Instructor: Incident response planning.
In this lesson, we're going to discuss
some of the different portions of an incident response plan.
We're going to talk about things like
the incident response team, communication plans,
and stakeholder management.
Let's get started with incident response teams.
Now, what exactly is an incident response team
and who is on that team?
Well, your incident response team
is key people that are going to be available
to respond to any incident that meets the severity
and priority thresholds that are set out
by your incident response plan,
because not everything that you run into
is going to require you
to activate your incident response team.
Some things can just be handled by your incident handlers
and you don't need the full team to do it.
But if you have a big issue,
like an ongoing data breach or something like that,
you're going to want the entire incident response team.
So what type of positions
are on this incident response team?
Well, first you're going to have
an incident response manager or team lead,
this person is going to oversee and prioritize actions
during the detection,
analysis and containment of an incident.
This is a position
that I have personally filled numerous times.
And I can tell you, it is a difficult position
that requires a lot of good soft skills
in addition to those traditional in depth technical skills
that some other positions are going to require.
This is because your incident response manager or team lead
is going to be responsible for conveying information
about the response and recovery efforts
to the executives and management within your organization.
And oftentimes, they also could be thrust into the role
of being the public face of the company to the media
or law enforcement during an incident response.
The second position we have is a security analyst.
Your team needs to have one or more security analysts
assigned in order to work directly on the affected network
and to play detective in order to determine
what happened up to this point.
Your security analyst may be assigned into two categories,
although some analysts may be working
in both categories simultaneously
when dealing with a smaller scale incident.
The first of these is known as a triage analyst.
A triage analyst is a security analyst
that's assigned to work on the network
during the incident response.
Triage analysts are going to help filter out false positives
by properly configuring intrusion detection
and protection systems,
as well as performing ongoing monitoring and analysis
to detect any new or potential intrusions
during your incident response.
Another type of security analysts we use
is what's known as a forensic analyst.
Now, a forensic analyst on the other hand,
is going to be more focused on the detective work
and trying to piece together
what has already occurred on the network.
They're going to focus on recovering key artifacts
and evidence from the network and then use these
to build a timeline of the different events
that led up to the incident itself
and that way we can understand
what happened up to this point.
Beyond that, you're also going to want to have
a threat researcher, this is another key part of your team.
These threat researchers are able
to compliment your analysts
by providing threat intelligence, and overall context
during your incident response.
These specialists work to always remain up to date
on the current threats that are facing your organization
and your specific industry, as well as keeping up to date
with previous incidents that may have occurred.
I like to think about these folks
as both a combination of a futurist
in terms of guessing what the bad guys might do,
as well as a historian because they know all the bad things
that the bad guys have done in the past.
By using this they can help us
build better security and defenses,
as well as trying to get one step ahead of the bad guys
who've already broken into our network.
Finally, we have cross functional support.
In addition to all the critical roles
I already talked about,
we also want to expand our team
with additional cross functional support.
This includes people from management or the executive team,
somebody from human resources,
if you're dealing with an employee insider threat,
or an attorney or lawyer in the case that the company
may want to take legal action
against the perpetrator or the attacker.
Sometimes you might even have somebody
from public relations, because you may have media interest
into the incident as well.
In addition, all this,
you may have to pull in technical experts
on specific systems, like system administrators,
network administrators, or database administrators
to help you recover back to normal operations
as part of your response.
All of these are considered cross functional support,
because they're coming from outside
of the incident response team itself
and across the entire organization.
Now, this incident response team is often known as a CSIRT,
a CSIRT is the computer security incident response team,
and your CSIRT should be the single point of contact
for security incidents.
Now, the CSIRT may be part of the SOC,
the Security Operations Center,
or they could be an independent team.
It just depends on how your organization has set this up.
In fact, some organizations have chosen to outsource
their security response and their CSIRT teams,
this way whenever there's an incident
they would call on this third party contractor
who will bring their experts
to help you bring your systems back online
and get the bad guys out of your network.
Now, regardless of which way you decide to do it,
it's important to realize that being a part
of an instant response team is a 24/7 job.
Instant response teams
will typically require 24/7 availability
and because of this,
this can become a very expensive thing to provide.
It's also important for you to consider
how you're going to rotate out members of your CSIRT,
because they can get burnout being on call 24/7.
If you happen to work
for a small or medium sized organization,
it's most likely the case that you're going to have
an outsourced CSIRT team,
because it's really expensive
to have your own CSIRT team for your organization.
So really, unless you have a really large organization,
you're probably not going to have a dedicated CSIRT team.
Because the expense of maintaining this
and having them ready to run
at any time 24/7 can get very costly.
The other benefit of using an external team
is these external agents can often deal more efficiently
with the different problems
that we have inside of our networks.
For example, if you're investigating an insider threat,
using a third party team
that's coming from outside your organization
would actually be better,
because they don't know all the people involved
and they will have a clear eye
to look through all of the logs
and try to figure out exactly who the insider threat is,
as opposed to having bias
based on knowing these people personally.
Another thing we need
to consider during our preparation phase
is our communication plan.
How are we going to make our communications
back to home station or up to leadership
in the event of an instant response.
This is extremely important,
especially in larger organizations
where have a distributed workforce
around the country or around the world.
Your teams have to have a secure method of communication
for managing your incidents.
Now, this is really important,
because if you have a VoIP system for instance,
and that's what you're going to use to communicate,
well, if an attacker is in that VoIP system
they can hear everything you're saying.
And they will be tipped off
as to all the things you're doing
and they can pivot away from you
and gain access further into your systems
before you can get them out.
And so often what we want to do
is use an out-of-band communication system.
Now, an out-of-band communication system
is one where your signals are being sent
between two parties or two devices
that are sent via a path or method that's different
from the primary communication
between those two parties or devices.
Now, that's a really complicated way of saying things,
but let's say, normally, you would go ahead
and report to your manager
using your corporate email account.
Well, if you think that corporate email
has been compromised, you don't want to use that
as your communication method.
Instead, you want to use an out-of-band communication,
that might mean you're going to make a call
from your cell phone to their cell phone,
or you might use an encrypted path.
For example, one of the most commonly used ones
is something like WhatsApp, or Signal, or Off the Record.
All of these apps have messaging systems
with end-to-end encryption
so no attacker can see the information
in between that's being sent.
And so oftentimes we'll have a channel set up
on WhatsApp as a way for us to use
as a communication mechanism.
So what is your backup communication plan going to be?
Well, you have to consider how you're going to communicate
with the people who are on call,
especially when you have a distributed workforce.
For example, one organization I worked at,
we had a team that was actually flown out to a center
that was dealing with an incident.
Now, that center they weren't allowed
to have smartphones inside that organization
and so when they went into the building
we couldn't communicate with them.
So we had to have a way for that team to respond to us,
even though they couldn't have smartphones in the building.
And what we ended up coming up with
was that every hour somebody would leave the building,
go to their car, get their smartphone,
and then call in to check with us.
This is a way of doing things.
But again, it was our backup plan
that if you can't do this, then you should do this,
and every hour send us a message.
That was one of the ways
we were able to work our way through this.
Maybe you're going to send them to an area
that doesn't have good cellular coverage.
So instead, they're going to have to use a satellite phone,
or some other method like that.
You're going to have to think these things through
for your organization based on your own incidents
and your own response plans.
Another thing you want to make sure you're doing
is always make sure you maintain an up-to-date contact list.
The best way to prepare for your team's activation
for a future response
is to ensure you have an accurate contact list
that is prepared ahead of time.
This will include things like their phone numbers
and email addresses
for each member of the incident response team,
as well as others within the organization
and your third party partner organizations.
By ensuring you have an accurate contact list,
that is ready to go, you're going to be better prepared
when you receive the message
that your team needs to be activated
and in the case of people who have to fly away
for a remote response, you're ready to go
because you have all the information you need.
Another part of your communication plan
that you have to consider
is what is your escalation procedure going to be
for your organization?
Basically, at what point should you call
the on call person or team?
What exactly defines when the event or incident
is bad enough that you need to call somebody
and wake them up at three in the morning,
instead of waiting till they show up
at work at 8am the next day.
This is especially important
if you're going to use contract personnel.
Because when you call in contract personnel after-hours,
this can actually have huge additional charges
per the contract.
So you need to make sure it's clearly outlined
based on your organizational priorities
when you're going to call somebody in.
If somebody's email was hacked,
you may not call them in for that.
But if your credit card system was hacked,
you may do that for that because it's protected information.
These are the type of things to think about
and make sure you have the right policies
and procedures are in place.
Now, another thing we have to think about
is how are we going to notify people.
Based on the prioritization
and categorization of an incident,
we're going to have different levels of notification
from none to an email to a 3am wake up call.
If it's a minor incident that's well understood
and considered to have low or no priority,
then our organization may have procedures in place,
so the incident handlers and security teams
will simply take care of the incident
without telling anybody about it.
If we have a more serious incident though
we're going to have to follow our procedures
to notify the right individuals up our organizational chain.
This may include people like the chief information officer
or the chief security officer,
or the chief information security officer,
the incident response team members, the system owners,
the system administrators, our point of contact
within human resources,
a legal department representative, or even public affairs,
maybe we even have to call law enforcement.
All of this and the exact methods
that you're going to use for these notifications
will be based on your organization's procedures
that are going to be developed
during this preparation phase
of the incident response lifecycle.
Now, in addition to figuring out who you're going to notify,
you have to know how you're going to notify them.
This can include using email, internal web portals,
telephone calls, an in-person update,
leaving a voicemail, setting up a formal report
or other forms of notification
that your company decides is desirable and effective.
In most organizations I've worked at in the past,
we've used a phone call or an in-person notification
for urgent and high priority incidents,
and then we follow it up with an email or report.
For a medium priority issue,
we generally will rely on an email, a voicemail,
or possibly even a phone call if it's during working hours.
If it's a low priority incident,
then we may handle this through a daily or weekly report
or through an internet portal.
This is definitely an area where you want to tailor it
to suit the needs and desires of your organization.
So you need to make sure
you're involving those who are going to be notified
and ask them what method works best for them
and for which categories they want you to use it.
The final thing we need to talk about
when we're dealing with communication plans
is how far out do you want this information to go.
And what I mean by that is we want to make sure
we're preventing unauthorized release of information
outside of your CSIRT.
This is because we don't want
this information prematurely hitting the front page news,
it is not helpful for an incident to be publicized
in the press or through social media,
outside of your planned communications.
Anything that's going to go outside of the team
needs to go through the appropriate parties
and through public relations.
You also need to ensure that parties
with privileged information do not release this information
to untrusted parties,
whether intentionally or inadvertently,
remember, if something is going to get out onto Facebook,
or Twitter, or onto Google News,
or any other news site, you want to make sure
it's done appropriately through the organization
with your public relations team
and making sure that they are in the know
and that they're the ones who are doing that.
When you have an incident
you need to start thinking about
who are the affected stakeholders?
There are lots of them out there,
inside and outside your organization.
For instance, you might have senior leadership
that gets involved, you might have regulatory bodies,
you might have your internal legal counsel,
or external law enforcement.
It might be your internal human resources
or your internal public relations
and externally with the media.
All of these are people who are valid stakeholders
when you have an incident.
And you have to consider
how does this incident affect them?
And how are you going to coordinate your response?
Let's go ahead and look at each of these.
First, we have senior leadership.
When we talk about senior leadership,
this is the executives and managers who are responsible
for business operations and various functional areas
within your company.
Now, the reason this is important
is because a lot of our incident responders
tend to be technical people.
And so we might as technical people say,
the quickest way to solve this incident
is to shut down that server.
But if we're not understanding
the business impact to those actions,
that could have second and third order effects
that'd be very bad for our organization.
So we're going to have to get senior leadership involved
to understand if I do this,
it's going to have this and that and the other effect,
and we have to mitigate those.
For example, if your credit card processing system
has been compromised, if you immediately shut it down,
you are cutting off your ability
to process new transactions.
Now, that might be the right answer technically,
but from a business standpoint
that could actually hurt you even worse.
And so you have to start weighing these factors
and working across your organization
to make sure you have another way to accept payments
before shutting down that credit card system.
Or maybe you're going to make the decision
that it's okay to shut down the system
but you understand you're going to be giving up the ability
to process credit cards right now.
That is a business decision not a technical decision
and so it is one you have to have
senior leadership's buy in on.
The next key stakeholder we have to consider
is regulatory bodies.
These are governmental organizations
that oversee the compliance
with specific regulations and laws.
For example, if we're talking about HIPAA,
which has to do with health care,
you're going to have to be overseen
by Health and Human Services,
because they're the ones who run the HIPAA program.
If you're dealing with something like
the California consumer Privacy Act, or CCPA,
you're going to be dealing with the state of California.
If you're dealing with something like credit card data,
you're going to be dealing with PCI DSS
and those people who run that program.
Again, these are the different regulatory bodies
you're going to have to consider.
Now, a quick note, when you're dealing with PCI DSS
they're not technically a regulatory body
from a legal standpoint, but they do oversee
all of the payment card systems.
Now, generally, when we talk about the term regulatory,
we are talking about legal,
and with PCI DSS, it is not a legal requirement
it is a contractual requirement
between you and your payment processor.
The next stakeholder we have to consider is legal.
Now, legal is the business or organization's legal counsel,
and they're going to be responsible
for mitigating risk from civil lawsuits.
For example, as you're planning out your response
of what you're going to do to stop the breach of data
you want to make sure legal is in the room
because your actions could come up later on
if your company is sued for its response.
And so you want to make sure they're in the room
and they understand what you're doing,
and they have input into it.
On the other side of the coin,
we have law enforcement,
and law enforcement is an external stakeholder,
they may provide services to assist
in your incident into handling efforts,
or to prepare for legal action
against the attacker in the future.
Now, one quick thing to note,
your decision to involve law enforcement
has to be made by senior executives
with guidance from your internal legal counsel.
You as an incident responder
should not immediately pick up the phone
and call the FBI or the local police,
this is something your business has to decide.
Now, there are cases where it is legally required
to bring in law enforcement.
But in a lot of cases, it is more of a civil issue.
And you have the determination
and the right to decide if you want to press charges
and bring in law enforcement.
So keep that in mind.
And remember, your senior executives
get to make that decision.
Our next stakeholder is human resources,
and this is an internal stakeholder.
They're going to be used to ensure
there's no breaches of employment law
or employee contracts during the incident response.
For example, if you're a suspecting
that there's an internal threat,
and you start questioning employees,
or you want start going through employee files,
you're going to have to consult human resources,
because you could be breaching employment law
or employee contracts,
so make sure you involve human resources.
And our final stakeholder we want to consult
is public relations or PR.
Public relations is used to manage
the negative publicity from a serious incident.
Now, this is important
because you want to make sure as the technical lead,
as an incident responder
you're not the one answering questions to the media.
You don't want to be the one up there,
behind all those microphones
with a sea of reporters asking you questions.
You have people in your organization
whose job it is to handle that.
And they're going to come up with a clear, concise,
message that can be said over and over again
to all the inquiries reporters have.
This way you're on brand
and on message across the organization.
Remember, public relations needs to be involved,
especially with larger breaches that may get the interest
of media and the press.
Now, as part of the CSIRT team,
it's going to be your role to help provide information
to these stakeholders.
As part of the CSIRT,
you're going to be asked for information
regarding the estimated downtime,
the scope of the systems and data affected
and other relevant details.
And by having that information
and providing that up to the appropriate senior leadership,
human resources, legal, public relations and others
it's going to make sure that we all are providing
a consistent message and that we're coordinating
our response efforts.