Breached Whale: Optimizing Incident Response to Get Your Business Swimming Again

Welcome to the Thunderdome

Sometimes I hate the term incident response. I get the feeling people think we’re just sitting around waiting for stuff to happen and, in turn, that’s what they do. The amount of planning that goes into preparing for an incident should be just as monstrous and well-resourced as the actual execution of a response to an incident.

The difference between a well-constructed, well socialized, well-practiced incident response strategy is being ahead of the headlines rather than being the headline. The purpose of our lesson today will be how to stay the former and avoid the latter. Where do we begin? By studying what happened to everyone else without a decent security incidence response program.

One of the greatest resources for incidence-response lessons learned is the Verizon Data Breach Investigations Report. It’s published every year to the public for free, and it’s a gold mine of statistics about breaches from across the industry.

Below is one of my favorite graphics from the DBIR. It highlights the disproportion between how quickly an incident can occur and the time it takes to respond. On average, it took 87 percent of the compromises just minutes to gain a foothold, and two-thirds of those went undiscovered for months. Keep in mind, that means months just to discover anything happened at all and doesn’t include the time it took to remediate the incident, during which the rats in the nest kept feasting.

It’s easy to get lost in all the vendor hype when it comes to protecting your environment. Some vendors will try and hit you with a tag line like, “Advanced threats demand advanced response.” Why should your aspirations of incident response capabilities be hung on advanced threats?

If you’re reading this article, wondering how to even define what incidence response looks like to your organization, chances are advanced threats are a long way off in regard to your detection capabilities. Do you know what will kill you before you even get there? Common vulnerabilities that have gone unpatched in your environment. The low-hanging fruit that can be exploited by someone with little to no technical skill, but access to stolen credit cards used to pay for a piece of malware someone else only slightly more skilled wrote for them.

No, we’re not talking about advanced threat actors here — yet. We need to start not with a definition of IR, but with a philosophy. It’s not “if” it’s “when.” When your preventative tactics fail, and they inevitably will, you’ll have yourself an incident. How will you detect that an incident occurred? How will you respond? What will the remediation steps be?

Incident response is the capability to determine the scope and nature of an incident in order to mitigate the impact on your organization. More succinctly, figuring out what computers the bad guys are on, what they are using them for and how to stop it.

Everyone’s Got a Plan Until They Get Punched in the Mouth

We have the What, we have the Why, now we move into the How. How do you even start to respond to an incident?

I travel a fair amount every year. Same airports, same seats, seemingly the same crying baby just an inch to the left of my last good ear drum. However, until recently I’ve never really paid attention to the safety instructions before takeoff. I figured “What are the chances it will happen to me?,” followed by “I’m sure someone will tell me what to do when it happens.” That second one is particularly stupid because I’m assuming that I’ll even know what the circumstances will look like when I need to know the information to get through it. Screaming and on fire is a poor time to figure out logistics.

Response starts before a single bad packet has traversed your network. The key to running a successful response is in the planning. Who are the people deputized to handle it? What tools will they use to scope, contain and remediate? What’s the game plan to get through all those steps not only quickly, but thoroughly? Incidentally, quick and thorough can often be at odds between what the incident response team needs and what management wants.

Nowhere is this point articulated more clearly than when you have confirmed evil on a box. It’s natural to want to pull the plug immediately. Every time you find a box that was touched by the adversary, yank it off the network, set it on fire, and move to the next one. This is most certainly quick(ish) but it’s far from thorough. You’re just asking for the bad guys to come back in some other way you haven’t seen yet. If you’re not taking steps to squeeze every last bit of intel out of these incidents, your program is failing. You will never get higher fidelity pieces of intel than what happened in your own back yard.

Teamwork Makes the Dreamwork

A good security incident response plan is not unlike a good disaster recovery plan. You hope you never have to use it, but you also need to have a plan and test it. Your team will consist of not just the hands-on-the-keyboard, battle-tested analysts. You need to reach across and down the aisle to bring all your stakeholders in. Do you even know who these people are and how to get them on a conference bridge at 3 a.m.?

Managers, coordinating team leads, who coordinates incident phone bridges, which security analysts research threats and reports back up the chain. Do you have a complete, updated network diagram at your disposal? (Hint: you don’t.) You’ll need to loop in the networking team.

Once the scope is established and you determine how much of the building is on fire, it’s time to talk to legal. Are you a publicly traded company? Anyone have the number for public relations? This all gets complicated before the first SIEM alert ever fires off and stays complicated even with the best incident response plan. Imagine the utter chaos that would ensue if you never prepared any of that information. Your incident response plan needs to be socialized to everyone I just mentioned before it ever needs to be enacted.

Tool Time

We’ve got our army, now we must arm them. Well, we should’ve armed them a long time before the gates came crashing down, but now’s not the time to point fingers. That comes later during the reporting phase and passive aggressive email chains with liberal use of the Bcc: line.

A large part of a solid security incidence response plan revolves around finding answers. How many assets are affected? Are some assets compromised, but sitting silently as a secondary backdoor? Are there multiple C2 channels? When did all of this start? If you didn’t already start to capture things like endpoint and network metadata, and application logs from every living, breathing thing on your wires then this first lesson in “How Not to IR” is going to be a painful one.

We don’t live and die by indicators of compromise (IOC’s) here. We deal in behavior. We live by analytics. We need to see what came before and what came after, so we can see what’s coming later. A great, properly deployed and tuned tool arsenal begets great visibility. The cool thing about that is it’s the closest thing we can get to being actually omniscient.

Get Some Insight

I’d be remiss if I didn’t take a beat here to tell you how Gigamon Insight can help give you some much needed visibility during a time when there is fog all around you. This isn’t your average “just buy our stuff” pitch. I’m including Insight as an option for your toolset because, as a career analyst, I adore it and think more people should know about it. Insight’s capabilities live in the detection, hunting and packet-level forensics space.

At Gigamon, we’re proud of the fact that we only partner with top-tier security consulting firms. Insight has come to be their preferred network detection and response technology that they deploy during incident response and compromise assessment services.

These consulting teams deploy Insight for broad network-level visibility, covering both North/South traffic (inbound and outbound) and East/West traffic (internal). We’ve talked a lot about the need for that visibility during this blog. We’ve also talked about behavioral analytics and the need for thorough and reliable intelligence capabilities.

During an incident response engagement, you not only need every one of these things, you also need them quickly. Insight works by rapidly deploying fully managed sensors that can validate network traffic you expect to see and reveal traffic that you don’t expect to see. These sensors are easily managed and monitored for their health status.

Forgetting all the technical bells and whistles for a second, the reason these partners continue to choose Insight is due to the ongoing support offered by our customer success team comprising actual subject matter experts in the areas of threat hunting and incident response.

Another differentiator is the high-quality detection capabilities generated by Gigamon Applied Threat Research (ATR). Their sole purpose in life is to gather actionable intel about threats facing your environment and put that intel to use inside Insight. Not just once a quarter, not just every release cycle. They do it constantly and consistently because that’s how the bad guys move.

Simply put, Insight was made for analysts by analysts.

A Failure to Plan Is a Plan to Fail

My mom used to tell me that bones get stronger after a break. Then I Googled it and discovered she totally made that up, and I felt betrayed. Then I Googled some more and found she wasn’t totally lying but she wasn’t fully informed, which explains the whole Santa Claus thing.

When a bone heals after a break it goes through three phases (McVean, 2018). The reactive phase when the area around it starts to swell. Then the reparative phase when the membrane that covers your bone converts itself to bone and cartilage cells, and the remodeling phase where that temporary bone is replaced with compact, permanent bone.

It’s during the reparative phase that callus forms at the site of the break, making it temporarily stronger than it had ever been before. Eventually this all evens out, and you’re back to where you started, no worse for wear but not exactly Wolverine-strong.

Incident response follows a similar pattern. During our reactive phase we realize our control bones have been broken — that swelling is the 4 a.m. phone call or the article in TechCrunch that nobody wants to see. The reparative phase is when you start cleaning up the mess and patching the Kool-Aid Man-sized holes in your living room. Finally, the remodeling phase is taking those hard lessons learned and applying them so that 4 a.m. call doesn’t happen again.

Here’s the thing though, unlike a broken bone we have the ability to actually get stronger than we were. We are not constrained by physiology and the rules of nature — we can literally be Wolverine. Install more bare metal, do your reps to build stronger muscles around the bones, which is to say, practice what you preach. If you have a plan for failure (and practice that plan), you’ll be less likely to have a plan that fails.

You really want to know what the secret is to using incident response to optimize your business? Have any kind of solid incidence response plan at all; one that’s actually been tested. You’ll already be ahead of most other companies out there that have no idea how their own seat turns into a floatation device.