Postscript from an Unexpected (Threat) Hunting Trip

We always appreciate hearing from our customers, whether it’s important feedback on improving our products, war stories from the trenches or just saying hello. Recently a customer, in the midst of doing all three of those, told us how their security team had recently used Gigamon ThreatINSIGHT cloud-based network detection and response solution to discover and root out a longstanding anomaly that had been secretly persisting for over a year.

To respect everyone’s privacy, we’ll refer to the organization as A Gigamon Customer, or AGC for short.

A Troubling Discovery

The incident began when AGC’s security team was scanning through the output of a Gigamon ThreatINSIGHT dashboard query to identify non-sanctioned outbound email traffic. That’s when they spotted something that wasn’t just strange, but possibly a major security breach.

They were checking the Gigamon ThreatINSIGHT query’s results, going through its list of IPs. ThreatINSIGHT shows associated context, so the PDNS (passive DNS) feature gave them human-readable domain names. One IP’s domain caught their attention because it matched the last name of an employee…who’d left the company more than a year ago.

They ran a WHOIS, which linked the domain owner to a physical address, with a variation on that surname attached to an email address. With more investigation, they were able to correlate that the former employee had once lived at the address. This was enough information to safely assume the mail server was operated by this former employee, who would have had access to network and storage devices.

At this point the team had enough information to start taking action. The team leader consulted with her manager, opened an incident ticket and made a request for assistance from engineering services. As she wrote in the ticket, “Critical infrastructure should not be sending status emails to former employees.”

Investigating Further

The machine hosting the unidentified email server had “csg.nas-3” in its name. NAS infrastructure is critical, so if email traffic from a storage device to a former employee was taking place, they had to first assume the worst-case scenario, namely that sensitive corporate data was being exfiltrated.

Connecting to the device’s IP brought up the NAS web interface. Other information received by the team suggested that the “nas csg” in the machine’s name was probably an abbreviation for Custom Solutions Group, the department the ex-employee belonged to. Due to the nature of its work, this group was typically granted some leeway by IT to manage its own infrastructure.

Knowing this, the AGC team’s suspicions of malfeasance dropped considerably. Applying Occam’s razor, it was likely the employee had been using a personal address to receive status emails from this storage device, and simply neglected to stop them upon leaving the company. The relevant engineering and lab teams would still conduct appropriate forensics, but this initial investigation suggested no major breach had actually occurred. Whew.

Leveraging the Rules of the Hunt

How one ultimately uses a tool like Gigamon ThreatINSIGHT can be tightly coupled to the network environment under watch. As AGC explained to us, they have the interesting approach of using something like hunting patterns to search out threats. Indeed, this incident began while their team was engaged in a fairly-standard network grid search, looking for anomalies and doing discovery.

Threat hunting is not a universally recognized cybersecurity strategy, but this type of “search and rescue” technique is used often in the physical world to find a lost person or animal. Physical searchers need to be alert to certain behaviors. What paths will a missing person likely take? In what direction are they likely to head? There are general rules, such as people often follow downward slopes and turn in the direction of their dominant hand.

Similarly, threats and malware tend to have set behaviors, and as the AGC team demonstrates, we can use knowledge of these to optimize our hunting strategies. For example, developing a threat hunting processes that is tunable to the network environment and proactive detection measures can help to better secure your environment.

We all do this implicitly, but we also expend a lot of time and effort figuring out (and ultimately, codifying) specific guidelines. The grid-search methodology regularly employed by AGC’s team, leveraging tools like Gigamon ThreatINSIGHT, is just one example of a strategy evolved over time that fit the parameters of a given organization’s mission and infrastructure. In this incident it paid off, surfacing an anomaly that had gone undetected.

Curiosity + Gigamon ThreatINSIGHT = Mitigation

Though in the end this breach wasn’t malicious, our AGC customer was sure glad they caught it. It’s sobering, they noted to note, that the outside-email situation persisted for more than 400 days. The only reason anyone finally noticed (and then achieved comfortable mitigation after seven hours) was a little old-fashioned human curiosity combined with the Gigamon ThreatINSIGHT solution’s solid hunt-and-analytics platform surfacing the strange network activity.

They ended up happy with the ease of using Gigamon ThreatINSIGHT, and how it helped them keep a little bit ahead of potential badness. And it showed once again the importance of network visibility, that the network is the single source of truth for security, and why we really, really like getting feedback from our customers. Cheers to AGC for the war story.