Culture Wars in the Data Center: When the Security and Network Teams Collide

Last year, I had the pleasure of spending a couple of days with Yahoo CISO Bob Lord who, at the time, had yet to join Yahoo and was doing some security consulting at Gigamon.

Bob pointed out that many IT departments have a cultural disparity in the way they operate. He described network teams as operationally mature, highly disciplined, and often KPI’d on very strict and measurable metrics like network uptime and availability.  One could even argue, he added, that their primary adversary is, ultimately, the laws of physics. Though unyielding, these laws are not malicious. Network teams are constantly being asked to do more with less budget as what they deliver is seen as a business function and subject to the constant need to improve efficiency.

Security teams, on the other hand, face an unknown and unpredictable enemy.  An enemy that is often malicious, will respond to security measures by finding ways around them, and can and will hit every single layer of the OSI stack, right up to layers 8^[1], 9^[2] and 10^[3].  The result is an “arms race” between attackers and defenders, with new features and even toolsets continually appearing to address new threats.  KPI’ing security teams – beyond “don’t get breached” – can be a challenge in many organisations.  But as security threats resonate to the top level of most organizations, these teams are often well funded in comparison to other IT functions.

Differing Objectives and Polarizing Passions

On one hand, a network team is goaled on uptime, does multi-year planning and capacity measurement, has strong processes in place to prevent disruption, and is constantly seeking ways to do more with less.  The flipside is that these processes can make the team seem slow, risk-averse to outages and yet apparently unconcerned about security, hide-bound, and ridiculously conservative.  It’s not unusual to hear venom in security engineers’ voices when the three words “network change control” are uttered.

On the other hand, a security team is fighting an intelligent, agile, and constantly morphing threat.  To have a chance at defeating the enemy, they deploy tools and features that require extreme agility, especially in incident response situations. Despite teams sometimes having generous budgets, security tools are expensive, and assessing, deploying and operating the best is highly challenging.  And yet to the network team, this behaviour looks ridiculously over-responsive, lacking in strategy, suffering from a “tool of the month” or even “shiny new toy” behavior, and puts the availability of the network at risk.

As Bob explained, security people care passionately about preventing security breaches; network people care passionately about preventing outages.  It is very easy for either side to see the other as disregarding something that is, to their unique mindsets, core to the organization’s success. I have seen this happen many times at companies—varying from a good-humoured wariness between groups to outright hostility bordering on open warfare at its most negative.

The Gigamon Security Delivery Platform Enables Deployment without Disruption

The traditional way to deploy a new monitoring tool was to configure a SPAN port at some traffic choke-point and attach the monitoring tool to that. Of course, that’s leaving aside the fact that the tool was already compromised by being attached to a single point on the network, and the fact that SPAN ports do not perform well at high-traffic volumes.  It’s also leaving aside the fact that most security tools are also developed by security engineers and often lack a realistic model for just how nasty and messy real-world networks can be.^[4]  SPAN ports also require switch reconfiguration, which is potentially disruptive and, therefore, requires strict change management.

The presence of a Gigamon SDP, however, can facilitate the whole process.  Both TAPs and the SDP itself function like a data diode: traffic passes unidirectionally from the TAP to the SDP to the tool.  Even if the monitoring tool were to inject traffic, it would be dropped by the SDP’s tool port; or if the SDP itself were misconfigured, the TAP itself would also prevent injection of any traffic.  In this way, the SDP provides a very high level of assurance against introducing availability risk into the network.^[5]

As an aside, an SDP also improves the efficiency of selecting security tools.  Multiple competing tools can be run, side by side, with exactly the same traffic and without the extensive change controls between them.  This means that the comparison is genuine, with the same traffic, and no vendor can make the tired excuse “Oh, we’d have seen the attack my competitor caught, but it didn’t happen when we were attached.”  Side-by-side evaluations remove their ability to make excuses.  For more details, see the blog, “Hasten Proof of Concepts—and Time to Value—with Pervasive Visibility.”

What about Inline Security Tools?

Inline is seeing a resurgence.  Traditionally deployed at the edge, where network speeds were low enough to be processed, processing speed improvements in x86 CPUs combined with the use of high-end network processing units for extreme data rates have seen tools move from the edge to the core where they become much more valuable.

The concept of having an inline security tool in the core network worries many network people, and rightly so.  A malfunctioning inline tool could take the core network down.  There are also many potential failure modes for inline tools, some of which may not be noticed quickly by monitoring tools.^[6]

The inline capability of the Gigamon SDP de-risks inline tools for a network team.  The presence of a configurable positive and negative heartbeat allows the detection of many inline tool failures, with an automated remediation response to maintain mandated network availability.  The physical bypass capability of Gigamon’s BPS Modules means that even a powered-down system will fail to fibre or wire.  It brings availability assurance to inline network deployments.

The biggest win for a network team, however, is the ability to instantly take a malfunctioning inline appliance out of the network—either automatically or manually.  This returns the ability to control and eliminates much of the concern about inline devices.

The win for the security team is that it makes objections from the network team manageable and facilitates these highly useful deployments.  As it was for the deployment of monitoring tools, both sides see a win from an SDP deployment.

There is no right or wrong in this clash of cultures: It is simply different priorities and focuses that so often show network and security teams as competitive.

Too often, technologists focus exclusively on the technology and fail to understand the organisational and business culture.  The Gigamon SDP bridges these challenges, de-risking security deployments for the network team and speeding up network access for the security team. In doing so, it can also save money and facilitate operational efficiencies.