Two sides of IT vs. OT Security and ICS Security Operations

Two sides of IT vs. OT Security and ICS Security Operations

People, Process, & Technology on Two sides of the Same Coin

Who should perform the security of the operations technology (OT) and industrial control system (ICS) in any given company? This is a question I commonly get, which is based on a need to identify the proper structure, governance, staffing, tooling, training, etc., of personnel in the organization. This question sometimes forms in the information technology (IT) vs. OT debate as to who “owns” the security function for the overall industrial environment. The intention of this blog is to share some of my insights and my thoughts on the classic debate, as well as recommendations for putting in place the right structure to support a security team focused on industrial security and how to find and build these professionals. My insights will largely draw from “what right looks like” in terms of dealing with targeted threats at Dragos and from teaching my SANS ICS515 class at the SANS Institute, to which I’ve now trained over 1,000 personnel.

The Core Debate (A Recap)

Right up front I want to note this debate isn’t new, and my focus on it is mostly going to be towards security operations and incident response. I posted a Little Bobby comic recently that got a few folks talking on social media about this debate again in a good way, so I wanted to add my thoughts beyond a three-pane comic. It’s useful to catch everyone up, though, on the core of the discussion. There are two articles by Dale Peterson on LinkedIn that largely capture the base argument between IT and Operations. I’ll capture a few of the highlights that I also agree with, which will allow me to build upon them for my points.

Article 1: Best Raw Material for an OT Security Team

Dale’s key points:

• Where you source talent (IT or Operations) for your OT security team doesn’t have to be absolute (should I train an operations person on security or a security person on operations?). The key is to find good people on either side. However, IT security background folks tend to adapt quicker to finding intrusions in OT where Operations background folks tend to adapt quicker to understanding the consequence of a cyber attack. In short, recruit from both when possible.
• The IT vs. OT debate is dying off. CEOs and boards expect their CIO/CISOs to understand and manage the cyber risk as it relates to the industrial networks and environments. There will have to be specialized IT teams to deal with OT in the same manner that specialized teams form if IT is asked to manage specialized equipment and software such as enterprise resource planning (ERP).

My additions:

• My favorite strategy is to pair people of different skills and backgrounds. When you’re dealing with complex environments and complex human threats, it’s important to have diversity of thought and expertise. One of the core things I tend to want for folks that will work ICS security operations is the ability to run an investigation instead of simply dealing with alerts. I will discuss that analyst journey below.
• I fully agree that the IT vs. OT debate is changing quickly. I expect this will still be an issue for many companies for years to come, but as a community, Dale’s spot on. There will be a dedicated team whether you want to call them an OT/ICS security team or IT security team with an OT focus; dedication is a theme I’ll hit again below.

Article 2: It’s Not OT v. IT

Dale’s key points:

• Finding skilled OT security personnel is going to be highly difficult and competitive (thus the focus on building your own), which is why he draws largely from IT security talent. Wanting a single person to have engineering, automation, IT, and IT security skills is a rare level of expertise that shouldn’t be how we measure everyone.
• OT is necessary. IT security processes and technology cannot be blindly applied to the industrial environment, and OT specialization is required to be effective.
• The main focus is not on “which side wins,” but more on who owns the governance piece. Operations will want to hold onto and control anything that impacts and affects their mission, but executive management is more aware of the cyber risks to ICS now, and they want their CIO/CISO leadership managing that risk.

My additions:

• Finding skilled OT security personnel is going to be difficult, and I agree with Dale that you cannot expect an analyst to know engineering, automation, IT, and IT security skills at a competent level to start. I think it’s important to have a keen understanding of the language and a core passion for the mission that’s taking place in engineering and automation; with that, the IT and IT security skills can translate quickly, and the other areas can grow.
• I completely agree that specialized process and technology will be required. As I’ll highlight below, when I have talked about OT security operations center (SOC) vs. IT SOC, I am referring to unique people, processes, and technology, not the idea that there has to be a new reporting chain, budget, or facilities.
• On the governance side, one of the biggest clashes I tend to see is not that CIO/CISOs are expected to manage the risk in Operations, but that the manner to which they do is often through the lens of standards, regulations, or best practices. Most of the time these are entirely just IT security practices copy/pasted into the ICS. This can lead to massive friction and form an IT vs. OT divide that does not need to exist. I’m good with the CISO advising on the cyber risk in the industrial networks (I don’t think it should be the CIO), but they must take more consideration on their customers’ needs, as Dale noted. I also don’t think the CISO should “own” the risk so much as advise on it and report to the CEO and board about it–but there’s a thin line between what works and what doesn’t for different organizations.

Extending the Discussion to Security Operations

The main points above all apply, but a little more guidance is useful when considering a security operations team for OT/ICS (I mostly just refer to these as industrial networks or industrial environments for simplicity’s sake). The industrial environment has different systems, mission requirements, risks, and cyber threats. Often a lot of focus is placed on the systems piece. “Well, they’re just Windows systems” has been a common push back against ICS-specific threat detection, as an example. I would argue that it’s not just a Windows system anymore when it’s running engineering software or mission-critical applications and ICS software such as a Human Machine Interface. How the system operates, what it does, the impact of its compromise, what you can do with it, what an adversary has to do to accomplish its goals, etc., all change, so breaking down the debate based on operating systems to me is naïve. However, it’s a moot point, because the prevalence of IT systems in ICS or OT is not the focus to me. The point is there is a unique mission and unique risks with unique cyber threats. Therefore, no matter how much convergence takes place in the form of technology, the reality of these environments is that they demand a unique security focus. Related: Facility Managers Guide to Building Systems and Cybersecurity

Ben Miller and I wrote a paper a couple years ago exploring the differences between an IT and OT SOC. One of the chief points we made was that an OT SOC was required, due to the unique mission and risks. It was also discussed that the structure of the OT SOC, in many ways, is flipped where your Tier 3 personnel are not necessarily your highest trained security personnel centralized together to deal with significant cases, but instead are decentralized and closest to industrial operations with the most knowledge of them. These Tier 3 personnel could be members of operations who act as the “champion” for security, or they can be (and if possible, should be) OT SOC-dedicated members who have the best understanding and relationships with the operations staff. As an example, you might have a Tier 3 OT SOC member who really understands offshore drilling for an oil and gas company and has the right connections and relationships at the facilities. This person is still expected to really understand security and still contribute to the daily tasks of alert triaging and analysis, but may also spend time working with operations to position them better against future threats or help tailor collection, so that the OT SOC gets the visibility required for the type of threat detection and response they want to achieve.

Thus, do you need a dedicated OT SOC with its own budget, reporting chain, separate room, and lots of big screens? No, that’s not what a SOC is. A SOC is simply people, process, and technology focused on security operations. It can have its own budget, but it might not. The reporting chain will likely be the same. You must absolutely sit them with or near the IT SOC, so that visibility and collaboration across the organization can take place. But the people, process, and technology you choose for the OT security operations mission is going to be different. You are going to look for different skillsets and training, you are going to measure and incentivize the people differently (the number of tickets opened and closed would be a horrible core measurement of an OT SOC analyst, as an example), and you must have technology that is well-suited for industrial networks–not only on the ICS protocols and communications, but also in the context of what it’s looking for and how it enables investigations for the analyst. (Cue a  Dragos Platform pitch).

The Analyst Journey

The analyst journey for industrial security operations is also different from the analyst journey for those working a classic enterprise security operations mission. It is a common occurrence that in an enterprise SOC you have analysts broken across Tier 1, Tier 2, and Tier 3 that have functions and tasks that would not be effective or appropriate for the industrial networks. As an example, right or wrong, in a decently large enterprise SOC, it would be completely routine for analysts not to have enough time to dive deep into every intrusion they encounter. Therefore, it would be appropriate on lower-risk items to simply activate the endpoint protection system to clean up a host or to reimage the system entirely. In the ICS, that could pose significant risk if done to the wrong systems or in the wrong time window. It is entirely reasonable that there are known systems infected in the ICS that an analyst determines isn’t causing risk, so they determine not to fully address it until there is a maintenance window.

The analyst journey in an enterprise SOC will facilitate, and sometimes demand, actions based on alerts without an investigation. The analyst journey in an OT SOC requires that the alerts be properly investigated to determine what is going on. This investigation is done across multiple data sets. In fact, industrial networks have more data collection opportunities than enterprise networks when factoring in controller events, alarms, and historian data. Thus, a core skill for an OT SOC member is the ability to ask a question of the data and follow it through an investigation across disparate sets of data. This is a great skill to have in the IT SOC, but it is usually reserved for higher tier analysts and is not always required.

The people, process, and technology required to facilitate the analyst journey in industrial security operations in comparison to enterprise security operations–inclusive of the different mission and risks–will dictate the need for an OT-specific SOC. This isn’t about more organizations or even bigger budgets. It’s an appreciation that if your organization is going to add another mission on (OT/ICS security), it’s going to need more resources anyway. The correct structuring of those resources and the appreciation of the analyst journey in industrial security operations will determine the success of the effort. To learn more about insights, strategies, and recommendations for building the appropriate ICS security approach for your specific organization, learn more from my 2018 ICS Year in Review whitepaper: Insights To Build An Effective Industrial Cybersecurity Strategy For Your Organization

 

Reference documents:

• SANS ICS ICS515: ICS Active Defense and Incident Response https://www.sans.org/course/industrial-control-system-active-defense-and-incident-response
• Little Bobby http://www.littlebobbycomic.com/projects/week-221/
• Best Raw Material for an OT Security Team
• It’s Not OT v. IT
• Insights into Building an Industrial Control System Security Operations Center https://dragos.com/resource/insight-into-ics-soc-pdf/
• Dragos Platform

 

This article was written by Robert M. Lee and originally it was published here. Robert is a recognized pioneer in the industrial security incident response and threat intelligence community. As a non-resident national security fellow at New America, Robert works to inform policy related to critical infrastructure cyber security and is regularly asked by various governments to brief to national level leaders.