GUIDANCE: Incident Response Planning for Industrial

Control Systems / Operational Technology - Meet Admin

Corp
Introduction

This article is designed to be read in conjunction with the ICS COI guidance article - Considerations
for Cyber Incident Response Planning within Industrial Control Systems/Operational Technology, in
addition to NCSC’s principal based incident response guidance.

If you are responsible for the management or maintenance of Industrial Control System (ICS)
/Operational Technology (OT) assets, this article is designed to support you in adopting a mature
cyber incident response planning process that covers ICS/OT environments.

While cyber Incident Response Plans (IRPs) should cater for both Information Technology (IT) and
ICS/OT systems, consideration must be made for the key differentiators found in ICS/OT. So, in this
article, we're going to walk through specific concerns in an ICS/OT cyber IRP. We have done this to
help those with ICS/OT environments and assets best understand how good cyber incident
response planning should be conducted, and how it can be very different in certain circumstances
from an IT cyber incident response planning process. The incident response plan also takes into
account elements needed to recover also from the incident.

Having an effective cyber IRP in place also supports several Outcomes within the NCSC’s cyber
Assessment Framework (CAF). A summary of the related Indicators of Good Practice (IGP)
outcomes covered in this guidance is shown at the end of this article.
Meet 'Admin Corp'

Let's imagine we're following a fictional organisation who are responsible for managing the cyber
security of a CNI processing plant.

Admin Corp runs a plant that produces Adminox, a highly volatile, refined form of administrative
paperwork that is essential to every organisation in the country. It is created from volatile raw
products using a continuous chemical process.

As an essential service, Admin Corp must comply with the UK NIS Regulation. This means that
Admin Corp's assets needed to produce Adminox must be protected from cyber-attack. Also,
because Admin Corp are regulated for safety by the UK Health and Safety Executive using OG86 ,
they must take steps to ensure the continued cyber Security of the Adminox production process.

The process for producing Adminox involves several steps, with the final product stored under
pressure in a tank. Admin Corp's system, therefore, has two critical non-functional requirements:

• As a responsible and safety regulated company, they need to keep the local environment
safe from release of Adminox.
• Maintaining the availability of the product for customers in order to continue as a profitable
company.

We will look at how Admin Corp applied NCSC’s and the ICS COI’s guidance for cyber incident
response planning within their ICS/OT environment.

The Adminox processing plant has a fairly typical arrangement for this type of facility:

• Process Control System (PCS) network. This network hosts the main ICS/OT for the
Adminox process. It comprises primarily of Programmable Logic Controllers (PLC)s that
automatically operate the process, with some Human Machine Interfaces (HMI)s providing
operators with localised viewing (as clients of the site Supervisory Control and Data
Acquisition (SCADA) system, while others provide independent control of individual
process areas. The network is an Internet Protocol (IP) based network compromising of
managed Ethernet switches. The network is logically separated into Virtual Local Area
Networks (VLAN)s to allow segregation of individual process areas.
• Control Room. This provides a centralised position for operators to control and monitor the
plant using a SCADA system comprising of servers and workstations. The SCADA system
operates in a dedicated VLAN.
• Business Network. This hosts IT infrastructure that is not involved in the production of
Adminox; however, has connectivity to the control room and PCS networks to allow:
o Consumption of process data by IT systems to support business operations.
o Verified system updates to be passed from IT systems to the ICS/OT infrastructure.
 • Network firewalls. These controls the data connection between the various ICS/OT VLANs,
including between the control room and remote sites, in addition to between the ICS/OT
and IT environments.
• Safety Instrumented System (SIS) network. This network is physically isolated from the
control room and PCS networks, and ensures the plant reverts to a safe operating condition
should a dangerous fault condition occur within the processing plant. It does, however,
have a data-diode on the network allowing outbound event data to be sent to the SCADA
system, but without any inbound communication paths that could be used as an ingress
point for threats.

Importance of ICS/OT incident response planning

Admin Corp’s cyber IRPs are designed to incorporate step-by-step procedures the organisation can
use in one of the most stressful situations their teams will face together. A detailed ICS/OT cyber
IRP prepares responders in reducing impact, collecting evidence, remediating compromise, and
quickly restoring systems and operations back to their normal conditions.

The standard practices they have for cyber incident response with their IT environments form a
fundamental baseline. However, critical parts of protecting ICS/OT assets differ drastically and
considerations were made for immutable secure backups, leveraging logs, and detecting malicious
activity in the ICS/OT systems.

Most critically responding to cyber incidents in ICS/OT environments has a high probability of real-
world impacts to the environment, equipment damage, and extended operational impacts that will
directly affect revenue streams.

A properly developed ICS/OT and IT cyber IRP will not only prepare the organisation to respond to
security incidents but realistically the proactive steps taken to build and test these capabilities
could also help detect, and restore the operations, lessening the impact from security incidents.

Considerations for cyber incident response planning in ICS/OT

Admin Corp needs a comprehensive cyber IRP that includes consideration for responding to
incidents within their ICS/OT environment. Critically the cyber IRP needed to take the following into
consideration:

• Identify and classify the IT services and applications running in their business network
which are critical to keeping ICS/OT operations/production running in the event of a
compromise within the IT environment.
• Identify connections to the business network, IT environment and external systems that can
be safely disabled to protect the ICS/OT environment from threats in those systems.
• Include preventative measures to reduce the risk of operational disruption or functional
degradation to necessary capacity.
 • Identify ICS/OT assets that may not be able to be restored without vendor support or
possibly require replacement.
• Detail of agreement with vendors for the level of support Admin Corp would expect during a
cyber incident.
• Identify ICS/OT logging and monitoring capabilities within ICS/OT environments that will
enable detection, response, and remediation of cyber incidents. Examples of these events
to be documented within the ICS/OT Cyber IRP.
• Ensure processes and procedures include operational and engineering teams with critical
knowledge of the Control Centre, PCS and SIS systems.
• Potential physical impacts due to cyber incident (fire, hazardous spill, explosion, etc.) to
provide collaboration with Health, Safety, and Environmental (HSE) and physical response
teams and resources that may have to be engaged in the event a cyber event/incident
transcends to a physical event/incident.
• Agreement which procedures/policies come above others, for example is any cyber IRP only
enacted after all safety policies have been enacted and the site made safe.
• Physical incidents (fire, hazardous spill, explosion, etc.) could be a result of an unknown
cyber security incident and cyber security teams would need to be considered as part of a
physical incident investigation process.
• Create situational exercises to test the effectiveness of procedures, and personnel
responsible for implementing measures. Include executive, technical, operations, and
engineering resources.

What part of an IT cyber IRP is relevant in ICS/OT?

Admin Corp is able to leverage the structure and processes from their IT cyber IRP which is based
on ISO/IEC 27035-1:2023 and have added the considerations detailed above to develop a
comprehensive cyber IRP for their ICS/OT environment.

Admin Corp leveraged their IT incident response plan for the following elements:

• Senior management support—ensures all the necessary resources are allocated and
reflects management’s dedication for making IRP a priority. This includes having clearly
assigned roles and it being clear who is accountable for what with regards the ICS/OT
environment, inclusive of executives, SME’s and incident coordinator.
• Combined Teams—bringing all the stakeholders to the table for a collaborative response
i.e. Service Owner, Business Owner, regulatory, communications, technical, etc including
the team responsible for the ICS/OT Network firewalls, in addition to the control centre
staff, and the teams supporting the PCS and SIS networks is key.
• Detailed but flexible—specific, actionable steps to follow when an incident occurs.
Develop a detailed plan but allow for flexibility to support a wide range of incidents. Avoid
rigid processes that create complexity and an inability to deal with the unexpected. Overly
complicated plans can also be difficult to follow in high stress environment of a real-life
incident.
 • Communication—Clearly define who on the incident team should communicate, contact
details, which communication methods, and what information is released. Clear guidelines
on what should be communicated to management, executives, affected departments,
affected customers, relevant agencies, and publicly.
• Consistent testing—Your plan is just another binder on the shelf if you don’t pressure test it
and practice the execution. Testing and updating the plan can help with flexibility by
allowing additional process for new issues, vulnerabilities, and threats relevant to your
industry.

As part of their IT cyber IRP, the Admin Corp security team, also highlighted that if there is an
incident within their IT environment, that elements of the ICS/OT IRP may be implemented as a
precaution or containment activity, while the incident is investigated further. A template for
documenting the actions and responses taken during an incident was also utilised from the IT
cyber IRP, however, it was slightly tweaked to highlight more strongly aspects relating to the ICS/OT
environment.

What is specific to an ICS/OT incident response plan?

Admin Corp worked closely with operational and engineering teams to ensure the need for the
ICS/OT cyber IRP is understood and supported. These teams have critical insight into the Control
Centre, PCS and SIS networks and could very well be the ones who initially identify an event that
becomes an incident.

The organisation determined it was critical to have teams that are ready to respond to critical
questions during an incident response, such as:

• Is a shutdown required?
• What are the implications to the overall business or other operational processes?
• Do local agencies or outside partners need to be notified?
• Are there relevant regulatory requirements?

By having their IT, ICS/OT, engineering, and HSE teams in the same room they can drive toward a
common goal of restoring normal operations and getting processes running while weighing against
the risk of whether it’s safe to do so.

To support this Admin Corp developed an ICS/OT specific cyber incident response decision
tree/play book, to identify who was responsible for specific elements and actions within the ICS/OT
environment. This decision tree/play book identified owners for key decisions that might possibly
need to be made (such as implanting ‘Island Mode’). It also covered specific owners for the Control
Centre, ICS/OT Firewalls, plus the PCS and SIS networks. Having identified the key roles within the
various teams that needed to be interacting, communicating together, in addition to making the
decisions in the event of a cyber incident, Admin Corp, ensured that the staff in these roles were
trained effectively. Admin Corp also ensured their ICS/OT IRP was integrated in the wider business
continuity and disaster recovery planning.
Admin Corp understands its requirements to Contain, Mitigate and Recover. After extensive vendor
and engineer engagement it now understands which devices, such as some PLCs and industrial
equipment that cannot be backed up or imaged like traditional IT assets. While some required
different forensics procedures, which were detailed and documented in the ICS/OT Cyber IRP,
some would require replacement to facilitate collecting and preserving forensic evidence, which
means that Admin Corp had to identify critical spares for assets, as part of the Cyber IRP process,
which would be needed to replace and restore operational processes at the same time. Admin
Corp in doing this work identified not only key spares that would be needed to be maintained (and
located where easily available to the response teams) but also key elements where vendor support
would be required in response and ensured that their SLAs and vendor contact details covered this
support. Admin Corp also documented the secure process for transporting and storing the forensic
evidence from their PCS and SIS networks within their ICS/OT Cyber IRP.

Admin Corp also identified specific ICS/OT environment logging and auditing capability by
developing a Collection Management Framework across their Business, PCS and SIS networks, to
help them to understand what they had in place to aid event detection. This work identified
significant gaps, that they then established a project to provide the enhanced logging and
monitoring capabilities needed.

Having undertaken a detailed ICS/OT environment network mapping and data flow activity across
the individual networks, Admin Corp also developed for the first time a series of pre-formed
documented containment activities to aid a quicker containment response. These responses took
into account which connections are essential to maintain normal operations and what can be
safely disconnected would support quicker containment activities (such as implementing “Island
Mode”) and included pre-defined separate firewall policy configurations (covering both internal
ICS/OT environment firewalls between sites/layers, but also between the ICS/OT and Business/IT
network.

As part of the planning, Admin Corp revisited its backup procedures to ensure they were robust
enough for the ICS/OT environments, especially the PCS and SIS networks, ensuring that the
backups were made as part of the configuration control mechanisms, and that the location of the
offline backups were fully documented as part of the ICS/OT IRP.

Events vs incidents in ICS/OT

Admin Corp knew they could possibly see some of the same intrusion events in their ICS/OT
environment as in their IT systems (given some of their ICS/OT environment was running on IT
based hosts and servers, especially their SCADA applications within the control centre, and
Windows based HMI’s in the PCS network), but with added complexity of process variables and
alarming. They worked closely with engineering and operations teams to:

• Define what constitutes an event and a cyber incident in their ICS/OT systems (covering the
PCS, SIS, and Business networks), developing an incident severity matrix aligned to their
ICS/OT operations.
 • Build a baseline for system process data to gain a clear understanding of what is normal, so
they are able to more quickly identify anomalies.
• Document the ‘tribal knowledge’ of operations teams and understand the process of
operational response to events that can be evolved to incident response tactics.
• Design an escalation process: Ops, Engineering and ICS/OT Security teams working closely
and train on when to initiate the cyber incident response process.
• Develop decision tree/play book that aids response teams in determining when to shut
down or isolate ICS/OT networks (implementing elements of ‘island mode’).

Proactive Approaches to ICS/OT cyber incident response plan

Admin Corp continue to apply learning from their wider processes to feed their cyber IRP with a
view to continuous improvement. They recognise a cyber IRP should not be static and so by
leveraging existing tools and technologies for threat intelligence, logging/monitoring, disaster
recovery, security awareness training, and regularly scheduled tabletop exercises they build up
familiarity with their cyber ICS/OT IRP, iteratively improving it through every cycle. This approach
identifies the need to add better defined resource roles and responsibilities as well as create new
roles that included an expanded dedicated ICS/OT security team. The exercises also cover
incidents in the Business Network, Control Room network, the PCS and the SIS networks
individually, in addition to combinations of them. This helps focus on the unique and specific
nature of each, especially with the PCS and SIS networks.

Conclusions

Admin Corp recognise that they had previously established the fundamentals of a wider ICS/OT
cyber IRP with their IT cyber IRP, but that they needed to expand upon it with ICS/OT related
practices and expertise and inputs from their engineering and operational teams.

By understanding that that cyber incidents in the ICS/OT environment have more than just financial
implications but potential impacts to life, environment, and infrastructure, Admin Corp has created
an ICS/OT cyber IRP that can be utilised for training, testing, and improving the organisations overall
readiness to respond to incidents. Admin Corp ensured in their ICS/OT cyber IRP that they had
taken into account all of the actions points identified in the ICS COI’s guidance - Considerations for
Cyber Incident Response Planning within Industrial Control Systems/Operational Technology -
detailed below:

• Action point: Develop an ICS/OT specific Incident Response decision tree/play book.
• Action point: Identify and train key individuals to act as the coordination point between IT
and ICS/OT Teams.
• Action point: Create an IRP that is specific to your ICS/OT environment.
• Action Point: Provide Training and Awareness to staff involved in the ICS/OT IRP.
• Action point: Document the event detection examples from your environment in your
ICS/OT Cyber IRP.
 • Action point: Develop a Collection Management Framework, sometimes referred to as a
logging inventory, this is documented result of determining what logging and monitoring is in
place across an environment.
• Action point: Document in the ICS/OT Cyber IRP where the incident response team can
find ICS/OT specific forensic collection procedures.
• Action point: Plan ahead to think about which collection tools can be used, by whom and
how they would be authorised for use, and how collected evidence can be securely
transferred to where it can be analysed.
• Action point: Document where and how containment can be implemented across the
ICS/OT environment. Include this information in the ICS/OT Cyber IRP alongside the
consequences and potential consequences associated with the action.
• Action point: Document in the ICS/OT Cyber IRP the support required for the recovery and
restoration of systems and industrial processes, including contact details for vendors
and/or system integrators.
• Action point: Reference Business Continuity Plan (BCP) and Disaster Recovery Plan (DRP)
outputs into the ICS/OT Cyber IRP, listing out or providing document references to where
and how backups are created, stored, and tested.
• Action point: Create a template that incident response providers can use to record and
track details on an incident and include this (or a reference to it) within the ICS/OT Cyber
IRP.
• Action point: Decide and document the ICS/OT incident severity matrix aligned to your
ICS/OT operations.
• Action Point: Schedule in an ICS/OT incident response exercise at least once a year.

Next steps

Having undertaken this work, Admin Corp believe they have an appropriate cyber IRP in place to
underpin secure operations across their ICS/OT environment.

Admin Corp decided to regularly practice their cyber IRP by conducting ICS/OT specific cyber
incident response tabletop exercises on core system elements every 6-months, while undertaking
every 12-months a combined ICS/OT and IT, technical and executive exercise across the business
to ensure their response teams are ready in the event of compromise. Admin Corp also decided to
instigate use of a specialised ICS/OT incident response company to provide external review of their
exercising activities periodically for an additional level of assurance. Admin Corp decided upon an
NCSC assured incident response company that was also an NCSC assured Cyber Security
consultancy, with a very competent team of consultants working in ICS/OT environments for a
number of Critical National Infrastructure organisations.

The ICS/OT engineering teams also identified that to truly test their response within their ICS/OT
environments, that they would require a representative lab environment, in order for them to safely
practice the likes of backing up configurations, forensic imaging of ICS/OT devices, transferring and
storing securely the forensic evidence, implementation of pre-defined containment policies and
configurations, and have thus started building an environment for them to hone their incident
response skills. This is intended to cover primarily the PCS and SIS networks, including network
devices such as firewalls and switches, in addition to providing a representation of the Control
Centre environment to help train the Control Centre Staff.

As an operator of Critical National Infrastructure, Admin Corp ensure they continue to manage their
business securely by joining NCSC’s CiSP platform to stay abreast of the latest threat
developments to their sector, Sign up for support from NCSC’s Early Warning Service, and also
maintain a relationship with NCSC’s Resilience team covering Private Sector CNI.

CAF IGP Summary

This case study discusses measures that contribute to the following CAF IGP outcomes:

• D1.a Response Plan - You have an up-to-date incident response plan that is grounded in a
thorough risk assessment that takes account of your essential function and covers a range
of incident scenarios.
• D1.b Response and Recovery Capability - You have the capability to enact your incident
response plan, including effective limitation of impact on the operation of your
essential function. During an incident, you have access to timely information on which to
base your response decisions.
• D1.c Testing and Exercising - Your organisation carries out exercises to test response
plans, using past incidents that affected your (and other) organisation, and scenarios that
draw on threat intelligence and your risk assessment.
• D2.a Incident Root Cause Analysis - When an incident occurs, steps must be taken to
understand its root causes and ensure appropriate remediating action is taken.
• D2.b Using Incidents to Drive Improvements - Your organisation uses lessons learned
from incidents to improve your security measures.

Statement of Support

This guidance has been produced with support from Mandiant, Bridewell Consulting, Wales and
West Utilities and members of the Industrial Control System Community of Interest (ICS-COI) for
publication via the Research Institute for Trustworthy Interconnected Cyber-Physical Systems
(RITICS), with support from the National Cyber Security Centre (NCSC). This guidance is not
intended to replace formal NCSC guidance where already available, and care has been taken to
reference such existing guidance where applicable. This document is provided on an information
basis only, Mandiant, Bridewell Consulting, Wales and West Utilities, ICS-COI members and NCSC
have used all reasonable care in verifying the guidance contained within using the data sources
available to it, they provide no warranty as to its accuracy or completeness. To the fullest extent
permitted by law, Mandiant, Bridewell Consulting, Wales and West Utilities, the NCSC and the ICS-
COI accept no liability whatsoever for any expense, liability, loss, damage, claim or proceedings
incurred or arising as a result of any error or omission in the report or arising from any person
acting, refraining from acting, relying upon or otherwise using this document. You should make your
own judgment as regards use of this document and seek independent professional advice on your
particular circumstances. Reference to any specific commercial product, process or service by
trade name, trademark, manufacturer or otherwise, does not constitute or imply its endorsement,
recommendation or favour by Mandiant, Bridewell Consulting, Wales and West Utilities, the ICS-
COI or NCSC. The views and opinions of authors expressed within this document shall not be used
for advertising or product endorsement purposes.