The following blog is adapted from the book Industrial Cybersecurity Case Studies and Best Practices, authored by Steve Mustard. More excerpts will be following in the coming weeks. See Excerpt #1 here. See Excerpt #2 here.
NEW: See Steve Mustard's October 2022 appearance on NasdaqTV here.
The Pitfalls of Project Delivery
One might think that a project involving a new facility, or an upgrade to an existing facility, would present the ideal opportunity to implement effective cybersecurity controls from the ground up. Unfortunately, this is rarely the case. Despite the widespread awareness of the cybersecurity threat and the availability of standards, certified products, certified professionals, and collective experience, systems are being deployed that lack the most basic security controls. In addition, the projects themselves create additional security vulnerabilities due to poor training, awareness, and oversight among personnel. In addition, a focus on efficiency and cost reduction means that many of the duties involved in managing cybersecurity are added to existing workloads, rather than to dedicated professionals with the right mix of skills and knowledge.
The key factors required to correct these issues are:
- Secure senior project leadership support – without this cybersecurity will be relegated to a “nice to have” requirement in the project.
- Embed cybersecurity throughout the project – a typical project may run for several years, and at each stage vulnerabilities can creep in, both within the project team and in the final facility design.
- Embed cybersecurity requirements in all contracts - Contracts should include cybersecurity requirements not only for systems, but also for project execution. There should be explicit milestones, deliverables, and payments related to cybersecurity.
- Raise awareness within the project team - Cybersecurity awareness is essential throughout all phases of a project. It should be a requirement for all members of the team, regardless of whether they work for the asset owner, a vendor, or a system integrator.
- Implement rigorous oversight processes - As with any aspect of project execution, cybersecurity requires constant, rigorous oversight to ensure success. Key oversight elements are requirement verification, risk and issue management, and performance management.
Good cybersecurity requires significant investment, often without an obvious financial return. Like any essential element, cutting costs in cybersecurity creates additional expenses down the line. These additional costs may crop up later in the project or during the operational phase.
Embedding Cybersecurity Throughout the Project
The following table summarizes the key stages in a typical project, and the key cybersecurity considerations for each stage. Cybersecurity needs to be considered from the earliest stage and be constantly managed throughout until handover, when it must be managed as part of ongoing operations.
Stage |
Key Cybersecurity Considerations |
Feasibility |
Cybersecurity risks in each phase of the project Cybersecurity risks in the final facility |
Conceptual engineering |
Cybersecurity risk comparison for high-level logical design options |
Preliminary engineering (front-end engineering design, or FEED) |
Cybersecurity requirements for systems or devices Verification requirements |
Detailed engineering |
Contractual requirements and payment milestones Detailed test specifications Cybersecurity design reviews |
Construction |
Management of change Incident response preparedness |
Commissioning |
Management of change Incident response preparedness Red-team assessment |
Start-up |
Management of change Incident response preparedness |
Handover and closeout |
As-built documentation Asset inventory |
Management of Change
Despite the best efforts of everyone involved in the engineering phase, errors and omissions will occur that must be corrected. A typical example is the need to run additional cables to accommodate system connections.
Often, some requirements are omitted during the project phase. For instance, equipment required for vendor remote access to its system may not be included in the project scope because it is considered part of a separate maintenance contract. As a result, changes may be needed to accommodate this equipment later, as well as additional cabling to provide connectivity.
Changes may not involve omissions related to known requirements. Due to the long-term nature of the project, new requirements may arise. For example, the asset owner may incorporate new equipment to support additional production capacity.
In all these cases, a rigorous change-management process should be established to address the impact and procedure for these changes. The impacts may include the effect on project timescale and cost. The process may also address such issues as performance and resilience.
Cybersecurity must be included in the management of change process, to ensure that:
- Drawings and other documents are reviewed and updated when making changes. Accurate drawings and documents are essential to successfully manage cybersecurity. The failure to indicate the connectivity of equipment, for instance, can lead to an incorrect assessment of product vulnerability.
- All procedural cybersecurity controls, such as multifactor authentication and anti-malware checks, are followed when executing the change.
- All technical controls are in place on completion of the change. For example, an industrial firewall may be disconnected during testing to resolve a system-to-system communications failure. This firewall must be reinstated once the issue is resolved to ensure all necessary controls are properly deployed. Figure 1 shows an example where this was not done, leaving the facility with a missing cybersecurity control that few were aware of.
Figure 1: An Industrial Firewall Left Disconnected After a Change
Incident Response Preparedness
Figure 2 shows a typical project construction site. There may be dozens, or even hundreds, of workers operating around technology that is vulnerable to cybersecurity incidents. Even if they are not working on the technology itself, personnel will need access to communications networks to collaborate, report, and work safely. This may take the form of a temporary wireless network with Internet access. Internet access may encourage workers to check their email or browse social media. If these workers are not aware of cybersecurity risks, their devices or the network itself could be compromised. Even if the impact is limited to worker devices and the temporary network, this disruption could delay the project.
Figure 2: Workers at a Typical Project Construction Site
Red-Team Assessment
A red-team assessment is an important tool in the verification of cybersecurity posture. The assessment gets its name from military wargaming, where conflicts are simulated between an aggressor, the red team, and a defending force, the blue team. Red-team assessments in cybersecurity involve experts attempting to achieve a target, such as access to a certain machine or other resource. The exercise identifies vulnerabilities that can then be addressed. There are other methods of identifying vulnerabilities, such as penetration testing. Red-team assessments, if conducted properly, reflect realistic scenarios that may occur. These assessments identify vulnerabilities in technology, people, or processes.
Although the commissioning phase is hectic, it is an opportune time to conduct a red-team assessment. It is likely impractical to conduct such an assessment earlier in the project. Prior to commissioning, many of the systems and networks are not fully operational. For similar reasons, the scope of testing security controls during factory acceptance testing (FAT) may be limited and still not fully representative of the final facility. For instance, the physical security element of a red-team assessment is not indicative of the actual controls that will be in place. However, a red-team assessment also provides realistic training for the operations personnel acting as the blue team in the exercise.
Figure 3: Red-team Testing Physical Security Controls During a Project Assessment
Summary
Projects that deliver new automation systems or enhancements to existing systems routinely introduce new cybersecurity vulnerabilities in organizations. In addition, the projects themselves contain vulnerabilities that can impact the organization. The lack of understanding of cybersecurity risks is a major factor, as is the failure to correctly manage cybersecurity.
There are many things that organizations can leverage to improve results, including the following:
- Use certified secure products from certified vendors.
- Define security controls at the design stage to avoid costly, less effective, implementation later.
- Regularly review cybersecurity implementation progress in the project.
- Link milestones and payments to cybersecurity requirements.
- Ensure the project resourcing and time plan includes a regular cybersecurity update of equipment during execution (e.g., anti-malware, software upgrades/patches, backups).
- Include a plan for changing over all user accounts and test code, as well as removing vendor accounts, default accounts, and test software or configurations.
- Define a cybersecurity incident response plan for the project that includes all stakeholders, so a process is in place when an incident occurs during project execution.
- Provide regular cybersecurity awareness training for everyone on the project, including users, vendors, and integrators.
- Plan for an independent red-team assessment of the final as-built environment, incorporating realistic scenarios to provide additional assurance that security is in place as expected.
Cybersecurity is a critical element of operations and must be treated as such during a project.