Our client is a UK-based water company with a large infrastructure of Industrial Control System (ICS) sites distributed over a wide geographical area. These sites did not have any remote network connectivity to them but could be monitored through a dial-up telemetry system. There were also leased lines in place inter-linking specific water treatment sites. There were several typical ICS architectures in operation, ranging from smaller sites with a simple programmable controller, to large complex sites with multiple controllers and operator interfaces. Some of the complex systems were interconnected with legacy proprietary network protocols, although some used Ethernet local area networks (LANs).
The Challenge
Our client identified a range of business needs for digitising communications to its ICS networks through IP communications, including:
- asset performance monitoring
- remote support
- extraction of valuable OT/ICS data into business systems
- monitoring of internal security threats
- replacement of leased lines
This level of interconnectivity would expose the infrastructure to external threats which it had not previously faced, nor been designed to combat, as some of the equipment had been in service for many years. The increased interconnectivity would also significantly magnify any breach, due to the potential to compromise multiple systems simultaneously.
Our client established an IP enablement program to deliver the required infrastructure to support secure IP communications to its ICS environments. Central to the IP enablement project was a cloud datacentre hosted in Microsoft Azure, which was delivered by Bridewell. It was agreed with the client that the Azure deployment would operate in a dedicated OT tenancy, employing managed subscriptions and resource groups to provide common internal and external gateway infrastructure shared across multiple applications whilst maintaining administrative segregation between resources.
The Solution
Over the course of the two-year program, we designed and delivered the following:
1. High-level design - Covering all areas of the IP enablement solution, including:
- A scalable Azure OT data centre design to allow for easy hosting of additional applications and services in the future.
- A centrally manageable security gateway infrastructure to secure data centre interfaces and remote site borders.
- A VPN solution to protect data across the company’s MPLS WAN providing secure inter-site communications.
- A secure remote access solution that mitigated the risk of a threat using the service to simultaneously impact multiple remote sites.
- Standard segregated network architecture for ICS sites.
- Active Directory for centralised identity and access management.
- Read-only domain controllers on remote sites to provide user authentication and authorisation should connection to the data centre be unavailable.
- Security event collection into SIEM to allow SOC monitoring.
2. Detailed-level designs - After approval of the high-level design, Bridewell developed detailed-level designs for the Azure datacentre (including the VPN infrastructure and secure remote access), Active Directory infrastructure, ICS network security design, and ICS endpoint protection.
3. IP addressing schema - To support the ability to assign unique addresses in a consistent manner whilst providing efficient routing and VPN configuration, we developed an OT infrastructure-wide IP addressing schema scalable for IP enablement of the entire estate.
4. Firewall rule base - To ensure traffic flows were restricted to the minimum necessary at all network borders, we designed the firewall rulesets to be deployed on the data centre and ICS site firewalls.
The Results
Bridewell delivered a fully functional OT data centre in Azure, comprised of:
External network virtual appliances (NVAs) - Providing central breakout to the Internet for the entire OT network.
Internal NVAs - Providing access via three ExpressRoute circuits into a private MPLS wide area network, separating traffic dependent upon business area.
Load balancers - External and internal NVAs were deployed in high-availability pairs with load balancers to direct traffic to the primary active node.
DMZ network - Connected between the external and internal NVAs.
Active Directory - A combination of Windows Server AD (hosted in Azure) and Azure AD with MFA providing group policy management and centralised authentication for the entire OT estate.
IaaS & PaaS - The data centre used PaaS resources where available, such as SQL server databases. However, most OT applications had no PaaS service offerings, requiring the deployment of virtual machines (VM)s to host the applications.
Resilience & Availability - We used availability sets for basic VM high-availability requirements. VMs running critical services requiring higher availability were deployed across availability zones protecting against failure of a single data centre.
Network Segregation - Within each resource group network components were segregated into individual VNETs with peering connections via the internal NVAs to control inter-VNET communications.
Management Platforms – Providing centralised management of ICS site security appliances, virtualised servers, anti-malware software, software updates, backups, network management and network performance monitoring.
Security Event Monitoring – Security events passed successfully out to our client’s SOC.
Secure Remote Access – Exploiting the capabilities of Azure to allow OT engineers to spin up remote access VM instances from pre-configured templates to provide a timebound remote access capability.
All core elements of the design were provisioned from Azure Resource Manager (ARM) formation scripts. Primarily this provided a coherent approach to adding additional resource in a structured, secure, centralised, and audited manner but also allowed for the quick and easy deployment of development and staging environments. ARM formation scripts will also provide the foundation for future iterations of the datacentre when the current platform reaches end of life, thus future proofing this design concept. The modular design of the solution provided further future proofing through full support to implement additional future datacentre solutions, both new and migrations that may use both PaaS and IaaS elements in Azure. This deployment method also allowed the data centre to be easily redeployed (potentially to a different environment if necessary) should a significant failure occur.
Bridewell provided detailed support guides for the full solution and training sessions for the client’s administrators who would be taking on management of the solution. We also provided early life support for 30 days post training and handover and warrantied the solution for 90 days.
