You are on page 1of 4

Secure IED Management Case Studies

A. Salvador, Network Integration Specialist, SUBNET Solutions Inc., D. Mack, Substation


Integration Specialist, SUBNET Solutions Inc., and C. Carnegie, System Integration Specialist,
SUBNET Solutions Inc.,

B. Existing System and Issues


Abstract--This paper presents a summary and analysis
deployment of a secure Intelligent Electronic Device (IED) The utilitys generation plants consist of 41 dam sites, 30
management system at two different utilities in North America. hydro facilities, and 9 thermal sites. They have approximately
Each of these utilities began their investigation into technologies 18,000 kilometers of transmission lines, 260 substations and
and methodologies for securing their system for North America 56,000 kilometers of distribution lines. The utility is providing
Reliability Corporation (NERC) Critical Infrastructure electricity to approximately 1.8 million customers in the region.
Protection (CIP) standards. This report is intended for anyone Due to the utilitys size and coverage area, approaching a
considering the implementation of an IED management system
solution to manage all their remote devices, which is estimated
Index TermsDevice Management, Interoperability, Legacy
to be around 10,000 devices, is not a simple task. The utility is
Devices, Multi-Vendor. not only faced with a quantity problem but also terrain issues.
Due to the mountain range nearby, certain substations and
I. INTRODUCTION devices are difficult to communicate with due to a complex
communication infrastructure as well as the possibility of bad
T HE two utilities featured in this paper had the same goals
when first initiating the their projects; to meet NERC CIP
[1] standards by implementing IED management systems that
weather. Depending on the location of the devices, bandwidth
can be a scarce commodity, and only operational data is
collected over the communication lines. To obtain important
assist with non-operational data collection, provide secure non-operational data such as event files and configuration files,
remote access and increase system reliability. However, due to personnel are forced to physically access these remote devices,
old, yet reliable protection equipment, both utilities which introduces expensive labor cost and safety issues due to
encountered issues with integrating legacy devices with modern the terrain. Since non-operational data was so difficult to
IED management systems. collect, analysis of the data is intermittent and fails to produce
This paper summarizes the lifecycle of both projects, a clear picture of the system for data trending and predictive
including a description of the problems and challenges the maintenance.
utilities faced during the implementation to meet their unique Another problem the utility faces is the vast number of
challenges, while keeping in mind their expansion plans over brands of devices that are implemented in the field. Each vendor
the next few years. provides devices that use a different protocol or protocol variant
and, most of the time, a proprietary configuration software
II. CASE STUDY #1 application that must be learned by the utilitys personnel that
are collecting the data.
A utility was dealing with the problem of trying to manage
all of the distribution automation devices they have on their C. Solution Design
network. They had too many devices to manage manually, and To reduce labor cost and increase safety, the utility focused
access to device event data was non-existent or difficult, which on providing their distribution engineers a solution for
prevented them performing root-cause analysis of circuit managing IEDs on their distribution network. The project scope
reliability. They had security and worker safety risks in included reclosers, voltage regulators, switchgear relays and
managing the device passwords, settings and configuration capacitor bank controllers and allowed for future expansion to
changes. Because of this, the utility executed a project to substation and transmission equipment. The solution was
develop a solution that overcame these challenges. required to be device/vendor agnostic with the ability to secure
access to, and manage non-operational data of these IEDs
A. Project Goals
remotely. Due to the unforgiving terrain of the utilitys
The utilitys main goal is to meet NERC CIP standard by distribution network, the solution needed to be robust enough
implementing an IED management system that securely allows to handle low performance communications such as satellite
personnel to access remote devices, manage its passwords, and cellular connected IEDs. As well, the solution would allow
obtain configuration files and retrieve event files without them to become NERC CIP compliant.
physically accessing these devices. By allowing personnel to The utility selected a software integration company as the
remotely manage the devices, the utility expects to save labor contractor to drive this project to completion. Together, they
cost, equipment maintenance cost, and increase safety and designed a central IED management solution to overcome these
reliability within its system. challenges. The platform is vendor agnostic, provides a means
for remotely maintaining non-operational IED data, and

978-1-4673-8848-1/16/$31.00 2016 IEEE


2

is scalable for future expansion. It supports remotely connecting work flow events that can alert specific personnel allowing
to IEDs regardless of the communication protocol or them to examine the new configuration file, and determine if
connection media that is utilized by the devices. This remote new changes should be approved or denied. The new
connectivity provides users with the ability to configure and/or configuration file can stay in the device or the old configuration
retrieve data, as if the user is directly connected to the device. file can be pushed down to the device if the changes were
Most importantly, it provides them a secure method of rejected. All versions of the configuration file are centrally
accessing the remote IEDs for various management tasks. stored, archived and are accessible to utilitys personnel. For
Furthermore, the system is designed to significantly aid the devices that have low bandwidth communication, the
utility for compliance with guidelines, standards, and policies automated retrieval of configuration files can be disabled and
for critical cyber assets as outlined within NERC CIP only on demand retrieval is allowed. This allows users to only
requirements. retrieve configuration files when required and communication
The solution is installed in the corporate local area network is reserved for operational data.
(LAN) and natively integrated with the utilitys active The Event file management module allows the system to
directory. The system is isolated in a demilitarized zone (DMZ) automatically retrieve new event files from field devices. When
utilizing firewalls with strict access rules. An example of the a new event file comes into the system, the module can alert
rules on the firewall between Corporate LAN and the DMZ subscribed personnel via email. Personnel can then analyze the
include ports required to access the system with Microsoft new event file and determine the importance of the event. All
Remote Desktop client, authenticate with the Microsoft Active event files are centrally stored and archived within the system
Directory and send outbound email notifications. The firewall where personnel can access multiple files and analyze them for
between DMZ and remote device network is configured to only predictive maintenance on the field device or its corresponding
allow access to the distribution IEDs intended to be managed equipment. Similar to configuration management, devices that
by the solution. have low bandwidth communication, the automated retrieval of
The solution is modular, allowing the utility to pick and configuration files can be disabled and only on demand retrieval
choose which modules were relevant to their system. The is allowed.
Modules available and implemented by the utility were for In addition to the modules that the IED Management system
remote engineering access, password management, provides, new software features and enhancements are required
configuration management and event file management. to meet all of the project requirements. One of the key concepts
The Remote engineering access module allows personnel at used to tackle the design of these new features is the process of
the utility to remotely access the field device securely from the Agile Development. This iterative approach to the software
centralized system. The module only displays the devices that development process introduced two important benefits to the
the user has permission to access, and other devices are project, evaluation and transparency.
completely hidden and are inaccessible to the user. The system Many times, high level requirements defined in a projects
is configured with pre-approved vendor applications and is scope of work are not enough to fully understand the expected
linked to the proper field devices. The user has the option of function or use case. To reduce wasted time and money, the
what pre-approved applications they want to use to connect to requirements are discussed and defined in detail with the utility.
each device. Once connected to the field device, the system After development of a function is complete, the utility is given
monitors the users activity such as commands sent and/or the opportunity to evaluate it. Going through this iterative
mouse clicks used within the system. If the user sends an un- process several times throughout the project provides
approved command or clicked on un-approved item, the system transparency, demonstrates progress, and confirms the solution
intercepts the action and prevents the command or action being for the utility. The Agile approach is a win-win scenario for
sent to the field device. All remote connections made by the both the utility and the integration company. The utility
user to field devices are logged and auditable for NERC CIP or receives the functionalities they are looking for and the
for internal purposes. integration company increased its products functionalities and
The Password management module allows the system to capabilities for other utilities to take advantage of.
manage the field devices passwords. Passwords are encrypted In addition to the benefits of the agile approach mentioned
and centrally stored within the system. The system above, the process helps drive innovation for advancements in
automatically accesses the device and changes its passwords vendor agnostic solutions, specifically, advancements in the
based on utilitys preference; for example, at a configurable ability to manage and interoperate with closed and proprietary
interval, such as once a year, or when an employee uses a IEDs. A key requirement for the utility is to automate time
password and no longer needs the current password. The consuming and inefficient tasks for their distribution teams.
Password management module assists the utilitys compliance Tasks such as collecting and organizing IED event data,
with NERC CIP where it requires that field devices have their collecting and looking for out of band IED configuration
passwords changed to its highest complexity every 15 months. changes, and changing IED passwords. For some IEDs, these
Making password changes on all field devices manually can be tasks can easily be automated if they support open standard
tedious and very expensive for the utility. protocols such as Modbus, SEL ASCII, and telnet. The utility
The Configuration management module allows the system to requires automation of these tasks for the several devices such
automatically retrieve configuration files from field devices. It as the SEL 751 and Eaton Form 6. The SEL 751 provides well-
compares any new incoming configuration file against a documented, open standard protocols and is easily automated
previous configuration file that was retrieved from the same in the system through a protocol driver. This type of driver
device. If the module detects any differences, it can initiate automates tasks to the end device using the native
3

communication protocol. However, it was clear in the initial completed successfully or if it failed for a specific reason.
investigations that the Eaton Form 6 was a closed system, and Furthermore, all IED specific documents that are managed by
did not support open standard protocols. Analysis of the the solution, such as configuration files, point mappings,
communication protocol showed messages were encrypted or drawings, stencils, etc. are versioned controlled. The version
hashed. Users are forced to manually manage these IEDs with history for each file indicates the modified date, modified user
vendor-specific software. and comments associated with the version. It also allowed users
Lacking the ability to communicate with the Form 6 device to retrieve previous versions for recovery situations. In
using an open standard, the concept of an Application Driver in addition, configuration files are integrated in document
the IED management system was developed during the project. workflow approval processes. For all files in the Working
This type of driver is based on a software wrapper developed to library, users are required to exclusively check out the file
remotely drive a vendor application to communicate with an before they are allowed to edit them in the system. When a user
IED. It provides the ability to automatically mimic the mouse checks out a file, the system indicates which user is working on
clicks, keyboard entries, and file saves normally performed by the file. When a user checks in a file and attempts to approve it
a human. Working with the utilitys distribution teams, several to a major revision, other users with authorized approval
workflows were created for all the mouse clicks and keyboard permissions are notified by email of the new configuration.
entries for communicating with the Eaton Form 6 using the Only the users with the approval permissions can approve or
ProView application. These workflows were turned into Jobs reject the file after verifying the changes.
within the system. These Jobs can be issued to a specific device
on demand by an authorized user or scheduled on a recurring III. CASE STUDY #2
basis by the system. The jobs are generalized to be vendor A utility was able to cost-effectively implement a secure and
agnostic providing users with a standard mechanism for powerful relay and meter integration and management solution.
interacting with all devices being managed by the system. Jobs A key to this projects cost-effectiveness was that the utility
such as change password, get fault files, get configuration, get was able to avoid significant capital and labor costs associated
SOE, and get data profiler can be issued against an Eaton or with relay firmware upgrades and/or relay replacement and
SEL device. Jobs can also be disabled by users in the event that recommissioning. Instead, the utility was able to implement a
a device is physically being worked on in the field. It is also the solution that met the objects listed above without changing any
responsibility of the system to determine if the device required of the relays they have deployed over the last two decades.
using an Application Driver or a Protocol Driver to complete
the job. The jobs would also only upload new information from A. Project Goals
an IED by comparing it to the last known configuration, last The utilitys main goal is to meet NERC CIP standard by
collected fault file, or last retrieved SOE. This comparison implementing additional security and record keeping for
becomes invaluable when configured on a scheduled basis, as protection relays and meters at minimum cost. However, the
the system can then watch for out-of-bounds configuration utilitys system consists of old devices and contains schemas
changes or event fault data for post-event analysis. that were designed before NERC CIP existed. The question of
Further, the Application Driver can permanently hide or How do you implement new functionality to existing
restrict access to menus, buttons, and/or dialogs within the infrastructure at minimum cost? presented itself to the utility.
vendor application. This concept is incorporated in the remote
engineering access module to improve security and reduce B. Existing System and Issues
human error. In the case of the Eaton ProView software, a user The existing system can be described as a large coal
is required to log into the application first, followed by also generation plant connected to a Bulk Electric System (BES) at
logging into the end device. By utilizing the Application Driver 230kV supplying power to a large vehicle manufacturer. The
capabilities, a workflow was developed to automatically log distribution grid for the system operates at 230kV, 120kV and
into the application and device, without any user interaction or 34.5kV. The plant generates 750 MW of power to supply the
knowledge of the passwords. Following this workflow, the user vehicle manufacturer, but has sufficient capacity so that it can
is then given control of the application to remotely sell some of the power to the state for other consumer purposes.
communicate with the IED. Extending this concept to security, The protection system is comprised of GE Universal Relay
depending on the users authorized permissions, certain menus (UR) relays that have version 2.xx firmware. These relays have
and buttons in the application like Trip or Close can be disabled been in place since the plant was commissioned and are
throughout the remote engineering access session with the user. functioning as expected.
This is important to the utility, as the ProView software The secondary protection scheme is comprised of Alstom
provides these buttons to the user, no matter what device access Micom and Schweitzer Engineering Laboratories (SEL) relays.
level or application role was used to connect to the device. The system metering is handled by Power Measurement
The concept of logging is also an important requirement for Laboratories model 7600 and 7650 revenue meters.
the utility. The goal was for the system to provide situational As part of the project to increase security and record keeping,
awareness. In terms of individual devices, the concept of device the utility deployed a new IED Management system and Human
state was created to allow users to immediately understand if a Machine Interface (HMI) system. The IED Management
device was normal, disabled, tagged or decommissioned when system allow the utilitys personnel to securely access the relay
users navigated to devices in the system. All jobs issued, either and meters for maintenance and automatically manage the
by a user or the system, are also displayed and logged for the devices passwords for NERC CIP compliance. The HMI
device. The jobs contain state information indicating if the job system allows the utility to have better visual knowledge of
4

their protection system. The HMI system provides real-time vendor to modify the Modbus protocol was a small fraction of
visual displays, digital alarm panels and data trending. As part what it would have cost to replace all the GE UR relays.
of the IED management system, the system can also In regards to labor cost, the implementation of the hotfix took
automatically retrieve event files from the SEL and GE UR only a few hours, and minimal labor cost, to install and test the
relays. functionality on the live system in comparison to the labor cost
The main issue that was observed during the IED associated with commissioning of new relays and its protection
management system deployment was the advanced capabilities schema. Event file collection for all GE UR relays was resumed
of the newer Modbus protocol used for Event File Collection in and the entire system performed as expected with no issues for
the GE UR relays caused the version 2.xx GE UR relay to go any relays.
into a restart sequence on every event file collection cycle. In
other words, during the event file collection sequence, the IV. CONCLUSION
command would begin power cycling every relay in the entire As part of any hardware vendor business model, hardware
system, one at a time. It was determined that the newer Modbus vendors will always implement features that will encourage
commands produced this unexpected result with the older GE utilities to purchase more hardware. As experienced in the first
UR relays. GE was unaware of the issue with the version 2.xx case study, it is not uncommon for hardware vendors to
firmware and recommended the replacement of the GE UR implement proprietary protocols within their devices and their
relays with newer hardware and firmware version 7.xx. The software application hoping the utility will keep their hardware
utility contemplated these two solutions for the project: because they do not want to go through another process of
Replace all GE UR version 2.xx relays with version. learning another vendors protocol or application. A proprietary
7.xx relays protocol encourages the utility to implement a single-vendor
Leave all GE UR version 2.xx relays in place and system instead of implementing the best equipment available
modify the Modbus protocol used for Event File on the market to do the job required regardless of its vendor. To
Collection implement a highly reliable and secure system, the utility needs
C. Solution 1: Replace all GE UR version 2.xx relays with to encourage multi-vendor systems. Multi-vendor systems will
version 7.xx relays. force vendors to move to open standard applications which will
promote cost savings, personnel safety and system reliability.
This solution is to remove and replace all existing UR relays
The second case study experienced a different problem with
with newer version of the relays. There are several problems
hardware vendors. With new functionalities being introduced to
with this solution. The first issue is the cost. Thirty-seven relays
the industry every year, its very easy for hardware vendors to
at a cost of $6000 per relay is approximately $222,000. This
implement these new features in new hardware and not old
significant number was not budgeted into the cost of the project.
hardware; thereby enticing utilities to upgrade their relays and
The second issue is the manpower and outage time required to
meters. However, depending on the functionality, it can be
replace every single relay. Allocating manpower resources and
implemented somewhere else. Utilities need to analyze their
scheduling time for systems to be down while relays are
current system and determine if certain functionalities can be
replaced would have added a significant increase in the budget.
implemented at a higher level which can be pushed to all their
The third issue is the need to recommission every device and
hardware to reduce implementation cost as well as labor cost.
connection because all the original wiring would have to be
As stated in this case study, a simple hotfix to an upper level
disconnected and then reconnected to all thirty-seven relays.
system that was communicating to all the relays saved this
Once again, there are significant time and financial costs
specific utility a large amount of money and labor.
associated with the commissioning effort.
Ultimately, the main goal for any utility is to make their
D. Solution 2: Leave all GE UR version 2.xx relays in place system more safe, reliable and cost effective. Some utilities lack
and modify the Modbus protocol internal expertise or teams that understand IT technologies. For
This solution is leaving the current relays in place and have some of these utilities, they are turning towards vendors to bring
them function as they had in the past. This was the best idea for that level of expertise and knowledge to secure and manage
the utility, but the issue with restart sequence caused by event their IEDs by bringing in proven IT technologies. Utilities need
file collection needed to be resolved. In discussions with the to ensure vendors recommendations are objective and cost
IED management system vendor, they determined that it would effective and simply not recommendations that will require
be possible to modify the Modbus protocol in the IED more purchasing of hardware.
management system to handle older GE UR relays.
After some testing with older GE UR relays, a modification V. REFERENCES
to the Modbus protocol was implemented and a hotfix for the Standards:
system was provided to the utility. [1] North American Electric Reliability Corporation [Online].
Available:
E. Solution Design http://www.nerc.com/pa/Stand/Pages/CIPStandards.aspx
The final decision made by the utility was to implement
Solution 2. A hotfix was created by the integration company
and the utility was able to successfully retrieve event files from
the old GE UR relays without restart issues. Once the Modbus
protocol change was proven to be working as expected, the
utility realized they made the right decision. The cost to pay the