Substation Communications Design Legacy to IEC 61850 Best Practices

Tim Wallaert Chris Jenkins

Overview

This presentation is Part 1 of 3 of our series: Substation Communications Design, Legacy to IEC 61850

At the end of this presentation you should take away AT LEAST ONE new idea that you can use on current projects Additional resources are provided at the end

Let’s get started!

© 2014 Belden Inc. | belden.com | @BeldenInc

2

2

Introduction
Belden’s Substation Communications Infrastructure design overview enables you to compare your designs to industry best-practices.
• • • •

10 easy steps Provides you with insight & ideas for improvement Helps us better understand your needs The things we’ll use are:

Switch / Router List - Network Design
Project Project Engineer
Use Backbone

My Factory David Adams
Location /Name Control Room / Ship / Receive Make Area Packaging Area Control Room / Ship / Receive L3 Make Area L3 Packaging Area L3 Make 1 East Make 1 West Make 2 East Make 2 West Line 1 East Line 1 Central Line 1 West Line 2East Line 2Central Line 2 West Line 3 East Line 3 Central Line 3 West Control Room North Control Room South Receiving West Receiving NorthEast Receiving SouthEast Shipping North L2 x x x x x x L3 10G 1G 10/100 PoE PoE 2x Redun USB ports ports ports ports Watts 1588 IP-67 power nets memory 3 3 3 1 1 1 3 2 3 x Physical Media List - Network Design x 1 7 x x electrical noise? x special requirements Model Catalog Number Accessories Accessory Catalog Numbers

L3

L2

x 8 8 x x 16 Project x 6 Project Engineer x 6 x 10 x 20 x 10 10 x x 20 x Switch Port #12 x 12 Make 1 East 1 x 20 2 x 20 3 20 x 4 12 x x 5 12 x 6 12 x 7 10

Considerations
electrical noise? x

speed
100 100
3100 21 100 3100 21

connector
IP-67 m12 x IP-67 m12 IP-67 m12 IP-67 m12 IP-67 m12 IP-67 m12 IP-67 m12 IP-67 m12

distance (meters)
0.3 10 2 2 7 6 spare spare

electrical noise? Make Area1 FW sec

To Electrical Hi To switch or device port # connector Noise Temp Chemical Flex
RJ-45 IP-67 m12 IP-67 m13 x x x x x x x x x x x x x x x x x x 3 1

other

Industrial Ethernet Cordset
M224PVCSTJG00.3M M224PVCSTMU10.0M M224PVCSTMU02.0M M224PVCSTMU02.0M M224PVCSTMU07.0M M224PVCSTMU06.0M

Make 1 West I/O block 22

I/O block 23 1 IP-67 m14 temp extremes
temp extremes I/O block 24 1 IP-67 m15

100 100 100

Flowmeter 8 temp extremes 1 IP-67 m16 temp extremes

8 Receiving NorthEast

1 2 3 4 5 6 7 8 9 10 11 12

1g 1g

Receiving SouthEast GigE camera

2
fiber

x

Excel Worksheets

Best Practices

Examples

Popular Configuration Drawings

Product Guides

© 2014 Belden Inc. | belden.com | @BeldenInc

3

Segment Ethernet communications into groups (subnets) Subnet: As your Ethernet network grows. | belden.com | @BeldenInc 4 . break a large network into smaller ones connected by routers or layer 3 switches Layer 3 Switch / Router Best Practice 1 Device Device Device Layer 3 Switch / Router • Connects subnets • limits data in and out of each subnet • provides security Subnet A Device Device Device Layer 2 Switch Layer 2 Switch Subnet B Rule of thumb 80% traffic stays in subnet 20% traffic travels in/out Devices on the same subnet easily talk Device Device Layer 2 Switch Layer 2 Switch Device Device Reliability. Maintenance One subnet can be stopped without affecting others Device Device Layer 2 Switch Layer 2 Switch Device Device Device Subnets are great for isolating: • High performance • High bandwidth traffic (video. voice) Device © 2014 Belden Inc. Management.

com | @BeldenInc 5 . | belden. we’ll do it as follows… © 2014 Belden Inc.Segment communications into groups (subnets) Substation Yard Example 1 Bay with Ethernet IEDs Substation Control House Transformer 1 IED layer 3 switch / router Feeder 1 IED Feeder 2 layer 2 switch IED A – Create physical groups based on function and location for maintenance ease and security Devices in a physical group Bay with Ethernet IEDs Transformer 2 IED (these will become SUBNETS) layer 2 switch Feeder 3 IED Feeder 4 IED layer 2 switch Station PC Printer VoiP Phone Bay with Serial IEDs Transformer 3 IED Feeder 5 IED Feeder 6 IED What would you do? This application could be: • 1 subnet • 2 subnets – process and the rest • Several subnets For this example.

Network Design Project Deb-Bay Substations 20-31 Project Engineer Andrew T. | belden.Record Your Infrastructure Choices Record Your Choices 1 Switch / Router List . External Secure External Comms WAN L3 / Router x x x x 10G ports Ethernet 1G ports 1 10/100 PoE ports ports 3 6 6 8 PoE+ ports 1588 precision time Use External comm Bay 1 Bay 2 Control House Location /Name Router 20 Switch B1 Switch B2 Switch CH Equip 3 L2 © 2014 Belden Inc.com | @BeldenInc 6 .

Segment Communications into Groups (vLANs – Virtual LANs) VLAN: Create LOGICAL groups of Ethernet devices that cannot be easily physically grouped Layer 3 Switch / Router Best Practice 2 VLAN 1 VLAN 2 Layer 3 Switch / Router are used • To configure VLANs • Limits data in/out of VLAN • Provides security to VLAN Device Device Subnet A Layer 2 Switch Layer 2 Switch Subnet B Device Device Device Rule of thumb 80% traffic stays in VLAN 20% traffic travels in/out Device OK for devices from multiple VLANs to connect to a switch Devices in the same VLAN can easily talk Device Device Layer 2 Switch Layer 2 Switch Device Device Device Device Layer 2 Switch Layer 2 Switch Device Device Device VLANs are great for isolating • • High bandwidth traffic (video. | belden. voice) Distributed groups of data Device © 2014 Belden Inc.com | @BeldenInc 7 .

4) IED Devices in Bay 2 (VLAN .1) Devices in Logical group 3 Transformer 3 Physical Security IED (VLAN .com | @BeldenInc 8 These groups will become VLANs .3) Feeder 5 IED Devices in Logical group 4 Control House Devices Feeder 6 (VLAN .2) © 2014 Belden Inc. | belden.Segment Ethernet Communications into Groups (vLANs – Virtual LANs) Substation Yard Example 2 Bay with Ethernet IEDs Substation Control House Transformer 1 IED layer 3 switch / router Feeder 1 IED Feeder 2 layer 2 switch IED Bay with Ethernet IEDs Transformer 2 IED Feeder 3 IED Feeder 4 IED Station PC Printer VoiP Phone Create logical groups based on function Bay with Serial IEDs (regardless of location) for maintenance ease Devices in Bay 1 (VLAN .

| belden.Add Serial Communications Devices to your Ethernet Infrastructure Connect to Legacy IEDs and other Serial Communications Devices to Ethernet Layer 3 Switch / Router Best Practice 3 Enet Device Enet Device Layer 2 Switch Layer 2 Switch Enet Device Enet Device Terminal Servers connect multiple serial devices to Ethernet (and the rest of the infrastructure) Enet Device copper Ethernet Terminal Server Terminal Server Enet Device copper Ethernet Serial Device Serial Device Serial Device Use media converters to change copper serial signals to fiber in high electro-mechanical noise areas Serial Device Media Converter serial fiber Media Converter serial copper Media Converter serial fiber Media Converter Serial Device Serial Device Serial Device Adding Serial Devices to an Ethernet Infrastructure provides: • Continued use of serial devices • Upgrade of their overall communications Serial Device Serial Device serial copper Serial Device © 2014 Belden Inc.com | @BeldenInc 9 .

Add Serial Communications Devices to your Ethernet Infrastructure Substation Yard Example 3 Bay with Ethernet IEDs Substation Control House layer 3 switch / router Transformer 1 IED Feeder 1 IED layer 2 switch Feeder 2 IED copper Ethernet Bay with Ethernet IEDs Transformer 2 Terminal Server IED Feeder 3 IED Feeder 4 IED serial copper Media Converter Station PC Fault Recorder Printer VoiP Phone serial fiber Bay with Serial IEDs Transformer 3 IED Feeder 5 IED Media Converter Feeder 6 IED © 2014 Belden Inc. | belden.com | @BeldenInc 10 .

External Secure External Comms WAN L3 / Router x x x x 10G ports Ethernet 1G ports 1 10/100 PoE ports ports 3 6 6 8 8 PoE+ ports 1588 precision time Serial IRIG-B Serial precision Ports time Use External comm Bay 1 Bay 2 Control House Bay 3 Location /Name Router 20 Switch B1 Switch B2 Switch CH Equip 3 Terminal Server B3 L2 © 2014 Belden Inc.Network Design Project Deb-Bay Substations 20-31 Project Engineer Andrew T. | belden.com | @BeldenInc 11 .Record Your Serial Choices Record Your Choices 3 Switch / Router List .

Identify the power consumption (in watts) of each device and total the power for all PoE devices wired to one PoE switch • • Most devices are “standard” PoE – up to 13 Watts Some devices are “PoE+” . Identify PoE devices you will use (cameras. etc. | belden. Coax Cable Signal Out Pan/Tilt/Zoom Signals – Serial Cable Common Examples of PoE Devices PoE Approach • • Fewer pieces/lower cost Mix in existing network infrastructure Low voltage with isolation Switch With PoE • Industrial Ethernet Cable (4-pair copper) © 2014 Belden Inc.5 Watts.Power over Ethernet (PoE) Power over Ethernet (PoE) : use a single industrial Ethernet cable to provide power and Ethernet communications to devices Best Practice 4 Traditional Approach Line Power Power Source Power In 1. telephones.com | @BeldenInc 12 .) and the switch it will connect to 2.up to 25.

up to 25. | belden. Identify the power consumption (in watts) of each device and total the power for all PoE devices wired to one PoE switch • • Most devices are “standard” PoE – up to 13 Watts Some devices are “PoE+” .) and the switch it will connect to: PoE Devices Switches layer 2 switch with PoE IED Bay with Ethernet IEDs Transformer 2 layer 2 switch with PoE IED Feeder 3 IED Feeder 4 IED Station PC Printer VoiP Phone Bay with Serial IEDs Transformer 3 IED 2.Power over Ethernet (PoE) Substation Yard Example 4 Bay with Ethernet IEDs Substation Control House Transformer 1 IED layer 3 switch / router Feeder 1 IED Feeder 2 1. © 2014 Belden Inc.5 Watts. telephones. etc. Identify PoE devices you will use (cameras.com | @BeldenInc Feeder 5 IED Feeder 6 IED 13 .

older technology Determine if application needs sub-millisecond time accuracy All devices synchronized to a GPS or master clock Layer 2 Switch Best Practice 5 Layer 3 Switch / Router Enet Device Ensure all switches / routers / terminal servers in the path between devices needing synchronization support the technology you are using Select devices for the application that support the technology (IEEE1588 or IRIG-B) Enet Device copper Ethernet Terminal Server Serial Device Serial Device Media Converter serial fiber Media Converter Serial Device Serial Device Time Synchronization is ideal for: • First fault detection • Measurement & testing serial copper Serial Device © 2014 Belden Inc.com | @BeldenInc 14 . IRIG-B is a similar. | belden.Time Synchronization Use IEEE-1588 precision time protocol (PTP) for devices on Ethernet requiring extremely precise timing accuracy (<1 microsecond).

| belden.Choose Environmental Ratings for your switches and routers IEC-61850 part 3 ratings describe a device’s protection against environmental and other hazards Best Practice 6 Ensure PT&D network infrastructure products are made for their environment … and meet / exceed relevant industry standards Consider appropriate ratings for: • Temperature • Humidity • Corrosion • Electromechanical Noise Appropriate environmental & electrical ratings specified now eliminates trouble later ! © 2014 Belden Inc.com | @BeldenInc 15 .

Record Your Environmental & Standards Choices Record Your Choices 6 Switch / Router List . | belden.com | @BeldenInc 16 .Network Design Project Deb-Bay Substations 20-31 Project Engineer Andrew T. External Secure External Comms WAN L3 / Router x x x x 10G ports Ethernet 1G ports 1 10/100 PoE ports ports 3 6 6 8 PoE+ ports 1588 precision time x x x Serial IRIG-B Serial precision time Ports Temp 170F 170F 170F 170F 8 x 170F Standards ElectroIEC NEBS 61850Corrosio Mechanic NEMA Level 3 al Noise UL60950 3 n Humid IEEE1613 TS-2 /ETSI EN50155 x x x x x x x x x x x Use External comm Bay 1 Bay 2 Control House Bay 3 Location /Name Router 20 Switch B1 Switch B2 Switch CH Equip 3 Terminal Server B3 L2 © 2014 Belden Inc.

or Terminal Server • Ethernet WAN • Cellular 3G • MPLS-PPP WAN Enet Device Best Practice 7 Perimeter Protection Layer 3 Switch / Router Subnet A Layer 2 Switch Layer 2 Switch Subnet B Enet Device Enet Device Enet Device Ensure your remote communications device provides robust security for the edge of your substation network • Or add a perimeter security & communications device Enet Device Enet Device Serial Device Terminal Server Terminal Server Serial Device Serial Device Configure subnets & vLANs Enable security features in routers. layer 3 switches. Router.Add remote access & network security Add Remote Access & Network Security (based on risk assessment) Add remote communications to your Layer3 Switch. | belden.com | @BeldenInc 17 . layer 2 switches & terminal servers Serial Device Media Converter Media Converter Serial Device Serial Device Network Security is part of Defense in Depth: • Policies & procedures • Physical security • Network security • Computer security • Device Security Serial Device Serial Device Media Converter Media Converter Serial Device Serial Device © 2014 Belden Inc.

Configure subnets & vLANs 3. Enable security features in layer 3 switches & routers 2. Bay with Ethernet IEDs Transformer 2 Terminal Server IED Feeder 3 Ensure any PCs don’t have additional external access or add a firewall here too. | belden. Also ensure good antivirus software is installed & configured correctly ! IED Feeder 4 IED Station PC Media Converter Printer VoiP Phone Bay with Serial IEDs Transformer 3 IED 1. Enable switch security features in layer 2 switches & terminal servers Feeder 5 IED Media Converter Feeder 6 IED © 2014 Belden Inc.com | @BeldenInc 18 .Add Remote Access & Network Security Substation Yard To Operations Center Router/Firewall Example 7 Bay with Ethernet IEDs Substation Control House layer 3 switch / router Transformer 1 IED Feeder 1 IED layer 2 switch Feeder 2 IED Protect the perimeter: Either of these devices may provide remote access & excellent security… choose one or add a standalone router/firewall.

expandable infrastructure © 2014 Belden Inc.Add communications infrastructure between master & substations Add Remote Communications Infrastructure Add remote communications between Master. | belden. etc Bay with Ether net IEDs Best Practice 8 Master Control Station servers SCADA Ethernet WAN MPLS / PPP Cellular Router/Firewall Substation Printers. backup and substations • Ethernet WAN • Cellular 3G • MPLS-PPP WAN Consider making these communications redundant • Robust communications keeps small issues small Redundancy Communication Options Include: • 2 different routes to the master • A master & backup station Make critical communications device power redundant: Power Supply #1 Input Power #1 Printers. etc Bay with Ether net IEDs Router/Firewall Backup Control Station servers SCADA Substa tion Bay with Ether net IEDs Bay with Ether net IEDs Bay with Serial IEDs Router/Firewall Substa tion Bay with Ether net IEDs Bay with Ether net IEDs Bay with Serial IEDs Input Power #2 Power Supply #2 Bay with Serial IEDs … all are key to form a reliable.com | @BeldenInc 19 .

per hour.Evaluate Redundancy Needs Best Practice 8 Math you can do to justify an investment in redundancy • Unplanned downtime calculator • How long will service be impacted? • Will end-customer service be lost? • How much effort is needed to recover and restart your process? • Calculate your downtime cost per minute. | belden.com | @BeldenInc 20 . per day © 2014 Belden Inc.

com | @BeldenInc 21 . External Secure External Comms WAN L3 / Router x x x x 10G ports Ethernet 1G ports 1 10/100 PoE ports ports 3 6 6 8 PoE+ ports 1588 precision time x x x Redundancy Redund.Record Your Redundancy Choices Record Your Choices 8 Switch / Router List .. x x x x x © 2014 Belden Inc. | belden.Network Design Project Deb-Bay Substations 20-31 Project Engineer Andrew T. nets WAN/Cell Use External comm Bay 1 Bay 2 Control House Bay 3 Location /Name Router 20 Switch B1 Switch B2 Switch CH Equip 3 Terminal Server B3 L2 2x power .

| belden.com | @BeldenInc 22 BondedPair . ALWAYS spec: • Bonded Pair (see “9 tests” data) • CAT5e or higher Nonbonded -Pair © 2014 Belden Inc.Specify Copper / Fiber Requirements Best Practice 9 Industrial Copper.A .

B . | belden.Specify Jacket Requirements Best Practice 9 © 2014 Belden Inc.com | @BeldenInc 23 .

and Buy vs.com | @BeldenInc 24 . | belden.C . Build Best Practice 9 or Washdown © 2014 Belden Inc.Specify Standards. Connector Type.

Fax & phone consultation Email. | belden.Keys to Project and Operations Success Industrial Networking Project Checklist Need Manage Design Manage my entire project Review my design & highlight areas of risk Assist with my design in a few key areas Assess my situation & create my design Preconfigure switches / routers Provide industrial installation guidelines Create custom installation instructions & drawings Peform the installation Peform security vulnerability testing Peform network validation Perform startup Provide troubleshooting Dedicated onsite engineering service Stock spares Stock preconfigured spares Firmware Switch warranty Industrial HiVision Service Contract Advanced replacement for faulty devices Remote troubleshooting Dedicated technical support contact On-site troubleshooting Troubleshooting procedures Troubleshooting tools Training for maintenance team Assess planned network changes & highlight areas of risk Onsite visit if needed We review your application & needs & provide recommendations Keep your hardware current Extended Warranty Keep your software current How Belden Can Help Provide a dedicated resource to work as customer staff Email.com | @BeldenInc 25 . Fax & phone consultation Onsite meeting & comprehensive network design Best Practice 10 Install Recommend experienced Belden System Integrator or partner Recommend experienced Belden System Integrator or partner Onsite testing and assesment Onsite testing and assesment Recommend experienced Belden System Integrator or partner Onsite troubleshooting Startup Operate Maintain Get help from someone that knows you and your application Upgrade Fax & phone consultation © 2014 Belden Inc.

| belden.Where do I start? Popular Config Diagrams © 2014 Belden Inc.com | @BeldenInc 26 .

| belden.Popular Config Diagram 1 – Ethernet WAN © 2014 Belden Inc.com | @BeldenInc 27 .

Popular Config Diagram 2 – MPLS WAN © 2014 Belden Inc. | belden.com | @BeldenInc 28 .

com | @BeldenInc 29 . | belden.Popular Config Diagram 3 – Cellular WAN © 2014 Belden Inc.

• • • • 10 easy steps Provides you with insight & ideas for improvement Helps us better understand your needs The things we’ll use are: Switch / Router List .3 10 2 2 7 6 spare spare electrical noise? Make Area1 FW sec To Electrical Hi To switch or device port # connector Noise Temp Chemical Flex RJ-45 IP-67 m12 IP-67 m13 x x x x x x x x x x x x x x x x x x 3 1 other Industrial Ethernet Cordset M224PVCSTJG00.com | @BeldenInc 30 .Summary Belden’s Substation Communications Infrastructure design overview enables you to compare your designs to industry best-practices.0M Make 1 West I/O block 22 I/O block 23 1 IP-67 m14 temp extremes temp extremes I/O block 24 1 IP-67 m15 100 100 100 Flowmeter 8 temp extremes 1 IP-67 m16 temp extremes 8 Receiving NorthEast 1 2 3 4 5 6 7 8 9 10 11 12 1g 1g Receiving SouthEast GigE camera 2 fiber x Excel Worksheets Best Practices Examples Popular Configuration Drawings Product Guides © 2014 Belden Inc.3M M224PVCSTMU10.0M M224PVCSTMU07.0M M224PVCSTMU02.Network Design Project Project Engineer Use Backbone My Factory David Adams Location /Name Control Room / Ship / Receive Make Area Packaging Area Control Room / Ship / Receive L3 Make Area L3 Packaging Area L3 Make 1 East Make 1 West Make 2 East Make 2 West Line 1 East Line 1 Central Line 1 West Line 2East Line 2Central Line 2 West Line 3 East Line 3 Central Line 3 West Control Room North Control Room South Receiving West Receiving NorthEast Receiving SouthEast Shipping North L2 x x x x x x L3 10G 1G 10/100 PoE PoE 2x Redun USB ports ports ports ports Watts 1588 IP-67 power nets memory 3 3 3 1 1 1 3 2 3 x Physical Media List . | belden.Network Design x 1 7 x x electrical noise? x special requirements Model Catalog Number Accessories Accessory Catalog Numbers L3 L2 x 8 8 x x 16 Project x 6 Project Engineer x 6 x 10 x 20 x 10 10 x x 20 x Switch Port #12 x 12 Make 1 East 1 x 20 2 x 20 3 20 x 4 12 x x 5 12 x 6 12 x 7 10 Considerations electrical noise? x speed 100 100 3100 21 100 3100 21 connector IP-67 m12 x IP-67 m12 IP-67 m12 IP-67 m12 IP-67 m12 IP-67 m12 IP-67 m12 IP-67 m12 distance (meters) 0.0M M224PVCSTMU02.0M M224PVCSTMU06.

| belden.Additional Resources & Assistance 1. Contact a Belden representative for assistance:   Call 510-438-9071 if you are in the U.belden.com | @BeldenInc 31 . Listen to the recorded webinar of this presentation:  Substation Communications Design Legacy to IEC 61850 Recorded Webinar 2.belden.com/contact/ Thank you for your interest in this presentation! © 2014 Belden Inc.   Obtain further Substation Communication resources from our website: www. or Canada Or complete the form at www.com/power-td/ This webpage includes substation communication diagrams and other useful tools 3.S.

com | @BeldenInc © 2014 Belden .Belden.