IEC 61850 Network Communication

Overview
Agenda

The Concept of IEC 61850

Communication Protocol

Substation Communication Practice

Topology Design

Virtual LAN

Time Synchronization

Confidential
The reliable networking infrastructure plays a critical role
to enable an IEC 61850 substation

Confidential
 The Concept of IEC 61850

4 Confidential
 Why Spend Money on IEC 61850?
Conventional Substation

➢ Reduce point-to-point copper wiring costs

➢ Enhance management efficiency by
message passing
➢ Retrofit to IEC 61850 substation
IEDs
compatibility
IEC 61850 Substation
➢ Support protocol interoperability
➢ Increase high reliability and maintainability

Confidential
Recap

Evolution of Substation Automation

1975 1995 2010

GOOSE/
MMS
MMS

GOOSE/
SMV

Conventional IEC 61850 Station Bus IEC 61850 Station and Process Bus
Substation Automation Substation Automation Substation Automation

A Journey from end-to-end hard-wired connectivity

to Ethernet-based communication
From Network Communication Perspective
What is IEC 61850?
• A framework for substation automation
• An Ethernet-based communication
• Multi-vendor interoperability
• High speed IED to IED communication and guaranteed delivery
times
The transfer time requirements for functions

Reference: IEC 61850-5

7 Confidential
 IEC 61850 Station Architecture
- with 3 Levels & 2 Communication Bus
Station Level
Power SCADA
Station Level Equipment
Monitoring

Station Bus

Bay Level
Bay Control Bay Protection
Protection &
Control

Process Bus
Process Level
Current/ Voltage
Data Acquisition &
Transformer Circuit Breaker Circuit Breaking
Levels in Substation Automation Systems
Reference: IEC 61850-5

8 Confidential
 IEC 61850 Station Architecture
- with 3 Levels & 2 Communication Bus
Station Level Computing Devices:
• Back-end Host
Station Level Equipment • Tele-control Computer
• Protocol Converter
• Protection Management Computer
Station Bus Communication Devices:
Bay Level • Secondary Device Networking
• IEEE 1588 Transparent Clock
Bay Control Bay Protection
• Protocol Converter
• Meter Networking
Process Bus
Process Level
Instrumental Transformer Switchgear

Levels in Substation Automation Systems

Reference: IEC 61850-5

9 Confidential
 IEC 61850 Communication

10 Confidential
 IEC 61850 Communication

Management Protection & Data Acquisition

Control

MMS traffic SV traffic

GOOSE traffic
to monitor the to transmit the power
substation status to update status and
line current and voltage
send control command
value

MMS = Manufacturing Messaging Specification

GOOSE = Generic Object Oriented Substation Event
SV = Sampled Values
Reference: IEC 61850-90-4

11 Confidential
 IEC61850 Communication
Step5.
ALARM
ALARM
ALARM
Power SCADA Server

Step4.
MMS The power has
been cut!
Station Bus

↑ Step3.
Okay I cut the power!
Protect Relay ↓ Step2.
SMV GOOSE It’s over current, trip the
circuit breaker!
Process Bus Step1.
Current
Current POWER LINE
Current
Over Current!!
Merging Unit Intelligent Control Unit

12
 MMS

Power SCADA Implementation with MMS

MMS is an internationally standardized messaging system for network
management within substation, for exchanging real-time data and supervisory
control information between networked devices and/or computer applications.

• Recommended by IEC 61850

• Friendly Power SCADA interface
• Easy and logical fault analysis on
single platform, the familiar
Power SCADA
• No more money and efforts spent
on different NMS
GOOSE Protocol
• GOOSE Characteristic
– Publisher & Subscriber (Layer 2 multicast)
– Always with VLAN tag and filter by VLAN
– Time critical
– Need fast failure recovery
IED
GOOSE
(Subscriber)

IED IED
GOOSE Ethernet GOOSE
(Publisher) (Subscriber)

GOOSE IED
(Subscriber)

Confidential
 GOOSE

GOOSE Communication

Switch

GOOSE.block

! Bay Level
Feeder Feeder Busbar Dist
Protection Protection Protection Protection

GOOSE.trip
SV SV SV SV

intelligent intelligent intelligent

Merging Merging Merging Merging intelligent
control control control
Unit unit Unit unit Unit unit Unit control unit

CT/ VT CB CT/ VT CB CT/ VT CB CT/ VT CB

Confidential
Performance Enhancement for SAS
IEC 61850 QoS – GOOSE/SMV/PTP prioritization

IEC 61850 QoS

• Packet type examination
and guarantee higher priority
to critical packet
GOOSE/SV/PTP • System-based configuration

Standard QoS
High Queue
GOOSE • Packet prioritization by port-
Medium Queue based configuration
• No packet type examination:
all packet in the high queue
Normal Queue
still FIFO

Low Queue

Confidential
 Substation Communication Practice

17 Confidential
Logical Network Topology Design
in Communication Bus

• Design principle of network topologies

– Hierarchy
– Modularity
– Redundancy
– Security

18 Confidential
 Logical Network Design in Communication Bus
- Design principle of network topologies
Hierarchy Modularity
-> Increase traffic performance -> Easy add a new bay or application
Station Bus
(Core Layer)

Topology
Station Bus Design
(Distribution Layer)

Bay1 Bay2 Bay3 Bay4 Bay5

Process Bus
(Edge Layer)

Redundancy Security
-> Increase availability -> Protect and defend against attacks
Firewall
Connect Redundancy Redundant Security Guide
Protocol
Planning for Physical Security

Bay1 Bay2 Bay3 Bay4 Bay5

Bay1 Bay2 Bay3 Bay4 Bay5

19 Confidential
Redundancy

Network Redundancy
Without Path Redundancy With Path Redundancy Ex: RSTP

switch switch switch switch

Backup path (Block)

switch switch switch switch

Backup path (Enable)

Without Device Redundancy With Device Redundancy

Backup path (Block)

switch switch switch

switch
Backup path (Block)
switch

Backup path (Enable)

switch switch
switch switch

switch

Confidential
Redundancy

Network Redundancy

Confidential
Redundancy Protocol

Redundancy Protocol Recovery Time Public / Private

RSTP (Rapid spanning Tree Protocol) 200ms-10s Public

Turbo Ring <300ms @120hops Moxa Private

Turbo Ring V2 <20ms @250hops Moxa Private

(Fast Ethernet)

Turbo Chain <20ms @250hops Moxa Private

(Fast Ethernet)

PRP (Parallel Redundancy Protocol) 0ms Public

HSR (High-availability Seamless Redundancy ) 0ms Public

22 Confidential
 Redundancy
LAN B LAN

PRP Operation
A
SAN switc
Inter Link h

To transmit or receive two independent active paths to/from Redundancy Single Attached General
Box Node (SAN) Ethernet Switch
different LANs simultaneously in a zero recovery time network
Source
SAN SAN PRP LAN A: Red Line
PRP LAN B: Green Line
SAN Inter Link: Blue
“C” frame Line
Inter Link Inter Link

RedBox PRP “B” frame RedBox

LAN A LAN B LAN A LAN B
PRP “A” frame

LAN A Malfunction !!
switch switch

LAN A LAN B
switch switch switch switch

PRP “B” frame

LAN A LAN A LAN B

LAN B
RedBox PRP “A” frame RedBox

Inter Link Inter Link Well received without

“D” frame any packet loss &
system downtime.

SAN SAN
Zero Recovery Time
Destination

23
 Redundancy LAN B LAN A

HSR Operation Inter Link

Redundancy
SAN

Single Attached Node

Every frame is duplicated and then transmitted in both directions of the Box (SAN)
HSR ring to deliver zero recovery time Source

SAN SAN SAN SAN HSR Ring: Orange Line

SAN Inter Link: Blue
Line
“C” frame
Original frames will Inter Link Inter Link Inter Link Inter Link
be dropped by the
RedBox RedBox RedBox RedBox
original RedBox to
prevent looping LAN A LAN B LAN A LAN B LAN A LAN B LAN A LAN B
Drop !! Drop !!

HSR “A” frame HSR “B” frame

Part of System HSR Ring
Malfunction!!

LAN B LAN A LAN B LAN A LAN B LAN A LAN B LAN A

RedBox RedBox RedBox RedBox

Inter Link Inter Link Inter Link Inter Link

“D” frame

SAN SAN SAN SAN

Destination Well received without

any packet loss &
system downtime.

Confidential
 Moxa PRP/HSR Solutions
Embedded-level to Management-level for Higher Interoperability

Managed-level Board-level Integration Device-level Integration

Integration

PRP/HSR Computer PRP/HSR Embedded Module PRP/HSR RedBox PRP/HSR 19” Ethernet Switch
DA-820 EOM-G103-PHR-PTP PT-G503-PHR-PTP PT-7728-PTP
PM-7200-PHR-PTP*
Efficient and Visually Easy integration to single Quickly enable legacy Enable multiple single
Represented PRP/HSR attached node(SAN) system devices to PRP/HSR network attached nodes(SANs) to
Network Management and enable PRP/HSR network PRP/HSR network

Module built-in

SAN SAN SAN SAN SAN

Available Now Available Now Available Now 2016. Q1

Confidential
PT-G503 Highlight
3 ports, 10/100/1000Mbps connection
- 2 x 100/1000MBaseT(X) for PRP/HSR redundant
- 1 x 10/100/1000MBaseT(X) for end device

IEEE 1588v2 PTP

One-step or (2 to 1-step) conversion end-to-end
and peer-to-peer* TC mode

Fiber CheckTM-- All fiber port monitoring

MMS Server built-in for easy integration

with Power SCADA

RSTP Transparent

Confidential
PRP/HSR Switch Highlight

• World’s 1st rackmount modular switch

– Flexible combination by modularity design
– Minimum mean time to repair(MTTR)
• Multiple redundant protocols on one device
– PRP/HSR + Turbo Ring or Turbo Chain or RSTP or RSTP transparent
• IEEE 1588 v2 PTP support (1-step, 2-step, TC/BC, P2P/E2E)

Confidential
 Topology

28
 Logical Network Design in Communication Bus
- Design principle of network topologies
Hierarchy Modularity
-> Increase traffic performance -> Easy add a new bay or application
Station Bus
(Core Layer)

Topology
Station Bus Design
(Distribution Layer)

Bay1 Bay2 Bay3 Bay4 Bay5

Process Bus
(Edge Layer)

Redundancy Security
-> Increase availability -> Protect and defend against attacks
Firewall
Connect Redundancy Redundant Security Guide
Protocol
Planning for Physical Security

Bay1 Bay2 Bay3 Bay4 Bay5

Bay1 Bay2 Bay3 Bay4 Bay5

29 Confidential
 Topology

Legacy Network Communication

Topology – Single switch
Network
Printer PSCADA Clock
Topology Characteristic
• Simple topology
• One level switch
• Easy to ensure adequate bandwidth
Switch • Low latency
IED IED IED IED • No redundancy
•The bridge becomes a bottleneck
IED IED IED IED
when the number of IEDs is large
Bay1 Bay2 Bay3 Bay4

Reference: IEC 61850-90-4 7.3.1.1.1

30 Confidential
 Topology

IEC 61850 Network Communication

Station Bus #1
Topology - Ring of RSTP
Network
Printer PSCADA Clock
Topology Characteristic
• Simple topology
• Slight difficult to ensure adequate
Switch bandwidth
RSTP • Latency increases with the number
Switch Switch Switch Switch of switches in series
• Redundancy recovery time is
IED IED IED IED
200ms-10s
IED IED IED IED (Redundancy against trunk link failures in the
ring)
Bay1 Bay2 Bay3 Bay4

Reference: IEC 61850-90-4 7.3.1.2.1

31 Confidential
 Topology

IEC 61850 Network Communication

Station Bus #2
Topology - Ring and subrings with RSTP
Network
Printer PSCADA Clock
Topology Characteristic
• Traffic can be segmented by the
switches
Switch
• Slight low latency
RSTP
Switch Switch
• One path redundancy on the rings
RSTP RSTP • Redundancy recovery time is
Switch Switch Switch Switch 200ms-10s

IED IED IED IED

IED IED IED IED

Bay1 Bay2 Bay3 Bay4

Reference: IEC 61850-90-4 7.3.1.3.1

32 Confidential
 Topology

IEC 61850 Network Communication

Station Bus #3
Topology - Ring and subrings with Turbo Ring/Turbo Chain
Network
Printer PSCADA Clock
Topology Characteristic
• Traffic can be segmented by the
switches
Switch
• Slight low latency
Turbo Ring
Switch Switch Switch Switch
• One path redundancy on the rings
Turbo Chain Turbo Chain • Turbo Ring/Turbo Chain recovery
Switch Switch Switch Switch time is <20ms (Fast Ethernet)

IED IED IED IED

IED IED IED IED

Bay1 Bay2 Bay3 Bay4

Reference: IEC 61850-90-4 7.3.1.3.1

33 Confidential
 Virtual LAN
34 Confidential
 VLAN

Applications by Using VLANs within

IEC 61850 Substation

• Substation Event Communication

– GOOSE Communication
• Asset Management System
– Video Surveillance System

Reference: IEC 61850-90-4 12.1

35 Confidential
 VLAN

The Benefit by Using VLANs

• Traffic filtering
– To segregate traffic flow throughout the network
– The traffic can be filtered or restricted to specific
domains in the network by VLAN
• Trunk traffic reduction
– Removing bottlenecks on the trunk links
• Device performance improvement
– Reduce the amount of messages that the end devices
must process

Reference: IEC 61850-90-4 12.1

36 Confidential
 VLAN

VLAN Setting within a Substation

Station Level
Control Room
VLAN 10-50 VLAN 90
HMI IP Surveillance Server

Printer IP Phone

VLAN 90 VLAN 90 VLAN 90 VLAN 90 VLAN 90

Bay1 Bay2 Bay3 Bay4 Bay5

IED IED IED IED IED IED IED IED IED IED
IED IED IED IED IED IED IED IED IED IED
IED IED IED IED IED IED IED IED IED IED
VLAN 10 VLAN 20/50 VLAN 10/30 VLAN 40 VLAN 50

Bay Level
37 Confidential
 Time Synchronization

38 Confidential
The Purpose of Using Time Synchronization within
Substation Automation (SA)

• Enable smooth power transfer and maintain power

supply integrity
• System control and data acquisition achieve
accurate control and precise global analysis of
network response and when, where and why any
faults have occurred

Confidential
 IEEE 1588 V2 PTP

IEEE 1588 V2 PTP Benefits

Fulfill Fulfill
NOT
Time Typical IEC 61850 IEC 61850 Scales well with
required
Synchronization Accuracy in Station bus Process bus large number of
Dedicated
Protocol substation requirement requirement device
cabling
(1ms) (1us)

IRIG-B 1ms/100us v
(AM/DC-shifted)

NTP/SNTP 1-10ms v v

IEEE 1588 V1 1us v v v

IEEE 1588 V2 1us v v v *v

Time Accuracy Cost effective Reduce Traffic Loading

* The most important feature in IEEE 1588 v2 is the peer-to-peer path delay measurement and Transparent clock

Confidential
Synchronization Process in IEEE 1588 v2 PTP

• PTP is a master-slave hierarchy and the

synchronization process has two steps:
1. Offset Correction:
• Correct the time difference between the master clock and
slave clock
2. Delay measurement:
• Determine the run time between the master clock and slave
clock

Confidential
Clock Modes
Clock Mode Description
• In Top-Level, located in a Time domain
Grand Master • Connects the Reference clock (GPS or atomic clock)
Clock

• Located in a Time sub-domain.

Master Clock • A Grand Master Clock or Boundary Clock.
• Send Sync Message

• Has a slave port to sync with Master clock and has multiple
Boundary Clock master port to control time in a sub-domain
(BC)

• Forward Sync Message

Transparent Clock • Calculated Residence time to the correction field
(TC) • Delay request-response & Peer-delay

• Has only one PTP port

Ordinary Clock • A Slave clock or Master Clock.
(OC)

Confidential
IEEE 1588 V2 PTP

What is the difference between Boundary Clock and

Transparent Clock?

Boundary Clock Boundary Clock Sync Mechanism

M Master Port Sync Sync Sync Sync

S Slave Port

M M M
* Nonlinear decreasing synchronization accuracy in highly
S cascaded networks

Transparent Clock Sync Mechanism

M M M Sync

* A boundary clock has one slave

ports and multi master ports

Confidential
 IEEE 1588 V2 PTP

IEEE 1588V2 PTP Application Architecture

Architecture Practice
Printer Master Clock
PSCADA
(IEEE 1588 Reference Clock)
Time Synchronization
GPS
Master Clock
Switch
Sync Message TC
TC TC TC TC

Transparent Switch Switch Switch Switch

Clock (TC) IED IED IED IED

Sla Sla Sla Sla
Sync Message ve
IED ve
IED ve ve
IED
IED
Sla Sla Sla Sla
Transparent Bay1ve Bay2
ve ve
Bay3 Bay4
ve
Clock (TC) TC
TC
Sync Message Switch Switch

Slave MU MU MU
Slav Slav Slav
e e e

44 Confidential
 Moxa PRP/HSR Solutions
Embedded-level to Management-level for Higher Interoperability

Managed-level Board-level Integration Device-level Integration

Integration

PRP/HSR Computer PRP/HSR Embedded Module PRP/HSR RedBox PRP/HSR 19” Ethernet Switch
DA-820 EOM-G103-PHR-PTP PT-G503-PHR-PTP PT-7728-PTP
PM-7200-PHR-PTP*
Efficient and Visually Easy integration to single Quickly enable legacy Enable multiple single
Represented PRP/HSR attached node(SAN) system devices to PRP/HSR network attached nodes(SANs) to
Network Management and enable PRP/HSR network PRP/HSR network

Module built-in

SAN SAN SAN SAN SAN

Available Now Available Now Available Now 2016. Q1

Confidential
Delay Measurement

• IEEE 1588 v2 PTP provides two delay measurements

– Delay request-respond
• Master- Slave, End-to-End Transparent Clock to Slave
– Peer delay
• Peer-to-Peer Transparent Clock to Slave

Confidential
 End-to-End TC and Peer-to-Peer TC mode

End-to-end TCs in ring topology Peer-to-peer TCs in ring topology

Confidential
The reason we prefer Peer-to-Peer TC

• End-to-End TC
– Path delay request messages must be answered by the
master clock that in a large network can cause a traffic and
processing bottleneck at the master

• Peer-to-Peer TC
– Path symmetry is guaranteed and there will never be
processing or traffic overloading due to the one to one
relationship
– Better time synchronization convergence than end-to-end
TC when the network recovery happens

Confidential
Confidential
 Thank You

50
© 2014 Moxa Inc. All rights reserved.