Monitoring of Ethernet

Messages

IEC 61850 Seminar

Dr. Alexander Apostolov

K02 03 20060309

Page: 1

OMICRON

K02 03 20060309

Scope
Discuss Ethernet architectures supporting
IEC 61850 projects
Not defined in the standard but essential for a
real project !

Background on Ethernet principles and

technologies
Practical examples

OMICRON

K02 03 20060309

Page: 2

Transmission Definitions

SIMPLEX
Transmission in one direction only
HALF DUPLEX
Two way means of transmission but data can only
travel in one direction at a time
FULL DUPLEX
Transmission in both directions simultaneously

OMICRON

K02 03 20060309

Page: 3

Transmission Definitions

PROTOCOL
Rules and procedures that communications networks
use to communicate on the communications medium
CONNECTION
Communications are Connection Oriented or
Connectionless

OMICRON

K02 03 20060309

Page: 4

Transmission Definitions
Point-to-point
Engineering
Station

Laptop

Modem

IED

Modem

IED

OMICRON

K02 03 20060309

Page: 5

Transmission Definitions
Shared Access
Ethernet

Substation
HMI

IED

IED

IED

Ethernet
Switch

Proxy
Server
RS485
Legacy
IED

Substation
HMI
Ethernet
IEC 61850
IED

IEC 61850
IED

OMICRON

K02 03 20060309

Page: 6

Transmission Definitions
NETWORK ACCESS
CARRIER SENSE METHODS
CSMA (Carrier Sense Multiple Access)/CD
(Collision Detection)
Medium Access Control (MAC)
TOKEN ACCESS METHOD
The device that has the Token has access to
transmit
RESERVATION METHOD
Each device has a predefined time slot to transmit

OMICRON

K02 03 20060309

Page: 7

Transmission Definitions
Medium - Ethernet devices attach to a common medium that
provides a path along which the electronic signals will travel:
- historically, this medium has been coaxial copper cable
- more commonly a twisted pair
-fiber optic cabling.
Segment - a single shared medium as an Ethernet segment.
Nodes - devices that attach to that segment are stations or
nodes.
Frame - The nodes communicate in short messages called
frames, which are variably sized chunks of information.

OMICRON

K02 03 20060309

Page: 8

Transmission Medium
RS 232
EIA 232

- 9/25 pin D connector

- Maximum data rate 20kb/s
- Maximum distance 50 feet

RS 485
EIA 485

- 2/4 conductors
- Maximum data rate 10Mb/s
- Maximum distance 4000 feet

Optical fiber - Immunity to electrical interference

- Advantages in distance and speed
Wireless

OMICRON

K02 03 20060309

Page: 9

Data Security
Noise corruption of data

Parity bit check

Two coordinate parity check
Checksums
Cyclic redundancy check

Unauthorized access

Password protection
Modem dial back
Firewalls

OMICRON

K02 03 20060309

Page: 10

Open Systems Interconnection

(OSI) Model
Application
Presentation
Session
Transport

Selects appropriate service for application

Provides code conversion, data reformatting
Coordinates interaction between end
application process
Provides for end to end data integrity and
quality of service

Network

Switches and routes information

Data Link

Transfers unit of information to other end

of physical link

Physical

Transmits bit stream to medium

OMICRON

K02 03 20060309

Page: 11

OSI Stack
PCI =Protocol Control Inform.

Commands or data

PDU =Protocol Data Unit

H =Header

PCI

T =Trailer

PCI
PCI

fragment

Transport
Network

PDU

PCI

Data link

PDU

PDU
fragment

Presentation
Session

PDU
PDU

PCI

PDU

PDU

Application

PDU
fragment

Physical

Frames (Ethernet, token ring, etc)

OMICRON

K02 03 20060309

Page: 12

Communications Process
Upper Layer

Upper Layer

SAP

SAP

Entity with

Peer-to-peer dialog

Entity with

services to offer

services to offer

SAP

SAP

Lower Layer

Lower Layer

OMICRON

K02 03 20060309

Page: 13

Ethernet Frame
Pre SFD DA SA Length
Type
7 1 6 6
2

MAC Data + Pad

FCS

46-1500bytes

Pre: The Preamble is an alternating pattern (7 bytes) of 1 and 0 that

tells receiving stations that a frame is coming
SFD: Start-of-frame delimiter (1 byte: 10101011) indicating that the
next bit is the left-most bit in the left-most byte of the destination
address.
DA: Destination address (6 bytes) identifies which station(s) should
receive the frame
SA: Source addresses (6 bytes) identifies the sending station

OMICRON

K02 03 20060309

Page: 14

Ethernet Frame
Pre SFD DA SA Length
Type
7 1 6 6
2

MAC Data + Pad

FCS

46-1500bytes

Length Type: Number of MAC-client data bytes that are contained

in the data field of the frame

MAC Client Data: A sequence of n bytes (46=< n =<1500) of any

value. (The total frame minimum is 64 bytes). The Pad contains (if
necessary) extra data bytes in order to bring the frame length up to
its minimum size. A minimum Ethernet frame size is 64 bytes from
the Destination MAC Address field through the Frame Check
Sequence.

FCS: The Frame Check Sequence is a 32-bit cyclic redundancy

check (CRC) value

OMICRON

K02 03 20060309

Page: 15

Ethernet
Media Access Control (MAC) Address This is the physical address of any device,
such as the NIC in a computer, on the
network.
The MAC address has two parts, each 3
bytes long.
The first 3 bytes identify the company that
made the NIC.
The second 3 bytes are the serial number
of the NIC itself.

OMICRON

K02 03 20060309

Page: 16

Ethernet
Unicast - A transmission from one node
addressed specifically to another node.
Multicast - When a node sends a
packet addressed to a special group
address. Devices that are interested in
this group register to receive packets
addressed to the group.
Broadcast - When a node sends out a
packet that is intended for transmission
to all other nodes on the network.

OMICRON

K02 03 20060309

Page: 17

Network Terminology
Network

An interconnected group of nodes or

stations linked by communication channels

Node

The interface point where one or more

functional units are connected

LAN

Local area network (<5km)

WAN

Wide area network

Network topology

Pattern of nodes and their interconnection

OMICRON

K02 03 20060309

Page: 18

Ethernet Basic Principle

1. Listen and broadcast if the wire is free
2. If there is a collision then re-transmit in an interval [0, 2N-1]

Ethernet

Collisions are an issue for real time automation

OMICRON

K02 03 20060309

Page: 19

Switches principles
Internal Switches
Queue

Switches eliminate collisions and are thus

systematically used for industrial applications
OMICRON

K02 03 20060309

Page: 20

Switches: Performances
Pure delay (store & forward): 5-10s
Maximum frame size: 1536 bytes
1536 bytes @ 100 Mbps = 123 s
Goose size << 1536 bytes

Performances bottleneck is no longer the communication

Network but possibly the applications (ms timefame)
OMICRON

K02 03 20060309

Page: 21

Switches: Unicast

OMICRON

K02 03 20060309

Page: 22

Switches: Multicast

OMICRON

K02 03 20060309

Page: 23

Switches: Broadcast

OMICRON

K02 03 20060309

Page: 24

To

Process
Binary Input
Filtering [2-10]
Logical [1-50]

Wall-to-wall performances
example (ms)
[5-75]

Communication [2-15]

Switching [0.1]

Communication [2-15]
Logical [1-50]

[4-72]

Relaying [1-7]
Process

To + 9/147

Binary Output
OMICRON

K02 03 20060309

Page: 25

Switches: other features used in

Substation Automation
Substation environment !
Electro-magnetic compatibility
DC power supply with 20ms voltage dips, possibly
redundant

Priority management (802.3p):

Capability to have priority queues in order to further
boost the communication performances

Virtual LAN VLAN (802.3q):

Capability to create logical groups of devices in order
to filter the messages not belonging to one group

OMICRON

K02 03 20060309

Page: 26

Switches: other features used in

Substation Automation
Management
Capability to detect a switch failure: SNMP or watchdog

Redundancy management
No standard today for hard real time redundancy
Redundancy between switches (not applications)
See further slides

OMICRON

K02 03 20060309

Page: 27

Base architectures
STAR

Basic scheme since

failure of the central
switch leads to the
total communication
failure

RING

The most common scheme since

good performance/cost ratio
(save 2 central switches vs. start
configuration)

DOUBLE STAR

The most secured

scheme since multiple
failures are tolerated
except the two central
switches
simultaneously

Failure of a single fiber or device

does affect the system once
reconfiguration is completed

OMICRON

K02 03 20060309

Page: 28

Redundancy management
A way to improve system availability and reliability
Behavior is not defined in IEC 61850
Typical example where the architect needs to define what
shall be done and first check on paper that it can be
supported by the IEDs

May be applied to:

Communication infrastructure: tolerate the loss of an optical
fiber or/and the loss of a central switch
Clients: capability for a client to continue the tasks initiated
by a first client (example: Graphical user interface)
Servers: capability for a client to switch to a redundant server
if the first one is not operational (example: CT/VT sensor)

99.99% availability requests the 3 types of redundancy

OMICRON

K02 03 20060309

Page: 29

Redundancy management at
communication level
Defined between Ethernet switches

If a device has two Ethernet port it must also have an integrated

switch

Spanning tree mechanisms

Recalculation of the route between switches using an internal

protocol between switches and avoiding loops
Base: 802.3d. Typically 30 s reconfiguration time
Fast: 803w. Typically 100 ms reconfiguration time, some
implementations claims 5 ms per switch (N x 5ms with a ring made of
N switches)

Other mechanisms

Detects the failure of the adjacent switch to reconfigure

For ring topology, less than 1 ms reconfigurations

Sends two signals at the same time and keep the first one arrived
No switch-over time, but need to keep a correct buffer size (dual homing)

Substation applications enable to get extremely fast

Switch-over time
OMICRON

K02 03 20060309

Page: 30

Example (ring)
Repeaters
(IEE 802.3 Ethernet switch)
+ Self Healing Manager (SHM)
Ethernet Ports 10/100 Base TX

EEPRO M

SW ITCH

Port MII

Failsafe
output relays

SHM

Self healing ring

m anager

100Mbs Full duplex

opto

opto

Prim ary Fiber

Optical Ring

Secondary Fiber

OMICRON

K02 03 20060309

Page: 31

Example (ring)
X During nominal situation, Ethernet packet goes in
primary fibre always in the same direction, and
only a checking frame (4 bytes) is sent every 5ms
in secondary fibre in the opposite direction.
Primary fibre

switch

switch
RP

10

switch

switch

switch

11

EP
B

ES

RS

Secondary fibre

OMICRON

K02 03 20060309

Page: 32

Example (ring)
If the connection between 2 switches is broken
X The Ethernet network will continue to run correctly.
X Both SHM start immediately the network self-healing.
At one side, the messages received are no more emitted to
the primary fibre but to the secondary fibre.

On the other side of the cut off, the messages received to secondary
are emitted to primary and the new topological loop is closed.
Primary fibre

switch

switch
RP

10

switch

switch

switch

11

EP
B

ES

RS

Secondary fibre

OMICRON

K02 03 20060309

Page: 33

Other communication elements

OSI Model

Technology

Application

Protocol
MMS/SNTP

IEC Com.

Com.

Services

Element

Report, Control,
File

Gateway/
Proxy

Presentation
Session

Software

Transport

TCP

Network

IP

Data Link

802.3

Physical

Hardware

Router
GOOSE

Bridge/Switch
/Proxy

802.3

Need to design the communication infrastructure

according to the system constraints and requirements
OMICRON

K02 03 20060309

Page: 34

Proxy use
GUI

T104,
http, etc.

Gateway

Client

Client

Client

Proxy

Proxy

IP Routable
Server

Server

Server

Fast Peer-to-Peer
Bay

Server (resp.
subscriber, clients)
performance is
dependant on the
number of clients
(resp. pubslisher,
servers)

Bay

Bay

Bay

Bay
Bay

A proxy can
replicate the real
time status &
measurement of a
series of servers
and be possibly
redundant

OMICRON

K02 03 20060309

Page: 35

IEC 62351
Data and Communication Security series of standard
Currently at CD stage (votes till August 2005)
IEC 62351-6: Security for IEC 61850 profiles
Relies on IEC 62351-4: Profiles including MMS
Relies on IEC 62351-3: Profiles including TCP
OSI Model

Technology

Application

Protocol

IEC Com.
Services

MMS/SNTP

Report, Control,
File

Security
Services
Authentification,
Replay

Presentation

Session

Software

Transport

TCP

Network

IP

Data Link

802.3

Physical

Hardware

Authentification,
Encryption, Replay

GOOSE

Authentification,
Replay

802.3

OMICRON

K02 03 20060309

Page: 36