You are on page 1of 61

RuggedCom

Industrial Strength Networks

Industrial Ethernet
Issues and Requirements
Marzio Pozzuoli
RuggedCom Inc. - Industrial Strength Networks
Concord, Ontario, Canada
Presented at University of Toronto
for the
IEEE Instrumentation & Measurement Society / IEEE Communications Society

RuggedCom
Industrial Strength Networks

Key Topics Covered

The Emergence of Industrial Ethernet Ethernet everywhere!


Industrial automation and process control applications
Electric power utilities substation automation applications
Future growth and emerging dominance of Ethernet for industrial applications

The Industrial Environment No place for the faint of heart!


Key issues and requirements for harsh industrial / substation environments
EPRI and Rockwell studies confirm what most engineers already knew.
IEC 61000-6-2, IEC 61850-3, IEEE P1613 Standards to the rescue

Real-time, Deterministic Performance Look Ma, No Collisions!


Technological improvements in modern Ethernet technology - facts and myths.
IEEE 802.3x, 802.1p, 802.1Q, 802.1w, IGMP Snooping / Filtering
Fault tolerant network architectures.

A comparison of Industrial Ethernet protocols Different Strokes..


Offerings from major industrial automation OEMs
Key highlights.

RuggedCom
Industrial Strength Networks

The Emergence of Industrial Ethernet


Ethernet everywhere!

RuggedCom
Industrial Strength Networks

Industrial Ethernet
Ethernet on the plant floor

RuggedCom

Ethernet A Brief History: From back office to factory floor

Industrial Strength Networks

Xerox PARC invention back in the 70s!


Original version based on CSMA/CD technology for use with multiple devices
in a bus architecture over coax cable.
Supported 10Base5 (5Mbps) and cheaper 10Base2 (2Mbps) speeds

Dominant LAN technology at the enterprise layer in the 80s & 90s!
Higher speeds (10/100/1000Mbps) help drive the PC networking revolution.
Switching Hub (Bridge) Technology eliminates collisions and improves
performance.

Emergence of Industrial Ethernet in the new millennium.


Major automation OEMs begin incorporating Ethernet ports into their devices.
All popular industrial Fieldbus protocols ported to work over Ethernet
Major automation OEMs begin to promote Ethernet for process control and
factory automation.
5

RuggedCom
Industrial Strength Networks

Schneiders
Schneiders
Transparent
Transparent Factory
Factory

Siemens
Siemens ITlution
ITlution

All Major Automation OEMs Have Offering

EtherNet/IP
EtherNet/IP

IT
ABBs
ABBs -- Industrial
IndustrialIT

6
All
All have
have one
one thing
thing in
in common
common Ethernet
Ethernet technology
technology on
on the
the factory
factory floor!
floor!

RuggedCom
Industrial Strength Networks

Schneider Electrics Transparent Factory

Schneiders
Schneiders -- Transparent
Transparent Factory,
Factory, based
based on
on Modbus/TCP
Modbus/TCP over
over
Ethernet.
Ethernet.

RuggedCom
Industrial Strength Networks

Allen-Bradleys EtherNet/IP

A-Bs
A-Bs -- EtherNet/IP:
EtherNet/IP: The
The IP
IP Stands
Stands for
for Industrial
Industrial Protocol.
Protocol.
ControlNet
ControlNet &
& DeviceNet
DeviceNet Application
Application Layers
Layers over
over Ethernet.
Ethernet.

RuggedCom
Industrial Strength Networks

Ethernet in the
Substation!
Even electric utilities do it

RuggedCom
Industrial Strength Networks

Ethernet in the SubstationWhy?

Back in the 90s North American Utilities were seeking a common


communications architecture for substation and utility automation.
Every IED vendor supported their own, often proprietary, protocols.
Protocol converters were required in order to integrate multi-vendor IEDs.
Costs were high, performance was low!

Utilities and vendors under the auspices of EPRI developed the Utility
Communications Architecture (UCA2.0 / IEC 61850)
A collection of standards to allow for a Utility communications architecture
Supporting: multi-vendor IED interoperability, real-time control over a
substation LAN and a seamless flow of information across the entire Utility
enterprise.

Ethernet chosen as the underlying technology for UCA / IEC 61850


Ethernet is the most prevalent local area network technology in the world.
Large installed base: over 95% of all back office or enterprise layers.
Non-proprietary technology with multiple vendors.
No single IED vendor would have an advantage over the other.
10

RuggedCom
Industrial Strength Networks

Traditional Substation

Inter
Inter IED
IED control
control signaling
signaling done
done via
via wiring
wiring or
or low
low speed
speed serial
serial communications.
communications.
Limited
Limited to
to simple
simple schemes
schemes due
due to
to cost
cost and
and complexity
complexity of
of wiring
wiring
and
and limited
limited performance
performance capabilities
capabilities of
of communications.
communications.

11

RuggedCom
Industrial Strength Networks

Connecting IEDs via the LAN

ALL
ALL IEDs
IEDs are
are connected
connected via
via aa single
single connection
connection (or
(or dual
dual redundancy)
redundancy) to
to the
the
LAN.
LAN.
Simple
Simple or
or complex
complex control
control schemes
schemes are
are possible
possible with
with no
no increased
increased wiring
wiring
costs
costs or
or complexity.
complexity.
Other
Other real-time
real-time data
data (e.g.
(e.g. analog,
analog, status)
status) can
can be
be shared
shared across
across ALL
ALL IEDs.
IEDs.
12

RuggedCom

The LAN Based Substation

Industrial Strength Networks

IEC 61850 (UCA)


Substation LAN

PC

10/100Mbps Ethernet
E th e rn e t S w itch

Digital Relay

Metering IED

IEC 61850-9-2
Process Bus LAN

RTU

PLC

LTC

E th e rn e t S w itch

100Mbps Ethernet

Digital
CT/VTs

Modern
Switchgear

13

RuggedCom
Industrial Strength Networks

GEs Substation Vision

Substation Automation Applications (GE Energy Services)


14

RuggedCom
Industrial Strength Networks

SELs Substation Vision

Substation Automation Applications (SEL Architecture)

15

RuggedCom
Industrial Strength Networks

ABB/ALSTOM/Siemens Vision IEC 61850-9-2 Substation

Typical Utility Process Bus (IEC 61850) Network (Siemens PTD)

16

RuggedCom
Industrial Strength Networks

The Proliferation of Ethernet for Industrial/Process Control

Worldwide Shipments of Industrial IED's by Network


Interfaces (% Shipped)

30
25

26.3

2000
2005

20

20.1
18

15
10
6

11.1

9.2

7.78.2
4

8.5

3.5

t
rn
e
Et
he

D
s

Pr
of
ib
u

FF

Bu
s

I/O

-B

Re
m

ot
e

Ne
t
ev
i

ce

bu
s
M
od

0.1

Source: VDC Global Markets and User Needs for Industrial Distributed/Remote I/O, Second Edition

Ethernet is becoming the new RS232 for Process Control.


17

RuggedCom

Industrial Ethernet Growth Projections

Industrial Strength Networks

Industrial Devices with Ethernet Ports

5,000,000

4600000

4,000,000
3,000,000

2900000

2,000,000
1,000,000
0

1200000
718,000

2002

2003

2004

2005

Source: Ethernet At The Device Level Worldwide Outlook by ARC Advisory Group, June 2001

CAGR = 110% (2002 2005)!

18

RuggedCom
Industrial Strength Networks

The Industrial Environment


No place for the faint of heart!

RuggedCom
Industrial Strength Networks

What is Industrial Ethernet

Ethernet designed for industry


Standard IEEE 802.3 in an industrialized design
Higher temperature ranges
Rugged and metal housing
Fan-less products that withstand vibrations
Industrial connectors and cables CAT5E
High speed redundancy
EMC Immunity
Source: Bill King Siemens Energy & Automation (ISA 2002 Conference Presentations)

Higher Reliability and Availability

Good qualitative definition but the devil is in the details


Higher temperature ranges - how much higher?
EMC Immunity which standards and what levels?
20

RuggedCom
Industrial Strength Networks

Industrial

The Substation / Industrial Environment


Substation

Electric Fields
Phenomena
Phenomena Encountered
Encountered
Magnetic Fields
Electrostatic Discharge
Conducted High Frequency Electrical Transients
High Energy Power Surges
Ground Potential Rise during ground faults
Climactic Variation: Temperature & Humidity
Seismic / Vibration
Pollution: Dust, Metallic Particles, Condensation, Solar Radiation
21

RuggedCom
Industrial Strength Networks

IEC 61000-6-2the bare minimum!

IEC 61000-6-2 Generic Immunity Requirements for Industrial Environments


IEC 61000-6-2 (Immunity)
IEC 61000-4-2
IEC 61000-4-3
IEC 61000-4-4

IEC 61000-4-5

IEC 61000-4-6
IEC 61000-4-8
IEC 61000-4-11

CE Requirements
Generic Immunity Standard for Industrial Environments
Enclosure Contact
ESD
Enclosure Air
Radiated RFI
Enclosure ports
Signal ports
Burst (Fast Transient)
D.C Power ports
A.C. Power ports
Signal ports
Surge
D.C Power ports
A.C. Power ports
Signal ports
D.C Power ports
Induced (Conducted) RFI
A.C. Power ports
Earth ground ports
Magnetic Field
Enclosure ports
A.C. Power ports
Voltage Dips & Interrupts

Test Levels
+/- 4kV
+/- 8kV
10 V/m, 80 to 1000Mhz
+/- 1kV @ 5kHz
+/- 2kV @ 5kHz
+/- 2kV @ 5kHz
+/- 1kV line-to-earth
+/- 0.5kV line-to-earth/line
+/- 2kV line-to-earth, +/- 1kV line-to-line
10V @ 0, 5-80 MHz
10V @ 0, 5-80 MHz
10V @ 0, 5-80 MHz
10V @ 0, 5-80 MHz
30 A/m @ 50, 60 Hz
>95% reduction for 250 periods

Pass/Fail Criteria
B
B
A
B
B
B
B
B
B
A
A
A
A
A
C

* Performace criterion A refers to continuous operation of the DUT as intednded during, and after test. Performance criterion B refers to spontaneous
recovery after the test with no loss of function or operational performance. Performance criterion C refers to allowable temporary loss of function with
recovery through the device controls.

IEC
IEC 61000-6-2
61000-6-2 required
required for
for CE
CE mark
mark compliance
compliance for
for electronic
electronic equipment
equipment
(e.g.
(e.g. PLCs)
PLCs) in
in industrial
industrial environments.
environments.
IEC
IEC 61000-6-2
61000-6-2 should
should be
be the
the minimum
minimum requirement
requirement for
for networking
networking
equipment
equipment in
in industrial
industrial environments.
environments.
22

RuggedCom
Industrial Strength Networks

What are industrial (factory floor) users specifying?

The following excerpt was taken from an RFP specification document


from a major manufacturer who is intent is to use Ethernet on the factory
floor:
Required and Desired Features of Industrial Ethernet Switches
1. The primary requirement for an industrial Ethernet switch is
that it be environmentally hardened to operate under the same
extremes in operating conditions (temperature, vibration,
humidity, etc.) as an industrial PLC (Programmable Logic
Controller).

Industrial
Industrial users
users are
are often
often taking
taking the
the common
common sense
sense approach
approach in
in specifying
specifying
that
that the
the Ethernet
Ethernet networking
networking equipment
equipment be
be us
us robust
robust as
as the
the IEDs
IEDs
connecting
connecting to
to it.
it.
23

RuggedCom
Industrial Strength Networks

EMI Phenomena - Frequency of Occurrence

Continuous
Continuous
Phenomena
Phenomena
Radiated
Radiated RFI
RFI
Induced
Induced RFI
RFI
Power
Power freq.
freq. Magnetic
Magnetic Field
Field
Slow
Slow Voltage
Voltage Variations
Variations
Harmonics,
Harmonics, Interharmonics
Interharmonics
Ripple
Ripple on
on d.c.
d.c. power
power supply
supply
Power
Power Frequency
Frequency Voltage
Voltage

Transient
Transient Phenomena
Phenomena
(High
(High Occurrence)
Occurrence)
Electrostatic
Electrostatic Discharge
Discharge
Voltage
Voltage Dips
Dips
Lightning
Lightning
HV
HV Switching
Switching by
by Isolators
Isolators
Reactive
Reactive Load
Load Switching
Switching

Transient
Transient Phenomena
Phenomena
(Low
(Low Occurrence)
Occurrence)
Power
Power Frequency
Frequency Variation
Variation
Power
Power System
System Faults
Faults
Short
Short Duration
Duration Power
Power Freq.
Freq.
Magnetic
Magnetic Fields
Fields

Devices in industrial / substation environments must deal with a


combination of EMI phenomena which are continuous and transient.
24

RuggedCom
Industrial Strength Networks

Devices are Hardened for the Substation


Substation LAN

E th e rn e t S w itch

Digital Relay

Metering IED

RTU

PLC

LTC

Type Test Standards


For Substation Equipment:
IEC 60255-x
IEEE C37.90.x

Devices connected to the substation LAN are specifically Hardened


for the substation environment.
What about the Ethernet LAN?
25

RuggedCom
Industrial Strength Networks

IEC 61850-3 Communications networks and systems in substations

New IEC Standard (Jan/2002)


Communications networks and systems
in substations

Reliability

EMC

Environmental

26

RuggedCom
Industrial Strength Networks

IEC 61850-3 (61000-6-5): Apparatus Port Definitions

Power
Power Ports
Ports
(P/S In)

Enclosure
Enclosure Port
Port

Signal
Signal Ports
Ports

APPARATUS

(Local)
(Field)

E th ern et S w itch

(P/S Out)
Functional
Functional Earth
Earth
Port
Port

(To HV)
(Telecom)

(Other than Safety)

A port is defined as a particular interface of the specified apparatus


with the external electromagnetic environment
Type tests are defined and assigned for each specific port type.
27

RuggedCom

IEC 61850-3 (61000-6-5): Substation Definitions

Industrial Strength Networks

CONTROL BUILDING

(Switch Yard)
High Voltage
Equipment

SHIELDED AREA
(if any)
TELECOM
ROOM

Power
Line
Carrier

(T)
Remote Earth

(L)

PROTECTION
IED KIOSK

(L)

(P)

(F)

(H)

(F)

(F)
Earth Network

Signal Port Connection


Types
Local (L)
Field (F)
HV Equipment (H)
Telecom Power Line Carrier (T)
Protected (P)

Ethernet Sw itch

Ethernet Sw itch

Typical Locations of
Substation Ethernet Equipment
28

RuggedCom

IEC 61850-3EMI Immunity Requirements

Industrial Strength Networks

UTILITY IEC 61850-3 (61000-6-5) Communications Networks and Systems In Substations (Jan 2002)
TEST

Description

IEC 61000-4-2

ESD

IEC 61000-4-3

Radiated RFI

IEC 61000-4-4

Burst (Fast Transient)

IEC 61000-4-5

Surge

IEC 61000-4-6

Induced (Conducted) RFI

IEC 61000-4-8

Magnetic Field

IEC 61000-4-29

Voltage Dips & Interrupts

IEC 61000-4-11
IEC 61000-4-12

Damped Oscillatory

IEC 61000-4-16

Mains Frequency Voltage

IEC 61000-4-17

Ripple on D.C. Power Supply

Enclosure Contact
Enclosure Air
Enclosure ports
Signal ports
D.C. Power ports
A.C. Power ports
Earth ground ports3
Signal ports
D.C. Power ports
A.C. Power ports
Signal ports
D.C Power ports
A.C. Power ports
Earth ground ports3
Enclosure ports
D.C. Power ports
A.C. Power ports
Signal ports
D.C. Power ports
A.C. Power ports
Signal ports
D.C. Power ports
D.C. Power ports

Test Levels
Severity Levels
+/- 6kV
3
+/- 8kV
3
10 V/m
3
+/- 4kV @ 2.5kHz
x
+/- 4kV
4
+/- 4kV
4
+/- 4kV
4
+/- 4kV line-to-earth, +/- 2kV line-to-line
4
+/- 2kV line-to-earth, +/- 1kV line-to-line
3
+/- 4kV line-to-earth, +/- 2kV line-to-line
4
10V
3
10V
3
10V
3
10V
3
40 A/m continuous, 1000 A/m for 1 s
N/A
30% for 0.1s, 60% for 0.1s, 100% for 0.05s
N/A
30% for 1 period, 60% for 50 periods
N/A
100% for 5 periods, 100% for 50 periods2
N/A
3
2.5kV common, 1kV differential mode @ 1MHz
3
2.5kV common, 1kV differential mode @ 1MHz
3
2.5kV common, 1kV differential mode @ 1MHz
30V Continous, 300V for 1s
4
30V Continous, 300V for 1s
4
10%
3

Issued
Issued January
January 2002
2002 in
in recognition
recognition of
of the
the proliferation
proliferation of
of Ethernet
Ethernet in
in the
the
Substation.
Substation.
More
More tests
tests and
and higher
higher test
test levels
levels than
than IEC
IEC 61000-6-2.
61000-6-2.
Reflects
Reflects the
the substation
substation environment.
environment.

29

RuggedCom
Industrial Strength Networks

IEC 61850-3 (61000-6-5): Performance Criteria

IEC 61000-6-5 Table 7 - Performance Criteria for the most relevant functions
(in descending order of criticality)
Functional requirements versus electromagnetic phenomena
Functions

Continous phenomena

Transient phenomena
with high occurance

Transient phenomena
with low occurance

Protection and teleprotection

On-line processing and


regulation
Metering
Short delay

Command and Control


Supervision

** No Delays or Data Loss **

Stop and reset

Man-machine interface
Alarm
Data transmission and
telecommunication
Data acquisition and storage

Measurement

Temporary loss, self


recovered

Short delay, temporary wrong indication


No loss, possible bit
error rate degradation

Temporary loss

Temporary degradation
Temporary degradation, self recovered

Off-line processing

Temporary degradation Temporary loss and reset

Passive monitoring

Temporary degradation

Self-diagnosis

Temporary loss

Temporary loss, self recovered

30

RuggedCom
Industrial Strength Networks

IEEE P1613 Coming soon

IEEE
IEEE working
working on
on North
North American
American equivalent
equivalent of
of IEC
IEC 61850-3.
61850-3.
31

RuggedCom

IEEE P1613EMI Immunity Requirements

Industrial Strength Networks

IEEE P1613 Draft Standard Environmental Requirements for Communications Devices Installed in Electric
Power Substations
TEST

Description

IEEE C37.90.3

ESD

IEEE C37.90.2

Radiated RFI

IEEE C37.90.1

Fast Transient

IEEE C37.90.1

Oscillatory

IEEE C37.90

Dielectric Strength

Enclosure Contact
Enclosure Air
Enclosure ports
Signal ports
D.C. Power ports
A.C. Power ports
Earth ground ports3
Signal ports
D.C. Power ports
A.C. Power ports
Signal ports
D.C. Power ports
A.C. Power ports

Test Levels
+/- 8kV
+/- 15kV
35 V/m
+/- 4kV @ 2.5kHz
+/- 4kV
+/- 4kV
+/- 4kV
2.5kV common mode @ 1MHz
2.5kV common & differential mode @ 1MHz
2.5kV common & differential mode @ 1MHz
2kVac
2kVac
2kVac

Severity Levels
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A

Borrows
Borrows heavily
heavily from
from existing
existing IEEE
IEEE C37.90.x
C37.90.x standards
standards for
for Protective
Protective
Relaying
Relaying devices.
devices.
Fewer
Fewer tests
tests than
than IEC
IEC 61850-3
61850-3 but
but with
with test
test levels
levels just
just as
as high
high and
and higher
higher in
in
the
the case
case of
of Radiated
Radiated RFI:
RFI: 35V/m
35V/m v.s.
v.s. 10V/m
10V/m !!
Defines
Defines two
two classes
classes of
of devices:
devices:
Class
Class 11 devices
devices allow
allow communications
communications errors
errors or
or loss
loss during
during EMI
EMI type
type tests.
tests.
Class
Class 22 devices
devices allow zero
zero communications
communications errors
errors or
or loss
loss during
during EMI
EMI type
type tests.
tests.
32

RuggedCom
Industrial Strength Networks

EPRI Tests: EMI Susceptibility of STP Copper Cables

EPRI (AEP) 1997 EMI Immunity Testing


Transient
Generator
Copper
CAT5 Cable

IEC 1000-4-4
E.F.T.
Coupling
Coupling

Data Frames

33

RuggedCom

The EPRI Tests (2002): A closer look

Industrial Strength Networks

Transient
Generator
Copper CAT5 Cable

Coupling
Coupling Medium
Medium

IEC 61000-4-4
Electrical Fast
Transients

Data Frames
Media
Converter
Fiber Optical
Cable

Media
Converter

Network Analyzer
Simulator
(SmartBits)

Electrical Fast Transients (IEC 61000-4-4) Applied to CAT5 cable


Resultant Frame Loss:
32% @ 1kV
Unacceptable Performance For
66%@ 2kV
Real-Time Control!
75% @ -2kV

34

RuggedCom
Industrial Strength Networks

Real-time Performance - An Illustrative Example During


a Power System Fault...

Power System
Current Waveform

Fault

= 16.7ms

= 16.7ms

Fault period will be period of:


High levels of transient EMI phenomena!
Sub-cycle (i.e. 16.67ms) processing by IEDs and critical
communications performance by the LAN in ( < 4ms)!

Cant afford frame errors, delays or loss of


communications!
35

RuggedCom
Industrial Strength Networks

Industry Studiesconfirm what most of us already knew!

Rockwell Automation - Study CMR of CAT-5 Cable


IEC 61000-4-6 (Induced RFI)
AC Mains

EUT
Signal
Port

AC
Metallic
Plane
(grounded)

Fiber Optical
Connections

DC

DC Power Source
Schaffner NSG2070-1

CDN-AF2

Power

150KHz 80MHz

Ground
Strap

SmarBits 600
Communications Tester

EUT

Grounding
Connection

Non-Metallic / Non-Conductive Table

80 cm

Metallic Ground Plane

Induced RFI (IEC 61000-4-6) Applied to CAT5 cable to test CMR.


Common mode noise coupling will occur via adjacent cabling

Resultant Bit Error Rate:


22% @ 10Vrms (noise coupled)!
Error rates of this magnitude render the network useless!

36

RuggedCom
Industrial Strength Networks

Real-time, Deterministic Performance


Look Ma, No Collisions!

RuggedCom
Industrial Strength Networks

Key advances in Ethernet technology for real-time


control!

IEEE 802.3x Full Duplex Operation


IEEE 802.3p Priority Queuing
IEEE 802.3Q VLAN
IEEE 802.3w Rapid Spanning Tree
IGMP Snooping (Multicast Filtering)
Modern
Modern Ethernet
Ethernet technology
technology is
is well
well suited
suited for
for real-time
real-time control!
control!
The
The age
age old
old knock
knock against
against Ethernet
Ethernet of
of lack
lack of
of deterministic
deterministic performance
performance
38
is
is more
more aa red
red herring
herring than
than reality
reality when
when it
it comes
comes to
to modern
modern Ethernet.
Ethernet.

RuggedCom

Look ma no collisions

Industrial Strength Networks

Q. How do you make Ethernet deterministic or how do you


prevent collisions from occurring on an Ethernet network?
Ans. You use an Ethernet switching hub (i.e. Switch) with fullduplex ports IEEE 802.3x!

S W
Tx

Rx

Tx

I T C

Rx

Tx

Rx

Full-Duplex
Ports

IED
IED
#1
#1

IED
IED
#2
#2

IED
IED
#N
#N

Ethernet Switches with full-duplex ports dont have collisions!


39

RuggedCom

Minding your ps and Qs

Industrial Strength Networks

Standard Ethernet Frame IEEE 802.1


SYNCH
PREAMBLE

MAC
DESTINATION
ADRESSS

MAC
LENGTH
SOURCE
OR
ADRESSS
TYPE

DATA

FCS

Ethernet Frame with new VLAN IEEE 802.1Q & Priority IEEE 802.1p Tag.
SYNCH
PREAMBLE

MAC
DESTINATION
ADRESSS

MAC
SOURCE
ADRESSS

802.1p
PRIORITY

802.1Q
VLAN
ID

LENGTH
OR
TYPE

DATA

FCS

IEEE 802.1Q VLAN Tag

Allows
Allows priority
priority tagging
tagging of
of mission
mission critical
critical frames
frames 802.1p
802.1p
Allows
Allows isolation
isolation &
& grouping
grouping of
of IEDs
IEDs into
into virtual
virtual LANs
LANs 802.1Q
802.1Q VLAN
VLAN

40

RuggedCom

Minding your ps and Qs

Industrial Strength Networks

Q. How do I make sure during periods of heavy traffic that


critical messages make to it through without delay?
Ans. You tag them with a higher priority IEEE 802.1p.
High-Priority Queue

In-coming Frames
Priority
Queue
Mapping

Low-Priority Queue

Port Transmit Queue

Inside the Switch


(per port)

IEEE 802.1p allows prioritization for critical messages!

41

RuggedCom

An IEEE 802.1p Priority example

Industrial Strength Networks

High-Priority Queue

In-coming Frames
Priority
Queue
Mapping

Low-Priority Queue

Port Transmit Queue

Inside the Switch


(per port)

Real-time
Real-time control
control packets
packets can
can be
be assigned
assigned to
to higher
higher priority.
priority.
Ensures
Ensures real-time
real-time control
control messages
messages will
will get
get through
through even
even during
during network
network congestion.
congestion.
Priority
Priority can
can be
be tag
tag header
header based
based or
or MAC
MAC address
address or
or port
port based
based for
for legacy
legacy IEDs.
IEDs.
42

RuggedCom

Minding your ps and Qs

Industrial Strength Networks

Q. How do you segregate network traffic between devices used


for data collection and those utilized for real-time control?
Ans. You assign them to different VLANs IEEE 802.1Q.
Substation
Substation
Computer
Computer

S W
VLAN
VLAN
11

IED
IED
11

IED
IED
22

IED
IED
33

I T C H

IED
IED
44

Protective
Protective Relaying
Relaying IEDs
IEDs

IED
IED
55

IED
IED
66

IED
IED
77

IED
IED
88

VLAN
VLAN
22

Data
Data collection
collection IEDs
IEDs

IEEE 802.1Q allows segregation of devices into VLANs !43

RuggedCom

An IEEE 802.1Q VLAN example..

Industrial Strength Networks

HMI
Computer

S W

I T C H

VLAN
VLAN
11

VLAN
VLAN
22
IED
IED
11

IED
IED
22

IED
IED
33

PLC
PLC IEDs
IEDs

IED
IED
44

IED
IED
55

IED
IED
66

IED
IED
77

IED
IED
88

Data
Data collection
collection IEDs
IEDs

Traffic
Traffic from
from data
data IEDs
IEDs in
in VLAN
VLAN 22 isolated
isolated from
from IEDs
IEDs in
in VLAN
VLAN 11
HMI
HMI Computer
Computer can
can communicate
communicate to
to both
both
44
VLAN
VLAN can
can be
be tag
tag header
header based
based or
or MAC
MAC address
address or
or port
port based
based for
for legacy
legacy IEDs.
IEDs.

RuggedCom
Industrial Strength Networks

IGMP Management for Multicast Messaging..


Substation
Substation
Computer
Computer

S W

Multicast Traffic

IED
IED
11

IED
IED
22

IED
IED
33

Producer
Producer IEDs
IEDs

I T C H

IED
IED
44

IED
IED
55

IED
IED
66

IED
IED
77

IED
IED
88

Consumer
Consumer IEDs
IEDs

Multicast
Multicast Traffic
Traffic from
from Sensor
Sensor IEDs
IEDs can
can be
be assigned
assigned by
by the
the Switch
Switch to
to specific
specific
Consumer
Consumer IEDs.
IEDs.
For
For example,
example, Multicast
Multicast Traffic
Traffic from
from the
the Producer
Producer IEDs
IEDs 1,
1, 2,
2, 33 &
& 44 can
can be
be assigned
assigned
only
only to
to the
the consumer
consumer IEDs
IEDs which
which require
require itit (e.g.
(e.g. IEDs
IEDs 66 &
& 8).
8).
45

RuggedCom
Industrial Strength Networks

How Real-Time can an Ethernet network be?

Industrial Grade Ethernet Switch

100m

10/100Mbps
10/100Mbps
RJ45
RJ45 Ports
Ports

Sensor IEDs that multicast their


respective sensor value every 1ms
(producer IEDs)
PLC

PLC is the
consumer IED
46

RuggedCom
Industrial Strength Networks

Bandwidth Example Calculations

Given X number of producer IEDs and one consumer IED:


1.

Port Bandwidth of consumer device (On uplink port) = BWConsumer (Mbits/second)

2.

Frame size generated by producer IEDs = FS (in bits)

3.

IED Period of reporting by producer devices =TProducer (ms)

Given a 16 port full-duplex 100Mbps Ethernet Switch with:


BWConsumer = 100Mb/s
The maximum number of possible frame bits @ 100% utilization of this port is defined as:
BWMAX = 100,000,000 bits/second
(This number represents total traffic including CRC and preamble)
For this analysis assume the following:
The frame size generated by producer devices , FS=64 bytes in length.
The period of IED reporting, Treporting = 1 ms (i.e. 1000 times / second)
Therefore, it can be calculated that each producer device will consume:
64 bytes x 8 bits/byte x 1000/s = 512kbits/s of bandwidth.
Therefore one can conclude the following:
Each IED consumes (512,000)/(100,000,000) = 0.512 % of bandwidth each second
(100,000,000) / (512,000) = 195. This implies 195 producer devices can be used before the
maximum channel BW is reached. At this point channel flow control would kick in to relieve the
pressure.
47

RuggedCom
Industrial Strength Networks

Bandwidth Calculations for Different Reporting


Periods and Frame Sizes

Effect of multi-cast messaging-based data collection on Uplink Bandwidth


Frame
Consumer IED
Uplink
Percent
Max # Possible
Size
Reporting
Bandwidth
Utilization of
Producer IEDs
(Bytes)
Period
Available
Consumer
(See Note)
(ms)
(Mb/s)
Uplink
(%)
64
1
100
0.512
195
64
0.5
100
1.024
97
64
0.25
100
2.048
48
64
0.1
100
5.12
19
128
1
100
1.024
97
256
1
100
2.048
48
512
1
100
4.096
24
128
0.5
100
2.048
48
256
0.25
100
8.192
12
512
0.1
100
40.96
2

Bandwidth
Bandwidth scales
scales linearly
linearly allowing
allowing for
for aa wide
wide variety
variety of
of possible
possible
configurations.
configurations.
Support
Support of
of IGMP
IGMP (Internet
(Internet Group
Group Management
Management Protocol)
Protocol) allows
allows for
for
multicast
multicast message
message filtering
filtering and
and producer-subscriber
producer-subscriber groupings.
groupings. 48
Equivalent
Equivalent Layer
Layer 22 protocol
protocol is
is GMRP.
GMRP.

RuggedCom
Industrial Strength Networks

Network Architectures for


Connecting IEDs on the
Factory Floor
SWITCH

PLC
PLC

HMI
HMI

RTU
RTU

Meter
Meter

Relay
Relay

Sensors
Sensors

Remote I/O
I/O
Remote

IED
IED
IED
IEDIED
IED
IED
IEDIED
IEDIED
IED

IED
IED
(Intelligent Electronic Device)

RuggedCom

CASCADING ARCHITECTURE

Industrial Strength Networks

Cost-effective
Cost-effective bus
bus architecture
architecture messages
messages cascade
cascade from
from switch
switch to
to switch.
switch.
Maximum
Maximum number
number of
of hops
hops (N)
(N) is
is determined
determined by
by worst
worst case
case latency
latency
requirements.
requirements.
1

SWITCH

SWITCH

SWITCH

SWITCH

SWITCH

IED IED

IED

IED IED
IED

IED

IED IED
IED

IED

IED IED
IED

IED

IED IED
IED

IED

Typical
Typical Switch
Switch Latency
Latency =5us
=5us ++ Frame
Frame Time
Time (100Mbps
(100Mbps Port)
Port)
Therefore,
Therefore, for
for aa 64
64 byte
byte frame
frame (512
(512 bits):
bits): the
the frame
frame time
time == 5.12us
5.12us (100Mbps)
(100Mbps)
Total
Total Latency
Latency per
per switch
switch hop
hop == 5us
5us ++ 5.12us
5.12us == 10.12us
10.12us
For
For N
N == 10
10 the
the worst
worst case
case latency
latency would
would be:
be: 10
10 xx 10.12us
10.12us == 101.2us.
101.2us.
50

IED

RuggedCom

RING ARCHITECTURE

Industrial Strength Networks

N
SWITCH
PATH
2

PATH
1

IED IED

IED

IED

Fault
Fault

SWITCH

SWITCH

SWITCH

SWITCH

IED IED

IED

IED IED
IED

N-1
SWITCH

IED IED Ring Architecture!


IED IED
Fault
Tolerant
Fault
Tolerant
Ring
Architecture!
IED
IED
IED
IED

IED

IED

IED IED

IED

Rapid
Rapid (i.e.
(i.e. ms)
ms) Reconfiguration
Reconfiguration via
via IEEE
IEEE 802.1w
802.1w Rapid
Rapid Spanning
Spanning Tree!
Tree!
Typical
Typical reconfiguration
reconfiguration times
times << 50ms
50ms for
for ring
ring with
with 10
10 switches.
switches. 51

IED

RuggedCom

STAR ARCHITECTURE

Industrial Strength Networks

N
SWITCH

SWITCH

SWITCH

SWITCH

SWITCH

IED IED

IED

IED IED
IED

IED

IED IED
IED

IED

IED IED
IED

IED

N-1
SWITCH

IED IED
IED

IED

Low-Latency
Low-Latency Architecture
Architecture Any
Any IED
IED to
to IED
IED communications
communications requires
requires
only
only two
two hops.
hops.

52

IED

RuggedCom
Industrial Strength Networks

Fault Tolerant Hybrid (Star/Ring) ARCHITECTURE


Fault
Fault

Fault
Fault

N-1

SWITCH

SWITCH

Fault
Fault

SWITCH

SWITCH

SWITCH

SWITCH

IED IED

IED

IED IED
IED

IED

IED IED
IED

IED

IED IED
IED

IED

N-2
SWITCH

IED IED
IED

IED

Low-Latency,
Low-Latency, Fault
Fault Tolerant
Tolerant Architecture.
Architecture.
Able
Able to
to tolerate
tolerate failure
failure of
of uplink
uplink or
or backbone
backbone switches.
switches.

53

IED

RuggedCom
Industrial Strength Networks

High Redundancy Architecture via IEDs with Dual Ethernet Ports


Fault
Fault

Fault
Fault
SWITCH

SWITCH

SWITCH

SWITCH

IED
IED
IED IED

IED
IED
IED
IED IED

IED
IED

IED
IED
IED
IED IED

IED

IED
IED

SWITCH
SWITCH

SWITCH

Fault
Fault

SWITCH
SWITCH

SWITCH
SWITCH

IED
IED

Fault
Fault

SWITCH
SWITCH

SWITCH
SWITCH

Fault
Fault

Able
Able to
to tolerate
tolerate multiple
multiple fault
fault types!
types!
IED
IED
IED IED

IED
IED

IED

SWITCH
SWITCH

SWITCH
SWITCH

Rapid
Rapid (i.e.
(i.e. ms)
ms) Reconfiguration
Reconfiguration via
via IEEE
IEEE 802.1w
802.1w Rapid
Rapid Spanning
Spanning Tree!
Tree!
54

RuggedCom
Industrial Strength Networks

A comparison of Industrial Ethernet protocols Different strokes for different folks!

RuggedCom
Industrial Strength Networks

7 Layer OSI Model


7 Layer OSI Model
Application
Presentation
Session

Software

Transport
Network
Data Link

Hardware

Physical

Used to describe any network or protocol.


Helps us compare apples to apples.
56

RuggedCom

Different strokes for different folks

Industrial Strength Networks

The Process Automation Sector

The Substation Automation Sector


IEC61850(UCA2)

ProfiNet

Modbus/TCP

DNP3/TCP

Application

Fieldbus HSE

Modbus/TCP
EtherNet/IP

Presentation
Session
Transport

TCP/UDP

IP

Network

IP

IEEE 802.1

Data Link

IEEE 802.1

Physical

IEEE 802.3

TCP/UDP

TCP/IP

Ethernet

IEEE 802.3

TCP/IP

Ethernet

Bottom half is pretty much consistent.

Bottom half is pretty much consistent.

Application Layer is dominated by a


few players. Trend is towards 61850.

Application Layer is still being hotly


contested by major OEMs.

57

RuggedCom
Industrial Strength Networks

Rockwells EtherNet/IP
Released by ControlNet International and ODVA. The IP
stands for Industrial Protocol
Uses the CIP (Control & Information Protocol) Application
Layer from ControlNet and DeviceNet.
PROS:
Supports Object Oriented communications model
Supports publisher-subscriber model
CONS:
Difficulty in supporting legacy A-B protocols.

Siemens (PTO) ProfiNet


Released by Profibus Trade Organization
Based on Microsofts DCOM protocol
PROS:
DCOM supports Object Oriented communications
Major vendor backing (Siemens)
CONS:
Does not support publisher-subscriber model
Poor real-time control?

Fieldbus Protocols Over Ethernet


Schneiders Modbus/TCP
Released in March 1999 Open Modbus/TCP Specification
Uses the Modbus Application Layer (MBAP) bundled with
TCP/IP
PROS:
TCP and Modbus are widely supported
Easy to implement
CONS:
Does not support object oriented communications
model (i.e. you still need to know register addresses
of every device!)
Does not support publisher-subscriber model

Foundation Fieldbus HSE


HSE = High Speed Ethernet - using 100Mbps Ethernet
Uses the Application Layer from their H1 standard with
TCP/IP (UDP)
PROS:
Supports Object Oriented Communications Model
Supports Publisher Subscriber model
Supports Network Management
Major Vendor Support
CONS:
Limited support outside process control sector

Source: ISA Training Institute Is Ethernet Ready for the Plant Floor online seminar.

No convergence (i.e. One protocol) on the horizon for industrial/process


58
automation sectorsyet!

RuggedCom

Cant we all just get along

Industrial Strength Networks

Q. How do you get multiple devices which communicate using


different protocols to coexist on a common network?
Ans. You use Ethernet!

S W

RTU
RTU
(DNP3)
(DNP3)

I T C H

Relay
Relay
(UCA2/IEC61850)
(UCA2/IEC61850)

Meter
Meter
(Modbus)
(Modbus)

PLC
PLC
(Fieldbus)
(Fieldbus)

Ethernet allows multiple protocols to coexist on the same network!


59

RuggedCom
Industrial Strength Networks

Key Takeaways

Industrial Ethernet requires network devices (e.g. Switches) to operate


reliably in industrial environments.
IEC 61850-3, IEEE P1613 specifically define EMI and Environmental
requirements for substation environments.
IEC 61000-6-2 should be minimum requirement for EMI immunity in industrial
environments.
Other requirements, depending on industry sector may be Class 1 Division 2
(Petrochem), IP67/NEMA 4x (Mining).

For real-time control applications Industrial Ethernet devices and


network equipment should support:
IEEE 802.3x Full-Duplex operation for collision free operation
IEEE 802.1p Prioritization to allow real-time critical messages to get through.
IEEE 802.1Q VLAN to allow isolation of critical IEDs from non-critical IEDs .
IEEE 802.1w Rapid Spanning Tree to allow fault-tolerant ring architectures
with rapid (i.e. <50ms) reconfiguration times.
IGMP Snooping / Multicast filtering to prevent multicast intensive protocols
(e.g. EtherNet/IP) from inundating non participating IEDs on the LAN.
60

RuggedCom
Industrial Strength Networks

Thank you.