ISA Seminars on the Web

Live Experts on Hot Topics

Standards
Certification
Education and Training
Publishing
Conferences and Exhibits

Designing Industrial
Ethernet Networks
FG21CW3 Version 1.5
© 2004

Standards
Certification
Education and Training
Publishing
Conferences and Exhibits

© 2004, ISA
FG21W3 (1.5)
 Seminar Logistics

• Seminar materials
– Downloadable presentation
– Question and Answer session (audio and email)
– Survey
– Earn 1 Professional Development Hour (PDH)

• Seminar length
– 60 minute presentation
– Three 10-minute question and answer sessions

Audio Instructions

• As a participant, you are in a “listen-only” mode.

• You may ask questions via the internet, using your keyboard, at
any time during the presentation. However, the presenter may
decide to wait to answer your question until the next Q&A Session.
• If you have audio difficulties, press *0.

© 2004, ISA
FG21W3 (1.5)
 Audio Instructions for Q&A Sessions

• Questions may be asked via your telephone line.

• Press the *1 key on your telephone key-pad.
• If there are no other callers on the line, the operator will
announce your name and affiliation to the audience and
then ask for your question.
• If other participants are asking questions, you will be placed
into a queue until you are first in line.
• While in the queue, you will be in a listen-only mode until
the operator indicates that your phone has been activated.
The operator will announce your name and affiliation and
then ask for your question.

Introduction of Presenter, Victor Weglin

• Victor Wegelin is the owner of

PMA Concepts, a consulting firm
in Westminster, CA with over 25
years experience in the design,
build and support of Industrial
Control Networks. In addition to
instructing at ISA, he is a faculty
member of the Industrial Control
Certificate program at California
State University. He holds a BS
in Chemical Engineering from the
University of Cincinnati, and an
MBA from the University of
Chicago. He is a senior member
and Fellow of ISA and a licensed
Professional Engineer. He
started teaching for ISA in 1990.

© 2004, ISA
FG21W3 (1.5)
 Introduction of Presenter, Eric Byres

• Professional engineer and research

faculty leader at the British Columbia
Institute of Technology.
• Specializes in the design of large-
scale
data communications systems in
industry, including fieldbuses, data
highways and Ethernet control LANs.
• Industrial Ethernet Conference Chair
for 2000/2001.
• Awarded IEEE Best Industrial
Applications Paper of 2000 for
“Designing Secure Networks for
Process Control”.
• Awarded Advanced Systems Institute
Fellowship for research into industrial
network security.

Key Benefits of Seminar

• Learn the ten key steps to successful industrial network

design.
• Understand how modern structured cabling systems can be
used in the typical industrial plant.
• Learn the principles of industrial network security.

© 2004, ISA
FG21W3 (1.5)
 Section 1: Designing the Physical Infrastructure for
Industrial Networks

• Measure the Existing Network

• Determine Long Term Strategy
• Distribution Design
• Cable Standards
• Cabinet Design Standards

Designing Physical Network Infrastructure

• The core of any network is the physical infrastructure (i.e.

cabling, connectors, distribution, cabinets, etc.):
– Responsible for the bulk of the cost (approx 60%)
– Responsible for most maintenance issues (50 - 80%)
– Longest life before replacement (10-15 years)
– Most difficult and costly to modify
• Since this is most important part of any industrial network it
needs to be:
– Standards based
– Flexible
– Manageable

© 2004, ISA
FG21W3 (1.5)
 Five Steps to the Design of Cable Systems

• Five steps when designing the physical infrastructure for an

industrial network:
1. Understand the Existing Network.
2. Determine Long Term Strategy.
3. Develop Distribution Design.
4. Develop Cable and Connector Standards.
5. Develop Cabinet Design Standards.

Understand the Existing System

• Start off by measuring and documenting the existing

communications systems:
– Cabling/network diagrams.
– Traffic analyzers.
– Downtime records/ service logs.
• Learn the current situation – especially the issues and trouble
spots (don’t repeat them).
• Doesn’t have to be a major study.

© 2004, ISA
FG21W3 (1.5)
 Determine Long Term Goals

• Learn the long term goals of the site network and the general
technologies and standards to be used:
– Future expansion plans.
– Future decommissioning plans.
– Technology and integration demands.
• Case History:
– Engineers install buried fiber run across plant site, ending in dispatch office.
– Management moved dispatch and demolished building.
– Fiber trunk too short to move to new building and too expensive to splice
extension on. Abandoned $35K.

Communications Cabling in the Industry

• No standards for communications cabling in industrial facilities

(but EIA/TIA TR42.9 is in progress).
• Most industrial communications are still designed and installed
on a as-needed basis.
• Good distribution design is needed.

PLC PLC
DCS

© 2004, ISA
FG21W3 (1.5)
 TIA-568B Commercial Standards

• TIA 568B is a design and cabling standard for

telecommunications in commercial buildings.
• Premise: The type of equipment should not influence the type
of cable being installed.
• Intended to allow engineers design a single communications
infrastructure, independent of the technology being run over it:
— Ethernet — EIA-422/485
— Voice — Video
— Alarms — Future Networks

EIA/TIA 568B Standard

• Very influential in the development of new LAN technology.

• Some of its most important points:
– Star topology is strongly recommended.
– Three Level Hierarchy (no more than two sub-levels).
– Only UTP, STP and fiber is recommended.
– Multimode fiber is either 62.5/125 or 50/125.
– UTP runs can be a maximum of 100 meters.
– Fiber runs can be a maximum of 2000 meters.

© 2004, ISA
FG21W3 (1.5)
 Communications Cabling Hierarchy

• Tree/Star-based. Main Cross-

connect (MC)
• Main communications
centers feed maximum 2
sub-layers of closets.
• Computer/controllers
Intermediate Intermediate Intermediate
connected at any layer. Cross-connect Cross-connect Cross-connect
(IC) (IC) (IC)
• IAONA Ethernet Planning
and Installation Guide
shows how to follow this
strategy in industrial Networked
Horizontal Horizontal Devices
settings. Cross-connect Cross-connect
(HC) (HC)

Networked
Devices

Main Closet and Cabling Distribution

• Plan the main communications centers and backbone cable

runs. Things to consider are:
– Locations of existing telecommunications closets
– Availability of cabling pathways
– Server farm locations
– Distribution to servers
– Distribution to workstations
• Pick your main centers and work from there. You should be
able to get from a center to any closet in 2 hops and to any
device in 3.

© 2004, ISA
FG21W3 (1.5)
 Choosing the Right Cable

• Network designers longer consider coaxial based Ethernet an option.

Manufacturers treat coaxial-based Ethernet hardware as legacy
equipment.
• Fiber optic and unshielded twisted pair (UTP) have both taken over:
– coax is not cost competitive with fiber and UTP for the data capacity provided.
– fiber and UTP can transmit Ethernet at gigabit rates while coax is stuck at 10
Mbps.
– coax is not recognized in the EIA/TIA-568 standard (mentioned only an
informational appendix).

Fiber Optic Cabling Standards

• Select common cabling media for as much business and

process communications as possible...
• Fiber optic cable for all cabling outside of offices and
control/electrical rooms:
– 62.5/125m or 50/125m multimode fibre
– Aluminium armour protection
– Rated for both indoor and outdoor use
– FT-4 flame rated
– 12 fibres minimum cable bundle
– 24 fibres cable bundle recommended for most runs.

© 2004, ISA
FG21W3 (1.5)
 Copper Cabling Standards

• Fiber can be expensive for connection to the end device

(Though not impossible).
• So… copper cabling can be specified for use inside offices and
electrical/control rooms:
– Category-5e/i Unshielded Twisted Pair (UTP)
– Screened Twisted Pair (ScTP).
– Shielded Twisted Pair (STP).
• Connectors/cables must be selected as a pair.
• IAONA and EtherNet/IP have excellent guides for detailed
specifications.

Cable and Cabinet Management

• It is essential to organize the cable system in the plant. This is

the wrong way…

Typical cable
management

Mixed UTP,
Fibre and
Power

No Power
Management

Difficult to
troubleshoot

© 2004, ISA
FG21W3 (1.5)
 Good Cable and Cabinet Management

• This is the same equipment room one year later!

Good cable management

Fiber patch panels
UTP, Fiber and Power separated with
space for expansion
Power management
Easy to troubleshoot

Standardized Communications Cabinets

• Create standard designs to

ease engineering and
maintenance. Fiber Patch
Panel
• Each room contains one
communications marshalling
cabinet.
• All communications pass Ethernet
through cabinet. Switch

• The conversion point where Fiber/UTP

fibre is converted to copper- Converters
based wiring.
UTP Patch
Panel

© 2004, ISA
FG21W3 (1.5)
 Standardized Communications Cabinets (cont’d)

• All fibre cabling brought into

the rear of the cabinet.
• Fiber terminated on swing-out
patch panels.
• Connections from field cables Fiber Patch-
Panel
to electronics made with
patch cords.
• Design with at least 30%
spare capacity. Fiber Patch-
Cords
Field
Cables

Live Question and Answer Session

• During Q&A, questions may be asked via your telephone line.

• Press the *1 key on your telephone key-pad.
• If there are no other callers on the line, the operator will
announce your name and affiliation to the audience and then ask
for your question.
• If other participants are asking questions, you will be placed into
a queue until you are first in line.
• While in the queue, you will be in a listen-only mode until the
operator indicates that your phone has been activated. The
operator will announce your name and affiliation and then ask for
your question.

© 2004, ISA
FG21W3 (1.5)
 Section 2: Designing the Logical
Infrastructure for Industrial Networks

• Things That Can Go Wrong . . .

• Switch Based Architectures
• Network Redundancy
• Network Security
• Documentation
• Network Management

Designing The Logical Network

• Selection of the logical network components (i.e. the routers,

switches, etc) are the next step major stage in design.
• In this stage we need to consider:
– Performance
– Reliability and Redundancy
– Security
– Long Term Management.
• We will start by looking at what we want to avoid…

© 2004, ISA
FG21W3 (1.5)
 Noise or Bad Packets

• Propagation of noise or bad packets throughout an entire

network is a serious risk.
• Pulp mill case history-
– Cable damage problem in one area creates bad packets from
reflections.
– “Dumb” network equipment spreads problem to other areas.

Repeater

Accounting Network Engineering Network

Cut

IP Address Duplication

• The TCP/IP protocol demands that every device has a

unique IP address.
• Paper Machine Profile Controller Case History:
– Controller and Scanners use TCP/IP to communicate.
– Printer in another mill area gets same address as controller.
– Scanners try to talk to printer rather than to controller.

Scanner sends Bridge

data to printer
Accounting Network Process Network

Scanner
Controller

© 2004, ISA
FG21W3 (1.5)
 Loading and Broadcast Storms

• Too much uncontrolled traffic will impact network access and

device performance.
• For example, broadcasts are messages addressed to all
network nodes. A few broadcasts are okay. Many are a
broadcast storm and will use up a device’s CPU time.
• Case History- Steam Plant DCS:
– DCS uses Ethernet to communicate between display server and
operator consoles.
– Broadcasts from mis-configured Windows 95 machine in another mill
area overloads screen server. Shuts down all DCS operator consoles.

Deliberate Intrusion

• Viruses and hacking on the plant floors is no longer a rare event:

– Viruses can infiltrate through firewalls, laptops, modems and wireless to infect
common operation systems and applications.
– Hackers look for CPUs with well known operating systems (such as UNIX and
Windows) that are poorly secured or not patched. DCS systems are a good
target.
• Passwords are not good enough security on their own.

© 2004, ISA
FG21W3 (1.5)
 Case History - Hacking a DCS

• Eastern plant does major upgrade of DCS.

• Several months later, head-office engineer connects to the mill
DCS from head office, using the company's wide area network
(WAN).

Router
Head Office
Router
East Coast Mill
A B C DCS Network Business Network

Head Office
X Y Z
Engineer
A
DCS

PLC PLC

Case History - Hacking a DCS (cont’d)

• Engineer loaded a small program onto a DCS graphics station

to send data back to head office.
• This new task overloaded DCS/PLC gateways
• Operators would lose control of the motors controlled by the
PLCs

Router
Head Office
Router
East Coast Mill
A B C DCS Network Business Network

Head Office
X Y Z
Engineer
A
DCS
PLC Gateway PLC PLC

© 2004, ISA
FG21W3 (1.5)
 Network Architecture Design

• How do we avoid these issues?

• We need a hierarchy of network devices that:
– control traffic and errors at the lowest device level.
– control addressing issues and broadcasts at the area level.
– control security and virus issues at the manufacturing to business
system interface.

Plant Floor Level - Layer 2 Switches

• First Layer-2 (bridging) switches are installed in place of hubs

on the plant floor. These only cost slightly more but offer many
benefits:
– No collisions - each device in private collision domain
– Automatic conversion between 10 and 100 Mbps Ethernet
– No “repeater rule” issues
– Prevents propagation of bad packets.

L2-Switch L2-Switch

PLC Network DCS Network

X
A B C
PLC

© 2004, ISA
FG21W3 (1.5)
 Industrial Level – Layer-3 Switches

• Next Layer-3 (routing) switches are installed at the top of the

plant floor network, connecting the different control areas.
• These offer containment of broadcast storms, address control
and basic packet security and in a very high through-put
device.

L3-Switch
L2-Switch L2-Switch

DCS Network PLC Network

X
A B C
PLC

Business to Plant Floor Interface Level

• Between plant floor and the rest of the company networks a

firewall is recommended.
• Provide more sophisticated security than the L3-switch to
prevent hackers/viruses entering.

DMZ/PIN Enterprise
Network

Data
Historian

L2-Switch L3-Switch
L2-Switch

PLC VLAN DCS VLAN

Server
PLC PLC
DCS

© 2004, ISA
FG21W3 (1.5)
 Security with Routing-Switches (VLANs)

• Routing-switches have the ability to create virtual LANs

(VLANs) where the switch allows defined devices on different
ports to act as if they are on the same LAN segment.
• VLANs group arbitrary collections of end nodes on multiple
LAN segments into separate domains.
– Packets between VLAN nodes are switched.
– Packets between VLANs are routed, allowing security filtering.
• Very useful for dividing up the plant floor.

Security with VLANs

• Separate VLANs have been set up for both the business users
and the process control users.
• Both VLANs contain the process information server, allowing
both groups to access the server, yet forming a secure
separation between different process networks.

L3-Switch
L2-Switch L2-Switch

DCS VLAN Engineering VLAN

Server
A B C
PLC PLC

© 2004, ISA
FG21W3 (1.5)
 Implementing Security Policy

• Once the Layer-3 switch and firewalls are installed it is

necessary to implement a security rules known as access
control lists.
• ACLs dictate who can access various network areas and what
actions are permitted.
• Options for an typical IP network:
– Filtering on specific IP addresses.
– Filtering on IP address ranges. This utilizes IP subnet masks to identify a
specific address range.
– Filtering TCP/UDP port numbers. For example, Telnet traffic always uses
the TCP port of 23.
– Filtering on connection status or direction.

Network Redundancy

• IEEE 802.1d Spanning Tree Algorithm (STA) allows backup links over
redundant fiber links.
• If a loop occurs in a network, bridges may forward traffic indefinitely,
which can result in network failure.
• STA prevents loops by allowing bridges to detect parallel paths and
blocking one of these paths.

Switch

Hub or Switch
Hub or Switch

Process Network Accounting Network

X
A B C
PLC

© 2004, ISA
FG21W3 (1.5)
 Network Redundancy (cont’d)

• Propriety schemes also allow backup links between switches

over redundant fiber links.
• Generally based on monitoring the state of a single pair of
links rather than a network wide view.
• Will have much faster switchover times.

Switch

Hub or Switch
Hub or Switch

Process Network Accounting Network

X
A B C
PLC

Documentation

• Provide staff with easily accessible network documentation:

– Cable plans with labelling strategy.
– Logical network diagrams.
– Equipment lists and configurations.
• Example: Web-based documentation package allows an
electrician to:
– View the overall network diagrams
– Read troubleshooting guides
– Drill down to actual photographs of equipment
– View configuration of equipment dip switches

© 2004, ISA
FG21W3 (1.5)
 Documentation

Network Management Software

• Software program that

automatically finds devices on
network and monitors them.
• Based on Simple Network
Management Protocol
(SNMP).
• Can warn of failed or high
error ports, bad lines, etc.
• Can also be a big security risk
if not careful!

© 2004, ISA
FG21W3 (1.5)
 Live Question and Answer Session

• During Q&A, questions may be asked via your telephone line.

• Press the *1 key on your telephone key-pad.
• If there are no other callers on the line, the operator will
announce your name and affiliation to the audience and then ask
for your question.
• If other participants are asking questions, you will be placed into
a queue until you are first in line.
• While in the queue, you will be in a listen-only mode until the
operator indicates that your phone has been activated. The
operator will announce your name and affiliation and then ask for
your question.

Five Steps to a Physical Infrastructure

for Industrial Networks

1. Measure Existing Network: Study existing networks and

develop measures to analyze project success.
2. Long Term Strategy: Determine long term goals of site
network and technology standards to be used.
3. Distribution Design: Locate main communication centers,
then determine cable runs to distribution centers.
4. Cable Standards: Select fiber and copper cable types and
create site purchasing/installation standards.
5. Cabinet Design Standards: Create consistent cabinets for
easier design and maintenance.

© 2004, ISA
FG21W3 (1.5)
 Five Steps to a Logical Infrastructure
for Industrial Networks

1. Logical Network Design: Determine the switches and

routers required and the interconnection.
2. Security Policy: Set policy on who needs to connect to
what systems and for what purposes… then implement it
in devices.
3. Redundancy Analysis: Analyze failure modes to
determine what really needs to be redundant and what is
too complex.
4. Documentation Plan: Create site standards for cable,
cabinet and equipment labeling and documentation.
5. Long Term Management Plan: Determine how the
network is going to be monitored and maintained.

How Many People Are at Your Site?

• Poll Slide
• Click on the appropriate number indicating the number of
people that are at your site.

© 2004, ISA
FG21W3 (1.5)
 Live Question and Answer Session

• During Q&A, questions may be asked via your telephone line.

• Press the *1 key on your telephone key-pad.
• If there are no other callers on the line, the operator will
announce your name and affiliation to the audience and then ask
for your question.
• If other participants are asking questions, you will be placed into
a queue until you are first in line.
• While in the queue, you will be in a listen-only mode until the
operator indicates that your phone has been activated. The
operator will announce your name and affiliation and then ask for
your question.

More Industrial Ethernet Resources:

• Good Books on Ethernet:

– Spurgeon, Charles E., Ethernet: The Definitive Guide,
O'Reilly and Associates, 2000
• Free Guides on Industrial Ethernet:
– IAONA Ethernet Planning and Installation Guide 4.0
www.iaona.org/home/downloads.php
– EtherNet/IP Developer Recommendations 1.00
www.ethernetip.de/files/EIP_Developer_WP.pdf

© 2004, ISA
FG21W3 (1.5)
 Related Courses from ISA

• One-day seminar - Ethernet and TCP/IP on the Plant Floor

(FG21C)

• Industrial Ethernet Web Seminar Series #1 - Understanding

Ethernet Cabling and Hardware

• Industrial Ethernet Web Seminar Series #3 - Designing

Industrial Ethernet Networks

• Industrial Data Communications Part I (TS05)

• Industrial Data Communications Part II (TS10)

• All ISA courses are available any time as on-site training

• For more information: www.isa.org/training or
(919) 549-8411

Other Related Resources from ISA

• Industrial Ethernet, 2nd Edition by Perry S. Marshall and John

S. Rinaldi from ISA Press

• Automation Network Selection by Dick Caro from ISA Press

• ISA Membership is just $100 per year, which includes free

membership in two Technical Divisions (a $20 value) - one
from each Department: Automation and Technology and
Industries and Sciences.

– For more information: http://www.isa.org/membership/meminfo or

(919) 549-8411

© 2004, ISA
FG21W3 (1.5)
 ISA Certifications

• Certified Automation Professionals ® (CAP ®)

– www.isa.org/CAP
• Certified Control Systems Technician® (CCST®)
– www.isa.org/CCST

• Please visit us online for more information on any of these

programs, or call (919) 549-8411.

Please take our Web Seminar Survey

via Zoomerang

The seminar survey was sent to you via email during

the seminar. Please do not forget to complete the
Zoomerang survey.

© 2004, ISA
FG21W3 (1.5)