(Redundant Communication System for Enhancing Security of Power Control System)

-. -. .
: 02-4362541 : 02-4362594
E-mail: thanyapatt.s@egat.co.th

(Redundant communication system)
(Availability)

Virtual Router Redundancy Protocol (VRRP) Ethernet-E1 Converter
Ethernet over SDH (EoSDH) (Virtual Gateway) (Remote Terminal
Unit: IP RTU ) Internet Protocol (IP) IP RTU IP RTU MultiProtocol Label Switching Virtual Private Network (MPLS VPN) . IP RTU
IP RTU
Operation Application
Phase Monitoring Unit (PMU) Meter

1. (INTERNET PROTOCOL - REMOTE TERMINAL UNIT, IP-RTU)

1.1 IP RTU (General Description of IP-RTU)
SCADA (Supervisory Control and Data Acquisition)


SCADA
(Master Station) , RTUs (Remote Terminal Units) .
6
(Internet Protocol - Remote Terminal Unit: IP-RTU) SCADA
IP RTU SCADA
(Analog) (Digital) Input Unit RTU
RTU
Output Unit RTU
Input Unit
-1-


(Communication Port) CPU Central Processing
Unit (CPU) RTU IP RTU . (Alarm)
Volt Amp .

SCADA


1.2 IP RTU
IP RTU .
IP/MPLS (Internet Protocol / Multiprotocol Label Switching) MPLS VPN
. IP/MPLS MPLS
VPN
IP RTU, AMR, DFR/FRS/PMU, CCTV, VoIP, Video Conference, Intranet Internet

IP RTU IP/MPLS .
IEC60870-5-104 2 Controlling station Controlled station 1

1 Controlled Station Controlling Station

1.2.1

Controlled Station (Server) IP RTU Information Data

Controlling Station . 2
-2-

1.2.2

Spontaneous Transmission IP RTU Controlling Station

Station Interrogation (Group) Controlling Station
Controlling Station (Client) . NCC RCC GCC CCS
(Command transmission) Controlled Station
. 2
Select and Execute Command On/Off Close/Open Circuit Breaker
Select and Execute Set Point Command Setting

LAN Port Router LAN Switch MPLS QoS

3 WAN Bandwidth 128 kbps 1 IP RTU
IP RTU IP RTU
Connection IP RTU IP RTU Station Interrogation
Control Center IP RTU Station Interrogation
Digital Input Analog Input Control Center Control Center
Update Bandwidth IP RTU 13.45 kbps 1
Bandwidth IP RTU 128 kbps Required Bandwidth
13.45 kbps
1 Required Bandwidth IP RTU
TYPE IDENT
1
2
3
4
5

STARTDT_ACTIVE
STARTDT_CONFIRM
INTERROGATION_ACTIVE
INTERROGATION_CONFIRM
1024 Digital Input
128 Analog Input
6 INTERROGATION_TERMINATE

Direction
Master to Slave
Slave to Master
Master to Slave
Slave to Master
Slave to Master
Slave to Master

ASDU (Byte)
(Data x Qty.)
6 x 1 =
6
6 x 1 =
6
14 x 1 =
14
14 x 1 =
14
2
140 x 8 = 1,120
2473 x 2 = 494
14 x 1 =
14

ASDU
+ Header (Byte)1
60
60
70
70
1,514 + 167 = 1,681

Required Bandwidth
, Assume 1sec (kbps)
0.48
0.48
0.56
0.56
13.45

70

0.56

1. ASDU + Header (Byte) Wireshark Capture Package

2. ASDU 140 Byte D/I 127 points
3. ASDU 247 Byte A/I 64 points
- IP RTU Shared bandwidth Link
capacity IP RTU Bandwidth 128 kbps

-3-

2. Operation applications
IP RTU (LAN) default gateway
IP RTU default gateway IP Address default gateway
static default gateway
IP RTU default gateway
2

2 default gateway

3 Virtual IP Address

VRRP static default gateway single point of failure

VRRP Router Virtual Router
Master router (routing and switching)
Backup router
Router Router
Virtual Router Master Virtual Router Backup 3
VRRP IP RTU default gateway Master router
Master router - default gateway Master Backup
router default gateway Master default gateway
Backup router Dynamic failover IP RTU
- Master router IP RTU
Gateway
VRRP VRRPv1 RFC 2338 (for IPv4 1998),
VRRPv2 RFC 3768 (for IPv4 2005) VRRPv3 RFC 5798 (for IPv4 and
IPv6)
-4-

 Operation applications
(Redundant Path) (Main Path) Operation application
NCC, RCC, BNCC
Operation application IP RTU IP
RTU Ethernet-to-E1 converter Ethernet-over-SDH card SDH E1
Provisioning 4

4 VRRP LAN Switch

3. (Availability)
Availability Uptime
/// 24
7 Uptime 165 (168 )
Availability 98.21 Availability
Availability = MTBF/(MTBF+MTBR)
MTBF (Mean Time Between Failure) Reliability uptime
uptime MTBF MTTR
Availability
MTTR (Mean Time To Repair) recovery
Failover Vendors Spare
4 6
-5-

 MTTR Single point of failure

downtime MTTR Availability 100%
Availability Availability
Redundancy Fault Tolerance Redundancy
(Redundancy communication)
Availability
(downtime)
Availability
Serial Availability
= A1*A2*An
Parallel Availability
= 1 {(1-A1)*(1-A2)*(1-An)}
Availability Availability
Availability

5 Availability topology Serial

Availability
= ASWA*ARA*ARB*ASWB
topology Serial
= 0.999*0.9995*0.9995*0.9999 = 0.9988 (99.98%)
Downtime
(1-Availability)*525,960 = (1-0.9988)* 525,960 = 631
Availability 0.9988 99.88% Downtime 1
24 ( 60x24x365.25=525,960) 631

6 Availability Parallel Serial

-6-

 2 parallel Availability
99.88% 99.9799% downtime 631 105

Availability
= ASWA1* ASWB1*{1-{(1-ARA1*ARB1)*(1- ARA2*ARB2)}}
topology Parallel
= 0.9999*0.9999*{ 1-{(1-(0.9995*0.9995))* (1-(0.9995*0.9995))} }
= 0.9999*0.9999*0.999999
= 0.999799 (99.9799%)
Downtime
= (1-Availability)*525,960 = (1-0.999799)* 525,960 = 105
4. (Availability) IP/MPLS
4.1 1 WAN Redundancy

7 WAN Redundancy
RTU (RTU Main) (RTU Backup) WAN
LAN Switch A1 IP/MPLS Router A1 (NCC, RCC)
7 ( WAN )
: Switch_A1 --> Router_A1 --> Router_B1 --> Router_B1 :
Availability
1 Availability
Availability of RTU System = { 1-{(1-ARB1*ASWB1)*(1- ARB2*ASWB2)} } * ASWA1*ARA1
= { 1-{(1-(0.9995*0.9999))* (1-(0.9995*0.9999))} } * 0.9999*0.9995
= 0.999399 (99.9399%)
Downtime of RTU System
= (1-Availability)*525,960 = (1-0.999399)* 525,960 = 316

-7-

4.2 2 WAN Redundancy Path Redundancy

8 WAN Redundancy Path Redundancy

RTU Main WAN LAN Switch A1 IP/MPLS
Router A1 (NCC, RCC) RTU Backup LAN Switch C
Protocol Ethernet
TDM (Time Division Multiplex)
SDH Protocol
Ethernet-to-E1 Converter Ethernet-over-SDH card 8
( WAN )
: Switch_A1 --> Router_A1 --> Router_B1 --> Router_B1
: Switch_C --> Router_C --> Router_B1 --> Router_B1
Availability
Availability of RTU Main
= 0.999399 (99.9399%)
Downtime of RTU Main
= 316
Availability of RTU System = 1-{(1-ASWA1*ARA1*ARB1*ASWB1)*(1-ARC*ASWC*ARB2*ASWB2)}
= 1-{(1-0.9988)*(1-0.9988)} = 0.9999985 (99.99985%)
Downtime of RTU System
= (1-Availability)*525,960 = (1-0.9999985)* 525,960 = 0.08394

-8-

4.3 3 WAN Redundancy Path Redundancy with VRRP (1)

9 WAN Redundancy Path Redundancy

RTU Main WAN LAN Switch A1 IP/MPLS
Router A1 (NCC, RCC) RTU Backup LAN Switch A1
LAN Switch C Protocol
Ethernet TDM 2 Topology
Local Router A1 Remote Router C Feature VRRP
RTU Main Router A1
Router C 9
( WAN )
: Switch_A1 --> Router_A1 --> Router_B1 --> Router_B1
: Switch_C --> Router_C --> Router_B1 --> Router_B1
Availability
Availability of RTU System = {1-{(1-ASWA1*ARA1*ARB1*ASWB1)*(1-ARC*ARB2*ASWB2)}}*ASWC
= {1-(0.0011*0.0010)}*0.9999 = 0.999899 (99.9899%)
Downtime of RTU System
= (1-Availability)*525,960 = (1-0.999899)* 525,960 = 53

-9-

4.4 4 WAN Redundancy Path Redundancy with VRRP (2)

10 WAN Redundancy Path Redundancy with VRRP (2)

4 LAN Switch A2
LAN Port RTU 10
( WAN )
: Switch_A1 --> Router_A1 --> Router_B1 --> Router_B1
: Switch_A2 --> Switch_C --> Router_C --> Router_B1 --> Router_B1
Availability
Availability of RTU Main
= 1-{(1-ASWA1*ARA1*ARB1*ASWB1)*(1-ASWA1*ASWA2*ASWC*ARC*ARB2*ASWB2)}
= 1-{(1-0.9988)*(1-0.9986)} = 0.999998
Downtime of RTU Main
= (1-Availability)*525,960 = (1-0.999998)* 525,960 = 1
Availability of RTU System
= 1-{(1-ASWA1*ARA1*ARB1*ASWB1)*(1-ASWA2*ASWC*ARC*ARB2*ASWB2)}
= 1-{(1-0.9988)*(1-0.9987)} = 0.999998 (99.9998%)
Downtime of RTU System
= (1-Availability)*525,960 = (1-0.999998)* 525,960 = 1
5.
2

- 10 -

1
WAN
Redundant

2
WAN & Path
Redundant

3
WAN & Path
& VRRP (1)

4
WAN & Path
& VRRP (2)

Availability

99.9399%

99.99985%

99.9899%

99.9998%

Downtime

316

0.08394

53

Additional
Equipment

Ethernet-to-E1
Converter, EoSDH

Ethernet-to-E1
Converter, EoSDH

Ethernet-to-E1
Converter, EoSDH,
LAN Switch

CAPEX

$$$

OPEX

$$

$$

$$

Pros

Local WAN

RTU
Backup

Local WAN

RTU
Main/Backup

Local LAN/WAN

RTU
System

Operation Application

Cons

Local
WAN

Local WAN
RTU Main

Local LAN

RTU Main/Backup

2 4 Fail-over RTU Main Virtual

Default Gateway Fail-over Operation Application
Availability
6.

[1] IP RTU
[2] Di Benedetto, Marco, et al. "High availability architecture for network devices." U.S. Patent No. 7,061,858. 13 Jun. 2006.
[3] Carthern, Chris, et al. "Introduction to Availability." Cisco Networks. Apress, 2015. 383-406.
[4] Shivakumar, V., M. F. Khan, and Amit Jain. "Communication infrastructure for Smart Grid." Power Engineer Journal 18.1 (2016):
19-24.
- 11 -