Brocade 02

Replication:
Connectrix Optimization between Data Centers

Replication:
Connectrix Optimization Between Data Centers
SRDF, SyncIQ, RPA, Data Domain, DLm…
Mark Detrick, Brocade

Director, Storage Networking Principal Architect
2
LEGAL DISCLAIMER
All or some of the products detailed in this presentation may still be under development and certain
specifications, including but not limited to, release dates, prices, and product features, may change. The
products may not function as intended and a production version of the products may never be released. Even
if a production version is released, it may be materially different from the pre-release version discussed in this
presentation.
Nothing in this presentation shall be deemed to create a warranty of any kind, either express or implied,
statutory or otherwise, including but not limited to, any implied warranties of merchantability, fitness for a
particular purpose, or non-infringement of third-party rights with respect to any products and services
referenced herein.
ADX, AnyIO, Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, HyperEdge, ICX, MLX,
MyBrocade, OpenScript, VCS, VDX, and Vyatta are registered trademarks, and The Effortless Network and
The On-Demand Data Center are trademarks of Brocade Communications Systems, Inc., in the United
States and/or in other countries. Other brands, products, or service names mentioned may be trademarks of
others.
Brocade makes no claims, promises or guarantees as to the achievable compression for customer specific
data.
B-Series Extension Leadership
Innovation
• IPEX • Extension HCL
• 40 Gig Extension • Extension Trunking
• 10 Gig Extension • FC Routing
• FICON Acceleration • WAN Optimized TCP
• FastWrite • Adaptive Rate Limiting
• Tape Pipelining • Extension Encryption
• PerPriority-TCP-QoS • High Efficiency
Encapsulation
DellEMC Connectrix B-Series Extension
The Art of Optimization
IPEX is short for IP Extension

Same principles as FCIP, except with IP
IPEX is IP Storage WAN-Security-Availability-Operations Optimization
5 © Copyright 2017 Dell Inc.

Connectrix 7840B Extension Switch
Serial, USB, 24x 2x 16x

mgmt 16G FC/FICON ports 40GE GE/10GE
3 hot-swappable fan modules

Dual redundant, hot-swappable power supplies
Connectrix B-Series SX6 Extension Blade
Only in the new X6 Director

Supports
16 Virtual Fabric
Logical Switches

Connectrix IPEX Solution
The IPEX Architecture
Connectrix 7840B
An Ethernet switch is integral to IPEX:
802.1AX LAG • Local switching
10GE Link Aggregation • More Ethernet ports
Ethernet Switch
Connectrix 7840B IPEX
5 Pillars of IPEX
• High Performance
• Operational Excellence
• Superior Availability
• Solid Security
• Easy Deployment

Performance
The IPEX Experience
The fastest speed replication can go between two devices…
Within the same data center

(300 meters or less)
fiber cable
• Local communications A device will not

• No latency communicate
• No packet loss faster than this
IPEX TCP Acceleration
Local Performance over distance
Brocade’s
High Speed
Heavy Duty
TCP Transport
WAN Optimized TCP
Local TCP session Local TCP session

No latency effects No latency effects
No packet loss No packet loss
Maximum possible bandwidth Maximum possible bandwidth

Connectrix B-Series Compression
Connectrix 7840B: Hybrid Mode
• Fast Deflate (FCIP only)

– 20 Gbps per DP FC side at 2:1
– 2:1 ratio (center of bell curve)
• Deflate
– 16 Gbps per DP IPEX LAN side
– 10 Gbps per DP FC side
• Aggressive Deflate
– 10 Gbps per DP IPEX LAN side
Brocade makes no claims, promises or guarantees as to the
– 10 Gbps per DP FC side achievable compression ratio for customer specific data.

Connectrix B-Series Extension
High Efficiency Encapsulation Remove TCP &
IP data flow to be sent IP Headers
IP IP IP IP IP IP IP IP IP IP IP IP IP IP IP IP Create
16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
32 KB Batches
Compression
Extension Assume 2:1
Batch Header
Compressed Byte Stream (16 frames) 28 B
… TCP Segment
1460 B
TCP
20 B
TCP Segment
1460 B
TCP
20 B
TCP Segment
1460 B
TCP
20 B
TCP Segment
1460 B
TCP
20 B
In this example the
MTU = 1500 bytes
Fill MTU = Max Efficiency

IP Payload IP IP Payload IP IP Payload IP IP Payload IP
1480 B 20 B 1480 B 20 B 1480 B 20 B 1480 B 20 B
IP Packet 4 IP Packet 3 IP Packet 2 IP Packet 1

1500 bytes 1500 bytes 1500 bytes 1500 bytes
Full MTU Full MTU Full MTU Full MTU
Schedule to
Ethernet Egress
High Efficiency Encapsulation Remove TCP &
IP data flow to be sent IP Headers
IP IP IP IP IP IP IP IP IP IP IP IP IP IP IP IP Create
16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 32 KB Batches
Compression
Assume 2:1
… Batch 2
More frames
17+
Extension
Header
28 B
Batch 1
Compressed Byte Stream (16 frames)
Extension
Header
28 B
TCP Segment TCP In this example the

9176 bytes 20 B MTU = 9216 bytes
Maximum Efficiency = fill MTU

IP Payload IP
9196 bytes 20 B
IP Packet 1
9216 bytes Schedule to
Full MTU Ethernet Egress

Use Case: Acceleration
Avoid Droop
10 Gig Throughput MB/s
• Droop 1200
– Transmission protocol incapable of full WAN utilization
1000
• Bandwidth Delay Product (BDP) 800
– Brocade IPEX greatly enhances BDP
600
–
BW
BDP = BW x RTT = bytes available to fill WAN BDP
400
• Brocade IPEX provides BDP 200
– Multiple SRDF TCP sessions are no longer necessary. RTT
0
– SRDF TCP becomes Point-to-Point 0 5 10 20 40 60 100 150 200 250 500

Connectrix B-Series TCP WAN Optimized TCP was
UltraFast Heavy Duty Transport
designed to maintain full
utilization of 2x 10 Gbps links
from Los Angeles to Hong Kong

IPEX Bandwidth Management
Bandwidth Optimization
• Extension Trunking
– Bandwidth pooling
– Always in-order delivery
– Data never lost inflight
– Failover & Failback
• WAN Optimized TCP

– Ultrafast Heavy Duty Transport
– Maintains and optimizes throughput
– Manages every streams independently (vWindows – no HoLB)
• ARL (Adaptive Rate Limiting)

Dual WAN Connections
Physical WAN Pipes
10 Gbps
• Multiple WAN connections
10 Gbps

Single logical WAN connection
Logical WAN Pipe
• Bandwidth Pooling
• Create single virtual connection
20 Gbps using multiple pathways

Bandwidth Allocation to Storage
Allocated Bandwidth
10 Gbps
Not for storage use
• Storage applications not typically
permitted to use all the enterprise
bandwidth
10 Gbps
Available to storage

Policing of Permitted Allocation
Policed WAN Bandwidth

• IP storage applications intolerant of
packet drops
• Results in extreme performance
degradation
• IP network imposes strict limits
• Policed by packet drops
• WAN BW itself is also a limit
• Oversubscribed WAN?

Manage Independent TCP Streams
WAN Pipe
• TCP Virtual Windows
TCP Sessions = Streams

– vWindows
• Many IP storage flows

• Prevent Head of Line Blocking
• Deal with slow-drain devices

QoS
WAN Pipe
• Offer application priorities

• Egress enforcement
• Marking:
High
• Differentiated Services
(DSCP at L3)
Medium
• 802.1P (L2 CoS at L2)
Low

Managing flows at the end-device is impossible!
WAN Pipe
• How do you manage hundreds of flows
over the WAN?
• Operationally, can rate limiting on end-
Maximum devices be effective? No!
Permitted • How could end-devices optimally
adjust their own rate limit? Not
possible!
• How could rate limiting be done in a
failure scenario?

Adaptive Rate Limiting (ARL)
WAN Pipe
• Optimize bandwidth utilization via flow

control
Ceiling • Adaptively adjust rate limit between floor
& ceiling
• Floor – Push at least that much
Floor • Ceiling – Never push more
• Prevent packet drops

End-to-End Flow-Control
TCP No Head of Line Blocking! TCP

Streams Streams
Stop!
Stop! Stop!
WO-TCP vWindows
IPEX controls Each stream has its End-device

stream into IP own WAN Optimized needs to slow
WAN by vWindow down!
Closing Rwnd Virtual TCP Window Closes Rwnd
Rwnd = TCP Receive Window

End-to-End Flow-Control
TCP TCP
Streams
Down Streams
Slow
WO-TCP vWindows
No WAN overflow
IPEX controls
all streams into
IP WAN by
Closing Rwnd
Rwnd = TCP Receive Window

Mortal Enemies to Replication Performance
Latency: Slows every acknowledgement across the WAN,

dramatically slowing performance
Packet Loss: Protocol response is to exponentially back-off,

slowing replication performance to a crawl

IPEX TCP Acceleration
The Results: % Improvement with IPEX
RTT No Loss 0.1% Loss RTT No 0.1% RTT No Loss 0.1% Loss
Loss Loss
10 26% 2748% 2 0% 1600%
10 22% 3900%
25 145% 7536% 10 0% 8300%
25 80% 5233%
60 645% 11467% 100 665% 52400%
50 416% 6300%
Data Domain SRDF/A (RE)
Isilon SyncIQ
• Without IPEX, RPA WAN • DLm uses VNX Data Mover or Note:
throughput drops significantly VMAX3 SRDF to replicate data. Encryption enabled
when latency or packet loss • Without IPEX, throughput drops on all tests
increase, especially with packet significantly when latency or
loss. packet loss increase, especially
Recoverpoint with packet loss.
DLm
Visibility, Monitoring & Alerting

IP Storage When do you have a problem?
Where do you have a problem?
No visibility or control!
What happens when there is a problem?
Storage vMotion
VMware Replication
Database Replication
NAS Replication
• No visibility
Host-Based Replication
Storage Clusters
IP Tape
IP •
•
•
No flow management
No flow rate limiting
No QoS
WAN • No encryption
Array IP Replication
? •
•
•
No compression
No SLA reporting
No end-to-end
diagnostics
Mainframe

Connectrix IPEX
See your storage flows
Storage vMotion
VMware Replication
Visibility, Monitoring,
Alerting, Reporting
NAS Replication
Tape
Storage
Storage Clusters
Array Replication
IP Manageability
Security/Availability
Brocade Tunnel
Mainframe

Connectrix B-Series Manager
Policies, Monitoring and Alerting
Extension Dashboard
• FC Throughput (Rx & Tx)
• Compression Ratio
• QoS Throughput
• WAN Utilization
• Round Trip Time (RTT)
• Jitter (Variance)
• Dropped Packets
• Slow Starts
• Fast Retransmits
• OOO Segments
• Duplicate ACKs
• Circuit Status
• Tunnel Status
WAN Test Tool (Wtool) Diagnostics
• Non-disruptive tunnel testing
• Generates test traffic between a pair of IP

addresses at a specified rate
• Reports
– Throughput
VE_Port
VE_Port
– RTT Circuit 0 operational
Circuit 0 In-Test
– Packet loss
– Packet re-order Circuit 1 operational
Circuit 1 In-Test
Previous run’s result is stored for

historical reference

High Availability
Connectrix B-Series Extension Trunking
VE_Ports, Tunnels, Circuits, and IP & Ethernet Interfaces
Logical Physical Physical Logical

Source 10GE 10GE Destination
IP Interfaces Interfaces Interfaces IP Interfaces
Circuits
IP interface GE2 GE2
VE_Port VE_Port
Tunnel IP WAN Tunnel
Extension Extension
Trunk IP interface GE5 GE5 Trunk

High Availability
Extension
TCP Sessions
FCIP IPEX
Trunk
Control
High High Circuit 0 Benefits
Med Med GE2 • Aggregate bandwidth
Low Low
• Load Balancing
Control
High High Circuit 1 • Lossless
Batches DWRR Med Med GE3 • In-order Delivery
across member Low Low
• Multiple link support
circuits within a Control
High High Circuit 2 • FastWrite
trunk
Med Med GE4 • OSTP
Low Low
• FICON Acceleration
Control
High High Circuit 3
Med Med GE5
Low Low
PTQ in each circuit

High Availability
Protocol Optimization Protocol Optimization

TCP Acceleration TCP Acceleration
FICON Acceleration
Supervisor TCP Session FICONData Received
Acceleration
FastWrite FastWrite
OSTP 3
1 OSTP
Circuit 1 with TCP sessions
Scheduler
Extension Circuit 2 with TCP sessions Extension
Batches 4
3
2 3? Batches
Lossless Link Loss

Metrics & Failover Groups
Extension Trunk
Circuit 0, Group 0, Metric 0 - Active

VE_Port
Group 0
Circuit 1, Group 0, Metric 1 - Backup
Circuit 2, Group 1, Metric 0 - Active

Group 1
Circuit 3, Group 1, Metric 1 - Backup
Lossless and In-order delivery

across failover/failback
Extension Hot Code Load

Extension Hot Code Load (eHCL)
• FCIP and IPEX non-disruptive firmware updates

– Unique to B-Series Extension
• Will not cause mainframe IFCC or SCSI/FCP errors

– Lossless guaranteed
– In-order delivery guaranteed
• Simple configuration
– Requires 1 additional IP address at each circuit end
– Configured once at time deployment

IPEX Security

Connectrix B-Series Encryption
IPEX Data Inflight Security
• Is encryption a requirement or just a good idea?
• IP storage encryption for

– devices with severe encryption penalty
– devices that cannot encrypt
• High-Speed, Line-Rate (HW implementation)
• Nearly no added latency (HW implementation)
• AES 256, SHA512 HMAC, IKEv2, Suite B
• Base unit, no added license cost

Use Case: IPsec Encryption (data inflight)
Prevent Eavesdropping and Attacks
AES
IP Storage
IP Storage
256
Streams Streams
Connectrix 7840B
FCIP+IPEX

IPEX DEPLOYMENT

Connectrix IPEX Deployment
Typical Connectrix B-Series Extension Architecture
IP Storage End Devices
IP Storage End Devices

10GE 10GE
10 Gbps
IP WAN IP WAN
10 Gbps
Service
Provider(s)
10GE 10GE
DC LAN Connectrix WAN Router WAN Router Connectrix DC LAN
7840B 7840B

Connectivity & Gateways
Storage vMotion IP Router

VMware Replication
Original gateway
Local flows come here
NAS Replication
Tape
Storage Clusters
Brocade Tunnel
DC LAN
switch
LAG VLAN Trunk Extension Trunk
802.1AX & 802.1Q towards WAN
Array IPEX gateway
Direct Connect Remote flows come here
49 © Copyright 2017 Dell Inc. Connectrix 7840B
Connectivity & Gateways
DC network
VMotion switch/router Default Gateway
Production network - Not part of
Oracle Data Guard IPEX. Local flows continue to come
Isilon SyncIQ here.
DLm Mainframe Tape
RecoverPoint
Data Domain Data Center
Replication VLAN
VMAX
SRDF RE
Trunked LAG Brocade Tunnel
802.1AX & 802.1Q
IPEX Gateway Circuit1

Flows to remote data center come here Extension
Trunk
Connectrix Core
7840B Routers
IPEX USE CASES
Common IPEX Use Cases
Solve Storage Mobility Problems
• Performance & Acceleration
• Aggregation & Congestion Avoidance
• Encryption
• Visibility, Monitoring, Reporting & Control
• HA, Load Balancing & Bandwidth Pooling
• Data Center Migrations
• Tape Grids
• Cloud Replication

Use Case: Aggregation
Bandwidth Management with Flow Control
• Oversubscribed WANs
• Link Aggregation
– Actual customer examples:
› 44x 1 GbE  2x 10 Gbps WAN – Oversubscribed
› 40x 1 GbE  3 Gbps WAN – Oversubscribed
– Simplifies network connectivity
– Aggregate all array & tape IP traffic into managed flows

Use Case: Aggregation
Bandwidth Management with Flow Control
RecoverPoint
Domain
VMAX3
VMAX3
Isilon
Isilon
DLm
DLm
Data
MF
MF
4x 10GE 4x 1GE 2x 10GE 2x 10GE 12x 10GE 12x 10GE 12x 10GE 4x 10GE 4x 10GE 4x 10GE
Data Center LAN switch
WAN Routers
WAN: 2x 10
Gbps links
WAN Routers
MF
MF
Data
Domain
VMAX3
VMAX3
DLm
DLm
Isilon
Isilon
RecoverPoint

RecoverPoint
Domain
VMAX3
VMAX3
Isilon
Isilon
DLm
DLm
Data
MF
MF
End-to-End Flow Control

Connectrix 7840B
WAN Routers
WAN: 2x 10
Gbps links
WAN Routers
Connectrix 7840B
MF
MF
Data
Domain
VMAX3
VMAX3
DLm
DLm
Isilon
Isilon
RecoverPoint

Use Case: Congestion Avoidance
Full-Mesh Causes Congestion
RE0 RE0
RE1 RE1
RE2 RE2
RE3 RE3
SRDF SRDF
Congestion points

Use Case: Congestion Avoidance
Point-to-Point – no congestion
RE0 RE0
RE1 RE1
RE2 RE2
RE3 RE3
DC LAN Connectrix Connectrix DC LAN
SRDF 7840B 7840B SRDF

Use Case: HA & Bandwidth Pooling
Increase Bandwidth with High Availability
WAN
Carrier 1 Connectrix 7840B
No requirements for:
• Same WAN BW
• Same WAN RTT
WAN
Carrier
2
Extension Trunks
Bandwidth Pooling
Use Case: Balance
WAN A
WAN B
WAN C
WAN D

Use Case: Mainframe Tape Grid
FCIP+IPEX
DLm path A
DLm path B
Connectrix
DLm DLm
DC LAN 7840B WAN Router
IP Network
LAG
10GE 10GE 10GE
Local Site Remote Site
DLm Mainframe Tape

High Availability Architecture
Use Case: Cloud Replication
NAS Replication: Tape Replacement
No more tape-by-truck
Remote
Site
Local Isilon Remote Isilon
SyncIQ Adjacent
Primary Cloud
Site Provider
VM VM
VM IP WAN VM
VM VM
DC LAN Connectrix
7840B IPEX

IPEX: The More EMC… The Better!
Brocade IPEX

IP Storage with and without IPEX
Test results: EMC VMAX3 Adaptive Copy RE ports
Latency ms RTT No packet loss 0.1% packet loss

0 215 241 217 225
5 213 229 216 69
10 212 238 213 43
25 213 218 190 8
50 191 129 128 16
100 110 71 70 13
200 69 36 44 10
EMC Adaptive Copy with IPEX
EMC Adaptive Copy without IPEX
Note: IPEX numbers are with AES 256
Results in MB/s encryption enabled

Test results: EMC RecoverPoint over IP

0 222 224
80 221 2.7
200
EMC RecoverPoint with IPEX
EMC RecoverPoint without IPEX
Results in MB/s
Note: IPEX numbers are with AES 256

encryption enabled

Test results: EMC RecoverPoint over FCIP

0 187.7 160.7
30 176.3 154.6
80
157.8 82.5
200
EMC RecoverPoint FastWrite Enabled

EMC RecoverPoint FastWrite Disabled
Results in MB/s Note: IPEX numbers are with AES 256
encryption enabled

Brocade 02

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brocade 02

Uploaded by

Copyright:

Available Formats

Replication:

Connectrix Optimization between Data Centers

Mark Detrick, Brocade

IPEX is short for IP Extension

IPEX is IP Storage WAN-Security-Availability-Operations Optimization

5 © Copyright 2017 Dell Inc.

Serial, USB, 24x 2x 16x

3 hot-swappable fan modules

Only in the new X6 Director

7 © Copyright 2017 Dell Inc.

9 © Copyright 2017 Dell Inc.

Within the same data center

• Local communications A device will not

WAN Optimized TCP

Local TCP session Local TCP session

12 © Copyright 2017 Dell Inc.

• Fast Deflate (FCIP only)

– 4:1 ratio (center of bell curve)

Fill MTU = Max Efficiency

IP Packet 4 IP Packet 3 IP Packet 2 IP Packet 1

TCP Segment TCP In this example the

Maximum Efficiency = fill MTU

15 © Copyright 2017 Dell Inc.

16 © Copyright 2017 Dell Inc.

17 © Copyright 2017 Dell Inc.

• WAN Optimized TCP

• ARL (Adaptive Rate Limiting)

19 © Copyright 2017 Dell Inc.

Physical WAN Pipes

20 © Copyright 2017 Dell Inc.

Logical WAN Pipe

21 © Copyright 2017 Dell Inc.

22 © Copyright 2017 Dell Inc.

Policed WAN Bandwidth

23 © Copyright 2017 Dell Inc.

• TCP Virtual Windows

TCP Sessions = Streams

• Many IP storage flows

24 © Copyright 2017 Dell Inc.

• Offer application priorities

25 © Copyright 2017 Dell Inc.

26 © Copyright 2017 Dell Inc.

• Optimize bandwidth utilization via flow

27 © Copyright 2017 Dell Inc.

TCP No Head of Line Blocking! TCP

IPEX controls Each stream has its End-device

Rwnd = TCP Receive Window

Rwnd = TCP Receive Window

Latency: Slows every acknowledgement across the WAN,

Packet Loss: Protocol response is to exponentially back-off,

30 © Copyright 2017 Dell Inc.

32 © Copyright 2017 Dell Inc.

33 © Copyright 2017 Dell Inc.

34 © Copyright 2017 Dell Inc.

• Non-disruptive tunnel testing

• Generates test traffic between a pair of IP

Previous run’s result is stored for

36 © Copyright 2017 Dell Inc.

Logical Physical Physical Logical

38 © Copyright 2017 Dell Inc.

PTQ in each circuit

Protocol Optimization Protocol Optimization

Lossless Link Loss