BRKOPT-2116 - High Speed Optics For DC Fabric & Optical Transport - Las Vegas - 2017

High Speed Optics for
Data Center Fabric &

Optical Transport
Errol Roberts, Distinguished Systems Engineer
Mala Krishnan, Director, Product Management
BRKOPT-2116
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
Cisco Spark spaces will be cs.co/clus17/#BRKOPT-2116

available until July 3, 2017.
BRKOPT-2116 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Introduction
• Industry and Market Trends
• Industry Standards
• Client Side Optics - Technologies
• Line Side Optics -Technologies
• Conclusion
Setting the stage......
ITU-T define OTN Transport Network
carrying Ethernet Traffic
IEEE defined Ethernet Regen

OA
TxP
Mux/Demux
ROADM
DC & Client Optics Line Optics

• Within a building or campus or city  Across country (a few hundred to 1000s km)
• Grey optics  Multiple channels / Fiber (DWDM)
• Ethernet  OTN
• Solve for Cost, Power, Density  Solve for Spectral efficiency and Performance
Last Mile Connectivity and Applications
that Drive Traffic
Wireless
STB
Residential
Corporate
Enterprise
WiFi
Hot Spot
Growth in the Cloud
By 2019, 83% of global data center traffic come from cloud
services and applications
18% of traffic between datacenters & end users
9% between data centers for replication & inter-

database links
73% will be coming from within the datacenter,

storage, production, development and
authentication traffic
1 zetabyte is trillion gigabytes

Industry Trends
Ethernet Roadmap
Evolution of New speeds:
25Gb/s, 50Gb/s, 200Gb/s, 400Gb/s

2.5Gb/s & 5Gb/s for wireless
Development and deployment at

points all along the Ethernet value
chain have exploded!!
6 Ethernet speeds in 5 years, same

amount as the previous 40 years
provided !!
The Impact of Cloud Data Center
Success of Ethernet protocol has
shifted consumption to wider range of
applications and markets
When market timing differences

occur, different solutions emerge.
When market timing overlaps,

common solutions emerge.
All markets benefit from economy of

scale and converged technologies
Network Migration Trends
Place in Network 2014/ 2015 2016/ 2017 2018/ 2019
Hyper scale Cloud server 10G → 40G 10/40G → 25/50G* 25/50G* → 50/100G
access
Other Cloud server 1G → 10G 10G → 25G 25G → 50G
access
Enterprise server access 1G → 10G 1G → 10G 10G → 25G
Enterprise Edge Routing 1G → 10G 10G → 100G 10G → 100G
SP/Cloud Core Routing 10G → 40/100G 10G → 100G 100G→ 200/400G

Each part of the network is migrating to a different speed
*2x25G
Data Center Switch Deployments
• Data Centers
• Cloud DC
• Service Provider
DC
• Enterprise DC
• Growth Segment:
• Cloud DC
• Service Provider
DC
• Growth Speeds:
• 100GbE
• 25GbE
Source: BRKOPT-2116 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Enterprise Switch Deployments
WLAN:
• 1GE to 2.5G/5G
Access points
Uplinks:
• 10G to 25G+
Source: DELL’ORO GROUP INC ADVANCED RESEARCH REPORT ETHERNET SWITCH DEPLOYMENT LOCATION VOL. 6, NO.1 ARR95B
Pluggable transceivers Trends
• Cost, reach, size, power optimization continues
• QSFP28 form factor is the de-facto for high density
100G
• Same port density as 40G QSFP
• SFP28 is the 25G form factor

• Same port density as 10G SFP+
• QSFP-DD emerging as the 400G form factor

• Same port density as QSFP 40G and QSFP 100G
• Advances in high order modulation techniques –

electrical and optical.
• Market need for flexible interfaces on client devices
• 4x10G breakout, 4x25G breakout …..
Data Center Fabric Trends
• High bandwidth Fabric • Early Integration of 100G north-south
• Redundancy Model focused DC edge
• Physical Infrastructure
• Hash Efficiency Improvements
w/high speed links
Multiple Spines 100G Links
40 -100G Links
Wider Spine 10 - 25G Servers
Higher Scale Compute

Ethernet Physical Interface Standards
1G 10G 40G 100G 25G 400G
40km
ER 40km 100G-ER4
SMF LX LR 10km 40G-LR4 100G-LR4 10km 400G-FR8 10km
40G-FR 2km, Optical

Transport only 400G-DR4 500m
LX4 300m
Ribbon cabling
MMF SX SR 300m (N x 4, N x 10)
LRM 220m 40G-SR4 100m OM3

100G-SR10
T 100G-SR4 25G-SR 400G-SR16 100m
OM4
Copper T CX4 15m
40G-CR4 100G-CR10
100G-CR4 25G-CR
IEEE continued to standardize a few reaches for each new speed……but these Standard PMDs do not sufficiently meet the
requirements of new and emerging applications.
Additional Interface Options …….
1G 10G 40G 100G 25G
10G DWDM 100G-ER4L
25G-LRL
10G ZR 40G-LRL4/Lite 1-2km
SMF 100G-CWDM4
40G-PLR4 1km
100G-SM-SR
40G-UNIV 500m SMF  And
100G-PSM4
40G-CSR4 300m
more ‘non-
MMF 40G-XSR4 300m standard
FET10 100m SWDM4 interfaces
SRL 100m Cisco BiDi 100m BiDi to come....
AOC Cables AOC <10m AOC <30m AOC <30m AOC <10m
So what was Industry’s response?

• Create MSA (Multi-Source Agreements) targeted to specific applications (CWDM4, PSM4 etc)
• Release Non-Standard optics BiDi, UNIV, FET, SM-SR etc
More choices, but also fragments the market…..
Datacenter Connectivity Trends
• Use of Multimode fiber for forward Enterprise & Mega-scale
and backward compatibility in DC Fiber deployments
migration from 10GE->40GE->100G
where possible
• Interconnect costs affect physical
architecture
• DC Architecture is driving cost of
short reach single mode optics down
• Single mode deployment for future-
proofing fiber infrastructure .. Why?
Single mode
fiber
deployments
Network fiber migration with increase in data rates
Multimode Fiber Single Mode Fiber
1G
1km 10km
10G
400m 10km
40G
150m 10km
2km
100G
100m 10km
500m 2km
Shorter link distances with higher data

rates on OM4 multimode fiber. Short reach optics for single mode fiber at higher data rates.
Cisco QSFP28 100G Transceivers For your
reference
IEEE compliant 100G-CR4 Multiple Mode Fiber, Parallel

<5m Copper
Multiple Mode Fiber, Duplex
4x SFP28 Breakout
IEEE compliant
<5m Copper Single Mode Fiber, Dual
(25G)
100G AOC Single Mode Fiber, Parallel
<30m Active Optical
Cables
IEEE compliant 100G SR4 12 fiber MPO (parallel fiber), MMF
100m on MMF (OM4)
100G BiDi Duplex fiber LC, MMF
100m on MMF (OM4)
100G LR4
IEEE compliant Duplex fiber LC, SMF
10km on SMF
100G PSM4 12 fiber MPO (parallel fiber), SMF
MSA compliant 500m on SMF
100G CWDM4
MSA compliant Duplex Fiber LC, SMF
2km on SMF
100G SM_SR
MSA Compatible Duplex Fiber LC, SMF
2km on SMF
100G ER4L Duplex Fiber LC, SMF

25km on SMF (40km with FEC)
5m 30m 100m 500m 2km 10km 25km 40km 80km
Transceiver Industry Trends
QSFP 40G to QSFP 100G Transition by reach
QSFP QSFP
5m 10m 30m 100m 150m 300m 500m 2km 10km 25km 40km 80km
40G 100G
COPPER CABLES
1 Direct Attach Copper cables YES YES
Breakout Copper Cables YES YES
OPTICAL CABLES
2 Active Optical Cables YES YES
Breakout Active Optical Cables YES YES
PARALLEL FIBER MULTIMODE OPTICS 100G 40G

3 SR4 (OM4) YES YES
CSR4 (OM4) YES YES
4
DUPLEX FIBER MULTIMODE OPTICS 100G 40G
BiDi YES YES
5
PARALLEL FIBER SINGLEMODE OPTICS 100G 40G
4x10G-LR PSM4
DUPLEX FIBER SINGLEMODE OPTICS

6 (SHORT REACH)
LR4-Lite CWDM4

7 LR4 YES YES 100G 40G
Transceiver Industry Trends
SFP 10G to SFP 25G Transition by reach
SFP SFP
5m 10m 30m 100m 150m 300m 500m 2km 10km 25km 40km 80km
10G 25G
1
COPPER CABLES
Direct Attach Copper cables YES YES
2
OPTICAL CABLES
Active Optical Cables YES YES
3
DUPLEX FIBER MULTIMODE OPTICS 25G 10G
SR YES YES

4
LR YES YES
ER YES YES
100G QSFP Cables
100G Copper Cables 100G Copper Breakout Cables

(upto 5m) 100G Active Optical Cables (AOC)
(upto 5m)
QSFP 100G to QSFP 100G (upto 30m)
QSFP 100G to 4 x SFP25G
QSFP 100G to QSFP 100G
Cable diameter
• AOC cables are thinner than twinax Copper cables

• Enables improved air Flow
• No Electromagnetic Interference
• Reach upto 30m (while Copper cables have reach
limitations with increasing speeds)
• Less weight and Flexible
3mm >10mm
• Significantly simplifies Cable management
Active Optical Twinax
Cable copper cable
Hyperscale Datacenter Trends
• Continued evaluation of Multimode vs. Single mode for
future bandwidth needs
• DC Campus can consume many acres and require
significant amounts of fiber interconnect
• DC Architecture is driving cost of short reach Single
mode optics down
• Interconnect costs affect DC physical architecture
• Increased choice of cabling options to address density,
bandwidth and reach
Enterprise Datacenter Trends
• Use of Multimode fiber for forward and backward
connectivity in migration from 10GE->40GE->100G
where possible
• Continued evaluation of Multimode vs. Single mode for
future bandwidth needs
• Interconnect costs affect DC physical architecture
• Increased choice of cabling options to address density,
bandwidth and reach
For your
reference
Data Center Cabling Architecture Considerations

• Optics will drive physical architecture
• Employ a main cross-connect at a MDA, for best management, scalability and growth
• Plan for future device optics density and reach;
• Fiber density in the trunk cables – parallel vs duplex fiber optics support
• Reach – multimode vs single mode
• Use low loss connectors to enable MDA
• Allocate rack space for orderly fiber count growth and network device interface scale
• Plan for future high density 10G/25G server access
• Can also be addressed by 25/40/100GE break out cables
• Switch Architecture Type for Access – Modular or ToR
MDA: Main Distribution Area

Optical connectivity increasingly important
for Performance, Scale, and Cost!
Need to optimize all

- cost, power, density -
simultaneously…
Design Choices will vary by

network use-case!
Never ending balancing act!
Industry Standards
New Standards Leverage Previous
Industry Investment
Technology Nomenclature Description Status
Backplanes 100GBASE-KR4 4 x 25 Gb/s (NRZ) Deployed. Legacy 4x installed

100GBASE-KP4 4 x 25 Gb/s (PAM-4) base requiring upgrade
Chip-to-Module CDAUI-8 8 x 50 Gb/s PAM-4 IEEE P802.3bs in Task Force

Review
Chip-to-Chip CEI-56G-VSR-PAM 56 Gb/s PAM-4 Straw Ballot
SMF Optical 400GBASE-FR8 8 x 50 Gb/s PAM-4 IEEE P802.3bs in Task Force

400GBASE-DR4 4 x 100 Gb/s PAM-4 Review
Module Form Factor SFP56 1 x 50 Gb/s Summary Document SFF-8402
QSFP56 4 x 50 Gb/s Summary Document SFF-8665
Client Side Optics -
Technologies
Optical Technology Trends
DC & Client optics
How can we go faster
• Increase Signaling speed
• Evolution of Ethernet speed roadmap
• Adopt advanced modulation formats
• carrying more bits/s, but keep the baud rate low
• Increase number of fibers (Parallelize)
• Aggregating data over several number of fibers/ lanes
• Increase number of wavelengths
• Combining data over multiple wavelengths (WDM)
Parallel Data Streams – Multi Layer Distribution (MLD)
MAC/ PM PM MAC/
PCS D D PCS
(4x25Gb/s)
• Multi-lane Distribution (in the PCS layer) provides a simple way to map
40G/100G to physical interfaces of different lane widths – with Virtual lanes
• Data from any particular virtual lane will reside on the same electrical and
optical lane across the link – No skew introduced between bits within the virtual
lane
Higher Order Modulation
Same Data and Data Rate; but lower frequency (baud rate).
PAM-2 1 (1 level)
0110 100011
1-bit Symbols
1 Level
0 (0 level)
0 Level
1 1 (3 level)
0110 100011
PAM-4 1 0 (2 level) 3 Level
2-bit Symbols
2 Level
0 1 (1 level)
But 4 levels 1 Level
0 0 (0 level)
0 Level
• Lessens many transmission penalties

Why higher order • Enables lower bandwidth components and materials
modulation? • Reduces wavelengths & fibers compared to NRZ
The new Normal – Multi-lane & Reuse
Since 10 GbE, Ethernet

has progressed by
defining pragmatic multi-
lane solutions and fastest
single lane technologies to
produce cost-effective
solutions.
Form factor evolution
ETHERNET SPEEDS
400 GbE 400G QSFP56-DD Form factor evolution

follows multi-lane to
200 GbE 200G QSFP56 single-lane serdes
evolution
100 GbE 100G QSFP28
50 GbE 50G SFP56
40 GbE 40G QSFP+
25 GbE 25G SFP28
10 GbE 10G SFP+
10 Gb/s 25 Gb/s 50 Gb/s
SERDES SPEEDS
How can we go faster? Some examples….
TX
TX
Duplex Duplex Duplex 1 x 25G Duplex Toolkit:
1 x 10G
Multimode Multimode Multimode Multimode Speed-up,
Fiber Fiber Fiber Fiber
WDM,
1 x 10G
1 x 25G Modulation
RX RX
4 x 10G TX 1 x 50G w/ Toolkit:

Parallel Parallel Duplex modulation Duplex Speed-up,
Multimode Multimode Multimode Multimode
Fiber Fiber Fiber
WDM,
Fiber
Modulation
4 x 10G
RX 1 x 50G w/
modulation
The Challenge With Migration:
Current 10G SR vs 40G QSFP SR4, 100G QSFP28 SR4
40G
10G
QSFP SR4
10G TX TX
10G 10G 4 x 10G
12-Fiber 12-Fiber
SR SR infrastructure infrastructure
10G RX 4 x 10G
RX
2 Fiber Strands 8 - 12 Fiber Strands
40G & 100G BiDi Technology
40G BiDi 100G BiDi
 4 electrical lanes of 10G from Host are converted to 20G  4 electrical lanes of 25G from Host are converted to 50G-PAM4
electrical lanes by a Gearbox IC electrical lanes by a Gearbox IC
 20G electrical lanes are converted to 20G optical lanes  50G PAM4 electrical lanes are converted to 50G PAM4 optical
and transmitted over 2 fibers lanes and transmitted over 2 fibers
 The same fibers are used to receive 20G signals and the  The same fibers are used to receive 50G PAM4 signals and the
Gearbox coverts the 20G signals to 4 lanes of 10G before Gearbox coverts the 50G PAM4 signals to 4 lanes of 25G before
sending to Host sending to Host
 The module uses Bi-directional Optical sub-assembly  The module uses Bi-directional Optical sub-assembly (BOSA) to
(BOSA) to transmit and receive 20G optical signals on the transmit and receive 50G PAM4 optical signals on the same
same fiber. fiber. BRKOPT-2116 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Single Lane .. Less Complexity
The 40G & 100G BiDi Example40G BiDi
TX 4 x 10G
TX/RX
2 x 20G Toolkit:
Switch Switch
Switch Switch Speed-up,
WDM,
2 x 20G Modulation
4 x 10G
RX
TX/RX
100G BiDi
TX 4 x 25G TX/RX
2 x 50G Toolkit:
Switch Switch PAM4 Speed-up,
Switch Switch
WDM,
2x 50G Modulation
4 x 25G
PAM4
RX
TX/RX
100G QSFP28 Multimode Optics
λ λ λ λ λ λ
1 2
100m 100m
R R R R R R
X X X X X1 X2
100G BiDi
100G SR4
Duplex Fiber (LC)
Parallel fiber (MPO)
850/910nm wavelengths
100G QSFP28 Single mode Optics
λ λ λ λ λ λ λ λ
λ λ λ λ
1 2 3 4 1 2 3 4
Optical Mux Optical Mux
500m/2km 2km
500m
Optical DMux Optical DMux
R R R R R R R R R R R R
X X X X X1 X2 X3 X4 X1 X2 X3 X4
100G SM-SR 100G CWDM4

100G PSM4
Duplex Fiber (LC) Duplex Fiber (LC)
Parallel fiber (MPO)
Coarse WDM wavelengths Coarse WDM wavelengths
λ λ λ λ λ λ λ λ
1 2 3 4 1 2 3 4
Optical Mux Optical Mux
10km 25km/ 40km
Optical DMux Optical DMux

R R R R R R R R
X1 X2 X3 X4 X1 X2 X3 X4
100G LR4 100G ER4L

Duplex fiber (LC) Duplex fiber (LC)
LAN WDM wavelengths LAN WDM4 wavelengths
Silicon Photonics
Potential for “transceiver on a chip”
• Optical devices can be made cheaply

using standard semiconductor CMOS
fabrication techniques
• Optics can be integrated with
microelectronic chips.
• Silicon integrated optical chips that can
generate, modulate, process and detect
light signals
Silicon Photonics is the most promising optical technology for:

Solving for Cost, Power, Density for DC & Client optics
Solve for Spectral efficiency and Performance for DCI & Long Haul optics
QSFP-DD 400G Module Comparison
QSFP-DD Provides the Highest BW Density of Any Pluggable Module
CXP QSFP-DD
CFP CPAK CFP2 CFP4 QSFP28 uQSFP CFP8 OSFP
86 21 35 42 22 18 14 41 22 18
75 29
50
50
50
Line card faceplate
76
130
91
92
92
83
100G Modules 400G Modules
16 16.2 11.6 12.4 9.5 13.5 12.4 13.5 13.5
Introducing QSFP-DD
Improved thermals supports

>2.5x QSFP power
QSFP-DD
75.85 mm
70.00 mm Essentially the same as QSFP

QSFP but with extra row of contacts.
Allows boards to be backwards
compatible to both.
Why QSFP-DD for 400G?
• The QSFP-DD module form factor has the highest BW density, enabling 14Tb/s in 1RU
• Avoids iteration of form factors for a particular speed, drives industry economy of scale
• QSFP-DD port is backward compatible with all prior QSFP modules
• This form factor leverages the industry’s manufacturing capability and cost structure of QSFP+ and
QSFP28, the de facto standards for 40GbE and 100GbE
Transceivers Evolution Drive Efficiency
Modules use less power for the same bandwidth for density
Transceiver Power for 1Tb/s of Line Card Bandwidth Enabled by

Bandwidth Module Form Factor
300
240
250 16 60
Line Card Bandwidth (Tb/s)
# of Line Card Ports

14 50
200 Line Card BW
Power (W)
12
10 40
150 120
100 8 30
87.5 6
100 75 20
35 4
50 25 2 10
0 0
0
Why QSFP-DD for 400G?
• The QSFP-DD module form factor has the highest BW density, enabling 14Tb/s in 1RU
• Avoids iteration of form factors for a particular speed, drives industry economy of scale
• QSFP-DD port is backward compatible with all prior QSFP modules
• This form factor leverages the industry’s manufacturing capability and cost structure of QSFP+ and
QSFP28, the de facto standards for 40GbE and 100GbE
• Up to the end of 2016 over 7M QSFP (40G & 100G) have been deploy since their introduction
Other Future Industry Trends - Onboard optics
Switch Optical Engine in a Pluggable module
• How do we go beyond 12TB+ ?

Switch Optical engine on-board the line card
• Will pluggables limit the faceplate density?
• >2020: board-level photonics
• COBO – Consortium On-board Optics [switch
face-plate, bandwidth density..
Switch Silicon Photonics Optical Engine
“On-board optics permit greater switch radix mounted on the package
with lower power consumption, which is really
important as we continue to increase speed
and bandwidth,” - COBO
Optical technology…
…Is the key enabler for systems to maintain pace with the
requirements of the bandwidth pressures
Challenge Goal
Form Factor Size reduction Port Density increase
Power reduction Port Density increase
Reach System/network scaling
Packaging simplification Increased Yield, lowers cost
The next speed (400G/1T) System Scale, lower costs

Beyond 40km at High Speeds
10 Gb
40 km
• Chromatic dispersion in SMF spreads

pulses as they propagate, limit the distance
25 Gb
• Limit is shorter at higher data rates
10 km
• Solutions
• 40km 40G in QSFP (40GBASE-ER4)
• High quality laser and receiver
• 40km 100G in QSFP
• compatible with 100GBASE-ER4
• “ER4-Lite”
• High quality laser and receiver
• Leverage FEC
Line Side Optics –
Technoloigies
Setting the stage......
ITU-T define OTN Transport Network
carrying Ethernet Traffic
IEEE defined Ethernet Regen

OA
TxP
Mux/Demux
ROADM
DC & Client Optics Line Optics

• Within a building or campus or city  Across country (a few hundred to 1000s km)
• Grey optics  Multiple channels / Fiber (DWDM)
• Ethernet  OTN
• Solve for Cost, Power, Density  Solve for Spectral efficiency and Performance
Goal: Enable Optical Networks go Farther and
Faster at Lower Cost
Optimize Data Rate vs Drive to Next Gen Data Rates

Distance vs Cost 600G, 800G, 1T, etc…
Figure of Merit: Increase bits per port to
$$/Gb-Km reduce cost per bit,
Go Further, leveraging Silicon advances
Faster at Reduce Operational Costs,
Lower Cost Simplify through Automation
New approaches to increase Optimize network continuously
fiber carrying capacity through Flexibility across Layers
Gain more from existing Automation & Simplification
Infrastructure via SW
Ethernet & OTN Evolution
• Ethernet has undergone continual innovation since standardization
• Transport systems evolving to 400G
FE GE 10GE 40/100GE
Standard Demand and Innovation
Ethernet continue
SDH Payload Eth Payload

OTN Standard Demand and Innovation
OTU1/2 OTU3 OTU4 continue
1985 1990 1995 2000 2005 2010 2015
OTN aligned with Ethernet speeds

G.709 Hierarchy and Frame Structures
G.709 Hierarchy
• G.709 defined a fixed “hierarchy” of Frame Payload (OPU)
payloads ODU0 1,238,954 kbit/s
OTU1 2,488,320 kbit/s
• G.709 started as a digital wrapper OTU2 9,995,276 kbit/s
around WDM client signals to OTU3 40,150,519 kbit/s
improve reach and manageability. OTU4 104,355,975 kbit/s
ODU-Flex Flexible Data Rates
• Recently it has developed into a
complex multiplexing structure. G.709 Digital Wrapper
• ODU-Flex allows flexible sub
wavelength grooming. Payload
MultiplexingStructure
Improve OSNR Performance with G.709 FEC
• FEC extends reach and design flexibility, at
“silicon cost” -3
-4 Pre-FEC
10Log(Bit Error Rate)

-5
• G.709 standard improves OSNR tolerance BER
-6
by 6.2 dB (at 10–15 BER) -7 Post-FEC
-8 BER
• Higher gains (8.4dB) possible by enhanced -9
-10
FEC (with same G.709 overhead) -11 CODING GAIN
-12
• Yet higher gains by soft decision (SD)FEC -13
4 5 6 7 8 9 10 11 12 13 14
• Offers intrinsic performance monitoring
OSNR (dB)
(pre and post-FEC error statitistics)
Industry Standards Groups
• IEEE Control Plane
• Physical interfaces for Backplane, Copper,
Fiber PMDs
IEEE ITU IEEE
• ITU Study Group 15: Optical and Transport OIF OIF
Networks
• OTU4 frame format
Client Interfaces Hardware
• Single mapping for 40GE/100GE into Layer 2/1 Vendors
OTU3/OTU4 (e.g.) interoperability Transport Component
• OTL protocol enabling OTU3/4 over multi- Networks Interoperability,
lane optics Layer 1/0 Commonality
interoperability
• OIF: 100G Long-distance DWDM
Transmission
• 100G standardization coordinated among ITU,
• Industry consolidation around a single
IEEE, and OIF
100G DWDM solution
• Proactively eliminate interop issues encountered
with 40G
Inter-DC connectivity in the Cloud-era
More than DCI
 Enterprise Data Center inter-connect
 Enterprise Data Center to Provider Data
Ent DC1 Ent DC2 Center
 Provider Data Center to Provider Data Center
PE PE “DCI” with varying requirements:

• Multiple data rates needs
SP EPN • Higher Density Interconnect in metro
CE DCPE CE
• Intra DC architecture extend beyond
DCPE metro
DCE
DCE
SP DC1 SP DC2
Tunable lasers and coherent receivers key enablers
Key enablers
Transmitter can tune its laser’s frequency to any Receiver can select any channel from of a
channel in the ITU grid. composite (unfiltered) signal.
Tunable lasers work with colorless add/drop to enable touchless changes in the frequency of an optical signal. Coherent
receivers simplify the construction of colorless and omni-directional ROADM nodes, by eliminating the need to de-multiplex
a signal down to the individual wavelength.
100G Technology: Coherent Detection
Direct Detection
• Must correct for impairments in the physical domain (insert DCU’s)
• Forced to live with non-correctable impairments via network design (limit distance, regenerate,
adjust channel spacing)
• Dumb detection (OOK), no Digital Signal Processing, only FEC
DCU DCU DCU
DD DD
Regen
Coherent Detection
• Moves impairment correction from the optical domain into the digital domain
• Allows for digital correction of impairments (powerful DSP) vs. physical correction of impairments
(DCU’s). Adds advanced FEC.
• Massive performance improvements over Direct Detection.
CD
BRKOPT-2116
– Dispersion
DCU© 2017 Compensation 62Unit
Cisco and/or its affiliates. All rights reserved. Cisco Public
Line Side Pluggable Optics- e.g. CFP2 ACO –
Coherent Optics Pluggable
32GBaud
Integrated Coherent Optical TX

Coherent 4 Transmitter
Optical Signal
DSP
Integrated Coherent Optical
Receiver RX
4
On Host Board CFP2-ACO Module
Optics in the pluggable CFP2, electronics on the line card

Reduces the components inside the module – improved power management
Only a CFP2-ACO can be used on the port; not gray optics
ACO  Analog Coherent Optics

Pluggable Optics
9.6Tbps 4000Km +
100G
19.2Tbps 800Km
200G
24Tbps 500Km
State of the art 28nm integrated ASIC Pluggable
Multiple Line rates – Distance vs reach trade- Field replaceable 250G
off
Deferrable
Soft Decision FEC – Double the reach
Thermal efficiency
Nyquist shaping – ~30% more capacity 96 channel DWDM system
High Speed Client Implementations
Interface Independent Functionality
IP-over-
Router Ethernet
Router Router
OTN
• IP-over-DWDM
DWDM Client Client
• Pre-FEC error threshold is monitored directly by
router
• RP initiates fast re-route based on internal trigger
directly from PLIM
FEC
• Gray Client - Ethernet OTN
OTN
• Pre-FEC error threshold is monitored by transponder Ethernet
Trigger
Trigger
• Ethernet trigger is generated by transponder and
sent to router which initiates fast re-route Transponder
Transponder
• Gray Client - OTN

FEC FEC
• Pre-FEC error threshold is monitored by transponder
• OTN PF-FDI trigger is generated by transponder and
sent to router which initiates fast re-route
• OTN interface monitors end-to-end path
Line Interfaces – Developments
• Doubling of baud rates
64 GB
• Higher data rates: 200G, 400G, 1T
32 GB 32 GB
• Flex Ethernet ...
• Flex Spectrum
{ DSP 200, 250, 300, 350, 400,450, 500, 550, 600
• Encryption
• Flexible modulation techniques
• PM-BPSK, PM-QPSK, PM-16QAM, PM-64QAM, ….
• All come at the expense of reduced distance
• Hybrid Modulation for Improved capacity

• Spectral Efficiency
• Nyquist Filtering
• Dynamic bandwidth filters (Flex spectrum ROADM technology)
Ethernet and signaling rates
Ethernet Rates
Signaling
10 GbE 40 GbE 100 GbE 400 GbE 25 GbE 50 GbE 200 GbE
Rates
Enabled by
2.5 Gb/s
availability
of…
Then drove Enabled by Enabled by

10 Gb/s
development availability availability
of… of… of…
Then drove Enabled by Enabled by

25 Gb/s
development availability availability
of… of… of…
Is driving
50 Gb/s Enabled by
development
availability of…
of …
Will be
optimized by Is driving
100 Gb/s development
of …
~ Chronological
What Will Change to Continue to Scale?
28G 56G Electrical Optical Fibers
SMF or MMF
28G 56G Electrical Electrical
Analog
Lanes Lanes
400G 400G
S&R Coherent 600G Line
ASIC DSP 200G System
400G 400G
DWDM
Multi-Terabit Multi-Rate Optic
Processor DSP
100G 200G/400G Grey Optics
SR-10, SR-4, CWDM-4, +8QAM
LR-4 Lite, LR-4
+32QAM
‣ Client Side Optics: QSFP28, QSFP56,
QSFP28-DD, QSFP56-DD, COBO +64QAM
‣ Flexibility: FlexE, FlexMod, and FlexRate Double Baud Rate
Network Optimizagtion .......
Flexible, efficient and dynamic mapping of packet services to optical transport
NPU Transponder DWDM
Superchannels
2x50G 2x200G
Flex FlexE Flex Mod FlexFlexspectrum

Spectrum
Flexible nx/25G 50G/100G/200/250G/+ Flexible nx12.5Ghz
12.5/50Ghz/100Ghz
interconnects DWDM interfaces channels
DWDM systems
Tools to Increase and Optimize System Capacity
Speed up & WDM & Modulation
Multi-Modulation Superchannel Formation

Adjust spectral efficiency by transmitting Optimize spectral efficiency by tightly
more or less bits per symbol at a given spacing sub-carriers using Flex-
baud rate Spectrum ROADM
Flex Baud Rate Hybrid Modulation

Increase the bit rate by increasing Mix different modulation formats in the
the rate at which symbols are time domain to achieve a greater
transmitted bandwidth/distance granularity
Higher order modulation schemes
BPSK QPSK 8-QAM 16-QAM 64-QAM
Bits/
Symbol 2 4 6 8 12
(2 pol.)
Capacity 50G 100G 150G 200G 300G

@32Gbaud
Increasing the capacity per WDM channel requires sending

more complex signals; trading between spectral efficiency
and reach
3x (64-QAM vs QPSK) gain
Optimize Data Rate vs Distance
Flexible Modulation (FlexMod)
• Different modulations provide

different capacity
• Different modulations provide
different reach
• Chip sets provide the ability
to SW config Capacity vs
Reach
• Flex Mod will close the Gap
and provide for 25Gig
granularity
Baud rate increase
• Higher baud rates

• 2x rate (1/2 carriers), but with Nyquist spectral
density improvements only below 5%
• While higher NLI and implementation
impairment might indeed affect reach/bandwidth.
Meng Qiu, Qunbi Zhuge, Xian Xu, M. Chagnon, M. Morsy-Osman, and
David V. Plant ‘Subcarrier Multiplexing Using DACs for Fiber
Nonlinearity Mitigation in Coherent Optical Communication Systems,’
in
Proc. OFC 2014, paper Tu3J.2, San Francisco (CA), Mar. 2014.
• Will higher baud rate be really required by bandwidth demand ?

• No, transport over multiple wavelengths will be supported by new protocols (FlexE,
OTUCn)
• But cost and real estate reduction will justify it.
DWDM Network capacity limited
by channel spacing imposed with fix ITU Grid
Superchannels require a new kind of ROADM, one that can switch
chunks of bandwidth larger and yet more granular than 50GHz
50 GHz ITU Grid 50 GHz ITU Grid

12.5 GHz Slices
Superchannel with Minimal Spacing

Rigid Spacing,Wasted Spectrum
Efficient Spectrum Use
Each 50GHz carrier provisioned and Superchannel switched through the
switched individually ROADM network as a single entity
This requires Flexible Spectrum allocation – Flex Spectrum

FlexSpectrum & Capacity Relationships
80ch system: From 4Tb/s to From 16Tb/s to 24Tb/s
6Tb/s per fiber Moving from Moving from 50GHz to
50GHz to 33GHz Chs spacing 33GHz Chs spacing
From 8Tb/s to 12Tb/s per fiber

Moving from 50GHz to 33GHz
Chs spacing
Adoption of hybrid modulation
Interleaving of different modulation schemes
symbols enables continuous optimization of the
capacity for every reach.
Flex Mod and Flex E allows for the mixing of
different Constellations providing for 25GHz of
Data Rate Granularity
16-QAM QPSK 16-QAM QPSK 16-QAM
time
Time Domain
Hybrid 32 GBaud BPSK BPSK BPSK BPSK BPSK BPSK BPSK 50G
Modulation BPSK
QPSK
QPSK
QPSK
BPSK
QPSK
QPSK
QPSK
BPSK
QPSK
QPSK
QPSK
BPSK
QPSK
QPSK QPSK QPSK 16QAM QPSK QPSK QPSK

 Mix of constellations in
QPSK 16QAM QPSK 16QAM QPSK 16QAM QPSK
the time domain
16QAM 16QAM QPSK 16QAM 16QAM 16QAM QPSK
 Variable reach 16QAM 16QAM 16QAM 16QAM 16QAM 16QAM 16QAM 200G
 Signal processing needs 16QAM 16QAM 16QAM 64QAM 16QAM 16QAM 16QAM
only small changes 16QAM 64QAM 16QAM 64QAM 16QAM 64QAM 16QAM
 Add one new modulation 64QAM 64QAM 16QAM 64QAM 64QAM 64QAM 16QAM
format 64QAM 64QAM 64QAM 64QAM 64QAM 64QAM 64QAM 300G

16QAM 16QAM 16QAM 64QAM 16QAM 16QAM 16QAM

42 GBaud
64QAM 64QAM 64QAM 64QAM 64QAM 64QAM 64QAM 400G


56 GBaud 64QAM 64QAM 16QAM 64QAM 64QAM 64QAM 16QAM
64QAM 64QAM 64QAM 64QAM 64QAM 64QAM 64QAM 600G

Dynamic Data Rates
Current Solutions are Still too Inflexible – FlexE and FlexMod optimize Capacity vs
Reach What we need:
400Gbps 64QAM More granularity and uniformity of
bandwidth / distance steps
Today’s Hierarchy defined in the IEEE and ITU provides
network limitations
Terminate predefined Standard Data Rates
10G, 100G, 400G, …. 1.6T?
200Gbps 16QAM Data Rate specified by reach
Bit Weakens ability to optimaze DR vs. Reach
Rate
Wasted Capacity
100Gbps QPPSK
50Gbps BPSK
Distance / OSNR
Optimize Data Rate vs Distance
Flex Ethernet (FlexE) – World’s first Demo OFC 2016
• Ability to leverage the full capacity of the NPU
• Ability to specify any Data Rate with no Hashing inefficiencies
• Ability to grow the Data Rate in 25/50G granularity upto max NPU capacity
independent of IEEE or ITU hierarchies
• Ability to dynamically adjust data rates to match the physical layer performance
400Gig
400Gig 400Gig
X
NPU NPU
50Gig
350Gig
Truly Optimize Data Rate vs Reach
Optimize Data Rate vs Reach
Flex Ethernet – Beak existing dependencies
FlexE Client a FlexE Client b FlexE Client z
64B/66B 64B/66B 64B/66B Error Control

encode encode encode Block
● ● ● Generator
Client
Clock
• Enable flexibility on Ethernet interface: Domain

FlexE
Clock
Idle insert/
delete
Idle insert/
delete
66B block distribution and
Idle insert/
delete
Domain Insertion into master calendar
• Eliminate existing Hierarchies - Custom rates Control Calendar
Calendar
Overhead Insertion FlexE Client a FlexE Client b Distribution
FlexE Client z
• Eliminate LAG Inefficiencies
Sub-calendar
Sub-calendar
Sub-calendar
● ● ●
64B/66B 64B/66B 64B/66B
PHY N
PHY B
PHY A
decode decode decode
● ● ●
• Match transport bandwidth FlexE defined
sublayers
Idle Idle Idle
Client
Clock
Doma
insert/ insert/ insert/
FlexE
in
• Break Ethernet and transport roadmap marriage

802.3 defined delete delete delete
sublayers 66B block extraction from Clock
Scramble Scramble master calendar Scramble Doma
in
Control
PCS lane distribution Calendar
and AM insertion
Deskew
100GBASE- 100GBASE- 100GBASE- Calendar
• Standardized under OIF Overhead extraction

R PMA R PMA R PMA Interleaving
Sub-calendar
Sub-calendar
Sub-calendar
LF generator
PHY A PHY B ● N
PHY ● ●
PHY B
PHY N
PHY A
Lower Layers Lower Layers Lower Layers
and PMD and PMD and PMD
• Effort initially patented by Cisco FlexE defined

sublayers
802.3 defined
sublayers
Descramble Descramble Descramble
• Implementation led by Cisco, Google, Juniper PCS lane deskew,

interleave, and
AM removal 100GBAS 100GBAS 100GBAS
E-R PMA E-R PMA E-R PMA
and others
Above figures are copied from the OIF FlexE 1.0 Implementation
agreement. Available on oiforum.com early in 2016.
Optimizing Transmission Efficiency
NPU Friction Ethernet Friction Transponder Friction DWDM
50G
10G 100G
25G 150G
200G 40G 200G ITU
 50G 250G 50GHz
Terabits 100G 300G Grid
200G 350G
400G 400G Flex Grid
450G, 500G, 550G, 600G
NPU Capacity Growing Interconnect Efficiency Spectrum Efficiency

We need to reduce friction between different layers of the network to get the
Client side, Line side optics, Flex modulation, FlexE, FlexGid innovations are the key to success
Summary
Agenda
• Introduction
• Industry and Market Trends
• Industry Standards
• Client Side Optics - Technologies
• Line Side Optics -Technologies
• Conclusion
Alignment Needed to Achieve TCO Goals
DC Network Infrastructure
ASIC BW & Port
Speed
• Innovation
• Standardization
• Open Ecosystems
Cabling Optics Speed and

Infrastructure Size
Summary - Trends
• Transition infrastructure to duplex fiber for lower overall cost
• Balance choice of optics with cost, reach, power, density, performance and
packaging
• Move to high speed optics in the DC fabric improves application performance
• CMOS photonics enables enable new generation of SMF optics (CPAK) and
beyond
• Move to SMF backbone in DC will require low cost short reach SM optics
• Evolution of high speed optics will enable flexible bandwidth, increased system
performance, drive network architecture and facilities consideration
• Cisco continues to invest and lead the industry in optics innovation to drive to
lower cost structure.
Summary
• New Ethernet rates combined with line side DWDN advances will drive
greater efficiency
• Higher cost sensitivity & shorter investment cycles at the edge
• Backward compatibility will de-couple investment cycles for server/ ToR &
aggregation switching
• Flex Spectrum and Flex Modulation are a good start towards flexibility of
trading off bandwidth vs. distance
• Hybrid Modulation plus Flex Ethernet will allow operators to squeeze every bit
of bandwidth out of their networks.
Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you
Reference / Backup Slides
For your
reference
ITU-T Superchannels
WDM channels with a granularity of 12.5 GHz, and

Ability to define an aggregate superchannel spectral width of N × 12.5 GHz
Accommodates any combination of optical carriers, modulations, and data rate.

BRKOPT-2116 - High Speed Optics For DC Fabric & Optical Transport - Las Vegas - 2017

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKOPT-2116 - High Speed Optics For DC Fabric & Optical Transport - Las Vegas - 2017

Uploaded by

Copyright:

Available Formats

High Speed Optics for

Data Center Fabric &

Cisco Spark spaces will be cs.co/clus17/#BRKOPT-2116

IEEE defined Ethernet Regen

DC & Client Optics Line Optics

9% between data centers for replication & inter-

73% will be coming from within the datacenter,

1 zetabyte is trillion gigabytes

25Gb/s, 50Gb/s, 200Gb/s, 400Gb/s

Development and deployment at

6 Ethernet speeds in 5 years, same

When market timing differences

When market timing overlaps,

All markets benefit from economy of

Enterprise Edge Routing 1G → 10G 10G → 100G 10G → 100G

SP/Cloud Core Routing 10G → 40/100G 10G → 100G 100G→ 200/400G

• SFP28 is the 25G form factor

• QSFP-DD emerging as the 400G form factor

• Advances in high order modulation techniques –

Multiple Spines 100G Links

Wider Spine 10 - 25G Servers

Higher Scale Compute

40G-FR 2km, Optical

LRM 220m 40G-SR4 100m OM3

So what was Industry’s response?

Shorter link distances with higher data

IEEE compliant 100G-CR4 Multiple Mode Fiber, Parallel

100G ER4L Duplex Fiber LC, SMF

PARALLEL FIBER MULTIMODE OPTICS 100G 40G

DUPLEX FIBER SINGLEMODE OPTICS

DUPLEX FIBER SINGLEMODE OPTICS

DUPLEX FIBER SINGLEMODE OPTICS

100G Copper Cables 100G Copper Breakout Cables

• AOC cables are thinner than twinax Copper cables

Data Center Cabling Architecture Considerations

MDA: Main Distribution Area

Need to optimize all

Design Choices will vary by

Never ending balancing act!

Backplanes 100GBASE-KR4 4 x 25 Gb/s (NRZ) Deployed. Legacy 4x installed

Chip-to-Module CDAUI-8 8 x 50 Gb/s PAM-4 IEEE P802.3bs in Task Force

SMF Optical 400GBASE-FR8 8 x 50 Gb/s PAM-4 IEEE P802.3bs in Task Force

• Lessens many transmission penalties

Since 10 GbE, Ethernet

400 GbE 400G QSFP56-DD Form factor evolution

50 GbE 50G SFP56

40 GbE 40G QSFP+

25 GbE 25G SFP28

10 GbE 10G SFP+

10 Gb/s 25 Gb/s 50 Gb/s

4 x 10G TX 1 x 50G w/ Toolkit:

2 Fiber Strands 8 - 12 Fiber Strands

40G BiDi 100G BiDi

Optical Mux Optical Mux

100G SM-SR 100G CWDM4

Optical Mux Optical Mux

10km 25km/ 40km

Optical DMux Optical DMux

100G LR4 100G ER4L

• Optical devices can be made cheaply

Silicon Photonics is the most promising optical technology for:

Improved thermals supports

70.00 mm Essentially the same as QSFP