INTRODUCTION

400 GIGABIT ETHERNET

John D’Ambrosia, Huawei

Chairman, Ethernet Alliance

March 14, 2018

1
www.ethernetalliance.org
© 2018 Ethernet Alliance
Regarding the Views Expressed
The views I am expressing on IEEE standards and related
products should NOT be considered the position,
explanation, or interpretation of the Ethernet Alliance.

Per IEEE-SA Standards Board Bylaws, Dec 2016

“At lectures, symposia, seminars, or educational courses,
an individual presenting information on IEEE standards
shall make it clear that his or her views should be
considered the personal views of that individual rather
than the formal position of IEEE. ”

2
© 2018 Ethernet Alliance
This Chart Helped Launch 400GbE
• Diverse
applications!
• Diverse bandwidth
growth rates!

Source: http://www.ieee802.org/3/ad_hoc/bwa/BWA_Report.pdf

3
© 2018 Ethernet Alliance
 4
© 2018 Ethernet Alliance
Families of Ethernet Lane Rates

5
© 2018 Ethernet Alliance
How Many Lanes?
• 400GbE has used 4, 8
and 16 lanes for various
links
• 100GbE electrical lanes
being developed now
to enable 4 x100 Gb/s
for 400GbE

6
© 2018 Ethernet Alliance
The 400 GbE Optical Family
Port Type Reach Description
16 x 25 Gb/s NRZ
400GBASE-SR16 At least 100m
Use of 16 parallel multi-mode fiber
4 x 100 Gb/s PAM4
400GBASE-DR4 At least 500m SMF
Use of 4 parallel single mode fibers
8 x 50 Gb/s PAM4
400GBASE-FR8 At least 2km SMF
Use of duplex single mode fiber
8 x 50 Gb/s PAM4
400GBASE-LR8 At least 10km SMF
Use of duplex single mode fiber

7
© 2018 Ethernet Alliance
400 Gigabit Ethernet Nomenclature

8
© 2018 Ethernet Alliance
400 Gigabit Ethernet Form Factors

CFP2

9
© 2018 Ethernet Alliance
The Importance of Multi-vendor Interoperability

• 400 GbE Standard ratified Dec 2017

• Ethernet Alliance 400 GbE Demonstrations
– OFC 2017
– SuperComputing 2017
– OFC 2018 (Booth #2648)
• Ethernet Alliance 400GbE Plugfests in
planning stages!

10
© 2018 Ethernet Alliance
100 Gb/s Lambda MSA
Enabling low-cost 100 GbE and 400 GbE solutions

Mark Nowell
Distinguished Engineer, Data Center Switching
Co-chair 100G Lambda MSA
March 2018, OFC
Technology Transitioning to 400 GbE
400 GbE Dominated
400 GbE
50G -> 100G ASIC SERDES

100 GbE Dominated

100 GbE

25G ASIC SERDES

10 GbE Dominated
10 GbE

10G ASIC SERDES

2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
© 2018 Cisco and/or its affiliates. All rights reserved.
Source: Dell’Oro
Market Transition Barriers

Optics cost is becoming a barrier to transition

Reduction of ASIC cost-per-bit is outpacing that of optics

Critical to upcoming 400 GbE transition that optics cost reductions are
accelerated
© 2018 Cisco and/or its affiliates. All rights reserved.
Market interest in 100 Gb/s optical modulation
Motivated by reducing the manufacturing complexity due to reduced
component count.
Initially specified in IEEE 802.3bs (400 Gb/s Ethernet) for:
• 400GBASE-DR4: 500m over parallel SMF

MSA formed to broaden application of technology

• concern over cost reduction opportunities for “existing” 400 GbE
solutions based on 8 optical lanes
• Ongoing cost pressures for 100 GbE optics
© 2018 Cisco and/or its affiliates. All rights reserved.
 100 Gb/s Lambda MSA
• Initiated in Sept 2017

• 22 initial member companies

• Membership open and growing
• Now 27 members

© 2018 Cisco and/or its affiliates. All rights reserved.

www.100GLambda.com
100 Gb/s Lambda MSA scope & progress

Phase 1

100G-FR 2 km D1.0 Published publicly

100G-LR 10 km Internal updates in progress**

400G-FR4 2 km Target public D2.0 May/June

Phase 2

400G-LR4 10 km Work about to initiate

** Strong member desire to maintain methodology alignment with IEEE

802.3cd

© 2018 Cisco and/or its affiliates. All rights reserved.

Implementation Cost Considerations
400 GbE SMF PAM4 Module internals
Mux/demux
(if nessecary)
Ser FEC EQ 4 or 8 x DAC Driver Mod Laser
4 or 8 4 or 8 4 or 8

400GAUI-8

4 or 8 4 or 8
Ser FEC EQ
4 or 8 x ADC TIA Det

400 GbE: Half the lane count

cost, yield, power, scalability
100 GbE: Quarter the lane count
(vs. CWDM4/LR4)
© 2018 Cisco and/or its affiliates. All rights reserved.
100 Gb/s Lambda MSA Next Steps
• Products are hitting
the labs right now!!
• Validation of
specifications
• Possible interop
testing
• Phase 2 work on 400G-LR4 specification
• Discussion around proposing IEEE project initiation with same
PMD objectives
© 2018 Cisco and/or its affiliates. All rights reserved.
Questions?

© 2018 Cisco and/or its affiliates. All rights reserved.

Click to edit
400Gbps FormMaster
Factorstitle
For Input/Output (I/O)
style
Connectivity
Click to edit Master subtitle style
Nathan Tracy
3/14/18
400G Form Factors Form Factor Electrical Interface
CDFP 16x25G
CFP8 16x25G
COBO 8x50G
OSFP 8x50G
QSFP-DD Type 1 8x50G
QSFP-DD Type 2 8x50G

This discussion will focus on the

8x50G form factors
Agenda
▪ Challenges
▪ Next Gen I/O
▪ Equipment Impact
o Density
o Electrical Performance
o PCB Issues
o Reach
o Thermal Management
o Air Flow
▪ Summary

3
What’s a Port? Key Equipment Considerations
▪ I/O ports are valued for their flexibility
▪ Consist of connectors and cages that
accept pluggable modules
o Passive direct attach copper cable
o Short reach optical modules
o Medium reach optical modules
o Long reach optical modules

▪ Allows end users to flexibly choose the

appropriate reach and cost solution
▪ Provide good signal integrity
▪ Optimize thermal dissipation from the optics
▪ Different channel counts
▪ Port selection determines aggregate bandwidth and granular bandwidth

4
COBO Form Factor
▪ Embedded optics solution
▪ Eight channel and sixteen channel versions
operate with 50Gbps electrical interfaces to
provide 1x400 and 2x400G versions
▪ Three sizes of both versions on common
footprint
▪ Allows at least 32 400G modules to be
distributed around the host PCB instead of
crowded at the faceplate
▪ Opens the face plate airflow and allows full
height heat sinks providing best thermal
capability At least 32 ports per 1RU

▪ Electrical connector validated for 100Gbps

operation
5
OSFP Form Factor
▪ OSFP is an eight channel port that Integrated heat sink
accommodates up to 36 modules in 1RU
▪ It achieves density by using a 0.6mm
connector contact pitch (vs. today’s
typical 0.8mm contact pitch)
▪ Like microQSFP, it implements a module
integrated heat sink to achieve higher
levels of power dissipation
▪ Can provide backward compatibility to
QSFP modules with the use of an adapter QSFP to OSFP
▪ Electrical connector has been validated to adapter

support 100Gbps operation

Up to 36 ports per 1RU

6
QSFP-DD Form Factor Riding heat sink

▪ QSFP-DD is a new form factor port that

enables backwards compatibility for
existing QSFP modules
▪ Because of the backwards compatibility, it
keeps the connector contacts on 0.8 mm
pitch and adds additional rows of
recessed contacts
▪ It uses the traditional riding heat sink
thermal management methodology
▪ QSFP-DD allows an extra 15mm of
module length outside the faceplate Up to 36 ports per 1RU
▪ QSFP-DD can support 36 modules in 1RU,
same as QSFP

7
 Signal Integrity
Equipment Impact
PCB layout

Density and Airflow

QSFP-DD, 35% Open

OSFP, 25% Open

COBO, 47% Open

Thermal Capacity

Thermal Management Connector Structure

COBO

OSFP

QSFP-DD

8
 Summary
Signal Thermal Larger Channel Backwards
Integrity mgmt Wire Density Compatibility
AWG

COBO
Result #2 for RL Up to 26AWG At least New form
and 20W per fits 32 factor, no
crosstalk module modules backwards
compatibility

OSFP
Result #1 for RL At least 26AWG Up to 36 QSFP with
and 15W per fits modules adapter
crosstalk module

QSFP-DD
Result #3 for RL 15W 26AWG Up to 36 Directly
and may be is modules accepts
crosstalk possible difficult legacy QSFP COBO OSFP QSFP-DD

9
400G Standards Update:
What Is on the Horizon?
Rob Stone

OFC
1 2018
| OFC- 400G
2018
Standards
400G Standards
Update:Update:
What Is on
What
theIsHorizon?
on the Horizon?
Package Design Considerations for Switches

LGA Larger Packages

Higher package loss
Additional Cost

 25G / Lane Products

Mature technology
Lowest cost packaging
BGA
50G / Lane Products

2 | OFC 2018 400G Standards Update: What Is on the Horizon?

Increasing Switch Bandwidth Forces Higher Lane Speeds

70mm package approx. limit @ 50G / Lane

50G / Lane

25G / Lane 100G IO

10G / Lane

3 | OFC 2018 400G Standards Update: What Is on the Horizon?

Port Speed vs ASIC Bandwidth

Total Number of Ports Possible

Total ASIC
@ 100GE @ 200GE @ 400GE
IO (Tb/s)
3.2 32 16 8
4.8 48 24 12
6.4 64 32 16
12.8 128 64 32

• For Leaf / Spine Switch Applications, market demands highest number of ports (largest radix)
– minimize number of switch stages in the fabric, enables an efficient design
• Current technology offers 32 ports at 400GE, 64 ports at 200GE, or 128 ports at 100GE
• Will future switches require higher port speeds > 400GE?
– Economics of higher rate optical interconnects needs to scale favorably (fiber plant + transceivers)

4 | OFC 2018 400G Standards Update: What Is on the Horizon?

 Summary

• Switch IO is being driven to higher per lane speed as total bandwidths increase
– Due to package size limitations
• Standardization activities to support these new electrical interface speeds
– 25 & 50G / lane electrical IO DONE
– 100G / lane electrical IO: Chip to Chip, Chip to Module, Backplanes, Copper Cables – IN PROCESS
• Challenges ahead:
– Continue to scale $/Gb/s and pJ/bit of optical interconnects to enable cost effective, high bandwidth
large radix switch systems

5 | OFC 2018 400G Standards Update: What Is on the Horizon?

Thank You

OFC
6 2018
| OFC- 400G
2018
Standards
400G Standards
Update:Update:
What Is on
What
theIsHorizon?
on the Horizon?
Recent Activities at TIA

Rakesh SAMBARAJU
Nexans Data Center Solutions
OFC 2018: 400G Standards, MSAs and Related Technologies: What is on the Horizon?
 Introduction to Nexans
Worldwide leader in the cable industry
-Started 120 years ago with electrical wiring
-Became part of Alcatel in 1991
-Became Nexans in 2000, listed on the European Stock
Exchange since 2001
-Today – have an industrial presence in 40 countries with
commercial activities worldwide, close to 26,000
employees, and sales of 7.8 billion (USD) in 2017

2 I OFC 2018, San Diego, CA. 03/13/2018

 Path to 400G

Parallel Transmission
-16F MPO
-32F MPO

Fiber
Capacity
-Performance guidelines for -New MMF – OM5
cable/connectivity -Study WDM over OM3/OM4
-Study VCSEL performance for DMD
measurements

3 I OFC 2018, San Diego, CA. 03/13/2018

 New Fiber Type – MMF MUX DeMux

 TIA-492AAAE standard for WDM capable MMF WDM

 OM5 is designed to supports 4 channel WDM (850 – 950 nm) over MMF
 Cable jacket, connector boot, adapters color – Lime Green

 TIA considering studying wideband performance of OM3/OM4

 Includes studying the VCSEL weighting functions at higher wavelengths
 Will provide for better system modeling to IEEE
 Q: Can WDM performance on existing OM3/OM4 be ‘guaranteed?’

Parameter OM3 (TIA 492AAAC) OM4 (TIA 492AAAD) OM5 (TIA 492AAAE)

EMB@ 850 nm 2000 MHz.km 4700 MHz.km 4700 MHz.km

EMB@ 953 nm Not specified Not specified 2470 MHz.km

4 I OFC 2018, San Diego, CA. 03/13/2018

 New Connector Types – MPO
 TIA 604-18/FOCIS-18: one/two row16F MPO Standard
 MPO-16
 Can support 8Tx/8Rx lanes
 Part of QSFP-DD and OSFP MSA
 Potential connector for 400G-SR8

 MPO-32
 2 rows of 16 fibers
 MDI for 400G-SR16; part of CDFP MSA

5 I OFC 2018, San Diego, CA. 03/13/2018

 Cable/Connectivity
 Cable:
 Reduce cable attenuation for MMF to 3.0 dB/km (previously 3.5 dB/km)
 Recommends armored/thick jacketed cables to avoid microbends in non-continuous cable
supports

 Connectivity:
 Increase SM Splice return loss requirement to 35 dB (previously 26 dB)
 Specified break-out configurations to accommodate 2-row connectors
 Recommends pre-term cabling for quality consistence

 Q: Address requirements for RL on SM MPO for PAM-4 systems?

6 I OFC 2018, San Diego, CA. 03/13/2018

 Summary
 Standardized new MMF called OM5
 Will enable consistent WDM over MMF

 New 16 and 32 Fiber MPO

 Connectors for upcoming 400G PMDs
 Already part of 8x and 16x form factors

 Tightened the specs for cabling/connectivity

 Will help with high baud-rate/multi-level signaling

 Need to address SM connectivity reflectance for PAM-4?

7 I OFC 2018, San Diego, CA. 03/13/2018