All Clouds Are Made of Metal BAV April 2432014 For Distribution

Ultimately All Clouds Are
Made of Metal
Harry@Quackenboss.com
HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 1

Why the Title?

Dave Roberts Tweet
“But all clouds are

ultimately made of
metal.”
Dave Roberts
Senior Vice President, ServiceMesh
Tweet, August 2, 2012
Most Everybody Thinks SDN
Is the Solution
SDN
Datacenter Mobile Network

Opinion (I)
The more you
understand the metal,
the more you will
understand SDN, NFV
and Network
Virtualization
Opinion (2)
WAN SDN ≠ LAN SDN
WAN Virtualization ≠ LAN Virtualization
Differences come from:

Physics
Economics
Opinion (3)
Mobile WAN ≠ Wired WAN

Physics & Econ Differences
WAN LAN
LATENCY ∞ 0
BANDWIDTH 0 ∞
COST ∞
HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com
0
4/23/2014 8
Do the same SDN
solutions apply?
Campus Wi-Fi
Mobile
Networks
Datacenter
LAN
Broadband and
Long haul Networks

WAN Providers’ Challenge
“A common refrain among the network
executives we’ve talked to is that, if IT
services can be provided in seconds
with cloud computing technologies, it
should not take days or weeks to
deliver the network services that
support them.”
Practical Implementation of SDN & NFV in the WAN

Heavy Reading
October 2013

London Mobile Infrastructure
600 Base Stations

See the Cell Tower?

Mobile Operators’ Finance Problem

3G Mobile Network Metal
Big device population
Complex
Heterogeneous
10 year depreciation

3G May Be the Future of IoT

Mobile SDN/NFV Perspective
Been “SDN” since before SDN
Salivating over IoT
NFV Mandatory Requirements
▪ Existing OSS/BSS integration
▪ Existing equipment interoperability
• Including legacy 3G

“Wired” WAN

Sprint Long Haul Network

Latency and Light Speed
300,000 km/sec (in vacuum)
~200,000 km/sec (in fiber)
Slower because of refraction
▪ Ask your neighborhood physicist
http://en.wikipedia.org/wiki/Optical_fiber

Cross-Country Latency
(4,100/200,000) =~.020 seconds

Source: Comcast
Packets go through routers, which buffer
adding delay
WAN Latency Is Speed-Independent
BANDWIDTH LATENCY
40Gbps 20 milliseconds
400Gbps 20 milliseconds
1Mbps 20 milliseconds
Source: Comcast.com 40G/sec network

Ping (Half Moon Bay to Palo Alto)

Half Moon Bay to New York

Half Moon Bay to Beijing

Half Moon Bay to New Delhi

Latency Vs. Bandwidth
Thought experiment:
▪ Transfer 1 Petabyte of data on 1TB
SATA drives from SFO to New
Delhi
• Note:SATA III throughput: 6 Gbits/sec
• SATA drive: ~100 Mbytes/sec (~1.1Gbps)
Alternatives
A. Lease dedicated 40G circuit
B. Airplane 1000 drives (~2000 lbs)

Latency Vs. Bandwidth
A: 40G pipe
▪ 1-way latency: ~100ms (best case)
• (first bit in to first bit out)
▪ ~70 hours
• Hypothetical ideal throughput
Option B: Airplane
▪ ~18 hours (assuming non-stop)
▪ 1-way latency: ~18 hours

WAN Cable Costs
Residential broadband installation
▪ CAPEX: ~$1,000/home passed
▪ OPEX: ~$5/month/home
http://fastnetnews.com/fiber-news/175-d/4835-fiber-economics-quick-and-dirtynth
 Long distance fiber installation

▪ $30K/mile for conduit in open
trench during road construction
(< 1% of cost of road)
Transportation Industry Lessons
Utilizationis key to low cost
Synchronize
Schedule in advance
Take time to pack (doesn’t
hurt overall time much)
Load big stuff first
Make sure there is a full
buffer at on ramp
Make sure there isn’t a
bottleneck unloading
Datacenter LANs
Samsung Backup
Datacenter Fire
April 20,2014

Mice + Elephants
Source: Facebook OIC 2013 presentation

East-West Traffic Example
Source: Facebook OIC 2013 presentation

Typical Cloud Data Center
20 to 30 1U(1.75”) servers/rack
2x1U Top of Rack switches
Switches are 5% to 15% of rack
space, CAPEX and electricity
BUT: about 3 switch ports per server
(15% to 45% of server CAPEX)

Datacenter Topology
(HVQ Note: For each TOR Switch server port, there will be ~1 to 2
aggregation ports)
Source: Nathan Farrington Facebook OIC13 Presentation

TOR (Top of Rack) Switch
▪ 48x10GE
+ 4x40GE
Vol. price (est.) Example OCP
▪ ~$2500-$4000
Bare Metal Switch
▪ Per port $50+
Source: Opencomute.org Intel draft

spec

Cabling Cost > TOR Cost
Copper Twinax: ~$75/link
▪ Intra-rack (server to TOR)
Fiber optics: ~$150 - $400
▪ (2Xoptics + cable)
▪ Rack to spine switch

Aggregation Cabling
MTP/MPO multifibre cables
▪ 12,24…up to72 fibers
Intel/Corning Silicon Photonics (future)

 Up to 64 fibers - 25Gbps each
http://www.intel.com/content/www/us/en/research/intel-labs-silicon-photonics-mxc-connector.html
Data Center Metal Economics

Datacenter Metal Key Points
Server networking switches CAPEX is
20% to 30% of server cost
 TOR switch hardware is pretty much
same for everybody
Weakening perception that brand name
SW is better is reducing margins
In-rack connectivity with copper cables
is cheap, but still as much or more than
the switch $/port
Limited opportunities to reduce
hardware CAPEX
Bandwidth Vs CPU
1 instruction needs 1 byte of
memory and generates 1 bit/sec of
bandwidth
(1994) SPARC10: 60Mhz+100Mbps
(2000) 1Ghz CPU+1GbE
(2014) (3Ghz*15 cores) * 2 sockets
▪ 90Gbps (!)
Do datacenters have
enough NIC bandwidth?
Performance in Datacenter LANs
Latency and Bandwidth

Server Latency (One Way)
Server With GbE
TCP/IP Note: numbers may vary based
Stack on specific benchmarks
Server With GbE

~15µs TCP/IP
Stack
~5µs
GBE NIC
10GE NIC
Switch Latency Is Less Than
Host Stack Latency
Data Link IP Data Link
PHY PHY
Firstbit in to first bit out

Measurement favors “cut-through”
(versus “store and forward” switches
Typical 10GE (per-chip) .3µs to 1.5µs

Switch Latency With Buffering
Data Link IP Data Link
PHY PHY
BUFFER
Switchbuffers if output port busy

Dramatic increase in latency
60µs observed in benchmark tests
Dropped packets/retransmits: ∞ latency
Round-trip user space to
user space
Source: HP Mellanox Low Latency Benchmark Report 2012

Who Cares About LAN Latency?
Not if traversing the WAN
But high latency and dropped hurt
server throughput
▪ Applications spin or get swapped
out
▪ HFT cares!

HFT: Milliseconds = $$$
 While You Were Blinking
High-frequency-trading activity
is not constant; it occurs in
microbursts. The line at the
bottom of this graphic is the
stock-market activity involving
General Electric shares over
100 milliseconds (one-tenth
of a second) at 12:44 p.m. on
Dec. 19, 2013. The gray box
magnifies a five-millisecond
window, during which GE
experienced heavy bid and
offer activity and a total of 44
trades.
44 trades in 5 milliseconds
Graphic: CLEVERºFRANKE.
Data source: IEX.
http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-lewis.html?_r=0
Leveling the HFT Playing Field
 To allow customers to
pick and choose
different pricing tiers
based on different
latencies, HFT systems
add latency using fiber
 38 miles of fiber cable =
~260 microseconds
http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-lewis.html?_r=0
Latency Key Points
From viewpoint of end-end across WAN, LAN
latency is near 0
10GE Host NIC and IP Stacks have improved
dramatically
1GE is > 10X higher than 10GE
▪ Few 1GE switches support cut through
▪ Higher chance of busy ports
Switch latency in cut through is great
If switches have to buffer because of busy
output port, latency goes way up
Latency spikes are short duration so hard to
see, let alone measure

OpenFlow For Datacenter LANs
Note: The control plane

network switches are
not shown in this
diagram

OpenFlow Reaction Time in LAN
OpenFlow Controller OpenFlow Switch
App + IP Stack Agent + IP Stack
1000µs OpenFlow OpenFlow 1000µs

Application Agent
Transport Layer Transport Layer
Internetwork Layer Internetwork Layer

30µs 30µs
NIC NIC
20-100µs

OpenFlow MicroControllers?
Master
OpenFlow
OpenFlow OpenFlow
MicroController
MicroController
Controller
http://www.theregister.co.uk/2012/12/11/pica8_sdn_bundles/
Lessons From Transportation

Multi-pathing
HOV lanes
▪ Priority based on
policy
▪ Inefficient utilization
What if nobody
changed lanes?
▪ (802.11ag and ECMP
are static)
Dynamic Load Balancing Can Increase
Throughput

“It's tough to make predictions,
especially about the future.”
― Yogi Berra

Lower Performance =
Higher Efficiency
HV Quackenboss 2014 http://www.pcper.com/news/General-Tech/Atom-aint-dead-yet-New-ultra-low-power-Avoton-chips-servers

All Rights Reserved harry@quackenboss.com 4/23/2014 57
New Rack Designs
Since the 1U server was invented by
Cobalt Networks (acquired by Sun)
the processors are > 100X faster
Network switching has
(mostly)lagged Moore’s Law
Will ratio of servers to network
switches decreases over time?
Time for new form factors?

TOR Switch Obselescence
Source: Intel IDF 14 Shenzhen presentation

Data Center Server Future

TOR Switch Evolves to Tray Switch

Conclusions

WAN Issues and Opportunities
Mobile: physical infrastructure
complexity has led to management
complexity
▪ NFV promise is to reducing
physical devices, simplify mgt.
Long haul: utilization of capacity is
still opportunity, SDN offers ways to
simplify mgt. and improve utilization

Datacenter LANs
Low cost of switching has limited
opportunities for independent SDN
controller sales
Distributed SDN MicroController
hierarchy should broaden appeal
Controller needs to be bundled with
switch or switch software
Effective dynamic load balancing will
improve throughput and latency
Important Differences
WAN LAN
COST ∞ 0
BANDWIDTH 0 ∞
LATENCY ∞ 0

Ultimately All Clouds Are
Made of Metal
Harry@Quackenboss.com

All Clouds Are Made of Metal BAV April 2432014 For Distribution

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

All Clouds Are Made of Metal BAV April 2432014 For Distribution

Uploaded by

Copyright:

Available Formats

Ultimately All Clouds Are

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 1

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 2

“But all clouds are

Datacenter Mobile Network

WAN SDN ≠ LAN SDN

WAN Virtualization ≠ LAN Virtualization

Differences come from:

Mobile WAN ≠ Wired WAN

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 7

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 9

Practical Implementation of SDN & NFV in the WAN

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 10

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 11

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 12

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 13

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 14

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 15

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 16

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 17

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 18

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 19

(4,100/200,000) =~.020 seconds

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 21

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 22

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 23

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 24

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 25

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 26

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 27

 Long distance fiber installation

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 30

Source: Facebook OIC 2013 presentation

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 31

Source: Facebook OIC 2013 presentation

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 32

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 33

Source: Nathan Farrington Facebook OIC13 Presentation

▪ Per port $50+

Source: Opencomute.org Intel draft

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 35

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 36

Intel/Corning Silicon Photonics (future)

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 38

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 41

Server With GbE

Firstbit in to first bit out

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 43

Switchbuffers if output port busy

Source: HP Mellanox Low Latency Benchmark Report 2012

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 45

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 46

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 49

Note: The control plane

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 50

1000µs OpenFlow OpenFlow 1000µs

Internetwork Layer Internetwork Layer

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 51

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 53

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 55

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 56

HV Quackenboss 2014 http://www.pcper.com/news/General-Tech/Atom-aint-dead-yet-New-ultra-low-power-Avoton-chips-servers

HV Quackenboss 2014 All Rights Reserved harry@quackenboss.com 4/23/2014 58

Source: Intel IDF 14 Shenzhen presentation