BRKDCT 2218

BRKDCT-2218
Data Center Network Design

for Midsize Enterprises
Fausto Vaninetti, Data Center

EMEAR DAT Consultant Systems Engineer
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKDCT-2218
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
…until you get to something
you’re happy with
Start small
… then grow
EEE education
Hint: leverage yet expand your skill set (even in IT)

BRKDCT-2218
exposure experience
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Agenda
• Introduction
• Beyond STP, from Networks to Fabrics
• Scaling Brownfield and Greenfield Data Center Networks
• Data Center Interconnect Solutions
• Feature-Specific Considerations
• Programmability, Automation & Orchestration
LOTS of Related Sessions, My Recommended List
Session ID Title Presenter Date / Time
BRKACI-2125 ACI Multi-Site Architecture and Deployment Max Ardica Check on CiscoLive App
BRKDCT-2949 Building Data Center networks with VXLAN BGP- Lukas Krattiger Check on CiscoLive App
EVPN Part I
BRKDCT-3378 Building Data Center networks with VXLAN BGP- Lukas Krattiger Check on CiscoLive App
EVPN Part II
BRKACI-2003 Cisco ACI MultiPod Design and Deployment John Weston Check on CiscoLive App
BRKARC-3222 Cisco Nexus 9000 Architecture Tim Stevenson Check on CiscoLive App
BRKDEV-1368 Effectively Understand and Leverage YANG with Hank Preston Check on CiscoLive App
NETCONF and RESTCONF for Model Driven
Programmability
BRKACI-2508 Nexus standalone fabrics migration to ACI Ricardo Trentin Check on CiscoLive App
BRKDCN-2657 Design and Implementation of DCI Networks Yves Louis Check on CiscoLive App
BRKDCT-2218 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Agenda
• Introduction
Some European Statistics
Out of 21M enterprises
Midsize enterprise:
Micro Enterprises
• 50 < employees < 250 Small Enterprises
Medium Enterpries
• € 10M < Revenue < € 50M Large Enterprises
http://ec.europa.eu/index_en.htm
Depends on Geography
Designing Data Center Networks for Midsize Enterprises
Defining “Midsize”
Client Access
 Require dedicated DC Switches, moving WAN / DCI
away from collapsed core with campus
Campus
 Mostly virtualized, some physical servers

Scalability
L3
-----------
 Size for current needs, reuse components L2
in larger designs FC
Design Options FCoE

iSCSI / NAS
 Feature choice + priority = tradeoffs

 No such a things as the perfect recipe’
 Where the industry is going:
 Programmability, Automation (SDN buzz)
What are you ready for?
Decisions will depend on where you draw the line:

• Want to stay with existing toolsets for config & management?
• Capable of consuming a new set of tools in the hope of some efficiency?
• New or traditional operational model?
• Resist/accept pressure from other teams or business owners?
Design Goals
Flexible
Reliable
Easy
Network Topologies
Ring Star Fully Connected Bus
Tree N-Tiered Spine Leaf
Which network model would YOU choose?
Example: 350 VM’s; 25 Baremetal Servers @ 1/10GE; HA; IP Based Storage
VXLAN
L3
L3
… …
96 front L2 96 front L2
facing ports facing ports
Collapsed/Single Tier Spine Leaf

1RU
Downlink ports: 48p Uplink/Breakout ports: 6-12p
Single-Tier, Dual-Tier, Spine/Leaf
Scalable Spine/Leaf DC Fabric
VXLAN
Dual Tier DC
Scale by Tiering
VXLAN
Single Layer DC
Scale Out
Small Spine/Leaf
Connectivity & Features Drive Design Choices
 Computing Form Factor  Connectivity Model
– Unified Computing Fabric – 25 or 10 or 1-GigE Server ports
– 3rd Party Blade Servers – NIC/HBA Interfaces per-server
– Rack Servers (Non-UCS Managed) – Copper vs Fiber cabling to ToR
 Storage & Storage Protocols  Virtual Networking Requirements

– Native Fibre Channel – vSwitch/DVS/OVS/Nexus1Kv/AVS
– Unified Ports, FCoE
 Programmabiltiy/Automation/Orchestration
– IP-based storage (iSCSI, NAS)
– Complete abstraction
– Device or fabric level programmability
iSCSI NFS/
FCoE FC CIFS
VM VM VM VM VM VM
Data Center Network Needs
• Server virtualization is here
• Higher P2V ratio, access port bandwidth up
Internet
• Oversubscription ratio from 20:1 to 1:1 Public Offsite DC
Cloud Enterprise
• Horizontal workload mobility Site B
NORTH - SOUTH TRAFFIC

Mobile Network
• Any VLAN everywhere, no physical boundaries
FC
• Highly distributed applications DATA FCoE

iSCSI / NAS
API CENTER
• Hadoop and microservices NETWORK Storage
• More server to server communication Orchestration/
Monitoring
• 80/20 traffic pattern reversed?
• Increasing high availability requirements Server/Compute Services
• Where did my maintenance window go? EAST – WEST TRAFFIC
• More execution speed, less human errors

From Networks To Fabrics
Definition: ensemble of switches that behave
and get configured like a single giant switch
Internet Offsite DC
Public
• Flexibility: allows workload mobility, VLANs everywhere Cloud Enterprise
Site B
NORTH - SOUTH TRAFFIC

Mobile Network
• Robustness: reduce L2 failure domains, L2/L3
boundary on leafs, anycast gateway FC
DATA FCoE
• (Virtual) Network services: moving out to border API CENTER
iSCSI / NAS
leafs, policy-based service chaining FABRIC Storage

Orchestration/
• Performance: full cross sectional bandwidth Monitoring
(any-to-any) with ECMP, avoid oversubscription
• Latency: deterministic at scale, single hop away Server/Compute Services
• Scalability: add end nodes, maintain oversubscription EAST – WEST TRAFFIC
• Cost: fixed switches vs modular switches
Network Implications of NVM/NVMe/NVMe-oF
NVM storage as a key force driving the transition to higher bit rates
• Rule of thumb: 1M IOPS need 40G of bandwidth

• 4x 10Gbps (or equivalent FC bandwidth) connectivity required in 2018
• 18x 40Gbps (or equivalent FC bandwidth) in 2022
• Making leaf-spine networks a necessity? More high-speed ports on access switches?
Agenda
• Introduction
History Lesson: Spanning tree
• Spanning Tree introduced around 1985

• 32 years ago, we also saw:
• Windows 1.0
• DNS come out of academia
• First Nintendo Entertainment System
• STP prevents loops, but blocked ports
Host or
and slow re-convergence Switch
• Successfully deployed for some time…

• …but since a few years, most DC
Designs built to work around STP
Virtual Port Channel (VPC)
• VPC invented to overcome STP limitations vPC Domain
• IEEE standard in 2000 (802.3ad)

• Not perfect, but a good workaround
• STP is still there on every link
• Human error, misconfiguration, bug can Host or
still cause issues Switch
• Cisco alternatives for campus design:

VSS, StackWise Virtual
Virtual Port Channel (VPC) “Mini-Fabric”
vPC Domain 1
• VPC Northbound & Southbound
• More efficient than native STP
Dual sided vPC
• STP is still running Back-to-Back vPC
“mini-fabric”
• Another good workaround
• Configuration can become complex
as switch counts grow vPC Domain 2
• vPC makes two switches look as

one….but what about 4 switches?
Host or
Switch
Cisco Live Session: BRKDCT-2949
Building Data Center networks with VXLAN BGP-EVPN Part I
Cisco Live Session: BRKDCT-3378

L3-Based Fabrics Building Data Center networks with VXLAN BGP-EVPN Part II
• Every link forwarding

• L3 ”routing” fast convergence
• VXLAN overlay (MAC in UDP) VXLAN
• Flood and learn vs VXLAN BGP-

EVPN control plane
• STP might still exist on edges, but
not within the “fabric”
• VPC still needed at edge
Host or
• Spine/Leaf: Switch
• Flexible and efficient design
• Consistent hop count & latency
Why Spine-Leaf Design? Flexibility and Efficiency
Scale ports and bandwidth independently
BW Fabric
40G fabric ports
M fabric links per leaf
BW Edge N ports
Oversubscription at leafs if:

10G host ports
M x BW Fabric < N x BW Edge
96 ports
2x48 10G (960 Gbps total)
To speed up FCT,
Need even more
boost the backplane
host ports?
Per Spine
Utilization
by spreading load
Add another leaf
across more spines
BW Fabric
FCT = Flow
40G fabric ports Completion Time
FCT
FCT
FCT
more ports
BW Edge N ports

10G host ports
192 ports
To speed up FCT,
Need even more
Utilization
Per Spine
boost the backplane
host ports?
by spreading load
Add another leaf more
bandwidth
across more spines
reduce failure domain
BW Fabric
FCT = Flow
40G fabric ports Completion Time
FCT
FCT
FCT
more ports
BW Edge N ports

10G host ports
Lower FCT = FASTER
192 ports
APPLICATIONS
Spine/Leaf DC Fabric ≅ Large Non-Blocking Fabric
Host Host Host Host Host Host Host

1 2 3 4 5 6 7
Spine/Leaf DC Fabric ≅ Large Non-Blocking Fabric
Host
4
Host
1
Host
5
Host
2
Host
6
Host
3
Host
7
Host Host Host Host Host Host Host
1 2 3 4 5 6 7
Spine/Leaf DC Fabric ≅ Large Modular Switch
Card
Card
Line
Line
Host
4
Module
Host
Fabric
1
Card
Line
Card
Line
Host
5
Host
CLOS
Module
2
Fabric
Card
Line
Card
Fabric
Line
Host
6
Module
Card
Fabric
Line
Card
Line
Host
3
Card
Line
Card
Line
Host
7
Impact of Link Speed – the Drive Past 10G Links
20×10Gbps 5×40Gbps 2×100Gbps

Bandwidth
Aggregate
200G
Uplinks Uplinks Uplinks

Bandwidth
Aggregate
200G

Downlinks Downlinks Downlinks
• 40 & 100Gbps fabric provide very similar performance for fabric links
• 40G provides performance, link redundancy, and low cost with BiDi
Statistical Probabilities of Efficient Forwarding
Probability of 100% throughput ≅ 3%
Uplinks Uplinks Uplinks
1 2 20
1 2 3 4 5
11×10Gbps flows
(55% load) 1 2
Impact of Link Speed on Flow Completion Times
20 Avg FCT: Large (10MB,∞) background flows
FCT (normalized to optimal) 18
16
Lower
FCT is 14 Series1
Better 12
Series2
10
Series3
8
6 Series4
4
2
0
30 40 50 60 70 80
Load (%) BRKDCT-2218 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
16
Lower
FCT is 14 Series1
Better 12
Series2
10
Series3
8
6 Series4
4
• 40/100Gbps
2 fabric links: ~ same FCT as non-blocking OQ-switch
0
30 40 50 60 70 80
• 10Gbps fabric links: FCT 40% worse than 40/100Gbps
Load (%)
16
Flow Completion Time dependent on
Lower
FCT is 14 queuing and latency
Series1
Better 12
Series2
10 40G is not just about faster ports &
8 higher bandwidth, it’s about Series3
6 shorter Flow Completion Time Series4

4
• 40/100Gbps
2 fabric links: ~ same FCT as non-blocking OQ-switch
0
30 40 50 60 70 80
• 10Gbps fabric links: FCT 40% worse than 40/100Gbps
Load (%)
40G BiDi Optics Preserve Existing MM 10G Cabling
MM Fiber Plant
MMF LC
Patch cord
Used Fiber Pair
MMF LC
Patch cord
Highlights:
SFP-10G-SR SFP-10G-SR • Reuse of existing
cable plant
QSFP-40G-SR4 QSFP-40G-SR4
MM Fiber Plant • Price of 40G BiDi
Used Fiber Pair
QSFP ~ price for 10G
MPO
MPO
Used Fiber Pair

Used Fiber Pair
Used Fiber Pair
SR SFP
• WDM technology
QSFP-40G-SR-BD QSFP-40G-SR-BD over MM fibers
MM Fiber Plant
MMF LC Used Fiber Pair
MMF LC • Excellent DC reach
Patch cord Patch cord
• Can we do the same

at 100G?
Distance up to 125m with OM4 BRKDCT-2218 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Networking Underlays and Overlays
IP Fabric
Robust Underlay/Fabric Flexible Overlay Virtual Network

• High Capacity Resilient Fabric • Mobility – Track end-point attach at edges
• Intelligent Packet Handling • Scale – Reduce core state, distribute to edge
• Full Cross Sectional Bandwidth • Flexibility/Programmability – Less touch points
• Programmable & Manageable • Physical + Virtual; L2 & L3 Connectivity
Do you really want to deal with two networks? Consider Cisco ACI instead
Why VXLAN Overlays?
VXLAN provides a Network with Segmentation, IP Mobility,

and Scale
• “Standards” based Overlay (RFC 7348)
• Leverages Layer-3 ECMP underlay – all links forwarding
• Increased Name-Space to 16M identifier
• Integration of Physical and Virtual Classical Ethernet Frame DMAC SMAC 802.1Q Etype Payload CRC
50 bytes Original CE Frame
• It’s the pillar of SDN Outer

IP UDP VxLAN 802.1Q CRC
MAC DMAC SMAC Etype Payload
(20) (8) (8) optional (new)
(14)
• Works better with a control plane 8 bits 24 bits 24 bits 8 bits
ags Reserved VNIVNI Reserved
Network Centric Deployment
• Leverage known networking constructs.
• VLANs, IP address / subnets, Flood domains etc.
Existing network – Nexus, Catalyst, etc
VRF VNI: 300100 VRF VNI: 300100
MAC_A,IP_A: 10.1.1.2 MAC_B,IP_B:20.1.1.2

VNI L2: 100100 VNI L2: 100200
VLAN 100:
10.1.1.0/24 VLAN 100 VLAN 200
VLAN 200:
20.1.1.0/24 VXLAN EVPN Fabric
Existing network – Nexus, Catalyst, etc TENANT
PRIVATE NETWORK (VRF)
BD: Blue BD: Red

10.1.1.0/24 20.1.1.0/24
VLAN 100:
10.1.1.0/24 EPG: Blue-100 EPG: Red-200
VLAN 200:
20.1.1.0/24 ACI Fabric
Cisco Live Session: PSOACI-4592
Why ACI: an overview and a customer (BBVA) perspective
Application Centric Deployment (ACI only)
• Leverage grouped constructs – Network Level; Application Level

• Abstraction with Application Network Profiles, Policy oriented
• Automation of network services
• APIC Controller: declaration of intention, translate to network constructs via policy
External
Connectivity L3 Outside
APIC
EPG EPG EPG

Web App DB
Web App DB
FW / SLB FW / SLB
App Profile
Controlling, Managing & Automating the Fabric
Application Policy
Datacenter Infrastructure
Network Manager Client Access Controller Client Access
Connection
Creation Expansion
WAN / DCI WAN / DCI
Reporting Fault Mgmt

APIC APIC
APIC
Network Centric only Network Centric

Policy Centric
Spine Spine
VXLAN EVPN ACI Fabric
Leaf Leaf
Programmable Fabric Application Centric Infrastructure (ACI)

Variety of Nexus models Nexus9000 only
ACI Anywhere
Any Workload, Any Location, Any Cloud
ACI ANYWHERE
Remote PoD Multi-Pod / Multi-Site Hybrid Cloud Extension
IP IP
WAN WAN
Remote Location On Premise Public Cloud
Security Everywhere Analytics Everywhere Policy Everywhere
Agenda
• Introduction
Scaling a VPC-based DC design
Access L3
Layer L2
VLANs
100-150 Host Host Host
Consolidated
Core/Agg
Layer
Access L3 Access
Layer L2 Layer
VLANs VLANs
100-150 Host Host Host Host Host Host 151-200
Distributed or
Consolidated
Core/Agg
Layer
L3
L2
Access Access
Layer Layer
VLANs VLANs
100-150 Host Host Host Host Host Host 151-200
Cisco Live Session: BRKACI-2508
Integrating ACI with an existing network Nexus standalone fabrics migration to ACI
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/migration_guides/migrating_existing_networks_to_aci.html
Integrating ACI with an existing network
ACI Pod
New DC
Row Upgrade
New Application
Distributed or
Consolidated
Core/Agg Spine
Layer Layer
L3
ACI Fabric
L2 (VXLAN based)
Dual sided
vPC
ACI Border Leafs

Access Access Access
Layer Layer Layer
VLANs VLANs VLAN
100-150 151-200
Host 201-250
Host Host
Integrating ACI with an existing network
ACI Pod
New DC
Row Upgrade
New Application
Distributed or
Consolidated
Core/Agg Spine
Layer Layer
L3
ACI Fabric
L2 (VXLAN based)
Access ACI
Layer
VLANs
Leafs
100-150
Host and Border Leafs
Host Host
Supported from ACI
Remote Leaf Concept 3.1(1) – Dec 2017
Remote Leaves
Any router with OSPF, logically connect
APIC and Spine Nodes (Proxy) DHCP relay and 9150
MTU support.
to Spines of one
remain at primary site(s) IP WAN POD in Main DC
IPN L2 / L3
1/10G ports
Tested with 500Mbps on WAN,
but the more the better
All local traffic is switched directly between

endpoints, both virtual and bare metal
Specific traffic requires use of spine proxy.
vSwitch
Hypervisor
Supported hardware and scale
Remote site contains Nexus 9300 connected
- All hardware from -EX onwards is supported
to IP Network via uplink ports and fully
- 40 RL (20 sites, 2 RL per site)
managed by primary site APIC cluster
ACI: Small Footprint Option Under Consideration
• Full featured ACI Fabric

Physical Server
• Reduced initial port count
Spine 32 ports
Spine 32 ports
• Optimized scalability for
Leaf 48 ports VM mid size deployments
Leaf 48 ports VM
• Lower entry cost (-30%)
What is your
opinion? • Reduced footprint (-40%)
Agenda
• Introduction
Cisco Live Session: BRKDCN-2657
Design and Implementation of DCI Networks
IT Trends - Distributed Data Centers
Building the Data Center Cloud
Distributed Data Center Goals:
• Seamless workload mobility between multiple
datacenters
• Distributed applications closer to end users
• Pool and maximize global compute resources
• Ensure business continuity and disaster
avoidance with workload mobility, distributed
deployments and clustered applications
• Midsized enterprises often rely on two Geographically
datacenters Dispersed Data Centers
NX-OS Overlay Transport Virtualization Technology
Extend VLANs Across Datacenters
OTV
Classical Pod Scalable Pod Leaf Spine Pod

Spanning Tree Protocol vPC, N-tier design VXLAN, ACI
Feature: Benefits: In the news:

• Simplified Layer 2 / VLAN Extensions • Field proven, very mature • M3 linecards with NX OS 8.0
• Restricted fault domain (STP domain), • Enable L2 elasticity across DC’s • OTV Loopback Join Interface for
loop prevention Multicast-based OTV control-plane
• Simplify Virtual Machine Mobility
• Optimized Multicast replication • Wire-rate 256-bit AES MACsec on all
• Extend Layer 2 without the risks of
ports at all speeds along with OTV
• Dual homing large fault domain
• Works over dark fibre, MPLS or IP • Simple – 3 easy commands !
Client Access
OTV options for DCI WAN / DCI

Client Access Campus
WAN / DCI
Campus
L3
-----------
L2
L3
-----------
L2
Virtual DC Virtual DC
Services in Services in
Software VM VM VM VM VM VM VM VM VM VM VM VM Software
Virtualized Servers, Nexus CSR1000v Virtualized Servers, Nexus

1000v, vPath, CSR 1000v 1000v, vPath, CSR 1000v
Client Access
OTV options for DCI ASR1000

WAN / DCI
Client Access Campus
WAN / DCI
Campus
ASR1000 L3
-----------
L2
L3
-----------
L2 N7K
Physical or Physical or
Virtual Virtual
Workloads Workloads
VM VM VM VM VM VM VM VM
and Services and Services
CSR1000v
VXLAN as a Data Center Interconnect? Building
• DCI is an architectural discussion
• VXLAN is just an encapsulation technique
• OTV is a well-proven technology that gets you

fully covered
• VXLAN can absolutely fit into a DCI

architecture…if you handle it CAREFULLY
• Protection at edge as well as control plane required
• Storm Control, BPDU Guard, HMM Route Tracking
• Control-Plane with MAC-learning, ARP suppression
ACI MultiPOD Design and Deployment

DCI with ACI Multi-Fabric Options ACI Multi-Site Architecture and Deployment
Single APIC Cluster/Single Domain (BC-DA) Multiple APIC Clusters/Multiple Domains (DR)
Stretched Fabric Multi-Fabric Connected (L2 and L3 Extension)
ACI Fabric ACI Fabric 1 ACI Fabric N
Site 1 Site 2
40G
L2/L3
More scale
Multi-Pod More flexibility Multi-Site (new)
More automation
Pod ‘A’ Inter Pod Pod ‘n’
Network@10G Site ‘A’ Inter Pod Site ‘n’
Network@10G
MP-BGP - EVPN
… MP-BGP - EVPN
APIC Cluster
Agenda
• Introduction
What Features Matter The Most to You? (as of Jan 2018)
Feature 3000 3500 5500 5600/ 7000/ 9300 9500
6000 7700
Unified Ports
FCoE
FEX
VXLAN Bridging and Routing
VXLAN F&L
VXLAN BGP EVPN
DCI - OTV
DCI – VXLAN
ISSU
VDC
LISP, MPLS
ITD
RISE
Extreme low latency
IPv6 support
Automation ACI
Automation DCNM
FCoE and Unified Fabric
Fibre Channel with simpler infrastructure and lower cost
FCoE Benefits
Encapsulate Fibre Channel frames

into Lossless Ethernet • Standards based
• Operationally same as existing LAN /SAN
Ethernet Carrier, DCB
• Transparent to OS and Apps
Individual • Fewer Cables
Ethernets
• Fewer switches
Individual • Fewer adapters
Storage
(IP, Eth, FC) • Overall less power
Byte 0 Byte 2229

Ethernet
Header
Header
Header
FCoE
CRC
EOF
FCS
FC Payload
FC
Nexus Fabric Switches Evolution (FC/FCoE View)
Nexus 9300 Unified Ports
Compelling Business Value and Flexibility
 Repurposing of any switch port as either

CAPEX Fibre Channel or Ethernet/FCoE avoids
CAPEX lock-in with specific port-types
Unified
Port
 Reduces physical management points
OPEX
 Extend adoption of FCoE in the
access with associated savings FC Eth
Native Fibre Lossless

 Simplifies the purchasing decision, makes
BUSINESS port-type decisions a thing of the past Channel Ethernet:
AGILITY 16/32G 1/10/25GbE, FCoE,
 Allows IT to respond faster to changing iSCSI, NFS/SMB
business needs
Introducing 16/32G FC on Nexus 93180YC-FX
Milestone Timeline Shipping

FCoE NPV mode Oct, 2017
(NX-OS and ACI) NX OS 7.0(3)I7(1)
16G Fibre Channel Nov, 2017

NPV mode (NX-OS) NX OS 7.0(3)I7(2)
32G Fibre Channel Soon

NPV mode (NX-OS)
Unified Ports: 48p 1/10/25G or 16/32G FC SFP 6p 40/100G QSFP
16/32G Fibre Channel Work in progress
NPV mode (ACI)
First Nexus 9300 switch with Fibre Channel Capabilities

Supports NX-OS and ACI Mode.
Seamless transition Multi-protocol UNIFIED PORT Interop with Cisco Cloud Scale
from N5K FC MDS and Nexus 5K technologies
FCoE
IP Storage MACSec
EX only supports FCoE NPV

FX supports both FCoE NPV (Oct 2017) and FC NPV (Nov 2017)
Example: Small Deployment For All Flash Array
All Flash Array
NPIV
feature
32G FC storage ports
MDS 9132T
16G FC uplink ports
MDS 9132T LAN
NPV mode (32G future)
Nexus 93180YC-FX Peer links (Ethernet) Nexus 93180YC-FX
10G FCoE access ports
NX OS mode: shipping
ACI mode: work in progress
Servers
What is ITD ?
Intelligent Traffic Director
• Traffic distribution and redirection
• ASIC based solution (HW-switched)
• Caters to multi-terabit traffic
• Works on Nexus switches – 5/6/7/9k,

similar but not identical capabilities
ITD does L3-L4 traffic distribution,
does not replace Layer-7 Load-balancers
Where to use ITD ?
ITD to load-balance to the destination

#1
Example: Server-Load Balancing
Clients Servers
Health checking supported
Where to use ITD ?
ITD for In-line traffic redirection
#2 Example: Firewalls, IDS, Web Caches etc.
Possible WCCP replacement? Possible MS NLB replacement?
Destination
Clients
Firewalls/other appliances
ITD Benefits
Intelligent Traffic Director
No service-module
Line-rate
or external
traffic-distribution
appliance required
Automatic Ease of deployment,

failure handling reduced configuration
Agenda
• Introduction
Why Network Programmability?
More Speed More Repeatability More Flexibility More Innovation
Fewer Mistakes
Programmability, Automation, Orchestration
Cisco Data Center Network Automation Types
Providing Choice in Automation and Programmability
Application Centric
Programmable Network Programmable Fabric
Infrastructure
DCNM
DB DB
Web Web App Web App
Modern NX-OS with enhanced VxLAN-BGP EVPN Turnkey integrated solution with
NX-APIs standard-based security, centralized management,
compliance and scale
DevOps toolset used for Network 3rd party controller support
Management Automated application centric-policy
(Puppet, Chef, Ansible etc.) Cisco Controller for software model with embedded security
overlay provisioning and
management across N2K-N9K Broad and deep ecosystem
Tools, APIs, Controllers and Automation© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Live Session: BRKDEV-1368
Effectively Understand and Leverage YANG with NETCONF and
RESTCONF for Model Driven Programmability
Nexus Programmability Nexus 7K Nexus 5K / 6K Nexus 9K

Puppet/Chef/Ansible Shipping Shipping Shipping
Provisioning &
PoAP Shipping Shipping Shipping
Orchestration
OpenStack Shipping Shipping Shipping
NetConf (SSH/XML) Shipping Shipping Shipping

NX API CLI (JSON/XML) Shipping Shipping Shipping
Protocols and NX API REST Future No Shipping

Data Models NetConf / YANG Future No Shipping
RESTconf/gRPC Future No Shipping
Streaming Telemetry Future No Shipping
Native Python Shipping Shipping Shipping

Linux container Shipping Shipping Shipping
Programmatic
Docker containers Future No Future
Interfaces
Guest Shell Future No Shipping
OpenFlow Shipping No Shipping
Programming a Fabric
• A lot of work is being done to provide customers maximum flexibility in
programming & automation interfaces
• Free Open Programmability book:
• http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/nexus9000/sw/open_nxos/program
mability/guide/Programmability_Open_NX-OS.pdf
• New community site dedicated to NXOS programmability:
• https://opennxos.cisco.com
• A lot of work has been done to increase available knowledge on network

programming across all Cisco products
• DevNet: If you haven’t visited, please do so
• https://devnet.cisco.com
• SANDBOX! – FREE 24 X 7 hosted labs
Programmable Application Centric
Programmable Fabric
Networking Infrastructure
DCNM
DB DB
Web Web App Web App
Along the Spectrum from CLI to ACI
A New Way To Do Fabric Management
Scripting to the
CLI and/or API
Basic
Element
Manager
CLI ACI
Fabric Management Functionalities
Common Customer Asks
• Fabric management automation – high interest
• Many Fabrics are based on things like VXLAN, BGP/EVPN, IS-IS
• New protocols, new configurations, new things to learn
• Simple tool to ease burden of adoption
• Simplified interaction – GUI and API
• CVD/Best practices – done for you!
Data Center Network Manager (DCNM) 10
• Network management system tool for

• SAN management (Nexus and MDS)
• LAN (full Nexus family)
• Programmable Fabrics (NX OS mode)
• Flow controller for IP Media Networks
• Advanced feature support (STP, VXLAN, POAP,
OTV, templates, topology, configuration backup)
• Underlay/Overlay manager for VXLAN-EVPN (IP)
Fabrics in NX-OS Mode
Functions Dashboard
Health Monitoring, Configuration/Backup Automation Trend Analysis & VM

Inventory & Analytics
Diagnostics
Host / Endpoint Visualization & Alert/Notifications Storage Management Professional Media

Monitoring Troubleshooting Net Controller
Programmable Application Centric
Programmable Fabric
Networking Infrastructure
DCNM
DB DB
Web Web App Web App
FCAPS and Automation
Application Centric
Programmable Network Programmable Fabric
Infrastructure
DCNM
Turnkey integrated solution with security, centralised

Modern NX-OS with enhanced NX-APIs Integrated stack management, compliance and scale
Or
DevOps toolset used for Network Management A-la-carte Automation Automated application centric-policy model with
(Puppet, Chef, Ansible etc.) embedded security
Streamlined Workflow Management
Custom Script based Operations and Workflows
Broad and deep ecosystem
Security
External
Performance Tools
External
Accounting Tools Integrated
Integrated Tools
Fault
Tools
Configuration
Effort vs Time – Traditional IT Build/Run
Ops
Config
Ops
Config
Ops
Config
Config Ops
Effort vs Time – Traditional IT Build/Run
Ops
Config
Ops
Config
Ops
Config
Config Ops
Effort vs Time – Utilizing Abstraction & Automation
Plan Implement
Ops
Config
Ops
Config
Ops
Ops
Config
Config Ops
Cisco UCS Director for Compute; Network & Storage
Infrastructure consumption made easier
Converged Stack
Secure Cloud Control Panel
Container
OS and
Network Compute VMs Storage Virtual Bare
VM VM
Metal
Machines
Single Pane of Glass for Virtualized and Bare-Metal
Policy-Driven Virtual AND Physical

Compute
Provisioning Speed with Accuracy Compute and Hypervisor
UCS Director More efficient use of People

Network A B C
& Time
Network and Services
Consistency, Less Error in

Tenant Tenant Tenant
Repetitive Tasks Storage A B C
UCS Director – Example Use Cases for SMB
 Provision new SP from template, provision zoning/masking/boot LUN, boot from SAN,
install hypervisor (ESX image on PXE server), associate VMWare Host Profile for vSwitch
settings, more customization
---------
-
------------
---------
------------
Start ------------
------------
-
---------
---------
------
---------
---
 Create VM Disk, install and format in OS, then create windows share or linux mount point
Start
Key Takeaways
 Cisco has many options for building data
center networks, you own the key
 All solutions can start small and grow
 No Cisco solution has to be a “rip and replace”
 Spine-Leaf does not have to be expensive
 Programmable fabrics provide new tools for
simplified operations
 Automated fabrics provide new methods of
managing DC Networking
Call to Action
 Check out the sessions listed at beginning of presentation

at www.ciscolive.com
 Visit the World of Solutions and look for “Data Center”
 Meet the Engineer
 Visit the DevNet Zone and see what they have to offer!
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKDCT-2218
• Please complete your Online Complete Your Online
Session Evaluations after each
session
Session Evaluation
• Complete 4 Session Evaluations
& the Overall Conference
Evaluation (available from
Thursday) to receive your Cisco
Live T-shirt
• All surveys can be completed via
the Cisco Live Mobile App or the
Communication Stations
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
www.ciscolive.com/global/on-demand-library/.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Tech Circle
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you

BRKDCT 2218

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCT 2218

Uploaded by

Copyright:

Available Formats

BRKDCT-2218

Data Center Network Design

Fausto Vaninetti, Data Center

Hint: leverage yet expand your skill set (even in IT)

 Mostly virtualized, some physical servers

Design Options FCoE

 Feature choice + priority = tradeoffs

Decisions will depend on where you draw the line:

Ring Star Fully Connected Bus

Tree N-Tiered Spine Leaf

facing ports facing ports

Collapsed/Single Tier Spine Leaf

Downlink ports: 48p Uplink/Breakout ports: 6-12p

 Storage & Storage Protocols  Virtual Networking Requirements

NORTH - SOUTH TRAFFIC

• Highly distributed applications DATA FCoE

• Where did my maintenance window go? EAST – WEST TRAFFIC

• More execution speed, less human errors

NORTH - SOUTH TRAFFIC

leafs, policy-based service chaining FABRIC Storage

• Scalability: add end nodes, maintain oversubscription EAST – WEST TRAFFIC

• Cost: fixed switches vs modular switches

• Rule of thumb: 1M IOPS need 40G of bandwidth

• Spanning Tree introduced around 1985

• Successfully deployed for some time…

• VPC invented to overcome STP limitations vPC Domain

• IEEE standard in 2000 (802.3ad)

• Cisco alternatives for campus design:

• vPC makes two switches look as

Cisco Live Session: BRKDCT-3378

• Every link forwarding

• Flood and learn vs VXLAN BGP-

Oversubscription at leafs if:

Oversubscription at leafs if:

reduce failure domain

Oversubscription at leafs if:

Host Host Host Host Host Host Host

20×10Gbps 5×40Gbps 2×100Gbps

Uplinks Uplinks Uplinks

20×10Gbps 20×10Gbps 20×10Gbps

6 shorter Flow Completion Time Series4

Used Fiber Pair

• Can we do the same

Robust Underlay/Fabric Flexible Overlay Virtual Network

• Intelligent Packet Handling • Scale – Reduce core state, distribute to edge

• Full Cross Sectional Bandwidth • Flexibility/Programmability – Less touch points

• Programmable & Manageable • Physical + Virtual; L2 & L3 Connectivity

VXLAN provides a Network with Segmentation, IP Mobility,

50 bytes Original CE Frame

• It’s the pillar of SDN Outer

• Works better with a control plane 8 bits 24 bits 24 bits 8 bits

ags Reserved VNIVNI Reserved

MAC_A,IP_A: 10.1.1.2 MAC_B,IP_B:20.1.1.2

Existing network – Nexus, Catalyst, etc TENANT

PRIVATE NETWORK (VRF)

BD: Blue BD: Red

Application Centric Deployment (ACI only)

• Leverage grouped constructs – Network Level; Application Level

EPG EPG EPG

Reporting Fault Mgmt

Network Centric only Network Centric

VXLAN EVPN ACI Fabric

Programmable Fabric Application Centric Infrastructure (ACI)