Cisco Complete Nexus Portfolio Deployment Best Practices: BRKDCT-2204

Cisco Complete Nexus portfolio
Deployment best practices

BRKDCT-2204
Presentation_ID © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 1
Session Goal
  Understand how to design a scalable data center based upon customer

requirements
  How to choose different flavor of the designs using Nexus family.
  Share a case study.
Recommended Sessions
BRKARC-3470: Cisco Nexus 7000 Hardware Architecture
BRKARC-3452: Cisco Nexus 5000/5500 and 2000 Switch Architecture
BRKARC-3471: Cisco NX-OS Software Architecture
BRKVIR-3013: Deploying and Troubleshooting the Nexus 1000v Virtual Switch
BRKDCT-2048: Deploying Virtual Port Channel in NX-OS
BRKDCT-2049: Overlay Transport Virtualization
BRKDCT-2081: Cisco FabricPath Technology and Design
BRKDCT-2202: FabricPath Migration Use Case
BRKDCT-2121: VDC Design and Implementation Considerations with Nexus 7000
BRKRST-2509: Mastering Data Center QoS
BRKDCT-2214: Ultra Low Latency Data Center Design - End-to-end design approach
BRKDCT-2218: Data Center Design for the Small and Medium Business
Session Agenda
  Nexus Platform Overview
  Data Center Design and Considerations
  Case Study #1: Green Field Data Center Design
  Key Takeaways
Data Center Architecture
Life used to be easy
  The Data Centre Switching Design was based on the hierarchical switching we
used everywhere
  Three tiers: Access, Aggregation and Core Core
  L2/L3 boundary at the aggregation
  Add in services and you were done Layer 3
Layer 2 Aggregation
  What has changed? Most everything 
  Hypervisors
  Cloud – Iaas, Pass, Sass
Services
  MSDC
  Ultra Low Latency
  Competition (Merchant Silicon, …)
Access
  We now sell compute !!
Presentation_ID © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Data Center Drivers
Business Regulatory Security Budget

Agility Compliance Threats Constraints
Business Challenges
Technology Trends
Proliferation
Cloud Big Data Energy Efficiency
of Devices
Data Centre Architecture
There is no ‘single design’ anymore
Spectrum of Design Evolution
blade1 blade1 blade1 blade1

slot 1
blade2 slot 1
blade2 slot 1
blade2 slot 1
blade2
slot 2
blade3 slot 2
blade3 slot 2
blade3 slot 2
blade3
slot 3
blade4 slot 3
blade4 slot 3
blade4 slot 3
blade4
slot 4
blade5 slot 4
blade5 slot 4
blade5 slot 4
blade5
slot 5
blade6 slot 5
blade6 slot 5
blade6 slot 5
blade6
slot 6
blade7 slot 6
blade7 slot 6
blade7 slot 6
blade7
slot 7
blade8 slot 7
blade8 slot 7
blade8 slot 7
blade8
slot 8 slot 8 slot 8 slot 8
slot 1
blade2 slot 1
blade2 slot 1
blade2 slot 1
blade2
slot 2
blade3 slot 2
blade3 slot 2
blade3 slot 2
blade3
slot 3
blade4 slot 3
blade4 slot 3
blade4 slot 3
blade4
slot 4
blade5 slot 4
blade5 slot 4
blade5 slot 4
blade5
slot 5
blade6 slot 5
blade6 slot 5
blade6 slot 5
blade6
slot 6
blade7 slot 6
blade7 slot 6
blade7 slot 6
blade7
slot 7
blade8 slot 7
blade8 slot 7
blade8 slot 7
blade8
slot 1
blade2 slot 1
blade2 slot 1
blade2 slot 1
blade2
slot 2
blade3 slot 2
blade3 slot 2
blade3 slot 2
blade3
slot 3
blade4 slot 3
blade4 slot 3
blade4 slot 3
blade4
slot 4
blade5 slot 4
blade5 slot 4
blade5 slot 4
blade5
slot 5
blade6 slot 5
blade6 slot 5
blade6 slot 5
blade6
slot 6
blade7 slot 6
blade7 slot 6
blade7 slot 6
blade7
slot 7
blade8 slot 7
blade8 slot
blade8 7 slot
blade8 7
Ultra Low Latency HPC/GRID Virtualized Data Center MSDC

•  High Frequency Trading •  Layer 3 & Layer 2 •  SP and Enterprise •  Layer 3 Edge (iBGP, ISIS)
•  Layer 3 & Mul:cast •  No Virtualization •  Hypervisor Virtualiza:on •  1000’s of racks
•  No Virtualization •  iWARP & RCoE •  Shared infrastructure •  Homogeneous Environment
•  Limited Physical Scale •  Nexus 2000, 3000, 5500, Heterogenous •  No Hypervisor virtualiza:on
•  Nexus 3000 & UCS 7000 & UCS •  1G Edge moving to 10G •  1G edge moving to 10G
•  10G edge moving to 40G •  10G moving to 40G •  Nexus 1000v, 2000, 5500, 7000 •  Nexus 2000, 3000, 5500, 7000 &
Presentation_ID & UCS
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS
Cisco DC Switching Portfolio
LAN LAN/SAN
Scalability
Nexus 7000
Nexus 5000
Nexus 4000
B22 FEX
Nexus 2000
Nexus 1010
Nexus 3000
Nexus 1000V
Cisco NX-OS: One OS from the Hypervisor to the Data Center Core
Convergence VM-Aware 10/40/100G Fabric Cloud Mobility
Networking Switching Extensibility
Nexus 7000 Series
Broad Range of Deployment Options
Highest 10GE Density in

Modular Switching
Nexus 7004 Nexus 7009 Nexus 7010 Nexus 7018
Height 7 RU 14 RU 21 RU 25 RU
Max BW per Slot 440 Gig/Slot 550 Gig/Slot 550 Gig/Slot 550 Gig/slot
Max 10/40/100GE ports 96/12/4 336/42/14 384/48/16 768/96/32
Air Flow Side-to-Rear Side-to-Side Front-to-Back Side-to-Side

2 x 6KW AC/DC 3 x 6KW AC/DC 4 x 6KW AC/DC
Power Supply Configurations 4 x 3KW AC
2 x 7.5KW AC 3 x 7.5KW AC 4 x 7.5KW AC
Small to Medium Core/ Data Center and Large Scale Data
Application Presentation_ID Edge Campus Core CiscoData
Public Center
© 2012 Cisco and/or its affiliates. All rights reserved. Center
Nexus 5596UP & 5548UP
Virtualized Data Center Access
  High density 1RU/2RU ToR Switches

  10GE / 1GE / FCoE / 8G FC
  Reverse airflow / DC-power
Native FC Lossless Ethernet –
FCoE, iSCSI, NAS
Innovations
  Unified Port capability
  Layer-2, -3 support
Benefits / Use-cases
  Investment protection in action!
  FEX support (24/L2)
  Proven, resilient NX-OS, designs
  Multihop FCoE/Lossless Ethernet
  Low, predictable latency at scale
  Cisco FabricPath (future)
Cisco Nexus 2000 Series
Platform Overview
Nexus B222 N2224TP

48 Port 1000M Host Interfaces 24 Port 100/1000M Host Interfaces
4 x 10G Uplinks 2 x 10G Uplinks
N2248TP N2232PP
48 Port 100/1000M Host Interfaces 32 Port 1/10G FCoE Host Interfaces
4 x 10G Uplinks 8 x 10G Uplinks
Changing the device paradigm
Cisco Nexus® 7000

Cisco Nexus® 5500
+
+
Distributed High Density Edge

Switching System
(up to 4096 virtual Ethernet
interfaces)
Cisco Nexus® 2000 FEX
Cisco Nexus® 2000 FEX
Nexus 3000: For Ultra Low Latency
1RU NX-OS Switches for 1/10/40G Connectivity
Major wins
in HFT/Web 2.0
Nexus 3048 Nexus 3064 Nexus 3016

48 ports 64 ports 16 ports
100M/1GE 1/10GE 10/40GE
Robust NX-‐OS with Differen:ated Feature Set

Wire-‐rate L2/L3 VPC, PTP, Configurable CoPP, Power-‐on Auto-‐Provisioning,
feature-‐set ERSPAN IPv6
FOR
Presentation_ID
High-‐Frequency Trading | Big Data | Web 2.0
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Nexus 1000V

VM VM VM VM VM VM VM VM
Nexus 1000V Nexus 1000V

VSM Nexus VSM Nexus
1000V 1000V
VEM VEM
VMware vSphere Windows 8 Hyper-‐V
VMware vCenter SCVMM
Consistent architecture, feature-‐set & network services ensures

opera]onal transparency across mul]ple hypervisors
Virtual Services for Nexus 1000V
Server Server Virtual Services
VM VM VM VM VM VM VM VM
Virtual ASA VSG
Nexus 1000V Switch

VMWare or Hyper-‐V VMWare or Hyper-‐V vWAAS NAM
Adapter Adapter VSG: Virtual Security Gateway

ASA: Adap]ve Security Appliance
WAAS: Wide Area Accelera]on Service
NAM: Network Analysis Module
(e.g. Nexus Network)
Customer Benefits
•  Operational consistency across physical and virtual networks
•  Network team manages physical and virtual networks
•  Integrated advanced Cisco NX-OS networking features
•  Support existing Cisco virtual network services
Session Agenda

Data Center Architecture Building Blocks
Data Center Data Center Data Center
(Intranet) (Internet/DMZ) Security (Extranet)
Intranet Extranet
Perimeter
Security
Intranet Extranet
Perimeter
Data Center Enterprise

Core Core
Core Core
BUSINESS & ENTERPRISE APPLICATIONS
DATA CENTER INTERCONNECT (DCI)
Core Core
VIRTUALIZATION
MANAGEMENT
SECURITY
Data Center Data Center
Aggregation Aggregation Aggregation DC Services
Service POD
Aggregation DC Services
Data Center
Access Access
Access
COMPUTE
STORAGE
FACILITIES
  Allow customization within blocks while maintain overall architecture
  Blocks aligned to meeting business and technical requirements

A Cloud Ready Data Center Architecture
Cisco Virtualized Multi-Tenant Data Center
Validated reference
architecture that delivers a
highly scalable, available,
secure, flexible, and efficient
data center infrastructure.
  Proven layered approach
  Reduced time to
deployment
  Reduced risk
  Increased flexibility
  Improved operational
efficiency
hap://www.cisco.com/en/US/partner/solu]ons/ns340/ns414/ns742/ns743/ns1050/landing_vmdc.html
What Makes Designing Networks for
the Data Center Different?
  Extremely high density of end nodes and
switching
  Power, cooling, and space management
constraints
  Mobility of servers a requirement, without DHCP
  The most critical shared end-nodes in the
network, high availability required with very
small service windows
  Multiple logical multi-tier application
architectures built on top of a common physical
topology
  Server load balancing, firewall, other services
required
The Evolving Data Centre Architecture
Data Center 2.0 (Physical Design == Logical Design)
  The IP portion of the Data Center Architecture

has been based on the hierarchical switching Core
design
  Workload is localized to the Aggregation Block
Layer 3
  Services localized to the applications running on
the servers connected to the physicalpod Layer 2 Aggregation
  Mobility often supported via a centralized cable

plant
  Architecture is often based on optimized design
Services
for control plane stability within the network fabric
  Goal #1: Understand the constraints of the
current approach (De-Couple the Elements of the
Design)
  Goal #2: Understand the options we have to build
a more efficient architecture (Re-assemble the
elements into a more flexible design)
Compute
Evolving Data Centre Architecture
Design Factor #1 to Re-Visit – Where are the VLAN’s
  As we move to new designs need to Where is the L2/

re-evaluate the L2/L3 scaling and design L3 boundary?
assumptions
  Need to consider VLAN Usage
Layer 3
  Policy assignment (QoS, Security, Closed User
Groups)
Layer 2 Aggregation
  IP Address Management
Scalability
  Some factors are fixed (e.g. ARP load) ARP, STP, FHRP
  Some factors can be modified by altering VLAN/
Subnet ratio
  Still need to consider L2/L3 Boundary Control
Plane Scaling
  ARP scaling (how many L2 adjacent devices)
  FHRP, PIM, IGMP
  STP logical port count (BPDUs generated per
second)
VLAN/Subnet Ratio?
  Goal: Evaluate which elements can change in VLAN span?
your architecture
Design Factor #2 to Re-Visit – Where are the Pods?
Network Pod:
Repeatable physical, compute and network infrastructure including
L2/L3 boundary equipment. The pod is traditionally the L2 failure
domain – fate-sharing domain
Access Pod:
Collection of compute nodes and network ports behind a pair
of access switches
10GE 10GE
Compute Pod:
Collection of compute nodes behind a single management
domain or HA domain
  Network and Fabric design ques]ons that depend on the choice of the
Compute Pod
  How Large is a Pod?
  Is Workload local to a Pod?
  Are Services local to a Pod?
Design Factor #2 to Re-Visit – Where are the Pods?
  The efficiency of the power

and cooling for the Data
Center is largely driven by the
physical/logical layout of the
Compute Pod
  The design of the network
and cable plant interoperate
to define the flexibility
available in the architecture
  Evolution of Server and
Storage connectivity driving
changes in the cable plant
Design Factor #2 to Re-Visit – Where are the cables?
Design Factor #2 to Re-Visit – Where are the cables?
Design Factor #3 to Re-Visit – How is the compute attached?
  How is “striping of workload” across the physical Data Center accomplished (Rack,
Grouping of Racks, Blade Chassis, …)?
  How is the increase in percentage of devices attached to SAN/NAS impacting the
aggregated I/O and cabling density per “compute unit”?
  Goal: Define the unit of Compute I/O and how it is managed (how does the cabling
connect the compute to the network and fabric)
blade1
slot 1
blade2
slot 2
blade3
slot 3
blade4
slot 4
blade5
slot 5
blade6
slot 6
blade7
slot 7
blade8
slot 8
blade1
slot 1
blade2
slot 2
blade3
slot 3
blade4
slot 4
blade5
slot 5
blade6
slot 6
blade7
slot 7
blade8
slot 8
blade1
slot 1
blade2
slot 2
blade3
slot 3
blade4
slot 4
blade5
slot 5
blade6
slot 6
blade7
slot 7
blade8
slot 8
Rack Mount Blade Chassis Integrated - UCS

Design Factor #4 to Re-Visit - Where Is the Edge?
veth vFC vFC vFC vFC

Eth vFC 1 2 3 4 126
Eth FC Eth FC 2/12 3 Converged SR-IOV adapter
2/12 provides
2/12 3/11 3/11 Network
10GbE
multiple PCIe
10GbE Fibre Channel

VETH
Adapter 10GE -‐ VNTag resources

Edge of the Network and Fabric Link provides
virtualization of
Ethernet
pNIC HBA Still 2 PCI the physical

NIC HBA Addresses Media Eth FC Eth FC Eth
PCIe 1 2 3 4 126
on the BUS
PCI-E Bus
VMFS PCI-E Bus
Edge of the VMFS
PCI-E Bus VETH SCSI Edge of PCI-E Bus
VETH SCSI
Fabric the Fabric Edge of
Pass VMFS
Thru SCSI
VNIC VNIC the Fabric
Operating VNIC
System and Hypervisor provides Hypervisor provides

Device Drivers virtualization of PCI-E virtualization of PCI-E Hypervisor provides
resources resources virtualization of PCI-E
resources
Compute and Fabric Edge are Merging
connected?
iSCSI iSCSI NAS NAS
FCoE SAN Appliance Gateway Appliance Gateway
Computer System Computer System Computer System Computer System Computer System
Applica:on Applica:on Applica:on Applica:on Applica:on

File System File System File System File System File System
Volume
Volume MManager
anager Volume Manager Volume Manager I/O Redirector I/O Redirector
SCSI Device Driver SCSI Device Driver SCSI Device Driver NFS/CIFS NFS/CIFS
The Flexibility of a
iSCSI Driver iSCSI Driver TCP/IP Stack
FCoE Driver TCP/IP Stack TCP/IP Stack
TCP/IP Stack
NIC NIC NIC NIC NIC
Unified Fabric Block I/O File I/O

Transport SAN IP IP IP IP
‘Any RU to Any
NIC NIC NIC NIC
Spindle’ TCP/IP Stack TCP/IP Stack TCP/IP Stack TCP/IP Stack
iSCSI Layer iSCSI Layer File System File System
FCoE Bus Adapter FC HBA Device Driver FC HBA
FC Block I/O FC
Design Factor #6 to Re-Visit—Where Are the Services?
Client
  In the non-virtualized model

services are inserted into the Data
Path at ‘choke points’
  Logical Topology matches the
Physical
  Virtualized workload may require a
re-evaluation of where the services
are applied and how they are
scaled
  Virtualized Services associated
with the Virtual Machine
(Nexus 1000v & vPath) VM VM VM
#2 #3 #4
Virtualized Services Nexus 1000v &

  Virtual Machine Isolation (VXLAN) vPath
VSG, vWAAS
Hierarchical Design Network Layers
Defining the Terms
  Data Center Core

‒  Routed layer which is distinct from enterprise network core Enterprise Network
‒  Provides scalability to build multiple aggregation blocks

Data Center
  Aggregation Layer Core
‒  Provides the boundary between layer-3 routing and layer-2
switching
‒  Point of connectivity for service devices (firewall, LB, etc.)
Layer 3 Links
Aggrega3on
  Access Layer Layer 2 Trunks
‒  Provides point of connectivity for servers and shared
resources
‒  Typically layer-2 switching
Access
Data Center Core Layer Design
Core Layer Function & Key Considerations
  High speed switching &100% layer 3

  Fault domain isolation between Enterprise and DC
Enterprise Network
  AS / Area boundary
  Routing table scale Data Center
Core
  Fast routing convergence
Layer 3 Links
Aggrega3on
Layer 2 Trunks
Access
Data Center Core Layer Design
Commonly Deployed Platform and Modules
M2-10G LC
  Platform: Nexus 7K M2-40G LC M1-10G LC F2-Series LC
M2-100G LC*
  Modules
4.0 and 6.0(1) and
Software 6.1 or above
M1: L2/L3/L4 with large forwarding later* later
tables Fabric
240G/200G* 80G 480G*
Connection
and rich feature set L3 IPv4
128K/1M 128K/1M 32K
Unicast
F2: Low-cost, high density with high L3 IPv4
N/A 32K 16K
Multicast
performance, low latency and low power
L3 IPv6
6K/350K Up to 350K 32K
Unicast
  Classic layer 3 Core: M1 or F2
L3 IPv6
N/A 16K 8K
  Large routing and ACL tables: M1 Multicast
ACL Entries 64/128K 128K 16K
  High density linerate10G: F2 MPLS ✓ ✓ ✗
  MPLS: M1 LISP and OTV ✓ ✓ ✗
Data Center Aggregation Layer Design
Virtualized Aggregation Layer provides
Enterprise Network
  L2 / L3 boundary
  Access layer connectivity point: STP root,
Data Center
  loop-free features Core
  Service insertion point

  Network policy control point: default GW,
  DHCP Relay, ACLs Aggrega3on
Access
Commonly Deployed Platform and Modules
M1-10G LC F1-Series LC F2-Series LC N5500 with L3

  Platform: N7K/N5K
  Features(L3, OTV
etc.) Min. Software 4.0* 5.1(1) 6.0(1) 5.0(3)N1(1)
Fabric Connection 80G 230G 480G* -

  Scalability L3 IPv4 Unicast 128K/1M - 32K 8K
(routing/mac table) L3 IPv4 Multicast 32K - 16K 2K
  Performance and MAC Entries 128K 16K (per SOC) 16K (per SOC) 32K
FEX Support Yes* No Yes Yes
port density L2 Portchannel 8 active 16 active 16 active 16 active
LISP and OTV ✓ ✗ ✗ ✗
FabricPath ✗ ✓ ✓ ✓
FCOE Support ✗ ✓ ✓ ✓
Key Design Considerations
Enterprise Network
  Data Center physical infrastructure
POD design & cabling infrastructure
Data Center
Size of the layer 2 domain Core
  Oversubscription ratio
Traffic flow
No. of access layer switches to aggregate Aggrega3on
Scalability requirement
  Service insertion
Service chassis vs. appliance Access
Firewall deployment model

Load balancer deployment model
Data Center Access Layer Design
Access Layer & Virtualized Edge
  Access Layer provides

Hosts connectivity point
Mapping from virtual to physical
L2 Services: LACP, VLAN Trunking
  Virtualized Edge provides

Virtual host connectivity point
Virtual extension of access services
Network policy enforcement point
Access Layer Key Considerations & Commonly Deployed Platform
  Physical infrastructure N5548/N5596 N5010/N5020 N7 F-Series LC

TOR vs. MoR
  Server Types Fabric Throughput 960G/1.92T 520G/1.04T 230G/480G
1 G vs 10G Port Density 48/96 26/52 32/48 per LC
A/A or A/S NIC No. of Vlans 4096 512 4096
Single attached MAC Entries 32K 16K 16K (per SOC)
No. of FEXs 24 12 32 (F2 only)
  Oversubscription ratio
1G FEX Ports 1152 576 32/48 per LC
No. of servers and uplinks
10G FEX Ports 768 384 32/48 per LC
  Virtual Access Requirements
8G Native FC Ports 48/96 6/12 -
Virtual Machine Visibility
FabricPath ✓ ✗ ✓
Virtual Machine Management
Boundary
Cisco FEXlink: Virtualized Access Switch
Changing the device paradigm
  De-Coupling of the Layer 1 and Layer 2 Topologies

  New Approach to Structured Building Block
  Simplified Management Model, plug and play provisioning,
centralized configuration
  Technology Migration with minimal operational impact
  Long Term TCO due to ease of Component Upgrade
...
Virtualized Switch
Evolutionary Fabric Edge
Mixed 1/10G, FC/FCoE, Rack and Blade
  Consolidation for all servers both rack and blade onto the same virtual switch
  Support for 1G, migration to 10G, FC and migration to FCoE
10G server racks are supported by the Support for direct connection of HBA to
1G server racks are supported by 1G FEX (2248TP, addition of 10G FEX (2232PP or 2232TM, Unified Ports on Nexus 5500UP
2224TP) or future proofed with 1/10G FEX 2248PQ)
(2232PP or 2232TM)
1G, 10G and FCoE connectivity for HP or Support for NPV attached blade switches
Dell Blade Chassis during FC to FCoE migration
Data Center Interconnect Design
Data Center Interconnect Drivers
IP Routed
  DC to DC IP connectivity Main Data
Center
L3
Service L3 Backup
Data Center
  DC to DC LAN extension EoMPLS

L2
L2
Workload scaling with vMotion WAAS
EoMPLSoGRE WAAS
L2 L2
GeoCluster
DWDM/
CWDM
Disaster recovery SAN SAN
Non-disruptive DC migration FC FC
  Storage extension and replication
Storage
Storage
Data Center Interconnect Design
DCI LAN Extension Key Considerations
  STP domain isolation

  Multihoming and loop avoidance
  Unknown unicast flooding and
broadcast storm control
  FHRP redundancy and localization
  Scalability and convergence time
  Three Nexus based options
OTV
vPC
Fabric Path
Cisco FabricPath
Switching Rou:ng
  Easy Configura:on   Mul:-‐pathing (ECMP)
  Plug & Play   Fast Convergence
  Provisioning Flexibility   Highly Scalable
FabricPath
“FabricPath brings Layer 3 rou5ng benefits to

flexible Layer 2 bridged Ethernet networks”
Blocked Links
Fully Non-Blocking
2:1
FabricPath
Oversubscription 16:1
Pods
4
8:1
2
Overlay Transport Virtualization
Technology Pillars
OTV is a “MAC in IP” technique to

extend Layer 2 domains
OVER ANY TRANSPORT
Dynamic Encapsulation Protocol Learning
No Pseudo-Wire State Nexus 7000 Preserve Failure

Maintenance First platform to support OTV Boundary
(since 5.0 NXOS Release)
Optimal Multicast
Built-in Loop Prevention
Replication
Multipoint Connectivity Automated Multi-homing

ASR 1000
Point-to-Cloud Model Now also supporting OTV Site Independence
(since 3.5 XE Release)
DCI Architectures
OTV
Greenfield ASR 1K Brownfield

Greenfield
L3
Si Si
OTV OTV
L2
Nexus 7K
L3
Si Si
L2
OTV OTV OTV OTV
Nexus 7K
Nexus 7K Nexus 7K
L3   Leverage OTV capabili]es on Nexus 7000 (Greenfield) and ASR

L3
L2 1000 (Brownfield)
L2   Build on top of the tradi]onal DC L3 switching model (L2-‐L3
FabricPath boundary in Agg, Core is pure L3)
OTV Virt. Link
  Possible integra]on with the FabricPath/TRILL model
Overlay Transport Virtualization
  Extensions over any transport (IP, MPLS)   Automated Built-in Multihoming
  Failure boundary preservation   End-to-End loop prevention
  Optimal BW utilization   ARP Optimization

(no head-end replication)
MAC TABLE Transport MAC TABLE

VLAN MAC IF Infrastructure VLAN MAC IF
100 MAC 1 Eth 2 IP A IP B 100 MAC 1 IP A
OTV OTV OTV OTV
100 MAC 2 Eth 1
MAC 1  MAC 3 IP A  IP B 100 MAC 2 IP A
100 MAC 3 IP B MAC 1  MAC 3 IP A  IP B
100 MAC 4 IP B
100 MAC 3 Eth 3
100 MAC 4 Eth 4
MAC 1  MAC 3 6
MAC 1  MAC 3 MAC 1 West East
Site Site MAC 3
Fabric Simplicity, Scale and Flexibility
Nexus Edge, Core & Boundary Nodes
  Isolation of function when possible
  Spine provides transport
Nexus Boundary   Compute Edge provides media type and scaled control plane
(OTV, LISP, MPLS)   Boundary provides localization of complex functions
Nexus Spine (Redundant

and Simple)
Nexus
Edge
blade1 blade1 blade1 blade1 blade1 blade1

slot 1 slot 1 slot 1
blade2 slot 1
blade2 slot 1
blade2 slot 1
blade2
blade2
slot 2 blade2
slot 2 slot 2
blade3 slot 2
blade3 slot 2
blade3 slot 2
blade3
blade3 blade3 slot 3 slot 3 slot 3
blade4 slot 3
blade4
slot 3
blade4 slot 3
blade4 blade4 blade4 slot 4
blade5 slot 4
blade5
slot 4
blade5 slot 4
blade5 slot 4
blade5 slot 4
blade5 slot 5
blade6 slot 5
blade6
slot 5
blade6 slot 5
blade6 slot 5
blade6 slot 5
blade6 slot 6
blade7 slot 6
blade7
slot 6
blade7 slot 6
blade7 slot 6
blade7 slot 6
blade7 slot 7
blade8 slot 7
blade8
slot 7
blade8 slot 7
blade8 slot 7
blade8 slot 7
blade8 slot 8 slot 8

blade2 slot 1
blade2 slot 1
blade2 slot 1
blade2
blade2
slot 2 blade2
slot 2 slot 2
blade3 slot 2
blade3 slot 2
blade3 slot 2
blade3
blade4 slot 3
blade4
slot 3
blade4 slot 3
blade5 slot 4
blade5
slot 4
blade5 slot 4
blade5 slot 4
blade5 slot 4
blade5 slot 5
blade6 slot 5
blade6
slot 5
blade6 slot 5
blade6 slot 5
blade6 slot 5
blade6 slot 6
blade7 slot 6
blade7
slot 6
blade7 slot 6
blade7 slot 6
blade7 slot 6
blade7 slot 7
blade8 slot 7
blade8
slot 7
blade8 slot 7
blade8 slot 7
blade8 slot 7
blade2 slot 1
blade2 slot 1
blade2 slot 1
blade2
blade2
slot 2 blade2
slot 2 slot 2
blade3 slot 2
blade3 slot 2
blade3 slot 2
blade3
blade4 slot 3
blade4
slot 3
blade4 slot 3
blade5 slot 4
blade5
slot 4
blade5 slot 4
blade5 slot 4
blade5 slot 4
blade5 slot 5
blade6 slot 5
blade6
slot 5
blade6 slot 5
blade6 slot 5
blade6 slot 5
blade6 slot 6
blade7 slot 6
blade7
slot 6
blade7 slot 6
blade7 slot 6
blade7 slot 6
blade7 slot 7
blade8 slot 7
blade8
slot 7
blade8 slot 7
blade8 slot 7
blade8 slot 7
Session Agenda

Case Study #1
Data Center High Level Requirements
  A leading online higher education   Customer Business Challenges
institution   x10G based virtualized next
  More than 500,000 students, 24,000 generation data center architecture
faculty members   No STP blocking topology
  Approximately 1200 servers and 600 VMs   Firewall protection for secured servers
across 5 data centers
  Support vMotion within and between
  Current data centers reach the limit of data centers
switching, power, and cooling capacity
  Network team gains visibility to VM
  Business decision made to build two new networking
green field data centers to consolidate and
provide DR capability
Virtualized Access Layer Requirements
  400 10G capable server connections

30 ESX servers with roughly 600 VMs
  800 1G connections for standalone servers, and

out of band management network
  Support both active/active and active/standby NIC
teaming configuration
  Network team manages network, Server team

manages server/virtual machines
  Network policies are retained during vMotion
  N2K being remote line card to reduce number of devices to

manage
  Migration to ToR for 10GE servers or selective 1GE server racks if
required (mix of ToR and EoR)
  Mixed cabling environment (optimized as required)
  Flexible support for Future Requirements
. . .
Nexus 5000/2000 Mixed
ToR & EoR
Combina:on of EoR (End of Row) and ToR (Top of Rack) cabling
Access Layer Port Counts & Oversubscription
  For 10G server off the 5596s   For 1G server off the 5548s
Total 10G ports = 20*32 = 640 Total 1G ports = 20*48 = 960
Server NIC utilization = 50% Server NIC utilization = 50%
Total uplink BW = 16*10 = 160G Total uplinks BW = 8*10 = 80G
Oversubscription ratio = 160/(640*0.5*10) = 1/20 Oversubscription ratio = 80/(960x0.5) = 1/6
N1KV Gains Visibility Into VM Environment
Cisco Nexus™ 1000V

Sokware Based
VM VM VM VM
  Built on Cisco NX-‐OS
  Compa]ble with all switching plalorms
  Maintain vCenter provisioning model Nexus
1000V
unmodified for server administra]on;

allow network administra]on of virtual
vSphere
network via familiar Cisco NX-‐OS CLI
Nexus 1000V
Policy-‐Based Mobility of Network and Non-‐Disrup]ve

VM Connec]vity Security Proper]es Opera]onal Model
Nexus 1000V Uplink Options
Spanning-Tree (Active/Passive)
Mac Pinning
•  UCS Blade Server Environment
•  3rd party blade server environment in non-MCEC Channel-group auto mode
topologies on mac-pinning
Single Switch Port-Channel

Port-Channel with single switch
•  Upstream switches do not support MCEC Channel-group auto mode
[active | passive]
Multi-Chassis EtherChannel
Port-channel with two switches Channel-group auto mode
•  Any server connected to upstream switches that
[active | passive]
supports Multi-Chassis EtherChannel (MCEC)
Access Layer Design Highlight
Requirement Solution
Flexible cabling N5K/2K provide mixed ToR & EoR
Ease of management Configurations only done on the 5Ks
1G, 10G server connectivity Straight-through FEX supports all the

with active/active, active/ NIC teaming options
standby NIC teaming Note: EVPC provides flexible server/
FEX topology
vMotion within the Data N5K operates in layer 2 only to make

Center larger layer 2 adjacency possible
Visibility to VMs N1KV provides network visibility to VM

Network team manages Clear management boundary defined
network, server team by N1KV
manages server
Aggregation Requirements
  Facility
Drop any server anywhere in the data center
  L2-L3
Layer 2 domain within data center
No STP blocking topology
  Service Layer
Secured zone and non-secured zone
FW protection between zones, no FW protection within the zone
LB service is required for Web server, server needs to track the client IP
High performance FW and LB are required
NAM and IPS solution are also required
Physical Infrastructure and Network Topology
Physical to Logical Mapping
Aggregation Oversubscription Ratio
  Large layer 2 domain with single pair of 7Ks

  Worse Case Calculation
Assume all the traffic is north-south bound
Assume 100% utilization from the 5Ks
All the ports operated in dedicated mode
Service Integration at Aggregation Layer
Service chassis vs. Appliance
  Virtual Service with Nexus 1000V

Virtual/Cloud Data Center
VDC-‐1
APP
OS
Hypervisor
VDC-‐2
Virtual •  Virtual appliance form factor

Service •  Dynamic instantiation/provisioning
Node •  Service transparent to VM mobility
(VSN) •  Support scale-out
•  Large scale multitenant operation
Service Integration-Physical Design
  High performance solution

ASA5585 Firewall and IPS
ACE30 module
  Low TCO
6500 repurpose
  Most scalable
NAM module inside service chassis
Available slot for future expansion
Firewall Logical Deployment Model
3.3.3.0 3.3.3.0
Bridging
Router GW Vlan 30 Vlan 31
1.1.1.0 1.1.1.0
FW
  Transparent Mode Router GW Vlan 10 Vlan 11
Pros : Easy to implement Router GW Vlan 40 Vlan 41

4.4.4.0 4.4.4.0
Cons: 8 bridge-group per contextRouting 3.3.

3.0
30
Vlan
  Routing Mode 1.1.1.0 GW 2.2.2.0
Router Vlan 10 FW GW Vlan 20
Pros: More scalable
GW V
lan 40
Cons: Configuration complexity 4.4.4.0
VRF sandwich 3.3.3.0

Vlan 30 GW Vlan 31
VDC sandwich 3.3.3.0
Nexus
Nexus 1.1.1.0
GW Vlan 31 (external FW (internal
vlan100 Vlan 10 GW Vlan 11
Nexus 1.1.1.0 Vrf) Vrf)
Nexus
FW (internal
(external vlan100 Vlan 10 GW Vlan 11 Vlan 40 GW Vlan 41
VDC)
VDC)
GW Vlan 41
4.4.4.0
Load Balancer Logical Deployment Model
3.3.3.0 3.3.3.0
Bridging
1.1.1.0
  Transparent Mode 1.1.1.0
Router GW Vlan 10 LB Vlan 11
Pros: Ease of deployment and multicast support

4.4.4.0
Cons: 8 bridge-group per context 4.4.4.0
  Routing Mode 3.3.

3.0
Routing 30
Vlan
Pros: Separate STP domain 1.1.1.0 GW 2.2.2.0
Router Vlan 10 LB GW Vlan 20
Cons: No routing protocol support
GW V
lan 40
4.4.4.0
LB
  One Arm Mode
Vlan 201
Pros: Non-LB traffic bypass the LB One Arm 3.3.3.0
GW Vlan 31
Cons: SNAT or PBR required
Nexus 1.1.1.0
Vlan 10 GW Vlan 11
GW Vlan 41
Presentation_ID © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 4.4.4.0 61
Service Integration – Logical Design
  VRF sandwich design

Three VRF created on Agg N7K
Server default gateway is Agg N7k
No VRF route leaking
  LB in transparent mode
Tracking client IP is possible
Multicast application behind LB is possible
Two ACE contexts plus admin context
  FW in routing mode
FW provides routing between VRFs
Two FW contexts plus admin and system context
Spanning Tree Recommendations
N Network port
E Edge or portfast port type
- Normal port type
Data Center Core B BPDUguard
R Rootguard
L Loopguard
Primary Secondary
vPC vPC
vPC
HSRP Domain HSRP
Layer 3
ACTIVE STANDBY
Aggregation N N Secondary
Primary
Root Root
- - - - - - - - Layer 2 (STP + Rootguard)

R R R R R R R R
-
Access
- - L
E E E E E
B B B B B
Layer 2 (STP + BPDUguard)
vPC Best Practice Features
Feature Benefit Overview

vPC auto-recovery Increase High-availability allows the one vPC device to assume STP / vPC primary role and
(reload restore) bring up all local vPCs in case other vPC peer device is down after DC
power outage
vPC Peer-Gateway Service continuity Allows a vPC switch to act as the active gateway for packets
addressed to the peer router MAC
vPC orphan-ports Increase High-availability When vPC peer-links go down, vPC secondary shuts down all the vPC
suspend member ports as well as orphan ports. It avoids single attached
devices like FW,LB or NIC teamed device get isolated during vPC
peer-link failure
vPC ARP SYNC Improve Convergence time Improve Convergence for Layer 3 flows after vPC peer-link is UP
vPC Peer-Switch Improve Convergence time Virtualize both vPC peer devices so they appear as a unique STP root
BRKDCT-2048: Deploying Virtual Port Channel in NX-OS
Aggregation Layer Design Highlight
Drop any server anywhere in Single pair of 7Ks provide data center Enterprise Network
the DC wide layer 2 domain
vMotion within the DC
No STP blocking Topology Double sided vPC between 7K and 5K Data Center
Core
eliminating blocking ports
FW protection between secure FW virtualization and VDC sandwich
zone and non-secure zone design to provide logical separation and
protection
Layer 3 Links
Aggrega3on
Web servers require load LB in transparent mode provides service Layer 2 Trunks
balancing service per Vlan basis
High throughput services are Mixed of service chassis and appliance
required and future scalability design is able to provide flexible and
scalable service choices Access
Low subscription ratio (target M1 10G line cards configured in
15:1) dedicated mode to provide lower
subscription ratio
Core Layer Design
Nexus 7010 with Redundant M1 line cards
  10G layer 3 port channels to Aggregation switches Enterprise Network
  OSPF as IGP
Data Center
Inject default into data center Core
  Fault domain separation via BGP

eBGP peering with enterprise network
Layer 3 Links
eBGP peering with remote data center Aggrega3on
Layer 2 Trunks
  DC Interconnects are connected onto the Core N7Ks
Access
DCI Requirements and Design Choices
Requirements IGP + PIM Peering
  L2 connectivity to provide OTV

VDC
OTV
VDC
OTV
VDC
OTV
VDC
workload scaling with IGMPv3

PIM Interface
vMotion L3 Join Interface

L2 Internal Interface
  Data replication between

the data centers
DC 1 DC 2
  Potential 3rd data center CO Dark Fiber
CO
RE
RE
DCI Design Choices
  OTV
AG
GR
GR
AG
  vPC
ACC
ESS
ACC
ESS
  Fabric Path Server Farms Server Farms
OTV Design
Dark fiber links

Why OTV
  Native STP and broadcast isolation
Data Center
  Easy to add 3rd site Core
  Existing multicast core

Aggrega]on
Design
  OTV VDC on Aggregation OTV
OTV OTV OTV
VPC VPC VDC
  No HSRP localization(phase1)
VDC VDC VDC
simplify configuration
minimal latency via dark fiber Data Center 1 Data Center 2
Nexus 1000v Deployment for VM Mobility
  Both VSM in the same data center

  Layer 3 control on Nexus 1000V
  Stretched cluster supports live vMotion (5ms latency)
Data Center #1 Data Center #2
VSM (Ac]ve) vCenter VSM Layer 2 Extension
(Ac]ve) (Standby) (OTV)
vSphere vSphere Virtualized Workload Mobility vSphere vSphere

Nexus 1000V VEM Nexus 1000V VEM Nexus 1000V VEM Nexus 1000V VEM
Stretched Cluster
vCenter SQL/Oracle Replicated vCenter SQL/Oracle

Database Database
Dark Fiber
Traffic Flow for VM Mobility
vMotion between Data Centers
  Virtual machines still use the original ACE and gateway after vMotion
  Traffic will ‘trombone’ the DCI link for vMotioned virtual machines
  No HSRP localization and Source NAT
Overall Design Highlight – Case Study #1
x10G based virtualized generation Nexus 7K,5K,2K provide scalable x10G
data center architecture architecture with end to end virtualization
No STP blocking Topology Double sided vPC between 7K and 5K

eliminating blocking ports
FW protection between secure FW virtualization and VRF sandwich design to

zone and non-secure zone provide logical separation and protection
Support vMotion within and L2/L3 boundary is placed at aggregation layer

between data centers to provide data center wide layer 2 domain
OTV provide layer 2 extension between data
centers
Network team gains visibility to Nexus 1000v provides clear management
VM networking boundary between network and server team.
Network policy through N1KV is implemented
on VMs
Key Takeaways
  Nexus family and NX-OS are designed for modern data center
architecture
  3 tier design model (Core, Aggregation, Access) ensure high availability &
scalability
  Nexus 5K/2K offer flexible cabling solution at Access
  Nexus 7K/5K double sided vPC supports non-blocking topology and
larger layer 2 domain. Fabric path is the new trend
  Nexus 7K virtualization provides flexible service insertion at Aggregation
  OTV/FabricPath/vPC simplify DCI and migration solution
  Nexus 1000v provides network policy control & visibility into VM, and
offers integrated virtual services (VSG, vWAAS, NAM, ASA) at VM level
THANK YOU for Listening & Sharing
Your Thoughts

Cisco Complete Nexus Portfolio Deployment Best Practices: BRKDCT-2204

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cisco Complete Nexus Portfolio Deployment Best Practices: BRKDCT-2204

Uploaded by

Copyright:

Available Formats

Cisco Complete Nexus portfolio

Deployment best practices

 Understand how to design a scalable data center based upon customer

 How to choose different flavor of the designs using Nexus family.

 Share a case study.

Business Regulatory Security Budget

Spectrum of Design Evolution

blade1 blade1 blade1 blade1

Ultra Low Latency HPC/GRID Virtualized Data Center MSDC

Highest 10GE Density in

Nexus 7004 Nexus 7009 Nexus 7010 Nexus 7018

Max 10/40/100GE ports 96/12/4 336/42/14 384/48/16 768/96/32

Air Flow Side-to-Rear Side-to-Side Front-to-Back Side-to-Side

 High density 1RU/2RU ToR Switches

Nexus B222 N2224TP

Cisco Nexus® 7000

Distributed High Density Edge

Nexus 3048 Nexus 3064 Nexus 3016

Robust NX-­‐OS with Diﬀeren:ated Feature Set

Nexus 1000V Nexus 1000V

VMware vCenter SCVMM

Consistent architecture, feature-­‐set & network services ensures

Nexus 1000V Switch

Adapter Adapter VSG: Virtual Security Gateway

(e.g. Nexus Network)

 Nexus Platform Overview

Data Center Enterprise

DATA CENTER INTERCONNECT (DCI)

 Allow customization within blocks while maintain overall architecture

 Blocks aligned to meeting business and technical requirements

 The IP portion of the Data Center Architecture

 Mobility often supported via a centralized cable

 As we move to new designs need to Where is the L2/

 The efficiency of the power

Rack Mount Blade Chassis Integrated - UCS

veth vFC vFC vFC vFC

10GbE Fibre Channel

Adapter 10GE -­‐ VNTag resources

System and Hypervisor provides Hypervisor provides

Compute and Fabric Edge are Merging

Applica:on Applica:on Applica:on Applica:on Applica:on

Unified Fabric Block I/O File I/O

 In the non-virtualized model

Virtualized Services Nexus 1000v &

 Data Center Core

‒ Provides scalability to build multiple aggregation blocks

 High speed switching &100% layer 3

 Service insertion point

M1-10G LC F1-Series LC F2-Series LC N5500 with L3

Fabric Connection 80G 230G 480G* -

Firewall deployment model

 Access Layer provides

 Virtualized Edge provides

 Physical infrastructure N5548/N5596 N5010/N5020 N7 F-Series LC

 De-Coupling of the Layer 1 and Layer 2 Topologies

 DC to DC LAN extension EoMPLS

Disaster recovery SAN SAN

 Storage extension and replication

 STP domain isolation

“FabricPath brings Layer 3 rou5ng beneﬁts to

OTV is a “MAC in IP” technique to

Dynamic Encapsulation Protocol Learning

No Pseudo-Wire State Nexus 7000 Preserve Failure

  Understand how to design a scalable data center based upon customer

  How to choose different flavor of the designs using Nexus family.

  Share a case study.

  High density 1RU/2RU ToR Switches

Robust NX-‐OS with Diﬀeren:ated Feature Set

Consistent architecture, feature-‐set & network services ensures

  Nexus Platform Overview

  Allow customization within blocks while maintain overall architecture

  Blocks aligned to meeting business and technical requirements

  The IP portion of the Data Center Architecture

  Mobility often supported via a centralized cable

  As we move to new designs need to Where is the L2/

  The efficiency of the power

Adapter 10GE -‐ VNTag resources

  In the non-virtualized model

  Data Center Core

‒  Provides scalability to build multiple aggregation blocks

  High speed switching &100% layer 3

  Service insertion point

  Access Layer provides

  Virtualized Edge provides

  Physical infrastructure N5548/N5596 N5010/N5020 N7 F-Series LC

  De-Coupling of the Layer 1 and Layer 2 Topologies

  DC to DC LAN extension EoMPLS

  Storage extension and replication

  STP domain isolation

L3   Leverage OTV capabili]es on Nexus 7000 (Greenﬁeld) and ASR

  Extensions over any transport (IP, MPLS)   Automated Built-in Multihoming

  Failure boundary preservation   End-to-End loop prevention

  Optimal BW utilization   ARP Optimization

  Nexus Platform Overview

  400 10G capable server connections

  800 1G connections for standalone servers, and

  Network team manages network, Server team

  N2K being remote line card to reduce number of devices to

network via familiar Cisco NX-‐OS CLI

Policy-‐Based Mobility of Network and Non-‐Disrup]ve

  Large layer 2 domain with single pair of 7Ks

  Virtual Service with Nexus 1000V

Virtual •  Virtual appliance form factor

  High performance solution

  Routing Mode 3.3.

  VRF sandwich design

  10G layer 3 port channels to Aggregation switches Enterprise Network

  Fault domain separation via BGP

  L2 connectivity to provide OTV

  Data replication between

  Existing multicast core

  Both VSM in the same data center

  No HSRP localization and Source NAT