Professional Documents
Culture Documents
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 1
Session Goal
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Recommended Sessions
BRKARC-3470: Cisco Nexus 7000 Hardware Architecture
BRKARC-3452: Cisco Nexus 5000/5500 and 2000 Switch Architecture
BRKARC-3471: Cisco NX-OS Software Architecture
BRKVIR-3013: Deploying and Troubleshooting the Nexus 1000v Virtual Switch
BRKDCT-2048: Deploying Virtual Port Channel in NX-OS
BRKDCT-2049: Overlay Transport Virtualization
BRKDCT-2081: Cisco FabricPath Technology and Design
BRKDCT-2202: FabricPath Migration Use Case
BRKDCT-2121: VDC Design and Implementation Considerations with Nexus 7000
BRKRST-2509: Mastering Data Center QoS
BRKDCT-2214: Ultra Low Latency Data Center Design - End-to-end design approach
BRKDCT-2218: Data Center Design for the Small and Medium Business
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Agenda
Nexus Platform Overview
Data Center Design and Considerations
Case Study #1: Green Field Data Center Design
Key Takeaways
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Data Center Architecture
Life used to be easy
The Data Centre Switching Design was based on the hierarchical switching we
used everywhere
Three tiers: Access, Aggregation and Core Core
L2/L3 boundary at the aggregation
Add in services and you were done Layer 3
Layer 2 Aggregation
What has changed? Most everything
Hypervisors
Cloud Iaas, Pass, Sass
Services
MSDC
Ultra Low Latency
Competition (Merchant Silicon, )
Access
We now sell compute !!
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Data Center Drivers
Business Challenges
Technology
Trends
Proliferation
Cloud Big Data Energy Efficiency
of Devices
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Data Centre Architecture
There is no single design anymore
Nexus 7000
Nexus
5000
Nexus
4000
B22
FEX
Nexus
2000
Nexus
1010
Nexus
3000
Nexus 1000V
Cisco NX-OS: One OS from the Hypervisor to the Data Center Core
Convergence VM-Aware 10/40/100G Fabric Cloud Mobility
Networking Switching Extensibility
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000 Series
Broad Range of Deployment Options
Height 7 RU 14 RU 21 RU 25 RU
Max BW per Slot 440 Gig/Slot 550 Gig/Slot 550 Gig/Slot 550 Gig/slot
N2248TP N2232PP
48 Port 100/1000M Host Interfaces 32 Port 1/10G FCoE Host Interfaces
4 x 10G Uplinks 8 x 10G Uplinks
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Changing the device paradigm
+
+
Major
wins
in
HFT/Web
2.0
FOR
Presentation_ID
High-Frequency
Trading
|
Big
Data
|
Web
2.0
2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Nexus 1000V
VM VM VM VM VM VM VM VM
Customer Benefits
Operational consistency across physical and virtual networks
Network team manages physical and virtual networks
Integrated advanced Cisco NX-OS networking features
Support existing Cisco virtual network services
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Agenda
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Data Center Architecture Building Blocks
Data Center Data Center Data Center
(Intranet) (Internet/DMZ) Security (Extranet)
Intranet Extranet
Perimeter
Security
Intranet Extranet
Perimeter
Core Core
VIRTUALIZATION
MANAGEMENT
SECURITY
Data Center Data Center
Aggregation Aggregation Aggregation DC Services
Service POD
Aggregation DC Services
Data Center
Access Access
Access
COMPUTE
STORAGE
FACILITIES
Validated reference
architecture that delivers a
highly scalable, available,
secure, flexible, and efficient
data center infrastructure.
Proven layered approach
Reduced time to
deployment
Reduced risk
Increased flexibility
Improved operational
efficiency
hap://www.cisco.com/en/US/partner/solu]ons/ns340/ns414/ns742/ns743/ns1050/landing_vmdc.html
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
What Makes Designing Networks for
the Data Center Different?
Extremely high density of end nodes and
switching
Power, cooling, and space management
constraints
Mobility of servers a requirement, without DHCP
The most critical shared end-nodes in the
network, high availability required with very
small service windows
Multiple logical multi-tier application
architectures built on top of a common physical
topology
Server load balancing, firewall, other services
required
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
The Evolving Data Centre Architecture
Data Center 2.0 (Physical Design == Logical Design)
Access Pod:
Collection of compute nodes and network ports behind a pair
of access switches
10GE 10GE
Compute Pod:
Collection of compute nodes behind a single management
domain or HA domain
Network
and
Fabric
design
ques]ons
that
depend
on
the
choice
of
the
Compute
Pod
How
Large
is
a
Pod?
Is
Workload
local
to
a
Pod?
Are
Services
local
to
a
Pod?
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Evolving Data Centre Architecture
Design Factor #2 to Re-Visit Where are the Pods?
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Evolving Data Centre Architecture
Design Factor #2 to Re-Visit Where are the cables?
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Evolving Data Centre Architecture
Design Factor #2 to Re-Visit Where are the cables?
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Evolving Data Centre Architecture
Design Factor #3 to Re-Visit How is the compute attached?
How is striping of workload across the physical Data Center accomplished (Rack,
Grouping of Racks, Blade Chassis, )?
How is the increase in percentage of devices attached to SAN/NAS impacting the
aggregated I/O and cabling density per compute unit?
Goal: Define the unit of Compute I/O and how it is managed (how does the cabling
connect the compute to the network and fabric)
blade1
slot
1
blade2
slot
2
blade3
slot
3
blade4
slot
4
blade5
slot
5
blade6
slot
6
blade7
slot
7
blade8
slot
8
blade1
slot
1
blade2
slot
2
blade3
slot
3
blade4
slot
4
blade5
slot
5
blade6
slot
6
blade7
slot
7
blade8
slot
8
blade1
slot
1
blade2
slot
2
blade3
slot
3
blade4
slot
4
blade5
slot
5
blade6
slot
6
blade7
slot
7
blade8
slot
8
10GbE
multiple PCIe
Ethernet
pNIC HBA Still 2 PCI the physical
NIC HBA Addresses Media Eth
FC
Eth FC
Eth
PCIe
1
2
3
4
126
on the BUS
PCI-E Bus
VMFS PCI-E Bus
Edge of the VMFS
PCI-E Bus VETH
SCSI Edge of PCI-E Bus
VETH
SCSI
Fabric the Fabric Edge of
Pass VMFS
Thru SCSI
VNIC
VNIC
the Fabric
Operating VNIC
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Evolving Data Centre Architecture
connected?
iSCSI
iSCSI
NAS
NAS
FCoE
SAN
Appliance
Gateway
Appliance
Gateway
Computer
System
Computer
System
Computer
System
Computer
System
Computer
System
The Flexibility of a
iSCSI
Driver
iSCSI
Driver
TCP/IP
Stack
FCoE
Driver
TCP/IP
Stack
TCP/IP
Stack
TCP/IP
Stack
NIC
NIC
NIC
NIC
NIC
FC Block I/O FC
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Evolving Data Centre Architecture
Design Factor #6 to Re-VisitWhere Are the Services?
Client
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Data Center Core Layer Design
Core Layer Function & Key Considerations
Layer
3
Links
Aggrega3on
Layer
2
Trunks
Access
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Data Center Core Layer Design
Commonly Deployed Platform and Modules
M2-10G LC
Platform: Nexus 7K M2-40G LC M1-10G LC F2-Series LC
M2-100G LC*
Modules
4.0 and 6.0(1) and
Software 6.1 or above
M1: L2/L3/L4 with large forwarding later* later
tables Fabric
240G/200G* 80G 480G*
Connection
and rich feature set L3 IPv4
128K/1M 128K/1M 32K
Unicast
F2: Low-cost, high density with high L3 IPv4
N/A 32K 16K
Multicast
performance, low latency and low power
L3 IPv6
6K/350K Up to 350K 32K
Unicast
Classic layer 3 Core: M1 or F2
L3 IPv6
N/A 16K 8K
Large routing and ACL tables: M1 Multicast
ACL Entries 64/128K 128K 16K
High density linerate10G: F2 MPLS
MPLS: M1 LISP and OTV
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Data Center Aggregation Layer Design
Virtualized Aggregation Layer provides
Enterprise
Network
L2 / L3 boundary
Access layer connectivity point: STP root,
Data
Center
loop-free features Core
Access
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Data Center Aggregation Layer Design
Commonly Deployed Platform and Modules
Performance and MAC Entries 128K 16K (per SOC) 16K (per SOC) 32K
FEX Support Yes* No Yes Yes
port density L2 Portchannel 8 active 16 active 16 active 16 active
LISP and OTV
FabricPath
FCOE Support
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Data Center Aggregation Layer Design
Key Design Considerations
Enterprise
Network
Data Center physical infrastructure
POD design & cabling infrastructure
Data
Center
Size of the layer 2 domain Core
Oversubscription ratio
Traffic flow
No. of access layer switches to aggregate Aggrega3on
Scalability requirement
Service insertion
Service chassis vs. appliance Access
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Data Center Access Layer Design
Access Layer Key Considerations & Commonly Deployed Platform
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Cisco FEXlink: Virtualized Access Switch
Changing the device paradigm
...
Virtualized Switch
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Evolutionary Fabric Edge
Mixed 1/10G, FC/FCoE, Rack and Blade
Consolidation for all servers both rack and blade onto the same virtual switch
Support for 1G, migration to 10G, FC and migration to FCoE
10G server racks are supported by the Support for direct connection of HBA to
1G server racks are supported by 1G FEX (2248TP, addition of 10G FEX (2232PP or 2232TM, Unified Ports on Nexus 5500UP
2224TP) or future proofed with 1/10G FEX 2248PQ)
(2232PP or 2232TM)
1G, 10G and FCoE connectivity for HP or Support for NPV attached blade switches
Dell Blade Chassis during FC to FCoE migration
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Data Center Interconnect Design
Data Center Interconnect Drivers
IP
Routed
DC to DC IP connectivity Main
Data
Center
L3
Service
L3
Backup
Data
Center
L2
L2
GeoCluster
DWDM/
CWDM
Non-disruptive DC migration FC FC
Storage
Storage
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Data Center Interconnect Design
DCI LAN Extension Key Considerations
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Cisco FabricPath
Switching
Rou:ng
Easy
Congura:on
Mul:-pathing
(ECMP)
Plug
&
Play
Fast
Convergence
Provisioning
Flexibility
Highly
Scalable
FabricPath
Blocked Links
Fully Non-Blocking
2:1
FabricPath
Oversubscription 16:1
Pods
4
8:1
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
2
Overlay Transport Virtualization
Technology Pillars
OTV OTV
L2
Nexus
7K
L3
Si Si
L2
OTV OTV OTV OTV
Nexus
7K
Nexus
7K
Nexus
7K
100 MAC 4 IP B
100 MAC 3 Eth 3
MAC 1 MAC 3
6
MAC 1 MAC 3 MAC
1
West East
Site Site MAC
3
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric Simplicity, Scale and Flexibility
Nexus Edge, Core & Boundary Nodes
Isolation of function when possible
Spine provides transport
Nexus Boundary Compute Edge provides media type and scaled control plane
(OTV, LISP, MPLS) Boundary provides localization of complex functions
Nexus
Edge
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Agenda
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Case Study #1
Data Center High Level Requirements
A leading online higher education Customer Business Challenges
institution x10G based virtualized next
More than 500,000 students, 24,000 generation data center architecture
faculty members No STP blocking topology
Approximately 1200 servers and 600 VMs Firewall protection for secured servers
across 5 data centers
Support vMotion within and between
Current data centers reach the limit of data centers
switching, power, and cooling capacity
Network team gains visibility to VM
Business decision made to build two new networking
green field data centers to consolidate and
provide DR capability
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Virtualized Access Layer Requirements
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Data Center Access Layer Design
.
.
.
Nexus
5000/2000
Mixed
ToR
&
EoR
Combina:on
of
EoR
(End
of
Row)
and
ToR
(Top
of
Rack)
cabling
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Access Layer Port Counts & Oversubscription
For 10G server off the 5596s For 1G server off the 5548s
Total 10G ports = 20*32 = 640 Total 1G ports = 20*48 = 960
Server NIC utilization = 50% Server NIC utilization = 50%
Total uplink BW = 16*10 = 160G Total uplinks BW = 8*10 = 80G
Oversubscription ratio = 160/(640*0.5*10) = 1/20 Oversubscription ratio = 80/(960x0.5) = 1/6
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
N1KV Gains Visibility Into VM Environment
Nexus 1000V
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Nexus 1000V Uplink Options
Spanning-Tree (Active/Passive)
Mac Pinning
UCS Blade Server Environment
3rd party blade server environment in non-MCEC Channel-group auto mode
topologies on mac-pinning
Multi-Chassis EtherChannel
Port-channel with two switches Channel-group auto mode
Any server connected to upstream switches that
[active | passive]
supports Multi-Chassis EtherChannel (MCEC)
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Access Layer Design Highlight
Requirement Solution
Flexible cabling N5K/2K provide mixed ToR & EoR
Ease of management Configurations only done on the 5Ks
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Aggregation Requirements
Facility
Drop any server anywhere in the data center
L2-L3
Layer 2 domain within data center
No STP blocking topology
Service Layer
Secured zone and non-secured zone
FW protection between zones, no FW protection within the zone
LB service is required for Web server, server needs to track the client IP
High performance FW and LB are required
NAM and IPS solution are also required
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Physical Infrastructure and Network Topology
Physical to Logical Mapping
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Aggregation Oversubscription Ratio
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Service Integration at Aggregation Layer
Service chassis vs. Appliance
VDC-1
APP
OS
Hypervisor
VDC-2
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Service Integration-Physical Design
Low TCO
6500 repurpose
Most scalable
NAM module inside service chassis
Available slot for future expansion
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Firewall Logical Deployment Model
3.3.3.0 3.3.3.0
Bridging
Router
GW Vlan 30 Vlan 31
1.1.1.0 1.1.1.0
FW
Transparent Mode Router
GW Vlan 10 Vlan 11
Vlan 201
Pros: Non-LB traffic bypass the LB One Arm 3.3.3.0
GW Vlan 31
Cons: SNAT or PBR required
Nexus
1.1.1.0
Vlan 10
GW Vlan 11
GW Vlan 41
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 4.4.4.0 61
Service Integration Logical Design
-
Access
- - L
E E E E E
B B B B B
Layer
2
(STP
+
BPDUguard)
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
vPC Best Practice Features
vPC Peer-Gateway Service continuity Allows a vPC switch to act as the active gateway for packets
addressed to the peer router MAC
vPC orphan-ports Increase High-availability When vPC peer-links go down, vPC secondary shuts down all the vPC
suspend member ports as well as orphan ports. It avoids single attached
devices like FW,LB or NIC teamed device get isolated during vPC
peer-link failure
vPC ARP SYNC Improve Convergence time Improve Convergence for Layer 3 flows after vPC peer-link is UP
vPC Peer-Switch Improve Convergence time Virtualize both vPC peer devices so they appear as a unique STP root
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Aggregation Layer Design Highlight
Requirement Solution
Drop any server anywhere in Single pair of 7Ks provide data center Enterprise
Network
the DC wide layer 2 domain
vMotion within the DC
No STP blocking Topology Double sided vPC between 7K and 5K Data
Center
Core
eliminating blocking ports
FW protection between secure FW virtualization and VDC sandwich
zone and non-secure zone design to provide logical separation and
protection
Layer
3
Links
Aggrega3on
Web servers require load LB in transparent mode provides service Layer
2
Trunks
balancing service per Vlan basis
High throughput services are Mixed of service chassis and appliance
required and future scalability design is able to provide flexible and
scalable service choices Access
Low subscription ratio (target M1 10G line cards configured in
15:1) dedicated mode to provide lower
subscription ratio
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Core Layer Design
Nexus 7010 with Redundant M1 line cards
OSPF as IGP
Data
Center
Inject default into data center Core
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
DCI Requirements and Design Choices
CO
RE
RE
DCI Design Choices
OTV
AG
GR
GR
AG
vPC
ACC
ESS
ACC
ESS
Fabric Path Server
Farms
Server
Farms
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
OTV Design
simplify configuration
minimal latency via dark fiber Data
Center
1
Data
Center
2
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Nexus 1000v Deployment for VM Mobility
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Traffic Flow for VM Mobility
vMotion between Data Centers
Virtual machines still use the original ACE and gateway after vMotion
Traffic will trombone the DCI link for vMotioned virtual machines
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Overall Design Highlight Case Study #1
Requirement Solution
x10G based virtualized generation Nexus 7K,5K,2K provide scalable x10G
data center architecture architecture with end to end virtualization
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Key Takeaways
Nexus family and NX-OS are designed for modern data center
architecture
3 tier design model (Core, Aggregation, Access) ensure high availability &
scalability
Nexus 5K/2K offer flexible cabling solution at Access
Nexus 7K/5K double sided vPC supports non-blocking topology and
larger layer 2 domain. Fabric path is the new trend
Nexus 7K virtualization provides flexible service insertion at Aggregation
OTV/FabricPath/vPC simplify DCI and migration solution
Nexus 1000v provides network policy control & visibility into VM, and
offers integrated virtual services (VSG, vWAAS, NAM, ASA) at VM level
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
THANK
YOU
for
Listening
&
Sharing
Your
Thoughts
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public