6 Data Centre Architecture

Data Center
Networking
Architecture
Presentation_ID © 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1
Before We Get Started:
Put cell phones into silent mode
Intermediate level session focused on data center front end
architecture
This session is based upon the Data Center Infrastructure Design
Guide 2.5.
http://www.cisco.com/application/pdf/en/us/guest/netsol/ns107/c649/cc
migration_09186a008073377d.pdf
Additional Cisco Validated Designs (CVD) can be found at;
http://www.cisco.com/go/cvd
Enterprise Data Center:
http://www.cisco.com/en/US/solutions/ns340/ns414/ns742/ns741/netw
orking_solutions_products_genericcontent0900aecd80601e1d.html#da
tacenter
Agenda
Data Center Infrastructure
Core Layer Design

Aggregation Layer Design
Access Layer Design
Density and Scalability Implications
Scaling Bandwidth and Density
Spanning Tree Design and Scalability
Increasing HA in the DC
Data Center Evolution
Data Center 1.0 Data Center 2.0 Data Center 3.0
Client-Server and Service Oriented and

Mainframe
Distributed Computing Web 2.0 Based
IT Relevance and Control
Consolidate
Virtualize
Automate
Centralized Decentralized Virtualized

Application Architecture Evolution
Application Centric Architecture
Two Sides Of the Same Coin
Data Center Architecture Overview
Layers of the Enterprise Multi-Tier Model
Multi-tier application architecture Enterprise Core

logically overlaid on network
Layer 2 and layer 3 access DC Core
topologies
Dual and single attached DC Aggregation/
Distribution
Servers, Mainframe and Blade
Chassis
Multiple aggregation modules
DC Access
L2 adjacency requirements
Stateful services for security and Blade Chassis w/
load balancing Integrated Switch
Blade Chassis w/ Mainframe

Pass Thru w/ OSA
L2 w/ Clustering L3 Access
and NIC Teaming
Core Layer Design
Core Layer Design Scaling
Requirements
Campus Access Layer
Campus
Distribution
Is a separate DC Core
Layer required?
Consider:
10GigE port density Campus Core
Administrative domains
Anticipate future
requirements
Key core characteristics DC Core
10GE scalability
Aggregation
Distributed forwarding
architecture
Advanced link load balancing
Scalable IP multicast support
Server Farm Access Layer
Scaling
Core Layer Design
L2/L3 Characteristics
Layer 3 Core
Equal cost multi-path (ECMP) load
balancing Campus Core
EIGRP/OSPF for fast convergence
L2 extension through core is
not recommended
DC Core
CEF Hash
CEF* hashing algorithm Applied to
Default hash is on L3 IP Packets on
addresses only Equal Cost
L3 + L4 port hash will improve Routes
L3
load distribution
L2
CORE1(config)#mls ip cef load full simple
Aggregation
Leverages automatic source port
randomization in client TCP stack
Access
Web Application Database

Servers Servers Servers
*CEF = Cisco Express Forwarding
Core Layer Design
Routing Protocol Design: OSPF
Isolate the DC network with a
dedicated OSPF Area.
Not So Stubby Area (NSSA) helps to Campus Core
limit LSA propagation, but permits DC Subnets
route redistribution (RHI) (Summarized)
Area 0
Advertise default into NSSA, NSSA DC Core
summarize routes out L0=10.10.1.1 L0=10.10.2.2
Default Default
Use “auto-cost reference-bandwidth”
to support 10G links
Loopback interfaces simplify L3 vlan-ospf
troubleshooting (neighbor ID)

Aggregation
Use passive-network default: enable
on L3 links to allow peering
L0=10.10.3.3 L0=10.10.4.4
Use authentication: more secure and
avoids undesired adjacencies Access
Interface hello-dead = 1/3

Core Layer Design
Routing Protocol Design: EIGRP
Use summarization and default to

isolate the DC from network events Campus Core
Advertise default into DC with
interface command on core:
ip summary-address eigrp 20 0.0.0.0 0.0.0.0 200 DC Core
Cost of 200 required to prefer Internet default route
over the local NULL0 default route installed by Default Default
summary-address DC Subnets
(Summarized)
If other default routes exist (from L3 vlan-eigrp
internet edge for example), may need Aggregation

to use distribute lists to filter out
Use passive-network default
Summarize DC subnets to core with Access
interface command on agg:
ip summary-address eigrp 20 10.20.0.0 255.255.0.0

Spanning Tree Design
Rapid-PVST+ (802.1w) or
MST (802.1s), Core
Choice of .1w/.1s based on scale

of logical+virtual ports required Root Primary Root Secondary
HSRP Primary HSRP Secondary
R-PVST+ is recommended as Active Context Standby Context
best replacement for 802.1d
Fast converging: inherits Cisco
enhancements (Uplink-fast,
Backbone-fast)
Combined with RootGuard,
BPDUGuard, LoopGuard ,and UDLD
achieves for STP stability
Access layer uplink failures:
~300ms – 2sec
Most flexible design options
UDLD global only enables on Fiber
ports, must enable manually on
copper ports Rootguard
LoopGuard
BPDU Guard
UDLD Global
Integrated Services
Services: Firewall, Load Balancing, SSL
Encryption/Decryption
+
L4-L7 services integrated in Cisco Catalyst® 6500
Server load balancing, firewall and SSL services may be
deployed in:
Active-standby pairs (ACE, FWSM 2.X)
Active-active pairs (ACE, FWSM 3.1)
Integrated blades optimize rack space, cabling, mgmt,
providing flexibility and economies of scale
Influences many aspects of overall design
Active-Standby Service Design
Active-standby services
Application Control Engine
Core
Firewall Service Module
SSL Module Root Primary Root Secondary
HSRP Primary HSRP Secondary
Under utilizes: Active Context Standby Context
Access layer uplinks

Service modules
Aggregation switch fabrics
Advantages:
Widely deployed
Predictable traffic patterns
Easier to configure and manage
Consistent performance under
failure conditions.
Active-Active Service Design
Active-Active Service Design

Application Control Engine
Core
(ACE)
Active-standby distribution VLAN5: VLAN6:
per context Root Primary Root Primary
HSRP Primary HSRP Primary
Active Context Active Context
Firewall Service Module (3.x)
Two active-standby groups
permit distribution of
contexts across two FWSM’s VLAN 6: VLAN 5:
Root Secondary Root Secondary
Permits uplink load balancing HSRP Secondary
Standby Context
HSRP Secondary
Standby Context
while having services applied
Increases overall service
performance vlan6 vlan6
vlan5 vlan6 vlan6 vlan5
Tech Tip: Virtual Context are key to Active/Active designs

Establishing Inbound Path Preference
Establish Route Preference for
Service Enabled Applications
Use Route Health Injection Core
feature of ACE
3. EIGRP/OSFP will
Aligns advertised route propagate RHI VIP to
Core to attract traffic 4. If Context Failover
of VIP with active context on Occurs, RHI and Route

Preference Follow
ACE, FWSM and SSL service

modules
Avoids unnecessary use 2. If Healthy, Installs
of inter-switch link and Host Route to VIP
on Local MSFC
asymmetrical flows
Introduce route-map to
RHI injected route to set
vlan6 vlan6
desired metric
vlan5 vlan6 vlan6 vlan5
1. ACE Probes to
Real Servers in VIP
to Determine Health
Scaling the Aggregation Layer
Aggregation modules provide:

Spanning tree scaling
HSRP Scaling Campus Core
Access layer density
Application services scaling DC Core
SLB/firewall
Fault domain sizing
Core layer provides inter-agg
module transport:
Provides inter-agg module Aggregation Aggregation Aggregation
transport in multi-tier model Module 1 Module 2 Module 3
Low latency distributed forwarding Server Farm Access Layer
(use DFC’s)
100,000’s PPS forwarding rate Scaling
Using Virtual Route Forwarding (VRF)
MPLS or
VRF-Green other Core
Enables virtualization/
partitioning of network VRF-Blue DC Core
resources (MSFC, VRF-Red
ACE, FWSM)
Agg1 Agg2
Permits use of application
services with multiple access
Alternate Primary
topologies Firewall and SLB Contexts on Agg1
Contexts for Green, and 2 to Achieve
Maps well to path isolation Blue, and Red Active-Active Design
MAN/WAN designs such as VLANs Isolate
with MPLS or Multi-VRF Contexts on Access
(VRF-Lite) 802.1Q
Trunks
Security policy management

and deployment by user
group/vrf
Access Layer Design
Access Layer Design
Defining Layer 2 Access
Access Layer connects servers &

hosts to the network
DC Core
Proprietary server protocols
require layer 2 adjacency
L3 Agg1 Agg2
L2 topologies consist of looped Inter-Switch Link
and loop-free models L2

Primary Root Secondary Root
Looped - VLANs are extended across Primary HSRP Secondary HSRP
inter-switch link trunk Active Services Standby Services
nks
Loop-free - VLANs are not extended
T ru
across inter-switch link trunk
.1q
L3 routing is typically performed in
802
the aggregation layer
Stateful services at Agg can be
provided across the L2 access
(FW, SLB, SSL, etc.)
Access Layer Design
Establish a Deterministic Model
Align active components in traffic
path on common Agg switch:
DC Core
Primary STP root L3+L4 Hash
Agg-1(config)#spanning-tree vlan 1-10 root primary Path
Pref
Primary HSRP (outbound) L3 Agg1 Agg2

standby 1 priority X L2
Primary Root
Active Service modules/contexts Primary HSRP
Active Services
nks
Def gwy
Path Preference (inbound)
T ru
.1q
Use RHI & Route map
802
Route-map preferred-path
match ip address x.x.x.x
set metric -30
(also see slide 17)
Access Layer Design
Balancing VLANs on Uplinks: L2 Looped Access
Distributes load across uplinks
STP blocks uplink path for
VLANs to secondary root switch DC Core
L3+L4 Hash
If active/standby service modules
Agg1 Agg2
are used; consider inter-switch L3
link utilization L2
Multiple service modules may be Blue-Primary Root Red-Primary Root
Blue-Primary HSRP Red-Primary HSRP
distributed on both agg switches to Red-Sec Root Blue-Sec Root
achieve balance Red-Sec HSRP Def gwy Def gwy Blue-Sec HSRP
Consider b/w of inter-switch link in a

failure situation
802.1q Trunks
If active/active service modules
are used;
(ACE and FWSM3.1)
Balance contexts+hsrp across agg
switches
Consider establishing inbound path
preference
Access Layer Design
Looped Design Model
VLANs are extended between Primary STP Root Secondary STP Root
aggregation switches, creating Primary HSRP Secondary HSRP
the looped topology Active Services Standby Services
Spanning Tree is used to L3 Inter-Switch Link
prevent actual loops (Rapid L2
PVST+, MST) F F
.1Q Trunk
Redundant path exists through a
F F F F F F F
second path that is blocking
Two looped topology designs:
Triangle and square
FB FB F F
VLANs may be load balanced B
across access layer uplinks
Inter-switch link utilization must
be considered as this may be
used to reach active services Looped Triangle Looped Square
Access Layer Design
L2 Looped Topologies
L2 Triangle L2 Square
L3 L3
L2 L2
VLAN VLAN
10 10
VLAN VLAN VLAN VLAN

10 20 10 20
Row 2, Cabinet 3 Row 9, Cabinet 8 Row 2, Cabinet 3 Row 9, Cabinet 8
Looped Triangle Access Looped Square Access

Supports VLAN extension/L2 adjacency across Supports VLAN extension/L2 adjacency across
access layer access layer
Resiliency achieved with dual homing and STP Resiliency achieved with dual homing and STP
Quick convergence with 802.1W/S Quick convergence with 802.1W/S
Supports stateful services at aggregation layer Supports stateful services at aggregation layer
Proven and widely used Active-active uplinks align well to ACE/FWSM
Achieves higher density access layer,
optimizing 10GE aggregation layer density
Access Layer Design
Benefits of L2 Looped Design
Services like firewall and load
balancing can easily be deployed at
the aggregation layer and shared DC Core
across multiple access layer switches
L3
VLANs are primarily
L2
contained between pairs
of access switches but─
VLANs may be extended
to different access
switches to support
NIC teaming
Clustering L2 adjacency
Administrative reasons
Geographical challenges
Row A, Cab 7 Row K, Cab 3
Access Layer Design
Loop-Free Design
Alternative to looped design
2 LoopFree Models:
U and Inverted U DC Core
Benefit: Spanning Tree L3

is enabled but no port is blocking L2
so all links are forwarding L3
L2
Benefit: less chance of loop condition
due to configuration errors or other
anomalies
L2-L3 boundary varies by
loop-free model used:
U or Inverted-U
Implication considerations with
service modules, L2 adjacency,
and single attached servers
LoopFree U LoopFree
Inverted U
Access Layer Design
Drawbacks of Layer 2 Looped Design
Main drawback: if frame looping occurs, the

network may become unmanageable due to
the infinite replication of frames
802.1w Rapid PVST+ combined with STP
0000.0000.3333
related features and best practices improve
stability and help to prevent loop conditions
UDLD
DST MAC 0000.0000.4444
Loopguard
Rootguard 3/2 3/2
BPDUguard
Limit domain size 3/1 3/1
Stay under STP watermarks for Switch 1 Switch 2
logical and virtual ports DST MAC 0000.0000.4444
Access Layer Design
Loop-Free Topologies
L2 Loop-Free U L2 Loop-Free Inverted U
L3
L2
L3
L2
VLAN
10
VLAN VLAN VLAN VLAN

10 20 10 20
Loop-Free U Access Loop-Free Inverted U Access

VLANs contained in switch pairs Supports VLAN extension
(no extension outside of switch pairs) No STP blocking; all uplinks active
No STP blocking; all uplinks active Access switch uplink failure black holes
Autostate implications for certain service single attached servers
modules (CSM) Supports all service module implementations
ACE supports autostate and per context Single attached servers prone to isolation
failover during failures
Potential for split subnets during failure
Access Layer Design
Loop-Free U Design and Service Modules (1)
Agg1, Active HSRP Agg2, Standby
Services Services
HSRP Secondary on
L3 Agg2 Takes Over as def-
L2 gwy, But How to Reach
Active Service Modules?
MSFC on Agg1
SVI for Interface vlan 10

goes to down state if
uplink is the only
interface in vlan 10
VLAN 10 VLAN 11
VLAN 20, 21
If the uplink connecting access and aggregation goes down, the VLAN interface on the MSFC
goes down as well due to the way autostate works
CSM and FWSM has implications as autostate is not conveyed (leaving black hole)
Tracking and monitoring features may be used to allow failover of service modules based on
uplink failure but would you want a service module failover for one access switch uplink failure?
Not recommended to use loop-free L2 access with active-standby service module
implementations. (See slide on ACE next)
Access Layer Design
Loop-Free U Design and Service Modules (2)
ACE Context 1 ACE Context 1
HSRP
L3
L2 HSRP Secondary on Agg2
Takes over as def-gwy
VLAN 10, 11 VLAN 20, 21

With ACE:
Per context failover with autostate
If uplink fails to Agg1, ACE can switchover to Agg2 (under 1sec)
Requires ACE on access trunk side for autostate failover
May be combined with FWSM3.1 for active-active design
Access Layer Design
Drawbacks of Loop-Free Inverted-U Design
Single attached servers are

black-holed if access switch DC Core
uplink fails
L3
Distributed EtherChannel® can L2
reduce chance of black holing
NIC teaming improves
resiliency in this design
Inter-switch link scaling needs
to be considered when using
active-standby service
modules
Virtual Switching System 1440
Network System Virtualization
Aggregation/Access Server Access
Si
Si Si
Si Si
Si Si
Si
Features Benefits of VSS

Network System Virtualization Increased Operational Efficiency
via Simplified Network
Inter-Chassis Stateful Switch Boost Non-stop Communication
Over (SSO)
Multi-Chassis EtherChannel Scale the System Bandwidth
(MEC) Capacity to 1.4 Tbps
Virtual Switching System
Data Center
A Virtual Switching in the Data Center increases bandwidth scalability, but still provides a
Layer 2 hierarchical architecture without relying on Spanning Tree…
Data Center VSS design guide Summer 2008
Single router node, Fast

convergence, Scalable
architecture
L2/L3 Core
Dual Active Uplinks,

Fast L2 convergence,
minimized L2 Control L2
Plane, Scalable Aggregation*
Dual-Homed Servers,
Single active uplink per
VLAN (PVST), Fast L2
convergence L2 Access
*Service Module support in August 2008.

Access Layer Design
Comparing Looped, Loop-Free and VSS
VLAN Service Single

Uplink vlans
Extension Module Black- Attached Access Switch Must Consider
on Agg Switch
Supported Holing on Server Black- Density per Inter-Switch
in Blocking or
Across Uplink Failure Holing on Agg Module (3) Scaling
Link
Standby State
Access Layer (5) Uplink Failure
Looped
Triangle
- + + + - (3) +
Looped
Square
+ + + + + -
(4)
Loop-Free U + - (4) - + + +
(1,2)
Loop-Free
Inverted U
+ + + (1,2) +/- + -
VSS (6) + + + + + +
1. Use of Distributed EtherChannel Greatly Reduces Chances of Black Holing Condition

2. NIC Teaming Can Eliminate Black Holing Condition
3. When Service Modules Are Used and Active Service Modules Are Aligned to Agg1
4. ACE Module Permits L2 Loopfree Access with per Context Switchover on Uplink failure
5. Applies to when using CSM or FWSM in active/standby arrangement
6. DCVSS design guide to be released in Summer’08
Access Layer Design:
L3 Design Model
Access Layer Design
Defining Layer 3 Access
L3 access switches connect to

aggregation with routed interfaces
DC Core DC Core
All uplinks are active (ECMP), no
spanning tree blocking occurs
.1Q trunks between pairs of L3 DC Aggregation
access switches to support L2
adjacency server requirements
Convergence time is usually better
than Spanning Tree DC Access
L3
Provides isolation/shelter for L2
hosts affected by broadcasts
Access Layer Design
Need L3 for Multicast Sources?
Multicast sources on L2 access

works well with IGMP snooping
IGMP snooping at access switch DC Core DC Core
automatically limits multicast flow
to interfaces with registered
clients in VLAN
DC Aggregation
Use L3 when IGMP snooping is
not available or when particular
L3 administrative functions are
required
DC Access
L3
L2
L3 Access with L3 Access with

Multicast Sources Multicast Sources
Access Layer Design
Benefits of L3 Access
Minimizes broadcast domains

attaining high level of stability
DC Core DC Core
Meet server stability requirements
or isolate particular application
environments
Creates smaller failure domains, DC Aggregation
increasing stability
All uplinks are active paths, no
blocking (up to ECMP maximum) DC Access
Fast uplink convergence: L3
L2
failover and fallback, no arp
table to rebuild for aggregation
switches
Access Layer Design
Drawbacks of Layer 3 Design
L2 adjacency is limited to access

pairs (clustering and NIC teaming
DC Core DC Core
limited)
IP address space management is
more difficult, small subnets. DC Aggregation
If migrating to L3 access, IP
address changes may be difficult
on servers (may break apps)
DC Access
Normally require services to L3
L2
be deployed at access layer
pair to maintain L2 adjacency
with server and provide stateful
failover
Access Layer Design
L2 or L3? What Are My Requirements?
Aggregation Aggregation
L3
OSPF, EIGRP
L2
Rapid PVST+ or MST

L3
Access L2 Access
Layer 2 Layer 3
The Choice of One Design Versus the Other:
Difficulties in managing loops Ability to extend VLANs
Staff skillset; time to resolution Specific application requirements
Broadcast domain sizing
Convergence properties
Oversubscription requirements
NIC teaming; adjacency
Link utilization on uplinks
HA clustering; L2 adjacency Service module support/placement
Access Layer Design:
BladeServers
Blade Server Requirements
Connectivity Options
Using Pass-Through Modules Using Integrated Ethernet Switches
Aggregation
Layer
External L2
Switches
Integrated L2
Switches
Interface 1
Interface 2
Blade Server Chassis Blade Server Chassis
Tech Tip – Avoid hierarchy of L2 switches. Traffic patterns

are difficult to predict during failure conditions.
Blade Server Requirements
Trunk Failover Feature
Switch takes down server Aggregation
interfaces if corresponding uplink
fails, forcing NIC teaming failover
Solves NIC teaming limitations;
prevents black-holing of traffic
Achieves maximum bandwidth
utilization:
No blocking by STP, but STP is Integrated L2
enabled for loop protection switches
Can distribute trunk failover groups

across switches
Dependent upon the NIC feature

set for NIC Teaming/failover Blade Server Chassis
Interface 1
Interface 2
Cisco Virtual Blade Switch (VBS)
Overview of Concept and Benefits
NEW
Management Simplification
– Operational simplification
• Single switch per rack to manage
• True Plug-n-Play of switches
– Design Simplification:
• Sharing Uplinks helps reduce cables
• Reduction in # of logical nodes in L2/L3
network helps improve network convergence
– Operational Consistency
• Familiar IOS CLI, MIBs and management tools
like CiscoWorks
• Consistent End-to-end features and functionality
Performance & Scalability

Traditional Virtual Blade – Up to 160G configurable bandwidth out of rack
Blade Switch Switch
– Rack switch allows server to double bandwidth
with no additional cost
Cisco Catalyst Virtual Blade Switch
Topology Highlighting Key Benefits
Access Layer (Virtual Blade Switch) Aggregation Layer
Mix-n-match
GE & 10GE
switches
Local Traffic
doesn’t go to
distribution
switch
Higher
Resiliency With Catalsyt
with 6500 VSS, all
Etherchannel links utilized
Single Switch / Node Greater Server BW –

(for Spanning Tree or via Active-Active
Layer 3 or Server Connectivity
Management)
Cisco Catalyst Virtual Blade Switch
Multiple Deployment Options
Common Scenario Separate Rings 4 NIC server Scenario

Single Virtual Separate VBS More Server Bandwidth
– VMware
Blade switch More resilient
Creates smaller Rings
* Design Guide recommendations have not been established yet.
Density and Scalability
Implications in the
Data Center
Modular, Top of Rack, Blade Access Switching Models
Where are the issues?

Cabling
Power
Cooling
Spanning Tree Scalability
Management
Oversubscription
Sparing
Redundancy
The right solution is usually
based on business requirements
Hybrid implementations can and
do exist
49
Server Farm Cabinet Layout
~30-40 1RU Servers per Rack
…… N- Racks per Row

……
Considerations:
How Many Interfaces per Server?
Top of Rack, Blade Switches or End of Row?
Separate Switch for Management Network?
Cabling overhead, under floor, patch systems?
Cooling capacity?
Power distribution/ Redundancy?
Density: How Many NICs to Plan For?
Front End Interface
Three to four NICs per server
OOB
are common Management
Front end or public interface Backup Network
Storage HBA or
Storage interface (GE, FC) GE NIC
Backup interface
Back end or private interface Back End Interface
integrated Lights Out (iLO)

for OOB mgmt
Cabling remains in cabinets

May require more than two
TOR switches per rack
30 servers@ 4 ports = 120 ports
required in a single cabinet (3x48 port
1RU switches)
May need hard limits on
cabling capacity
Avoid cross cabinet and other cabling
nightmares
Single Rack-2 Switches Dual Rack-2 Switches
Cabinet Design with Top of Rack Switching
Servers Connect to a Top of Rack (TOR) Switch
Minimizes the number of cables to

run from each cabinet/rack
If NIC teaming support: two - TOR
switches are required
Will two TOR switches provide
enough port density?
Cabling remains in cabinets

Cooling requirements usually do not
permit a full rack of servers
Redundant switch power supply are
option
No Redundant switch processors
GEC or 10GE Uplink Considerations
Single Rack-2 Switches Dual Rack-2 Switches
Top of Rack (TOR) Switching Model
Pro: Efficient Cabling DC Core 4 Uplinks per Cabinet = 100

Pro: Improved Cooling Uplinks Total to Agg Layer
Con: Number of Devices/Mgmt
Con: Spanning Tree Load
Aggregation
GEC or 10GE Uplinks?
Access
Cabinet 1 Cabinet 2
… Cabinet 25
With ~1,000 Servers/25 Cabinets = 50 Switches
Cabinet Design with Modular Access Switches
Servers Connect Directly to a Modular Switch
Cable bulk can be difficult to

manage and block cool air flow
Switch placement at end of row
or within row
Minimizes cabling to Aggregation
Reduces number of
uplinks/aggregation ports
Redundant switch power and
processors are options
GEC or 10GE Uplink
Considerations
NEBS Considerations
Cables route under raised floor or in overhead trays
Access Network Topology w Modular Switches
Pro: Fewer Devices/Mgmt DC Core 2 Uplinks per 6509 = 16

Con: Cabling Challenges Uplinks to Agg Layer
Con: Cooling Challenges
Aggregation
GEC or 10GE Uplinks?
Access
With ~1,000 Servers/9 Slot Access Switches= 8 Switches

~8 Access Switches to Manage
Modular, Top of Rack and Blade Comparison
Lower STP Proc
More I/O Cables
Modular Fewer Uplinks

Access
Aggregation
Top of Rack & Higher STP Proc

Blade Switch
Access
Fewer I/O Cables
More Uplinks
Scaling Bandwidth and Density
Scaling B/W with GEC and 10GE
Optimizing EtherChannel Utilization
Ideal is graph on top right

Bottom left graph more typical
Analyze the traffic flows in and out of
the server farm:
IP addresses (how many?)
L4 port numbers (randomized?)
Default L3 hash may not be optimal

for GEC: L4 hash may improve
agg(config)# port-channel load balance
src-dst-port
10 GigE gives you effectively the full

bandwidth without hash implications
Migrating Access Layer Uplinks to 10GE
How do I migrate from GEC to 10GE uplinks?

How do I increase the 10GE port density at the agg layer?
Is there a way to regain slots used by service modules?
DC Core
Aggregation
Access Pair 1 … …
Consolidate to ACE
DC Core
Consider consolidating multiple

service modules onto ACE Module
–SLB
–Firewall
Aggregation
–SSL
4/8/16G Fabric Connected
Active-Active Designs
Higher CPS + Concurrent CPS
Single TCP termination, lower latency
Firewall feature gap needs Access
consideration
Service Layer Switch
Move certain services out of

aggregation layer DC Core
Ideal for CSM, SSL modules
Opens slots in agg layer for
10GE ports
Service Service
Use separate links for FT Switch1 Aggregation Switch2
paths Si Si
(Redundant)
Extend only necessary L2
VLANs to service switches
via .1Q trunks (GEC/TenG)
Access
Increasing Throughput with
Virtual Switching System 1440
VSS combines a pair of DC Core

Catalyst 6500s into a single
logical switch
Eliminates Spanning-Tree
blocking ports. Service Service
Etherchannel load balancing Switch1 Switch2
(Redundant)
Service module support in
12.2(33)SXI Summer08
(FWSM, ACE, NAM)
Spanning-tree is a protection
mechanism against rogue
switches and cabling errors Access
Scaling with 10GE Density
Nexus 7000
Nexus 7000 provides high

density 10GE scalability. DC Core
Virtual Device Contexts
(VDC) to logically partition a
single chassis.
Multi-Chassis Etherchannel Service Service
Switch1 Aggregation Switch2
support (future)
NX-OS purpose built data Nexus Nexus
center operating system 7000 7000
Unified Fabric capable
Access
Common Questions
How many VLANs can I
support in a single
aggregation module? DC Core
Can a “VLAN Anywhere”
model be supported?
Aggregation
How many access switches
can I support in each
aggregation module?
What are the maximum
number of logical ports?
Are there STP hardware
restrictions?
Access Pair 1 … …
Spanning Tree Protocols Used in the DC
Rapid PVST+ (802.1w)

Most common in data center today
Scales to large size ~10,000 logical ports
Coupled with UDLD, Loopguard, RootGuard and BPDU Guard,
provides a strong-stable L2 design solution
Easy to implement, proven, scales
MST (802.1s)
Permits very large scale STP deployments ~30,000 logical ports
Not as flexible as Rapid PVST+
Service module implications (FWSM transparent mode)
More common in service providers and ASPs
This CVD focuses on the Use of Rapid PVST+

Spanning Tree Protocol Scaling
MST RPVST+ PVST+
Total Active STP

50,000 Total 10,000 Total 13,000 Total
Logical Interfaces
1,8001 per
Switching
Total Virtual Ports 6,0001 per Module(6700) 1,8001 per
per LineCard Switching Module Switching Module
1200 for Earlier
Modules
1 10 Mbps, 10/100 Mbps, and 100 Mbps Switching Modules Support a Maximum of 1,200
Logical Interfaces per Module
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/122sx/ol_4164.htm#wp26366
Number of Total STP Active
Logical Interfaces= DC Core
Trunks on the switch * active
VLANs on the trunks + number
of non-trunking interfaces on the switch
30 VLANs
In this example, aggregation 1 will have:
10 + 20 + 30 = 60 STP active logical interfaces
Te7/4
AGG1#sh spann summ tot
Switch is in rapid-pvst mode Te7/3
Root bridge for: VLAN0010, VLAN0020, VLAN0030
EtherChannel misconfig guard is enabled
Extended system ID is enabled
Portfast Default is disabled
PortFast BPDU Guard Default is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default is enabled
UplinkFast is disabled 10 VLANs 20 VLANs
BackboneFast is disabled
Pathcost method used is long
Name Blocking Listening Learning Forwarding STP Active

---------------------- -------- --------- -------- ---------- ----------
30 VLANs 0 0 0 60 60
AGG1#
STP Active Column = STP Total Active Logical Interfaces
Example: Calculating Total Active
Logical Ports
120 VLANs system wide
No manual pruning performed on Core
trunks
1RU access layer environment Layer 3
Layer 2
45 access switches each Aggregation 1 Aggregation 2
connected with 4GEC Primary Root Loop Secondary Root
Dual homed, loop topology

(120 * 45 access links)+120
instances on link to
agg2=5400+120=5520 Access 1 ..… Access 45
This is under the maximum
recommendation of 10,000 when
using Rapid PVST+
Number of Virtual Ports per
Line Card= DC Core
For line card x: sum of all trunks * VLANs *
(the number of ports in a port-channel if used)
10 + 20 + (30*2) 30 VLANs
=90 Virtual Port’s on line card 7 Te7/1
Te7/2
Te7/4
AGG1#sh vlan virtual-port slot 7
Slot 7 Te7/3
Port Virtual-ports
-------------------------
Te7/1 30 EtherChannel
Te7/2 30 10 VLANs 20 VLANs
Te7/3 10
Te7/4 20
Total virtual ports:90
AGG1#
NOTE: VPs Are Calculated per Port in Channel Groups
Example: Calculating Virtual Ports per
Line Card
Core
120 VLANs system wide

no manual pruning performed on
Layer 3
trunks Layer 2
Aggregation 2
12 access switches, each Aggregation 1
Primary Root loop Secondary Root
connected with 4GEC across

6700 line card 6748 Line Card
Acc1Acc2 …. Acc12
Access 1 ..… Access 12
EtherChannels to Acces Layer
(120 * 48 access links)

=5,760 virtual ports
This is above the recommended Maximum number VLANs per port = 37
1800/48=37.5 per port
watermark
Why STP Watermarks Are Important
Watermarks Are Not Hard Limits But─

If exceeded, performance is unpredictable
Larger impact when interface flaps, or shut/no_shut
Small networks may not see a problem
Large networks will usually see problems
Convergence time will be affected
Pruning off unneeded VLANs will reduce active
logical and virtual port instances
Design Guidelines
Add aggregation modules to

scale, dividing up the STP DC Core
domain
Maximum five hundred HSRP
instances on Sup720 (depends
on other cpu driven processes)
If logical/virtual ports near upper Aggregation
Module 1
limits perform:
–Manual pruning on trunks
–Add aggregation modules
–Use MST if necessary
Access
Increasing HA in the Data Center
Server High Availability
Common Points of Failure
1. Server network adapter
2. Port on a multi-port
server adapter
L3
L2 3. Network media
(server access)
4. Network media (uplink)
5. Access switch port
6. Access switch module
7. Access switch
These Network Failure Issues Can Be

Without Data Center HA With Data Center HA Addressed by Deployment of Dual
Recommendations Recommendations Attached Servers Using Network Adapter
Teaming Software
Common NIC Teaming Configurations
AFT—Adapter Fault Tolerance SFT—Switch Fault Tolerance ALB—Adaptive Load Balancing
Default GW Default GW Default GW

10.2.1.1 10.2.1.1 10.2.1.1
HSRP HSRP HSRP
Heartbeats
Heartbeats
Heartbeats
Eth0: Active Eth1: Standby Eth0: Active Eth1: Standby Eth0: Active Eth1-X: Active
IP=10.2.1.14 IP=10.2.1.14
IP=10.2.1.14 IP=10.2.1.14
MAC =0007.e910.ce0f MAC =0007.e910.ce0e
MAC =0007.e910.ce0f MAC =0007.e910.ce0f
One Port Receives, All Ports Transmit
On Failover, Src MAC Eth1 = Src MAC Eth0 On Failover, Src MAC Eth1 = Src MAC Eth0
Incorporates Fault Tolerance
IP Address Eth1 = IP Address Eth0 IP Address Eth1 = IP Address Eth0
One IP Address and Multiple MAC Addresses
Note: NIC manufacturer drivers are changing and may

operate differently. Also, server OS have started integrating
NIC teaming drivers which may operate differently.
Server Attachment: Multiple NICs
You Can Bundle Multiple Links to Allow
Generating Higher Throughputs Between
Servers and Clients
L3
L2
EtherChannel Hash May
Not Permit Full Utilization
for Certain Applications
(Backup Example)
Only One Link Active:

Fault Tolerant Mode
All Links Active: EtherChannels

Load Balancing
Failover: What Is the Time to Beat?
The overall failover time is the

combination of convergence
at L2, L3, + L4 components
Stateful devices can replicate connection
information and typically failover within 3-5sec
EtherChannels < 1sec
STP converges in ~1 sec (802.1w)
HSRP can be tuned to <1s
Where does TCP break? Microsoft,

Linux, AIX, etc.
Linux and
Microsoft XP Others Tolerate
2003 Server a Longer
Failover Time
TCP Stack Outage
L4 Convergence Tolerance
~ 5s ~ 9s
L3 Convergence L2 Convergence
Failover Time Comparison
STP-802.1w—One sec
OSPF-EIGRP—Sub sec
ACE Module with Autostate
HSRP—Three sec (using 1/3)
FWSM Module—Three sec
CSM Module—Five sec
WinXP/2003 ServerTCP Stack—Nine sec
Failover Time
TCP Stack
Tolerance
Content ~ 9s
Service
Module
HSRP FireWall ~ 5s
~ 3s Service
(may be tuned Module
Spanning Tree ACE to less) ~ 3s
OSPF/EIGRP ~ 1s
~1sec
Sub-second
Non-Stop Forwarding/Stateful Switch-Over
NSF/SSO is a redundancy NSF-Aware NSF-Aware

mechanism for supervisor failover
SSO synchronizes layer 2 protocol
state, hardware L2/L3 tables (MAC,
FIB, adjacency table), ACL and QoS
tables
SSO synchronizes state for:
trunks, interfaces, EtherChannels,
port security, SPAN/RSPAN, STP,
UDLD, VTP
NSF with EIGRP, OSPF, IS-IS, BGP
makes it possible to have no route
flapping during the recovery
Aggressive EIGRP/OSPF timers do
not work in NSF/SSO environment
NSF/SSO in the Data Center
Redundant Supervisors with SSO in the
access layer:
Improves availability for single attached
servers
Redundant Supervisors with SSO in the
aggregation layer:
Consider in primary agg layer switch
Prevents service module switchover
(up to ~5sec depending on module)
SSO switchover time less than two sec
12.2.18SXD3 or higher
Possible implications
HSRP state between Agg switches is
not tracked and will show switchover until
control plane recovers
IGP Timers cannot be aggressive (tradeoff)
Best Practices: STP, HSRP, Other
Rapid PVST+
UDLD Global
Spanning Tree Pathcost Method=Long
LACP+L4 Port Hash

L3+ L4 CEF Hash Dist EtherChannel for FT
and Data VLANs
Agg1: FT
Agg2:
STP Primary Root STP Secondary Root
HSRP Primary Data
HSRP Secondary
HSRP Preempt and Delay HSRP Preempt and Delay
Dual Sup with NSF+SSO Single Sup
LACP+L4 Hash
Dist EtherChannel
Min-Links
Rootguard
LoopGuard Blade Chassis
Portfast + with Integrated
BPDUguard Switch
UDLD Global Rapid PVST+: Maximum Number of STP Active Logical

Ports- 8000 and Virtual Ports Per Linecard-1500
Q and A
Recommended Reading
Continue your Networkers at
Cisco Live learning
experience with further
reading from Cisco Press
Check the Recommended
Reading flyer for suggested
books
Visit www.cisco.com/go/cvd
for Design Guides and
Whitepapers about all of
these subjects and more
Available Onsite at the Cisco Company Store


6 Data Centre Architecture

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

6 Data Centre Architecture

Uploaded by

Copyright:

Available Formats

Data Center

 Core Layer Design

Data Center 1.0 Data Center 2.0 Data Center 3.0

Client-Server and Service Oriented and

Centralized Decentralized Virtualized

 Multi-tier application architecture Enterprise Core

Blade Chassis w/ Mainframe

Web Application Database

troubleshooting (neighbor ID)

Web Application Database

 Use summarization and default to

internet edge for example), may need Aggregation

Web Application Database

 Choice of .1w/.1s based on scale

Access layer uplinks

 Active-Active Service Design

vlan5 vlan6 vlan6 vlan5

Tech Tip: Virtual Context are key to Active/Active designs

of VIP with active context on Occurs, RHI and Route

ACE, FWSM and SSL service

 Aggregation modules provide:

 Security policy management

 Access Layer connects servers &

and loop-free models L2

Primary HSRP (outbound) L3 Agg1 Agg2

Path Preference (inbound)

Consider b/w of inter-switch link in a

VLAN VLAN VLAN VLAN

Row 2, Cabinet 3 Row 9, Cabinet 8 Row 2, Cabinet 3 Row 9, Cabinet 8

Looped Triangle Access Looped Square Access

Row A, Cab 7 Row K, Cab 3

 Benefit: Spanning Tree L3

 Main drawback: if frame looping occurs, the

VLAN VLAN VLAN VLAN

Loop-Free U Access Loop-Free Inverted U Access

SVI for Interface vlan 10

VLAN 10, 11 VLAN 20, 21

 Single attached servers are

Features Benefits of VSS

Data Center VSS design guide Summer 2008

Single router node, Fast

Dual Active Uplinks,

*Service Module support in August 2008.

VLAN Service Single

1. Use of Distributed EtherChannel Greatly Reduces Chances of Black Holing Condition

 L3 access switches connect to

 Multicast sources on L2 access

L3 Access with L3 Access with

 Minimizes broadcast domains

 L2 adjacency is limited to access

Rapid PVST+ or MST

Using Pass-Through Modules Using Integrated Ethernet Switches

Tech Tip – Avoid hierarchy of L2 switches. Traffic patterns

Can distribute trunk failover groups

 Dependent upon the NIC feature

 Performance & Scalability

Single Switch / Node Greater Server BW –

Common Scenario Separate Rings 4 NIC server Scenario

 Where are the issues?

…… N- Racks per Row

integrated Lights Out (iLO)

Cabling remains in cabinets

 Minimizes the number of cables to

Cabling remains in cabinets

Core Layer Design

Multi-tier application architecture Enterprise Core

Use summarization and default to

Choice of .1w/.1s based on scale

Active-Active Service Design

Aggregation modules provide:

Security policy management

Access Layer connects servers &

Benefit: Spanning Tree L3

Main drawback: if frame looping occurs, the

Single attached servers are

L3 access switches connect to

Multicast sources on L2 access

Minimizes broadcast domains

L2 adjacency is limited to access

Dependent upon the NIC feature

Performance & Scalability

Where are the issues?

Minimizes the number of cables to

Cable bulk can be difficult to

Ideal is graph on top right

Default L3 hash may not be optimal

10 GigE gives you effectively the full

How do I migrate from GEC to 10GE uplinks?

Consider consolidating multiple

Move certain services out of

VSS combines a pair of DC Core

Nexus 7000 provides high

Unified Fabric capable

Rapid PVST+ (802.1w)

Dual homed, loop topology

120 VLANs system wide

Add aggregation modules to

The overall failover time is the

Where does TCP break? Microsoft,

NSF/SSO is a redundancy NSF-Aware NSF-Aware