RST-3009

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

1

Cisco Nexus 7000 Switch Architecture

RST-3009

RST-3009

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

2

© 2008, Cisco Systems, Inc. All rights reserved. RST-3009

Introduction to Cisco Nexus 7000 Series
10G Core Performance 10G Aggregation Density Access 1G/10G to the Host Data Centre Ethernet (DCE) (future)
LAN SAN
Nexus 7000 Agg Layer Nexus 7000 Agg Layer

Nexus 7000

Core Layer

Unified Fabric

IPC

Blade Servers Top of Rack

Nexus 7000

Access Layer

High performance, highly available 10GE core connectivity

Full featured 10G density for aggregating 10G top of rack and 10G blade servers

As virtualisation drives host I/O utilisation, 10G to the host requirements are becoming reality

Enables new Ethernet capabilities such as lossless Ethernet, L2 multipathing, and FCoE

RST-3009

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

3

Session Goal
To provide you with a thorough understanding of the Cisco Nexus 7000 switching architecture, I/O module design, packet flows, and key forwarding engine functions.
This session will NOT examine Unified I/O, FCoE, DCE, Nexus 5000, or the NX-OS software architecture. Related sessions: RST-2017: NX-OS Software Architecture DCT-2012: Fibre Channel over Ethernet: First step to Unified Fabric & Introducing Nexus 5000 Architecture DCT-2007: Evolution of Ethernet in the Data Centre

RST-3009

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

4 4

© 2008, Cisco Systems, Inc. All rights reserved. RST-3009

Agenda
 Chassis Architecture
 Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow

RST-3009

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

5

Nexus 7010 Chassis
Integrated cable management with cover Optional locking front doors Locking ejector levers Supervisor slots (5-6) Payload slots (1-4, 7-10)

System status LEDs

ID LEDs on all FRUs

Front-toback airflow

Air exhaust

System fan trays Fabric fan trays
21RU

Two chassis per 7’ rack Crossbar fabric modules

Power supplies Air intake with optional filter
RST-3009

Front © 2008 Cisco Systems, Inc. All rights reserved.

N7K-C7010
Cisco Public

Rear

Common equipment removes from rear
6

© 2008, Cisco Systems, Inc. All rights reserved. RST-3009

Power and Cooling
 6000W AC power supply for Nexus 7000 series chassis  Dual inputs at 220/240V or 110/120V  Proportional load-sharing among supplies  Variable speed redundant fans provide system cooling  Redundant system fan trays provide cooling of I/O modules and supervisor engines  Redundant fabric fans provide cooling of crossbar fabric modules

Fabric Fan Tray N7K-C7010-FAN-F N7K-AC-6.0KW 6000W AC Power Supply
RST-3009 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

System Fan Tray N7K-C7010-FAN-S
7

N+1 redundancy Grid redundancy

Power Redundancy
Power redundancy mode dictates how system budgets power:  N+1 redundancy – Reserves capacity equal to sum of lowest two power supplies (default)  Grid/input source redundancy – Reserves capacity equal to sum of half capacity of each power supply

Available Power

12kW 18kW 9kW

 Note: power budget and actual power draw are typically not equal! Actual draw ~30% lower than budget under normal operating conditions.

220V

220V

Grid 1
RST-3009 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Grid 2
8

© 2008, Cisco Systems, Inc. All rights reserved. RST-3009

Agenda
 Chassis Architecture

 Supervisor Engine Architecture
 I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow

RST-3009

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

9

Supervisor Engine
 Performs control plane and management functions  Dual-core 1.66GHz Intel Xeon processor with 4GB DRAM  2MB NVRAM, 2GB internal bootdisk, 2 external compact flash slots  Out-of-band 10/100/1000 management interface  Connectivity Management Processor (CMP)
Always-on Ethernet connectivity for lights-out management

 Console & auxiliary serial ports  USB ports for file transfer
N7K-SUP1

ID LED Status LEDs
RST-3009

AUX Port Console Port Management Ethernet
Cisco Public

USB Ports Compact Flash Slots Reset Button

CMP Ethernet

© 2008 Cisco Systems, Inc. All rights reserved.

10

© 2008, Cisco Systems, Inc. All rights reserved. RST-3009

Management Interfaces  Management Ethernet 10/100/1000 interface used exclusively for system management Belongs to dedicated “management” VRF Prevents data plane traffic from entering/exiting from mgmt0 interface Cannot move mgmt0 interface to another VRF Cannot assign other system ports to management VRF  Connectivity Management Processor (CMP) Ethernet Connects to standalone. Cisco Systems.66GHz Dual-Core PHY 10/100/1000 10/100/1000 Console AUX Mgmt Enet usb usb usb CMP Enet log-flash: RST-3009 © 2008 Cisco Systems. All rights reserved. always-on microprocessor on supervisor engine Runs lightweight Linux kernel and network stack Completely independent of DC-OS on main CPU Provides ‘lights out’ remote management and disaster recovery via 10/100/1000 interface Removes need for terminal servers RST-3009 © 2008 Cisco Systems. RST-3009 . All rights reserved. Inc. All rights reserved. Inc. Cisco Public 12 © 2008. Inc. Cisco Public 11 Supervisor Engine Architecture To Modules To Fabrics To Modules n * 23G Switched Gigabit Ethernet Arbitration Path Fabric ASIC Fabric Interface and VOQ 1GE Inband Arbitration Path Switched EOBC 1GE EOBC Central Arbiter 128MB 16MB DRAM Flash System Controller 4GB CMP Main CPU Security Processor 266MHz DRAM Link Encryption 2GB 2MB PHY Internal CF NVRAM OBFL Flash slot0: 1.

RST-3009 . Inc. Inc. Cisco Systems. All rights reserved. Inc.1AE LinkSec N7K-M132XP-12 RST-3009 © 2008 Cisco Systems.Agenda  Chassis Architecture  Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow RST-3009 © 2008 Cisco Systems. All rights reserved. All rights reserved. Cisco Public 14 © 2008. Cisco Public 13 32-Port 10GE I/O Module  32 10GE ports  SFP+ transceivers  80G full-duplex fabric connectivity  Integrated forwarding engine  4:1 port-level oversubscription  Virtual output queuing (VOQ)  802.

7) To fabric rate-mode dedicated 10G Dedicated mode 9 11 13 15  One interface in port group gets 10G bandwidth  Other three interfaces in port group disabled RST-3009 © 2008 Cisco Systems.19.27.3.4. Cisco Systems.30..g.32-Port 10GE I/O Module Architecture EOBC To Central Arbiter To Fabrics n * 46G Fabric ASIC LC CPU Fabric Interface and VOQ Inband (to Port ASIC) 23G 23G FE Daughter Card Layer 3 Engine Layer 2 Engine Fabric Interface and VOQ (to LC CPU) 23G 23G Replication MET Engine Replication MET Engine Replication MET Engine Replication MET Engine Port ASIC 10G CTS and 4:1 Mux 2.22.5. Inc.8 Port ASIC 10G CTS and 4:1 Mux 10.14.12.24 Port ASIC 10G CTS and 4:1 Mux 26. RST-3009 . Inc. All rights reserved.29. All rights reserved. Cisco Public 15 Shared versus Dedicated Mode To fabric rate-mode shared (default) 9 10G 11 13 15 Shared mode  Four interfaces in port group share 10G bandwidth “Port group” — group of contiguous even or odd ports that share 10G of bandwidth (e.7 Port ASIC 10G CTS and 4:1 Mux 9.21.11.5. ports 1.3.28.20. Cisco Public 16 © 2008. All rights reserved.31 Mezzanine Card RST-3009 © 2008 Cisco Systems.13.15 Port ASIC 10G CTS and 4:1 Mux 17.23 Port ASIC 10G CTS and 4:1 Mux 25.32 Port ASIC 10G CTS and 4:1 Mux 1.6. Inc.16 Port ASIC 10G CTS and 4:1 Mux 18.

Inc.1AE LinkSec N7K-M148GT-11 RST-3009 © 2008 Cisco Systems. All rights reserved. Cisco Systems. Inc. RST-3009 . All rights reserved. Cisco Public 18 © 2008. Inc.48-Port 10/100/1000 I/O Module  48 10/100/1000 RJ-45 ports  40G full duplex fabric connectivity  Integrated forwarding engine  Virtual output queuing (VOQ)  802. Cisco Public 17 48-Port 10/100/1000 I/O Module Architecture EOBC To Fabrics n * 46G To Central Arbiter LC CPU Inband (to Port ASIC) 23G FE Daughter Card Fabric ASIC Fabric Interface and VOQ Layer 3 Engine Layer 2 Engine 23G (to LC CPU) Replication MET Engine Replication MET Engine Port ASIC 12G Port ASIC 12G 12G Port ASIC 12G Port ASIC CTS 1-4 CTS 5-8 CTS 9-12 CTS 25-28 CTS 29-32 CTS 33-36 CTS 13-16 CTS 17-20 CTS 21-24 CTS 37-40 CTS 41-44 CTS 45-48 RST-3009 © 2008 Cisco Systems. All rights reserved.

Cisco Systems. RST-3009 . Inc. bidir)  RACL/VACL/PACLs  Cisco TrustSec security group tag support  Unicast RPF check and IP source guard  QoS remarking and policing policies  Ingress and egress NetFlow (full and sampled)  GRE tunnels Table sizes optimised for Data Centre FIB TCAM MAC table Classification TCAM (ACL and QoS) NetFlow Table RST-3009 © 2008 Cisco Systems. All rights reserved. Cisco Public 19 Forwarding Engine Hardware Hardware forwarding engine integrated on every I/O module  60Mpps Layer 2 bridging with hardware MAC learning  60Mpps IPv4 and 30Mpps IPv6 unicast  IPv4 and IPv6 multicast (SM. All rights reserved.Agenda  Chassis Architecture  Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow RST-3009 © 2008 Cisco Systems. SSM. Inc. All rights reserved. Inc. Cisco Public 128K 128K 64K 512K 20 © 2008.

Inc.Forwarding Engine Architecture Forwarding engine chipset consists of two ASICs:  Layer 2 Engine Ingress and egress SMAC/DMAC lookups Hardware MAC learning IGMP snooping and IP-based Layer 2 multicast constraint  Layer 3 Engine IPv4/IPv6 Layer 3/Layer 4 lookups FIB. Inc. Cisco Public © 2008. Cisco Public 21 Forwarding Engine Pipelined Architecture FE Daughter Card Ingress Pipeline  Ingress NetFlow collection Egress Pipeline  Ingress ACL and QoS classification lookups Layer 3 Engine  Egress policing  FIB TCAM and adjacency table lookups for Layer 3 forwarding  ECMP hashing  Multicast RPF check  Ingress policing  Egress NetFlow collection  Unicast RPF check  Ingress MAC table lookups  IGMP snooping lookups  IGMP snooping redirection  Egress ACL and QoS classification lookups Layer 2 Engine  Egress MAC lookups  IGMP snooping lookups Final lookup result to I/O Module Replication Engine 22 Packet Headers from I/O Module Replication Engine RST-3009 © 2008 Cisco Systems. QoS. Inc. All rights reserved. RST-3009 . pipelined architecture RST-3009 © 2008 Cisco Systems. ACL. All rights reserved. Cisco Systems. All rights reserved. NetFlow processing Linear.

Cisco Public 24 © 2008. All rights reserved.Forwarding Engine Details  Every packet subjected to both ingress and egress pipeline in forwarding engine Enabling features does not affect forwarding engine performance  Forwarding engine on INGRESS I/O module performs lookups for both ingress interface/VLAN and egress interface/VLAN  However. Inc. Cisco Public 23 Agenda  Chassis Architecture  Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow RST-3009 © 2008 Cisco Systems. Cisco Systems. RST-3009 . Inc. forwarding engine on EGRESS I/O module also performs lookups: Layer 2-only lookup to ensure current MAC table information Layer 2/3/4 lookups for multicast egress replicated packets RST-3009 © 2008 Cisco Systems. All rights reserved. All rights reserved. Inc.

Cisco Systems.Fabric Module  Nexus 7000 implements multistage crossbar switch fabric  Each fabric module provides 46Gbps per I/O module slot Up to 230Gbps per slot with 5 fabric modules  Initially shipping I/O modules do not leverage full fabric bandwidth Maximum 80G per slot with 10G module  Traffic load-sharing across all active fabric modules  Access to fabric controlled using QoS-aware central arbitration with VOQ N7K-C7010-FAB-1 RST-3009 © 2008 Cisco Systems. RST-3009 . Cisco Public 25 Multistage Crossbar Switch Fabric  Three-stage architecture crossbar architecture  Fabric modules form 2nd stage of switch fabric Ingress I/O Module Fabric Interface and VOQ Fabric Modules Crossbar Fabric ASIC 1 Crossbar Fabric ASIC 2 Egress I/O Module Fabric Interface and VOQ Traffic Flow Fabric Interface and VOQ Crossbar Fabric ASIC Crossbar Fabric ASIC 3 Crossbar Fabric ASIC Fabric Interface and VOQ Crossbar Fabric ASIC 4 Crossbar Fabric ASIC 5 3rd Stage Crossbar (Egress I/O module) Egress Fabric Interface RST-3009 Stage Crossbar Ingress (Ingress Fabric I/O Interface © 2008 Cisco Systems. All rights reserved. All rights reserved. module) Cisco Public 1st 2nd Stage Crossbar (Fabric modules) 26 © 2008. Inc. Inc. All rights reserved. Inc.

Cisco Systems. RST-3009 . Cisco Public 27 Fabric Modules I/O Module Capacity 46Gbps/slot 46Gbps/slot Crossbar Fabric ASIC 1 Crossbar Fabric ASIC 2 10G module  Requires 2 fabrics for full bandwidth  Requires 3 fabrics for N+1 redundancy 80G 40G 46Gbps/slot Crossbar Fabric ASIC 3 230Gbps 184Gbps 138Gbps 92Gbps 46Gbps per slot bandwidth  4th and 5th fabric modules provide additional redundancy and future-proofing 46Gbps/slot Crossbar Fabric ASIC 4 46Gbps/slot Crossbar Fabric ASIC 5 10/100/1000 module  Requires 1 fabric for full bandwidth  Requires 2 fabrics for N+1 redundancy RST-3009 © 2008 Cisco Systems. All rights reserved. Cisco Public 28 © 2008. Inc.Fabric Module Capacity 46Gbps/slot 2 x 23G channels per I/O module slot 1 x 23G channel per supervisor slot 46Gbps/slot Fabric Modules Crossbar Fabric ASIC 1 Crossbar Fabric ASIC 2 46Gbps/slot Crossbar Fabric ASIC 3 230Gbps 184Gbps 138Gbps 92Gbps 46Gbps per slot bandwidth 46Gbps/slot Crossbar Fabric ASIC 4 46Gbps/slot Crossbar Fabric ASIC 5 RST-3009 © 2008 Cisco Systems. All rights reserved. All rights reserved. Inc. Inc.

Cisco Public 5 30 © 2008. All rights reserved. Cisco Systems. All rights reserved. Inc. Inc. RST-3009 . Inc. All rights reserved. Cisco Public 29 Fabric Load-Sharing  Ingress fabric interface ASIC knows all active paths through 3-stage crossbar to each destination  Unicast – Pseudo round-robin traffic distribution across all active paths to egress fabric interface ASIC  Multicast – Selects one of the active paths to egress fabric interface ASIC for the packet based on hash algorithm Fabric Modules Crossbar Fabric ASIC 1 Ingress I/O Module Fabric Interface and VOQ Crossbar Fabric ASIC 2 Egress I/O Module Fabric Interface and VOQ Ingress port Fabric Interface and VOQ Crossbar Fabric ASIC Crossbar Fabric ASIC 3 Crossbar Fabric ASIC Fabric Interface and VOQ Egress port Crossbar Fabric ASIC 2 possible paths Crossbar Fabric ASIC 4 2 possible paths 2 possible paths 10 possible paths RST-3009 © 2008 Cisco Systems.Fabric Module Redundancy 46Gbps/slot Crossbar Fabric ASIC 1  Fabric removal or failure results in reduction of overall system bandwidth 46Gbps/slot Crossbar Fabric ASIC 2 80G 40G 46Gbps/slot Crossbar Fabric ASIC 3 230Gbps 184Gbps 138Gbps 92Gbps 46Gbps per slot bandwidth 46Gbps/slot Crossbar Fabric ASIC 4 46Gbps/slot Crossbar Fabric ASIC 10G module 5 10/100/1000 module RST-3009 © 2008 Cisco Systems.

RST-3009 . Inc. Cisco Systems. Cisco Public 31 What Are VOQs?  Virtual Output Queues (VOQs) on ingress modules represent bandwidth capacity on egress modules  Guaranteed delivery across fabric for arbitrated packets If VOQ available on ingress. Inc. All rights reserved. Cisco Public 32 © 2008. All rights reserved. Inc.Access to Fabric Bandwidth  Access to fabric controlled using central arbitration Arbiter ASIC on supervisor engine provides fabric arbitration  Egress module bandwidth represented by Virtual Output Queues (VOQs) at ingress to fabric RST-3009 © 2008 Cisco Systems. capacity exists at egress  VOQ is NOT equivalent to ingress or egress port buffer or queues Relates ONLY to ASICs at ingress and egress to fabric  VOQ is “virtual” because it represents EGRESS capacity but resides on INGRESS modules It is PHYSICAL buffer where packets are stored RST-3009 © 2008 Cisco Systems. All rights reserved.

even with traffic sources on different modules  Prevents congested egress ports from blocking ingress traffic destined to other ports Mitigates head-of-line blocking by providing dedicated buffer for individual destinations across the fabric  In future.Benefits of Central Arbitration with VOQ  Ensures priority traffic takes precedence over best-effort traffic across fabric Four levels of priority for each VOQ destination  Ensures fair access to bandwidth for multiple ingress ports transmitting to one egress port Central arbiter ensures all traffic sources get appropriate access to fabric bandwidth. Inc. will provide lossless service for FCoE traffic across the fabric Can provide strict priority and backpressure (blocking instead of dropping) for certain traffic classes. All rights reserved. All rights reserved. Inc. RST-3009 . Cisco Public 10/100/1000 I/O module 34 © 2008. such as SAN traffic RST-3009 © 2008 Cisco Systems. All rights reserved. Inc. Cisco Systems. each module has corresponding VOQ with four priority levels  One VOQ with four priority levels serves one of the following “VOQ destinations” on an egress module: One front-panel 10G port (dedicated mode) -orFour front-panel 10G ports (shared mode) -orTwelve front-panel 10/100/1000 ports 10G I/O module RST-3009 © 2008 Cisco Systems. Cisco Public 33 VOQ Destinations  For every “destination” on other modules in system.

Inc. All rights reserved. Cisco Public 36 © 2008. Cisco Public 35 Layer 2 Forwarding  MAC table is 128K entries (115K effective)  Hardware MAC learning CPU not directly involved in learning  All modules have copy of MAC table New learns communicated to other modules via hardware “flood to fabric” mechanism Software process ensures continuous MAC table sync  Spanning tree (PVRST or MST) ensures loop-free Layer 2 topology RST-3009 © 2008 Cisco Systems. All rights reserved.Agenda  Chassis Architecture  Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow RST-3009 © 2008 Cisco Systems. RST-3009 . Inc. Cisco Systems. Inc. All rights reserved.

All rights reserved. RST-3009 .6.. Cisco Public Hardware MAC Table Hardware MAC Learning 37 Hardware Layer 2 Forwarding Process  MAC table lookup in Layer 2 Engine based on {VLAN.Layer 2 Forwarding Architecture  Layer 2 Forwarding Manager (L2FM) maintains central database of MAC tables  L2FM keeps MAC table on all forwarding engines in sync  L2FM-Client process on I/O modules interfaces between L2FM and hardware MAC table L2FM-C Hardware I/O Module Supervisor Engine L2FM L2FM-C Hardware I/O Module L2FM-C Hardware I/O Module n7010# sh processes cpu | egrep PID|l2fm PID Runtime(ms) Invoked uSecs 1Sec Process 3848 1106 743970580 0 0 l2fm n7010# attach mod 9 Attaching to module 9 .' Last login: Mon Apr 21 15:58:12 2008 from sup02 on pts/0 Linux lc9 2.MAC} pairs  Source MAC and destination MAC lookups performed for each frame Source MAC lookup drives new learns and refreshes aging timers Destination MAC lookup dictates outgoing switchport RST-3009 © 2008 Cisco Systems.0 l2fmc module-9# RST-3009 © 2008 Cisco Systems.. Cisco Public 38 © 2008. Cisco Systems. Inc. Inc. All rights reserved. All rights reserved.10_mvl401-pc_target #1 Fri Mar 21 23:26:28 PDT 2008 ppc GNU/Linux module-9# sh processes cpu | egrep l2fm 1544 6396 388173 16 0. Inc. to abort type '$. To exit type 'exit'.

eeee 60 | 0000. Inc.cccc. Cisco Systems. All rights reserved. RST-3009 .bbbb 10 | 0000. Inc.cccc 4 Bank 1 Row HIT! 10 | 0000.ffff.dddd 6 Update Entry SMAC lookup 40 RST-3009 © 2008 Cisco Systems. Cisco Public 39 Layer 2 Lookup Frame 1 Compare 2 Lookup Key 10 0000.aaaa 30 | 0000. ffff Compare Bank 1 3 Hash Bank 2 Hash 100 | 0000.eeee.aaaa.abab 200 | 0000.dddd. Cisco Public Bank 1 © 2008.Layer 2 Forwarding Table Design 115 bits 4K*16*2 = 128K entries 16 pages 4096 rows Bank 1 MAC Table Bank 2 RST-3009 © 2008 Cisco Systems. Inc.aaaa VLAN |MAC Address 5 40 | 0000.bbbb.aaaa.abab.acac. All rights reserved.acac Bank 2 DMAC lookup Destination interface(s) Bank 2 Row 20 | 0000. All rights reserved.

Module 1 Cisco Public Module 2 e2/7 Transmit packet on wire 41 Agenda  Chassis Architecture  Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow RST-3009 © 2008 Cisco Systems. All rights reserved. Inc. Inc. Cisco Public 42 © 2008. RST-3009 . All rights reserved.L2 Unicast Packet Flow  Supervisor Engine  Credit grant for fabric access Central Arbiter Packet transmission Fabric Module 1 Fabric ASIC  Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC  Packet transmission Packet transmission  Transmit to fabric Fabric ASIC Forwarding Engine Layer 3 Engine   Queuing and VOQ arbitration request ACL/QoS/ NetFlow lookups  Forwarding Engine Layer 3 Engine Layer 2 Engine Fabric ASIC  Receive from fabric Return buffer credit Fabric Interface and VOQ  Layer 2 Engine  Fabric Interface and VOQ  Submit packet for lookup 2nd stage ingress queuing and scheduling Replication Engine  Layer 2 only SMAC/DMAC lookup  Replication Engine Egress queuing and scheduling Port ASIC CTS and 4:1 Mux Submit packet for egress L2 lookup  Port ASIC CTS and 4:1 Mux   Layer 2 SMAC/DMAC lookups  Receive packet from wire RST-3009 CTS LinkSec decryption and verification 1st stage ingress queuing and scheduling  CTS LinkSec encryption  e1/1 © 2008 Cisco Systems. Cisco Systems. All rights reserved. Inc.

RIP. All rights reserved. RST-3009 . ipfib Cisco Public 44 © 2008.IP Forwarding  Nexus 7000 decouples control plane and data plane  Forwarding tables built on control plane using routing protocols or static configuration OSPF. BGP for dynamic routing  Tables downloaded to forwarding engine hardware for data plane forwarding RST-3009 © 2008 Cisco Systems.?rib 44722390 0 0 34200830 0 0 cpu | egrep ufdm 743933460 0 0 Supervisor Engine BGP OSPF ISIS RIP EIGRP URIB/U6RIB UFDM IP FIB Hardware IP FIB Hardware I/O Module IP FIB Hardware I/O Module Process ospf u6rib urib ufdm I/O Module Hardware FIB TCAM ADJ Table module-9# sh processes cpu | egrep fib 1534 80042 330725 242 0. Inc. All rights reserved.0 module-9# RST-3009 © 2008 Cisco Systems. EIGRP. Inc. Cisco Systems. All rights reserved. Inc. Cisco Public 43 IP Forwarding Architecture  Routing protocol processes learn routing information from neighbours  IPv4 and IPv6 unicast RIBs calculate routing/next-hop information  Unicast Forwarding Distribution Manager (UFDM) interfaces between URIBs on supervisor and IP FIB on I/O modules  IP FIB process programs forwarding engine hardware on I/O modules FIB TCAM contains IP prefixes Adjacency table contains next-hop information n7010# PID 20944 n7010# 3573 3574 n7010# 3836 sh processes Runtime(ms) 93 sh processes 117 150 sh processes 1272 cpu | egrep ospf|PID Invoked uSecs 1Sec 33386880 0 0 cpu | egrep u. IS-IS.

Cisco Systems. Inc. Inc. Cisco Public 46 © 2008. IPv6 unicast prefixes (shared) IPv6 multicast routes Logical Entries 56K 32K 2K Physical Entries 56K 64K 8K RST-3009 © 2008 Cisco Systems. QoS. affecting final forwarding result RST-3009 © 2008 Cisco Systems.Hardware IP Forwarding Process  FIB TCAM lookup based on destination prefix (longestmatch)  FIB “hit” returns adjacency. All rights reserved. and NetFlow lookups. All rights reserved. All rights reserved. Inc. RST-3009 .0 release) FIB TCAM Partitioning Protocol IPv4 unicast prefixes IPv4 multicast routes. adjacency contains rewrite information (next-hop)  Pipelined forwarding engine architecture also performs ACL. Cisco Public 45 FIB TCAM  128K FIB TCAM entries  FIB TCAM hardware statically partitioned (4.

100 10.1.xx FIB TCAM Adj Index Result Adjacency Table /24 entries (mask last octet) RST-3009 © 2008 Cisco Systems.1.1.  Hardware adjacency table shared among protocols 1M adjacency entries shared between IPv4/IPv6 unicast and IPv4/IPv6 multicast  Individual adjacency table entries are not shared among protocols For example.1.10 10. MACs. Cisco Public 48 © 2008.1.0.1. All rights reserved.10. MTU Hash Offset 5 IF. same next-hop device for IPv4 and IPv6 will use two adjacency entries RST-3009 © 2008 Cisco Systems.3 10.1 10.10 10.100.xx 4 10.1. MACs. MACs.2 10.xx 10.1.1.xx HIT! 10. MTU 6 IF. MTU Load-Sharing IF. All rights reserved.2 10.1. Inc. Cisco Systems.xx 10.0.1.1. MTU Flow Data 1 DIP 10.100.1. Inc.3. MACs.10. RST-3009 .2. Cisco Public 47 IPv4 FIB TCAM Lookup Generate Lookup Key Compare 3 10.1. Inc.xx Lookup Key 2 10.1.0. All rights reserved.100. destination MAC address.10.100. MTU.xx 10.1.10 FFFFFFFF Packet /32 entries (compare all bits) 10.1.1.100. etc.Hardware Adjacency Entries  Contains information about next-hops Outgoing interface.1.33 IF.10.0.1.xx 10.4 10.10.

Cisco Systems. Inc. All rights reserved. RST-3009 .“Routing” versus “Forwarding”  “Routing” information refers to unicast RIB contents in supervisor control plane  “Forwarding” information refers to FIB contents at I/O module RST-3009 © 2008 Cisco Systems. Inc. Cisco Public 50 © 2008. All rights reserved. Cisco Public 49 Displaying Routing and Forwarding Information  show routing [ipv4|ipv6] [<prefix>] [vrf <vrf>] Displays software routing (URIB) information Can also use traditional show ip route command  show forwarding [ipv4|ipv6] route module <mod> [vrf <vrf>] Displays routing (FIB) information on per-module basis  show forwarding adjacency module <mod> Displays hardware adjacency table information on per-module basis RST-3009 © 2008 Cisco Systems. Inc. All rights reserved.

0/16 via Rtr-A via Rtr-B  Configure load-sharing hash options with ip load-sharing command: Source and Destination IP addresses (default) Source and Destination IP addresses plus L4 ports Destination IP address and L4 port A B  Additional randomised number added to hash prevents polarisation Automatically generated or user configurable value 10. Inc.0/16 RST-3009 © 2008 Cisco Systems.7. [110/5].0/24 module 9 IPv4 routes for table default/base ------------------+------------------+--------------------Prefix | Next-hop | Interface ------------------+------------------+--------------------10.0001 interface ---------Ethernet9/2 51 ECMP Load Sharing  Up to 16 hardware load-sharing paths per prefix  Use maximum-paths command in routing protocols to control number of load-sharing paths  Load-sharing is per-IP flow No per-packet load-balancing today 10. 1 ucast next-hops.1. Inc.0/24 IP Route Table for VRF "default" 10.2. Cisco Public 52 © 2008. adjacency count 1 next-hop --------------10. ospf-1. 0 mcast next-hops *via 10.10. Ethernet9/2.1. Cisco Public rewrite info -------------0010.2.100.10.0/24 10.9400.2.100.2.1.7. Inc.2.100.2 Ethernet9/2 n7010# show forwarding adjacency 10.Displaying Routing and Forwarding Information (Cont) n7010# sh routing ipv4 10. Cisco Systems.7.100.7. type-1 n7010# show forwarding ipv4 route 10. 00:02:30.2 module 9 IPv4 adjacency information.0.1.0.2 n7010# RST-3009 © 2008 Cisco Systems. All rights reserved. RST-3009 . All rights reserved. All rights reserved.0/24.

Cisco Systems. All rights reserved.2 10. 0 mcast next-hops *via 10. RST-3009 .0 IP Route Table for VRF "default" 10.0 module 9 IPv4 routes for table default/base ------------------+------------------+--------------------Prefix | Next-hop | Interface ------------------+------------------+--------------------10.0. 2 ucast next-hops.0/16. inter *via 10.1.1.12 10. Cisco Public 54 © 2008.200.0/16. All rights reserved.44.71.168. Ethernet9/2.2.2 (hash: 0x29). Ethernet9/2.188 Load-share parameters used for software forwarding: load-share type: 1 Randomizing seed (network order): 0xebae8b9a Hash for VRF "default" Hashing to path *10. 00:14:18.1.2. [110/5]. [110/5]. Inc. inter n7010# sh forwarding ipv4 route 10.1. for route: 10. inter n7010# Same hash algorithm applies to both hardware and software forwarding RST-3009 © 2008 Cisco Systems.200.1. ospf-1.ECMP Prefix Entry Example n7010# sh routing ipv4 10.2. ospf-1.0. inter *via 10. Ethernet9/1.200. Cisco Public 53 Identifying the ECMP Path for a Flow show routing [ipv4|ipv6] hash <sip> <dip> [<sport> <dport>] [vrf <vrf>] n7010# sh routing hash 192.1.2.0/16 n7010# 10.1.0.1.1. 00:03:33. 0 mcast next-hops *via 10. 00:00:13. ospf-1.2.200. Inc.2.2. ospf-1.0. Inc. [110/5]. All rights reserved.0.2 Ethernet9/1 Ethernet9/2 RST-3009 © 2008 Cisco Systems.2.1. 00:10:58.200. [110/5].200. 2 ucast next-hops. Ethernet9/1.

Inc. Cisco Public 56 © 2008. Cisco Systems.L3 Unicast Packet Flow  Supervisor Engine  Credit grant for fabric access Central Arbiter Packet transmission Fabric Module 1 Fabric ASIC  Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC  Packet transmission Packet transmission  Transmit to fabric Fabric ASIC Forwarding Engine Layer 3 Engine     Queuing and VOQ arbitration request Fabric Interface and VOQ L3 FIB lookup Ingress/egress ACL/QoS/ NetFlow lookups Forwarding Engine Layer 3 Engine Layer 2 Engine Fabric ASIC  Receive from fabric Return buffer credit Layer 2 Engine   Fabric Interface and VOQ  Submit packet for lookup 2nd stage ingress queuing and scheduling Replication Engine Port ASIC CTS and 4:1 Mux    Layer 2 ingress and egress SMAC/DMAC lookups Layer 2 only egress SMAC/DMAC lookups Replication Engine   Submit packet for lookup Egress queuing and scheduling Port ASIC CTS and 4:1 Mux  CTS LinkSec decryption and verification 1st stage ingress queuing and scheduling  CTS LinkSec encryption  Receive packet from wire RST-3009 e1/1 © 2008 Cisco Systems. Inc. All rights reserved. Inc. RST-3009 . All rights reserved. All rights reserved. Module 1 Cisco Public Module 2 e2/7 Transmit packet on wire 55 Agenda  Chassis Architecture  Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow RST-3009 © 2008 Cisco Systems.

ipfib Cisco Public 58 © 2008. RST-3009 . MLD  Tables downloaded to: Forwarding engine hardware for data plane forwarding Replication engines for data plane packet replication RST-3009 © 2008 Cisco Systems. Inc. Cisco Systems.G) and (S. IGMP. Cisco Public 57 IP Multicast Forwarding Architecture  Multicast routing processes learn routing information from neighbours/hosts  IPv4 and IPv6 multicast RIBs calculate multicast routing/RP/RPF/OIL information  Multicast Forwarding Distribution Manager (MFDM) interfaces between MRIBs on supervisor and IP FIB on I/O modules  IP FIB process programs hardware: FIB TCAM in forwarding engine contains (*.?rib 33436550 0 0 mrib 47169180 0 0 m6rib cpu | egrep mfdm 743581240 0 0 mfdm Supervisor Engine PIM IGMP PIM6 ICMP6 BGP MSDP MRIB/M6RIB MFDM IP FIB Hardware I/O Module IP FIB Hardware I/O Module IP FIB Hardware I/O Module Hardware FIB TCAM MET ADJ Table module-9# sh processes cpu | egrep fib 1534 80153 330725 242 0.0 module-9# RST-3009 © 2008 Cisco Systems.G) forwarding entries and RPF information Adjacency table in forwarding engine contains MET pointer MET in replication engines contains OILs n7010# PID 3842 3850 n7010# 3843 3847 n7010# 3846 sh processes Runtime(ms) 109 133 sh processes 177 115 sh processes 2442 cpu | egrep pim|igmp|PID Invoked uSecs 1Sec Process 32911620 0 0 pim 33279940 0 0 igmp cpu | egrep m. Inc. Inc.IP Multicast Forwarding  Forwarding tables built on control plane using multicast protocols PIM-SM. All rights reserved. PIM-Bidir. PIM-SSM. All rights reserved. All rights reserved.

1.1.1.10.44.1  Hit in FIB returns result in FIB DRAM RPF interface. Inc.1.1.1. lists of interfaces requiring replication RST-3009 © 2008 Cisco Systems. RST-3009 . 239. All rights reserved. i.1.8 HIT! 10. Cisco Public OIL MET  Replication engine uses MET index in lookup result to find correct OIL for replication 60 © 2008. ADJ Index RPF interface. Inc. control fields  Multicast Expansion Table (MET) Part of replication engine ASIC on I/O modules Contains output interface lists (OILs).G) entries as well as RPF interface  Adjacency Table (ADJ) Part of Layer 3 Engine ASIC on forwarding engine Contains MET indexes.3 10. 239.1.1.1. 232. 239.1.1.G) and (*.8.6.6.1 10. Cisco Public 59 Multicast FIB TCAM Lookup  Generate TCAM lookup key based on packet header data (source and group IP addresses) Ingress multicast packet header Generate Lookup Key 10. 239.4. Inc.1.7. ADJ Index RPF interface.10.10.1.1.1 FIB TCAM FIB DRAM Replication Engine OIL OIL OIL  Replication engine replicates to OIFs specified in MET (one copy per OIF listed) RST-3009 © 2008 Cisco Systems. packet rewrite data. Cisco Systems.12.Hardware Programming IP FIB process on I/O modules programs hardware:  FIB TCAM Part of Layer 3 Engine ASIC on forwarding engine Consists of (S.1 MET Index MET Index MET Index MET Index MET Index Adjacency  Adjacency contains MET index to drive replication 10. ADJ Index RPF interface. All rights reserved.2.e. ADJ Index  Compare lookup key to multicast entries in FIB TCAM  FIB DRAM contains RPF interface and index to rewrite data in adjacency table Forwarding Engine 10. ADJ Index RPF interface. 225..8.10.2.10. All rights reserved.

Inc. Cisco Public 61 Displaying Multicast Routing and Forwarding Information  show routing [ipv4|ipv6] multicast [vrf <vrf>] [<source-ip>] [<group-ip>] [summary] Displays software multicast routing (MRIB) information Can also use traditional show ip mroute command  show forwarding [ipv4|ipv6] multicast route [source <ip>] [group <ip>] [vrf <vrf>] module <mod> Displays hardware multicast routing (FIB) information on per-module basis RST-3009 © 2008 Cisco Systems.100 MET Block Index 1 from ADJ 1 tun0 e4/4 Index 2 from ADJ 2 e4/3.Multicast Expansion Table (MET)  MET blocks are shared by mroutes with identical fan-out Entry OIFs Index 0 from ADJ 0 e4/12 vlan100 e7/1. Inc. All rights reserved. Cisco Systems.44 e8/1 Replication MET Engine po100 vlan777 RST-3009 © 2008 Cisco Systems. All rights reserved. Inc. RST-3009 . Cisco Public 62 © 2008. All rights reserved.

Inc.1/32).4 9917127 1269391824 <…> RST-3009 © 2008 Cisco Systems.1.1 summary IP Multicast Routing Table for VRF "default" Total Total Total Total Group number number number number count: of of of of 1.1.Displaying Multicast Routing and Forwarding Information (Cont) n7010# sh routing multicast 10. All rights reserved. mrib n7010# sh routing multicast 239. flags: Received Packets: 10677845 Bytes: 1366764160 Number of Outgoing Interfaces: 2 Outgoing Interface List Index: 15 Ethernet9/2 Outgoing Packets:432490865 Bytes:55358830720 Ethernet9/17 Outgoing Packets:419538767 Bytes:53700962176 n7010# RST-3009 © 2008 Cisco Systems. Cisco Systems.1/32).1.2.0 Group: 239. uptime: 00:06:12.1 IP Multicast Routing Table for VRF "default" (10. 239.1.1/32.1.1.1.1. RPF nbr: 10.1. ip mrib pim Incoming interface: Ethernet9/1. uptime: 00:05:57. RPF Interface: Ethernet9/1. internal Outgoing interface list: (count: 2) Ethernet9/17. All rights reserved.1.1.1. 239.1. Inc.1.2/32. routes: 202 (*. mrib Ethernet9/2.1.1. Cisco Public 64 © 2008. RST-3009 .1. All rights reserved.2/32.1. Cisco Public aps 110 127 127 127 pps 0 4227 4227 4227 bit-rate 0 bps 4 mbps 4 mbps 4 mbps oifs 2 2 2 2 63 Displaying Multicast Routing and Forwarding Information (Cont) n7010# sh forwarding ipv4 multicast route group 239.1.G) routes: 1 (S.1.1.1.3 9917143 1269393890 10.1 source 10. Source count: 200 Source packets bytes (*.1.G) routes: 200 (*.2 module 9 (10.1.1.1.1.2 239. Inc.1. uptime: 00:40:31.G-prefix) routes: 1 average sources per group: 200.G) 767 84370 10.2 9917158 1269395810 10.

All rights reserved. Cisco Public Local OIFs Local OIFs 65 L3 Multicast Packet Flow  Packet transmission Fabric Module 1  Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC  Packet transmission Fabric ASIC Packet transmission  Queuing and transmitting multicast distribution packet to fabric   Fabric ASIC Forwarding Engine Layer 3 Engine L3 multicast FIB lookup Ingress ACL/QoS/Net Flow lookups  Forwarding Engine Layer 3 Engine Layer 2 Engine Fabric ASIC Receive from multicast fabric plane  Ingress multicast replication  Fabric Interface and VOQ Layer 2 Engine  Submit packet for lookup 2nd stage ingress queuing and scheduling Receive packet from wire RST-3009 Replication Engine  Egress ACL/QoS/ NetFlow lookups Fabric Interface and VOQ  Replication Engine   Egress multicast replication Submit packet for lookup Port ASIC  Ingress L2 and IGMP snooping lookups  Egress L2 and IGMP snooping lookups Egress queuing and scheduling Port ASIC CTS and 4:1 Mux  CTS and 4:1 Mux    CTS LinkSec decryption and verification 1st stage ingress queuing and scheduling  CTS LinkSec encryption e1/1 Module 1 Cisco Public Module 2 e2/7 Transmit packet on wire © 2008 Cisco Systems.Egress Replication  Distributes multicast replication load among replication engines of all I/O modules with OIFs  Input packets get lookup on ingress FE  For OIFs on ingress module. All rights reserved. RST-3009 . All rights reserved. ingress replication engine replicates a single copy of packet over fabric to all egress modules  Replication engine on egress module performs replication for local OIFs 2 Fabric ASIC IIF Local OIF Module 1 Replication MET Engine Fabric ASIC Fabric Copy Fabric Module Fabric ASIC 3 Fabric ASIC Fabric ASIC 4 Replication MET Engine Replication MET Engine Replication MET Engine Local OIFs RST-3009 © 2008 Cisco Systems. Inc. Inc. ingress replication engine performs the replication  For OIFs on other modules. Inc. 66 © 2008. Cisco Systems.

Inc. MAC PACLs supported  Security Group ACLs (SGACLs)—Part of Cisco Trusted Security. and Layer 4 information  Classification TCAM (CL TCAM) provides ACL lookups in forwarding engine  Router ACL (RACL)—Enforced for all traffic crossing a Layer 3 interface in a specified direction IPv4 RACLs supported  VLAN ACLs (VACLs)—Enforced for all traffic in the VLAN IPv4. Cisco Systems. Layer 3. enforces policies based on tags RST-3009 © 2008 Cisco Systems. MAC VACLs supported  Port ACLs (PACLs)—Enforced for all traffic input on a Layer 2 interface IPv4. Cisco Public 67 Security ACLs  Enforce security policies based on Layer 2. Inc. Inc. All rights reserved. All rights reserved. Cisco Public 68 © 2008. RST-3009 . All rights reserved.Agenda  Chassis Architecture  Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow RST-3009 © 2008 Cisco Systems.

Inc. Cisco Public 70 © 2008. Inc.0 module-9# Supervisor Engine CLI XML ACL Manager ACL/QoS-C Hardware I/O Module ACL/QoS-C Hardware I/O Module Hardware CL TCAM aclqos RST-3009 © 2008 Cisco Systems. All rights reserved. All rights reserved. Cisco Public 69 Classification TCAM  Hardware-based packet classification for ACLs and QoS  CL TCAM stores entries in hardware Resources shared between security ACLs and QoS Classification Resources Resource CL TCAM entries LOUs Labels L4Ops per label Entries 64K (16K/bank) 104 (208 registers) 16K 10  CL TCAM Entries: Total unique ACEs  LOUs: Logical Operation Units. Cisco Systems. All rights reserved.ACL Architecture  ACL manager receives policy via configuration  ACL manager distributes policies to ACL/QoS Clients on I/O modules  Clients perform ACL merge and program ACEs in Classification (CL) TCAM in forwarding engines ACL/QoS-C Hardware I/O Module n7010# sh processes cpu | egrep aclmgr|PID PID Runtime(ms) Invoked uSecs 1Sec Process 3589 1662 516430000 0 0 aclmgr module-9# sh processes cpu | egrep aclqos 1532 9885 671437 14 0. Inc. RST-3009 . registers that allow more efficient storage and matching for L4 operations  Labels: Identifies a unique policy configuration applied to an interface or VLAN  L4ops per Label: Number of LOU register pointers a single label can reference RST-3009 © 2008 Cisco Systems.

Cisco Public 1 4121 4013 4078 2 0 2 0 4 0 3 3 0 1 16383 12263 12371 12306 102 0.2.1. Bank 1 Tcam 1.000 0.000 16 3 14 6140 6140 2047 2046 0.1.000 0. Bank 0 Tcam 1. Inc.1. RST-3009 .Displaying Classification Resources  show system internal access-list resource utilization module <mod> n7010# sh system internal access-list resource utilization module 9 Hardware Modules Used Free Percent Utilization ----------------------------------------------------Tcam 0.101 | xx | xxx | xxx xxxxxxx | 10.000 71 Security ACL ip access-list example ACL CL TCAM Lookup Generate TCAM lookup key based on packet header data (source and dest IP addresses.000 25. Bank 1 LOU Both LOU Operands Single LOU Operands TCP Flags Protocol CAM Mac Etype/Proto CAM Non L4op labels.000 0.2.) permit ip any host 10.2.1 DIP: 10.000 24.25 permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq 514 permit tcp any any eq 80 Packet header: SIP: 10.2 | | xx | xxx | xxx xxxxxxx | 10.2.100 xx | xxx | xxx xxxxxxx | 10.000 24. protocol. Inc. Bank 0 Tcam 0. Tcam 1 L4 op labels.33.68.1. Inc.101 deny ip any host 10. Cisco Systems.25 | xx | xxx | xxx Permit Deny Deny Permit Deny Deny Permit Permit X=“Mask” xxxxxxx || xxxxxxx || 06 || xxx || 0050 xxxxxxx xxxxxxx 06 xxx 0016 xxxxxxx | xxxxxxx | 06 | xxx | 0017 xxxxxxx | xxxxxxx | 11 | xxx | 0202 HIT! xxxxxxx | xxxxxxx | 06 | xxx | 0050 xxxxxxx | xxxxxxx | 11 | xxx | 00A1 CL TCAM Hit in CL TCAM returns contents of results SRAM Results SRAM Result affects final packet handling RST-3009 © 2008 Cisco Systems. Tcam 0 Non L4op labels.2 Protocol: TCP SPORT: 33992 DPORT: 80 permit udp any any eq 161 Generate Lookup Key SIP | DIP | Protocol | SPORT | DPORT 10.2.33.1. etc. Tcam 1 n7010# RST-3009 © 2008 Cisco Systems.2 | 06 | 84C8 | 0050 Compare lookup key to ACL entries in CL TCAM xxxxxxx | 10.1.000 0.2.000 57.1. All rights reserved. L4 ports.1 | 10. All rights reserved.000 1.2.000 0.1. Tcam 0 L4 op labels.68. All rights reserved.2.2. Cisco Public 72 © 2008.100 deny ip any host 10.2.

Inc. Cisco Public 74 © 2008. All rights reserved.100/32 [match=3452] 20 deny ip any 10.ACL Statistics  ACL statistics NOT enabled by default  Enable statistics on per-ACL basis using statistics keyword in ACL configuration mode  Use show [ip|mac] access-list to view ACL matches  Use clear [ip|mac] access-list to clear ACL statistics RST-3009 © 2008 Cisco Systems. Cisco Systems. Inc.68.2.33. Inc. All rights reserved.101/32 [match=49920] 30 deny ip any 10. RST-3009 .1.2. All rights reserved.1.25/32 [match=232324] 40 permit tcp any any eq 22 [match=9881] 50 deny tcp any any eq telnet [match=442] 60 deny udp any any eq syslog [match=87112] 70 permit tcp any any eq www [match=4345667] 80 permit udp any any eq snmp [match=234222] n7010# RST-3009 © 2008 Cisco Systems. Cisco Public 73 Displaying ACL Statistics  show [ip|mac|arp] access-lists n7010# sh ip access example IP access list example statistics 10 permit ip any 10.

RST-3009 . Cisco Public 75 Quality of Service  Comprehensive LAN QoS feature set  Ingress and egress queuing and scheduling Applied in I/O module port ASICs  Ingress and egress mutation. All rights reserved. policing Applied in I/O module forwarding engines  All configuration through Modular QoS CLI (MQC) All QoS features applied using class-maps/policy-maps/servicepolicies RST-3009 © 2008 Cisco Systems. marking. All rights reserved.Agenda  Chassis Architecture  Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow RST-3009 © 2008 Cisco Systems. Inc. Inc. Cisco Systems. All rights reserved. classification. Cisco Public 76 © 2008. Inc.

All rights reserved. Inc.0 module-9# Supervisor Engine CLI XML QoS Manager ACL/QoS-C Hardware I/O Module ACL/QoS-C Hardware I/O Module Process ipqosmgr Hardware CL TCAM I/O Module ASICs aclqos RST-3009 © 2008 Cisco Systems. Inc. Cisco Public 78 © 2008. RST-3009 . All rights reserved. Cisco Public 77 Port QoS—32-Port 10G Module  Buffers Ingress (2-stage ingress buffering) Dedicated mode: 1MB per port + 65MB per port Shared mode: 1MB per port + 65MB per port group Egress Dedicated mode: 80MB per port Shared mode: 80MB per port-group  Queue structure 8q2t + 2q1t ingress 1p7q4t egress Dedicated mode: per port Shared mode: per port-group RST-3009 © 2008 Cisco Systems. Inc. Cisco Systems. All rights reserved.QoS Architecture  QoS manager receives policy via configuration  QoS manager distributes policies to ACL/QoS Clients on I/O modules  Clients perform ACL merge and program hardware: ACEs in Classification (CL) TCAM in forwarding engines Queuing policies in I/O module port ASICs ACL/QoS-C Hardware I/O Module n7010# sh processes cpu | egrep qos|PID PID Runtime(ms) Invoked uSecs 1Sec 3849 1074 66946870 0 0 module-9# sh processes cpu | egrep aclqos 1532 9885 671437 14 0.

8 65MB Port 2. Cisco Systems. All rights reserved.8 Port Group Egress RST-3009 © 2008 Cisco Systems.6. RST-3009 .10G Module Buffering—Shared Mode Replication Engine Ingress (Fixed) 12 2q1t Ports 2.6. Inc.4. Cisco Public 80 © 2008. All rights reserved. Inc.4.8 Port Group Egress RST-3009 © 2008 Cisco Systems.6.4.8 80MB 12345678 Port ASIC Port 4 1MB 1p7q4t 12345678 Port 2 1MB 8q2t Port 6 1MB Port 8 1MB CTS and 4:1 Mux 2.6. Inc. Cisco Public 79 10G Module Buffering—Dedicated Mode Replication Engine Ingress (Fixed) 12 2q1t Port 2 65MB Port 2 80MB 12345678 Port ASIC 1p7q4t 12345678 Port 2 1MB 8q2t CTS and 4:1 Mux 2.4. All rights reserved.

6MB 7.6MB Port 10 6.6MB Port 9 6 Port 7.2MB 6. All rights reserved.2MB 12 1234 2q4t 1p3q4t Port ASIC CTS 1-4 CTS 5-8 CTS 9-12 Egress RST-3009 © 2008 Cisco Systems.2MB Port 2 Port 12 6.6MB Port 8 5 Port 7.6MB 7. Cisco Public 81 10/100/1000 Module Buffering Ingress Replication Engine Port 10 7.2MB 6. Inc. Inc.2MB 6. Inc. RST-3009 .2MB Port 3 6.6MB Port 1 Port 11 7.2MB Port 7 4 Port 6.15MB egress per port  Queue structure 2q4t ingress 1p3q4t egress RST-3009 © 2008 Cisco Systems.6MB 7. All rights reserved.6MB 7.2MB 6.6MB Port 7 4 Port 7.56MB ingress per port 6.2MB 6. Cisco Systems.6MB Port 2 Port 12 7.2MB Port 1 Port 11 6. Cisco Public 82 © 2008.6MB Port 3 7.2MB Port 9 6 Port 6.Port QoS—48-Port 10/100/1000  Buffers 7. All rights reserved.2MB Port 8 5 Port 6.6MB 7.

3.2.3.1.1.12.0/24 any ip access-list remark-prec-3 permit tcp any 10.xx | xx | xxx | xxx 10.) permit ip any 10.1.2.2.2 Protocol: TCP SPORT: 33992 DPORT: 80 Generate Lookup Key SIP | DIP | Protocol | SPORT | DPORT 10.1 | 10.1.24.2 | 06 | 84C8 | 0050 Compare lookup key to QoS entries in CL TCAM xxxxxxx | 10.2. Inc. All rights reserved.xx xxxxxxx | 10. L4 ports.0/24 permit ip any 10.3.4.5. etc.1.2. All rights reserved.xx | xxxxxxx | 11 | xxx | xxx 06 xxx| xxx Policer ID 1 Policer ID 1 Remark DSCP 32 Remark DSCP 40 Remark IP Prec 3 HIT! 10.1.1.xx | xxxxxxx | 06 | xxx | xxx xxxxxxx | 10. Inc.0/24 eq 23 Packet header: SIP: 10.1. RST-3009 .0/24 any ip access-list remark-dscp-40 permit tcp 10.1. Cisco Public 83 QoS Classification ACLs ip access-list police QoS CL TCAM Lookup Generate TCAM lookup key based on packet header data (source and dest IP addresses. All rights reserved.1.0/24 ip access-list remark-dscp-32 permit udp 10.5.Marking and Policing  After classification.1 DIP: 10.3.xx | xx | xxx | xxx 10. Inc.xx| 06 | xxx | 0017 CL TCAM Hit in CL TCAM returns contents of results SRAM Results SRAM Result affects final packet handling RST-3009 © 2008 Cisco Systems.2.5.1. Cisco Systems. protocol. traffic can be marked or policed  Marking policies statically set QoS values for each class  Policing performs markdown and/or policing (drop)  Policers use classic token-bucket scheme Uses Layer 2 frame size when determining rate  Note: policing performed on per-forwarding engine basis Shared interfaces (such as SVI/EtherChannel) and egress policies could be policed at <policing rate> * <number of forwarding engines> RST-3009 © 2008 Cisco Systems.1.5. Cisco Public 84 © 2008.4.

All rights reserved. All rights reserved.Monitoring QoS Service Policies show policy-map interface [[<interface>] [type qos|queuing]]|brief] n7010# show policy-map interface e9/1 Global statistics status : enabled Ethernet9/1 Service-policy (qos) input: policy statistics status: mark enabled Class-map (qos): udp-mcast (match-all) 432117468 packets Match: access-group multicast set dscp cs4 Class-map (qos): udp (match-all) 76035663 packets Match: access-group other-udp police cir 2 mbps bc 1000 bytes pir 4 mbps be 1000 bytes conformed 587624064 bytes. 3999632 bps action: transmit exceeded 293811456 bytes. Cisco Public 86 © 2008. Inc. Inc. 153221133 bps action: drop n7010# RST-3009 © 2008 Cisco Systems. Inc. Cisco Public 85 Agenda  Chassis Architecture  Supervisor Engine Architecture  I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  Layer 2 Forwarding  IP Forwarding  IP Multicast Forwarding  ACLs  QoS  NetFlow RST-3009 © 2008 Cisco Systems. 1999812 bps action: set dscp dscp table cir-markdown-map violated 22511172352 bytes. RST-3009 . All rights reserved. Cisco Systems.

NetFlow  NetFlow table is 512K entries (490K effective). Inc. Cisco Systems. Cisco Public 87 NetFlow Architecture  NetFlow manager receives configuration via CLI/XML  NetFlow manager distributes configuration to NetFlow-Clients on I/O modules  NetFlow-Clients apply policy to hardware n7010# sh processes cpu | egrep nfm|PID PID Runtime(ms) Invoked uSecs 1Sec Process 24016 1463 735183570 0 0 nfm module-9# sh processes cpu | egrep nfp 1538 68842 424290 162 0. Inc. All rights reserved. RST-3009 . Inc. Cisco Public 88 © 2008.0 module-9# Supervisor Engine CLI XML NetFlow Manager NF-C Hardware I/O Module NF-C Hardware I/O Module NF-C Hardware I/O Module Hardware NF Table nfp Hardware NetFlow Creation RST-3009 © 2008 Cisco Systems. All rights reserved. shared between ingress/egress NetFlow  Hardware NetFlow creation CPU not involved in NetFlow entry creation/update  All modules have independent NetFlow table  Full and sampled NetFlow supported by hardware RST-3009 © 2008 Cisco Systems. All rights reserved.

All rights reserved. Cisco Public 89 Forwarding Engine NetFlow Tables Entry Key Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Entry Key Index Entry Key Index EntryKey Index Key Index EntryKey Index EntryKey Index Entry EntryKey Index Key Index EntryKey Index Entry Entry Key Index EntryKey Index Key Index EntryKey Index EntryKey Index Entry Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data 512K entries Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics 512K entries 512K entries Netflow Lookup Table 4 pages Netflow Entry Table Netflow Statistics Table 90 RST-3009 © 2008 Cisco Systems. Inc. Cisco Systems. RST-3009 . All rights reserved. Inc. Cisco Public © 2008. Inc. All rights reserved.NetFlow Table  NetFlow “Table” actually consists of three components in forwarding engine: NetFlow Lookup Table—Contains NetFlow Entry Keys and associated NetFlow Entry Table indexes NetFlow Entry Table—Contains actual NetFlow flow data NetFlow Statistics Table—Contains statistics for corresponding flow entries RST-3009 © 2008 Cisco Systems.

10 DIP=10. All rights reserved.NetFlow Lookup Packet Entry Key Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Key Index EntryKey Index EntryKey Index Entry EntryKey Index Key Index EntryKey Index Entry Entry Key Index EntryKey Index Key Index EntryKey Index EntryKey Index Entry EntryKey Index Key Index EntryKey Index Entry Entry Key Index EntryKey Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index HIT! Entry Key Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index Entry Key Index EntryKey Index Entry Key Index Entry Key Index SIP=10. Inc. Cisco Public Netflow Statistics Table 91 Full versus Sampled NetFlow  NetFlow configured per-direction and per-interface Ingress and/or egress on per-interface basis  Each interface can collect full or sampled flow data  Full NetFlow: accounts for every packet of every flow on interface. Cisco Public 92 © 2008. All rights reserved.1.11 Protocol=TCP (6) SPORT=33992 DPORT=80 1 Flow Key Flow Key Compare Flow Data 6 5 Index to NF Entry Table 2 Hash Function Compare all pages 4 Lookup Key 3 Entry Key Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data Flow Data HIT! Flow Data Flow Data Flow Data Flow Data 7 Update Stats Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Statistics Indexes row in Lookup Table Netflow Entry Table Netflow Lookup Table RST-3009 © 2008 Cisco Systems. All rights reserved.1. Inc.1. RST-3009 . up to capacity of NetFlow table RST-3009 © 2008 Cisco Systems. Inc. Cisco Systems.2. up to capacity of NetFlow table  Sampled NetFlow: accounts for M in N packets on interface.

Inc. Cisco Public 93 NetFlow Aging  Process of removing stale NetFlow entries  Each I/O module CPU ages entries independently  Types of aging Active—Maximum lifetime for flows (30m by default. select M consecutive packets and account only for those flows in the hardware NetFlow table  Sampled flows aged and exported from NetFlow table normally  Advantages Reduces NetFlow table utilisation Reduces CPU load on switch and collector  Disadvantages Accuracy may be sacrificed—Collector or user must extrapolate total traffic load based on configured sampling rate RST-3009 © 2008 Cisco Systems. Cisco Systems. 15s minimum) Fast—More aggressive aging of active flows (disabled by default) Aggressive—Table-utilisation based aging of flows (disabled by default) Session—Session-based aging (uses TCP FIN/RST flags) (disabled by default) n7010# sh flow timeout Flow timeout values Active timeout: Inactive timeout: Fast timeout: Session aging timeout: Aggressive aging timeout: n7010# RST-3009 © 2008 Cisco Systems.Sampled NetFlow  Random packet-based sampling  M:N sampling: Out of N consecutive packets. All rights reserved. Inc. All rights reserved. Cisco Public 1800 seconds 15 seconds Disabled Disabled Disabled 94 © 2008. All rights reserved. Inc. RST-3009 . 60s minimum) Inactive—Fixed idle time for flows (15s by default.

Sampler/Policer ID.Last Used Time.001.002 010.Protocol:Source Port:Destination Port TCP Flags: Ack.Creation Time. . FA . L4 Info . . Push. RST-3009 .001. . S .Direction. FR . . <…> n7010# sh system internal flow ip interface e9/1 detail module 9 D .004 010.001. . L4 Info .Protocol:Source Port:Destination Port TCP Flags: Ack. S . Inc. Flush. Inc.Adjacency/RIT Pointer CRT . S . Reset.NT Table Address D IF SrcAddr DstAddr L4 Info PktCnt TCP Flags -+-----+---------------+---------------+---------------+----------+----------ByteCnt TOS FR FA SID AP CRT LUT NtAddr -------------+---+--+--+-----+--------+-----+-----+-------I 9/1 010. Cisco Public 95 NetFlow Data Export To NetFlow Collector Generate NetFlow v5 or v9 export packets I/O Module Fabric ASIC via Inband Fabric Interface and VOQ Supervisor Engine LC CPU NetFlow Table Aged Flows Forwarding Engine Hardware Flow Creation I/O Module Main CPU Switched EOBC LC CPU NetFlow Table Aged Flows Forwarding Engine Hardware Flow Creation via mgmt0 Mgmt Enet I/O Module LC CPU NetFlow Table To NetFlow Collector Aged Flows Forwarding Engine Cisco Public Hardware Flow Creation RST-3009 © 2008 Cisco Systems.001.Direction. All rights reserved.Intf/VLAN. .001. AP . All rights reserved.001. Syn.001.003 006:01024:01024 0001403880 A .002. Urgent. Flush. I 9/1 010.001.Viewing NetFlow Records show system internal flow ip [detail] module <mod> n7010# sh system internal flow ip interface e9/1 module 9 D . NtAddr . Reset.001.002. IF . 0000218460416 000 N Y 0x000 0x000000 02168 02571 0x000331 RST-3009 © 2008 Cisco Systems.Intf/VLAN.004 006:01024:01024 0001403880 .002. Inc.001.FRagment.002 006:01024:01024 0001706722 A . Push.FastAging SID .002 010. All rights reserved. . S . IF . I 9/1 010.002 006:01024:01024 0001403880 A .001. Cisco Systems. . LUT . 96 © 2008.003 010. Syn.001. . Urgent D IF SrcAddr DstAddr L4 Info PktCnt TCP Flags -+-----+---------------+---------------+---------------+----------+----------I 9/1 010.002.

Cisco Public 97 Conclusion  You should now have a thorough understanding of the Nexus 7000 switching architecture.151. I/O module design. Inc.20. Inc.12 VRF: management (1) Destination UDP Port 10000 Source Interface mgmt0 (172. RST-3009 .151. and key forwarding engine functions…  ANY QUESTIONS? RST-3009 © 2008 Cisco Systems. Cisco Public 98 © 2008.20. All rights reserved. Inc. All rights reserved. packet flows.Viewing Flow Exporter Statistics show flow exporter [<name>] n7010# sh flow exporter Flow exporter nw: Destination: 172. All rights reserved. Cisco Systems.40) Export Version 9 Exporter Statistics Number of Flow Records Exported 988399 Number of Templates Exported 236 Number of Export Packets Sent 22686 Number of Export Bytes Sent 32189280 Number of Destination Unreachable Events 0 Number of No Buffer Events 0 Number of Packets Dropped (No Route to Host) 0 Number of Packets Dropped (other) 0 Number of Packets Dropped (LC to RP Error) 0 Number of Packets Dropped (Output Drops) 0 Time statistics were last cleared: Never n7010# RST-3009 © 2008 Cisco Systems.

Q and A RST-3009 © 2008 Cisco Systems. Inc. Inc. Inc. All rights reserved. All rights reserved. Cisco Systems. RST-3009 . Cisco Public 101 © 2008. All rights reserved. Cisco Public 100 Recommended Reading  Check the Recommended Reading flyer for suggested books  Continue your Cisco Networkers learning experience by visiting the following Demos located in the World of Solutions RST-3009 © 2008 Cisco Systems.

Cisco Systems.World of Solutions Demos Continue your Cisco Networkers learning experience by visiting the following Demos located in the World of Solutions  Nexus Range of Switches  Unified Communications Manager. All rights reserved. your preferred expert in that field and to set up a specific time to meet onsite. This is an invaluable opportunity so don’t miss out!  Visit the Meeting Centre in the World of Solutions to select your topic of interest.0  Cisco Wireless & Cisco Motion  Cisco and Ironport Security  Cisco ASR and Triple Play solutions with FTTx and Cisco IPTV  Infiniband and Virtual Blade Switches RST-3009 © 2008 Cisco Systems. All rights reserved. RST-3009 . Cisco Public 103 © 2008. Inc. Cisco Public 102 Meet the Expert  Make the most of your time at Cisco Networkers by meeting one-on-one with a Cisco Expert. Unity & MeetingPlace 7. Inc. RST-3009 © 2008 Cisco Systems. All rights reserved. Inc.0  Cisco Contact Centre Express 7.

All rights reserved. All rights reserved. Cisco Systems. Inc.Complete Your Online Session Evaluation  Win fabulous prizes by giving us your feedback!  Go to the Internet stations located throughout the Convention Centre to complete your session evaluation. Inc. Inc. RST-3009 © 2008 Cisco Systems. All rights reserved. Cisco Public 105 © 2008. Cisco Public 104 RST-3009 © 2008 Cisco Systems. RST-3009 .