ACI-FE M02

ACI Fabric Switches

Agenda

§  Nexus 9000 Overview

§  Nexus 9000 ASICs
§  Hardware Installation

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
Nexus 9000 Product Family
9500 Series
•  9504
•  9508
•  9516

9300 Series
•  9332PQ
•  9372PX / 9372TX
•  9396PX / 9396TX
•  93128TX
•  9336PQ (ACI Spine Switch)
© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
Nexus 9000 Platform: Two Modes
Application Centric Infrastructure NX-OS

APIC

Two modes of operation to support different operational models

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
Portfolio – ACI FCS
1.  Spine
•  Modular Spine: 4/8x 36p 40G
•  Fixed Spine: 36p 40G

2.  Leaf (Fixed)

•  48p 1/10GT + 6Q
•  48p 1/10GF + 12Q
•  48p 1/10GT + 12Q
•  96p 1/10GT + 8Q
•  32p 40GQ

3.  APIC Appliance APIC

•  Small < 1,000 edge ports
•  Large > 1,000 edge ports

4.  Interconnect
•  40G links
© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
 Nexus 9508 Hardware Overview
Nexus 9508 Front View Nexus 9508 Rear View

13 RU
8 Line Card Slots
Max 3.84 Tbps/Slot duplex
3 Fan Trays

3 or 6 Fabric Modules
(behind fan trays)

Redundant No Mid-plane for

Supervisor Engines LC to FM connectivity

Redundant System
3000W AC Power Supplies Controller Cards
2+0, 2+1, 2+2 Redundancy
Support up to 8 Power supports
Chassis Dimensions: 13 RU x 30 in. x 17.5 in (HxWxD)

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
Nexus 9336 and 9500 Spine Line Card

9336 fixed chassis, aka “Baby Spine”

9736 Spine line card for 95xx chassis

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Nexus 9396 and 93128 Leaf Switches

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Nexus C9396PX Hardware Overview
Interface 1/60
Interface 1/49
48-Port 1/10G SFP+
•  Fixed Chassis
•  12 Port QSFP+ Uplink Module
•  2 RU Interface 1/1
Interface 1/48

•  650W AC Power Supply

•  Front-to-Back or Back-to-Front
Airflow (“port-side intake/
exhaust”) PS 1: N9K-650W (hot air out)
FAN #3
PS 2
Or
•  Latency: 1-2 usec N9K-650W-B (cold air in)

FAN #1
•  Wire-Speed L2/L3 Forwarding N9K-C9300-FAN1 (hot air out)
Or
N9K-C9300-FAN1-B (cold air in)

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Modular: Nexus 9500 Line Card Types - Reference
Fabric
Line Cards Ports ASICs on Line Card OS Chassis Support
Modules
X9600
X9636PQ 36p QSFP+ 3 T2 NX-OS 6 N9504, N9508
X9612PC 12p 100G (form factor TBD) 3 T2 NX-OS 6 N9504, N9508
X9500

X9564PX 48p 1/10G SFP+ and 4p QSFP+ 2 T2 & 2 ALE NX-OS, ACI 3 N9504, N9508

X9564TX 48p 1/10G-T and 4p QSFP+ 2 T2 & 2 ALE NX-OS, ACI 3 N9504, N9508

X9536PQ 36p QSFP+ (1.5:1) 2 T2 & 2 ALE NX-OS, ACI 3 N9504, N9508

X9700
X9736PQ 36p QSFP+ 2 ASE ACI 6 N9504, N9508
X9400
X9464PX 48p 1/10G SFP+ and 4p QSFP+ 1 T2 NX-OS 4 N9504, N9508
X9464TX 48p 1/10G-T and 4p QSFP+ 1 T2 NX-OS 4 N9504, N9508
X9432PQ 32p QSFP+ 2 T2 NX-OS 4 N9504, N9508
© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
Fixed: Nexus 9300 - Reference
Model Ports ASICs OS RU Uplink Module

N9332PQ 32p 40G QSFP+ 1 T2, 1 ALE NX-OS, ACI 1 N

N9372PX 48p 1/10G SFP+ and 6p QSFP+ 1 T2, 1 ALE NX-OS, ACI 1 N

N9372TX 48p 1/10G-T and 6p QSFP+ 1 T2, 1 ALE NX-OS, ACI 1 N

N9396PX 48p 1/10G SFP+ and 12p QSFP+ 1 T2, (1 ALE) NX-OS, ACI 2 Y

N9396TX 48p 1/10G-T and 12p QSFP+ 1 T2, (1 ALE) NX-OS, ACI 2 Y

N93128TX 96p 1/10G-T and 8p QSFP+ 1 T2, (1 ALE) NX-OS, ACI 3 Y

N9336PQ 36p QSFP+ 2 ASE, 2 T2 ACI 2 N

Uplink Modules
N9K-M12PQ 12p 40-Gigabit port QSFP+ 1 ALE NX-OS, ACI

N9K-M6PQ 6p 40-Gigabit port QSFP+ 1 ALE NX-OS, ACI

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
Current Fabric Scalability

§  6 Spines §  Spine switches:

§  9504/8 with spine line card
§  50 Leafs, 5000 Ports (X9736PQ)
§  “baby spine” 9336PQ
§  Single Spine Layer §  Leaf switches:
§  9332PQ
§  100k End Points §  9372PX / 9372TX
§  100 Tenants §  9396PX / 9396TX
§  93128TX
§  FEX:
§  3/leaf switch
§  18/fabric
© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
FEX Support
Part Number Description

N2K-C2248PQ-10GE Cisco Nexus 2248PQ 10GE Fabric Extender

N2K-C2248TP-E-1GE Cisco Nexus 2248TP-E Fabric Extender

N2K-C2248TP-1GE Cisco Nexus 2248TP GE Fabric Extender

N2K-C2232PP-10GE Cisco Nexus 2232PP 10GE Fabric Extender

N2K-C2232TM-E-10GE Cisco Nexus 2232TM 10GE Fabric Extender

N2K-B22DELL-P B22 FEX for Dell

N2K-B22HP-P B22 FEX for HP

N2K-B22IBM-P B22 FEX for IBM

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
 ACI Customer Bundles Configuration
ü  ACI-C9336-APIC-B1 $100K ü  ACI-C9508-APIC-B1 $250K ü  ACI-C9504-APIC-B1 $200K
1)  Choose
2 x N9K-C9336 2xN9K-C9508-B2 + 2xN9K-X9736PQ 2xN9K-C9504-B2 + 2xN9K-X9736PQ
Spine
+ APIC-CLUSTER-M1 + APIC-CLUSTER-M1 + APIC-CLUSTER-M1
Bundle

ü  N9K-C9396PX-B18Q $46K

2xN9396PX, 6x40G uplink, 8 BIDI
ü  N9K-C93120TX-B18Q $56K
2xN93120TX, 6x40G uplink, 8 BIDI
2) Choose Leaf ü  N9K-C9372PX-B18Q $45K
Bundle(s) 2xN9372PX, 6x40G uplink, 8 BIDI
ü  N9K-C9372TX-B18Q $45K
2xN9372TX, 6x40G uplink, 8 BIDI
ü  N9K-C9396TX-B18Q $45K
2xN9396TX, 6x40G uplink, 8 BIDI

3) Add ACI License Per Leaf ACI-N9K-48X/96X

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
 ACI Lab Bundles
ACI Lab Bundle ACI Simulator Appliance
•  19% Bundle Discount
•  PID: ACI-SIM-S
•  PID: ACI-C9336-BL-96PX
ü  UCS C220 M3
ü  SPINES : (1) N9K-C9336PQ (36p 40G QSFP)
ü  (2) Xeon E5-2620 v2 6-core
ü  LEAFS : (2) N9K-C9396PX (48p 1/10G SFP+) ü  (4) 16-GB DDR3 RAM
ü  OPTICS : (2) QSFP AOC
ü  (1) 100-GB SSD
ü  APIC : (1) APIC--M1 (<1000 Leaf Ports)
ü  ACI S/W LICENSES: (2) ACI-N9K-48X= (96 Leaf Ports)

ACI
KITS

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
 ACI – Host Routed IP Fabric
•  Decoupling of end point identity, location and Spine
Inline overlay hardware database 576 x 40G
associated policy all of which are independent from ports (100G capable) Higher capacity & APIC
the underlying forwarding graph lower cost

•  Full normalization of the ingress encapsulation

mechanism used, 802.1Q VLAN, IETF VXLAN, IETF
NVGRE
•  Distributed Layer 3 gateway ensures optimal
forwarding for layer 3 and layer 2
•  Support for standard bridging and routing
semantics without standard location constraints
(any IP address anywhere)
•  Service insertion and redirection Scale
Fabric Optimisation Intelligent caching
•  Removal of flooding requirements for IP control Improved Utilisation Overlay hardware offload
plane (ARP/GARP) 1588 Timing & Latency Improved Analytics
ECMP based approaches

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
 MERCHANT +

DYNAMIC LOAD BALANCING

FLOW PRIORITIZATION
MULTICAST MULTIPATH
INLINE HW OVERLAY DB
FAST RE-ROUTE
CISCO
ASIC
INNOVATIONS

+ VXLAN Routing

Normalized Forwarding
Incremental 40 MB
Buffer, Active Queue
Management
APPLICATION VISIBILITY
Elephant Traps,
VXLAN Aware Trace-route
Atomic

End-end Latency
VXLAN/NVGRE/VLAN Measurements

INDUSTRY Counters per

VXLAN Bridging 12 MB Buffer Visibility per Port
BASELINE Packet and Bytes
© 2013-2014 Cisco Cisco Confidential 19
Trident 2 and/or its affiliates. All rights reserved.
 Nexus 9300 – ACI Physical Leaf
GEM 12 x 40 GE QSFP+ Uplinks
GEM 12 x 40 GE QSFP+ Uplinks (only 8 ports are active)

ALE ALE

12 x 40 Gbps 8 x 40 Gbps
Hi-Gig2
NFE-128 NFE-128
12 x 40G 24 x 40E
Ethernet Ethernet
Network Interfaces Network Interfaces

Front Panel 48 x 1 GE / 10 GE Ports Front Panel 96 x 1 GE / 10 GE Ports

Nexus® 9396PX / TX
Nexus 93128TX

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
Switch Installation best practices

§  Verify airflow

§  Different fan & PS model #’s
§  Front-to-back or Back-to-front
§  Red or blue color code
Red = hot air out
Blue = cold air in

§  Verify ACI image load with ‘show version’ command

NXOS image file is: bootflash:///aci-n9000-dk9.6.1.2.I2.1.bin!
bootflash:///n9000-... Means standalone image

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
Where to connect the spines?

§  Spines can only be connected to the Leaf’s 40G GEM module
§  All 10GE interfaces are for hosts or APICs
40G GEM

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
Agenda

§  Nexus 9000 Overview

§  Nexus 9000 ASICs
§  Hardware Installation

© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24