You are on page 1of 168

Nexus Validation Test

Phase 2
Contents
1. Introduction........................................................................................................................................ 4
2. NVT Topology Design Overview ......................................................................................................... 4
2.1 Network Logical Topology Design Overview.............................................................................. 4
2.1.1 Description of the Test Network ........................................................................................ 4
2.1.2 Test Network Configuration ............................................................................................. 11
2.2 Hardware and Software Overview ........................................................................................... 13
2.2.1 Network Hardware and Software version Details ........................................................... 13
2.2.2 Nexus 7000 Line Cards and Fabric Extenders (FEX) .......................................................... 14
2.2.3 Unified Computing System (UCS) Physical ....................................................................... 14
3. NVT Network Implementation and Configuration .......................................................................... 15
3.1 Configuration of Platform specific features ............................................................................. 15
3.1.1 Licensing ........................................................................................................................... 15
3.1.2 Common Configs............................................................................................................... 17
3.1.3 Out-of-Band Management Network ................................................................................ 19
3.1.4 CoPP .................................................................................................................................. 20
3.1.5 Rate Limiters ..................................................................................................................... 24
3.1.6 VDCs and Resource Allocation ......................................................................................... 24
3.2 Image Upgrade and Downgrade .............................................................................................. 26
3.3 Routing Design Overview ......................................................................................................... 29
3.3.1 Unicast Routing Design..................................................................................................... 29
3.3.2 Multicast Routing Design with PIM-ASM ......................................................................... 36
3.4 Layer-2/ Layer-3 Aggregation/Access Layer Network Design Overview ................................. 42
3.4.1 vPC .................................................................................................................................... 42
3.4.2 FabricPath ......................................................................................................................... 60
3.4.3 Fabric Extenders (FEX) ...................................................................................................... 81
3.5 Unified Computing System (UCS) Overview ............................................................................ 83
3.5.1 UCS Management & Monitoring ...................................................................................... 83
3.5.2 UCS Blade Management ................................................................................................... 84
3.5.3 UCS Uplink Port Infrastructure ......................................................................................... 85
3.5.4 UCS Server Port Infrastructure ......................................................................................... 87
3.5.5 UCS Distributed Virtual Switches (DVS) ........................................................................... 88
4. NVT Test Methodology .................................................................................................................... 88
4.1 Host/Server Configuration ................................................................................................... 88
4.2 Test Cycle .............................................................................................................................. 90
4.3 Network Disruption Test Cases ............................................................................................ 90
4.4 Automation........................................................................................................................... 95
5. NVT Findings/Conclusion/Recommendations ................................................................................. 97
5.1 Caveats for NVT 2.1-2.3 ............................................................................................................ 97
6. Test Results .................................................................................................................................... 107
1. Introduction

Cisco Nexus line of Data-center product hardware and software releases must pass Cisco's
comprehensive quality assurance process, which includes a multistage approach comprising extensive
unit test, feature test, and system-level test. Each successive stage in the process adds increasingly
higher levels of complexity in a multidimensional mix of features and topology.

Nexus Validation Test (NVT) has been established as an additional quality assurance stage in order to
leverage customer feedback and requirements into the product development cycle. NVT will validate
and publish guidelines for deploying NX-OS switching and UCS solutions for datacenter networks.

This document describes the NVT topologies, hardware & software configurations, test procedures and
findings. Addendums to this document will be published when NVT completes any future test cycles
using the same test topology and procedures.

2. NVT Topology Design Overview

2.1 Network Logical Topology Design Overview

The topologies and test cases validate highly-available data-center networks in order to provide unified
fabric and computing services. This is achieved by using Nexus line of switches and UCS B-series servers
with features such as vPC and FabricPath.

2.1.1 Description of the Test Network

The following figure illustrates the test network topology, consisting of two datacenter sites
interconnected through a public IP cloud. The first data center site is built around Nexus 7000 with Sup 1
and the second data center is built with Sup 2E.

Within each datacenter site the network is split into two halves:
 Nexus 7000 with vPC to Nexus 5000 for access
 Nexus 7000 with FabricPath with Nexus 5000

While the majority of test cases focus on integrated solutions using Nexus switching and UCS products,
modular Catalyst switches are also included for interoperability between NX-OS and IOS.
Figure 1 DC1 Topology
UCS-B-101- UCS-B-101-
UCS-B-101- UCS-B-101- 01-01
02-01 02-02
UCS 5108
01-02 !
UCS 5108

Ixia Ixia Expand to 2 Distributions:


UCS
SLOT SLOT !
1 2 SLOT SLOT
1 2

SLOT SLOT
3 4 SLOT SLOT
3 4


UCS 5108

UCS
UCS 5108
SLOT SLOT
5 6 SLOT SLOT
5 6 !
!
SLOT SLOT
SLOT SLOT 1 2
1 2

Sup720-10G VSS
SLOT SLOT
7 8 SLOT SLOT
7 8

OK FAIL OK FAIL OK FAIL OK FAIL


OK FAIL OK FAIL OK FAIL OK FAIL

SLOT SLOT
SLOT SLOT 3 4
3 4

UCS-B-101- UCS-B-101-
FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK

SLOT SLOT
SLOT

UCS-B-101- UCS-B-101- UCS-B-101- UCS-B-101-


SLOT 5 6
5 6
8 7 8 7
SLOT SLOT SLOT SLOT


6 5 6 5
SLOT SLOT SLOT SLOT

SLOT SLOT
SLOT SLOT 7 8
7 8

Sup2T VSS
OK FAIL
4 3 4 3
SLOT SLOT SLOT SLOT
OK FAIL OK FAIL OK FAIL
OK FAIL OK FAIL OK FAIL OK FAIL

02-01 02-02
2 1 2 1
SLOT SLOT SLOT SLOT

! !

03-01
UCS 5108 UCS 5108

03-01 03-02 03-02


UCS
UCS 5108 UCS 5108

! !
SLOT SLOT SLOT SLOT

UCS
1 2 1 2
UCS 5108 UCS 5108

! !

UCS
UCS 5108 UCS 5108 SLOT SLOT SLOT SLOT
1 2 1 2
! !
SLOT SLOT SLOT SLOT

FI
1 2 1 2

SLOT SLOT SLOT SLOT


3 4 3 4

SLOT SLOT SLOT SLOT


3 4 3 4

SLOT SLOT SLOT SLOT


3 4 3 4

SLOT SLOT SLOT SLOT


5 6 5 6

SLOT SLOT SLOT SLOT


5 6 5 6

SLOT SLOT SLOT SLOT


5 6 5 6

SLOT SLOT SLOT SLOT


7 8 7 8

10.2.250.3 UCS-B-101- UCS-B-101-


SLOT SLOT

10.2.250.2
SLOT SLOT
7 8 7 8
OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL

10.2.250.5
SLOT SLOT SLOT SLOT
7 8 7 8

10.2.250.6
OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL

OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL

FI-101-02-A FI-101-02-B FI-101-01-A FI-101-01-B 01-01


FI 10.2.250.5 10.2.250.6 10.2.250.9 10.2.250.8 10.2.250.9 01-02
10.2.250.8
UCS
UCS 5108

FI-101-02-A FI-101-02-A FI-101-03-B


!
SLOT SLOT

FI-101-03-B FI-101-03-A
1 2

UCS 5108

!
SLOT SLOT
1 2

FI-101-03-A
SLOT SLOT
3 4

SLOT SLOT
3 4

FEX
SLOT SLOT

Ixia
5 6

Ixia
SLOT SLOT
5 6

UCS-B-103- UCS-B-103-
SLOT SLOT
7 8

OK FAIL OK FAIL OK FAIL OK FAIL

SLOT SLOT
7 8

OK FAIL OK FAIL OK FAIL OK FAIL

FI FI FI vPC 01-01 01-02


40.101.0.7 40.101.0.8 UCS 5108

SLOT
1
SLOT
2
!
UCS 5108

SLOT
1
SLOT
2
!
UCS 5108

SLOT

SLOT
1

3
SLOT
2

SLOT
4
!
UCS 5108

SLOT
1

SLOT
3

SLOT
5
SLOT
2

SLOT
4

SLOT
6
!

DC101-7
SLOT SLOT

UCS
3 4

10.2.250.3 DC101-8
SLOT SLOT

40.101.0.17 40.101.0.18
SLOT SLOT 5 6
3 4

10.2.250.2
SLOT SLOT
7 8

40.101.0.27 40.101.0.28
SLOT SLOT OK FAIL OK FAIL OK FAIL OK FAIL
7 8
SLOT SLOT
5 6

SLOT SLOT
5 6 OK FAIL OK FAIL OK FAIL OK FAIL

40.101.0.37 40.101.0.38 FI-101-01-A FI-101-01-B L2 Switch


SLOT
7
SLOT
8
SLOT
7
SLOT
8

DC101-28 DC101-17
OK FAIL OK FAIL OK FAIL OK FAIL

40.101.0.47 DC101-5, DC101-6, DC101-18


OK FAIL OK FAIL OK FAIL OK FAIL

DC101-37 DC101-27 40.103.0.47


DC101-47 F105 FEX F105 L2 Switch
DC101-38 FEX N5K N5K FI orphan orphan DC103-47
vPC N3K
Non-vPC vlans.
Non- vPC
N3k vPC
vPC vlans vPC FI
vlans FI-103-01-A FI-103-01-B

UCS 5108

SLOT
1
Ixia SLOT
2
!
UCS 5108

SLOT
1
SLOT
2
!

SLOT SLOT
3 4
SLOT SLOT
3 4

SLOT SLOT
SLOT SLOT 5 6
5 6

SLOT SLOT
SLOT SLOT 7 8
7 8

OK FAIL OK FAIL OK FAIL OK FAIL


OK FAIL OK FAIL OK FAIL OK FAIL

40.103.0.7 40.103.0.8
DC103-7 DC103-8
Catalyst 6509e Catalyst 6509e
9 slot 9 slot

DC101-5 DC101-6 40.103.0.19


40.101.0.19 40.101.0.21 40.103.0.21
Nexus C7010 Nexus C7010 DC103-5 DC103-6
10 slot 10 slot
P3 40.101.1.19 P3 40.101.3.21 Catalyst 6509e Catalyst 6509e
P4 40.101.2.19 P4 40.101.4.21 9 slot VSS 9 slot
P9 40.101.5.19 P9 40.101.6.21

DC1-Dist-N7k-001
vdc2 P3 40.103.1.21
P4 40.103.2.21
P9 40.103.5.21

DC1-Dist-C6kE8-103-VSS
P1
P2 DC1-Dist-C6kE7-105-VSS
P15 40.101.2.17
P15 40.101.1.15 P16 40.101.4.17 P36 40.103.1.15 P36 40.103.2.17
P16 40.101.3.15 P36 40.103.5.13
P15 40.101.5.13

BGP Core AS1-1


Core Routing Nexus C7010 Nexus C7010
P16 40.101.6.13
ASR9006
10 slot 10 slot 6 slot
1 DC1-Core-N7k-1 DC1-Core-N7k-2 DC1-Core-ASR9k-3
40.1.0.15 40.1.0.17 40.1.0.13
AS1-2
P51 40.102.9.13
P1 P52 40.102.10.13
P21 40.102.2.17 P45 40.104.5.13
P2 P53 40.102.11.13 P46 40.104.6.13
P21 40.102.1.15

P1
P22 40.102.3.15
P22 40.102.4.17 P54 40.102.12.13

P45 40.104.3.17
DC1-Dist-C4k-107
P23 40.102.6.17
P2 P46 40.104.4.17
P23 40.102.5.15
P24 40.102.8.17 DC1-Dist-C6kE7-106
DC2 P24 40.102.7.15 P45 40.104.1.15

DC1-Dist-N7k-001 P46 40.104.2.15

DC1-Dist-C6kE8-104
N7K with Sup2 M1/F1 vdc2 P3 40.104.2.21
P4 40.104.4.21
P9 40.104.6.21
P3 40.102.1.51 P3 40.102.3.52 P3 40.102.5.53 40.104.0.21
P4 40.102.2.51 P4 40.102.4.52 P4 40.102.6.53
P3 40.102.7.54
P4 40.102.8.54
P3 40.104.1.19
P4 40.104.3.19
40.104.0.19
P15 40.102.9.51 P15 40.102.10.52 DC104-6
P15 40.102.11.53 P15 40.102.12.54 P9 40.104.5.19 DC104-5
40.102.0.51 40.102.0.52 40.102.0.53 40.102.0.54 Catalyst 6509e Catalyst 6509e
DC102-52 9 slot
DC102-51 DC102-53 DC102-54 9 slot
Nexus C7010 Nexus C7010 Nexus C7010
Nexus C7010 10 slot 10 slot
10 slot
10 slot

DC102-27
40.102.0.27
DC104-7 DC104-8
L2 Switch N3K DC104-47 40.104.0.7 40.104.0.8
Non FP vlan, 40.104.0.47 UCS 5108

SLOT
1

SLOT
3

SLOT
5
SLOT
2

SLOT
4

SLOT
6
!

UCS 5108

SLOT
1

SLOT
3

SLOT
SLOT
2

SLOT
4

SLOT
!

5 6
SLOT SLOT
7 8

DC102-706
OK FAIL OK FAIL OK FAIL OK FAIL

FP vlans
SLOT SLOT
7 8

OK FAIL OK FAIL OK FAIL OK FAIL

DC102-701
UCS 5108

!
SLOT SLOT
1 2

SLOT SLOT
3 4

40.102.0.71 N5K DC102-702 DC102-703 DC102-704 N5K 40.102.0.76 SLOT


5
SLOT
6

Ixia
N3k
SLOT SLOT
7 8

OK FAIL OK FAIL OK FAIL OK FAIL

DC102-17 DC102-18 40.102.0.73 40.102.0.74 DC102-705


DC102-47 40.102.0.17 40.102.0.18
40.102.0.72 40.102.0.75 Ixia
40.102.0.47
L2-7004 DC102- DC102-
703, F102 704, F102
6k FEX
FI FI-102-02-B FI-102-01-A
FI FI-102-01-B FI-102-01-A FI-102-01-B
FI-102-02-A 6k
10.2.251.5 10.2.251.6 10.2.251.2 10.2.251.2 10.2.251.3
10.2.251.3
L2-7001 L2-7002 L2-7003
FI-102-02-A FI FI-102-02-B 6k 6k 6k
UCS 5108

SLOT
1
SLOT
2
!
UCS 5108

SLOT
1
SLOT
2
!
UCS 5108

SLOT
1

SLOT
3
SLOT
2

SLOT
4
!
UCS 5108

SLOT
1

SLOT
3
SLOT
2

SLOT
4
!

FI
10.2.251.5 10.2.251.6
SLOT SLOT SLOT SLOT
3 4 3 4

SLOT SLOT SLOT SLOT


5 6 5 6

SLOT SLOT SLOT SLOT


5 6 5 6

SLOT SLOT SLOT SLOT


UCS 5108
7 8 7 8

!
SLOT SLOT
1 2
SLOT SLOT SLOT SLOT
7 8 7 8

UCS
OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL

OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL

SLOT SLOT
3 4

UCS
SLOT SLOT
5 6

SLOT SLOT
7 8

UCS-B-102- UCS-B-102-
OK FAIL OK FAIL OK FAIL OK FAIL

UCS-B-102- UCS-B-102-
FAIL OK FAIL OK FAIL OK FAIL OK

8 7
SLOT SLOT

UCS 5108

01-01 01-02 Ixia


!
SLOT SLOT
1 2
UCS 5108

!
UCS 5108 UCS 5108
SLOT SLOT
6 5 1 2
SLOT SLOT ! !
SLOT SLOT SLOT SLOT
1 2 1 2
SLOT SLOT
3 4

UCS 5108

!
SLOT SLOT

02-02
3 4 SLOT SLOT
1 2

02-01
SLOT SLOT SLOT SLOT SLOT SLOT
3 4 3 4 5 6
4 3
SLOT SLOT

SLOT SLOT
5 6 SLOT SLOT
3 4

UCS 5108 UCS 5108


SLOT SLOT
7 8
SLOT SLOT SLOT SLOT
! !
5 6 5 6
SLOT SLOT SLOT SLOT
1 2 1 2

SLOT SLOT
OK FAIL OK FAIL OK FAIL OK FAIL 7 8 SLOT SLOT
5 6

2 1
SLOT SLOT
OK FAIL OK FAIL OK FAIL OK FAIL
!
SLOT SLOT SLOT SLOT
7 8 7 8
SLOT SLOT SLOT SLOT
UCS 5108 3 4 3 4
SLOT SLOT
7 8

UCS
OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL

OK FAIL OK FAIL OK FAIL OK FAIL

SLOT SLOT SLOT SLOT


5 6 5 6

SLOT SLOT SLOT SLOT

Ixia
7 8 7 8

UCS
OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL

UCS-B-102- UCS-B-102- Ixia Ixia


UCS-B-102- UCS-B-102- 02-01 02-02
01-01 01-02
Figure 2 DC2 Topology
Ixia
UCS 5108
Expand to 2 Distributions:

!

UCS
SLOT SLOT
1 2

SLOT
8
FAIL OK FAIL OK FAIL OK FAIL OK

7
SLOT SLOT
8
FAIL OK FAIL OK FAIL OK FAIL OK

7
SLOT
SLOT
3

SLOT
5
SLOT
4

SLOT
6

Sup720-10G VSS

SLOT SLOT
6 5 6 5
7 8
SLOT SLOT SLOT SLOT

OK FAIL OK FAIL OK FAIL OK FAIL

Sup2T VSS
4 3 4 3
SLOT SLOT SLOT SLOT

2 1 2 1
SLOT SLOT SLOT SLOT

! !

UCS 5108 UCS 5108

UCS
UCS 5108 UCS 5108

! !
SLOT SLOT SLOT SLOT

UCS
1 2 1 2
UCS 5108 UCS 5108

! !
SLOT SLOT SLOT SLOT
1 2 1 2

SLOT SLOT SLOT SLOT


3 4 3 4

SLOT SLOT SLOT SLOT


3 4 3 4

SLOT SLOT SLOT SLOT


5 6 5 6

SLOT SLOT SLOT SLOT


5 6 5 6

SLOT SLOT SLOT SLOT


7 8 7 8

40.201.0.107
SLOT SLOT SLOT SLOT
7 8 7 8
OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL

40.201.0.108
OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL OK FAIL

40.201.0.111 40.201.0.103
FI-201-03-A FI FI-201-03-B 40.201.0.112 40.201.0.104 UCS 5108
UCS 5108

FI-201-03-B
SLOT SLOT
!
1 2
SLOT SLOT

FI-201-02-A FI-201-02-B FI-201-03-A


1 2

SLOT SLOT
3 4
SLOT SLOT
3 4

UCS
SLOT SLOT
5 6
SLOT SLOT

Ixia
5 6

Ixia
SLOT SLOT
7 8
SLOT SLOT
7 8

OK FAIL OK FAIL OK FAIL OK FAIL

OK FAIL OK FAIL OK FAIL OK FAIL

FI FI
UCS 5108

!
UCS 5108
SLOT SLOT
1 2

40.201.0.109 40.201.0.110
!
SLOT SLOT
1 2

SLOT SLOT
3 4

40.201.0.17 40.201.0.18
SLOT SLOT
3 4

DC201-6,40.201.0.27 40.201.0.28
SLOT SLOT
5 6

SLOT SLOT
5 6

DC201-5, FI-201-01-B
SLOT
7
SLOT
8

DC201-18 FI-201-01-A
SLOT SLOT