Deploying a

Cisco Data Center

 Nexus 7000 Nexus 7000
PSU FAN SUP PSU FAN SUP
IOM IOM

FAN
STATUS
N7K-Spine-A FAN
N7K-Spine-B
ID STATUS ID
Cisco Nexus 7004 Cisco Nexus 7004
N C N
N7K-SUP2E E M LI AC
T U
R E N7K- US E V CONSOLE
LI AC
T EJ U E

Pod Topology
CONSOLE KM M T 1 EQ 1 T
D
S
I U
JT U E SECTT T
SUP2E
IT
D
MS MT K EE
CTSR EJ EUSTC S
EEST
1 TM
E JR E A E Q E
YTG C AR
G S P
SE T 2 Q E
RE TT
V
IT
T

R
SERIAL PORT
ETH
MGMT 2
EQ
RE
LOG
SLOT0
1 S WI
SERIAL PORT
ETH
MGMT
E
ASH HOST PORTS
LOG
FL SLOT
0
EJ
R

S T T N FLASH ET
JE T N
N7K- A CONSOLELI AC HOST U E N7K-SUP2E LI AC T
T T EC R T 1
I
SUP2E SEMCM AC P
KW PORTS T CONSOLE K R E
D TUS MG CT SE ID S CT SE
E Q T

2 TA T VE YI JE S U TE CVEAYR S
TA SS EQ
E TC
SM TTM P
ST
S
YS
R
A W P S
SERIAL PORT
ETH
MGMT
1
E

S
R
T
EQ
2 W TIM
G
SERIAL
PORT
MGMT
HOST
PORTS
2 U

EJ U
R
LOG
FLASH
SLOT0
ES
E
T
E
R
ETH
ESD N7K- U PORT N7K- PORT GROU3P PORT
F312FQ- E GROU6P ESD F312FQ- PORT GROU4P GROU6P
25 E 10 11 12 25 4 5 6 7 8 10 11 12
4 5 Q6 7 8 9
PORT PORT GROU3P PORT GROU1P
1 GROU1P 2 9 R
2 GROU4P
PORT 1 2

3 STATUS
ID
3

PORTU2P
HOST
PORTS
LOG
SLOT0
FLASH
PORT 3 STATUS
ID
3

PORT U2P PORT

GRO GROU5P GRO GROU5P
PORT GROU1P PORT PORT GROU1P PORT GROU3P PORT
PORT GROU2P PORT GROU3P GROU5P PORT GROU2P PORT GROU4P GROU5P
PORT GROU4P

4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 F3 15 16 17 18 19 20 21 22 23 24 25 26 27 28 4 1 2 3 4 5 6 7 8 9 10 11 12 13 F3 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 PORT 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 PORT
I I I GROU6P I I GROU6P
I
N7K-AC-3KW IN N7K-AC-3KW IN
N7K-AC-3KW IN N7K-AC-3KW IN N7K-AC-3KW IN N7K-AC-3KW IN N7K-AC-3KW IN N7K-AC-3KW IN
OUT FAULT ID OUT FAULT ID OUT FAULT ID OUT FAULT ID OUT OUT
I FAULT ID
I FAULT ID OUT FAULT ID OUT FAULT ID

100 -240V ~ 50 /60 100 -240V ~ 50 /60 100 -240V ~ 50 /60 100 -240V ~ 50 /60 100 -240V ~ 50 /60 100 -240V ~ 50 /60 100 -240V ~ 16 A 100 -240V ~ 16 A
16 A Hz 16 A Hz 16 A Hz 16 A Hz 16 A Hz 16 A Hz 50 /60Hz 50 /60Hz

3/ {1,5,9} 3/ {1,5,9}
3/ {2,6,10} 3/ {2,6,10}

1/51 1/51 1/52

1/52

N9K-Leaf-A N9K-Leaf-B
Nexus 9300 1/49-50 1/49-50 Nexus 9300
PoX0

1/9-16 1/9 -16

v PC X1
v vPC X2 vPC X4
vPC X3
CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
32 32

UCS6200-A UCS6200-B
ID ID
STAT STAT

18 UCS 6200 UCS 6200 1/

1/17- 17-18
5 1/1-8 1/1-8
PoX PoX6
UCS 5108
Nexus 5500 A
2
4 x B240 M2
A
2 A
2
Nexus 5500
CISCO NEXU5S548
NUP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 ! ! ! ! CISCO NEXU5S548
NUP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
30 31 32 UCS 5108 UCS B440 30 31 32
3 20G
A
G G G
M1/M2 3
00 S 00 S 00 S
0 A
3 S 3 A 3 A 3 A

N5K-SAN-B
0
B
A 3 10k
-0G 0
-B3 10k
0G
A 0
-B3 10k
0G
A 0
-B3 10k
0G
A

N5K-SAN-A
0
S 0 0
S0 0
S0 0
S0
3 3 3 3
SL !
1 ! ID
ID SLOT SL
OT
STAT 2 STAT
OT SLOT 2
1 ! Console
Reset

1/19 1/19
A
2 A
2 A
2
! ! ! UCS B440 M1/M2
20G
A G G G !
0 A S 00 S 00 S 00 S
3 3 A 3 A 3 A
-03 10S
B
0
A k -B
03 1S
0
A
0
k -B
03 1S
0
A
0
k -B
03 1S
0
A
0
k
G
0
3
0 G
0
3
0 G
0
3
0 G
0
3
0

!
SLOT SLOT
SLOT SLOT 2
3 1 !
Console 4
Reset

UCS B440
2 ! 2 ! 2 ! 2 !
0A
G 0A
G 0A
G 0A
G M1/M2
-3S
0
0 S
03 0k -3S
0
0 S
03 0k -3S
0
0 S
03 0k -3S
0
0 S
03 0k
AG
B
A 1 AG
B
A 1 AG
B
A 1 AG
B
A 1
00 00 00 00
3 3 3 3
OT SLOT SLOT
5 1 !
SL SLOT
2 6
! Console
Reset
A
2 A
2 A
2
! ! !
UCS B440
20G
A
G G G !
00 S 00 S 00 S M1/M2
0 A
3 S 3 A 3 A 3 A
-A
03 10S
B
0 k -A
03 10S
B
0 k -A
03 10S
B
0 k -A
03 10S
B
0 k
G0 G0 G0 G0 !
0
3 0
3 0
3 0
3
SLOT SLOT
OT SL

SL OT
1 2 8
7 !
Console
Reset

S-X9
D 0
7
8-K9
SK9
D-X9 STATUS
0
7
8- OK OK OK OK S-X9
D 0
7
8-K9
0-Gbps8-portFCoEModule1234L
1 K5678
IN FAIL FAIL FAIL FAIL 0-Gbps8-portFCoEModule1234L
1 STATUS K5678
IN
SK9
D-X9 0
7
8-
STAT
US
portFCMo
8Gbps24-
96K91/2/4/
X9224-
DS- STAT
US
dule
0-Gbps8-portFCoEModule1234L
1 K5678
IN portFCMo
8Gbps24-
96K91/2/4/
X9224-
DS-

VSAN 901
dule
STAT
USL 0-Gbps8-portFCoEModule1234L
1 K5678
IN

VSAN 902
K1234567891
IN 0111
21
31
41
51
61
71
81
92
02
12
STAT
USL
K1234567891
IN 0111
21
31
41
51
61
71
81
92
02
122
324
32
24

SF2
X9
-K9
S
D3
5
0-
E1
EMV 0MMG
SSAPWRMT
SEGCONSOL
Y0T/1KM
IN
L C
O
TUSTTIMET1
ES
TACR USB
1 S3
D
X9
-K9
SF2
5
--
0
ACTUSB2 E1
EMV T
0MMG
MT
SEGCONSOL
0/1 KM
IN
L C
O
TUSTTIMET1
ESYUSB1
TACR
SUPE
RVIS
OR2 WR
SSAP ACTUSB2
S3
D
X9
-K9
SF2
5
--
0 SUPE
RVIS
OR2
EMV
E1
0/1T
0MMG
MT
SEGCONSOL
KM
IN
L C
O
TUSTTIMET1
RES
TAYC USB
1 S3
D
X9
-K9
SF2
5
--
0
WR
SSAP YEMV
SUPE
RVIS
OR2
ACTUSB2 E1
WR
SSAP 0MMG
MT
TL
SEGCONSOL
0/1 KM
INC
O
TUSTTIMET1
ES
TACR USB1
SUPERVISOR2 ACTUSB2

Core 9506 Core 9506

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Lab 1
VXLAN EVPN
VXLAN Overview Classic Ethernet Frame
802.1Q
Limitations in Layer-2 Data Centers DMAC SMAC Etype Payload CRC

• Traditionally VLAN is expressed

over 12 bits (802.1Q tag)
• Limits the maximum number of Destination MAC (DMAC)

segments in a Data Center to 4096 Source MAC (SMAC)

VLANs TPID
TCI

4 bytes 802.1Q 0x8100 PCP CFI VID

• Spanning-Tree for loop prevention (16 bits) (3 bits) (1 bits) (12 bits)

Ether Type (Etype)

• Inefficient use of links
Data (Payload)
• Slow convergence CRC/FCS

VLAN ID
12 bits

TPID = Tag Protocol Identifier, TCI = Tag Control Information,

PCP = Priority Code Point,
CFI = Canonical Format Indicator, VID = VLAN Identifier
VXLAN Overview
Enter VXLAN... • VXLAN leverages the VXLAN
Network Identifier (VNI/VNID) field
• Uses Layer 3 Data Center Fabric
• Total address space of 24 bits
• No need for Spanning-Tree Protocol
• Support of ~16M segments
• Equal-Cost Multi-Path (ECMP)
• Uses IP/UDP header (MAC in IP/UDP)
• Divided into Underlay and Overlay(s)

LT
VXLAN Overview
Enter VXLAN... • VXLAN leverages the VXLAN
Network Identifier (VNI/VNID) field
• Uses Layer 3 Data Center Fabric • Total address space of 24 bits
• No need for Spanning-Tree Protocol
• Support of ~16M segments
• Equal-Cost Multi-Path (ECMP)
• Uses IP/UDP header (MAC in IP/UDP)
• Divided into Underlay and Overlay(s) 802.1Q header removed

802.1Q 802.1Q
C Classic Ethernet
DMA SMAC Etype Payload CRC
Frame

Original Layer 2 Frame

VXLAN Outer Outer C CRC

UDP VXLAN DMA SMAC Etype Payload
Frame MAC IP (new)

14B 20B 8B 8B 14B 4B

20B + 8B +8B + 14B* = 50 Bytes

of total overhead
 Next-Hop MAC Address

Src VTEP MAC Address

VXLAN Frame Format Dest. MAC Address 48

Src. MAC Address 48

VLAN Type 14 Bytes

0x8100 16 IP Header
(4 Bytes Optional) 72
Misc. Data
VLAN ID
16
Outer MAC Header Tag
Protocol 0x11 (UDP) 8
Ether Type
16 Header
0x0800

n de ay
Checksum 16 20 Bytes
lr
50 (54) Bytes of Overhead

Source IP 32
Outer IP Header U Src and Dst
addresses of the
Source 32 VTEPs
16 Dest. IP
Port

UDP Header VXLAN Port 16 Hash of the inner L2/L3/L4 headers of

8 Bytes the original frame.
Enables entropy for ECMP Load
UDP Length 16
balancing in the Network.
VXLAN Header UDP 4789
Checksum 0x0000 16
Overlay

VXLAN Flags
RRRRIRRR 8
Allows for 16M
Original Layer-2 Frame possible
Reserved 24 Segments
8 Bytes
VNI 24

Reserved 8
VXLAN EVPN Overview
• Workload MAC and IP Addresses learnt by VXLAN Edge Devices (NVEs)
• Multi-Protocol BGP (MP-BGP) based Control-Plane using EVPN NLRI (Network
Layer Reachability Information)
• Forward based on MAC or IP address learnt via Control-Plane (MP-BGP EVPN)

• Make Forwarding decisions at VTEPs for Layer-2 (MAC) and Layer-3 (IP);
Integrated Route/Bridge (IRB)
• Bridge – L2 VNI
• Route – L3VNI

• Reduce impact of ARP on the Network using ARP Suppression

• Leverage Distributed Anycast Gateway capabilities
• Host/IP Mobility – Location (VTEP), Identifier (MAC, IP of End-Host)
VXLAN EVPN Overview – Control Plane
• Cisco’s VXLAN/EVPN does ide
prov automated Route (RD) VRF Info
Name: VRF-A
Distinguisher VRF Info
Name: VRF-A RD: 15:10.0.0.2 (auto)
• Automatic uses Type 1 format RD: 3:10.0.0.1 (autoIm) pRRRoute-TRarRget 65500:50000
V5R50F0I:n50fo000 (aEuxtpo)Route-Target 65500:50000
Imp Route-Target 6(auto)
• 4-byte IP Address (Router ID) Exp Route-Target 6N5a5m00e::50000 (auto)VRF-
(auto)
• 4-byte Value (VRF ID) A RD: 62:10.0.0.3 (auto)
Imp Route-Target 65500:50000 (auto)
vrf context VRF-A Exp Route-Target 65500:50000 (auto)
vni 50000
rd auto V1 V2
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto RR BGP Route-Reflector
route-target both auto evpn
V3 iBGP Peering
VXLAN EVPN Overview – Control Plane
BGP Advertisement
• Cisco’s VXLAN/EVPN does provide VPN-EVPN: RD:[MAC_A][IP_A]
BGP Next-Hop: V1
automated Route Target (RT) Route Target: 65500:50000
• 8-byte Route Target (2 x 4-byte) Label (L3VNI): 50000

• ASN : VNI RR RR

MAC_A / IP_A >> LOCAL

Route-Type2
vrf context VRF-A
vni 50000 V2
V1
rd auto
address-family ipv4 unicast MAC_A / IP_A >> V1
Route-Type2
route-target both auto
route-target both auto evpn
address-family ipv6 unicast RR BGP Route-Reflector
route-target both auto
route-target both auto evpn V3 iBGP Peering

Host A
MAC_A /
IP_A
VXLAN EVPN – ARP Handling on Lookup “Miss” ( 1)
MAC, IP VNI NH
RR RR
Missing MAC_A, IP_A 30001 IP_V1
MAC, IP “B”VNI NH
MAC_C, IP_C 30001 IP_V3
MAC _C, IP_C 30001 IP_V3
MAC_Y, IP_Y 30002 IP_V3
2 MAC_Y, IP_Y 30002 IP_V3

ARP Request for IP_B

ARP Request for IP_B
Src MAC: MAC_A V1 V2 Src MAC: MAC_A
Dst MA C: FF:FF:FF:FF:FF:FF Dst MAC: FF:FF:FF:FF:FF :FF

1 2

MAC, IP VNI NH

MAC_A, IP_A 30001 IP_V1

Host A
MAC_A / IP_A V3 Host B
MAC_B / IP_B
ARP Request
Request for
f IP_B
Src MAC: MAC_A
Dst MAC: FF:
FF:FF:FF:FF:FF:FF
1 ARP Request sent for IP_B sent from Host A
Virtual Switch

2 Miss of IP_B. Forward ARP Request to all

Ports except source-port (ARP snooping)
Host C Host Y
MAC_C / IP_C MAC_Y / IP_Y
VXLAN EVPN – ARP Handling on Lookup “Miss” ( 2)
RR RR MAC, IP VNI NH
MAC, IP VNI NH MAC_A, IP_A 30001 IP_V1
MAC_C, IP_C 30001
30000 IP_V3
V3 MAC_C, IP_C 30001 IP_V3
4
MAC_Y, IP_Y 30002
30001 IP_V3
V3 MAC_Y, IP_Y 30002 IP_V3
ARP Response for I P_B MAC_B, IP_B 30001 IP_V2 ARP Response from IP_B
Src MAC: MAC_B
Dst MAC: MAC_A V1 V2 Src MAC: MAC_B
Dst MAC: MAC_A

4
3
MAC, IP VNI NH

MAC_A, IP_A 30001

30000 IP_V1
V1

MAC_B, IP_B 30001 IP_V2

Host A
MAC_A / IP_A V3 Host B
MAC_B / IP_B

3 ARP Response is sent to V2

Virtual Switch

4 V2 will populate this information in the

control-plane (learn) and forward it subsequently
Host C Host Y
MAC_C / IP_C MAC_Y / IP_Y
VXLAN EVPN – Packet Forwarding (Bridging)

RR RR MAC, IP VNI NH
MAC, IP VNI NH

MAC_A, IP_A 30001 Local MAC_B, IP_B 30001 Local

MAC_B, IP_B 30001 IP_V2 MAC_A, IP_A 30001 IP_V1

SMAC: MAC_A SMAC: MAC_A

DMAC: MAC_B DMAC: MAC_B

SIP: IP_A
V1 V2 SIP: IP_A
DIP: IP_B DIP: IP_B

1
2 3 4
SMAC: MA C_V1 SMAC: ho p-by-hop

Underlay
DMAC: hop-by-hop DMAC: MAC_V2
y
l ar
SIP: IP_V1 SIP: IP_V1
e
DIP: IP_V2 d DIP: IP_V2
n

Host A U
MAC_A / IP_A
UDP
V3 UDP Host B
MAC_B / IP_B
VXLAN VNID: 30001 VXLAN VNID: 30001

Overlay
SMAC: MAC_A SMAC: MAC_A
Overlay

DMAC: MAC_B DMAC: MAC_B

SIP: IP_A SIP: IP_A

DIP: IP_B DIP: IP_B
VXLAN EVPN – Packet Forwarding (Routing)

RR RR MAC, IP VNI NH VRF

MAC, IP VNI NH VRF

MAC_A, IP_A 30001 Local 50001 MAC_A, IP_A 30001 Local 50001
MAC_F, IP_F 30005 E1/4 50001
MAC_F, IP_F 30005 IP_V2 50001
SMAC: MAC_A SMAC: MAC_GW
DMAC: MAC_GW DMAC: MAC_F

SIP: IP_A
V1 V2 SIP: IP_A
DIP: IP_F DIP: IP_F

1
2 3 4
SMAC: MA C_V1 SMAC: ho p-by-hop

Underlay
DMAC: hop-by-hop DMAC: MAC_V2
y
l ar
SIP: IP_V1 SIP: IP_V1
e
DIP: IP_V2 d DIP: IP_V2
n

Host A U
MAC_A / IP_A
UDP
V3 UDP Host F
MAC_F, IP_F
VXLAN VNID: 50001 VXLAN VNID: 50001

Overlay
SMAC: MAC_A SMAC: MAC_GW
Overlay

DMAC: MAC_GW DMAC: MAC_F

SIP: IP_A SIP: IP_A

DIP: IP_F DIP: IP_F
Lab 1 Modules
1. Configure MP-BGP EVPN on the Spines
2. Configure MP-iBGP EVPN on the VTEPs
3. Configure VLAN to VXLAN VNI Mapping on the VTEPs
4. Configure NVE Tunnel Endpoint Interface on the VTEPs
5. Configure VXLAN Bridging on the VTEPs
6. Configure VXLAN Routing on the VTEPs
7. Verify VXLAN Configuration
Lab 2
UCSM LAN
UCSM
UCSM Service Profile Flow
UCS Manager Service Profile High Level Flow

Create QoS Policy

Create Server
Pool “PODXY”
Create Network
Control Policy
Create Boot
Policy “PODXY”
Create FI-A MAC
Address Pool Create FI-A
Create FI-B MAC vNIC Template
Address Pool Create FI-B
vNIC Template
Create Service Profile
Create VLANs
PODXY-1 from Template
Create Create Service Profile
WWNN Pool Template “PODXY”
Create FI-A Create Service Profile
WWPN Pool
Create FI-A PODXY-2 from Template
Create FI-B vHBA Template
WWPN Pool
Create FI-B
vHBA Template
Create VSANs
Create UUID
Pool

Create
Maintenance Lab 3 Lab 4 Lab 5
Policy
 UCSM LAN

N9K-Leaf-A N9K-Leaf-B
Nexus 9300 Nexus 9300

v PC X1
v vPC X3 vPC X2 vPC X4
CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
31 32 31 32

Fabric Ports vPC ID

UCS6200-A UCS6200-B Fabric Ports vPC
ID
STAT

ST

UCS 6200 UCS 6200

AT

A 1/22-25 X1 A 1/22-25 X3
UCS 5108
B 1/22-25 X2 A
2
4 x B240 M2
A
2 A
2
B 1/22-25 X4
UCS 5108 ! ! ! !
UCS B440
2
0A M1/M2
0 S 0 S 0 S
G 0G
3 AB 0G
3 AB 0G
3 AB
0 S
0 3 A
-30G 0k 0
-03 0k 0
-03 0k 0
-03 0k
A 1 B A0G 1 A0G 1 A0G 1
0
S 0 0
S 0
S 0
S !
3 3 3 3 !
SL
1
SLOT SL
2 2OT
OT SLOT !
1 Console
Reset
A
2 A
2 A
2
! ! !
UCS B440
2 !
0A
G 0G 0 S
3 AB 0G 0 S
3 AB 0G 0 S
3 AB M1/M2
0 0S
-303 S -03 10S -03 10S -03 10S
0
A 1A
kB 0
A k 0
A k 0
A k
G
0
0 G
0
0 G
0
0 G
0
0
3 3 3 3

!
SL
3 SLOT
1 OT
OT SLOT SL
! Console 2 4
Reset

UCS B440
2 ! 2 ! 2 ! 2 !
0A
G 0A
G 0A
G 0A
G M1/M2
3
0 S 3
0 S 3
0 S 3
0 S
-0
30 0
S
AkB -0
30 0
S
AkB -0
30 0
S
AkB -0
30 0
S
AkB
A
G
00 1 A
G
00 1 A
G
00 1 A
G
00 1
3 3 3 3
5OT SLOT SLOT
!
SL
1
SL OT
2 6
!
Console
Reset
A
2 A
2 A
2
! ! !
UCS B440
2 !
0A
0 S 0 S 0 S M1/M2
G 0G
3 AB 0G
3 AB 0G
3 AB
0 0
S
0 3 A
-30G k 0 3 0k
-0G 0 3 0k
-0G 0 3 0k
-0G
A 1 B A 1 A 1 A 1 !
0
S 0 0
S 0 0
S 0 0
S 0
3 3 SLOT
3 3 SLOT
OT SL

SL 1 2 OT
7 8
! Console
Reset

OK OK OK OK
FAIL FAIL FAIL FAIL

VLANs

11-12,21-22
 UCSM LAN
N9K-Leaf-A N9K-Leaf-B
MAC Pool First MAC Address
Nexus 9300
MACNexus
Pool9300 First MAC Address

PODXA-A-MAC-POOL 00:25:B5:0X:AA:00 PODXB-A-MAC-POOL 00:25:B5:0X:BA:00

PODXA-B-MAC-POOL 00:25:B5:0X:AB:00 PODXB-B-MAC-POOL 00:25:B5:0X:BB:00

vPC X1 vPC X3 vPC X2 vPC X4
CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
31 32 31 32

UCS6200-A ID
UCS6200-B
ID
STAT STAT

UCS 6200 UCS 6200

UCS 5108
A
2
4 x B240 M2
A
2 A
2
UCS 5108 ! ! ! !
UCS B440
2
0A
0 S 0 S 0 S M1/M2
G 0G
3 AB 0G
3 AB 0G
3 AB
-3003 AS
S
0 -03 0S -03 0S -03 0S
0
A 1k B 0
A 1k 0
A 1k 0
A 1k
G
0
3
0 G
0
3
0 G
0
3
0 G
0
3
0 !
SL !
1
SLOT SL
2 2OT
OT SLOT
1 ! Console
Reset
A
2 A
2 A
2
! ! !
UCS B440
20G
A G G G !
00 S 00 S 00 S M1/M2
0 A
3 S 3 A 3 A 3 A
-03 0S
B
0 -0
03 0S
B -0
03 0S
B -0
03 0S
B
A 1k A 1k A 1k A 1k
G
0
3
0 G
0
3
0 G
0
3
0 G
0
3
0

!
SL
3 SLOT
1 OT
OT SLOT SL
! Console 2 4
Reset

UCS B440
2 ! 2 ! 2 ! 2 !
0A
G 0A
G 0A
G 0A
G M1/M2
0 S 0 S 0 S 0 S
0 3 A
-30G 0k 0 3 A
-30G 0k 0 3 A
-30G 0k 0 3 A
-30G 0k
A 1 B A 1 B A 1 B A 1 B
0
S 0 0
S 0 0
S 0 0
S 0
3 3 3 3
OT SLOT SLOT SL
5 1 !
SL OT
2 6
! Console
Reset
A
2 A
2 A
2
! ! !
UCS B440
20G
A G G G !
00 S 00 S 00 S M1/M2
0 A
3 S 3 A 3 A 3 A
-03 10S
B
0 k -B
03 1S
0
0
k -B
03 1S
0
0
k -B
03 1S
0
0
k
A A A A
G
0
0 G
0
0 G
0
0 G
0
0
3 3 3 3
SLOT SLOT
7 1 !

SLOT SLOT
2 8
! Console
Reset

OK OK OK
FAIL FAIL FAIL OK FAIL
Lab 2 Modules
1. Configure Named VLANs
2. Configure Ethernet Uplink Port-Channels
3. Create MAC Address Pools
4. Create vNIC Templates
Lab 3
UCSM SAN
UCSM Service Profile Flow
UCS Manager Service Profile High Level Flow

Create QoS Policy

Create Server
Pool “PODXY”
Create Network
Control Policy
Create Boot
Policy “PODXY”
Create FI-A MAC
Address Pool Create FI-A
Create FI-B MAC vNIC Template
Address Pool Create FI-B
vNIC Template
Create Service Profile
Create VLANs
PODXY-1 from Template
Create Create Service Profile
WWNN Pool Template “PODXY”
Create FI-A Create Service Profile
WWPN Pool
Create FI-A PODXY-2 from Template
Create FI-B vHBA Template
WWPN Pool
Create FI-B
vHBA Template
Create VSANs
Create UUID
Pool

Create
Maintenance Lab 3 Lab 4 Lab 5
Policy
SAN Concepts
Legend
VNP_PORT
VF_PORT
zone name PODXA-SP-1 vsan 901 VE_PORT zone name PODXA-SP-1 vsan 902
member pwwn member pwwn
50:0a:09:81:8a:72:87:8D
FCoE 50:0a:09:82:8a:72:87:8D
member pwwn FC member pwwn
20:0X:00:25:B5:0B:0A:00
20:0X:00:25:B5:0A:0A:00
zone name PODXA-SP-2 vsan 901 zone name PODXA-SP-2 vsan 902
member pwwn
member pwwn
50:0a:09:82:8a:72:87:8D
50:0a:09:81:8a:72:87:8D
No FC Domain ID member pwwn
member pwwn
20:0X:00:25:B5:0B:0A:01
20:0X:00:25:B5:0A:0A:01
zoneset name PODX-B vsan 902
zoneset name PODX-A vsan 901
member PODXA-B-SP-1
member PODXA-A-SP-1 CISCO UCS6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 CISCO UCS6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

member PODXA-B-SP-2
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32

member PODXA-A-SP-2 ID
NPV UCS6200-A ID
NPV
PV UCS6200-B
ST
AT

UCS 6200 N 6200

STA
T

UCS
zoneset activate name PODX-B
zoneset activate name PODX-A
vsan 902
vsan 901 UCS 5108
Nexus 5500 4 x B240 M2 Nexus 5500
CISCO NEXU5S548
NUP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 CISCO NEXU5S548
NUP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
24 25 26 27 28 29 30 31 32 UCS B440 M1/M2 ! 24 25 26 27 28 29 30 31 32

N5K-SAN-A N5K-SAN-B
!
2
A ! 2 A ! 2 A 2 A

WWPN-A WWPN-B
3 03
3
0Gk
-0
0 S0
A
S BG 00
-0
3G
3SS
A
k B0G 300G
-0 SkB 0G
03SA -0300G0SSB
3Ak 0G 3

NPI
UCS 5108 A 10 30 A 10 30 A 10 30

ID
STA
T
NPIV A 10 30

! !
ID
STA
T
NPIV
WWNN
SLOT SLOT

WWNN
SLOT SLOT
1 1 !
2 2

WWPN-A
Console Reset

0G
2
!A ! 2 A
0G ! 2 A0G UCS B440 M1/M2 ! 2 A0G
33
0 k
-0
0 AS 0G
S B 3003 S
-0 A
kS 0B
G 33
-000 Sk
AS 0B
G -03003S k
AS 0B
G
A 10 30 A 10 30 A 10 30 A 10 30

WWPN-A WWPN-B !

WWPN-B WWNN
SLOT
SLOTSLOT
3 1 2 4
!
Console Reset

WWNN
2
!A ! 2 A ! 2 A UCS B440 M1/M2 ! 2 A
00
A
3G10
-0
03 S
A
S
k 3G
B00 A0G
3-0
001AS3
3S 0kG
B 00 01
A-0
30G0S
33SG
A Bk 00
0 010G3
A 3-0 A0S
S 3G
B0k

SLOT
SLOT
SLOT
SLOT 5
1
WWPN-A WWPN-B !
!

2 6
Console Reset

WWPN-A WWPN-B
S-X9
D 0
7
8-K9
STATUS
Eth1/28
SLOT SLOT
2G
!A
0
30
0 10
-0
0
A3 AkS3G
S B0
0

WWNN
! 2 A0G
30
A-001
3SA3
SkGB00
0

WWPN-A
! 2 A0G
A-01300S
33AG
Sk B00
0
UCS B440 M1/M2 !

!
2 A0G
1 303
A -0 AS0kB
S03G

Eth1/28 S-X9
D 0
7
8-K9

WWNN
0D
1
S
D -Gbps8-portFCoEModule1234L
-X90
7
80 STATUS
STATSUS-X9 8-K9
7-K9 K5678
IN SLOT SLOT
0-Gbps8-portFCoEModule1234L
1
S
D -X9
0
7
8-K9 K5678
IN
0
1
-p
Gbps8
CoEM
-STATUS
ortF
odule 1234L K5678
IN 7 1 2 8
S-X9
D K9 0
7
8- ! 0
1
-p
Gbps8
CoEM
SortF
D --X9
odule0
7
8-K9
STATUS 1234L K5678
IN
0US
1
portFCModule
DS-X9224-96K91/2/4/8Gbps24-
STAT -Gbps8-portFCoEModule1234L K5678
IN Console Reset STATUS
0STATUS
1
-p
Gbps8
CoEM
-L
ortF
odule 1234L 4IN
K1
55678 0US
1
portFCModule
DS-X9224-96K91/2/4/8Gbps24-
-Gbps8-portFCoEModule1234L K5678
IN

WWPN-B WWNN
24-K1234567891
IN
4/8Gbps
96K91/2/
X9224-
DS- 011121
31 61
17181
92
02122
32
4 STAT

VSAN 901
odule
portFCM 0DS-X9224-
1
-p
Gbps8
CoEM
-L
ortF
portFCModule
96K91/2/4/8Gbps24-
STATUS
odule
K1234567891
IN 1234L
0111214IN
31 K1
55678
61
171
81
92
02
122
32
4

NPIV VSAN 902

STAT
US K1234567891
IN
L 011121
31
41 5161
71
8192
02
12232
4

NPIV
STAT
US K1234567891
IN
L 0111
21
31
41
51
61
71
819202122
32
4
S-X9
D 3
5
0-SF2-K9
TUSTTIMET1USB1
TAYCRES
S-X9
D SEMEGCONSOLE1
0VM MGMT
T ACTUSB2 0/1
SUPERVISOR2
SSAPWR
3
5-SF2-K9 0L KM
IN C
O OK FAIL OK FAIL OK FAIL S-X9
D 3
5
0-SF2-K9
SSF2-
D -X9
K9 3
5
0SSAPWR
-SEMEGCONSOLE1
TUSTTIMET1USB1
M MGMT
T ACTUSB2 0/1
SUPERVISOR2
TAYCRES
VMMGMT 0L KM
INC
O S-X9
D TUSTTIMET1USB1
TAYCRES
SEMEGCONSOLE1
0V M MGMT
T
SUPERVISOR2
SSAPWR
3
5-SF2-K9 ACTUSB2 0/10L
KM
IN C
O
T1VUSB1
TUSTTIME
RES
TAYC OK FAIL SSF2-
D -X9
K9 3
5
0SSAPWR
-SEMEGCONSOLE1
TUSTTIMET1USB1
M0MGMT
T 0/1
0LKM
INC
O

WWPN-A WWPN-B
T SUPERVISOR2
TAYCRES
T1VTMMG
MT
OLE1
SEGCONS
/1
TUSTTIME
RES
TAYC KACTUSB2
0L
IN C
M
O USB
1
S-X9
D
SUPERVISOR2
SSAPWR
SEMEGCONSOLE1
3
5
0-SF2-K9
ACTUSB2 0/1
0L
KM
IN C
O
SSAPWR
SEMEGCONSOLE1
T SUPERVISOR2
S-X9
D 3
5
0-SF2-K9
SSAPWR
EMV ACTUSB2
T1VUSB1ACTUSB2 0/1
MMGMT
SUPERVISOR2
TUSTTIME
RES
TAYC 0LKM
IN C
O
SUPERVISOR2
SSAPWR
EMV T1TMMGMT
OLE1
SEGCONS
0/1
0L
TUSTTIME KM
INC
O ACTUSB2
USB1
RES
TAYC

Core 9506 Core 9506

WWNN
FC Domain ID FC Domain ID

World Wide Port Name World Wide Node Name

 UCSM SAN

Fabric Ports vPC Fabric Ports PO

A 1/17-18 X7 B 1/17-18 X8
VSAN VSAN

901 902
CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
30 31 32 30 31 32

ID
UCS6200-A ID
UCS6200-B
STAT STAT

UCS 6200 UCS 6200

UCS 5108
Nexus 5500 4 x B240 M2 Nexus 5500
CISCO NEXU5S548NUP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 CISCO NEXU5S548NUP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
29 30 31 32 0G S
2
!A ! 2 A0G S ! 2 A0GS UCS B440 M1/M2
! 2 A 0GS 29 30 31 32
3
UCS 5108
-33
0
00 S
A BG
k 0 0 0
A 1 3
WWPN-A 0 0
A 1 3
3-0
03 S
0
Ak B0G 0 0
A 1 3
WWPN-B 0 0
A 1 3
30
-003ASkB 0G 30-0 3
AS k
0
B 0G 3

ID

STAT
N5K-SAN-A SLOT
SLOT
SLOT
SLOT
!
!

ID
STAT
N5K-SAN-B
1 1 !
2 2
Console Reset

2G
!A
0
-300 S
0G
AS BG
3 k 00 0
0 A 1 3
G
03 k0 00
A 1 3
!

WWNN
2 A0G
-030 SASB G
03 0k 00
A 1 3
!
03 0 k0
A 1 3
2 A0G
-0 3S
0 A
S
B
UCS B440 M1/M2
!

!
SLOT
2 A 0G
-0 S30 BAS

SLOT SLOT
3 1 2 4

WWPN-A WWPN-B !
Console Reset

2G S
!A
0 ! 2 A0G S ! 2 A0GS UCS B440 M1/M2
! 2 A 0GS
-300 S
A G B 3-0
0 S
A G B 3-00 AS BG 30-0 AS BG
03 10k 300
A A 031 03
k 00 A 103 30k 00 A 1 033 0k 00

SLOT
SLOT
5
1
SLOT
SLOT WWNN !
Console
2 6
Reset
!

2G S
!A
0 ! 2 A0G S ! 2 A0GS UCS B440 M1/M2
! 2 A 0GS
-300 S
A BG G 3-0
0 S
A B G 30
-0 AS B 30-0 AS B
03 k 00 0 03 k0 00 03 0k 00 03 0 k0

WWPN-A WWPN-B
G
0 A 1 3 A 1 3 A 1 3 A 1 3

S-X9
D 0
7
8-K9
STATUS
Eth1/28 SLOT
SLOT
7
SLOT
SLOT
1 !
Console
2
Reset
8 Eth1/28 S-X9
D 0
7
8-K9

WWNN
0D
1S-Gbps8-portFCoEModule1234L
-X9
0
7
8-K9 K5678
IN STATUS
S-X9
D 0
7
8-K9 0-Gbps8-portFCoEModule1234L
1 K5678
IN
STATUS S-X9
D -X9 80
7
8-K9
-K9
STATUS
0-Gbps8-portFCoEModule1234L
1 K5678
IN 0D
1S 07
portFCoEModule1234L
-Gbps8-
STATUS K5678
IN
0D
1 -Gbps8-portFCoEModule1234L K5678
IN SK9
D-X9 0
7
8-
STATUS
STAT
SUS
-X9
0
7
8-K9
portFCoEModule1234L
0
1 -Gbps8- K5678
IN
DS-X9224-96K91/2/4/8Gbps24-portFC Module 0-Gbps8-portFCoEModule1234L
1 STATUS K5678
IN

VSAN 901
portFCMo
8Gbps24-
96K91/2/4/
X9224-
DS-
dule DS-X9224-96K91/2/4/8Gbps24-portFC Module

VSAN
902902
OK FAIL OK FAIL OK FAIL OK FAIL portFCMo
8Gbps24-
96K91/2/4/
X9224-
DS- 1

V
dule

SAN 901 VSAN

STATUS
STAT
US K1234567891
IN
L 0111
21
3141
51
6171
8192 02122
32
4
K1234567891
IN
L
S-X9
D 0-SF2-K90111
3
5 21
31
4151
61
7181
92 02
12232
4 STATUS
K1234567891
IN
L 0111
2131
41
516171
8192
02
122
32 4
STAT
US K1234567891
IN
L 0111
21
3141
51
61 71
81
9202
122
32
4
S-X9
D
SUPERVISOR2
SSAPWRACTUSB2
3
5
SEMEGCONSOLE1
0 -SF2-K9
T 0/1
0L
KM
IN C
O
SEMEGCONSOLE1
T
SUPERVISOR2
S-X9
D SSAPWR
3
5
0-SF2-K9 ACTUSB20/1 0LKM
IN C
O
S-X9
D SSAPWR
TAYCRES
TUSTTIMET1USB1
SEMEGCONSOLE1
VMMGMT
T
SUPERVISOR2
3
5
0-SF2-K9
VUSB1
MMGMTACTUSB20/1 0L KM
IN C
O S
D0-Gbps8-portFCoEModule234L
1-X93
5
0-SF2-K9 K5678
IN
TUSTTIMET1
RES
TAYC

WWPN-A
TUSTTIMET1USB1
SEMEGCONSOLE1
VMMGMT
T
SUPERVISOR2
SSAPWR
TAYCRES ACTUSB20/1 0L KM
INC
O S-X9
D
SSF2-
D -X9
K9 SSAPWR
TAYCRES
TUSTTIMET1USB1
SEGCONSOLE1
EMV
MMGMT
T
SUPERVISOR2
3
5
0EMV
-SF2-K9
0-SSAPWR
3
5 MMGMT 0/1
ACTUSB20L
KM
IN C
O USB1

Core 9506 WWPN-B

TUSTTIMET1
D
SUPE
SRVIS
OR2
SF2-
-X9
K9 TUSTTIMET1USB1
SEGCONSOLE1
EMV
STMMGMT
TAYCRE
3
5 T
SUPERVISOR2
0-TAYCRES
SSAPWR
SEGCONSOLE1
/1
0L
KM
IN C 0 0/1
O ACTUSB20LKM
INC
O ACTUSB2
V MMG
MT
TUSTTIMET1
TAYC
RES USB
1 SUPE
RVIS
OR2TAYCRE
SEGCONSOLE1
STMMG
/1
0LKM
INC
O 0 ACTUSB2
SSAPWR
EMV
MT
TUSTTIMET1 USB
1

Core 9506

WWNN
UCSM SAN
Student Name: Assignment Order First WWNN: Size:

A PODXA-WWNN-POOL Default 20:0X:00:25:B5:00:0A:00 4

WWNN Pools
Student Name: Assignment Order First WWNN: Size:

B PODXB-WWNN-POOL Default 20:0X:00:25:B5:00:0B:00 4

Student Name: Assignment Order First WWNN: Size:

A PODXA-A-WWPN-POOL Default 20:0X:00:25:B5:0A:0A:00 4

A PODXA-B-WWPN-POOL Default 20:0X:00:25:B5:0B:0A:00 4

WWPN Pools
Student Name: Assignment Order First WWNN: Size:

B PODXB-A-WWPN-POOL Default 20:0X:00:25:B5:0A:0B:00 4

B PODXB-B-WWPN-POOL Default 20:0X:00:25:B5:0B:0B:00 4

UCSM SAN

CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 CISCO UCS
6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
30 31 32 30 31 32

ID
UCS6200-A ID
UCS6200-B
STAT STAT

UCS 6200 UCS 6200

UCS 5108
4 x B240 M2
2
!A ! 2 A ! 2 A UCS B440 M1/M2
! 2 A

UCS 5108
0G S
-33
0
00 k
A B
S 0G 0 0
A 1 3 WWPN-A 0 0
A 1 3
0G S
-30003 S
Ak B0G 0 0
A 1 3 WWPN-B 0 0
A 1 3
0G S
-30003 S
Ak B0G

!
0G S
-300 03S
A kB 0G

SLOT SLOT
SLOT SLOT
1 1 ! 2 2
Console Reset

2G
!A
0
-300 S
0G
AS BG
3 k 00 0
0 A 1 3
G
03 k0 00
A 1 3
!

WWNN
2 A0G
-030 SASB G
03 0k 00
A 1 3
!
030 k0
A 1 3
2 A0G
-0 30S AS
B
UCS B440 M1/M2
!

!
2 A 0G
-0 3S
0 B
AS

SLOT
SLOT SLOT
3 1

WWPN-A WWPN-B !
Console
2 4
Reset

2G
!A
0 ! 2 A0G ! 2 A0G UCS B440 M1/M2
! 2 A 0G
-300 S
AS B
G G -030 SASB G -0 30S AS
B -0 3S
0 B
AS
03G k 00 0 03 k0 00 03 k
0 00 030 k0

WWNN
0 A 1 3 A 1 3 A 1 3 A 1 3

SLOT SLOT
SLOT
5 SLOT
1 !
2 6
Console Reset

2G S
!A
0 ! 2 A0G S ! 2 A0GS UCS B440 M1/M2
! 2 A 0GS

WWPN-A WWPN-B
-300 S
A BG G -300 S
A B G 30
-0 AS B 30
-0 AS B
03G
k 00 0 03 k0 00 03 0k 00 030 k0
0 A 1 3 A 1 3 A 1 3 A 1 3

SLOT SLOT
SLOT SLOT
7 1 !
2 8
Console Reset

OK FAIL OK FAIL WWNN OK FAIL OK FAIL

WWPN-A WWPN-B

WWNN
Lab 3 Modules
1. Configure UCS 6200 Fabric Interconnect VSANs
2. Configure FCoE Port-Channels
3. Create SAN Pools
4. Create vHBA Templates
Lab 4
UCSM Server
UCSM Service Profile Flow
UCS Manager Service Profile High Level Flow

Create QoS Policy

Create Server
Pool “PODXY”
Create Network
Control Policy
Create Boot
Policy “PODXY”
Create FI-A MAC
Address Pool Create FI-A
Create FI-B MAC vNIC Template
Address Pool Create FI-B
vNIC Template
Create Service Profile
Create VLANs
PODXY-1 from Template
Create Create Service Profile
WWNN Pool Template “PODXY”
Create FI-A Create Service Profile
WWPN Pool
Create FI-A PODXY-2 from Template
Create FI-B vHBA Template
WWPN Pool
Create FI-B
vHBA Template
Create VSANs
Create UUID
Pool

Create
Maintenance Lab 3 Lab 4 Lab 5
Policy
UCSM Service Profiles
IPv4 Address block for KVM/CIMC
UCSM Service Profiles
SAN Boot Policy
UCSM Service Profiles
UUID Pool Policy
UCSM Service Profiles
Server Pool Policy
UCSM Service Profiles
Service Profile Template
UCSM Service Profiles
Service Profiles from Template
Lab 4 Modules
1. Configure an IP Pool for External Blade Management
2. Configure Primary and Secondary Server SAN Boot Policy
3. Create a UUID Suffix Pool
4. Create a Server Pool
5. Create a Service Profile Template
6. Create Service Profiles from a Service Profile Template
Lab 5
VXLAN Multi-Pod using NX-API
VXLAN Multi-Pod Overview
Lab 5 Modules
1. Configure VXLAN Multi-Pod using NX-API
2. Verify VXLAN Multi-Pod

LTRDCN-3077 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Lab 6
Verify UCS and VXLAN EVPN
Verify Blade Servers
Verify VXLAN Bridging & Routing

LTRDCN-3077 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Summary
Summary
• A CLOS Fabric (Spine/Leaf Architecture) was configured for VXLAN EVPN to
extend Layer 2 domains
• VLANs were mapped to VXLAN VNIs
• VXLAN Bridging and Routing were configured
• All the logical components were built for compute services using the UCS B-
Series; UCS 5108, UCS 6200 Fabric Interconnects, and Blade Servers
• Servers were built with Service Profiles created within the UCSM GUI
• Server Blades SAN booted over multi-hop FCoE