You are on page 1of 313

Nexus Technology Labs - Fabric Extenders

(FEX)
FEX Active Standby
Task
Configure N5K1 to pair with the Fabric Extender N2K1 as follows:
Enable the Fabric Extender feature.
Configure N5K1's link connecting to N2K1 as a FEX port.
N2K1 should be module number 101.
Configure N5K2 to pair with the Fabric Extender N2K2 as follows:
Enable the Fabric Extender feature.
Configure N5K2's link connecting to N2K2 as a FEX port.
N2K2 should be module number 102.
Configure the links between N5K1 & N5K2 as 802.1q trunk links.
Configure N5K1's links to Server 1 and the Emulex CNA Server in VLAN 10.
Configure N5K2's links to Server 2 and the Emulex CNA Server in VLAN 10.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.
Configure Server 2 with the IP address 10.0.0.2/24 on this link.
Configure the Emulex CNA Server to do Active Standby NIC teaming as follows:
Use the IP address 10.0.0.10/24 for the NIC Team.
Use the link to N2K1 as the primary active path and the link to N2K2 as the
secondary standby path.
Verify that both Server 1 and Server 2 have connectivity to the Emulex CNA Server,
and that traffic to the server is flowing only through N2K1.
Disable the FEX port from N5K1 to N2K1, and verify that connectivity to the CNA
Server is maintained by using the backup path through N2K2.

Configuration
N5K1:
feature fex
!
vlan 10

!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
switchport access vlan 10
N5K2:

feature fex
!
vlan 10
!
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102
!
interface Ethernet102/1/1
switchport access vlan 10

Verification
In Active/Standby FEX topologies, hosts are physically attached to multiple FEXes,
but only actively forward on one path. Note that this topology is not related to vPC,
as vPC is used to achieve active/active forwarding, not active/standby. This

topology also requires the end hosts support of teaming through software. In this
particular case, the teaming is achieved through the Emulex OneCommand utility,
which manages the NIC Team/Port Channel config of the end adapter.
First, the end host is configured to team its links together, with the type defined as
failover in this case. Some other utilities call this active/standby or
primary/secondary, but they essentially mean the same thing. Note that the first
connection is listed as Primary, which is the link to N2K1 (hence N5K1), whereas
the second connection goes to N2K2 (hence N5K2).

IP addressing is configured on the logical Team adapter, similar to how IOS or NXOS puts logical configuration on a port channel interface.

To test the traffic flows, you can use the iPerf application to generate bulk TCP or
UDP traffic. In the output below, we see the CNA Server receiving two TCP streams
of approximately 1Gbps each, one from Server 1 and one from Server 2.

From the network side, the interface counters indicate that both of these flows are
going through the link to N2K1/N5K1, while the backup link through N2K2/N5K2 is
unused.
N5K1# show interface e101/1/1 | include rate
30 seconds input rate 38455104 bits/sec, 75106 packets/sec
30 seconds output rate 1819517120 bits/sec, 149849 packets/sec

input rate 38.46 Mbps, 75.11 Kpps;

output rate 1.82 Gbps


, 149.85 Kpps
N5K2# show interface e102/1/1 | include rate
30 seconds input rate 1072 bits/sec, 2 packets/sec
30 seconds output rate 200 bits/sec, 0 packets/sec

input rate 1.07 Kbps, 2 pps; output rate 200 bps

, 0 pps

A failure of the FEX port from N5K1 to N2K1 signals a link-down event to the end
host.
N5K1# config t
Enter configuration commands, one per line.

End with CNTL/Z. N5K1(config)# int e1/10

N5K1(config-if)# shut

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/10 is down(Config change)

2013 Mar

2 19:19:46 N5K1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 101 that is connected with Ethernet1/10 ch

2013 Mar

2 19:19:46 N5K1 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-101 Off-line (Serial Number SSI16330GT8)

2013 Mar

2 19:19:46 N5K1 %PFMA-2-FEX_STATUS: Fex 101 is offline

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/1 is down (module removed)

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/2 is down (module removed)

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/3 is down (Interface removed

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/4 is down (Interface removed

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/5 is down (Interface removed

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/6 is down (Interface removed

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/7 is down (Interface removed

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/8 is down (Interface removed

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/9 is down (Interface removed

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/10 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/11 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/12 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/13 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/14 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/15 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/16 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/17 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/18 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/19 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/20 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/21 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/22 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/23 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/24 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/25 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/26 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/27 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/28 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/29 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/30 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/31 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/32 is down (Interface remove

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/1 is down (Interface removed

2013 Mar

2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/2 is down (Interface removed

N5K1(config-if)# 2013 Mar


2013 Mar

2 19:19:47 N5K1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 101 that is connected wit

2 19:19:47 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/10 is down (Administratively down)

The end hosts NIC Teaming software detects the primary link failure and begins to
forward via the backup path.

From the network view, we see that traffic is now re-routed through the backup path
via N2K2/N5K2.
N5K2# show interface e102/1/1 | include rate
30 seconds input rate 36073720 bits/sec, 70450 packets/sec
30 seconds output rate 1706995208 bits/sec, 140583 packets/sec
output rate 1.71 Gbps
, 140.58 Kpps

input rate 36.07 Mbps, 70.45 Kpps;

Nexus Technology Labs - Fabric Extenders


(FEX)
FEX Active Active Host vPC
Task
Configure N5K1's link to Server 1 in VLAN 10.
Configure N5K2's link to Server 2 in VLAN 10.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.
Configure Server 2 with the IP address 10.0.0.2/24 on this link.
Configure FEX support as follows:
Configure N5K1 to pair with N2K1 using FEX number 101.
Configure N5K2 to pair with N2K2 using FEX number 102.
Configure a vPC between N5K1 and N5K2 as follows:
Configure vPC domain 1 on the vPC peers N5K1 and N5K2.
Use the mgmt0 port as the vPC Peer Keepalive link.
Use LACP for negotiation of all port channels.
Configure all links between the vPC peers as Port-Channel 1, and use this
as the vPC Peer Link.
Configure N5K1 and N5K2's links to the Emulex CNA Server as PortChannel 10 and vPC 10.
Port-Channel 10 should be an access port in VLAN 10.
Configure the Emulex CNA Server with LACP NIC Teaming, and use the IP address
10.0.0.10/24 for the NIC Team.
Verify that both Server 1 and Server 2 have connectivity to the Emulex CNA Server,
and that traffic to the server is being load balanced across both links through
N2K1/N5K1 and N2K2/N5K2.

Configuration
N5K1:
feature lacp
feature vpc
feature fex

!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.52
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel10
switchport access vlan 10
vpc 10
!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
switchport access vlan 10
channel-group 10 mode active
N5K2:

feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.51
!
interface port-channel1
switchport mode trunk
spanning-tree port type network

vpc peer-link
!
interface port-channel10
switchport access vlan 10
vpc 10
!
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102
!
interface Ethernet102/1/1
switchport access vlan 10
channel-group 10 mode active

Verification
Fabric Extender (FEX) and vPC topologies currently come in three forms. The first is
a vPC from the FEX southbound to the end server, sometimes called a Host vPC;
the second is a vPC from the FEX northbound to the parent switches, sometimes
called a Fabric vPC; and the third is both a southbound and northbound vPC from
the FEX, which is considered an Enhanced vPC or EvPC. Note that EvPC is only
supported on newer hardware platforms with corresponding newer software
releases. This particular configuration is considered the first variation, a Host vPC.
This example uses the same physical topology as before, except now the server
attached to the Fabric Extenders can do active/active forwarding. This is
accomplished by configuring a vPC between the parent switches of the FEXes.
Logically, this topology would be the same as if the CNA server were physically
wired to N5K1 with one link of its NIC, and then to N5K2 with the other link. This is
again because the FEX simply acts as a remote line card of the parent switch and
behaves just like a module of a modular switch. Because of the vPC configuration,
N5K1 and N5K2 appear to be the same upstream switch from the CNA servers
perspective; therefore, it can do active/active forwarding and load balancing just as

if it was dual attached to a single switch.


From the server side, the NIC Teaming software is configured to form an LACPbased team. Note that although the terms LACP and 802.3ad are normally
interchangeable, some variations of NIC Teaming software use one term to define a
channel as mode on and the other as mode active. In the case of the Emulex
OneCommand, if you choose 802.3ad teaming, you would need to configure the
channel-group 10 mode on on the NX-OS side, while LACP means that the
channel mode can be active. As shown below, the load balancing method can also
be chosen based on load, IP address, or MAC address.

Like before, the IP address goes on the logical team adapter, not the physical links.

From the network side, the first major verification is to ensure that the vPC peering
is up between the 5Ks. Only after the keepalive is confirmed and the vPC peer link
is formed can the vPC to the end host actually form. Note that the same vPC
consistency rules apply to FEX-based vPCs as to regular vPCs.

N5K1# show vpc

Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

Peer status

: peer adjacency formed ok

: peer is alive

vPC keep-alive status

Configuration consistency status: success


Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: primary

Number of vPCs configured

: 1

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ -------------------------------------------------- 1 Po1

up

1,10
vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------- 10 Po10


success

success

up

10

The FEX configuration itself it technically unrelated to the vPC, as the FEX Fabric
Ports are configured the same as before.
N5K1# show interface fex-fabric
Fabric
Fex

Port

Fabric
Port State

Fex
Uplink

FEX
Model

Serial

--------------------------------------------------------------- 101
1

N2K-C2232PP-10GE

Eth1/10

Active

Eth1/11

Active

SSI16330GT8

N5K2# show interface fex-fabric


Fabric
Fex

Port

Fabric
Port State

Fex
Uplink

FEX
Model

Serial

--------------------------------------------------------------- 102
2

N2K-C2232PP-10GE

SSI15030C1R

For final verification, generate traffic flows between the servers and note the
interface statistics of the FEX host ports to the Emulex CNA server. In the below
output, iPerf is used to generate bulk TCP flows from Server 1 and Server 2 to the

CNA server.

These two flows are near line-rate for the 1GigE attached Server 1 and Server 2.
The difference between this example and the last one, however, is that these 2 x
1Gbps flows are distributed between the vPC member ports to the CNA server. This
can be verified as seen through the interface counters below:
N5K1# show interface e101/1/1 | include rate
30 seconds input rate 19755792 bits/sec, 38585 packets/sec
30 seconds output rate 934557000 bits/sec, 76969 packets/sec

input rate 19.75 Mbps, 38.58 Kpps;

output rate 934.56 Mbps


, 76.97 Kpps
N5K2# show interface e102/1/1 | include rate
30 seconds input rate 19651672 bits/sec, 38381 packets/sec
30 seconds output rate 934679720 bits/sec, 76978 packets/sec

input rate 19.65 Mbps, 38.38 Kpps;

output rate 934.68 Mbps


, 76.98 Kpps

In the case of a link failure, traffic will automatically be rerouted to the other
available member links after LACP detects the fault. As shown below, when N5K2's
link to the downstream N2K2 FEX goes down, both 1Gbps traffic flows are rerouted
to the other FEX.
N5K2# config t
Enter configuration commands, one per line.

End with CNTL/Z. N5K2(config)# int e1/11

N5K2(config-if)# shut
2013 Mar

2 22:10:07 N5K2 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/11 is down(Config change)

2013 Mar

2 22:10:07 N5K2 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 2 of Fex 102 that is connected with Ethernet1/11 ch

2013 Mar

2 22:10:07 N5K2 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-102 Off-line (Serial Number SSI15030C1R)

2013 Mar

2 22:10:07 N5K2 %PFMA-2-FEX_STATUS: Fex 102 is offline

<snip>
N5K1# show interface e101/1/1 | include rate
30 seconds input rate 38782896 bits/sec, 75746 packets/sec
30 seconds output rate 1838416440 bits/sec, 151406 packets/sec
output rate 1.84 Gbps
, 151.41 Kpps

input rate 38.78 Mbps, 75.75 Kpps;

Nexus Technology Labs - Fabric Extenders


(FEX)
Fabric Extenders (FEX)
Task
Configure N5K1 to pair with the Fabric Extender N2K1 as follows:
Enable the Fabric Extender feature.
Configure N5K1's link connecting to N2K1 as a FEX port
N2K1 should be module number 101.
Configure N5K1's links to Server 1 and the Emulex CNA Server in VLAN 10.
These links should both be STP Edge Ports.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.
Configure the Emulex CNA Server with the IP address 10.0.0.10/24 on this link.
When complete, Server 1 and the Emulex CNA Server should have IP reachability to
each other.

Configuration

N5K1:

feature fex
!
vlan 10
!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
switchport access vlan 10

Verification
Fabric Extenders (FEXes) are access switches that behave as remote line cards of
a parent switch. After the FEX is paired with the parent switch, such as a Nexus 5K
or 7K, all configuration occurs on the upstream parent. From the parent switchs
perspective, the FEX is simply another module or line card, and is configured as
such.
In the output below, we can see that a Nexus 2232PP FEX is paired with the parent
switch N5K1 as FEX number 101. This means that the FEX is simply treated as
module 101 from the 5Ks perspective.
N5K1# show fex
FEX
Number

FEX

FEX

Description

State

FEX
Model

Serial

-----------------------------------------------------------------------101

FEX0101

Online

N2K-C2232PP-10GE

SSI16330GT8

The detailed output below shows the specifics of this FEX, such as the software
version downloaded from the parent and what the state is. When the state is online,
the most important portion of the output shown below is how the downstream FEX
ports are pinned to the upstream Fabric Ports. In this topology, there is only one

physical uplink from the FEX to N5K1, so all FEX ports are pinned to the Fabric Port
E1/10. In a case in which more physical links are used, the pinning of FEX ports can
be controlled with the pinning max-links command under the global FEX
configuration, or the Fabric Port can be configured as a port-channel, essentially
dynamically pinning all FEX ports to all ports in the channel at the same time. In the
latter case, traffic is then load balanced based on the Port-Channel load balancing
method.
N5K1# show fex detail
FEX: 101 Description: FEX0101 state: Online
FEX version: 5.1(3)N1(1a) [Switch version: 5.1(3)N1(1a)]
FEX Interim version: 5.1(3)N1(1a)
Switch Interim version: 5.1(3)N1(1a) Extender Serial: SSI16330GT8
Extender Model: N2K-C2232PP-10GE
,

Part No: 73-12533-05


Card Id: 82, Mac Addr: 54:78:1a:30:3d:c2, Num Macs: 64
Module Sw Gen: 12594

[Switch Sw Gen: 21]

post level: complete pinning-mode: static

Max-links: 1

Fabric port for control traffic: Eth1/10


FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Fex Port
Eth101/1/2

State
Up

Eth1/10 - Interface Up. State: Active


Fabric Port Eth101/1/1

Eth1/10

Eth101/1/3

Down

None

Eth101/1/4

Down

None

Eth101/1/5

Down

None

Eth101/1/6

Down

None

Eth101/1/7

Down

None

Eth101/1/8

Down

None

Eth101/1/9

Down

None

Eth101/1/10

Down

None

Eth101/1/11

Down

None

Eth101/1/12

Down

None

Eth101/1/13

Down

None

Eth101/1/14

Down

None

Eth101/1/15

Down

None

Eth101/1/16

Down

None

Eth101/1/17

Down

None

Eth101/1/18

Down

None

Eth101/1/19

Down

None

Eth101/1/20

Down

None

Eth101/1/21

Down

None

Eth101/1/22

Down

None

Up

Eth1/10

Eth101/1/23

Down

None

Eth101/1/24

Down

None

Eth101/1/25

Down

None

Eth101/1/26

Down

None

Eth101/1/27

Down

None

Eth101/1/28

Down

None

Eth101/1/29

Down

None

Eth101/1/30

Down

None

Eth101/1/31

Down

None

Eth101/1/32

Down

None

Logs:
03/02/2013 18:17:32.337586: Module register received
03/02/2013 18:17:32.339470: Registration response sent
03/02/2013 18:17:32.465539: Module Online Sequence 03/02/2013 18:17:35.611664: Module Online

When pairing between the FEX and the parent switch is complete, further
configuration of the FEX ports is the same as any other physical link. Note that there
are some behavioral differences between FEX host ports and other physical links;
for example, the FEX ports always run as STP Edge Ports with BPDU Filter and
Guard enabled. This can be seen below; although the spanning-tree port type edge
is not configured on the FEX port, it still operationally runs in that mode.
N5K1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID

Priority

32778

Address

000d.eca2.edbc

This bridge is the root

Bridge ID

Interface

Hello Time

sec

Priority

32778

Address

000d.eca2.edbc

Hello Time

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Eth1/1

Desg FWD 4

128.129

Eth101/1/1

Desg FWD 2

128.1153 Edge P2p

Edge P2p

N5K1# show spanning-tree interface e101/1/1 detail

Port 1153 (Ethernet101/1/1) of VLAN0010 is designated forwarding


Port path cost 2, Port priority 128, Port Identifier 128.1153

Designated root has priority 32778, address 000d.eca2.edbc


Designated bridge has priority 32778, address 000d.eca2.edbc
Designated port id is 128.1153, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1 The port type is edge
Link type is point-to-point by default Bpdu guard is enabled
Bpdu filter is enabled by default

BPDU: sent 11, received 0

Nexus Technology Labs - Fabric Extenders


(FEX)
FEX Active Active Fabric vPC
Task
Configure N5K1's link to Server 1 in VLAN 10.
Configure N5K2's link to Server 2 in VLAN 10.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.
Configure Server 2 with the IP address 10.0.0.2/24 on this link.
Configure FEX support as follows:
Configure N5K1 and N5K2 to pair with N2K1 using FEX number 101.
Configure N5K1 and N5K2 to pair with N2K2 using FEX number 102.
Configure a vPC between N5K1 and N5K2 as follows:
Configure vPC domain 1 on the vPC peers N5K1 and N5K2.
Use the mgmt0 port as the vPC Peer Keepalive link.
Configure all links between the vPC peers as Port-Channel 1, and use this
as the vPC Peer Link.
Configure the FEX Fabric ports from N5K1 and N5K2 to N2K1 as PortChannel 101, and as vPC 101.
Configure the FEX Fabric ports from N5K1 and N5K2 to N2K2 as PortChannel 102, and as vPC 102.
Configure the Emulex CNA Server to do Active Standby NIC teaming as follows:
Use the link to N2K2 as the primary active path and the link to N2K1 as the
secondary standby path.
Use the IP address 10.0.0.10/24 for the NIC Team, and assign its links to
VLAN 10.
Verify that both Server 1 and Server 2 have connectivity to the Emulex CNA Server,
and that traffic to the CNA Server is flowing only through N2K2.
Disable the link from the CNA Server to N2K2, and verify that connectivity is
maintained by using the backup path through N2K1.

Configuration
N5K1:
feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.52
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel101
switchport mode fex-fabric
fex associate 101
vpc 101
!
interface port-channel102
switchport mode fex-fabric
fex associate 102
vpc 102
!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
channel-group 101 mode on
!
interface Ethernet1/11

switchport mode fex-fabric


fex associate 102
channel-group 102 mode on
!
interface Ethernet101/1/1
switchport access vlan 10
!
interface Ethernet102/1/1
switchport access vlan 10
N5K2:

feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.51
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel101
switchport mode fex-fabric
fex associate 101
vpc 101
!
interface port-channel102
switchport mode fex-fabric
fex associate 102
vpc 102
!
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/10

switchport mode fex-fabric


fex associate 101
channel-group 101 mode on
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102
channel-group 102 mode on
!
interface Ethernet101/1/1
switchport access vlan 10
!
interface Ethernet102/1/1
switchport access vlan 10

Verification
Fabric Extender (FEX) and vPC topologies currently come in three forms. The first is
a vPC from the FEX southbound to the end server, sometimes called a Host vPC,
the second is a vPC from the FEX northbound to the parent switches, sometimes
called a Fabric vPC, and the third is both a southbound and northbound vPC from
the FEX, which is considered an Enhanced vPC, or EvPC. Note that EvPC is only
supported on newer hardware platforms with corresponding newer software
releases. This particular configuration is considered the second variation, the Fabric
vPC.
In the Fabric vPC, the end host may be single or dual attached to one or more
FEXes, but the FEX does not perform port channeling southbound to the end host.
Instead, the FEX forms a vPC northbound to multiple parent switches. Although this
does not contribute to any load distribution between the end server and the FEX, it
does more evenly distribute the load from the FEXes northbound to their parents.
The potential danger with this design, however, is that multiple parent switches,
each with separate management and control planes, are referencing the same FEX
host ports. This means that if the configuration becomes out of sync between the
parent switches, there could be a problem in the data plane of the FEX host ports. A
possible resolution to this problem is to use the Configuration Synchronization
feature, which is demonstrated in a separate scenario.
The configuration of this scenario is similar to the other FEX pairings, with the
exception that the port channel southbound from the parent switch to the FEX does
not run LACP. This is because the FEX uplink ports do not support LACP: they only
support static channels. When complete, both parent switches must agree on

identical configurations down to the FEX Fabric Ports and to the FEX Host Ports.
The consistency of the parent switches configurations is protected against using the
vPC consistency check. Note that the show vpc output below indicates that each of
the FEX Host Ports now participates in the vPC, even though there is not
channeling configured on the host ports. Like other vPC configurations, the first
verification should be that the vPC Peer Keepalive is up and the vPC Peer Link
adjacency has been formed.
N5K1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

Peer status

vPC keep-alive status

: peer is alive

: peer adjacency formed ok

Configuration consistency status: success


Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: primary

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

Number of vPCs configured

: 66

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ -------------------------------------------------- 1

Po1 up

1,10
vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------- 101


success

success

success

success

102

Po102

102400 Eth101/1/1

up

success

success

10

102401 Eth101/1/2

up

success

success

102402 Eth101/1/3

down*

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

102403 Eth101/1/4

102404 Eth101/1/5

down*

down*

up

Po101

up

102405 Eth101/1/6

102406 Eth101/1/7

102407 Eth101/1/8

102408 Eth101/1/9

down*

down*

down*

down*

102409 Eth101/1/10 down*

102410 Eth101/1/11 down*

102411 Eth101/1/12 down*

102412 Eth101/1/13 down*

102413 Eth101/1/14 down*

102414 Eth101/1/15 down*

102415 Eth101/1/16 down*

102416 Eth101/1/17 down*

102417 Eth101/1/18 down*

102418 Eth101/1/19 down*

102419 Eth101/1/20 down*

102420 Eth101/1/21 down*

102421 Eth101/1/22 down*

102422 Eth101/1/23 down*

102423 Eth101/1/24 down*

102424 Eth101/1/25 down*

102425 Eth101/1/26 down*

102426 Eth101/1/27 down*

102427 Eth101/1/28 down*

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

102428 Eth101/1/29 down*

102429 Eth101/1/30 down*

102430 Eth101/1/31 down*

102431 Eth101/1/32 down*

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

103424 Eth102/1/1

up

success

success

10

103425 Eth102/1/2

up

success

success

103426 Eth102/1/3

down*

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

103427 Eth102/1/4

103428 Eth102/1/5

103429 Eth102/1/6

103430 Eth102/1/7

103431 Eth102/1/8

103432 Eth102/1/9

down*

down*

down*

down*

down*

down*

103433 Eth102/1/10 down*

103434 Eth102/1/11 down*

103435 Eth102/1/12 down*

103436 Eth102/1/13 down*

103437 Eth102/1/14 down*

103438 Eth102/1/15 down*

103439 Eth102/1/16 down*

103440 Eth102/1/17 down*

103441 Eth102/1/18 down*

103442 Eth102/1/19 down*

103443 Eth102/1/20 down*

103444 Eth102/1/21 down*

103445 Eth102/1/22 down*

103446 Eth102/1/23 down*

103447 Eth102/1/24 down*

103448 Eth102/1/25 down*

103449 Eth102/1/26 down*

103450 Eth102/1/27 down*

103451 Eth102/1/28 down*

103452 Eth102/1/29 down*

103453 Eth102/1/30 down*

103454 Eth102/1/31 down*

103455 Eth102/1/32 down*

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

N5K2# show vpc


Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

vPC keep-alive status

: peer adjacency formed ok

Peer status
: peer is alive

Configuration consistency status: success


Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: secondary

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

Number of vPCs configured

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ --------------------------------------------------

: 66

1 Po1

up
1,10

vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------- 101


success

success

success

success

102

Po102

102400 Eth101/1/1

up

success

success

10

102401 Eth101/1/2

up

success

success

102402 Eth101/1/3

down*

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

102403 Eth101/1/4

102404 Eth101/1/5

102405 Eth101/1/6

down*

down*

down*

102406 Eth101/1/7

down*

102407 Eth101/1/8

down*

102408 Eth101/1/9

down*

102409 Eth101/1/10 down*

102410 Eth101/1/11 down*

102411 Eth101/1/12 down*

102412 Eth101/1/13 down*

102413 Eth101/1/14 down*

102414 Eth101/1/15 down*

102415 Eth101/1/16 down*

102416 Eth101/1/17 down*

102417 Eth101/1/18 down*

up

Po101

up

102418 Eth101/1/19 down*

102419 Eth101/1/20 down*

102420 Eth101/1/21 down*

102421 Eth101/1/22 down*

102422 Eth101/1/23 down*

102423 Eth101/1/24 down*

102424 Eth101/1/25 down*

102425 Eth101/1/26 down*

102426 Eth101/1/27 down*

102427 Eth101/1/28 down*

102428 Eth101/1/29 down*

102429 Eth101/1/30 down*

102430 Eth101/1/31 down*

102431 Eth101/1/32 down*

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

103424 Eth102/1/1

up

success

success

10

103425 Eth102/1/2

up

success

success

103426 Eth102/1/3

down*

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

103427 Eth102/1/4

down*

103428 Eth102/1/5

down*

103429 Eth102/1/6

down*

103430 Eth102/1/7

103431 Eth102/1/8

103432 Eth102/1/9

down*

down*

down*

103433 Eth102/1/10 down*

103434 Eth102/1/11 down*

103435 Eth102/1/12 down*

103436 Eth102/1/13 down*

103437 Eth102/1/14 down*

103438 Eth102/1/15 down*

103439 Eth102/1/16 down*

103440 Eth102/1/17 down*

103441 Eth102/1/18 down*

103442 Eth102/1/19 down*

103443 Eth102/1/20 down*

103444 Eth102/1/21 down*

103445 Eth102/1/22 down*

103446 Eth102/1/23 down*

103447 Eth102/1/24 down*

103448 Eth102/1/25 down*

103449 Eth102/1/26 down*

103450 Eth102/1/27 down*

103451 Eth102/1/28 down*

103452 Eth102/1/29 down*

103453 Eth102/1/30 down*

103454 Eth102/1/31 down*

103455 Eth102/1/32 down*

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Note that both N5K1 and N5K2 are pairing with the same downstream FEXes.
N5K1# show interface fex-fabric
Fabric
Fex

Port

Fabric
Port State

Fex
Uplink

FEX
Model

Serial

--------------------------------------------------------------101

Eth1/10

Active

N2K-C2232PP-10GE

SSI16330GT8

102

Eth1/11

Active

N2K-C2232PP-10GE

SSI15030C1R

N5K2# show interface fex-fabric


Fabric
Fex

Port

Fabric
Port State

Fex
Uplink

FEX
Model

Serial

--------------------------------------------------------------101

Eth1/10

Active

N2K-C2232PP-10GE

SSI16330GT8

102

Eth1/11

Active

N2K-C2232PP-10GE

SSI15030C1R

From the end servers perspective, their links have been configured in
active/standby failover teaming, with the primary path being to N2K2.

For final verification of traffic distribution, Server 1 and Server 2 generate TCP flows
toward the CNA server. The end result is multiple flows for a total nearing 2Gbps,
which is the combined line rates of Server 1 and 2.

From the network side, both N5K1 and N5K2 see that all flows towards the CNA
server exit via N2K2.
N5K1# show interface e101/1/1 | include rate
30 seconds input rate 1072 bits/sec, 2 packets/sec
30 seconds output rate 880 bits/sec, 1 packets/sec

input rate 1.07 Kbps, 2 pps; output rate 912 bps

, 1 pps
N5K1# show interface e102/1/1 | include rate
30 seconds input rate 43354968 bits/sec, 84644 packets/sec
30 seconds output rate 1965663832 bits/sec, 161924 packets/sec

input rate 43.14 Mbps, 84.21 Kpps;

output rate 1.95 Gbps


, 160.63 Kpps
N5K2# show interface e101/1/1 | include rate
30 seconds input rate 1072 bits/sec, 2 packets/sec
30 seconds output rate 704 bits/sec, 1 packets/sec

input rate 1.07 Kbps, 2 pps; output rate 912 bps

, 1 pps
N5K2# show interface e102/1/1 | include rate
30 seconds input rate 43728760 bits/sec, 85367 packets/sec
30 seconds output rate 1967795096 bits/sec, 162099 packets/sec

input rate 43.15 Mbps, 84.23 Kpps;

output rate 1.95 Gbps


, 160.63 Kpps

Note that these outputs are nearly identical on both parent switches, as they are
both referencing the same physical FEX host ports. The key difference with this
configuration design, however, is that traffic is load balanced from the parent
switches down to the N2K2 FEX. This can be verified by viewing the counters of the
FEX Fabric Ports of the parent switches, as seen below.
N5K1# show interface e1/10 - 11 | include rate|Ethernet1
Ethernet1/10 is up
30 seconds input rate 19152 bits/sec, 2 packets/sec

30 seconds output rate 3184 bits/sec, 2 packets/sec

input rate 18.55 Kbps, 2 pps;

output rate 2.94 Kbps


, 2 pps
Ethernet1/11 is up
30 seconds input rate 17752 bits/sec, 2 packets/sec
30 seconds output rate 989910144 bits/sec, 81007 packets/sec

input rate 23.91 Kbps, 2 pps;

output rate 985.18 Mbps


, 80.56 Kpps
N5K2# show interface e1/10 - 11 | include rate|Ethernet1
Ethernet1/10 is up
30 seconds input rate 9232 bits/sec, 1 packets/sec
30 seconds output rate 3288 bits/sec, 2 packets/sec

input rate 16.00 Kbps, 2 pps;

output rate 2.84 Kbps


, 2 pps
Ethernet1/11 is up
30 seconds input rate 50142584 bits/sec, 84628 packets/sec
30 seconds output rate 990521328 bits/sec, 81056 packets/sec

input rate 50.03 Mbps, 84.39 Kpps;

output rate 986.24 Mbps


, 80.64 Kpps

Note that neither parent switch is sending traffic to FEX Fabric Port E1/10 because
this is the link to N2K1, which the CNA server is using as the standby connection.
Although the total of flows from Server 1 and 2 are 2Gbps, they are nearly equally
split between the FEX Fabric Ports going from N5K1 and N5K2 southbound to the
N2K2 FEX.
In the case of a failure of the servers primary uplink, traffic is automatically rerouted to the backup link via FEX N2K1, as shown below.

N5K1# show interface e1/10 - 11 | include rate|Ethernet1


Ethernet1/10 is up
30 seconds input rate 20552 bits/sec, 2 packets/sec

30 seconds output rate 987383384 bits/sec


, 80800 packets/sec
input rate 17.62 Kbps, 2 pps; output rate 518.97 Mbps, 42.43 Kpps
Ethernet1/11 is up
30 seconds input rate 8624 bits/sec, 1 packets/sec 30 seconds output rate 3128 bits/sec
, 2 packets/sec
input rate 16.26 Kbps, 2 pps; output rate 463.30 Mbps, 37.85 Kpps
N5K2# show interface e1/10 - 11 | include rate|Ethernet1
Ethernet1/10 is up
30 seconds input rate 50370824 bits/sec, 85031 packets/sec 30 seconds output rate 990017536 bits/sec
, 81015 packets/sec
input rate 24.29 Mbps, 40.97 Kpps; output rate 478.05 Mbps, 39.09 Kpps
Ethernet1/11 is up
30 seconds input rate 24968 bits/sec, 2 packets/sec 30 seconds output rate 3040 bits/sec
, 2 packets/sec
input rate 25.69 Mbps, 43.28 Kpps; output rate 505.58 Mbps, 41.32 Kpps

Nexus Technology Labs - Fabric Extenders


(FEX)
FEX and N5K Config Sync
Task
Enable the vPC, FEX, and LACP features on N5K1 and N5K2.
Enable Cisco Fabric Services over IP (CFSoIP) distribution between N5K1 and N5K2.
Configure vPC domain 1 between N5K1 and N5K2, and use the mgmt0 link for the
vPC Peer Keepalive.
Create a Config Sync session on both N5K1 and N5K2, and use the switch profile
name N5K.
Use the mgmt0 IP addresses as the config sync peers destination.
Verify that N5K1 and N5K2 can reach each other over CFSoIP for the config sync
session.
Without making any additional changes on N5K2, use the switch profile on N5K1 to
replicate the following configuration to both switches:
Pre-provision FEX modules 101 and 102, both of type N2K-C2232P.
Create VLAN 10.
All links to Server 1 and Server 2 should be access ports in VLAN 10.
Configure all links between the vPC peers as Port-Channel 1, and use this
as the vPC Peer Link.
Configure N5K1 and N5K2 to pair with N2K1 using FEX number 101.
Configure N5K1 and N5K2 to pair with N2K2 using FEX number 102.
Configure the FEX Fabric ports from N5K1 and N5K2 to N2K1 as PortChannel 101, and as vPC 101.
Configure the FEX Fabric ports from N5K1 and N5K2 to N2K2 as PortChannel 102, and as vPC 102.
Configure the links to the Emulex CNA Server in VLAN 10.
Commit the config and verify that both N5K1 and N5K2 identically accept it into their
running configuration.

Configuration
N5K2:
feature vpc
feature fex
feature lacp
cfs ipv4 distribute
!
vpc domain 1
peer-keepalive destination 192.168.0.51 vrf management
!
end
config sync
switch-profile N5K
sync-peers destination 192.168.0.51
verify
N5K1:

feature vpc
feature fex
feature lacp
cfs ipv4 distribute
!
vpc domain 1
peer-keepalive destination 192.168.0.52 vrf management
!
end
config sync
switch-profile N5K
sync-peers destination 192.168.0.52
verify
show switch-profile status

slot 101
provision model N2K-C2232P
!
slot 102
provision model N2K-C2232P
!
vlan 10
!
interface port-channel1

switchport mode trunk


spanning-tree port type network
vpc peer-link
!
interface port-channel101
switchport mode fex-fabric
fex associate 101
vpc 101
!
interface port-channel102
switchport mode fex-fabric
fex associate 102
vpc 102
!
interface Ethernet1/1 - 2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
channel-group 101 mode on
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102
channel-group 102 mode on
!
interface Ethernet101/1/1
switchport access vlan 10
!
interface Ethernet102/1/1
switchport access vlan 10
!
commit

Verification
Configuration Synchronization, also known as Config Sync for short or Switch
Profiles, is a way to apply a template of configuration onto multiple Nexus switches
at the same time. This feature is especially useful between vPC peers, or when FEX
deployments are used in active/active, where a downstream FEX peers with more
than one upstream parent switch. This feature helps to ensure that configurations
stay consistent between the vPC peers or FEX parents (or both as in the case
shown below), and avoid problems such as vPC failure caused by a consistency
check error.
Note that of current releases, this feature is not supported on the Nexus 7K.
Additionally, not all commands are supported in the switch profile mode, and instead
must be configured in regular global config. In this particular case, the unsupported
commands are the feature enablement of vPC, FEX, and LACP, as well as the vPC
domain creation, as the configuration is different between the vPC peers
(specifically the peer keepalive destination address).
To use config sync, the switches must first be configured to use Cisco Fabric
Services over IP (CFSoIP), as this is the control plane protocol that is actually used
to sync the config between the switches. This is enabled simply as follows:
N5K1# conf t
Enter configuration commands, one per line.

End with CNTL/Z. N5K1(config)# cfs ipv4 distribute

N5K1(config)# show cfs peers

Physical Fabric
------------------------------------------------------------------------Switch WWN

IP Address

------------------------------------------------------------------------- 20:00:00:0d:ec:a2:ed:80
192.168.0.51
[Local] N5K1
20:00:00:0d:ec:a4:74:00 192.168.0.52

Total number of entries = 2

Next, start a config sync session, create an identically named switch profile on each
switch, specify the IP address of the peer to sync with, and then verify their
connectivity.
N5K1# config sync

Enter configuration commands, one per line.

End with CNTL/Z. N5K1(config-sync)# switch-profile N5K

Switch-Profile started, Profile ID is 1 N5K1(config-sync-sp)# sync-peers destination 192.168.0.52


N5K1(config-sync-sp)# verify
Verification Successful
N5K1(config-sync-sp)#
vN5K2# config sync
Enter configuration commands, one per line.

End with CNTL/Z. N5K2(config-sync)# switch-profile N5K

Switch-Profile started, Profile ID is 1 N5K2(config-sync-sp)# sync-peers destination 192.168.0.51


N5K2(config-sync-sp)# verify
Verification Successful

N5K2(config-sync-sp)#

If the switch profile is in sync between the peers, they should both agree on the
profile revision number and show the sync status as in sync.
N5K1(config-sync-sp)# show switch-profile status

switch-profile

: N5K

----------------------------------------------------------

Start-time: 382018 usecs after Sun Mar


End-time: 441035 usecs after Sun Mar

3 15:44:48 2013
3 15:44:50 2013

Profile-Revision: 1
Session-type: Initial-Exchange
Session-subtype: Init-Exchange-All
Peer-triggered: Yes
Profile-status: Sync Success

Local information:
---------------- Status: Commit Success
Error(s):

Peer information:
---------------- IP-address: 192.168.0.52
Sync-status: In sync
Status: Commit Success
Error(s):
N5K2(config-sync-sp)# show switch-profile status

switch-profile

: N5K

----------------------------------------------------------

Start-time: 831674 usecs after Sun Mar


End-time: 875222 usecs after Sun Mar

3 16:35:38 2013
3 16:35:40 2013

Profile-Revision: 1

Session-type: Initial-Exchange
Session-subtype: Init-Exchange-All
Peer-triggered: No
Profile-status: Sync Success

Local information:
---------------- Status: Commit Success
Error(s):

Peer information:
---------------- IP-address: 192.168.0.51
Sync-status: In sync
Status: Commit Success

Error(s):

Now the switches are ready to accept the configuration changes to synchronize.
Commands are entered just like in global config, but they are not immediately
applied. Instead they are sent to the switch profile buffer, as shown below. Before
the buffer is committed, the buffer can be deleted or modified as desired. The line
numbers of the buffer show how the config will sequentially be applied. Therefore,
any configurations that are sensitive to order of operations must have the correct
line numbering in the buffer before a commit is executed.
N5K1(config-sync-sp)# slot 101
N5K1(config-sync-sp-slot)#

provision model N2K-C2232P

N5K1(config-sync-sp-slot)# !
N5K1(config-sync-sp-slot)# slot 102
N5K1(config-sync-sp-slot)#

provision model N2K-C2232P

N5K1(config-sync-sp-slot)# !
N5K1(config-sync-sp-slot)# vlan 10
N5K1(config-sync-sp-vlan)# !
N5K1(config-sync-sp-vlan)# interface port-channel1
N5K1(config-sync-sp-if)#

switchport mode trunk

N5K1(config-sync-sp-if)#

spanning-tree port type network

N5K1(config-sync-sp-if)#

vpc peer-link

N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface port-channel101
N5K1(config-sync-sp-if)#

switchport mode fex-fabric

N5K1(config-sync-sp-if)#

fex associate 101

N5K1(config-sync-sp-if)#

vpc 101

N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface port-channel102

N5K1(config-sync-sp-if)#

switchport mode fex-fabric

N5K1(config-sync-sp-if)#

fex associate 102

N5K1(config-sync-sp-if)#

vpc 102

N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface Ethernet1/1 - 2
N5K1(config-sync-sp-if-range)#

switchport access vlan 10

N5K1(config-sync-sp-if-range)#

spanning-tree port type edge

N5K1(config-sync-sp-if-range)#

speed 1000

N5K1(config-sync-sp-if-range)# !
N5K1(config-sync-sp-if-range)# interface Ethernet1/3 - 5
N5K1(config-sync-sp-if-range)#

switchport mode trunk

N5K1(config-sync-sp-if-range)#

spanning-tree port type network

N5K1(config-sync-sp-if-range)#

channel-group 1 mode active

N5K1(config-sync-sp-if-range)# !
N5K1(config-sync-sp-if-range)# interface Ethernet1/10
N5K1(config-sync-sp-if)#

switchport mode fex-fabric

N5K1(config-sync-sp-if)#

fex associate 101

N5K1(config-sync-sp-if)#

channel-group 101 mode on

N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface Ethernet1/11
N5K1(config-sync-sp-if)#

switchport mode fex-fabric

N5K1(config-sync-sp-if)#

fex associate 102

N5K1(config-sync-sp-if)#

channel-group 102 mode on

N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface Ethernet101/1/1
N5K1(config-sync-sp-if)#

switchport access vlan 10

N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface Ethernet102/1/1
N5K1(config-sync-sp-if)#

switchport access vlan 10

N5K1(config-sync-sp-if)#
N5K1(config-sync-sp-if)# show switch-profile buffer

switch-profile

: N5K

---------------------------------------------------------Seq-no

Command

---------------------------------------------------------1
1.1
2
2.1

slot 101
provision model N2K-C2232P
slot 102
provision model N2K-C2232P

vlan 10

interface port-channel1

4.1

switchport mode trunk

4.2

spanning-tree port type network

4.3

vpc peer-link

interface port-channel101

5.1

switchport mode fex-fabric

5.2

fex associate 101

5.3

vpc 101

interface port-channel102

6.1

switchport mode fex-fabric

6.2

fex associate 102

6.3

vpc 102

interface Ethernet1/1-2

7.1

switchport access vlan 10

7.2

spanning-tree port type edge

7.3

speed 1000

interface Ethernet1/3-5

8.1

switchport mode trunk

8.2

spanning-tree port type network

8.3

channel-group 1 mode active

interface Ethernet1/10

9.1

switchport mode fex-fabric

9.2

fex associate 101

9.3

channel-group 101 mode on

10

interface Ethernet1/11

10.1

switchport mode fex-fabric

10.2

fex associate 102

10.3
11
11.1
12
12.1

channel-group 102 mode on


interface Ethernet101/1/1
switchport access vlan 10
interface Ethernet102/1/1
switchport access vlan 10

When the commands in the buffer are acceptable, the profile is committed. During
the commit procedure, the config is synchronized across to the other peer using
CFSoIP, and applied sequentially. If there is an error in applying the config, all
commands in the buffer are rolled back and the commit fails. In other words, either
the commit succeeds 100 percent, or no config is applied to either peer. In the
output below, we see that the commit was successful, and syslog messages begin
to appear as config changes, link up/down events, etc. occur just as if you had
applied the commands manually on each switch individually.

N5K1(config-sync-sp-if)# commit
Verification successful...

Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.

2013 Mar

3 15:49:31 N5K1 %ETH_PORT_CHANNEL-5-CREATED: port-channel1 created

2013 Mar

3 15:49:31 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operatio

2013 Mar

3 15:49:33 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operatio

2013 Mar

3 15:49:33 N5K1 %ETH_PORT_CHANNEL-5-CREATED: port-channel101 created

2013 Mar

3 15:49:33 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel101 is down (No operat

2013 Mar

3 15:49:33 N5K1 last message repeated 2 times

2013 Mar

3 15:49:33 N5K1 %ETH_PORT_CHANNEL-5-CREATED: port-channel102 created

<snip> Commit Successful

N5K1(config-sync)#
N5K2(config-sync-sp)#

2013 Mar

3 16:40:22 N5K2 %ETH_PORT_CHANNEL-5-CREATED: port-channel1 created

2013 Mar

3 16:40:22 N5K2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operatio

2013 Mar

3 16:40:24 N5K2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operatio

2013 Mar

3 16:40:24 N5K2 %ETH_PORT_CHANNEL-5-CREATED: port-channel101 created

2013 Mar

3 16:40:24 N5K2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel101 is down (No operat

2013 Mar

3 16:40:24 N5K2 last message repeated 2 times

2013 Mar

3 16:40:24 N5K2 %ETH_PORT_CHANNEL-5-CREATED: port-channel102 created

<snip>

When the commit is successful, N5K1 automatically exits out of the switch profile
configuration mode. If additional config changes are required, a new switch profile
session must be started, using the same session name as before. Note in the output
below that both switches agree on the switch profile revision number, and the
switches are in sync.
N5K1(config-sync)# show switch-profile status

switch-profile

: N5K

----------------------------------------------------------

Start-time: 130047 usecs after Sun Mar


End-time: 663864 usecs after Sun Mar

3 15:49:28 2013
3 15:49:38 2013

Profile-Revision: 2

Session-type: Commit
Session-subtype: Peer-triggered: No Profile-status: Sync Success

Sync-status: In sync Status: Commit Success

Error(s):
N5K2(config-sync-sp)# show switch-profile status

switch-profile

: N5K

----------------------------------------------------------

Start-time: 830375 usecs after Sun Mar


End-time: 361267 usecs after Sun Mar

3 16:40:18 2013
3 16:40:29 2013

Profile-Revision: 2

Session-type: Commit
Session-subtype: Peer-triggered: Yes Profile-status: Sync Success

Local information:
---------------- Status: Commit Success

Error(s):

Peer information:
---------------IP-address: 192.168.0.51
Sync-status: In sync Status: Commit Success

Error(s):

From N5K2s perspective, the configuration commands appear in the running config
just as if they had been entered manually in global configuration.
N5K2# show run interface

!Command: show running-config interface


!Time: Sun Mar

3 17:12:09 2013

version 5.1(3)N1(1a)

interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link

interface port-channel101
switchport mode fex-fabric
fex associate 101
vpc 101

interface port-channel102
switchport mode fex-fabric
fex associate 102
vpc 102
<snip>

Further verification shows that the configured features such as the vPC, FEX Fabric
Ports, VLANs, etc. are functional.
N5K2# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status: success


Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: secondary

Number of vPCs configured

: 66

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ --------------------------------------------------

Po1

up

1,10

vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------101

Po101

up

success

success

102

Po102

up

success

success

up

success

success

10

102400 Eth101/1/1

102401 Eth101/1/2

up

success

success

102402 Eth101/1/3

down*

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

Applicable

Performed

Not

Consistency Check Not

102403 Eth101/1/4

102404 Eth101/1/5

down*

down*

102405 Eth101/1/6

down*

102406 Eth101/1/7

down*

102407 Eth101/1/8

102408 Eth101/1/9

down*

down*

<snip>
N5K2# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID

Bridge ID

Interface

Priority

32778

Address

000d.eca2.edbc

Cost

Port

4096 (port-channel1)

Hello Time

Priority

32778

Address

000d.eca4.743c

Hello Time

sec

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Root FWD 1

128.4096 (vPC peer-link) Network P2p

Eth1/1

Desg FWD 4

128.129

Edge P2p

Eth1/2

Desg FWD 4

128.130

Edge P2p

Eth101/1/1

Desg FWD 1

128.1153 (vPC) Edge P2p

Eth102/1/1

Desg FWD 1

128.1281 (vPC) Edge P2p

Nexus Technology Labs - Virtual Port


Channels (vPC)
vPC and HSRP
Task
This task applies only to INE Bootcamp partipants. Load
balancing is not included in the self-paced training curriculum at
this time.
Configure vPC between N7K1 and N7K2 as follows:
N7K1 and N7K2 are the vPC Peers.
Configure all available F1 ports between the vPC peers as Port-Channel 1,
and use this as the vPC Peer Link.
Use the mgmt0 ports for the Peer Keepalive Link.
Configure all available links from N7K1 and N7K2 to N5K1 in Port-Channel
51, and as vPC 51.
All port- channels should be trunks, STP Network Ports, and use LACP for
negotiation.
Configure VLAN assignments and the servers as follows:
Configure the link from N5K1 to Server 1 as an access port in VLAN 10.
Configure the link from N5K1 to Server 2 as an access port in VLAN 20.
Server 1 should use the IP address 10.0.0.1/24, and a default gateway of
10.0.0.254.
Server 2 should use the IP address 20.0.0.2/24, and a default gateway of
20.0.0.254.
Configure Inter-VLAN Routing and HSRP on N7K1 and N7K2 as follows:
Create interfaces VLAN 10 and VLAN 20 on N7K1 and N7K2, using the IP
address 10.0.0.X/24, where X is the last octet of the IP address on their
mgmt0 interfaces.
Configure HSRP group 10 for VLAN 10 on N7K1 and N7K2 using the virtual
address 10.0.0.254/24.

Configure HSRP group 20 for VLAN 20 on N7K1 and N7K2 using the virtual
address 20.0.0.254/24.
Set the Port Channel load balancing method on the Nexus switches to include the
source and destination layer 4 port numbers.
When complete, Server 1 and Server 2 should have IP reachability to each other,
and traffic between them should be load distributed across all links in the vPC.

Configuration
N5K1:
feature lacp
!
vlan 10,20
!
port-channel load-balance ethernet source-dest-port
!
interface Ethernet1/1
switchport mode access
switchport access vlan 10
speed 1000
!
interface Ethernet1/2
switchport mode access
switchport access vlan 20
speed 1000

interface Ethernet1/6-9
switchport mode trunk
spanning-tree port type network
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport mode trunk
spanning-tree port type network
N7K1-1:
feature vpc
feature lacp
!
vlan 10,20
!
port-channel load-balance src-dst ip-l4port-vlan
!

vpc domain 1
peer-keepalive destination 192.168.0.75
!
interface Ethernet2/1-2
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/3-4
switchport mode trunk
spanning-tree port type network
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk
spanning-tree port type network
vpc 51
!
interface Vlan10
no shutdown
ip address 10.0.0.71/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.71/24
hsrp 20
ip 20.0.0.254
N7K2-1:

feature vpc
feature lacp
!
vlan 10,20
!
port-channel load-balance src-dst ip-l4port-vlan

!
vpc domain 1
peer-keepalive destination 192.168.0.71
!
interface Ethernet2/1-2
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/5-6
switchport mode trunk
spanning-tree port type network
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk
spanning-tree port type network
vpc 51
!
interface Vlan10
no shutdown
ip address 10.0.0.75/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.75/24
hsrp 20
ip 20.0.0.254

Verification
This scenario demonstrates how the forwarding pattern of vPC and HSRP combined
differs from that of just HSRP on its own. The results of this scenario would be

similar if either VRRP or GLBP were used, because all of the First Hop Redundancy
Protocols (FHRPs) have special behavior that interacts with vPC.
First, the layer 2 only switch N5K1 has access ports in VLANs 10 and 20, and a
trunking port channel that carries both VLANs. From N5K1s perspective, this port
channel logically connects to just one upstream switch, but in reality it is the two
physical vPC Peers, N7K1 and N7K2.
N5K1# show spanning-tree vlan 10,20
VLAN0010
Spanning tree enabled protocol rstp
Root ID

Bridge ID

Priority

4106

Address

68bd.abd7.6041

Cost

Port

4146 (port-channel51)

Hello Time

Priority

32778

Address

000d.eca2.edbc

Hello Time

Interface

sec

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------- Po51


Root FWD 1

128.4146 Network P2p Eth1/1

Desg FWD 4

128.129

P2p

VLAN0020
Spanning tree enabled protocol rstp
Root ID

Bridge ID

Interface

Priority

4116

Address

68bd.abd7.6041

Cost

Port

4146 (port-channel51)

Hello Time

Priority

32788

Address

000d.eca2.edbc

Hello Time

sec

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 20)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------- Po51


Root FWD 1
Desg FWD 4

128.4146 Network P2p Eth1/2


128.130

P2p

According to normal layer 2 switching vs. layer 3 routing logic, any hosts in VLAN 10

that want to talk to hosts in VLAN 20 must have their traffic switched up to the
default gateway and have the layer 2 header re-written with a new source and
destination MAC address, then switched to the final destination. In this case, there
are two default gateways for each VLAN 10 and 20, both N7K1 and N7K2 that share
the HSRP virtual IP address. In the output below we can see that N7K1 is the active
HSRP router for both groups.
N7K1-1# show hsrp
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Active
, priority 100 (Cfged 100)
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 1.595000 sec(s)
Virtual IP address is 10.0.0.254 (Cfged)
Active router is local
Standby router is 10.0.0.75 , priority 100 expires in 5.957000 sec(s)
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac0a (Default MAC)
4 state changes, last state change 00:48:39
IP redundancy name is hsrp-Vlan10-10 (default)
Vlan20 - Group 20
(HSRP-V1) (IPv4) Local state is Active
, priority 100 (Cfged 100)
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 1.594000 sec(s)
Virtual IP address is 20.0.0.254 (Cfged)
Active router is local
Standby router is 20.0.0.75 , priority 100 expires in 6.264000 sec(s)
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac14 (Default MAC)
2 state changes, last state change 02:33:39
IP redundancy name is hsrp-Vlan20-20 (default)
N7K2-1# show hsrp
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Standby
, priority 100 (Cfged 100)
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 2.399000 sec(s)
Virtual IP address is 10.0.0.254 (Cfged)
Active router is 10.0.0.71, priority 100 expires in 9.872000 sec(s)
Standby router is local
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac0a (Default MAC)

7 state changes, last state change 00:23:01


IP redundancy name is hsrp-Vlan10-10 (default)
Vlan20 - Group 20
(HSRP-V1) (IPv4) Local state is Standby
, priority 100 (Cfged 100)
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.602000 sec(s)
Virtual IP address is 20.0.0.254 (Cfged)
Active router is 20.0.0.71, priority 100 expires in 5.152000 sec(s)
Standby router is local
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac14 (Default MAC)
6 state changes, last state change 00:23:01
IP redundancy name is hsrp-Vlan20-20 (default)

The potential problem with this design is that if traffic is switched to N7K2, the
standby HSRP router, because of the Port Channel load balancing method of N5K1,
it would have to be sent to the active HSRP router, N7K1, to be routed. This means
that traffic would have to transit the vPC Peer Link, which is undesirable because
the aggregate of flows from vPC Member Ports would quickly overwhelm the vPC
Peer Link. To prevent this from being necessary, vPC changes the behavior of the
FHRPs so that the standby router can forward the same as the active router. The
result of this can be seen below.
Server 2 generates bulk TCP flows to Server 1 using the iPerf app. The aggregate
of flows nears 1Gbps.

When the access switch, N5K1, receives these flows, they have the destination
MAC address of the virtual HSRP address. This MAC address is reachable via the

port channel to the upstream 7K, and is then load balanced based on the layer 4
port information of the flows as configured.
N5K1# show port-channel summary
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------51

Po51(SU)

Eth

LACP Eth1/6(P)

Eth1/7(P)

Eth1/8(P)

Eth1/9(P)

N5K1# show interface e1/6 - 9 | include rate|Ethernet


Ethernet1/6 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8d (bia 000d.eca2.ed8d)
30 seconds input rate 2930112 bits/sec, 5292 packets/sec
30 seconds output rate 117990496 bits/sec, 13644 packets/sec
input rate 2.96 Mbps, 5.28 Kpps; output rate 111.59 Mbps
, 13.04 Kpps
Ethernet1/7 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8e (bia 000d.eca2.ed8e)
30 seconds input rate 232864888 bits/sec, 21953 packets/sec
30 seconds output rate 117809336 bits/sec, 13602 packets/sec
input rate 222.07 Mbps, 20.96 Kpps; output rate 113.44 Mbps
, 13.19 Kpps
Ethernet1/8 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8f (bia 000d.eca2.ed8f)
30 seconds input rate 451188568 bits/sec, 37074 packets/sec
30 seconds output rate 342359448 bits/sec, 29597 packets/sec
input rate 433.19 Mbps, 35.54 Kpps; output rate 326.19 Mbps
, 28.20 Kpps
Ethernet1/9 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed90 (bia 000d.eca2.ed90)
30 seconds input rate 229147160 bits/sec, 21784 packets/sec
30 seconds output rate 337983576 bits/sec, 29262 packets/sec
input rate 220.18 Mbps, 20.96 Kpps; output rate 327.17 Mbps
, 28.28 Kpps

In the output above, we see that some traffic goes from N5K1 to N7K1, and some

from N5K1 to N7K2. Without the vPC modification to HSRP, this traffic should have
to be switched from N7K2 to N7K1 before it can be routed, because N7K2 isnt the
active HSRP router. However, the interface counters of the vPC Peer Link, as seen
below, indicate that the flows are not switched in that direction, and instead N7K2 is
routing them itself even though it is HSRP standby.
N7K2-1# show interface port-channel 1 | include rate
30 seconds input rate 1560 bits/sec, 1 packets/sec
30 seconds output rate 1544 bits/sec, 1 packets/sec input rate 1.56 Kbps, 1 pps; output rate 1.54 Kbps
, 1 pps

This behavior can be further verified by disabling the uplinks from N5K1 to N7K1, as
shown below.
N5K1# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute

Device-ID

Local Intrfce Hldtme Capability

Platform

Port ID

Nexus-MGMT-SW

mgmt0

175

S I

WS-C3550-48

N5K2(FLC12480280)

Eth1/3

140

S I s

N5K-C5020P-BF Eth1/3

N5K2(FLC12480280)

Eth1/4

140

S I s

N5K-C5020P-BF Eth1/4

N5K2(FLC12480280)

Eth1/5

140

S I s

N5K-C5020P-BF Eth1/5

N7K1-1(JAF1510CMLQ)

Eth1/6

170

R S s

N7K-C7010

Eth2/3

169

R S s

N7K-C7010

Eth2/4

Fas0/31

N7K1-1(JAF1510CMLQ)

N7K2-1(TBM14311481)

Eth1/8

167

R S s

N7K-C7010

Eth2/5

N7K2-1(TBM14311481)

Eth1/9

167

R S s

N7K-C7010

Eth2/6

Eth1/7

N5K1# config t
Enter configuration commands, one per line.

End with CNTL/Z. N5K1(config)# int e1/6 7

N5K1(config-if-range)# shutdown

2013 Mar

5 21:54:43 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel51: Ethernet1/7 is down

2013 Mar

5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/7 is down(Config change)

2013 Mar

5 21:54:43 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel51: Ethernet1/6 is down

2013 Mar

5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/6 is down(Config change)

2013 Mar

5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/7 is down (Administratively down)

2013 Mar

5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/6 is down (Administratively down)

With the links from N5K1 to N7K1 disabled, the only way for them to be switched

northbound is via N7K2, as shown below.


vN5K1# show interface e1/6 - 9 | include Ethernet1|rate
Ethernet1/6 is down (Administratively down)
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
Ethernet1/7 is down (Administratively down)
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
Ethernet1/8 is up
30 seconds input rate 231436152 bits/sec, 21853 packets/sec
30 seconds output rate 455582888 bits/sec, 42864 packets/sec
input rate 231.44 Mbps, 21.85 Kpps; output rate 455.58 Mbps
, 42.86 Kpps
Ethernet1/9 is up
30 seconds input rate 682938608 bits/sec, 63848 packets/sec
30 seconds output rate 458806416 bits/sec, 42838 packets/sec
input rate 682.94 Mbps, 63.85 Kpps; output rate 458.81 Mbps
, 42.84 Kpps

Because N7K1 still has the vPC Peer Link forwarding VLANs 10 and 20, it is still the
HSRP Active router.
N7K1-1# show hsrp | include state|Group
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Active
, priority 100 (Cfged 100)
4 state changes, last state change 02:02:24 Vlan20 - Group 20
(HSRP-V1) (IPv4) Local state is Active
, priority 100 (Cfged 100)
2 state changes, last state change 03:47:25
N7K2-1# show hsrp | include state|Group
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Standby
, priority 100 (Cfged 100)
7 state changes, last state change 01:36:48 Vlan20 - Group 20
(HSRP-V1) (IPv4) Local state is Standby
, priority 100 (Cfged 100)
6 state changes, last state change 01:36:48

If N7K1 is the router that is doing the layer 2 header re-write, the vPC Peer Link

should show 1Gbps input and output, which it does not according to the output
below.
N7K2-1# show interface port-channel 1 | include rate

30 seconds input rate 1576 bits/sec, 1 packets/sec


30 seconds output rate 1504 bits/sec, 1 packets/sec input rate 1.58 Kbps, 1 pps; output rate 1.50 Kbps
, 1 pps

Note that this behavior, in which both the active and standby HSRP routers are able
to forward traffic, is the default. There is no additional configuration needed to
accomplish this. As long as HSRP/VRRP/GLBP is configured in conjunction with
vPC, this behavior will be seen.

Nexus Technology Labs - Virtual Port


Channels (vPC)
Back-to-Back vPC
Task
Configure a vPC domain between N7K1 and N7K2 as follows:
N7K1 and N7K2 are the vPC Peers.
Configure vPC domain 70 between the vPC Peers, and use their mgmt0
interfaces for the vPC Peer Keepalive Link.
Configure all available F1 ports between the vPC peers as Port-Channel 1,
and use this as the vPC Peer Link.
Configure all available links from N7K1 and N7K2 to N5K1 and N5K2 in PortChannel 70, and as vPC 70.
All port channels should be 802.1q trunks, be configured as STP Network
Ports, and use LACP for negotiation.
Configure a vPC domain between N5K1 and N5K2 as follows:
N5K1 and N5K2 are the vPC Peers.
Configure vPC domain 50 between the vPC Peers, and use their mgmt0
interfaces for the vPC Peer Keepalive Link.
Configure all available ports between the vPC peers as Port-Channel 1, and
use this as the vPC Peer Link.
Configure all available links from N5K1 and N5K2 to N7K1 and N7K2 in PortChannel 50, and as vPC 50.
Port Channels 1 and 50 should be 802.1q trunks, be configured as STP
Network Ports, and use LACP for negotiation.
Configure a vPC from N5K1 and N5K2 to Server 1 as follows:
Use LACP negotiation for all port channels and NIC Teams.
Configure the links from N5K1 and N5K2 to Server 1 as Port-Channel 10,
vPC 10, and as an access port in VLAN 10.
Configure the links from N5K1 and N5K2 to Server 2 as Port-Channel 20,
vPC 20, and as an access port in VLAN 20.
Server 1 should do NIC Teaming for its links to N5K1 and N5K2, use the IP

address 10.0.0.1/24 for the NIC Team, and have a default gateway of
10.0.0.254.
Server 2 should do NIC Teaming for its links to N5K1 and N5K2, use the IP
address 20.0.0.2/24 for the NIC Team, and have a default gateway of
20.0.0.254.
Configure Inter-VLAN Routing on N7K1 and N7K2 as follows:
Create interfaces VLAN 10 and VLAN 20 on N7K1 and N7K2, using the IP
address 10.0.0.X/24, where X is the last octet of the IP address on their
mgmt0 interfaces.
Configure HSRP group 10 for VLAN 10 on N7K1 and N7K2 using the virtual
address 10.0.0.254/24.
Configure HSRP group 20 for VLAN 20 on N7K1 and N7K2 using the virtual
address 20.0.0.254/24.
Set the Port Channel load balancing method on the Nexus switches to include the
source and destination layer 4 port numbers.
When complete, Server 1 and Server 2 should have IP reachability to each other,
and traffic between them should be load distributed across all links in the vPCs.

Configuration
N5K1:
feature lacp
feature vpc
!
vlan 10,20
!
port-channel load-balance ethernet source-dest-port
!
vpc domain 50
peer-keepalive destination 192.168.0.52
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel10
switchport mode access
switchport access vlan 10
speed 1000
vpc 10

!
interface port-channel20
switchport mode access
switchport access vlan 20
speed 1000
vpc 20
!
interface port-channel50
switchport mode trunk
spanning-tree port type network
vpc 50
!
interface Ethernet1/1
switchport mode access
switchport access vlan 10
channel-group 10 mode active
speed 1000
!
interface Ethernet1/2
switchport mode access
switchport access vlan 20
channel-group 20 mode active
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/6 - 9
switchport mode trunk
spanning-tree port type network
channel-group 50 mode active
N5K2:
feature lacp
feature vpc
!
vlan 10,20
!
port-channel load-balance ethernet source-dest-port
!
vpc domain 50
peer-keepalive destination 192.168.0.51
!
interface port-channel1
switchport mode trunk

spanning-tree port type network


vpc peer-link
!
interface port-channel10
switchport mode access
switchport access vlan 10
speed 1000
vpc 10
!
interface port-channel20
switchport mode access
switchport access vlan 20
speed 1000
vpc 20
!
interface port-channel50
switchport mode trunk
spanning-tree port type network
vpc 50
!
interface Ethernet1/1
switchport mode access
switchport access vlan 10
channel-group 10 mode active
speed 1000
!
interface Ethernet1/2
switchport mode access
switchport access vlan 20
channel-group 20 mode active
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/6 - 9
switchport mode trunk
spanning-tree port type network
channel-group 50 mode active
N7K1:
feature vpc
feature lacp
feature hsrp
feature interface-vlan

!
vlan 10,20
!
port-channel load-balance src-dst ip-l4port-vlan
!
vpc domain 70
peer-keepalive destination 192.168.0.75
!
interface Ethernet2/1-2
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/3-6
switchport
switchport mode trunk
spanning-tree port type network
channel-group 70 mode active
no shutdown
!
interface port-channel70
switchport
switchport mode trunk
spanning-tree port type network
vpc 70
!
interface Vlan10
no shutdown
ip address 10.0.0.71/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.71/24
hsrp 20
ip 20.0.0.254
N7K2:

feature vpc
feature lacp
feature hsrp
feature interface-vlan
!
vlan 10,20
!
port-channel load-balance src-dst ip-l4port-vlan
!
vpc domain 70
peer-keepalive destination 192.168.0.71
!
interface Ethernet2/1-2
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/3-6
switchport
switchport mode trunk
spanning-tree port type network
channel-group 70 mode active
no shutdown
!
interface port-channel70
switchport
switchport mode trunk
spanning-tree port type network
vpc 70
!
interface Vlan10
no shutdown
ip address 10.0.0.75/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.75/24

hsrp 20
ip 20.0.0.254

Verification
Back-to-Back vPC is not a specific configuration, but instead refers to a design in
which two vPC peers have a vPC configured toward another set of vPC peers,
which likewise have a vPC configured back the other direction. In this particular
design, the Back-to-Back vPC consists of the vPC from the Nexus 7Ks southbound
to the Nexus 5Ks, and then from the Nexus 5Ks northbound back to the Nexus 7Ks.
The resulting logical topology is a single logical 8-way port channel between the
boxes.
From the Nexus 7K's perspective, this simply looks like a normal vPC configuration,
with Port-Channel 70 being the vPC Member Port.
N7K1# show vpc

Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 70

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status

: success

Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: secondary

Number of vPCs configured

: 1

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

Auto-recovery status

: Disabled

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ --------------------------------------------------

Po1

up

1,10,20

vPC status
---------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

--

----

------ ----------- ------

------------

70

Po70

up

success

success

1,10,20

From the Nexus 5K's perspective, this likewise looks like a normal vPC
configuration, with Port-Channels 10 and 20 going southbound to the end-hosts,
and Port-Channel 50 going northbound to the 7Ks.
N5K1# show vpc

Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 50

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status: success


Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: primary

Number of vPCs configured

: 3

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ --------------------------------------------------

Po1

up

1,10,20

vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------10

Po10

up

success

success

10

20

Po20

up

success

success

20

50

Po50

up

success

success

1,10,20

To test the final distribution of traffic flows, Server 2 uses the iPerf app to generate a
number of TCP flows to Server 1, as shown below. The aggregate of flows is near
1Gbps. Although you might expect the aggregate to be closer to 2Gbps, which is
both links in the NIC Team of the end servers, the load balancing method of the Intel
NIC card is always choosing the same member of the NIC Team because the
source and destination IP address is the same for all flows. If two or more

destinations had flows going to them, both links of the team could be utilized.

When the traffic flows enter the 5Ks, their more granular method of load balancing
based on both source and destination TCP ports allows the flows to be more evenly
split among the members of the vPCs. As shown below, N5K1 is receiving the
majority of the traffic in from Server 2.
N5K1# show interface e1/2 | include rate
30 seconds input rate 950132072 bits/sec, 78257 packets/sec
30 seconds output rate 2781568 bits/sec, 5402 packets/sec input rate 950.13 Mbps
, 78.26 Kpps; output rate 2.78 Mbps, 5.40 Kpps
N5K2# show interface e1/2 | include rate
30 seconds input rate 592 bits/sec, 0 packets/sec
30 seconds output rate 4094712 bits/sec, 7938 packets/sec input rate 592 bps
, 0 pps; output rate 4.09 Mbps, 7.94 Kpps

N5K1 then distributes the load of these ~ 1Gbps flows between the members of vPC 50, which go northbound to the 7Ks.
N5K1# show port-channel summary interface po50
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------50

Po50(SU)

Eth

LACP

Eth1/6(P)

Eth1/7(P)

Eth1/9(P)
N5K1# show interface E1/6 - 9 | include Ethernet1|rate

Eth1/8(P)

Ethernet1/6 is up
30 seconds input rate 251447800 bits/sec, 20649 packets/sec
30 seconds output rate 237787808 bits/sec, 22349 packets/sec

input rate 251.45 Mbps, 20.65 Kpps;

output rate 237.79 Mbps


, 22.35 Kpps
Ethernet1/7 is up
30 seconds input rate 128373136 bits/sec, 10541 packets/sec
30 seconds output rate 255434592 bits/sec, 23809 packets/sec

input rate 128.37 Mbps, 10.54 Kpps;

output rate 255.43 Mbps


, 23.81 Kpps
Ethernet1/8 is up
30 seconds input rate 2013120 bits/sec, 3680 packets/sec
30 seconds output rate 226323776 bits/sec, 22035 packets/sec

input rate 2.01 Mbps, 3.68 Kpps;

output rate 226.32 Mbps


, 22.03 Kpps
Ethernet1/9 is up
30 seconds input rate 962168 bits/sec, 1751 packets/sec
30 seconds output rate 227361664 bits/sec, 22371 packets/sec

input rate 962.17 Kbps, 1.75 Kpps;

output rate 227.36 Mbps


, 22.37 Kpps

When the 7Ks receive the flows in, they likewise load-balance them back down to
the 5Ks after the routing process occurs on the layer 3 VLAN interfaces. Note that if
the traffic was Intra-VLAN, the flows would not need to go all the way north to the
7Ks, but instead would be directly locally switched by the 5Ks. Also note that the
distribution of flows is not equal from the 7Ks back down to the 5Ks, simply because
of the uneven hashing results of the flows' input src/dst IP addresses and TCP
ports. Over a longer-term average and with a more diverse distribution of flows, the
distribution of traffic among the vPC member ports would be more even.
N7K1# show port-channel summary interface port-channel 70
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------70

Po70(SU)

Eth

LACP

Eth2/3(P)

Eth2/4(P)

Eth2/6(P)
N7K1# show interface e2/3 - 6 | include Ethernet2|rate

Eth2/5(P)

Ethernet2/3 is up
30 seconds input rate 236864320 bits/sec, 22331 packets/sec
30 seconds output rate 252770928 bits/sec, 20759 packets/sec

input rate 236.86 Mbps, 22.33 Kpps;

output rate 252.77 Mbps


, 20.76 Kpps
Ethernet2/4 is up
30 seconds input rate 260953864 bits/sec, 24287 packets/sec
30 seconds output rate 131586224 bits/sec, 10805 packets/sec

input rate 260.95 Mbps, 24.29 Kpps;

output rate 131.59 Mbps


, 10.81 Kpps
Ethernet2/5 is up
30 seconds input rate 696 bits/sec, 0 packets/sec
30 seconds output rate 3292624 bits/sec, 5998 packets/sec

input rate 696 bps, 0 pps;

output rate 3.29 Mbps


, 6.00 Kpps
Ethernet2/6 is up
30 seconds input rate 40 bits/sec, 0 packets/sec
30 seconds output rate 110174952 bits/sec, 9046 packets/sec

input rate 40 bps, 0 pps;

output rate 110.17 Mbps


, 9.05 Kpps
N7K2# show port-channel summary interface port-channel 70
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------70

Po70(SU)

Eth

LACP

Eth2/3(P)

Eth2/4(P)

Eth2/5(P)

Eth2/6(P)
N7K2# show interface e2/3 - 6 | include Ethernet2|rate
Ethernet2/3 is up
30 seconds input rate 48 bits/sec, 0 packets/sec
30 seconds output rate 454982944 bits/sec, 37376 packets/sec

input rate 48 bps, 0 pps;

output rate 454.98 Mbps


, 37.38 Kpps
Ethernet2/4 is up
30 seconds input rate 48 bits/sec, 0 packets/sec
30 seconds output rate 1104384 bits/sec, 2016 packets/sec
output rate 1.10 Mbps
, 2.02 Kpps
Ethernet2/5 is up
30 seconds input rate 232675160 bits/sec, 22524 packets/sec

input rate 48 bps, 0 pps;

30 seconds output rate 2013416 bits/sec, 3681 packets/sec

input rate 232.68 Mbps, 22.52 Kpps;

output rate 2.01 Mbps


, 3.68 Kpps
Ethernet2/6 is up
30 seconds input rate 226382000 bits/sec, 22279 packets/sec
30 seconds output rate 945056 bits/sec, 1719 packets/sec
output rate 945.06 Kbps
, 1.72 Kpps

input rate 226.38 Mbps, 22.28 Kpps;

Nexus Technology Labs - Virtual Port


Channels (vPC)
vPC Peer Switch
Task
Configure a vPC between N7K1, N7K2, and N5K1 as follows:
N7K1 and N7K2 are the vPC Peers.
Configure all available F1 ports between the vPC peers as Port-Channel 1,
and use this as the vPC Peer Link.
Use the mgmt0 ports for the Peer Keepalive Link.
Configure all available links from N7K1 and N7K2 to N5K1 in Port-Channel
51, and as vPC 51.
All port channels should be trunks, and use LACP for negotiation.
Create VLAN 10 on N7K1, N7K2, and N5K1, and set the spanning-tree priority to
4096 on both N7K1 and N7K2 for this VLAN.
Using the vPC peer-switch feature, configure N7K1 and N7K2 so that they both
share the STP root bridge role for VLAN 10.

Configuration
N5K1:
feature lacp
!
vlan 10
!
interface Ethernet1/6-9
switchport mode trunk
channel-group 51 mode active
no shutdown
N7K1-1:
feature vpc
feature lacp
!
vlan 10

spanning-tree vlan 10 priority 4096


!
vpc domain 1
peer-switch
peer-keepalive destination 192.168.0.75
!
interface Ethernet2/1-2
switchport mode trunk
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/3-4
switchport mode trunk
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk
vpc 51
N7K2-1:

feature vpc
feature lacp
!
vlan 10
spanning-tree vlan 10 priority 4096
!
vpc domain 1
peer-switch
peer-keepalive destination 192.168.0.71
!
interface Ethernet2/1-2
switchport mode trunk
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk

spanning-tree port type network


vpc peer-link
!
interface Ethernet2/5-6
switchport mode trunk
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk
vpc 51

Verification
Before applying the peer-switch feature, both N7K1 and N7K2 share the same STP
priority value, but the switch with the lower MAC address portion of their STP BridgeID is elected the root bridge. This is the normal expected behavior of STP.
N7K1-1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Address

Cost

Port

4096 (port-channel1)

Hello Time

sec

4106

Max Age 20 sec

Forward Delay 15 sec

(priority 4096 sys-id-ext 10)

68bd.abd7.6041
Hello Time

Interface

4106

0026.980c.2141

Bridge ID Priority
Address

Root ID Priority

sec

Max Age 20 sec

Role Sts Cost

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Root FWD 1

128.4096 (vPC peer-link) Network P2p

Po51

Desg FWD 1

128.4146 (vPC) P2p

N7K2-1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Address

Root ID Priority

4106

0026.980c.2141

This bridge is the root


Hello Time
Bridge ID Priority

2
4106

sec

Max Age 20 sec

Forward Delay 15 sec

(priority 4096 sys-id-ext 10)

Address

0026.980c.2141

Hello Time

Interface

sec

Max Age 20 sec

Role Sts Cost

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Desg FWD 1

128.4096 (vPC peer-link) Network P2p

Po51

Desg FWD 1

128.4146 (vPC) P2p

N5K1# sh spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp

Root ID Priority
Address

Bridge ID

Interface

Cost

Port

4146 (port-channel51)

Hello Time

Priority

32778

Address

547f.ee79.137c

Hello Time

sec

Max Age 20 sec

0026.980c.2141

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

sec

Max Age 20 sec

Role Sts Cost

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po51

Root FWD 1

128.4146 P2p

After enabling the peer-switch feature, both N7K1 and N7K2 share the same STP
Bridge-ID, so both see themselves as the STP Root Bridge. This is an exception to
normal STP rules that is specific to the vPC.
vN7K1-1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Address

Root ID Priority

4106

0023.04ee.be01

This bridge is the root


Hello Time
Bridge ID Priority
Address

sec

4106

Max Age 20 sec

Forward Delay 15 sec

(priority 4096 sys-id-ext 10)

0023.04ee.be01
Hello Time

Interface

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Root FWD 1

128.4096 (vPC peer-link) Network P2p

4106

Po51

Desg FWD 1

128.4146 (vPC) P2p

N7K2-1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Address

Root ID Priority

4106

0023.04ee.be01

This bridge is the root


Hello Time
Bridge ID Priority
Address

sec

4106

Max Age 20 sec

Forward Delay 15 sec

(priority 4096 sys-id-ext 10)

0023.04ee.be01
Hello Time

Interface

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Desg FWD 1

128.4096 (vPC peer-link) Network P2p

Po51

Desg FWD 1

128.4146 (vPC) P2p

N5K1# sh spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Address

Bridge ID

Interface

Root ID Priority

4106

0023.04ee.be01

Cost

Port

4146 (port-channel51)

Hello Time

Priority

32778

Address

547f.ee79.137c

Hello Time

sec

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po51

Root FWD 1

128.4146 P2p

Nexus Technology Labs - Virtual Port


Channels (vPC)
vPC Customization
Task
Configure a vPC between N7K1, N7K2, and N5K1 as follows:
N7K1 and N7K2 are the vPC Peers.
Configure all available F1 ports between the vPC peers as Port-Channel 1,
and use this as the vPC Peer Link.
Use the mgmt0 ports for the Peer Keepalive Link.
Configure all available links from N7K1 and N7K2 to N5K1 in Port-Channel
51, and as vPC 51.
All port channels should be trunks and use LACP for negotiation.
Customize the vPC configuration on N7K1 and N7K2 as follows:
The vPC peers should use the vPC system MAC address of
1234.5678.9ABC and an LACP system priority of 1000.
N7K1 should be the primary vPC peer with a role priority of 10, and N7K2
should be the secondary vPC peer with a role priority of 20.
In the case that one of the vPC peers is reloaded, vPCs should be delayed
from coming up for 60 seconds after the vPC peer adjacency is reestablished.
In the case that a Type 1 mismatch occurs for a particular VLAN, that VLAN
should be suspended on the vPCs of both the primary and the secondary
vPC peers.
In the case that the vPC Peer Link flaps, both the IPv4 ARP Cache and IPv6
ICMP ND Cache should be syncronized between the vPC peers upon
reconvergence of the vPC Peer Link.
In the case that both vPC Peers fail but only one of them is able to be
restored, the newly restored peer should wait five minutes before declaring
itself the active vPC Peer and bringing its vPCs up.

Configuration
N5K1:
feature lacp
!
interface Ethernet1/6-9
switchport mode trunk
channel-group 51 mode active
no shutdown
N7K1-1:
feature vpc
feature lacp
!
vpc domain 1
role priority 10
system-mac 12:34:56:78:9a:bc
system-priority 1000
peer-keepalive destination 192.168.0.75
delay restore 60
no graceful consistency-check
auto-recovery reload-delay 300
ipv6 nd synchronize
ip arp synchronize
!
interface Ethernet2/1-2
switchport mode trunk
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/3-4
switchport mode trunk
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk

vpc 51
N7K2-1:

feature vpc
feature lacp
!
vpc domain 1
role priority 20
system-mac 12:34:56:78:9a:bc
system-priority 1000
peer-keepalive destination 192.168.0.71
delay restore 60
no graceful consistency-check
auto-recovery reload-delay 300
ipv6 nd synchronize
ip arp synchronize
!
interface Ethernet2/1-2
switchport mode trunk
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/5-6
switchport mode trunk
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk
vpc 51

Verification
N5K1# show lacp neighbor interface port-channel 51
Flags:

S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs


A - Device is in Active mode

port-channel51 neighbors

P - Device is in Passive mode

Partner's information

Port

Partner

Partner

System ID

Port Number

0x4203

2590

Partner
Age

Flags Eth1/6

1000,12-34-56-78-9a-bc

SA

LACP Partner

Partner

Partner

Port Priority

Oper Key

Port State

32768

0x8033

0x3d

Partner

Partner

Partner

System ID

Port Number

Partner's information

Port
0x4204

2590

Age

Flags Eth1/7

1000,12-34-56-78-9a-bc

SA

LACP Partner

Partner

Partner

Port Priority

Oper Key

Port State

32768

0x8033

0x3d

Partner

Partner

Partner

System ID

Port Number

Partner's information

Port
0x205

2591

Age

Flags Eth1/8

1000,12-34-56-78-9a-bc

SA

LACP Partner

Partner

Partner

Port Priority

Oper Key

Port State

32768

0x8033

0x3d

Partner

Partner

Partner

System ID

Port Number

Partner's information

Port
0x206

2590

Age

Flags Eth1/9

1000,12-34-56-78-9a-bc

SA

LACP Partner

Partner

Partner

Port Priority

Oper Key

Port State

32768

0x8033

0x3d

N7K1-1# show vpc role

vPC Role status


---------------------------------------------------- vPC role

Dual Active Detection Status

: 0 vPC system-mac

N7K2-1# show vpc role

: 12:34:56:78:9a:bc

: 1000

vPC system-priority
vPC local system-mac

: primary

: 68:bd:ab:d7:60:41

vPC local role-priority

: 10

vPC Role status


---------------------------------------------------- vPC role

Dual Active Detection Status

: secondary

: 0 vPC system-mac

vPC system-priority

: 12:34:56:78:9a:bc

vPC local system-mac

: 1000
: 00:26:98:0c:21:41

vPC local role-priority

N7K1-1# show vpc


Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status

: success

Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: primary

Number of vPCs configured

: 1

Peer Gateway

: Disabled

Dual-active excluded VLANs

: - Graceful Consistency Check

Auto-recovery status

: Enabled (timeout = 300 seconds)

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ --------------------------------------------------

Po1

up

vPC status
---------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

--

----

------ ----------- ------

------------

51

Po51

up

success

success

N7K1-1# show ip arp vpc-statistics


ARP sync Enabled

ARP vPC global statistics


Ignore to send pull request using CFSoE as the feature is off : 6
Ignore to send push message using CFSoE as the feature is off : 8
Response sent via CFSoE : 1

: Disabled

: 20

Received message via CFSoE : 1

Nexus Technology Labs - Virtual Port


Channels (vPC)
Nexus 7K Virtual Port Channels (vPC)
Task
Configure a vPC between N7K1, N7K2, and N5K1 as follows:
N7K1 and N7K2 are the vPC Peers.
Configure all available F1 ports between the vPC peers as Port-Channel 1,
and use this as the vPC Peer Link.
Use the mgmt0 ports for the Peer Keepalive Link.
Configure all available links from N7K1 and N7K2 to N5K1 in Port-Channel
51, and as vPC 51.
All port channels should be trunks, and use LACP for negotiation.

Configuration
N5K1:
feature lacp
!
interface Ethernet1/6-9
switchport mode trunk
channel-group 51 mode active
no shutdown
N7K1-1:
feature vpc
feature lacp
!
vpc domain 1
peer-keepalive destination 192.168.0.75
!
interface Ethernet2/1-2
switchport mode trunk
channel-group 1 mode active
no shutdown
!

interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/3-4
switchport mode trunk
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk
vpc 51
N7K2-1:

feature vpc
feature lacp
!
vpc domain 1
peer-keepalive destination 192.168.0.71
!
interface Ethernet2/1-2
switchport mode trunk
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/5-6
switchport mode trunk
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk
vpc 51

Verification
N5K1# show port-channel summary
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------- 51 Po51(SU)
Eth

LACP Eth1/6(P)

Eth1/7(P)

Eth1/8(P)

Eth1/9(P)

N7K1-1# show vpc


Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

vPC keep-alive status

: peer adjacency formed ok

Peer status
: peer is alive

Configuration consistency status

: success

Per-vlan consistency status

: success

Type-2 inconsistency reason

: Consistency Check Not Performed

vPC role

: secondary

Number of vPCs configured

: 1

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

Auto-recovery status

: Disabled

vPC Peer-link status


--------------------------------------------------------------------id
--

Port

Status Active vlans

----

------ -------------------------------------------------- 1

Po1 up

vPC status
---------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

--

----

------ ----------- ------

------------ 51

success

Po51 up

success

N7K2-1# show vpc


Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

Peer status

: peer adjacency formed ok

: peer is alive

vPC keep-alive status

Configuration consistency status

: success

Per-vlan consistency status

: success

Type-2 inconsistency reason

: Consistency Check Not Performed

vPC role

: primary

Number of vPCs configured

: 1

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

Auto-recovery status

: Disabled

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ -------------------------------------------------- 1

Po1 up

vPC status
---------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

--

----

------ ----------- ------

------------ 51

success

Po51 up

success

Nexus Technology Labs - Virtual Port


Channels (vPC)
Nexus 7K Virtual Port Channels (vPC) with Layer 3
Routed Keepalive
Task
Configure a vPC between N7K1, N7K2, and N5K1 as follows:
N7K1 and N7K2 are the vPC Peers.
Configure all available F1 ports between the vPC peers as Port-Channel 1,
and use this as the vPC Peer Link.
Configure all available M1 ports between the vPC peers as Port-Channel 2,
and use this as the Peer Keepalive Link. This link should be a native layer 3
routed interface in its own VRF.
Configure all available links from N7K1 and N7K2 to N5K1 in Port-Channel
51, and as vPC 51.
All layer 2 port channels should be trunks, and all channels should use
LACP for negotiation.

Configuration
N5K1:
feature lacp
!
interface Ethernet1/6-9
switchport mode trunk
channel-group 51 mode active
no shutdown
N7K1-1:
feature vpc
feature lacp
!
vrf context VPC_KEEPALIVE
!
vpc domain 1

peer-keepalive destination 169.254.0.2 source 169.254.0.1 vrf VPC_KEEPALIVE


!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel2
vrf member VPC_KEEPALIVE
ip address 169.254.0.1/30
!
interface port-channel51
switchport
switchport mode trunk
vpc 51
!
interface Ethernet1/1-2
channel-group 2 mode active
no shutdown
!
interface Ethernet2/1-2
switchport mode trunk
channel-group 1 mode active
no shutdown
!
interface Ethernet2/3-4
switchport mode trunk
channel-group 51 mode active
no shutdown
N7K2-1:

feature vpc
feature lacp
!
vrf context VPC_KEEPALIVE
!
vpc domain 1
peer-keepalive destination 169.254.0.1 source 169.254.0.2 vrf VPC_KEEPALIVE
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!

interface port-channel2
vrf member VPC_KEEPALIVE
ip address 169.254.0.2/30
!
interface port-channel51
switchport
switchport mode trunk
vpc 51
!
interface Ethernet1/1-2
channel-group 2 mode active
no shutdown
!
interface Ethernet2/1-2
switchport mode trunk
channel-group 1 mode active
no shutdown
!
interface Ethernet2/5-6
switchport mode trunk
channel-group 51 mode active
no shutdown

Verification
N5K1# show port-channel summary
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------- 51 Po51(SU)
Eth

LACP Eth1/6(P)

Eth1/7(P)

Eth1/8(P)

Eth1/9(P)

N7K1-1# show vpc


Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

Peer status

: peer adjacency formed ok


: peer is alive

vPC keep-alive status

Configuration consistency status

: success

Per-vlan consistency status

: success

Type-2 inconsistency reason

: Consistency Check Not Performed

vPC role

: secondary

Number of vPCs configured

: 1

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

Auto-recovery status

: Disabled

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ -------------------------------------------------- 1

Po1 up

vPC status
---------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

--

----

------ ----------- ------

------------ 51

success

Po51 up

success

N7K2-1# show vpc


Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

vPC keep-alive status

Peer status
: peer is alive

Configuration consistency status

: success

Per-vlan consistency status

: success

Type-2 inconsistency reason

: Consistency Check Not Performed

vPC role

: primary

Number of vPCs configured

: 1

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

Auto-recovery status

: Disabled

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ --------------------------------------------------

: peer adjacency formed ok

Po1 up
1

vPC status
---------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

--

----

------ ----------- ------

------------ 51

success

Po51 up

N7K1-1# show port-channel summary


Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------1

Po1(SU)

Po2(RU)

51

Eth

Eth

Po51(SU)

LACP
LACP

Eth

Eth2/1(P)
Eth1/1(P)

LACP

Eth2/2(P)

Eth1/2(P)

Eth2/3(P)

Eth2/4(P)

N7K1-1# show ip route vrf VPC_KEEPALIVE


IP Route Table for VRF "VPC_KEEPALIVE"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
169.254.0.0/30
, ubest/mbest: 1/0, attached

* via 169.254.0.1, Po2

, [0/0], 00:12:45, direct


169.254.0.1/32, ubest/mbest: 1/0, attached
*via 169.254.0.1, Po2, [0/0], 00:12:45, local
N7K1-1# ping 169.254.0.2 vrf VPC_KEEPALIVE

PING 169.254.0.2 (169.254.0.2): 56 data bytes


64 bytes from 169.254.0.2: icmp_seq=0 ttl=254 time=1.125 ms
64 bytes from 169.254.0.2: icmp_seq=1 ttl=254 time=0.746 ms
64 bytes from 169.254.0.2: icmp_seq=2 ttl=254 time=0.934 ms
64 bytes from 169.254.0.2: icmp_seq=3 ttl=254 time=0.826 ms
64 bytes from 169.254.0.2: icmp_seq=4 ttl=254 time=0.821 ms

success

Nexus Technology Labs - Virtual Port


Channels (vPC)
Nexus 7K Virtual Port Channels (vPC) with SVI
Keepalive
Task
Configure a vPC between N7K1, N7K2, and N5K1 as follows:
N7K1 and N7K2 are the vPC Peers.
Configure all available F1 ports between the vPC peers as Port-Channel 1,
and use this as the vPC Peer Link.
Configure all available M1 ports between the vPC peers as a layer 2 trunk,
and as Port-Channel 2.
Configure a layer 3 VLAN 999 interface for the Peer Keepalive of the vPC
peers. This SVI should be in its own VRF, and should not forward over the
vPC Peer Link or vPC member ports.
Configure all available links from N7K1 and N7K2 to N5K1 in Port-Channel
51, and as vPC 51.
All layer 2 port channels should be trunks, and all channels should use
LACP for negotiation.

Configuration
N5K1:
feature lacp
!
interface Ethernet1/6-9
switchport mode trunk
channel-group 51 mode active
no shutdown
N7K1-1:
feature interface-vlan
feature lacp
feature vpc
!

vrf context VPC_KEEPALIVE


!
vlan 1,999
!
vpc domain 1
peer-keepalive destination 169.254.0.2 source 169.254.0.1 vrf VPC_KEEPALIVE
!
interface Vlan999
no shutdown
vrf member VPC_KEEPALIVE
ip address 169.254.0.1/30
!
interface port-channel1
switchport
switchport mode trunk
switchport trunk allowed vlan 1-998,1000-4094
spanning-tree port type network
vpc peer-link
!
interface port-channel2
switchport
switchport mode trunk
!
interface port-channel51
switchport
switchport mode trunk
switchport trunk allowed vlan 1-998,1000-4094
vpc 51
!
interface Ethernet1/1-2
switchport
switchport mode trunk
channel-group 2 mode active
no shutdown
!
interface Ethernet2/1-2
switchport mode trunk
switchport trunk allowed vlan 1-998,1000-4094
channel-group 1 mode active
no shutdown
!
interface Ethernet2/3-4
switchport mode trunk
switchport trunk allowed vlan 1-998,1000-4094
channel-group 51 mode active
no shutdown

N7K2-1:

feature interface-vlan
feature lacp
feature vpc
!
vrf context VPC_KEEPALIVE
!
vlan 1,999
!
vpc domain 1
peer-keepalive destination 169.254.0.1 source 169.254.0.2 vrf VPC_KEEPALIVE
!
interface Vlan999
no shutdown
vrf member VPC_KEEPALIVE
ip address 169.254.0.2/30
!
interface port-channel1
switchport
switchport mode trunk
switchport trunk allowed vlan 1-998,1000-4094
spanning-tree port type network
vpc peer-link
!
interface port-channel2
switchport
switchport mode trunk
!
interface port-channel51
switchport
switchport mode trunk
switchport trunk allowed vlan 1-998,1000-4094
vpc 51
!
interface Ethernet1/1-2
switchport
switchport mode trunk
channel-group 2 mode active
no shutdown
!
interface Ethernet2/1-2
switchport mode trunk
switchport trunk allowed vlan 1-998,1000-4094
channel-group 1 mode active
no shutdown

!
interface Ethernet2/5-6
switchport mode trunk
switchport trunk allowed vlan 1-998,1000-4094
channel-group 51 mode active
no shutdown

Verification
N7K1-1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status

: success

Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: secondary

Number of vPCs configured

: 1

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

Auto-recovery status

: Disabled

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ --------------------------------------------------

Po1

up

vPC status
---------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

--

----

------ ----------- ------

------------

51

Po51

up

success

success

N7K1-1# show vpc peer-keepalive

vPC keep-alive status

: peer is alive

--Peer is alive for

: (1825) seconds, (714) msec

--Send status

: Success

--Last send at

: 2013.02.12 19:01:15 528 ms

--Sent on interface

: Vlan999

--Receive status

: Success

--Last receive at

: 2013.02.12 19:01:15 941 ms

--Received on interface

: Vlan999

--Last update from peer

: (0) seconds, (221) msec

vPC Keep-alive parameters


--Destination

: 169.254.0.2

--Keepalive interval

: 1000 msec

--Keepalive timeout

: 5 seconds

--Keepalive hold timeout

: 3 seconds

--Keepalive vrf

: VPC_KEEPALIVE

--Keepalive udp port

: 3200

--Keepalive tos

: 192

Note that VLAN 999 does not forward over the vPC Peer Link or the vPC member
ports because VLAN 999 was removed from the trunking allowed list.
N7K1-1# show spanning-tree vlan 999

VLAN0999
Spanning tree enabled protocol rstp
Root ID

Bridge ID

Interface

Priority

33767

Address

0026.980c.2141

Cost

Port

4097 (port-channel2)

Hello Time

Priority

33767

Address

68bd.abd7.6041

Hello Time

sec

sec

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 999)

Max Age 20 sec

Role Sts Cost

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po2

Root FWD 1

128.4097 P2p

N7K1-1# show interface trunk

<snip>
-------------------------------------------------------------------------------Port

Vlans Allowed on Trunk

-------------------------------------------------------------------------------Eth1/1

1-4094

Eth1/2

1-4094

Eth2/1

1-998,1000-4094

Eth2/2

1-998,1000-4094

Eth2/3

1-998,1000-4094

Eth2/4

1-998,1000-4094

Po1

1-998,1000-4094

Po2

1-4094

Po51

1-998,1000-4094

<snip>
-------------------------------------------------------------------------------Port

Vlans in spanning tree forwarding state and not pruned

-------------------------------------------------------------------------------Eth1/1

none

Eth1/2

none

Eth2/1

none

Eth2/2

none

Eth2/3

none

Eth2/4

none

Po1

Po2

999

Po51

<snip>

Nexus Technology Labs - Virtual Port


Channels (vPC)
Active Standby NIC Teaming without vPC
Task
Configure the links connecting N5K1 and N5K2 as Port-Channel1.
This port channel should be an 802.1q trunk link and be configured as an STP
Network Port.
Configure VLAN 10 on N5K1 and N5K2.
Assign VLAN 10 as the access VLAN for all of N5K1 and N5K2's connections to
Server 1 and Server 2, and configure these as STP Edge Ports.
Configure Active/Standby NIC Teaming on Server 1 and Server 2 as follows:
Server 1 should use the link to N5K1 as primary, and the link to N5K2 as a
backup.
Server 2 should use the link to N5K2 as primary, and the link to N5K1 as a
backup.
Server 1 should use the IP address 10.0.0.1/24, and Server 2 should use
10.0.0.2/24.
When complete, ensure that Server 1 and Server 2 have IP connectivity to each
other, and that they maintain connectivity if either of the primary links of the NIC team
goes down.

Configuration
N5K1:
vlan 10
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
speed 10000
!
interface Ethernet1/1
switchport access vlan 10

spanning-tree port type edge


speed 1000
!
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3
switchport mode trunk
spanning-tree port type network
channel-group 1
!
interface Ethernet1/4
switchport mode trunk
spanning-tree port type network
channel-group 1
!
interface Ethernet1/5
switchport mode trunk
spanning-tree port type network
channel-group 1
N5K2:

vlan 10
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
speed 10000
!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3
switchport mode trunk
spanning-tree port type network
channel-group 1
!

interface Ethernet1/4
switchport mode trunk
spanning-tree port type network
channel-group 1
!
interface Ethernet1/5
switchport mode trunk
spanning-tree port type network
channel-group 1

Verification
In this design, the end servers are dual attached to separate access switches, N5K1
and N5K2. Because the 5Ks do not have vPC configured, the end host cannot form
a port channel to the switches. Instead, the NIC Teaming software of the server is
configured to use one link as primary and the other link as backup. This is then
considered an Active/Standby design, because only one link can forward at a time.
From the network point of view, there is no special configuration. The links
connecting to the servers are simply configured as normal access ports or trunk
ports, and then the load distribution is up to the end host.
In this case specifically, Server 1 and Server 2 both have Intel NICs, so the teaming
is configured under the NIC properties and then the Teaming tab to get to the Intel
Advanced Networking Services (ANS) properties.

Create the team and add the physical adapters to it, and then define the team type
as Fault Tolerance. This is what Intel ANS calls Active/Standby teaming, in which
one link of the team forwards and the other links are backup.

Go back to the NIC Team properties and choose the adapter that you want to be the
primary or secondary.

IP configuration goes under the new logical Team adapter, similar to how NX-OS or
IOS uses the logical Port-Channel interface to configure properties of the physical
members of a channel.

After connectivity is established between the servers, you can generate bulk TCP or
UDP flows by using the iPerf application. In the following output, Server 2 is listening
for a TCP connection from Server 1, and Server 1 is generating a TCP flow near its
line rate of 1Gbps. Note that the flows do not total 2Gbps, as only one adapter in the
NIC Team is actively forwarding.

From the network side, we can verify which links are being used by checking the
interface counters. Below we see that N5K1 is receiving the near 1Gbps flow in from
Server 1 on Ethernet1/1.
N5K1# show interface e1/1 - 2 | include Ethernet1|rate
Ethernet1/1
is up
30 seconds input rate 928586320 bits/sec, 76475 packets/sec
30 seconds output rate 3465280 bits/sec, 6756 packets/sec input rate 955.00 Mbps
, 78.59 Kpps; output rate 3.59 Mbps, 6.94 Kpps
Ethernet1/2 is up
30 seconds input rate 456 bits/sec, 0 packets/sec

30 seconds output rate 2024 bits/sec, 2 packets/sec


input rate 432 bps, 0 pps; output rate 1.61 Kbps, 2 pps

N5K1 then sends this flow over the trunk to N5K2, Port-Channel1.
N5K1# show interface port-channel1 | include rate
30 seconds input rate 3761048 bits/sec, 6902 packets/sec
30 seconds output rate 944914832 bits/sec, 77614 packets/sec

input rate 3.80 Mbps, 6.91 Kpps;

output rate 953.70 Mbps


, 78.28 Kpps

N5K2 then forwards the traffic to Server 2 via the local link Ethernet1/2.
N5K2# show interface e1/1 - 2 | include Ethernet1|rate
Ethernet1/1 is up
30 seconds input rate 496 bits/sec, 0 packets/sec
30 seconds output rate 2008 bits/sec, 3 packets/sec
input rate 424 bps, 0 pps; output rate 1.66 Kbps, 2 pps
Ethernet1/2 is up
30 seconds input rate 3499416 bits/sec, 6825 packets/sec
30 seconds output rate 932333608 bits/sec, 76786 packets/sec

input rate 3.55 Mbps, 6.88 Kpps;

output rate 946.85 Mbps


, 77.93 Kpps

The key point about the above output is that the standby links to the servers,
specifically E1/2 on N5K1 and E1/1 on N5K2, are essentially unused. Regardless of
the amount of flows being sent between the servers, they cannot exceed 1Gbps as
an aggregate, because only one of the physical links of the team is being used. The
only real advantage of this type of design is that there is fault tolerance for the NICs.
If N5K1s link to Server 1 goes down, as shown below, the traffic will re-route to
N5K2.
N5K1# config
Enter configuration commands, one per line.

End with CNTL/Z.

N5K1(config)# int e1/1


N5K1(config-if)# shut
2013 Mar

3 18:00:47 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/1 is down(Config change)

2013 Mar

3 18:00:47 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/1 is down (Administratively down)

N5K2# show interface e1/1 - 2 | include Ethernet1|rate


Ethernet1/1
is up
30 seconds input rate 942865872 bits/sec, 77650 packets/sec
30 seconds output rate 3521168 bits/sec, 6867 packets/sec input rate 942.86 Mbps
, 77.65 Kpps; output rate 3.52 Mbps, 6.87 Kpps Ethernet1/2
is up

30 seconds input rate 3520152 bits/sec, 6865 packets/sec


30 seconds output rate 942866792 bits/sec, 77652 packets/sec

input rate 3.52 Mbps, 6.86 Kpps;

output rate 942.87 Mbps


, 77.65 Kpps

Now that the connection from Server 1 to N5K1 is down, N5K2 receives all traffic
between the servers.

Nexus Technology Labs - Virtual Port


Channels (vPC)
Active Active NIC Teaming with vPC
Task
Configure a vPC Domain between N5K1 and N5K2 as follows:
N5K1 and N5K2 are the vPC Peers.
Create vPC Domain 1 on the peers, and use the mgmt0 ports for the vPC
Peer Keepalive Link.
Configure all links between the vPC peers as Port-Channel 1, and use this
as the vPC Peer Link.
The vPC Peer Link should use LACP negotiation, be an 802.1q trunk link,
and be an STP Network Port.
Configure vPCs from N5K1 and N5K2 to Server 1 and Server 2 as follows:
Configure N5K1 and N5K2's links to Server 1 as Port-Channel 101.
Port-Channel 101 should be configured as an access port in VLAN 10, an
STP Edge Port, and as vPC 101.
Configure N5K1 and N5K2's links to Server 2 as Port-Channel 102.
Port-Channel 102 should be configured as an access port in VLAN 10, an
STP Edge Port, and as vPC 102.
Configure Active/Active NIC Teaming on Server 1 and Server 2 as follows:
Configure a NIC Team on Server 1 using 802.3ad (LACP); both links to
N5K1 and N5K2 should be in this team, and it should use the IP address
10.0.0.1/24.
Configure a NIC Team on Server 2 using 802.3ad (LACP); both links to
N5K1 and N5K2 should be in this team, and it should use the IP address
10.0.0.2/24.
When complete, ensure that Server 1 and Server 2 have IP connectivity to each
other, and that traffic between them uses both uplinks to N5K1 and N5K2
simultaneously.

Configuration
N5K1:
feature lacp
feature vpc
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.52
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel101
switchport mode access
switchport access vlan 10
spanning-tree port type edge
vpc 101
!
interface port-channel102
switchport mode access
switchport access vlan 10
spanning-tree port type edge
vpc 102
!
interface Ethernet1/1
switchport mode access
switchport access vlan 10
channel-group 101 mode active
speed 1000
!
interface Ethernet1/2
switchport mode access
switchport access vlan 10
channel-group 102 mode active
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network

channel-group 1 mode active


N5K2:

feature lacp
feature vpc
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.51
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel101
switchport mode access
switchport access vlan 10
spanning-tree port type edge
vpc 101
!
interface port-channel102
switchport mode access
switchport access vlan 10
spanning-tree port type edge
vpc 102
!
interface Ethernet1/1
switchport mode access
switchport access vlan 10
channel-group 101 mode active
speed 1000
!
interface Ethernet1/2
switchport mode access
switchport access vlan 10
channel-group 102 mode active
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network

channel-group 1 mode active

Verification
In this design, the end servers are dual attached to separate access switches, N5K1
and N5K2. Additionally, N5K1 and N5K2 are configured for Virtual Port Channel
(vPC), which is a type of Multi-Chassis EtherChannel (MEC). vPC means that the
downstream devices, Server 1 and Server 2 in this case, see the upstream switches
(the vPC Peers) as a single switch. In other words, while the physical topology is a
triangle, the logical topology is a point-to-point port channel.
vPC configuration is made up of three main components, the vPC Peer Keepalive
Link, the vPC Peer Link, and the vPC Member Ports. The vPC Keepalive Link is any
layer 3 interface, including the mgmt0 port, that is used to send UDP pings between
the vPC peers. If the UDP ping is successful over the keepalive link, the peers are
considered to be reachable. The second portion, the vPC Peer Link, is used to
synchronize the control plane between the vPC Peers. The Peer Link is used for
operations such as MAC address table synchronization, ARP table synchronization,
IGMP Snooping synchronization, and so on. The Peer Link is a port channel made
up of at least two 10Gbps links, and it should be configured as a layer 2 trunk link
that runs as STP port type network. The final portions, the vPC member ports, are
the port channel interfaces that go down the end hosts or downstream devices.
The first step in vPC verification is to ensure that the vPC Peer Keepalive is up and
that the vPC Peer Link is up, as shown below.
N5K1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

vPC keep-alive status

Peer status
: peer is alive

Configuration consistency status: success


Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: primary

Number of vPCs configured

: 2

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

vPC Peer-link status


---------------------------------------------------------------------

: peer adjacency formed ok

id

Port

Status Active vlans

--

----

------ --------------------------------------------------

Po1

up

1,10

<snip>

Next, the vPC Member Ports are configured to the end hosts. In the output below,
Port-Channel101 to Server 1 shows its vPC as down, because the vPC has been
configured on the switch side but not yet on the server side. The end result is that
the link runs as a normal access port, as indicated by the Individual flag of the
show port-channel summary.
N5K1# show vpc 101

vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------- 101


*

success

success

Po101

N5K1# show port-channel summary


Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------1

Po1(SU)

Eth

LACP

Eth1/3(P)

101

Po101(SD)

Eth

LACP

Eth1/1(I)

102

Po102(SU)

Eth

LACP

Eth1/2(P)

Eth1/4(P)

Eth1/5(P)

Next, the end server is configured for NIC Teaming. In the case of the Intel ANS
software, an LACP-based channel is called 802.3ad Dynamic Link Aggregation.

down

After the server signals the switch with LACP, the channel can form and the vPC
comes up, as shown below.
N5K1#
2013 Mar

3 18:58:39 N5K1 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/1 is down (Initializing)

2013 Mar

3 18:58:39 N5K1 %ETH_PORT_CHANNEL-5-PORT_INDIVIDUAL_DOWN: individual port Ethernet1/1 is down

2013 Mar

3 18:58:39 N5K1 %ETHPORT-5-SPEED: Interface port-channel101, operational speed changed to 1 Gbps

2013 Mar

3 18:58:39 N5K1 %ETHPORT-5-IF_DUPLEX: Interface port-channel101, operational duplex mode changed to Full

2013 Mar

3 18:58:39 N5K1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel101, operational Receive Flow Control

2013 Mar

3 18:58:39 N5K1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel101, operational Transmit Flow Contro

2013 Mar

3 18:58:42 N5K1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel101: Ethernet1/1 is up

N5K1# 2013 Mar

3 18:58:51 N5K1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel101: first operational port changed fro

2013 Mar

3 18:58:51 N5K1 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode access

2013 Mar

3 18:58:51 N5K1 %ETHPORT-5-IF_UP: Interface port-channel101 is up in mode access

N5K1# show vpc 101

vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------- 101


success

success

Po101

up

10

The IP configuration of the server goes on the logical NIC Team interface, similar to
how NX-OS and IOS use the logical Port-Channel interface to reference the
physical members of the channel.

Testing the traffic flows over the vPC in the data plane becomes a little difficult in
this case. Each device that has a port channel configured ultimately controls the
decision of how its outbound traffic flows. For example, if a traffic flow is moving
from Server 1 to Server 2, Server 1 first determines which links to send the flows out
on, and then the upstream switches choose which outbound links to send the flows
out on, until the final destination is reached. This is an issue because you will not
see an even distribution of traffic among the NIC Team and vPC Member Ports
unless there is a sufficiently large number of flows from diverse source and
destination addresses. Although the port-channel load balancing method can be
changed on the Nexus switches, it cannot be changed in the Intel NIC drivers in this
design. Therefore, to fully verify that Active/Active forwarding is working, we need
more than one destination address to send to. This is achieved below by configuring
a secondary IP address on the NIC Team of Server 1.

Next, Server 2 is configured to send separate UDP flows to each of the addresses
on Server 1 with the iPerf app, as shown below.

On the network side, the traffic flows in the data plane can be verified by looking at
the interface counters of the vPC Member Ports. If the input bandwidth counter from
Server 2 is split between both N5K1 and N5K2, we would then know that Server 2 is
distributing the load between both members of its NIC Team in an Active/Active
manner. Furthermore, if we see that the output bandwidth counters from N5K1 and
N5K2 to Server 1 is split between them, we would also know that the switches are
doing Active/Active forwarding to the destination. This can be seen in the output
below.
N5K1# show interface e1/1-2 | in rate|Ethernet
Ethernet1/1 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed88 (bia 000d.eca2.ed88)
30 seconds input rate 946992 bits/sec, 198 packets/sec
30 seconds output rate 5899400 bits/sec, 926 packets/sec

input rate 946.99 Kbps, 198 pps;

output rate 5.90 Mbps


, 926 pps
Ethernet1/2 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed89 (bia 000d.eca2.ed89)
30 seconds input rate 5899032 bits/sec, 926 packets/sec
30 seconds output rate 947384 bits/sec, 199 packets/sec input rate 5.90 Mbps
, 926 pps; output rate 947.38 Kbps, 199 pps
N5K2# show interface e1/1-2 | in rate|Ethernet
Ethernet1/1 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca4.7408 (bia 000d.eca4.7408)
30 seconds input rate 40 bits/sec, 0 packets/sec
30 seconds output rate 6211424 bits/sec, 975 packets/sec

input rate 40 bps, 0 pps;

output rate 6.21 Mbps


, 975 pps
Ethernet1/2 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca4.7409 (bia 000d.eca4.7409)
30 seconds input rate 6211216 bits/sec, 975 packets/sec
30 seconds output rate 144 bits/sec, 0 packets/sec

input rate 6.21 Mbps


, 975 pps; output rate 144 bps, 0 pps

Note that on N5K1 the input rate of E1/2, which connects to Server 2, matches the
output rate of E1/1, which connects to Server 1. Likewise, on N5K2 the input rate of
E1/2, which connects to Server 2, matches the output rate of E1/1, which connects
to Server 1. Also note that these traffic flows do not cross the vPC Peer Link
between the Nexus 5Ks, because this link is excluded from the data plane under
normal correct operations. Verification of the counters of Port-Channel1, the vPC
Peer Link, show little to no traffic being sent or received on the port.
N5K1# show interface port-channel 1 | include rate

30 seconds input rate 944 bits/sec, 1 packets/sec


30 seconds output rate 1168 bits/sec, 1 packets/sec
input rate 976 bps, 1 pps; output rate 1.07 Kbps, 1 pps

The output shown above indicates the normal forwarding logic of vPC, which is that
the vPC Peer will first attempt to forward traffic to a local vPC Member Port instead
of crossing the vPC Peer Link. The only time that this rule is normally broken for
known unicast traffic is if the local vPC Member Port is down. For example, if a
failure occurs between N5K1 and Server 1, N5K1s traffic received from Server 1
going to Server 2 must be sent over the vPC Peer Link; otherwise it would be
blackholed. This can be seen below.

Normally this detection is immediate based on link failure, but in this topology design
Server 1 is a Virtual Machine that is not directly physically connected to N5K1.
When the LACP timer expires, N5K1 detects that the vPC peer is gone, and the
vPC Member Port goes down.
N5K1#
2013 Mar

3 22:54:34 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel101: Ethernet1/1 is down

2013 Mar

3 22:54:34 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel101: port-channel101 is down

<snip>
N5K1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id

: 1

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status: success


Per-vlan consistency status

: success

Type-2 consistency status

: success

vPC role

: primary

Number of vPCs configured

: 2

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

Graceful Consistency Check

: Enabled

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ --------------------------------------------------

Po1

up

1,10

vPC status
---------------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

------ ----------- ------ ----------- -------------------------- ----------- 101


success
102

Po102

success
up

Po101

success

success

10

Now any traffic that comes in on N5K1 from Server 2 that is going toward Server 1
must transit the vPC Peer Link.

down*

N5K1# show interface port-channel 1 | include rate

30 seconds input rate 1784 bits/sec, 1 packets/sec


30 seconds output rate 5520864 bits/sec, 862 packets/sec

input rate 992 bps, 1 pps;

output rate 5.67 Mbps


, 856 pps

This situation normally only happens during a failure event. It is highly undesirable
for vPC because the vPC Peer Link is usually much lower bandwidth (such as 20
Gbps) than the aggregate of the vPC Member Ports (such as 400Gbps+, depending
on port density), so the Peer Link can quickly become overwhelmed if it needs to be
used in the data plane.

Nexus Technology Labs - Storage Area


Network (SAN)
FCoE QoS
Task
Configure N5K1 to pair with the Fabric Extender N2K1 using FEX number 101.
Configure FCoE support for the Emulex CNA server on N5K1 as follows:
Enable the FCoE feature set on N5K1.
Create VSAN 101 on N5K1.
Configure VLAN 1101 on N5K1, and map it to the FCoE VSAN 101.
Configure N5K1's link to the Emulex CNA server as an 802.1q trunk and as
STP port type edge trunk.
Configure VLAN 10 on N5K1, which will be the server's access VLAN for
regular LAN traffic (that is, not FCoE traffic).
Configure N5K1's link to the CNA server with VLAN 10 as the native VLAN,
and limit the trunk link to only VLANs 10 and 1101.
Create interface Virtual Fibre Channel 111 on N5K1, and bind this to N5K1's
link to the CNA server.
Assign this interface VFC111 to VSAN 101, and edit the trunking allowed list
so that it only includes VSAN 101.
Modify the QoS policy on N5K1 so that the FCoE traffic to and from the
Emulex CNA server is reserved 75% of the link bandwidth, while normal
LAN traffic is reserved 25%.
When complete, verify that the Emulex CNA server is registered to the fabric and has
learned the QoS policy from the N5K.

Configuration
N5K1:

feature fex
feature fcoe
!

vlan 10
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 10,1101
spanning-tree port type edge trunk
no shutdown
!
interface vfc111
bind interface Ethernet101/1/1
switchport trunk allowed vsan 101
no shutdown
!
policy-map type queuing FCoE_75_PERCENT_POLICY
class type queuing class-fcoe
bandwidth percent 75
class type queuing class-default
bandwidth percent 25
!
system qos
service-policy type queuing input FCoE_75_PERCENT_POLICY
service-policy type queuing output FCoE_75_PERCENT_POLICY

Verification

One important behavioral difference between native Fibre Channel and Ethernet
networks is that Fibre Channel is by design a lossless protocol and Ethernet is not.
To provide guaranteed delivery, Ethernet networks rely on upper-layer protocols
such as TCP to do flow-control and retransmission. In Fibre Channel networks,
these functions are built directly into the equivalent of the layer 2 transport.
Therefore, to meet the application requirements of running SCSI reads and writes
over an Ethernet transport (FCoE), Ethernet had to be extended to allow for
guaranteed delivery.
These Ethernet extensions are grouped together under the terms Data Center
Ethernet (DCE), Data Center Bridging (DCB), or Converged Enhanced Ethernet
(CEE), depending on which vendors documentation you are reading. All of these
terms essentially mean the same thing, which in one form or another is QoS support
for FCoE traffic. The relevant standards that emerged from these protocols are
802.1Qbb Priority-based Flow Control (PFC), 802.1Qaz Enhanced Transmission
Selection (ETS) & Data Center Bridging Capabilities Exchange Protocol (DCBX),
and 802.1Qau Quantized Congestion Notification (QCN). Of these, DCBX is the
negotiation protocol that runs between the FCoE-capable switch (ithe FCF) and the
end host (the ENode), PFC provides the pause functionality so that FCoE doesnt
get dropped, ETS provides the QoS scheduling (similar to CBWFQ), and QCN
performs end-to-end flow control. Ciscos NX-OS implementation of FCoE supports
DCBX, PFC, and ETS, but not QCN.
From an implementation point of view, we must ensure that all FCFs and ENodes
have DCBX enabled, and that through DCBX they are able to negotiate PFC and
ETS. The actual signaling of the DCBX exchanges happens through the Link Layer
Discovery Protocol (LLDP), which is essentially the open standard version of CDP.
This means that the first step of FCoE support is that the Nexus switch and the end
hosts CNA become LLDP adjacent, as shown below.
N5K1# show lldp neighbors interface e101/1/1

Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID

Local Intf

Hold-time

OCe11102-FX FWVer:4.1.402.18Eth101/1/1

120

Capability
S

Port ID
0000.c9bb.199f

Total entries displayed: 1

The device ID shown above indicates that the end station has an Emulex
OCe11102-FX Converged Network Adapter (CNA), along with the particular
firmware version. This particular CNA supports the standardized FCoE negotiation
protocols, including FCoE Initialization Protocol (FIP), DCBX, PFC, and ETS. This is

important to note because some older generation 1 CNAs dont support the new
standards, which means they cannot negotiate FCoE with all switches. Specifically,
this CNAs support for these standard protocols can be verified as follows:
N5K1# show system internal dcbx info interface ethernet 101/1/1

Interface info for if_index: 0x1f640000(Eth101/1/1)

Interface info for if_index: 0x1f640000(Eth1/1)


tx_enabled: TRUE
rx_enabled: TRUE dcbx_enabled: TRUE

DCX Protocol: CEE


DCX CEE NIV extension: disabled
<snip>

Decoding of the output of this command is not very intuitive; it assumes you know
the specific Type Length Values (TLVs) that LLDP and DCBX use to negotiate the
PFC and ETS features. If for some reason negotiation fails between the FCF and
the ENode, and you need to pinpoint the exact reason, visit the Troubleshooting
FCoE Issues section of the Nexus 5K documentation at
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/n5K_ts_
. For example, in this case we would want to know that the CNA has the PFC
feature on, and has agreed to mark the FCoE traffic with the particular Class of
Service (CoS) value that the switch has requested. This is verified from the switchs
side per the output below.
N5K1# show system internal dcbx info interface ethernet 101/1/1 | include TLV|willing
LLDP TLV's
LLDP TLV type:Chassis ID
LLDP TLV type:Port ID

LLDP TLV Length: 7

LLDP TLV Length: 7

LLDP TLV type:Time to Live

LLDP TLV Length: 2

LLDP TLV type:Port Description


LLDP TLV type:System Name

LLDP TLV Length: 8

LLDP TLV Length: 28

LLDP TLV type:System Description

LLDP TLV Length: 45

LLDP TLV type:System capabilities

LLDP TLV Length: 4

LLDP TLV type:LLDP Organizationally Specific


LLDP TLV type:END of LLDP PDU

LLDP TLV Length: 55

LLDP TLV Length: 0

Peer's DCX TLV:


DCBX TLV Proto(1) type: 1(Control) DCBX TLV Length: 10 DCBX TLV Value
sub_type 0, error 0, willing 0, enable 0, max_version 0, oper_version 0
DCBX TLV Proto(1) type: 2(PriGrp)
DCBX TLV Length: 17 DCBX TLV Value

sub_type 0, error 0, willing 1, enable 1


, max_version 0, oper_version 0 DCBX TLV Proto(1) type: 3(PFC)
DCBX TLV Length: 6 DCBX TLV Value

sub_type 0, error 0, willing 1, enable 1

, max_version 0, oper_version 0
DCBX TLV Proto(1) type: 4(App) DCBX TLV Length: 10 DCBX TLV Value
sub_type 0, error 0, willing 1, enable 1, max_version 0, oper_version 0
Feature Register Params: max_version 0, enable 1, willing 0 advertise 1
Feature Register Params: max_version 0, enable 1, willing 0 advertise 1
Feature Register Params: max_version 0, enable 1, willing 0 advertise 1
Feature Register Params: max_version 0, enable 1, willing 0 advertise 1
Feature Register Params: max_version 0, enable 1, willing 0 advertise 1
Feature Register Params: max_version 0, enable 1, willing 0 advertise 1
Feature Register Params: max_version 0, enable 1, willing 0 advertise 1
Feature Register Params: max_version 0, enable 1, willing 0 advertise 1
lldp feature register params max_version 0, enable 1, willing 0 advertise 1, disruptive_e
rror 0 mts_addr_node 0x2201 mts_addr_sap 0x179
lldp feature register params max_version 0, enable 1, willing 0 advertise 1, disruptive_e
rror 0 mts_addr_node 0x2201 mts_addr_sap 0xaf
Total TLVs unrecognized: 0

In reality, this negotiation is much more easily verified from the end station itself,
because the CNAs management application, such as the Emulex OneCommand
Manager, will simply tell you whether negotiation was successful. In the output
shown below, we see that FCoE negotiation was successful and the switch has
informed the CNA to do a 50/50 split between LAN and SAN traffic. SAN traffic is
reserved the Class of Service (CoS) 3, whereas LAN traffic gets the other CoS
values 0, 1, 2, 4, 5, 6, and 7.

Note that on the first-generation Nexus 5000s (5010 and 5020), this QoS policy is
enabled by default. However, in the second-generation Nexus 5500s (5548 and
5596), this policy is not on by default and must be manually enabled. On a Nexus

5500, this can be verified as follows:


Nexus-5548UP# show run all | section "system qos"
system qos
service-policy type queuing input default-in-policy
service-policy type queuing output default-out-policy
service-policy type qos input default-in-policy service-policy type network-qos default-nq-policy
Nexus-5548UP# show policy-map type network-qos

Type network-qos policy-maps


===============================

policy-map type network-qos default-nq-policy


class type network-qos class-default

mtu 1500

multicast-optimize

policy-map type network-qos fcoe-default-nq-policy


class type network-qos class-fcoe

pause no-drop
mtu 2158
class type network-qos class-default

mtu 1500
multicast-optimize

Note that there is a default QoS policy that would support FCoE, the fcoe-default-nqpolicy, but it is not the one that is enabled. In the case of a Nexus 5000, this policy
differed as follows:

Nexus-5020# show run all | section "system qos"

system qos
service-policy type queuing input default-in-policy
service-policy type queuing output default-out-policy
service-policy type qos input default-in-policy service-policy type network-qos default-nq-policy

Nexus-5020# show policy-map type network-qos

Type network-qos policy-maps


===============================

policy-map type network-qos default-nq-policy


class type network-qos class-fcoe

pause no-drop
mtu 2158
class type network-qos class-default
mtu 1500

The specific relevant output above is the fact that there is a pause no-drop class,
which is enabling Priority Flow Control (PFC) for the FCoE class. The actual
reservation of SAN vs. LAN traffic, however, does not use this network-qos policy,
but is instead controlled by the queuing policy. The default queuing policies can be
seen below.
N5K1# show policy-map type queuing

Type queuing policy-maps


========================

policy-map type queuing default-in-policy


class type queuing class-fcoe
bandwidth percent 50
class type queuing class-default

bandwidth percent 50

policy-map type queuing default-out-policy


class type queuing class-fcoe
bandwidth percent 50
class type queuing class-default

bandwidth percent 50

These separate input and output policies both call FCoE and default classes, and
assign them 50% of the bandwidth. The matches in these classes can be verified as
follows:
N5K1# show class-map type queuing

Type queuing class-maps


=======================

class-map type queuing class-fcoe

match qos-group 1

class-map type queuing class-default


match qos-group 0

class-map type queuing class-all-flood


match qos-group 2

class-map type queuing class-ip-multicast

The queuing class-map class-fcoe is matching qos-group 1, and then in turn is


guaranteed 50% bandwidth from the queuing policy-map. The qos-group, however,
is only a temporary internal placeholder, not an actual marking in the packet. Where
this qos-group 1 comes from is as follows:
N5K1# show class-map type qos

Type qos class-maps


===================

class-map type qos match-any class-fcoe

match cos 3

class-map type qos match-any class-default


match any

class-map type qos match-any class-all-flood


match all flood

class-map type qos match-any class-ip-multicast


match ip multicast

N5K1# show policy-map type qos

Type qos policy-maps


====================

policy-map type qos default-in-policy


class type qos class-fcoe

set qos-group 1

class type qos class-default


set qos-group 0

These qos type class-maps and policy-maps are where the actual classification of
FCoE occurs. The workflow of the QoS policy is then as follows:
Match cos 3 with the qos type class-map called class-fcoe.
Set the qos-group to 1 from the qos type policy-map called default-in-policy .
Match qos-group 1 with the queuing type class-map called class-fcoe.
Perform the bandwidth reservation on qos-group 1 with the queuing type policy-maps
called default-in-policy and default-out-policy.
You may ask, Why is this workflow so important? It is because you must ensure that
only your FCoE traffic lands in these particular classes; otherwise, other LAN traffic
types could potentially break the lossless behavior that we are trying to achieve for
FCoE. With the default built-in policies (assuming they are enabled on Nexus 5500),
this behavior will be achieved, but if you are modifying the policy for other LAN
traffic types, you must be careful not to accidentally mark anything as CoS 3 or qosgroup 1, because these land in the FCoE classes.
To meet the requirements of this specific task, if we want to change how much
bandwidth is allocated to the SAN traffic vs. the LAN traffic, we simply need to
create a new queuing policy-map that calls the already in-place FCoE queuing classmaps. In our case, this new policy is defined as follows:
N5K1# show policy-map system type queuing

Service-policy (queuing) input


:

FCoE_75_PERCENT_POLICY
policy statistics status:
Class-map (queuing):

disabled

class-fcoe

(match-any)
Match: qos-group 1

Class-map (queuing):

bandwidth percent 75

class-default (match-any)

Match: qos-group 0
bandwidth percent 25
Service-policy (queuing) output
:

FCoE_75_PERCENT_POLICY
policy statistics status:
Class-map (queuing):

disabled

class-fcoe

(match-any)
Match: qos-group 1

bandwidth percent 75

Class-map (queuing):

class-default (match-any)

Match: qos-group 0
bandwidth percent 25

From the end hosts CNA point of view, these new values should be learned through
the DCBX negotiation, and then enforced in the data plane as the QoS policy of the
NIC itself. The changes can be seen below: Now the CoS 3 traffic is guaranteed
75%.

Nexus Technology Labs - Storage Area


Network (SAN)
Fibre Channel Zoning
Task
Create VSAN 101 on MDS3 and N5K1, and VSAN 102 on MDS4 and N5K2.
Configure MDS3 and MDS4's links to the SAN as 2Gbps F_Ports in VSANs 101 and
102, respectively.
Configure trunking on the links between MDS3 and N5K1 and MDS4 and N5K2 as
2Gbps TE_Ports for VSANs 101 and 102, respectively.
Configure N5K1 to pair with the Fabric Extender N2K1 using FEX number 101.
Configure N5K2 to pair with the Fabric Extender N2K2 using FEX number 102.
Configure FCoE support for the Emulex CNA Server on N5K1 and N5K2 as follows:
Enable the FCoE feature set on N5K1 and N5K2.
Configure VLAN 1101 on N5K1, and map it to the FCoE VSAN 101.
Configure VLAN 1102 on N5K2, and map it to the FCoE VSAN 102.
Configure N5K1 and N5K2's links to the Emulex CNA server as 802.1q
trunks and as STP port type edge trunk.
Configure VLAN 10 on N5K1 and N5K2, which will be the server's access
VLAN for regular LAN traffic (that is, not FCoE traffic).
Configure N5K1's link to the CNA Server with VLAN 10 as the native VLAN,
and limit the trunk link to only VLANs 10 and 1101.
Configure N5K2's link to the CNA Server with VLAN 10 as the native VLAN,
and limit the trunk link to only VLANs 10 and 1102.
Create interface Virtual Fibre Channel 111 on N5K1, and bind this to N5K1's
link to the CNA server.
Assign this interface VFC111 to VSAN 101, and edit the trunking allowed list
so that it only includes VSAN 101.
Create interface Virtual Fibre Channel 211 on N5K2, and bind this to N5K2's
link to the CNA server.
Assign this interface VFC211 to VSAN 102, and edit the trunking allowed list
so that it only includes VSAN 102.

Configure zoning in the Fibre Channel fabric for VSAN 101 as follows:
Configure enhanced zoning for VSAN 101 on MDS3 and N5K1.
Configure a zoneset for VSAN 101 named SAN_A_ZONESET.
Create a zone in this zoneset named EMULEX_TO_NTFS1_SAN_A.
Add the pWWNs of the Emulex CNA attached to N5K1 and the SAN
attached to MDS3's port FC1/9 to the zone.
Activate the zoneset.
Commit the VSAN 101 enhanced zoning session.
Configure zoning in the Fibre Channel fabric for VSAN 102 as follows:
Configure enhanced zoning for VSAN 102 on MDS4 and N5K2.
Configure a zoneset for VSAN 102 named SAN_B_ZONESET.
Create a zone in this zoneset named EMULEX_TO_NTFS1_SAN_B.
Add the pWWNs of the Emulex CNA attached to N5K2 and the SAN
attached to MDS4's port FC1/9 to the zone.
Activate the zoneset.
Commit the VSAN 102 enhanced zoning session.
When complete, the Emulex CNA server should be able to mount only one volume,
NTFS1, out both SAN A and SAN B.

Configuration
N5K1:
feature fex
feature fcoe
!
vlan 10
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 101
no shutdown
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101

!
interface Ethernet101/1/1
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 10,1101
spanning-tree port type edge trunk
no shutdown
!
interface vfc111
bind interface Ethernet101/1/1
switchport trunk allowed vsan 101
no shutdown
!
vsan database
vsan 101 interface vfc111
!
zone name EMULEX_TO_NTFS1_SAN_A vsan 101
member pwwn 21:00:00:1b:32:07:32:23
member pwwn 10:00:00:00:c9:bb:19:9f
!
zoneset name SAN_A_ZONESET vsan 101
member EMULEX_TO_NTFS1_SAN_A
!
zoneset activate name SAN_A_ZONESET vsan 101
!
zone commit vsan 101
N5K2:
feature fex
feature fcoe
!
vlan 10
vlan 1102
fcoe vsan 102
!
vsan database
vsan 102
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 102
no shutdown
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102

!
interface Ethernet102/1/1
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 10,1102
spanning-tree port type edge trunk
no shutdown
!
interface vfc211
bind interface Ethernet102/1/1
switchport trunk allowed vsan 102
no shutdown
!
vsan database
vsan 102 interface vfc211
!
zone default-zone permit vsan 102
MDS3:
interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
vsan database
vsan 101
vsan 101 interface fc1/9
vsan 101 interface fc1/10
!
interface fc1/1 - 2
switchport speed 2000
switchport rate-mode dedicated
switchport mode E
switchport trunk allowed vsan 101
no shutdown
!
zone name EMULEX_TO_NTFS1_SAN_A vsan 101
member pwwn 21:00:00:1b:32:07:32:23
member pwwn 10:00:00:00:c9:bb:19:9f
!
zoneset name SAN_A_ZONESET vsan 101
member EMULEX_TO_NTFS1_SAN_A
!
zoneset activate name SAN_A_ZONESET vsan 101
!
zone commit vsan 101
MDS4:

interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
vsan database
vsan 102
vsan 102 interface fc1/9
vsan 102 interface fc1/10
!
interface fc1/1 - 2
switchport speed 2000
switchport rate-mode dedicated
switchport mode E
switchport trunk allowed vsan 102
no shutdown
!
zone name EMULEX_TO_NTFS1_SAN_B vsan 102
member pwwn 21:02:00:1b:32:47:32:23
member pwwn 10:00:00:00:c9:bb:19:a3
!
zoneset name SAN_B_ZONESET vsan 102
member EMULEX_TO_NTFS1_SAN_B
!
zoneset activate name SAN_B_ZONESET vsan 102
zone commit vsan 102

Verification
Zoning in Fibre Channel is analogous to Access Lists in the IP world, meaning that
zoning is used to control which devices can talk to each other through the Fibre
Channel Fabric. Specifically, zoning is used to control which FC Initiators can talk to
which FC Targets. Zoning helps to prevent problems with file locking if multiple
hosts are mounting the same volumes, or data corruption if hosts are mounting the
wrong types of volumes. For example, if a Windows host accidentally mounts a
volume formatted as VMFS for ESXi hosts, it might corrupt the datastore, or if an
ESXi host mounts a NTFS formatted volume, it would be corrupted.
A very important point to remember about zoning is that it is required. It is not an
optional step in the configuration of the Fibre Channel Fabric. MDS and Nexus
switches run what is known as hard zoning, which means that the zoneset is
enforced not only in the control plane during the Fabric Discovery process of an
initiator, but also in the data plane on all switches in the fabric. In the Ethernet world,

this would be analogous to having a VLAN ACL (VACL or VLAN Access Map)
configured on every single switch that knows about the VLAN in question. This also
means that for zoning to work properly, all switches in the fabric must agree on the
zoneset configuration. The easiest and most commonly implemented way of having
the switches agree on the zoning configuration is to use what is known as enhanced
zoning.
Enhanced zoning allows zoning configuration changes to be made on one device in
the fabric, and then have these changes dynamically advertised to other switches in
the fabric. Additionally, while a zoning configuration change is in progress, a lock is
put on the fabric to ensure that someone else doesnt make a separate zoning
change that would conflict. Assuming that the fabric is already end-to-end (VSANs
are created, Trunking Expansion ports are active, etc.), the zoning mode can be
changed from basic to enhanced on one switch in the fabric, and then advertised to
the other switches. In the output below, we see that MDS3 sets the zoning mode to
enhanced for VSAN 101, and then N5K1 learns about this change. Likewise, N5K2
sets VSAN 102 to enhanced zoning mode, and MDS4 learns about it.
MDS3# conf t
Enter configuration commands, one per line.

End with CNTL/Z. MDS3(config)# zone mode enhanced vsan 101

WARNING: This command would distribute the zoning database of this switch throughout the fabric. Do you want to cont
Set zoning mode command initiated. Check zone status
N5K1# show zone status vsan 101
VSAN: 101 default-zone: deny distribute: full Interop: default mode: enhanced
merge-control: allow
session: none
hard-zoning: enabled broadcast: enabled
Default zone:
qos: none broadcast: disabled ronly: unsupported
Full Zoning Database :
DB size: 44 bytes
Zonesets:0

Zones:0 Aliases: 0 Attribute-groups: 1

Active Zoning Database :


Database Not Available Status: Set zoning mode success
at 19:47:54 UTC Mar 20 2013
N5K2# conf t
Enter configuration commands, one per line.

End with CNTL/Z. N5K2(config)# zone mode enhanced vsan 102

WARNING: This command would distribute the zoning database of this switch throughout the fabric. Do you want to cont
Set zoning mode command initiated. Check zone status
MDS4# show zone status vsan 102
VSAN: 102 default-zone: deny distribute: active only Interop: default mode: enhanced
merge-control: allow
session: none
hard-zoning: enabled broadcast: enabled
Default zone:

qos: none broadcast: disabled ronly: disabled


Full Zoning Database :
DB size: 44 bytes
Zonesets:0

Zones:0 Aliases: 0 Attribute-groups: 1

Active Zoning Database :


Database Not Available Status: Set zoning mode success
at 02:24:13 UTC Jan 13 1982

When all devices in the fabric agree that the zoning mode is enhanced, any of the
switches can start an enhanced zone session. This is where a lock is advertised
through the fabric to other switches, zoning configuration changes are made, the
changes are committed and advertised to other switches in the fabric, and finally the
lock is released. Below we see MDS3 making zoning changes and then N5K1
learning about them after the changes are committed.
MDS3# conf t
Enter configuration commands, one per line.

End with CNTL/Z. MDS3(config)#

zoneset name SAN_A_ZONESET vsan 101


Enhanced zone session has been created. Please 'commit' the changes when done.
MDS3(config-zoneset)# zone name EMULEX_TO_NTFS1_SAN_A
MDS3(config-zoneset-zone)# member pwwn 21:00:00:1b:32:07:32:23
MDS3(config-zoneset-zone)# member pwwn 10:00:00:00:c9:bb:19:9f
MDS3(config-zoneset-zone)# exit
MDS3(config-zoneset)# exit MDS3(config)# zoneset activate name SAN_A_ZONESET vsan 101
MDS3(config)# zone commit vsan 101
Commit operation initiated. Check zone status
N5K1# show zoneset active vsan 101
zoneset name SAN_A_ZONESET vsan 101
zone name EMULEX_TO_NTFS1_SAN_A vsan 101
* fcid 0x630000 [pwwn 21:00:00:1b:32:07:32:23]
N5K1# show run zone

!Command: show running-config zone


!Time: Wed Mar 20 20:06:40 2013

version 5.1(3)N1(1a)
zone mode enhanced vsan 101
!Active Zone Database Section for vsan 101
zone name EMULEX_TO_NTFS1_SAN_A vsan 101
member pwwn 21:00:00:1b:32:07:32:23
member pwwn 10:00:00:00:c9:bb:19:9f

zoneset name SAN_A_ZONESET vsan 101


member EMULEX_TO_NTFS1_SAN_A

* fcid 0xef0000 [pwwn 10:00:00:00:c9:bb:19:9f]

zoneset activate name SAN_A_ZONESET vsan 101


do clear zone database vsan 101
!Full Zone Database Section for vsan 101
zone name EMULEX_TO_NTFS1_SAN_A vsan 101
member pwwn 21:00:00:1b:32:07:32:23
member pwwn 10:00:00:00:c9:bb:19:9f

zoneset name SAN_A_ZONESET vsan 101


member EMULEX_TO_NTFS1_SAN_A

zone commit vsan 101

At any given time if the zoning database is synchronized (which it should be), all
devices that are registered to the fabric for a particular VSAN should see the same
active zoneset. The active zoneset is the one that is actually being applied as the
filter in hardware, whereas the full zoneset is the one in the local configuration. In
the output below, we see that the VSAN 101 devices and the VSAN 102 devices
agree on the active zonesets for those particular VSANs.
MDS3# show zone active vsan 101
zone name EMULEX_TO_NTFS1_SAN_A vsan 101
* fcid 0x630000 [pwwn 21:00:00:1b:32:07:32:23]
* fcid 0xef0000 [pwwn 10:00:00:00:c9:bb:19:9f]
N5K1# show zone active vsan 101
zone name EMULEX_TO_NTFS1_SAN_A vsan 101
* fcid 0x630000 [pwwn 21:00:00:1b:32:07:32:23]
* fcid 0xef0000 [pwwn 10:00:00:00:c9:bb:19:9f]
MDS4# show zone active vsan 102
zone name EMULEX_TO_NTFS1_SAN_B vsan 102
* fcid 0xef0000 [pwwn 21:02:00:1b:32:47:32:23]
* fcid 0x280000 [pwwn 10:00:00:00:c9:bb:19:a3]
N5K2# show zone active vsan 102

zone name EMULEX_TO_NTFS1_SAN_B vsan 102


* fcid 0xef0000 [pwwn 21:02:00:1b:32:47:32:23]
* fcid 0x280000 [pwwn 10:00:00:00:c9:bb:19:a3]

Whenever an initiator registers to the fabric, a Fabric Login (FLOGI) and Port Login
(PLOGI) are performed. Through these login processes, the fabric learns about the
initiators pWWN, assigns them an FCID, and advertises to them with Registered
State Change Notifications (RSCNs) their zoning configuration. In other words, an
initiator will be able to discover only the targets in the fabric that zoning permits. In

our particular case, this means that the Emulex CNA server can mount the NTFS1
volume, but not the NTFS2 volume, because it is not zoned to permit that particular
pWWN. The final verification of this can be seen in the Windows Disk Management
utility below.

Note that although the LUN for the NTFS1 volume exists on both SAN A and SAN
B, separate VSAN numbers and pWWNs are used on the SAN A side vs. the SAN B
side. This means that separate zonesets are needed on A vs. B with the appropriate
numbers. If this configuration is successful, the end host should be able to use the
LUN on both SAN A and SAN B, and as long as one of these sides is reachable,
read/writes to the volume will be successful.
To verify this, the server is configured to write to the volume with the Disk Speed
Test application.

Interface counters on both N5K1 and N5K2 indicate that the server is using
Multipath IO (MPIO) to load balance SCSI read and write traffic to both fabrics.
MPIO is preconfigured on the server and should work as long as the same volume

is seen on both SAN A and SAN B by Windows Disco Management.


N5K1# show interface vfc111 | include rate
1 minute input rate 283775808 bits/sec, 35471976 bytes/sec, 16610 frames/sec
1 minute output rate 209903280 bits/sec, 26237910 bytes/sec, 12577 frames/sec
N5K2# show interface vfc211 | include rate

1 minute input rate 264992248 bits/sec, 33124031 bytes/sec, 15774 frames/sec


1 minute output rate 257687296 bits/sec, 32210912 bytes/sec, 15404 frames/sec

Next, a failure of a link to the server is simulated by shutting a port down on the SAN
A side. Note that the rest of the FC Fabric is informed about this with a Fabric
Logout message.
N5K1# config t
Enter configuration commands, one per line.

End with CNTL/Z. N5K1(config)# int e101/1/1

N5K1(config-if)# shutdown
2013 Mar 20 21:02:45 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet101/1/1 is down(Config change)
2013 Mar 20 21:02:45 N5K1

%FLOGI-5-MSG_PORT_LOGGED_OUT: %$VSAN 101%$ [VSAN 101, Interface vfc111: mode[TF]] Nx Port 10:00:00:00:c9:bb:19:9f lo

2013 Mar 20 21:02:45 N5K1 %LLDP-FEX101-5-SERVER_REMOVED: Server with Chassis ID 0000.c9bb.199f Port ID 0000.c9bb.199
2013 Mar 20 21:02:45 N5K1 %PORT-5-IF_DOWN_NONE: %$VSAN 101%$ Interface vfc111 is down (None)
N5K1(config-if)# 2013 Mar 20 21:02:45 N5K1 %PORT-5-IF_DOWN_NONE: %$VSAN 101%$ Interface vfc111 is down (None)
2013 Mar 20 21:02:45 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet101/1/1 is down (Administratively down)

All traffic from the server is now re-routed to SAN B. This can be verified below, as
the I/O rate of N5K2s link to the CNA is effectively double that of before:
N5K1(config-if)# show interface vfc111 | include rate
1 minute input rate 0 bits/sec
, 0 bytes/sec, 0 frames/sec 1 minute output rate 0 bits/sec
, 0 bytes/sec, 0 frames/sec
N5K2# show interface vfc211 | include rate
1 minute input rate 544032640 bits/sec
, 68004080 bytes/sec, 31427 frames/sec 1 minute output rate 413997312 bits/sec
, 51749664 bytes/sec, 24797 frames/sec

Nexus Technology Labs - Storage Area


Network (SAN)
Fibre Channel over Ethernet (FCoE)
Task
Create VSAN 101 on MDS3 and N5K1, and VSAN 102 on MDS4 and N5K2.
Configure MDS3 and MDS4's links to the SAN as 2Gbps F_Ports in VSANs 101 and
102, respectively.
Configure trunking on the links between MDS3 and N5K1, and MDS4 and N5K2, as
2Gbps TE_Ports for VSANs 101 and 102, respectively.
Configure N5K1 to pair with the Fabric Extender N2K1 using FEX number 101.
Configure N5K2 to pair with the Fabric Extender N2K2 using FEX number 102.
Configure FCoE support for the Emulex CNA server on N5K1 and N5K2 as follows:
Enable the FCoE feature set on N5K1 and N5K2.
Configure VLAN 1101 on N5K1, and map it to the FCoE VSAN 101.
Configure VLAN 1102 on N5K2, and map it to the FCoE VSAN 102.
Configure N5K1 and N5K2's links to the Emulex CNA server as 802.1q
trunks and as STP port type edge trunk.
Configure VLAN 10 on N5K1 and N5K2, which will be the server's access
VLAN for regular LAN traffic (that is, not FCoE traffic).
Configure N5K1's link to the CNA server with VLAN 10 as the native VLAN,
and limit the trunk link to only VLANs 10 and 1101.
Configure N5K2's link to the CNA server with VLAN 10 as the native VLAN,
and limit the trunk link to only VLANs 10 and 1102.
Create interface Virtual Fibre Channel 111 on N5K1, and bind this to N5K1's
link to the CNA Server.
Assign this interface VFC111 to VSAN 101, and edit the trunking allowed list
so that it only includes VSAN 101.
Create interface Virtual Fibre Channel 211 on N5K2, and bind this to N5K2's
link to the CNA server.
Assign this interface VFC211 to VSAN 102, and edit the trunking allowed list
so that it only includes VSAN 102.

Configure N5K1 and MDS3 so that the default zoning policy for VSAN 101 is permit.
Configure N5K2 and MDS4 so that the default zoning policy for VSAN 101 is permit.
When complete, the Emulex CNA server should be able to mount two volumes,
NTFS1 and NTFS2, out both SAN A and SAN B.

Configuration
N5K1:
feature fex
feature fcoe
!
vlan 10
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 101
no shutdown
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 10,1101
spanning-tree port type edge trunk
no shutdown
!
interface vfc111
bind interface Ethernet101/1/1
switchport trunk allowed vsan 101
no shutdown
!
vsan database
vsan 101 interface vfc111
!
zone default-zone permit vsan 101

N5K2:

feature fex
feature fcoe
!
vlan 10
vlan 1102
fcoe vsan 102
!
vsan database
vsan 102
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 102
no shutdown
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102
!
interface Ethernet102/1/1
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 10,1102
spanning-tree port type edge trunk
no shutdown
!
interface vfc211
bind interface Ethernet102/1/1
switchport trunk allowed vsan 102
no shutdown
!
vsan database
vsan 102 interface vfc211
!
zone default-zone permit vsan 102
MDS3:
interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
vsan database
vsan 101

vsan 101 interface fc1/9


vsan 101 interface fc1/10
!
interface fc1/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 101
no shutdown
!
zone default-zone permit vsan 101
MDS4:

interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
vsan database
vsan 102
vsan 102 interface fc1/9
vsan 102 interface fc1/10
!
interface fc1/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 102
no shutdown
!
zone default-zone permit vsan 102

Verification
Similar to how a physical FC port on an MDS or Nexus switch is used to reference a
server that is physically attached with a regular Fibre Channel HBA, a Virtual Fibre
Channel (VFC) interface is used to reference FCoE attached hosts. Each VFC is
bound to one underlying physical link, or potentially a port channel in the case of
Virtual Trunking Expansion Ports (VTE_Ports) for multi-hop FCoE. In the output
below, we see the interface VFC111 that is bound to N5K1s physical Ethernet link
to the Emulex CNA Server.
N5K1# show interface vfc111
vfc111 is trunking
Bound interface is Ethernet101/1/1

Hardware is Ethernet
Port WWN is 20:6e:00:0d:ec:a2:ed:bf Admin port mode is F, trunk mode is on
snmp link state traps are enabled Port mode is TF
Port vsan is 101
Trunk vsans (admin allowed and active) (101)
Trunk vsans (up)

(101)

Trunk vsans (isolated)

()

Trunk vsans (initializing)

()

1 minute input rate 64 bits/sec, 8 bytes/sec, 0 frames/sec


1 minute output rate 240 bits/sec, 30 bytes/sec, 0 frames/sec
72491639 frames input, 151963572580 bytes
0 discards, 0 errors
63800061 frames output, 132994718596 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
Interface last changed at Sat Mar

9 16:10:22 2013

The VFC interface above is configured as a Virtual Fabric Port (VF_Port), or more
specifically in this case, a Virtual Trunking Fabric Port (VTF_Port) because trunking
is on. Similar to how a Fabric Port (F_Port) on the switch side connects to a Node
Port (N_Port) on the end host side (either the FC Target or FC Initiator), the VF_Port
connects to a Virtual Node Port (VN_Port). The VN_Port is the Converged Network
Adapter (CNA) of the end server, or what is sometimes also called the Ethernet
Node (ENode).
The significance of this output is that the VSAN is up on the VFC port. If the VSAN
is stuck in either initializing or isolated mode, this normally indicates some sort of
negotiation error with the end host. For example, if the FCoE Forwarder (FCF),
which is the switch, and the ENode, which is the end host, did not properly negotiate
Data Center Bridging Exchange (DCBX), the VSAN would not be up.

There are several hardware and software reasons that this could occur, which are
outside the scope of this scenario. For more information on problems that cause the
VFC to fail, and their resolution, see Troubleshooting FCoE Issues (
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/n5K_ts_
) in the Nexus 5000 documentation.
In the output below, the vfc interface appears just like any other Fibre Channel
interface, but it doesnt tell us which physical Ethernet port it is actually bound to.
N5K1# show interface fc2/1 - 2 brief

------------------------------------------------------------------------------Interface

Vsan

Admin

Admin

Mode

Trunk

Status

SFP

Oper

Oper

Port

Mode

Speed

Channel

Mode

(Gbps)

------------------------------------------------------------------------------fc2/1

on

trunking

swl

TE

--

fc2/2

on

trunking

swl

TE

--

N5K1# show interface vfc111 brief

------------------------------------------------------------------------------Interface

Vsan

Admin

Admin

Mode

Trunk

Status

SFP

Oper

Oper

Port

Mode

Speed

Channel

Mode

(Gbps)

------------------------------------------------------------------------------vfc111

101

on

trunking

--

TF

auto --

During the FCoE Initialization Protocol (FIP) exchange between the FCoE
Forwarder (FCF) and the ENode, the ENode is assigned a Fabric Provided MAC
Address (FPMA) as well as a regular Fibre Channel Identifier (FCID). The FCID can
be verified through either of the below outputs.
N5K1# show fcoe database

------------------------------------------------------------------------------INTERFACE

FCID

PORT NAME

MAC ADDRESS

------------------------------------------------------------------------------- vfc111

0xef0000

10:00:00:00:c9:bb:19:9f 00:00:c9:bb:19:9f

Total number of flogi count from FCoE devices = 1.


N5K1# show flogi database
-------------------------------------------------------------------------------INTERFACE

VSAN

FCID

PORT NAME

NODE NAME

-------------------------------------------------------------------------------vfc111

101

0xef0000

10:00:00:00:c9:bb:19:9f 20:00:00:00:c9:bb:19:9f

The FPMA of the host is the combination of the FC-MAP of the FCF, and the FCID
of the Enode. Below we see that N5K1s FC-MAP is 0e:fc:00, and from above we
see that the ENode has the FCID 0xef0000. This means that their MAC address for
FCoE data plane traffic is 0e:fc:00:ef:00:00.
N5K1# show fcoe
Global FCF details
FCF-MAC is 00:0d:ec:a2:ed:80 FC-MAP is 0e:fc:00

FCF Priority is 128

FKA Advertisement period for FCF is 8 seconds

VFC MAC details


vfc111 FCF-MAC is 00:0d:ec:a2:ed:80

Now that the ENode has been verified as logged in to the fabric, we want to make
sure that the SAN itself is logged in, that the N5K can see the SAN in the fabric, and
that the MDS can see the ENode in the fabric. All of these can be verified by viewing
the FCNS database, as shown below.
N5K1# show fcns database

VSAN 101:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0x630000

21:00:00:1b:32:07:32:23 (Qlogic)

scsi-fcp:target

0x630100

21:01:00:1b:32:27:32:23 (Qlogic)

scsi-fcp:target

0xef0000

10:00:00:00:c9:bb:19:9f (Emulex)

ipfc scsi-fcp:init

Total number of entries = 3


MDS3# show fcns database

VSAN 101:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0x630000

21:00:00:1b:32:07:32:23

scsi-fcp:target

0x630100

21:01:00:1b:32:27:32:23

scsi-fcp:target

0xef0000

10:00:00:00:c9:bb:19:9f (Emulex)

ipfc scsi-fcp:init

Total number of entries = 3

In VSAN 102 on the SAN B side, a similar output would be seen, but with different
pWWNs and FCIDs, as the FC domain services, FCNS, FLOGI, etc. happens
separately on a per-VSAN basis.
N5K2# show fcns database

VSAN 102:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

--------------------------------------------------------------------------

0x280000

10:00:00:00:c9:bb:19:a3 (Emulex)

ipfc scsi-fcp:init

0xef0000

21:02:00:1b:32:47:32:23 (Qlogic)

scsi-fcp:target

0xef0100

21:03:00:1b:32:67:32:23 (Qlogic)

scsi-fcp:target

Total number of entries = 3


MDS4# show fcns database

VSAN 102:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0x280000

10:00:00:00:c9:bb:19:a3 (Emulex)

ipfc scsi-fcp:init

0xef0000

21:02:00:1b:32:47:32:23

scsi-fcp:target

0xef0100

21:03:00:1b:32:67:32:23

scsi-fcp:target

Total number of entries = 3

Another important point about FCoE is that the switches must ensure a lossless
fabric for the ENodes. In other words, the FCoE switches need a QoS policy to
make sure that FCoE isnt dropped. On the Nexus 5000s, this policy is on by
default, as shown below.
N5K1# show policy-map system type network-qos

Type network-qos policy-maps


===============================

policy-map type network-qos default-nq-policy


class type network-qos class-fcoe
match qos-group 1

pause no-drop

mtu 2158

class type network-qos class-default


match qos-group 0

mtu 1500

On the Nexus 5500 switches, this QoS policy is not enabled by default and must be
configured before the FIP negotiation is successful. Configuring and modifying this
policy is covered in a later scenario.
One more step on the network side now remains before the end hosts can mount
their LUNs on the SAN. Fibre Channel Zoning must be configured so that the N5Ks
and MDSes allow traffic in the data plane between the hosts and their LUNs. Zoning
is analogous to an Access List in the IP world, and the default zoning behavior is to
deny all traffic. For the purpose of simplicity in this scenario, the zoning policy was
changed to explicit permit, which can be seen below. In a real design, zoning would
be manually configured to associate which servers can map which storage. Zoning
techniques are covered in a separate scenario.
N5K1# show zone status vsan 101
VSAN: 101 default-zone: permit
distribute: active only Interop: default
mode: basic merge-control: allow
session: none
hard-zoning: enabled broadcast: disabled
Default zone:
qos: none broadcast: disabled ronly: unsupported
Full Zoning Database :
DB size: 4 bytes
Zonesets:0

Zones:0 Aliases: 0

Active Zoning Database :


Database Not Available Status: Default zoning policy changed to permit
at 18:42:25 UTC Mar 20 2013

The next step in verification is to check the end host itself. In the output below, in the
Emulex OneCommand Manager app we can see that both ports of the CNA have
logged into the fabric and detected LUNs available from the SAN. If Automapping is
turned on in OneCommand, which it is below, no extra steps are needed in the
management app.

The LUNs on the SAN should now be reported to Windows Disk Management so
that the volumes can actually be mounted. In the output below, we see that the
LUNs are mounted as the NTFS1 and NTFS2 volumes.

If you want to further verify storage traffic in the data plane, you can use a Disk
Speed Test application to stress test reads and writes to the disks, as shown below.

Because the end host is pre-configured to do Multipath I/O (MPIO), we should see
traffic coming in from the ENode on both the SAN A and SAN B sides, as shown
here.
N5K1# show interface vfc111 | include rate
1 minute input rate 234900256 bits/sec
, 29362532 bytes/sec, 13779 frames/sec
1 minute output rate 140410432 bits/sec, 17551304 bytes/sec, 8434 frames/sec
N5K2# show interface vfc 211 | include rate
1 minute input rate 217952752 bits/sec
, 27244094 bytes/sec, 12960 frames/sec
1 minute output rate 157412920 bits/sec, 19676615 bytes/sec, 9206 frames/sec

Both of the 5Ks should then be trunking the SCSI reads and writes out to the
MDSes, as shown below.

N5K1# show interface fc2/1 - 2 | include fc|rate

fc2/1 is trunking 1 minute input rate 102600368 bits/sec


, 12825046 bytes/sec, 6261 frames/sec 1 minute output rate 86265328 bits/sec
, 10783166 bytes/sec, 5206 frames/sec
fc2/2 is trunking 1 minute input rate 50599760 bits/sec
, 6324970 bytes/sec, 3089 frames/sec 1 minute output rate 173235168 bits/sec
, 21654396 bytes/sec, 10456 frames/sec
N5K2# show interface fc2/1 - 2 | include fc|rate

fc2/1 is trunking 1 minute input rate 92773712 bits/sec


, 11596714 bytes/sec, 5655 frames/sec 1 minute output rate 145061296 bits/sec
, 18132662 bytes/sec, 8757 frames/sec
fc2/2 is trunking 1 minute input rate 91289928 bits/sec
, 11411241 bytes/sec, 5567 frames/sec 1 minute output rate 145173248 bits/sec
, 18146656 bytes/sec, 8764 frames/sec

Note that all four uplinks to the MDSes, two from N5K1 and two from N5K2, are
being used because the 5Ks have multiple equal cost paths to reach the SAN
targets in the fabric.

Nexus Technology Labs - Storage Area


Network (SAN)
SAN Port Channeling
Task
Enable the FCoE feature on N5K1 and N5K2.
Create VSAN 101 on MDS3 and N5K1.
Create VSAN 102 on MDS4 and N5K2.
Configure MDS3's links to the Fibre Channel SAN as follows:
Hardcode the links to 2Gbps.
Set the links to be a Fabric Ports (F_Ports).
Assign VSAN 101 to these links.
Configure MDS4's link to the Fibre Channel SAN for your rack as follows:
Hardcode the links to 2Gbps.
Set the links to be a Fabric Ports (F_Ports).
Assign VSAN 102 to these links.
Configure the links between MDS3 and N5K1 as follows:
Hardcode the links to 2Gbps.
Aggregate these links together into Port Channel 11 using the Port Channel
Protocol for negotiation.
Set the port channel to be a Trunking Expansion Port (TE_Port).
Limit the VSANs on the trunk to just VSAN 101.
Configure the links between MDS4 and N5K2 as follows:
Hardcode the links to 2Gbps.
Statically aggregate these links together into Port Channel 12.
Set the port channel to be a Trunking Expansion Port (TE_Port).
Limit the VSANs on the trunk to just VSAN 102.
When complete, ensure that the MDSs see the FC SAN ports in the FLOGI database
and that the N5Ks see the FC SAN ports in the FCNS database.

Configuration

N5K1:

feature fcoe
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
channel-group 11 force
no shutdown
!
interface san-port-channel 11
channel mode active
switchport mode E
switchport trunk allowed vsan 101
switchport speed 2000
N5K2:
feature fcoe
!
vsan database
vsan 102
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
channel-group 12 force
no shutdown
!
interface san-port-channel 12
no channel mode active
switchport mode E
switchport trunk allowed vsan 102
switchport speed 2000
MDS3:
interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
vsan database
vsan 101
vsan 101 interface fc1/9
vsan 101 interface fc1/10
!

interface fc1/1 - 2
switchport speed 2000
switchport mode E
channel-group 11 force
no shutdown
!
interface port-channel 11
channel mode active
switchport mode E
switchport trunk allowed vsan 101
switchport speed 2000
MDS4:

interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
vsan database
vsan 102
vsan 102 interface fc1/9
vsan 102 interface fc1/10
!
interface fc1/1 - 2
switchport speed 2000
switchport mode E
channel-group 12 force
no shutdown
!
interface port-channel 12
no channel mode active
switchport mode E
switchport trunk allowed vsan 102
switchport speed 2000

Verification
Like Ethernet port channels, SAN port channels are used to aggregate the
bandwidth of multiple physical links. Fibre Channel traffic is then load balanced over
the member links of the SAN port channel using the source ID, destination ID, and
exchange ID. Note that the Nexus 5000 switches denote these as san-portchannels, whereas the MDS denotes them simply as port-channels.
Below we see that the SAN port channel has formed between N5K1 and MDS3

using the Port Channel Protocol for negotiation.


N5K1# show san-port-channel summary

U-Up D-Down B-Hot-standby S-Suspended I-Individual link

summary header
-------------------------------------------------------------------------------Group

Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------11

San-po11

FC

PCP

(U)

FC

fc2/1(P)

fc2/2(P)

MDS3# show port-channel database interface port-channel 11


port-channel 11
Administrative channel mode is active Operational channel mode is active
Last membership update succeeded
First operational port is fc1/2
2 ports in total, 2 ports up Ports:
fc1/2

fc1/1

[up]

[up]

*
N5K1# show san-port-channel database interface san-port-channel 11
san-port-channel 11
Last membership update is successful
2 ports in total, 2 ports up
First operational port is fc2/2
Age of the port-channel is 0d:00h:03m:06s Ports:
fc2/2

fc2/1

[up]

[up]

Below we see that the port channel is running as a Trunking Expansion Port
(TE_Port) and is trunking VSAN 101. The speed of the channel is 4Gbps, the
aggregate of 2 x 2Gbps member ports.
N5K1# show interface san-port-channel 11 brief

-------------------------------------------------------------------------------Interface

Vsan

Admin

Status

Trunk

Oper

Oper

IP

Mode

Speed

Address

Mode

(Gbps)

-------------------------------------------------------------------------------san-port-channel 11

on

trunking

TE

-N5K1# show interface san-port-channel 11 trunk vsan


san-port-channel 11 is trunking

Vsan 101 is up (None)

From a Fibre Channel routing point of view, N5K1 sees MDS3 (Domain ID 0x51)
reachable through the single san-port-channel 11 interface.
N5K1# show fspf internal route vsan 101

FSPF Unicast Routes


--------------------------VSAN Number

Dest Domain

Route Cost

Next hops

----------------------------------------------- 201

0x51(81)

250 san-port-channel 11

The Fibre Channel SAN sends a Fabric Login (FLOGI) to MDS3 to join the fabric,
and is assigned an FCID.
MDS3# sh flogi database vsan 101

-------------------------------------------------------------------------------INTERFACE

VSAN

FCID

PORT NAME

NODE NAME

-------------------------------------------------------------------------------fc1/9

101

0x510000

21:00:00:1b:32:07:32:23 20:00:00:1b:32:07:32:23

fc1/10

101

0x510100

21:01:00:1b:32:27:32:23 20:01:00:1b:32:27:32:23

Total number of flogi = 2.

N5K1 can see the pWWN and FCID of this target as registered to the Fibre Channel
Name Server database, which indicates that the fabric communication is end-to-end
for VSAN 101.

N5K1# show fcns database vsan 101

VSAN 101:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0x510000

21:00:00:1b:32:07:32:23 (Qlogic)

scsi-fcp:init

0x510100

21:01:00:1b:32:27:32:23 (Qlogic)

scsi-fcp:init

Total number of entries = 2

Nexus Technology Labs - Storage Area


Network (SAN)
VSANs and VSAN Trunking
Task
Enable the FCoE feature on N5K1 and N5K2.
Create VSAN 101 on MDS3 and N5K1.
Create VSAN 102 on MDS4 and N5K2.
Configure MDS3's links to the Fibre Channel SAN as follows:
Hardcode the links to 2Gbps.
Set the links to be a Fabric Ports (F_Ports).
Assign VSAN 101 to these links.
Configure MDS4's link to the Fibre Channel SAN for your rack as follows:
Hardcode the links to 2Gbps.
Set the links to be a Fabric Ports (F_Ports).
Assign VSAN 102 to these links.
Configure the links between MDS3 and N5K1 as follows:
Hardcode the links to 2Gbps.
Set the links to be Trunking Expansion Ports (TE_Ports).
Limit the VSANs on the trunk to just VSAN 101.
Configure the links between MDS4 and N5K2 as follows:
Hardcode the links to 2Gbps.
Set the links to be Trunking Expansion Ports (TE_Ports).
Limit the VSANs on the trunk to just VSAN 102.
When complete, ensure that the MDSs see the FC SAN ports in the FLOGI database
and that the N5Ks see the FC SAN ports in the FCNS database.

Configuration
N5K1:
feature fcoe
!
vsan database

vsan 101
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 101
no shutdown
N5K2:
feature fcoe
!
vsan database
vsan 102
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 102
no shutdown
MDS3:
interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
vsan database
vsan 101
vsan 101 interface fc1/9
vsan 101 interface fc1/10
!
interface fc1/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 101
no shutdown
MDS4:

interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
vsan database
vsan 102
vsan 102 interface fc1/9
vsan 102 interface fc1/10
!

interface fc1/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 102
no shutdown

Verification
This scenario shows how to establish basic VSAN connectivity between the MDS
and Nexus 5000 switches. First, we verify that the VSAN is assigned to the correct
Fabric Ports to the storage array and that the VSAN is trunking between the
switches.
MDS3# show vsan 101 membership
vsan 101 interfaces
: fc1/9

fc1/10

MDS3# show interface fc1/1 - 2 trunk vsan


fc1/1 is trunking
Vsan 101 is up
(None)
fc1/2 is trunking
Vsan 101 is up (None)

The port types can also be verified with

show interface brief

, as you can see below.

MDS3# show interface fc1/1-2,fc1/9-10 brief

------------------------------------------------------------------------------Interface

Vsan

Admin

Admin

Mode

Trunk

Status

SFP

Oper

Oper

Port

Mode

Speed

Channel

Mode

(Gbps)

------------------------------------------------------------------------------fc1/1

on trunking

swl TE

-- fc1/2

on trunking

-- fc1/9

101

on

up

swl F

-- fc1/10

101

on

up

swl F

--

swl TE

If vsan 101 is end-to-end through the fabric, the switches should know about each
others Domain IDs. In the output below, we see that MDS3 is assigned the Domain
ID 0x51 and N5K3 is assigned 0x12.
MDS3# show fcdomain domain-list vsan 101

Number of domains: 2
Domain ID
---------

WWN
----------------------- 0x51(81)

20:65:00:0d:ec:26:ea:81 [Local]

[Principal]
0x12(18)

20:65:00:0d:ec:a2:ed:81

N5K1# show fcdomain domain-list vsan 101

Number of domains: 2
Domain ID

WWN

---------

-----------------------

0x51(81)

20:65:00:0d:ec:26:ea:81 [Principal] 0x12(18)

20:65:00:0d:ec:a2:ed:81 [Local]

Because MDS3 and N5K3 have two trunk links connected between them, they
should both be seen as equal cost paths to reach the remote switchs Domain ID, as
shown below.
N5K1# show fspf internal route vsan 101

FSPF Unicast Routes


--------------------------VSAN Number

Dest Domain

Route Cost

Next hops

----------------------------------------------- 101

0x51(81)

500

fc2/1

fc2/2

Now that the inter-switch communication has been verified, we can focus on the
Fibre Channel Target and Initiator verification. The FC Target is the SAN directly
attached to MDS3. If this communication is successful, we should see that the
SANs FC HBAs have sent a Fabric Login (FLOGI) into the switch fabric. This can
be verified below, which shows the ports that the SAN is connected on, its VSAN
assignments, its Fibre Channel Identifiers (FCID), and both the Port World Wide
Names (pWWN/WWPN) and the Node World Wide Names (nWWN/WWNN).
MDS3# show flogi database vsan 101v
-------------------------------------------------------------------------------INTERFACE

VSAN

FCID

PORT NAME

NODE NAME

-------------------------------------------------------------------------------fc1/9

101

0x510000

21:00:00:1b:32:07:32:23 20:00:00:1b:32:07:32:23

Total number of flogi = 2.

When FLOGI is complete, targets and initiators register with the Fibre Channel
Name Server/Service (FCNS). Unlike the FLOGI database, which is only local to the
switch where the initiator/target is directly connected, the FCNS database is fabricwide. This means that if the Nexus 5K and the MDS are properly communicating,
the N5K should see the FCIDs and pWWNs of the SAN in the FCNS database, as
shown below.
N5K1# show fcns database vsan 101

VSAN 101:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0x510000

21:00:00:1b:32:07:32:23

(Qlogic)

scsi-fcp:init 0x510100

(Qlogic)

scsi-fcp:init

Total number of entries = 2

21:01:00:1b:32:27:32:23

Nexus Technology Labs - Storage Area


Network (SAN)
Fibre Channel Device Aliases
Task
Create VSAN 101 on MDS3 and N5K1.
Configure MDS3's links to the SAN as 2Gbps F_Ports in VSAN 101.
Configure trunking on the links between MDS3 and N5K1 as 2Gbps TE_Ports for
VSAN 101.
Configure N5K1 to pair with the Fabric Extender N2K1 using FEX number 101.
Configure FCoE support for the Emulex CNA server on N5K1 as follows:
Enable the FCoE feature set on N5K1.
Configure VLAN 1101 on N5K1, and map it to the FCoE VSAN 101.
Configure N5K1's link to the Emulex CNA Server as an 802.1q trunk and as
STP port type edge trunk.
Configure VLAN 10 on N5K1, which will be the server's access VLAN for
regular LAN traffic (not FCoE traffic).
Configure N5K1's link to the CNA server with VLAN 10 as the native VLAN,
and limit the trunk link to only VLANs 10 and 1101.
Create interface Virtual Fibre Channel 111 on N5K1, and bind this to N5K1's
link to the CNA server.
Assign this interface VFC111 to VSAN 101, and edit the trunking allowed list
so that it only includes VSAN 101.
Configure device aliases in the Fibre Channel fabric as follows:
Change the Device Alias mode to Enhanced.
Start a device-alias database session on MDS3.
Create a device alias named ARRAY_1_SAN_A_PORT_1 for the pWWN
learned on MDS3's first link to the SAN.
Create a device alias named ARRAY_1_SAN_A_PORT_2 for the pWWN
learned on MDS3's second link to the SAN.
Create a device alias named EMULEX_CNA_SERVER_SAN_A for the
pWWN of the CNA server.

Create a device alias named NEW_INITIATOR for the pWWN


10:00:00:00:00:00:00:01.
Commit the Device Alias database, and ensure that N5K1 learns the name
to pWWN mappings.
Configure zoning for VSAN 101 as follows:
Enable enhanced zoning for VSAN 101.
Create a Zoneset named VSAN_101_ZONESET.
Add a Zone named ZONE_ONE to this set with members
ARRAY_1_SAN_A_PORT_1 and EMULEX_CNA_SERVER_SAN_A.
Add another zone named ZONE_TWO to this set with members
ARRAY_1_SAN_A_PORT_2 and NEW_INITIATOR.
Zone membership should be based on the device aliases instead of directly
on the pWWNs.
Commit the Zoneset, and verify that both MDS3 and N5K1 have activated
the Zoneset.
When complete, the Emulex CNA server should be able to mount only one volume,
NTFS1.

Configuration
N5K1:
feature fex
feature fcoe
!
vlan 10
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101
!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 101
no shutdown
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!

interface Ethernet101/1/1
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 10,1101
spanning-tree port type edge trunk
no shutdown
!
interface vfc111
bind interface Ethernet101/1/1
switchport trunk allowed vsan 101
no shutdown
!
vsan database
vsan 101 interface vfc111
!
device-alias mode enhanced
device-alias database
device-alias name NEW_INITIATOR pwwn 10:00:00:00:00:00:00:01
device-alias name ARRAY_1_SAN_A_PORT_1 pwwn 21:00:00:1b:32:07:32:23
device-alias name ARRAY_1_SAN_A_PORT_2 pwwn 21:01:00:1b:32:27:32:23
device-alias name EMULEX_CNA_SERVER_SAN_A pwwn 10:00:00:00:c9:bb:19:9f
!
device-alias commit
!
zone mode enhanced vsan 101
!Active Zone Database Section for vsan 101
zone name ZONE_ONE vsan 101
member pwwn 21:00:00:1b:32:07:32:23
!

[ARRAY_1_SAN_A_PORT_1]
member pwwn 10:00:00:00:c9:bb:19:9f

[EMULEX_CNA_SERVER_SAN_A]

!
zone name ZONE_TWO vsan 101
member pwwn 21:01:00:1b:32:27:32:23
!

[ARRAY_1_SAN_A_PORT_2]
member pwwn 10:00:00:00:00:00:00:01

[NEW_INITIATOR]

!
zoneset name VSAN_101_ZONESET vsan 101
member ZONE_ONE
member ZONE_TWO
!
zoneset activate name VSAN_101_ZONESET vsan 101
do clear zone database vsan 101
!Full Zone Database Section for vsan 101
zone name ZONE_ONE vsan 101

member pwwn 21:00:00:1b:32:07:32:23


!

[ARRAY_1_SAN_A_PORT_1]
member pwwn 10:00:00:00:c9:bb:19:9f

[EMULEX_CNA_SERVER_SAN_A]

!
zone name ZONE_TWO vsan 101
member pwwn 21:01:00:1b:32:27:32:23
!

[ARRAY_1_SAN_A_PORT_2]
member pwwn 10:00:00:00:00:00:00:01

[NEW_INITIATOR]

!
zoneset name VSAN_101_ZONESET vsan 101
member ZONE_ONE
member ZONE_TWO
!
zone commit vsan 101
MDS3:

interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
vsan database
vsan 101
vsan 101 interface fc1/9
vsan 101 interface fc1/10
!
interface fc1/1 - 2
switchport speed 2000
switchport mode E
switchport trunk allowed vsan 101
no shutdown
!
device-alias mode enhanced
device-alias database
device-alias name NEW_INITIATOR pwwn 10:00:00:00:00:00:00:01
device-alias name ARRAY_1_SAN_A_PORT_1 pwwn 21:00:00:1b:32:07:32:23
device-alias name ARRAY_1_SAN_A_PORT_2 pwwn 21:01:00:1b:32:27:32:23
device-alias name EMULEX_CNA_SERVER_SAN_A pwwn 10:00:00:00:c9:bb:19:9f
!
device-alias commit
!
zone mode enhanced vsan 101
!Active Zone Database Section for vsan 101
zone name ZONE_ONE vsan 101

member pwwn 21:00:00:1b:32:07:32:23


!

[ARRAY_1_SAN_A_PORT_1]
member pwwn 10:00:00:00:c9:bb:19:9f

[EMULEX_CNA_SERVER_SAN_A]

!
zone name ZONE_TWO vsan 101
member pwwn 21:01:00:1b:32:27:32:23
!

[ARRAY_1_SAN_A_PORT_2]
member pwwn 10:00:00:00:00:00:00:01

[NEW_INITIATOR]

!
zoneset name VSAN_101_ZONESET vsan 101
member ZONE_ONE
member ZONE_TWO
!
zoneset activate name VSAN_101_ZONESET vsan 101
do clear zone database vsan 101
!Full Zone Database Section for vsan 101
zone name ZONE_ONE vsan 101
member pwwn 21:00:00:1b:32:07:32:23
!

[ARRAY_1_SAN_A_PORT_1]
member pwwn 10:00:00:00:c9:bb:19:9f

[EMULEX_CNA_SERVER_SAN_A]

!
zone name ZONE_TWO vsan 101
member pwwn 21:01:00:1b:32:27:32:23
!

[ARRAY_1_SAN_A_PORT_2]
member pwwn 10:00:00:00:00:00:00:01

[NEW_INITIATOR]

!
zoneset name VSAN_101_ZONESET vsan 101
member ZONE_ONE
member ZONE_TWO
!
zone commit vsan 101

Verification
Fibre Channel device aliases are loosely analogous to Fully Qualified Domain
Names (FQDNs) and DNS in the IP world. The goal of using aliases is to give a
human-friendly name to an FC initiator or targets hardware address, the Port World
Wide Name (pWWN). Enhanced device aliases, similar to Enhanced Zoning, can be
administered from any switch participating in the fabric and can have changes

dynamically advertised through Cisco Fabric Services (CFS), while at the same time
preventing multiple people from editing the database at the same time by adverting
a lock to the fabric. Like zoning, the device alias mode can be verified as follows:
N5K1# show device-alias status
Fabric Distribution: Enabled Database:- Device Aliases 4 Mode: Enhanced

Checksum: 0x161758502cec8377ff1ffed99edf6bf

After a device-alias database session is started, a lock is advertised to the fabric


and no one else can make changes. As shown below, N5K1 starts a session by
creating an alias; MDS3 is unable to start a session because N5K1s changes have
not yet been committed.
N5K1# config t
Enter configuration commands, one per line.

End with CNTL/Z.

N5K1(config)# device-alias database


N5K1(config-device-alias-db)# show device-alias status
Fabric Distribution: Enabled
Database:- Device Aliases 4

Mode: Enhanced

Checksum: 0x161758502cec8377ff1ffed99edf6bf
N5K1(config-device-alias-db)# device-alias name ALIAS1 pwwn 11:22:33:44:55:66:77:88
N5K1(config-device-alias-db)#
MDS3# config t
Enter configuration commands, one per line.

End with CNTL/Z.

MDS3(config)# device-alias database


MDS3(config-device-alias-db)# device-alias name ALIAS2 pwwn AA:BB:CC:DD:00:11:22:33
Operation failed. Fabric is already locked
MDS3(config-device-alias-db)# show device-alias status
Fabric Distribution: Enabled
Database:- Device Aliases 4

Mode: Enhanced

Checksum: 0x161758502cec8377ff1ffed99edf6bf
Locked By:- User "CLI/SNMPv3:admin" SWWN 20:00:00:0d:ec:a2:ed:80

After they are committed, the changes are advertised to everyone in the fabric.
N5K1(config-device-alias-db)# device-alias commit
N5K1(config)# show device-alias database
device-alias name ALIAS1 pwwn 11:22:33:44:55:66:77:88
device-alias name NEW_INITIATOR pwwn 10:00:00:00:00:00:00:01
device-alias name ARRAY_1_SAN_A_PORT_1 pwwn 21:00:00:1b:32:07:32:23
device-alias name ARRAY_1_SAN_A_PORT_2 pwwn 21:01:00:1b:32:27:32:23
device-alias name EMULEX_CNA_SERVER_SAN_A pwwn 10:00:00:00:c9:bb:19:9f

Total number of entries = 5


MDS3# show device-alias database

device-alias name ALIAS1 pwwn 11:22:33:44:55:66:77:88


device-alias name NEW_INITIATOR pwwn 10:00:00:00:00:00:00:01
device-alias name ARRAY_1_SAN_A_PORT_1 pwwn 21:00:00:1b:32:07:32:23
device-alias name ARRAY_1_SAN_A_PORT_2 pwwn 21:01:00:1b:32:27:32:23
device-alias name EMULEX_CNA_SERVER_SAN_A pwwn 10:00:00:00:c9:bb:19:9f

Total number of entries = 5

Another advantage of using device aliases is that these names will now appear in
various show commands to more clearly identify a pWWN, as shown below.
N5K1# show zoneset active
zoneset name VSAN_101_ZONESET vsan 101
zone name ZONE_ONE vsan 101

* fcid 0x630000 [pwwn 21:00:00:1b:32:07:32:23] [ARRAY_1_SAN_A_PORT_1]

* fcid 0xef0000 [pwwn 10:00:00:00:c9:bb:19:9f] [EMULEX_CNA_SERVER_SAN_A]

zone name ZONE_TWO vsan 101

* fcid 0x630100 [pwwn 21:01:00:1b:32:27:32:23] [ARRAY_1_SAN_A_PORT_2]

pwwn 10:00:00:00:00:00:00:01 [NEW_INITIATOR]


N5K1# show flogi database
-------------------------------------------------------------------------------INTERFACE

VSAN

FCID

PORT NAME

NODE NAME

-------------------------------------------------------------------------------vfc111

101

0xef0000

10:00:00:00:c9:bb:19:9f 20:00:00:00:c9:bb:19:9f

[EMULEX_CNA_SERVER_SAN_A]

Total number of flogi = 1.


N5K1# show fcns database

VSAN 101:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0x630000

21:00:00:1b:32:07:32:23 (Qlogic)

scsi-fcp:target

0x630100

21:01:00:1b:32:27:32:23 (Qlogic)

scsi-fcp:target

0xef0000

10:00:00:00:c9:bb:19:9f (Emulex)

ipfc scsi-fcp:init

Total number of entries = 3

[ARRAY_1_SAN_A_PORT_1]
[ARRAY_1_SAN_A_PORT_2]
[EMULEX_CNA_SERVER_SAN_A]

Nexus Technology Labs - Storage Area


Network (SAN)
iSCSI Virtual Targets
You may download or view the physical topology for your rack by
clicking the Resources button in the upper-right.

Task

Create VSAN 101 on MDS3, and configure its links to the SAN as 2Gbps F_Ports in
VSAN 101.
Establish IP connectivity between MDS3 and Server 1 as follows:
Configure MDS3's Gig1/1 interface with the IP address 10.0.0.63/24.
Configure Server 1's link to N5K2 with the IP address 10.0.0.11/24.
Configure N5K2's link to Server 1 in VLAN 101.
Configure 3750G-2's link the MDS3's Gig1/1 in VLAN 101.
Configure VLAN 101 trunking between N5K2 and 3750G-2.
Enable Jumbo Frame support on MDS3's Gig1/1, Server 1's link the N5K2,
and N5K2.
Verify that Server 1 can ping MDS3 in the 10.0.0.0/24 network.
Configure iSCSI Virtual Target support on MDS3 as follows:
Enable the iSCSI feature set on MDS3 for module 1.
Create interface iSCSI1/1, and assign it to VSAN 101.
Configure MDS3 to dynamically import all Fibre Channel targets into iSCSI.
Create an iSCSI Virtual Target using Server 1's IP address 10.0.0.11 as the
initiator.
MDS3 should represent this iSCSI initiator to the SAN as a Fibre Channel
initiator in VSAN 101 with pWWN 10:00:00:00:15:c5:10:01.
Configure Fibre Channel Zoning so that Server 1 has access to LUNs on
MDS3's first link to the SAN.
When complete, open the Windows iSCSI Initiator on Server 1, enter the target
address 10.0.0.63, and click Quick Connect. If successful, Server 1 should mount the
volume named "iSCSI1" from the Fibre Channel SAN.

Configuration
3750G-2:
system mtu jumbo 9000
!
vlan 101
!
interface GigabitEthernet1/0/5
description To MDS3 Gig1/1
switchport access vlan 101
switchport mode access
!
interface GigabitEthernet1/0/9
description To N5K2 E1/14
switchport trunk encapsulation dot1q

switchport trunk allowed vlan 101


switchport mode trunk
N5K2:
interface Ethernet1/1
switchport access vlan 101
speed 1000
!
interface Ethernet1/14
switchport mode trunk
switchport trunk allowed vlan 101
speed 1000
!
policy-map type network-qos JUMBO_FRAMES
class type network-qos class-default
mtu 9216
!
system qos
service-policy type network-qos JUMBO_FRAMES
MDS3:

feature iscsi
iscsi enable module 1
!
iscsi import target fc
iscsi initiator ip-address 10.0.0.11
static pWWN 10:00:00:00:15:c5:10:01
vsan 101
!
interface fc1/9 - 10
switchport speed 2000
switchport mode F
no shutdown
!
zone mode enhanced vsan 101
zone name iSCSI_VIRTUAL_TARGET vsan 101
member pwwn 10:00:00:00:15:c5:10:01
member pwwn 21:00:00:1b:32:07:32:23
!
zoneset name VSAN_101_ZONESET vsan 101
member iSCSI_VIRTUAL_TARGET
!
zoneset activate name VSAN_101_ZONESET vsan 101
zone commit vsan 101
!
interface GigabitEthernet1/1
ip address 10.0.0.63 255.255.255.0

switchport mtu 8000


no shutdown
!
interface iscsi1/1
no shutdown
!
vsan database
vsan 101
vsan 101 interface iscsi1/1
vsan 101 interface fc1/9
vsan 101 interface fc1/10

Verification
In addition to normal Fibre Channel switching, Cisco MDS switches support IP
storage (IPS) services. One of these IPS services is the iSCSI Virtual Target,
sometimes called the iSCSI Gateway feature, which allows the MDS to be a proxy
between an iSCSI-based initiator and a Fibre Channel-based target. The advantage
of this feature is that the MDS can attach to a normal Fibre Channel-based SAN and
then support any iSCSI initiator, such as the Microsoft iSCSI Initiator, as shown in
this scenario, or a NIC that supports boot from iSCSI, without needing native Fibre
Channel infrastructure end to end. From the perspective of the FC SAN, the MDS is
the FC initiator, and from the perspective of the iSCSI initiator, the MDS is the iSCSI
target.
The first step in configuring this feature is obtaining IP reachability between MDS
and the end server that is the iSCSI initiator. When considering iSCSI vs. native
Fibre Channel, it is important to remember the frame and payload size. Fibre
Channel has a max frame length of 2148 bytes, with a usable max payload of 2112
bytes. Ethernet transports normally default to a 1500-byte MTU. Running IP and
TCP on top of this adds an additional 20 bytes of overhead each, which means that
the usable TCP payload called the Maximum Segment Size (MSS) - is 1460
bytes. Using this default lower MTU with iSCSI can result in poor performance, or
dropped frames in the worst case scenario. Therefore it is normal best practice to
enable Jumbo Frame support for Ethernet whenever iSCSI is being used. On the
MDS, this is configured as the switchport MTU, and verified below.
MDS3# show interface gigabitethernet 1/1
GigabitEthernet1/1 is up
Hardware is GigabitEthernet, address is 000d.bd86.a4ec
Internet address is 10.0.0.63/24 MTU 8000

Port mode is IPS


Speed is 1 Gbps

bytes

Beacon is turned off


Auto-Negotiation is turned on
5 minutes input rate 48 bits/sec, 6 bytes/sec, 0 frames/sec
5 minutes output rate 8 bits/sec, 1 bytes/sec, 0 frames/sec
2745528 packets input, 5823955862 bytes
5119 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
4149531 packets output, 4783045498 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
0 carrier errors

On the Nexus 5K, Jumbo Frame support is enabled by changing the Network QoS
policy, as shown below.
N5K2# show policy-map system type network-qos

Type network-qos policy-maps


===============================
policy-map type network-qos JUMBO_FRAMES
class type network-qos class-fcoe
match qos-group 1

pause no-drop
mtu 2158
class type network-qos class-default
match qos-group 0
mtu 9216

When IP connectivity is established, the MDS presents itself as an iSCSI virtual


target to the end host. Below we see that the Windows machine has initiated an
iSCSI session to the MDS, and the MDS has assigned it virtual pWWN of
10:00:00:00:15:c5:10:01. Typically, you would statically assign a pWWN to an iSCSI
initiator if you wanted to do your Fibre Channel Zoning based on pWWNs, as shown
in this scenario. Zoning can also be configured based on the iSCSI Qualified Name
(IQN), the IP address of the initiator, or even login authentication information. In
these cases the pWWN assignment would be arbitrary, assuming that the SAN itself
does not need to enforce some type of LUN Masking.
MDS3# show iscsi initiator
iSCSI Node name is 10.0.0.11

iSCSI Initiator name: iqn.1991-05.com.microsoft:rack1server1

iSCSI alias name:


Configured node (iSCSI)
Node WWN is 22:02:00:0d:ec:26:ea:82 (dynamic)
Member of vsans: 101
Number of Virtual n_ports: 1 Virtual Port WWN is 10:00:00:00:15:c5:10:01 (configured)

Interface iSCSI 1/1, Portal group tag: 0x3000


VSAN ID 101, FCID 0x630200

Below we see that the MDS appears to the iSCSI initiator as the iSCSI target with
IQN iqn.1987-05.com.cisco:05.mds3.01-01.2100001b32073223.
MDS3# show iscsi virtual-target
target: iqn.1987-05.com.cisco:05.mds3.01-01.2100001b32073223

* Port WWN 21:00:00:1b:32:07:32:23 , VSAN 101


Auto-created node (iSCSI)
MDS3# show iscsi session
Initiator 10.0.0.11 Initiator name iqn.1991-05.com.microsoft:rack1server1

Session #1 Target iqn.1987-05.com.cisco:05.mds3.01-01.2100001b32073223


VSAN 101 , ISID 400001370000, Status active
, no reservation

This IQN in what appears in the Windows iSCSI Initiator when the session is
established, as shown below.

When the volume is mounted, you can use the Disk Speed Test app shown below to
generate bulk SCSI read and write traffic to the disk.

From the network side, the MDS maintains interface counters for the logical iSCSI

interface.
MDS3# show iscsi stats iscsi1/1

iscsi1/1
5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
iSCSI statistics
64734 packets input, 1377350972 bytes
Command 43407 pdus, Data-out 21269 pdus, 1374170112 bytes, 2723133 fragments, unsolicited 1489841152 bytes
output 64733 packets, 260484838 bytes
Response 43390 pdus (with sense 9), R2T 21285 pdus
Data-in 70674 pdus, 257373134 bytes

A data plane capture from the end server indicates that Jumbo Frames are being
used for the iSCSI reads and writes. Below we see a Wireshark output with an IP
packet length of 4184, which without our MTU modification would be limited to 1500
bytes.

Nexus Technology Labs - Storage Area


Network (SAN)
Fibre Channel over IP (FCIP)
Task
Configure basic Fibre Channel reachability throughout the fabric as follows:
Create VSAN 101 on N5K1, MDS3, and MDS4.
Configure MDS4's first link to the SAN as a 2Gbps F_Port in VSAN 101.
Configure trunking on the links between MDS3 and N5K1 as 2Gbps
TE_Ports for VSAN 101.
Configure N5K1 to pair with the Fabric Extender N2K1 using FEX number
101.
Configure FCoE support on N5K1 to the the Emulex CNA server.
The CNA server should use VLAN 1101 for FCoE traffic, which should map
to VSAN 101.
When complete, N5K1 and MDS3 should see the CNA server registered to
the fabric, and MDS4 should see the SAN.
Configure Fibre Channel over IP between MDS3 and MDS4 as follows:
Configure 3750G-2 so that the Gig1/1 ports of MDS3 and MDS4 are in one
VLAN, and the Gig1/2 ports are in a separate VLAN.
Configure the Gig1/1 ports on MDS3 and MDS4 with the IP addresses
169.254.1.63/24 and 169.254.1.64/24, respectively.
Configure the Gig1/2 ports on MDS3 and MDS4 with the IP addresses
169.254.2.63/24 and 169.254.2.64/24, respectively.
Enable Jumbo Frame support on the Gig ports of MDS3, MDS4, and 3750G2.
Enable the FCIP feature on MDS3 and MDS4.
Configure FCIP profile 1 on MDS3 and MDS4 that uses the source IP
address of their Gig1/1 interfaces.
Configure interface FCIP1, which uses FCIP profile 1, and references the
remote peer's IP address in the 169.254.1.0/24 subnet.
Configure FCIP profile 2 on MDS3 and MDS4 that uses the source IP

address of their Gig1/2 interfaces.


Configure interface FCIP2, which uses FCIP profile 2, and references the
remote peer's IP address in the 169.254.2.0/24 subnet.
Both of these FCIP interfaces should be TE_Ports that carry VSAN 101.
Configure device aliases so that the pWWN of the SAN connected to MDS4 gets the
alias ARRAY_1_SAN_B_PORT_1, and the pWWN of the Emulex CNA server gets
the alias EMULEX_CNA_SERVER_SAN_A.
Configure zoning in VSAN 101 using these aliases so that the CNA server can talk to
the SAN.
When complete, the CNA server should mount a volume called FCIP1, and SCSI
reads and writes to this volume should be load balanced across both FCIP tunnels
between the MDS switches.

Configuration
3750G-2:
vlan 1691,1692
!
interface GigabitEthernet1/0/5
description To MDS3 Gig1/1
switchport access vlan 1691
!
interface GigabitEthernet1/0/6
description To MDS3 Gig1/2
switchport access vlan 1692
!
interface GigabitEthernet1/0/7
description To MDS4 Gig1/1
switchport access vlan 1691
!
interface GigabitEthernet1/0/8
description To MDS4 Gig1/2
switchport access vlan 1692
N5K1:
feature fex
feature fcoe
!
vlan 1101
fcoe vsan 101
!
vsan database
vsan 101

!
interface fc2/1 - 2
switchport speed 2000
switchport mode E
no shutdown
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
switchport mode trunk
spanning-tree port type edge trunk
no shutdown
!
interface vfc111
bind interface Ethernet101/1/1
no shutdown
!
vsan database
vsan 101 interface vfc111
!
device-alias mode enhanced
!
device-alias database
device-alias name ARRAY_1_SAN_B_PORT_1 pwwn 21:02:00:1b:32:47:32:23
device-alias name EMULEX_CNA_SERVER_SAN_A pwwn 10:00:00:00:c9:bb:19:9f
device-alias commit
!
zone mode enhanced vsan 101
!
zone name CNA_SERVER_TO_SAN_OVER_FCIP vsan 101
member device-alias EMULEX_CNA_SERVER_SAN_A
member device-alias ARRAY_1_SAN_B_PORT_1
!
zoneset name VSAN101_ZONESET vsan 101
member CNA_SERVER_TO_SAN_OVER_FCIP
!
zoneset activate name VSAN101_ZONESET vsan 101
zone commit vsan 101
MDS3:
vsan database
vsan 101
!
device-alias mode enhanced
zone mode enhanced vsan 101

!
interface fc1/1 - 2
switchport speed 2000
switchport mode E
no shutdown
!
feature fcip
!
interface GigabitEthernet1/1
ip address 169.254.1.63 255.255.255.0
switchport mtu 8000
no shutdown
!
interface GigabitEthernet1/2
ip address 169.254.2.63 255.255.255.0
switchport mtu 8000
no shutdown
!
fcip profile 1
ip address 169.254.1.63
!
fcip profile 2
ip address 169.254.2.63
!
interface fcip1
use-profile 1
peer-info ipaddr 169.254.1.64
no shutdown
!
interface fcip2
use-profile 2
peer-info ipaddr 169.254.2.64
no shutdown
MDS4:

vsan database
vsan 101
!
device-alias mode enhanced
zone mode enhanced vsan 101
!
interface fc1/9
switchport speed 2000
switchport mode F
no shutdown
!

feature fcip
!
interface GigabitEthernet1/1
ip address 169.254.1.64 255.255.255.0
switchport mtu 8000
no shutdown
!
interface GigabitEthernet1/2
ip address 169.254.2.64 255.255.255.0
switchport mtu 8000
no shutdown
!
fcip profile 1
ip address 169.254.1.64
!
fcip profile 2
ip address 169.254.2.64
!
interface fcip1
use-profile 1
peer-info ipaddr 169.254.1.63
no shutdown
!
interface fcip2
use-profile 2
peer-info ipaddr 169.254.2.63
no shutdown

Verification
Fibre Channel over IP (FCIP) is a tunneling technique that is used to transport
native Fibre Channel traffic inside of TCP and IP. Unlike the iSCSI Virtual Target
feature of the MDS, an FCIP tunnel does not perform any sort of protocol
conversion, but instead is more analogous to how a GRE tunnel works in the IP
world. FCIP is sometimes called SAN Extension, because by adding the
retransmission and resequencing capabilities native to TCP on top of Fibre Channel,
the SAN can be extended over further physical distances while at the same time
guaranteeing reliability.
The FCIP tunnel configuration is very straightforward, as shown in this example.
First, IP connectivity is established between the switches over their GigE ports. In
this case the switches are in the same IP subnet, but if routing was required it would
be configured simply with static routes. Jumbo Frame support should be enabled,

just like in the iSCSI Virtual Target configuration, to allow for the larger native FC
packets to be fit inside a TCP payload.
MDS3# sh run interface gigabitethernet 1/1

version 4.1(3a)

interface GigabitEthernet1/1
ip address 169.254.1.63 255.255.255.0
switchport mtu 8000
no shutdown

MDS3# ping 169.254.1.64


PING 169.254.1.64 (169.254.1.64) 56(84) bytes of data.
64 bytes from 169.254.1.64: icmp_seq=1 ttl=255 time=0.456 ms
64 bytes from 169.254.1.64: icmp_seq=2 ttl=255 time=0.389 ms
64 bytes from 169.254.1.64: icmp_seq=3 ttl=255 time=0.452 ms
64 bytes from 169.254.1.64: icmp_seq=4 ttl=255 time=0.381 ms
64 bytes from 169.254.1.64: icmp_seq=5 ttl=255 time=0.390 ms

--- 169.254.1.64 ping statistics --5 packets transmitted, 5 received, 0% packet loss, time 3996ms
rtt min/avg/max/mdev = 0.381/0.413/0.456/0.039 ms

Next, an FCIP profile is created that references the IP address on the GigE link as
the source. This is the logical equivalent of the tunnel source command for a GRE
tunnel in IOS. Note from the output below that other TCP parameters, such as the
keepalive and retransmission time, can be modified, but the default values are fine
for this example.
MDS3# show fcip profile 1
FCIP Profile 1 Internet Address is 169.254.1.63

(interface GigabitEthernet1/1)

Tunnels Using this Profile: fcip1

Listen Port is 3225


TCP parameters
SACK is enabled
PMTU discovery is enabled, reset timeout is 3600 sec
Keep alive is 60 sec
Minimum retransmission timeout is 200 ms
Maximum number of re-transmissions is 4
Send buffer size is 0 KB
Maximum allowed bandwidth is 1000000 kbps
Minimum available bandwidth is 500000 kbps
Configured round trip time is 1000 usec

Congestion window monitoring is enabled, burst size is 50 KB


Auto jitter detection is enabled

Now that we know from where to source the FCIP tunnel, a logical FCIP interface is
created that calls the profile and specifies the other end of the tunnel. The peer-info
command below is then the logical equivalent of the tunnel destination command
for a GRE tunnel.
MDS3# show run interface fcip1

version 4.1(3a)

interface fcip1
use-profile 1
peer-info ipaddr 169.254.1.64
no shutdown

Assuming that the switch on the other end of the tunnel is properly configured, a
TCP session should now form between the endpoints, as shown below.
MDS3# show fcip counters
fcip1
TCP Connection Information
2 Active TCP connections
Control connection: Local 169.254.1.63:3225, Remote 169.254.1.64:65500
Data connection: Local 169.254.1.63:3225, Remote 169.254.1.64:65502

32 Attempts for active connections, 6 close of connections


TCP Parameters
Path MTU 8000 bytes
Current retransmission timeout is 200 ms
Round trip time: Smoothed 2 ms, Variance: 1 Jitter: 195 us
Advertized window: Current: 32 KB, Maximum: 31 KB, Scale: 5
Peer receive window: Current: 25 KB, Maximum: 25 KB, Scale: 5
Congestion window: Current: 43 KB, Slow start threshold: 112 KB
Current Send Buffer Size: 31 KB, Requested Send Buffer Size: 0 KB
CWM Burst Size: 50 KB
Measured RTT : 0 us Min RTT: 0 us Max RTT: 0 us
5 minutes input rate 288 bits/sec, 36 bytes/sec, 0 frames/sec
5 minutes output rate 360 bits/sec, 45 bytes/sec, 0 frames/sec
5272 frames input, 779640 bytes
2288 Class F frames input, 232548 bytes
2984 Class 2/3 frames input, 547092 bytes
0 Reass frames
0 Error frames timestamp error 0

309765 frames output, 649335820 bytes


2207 Class F frames output, 260232 bytes
307558 Class 2/3 frames output, 649075588 bytes
0 Error frames

Although the FCIP tunnel transport is IP, it is now treated like a normal Fibre
Channel link. Specifically, it is treated as a Trunking Expansion Port (TE_Port), just
like a native Fibre Channel link between the switches.
MDS3# show interface fcip1
fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 20:10:00:0d:ec:26:ea:80
Peer port WWN is 20:10:00:0d:ec:3f:8a:00
Admin port mode is auto, trunk mode is on
snmp link state traps are enabled Port mode is TE
Port vsan is 1
Speed is 1 Gbps
Trunk vsans (admin allowed and active) (1,101) Trunk vsans (up)

Trunk vsans (isolated)

()

Trunk vsans (initializing)

()

Interface last changed at Sat Sep

(1,101)

4 04:24:23 1982

<snip>

The link also participates in normal Fabric Services, such as FCNS, FSPF, Zoning,
Device Aliases, etc., just like any other native FC link in the topology. Because we
configured two separate FCIP tunnels, they are seen as equal-cost paths between
the MDS switches.
MDS3# show fspf internal route vsan 101

FSPF Unicast Routes


--------------------------VSAN Number

Dest Domain

Route Cost

Next hops

----------------------------------------------- 101
fcip2

101

0xef(239)

500

fc1/1
fc1/2

0xee(238)

1000

fcip1

If the fabric is end to end, we should see the pWWNs of the target and initiator in the
FCNS database of the MDSes and the N5K.
MDS3# show fcns database

VSAN 101:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0xee0000

21:02:00:1b:32:47:32:23

scsi-fcp:target

[ARRAY_1_SAN_B_PORT_1]
0xef0000

10:00:00:00:c9:bb:19:9f (Emulex)

ipfc scsi-fcp:init

[EMULEX_CNA_SERVER_SAN_A]

Total number of entries = 2


MDS4# show fcns database

VSAN 101:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0xee0000

21:02:00:1b:32:47:32:23

scsi-fcp:target

[ARRAY_1_SAN_B_PORT_1]
0xef0000

10:00:00:00:c9:bb:19:9f (Emulex)

ipfc scsi-fcp:init

[EMULEX_CNA_SERVER_SAN_A]

Total number of entries = 2


N5K1# show fcns database

VSAN 101:
-------------------------------------------------------------------------FCID

TYPE

PWWN

(VENDOR)

FC4-TYPE:FEATURE

-------------------------------------------------------------------------0xee0000

21:02:00:1b:32:47:32:23 (Qlogic)

scsi-fcp:target

[ARRAY_1_SAN_B_PORT_1]
0xef0000

10:00:00:00:c9:bb:19:9f (Emulex)

ipfc scsi-fcp:init

[EMULEX_CNA_SERVER_SAN_A]

Total number of entries = 2

Before the initiator can actually talk to the target, zoning must also be configured.
MDS4# show zoneset active vsan 101

zoneset name VSAN101_ZONESET vsan 101


zone name CNA_SERVER_TO_SAN_OVER_FCIP vsan 101
* fcid 0xef0000 [device-alias EMULEX_CNA_SERVER_SAN_A]
* fcid 0xee0000 [device-alias ARRAY_1_SAN_B_PORT_1]

The Emulex CNA server should now automatically mount the LUN presented by the
SAN, as shown below.

Traffic can be generated in the data plane using the Disk Speed Test app on the
server.

Throughput to the disks should be higher than the iSCSI Virtual Target
configuration, as less processing is required to simply encapsulate the traffic into
TCP and IP than to do an upper-layer protocol conversion, but it should be a little
slower than native 2Gbps Fibre Channel because some overhead is still incurred by
the tunneling. Verification of the counters of the FCIP tunnel interfaces should
indicate that both links are being used.
MDS4# show interface fcip1 - 2 | include "fcip|input rate|output rate"

fcip1 is trunking
5 minutes input rate 11739632 bits/sec, 1467454 bytes/sec, 696 frames/sec
5 minutes output rate 7675984 bits/sec, 959498 bytes/sec, 460 frames/sec
fcip2 is trunking
5 minutes input rate 62896248 bits/sec, 7862031 bytes/sec, 3730 frames/sec
5 minutes output rate 35324400 bits/sec, 4415550 bytes/sec, 2119 frames/sec

Nexus Technology Labs - Classical Ethernet


Switching
Core Nexus LAN Switching
Task
Configure N7K1, N7K2, N5K1, and N5K2 as follows so that IP connectivity is
established between Server 1 and Server 2:
Create VLAN 10 on N7K1, N7K2, N5K1, and N5K2.
Assign VLAN 10 as the access VLAN for N5K1's connection to Server 1;
Server 1 should use the IP address 10.0.0.1/24.
Assign VLAN 10 as the access VLAN for N5K2's connection to Server 2;
Server 2 should use the IP address 10.0.0.2/24.
Statically set these links to 1Gbps on N5K1 and N5K2, respectively.
Configure 802.1Q Trunking between the switches as follows:
N5K1's links to N7K1 should all be trunks.
N5K2's links to N7K2 should all be trunks.
N7K1's links to N7K2 should all be trunks.
Edit the trunking allowed list so that only VLANs 1 and 10 will forward over
these links.
Disable all other interconnections between the switches.
When complete, Server 1 and Server 2 should have IP connectivity to each other.

Configuration
N5K1:
vlan 1,10
!
interface Ethernet1/1
switchport access vlan 10
speed 1000
no shutdown
!
interface Ethernet1/2 - 5
shutdown

!
interface Ethernet1/6
switchport mode trunk
switchport trunk allowed vlan 1,10
no shutdown
!
interface Ethernet1/7
switchport mode trunk
switchport trunk allowed vlan 1,10
no shutdown
!
interface Ethernet1/8 - 9
shutdown
N5K2:
vlan 1,10
!
interface Ethernet1/1
shutdown
!
interface Ethernet1/2
switchport access vlan 10
speed 1000
no shutdown
!
interface Ethernet1/3 - 5
shutdown
!
interface Ethernet1/6
switchport mode trunk
switchport trunk allowed vlan 1,10
no shutdown
!
interface Ethernet1/7
switchport mode trunk
switchport trunk allowed vlan 1,10
no shutdown
!
interface Ethernet1/8 - 9
shutdown
N7K1:
vlan 1,10
!
interface Ethernet1/1 - 2, Ethernet2/1 - 4
switchport
switchport mode trunk
switchport trunk allowed vlan 1,10

no shutdown
N7K2:

vlan 1,10
!
interface Ethernet1/1 - 2, Ethernet2/1 - 4
switchport
switchport mode trunk
switchport trunk allowed vlan 1,10
no shutdown

Verification
Server 1 is able to reach Server 2 on the local LAN segment:

Only the required interfaces are connected. Links to the end servers run at 1Gbps,
whereas the trunks between the switches run at 10Gbps:
N5K1# show interface status

-------------------------------------------------------------------------------Port

Name

Status

Vlan

Duplex

Speed

Type

-------------------------------------------------------------------------------Eth1/1

--

connected 10

full

1000

SFP-1000BAS
Eth1/2

--

disabled

full

10G

SFP-1000BAS

Eth1/3

--

disabled

full

10G

SFP-H10GB-C

Eth1/4

--

disabled

full

10G

SFP-H10GB-C

Eth1/5

--

disabled

full

10G

SFP-H10GB-C

Eth1/6

--

connected trunk

full

10G

SFP-H10GB-C Eth1/7
SFP-H10GB-C

--

connected trunk

full

10G

Eth1/8

--

disabled

full

10G

SFP-H10GB-C

Eth1/9

--

disabled

full

10G

SFP-H10GB-C

VLAN 10 is created and forwarding on the required interfaces:


N5K1# show vlan brief

VLAN Name

Status

Ports

---- -------------------------------- --------- ------------------------------1

default

active

Eth1/2, Eth1/3, Eth1/4, Eth1/5


Eth1/6, Eth1/7, Eth1/8, Eth1/9
Eth1/10, Eth1/11, Eth1/12
Eth1/13, Eth1/14, Eth1/15
Eth1/16, Eth1/17, Eth1/18
Eth1/19, Eth1/20, Eth1/21
Eth1/22, Eth1/23, Eth1/24
Eth1/25, Eth1/26, Eth1/27
Eth1/28, Eth1/29, Eth1/30
Eth1/31, Eth1/32

10

VLAN0010

active

Eth1/1, Eth1/6, Eth1/7

VLANs 1 and 10 are trunking on the required interfaces, and other VLANs have
been removed from the allowed list:
N7K1# show interface trunk

-------------------------------------------------------------------------------- Port

Native

Status
Port
Vlan

Channel

-------------------------------------------------------------------------------- Eth1/1
-- Eth1/2

1 trunking

-- Eth2/1

1 trunking

-- Eth2/2

1 trunking

-- Eth2/3

1 trunking

-- Eth2/4

1 trunking

1 trunking

--

-------------------------------------------------------------------------------- Port
Vlans Allowed on Trunk
-------------------------------------------------------------------------------- Eth1/1
Eth1/2

1,10

Eth2/1

1,10

Eth2/2

1,10

Eth2/3

1,10

1,10

-------------------------------------------------------------------------------Port

Vlans Err-disabled on Trunk

-------------------------------------------------------------------------------Eth1/1

none

Eth1/2

none

Eth2/1

none

Eth2/2

none

Eth2/3

none

Eth2/4

none

-------------------------------------------------------------------------------- Port STP Forwarding


-------------------------------------------------------------------------------- Eth1/1
Eth1/2

none

Eth2/1

none

Eth2/2

none Eth2/3

Eth2/4

1,10

1,10

1,10

-------------------------------------------------------------------------------- Port
Vlans in spanning tree forwarding state and not pruned
-------------------------------------------------------------------------------- Eth1/1
Eth1/2

none

Eth2/1

none

Eth2/2

none Eth2/3

Eth2/4

1,10

1,10

<snip>

All four switches agree on who the root of the Spanning-Tree is for VLAN 10, and
alternate paths to the root bridge are blocked:
N5K1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp Root ID
Address

Bridge ID

Interface

Priority

32778

0026.980c.2141
Cost

Port

134 (Ethernet1/6)

Hello Time

Priority

32778

Address

547f.ee79.137c

Hello Time

sec

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

Max Age 20 sec

Prio.Nbr Type

Forward Delay 15 sec

1,10

---------------- ---- --- --------- -------- -------------------------------- Eth1/1


4

128.129

P2p Eth1/6

Root FWD

128.134

P2p Eth1/7

Altn BLK

128.135

P2p

Desg FWD

N5K2# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp Root ID
Address

Priority

32778

0026.980c.2141

Bridge ID

Cost

Port

134 (Ethernet1/6)

Hello Time

Priority

32778

Address

547f.ee7a.4d7c

Hello Time

Interface

sec

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------- Eth1/2


4

128.130

P2p Eth1/6

Root FWD

128.134

P2p Eth1/7

Altn BLK

128.135

P2p

Desg FWD

N7K1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp Root ID
Address

32778

0026.980c.2141

Bridge ID

Interface

Priority

Cost

Port

129 (Ethernet1/1)

Hello Time

Priority

32778

Address

68bd.abd7.6041

Hello Time

sec

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------- Eth1/1


2

128.129

P2p Eth1/2

Altn BLK

128.130

P2p Eth2/1

Altn BLK

128.257

P2p Eth2/2

Altn BLK

128.258

P2p Eth2/3

Desg FWD

128.259

P2p Eth2/4

Desg FWD

128.260

P2p

N7K2# show spanning-tree vlan 10

VLAN0010

Root FWD

Spanning tree enabled protocol rstp Root ID


Address

Priority

32778

0026.980c.2141

This bridge is the root

Bridge ID

Interface

Hello Time

sec

Max Age 20 sec

Priority

32778

Address

0026.980c.2141

Hello Time

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

sec

Max Age 20 sec

Role Sts Cost

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------- Eth1/1


2

128.129

P2p Eth1/2

Desg FWD

128.130

P2p Eth2/1

Desg FWD

128.257

P2p Eth2/2

Desg FWD

128.258

P2p Eth2/3

Desg FWD

128.259

P2p Eth2/4

Desg FWD

128.260

P2p

Server 1 and Server 2's MAC addresses are learned in the VLAN 10 CAM table:
N5K1# show mac address-table dynamic vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN

MAC Address

Type

age

Secure NTFY

Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------ *
10
* 10

000c.29d0.a588
000c.29f8.57ec

dynamic
dynamic

180
250

F
F

Eth1/6
F

Eth1/1

Desg FWD

Nexus Technology Labs - Classical Ethernet


Switching
VLAN Trunking Protocol (VTP)
Task
Configure trunking between N7K1, N7K2, N5K1, and N5K2 as follows:
N5K1's links to N7K1 should all be trunks.
N5K2's links to N7K2 should all be trunks.
N7K1's links to N7K2 should all be trunks.
Disable all other interconnections between the switches.
Configure VLAN Trunking Protocol (VTP) on the switches as follows:
Enable VTP on all switches.
Use the VTP domain name NEXUS.
N5K1 should run in VTP client mode.
N5K2 should run in VTP off mode.
N7K1 should run in VTP transparent mode.
N7K2 should run in VTP server mode.
Configure VTP authentication on N5K1 and N7K2 using the password
VTPPASSWORD.
Configure the following VLANs on N7K2:
VLAN 10 named TEN.
VLAN 20 named TWENTY.
VLAN 1000 named ONETHOUSAND.
VLAN 2000 named TWOTHOUSAND.
Configure VLAN 1500 named FIFTEENHUNDRED on N5K1.
Configure VLAN 3000 named THREETHOUSAND on N7K1 and N5K2.
Verify the resulting VLAN databases on the switches.

Configuration
N5K1:

feature vtp
vtp domain NEXUS
vtp mode client
vtp password VTPPASSWORD
!
vlan 1500
name FIFTEENHUNDRED
!
interface Ethernet1/1 - 5
shutdown
!
interface Ethernet1/6 - 7
switchport mode trunk
no shutdown
!
interface Ethernet1/8 - 9
shutdown
N5K2:
feature vtp
vtp domain NEXUS
vtp mode off
vlan 3000
name THREETHOUSAND
!
interface Ethernet1/1 - 5
shutdown
!
interface Ethernet1/6 - 7
switchport mode trunk
no shutdown
!
interface Ethernet1/8 - 9
shutdown
N7K1:
feature vtp
vtp domain NEXUS
vtp mode transparent
!
vlan 3000
name THREETHOUSAND
!
interface Ethernet1/1 - 2, Ethernet2/1 - 4
switchport
switchport mode trunk
no shutdown
N7K2:

feature vtp
vtp domain NEXUS
vtp mode server
vtp password VTPPASSWORD
!
vlan 10
name TEN
vlan 20
name TWENTY
vlan 1000
name ONETHOUSAND
vlan 2000
name TWOTHOUSAND
!
interface Ethernet1/1 - 2, Ethernet2/1 - 4
switchport
switchport mode trunk
no shutdown

Verification
Like Catalyst IOS, Nexus NX-OS supports VLAN Trunking Protocol (VTP) to ease in
the administration of VLAN creation, deletion, and naming throughout the LAN.
However, there are a few key functional differences between NX-OSs
implementation of VTP and most other platforms.
The first, as shown in this example, is that NX-OS supports VTP mode off. In this
mode, all received VTP updates are filtered, and the VLAN database is only locally
significant. This is unlike VTP Transparent mode, which passes VTP updates
between interfaces, even though its database is locally significant as well. Another
difference, as of the version in this example, is that Nexus 5K does not support VTP
pruning. Also note that although extended VLANs can be configured in all four VTP
modes, they still cannot be advertised by either servers or clients because there is
currently no support for VTP version 3.
The end result is that it is fairly easy for the VLAN databases to become out of sync
when there is a mix of VTP modes in the network, not to mention the same potential
problems as in Catalyst IOS of mistakes in the configuration of VTP servers possibly
deleting VLANs, or the entire VLAN database, in the entire network. For this reason,
it is very typical that VTP is not used in Data Center LAN environments, although
the feature is still technically supported.
Verification of VTP is similar to that of Catalyst IOS, as follows:

N5K1# show vtp status

VTP Status Information


---------------------VTP Version

: 2 (capable) Configuration Revision

: 16

Maximum VLANs supported locally : 1005


Number of existing VLANs

: 8 VTP Operating Mode

VTP Domain Name

: NEXUS

VTP Pruning Mode

: Disabled (Operationally Disabled)

VTP V2 Mode

: Disabled

VTP Traps Generation

: Disabled MD5 Digest

: Client

0xEC 0x88 0x7C 0xB6 0x3E 0x9E 0x73 0xF2


Configuration last modified by 0.0.0.0 at 2-13-13 17:47:19
VTP version running

: 1

N5K1# show vlan brief

VLAN Name

Status

Ports

---- -------------------------------- --------- ------------------------------1

default

active

Eth1/2, Eth1/3, Eth1/4, Eth1/5


Eth1/6, Eth1/7, Eth1/8, Eth1/9
Eth1/10, Eth1/11, Eth1/12
Eth1/13, Eth1/14, Eth1/15
Eth1/16, Eth1/17, Eth1/18
Eth1/19, Eth1/20, Eth1/21
Eth1/22, Eth1/23, Eth1/24
Eth1/25, Eth1/26, Eth1/27
Eth1/28, Eth1/29, Eth1/30
Eth1/31, Eth1/32 10

active
active
active

Eth1/1, Eth1/6, Eth1/7 20

TEN
TWENTY

Eth1/6, Eth1/7 1000 ONETHOUSAND

Eth1/6, Eth1/7

1002 fddi-default

suspended Eth1/6, Eth1/7

1003 token-ring-default

suspended Eth1/6, Eth1/7

1004 fddinet-default

suspended Eth1/6, Eth1/7

1005 trnet-default

suspended Eth1/6, Eth1/7 1500 FIFTEENHUNDRED


active

Eth1/6, Eth1/7

N5K2# show vtp status


VTP Status Information
---------------------VTP Version

: 2 (capable) Configuration Revision

:0

Maximum VLANs supported locally : 1005


Number of existing VLANs

: 1 VTP Operating Mode

VTP Domain Name

: NEXUS

VTP Pruning Mode

: Disabled (Operationally Disabled)

VTP V2 Mode

: Disabled

VTP Traps Generation

: Disabled

: Off

MD5 Digest

: 0x04 0x6C 0xBC 0xB5 0x6F 0xCE 0x92 0xC9

Configuration last modified by 0.0.0.0 at 2-13-13 17:26:49


VTP version running

: 1

N5K2# show vlan brief

VLAN Name

Status

Ports

---- -------------------------------- --------- ------------------------------1

default

active

Eth1/1, Eth1/3, Eth1/4, Eth1/5


Eth1/6, Eth1/7, Eth1/8, Eth1/9
Eth1/10, Eth1/11, Eth1/12
Eth1/13, Eth1/14, Eth1/15
Eth1/16, Eth1/17, Eth1/18
Eth1/19, Eth1/20, Eth1/21
Eth1/22, Eth1/23, Eth1/24
Eth1/25, Eth1/26, Eth1/27
Eth1/28, Eth1/29, Eth1/30
Eth1/31, Eth1/32 3000 THREETHOUSAND

active

Eth1/6, Eth1/7

N7K1-1# show vtp status


VTP Status Information
---------------------VTP Version

: 2 (capable) Configuration Revision

:0

Maximum VLANs supported locally : 1005


Number of existing VLANs

: 1 VTP Operating Mode

VTP Domain Name

: NEXUS

: Transparent

VTP Pruning Mode

: Disabled (Operationally Disabled)

VTP V2 Mode

: Disabled

VTP Traps Generation

: Disabled

MD5 Digest

: 0x93 0x5A 0xCB 0xBB 0xDA 0xC8 0x15 0x37

Configuration last modified by 0.0.0.0 at 2-13-13 17:44:06


VTP version running

: 1

N7K1-1# show vlan brief

VLAN Name

Status

Ports

---- -------------------------------- --------- ------------------------------1

default

active

Eth1/1, Eth1/2, Eth2/1, Eth2/2


Eth2/3, Eth2/4, Eth2/5, Eth2/6
Eth2/7, Eth2/8

3000 THREETHOUSAND

active

Eth1/1, Eth1/2, Eth2/1, Eth2/2


Eth2/3, Eth2/4N7K2-1# show vtp status

VTP Status Information


---------------------VTP Version

: 2 (capable) Configuration Revision

Maximum VLANs supported locally : 1005


Number of existing VLANs

: 8 VTP Operating Mode

: Server

: 16

VTP Domain Name

: NEXUS

VTP Pruning Mode

: Disabled (Operationally Disabled)

VTP V2 Mode

: Disabled

VTP Traps Generation

: Disabled MD5 Digest

0xEC 0x88 0x7C 0xB6 0x3E 0x9E 0x73 0xF2


Configuration last modified by 0.0.0.0 at 2-13-13 17:47:19
Local updater ID is 0.0.0.0
VTP version running

: 1

N7K2-1# show vlan brief

VLAN Name

Status

Ports

---- -------------------------------- --------- ------------------------------1

default

active

Eth1/1, Eth1/2, Eth2/1, Eth2/2


Eth2/3, Eth2/4, Eth2/5, Eth2/6
Eth2/7, Eth2/8 10

active

Eth1/1, Eth1/2, Eth2/1, Eth2/2


Eth2/3, Eth2/4 20

active

TEN

TWENTY

Eth1/1, Eth1/2, Eth2/1, Eth2/2


Eth2/3, Eth2/4 1000 ONETHOUSAND

active

Eth1/1, Eth1/2, Eth2/1, Eth2/2


Eth2/3, Eth2/4

1002 fddi-default

suspended Eth1/1, Eth1/2, Eth2/1, Eth2/2

1003 token-ring-default

suspended Eth1/1, Eth1/2, Eth2/1, Eth2/2

Eth2/3, Eth2/4

Eth2/3, Eth2/4
1004 fddinet-default

suspended Eth1/1, Eth1/2, Eth2/1, Eth2/2


Eth2/3, Eth2/4

1005 trnet-default

suspended Eth1/1, Eth1/2, Eth2/1, Eth2/2


Eth2/3, Eth2/4 2000 TWOTHOUSAND
active

Eth1/1, Eth1/2, Eth2/1, Eth2/2


Eth2/3, Eth2/4

Nexus Technology Labs - Classical Ethernet


Switching
Port Channels
Task
Configure port channels between N7K1, N7K2, N5K1, and N5K2 as follows:
The links between N5K1 and N7K1 should be grouped together in PortChannel 1. Do not use a negotiation protocol for this channel.
The links between N7K1 and N7K2 should be grouped into Port-Channels 2
and 3. Both switches should initiate LACP negotiation for both of these
channels and use LACP fast timers.
The links between N5K2 and N7K2 should grouped together into PortChannel 4. N7K2 should initiate LACP negotiation and N5K2 should
respond.
Disable all other interconnections between the switches.
Configure all of these port channels as 802.1Q trunk links.
Configure N7K2 with an LACP priority of 16384 so that it is the preferred device for
managing negotiation of its port channels.
Configure all switches to use source and destination TCP/UDP ports for load
balancing flows across the port channel members.
Configure N5K1's link to Server 1 and N5K2's link to Server 2 in VLAN 10.
Server 1 should use the IP address 10.0.0.1/24 and Server 2 should use the IP
address 10.0.0.2/24. When complete, Server 1 and Server 2 should have IP
reachability to each other.
Use the iPerf application to verify that traffic flows between Server 1 and Server 2 are
being distributed among the member links of the port channels.

Configuration
N5K1:
vlan 10
!
port-channel load-balance ethernet source-dest-port

!
interface port-channel1
switchport mode trunk
speed 10000
!
interface Ethernet1/1
switchport access vlan 10
speed 1000
!
interface Ethernet1/2 - 6
shutdown
!
interface Ethernet1/6
switchport mode trunk
channel-group 1
!
interface Ethernet1/7
switchport mode trunk
channel-group 1
!
interface Ethernet1/8 - 9
shutdown
N5K2:
feature lacp
!
vlan 10
!
port-channel load-balance ethernet source-dest-port
!
interface port-channel4
switchport mode trunk
speed 10000
!
interface Ethernet1/1
shutdown
!
interface Ethernet1/2
switchport access vlan 10
speed 1000
!
interface Ethernet1/3 - 5
shutdown
!
interface Ethernet1/6
switchport mode trunk
channel-group 4 mode passive

!
interface Ethernet1/7
switchport mode trunk
channel-group 4 mode passive
!
interface Ethernet1/8 - 9
shutdown
N7K1:
feature lacp
!
vlan 10
!
port-channel load-balance src-dst l4port
!
interface port-channel1
switchport
switchport mode trunk
!
interface port-channel2
switchport
switchport mode trunk
!
interface port-channel3
switchport
switchport mode trunk
!
interface Ethernet1/1
lacp rate fast
switchport
switchport mode trunk
channel-group 2 mode active
no shutdown
!
interface Ethernet1/2
lacp rate fast
switchport
switchport mode trunk
channel-group 2 mode active
no shutdown
!
interface Ethernet2/1
lacp rate fast
switchport mode trunk
channel-group 3 mode active
no shutdown
!

interface Ethernet2/2
lacp rate fast
switchport mode trunk
channel-group 3 mode active
no shutdown
!
interface Ethernet2/3
switchport mode trunk
channel-group 1
no shutdown
!
interface Ethernet2/4
switchport mode trunk
channel-group 1
no shutdown
N7K2:

feature lacp
!
vlan 10
!
lacp system-priority 16384
port-channel load-balance src-dst l4port
!
interface port-channel2
switchport
switchport mode trunk
!
interface port-channel3
switchport
switchport mode trunk
!
interface port-channel4
switchport
switchport mode trunk
!
interface Ethernet1/1
lacp rate fast
switchport
switchport mode trunk
channel-group 2 mode active
no shutdown
!
interface Ethernet1/2
lacp rate fast
switchport

switchport mode trunk


channel-group 2 mode active
no shutdown
!
interface Ethernet2/1
lacp rate fast
switchport mode trunk
channel-group 3 mode active
no shutdown
!
interface Ethernet2/2
lacp rate fast
switchport mode trunk
channel-group 3 mode active
no shutdown
!
interface Ethernet2/3
switchport mode trunk
channel-group 4 mode active
no shutdown
!
interface Ethernet2/4
switchport mode trunk
channel-group 4 mode active
no shutdown

Verification
Port channels in NX-OS, just like in Catalyst IOS and other platforms, require that
the member interfaces first have compatible parameters for the channel to form. In
NX-OS, these parameters can be verified with the command show port-channel
compatibility-parameters . Some of these parameters can be seen below:
N7K1-1# show port-channel compatibility-parameters | include \*

* port mode
* speed
* MTU
* MEDIUM
* Span mode
<snip>

In this topology, the links between the Nexus 7000 switches include both M series

and F series modules. Because these modules have different port level capabilities,
they are not compatible to channel together. This is why the above solution has one
port channel for the M1 ports and another for the F1 ports. The NX-OS parser will
detect this and return an error message if you attempt to channel together
incompatible port types, as shown below:
N7K1-1# config t

Enter configuration commands, one per line.

End with CNTL/Z.

N7K1-1(config)# int e1/1 - 2, e2/1 - 2


N7K1-1(config-if-range)# channel-group 10 mode active
N7K1-1 %ETH_PORT_CHANNEL-3-COMPAT_CHECK_FAILURE: rate mode is not compatible
command failed: port not compatible [rate mode]

** You can use force option to override the port's parameters


** (e.g. "channel-group X force")
** Use "show port-channel compatibility-parameters" to get more information on failure

After the channels are successfully formed, the show port-channel summary output
should indicate that the member links are "Up in the port-channel" with flag (P). This
output also shows whether LACP negotiation was used or not.
N5K1# show port-channel summary
Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------1

Po1(SU)

Eth

NONE

Eth1/6(P)

Eth1/7(P)

N5K2# show port-channel summary


Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group PortChannel

Type

Protocol

Member Ports

-------------------------------------------------------------------------------4

Po4(SU)

Eth

LACP

Eth1/6(P)

Eth1/7(P)

N7K1-1# show port-channel summary


Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------1

Po1(SU)

Eth

NONE

Eth2/3(P)

Eth2/4(P)

Po2(SU)

Eth

LACP

Eth1/1(P)

Eth1/2(P)

Po3(SU)

Eth

LACP

Eth2/1(P)

Eth2/2(P)

N7K2-1# show port-channel summary


Flags:

D - Down

P - Up in port-channel (members)

I - Individual

H - Hot-standby (LACP only)

s - Suspended

r - Module-removed

S - Switched

R - Routed

U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------Group Port-

Type

Protocol

Member Ports

Channel
-------------------------------------------------------------------------------2

Po2(SU)

Eth

LACP

Eth1/1(P)

Eth1/2(P)

Po3(SU)

Eth

LACP

Eth2/1(P)

Eth2/2(P)

Po4(SU)

Eth

LACP

Eth2/3(P)

Eth2/4(P)

Spanning-Tree Protocol should see the port channels as one logical link, as shown
below. Separate channels that point the same direction in the spanning-tree, such
as Port-Channels 2 and 3 below, are still subject to the normal forwarding and
blocking rules.
N7K1-1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID

Priority

32778

Address

0026.980c.2141

Bridge ID

Cost

Port

4097 (port-channel2)

Hello Time

Priority

32778

Address

68bd.abd7.6041

Hello Time

Interface

sec

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

sec

Max Age 20 sec

Role Sts Cost

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------- Po1


1

128.4096 P2p Po2

Root FWD

128.4097 P2p Po3

Altn BLK

128.4098 P2p

Desg FWD

LACP neighbors of N7K2 should see that its System Priority (the first portion of the
System ID) has been reduced to a more preferred value of 16384. The output below
also shows whether the neighbor is running LACP in active or passive mode, and
whether slow or fast LACP hellos are being used.
N7K1-1# show lacp neighbor interface port-channel 2
Flags:

S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs

A - Device is in Active mode


P - Device is in Passive mode
port-channel2 neighbors
Partner's information Partner
Port System ID
Eth1/1 16384,0-26-98-c-21-41

Partner Partner

Port Number
0x101

Age Flags
2077 FA

LACP Partner

Partner

Partner

Port Priority

Oper Key

Port State

32768

0x1

0x3f

Partner

Partner

Partner

Port

System ID

Port Number

Age

Flags

Eth1/2

16384,0-26-98-c-21-41

0x102

2076

FA

LACP Partner

Partner

Partner

Port Priority

Oper Key

Port State

32768

0x1

0x3f

Partner's information

To verify the configured load balancing method of the switches, use the
channel load-balance command, as seen below.

show port-

Note that on the Nexus 7000 the load balancing method can only be
changed in the default VDC, as this change is chassis-wide between
all VDCs.

N5K1# show port-channel load-balance

Port Channel Load-Balancing Configuration:


System: source-dest-port

Port Channel Load-Balancing Addresses Used Per-Protocol:


Non-IP: source-dest-mac IP: source-dest-port source-dest-ip source-dest-mac
N7K1# show port-channel load-balance

Port Channel Load-Balancing Configuration:


System: src-dst l4port

Port Channel Load-Balancing Addresses Used Per-Protocol:


Non-IP: src-dst mac IP: src-dst l4port

To verify load distribution among member ports of the channels in the data plane,
Server 1 generates multiple TCP flows to Server 2 using the iPerf testing app.

Verification of the port channel from N5K1 to N7K1 shows a 30-second average
output rate of around 750Mbps. This, however, does not tell us which particular
member interfaces of the channel are being used.

N5K1# show interface port-channel 1 | include "output rate"

30 seconds output rate 754266632 bits/sec, 61966 packets/sec

input rate 3.40 Mbps, 6.13 Kpps;

output rate 754.27 Mbps


, 61.97 Kpps

By looking at the individual counters of the member interfaces, we can see that the
load is being distributed among them. Note that it is not a 1:1 even distribution,
because not every TCP flow is getting the same about of throughput based on other
variables such as burstiness, queueing, etc.
N5K1# show interface e1/6 - 7 | include "Ethernet1/|output rate"
Ethernet1/6 is up
30 seconds output rate 477799744 bits/sec, 39256 packets/sec

input rate 1.67 Mbps, 3.01 Kpps;

output rate 477.80 Mbps


, 39.26 Kpps
Ethernet1/7 is up
30 seconds output rate 276466888 bits/sec, 22710 packets/sec
output rate 276.47 Mbps
, 22.71 Kpps

input rate 1.73 Mbps, 3.12 Kpps;

Nexus Technology Labs - Classical Ethernet


Switching
Rapid PVST Traffic Engineering
Task
Configure all links connecting N7K1, N7K2, N5K1, and N5K2 as 802.1Q trunk ports.
Configure the links connecting N5K1 and N5K2 as a port channel.
Create VLANs 10 and 20 on all switches, and assign them as follows:
Server 1s link to N5K1 should be in VLAN 10 and use the IP address
10.0.0.1/24.
Server 1s link to N5K2 should be in VLAN 20 and use the IP address
20.0.0.1/24.
Server 2s link to N5K1 should be in VLAN 20 and use the IP address
20.0.0.2/24.
Server 2s link to N5K2 should be in VLAN 10 and use the IP address
10.0.0.2/24.
Configure Spanning-Tree Protocol between the switches as follows:
All switches should use 32 bits for spanning-tree port path costs.
N7K1 should be the STP Root Bridge for VLAN 10, with N7K2 being the
backup Root Bridge.
N5K2 should be the STP Root Bridge for VLAN 20, with N5K1 being the
backup Root Bridge.
Server 1s VLAN 10 traffic to Server 2 should follow the path of N5K1 ->
N7K2 -> N7K1 -> N5K2 -> Server 2.
Server 2s VLAN 20 traffic to Server 1 should follow the path of N5K1 ->
N7K1 -> N7K2 -> N5K2 -> Server 1.

Configuration
N5K1:
vlan 10,20
!
spanning-tree pathcost method long

spanning-tree vlan 20 priority 8192


!
interface port-channel1
switchport mode trunk
spanning-tree vlan 10,20 cost 99999
speed 10000
!
interface Ethernet1/1
switchport access vlan 10
speed 1000
!
interface Ethernet1/2
switchport access vlan 20
speed 1000
!
interface Ethernet1/3
switchport mode trunk
channel-group 1
!
interface Ethernet1/4
switchport mode trunk
channel-group 1
!
interface Ethernet1/5
switchport mode trunk
channel-group 1
!
interface Ethernet1/6
switchport mode trunk
spanning-tree vlan 10 cost 99999
!
interface Ethernet1/7
switchport mode trunk
spanning-tree vlan 10 cost 99999
!
interface Ethernet1/8
switchport mode trunk
spanning-tree vlan 20 cost 99999
!
interface Ethernet1/9
switchport mode trunk
spanning-tree vlan 20 cost 99999
N5K2:
vlan 10,20
!
spanning-tree pathcost method long

spanning-tree vlan 20 priority 4096


!
interface port-channel1
switchport mode trunk
speed 10000
!
interface Ethernet1/1
switchport access vlan 20
speed 1000
!
interface Ethernet1/2
switchport access vlan 10
speed 1000
!
interface Ethernet1/3
switchport mode trunk
channel-group 1
!
interface Ethernet1/4
switchport mode trunk
channel-group 1
!
interface Ethernet1/5
switchport mode trunk
channel-group 1
!
interface Ethernet1/6
switchport mode trunk
!
interface Ethernet1/7
switchport mode trunk
!
interface Ethernet1/8
switchport mode trunk
!
interface Ethernet1/9
switchport mode trunk
N7K1:
vlan 10,20
!
spanning-tree pathcost method long
spanning-tree vlan 10 priority 4096
!
interface Ethernet1/1
switchport
switchport mode trunk

no shutdown
!
interface Ethernet1/2
switchport
switchport mode trunk
no shutdown
!
interface Ethernet2/1
switchport mode trunk
no shutdown
!
interface Ethernet2/2
switchport mode trunk
no shutdown
!
interface Ethernet2/3
switchport mode trunk
no shutdown
!
interface Ethernet2/4
switchport mode trunk
no shutdown
!
interface Ethernet2/5
switchport mode trunk
spanning-tree vlan 20 cost 99999
no shutdown
!
interface Ethernet2/6
switchport mode trunk
spanning-tree vlan 20 cost 99999
no shutdown
N7K2:

vlan 10,20
!
spanning-tree pathcost method long
spanning-tree vlan 10 priority 8192
!
interface Ethernet1/1
switchport
switchport mode trunk
no shutdown
!
interface Ethernet1/2
switchport

switchport mode trunk


no shutdown
!
interface Ethernet2/1
switchport mode trunk
no shutdown
!
interface Ethernet2/2
switchport mode trunk
no shutdown
!
interface Ethernet2/3
switchport mode trunk
no shutdown
!
interface Ethernet2/4
switchport mode trunk
no shutdown
!
interface Ethernet2/5
switchport mode trunk
no shutdown
!
interface Ethernet2/6
switchport mode trunk
no shutdown

Verification
NX-OS runs Rapid Per-VLAN Spanning-Tree Protocol by default. This means that
for each VLAN that is created, a separate instance of STP is created, with each of
these running the 802.1w RSTP algorithm. Beyond this, the default behavior of STP
on NX-OS is essentially identical to that of Catalyst IOS.
For the purposes of traffic engineering, the Root Bridge election can be modified on
a per-VLAN basis by changing the STP Bridge-ID Priority value (lower is preferred),
and the Root Port election can be modified on a per-port per-VLAN basis by
changing either the STP Port Cost or STP Port Priority. Changing the Port Cost is
the more common modification, as the Port Priority only comes into play when you
are choosing between multiple links with the same Root Path Cost, and the links
connect to the same upstream switch.
Note that in this example the range of STP cost values was increased from a 16-bit
field to a 32-bit field (the long pathcost method), as newer higher-speed links such

as 10GigE, 40GigE, 100GigE, and beyond start to look the same from a cost point
of view when the default short pathcost method is used. Also note that this
command is applicable only when running Rapid PVST on NX-OS, as Multiple
Spanning-Tree (MST) always uses the longer pathcost length.
Verification of this task can be performed by viewing the Root Bridge and Root Port
election on a per-switch basis, or by viewing the MAC address table, as the STP
topology ultimately controls which interfaces can participate in MAC address
learning. Below we see that for VLAN 10, N7K1 is elected the Root Bridge. This
implies that all of its VLAN 10 links will be Designated ports in the Forwarding state.
N7K1-1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID

Bridge ID

Interface

Priority

4106

Address

68bd.abd7.6041 This bridge is the root

Hello Time

Priority

4106

Address

68bd.abd7.6041

Hello Time

sec

Max Age 20 sec

Forward Delay 15 sec

(priority 4096 sys-id-ext 10)

sec

Max Age 20 sec

Role Sts Cost

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------- Eth1/1 Desg FWD


2000

128.129

P2p Eth1/2 Desg FWD

2000

128.130

P2p Eth2/1 Desg FWD

2000

128.257

P2p Eth2/2 Desg FWD

2000

128.258

P2p Eth2/3 Desg FWD

2000

128.259

P2p Eth2/4 Desg FWD

2000

128.260

P2p Eth2/5 Desg FWD

2000

128.261

P2p Eth2/6 Desg FWD

2000

128.262

P2p

MAC addresses for VLAN 10 are being learned in ports Eth2/5 and Eth1/1, which
implies that N5K2 and N7K2 on the other end of these links, respectively, have
chosen those ports as their Root Ports.
N7K1-1# show mac address-table dynamic vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN

MAC Address

Type

age

Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+-----------------* 10

000c.29d0.a588

dynamic

30

F Eth2/5

* 10

000c.29f8.57ec

dynamic

30

F Eth1/1

Per the output below, N7K2 chose E1/1 as the Root Port to reach N7K1. Although
all ports have the same cost of 2000, E1/1 has the lowest Port ID (port priority and
port number) on the other end of the link.
N7K2-1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID

Bridge ID

Interface

Priority

4106

Address

68bd.abd7.6041

Cost

2000

Port

129 (Ethernet1/1)

Hello Time

Priority

8202

Address

0026.980c.2141

Hello Time

sec

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 8192 sys-id-ext 10)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Eth1/1


128.129

Root FWD 2000


P2p

Eth1/2

Altn BLK 2000

128.130

P2p

Eth2/1

Altn BLK 2000

128.257

P2p

Eth2/2

Altn BLK 2000

128.258

P2p

Eth2/3

Desg FWD 2000

128.259

P2p

Eth2/4

Desg FWD 2000

128.260

P2p

Eth2/5

Desg FWD 2000

128.261

P2p

Eth2/6

Desg FWD 2000

128.262

P2p

Per the view of the CAM table below, we see that N7K2 learns MAC addresses for
VLAN 10 in Eth1/1, its root port, and Eth2/5, the downstream link connecting to
N5K1.

N7K2-1# show mac address-table dynamic vlan 10

Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN

MAC Address

Type

age

Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+-----------------* 10

000c.29d0.a588

dynamic

30

Eth1/1

* 10

000c.29f8.57ec

dynamic

30

Eth2/5

On the next downstream switch, N5K1, we see that it has chosen Eth1/8, a link to
N7K2, as its Root Port. This is because other possible paths to the Root Bridge
have had their cost raised to 99999. The end result is that traffic received from
Server 1 in VLAN 10 going to Server 2 is first forwarded to N7K2, then to N7K1,
then to N5K2, and finally to Server 2.
N5K1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID

Bridge ID

Interface

Priority

4106

Address

68bd.abd7.6041

Cost

4000

Port

136 (Ethernet1/8)

Hello Time

Priority

32778

Address

547f.ee79.137c

Hello Time

sec

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Altn BLK 99999

128.4096 P2p

Eth1/1

Desg FWD 20000

128.129

P2p

Eth1/6

Altn BLK 99999

128.134

P2p

Eth1/7

Altn BLK 99999

128.135

P2p Eth1/8

128.137

P2p

128.136
Eth1/9

Root FWD 2000

P2p
Altn BLK 2000

N5K1# show mac address-table dynamic vlan 10v


Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN

MAC Address

Type

age

Secure NTFY

Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------ *
10

000c.29d0.a588

* 10

000c.29f8.57ec

dynamic

130

dynamic

130

F
F

Eth1/8
F

Eth1/1

Likewise, traffic in VLAN 20 from Server 2 can be verified to follow the path of
N5K1 -> N7K1 -> N7K2 -> N5K2 -> Server 1 by the CAM tables below.
N5K1# show mac address-table dynamic vlan 20
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN

MAC Address

Type

age

Secure NTFY

Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------ *
20

000c.29d0.a57e

* 20

000c.29f8.57f6

dynamic

10

dynamic

10

F
F

Eth1/2
F

Eth1/6

N7K1-1# show mac address-table dynamic vlan 20


Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN

MAC Address

Type

age

Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------ *
20

000c.29d0.a57e

* 20

000c.29f8.57f6

dynamic
dynamic

F
30

F
F

Eth2/3
F

Eth1/1

N7K2-1# show mac address-table dynamic vlan 20


Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN

MAC Address

Type

age

Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------ *
20

000c.29d0.a57e

* 20

000c.29f8.57f6

dynamic
dynamic

30

30

F
F

Eth1/1
F

Eth2/3

N5K2# show mac address-table dynamic vlan 20


Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN

MAC Address

Type

age

Secure NTFY

Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------ *
20
* 20

000c.29d0.a57e
000c.29f8.57f6

dynamic
dynamic

20
20

F
F

Eth1/6
F

Eth1/1

Nexus Technology Labs - Classical Ethernet


Switching
MSTP Traffic Engineering
Task
Configure the links between N7K1, N7K2, N5K1, and N5K2 as port channels and
802.1q trunk links as follows:
Use Port-Channel 1 on the F ports between N7K1 and N7K2.
Use Port-Channel 2 between N7K1 and N5K1.
Use Port-Channel 3 between N7K1 and N5K2.
Use Port-Channel 4 between N7K2 and N5K1.
Use Port-Channel 5 between N7K2 and N5K2.
Use Port-Channel 6 between N5K1 and N5K2.
Disable all other ports.
Create VLANs 10, 20, 30, 40, 50, and 60 on all switches.
Configure Multiple Spanning-Tree Protocol between the switches as follows:
Configure the switches in MST region 100 named MST100.
VLANs 10, 20, and 30 should be in MST instance 1.
VLANs 40, 50, and 60 should be in MST instance 2.
N7K1 should be the root bridge for MST instance 1.
N7K2 should be the root bridge for MST instance 2.
N5K2 should be the CIST root bridge.
Configure MST traffic engineering as follows:
N5K2 should forward traffic through N5K1 to get to N7K2.
N7K2 should forward traffic through N5K2 to get to N7K1.

Configuration
N5K1:
feature lacp
!
vlan 10,20,30,40,50,60
!

spanning-tree mode mst


spanning-tree mst configuration
name MST100
revision 100
instance 1 vlan 10,20,30
instance 2 vlan 40,50,60
!
interface Ethernet1/3 - 5
switchport mode trunk
channel-group 6 mode active
!
interface Ethernet1/6 - 7
switchport mode trunk
channel-group 2 mode active
!
interface Ethernet1/8 - 9
switchport mode trunk
channel-group 3 mode active
N5K2:
feature lacp
!
vlan 10,20,30,40,50,60
!
spanning-tree mode mst
spanning-tree mst 0 priority 4096
spanning-tree mst configuration
name MST100
revision 100
instance 1 vlan 10,20,30
instance 2 vlan 40,50,60
!
interface Ethernet1/3 - 5
switchport mode trunk
channel-group 6 mode active
!
interface Ethernet1/6 - 7
switchport mode trunk
channel-group 5 mode active
!
interface Ethernet1/8 - 9
switchport mode trunk
channel-group 3 mode active
!
interface port-channel3
spanning-tree mst 2 cost 99999
!

interface port-channel5
spanning-tree mst 2 cost 99999
N7K1:
feature lacp
!
vlan 10,20,30,40,50,60
!
spanning-tree mode mst
spanning-tree mst 1 priority 4096
spanning-tree mst configuration
name MST100
revision 100
instance 1 vlan 10,20,30
instance 2 vlan 40,50,60
!
interface Ethernet2/1 - 2
switchport mode trunk
channel-group 1 mode active
!
interface Ethernet2/3 - 4
switchport mode trunk
channel-group 2 mode active
!
interface Ethernet2/5 - 6
switchport mode trunk
channel-group 3 mode active
N7K2:

feature lacp
!
vlan 10,20,30,40,50,60
!
spanning-tree mode mst
spanning-tree mst 2 priority 4096
spanning-tree mst configuration
name MST100
revision 100
instance 1 vlan 10,20,30
instance 2 vlan 40,50,60
!
interface Ethernet2/1 - 2
switchport mode trunk
channel-group 1 mode active
!
interface Ethernet2/3 - 4
switchport mode trunk

channel-group 5 mode active


!
interface Ethernet2/5 - 6
switchport mode trunk
channel-group 4 mode active
!
interface port-channel1
spanning-tree mst 1 cost 99999
!
interface port-channel4
spanning-tree mst 1 cost 99999

Verification
N5K2# show spanning-tree mst 0
##### MST0

vlans mapped:

Bridge

address 000d.eca4.743c

Root

this switch for the CIST

1-9,11-19,21-29,31-39,41-49,51-59,61-4094
priority

4096

(4096 sysid 0)

Regional Root this switch


Operational

hello time 2 , forward delay 15, max age 20, txholdcount 6

Configured

hello time 2 , forward delay 15, max age 20, max hops

Interface

Role Sts Cost

20

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po3

Desg BLK 1000

128.4098 P2p

Po5

Desg FWD 1000

128.4100 P2p

Po6

Desg FWD 660

128.4101 P2p

N7K1-1# show spanning-tree mst 1


##### MST1

vlans mapped:

Bridge

address 68bd.abd7.6041

Root

this switch for MST1

Interface

10,20,30

Role Sts Cost

priority

4097

(4096 sysid 1)

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Desg FWD 1000

128.4096 P2p

Po2

Desg FWD 1000

128.4097 P2p

Po3

Desg FWD 1000

128.4098 P2p

N7K2-1# show spanning-tree mst 1

##### MST1

vlans mapped:

Bridge

address 0026.980c.2141

priority

32769 (32768 sysid 1)

Root

address 68bd.abd7.6041

priority

4097

port

cost

2000

Po5

10,20,30

(4096 sysid 1)
rem hops 18

Interface

Role Sts Cost

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Altn BLK 99999

128.4096 P2p

Po4

Altn BLK 99999

128.4099 P2p Po5

FWD 1000

Root

128.4100 P2p

N7K2-1# show spanning-tree mst 2


##### MST2

vlans mapped:

Bridge

address 0026.980c.2141

Root

this switch for MST2

Interface

40,50,60

Role Sts Cost

priority

4098

(4096 sysid 2)

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Desg FWD 1000

128.4096 P2p

Po4

Desg FWD 1000

128.4099 P2p

Po5

Desg FWD 1000

128.4100 P2p

N5K2# show spanning-tree mst 2

##### MST2

vlans mapped:

Bridge

address 000d.eca4.743c

priority

32770 (32768 sysid 2)

Root

address 0026.980c.2141

priority

4098

port

cost

1660

Interface

40,50,60

Po6

Role Sts Cost

(4096 sysid 2)
rem hops 18

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po3

Altn BLK 99999

128.4098 P2p

Po5

Altn BLK 99999

128.4100 P2p Po6

660

128.4101 P2p

Root FWD

Nexus Technology Labs - Classical Ethernet


Switching
STP Bridge Assurance
Task
Configure the links between N7K1, N7K2, N5K1, and N5K2 as port channels and
802.1q trunk links as follows:
Use Port-Channel 1 on the F ports between N7K1 and N7K2.
Use Port-Channel 2 between N7K1 and N5K1.
Use Port-Channel 3 between N7K1 and N5K2.
Use Port-Channel 4 between N7K2 and N5K1.
Use Port-Channel 5 between N7K2 and N5K2.
Use Port-Channel 6 between N5K1 and N5K2.
Disable all other ports.
Create VLANs 10, 20, 30, 40, 50, and 60 on N7K1 and N7K2.
Create VLANs 10, 20, and 30 on N5K1.
Create VLANs 40, 50, and 60 on N5K2.
Configure N7K1 as the STP root bridge for all VLANs.
Configure all trunk links as STP port type network.

Configuration
N5K1:
feature lacp
!
vlan 10,20,30
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 6 mode active
!
interface Ethernet1/6 - 7
switchport mode trunk

spanning-tree port type network


channel-group 2 mode active
!
interface Ethernet1/8 - 9
switchport mode trunk
spanning-tree port type network
channel-group 3 mode active
N5K2:
feature lacp
!
vlan 40,50,60
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 6 mode active
!
interface Ethernet1/6 - 7
switchport mode trunk
spanning-tree port type network
channel-group 5 mode active
!
interface Ethernet1/8 - 9
switchport mode trunk
spanning-tree port type network
channel-group 3 mode active
N7K1:
feature lacp
!
vlan 10,20,30,40,50,60
!
spanning-tree vlan 10,20,30,40,50,60 priority 4096
!
interface Ethernet2/1 - 2
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet2/3 - 4
switchport mode trunk
spanning-tree port type network
channel-group 2 mode active
!
interface Ethernet2/5 - 6
switchport mode trunk
spanning-tree port type network

channel-group 3 mode active


N7K2:

feature lacp
!
vlan 10,20,30,40,50,60
!
interface Ethernet2/1 - 2
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet2/3 - 4
switchport mode trunk
spanning-tree port type network
channel-group 5 mode active
!
interface Ethernet2/5 - 6
switchport mode trunk
spanning-tree port type network
channel-group 4 mode active

Verification
Spanning-tree Bridge Assurance is an STP enhancement to help prevent against
unidirectional links, and to also automatically prune unneeded VLANs from trunk
links. Like STP Loopguard or UDLD, STP Bridge Assurance uses a keepalive, the
STP BPDU in this case, to make sure that both ends of the link can both send and
receive packets. Unlike UDLD, though, this keepalive is on a per-VLAN basis (or perSTP instance basis in the case of MST). The advantage of using this feature is that
it not only prevents against unidirectional links, but has a behavior similar to VTP
pruning, which means that you do not need to manually edit the trunking allowed list
to remove unneeded VLANs off trunks. Bridge Assurance is configured by setting
the spanning-tree port-type network at the link level.
In the below output on N7K1, we see that VLAN 10 is not forwarding toward N5K2
on Port-Channel 3, and that VLAN 40 is not forwarding toward N5K1 on PortChannel 2. Note that in addition to the port being in the blocking state, it is denoted
as Bridge Assurance Inconsistent. This essentially means that when N7K1 sent a
BPDU keepalive for VLAN 10 out Po3, it did not receive a response back in. The
end result is that the VLAN is blocked, and hence pruned.
N7K1-1# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID

Priority

4106

Address

68bd.abd7.6041

This bridge is the root

Bridge ID

Hello Time

Priority

4106

Address

68bd.abd7.6041

Hello Time

Interface

sec

sec

Max Age 20 sec

Forward Delay 15 sec

(priority 4096 sys-id-ext 10)

Max Age 20 sec

Role Sts Cost

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Desg FWD 1

128.4096 Network P2p

Po2

Desg FWD 1

128.4097 Network P2p

Po3

Desg BKN*1

128.4098 Network P2p *BA_Inc

N7K1-1# show spanning-tree vlan 40

VLAN0040
Spanning tree enabled protocol rstp
Root ID

Priority

4136

Address

68bd.abd7.6041

This bridge is the root

Bridge ID

Hello Time

Priority

4136

Address

68bd.abd7.6041

Hello Time

Interface

sec

sec

Role Sts Cost

Max Age 20 sec

Forward Delay 15 sec

(priority 4096 sys-id-ext 40)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Po1

Desg FWD 1

128.4096 Network P2p

Po2

Desg BKN*1

128.4097 Network P2p *BA_Inc

Po3

Desg FWD 1

128.4098 Network P2p

If a new VLAN is added, Bridge Assurance automatically reacts to it by either


pruning or un-pruning the VLAN, depending on whether the keepalive is received bidirectionally, as seen below.
N7K1-1# conf t
Enter configuration commands, one per line.

End with CNTL/Z. N7K1-1(config)# vlan 100

N7K1-1(config-vlan)# end
N7K1-1# 2013 Mar

2 17:41:56 N7K1-1 last message repeated 1 time

2013 Mar

2 17:41:56 N7K1-1

%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port port-channel1 VLAN0100.


2013 Mar

2 17:41:56 N7K1-1

%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port port-channel2 VLAN0100.


2013 Mar

2 17:41:56 N7K1-1

%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port port-channel3 VLAN0100.

After the VLAN is configured on the other side of a trunk link, BA unblocks it.
N7K2-1# config t
Enter configuration commands, one per line.

End with CNTL/Z. N7K2-1(config)# vlan 100

N7K2-1(config-vlan)# end
N7K2-1#

N7K1-1# 2013 Mar

2 17:42:24 N7K1-1

%STP-2-BRIDGE_ASSURANCE_UNBLOCK: Bridge Assurance unblocking port port-channel1 VLAN0100.

This automatic pruning behavior can also be verified by checking the VLANs
forwarding on the trunk links through the show interface trunk command, as shown
below.
N5K1# show interface trunk | begin Forwarding
Port

STP Forwarding

-------------------------------------------------------------------------------Eth1/3

none

Eth1/4

none

Eth1/5

none

Eth1/6

none

Eth1/7

none

Eth1/8

none

Eth1/9

none Po2

Po3

1,10,20,30

Po6

1,10,20,30

<snip>
N5K2# show interface trunk | begin Forwarding
Port

STP Forwarding

-------------------------------------------------------------------------------Eth1/3

none

Eth1/4

none

Eth1/5

none

Eth1/6

none

Eth1/7

none

Eth1/8

none

Eth1/9

none Po3

Po5

1,40,50,60

Po6

1,40,50,60

<snip>
N7K1-1# show interface trunk | begin Forwarding
Port

STP Forwarding

-------------------------------------------------------------------------------Eth1/1

none

Eth1/2

none

Eth2/1

none

Eth2/2

none

Eth2/3

none

Eth2/4

none

Eth2/5

none

Eth2/6

none Po1

Po2

1,10,20,30

Po3

40,50,60

10,20,30,40,50,60

<snip>
N7K2-1# show interface trunk | begin Forwarding
Port

STP Forwarding

-------------------------------------------------------------------------------Eth1/1

none

Eth1/2

none

Eth2/1

none

Eth2/2

none

Eth2/3

none

Eth2/4

none

Eth2/5

none

Eth2/6

none Po1

Po4

Po5

none

<snip>

1,10,20,30,40,50,60

Nexus Technology Labs - Classical Ethernet


Switching
STP Edge Ports
Task
Configure the links between N7K1, N5K1, and N5K2 as follows:
Configure the ports between N7K1 and N5K1 as 802.1q trunks.
Configure the ports between N7K1 and N5K2 as 802.1q trunks.
Configure these links as STP Network Ports.
Disable all other links between the switches.
Configure VLANs between N7K1, N5K1, and N5K2 as follows:
Create VLAN 10 on these three switches.
Configure N5K1's link to Server 1 as an access port in VLAN 10.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.
Configure N5K2's link to Server 2 as an access port in VLAN 10.
Configure Server 2 with the IP address 10.0.0.2/24 on this link.
Configure N5K1 and N5K2's links to the servers as STP Edge Ports.
When complete, Server 1 & 2 should have IP reachability to each other, and should
not be subject to the STP forwarding delay if one of their links flap.

Configuration
N5K1:
vlan 10
!
interface Ethernet1/1
speed 1000
switchport mode access
switchport access vlan 10
spanning-tree port type edge
!
interface Ethernet1/6 - 7
switchport mode trunk
spanning-tree port type network

N5K2:

vlan 10
!
interface Ethernet1/2
speed 1000
switchport mode access
switchport access vlan 10
spanning-tree port type edge
!
interface Ethernet1/8 - 9
switchport mode trunk
spanning-tree port type network
N7K1:

vlan 10
!
interface Ethernet2/3 - 6
switchport mode trunk
spanning-tree port type network

Verification
Rapid STP Edge Ports are the logical equivalent of the previously Cisco proprietary
PortFast feature. Syntax-wise, the spanning-tree port type edge is the equivalent of
Catalyst IOS spanning-tree portfast. Like PortFast ports, STP Edge Ports are not
subject to the STP Forwarding Delay, because they are the leaf nodes of the tree.
The end result of using Edge Ports is two-fold. The first is that when an edge port
comes up, it immediately transitions to the forwarding state without having to wait
the default 30-second forwarding delay. Second, if the link goes down, a Topology
Change Notification (TCN) is not generated. TCNs in RSTP means that the CAM
table must immediately be flushed. Frequent TCN generation in RSTP could mean a
lot of inefficient flooding in the network, as MAC addresses would constantly have to
be re-learned if TCNs are causing the CAM table to flush. To prevent this, all links
that go to non-STP devices, such as end servers, storage arrays, etc., should run as
STP edge ports.
Verification that a link is running as edge is shown below.

N5K2# show spanning-tree interface e1/2

Vlan

Role Sts Cost

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------VLAN0010

Desg FWD 4

128.130 Edge P2p

The default behavior is that all ports run as STP port type normal, as seen by the
Type P2p from the show spanning-tree output below.
N5K1# show spanning-tree interface e1/1
Vlan

Role Sts Cost

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------VLAN0010

Desg FWD 4

128.129 P2p

When normal ports are activated, RSTP initiates the sync process to attempt rapid
convergence. Because the device on the other end of the link, a server in this case,
is not running RSTP, no sync response is received. The result is that RSTP must fall
back to legacy STP rules and wait for the forward delay to expire before the link can
transition to forwarding state. From the debug output and the timestamps below, we
see that the link is activated at 16:52:44 and begins in the blocking state, transitions
to learning, and then finally to forwarding at 16:53:14, which is 30 seconds after the
link up event.
N5K1# debug spanning-tree event interface e1/1
N5K1# conf t
Enter configuration commands, one per line.

End with CNTL/Z. N5K1(config)# int e1/1

N5K1(config-if)# shut
N5K1(config-if)# no shut
2013 Mar

2 16:52:39 N5K1 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/1 is admin up .

N5K1(config-if)# 2013 Mar

2 16:52:44 N5K1 %ETHPORT-5-SPEED: Interface Ethernet1/1, operational speed changed to 1 G

2013 Mar

2 16:52:44 N5K1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/1, operational duplex mode changed to Full

2013 Mar

2 16:52:44 N5K1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/1, operational Receive Flow Control sta

2013 Mar

2 16:52:44 N5K1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/1, operational Transmit Flow Control st

2013 Mar

2 16:52:44 N5K1 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode access

2013 Mar

2 16:52:44.824150 stp: stp_im_l2_protocol_state_change(341): IM L2 Port Proto UP Notif : port_mode 0x80000

2013 Mar

2 16:52:44.824484 stp: stp_mcs_set_slave_mcec_vlans_multi_port_state(928): Ignore - Not a vPC Master

2013 Mar

2 16:52:44.824514 stp: vb_vlan_shim_set_vlans_multi_port_state(2738): Req (type=9) to PIXM vdc 1, inst 0,

2013 Mar

2 16:52:44 .824700 stp

: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state BLK


, rr_token 0x1f65af msg_size 544

2013 Mar

2 16:52:59.765655 stp: stp_mcs_set_slave_mcec_vlans_multi_port_state(928): Ignore - Not a vPC Master

2013 Mar

2 16:52:59.765701 stp: vb_vlan_shim_set_vlans_multi_port_state(2738): Req (type=9) to PIXM vdc 1, inst 0,

2013 Mar

2 16:52:59 .765901 stp

: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state LRN


, rr_token 0x1f6686 msg_size 544
2013 Mar

2 16:53:14.766632 stp: stp_mcs_set_slave_mcec_vlans_multi_port_state(928): Ignore - Not a vPC Master

2013 Mar

2 16:53:14.766676 stp: vb_vlan_shim_set_vlans_multi_port_state(2738): Req (type=9) to PIXM vdc 1, inst 0,

2013 Mar

2 16:53:14 .772174 stp

: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state FWD


, rr_token 0x1f6741 msg_size 544
2013 Mar

2 16:53:14.772296 stp: stp_send_pvlan_port_state_change_notif[4224]:idx=0:state=4:if_index=Ethernet1/1:pri

When the link is configured as edge, the transition from down to forwarding happens
immediately, as shown below.

N5K1# debug spanning-tree event interface e1/1

N5K1# conf t
Enter configuration commands, one per line.

End with CNTL/Z.

N5K1(config)# int e1/1


N5K1(config-if)# shut N5K1(config-if)# spanning-tree port type edge

Warning: Edge port type (portfast) should only be enabled on ports connected to a single
host. Connecting hubs, concentrators, switches, bridges, etc... to this
interface

when edge port type (portfast) is enabled, can cause temporary bridging loops.

Use with CAUTION

Edge Port Type (Portfast) has been configured on Ethernet1/1 but will only
have effect when the interface is in a non-trunking mode.

N5K1(config-if)# no shut
2013 Mar

2 17:07:44 N5K1 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/1 is admin up .

2013 Mar

2 17:07:49 N5K1 %ETHPORT-5-SPEED: Interface Ethernet1/1, operational speed changed to 1 Gbps

2013 Mar

2 17:07:49 N5K1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/1, operational duplex mode changed to Full

2013 Mar

2 17:07:49 N5K1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/1, operational Receive Flow Control sta

2013 Mar

2 17:07:49 N5K1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/1, operational Transmit Flow Control st

2013 Mar

2 17:07:49.516552 stp: stp_im_l2_protocol_state_change(341): IM L2 Port Proto UP Notif : port_mode 0x80000

2013 Mar

2 17:07:49.516882 stp: stp_mcs_set_slave_mcec_vlans_multi_port_state(928): Ignore - Not a vPC Master

2013 Mar

2 17:07:49.516912 stp: vb_vlan_shim_set_vlans_multi_port_state(2738): Req (type=9) to PIXM vdc 1, inst 0,

2013 Mar

2 17:07:49.517105 stp: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state

2013 Mar

2 17:07:49 N5K1 %ETHPORT-5-IF_UP

: Interface Ethernet1/1 is up in mode access


2013 Mar

2 17:07:49.525471 stp: stp_mcs_set_slave_mcec_vlans_multi_port_state(928): Ignore - Not a vPC Master

2013 Mar

2 17:07:49.525540 stp: vb_vlan_shim_set_vlans_multi_port_state(2738): Req (type=9) to PIXM vdc 1, inst 0,

2013 Mar

2 17:07:49.525715 stp

: Success: pixm_send_set_mult_cbl_vlans_for_multiple_ports, num ports 1 VDC 1, state FWD


, rr_token 0x1fb7db msg_size 544

Nexus Technology Labs - Overlay Transport


Virtualization (OTV)
Overlay Transport Virtualization (OTV)
Task
Configure basic layer 2 reachability as follows:
Configure Server 1s link to N5K1 with the IP address 10.0.0.1/24.
Configure Server 2s link to N5K2 with the IP address 10.0.0.2/24.
Configure N5K1s link to Server 1 in VLAN 10.
Configure N5K2s link to Server 2 in VLAN 10.
Configure VLAN 10 trunking on the links between N7K1 and N5K1, and
between N7K2 and N5K2.
Configure N7K1 and N7K2 as the STP root bridges for VLAN 10.
Disable all other unused ports on all four switches.
Configure Overlay Transport Virtualization (OTV) between N7K1 and N7K2 to tunnel
traffic between Server 1 and Server 2 as follows:
Enable the OTV feature on N7K1 and N7K2.
Create VLAN 999 on N7K1 and N7K2 and configure it as the OTV Site
VLAN.
Configure N7K1 with the OTV Site Identifier 0x101.
Configure N7K2 with the OTV Site Identifier 0x102.
Configure the first M1 port connecting N7K1 and N7K2 as a native layer 3
routed interface using the addresses 169.254.0.71/24 and 169.254.0.72/24,
respectively. This will be the OTV Join Interface.
Enable IGMPv3 on the OTV Join Interface.
Configure interface Overlay 1 on both N7K1 and N7K2. Use the routed
interface between them as the OTV Join Interface, 224.1.1.1 as the OTV
Control Group, 232.1.1.0/24 as the OTV Data Group range, and VLAN 10 as
an Extend VLAN.
When complete, N7K1 and N7K2 should form an OTV IS-IS adjacency, and traffic
between Server 1 and Server 2 should be bridged over the OTV tunnel.

Configuration
N5K1:
vlan 10
!
interface Ethernet1/1
switchport access vlan 10
speed 1000
!
interface Ethernet1/6 - 7
switchport mode trunk
N5K2:
vlan 10
!
interface Ethernet1/2
switchport access vlan 10
speed 1000
!
interface Ethernet1/6 - 7
switchport mode trunk
N7K1:
feature otv
!
vlan 10
spanning-tree vlan 10 priority 4096
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
otv site-identifier 0x101
!
interface Overlay1
otv join-interface Ethernet1/1
otv control-group 224.1.1.1
otv data-group 232.1.1.0/24
otv extend-vlan 10
no shutdown
!
interface Ethernet1/1
ip address 169.254.0.71/24
ip igmp version 3
no shutdown

!
interface Ethernet2/3 - 4
switchport mode trunk
N7K2:

feature otv
!
vlan 10
spanning-tree vlan 10 priority 4096
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
otv site-identifier 0x102
!
interface Overlay1
otv join-interface Ethernet1/1
otv control-group 224.1.1.1
otv data-group 232.1.1.0/24
otv extend-vlan 10
no shutdown
!
interface Ethernet1/1
ip address 169.254.0.72/24
ip igmp version 3
no shutdown
!
interface Ethernet2/3 - 4
switchport mode trunk

Verification
Overlay Transport Virtualization (OTV) is an Ethernet over IP tunnel that allows you
to span the broadcast domain of two Data Center sites together over an IP
transport. In essence, OTV accomplishes the same goal as other L2 tunneling
protocols such as L2TPv3, Any Transport over MPLS (AToM), or Virtual Private
LAN Services (VPLS), but it includes special enhancements specific to Data Center
Interconnect (DCI) environments. The first of these enhancements is that OTV is
transport agnostic.
In this example, our OTV Edge Devices N7K1 and N7K2 are directly connected over
a point-to-point layer 3 routed interface. However, OTV can run over any transport

as long as both unicast and multicast IP connectivity exist. This means that OTV
can run over Dark Fibre, MPLS L2VPN/L3VPN, DMVPN, etc., as long as
reachability is there. In later examples, we will see how to lessen this connectivity
requirement so that only unicast IP reachability is required, as opposed to both
unicast and multicast.
The layer 3 routed interface used to form the OTV tunnel is called the Join Interface.
Note that as of the release used in this example, the Join Interface cannot be an SVI
or a Loopback; it must be a native layer 3 routed physical interface, subinterface, or
port channel. This link is essentially what you would think of as the tunnel source
in a normal GRE tunnel configuration. The logical interface where traffic is OTV
encapsulated is called the Overlay Interface, which is analogous to the interface
tunnel in normal GRE tunnel configurations. A concise verification of the state of
both the Join and Overlay interfaces is shown below.
N7K1# show otv

OTV Overlay Information


Site Identifier 0000.0000.0101

Overlay interface Overlay1

VPN name

: Overlay1

VPN state

: UP

Extended vlans

: 10 (Total:1)

Control group

: 224.1.1.1

Data group range(s) : 232.1.1.0/24


Join interface(s)

: Eth1/1 (169.254.0.71)

Site vlan

: 999 (up)

AED-Capable

: Yes

Capability

: Multicast-Reachable

The above output shows us both the Unicast and Multicast transport addresses
used, as well as other pertinent information such as the VLANs extended (bridged)
over the tunnel, whether the local Site VLAN is up (which it must be), and whether
the edge router has elected itself as Authoritative (which it must be). Another
important output from above is the Site Identifier, which must be the same for Edge
Devices in the same DC site, and different between different sites (as is this case).
The specific need for the Site VLAN and Site Identifier will be explored in more
detail in later scenarios.
For the transport addresses, Unicast reachability is simple to verify. If the OTV Edge
Devices can ping the IP address on their Join Interfaces, thats basically all there is

to it. For multicast reachability, we would, of course, require a multicast-capable


transport, such as a point-to-point link like in this case, or MPLS L3VPN that has
MDT service enabled. Note that the multicast transport must be both ASM (*,G) and
SSM (S,G) capable, because the discovery of other OTV edge devices uses shared
trees (*,G), whereas actual multicast data plane over the tunnel uses shortest path
trees (S,G). Because SSM is used for certain flows, IGMPv3 support must be
enabled on the Join Interface as well as the actual PIM router on the other side of
the link (if applicable).
Without doing a verification in the DCI transit network itself, multicast reachability
can be quickly verified from the OTV edge routers themselves simply by checking
whether the OTV IS-IS adjacencies have formed. IS-IS traffic is encapsulated in the
OTV control-group multicast, which is the (*,G) feed. This means that if IS-IS is up,
multicast likewise is up. This verification is shown below.
N7K1# show ip mroute
IP Multicast Routing Table for VRF "default"
(*, 224.1.1.1/32)
, uptime: 00:02:22, otv ip

Incoming interface: Ethernet1/1

, RPF nbr: 169.254.0.71 Outgoing interface list


: (count: 1) Overlay1
, uptime: 00:02:22, otv
N7K1# show otv isis adjacency
OTV-IS-IS process: default VPN: Overlay1
OTV-IS-IS adjacency database:
System ID

SNPA

Level

State

Hold Time

Interface Site-ID

N7K2

0026.980c.2141

UP

00:00:25

Overlay1 0000.0000.0102

Unlike FabricPath, which uses IS-IS simply to establish a shortest path tree between
the FabricPath Spine and Leaf switches, the OTV IS-IS process is used to advertise
reachability information about the end hosts. Before traffic can actually flow over the
tunnel, the end devices MAC addresses must be advertised into IS-IS, and can be
verified as follows.

N7K1# show otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address

Metric

Uptime

Owner

Next-hop(s)

---- --------------

------

--------

---------

-----------

10 000c.2922.167d

10 000c.29bc.5bea

42

00:26:16

00:26:17

overlay

site

N7K2

Ethernet2/3

MAC addresses local to the site should be reachable out a physical link or portchannel, whereas MAC addresses in other sites will be reachable out the overlay
tunnel interface. The detailed advertisements in the IS-IS database are shown here.

N7K1# show otv isis database detail

OTV-IS-IS Process: default LSP database VPN: Overlay1

OTV-IS-IS Level-1 Link State Database


LSPID

Seq Number

.00-00

0x00000006

Checksum

0x2014

Lifetime

751

0/0/0/1

Instance

0x00000003

Area Address

00

NLPID

0xCC 0x8E

Hostname

N7K2

Length : 4

Extended IS

N7K1.01

Metric : 40

MAC Address

Vlan

: 10 : Metric

: 1

Vlan

: 10 : Metric

: 1

: 000c.2922.167d

Digest Offset :
.00-00

A/P/O/T N7K2

0 N7K1

* 0x00000006

0x8E3B

790

0/0/0/1

Instance

0x00000006

Area Address

00

NLPID

0xCC 0x8E

Hostname

N7K1

Length : 4

Extended IS

N7K1.01

Metric : 40

: 000c.29bc.5bea

MAC Address

Digest Offset :

N7K1.01-00

* 0x00000003

0xBC4E

645

Instance

0x00000003

Extended IS

N7K2.00

Metric : 0

Extended IS

N7K1.00

Metric : 0

Digest Offset :

0/0/0/1

The database output above indicates that N7K1 is advertising Server 1s MAC
address 000c.29bc.5bea in VLAN 10, whereas N7K2 is advertising Server 2s MAC
address 000c.2922.167d in VLAN 10.
Another indication that the data plane is working in OTV is to check the ARP and
ICMP ND cache on the OTV Edge Devices. As a flooding optimization, the AEDs
cache ARP and ND responses that they have learned so that further ARP/ND
requests do not need to flood over the overlay tunnel. This can be seen below.
N7K1# show otv arp-nd-cache

OTV ARP/ND L3->L2 Address Mapping Cache

Overlay Interface Overlay1

VLAN

MAC Address

Layer-3 Address

Age

Expires In

10

000c.2922.167d

10.0.0.2

00:00:01

00:07:58

From the above output, we see that N7K1 knows about Server 2 with the IP address
10.0.0.2 across the Overlay1 interface. If Server 1 were to clear its ARP cache and
send a new ARP request for Server 2, we should see N7K1 respond locally instead
of forwarding the request over the Overlay, as shown here.
N7K1# debug otv arp-nd

N7K1#
otv: IPv4 ARP Request packet received from source 10.0.0.1 on interface Ethernet2/3

otv: Found ORIB mac route next hop Ethernet2/3 for target IP 10.0.0.2 with source MAC 0010-000c.29bc.5bea , hence se

otv: Send proxy ARP-Reply to 10.0.0.1 (0010-000c.29bc.5bea) for target IP 10.0.0.2 with target MAC 000c.2922.167d on

An important point to remember about OTV and data plane traffic is that the OTV
tunnel adds 42 bytes of overhead and does not support fragmentation. This means
that if the DCI does not support Jumbo Frames, it is possible that traffic flows that
work over other interconnects such as dark fiber wont work over OTV. From the
end host perspective, this can be seen here.

ICMP PINGs from Server 1 to Server 2 with the Dont Fragment bit set work up to
1430 bytes. With the ICMP header of 8 bytes and IP header of 20 bytes, it means
that the entire IP packet length is 1458. An additional 42 bytes of OTV bring us to a
total length of 1500 bytes, which is the default MTU. The next ping with an ICMP
payload of 1431 is dropped, as the OTV edge device silent discards the packet
instead of fragmenting it. To increase the payload size that the tunnel supports, the
DCI links (the Join Interface and everything else in the transit path) must support
Jumbo Frames. On Nexus 7K this is simply one command at the interface level,

shown here.
N7K1# conf t

Enter configuration commands, one per line.

End with CNTL/Z.

N7K1(config)# int e1/1


N7K1(config-if)# mtu 9216

Now the servers can exchange packets up to 1500 bytes, as shown above. Jumbo
Frames above this size are still dropped, but now due to the normal layer 2
infrastructure such as the 5Ks and the Server NIC cards themselves not having
Jumbo Frames enabled, not the OTV tunnel or the DCI itself.

Nexus Technology Labs - Overlay Transport


Virtualization (OTV)
OTV Authentication
Task
Configure basic layer 2 reachability as follows:
Configure Server 1s link to N5K1 with the IP address 10.0.0.1/24.
Configure Server 2s link to N5K2 with the IP address 10.0.0.2/24.
Configure N5K1s link to Server 1 in VLAN 10.
Configure N5K2s link to Server 2 in VLAN 10.
Configure VLAN 10 trunking on the links between N7K1 and N5K1, and
between N7K2 and N5K2.
Configure N7K1 and N7K2 as the STP root bridges for VLAN 10.
Disable all other unused ports on all four switches.
Configure Overlay Transport Virtualization (OTV) between N7K1 and N7K2 to tunnel
traffic between Server 1 & Server 2 as follows:
Enable the OTV feature on N7K1 and N7K2.
Create VLAN 999 on N7K1 and N7K2 and configure it as the OTV Site
VLAN.
Configure N7K1 with the OTV Site Identifier 0x101.
Configure N7K2 with the OTV Site Identifier 0x102.
Configure the first M1 port connecting N7K1 and N7K2 as a native layer 3
routed interface using the addresses 169.254.0.71/24 and 169.254.0.72/24,
respectively. This will be the OTV Join Interface.
Enable IGMPv3 on the OTV Join Interface.
Configure interface Overlay 1 on both N7K1 and N7K2. Use the routed
interface between them as the OTV Join Interface, 224.1.1.1 as the OTV
Control Group, 232.1.1.0/24 as the OTV Data Group range, and VLAN 10 as
an Extend VLAN.
Configure OTV Authentication as follows:
Authenticate the OTV IS-IS adjacency between N7K1 and N7K2 using an
MD5 hash of the key-string ADJACENCY.

Authenticate the OTV IS-IS LSP advertisements between N7K1 and N7K2
using an MD5 hash of the key-string UPDATES.
When complete, N7K1 and N7K2 should form an OTV IS-IS adjacency, and traffic
between Server 1 and Server 2 should be bridged over the OTV tunnel.

Configuration
N5K1:
vlan 10
!
interface Ethernet1/1
switchport access vlan 10
speed 1000
!
interface Ethernet1/6 - 7
switchport mode trunk
N5K2:
vlan 10
!
interface Ethernet1/2
switchport access vlan 10
speed 1000
!
interface Ethernet1/6 - 7
switchport mode trunk

N7K1:
feature otv
!
vlan 10
spanning-tree vlan 10 priority 4096
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
otv site-identifier 0x101
!
key chain OTV_ISIS_ADJACENCY
key 1
key-string 0 ADJACENCY
!
key chain OTV_ISIS_LSP
key 1

key-string 0 UPDATES
!
interface Overlay1
otv isis authentication-type md5
otv isis authentication key-chain OTV_ISIS_ADJACENCY
otv join-interface Ethernet1/1
otv control-group 224.1.1.1
otv data-group 232.1.1.0/24
otv extend-vlan 10
no shutdown
!
otv-isis default
vpn Overlay1
authentication-type md5
authentication key-chain OTV_ISIS_LSP
!
interface Ethernet1/1
ip address 169.254.0.71/24
ip igmp version 3
no shutdown
!
interface Ethernet2/3 - 4
switchport mode trunk
N7K2:

feature otv
!
vlan 10
spanning-tree vlan 10 priority 4096
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
otv site-identifier 0x102
!
key chain OTV_ISIS_ADJACENCY
key 1
key-string 0 ADJACENCY
!
key chain OTV_ISIS_LSP
key 1
key-string 0 UPDATES
!
interface Overlay1
otv isis authentication-type md5

otv isis authentication key-chain OTV_ISIS_ADJACENCY


otv join-interface Ethernet1/1
otv control-group 224.1.1.1
otv data-group 232.1.1.0/24
otv extend-vlan 10
no shutdown
!
otv-isis default
vpn Overlay1
authentication-type md5
authentication key-chain OTV_ISIS_LSP
!
interface Ethernet1/1
ip address 169.254.0.72/24
ip igmp version 3
no shutdown
!
interface Ethernet2/3 - 4
switchport mode trunk

Verification
OTV uses IS-IS in the control plane behind the scenes to advertise MAC address
reachability between different DC sites. This means that many of the features native
to IS-IS are also inherently native to OTV, such as authentication. Specifically, IS-IS
supports two different types of authentication, authentication of the IS-IS adjacency,
and authentication of the Link State Packet (LSP) exchange.
The configuration of OTV IS-IS adjacency authentication is very similar to what you
would see in regular IOS or in NX-OS for the regular IPv4/IPv6 IS-IS process. A key
chain is defined globally and then applied at the link level where the adjacency is
occurring. In our case, the OTV IS-IS adjacency forms on the logical Overlay1
interface, as shown below.

N7K1# show otv adjacency

Overlay Adjacency database


Overlay-Interface Overlay1
:
Hostname

System-ID

Dest Addr

N7K2

0026.980c.2141 169.254.0.72

Up Time

State

04:11:11

UP

The authentication configuration at the link level can be verified as follows.


N7K1# show otv isis interface overlay 1
OTV-IS-IS process: default VPN: Overlay1
Overlay1, Interface status: protocol-up/link-up/admin-up
IP address: none
IPv6 address: none
IPv6 link-local address: none
Index: 0x0001, Local Circuit ID: 0x01, Circuit Type: L1
Level1
Adjacency server (local/remote) : disabled / none
Adjacency server capability : multicast Authentication type is MD5
Authentication keychain is OTV_ISIS_ADJACENCY
Authentication check specified

LSP interval: 33 ms, MTU: 1400


Level

Metric

CSNP

40

10

Level
1

Adjs
1

Next CSNP

AdjsUp Pri
1

Hello

00:00:08

64

Circuit ID

N7K1.01

Multi

Next IIH

00:00:01

Since
* 04:12:58

In the case of an authentication failure, the IS-IS adjacency would fail to form. If
authentication is enabled, as verified above, and the adjacency is forming, we can
assume that the authentication is successful.
Troubleshooting a mismatch of authentication can be seen below, where N7K1
modifies its key chain so that the password is not the same as the remote OTV edge
device, and the end result is that the IS-IS adjacency is torn down.
N7K1# config t
Enter configuration commands, one per line.
N7K1(config)# key chain OTV_ISIS_ADJACENCY
N7K1(config-keychain)# key 1

End with CNTL/Z.

N7K1(config-keychain-key)# key-string WRONGPASSWORD


N7K1(config-keychain-key)# end
N7K1#
N7K1# show otv isis adjacency
OTV-IS-IS process: default VPN: Overlay1
OTV-IS-IS adjacency database: System ID
Interface Site-ID N7K2

Level State

SNPA

0026.980c.2141

1 UP

Hold Time

00:00:10

Overlay1 0000.0000.0102
N7K1# 2013 Apr 19 21:14:21 N7K1 %ISIS_OTV-5-ADJCHANGE:

isis_otv-default [2770]

LAN adj L1 N7K2 over Overlay1 - DOWN (Hold timer expired)


on MT-0

N7K1# show otv isis adjacency


OTV-IS-IS process: default VPN: Overlay1
OTV-IS-IS adjacency database: System ID
Interface Site-ID N7K2

SNPA

0026.980c.2141

Level State
1 LOST

Hold Time

00:05:19

Overlay1 0000.0000.0102

By debugging the OTV IS-IS process, we can see that the authentication
configuration is the reason that the adjacency is not forming.
N7K1# debug otv isis iih
<snip> isis_otv: default [2770] Receive L1 LAN IIH over Overlay1 from N7K2 (0026.980c.2141)
len 1397 prio 7
isis_otv: default [2770] Receive link cap tlv, site cap sub-tlv, len 9 isis_otv: default [2770]
LAN IIH authentication failure on Overlay1 from 0026.980c.2141

<snip>

LSP authentication, like in regular IOS, is configured under the global process,
which in our case is the global OTV IS-IS process. The below output shows us that
LSP authentication is enabled.
N7K1# show otv isis

ISIS process : default


VPN: Overlay1
System ID : 68bd.abd7.6041
SAP : 439

IS-Type : L1

Queue Handle : 11

Maximum LSP MTU: 1392


Graceful Restart enabled. State: Inactive
Last graceful restart status : none
Metric-style : advertise(wide), accept(narrow, wide)
Area address(es) :
00

Process is up and running


VPN ID: 214
Incremental update routes during SPF run
Stale routes during non-graceful controlled restart
Interfaces supported by OTV-IS-IS :
Overlay1
Level 1 Authentication type is MD5
Authentication keychain is OTV_ISIS_LSP

Authentication check is specified


Address family IPv4 unicast :
Number of interface : 1
Adjacency check disabled
Distance : 115
Address family IPv6 unicast :
Number of interface : 1
Adjacency check disabled
Distance : 115
Address family MAC unicast :
Number of interface : 1
Adjacency check disabled
Distance : 115
L1 Next SPF: Inactive

A problem in LSP authentication is a little harder to troubleshoot, because a


mismatch in authentication does not prevent the adjacency from forming, but it will
prevent traffic in the data plane from actually forwarding. To illustrate this, N7K1
changes its LSP authentication key chain to be incorrect, as shown below. Note that
before changing the authentication, N7K1 has OTV routes installed via the overlay.
N7K1# show otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address
---- -------------10 000c.2922.167d

10 000c.29bc.5bea

Metric

Uptime

-----42

-------01:26:12

01:34:46

Owner

Next-hop(s)

--------overlay

----------N7K2

site

Ethernet2/3

Enter configuration commands, one per line.

End with CNTL/Z.

N7K1# config t

N7K1(config)# key chain OTV_ISIS_LSP


N7K1(config-keychain)# key 1
N7K1(config-keychain-key)# key-string 0 WRONGPASSWORD

N7K1(config-keychain-key)# end
N7K1#

Now that the authentication is mismatched, we try to send traffic between the
servers.

The result is a little different than expected, because even though the OTV AEDs
dont match their authentication, traffic is still allowed to pass. In reality, whats
happening is that because the topology has not changed at all, and neither of the
AEDs has had to do IS-IS LSP flooding, they dont yet know that a mismatch as
occurred. Traffic will not begin to drop until a topology event occurs that. To illustrate
this, Server 1 changes the MAC address of its NIC card, which will cause a new
OTV IS-IS advertisement.

After Server 1 generates traffic to Server 2, the routing table of the AEDs is as
follows.

N7K1# show otv route

OTV Unicast MAC Routing Table For Overlay1


VLAN MAC-Address

Metric

Uptime

Owner Next-hop(s)

---- --------------

------

--------

---------

00:00:13

-----------

10 0000.0000.0001

site Ethernet2/3

10 000c.29bc.5bea

01:53:08

site Ethernet2/3

N7K2# show otv route

OTV Unicast MAC Routing Table For Overlay1


VLAN MAC-Address

Metric

Uptime

Owner Next-hop(s)

---- --------------

------

--------

---------

01:56:46

site Ethernet2/3

10 000c.2922.167d

-----------

Note that the AEDs only know about MAC addresses local to their site, because the
Overlay1 interface is not listed as a next-hop for any MAC address. Additionally, we
see that on N7K1 the new MAC address of Server 1 was learned. A debug of the
OTV IS-IS process will now tell us what the problem is.
N7K1# debug otv isis lsp flooding
isis_otv: default [2770] Receive Overlay1 L1 LSP
0026.980c.2141.00-00 Seq 0x0000003A Chksum 0x705F Lifetime 1199 over Overlay1 from 0026.980c.2141 prio 7
isis_otv: default [2770] LSP authentication failure

This output indicates that N7K2 is trying to send N7K1 an IS-IS update over the
OTV tunnel, but the authentication is failing. Note, however, that it does not stop the
adjacency from forming.
N7K1# show otv isis adjacency

OTV-IS-IS process: default VPN: Overlay1


OTV-IS-IS adjacency database:
System ID

SNPA

Level

State

Hold Time

Interface Site-ID

0026.980c.2141

0026.980c.2141

UP

00:00:30

Overlay1 0000.0000.0102

Nexus Technology Labs - Overlay Transport


Virtualization (OTV)
OTV Adjacency Server
Task
Configure basic layer 2 reachability as follows:
Configure Server 1s link to N5K1 with the IP address 10.0.0.1/24.
Configure Server 2s link to N5K2 with the IP address 10.0.0.2/24.
Configure N5K1s link to Server 1 in VLAN 10.
Configure N5K2s link to Server 2 in VLAN 10.
Configure VLAN 10 trunking on the links between N7K1 and N5K1, and
between N7K2 and N5K2.
Configure N7K1 and N7K2 as the STP root bridges for VLAN 10.
Disable all other unused ports on all four switches.
Configure Overlay Transport Virtualization (OTV) between N7K1 and N7K2 to tunnel
traffic between Server 1 and Server 2 as follows:
Enable the OTV feature on N7K1 and N7K2.
Create VLAN 999 on N7K1 and N7K2 and configure it as the OTV Site
VLAN.
Configure N7K1 with the OTV Site Identifier 0x101.
Configure N7K2 with the OTV Site Identifier 0x102.
Configure the first M1 port connecting N7K1 and N7K2 as a native layer 3
routed interface using the addresses 169.254.0.71/24 and 169.254.0.72/24,
respectively. This will be the OTV Join Interface.
Configure interface Overlay 1 on both N7K1 and N7K2, using the routed
interface between them as the OTV Join Interface.
Configure VLAN 10 as an Extend VLAN on Overlay 1.
Instead of configuring multicast control and data groups, configure N7K1 as
the OTV Adjacency Server.
When complete, N7K1 and N7K2 should form an OTV IS-IS adjacency, and traffic
between Server 1 and Server 2 should be bridged over the OTV tunnel.

Configuration
N5K1:
vlan 10
!
interface Ethernet1/1
switchport access vlan 10
speed 1000
!
interface Ethernet1/6 - 7
switchport mode trunk
N5K2:
vlan 10
!
interface Ethernet1/2
switchport access vlan 10
speed 1000
!
interface Ethernet1/6 - 7
switchport mode trunk
N7K1:
feature otv
!
vlan 10
spanning-tree vlan 10 priority 4096
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
otv site-identifier 0x101
!
interface Overlay1
otv join-interface Ethernet1/1
otv adjacency-server unicast-only
otv extend-vlan 10
no shutdown
!
interface Ethernet1/1
ip address 169.254.0.71/24
no shutdown
!
interface Ethernet2/3 - 4

switchport mode trunk


N7K2:

feature otv
!
vlan 10
spanning-tree vlan 10 priority 4096
!
vlan 999
name OTV_SITE_VLAN
!
otv site-vlan 999
otv site-identifier 0x102
!
interface Overlay1
otv join-interface Ethernet1/1
otv use-adjacency-server 169.254.0.71 unicast-only
otv extend-vlan 10
no shutdown
!
interface Ethernet1/1
ip address 169.254.0.72/24
no shutdown
!
interface Ethernet2/3 - 4
switchport mode trunk

Verification
With the default operation, OTV requires that the Data Center Interconnect (DCI)
transit network between the DC sites be multicast capable, both for Any Source
Multicast (ASM) and for Source Specific Multicast (SSM). This can potentially limit
the deployment options for OTV if the DCI transit network doesn't natively support
multicast, such as the Internet or a DMVPN tunnel, for example. To alleviate these
design constraints, newer versions of the OTV implementation introduce the OTV
Adjacency Server feature.
The OTV Adjacency Server is one or more OTV Authoritative Edge Devices (AEDs)
that can be used to discover all other endpoints of the OTV tunnel without the need
for multicast. Without the Adjacency Server, all AEDs join the ASM (*,G) OTV
control group. This group is used to discover the other AEDs in the network,
because all AEDs join the same PIM shared tree. In addition to endpoint discovery,
the control group is used to transmit certain control plane packets such as the IS-IS

routing protocol traffic. To see the Adjacency Server in action, first lets review how
the control and data planes of OTV are built in the default design.
Below we see N7K1 configured with both an OTV control group and data group.
Additionally, the interface facing toward the DCI (the OTV Join Interface) is running
IGMPv3.
N7K1# show run interface overlay 1

!Command: show running-config interface Overlay1


!Time: Thu Apr 25 23:53:10 2013

version 6.0(2)

interface Overlay1
otv join-interface Ethernet1/1 otv control-group 224.1.1.1
otv data-group 232.1.1.0/24
otv extend-vlan 10
no shutdown
N7K1# show running-config interface Ethernet1/1

!Command: show running-config interface Ethernet1/1


!Time: Fri Apr 26 00:15:33 2013

version 6.0(2)

interface Ethernet1/1
ip address 169.254.0.71/24 ip igmp version 3

no shutdown

Next, both unicast and multicast connectivity is verified over the DCI. Server 1
sends unicast ICMP PINGs to Server 2, as well as a traceroute. The PINGs test
basic unicast connectivity, and the traceroute shows that they are on the same layer
2 segment.
To use the iPerf app for sending and receiving multicast traffic, the
Windows routing table must know which link to send the multicast
UDP traffic out and where to send the IGMP Report (Join) messages
out. This is what the static multicast route is accomplishing. This is
only needed here because there are multiple NICs on the Virtual
Machines, and we need to make sure these messages go out the
correct link.

Next, iPerf is used to generate and receive multicast traffic. R1 is the multicast
server and chooses the group address 239.0.0.1. R2 is the multicast receiver, and
joins 239.0.0.1. Note that the UDP packet length must be lowered because the OTV
tunnel does not support fragmentation.

After the multicast tree is built end to end, multicast data plane traffic through the
OTV tunnel is encapsulated in the multicast data group that was configured.
Essentially, this is the multicast UDP packets generated with iPerf from the end
server being encapsulated across the DCI as multicast GRE (the OTV tunnel). From
the below output, we see that N7K1 is a sender for the (S,G) pair (169.254.0.71/32,
232.1.1.0/32), which is this new multicast GRE tunnel. The address 232.1.1.0/32
comes from the fact that the multicast data group range of interface Overlay1 is
232.1.1.0/24, so simply the first address is chosen.
N7K1# show ip mroute
IP Multicast Routing Table for VRF "default"

(*, 224.1.1.1/32), uptime: 03:47:33, otv ip


Incoming interface: Ethernet1/1, RPF nbr: 169.254.0.71
Outgoing interface list: (count: 1)
Overlay1, uptime: 03:47:33, otv
(169.254.0.71/32, 232.1.1.0/32)
, uptime: 00:00:55, igmp ip
Incoming interface: Ethernet1/1, RPF nbr: 169.254.0.71
Outgoing interface list: (count: 1)
Ethernet1/1, uptime: 00:00:55, igmp, (RPF)

The show otv mroute output below tells us what specific multicast traffic is being
encapsulated into the new multicast tunnel. In this case we see that in VLAN 10,
there is a sender 10.0.0.1 using the group address 239.0.0.1.
N7K1# show otv mroute

OTV Multicast Routing Table For Overlay1

(10, *, 239.0.0.1), metric: 0, uptime: 00:00:57, overlay(r)


Outgoing interface list: (count: 1)
Overlay1, uptime: 00:00:57, isis_otv-default
(10, 10.0.0.1, 239.0.0.1)
, metric: 0, uptime: 00:02:07, site Outgoing interface list
: (count: 1) Overlay1
, uptime: 00:00:57, otv

The OTV AED on the other end of the tunnel, N7K2, sees the new multicast GRE
tunnel coming in, as shown below.
N7K2# show ip mroute
IP Multicast Routing Table for VRF "default"

(*, 224.1.1.1/32), uptime: 04:19:02, otv ip


Incoming interface: Ethernet1/1, RPF nbr: 169.254.0.72
Outgoing interface list: (count: 1)
Overlay1, uptime: 04:19:02, otv
(169.254.0.71/32, 232.1.1.0/32)
, uptime: 00:00:31, otv ip
Incoming interface: Ethernet1/1, RPF nbr: 169.254.0.71
Outgoing interface list: (count: 1)
Overlay1, uptime: 00:00:31, otv

In summary, we see that the control plane traffic such as the IS-IS routing protocol
hellos use the OTV control group (224.1.1.1/32 in this case), whereas actual data
multicast feeds use a new address in the 232.1.1.0/24 range. Now the AEDs are
changed so that an OTV Adjacency Server is specified, and the multicast related
configuration is removed.
N7K1# show run interface overlay 1

!Command: show running-config interface Overlay1


!Time: Fri Apr 26 00:31:57 2013

version 6.0(2)

interface Overlay1
otv join-interface Ethernet1/1
otv extend-vlan 10 otv adjacency-server unicast-only
no shutdown
N7K1# show run interface e1/1

!Command: show running-config interface Ethernet1/1

!Time: Fri Apr 26 00:32:02 2013

version 6.0(2)

interface Ethernet1/1
ip address 169.254.0.71/24
no shutdown
N7K2# show run interface overlay 1

!Command: show running-config interface Overlay1


!Time: Fri Apr 26 00:27:14 2013

version 6.0(2)

interface Overlay1
otv join-interface Ethernet1/1
otv extend-vlan 10 otv use-adjacency-server 169.254.0.71 unicast-only
no shutdown
N7K2# show run interface ethernet 1/1

!Command: show running-config interface Ethernet1/1


!Time: Fri Apr 26 00:27:15 2013

version 6.0(2)

interface Ethernet1/1
ip address 169.254.0.72/24
no shutdown

N7K1 is configured as the Adjacency Server, and N7K2 is configured with the
unicast address of the Adjacency Server. This means that N7K1 will learn N7K2s
unicast address when N7K2 registers. Also, if there were more AEDs at the remote
site or if there were multiple remote sites, they would all register with N7K1, then
N7K1 would distribute the endpoint addresses to all other AEDs. Additionally, there
is an option to configure redundant Adjacency Servers, as with the current
configuration shown in this task; a single failure of the Adjacency Server would
effectively cause a loss in connectivity of the entire OTV network.
From the end servers perspective, nothing changes. Both unicast and multicast
reachability are maintained; however, in the core of the DCI, the traffic pattern
changes. The first change of note is that there are no longer any entries in the
multicast routing table over the DCI, as seen below.

N7K1# show ip mroute

IP Multicast Routing Table for VRF "default"

The AED does still know about the multicast sender in its local site, as seen with the
show otv mroute below.
N7K1# show otv mroute

OTV Multicast Routing Table For Overlay1

(10, *, 239.0.0.1), metric: 0, uptime: 00:01:29, overlay(r)


Outgoing interface list: (count: 1)
Overlay1, N7K2, uptime: 00:01:29, isis_otv-default
(10, 10.0.0.1, 239.0.0.1)
, metric: 0, uptime: 00:06:08, site
Outgoing interface list: (count: 1)
Overlay1, N7K2, uptime: 00:01:29, otv

An inline EthAnalyzer capture shows us that control plane traffic is unicast only
between the AEDs, where previously this would have been multicast using the
control group.
On Nexus 7K, EthAnalyzer can only be run by a user with the admin
role in the default VDC.

N7K2# ethanalyzer local interface inband capture-filter "ip proto 47" limit-captured-frames 1 detail | no-more

Capturing on inband
Frame 1 (228 bytes on wire, 196 bytes captured)
Arrival Time: Apr 26, 2013 01:52:50.984764000
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 228 bytes
Capture Length: 196 bytes
[Frame is marked: False]
[Protocols in frame: eth:ip:gre:mpls:pwethheuristic:pwethcw:eth:mdshdr:fc]
Ethernet II, Src: 68:bd:ab:d7:60:41 (68:bd:ab:d7:60:41), Dst: 00:26:98:0c:21:41 (00:26:98:0c:21:41)
Destination: 00:26:98:0c:21:41 (00:26:98:0c:21:41)
Address: 00:26:98:0c:21:41 (00:26:98:0c:21:41)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)

Source: 68:bd:ab:d7:60:41 (68:bd:ab:d7:60:41)


Address: 68:bd:ab:d7:60:41 (68:bd:ab:d7:60:41)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Type: IP (0x0800)
Internet Protocol, Src: 169.254.0.71 (169.254.0.71), Dst: 169.254.0.72 (169.254.0.72)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 182
Identification: 0x0824 (2084)
Flags: 0x02 (Don't Fragment)
0.. = Reserved bit: Not Set
.1. = Don't fragment: Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 254 Protocol: GRE
(0x2f)
Header checksum: 0x1f69 [correct]
[Good: True]
[Bad : False] Source: 169.254.0.71 (169.254.0.71)
Destination: 169.254.0.72 (169.254.0.72)

Generic Routing Encapsulation (0x8848 - unknown)


Flags and version: 0000
0... .... .... .... = No checksum
.0.. .... .... .... = No routing
..0. .... .... .... = No key
...0 .... .... .... = No sequence number
.... 0... .... .... = No strict source route
.... .000 .... .... = Recursion control: 0
.... .... 0000 0... = Flags: 0
.... .... .... .000 = Version: 0
Protocol Type: Unknown (0x8848)
MultiProtocol Label Switching Header, Label: 42, Exp: 0, S: 1, TTL: 254
MPLS Label: 42
MPLS Experimental Bits: 0
MPLS Bottom Of Label Stack: 1
MPLS TTL: 254
PW Ethernet Control Word
Sequence Number: 1
Ethernet II, Src: 5b:ea:86:dd:60:00 (5b:ea:86:dd:60:00), Dst: 00:02:00:0c:29:bc (00:02:00:0c:29:bc)
Destination: 00:02:00:0c:29:bc (00:02:00:0c:29:bc)

Address: 00:02:00:0c:29:bc (00:02:00:0c:29:bc)


.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: 5b:ea:86:dd:60:00 (5b:ea:86:dd:60:00)
Address: 5b:ea:86:dd:60:00 (5b:ea:86:dd:60:00)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
.... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
Type: Unknown (0x0000)
MDS Header(SOFi2/)
MDS Header
...1 0001 0000 0001 = Packet Len: 4353
.... 0000 0000 00.. = Dst Index: 0x0000
.... ..00 0000 0000 = Src Index: 0x0000
.... 0010 0000 1000 = VSAN: 520
Fibre Channel
[Exchange First In: 0]
[Exchange Last In: 0]
R_CTL: 0xc1(Link_Control Frame/ACK0)
Dest Addr: 16.13.dc
CS_CTL: 0xff
Src Addr: 02.00.00
Type: Basic Link Svc (0x00)
F_CTL: 0x000000 Exchange Originator Seq Initiator CS_CTL
0... .... .... .... .... .... = ExgRpd: Exchange Originator
.0.. .... .... .... .... .... = SeqRec: Seq Initiator
..0. .... .... .... .... .... = ExgFst: NOT exchg first
...0 .... .... .... .... .... = ExgLst: NOT exchg last
.... 0... .... .... .... .... = SeqLst: NOT seq last
.... ..0. .... .... .... .... = Pri: CS_CTL
.... ...0 .... .... .... .... = TSI: NOT transfer seq initiative
.... .... 00.. .... .... .... = LDF: Last Data Frame - No Info (0x000000)
.... .... ..00 .... .... .... = A01: no ack required (0x000000)
.... .... .... ..0. .... .... = RetSeq: NOT retransmitted sequence
.... .... .... .... ..00 .... = AA: ABTS - Cont (0x000000)
.... .... .... .... .... 0... = RelOff: rel offset NOT set
SEQ_ID: 0x00
DF_CTL: 0x00
SEQ_CNT: 0
OX_ID: 0x0001
RX_ID: 0x0002
Parameter: 0x02220223

1 packet captured

Program exited with status 0.

Nexus Technology Labs - Data Center


Technology Lab Resources
Data Center Rack Rental Access Guide
This guide describes how to use the INE CCIE Data Center Rack Rentals that
complement the CCIE Data Center Technology Lab Online Workbook.

Important Notes
To rent Data Center racks, you must have purchased the Data Center Online
Workbook. You can read about, preview, and purchase the workbook here.
You are limited to 3 concurrent scheduled sessions with a maximum time of 9 hours
per session.

Common Problems
Scheduling a Rack Session
Passwords
Scheduling Data Center Rack Add-ons
Loading and Saving Configurations
Canceling a Rack Session
Connecting to NX-OS Devices CLI
Using the Rack Control Panel
Connecting to Windows Virtual Machines
Connecting to UCS Blade ESXi Instances

Common Problems
Telnet Connection Warning

Some Telnet clients will close the Telnet window if the connection cannot be
established. This behavior prevents you from seeing error messages that indicate
the line is in use, so its a good practice to disable this behavior in your Telnet client.
If you do not see a command prompt when you establish a Telnet connection, you
may need to press Enter to wake up the device.
Important Configuration Changes
When you load a saved configuration into the UCS, you must change the interfaces
in use to match the Data Center rack you are using; our automation cannot make
this modification for you. You will find the necessary configuration changes in the
reminder email you receive before your rack session.
Port Speed Information
If you receive an error message stating, SFP validation failed, you must manually
set the port speed to 1G (1000) because Nexus does not support auto-negotiation
from 10G to 1G.

Scheduling a Rack Session


You schedule your Data Center rack session through your Members account.
1. Sign in to the Members site, and then click Rack Rentals on the left side of the page.
2. Find the rack rental type you want, and click Schedule to see the rack session
booking page.
3. On this page, select a preferred range of start and end dates and start and end times
for your rack rental session.
4. Click Search to find and select available sessions. Click the Book button to reserve
your time.
Data Center rack sessions start at designated, set intervals every
three hours. You cannot choose a custom start time for Data Center
rack rentals at this time. The last half hour of your session is for
intersession (the rack is reset for the next customer). For example, if
you reserve a 3-hour session, your session will last 2.5 hours; if you
reserve a 6-hour session, your session will last 5.5 hours. You are not
charged for the intersession portion of your rack rental.
FULLY BOOKED means that the session is not available to rent.
Book means that you may reserve this session. It also displays the number of

tokens required to reserve this session.


Rent Now means that the session is available for immediate rental.
Before booking or reserving a session, select any add-ons you want
by using the check boxes above the calendar tool.
In the Reserved Sessions area, the Active tab displays any sessions that have
already started. These sessions cannot be cancelled. The Upcoming tab displays
your reserved sessions, which can be cancelled at any time before the start time of
the session.

Passwords

Device

IP Address

Username/password (casesensitive)

ESXi1

10.0.115.11

root / cciedc01

ESXi2

10.0.115.12

root / cciedc01

ESXiVMFEX

10.0.115.21

root / cciedc01

Win2k8-BM

10.0.114.21

Administrator / Cciedc01

N7K and
N5K

cisco / cisco

MDS

cisco / cisco (role = cisco)

UCS

192.168.0.100

cisco / rackpassword (role =


admin)

The UCS password is now dynamically changed to match your


current rack session password.

Scheduling Data Center Rack Add-ons


Data Center Rack Rentals are modular.
By default, each 'Base 5K/7K' rental includes access to 2 x Nexus 7000 VDCs, 2 x
Nexus 5000s, and 2 x Windows Server Virtual Machines.
By selecting the UCS/SAN Add-On, you also get access to the UCS Blade Server
chassis, UCS 6200 Fabric Interconnects, MDS 9200 SAN Switches, and the Fibre
Channel SAN.
By selecting the N2K/SAN Add-On, you also get access to the Nexus 2000 Fabric
Extenders with a 10GigE attached server, MDS 9200 SAN Switches, and the Fibre
Channel SAN.

Note that not all labs require each Add-On. You can determine which labs need
which Add-Ons by comparing the Data Center Rack Rental Access Guide CCIE DC
Physical Diagram, located in the Resources section in the upper-right corner of this
page, with the diagram for the individual workbook section that you are working on.
Use the following general guidelines to help determine which add-ons you need:
The Nexus labs require only the Base N5K/N7K rental, unless you are working on
the Fabric Extender or SAN labs, in which case they require the N2K/SAN Add-On.
The UCS labs require the Base N5K/N7K rental plus the UCS Add-On.

Loading and Saving Configurations


When you load a saved configuration into the UCS, you must change the interfaces
in use to match the Data Center rack you are using; our automation cannot make
this modification for you. You will find the necessary configuration changes in the
reminder email you receive before your rack session.
Before saving any configuration, make sure that all of your devices are at the
command prompt or a login prompt.

Canceling a Rack Session


You can cancel a rack session in your Members account.
1. Click Rack Rentals on the left side of the page, and then click the Upcoming
Rentals tab at the top.
2. To cancel a session, click the red Cancel button in the Upcoming Rentals window.
3. You will be asked to confirm your cancellation. Another message states the number
of tokens that have been returned to your account. Click OK. Your session will be
cancelled and your tokens refunded.

Connecting to NX-OS Devices CLI


When your session begins, find your login information on the Rack Rentals page as
shown above. Open your terminal software, such as PuTTY, SecureCRT, etc., and
telnet to dc.ine.com. Log in with the provided username and password.

When you are logged in, a menu displays the devices that you have access to. The
example shown below indicates that this session is assigned the Base 5K/7K rental,
with both the UCS/SAN and N2K/SAN add-ons.

Choose the device that you want to connect to, such as N7K1, and log in with the
username cisco and password cisco. Note that the default ACE
username/password is cisco/ciscocisco.

Certain commands, such as erasing and reloading the device or


editing the management IP address, are purposely restricted to allow
the automation system to function properly.

Using the Rack Control Panel


The Rack Control Panel allows you to reset devices to their default configurations
and access the remote desktops of the virtual machines. To access the control
panel during your active rack session, click Rack Rentals on the Members site, find
your session by clicking the Upcoming Rentals tab, and then click the Control
Panel button.
Save config and load config functionality will be added to the DC
control panel shortly.

Connecting to Virtual Machines


Each rack session is assigned access to certain virtual machines, based on your
rack number assignment and the add-ons that you chose. To access the virtual
machines:
1. Go to the Rack Control Panel on the Members site.
2. Click the Remote Desktop Access tab to see a list of available virtual machines
below. If this page asks you for a username and password, use your rack username
and password, e.g. username dcrack1 password abcdef.

3. Click the icon for the VM that you want to access, and the remote desktop will
automatically open in another window. The remote desktop's resolution automatically
resizes to your browser window size, so if you simply refresh this page the resolution
will update to the browser size.

If for some reason you lock yourself out of a Virtual Machine, you can reset them
from the control panel. All changes you made during your rack session will be
reverted when you use this option.

Connecting to UCS Blade ESXi Instances


When it becomes necessary to connect to the UCS blade ESXi (or WindowsBareMetal) instances, follow the directions in the task to connect to them, using the
VMWare Infrastructure Client provided (preinstalled on any of the Windows VM
desktops). The IP addresses for the UCS blade ESXi and Windows Win2k8R2 bare
metal instances are:
ESXi1: 10.0.115.11
ESXi2: 10.0.115.12
ESXi-VMFEX: 10.0.115.21
Win2k8-BM: 10.0.114.21
The username and password for all ESXi instances are root and cciedc01.

The username and password for the Win2k8R2 instance is Administrator and
cciedc01 or Cciedc01.

Nexus Technology Labs - Release Notes


Nexus Technology Labs: January 3, 2014
Changes made January 3, 2014
This workbook was updated with new rack number associations, and redundant
MDS links were removed.

Nexus Technology Labs - FabricPath


FabricPath
Task
Configure basic layer 2 connectivity between the Nexus 7Ks and 5Ks as follows:
Create VLANs 10 and 20 on N7K1, N7K2, N5K1, & N5K2.
Create SVIs for VLANs 10 and 20 on N5K1 & N5K2. Use IP addresses
10.0.0.5Y/24 and 20.0.0.5Y/24, where Y is the 5K's device number per the
diagram.
Configure trunk links between N7K1 & N5K1, and between N7K2 & N5K2.
All other links on the 5Ks should be disabled.
Configure FabricPath on N7K1 & N7K2 as follows:
Install and enable the FabricPath feature set.
Configure VLANs 10 & 20 as FabricPath VLANs.
Set the FabricPath Switch-ID to by 7Y, where Y is the 7K's device number
per the diagram.
Enable FabricPath on the F1 ports between the 7Ks; the links down to the
5Ks will be running Classical Ethernet.
Ensure that the 7Ks are the root of the Spanning-Tree for both VLANs 10 &
20.
Once complete, N5K1 & N5K2 should have IP reachability between their VLAN 10 &
20 interfaces, and the 7Ks should be routing this traffic over FabricPath between
each other.

Configuration
N5K7:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20
!

interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N5K8:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N7K7:
feature-set fabricpath
!
vlan 10,20
mode fabricpath
!
spanning-tree vlan 10,20 priority 0
!

fabricpath switch-id 77
!
interface Ethernet2/25
switchport mode fabricpath
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown
N7K8:

feature-set fabricpath
!
vlan 10,20
mode fabricpath
!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 78
!
interface Ethernet2/25
switchport mode fabricpath
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown

Verification
Note that the installation of the FabricPath feature in the
Admin/Default VDC is already done for you in the INE rack rentals.
For other topologies the following steps would also be required before
running FabricPath.

DC2-N7K1# conf t
Enter configuration commands, one per line.

End with CNTL/Z.

DC2-N7K1(config)# install feature-set fabricpath

feature set is installed already(0x40aa0011)DC2-N7K1(config)# vdc N7K7


DC2-N7K1(config-vdc)# allow feature-set fabricpath

DC2-N7K1(config-vdc)# end
DC2-N7K1#

Once FabricPath is enabled at the interface level in the user VDC, the 7Ks should
form an IS-IS adjacency, as seen below.
N7K7# show fabricpath isis adjacency
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID

SNPA

Level

State

Hold Time

Interface

N7K8

N/A

UP

00:00:25

Ethernet2/25

N7K8

N/A

UP

00:00:24

Ethernet2/26

The show fabricpath switch-id command shows how many switches (both Leaf and
Spine nodes) are in the FabricPath topology, and what their FabricPath Switch-IDs
are (essentially the IS-IS Router-ID). This would also be where we would see any
switches running vPC+ or Anycast HSRP.
N7K7# show fabricpath switch-id
FABRICPATH SWITCH-ID TABLE
Legend: '*' - this system
'[E]' - local Emulated Switch-id
'[A]' - local Anycast Switch-id
Total Switch-ids: 2
=============================================================================
SWITCH-ID

SYSTEM-ID

FLAGS

STATE

STATIC

EMULATED/
ANYCAST

--------------+----------------+------------+-----------+--------------------

* 77
64a0.e742.8dc5

Primary

Confirmed Yes

No

4055.390b.c145

Primary

Confirmed Yes

No

78

The FabricPath routing table shows how the local Leaf or Spine makes its path
selection to the other Switch-IDs in the Fabric.
N7K7# show fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/77/0, number of next-hops: 0


via ---- , [60/0], 0 day/s 00:02:32, local 1/ 78
/0, number of next-hops: 2 via Eth2/25, [115/40]
, 0 day/s 00:02:24, isis_fabricpath-default via Eth2/26, [115/40]
, 0 day/s 00:02:24, isis_fabricpath-default

The details of the IS-IS database show how the switches are specifically connected
together in the SPF graph, their FabricPath Switch-ID, which Topologies are
enabled, and which switches are the Multi-Destination Root for those trees.
N7K7# show fabricpath isis database detail
Fabricpath IS-IS domain: default LSP database
LSPID

Seq Number
0x00000006

0x5E1F

Instance

0x00000002

Area Address

00

NLPID

0xC0

Hostname

N7K8

Extended IS

Lifetime

A/P/O/T N7K8.00-00

0/0/0/1

Length : 4

N7K7.00

Extended IS

Capability

Checksum
1157

Metric : 40
N7K7.00

Metric : 40

: Device Id: 78 Base Topology

Base Topo Trees :


Trees desired: 2
Nickname

Trees computed: 2

Trees usable: 2

Priority: 0 Nickname: 78 BcastPriority: 64


Version

Version: 1 Flags: 0

Nickname Migration :
Swid: 78 Sec. Swid: 0
0 N7K7

Digest Offset :
.00-00

* 0x00000009

0xB2C5

Instance

0x00000009

Area Address

00

NLPID

0xC0

Hostname

N7K7

Extended IS

N7K8.00

Extended IS

0/0/0/1

Length : 4

Capability

1158

Metric : 40
N7K8.00

Metric : 40

: Device Id: 77 Base Topology

Base Topo Ftag

Graph 1: Root: N7K7 Primary: 1, Secondary: 0 Nickname 77

Graph 2: Root: N7K8 Primary: 2, Secondary: 0 Nickname 78

Base Topo Trees :


Trees desired: 2

Trees computed: 2

Trees usable: 2

Base Topo Roots :


Graph 1: Root Nickname: 77
Graph 2: Root Nickname: 78
Nickname

Priority: 0 Nickname: 77 BcastPriority: 64


Version

Version: 1 Flags: 0
Nickname Migration :
Swid: 77 Sec. Swid: 0
Digest Offset :

From the devices in the Classical Ethernet domain (the 5Ks in this case), they
simply see the network as a normal Ethernet switched LAN.
N5K7# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID

Bridge ID

Priority

10

Address

c84c.75fa.6000

Cost

Port

134 (Ethernet1/6)

Hello Time

Priority

32778

Address

547f.ee79.137c

Hello Time

sec

sec

Max Age 20 sec

Forward Delay 15 sec

(priority 32768 sys-id-ext 10)

Max Age 20 sec

Forward Delay 15 sec

Interface

Role Sts Cost

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------- Eth1/6


2

128.134

Eth1/7

Root FWD

P2p

Altn BLK 2

128.135

P2p

N5K7# ping 10.0.0.58


PING 10.0.0.58 (10.0.0.58): 56 data bytes
Request 0 timed out
64 bytes from 10.0.0.58: icmp_seq=1 ttl=254 time=3.038 ms
64 bytes from 10.0.0.58: icmp_seq=2 ttl=254 time=4.825 ms
64 bytes from 10.0.0.58: icmp_seq=3 ttl=254 time=4.976 ms
64 bytes from 10.0.0.58: icmp_seq=4 ttl=254 time=4.928 ms

--- 10.0.0.58 ping statistics --5 packets transmitted, 4 packets received, 20.00% packet loss
round-trip min/avg/max = 3.038/4.441/4.976 ms
N5K7# show mac address-table dynamic vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN

MAC Address

Type

age

Secure NTFY

Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------ *
10

547f.ee7a.4d7c

dynamic

10

Eth1/6

When frames are received by the FabricPath Leaf nodes, a MAC address table
lookup is performed, and the destination FabricPath Switch-ID is found. This field is
used as the Outer Destination Address (ODA) when the Ethernet frame is
FabricPath encapsulated. The source address of the FabricPath frame is the local
Switch-Id, which is then considered the Outer Source Address (OSA).

N7K7# show mac address-table dynamic vlan 10

Note: MAC table entries displayed are getting read from software.
Use the 'hardware-age' keyword to get information related to 'Age'

Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False ,
VLAN

MAC Address

~~~ - use 'hardware-age' keyword to retrieve age info

Type

Secure NTFY Ports/ SWID.SSID.LID

age

---------+-----------------+--------+---------+------+----+-----------------* 10

547f.ee79.137c

dynamic

~~~

dynamic

~~~

Eth2/27

10 547f.ee7a.4d7c

F 78.0.26

In this example there are no pure Spine switches in the FabricPath topology, but in
a real deployment the Spine switches will not populate the MAC address table for
destinations reachable via FabricPath. Instead they simply use the FabricPath
routing table to forward towards the ODA address (destination Switch-ID).
Note that the FabricPath Leaf Switches must be the root of the STP
for the Classical Ethernet domain, otherwise their CE facing ports will
be disabled, as seen below. This is similar to the STP RootGuard
feature.

N7K7# config
Enter configuration commands, one per line.

End with CNTL/Z.

N7K7(config)# spanning-tree vlan 10 priority 61440


N7K7(config)#
2015 Feb 13 20:32:14 N7K7 %STP-2-L2GW_BACKBONE_BLOCK:
L2 Gateway Backbone port inconsistency blocking port Ethernet2/27 on VLAN0010
.

2015 Feb 13 20:32:14 N7K7 %STP-2-L2GW_BACKBONE_BLOCK: L2 Gateway Backbone port inconsistency blocking port Ethernet2
N7K7(config)# show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID

Priority

61450

Address

c84c.75fa.6000

This bridge is the root


Hello Time

sec

Max Age 20 sec

Forward Delay 15 sec

Bridge ID

Interface

Priority

61450

Address

c84c.75fa.6000

Hello Time

sec

Role Sts Cost

(priority 61440 sys-id-ext 10)

Max Age 20 sec

Forward Delay 15 sec

Prio.Nbr Type

---------------- ---- --- --------- -------- -------------------------------Eth2/27

Desg BKN*2

128.283

P2p *L2GW_Inc

Eth2/28

Desg BKN*2

128.284

P2p *L2GW_Inc

Nexus Technology Labs - FabricPath


FabricPath with Port-Channels
Task
Configure basic layer 2 connectivity between the Nexus 7Ks and 5Ks as follows:
Create VLANs 10 and 20 on N7K1, N7K2, N5K1, & N5K2.
Create SVIs for VLANs 10 and 20 on N5K1 & N5K2. Use IP addresses
10.0.0.5Y/24 and 20.0.0.5Y/24, where Y is the 5K's device number per the
diagram.
Configure trunk links between N7K1 & N5K1, and between N7K2 & N5K2.
All other links on the 5Ks should be disabled.
Configure FabricPath on N7K1 & N7K2 as follows:
Install and enable the FabricPath feature set.
Configure VLANs 10 & 20 as FabricPath VLANs.
Set the FabricPath Switch-ID to by 7Y, where Y is the 7K's device number
per the diagram.
Aggregate the F1 ports between the 7Ks as interface Port-Channel1.
Enable FabricPath on Port-Channel1 of the 7Ks; the links down to the 5Ks
will be running Classical Ethernet.
Ensure that the 7Ks are the root of the Spanning-Tree for both VLANs 10 &
20.
Once complete, N5K1 & N5K2 should have IP reachability between their VLAN 10 &
20 interfaces, and the 7Ks should be routing this traffic over FabricPath between
each other.

Configuration
N5K7:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20

!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N5K8:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N7K7:

feature-set fabricpath
!
vlan 10,20
mode fabricpath
!

spanning-tree vlan 10,20 priority 0


!
fabricpath switch-id 77
!
interface port-channel1
switchport
switchport mode fabricpath
!
interface Ethernet2/25
switchport mode fabricpath
channel-group 1
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
channel-group 1
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown
N7K8:

feature-set fabricpath
!
vlan 10,20
mode fabricpath
!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 78
!
interface port-channel1
switchport
switchport mode fabricpath
!
interface Ethernet2/25
switchport mode fabricpath
channel-group 1
no shutdown
!
interface Ethernet2/26

switchport mode fabricpath


channel-group 1
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown

Verification
As seen in previous examples, the underly switchport mode (access, trunk, layer 3,
etc.) is independent of the Port-Channel configuration. Per the output below we can
see that Port-Channel1 is running in FabricPath mode.
N7K7# show int port-channel1 status

-------------------------------------------------------------------------------Port

Name

Status

Vlan

Duplex

Speed

Type

-------------------------------------------------------------------------------Po1

connected f-path

-full

a-10G

--

The difference between this example and the previous one is that the switches will
only form one FabricPath IS-IS adjacency over the Port-Channel, instead of one
adjacency for each underlying member port. From a control-plane point of view, the
port-channel design is more efficient, because if one of the member interfaces is
lost, IS-IS does not need to reconverge. Also the IS-IS state machine does not need
to maintain multiple adjacencies between the same switches.
N7K7# show fabricpath isis adjacency
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID

SNPA

Level

State

Hold Time

Interface

N7K8

N/A

UP

00:00:26

port-channel1

From a traffic forwarding point of view, packets are still load balanced on the links
between the 7Ks, but now it uses the port-channel hashing mechanism instead of
the IS-IS load balancing method.
N7K7# show fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/77/0, number of next-hops: 0


via ---- , [60/0], 0 day/s 00:23:10, local 1/ 78
/0, number of next-hops: 1 via Po1, [115/20]
, 0 day/s 00:02:51, isis_fabricpath-default

Note that the cost of the route between the 7Ks is now 20, where in the previous
example it was two links each with a cost of 40. This is because FabricPath IS-IS
uses a linear reference bandwidth scale, that if the bandwidth of the links is doubled
(2 x 10GigE), the cost is halved. This also means from a path selection point of
view, FabricPath IS-IS would prefer a 2 x 10GigE channel vs. a single 10GigE link.
N7K7# show fabricpath isis database detail
Fabricpath IS-IS domain: default LSP database
LSPID

Seq Number

.00-00

0x0000000F

0xA796

Instance

0x00000008

Area Address

00

NLPID

0xC0

Hostname

N7K8

Extended IS

Capability

Checksum

Lifetime

1032

A/P/O/T N7K8

0/0/0/1

Length : 4

N7K7.00

Metric : 20

: Device Id: 78 Base Topology

Base Topo Trees :


Trees desired: 2
Nickname

Trees computed: 2

Trees usable: 2

Priority: 0 Nickname: 78 BcastPriority: 64


Version

Version: 1 Flags: 0
Nickname Migration :
Swid: 78 Sec. Swid: 0
Digest Offset :

N7K7
.00-00

* 0x00000012

0x9601

Instance

0x00000012

Area Address

00

NLPID

0xC0

Hostname

N7K7

Extended IS

0/0/0/1

Length : 4

Capability

1026

N7K8.00

Metric : 20

: Device Id: 77 Base Topology

Base Topo Ftag

Graph 1: Root: N7K7 Primary: 1, Secondary: 0 Nickname 77


Graph 2: Root: N7K8 Primary: 2, Secondary: 0 Nickname 78
Base Topo Trees :
Trees desired: 2

Trees computed: 2

Trees usable: 2

Base Topo Roots :


Graph 1: Root Nickname: 77
Graph 2: Root Nickname: 78
Nickname

Priority: 0 Nickname: 77 BcastPriority: 64


Version

Version: 1 Flags: 0
Nickname Migration :
Swid: 77 Sec. Swid: 0
Digest Offset :

From the MAC address table perspective, the destination MAC addresses are still
associated with the destination Switch-ID, so the Port-Channel configuration does
not impact this operation.
N7K7# show mac address-table dynamic vlan 10
Note: MAC table entries displayed are getting read from software.
Use the 'hardware-age' keyword to get information related to 'Age'

Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False ,
VLAN

MAC Address

~~~ - use 'hardware-age' keyword to retrieve age info

Type

age

Secure NTFY Ports/ SWID.SSID.LID

---------+-----------------+--------+---------+------+----+-----------------* 10

547f.ee79.137c

dynamic

~~~

Eth2/27

10

547f.ee7a.4d7c

dynamic

~~~

F 78.0.26

Nexus Technology Labs - FabricPath


FabricPath Hello Authentication
Task
Configure basic layer 2 connectivity between the Nexus 7Ks and 5Ks as follows:
Create VLANs 10 and 20 on N7K1, N7K2, N5K1, & N5K2.
Create SVIs for VLANs 10 and 20 on N5K1 & N5K2. Use IP addresses
10.0.0.5Y/24 and 20.0.0.5Y/24, where Y is the 5K's device number per the
diagram.
Configure trunk links between N7K1 & N5K1, and between N7K2 & N5K2.
All other links on the 5Ks should be disabled.
Configure FabricPath on N7K1 & N7K2 as follows:
Install and enable the FabricPath feature set.
Configure VLANs 10 & 20 as FabricPath VLANs.
Set the FabricPath Switch-ID to by 7Y, where Y is the 7K's device number
per the diagram.
Enable FabricPath on the F1 ports bewteen the 7Ks; the links down to the
5Ks will be running Classical Ethernet.
Configure the 7Ks to authenticate each others IS-IS hellos with an MD5
hash of the string CCIEDC.
Ensure that the 7Ks are the root of the Spanning-Tree for both VLANs 10 &
20.
Once complete, N5K1 & N5K2 should have IP reachability between their VLAN 10 &
20 interfaces, and the 7Ks should be routing this traffic over FabricPath between
each other.

Configuration
N5K7:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan

!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N5K8:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N7K7:
feature-set fabricpath
!
vlan 10,20
mode fabricpath

!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 77
!
key chain FPATH_HELLO_AUTH
key 1
key-string CCIEDC
!
interface Ethernet2/25
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FPATH_HELLO_AUTH
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FPATH_HELLO_AUTH
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown
N7K8:

feature-set fabricpath
!
vlan 10,20
mode fabricpath
!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 78
!
key chain FPATH_HELLO_AUTH
key 1
key-string CCIEDC
!
interface Ethernet2/25
switchport mode fabricpath
fabricpath isis authentication-type md5

fabricpath isis authentication key-chain FPATH_HELLO_AUTH


no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FPATH_HELLO_AUTH
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown

Verification
FabricPath IS-IS supports two types of authentication, hello authentication
configured at the interface level, and Link State Packet (LSP) authentication
configured under the global IS-IS process.
In the below output, one side of the link is configured with authentication, while the
other is not. After the IS-IS hold timer expires, the adjacency is torn down.
N7K8# sh run int e2/25 - 26

!Command: show running-config interface Ethernet2/25-26


!Time: Fri Feb 13 21:59:27 2015

version 6.2(10)

interface Ethernet2/25
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FPATH_HELLO_AUTH
no shutdown

interface Ethernet2/26
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FPATH_HELLO_AUTH
no shutdown
N7K8# show fabricpath isis adjacency

Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:


System ID

SNPA

Level

N7K7

N/A

1 UP

00:00:12

Ethernet2/25 N7K7

00:00:10

Ethernet2/26

State

Hold Time

Interface

N/A

1 UP

N7K8# 2015 Feb 13 21:59:42 N7K8 %ISIS_FABRICPATH-5-ADJCHANGE:

isis_fabricpath-default [906]

P2P adj L1 N7K7 over Ethernet2/26 - DOWN


(Hold timer expired) on MT-0
2015 Feb 13 21:59:43 N7K8 %ISIS_FABRICPATH-5-ADJCHANGE:

isis_fabricpath-default [906]

P2P adj L1 N7K7 over Ethernet2/25 - DOWN


(Hold timer expired) on MT-0
N7K8# show fabricpath isis adjacency
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID

SNPA

Level

N7K7

N/A

1 LOST

00:05:31

Ethernet2/25 N7K7

00:05:13

Ethernet2/26

State

Hold Time

N/A

Interface

1 LOST

Debugging the IS-IS hello exchange between peers can be used to troubleshoot
authentication issues, as seen below.
N7K8# debug fabricpath isis iih

2015 Feb 13 22:02:22.852373 isis_fabricpath: default [906] Receive P2P IIH over Ethernet2/25 from N7K7 len 1500 prio
2015 Feb 13 22:02:22.852412 isis_fabricpath: default [906]
P2P IIH authentication failure over Ethernet2/25
2015 Feb 13 22:02:22.852428 isis_fabricpath: default [906] P2P IIH auth failed!

2015 Feb 13 22:02:23.048618 isis_fabricpath: default [906] IIH timer expired for interface Ethernet2/25

2015 Feb 13 22:02:23.048699 isis_fabricpath: default [906] isis_add_auth_tlv: keychain : , type 36, handle0x0x896d44

Interface level FabricPath authentication can be verified as follows:


N7K7# sh fabricpath isis interface e2/25
Fabricpath IS-IS domain: default
Interface: Ethernet2/25
Status: protocol-up/link-up/admin-up
Index: 0x0002, Local Circuit ID: 0x01, Circuit Type: L1 Authentication type MD5
Authentication keychain is FPATH_HELLO_AUTH
Authentication check specified

Extended Local Circuit ID: 0x1A098000, P2P Circuit ID: 0000.0000.0000.00


Retx interval: 5, Retx throttle interval: 66 ms
LSP interval: 33 ms, MTU: 1500

P2P Adjs: 1, AdjsUp: 1, Priority 64


Hello Interval: 10, Multi: 3, Next IIH: 00:00:05
Level

Adjs

AdjsUp

Metric

CSNP

40

60

Next CSNP

Last LSP ID

Inactive

ffff.ffff.ffff.ff-ff

Topologies enabled:
Level Topology Metric

MetricConfig Forwarding

40

no

UP

40

no

UP

Nexus Technology Labs - FabricPath


FabricPath LSP Authentication
Task
Configure basic layer 2 connectivity between the Nexus 7Ks and 5Ks as follows:
Create VLANs 10 and 20 on N7K1, N7K2, N5K1, & N5K2.
Create SVIs for VLANs 10 and 20 on N5K1 & N5K2. Use IP addresses
10.0.0.5Y/24 and 20.0.0.5Y/24, where Y is the 5K's device number per the
diagram.
Configure trunk links between N7K1 & N5K1, and between N7K2 & N5K2.
All other links on the 5Ks should be disabled.
Configure FabricPath on N7K1 & N7K2 as follows:
Install and enable the FabricPath feature set.
Configure VLANs 10 & 20 as FabricPath VLANs.
Set the FabricPath Switch-ID to by 7Y, where Y is the 7K's device number
per the diagram.
Enable FabricPath on the F1 ports bewteen the 7Ks; the links down to the
5Ks will be running Classical Ethernet.
Configure the 7Ks to authenticate each others IS-IS LSPs with an MD5 hash
of the string CCIEDC.
Ensure that the 7Ks are the root of the Spanning-Tree for both VLANs 10 &
20.
Once complete, N5K1 & N5K2 should have IP reachability between their VLAN 10 &
20 interfaces, and the 7Ks should be routing this traffic over FabricPath between
each other.

Configuration
N5K7:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan

!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N5K8:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N7K7:
feature-set fabricpath
!
vlan 10,20
mode fabricpath

!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 77
!
key chain FPATH_LSP_AUTH
key 1
key-string CCIEDC
!
fabricpath domain default
authentication-type md5
authentication key-chain FPATH_LSP_AUTH
!
interface Ethernet2/25
switchport mode fabricpath
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown
N7K8:

feature-set fabricpath
!
vlan 10,20
mode fabricpath
!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 78
!
key chain FPATH_LSP_AUTH
key 1
key-string CCIEDC
!
fabricpath domain default
authentication-type md5
authentication key-chain FPATH_LSP_AUTH

!
interface Ethernet2/25
switchport mode fabricpath
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown

Verification
FabricPath IS-IS supports two types of authentication, hello authentication
configured at the interface level, and Link State Packet (LSP) authentication
configured under the global IS-IS process.
An LSP authentication mismatch can be difficult to troubleshoot, as it appears as
though the switches form an IS-IS adjacency, as seen below.
N7K7# show fabricpath isis adjacency
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID

SNPA

Level

4055.390b.c145

N/A

1 UP

State

00:00:32

Ethernet2/25 4055.390b.c145

00:00:26

Ethernet2/26

Hold Time

N/A

Interface

1 UP

However, when LSP authentication fails, the switches will not install routes learned
from each other into the routing table. Without routes installed, the end result is a
failure in the data plane of the actual user traffic sent over the fabric.
N7K7# show fabricpath route

FabricPath Unicast Route Table


'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/77/0, number of next-hops: 0


via ---- , [60/0], 0 day/s 00:03:30, local

LSP authentication failure can be debugged as follows:


N7K7# debug fabricpath isis lsp flooding
2015 Feb 13 20:58:14.289898 isis_fabricpath: default [6331] Receive default L1 LSP
4055.390b.c145.00-00 Seq 0x00000007 Chksum 0x5C20 Lifetime 1139 over Ethernet2/26 prio 6
2015 Feb 13 20:58:14.289937 isis_fabricpath: default [6331] LSP authentication failure

2015 Feb 13 20:58:14.609373 isis_fabricpath: default [6331] Receive default L1 LSP 4055.390b.c145.00-00 Seq 0x000000
2015 Feb 13 20:58:14.609407 isis_fabricpath: default [6331] LSP authentication failure

2015 Feb 13 20:58:17.439426 isis_fabricpath: default [6331] Receive default L1 GM-LSP 4055.390b.c145.00-00 Seq 0x000
2015 Feb 13 20:58:17.439462 isis_fabricpath: default [6331] GM-LSP authentication failure

2015 Feb 13 20:58:18.576989 isis_fabricpath: default [6331] Retx L1 GM-LSP N7K7.00-00 Seq 0x00000002 Chksum 0xA670 L

After the authentication problem is corrected, normal adjacencies form and the
routing table is populated.

N7K8# show fabricpath isis adjacency

Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:


System ID

SNPA

Level

00:00:30

Ethernet2/25 N7K7

00:00:28

Ethernet2/26

State

Hold Time
N/A

Interface N7K7
1 UP

N7K8# show fabricpath route

FabricPath Unicast Route Table


'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/78/0, number of next-hops: 0


via ---- , [60/0], 0 day/s 00:03:10, local 1/ 77
/0, number of next-hops: 2 via Eth2/25
, [115/40], 0 day/s 00:00:15, isis_fabricpath-default via Eth2/26
, [115/40], 0 day/s 00:00:15, isis_fabricpath-default

N/A

1 UP

Nexus Technology Labs - FabricPath


FabricPath Root Tree Election
Task
Configure basic layer 2 connectivity between the Nexus 7Ks and 5Ks as follows:
Create VLANs 10 and 20 on N7K1, N7K2, N5K1, & N5K2.
Create SVIs for VLANs 10 and 20 on N5K1 & N5K2. Use IP addresses
10.0.0.5Y/24 and 20.0.0.5Y/24, where Y is the 5K's device number per the
diagram.
Configure trunk links between N7K1 & N5K1, and between N7K2 & N5K2.
All other links on the 5Ks should be disabled.
Configure FabricPath on N7K1 & N7K2 as follows:
Install and enable the FabricPath feature set.
Configure VLANs 10 & 20 as FabricPath VLANs.
Set the FabricPath Switch-ID to by 7Y, where Y is the 7K's device number
per the diagram.
Enable FabricPath on the F1 ports between the 7Ks; the links down to the
5Ks will be running Classical Ethernet.
Configure N7K1 to be the root of the first multi-destination tree for the
default FabricPath topology.
Configure N7K2 to be the root of the second multi-destination tree for the
default FabricPath topology.
Ensure that the 7Ks are the root of the Spanning-Tree for both VLANs 10 &
20.
Once complete, N5K1 & N5K2 should have IP reachability between their VLAN 10 &
20 interfaces, and the 7Ks should be routing this traffic over FabricPath between
each other.

Configuration
N5K7:
interface Ethernet1/1 - 32
shutdown

!
feature interface-vlan
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N5K8:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N7K7:
feature-set fabricpath
!

vlan 10,20
mode fabricpath
!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 77
!
fabricpath domain default
root-priority 255
!
interface Ethernet2/25
switchport mode fabricpath
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown
N7K8:

feature-set fabricpath
!
vlan 10,20
mode fabricpath
!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 78
!
fabricpath domain default
root-priority 254
!
interface Ethernet2/25
switchport mode fabricpath
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
no shutdown

!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown

Verification
The FabricPath Multi-Destination tree is used for forwarding broadcast, unknown
unicast, and multicast traffic over the FabricPath network. Each FabricPath topology
has two trees, each denoted with a unique ftag. The first of these trees is used to
forward broadcast and unknown unicast, while the second tree is for forwarding
multicast. In a real deployment, the roots of the trees should be placed on the
FabricPath Spine nodes.
To influence the tree election, the root-priority can be set under the global
FabricPath process. The device with the highest priority is elected root for tree 1,
and the next-highest priority device is elected root for tree 2. The result of the
election can be verified as follows:
N7K7# show fabricpath isis topology summary
FabricPath IS-IS Topology Summary
Fabricpath IS-IS domain: default MT-0
Configured interfaces:
Max number of trees: 2

Ethernet2/25

Ethernet2/26

Number of trees supported: 2

Tree id: 1, ftag: 1, root system: 64a0.e742.8dc5, 77


Tree id: 2, ftag: 2, root system: 4055.390b.c145, 78

Ftag Proxy Root: 64a0.e742.8dc5

The field MT-0 means Multi-Topology 0, which is the default topology that all
FabricPath links belong to. The root of the trees are denoted by their FabricPath
Switch-IDs, which in this case have been statically assigned. The below output
shows the specific IS-IS cost values to reach the roots of the trees. Note that a
metric of 0 to the root means that device is the root for that tree.
Per the below output we can see that Switch-ID 78 has a cost of 40 to reach the root
of tree 1, while 78 is the root for tree 2 (it has a cost of 0 to itself).
N7K7# show fabricpath isis trees

Fabricpath IS-IS domain: default


Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id
*:directly connected neighbor or link
P:Physical switch-id, E:Emulated, A:Anycast

MT-0 Topology 0, Tree 1


, Swid routing table
78, L1
via Ethernet2/26, metric 40
Topology 0, Tree 2
, Swid routing table 78
, L1 via Ethernet2/26, metric 0

Nexus Technology Labs - FabricPath


FabricPath Path Selection
Task
Configure basic layer 2 connectivity between the Nexus 7Ks and 5Ks as follows:
Create VLANs 10 and 20 on N7K1, N7K2, N5K1, & N5K2.
Create SVIs for VLANs 10 and 20 on N5K1 & N5K2. Use IP addresses
10.0.0.5Y/24 and 20.0.0.5Y/24, where Y is the 5K's device number per the
diagram.
Configure trunk links between N7K1 & N5K1, and between N7K2 & N5K2.
All other links on the 5Ks should be disabled.
Configure FabricPath on N7K1 & N7K2 as follows:
Install and enable the FabricPath feature set.
Configure VLANs 10 & 20 as FabricPath VLANs.
Set the FabricPath Switch-ID to by 7Y, where Y is the 7K's device number
per the diagram.
Enable FabricPath on the F1 ports between the 7Ks; the links down to the
5Ks will be running Classical Ethernet.
Modify the FabricPath path selection so that 7K1 prefers to route to 7K2
over their first link, while 7K2 prefers to route to 7K1 over their second link.
Ensure that the 7Ks are the root of the Spanning-Tree for both VLANs 10 &
20.
Once complete, N5K1 & N5K2 should have IP reachability between their VLAN 10 &
20 interfaces, and the 7Ks should be routing this traffic over FabricPath between
each other.

Configuration
N5K7:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan

!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N5K8:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown
ip address 20.0.0.58/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N7K7:
feature-set fabricpath
!
vlan 10,20
mode fabricpath

!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 77
!
interface Ethernet2/25
switchport mode fabricpath
fabricpath isis metric 20
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown
N7K8:

feature-set fabricpath
!
vlan 10,20
mode fabricpath
!
spanning-tree vlan 10,20 priority 0
!
fabricpath switch-id 78
!
interface Ethernet2/25
switchport mode fabricpath
no shutdown
!
interface Ethernet2/26
switchport mode fabricpath
fabricpath isis metric 20
no shutdown
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28

switchport mode trunk


no shutdown

Verification
Modifying FabricPath path selection works the same way as in OSPF or IS-IS for
regular IP routing, where the cost value is a function of the link (i.e. the Link State)
as opposed to the route itself (as in EIGRP or BGP). Therefore in order to influence
path selection, we can simply raise or lower the IS-IS metrics on a per-link basis.
Since we are currently using only one default topology, modifying the
cost of a link affects the path selection of all VLANs being routed over
FabricPath.
Prior to modifying metrics, the 7Ks install ECMP routes to each others Switch-IDs.
N7K7# show fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/77/0, number of next-hops: 0


via ---- , [60/0], 0 day/s 00:02:37, local 1/ 78
/0, number of next-hops: 2 via Eth2/25, [115/40]
, 0 day/s 00:01:29, isis_fabricpath-default via Eth2/26, [115/40]
, 0 day/s 00:01:29, isis_fabricpath-default

After modification of the metrics, only the lowest cost path is installed in the routing
table.
N7K7# config t
Enter configuration commands, one per line.

End with CNTL/Z.N7K7(config)# int e2/25

N7K7(config-if)# fabricpath isis metric 20


N7K7(config-if)# end
N7K7# 2015 Feb 13 21:04:37 N7K7 last message repeated 1 time
N7K7# show fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]

ftag 0 is local ftag


subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/77/0, number of next-hops: 0


via ---- , [60/0], 0 day/s 00:03:09, local 1/ 78
/0, number of next-hops: 1 via Eth2/25, [115/20]
, 0 day/s 00:02:01, isis_fabricpath-default

The result of this change can be seen in the FabricPath IS-IS database.
N7K7# show fabricpath isis database detail
Fabricpath IS-IS domain: default LSP database
LSPID

Seq Number

.00-00

0x00000010

0x6324

Instance

0x0000000A

Area Address

00

NLPID

0xC0

Hostname

N7K8

Extended IS

Checksum
1138

N7K7.00

Capability

A/P/O/T N7K8

0/0/0/1

Length : 4

Extended IS

Lifetime

Metric : 40
N7K7.00

Metric : 20

: Device Id: 78 Base Topology

Base Topo Trees :


Trees desired: 2
Nickname

Trees computed: 2

Trees usable: 2

Priority: 0 Nickname: 78 BcastPriority: 64


Version

Version: 1 Flags: 0
Nickname Migration :
Swid: 78 Sec. Swid: 0
Digest Offset :
.00-00

0 N7K7

* 0x00000010

0x9AEA

Instance

0x00000010

Area Address

00

NLPID

0xC0

Hostname

N7K7

Extended IS

:
:

0/0/0/1

Length : 4
N7K8.00

Extended IS

Capability

1139

Metric : 20
N7K8.00

Metric : 40

: Device Id: 77 Base Topology

Base Topo Ftag

Graph 1: Root: N7K7 Primary: 1, Secondary: 0 Nickname 77

Graph 2: Root: N7K8 Primary: 2, Secondary: 0 Nickname 78


Base Topo Trees :
Trees desired: 2

Trees computed: 2

Trees usable: 2

Base Topo Roots :


Graph 1: Root Nickname: 77
Graph 2: Root Nickname: 78
Nickname

Priority: 0 Nickname: 77 BcastPriority: 64


Version

Version: 1 Flags: 0
Nickname Migration :
Swid: 77 Sec. Swid: 0
Digest Offset :

Nexus Technology Labs - FabricPath


FabricPath Multi Topology Routing
Task
Configure basic layer 2 connectivity between the Nexus 7Ks and 5Ks as follows:
Create VLANs 10, 20, and 30 on N7K1, N7K2, N5K1, & N5K2.
Create SVIs for VLANs 10, 20, and 30 on N5K1 & N5K2. Use IP addresses
10.0.0.5Y/24, 20.0.0.5Y/24, and 30.0.0.5Y/24, where Y is the 5K's device
number per the diagram.
Configure trunk links between N7K1 & N5K1, and between N7K2 & N5K2.
All other links on the 5Ks should be disabled.
Configure FabricPath on N7K1 & N7K2 as follows:
Install and enable the FabricPath feature set.
Configure VLANs 10, 20, & 30 as FabricPath VLANs.
Set the FabricPath Switch-ID to by 7Y, where Y is the 7K's device number
per the diagram.
Enable FabricPath on the F1 ports between the 7Ks; the links down to the
5Ks will be running Classical Ethernet.
Ensure that the 7Ks are the root of the Spanning-Tree for VLANs 10, 20, &
30.
Configure FabricPath Multi-Topology Routing as follows:
Create two new topology on N7K1 and N7K2.
VLAN 10 should belong to the first new topology.
VLAN 20 should belong to the second new topology.
All other VLANs should belong to the default topology.
The first new topology should only route over the first link between the 7Ks.
The second new topology should only route over the second link between
the 7Ks.
Once complete, N5K1 & N5K2 should have IP reachability between their VLAN 10,
20, & 30 interfaces; the 7Ks should be installing only one route for VLAN 10, only
one route for VLAN 20, but two ECMP routes for VLAN 30 over FabricPath.

Configuration
N5K7:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20,30
!
interface Vlan10
no shutdown
ip address 10.0.0.57/24
!
interface Vlan20
no shutdown
ip address 20.0.0.57/24
!
interface Vlan30
no shutdown
ip address 30.0.0.57/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N5K8:
interface Ethernet1/1 - 32
shutdown
!
feature interface-vlan
!
vlan 10,20,30
!
interface Vlan10
no shutdown
ip address 10.0.0.58/24
!
interface Vlan20
no shutdown

ip address 20.0.0.58/24
!
interface Vlan30
no shutdown
ip address 30.0.0.58/24
!
interface Ethernet1/6
switchport mode trunk
no shutdown
!
interface Ethernet1/7
switchport mode trunk
no shutdown
N7K7:
feature-set fabricpath
!
vlan 10,20,30
mode fabricpath
!
fabricpath topology 1
member vlan 10
!
fabricpath topology 2
member vlan 20
!
spanning-tree vlan 10,20,30 priority 0
!
fabricpath switch-id 77
!
interface Ethernet2/25
fabricpath topology-member 1
switchport mode fabricpath
!
interface Ethernet2/26
fabricpath topology-member 2
switchport mode fabricpath
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown
N7K8:

feature-set fabricpath
!
vlan 10,20,30
mode fabricpath
!
fabricpath topology 1
member vlan 10
!
fabricpath topology 2
member vlan 20
!
spanning-tree vlan 10,20,30 priority 0
!
fabricpath switch-id 78
!
interface Ethernet2/25
fabricpath topology-member 1
switchport mode fabricpath
!
interface Ethernet2/26
fabricpath topology-member 2
switchport mode fabricpath
!
interface Ethernet2/27
switchport mode trunk
no shutdown
!
interface Ethernet2/28
switchport mode trunk
no shutdown

Verification
This feature comes from the legacy Multi-Topology Routing (MTR) feature of link
state protocols such as OSPF and IS-IS, which was originally designed to allow for
class-based forwarding without the need for MPLS Traffic Engineering. In the case
of FabricPath, Multiple Topologies allow you to control path selection on a per-VLAN
basis or per a group of VLANs, analogous to how Multiple Spanning-Tree path
selection modification works.
A FabricPath Topology is made up of links and VLANs assigned to the topology.
SPF is run separately for each topology, which means that path selection can be
modified from one topology independent of another. The end result of this

configuration is that there are multiple routing tables, one for each topology, as seen
below:
N7K7# show fabricpath route topology all
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/77/0, number of next-hops: 0


via ---- , [60/0], 0 day/s 00:14:12, local 1/ 78
/0, number of next-hops: 2 via Eth2/25
, [115/40], 0 day/s 00:10:33, isis_fabricpath-default via Eth2/26
, [115/40], 0 day/s 00:08:59, isis_fabricpath-default

FabricPath Unicast Route Table for Topology 1

0/77/0, number of next-hops: 0


via ---- , [60/0], 0 day/s 00:14:12, local 3/ 78
/0, number of next-hops: 1 via Eth2/25
, [115/40], 0 day/s 00:02:59, isis_fabricpath-default

FabricPath Unicast Route Table for Topology 2

0/77/0, number of next-hops: 0


via ---- , [60/0], 0 day/s 00:14:12, local 5/ 78
/0, number of next-hops: 1 via Eth2/26
, [115/40], 0 day/s 00:02:57, isis_fabricpath-default

Topologies, their associated links, and SPF metrics can be verified as follows.
All links running FabricPath always belong to the Default topology,
MT-0.

N7K7# show fabricpath isis topology


FabricPath IS-IS Topology
Fabricpath IS-IS domain: default MT-0
Fabricpath IS-IS Graph 0 Level-1 for MT-0 IS routing table
N7K8.00, Instance 0x00000020
, metric 40
, metric 40
MT-1

*via N7K8, Ethernet2/25

*via N7K8, Ethernet2/26

Fabricpath IS-IS Graph 0 Level-1 for MT-1 IS routing table


N7K8.00, Instance 0x00000009

*via N7K8, Ethernet2/25

, metric 40
MT-2
Fabricpath IS-IS Graph 0 Level-1 for MT-2 IS routing table
N7K8.00, Instance 0x00000009

*via N7K8, Ethernet2/26

, metric 40

VLAN to topology mappings can be verified as follows.


N7K8# show fabricpath topology vlan

Topo-Description

Topo-ID

Configured VLAN List

-------------------------------- --------- ------------------------------------0

1-9, 11-19, 21-4095

10

20

For each new topology in FabricPath, a new Root Tree election is performed. Note
that each tree gets a unique ftag value, which is used to classify packets in the data
plane as part of the FabricPath header encapsulation.
N7K7# show fabricpath isis topology summary
FabricPath IS-IS Topology Summary
Fabricpath IS-IS domain: default MT-0
Configured interfaces:
Max number of trees: 2

Ethernet2/25

Ethernet2/26

Number of trees supported: 2

Tree id: 1, ftag: 1, root system: 64a0.e742.8dc5, 77


Tree id: 2, ftag: 2, root system: 4055.390b.c145, 78
Ftag Proxy Root: 64a0.e742.8dc5 MT-1
Configured interfaces:
Max number of trees: 2

Ethernet2/25
Number of trees supported: 2

Tree id: 1, ftag: 3, root system: 64a0.e742.8dc5, 77


Tree id: 2, ftag: 4, root system: 4055.390b.c145, 78
Ftag Proxy Root: 64a0.e742.8dc5 MT-2
Configured interfaces:
Max number of trees: 2

Ethernet2/26
Number of trees supported: 2

Tree id: 1, ftag: 5, root system: 64a0.e742.8dc5, 77


Tree id: 2, ftag: 6, root system: 4055.390b.c145, 78

Ftag Proxy Root: 64a0.e742.8dc5