Disjointed Layer 2 Networks in X-UCS Intersight MM

#CiscoLive
Disjointed Layer 2 Networks in

Intersight Managed Mode (IMM)
Chance Hinchman
TACDCN-2005
#CiscoLive
Cisco Webex App
Questions?
Use Cisco Webex App to chat
with the speaker after the session
How
https://eurl.io/#EMbNoJMRn
I will be moderating the WebEx space

until June 9, 2023.
#CiscoLive TACDCN-2005 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Objective
• Disjointed Layer 2 configuration is a common task for server
administrators. However, a simple misconfiguration could lead to
major network outage. This presentation will be focused on
implementation of Disjointed Layer 2 in Intersight Managed Mode, as
opposed to UCS Manager. We will cover common misconfiguration
scenarios, how to identify those issues, and how to avoid those
configuration mistakes.
• Ideally the target audience for this presentation will have a working
knowledge of UCS in IMM.
#CiscoLive TACDCN-2005 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
• Disjointed Layer 2 Concept

• Disjointed Layer 2 Implementation
• Misconfiguration Scenarios and Impact
• Overlapping VLANs
• ENM Source Pin Fail
Disjointed Layer 2
Concept
Disjointed Layer 2 Concept
• By default, UCS data traffic works on the principle of mutual inclusion. All
traffic for all VLANs travels along all uplinks.
• Disjoint Layer 2 works on the principle of selective exclusion. Traffic for a

VLAN that is designated as part of a disjoint network can only travel along
an uplink that is specifically assigned to that VLAN, and is selectively
excluded from other uplinks.
*This applies to Fabric Interconnects in End Host Mode, which is the default.
Disjointed Layer 2 Topology
Prod
DMZ
Newly added Preexisting
VLANs 20-30 VLANs 101-998
FI-A FI-B
interface Ethernet1/15 interface Ethernet1/15

allowed vlan 1,20-30 allowed vlan 1,20-30

allowed vlan 1,101-998 allowed vlan 1,101-998
chassis
No overlap between Prod
Server 3/1 A
Server 3/1 B I.E., the selective exclusion
and DMZ VLANs on either
the uplinks or vNIC's vNIC eth0
vNIC eth1 that we mentioned earlier
vlan 1,101-998
vlan 1,101-998
vNIC eth3
vNIC eth2
vlan 1,20-30
vlan 1,20-30
Disjointed Layer 2
Implementation
Deploying Disjoint Layer 2 Order of Operations
• When adding disjoint VLANs, deploy the domain profile first, and confirm it
is successful before deploying the server profile. If the VLANs are added to
the server vNICs before the uplinks, a pinning failure may occur.
• When removing disjoint VLANs the order of operations is reversed. Remove

the the VLANs from the vNICs first, and deploy the server profile, before
removing the VLANs from the uplinks and deploying the domain profile.
A breakdown of the configuration steps
1. Add the disjoint VLAN range to the VLAN Configuration Policy used in the domain profile
2. Create Ethernet Network Groups
3. Apply the Network Groups to the appropriate Port Policy
4. Apply the Network Groups to the LAN Connectivity Policy
5. Verify that the configuration is successful
Configure and Deploy
Domain Profile
Adding Disjoint VLANs
Creating Ethernet Network Groups (Prod)
Although the production VLANs were preexisting

they were not previously in a network group.
Again, by default, UCS data traffic works on the

principle of mutual inclusion.
Creating Ethernet Network Groups (DMZ)
Adding Network Group to Uplinks (DMZ)
Remember, we disabled the “auto allow on

uplink” option in the VLAN configuration
policy.
Adding Network Group to Uplinks (Prod)
Deploy Domain Profile
Deploy Domain Profile
Configure and Deploy
LAN Connectivity Policy
Adding Network Group to LAN Connectivity Policy (DMZ)
You'll want at least 2 vNIC's for each side of

the fabric, one vNIC for each disjoint network.
Adding Network Group to LAN Connectivity Policy (Prod)
You'll rinse and repeat for all vNIC's using the DMZ
and Prod ethernet network groups accordingly.
Deploy Server Profile
Deploy Server Profile
Verifying the
configuration
Determining a Servers vNICs
F340-24-21-IMM-1-A# connect nxos F340-24-21-IMM-1-B# connect nxos
F340-24-21-IMM-1-A(nx-os)# show run interface | grep prev 1 FCH251372LZ F340-24-21-IMM-1-B(nx-os)# show run interface | grep prev 1 FCH251372LZ
interface Vethernet800 interface Vethernet803
description SP chhinchm-1, vNIC eth0, Blade:FCH251372LZ description SP chhinchm-1, vNIC eth1, Blade:FCH251372LZ
-- --
• GREPing the running config for the serial number of a server is an

easy way to determine that servers vNICs and Vethernet interfaces.
Confirming vNIC Programming
F340-24-21-IMM-1-A(nx-os)# show running-config interface vethernet 800 F340-24-21-IMM-1-B(nx-os)# show running-config interface vethernet 803

no lldp transmit no lldp transmit
no lldp receive no lldp receive
no pinning server sticky no pinning server sticky
pinning server pinning-failure link-down pinning server pinning-failure link-down
no cdp enable no cdp enable
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan 1,101-998 switchport trunk allowed vlan 1,101-998
no hardware vethernet mac filtering per-vlan no hardware vethernet mac filtering per-vlan
bind interface port-channel1319 channel 800 bind interface port-channel1319 channel 803
service-policy type qos input default-IMM-QoS service-policy type qos input default-IMM-QoS
no shutdown no shutdown

Confirming FI Configuration and Veth Pinning
F340-24-21-IMM-1-A(nx-os)# show running-config interface ethernet 1/15-16 F340-24-21-IMM-1-B(nx-os)# show running-config interface ethernet 1/15-16

description Uplink description Uplink
pinning border pinning border
udld disable udld disable

F340-24-21-IMM-1-A(nx-os)# show pinning server-interfaces | include Veth F340-24-21-IMM-1-B(nx-os)# show pinning server-interfaces | include Veth
---------------+-----------------+------------------------+----------------- ---------------+-----------------+------------------------+-----------------
SIF Interface Sticky Pinned Border Interface Pinned Duration SIF Interface Sticky Pinned Border Interface Pinned Duration
---------------+-----------------+------------------------+----------------- ---------------+-----------------+------------------------+-----------------
Veth800 No Eth1/16 1:14:42
Veth807 No - - Veth803 No Eth1/16 1:20:5
Veth810 No - - Veth808 No - -
Veth813 No Eth1/15 1:14:42 Veth812 No - -
Veth822 No Eth1/15 1:20:5
Failure Scenario 1
Overlapping VLANs
Failure Scenario 1 – Overlapping VLANs
I then apply the network group to the

DMZ uplinks but not the prod uplink.
DMZ Prod
20-30,101-998 Overlapping VLANs VLANs 101-998
Attempting to allow both DMZ and

Prod VLANs on the DMZ uplinks fails.
FI-A FI-B
chassis
Server 3/1 A FI-B
vNIC eth0 interface Ethernet1/15

vlan 1,101-998 allowed vlan 1,20-30
vNIC eth2 interface Ethernet1/16

vlan 1,20-30 allowed vlan 1,101-998
This mistake is not service impacting in IMM. Intersight catches

this misconfiguration and deployment of the domain profile fails.
• The absence of Spanning Tree on the FIs in End Host Mode means we
rely on other mechanisms to avoid loops. This mechanism is referred to
as the Designated Receiver.
• Broadcast and multicast traffic is pinned on a per-VLAN basis to an uplink

port; it is dropped when received on other uplinks.
• Overlapping VLANs on uplinks could cause multicast and/or broadcast

traffic to be black holed. Configurations that may result in this behavior are
not permitted.
Determining a VLANs Designated Receiver
F340-24-21-IMM-1-A(nx-os)# show platform software enm internal info F340-24-21-IMM-1-B(nx-os)# show platform software enm internal info
vlandb id 20 vlandb id 20
vlan_id 20 vlan_id 20
------------- -------------
Designated receiver: Eth1/15 Designated receiver: Eth1/15
Membership: Membership:
Eth1/15 Eth1/15
F340-24-21-IMM-1-A(nx-os)# show platform software enm internal info F340-24-21-IMM-1-B(nx-os)# show platform software enm internal info
vlandb id 101 vlandb id 101
vlan_id 101 vlan_id 101

------------- -------------
Designated receiver: Eth1/16 Designated receiver: Eth1/16
Membership: Membership:
Eth1/16 Eth1/16
• This can be a useful command when t-shooting network connectivity issues in DJL2 networks.
• For instance, if you were performing a packet capture or SPAN and you wanted to know
what interface broadcast traffic should be received on.
Overlapping VLANs - Remediation
• Remove Production VLANs from the DMZ uplinks and reapply Domain Profile.
Failure Scenario 2
ENM Source Pin Fail
Failure Scenario 2 – ENM Source Pin Fail
In this scenario the uplinks Prod
DMZ
VLANs 20-30
carry EITHER production or VLANs 101-998
DMZ VLANs…
FI-A FI-B
…while the vNIC's allow both

chassis
production AND DMZ VLANs.
The uplinks are disjoint
while the vNIC's are not. Server 3/1 A Server 3/1 B
vNIC eth0 vNIC eth1
Unable to pin. vlan 1,20-30,101-998 vlan 1,20-30,101-998
F340-24-21-IMM-1-A(nx-os)# show interface brief | include Veth F340-24-21-IMM-1-B(nx-os)# show interface brief | include Veth
Vethernet VLAN Type Mode Status Reason Vethernet VLAN Type Mode Status Reason
Speed Speed
Veth800 1 virt trunk down ENM Source Pin Fail auto Veth803 1 virt trunk down ENM Source Pin Fail auto
Veth807 110 virt trunk down nonPartcipating auto Veth808 110 virt trunk down nonPartcipating auto
Veth821 1010 virt access down nonPartcipating auto Veth820 1011 virt access down nonPartcipating auto
ENM = End Node Manager

Interface down reason “ENM Source Pin Fail” is indicative of DJL2 misconfiguration.
F340-24-21-IMM-1-A(nx-os)# show running-config interface ethernet 1/15-16 F340-24-21-IMM-1-B(nx-os)# show running-config interface ethernet 1/15-16

we can see that we have a
description Uplink disjoint upstream network description Uplink

switchport trunk allowed vlan 1,20-30,101-998 switchport trunk allowed vlan 1,20-30,101-998
however, we do not have disjoint vNICs
no selective exclusion
ENM Source Pin Fail - Remediation
Create additional vNICs for disjoint networks in the LAN connectivity

policy, then apply Ethernet Network Groups accordingly.
Resources
Configure Disjoint Layer 2 in Intersight Managed Mode Domain

https://www.cisco.com/c/en/us/support/docs/cloud-systems-
management/intersight/217804-configure-disjoint-layer-2-in-intersight.html
Deploy Layer 2 Disjoint Networks Upstream in End-Host Mode

https://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-
virtualization/unified-computing/white_paper_c11-692008.pdf
#CiscoLive TACDCN-2005 © 2022

2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fill out your session surveys!
Attendees who fill out a minimum of four session

surveys and the overall event survey will get
Cisco Live-branded socks (while supplies last)!
Attendees will also earn 100 points in the

Cisco Live Game for every survey completed.
These points help you get on the leaderboard and increase your chances of winning daily and grand prizes
• Visit the Cisco Showcase
for related demos
• Book your one-on-one

Meet the Engineer meeting
• Attend the interactive education

with DevNet, Capture the Flag,
Continue and Walk-in Labs
your education • Visit the On-Demand Library

for more sessions at
www.CiscoLive.com/on-demand
TACDCN-2005 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Webex App
Questions?
Use Cisco Webex App to chat
with the speaker after the session
How
https://eurl.io/#EMbNoJMRn
I will be moderating the WebEx space

until June 9, 2023.
#CiscoLive TACDCN-2005 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Thank you
#CiscoLive
#CiscoLive

Disjointed Layer 2 Networks in X-UCS Intersight MM

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Disjointed Layer 2 Networks in X-UCS Intersight MM

Uploaded by

Copyright:

Available Formats

#CiscoLive

Disjointed Layer 2 Networks in

I will be moderating the WebEx space

• Disjointed Layer 2 Concept

• Disjoint Layer 2 works on the principle of selective exclusion. Traffic for a

interface Ethernet1/15 interface Ethernet1/15

interface Ethernet1/16 interface Ethernet1/16

• When removing disjoint VLANs the order of operations is reversed. Remove

2. Create Ethernet Network Groups

3. Apply the Network Groups to the appropriate Port Policy

4. Apply the Network Groups to the LAN Connectivity Policy

5. Verify that the configuration is successful

Although the production VLANs were preexisting

Again, by default, UCS data traffic works on the

Remember, we disabled the “auto allow on

You'll want at least 2 vNIC's for each side of

• GREPing the running config for the serial number of a server is an

interface Vethernet800 interface Vethernet803

interface Vethernet813 interface Vethernet822

interface Ethernet1/15 interface Ethernet1/15

interface Ethernet1/16 interface Ethernet1/16

I then apply the network group to the

20-30,101-998 Overlapping VLANs VLANs 101-998

Attempting to allow both DMZ and

Server 3/1 A FI-B

vNIC eth0 interface Ethernet1/15

vNIC eth2 interface Ethernet1/16

This mistake is not service impacting in IMM. Intersight catches

• Broadcast and multicast traffic is pinned on a per-VLAN basis to an uplink

• Overlapping VLANs on uplinks could cause multicast and/or broadcast

vlan_id 101 vlan_id 101

…while the vNIC's allow both

ENM = End Node Manager

interface Ethernet1/15 interface Ethernet1/15

interface Vethernet800 interface Vethernet803

Create additional vNICs for disjoint networks in the LAN connectivity

Configure Disjoint Layer 2 in Intersight Managed Mode Domain

Deploy Layer 2 Disjoint Networks Upstream in End-Host Mode

#CiscoLive TACDCN-2005 © 2022

Attendees who fill out a minimum of four session

Attendees will also earn 100 points in the

• Book your one-on-one

• Attend the interactive education

your education • Visit the On-Demand Library

I will be moderating the WebEx space

You might also like