- BOM HCI, TOR switch part

Q2F22A HPE SN2410M 25GbE 48SFP28 8QSFP28 Switch 2

Q7F11A HPE M-series 40GbE QSFP28 SR4 100m Transceiver 4
844477-B21 HPE 25Gb SFP28 to SFP28 3m Direct Attach Copper Cable 16
H7J34A3 HPE 3Y Foundation Care 24x7 SVC 1
H7J34A3 RCD HPE SN2410M Storage Switch Support 2
Shady: ask Amr/Servers team about how many cables will be connected between two TOR switches for high availability

- Review IP scheme file

Sohag university

Name of Subnet Subnet Mask Vlan

Vmware Subnets
OOB Vlan for servers iDRAC &
TORs 10.93.1.0 /24 901 Shady: this will be used in switches OOB
MGMT 10.93.2.0 /24 902
Vmotion-vlan 10.93.4.0 /24 904
Vsan-vlan 10.93.5.0 /24 905
Production-1 vlan 10.93.6.0 /24 906
Production-2 vlan 10.93.7.0 /24 907

Network Subnets
P2P between Distribution and
router 192.168.93.0 /27 601
MGMT vlan for Network
devices 172.16.93.0 /24 1000 Shady: this is Iband management VLAN

Users Subnets
Thin Clients Vlan (Users) 10.93.16.0 /20 900

Security Subnets
External WAF subnet (Off
Campus Sites) 10.93.32.0 /24 908
Internal WAF subnet (Off
Campus Sites) 10.93.33.0 /24 909
In Campus WAF Subnet 10.93.35.0 /24 910  

Caching Server Subnets

Vlan for MGMT of servers. 10.93.36.0 /24 911
Vlan for Hypervisor of servers. 10.93.37.0 /24 912
Application Vlan 1 10.93.38.0 /24 913
Application Vlan 2 10.93.39.0 /24 914
Application Vlan 3 10.93.40.0 /24 915
Application Vlan 4 10.93.41.0 /24 916
Application Vlan 5 10.93.42.0 /24 917
Application Vlan 6 10.93.43.0 /24 918

System Subnets
Type IP Mask Vlan Domain Name Space NetBiosName
Additional Active Directory 10.93.34.0 /24 919 EEXAM.LOCAL SohagEE

- Server team responsibilities (Shady):

o Connections between different components
o Switches mounting in the rack
o Cable’s handling and connection within the rack
o All needed VLANs, IP addresses and physical connections

- Configuration items (expected, Shady):

o VLAN
o Inter-VLAN routing
o Stacking (ask for stacking cables)
o MLAG ports
o Layer 2
- Multi Chassis LAG (MLAG)
- VLAN
o Layer 3
- IP Addressing
- MAGP
Chapter: Getting Started
Configuring the Switch for the First Time
To initialize the switch do the following:

1. Connect the host PC to the console (RJ-45) port of the switch system using the supplied cable

2. Configure a serial terminal with the settings described below.

3. The boot menu is prompted.

Onyx Boot Menu:
1: <image #1>
2: <image #2>
u: USB menu (if USB device is connected) (password required)
c: Command prompt (password required)
Choice:
The boot menu features a countdown timer. It is recommended to allow the timer to run out by not selecting
any of the options.

4. Login as admin and use admin as password.

5. Go through the Switch Management configuration wizard. (check page 56 for details)
a. Shady: we should start with page 58 for “static configuration” this is for management port (OOB)
b. IP address for management port (Shady: from IP Scheme):
i. SOHAG configured IP address MGMT port: 10.93.1.1/24 for 1st TOR switch
ii. SOHAG configured IP address MGMT port: 10.93.1.2/24 for 2nd TOR switch
c. Passwords for Admin: ………… (Shady: as we agreed don’t share with anyone)

Step 1: Hostname? [switch-112126]

Step 2: Use DHCP on mgmt0 interface? [yes] n

Step 3: Use zeroconf on mgmt0 interface? [no]

Step 4: Primary IP address? 10.93.1.1 (for 1st switch) & 10.93.1.2 (for 2nd switch)

Mask length may not be zero if address is not zero (interface mgmt0)

Step 5: Netmask? [0.0.0.0] 255.255.255.0

Step 6: Default gateway? 10.93.1.254

Step 7: Primary DNS server?

Step 8: Domain name?

Step 9: Enable IPv6? [yes] no

Step 10: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] no

Step 11: Update time? [yyyy/mm/dd hh:mm:ss]

Step 12: Enable password hardening? [yes] yes

Step 13: Admin password (Enter to leave unchanged)? Shady: use a one and write it down & don’t share with anyone

6. Check the mgmt0 interface configuration before attempting a remote (for example, SSH) connection to the
switch. Specifically, verify the existence of an IP address.

switch # show interfaces mgmt0

To rerun the wizard:

switch > enable

switch # config terminal

switch (config) # configuration jump-start

Starting the Command Line (CLI):

Set up an Ethernet connection between the switch and a local network machine using a standard RJ-45 connector.

Start a remote secured shell (SSH) to the switch using the command “ssh -l <username> <switch ip address>”.
rem_mach1 > ssh -l <username> <ip address>

Log into the switch (default username is admin, password admin).

Read and accept the EULA when prompted. Once the following prompt appears, the system is ready to use.
Chapter: System Management
OOB Management interface: interface mgmt0
switch (config) # interface mgmt0 ip address <IP address> <netmask> (already done during startup)

switch # show interfaces mgmt0

Default Gateway
To configure manually the default gateway, use the “ip route” command, with “0.0.0.0” as prefix and mask.

The nexthop address must be within the range of one of the IP interfaces on the system.

In-Band Management
switch (config)# vlan 1000
switch (config vlan 10)# name MGMT
switch (config)# interface vlan 1000
switch (config interface vlan 1000)# ip address 172.16.93.1 /24 (for TOR Switch 1)
switch (config interface vlan 1000)# ip address 172.16.93.2 /24 (for TOR Switch 2)
switch (config)# show interfaces vlan 1000

Hostname
switch (config)# hostname SOHAG-HCI-TOR-1 (for TOR Switch 1)
switch (config)# hostname SOHAG-HCI-TOR-2 (for TOR Switch 2)

Upgrading Operating System Software, page 227

Shady: to check the version and see if it close to latest so it is OK and if not, we should upgrade
Shady: latest version I saw is 3.9, please double check

Saving a Configuration File, page 246

switch (config) # configuration write
System Synchronization NTP, page 302
Shady: this is should be done after the WAN router is up and running to sync with it
Chapter: Ethernet Switching
Page 777

VLANs configuration, page 859

switch (config) # vlan 6
switch (config vlan 6) # name ….
switch (config) # interface ethernet 1/22
switch (config interface ethernet 1/22) # switchport mode access
switch (config interface ethernet 1/22) # switchport access vlan 6
or
switch (config interface ethernet 1/35) # switchport mode trunk
By default, a Trunk port is, automatically, a member on all current VLANs.

Link Aggregation Group (LAG), page 824

switch (config) # interface port-channel 1
switch (config interface port-channel 1) # exit
Static LAG
switch (config interface ethernet 1/4) # channel-group 1 mode on
LACP
switch (config interface ethernet 1/4) # channel-group 1 mode active OR passive

MLAG, page 911

multi-chassis link aggregation
Each switch configuration is independent and it is user responsibility to make sure to configure both switches
similarly pertaining MLAG (e.g. MLAG port-channel VLAN membership, static MAC, ACL, etc).

A peered device (host or switch) connecting to switches running an MLAG runs a standard LAG and is unaware
of the fact that the LAG connects to two separate switches.

The MLAG switches share an inter-peer link (IPL) between them for carrying control messages in a steady state
or data packages in failure scenarios. Thus, the bandwidth of the IPL should be defined accordingly. The IPL
itself can be a LAG and may be constructed of links of any supported speed.

The IPL serves the following purposes:

MLAG protocol control – keepalive messages, MAC sync, MLAG port sync, etc.
MLAG port failure – serves redundancy in case of a fallen link on one of the MLAG switches
Layer-3 failure – serves redundancy in case of a failed connection between the MLAG switches and the
rest of the L3 network should there be one

The IPL VLAN interface must be used only for MLAG protocol and must not be used by any other interfaces
(e.g. LAG, Ethernet).

When positioned at the top of rack (ToR) and connecting with a Layer-3 uplink, the MLAG pair acts as the L3
border for the hosts connected to it. To allow default gateway redundancy, both MLAG switches should be
addressed by the host via the same default gateway address.
MLAG uses an IP address (VIP) that points to all MLAG member nodes.
When running MLAG as L2/L3 border point, an MAGP VIP must be deployed as the default GW for MLAG
portchannels (MPOs).
When MLAG is connected through a Layer-2 based uplink, there is no need to apply default gateway
redundancy towards hosts since this function is implemented on the L2/L3 border points of the network. For
more information, refer to the “MAGP” page.
Shady: from this no need for MAGP now, since no connection to layer 3 node.

MLAG Keepalive and Failover

Master election in MLAG is based on the IPs of the nodes taking part of the MLAG. The master elected is that
which has the highest IPL VLAN interface local IP address.
MLAG master/slave roles take effect in fault scenarios such as split-brain, peer faults, and during software
upgrades.
The MLAG pair of switches periodically exchanges a keepalive message on a user configurable interval. If the
keepalive message fails to arrive for three consecutive intervals the switches break into two standalone
switches. In such a case, the remaining active switch begins to act as a standalone switch and assumes that its
previously peering MLAG switch has failed.
To avoid a scenario where failure on the IPL causes both MLAG peers to assume that their peer has failed, a
safety mechanism is maintained based on UDP packets running via the management plane which alerts both
MLAG switches that its peer is alive. In such case where keepalive packets are not received the slave shuts
down its MLAG interfaces and the master becomes a standalone switch in order to avoid misalignment in
MLAG configuration.

MLAG Port Sync

Under normal circumstances, traffic from the IPL cannot pass through the MLAG ports (the IPL is isolated from
the MLAG ports). If one of the MLAG links break, the other MLAG switch opens that isolation and allows traffic
from its peer through the IPL to flow via the MLAG port which accesses the destination of the fallen link.

MLAG Virtual System-MAC

A pair of MLAG switches uses a single virtual system MAC for L2 protocols (such as LACP) operating on the
MLAG ports. This virtual system MAC is served also as the STP bridge ID.
The virtual system MAC is automatically computed based on the MLAG VIP name, but can be manually set
using the command “system-mac”.
MLAG relies on systems to have the same virtual system MAC. Therefore, if a system MAC mismatch is
detected, the slave shuts down its interfaces.
Configuring MLAG, page 916

Configuring L2 MLAG, page 917

Prerequisites:
switch (config)# ip routing
switch (config)# lacp
switch (config)# dcb priority-flow-control enable force
switch (config)# protocol mlag

Configuring the IPL:

switch (config)# vlan 4000
switch (config vlan 4000)#

switch (config)# interface port-channel 1

switch (config interface port-channel 1)#

switch (config)# interface ethernet 1/1 channel-group 1 mode active

 switch (config interface port-channel 1)# ipl 1
switch (config interface port-channel 1)# dcb priority-flow-control mode on force

switch (config)# interface vlan 4000

switch (config interface vlan 4000)#
switch (config interface vlan 4000)# mtu 9216

Set an IP address and netmask for the VLAN interface. Configure IP address for the IPL link on both switches:
The IPL IP address should not be part of the management network, it could be any IP address and subnet that
is not in use in the network. This address is not advertised outside the switch.

On SwitchA, run:
switch (config interface vlan 4000)# ip address 1.1.1.1 /30

On SwitchB, run:
switch (config interface vlan 4000)# ip address 1.1.1.2 /30

On SwitchA, run:
switch (config interface vlan 4000)# ipl 1 peer-address 1.1.1.2

On SwitchB, run:
switch (config interface vlan 4000)# ipl 1 peer-address 1.1.1.1

(Optional) Configure a virtual IP (VIP) for the MLAG. MLAG VIP is important for retrieving peer information.
If you have a mgmt0 interface, the IP address should be within the subnet of the management interface. Do
not use mgmt1. The management network is used for keepalive messages between the switches. The MLAG
domain must be unique name for each MLAG domain. In case you have more than one pair of MLAG switches
on the same network, each domain (consist of two switches) should be configured with different name.
On SwitchA, run:
switch (config)# mlag-vip my-vip ip 10.234.23.254 /24
On SwitchB, run:
switch (config)# mlag-vip my-vip

(Optional) Configure a virtual system MAC for the MLAG. Run:

switch (config)# mlag system-mac 00:00:5E:00:01:5D

Creating an MLAG interface:

switch (config)# interface mlag-port-channel 1
switch (config interface ethernet 1/2)# mlag-channel-group 1 mode on
Shady: it seems that it is the same interface in both switches and configured from single side
switch (config interface mlag-port-channel 1)# no shutdown

Enabling MLAG:
switch [my-vip: master] (config mlag)# no shutdown

When running MLAG as L2/L3 border point, MAGP VIP must be deployed as the default GW for MPOs. For
more information, refer to “MAGP”.
Verifying MLAG Configuration
SX2 [master] (config)# show mlag
Chapter: IP Routing
Page 1121
Onyx supports the following 3 types of IP interfaces:
VLAN interface
switch (config)# vlan 10
switch (config vlan 10)# exit
switch (config)# interface ethernet 1/1
switch (config interface ethernet 1/1)# switchport mode access
switch (config interface ethernet 1/1)# exit
switch (config)# show interface ethernet 1/1 status Port
Operational state UP
There must be at least one interface in the operational state “UP”.
switch (config)# interface vlan 10
switch (config interface vlan 10)#
switch (config interface vlan 10)# ip address 10.10.10.10 /24

switch (config interface vlan 10) # show interfaces vlan 10

Loopback interface

Router port interface

Router Port Interfaces

Router port interface is a regular switch port configured to operate as an L3 interface. Router port interfaces
are assigned an IP address and all L3 commands become applicable to them.
Once configured, router port interfaces no longer partake in the bridging activities of the switch and VLANs
configured on them are separate from the pool allocated for the switch ports.
switch (config)# interface ethernet 1/10
switch (config interface ethernet 1/10)#
switch (config interface ethernet 1/10)# no switchport force
switch (config interface ethernet 1/10)# ip address 100.100.100.100 /24
switch (config interface ethernet 1/10)# show interfaces ethernet 1/10

MAGP, page 1420

Shady: I think this is will be used in case connecting the two MLAG TOR switches to a single layer 3 node.
Multi-active gateway protocol (MAGP) is aimed to solve the default gateway problem when a host is
connected to a set of switch routers (SRs) via MLAG.
The network functionality in that case requires that each SR is an active default gateway router to the host,
thus reducing hops between the SRs and directly forwarding IP traffic to the L3 cloud regardless which SR
traffic comes through.
switch (config)# ip routing
switch (config)# vlan 20
The VLAN cannot be the same one configured for the MLAG IPL, if MLAG is used.
switch (config)# interface ethernet 1/1
switch (config interface ethernet 1/1)# switchport access vlan 20
switch (config)# interface vlan 20
switch (config interface vlan 20)# ip address 11.11.11.11 /8
switch (config interface vlan 20)# no shutdown
switch (config)# protocol magp
switch (config interface vlan 20)# magp 100
switch (config interface vlan 20 magp 100)# ip virtual-router address 11.11.11.254

switch (config) # show magp