How to Deploy a Nexus 1000v lab with a single ESX host.

By Robert Burns CCIE Data Center #37856

*Slight variation works with VMware Workstation/Fusion also.

*Details of third party OS & application installation is beyond the scope of this guide.
Pre-requisites:
-Disable Windows firewalls on your client
-Java must be installed
Example Host List
Host IP
10.85.49.215
10.85.49.216
10.85.49.217
10.85.49.218
10.85.49.219
10.85.49.220

VM Name
RHEL62-Test-1
vCenter-5
ESX5-Nested-1
ESX5-Nested-2
N1000v

Description
Bare Metal ESX host
Redhat test VM
vCenter server
Nested ESX 5.1 (VM 1)
Nested ESX 5.1 (VM 2)
VSM Management IP

1. Infrastructure setup
a. Install ESX bare metal host. Ensure you have > 75GB available space on your VMFS if
you plan on installing a vCenter VM with Update Manager.
b. Configure management network interface.
c. Install one WIN2K8/2012 VM (for vCenter) allocated 30GB or more for virtual disk. Next
install vCenter 5.1 on Windows VM. Alternately you can use the vCenter appliance.
d. On the vCenter server install VMware Update Manager (VUM).
e. Install two ESX 5.1 VMs following Nested ESX Instructions here:
http://www.vcritical.com/2011/07/vmware-vsphere-can-virtualize-itself/
i. Configured four vNICs each using the default vSwitch VM Network port group
for now.
ii. Boot Nested ESX VMs and configure Management network interfaces from the
VM console within VI Client.
f. Test all IP connectivity between vCenter and all three ESX hosts.
g. Add Nested ESX hosts to vCenter.
See Fig. 1

Tip: Put the Nested ESX hosts in their own Cluster for easier management.
Note: The two ESX5-Nested-X VMs below correspond to 10.85.49.218 & 10.85.49.219 hosts connected to vCenter in the ESXNested cluster.

Fig. 1 Infrastructure setup with Nested ESX VMs installed & added to vCenter.

2. Deploy the 1000v VSM

a. Download the latest 1000v SW from CCO.
*For ESX 5.1 and later you must use 1000v version Nexus1000v.4.2.1.SV2.1.1 or later.
http://software.cisco.com/download/release.html?mdfid=282646785&flowid=3090&softwa
reid=282088129&release=4.2%281%29SV2%281.1a%29&relind=AVAILABLE&rellifecycle=&r
eltype=latest

b. Unzip the bundle and navigate to the Install_App folder & launch the application.

c. Start with the VSM Complete Installation selecting Custom install.

d. Review the pre-reqs & click Next.

e. Enter the appropriate details for your vCenter

f.

Enter the appropriate info for your VSM. Since Im hosting the VSM pair on my single
bare metal ESX host, Ive used the same ESX Host IP twice. Click Next when complete.
Notes:
- Recommend setting your Domain ID to something other than default 1.
- Im setting my Management VLAN to what my physical switches use for
their native VLAN.
- I chose to use L2 mode for simplicity versus L3, but either will work.
- Do not migrate hosts at this time.

g. Review configuration and click Next. Be patient, deployment will take up to 15mins.

h. Next step will prompt to add additional Modules.

i.

Select the hosts you wish to have the VEM agent installed. Click Next.
Note: This method requires VUM to be previously installed. If not, youll need to manually install the VEM
agent vibs.

j.

Review and then click Finish to proceed.

You can monitor the progress from the VI Client Recent Task log

k. The Install App hopefully completed successfully for all hosts.

**If the VEM installation fails, it likely points to a problem with VMware Update
Manager (VUM).

l.

From the VI Client go to Home -> Inventory -> Networking and you should see your two
new hosts as part of the 1000v DVS. Ensure you click on the 1000v DVS in the left pane.

m. SSH into the VSM and and check the modules

N1000v(config)# show mod
Mod

Ports

Module-Type

Model

Status

---

-----

--------------------------------

------------------

------------

Virtual Supervisor Module

Nexus1000V

active *

Virtual Supervisor Module

Nexus1000V

ha-standby

248

Virtual Ethernet Module

NA

ok

248

Virtual Ethernet Module

NA

ok

Mod
--

3.

Sw

Hw

------------------

------------------------------------------------

4.2(1)SV2(1.1)

0.0

4.2(1)SV2(1.1)

0.0

4.2(1)SV2(1.1)

VMware ESXi 5.1.0 Releasebuild-838463 (3.1)

4.2(1)SV2(1.1)

VMware ESXi 5.1.0 Releasebuild-838463 (3.1)

Mod

MAC-Address(es)

Serial-Num

---

--------------------------------------

----------

00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8

NA

00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8

NA

02-00-0c-00-03-00 to 02-00-0c-00-03-80

NA

02-00-0c-00-04-00 to 02-00-0c-00-04-80

NA

Mod

Server-IP

Server-UUID

Server-Name

---

---------------

------------------------------------

--------------------

10.85.49.220

NA

NA

10.85.49.220

NA

NA

10.85.49.218

422954ef-1f4d-f096-2a47-4b64cd67b932

10.85.49.218

10.85.49.219

42299b1d-8226-47bc-f375-2432c7cbe87e

10.85.49.219

Migrate Test VM & Test Connectivity

a.

Cold migrate (powered off) your Test VM to one the Nested ESX hosts. If you get any errors, youve likely done one of
the following:
Didnt modify your Nested ESX VM to Virtual Machine Version 9 prior to install
Didnt enable the Expose NX/XD flag to guest option in the Nested ESX VM.

b.

Before we power it up were going to create a Port Profile for it on the 1000v.
N1000v(config)# port-profile type vethernet rhel-pp
N1000v(config-port-prof)# switchport mode access
N1000v(config-port-prof)# switchport access vlan 711
N1000v(config-port-prof)# state enabled
N1000v(config-port-prof)# no shut
N1000v(config-port-prof)# vmware port-group

c.

Now change the virtual network binding of your test VM from the vSwitch to the 1000v port profile.

d.

Power up your test VM, and verify the interface on the 1000v.
N1000v(config)# show interface virtual
------------------------------------------------------------------------------Port
Adapter
Owner
Mod Host
------------------------------------------------------------------------------Veth1
Net Adapter 1 RHEL62-Test-1
3
10.85.49.218
N1000v(config)#

Assuming your networking & port profiles are setup correctly you should have connectivity to your Test VM.

4.

Advanced Configuration (optional)

a.

Now that we have basic connectivity, lets add the remaining uplinks to your Nested ESX VEM hosts.
Select the host Configuration Networking vSphere Distributed Switch tab Manage Physical Adapters

b.

Find the uplink port profile and click Add NIC. Add each of the 2 remaining NICs from each host.

c.

Verify the uplinks on the 1000v. Your uplink port profile should be configured for mac pinning in which case you should
see two new Port channels automatically created.
N1000v(config)# show int brief
-------------------------------------------------------------------------------Port
VRF
Status IP Address
Speed
MTU
-------------------------------------------------------------------------------mgmt0
-up
10.85.49.220
1000
1500
-------------------------------------------------------------------------------Ethernet
VLAN
Type Mode
Status Reason
Speed
Port
Interface
Ch #
-------------------------------------------------------------------------------Eth3/2
711
eth trunk up
none
1000
1
Eth3/3
711
eth trunk up
none
1000
1
Eth3/4
711
eth trunk up
none
1000
1
Eth4/2
711
eth trunk up
none
1000
2
Eth4/3
711
eth trunk up
none
1000
2
Eth4/4
711
eth trunk up
none
1000
2
-------------------------------------------------------------------------------Port-channel VLAN Type Mode
Status Reason
Speed Protocol
Interface
-------------------------------------------------------------------------------Po1
711
eth trunk up
none
a-1000(D) none
Po2
711
eth trunk up
none
a-1000(D) none
<snip>

5.

Exercise - Determine which uplink your Test VM is utilizing

i.

Find which module the VM is hosted by.

N1000v(config)# show int virtual
------------------------------------------------------------------------------Port
Adapter
Owner
Mod Host
------------------------------------------------------------------------------Veth1
Net Adapter 1 RHEL62-Test-1
3
10.85.49.218

ii.

Identify the Sub Group IDs of all uplinks on that host.

N1000v(config)# module vem 3 execute vemcmd show port
LTL
VSM Port Admin Link State PC-LTL SGID Vem Port Type
18
Eth3/2
UP
UP
FWD
305
1
vmnic1
19
Eth3/3
UP
UP
FWD
305
2
vmnic2
20
Eth3/4
UP
UP
FWD
305
3
vmnic3
49
Veth1
UP
UP
FWD
0
2 RHEL62-Test-1.eth0
305
Po1
UP
UP
FWD
0
<snip>
You can see from the output, SGID 1 = vmnic1, 2 = vmnic2 and 3 = vmnic3

iii.

Find the VMs pinned Sub Group ID from the same ouput.
N1000v(config)# module vem 3 execute vemcmd show port
LTL
VSM Port Admin Link State PC-LTL SGID Vem Port Type
18
Eth3/2
UP
UP
FWD
305
1
vmnic1
19
Eth3/3
UP
UP
FWD
305
2
vmnic2
20
Eth3/4
UP
UP
FWD
305
3
vmnic3
49
Veth1
UP
UP
FWD
0
2 RHEL62-Test-1.eth0
305
Po1
UP
UP
FWD
0
<snip>
From this we can see that the Test VM is assigned to SGID 2, which will use vmnic2 for external
communication.

6.

7.

Exercise Force your Test VM to utilize vmnic1.

a.

First determine what the SGID of vmnic1 is. From our previous output, this would be SGID 1.

b.

Configure either the port profile or the individual interface to prefer this Sub Group.
N1000v(config)# port-profile rhel-pp
N1000v(config-port-prof)# pinning id 1

c.

Verify the change.

N1000v(config-port-prof)# module vem 3 execute vemcmd show port
LTL
VSM Port Admin Link State PC-LTL SGID Vem Port Type
18
Eth3/2
UP
UP
FWD
305
1
vmnic1
19
Eth3/3
UP
UP
FWD
305
2
vmnic2
20
Eth3/4
UP
UP
FWD
305
3
vmnic3
49
Veth1
UP
UP
FWD
0
1 RHEL62-Test-1.eth0
305
Po1
UP
UP
FWD
0
<snip>

Explore & play with various features - ACLs, QoS, PVLANs, etc!