Nutanix - Advanced-Setup-Guide-AOS-v51

Acropolis Advanced Setup Guide
Acropolis 5.1
27-Apr-2017
Notice
Copyright
Copyright 2017 Nutanix, Inc.
Nutanix, Inc.
1740 Technology Drive, Suite 150
San Jose, CA 95110
All rights reserved. This product is protected by U.S. and international copyright and intellectual property
laws. Nutanix is a trademark of Nutanix, Inc. in the United States and/or other jurisdictions. All other marks
and names mentioned herein may be trademarks of their respective companies.
License
The provision of this software to you does not grant any licenses or other rights under any Microsoft
patents with respect to anything other than the file server implementation portion of the binaries for this
software, including no licenses or any other rights in any hardware or any devices or software that are used
to communicate with or in connection with this software.
Conventions
Convention Description
variable_value The action depends on a value that is unique to your environment.
ncli> command The commands are executed in the Nutanix nCLI.
user@host$ command The commands are executed as a non-privileged user (such as nutanix)
in the system shell.
root@host# command The commands are executed as the root user in the vSphere or Acropolis
host shell.
> command The commands are executed in the Hyper-V host shell.
output The information is displayed as output from a command or in a log file.
Default Cluster Credentials

Interface Target Username Password
Nutanix web console Nutanix Controller VM admin Nutanix/4u
vSphere Web Client ESXi host root nutanix/4u
Copyright | Acropolis Advanced Setup Guide | AOS | 2

Interface Target Username Password
vSphere client ESXi host root nutanix/4u
SSH client or console ESXi host root nutanix/4u
SSH client or console AHV host root nutanix/4u
SSH client or console Hyper-V host Administrator nutanix/4u
SSH client Nutanix Controller VM nutanix nutanix/4u
SSH client Nutanix Controller VM admin Nutanix/4u
SSH client or console Acropolis OpenStack root admin

Services VM (Nutanix
OVM)
Version
Last modified: April 27, 2017 (2017-04-27 23:16:28 GMT-7)
Copyright | Acropolis Advanced Setup Guide | AOS | 3

Contents
1: Cluster IP Address Configuration..........................................................5

Configuring the Cluster in a VLAN-Segmented Network.................................................................... 5
Assigning VLAN Tags to Nutanix Nodes (vSphere)................................................................. 6
Assigning VLAN Tags to Nutanix Nodes (Hyper-V)................................................................. 7
Assigning an Acropolis Host to a VLAN...................................................................................7
Preparing to Set IP Addresses............................................................................................................7
Configuring the Cluster........................................................................................................................ 9
2: Manual Cluster Configuration Methods.............................................. 13

Verifying IPv6 Link-Local Connectivity...............................................................................................13
Summary: Configuring the Cluster (Manual)..................................................................................... 15
Creating the Cluster (Manual)........................................................................................................... 15
Creating the Cluster (Configuration File)...........................................................................................17
3: Manual IP Address Configuration Methods........................................20

Remote Console IP Address Configuration...................................................................................... 20
Configuring the Remote Console IP Address (BIOS).............................................................20
Configuring the Remote Console IP Address (command line)...............................................21
Configuring Host Networking (ESXi)................................................................................................. 22
Enabling 1 GbE Interfaces (Hyper-V)................................................................................................23
Configuring Host Networking for Hyper-V Manually..........................................................................24
Joining a Host to a Domain Manually............................................................................................... 26
Configuring Host Networking (AHV).................................................................................................. 26
Enabling 1 GbE Interfaces (AHV)..................................................................................................... 27
Configuring the Controller VM IP Address........................................................................................ 27
4
1
Cluster IP Address Configuration
AOS includes a web-based configuration tool that automates assigning IP addresses to cluster
components and creates the cluster.
Requirements
The web-based configuration tool requires that IPv6 link-local be enabled on the subnet. If IPv6 link-local is
not available, you must configure the Controller VM IP addresses and the cluster manually. The web-based
configuration tool also requires that the Controller VMs be able to communicate with each other.
All Controller VMs and hypervisor hosts must be on the same subnet. The hypervisor can be multihomed
provided that one interface is on the same subnet as the Controller VM.
Guest VMs can be on a different subnet.
Configuring the Cluster in a VLAN-Segmented Network

The automated IP address and cluster configuration utilities depend on Controller VMs being able to
communicate with each other. If the customer network is segmented using VLANs, that communication is
not possible until the Controller VMs are assigned to a valid VLAN.
1. Configure the IPMI IP addresses.

→ Configuring the Remote Console IP Address (BIOS) on page 20
→ Configuring the Remote Console IP Address (command line) on page 21
For nodes from other hardware vendors, refer to the manufacturer's instructions.
2. Configure the hypervisor host IP addresses.

→ vSphere: Configuring Host Networking (ESXi) on page 22
→ Hyper-V: Configuring Host Networking for Hyper-V Manually on page 24
→ AHV: Configuring Host Networking (AHV) on page 26
Cluster IP Address Configuration | Acropolis Advanced Setup Guide | AOS | 5

3. Assign VLAN tags to the hypervisor hosts and Controller VMs by following the hypervisor-specific
procedure.
→ vSphere: Assigning VLAN Tags to Nutanix Nodes (vSphere) on page 6
→ Hyper-V: Assigning VLAN Tags to Nutanix Nodes (Hyper-V) on page 7
→ AHV: Assigning an Acropolis Host to a VLAN on page 7
4. Configure the Controller VM IP addresses and create the cluster.

→ If IPv6 link-local is enabled on the subnet, set the Controller VM IP addresses and create the cluster
by following Configuring the Cluster on page 9.
→ If IPv6 link-local is not enabled on the subnet, set the Controller VM IP addresses by following
Configuring the Controller VM IP Address on page 27 for each Controller VM in the cluster, then
create the cluster by following Creating the Cluster (Manual) on page 15.
Assigning VLAN Tags to Nutanix Nodes (vSphere)
1. Assign the ESXi hosts to the pre-defined host VLAN.
a. Connect to the ESXi host with the IPMI remote console or by attaching a keyboard and monitor.
b. Press F2 and then provide the ESXi host logon credentials.
c. Press the down arrow key until Configure Management Network is highlighted and then press
Enter.
d. Select VLAN (optional) and press Enter.
e. Type the VLAN ID specified by the customer and press Enter.
f. Press Esc and then Y to apply all changes and restart the management network.
g. Repeat this process for all remaining ESXi hosts.
2. Assign the Controller VMs to the pre-defined virtual machine VLAN.
a. Log on to an ESXi host with the vSphere client.
b. Select the host and then click the Configuration tab.
c. Click Networking.
d. Click the Properties link above vSwitch0.
e. Select VM Network and then click Edit.
f. Type the VLAN ID specified by the customer and click OK.
g. Click Close.
h. Repeat this process for all remaining ESXi hosts.

Assigning VLAN Tags to Nutanix Nodes (Hyper-V)
To configure VLANs for the Controller VM, do the following on each host:
1. Connect to the Hyper-V host with the IPMI remote console or by attaching a keyboard and monitor.
2. Start PowerShell.
3. Find the name of the Controller VM and set it to a variable $cvm.

> $cvm = Get-VM | Where {$_.Name -match 'NTNX.*CVM'}
4. Set the VLAN of the Controller VM.

> Set-VMNetworkAdapterVlan -VMName $cvm.name -Access -VlanId vlan_id `
-VMNetworkAdapterName External
• Replace vlan_id with the identifier of the VLAN.
5. Set the VLAN of the hypervisor host.

> Set-VMNetworkAdapterVlan -ManagementOS -Access -VlanId vlan_id `
-VMNetworkAdapterName ExternalSwitch
Replace vlan_id with the identifier of the VLAN.
Assigning an Acropolis Host to a VLAN

To assign an AHV host to a VLAN, do the following on every AHV host in the cluster:
1. Log on to the AHV host with SSH.
2. Assign port br0 (the internal port on the default OVS bridge, br0) to the VLAN that you want the host be
on.
root@ahv# ovs-vsctl set port br0 tag=host_vlan_tag
Replace host_vlan_tag with the VLAN tag for hosts.
3. Confirm VLAN tagging on port br0.

root@ahv# ovs-vsctl list port br0
4. Check the value of the tag parameter that is shown.
5. Verify connectivity to the IP address of the AHV host by performing a ping test.
Preparing to Set IP Addresses

Before you configure the cluster, check that these requirements are met.
• Confirm that IPv6 link-local is enabled on the subnet.

IPv6 link-local is required only for discovery of nodes. It is not required after cluster creation except to
add nodes to an existing cluster.
• Confirm that the system you are using to configure the cluster meets the following requirements:
• IPv6 link-local enabled.
• Windows 7, Vista, or MacOS.

• (Windows only) Bonjour installed (included with iTunes or downloadable from http://
support.apple.com/kb/DL999).
• (Hyper-V only) Confirm that the hosts have only one type of NIC (10 GbE or 1 GbE) connected during
cluster creation. If the nodes have multiple types of network interfaces connected, disconnect them until
after you join the hosts to the domain.
• Determine the IPv6 service name of any Controller VM in the cluster.

The service name depends on a unique identifier for the system.
Nutanix Serial Number
IPv6 service names are uniquely

generated at the factory and have the
following form (note the final period):
NTNX-block_serial_number-node_location-
CVM.local.
On the right side of the block toward the front

is a label that has the block_serial_number (for
example, 12AM3K520060).
The node_location is A for one-node blocks, A-
B for two-node blocks, and A-D for four-node
blocks.
If you do not have access to get the block serial
number, see the Nutanix support knowledge base
for alternative methods.
Dell Service Tag
IPv6 service names are uniquely

generated at the factory and have the
following form (note the final period):
NTNX-system_service_tag-node_location-
CVM.local.
On the front left side of the system is a slide-out

label that contains the system_service_tag (for
example, B57PW12).
The node_location is A for one-node blocks.

Configuring the Cluster
Before you begin: Check that the cluster is ready to be configured by following Preparing to Set IP
Addresses on page 7.
Note: This procedure has been deprecated (superseded) in AOS 4.5 and later releases. Instead,
use the Foundation tool to configure a cluster. See the "Creating a Cluster" topics in the Field
Installation Guide for more information.
Video: Click here to see a video (MP4 format) demonstration of this procedure. (The video may
not reflect the latest features described in this section.)
Figure: Cluster IP Address Configuration Page
1. Open a web browser.

Nutanix recommends using Internet Explorer 9 for Windows and Safari for Mac OS.
Note: Internet Explorer requires protected mode to be disabled. Go to Tools > Internet
Options > Security, clear the Enable Protected Mode check box, and restart the browser.
2. In the browser, go to http://[cvm_ipv6_addr]:2100/cluster_init.html.

Replace [cvm_ipv6_addr] with the IPv6 address of any Controller VM that should be added to the
cluster.
Following is an example URL to access the cluster creation page on a Controller VM:
http://[fe80::5054:ff:fea8:8aae]:2100/cluster_init.html

If the cluster_init.html page is blank, then the Controller VM is already part of a cluster. Connect to a
Controller VM that is not part of a cluster.
You can obtain the IPv6 address of the Controller VM by using the ifconfig command.
Example
nutanix@cvm$ ifconfig
eth0 Link encap:Ethernet HWaddr 52:54:00:A8:8A:AE
inet addr:10.1.65.240 Bcast:10.1.67.255 Mask:255.255.252.0
inet6 addr: fe80::5054:ff:fea8:8aae/64 Scope:Link
...etc...
The value of the inet6 addr field up to the / character is the IPv6 address of the Controller VM.
3. Type a meaningful value in the Cluster Name field.

This value is appended to all automated communication between the cluster and Nutanix support. It
should include the customer's name and, if necessary, a modifier that differentiates this cluster from any
other clusters that the customer might have.
Note: This entity has the following naming restrictions:
• The maximum length is 75 characters (for vSphere and AHV) and 15 characters (for Hyper-
V).
• Allowed characters are uppercase and lowercase standard Latin letters (A-Z and a-z),
decimal digits (0-9), dots (.), hyphens (-), and underscores (_).
4. Type a virtual IP address for the cluster in the Cluster External IP field.
This parameter is required for Hyper-V clusters and is optional for vSphere and AHV clusters.
You can connect to the external cluster IP address with both the web console and nCLI. In the event
that a Controller VM is restarted or fails, the external cluster IP address is relocated to another
Controller VM in the cluster.
5. (Optional) If you want to enable redundancy factor 3, set Cluster Max Redundancy Factor to 3.
Redundancy factor 3 has the following requirements:
• Redundancy factor 3 can be enabled only when the cluster is created.
• A cluster must have at least five nodes for redundancy factor 3 to be enabled.
• For guest VMs to tolerate the simultaneous failure of two nodes or drives in different blocks, the data
must be stored on storage containers with replication factor 3.
• Controller VMs must be configured with 24 GB of memory.
6. Type the appropriate DNS and NTP addresses in the respective fields.
Note: You must enter NTP servers that the Controller VMs can reach in the CVM NTP Servers
field. If reachable NTP servers are not entered or if the time on the Controller VMs is ahead of
the current time, cluster services may fail to start.
For Hyper-V clusters, the CVM NTP Servers parameter must be set to the IP addresses of one or more
Active Directory domain controllers.
The Hypervisor NTP Servers parameter is not used in Hyper-V clusters.
7. Type the appropriate subnet masks in the Subnet Mask row.
8. Type the appropriate default gateway IP addresses in the Default Gateway row.
9. Select the check box next to each node that you want to add to the cluster.

All unconfigured nodes on the current network are presented on this web page. If you are going to
configure multiple clusters, be sure that you only select the nodes that should be part of the current
cluster.
10. Provide an IP address for all components in the cluster.

Ensure that all components satisfy the cluster subnet requirements. See Cluster IP Address
Configuration on page 5. The use of a DHCP server is not supported for Controller VMs, so make
sure to not use DHCP for Controller VMs.
Note: The unconfigured nodes are not listed according to their position in the block. Ensure
that you assign the intended IP address to each node.
11. Click Create.

Wait until the Log Messages section of the page reports that the cluster has been successfully
configured.
Output similar to the following indicates successful cluster configuration.
Configuring IP addresses on node 13SM71450003/A...
Configuring the Hypervisor DNS settings on node 13SM71450003/A...
Configuring the Hypervisor NTP settings on node 13SM71450003/A...
Configuring Zeus on node 13SM71450003/A...
Initializing cluster...
Cluster successfully initialized!
Initializing the CVM DNS and NTP servers...
Successfully updated the CVM NTP and DNS server list
The cluster is started automatically after creation.
12. Log on to any Controller VM in the cluster with SSH.
13. Verify that all services are up on all Controller VMs.

nutanix@cvm$ cluster status
If the cluster is running properly, output similar to the following is displayed for each node in the cluster:
CVM: 10.1.64.60 Up
Zeus UP [5362, 5391, 5392, 10848, 10977, 10992]
Scavenger UP [6174, 6215, 6216, 6217]
SSLTerminator UP [7705, 7742, 7743, 7744]
SecureFileSync UP [7710, 7761, 7762, 7763]
Medusa UP [8029, 8073, 8074, 8176, 8221]
DynamicRingChanger UP [8324, 8366, 8367, 8426]
Pithos UP [8328, 8399, 8400, 8418]
Hera UP [8347, 8408, 8409, 8410]
Stargate UP [8742, 8771, 8772, 9037, 9045]
InsightsDB UP [8774, 8805, 8806, 8939]
InsightsDataTransfer UP [8785, 8840, 8841, 8886, 8888, 8889,
8890]
Ergon UP [8814, 8862, 8863, 8864]

Cerebro UP [8850, 8914, 8915, 9288]
Chronos UP [8870, 8975, 8976, 9031]
Curator UP [8885, 8931, 8932, 9243]
Prism UP [3545, 3572, 3573, 3627, 4004, 4076]
CIM UP [8990, 9042, 9043, 9084]
AlertManager UP [9017, 9081, 9082, 9324]
Arithmos UP [9055, 9217, 9218, 9353]
Catalog UP [9110, 9178, 9179, 9180]
Acropolis UP [9201, 9321, 9322, 9323]
Atlas UP [9221, 9316, 9317, 9318]
Uhura UP [9390, 9447, 9448, 9449]
Snmp UP [9418, 9513, 9514, 9516]
SysStatCollector UP [9451, 9510, 9511, 9518]
Tunnel UP [9480, 9543, 9544]
ClusterHealth UP [9521, 9619, 9620, 9947, 9976, 9977,
10301]
Janus UP [9532, 9624, 9625]
NutanixGuestTools UP [9572, 9650, 9651, 9674]
MinervaCVM UP [10174, 10200, 10201, 10202, 10371]
ClusterConfig UP [10205, 10233, 10234, 10236]
APLOSEngine UP [10231, 10261, 10262, 10263]
APLOS UP [10343, 10368, 10369, 10370, 10502,
10503]
Lazan UP [10377, 10402, 10403, 10404]
Orion UP [10409, 10449, 10450, 10474]
Delphi UP [10418, 10466, 10467, 10468]

2
Manual Cluster Configuration Methods
Verifying IPv6 Link-Local Connectivity

The automated IP address and cluster configuration utilities depend on IPv6 link-local addresses, which
are enabled on most networks. Use this procedure to verify that IPv6 link-local is enabled.
1. Connect two Windows, Linux, or Apple laptops to the switch to be used.
2. Disable any firewalls on the laptops.
3. Verify that each laptop has an IPv6 link-local address.

→ Windows (Control Panel)
Start > Control Panel > View network status and tasks > Change adapter settings > Local
Area Connection > Details
→ Windows (command-line interface)

> ipconfig
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . : corp.example.com

Link-local IPv6 Address . . . . . : fe80::ed67:9a32:7fc4:3be1%12
IPv4 Address. . . . . . . . . . . : 172.16.21.11
Subnet Mask . . . . . . . . . . . : 255.240.0.0
Default Gateway . . . . . . . . . : 172.16.0.1
Manual Cluster Configuration Methods | Acropolis Advanced Setup Guide | AOS | 13

→ Linux
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:0c:29:dd:e3:0b

inet addr:10.2.100.180 Bcast:10.2.103.255 Mask:255.255.252.0
inet6 addr: fe80::20c:29ff:fedd:e30b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2895385616 errors:0 dropped:0 overruns:0 frame:0
TX packets:3063794864 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2569454555254 (2.5 TB) TX bytes:2795005996728 (2.7 TB)
→ Mac OS
$ ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

ether 70:56:81:ae:a7:47
inet6 fe80::7256:81ff:feae:a747 en0 prefixlen 64 scopeid 0x4
inet 172.16.21.208 netmask 0xfff00000 broadcast 172.31.255.255
media: autoselect
status: active
Note the IPv6 link-local addresses, which always begin with fe80 . Omit the / character and anything
following.
4. From one of the laptops, ping the other laptop.

→ Windows
> ping -6 ipv6_linklocal_addr%interface
→ Linux/Mac OS
$ ping6 ipv6_linklocal_addr%interface
• Replace ipv6_linklocal_addr with the IPv6 link-local address of the other laptop.
• Replace interface with the interface identifier on the other laptop (for example, 12 for Windows, eth0
for Linux, or en0 for Mac OS).
If the ping packets are answered by the remote host, IPv6 link-local is enabled on the subnet. If the
ping packets are not answered, ensure that firewalls are disabled on both laptops and try again before
concluding that IPv6 link-local is not enabled.
5. Reenable the firewalls on the laptops and disconnect them from the network.
Results:
• If IPv6 link-local is enabled on the subnet, you can use automated IP address and cluster configuration
utility.
• If IPv6 link-local is not enabled on the subnet, you have to manually set IP addresses and create the
cluster.
Note: IPv6 connectivity issue might occur if mismatch occurs because of VLAN tagging. This
issue might occur because ESXi that is shipped from the factory does not have VLAN tagging,
hence it might have VLAN tag as 0. The workstation (laptop) that you have connected might be
connected to access port, so it might use different VLAN tag. Hence, ensure that ESXi port must
be in the trunking mode.

Summary: Configuring the Cluster (Manual)
Use this procedure if IPv6 link-local is not enabled on the subnet.
1. Configure the IPMI IP addresses.

→ Configuring the Remote Console IP Address (command line) on page 21
→ Configuring the Remote Console IP Address (BIOS) on page 20
For nodes from other hardware vendors, refer to the manufacturer's instructions.
2. Configure networking on the node by following the hypervisor-specific procedure.

→ vSphere: Configuring Host Networking (ESXi) on page 22
→ Hyper-V
1. (Optional) Enabling 1 GbE Interfaces (Hyper-V) on page 23
2. Configuring Host Networking for Hyper-V Manually on page 24
3. Joining a Host to a Domain Manually on page 26
→ AHV: Configuring Host Networking (AHV) on page 26
3. Configure the Controller VM IP addresses by following Configuring the Controller VM IP Address on

page 27.
4. Create the cluster by following Creating the Cluster (Manual) on page 15.
Creating the Cluster (Manual)
1. Log on to any Controller VM in the cluster with SSH.
2. Create the cluster.

nutanix@cvm$ cluster -s cvm_ip_addrs create
Replace cvm_ip_addrs with a comma-separated list of Controller VM IP addresses. Include all

Controller VMs that are part of the cluster.
If you want to configure redundancy factor 3, add the parameter --redundancy_factor=3 before create.
For example, if the new cluster should comprise all four nodes in a block, include all the IP addresses of
all four Controller VMs.
3. Start the Nutanix cluster.

nutanix@cvm$ cluster start
If the cluster starts properly, output similar to the following is displayed for each node in the cluster:
CVM: 10.1.64.60 Up
Zeus UP [5362, 5391, 5392, 10848, 10977, 10992]

Scavenger UP [6174, 6215, 6216, 6217]
Medusa UP [8029, 8073, 8074, 8176, 8221]
Pithos UP [8328, 8399, 8400, 8418]
Hera UP [8347, 8408, 8409, 8410]
Stargate UP [8742, 8771, 8772, 9037, 9045]
InsightsDB UP [8774, 8805, 8806, 8939]
8890]
Ergon UP [8814, 8862, 8863, 8864]
Cerebro UP [8850, 8914, 8915, 9288]
Chronos UP [8870, 8975, 8976, 9031]
Curator UP [8885, 8931, 8932, 9243]
Prism UP [3545, 3572, 3573, 3627, 4004, 4076]
CIM UP [8990, 9042, 9043, 9084]
AlertManager UP [9017, 9081, 9082, 9324]
Arithmos UP [9055, 9217, 9218, 9353]
Catalog UP [9110, 9178, 9179, 9180]
Acropolis UP [9201, 9321, 9322, 9323]
Atlas UP [9221, 9316, 9317, 9318]
Uhura UP [9390, 9447, 9448, 9449]
Snmp UP [9418, 9513, 9514, 9516]
Tunnel UP [9480, 9543, 9544]
10301]
Janus UP [9532, 9624, 9625]
MinervaCVM UP [10174, 10200, 10201, 10202, 10371]
APLOSEngine UP [10231, 10261, 10262, 10263]
APLOS UP [10343, 10368, 10369, 10370, 10502,
10503]
Lazan UP [10377, 10402, 10403, 10404]
Orion UP [10409, 10449, 10450, 10474]
Delphi UP [10418, 10466, 10467, 10468]
4. Set cluster parameters.
a. Define the name of the cluster.

nutanix@cvm$ ncli cluster edit-params new-name=cluster_name
Replace cluster_name with a name for the cluster chosen by the customer.
b. Configure the DNS servers.

nutanix@cvm$ ncli cluster add-to-name-servers servers="dns_server"
Replace dns_server with the IP address of a single DNS server or with a comma-separated list of
DNS server IP addresses.
c. Configure the NTP servers.

nutanix@cvm$ ncli cluster add-to-ntp-servers servers="ntp_server"
Replace ntp_server with the IP address or host name of a single NTP server or a with a comma-
separated list of NTP server IP addresses or host names.
d. Configure an external IP address for the cluster.

nutanix@cvm$ ncli cluster set-external-ip-address \
external-ip-address="cluster_ip_address"

Replace cluster_ip_address with a virtual IP address for the cluster.
e. (Hyper-V only) Add a record for the cluster external IP address to the domain DNS server.
Creating the Cluster (Configuration File)

Use this procedure if IPv6 link-local is enabled on the subnet but you cannot use the browser method of
configuring the cluster.
This method of cluster creation works only to define a new cluster. It cannot be used for IP address
configuration or cluster expansion.
1. Log on to any Controller VM on the same subnet as the Controller VMs that you want to include in the
new cluster.
This can be a Controller VM that is already part of a cluster. Connect to the IPv6 address because an
IPv4 connection is lost during configuration.
2. Create a JSON file that defines the networking configuration for the new cluster.
{
"Subnet Mask": {
"Controller": "Subnet mask",
"Hypervisor": "Subnet mask",
"IPMI": "Subnet mask"
},
"Default Gateway": {
"Controller": "IPv4 address",
"Hypervisor": "IPv4 address",
"IPMI": "IPv4 address"
},
"IP Addresses": {
"block_serial_number/A": {
},
"block_serial_number/B": {
},
"block_serial_number/C": {
},
"block_serial_number/D": {
}
}
}
The IP Addresses block requires one entry for each node that you want to include in the cluster
(minimum 3 nodes). Each node is identified by the block serial number and the node position (A, B, C,
or D).

3. Create the cluster.
nutanix@cvm$ cluster -j cluster_config_json_file create
Replace cluster_config_json_file with the name of the JSON file that defines the networking
configuration for the new cluster.
If you want to configure redundancy factor 3, add the parameter --redundancy_factor=3 before create.
If the cluster can be created, a success message for each Controller VM is displayed and the cluster
starts. If the cluster cannot be created, ensure the JSON file is correct and attempt the creation again.
4. Log on to any Controller VM in the newly-created cluster with SSH.
5. Verify that all services are up on all Controller VMs.

nutanix@cvm$ cluster status
If the cluster is running properly, output similar to the following is displayed for each node in the cluster:
CVM: 10.1.64.60 Up
Zeus UP [5362, 5391, 5392, 10848, 10977, 10992]
Scavenger UP [6174, 6215, 6216, 6217]
Medusa UP [8029, 8073, 8074, 8176, 8221]
Pithos UP [8328, 8399, 8400, 8418]
Hera UP [8347, 8408, 8409, 8410]
Stargate UP [8742, 8771, 8772, 9037, 9045]
InsightsDB UP [8774, 8805, 8806, 8939]
8890]
Ergon UP [8814, 8862, 8863, 8864]
Cerebro UP [8850, 8914, 8915, 9288]
Chronos UP [8870, 8975, 8976, 9031]
Curator UP [8885, 8931, 8932, 9243]
Prism UP [3545, 3572, 3573, 3627, 4004, 4076]
CIM UP [8990, 9042, 9043, 9084]
AlertManager UP [9017, 9081, 9082, 9324]
Arithmos UP [9055, 9217, 9218, 9353]
Catalog UP [9110, 9178, 9179, 9180]
Acropolis UP [9201, 9321, 9322, 9323]
Atlas UP [9221, 9316, 9317, 9318]
Uhura UP [9390, 9447, 9448, 9449]
Snmp UP [9418, 9513, 9514, 9516]
Tunnel UP [9480, 9543, 9544]
10301]
Janus UP [9532, 9624, 9625]
MinervaCVM UP [10174, 10200, 10201, 10202, 10371]
APLOSEngine UP [10231, 10261, 10262, 10263]

APLOS UP [10343, 10368, 10369, 10370, 10502,
10503]
Lazan UP [10377, 10402, 10403, 10404]
Orion UP [10409, 10449, 10450, 10474]
Delphi UP [10418, 10466, 10467, 10468]
6. Set cluster parameters.
a. Define the name of the cluster.

nutanix@cvm$ ncli cluster edit-params new-name=cluster_name
Replace cluster_name with a name for the cluster chosen by the customer.
b. Configure the DNS servers.

nutanix@cvm$ ncli cluster add-to-name-servers servers="dns_server"
Replace dns_server with the IP address of a single DNS server or with a comma-separated list of
DNS server IP addresses.
c. Configure the NTP servers.

nutanix@cvm$ ncli cluster add-to-ntp-servers servers="ntp_server"
Replace ntp_server with the IP address or host name of a single NTP server or a with a comma-
separated list of NTP server IP addresses or host names.
d. Configure an external IP address for the cluster.

nutanix@cvm$ ncli cluster set-external-ip-address \
external-ip-address="cluster_ip_address"
Replace cluster_ip_address with a virtual IP address for the cluster.

e. (Hyper-V only) Add a record for the cluster external IP address to the domain DNS server.

3
Manual IP Address Configuration Methods
Remote Console IP Address Configuration

The Intelligent Platform Management Interface (IPMI) is a standardized interface used to manage a host
and monitor its operation. To enable remote access to the console of each host, you must configure the
IPMI settings within BIOS.
The Nutanix cluster provides a Java application to remotely view the console of each node, or host server.
You can use this console to configure additional IP addresses in the cluster.
The procedure for configuring the remote console IP address is slightly different for each hardware
platform.
Configuring the Remote Console IP Address (BIOS)
1. Connect a keyboard and monitor to a node in the Nutanix block.
2. Restart the node and press Delete to enter the BIOS setup utility.
There is a limited amount of time to enter BIOS before the host completes the restart process.
3. Press the right arrow key to select the IPMI tab.
4. Press the down arrow key until BMC network configuration is highlighted and then press Enter.
5. Press down the arrow key until Update IPMI LAN Configuration is highlighted and press Enter to
select Yes.
6. Select Configuration Address source and press Enter.
7. Select Static and press Enter.
8. Assign the Station IP address, Subnet mask, and Router IP address.
Manual IP Address Configuration Methods | Acropolis Advanced Setup Guide | AOS | 20

9. Review the BIOS settings and press F4 to save the configuration changes and exit the BIOS setup
utility.
The node restarts.
Configuring the Remote Console IP Address (command line)

You can configure the management interface from the hypervisor host on the same node.
Perform these steps once from each hypervisor host in the cluster where the management network
configuration need to be changed.
1. Log on to the hypervisor host with SSH (vSphere or AHV) or remote desktop connection (Hyper-V).
2. Set the networking parameters.

→ vSphere
root@esx# /ipmitool -U ADMIN -P ADMIN lan set 1 ipsrc static
root@esx# /ipmitool -U ADMIN -P ADMIN lan set 1 ipaddr mgmt_interface_ip_addr
root@esx# /ipmitool -U ADMIN -P ADMIN lan set 1 netmask mgmt_interface_subnet_addr
root@esx# /ipmitool -U ADMIN -P ADMIN lan set 1 defgw ipaddr mgmt_interface_gateway
→ Hyper-V
> ipmiutil lan -I mgmt_interface_ip_addr -G mgmt_interface_gateway `
-S mgmt_interface_subnet_addr -U ADMIN -P ADMIN
→ AHV
root@ahv# ipmitool -U ADMIN -P ADMIN lan set 1 ipsrc static
root@ahv# ipmitool -U ADMIN -P ADMIN lan set 1 ipaddr mgmt_interface_ip_addr
root@ahv# ipmitool -U ADMIN -P ADMIN lan set 1 netmask mgmt_interface_subnet_addr
root@ahv# ipmitool -U ADMIN -P ADMIN lan set 1 defgw ipaddr mgmt_interface_gateway
• Replace mgmt_interface_ip_addr with the new IP address for the remote console.
• Replace mgmt_interface_gateway with the gateway IP address.
• Replace mgmt_interface_subnet_addr with the subnet mask for the new IP address.
3. Show current settings.

→ vSphere
root@esx# /ipmitool -v -U ADMIN -P ADMIN lan print 1
→ Hyper-V
> ipmiutil lan -r -U ADMIN -P ADMIN
→ AHV
root@ahv# ipmitool -v -U ADMIN -P ADMIN lan print 1
Confirm that the parameters are set to the correct values.
Configuring Host Networking (ESXi)
You can access the ESXi console either through IPMI or by attaching a keyboard and monitor to the node.
1. On the ESXi host console, press F2 and then provide the ESXi host logon credentials.
2. Press the down arrow key until Configure Management Network is highlighted and then press Enter.
3. Select Network Adapters and press Enter.
4. Ensure that the connected network adapters are selected.

If they are not selected, press Space to select them and press Enter to return to the previous screen.
5. If a VLAN ID needs to be configured on the Management Network, select VLAN (optional) and press
Enter. In the dialog box, provide the VLAN ID and press Enter.
6. Select IP Configuration and press Enter.
7. If necessary, highlight the Set static IP address and network configuration option and press Space
to update the setting.

8. Provide values for the following: IP Address, Subnet Mask, and Default Gateway fields based on your
environment and then press Enter .
9. Select DNS Configuration and press Enter.
10. If necessary, highlight the Use the following DNS server addresses and hostname option and press
Space to update the setting.
11. Provide values for the Primary DNS Server and Alternate DNS Server fields based on your
environment and then press Enter.
12. Press Esc and then Y to apply all changes and restart the management network.
13. Select Test Management Network and press Enter.
14. Press Enter to start the network ping test.
15. Verify that the default gateway and DNS servers reported by the ping test match those that you
specified earlier in the procedure and then press Enter.
Ensure that the tested addresses pass the ping test. If they do not, confirm that the correct IP
addresses are configured.
Press Enter to close the test window.
16. Press Esc to logoff.
Enabling 1 GbE Interfaces (Hyper-V)

If 10 GbE networking is specified during cluster setup, 1 GbE interfaces are disabled on Hyper-V nodes.
Follow these steps if you need to enable the 1 GbE interfaces later.
To enable the 1 GbE interfaces, do the following on each host:
1. Log on to the Hyper-V host with Remote Desktop Connection and start PowerShell.
2. List the network adapters.

> Get-NetAdapter | Format-List Name,InterfaceDescription,LinkSpeed
Output similar to the following is displayed.
Name : vEthernet (InternalSwitch)

InterfaceDescription : Hyper-V Virtual Ethernet Adapter #3

LinkSpeed : 10 Gbps
Name : vEthernet (ExternalSwitch)

LinkSpeed : 10 Gbps
Name : Ethernet
InterfaceDescription : Intel(R) 82599 10 Gigabit Dual Port Network Connection
LinkSpeed : 10 Gbps
Name : Ethernet 3
InterfaceDescription : Intel(R) 82599 10 Gigabit Dual Port Network Connection #2
LinkSpeed : 10 Gbps
Name : NetAdapterTeam
InterfaceDescription : Microsoft Network Adapter Multiplexor Driver
LinkSpeed : 20 Gbps
Name : Ethernet 4
InterfaceDescription : Intel(R) I350 Gigabit Network Connection #2
LinkSpeed : 0 bps
Name : Ethernet 2
InterfaceDescription : Intel(R) I350 Gigabit Network Connection
LinkSpeed : 1 Gbps
Make a note of the Name of the 1 GbE interfaces you want to enable.
3. Configure the interface.

Replace interface_name with the name of the 1 GbE interface as reported by Get-NetAdapter.
a. Enable the interface.

> Enable-NetAdapter -Name "interface_name"
b. Add the interface to the NIC team.

> Add-NetLBFOTeamMember -Team NetAdapterTeam -Name "interface_name"
If you want to configure the interface as a standby for the 10 GbE interfaces, include the parameter -
AdministrativeMode Standby
Perform these steps once for each 1 GbE interface you want to enable.
Configuring Host Networking for Hyper-V Manually
1. Log on to the Hyper-V host with the IPMI remote console and start a Powershell prompt.
2. List the network adapters.

> Get-NetAdapter | Format-List Name,InterfaceDescription,LinkSpeed
Name : vEthernet (InternalSwitch)

LinkSpeed : 10 Gbps
Name : vEthernet (ExternalSwitch)

LinkSpeed : 10 Gbps
Name : Ethernet
InterfaceDescription : Intel(R) 82599 10 Gigabit Dual Port Network Connection
LinkSpeed : 10 Gbps
Name : Ethernet 3
InterfaceDescription : Intel(R) 82599 10 Gigabit Dual Port Network Connection #2
LinkSpeed : 10 Gbps
Name : NetAdapterTeam
InterfaceDescription : Microsoft Network Adapter Multiplexor Driver
LinkSpeed : 20 Gbps
Name : Ethernet 4
InterfaceDescription : Intel(R) I350 Gigabit Network Connection #2
LinkSpeed : 0 bps
Name : Ethernet 2
InterfaceDescription : Intel(R) I350 Gigabit Network Connection
LinkSpeed : 0 bps
Make a note of the InterfaceDescription for the vEthernet adapter that links to the physical interface
you want to modify.
3. Start the Server Configuration utility.

> sconfig
4. Select Networking Settings by typing 8 and pressing Enter.
5. Change the IP settings.
a. Select a network adapter by typing the Index number of the adapter you want to change (refer to the
InterfaceDescription you found in step 2 on page 24) and pressing Enter.
Warning: Do not select the network adapter with the IP address 192.168.5.1. This IP
address is required for the Controller VM to communicate with the host.
b. Select Set Network Adapter Address by typing 1 and pressing Enter.
c. Select Static by typing S and pressing Enter.
d. Enter the IP address for the host and press Enter.
e. Enter the subnet mask and press Enter.
f. Enter the IP address for the default gateway and press Enter.
The host networking settings are changed.
6. (Optional) Change the DNS servers.

DNS servers must be configured for a host to be part of a domain. You can either change the DNS
servers in the sconfig utility or with setup_hyperv.py.
a. Select Set DNS Servers by typing 2.
b. Enter the primary and secondary DNS servers and press Enter.
The DNS servers are updated.

7. Exit the Server Configuration utility by typing 4 and pressing Enter then 15 and pressing Enter.
Joining a Host to a Domain Manually

For joining a host to a domain by using utilities provided by Nutanix, see the Nutanix Hyper-V
Administration guide. Perform these steps for each Hyper-V host in the cluster to manually join a host to a
domain.
1. Log on to the Hyper-V host with the IPMI remote console and start a Powershell prompt.
2. Join the host to the domain and rename it.

> Add-Computer -DomainName domain_name -NewName node_name `
-Credential domain_name\domain_admin_user -Restart -Force
• Replace domain_name with the name of the join for the host to join.
• Replace node_name with a new name for the host.
• Replace domain_admin_user with the domain administrator username.
The host restarts and joins the domain.
Configuring Host Networking (AHV)
To configure host networking, do the following:
1. Edit the settings of port br0, which is the internal port on the default bridge br0.
a. Log on to the host console as root.

You can access the hypervisor host console either through IPMI or by attaching a keyboard and
monitor to the node.
b. Open the network interface configuration file for port br0 in a text editor.
root@ahv# vi /etc/sysconfig/network-scripts/ifcfg-br0
c. Update entries for host IP address, netmask, and gateway.

The block of configuration information that includes these entries is similar to the following:
ONBOOT="yes"
NM_CONTROLLED="no"
PERSISTENT_DHCLIENT = 1
NETMASK="subnet_mask"
IPADDR="host_ip_addr"
DEVICE="br0"
TYPE="ethernet"
GATEWAY="gateway_ip_addr"
BOOTPROTO="none"
• Replace host_ip_addr with the IP address for the hypervisor host.

• Replace subnet_mask with the subnet mask for host_ip_addr.
• Replace gateway_ip_addr with the gateway address for host_ip_addr.
d. Save your changes.

2. Update the resolver configuration file with the details of your domain name server.
a. Open the resolver configuration file.

root@ahv# vi /etc/resolv.conf
b. Update the values of the nameserver parameter and then save and close the file.
c. Restart the networking service.

root@ahv# service network restart
3. Assign the AHV host to a VLAN. For information about how to add the AHV host to a VLAN, see
Assigning an Acropolis Host to a VLAN in the Acropolis Hypervisor Administration Guide.
Enabling 1 GbE Interfaces (AHV)

This procedure describes how to add the 1 GbE uplinks to the default OVS bridge br0, which is used by the
Controller VM, AHV, and guest VMs, and must be performed on each Nutanix Controller VM in the cluster.
To add the 1 GbE uplinks to the default OVS bridge named br0, do the following:
1. Log on to the Controller VM with SSH.
2. Determine the names of the 1 GbE interfaces.

nutanix@cvm$ manage_ovs show_interfaces
name mode link speed

eth0 1000 True 1000
eth1 1000 False None
eth2 10000 True 10000
eth3 10000 True 10000
3. Verify which interfaces are assigned to bridge br0.

nutanix@cvm$ manage_ovs --bridge_name br0 show_uplinks
Output similar to the following indicates that interfaces eth2 and eth3 are bonded and that the bonded
port is assigned to the br0 switch. The output also indicates that the interfaces eth0 and eth1 are not on
bridge br0.
Uplink ports: bond0

Uplink ifaces: eth2 eth3
4. Add the 1 GbE interfaces to a bonded port bond1 on bridge br0.

nutanix@cvm$ manage_ovs --interfaces 1g --bond_name bond1 update_uplinks
Configuring the Controller VM IP Address

You can manually assign an IP address or change the IP address that you assigned through Foundation.
Do not use a DHCP server to assign an IP address to a Controller VM. The use of a DHCP server is not
supported for Controller VMs. Assign a static IP address instead.

If IPv6 is not enabled on the subnet, manually change the Controller VM IP address for each Controller VM
in the cluster.
If IPv6 is enabled on the subnet, you can change the Controller VM IP address by following Changing
Controller VM IP Addresses in Acropolis Advanced Administration Guide.
To manually change the Controller VM IP address, do the following:
1. Stop the cluster.
2. Run the external IP address reconfiguration script (external_ip_reconfig).
3. Restart all the Controller VMs in the cluster after the script has been successfully executed.
4. Start the cluster.
The external IP address reconfiguration script (external_ip_reconfig) performs the following tasks:
1. Puts the cluster in reconfiguration mode.
2. Restarts Genesis.
3. Prompts you to type the new netmask, gateway, and external IP addresses, and updates them.
4. Updates the IP addresses of the Zookeeper hosts.
Perform the following procedure to manually change the Controller VM IP address.
Warning: If you are changing the Controller VM IP addresses to another subnet, network, IP
address range, or VLAN, you should also change the hypervisor management IP addresses to the
same subnet, network, IP address range, or VLAN.
1. Log on to the hypervisor with SSH (vSphere or AHV) or remote desktop connection (Hyper-V), or the
IPMI remote console.
2. Log on to the Controller VM.

→ vSphere or AHV root@host# ssh nutanix@192.168.5.254
→ Hyper-V > ssh nutanix@192.168.5.254
Accept the host authenticity warning if prompted, and enter the Controller VM nutanix password.
3. Stop the Nutanix cluster.
Warning: This step affects the operation of a Nutanix cluster. Schedule a down time before
performing this step.
nutanix@cvm$ cluster stop
Note: If necessary, change the hypervisor management IP address or IPMI IP address before
you execute the external_ip_reconfig script.
4. Run the external IP address reconfiguration script (external_ip_reconfig) from any one Controller VM in
the cluster.
nutanix@cvm$ external_ip_reconfig
5. Follow the prompts to type the new netmask, gateway, and external IP addresses.
A message similar to the following is displayed after the reconfiguration is successfully completed:
External IP reconfig finished successfully. Restart all the CVMs and start the cluster.
6. Restart each Controller VM in the cluster.

nutanix@cvm$ sudo reboot
Enter the nutanix password if prompted.

7. Start the Nutanix cluster.
nutanix@cvm$ cluster start
If the cluster starts properly, output similar to the following is displayed for each node in the cluster:
CVM: 10.1.64.60 Up
Zeus UP [3704, 3727, 3728, 3729, 3807, 3821]
Scavenger UP [4937, 4960, 4961, 4990]
Hyperint UP [5059, 5082, 5083, 5086, 5099, 5108]
Medusa UP [5534, 5559, 5560, 5563, 5752]
Pithos UP [5877, 5899, 5900, 5962]
Stargate UP [5902, 5927, 5928, 6103, 6108]
Cerebro UP [5930, 5952, 5953, 6106]
Chronos UP [5960, 6004, 6006, 6075]
Curator UP [5987, 6017, 6018, 6261]
Prism UP [6020, 6042, 6043, 6111, 6818]
AlertManager UP [6070, 6099, 6100, 6296]
Arithmos UP [6107, 6175, 6176, 6344]
Tunnel UP [6263, 6312, 6313]
ClusterHealth UP [6317, 6342, 6343, 6446, 6468, 6469, 6604, 6605,
6606, 6607]
Janus UP [6365, 6444, 6445, 6584]
NutanixGuestTools UP [6377, 6403, 6404]
What to do next: Run the following NCC checks to verify the health of the Zeus configuration:
• nutanix@cvm$ ncc health_checks system_checks zkalias_check_plugin
• nutanix@cvm$ ncc health_checks system_checks zkinfo_check_plugin

Nutanix - Advanced-Setup-Guide-AOS-v51

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Nutanix - Advanced-Setup-Guide-AOS-v51

Uploaded by

Copyright:

Available Formats

Acropolis Advanced Setup Guide

variable_value The action depends on a value that is unique to your environment.

ncli> command The commands are executed in the Nutanix nCLI.

output The information is displayed as output from a command or in a log file.

Default Cluster Credentials

Nutanix web console Nutanix Controller VM admin Nutanix/4u

vSphere Web Client ESXi host root nutanix/4u

Copyright | Acropolis Advanced Setup Guide | AOS | 2

vSphere client ESXi host root nutanix/4u

SSH client or console ESXi host root nutanix/4u

SSH client or console AHV host root nutanix/4u

SSH client or console Hyper-V host Administrator nutanix/4u

SSH client Nutanix Controller VM nutanix nutanix/4u

SSH client Nutanix Controller VM admin Nutanix/4u

SSH client or console Acropolis OpenStack root admin

Copyright | Acropolis Advanced Setup Guide | AOS | 3

1: Cluster IP Address Configuration..........................................................5

2: Manual Cluster Configuration Methods.............................................. 13

3: Manual IP Address Configuration Methods........................................20

Configuring the Cluster in a VLAN-Segmented Network

1. Configure the IPMI IP addresses.

2. Configure the hypervisor host IP addresses.

Cluster IP Address Configuration | Acropolis Advanced Setup Guide | AOS | 5

4. Configure the Controller VM IP addresses and create the cluster.

Assigning VLAN Tags to Nutanix Nodes (vSphere)

1. Assign the ESXi hosts to the pre-defined host VLAN.

b. Press F2 and then provide the ESXi host logon credentials.

d. Select VLAN (optional) and press Enter.

e. Type the VLAN ID specified by the customer and press Enter.

g. Repeat this process for all remaining ESXi hosts.

2. Assign the Controller VMs to the pre-defined virtual machine VLAN.

a. Log on to an ESXi host with the vSphere client.

b. Select the host and then click the Configuration tab.

d. Click the Properties link above vSwitch0.

e. Select VM Network and then click Edit.

f. Type the VLAN ID specified by the customer and click OK.

h. Repeat this process for all remaining ESXi hosts.

Cluster IP Address Configuration | Acropolis Advanced Setup Guide | AOS | 6

3. Find the name of the Controller VM and set it to a variable $cvm.

4. Set the VLAN of the Controller VM.

• Replace vlan_id with the identifier of the VLAN.

5. Set the VLAN of the hypervisor host.

Replace vlan_id with the identifier of the VLAN.

Assigning an Acropolis Host to a VLAN

1. Log on to the AHV host with SSH.

Replace host_vlan_tag with the VLAN tag for hosts.

3. Confirm VLAN tagging on port br0.

4. Check the value of the tag parameter that is shown.

Preparing to Set IP Addresses

• Confirm that IPv6 link-local is enabled on the subnet.

Cluster IP Address Configuration | Acropolis Advanced Setup Guide | AOS | 7

• Determine the IPv6 service name of any Controller VM in the cluster.

Nutanix Serial Number

IPv6 service names are uniquely

On the right side of the block toward the front

Dell Service Tag

IPv6 service names are uniquely

On the front left side of the system is a slide-out

Cluster IP Address Configuration | Acropolis Advanced Setup Guide | AOS | 8

Figure: Cluster IP Address Configuration Page

1. Open a web browser.

2. In the browser, go to http://[cvm_ipv6_addr]:2100/cluster_init.html.

Cluster IP Address Configuration | Acropolis Advanced Setup Guide | AOS | 9

3. Type a meaningful value in the Cluster Name field.