VMware ESXi Host upgrades and patching on VRTX infrastructure

VMware ESXi Host upgrades and patching on VRTX infrastructure

The VRTX infrastructure is designed for host redundancy so hosts can be upgraded without impact to the running VMs or
hotel operations. On VRTX with two hosts (Small and Medium), all VMs must be able to run on one host. On VRTX with
three hosts (Large), all VMs must be able to run on two hosts. The only thing that will prevent this from being possible is
if you have over-allocated RAM on the VMs. In that case, review the VM needs/settings and adjust as necessary.

This document has been created using vCenter 6.7 and hosts being upgraded from ESXi 6.0 to ESXi 6.7. The specific
versions, file name, patch numbers, etc. can and will change at any time. The overall process will remain mostly the
same. A minimum number of screenshots will be used as well to reduce file size and possible paper when printed.

Contents
Pre-requisite............................................................................................................................................................................ 1
Upgrade vCenter ..................................................................................................................................................................... 2
Move VMs from host being upgraded .................................................................................................................................... 4
Methods of upgrading the ESXi version - use iDRAC interactively or use VMware Update Manager ................................... 4
Upgrade ESXi using iDRAC interactively .................................................................................................................................. 5
Upgrade ESXi using VMware Update Manager ...................................................................................................................... 8
Remediate each host ........................................................................................................................................................ 11
Patching ESXi host to current build using VMware Update Manager .................................................................................. 13
Remediate CVE-2018-3646 ................................................................................................................................................... 17
Appendix A - Missing VIBS .................................................................................................................................................... 19

Pre-requisite
Download the ISO file for your ESXi host upgrade from BOX. These are vendor specific and the files for Dell servers are
in: /Americas IT - TaPS/VMware/Dell ESXi Images. The file names will include the major version, update level and patch
number. The TaPS team will update these monthly as new versions are posted by the vendor.

For the purpose of this document, we will use VMware-VMvisor-Installer-6.7.0.update03-15160138.x86_64-

DellEMC_Customized-A03.ISO.

Note: ESXi 6.7 is not on the VMware Compatibility Guide for the oldest VRTX with M620 hosts. Any hosts with M620
servers should use the 6.5 update.

For reference, the VMware Compatibility Guide can be found here:

https://www.vmware.com/resources/compatibility/search.php
Select the product you would like to upgrade to, the Partner Name (Manufacturer), System Type, enter your model as a
Keyword and click Update and View Results.

1
VMware ESXi Host upgrades and patching on VRTX infrastructure

Upgrade vCenter
Before upgrading the hosts, it is highly recommended (not required) to upgrade vCenter to the latest version. This is
currently VMware-VIM-all-6.7.0-15132721.iso as of January 2020. The file is on BOX under: All Files > Americas IT - TaPS
> VMware > vCenter.

 Download the ISO and double click to mount it.

 Double click autorun.exe.
 Under VMware vCenter Server, select vCenter Server for Windows - click the Install button.
o Click OK when prompted about the administrators group not having Bypass traverse checking rights.
o Next > Agree > Update.
o Note: Be patient! This upgrade can take a long time - even hours!
 Back on the Installer screen, under vSphere Update Manager, select Server.
o On the right, select Use Microsoft SQL Server 2012 Express as the embedded database AND Install
Microsoft .Net Framework 4.7. Both should already be installed. Click Install.
o Click Yes when prompted to continue to upgrade VMware vSphere Update Manager.
o Click OK when prompted to upgrade to the current version.
o Next > Accept > Next > Next
o Ensure the IP address of the vCenter is correct and enter the password for administrator@vsphere.local.
Click Next.

2
VMware ESXi Host upgrades and patching on VRTX infrastructure

o IF PROMPTED for Database Information, you will need to use your ADM account and password from
ADSP. Click Next.

o Select Yes, I want to upgrade my Update Manager Database.

Make sure a check is in the box for I have taken a backup. Click Next.

o Click Install.
o Click OK to Automatically close and attempt to restart applications.
o Click Finish when complete
 Close the Installer screen and Eject the ISO.
 Open https://marshavcenter.domain.marrcorp.marriott.com
3
VMware ESXi Host upgrades and patching on VRTX infrastructure

Move VMs from host being upgraded

 Select a host > VM tab > move all VMs off the first host you wish to patch.
o Right click the VM (or select multiple)
o Migrate
o Change compute resource only
o Select a different host (keep in mind any load balancing needs)
 It may be necessary to resolve any conflicts
 No heartbeats as a conflict can be ignored

o Confirm network
o Schedule vMotion with high priority
o Finish
o Monitor Recent Tasks until the VM(s) have completed migrating to the other host

Methods of upgrading the ESXi version - use iDRAC interactively or use VMware
Update Manager
There are two different methods that can be used to upgrade ESXi hosts.
 Use iDRAC to view the console, mount the ISO, reboot and run the upgrade interactively.
 Use VMware Update Manager to import the ISO, set a Baseline and Remediate to the Baseline.

Both methods are detailed below. There isn't a recommended method and both have pros and cons.

4
VMware ESXi Host upgrades and patching on VRTX infrastructure

Upgrade ESXi using iDRAC interactively

There are two methods to upgrade ESXi. This option uses the iDRAC console to boot to the ISO image.

 In vCenter, right click the host > Maintenance Mode > Enter Maintenance Mode
o You should have already moved the VMs so the question if you want to relocate VMs is not relevant.
 Access the iDRAC IP address using IE or another web browser.
o Login with your adm-eid@marrcorp.marriott.com.
o On the left, open the Server section and click on Virtual Console.
o It is recommended to use Plug-in Type: Native. This will install/launch and ActiveX control. The
alternate is to set this to HTML5 which will make the install slower. Screenshots shown here are using
the Native client.

 Click Launch Virtual Console

o Install ActiveX control if needed, click Run when prompted.
 On the Menu bar, click Virtual Media > Connect Virtual Media.
 On the Menu bar, click Virtual Media > < Map CD/DVD.
o Browse to the ESXi upgrade ISO.
o Click Map Device.
 On the Menu bar, click Next Boot > select Virtual CD/DVD/ISO. Click OK.
 On the host desktop, press F12
o Enter the root password > press Enter.
o You should have already moved the VMs so the question if you want to terminate VMs is not relevant.
o Press F11 to restart.
 The host will reboot and should automatically boot to the ISO/installer.
 At the Welcome to the VMware ESXi Installation screen, press Enter to continue.
 Press F11 to Accept and Continue.

5
VMware ESXi Host upgrades and patching on VRTX infrastructure

 At the "Select a disk to Install or Upgrade" screen, ensure the boot disk is selected. This should be the only
"Local:" disk listed. On most VRTX purchased prior to mid-2019, this will be the Internal Dual SD. DO NOT select
any of the "Remote:" disks since you could overwrite your Datastores.

 The installation will scan the disk and see the previous installation. When the ESXi Found screen appears,
ensure (X) Upgrade is selected and press Enter to continue.

 At the Confirm Upgrade screen, press F11 to Upgrade.

 NOTE: If you received an error "MISSING_DEPENDENCY_VIBS ERROR" please refer to Appendix A.

 When the upgrade is complete, you must disconnect the ISO. On the Menu bar, click Virtual Media > Disconnect
Virtual Media. Click Yes at the Close Virtual Media warning.

6
VMware ESXi Host upgrades and patching on VRTX infrastructure

 After disconnecting the virtual media, press Enter to reboot.

 Once the host reboots, it will take some time for the host to re-appear in vCenter. Once the host no longer says
"not responding" you can right click > Maintenance Mode > Exit Maintenance Mode.
o If the host does not reconnect, you may need to right click and Connect. This may involve entering the
root login/password and accepting the new fingerprint.
 Repeat on remaining hosts. Refer to the section " Move VMs from host being upgraded" above to move VMs
before remediating each host.

7
VMware ESXi Host upgrades and patching on VRTX infrastructure

Upgrade ESXi using VMware Update Manager

There are two methods to upgrade ESXi. This option uses the VMware Update Manager in the vCenter GUI.

 Open https://marshavcenter.domain.marrcorp.marriott.com
 There will be two options - LAUNCH VSPHERE CLIENT (HTML5) and LAUNCH VSPHERE WEB CLIENT (FLEX). Select
LAUNCH VSPHERE WEB CLIENT (FLEX).
o The FLEX option uses Adobe Flash. If Flash is not enabled, run PowerShell as Administrator and run the
following command:
dism /online /add-package /packagepath:”C:\Windows\servicing\Packages\Adobe-Flash-For-Windows-
Package~31bf3856ad364e35~amd64~~10.0.14393.0.mum”
o Note: VMware Update Manager is not present in the HTML5 interface when vCenter is installed on a
Windows OS.
o Note: If the interface does not load in IE, use Chrome. You will need to allow the plug-in to run.
 Click on the Home button and select Update Manager

 Click on the server > click the Manage tab > click ESXi Images > click Import ESXi Image.

 Brose for the ISO and upload it.

Note: The ESXi version on the ISO must be equal or lower to the version of vSphere being used. You can check
the vSphere version by clicking Help > About VMware vSphere. If it is a newer build, you will receive an error
that the ISO could not be imported. You must either upgrade the vCenter installation or use an older version of

8
VMware ESXi Host upgrades and patching on VRTX infrastructure

the ESXi ISO. Using an older version of ESXi will be OK since it will be patched in following steps.

 Still in Update Manager, click + New Baseline

9
VMware ESXi Host upgrades and patching on VRTX infrastructure

 Use a unique name that will indicate specifically what the baseline is for.
Under Baseline type, select Host Upgrade

 Select the image you uploaded

 Click Finish. You will see the Custom baseline listed.

10
VMware ESXi Host upgrades and patching on VRTX infrastructure

Remediate each host

After loading the ISO as an upgrade option in VMware Update Manager, you can upgrade each host individually. Refer
to the section " Move VMs from host being upgraded" above to move VMs before remediating each host.

 NOTE: If the host shows a status of "incompatible" please refer to Appendix A.

 Go back to Hosts and Clusters > select a host to upgrade > select the Update Manager tab > click Attach Baseline
> select the new baseline you created > OK.

 Click the Remediate button, select Upgrade Baselines, select the Baseline Name you created, click Next

 Select your target (the host), accept the EULA, skip the Advanced options and leave Host remediation options at
the defaults.

11
VMware ESXi Host upgrades and patching on VRTX infrastructure

 Under Cluster remediation options, add a check to "Disable High Availability admission control if it is enabled for
any of the selected clusters. Click Next, then click Finish.

 Monitor Recent Tasks for progress on the Remediate entity task.

 The host will be automatically put into Maintenance Mode while it is upgraded and rebooted.
o The process should take 20-30 minutes
 Repeat on remaining hosts. Refer to the section " Move VMs from host being upgraded" above to move VMs
before remediating each host.

12
VMware ESXi Host upgrades and patching on VRTX infrastructure

Patching ESXi host to current build using VMware Update Manager

After upgrading the major version of ESXi, you may need to update the host with the latest Critical and Non-Critical Host
Patches. VMware Update Manager makes this much easier than the alternate of downloading the compressed patch
file, uploading to a datastore and using SSH to issue CLI commands.

 Note: After upgrading the ESXi version, it may be necessary to go into VMware Update Manager and select
Download patches and upgrades. VUM only downloads patches for the software it knows is in its inventory.
Until now, it did not have the new ESXi version so it must download new patches associated with it. This
happens automatically on a regular basis but can also be forced.

13
VMware ESXi Host upgrades and patching on VRTX infrastructure

 Click on the host to be patched > Update Manager tab > Attach Baseline > select both Critical Host Patches
(Predefined) and Non-Critical Host Patches (Predefined) > click OK.

 Click Scan for Updates > ensure both Patches and Extensions and Upgrades are checked > click OK. Monitor
progress under Recent Tasks.

 Click Stage Patches > ensure both Critical Host Patches are selected and click Next.

 Ensure the host is selected and click Next.

14
VMware ESXi Host upgrades and patching on VRTX infrastructure

 Ensure all available patches and updates are selected and click Next.

 Click Finish. Monitor progress under Recent Tasks.

 Click Remediate
 Ensure both Critical Host Patches and Non-Critical Host Patches are selected and click Next.

 Ensure the host is select and click Next.

15
VMware ESXi Host upgrades and patching on VRTX infrastructure

 Ensure all patches/upgrades are selected and click Next.

 Click Next on Advanced options.

 Click Next on Host remediation options.
 On the Cluster remediation options, select "Disable High Availability adminssion control if it is enabled for any of
the selected clusters" and click Next.

 Click Finish.
 The host will have all selected patches applied and reboot. During this time, it will automatically be placed into
and removed from Maintenance Mode.

16
VMware ESXi Host upgrades and patching on VRTX infrastructure

Remediate CVE-2018-3646
These steps will properly mitigate the VMware vulnerability for ‘L1 Terminal Fault - VMM’ (L1TF - VMM) Speculative-
Execution vulnerability in Intel processors for vSphere: CVE-2018-3646

Do not suppress the warning!

 Migrate all VMs from host

 Place host in Maintenance Mode
 Select host – Configure – System – Advance System Settings – Edit

 Use the filter search box to filter on “hyperthread”

Ensure UserVars.SuppressHyperthreadWarning = 0 (or true)
Ensure the VMkernel.Boot.hyperthreadingMitigation box for Enabled is checked
click OK

17
VMware ESXi Host upgrades and patching on VRTX infrastructure

 Note, if the warning that the host is potentially vulnerable appears after an upgrade, but the
SuppressHyperthreadWarning is set to 0 (or true), reboot the host and it should resolve the warning.

18
VMware ESXi Host upgrades and patching on VRTX infrastructure

Appendix A - Missing VIBS

If you received this error, you will need to access the host console to remove the VIBs:

MISSING_DEPENDENCY_VIBS ERROR: Found=['LSI_bookbank_scsi-mpt3sas_04.00.00.00.lvw-10EM.500.0.0.472560'

These vibs on the host are missing dependency if continue to upgrade.
Remove these vibs before upgrade or use Image Builder to resolve this missing dependency issue.

The ‘mpt3sas’ driver enables support for AVAGO MPT Fusion based SAS3 Controllers. Dell doesn’t support this device
and so this driver can be safely removed. You’ll need to enable command line access
 Open the iDRAC Virtual Console to access the host desktop.
 Log in to the desktop console using F2, then go into Troubleshooting > Enable Console access.
 Press Alt+F1 to get to the command line interface, login and run the command below.
esxcli software vib remove --vibname scsi-mpt3sas
 When it is done, press Alt+F2 to get back to the console GUI and disable the Console access.
Then you can upgrade without running into that error.

If the above doesn’t work, use the –f to force it:

esxcli software vib remove -f --vibname scsi-mpt3sas

19