You are on page 1of 13




Hypervisor management and optimization
All hypervisors essentially do the same job, but not in the same way. The three top hypervisors all have differences that require specific management tactics.





8 HYPERVISOR MANAGEMENT AND OPTIMIZATION management TAKE STOCK OF YOUR RAM HYPERVISOR and optimization BY MIKE LAVERICK EVALUATE CPU NEEDS SIZE UP YOUR NETWORK VMWARE ESX MICROSOFT HYPER-V CITRIX XENSERVER few best practices and recommendations you can follow to optimize and maintain the three core data center hypervisors— VMware ESX. Microsoft Hyper-V and Citrix XenServer. But that’s a rather simplistic view considering that a server has a finite number of DIMM slots. Failure to purchase the correct amount of memory for current and future demands can lead to unwanted and expensive consequences—having to upgrade your memory and having to purchase additional hosts. On the surface. A little research reveals that there is a marked difference in recommendations and best practices from one hypervisor to another. No matter which hypervisor you select to manage your virtual machines (VMs). TAKE STOCK OF YOUR RAM RAM is the key resource to consider before purchasing a hypervisor. and the ADVANCED VIRTUALIZATION E-BOOK 2 . CPU. In general. all three platforms have similar resource demands as well as challenges to solve. mainly because it’s what most people run out of first. It’s very common for virtualized environments to run low on memory long before they run out of CPU cycles. the discussion of hypervisor optimization might seem trivial since hypervisors are fairly basic and don’t come with the usual baggage that a full-blown operating system has. you can say that the right amount of memory to purchase is as much as you can afford. But it is possible to modify key default settings and tweak various vendors’ hypervisors. The easiest way to control these demands is to systematically exTHERE ARE A plore the four-core resources of any system: memory. disk utilization and network resources.

In the long run. you may be attracted to the performance gains that x64 OSes can deliver from both a memory and CPU perspective. for example. Additionally. on a VM level. But you may discover that your application doesn’t notice all of the benefits of the new architecture because it is 32-bit infrastructure. having fewer servers with a large amount of memory will be more cost efficient than having more smaller servers. 4 GB. In general. the more VMs you create with more than one vCPU. machines and the software applications they contain and ask if virtual symmetric multiprocessing. “Will my application run on 32bit architecture in a way that the vendor will support?” Finally. requiring more memory just to power on the virtual machine. x64 OSes come with greater virtualization overhead. Additionally. it forms the bedrock of server consolidation where two. the more work the underlying hypervisor has to do to schedule each CPU within the guest vCPU. will really pay any dividends. this total cost is bound to the number of completed sockets in a rackmounted server or chassis of blades. cooling and licensing. if you’re working with a hypervisor that doesn’t allow for intelligent placement of the virtual machine on the correct host. SIZE UP YOUR NETWORK EVALUATE CPU NEEDS VMWARE ESX MICROSOFT HYPER-V CITRIX XENSERVER Despite the rise of four-core and sixcore CPUs. then ADVANCED VIRTUALIZATION E-BOOK 3 . requiring more memory just to power on the VM. 8 GB and 16 GB DIMMs is not a small one from a cost perspective. Therefore. But resources do not scale to “infinity and beyond. be careful about the excessive use of multiple vCPUs within a VM because it may not produce the performance dividend you’re looking for. In the case of VMware. this isn’t a bad thing in its own right. The question to ask is.” so it’s important not to overuse the physical CPUs. Closely examine your virtual Pound for pound. the excessive use of vCPU can increase scheduling collisions—more than one vCPU executes on the same core at the same time. or SMP. x64 OSes come with greater virtualization overhead. Pound for pound. three and four VMs share the same underlying physical CPU. especially when you consider the total cost of power.8 HYPERVISOR MANAGEMENT AND OPTIMIZATION TAKE STOCK OF YOUR RAM EVALUATE CPU NEEDS jump from 2 GB. A legacy service that isn’t multiprocessor aware will simply not use all of the CPU efficiently.

I recommend using discrete and different network interfaces for each type of traffic that you intend to push in and out of the physical box. but recent enhancements on the Intel Nehalem CPUs indicate that Hyper-Threading is back and worth enabling for most workloads. You can find some helpful information online regarding which processor works well in virtualized environments. Often. Most vendors often have a confusing array of different CPU specifications for the same rack-mounted server or blade.” which degrades performance when too many virtual machines try to access the same physical CPU. These packets are “tagged” in such a way that when the packets arrive at the physical network switch. Additionally. It makes no sense to put two or more resource-intensive VMs on the same physical host. it can read the data and send the frame to the appropriate network. In short. In the past. there can be a huge price difference between one CPU type and another—with an almost negotiable gain in performance. The process of VLAN tagging significantly reduces the number of physical network interface cards (NICs) that a hypervisor needs. You want to avoid excessive over- TAKE STOCK OF YOUR RAM EVALUATE CPU NEEDS It makes no sense to put two or more resourceintensive VMs on the same physical host. subscription or “contention.8 HYPERVISOR MANAGEMENT AND OPTIMIZATION it’s up to you to blend the VMs on an array of hosts. most virtualization specialists would have recommended disabling Hyper-Threading on physical CPUs. ADVANCED VIRTUALIZATION E-BOOK 4 . the new Intel Nehalem and AMD Shanghai processors offer significant CPU gains over SIZE UP YOUR NETWORK VMWARE ESX older Xeon and Opteron chipsets. then the amount of onboard memory cache on the CPU will make more of a difference than the overall amount of CPU cycles. Each type of traffic should reside in its own broadcast domain. ideally using VLAN TAGGING is a process in which the hypervisor adds an additional identifier to Ethernet frames. if you think your workload is going to be CPU bound. MICROSOFT HYPER-V SIZE UP YOUR NETWORK CITRIX XENSERVER From a network perspective.

It’s possible for network traffic to be “blended” together. it’s less expensive to drive live migration traffic across a man- agement network. the actual amount of ADVANCED VIRTUALIZATION E-BOOK 5 . however. so it’s not uncommon for organizations to also use the management network to carry out live migration tasks. In these few instances. Here’s a look at the three biggest hypervisors by vendor and considerations when implementing each. Generally. HYPERVISOR TEARDOWN: VENDOR BY VENDOR MICROSOFT HYPER-V CITRIX XENSERVER All hypervisors essentially have the same job—spawn virtual machines and allocate a slice of memory. Most live migration traffic is unencrypted and contains sensitive VM memory data.8 HYPERVISOR MANAGEMENT AND OPTIMIZATION TAKE STOCK OF YOUR RAM VLANs and VLAN tagging to maintain network boundaries. the following numbers are recommended: EVALUATE CPU NEEDS SIZE UP YOUR NETWORK VMWARE ESX 1 At least one NIC for management traffic 1 At least two NICs connected for VM traffic 1 At least two NICs connected for any IP-based storage system used (network-attached storage or iSCSI) 1 At least one NIC for any live migration or VMotion feature Hypervisors network demands alone can be daunting. 1 VMware ESX Generally recognized to have one of the smallest hypervisor footprints on the market. In some cases. With that said. So companies will inevitably have to make compromises based on cost. As a network gateway. Few customers have experienced poor network performance in the virtual data center. However. most management networks are secured and private. CPU. vendor-specific tweaking is required to gain the most from the virtualization layer you have selected. not all hypervisors are the same architecturally. For example. which is a shame because it would be suitable as spare bandwidth for something else the network needs. The features you select will ultimately dictate the number of physical network interface cards (NICs) you need and whether the traffic they carry requires redundancy and fault-tolerance. but that has its security consequences. most hypervisor vendors caution that this is against best practices. the root cause was attributed to external configuration issues such as poor DNS queries and structures or out-ofdate routing table settings that needed immediate attention. And while there are some fundamental aspects of hypervisors where performance optimization will be very similar. the management network isn’t heavily used. performance and security. disk space and network usage to the workload.

’s publically stated position that ESXi is the future of its hypervisor. is commonly recognized as the more mature cousin to ESXi.8 HYPERVISOR MANAGEMENT AND OPTIMIZATION TAKE STOCK OF YOUR RAM EVALUATE CPU NEEDS SIZE UP YOUR NETWORK space VMware ESX consumes depends on which flavor of ESX you purchase. which has a fully interactive service console. VMware assigns a block of memory to the service console on a sliding scale with 300 MB allocated on an 8 GB server and up to 703 MB allocated to a 128 GB server. you’re more likely to see ESX Classic in most data centers. CITRIX XENSERVER ADVANCED VIRTUALIZATION E-BOOK 6 . Currently. ESX Classic. it’s good practice to max out the amount of memory assigned to it. The value can be adjusted by modifying the settings in the Configuration tab as well as the Memory option in the Hardware pane (Figure 1). With the amount of memory in most hosts moving from 32 GB to the 64 GB or 72 GB range. Adopting the ESXi version of the VMWARE ESX FIGURE 1: MICROSOFT HYPER-V Use VMware ESX's Configuration tab to adjust the value of memory allotted to the host. Despite VMware Inc. I recommend allocating the maximum of 800 MB to the service console when possible to reduce the impact of management agents that are installed on it. The Classic edition allocates a block of memory to the ancillary management front end called the service console.

which possesses AMD-V and Rapid Virtualization Indexing (RVI) support. HBA and LUN as well as LUN masking. A common problem with enabling VMware fault tolerance is that using power-saving features like Intel SpeedStep and AMD PowerNow can undermine the feature’s reliability. ESXi presents a smaller attack service area that should leave it less vulnerable to attacks and easier to patch. review the BIOS settings that control these features sets. Intel VT-X CPU. When comparing SAN boot or USB boot. both have this capability. available memory probably matters more than overall speed. which complete the boot process. if you want to take advantage of VMware’s fault tolerance feature. With that said.8 HYPERVISOR MANAGEMENT AND OPTIMIZATION TAKE STOCK OF YOUR RAM EVALUATE CPU NEEDS SIZE UP YOUR NETWORK platform will avoid this issue altogether. Additionally. which possesses Extended Page Tables (EPTs) and the AMD CPU. you must select an x64 process because VMware no longer produces a 32-bit version of its platform. In the future. It involves a complex interlocking series of settings from the server BIOS. As a general rule. VMware will likely support the Preboot Execution Environment (PXE) booting for ESX. The goal should be to use a stateless ESX host in which there is no configuration or enduser data held on the host itself. VMware claims that it will not produce the service console edition forever. VMWARE ESX MICROSOFT HYPER-V CITRIX XENSERVER 3 Buying the right amount of CPUs: From a CPU perspective. 3 Storage recommendations for ESX: I also recommend that businesses using ESX Classic edition use a boot-from-SAN configuration as well as USB or SD cards to boot ESXi systems. you should look for a CPU that supports the vLockstep attribute. Once loaded in memory. I recommend buying the CPU that has the largest onboard memory cache you can reasonably afford. which allows for the successful mirroring of the primary and mirrored secondary virtual machine. SAN booting is more complicated and troublesome from a configuration perspective. and it makes VMware ESXi an attractive option for many data centers. 2 Microsoft Hyper-V Microsoft Hyper-V is available in several formats: as a standalone ADVANCED VIRTUALIZATION E-BOOK 7 . Although ESX will manage the mapping of VM memory to physical memory using its internal memory management unit (MMU). Since it doesn’t have a service console. it’s better to have a CPU that offers this MMU functionality as a hardwareassist feature. However. ESXi can be run completely from either USB or SD memory cards. you don’t need to worry about configuration issues.

As is the case with CPU resources. which come with generous licensing conditions that allow unlimited copies of the Datacenter edition. features and quality of support than raw throughput. Using less than 500 GB creates a situation where you need to manage too many volumes that will fill up rapidly. which causes you to run out of IOPS before reaching capacity. In the case of excessive IOPS. Disk I/O rarely constitutes a bottleneck unless excessive queue depths are being created from storing too many virtual machines on too few LUNs/volumes without sufficient spindles. although storage vendors battle over which is better— Fibre Channel. manageability. As is the case with VMware ESXi. however. you want to avoid oversubscribing the same LUN/volume while storing too many VMs on the same underlying spindles. Microsoft Hyper-V SIZE UP YOUR NETWORK VMWARE ESX The history of storage protocol wars perspective. You have the option to run them all in a core mode where the graphical shell is suppressed in favor of a command-line-only distribution. That way. A more appealing version would be to run the server core versions of either the Enterprise or Datacenter edition. iSCSI or NFS—each method offers about the same performance characteristics.8 HYPERVISOR MANAGEMENT AND OPTIMIZATION TAKE STOCK OF YOUR RAM EVALUATE CPU NEEDS platform. I FROM A DISK MICROSOFT HYPER-V CITRIX XENSERVER ADVANCED VIRTUALIZATION E-BOOK 8 . creating LUNS/volumes in excess of 1. Currently.500 GB is a safe bet. I don’t recommend any hard and fast numbers. Enterprise edition and Datacenter edition. Additionally. it’s possible to dedicate an entire LUN/volume and the bandwidth of your Fibre Channel HBAs or network interface cards to service read/ writes. between 500 GB and 1.5 TB usually wastes space. in graphical user interface (GUI) mode Standard edition. The correct argument is more about usability. For corporate customers. running a stripped down version of Hyper-V also reduces the attack surface from a security perspective and frees up valuable resources. Organizations often ask what the optimum LUN/volume size is. the standalone model will probably be insufficient given that it is limited to addressing 32 GB of RAM. companies can create VMs without worrying about the licensing implication of VM sprawl.

8 HYPERVISOR MANAGEMENT AND OPTIMIZATION does not support memory overcommitment. Microsoft has indicated that Service Pack 1 will offer a feature for handling memory over-commitment called “Dynamic Memory. use up to two quad-port NICs to complement the two onboard network interfaces common on most rack-mounted Although dynamic disks offer better disk space utilization. Older Windows OSes such as Windows 2000 and NT4 lack these enhancements. At the upper end. Microsoft recommends using versions of Windows that have been “enlightened” for use on Hyper-V such as Windows Vista. but in parts of the user interface that concern networking. test and development environments where raw I/O may be less of a pain point. you might experience heavy traffic loss combined with subsequent retransmits. it’s important to choose synthetic devices over emulated devices when possible. Microsoft doesn’t use this specific terminology in the Hyper-V suite. Windows 7 and Windows Server 2008. mends increasing the buffer size on a per-virtual-machine basis from 1 MB to 2 MB to alleviate this. Hyper-V will need a significant number of physical NICs—especially if you’re using IP-based technologies such as iSCSI. so optimizing your memory efficiency is critical. Microsoft recom- EVALUATE CPU NEEDS SIZE UP YOUR NETWORK VMWARE ESX MICROSOFT HYPER-V CITRIX XENSERVER 3 Network selection and performance: From a network perspective. these appear as “Network Adapter” and “Legacy Network Adapter. Use fixed disks for production VMs and dynamic virtual disks for test and dev. If you have a large network load.” TAKE STOCK OF YOUR RAM servers. At the physical level. many people ADVANCED VIRTUALIZATION E-BOOK 9 . As a rule of thumb. Although dynamic disks offer better disk space utilization.” You can switch from one to the other after you have installed the Integration Components to the VM. they’re not without a performance hit. they’re not without a performance hit. There also may be extreme cases where you’ll want to adjust the guest OS’s ReceiveBufferSize and SendBufferSize in the Microsoft registry. Use both synthetic devices with integration tools to get the best performance from a CPU perspective. The best bet is to use fixed disks for production VMs and use dynamic virtual disks for desktops.

but reduce the number of VMs per host. This means that a 30 GB virtual disk would consume 30 GB of physical storage. At the disk-level.6. some data center policies may be non-negotiable. The problem is that you never quite ADVANCED VIRTUALIZATION E-BOOK 10 . which gives a Hyper-V host the capacity to present LUNs/volumes directly to the virtual machine.exe 3 Citrix XenServer CITRIX XENSERVER 3 Hyper-V trouble spots: Antivirus software configuration can be an area of contention in Microsoft Hyper-V because it can degrade performance. a 30 GB virtual disk would use just a couple of megabytes of physical storage and would grow dynamically on demand as data is written inside it. such as C:\Users\ Public\Documents\Hyper-V\ Virtual Hard Disks 1 Snapshot directories 1 Vmms. a virtual disk is created upfront as a flat file whenever a VM is created.exe and Vmwp. Microsoft recommends using fixed virtual disks over the newer Dynamic VHD format. This could possibly solve the virtual machine memory management paradox— allocating a large amount of memory to VMs will improve their performance. which frees an administrator from allocating a fixed amount of memory to the virtual desktop before powering it on. This represents a better use of physical disk space because you don’t waste space storing empty blocks of data that are waiting for the VM to write to them. so if you’re forced to install an AV product. With fixed disks. Of course. Allocating a small amount of memory increases the consolidation ratio but reduces performance. One of the downsides of virtual desktops is that they consume resources even when end users aren’t connected to them. the general guidelines should not include these locations or these executables until a systemwide scan has been completed: There are several tricks for tuning XenServer specifically for virtual desktop loads. When correctly configured. such as C:\ProgramData\ Microsoft\Windows\Hyper-V 1 Custom VM configuration directories. With a dynamic disk.8 HYPERVISOR MANAGEMENT AND OPTIMIZATION TAKE STOCK OF YOUR RAM EVALUATE CPU NEEDS SIZE UP YOUR NETWORK VMWARE ESX MICROSOFT HYPER-V recommend using the Pass-Thru Disk format (PTD). 1 Default and custom VM configuration directory. Hyper-V should not be used as a workstation nor should it access the Internet from its management network. and many of Citrix Systems Inc.’s performance optimization methods center on the company’s long history of delivering a client experience to end users. Citrix introduced Dynamic Memory Control (DMC). In XenServer 5.

when it needs it or if it will only need it for a certain period of time. the RAM allocation that Y never quite ou know how much memory a VM needs. leaving approximately 352 MB from which each VM (DomU) can take its 6 MB. Each VM lives inside a DomU. when it needs it or if it will only need it for a certain period of time. a bottleneck could occur if the Dom0 isn’t configured correctly. It acts like a broker for all network communication from VMs to the physical world. and although this isn’t a huge amount of memory. Once you reach about 60 MICROSOFT HYPER-V CITRIX XENSERVER Storage and capacity planning library researching storage optimization and planning your virtual environment capacity for Citrix XenServer. look to these resources for help. So. takes 400 MB of this for itself.8 HYPERVISOR MANAGEMENT AND OPTIMIZATION know how much memory a VM needs. Dom0 TAKE STOCK OF YOUR RAM EVALUATE CPU NEEDS SIZE UP YOUR NETWORK VMWARE ESX 3 Configuring XenServer’s Dom0: XenServer has a privilege VM called Dom0. WHEN YOU START q XenServer Performance Monitoring for Scalability Testing q Evaluate Performance of Block Type Storage Repositories for XenServer q Proper Storage Allocation for Hosted Virtual Desktops q Software iSCSI Storage Repository Detachment ADVANCED VIRTUALIZATION E-BOOK 11 . Dom0 receives an allocation of 752 MB. comes from the Dom0 partition can add up. such as when the user is connected to his virtual desktop. and network communication occurs from the netfront component in DomU to its netback component. Each VM that’s created (DomU) will use 6 MB of RAM at power on. At boot up.

And. the memory pool is Jo Maitland Executive Editor jmaitland@techtarget. dd. incorrect planning can reduce VM performance and degrade the manageability of the host through the Colin Steele Site Editor Michelle Boisvert Managing Editor mboisvert@techtarget. netstat and ifconfig. If your performance woes are not RAM related but IOPS bound. In 2009. ©2010 TECHTARGET. I recommend that you avoid individual commands that let you monitor network and desk activity such as iostat. VMware vSphere4 and VMware Site Recovery Manager. organizations that previously used different versions of VMware ESX will know it has a memory reservation for its management layer. He is also the owner and author of the virtualization website and blog RTFM Education. Windows and Citrix. tcpdump. The memory allocations in Citrix XenServer are much more specific.8 HYPERVISOR MANAGEMENT AND OPTIMIZATION TAKE STOCK OF YOUR RAM EVALUATE CPU NEEDS SIZE UP YOUR NETWORK VMWARE ESX MICROSOFT HYPER-V CITRIX XENSERVER VMs. Laverick received the VMware vExpert award and helped found the Irish and Scottish user groups. Microsoft Hyper-V needs a chunk of memory just to be viable. Cathy Gagne Editorial Director Marc Laplante Publisher mlaplante@techtarget. ALL RIGHTS RESERVED.” which pull all of these tools together into handy scripts that send out all of the data you actually need. ADVANCED VIRTUALIZATION E-BOOK 12 . and he has been involved with the VMware community since 2003. I Mike Laverick is a professional instructor with 15 years experience in technologies such as Novell. it can crash XenServer. This situation is common in a Linda Koury Art Director of Digital Content Martha Moore Copy Editor mmoore@techtarget. Laverick is a VMware forum moderator and member of the London VMware User Group Steering Committee. Of course. However. where he publishes free guides and utilities aimed at VMware ESX/VirtualCenter users. in a worst-case scenario. Focus instead on Citrix’s various “shell scripts. He has had books published on VMware Virtual Infrastructure 3.

go to visit www. or to communicate directly with Dell via a variety of online channels. Dell is a leading global systems and services company and No. Inc. ADVANCED VIRTUALIZATION E-BOOK 13 . Uniquely enabled by its direct business (NASDAQ: DELL) listens to customers and delivers innovative technology and services they trust and value.: Dell Inc. visit www. For more 34 on the Fortune 500.FROM OUR SPONSOR q Virtualization: Consolidation to Cloud Foundation q Forecast: Cloudy with a Chance of Virtualization q Implementing and Expanding a Virtualized Environment About To get Dell news direct.