Hyper-V in the Real World

Performance Tuning and Best Practices

Dave Kawula TriCon Technical Services Inc
B-daveka@microsoft.com / dkawula@triconts.com www.triconts.com www.nitandgritofit.com (BLOG)

Session Objectives
Tools used on the field for the Hyper-V Health Checks Hyper-V BPA (Full and Server Core) MAP Toolkit Hyper-V R2 Best Practices HAVM, Failover Clustering Cluster Validation Wizard (in case of Hyper-V Cluster) Performance Monitor and PAL for Hyper-V R2 Update of Memory Management for Hyper-V R2 SP1

Best Practices Analyzer (BPA)

REQUIREMENTS

Windows Server 2008 R2

Hyper-V Role Update 977238 required on Hyper-V Hosts Server Manager remote access (Optional) Server Manager and BPA modules for Powershell execution
Core or Full Edition

Why Hyper-V BPA?

Integration Components
Installed? What version? Are all of the ICs added to the VMs?

Number of logical processors Supported LPs number on host Number of VPs on each VM VHD type detection Legacy Network Adapters Much more
Inventory and supported number of VPs for each OS Fixed, Dynamic or Differencing Disk inventory Only use it if its mandatory (i.e Citrix provisioning VDI scenarios) http://technet.microsoft.com/en-us/library/ee941122(WS.10).aspx

Hyper-V BPA
Export scan results to HTML format Configure Server Manager for remote administration
http://blogs.technet.com/b/cedward/archive/2011/01/11/hyper-v-bpa-html-report.aspx

Apply specific filters to avoid false positives or known errors

http://blogs.technet.com/b/mghazai/archive/2010/06/08/hyper-v-best-practicesanalyzer-powershell-automation.aspx
Some warning or errors can be safely ignored depending on the your environment or requirements Apply Hyper-V BPA Updates (i.e. KB2485986 ) http://gallery.technet.microsoft.com/scriptcenter/en-us/84d3e608-1404-4ee9-b5e224452ecc8b94

Run BPA remotely with Powershell to a group of servers

Sample PS Script: Hyper-V R2 BPA

Import-Module failoverclusters $OutputPath = "c:\temp\hvbpa.csv" $clustername = "ServerName" $s = Get-ClusterNode -cluster $clustername | New-PSSession invoke-command -session $s -scriptblock { import-module BestPractices Invoke-BpaModel Microsoft/Windows/Hyper-V | out-null Get-BPAResult Microsoft/Windows/Hyper-v | select ResultNumber,@{Name="Server Name"; Expression={hostname}},ModelId,RuleId,ResultId,Severity,NeutralSeverity,Categor y,Title,Problem,Impact,Resolution,Compliance,Help,Excluded } | Export-Csv $OutputPath Remove-PSSession $s

demo

Hyper-V R2 BPA

MAP Toolkit (6.0)

Analyze your portfolio of application for move to Azure Accelerate private cloud planning with Hyper-V Cloud Fast Track onboarding Identify migration opportunities with heterogenous server environment inventory Assess you client environment for Office 365 readiness Determine readiness for Migration to Windows 7 and IE 9 Discover heterogeneous database instance for migrating to SQL Server http://connect.microsoft.com/site297

Hyper-V Security
Hardening Hyper-V and Parent Partition
Hyper-V R2 SKU or Server Core No Other Apps/Roles/Features Patch and A/V Isolate network traffic (separate NICs and VLANs) Bitlocker

Hyper-V Security
Delegating Virtual Machine Management VM Admins no access to Parent Authorization Manager (AzMan.msc) SCVMM and Self Service Portal Harden Management host OS Virtual Machine Servicing Toolkit (VMST) http://technet.microsoft.com/en-us/library/cc501231.aspx Offline VMs in SCVMM library Stopped and saved state VMs on a host VM Templates. Offline VHDs in SCVMM library by injecting update packages

Hyper-V Security
Protecting Virtual Machines Patch and A/V OS hardening Encryption (i.e. EFS) Audit Permission Hyper-V Security Guide http://technet.microsoft.com/en-us/library/dd569113.aspx Microsoft Security Compliance Manager http://technet.microsoft.com/en-us/library/cc677002.aspx

MBSA for Windows Server 2008 R2

MBSA 2.1.1 or greater
MBSA 2.2 recommended

Admin rights on target host to run it remotely Internet access and Firewall exceptions for remote scanning
TCP port 135, 139 and 445 and UDP 137 and 138

Remote Registry service, Server service, Workstation service, File and Printer Sharing service, and Automatic Updates service. MBSA FAQ
http://technet.microsoft.com/en-us/security/cc184922#EWBAC

Why Microsoft Baseline Security Analyzer?

Security Updates Check
Other security checks Local and remote execution
Fast and automated security updates report Administrators number, Administrator blank passwords, Firewall configuration,

Local or remote scan to one or many servers

demo

MAP 6.0 and MBSA (Hyper-V)

CSV SAN Fault Tolerance

I/O Redirected Over CSV Network

VM running on Node 2

Coordinator Node

SAN
SAN Connectivity Failure

VHD VHD

VHD

20

Live Migration
Identical Processors
May enable Processor Compatibility Must be same processor manufacturer

Identically named Virtual Switches Ensure that no ISOs are mounted VM may need to be refreshed in Failover Cluster Specify the network for the VMs to use for Live Migration Dont use IPSec on LM network unless required for security Linux HAVM needs Static MAC Address

CSV Volume
Nodes running Windows Server 2008 R2 Drive letter of the System Disk must be identical %SystemDrive%\ClusterStorage NTLM Protocol enabled on all nodes Latest Identical HBA/Storage/Drivers/Firmware SMB used by CSV Client for Microsoft Networks File and Printer Sharing for Microsoft Networks Check Free Space Use PowerShell Script http://blogs.msdn.com/b/clustering/archive/2010/06/19/10027366.aspx

CSV Settings
Highest priority network = CSV network
Network with lowest metric is the highest priority

Manually set CSV network if not set properly:

Get-ClusterNetwork | ft Name, Metric, AutoMetric, Role ( Get-ClusterNetwork Cluster Network ).Metric=900

Great Guide on Configuring Hyper-V for HA from NetApp http://media.netapp.com/documents/tr-3702.pdf

CSV Performance Baseline

What is the data telling us?
Object: Physical Disk

Counters from Parent Partition:

Disk Reads/sec, Disk Writes/sec, Avg Disk Queue, Current Disk Queue

Multiple nodes are accessing the same CSV

Aggregate data across nodes
Reads/Sec, Writes/Sec

Disk latency issues Check counters within VMs

http://blogs.technet.com/b/kaitling/archive/2011/02/11/deciph ering-performance-data-csv-volumes.aspx

Networking
Best Scenario 4 physical network access points
1 Virtual MachineAccess 2 Management 3 Cluster and CSV 4 Live Migration (5th network when using ISCSI)

Gigabit or faster NICs Teaming ok Ensure latest drivers, use good Server Class NICs All nodes must be on the same logical subnet Configure Network Priority Consider QoS policy for Live Migration Network Ensure networks are aptly named http://technet.microsoft.com/en-us/library/ff428137(WS.10).aspx

Other
Use Possible and Preferred Owners with VM resources Proper Exclusion in Parents Anti-Virus Proper Page File sizing on parent and VMs Ensure proper amounts of RAM per node Base performance on least powerful node (as applicable)
Each node must be able to handle the workload it may be given

SCVMM PRO and SCOM feature for load balancing Use VMQ in place of Chimney Offload
More efficient copying between networks

With SP1 use MemoryReserve registry value on hosts

Hyper-V Cluster Best Practice Checks

Validate Report? Network adapters- Number of? Speed? Metric (Priority) Settings? Role (1-3)? Client Access - Private/Live Migration network, AutoMetric enabled Selected Network for Live Migration? Network configuration - Other iSCSI in use? Verify Proper/Dedicated Network Path Any VMs running on disks that are in "Redirected Access Mode"? Maintenance Mode? Heartbeat monitoring for the Virtual Machine - Enabled/Disabled? Policy for VM recovery/actions Is Persistent Mode/Auto Start Enabled? VMStates? Paused/Online/Offline/Failed/ CSV enabled? State of the disks? (R2 ONLY) How many VMs per (CSV) volume? Optimization available? GPT disk/MBR disk? Multi-site cluster? H/W Logo'd for Win2008/R2? Filter Drivers Loaded? Backup/Restore Process of (CSV or other) volumes Perfmon counters for CSV volumes Processors must be identical across cluster nodes - Processor Compat mode enabled? Is Live Migration working properly? Event logs for failed migrations? System Drive letter must be the same across nodes Verify supported workloads are running on each VM Check firmware version of HBA from validation report and matches on all nodes Verify DSM Module in use Check if network teaming used for ISCSI Is MPIO feature installed Check for proper cluster Quorum Model Storage configured to vendor tested configuration Chkdsk set to run on any drives?

Microsoft Approved!

demo

Hyper-V Cluster Configuration Post-Install steps

Performance monitor and PAL

New PAL version available http://pal.codeplex.com
Version 2.06 include Hyper-V R2 SP1 counters thresholds for Dynamic Memory analysis

Performance monitor and PAL

LPs and Hyper threading
Hyper threadingis not multiplying processors forreal Capacity planningmust bedonewithout Hyperthreading

Support Statements are based on LPs/VPs ratios 1:8 Ratio is the supported limit for Server Virtualizations scenarios 1:12 Ratio is the supported limit for VDI scenarios with Windows 7 w/SP1 > 32 Procs Do NOT enable HT # Physical Processors 2 (1 Dual Core) 2 (1 Dual Core) Hyperthreading Disabled Enabled # Logical Processors 2 4 # Virtual Processors 16 16 LP / VP Ratio 1:8 1:4

Performance monitor and PAL

demo

PAL in action

Customer: Not certain how to size

New virtual machines get 1GB of RAM [no matter what the VM is running]. I only give people more memory if they complain about performance

All VMs get 4GB of RAM [I have no idea what is happening with that memory] and no one complains I take the minimum system requirements and add (insert one: 50%, 100%, 150%)
A vendor tells me their app needs 4GB of RAM. I do not have the time to test this to find out if it is true or not

Virtualization & Memory

Memory
Key Factor to the number of running VMs The most utilized asset in the system, but also a more expensive asset

Statistics on resource utilization of workloads

CPU 10% Memory 40% Network I/O <5% Disk I/O <5%

Customer Requirements
Maximum density, without sacrificing performance Maintain consistent performance Dont provide a feature thats unsuitable for production use

Ballooning
How it works: Increasing the size of the balloon forces the guest to react to memory pressure by releasing unused pages Decreasing the size makes more memory available to the guest

Memory
Balloon

Memory
Balloon

Swap Out

Memory
Balloon Ballloon Deflate Deflate

Swap In

Disk

Disk

Disk

Inflate

External Page Sharing

How it works:
Eliminate redundant copies of memory pages common to more than one virtual machine Hash all memory and store it in a table Identify the common hashes and then Perform a bit by bit comparison

Problems
Page Sharing not dynamic Can take hours to share pages The largest benefit are zero pages Doesnt work with large pages

Dynamic Memory Goals

Higher VM consolidation ratios with minimal performance impact

Dependent on:
How much variation in memory utilization the workloads have How good a job you did of sizing the systems in the first place Work well for both server and desktop workloads Add minimal overhead to the system Pass the that looks right test

Dynamic Memory
Benefits
Enables higher consolidation ratios per host by addressing the greatest limiting factor to consolidation: Memory A Production feature.

Overview
Memory is pooled and dynamically and securely distributed across VMs Memory is dynamically added/removed based VM usage with no service interruption Guest enlightened: guests & Hyper-V work TOGETHER

Adding/Removing Memory
Adding Memory
Enlightened fashion
Synthetic Memory Driver (VSP/VSC Pair)

No hardware emulation Light weight

Removing Memory
Ballooning is more efficient
Messes up task manager in the guest OS

System Requirements
Parent Requirements:

Windows Server 2008 R2 SP1 Microsoft Hyper-V Server 2008 R2 SP1

Guest Requirements:

Windows Server 2003, 2008 & 2008 R2

32-bit & 64-bit versions

Windows Vista and Windows 7

Enterprise and Ultimate Editions only 32-bit & 64-bit versions

Dynamic Memory

Startup & Max

Startup: amount of memory to boot VM Max: dont let the VM above this amount
Default: 64GB
BIOS does not know about DM Guest OS may not know about DM Default: 512MB

Memory Buffer & Priority

Buffer: How much free memory should we try and keep in the VM?
Priority: which VM gets the memory first

Allows for responsiveness to bursty workloads Can be used for file cache I like to configure my virtual machines so that they have ~20% free memory

1-10,000: default is 5,000 The higher the priority, the higher the availability

demo

Dynamic Memory in Action

Call to Action
Contact your Microsoft Partner or Account Manager about available programs PVDPS Use your SA Vouchers Share Jumpstart Hyper-V PoC Hyper-V Cloud Accelerate Program There is help for you Contact us about about your Virtualization needs and how Microsoft can help.

Remember To Complete Your Evaluations!

You could WIN a Samsung Focus Windows Phone 7! Let us know what you liked & disliked! Remember, 1=Bad, 5=Good Please provide comments!
No purchase necessary. The contest is open to residents of Canada (excluding government employees). The Toronto TechDays evaluation form contest begins on October 25th, 2011 and ends on October 26th, 2011. The Vancouver TechDays evaluation form contest begins on November 15th, 2011 and ends on November 16th, 2011. The Montreal TechDays evaluation form contest begins on November 29th, 2011 and ends on November 30th, 2011. Participants can enter the contest in one of two ways: (1) complete and submit an evaluation form by the contest close date; or (2) provide contact information by the contest close date. The draw for Toronto will take place on October 31st, 2011. The draw for Vancouver will take place on November 21st, 2011. The draw for Montreal will take place on December 5th, 2011. The chances of being selected depend upon the number of eligible entries. Selected participants will be contacted by phone and/or e-mail and will be required to answer correctly a time-limited skill-testing question. There are three (3) prizes available to be won. One (1) prize will be given away for each TechDays event in Toronto (October 25-26 2011), Vancouver (November 15-16 2011) and Montreal (November 29-30 2011). The prize consists of a Samsung Focus Windows Phone 7 (handset only; voice and/or data plan not included) (approximate retail value of $499 CAD). The prize will be delivered to the shipping address designated by the winner within 6-8 weeks. The winner may be required to sign a declaration and release form. For full contest rules, please see a Microsoft TechDays representative.

You can email any additional comments directly to td_can@microsoft.com at any time.

Q&A

 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.