Information Guide

VMware VMotion and CPU Compatibility

VMware Infrastructure 3
VMwareVMotiontechnologyallowsuserstomigraterunningvirtualmachinesbetweencompatiblephysical serverswithzerodowntime.Toensuresuccessfulmigrationandsubsequentfunctioningofthevirtual machine,youmustrespectcertaincompatibilityconstraints.ThisinformationguidedescribesCPU compatibilityissuesandoutlinessomeCPUcompatibilitychecksthatVMwareVirtualCenterperformsbefore allowingmigrationwithVMotion.ItdescribeswhysomeCPUcompatibilityconstraintsmakeVMotion possibleonlybetweencertainrevisionsofCPUs.Theappendicesdetailsomedifferencesinfeaturesand extensionsincurrentCPUsanddescribeproceduresyoucanusetorelaxsomeCPUcompatibilityconstraints tofacilitateVMotion. Thisguideisprimarilyintendedfor:

Purchasingmanagersandsystemadministratorswhoconfigureandpurchasenewserversforadata centerupgrade SystemadministratorswhowanttoperformmigrationswithVMotionorwanttorelaxCPUcompatibility constraintsinordertocircumventVMotionchecks HighlevelITstaffwhomakedecisionsonpotentialdatacenterupgrades AnyonewhowantstounderstandhowCPUcompatibilityconstraintscanaffectlivemigrationofvirtual machines

Thisguidecoversthefollowingtopics:

IntroductiontoVMwareVMotiononpage 1 CPUCompatibilityRequirementsforVMotiononpage 3 EnhancedVMotionCompatibilityonpage 6 Conclusiononpage 6 Referencesonpage 7 AppendixA:CPUIDandx86ImplementationDifferencesonpage 7 AppendixB:CPUIdentificationMasksonpage 10 AppendixC:RelaxingVMwareVMotionCPUCompatibilityConstraintsonpage 11

Introduction to VMware VMotion

TheVMwareVMotionfeature,partofVirtualCenter1.0andlaterreleases,allowsyoutomigraterunning virtualmachinesfromonephysicalmachinetoanotherwithnoperceivableimpacttotheenduser.Youcan useVMotiontoupgradeandrepairserverswithoutanydowntimeordisruptionsandalsotooptimize resourcepoolsdynamically,resultinginanimprovementintheoverallefficiencyofadatacenter.

Copyright 2008 VMware, Inc. All rights reserved.

VMware VMotion and CPU Compatibility

Key Concepts
ThefollowingtermsandconceptsareimportanttounderstandingtheVMotiontechnologyandthe compatibilityconstraintsthataffectuseofVMotion. Host AphysicalserverthatispartoftheVMwareInfrastructurehardwareresourcepool.Thishostmaybe usedtorunoneormorevirtualmachines. Migration of a virtual machine Theprocessofmovinganentirevirtualmachinefromonehosttoanother. Completevirtualizationofallcomponentsofamachine,suchasCPU,BIOS,storagedisks,networking,and memoryallowstheentirestateofavirtualmachinetobecapturedbyasetofdatafiles.Therefore,movinga virtualmachinefromonehosttoanotheris,inessence,aspecializeddatatransferbetweentwohosts.Based onthestateofthevirtualmachineduringmigration,themigrationisreferredtoascoldorhot.

Coldmigration:Migrationofavirtualmachinethathasbeenpoweredoffonthesourcehost.Thevirtual machineispoweredonagainonthedestinationhostafterthetransferofthevirtualmachinestateis completed. Hotmigration:Migrationofavirtualmachinethatispoweredon.Thevirtualmachine(andapplications) previouslyrunningonthesourcehostcontinueexecutiononthedestinationhost,withoutdetectingany changes,afterthehotmigrationiscomplete.Basedonavailabilityofthevirtualmachineduring migration,hotmigrationfallsintothefollowingsubcategories:

Suspend/resumemigration:Involvessuspendingarunningvirtualmachine,thenresumingthe suspendedvirtualmachineonadifferenthost. Livemigration(VMotion):Migrationofarunningvirtualmachinefromasourcehosttoa destinationhostwithoutanydisruptions(downtime)totherunningvirtualmachine.

CPU microarchitecture Theinternalarchitecture,includingdesignandinteractionofdifferentcomponents, oftheCPU.ThemicroarchitectureofaCPUisanimplementationofaparticularinstructionsetarchitecture (ISA).DifferentmicroarchitecturesmightimplementthesameISAirrespectiveofCPUvendor.Forexample, IntelCorebasedCPUsdifferfromtheirP4counterpartinmicroarchitecture,butimplementthesamex86ISA. Privileged code Setofinstructionsthatrunatthehighestprivilegelevelandcanthereforeperform operationscriticaltothefunctioningofamachineandmayaffectallapplications.Onx86CPUs,thehighest privilegelevelis0.Typically,operatingsystemrelated(kernel)instructionsrunatthehighestprivilegelevel. Anyinstructionmayrunintheprivilegedmodeiftheoperatingsystemspecifiesthatitdoso. Nonprivileged code Instructionsthatbelongtoapplicationsoftware.Notallinstructionscanrunin nonprivilegedmode.Applicationssometimesneedtoperformprivilegedoperationsanduseservices(suchas APIs)providedbytheoperatingsystemtoachievethis.Onx86systems,applicationstypicallyrunata privilegelevelof3,thoughitiscorrectforapplicationsrunningatprivilegelevelsof1and2tobecategorized asnonprivilegedcode. CPUID instruction Anx86assemblyinstructionusedforprocessorandfeatureidentification.This instructionistheprescribedandstandardizedmethodforsoftwaretodeterminethesetoffeaturesand instructionsavailableonthecurrentCPU.

Applications of VMotion
Theabilitytomigratearunningvirtualmachineacrossdifferenthostswithoutanyperceivableimpacttothe enduseropensupawidearrayofapplications.SomeimportantapplicationsofVMotionare:

Hardwaremaintenance:VMotionallowsyoutorepairorupgradetheunderlyinghardwarewithout schedulinganydowntimeordisruptingbusinessoperations. Optimizinghardwareresources:VMotionletsyoumovevirtualmachinesawayfromfailingor underperforminghosts. VMwareproductupgrades:VMotion,coupledwithdiskmigrationcapabilities,allowyoutoupgrade fromanolderversionofVMwareESX.Youcanalsousethisfeaturetoupgradebetweenincompatible versionsoftheVMwareVirtualMachineFileSystem(VMFS)whilekeepingthevirtualmachinelive.

Copyright 2008 VMware, Inc. All rights reserved.

VMware VMotion and CPU Compatibility

VMwareDistributedResourceScheduler:VMwareDistributedResourceScheduler(DRS)continuously monitorsutilizationacrossresourcepoolsandallocatesresourcesamongvirtualmachinesbasedon currentneedsandpriorities.Whenvirtualmachineresourcesareconstrained,DRSmakesadditional capacityavailablebymigratinglivevirtualmachinestoalessutilizedhostusingVMotion.

AmoredetailedlistofadvantagesandapplicationsofVMotionisavailableintheVirtualCenterVMotion featureintroductionontheVMwareWebsite(seeReferencesonpage 7foralink).

Requirements for VMotion

InordertofacilitateVMotionbetweenhosts,eachhostmustmeetcertainbasicrequirements:

Datastorecompatibility:Thesourceanddestinationhostsmustusesharedstorage.Youcanimplement thissharedstorageusingaSANoriSCSI.ThesharedstoragemayuseVMFSorsharedNAS.Disksofall virtualmachinesusingVMFSmustbeavailabletobothsourceandtargethosts. Networkcompatibility:VMotionitselfrequiresaGigabitEthernetnetwork.Additionally,virtual machinesonsourceanddestinationhostsmusthaveaccesstothesamesubnets,implyingthatnetwork labelsforeachvirtualEthernetadaptershouldmatch.YoushouldconfigurethesenetworksoneachESX host. CPUcompatibility:ThesourceanddestinationhostsmusthavecompatiblesetsofCPUs.This requirementisdiscussedindetailinthisguide.

ThisguideoutlinestheCPUcompatibilityrequirementsforVMwareVMotion.RefertotheVMware Infrastructure3BasicSystemAdministrationguide(seeReferencesonpage 7)formoreinformationabout othercompatibilityrequirementsforVMotion.

CPU Compatibility Requirements for VMotion

VirtualCenterperformsCPUcompatibilitycheckstoensurethatavirtualmachinecanperformnormallyon thedestinationhostaftermigrationwithVMotion.ItexpectstheCPUsonthesourceanddestinationhoststo providethesamesetoffeaturestothevirtualmachinesothelivevirtualmachineandaccompanying applicationsdonotcrash.ThesameCPUcompatibilitychecksapplytosuspend/resumemigration.This sectionexaminesthereasonsthatCPUcompatibilitychecksforVMotionarenecessary.

Rapid Evolution of CPUs and the x86 Architecture

CPUsfrequentlyundergotremendousimprovements,manyofwhichrequireaugmentationstothex86 architectureitself.Forexample,theadditionofSSE3instructionsinIntelCPUsfacilitatesfasterfloatingpoint operations,whichinturnresultsinbetterperformanceformultimediaapplications.Severalsuchadditions supportspecificclassesofapplications.Furthermore,differentCPUvendorsmaychoosetoextendthex86 instructionsetindifferentwaysbasedontheapplicationstheydecidetotarget.Forexample,manyAMD CPUssupportasetof3DNow!instructionsformultimediaprocessingcapabilities.IntelCPUsaddressthe sametargetapplicationswithasetofSSEinstructions.Thesesetsofinstructionsdifferinmanyrespects. Tosummarize:

ManufacturersconstantlyreviseCPUs. NewerrevisionsofCPUsmighthavenewinstructionsorfeaturesthatwereunavailableinolderrevisions. DifferentCPUvendorsmaychoosetoimplementdifferentinstructionstoaddressthesameconcerns.

Feature Determination and CPUID

SoftwarecanusetheCPUIDassemblylanguageinstructiontoobtaintheexactsetoffeaturessupportedbyan x86CPU.DetailsabouttheCPUIDinstructionareavailableinAppendixA:CPUIDandx86Implementation Differencesonpage 7.IfyouwanttodeterminethesetoffeaturessupportedbyaCPU,youcanusethe bootableCPUIDimageavailablefromtheVMwareWebsite

Copyright 2008 VMware, Inc. All rights reserved.

VMware VMotion and CPU Compatibility

(http://www.vmware.com/download/shared_utilities.html).ForinformationonusingthisCPUIDISOimage, seeAppendixC:RelaxingVMwareVMotionCPUCompatibilityConstraintsonpage 11.Youcanalsouse theManagedObjectBrowserfeatureofESX,whichyoucanaccessusingaWebbrowser,todeterminetheset offeaturesavailableonthehostCPU. CPUvendorsrecommendthatapplicationsusetheCPUIDinstructionforfeaturedetermination.Wellbehaved systemandapplicationsoftwaredoesso.AtypicalapplicationdeterminesthesetofavailableCPUfeaturesby executingCPUIDonlyonceasitisstarting.Basedontheresults,theapplicationmightcontinuetousesomeof thesefeaturesuntilitisshutdown. However,notallapplicationsarewellbehaved.SincetheCPUIDinstructionwasintroducedonx86CPUsonly intheearly1990s,legacysoftwaremightnotfollowthisrecommendedmethodoffeaturedetectionandmight usecustommethodsinstead.ApplicationsmightusethesupportforaspecialinstructiononaparticularCPU asanindicationoftheavailabilityoffeaturesrepresentedbythatinstruction.Forinstance,existenceofthe ADDPDassemblyinstructioncouldimplypresenceofallSSE2instructions.Iftheapplicationexecutesthis specialinstructionoptimisticallyandreceivesanUnDefinedinstructionexception(#UD)fromtheCPU,it handlesthisexceptionandassumestheclassoffeaturesisunsupported.Becausetheapplicationhandlesand consumestheexception,thisprocessmaybetotallytransparenttotheenduser.Suchtrycatchabort exceptionsarebuiltintoolderapplicationsoftware,andtheapplicationsmaycontinuetoworknormallyeven iftheydonotadheretomethodsrecommendedbyCPUvendors. Therefore,itisimportanttonotethefollowing:

AtypicalapplicationqueriestheCPUforsupportedfeaturesonlyonceduringstartupor initializationandcontinuestousethesefeatures(ifappropriate)aslongastheapplicationisactive. ThoughtherearerecommendedmethodsofdeterminingCPUfeatures(viaaCPUIDinstruction),different applicationsmayhavedifferentmethodsofdeterminingCPUsupportforcertainfeaturesorinstructions.

ForinformationonwritingapplicationsthattestCPUfeaturesinawaythatfacilitatesuseofVMotion,seethe VMwareknowledgebasearticleTestingandUsingNewFeaturesinCPUs(foralink,seeReferenceson page 7.)

High-Performance Virtualization
Highperformancevirtualizationrequiresdeliveringvirtualizationatnearnativeperformance.Forthistobe technicallyfeasiblewhilemaintainingcorrectness,itisnecessarytoexecutemanyinstructionsdirectlyonthe underlyinghardware.Utilizingunderlyinghardwareresourcestofacilitatehighperformanceexecutionis trueofanyvirtualizationlayer.Thisensuresapplicationsrunninginsideavirtualmachinedonotencounter majorslowdowns.Thevirtualizationlayerinterceptsonlycertainselectinstructionsthatreallyrequire interceptionandsubsequentemulation. Thex86ISAallowsonlyinstructionsbelongingtotheoperatingsystemorkerneltobeinterceptedbythe traditionalvirtualizationlayerthatis,avirtualizationlayerthatdoesnotrelyonhardwaresupportfor virtualizationoronparavirtualization.Theseinstructionsrunatthehighestprivilegelevel,makingthem privilegedcode.Typically,instructionsthatrunatlowerprivilegelevel(nonprivilegedcode)arepassed throughtotheunderlyinghardwareunimpeded.TheCPUIDinstructioncanbeexecutedbytheoperating system(privilegedcode)aswellasbyapplications(nonprivilegedcode).Ifthisinstructionispartofan application,itexecutesdirectlyontheunderlyinghardware,therebygivinganapplicationalistoffeatures availableontheunderlyinghardware.IfCPUIDispartofprivilegedcode,thevirtualizationlayercanintercept itifemulationisneeded.Thismethodologyarisesfromattemptingtovirtualizetheinherently nonvirtualizablex86ISA(seeFormalRequirementsforVirtualizableThirdGenerationArchitecturesin Referencesonpage 7). Therefore,evenwhenapplicationsarerunningwithinavirtualmachine,thefollowingmaybetrue:

Applicationshavemanyoftheirinstructionsrunningdirectlyonunderlyinghardware. InstructionssuchasCPUIDmightbeexecutedasanonprivilegedinstructionsandthereforerundirectly ontheunderlyinghardware,potentiallytyinganapplicationtothesetoffeaturessupportedbytheCPU onaparticularhost.

Copyright 2008 VMware, Inc. All rights reserved.

VMware VMotion and CPU Compatibility

Live Migration
Migratinganentirevirtualmachinewhileitisrunningisusefulonlyifapplicationscontinuetooperate normallyaftermigration.Applicationstypicallyincludenonprivilegedinstructions,whichmightexecute directlyontheunderlyinghardwareCPU.Ifapplicationsaretocontinueperformingnormallyafteralive migration,theyneedtoexecuteinstructionsandutilizefeaturesthatwereavailableonthesourcehost hardwareevenwhentheyarerunningonthedestinationhosthardware.Thisisrequiredeventhoughthese applicationsareexecutingwithinthesamevirtualmachine. Coldmigration,ontheotherhand,ensuresthatthevirtualmachineandunderlyingapplicationsrestartafter migration.Becausetheapplicationsrestart,theyinitializeandquerytheCPUforavailablefeatures.This impliestheapplicationsutilizeonlyinstructionsandfeaturessupportedbythedestinationhosthardware (wherethevirtualmachineispoweredon).

x86 ISA Implementation Differences

Mostadditionstothex86ISAwillcontinuetobepartofthearchitecturetoensurebackwardcompatibility. AdditionoffeaturesandinstructionsinnewerCPUsanddifferencesinimplementationofanyparticular featurebydifferentCPUvendorsaffectallapplicationsthatusethem.Portingapplicationstoutilizefeatures specifictoaCPUvendorisanarduoustask,andtherapidpaceofCPUimprovementsmakesthisharder.For example,recentIntelCPUssupporttheSSSE3instructionset.Ifapplicationsaretunedtoutilizeandbenefit fromtheseextensions,theyneedtobedesignedtoaccountforthefactthatsuchafeaturemaynotexiston CPUsmanufacturedbyothervendors.ItmayalsobethecasethatothervendorsoffersomeSSSE3like extensionsthatservethesamepurposebutcouldbenamedandimplementeddifferently. AppendixA:CPUIDandx86ImplementationDifferencesonpage 7listssomefeaturesandextensionsthat illustrateimplementationdifferencesbetweenCPUvendorsorevenwithinCPUrevisions.Ifafeatureoran instructionsetextensionexistsonasourcehostmachineandismadeavailabletovirtualmachines, VirtualCenterrequirestheseextensionstobeavailableonthedestinationhostforcompatibilityduring VMotion.

VirtualCenter Compatibility Checks for VMotion

UnlessyouareusingenhancedVMotioncompatibility,VirtualCentermustmatchupthefeaturesofthesource anddestinationhostCPUstoensurethatavirtualmachineanditsapplicationswilloperatenormallyafter migrationwithVMotion.VirtualCenterusesfeaturesreportedbytheCPUIDinstructiontodoso.VirtualCenter matchesfeaturesbetweenthesourceanddestinationhostCPUstoensurethatallfeaturesarecompatible. However,VirtualCenteralsohasasetofmasksyoucanapplytohidefeaturesreportedbytheCPUID instruction,sothatthequerycanviewfewerfeatures.ThesemasksareusedonlywhenCPUIDisexecutedas apartofprivilegedcode.Theyensurethatonlyfeaturesthatmattertotheexecutionofthevirtualmachineare compared.SeeAppendixB:CPUIdentificationMasksonpage 10formoreinformationaboutdefaultand overridemasks. Becausethesefeaturesarepresentinhardware,nonprivilegedcodequeryingCPUIDisnotaffectedbyanypart ofthevirtualizationlayer,thusthesemasksarenotappliedwhenthenonprivilegedcodemakesthequery. Therefore,applyingCPUIDoverridemasksasshownintheexampleinAppendixCcircumventsCPU compatibilitychecksforVMotion,butanapplicationrunninginavirtualmachinestillmightcrashifitdoes notfindthesetofinstructionsthatwereavailabletoitonthesourcehosthardware.Thisproblemiseffectively managedbyhardwaresupportforlivemigration,asdescribedinEnhancedVMotionCompatibilityon page 6. TheabovefactsandexamplesdemonstratewhyCPUcompatibilitychecksarenecessaryformigrationwith VMotion.Thesechecksnotonlyensurethatthemigrationsucceeds,theyalsoensuretheapplicationsrunning insideavirtualmachinecontinuetooperatenormallyonthedestinationhosthardware.ThesechecksforCPU compatibilitybeforeallowingVMotionhelpmakeESXastable,reliableplatform.

Copyright 2008 VMware, Inc. All rights reserved.

VMware VMotion and CPU Compatibility

Enhanced VMotion Compatibility

BecausenewfeaturesareconstantlyincorporatedintonewCPUs,youarelikelytoencounterincompatibilities andthereforefaceproblemswhileattemptingmigrationwithVMotion.SuchCPUincompatibilitieslimit VMotiontoacertainrangeofCPUs.VMwarehasinvestigatedtheissuesandconcludedthatthereisno softwareonlysolutiontothisproblem.Tominimizeexacerbationofthecompatibilityproblemwithtime, VMwarehasworkedwithCPUvendorstofacilitatelivemigrationofvirtualmachinesacrossdifferentCPU revisions. VMwareEnhancedVMotionCompatibility(EVC)availableinVMwareInfrastructure3beginningwith version3.5Update2facilitatesVMotionbetweendifferentCPUgenerations,takingadvantageofIntelFlex MigrationandAMDVExtendedMigrationtechnologies.Whenenabledforacluster,EVCensuresthatall CPUswithintheclusterareVMotioncompatible.CPUsstartingwithIntel45nmCore2(Penryn)andAMD SecondGenerationOpteron(revisionEorF)incorporateFlexMigrationandExtendedMigrationtechnologies, respectively. TheEVCfeatureallowsthevirtualizationlayertomaskorhidecertainfeaturesbymodifyingthesemantics oftheCPUIDinstructionandhidescertainCPUIDfeaturebits,evenfromnonprivilegedcode.Forexample, withsupportfromhardware,thevirtualizationlayermodifiesthesemanticsoftheCPUIDinstructiontomask orhidetheSSE4.1featurefromanycode(privilegedornonprivileged)tomakeCPUsdifferinginthisfeature compatibleforVMotion.SpecificsonCPUcompatibilitywiththeEnhancedVMotionCompatibilityfeature areavailableintheBasicSystemAdministrationguideforeachESXreleasestartingwithversion3.5Update2. EVCutilizeshardwaresupporttomodifythesemanticsoftheCPUIDinstructiononly.Itdoesnotdisablethe featureitself.Forexample,ifanattempttodisableSSE4.1ismadebyapplyingtheappropriatemaskstoaCPU thathasthesefeatures,thisfeaturebitindicatesSSE4.1isnotavailabletotheguestortheapplication,butthe featureandtheSSE4.1instructionsthemselves(suchasPTESEandPMULLD)arestillavailableforuse.This impliesapplicationsthatdonotusetheCPUIDinstructiontodeterminethelistofsupportedfeatures,butuse trycatchundefinedinstructions(#UD)instead,canstilldetecttheexistenceofthisfeature. Therefore,forEVCtobeuseful,applicationdevelopersmustadheretorecommendedguidelinesonfeature detection.CPUvendorsrecommendthatsoftwareprogrammersqueryCPUIDpriortousingspecial instructionsandfeaturesavailableontheirCPUs.Ifthisguidelineisfollowedbyprogrammers,EVCisa reliablemechanismforlivemigrationofx86virtualmachinesacrossvariedhardware.Thus,youcanuseEVC toenableanentireclustertousethesamesetofbasicfeatures,allowingmigrationwithVMotionacrossany twonodesinthecluster.VirtualCentercanalsosetupnewhardwareaddonstotheclusterandapplythese masks.

Conclusion
Virtualizationisstillarelativelynewtechnologicalinnovation,andtheareaoflivemigrationofvirtual machinesisstillinitsinfancy.VMwareVMotionisanextremelyusefulandcriticalfeatureindatacentersto ensurebusinesscontinuity,resourceoptimization,andahostofotherbenefits.VirtualCenterperforms numerouscompatibilitychecksbeforeallowingmigrationwithVMotion.SomestringentCPUcompatibility checksforVMotionarenecessaryforproperfunctioningofavirtualmachineaftermigration.Thoughusers canoverridethesechecksandcompleteVMotionbyapplyingappropriatemasks,thevirtualmachinesand applicationsmaynotfunctionproperlyiftheyrelyonfeaturesparticulartotheunderlyinghardware.Thex86 CPUvendorsdidnotinitiallyenvisionlivemigrationofvirtualmachinesanddesignCPUstosupportit. VMwarehasworkedwithCPUvendorstosupportsuchfeatures,takingadvantageofIntelFlexMigrationand AMDVExtendedMigrationtechnologies.WithEnhancedVMotionCompatibility,VMwareInfrastructure worksinconjunctionwithhardwaretosupportlivemigrationofvirtualmachinesinawiderrangeof environments.EffectiveuseofenhancedVMotionalsodependsonusingapplicationsoftwarethatfollows recommendedguidelinestosupportCPUfeaturedetection.

Copyright 2008 VMware, Inc. All rights reserved.

VMware VMotion and CPU Compatibility

References
VMware Infrastructure Documentation

VMwareInfrastructure3OnlineLibrary http://www.vmware.com/support/pubs/vi_pubs.html VMwareInfrastructure3BasicSystemAdministration http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_admin_guide.pdf

VMware Knowledge Base Articles

TestingandUsingNewFeaturesinCPUs http://kb.vmware.com/kb/1005763 VMotionandCPUCompatibilityFAQ http://kb.vmware.com/kb/1005764 VMotionCPUCompatibilityRequirementsforIntelProcessors http://kb.vmware.com/kb/1991 VMotionCPUCompatibilityRequirementsforAMDProcessors http://kb.vmware.com/kb/1992 VMotionCPUCompatibilityMigrationsPreventedDuetoCPUMismatchHowtoOverrideMasks http://kb.vmware.com/kb/1993

CPU Vendor Documentation

AP485IntelProcessorIdentificationandtheCPUIDInstruction http://developer.intel.com/design/processor/applnots/241618.htm AMDCPUIDSpecification http://www.amd.com/usen/assets/content_type/white_papers_and_tech_docs/25481.pdf AMDProcessorRecognitionapplicationnote http://www.amd.com/usen/assets/content_type/white_papers_and_tech_docs/20734.pdf

Other References

BootableCPUIDimage http://www.vmware.com/download/shared_utilities.html FormalRequirementsforVirtualizableThirdGenerationArchitectures,GeraldJ.PopekandRobertP. Goldberg,CommunicationsoftheACM17(7):412421(1974) http://portal.acm.org/citation.cfm?id=361073 VMwareVMotionfeatureintroduction http://www.vmware.com/products/vi/vc/vmotion_features.html

Appendix A: CPUID and x86 Implementation Differences

TounderstandCPUcompatibilityrequirementsforVMotion,considerhowapplicationsdeterminethesetof featuressupportedonanx86CPU.TheCPUIDinstructionprovidesapplicationswithastandardizedmethod forprocessorandfeatureidentification.Theavailabilityofaparticularfeatureisindicatedbythevalueofthe correspondingfeaturebitinthegeneralpurposeregister.TheCPUIDinstructionrequiresalevelnumberoran extendedlevelnumberintheEAXregisterinordertoexecute.ExecutingCPUIDwithalevelnumberinEAX returnsinformationinthegeneralpurposeregisters(suchasEAX,EBX,ECX,andEDX).Theregularlevel numbersyieldgeneralinformation,suchasCPUvendor,family,model,andstepping.Theextendedlevel numbersyieldinformationonspecificextendedfeaturessupportedbythatCPUvendorforafamily, generation,orrevision.Theextendedlevelsalsoutilizegeneralpurposeregisterstoconveyextendedfeature information.ThelevelnumberinEAXissometimesreferredtoasfunctionnumberorleaf.
Copyright 2008 VMware, Inc. All rights reserved. 7

VMware VMotion and CPU Compatibility

AregularfunctionnumberinEAXisoftheform0x000000XX,whereXXisahexadecimalvalue.Forexample, with0h(0x00000000)presentinEAX,CPUIDreturnsthefollowingonanIntelCPU: Table 1. Sample Returns from CPUID

Value in EAX 0x00000000 Returned Values EAX EBX ECX EDX Meaning LargestregularfunctionnumberinEAXsupportedbythisCPU Processorvendorstring

Theextendedlevelnumberlistsspecificfeatureinformation.AnextendedlevelnumberinEAXisoftheform 0x800000XX,whereXXisahexadecimalvalue.Forexample,todetectNXbitsupportinanInteloranAMD CPU,EAXisloadedwith0x80000001handtheCPUIDinstructionisexecuted.Thevalueofbit20intheEDX registerindicateswhethertheCPUsupportsNX(0isunsupported,1issupported).Thisguideusesthe followingnotationtoexaminethevalueofaparticularCPUIDbit:

CPUID level <feature/extended feature> <Register>:<bit>

Forinstance,forNXsupport,CPUID level 0x80000001 EDX:20shouldbeset. Thefamilyorextendedfamily,model,andsteppingbitssimplydescribeaCPU,whereasthefeatureand extendedfeaturebitsprovideinformationforusebyvariouskindsofsoftware.TheCPUIDinstructionbyitself doesnotrequireanyspecialprivilegesandthereforetheoperatingsystem(privilegedmode)oranyother application(nonprivilegedmode)canexecuteit. SpecificfeaturesandextendedfeaturesreturnedbytheCPUIDinstructionarecloselytiedtotheCPUsandare decidedbyCPUvendors.ForinformationaboutCPUDonIntelCPUs,refertoAP485IntelProcessor IdentificationandtheCPUIDInstruction,andforinformationaboutCPUIDonAMDCPUs,refertotheAMD CPUIDSpecification(seeReferencesonpage 7forlinks).

x86 Feature Bits and Implementation Differences

Thissectionlistssomeinstructionsandfeaturesthatdemonstrateimplementationdifferencesbetween vendorsorevenwithinCPUrevisionsinstructionsandfeaturesthatVirtualCenterchecksforcompatibility beforeallowingmigrationwithVMotion.ThislistisbasedonproblemscommonlyencounteredduringCPU compatibilitycheckswhenperformingVMotion.Itisnotexhaustiveandshouldbetreatedasasetof examples.ThislistisbasedonlyoncurrentCPUs.DetailsmayvaryandchangewithnewerrevisionsofCPUs.

CPUvendorstring:TheCPUvendorstringispresentinCPUIDlevel0x0EBX,ECX,andEDXregisters. ThedefaultVirtualCentermasksrequirethisstringtomatchbetweensourceanddestinationCPUsfor VMotioncompatibility.Thismatchisrequiredbecausedifferentvendorsmaychoosetointerpret, implement,andextendthex86ISAdifferently. CPUfamily,extendedfamily:TheCPUfamilyispresentinCPUIDlevel0x1EAX:811.Theextended familyfieldispresentinCPUIDlevel0x1registerEAX:2027.ACPUfamilyiscompletelydefinedbythe combinationoffamily(sometimesreferredtoasthebasefamily)andtheextendedfamily.Thedefault VirtualCentermasksrequirethisvaluetomatchbecausedifferentCPUfamilies,evenfromthesame vendor,havedifferentsupportedinstructionsetsandfeatures. RDTSCP:ThisinstructioncanbeexecutedonlybyprivilegedcodeandisspecifictoAMDCPUs.Ifthis instructionissupportedbytheCPU,itisindicatedbyCPUIDlevel0x80000001hEAX:27beingset. VirtualCentersupportsmaskingthisfeaturebitandthereforedisablingitforapplicationsrunninginside VMwarevirtualmachines.ThisallowssuccessfulvalidationofCPUcompatibilitychecksandfacilitates VMotion. StreamingSIMDextensions(SSE)and3DNow!:Thesesetsofinstructionsareextensionstothex86ISA allowingCPUstoprocesssingleinstructionmultipledata(SIMD)instructionsthatarewidelyusedin multimediaapplications.DifferentCPUvendorschosetoextendthex86instructionsetindifferentways. InteldesignedtheSSEinstructions.AMDdesignedthe3DNow!extensions.

Copyright 2008 VMware, Inc. All rights reserved.

VMware VMotion and CPU Compatibility

SSE3:SSE3isthethirdgenerationofSSEaddedtothex86instructionset.SupportforSSE3is indicatedbythevalueofCPUIDlevel0x1ECX:0.Ifthisfeatureisavailableonamachine,itcould potentiallybeusedbycertainmultimediaapplications.Ifsuchapplicationsaretocontinueoperating normallyaftermigrationwithVMotion,thisfeaturemustbeavailableinhardwareonthedestination host.VirtualCenterthereforerequiresthatthisfeaturematchbetweenthesourceanddestination hostsforVMotioncompatibility. SSSE3:SSSE3isthesetofsupplementalSSE3instructionsaddedonCPUsbasedontheIntelCore architecture.ThisfeatureiscurrentlyavailableonlyonIntelCPUs.Itisconsideredarevisionofthe SSE3instructionset.AswiththeSSE3instructions,ifthisfeatureisavailableonthesourcehost,it mustbeavailableonthedestinationhost.ExistenceofthisfeatureisconfirmedbythevalueofCPUID level0x1ECX:9. SSE4.1:Atthetimewewrotethisguide,SSE4.1instructionswereavailableonlyonIntelCPUs.This featureisthefirstsubsetofthefourthrevisionoftheSSEinstructionsandisavailableforapplications touse,ifsupportedbytheCPU.ThisfeatureisindicatedbythevalueofCPUIDlevel0x1ECX:19. SSE4.2:Atthetimewewrotethisguide,SSE4.2wasyettobeintroduced.Itwillbethesecondsubset ofthefourthgenerationofSSEinstructions.ThisfeatureisindicatedbythevalueofCPUIDlevel0x1 ECX:20. 3DNow!:The3DNow!instructionsprocessvectorfloatingpointoperations,whichaidmultimedia processing.TheseinstructionswerefirstmadeavailablebyAMD.Presenceofthisfeatureisdetected bythevalueofCPUIDlevel0x80000001EDX:31. 3DNow!Extensions:Extensionstothe3DNow!instructionswereintroducedbyAMD.Thisfeature isdetectedbythevalueofCPUIDlevel0x80000001EDX:30.

NoeXecute(NX)oreXecuteDisable(XD):Thisfeatureallowsprivilegedcode(kernellevel)tomark certainsectionsofmemoryascodeandothersasdata.ThismarkingpreventstheCPUfromexecuting dataregionsofmemory,onemethodusedbymalicioussoftwaretotakeunauthorizedcontrolofa machine.Suchusurpationisreferredtoasabufferoverflowattack.Presenceofthisfeatureisindicated byCPUIDlevel0x80000001EDX:20.Becausethisfeaturemaybeusedonlybyoperatingsystems (privilegedcode),VirtualCenterallowsuserstodisablethisfeatureandhideitfromtheguestoperating systemevenifitisavailableonhardware. Longmodesupport:CPUssupportinglongmodeallow64bitinstructions(64bitmode)aswellas32bit instructions(compatibilitymode)toexecute.ThissupportisindicatedbyCPUIDlevel0x80000001 EDX:29.IfaCPUsupportslongmode,capableapplicationsexecute64bitinstructions.VMwareproducts requirethisfeaturetobepresentwhenyoucreatea64bitvirtualmachinescapableofrunninga64bit guestoperatingsystem.VirtualCenterattemptstomatchthisfeaturebitwhenperformingcompatibility checksbeforeallowingmigrationwithVMotion.Thecompatibilitychecksforlongmodesupportare basedontheconfiguredoperatingsystemandsettingsforthevirtualmachine.VMwareproductssupport longmodeonlyifhardwarevirtualizationsupport(VTonIntelCPUsandAMDVonAMDCPUs)isalso availableonthehost. CMPXCHGinstructions:The8byteand16bytecompareandexchangeinstructionsarenewadditions availableoncertainnewerCPUs.TheirpresenceisindicatedbythevalueofCPUIDlevel0x1EDX:8(for CMPXCHG8B)andCPUIDlevel0x1ECX:13forCMPXCHG16B.Theseinstructionsmaybeusedbyany application,ifavailable.Therefore,VirtualCentermustmatchthisfeaturebetweensourceanddestination hostsbeforeallowingmigrationwithVMotion. FFXSR:TheFXSAVEandFXRSTORinstructionssaveandrestorex87,MMX,andXMMregistersusedin floatingpointoperations.CurrentAMDCPUsalsohaveoptimizedversionsoftheseinstructions:fast FXSAVEandfastFXRSTOR(FFXSR),indicatedbyCPUIDlevel0x80000001EDX:25,whichdonotsave andrestoreXMMregistersinthe64bitmodewhenexecutedbyprivilegedcode.Thisfeaturebitis unused(reserved)onIntelCPUs.BecauseapplicationscanquerytheCPUforthesefeaturesandusethem, VirtualCenterrequiresthesefeaturestomatchbetweensourceanddestinationwhenitperformsVMotion compatibilitychecks.

Copyright 2008 VMware, Inc. All rights reserved.

VMware VMotion and CPU Compatibility

Prefetchinstructions:AMDCPUssupportPREFETCHandPREFETCHWinstructionstoloaddatainto theCPUsL1cache.Applicationsmayusetheseinstructions,ifavailable.Theseinstructionsareavailable onlyifCPUIDlevel0x80000001ECX:8isset.IntelCPUsdonotcurrentlysupportthisinstruction,andthis bitisreserved. MONITOR/MWAIT:IntelCPUsintroducedtheMONITORandMWAITinstructionsalongwiththeSSE3 extensions.Theyareprimarilyusedforthreadsynchronizationandthereforehavegenericusage. AvailabilityoftheseinstructionsisindicatedbyCPUIDlevel0x1ECX:3.RecentAMDCPUsnowalso supporttheseinstructions. Modelspecificregisters(MSR):Processorimplementationsprovidecertaincontrolregistersthatallow applicationstousefeaturesspecifictoaprocessor.SinceMSRsarespecifictoaprocessormodeland implementation,theymaynotbeavailableonothermodelsofthesameprocessorfamilyorothermodels fromthesamevendor.

Appendix B: CPU Identification Masks

VirtualCenterqueriesthehostCPUforthesetofavailablefeatures.Itthenappliesadefaultmasktothisvalue. Thedefaultmasksarecreatedbasedontheguestoperatingsystemanddeterminethesetoffeaturesavailable totheguestoperatingsystem.PrivilegedcodeexecutingCPUIDwillbeabletoseeonlythefeaturesthatare notmasked.Inordertogivesystemadministratorstheflexibilitytocontrolthefeaturestheirvirtualmachine andtheguestoperatingsystemobserve,VirtualCenterallowsyoutooverrideitsdefaultmaskswithoverride masks.YoucanmaskoutanyfeaturebitatanyCPUIDlevel.VirtualCentercombinesitsdefaultmaskwithyour overridemaskandappliesthecombinedmasktothevaluereturnedbyCPUIDtodisplaythefinalsetof featuresavailabletothevirtualmachine.NotethatintheabsenceofsupportforenhancedVMotion, nonprivilegedcodeexecutingCPUIDrunsdirectlyonhardware,overridinganymasksdefinedin VirtualCenter.Table2showsthenotationsusedintheoverrideanddefaultmasks. Table 2. Notations Used in CDP Feature Masks
Mask X T F 1 0 Explanation Usevaluepresentinthedefaultmask;donotoverride.Usedinoverridemasksonly. Dontcare.Valueunusedbyguestsoftware.NotcheckedforVMotion. Bit(feature)mustbesetinhardwareandisrequiredbysoftwareinthevirtualmachine.Maybeused indefaultoroverridemasks.Bitmustmatch(set)forVMwareVMotion. Bit(feature)mustnotbesetinhardware,hencefeaturemustnotbeenabledforsoftwarerunningin virtualmachine.Maybeusedindefaultoroverridemasks.Bitmustmatch(clear)forVMotion. SetthisbitifCPUIDisexecutedbyguestsoftware.Maybeusedindefaultoroverridemasks.Not checkedforVMotion. ClearthisbitifCPUIDisexecutedbysoftwareinthevirtualmachine.Thisisusedtohidefeature fromsoftwareinthevirtualmachine.Maybeusedindefaultoroverridemasks.Notcheckedfor VMotion. AllowsoftwareinthevirtualmachinetoseeactualfeaturevaluereturnedbyCPUIDonhardware. Likelytobeusedindefaultmasksbutmaybeusedinoverridemasksaswell.Mustmatchfor VMotion. Hidefeaturefromsoftwareinthevirtualmachinebyclearingthisbit.Likelytobeusedondefault masksbutmaybeusedinoverridemasksaswell.MustmatchforVMotion.

Default Masks
StartingwithVirtualCenter2,defaultmasksaredefinedforeachguestoperatingsystem.ForVMotionchecks, eventhoughtheRentriesarehiddenfromthesoftwareinthevirtualmachine,theyarerequiredtomatch, becauseapplicationlevelsoftwaremaybeutilizingthefeaturesbyqueryingCPUIDdirectlyonhardware (withoutenhancedVMotionsupport).TheHvaluesarerequiredtomatchforVMotion,becausesoftwarein thevirtualmachineutilizesthesefeaturesinhardware.TheTvaluesindicatetheguestoperatingsystem cannotfunctionwithoutthesefeatures,thereforethefeaturesmustbesetinhardware(forexample,thelong

Copyright 2008 VMware, Inc. All rights reserved.

10

VMware VMotion and CPU Compatibility

modebitCPUIDlevel0x80000001hEDX:29for64bitguestoperatingsystems).Thisimpliesthatthevirtual machinerequiresthisfeatureevenonthenewdestinationhosthardware.Therefore,thisvalueisrequiredfor VMotion.Similarly,theFvaluesindicateaguestcannotfunctionifthisfeatureispresent,thereforethefeature mustnotbepresentonthedestinationhardwarewhenperformingVMotion. ThedefaultmasksforavirtualmachinearestoredinanXMLfilenamed vmconfigoption-esx-<version>.xml,where<version>isthethreedigitversionofESXforexample, 3.0.0.ThisXMLfileislocatedinthe/etc/vmware/hostd/envdirectory.

Override Masks
AnoverridemaskreplacestheexistingdefaultmaskvalueforaparticularCPUIDbit.Overridemaskshave animplicitemptyvalue(),whichmeansthereisnooverridevalueforthatbit.Overridemasksarepervirtual machineandarestoredinthevirtualmachinesconfigurationfile(.vmx).TooverrideaparticularCPUIDbit, youmustenteroverridevaluesfortheentireregisterforthecorrespondingCPUIDlevel.Forinstance,to disabletheSSSE3featureforsoftwareinthevirtualmachine,clearCPUIDlevel0x1hECX:9.Theoverride masktoaccomplishthisis:
Level 0x1h ECX: ---- ---- ---- ---- ---- --0- ---- ----

ThisoverridemaskdisablesonlytheSSSE3bitinCPUIDlevel0x1hECX:9whilepreservingallotherdefault valuesthatVirtualCentersetsforthiscombinationofregisterandlevel. NOTEYoucancreateoverridesonlywhenthevirtualmachineispoweredoff,andtheoverridemasktakes effectonlywhenthemachineisstarted.

Final Mask
VirtualCentercalculatesthefinalmaskbyoverridingvaluesinthedefaultmaskusingthevaluesinthe overridemask,ifthereisone.ThisfinalmaskthenbecomespartoftheVirtualCenterconfigurationandis appliedtotheresultsoftheCPUIDinstruction. ContinuingwiththeSSSE3exampleabove,assumethedefaultmaskforCPUIDlevel0x1hECXis:
Level 0x1h ECX: RRRR RRRR RRRR RRR0 00xR R0H0 000H 0RRH

Youthenapplythefollowingoverridemask:
Level 0x1h ECX: ---- ---- ---- ---- ---- --0- ---- ----

ThisgivesyouafinalmaskforCPUIDLevel0x1hregisterECXof:
Level 0x1h ECX: RRRR RRRR RRRR RRR0 00xR R000 000H 0RRH

TotestVMotioncompatibility,VirtualCenterappliesthismasktotheresultsoftheCPUIDinstructiononthe sourceanddestinationhardwareandcomparesallregistersatvariouslevels.CPUcompatibilitychecksmatch onlyifallbitsmatchafterthemaskisapplied.

Appendix C: Relaxing VMware VMotion CPU Compatibility Constraints

IfyouneedtouseVMotionbetweenhostswithdifferentCPUsinenvironmentsthatdonotsupportEnhanced VMotionCompatibilityclusters,youmayfinditusefultofollowtheguidelinesinthisappendix.Ifyour environmentallowsyoutouseEVCclusters,youdonotneedtousethetechniquesdescribedinthisappendix. VirtualCentercheckstoensurethatthesourceandtargetCPUsarecompatiblebeforeallowingmigrationwith VMotion.Itdoessobyensuringfeatureandextendedfeatureflagsmatch.Itispossibletocircumventthese checksbyapplyingoverridemasks.Theseoverridemasksdisablecertainfeaturesforguestoperatingsystems onthehostanddestinationCPUsandthereforeallowVirtualCentertocompleteVMotioncheckssuccessfully.

Copyright 2008 VMware, Inc. All rights reserved.

11

VMware VMotion and CPU Compatibility

AssumingtheguestoperatingsystemusestheCPUIDinstructiontodetectfeaturessupportedbyaCPU,the featuresmaskedoutarenotavailabletotheguestoperatingsystemsoftware.Thevirtualmachinemustbe restartedsoapplicationsoftwareinthevirtualmachinecanrunfeaturedetection(viatheCPUIDinstruction) anduseonlythesetoffeaturesprescribedbytheoverridemasks. NOTETheserelaxationsarenotsupported. ThefollowingexampleshowshowtodisableSSE4.1instructionsonanIntelCPUsothatavirtualmachinecan bemigratedtoahostwithaCPUthatlacksthisfeature. NOTEThisfeatureisnotsupportedbecauseapplicationsoftwarethatreliesonrunningtheCPUIDinstruction forfeaturedetectiondirectlyonhardwarecandetectthisfeatureandmightrelyonit.Inanenvironmentthat supportsEVC,youdonotneedtobeconcernedwiththemaskingdescribedinthisappendix. TheexampleusesthefollowinghostsandCPUs:

host1.vmware.com:twocoreIntelPenrynCPU host2.vmware.com:fourcoreIntelKentsfieldCPU

AssumeallcompatibilitychecksforVMotionotherthantheCPUcheckcompletesuccessfully.While performingVMotionfromhosthost1.vmware.comtohosthost2.vmware.comonVMwareInfrastructure3, thefollowingmessageappears:

ThismessageindicatesthattheCPUcompatibilitychecksperformedpriortomigrationfailed.TheCPUin host1.vmware.compossessesSSE4.1extensions,buttheCPUinhost2.vmware.comdoesnot.Youcan determinetheseCPUfeaturesbyexecutingtheCPUIDinstructiononthesehosts.Youcanobtaintheexact valueofthemismatchedbitsbyrunningtheCPUinformationutility(cpuid.iso)providedwithVMware Infrastructure3.YoucanuncompresstheISOimagefile(\images\cpuid.iso.gz)anduseittocreatea bootableCDROMthatprovidesCPUinformationaboutagivenhostbeforeyouinstallanoperatingsystem orESX.Runningthisutilityonthesourcehost(host1.vmware.com)givesthevalueofregisterswhenCPUIDis executedatvariouslevels.Youcanuseittoexaminethecauseofthemismatch. RuntheCPUIDimageonhost1.vmware.com(IntelPenrynCPU).Theoutputincludesthefollowingrelevant information:
Family: 06 Model: 17 Stepping: 4 ID1ECX ID1EDX ID81ECX ID81EDX 0x0008e3bd 0xbfebfbff 0x00000001 0x20100800

RuntheCPUIDimageonhost2.vmware.com(IntelKentsfieldCPU).Theoutputincludesthefollowing relevantinformation:
Family: 06 Model: 0f Stepping: 7 ID1ECX ID1EDX ID81ECX ID81EDX 0x0000e3bd 0xbfebfbff 0x00000001 0x20100800

Copyright 2008 VMware, Inc. All rights reserved.

12

VMware VMotion and CPU Compatibility

ThecompatibilityerrormessageindicatesthatCPUIDlevel1ECXbitsdonotmatch.(VirtualCenterchecksfor compatiblefamiliesonlyandignoresmodelandsteppinginformation.)Analternativemethodofdetermining featurespresentonahostCPUisusingthemanagedobjectbrowser(MOB)presentinESX.Youcanaccess CPUIDbitsforalllevelsusingtheMOB.Thevaluesofregistersareindicatedinbinaryformat.Thelevel numbersarereportedinintegerformat. LookmorecloselyatID1ECXonthetwomachines. CPUIDlevel0x1hECX(host1.vmware.com)0x0008e3bd

31 0 0 0 0 27 0 0 0 0 23 0 0 0 0 19 1 0 0 0 15 1 1 1 0 11 0 0 1 1 7 1 0 1 1 3 1 1 0 0 1

CPUIDlevel0x1hECX(host2.vmware.com)0x0000e3bd
31 0 0 0 0 27 0 0 0 0 23 0 0 0 0 19 0 0 0 0 15 1 1 1 0 11 0 0 1 1 7 1 0 1 1 3 1 1 0 0 1

Theregistersdifferinthevalueofbit19,whichindicatessupportfortheSSE4.1setofinstructions.Youcan alsoobtainthebitsdifferingbetweenthetworegistersbyusingabitwiseXORfunction. LookatthedefaultmasksinVirtualCenter2.0.TheyaresettorequireamatchforthevalueofCPUIDlevel1 ECX:19betweensourceanddestination.Tocircumventthischeck,youmust: 1 2 3 Applyappropriatemaskstohidethisfeaturefromthevirtualmachine. Establishthatnoapplicationrequiresthefeatureyoumasked. Rebootthevirtualmachine.

Theoverridemaskforeachvirtualmachineisstoredinitsconfiguration(.vmx)file. TomaskCPUIDlevel0x1hECX:19,applythefollowingmaskinVirtualCenter.
31 27 23 19 0 15 11 7 3 0

Themask---- ---- ---- 0--- ---- ---- ---- ----appliedtolevel1ECXhidestheSSE4.1feature fromsoftwareinthevirtualmachine.

Copyright 2008 VMware, Inc. All rights reserved.

13

VMware VMotion and CPU Compatibility

Forthenewmaskstotakeeffect,youmustrestartthevirtualmachine.Whenthenewmaskisineffect, migrationfromhost1.vmware.comtohost2.vmware.comshouldsucceed.

SeetheVMwareknowledgebasearticleVMotionCPUCompatibilityMigrationsPreventedDuetoCPU MismatchHowtoOverrideMasksformoreexamplesshowinghowotherfeaturescanbemasked.See Referencesonpage 7foralink.

If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com VMware, Inc. 3401 Hillview Ave., Palo Alto, CA 94304 www.vmware.com Copyright 2008 VMware, Inc. All rights reserved. Protected by one or more of U.S. Patent Nos. 6,397,242, 6,496,847, 6,704,925, 6,711,672, 6,725,289, 6,735,601, 6,785,886, 6,789,156, 6,795,966, 6,880,022, 6,944,699, 6,961,806, 6,961,941, 7,069,413, 7,082,598, 7,089,377, 7,111,086, 7,111,145, 7,117,481, 7,149, 843, 7,155,558, 7,222,221, 7,260,815, 7,260,820, 7,269,683, 7,275,136, 7,277,998, 7,277,999, 7,278,030, 7,281,102, 7,290,253, and 7,356,679; patents pending. VMware, the VMware boxes logo and design, Virtual SMP and VMotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. Revision 20080609 Item: EN-000039-00

14