You are on page 1of 19

sponsored by

Greg Shields
Resolving VMware
vSpheres Six Biggest
Performance Issues
The Essentials Series
TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

i
IntroductiontoRealtimePublishers
by Don Jones, Series Editor

Forseveralyearsnow,Realtimehasproduceddozensanddozensofhighqualitybooks
thatjusthappentobedeliveredinelectronicformatatnocosttoyou,thereader.Weve
madethisuniquepublishingmodelworkthroughthegeneroussupportandcooperationof
oursponsors,whoagreetobeareachbooksproductionexpensesforthebenefitofour
readers.
Althoughwevealwaysofferedourpublicationstoyouforfree,dontthinkforamoment
thatqualityisanythinglessthanourtoppriority.Myjobistomakesurethatourbooksare
asgoodasandinmostcasesbetterthananyprintedbookthatwouldcostyou$40or
more.Ourelectronicpublishingmodeloffersseveraladvantagesoverprintedbooks:You
receivechaptersliterallyasfastasourauthorsproducethem(hencetherealtimeaspect
ofourmodel),andwecanupdatechapterstoreflectthelatestchangesintechnology.
Iwanttopointoutthatourbooksarebynomeanspaidadvertisementsorwhitepapers.
Wereanindependentpublishingcompany,andanimportantaspectofmyjobistomake
surethatourauthorsarefreetovoicetheirexpertiseandopinionswithoutreservationor
restriction.Wemaintaincompleteeditorialcontrolofourpublications,andImproudthat
weveproducedsomanyqualitybooksoverthepastyears.
Iwanttoextendaninvitationtovisitusathttp://nexus.realtimepublishers.com,especially
ifyouvereceivedthispublicationfromafriendorcolleague.Wehaveawidevarietyof
additionalbooksonarangeoftopics,andyouresuretofindsomethingthatsofinterestto
youanditwontcostyouathing.WehopeyoullcontinuetocometoRealtimeforyour
farintothefuture. educationalneeds
enjoy. Untilthen,
DonJones

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

ii

IntroductiontoRealtimePublishers.................................................................................................................i
Ar

ticle1:WhatSixvSphereIssuesMostImpactVMPerformance?...................................................1
MonitoringBehaviorstoFindPerformanceIssues..............................................................................1
vS pheresSixBiggestPerformanceIssues................................................................................................3
PerformanceIssue#1:CPUUtilization.................................................................................................3
PerformanceIssue#2:MemoryUtilization........................................................................................3
PerformanceIssue#3:StorageUtilizationandDiskI/O..............................................................4
PerformanceIssue#4:ApplicationIssues..........................................................................................4
PerformanceIssue#5:HypervisorProblems....................................................................................4
PerformanceIssue#6:OverheadUtilizationandScalability......................................................5
ActionableIntelligenceIsMorethanMonitoring..................................................................................5
Ar ticle2:WhatTenCountersQuantifythoseBehaviors?......................................................................6
Te nBehaviorQuantifyingCounters...........................................................................................................7
Counter#1:CPU.ready.summation........................................................................................................7
Counter#2:CPU.usagemhz.average.......................................................................................................7
Counter#3:mem.active.average.............................................................................................................7
Counter#4:mem.consumed.average....................................................................................................7
Counter#5:mem.swapped.average.......................................................................................................7
Counter#6:mem.vmmemctl.average....................................................................................................8
Counter#7:disk.busResets.summation...............................................................................................8
Counter#8:disk.totalLatency.average.................................................................................................8
Counter#9:disk.usage.average...............................................................................................................8
Counter#10:net.usage.average...............................................................................................................8
CountersArentEverything.............................................................................................................................9
Ar ticle3:WhatTwelvePracticesConvertRawDataintoResolutions?........................................10
Th eDailyPractices...........................................................................................................................................11
Practice#1:SNMPMonitoring...............................................................................................................11
TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

iii
Practice#2:ResourceUtilizationMonitoring.................................................................................11

Practice#3:AlertMonitoring.................................................................................................................12
Th eMonthlyPractices....................................................................................................................................12
Practice#4:DiskSpaceUtilization......................................................................................................12
Practice#5:ApplicationRestarts.........................................................................................................12
Practice#6:ServerReboots....................................................................................................................12
Practice#7:OfflineMaintenance..........................................................................................................13
Practice#8:OverallHealth......................................................................................................................13
Th eYearlyPractices........................................................................................................................................13
Practice#9:BudgetingforReplacementorAugmentation......................................................13
Practice#10:UserFeedback..................................................................................................................13
Practice#11:LargeScaleChangeInitiatives..................................................................................14
Practice#12:TheOngoingPractice.........................................................................................................14

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

iv
Copyright Statement
2011 Realtime Publishers. All rights reserved. This site contains materials that have
been created, developed, or commissioned by, and published with the permission of,
Realtime Publishers (the Materials) and this site and any such Materials are protected
by international copyright and trademark laws.
THE MATERIALS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE,
TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice
and do not represent a commitment on the part of Realtime Publishers its web site
sponsors. In no event shall Realtime Publishers or its web site sponsors be held liable for
technical or editorial errors or omissions contained in the Materials, including without
limitation, for any direct, indirect, incidental, special, exemplary or consequential
damages whatsoever resulting from the use of any information contained in the Materials.
The Materials (including but not limited to the text, images, audio, and/or video) may not
be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any
way, in whole or in part, except that one copy may be downloaded for your personal, non-
commercial use on a single computer. In connection with such use, you may not modify
or obscure any copyright or other proprietary notice.
The Materials may contain trademarks, services marks and logos that are the property of
third parties. You are not permitted to use these trademarks, services marks or logos
without prior written consent of such third parties.
Realtime Publishers and the Realtime Publishers logo are registered in the US Patent &
Trademark Office. All other product or service names are the property of their respective
owners.
If you have any questions about these terms, or if you would like information about
licensing materials from Realtime Publishers, please contact us via e-mail at
info@realtimepublishers.com.

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

[EditorsNote:ThisbookwasdownloadedfromRealtimeNexusTheDigitalLibraryfor
ITProfessionals.AllleadingtechnologybooksfromRealtimePublisherscanbefoundat
ttp://nexus.realtimepublishers.com h .]

Article1:WhatSixvSphereIssuesMost
ImpactVMPerformance?
VMwarevSphereisacomplicatedbeast.vSphereisfullofmovingparts,deepintegrations,
andelementsthatsometimesdoandsometimesdontimpacteachother,whichmakes
keepingitrunningwithbestperformanceaneverendingexerciseincapacitymanagement.
Assistingyouwiththetaskisalonglistofcounters.Thesecounters,measuredbyvCenter
ServerandexposedinthevCenterClient,helpyouquantifythebehaviorsexperiencedby
yourvirtualmachines.
Thehardpart,however,isinrecognizinghowthosecounterstranslateintoactual
performanceissues.Withhundredsatyourdisposal,whichcounterspresentactually
usefulinformation?Whichonesunnecessarilymuddythewatersbydeliveringtoomuch?
Andwhichhelpyoutrulyunderstandthecapacityandperformanceissuesthataretruly
impactful.ThisEssentialsSeriesexposestencountersatthecenterofthoseactivities.
Butcountersthemselvesarenttheonlyfactorinsuccessfulcapacitymanagement.They
are,infact,onlyonepieceofamuchgreaterpuzzle.Alsocriticaltomaintainingyour
environmentareeffectiveprocesses;daily,monthly,andyearlyactivitiesthatpreserve
vCentersoverarchinghealth.Thisguidedeliversasetofimportantprocessesyoullwant
toimplement.
Mostimportant,anunderstandingofVMwaresbiggestperformanceissuescanonlybe
achievedwhenyoucantranslatetherawdataitsuppliesintoactionableintelligence.What
doesitreallymeanwhenmem.active.averagereads3822today?Shouldyoudosomething?
Ifso,what?Thisguideconcludeswithalookatexactlythatactionableintelligenceevery
administratorreallywants,showingyouwhyyoufeeloverloadedwithdataandhowto
gleanrealresolutionsfromrawdata.Heresahint:Theanswerisntcompletelyinthe
numbers.Itsalsointhehumanprocessesyoumustimplementtocontrolandstabilizeyour
vCenterenvironment.
MonitoringBehaviorstoFindPerformanceIssues
Beforewecandelveintothetechnicalinformation,itisnecessarytorecognizethebiggest
issuesvCenterenvironmentsface.WhenyoustepintoyourofficeonaMondaymorningto
findadozenworkordersandvoicemails,itsyourjobtofigureoutwhytheserverisslow
today.

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

Thattroubleshootingprocesshasfortoolongbeenasubjectiveactivity.Partofthereason
forourgutfeelingapproachtoperformanceandcapacitymanagementhascenteredonthe
serverslackofinstrumentation.Inthephysicalworld,instrumentingaserverrequired
extraeffort,additionalandsometimesexpensivesoftware,andanadvanceddegreein
statisticsanddataanalysis.Today,evenasvirtualizationcomplicatestheseactivities
throughitscollocationofvirtualmachines,italsoeasesperformanceandcapacity
managementbyautomaticallyinstrumentingvirtualmachineactivitieswitharangeof
behavioralmonitors.
BehavioralmonitorsforCPUutilization,memory,disk,andstorage,atthehostlevelaswell
asinsidevirtualmachinescreateanendlesssupplyofbeautifullycomplexgraphsthat
highlightvSpheresrawexperiences.YetalthoughvSpheresgraphsFigure1providesan
exampleareacademicallyinteresting,canyouasahumandivineactionableinformation
fromitsjaggedandoverlappinglines?

Figure1:RawdatafromvSpheresperformancetab.
Theresanargumentthatyoucant.Withevenjustafewdatapointsbeingcollectedper
server,monitoring100virtualmachinesrequiresanalyzingoveramillionofthemperday.
Asanenvironmentscalesup,nohumanunaidedcanusethosegraphstoanswerthe
uestion:WhatshouldIdo? q

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

3
vSpheresSixBiggestPerformanceIssues
ThatswhythislookatvSpheresbiggestperformanceissuesmuststartwiththatreality
check.Notwithstandingitssizeorcomplexity,almosteveryvSphereenvironmentsuffers
fromasimilarseriesofissues.Theyexistnotbecausetheresalackofvisibilityintokey
capacityandperformancemetrics.Farfromit.Rather,theseissuescausepainbecause,
unaided,peoplecantturnthatdataintoactionableintelligence.
Keepthattermfreshinyourmindasyoureadon.Itstheintelligenceyoucanperform
actionuponthatsuppliesyouwiththeanswertothequestion:WhatshouldIdo?
Sowhatarethesixbiggestperformanceissuesyouandyourvirtualadministratorpeers
areexperiencing?Takealookthroughthefollowingissuestoseeiftheyrelatetothe
behaviorsyoususpectaresappingperformanceoutofyourvSphereenvironment.
PerformanceIssue#1:CPUUtilization
Thefirstoftheseissuesislikelytheoneyouremostfamiliarwith.vSpherevirtual
machinesdontworkverywellwhentheyrunlowonCPUcapacity.Thelinkagebetween
CPUsupplyandvirtualmachineperformanceiswellknownandseeminglyeasytotrack
down.Formostofus,seeingahostwhoseCPUsareconstantlypeggedorwhereCPUReady
timeishighimmediatelypointsustoanoverusecondition.
YetwhatmanydontrealizeisthatCPUoversubscriptionisasmuchacapacityissueasone
ofperformance.WhenalleightofaneightwayhostsCPUsrunconsistentlyat90%,that
hostdoesnothavethecapacitytosupportitsworkloads.Neededisadditionalhardwareto
offloadvirtualmachinesandrebalancetheload.
MostenvironmentsleanonvSpheresbuiltinDynamicResourceScheduler(DRS)to
automatetherebalancingontheirbehalf.Yetdoingsowithoutappropriatemonitoringwill
quicklycreateadistributedcapacityshortfallastheenvironmentgrows.Inessence,youll
loadbalanceyourselftoaclusterwidecapacityshortfallasyoukeepaddingvirtual
machines.VMwarevSpheresnumberoneperformanceissuehappenswhenhighly
automatedenvironmentscantplanforthatsituationbeforeithappens.
PerformanceIssue#2:MemoryUtilization
Issuenumbertwoisslightlymoredifficulttospot.Thereasonforthisdifficultyliesin
VMwaresmuchtoutedmemoryovercommitmentcapabilities.Withthem,virtual
machinesonahostcanbeassignedmorememorythanisphysicallyavailableonthathost.
AlthoughgreatforconsolidationandanabsoluteboontoDRSrebalancingactivities,over
commitmentisinrealityasituationyoushouldavoidwheneverpossible.Avoidingover
commitmentmeansnotforcingvSpheretoengageinitsextramemorymanagement
activitiesthatfacilitatethesharingofmemory.Thoseactivitiesconsumeunnecessary
resourcesthatwillimpactperformance.

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

Amuchbetterapproachistorightsizeassignedmemorytoactualvirtualmachine
requirements,particularlytakingintoaccountthememorythatsneededatpeakusage
times.Thisisobviouslyadifficulttaskwithouteffectivemonitoringthatprovides
actionableintelligencetotellyouhowtoadjustyourconfiguration.
PerformanceIssue#3:StorageUtilizationandDiskI/O
Agrowingsourceofconcerninthevirtualworldistheimpactofstorageonoverallvirtual
machineperformance.IOPSandtotalstoragethroughputaremeasurementsyouve
probablybeenhearingaboutveryrecently;yet,itsbearingonperformanceisonlynow
beingrecognizedasextremelyimportant.
TodaysvSpherefailsinthisregard.Itscountersdonotdeliverinformationaboutstorage
performanceinaneasytounderstandformat.UnlessyoureskilledinreadingvSphere
countersandrelatingthemtoyourenvironmentcomposition,yourenotlikelytoglean
informationfromtheirdatathatcantellyouwhattodo.
Storageutilizationisabetterunderstoodtopic,althoughnotonethatisnecessarilywell
alertedinsidevSpheresinterface.Complicatingthesituationisthevastarrayofstorage
optionsavailableinsidethetypicaldatacenter.vSpherealonedoesnotdoagoodjobof
helpingyouunderstandwhichofyourarrayofstorageoptionsmakesbestsenseforvirtual
machinelocation.Factoringincost,capacity,andevenIOPSintothiscalculationrequires
extraeffortoroutsidesupport.
PerformanceIssue#4:ApplicationIssues
CPU,memory,andstorageareoftentreatedasaggregatecountersoncevirtualmachines
arevirtualized.Knowingthatyouhavesomenumberofmegahertzofprocessingcapacityis
usefulforplanningwhatfuturedatemoresupplymustbepurchased.But,asisoftensaid,
virtualizationrewardssmartadministration,meaningthatasmarterworkload
configurationresultsinneedingtobuylesshardware.
vCentersinstrumentationintoavirtualmachinesbehaviorsislimitedtotheaggregate
behaviorsexposedatthehost.However,sometimesadrainoncapacityhaslittletodowith
thevirtualmachineitselfandmoretodowiththeworkloadontop.
Untamedapplications,particularlywhencombinedwithoverallocatedresources,canhave
adeleteriouseffectontotalcapacity.Considerthepoorlytuneddatabaseormiddleware
applicationthatconsumeseveryresourceavailable.Leveragingsmartmonitoringtools
thatinformwhenthesesituationsoccurcanhelpthevirtualadministratorfinelytunethe
applicationratherthanresortingtoother,morebruteforceapproaches.
PerformanceIssue#5:HypervisorProblems
Althoughtodayshypervisorsaremostlybombproof,theyarentcompletelydevoidof
issues.Manyofthoseissuesareinfactcreatedbywellmeaningadministrators.A
hypervisorthathasbeenaskedtodotoomuchwillsufferanexcessivelossofperformance.
Onewhosevirtualmachinecommunicationchannels(visvistheVMwareToolsamong
others)aresevered,disabled,ornonpresentcausesextraunnecessarywork.
TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

Thekindsofactionableintelligenceanadministratordesireshelpsidentifywhen
hypervisorsarentconfiguredappropriately.Thatsameinformationalertsthemwhentoo
manyCPUs,memory,orotherresourcesareassignedtovirtualmachines.Italsosendsup
redflagswhencommunicationpathsarenotavailableoroptimized.
PerformanceIssue#6:OverheadUtilizationandScalability
Lastlyandmostimportantlyisgettingonesarmsaroundtheactivitiesoftheentiredata
center.TodayssweetspotforvSphereclustersissaidtorangebetween16and24hosts.
Manydatacentersrequirefarmorehardwarethanthat.Optimizingperformanceacross
multiplehostsandentireclustersisatasknotwellvisualizedinsidevSpheres
Performancetabsalone.
Whenworkinginsuchadistributedenvironment,youwillneedtohavevisibilityoutside
theboundaryoftheindividualclustertobestoptimizeyourresources.Seeingperformance
andcapacityinformationthatspanstheirboundarieshelpsyoudeterminewhenvirtual
machinesarebestrebalancedacrossclusters(orevendatacenters).
Scalabilityisntonlyaclusterspecificcalculation.Thereisacertainquantityofextra
resourcesthatarerequiredtomanagetheassignedresourcesofeachvirtualmachine.
Theseextraresourcesrepresentadrainonthosethatcanbeassignedelsewhere.Asa
result,oversizedvirtualmachinestendtoconsumeagreaterlevelofoverheadthanthose
thatareproperlyconfigured.Inshort,oversizingvirtualmachinespaysakindofdoubletax
onavailableresources.
ActionableIntelligenceIsMorethanMonitoring
SolvingthesesixbigissuesrequiresasuperioranalysisofthedataVMwarevCenter
exposes.Itrequirestheassistivesupportofexternalservicesthatwatchthedataforyou,
crunchitsnumbersonyourbehalf,anddelivertoyouactionableintelligenceinsteadofjust
rawdata.
Trulyappreciatingthisstatement,however,requiresfirstalookatthecounters
themselves.OnlybyseeingtheintrinsiccomplexitywithinjusttenofVMwaresmost
importantcounterscanyoutrulyrecognizethatyoullneedhelptoanswerthequestion
hatshouldIdo? W


TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

6
Article2:WhatTenCountersQuantify
thoseBehaviors?
Avirtualenvironmentisbynatureaninvisibleenvironment.Yousimplycantcrackthe
caseonavSpherehostandexpecttoseethebehaviorsgoingoninside.Thatswhyits
countersaresoimportant.Theyrepresentyouronlywaytounderstandthebehaviorsand
quantifypotentialresolutions.
Butcountersbythemselvesareveryscarythings.Acounterisbydefinitionjustanumber.
Puttogetherenoughofthosenumbers,andyoullcreateagraphnotunlikeFigure1inthe
previousarticle.Diviningmeaningfromthepointsonthatgraph,however,isanotherthing
entirely.Studyingchartsandgraphsisanactivitythatcanconsumeeverypartofyour
workday.Withthosegraphsconstantlyevolvingwithavirtualenvironmentsbehaviors,
justkeepingupisachallengeallitsown.
Yetmonitoringvirtualmachineperformanceisavirtualenvironmentsmostimportant
activity.Thatswhythisseriespreviousarticlesuggestedthatanunaidedpersoncannever
effectivelyconvertrawdataintoactionableintelligence.Oh,yes,inatinyenvironmentwith
justafewinterdependencies,youprobablymight.ButmostofourVMwarevSpheredata
centersarelargeanddistributed.Findingthesourceofaperformanceissueisnteasywhen
yourestartingatitsunendingintegers.
Relatingthosenumberstotheactionsyoushouldtakeiswhatyoureallywant.Figure1
showsthatlineofthinkinginrelationtothefirstarticlesnotionofactionableintelligence.
Init,youcanseehowtherawmonitoringdatafromavSpheresystemcanflowthrough
somekindofbuiltbysomebodyelsemathematicalmodelthatconvertsrawdatainto
suggestedactions.

Figure1:Howactionableintelligenceisgenerated.
Whatdoesthatrawdatalooklike?Whattoptencountersmightyouplugintosucha
modeltorepresentvirtualmachineandvirtualhostbehavior?Thesetenarethetopicof
thissecondarticle.Inthenextsection,youllgainanappreciationfortheamountofeffortit
takestoconvertjusttenintegersintoanswers.

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

7
TenBehaviorQuantifyingCounters
Whatfollowsaretencountersconsideredtobemostimportantindeterminingtherough
behaviorswithinavSphereenvironment.Thinkofthemasanequationwithtenvariables.
Withtherightequation,plugginginthesevalueswillnetyouanapproximationofyour
vSphereenvironmentsbehaviors.
Counter#1:CPU.ready.summation
CPUreadytimerelatestothepercentageoftimeavirtualmachinewasreadytousea
physicalCPUbutcouldnotgetscheduledtorunonit.AhighCPU.ready.summationtime
meansthatvirtualmachinesarewaitingforphysicalCPUresourcesthatarentavailable.A
highcountheretendstoindicatethatphysicalCPUisabottlenecktoperformance.
Counter#2:CPU.usagemhz.average
TheCPU.usagemhz.averagecountermeasuresaverageCPUusageinmegahertzduringa
configuredinterval.Measuredoverallphysicalprocessors,thiscounterisaprimary
indicatoroftheamountofCPUloadbeingplacedonthehost.
Counter#3:mem.active.average
mem.active.averagerepresentsanestimationofhowmuchmemoryisactivelybeingused
byvirtualmachines.ThisestimationismadebytheVMkernelandisbasedonrecently
touchedmemorypages.Thismemorycounterreferencesthequantityofguestmemorythe
guestisactuallyusingtoaccomplishitsworkloadrequirements.
Counter#4:mem.consumed.average
Slightlydifferentthanthemem.active.averagecounter,mem.consumed.averagemeasures
theamountofguestphysicalmemoryconsumedbyavirtualmachine.Whenmeasuredfor
avirtualmachine,thislevelofmemoryincludessharedmemoryandmemorythatis
reservedbutnotused.Thiscountercanalsobemeasuredforhostsandclusters.When
measuredforahost,thecountermeasurestheamountofmachinememoryusedonthe
host.Foracluster,itmeasurestheamountofmemoryusedbyallpoweredonvirtual
machineinthecluster.
Counter#5:mem.swapped.average
HostswappingisalastditchapproachusedbytheVMkernelduringperiodsofcontention
toensurevirtualmachinesneverrunoutofavailablememory.Hostswappingrequires
transferringmemoryfromRAMtodisk,whichsignificantlyreducesitsperformance.The
mem.swapped.averagecounterreferencesthecurrentamountofguestphysicalmemory
thathasbeenswappedouttoitsswapfile.Avalueheregreaterthanzerocanindicatethat
emoryisabottlenecktoperformance. m

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

8
Counter#6:mem.vmmemctl.average
OvercommitmentinvSphereenvironmentsishandledthroughaprocesscalled
ballooning.Theprocessreclaimsunusedmemoryfromrunningvirtualmachinestomakeit
availableforothers.Anexcessivevalueformem.vmmemctl.averagemeansthatvirtual
machineshavebeenassignedtoomuchmemorythattheyarenotusingbutisneededby
othervirtualmachines.Alargeamountofballooningcanhaveanimpactonoverallhost
performance.
Counter#7:disk.busResets.summation
SCSIbusresetsoccurwhenareadorwritecommandcannotbecompletedwithinan
acceptableamountoftime.Theseresetsoftenindicateanunderlyingperformanceissue
withinstoragehardwareandaremeasuredusingthedisk.busResets.summationcounter.A
nonzerovalueherecanindicatethatstorageisabottleneck.
Counter#8:disk.totalLatency.average
Totallatencyreferstotheabsolutequantityoftimeelapsedbetweensubmittinga
commandtostorage,processingthatcommand,andreceivingtheanticipatedresponse.
Measuredbythedisk.totalLatency.averagecounter,itsinformationhelpstoidentifythe
totalamountofvirtualmachineprocessingdelaycausedbystoragehardware,andis
anotherindicationthatstorageisabottleneck.
Counter#9:disk.usage.average
Relatedtothedisk.totalLatency.averagecounter,disk.usage.averagemeasuresthetotal
diskI/Orate.Thisinformationusefulforidentifyingwhenstorageeitherwithinthe
storageitselfortheconnectiontothatstorageisabottlenecktovirtualmachine
performance.
Counter#10:net.usage.average
Thisfinalcountermeasuresthecombinedsendandreceiveratesfornetworktrafficduring
aconfiguredinterval.Thenet.usage.averagecountercanbemeasuredagainstasingle
virtualmachineortheentirehost,andisusedtoidentifyhowmuchdatatrafficispassing
nandoutofthemeasuredhostorvirtualmachine. i

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

9
CountersArentEverything
Thesecountersmightcreatethattenvariableequation,buttheybynomeansfully
approximateeverybehavioryourvSphereenvironmentexperiences.Theactualsetof
countersisfarhigher,makingthemodelfarmorecomplex.Thatsaid,knowingthese
criticaltengetsyoustarteddowntheroadofquantifyingvSpheresbehaviors.Italsogives
youamuchbetterappreciationforhownecessaryassistivesupportisinresolvingVMware
vSpheresbiggestperformanceissues.
Thatsaid,countersarenteverythingnoraremathematicalmodels.Bothcountersand
themodelstheyfeedonlyworkwhentheenvironmentremainspredictablystable.
Althoughvirtualizationisbynatureadynamicarchitecture,youcangaintherightkindsof
stabilitybyobservingasetofgoodpractices.Thethirdandfinalarticleinthisseries
eintelligenceisinfactintelligent. explainstwelvepracticesthatwillensureyouractionabl

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

10
Article3:WhatTwelvePracticesConvert
RawDataintoResolutions?
Keepingeyesontencountersforasinglevirtualmachineisnteasy.Doingthesamefor
dozensorhundredsofvirtualmachinesisfunctionallyimpossibleforanyhumanbeing.
Thatswhyassistivetoolsarenecessarytoconvertthosecountersrawdataintoactionable
intelligence.AnsweringthatallimportantquestionofWhatshouldIdo?requiresaligning
whatsgoingwiththerangeofpossibleresolutions.
Thislastarticlewaswrittenspecificallytohighlighthowdifficultthatprocessiswith
countersalone.Ifnet.usage.averageishightodaybutsoisdisk.busResets.summation,what
shouldyoudo?Isthebottleneckrelatedtonetworkoversubscriptionortoasituationin
yourstoragelayer?Evenworse,arebothsubsystemsexperiencingaproblem,orisone
problemcausingtheother?
Evenmoreinsidiousisthesituationwheretheissueisntaproblematall.Insteadof
sourcingfromsomehardwareshortcoming,perhapstheproblemrelatestoanother
administratorsstorageornetworkingactivities.Maybetheyvejustbegunalargeand
unthrottledmigrationofdataoverthenetwork.Numberslie.Theydosoparticularlywhen
nogovernanceexistsovertheactivitiesthosenumbersaremeasuring.
Thus,thisfinalarticleintendstobringstabilitytoyourvSphereenvironment.Indeedthat
environmentishighlydynamic.Thatsthenatureofvirtualizationanditstechnologiesthat
aggregateITworkloads.Buttamingitsdynamicsrequiresasetofstabilizingpracticesthat
ensurecounterdataretainsmeaning(seeFigure1).GoverningyourvSphere
environmentsactivitiesgoesalongwayinensuringitsbehaviorscanbepredictably
categorized.

Figure1:StabilizingpracticesarethefinalpiecetoresolvingvSpheresbiggest
performanceissues.

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

11

Theelevenprocessesdescribedinthisarticleassistwiththistask.Eachdealsmorewith
thepeoplesideofvirtualizationstechnology,butallarenecessarytoinstillthat
predictablestabilitythemathematicalmodelsrequire.
Alongwiththefirstelevenisatwelfthandfinalprocessthatinvolvesthemodelitself.You
couldabsolutelycreateyourownmodel,onethattakesintoaccounteachbehaviorsrange
ofpossibilitiesanditsimpactsoncountersbutwhydothat?Leveragingonebuiltby
expertsmeansimmediatelyincorporatingtheirexperienceintoyourvSphereenvironment.
Inaway,itsalotlikehavingtheworldsgreatestperformanceandcapacitymanagement
expertsrightatyourfingertips.Considerthesetwelveprocessesasyourfinalpiecein
resolvingVMwarevSpheresbiggestperformanceissues.
TheDailyPractices
VMwaresactivitiesrepresentanalwaysonservice.Unlikeafileserverthatcangodown
onceinawhileandnotharmthebottomlinetoomuch,VMwaresservicesarethe
foundationuponwhichallotherdatacenteractivitiesreside.WhenVMwaregoesdown,
everythinggoesdown.
ThatswhyvSpheresdailypracticesdealmostwithmonitoring.Discoveringinappropriate
behaviorsearlyandbeforetheyimpactusersisofgreatestimportanceinthesedaily
practices.Doingsoviaadashboardthatincorporateshardwareandsoftwarebehaviors
beneathasinglepaneofglassshouldbeadesiredgoal.Gettherebyincorporatingthefirst
threepractices:
Practice#1:SNMPMonitoring
VMwarebydefaultdoesntdoaterrificallygoodjobwithitsSNMPexposure.Enablingand
tuningsuchmonitoringrequiresextrastepsthatarentimmediatelyobviouswithinits
interface.ButSNMPmonitoringiscriticallyimportantwhenthatsinglepaneofglass
managementisyourdesiredendstate.Forthefirstpractice,brushuponyourSNMP
technologies,orfindasolutionthatllautomatetheirimplementation(and,more
importantly,theirtuningoncethosetechnologiesareinplace!).
Practice#2:ResourceUtilizationMonitoring
ResourcesareconstantlyinfluxinsidevSphere.VirtualmachinesusemoreCPUforawhile,
thenuseless.Theirusedependsontheneedsofprocessesandusers.Monitoringthat
resourceutilizationacrossvirtualmachines,hosts,andclusters,isfundamentally
importanteveninthefullyHA/DRSautomatedenvironmenttogetahandleoncapacity
ssuesbeforetheyimpactusers. i

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

12
Practice#3:AlertMonitoring
vSpherebydefaultwillalertyouwhenpreconfiguredconditionsoccur.Butwhenwasthe
lasttimeyoulookedthroughitsalerts?Doyouknowwhichareenabledandwhichare
properlytuned?Haveyoualsointegratedthemintoyourgreateralertmanagement
system?Ifnot,youllbemissingthemwhentheyreannouncedinthevCenterClientbut
nowhereelse,oryoullnevergetthematallbecausetheywereneverenabled.Thepractice
ofcheckingvCenteralertsonadailybasisisyourfirstlineofdefenseagainstavSphere
environmentthatisntmeetingtheneedsofitsvirtualmachines.
TheMonthlyPractices
Inadditiontothedailypracticesareasetthatrequirelessfrequentattention.This
reductioninfrequencydoesnotinsinuatethatthesetasksarelessmeaningful,onlythat
theyrequirelessfrequentattention.Arguably,themonthlytaskscomprisethemore
importantgroupbecausetheyaremorelikelytobeforgottenovertime.Setupascheduled
activityonyourcalendar,orincorporateasolutiontohelpyourememberthesefivein
depthpractices.
Practice#4:DiskSpaceUtilization
AnotherareawhereVMwarevSpherehasneverdoneaterrificallygoodjobisalerting
whenavailablediskspaceislow.Yetatthesametime,VMwarewarnsthatadatastorethat
fillscompletelyisoneoftheworstsituationsanyenvironmentcanexperience.Younever
wantthesituationwherediskspacerunsout,particularlyinthinprovisioned
environmentswherevirtualmachinesthinktheyhavemorediskspacethantheyreallydo.
ThatswhyPractice#4remindsyoutoverifyyourdiskspaceonatleastamonthlybasis
ifnoteverydayduringtimeswhenspaceislow.
Practice#5:ApplicationRestarts
Thevirtualadministratorspendssomuchtimeworryingaboutresourcesandhardware
thattheysometimesforgetvSphereisreallyabouttheapplications.Thoseapplications
sometimesexperiencebadbehaviorslikerestartsduringinappropriatetimes.Othersneed
restartingfromtimetotimetoreturnthemtohealth.Takingamonthlylookatapplication
historiesandbehaviorshelpskeepyourapplicationsinline.
Practice#6:ServerReboots
Virtualserverstoocanhaveoddrebootrequirementsandbehaviors.Somereboot
spontaneously,givinglittlewarningthattheyreabouttoincuraserviceoutage.Others
needregularrebootstoclearmemoryandcollectedprocesses.Maintainingaserverreboot
logonamonthlybasisandmonitoringforresourceodditieskeepstheirhealthat
aximum. m

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

13
Practice#7:OfflineMaintenance
Oneclassofrebootsnoonelikesbuteveryonedoesarethosesurroundingupdates.The
monthlyupdatecyclehasbecomederigueurinmostdatacenters,withpatchesthemselves
oftenreleasedonamonthlybasis.Usethatdowntimeasyouropportunitytorightsize
assignedresourcelevelstothevaluesyouvedeterminedoverthepastmonth.Alsousethat
timetohandleanyspecialbackups,snapshots,andothermaintenanceactivitiesthatwork
bestwhilethevirtualmachineisoffline.
Practice#8:OverallHealth
Last,anoverallhealthcheckisinorderonamonthlybasis.Thishealthcheckisnt
necessarilyjusttothevirtualmachinesrunningatopvSpherebutalsotothevSphere
environmentitself.Reviewlogs,validateloadbalancingeffectiveness(includingmoving
virtualmachinesbetweenclusters),verifydatacenterhealth,andperformallthecareand
feedingactivitiesonamonthlybasisyoukeeppromisingtodo.Leaningonassistive
technologyherethatremindsyouofsuchhealthcheckscanbeuseful,particularlywhen
thattechnologygivesyouspecificadviceforwhatstepstotake.
TheYearlyPractices
YourVMwarevSphereyearlypracticesdontcomearoundthatoften,buttheyrenoless
important.Theycenteronevaluatingfutureinitiatives,incorporatingfeedbackfrom
monitoringsolutionsandusersalike,andreflectingonandoptimizingtheprocessesyouve
laidintoplace.Youryearlychecksareimportantopportunitiesforyoutoimproveuponthe
governanceactivitiesyouincorporatedinthepastyear.
Practice#9:BudgetingforReplacementorAugmentation
Thefirsttaskofcapacitymanagementisresourceassurance,ensuringthatvirtual
machineshavethephysicalresourcestheyneedtodotheirjobs.Thesecondtaskis
planning,analyzingresourceutilizationoverthelongtermtolookfortrends.Awell
managedenvironmentshouldbeabletodrawastraightlinethatbeginswiththeir
historicalusageandendsatthedatetheyllneedmoreresources.Withtherightdailyand
monthlypracticesinplace,Practice#9becomeseasywhenyourannualbudgetnumbers
aredue.
Practice#10:UserFeedback
Thatplanningactivityalsorequiresanunderstandingoftheusersexperience.Your
numbersmay,forexample,showthatserversandhosteddesktopsareperformingto
expectations;however,yourexpectationsandthoseofyourusersmightbemismatched.
Interviewingthemandincorporatingtheirfeedbackensuresthattheservicesyoure
eliveringaremeetingtheirneeds. d

TheEssentialsSeries:ResolvingVMwarevSpheresSixBiggestPerformanceIssues GregShields

14
Practice#11:LargeScaleChangeInitiatives
Theeleventhtaskgoesonestepfurther.Justlikeyourvirtualenvironment,yourbusinessis
ahighlydynamicentity.Planningforthefutureinavoidofbusinessinitiativesmeansyou
wonthavethecapacityinplacewhenthebusinessrequiresit.Planningforand
implementingthatcapacitybeforehandmeansavoidingperformanceconflictsonesthat
willhaveyouupdatingyourresumeasyousearchfornewemploymentdowntheroad.
Practice#12:TheOngoingPractice
LastistheentirereasonforthisEssentialsSeries.Youasahumancantmonitorallthose
countersalone.Youneedtheassistivesupportofsoftwaresolutionsthattranslaterawdata
intoactionableintelligence.Thefinalpracticesuggeststhatyoulooktosolutionsthat
accomplishthattask.OnlywithalittlehelpcanyoutameVMwarescomplicatedbeast,full
fmovingparts,deepintegrations,andunexpectedimpacts. o

DownloadAdditionalBooksfromRealtimeNexus!
RealtimeNexusTheDigitalLibraryprovidesworldclassexpertresourcesthatIT
professionalsdependontolearnaboutthenewesttechnologies.Ifyoufoundthisbookto
beinformative,weencourageyoutodownloadmoreofourindustryleadingtechnology
booksandvideoguidesatRealtimeNexus.Pleasevisit
ttp://nexus.realtimepublishers.com h .