You are on page 1of 33

nmon(NigelsMonitor)

thebasics
newfeatures
nowpartofAIXitself
NigelGriffiths
IBMEurope
PresentationVersion14

2009IBMCorporation

Trademarks

2009IBM
2

ThefollowingaretrademarksoftheInternationalBusinessMachinesCorporationintheUnitedStates,othercountries,orboth.
NotallcommonlawmarksusedbyIBMarelistedonthispage.FailureofamarktoappeardoesnotmeanthatIBMdoesnotusethemarknordoesitmeanthattheproductisnot
activelymarketedorisnotsignificantwithinitsrelevantmarket.
ThosetrademarksfollowedbyareregisteredtrademarksofIBMintheUnitedStatesallothersaretrademarksorcommonlawmarksofIBMintheUnitedStates.
Some,allornoneofthispresentationmight,mayorwillbetrueornot,asapplicable.

ForacompletelistofIBMTrademarks,seewww.ibm.com/legal/copytrade.shtml:
*,AS/400,ebusiness(logo),DBE,ESCO,eServer,FICON,IBM,IBM(logo),iSeries,MVS,OS/390,pSeries,RS/6000,S/30,VM/ESA,VSE/ESA,
WebSphere,xSeries,z/OS,zSeries,z/VM,Systemi,Systemi5,Systemp,Systemp5,Systemx,Systemz,Systemz9,BladeCenter

Thefollowingaretrademarksorregisteredtrademarksofothercompanies.
Adobe,theAdobelogo,PostScript,andthePostScriptlogoareeitherregisteredtrademarksortrademarksofAdobeSystemsIncorporatedintheUnitedStates,and/orothercountries.
CellBroadbandEngineisatrademarkofSonyComputerEntertainment,Inc.intheUnitedStates,othercountries,orbothandisusedunderlicensetherefrom.
JavaandallJavabasedtrademarksaretrademarksofSunMicrosystems,Inc.intheUnitedStates,othercountries,orboth.
Microsoft,Windows,WindowsNT,andtheWindowslogoaretrademarksofMicrosoftCorporationintheUnitedStates,othercountries,orboth.
Intel,Intellogo,IntelInside,IntelInsidelogo,IntelCentrino,IntelCentrinologo,Celeron,IntelXeon,IntelSpeedStep,Itanium,andPentiumaretrademarksorregisteredtrademarksofIntel
CorporationoritssubsidiariesintheUnitedStatesandothercountries.
UNIXisaregisteredtrademarkofTheOpenGroupintheUnitedStatesandothercountries.
LinuxisaregisteredtrademarkofLinusTorvaldsintheUnitedStates,othercountries,orboth.
ITILisaregisteredtrademark,andaregisteredcommunitytrademarkoftheOfficeofGovernmentCommerce,andisregisteredintheU.S.PatentandTrademarkOffice.
ITInfrastructureLibraryisaregisteredtrademarkoftheCentralComputerandTelecommunicationsAgency,whichisnowpartoftheOfficeofGovernmentCommerce.
*Allotherproductsmaybetrademarksorregisteredtrademarksoftheirrespectivecompanies.
Notes:
PerformanceisinInternalThroughputRate(ITR)ratiobasedonmeasurementsandprojectionsusingstandardIBMbenchmarksinacontrolledenvironment.Theactualthroughputthatanyuserwill
experiencewillvarydependinguponconsiderationssuchastheamountofmultiprogrammingintheuser'sjobstream,theI/Oconfiguration,thestorageconfiguration,andtheworkloadprocessed.
Therefore,noassurancecanbegiventhatanindividualuserwillachievethroughputimprovementsequivalenttotheperformanceratiosstatedhere.
IBMhardwareproductsaremanufacturedfromnewparts,ornewandserviceableusedparts.Regardless,ourwarrantytermsapply.
AllcustomerexamplescitedordescribedinthispresentationarepresentedasillustrationsofthemannerinwhichsomecustomershaveusedIBMproductsandtheresultstheymayhaveachieved.Actual
environmentalcostsandperformancecharacteristicswillvarydependingonindividualcustomerconfigurationsandconditions.
ThispublicationwasproducedintheUnitedStates.IBMmaynotoffertheproducts,servicesorfeaturesdiscussedinthisdocumentinothercountries,andtheinformationmaybesubjecttochangewithout
notice.ConsultyourlocalIBMbusinesscontactforinformationontheproductorservicesavailableinyourarea.
AllstatementsregardingIBM'sfuturedirectionandintentaresubjecttochangeorwithdrawalwithoutnotice,andrepresentgoalsandobjectivesonly.
InformationaboutnonIBMproductsisobtainedfromthemanufacturersofthoseproductsortheirpublishedannouncements.IBMhasnottestedthoseproductsandcannotconfirmtheperformance,
compatibility,oranyotherclaimsrelatedtononIBMproducts.QuestionsonthecapabilitiesofnonIBMproductsshouldbeaddressedtothesuppliersofthoseproducts.
Pricessubjecttochangewithoutnotice.ContactyourIBMrepresentativeorBusinessPartnerforthemostcurrentpricinginyourgeography.

v
2009IBM
3

nmonWhyandPrinciples?
Whynmon?
AIXMonitoring&TuningSimple,small&safe
Producebenchmarkreportswithgraphs(Iamlazy)
Formyself(personalpetproject)buteveryonewantedacopy!!

Designprinciples
Zeroinstallationtime&simpletouse
Fortheperformanceexpertmaxinfoonthescreen
Lessthan1%CPU
Forlargemachines
64CPUs,4000+disks,35,000processes

Freelyavailable
Performancemonitorfor
AIX5and6
AIX4(viaoldernmonversion)
Linux
POWER,x86&mainframe

nmonbasics

2009IBM
4

nmon

CSVfilein
nmon
format

Online

DataCollection

nmonOnlineorDataCollector

2009IBM
5

nmon

Stephens

nmon
file

Screen

forFoption

2009IBM

nmon
6
Consolidator

Stephens

Others
Federicos

pGraph

Bruces

nmon2web
Perl

nmon2rrd
Cfilter

nmon
Analyser

18 0
16 0
14 0
12 0
10 0
8 0
6 0
4 0
2 0
0

JavaDynamicGraphs

Excelgraphs
Scripts

Scripts

rrdtoolscripts
+CPU&RAM
aggregation

Createrrd
Loadrrd
Graphrrd
index.html

10 0
9 0
8 0
7 0
6 0
5 0
4 0
3 0
2 0
10
0

nmonflow
Website+.jpggraphs
rrdtool

rrdtool

open
source

open
source

v
2009IBM
7

nmonishalfthestory

Atleast,50%ofthesuccessofnmon
isdowntothenmonAnalyserfromStephenAtkins

v
2009IBM
8

nmonConsolidatorv1.4.1
ProducesoverviewchartsforCPU,Memory,Network,Disk
Fast&simplecode,pregeneratedcharts
Multiplenmon/topasoutfilesfor:
ALLLPARsofonemachine
ClusteredsystemOracleRAC/HPC
ModellingchangesfromDedicatedtoSharedCPULPARs
ServerconsolidationModelling
Reportingactualorpotentialsavingsfromvirtualisation
Trendchartsprocessingmultiplefilesfrom1LPAR

v
2009IBM
9

QuickDemonstrationHere

nmononline
nmonanalyser

v
2009IBM
10

nmonOfferedtoAIXLabsinAustin,Texas
nmonstartedin1996
ManycustomershavewantedIBM/AIXSupport
LabattitudewasclassicNotinventedheresyndrome
&notaproblemdeterminationtoolfairenough

IhaveprovidedSupport
Ihopequitewellorbrilliantly

PerhapswedontwantAIXsupport
Icanproduceanewversionoveraweekend
IcanaddfeaturesasandwhenIwantthem
LabsdontgetthebackportingconceptonlylatestTL

Decidedhappytocontinueforyearstocomebutthen

v
2009IBM
11

AIXLabs:

nmonwillbe
assimilated!

v
2009IBM
12

Codestuffing

Linesofcodefrom65009000

nmononPOWER6&AIX6+NewFeaturesforV12
UserRequested

Advanced,POWER6andAIX6items

1. DiskServiceTimes

8. VirtualI/OServerSEAviaentstat

2. SelectingParticularDisks

9. PartitionMobility

3. TimeDrift

10.WPAR&ApplicationMobility

4. MultiplePageSizes

4KB,64KB,++ 11.DedicatedDonating

5. TimestampsinUTC&no.ofdigits
6. MoreKernel&HypervisorStats*
7. HighPrioritynmon

2009IBM
13

(POWER6)
(AIX6)

(POWER6)

12.FoldedCPUcount

(SPLPAR)

13.MultipleSharedPools

(POWER6)

14.FibreChannelstatsviafcstat

Housekeepingitems
15.Bugfixessmallorfinetuning,seenmonwikiforalist
16.Networkpacketsizesnowsavedtofile
17.Warningsfromnmonlike:networkoverflow
18.TOPOnlinesizesinKB,MB,GBtokeepcolumnsaligned
19.FastabortincapturemodeSIGUSR1duringconfigcollectionstopsnmon
20.Returnanerror&oninvalidoptionsreturncodestocheckitfinishednormally
21.Adapter"notavailable"workaroundseenmonFAQwikiabouttheseAIXbugs
22.EMChdiskpowerrenamedtopowertosavescreenspace
23.NFSv4forAIX5.3ML5+

WPAR

2009IBM
14

App/Sys
Defined/Active
Usr/Sys%CPU
RAMuse&
FScache
RunQ/pSwich
andFork
Dynamic
WPARclass
DefaultNo
WLMcontrol

TopProcs
WPARname
System/Default
Etc=GlobalAIX

v
2009IBM
15

DirectFibreChannelandVIOSSEA
SharedEthernetwithO
FibreChannelwith^alsoregularLPAR&FCtapedrives
Actuallyusingmoreexpensiveentstatandfcstatcommands
nmon12h=HelpHost=silver_vios1Refresh=2secs10:03.36

SharedEthernetAdapter

AdapterReceiveTransmit
Requests
SizeKB

NumberNameKB/s
KB/s
InOutInOut

1
ent3 141.41.4211.622.567.01024.0

Totals0.0
0.0MB/s21.622.5

FibreChannelAdapter

AdapterReceiveTransmit
Requests
SizeKB

NumberNameKB/s
KB/s
InOutInOut

1
fcs0 9945.80.0420.20.017.9

Totals0.0
0.0MB/s0.0
0.0

v
2009IBM
16

IaminternalbetatesterforAIX6.1TL02(Nov2008)
Idoproductintroductionneedaccesstonewstuff
InstalledAIX6.1TL02&wasusingnmonfora
coupleofdaysbeforeIrealiseditwastopas_nmon!!

Syntax,toggleandlayoutperfect
Gobsmacked

v
2009IBM
17

nmontopas_nmon

v
2009IBM
18

topasnmonInstall

Clarification:
Donothingasitis
InstalledbyDefault
withAIX

v
2009IBM
19

topasnmonCommand

Typicalplace
usersbutthe
nmonshellscript

$nmon
script/usr/bin/nmon
binary/usr/bin/topas_nmon
hardlinktotopas&topasrec
$topas
andhit~

$nmonfs300c288
Justworksasnormal

v
2009IBM
20

Menubasedaccesstonmonfunctions
smittytopas,websmorpConsole
Allrunoffthesametemplate
Performance&ResourceConfigureTopas

v
2009IBM
21

Using:smittytopas

#smittytopas

AddHosttotopasexternalsubnetsearchfile(Rsi.hosts)
Listhostsintopasexternalsubnetsearchfile(Rsi.hosts)
ListactiveRecordings
StartPersistentlocalrecording
StartNewRecording
StartPersistentCECrecording
StopRecording
Startlocalrecording
Listcompletedrecordings
StartCECrecording
GenerateReport

Period
Day
Hour
Custom

Binary
nmon

Startnow
&onceonly

ViaCron
everyday

StartPersistentXXXXXrecording
Startnmonrecording
[EntryFields]
TypeofRecording
nmon
*LengthofRecording
custom
*RecordingIntervalinseconds[300]#
*NumberofSamples[288]#
*Overwriteexistingrecordingno+
OutputPath[]
RunningPriority[]#
*RecordingtimezoneLocaltime+
IncludetopprocessesIncludetopprocess+

Typeorselectvaluesinentryfields.
PressEnterAFTERmakingalldesired
changes.
[EntryFields]
TypeofRecordingCEC
LengthofRecording
persistent
*RecordingIntervalinseconds[60]#
*NumberofDaystostoreperfile[1]#
*NumberofDaystoretain[31]#
OutputPath[]
*Overwriteexistingrecordingfileno+

v
2009IBM
22

FilePostProcessing
Sodothefollowingwork?
nmonanalyser
nmonconsolidator
nmon2rrd Cprogramtogenerategraphs+html
nmon2webPerlprogram ditto

Iassumeso,
usershavethe
codetofixit!

v
2009IBM
23

SmittyandpConsole

smittytopasorpConsole
Runoffthesametemplate
Performance&ResourceConfigureTopas

topas
nmon

Classic

nmon
Classic

2009IBM
24

v
2009IBM
25

nmonSupport

AIX5.3TL09+&AIX6.1TL02+
YouhavefullAIXSupportwithSWMA

topas
nmon

UsenewOfficialnmon

PleaseONLYversion12e+
Google:AIXPerformanceToolForum

nmon
nmon

OlderAIXreleasesusenmonClassic

Classic

RaiseaPMR

nmonclassicmaybea12fthenstop
ALLnewdevelopmentintopas_nmon
topas
nmon

topas
nmon

topas
nmon

topas
nmon

topas
nmon

topas
nmon

topas
nmon

topas
nmon

topas
nmon

topas
nmon

Classic

nmon

Classic

nmon

12e

topas
nmon

topas
nmon

topas
nmon

2009IBM
26

nmonDevelopment
12f

v
2009IBM
27

nmonisnow

GREEN
fullyrecycled
sourcecode
Noanimals
harmedinthe
testingofnmon

BlatantAdvertforHandsOnMovies

2009IBM
28

AIX6&Power6themovie!
10onWPAR
8onAIX
7onPOWER6SPLPAR/SMT
10onIVM
5onCoolstuff
15onPowerVM
Eachmovie5to40minutes
Downloadablefilesin.wmvformat
Watchactualuseofthesenewfeatures
Intotal~11hours

AIXmovie

v
2009IBM
29

SearchGooglefor:AIXMovie

TenonWorkloadPartitions(WPAR)
TenonIntegratedVirtualisationManager(IVM)
AIX6RolebasedAccessControl(RBAC)
AIX6JFS2withthenologoption
AIX6JFS2withinlinesnapshotoption
AIX6Probevuenewwavemonitoring
AIX6EncryptedFileSystem
AIX6SystemDirectorpConsole
AIX6Securitywithaixpert
AIX6Securitywithfpm
POWER6DecimalFloatingPoint(DFP)
HMC7HardwareManagementConsole
POWER6PartitionMobility
POWER6MemoryProtectionKeys
POWER6PartitionPriority
POWER6MultipleSharedCPUPools
POWER6SharedCPU&PoolMonitoring

GangliaPerformanceMonitoring
LinuxforPowerPowerVMLx86Translator
nmonGetYouStarted
nmonversion12
nmonnowinAIX
POWERIntroduction
HMCIntroduction
CreatethefirstLogicalPartition(LPAR)
InstallingAIXfromCDROM
InstallingAIXfromNIM
DynamicallychangingLPARs
EthernetOptionswithVirtualEthernet
VirtualEthernetSetup
DiskOptionswithVirtualSCSI
VirtualDiskSetup
HowmanyCPUsinaPowerChip?
AreLogicalCPUsReal?WhatisSMT?
WhatisaVirtualProcessor?And
controllingSharedProcessorLPARs?
VIOS2.1Principles
VIOS2.1Features

v
2009IBM
30

Topas2secrets:
1)nmoninterworking
2)VirtualI/OServer
specificstatistics

topasC

2009IBM
31

TopasCECMonitorInterval:10ThuNov1304:06:542008
PartitionsMemory(GB)Processors
Shr:4Mon:8.0 InUse:6.1 Shr:2 PSz:4Don:0.0Shr_PhysB 0.02
Ded:0
Avl:
Ded:0APP:4.0Stl:0.0Ded_PhysB 0.00
HostOSM MemInULp UsSyWaId PhysB VcswEnt%EntCPhI pmem
shared
silver_lpar3A61UM2.01.624500550.012100.501.10
silver_lpar2A61UM2.01.327200270.002190.500.901.03
silver_lpar4A61UM2.01.62000990.002180.500.800.97
silver_lpar5A61UM2.01.621000890.002300.500.701.15
silver8A61U2.01.34010980.019270.50 2.24

C
~

topas

TopasMonitorforhost:orangeEVENTS/QUEUES
FILE/TTY
MonMar1608:04:052009Interval:2Cswitch184 Readch0
Syscall71 Writech322
CPUUser%Kern%Wait%Idle%PhyscReads0 Rawin0
10.033.10.066.90.00Writes1 Ttyout322
019.462.20.018.50.01Forks0 Igets0
20.027.50.072.50.00Execs0 Namei2
30.029.30.070.70.00Runqueue1.0 Dirblk0
Waitqueue0.0
NetworkKBPSIPackOPackKBInKBOutMEMORY
Total0.72.52.50.20.5PAGING
Real,MB2048
Faults0 %Comp64.4
DiskBusy%KBPSTPSKBReadKBWritSteals0%Noncomp21.1
Total0.00.00.00.00.0PgspIn0 %Client21.1
PgspOut0
FileSystemKBPSTPSKBReadKBWritPageIn0PAGINGSPACE
Total0.00.00.00.0PageOut0 Size,MB512
Sios0 %Used1.1

~
topas_nmonHost=orangeRefresh=2secs 08:01.24
CPUUtilisationSmallView
EntitledCPU=0.40UsedCPU=0.008
LogicalCPUs0255075100
CPUUser%Sys%Wait%Idle%|||
||
00.0 0.0 0.0100.0|>|
10.0 0.0 0.0100.0|>|
20.0 0.0 0.0100.0|>|
30.0 0.0 0.0100.0|>|
EntitleCapacity/VirtualCPU+|||+
EC0.41.00.00.7||
VP0.10.20.00.1||
EC=2.1%VP=0.4%+NoCap|Folded=1|100%VP=2CPU+

nmon
topas_nmon

HitEVirtualEnternetincludingSEA

VIOSmonitoringviatopas

VI

2009IBM
32

2.
1

TopasMonitorforhost:
bronze_ivm Interval:2FriDec508:08:192008
===============================================================================
NetworkKBPSIPackOPackKBInKBOut
ent10(SEA)31.343.0
43.0
15.915.4
|\ent2(VETH)15.915.527.014.31.6
\ent0(PHYS)15.527.516.01.6
13.9
lo0
0.0
0.0
0.0
0.0
0.0

HitDandthendVirtualSCSIovertheVIOS
TopasAdapterView:
bronze_ivm Interval:2FriDec508:29:172008
===============================================================================
AdapterKBPS
TPS KBRKBW
sissas0
9.9K92.00.09.9K
vhost0
0.0
0.0
0.0
0.0
vhost1
9.9K184.092.0 92.0
===============================================================================
Vtargets/DisksBusy%KBPS TPS
KBRART MRT KBW AWT MWT AQW AQD
hdisk00.0
0.0 0.0
0.0 0.0 3.60.0 0.0 62.60.0 0.0
hdisk125.09.9K92.00.0 0.0 3.59.9K4.228.90.0 0.0
hdisk20.0
0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
hdisk30.0
0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
hdisk40.0
0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

v
2009IBM
33

AndFinallytheEnd
Google:nmonwiki

Download
FAQ&Manual
Analyser/Consolidator
PerformanceToolsForum
QuestionsmostlyaboutnmonJ

Google:AIXMovies

nmonandloadsmore

Runthelatestversionofnmon
Changingonefileishardlyanupgrade!
UncappedSharedProcessorLPARs(SPLPAR)monitor
PhysicalCPUuse
NotUtilisation