Professional Documents
Culture Documents
Nigel Griffiths
IBM Power Systems © 2012 IBM Corporation
Advanced Technology Support, Europe
To
© 2010 IBM
p
as
2
&
nm
nmon
Year 16
2 of 33
v
© 2010 IBM
p
as
3
&
nm
now Open Source
on
nmon for Linux on Power
then added x86, Z, x86_64
Released by IBM - July 2009
– http://nmon.sourceforge.net
http://nmon.sourceforge.net
– Binaries for 66 Linux flavours
Largely thanks for MPG
Compile for your Linux flavour
– Cell blades, disk engines,
– Diskless, embedded ++
– 6000 lines of C code200 KB
Fully supported by the nmon analyser
On a Linux Distro's?
– Ubuntu/Debain (get-apt), IBM on PowerLinux, x86 & secret, …
3 of 33
To
© 2010 IBM
p
embedded
as
4
&
x86_64
nm
60+ flavours!
nmon for Linux Open
Source’d
1997 …. 2000 2003 2006 2009 2011
4.3.3
1999 2001 2007 2010
AIX command
2003 2005 2008
topas_nmon
merged in to topas
nmon12f function
AIX 3.1.5 - 4.2 nmon9f AIX4 nmon10 AIX5 nmon12f AIX6 Released with AIX
Benchmarking Last AIX 4.3.3 Uses public API AIX 5.3 TL09 +
Tuning/Reports Used /dev/kmem & libperfstat AIX 6.1 TL02 +
topas CEC
nmon2web
nmon reports
analyser analyser
nmon
nmon2rrd Consolidator
Lotus 1-2-3
Then Excel Multiple machines pGraph
Website generator
4 of 33
v
© 2010 IBM
nmon
p
-f or –F option nmon
as
Screen nmon file 5
&
Consolidator
nm
on
Stephen’s
Others
nmon 18 0
Federico’s 16 0
Bruce’s Analyser 14 0
12 0
pGraph nmon2rrd 10 0
nmon2web
80
60
C filter 40
20
Perl 0
index.html
nmon flow
Website +.jpg graphs
rrdtool rrdtool
open open
source source
5 of 33
To
© 2010 IBM
p
as
6
&
nm
6 of 33
v
ER
Email 6: © 2013 IBM
7
Pe
rf
What 1st things do I look for in an nmon file?
or
m
an
ce
Firmware level (BBBP lsconf)
AIX level - release, TL and SP (BBBP oslevel -s)
Not using classic nmon (AAA)
If any over 2 years old the nmon user are either a newbie
or a time waster.
From this I know how professional they are!!
If you don’t service your car for three+ years:
Why should the car maker spend time investigating
your problem or take the blame for the top speed !
Then period of time covered and the interval
– Long intervals hide peaks
Then
PO
W
© 2013 IBM
7
Pe
rf
or
m
PO
W
v
ER
© 2013 IBM
What 3rd things do I look for in an nmon file?
7
Pe
rf
or
m
NET tab
an
ce
– Is it pretty!
– Does it peak at a well know network speed?
Page tab
– Should be low but do see high filesystem paging
these days = memory mapped files.
POOL tab
– See if Pool is exhausted
Proc
– Compare RunQ with SMT thread available
Top - if present
– Check is the top CPU processes expected
3 of 3
To
© 2010 IBM
p
as
10
&
nm
10 of 33
v
© 2010 IBM
p
as
11
&
nm
on
11 of 33
To
© 2010 IBM
p
as
12
Mode Changing
&
nm
on
Nigel’s Monitor
nmon
topas_nmon
mode
~
~
topas
topas
default C
DETAIL a CEC topas -C
12 of 33
v
© 2010 IBM
p
as
13
&
nm
Let talk about
on
topas
13 of 33
To
© 2010 IBM
topas problems
p
as
14
&
nm
No marketing
on
No Redbook
Manual is 30+ pages
KB-W
AQW
MWT
PSz
Vcsw
%bdon
MPuse
hpit
10 different online modes physb
7 file formats (with friends) pmem
14 of 33 RTFM
v
© 2010 IBM
Modes
p
as
15
&
nm
Topas
on
nmon
Classic for
Local AIX mode Cross-partition Mode AIX & Linux
nmon mode
[CEC/whole machine]
CSV file
nmon Analyser
Other Local topasout Text
nmon Consolidator
Save Tools Reports
nmon2rrd/nmon2web
Other tools or databases
15 of 33
To
© 2010 IBM
p
as
16
&
nm
16 of 33
v
© 2010 IBM
p
as
17
&
nm
on
cndfp
17 of 33
To
© 2010 IBM
p
as
18
&
nm
on
P L
V
D
F
18 of 33
v
© 2010 IBM
p
as
19
&
nm
Topas VIOS view
on
19 of 33
To
© 2010 IBM
p
as
20
&
nm
Topas Monitor for host: bronze_ivm Interval: 2 Fri Dec 5 08:08:19 2008
===============================================================================
Network KBPS I-Pack O-Pack KB-In KB-Out
ent10 (SEA) 31.3 43.0 43.0 15.9 15.4
|\--ent2 (VETH) 15.9 15.5 27.0 14.3 1.6
\--ent0 (PHYS) 15.5 27.5 16.0 1.6 13.9
lo0 0.0 0.0 0.0 0.0 0.0
VOIS
[SEA] Client
31.3KB/s
Physical Network Virtual Network
13.9KB/s 14.3KB/s
20 of 33
v
© 2010 IBM
p
as
21
&
nm
Another Example two SEA + HEA
on
Topas Monitor for host: bluevio44 Interval: 2 Mon Nov 9 18:28:12 2009
===============================================================================
Network KBPS I-Pack O-Pack KB-In KB-Out
ent5 (SEA) 30.9 35.5 35.0 15.5 15.5
|\--ent1 (PHYS) 15.5 25.0 10.5 14.7 0.8
\--ent4 (VETH) 15.5 10.5 24.5 0.8 14.6
ent6 (SEA) 1.9 15.5 16.0 0.9 1.0
|\--ent0 (HEA PHYS) 1.0 15.5 0.5 0.9 0.1
\--ent3 (VETH) 0.9 0.0 15.5 0.0 0.9
lo0 0.0 0.0 0.0 0.0 0.0
21 of 33
To
© 2010 IBM
p
as
22
&
nm
Hit “D” and then “d” You can see VIOS virtual/physical Disks
And which hosts adapters is generating the I/O
Topas Adapter View : bronze_ivm Interval: 2 Fri Dec 5 08:29:17 2008
===============================================================================
Adapter KBPS TPS KB-R KB-W
sissas0 9.9K 92.0 0.0 9.9K
vhost0 0.0 0.0 0.0 0.0
vhost1 9.9K 184.0 92.0 92.0
===============================================================================
Vtargets/Disks Busy% KBPS TPS KB-R ART MRT KB-W AWT MWT AQW AQD
hdisk0 0.0 0.0 0.0 0.0 0.0 3.6 0.0 0.0 62.6 0.0 0.0
hdisk1 25.0 9.9K 92.0 0.0 0.0 3.5 9.9K 4.2 28.9 0.0 0.0
hdisk2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
hdisk3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
hdisk4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Virtual Disks
Client Server
Logical
Volume
vhost vhost1 sissas0
hdisk0 to 4
22 of 33
v
© 2010 IBM
CEC
pas
23
&
nm
on
Whole machine
Cross Partition
All LPAR
View
23 of 33
To
© 2010 IBM
pas
24
&
nm
LPAR: topas –C
VIOS Once only
- Can take 20 seconds
– topas then hit “C” - Newer version polls
the local network for
– topas –cecview (ugly) AIX LPARs
Totally different program inside topas
xmtopas AIX
xmperf AIX
v
© 2010 IBM
p
as
25
&
nm
Active
on
Memory
Sharing view
AMS has a renewed focus due to new De-Duplication features
25 of 33
To
© 2010 IBM
p
as
26
&
nm
on
Monitoring 1 LPAR
for AMS is “insane”
26 of 33
v
© 2010 IBM
p
topas –C AMS Have to monitor all LPARs
as
27
&
nm
No details for Linux on POWER or IBM i = no daemon
on
Regular extra “g” details
Topas CEC Monitor Interval: 10 Wed Dec 3 10:15:06 2008
Partition Info Memory (GB) Processor Virtual Pools : 0
Monitored : 4 Monitored : 8.0 Monitored :2.0 Avail Pool Proc: 3.7
UnMonitored: - UnMonitored: - UnMonitored: - Shr Physical Busy: 0.28
Shared : 4 Available : - Available : - Ded Physical Busy: 0.00
Uncapped : 4 UnAllocated: - UnAllocated: - Donated Phys. CPUs 0.00
Capped : 0 Consumed : 6.5 Shared : 2 Stolen Phys. CPUs : 0.00
Dedicated : 0 Dedicated : 0 Hypervisor
Donating : 0 Donated : 0 Virt. Context Switch: 976
Pool Size : 4 Phantom Interrupts : 1
Dedicated
Inu
InUse LPAR Working Set
Shared memory LPAR Memory
Logical Memory Mem Physical Memory pmem LPAR
27 of 33
To
© 2010 IBM
p
as
28
&
nm
Hypervisor Page-in/second
Pool Size
In use I/O Memory
Pool Used I/O Memory assign for device driver DMA I/O
For more on I/O Memory see:
• lparstat -m
Machine memory • lparstat -me
Machine memory Used
28 of 33
v
© 2010 IBM
topasout
p as
29
&
nm
on
Takes binary topas recording files /etc/perf/…
Reports for
LPAR,
All Partitions or
Whole Machine
In formats
text report
CSV data files
-c
To
© 2010 IBM
topasout
p as
30
&
nm
v
© 2010 IBM
p
as
31
&
topasrec topasout CEC analyser (Excel)
nm
on
topas –C working OK but want to record it
1. http://tinyurl.com/topas-cec
– Download CEC Analyser from Stephen Atkins
2. topasrec -C -o sample -s 120 -c 720 (every 2 minutes all day)
– Creates sample_cec_101222_2149.topas
3. topasout -a sample_cec_101222_2149.topas
– Creates sample_cec_101222_21492.topas.csv
4. Start CEC Analyser & load the .csv output file & you get
31 of 33
To
© 2010 IBM
Whole machine performance graphs
p
as
32
&
nm
on
32 of 33
v
© 2010 IBM
p
as
33
&
nm
New Stuff
on
1. nmon analyser improvements from Stephen Atkins
– Make sure you use the latest version
– http://tinyurl.com/nmon-analyser
2. Topas Active Memory Expansion
3. nmon Active Memory Expansion file output
4. Topas Memory Affinity
5. nmon gets WPAR
6. Too Many CPUs problem
7. nmon online disk columns
8. nmon file output more stats
33 of 33
purple7.aixncc.uk.ibm.com
To
© 2010 IBM
p
as
34
&
nm
Nigel’s opinion
– “It’s a no brainer & here to stay”
– “Future POWER processors will … err!
Blow your socks off!”
34 of 33
v
© 2010 IBM
p
as
35
Active Memory Expansion
&
nm
on
Conceptual Model
Compressed
Memory Pages Memory pages
To
© 2010 IBM
p
as
36
&
nm
10GB 10GB
36 of 33
v
© 2010 IBM
p
as
37
&
nm
topas
on
TMEM = True Memory
CMEM = Compressed Memory EF = Expansion Factor
T = Target
CI = Compressed Page In A = Actual
37 of 33 CO = Compressed Page Out
To
© 2010 IBM
p
as
38
&
nm
MEM tab
– Size of the Compressed pool (MB)
– Size of true memory (MB)
– Expanded memory size (MB),
– Size of the Uncompressed pool (MB)
MEMNEW tab
– Compressed Pool%
PAGE tab
– Paging rates (pages per second) but this time very quickly
in & out of the compressed memory area
– Compressed pool pgins - other tools like topas call this CI
– Compressed pool pgouts - other tools like topas call this CO
38 of 33
v
© 2010 IBM
p
as
39
&
nm
Topas AME - Very small change
on
Units have moved
– TMEM, MB 2000 TMEM 2.00G
39 of 33
To
© 2010 IBM
p
as
40
&
nm
See my session
– PE07
– POWER7, AIX, Affinity and Performance
40 of 33
v
© 2010 IBM
p
as
41
&
nm
topas –M or
on
topas and the hit M or
nmon then ~ then M
41 of 33
To
© 2010 IBM
p
as
42
&
nm
on
42 of 33
v
© 2010 IBM
p
as
43
&
nm
on
SRAD Memory View
Home processes/threads
43 of 33
To
© 2010 IBM
p
as
44
&
nm
on
44 of 33
v
© 2010 IBM
Topas New: Affinity stats M (memory!)
p
as
45
&
nm
Power 770
on
LPAR = 64 GB
Across 3 SRAD’s
CPU’s are
0- 7 8 = 2
8-11 4 = 1 Excellent
16-23 4 = 1
VP = 4 Good
Very little
use so 100%
not that
important
45 of 33
To
© 2010 IBM
p
as
46
&
nm
v
© 2010 IBM
p
as
nmon WPAR stats are back in AIX 7 TL1/AIX6 TL7 47
&
nm
on
Memory stats Kernel stats
numperm= Process switches
WPAR hit @ & WLM hit W filesystem cache Fork new process
Type =
System or App
State =
Active or Defined
47 of 33
To
© 2010 IBM
p
as
Busy with
Not just WLM hit W WPAR work
48 of 33
v
© 2010 IBM
p
as
49
&
nm
Inside a WPAR
on
Bonkers – like driving a car with your eyes closed
To
© 2010 IBM
p
as
50
&
nm
50 of 33
v
© 2010 IBM
p
as
51
&
nm
Too many CPUs
on
Power 795 = 256 cores = 1024 logical CPUs
Can’t watch this on screen 1024 pixel screen!
Actually logical CPUs not so interesting
– Once we know they are all in use
To
© 2010 IBM
p
as
52
&
nm
52 of 33
To
v
© 2010 IBM
p
as
53
&
nm
Stopped all work
on
Long term Physical CPU view
Max=0.5
– Autoscaling
Max=0.031
Max=0.06
53 of 33
To
© 2010 IBM
p
as
54
&
nm
54 of 33
v
© 2010 IBM
nmon More data saved for AIX6 TL7/AIX7 TL1
p
as
55
&
nm
on
In the BBBP section = AIX Parameters
Uptime long AIX has been running
– BBBP,000,uptime
– BBBP,001,uptime," 01:13AM up19:18,2 users, load average: 1.91, 1.54, 1.18“
lssrad –av SRAD and CPU-Memory Affinity
– BBBP,174,lssrad
– BBBP,175,lssrad,"REF1 SRAD MEM CPU"
– BBBP,176,lssrad,"0"
– BBBP,177,lssrad," 0 1699.37 0-7"
Next service pack: emgr -l
– For determining if or when eFixes are installed
I forget when added but it includes
– mpstat –b output = Memory Affinity stats Local, Near, Far references
– ifconfig & netstat – may have been there a while!!
55 of 33
To
© 2010 IBM
p
as
56
&
nm
Experimental
– DISKAVGRIO - Disk IO Average Reads per second
– DISKAVGWIO - Disk IO Average Writes per second
56 of 33
v
© 2010 IBM
p
as
57
&
nm
nmon More data saved for AIX6 TL7/AIX7 TL1
on
CPU PURR and SPURR based stats
– PCPU_ALL and PCPU<number 1 to logical processors>
– SCPU_ALL and SCPU<number 1 to logical processors>
To
© 2010 IBM
p
as
58
&
nm
topas_nmon PURR/SPURR
on
Note:
58 of 33
Goes over the VP. It has more clock ticks due to over clocking by ~9% - I think!!
v
© 2010 IBM
p
as
59
&
nm
Online Disks
on
nmon then
DDDD
New columns
59 of 33
To
© 2010 IBM
p
as
60
&
nm
nmon
on
60 of 33
v
© 2010 IBM
p
as
61
&
nm
topas
on
Hit Space to FREEZE the output
– if you want to study the numbers in detail
61 of 33