Performance Monitoring on AIX

by Peter Lee IT Specialist

Performance Tuning Process
Indentify Workloads Define and prioritize goals Identify the required resoureces Minimize resource requirements Control the allocation of resource Apply additional resources as indicated

Performance Analysis Tools
CPU vmstat, iostat ps sar gprof/prof/tprof time/timex netpmon stem syscalls lockstat emstat performance toolbox trace, trcrpt, utld Memory Subsystem vmstat lsps svmon filemon bf, bfrpt I/O Subsystem iostat vmstat lsps lsattr lsdev lspv/lsls/lsvg fileplace filemon Network Subsystem lsattr netstat nfsstat entstat netpmon ifconfig iptrace/ipreport tcpdump

performance toolbox trace, trcrpt

performance toolbox trace, trcrpt

performance toolbox trace, trcrpt

Performance Tuning Tools

CPUs nice/renice schedtune bindprocessor chdev setpri

Memory Subsystem vmtune chps/mkps fdpr chdev rmss

I/O Subsystem vmtune chdev migratepv chlv reorgvg

Network Subsystem no nfso chdev ifconfig

Performance Tuning Flowchart
YES
Actions Is System CPU-Bound?

NO YES
Is System Memory-Bound?

NO

Is System Disk-Bound?

YES

Actions

NO YES
Is System Network-Bound?

Actions

NO

Actions

Additonal

Actions

vmstat Command vmstat Command Reports: Summary of paging and system events since initialization (-s) Number of forks since initilization (-f) Device interrupts since initilization or per interval (-i) Periodic reports of processes. paging. real memory.----------r b avm fre re pi po fr sr cy in sy cs us sy id wa 0 0 2514 24166 0 0 0 0 0 0 121 1 18 0 0 99 0 0 0 2514 24165 0 0 0 0 0 0 126 178 19 1 0 99 0 0 0 2514 24165 0 0 0 0 0 0 118 60 24 0 0 99 0 0 0 2514 24165 0 0 0 0 0 0 114 54 20 0 0 99 0 0 0 2514 24165 0 0 0 0 0 0 121 117 21 0 0 99 0 0 0 2514 24165 0 0 0 0 0 0 119 54 20 0 1 99 0 0 0 2514 24165 0 0 0 0 0 0 126 54 22 1 0 99 0 0 0 2514 24165 0 0 0 0 0 0 127 54 19 0 1 99 0 0 0 2514 24165 0 0 0 0 0 0 125 54 21 0 0 99 0 0 0 2514 24165 0 0 0 0 0 0 127 54 18 0 0 99 0 .----------.-----------------------.-----------. and CPU Usage : vmstat [ -ifs ] [drives] [interval] [count] vmstat 1 10 kthr memory page faults cpu ----.

vmstat Command root@bmserver[/tmp]>vmstat -i priority level 0 0 0 24 0 24 0 48 1 2 3 4 3 5 3 7 3 14 5 62 10 63 type hardware hardware hardware hardware hardware hardware hardware hardware hardware hardware hardware count module(handler) 0 i_misc_pwr(b0e5c) 0 i_epow(9d820) 0 /etc/drivers/hscsidd(18bc578) 0 i_scu(b0e50) 66 /etc/drivers/rsdd(196de4c) 3 /etc/drivers/middd_loadpin(19ce63c) 1736 /etc/drivers/hscsidd(18bc56c) 21636015 /etc/drivers/tokdd(19e78fc) 54189 /etc/drivers/hscsidd(18bc56c) 117295403 clock(9d670) 7647 i_softoff(9d628) .

faults page ins page outs paging space page ins paging space page outs total reclaims zero filled pages faults executable filled pages faults pages examined by clock revolutions of the clock hand pages freed by the clock backtracks lock misses free frame waits extend XPT waits pending I/O waits start I/Os iodones cpu context switches device interrupts software interrupts traps syscalls .vmstat Command root@bmserver[/tmp]>vmstat -s 210376 5724 55944 0 0 0 78419 1551 0 0 0 8790 0 0 0 3760 56326 56326 21347782 139338639 0 0 1873766 total address trans.

pi or po : the number of page ins or page outs per second from/to paging space during the sample interval fr/sr : the number of pages per second that were freed/examined by the page replacement algorithm during the interval cy : the number of cycles per second of the clock algorithm in/sy/cs : the number of device interrupts/system calls/thread context switches per second observed in the interval CPU : CPU utilization observed in the interval. This is not to be confused with the term repage which refers to a page which has already incurred a page-fault (the page could be currently on the freelist. If a page fault occurs and this page is currently on the freelist and has not yet been reused. this is considered a reclaim since no new I/O request has to be initiated.vmstat Command fre : the number of pages of real memory on the free list at the time of the interval sample re : the number of page reclaims per second observed in the sample interval. filesystem. or in paging space). id-idle. divided into us-user. sy-system. wa-wait disk xfer the number of disk requests per second observed to the PV .

topas Command .

iostat Command iostat Command Reports: CPU utilization is %user + %system approaching 100%? Terminal I/O Disk I/O Balanced? CD-ROM I/O Hardware Dependent Local disk information only No special authority required .

1 % idle 99.0 Kbps 0.0 Kb_read 40264 0 500 Kb_wrtn 220277 0 0 tin 0.0 % sys 0.2 0.8 .0 0.1 0.0 tps 0.iostat Command usage: iostat [-t] [-d] [drives] [interval [count]] -t specifies tty/cpu report only -d specifies drive report only -t and -d cannot both be specified root@bmserver[/tmp]>iostat tty: % iowait 0.5 avg-cpu: % user 0.1 Disks: hdisk0 hdisk1 cd0 % tm_act 0.0 0.0 0.0 0.0 tout 0.

only specified data types reported on Timing Intervals: the extraction interval can differ from the collection interval Permissions: need to be in system group sadc: back end for sar. responsible for the data collection timex -s: displays sar statistics for the duration of the command executed by timex .sar Command sar Command Data Types All data types collected.

the second form is used to process and display previously-captured data.g Collection: sar -o filename 60 300 > /dev/null & Extraction: sar -A -f filename Extraction: sar -u -f filename -s 10:20 -e 14:50 Monitoring: sar -c 60 300 . e. and/or save systemstatistics.sar Command There are two forms of the sar command: the first form is used to sample. display.

] | ALL] Show individual processor utilization on SMP systems .... process and file -w process switching activity -y reports on tty statistics -A Show all of the above -P [processor_id[.sar Command Data Types: -a file access system routines -b buffer activity -c system calls -k kernel process statistics -m message and semaphore activity -q statistics on run queue and wait queue -r paging statistics -u CPU utilization -v kernel tables: i-node.

00 0.0 100 1.00 0.0 100 Average 1.0 95 .0 100 19:31:52 2.sar Command # sar -u 5 3 19:28:02 %usr %sys %wio %idle 19:28:07 0 6 0 94 19:28:12 1 4 0 95 19:28:07 7 4 0 90 Average 3 4 0 93 # sar -c 1 3 19:28:25 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s 19:28:26 134 36 1 0.3 95 1.00 2691306 1517 19:28:27 46 34 1 0.00 2708329 1527 # sar -q 5 3 19:31:42 runq-sz %runocc swpq-sz %swpocc 19:31:47 1.00 2716922 1531 Average 75 35 1 0.00 0.00 2716922 1531 19:28:28 46 34 1 0.0 100 19:31:57 1.0 100 1.00 0.0 100 1.

If you see the -m option as well (-mo THREAD).ps Command ps Command The ps command writes the current status of active processes and (if the -m flag is given) associated kernel threads to standard output.. Usage : ps [-Aaedfklm] [-n namelist] [-F Format] [-o specifier[=header]. then you can see the thread or threads that belongs to each process.] [-p proclist][-G|-g grouplist] [-t termlist] [-U|-u userlist] Usage: ps [aceglnsuvwxU] [t tty] [processnumber] . A new option to ps in AIX Version 4 shows if a process is bound to a particular CPU.. This option is -o THREAD..

lockd install_as ps telnetd * 9ec0170 .ps Command root@bmserver[/tmp]>ps -el F S UID PID PPID C PRI NI ADDR 200003 A 0 1 0 0 60 20 2805 240001 A 0 2548 1 0 60 20 8831 40401 A 0 2814 1 0 60 20 1923 240001 A 0 2902 1 0 60 20 921 240001 A 0 3528 1 0 60 20 d21a 240001 A 0 3694 2902 0 60 20 59eb 240001 A 0 3968 2902 0 60 20 c9f9 240001 A 0 4218 2902 0 60 20 a1f4 240001 A 0 4468 2902 0 60 20 79ef 240001 A 0 4712 2902 0 60 20 9e1 240001 A 0 4998 2902 0 60 20 2204 240001 A 0 5866 8424 0 60 20 5acb 240001 A 0 6054 2902 0 60 20 8a11 240001 A 0 6326 2902 0 60 20 4208 240001 A 0 6598 2902 0 60 20 f21e 240001 A 0 6858 1 0 60 20 1a23 200001 A 0 6932 5866 11 65 20 7acf 200001 A 0 8424 4218 2 61 20 82d0 SZ 252 68 276 244 168 608 984 172 292 128 468 348 68 144 68 32 188 268 WCHAN 5a6cf58 95378 54aeccc TTY pts/0 pts/0 TIME 0:00 2:32 0:00 0:00 0:13 0:00 0:42 0:00 0:00 0:02 0:00 0:00 0:00 0:00 0:03 0:00 0:00 0:00 CMD init syncd errdemon srcmstr cron sendmail snmpd inetd portmap syslogd dpid2 ksh biod rpc.statd rpc.

7105 S 0 60 1 8400 root 6598 2902 .4233 S 0 60 1 8400 root 4218 2902 .A 0 60 1 240001 .statd /usr/sbin/rpc.A 0 60 1 240001 .A 0 60 1 5a6cf58 240001 .4989 S 0 60 1 8400 root 4712 2902 .3325 S 0 60 1 5a6cf58 400 root 2814 1 .3335 S 0 60 1 95378 400 root 2902 1 .lockd ps -emo THREAD - .4727 S 0 60 1 8400 root 3968 2902 .7897 S 0 60 1 2735dc 1400 root 7008 5866 .5519 S 0 60 1 8400 root 5866 8424 .9071 R 12 66 1 0 F pts/0 pts/0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 TT BND COMMAND /etc/init /usr/sbin/syncd 60 /usr/lib/errdemon /usr/sbin/srcmstr /usr/sbin/cron sendmail: accepting /usr/sbin/snmpd /usr/sbin/inetd /usr/sbin/portmap /usr/sbin/syslogd /usr/sbin/dpid2 -ksh /usr/sbin/biod 6 /usr/sbin/rpc.A 0 60 1 200003 463 S 0 60 1 400 root 2548 1 .4483 S 0 60 1 8400 root 4468 2902 .A 0 60 1 240001 .A 0 60 1 240001 .6833 S 0 60 1 400 root 6326 2902 .8185 S 0 60 1 400 root 6054 2902 .A 12 66 1 200001 .A 0 60 1 95378 40401 .A 0 60 3 * 240001 .2399 S 0 60 1 8400 root 3528 1 .5233 S 0 60 1 8400 root 4998 2902 .7639 S 0 60 1 54aa988 1400 .3795 S 0 60 1 54aeccc 400 root 3694 2902 .A 0 60 1 54aeccc 240001 .A 0 60 1 240001 .A 0 60 1 240001 .A 0 60 1 240001 .A 0 60 1 240001 .A 0 60 1 240001 .7377 S 0 60 1 9f64c98 400 .A 0 60 1 240001 connections .ps Command root@bmserver[/tmp]>ps -emo THREAD USER PID PPID TID ST CP PRI SC WCHAN root 1 0 .

0 348 root 6054 0.lock 06:01:57 0:01 telnetd 07:27:18 0:00 ps aux Aug 20 0:00 /usr/sbin/rpc.0 1.9 0.0 68 root 5264 0.ps Command root@bmserver[/tmp]>ps aux USER PID %CPU %MEM SZ root 5522 0.0 0.0 208 root 1032 0.0 468 root 7308 0.0 128 root 4998 0.0 8 root 2228 0.0 4 root 1 0.0 1.0 1.0 0.0 0.0 40 root 1896 0.0 1.0 292 root 4712 0.0 8 root 774 0.0 144 root 6858 0.0 0.0 268 root 7534 0.0 0.0 0.0 0.0 68 root 8424 0.0 0 RSS 16 408 96 12 360 172 548 12 108 324 232 196 44 252 48 12 12 12 8 304 4 TTY pts/0 pts/0 pts/0 STAT A A A A A A A A A A A A A A A A A A A A A STIME TIME COMMAND Aug 20 0:00 kproc 06:01:57 0:00 -ksh Aug 20 0:00 /usr/sbin/biod 6 Aug 20 0:01 kproc Aug 20 0:00 /usr/sbin/portmap Aug 20 0:02 /usr/sbin/syslogd Aug 20 0:00 /usr/sbin/dpid2 Aug 30 0:00 kproc Aug 20 0:03 /usr/sbin/rpc.0 1.0 8 root 4468 0.0 1.0 8 root 6598 0.0 1.0 188 root 6326 0.0 8 root 5866 0.stat Aug 20 0:00 /usr/sbin/install_ 07:27:18 0:00 more Aug 20 2:25 kproc Aug 20 0:00 kproc Aug 20 0:00 kproc Aug 20 0:25 kproc Aug 20 2:51 swapper Aug 20 0:00 /etc/init Aug 20 19067:57 kproc .0 2.0 0.0 32 root 7024 0.0 8 root 0 0.0 252 root 516 99.0 0.0 0.0 0.0 0.

emstat Command Instruction Emulation can be determined Different processors have different instruction sets PowerPC architecture deletes about 35 POWER instruc-tions To maintain binary compatibility. deleted instructions are emulated Instruction emulation increases the pathlength for the instruction Emulation can be avoided by compiling applications using Common Mode option .

adt.shipped with bos.samples Usage : /usr/samples/kernel/emstat [interval] [count] # /usr/samples/kernel/emstat 1 7 emstat total count emstat interval count 21951937 21951937 21967488 15551 21974877 7389 21994354 19477 22007329 12975 22018171 10842 22018171 0 .emstat Command New tool to detect instruction emulation .

Scheduling Policies SCHED_FIFO Non-preemptive scheduling Fixed piority SCHED_RR Round-Robin scheduling Fixed piority Run until interrupted or end of time slice SCHED_OTHER Default scheduling policy Same as SCHED_RR but applies to threads with non-fixed piorities Piority degrades with CPU Usage .

Process and Thread Piority Process Piorities High = 0 Real-Time Process Piorities 40 User Process Piorities setpri( ) Parent Process Inherited Process Piority Low = 127 Initial PRI = NICE + 40 pe -el (to view values) ps -emo THREAD (to view threads) nice renice fork() Child Process /Threads .

Syntax of renice: usage: renice [-n increment] [ -g| -p| -u] ID. Reducing nice value makespriority more favored.. Increasing nice value makes priority less favored. Nice value can be set at thread creation time by using the nice command.. Nice values are reduced by using negative increment values and increased by using positive increment values. Syntax of nice: usage: nice [-n increment] command usage: nice [-increment] command Nice value can be changed after thread has already been created using the renice command. NICE can range from 0-40. .nice/renice Commands Initial priority of non-fixed priority thread is: 40 + NICE where NICE has a default value of 20.

schedtune Command Modifies/Disables Memory Load Control Modifies CPU scheduling algorithms Modifies CPU time slices Restores defaults Must repeat at every boot .

Default timeslice is 1 clock tick and is tunable using /usr/samples/kernel/schedtune -t <# of ticks> .schedtune Command # /usr/samples/kernel/schedtune THRASH -p -m PROC MULTI 4 2 SUSP -w -e WAIT GRACE 1 2 FORK -f TICKS 10 SCHED -r SCHED_R 16 -h SYS 6 -d SCHED_D 16 -t TIMESLICE 1 -s MAXSPIN 1 Change the Scheduling policy from default to Round-Robin using the following command: /usr/samples/kernel/schedtune -r 0 -d 0 Each timeslice is in units of clock ticks (10ms).

bindprocessor Command bindprocessor Process [ ProcessorNum ] | -q | -u Process Purpose Binds or unbinds the kernel threads of a process to a processor. .

svmon Command svmon Command is an AIX Specific analysis tool for virtual memory Purpose: Captures a snapshot of the current state of memory.tools fileset. svmon is packaged as part of perfagent. . The display of information can be analyzed using four different reports: global [ -G] process [-P] segment [-S] detailed segment [-D] In AIX Version 4.

sidN ] [ -i interval [ numintervals ] ] [ -r ] # svmon -G -i 5 3 m e m o r y size inuse free 49152 48134 1018 49152 48139 1013 49152 48139 1013 i n u s e pin work pers clnt 2613 10674 37460 0 2613 10678 37461 0 2613 10678 37461 0 p i n work pers clnt 2613 0 0 2613 0 0 2613 0 0 p g s p a c e size inuse 98304 8104 98304 8105 98304 8105 ...svmon Command usage: svmon [ -G ] [ -P[n|s|a] [ pid1 .. pidN ] ] [ -P[n|s|a][u|p|g|r] [ count ] ] [ -S sid1 . sidN ] [ -S{[n|s|a][u|p|g|r]} [ count ] ] [ -D sid1 ....

To sort system segments by the number of pages in real memory. repeated 10 times at 30 second intervals: $ svmon -G -i 30 10 . To print out the global statistics.svmon Command 1. To print out the memory usage statistics for all processes (limit statistics to non-system segments): $ svmon -Pn 3. To print out the memory usage statistics for processes 6746 and 10078 (include statistics on all segments): $ svmon -Pa 6746 10078 4. and print out the top 10 segments of the resulting list: $ svmon -Ssu 10 5. To print detailed memory usage statistics on segment 340d: $ svmon -D 340d 6. To print out global statistics: $ svmon -G 2.

rmss Command Reduced Memory System Simulator Simulate RISC System/6000 with various size memories Run an application over a range of memory sizes rmss is packaged as part of perfagent.tools usage: rmss [-s startmemsize] [-f finalmemsize] [-d deltamemsize] [-n numiterations] [-o outputfile] command rmss -c memsize rmss -r rmss -p .

out rmss -s 32 -f 8 -d 8 -n 1 -o cc.rmss. run and measure the command once at each memory size and write the report to the file cc.rmss Command To change memory size # rmss -c 32 Simulated memory size changed to 32 Mb. To reset the memory size to the real size: rmss -r Simulated memory size changed to 192 Mb. To display the current memory size: # rmss -p Simulated memory size is 32 Mb.c . To measure the performance of the command cc -O foo. 24. 8MB.out cc -O foo.c on memory size 32.rmss. 16.

filemon will only collect data for those files opened after filemon was started unless you specify the -u flag.filemon Command The filemon tool collects and presents trace data on the various layers of file system utilization including the logical file system. and physical layers. LVM. The default is to collect data on the VM. Data can be collected on all the layers. only the top 20 logical files and segments are reported unless the -v (verbose) flag is used. Normally. and physical disk layers. . virtual memory segments. or layers can be specified with the -O layer option. LVM.

lv.vm.all lf: monitor logical file I/O vm: monitor virtual memory I/O lv: monitor logical volume I/O pv: monitor physical volume I/O all: short for lf.vm.pv.lv.pv (default is: vm.lv.pv) .filemon Command Usage: filemon [-i file] [-o file] [-d] [-v] [-u] [-O opt] -i file: open input file (default is real-time trace) -o file: open output file (default is stdout) -d: deferred trace (until ‘ trcon’ ) -T num: set trace kernel buf sz (default 32000 bytes) -P: pin monitor process in memory -v: verbose mode (print extra details) -u: print unnamed file activity via pid -O opt: other monitor-specific options valid -O options: lf.

450 secs in measured interval Cpu utilization: 72.7 /dev/lv04 /tstc Compressed: Frag_Sz.=1024 0.36 0 1588 145.out # dd if=/dev/rhdisk1 of=/dev/null bs=512 count=1024 # cp /unix /tstc/junk # trcstop # more fmon.5 /dev/loglv00 jfslog Most Active Physical Volumes -----------------------------------------------------------------------util #rblk #wblk KB/s volume description -----------------------------------------------------------------------0.75 1024 1604 241.out Fri Aug 2 15:11:21 1996 System: AIX magic Node: 4 Machine: 000001455C00 5.6% Most Active Logical Volumes -----------------------------------------------------------------------util #rblk #wblk KB/s volume description -----------------------------------------------------------------------0.00 0 16 1.filemon Command # filemon -O lv.1 /dev/hdisk1 .pv -o fmon.

netstat Command netstat Command Shows state of interfaces Shows interface statistics Shows routing statistics Shows per-protocol statistics Shows network device driver statistics Shows network memory usage statistics .

c9.0.a4 tr0 1492 9.64 bmserver Ipkts Ierrs 473 0 473 0 10832411 0 10832411 0 Opkts Oerrs 473 0 473 0 22191 0 22191 0 Coll 0 0 0 0 root@bmserver[/tmp]>netstat -I tr0 1 input (tr0) output input (Total) output packets errs packets errs colls packets errs packets errs colls 10832898 0 22267 0 0 10833371 0 22740 0 0 3 0 1 0 0 3 0 1 0 0 7 0 1 0 0 7 0 1 0 0 4 0 1 0 0 4 0 1 0 0 5 0 1 0 0 5 0 1 0 0 10 0 1 0 0 10 0 1 0 0 7 0 1 0 0 7 0 1 0 0 6 0 1 0 0 6 0 1 0 0 11 0 1 0 0 11 0 1 0 0 10 0 1 0 0 10 0 1 0 0 9 0 1 0 0 9 0 1 0 0 4 0 1 0 0 4 0 1 0 0 9 0 1 0 0 9 0 1 0 0 7 0 1 0 0 7 0 1 0 0 7 0 1 0 0 7 0 1 0 0 .netstat Command root@bmserver[/tmp]>netstat -i Name Mtu Network Address lo0 16896 <Link> lo0 16896 127 loopback tr0 1492 <Link>10.184.5a.62.

netstat Command root@bmserver[/tmp]>netstat -m 34 mbufs in use: 16 mbuf cluster pages in use 72 Kbytes allocated to mbufs 0 requests for mbufs denied 0 calls to protocol drain routines Kernel malloc statistics: ******* CPU 0 ******* By size inuse calls failed 32 108 115 0 64 122 765 0 128 111 1666 0 256 134 21771388 0 512 37 2470 0 1024 1 481 0 2048 0 773 0 4096 21 1510233 0 8192 0 66 0 16384 1 6 0 By type mbuf mcluster socket pcb routetbl ifaddr mblk mblkdata strhead strqueue strmodsw strosr strsyncq streams kernel table temp inuse 34 16 111 78 8 5 30 3 13 21 21 0 29 148 14 5 calls failed 21747482 0 1512910 0 698 0 427 0 13 0 5 0 1198 0 1070 0 57 0 132 0 21 0 23261 0 388 0 275 0 15 0 12 0 free 20 70 17 42 11 3 6 6 2 19 memuse 8704 65536 18048 12032 1312 576 6272 12288 3776 10752 1344 0 2944 16000 45920 5312 hiwat 640 320 160 384 40 20 10 120 10 24 memmax 17920 95744 19008 12864 2080 576 6528 22528 4800 13312 1344 768 3648 17280 46432 9920 freed 0 0 0 0 0 0 0 0 0 7 mapb 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Streams mblk statistic failures: 0 high priority mblk failures 0 medium priority mblk failures 0 low priority mblk failures .

netstat Command root@bmserver[/tmp]>netstat -D Source Ipkts Opkts Idrops Odrops ------------------------------------------------------------------------------ent_dev0 0 0 0 0 tok_dev0 21755956 24738 58384 0 --------------------------------------------------------------Devices Total 21755956 24738 58384 0 ------------------------------------------------------------------------------ent_dd0 0 0 0 0 tok_dd0 21697572 24738 0 0 --------------------------------------------------------------Drivers Total 21697572 24738 0 0 ------------------------------------------------------------------------------ent_dmx0 0 N/A 0 N/A tok_dmx0 10836721 N/A 10860851 N/A --------------------------------------------------------------Demuxer Total 10836721 N/A 10860851 N/A ------------------------------------------------------------------------------IP 5171634 2654588 18061 18061 TCP 26292 21120 0 0 UDP 2628010 473 2 2370 --------------------------------------------------------------Protocols Total 7825936 2676181 18063 20431 ------------------------------------------------------------------------------lo_if0 473 473 0 0 tr_if0 10834304 22321 0 7 --------------------------------------------------------------Net IF Total 10834777 22794 0 7 --------------------------------------------------------------------------------------------------------------------------------------------NFS/RPC Total N/A 12 0 0 ------------------------------------------------------------------------------(Note: N/A -> Not Applicable) .

netstat Command root@bmserver[/tmp]>netstat -s ip: 1079372 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 266328 packets for this host 13 packets for unknown/unsupported protocol 0 packets forwarded 14007 packets not forwardable 0 redirects sent 14202 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs. etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 799024 IP Multicast packets dropped due to no receiver 37 successful path MTU discovery cycles 8 path MTU rediscovery cycles attempted 0 path MTU discovery no-response estimates 1 path MTU discovery response timeout 0 path MTU discovery decreases detected 39 path MTU discovery packets sent 0 path MTU discovery memory allocation failures 0 ipintrq overflows 0 with illegal source 0 packets processed by threads 0 packets dropped by threads 0 packets dropped due to the full socket receive buffer 0 dead gateway detection packets sent 0 dead gateway detection packet allocation failures 0 dead gateway detection gateway allocation failures icmp: 4559 calls to icmp_error 0 errors not generated because old message was icmp Output histogram: echo reply: 6 destination unreachable: 25 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: echo reply: 37 destination unreachable: 13 echo: 6 6 message responses generated igmp: 1 0 0 0 0 1 1 0 4 message received messages received with too few bytes messages received with bad checksum membership queries received membership queries received with invalid field(s) membership report received membership report received with invalid field(s) membership reports received for groups to which we belong membership reports sent .

data (1572 bytes duped) 307 out-of-order packets (4588 bytes) 0 packets (0 bytes) of data after window 0 window probes 262 window update packets 0 packets received after close 0 packets with bad hardware assisted checksum 2 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded by listeners 579 ack packet headers correctly predicted 3412 data packet headers correctly predicted 10 connection requests 311 connection accepts 316 connections established (including accepts) 318 connections closed (including 5 drops) 0 connections with ECN capability 0 times responded to ECN 5 embryonic connections dropped 11180 segments updated rtt (of 11262 attempts) 0 segments with congestion window reduced bit set 0 segments with congestion experienced bit set 0 resends due to path MTU discovery 0 path MTU discovery terminations due to retransmits 86 retransmit timeouts 0 connections dropped by rexmit timeout 2 fast retransmits 0 when congestion window less than 4 segments 0 newreno retransmits 0 times avoided false fast retransmits 6 persist timeouts 0 connections dropped due to persist timeout 149 keepalive timeouts 149 keepalive probes sent 0 connections dropped by keepalive 0 times SACK blocks array is extended 0 times SACK holes array is extended 0 packets dropped due to memory allocation failure 0 connections in timewait reused 0 delayed ACKs for SYN 0 delayed ACKs for FIN 0 send_and_disconnects 0 spliced connections 0 spliced connections closed 0 spliced connections reset 0 spliced connections timeout 0 spliced connections persist timeout 0 spliced connections keepalive timeout udp: 250354 datagrams received 0 incomplete headers 0 bad data length fields 0 bad checksums 4559 dropped due to no socket 245436 broadcast/multicast datagrams dropped due to no socket 0 socket buffer overflows 359 delivered 381 datagrams output .netstat Command tcp: 13593 packets sent 11369 data packets (4102411 bytes) 89 data packets (108779 bytes) retransmitted 1842 ack-only packets (1062 delayed) 0 URG only packets 0 window probe packets 1 window update packet 292 control packets 15930 packets received 11390 acks (for 4104523 bytes) 217 duplicate acks 0 acks for unsent data 9910 packets (154901 bytes) received in-sequence 195 completely duplicate packets (592 bytes) 0 old duplicate packets 5 packets with some dup.

This flag can only be issued by privileged users. -r Resets all the statistics back to their initial values. including the device-specific statistics. -t Toggles debug trace in some device drivers. This command is also invoked when the netstat command is run with the -v flag.entstat Command The entstat command displays the statistics gathered by the specified Ethernet device driver. . usage: entstat [-drt] <device name> -d Displays all the statistics. The netstat command does not issue any entstat command flags.

entstat Command root@bmserver[/tmp]>entstat -d ent0 ------------------------------------------------------------ETHERNET STATISTICS (ent0) : Device Type: IBM 10/100 Mbps Ethernet PCI Adapter (23100020) Hardware Address: 00:04:ac:17:74:a0 Elapsed Time: 4 days 18 hours 8 minutes 43 seconds Transmit Statistics: Statistics: -------------------------------------Packets: 0 Bytes: 0 Interrupts: 0 Transmit Errors: 6 Packets Dropped: 1 0 Max Packets on S/W Transmit Queue: 2 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0 Broadcast Packets: 0 Packets: 0 Multicast Packets: 0 Packets: 0 No Carrier Sense: 6 DMA Underrun: 0 Lost CTS Errors: 0 0 Max Collision Errors: 0 Errors: 0 Late Collision Errors: 0 Errors: 0 Deferred: 0 Errors: 0 SQE Test: 0 Errors: 0 Timeout Errors: 0 by Adapter: 0 Single Collision Count: 0 Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0 Broadcast Multicast CRC Errors: 0 DMA Overrun: 0 Alignment Errors: No Resource Receive Collision Packet Too Short Packet Too Long Packets Discarded Receiver Start Receive General Statistics: ------------------No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 200 Driver Flags: Up Broadcast Running Simplex AlternateAddress 64BitSupport PrivateSegment DataRateSet IBM 10/100 Mbps Ethernet PCI Adapter Specific Statistics: -----------------------------------------------Chip Version: 26 RJ45 Port Link Status : down Media Speed Selected: Auto negotiation Media Speed Running: Unknown Receive Pool Buffer Size: 384 Free Receive Pool Buffers: 128 No Receive Pool Buffer Errors: 0 Inter Packet Gap: 96 Adapter Restarts due to IOCTL commands: 1 Packets with Transmit collisions: 1 collisions: 0 6 collisions: 0 11 collisions: 0 2 collisions: 0 7 collisions: 0 12 collisions: 0 3 collisions: 0 8 collisions: 0 13 collisions: 0 4 collisions: 0 9 collisions: 0 14 collisions: 0 5 collisions: 0 10 collisions: 0 15 collisions: 0 Excessive deferral errors: 0x0 Packets: 0 Bytes: 0 Interrupts: 0 Receive Errors: 0 Packets Dropped: Bad Packets: 0 .

NFS Tuning Concepts NFS Version 3 or Version 2 (Version 3 is recommended) UDP or TCP protocol (NOTE: TCP is recommended) Increase/decrease number of biods/nfsds nfs socket size and sb_max adapter transmit queues Filesystem read/write buffer sizes RAM for caching file pages (vmtune -P) I/O tuning (multiple disks. separate log disk) Tuning with nfso .

nfsstat Command Displays statistics on: Network File System (NFS) Remote Procedure Calls (RPC) Syntax ./usr/sbin/nfsstat: -c client information -s server information -n NFS information only -r RPC information only -z reset statistics (root only) -m mount statistics default: -csnr .

nfsstat Command root@bmserver[/tmp]>nfsstat -rc Client rpc: Connection oriented: calls badcalls 1 0 nomem cantconn 0 0 Connectionless: calls badcalls 1448 0 timers nomem 22 0 badxids timeouts 0 0 interrupts 0 retrans 12 cantsend 0 badxids 0 newcreds 0 badverfs 0 timers 0 timeouts 12 newcreds 0 badverfs 0 .

cur=4 (80ms) All: srtt=15 (37ms). rsize=32768.intr.intr.ret rans=5 Lookups: srtt=2 (5ms).dynamic.hard.rsize=8192.soft.proto=udp.rsize=8192. cur=5 (100ms) /nfs/cust from /nfs/cust:zen Flags: vers=3.dynamic.wsize=8192. cur=4 (80ms) Reads: srtt=18 (45ms). dev=5 (25ms).proto=udp.auth=unix. dev=7 (35ms).wsize=8192. cur=1 (20ms) All: srtt=2 (5ms).nfsstat Command # nfsstat -m /nfs/retain/pmrs from /nfs/retain/pmrs:cia Flags: vers=2.wsize=32 768.proto=tcp.ret rans=5 Lookups: srtt=14 (35ms).auth=unix. dev=6 (30ms).retrans=5 All: srtt=0 (0ms).link.auth=unix. dev=2400 (12000ms).soft.intr.sym-link. dev=2 (10ms). dev=2400 (12000ms). cur=600 (12000ms) /nfs/retain/bin from /nfs/retain/bin:cia Flags: vers=2. cur=600 (12000ms) .

nfsstat Command # netstat -p udp udp: 27136572 datagrams received 0 incomplete headers 0 bad data length fields 0 bad checksums 20 dropped due to no socket 8 broadcast/multicast datagrams dropped due to no socket 39476 socket buffer overflows 27097068 delivered 177636 datagrams output socket buffer overflows = Too few nfsd daemons and/or too small nfs_socketsize Increase the number of nfsd daemons (NFS V2) Increase value of nfs_socketsize and sb_max .

nfso Command nfso Command # nfso -a portcheck= 0 udpchecksum= 1 nfs_socketsize= 60000 nfs_tcp_socketsize= 60000 nfs_setattr_error= 0 nfs_gather_threshold= 4096 nfs_repeat_messages= 0 nfs_udp_duplicate_cache_size= 0 nfs_tcp_duplicate_cache_size= 0 nfs_server_base_priority= 0 nfs_dynamic_retrans= 1 nfs_iopace_pages= 0 nfs_max_connections= 0 nfs_max_threads= 8 nfs_use_reserved_ports= 0 nfs_device_specific_bufs= 1 nfs_server_clread= 1 nfs_rfc1323= 0 nfs_max_write_size= 0 nfs_max_read_size= 0 nfs_allow_all_signals= 0 .

Sign up to vote on this title
UsefulNot useful