Relevance of High %iowait in Server Performance
High
%iowait
has historically indicated a problem in I/O performance. However, due toadvances in CPU performance, high
%iowait
may be a misleading indicator, especially inrandom I/O workloads. It's misleading because
%iowait
measures CPU performance, notI/O. To be precise,
%iowait
measures the percent of time the CPU is idle, but waiting for an I/O to complete. As such, it is only indirectly related to I/O performance, which canresult in false conclusions. It is possible to have healthy system with nearly 100% iowait,or have a disk bottleneck with 0% iowait.High
%iowait
is becoming more common as processor speeds increase. Gains in processor performance have significantly outpaced disk performance. While processor performance has doubled every 12 to 18 months, disk performance (in IOPS per disk) hasremained relatively constant* . This imbalance has resulted in a trend toward higher
%iowait
on healthy systems. (* IOPS depend on seek time, which hasn't increased at therate of processor performance. The improvements in storage have been in areas such asareal density (MB/sq. in.) and rotational speed.)The following example illustrates how faster CPU's can increase
%iowait
. Assume weupgrade a system with CPU's that are 4 times faster. All else remains unchanged. Beforethe upgrade, a transaction takes 60 ms which includes 40 ms of CPU time plus 20 ms to perform an IO, and that our application performs one transaction after another in a serialstream.
Before CPU Upgrade
CPU time = 40 msIO time = 20 msTotal transaction time = CPU + IO = 40 + 20 = 60 ms
%iowait
= IO time/total time = 20/60 = 33%
After CPU Upgrade
CPU time = 40 ms/ 4 = 10 msIO time = 20 msTotal transaction time = CPU + IO = 10 + 20 = 30 ms
%iowait
= 20/30 = 66%In this example, transaction performance doubled, despite a 2x increase in
%iowait
. Inthis case, the absolute value of
%iowait
is a misleading indicator of an I/O problems.So, how do you identify an I/O problem if you can't rely on
%iowait
? The best way is tomeasure I/O response times using filemon. As a rule of thumb, read/write time shouldaverage 15-20 ms on non-cached disk subsystems. On cached disk subsystems, readsshould average 5-20 ms, and writes should average 2-3 ms. Higher response timesindicate the storage subsystem is possibly overloaded.Here's an example of a 90 second filemon trace from an actual customer system that washeavily utilized. The filemon command was:
# filemon -o /tmp/filemon.out -O lv,pv -T 320000; sleep 90; trcstopThe output is in /tmp/filemon.out. From the Detailed Physical Volume Section:
VOLUME: /dev/hdisk60 description: EMC Symmetrix FCP Raid1reads: 9217 (0 errs)read sizes (blks): avg 71.8 min 8 max 256 sdev 93.4 read times (msec):avg 61.515 min 0.011 max 1643.486 sdev 130.135read sequences: 6249read seq. lengths:avg 105.9 min 8 max 3920 sdev 309.8writes: 7023 (0 errs)write sizes (blks): avg 43.0 min 8 max 256 sdev 37.6 write times (msec):avg 40.651 min 0.003 max 1544.865 sdev 88.734write sequences: 6939write seq. lengths:avg 76.6 min 8 max 1696 sdev 88.9seeks: 10188(62.7%)seek dist (blks):init 0, avg 16784566.3 min 8 max 78792992sdev 19185871.9seek dist (%tot blks):init 0.00000, avg 17.80295 min 0.00001 max83.57367 sdev 20.34995time to next req(msec): avg 22.074 min 0.006 max 2042.710 sdev54.050throughput: 1598.1 KB/secutilization: 0.73
The average read and write service times are highlighted: the average read time for hdisk60 was 61.515 ms, and the average write time was 40.651 ms. In this case, we havea disk bottleneck. High IO service times are typically due to overloaded disks in the disk subsystem (i.e. we're sending more IOPS than the disks can handle), an overloaded processor in the disk subsystem, or bottlenecks or problems in the interconnect to thedisk.Here are some alternatives to alleviate this problem:
•
Tune AIX: Use asynch I/O, read larger blocks (vmtune- maxpgahead), etc
•
Reduce the number of IOs to the disk subsystem. Increase memory for datacaching, or by use a RAM filesystem.
of 00
Iowa It
207 Reads
Info and Rating
| Category: | Uncategorized. |
| Rating: | |
| Upload Date: | 11/18/2009 |
| Copyright: | Attribution Non-commercial |
| Tags: |
Leave a Comment