7

Performance Diagnostics

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Objectives

After completing this lesson, you should be able to:

• Boot into rescue mode to repair a boot problem
• Describe frequently encountered performance problems in
Oracle Linux
• Describe performance tuning kernel parameters
• Use system resource reporting utilities
• Use OS Watcher Black Box (OSWbb)
• Use OS Watcher Black Box Analyzer (OSWbba)
• Monitor system latency using latencytop
• Trace system calls and signals using the strace utility

7-2 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
 Rescue Mode

• Boot into rescue mode to correct boot problems.

• Rescue mode boots from installation media and allows
access to system hard drive.
• Enter rescue mode by typing the following from the boot
prompt:
boot: linux rescue
• Provide language, keyboard type, media, and network (if
needed).
• File systems are mounted under /mnt/sysimage. To
change the root partition of the rescue mode environment:
# chroot /mnt/sysimage
• Use vi, fsck, rpm, grub-install, and other utilities to
fix the boot problem.

7-3 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
 Frequently Encountered Performance Problems

• Out-of-memory errors, slow performance

– Cause: SGA mapped in 4 KB pages instead of 2 MB
– Solution: Use HugePages.
• System slowing down when many console messages are
written to the logging buffer
– Cause: Slow baud rate to serial port
– Solution: Increase baud rate for serial console in
grub.conf.
kernel /vmlinuz-2.6.36-200.24.1.el6uek.x86_64 ro
root=/dev/mapper/vg_host... console=ttyS0,115200
console=tty debug

• Slow performance at 95% RAM

– Cause: Missing shrink_zone patch (with OL 5.3 or 5.5)
– Solution: Install Unbreakable Enterprise Kernel.

7-5 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
 Performance Tuning Kernel Parameters

• Many parameters are application-specific:

– Use validated configurations.
– See release notes for recommendations.
• Parameters that are almost always set:
– TCP/IP read buffer size (net.core.rmem_max)
– TCP/IP write buffer size (net.core.wmem_max)
– File descriptors (fs.file-max)
• Parameters that are sometimes useful:
– Swap tendency (vm.swappiness)
– Memory conditions (vm.panic_on_oom)

7-7 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
 System Resource Reporting Utilities

Tools for collecting system resource usage and errors:

• mpstat – Reports CPU statistics
• sar – Reports system activity information
• dmesg – Reports physical resource errors
• dstat – Reports system resource statistics
• vmstat – Reports virtual memory statistics
• free – Reports amount of free and used memory
• top – Provides a real-time view of running Linux processes
• ip – Reports network interface statistics and errors
• ifconfig – Reports network interface statistics and errors
• netstat – Reports network interface statistics and errors
• iostat – Reports I/O statistics for devices and partitions
• iotop – Monitors swap and disk I/O on a per-process basis

7-8 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
 CPU Reporting Utilities

• Display CPU usage:

# mpstat -P ALL 1
# sar -P ALL
# sar -u

– Observe the %idle value.

– Low idle value indicates high CPU usage.
• Display queue length and load averages:
# sar -q

– Observe the runq-sz value.

– High run queue value is an indication of CPU saturation.

7 - 10 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Memory Reporting Utilities

• Display memory utilization statistics:

# sar -r
– Observe the %memused value.
• Display memory paging statistics:
# sar -B
– Observe the pgscank and pgscand values.
• Display swapping statistics:
# sar -W
– Observe the pswpin/s and pswpout/s values.
• Display physical memory failures:
# dmesg

7 - 11 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Using the dstat Utility

• Display CPU utilization statistics:

# dstat -c
– Observe the idl value.
• Display process statistics:
# dstat -p
– Observe the run value.
• Display memory statistics:
# dstat -m
– Observe the free value.
• Display file system statistics:
# dstat --fs
# sar -v
– Observe the files and file-nr values, respectively.

7 - 13 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Using vmstat and free Utilities

• Display CPU statistics:

# vmstat 1
– Observe the id value in the cpu column.
– Observe the r value in the proc column.
• Display memory statistics:
# vmstat 1
– Observe the free value in the memory column.
– Observe the si and so values in the memory column.
• Display the amount of free and used memory:
# free -m
– The -m option displays amounts in megabytes.

7 - 14 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Using the top Utility

• The top command provides an ongoing look at processor

activity in real time:
# top
• The upper section displays:
– Load averages during the last 1-, 5-, and 15-minute periods
– Number of running and sleeping tasks
– Overall CPU and memory usage
• The lower section displays a sorted list of processes:
– By default, processes are sorted by CPU usage (%CPU).
• To sort by memory usage:
– Press F or O to change the sort field.
– Select either n: %MEM or Q: RES.

7 - 15 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Reporting Top Consumers of System Resources

• The dstat command reports the “top” consumers of

system resources.
• Display top consumers of multiple resources:
# dstat --top-cpu --top-bio --top-mem –top-latency
• Some of the available --top options are listed as follows:
– --top-bio – Show the most expensive block I/O process.
– --top-cpu – Show the most expensive CPU process.
– --top-io – Show the most expensive I/O process.
– --top-latency – Show the process with the highest total
latency in milliseconds.
– --top-mem – Show the process that is using the most
memory.
– --top-oom – Show the process to be killed first by OOM.

7 - 17 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Network Reporting Utilities

• Display network statistics:

# ip –s link
– Observe the TX bytes and RX bytes values.
– Observe the dropped, overrun, and errors values.
• Display network statistics:
# ifconfig
– Observe dropped, overrun, and errors values.
• Display summary statistics for each protocol:
# netstat -s
– Observe the segments retransmitted value.
• Display network interface errors (RX-ERR and TX-ERR):
# netstat -i

7 - 18 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 I/O Reporting Utilities

• Display I/O statistics for devices:

# iostat –xz 1
– Observe the %util value.
– Observe the avgqu-sz value.
• Display NFS statistics (-n):
# iostat –xnz 1
– Includes statistics for NFS-mounted file systems
• Display I/O statistics for devices:
# sar -d
– Observe the %util value.
– Observe the avgqu-sz value.

7 - 19 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Using the iotop Utility

• The iotop command monitors swap and disk I/O on a

per-process basis:
# iotop
• The upper section displays the sum of the DISK READ and
DISK WRITE bandwidth in B/s (bytes per second).
• The lower section displays the following:
– DISK READ – I/O bandwidth read by each process or thread
– DISK WRITE – I/O bandwidth written by each process or
thread
– SWAPIN – Percentage of time the thread or process spent
while swapping in
– IO – Percentage of time the thread or process spent while
waiting on I/O
– COMMAND – The name of the process
7 - 20 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
 System Monitor GUI

7 - 21 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 OS Watcher Black Box (OSWbb)

• OSWbb collects and archives operating system and

network metrics to aid in diagnosing performance issues.
• OSWbb includes a built-in analyzer called OSWbba.
• Download the OSWbb tar file from My Oracle Support
(MOS).
• To install OSWbb, use the tar command:
# tar xvf oswbb403.tar
• To start OSWbb, use the following command:
# ./startOSWbb.sh
• To stop OSWbb, use the following command:
# ./stopOSWbb.sh
• Diagnostic data output is stored in the archive
subdirectory.
7 - 22 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
 OS Watcher Black Box Analyzer (OSWbba)

• OSWbba is a graphing and analysis utility that is included

with OSWbb v4.0.0 and higher.
• OSWbba graphically displays data collected, and
generates reports.
• OSWbba includes a built in analyzer to provide details on
performance problems.
– The ability to create a graph and analyze this information
relieves you of manually inspecting all the files.
• To start OSWbba, use the following command:
# java -jar oswbba.jar -i ~/oswbb/archive
• The OSWbba menu provides options to graph and analyze
the collected data.

7 - 24 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Analyze OSWbb Archive Files

• Start the analyzer in the directory where OSWbba is

installed.
• Select Option A from the OSWbba menu.
• You can also run the analyzer from the command line:
# java -jar oswbba.jar -i ~/oswbb/archive -A
• The analyzer output is divided into six sections:
– Section 1: Overall Status
– Section 2: System Slowdown Summary Ordered By Impact
– Section 3: Other General Findings
– Section 4: CPU Detailed Findings
– Section 5: Memory Detailed Findings
– Section 6: Disk Detailed Findings

7 - 27 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Monitoring System Latency: latencytop

7 - 29 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 strace Utility

• Use strace to trace system calls and signals:

# strace ls
execve("/bin/ls", ["ls"], [/* 29 vars */]) = 0
brk(0) = 0xf47000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
MAP_ANONYMOUS, -1, 0) = 0x7f4e1a7b5000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No
such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=32876, ...}) = 0
...
• Input and output arguments are surrounded by ().
• The return value is displayed after the = sign.
• The ltrace command is similar to strace, except it is a
library call tracer.

7 - 30 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Quiz

Which of the following statements are true?

a. The vmstat command is primarily used to monitor your
system’s memory usage, but the command output contains
more than just memory statistics.
b. The sar command, by default, produces system activity
information from the current daily data file.
c. The top utility is a Python program that is used for
monitoring swap and disk I/O on a per-process basis.
d. OS Watcher Black Box (OSWbb) is a collection of UNIX
shell scripts that can continuously collect and archive
operating system and network metrics to help diagnose
performance issues.

7 - 32 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Summary

In this lesson, you should have learned how to use the

following:
• Booting into rescue mode to repair boot problems
• System resource reporting utilities
• OS Watcher Black Box (OSWbb)
• OS Watcher Black Box Analyzer (OSWbba)
• strace utility

7 - 33 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

 Practice 7: Overview

This practice covers the following topics:

• Using rescue mode to repair a boot problem
• Using standard Linux utilities to monitor system resources
• Installing OSWbb and collecting data
• Analyzing OSWbb data using OSWbba

7 - 34 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.