Using a Systematic Five-Step Tuning Method

Once upon a time, a database-tuning expert saw a drunken man on his hands and knees
searching for
something under a bright streetlight. �Have you lost something?� he asked the
drunk.
�My keys,� said the drunk.
The performance expert offered to help, and he too got on his hands and knees to
search for the lost
keys. After concluding that no keys were to be found under the streetlight, he
began questioning the drunk.
�When was the last time you remember seeing your keys?� he asked.
�At the bar,� said the drunk.
�Then why are you searching here?�
�Because it is so much brighter here!�
The story is fictional, and the moral is that you must concentrate your efforts in
the appropriate place.
You shouldn�t focus your exertion wherever it is most convenient to do so�the
problem might not be in the
Oracle database at all.
Here is an example of a case where poor performance was reported but the database
was not the
problem. The output shown in Listing 16-1 was produced by using the
dbms_monitor.session_trace:enable
procedure to trace a poorly performing batch process and then using the tkprof
utility to summarize the trace
data. The session was traced for about 40 minutes, and the data shows that the
Oracle database spent only
140.08 seconds executing SQL queries. The bulk of the time�2,138.67 seconds�was
spent waiting for further
instructions from the program. This clearly showed that the database was not the
problem.
Listing 16-1. Sample Output of the tkprof Utility
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 14114 0.37 0.40 0 0 0 0
Execute 78466 107.84 109.89 195 1105246 26768 13139
Fetch 72200 19.88 29.78 2100 432976 0 84080
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 164780 128.09 140.08 2295 1538222 26768 97219
Misses in library cache during parse: 4
Misses in library cache during execute: 4
Elapsed times include waiting on following events:
Event waited on Times Max. Wait Total Waited
Waited
---------------------------------------- ------ ---------- ------------
SQL*Net message to client 87649 0.00 0.09
SQL*Net message from client 87649 0.57 2138.67
db file sequential read 2295 0.26 10.80
SQL*Net break/reset to client 1084 0.00 0.07
log file sync 1813 0.15 18.02
latch: session allocation 8 0.29 2.14
latch: cache buffers chains 1 0.00 0.00
latch: library cache 3 0.00 0.01
log file switch completion 1 0.97 0.97
latch: redo writing 1 0.04 0.04
In his book The Art and Science of Oracle Performance Tuning, Christopher Lawson
outlines a
systematic five-step method for solving any performance-tuning problem:2
1. Define the problem. This requires patient listening, skillful questioning, and
even
careful observation. �The database is slow� is an example of a poorly defined
problem. �The database is slow between 10 a.m. and 11 a.m. every day� is more
precise. �This report now takes twice as long as it used to take only a week ago�
is another example of a precisely defined problem. Ask the user for the history of
the problem. Ask what previous efforts have been made to solve the problem. Ask
what changed recently in the environment�for example, software or hardware
upgrades. Ask whether all users are affected or only some. Ask whether the
problem occurs at specific times of the day or week. Ask whether all parts of the
application are equally affected or just certain parts.
2This was presented in Chapter 11 as a general method of solving any performance
problems; refer to the flowchart
shown in Figure 11-2.
2. Investigate the problem, and collect as much pertinent evidence as possible.
Examples include Statspack reports, graphs of database workload and DB time,
and session traces. Study the environment, and find out what may be affecting
the database�for example, other databases and applications may be competing
for resources with your database.
3. Analyze the data collected in the second step, and isolate the cause of the
performance problem. This is often the most challenging part of the
performance-tuning exercise. If the cause of the problem is not found, go back to
step 1 or step 2.
4. Solve the problem by creating a solution that addresses the root cause.
Solutions
are not always obvious, and, therefore, this step may require a great deal of
ingenuity and creativity.
5. Implement the solution in a safe and controlled manner. Conduct an appropriate
level of testing. Obtain �before� and �after� measurements, if possible, in order
to quantify the performance improvement. If the expected improvement is not
obtained, go back to step 2.
Oracle Database versions may change, and software tools may change, but the five
performance-tuning
steps never change. A problem may be simple and require only a few minutes of your
time, or it may be
tremendously complex and require weeks of your time, but the five steps always
remain the same. A simple
example follows; you will be asked to duplicate it in the �Exercises� section at
the end of the chapter:
1. The users told the DBA that the problem was restricted to certain parts of
the application and that there had not been a problem the previous day. On
questioning the application developers, the DBA learned that there had been an
application upgrade the previous night.
2. The DBA used the spreport.sql script to generate Statspack reports for a
one-hour period for the previous day and the current day. The DBA also traced
a number of database sessions by using the DBMS_MONITOR.SET_SQL_TRACE
procedure.
3. Examination of the Statspack report for the current day showed large numbers
of enq: TM - contention events; sessions were waiting, trying to lock an entire
table. These events were not observed in the Statspack report for the previous
day. The table in question was identified by searching through the trace files.
4. The enq: TM - contention event indicates an unindexed foreign key. If a
foreign key is not indexed, Oracle must lock the entire child table when a record
is deleted from the parent table in the foreign-key relationship. The DBA queried
the DBA_CONSTRAINTS, DBA_CONS_COLUMNS, and DBA_IND_COLUMNS views to
identify the missing index and brought this to the attention of the developers.
5. All that was left to do was to implement the solution in a safe and controlled
manner. The developers followed the necessary change-control procedures
before creating the necessary index.
Analyzing DB Time
The best way to analyze database performance is to find out where the database is
spending time. This is
powerfully summarized by the Statspack and AWR reports. To demonstrate this, I used
the Swingbench tool3
to perform large numbers of transactions in my test database on my laptop. I then
used the spreport.sql
and awrrpt.sql scripts to generate the Statspack and AWR reports for a one-hour
period; both scripts can
be found in ORACLE_HOME\rdbms\admin. Listing 16-2 shows the first page of the
Statspack report, and
Listing 16-3 shows the first page of the corresponding AWR report.
Listing 16-2. First Page from a Sample Statspack Report
STATSPACK report for
Database DB Id Instance Inst Num Startup Time Release RAC
~~~~~~~~ ---------- --------- -------- --------- ----- ---------- ---
1365223133 orcl 1 05-Jul-14 22:59 12.1.0.1.0 NO
Host Name Platform CPUs Cores Sockets Memory (G)
~~~~ ---------------- ---------------------- ----- ----- ------- ------------
localhost.locald Linux x86 64-bit 1 0 0 1.5
Snapshot Snap Id Snap Time Sessions Curs/Sess Comment
~~~~~~~~ ---------- ------------------ -------- --------- --------
Begin Snap: 111 18-Apr-15 14:00:15 2 4.5
End Snap: 171 18-Apr-15 15:00:10 3 4.3
Elapsed: 59.92 (mins) Av Act Sess: 0.9
DB time: 56.77 (mins) DB CPU: 1.30 (mins)
Cache Sizes Begin End
~~~~~~~~~~~ ------ ----------
Buffer Cache: 52M Std Block Size: 8K
Shared Pool: 172M Log Buffer: 3,912K
Load Profile Per Second Per Transaction Per Exec Per Call
~~~~~~~~~~~~ ------------------ ----------------- ----------- -----------
DB time(s): 1.0 0.1 0.01 0.02
DB CPU(s): 0.0 0.0 0.00 0.00
Redo size: 30,328.6 2,658.6
Logical reads: 1,520.9 133.3
Block changes: 288.9 25.3
Physical reads: 195.8 17.2
Physical writes: 0.1 0.0
User calls: 43.9 3.9
Parses: 30.6 2.7
Hard parses: 5.3 0.5
W/A MB processed: 0.3 0.0
Logons: 1.4 0.1
Executes: 153.6 13.5
Rollbacks: 0.0 0.0
Transactions: 11.4
Instance Efficiency Indicators
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 99.99
Buffer Hit %: 87.13 Optimal W/A Exec %: 100.00
Library Hit %: 93.56 Soft Parse %: 82.80
Execute to Parse %: 80.05 Latch Hit %: 100.00
Parse CPU to Parse Elapsd %: 6.93 % Non-Parse CPU: 82.42
Shared Pool Statistics Begin End
------ ------
Memory Usage %: 90.17 94.15
% SQL with executions>1: 79.38 80.00
% Memory for SQL w/exec>1: 74.63 79.55
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time
----------------------------------------- ------------ ----------- ------ ------
PL/SQL lock timer 143,791 1,398 10 37.3
db file sequential read 311,592 1,254 4 33.5
log file sync 20,819 684 33 18.2
write complete waits 212 208 979 5.5
CPU time 155 4.1
The most important part of the Statspack report is the Top 5 Timed Events section.
It is preceded by
details concerning the workload processed by the database. In Listing 16-2, you can
see that the DB time was
56.77 minutes during an elapsed interval of 59.92 minutes. DB time is the sum of
the response times of all
database calls made by the connected sessions; it can be much larger than the clock
time if multiple sessions
are simultaneously active. Note that actual CPU usage was only 55 seconds during
the entire period�just
4.1% of the total DB time. The rest of the time was spent waiting for overhead
activities such as reading data
from disk (db file sequential read operations) and writing redo information to the
online redo logs
(log file sync operations). Each db file sequential read operation retrieves one
database block from disk.
The average time for each operation was 4 milliseconds. The average time for each
log file sync operation
was 33 milliseconds�too large for comfort�but such operations accounted for only
18.2% of DB time.
Listing 16-3 shows the first page of the AWR report for the same period. The format
is very similar to that
of the Statspack report, but there is some variance in the numbers because the
methods used by AWR are
different than those of Statspack.
Listing 16-3. First Page from a Sample AWR Report
WORKLOAD REPOSITORY report for
DB Name DB Id Instance Inst Num Startup Time Release RAC
------------ ----------- ------------ -------- --------------- ----------- ---
ORCL 1365223133 orcl 1 05-Jul-14 23:00 12.1.0.1.0 NO
Host Name Platform CPUs Cores Sockets Memory(GB