Unit1 Intro To Waits

Waits Defined
Kyle Hailey http://perfvision.com
Top 36 Foreground Waits

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. db file sequential read log file sync db file scattered read latch free enqueue SQL*Net more data from client direct path read direct path write buffer busy waits SQL*Net more data to client log buffer space log file switch completion library cache pin SQL*Net break/reset to client io done file open free buffer waits db file parallel read 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. write complete waits library cache lock SQL*Net more data from dblink log file switch (checkpoint incomplete) library cache load lock row cache lock local write wait sort segment request process startup unread message file identify pipe put switch logfile command SQL*Net break/reset to dblink log file switch (archiving needed) Wait for a undo record direct path write (lob) undo segment extension
#.2
minus OPS, RAC, PQ, Resource Mgr
http://www.oraperf.com
Copyright 2006 Kyle Hailey
#.3
Tuning Methodology
Machine
Run
queue (CPU)
reduce CPU usage or add CPUs Reduce memory usage or add memory
Paging
Oracle
Waits >> CPU
Tune waits
We are going to concentrate here on WAITS

#.4
For Analysis
CPU Time
For
comparision with Wait Time
Wait Time Top Waits

If
Wait Time >> CPU Time, need top waits detailed analysis of some waits detailed analysis of some waits
Wait Arguments p1, p2, p3

For
SQL that caused waits

For
#.5
Wait and CPU Data Sources

I Statspack
Top 5 Timed Events
II 10g ASH
OEM ASH Report : ashrpt.sql Custom queries on v$active_session_history
III V$session & v$session_wait

Custom queries
#.6
I Statspack
Look at Top 5 Timed Events
Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ------------------buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51
II OEM 10g Performance Tab
#.7
#.8
II OEM 10g Zoom-In
II Top Activity Page
#.9
#.10
II 10g v$active_session_history
The best source Only in 10g Can simulate in any version since v7
II Querying ASH
#.11
Top 5 Timed Events Count ------------------------------------ --------log file parallel write 8 CPU 7 select 3 event as "TOP 5 Timed Events", control file parallel write null event 3 cnt as "Count" from ( log file sync 2 select decode(ash.session_state, 'ON CPU','CPU',en.name) event , count(decode(ash.session_state, 'ON CPU','CPU',en.name)) cnt from v$active_session_history ash, v$event_name en where en.event# (+) = ash.event# and SAMPLE_TIME > sysdate - (5/(24*60)) group by decode(ash.session_state,'ON CPU','CPU',en.name) order by count(decode(ash.session_state, 'ON CPU','CPU',en.name)) desc ) where rownum < 6; Copyright 2006 Kyle Hailey
#.12
II Querying ASH for P1,P2,P3

select session_id, event, p1, p2, p3
SID EVENT
------------------------236 db file scattered read 245 db file scattered read 245 db file scattered read 248 db file scattered read
P1 P2 P3
--- --- --3 35291 16 1 13646 2 1 8457 7 1 42226 8
from v$ash
248 read by other session 1 13646 1
245 db file sequential read Copyright 1 13591 1 2006 Kyle Hailey
#.13
II ASH Report
ASH report @?/rdbms/admin/ashrpt.sql Pick interval over last 7 days !
1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***
#.14
III V$session_wait
Moment in Time data
select decode(w.wait_time, 0, w.event , 'CPU') as "TOP 5 Timed Events, count(*) from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply') group by decode(w.wait_time, 0, w.event , 'CPU') order by count(*) desc; Copyright 2006 Kyle Hailey
#.15
III V$session_wait
Top 5 Timed Events ----------------------------------enq: US contention CPU buffer busy waits latch: row cache objects COUNT ---------4 3 1 1
#.16
III V$session_wait
col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event , 'CPU') as status, w.p1, w.p2, w.p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply');
#.17
III V$session_wait
Moment in Time data
SID SQL_HASH STATUS ---------- --------------------------------------------234 82347421 CPU 235 3336613934 enq: US - contention 236 1772152815 enq: US - contention 238 2750335498 enq: US - contention 240 343101472 enq: US - contention 246 1782401401 enq: US - contention 248 3333220954 CPU 252 323960517 enq: US - contention 260 1272059733 CPU P1 P2 P3 ------------------- ---------1431502854 39 0 1431502854 44 0 1431502854 42 0 1431502854 44 0 1431502854 44 0 1431502854 44 0 1650815232 1 0 1431502854 44 0 1431502854 44 0
#.18
Waits we will Ignore

Background Idle RAC Resource Manager
#.19
Background Waits
Filter Out Background Waits
Statspack ASH : SESSION_TYPE='FOREGROUND V$session_wait : type='USER'
#.20
Background Waits
ASH
Avoid
Background waits in ASH with
Select from v$active_session_history where SESSION_TYPE='FOREGROUND' V$session_wait joined to v$session

select from v$session s, v$session_wait w where w.sid=s.sid and s.type='USER'
#.21
Idle Waits
Filter Out 10g
where
wait_class != Idle Create a list
9i

Select name from v$event_name where wait_class=Idle;

a list with
Documentation List created from 10g
Create
#.22
Parallel Query Waits

Filter Out Parallel Query Wait events are unusable
Save
waits are both idle and waits
Parallel Query Waits start with PX or KX

PX
Deq: Par Recov Reply PX Deq: Parse Reply
#.23
RAC Waits
You are on your own Check documentation If you are not using RAC then no worries 10g
Select event from v$event_name where wait_class=Cluster;

9i
RAC and OPS waits usually contain the word global
#.24
Resource Manager Waits

Resource manager throttles user
Creates
wait Obfuscates problems
10g
select name from v$event_name where wait_class='Scheduler';
#.25
Wait Arguments: P1,P2,P3

Each Wait has a 3 parameters P1,P2,P3 Give detailed information Meaning different for each wait Meaning definitions in V$event_name
Select name, parameter1, parameter2, parameter3 from v$event_name;
#.26
Wait Arguments Example

NAME PARAMETER1 PARAMETER2 PARAMETER3 ----------------------------------- --------------- --------------- --------------latch: cache buffers chains free buffer waits buffer busy waits latch: redo copy log buffer space switch logfile command log file sync db file sequential read enq: TM - contention undo segment extension enq: TX - row lock contention row cache lock library cache pin library cache load lock pipe put address file# file# address number block# block# number tries set-id# class# tries
buffer# file# name|mode segment# name|mode cache id handle address object address handle address
block# object # usn<<16 | slot mode pin address lock address record length
blocks table/partition sequence request 100*mode+namesp 100*mask+namesp timeout
#.27
Wait Tree
Buffer Cache I/O Locks Waits
Library Cache
Redo SQL*Net Undo
Top 12 Waits
NAME Count % Total
#.29
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.
db file sequential read 23,850.00 log file sync 20,594.00 db file scattered read 15,505.00 latch free 11,078.00 enqueue 7,732.00 SQL*Net more data from client 7,510.00 direct path read 5,840.00 direct path write 4,868.00 buffer busy waits 4,589.00 SQL*Net more data to client 3,805.00 log buffer space 2,990.00 log file switch completion 2,878.00
11.67% 10.08% 7.59% 5.42% 3.78% 3.67% 2.86% 2.38% 2.25% 1.86% 1.46% 1.41%
Above is over 80% of wait times reported
Anjo Kolk
Copyright 2006 Kyle Hailey www.oraperf.com
#.30
Top 36 Foreground Waits

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. db file sequential read log file sync db file scattered read latch free enqueue SQL*Net more data from client direct path read direct path write buffer busy waits SQL*Net more data to client log buffer space log file switch completion library cache pin SQL*Net break/reset to client io done file open free buffer waits db file parallel read 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. write complete waits library cache lock SQL*Net more data from dblink log file switch (checkpoint incomplete) library cache load lock row cache lock local write wait sort segment request process startup file identify pipe put switch logfile command SQL*Net break/reset to dblink log file switch (archiving needed) Wait for a undo record direct path write (lob) undo segment extension undo segment tx slot
Copyright 2006 Kyle Hailey minus OPS, RAC, PQ, Resource Mgr
Waits 1-9
1.
2. 3. 4. 5. 6. 7. 8. 9.
#.31
db file sequential read

Tune SQL, speed up disks (5-15ms), increase buffer cache
log file sync

Commit less, put redo logs on faster disks
db file scattered read

FTS - Tune SQL, add indexes, speed up disks (5-15ms)
latch free need p1,p2,p3 enqueue need p1,p2,p3 and SQL SQL*Net more data from client
Usually OK, reduce data transferred, possible Network problems
direct path read

sorts or PQO - tune IO, sort less
direct path write

direct path load or temp io, improve disk speed
buffer busy waits need p1,p2,p3 and SQL

Waits 10-18
10. SQL*Net more data to client
Usually OK, reduce amount of data transferred, possible Network tuning needed
#.32
11. log buffer space

Increase log buffer
12. log file switch completion

Increase log file sizes
13. library cache pin

Reduce parsing the same cursor concurently
14. SQL*Net break/reset to client

Check for errors in sql statement
15. io done
Io done by IO slaves, oracleIxxx could also be an idle wait sort of worthless
16. file open

Reduce logons/logoffs or keep log files open
17. free buffer waits

Increase buffer cache size
18. db file parallel read

Tune sql, tune io subsystem, increase buffer cache
Waits 19-27
19. write complete waits
Increase buffer cache (or tune DBWR)
#.33
20. library cache lock

Avoid hard parsing same cursor at same time
21. SQL*Net more data from dblink

Reduce data transfer, check net response
22. log file switch (checkpoint incomplete)

Add log files (or increase log file size)
23. library cache load lock

Avoid hard parsing same cursor at same time
24. row cache lock

need p1
25. local write wait

truncates / reduce cache size
26. sort segment request

SMON busy, process rollback problem
27. process startup

If MTS, increase min servers and dispatchers in init.ora
#.34
Waits 28-36
28. file identify
Keep log files open, reduce checkpoints
29. pipe put

Speed up pipe readers
30. switch logfile command

Avoid switching log files
31. SQL*Net break/reset to dblink

Check for errors in sql statement sent
32. log file switch (archiving needed)

Archive log running out of space
33. Wait for a undo record

??
34. direct path write (lob)

Improve IO, reduce lob write size
35. undo segment extension

Use UNDO or with RBS, increase RBS size, avoid OPTIMAL
36. undo segment tx slot

Use UNDO, increae # of RBS segs
#.35
Difficult Waits
Latches Locks Buffer Busy Row Cache Lock
#.36
Concepts: Latches
Protect areas of memory from concurrent use Light weight locks
Usually
a held in a bit in memory Often supported by atomic processor call Fast and cheap Gone if memory is lost
Often used in cache coherency management

Changes
to a datablock
Exclusive Generally
Sharing
reading has been introduced for some latches

#.37
Finding Latches
latch free
Covers many latches, find the problem latch by 1. select name from v$latchname where latch# = p1;
OR
2.
Find highest sleeps in Statspack latch section
In 10g, important latches have a wait event

latch: cache buffers chains latch: shared pool latch: library cache
#.38
Latches : Find Highest Sleeps

Use Statspack (or AWR report)
Latch Name ------------------shared pool Get Requests Misses Sleeps Spin Gets Sleep1
-------------- ----------- ----------- ---------- ----3,044,462 184,517 19,098 169,226 0
library cache
kks stats row cache object library cache lock library cache pin enqueues enqueue hash chains
2,242,805
263,464 434,844 666,668 741,963 133,577 133,751
79,554
2,842 955 734 635 373 89
15,198
1,774 147 54 42 40 19
66,497
1,152 816 680 593 333 70
0
0 0 0 0 0 0
Important Latches
Cache Buffers Chains
#.39
Hot blocks, distribute activity, reduce update time, avoid select for update
Cache Buffers LRU Chain
Too much activity , uses multiple buffer caches and/or increse LRU latches
Shared Pool
Too much hard parsing, too small a shared pool
Library Cache Latch
Hard Parsing

Use bind variables Uses cursor_sharing=force Increase shared pool size if reloads are high
Avoid invalidations
Hold cursors open Use session_cached_cursors
Soft Parsing

Use cursor_space_for_time
#.40
Concepts: Locks
Enqueue wait covers all locks pre 10 Protect data against concurrent changes Lock info written into data structures
Block
headers Data blocks Written in cache structures
Shareable in compatible modes
Finding Locks
Statspack not much help V$session_wait needs lots of decoding
P1
#.41
tells Lock Type and Mode
P2,P3 give more data
Usually Need SQL to solve

select sid, event, chr(bitand(P1,-16777216)/16777215)|| chr(bitand(P1,16711680)/65535) as "Type", mod(p1,16) as "mode" SID EVENT Ty mode P2 P3 from v$session_wait where event = 'enqueue; --- ------- -- ---- ------- ---240 enqueue TX 6 2686995 433
#.42
Locks 10g
10g breaks Enqueues out
enq: enq: enq: enq: enq: enq:
HW TM TX TX TX UL
contention contention allocate ITL entry index contention row lock contention contention
Configuration Application Configuration Concurrency Application Application
#.43
Locks : TM & TX
select * from v$lock where type in ('TX', 'TM'); SID --151 135 151 135 TY ID1 ID2 LMODE REQUEST CTIME BLOCK -- ---------- ---------- ----- ------- ----- ----TX 589855 1592 0 6 4049 0 TM 53737 0 3 0 4058 0 TM 53737 0 3 0 4049 0 TX 589855 1592 6 0 4058 1
#.44
Buffer Busy Cases

Undo Header - Not enough rollback segments Undo Block hot spot in RBS Data Block
Table Block w/ DML : Table lacks Free lists Table I/O : Multiple sessions reading same block into data cache Note: in 10g, becomes read by other session Index : has hot block, partition or change SQL
Segment Header add free Lists (with Datablock) Free List add free lists groups
#.45
Buffer Busy Waits

Statspack
Buffer wait Statistics for DB: ORA9 Instance: ora9 Snaps: 2 -3 -> ordered by wait time desc, waits desc Tot Wait Avg
Class
Waits
Time (s) Time (ms)
------------------ ----------- ---------- --------data block undo header 1,490 2 2 0 1 0
#.46
Buffer Busy Solutions

IF Buffer Busy Waits high
First

look at v$waitstat
Class Time/Count
SQL> select
* from v$waitstat;
CLASS COUNT TIME ------------------ ---------- ---------data block 1 1 segment header 0 0 free list 0 0 undo header 0 0 undo block 0 0
#.47
Row Cache Lock

Need p1 to see the cache type
SQL> select cache#, parameter from v$rowcache; CACHE# PARAMETER ---------- -------------------------------1 dc_free_extents 4 dc_used_extents 2 dc_segments 0 dc_tablespaces 5 dc_tablespace_quotas 6 dc_files 7 dc_users 3 dc_rollback_segments 8 dc_objects 17 dc_global_oids 12 dc_constraints
#.48
Row Cache Lock

Statspack
^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: 1 -2 ->"Pct Misses" should be very low (< 2% in most cases) ->"Cache Usage" is the number of cache entries being used ->"Pct SGA" is the ratio of usage to allocated size for that cache Get Pct Scan Pct Mod Final Cache Requests Miss Reqs Miss Reqs Usage ----------------- --------- ------ ------- ----- -------- ---------dc_object_ids 45 0.0 0 0 958 dc_objects 89 0.0 0 0 1,129 dc_segments 69 0.0 0 0 807 dc_tablespaces 12 0.0 0 0 13 dc_usernames 22 0.0 0 0 19 dc_sequences 120,003 0.0 0 120,003 5
#.49
Areas of Contention
Buffer Cache Disk I/O Library Cache Redo SQL*Net Undo
We will go through these areas and the wait events that are relevant to them.

Unit1 Intro To Waits

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Unit1 Intro To Waits

Uploaded by

Copyright:

Available Formats

Waits Defined

Kyle Hailey http://perfvision.com

Top 36 Foreground Waits

minus OPS, RAC, PQ, Resource Mgr

Copyright 2006 Kyle Hailey

Waits >> CPU

We are going to concentrate here on WAITS

comparision with Wait Time

Wait Time Top Waits

Wait Arguments p1, p2, p3

SQL that caused waits

Wait and CPU Data Sources

III V$session & v$session_wait

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

II OEM 10g Performance Tab

Copyright 2006 Kyle Hailey

II OEM 10g Zoom-In

Copyright 2006 Kyle Hailey

II Top Activity Page

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

II Querying ASH for P1,P2,P3

248 read by other session 1 13646 1

245 db file sequential read Copyright 1 13591 1 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Waits we will Ignore

Copyright 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Background waits in ASH with

Select from v$active_session_history where SESSION_TYPE='FOREGROUND' V$session_wait joined to v$session

wait_class != Idle Create a list

Select name from v$event_name where wait_class=Idle;

Parallel Query Waits

waits are both idle and waits

Parallel Query Waits start with PX or KX

Deq: Par Recov Reply PX Deq: Parse Reply

Copyright 2006 Kyle Hailey

Select event from v$event_name where wait_class=Cluster;

RAC and OPS waits usually contain the word global

Copyright 2006 Kyle Hailey

Resource Manager Waits

wait Obfuscates problems

Copyright 2006 Kyle Hailey

Wait Arguments: P1,P2,P3

Wait Arguments Example

blocks table/partition sequence request 100*mode+namesp 100*mask+namesp timeout

Copyright 2006 Kyle Hailey

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12.

Above is over 80% of wait times reported

Copyright 2006 Kyle Hailey www.oraperf.com

Top 36 Foreground Waits

db file sequential read

log file sync

db file scattered read

direct path read

direct path write

buffer busy waits need p1,p2,p3 and SQL

11. log buffer space

12. log file switch completion

13. library cache pin

blocks table/partition sequence request 100mode+namesp 100mask+namesp timeout