Part II : Waits Events and the Geeks who love them

Kyle Hailey


Wait Events Wait Events

Copyright 2006 Kyle Hailey


And the Geeks Who Love Them

Copyright 2006 Kyle Hailey


In this Presentation: 
Introduction to Waits  Tuning Methodology
of Action  Statspacks, AWR or OEM for Collection Data  Based on Waits 

Using Waits to Solve Bottlenecks

Copyright 2006 Kyle Hailey

what do we do? Copyright 2006 Kyle Hailey .5 Database is Hung!  Everybody blames the database  Yet 9 out of 10 dba¶s agree it¶s not the database  How do you prove it to management? On the off chance it¶s the database.#.

Database: Guilty until proven innocent #.6 *$%@!! Copyright 2006 Kyle Hailey .

7 Oracle¶s Defense  After years of false accusations  Oracle took action and created a defense system: WAIT EVENTS To the rescue  Oracle is the best instrumented database on the market which can save time and money on development and tuning Copyright 2006 Kyle Hailey .#.

8 Redo Lib Cache Buffer Cache Network IO Copyright 2006 Kyle Hailey .Oracle Instrumentation CPU Locks #.

procedure) Where (CPU. Wait) When (time) What (SQL statement) Copyright 2006 Kyle Hailey .9 from Ratio Guesswork to empirical measure of time lost to bottlenecks only identifies bottlenecks but  10g added the crucial addition ASH  Not     Who (session. service.Waits  Introduced in v7  Revolutionized tuning  Changed #. package.

#.10 Tuning Methodology 1. Machine  Run queue (CPU)   Check other applications reduce CPU usage or add CPUs Reduce memory usage or add memory  Paging  2. available CPU then Tune waits    It¶s the application Copyright 2006 Kyle Hailey . Oracle   Waits + CPU > Available CPU We are going to concentrate here CPU 100% on WAITS  Tune SQL Else low waits.

sql  OEM 10g  Performance Page does everything If there is a wait bottleneck tune the wait Copyright 2006 Kyle Hailey .#.11 Dependable Tuning Strategy Determine AAS :  Run Statspack or AWR Report  Top  5 Timed Events Available CPU ~50 lines down from top Elapsed Time CPU_COUNT  Need    ASH Report : ashrpt.

#. it¶s the application Get to Work! Copyright 2006 Kyle Hailey .12 Tuning Methodology Graphics Relax.

13 Waits beyond OEM  OEM identifies Wait problems  Provides solutions with ADDM sometimes  But   What do you do when ADDM isn¶t sufficient? What do you do if you don¶t have OEM 10g? Need to know about waits How they work How to analyze them Copyright 2006 Kyle Hailey  Waits    .#.

14 We¶ll discuss Waits in these logical database areas Copyright 2006 Kyle Hailey .Wait Areas Buffer Cache I/O Locks Waits Library Cache Redo SQL*Net #.

15 Rollback Free lists IO Read Copyright 2006 Kyle Hailey .Wait Tree IO Buffer Cache Library Cache Waits Lock Redo SQL Net Write IO Read IO Buffer Busy Cache Latches Library Cache Shared Pool TX Row Lock TX ITL Lock HW Lock Log File Log Buffer Log File Sync #.

CURRENT_FILE# . client.16 CURRENT_OBJ# . service. package. etc)  SQL statement  For IO related waits   #.CURRENT_BLOCK# Blocking_Session  P1  P2  P3  Copyright 2006 Kyle Hailey . procedure.v$active_session_history  When ADDM fails or we don¶t have ADDM we can collect the necessary information from  v$active_session_history Session (user.

col parameter2 for a10 parameter1. select parameter1 .P2.parameter2 .P3 ?  Each Wait has a 3 parameters P1.P2. Copyright 2006 Kyle Hailey .P3  Give detailed information  Meaning different for each wait  Meaning definitions in V$event_name Select col parameter1 for a10 name.#. col parameter3 for a10 parameter2. where name = '&1'.17 What are P1. parameter3 parameter3 from v$event_name from v$event_name.

--------------latch: cache buffers chains free buffer waits buffer busy waits latch: redo copy log buffer space switch logfile command log file sync db file sequential read enq: TM . NAME PARAMETER1 PARAMETER2 PARAMETER3 -----------------------------. parameter3 from v$event_name.contention undo segment extension enq: TX .--------------.row lock contention row cache lock library cache pin library cache load lock pipe put address file# file# address number block# block# number tries set-id# class# tries #.Wait Arguments Example select parameter1 .18 buffer# file# name|mode segment# name|mode cache id handle address object address handle address block# object # usn<<16 | slot mode pin address lock address record length blocks table/partition sequence request 100*mode+namesp 100*mask+namesp timeout Copyright 2006 Kyle Hailey .----------.parameter2 .

#.19 Wait Analysis requires p1.P2.P3 Copyright 2006 Kyle Hailey .p3  Of the top 30 wait events 8 can be solved without ASH free buffer waits log buffer space log file switch (archiving needed) log file switch (checkpoint incomplete) log file switch completion log file sync switch logfile command write complete waits  The rest need  Sql_id and/or P1.p2.

20 Difficult Waits These 4 waits have multiple causes  Latches  p2 = latch # (p1= address. p3=request) Copyright 2006 Kyle Hailey . p1= file. p3= tries) p1 = lock type and mode ( p2 = id1. p2=block  (in 9i p3 was the bbw type)   Row Cache Lock  p1 = cache id (p2 = mode. p3= id2)  Locks   Buffer Busy p3 = block class#.#.

 Sometimes the wait events that are found are not in the documentation and it takes some educated guesswork to figure out the problem . and changing it Parameter2.#. Parameter3  Find extended wait information  Parameter1.21 Wait Analysis  Find SQL waiting  Most often the tuning answer lies in looking at what the application is doing.

#.  Background  Idle  Resource Manager Copyright 2006 Kyle Hailey .22 Waits we will Ignore One thing that makes waits difficult is knowing which ones to look at and which ones to ignore.

Background Processes PMON SMON SGA Log Library Buffer Buffer Cache Cache Buffer Cache Log Buffer DBWR User1 User2 User3 REDO Log Files Copyright 2006 Kyle Hailey #.23 LGWR Data Files .

24 Background & Foreground  Background Processes      DBWR LGWR PMON SMON Etc SQL*Plus Pro*C SQL*Forms Oracle applications  Foreground Processes     Only interested in Foreground waits Copyright 2006 Kyle Hailey .#.

25 Background Waits  ASH  Avoid Background waits in ASH with Select «from v$active_session_history where SESSION_TYPE='FOREGROUND'  V$session_wait joined to v$session select from « v$session s.sid and s.type='USER' Copyright 2006 Kyle Hailey . v$session_wait w where w.#.sid=s.

Idle Waits  Filtered Out of ASH by default  10g wait_class != µIdle¶  Create a list  where #.26  9i    Select name from v$event_name where wait_class=µIdle¶. a list with Documentation List created from 10g Stats$idle_events from statspack  Create SQL*Net message from client Copyright 2006 Kyle Hailey .

#.27 Parallel Query Waits Filter Out  Parallel Query Wait events are unusable  Save waits are both idle and waits  Parallel Query Waits start with µPX¶ or µKX¶ Deq: Par Recov Reply  PX Deq: Parse Reply  PX Copyright 2006 Kyle Hailey .

28 Resource Manager Waits  Resource manager throttles user wait  Obfuscates problems  Creates  10g select name from v$event_name where wait_class='Scheduler'.#. Copyright 2006 Kyle Hailey .

29 RAC Waits RAC waits are certainly interesting but will be covered outside of this presentation.     You are on your own Check documentation If you are not using RAC then no worries 10g Select event from v$event_name where wait_class=µCluster¶.  9i  RAC and OPS waits usually contain the word ³global´ Copyright 2006 Kyle Hailey .#.

30 Latches  Protect areas of memory from concurrent use  Light weight locks  Bit in memory  Atomic processor call  Fast and cheap  Gone if memory is lost  Often used in cache coherency management  Changes  Sharing to a data block  Exclusive Generally reading has been introduced for some latches Copyright 2006 Kyle Hailey .#.

find the problem latch by 1.31 Finding Latches  ³latch free´  Covers many latches. OR 2. Find highest sleeps in Statspack latch section  In 10g.#. select name from v$latchname where latch# = p1. important latches have a wait event    latch: cache buffers chains latch: shared pool latch: library cache Copyright 2006 Kyle Hailey .

32 Enqueues aka Locks  ³Enqueue´ wait ± covers all locks pre 10  Protect data against concurrent changes  Lock info written into data structures  Block headers  Data blocks  Written in cache structures  Shareable in compatible modes Copyright 2006 Kyle Hailey .#.

#.33 Locks 10g  10g breaks all Enqueues out enq:  enq:  enq:  enq:  enq:  enq:  HW TM TX TX TX UL - contention contention allocate ITL entry index contention row lock contention contention Configuration Application Configuration Concurrency Application Application Copyright 2006 Kyle Hailey .

CACHE# ---------1 4 2 0 5 6 7 3 8 17 12 PARAMETER -------------------------------dc_free_extents dc_used_extents dc_segments dc_tablespaces dc_tablespace_quotas dc_files dc_users dc_rollback_segments dc_objects dc_global_oids dc_constraints Copyright 2006 Kyle Hailey #.34 .Row Cache Lock  Need p1 to see the cache type SQL> select cache#. parameter from v$rowcache.

0 0 0 13 dc_usernames 22 0.-------.0 0 0 958 dc_objects 89 0.---------dc_object_ids 45 0.0 0 0 19 dc_sequences 120.-----.------.#.----.003 0.0 0 0 1.0 0 120.129 dc_segments 69 0.--------.0 0 0 807 dc_tablespaces 12 0.35 Row Cache Lock  Statspack ^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: 1 -2 ->"Pct Misses" should be very low (< 2% in most cases) ->"Cache Usage" is the number of cache entries being used ->"Pct SGA" is the ratio of usage to allocated size for that cache Get Pct Scan Pct Mod Final Cache Requests Miss Reqs Miss Reqs Usage ----------------.003 5 Copyright 2006 Kyle Hailey .

min  DBA_HIST_SEG_STAT     DBA_HIST_SYSTEM_EVENT    DBA_HIST_SQLSTAT DBA_HIST_SYSMETRIC_SUMMARY  Metric Tables ± in memory deltas  V$EVENTMETRIC Copyright 2006 Kyle Hailey .#. max.36 Additional Support  AWR Tables ± on disk for 7 days by default  DBA_HIST_ACTIVE_SESS_HISTORY  1 in 10 ASH samples Good for ITL and buffer busy wait Important for getting avg wait times sql execution deltas Statistics avg.

event from ( select event from DBA_HIST_ACTIVE_SESS_HISTORY where sample_time < ( select min(sample_time) from min(sample_time) v$active_session_history) v$active_session_history) union all select event from v$active_session_history ) group by event order by event / Copyright 2006 Kyle Hailey .#.37 All Events over 7 days select count(*).

CURRENT_FILE# filen. o.p2.p3 and o.38 P1 P2 OBJN OTYPE FILEN BLOCKN SQL_ID BLOCK_TYPE -.------1 112796 66053 BBW_INDEX_VAL_I INDEX 1 112796 6avm49ys4k7t6 data block 1 1 112401 66053 BBW_INDEX_VAL_I INDEX 1 112401 5wqps1quuxqr4 data block 1 1 112796 66053 BBW_INDEX_VAL_I INDEX 1 112796 5wqps1quuxqr4 data block 1 1 113523 66053 BBW_INDEX_VAL_I INDEX 1 113523 5wqps1quuxqr4 data block 1 Copyright 2006 Kyle Hailey .------------. class from v$waitstat ) w.--------------------.object_id (+)= ash. CURRENT_BLOCK# blockn. ash.sample_time > sysdate .----.class#(+)=ash.CURRENT_OBJ# and ash.-----. w.SQL_ID.&1/(60*24) Order by sample_time Example ASH Query #. ash.object_name objn.p1.session_state='WAITING' and ash.Select ash.class ||' '||to_char(ash.object_type otype.----. CURRENT_OBJ#||' '||o.p3) block_type from v$active_session_history ash.-----. all_objects o where event='buffer busy waits' and w. ( select rownum class#.

0) avg_ms from ( select to_char(s.39 Average Wait Times Historic select btime.event_name ORDER BY s.046 order by btime.snap_id) time_ms_beg.'DD-MON-YY HH24:MI') btime.total_waits) OVER( PARTITION BY e. time_waited_micro/1000 time_ms_end.#.event_name= '&1' 08-JAN08-JAN-08 03:00 .event_name ORDER BY s. total_waits count_end.017 s. Lag (e.720 and e.BEGIN_INTERVAL_TIME.snap_id=e.snap_id) count_beg from DBA_HIST_SYSTEM_EVENT e.time_waited_micro/1000) OVER( PARTITION BY e. BTIME AVG_MS DBA_HIST_SNAPSHOT s -------------------. 08-JAN08-JAN-08 06:00 Copyright 2006 Kyle Hailey 1. Lag (e.747 ) 08-JAN08-JAN-08 05:00 1.snap_id 08-JAN08-JAN-08 02:00 .-----------where 08-JAN08-JAN-08 01:00 1.621 order by begin_interval_time 08-JAN08-JAN-08 04:00 1. (time_ms_end-time_ms_beg)/nullif(count_end-count_beg.444 .

Avg Wait times now select 6420 .---------.549427419 186 .event# and en.40 AVG_MS WAIT_COUNT ---------.event# = en. (time_waited)/nullif(wait_count. NAME ------------------------db file sequential read db file scattered read db file parallel write #.0) avg_ms.089073438 64 Copyright 2006 Kyle Hailey . v$event_name en where like '%&1%µ. wait_count from v$eventmetric e.

41 Object Translation  Object ID  File # and Block # .#. Copyright 2006 Kyle Hailey .42 Wait interface Weaknesses  Logons EM 10g shows these on perf page  Time model helps    V$SYS_TIME_MODEL  connection management call elapsed time I¶ve had problems     Paging/Memory issues CPU starvation Null Events Bugs ± read external table reports CPU  http://blog.

#.43 Dependable Tuning Strategy  Run Statspack/AWR report  Top  5 Timed Events Available CPU ~50 lines down from top Elapsed Time CPU_COUNT  Need    OEM 10g  Performance  Query Page does everything !  OEM doesn¶t solve the problem v$active_session_history directly Copyright 2006 Kyle Hailey .

PQO  Use ASH if OEM fails  See http://perfvision.  S/ASH   Ignore Background.Summary  Waits make Tuning Easy Check Machine Health  Tune Waits  Tune CPU    #. Resmgr. Idle.44 Tune SQL Change Application Architecture  Use OEM10g  Statspack/ for more info Copyright 2006 Kyle Hailey .

Sign up to vote on this title
UsefulNot useful