Understanding LGWR, Log File Sync Waits and Commit Performance

Tanel Põder http://blog.tanelpoder.com http://tech.e2sn.com

Enterprise 2.0 Service Networks

© 2009-2010 www.e2sn.com

WWW.E2SN.COM

1

Intro: About me
• Tanel Põder
• http://tech.e2sn.com - My company and technical Oracle stuff • http://blog.tanelpoder.com - Personal Blog (more tech stuff) • tanel@tanelpoder.com - Questions & enquiries

• Consulting, Training, Seminars • tanel@tanelpoder.com • http://tech.e2sn.com/oracle-training-seminars
• Online seminars coming soon!

Enterprise 2.0 Service Networks

© 2009-2010 www.e2sn.com

WWW.E2SN.COM

2

Topics
• How does commit & log file sync work
• Overview

• Reasons for “too long” log file sync waits

• How to measure where’s the problem?

Enterprise 2.0 Service Networks

© 2009-2010 www.e2sn.com

WWW.E2SN.COM

3

Reasons for log file sync waits • Commits wait for log file sync by default • User commits • There’s an user commits statistic in v$sesstat • DDL • Resulting recursive transactions commit • Recursive data dictionary DML • Rollbacks wait too! • User rollbacks • User/application issued a rollback command • Transaction rollbacks • We had an internal rollback (because of some failure) • Space allocation/ASSM problems.e2sn.E2SN.com WWW.0 Service Networks © 2009-2010 www. killed sessions Enterprise 2. cancelled queries.COM 4 .

E2SN.com WWW.e2sn.Commit & log file sync flow – idealistic overview Time 1) User issues a COMMIT 2) Foreground proc posts LGWR 3) LGWR issues the physical write syscall 5) LGWR posts the FG proc. 6) Commit complete FG proc. LGWR IO 4) The physcical write syscall completes log file parallel write log file sync Enterprise 2.COM 5 .0 Service Networks © 2009-2010 www.

COM Enterprise 2.e2sn.0 Service Networks © 2009-2010 www.com 6 . IPC were small WWW. LGWR IO The physical write IO (log file parallel write) takes most of the time Most of the log file sync time was spent waiting on log file parallel write The other components.Log file sync performance -> disk IO speed Time FG proc.E2SN. scheduling latency.

com 7 . gets onto CPU runqueue LGWR IO 3) LGWR submits the IO. goes to sleep 4) IO completes.Log file sync performance -> scheduling latency Time 1) User issues a COMMIT FG proc.0 Service Networks © 2009-2010 www. OS puts LGWR to CPU runqueue WWW.COM 5) LGWR gets onto CPU. 2) LGWR waits in CPU runqueue 6) Foreground proc gets posted.e2sn.E2SN. Enterprise 2. posts foreground proc.

Log file sync flow 1.COM 8 . LGWR wakes up. Foreground process (FG) posts LGWR and goes to sleep • The “log file sync” wait starts • Posting is done via a semaphore operation on Unix/Linux 2.0 Service Networks © 2009-2010 www.E2SN.com WWW. Hardware completes the IO and OS wakes up LGWR • LGWR gets onto CPU • Marks “log file parallel write” event complete and posts the FG 4. Foreground process is woken up by LGWR post • Foreground process gets onto CPU and completes the “log file sync” wait Enterprise 2.e2sn. waiting for “log file parallel write” wait 3. gets onto CPU • Issues the IO request(s) • LGWR goes to sleep.

seconds=1 Enterprise 2. 1096. 10 . 1. (LGWR) . 1096.4%. GRAPH --------------------------------------------------------------------------------1096. %TIME. 1096. 1096. 273.Session Snapper v2.88M.1%. 2817 . HDELTA/SEC. 1096.|@@@@@@@@@@| -. STAT. messages received .e2sn. events in waitclass Other . (LGWR) . redo writes . 1. STAT. 38. 1096. background timeouts . 1 . 1096. STAT.82k. STAT. |@@@@@@@@@@| 1096. 25 . (LGWR) . STAT. WAIT. (LGWR) .Measuring LGWR "speed" SQL> @snapper out 1 10 1096 -. redo blocks written . 4548 .55k. redo wastage . 40. 10 .End of snap 1.04s. 1035172 .0 Service Networks © 2009-2010 www. DELTA. 12.01 by Tanel Poder ( http://www.E2SN. 1096. 2. 1. (LGWR) . 103. (LGWR) . end=2010-03-23 12:46:04. redo write time . 20. physical write total bytes. STATISTIC .84ms. STAT.COM 9 .tanelpoder. 273837 .5%. 40.com WWW. (LGWR) .04s . 25. 1040575 . messages sent . 1096. 2884608 . 104. STAT. (LGWR) . 1096. LGWR wait on LNS . STAT. TYPE. 27. STAT. physical write total multi block request. 12 . WAIT. 1096. 38. calls to kcmgcs . (LGWR) . WAIT. (LGWR) .com ) --------------------------------------------------------------------------------SID. 20 . (LGWR) . STAT. log file parallel write . physical write total IO requests . STAT. 4. 10. 2. (LGWR) . (LGWR) . USERNAME . (LGWR) .|@@@ | 1096. 10.

interrupt to quit ^CProcess 12457 detached % time seconds usecs/call calls errors This is what the log -----.----------.000000 0 701 0.----------.COM 10 .0 Service Networks © 2009-2010 www.LGWR and Asynch IO oracle@linux01:~$ strace -cp `pgrep -f lgwr` Process 12457 attached .000000 0 2 0.00 0.--------file263 parallel write wait 100.00 0.000000 0 37 instrumented by wait -----.--------Interface! 100.00 0.00 0.000000 0 41 This system call is not 0.com WWW.000000 0 8 reaping duration 0.00 0.000000 0 41 0.E2SN.--------.000000 213 event0measures – AIO 0.00 0.00 0.010000 1081 3 syscall -------------semtimedop times getrusage gettimeofday io_getevents io_submit semop semctl -------------total Enterprise 2.----------.00 0.00 0.--------.010000 38 3 0.e2sn.----------.

for example 11.Warning • The (background processes) IO instrumentation has quite a few bugs • Different IO modes. IO syscall waits are instrumented ok (but you don’t wan to use this option) • When filesystem_io_options = ASYNC.COM 11 . IO reaping waits are instrumented properly Enterprise 2.1 on Linux • When filesystemio_options = NONE. the log file parallel write (and db file parallel write) aren’t properly instrumented • Version dependent.e2sn. asynd. direct.com WWW. IO reaping waits are all very short • However there’s unaccounted time in LGWR’s wait profile • When filesystem_io_options = SETALL. buffered etc • On some versions.0 Service Networks © 2009-2010 www.E2SN.0.2. sync.

we should fix only problems which exist • In other words.Redo. if wait interface doesn’t show anyone waiting for them.E2SN. then don’t bother “tuning” them! WWW.0 Service Networks © 2009-2010 www.COM Enterprise 2.com 12 .e2sn. commit related latches and “tuning” • Redo related latches • redo allocation latches • Protect allocating space in log buffer / RBA ranges in redolog stream • redo copy latches • Used only for keeping track of whether anyone’s copying data into redo log buffer • …so that LGWR would know to wait for these memory copies to complete before it tries to write buffers to disk • LGWR will wait for LGWR wait for redo copy wait event in such cases • Used to be tuned by _log_simultaneous_copies • Should we “tune” any of these? • No.

com WWW.E2SN.Instrumentation • Wait Events: • log file sync • log file parallel write • log file single write • Performance Counters (V$SESSTAT.COM 13 . V$SYSSTAT) • • • • • redo size redo writing time user commits user rollbacks transaction rollbacks Enterprise 2.e2sn.0 Service Networks © 2009-2010 www.

Wait event: log buffer space • (Not a commit problem) • LGWR is too slow flushing redo log buffer contents to disk • Either because too slow IO subsystem • Or LGWR not getting enough (quality) CPU time • Sometimes pops up due large (unplanned) transactions • Of course. it can also be because of a too small log buffer • Which is not the case anymore in modern days • Log buffer is usually multiple MB due how it is allocated from SGA • You shouldn’t even set the log_buffer parameter in 10g+ Enterprise 2.com WWW.e2sn.E2SN.COM 14 .0 Service Networks © 2009-2010 www.

e2sn.Wait event: log file single write • Single block redo IO is used mostly for logfile header block reading/writing • Log switch is the main cause • Archiving as well as it updates log header • Who wait: LGWR & ARCH • Example of what LGWR does during a log switch: WAIT #0: nam='log file sequential read' ela= 12607 log#=0 block#=1 WAIT #0: nam='log file sequential read' ela= 21225 log#=1 block#=1 WAIT #0: nam='control file sequential read' ela= 358 file#=0 WAIT #0: nam='log file single write' ela= 470 log#=0 block#=1 WAIT #0: nam='log file single write' ela= 227 log#=1 block#=1 Enterprise 2.E2SN.COM 15 .com WWW.0 Service Networks © 2009-2010 www.

size 19KB *** 2010-03-10 11:37:23.778 Warning: log write time 52710ms.2.e2sn. size 144KB Enterprise 2. size 0KB *** 2010-03-10 11:37:27.com WWW.E2SN.302 Warning: log write time 3520ms.4) LGWR is trying to be helpful and dump warnings when the actual log write IO takes too long: • New parameter: _side_channel_batch_timeout_ms • timeout before shipping out the batched side channelmessages in milliseconds • LGWR trace file: *** 2010-03-10 11:36:06.2.0.3 (or was it 10.COM 16 .LGWR trace warnings… • Starting from 10.0.0 Service Networks © 2009-2010 www.759 Warning: log write time 690ms.

com WWW.COM 17 .E2SN.Log file sync in Statspack / AWR • How much of the end-to-end response time goes to log file sync? • How big it is compared to the full response time of the end user? • Log file sync may take 20% of your DB Time… • …but the DB Time itself may take only 10% of the total end user response time! Enterprise 2.e2sn.0 Service Networks © 2009-2010 www.

.-------.8 db file sequential read 61.562 25 12.3 log file sync 5... %Tim Total Wait out Time (s) ---.3 16 0.0 Event Waits -----------------------.-----------log file parallel write 13..e2sn.-----8 0.Log file sync in Statspack / AWR • If log file sync waits take a significant part of the response time..COM 18 .com WWW.258 1.470 db file sequential read 257 Enterprise 2.-----------..6 direct path write 235.-----.754 153 68..---------0 108 0 39 0 4 Avg %Total wait Waits Call (ms) /txn Time -----.606 155 1 1.E2SN.2 ..292 db file parallel write 3.159 8..8 11 0.0 CPU time 463 3.-----PL/SQL lock timer 57.9 .----------. look into the “Avg wait (ms)” column: Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time ------------------------------------.0 .2 ..873 1..0 Service Networks © 2009-2010 www.522 259 12.

COM Enterprise 2.--------------. wait_time_milli.E2SN.com 19 . breaks wait times into buckets SQL> select event.Better breakdown of wait times • V$EVENT_HISTOGRAM • Instead of a single wait time average.wait_count 2 from v$event_histogram 3 where event = 'log file parallel write'.e2sn.0 Service Networks © 2009-2010 www. EVENT WAIT_TIME_MILLI WAIT_COUNT ------------------------.---------log file parallel write 1 22677 log file parallel write 2 424 log file parallel write 4 141 log file parallel write 8 340 log file parallel write 16 1401 log file parallel write 32 812 log file parallel write 64 391 log file parallel write 128 21 log file parallel write 256 6 WWW.

it can’t be directly measured with standard tools • Indirectly.E2SN.com WWW.0 Service Networks © 2009-2010 www.Identifying scheduling latency Easy on Solaris • prstat –m • Works since Solaris 8 • Timed_os_statistics • OS Wait-cpu (latency) time in v$sesstat On other OS’es.COM 20 . huh? Enterprise 2. you can look into system wide average CPU runqueue length and assume that LGWR was also queueing • Not a too systematic approach.e2sn.

0 0.0 0.0 TRP 0.0 0.0 44 0 90 0 oracle/7 0.0 0.2 5124 oracle 0.Measure scheduling latency (Solaris) • Reading thread-level microstate accounting data with prstat: • • • • • • • USR SYS TRP TFL/DFL LCK SLP LAT % % % % % % % Time spent on CPU in user mode Time spent on CPU in kernel mode Time spent processing system traps (CPU traps) Time spent processing text/data page faults Time spent waiting for user locks Time spent sleeping (other than user locks) CPU scheduling latency # prstat -mLp 5124 PID USERNAME USR 5124 oracle 4.0 0.0 DFL 0.0 0.0 0.0 Service Networks © 2009-2010 www.0 5124 oracle 0.0 0.0 0.0 46 0 90 0 oracle/9 0.0 0.0 0.0 5124 oracle 0.0 0.0 0.1 412 20 3K 0 oracle/1 0.0 100 0.0 5124 oracle 0.6 0.0 5124 oracle 0.0 0.6 0.0 1 0 5 0 oracle/3 0.0 5124 oracle 0.E2SN.0 LAT VCX ICX SCL SIG PROCESS/LWPID 0.0 0.0 99 99 99 99 0.0 SYS 1.com WWW.0 0.0 0.6 0.0 0.0 0.0 0.0 0.e2sn.0 0.0 24 0 7 0 oracle/2 0.0 100 100 100 SLP 94 0.0 LCK 0.6 0.0 1 0 5 0 oracle/8 Enterprise 2.0 44 1 90 0 oracle/5 0.6 0.0 TFL 0.0 5124 oracle 0.0 0.7 0.0 44 3 90 0 oracle/11 0.COM 21 .0 0.0 0.0 0.0 0.6 0.0 0.0 5124 oracle 0.0 0.0 0.0 1 0 5 0 oracle/10 0.

com WWW.COM 22 .x on Solaris for example) • For LGWR you can use V$SESSTAT redo write time statistic instead • It’s in centiseconds • 1-second log file sync bug • Most log file syncs took ~1 second to complete • The posts sent back by LGWR were missed by foreground process • Thus the FG always waited until the 1 second log file sync wait timeout happened Enterprise 2.2-10.0.2.0 Service Networks © 2009-2010 www.E2SN. problems • No instrumentation • On some version/platform/IO configuration the wait interface doesn’t record log file parallel write waits at all • The same goes for db file parallel writes • (I’ve noticed it on 9.Bugs.e2sn.

Tuning • No need for tuning! • Log buffer is quite large by default • All memory remaining in a granule after the allocation for fixed SGA is given to log buffer • Oracle used to have a single redo log buffer until v9.E2SN. Oracle can keep lots of small private redo strands in shared pool • Each protected by a separate redo allocation latch • @rs.COM Enterprise 2.2. Oracle can have the log buffer split into multiple buffers • Each protected by a separate redo allocation latch • From 10g.e2sn.com 23 .0 Service Networks © 2009-2010 www.sql – Show redo strands available WWW.0 • Redo allocation latch could become the ultimate contention point • Since 9.

E2SN. then: • 10gR1: • commit_logging transaction commit log write behavior • 10gR2: • commit_write • commit_wait transaction commit log write behavior transaction commit log wait behavior • Old undocumented stuff • _wait_for_sync wait for sync on commit MUST BE ALWAYS TRUE • Old: Put redologs to /tmp (on Solaris) or in-memory disks (/dev/shm) for duration of a migration/upgrade • If your OS / server crashes.Evil tuning • If you don’t care about the D in ACID (and want to occasionally lose data for fun).COM 24 .e2sn.com WWW. you’ll need to restore from a backup! Enterprise 2.0 Service Networks © 2009-2010 www.

E2SN. • No log file sync waits • log buffer space / log file switch completion waits more likely! Enterprise 2. end loop.e2sn.0 Service Networks © 2009-2010 www.Optimizations for working around bad applications • Commit optimization • In PL/SQL since Oracle 9i • The log file sync is deferred until the end of the PL/SQL call! SQL> exec while true loop update t set a=a+1 .COM 25 . commit .com WWW.

com LMS*|VKTM 26 WWW.COM .LGWR configuration – CPU • Prevent priority decay • Put LGWR into fixed priority scheduling class (FX60 on Solaris) • LGWR should get onto CPU faster when waking up • LGWR isn’t thrown off CPU as likely • If LGWR is still experiencing significant scheduling latency • You can put LGWR into a higher priority class • You should not put LGWR into the highest real-time class • Real time is tricky – your process can monopolize a CPU for itself • You don’t want to make LGWR pre-empt the OS kernel! • Note that Oracle sets some processes into higher priority by default: • _high_priority_processes Enterprise 2.0 Service Networks © 2009-2010 www.E2SN.e2sn.

COM 27 .IO • Reduce the amount of work and waiting a “log file parallel write” has to do • Unbuffered concurrent IO • Verify with truss/strace whether proper flags are used • (O_DIRECT. O_DIO. moving redo log files to SSD may not give any advantage! • Verify what’s your current “log file parallel write” latency using v$event_histogram Enterprise 2. O_CIO etc) • Or use raw devices • ASM is essentially a raw device • Or ODM for some cases • And optimize the whole IO hardware stack.com WWW.0 Service Networks © 2009-2010 www.E2SN.e2sn. of course! • Note shat mid-large size storage arrays do have write cache built in • So.LGWR configuration .

0 Service Networks © 2009-2010 www.log file sync magic tuning super-secret!!! COMMIT LESS!!! Commit when your business transaction ends.E2SN. not after every single update! Enterprise 2.com WWW.e2sn.COM 28 .

0 Service Networks © 2009-2010 www.COM 29 .e2sn.Summary • Application: Commit less! • Ideally only when your logical business transaction ends • Troubleshooting: Measure log file sync at session level detail • CPU: If waits for log file sync are significant .see whether LGWR gets: • Enough (quality) CPU time • Onto the CPU fast enough • IO: See how much LGWR waits for log file parallel write event • What’s the log file parallel write completion time • V$EVENT_HISTOGRAM for better detail Enterprise 2.com WWW.E2SN.

e2sn.Thanks! • Download slides from: • http://tech.tanelpoder.E2SN.e2sn.com/oracle-slides-and-whitepapers • Download Snapper from: • http://tech.com/oracle-scripts-and-tools/session-snapper • Blog: • http://blog.0 Service Networks © 2009-2010 www.com Enterprise 2.COM 30 .e2sn.com • Email: • tanel@tanelpoder.com WWW.

Sign up to vote on this title
UsefulNot useful