This action might not be possible to undo. Are you sure you want to continue?
Advisory Team SQL Server Development
Why waits & queues?
What impacts performance?
App life cycle - before, during, after development
A Performance Tuning methodology
Waits & Queues Performance metrics & ratios Other performance tools
Correlating Waits & Queues Other sources of information Conclusion Q&A
Performance ± Why waits & queues? Performance is a broad topic
Art or Science?
Waits & Queues Methodology
Based on SQL Customer Lab experiences Objective - analyze customer app over 3-4 day period Repeatable process Useful in quickly identifying some bottlenecks
Performance, Architecture, Code
Not a replacement for good design
Application Life Cycle
Production workloads Test Matrix
Impact on Performance Application & Database Design and T-SQL have major impact on performance µTypical¶ performance tuning plays smaller role Performance monitoring can point out some deficiencies (or opportunities for improvement!) throughout application life cycle. App DB Design SQL Design Hardware tuning 25% 25% 20% 10% Perf Monitoring workload changes 20% .
Performance Life Cycle Performance monitoring plays a key role throughout the application life cycle. not just at the end (prior to production) Provides clues that identifies problem areas in an application including SQL App architecture Database design Configuration Hardware .
What can Performance Monitoring do? Some typical bottlenecks What are some of the typical problem areas in application? Cache management Query plan reuse Recompilation Transaction management e. concurrency Resource utilization Doesn¶t replace need for good app design .g.
Start Demo SQLStressTest application PerfMon counters Track_waitstats .
explains application performance Most are familiar with queues ( ½ the story) Both are essential for problem determination Queues Perfmon counters .Waits & Queues Complementary information Together.measure resource utilization Queues indicate unfulfilled resource requests SQL Waits Waits from an Application or User connection perspective .
sorts 50+ wait types for SQL2000 Anytime a user connection waits for results. acquire locks. CPU ± E.g. SQL sets a wait type . read | write data pages.Waits SQL waits Application sends SQL queries SQL Server in turn issues resource requests ± Translates to subsystem IO. memory.
Queues PERFMON counters provide a view of system performance from a resource standpoint Queues measure resource utilization Requests that cannot be immediately provided are queued Key Perfmon counters (a few«) Physical Disk (IO) SQL Buffer Cache SQL Statistics .
spid history .Wait types If a thread is not currently executing. a wait type or state is set in sysprocesses Sysprocesses contains Lastwaittype Waitype Waittime Limitation: Transience of sysprocesses.
DBCC sqlperf(waitstats. what you can control? Wait types .[clear]) Cumulative waittypes & waittimes Limitation: relevance of types.
Track_waitstats stored proc captures SQL waitstats from DBCC SQLPERF ranks wait types in descending order based on percentage of total waits identifies greatest opportunites for performance improvement Can be graphed to see how waits vary over a workload Limitation: long durations can ³hide´ or smooth out waits .
@totalwait numeric(20. default delay interval is 1 minute -. and confers no rights.create waitstats table if it doesn't exist.1).This stored procedure is provided "AS IS" with no warranties.htm -.@num_samples is the number of times to capture waitstats. -. @sec int select @i = 1 select @dt = case lower(@delaytype) when 'minutes' then 'm' when 'minute' then 'm' when 'min' then 'm' when 'mm' then 'm' when 'mi' then 'm' when 'm' then 'm' when 'seconds' then 's' when 'second' then 's' when 'sec' then 's' when 'ss' then 's' when 's' then 's' else @delaytype end if @dt not in ('s'.@delaynum int=1. @now datetime. [signal wait time] numeric(20. now datetime default getdate()) else truncate table waitstats dbcc sqlperf (waitstats.@delaytype nvarchar(10)='minutes') as -.1) .@hr int. delaytype specifies whether the delay interval is minutes or seconds -.microsoft. [wait time] numeric (20. @min int.Use of included script samples are subject to the terms specified at http://www.T.@dt varchar(3).'m') begin print 'please supply delay type e. default is 10 times.delaynum is the delay interval. otherwise truncate set nocount on if not exists (select 1 from sysobjects where name = 'waitstats') create table waitstats ([wait type] varchar(80).Track_waitstats code (1 of 2) CREATE proc track_waitstats (@num_samples int=10. requests numeric(20.@endtime datetime.com/info/cpyright.1).@begintime datetime .g. seconds or minutes' return end if @dt = 's' begin select @sec = @delaynum % 60 select @min = cast((@delaynum / 60) as int) select @hr = cast((@min / 60) as int) select @min = @min % 60 end .@delay varchar(8).clear out waitstats declare @i int. Davidson -.clear) -.1).
'Total'.'RESOURCE_QUEUE'.@hr). sleep. datediff(mi.2) if @hr > 23 or @min > 59 or @sec > 59 begin select 'hh:mm:ss delay time cannot > 23:59:59' select 'delay interval and type: ' + convert (varchar(10). requests.@now select 'start.@begintime=min(now). 'delay. rank by percentage descending insert into waitstats select '***total***'.@delaynum) + '.20) + ' duration (minutes): ' + convert(varchar(10). '***total***') and now = @now -.insert adjusted totals.@endtime.@begintime.percentage=cast (100*[wait time]/@totalwait as numeric(20.@begintime. end.'waitfor delay interval'=@delay select [wait type].' + @delaytype + ' converts to ' + @delay return end while (@i <= @num_samples) begin insert into waitstats ([wait type].'Total') and now = @now order by percentage desc .2) + ':' + + right('0'+convert(varchar(2). type'= convert (varchar(10).@sec). [wait time].@min). and resource_queue from Total select @totalwait = sum([wait time]) + 1 from waitstats where [wait type] not in ('WAITFOR'.1)) from waitstats where [wait type] not in ('WAITFOR'.[signal wait time]) exec ('dbcc sqlperf(waitstats)') select @i = @i + 1 waitfor delay @delay end select @now=max(now).20) + ' end: ' + convert(varchar(20).'SLEEP'.@totalwait.2) + ':' + + right('0'+convert(varchar(2).'SLEEP'.'RESOURCE_QUEUE'.[wait time].@endtime)) select 'waitstats samples'=@num_samples.Track_waitstats code (2 of 2) if @dt = 'm' begin select @sec = 0 select @min = @delaynum % 60 select @hr = cast((@delaynum / 60) as int) end select @delay= right('0'+ convert(varchar(2).@delaynum) + '.0.@endtime=max(now) from waitstats where [wait type] = 'Total' --. duration'='start: ' + convert(varchar(20).subtract waitfor.' + @delaytype.@totalwait.
Track_waitstats Sample output .
Sample Wait Stats ± Bank (Linked Server) .
Demo Walk through perfmon. track_waitstats .
or DB design. rather than resource limitations Helpful to know performance profile when things are good Now performance is bad .What changed? . SQL.Correlating Waits and Queues Waits should be corroborated with Queues Look beyond the symptom Symptom can mask the problem Identifies the specific resource constraint along with the associated wait types Actual problem could be app.
SQL 2000 Waits & Queues * coming very soon on MSDN .
Physical Disk average page Avg disk sec/read life indicates Avg disk sec/write memory Disk queues pressure e.g. cache flushing Avg Page Life Expectancy (seconds) Checkpoint pages/sec Lazywrites/sec .Correlating Waits & Queues: IO or Memory Pressure? Waits IO_Completion Async_IO_Completion PageIOLatch_x PageLatch_x Queues Explanation 1. SQL Buffer Mgr Waits indicate IO issue High Avg disk seconds indicates IO issue HOWEVER Low 2.
If Showplan shows Table scans Clustered index range scans Nonclustered index range scans Sorts 2. Physical Disk Avg disk sec/read Avg disk sec/write Disk queues . If Profiler shows: Avg Page Life Expectancy (seconds) Checkpoint pages/sec Lazywrites/sec Scan started Reads Writes 3. SQL Buffer Mgr 1. Async_IO_Completion Queues Explanation 1.Correlating Waits & Queues ± IO or DB Design? Waits 1. Writelog 2. IO_Completion 2.
Other sources of information Virtual tables Undocumented DBCC options .
g. query plans Sysprocesses Contains each SQL thread & wait types Syslocks . [fileId | -1]) Provides breakdown of IO by file Look for IostallMS compare to Physical Disk reads/writes Indentifies SQL component of IO Syscacheobjects Contains compiled objects e.Virtual System Tables ::Fn_virtualfilestats (dbid.
Concurrency/CPU Issues: Cached Objects Master. Views Defaults.Syscacheobjects Procedure or batch name Set options for plans Ref counts. rules adhoc SQL. Triggers.. Use counts Compiled plan Single copy (serial and parallel) Re-entrant and re-usable Re-comps place **lock** on compile plan Data structure for user context. Check contraints. not re-entrant Look for plan reuse: usecounts > 1 Executable plan Plan re-use of Procs. sp_executesql .
Syscacheobjects & Re-use .
.Plan re-use SQL Batch requests/sec Compare to initial SQL Compilations/sec SQL Compilations/sec Includes initial compiles AND re-compiles Eliminate re-compilations for initial compiles Look for identical SQL statements with low usecounts in syscacheobjects SQL Re-compilations/sec Just re-compiles Check profiler for sp:recompile event to identify SQL statement.
discards Index usage (or lack of) Object access . misses.Profiler events Plan re-use (or lack of) Compare batch requests to SQL compiles/sec IO Reads and writes Recompilation Cache hits.
Concurrency/CPU: Recompilation Plan determination is CPU Intensive Recomp good if benefit of new plan > CPU cost Profiler Lists recomp events and statements Data column for reason: EventSubClass locks on system tables Re-compiling stored procedure plans serialize other users during high concurrency places lock on single compile plan Re-compilation based on Rows changed thresholds (rowmodctr) DDL placement. schema changes Code practice & temp tables (P1 & P2) .
deferred check at run-time. Statistics changed. . binding or permission changed. Object not found at compile time. bindings or permissions changed between compile or execute. Set option changed in batch. binding or permission changed. Remote rowset schema. Temp table schema.EventSubClass: Reason for recompilation 1 2 3 4 5 6 Schema.
memory pressure. DB design UMS ± context Dbcc switching sqlperf(umsstats) .Uses of CPU resources Sorts Joins Lack of plan re-use Recompilation Manages IO Distincts. order by. group by Worktables. Match big jobs with CPUs Profiler Scans.types syscacheobjects usecounts Compare to batch requests Look at parameterization Compare to batch requests Index utilization? Page Life high enough? Look for BIG differences. temp tables SQL Compiles ± SQL Recompiles SQL Recompiles/sec Compare to index & IX overhead Query plans indicate joins.
Page Life Expectancy is higher.When does more memory help? Additional memory may help if«.. more memory helps performance . SQL Buffer Mgr:Page Life Expectancy is low Average time in seconds a page stays in cache SQL Server Memory Mgr: Memory Grants Pending is high SQL won¶t execute a query if there isn¶t enough memory for it If after adding.
SQL Server Scheduler ± understand it! UMS UMS UMS UMS 4 way CPU 1) 2) 3) 4) Connections get assigned to UMS UMS schedules across Processors A connection stayed on UMS for life of thread Two heavy threads will fight on the same UMS .
SQL Server Scheduler ± example Task 1 2 3 4 5 5 BCP.exe jobs 1) 2) 3) 4) 1. 5 2 3 4 UMS UMS UMS UMS 4 way CPU Each load job takes 2 minutes to run Task 2.3 and 4 each finished in 2 minutes Task 1 & 5 took closer to 4 minutes Using 4 threads instead of 5 would be a better solution for this example. 5) Monitor using DBCC SQLPERF (UMSTATS) .
Tracking UMS issues Dbcc sqlperf(umsstats.clear) Clears out UMS info run workload Dbcc sqlperf(umsstats) Look for ***BIG*** differences in context switching .
Concurrency: Helpful Lock scripts sp_blockinfo ± lists locking chain .
deletes Order imposed by index. doesn¶t fit back on page Balance with reads . New row doesn¶t fit on page Small record updated. cause memory pressure Look at Page life. Data drive block size=64K for large IO intensive apps Physical disk:split IO/sec Page splits SQL Access Methods: Page splits/sec Forwarded records SQL Access Methods: Forwarded records/sec Index maintenance Inserts.Uses of IO resources Scans Can flush cache. updates.
Use IOMETER(shareware) or just copy large files You should get ~ 120mb/sec throughput per channel If heavy updated system then make cache more write through vs read. RAID5 ± write penalty for write intensive apps . Individual disks are still the slowest component for database activity. Tranlog ± RAID10 is popular Set controller or IO Subsystem cache for log as 100% write through (unless reading log with replication or a lot of trigger activity) Set Goal for 1-2ms / write (Avg Disk sec/write) IO issues Test your SAN throughput prior to database activity. make sure data is spread across as many drives as possible.
Add more memory Check for correct plan re-use.asp Redo transaction management use correct transaction isolation levels High CPU Utilization 1. parameterization. recompilation .com/libra ry/default. see http://msdn. 2.microsoft. 4. Add correct indexes to minimize IO. Bad query plans resulting from improper indexing. High Blocking / Low concurrency 1. 2.asp?url=/library/enus/dnsql2k/html/sql_queryreco mpilation. Memory pressure Plan re-use Recompilation Parameterization 1. Database design Memory Pressure Possible remedies 1.Design Issues There are design considerations resulting from the Waits and Queues methodology. . 3. Transaction management 1. 2. 2. 3. Observation High IO waits Application issue 1.
maximizing system capabilities Figure out the problem. shifting waits & queues System performance is all about identifying (& resolving) bottlenecks.Summation Application performance all about waits and queues Workloads may vary. if it ain¶t broke«. .
Conclusions SQL Server 2000 is a high performance enterprise class database. Performance Monitoring is a tool that should be used throughout the application life cycle Identify bottlenecks in application database resource utilization CPU I/O Network Data (concurrency vs. consistency) Exploit all resources .
microsoft. Thompson ³SQL Server 2000 Recompilation´ at http://msdn. Garcia.com SQL Server 2000 Waits & Queues (part 1) SQL Magazine (October 2003) .References ³Inside SQL Server 2000´ by Kalen Delaney ³SQL Server 2000 Performance Tuning´ by Whalen.com ³SQL Server 2000 Performance Tuning using Waits & Queues Methodology´ at http://msdn. DeLuca.microsoft.