SQL Server Performance Internals

for DBA’s
Gert E.R. Drapers
Software Architect
SQL Server Development
Performance
When under load, every system has a bottleneck
that limits performance
In real systems as utilization  100%, other factors
provide back pressure (for example: Only so many
connections to service)
May depend on workload, time of day
For example OLTP, Decision Support and Batch may
have different profiles and bottlenecks
Everything slows down due to bottleneck; fix one
bottleneck, and another surfaces under load
CPU, I/O, Locking (concurrency), Network
In a perfect world, CPU is the last bottleneck!
So, how do you find the bottleneck?
Look for longest queues and response times…
Scalability Rules
Database scalability is limited by the
maximum throughput of the transaction log
Disk I/O
Instance scalability is limited by shared
“process” level resources
Memory
Server scalability is limited by shared
“server”/”machine” level resources
CPU (incl. L1 & L2 cache)
Network bandwidth
Architecture TDS Packets
Network Libraries
Relational Engine
Language RPC
Open Data Services (ODS)
Events Events
User Mode Scheduler (UMS)

Parser
Expression Manager
Query SQL
Compiler
Optimizer Manager
SQL Executor
Normalizer

Storage Engine Access Methods Manager

Log Transaction Lock Page Index Row
Manager Manager Manager Manager Manager Manager

File Manager Buffer Manager

Data / Procedure Cache

Windows I/O Manager

Databases
User
User
User
User
UserDBs
User tempdb
TempDB master
Master msdb
MSDB
How to determine bottlenecks?
Distinct Queues and Waits
Queues
Outstanding (unfulfilled) resource requests
Performance Monitor counters, measure resource utilization
(e.g. Avg Disk Queue Length)
SQL Waits
Anytime a SQL thread is not executing, a wait type is set
reflecting the reason for waiting
Waits from an Application or User connection perspective
76 distinct wait types for SQL Server 2000
Queues and Waits are complementary information
Together, explain application performance
Both are essential for problem determination
Performance Monitor counters provide a view of
system performance from a resource standpoint
Wait types and sysprocesses
If a worker thread is not currently executing, a wait
type or state is set in sysprocesses
sysprocesses contains:
lastwaittype
waitype
waittime
Example:
select spid, blocked, waittype, waittime, lastwaittype
from master.dbo.sysprocesses
where spid > 51
Limitation: Transience of sysprocesses, spid
history
 Wait types and dbcc sqlperf()
dbcc sqlperf(waitstats, [clear])
Cumulative wait types and times
Limitations: Relevance of types and what you
can control?
DBCC SQLPERF (WAITSTATS)
Wait Type Requests Wait Time Signal Wait Time
----------------------- ------------------------ ------------------------ ----------------
MISCELLANEOUS 0.0 0.0 0.0
LCK_M_SCH_S 0.0 0.0 0.0
LCK_M_SCH_M 0.0 0.0 0.0
LCK_M_IX 0.0 0.0 0.0
SLEEP 804037.0 8.0404858E+8 8.0404858E+8
IO_COMPLETION 132.0 20763.0 3201.0
ASYNC_IO_COMPLETION 4.0 4109.0 0.0
RESOURCE_SEMAPHORE 0.0 0.0 0.0
DTC 0.0 0.0 0.0
OLEDB 1.0 62.0 0.0
PIPELINE_INDEX_STAT 0.0 0.0 0.0
PIPELINE_LOG 0.0 0.0 0.0
PIPELINE_VLM 0.0 0.0 0.0
WRITELOG 14.0 2407.0 15.0
PSS_CHILD 0.0 0.0 0.0
CMEMTHREAD 24.0 0.0 0.0
CXPACKET 0.0 0.0 0.0
PAGESUPP 0.0 0.0 0.0
SHUTDOWN 0.0 0.0 0.0
WAITFOR 0.0 0.0 0.0
dbcc sqlperf()
Syntax:
dbcc sqlperf (<key>) tableresults, no_infomsgs
Options:
umsstats – SQL thread management (*)
waitstats – resources, wait types (*)
iostats – outstanding reads & writes (*)
rastats – read ahead activity (*)
threads – I/O / CPU / memory usage per thread
(*) = Undocumented!
Resetting the counters:
dbcc sqlperf (<key>, clear) with no_infomsgs
Wait types explained
See
http://SQLDev.Net/misc/waittypes.htm
KB article 822101
Correlating Waits & Queues:
I/O or Memory Pressure?
Waits Queues Explanation
IO_Completion 1. SQL Buffer Mgr Waits indicate I/O issue
Async_IO_Completion Avg Page Life High Avg disk seconds
PageIOLatch_x Expectancy (seconds) indicates I/O issue
PageLatch_x Checkpoint HOWEVER Low average
pages/sec page life indicates
Lazywrites/sec memory pressure e.g.
cache flushing
2. Physical Disk
Avg disk sec/read
Avg disk sec/write
Disk queues
Correlating Waits & Queues:
I/O or DB Design?
Waits Queues Explanation
1. IO_Completion 1. SQL Buffer Mgr 1. If Profiler shows:
Avg Page Life Scan started
2. Async_IO_Completion Expectancy (seconds) Reads
Checkpoint Writes
pages/sec
3. Writelog 2. If Showplan shows
Lazywrites/sec
Table scans
2. Physical Disk
Clustered index
Avg disk sec/read range scans
Avg disk sec/write Nonclustered index
Disk queues range scans
Sorts
dbcc sqlperf(umsstats)
SQL Server implements its own cooperative scheduler
UMS = User Mode Scheduler
Underneath uses native Win32/64 threads or fibers
Fiber support is enable via sp_configure
exec sp_configure 'lightweight pooling', [0|1]
dbcc sqlperf (umsstats) will show UMS statistics including
runnable queue length
Output dbcc sqlperf(umsstats)
Statistic Value
Scheduler ID 0.0
num users 8.0
num runnable 0.0
num workers 6.0
idle workers 3.0
work queued 0.0
cntxt switches 383453.0
cntxt switches(idle) 1807557.0
dbcc sqlperf(umsstats)
Item Definition
num users Number of SQL threads on the scheduler

num runnable Number of SQL threads that are “runnable”

(Scheduler queue length)
num workers Number of actual workers to process threads
(thread pool)
idle workers Number of idle workers

cntxt switches Context switches between runnable threads

cntxt switches(idle) Context switches to the “idle” thread

Disk I/O
Determine I/O pattern
Write
Transaction Log (~100% sequential)
Lazy Writer (random)
Read
Random
Sequential
Establish disk I/O baseline or SLA outside
SQL Server, using:
SQLIOStress or IOMeter (Intel, public domain)
Special cases:
Transaction log
tempdb contention (see KB 328551)
 ::fn_virtualfilestats
::fn_virtualfilestats (dbid, [fileId | -1])
Provides breakdown of physical I/O by file, SQL
Server I/O only!
Look for IostallMS, compare with reads/writes
Compare SQL I/O to Performance Monitor
PhysicalDisk:Disk Reads/sec
PhysicalDisk:Disk Writes/sec
PhysicalDisk:Disk Read Bytes/sec
-- DbId PhysicalDisk:Disk Write Bytes/sec
= -1 == all databases
-- FileId = -1 == all files
declare @dbid int
select @dbid = db_id('pubs')
select DbId, FileId, TimeStamp, NumberReads, NumberWrites, BytesRead,
BytesWritten, IoStallMS
from ::fn_virtualfilestats(@dbid, -1)
Memory
Understand your memory models:
Default 2GB Virtual Memory
/3GB
Physical Address Extension (PAE) and Address
Windowing Extensions (AWE), address memory
between 4  48GB range
See KB articles:
283037
274750
329914
811891
Memory…
When using AWE who can benefit?
Only the Buffer Cache!
At a cost of increase kernel CPU time
Each memory access is effectively a double operation
One to bring memory from upper memory into lower
memory window
Second from memory window to application
Note: When addressing more then 16GB of
AWE memory, you can no longer use the
/3GB option, because memory pressure in
OS address space become too high!
Time to evaluate a 64-bit solution
Memory…
How to determine memory problems
Heavy OS swapping
“Memory : Page faults/sec”
Buffer cache problems
Low “Buffer Manager : Buffer cache hit ratio”
Low “Buffer Manager : Page Life Expectancy”
Procedure cache problems:
“SQL Statistics : Batch Request/sec” vs. “SQL
Statistics : SQL Compilations/sec”
Misc. DBCC commands
dbcc memorystatus
cache broken down: data, stolen, memory grants
outstanding, procs, query (compile), locks, latches etc.
dbcc proccache
total proc objects
dbcc cachestats
breakdown of proc cache: hit rates, compiled objects,
compiled plans & execution contexts
dbcc dropcleanbuffers
Clear out cache, “cold” cache without recycle
dbcc freeproccache
Clear out proc cache plans
 Common (design) issues…
Observation Application issue Possible remedies
High IO waits 1. Database design 1. Bad query plans resulting
2. Memory Pressure from improper indexing.
2. Add correct indexes to
minimize IO.
3. Add more memory
High CPU Utilization 1. Memory pressure 1. Check for correct plan re-use,
2. Plan re-use parameterization, re-
compilation , see http://
3. Recompilation msdn.microsoft.com/library/
4. Parameterization default.asp?url
=/library/en-us/dnsql2k/html
/sql_queryrecompilation.asp

High Blocking / Low 1. Transaction 1. Redo transaction

concurrency management management
2. use correct transaction
isolation levels
Summary
Performance management is a key part of the complete
application life cycle
Use it to identify bottlenecks in: application, database and resource
utilization
Maximize performance by exploiting all resources: CPU
I/O, memory, network and data (concurrency vs. consistency)
Application performance is all about waits and queues
Workloads may vary, shifting resource usage and utilization
System performance is all about identifying and resolving
bottlenecks, maximizing system capabilities

“A well performing system, is a balanced system!”

Resources
KB Articles:
Q146005: Optimizing Windows NT for Perf.
Q110352: Optimizing SQL Server Performance (bit outdated)
Q314648: Differences in STATISTICS IO, SQL Profiler and sysprocesses
Q322883: Microsoft SQL Server 7.0 Storage Engine Capacity Planning
Tips
Q322873: SAP R/3 Performance Tuning Guide for Microsoft SQL
Server 7.0
Q283696: Job to Monitor SQL Server 2000 Performance and Activity
Q283784: How to View SQL Server 2000 Activity Data
Q283725: How to View SQL Server 2000 Blocking Data
Q283886: How to View SQL Server 2000 Performance Data
Q283786: How to Monitor SQL Server 2000 Traces
Q279113: FILE: Use xpvmlog to Dump the Layout of Virtual Memory in
SQL Server
MSDN (white papers)
SQL Server 2000 Recompilation
SQL Server 2000 Performance Tuning using Waits & Queues
Methodology
 © 2004 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.