Professional Documents
Culture Documents
Server
Health
Check
Josh Berkus
PostgreSQL Experts Inc.
pgCon 2010
DATABASE SERVER
HELP 5¢
Program of Treatment
● What is a Healthy Database?
● Know Your Application
● Load Testing
● Doing a database server checkup
● hardware
● OS & FS
● PostgreSQL
● application
● Common Ailments of the Database Server
What is a Healthy Database
Server?
What is a Healthy Database
Server?
● Response Times
What is a Healthy Database
Server?
● Response Times
● lower than required
● consistent & predicable
● Capacity for more
● CPU and I/O headroom
● low server load
30
25
Median Response Time
20
Expected Load
15
0
25 50 75 100 125 150 175 200 225 250
Number of Clients
What is an Unhealthy Database
Server?
● Slow response times
● Inconsistent response times
● High server load
● No capacity for growth
30
25
Median Response Time
20
Expected Load
15
0
25 50 75 100 125 150 175 200 225 250
Number of Clients
A healthy database server is
able to maintain consistent
and acceptable response times
under expected loads with
margin for error.
30
25
Median Response Time
20
15
10
0
25 50 75 100 125 150 175 200 225 250
Number of Clients
Hitting The Wall
CPUs Floored
FATAL:
connection limit
exceeded for non-
superusers
How close are you
to the wall?
The Checkup
(full physical)
1. Analyze application
2. Analyze platform
3. Correct anything obviously wrong
4. Set up load test
5. Monitor load test
6. Analyze Results
7. Correct issues
The Checkup
(semi-annual)
1. Check response times
2. Check system load
3. Check previous issues
4. Check for Signs of Illness
5. Fix new issues
Know
your
application!
Application database usage
Which does your application do?
✔ small reads
✔ large sequential reads
✔ small writes
✔ large writes
✔ long-running procedures/transactions
✔ bulk loads and/or ETL
What Color Is My Application?
W ● Web Application (Web)
70
60
Requests Per Second
50
40
30
20
10
0
02:00:00 AM 06:00:00 AM 10:00:00 AM 02:00:00 PM 06:00:00 PM 10:00:00 PM
12:00:00 AM 04:00:00 AM 08:00:00 AM 12:00:00 PM 04:00:00 PM 08:00:00 PM
Time
80
70
DOWNTIME
60
Requests Per Second
50
40
30
20
10
0
02:00:00 AM 06:00:00 AM 10:00:00 AM 02:00:00 PM 06:00:00 PM 10:00:00 PM
12:00:00 AM 04:00:00 AM 08:00:00 AM 12:00:00 PM 04:00:00 PM 08:00:00 PM
Time
When preventing downtime,
it is not average load which
matters, it is peak load.
What to load test
● Load should be as similar as possible to your
production traffic
● You should be able to create your target level of
traffic
● better: incremental increases
● Test the whole application as well
● the database server may not be your weak point
How to Load Test
1. Set up a load testing tool
you'll need test servers for this*
2. Turn on PostgreSQL, HW, application
monitoring
all monitoring should start at the same time
3. Run the test for a defined time
1 hour is usually good
4. Collect and analyze data
5. Re-run at higher level of traffic
Test Servers
● Must be as close as reasonable to production
servers
● otherwise you don't know how production will be
different
● there is no predictable multiplier
● Double them up as your development/staging
or failover servers
● If your test server is much smaller, then you
need to do a same-load comparison
Tools for Load Testing
Production Test
1. Determine the peak load hour on the
production servers
2. Turn on lots of monitoring during
that peak load hour
3. Analyze results
● http://pgreplay.projects.postgresql.org/
tsung
● Generic load generator in erlang
● a load testing kit rather than a tool
● Generate a tsung file from your actvity logs using
pgFouine and test the database
● Generate load for a web application using custom
scripts
● Can be time consuming to set up
● but highly configurable and advanced
● very scalable - cluster of load testing clients
● http://tsung.erlang-projects.org/
pgBench
● Simple micro-benchmark
● not like any real application
● Version 9.0 adds multi-threading, customization
● write custom pgBench scripts
● run against real database
● Fairly ad-hoc compared to other tools
● but easy to set up
log_min_duration_statement = 0
log_connections = on
log_disconnections = on
log_temp_files = 100kB
log_lock_waits = on
monitoring hardware during
load test
sar -A -o load_test_1.sar 30 240
iostat
or fsstat
or zfs iostat
monitoring application during
load test
● Collect response times
● with timestamp
● with activity
● pgTop
in praise of sar
● collects data about all aspects of HW usage
● available on most OSes
● but output is slightly different
● easiest tool for collecting basic information
● often enough for server-checking purposes
● BUT: does not report all data on all platforms
sar
CPUs: sar -P ALL and sar -u
Memory: sar -r and sar -R
I/O: sar -b and sar -d
network: sar -n
sar CPU output
Linux
06:05:01 AM CPU %user %nice %system %iowait %steal %idle
06:15:01 AM all 14.26 0.09 6.01 1.32 0.00 78.32
06:15:01 AM 0 14.26 0.09 6.01 1.32 0.00 78.32
Solaris
15:08:56 %usr %sys %wio %idle
15:09:26 10 5 0 85
15:09:56 9 7 0 84
15:10:26 15 6 0 80
15:10:56 14 7 0 79
15:11:26 15 5 0 80
15:11:56 14 5 0 81
Memory
● Only one statistic: how much?
● Not generally an issue on its own
● low memory can cause more I/O
● low memory can cause more CPU time
memory sizing
In Buffer
In Cache
On Disk
Figure out Memory Sizing
● What is the active portion of your database?
● i.e. gets queried frequently
● How large is it?
● Where does it fit into the size categories?
● How large is the inactive portion of your
database?
● how frequently does it get hit? (remember backups)
Memory Sizing
● Other needs for RAM – work_mem:
● sorts and aggregates: do you do a lot of big ones?
● GIN/GiST indexes: these can be huge
● hashes: for joins and aggregates
● VACUUM
I/O Considerations
● Throughput
● how fast can you get data off disk?
● Latency
● how long does it take to respond to requests?
● Seek Time
● how long does it take to find random disk pages?
I/O Considerations
● Throughput
● important for large databases
● important for bulk loads
● Latency
● huge effect on small writes & reads
● not so much on large scans
● Seek Time
● important for small writes & reads
● very important for index lookups
I/O Considerations
● Web
● concerned about read latency & seek time
● OLTP
● concerned about write latency & seek time
● DW/BI
● concerned about throughput & seek time
------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-
Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
32096M 79553 99 240548 45 50646 5 72471 94 185634 10 1140 1
If your database
doesn't fit in RAM,
don't host it
on a public cloud
Checking Operating System
and Filesystem
OS Basics
● Use recent versions
● large performance, scaling improvements in Linux &
Solaris in last 2 years
● Check OS tuning advice for databases
● advice for Oracle is usually good for PostgreSQL
● Keep up with information about issues &
patches
● frequently specific releases have major issues
● especially check HW drivers
OS Basics
● Use Linux, BSD or Solaris!
● Windows has poor performance and weak
diagnostic tools
● OSX is optimized for desktop and has poor
hardware support
● AIX and HPUX require expertise just to install, and
lack tools
Filesystem Layout
● One array / one big pool
● Two arrays / partitions
● OS and transaction log
● Database
● Three arrays
● OS & stats file
● Transaction log
● Database
Linux Tuning
● XFS > Ext3 (but not that much)
● Ext3 Tuning: data=writeback,noatime,nodiratime
● XFS Tuning: noatime,nodiratime
– for transaction log: nobarrier
● “deadline” I/O scheduler
● Increase SHMMAX and SHMALL
● to ½ of RAM
● Cluster filesystems also a possibility
● OCFS, RHCFS
Solaris Tuning
● Use ZFS
● no advantage to UFS anymore
● mixed filesystems causes caching issues
● set recordsize
– 8K small databases
– 128K large databases
– check for throughput/latency issues
Solaris Tuning
● Set OS parameters via “projects”
● For all databases:
● project.max-shm-memory=(priv,12GB,deny)
● For high-connection databases:
● use libumem
● project.max-shm-ids=(priv,32768,deny)
● project.max-sem-ids=(priv,4096,deny)
● project.max-msg-ids=(priv,4096,deny)
FreeBSD Tuning
● ZFS: same as Solaris
● definite win for very large databases
● not so much for small databases
● Other tuning per docs
PostgreSQL Checkup
postgresql.conf: formulae
shared_buffers =
available RAM / 4
postgresql.conf: formulae
max_connections =
web: 100 to 200
OLTP: 50 to 100
DW/BI: 5 to 20
DW/BI:
work_mem AvRAM /
max_connections
postgresql.conf: formulae
Web/OLTP:
maintenance_work_mem =
Av.RAM * 16
DW/BI:
maintenance_work_mem =
AvRAM / 8
postgresql.conf: formulae
autovacuum = on
DW/BI & bulk loads:
autovacuum = off
autovacuum_max_workers =
1/2
postgresql.conf: formulae
checkpoint_segments =
web: 8 to 16
oltp: 32 to 64
BI/DW: 128 to 256
postgresql.conf: formulae
wal_buffers = 8MB
effective_cache_size =
AvRAM * 0.75
How much recoverability do
you need?
● None:
● fsync=off
● full_page_writes=off
● consider using ramdrive
● Some Loss OK
● synchronous_commit = off
● wal_buffers = 16MB to 32MB
● Data integrity critical
● keep everything on
File Locations
● Database
● Transaction Log
● Activity Log
● Stats File
● Tablespaces?
Database Checks: Indexes
select relname, seq_scan, seq_tup_read,
pg_size_pretty(pg_relation_size(relid)) as size, coalesce(n_tup_ins,0)
+ coalesce(n_tup_upd,0) + coalesce(n_tup_del,0) as update_activity
from pg_stat_user_tables where seq_scan > 1000 and
pg_relation_size(relid) > 1000000 order by seq_scan desc limit 10;
index | table
acct_acctdom_idx | accounts
hitlist_acct_idx | hitlist
hitlist_number_idx | hitlist
custom_field_acct_idx | custom_field
user_log_accstrt_idx | user_log
user_log_idn_idx | user_log
user_log_feed_idx | user_log
user_log_inbdstart_idx | user_log
user_log_lead_idx | user_log
Database Checks:
Large Tables
Application
Middleware
PostgreSQL
OS
HW
Check PostgreSQL Drivers
● Does the driver version match the PostgreSQL
version?
● Have you applied all updates?
● Are you using the best driver?
● There are several Python, C++ drivers
● Don't use ODBC if you can avoid it.
● Does the driver support cached plans & binary
data?
● If so, are they being used?
Check Caching
Check Caching
● Does the application use data caching?
● what kind?
● could it be used more?
● what is the cache invalidation strategy?
● is there protection from “cache refresh storms”?
● Does the application use HTTP caching?
● could they be using it more?
Check Connection Pooling
● Is the application using connection pooling?
● all web applications should, and most OLTP
● external or built into the application server?
● Is it configured correctly?
● max. efficiency: transaction / statement mode
● make sure timeouts match
Check Query Design
● PostgreSQL does better with fewer, bigger
statements
● Check for common query mistakes
● joins in the application layer
● pulling too much data and discarding it
● huge OFFSETs
● unanchored text searches
Check Transaction
Management
● Are transactions being used for loops?
● batches of inserts or updates can be 75% faster if
wrapped in a transaction
● Are transactions aborted properly?
● on error
● on timeout
● transactions being held open while non-database
activity runs
Common Ailments
of the
Database Server
Check for them,
monitor for them
● ailments could throw off your response time
targets
● database could even “hit the wall”
● check for them during health check
● and during each checkup
● add daily/continuous monitors for them
● Nagios check_postgres.pl has checks for many of
these things
Database Growth
● Checkup:
● check both total database size and largest table(s)
size daily or weekly
● Symptoms:
● database grows faster than expected
● some tables grow continuously and rapidly
Database Growth
● Caused By:
● faster than expected increase in usage
● “append forever” tables
● Database Bloat
● Leads to:
● slower seq scans and index scans
● swapping & temp files
● slower backups
Database Growth
● Treatment:
● check for Bloat
● find largest tables and make them smaller
– expire data
– partitioning
● horizontal scaling (if possible)
● get better storage & more RAM, sooner
Database Bloat
-[ RECORD 1 ]+-----
schemaname | public
tablename | user_log
tbloat | 3.4
wastedpages | 2356903
wastedbytes | 19307749376
wastedsize | 18 GB
iname | user_log_accttime_idx
ituples | 941451584
ipages | 9743581
iotta | 40130146
ibloat | 0.2
wastedipages | 0
wastedibytes | 0
wastedisize | 0 bytes
Database Bloat
● Caused by:
● Autovacuum not keeping up
– or not enough manual vacuum
– often on specific tables only
● FSM set wrong (before 8.4)
● Idle In Transaction
● Leads To:
● slow response times
● unpredictable response times
● heavy I/O
Database Bloat
● Treatment:
● make autovacuum more aggressive
– on specific tables with bloat
● fix FSM_relations/FSM_pages
● check when tables are getting vacuumed
● check for Idle In Transaction
Memory Usage Growth
00:00:01 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
01:00:00 0 0 100 0 0 100 0 0
02:00:00 0 0 100 0 0 100 0 0
03:00:00 0 0 100 0 0 100 0 0
04:00:00 0 0 100 0 0 100 0 0
Copyright 2010 Josh Berkus & PostgreSQL Experts Inc. Distributable under the creative
commons attribution license,
except for 3rd-party images which are property of their respective owners.