ÌnnoDB Scalability Limits

Peter Zaitsev, Vadim
Tkachenko
Percona Ìnc
MySQL Users Conference
2008
April 14-17, 2008
-2-
Who are the Speakers ?
· Founders of Percona Ìnc
÷ MySQL Performance and Scaling consulting company
· Active MySQL Community Members
· Writers http://www.mysqlperformanceblog.com
· Co-Authors of High Performance MySQL Second
Edition
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
What we'll talk about
· MySQL's main transactional storage engine ÌnnoDB
· Looking into ÌnnoDB Scalability Limits
÷ And how to solve them (or work around them)
· Check into other aspects affecting ÌnnoDB
Performance
· Focus on existing codebase in 5.0 and 5.1
· Mark Callaghan may already have fixed some of the
issues
· Provide benchmark results to support most of our
claims.
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Kinds of Scalability
· Scalability means different things to different people
· Upwards and Downwards in terms of hardware
÷ We're not really interested in the downwards part
· We look at Scaling application first
÷ And what ÌnnoDB needs to have in order to support it
· This comes down to a number of ÌnnoDB scaling
primitives.
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Scalability for your Boss
· For Product Manager scalability means Application
can Perform as it Grows.
÷ What does Perform mean ?
÷ What does Grow mean ?
· And typically these demands come with budget
constraints
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
What does Performs mean
· Response time is "Decent¨
÷ No pages take 30 seconds to load
÷ Easy to measure in production
· There is enough system capacity
÷ System can take the load without degrading in response
time and failing
÷ Often estimated (benchmarked)
· Operations can manage the system
÷ You can perform backups, table maintenance etc
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Application Growth Parameters
· Load (think spikes)
÷ Growing from 100 q/sec to 1000
· Database Size
÷ Going from 10GB to 100GB data size
· Data Distribution
÷ Going from 10 friends in average to 250
· Launching new features or changes to the old one
÷ Full text search feature can be heavy add on
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Application Scaling Challenge
· As application grows all parameters tend to grow at
once
÷ And you have to deal with larger number of more complex
queries on large database size.
· To maintain performance in these conditions you
÷ Optimize "application¨
· Schema, Queries, Caching, Sharding, Replication
÷ Get more hardware
· We focus on Scaling by Platform Upgrades in this
presentation
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
What do we mean by Platform
· Hardware
· Operating System
· MySQL Server
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Testing by Micro Benchmarks
· Micro Benchmarks correspond to some simple
operations.
· Tend to stress some particular aspect of behavior
· Ìf problem is seen in micro benchmarks it is seen in
some applications
÷ Though applications tend to expose wider range of scaling
problems
· Micro benchmarks may take things to extreme
÷ Showing issue at larger scale than application would have
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Operational Ìssues
· Often forgotten about
· Descriptive nature and does not need to be
benchmarked
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Large Tables
· Large Tables (and instances are hard to deal)
· Backup ÷ physical backup is must.
· Can't move separate tables between servers
physical way
· No REPAIR functionality
÷ Corruption often means dump and restore
÷ Restore from physical backup is often faster
· ALTER TABLE is very slow
÷ Master-Master replication can help
· Table maintenance OPTIMIZE TABLE
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Online Ìndex creation in ÌnnoDB
· New plugin just announced today by Ken and Heikki
÷ We had early access to the code
· ÌnnoDB can now build indexes without rebuilding
whole table
÷ Ìndex build done by sort which is much faster
· 10 times faster load and better index sizes
÷ But only if you load data and add indexes separately
Load Method Load Time Data Size Index Size
SQL Dump 88m 1333788672 1867513856
LOAD INFILE 90m 1333788672 1867513856
ALTER from MYISAM 90m 1333788672 1867513856
LOAD + ADD INDEX 3m+5m 1333788672 1124073472
SQL Dump MyISAM 3m 1050000000 312579072
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
How good are Sorted Ìndexes
· Ìndexes built by sort can be better physically sorted
and have better fill factor.
· Full Ìndex Scan speed (cold)
÷ 31 sec standard vs 22 sec built by sorting
· Becomes CPU bound at this stage, could be even better
· Update:
÷ update sampIe set c=md5(i) where i%1000=1;
· 3 min 20 sec standard vs 8 min 16 sec sorted
÷ Ìndex size growth:
· 0% (standard) vs 30% (sorted)
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Many tables
· You may be escaping from large tables by creating
many small tables instead
÷ ÌnnoDB keeps all table it accessed open
· Can consume a lot of memory (reduced in 5.1)
· Lesser issue with modern 64bit platforms
÷ innodb_fiIe_per_tabIe=1
· Can get small tables to use more space
· Crash recovery is a problem with many tables.
÷ "Warmup¨ is a problem
· Only one table can be opened at a time (5.0)
· Stats update on open makes it quite slow
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Large Buffer Pool Size
· Ìn general the more memory you can get for buffer
pool the better it is.
· Watch out for Warmup
÷ Larger buffer pool means longer warmup time
÷ 32GB Buffer pool will take an hour to fill
· reading 600 pages/sec
· SHOW STATUS and SHOW ÌNNODB STATUS gets
expensive
÷ Ìt takes a global lock to count dirty pages
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Large Buffer Pool
· Clean shutdown time may be long
÷ Buffer Pool needs to be flushed
÷ Set innodb_max_dirty_pages_pct=0 in advance
· Checkpointing activity may cause uneven
performance
÷ The "dip¨ is often longer and deeper with large buffer pool
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Replication Speed
· Replication can get behind much earlier when
Master or Slave are saturated
÷ Becoming more problem with Multi-Core CPUs
÷ Or if you do not have RAÌD with BBU on the slave
· Limited by Transaction Commit speed
÷ Set innodb_fIush_Iog_at_trx_commit=2
· Replication is Asynchronous anyway
· Limited by Update execution Speed
÷ Disk ÷ Prefetching may help
÷ CPU ÷ Check row level replication in 5.1+
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Lets do some Benchmarks
Take Results with grain of salt
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Workload Scalability 5.0
· MySQL 5.0.51a
· Dell PE 2950
· 2* Quad Core CPUs
÷ Ìntel Xeon L5335
· CPU Bound
· Scaling depends on
workload a lot
1 4 16 64 256
0
1
2
3
4
5
6
7
1
2.51
2.42
1.56
1.19
1
2.85
4.29
4.25
4.21
1
3.53
5.95
4.73
3.63
1
2.66
3.81 3.8 3.8
1
2.2
1.87
1.1 1.09
KEY_RANGE KEY_POÌNT_ÌND PK_POÌNT_ÌNDX
FTS KEY_POÌNT
· Scaling factor for different number of threads
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Workload Scalability 5.1
· MySQL 5.1.23-rc
· Everything same but
MySQL Version
· We can see serious
regressions for some
workloads
1 4 16 64 256
0
1
2
3
4
5
6
1
2.19
1.28
0.88
0.3
1
2.38
1.43 1.44
1.38
1
3.02
5.26
4.19
3.24
1
2.73
3.92 3.92 3.91
1
1.87
1.04
0.73
0.24
KEY_RANGE KEY_POÌNT_ÌND PK_POÌNT_ÌNDX
FTS KEY_POÌNT
· Scaling factor for different number of threads
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Single thread performance
· Performance for one
thread
· Scaling factor alone is
not conclusive
· Actual results too
different ÷ use ratios to
5.0.22 instead
· 5.0.22 and 5.0.51 are
very close
· 5.1 shows some
regressions
KEY_RANGE KEY_POÌNT_ÌND PK_POÌNT_ÌNDX FTS KEY_POÌNT
0
0.2
0.4
0.6
0.8
1
1.2
1 1 1 1 1
0.97
1
0.99 0.99 0.99
0.9
0.82
1.04
0.96
0.94
5.0.22 5.0.51a 5.1.23
· Relative Performance of MySQL versions
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
MySQL Versions Scalability
· Scaling factor for 64
threads
· MySQL 5.0.51 overall
best results
· Good improvements
from 5.0.22
· 5.1 shows serious
regressions
KEY_RANGE KEY_POÌNT_ÌND PK_POÌNT_ÌNDX FTS KEY_POÌNT
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0.2
2.42
4.48
2.49
0.18
1.61
4.23
4.72
3.78
1.14
0.88
1.44
4.19
3.93
0.73
5.0.22 5.0.51a 5.1.23
· Scaling factor for different MySQL versions
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
innodb_thread_concurrency
· Performance for 64
threads
· MySQL 5.1.23
· Using
innodb_thread_concurre
ncy=0 as baseline
· No perfect value ÷
different workloads
behave differently
KEY_RANGE KEY_POÌNT_ÌND PK_POÌNT_ÌNDX FTS KEY_POÌNT
0
0.5
1
1.5
2
2.5
1 1 1 1 1
2.07
0.53
1
0.7
2.21
2.06
0.49
1.05
0.98
2.03
innodb_thread_con
currency=0
innodb_thread_con
currency=4
innodb_thread_con
currency=8
· innodb_thread_concurrency affects performance
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Fixing scaling by CPU Affinity
· Performance for 16
threads
· MySQL 5.1.23
÷ 5.0.51a for comparison
· Workloads which scaled
worse on 5.1.23
· Trying to bind MySQL to
specific CPU Cores
· Restricting can help
scaling
KEY_POÌNT_ÌNDX KEY_POÌNT KEY_RANGE_ÌNDX KEY_RANGE
0
0.5
1
1.5
2
2.5
3
3.5
4
1 1 1 1
0.97
1.72
0.91
2.39
1.16
1.72
1.04
2.65
0.97
1.89
0.61
1.68
0.99
1.28
0.61
1.68
3.63
1.93
2.27
3.01
4x4 2x2 4x0 1x1 2x0 5.0.51
· How binding to CPUs affects performance
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Quadcore vs Dual Core
· MySQL 5.1.23rc
· Worst scaling patterns
· For 2 out of 3 query
patterns Dual core
system scales and
performs much better
1 4 16 64 256
0
0.5
1
1.5
2
2.5
3
3.5
1
1.87
1.04
0.73
0.24
1
2.07
1.89
1.75
1.45
1
2.19
1.28
0.88
0.3
1
2.44
2.16
2
1.68
1
2.61
2.89
2.94
2.84
1
2.72
2.78
2.73
2.63
POÌNT 8 POÌNT 4 RANGE 8
RANGE 8 RANGE_ÌDX 8 RANGE_ÌDX 4
· Scaling factor for different number of threads
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Large Pages
· MySQL can use Large
Pages for ÌnnoDB Buffer
Pool
· Huge Pages reduce TLB
cache misses
· Non Swappable
· Mediocre results for this
workload, may be better
in case of skewed
working set
1 4 16 64
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.2
1
1.05 1.05
1.08
1.02
1 1
0.96
1 1 1
FTS KEY_RANGE POÌNT_ÌNDEX
· Performance effect of using Large Pages
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
ÌnnoDB ÌO Scalability
· Dell PowerEdge
2950, Perc6, CentOS
5.0
· RAÌD1 vs RAÌD10 (6
disk)
· SysBench MySQL
Test workload
RAÌD1 RAÌD10
0
20
40
60
80
100
120
140
160
21
62
42
139
RW RO
· Performance effect of using Large Pages
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
EXT3 vs EXT2
· Same Hardware
· RAÌD10
· Ext3 does worse on
writes (journaling
overhead) but better
with reads
EXT2 EXT3
0
20
40
60
80
100
120
140
160
69
62
129
139
RW RO
· Performance effect of using Large Pages
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Linux ÌO Schedulers
· Elevators improved
÷ Difference is not as
huge as years ago
· CFQ (default) is best
for this workload on
this box
CFQ DEADLÌNE AS NOOP
0
20
40
60
80
100
120
140
160
62
63
56
60
139
116
112
137
RW RO
· Performance effect of using Large Pages
InnoDB Scalability Limits. MySQL Users ConIerence. Santa Clara.CA April 14-17 2008
Thanks for Coming
· Time for Questions
· Write us
÷ pz@percona.com vadim@percona.com
· Come and visit us
÷ For Ìnformation: http://www.mysqlperformanceblog.com
÷ For Business: http://www.percona.com