Best Perf in Products Part2 Arubin 15 Sept 2010

<Insert Picture Here>
Getting the Best MySQL Performance in Your Products:

Part 2, Beyond the Basics
Alexander Rubin
Principle Consultant MySQL
Copyright 2010 Oracle The World’s Most Popular Open Source Database
About MySQL
• Founded, first release in 1995

• Acquired by Sun in February 2008
• Acquired by Oracle in January 2010
• #1 Most Popular Open Source Database
• Market-leading customers
Oracle’s Plans for MySQL
• Fill-in Oracles database product line
• MySQL Global Business Unit

• Managed by Edward Screven, Chief Corporate Architect
• Invest in MySQL!
• “Make MySQL a Better MySQL”
• Develop, Promote and Support MySQL
• Improve engineering, consulting, and support
• MySQL Sunday at Oracle Open World
• Leverage World-Wide, 24x7 Oracle Support
• MySQL Community Edition

• Source and binary releases
• GPL license
MySQL’s Market Position
• Ubiquitous
• Over 100 million copies downloaded worldwide
• Over 12 million installations
• Over 70,000 MySQL downloads each day
• Popular
• MySQL is the number one most popular database
• 31% use MySQL as primary DB for deployed applications
(Eclipse Community Survey, 2010)
• MySQL is the 3rd most deployed database worldwide
• Behind MSFT SQL Server and Oracle (Gartner 2008)
• MySQL is the most popular open source database
• 40% of OSDB market with 25% growth (Evans Data, 2007)
• Majority of Oracle database users also use MySQL
Industry-Leading Customers
Web / Web 2.0 OEMs / ISVs
On Demand, SaaS, Hosting Telecommunications Enterprise 2.0
Rely on MySQL
Agenda
Presentation Overview
MySQL versions
Configuration
Monitoring
Queries Tuning
+ Tips
Resources
The World’s Most Popular Open Source Database
Overview
 Covers
 MySQL configuration
 Monitoring
 Query tuning
 Based on what customers are asking
 This will be technical
 No new tools required; everything you need
comes with MySQL!
 You cannot become a performance tuning
wizard in 45 minutes - PT Class is 4 day
class
– http://www.mysql.com/training/courses/performanc
e_tuning.html
 MySQL Performance Forum
– http://forums.mysql.com/list.php?24
MySQL Versions and Performance
Latest GA release: MySQL 5.1.50, InnoDB plugin
MySQL Configuration
MySQL Configuration
• Single configuration file: my.cnf (Linux/Unix/Mac),
my.ini (Windows)
• Good example (with comments): my-innodb-heavy-
4g.cnf
• Treat it as an example only
• Variables with different scope:
• Global variables
• Session variables
• Engine Specific variables
• Configure buffers
• Global, allocated per MySQL instance
• Session, allocated per each connections
• Make sense only for InnoDB or MyISAM or other engine
Table Cache
• table_cache – Defines the number of open tables for all threads that
the server should allow
• Open_table_definitions – Displays the number of cached .frm
files
• Open_tables - Displays the number of currently open tables
• Opened_tables - Displays the number of tables that have been
opened
Temporary Tables
• tmp_table_size – maximum size for in Memory
temporary tables created by MySQL
• max_heap_table_size - Sets the maximum size to
which MEMORY tables are allowed to grow
• Watch:
• Created_tmp_tables – number of temporary table MySQL
created in RAM
• Created_tmp_disk_tables - number of temporary table
MySQL created on DISK
Session Variables
• Some session variables control space allocated by
each session (connection)
• Setting these to small can give bad performance
• Setting these too large can cause the server to swap!
• Can be set by connection
• SET SORT_BUFFER_SIZE=8*1024*1024; (8M)
• Set small be default, increase in connections that need it
• sort_buffer_size - Used for ORDER BY, GROUP
• BY, SELECT DISTINCT, UNION DISTINCT
• Monitor Sort_merge_passes < 1-2 an hour optimal
• Usually a problem in a reporting or data warehouse database
• Other important session variables (better to keep small)
• read_rnd_buffer_size - Set to 1/2 sort_buffer_size
• join_buffer_size - (BAD) Watch Select_full_join
• read_buffer_size - Used for full table scans, watch
Select_scan
InnoDB Concepts: Buffer Pool
The In-Memory
buffer pool writes
and reads data
pages from the DB
server
Read Read
Buffer Database
Disk Storage Pool Server
Write
Write
The In-Memory
buffer pool writes
and reads data
pages from O/S
Internal InnoDB Buffers and Logs
Log
Buffer
Buffer
Writes to disk Pool
at COMMIT and
checkpoints
• Log buffer – Cached index

pages Writes to disk
at checkpoints
• innodb_log_buffer_size Internal
Internaldata
data
Transaction dictionary
• Transaction Log Transaction dictionary
--------------------
records, --------------------
Insert undo
records, Insert undo
redo
redolog
log logs
logs
• InnoDB will perform a dual- records,
records,etc.
etc. Update
Update
undo
undologs
logs
buffer write when a record Ib_logfile files ibdata files
is changed issued under a
transaction
innodb_flush_log_at_trx_commit
• This server variable tells when to execute a

transaction commit to flush log buffer to disk
• 0 - The logs are flushed to disk during checkpoints which occurs
approximately once per second
• 1 - The COMMIT statement initiates the flush
• 2 - The log_buffer is written to the file (an fsync occurs every second)
Log Files
• The size of each log file should be chosen to avoid

executing checkpoint operations too often
• The bigger log file size reduces disk I/O in checkpointing
• innodb_log_file_size – Defines the size of each log file in a log
group in megabytes
• The larger the value, the less checkpoint flush activity is
needed in the buffer pool, saving disk I/O
• Larger log files mean that recovery will be slower in case of
a crash
• The combined size of log files must be less than 4 GB on
32-bit computers
Other InnoDB settings
• innodb_file_per_table
• Create tablespace (.ibd) per table
• innodb_flush_method=O_DIRECT
• Prevent unnecessary OS buffering
• For MySQL 5.1.46+

• ignore-builtin-innodb
• plugin-load=innodb=ha_innodb_plugin.so
• Enable new InnoDB plugin with better performance
Storage Medium
MySQL
MySQL
Multiple Tablespaces
Data
Data
mysql
mysql
InnoDB multiple tablespaces
test
test City.ibd
City.ibd
Country.ibd
Country.ibd
world
world
Countrylanguage.ibd
Countrylanguage.ibd
Internal data Multiple
Multipletablespaces
tablespaces
Internal data
requires
Country.frm
dictionary
dictionary
--------------------
Transaction
Transaction requiresthe
the (Metadata)
--------------------
records, innodb_file_per_table
Insert innodb_file_per_table
Insertundo
undo records, clause Country.frm
logs redo
redolog
log clause
logs records, etc. (Metadata)
Update records, etc.
Update
undo logs
undo logs Countrylanguage.frm
ibdata files Ib_logfile files
(Metadata)
InnoDB configuration example
6-8G RAM, dedicated box, v.5.1.50
ignore-builtin-innodb
plugin-load=innodb=ha_innodb_plugin.so
innodb_buffer_pool_size = 4G # OR 6G
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 512M
innodb_log_files_in_group = 2
innodb_file_per_table=1
innodb_flush_method=O_DIRECT
MySQL Monitoring
MySQL monitoring
• SHOW PROCESSLIST – This command displays a real-time

list of all the connections to the server and the actions that each
one is performing
• INFORMATION_SCHEMA.PROCESSLIST -
• SHOW [GLOBAL|SESSION] STATUS – This command
provides server status information
• SHOW [GLOBAL|SESSION] VARIABLES – This
command provides the values of MySQL system variables
• SHOW ENGINE INNODB STATUS – shows innodb specific
information
MySQL monitoring: mysqladmin
• SHOW GLOBAL STATUS – provides global information
since uptime
• Real-time information:
• > mysqladmin .. -uroot -i 10 -r ex
• Will show interactive statistics per 10 seconds, example:
mysqladmin … -uroot -i 10 -r ex| grep tmp
| Created_tmp_disk_tables |0 |
| Created_tmp_files |5 |
| Created_tmp_tables | 543352 |
| Created_tmp_disk_tables |0 |
| Created_tmp_files |0 |
| Created_tmp_tables | 20 |
• 2 tmp tables per second
Slow Query Log
• Contains text of long running queries
• log queries executing longer than long_query_time server variable (in seconds, but
supports microseconds resolution when logging to file)
• Helps identify candidates for query optimization

• Enabling log
• --log-slow-queries or --log-slow-queries=file_name
• Can log non-indexed queries

• --log-queries-not-using-indexes
• Written to log file or table

• slow_log table in mysql database
• Use --log-short-format option for less verbose logging
Show InnoDB Status: Buffer Pool
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 4391436288; in additional pool allocated 0
Buffer pool size 262143
Free buffers 262124
Database pages 19
Modified db pages 0
Show InnoDB Status: Row Operations
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Main thread process no. 31744, id 1189128544, state: waiting for server activity
Number of rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Trick with InnoDB : “are we done yet?”
• Delete from a where t1 > 2000000
• How long it will take? How much rows already deleted? What is
the speed?
• 4M rows in table, deleting half of rows
• Show innodb status:
---TRANSACTION 0 97281, ACTIVE 21 sec…
1179 lock struct(s), .. undo log entries 650910
… delete from a where t1 > 2000000
• 1 undo log entries = 1 row
• Deleted 650K rows for 21 sec, avg. speed: 30K rows/sec
• 2M rows will be deleted for 66.6 sec
• 1 min 6 sec – raw estimate
• mysql> delete from a where t1 > 2000000 ;
• Query OK, 2000000 rows affected (1 min 13.64 sec)
Monitoring GUI tools
• MySQL Enterprise Monitor
• Apply for trial: http://www.mysql.com/trials/
• MySQL Workbench
• http://www.mysql.com/products/workbench/
• Plugins for Nagios
• Lots of other tools
Query Optimization
MySQL Optimizer Overview
MySQL DBMS
Query Compiler
Parser
Pars
erSyntax
Analyze
Analyze
Syntax
Preprocess –
Preprocessor
Semantic
Semantic orname resolution
checking,
checking, name
resolution
Parse
Optimizer Tree
Generate
generateOptimal
optimal Query
Query Execution Plans QEP
Execution Plans QE ’s)‫‏‬
’s)‫‏‬
(( P
Query
QueryTransformations
Transformations
Search
Searchforfor
optimal
optimal
execution
execution
plan
plan
Plan
Planrefinement
refinement Query
Execution
QEP Plan
Query execution engine
Query Performance 17.1 Query Optimization
Using Indexes for Optimization
• Benefits of indexes
• Contain sorted values
• Result in less disk I/O
• MySQL supports three general types of indexes

• Primary key
• Unique index
• Non-unique index
• Specialized indexes
• FULLTEXT
• SPATIAL
Principles for Index Creation
• Use NOT NULL if possible
• Avoid “over-indexing”
• Over-indexing may slow down writes
• Avoid creating redundant indexes
• Estimate whether indexing is efficient
• Indexes that are not selective enough will not be used (efficiently)
• Choose unique and non-unique indexes appropriately
• Make indexes as small as possible
• Index column prefix rather than entire column
• Use a single ALTER TABLE to perform creation/alteration of multiple
indexes (rebuild the table only once)
Composite Indexes
• Composite indexes facilitate quick lookup
• MySQL can use left-most part of any index
• Leftmost prefix of an index is not the same thing

as an index on a column prefix
Leftmost Index Prefixes (2/2)
• SHOW INDEX displays composite index information
mysql> SHOW INDEX FROM CountryLanguage\G
*************************** 1. row ***************************
Table: CountryLanguage
Non_unique: 0
Key_name: PRIMARY
Seq_in_index: 1
Column_name: CountryCode
Collation: A
Cardinality: NULL
Sub_part: NULL
Packed: NULL
Null:
Index_type: BTREE
Comment:
*************************** 2. row ***************************
Table: CountryLanguage
Non_unique: 0
Key_name: PRIMARY
Seq_in_index: 2
Column_name: Language
Collation: A
Cardinality: 984
Sub_part: NULL
Packed: NULL
Null:
Index_type: BTREE
Comment:
Duplicate Indexes
• Example 1
CREATE TABLE `dupl_index` (
ì` int(11) NOT NULL,
à` int(11) DEFAULT NULL,
PRIMARY KEY (ì`),
UNIQUE KEY ì` (ì`),  DUPLICATE KEY
KEY ì_2` (ì`)  DUPLICATE KEY
• Example 2
CREATE TABLE `dupl_index1` (
à` int(11) DEFAULT NULL,
`b` int(11) DEFAULT NULL,
`c` int(11) DEFAULT NULL,
KEY ì1` (à`,`b`,`c`),
KEY ì2` (à`,`b`),  DUPLICATE KEY
KEY ì3` (à`)  DUPLICATE KEY
Using EXPLAIN to Analyze Queries
• Find out how the query optimizer would improve a

SELECT query
• Useful optimizer information
• Points out need to index
• Finds out if optimizer is using existing indexes
• Can help qualify query rewrites
• Can also be used with UPDATE and DELETE
How EXPLAIN Works (2/3)
• Shows the efficiency of SELECT statements
mysql> EXPLAIN SELECT * FROM Country WHERE Name = 'France'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: Country
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 239
Extra: Using where
mysql> EXPLAIN SELECT * FROM Country WHERE Code = 'FRA'\G

*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: Country
type: const
possible_keys: PRIMARY
key: PRIMARY
key_len: 3
ref: const
rows: 1
Extra:
How EXPLAIN Works (3/3)
• Performs a “dry run” of the query

• Indicates number of rows which need to be
examined during processing the query
• EXPLAIN is especially important for join analysis
Analyzing a Query (1/9)
• Example of using EXPLAIN to analyze and optimize
a query
• Starting out with no indexes ...
mysql> DESC CountryList;
+-------+----------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+----------+------+-----+---------+-------+
| Code | char(3) | NO | | | |
| Name | char(52) | NO | | | |
+-------+----------+------+-----+---------+-------+
mysql> DESC CityList;

+-------------+----------+------+-----+---------+-------+
+-------------+----------+------+-----+---------+-------+
| CountryCode | char(3) | NO | | | |
| Name | char(35) | NO | | | |
| Population | int(11) | NO | | 0 | |
+-------------+----------+------+-----+---------+-------+
...
• Query used:
mysql> SELECT Co.Name, Ci.Name, Ci.Population
-> FROM CountryList Co, CityList Ci
-> WHERE Co.Code = Ci.CountryCode
-> AND Ci.Population > 8000000;
• Obtain the query execution plan:

mysql> EXPLAIN
-> SELECT Co.Name, Ci.Name, Ci.Population
-> FROM CountryList Co, CityList Ci
-> WHERE Co.Code = Ci.CountryCode
-> AND Ci.Population > 8000000\G
• EXPLAIN output:
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: Co
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 209
Extra:
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: Ci
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 4018
Extra: Using where; Using join buffer
• Example (continued)‫‏‬
• EXPLAIN shows the need for indexing:
• type: ALL – all rows are retrieved
• possible_keys: NULL – there are no indexes whatsoever
• Extra: Using where – criteria tested on data of each
scanned row
• Extra: Using join buffer – A row cache is used to
scan the table for each row of previous table
• Good columns to index are those used for searching, grouping
and sorting
• Use Co.Code and Ci.CountryCode to match rows
• Uses CityList.Population to cull rows
• Adding indexes:
• ALTER TABLE CountryList ADD PRIMARY KEY (Code)
+-------+----------+------+-----+---------+-------+
+-------+----------+------+-----+---------+-------+
| Code | char(3) | NO | PRI | | |
| Name | char(52) | NO | | | |
+-------+----------+------+-----+---------+-------+
• ALTER TABLE CityList ADD INDEX (CountryCode)

+-------------+----------+------+-----+---------+-------+
+-------------+----------+------+-----+---------+-------+
| CountryCode | char(3) | NO | MUL | | |
| Name | char(35) | NO | | | |
| Population | int(11) | NO | | 0 | |
+-------------+----------+------+-----+---------+-------+
• EXPLAIN after adding indexes:
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: Co
type: ALL
key: NULL
key_len: NULL
ref: NULL
rows: 225
Extra:
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: Ci
type: ref
possible_keys: CountryCode
key: CountryCode
key_len: 3
ref: world.Co.Code
rows: 7
Extra: Using where
• Indicates which indexes are used:
• possible_keys: CountryCode – there is an index called
CountryCode that might be usable
• key: CountryCode – the index called CountryCode will be
used
• key_len: 3 – the number of bytes per index entry that can
be used (less than total length in case an index prefix is used)
• Indicates how indexes are used
• type: ref – the index is scanned to look if rows match the
combination of values from the previous table
• ref: world.Co.Code – the index (or index prefix) from this
table is compared against this value to look for matches
• Adding an index for Population too changes the order:
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: Ci
type: range
possible_keys: CountryCode,Population
key: Population
key_len: 4
ref: NULL
rows: 10
Extra: Using where
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: Co
type: eq_ref
key: PRIMARY
key_len: 3
ref: world.Ci.CountryCode
rows: 1
Extra:
• For CityList:
• possible_keys: CountryCode, Population – two
indexes are considered
• key: Population – the index called Population will be used
• type: range – only a partial index scan is required
• rows: 10 – the scan likely yields only 10 rows
• For CountryList
• type: eq_ref – values from the previous table are used to
perform single row lookups in this table
• In this case the range scan drastically reduced number of rows,
reversing the join order as compared to the previous queries
EXPLAIN for Table Processing (1/2)
• The Extra column provides process information

• Can indicate an efficient or inefficient query
• Efficient query Extra output

• Using index (index sufficient to obtain all data without als looking up
table data)
• Using where (rows tested for criteria before joining with next table)
• Distinct (only unique value combinations are required instead of

scanning all rows)
• Not exists (scan for an outer joined table that is required not to exist
can be skipped once the first row is found to match)
EXPLAIN for Table Processing (2/2)
• Indicators for inefficient query Extra output
• Using filesort (extra pass to sort rows required to retrieve
actual rows in sorted order)
• Using temporary (result copied to temporary table, indicates
ORDER BY and GROUP BY with different implied row order)
• Using join buffer (repeated scans buffered, might indicate a
missing join condition or index)
• Indicators of poor performance
• Range checked for each record (a series of range scans
are performed preventing efficient caching of scanned rows)
• Just indicators – it’s not a black and white thing
How to Avoid “Using Filesort”?
• Covered (combined) index can help. Before:
CREATE TABLE `City` (
ÌD` int(11) NOT NULL AUTO_INCREMENT,
`Name` char(35) NOT NULL DEFAULT '',
`CountryCode` char(3) NOT NULL DEFAULT '',
`District` char(20) NOT NULL DEFAULT '',
`Population` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (ÌD`),
KEY `CountryCode` (`CountryCode`)
mysql> explain select name, District from City where

CountryCode ='USA' order by Population desc limit 10\G
table: City
type: ref
possible_keys: CountryCode
key: CountryCode
key_len: 3
ref: const
rows: 267
Extra: Using where; Using filesort
How to Avoid “Using Filesort”?
• After:
mysql> alter table City

add key comb_ind(CountryCode, Population);
mysql> explain select name, District from City

where CountryCode ='USA'
order by Population desc limit 10\G
...
table: City
type: ref
possible_keys: CountryCode,comb_ind
key: comb_ind
key_len: 3
ref: const
rows: 267
Extra: Using where
Full covered index
mysql> alter table City add key
covered_ind (CountryCode, Population, name, District);
mysql> explain select name, District from City

where CountryCode ='USA'
order by Population desc limit 10\G
...
table: City
type: ref
possible_keys: CountryCode,comb_ind,covered_ind
key: covered_ind
key_len: 3
ref: const
rows: 457
Extra: Using where; Using index
Event, Resources and Q&A
• MySQL Sunday at Oracle Open World, 19 Sept 2010
– http://www.oracle.com/us/openworld/mysql-sunday-078000.html
• “Getting the Best MySQL Performance in Your Products:
Part 1, The Fundamentals” replay
– http://mysql.com/news-and-events/on-demand-webinars/display-od-
552.html
• mysql.com/performance
– http://mysql.com/why-mysql/performance/index.html
• Webinar replay
– See: “Embedded Server for ISVs and OEMs” section on
http://mysql.com/news-and-events/on-demand-webinars/
• Questions?
– http://www.mysql.com/about/contact/sales.html?s=oem
– Phone: USA=+1-866-221-0634; Outside USA = +1-208-327-6494
The presentation is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.

Best Perf in Products Part2 Arubin 15 Sept 2010

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Best Perf in Products Part2 Arubin 15 Sept 2010

Uploaded by

Copyright:

Available Formats

<Insert Picture Here>

Getting the Best MySQL Performance in Your Products:

• Founded, first release in 1995

• MySQL Global Business Unit

• MySQL Community Edition

Web / Web 2.0 OEMs / ISVs

On Demand, SaaS, Hosting Telecommunications Enterprise 2.0

• Log buffer – Cached index

• This server variable tells when to execute a

• The size of each log file should be chosen to avoid

• For MySQL 5.1.46+

• SHOW PROCESSLIST – This command displays a real-time

• 2 tmp tables per second

• Helps identify candidates for query optimization

• Can log non-indexed queries

• Written to log file or table

• Use --log-short-format option for less verbose logging

BUFFER POOL AND MEMORY

Total memory allocated 4391436288; in additional pool allocated 0

Buffer pool size 262143

Free buffers 262124

0 queries inside InnoDB, 0 queries in queue

1 read views open inside InnoDB

Number of rows inserted 0, updated 0, deleted 0, read 0

0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s

Query execution engine

• MySQL supports three general types of indexes

• Composite indexes facilitate quick lookup

• MySQL can use left-most part of any index

• Leftmost prefix of an index is not the same thing

• Find out how the query optimizer would improve a

• Can also be used with UPDATE and DELETE

mysql> EXPLAIN SELECT * FROM Country WHERE Code = 'FRA'\G

• Performs a “dry run” of the query

mysql> DESC CityList;

• Obtain the query execution plan:

• ALTER TABLE CityList ADD INDEX (CountryCode)

• The Extra column provides process information

• Efficient query Extra output

• Distinct (only unique value combinations are required instead of

mysql> explain select name, District from City where

mysql> alter table City

mysql> explain select name, District from City

mysql> explain select name, District from City

You might also like