Applied Partitioning and

Scaling Your (OLTP)
Database System
Presented by,
MySQL AB® & O’Reilly Media, Inc.

Phil Hildebrand
phil.hildebrand@gmail.com
thePlatform for Media, Inc.

Objectives

 Review classic uses of database partitioning
 Applying partitioning to MySQL OLTP applications
 Hash partitioning with MySQL OLTP applications
•Implementation examples
 Q&A

Classic Partitioning
 Old School – union in the archive tables
 Auto partitioning and partition pruning
 Lends itself to Data Warehouses
•Archival and Date based partitioning
•Predictable growth patterns

 Benefits within Data Warehouses
•Maintenance benefits
•Query performance improved

Applying Partitioning to OLTP
 Design Issues
•Often id driven access vs. date driven access
•Difficulties in estimating partition ranges / sizes
•Intelligent keys increase complexity in partitions

 Operational Issues
•Difficult to schedule downtime for DDL changes
•General lack of use outside of data warehousing

Applying Partitioning to OLTP

 Understanding the Benefits
•Reducing seek and scan set sizes
•Limiting insert / update transaction durations
•Creates additional options for Maint processes

Reducing scan/seek set sizes
mysql> explain partitions select

my_store.city,my_employee_old.name

from

my_store, my_employee_old where my_store.id in (5,8,10)

and

my_store.id = my_employee_old.store_id and

my_employee_old.id =

(ROUND(RAND()*50000,0));

+---+-------------+-----------------+------------+-------+---------------+---------+---------+------+-------+--------------------------------+
|id | select_type | table

| partitions | type

| possible_keys | key

| key_len | ref

| rows

| Extra

|

+---+-------------+-----------------+------------+-------+---------------+---------+---------+------+-------+--------------------------------+
| 1 | SIMPLE

| my_store

| p5,p8,p10

| 1 | SIMPLE

| my_employee_old | NULL

| range | PRIMARY

| PRIMARY | 8

| NULL |

| ALL

| NULL

| NULL | 47483 | Using where; Using join buffer |

| NULL

| NULL

3 | Using where

|

+_--+-------------+-----------------+------------+-------+---------------+---------+---------+------+-------+--------------------------------+

mysql> explain partitions select

my_store.city,my_employee.name

from

my_store, my_employee where my_store.id in (5,8,10)

and

my_store.id = my_employee.store_id and

my_employee.id =

(ROUND(RAND()*50000,0));

+----+-------------+-------------+------------+-------+---------------+---------+---------+------+------+--------------------------------+
| id | select_type | table

| partitions | type

| possible_keys | key

| key_len | ref

| rows | Extra

|

+----+-------------+-------------+------------+-------+---------------+---------+---------+------+------+--------------------------------+
|

1 | SIMPLE

| my_store

| p5,p8,p10

|

1 | SIMPLE

| my_employee | p5,p8,p10

| range | PRIMARY

| PRIMARY | 8

| NULL |

| ALL

| NULL

| NULL | 2979 | Using where; Using join buffer |

| NULL

| NULL

3 | Using where

|

+----+-------------+-------------+------------+-------+---------------+---------+---------+------+------+--------------------------------+

Simple join with out partitions
$ time mysqlslap -u root --create-schema=conf --query=sel_store_employee_old.sql -c 5 -i 1000 -F ";"

Benchmark
Average number of seconds to run all queries: 0.141 seconds
Minimum number of seconds to run all queries: 0.101 seconds
Maximum number of seconds to run all queries: 0.213 seconds
Number of clients running queries: 5
Average number of queries per client: 1

real

2m22.018s

user

0m0.217s

sys

0m0.445s

Simple join with partitions
$ time mysqlslap -u root --create-schema=conf --query=sel_store_employee.sql -c 5 -i 1000 -F ";"

Benchmark
Average number of seconds to run all queries: 0.006 seconds
Minimum number of seconds to run all queries: 0.005 seconds
Maximum number of seconds to run all queries: 0.025 seconds
Number of clients running queries: 5
Average number of queries per client: 1

real

0m6.660s

user

0m0.133s

sys

0m0.306s

Rebuilding by partition
mysql> optimize table my_employee_old;
+----------------------+----------+----------+----------+
| Table

| Op

| Msg_type | Msg_text |

+----------------------+----------+----------+----------+
| conf.my_employee_old | optimize | status

| OK

|

+----------------------+----------+----------+----------+
1 row in set (1.14 sec)
mysql> alter table my_employee rebuild partition p1;
Query OK, 0 rows affected (0.03 sec)
Records: 0

Duplicates: 0

Warnings: 0

mysql> alter table my_employee rebuild partition
p1,p2,p3,p4,p5,p6,p7,p8,p9,p10;
Query OK, 0 rows affected (0.27 sec)
Records: 0

Duplicates: 0

Warnings: 0

Applying Partitioning to OLTP
 Design Considerations
•Table sizes and predicted growth patterns
•Access patterns
•Keys and indexes
•Availability and Scalability requirements
•Manageability considerations
•Reuse considerations

Choosing a Partitioning Method
 Range Partitioning
•Data usually accessed by date
•Limited number of (primary) partitions needed
•Ordered Intelligent keys
•Supports Sub Partitions

 List Partitioning
•Grouping data in partitions out of order (1,5,7 in partition x)
•Limited number of (primary) partitions needed
•Intelligent keys
•Supports Sub Partitions

 Hash Partitioning
•Low maintenance
•Works with limited or large number of partitions
•Non-intelligent keys (can work with some cases of intelligent keys)

 Key Partitioning
•Non-integer based partitioned keys (md5 hash)
•Low maintenance

Hash Partitioning and OLTP
 Applying a hash to the partitioning key
•Hash Partitions
•Key Partitions

 Fixed number of partitions
•Number of partitions determined by hash (mod%num_partitions)

My Retail Store App
mysql> show columns from my_store;

mysql> show columns from my_inventory;

+---------+--------------+------+-----+---------+

+----------+-------------+------+-----+---------+

| Field

| Field

| Type

| Null | Key | Default |

| Type

| Null | Key | Default |

+---------+--------------+------+-----+---------+

+----------+-------------+------+-----+---------+

| id

| bigint(20)

| city

| NO

| PRI | NULL

|

| id

| bigint(20)

| NO

| PRI | NULL

|

| varchar(128) | YES

|

| NULL

|

| store_id | bigint(20)

| NO

| PRI | NULL

|

| country | varchar(128) | YES

|

| NULL

|

| name

|

| NULL

|

+---------+--------------+------+-----+---------+

mysql> show columns from my_employee;
+----------+-------------+------+-----+---------+
| Field

| Type

| Null | Key | Default |

+----------+-------------+------+-----+---------+
| id

| bigint(20)

| NO

| PRI | NULL

|

| store_id | bigint(20)

| NO

| PRI | NULL

|

|

|

| name

| varchar(56) | YES

| NULL

+----------+-------------+------+-----+---------+

| varchar(56) | YES

| in_stock | bit(1)

| YES

|

| NULL

|

| on_order | bit(1)

| YES

|

| NULL

|

| item_cnt | bigint(20)

| YES

|

| NULL

|

| cost

| YES

|

| NULL

|

| float

+----------+-------------+------+-----+---------+

Applying Hash Partitioning
 Partition on Store ID
mysql> ALTER TABLE MY_STORE PARTITION BY HASH (id) PARTITIONS 50 ;
Query OK, 50 rows affected (0.76 sec)
Records: 50 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE MY_EMPLOYEE PARTITION BY HASH (store_id) PARTITIONS 50 ;
Query OK, 50000 rows affected (25.28 sec)
Records: 50000 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE MY_INVENTORY PARTITION BY HASH (store_id) PARTITIONS 50 ;
Query OK, 250000 rows affected (2 min 8.32 sec)
Records: 250000 Duplicates: 0 Warnings: 0

Splitting Partitions
 Expanding into Australia with 2 new stores:
mysql> ALTER TABLE MY_STORE ADD PARTITION PARTITIONS 2;
Query OK, 0 rows affected (0.86 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE MY_EMPLOYEE ADD PARTITION PARTITIONS 2;
Query OK, 0 rows affected (2.43 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE MY_INVENTORY ADD PARTITION PARTITIONS 2;
Query OK, 0 rows affected (7.60 sec)
Records: 0 Duplicates: 0 Warnings: 0

Splitting Partitions
mysql> select table_name,partition_name,table_rows
-> from information_schema.partitions
-> where table_schema = 'conf'
-> and table_name in ('MY_STORE','MY_INVENTORY','MY_EMPLOYEE')
-> and table_rows < 1;
+--------------+----------------+------------+
| table_name

| partition_name | table_rows |

+--------------+----------------+------------+
| my_employee

| p0

|

0 |

| my_employee

| p51

|

0 |

| my_inventory | p0

|

0 |

| my_inventory | p51

|

0 |

| my_store

| p0

|

0 |

| my_store

| p51

|

0 |

+--------------+----------------+------------+

Merging Partitions
 Closing All Stores in China (4 stores) :
mysql> ALTER TABLE MY_STORE COALESCE PARTITION 4;
Query OK, 0 rows affected (0.40 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE MY_EMPLOYEE COALESCE PARTITION 4;
Query OK, 0 rows affected (2.71 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> ALTER TABLE MY_INVENTORY COALESCE PARTITION 4;
Query OK, 0 rows affected (7.81 sec)
Records: 0 Duplicates: 0 Warnings: 0

Merging Partitions
 Closing All Stores in China (4 stores) :
mysql> select table_name,count(*)
-> from information_schema.partitions
-> where table_schema = 'conf'
-> and table_name in ('MY_STORE','MY_INVENTORY','MY_EMPLOYEE')
-> group by table_name;
+--------------+----------+
| table_name

| count(*) |

+--------------+----------+
| my_employee

|

48 |

| my_inventory |

48 |

| my_store

48 |

|

+--------------+----------+

A Few More Stats… (No Partitions)
mysql> explain partitions select my_store_no_part.city,my_employee_no_part.name,count(*) from
my_store_no_part, my_employee_no_part, my_inventory_no_part where my_store_no_part.id in (5,8,10,23,80)
and my_store_no_part.id = my_employee_no_part.store_id and my_store_no_part.id =
my_inventory_no_part.store_id and my_employee_no_part.id < 2000 and my_inventory_no_part.in_stock =
(ROUND(RAND(),0)) group by my_store_no_part.city,my_employee_no_part.name;
+---+-------------+----------------------+------------+--------+---------------+---------+---------+----------------------------------+--------+----------------------------------------------+
|id | select_type | table
| rows | Extra

| partitions | type

| possible_keys | key

| ref

|

+---+-------------+----------------------+------------+--------+---------------+---------+---------+----------------------------------+--------+----------------------------------------------+
| 1 | SIMPLE
| my_employee_no_part | NULL
| range | PRIMARY
3962 | Using where; Using temporary; Using filesort |

| PRIMARY | NULL

| 1 | SIMPLE
| my_store_no_part
| NULL
conf.my_employee_no_part.store_id |
1|

| eq_ref | PRIMARY
|

| PRIMARY |

| 1 | SIMPLE
| my_inventory_no_part | NULL
508243 | Using where; Using join buffer

| ALL
|

| NULL

| NULL

| NULL

|

|

+----+-------------+----------------------+------------+--------+---------------+---------+---------+----------------------------------+--------+----------------------------------------------+

mysql> select my_store_no_part.city,my_employee_no_part.name,count(*) from my_store_no_part,
my_employee_no_part, my_inventory_no_part where my_store_no_part.id in (5,8,10,23,80) and
my_store_no_part.id = my_employee_no_part.store_id and my_store_no_part.id =
my_inventory_no_part.store_id and my_employee_no_part.id < 2000 and my_inventory_no_part.in_stock =
(ROUND(RAND(),0)) group by my_store_no_part.city,my_employee_no_part.name;
+----------+-------------+----------+
| city

| name

| count(*) |

+----------+-------------+----------+
| Delhi

| Employee #0 |

60453 |

| Istanbul | Employee #0 |

79707 |

| Karachi

59872 |

| Employee #0 |

A Few More Stats… (Partitions)
mysql> explain partitions select my_store_lrg.city,my_employee_lrg.name,count(*) from my_store_lrg,
my_employee_lrg, my_inventory_lrg where my_store_lrg.id in (5,8,10,23,80) and my_store_lrg.id =
my_employee_lrg.store_id and my_store_lrg.id = my_inventory_lrg.store_id and my_employee_lrg.id < 2000
and my_inventory_lrg.in_stock = (ROUND(RAND(),0)) group by my_store_lrg.city,my_employee_lrg.name;
+---+-------------+------------------+-------------------+--------+---------------+---------+---------+-------------------------------+------+----------------------------------------------+
|id | select_type | table
rows | Extra |

| partitions

| type

| possible_keys | key

| ref

|

+---+-------------+------------------+-------------------+--------+---------------+---------+---------+-------------------------------+------+----------------------------------------------+
|1 | SIMPLE | my_employee_lrg | p5,p8,p10,p23,p80 | range | PRIMARY
94 | Using where; Using temporary; Using filesort |

| PRIMARY | NULL

|1 | SIMPLE | my_store_lrg
| p5,p8,p10,p23,p80 | eq_ref | PRIMARY
conf.my_employee_lrg.store_id |
1|
|

| PRIMARY |

|1 | SIMPLE | my_inventory_lrg | p5,p8,p10,p23,p80 | ALL
47938 | Using where; Using join buffer
|

| NULL

| NULL

| NULL

|

|

+----+-------------+------------------+-------------------+--------+---------------+---------+---------+------------------------------+-------+----------------------------------------------+

mysql> select my_store_lrg.city,my_employee_lrg.name,count(*) from my_store_lrg, my_employee_lrg,
my_inventory_lrg where my_store_lrg.id in (5,8,10,23,80) and my_store_lrg.id = my_employee_lrg.store_id
and my_store_lrg.id = my_inventory_lrg.store_id and my_employee_lrg.id < 2000 and
my_inventory_lrg.in_stock = (ROUND(RAND(),0)) group by my_store_lrg.city,my_employee_lrg.name;
+----------+-------------+----------+
| city

| name

| count(*) |

+----------+-------------+----------+
| Delhi

| Employee #0 |

60041 |

| Istanbul | Employee #0 |

77721 |

| Karachi

59786 |

| Employee #0 |

Summing it Up




Partitioning provides an easy way to scale within a database
Partitioning has a place in OLTP
Remember access methods and maintenance
Use Range/List for intelligent partitioning
Use Hash/Key for low maintenance, many partitions

Questions Anyone?

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.