May 20, 2008

Sheraton Gateway Suite O'Hare

May 20, 2008

Sheraton Gateway Suite O'Hare

Designing for Higher Availability

May 20, 2008

Sheraton Gateway Suite O'Hare

Designing for Higher Availability

Morgan Tocker
MySQL Canada Inc. http://tocker.id.au/ morgan@mysql.com May 20, 2008 Sheraton Gateway Suite O'Hare

Audience Poll! Who needs 100% availability?

May 20, 2008

Sheraton Gateway Suite O'Hare

2

Sorry, It doesn’t exist.

Sheraton Gateway Suite O'Hare

3

Let’s just say in a magical world, it did exist.....

Sheraton Gateway Suite O'Hare

4

You can’t afford it.

Sheraton Gateway Suite O'Hare

5

Next question. What is Availability?

Sheraton Gateway Suite O'Hare

6

I’m going to argue this one until I die.

Sheraton Gateway Suite O'Hare

7

Is it the time your application is running minus unpredicted outages?

Sheraton Gateway Suite O'Hare

8

Maybe. But, does scheduled maintenance count?

Sheraton Gateway Suite O'Hare

9

How about network downtime?

Sheraton Gateway Suite O'Hare

10

The Marketing Guys have made it easier....

• Availability is measured in “nines”;
–Two, three, four, five and six nines.

• But what downtime that includes is not consistently applied.

Sheraton Gateway Suite O'Hare

11

Two Nines

• 99% Availability • 87 hours, 39 minutes, 29 seconds downtime.

http://www.flickr.com/photos/maebmij/122564955/ Sheraton Gateway Suite O'Hare 12

Three Nines

• 99.9% Availability • 8 hours, 45 minutes, 56 seconds, downtime.

http://www.flickr.com/photos/mshades/276083406/

Sheraton Gateway Suite O'Hare

13

Four Nines

• 99.99% Availability • 52 minutes, 35 seconds, downtime.

http://www.flickr.com/photos/osakajon/315338262/

Sheraton Gateway Suite O'Hare

14

Five Nines

• 99.999% Availability • 5 minutes, 15 seconds, downtime.
http://www.flickr.com/photos/alexandrechang/1201385569/

Sheraton Gateway Suite O'Hare

15

Six Nines

• Still off Limits for us mere mortals. • For Government, banks and royalty.

<< insert picture of a roman eating grapes >>

Sheraton Gateway Suite O'Hare

16

Let me rephrase the question this time....

Sheraton Gateway Suite O'Hare

17

Availability is whatever your customer thinks it is.

Sheraton Gateway Suite O'Hare

18

Applying some context...

<?php /* Change from 60 seconds (way too long) to 2 seconds If a connection fails, reduce functionality slightly. */ ini_set('mysql.connect_timeout', 2); mysql_connect($hostname, $username, $password)   or handle_mysql_failure(); ..

Sheraton Gateway Suite O'Hare

19

Context (cont.) .. /*  Check if we have a database link  before rendering optional functionality. */ if (!degrade_gracefully_mode()) {     show_add_comment_box(); } else {     print "Comments is under maintenance, check back  soon!"; } ?>
Sheraton Gateway Suite O'Hare 20

Tools of the Trade

+Replication

Heartbeat

Sheraton Gateway Suite O'Hare

21

MySQL Replication

• Asynchronous
–I’ll explain that in a second.

• Statement-Based (until 5.1 at least) • Spreads read load. • Does not spread writes.

Sheraton Gateway Suite O'Hare

22

MySQL Replication (cont.)

Sheraton Gateway Suite O'Hare

23

MySQL Replication (cont.)

Sheraton Gateway Suite O'Hare

23

MySQL Replication (cont.)

Sheraton Gateway Suite O'Hare

23

MySQL Replication (cont.)

Sheraton Gateway Suite O'Hare

23

MySQL Replication (cont.)

Sheraton Gateway Suite O'Hare

23

DRBD

• A Linux Kernel Module
–Think of it as “Network Level RAID1”.

• Not just for MySQL!
–Can be used to mirror other files.

• Doesn’t spread reads. • Doesn’t spread writes. • Can be Synchronous.

Sheraton Gateway Suite O'Hare

24

How DRBD works

Sheraton Gateway Suite O'Hare

25

How DRBD works

Sheraton Gateway Suite O'Hare

25

How DRBD works

Sheraton Gateway Suite O'Hare

25

How DRBD works

Sheraton Gateway Suite O'Hare

25

How DRBD works

Sheraton Gateway Suite O'Hare

25

How DRBD works

Sheraton Gateway Suite O'Hare

25

How DRBD works

Sheraton Gateway Suite O'Hare

25

How DRBD works

Sheraton Gateway Suite O'Hare

25

How DRBD works

Sheraton Gateway Suite O'Hare

25

Comparing DRBD to MySQL Slaves

Feature
Fast Resync on inconsistency Online Resync Syncronous Network Traffic Corruption on primary spreads Secondary can be time delayed from primary Secondary can be utilized OS Support

Replication
No Yes if InnoDB No Low to Medium Unlikely Yes Yes (reads) All

DRBD
Yes Yes Yes High Certain No Yes (backups) Linux

Sheraton Gateway Suite O'Hare

26

Comparing DRBD to MySQL Slaves (cont.)

Feature
Consistency Check Failover Capability Multiple Spares Remote Spares myslqdump backup from spare Secondary hardware requirements Allows replicating partial dataset Replication Event Checksums

Replication
Not Built in Fairly Easy Easy Yes Yes Either faster/slower than master Yes No

DRBD
Yes Easy No Unsuitable No* At least as fast as master No Yes

Sheraton Gateway Suite O'Hare

27

Heartbeat

• A Linux cluster manager. • Can monitor just about anything
–Including DRBD and MySQL Replication.

• Common setup is to use shared IP address between two nodes.
–Can script whatever you like happens on node failure.

Sheraton Gateway Suite O'Hare

28

Heartbeat Monitoring

192.168.1.1

192.168.1.2

192.168.1.3

#2

#1

Webservers

Database Servers

Sheraton Gateway Suite O'Hare

29

Heartbeat Monitoring

192.168.1.1

192.168.1.2

192.168.1.3

#2

#1

Webservers

Database Servers

Sheraton Gateway Suite O'Hare

29

Heartbeat Monitoring

192.168.1.1

192.168.1.2

192.168.1.3

#2

Webservers

Database Servers

Sheraton Gateway Suite O'Hare

29

Heartbeat Monitoring

192.168.1.1

192.168.1.2

192.168.1.3

Webservers

Database Servers

Sheraton Gateway Suite O'Hare

29

Heartbeat Monitoring

192.168.1.1

192.168.1.2

192.168.1.3

#1

Webservers

Database Servers

Sheraton Gateway Suite O'Hare

29

Memcached

• Dead simple front-end caching. • Access data from cache, alleviate database load server.

Sheraton Gateway Suite O'Hare

30

When we combine our powers together, we have topologies.....

Sheraton Gateway Suite O'Hare

31

Start Small

Web Servers

Database Server

Database Slave

Reports + Backup

Sheraton Gateway Suite O'Hare

32

A little larger

Web Servers

Database Server

Database Slave

Reports + Backup

Sheraton Gateway Suite O'Hare

33

“May work”

Web Servers

• Master-Master replication
–Strong warnings not to send writes to either node. You may be asking for trouble.

Database Servers in Master-Master.

Sheraton Gateway Suite O'Hare

34

Larger Again

Web Servers

Front End Caching (Memcached) Database Slave

Reports + Backup Database Server

Sheraton Gateway Suite O'Hare

35

Think HA is expensive?

• You’d be right. • HA is like the Noah’s Ark of computing.
–You need at least two of everything.

• Anything less less than two and you have a single point of failure (“SPOF”).

Sheraton Gateway Suite O'Hare

36

Clusters can’t be all fun and games!

Sheraton Gateway Suite O'Hare

37

Cluster and HA caveats and Terminology

• Split Brains
–A situation where both nodes think that they are the primary.
I’m Primary I’m Primary

Database Servers

Sheraton Gateway Suite O'Hare

38

• Fencing and STONITHs
–Force a take down of a failed node.
I kill you!

I won’t die.

Database Servers

Sheraton Gateway Suite O'Hare

39

No HA talk would be complete without plugging the importance of backups!

Sheraton Gateway Suite O'Hare

40

What makes a good backup?

• What are you insulating against?
–Require protection from failures as well as accidental/Malicious deletes.

• Quick to Backup • Quick to Recovery • Flexible recovery granularity
–Can recover just one row, or an entire table.

• Consistent

Sheraton Gateway Suite O'Hare

41

An “Inconsistent” Backup

Users id 1 2 name Fred Mary email emo_kid5000@hotmail.com daddys_princess@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 user_2 1 2

Sheraton Gateway Suite O'Hare

42

An “Inconsistent” Backup

Users id 1 2 name Fred Mary email emo_kid5000@hotmail.com daddys_princess@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 user_2 1 2

Sheraton Gateway Suite O'Hare

42

An “Inconsistent” Backup

Users id 1 2 name Fred Mary email emo_kid5000@hotmail.com daddys_princess@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 user_2 1 2

Sheraton Gateway Suite O'Hare

42

An “Inconsistent” Backup

Users id 1 2 3 name Fred Mary Barry email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 user_2 1 2

Sheraton Gateway Suite O'Hare

42

An “Inconsistent” Backup

Users id 1 2 3 name Fred Mary Barry email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 user_2 1 2

Sheraton Gateway Suite O'Hare

42

An “Inconsistent” Backup

Users id 1 2 3 name Fred Mary Barry email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 3 2 user_2 1 2 2 3

Sheraton Gateway Suite O'Hare

42

An “Inconsistent” Backup

Users id 1 2 3 name Fred Mary Barry email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 3 2 user_2 1 2 2 3

Sheraton Gateway Suite O'Hare

42

An “Inconsistent” Backup

Users id 1 2 3 name Fred Mary Barry email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 3 2 user_2 1 2 2 3

Sheraton Gateway Suite O'Hare

42

An “Inconsistent” Backup

Users id 1 2 3 name Fred Mary Barry email

emo_kid5000@hotmail.com

daddys_princess@hotmail.com bazza@somewhere.com

user_blog_posts id 1 2 user_id 1 2

post_contents

I hate my parents. Ponnies! lol. :D

What’s wrong here?
Sheraton Gateway Suite O'Hare

user_friendship user_1 1 2 3 2 user_2 1 2 2 3

42

An “Inconsistent” Backup

Users id 1 2 name Fred Mary email emo_kid5000@hotmail.com daddys_princess@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 3 2 user_2 1 2 2 3

Sheraton Gateway Suite O'Hare

43

An “Inconsistent” Backup

Users id 1 2 3 name Fred Mary Jim email emo_kid5000@hotmail.com daddys_princess@hotmail.com i_hate_mary@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 3 2 user_2 1 2 2 3

Sheraton Gateway Suite O'Hare

43

An “Inconsistent” Backup

Users id 1 2 3 name Fred Mary Jim email emo_kid5000@hotmail.com daddys_princess@hotmail.com i_hate_mary@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 3 2 user_2 1 2 2 3

Sheraton Gateway Suite O'Hare

43

A “Consistent” Backup

Users id 1 2 name Fred Mary email emo_kid5000@hotmail.com daddys_princess@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 user_2 1 2

Sheraton Gateway Suite O'Hare

44

A “Consistent” Backup

Users id 1 2 name Fred Mary email emo_kid5000@hotmail.com daddys_princess@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 user_2 1 2

Sheraton Gateway Suite O'Hare

44

A “Consistent” Backup

Users id 1 2 name Fred Mary email emo_kid5000@hotmail.com daddys_princess@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 user_2 1 2

Sheraton Gateway Suite O'Hare

44

A “Consistent” Backup

Users id 1 2 name Fred Mary email emo_kid5000@hotmail.com daddys_princess@hotmail.com user_friendship user_1 1 user_blog_posts id 1 2 user_id 1 2 post_contents I hate my parents. Ponnies! lol. :D 2 user_2 1 2

Sheraton Gateway Suite O'Hare

44

InnoDB Backup with Multiversional powers!

Users id 1 2 3 4 name Fred Mary Barry Jim email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com i_hate_mary@hotmail.com

Sheraton Gateway Suite O'Hare

45

InnoDB Backup with Multiversional powers!

Users id 1 2 3 4 name Fred Mary Barry Jim email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com i_hate_mary@hotmail.com

Sheraton Gateway Suite O'Hare

45

InnoDB Backup with Multiversional powers!

Users id 1 2 3 4 name Fred Mary Barry Jim email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com i_hate_mary@hotmail.com

Sheraton Gateway Suite O'Hare

45

InnoDB Backup with Multiversional powers!

Users id 1 2 3 4 name Fred Mary Barry Jim email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com i_hate_mary@hotmail.com

Sheraton Gateway Suite O'Hare

45

InnoDB Backup with Multiversional powers!

Users id 1 2 3 4 name Fred Mary Barry Jim email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com i_hate_mary@hotmail.com 3 Barry bazza@hotmail.com

Sheraton Gateway Suite O'Hare

45

InnoDB Backup with Multiversional powers!

Users id 1 2 3 4 name Fred Mary Barry Jim email emo_kid5000@hotmail.com daddys_princess@hotmail.com bazza@somewhere.com i_hate_mary@hotmail.com 3 Barry bazza@hotmail.com

Sheraton Gateway Suite O'Hare

45

Your backup Options

• mysqldump
–Text based backups –Easier to recover individual rows

• Raw Backup
–Copy the data files –Filesystem snapshot

• InnoDB Hot Backup (commercial) • Backup from a Slave

Sheraton Gateway Suite O'Hare

46

Backup Options

mysqldump No (if all InnoDB) Medium

Raw backups

Snapshots

InnoDB Hot Backup* No (if all InnoDB) Fast

Blocking

Yes

Partially

Backup Speed Recovery Speed Recovery Granularity

Fast

Fast

Slow

Fast

Medium

Fast

Best

Depends

Not Good

Good

* Commercial Tool
Sheraton Gateway Suite O'Hare 47

Point in Time Recovery

• Really slick with Statement basedreplication, just enable binary logging. • An example:
mysqldump --flush-logs --all-databases --master-data=2 > mybackup.sql

Recovery:
mysql < mybackup.sql mysqlbinlog mylog.000005 | mysql
Sheraton Gateway Suite O'Hare 48

The End.

Do you have any questions? Lemmeno.

Sheraton Gateway Suite O'Hare

49

Sign up to vote on this title
UsefulNot useful