MySQL HA Using different solutions

Robert Krzykawski DB Team Coordinator, bwi n games. Anders Karlsson Principal Sales Engineer, MySQL

Agenda
 Who are we?  HA Basics – Anders  How we did it; Success or failure – Robert  Summary  Questions?

Anders Karlsson
 Sales Engineer with Sun / MySQL for 5+ years  I have been in the RDBMS business for 20+ years  I have worked for many of the major vendors and with most of the vendor products  I’ve been in roles as
> > > > > Sales Engineer Consultant Porting engineer Support engineer Etc.

 Outside MySQL I build websites (www.papablues.com), develop Open Source software (MyQuery, ndbtop etc), am a keen photographer and drives sub-standard cars, among other things. Also: www.makezfsgpl.com ! Right now!

Robert Krzykawski
 DB Team Coordinator @ bwi n Games AB  Have been working with MySQL in every way from system admin, DBA, DBD and now taking a more system architectural role.  Been involved in building both small and big web based solutions since 1998 using MySQL.  My roles throughout my professional life have varied. System administrator, Technical Sales support, DBA, DBD, Programmer, Application architect and System architect.  Off work I am trying to automate things with scripts and programs to off load myself when “on work”.   I am also trying to find time to snowboard, play some paintball and a recently introduced hobby is our Maine Coon kittens. 

Why do you need HA
 Something can break. It usually will, eventually  You will need to maintain your database eventually, without shutting the whole system down  Adding HA to an existing running system is difficult, Much more so than to provide HA from the start  You want a good nights sleep! You want failover to be automatic!

HA Concepts
 Fault tolerant architectures
> These are hardware architectures with supporting software that prevents against even individual component failures

 Single Point of Failure (SPOF)
> In any fault tolerant setup, you want to avoid a SPOF, as a link is not better than it’s weakest link

 Fail over and Fail back
> Fail over is the process of switching from a failed component to another component, dormant or also active. Fail back is the process of failing back from the backup component to the original one.

Some HA Components
 Heartbeat
> Heartbeat is an HA component that checks that the services that are being failed over, are alive. Heartbeat can check individual servers, software services, networking etc.

 HA Monitor
> The HA Monitor has different names in different frameworks. This is the component that allows configuration of the services, ensures proper shutdown and startup and allows manual control

 Replication
> Replication is a common component that ensures that the data content of managed data rich components are in sync

What should I require?
 Don’t aim too high, aim for what is reasonable for your needs  Aim to ensure that no important data is lost
> What is “important data”? You decide! Different data means different “needs”!

 Aim to ensure that the solution can be automated. You will want this eventually anyway  Aim to ensure a solution that can easily be tested and administered  Aim to ensure that the solution is performant and scalable

HA with MySQL – In short
 MySQL Replication
> Easy to use and set up. Low performance impact > Asynchronous only. Failback can be difficult. Need additional components

 MySQL with DRBD / ZFS / AVS
> Easy to use. Low cost software only. Synchronous. Good HA software integration. > Certain performance impact. Limited data size and transaction rates.

HA with MySQL – In short
 MySQL with Shared storage
> Good performance. Eases hardware management. Good integration with HA software. > Costly. SAN itself is a SPOF.

 MySQL Cluster
> Very good performance. Self contained. Very short fail-over times. Software only solution. > Needs several physical servers. Not optimized for all MySQL applications.

bwin games ab

Our goal at bwin
 We were faced with a requirement; establish a highly available database platform.  We had some rules to follow from management.
> interruptions due to hardware failure should not require hands-on work. > Downtime should be minimized during interruptions. > Performance of DB platform should not decrease when operating as usual > Performance can decrease if a failure has occurred but should not deem the service unusable. > Implementation should be done by the operations department. Developers should not be involved.

What solutions did we consider?
 Master/Master  Linux HA  HP Service Guard  Sun Cluster  Combination of the above  MySQL Cluster  Will walk through all of the above

Master/Master
 Master/Master with two active nodes would give us a seamless switch if we have a good load balancer.
> Will give us the ability to do schema changes “on line” > Not only higher availability when both nodes are up, but better performance. > Can eliminate the use of production slaves. > One entry point for application when using “LB”

Linux HA/ServiceGuard/SunCluster
 Service IP switch will cause a glitch in service.  Since we are running 4.0 we can’t really do a master/master setup with service IP switching.  Slave integrity is important and we are running 4.0; One master data. Can’t switch to slave and hope that everything was replicated.  We are using SAN – Shared storage possible.  One instance, two machines – One active, one standby.  Innodb log size will be a problem.  Timeout during recovery can cause problems during switch.

MySQL Cluster
 High availability built in if implemented correct  Requires more hardware.  More complex solution  Requires application to support NDB  Not full feature set.

Obstacles
 We are using MySQL 4.0 in our biggest database  Master/Master scenario on 4.0 requires higher level of application awareness.  LinuxHA/ServiceGuard/Sun Cluster will cause small glitch when we move resources.  MySQL Cluster will require even more application changes in our case.

Our Choice
 LinuxHA because it is GPL/LGPL. Free and not owned by an organization.  Fastest way to implement, did not require any support from dev. Department.  All other ways required changes in application.

Layout
 Two versions

Master

Slave

HA Standby1

SAN01

SAN02

Master2

Master3

Master4

Master5

HA Standby2

SAN01

SAN02

We do..
 Use Linux HA 2.0. Needed for setup of “cluster”  Use SAN. Shared storage is easier and faster, but Expensive.
> DRBD can be used but saves the same data twice Also comes with a performance decrease.

 Heartbeat on two bonds. Primary database interconnect network, secondary on database service network  We have LUNs presented to multiple hosts  Services have rules to be run on specific hosts only.  We fence using RiLOE
> Have plans to fence on port level in FC switches.

What’s good and what’s bad..
 Easy and fast implementation  Our config does not increase/decrease performance.  Innodb log size causes long recovery times. Testing to decrease it has caused performance penalties.  Our solution is not fool proof because of long recovery times.  It causes interruption of service.  We can say it’s HA, but true HA solution would give us 100% uptime.  2nd Setup is complicated. We should aim for having simple setups. More common

What can we do better.
 Fine tune config for faster recovery/startup  Add better fencing  Monitor failover in case recovery takes long  Master/Master or Multi master.
> If application can reconnect or if we have a smart load balancer we have no outages. > Upgrades or schema changes can be made “online” > No separation between writes and reads. Less complicated for developers. One entry point.

Summary
 Concepts  Components  Requirements  Technologies  Your goal  Considerations  Obstacles  How we did it @ bwi n games AB  HA recommendations

Questions
The question is not, ‘What is the answer?’ The question is, ‘What is the question?’ Henri Poincaré

Thank you for your time!
 And thank you for listening so kindly.

 We can be found on:  Robert Krzykawski – http://krzykawski.com  Anders Karlsson – http://papablues.com http://karlssonondatabases.blogspot.com/