High-Availability MySQL with DR:BD and Heartbeat: MTV Japan mobile services

©2008 MTV Networks Japan K.K.

Patrick Bolduan
patrick.bolduan@mtvn.jp

System Manager MTV Networks Japan K.K.

©2008 MTV Networks Japan K.K.

Let’s get started

©2008 MTV Networks Japan K.K.

About MTVNJ
Company Profile
Company Name President Address Business Description MTV Networks Japan K.K. Peter Robert Bullard 2-8-2 Jingumae, Shibuya-ku, Tokyo 150-0001 TEL: 03-6434-3111 FAX: 03-6434-3150 MTV Networks Japan K.K. (MTVNJ) operates the mobile and PC digital media services for all MTV Networks brands including MTV and Nickelodeon in Japan. In addition to digital media services, MTVNJ also operates a 24-hour Nickelodeon channel on CS/CATV, and the consumer products business for Nickelodeon's character franchises in Japan.

©2008 MTV Networks Japan K.K.

MTVNJ Mobile Brands

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project intro

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project DB Selection

{database}

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project DB Selection

Project summary:
Merge all the database services for the MTV Flux and MTV Mobile services on to a single database platform.

Database requirements:
• flexibility • performance • scalability • redundancy • support • cost
©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project DB Selection reqs/DB
flexibility performance scalability redundancy support cost Oracle △/ × ○ ○ /△ ○ /△ ○ × MS SQL × ○ ○ ○ ○ × MySQL Community ○ ○ ○ ○ MySQL Enterprise ○ ○ ○ ○ ○

× ◎

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project DB Selection
We chose MySQL Enterprise

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project MySQL Architectures

MySQL Architectures:
• Single/multi Master • Master/Slave • Cluster • DR:BD/Heartbeat HA Master

What we looked for:
• flexibility • performance • scalability • redundancy • support • cost
©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project MySQL Architectures

Single/multi Master
www www www

• cheap(!) • easy configuration • load ↑ ⇒ performance ↓ • high risk (if master dies, no more DB)

×

Master

flexibility performance scalability redundancy support cost


©2008 MTV Networks Japan K.K.

○/△

×

×

MTV Flux/MTV Mobile DB project MySQL Architectures

Master/Slave
www www www

• some redundancy (read/writes separate) • commonly used architecture • still some risk (operator needed for outage recovery) • manual failover for outages

×

Master

Slave

flexibility performance scalability redundancy support cost


©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project MySQL Architectures

Cluster
www www www

• redundant! • can scale • complicated configuration • limited support in Japan (2007)

Cluster

flexibility performance scalability redundancy support cost


©2008 MTV Networks Japan K.K.

○/△

MTV Flux/MTV Mobile DB project MySQL Architectures

DR:BD/Heartbeat HA Master
www www www

Primary

Secondary

• redundant (master fully redundant) • can scale (add salves) • reduced outage risk • can use failover for maintenance tasks • complicated configuration


flexibility performance scalability redundancy support cost


©2008 MTV Networks Japan K.K.

○/△

○/△

MTV Flux/MTV Mobile DB project MySQL HA Master

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA Master
Application Application

VIP

heartbeat

/etc/my.cnf local /var/lib/mysql DRBD
Primary
©2008 MTV Networks Japan K.K.

/etc/my.cnf local drbd /var/lib/mysql DRBD
Secondary

MTV Flux/MTV Mobile DB project HA Master
Application Web site or whatever Manages/monitors DB process and mounts: • provides application connection point (Virtual IP)
• monitors network status • watches the mysqld process • manages the /var/lib/mysql mount point

VIP heartbeat

mysqld (the database)

DRBD
©2008 MTV Networks Japan K.K.

Synchronous device layer data replication

MTV Flux/MTV Mobile DB project HA Master
Application

3 – heartbeat does: Application

Failover process
network or mysqld failure

×
/var/lib/mysql DRBD
Primary

VIP

• 4 – Failoverdir unmount pDB mysql complete! • DRBD2switch 1 – Application looses – heartbeat • sDB mysqlthe mount connection to mysqld detects dir problem • sDB mysqld start

heartbeat

/etc/my.cnf local drbd drbd /var/lib/mysql DRBD
Secondary

/etc/my.cnf local

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA Master
Application Application

Failover process
server outage

VIP

3 – heartbeat does: 4 – Failover complete! • sDB 32 – drbd mount mysql dir – heartbeat 1 – primary server fails! • sDBdetects outage mysqld start fails over

heartbeat

/etc/my.cnf local /var/lib/mysql DRBD
Primary
©2008 MTV Networks Japan K.K.

/etc/my.cnf local drbd /var/lib/mysql DRBD
Secondary

MTV Flux/MTV Mobile DB project HA MySQL install

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA Master Install

Old DB architecture

A Master

B Master

Master

local disk

local disk

local disk

Dual Master
©2008 MTV Networks Japan K.K.

Single Master

MTV Flux/MTV Mobile DB project HA Master Install

New DB architecture
VIP

heartbeat

/etc/my.cnf local /var/lib/mysql DRBD
Primary
©2008 MTV Networks Japan K.K.

/etc/my.cnf local drbd /var/lib/mysql DRBD
Secondary

MTV Flux/MTV Mobile DB project HA Master Install

Database migration work plan:
• stop production services • perform full data backup (mysqldump) • Replace MySQL architecture • restore data from backup • test database failover • test application • re-open production services

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA Master Install

Migration process

1 – remove old VIP database components

2 – install/configure 5 – test application 3 – install 4 – configure VIP drbd and mysql connectivity heartbeat

heartbeat

/etc/my.cnf local disk local

×

A Master

local disk /var/lib/mysql DRBD
Primary

×

B Master

Master

/etc/my.cnf local disk local drbd /var/lib/mysql DRBD
Secondary

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA Master Install

Actual migration work times:
•production services stop – 00:10 • full data backup (mysqldump) – 00:05 • MySQL architecture replacement – 01:30 • data restore – 00:10 • database failover test – 00:15 • application test – 03:00 • production services fully restored!

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project HA MySQL lessons learned

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project Lessons learned

Impact on our application:
• essentially the same as a single master • can use same HA for: • single master - master/slave - multi-master • failover fast enough that DB outages don’t affect application

Heartbeat:
• no room for configuration mistakes • pay special attention to peer connectivity • run lots of tests before going to production!!!

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project Lessons learned

mysqld:
• can take advantage of redundancy to swap out nodes • never try to run both primary and secondary at once!! • leave all the redundancy to heartbeat

DR:BD:
• requires separate partition (can be LVM) • like heartbeat – test your configurations!!! • always check your DRBD status after a failover • peers sometime require a manual re-sync

©2008 MTV Networks Japan K.K.

MTV Flux/MTV Mobile DB project Lessons learned

Overall – good points:
• we were able to deploy HA in a short maintenance window • commercial support available (MySQL – LINBIT) • MySQL consultative services were a big help • also usable with MySQL Community • can be part of other architectures • multi-master - master/slave

Overall – weak points:
• lots to learn, lots to test (heartbeat and DRBD) • configuration and failover is complex • mysqld restart requires some extra steps • all admins/vendors need to understand failover basics
©2008 MTV Networks Japan K.K.

HA MySQL in other MTVNJ environments

©2008 MTV Networks Japan K.K.

Questions?

DRBD related links http://www.drbd.org/users-guide/s-heartbeat-r1.html http://www.drbd.org/users-guide/s-heartbeat-crm.html MTVNJ links http://www.mtvn.jp/en/index.html http://www.mtvn.jp/en/service.html Patrick Bolduan MTV Networks Japan KK 2-8-2 Jingu-mae, Shibuya-ku, Tokyo 150-0001 patrick.bolduan@mtvn.jp http://www.mtvjapan.com

©2008 MTV Networks Japan K.K.