Professional Documents
Culture Documents
System Manager
MTV Networks Japan K.K.
Business MTV Networks Japan K.K. (MTVNJ) operates the mobile and PC
Description digital media services for all MTV Networks brands including MTV and
Nickelodeon in Japan. In addition to digital media services, MTVNJ
also operates a 24-hour Nickelodeon channel on CS/CATV, and the
consumer products business for Nickelodeon's character franchises in
Japan.
{database}
Project summary:
Merge all the database services for the MTV Flux and
MTV Mobile services on to a single database platform.
Database requirements:
• flexibility
• performance
• scalability
• redundancy
• support
• cost
MySQL MySQL
reqs/DB Oracle MS SQL
Community Enterprise
flexibility △/ × × ○ ○
performance ○ ○ ○ ○
scalability ○ /△ ○ ○ ○
redundancy ○ /△ ○ ○ ○
support ○ ○ × ○
cost × × ◎ △
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
DB Selection
MySQL Architectures:
• Single/multi Master
• Master/Slave
• Cluster
• DR:BD/Heartbeat HA Master
Single/multi Master
www www www • cheap(!)
• easy configuration
×
• load ↑ ⇒ performance ↓
• high risk (if master dies, no more DB)
Master
○ ○/△ × × ○ ◎
Master/Slave
www www www
• commonly used architecture
×
• some redundancy (read/writes separate)
○ △ ○ △ ○ ◎
Cluster
www www www • redundant!
• can scale
△
• complicated configuration
• limited support in Japan (2007)
Cluster
○ ○ ◎ ◎ ○/△ △
DR:BD/Heartbeat HA Master
www www www • redundant (master fully redundant)
• can scale (add salves)
• reduced outage risk ◎
• can use failover for maintenance tasks
Primary Secondary
• complicated configuration
◎ ○/△ ○/△ ◎ ○ ○
Application Application
VIP
heartbeat
/etc/my.cnf /etc/my.cnf
local local
×
1 – Application looses
mysqld start VIP connection to mysqld
process
network or mysqld
failure
heartbeat
2 – heartbea
detects the prob
/etc/my.cnf /etc/my.cnf
local local
/var/lib/mysql drbd
drbd /var/lib/mysql
DRBD DRBD
Primary Secondary
©2008 MTV Networks Japan K.K.
heartbeat does: MTV Flux/MTV Mobile DB project
B mysql dir mount
B mysqld start HA Master 4 – Failover complete!
Application Application
2 – drbd
fails over
Failover 1 – primary server fails!
VIP
process
server outage 3 – heartb
detects out
heartbeat
/etc/my.cnf /etc/my.cnf
local local
Old DB
architecture
New DB
architecture
VIP
heartbeat
/etc/my.cnf /etc/my.cnf
local local
3 – install
heartbeat
Migration 5 – test application
1 – remove
VIP old
process database components
connectivity
heartbeat
×
/etc/my.cnf
local disk
local ×
local disk
DRBD DRBD
Primary Secondary
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
HA Master Install
Heartbeat:
• no room for configuration mistakes
• pay special attention to peer connectivity
• run lots of tests before going to production!!!
mysqld:
• can take advantage of redundancy to swap out nodes
• never try to run both primary and secondary at once!!
• leave all the redundancy to heartbeat
DR:BD:
• requires separate partition (can be LVM)
• like heartbeat – test your configurations!!!
• always check your DRBD status after a failover
• peers sometime require a manual re-sync
MTVNJ links
http://www.mtvn.jp/en/index.html
http://www.mtvn.jp/en/service.html
Patrick Bolduan
MTV Networks Japan KK
2-8-2 Jingu-mae, Shibuya-ku, Tokyo 150-0001
patrick.bolduan@mtvn.jp
http://www.mtvjapan.com