Migrating from PostgreSQL to MySQL

at Cocolog

Naoto Yokoyama, NIFTY Corporation

Garth Webb, Six Apart
Lisa Phillips, Six Apart

Credits:
Kenji Hirohama, Sumisho Computer Systems Corp.
Agenda

 1. What is Cocolog
 2. History of Cocolog
 3. DBP: Database Partitioning
 4. Migration From PostgreSQL to MySQL
1. What is Cocolog
What is Cocolog
 NIFTY Corporation
 Established in 1986
 A Fujitsu Group Company
 NIFTY-Serve (licensed and interconnected with CompuServe)
 One of the largest ISPs in Japan
 Cocolog
 First blog community at a Japanese ISP
 Based on TypePad technology by SixApart
 Several hundred million PV/month
 History
 Dec/02/2003: Cocolog for ISP users launch
 Nov/24/2005: Cocolog Free for free launch
 April/05/2007: Cocolog for Mobile Phone launch
Cocolog (Screenshot of home page)

2008/04
700 Thousand Users
Cocolog (Screenshot of home page)

Cocolog TypePad
Cocolog template sets
Cocolog Growth (User) 　■ Cocolog 　■ Cocolog Free

phas phas phas phas

e1 e2 e3 e4
Cocolog Growth (Entry) 　■ Cocolog 　■ Cocolog Free

phas phas phas phas

e1 e2 e3 e4
Technology at Cocolog

 Core System
 Linux 2.4/2.6
 Apache 1.3/2.0/2.2 　＆ mod_perl
 Perl 5.8+CPAN
 PostgreSQL 8.1
 MySQL 5.0
 memcached/TheSchwartz/cfengine
 Eco System
 LAMP,LAPP,Ruby+ActiveRecord, Capistrano
 Etc...
Monitoring
 Management Tool
 Proprietary in-house development with PostgreSQL, PHP,
and Perl
 Monitoring points (order of priority)
 response time of each post
 number of spam comments/trackbacks
 number of comments/trackbacks
 source IP address of spam
 number of entries
Service  number of comments via mobile devices
 page views via mobile devices
 time of batch completion
APL  amount of API usage
 bandwidth usage
 DB
DB  Disk I/O
 Memory and CPU usage
 time of VACUUM analyze
Hard  APP
 number of active processes
 CPU usage
 Memory usage
Tips for migration

 Troubles with PostreSQL 7.4-8.1&Linux

2.4/2.6
 VACUUM
 Data size
 Character set
 Cleaning data
 Troubles with MySQL
 convert_tz function
 sort order
2. History of Cocolog
 Before DBP
Phase1 2003/12 ～ (Entry: 0.04Million)
10servers

Postgre
SQL
Register

TypePad
Static contents
Published

NAS

WEB
 Before DBP
Phase2 2004/12 ～ (Entry: 7Million)
50servers
Publish Book Rich template

Tel Operator Postgre

Support SQL
Register
2005/5 ～
Podcast
Portal
2004/12 ～TypePad
Profile
Etc.. Static contents
Published

NAS

WEB
Phase2 - Problems
 The system is tightly coupled.
 Database server is receiving from multiple
points.
 It is difficult to change the system design and
database schema.
 Before DBP
Phase3 2006/3 ～ (Entry: 12Million)
200servers
Publish Book Rich template
Web-API
memcached

Tel Operator Postgre

Support SQL
Register
TypePad
Podcast
Portal Static contents
Published
Profile
Etc..
NAS

WEB
 Before DBP
Phase4 2007/4 ～ (Entry: 16Million)
300servers
Publish Book Rich template
Web-API
Typepad memcached

Tel Operator Postgre

Support SQL
Register
Static contents Atom
Published

Mobile
WEB
N
A
S
W
EB
 After DBP
Now 2008/4 ～
150servers
Publish Book Rich template
Web-API
memcached

Tel Operator Multi

Support MySQL
Register
Typepad
Static contents Atom
Published

Mobile
WEB
N
A
S
W
EB
3. TypePad Database Partitioning
Steps for Transitioning
• Server Preparation
　　　　 Hardware and software setup
• Global Write
　　　　 Write user information to the global DB
• Global Read
　　　　 Read/write user information on the global DB
• Move Sequence
　　　　 Table sequences served by global DB
• User Data Move
　　　　 Move user data to user partitions
• New User Partition
　　　　 All new users saved directly to user partition 1
• New User Strategy
　　　　 Decide on a strategy for the new user partition
• Non User Data Move
　　　　 Move all non-user owned data
TypePad Overview (PreDBP)
Internet
Mobile Blog
Blog Owners Readers

https(443) smtp(25) / pop(110)

Blog Readers
http(80)

Web Application TypeCast Mail

Server Server Server Server

smtp(25) / pop(110)
memcached(11211) http(80) : atom api

nfs(2049) postgres(5432)

ATOM ADMIN(CRON)
Da tab ase MEMCACHED
Storage Server Server
(Post gres)

Data Caching servers to Dedicated Server for Cron Server for periodic
Static Content (HTML, TypeCast (via ATOM)
reduce DB load asynchronous tasks
Images, etc)
Why Partition?
TypePad TypePad
TypePad TypePad
TypePad TypePad
TypePad TypePad

User Rol e Global Non- User

(U ser 1) Rol e Rol e

User Rol e
User Rol e
(U ser 2)
(U ser 0)

Non- User Rol e

User Rol e (U ser 3)

Current setup After DBP

All inquires (access) go to one Inquiries (access) are divided among

DB(Postgres) several DB(MySQL)
Server Preparation
Information that does not
need to be partitioned

TypePad
(such as session
information)

Global Non-U ser Schw art z

Rol e Ro le DB

Maintains user mapping

and primary key generation Stores job
User Rol e details
(User 1)

User Rol e User information is

DB(PostgreSQL) (User 2) partitioned
Server for
executing Jobs
User Rol e
User Rol e
(U ser 0)
(User 3) Job Server
Non- + TypePad
User Role DB(MySQL) for partitioned data + Schwartz

Asynchronous Job Server

Current Setup New expanded setup

※Grey areas are not used in current

steps
 Global Write
Creating the user map

TypePad
①
Global Non-U ser Schw art z
Rol e Ro le DB

② Maintains user mapping

and primary key generation
User Rol e
(User 1)

User Rol e
DB(PostgreSQL) (User 2)

User Rol e
User Rol e
(U ser 0)
(User 3) Job Server
Non- + TypePad
User Role DB(MySQL) for partitioned data + Schwartz

Asynchronous Job Server

Explanation
　 ①： For new registrations only, uniquely identifying user data is written to the global DB
　 ②： This same data continues to be written to the existing DB

※Grey areas are not used in current steps

 Global Read
Use the user map to find the user partition

TypePad
②
Global Non-U ser Schw art z
③ Rol e Ro le DB
①
Maintains user mapping
Migrate existing and primary key generation
user data User Rol e
(User 1)

User Rol e
DB(PostgreSQL) (User 2)

User Rol e
User Rol e
(U ser 0)
(User 3) Job Server
Non- + TypePad
User Role DB(MySQL) for partitioned data + Schwartz

Asynchronous Job Server

Explanation
　 ①： Migrate existing user data to the global DB
　 ②： At start of the request, the application queries global DB for the location of user data
　 ③： The application then talks to this DB for all queries about this user. At this stage the global DB points
to the user0 partition in all cases. ※Grey areas are not used in current steps
 Move Sequence
Migrating primary key generation

TypePad
②
Global Non-U ser Schw art z
Rol e Ro le DB
①
Maintains user mapping
Migrate sequence and primary key generation
management User Rol e
(User 1)

User Rol e
DB(PostgreSQL) (User 2)

User Rol e
User Rol e
(U ser 0)
(User 3) Job Server
Non- + TypePad
User Role DB(MySQL) for partitioned data + Schwartz

Asynchronous Job Server

Explanation
　 ①： Postgres sequences (for generating unique primary keys) are migrated to tables on the global DB that
act as “pseudo-sequences”.
　 ② Application requests new primary keys from global DB rather than the user partition.
※Grey areas are not used in current steps
 User Data Move
Moving user data to the new user-role partitions

②
TypePad

Global Non-U ser Schw art z

Rol e Ro le DB

③ Maintains user mapping

and primary key generation Stores job
User Rol e details
(User 1)
④
User Rol e User information is
DB(PostgreSQL) (User 2) partitioned
Server for
executing Jobs
User Rol e
User Rol e
(U ser 0) Migrating each (User 3) Job Server
user data
Non- + TypePad
① + Schwartz
User Role DB(MySQL) for partitioned data

Explanation DB(MySQL) for partitioned data

　 ①： Existing users that should be migrated by Job Server are submitted as new Schwartz jobs. User data is
then migrated asynchronously
　 ②： If a comment arrives while the user is being migrated, it is saved in the Schwartz DB to be published later.
　 ③： After being migrated all user data will exist on the user-role DB partitions
　 ④： Once all user data is migrated, only non-user data is on Postgres ※Grey areas are not used in current steps
 New User Partition
New registrations are created on one user role partition

TypePad

① Global Non-U ser Schw art z

Rol e Ro le DB

Maintains user mapping

and primary key generation
User Rol e
(User 1)
②

User Rol e User information is

DB(PostgreSQL) (User 2) partitioned

User Rol e
User Rol e
(U ser 0)
(User 3) Job Server
Non- + TypePad
User Role DB(MySQL) for partitioned data + Schwartz

Asynchronous Job Server

Explanation
　 ①： When new users register, user data is written to a user role partition.
　 ②： Non-user data continues to be served off Postgres

※Grey areas are not used in current steps

 New User Strategy
Pick a scheme for distributing new users

TypePad

① Global Non-U ser Schw art z

Rol e Ro le DB

Maintains user mapping

and primary key generation
User Rol e
(User 1)
②

User Rol e User information is

DB(PostgreSQL) (User 2) partitioned

User Rol e
User Rol e
(U ser 0)
(User 3) Job Server
Non- + TypePad
User Role DB(MySQL) for partitioned data + Schwartz

Asynchronous Job Server

Explanation
　 ①： When new users register, user data is written to one of the user role partitions, depending on a set
distribution method (round robin, random, etc)
　 ②： Non-user data continues to be served off Postgres
※Grey areas are not used in current steps
 Non User Data Move
Migrate data that cannot be partitioned by user
Information that does not
need to be partitioned

TypePad
(such as session
information)

Global Non-U ser Schw art z

Rol e Ro le DB

Maintains user mapping

and primary key generation
User Rol e
(User 1)

User Rol e User information is

DB(PostgreSQL) (User 2) partitioned

User Rol e
User Rol e
(U ser 0) Migrate non-User
data (User 3) Job Server
Non- + TypePad
①
User Role DB(MySQL) for partitioned data + Schwartz

Asynchronous Job Server

Explanation
　 ①： Migrate non-user role data left on PostgreSQL to the MySQL side.

※Grey areas are not used in current steps

Data migration done
Information that does not
need to be partitioned
①
TypePad
(such as session
information)

① Global Non-U ser Schw art z

Rol e Ro le DB

Maintains user mapping

and primary key generation Stores job
User Rol e details
(User 1)

User Rol e User information is

DB(Postgres) (User 2) partitioned
Server for
executing Jobs
User Rol e
User Rol e
(U ser 0)
(User 3) Job Server
Non- + TypePad
User Role DB(MySQL) for partitioned data + Schwartz

② Asynchronous Job Server

Explanation
　 ①： All data access is now done through MySQL
　 ②： Continue to use The Schwartz for asynchronous jobs

※Grey areas are not used in current steps

The New TypePad configuration

Internet
Blog Readers Mobile Blog
Readers
Blog Owners
(management
interface) https(443) smtp(25) / pop(110)
http(80)

Web Application TypeCast Mail

Server Server Server Server

http(80) : atom api

smtp(25) / pop(110)
memcached(11211)

nfs(2049) MySQL(3306)

ATOM ADMIN(CRON) Job

Database MEMCACHED
St orage Server Server Server
(MyS QL )

Static Content Dedicated Server for Cron Server for periodic TheSchwartz server for
Data Caching servers to
(HTML, TypeCast (via ATOM) asynchronous tasks running ad-hoc jobs
reduce DB load
Images, etc) asynchronously
4. Migration from PostgreSQL to MySQL
History of scale up PostgreSQL server, Before DBP

 DB Node Spec History

Time OS(RedHat) CPU Xeon MEM DiskArray

2003/12 7.4(2.4.9) 1.8GHz/512k×1 1GB No

ES2.1(2.4.9) 3.2GHz/1M×2 4GB No

ES2.1(2.4.9) 3.2GHz/1M×2 4GB Yes

AS2.1(2.4.9) 3.2GHz/1M×4 12GB Yes

AS4 (2.6.9) 3.2GHz/1M×4 12GB Yes

2007/11
AS4 (2.6.9) MP3.3GHz/1M×4 16GB Yes
〔 2Core×4 〕
History of scale up PostgreSQL server, Before DBP

 DB DiskArray Spec
 [FUJITSU ETERNUS8000]
http://www.computers.us.fujitsu.com/www/prod
ucts_storage.shtml?products/storage/fujitsu/e
8000/e8000

 Best I/O transaction performance in the world

 146GB (15 krpm) * 32disk with RAID - 10
 MultiPath FibreChannel 4Gbps
 QuickOPC (One Point Copy)
 OPC copy functions let you create a duplicate copy
of any data from the original at any chosen time.
 Scale out MySQL servers, After DBP

 A role configuration
 Each role is configured as HA cluster
 HA Software: NEC ClusterPro
 Shared Storage
 Scale out MySQL servers, After DBP heart beat
FibreChannel SAN
DiskArray

Postgre MySQL MySQL MySQL

SQL Role1 Role2 Role3

TypePad
Application
Scale out MySQL servers, After DBP
 Backup
 Replication w/ Hot backup
 Scale out MySQL servers, After DBP heart beat
FibreChannel SAN
DiskArray

Postgre MySQL MySQL MySQL

SQL Role1 Role2 Role3

mysqld mysqld mysqld

…

rep rep rep

mysqld
mysqld
mysqld

TypePad opc

Application MySQL
BackupRole
 Troubles with PostreSQL 7.4 – 8.1
 Data size
 over 100 GB
 40% is index
 Severe Data Fragmentation
 VACUUM
 “VACUUM analyze” cause the performance problem
 Takes too long to VACUUM large amounts of data
 dump/restore is the only solution for de-fragmentation
 Auto VACUUM
 We don’t use Auto VACUUM since we are worried about
latent response time
 Troubles with PostgreSQL 7.4 – 8.1
 Character set
 PostgreSQL allow the out of boundary UTF-8
Japanese extended character sets and multi
bytes character sets which normally should
come back with an error - instead of
accepting them.
 “Cleaning” data
 Removing characters set that are out of the
boundries UTF-8 character sets.
 Steps
 PostgreSQL.dumpALL
 Split for Piconv
 UTF8 -> UCS2 -> UTF8 & Merge
 PostgreSQL.restore

dump Split restore

UTF8->UCS2->UTF8 Merge
Migration from PostgreSQL to MySQL using TypePad script

 Steps
 PostgreSQL -> PerlObject & tmp publish
-> MySQL -> PerlObject & last publish
 diff tmp ＆ last Object （ data check ）
 diff tmp ＆ last publish （ file check ）
data check
Object Object
TypePad TypePad

PostgreSQL
Document Document

File check
tmp last
 Troubles with MySQL

 convert_tz function
 doesn't support the input value outside the
scope of Unix Time
 sort order
 different sort order without “order by” clause
Cocolog Future Plans

 Dynamic
 Job queue
 Consulting by

 Sumisho Computer Systems Corp.

 System Integrator
 first and best partner of MySQL in Japan
since 2003
 provide MySQL consulting, support, training
service
 HA
 Maintenance
 online backup
 Japanese character support
Questions