Gii thiu MySQL Replication

kipalog.com/posts/Gioi-thieu-MySQL-Replication

1. Gii thiu
Right tool for right job. Trc tin phi hiu l MySQL Replication khng phi l gii php gii quyt mi bi ton v qu ti h thng c s d liu. m rng mt h thng ta c hai
phng php m rng l scale up v scale out. Bt u vi 1 my ch th hai phng php trn c din gii nh sau:

Scale up c ngha l vi mt my ch ta lm cch no n c th phc v nhiu hn s lng kt ni, truy vn. Ngha l gi tr 1/(s kt ni phc v) cng nh th cng tt. t
c mc ch ny th c 2 phng php:

Tng phn cng ln cho my ch. Ngha l vi CPU l 4 core, RAM l 8 GB phc v c 500 truy vn th gi ta tng CPU ln 24 core, RAM tng ln 32GB -> my ch c th
phc v c s lng kt ni truy vn nhiu hn.
Optimize ng dng, cu truy vn. V d vi cu truy vn ly d liu tn 5s ly c d liu, sau mi tr li ti nguyn cho h thng phc v cc truy vn khc. My ch c
th ng thi phc v 500 truy vn dng nh vy th nu ta ti u truy vn ly d liu ch tn 1s => My ch c th phc v ng thi nhiu truy vn hn

Scale out l gii php tng s lng server v dng cc gii php load-balacer phn phi truy vn ra nhiu server. V d bn c 1 server c kh nng phc v 500 truy vn. Nu ta
dng thm 5 server na c cu hnh tng t, t thm mt LB pha trc phn phi th c kh nng h thng c th phc v c 5x500 truy vn ng thi.

MySQL Replication l mt gii php scale out (tng s lng instance MySQL) nhng khng phi bi ton no cng dng c. Cc bi ton m MySQL Replication s gii quyt tt:

Scale Read
Data Report
Real time backup

1.1 Scale Read

Scale Read thng gp cc ng dng m s truy vn c d liu nhiu hn ghi, t l read/write c th 80/20 hoc hn. Cc ng dng thng gp l bo, trang tin tc.

Vi scale read ta s ch c mt Master instance phc v cho vic c/ghi d liu. C th c mt hoc nhiu Slave instance ch phc v cho vic c d liu

Mt s ng dng write nhiu (thng mi in t) cng c s dng MySQL Replication scale out h thng

1.2 Data Report

Mt s h thng cho php mt s ngi (leader, manager, ngi lm report, thng k, data) truy cp vo d liu trn production phc v cho cng vic ca h. Vic chc thng vo data
production s rt nguy him v:

V tnh chnh sa lm sai lnh d liu (nu c quyn insert, update)

V tnh thc thi cc cu truy vn tn nhiu ti nguyn, thi gian truy vn di lm treo h thng

Vic setup mt my ch lm data report (application cng s khng kt ni ti server ny) lm gim thiu 2 ri ro trn

1.3 Real time backup

Vi c s d liu ln vic backup khng th thc hin thng xuyn c (hng gi, hng pht). Vi cc ng dng giao dch ti chnh, thanh ton, TMDT nu b mt d liu 1 gi, 1
ngy th thit hi s rt ln (my ch chnh t dng b hng). Real time backup l mt gii php b sung cho offline backup, chy ng thi c 2 phng php ny bo m s an ton
cho d liu.

2. Hot ng nh th no?

2.1 Mt s m hnh

Vi c hai m hnh ta lun ch c 1 Master database phc v cho Write d liu, c th c mt hoc nhiu Slave database. Ty tng m hnh ta c th cu hnh mi web node connect vo
mt Slave DB tng ng hoc c th dng mt LB t trc cm Slave LB t ng phn phi connection vo tng Slave DB theo thut ton ca LB

1/6
2.2 Cch hot ng
Trn Master:

Cc kt ni t web app ti Master DB s m mt Session_Thread khi c nhu cu ghi d liu. Session_Thread s ghi cc statement SQL vo mt file binlog (v d vi format
ca binlog l statement-based hoc mix). Binlog c lu tr trong data_dir (cu hnh my.cnf) v c th c cu hnh cc thng s nh kch thc ti a bao nhiu, lu li
trn server bao nhiu ngy.
Master DB s m mt Dump_Thread v gi binlog ti cho I/O_Thread mi khi I/O_Thread t Slave DB yu cu d liu

Trn Slave:

Trn mi Slave DB s m mt I/O_Thread kt ni ti Master DB thng qua network, giao thc TCP (vi MySQL 5.5 replication ch h tr Single_Thread nn mi Slave DB s
ch m duy nht mt kt ni ti Master DB, cc phin bn sau 5.6, 5.7 h tr m ng thi nhiu kt ni hn) yu cu binlog.
Sau khi Dump_Thread gi binlog ti I/O_Thead, I/O_Thread s c nhim v c binlog ny v ghi vo relaylog.
ng thi trn Slave s m mt SQL_Thread, SQL_Thread c nhim v c cc event t relaylog v apply cc event vo Slave => qu trnh replication hon thnh.

V logic mi Slave DB s ch nhn d liu t Master DB, mi hnh ng cp nht d liu BT BUC phi c thc hin trn Master. V nguyn tc nu ghi d liu trc tip ln Slave
DB => hng replication. Nhng thc cht ta hon ton c th ghi d liu trn Slave min sao khi Slave c binlog v apply khng ng g ti nhng trng d liu m ta mi ghi vo th
s khng b li (mc ny s ni thm cc phn sau)

Vi MySQL 5.5 th mi slave s ch c mt slave_thread connect ti Master, tuy nhin t phin bn 5.6 chng ta c th cu hnh nhiu slave_thread vic apply bin log ti cc
slave nhanh hn.

3. Hng dn ci t v cu hnh
M hnh:

Master DB: 172.17.0.1

Slave DB: 172.17.0.2

Trn Master DB

Cu hnh my.cnf

event-scheduler = on

2/6
bind-address = 172.17.0.1
server-id = 1

log-bin
binlog-format=row
binlog-do-db=dwh_prod
binlog-ignore-db=mysql
binlog-ignore-db=test

sync_binlog=0
expire_logs_days=2

To user replication

GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'172.16.0.2' IDENTIFIED BY 'p@ssword';

FLUSH PRIVILEGES;

To schema, d liu test

CREATE SCHEMA dwh_prod CHARACTER SET utf8 COLLATE utf8_general_ci;

CREATE TABLE tb1 (

id INT,
data VARCHAR(100)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE tb2 (

id INT,
data VARCHAR(100)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

SHOW TABLES;

Trn Slave DB

Cu hnh my.cnf

event_scheduler=off
bind-address = 172.17.0.2
server-id=2

log-bin
binlog-format=row
binlog-do-db=dwh_prod
binlog-ignore-db=mysql
binlog-ignore-db=test

transaction-isolation=read-committed
sync_binlog=0
expire_logs_days=2

To replication v kim tra

Nguyn tc khi to replication l phi LOCK tt c cc table trn Master DB, d liu khng thay i, sau xc nh binlog v position, 2 thng s dng cu hnh trn Slave xc
nh on d liu bt u ng b

Trn Master DB

FLUSH TABLES WITH READ LOCK;

SHOW MASTER STATUS;

+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+
| m01-bin.000001 | 827 | dwh_prod | mysql,test |
+

Gi tr cn quan tm l

m01-bin.000001
827

Sau ta s dump d liu t Master DB v y qua Slave DB (sau khi dump xong c th UNLOCK TABLES; Master DB c th hot ng li).

mysqldump -uroot -p dwh_prod > dwh_prod_03072015.sql

rsync -avz -P -e'ssh' dwh_prod_03072015.sql root@172.17.0.2:/root/

Trn Slave

mysql -uroot -p dwh_prod < /root/dwh_prod_03072015.sql

> CHANGE MASTER TO MASTER_HOST='172.17.0.1',MASTER_USER='slave_user', MASTER_PASSWORD='p@ssword', MASTER_LOG_FILE='m01-bin.000001',

MASTER_LOG_POS=827;
> START SLAVE

3/6
> SHOW SLAVE STATUS\G

Mt s thng tin c lc b cho d nhn

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event
Master_Host: 172.0.0.1
Master_User: slave_user
Master_Log_File: m01-bin.000001
Read_Master_Log_Pos: 827
Relay_Log_File: m02-relay-bin.000002
Relay_Log_Pos: 251
Relay_Master_Log_File: m01-bin.000001
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 827
Relay_Log_Space: 405
Seconds_Behind_Master: 0
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Master_Server_Id: 1

Cc thng s cn quan tm l

Last_Error: 0
Last_SQL_Error
Seconds_Behind_Master: 0

Hai thng s u tin l li khi Slave DB thc thi cc event c t relay log. Thng s Seconds_Behind_Master cho ta bit d liu ca Slave DB ang b tr (delay, lag) bao nhiu giy
so vi Master DB. Cc phn sau ta s ni k hn v replication lag ny.

4. Vn hnh h thng MySQL Replicatione

4.1 Test logic replication

trng thi bnh thng d liu trn Slave DB ng b vi Master DB. Kim tra

Trn Master

mysql> USE dwh_prod

mysql> SHOW TABLES;
+--------------------+
| Tables_in_dwh_prod |
+--------------------+
| tb1 |
| tb2 |
| tb3 |
| tb4 |
+--------------------+

Trn Slave

mysql> USE dwh_prod

mysql> SHOW TABLES;
+--------------------+
| Tables_in_dwh_prod |
+--------------------+
| tb1 |
| tb2 |
| tb3 |
| tb4 |
+--------------------+

mysql -e -p 'SHOW SLAVE STATUS\G' | grep -i 'error\|seconds'

Last_Error:
Seconds_Behind_Master: 0
Last_IO_Error:
Last_SQL_Error:

Mi th u n, khng li v khng c Lag.

Gi gi s ta s to mt table vi tn l tb00 trn Slave v kim tra xem c ng l khi ghi d liu ln Slave DB th replication c b hng hay khng.

mysql> CREATE TABLE tb00 (

id INT,
data VARCHAR(100)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

mysql> SHOW TABLES;

4/6
+
| Tables_in_dwh_prod |
+
| tb00 |
| tb1 |
| tb2 |
| tb3 |
| tb4 |
+
5 rows in set (0.00 sec)

Kim tra cc table trn Master DB

mysql> SHOW TABLES;

+--------------------+
| Tables_in_dwh_prod |
+--------------------+
| tb1 |
| tb2 |
| tb3 |
| tb4 |
+--------------------+

V kim tra li trng thi ca replication

mysql -e 'SHOW SLAVE STATUS\G' | grep -i 'error\|seconds'

Last_Error:
Seconds_Behind_Master: 0
Last_IO_Error:
Last_SQL_Error:

=> Nh ta thy r rng l d liu trn Slave v Master khc nhau (Slave c tb00 nhng Master th khng) nhng trng thi ca replication vn hon ton n.

Gi chng ta s th thm mt trng hp na l trn Master ta s to mt table tn l tb6 kim tra xem chuyn g s xy ra

mysql> CREATE TABLE tb6 (

id INT,
data VARCHAR(100)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

mysql> SHOW TABLES;

+
| Tables_in_dwh_prod |
+
| tb1 |
| tb2 |
| tb3 |
| tb4 |
| tb6 |
+

Kim tra cc table trn Slave DB

mysql> SHOW TABLES;

+--------------------+
| Tables_in_dwh_prod |
+--------------------+
| tb00 |
| tb1 |
| tb2 |
| tb3 |
| tb4 |
| tb6 |
+--------------------+

=> bng tb6 c ng b t Master qua, kim tra trng thi replication

mysql -e 'SHOW SLAVE STATUS\G' | grep -i 'error\|seconds'

Last_Error:
Seconds_Behind_Master: 0
Last_IO_Error:
Last_SQL_Error:

=> Mi th vn n, ngha l d ta c ghi d liu vo Slave, nhng nu Master thc thi cc cu truy vn khng ng g ti d liu c ghi mi Slave th trng thi ca replication vn
n.

Gi ta s thc hin thm mt th nghim na l trn Master ta to mt table tn l tb00, trng vi table to lc trc Slave pha trn v kim tra li trng thi ca replication

Kim tra trng thi replication trn Slave

mysql -e 'SHOW SLAVE STATUS\G' | grep -i 'error\|seconds'

Last_Error: Error 'Table 'tb00' already exists' on query. Default database: 'dwh_prod'. Query: 'CREATE TABLE tb00 (
Seconds_Behind_Master: NULL
Last_IO_Error:

5/6
 Last_SQL_Error: Error 'Table 'tb00' already exists' on query. Default database: 'dwh_prod'. Query: 'CREATE TABLE tb00 (

=> nh ta thy h thng bo li do trn Slave khng th thc thi hnh ng to table tb00 t Master y xung (do table ny tn ti trc )

Kt Lun: Vic ghi d liu vo Slave l c th thc hin c, nhng s gy ra ri ro hng replication mt lc no . Nht l cc cu truy vn dng SELECT ... UPDATE. Tt nht l
nn trnh ghi d liu vo Slave

4.2 Replication Lag

Replication Lag l tr d liu ca Slave so vi Master. Khi trin khai mt h thng MySQL Replication th Lag l vn chc chn gp phi. Ta ch c th gim thiu tr d liu
trong mc chp nhn c ch khng th khng c lag. L do l vic ng b d liu l Asynchronous, ngha l cc Slave server khng cn thng bo cho Master bit khi transaction
thc hin trn Slave thnh cng -> iu ny gip gi nguyn hiu sut (khc vi c ch ng b synchronous, mt transaction c gi l thnh cng khi n committed trn master
server v master server nhn c mt thng bo t slave server l transaction ny c write v committed. Qu trnh ny m bo tnh thng nht gia master v slave server
nhng ng thi n lm gim hiu sut i mt na do cc vn v network, bandwidth, location.)

Vn ca replication lag nh hng ti cc truy vn va write d liu xung l read d liu ln lin. V d

Mt trang thng mi in t c tnh nng add vo gi hng mt sn phm. Sau khi sn phm c add vo gi hng s tr s lng trong tn kho. 2 user thc hin mua sn phm
(sn phm c s lng tn kho l 1). C 2 u thy sn phm trn website hin th trng thi CN HNG. Khi mt user mua sn phm v thanh ton thnh cng. Do tr d
liu (v d 5s) nn d liu cha c cp nht tn kho xung Slave l sn phm ht hng. Khi user add gi hng v thanh ton th lc ny d liu mi cp nht v tr v m li l
thanh ton khng thnh cng do s lng tn kho khng p ng => nh hng ti tri nghim ca user trn h thng.

Thng vi nhng trng hp ny (truy vn write xong l read lin) th nn s dng cu hnh truy vn trn Master (y l l do Master c th va write va read ch khng nht thit l
ch c write)

4.3 Lb mysql v healthchk

Nh 1 trong 2 m hnh pha trn th vi m hnh th 2 ta c th dng haproxy lm lb cho cc MySQL Instance.

Vi m hnh 1 nhc im l nu MySQL instance b delay qu lu, server qu ti hoc ri ro nht l instace b down th ta khng c cch no check hoc remove instance ra
c.

Vi m hnh 2 nhc im l ta mt thm 1 layer (haproxy) na mi c th kt ni ti MySQL (tn thi gian, x l nhiu lp) nhng li im l c th cu hnh healthchk, hoc remove
instance theo mt s iu kin.

5. Troubleshoot

6. Mt s lu

6.1 Vn v server, phn cng

Cc vn v CPU, RAM, a cng (kch thc, loi a cng, SSD hay HDD, tc c ghi ca a cng)

Vi mt h thng DB cc thng s phn cng NN quan tm l

CPU: Cng nhiu core cng tt, tc cng nhanh cng tt

RAM: RAM cng nhiu cng tt

Vi a cng

Nn s dng RAID 5, 6, 10
Nn s dng SSD (Enterprise th cng tt) IOPS cng cao cng tt
a cng nn c dung lng t nht l x2 ln dung lng ca CSDL (s cn thit trong trng hp dump, backup d liu fix replication)

Khc vi cc ng dng khc nh web, static (thng CPU khng cn nhiu core, a cng khng cn nhiu v nhanh), my ch CSDL s cn nhiu cc thng s trn

Vi AWS khi chn Instance cng nn ch cc im trn

6.2 Cc vn v kch thc d liu

Vn kch thc d liu nh hng kh nhiu n vn hnh mt h thng MySQL Replication. D liu ln th qu trnh replication u tin hoc khi hng replication s rt lu => Slave
khng th s dng c trong thi gian replication, n khi Second_Behind_Master = 0 th mi c th s dng c.

Ngoi ra cc yu t v a cng (SSD, tc c ghi) cng nh hng nhiu n vic import hoc apply cc binlog t Master

Di y l mt m t thc t:

D liu th /var/lib/mysql c kch thc 80-100GB

D liu dump ra cha nn 18-30GB
D liu nn bng chun tgz ~ 2-3GB
My ch 24 core, 32GB RAM, SSD Plextor M6 PRO (4x256, RAID 10)
Thi gian dump d liu l 1h-1h30
Thi gian sync bn dump qua cc server (local, port 1Gb) -> khng nh
Thi gian import d liu -> khng nh
Thi gian Second_Behind_Master sau khi import xong -> khng nh

6/6