You are on page 1of 32




Building Elastix-2.4 High Availability Clusters with
DRBD and Heartbeat (using a single NIC)

This information has been modified and updated by Nick Ross.


Please refer to the original document found at:

KWWSZZZHODVWL[EUDVLOFRPEUGRZQORDGV(ODVWL[BB+$B&OXVWHUSGI

Changes made to this document will be explained at the very end


in Appendix A.

Document Last updated October 15th, 2015.


Credits
A great deal of credit goes out to Daniel Guevara and Amjad Jabali,
who authored previous versions of this document. Daniel Guevara's

document is linked above, but it appears Amjad Jabali's is offline.

While I have added a great deal to this document, and made many
changes, a great deal of work was done by these other authors, to
the point where this document would not exist without them.

Thanks for the great work guys.












1







INDEX

Operational Overview……………………………………………………
 What Is DRBD…………………………………………………….
 :KDW+HDUWEeat does……………………………………………
(TXLSPHQW2YHUYLHZ……………………………………………………….4
'5%',QVWDOODQG&RQILJXUDWLRQ………………………………………….

+HDUWEHDW&RQILJXUDWLRQ……………………………………………….…
Credits…………………………………………………………………..
5HIHUHQFHV……………………………………………………………..
$SSHQGL[ $ FKDQJHV PDGH ««««««««««««««
$SSHQGL[ % )ODVK RS SDQHO IL[ «««««««««««««
$SSHQGL[ & 8SGDWH (ODVWL[ Z '5%' «««««««
$SSHQGL[ ' 5HVL]H '5%' SDUWLWLRQ «««««««
$SSHQGL[ ( '5%' Z  1,&V «««««««
$SSHQGL[ ) 7)73%227 RQ '5%' ««««««
$SSHQGL[ * 7KUHH 1RGHV «««««««
$SSHQGL[ + ,3 6RXUFLQJ «««««««
$SSHQGL[, ,36RXUFLQJ3W «««««««
$SSHQGL[- ,36HFIRU'5%' «««««««

2
Operational Overview

What is DRBD?

'5%'Š UHIHUV WR EORFN GHYLFHV GHVLJQHG DV D EXLOGLQJ EORFN WR IRUP KLJK DYDLODELOLW\ +$
FOXVWHUV 7KLV LV GRQH E\ PLUURULQJ D ZKROH EORFN GHYLFH YLD DQ DVVLJQHG QHWZRUN '5%'
FDQ EH XQGHUVWRRG DV QHWZRUN EDVHG UDLG

,Q WKH LOOXVWUDWLRQ DERYH WKH WZR RUDQJH ER[HV UHSUHVHQW WZR VHUYHUV WKDW IRUP DQ +$
FOXVWHU 7KH ER[HV FRQWDLQ WKH XVXDO FRPSRQHQWV RI D /LQX[ΠNHUQHO ILOH V\VWHP EXIIHU
FDFKH GLVN VFKHGXOHU GLVN GULYHUV 7&3,3 VWDFN DQG QHWZRUN LQWHUIDFH FDUG 1,& GULYHU
7KH EODFN DUURZV LOOXVWUDWH WKH IORZ RI GDWD EHWZHHQ WKHVH FRPSRQHQWV

7KH RUDQJH DUURZV VKRZ WKH IORZ RI GDWD DV '5%' PLUURUV WKH GDWD RI D KLJKO\ DYDLODEOH
VHUYLFH IURP WKH DFWLYH QRGH RI WKH +$ FOXVWHU WR WKH VWDQGE\ QRGH RI WKH +$ FOXVWHU

,Q RXU LPSOHPHQWDWLRQ ZH ZLOO EH FUHDWLQJ D '5%' V\QFKURQL]HG SDUWLWLRQ RQ GHYVGD


FDOOHG ³UHSOLFD´ 7KLV SDUWLWLRQ ZLOO FRQWDLQ RQO\ WKRVH GLUHFWRULHV DQG ILOHV ZH ZDQW
V\QFKURQL]HG EHWZHHQ RXU SULPDU\ DQG VHFRQGDU\ VHUYHU 1DPHO\ WKH LPSRUWDQW $VWHULVN
DQG (ODVWL[ UHODWHG GLUHFWRULHV DQG ILOHV

3
7KH XSSHU SDUW RI WKLV SLFWXUH VKRZV D FOXVWHU ZKHUH WKH OHIW QRGH LV FXUUHQWO\ DFWLYH LH
WKH VHUYLFH
V ,3 DGGUHVV WKDW WKH FOLHQW PDFKLQHV DUH WDONLQJ WR LV FXUUHQWO\ RQ WKH OHIW QRGH

7KH VHUYLFH LQFOXGLQJ LWV ,3 DGGUHVV FDQ EH PLJUDWHG WR WKH RWKHU QRGH DW DQ\ WLPH HLWKHU
GXH WR D IDLOXUH RI WKH DFWLYH QRGH RU DV DQ DGPLQLVWUDWLYH DFWLRQ 7KH ORZHU SDUW RI WKH
LOOXVWUDWLRQ VKRZV D GHJUDGHG FOXVWHU ,Q +$ VSHDN WKH PLJUDWLRQ RI D VHUYLFH LV FDOOHG
failover WKH UHYHUVH SURFHVV LV FDOOHG failback DQG ZKHQ WKH PLJUDWLRQ LV WULJJHUHG E\ DQ
DGPLQLVWUDWRU LW LV FDOOHG switchover.

,Q RXU LPSOHPHQWDWLRQ ZH ZLOO XWLOL]H +HDUWEHDW WR PRQLWRU WKH VWDWH RI WZR VHUYHUV DQG
GXULQJ D IDLORYHU PRXQW RXU V\QFKURQL]HG SDUWLWLRQ RQ WKH VHFRQGDU\ VHUYHU DQG VWDUW XS
WKH IROORZLQJ UHVRXUFHVDSSOLFDWLRQV DVWHULVN P\VTO DQG KWWS 'XULQJ IDLORYHU RXU IORDWLQJ
,3 DGGUHVV ZLOO PRYH IURP WKH SULPDU\ WR WKH VHFRQGDU\ VHUYHU 7KLV ,3 DGGUHVV VKRXOG EH
XVHG WR UHJLVWHU 6,3 DQG RWKHU 9R,3 HQGSRLQWV

Equipment Overview

7KLV LQVWDOODWLRQ VFHQDULR DVVXPHV WZR VHUYHUV HDFK ZLWK RQH (WKHUQHW LQWHUIDFHV DQG D
VLQJOH 6$7$ KDUG GULYH <RX PD\ KDYH D GLIIHUHQW W\SH RI KDUG GULYH ,'( 6&6, HWF DQG
WKHUHIRUH VRPH RI WKHVH VWHSV PD\ QHHG WR EH PRGLILHG WR EHWWHU UHIOHFW \RXU HQYLURQPHQW

4
yum –y update
x
3ULPDU\ S 
x
3DUWLWLRQQXPEHU  
x
3UHVVHQWHUXQWLOUHWXUQHGWRIGLVNFRPPDQGSURPSW
x
NOTELI\RXUVHUYHUVKDYHWZRGLIIHUHQWVL]HGKDUGGULYHVLWLVLPSHUDWLYHWKDWWKH
WKLUGSDUWLWLRQLVLGHQWLFDOLQVL]HRUWKH\ZLOOQHYHUV\QFKURQL]HRYHU'5%''RWKLV
E\DFFHSWLQJWKHGHIDXOWILUVWF\OLQGHUDQGWKHQVSHFLI\LQJWKH/DVWF\OLQGHUZLWKWKH
VL]H0RSWLRQ([00DNHWKHVHVDPHVSHFLILFDWLRQVRQERWKVHUYHUV
x Press “t” to change the partition system ID 
x Press “3” to choose partition number 
x &KRRVH+(;IRUW\SH
x Press “w” to save changes 

RESTART SERVER 

 )RUPDWQHZO\PDGHSDUWLWLRQ
mke2fs -j /dev/sda3

 1RZZHGHOHWHWKHILOHV\VWHPIURPWKHGLVNZHMXVWFUHDWHG
 dd if=/dev/zero bs=1M count=500 of=/dev/sda3; sync

 ,QVWDOO'5%'+HDUWEHDWDQGGHSHQGHQFLHVZLWK\XP

yum install heartbeat drbd83 kmod-drbd83

Note:,IE\DQ\FKDQFH\RXH[SHULHQFHSUREOHPVZLWKGUEGXVHGUEGYHUVLRQ ELW
YHUVLRQV 

7RHQVXUHSURSHUKRVWQDPHWR,3UHVROXWLRQLWLVUHFRPPHQGHGWKDW\RXPDQXDOO\
XSGDWHWKHHWFKRVWVILOHWRUHIOHFWSURSHUKRVWWR,3PDSSLQJ$GG WKH IROORZLQJ

192.168.1.243 voipserver.drbd
192.168.1.242 voipbackup.drbd




(GLWHWFGUEGFRQIRQ6HUYHUGUEG0RGLI\WKLVVDPSOHWRPHHW\RXUSDUWLFXODU
QHHGV

global { usage-count no; }


resource r0 {
protocol C;
startup { wfc-timeout 10; degr-wfc-timeout 30; } #change timers to your need
disk { on-io-error detach; } # or panic, ...
net {

6
after-sb-0pri discard-least-changes;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;
cram-hmac-alg "sha1";
shared-secret "Cent0Sru!3z";
}
syncer { rate 5M; }
on voipserver.drbd {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.1.242:7788;
meta-disk internal;
}
on voipbackup.drbd {
device /dev/drbd0;
disk /dev/sda3;
address 192.168.1.243:7788;
meta-disk internal;
}
}


Note:7KHIROORZLQJOLQHVDUHXVHGWRKHOSWKHVHUYHUVUHVROYHVSOLWEUDLQUHFRYHU\6SOLW
EUDLQLVZKHQWZRVHUYHUVDUHLQSULPDU\PRGHDQGQHHGWRNQRZKRZWRUHVROYHZKR
VKRXOGDVVXPHSULPDU\VHFRQGDU\UROH GLVFDUGLQJRUDFFHSWLQJFKDQJHVPDGHLQ
SULPDULHV 
 after-sb-0pri discard-least-changes;
after-sb-1pri discard-secondary;
after-sb-2pri call-pri-lost-after-sb;

Reference:KWWSZZZGUEGRUJXVHUVJXLGHVFRQILJXUHVSOLWEUDLQEHKDYLRUKWPO


5HSOLFDWHWKLVFRQILJILOH HWFGUEGFRQI WRWKHVHFRQGVHUYHU

scp /etc/drbd.conf root@voipbackup.drbd:/etc/

,QLWLDOL]HWKHPHWDGDWDDUHDRQGLVNEHIRUHVWDUWLQJGUEG RQERWKVHUYHU 

drbdadm create-md r0

6WDUWGUEGRQERWKQRGHV VHUYLFHGUEGVWDUW 

service drbd start

9HULI\WKDWERWKVHUYHUDUHVHFRQGDU\

cat /proc/drbd

7
$V\RXFDQVHHERWKQRGHVDUHVHFRQGDU\ZKLFKLVQRUPDOZHQHHGWRGHFLGH
ZKLFKQRGHZLOODFWDVDSULPDU\QRZ YRLSVHUYHUGUEG WKDWZLOOLQLWLDWHWKHILUVW
IXOO
V\QF
EHWZHHQWKHWZRQRGHV

drbdadm -- --overwrite-data-of-peer primary r0


/DXQFKWKHFRPPDQGDQGZDLWXQWLOLW’VILQLVKV\QFKURQL]LQJ


watch -n 1 cat /proc/drbd


:HFDQQRZIRUPDWGHYGUEGDQGPRXQWLWRQYRLSVHUYHUGUEG

 mkfs.ext3 /dev/drbd0

mkdir /replica

mount /dev/drbd0 /replica



:HFDQGHWHUPLQHWKHUROHRIDVHUYHUE\H[HFXWLQJWKHIROORZLQJ

drbdadm role r0

7KHSULPDU\VHUYHUVKRXOGUHWXUQ

3ULPDU\6HFRQGDU\

1RZZHZLOOFRS\DOORIWKHGLUHFWRULHVZHZDQWV\QFKURQL]HGEHWZHHQWKHWZR
VHUYHUVWRRXUQHZSDUWLWLRQUHPRYHWKHRULJLQDOGLUHFWRULHVDQGWKHQFUHDWH
V\PEROLFOLQNVWRUHSODFHWKHPRQYRLSVHUYHUGUEG

cd /replica
amportal chown
tar -zcvf etc-asterisk.tgz /etc/asterisk
tar -zxvf etc-asterisk.tgz
tar -zcvf var-lib-asterisk.tgz /var/lib/asterisk
tar -zxvf var-lib-asterisk.tgz
tar -zcvf usr-lib-asterisk.tgz /usr/lib/asterisk/
tar -zcvf var-www.tgz /var/www/
tar -zxvf usr-lib-asterisk.tgz
tar -zcvf var-spool-asterisk.tgz /var/spool/asterisk/
tar -zxvf var-spool-asterisk.tgz
tar -zcvf var-lib-mysql.tgz /var/lib/mysql/
tar -zxvf var-lib-mysql.tgz
tar -zcvf var-log-asterisk.tgz /var/log/asterisk/
tar -zxvf var-log-asterisk.tgz
tar -zxvf var-www.tgz
rm -rf /etc/asterisk
rm -rf /var/lib/asterisk
rm -rf /usr/lib/asterisk/
rm -rf /var/spool/asterisk
rm -rf /var/www
rm -rf /var/lib/mysql/
rm -rf /var/log/asterisk/
ln -s /replica/etc/asterisk/ /etc/asterisk
ln -s /replica/var/lib/asterisk/ /var/lib/asterisk
ln -s /replica/usr/lib/asterisk/ /usr/lib/asterisk
ln -s /replica/var/spool/asterisk/ /var/spool/asterisk
ln -s /replica/var/lib/mysql/ /var/lib/mysql
ln -s /replica/var/log/asterisk/ /var/log/asterisk
ln -s /replica/var/www /var/www
cd /

6WRSP\VTOGDVWHULVNDQGKWWSGVHUYLFHVRQYRLSVHUYHUGUEG
service mysqld restart
service mysqld stop
service asterisk stop
service httpd stop
service elastix-updaterd stop
service elastix-portknock stop


9HULI\VHUYLFHVDUHGRZQDQGSURFHHGWRVZLWFKPDQXDOO\WRWKHVHFRQGVHUYHU
>URRW#YRLS6(59(5GUEG@ umount /replica ; drbdadm secondary r0
1RZ VZLWFK WR WKH 92,3%$&.83 VHUYHU
>URRW#YRLS%$&.83GUEG@
 mkdir /replica ; drbdadm primary r0 ; mount /dev/drbd0 /replica
>URRW#YRLS%$&.83GUEG@ ls /replica/

Note:7KLVLVXVHGWRFKHFNLI\RXDUHUHSOLFDWLQJLQIRUPDWLRQRQERWKVHUYHUV<RXVKRXOG
VHHDOOGDWDUHSOLFDWHGLQWKHVHFRQGDU\VHUYHUMXVWOLNHGDWDLQWKHSULPDU\
'2 127 SHUIRUP WKLV DFWLRQ ZLWK WKH SK\VLFDO WHUPLQDO ORJJHG LQ 8VH 66+ 2WKHUZLVH LW ZLOO IDLO WR
XQPRXQW WKH UHSOLFD IROGHU IRU VRPH UHDVRQ$OVR PDNH VXUH \RX DUH QRW ,1 WKH UHSOLFD IROGHU 7\SH FG  

9HULI\YRLSVHUYHUGUEGVWDWXV 3ULPDU\6HFRQGDU\ 

drbdadm role r0

Execute ‘df –h’ on the primary to confirm that our /dev/drbd0 partition is
PRXQWHGDQGLQXVH

>URRW#YRLSVHUYHU a@ GI K
)LOHV\VWHP 6L]H 8VHG $YDLO 8VH 0RXQWHG RQ
GHYVGD * * *  
WPSIV *  *  GHYVKP
GHYGUEG * 0 *  UHSOLFD

Note:([HFXWLQJWKLVVDPHFRPPDQGLQYRLSEDFNXSGUEGZKLOHLQVHFRQGDU\PRGHVKRXOG
not display the /dev/drbd0 partition unless it’s assuming primary mode.

1RZZHZLOOUHPRYHDQGOLQNRQYRLSEDFNXSGUEG
rm -rf /etc/asterisk
rm -rf /var/lib/asterisk
rm -rf /usr/lib/asterisk/
rm -rf /var/spool/asterisk
rm -rf /var/lib/mysql/
rm -rf /var/log/asterisk/
rm -rf /var/www
ln -s /replica/etc/asterisk/ /etc/asterisk
ln -s /replica/var/lib/asterisk/ /var/lib/asterisk
ln -s /replica/usr/lib/asterisk/ /usr/lib/asterisk
ln -s /replica/var/spool/asterisk/ /var/spool/asterisk
ln -s /replica/var/lib/mysql/ /var/lib/mysql
ln -s /replica/var/log/asterisk/ /var/log/asterisk
ln -s /replica/var/www /var/www

6WRSP\VTOGDVWHULVNDQGKWWSGVHUYLFHVRQYRLSEDFNXSGUEG
service mysqld restart
service mysqld stop
service asterisk stop
service httpd stop
service elastix-updaterd stop
service elastix-portknock stop


 1RZVZLWFKEDFNWRWKHILUVWVHUYHU
>URRW#YRLS%$&.83GUEG@ umount /replica/ ; drbdadm secondary r0

1RZ VZLWFK WR WKH 92,36(59(5 VHUYHU
>URRW#YRLS6(59(5GUEG@ drbdadm primary r0 ; mount /dev/drbd0 /replica


'UEGLVZRUNLQJOHW
VEHVXUHWKDWLWZLOODOZD\VEHVWDUWHG
 chkconfig drbd on


Heartbeat Configuration

5HPHPEHUWRVWRSDQ\ERRWXSVHUYLFHVRQERWKVHUYHUVWKDWVKRXOGEHFRQWUROOHG
E\KHDUWEHDW7KHVHVHUYLFHVZLOOEHFRQWUROOHGE\KHDUWEHDWRQWKHVHUYHUWKDWLVLQ
FRQWURO
 chkconfig asterisk off
 chkconfig mysqld off
chkconfig httpd off
chkconfig elastix-updaterd off
chkconfig elastix-portknock off
service mysqld stop
service asterisk stop
service httpd stop
 service elastix-portknock stop
 service elastix-updaterd stop

 /HW
VFRQILJXUHDVLPSOHHWFKDGKDFIILOHRQYRLSVHUYHUGUEG
debugfile /var/log/ha-debug
logfile /var/log/ha-log

10
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694
bcast eth0
auto_failback off
node voipserver.drbd
node voipbackup.drbd

NOTE: I've set auto_failback to off. This seems more appropriate to me.
use the following command on the current secondary to switch back:

sh /usr/lib/heartbeat/hb_takeover

&UHDWHDOVRWKHHWFKDGDXWKNH\VRQYRLSVHUYHUGUEG
auth 1

1 sha1 MySecret

 &KDQJHSHUPLVVLRQVRQWKHHWFKDGDXWKNH\VILOHRQYRLSVHUYHUGUEG

chmod 600 /etc/ha.d/authkeys

(GLWHWFKDGKDUHVRXUFHVRQYRLSVHUYHUGUEG ,WLVWZROLQHV)RUPDWLQJLV
LPSRUWDQW  5HSODFH WKH HPDLO DGGUHVVHV ZLWK \RXU RZQ RQ WKH VHFRQG OLQH

voipserver.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::192.168.1.245/24/eth0/192.168.1.255 mysqld asterisk httpd elastix-updaterd elastix-portknock

voipserver.drbd MailTo::your@emailgoeshere.com,your@emailgoeshere.com::DRBD/HA-ALERT

6WDUWWKHKHDUWEHDWVHUYLFHRQYRLSVHUYHUGUEG

service heartbeat start

5HSOLFDWHQRZWKHKDFIDXWKNH\VDQGKDUHVRXUFHVWRYRLSEDFNXSGUEGDQGVWDUW
KHDUWEHDW


[root@voipserver.drbd ha.d]#VFSHWFKDGKDFIHWFKDGDXWKNH\VHWFKDGKDUHVRXUFHV
URRW#YRLSEDFNXSGUEGHWFKDG

[root@svoipbackup.drbd ha.d]# VHUYLFHKHDUWEHDWVWDUW


&RQILJXUHKHDUWEHDWWRLQLWLDOL]HDWERRWRQERWKVHUYHU

chkconfig --add heartbeat
chkconfig heartbeat on

9HULI\YRLSVHUYHUGUEGVWDWXV 3ULPDU\6HFRQGDU\ 
 drbdadm role r0

Execute ‘GI–h’ on the primary to confirm that our /dev/drbd0 partition is
PRXQWHGDQGLQXVH


11
>URRW#YRLSVHUYHU a@ GI K
)LOHV\VWHP 6L]H 8VHG $YDLO 8VH 0RXQWHG RQ
GHYVGD * * *  
WPSIV *  *  GHYVKP
GHYGUEG * 0 *  UHSOLFD


7HVW\RXUZRUNE\FUHDWLQJD6,3H[WHQVLRQRUDQ\WKLQJLQVLGH(ODVWL[:HE
,QWHUIDFHWKHQVKXWGRZQ\RXUSULPDU\VHUYHUZKLOHPDNLQJDFRQWLQXRXVSLQJWR
 IORDWLQJ,3DGGUHVV YHULI\LQJit doesn’t lose connectivity. Make
DQRWKHUFKDQJHLQWKHVHFRQGDU\VHUYHUWXUQ\RXUSULPDU\EDFNRQDQGDOO
FKDQJHVVKRXOGEHNHSWLQWDFW

Special Note:$Q\FKDQJHVPDGHWRDVWHULVNILOHVVKRXOGEHGRQHYLDZHE,QWHUIDFH
21/<'RQRWDWWHPSWWRXSJUDGH(ODVWL[YHUVLRQRQFHILQLVKHGWKHFOXVWHURUHOVHLWZLOO
ZULWHLWVRZQILOHVDJDLQGLVFDUGLQJOLQNVWRWKHUHSOLFDGLUHFWRU\

Troubleshooting:
tcpdump –i eth0:0 –s 1500 –w captura.pcap #capture traffic

mv captura.pcap /var/www/html #move file to web for download

Credits 

0D[L'LVWULEXFLRQHV&tD/WGD
ZZZJUXSRPD[LFRPHF

(VFXHOD3ROLWpFQLFDGHO(MpUFLWR
ZZZHVSHHGXHF

7HOHVRIW,QWHJUDQGR7HFKQRORJLHV

5HGIRQH

References

KWWSZLNLFHQWRVRUJ+RZ7RV+D'UEG
KWWSVXSSRUWUHGIRQHFRPGRZQORDGVHODVWL[(ODVWL[ +$ &OXVWHUSGI
KWWSGDQLHODOLDPDQFRPEORJILOHVSKRQHFXEHFOXVWHU$VWHULVN&OXVWHUSGI
KWWSZZZGUEGRUJXVHUVJXLGHVFRQILJXUHVSOLWEUDLQEHKDYLRUKWPO



Author:1LFN 5RVV EDVHG RQ WKH ZRUN RI 'DQLHO *HXYDUD DQG $PMDG -DEDOL


12
$33(1',; $

, FKDQJHG D IHZ WKLQJV ZLWKLQ WKH RULJLQDO  GRFXPHQW %HORZ LV P\ GHVFULSWLRQ DQG
UHDVRQLQJ IRU DOO RI WKHVH FKDQJHV

&+$1*(   , HGLWHG WKH LQVWUXFWLRQV IRU RQO\ RQH QHWZRUN FDUG :KLOH LWV EHVW LQ SUDFWLFH WR XVH D VHFRQG QHWZRUN
FDUG IRU '5%'LWVDFRPSOHWHO\RSWLRQDOVWHS 7R LPSOHPHQW WKLV , ORZHUHG WKH UDWH DW ZKLFK '5%' UHSOLFDWHV
IURP 0 WR 0  0ESV  7KLV LV WR DYRLG FUHDWLQJ FRQJHVWLRQ RI 9R,3 WUDIILF RQ UHSOLFDWLRQ $IWHU WKH LQLWLDO V\QF
YHU\ OLWWOH GDWD ZLOO EH WUDQVIHUUHG VR WKLV LV OLNHO\ QRW WR EH DQ LVVXH $GGLWLRQDOO\ '5%' QRZ XVHV WKH VDPH
,3 DGGUHVV    UDWKHU WKDQ D VHSDUDWH VXEQHW 5HPHPEHU WR FKDQJH WKH ,3 DGGUHVVHV WR WKRVH
RI \RXU VHUYHUV 7KH HWFKDGKDUHVRXUFHV ILOH FRQDWLQV WKH ,3 DGGUHVV IRU WKH FOXVWHU ZKLFK VKRXOG DOVR EH VHW E\ \RX
B
,) <28 :,6+ 72 86( $ 6(&21' 1,& \RX FDQ HDVLO\ GR VR E\ JRLQJ WR 67(3  WR HGLW GUEGFRQI DQG FKDQJLQJ
WKH ,3 DGGUHVVHV LQ WKLV ILOH WR PDWFK WKRVH RI \RXU GHGLFDWHG 1,&V <RX ZLOO DOVR ZDQW WR JR WR 67(3  DQG FKDQJH
KDFI WR VWDWH EFDVW HWK LQVWHDG RI HWK

&+$1*(  ,Q 67(3  DIWHUVESUL LV VHW WR GLVFDUGOHDVWFKDQJHV  , IHOW WKLV ZDV D EHWWHU RSWLRQ IRU DVWHULVN ,Q
WKH HYHQW WKDW D VSOLW EUDLQ RFFXUV LW UHDOO\ GRHVQ
W PDWWHU ZKR ZDV SULPDU\ ODVW :KDW PDWWHUV LV ZKR ZDV WKH IXQFWLRQLQJ
VHUYHU IRU RXWJRLQJ FDOOV 7KH IXQFWLRQLQJ VHUYHU LV OLNHO\ WR KDYH KDG PRUH FKDQJHV PDGH DQG WKHUHIRUH KDYH WKH PRVW
FXUUHQW FRQILJXUDWLRQ 7KLV LV LPSRUWDQW EHFDXVH WKH VESUL VHWWLQJ UHIHUV EDFN WR WKH VESUL VHWWLQJ

&+$1*(   %HLQJ ORJJHG LQWR WKH SK\VLFDO FRQVROHWHUPLQDO FDXVHG 6WHS  WR IDLO , PDGH DQ HQWU\ LQ WKH QRWHV

&+$1*(   6WHS  ZDV FKDQJHG WR FKNFRQILJ GUEG RQ  ,W SUHYLRXVO\ KDG FKNFRQILJ GUEG RQ  7KLV FRPPDQG
GLG QRW ZRUN IRU PH RQ  EXW P\ FRPPDQG GLG ZRUN

&+$*(   6WHS  ZDV FKDQJHG WR VHW DXWRBIDLOEDFN WR RII (YHU\ WLPH WKH VHUYHU IDLOV RYHU RU IDLOV EDFN VHUYLFH LV LQWHUUXSWHG
3HUVRQDOO\ ,
G OLNH WR LQYHVWLJDWH WKH UHDVRQ IRU WKH IDLORYHU EHIRUH , VZLWFK EDFN ,I WKH EDFNXS VHUYHU IDLOV
\RX ZLOO VWLOO EH DEOH WR DXWRPDWLFDOO\ IDLORYHU WR WKH RULJLQDO SULPDU\ VHUYHU VR WKHUH LV OLWWOH ULVN WR WKLV 2Q WKH RWKHU KDQG DXWRPDWLFDOO\
IDLOLQJ %$&. WR WKH RULJLQDO VHUYHU FRXOG FDXVH FRQVWDQW LQWHUUXSWLRQV LI LW KDG D ERXQFLQJ 1,& RU WKH VHUYHU
NHHSV UHVWDUWLQJ GXH WR D KDUGZDUH LVVXH
B
0DQXDO IDLOEDFN FDQ EH FRPSOHWHG E\ WKH FRPPDQG VK XVUOLEKHDUWEHDWKEBWDNHRYHU RQ
ZKLFKHYHU VHUYHU LV VHFRQGDU\ RU VK XVUOLEKHDUWEHDWKEBVWDQGE\ RQ ZKDWHYHU VHUYHU LV
FXUUHQWO\ WKH SULPDU\

&+$1*(   6WHS  ZDV HGLWHG WR DGG FKNFRQILJ KHDUWEHDW RQ  +HDUWEHDW GLG QRW DXWRPDWLFDOO\
VWDUW XQWLO WKLV FRPPDQG ZDV DGGHG

&+$1*(   6WHS  DGGHG HODVWL[XSGDWHUG HODVWL[SRUWNQRFN WR KDUHVRXUFHV ILOH 7KH VHUYLFHV ZHUH DOVR DGGHG WR
6WHSV  DQG  7KH\ UHO\ RQ YDUZZZ UHVRXUFHV VR WKH\ FDQ
W EH ORDGHG DW ERRW

&+$1*(   6WHS   DGGHG UP UI YDUZZZ DV LW ZDV PLVVLQJ IURP SUHYLRXV GRFXPHQWV

&+$1*(   6WHS  DGGHG D OLQH IRU HPDLO QRWLILFDWLRQV

&+$1*(  6WHS  DGGHG LQ DPSRUWDO FKRZQ 7KLV LV RXU ODVW FKDQFH WR GR WKLV DQG HQVXUH RZQHUVKLS DV LW GRHV QRW ZRUN
RQFH WKH ILOHV DUH PRYHG WR WKH GUEG GDWDEDVH 2WKHUZLVH ZH FRXOG EH VWXFN ZLWKRXW RZQHUVKLS ZKLFK FDQ KDSSHQ DIWHU DQ XSGDWH

 $SSHQGL[ DQG &KDQJHV E\ 1LFN 5


$33(1',; %
)ODVK 2SHUDWRU 3DQHO )L[
7KH )ODVK 2SHUDWRU 3DQHO LV GHSHQGHQW RQ D SURFHVV LQ RUGHU WR UHFHLYH XSGDWHV PDGH LQ )UHH3%;(ODVWL[
7KLV SURFHVV LV QRUPDOO\ ORDGHG RQ ERRW DQG FDOOHG ZLWKLQ WKH HWFLQLWGUFORFDO ILOH ,W UHTXLUHV ILOHV LQ
YDUZZZ WR VXFFHVVIXOO\ ODXQFK 8QIRUWXQDWHO\VLQFH ZH DUH XVLQJ '5%' WKH FOXVWHUHG GDWD KROGV YDUZZZ
DQG LV QRW DYDLODEOH RQ ERRW :H PXVW FRQILJXUH KHDUWEHDW WR ODXQFK WKLV SURFHVV EXW ILUVW ZH PXVW PDNH
D /LQX[ 6WDQGDUG %DVH VHUYLFH VFULSW <28 0867 '2 7+,6 21 %27+ 7+( 35,0$5< $1' 7+( 6(&21'$5<
6(59(56

7KLV RQO\ PDWWHUV LI \RX DFWXDOO\ XVH WKH IODVK RSHUDWRU SDQHO ,I \RX GRQ
W WKHQ WKHUH LV OLWWOH UHDVRQ WR GR WKLV

6WHS   7\SH WKH IROORZLQJ FRPPDQG RQ 92,36(59(5 ZKLFK ,
OO DVVXPH LV LQ WKH 35,0$5< UROH
QDQR HWFLQLWGIRSBVWDUW

6WHS   ,Q WKH HGLWRU FRS\ WKH IRSBVWDUW VFULSW WKHQ VDYH H[LW &75/2 WKHQ &75/;  7KH IRSBVWDUW VFULSW LV IRXQG
RQ WKH IROORZLQJ SDJH VRUU\ LW GLGQ
W ILW RQ WKLV SDJH 

6WHS   7\SH WKH FRPPDQG WR VHW WKH DSSURSULDWH SHUPLVVLRQV
FKPRG  HWFLQLWGIRSBVWDUW

6WHS   (GLW WKH HWFKDGKDUHVRXUFHV ILOH \RX FDQ W\SH QDQR HWFKDGKDUHVRXUFHV DQG DGG IRSBVWDUW WR WKH HQG
RI WKH ILUVW OLQH ,W VKRXOG QRZ ORRN OLNH WKLV
YRLSVHUYHUGUEG GUEGGLVN U )LOHV\VWHP GHYGUEGUHSOLFDH[W 3DGGUHWK P\VTOG DVWHULVN KWWSG HODVWL[XSGDWHUG HODVWL[SRUWNQRFN IRSBVWDUW
YRLSVHUYHUGUEG 0DLO7R\RXU#HPDLOJRHVKHUH FRP\RXU#HPDLOJRHVKHUH FRP'5%'+$$/(57

6WHS  <RX FDQ WHVW WR PDNH VXUH WKH IRSBVWDUW VHUYLFH LV ZRUNLQJ E\ W\SLQJ
VHUYLFH IRSBVWDUW VWDUW
VHUYLFH IRSBVWDUW VWDWXV
VHUYLFH IRSBVWDUW VWRS

7KLV VKRXOG SURGXFH D EXQFK RI PHVVDJHV DERXW DPSRUWDOFRQI ,W WKHQ VKRXOG HQG ZLWK DQ 2. IRU WKH VWRS DQG WKH
VWDUW FRPPDQG ,I LW GRHV QRW WKHQ \RXU FXUUHQW VHUYHU PLJKW EH WKH VHFRQGDU\ LQ ZKLFK FDVH D )$,/(' UHVXOW LV
WR EH H[SHFWHG  7R FKHFN LI WKH FXUUHQW VHUYHU LV WKH SULPDU\ W\SH GUEGDGP UROH U

6WHS   5(3($7 67(36  21 7+( 27+(5 6(59(5 92,3%$&.83  6WHS  ZLOO UHVXOW LQ D )$,/(' PHVVDJH
XQWLO LW DVVXPHV WKH SULPDU\ UROH 5XQQLQJ VK XVUOLEKHDUWEHDWKEBWDNHRYHU ZLOO PDNH \RX DVVXPH WKH SULPDU\ UROH
6FULSW IRU HWFLQLWGIRSBVWDUW
 ELQEDVK
 GHVFULSWLRQ )23 GDHPRQ
 SURFHVV QDPH IRSBVWDUW
 $XWKRU 1LFN 5RVV
 HWFLQLW GIXQFWLRQV
> I XVUVELQDPSRUWDO @ __ H[LW 
)23 XVUVELQDPSRUWDO
);2 XVUVELQI[RWXQH
5(79$/ 
JHWSLG ^
SLG SV HR SLGFRPP _ JUHS RSBVHUYHUSO _ DZN
^ SULQW  `

`
VWDUW ^
HFKR Q 6WDUWLQJ )23 
JHWSLG
LI > ] SLG @ WKHQ
);2 V !GHYQXOO
)23 VWDUWBIRS ! GHYQXOO
5(79$/ "
IL
LI > 5(79$/ HT  @ WKHQ
WRXFK YDUORFNVXEV\VIRSBVWDUW
HFKRBVXFFHVV
HOVH
HFKRBIDLOXUH
IL
HFKR
UHWXUQ 5(79$/
`
VWRS ^
HFKR Q 6WRSSLQJ )23 
JHWSLG
5(79$/ "
LI > Q SLG @ WKHQ
)23 VWRSBIRS ! GHYQXOO
VOHHS 
JHWSLG
LI > ] SLG @ WKHQ
UP I YDUORFNVXEV\VIRSBVWDUW
HFKRBVXFFHVV
HOVH
HFKRBIDLOXUH
IL
HOVH
HFKRBIDLOXUH
IL
HFKR
UHWXUQ 5(79$/
`
 6HH KRZ ZH ZHUH FDOOHG
FDVH  LQ
VWDUW
VWDUW

VWRS
VWRS

VWDWXV
JHWSLG
LI > Q SLG @ WKHQ
HFKR )23 SLG SLG LV UXQQLQJ
HOVH
5(79$/ 
HFKR )23 LV VWRSSHG
IL

UHVWDUW
VWRS
VWDUW


HFKR 8VDJH  ^VWDUW_VWRS_VWDWXV_UHVWDUW`
H[LW 

HVDF
H[LW 5(79$/


$33(1',; &
8SGDWLQJ $VWHULVN :LWK '5%'

6WHS   .HHS GDWD RQ ERWK VHUYHUV E\ VKXWWLQJ GRZQ RQH VHUYHU :H ZDQW WR PDNH VXUH
RQO\ RQH VHUYHU LV RQ DW D WLPH , VKXW GRZQ YRLSEDFNXS DQG OHDYH YRLSVHUYHU RQ

6WHS   6WRS 6HUYLFHV RQ WKH DFWLYH PDFKLQH DQG PDNH VXUH WKH\ DOZD\V VWD\ RII KHDUWEHDW PDQDJHV WKHP
FKNFRQILJ DVWHULVN RII
FKNFRQILJ P\VTOG RII
FKNFRQILJ KWWSG RII
VHUYLFH P\VTOG UHVWDUW
VHUYLFH P\VTOG VWRS
VHUYLFH DVWHULVN VWRS
VHUYLFH KWWSG VWRS
VHUYLFH HODVWL[XSGDWHUG VWRS
VHUYLFH HODVWL[SRUWNQRFN VWRS
VHUYLFH IRSBVWDUW VWRS

6WHS   5HPRYH OLQNV IURP WKH UHSOLFD IROGHU WR WKH ILOH V\VWHP
XQOLQN HWFDVWHULVN
XQOLQN YDUOLEDVWHULVN
XQOLQN XVUOLEDVWHULVN
XQOLQN YDUVSRRODVWHULVN
XQOLQN YDUOLEP\VTO
XQOLQN YDUORJDVWHULVN
XQOLQN YDUZZZZZZ
XQOLQN YDUZZZ

6WHS   &RS\ DOO WKH ILOHV LQ WKH UHSOLFD IROGHU EDFN WR WKH URRW SDUWLWLRQ
PNGLU HWFDVWHULVN
PNGLU YDUOLEDVWHULVN
PNGLU XVUOLEDVWHULVN
PNGLU YDUVSRRODVWHULVN
PNGLU YDUOLEP\VTO
PNGLU YDUORJDVWHULVN
PNGLU YDUZZZ
FS DUI UHSOLFDHWFDVWHULVN HWFDVWHULVN
FS DUI UHSOLFDYDUOLEDVWHULVN YDUOLEDVWHULVN
FS DUI UHSOLFDXVUOLEDVWHULVN XVUOLEDVWHULVN
FS DUI UHSOLFDYDUVSRRODVWHULVN YDUVSRRODVWHULVN
FS DUI UHSOLFDYDUOLEP\VTO YDUOLEP\VTO
FS DUI UHSOLFDYDUORJDVWHULVN YDUORJDVWHULVN
\HV _ FS DUI UHSOLFDYDUZZZ YDUZZZ


67(3   5HPRYH HYHU\WKLQJ LQ WKH UHSOLFD IROGHU
UP UI UHSOLFD

67(3   5XQ WKH \XP XSGDWH IRU HODVWL[ WKHQ DPSRUWDO FKRZQ WR HQVXUH SURSHU RZQHUVKLS DIWHU WKH XSGDWH
\XP \ XSGDWH HODVWL[
DPSRUWDO FKRZQ

67(3  5HFRS\ DQG OLQN HYHU\WKLQJ WR WKH UHSOLFD IROGHU

cd /replica
tar -zcvf etc-asterisk.tgz /etc/asterisk
tar -zxvf etc-asterisk.tgz
tar -zcvf var-lib-asterisk.tgz /var/lib/asterisk
tar -zxvf var-lib-asterisk.tgz
tar -zcvf usr-lib-asterisk.tgz /usr/lib/asterisk/
tar -zcvf var-www.tgz /var/www/
tar -zxvf usr-lib-asterisk.tgz
tar -zcvf var-spool-asterisk.tgz /var/spool/asterisk/
tar -zxvf var-spool-asterisk.tgz
tar -zcvf var-lib-mysql.tgz /var/lib/mysql/
tar -zxvf var-lib-mysql.tgz
tar -zcvf var-log-asterisk.tgz /var/log/asterisk/
tar -zxvf var-log-asterisk.tgz
tar -zxvf var-www.tgz
rm -rf /etc/asterisk
rm -rf /var/lib/asterisk
rm -rf /usr/lib/asterisk/
rm -rf /var/spool/asterisk
rm -rf /var/lib/mysql/
rm -rf /var/log/asterisk/
rm -rf /var/www
ln -s /replica/etc/asterisk/ /etc/asterisk
ln -s /replica/var/lib/asterisk/ /var/lib/asterisk
ln -s /replica/usr/lib/asterisk/ /usr/lib/asterisk
ln -s /replica/var/spool/asterisk/ /var/spool/asterisk
ln -s /replica/var/lib/mysql/ /var/lib/mysql
ln -s /replica/var/log/asterisk/ /var/log/asterisk
ln -s /replica/var/www /var/www
cd /

6WHS   6KXW GRZQ WKH 3ULPDU\ VHUYHU YRLSVHUYHU



6WHS  6WDUW XS VHFRQGDU\ YRLSEDFNXS  UXQ WKH FRPPDQG
VK XVUOLEKHDUWEHDWKEBWDNHRYHU

6WHS   5HSHDW 6WHSV  RQ YRLSEDFNXS

6WHS   %RRW XS WKH YRLSVHUYHU WKHQ RQFH LWV XS IRU D IHZ PLQXWHV DQG V\QFHG
UHVWDUW YRLSEDFNXS 8VH WKH FRPPDQG VHUYLFH GUEG VWDWXV WR FKHFN V\QFLQJ ,I \RX
GLG HYHU\WKLQJ ULJKW \RX ZLOO KDYH DQ XSGDWHG HODVWL[DVWHULVN V\VWHP DQG WKH '5%'
GDWH ZLOO UHFRQFLOH


$33(1',; '
5(6,=,1* 25 029,1* WKH '5%' 3DUWLWLRQ

7KLV LV D YHU\ ORQJ DQG GUDZQ RXW SURFHVV DQG LV QRW IRU WKH ZHDN RI KHDUW 7KH FRQFHSW LV VLPLODU WR WKH SUHYLRXV VHFWLRQ
ZKHUH ZH PRYH HYHU\WKLQJ RII WKH '5%' SDUWLWLRQ DQG WKHQ EDFN RQ ,I \RX
YH FKDQJHG WKH SDUWLWLRQ RU ZDQW WR VSHFLI\ D
GLIIHUHQW VL]H WKDQ ZKDW , KDYH WKHQ PDNH WKH FKDQJHV DFFRUGLQJO\ 3OHDVH HQVXUH \RX SHUIRUP WKH VWHSV RQ WKH FRUUHFW
VHUYHU

 21 92,3%$&.83  
67(3   :H KDYH WR PDNH VXUH WKDW 92,3%$&.83 ZLOO QRW VWDUW '5%' RU +($57%($7 RQ UHERRW
FKNFRQILJ KHDUWEHDW RII
FKNFRQILJ GUEG RII
6KXW GRZQ WKH 92,3 %$&.83 VHUYHU!

#### 21 92,36(59(5  ####
6WHS   :H PXVW WXUQ DOO WKH VHUYLFHV RII XQOLQN WKH IROGHUV LQ UHSOLFD WKHQ FRS\ WKH GDWD EDFN WR WKH URRW SDUWLWLRQ
FKNFRQILJ DVWHULVN RII
FKNFRQILJ P\VTOG RII
FKNFRQILJ KWWSG RII
VHUYLFH P\VTOG UHVWDUW
VHUYLFH P\VTOG VWRS
VHUYLFH DVWHULVN VWRS
VHUYLFH KWWSG VWRS
VHUYLFH HODVWL[XSGDWHUG VWRS
VHUYLFH HODVWL[SRUWNQRFN VWRS
VHUYLFH IRSBVWDUW VWRS

XQOLQN HWFDVWHULVN
XQOLQN YDUOLEDVWHULVN
XQOLQN XVUOLEDVWHULVN
XQOLQN YDUVSRRODVWHULVN
XQOLQN YDUOLEP\VTO
XQOLQN YDUORJDVWHULVN
XQOLQN YDUZZZZZZ
XQOLQN YDUZZZ
PNGLU HWFDVWHULVN
PNGLU YDUOLEDVWHULVN
PNGLU XVUOLEDVWHULVN
PNGLU YDUVSRRODVWHULVN
PNGLU YDUOLEP\VTO
PNGLU YDUORJDVWHULVN
PNGLU YDUZZZ

FS DUI UHSOLFDHWFDVWHULVN HWFDVWHULVN
FS DUI UHSOLFDYDUOLEDVWHULVN YDUOLEDVWHULVN
FS DUI UHSOLFDXVUOLEDVWHULVN XVUOLEDVWHULVN
FS DUI UHSOLFDYDUVSRRODVWHULVN YDUVSRRODVWHULVN
FS DUI UHSOLFDYDUOLEP\VTO YDUOLEP\VTO
FS DUI UHSOLFDYDUORJDVWHULVN YDUORJDVWHULVN
\HV _ FS DUI UHSOLFDYDUZZZ YDUZZZ
DPSRUWDO FKRZQ 
#### 6WLOO RQ 92,36(59(5  ####
6WHS  :H PXVW QRZ WXUQ WKH VHUYLFHV RII WHPSRUDULO\ SUHYHQW WKHP IURP VWDUWLQJ XS DQG UHPRYH UHFUHDWH WKH
'5%' SDUWLWLRQ :KHQ SDUWLWLRQLQJ SOHDVH W\SH LQ WKH FRPPDQGV PDQXDOO\ ,W ZLOO QRW FRS\ DQG SDVWH LQ DQG \RX
QHHG WR EH VXUH \RX DUH XVLQJ WKH ULJKW SDUWLWLRQV 0DNH VXUH\RX
YH UHDG WKH LQIRUPDWLRQ DQG DUH DQVZHULQJ FRUUHFWO\
,I \RX DUH FKDQJLQJ WKH SDUWLWLRQ RU GULYH \RX ZLOO QHHG WR PDNH PRGLILFDWLRQV DFFRUGLQJO\

FKNFRQILJ KHDUWEHDW RII
FKNFRQILJ GUEG RII
VHUYLFH KHDUWEHDW VWRS
VHUYLFH GUEG VWRS

IGLVN GHYVGD
S
G


S
Q
S
SUHVV HQWHU!
0 RU ZKDWHYHU VL]H \RX GHVLUH

W


Z

67(3  5HERRW 92,36(59(5 6WDUW XS 92,3%$&.83

 21 92,3%$&.83  
67(3  :H PXVW QRZ IRUFH WKH EDFNXS VHUYHU WR D '5%' SULPDU\ UROH DQG UHPRXQW WKH UHSOLFD IROGHU
VHUYLFH GUEG VWDUW
GUEGDGP SULPDU\ U  PRXQW GHYGUEG UHSOLFD

67(3   :H PXVW SHUIRUP WKH FRS\ IURP WKH UHSOLFD IROGHU EDFN WR WKH URRW SDUWLWLRQ DV ZH GLG RQ WKH RWKHU VHUYHU
XQOLQN HWFDVWHULVN
XQOLQN YDUOLEDVWHULVN
XQOLQN XVUOLEDVWHULVN
XQOLQN YDUVSRRODVWHULVN
XQOLQN YDUOLEP\VTO
XQOLQN YDUORJDVWHULVN
XQOLQN YDUZZZZZZ
XQOLQN YDUZZZ

67(3  LV FRQWLQXHG RQ WKH QH[W SDJH 1ϵ
67(3  LV FRQWLQXHG KHUH


PNGLU HWFDVWHULVN
PNGLU YDUOLEDVWHULVN
PNGLU XVUOLEDVWHULVN
PNGLU YDUVSRRODVWHULVN
PNGLU YDUOLEP\VTO
PNGLU YDUORJDVWHULVN
PNGLU YDUZZZ
FS DUI UHSOLFDHWFDVWHULVN HWFDVWHULVN
FS DUI UHSOLFDYDUOLEDVWHULVN YDUOLEDVWHULVN
FS DUI UHSOLFDXVUOLEDVWHULVN XVUOLEDVWHULVN
FS DUI UHSOLFDYDUVSRRODVWHULVN YDUVSRRODVWHULVN
FS DUI UHSOLFDYDUOLEP\VTO YDUOLEP\VTO
FS DUI UHSOLFDYDUORJDVWHULVN YDUORJDVWHULVN
\HV _ FS DUI UHSOLFDYDUZZZ YDUZZZ
DPSRUWDO FKRZQ

67(3   :H PXVW QRZ HQG WKH '5%' VHUYLFH DQG UHSDUWLWLRQ 92,3%$&.83 DV ZH GLG RQ WKH SUHYLRXV VHUYHU
VHUYLFH GUEG VWRS

IGLVN GHYVGD
S
G


S
Q
S
SUHVV HQWHU!
0 RU ZKDWHYHU VL]H \RX GHVLUH  MXVW PDNH VXUH LWV WKH 6$0( 6,=( 21 %27+ 6(59(56

W


Z

67(3   5HERRW 92,3%$&.83 92,36(59(5 VKRXOG DOVR EH RQ DQG UXQQLQJ

 21 %27+ 92,36(59(5 $1' 92,3%$&.83 
67(3  3HUIRUP WKHVH FRPPDQGV RQ 92,36(59(5 ILUVW WKHQ SHUIRUP WKHP DJDLQ RQ 92,3%$&.83 <RX PXVW
GR WKLV RQ %27+ PDFKLQHV :H UXQ GUEGDGP FUHDWHPG U WR HQVXUH WKDW WKH GUEG GDWD LV D IUHVK UHZULWH DQG QRW
WKH ROG PHWD GDWD 7KLV HQVXUHV LW XVHV WKH IXOO VL]H RI WKH QHZ SDUWLWLRQ $OVR GR QRW FXW DQG SDVW HQWHU \HV!

PNHIV ±M GHYVGD
GG LI GHY]HUR EV 0 FRXQW  RI GHYVGD V\QF
GUEGDGP FUHDWHPG U
HQWHU \HV!
GUEGDGP FUHDWHPG U
VHUYLFH GUEG VWDUW

#### 21 92,36(59(5  ####
67(3   2Q YRLSVHUYHU RQO\ LVVXH WKH IROORZLQJ FRPPDQGV 'R QRW FRS\ DQG SDVWH WKH VHFRQG OLQH ,W LV DQ
LQVWUXFWLRQ IRU \RX WR ZDLW 8VH VHUYLFH GUEG VWDWXV WR FKHFN LI \RX DUH  V\QFHG

GUEGDGP  RYHUZULWHGDWDRISHHU SULPDU\ U
ZDLW IRU WKH V\VWHPV WR V\QF   FKHFN ZLWK VHUYLFH GUEG VWDWXV
PNIVH[W GHYGUEG
PNGLU UHSOLFD
PRXQW GHYGUEG UHSOLFD

67(3   :H DUH QRZ JRLQJ WR WDNH WKH GDWD RQ 92,36(59(5 DQG XVH LW WR ILOO WKH UHSOLFD IROGHU :H ZLOO WKHQ
XQPRXQW WKH UHSOLFD SXW 92,36(59(5 LQ D VHFRQGDU\ VWDWXV DQG WKHQ VHW WKH VHUYLFHV WR EH DFWLYH ZKHQ ZH UHERRW
'2 127 5(%227 92,36(59(5 <(7

FG UHSOLFD
WDU ]FYI HWFDVWHULVNWJ] HWFDVWHULVN
WDU ][YI HWFDVWHULVNWJ]
WDU ]FYI YDUOLEDVWHULVNWJ] YDUOLEDVWHULVN
WDU ][YI YDUOLEDVWHULVNWJ]
WDU ]FYI XVUOLEDVWHULVNWJ] XVUOLEDVWHULVN
WDU ]FYI YDUZZZWJ] YDUZZZ
WDU ][YI XVUOLEDVWHULVNWJ]
WDU ]FYI YDUVSRRODVWHULVNWJ] YDUVSRRODVWHULVN
WDU ][YI YDUVSRRODVWHULVNWJ]
WDU ]FYI YDUOLEP\VTOWJ] YDUOLEP\VTO
WDU ][YI YDUOLEP\VTOWJ]
WDU ]FYI YDUORJDVWHULVNWJ] YDUORJDVWHULVN
WDU ][YI YDUORJDVWHULVNWJ]
WDU ][YI YDUZZZWJ]
UP UI HWFDVWHULVN
UP UI YDUOLEDVWHULVN
UP UI XVUOLEDVWHULVN
UP UI YDUVSRRODVWHULVN
UP UI YDUOLEP\VTO
UP UI YDUORJDVWHULVN
UP UI YDUZZZ
OQ V UHSOLFDHWFDVWHULVN HWFDVWHULVN
OQ V UHSOLFDYDUOLEDVWHULVN YDUOLEDVWHULVN
OQ V UHSOLFDXVUOLEDVWHULVN XVUOLEDVWHULVN
OQ V UHSOLFDYDUVSRRODVWHULVN YDUVSRRODVWHULVN
OQ V UHSOLFDYDUOLEP\VTO YDUOLEP\VTO
OQ V UHSOLFDYDUORJDVWHULVN YDUORJDVWHULVN
OQ V UHSOLFDYDUZZZ YDUZZZ
FG 

XPRXQW UHSOLFD  GUEGDGP VHFRQGDU\ U

FKNFRQILJ KHDUWEHDW RQ
FKNFRQILJ GUEG RQ

Ϯϭ
 21 92,3%$&.83  
67(3   2Q 92,3%$&.83 ZH DUH QRZ JRLQJ WR DVVXPH WKH SULPDU\ UROH DQG PRXQW WKH UHSOLFD SDUWLWLRQ
2QFH WKLV LV GRQH ZH UHPRYH WKH RULJLQDO IROGHUV FUHDWH WKH VRIW OLQNV DQG WKHQ VHW WKH VHUYLFHV WR WXUQ RQ DIWHU D
UHERRW

PNGLU UHSOLFD  GUEGDGP SULPDU\ U  PRXQW GHYGUEG UHSOLFD

UP UI HWFDVWHULVN
UP UI YDUOLEDVWHULVN
UP UI XVUOLEDVWHULVN
UP UI YDUVSRRODVWHULVN
UP UI YDUOLEP\VTO
UP UI YDUORJDVWHULVN
UP UI YDUZZZ
OQ V UHSOLFDHWFDVWHULVN HWFDVWHULVN
OQ V UHSOLFDYDUOLEDVWHULVN YDUOLEDVWHULVN
OQ V UHSOLFDXVUOLEDVWHULVN XVUOLEDVWHULVN
OQ V UHSOLFDYDUVSRRODVWHULVN YDUVSRRODVWHULVN
OQ V UHSOLFDYDUOLEP\VTO YDUOLEP\VTO
OQ V UHSOLFDYDUORJDVWHULVN YDUORJDVWHULVN
OQ V UHSOLFDYDUZZZ YDUZZZ

FKNFRQILJ KHDUWEHDW RQ
FKNFRQILJ GUEG RQ

67(3  ),1$/ 67(3  5HERRW 92,3%$&.83 ILUVW DQG WKHQ UHERRW 92,36(59(5 $W WKLV SRLQW WKH VHUYHUV ZLOO
FRPH EDFN XS DQG RQH VKRXOG DVVXPH WKH 35,0$5< UROH DQG EHJLQ UHSOLFDWLQJ FRUUHFWO\ <RXU QHZ '5%' SDUWLWLRQ
VKRXOG EH VL]HG FRUUHFWO\ UXQ GI K WR FKHFN WKLV RQ WKH SULPDU\ 

,I \RX KDYH DXWRBIDLOEDFN VHW WR RII DQG \RX GR QRW OLNH ZKLFK VHUYHU DVVXPHG WKH SULPDU\ UROH WKHQ VZLWFK WR
ZKLFKHYHU VHUYHU LV VHFRQGDU\ DQG W\SH

VK XVUOLEKHDUWEHDWKEBWDNHRYHU


7KLV ZLOO VHW WKDW VHUYHU DV WKH '5%' SULPDU\ QRGH :KHZ 7KDW ZDV D ORW RI ZRUN , KRSH \RX QHYHU KDYH WR GR WKLV
,W ZDV D FRPSOHWH SDLQ LQ WKH DVV WR ERWK GR DQG WR GRFXPHQW 7KH EHWWHU VWUDWHJ\ LV WR JLYH \RXUVHOI SOHQW\ RI VSDFH
LQ WKH EHJLQQLQJ

Ϯ2
$33(1',; (
8VLQJ 7ZR 1HWZRUN ,QWHUIDFH &DUGV

7KH RULJLQDO JXLGH WKDWPLQHLV EDVHG RQ XVHG D GXDO 1,& FRQILJXUDWLRQ 8VLQJ D VHSDUDWH 1,& IRU '5%' LV XVXDOO\ D
JRRG LGHD <RX ZDQW WKH UHSOLFDWLRQ WR ZRUN WKURXJK DQ LQGHSHQGHQW FKDQQHO W\SLFDOO\ ZKHUH EDQGZLGWK IRU \RXU
VHUYLFHV ZLOO QRW EH VWDUYHG E\ '5%' UHSOLFDWLRQ WUDIILF 7KHUH DUH PDQ\ VLWXDWLRQV KRZHYHU ZKHUH WKLV LV QRW
QHFHVVDULO\SRVVLEOH0DQ\HODVWL[VHUYHUVDUHVPDOO RUHYHQYLUWXDOL]HG DSSOLDQFHVDQGWKHVHFRQG1,&LVDVRXUFHRI
XQQHFHVVDU\UHTXLUHPHQWVDQGFRPSOH[LWLHV(LWKHULPSOHPHQWDWLRQZLOOOLNHO\JHWWKHMREGRQH<RXPD\ZDQWWR
FRQVLGHUWKLVGXDO1,&FRQILJXUDWLRQLI\RXDUHLQDFDOOFHQWHUDQGKDYHUHFRUGLQJUHTXLUHPHQWV

$VWHULVN VHUYHUV DUH XVXDOO\ QRW YHU\ GLVN VHQVLWLYH VR LWV XQOLNHO\ '5%' ZLOO EH VHQGLQJ PXFK GDWD ,I \RX DUH ZRUULHG
DERXW '5%' XVLQJ WRR PXFK EDQGZLGWK WKHQ \RX FDQ ORZHU WKH UHSOLFDWLRQ UDWH OLPLW EHORZ  0%V ZKLFK LV  0ELWV 

,I \RX ZLVK WR XVH WZR QHWZRUN FDUGV WKH FKDQJHV WR WKH RULJLQDO LPSOHPHQWDWLRQ DUH YHU\ VLPSOH ,Q WKLV VFHQDULR ZH
ZLOO DVVXPH 92,36(59(5 KDV D VHFRQGDU\ 1,& HWK ZLWK DQ ,3 DGGUHVV RI   :H ZLOO DVVXPH 92,3%$&.83
KDV D VHFRQGDU\ 1,& HWK ZLWK DQ ,3 DGGUHVV RI  

1RWH WKH LS DGGUHVVHV GR QRW QHHG WR EH LQ D GLIIHUHQW 7KH FDQ EH ZKDWHYHU \RX DVVLJQ WKHP WR -XVW PDNH VXUH WKH\
DUH VWDWLF DGGUHVVHV RU \RX KDYH VHW D '+&3 UHVHUYDWLRQ IRU WKHVH 1,&V

 0$.( 7+( &+$1*(6 21 %27+ 6(59(56
6WHS  &KDQJH WKH IROORZLQJ VHFWLRQ RI WKH HWFGUEGFRQI

V\QFHU ^ UDWH 0 `


RQ YRLSVHUYHUGUEG ^
GHYLFH GHYGUEG
GLVN GHYVGD
DGGUHVV 
PHWDGLVN LQWHUQDO
`
RQ YRLSEDFNXSGUEG ^
GHYLFH GHYGUEG
GLVN GHYVGD
DGGUHVV 
PHWDGLVN LQWHUQDO
`
`

6WHS  &KDQJH WKH IROORZLQJ VHFWLRQ RI WKH HWFKDGKDFI ILOH

ZDUQWLPH 
LQLWGHDG 
XGSSRUW 
EFDVW HWK
DXWRBIDLOEDFN RII
QRGH YRLSVHUYHUGUEG
QRGH YRLSEDFNXSGUEG

6WHS   5HERRW ERWK VHUYHUV DQG \RX DUH GRQH

Ϯϯ
$33(1',; )
0RYLQJ 7)73%227
7KH 7)73 ERRW FDQ EH HDVLO\ PRYHG E\ FRS\LQJ LW WR WKH UHSOLFD IROGHU GHOHWLQJ WKH RULJLQDO IROGHU DQG WKHQ FUHDWLQJ D V\PEROLF
OLQN WR WKH UHSOLFD IROGHU $V \RX FDQ VHH LW LV YHU\ VLPLODU WR WKH SURFHGXUH ZH
YH XVHG IRU DOO RWKHU IROGHUV
6WHS  *R RQ WKH DFWLYHSULPDU\ FRPSXWHU DQG W\SH WKH IROORZLQJ FRPPDQGV
FG UHSOLFD
WDU ]FYI WIWSERRWWJ] WIWSERRW
WDU ][YI WIWSERRWWJ]
UP UI WIWSERRW
OQ V UHSOLFDWIWSERRW WIWSERRW
6WHS  *R RQ WKH VHFRQGDU\ FRPSXWHU DQG W\SH WKHVH FRPPDQGV
UP UI WIWSERRW
OQ V UHSOLFDWIWSERRW WIWSERRW
7KDW LV DOO WKHUH LV WR LW

Ϯϰ
$33(1',; *
,QVHUWLQJ D SDVVLYH WKLUG QRGH QR +$
:KLOH '5%' DQG +HDUWEHDW GR QRW RIIHU WKH DELOLW\ WR IDLO RYHU WR D WKLUG QRGH \RX FDQ VHW XS D WKLUG QRGH IRU SDVVLYH UHSOLFDWLRQ ,W
ZLOO FRS\ HYHU\WKLQJ RQ WKH FXUUHQW '5%' FOXVWHU DQG DOZD\V EH D VHFRQGDU\ QRGH 7KHUH DUH PDQ\ FRPSOLFDWLRQV ZLWK WKLV SURFHVV
DQG WKH RQO\ UHDO SUDFWLFDO XVH LV IRU GLVDVWHU UHFRYHU\ 7KDW EHLQJ VDLG WKLV LV DQ RUGHU RI PDJQLWXGH PRUH GLIILFXOW WKDQ WKH WZR QRGH
FOXVWHU DQG WKLV GRFXPHQWDWLRQ SURYHG WRR WLPH LQWHQVLYH WR ZULWH WKRURXJKO\ +RSHIXOO\ WKLV ZLOO DW OHDVW JXLGH \RX

-XVW D QRWH  LQ RUGHU WR GR WKLV \RX EDVLFDOO\ KDYH WR UHFUHDWH WKH RULJLQDO '5%' VHWXS RQ \RXU FOXVWHUHG SDUWLWLRQV ,I \RX DOUHDG\
DUH XVLQJ '5%' IROORZ WKH JXLGH LQ $SSHQGL[ ' WR WKH SRLQW ZKHUH \RX IGLVN \RXU SDUWLWLRQV 7KLV ZLOO FRS\ \RXU DVWHULVN
FRQILJV EDFN WR ERWK VHUYHUV VR \RX GR QRW ORVH \RXU FRSLHV

:H KDYH  VHUYHUV 92,36(59(5 92,3%$&.83 DQG 92,3'5 RXU SDVVLYH QRGH IRU GLVDVWHU UHFRYHU\ 
92,3'5 ZLOO EH 

6WHS  )ROORZ WKH PDLQ JXLGH IRU VWHSV  RQ $// VHUYHUV

6WHS  )RU VWHS  RQ WKH RULJLQDO JXLGH RQ DOO VHUYHUV \RX ZLOO QHHG WR DGG WKLV OLQH LQ DGGLWLRQ WR WKH RWKHUV LQ VWHS 
 YRLSGUGUEG

6WHS  )RU 6WHS  RQ WKH RULJLQDO JXLGH RQ DOO VHUYHUV DGG WKH IROORZLQJ WR GUEGFRQI $)7(5 WKH YHU\ ODVW EUDFNHW
UHVRXUFH U ^
SURWRFRO $
QHW
^
`
V\QFHU^ UDWH 0 `
VWDFNHGRQWRSRI U ^
GHYLFH GHYGUEG
DGGUHVV 
`
RQ YRLSGUGUEG ^
GHYLFH GHYGUEG
GLVN GHYVGD
DGGUHVV 
PHWDGLVN LQWHUQDO
`
`
6WHS  &RQWLQXH ZLWK WKH RULJLQDO JXLGH RQ RQO\ YRLSVHUYHU DQG YRLSEDFNXS 6723 DW VWHS  '2 127 SHUIRUP WKHVH VWHSV RQ 9RLS'5

 VNLS IURP VWHS  WR  RQ WKH RULJLQDO JXLGH

6WHS  3HUIRUP VWHS   LQ WKH RULJLQDO JXLGH RQ $// 6(59(56

6WHS   :H QRZ PXVW FRQILJXUH KHDUWEHDW WHPSRUDULO\ 7KLV LV PXFK HDUOLHU WKDQ LQ WKH RULJLQDO JXLGH EXW LW LV UHTXLUHG
WR HYHQ JHW WKH WKLUG QRGH XS 3HUIRUP VWHSV  DQG  RQ 92,36(59(5 DQG 92,3%$&.83 7KHUH ZLOO EH
12 +($57%($7 RQ 92,3'5 (9(5 ,I \RX WU\ WR GR WKLV RQ 92,3'5 \RX ZLOO EUHDN HYHU\WKLQJ

Ϯϱ
6WHS  ,QVWHDG RI VWHS  ZH PXVW SURYLGH D YHU\ GLIIHUHQW HWFKDGKDUHVRXUFHV ILOH 7KH KDUHVRXUFHV ILOH PXVW
QRZ KDYH RQO\ WKH VLQJOH IROORZLQJ OLQH DQG 12 27+(56

YRLSVHUYHUGUEG ,3DGGUHWK

6WHS   <RX PD\ QRZ IROORZ VWHSV  RQ WKH IROORZLQJ JXLGH 5HPHPEHU 92,3'5 VKRXOG 127 KDYH KHDUWEHDW UXQQLQJ
VR \RX GR QRW QHHG WR UHSOLFDWH WKHVH VWHSV WR 92,3'5

6WHS   1RZ LV ZKHUH ZH UHDOO\ GHYLDWH :H PXVW PDNH UHVRXUFH U RQ WRS RI U %DVLFDOO\ ZH DUH SXWWLQJ D VHFRQG
OD\HU RQ RXU H[LVWLQJ '5%' ZKLFK ZH ZLOO UHSOLFDWH ZLWK 92,3'5

7\SH WKH IROORZLQJ :H ZLOO VWDUW RQ WKH SULPDU\ ZKLFK VKRXOG EH 92,36(59(5
92,36(59(5
GUEGDGP VWDFNHG FUHDWHPG U
GUEGDP VWDFNHG XS U
GUEGDGP VWDFNHG DGMXVW U

6WHS   1RZ ZH SUHSDUH 92,3'5 EXW LQ D VOLJKWO\ GLIIHUHQW ZD\
 21 92,3'5 
GUEGDGP FUHDWHPG U
VHUYLFH GUEG VWDUW
GUEGDGP DGMXVW U

6WHS  1RZ ZH PXVW KDYH 92,36(59(5 DVVXPH WKH SULPDU\ UROH IRU VWDFNHG UHVRXUFH U DQG IRUFH D UHSOLFDWLRQ
 21 92,36(59(5 
GUEGDGP VWDFNHG  RYHUZULWHGDWDRISHHU SULPDU\ U

<RX VKRXOG ZDLW XQWLO WKLV KDV ILQLVKHG EHIRUH FRQWLQXLQJ 8VH VHUYLFH GUEG VWDWXV WR FKHFN WKH UHSOLFDWLRQ SURJUHVV
3OHDVH QRWH ZH KDYH QRW WRXFKHG 92,3%$&.83 DW DOO LQ WKLV SURFHVV DQG WKDW LV RN ,W LV QRW QHFHVVDU\ $OO RI WKLV
GDWD ZLOO DXWRPDWLFDOO\ UHSOLFDWH RYHU WR 92,3$&.83 LI 92,36(59(5 IDLOV RU LW ZLOO DIWHU ZH ILQDOL]H WKH KHDUWEHDW VHWWLQJV 

6WHS   2QFH WKLV LV FRPSOHWHG ZH FDQ PRXQW WKH UHVRXUFH RQ 92,36(59(5 5XQ WKH IROORZLQJ FRPPDQGV
 21 92,36(59(5 
PNIVH[W GHYGUEG
PNGLU UHSOLFD
PRXQW GHYGUEG UHSOLFD

1RWLFH WKDW ZH DUH XVLQJ GHYGUEG LQVWHDG RI GUEG GUEG LV WKH GHYLFH WKDW UHSUHVHQWV VWDFNHG UHVRXUFH U ZKLFK LV RQ WRS
RI UHVRXUFH U DQG GHYGUEG $OO WKUHH GHYLFH ZLOO EH XVLQJ WKH VWDFNHG UHVRXUFH U GHYGUEG WR PRXQW WKH GDWD WKLV ZD\
WKH GDWD LV DFFHVVLEOH WR DOO WKUHH QRGHV ,I ZH XVHG U GUEG 92,3'5 ZRXOG QHYHU UHFHLYH DQ\ UHSOLFDWHG GDWD

6WHS  :H QRZ PXVW PRYH WKH DVWHULVNHODVWL[ FRQILJXUDWLRQ RQWR RXU PRXQWHG YROXPH 6WLOO RQ 92,36(59(5
IROORZ RQO\ 67(3  IURP WKH RULJLQDO JXLGH 'R QRW SURFHHG SDVVHG VWHS 

Ϯϲ
6WHS  1RZ ZH DUH JRLQJ WR RXW LQ WKH UHDO KHDUWEHDW FRQILJXUDWLRQ 7KH FRQILJXUDWLRQ ZH KDG EHIRUH ZDV RQO\ WHPSRUDU\
DQG LW ZDV QHHGHG WR VLPSO\ EULQJ 92,3'5 LQWR WKH UHSOLFDWLRQ 7KH IROORZLQJ LV D FRS\ RI WKH KD UHVRXUFHV ILOH \RX VKRXOG
KDYH ,W QHHGV WR PDWFK (;$&7/< 7KLV PXVW EH GRQH RQ %27+ 92,36(59(5 DQG 92,3%$&.83 'R 127 GR WKLV RQ
92,3'5
YRLSVHUYHUGUEG 3DGGUHWK GUEGXSSHUU )LOHV\VWHPGHYGUEG UHSOLFDH[W P\VTOG DVWHULVN KWWSG HODVWL[XSGDWHUG HODVWL[SRUWNQRFN
YRLSVHUYHUGUEG 0DLO7R\RXU#HPDLOKHUHFRP\RXU#HPDLOKHUHFRP '5%'+$B0$ 112'(

3OHDVH QRWH LI \RX KDYH LPSOHPHQWHG WKH IODVK RSHUDWRU SDQHO IL[ $SSHQGL[ %  SOHDVH PDNH WKH DSSURSULDWH FKDQJHV

6WHS  $W WKLV SRLQW ZH QHHG WR VWDUW SUHSDULQJ 92,3%$&.83 ZKLFK LV D OLWWOH WULFN\ :LWK D VWDFNHG UHVRXUFH LWV D OLWWOH
PRUH FRPSOLFDWHG WR XQPRXQW DQG VZLWFK SULPDULHV EXW ILUVW WKLQJV ILUVW /HWV SXW D UHSOLFD IROGHUR Q 92,3%$&.83
 21 92,3%$&.83 
PNGLU UHSOLFD

6WHS  $W WKLV SRLQW WKH HDVLHVW ZD\ WR ILQLVK XS WKH SURFHVV LV WR UHERRW DQG KDYH 92,3%$&.83 WDNH WKH SULPDU\ UROH
YLD KHDUWEHDW 5HERRW ERWK 92,36(59(5 DQG 92,3%$&.83 EXW JLYH 92,3%$&.83 D  VHFRQG KHDGVWDUW
,I 92,3%$&.83 GRHV QRW FRPH XS LQ WKH SULPDU\ UROH \RX FDQ UXQ VK XVUOLEKHDUWEHDWKEBWDNHRYHU RQ LW

6WHS  7KH UHSOLFD IROGHU VKRXOG EH SRSXODWHG QRZ RQ 92,3%$&.83 ,I VR \RX FDQ FRPSOHWH WKH WUDQVIHU E\
IROORZLQJ VWHS  RQ WKH RULJLQDO JXLGH 2QFH \RX DUH GRQH \RX FDQ JLYH 92,36(59(5 FRQWURO E\ W\SLQJ LQ
VK XVUOLEKHDUWEHDWKEBWDNHRYHU RQ WKDW VHUYHU

7+( *8,'( ,6 29(5 6723 +(5(
7+( ,16758&7,216 %(/2: $5( )25 ',6$67(5 5(&29(5< 21/<
'2 127 )2//2: 75< 72 )2//2: 7+(6( 67(36 81/(66 <28 +$9( 72

<RXPD\ EH ZRQGHULQJ ZK\ \RX
YH GRQH VR OLWWOH RQ WKH WKLUG QRGH 92,3'5 7KH VKRUW DQVZHU LV WKDW 92,3'5 GRHVQ
W GR PXFK
2QFH \RX VHW XS '5%' RQ 92,3'5 LW MXVW VLWV WKHUH DQG UHSOLFDWHV 7KHUH LV QR KHDUWEHDW QR IDLORYHU HWF ,I VRPHWKLQJ HYHU
ZHUH WR KDSSHQ WR WKH RWKHU WZR QRGHV 92,3'5 ZRXOG KDYH WR EH DFWLYDWHG PDQXDOO\

$VVXPLQJ VRPH GLVDVWHU ZKHUH WKH RWKHU WZR QRGHV FDXJKW ILUH DQG EXUQHG KHUH LV D TXLFN UXQGRZQ RI ZKDW \RX
G GR
 21 92,3'5  %87 21/< ,1 7+( (9(17 2) $ ',6$67(5 
6WHS 
PNGLU UHSOLFD
GUEGDGP SULPDU\ U
PRXQW GHYGUEG UHSOLFD

6WHS  )2//2: 67(3  21 7+( 25,*,1$/ *8,'( WR SUHSDUH WKH OLQNV DQG UHPRYH WKH ROG DVWHULVN ILOHV

6WHS  LVVXH WKH IROORZLQJ FRPPDQGV
VHUYLFH P\VTOG VWDUW
VHUYLFH DVWHULVN VWDUW
VHUYLFH KWWSG VWDUW
VHUYLFH HODVWL[XSGDWHUG VWDUW
VHUYLFH HODVWL[SRUWNQRFN VWDUW
I[RWXQH V
DPSRUWDO VWDUWBIRS

1RZ HGLW WKH SKRQH FRQILJXUDWLRQV VR WKDW WKH\ FRQQHFW WR 92,3'5 ,WV QRW QHDUO\ DV HOHJDQW EXW LW ZLOO ZRUN $OWHUQDWLYHO\
LQVWHDG RI 67(3  \RX FDQ MXVW GHOHWH WKH RULJLQDO IROGHUV WKHQ FRS\ WKHP IURP WKH UHSOLFD IROGHU EDFN WR WKHLU UHDO
ORFDWLRQV $IWHU \RX GR WKLV \RX FDQ VWRS GUEG VHUYLFH GUEG VWRS  DQG FKDQJH WKH LS DGGUHVV RI WKH VHUYHU WR WKDW RI WKH
ROG FOXVWHU   7KH SKRQHV ZLOO QRZ DXWRPDWLFDOO\ ILQG WKH VHUYHU DQG UHVXPH RSHUDWLRQV

Ϯϳ
$33(1',; +
,3 6RXUFLQJ
2QH RI WKH SUREOHPV ,
YH HQFRXQWHUHG LQ D FOXVWHUHG HQYLURQPHQW LV WUDIILF RULJLQDWLQJ IURP WKH RULJLQDO ,3 DGGUHVV RI D GHYLFH
LQVWHDG RI WKH FOXVWHU
V ,3 DGGUHVV 7KLV FDQ FDXVH SUREOHPV ZLWK VWDWLF 1$7 ILUHZDOO UXOHV DQG SRUW IRUZDUGLQJ DPRQJ RWKHU
WKLQJV  ,GHDOO\ DOO RI RXU KHDUWEHDW PRQLWRUHG VHUYLFHV ZLOO VHQG WUDIILF IURP WKH FOXVWHU ,3 DGGUHVV RI   /XFNLO\
WKHKHDUWEHDW VHUYLFH FDQ GR MXVW WKDW ZLWK WKH FRPPDQG ,3VUFDGGU  7KLV FRPPDQG VKRXOG EH DGGHG DIWHU
GHFODULQJ WKH FOXVWHU ,3 DGGUHVV LQ HWFKDGKDUHVRXUFHV 

,W ZRXOG ORRN VRPHWKLQJ OLNH WKH IROORZLQJ

YRLSVHUYHUGUEG ,3DGGUHWK ,3VUFDGGU GUEGXSSHUU 

%HORZ DUH H[DPSOHV IRU RXU ZD\ DQG ZD\ UHSOLFDFDWLRQ JXLGHV

7ZR ZD\ UHSOLFDWLRQ

YRLSVHUYHUGUEG ,3DGGUHWK  ,3VUFDGGU  GUEGGLVNU )LOHV\VWHPGHYGUEGUHSOLFDH[W P\VTOG DVWHULVN KWWSG HODVWL[XSGDWHUG HODVWL[SRUWNQRFN
YRLSVHUYHUGUEG 0DLO7R\RXU#HPDLOKHUH FRP\RXU#HPDLOKHUH FRP'5%'+$B0$,112'(

7KUHH ZD\ UHSOLFDWLRQ

YRLSVHUYHUGUEG 3DGGUHWK ,3VUFDGGU GUEGXSSHUU )LOHV\VWHPGHYGUEG UHSOLFDH[W P\VTOG DVWHULVN KWWSG HODVWL[XSGDWHUG HODVWL[SRUWNQRFN
YRLSVHUYHUGUEG 0DLO7R\RXU#HPDLOKHUHFRP\RXU#HPDLOKHUHFRP '5%'+$B0$ 112'(

,I \RX XVH WKH )ODVK RSHUDWRU SDQHO DQG XVH WKH IRSBVWDUW JXLGH LQ WKH DSSHQGL[ UHPHPEHU WR DGG WKH FRUUHVSRQGLQJ HQWU\ WR
WKH HQG RI \RXU HWFKDGKDUHVRXUFHV ILOH
APPENDIX I
IP Sourcing Part 2
The previous section ensures that external traffic will be sent from the box using the cluster IP address. What it does not
do, is use the cluster IP address on the internal LAN. This could be a problem for certain equipment on your LAN. For devices that
register with your asterisk PBX, the line "bindaddr=192.168.1.245" in sip_general_custom.conf will take care of the issue.
HOWEVER, a problem still exists with devices that your PBX registers with. For instance, VoipServer.drbd will try registering itself
to another device on the LAN using the IP address 192.168.1.242.
-
The only solution to this problem is to specify an IP source address when trying to reach individual hosts on the network.
This is not often an issue, but nevertheless is something that you may run into. To fix this, we need to implement a new service
on our linux system. These steps must be implemented on both the primary and secondary servers.
-
Step 1- Type the following command:
nano /etc/init.d/pbxiprouting
-
Step 2- Paste the code found on the following page into the editor. YOU MUST CHANGE THE IP ADDRESSES IN THE SCRIPT
There are two entries. One is under start(), the other is under stop(). I've used 192.168.1.29 as an arbitrary IP address. The IP
address that you use here should represent another system on the internal network that your asterisk PBX will INITIATE
communication with. A good example would be an Analog Gateway Device, where your server reaches out to it in order to register.
It can really be any device on the local network, aside from the servers in our drbd cluster. If you wish to do this for multiple devices,
you can copy and paste, entering multiple lines with different IP addresses. Use CTRL+ O and CTRL + X to save & exit.
-
Step 3- Enter the following command:
chmod 755 /etc/init.d/pbxiprouting
-
Step 4- Verify that the script works, with the commands:
-
service pbxiprouting start
service pbxiprouting stop
-
Step 5- If the above works normally, the last step is to add an entry within your /etc/ha.d/haresources file. Change:
(....)IPaddr::192.168.1.245/24/eth0/192.168.1.255 mysqld asterisk httpd(...)
to
(....)IPaddr::192.168.1.245/24/eth0/192.168.1.255 pbxiprouting mysqld asterisk httpd(...)
-
This change ensures that the necessary routing changes are only made when the cluster is owned by THAT host. It also ensures
that the routing changes are removed when the host releases the cluster.

Ϯϵ
Script for /etc/init.d/pbxiprouting

#!/bin/bash
# description: pbxiprouting
# process name: pbxiprouting
# Author: Nick Ross
. /etc/init.d/functions
RETVAL=0
getpid() {
pid=`ps -eo pid,comm | grep "asterisk" | awk '{ print $1 }'`
}
start() {
echo -n $"Starting PBXIPRouting: "
route add -host 192.168.1.29 dev eth0:0
RETVAL=0
if [ $RETVAL -eq 0 ]; then
touch /var/lock/subsys/pbxiprouting
echo_success
else
echo_failure
fi
echo
return $RETVAL
}
stop() {
echo -n $"Stopping PBXIPRouting: "
route delete -host 192.168.1.29
RETVAL=0
rm -f /var/lock/subsys/pbxiprouting
echo_success
return $RETVAL
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
status)
getpid
if [ -n "$pid" ]; then
echo "PBXIPRouting (pid $pid) is running..."
else
RETVAL=1
echo "PBXIPRouting is stopped"
fi
;;
restart)
stop
start
;;
*)
echo $"Usage: $0 {start|stop|status|restart}"
exit 1
;;
esac
exit $RETVAL

ϯϬ
APPENDIX J
IPSec for DRBD
If you are not using a two NIC configuration, with a secured and separate network for DRBD, its very likely that your
DRBD data is vulnerable while in transit. DRBD transmits raw disk data, without any encryption. Changes to your configuration,
passwords, etc., are all transmitted over the wire and vulnerable to interception. Luckily, this is very easy to secure in a linux
environment, via IPSec. This will have to be done on BOTH the primary and secondary server.
-
Step 1- Install the ipsec-tools package. Use the following command:
yum install ipsec-tools
-
Step 2- Make a file to start the ipsec connection. Use the command:
nano /etc/sysconfig/network-scripts/ifcfg-ipsec0
-
Step 3- Enter the following in the test editor (this assumes you are on VoipMain.drbd):
DST=192.168.1.243
TYPE=IPSEC
ONBOOT=yes
IKE_METHOD=PSK
-
(note: on voipserver, you would change the DST field to DST=192.168.1.242)
The DST field always contains the ip of the REMOTE server, NOT the ip of the server you are on.
CTRL+O saves, CTRL+X exits the editor.
-
Step 4- Make a key file. Type the command:
nano /etc/sysconfig/network-scripts/keys-ipsec0
-
Step 5- Choose a key for the ipsec connection (change it from what I put below): Type in something like this in the editor:
IKE_PSK=supersecretpassword12345!
-
CTRL+O saves, CTRL+X exits
-
Step 6- Secure the file by typing the following command:
chmod 600 /etc/sysconfig/network-scripts/keys-ipsec0
-
Step 7- Repeat this on the secondary server. Please remember to enter the proper IP address on the secondary, and do not simply
copy and paste the same IP address. See step 3 again for clarification of the DST field.
-
Step 8- To get the tunnel working without a reboot, you'll have to start it manually. On both servers, type the command:
ifup ipsec0
-
That's it, you are done. The ipsec connection should come online automatically when you reboot.
-
If you'd like to verify the ipsec connection is working, you can use tcpdump like so:
tcpdump -n host 192.168.1.242 and host 192.168.1.243
-
Tcpdump should should an AH and ESP field, indicating the header and payload are protected by ipsec. It may take up to ten seconds
before you see results.
-
If you ever want to turn of ipsec, the "ifdown ipsec0" command should be executed on both hosts. To prevent IPSec from starting
automatically upon boot, go back to step 3 and set ONBOOT=no .

ϯϭ
, KRSH \RX IRXQG WKHVH GRFXPHQWV XVHIXO

6LQFHUHO\

1LFN 5RVV RI 7KH6HUYHU([SHUWFRP DQG &7&

3OHDVH FRQWDFW PH  QLFNDW WKHVHUYHUH[SHUW GRW FRP

LI \RX KDYH DQ\ FRPPHQWV RU ILQG DQ\ PLVWDNHV ZLWK WKH JXLGH