October 18-21, 2005

San Francisco, CA

Hardware Layouts for LAMP Installations

John Allspaw, Flickr Plumbr
Flickr (Yahoo) jallspaw@yahoo.com October 18, 2005

Hyatt Regency San Francisco Airport Burlingame, CA

Hardware Layouts for LAMP Installations

Hardware requirements for LAMP installs have to do with:

o A decent amount about the actual hardware

(in-box stuff)

o A bit more about the hardware architecture o Which should complement the application
architecture

Oct. 18, 2005

#2

Hardware Layouts for LAMP Installations

What well talk about here:

o Database (MySQL) layouts and considerations o Some miscellaneous/esoteric stuff (lessons

learned)

o Caching content and considerations

Oct. 18, 2005

#3

Hardware Layouts for LAMP Installations

Growing Up, One Box solution

   
Basic web application (discussion board, etc.) Low traffic Apache/PHP/MySQL on one machine Bottlenecks will start showing up:
Most likely database before apache/php Disk I/O (Innodb) or locking wait states (MyISAM) Context switching between memory work (apache) and CPU work (MySQL)

Oct. 18, 2005

#4

Hardware Layouts for LAMP Installations

ONE BOX

Oct. 18, 2005

#5

Hardware Layouts for LAMP Installations

Growing Up, Two Box solution

 Higher traffic application (more demand)  Apache/PHP on box A, MySQL on box B  Same network = bad (*or is it ?), separate network =
good  Bottlenecks with start to be: Disk I/O on MySQL machine (Innodb) Locking on MyISAM tables Network I/O

Oct. 18, 2005

#6

Hardware Layouts for LAMP Installations

TWO BOX

Oct. 18, 2005

#7

Hardware Layouts for LAMP Installations

Growing Up, Many Boxes with Replication

solution
Yet even higher traffic Writes are separated from reads (master gets IN/UP/DEL, slaves get SELECTs) Diminishes network bottlenecks, disk I/O, and other in-box issues SELECTs, IN/UP/DEL can be specified within the application, OR. Load-balancing can be used

Oct. 18, 2005

#8

Hardware Layouts for LAMP Installations

MANY BOX

Oct. 18, 2005

#9

Hardware Layouts for LAMP Installations

Slave Lag
When slaves cant keep up with replication Theyre too busy:
Reading (production traffic) Writing (replication)

Manifests as:
Comments/photos/any user-entered data doesnt show up on the site right away So users will repeat the action, thinking that it didnt take the first time, makes situation worse

Oct. 18, 2005

# 10

Hardware Layouts for LAMP Installations

Insert funny photo here about slave lag*

*slave lag isnt funny

Oct. 18, 2005 # 11

Hardware Layouts for LAMP Installations

Hardware Load Balancing MySQL

Oct. 18, 2005

# 12

Hardware Layouts for LAMP Installations

How Its Usually Done
Standard MySQL master/slave replication All writes (inserts/updates/deletes) from application go to Master All reads (selects) from application go to a loadbalanced VIP (virtual IP) spreading out load across all slaves

Oct. 18, 2005

# 13

Hardware Layouts for LAMP Installations

Oct. 18, 2005

# 14

Hardware Layouts for LAMP Installations

What Is Good About Load Balancing
you can add/remove slaves without affecting application, since queries are atomic (sorta/kinda) additional monitoring point and some automatic failure handling you can treat all of your slave pool as one resource, and makes capacity planning a lot easier if you know the ceiling of each slave

Oct. 18, 2005

# 15

Hardware Layouts for LAMP Installations

How do you know the ceiling (maximum QPS capacity) of
each slave ? First make a guess based on benchmarking (or look up some bench results from Toms Hardware or anandtech.com, etc. Then get more machines than that :) Scary: in production during a lull in traffic, remove machines from the pool until you detect lag The QPS you saw right before slave lag set in: THAT is your ceiling
Oct. 18, 2005 # 16

Hardware Layouts for LAMP Installations

Oct. 18, 2005

# 17

Hardware Layouts for LAMP Installations

What Can Be Bad/Tough About Load Balancing:
not all load-balancers are created equal, not all loadbalancing companies expect this product use, so support may still be thin not that many people are doing it in high-volume situations yet, so support from community isnt large either Gotchas:
port exhaustion, health checks, and balance algorithms

Oct. 18, 2005

# 18

Hardware Layouts for LAMP Installations

Port Exhaustion
PROBLEM: LB is basically a traffic cop, nothing more Side effect of having a lot of connections: only ~64,511 ports per each IP (VIP) to use 64,511 ports/120 sec per port. ~535 max concurrent connections per IP*

* Not really, but close to it: tcp_tw_recycle and tcp_tw_reuse

Oct. 18, 2005 # 19

Hardware Layouts for LAMP Installations

Oct. 18, 2005

# 20

Hardware Layouts for LAMP Installations

Port Exhaustion (contd)
SOLUTION: Use a pool of IPs on the database slave/farm side (Netscaler calls these subnet IPs, Alteon calls them PiPs) Monitor port/connection usage, know when its time to add more

Oct. 18, 2005

# 21

Hardware Layouts for LAMP Installations Health checks

LB wont know anything about how well each MySQL slave is doing, and will pass traffic as long as port 3306 is answering Load balancers dont talk SQL, only things like plain old TCP, HTTP/S, maybe FTP

Oct. 18, 2005

# 22

Hardware Layouts for LAMP Installations Health checks (contd)

Two options: 1.Dirty, but workable:  Have each server monitor itself, and shut off/firewall its own port 3306, even if MySQL is still running

Oct. 18, 2005

# 23

Hardware Layouts for LAMP Installations Health checks (contd)

2. Cleaner, but a bit more work:
Have each server monitor itself, and run a check

via xinetd (for example, a nagios monitor)  So the LB can tickle that port, and expect back an OK string. If not, itll automatically take that server out of the pool  Good for detecting and counteracting isolated incidents of slave lag and automatically handling it

Oct. 18, 2005

# 24

Hardware Layouts for LAMP Installations

Health Checks

Oct. 18, 2005

# 25

Hardware Layouts for LAMP Installations

Balancing Algorithms
Load balancers know HTTP, FTP, basic TCP, but not SQL Two things to care about: Should the server still be in the pool ? (health checks) How should load get balanced ?  least connections or least bandwidth or least anything = BAD  Because not all SQL queries are created equal  Use round-robin or random  What happens if you dont: Evil Favoritism

Oct. 18, 2005

# 26

Hardware Layouts for LAMP Installations

Evil Favoritism

Oct. 18, 2005

# 27

Hardware Layouts for LAMP Installations

Oct. 18, 2005

# 28

Hardware Layouts for LAMP Installations

Meanwhile.for in-the-box considerations
 Interleaving memory *does* make a difference  Always RAID10 (or RAID0 if youre crazy*) but NEVER RAID5
(for Innodb, anyway)

 RAID10 has much more read capacity, and a write penalty,

but not as much as RAID5

 Always have battery backup for HW RAID write caching  Or, dont use write caching at all

Oct. 18, 2005

# 29

Hardware Layouts for LAMP Installations

IN-THE-BOX considerations (contd)
 Always have proper monitoring (nagios, etc.) for
failed/rebuilding drives  SATA or SCSI ? SCSI ! Its worth it!  10k or 15k RPM SCSI ? 15k! Its worth it! (~20% performance increase when youre disk bound)

 For 64bit Linux (AMD64 or EM64T):

Crank up the RAM for Innodbs buffer pool Swapping = very very bad either: Turn it off (slightly scary) Leave it on and set /proc/sys/vm/swapiness = 0

Oct. 18, 2005

# 30

Hardware Layouts for LAMP Installations

10k versus 15k drives ? Does it really matter that much ? Some in-the-wild proof.

Oct. 18, 2005

# 31

Hardware Layouts for LAMP Installations

Slave Lag in production

10K drives

15K drives

Oct. 18, 2005

# 32

Hardware Layouts for LAMP Installations

Using MySQL with a SAN (Storage Area Network)

Do layout storage same as if they would be local Do make sure that the HBA (fiber card) driver is well
supported by Linux Dont share volumes across databases Dont forget to correctly tune Queue Depth Size, which should be increasing, from server HBA -> switch -> storage

Oct. 18, 2005

# 33

Hardware Layouts for LAMP Installations

Caching your static content

Oct. 18, 2005

# 34

Hardware Layouts for LAMP Installations

Caching Static Content

 SQUID = good  Relieve your front-end PHP machines from looking up
data that will never (or rarely) change  Generate static pages, and cache them in squid, along with your images

Oct. 18, 2005

# 35

Hardware Layouts for LAMP Installations

Caching Static Content (contd)

 Use SQUID to accelerate plain-old origin webservers,
also known as reverse-proxy HTTP acceleration

 Described here and elsewhere:  http://www.squid-cache.org/Doc/FAQ/FAQ-20.html

Oct. 18, 2005

# 36

Hardware Layouts for LAMP Installations

Basic SQUID layout

squid accepts requests on 80 passes on cache misses to apache on 81 apache uses as its docroot an NFS mounted dir should be on local subnet, or dedicated net

Oct. 18, 2005

# 37

Hardware Layouts for LAMP Installations

Good HW layout for high-volume SQUIDing

 Do use SCSI, and many spindles for disk cache dirs  Dont use RAID  Do use network attached storage, or place the origin
servers on separate machines  Do use ext3 with noatime for disk cache dirs  Do monitor squid stats

Oct. 18, 2005

# 38

Hardware Layouts for LAMP Installations

Flickr: How We Roll

Oct. 18, 2005

# 39

Hardware Layouts for LAMP Installations

Yummy SQUID stats: >2800 images/sec, ~75-80% are cache hits ~10 million photos cached at any time 1.5 million cached in memory

Oct. 18, 2005

# 40

Hardware Layouts for LAMP Installations

The End

Oct. 18, 2005

# 41