Scalability and Performance Best

Practices

Presented by Laura Thomson

OmniTI
laura@omniti.com

(Based on slides by George Schlossnagle,

george@omniti.com)
1
 Scalability vs. Performance

Scalability: Ability to gracefully handle

additional traffic while maintaining
service quality.
Performance: Ability to execute a single
task quickly.

Often linked, not the same.

2
 Why are Scalability and
Performance Important?
No hope of growth otherwise.
Scalability means you can handle service
commitments of the future.
Performance means you can handle the
service commitments of today.
Both act symbiotically to mean cost-
efficient growth.

3
 Why PHP?

PHP is a completely runtime language

Compiled, statically typed languages are faster.
BUT:
Scalability is (almost) never a factor of the
language you use
Most bottlenecks are not in user code
PHPs heavy lifting is done in C
PHP is fast to learn
PHP is fast to write
PHP is easy to extend

4
 When to Start

Premature optimization is the root of all evil Donald Knuth

Without direction and goals, your code

will only get more obtuse with little
hope of actual improvement in
scalability or speed.
Design for refactoring, so that when you
need to make changes, you can.

5
 Knowing When to Stop

Optimizations get exponentially more

expensive as they are accrued.
Strike a balance between performance,
scalability and features.
Unless you ship, all the speed in the
world is meaningless.

6
 No Fast = True

Optimization takes effort.

Some are easier than others, but no
silver bullet.
Be prepared to get your hands dirty.

7
 General Best Practices

1. Profile early, profile often.

2. Dev-ops cooperation is essential.
3. Test on production data.
4. Track and trend.
5. Assumptions will burn you.

8
 Scalability Best Practices

1. Decouple.
2. Cache.
3. Federate.
4. Replicate.
5. Avoid straining hard-to-scale
resources.

9
 Performance Best Practices

1. Use a compiler cache.

2. Be mindful of using external data
sources.
3. Avoid recursive or heavy looping
code.
4. Dont try to outsmart PHP.
5. Build with caching in mind.

10
 1. Profiling

Pick a profiling tool and learn it in and out.

APD, XDebug, Zend Platform
Learn your system profiling tools
strace, dtrace, ltrace
Effective debugging profiling is about spotting
deviations from the norm.
Effective habitual profiling is about making the
norm better.
Practice, practice, practice.

11
 2. Dev-Ops Cooperation

The most critical difference in organizations that

handles crises well.
Production problems are time-critical and usually hard
to diagnose.
Build team unity before emergencies happen.
Operations staff should provide feedback on behavior
changes when code is pushed live.
Development staff must heed warnings from
operations staff.
Established code launch windows, developer escalation
procedures, and fallback plans are very helpful.

12
 3. Test on Production(-ish) Data

Code behavior (especially performance)

is often data driven.
Using data that looks like production
data will minimize surprises.
Having a QA environment that simulates
production load on all components will
highlight problems before they occur.

13
 4. Track and Trend

Understanding your historical

performance characteristics is essential
for spotting emerging problems.
Access logs (with hi-res timings)
System metrics
Application and query profiling data

14
 Access log timings

Apache 2 natively supports hi-res timings

For Apache 1.3 youll need to patch it
(timings in seconds = not very useful)

15
 5. When you assume

Systems are complex and often break in

unexpected ways.
If you knew how your system was
broken, you probably would have
designed it better in the first place.
Confirming your suspicions is almost
always cheaper than acting on them.
Time is your most precious commodity.

16
 6. Decouple

Isolate performance failures.

Put refactoring time only where needed.
Put hardware only where needed.
Impairs your ability to efficiently join two
decoupled application data sets.

17
 Example:
Static versus dynamic content
Apache + PHP is fast for dynamic content
Waste of resources to serve static
content from here: images, CSS,
JavaScript
Move static content to a separate faster
solution for static content e.g. lighttpd
on a separate box -> on a
geographically distributed CDN

18
 Example: Session data

Using the default session store limits

scale out
Decouple session data by putting it
elsewhere:
In a database
In a distributed cache
In cookies

19
 7. Cache

Caching is the core of most optimizations.

Fundamental question is: how dynamic does this bit
have to be.
Many levels of caching
Algorithmic
Data
Page/Component
Good technologies out there:
APC (local data)
Memcache (distributed data)
Squid (distributed page/component/data)
Bespoke

20
 Caching examples

Compiler cache (APC or Zend)

MySQL query cache (tune and use where
possible)
Cache generated pages or iframes (disk
or memcache)
Cache calculated data, datasets, page
fragments (memcache)
Cache static content (squid)
21
 8. Federate

Data federation is taking a single data set and

spreading it across multiple database/
application servers.
Great technique for scaling data.
Does not inherently promote data reliability.
Reduces your ability to join within the data set.
Increases overall internal connection
establishment rate.

22
 9. Replicate

Replication is making synchronized

copies of data available in more than
one place.
Useful scaling technique, very popular in
modern PHP architectures.
Mostly usable for read-only data.
High write rates can make it difficult to
keep slaves in sync.

23
 Problems

On the slave, you should see two threads

running: an I/O thread, that reads data from
the master, and an SQL thread, that updates
the replicated tables.
(You can see these with SHOW PROCESSLIST)
Since updates on the master occur in
*multiple* threads, and on the slave in a
*single* thread, the updates on the slave
take longer.
Slaves have to use a single SQL thread to make
sure queries are executed in the same order
as on the master
24
The more writes you do, the more likely
the slaves are to get behind, and the
further behind they will get.
At a certain point the only solution is to
stop the slave and re-image from the
master.
Or use a different solution: multi master,
federation, split architectures between
replication and federation, etc
25
 Other uses of replication

Remember replication has other uses

than scale out
Failover
Backups

26
 10. Avoid Straining Hard-to-Scale
Resources
Some resources are inherently hard to scale
Uncacheable data
Data with a very high read+write rate
Non-federatable data
Data in a black-box
Be aware of these limitations and be extra
careful with these resources.
Try and poke holes in the assumptions about
why the data is hard to manage.

27
 11. Compiler Cache

PHP natively reparses a script and its

includes whenever it executes it.
This is wasteful and a huge overhead.
A compiler cache sits inside the engine
and caches the parsed optrees.
The closest thing to fast = true
In PHP5 the real alternatives are APC and
Zend Platform.
28
 12. Xenodataphobia

External data (RDBMS, App Server, 3rd

Party data feeds) are the number one
cause of application bottlenecks.
Minimize and optimize your queries.
3rd Party data feeds/transfers are
unmanageable. Do what you can to
take them out of the critical path.

29
 Managing external data and
services
Cache it (beware of AUPs for APIs)

Load it dynamically (iframes/

XMLHttpRequest)

Batch writes

Ask how critical the data is to your app.

30
 Query tuning

Query tuning is like PHP tuning: what you think

is slow may not be slow.
Benchmarking is the only way to truly test this.
When tuning, change one thing at a time
Your toolkit:
EXPLAIN
Slow Query Log
mytop
Innotop
Query profilers

31
 Indexing problems

Lack of appropriate indexing

Create relevant indexes. Make sure your
queries use them. (EXPLAIN is your
friend here.)
The order of multi-column indexes is
important
Remove unused indexes to speed writes

32
 Schema design (MySQL)

Use the smallest data type possible

Use fixed width rows where possible (prefer
char over varchar: disk is cheap)
Denormalize where necessary
Take static data out of the database or use
MEMORY tables
Use the appropriate storage engine for each
table

33
 Queries

Minimizing the number of queries is always a good

start. Web pages that need to make 70-80 queries
to be rendered need a different strategy:
Cache the output
Cache part of the output
Redesign your schema so you can reduce the number of
queries
Decide if you can live without some of these queries.
Confirm that your queries are using the indexes you
think that they are
Avoid correlated subqueries where possible
Stored procedures are notably faster

34
 13. Be Lazy

Deeply recursive code is expensive in

PHP.
Heavy manual looping usually indicates
that you are doing something wrong.
Learn PHPs idioms for dealing with large
data sets or parsing/packing data.

35
 14. Dont Outsmart Yourself

Dont try to work around perceived inefficiencies in

PHP (at least not in userspace code!)
Common bad examples here include:
Writing parsers in PHP that could be done with a simple
regex.
Trying to circumvent connection management in
networking/database libraries.
Performing complex serializations that could be done with
internal extensions.
Calling out to external executables when a PHP extension
can give you the same information.
Reimplementing something that already exists in PHP

36
 15. Caching

Mentioned before, but deserves a second

slide: caching is the most important
tool in your tool box.
For frequently accessed information,
even a short cache lifespan can be
productive.
Watch your cache hit rates. A non-
effective cache is worse than no cache.

37
 Thanks!

There are longer versions of this talk at

http://omniti.com/~george/talks/
There are good books on these topics as well:
Advanced PHP Programming, G. Schlossnagle
Building Scalable Web Sites, C. Henderson
Scalable Internet Architectures, T. Schlossnagle
Compulsory plug: OmniTI is hiring for a
number of positions (PHP, Perl, C, UI design)
http://omniti.com/careers

38