Framework and Application Benchmarking

Paul M. Jones php|works atlanta September 12-14, 2007

“Statistical Histories for Fun and Profit”
• Benchmarking as economic predictor • Benchmarking as engineering analysis • Methodologies, results, and interpretation • Application • Single frameworks • Framework comparisons

Read These

About Paul
• Savant template system • DB_Table, Text_Wiki • PEAR Group
representative

• http://paul-m-jones.com

Full Disclosure
• Solar framework • Zend framework • Zend_Db,
Zend_View

• “Rigor and integrity” • Public project

Profiling vs Benchmarking
• Profiling applies to sub-systems
(component design)

• Benchmarking applies to entire
systems (architecture and interaction)

• Profile is to unit test, as

benchmark is to system test

Benchmarking Subjects
• CPU • RAM • Disk access • Database access • Network access • Requests/second • Programmer
productivity

• Time to initial
major feature

implementation

• Time to add new • Time to fix bugs

-- controlling for variables is key --- must be numerically measurable --

What You Tell The Boss
• New site linked by
TechCrunch

• Could we handle
the traffic spike?

• “Sure thing, boss.”

What You Tell The Team
• “...(time to load-test.)” • Ran “ab”, optimized,
handled spike

• But how many req/sec
were possible?

• Where were the limits?

Relative Limits
• There is a top limit • When is effort
warranted? imposed by each stack component

• “Fast” and “slow” are

relative (need context or requirements)

Benchmarking Tools
• Use “ab”, “http_load”, “httperf”, “flood”, etc • Emulate concurrent users hitting a URI • Reports req/sec, latency, min/max times, etc
ab -c 10 -t 60 http://localhost/...

Outer Limits

Hardware & OS

• • • •

Amazon EC2, 1.7 GHz, 1.75 GB Fedora ~2100 req/sec ~1160 req/sec

• • •

Apache 2.2 mod_php (5.2.1) & Xcache Thanks Clay “Killersoft” Loveless

Application Benchmarking
• Run “ab” against a test installation • As similar to production as possible • Hardware, OS, server, PHP installation,
database, data, application code, etc.

• Network latency (use localhost) • Server processes (turn everything off)

Controlling for Variables
• Run “ab” multiple times • Sessions (esp. file-based) will • session_id(‘constant_value’)

cause exponentially declining returns

Benchmarking as Economics
• About resource planning, not
speed (per se)

• With supply of “R” requests/ • What stack components can

second, you need “S” servers for demand of “T” traffic you modify to increase “R” so you can handle more “T” with same or fewer “S”?

Framework Benchmarking
Web Server + PHP Framework
Boot Front Page Action View

• Dynamic dispatch cycle as the limiting factor • Improvements off critical path will not
increase responsiveness

• Page cache? Action cache?

Single-System Methodology
• Use “ab” tool on a minimal application • Sane config • No action code • No view layout or helpers • Static view text • No caching • No database

Single-System Analysis
• Now you know • Work on the
critical path how much you need to improve

• Iterative

benchmarking

Benchmarking as Engineering Aid
• Every architecture
decision has a cost stats between versions worth req/sec cost? Thomson.)

• Compare benchmark • (Thanks, Laura

• Are added features are

Multiple-System Comparison
• Examine responsiveness in
context of other systems between systems

• Compare architecture costs • Use the same minimal app to
remove variability in application code

Compare Like With Like
• • • •
Frameworks to be compared should be reasonably similar in “style” or “class” PHP5, design patterns, front & page controllers, controller & view separation, plugins or extensions, no globals Cake, Solar, Symfony, Zend Code Igniter? Prado?

Multiple-System Methodology (1/2)
• “Minimal” application • Reduced or optimized config • Especially Symfony • Production mode • No debugging or logging • No database connection • No layouts or view helpers

Multiple-System Methodology (2/2)
• A user session • Xcache to reduce filesystem access • Restart web server between runs (Xcache) • 5 runs of “ab -c 10 -t 60”, averaged • Calculate req/sec as percent of PHP (thanks,
Richard “Cyberlot” Thomas)

Multiple-System Table
pct-php
baseline-html baseline-php cake-1.1.17 solar-0.28.0 symfony-1.0.6 symfony-1.0.6-dw zend-1.0.1

avg

1

2

3

4

5

1.7993 2082.8

2057 2156.7

1968 2112.2 2120.2

1.0000 1157.6 1264.4 1107.1 1131.7 1162.5 1122.1 0.0850 98.38 99.95 96.98 97.69 97.66 99.6

0.1234 142.82 0.0572 0.0613 0.0833 66.17 70.93 96.45

142.5 142.79 141.26 143.92 143.62 65.4 71.29 93.98 65.95 71.52 98.58 66.01 70.5 95.44 66.79 69.85 98.16 66.71 71.51 96.11

Multiple-System Graph
cake-1.1.17 solar-0.28.0 symfony-1.0.6 symfony-1.0.6-dw zend-1.0.1 0 0.0325 0.0650 0.0975 0.1300

What Does This Mean?
• Both more and less than you think it might • Speed is not the only consideration • Features • Intangibles • Even the slowest is still pretty fast • Each AJAX call is one request • Given comparable or equivalent features,
faster (probably) ~= better

Code Igniter, Prado
pct-php baseline-html baseline-php ci-1.5.4 prado-3.1.0 avg 1 2 3 4 5 1.9364 2330.53 2677.95 2209.94 2316.75 2250.23 2197.80 1.000 1203.53 1446.63 1112.10 1144.40 1171.62 1142.88 0.1830 0.0289 220.30 34.78 222.44 34.42 217.22 34.56 220.43 34.50 221.05 34.36 220.38 36.06

ci-1.5.4

prado-3.1.0 0 0.05 0.10 0.15 0.20

Failure Modes
• Apache/Lighty, Xcache/APC, mod_php/fastcgi • EC2 load fluctuation between/during runs • Order of testing • Different values of -c, -t • Memory and CPU usage • Added features and app code will reduce
dynamic response ... but curve is unknown.

• Differences might be too great to minimize

Links
• • • •
Google: “web framework benchmarks code”

• • •

http://code.google.com/p/web-frameworkbenchmarks/ http://paul-m-jones/blog/?p=238 http://www.laurathomson.com/2007/07/do-allframewor.html

“New Year’s Benchmarks” “Do All Frameworks Suck?”

Google: "cargo cult science"

• Questions? • Comments? • Criticism?

Avoid Frameworks?
• If framework code is only at 7-12% of max
PHP, should we not use frameworks?

• “Slow” relative to what? Must compare like
with like, and proposed systems to existing systems. off applications.

• You can’t avoid them, unless you write one-