Web Workloads for

Comparing, Testing and

Tuning MySQL Performance ,
Presented by, SPECjAppServer2004,
MySQL AB® & EAStress and Faban
O’Reilly Media, Inc.

Tom Daly
Senior Performance Engineer
Sun Microsystems
Agenda

MySQL and SPECjAppServer2004

resources you probably never knew were there

MySQL and EAStress

how you can run SPECjAppServer and why you would

MySQL and Faban ,

create your own web based transaction oriented benchmarks
Aims/Objectives

To introduce you to some tools and resources

for testing and tuning MySQL with web
applications that you might find useful

To give you an idea of the sorts of analysis you

can do with these tools

We work from least work to most work

Testing MySQL using web workload
Advantages
typical and realistic
often SQL is generated / is typical of modern
web based transactions
tests DB and MySQL dependencies like
network and JDBC
Tests connector, connection pooling
stresses system with many users
Can test usage patterns available from web
statistics reports directly
Due to the above factors testing at the web tier will more
often give more representative results for DBMSs used
for web applications
Testing MySQL using web workload

Disadvantages

need to ensure that middle-tier is not clouding

results
some middle ware expertise required
require hardware for the middle tier
can be more work
SPECjAppServer2004

Benchmark =
workload + rules and basis for comparison +
process to ensure rule compliance

Benchmarks can be useful for end user

comparisons and sizing!

Benchmarks are potentially much more useful

for comparisons than workloads
SPECjAppServer2004

Is an industry standard benchmark

Models a web + mfg business
Extensive run rules
No changes to source code
No benchmark specials
All products available and supported
lots more
Peer review of all results prior to publishing
SPECjAppServer2004
SPECjAppServer2004
So it turns out ...

SPECjAppServer2004 is a reasonably useful

benchmark for comparing and improving
database (MySQL performance)
It presents an OLTP workload to MySQL
Uses a DB that may or may not be in DB cache
Representative to MySQL of loads from
applications developed in Java,Ruby on Rails,
web services PhP etc.
SPECjAppServer2004 typical configuration
Sun 720
SPECjAppServer2004
JOPS@Standard


models > 5500,
concurrent users

with 90% of results
under 2 secs (or 5 secs)

MySQL 5.0.3

approx. 800 writes per

sec to the log, approx.
4kw sec
SPECjAppServer2004
Results list is a useful database sizing and
comparison resource
Also valuable resource for tuning information

For instance if your application today requires less than

say 6000 users and 8GB Ram then perhaps MySQL on
4 (modern) cores will meet your OLTP performance
requirements
Demo – Benchmark Results Page
walk through (FDR and FDA)
SPECjAppServer2004
Run rules require that all submissions include
“complete bill of materials for the SUT”
Supplier Description Product # Qty Unit Price Price
-------- ---------------------------------------- ------------------ --- ----------- --------------
Sun Sun Fire X4100 (2x285,4x2GB,2X73GB) A64-EGB2-2H-8G-CB7 3 $8995 $26,985
Sun Solaris 10 RTU 3 $0 $0
Sun SunSpectrum Upgrade: 3YGOLD, 24x7 W9D-A64-24-3G 3 not priced not priced
Sun Sun Fire X4100 (2x285,4x2GB,2X73GB) A64-EGB2-2H-8G-CB7 1 $8995 $8995
Sun Solaris 10 RTU 1 $0 $0
Sun Single-Port PCI Ultra320 SCSI HBA SGXPCI1SCSILM320-Z 1 $340 $340
Sun SunSpectrum Upgrade: 3YGOLD, 24x7 W9D-A64-24-3G 1 not priced not priced
Sun Sun StorEdge 3320, 12x73GB, 1 RAID CONT XTA3510R01A1R876 1 $22,495 $22,495
Sun Single-Port PCI Ultra320 SCSI HBA SGXPCI1SCSILM320-Z 1 $445 $445
Sun SunSpectum Upgrade: 3Y GOLD, 24x7 W9D-SE3510-24-3G 1 not priced not priced
Sun 17" Entry Color Monitor X7147A 1
Sun PS/2 Keyboard & Mouse #320-1261 1
Sun Sun Java System Application Server $0 $0
Sun Sun Java System Application support cpu/1 yr SJSAS-PE9F-1PR 9 not priced not priced

MySQL MySQL Database 5.0 $0 $0

MySQL MySQL Network Gold Support 1 yr 3 not priced not priced
=======
$59,260
=======
 performance and cost example
JOPS Througput Acquisition Cost Comparison
900 874 200000
813
800 180000
720
700 160000

Cost in US Dollars
140000
600
120000
500
JOPS

100000
400
80000
300
60000

200 40000

100 20000

0
0
MySQL PostgreSQL Oracle
MySQL PostgreSQL Oracle
SPECjAppServer2004 JOPS@standard
SPEC, SPECjAppServer reg tm of Standard Performance Evaluation Corporation.
Sun 813.73 SPECjAppServer2004 JOPS@Standard (Sun Fire X4200, 4 chips / 8 cores, T2000 1 chip / 8 cores)
Sun 720.56 SPECjAppServer2004JOPS@Standard(Sun Fire, X4100, 4 chips/8 cores, X4100 1 chip / 4 cores )
HP 874.17 SPECjAppServer2004 JOPS@standard (rx2660 , 4 cores, 2 chips, rx2660 4 cores. 2chips)
SPECjAppServer2004

Models an enterprise class web application

not useful for older style “stored procedure
code”
not a good BIDW application model
not necessarily a good Web 2.0 model, i.e.
wouldn't be a good workload to test MySQL
use by facebook
(see EAStress or Faban for this)
SPECjAppServer

Is in active development at SPEC

SPECjAppServer2004 research mode “EAStress”
SPECjAppServer2004 research mode “EAStress2004”

More open and easy to use version of

SPECjAppServer2004 intended for R&D
can use results w/o going to SPEC for approval
Very similar workload but changed enough so
as to ensure non comparability with published
results
“EAStress2004” - mandatory notification

SPECjAppServer is a trademark of the Standard

Performance Evaluation Corp. (SPEC). The
EAStress workload results or findings in this
publication have not been reviewed or accepted
by SPEC, therefore no comparison nor
performance inference can be made against
any published SPEC result. The official web site
for SPECjAppServer2004 is located at
http://www.spec.org/osg/jAppServer2004.
Demo
EAStress2004 – reports walk
through
“EAStress” - why you might care

Good representative pre-written web application

you can use to tune, test, size and even
compare MySQL performance

Allows testing of many what-if scenarios

Examples follow
“EAStress” - experiments / results
query_cache_size
750
701
700
650 638
600
550
500
450
400 EAStress2004 HASOPM
350
300
250
200
150
100
50
0
cache=0 cache=256mb

Note difference is understated as response criteria failed

for cache=256mb Mfg = 7 secs
“EAStress” - experiments / results
Bug fix #15815
Bug #15815 fixed 750
700
701

in MySQL 5.1.24rc 650

600
631

IR=718 for both 550

500
runs but 5.1.23 450
400
failed to meet
HASOPM
350

response time 300

250

criteria 200
150
100
50
0
5.1.23rc 5.1.24rc
“EAStress” - experiments / results
directio
/data mounted 750
700 682.58
701

forcedirectio or 650

ufs
600
550
500
Note /data as 450
400
UFS results did
HASOPM
350

not meet the 300

250

response time 200

150

criteria 100
50
0
/data = UFS /data = directio
“EAStress” - experiments / results

EAStress Observations
query cache
txn log tables beware single contention point
compiler options
Big impact of Bug #15815
5.0, 5.1 performance improvement
UseServerPrepared Statements caused errors
Supported > 9000 of concurrent users with
Sun 8 core Intel server as the DB
EAStress

You can purchase EAStress

(SPECjAppServer2004) from
http://www.spec.org/order.html
(and there are academic and research prices )
Faban – building your own web workload for MySQL
Faban : create your own benchmarks

Faban is transaction oriented pert testing

tool, closely related to the
SPECjAppServer work
Can use for MySQL/Java or anything web
based
Faban

Infrastructure for developing transaction

oriented web benchmarks
Written by Akara Sucharitakul of Sun
Faban components
• Harness – benchmark process
automation, collect statistics, display
results
• Driver Framework - benchmark
development framework and
component model
• Analysis tools - graph, tabulate, and
compare results
The Faban Harness
Provides a Web UI for managing
benchmark runs
Allows users to ...
Submit/queue up benchmark runs
View progress and centralized log of run
View/compare results, statistics, graphs,
etc.
Also ...
Configures/prepares the benchmarking
rig
Automatically collects all system
configuration
Faban Harness Architecture
Master
Web Interface
(Tomcat)
Log Server

Run Daemon
Registry
Benchmark

Agent Agent (Remote)

The Faban Driver Framework
A driver developer's component
model
Container/driver distributable across
multiple systems
Developer defines driver operations
Operation: a unit of work to be measured
Operation defines data generation and
call to SUT
Utilizes Java SE 5 annotations to
describe benchmark spec, policy,
stochastic model
XML-based configuration for
Specifying the Policies
Mix
Determines the statistical model to call
the operations
Cycle
Determines the time duration between
operation calls
Mixes
Detemines the operation to select
Current mixes:
@MatrixMix(
FlatMix operations = {“Operation1”,
“Operation2”,
MatrixMix mix
“Operation3”},
= {@Row({10, 50, 40}),
@Row({30, 10, 60}),
FlatSequenceMix deviation = 2
@Row({45, 45, 10})},
)
FixedSequence
Mixes are extensible – new mix
policies can be added as needed
Well defined interface for adding
mixes
Cycles
Determines the time to fire an
operation
Can be set to think or cycle time
Current cycle policies:
@NegativeExponential(
cycleType =
CycleType.CYCLETIME,
FixedTime cycleMean
cycleMax
= 5,
= 25,
Uniform )
cycleDeviation = 1

Negative Exponential
Cycles are extensible – new cycle
policies can be added as needed
Well defined interface for adding
cycles
Metric Recording
Automatic
No timer call in code, cleaner code
More accurate timing of server response
times
Requires TimedSocket installed in
transport protocol
Manual
Timer calls demarcates critical section of
the operation
Works with all protocols, or even
benchmarks running on the local system
Faban code
Other Features
Distributed or centralized
Time syncronization
Sleep time calibration
Transport protocol handlers
@Background
Mix and cycle extensibility
Open Source
Faban is open source, period
http://faban.sunsource.net/
Licensed under CDDL
http://www.sun.com/cddl/
Same as GlassFish, etc
CVS repository is public, nothing
hidden
Repository is usually ahead of builds
You shall request changes to fit your
needs
You shall make the changes if you want it
Summary
Driver framework promotes high
quality benchmarks/workloads
Harness provides process automation
for
Running the benchmarks
Collect results
Monitor system stats across rig
Use web interface to view the status,
results, and analyze the results
Faban - demo
Q&A