Presented by, MySQL AB® & O’Reilly Media, Inc.

Web Workloads for Comparing, Testing and Tuning MySQL Performance , SPECjAppServer2004, EAStress and Faban
Tom Daly Senior Performance Engineer Sun Microsystems


MySQL and SPECjAppServer2004 MySQL and EAStress MySQL and Faban ,

resources you probably never knew were there

how you can run SPECjAppServer and why you would

create your own web based transaction oriented benchmarks

To introduce you to some tools and resources for testing and tuning MySQL with web applications that you might find useful To give you an idea of the sorts of analysis you can do with these tools We work from least work to most work

Testing MySQL using web workload
Advantages typical and realistic often SQL is generated / is typical of modern web based transactions tests DB and MySQL dependencies like network and JDBC Tests connector, connection pooling stresses system with many users Can test usage patterns available from web statistics reports directly
Due to the above factors testing at the web tier will more often give more representative results for DBMSs used for web applications

Testing MySQL using web workload
Disadvantages need to ensure that middle-tier is not clouding results some middle ware expertise required require hardware for the middle tier can be more work

Benchmark = workload + rules and basis for comparison + process to ensure rule compliance Benchmarks can be useful for end user comparisons and sizing! Benchmarks are potentially much more useful for comparisons than workloads

Is an industry standard benchmark Models a web + mfg business Extensive run rules No changes to source code No benchmark specials All products available and supported lots more Peer review of all results prior to publishing


So it turns out ... SPECjAppServer2004 is a reasonably useful benchmark for comparing and improving database (MySQL performance) It presents an OLTP workload to MySQL Uses a DB that may or may not be in DB cache Representative to MySQL of loads from applications developed in Java,Ruby on Rails, web services PhP etc.

SPECjAppServer2004 typical configuration
Sun 720 SPECjAppServer2004 JOPS@Standard models > 5500, concurrent users  with 90% of results under 2 secs (or 5 secs)
 

MySQL 5.0.3

approx. 800 writes per sec to the log, approx. 4kw sec

Results list is a useful database sizing and comparison resource Also valuable resource for tuning information

For instance if your application today requires less than say 6000 users and 8GB Ram then perhaps MySQL on 4 (modern) cores will meet your OLTP performance requirements

Demo – Benchmark Results Page walk through (FDR and FDA)

Run rules require that all submissions include “complete bill of materials for the SUT”
Supplier Description Product # -------- --------------------------------------------------------Sun Sun Fire X4100 (2x285,4x2GB,2X73GB) A64-EGB2-2H-8G-CB7 Sun Solaris 10 RTU Sun SunSpectrum Upgrade: 3YGOLD, 24x7 W9D-A64-24-3G Sun Sun Fire X4100 (2x285,4x2GB,2X73GB) A64-EGB2-2H-8G-CB7 Sun Solaris 10 RTU Sun Single-Port PCI Ultra320 SCSI HBA SGXPCI1SCSILM320-Z Sun SunSpectrum Upgrade: 3YGOLD, 24x7 W9D-A64-24-3G Sun Sun StorEdge 3320, 12x73GB, 1 RAID CONT XTA3510R01A1R876 Sun Single-Port PCI Ultra320 SCSI HBA SGXPCI1SCSILM320-Z Sun SunSpectum Upgrade: 3Y GOLD, 24x7 W9D-SE3510-24-3G Sun 17" Entry Color Monitor X7147A Sun PS/2 Keyboard & Mouse #320-1261 Sun Sun Java System Application Server Sun Sun Java System Application support cpu/1 yr SJSAS-PE9F-1PR MySQL MySQL MySQL Database 5.0 MySQL Network Gold Support 1 yr Qty Unit Price --- ----------3 $8995 3 $0 3 not priced 1 $8995 1 $0 1 $340 1 not priced 1 $22,495 1 $445 1 not priced 1 1 $0 9 not priced 3 $0 not priced Price -------------$26,985 $0 not priced $8995 $0 $340 not priced $22,495 $445 not priced $0 not priced $0 not priced ======= $59,260 =======

performance and cost example
JOPS Througput
900 800 700 600
720 813 874

Acquisition Cost Comparison
200000 180000 160000

Cost in US Dollars

140000 120000 100000 80000 60000 40000 20000 0


500 400 300 200 100 0 MySQL PostgreSQL Oracle




SPECjAppServer2004 JOPS@standard SPEC, SPECjAppServer reg tm of Standard Performance Evaluation Corporation. Sun 813.73 SPECjAppServer2004 JOPS@Standard (Sun Fire X4200, 4 chips / 8 cores, T2000 1 chip / 8 cores) Sun 720.56 SPECjAppServer2004JOPS@Standard(Sun Fire, X4100, 4 chips/8 cores, X4100 1 chip / 4 cores ) HP 874.17 SPECjAppServer2004 JOPS@standard (rx2660 , 4 cores, 2 chips, rx2660 4 cores. 2chips)

Models an enterprise class web application not useful for older style “stored procedure code” not a good BIDW application model not necessarily a good Web 2.0 model, i.e. wouldn't be a good workload to test MySQL use by facebook (see EAStress or Faban for this)

Is in active development at SPEC

SPECjAppServer2004 research mode “EAStress”

SPECjAppServer2004 research mode “EAStress2004” More open and easy to use version of SPECjAppServer2004 intended for R&D can use results w/o going to SPEC for approval Very similar workload but changed enough so as to ensure non comparability with published results

“EAStress2004” - mandatory notification SPECjAppServer is a trademark of the Standard Performance Evaluation Corp. (SPEC). The EAStress workload results or findings in this publication have not been reviewed or accepted by SPEC, therefore no comparison nor performance inference can be made against any published SPEC result. The official web site for SPECjAppServer2004 is located at

Demo EAStress2004 – reports walk through

“EAStress” - why you might care

Good representative pre-written web application you can use to tune, test, size and even compare MySQL performance Allows testing of many what-if scenarios Examples follow

“EAStress” - experiments / results
750 700 650 600 550 500 450 400 350 300 250 200 150 100 50 0 cache=0 cache=256mb
EAStress2004 HASOPM 701 638

Note difference is understated as response criteria failed for cache=256mb Mfg = 7 secs

“EAStress” - experiments / results Bug #15815 fixed in MySQL 5.1.24rc IR=718 for both runs but 5.1.23 failed to meet response time criteria
Bug fix #15815
750 700 650 600 550 500 450 400 350 300 250 200 150 100 50 0 5.1.23rc 5.1.24rc
HASOPM 631 701

“EAStress” - experiments / results /data mounted forcedirectio or ufs Note /data as UFS results did not meet the response time criteria
750 700 650 600 550 500 450 400 350 300 250 200 150 100 50 0 /data = UFS /data = directio
HASOPM 682.58 701

“EAStress” - experiments / results EAStress Observations query cache txn log tables beware single contention point compiler options Big impact of Bug #15815 5.0, 5.1 performance improvement UseServerPrepared Statements caused errors Supported > 9000 of concurrent users with Sun 8 core Intel server as the DB


You can purchase EAStress (SPECjAppServer2004) from (and there are academic and research prices )

Faban – building your own web workload for MySQL

Faban : create your own benchmarks

Faban is transaction oriented pert testing tool, closely related to the SPECjAppServer work Can use for MySQL/Java or anything web based

Faban Infrastructure for developing transaction oriented web benchmarks Written by Akara Sucharitakul of Sun

Faban components
• Harness – benchmark process automation, collect statistics, display results • Driver Framework - benchmark development framework and component model • Analysis tools - graph, tabulate, and compare results

The Faban Harness
Provides a Web UI for managing benchmark runs Allows users to ...
Submit/queue up benchmark runs View progress and centralized log of run View/compare results, statistics, graphs, etc.

Also ...
Configures/prepares the benchmarking rig Automatically collects all system configuration

Faban Harness Architecture
Master Web Interface (Tomcat) Log Server Run Daemon Benchmark Registry


Agent (Remote)

The Faban Driver Framework
A driver developer's component model
Container/driver distributable across multiple systems Developer defines driver operations Operation: a unit of work to be measured Operation defines data generation and call to SUT

Utilizes Java SE 5 annotations to describe benchmark spec, policy, stochastic model XML-based configuration for

Specifying the Policies
Determines the statistical model to call the operations

Determines the time duration between operation calls

Detemines the operation to select Current mixes:
FlatMix MatrixMix FlatSequenceMix FixedSequence

@MatrixMix( operations = {“Operation1”, “Operation2”, “Operation3”}, mix = {@Row({10, 50, 40}), @Row({30, 10, 60}), @Row({45, 45, 10})}, deviation = 2 )

Mixes are extensible – new mix policies can be added as needed Well defined interface for adding mixes

Determines the time to fire an operation Can be set to think or cycle time @NegativeExponential( Current cycle policies: cycleType =
FixedTime Uniform Negative Exponential

CycleType.CYCLETIME, cycleMean = 5, cycleMax = 25, cycleDeviation = 1 )

Cycles are extensible – new cycle policies can be added as needed Well defined interface for adding cycles

Metric Recording
No timer call in code, cleaner code More accurate timing of server response times Requires TimedSocket installed in transport protocol

Timer calls demarcates critical section of the operation Works with all protocols, or even benchmarks running on the local system

Faban code

Other Features
Distributed or centralized Time syncronization Sleep time calibration Transport protocol handlers @Background Mix and cycle extensibility

Open Source
Faban is open source, period Licensed under CDDL Same as GlassFish, etc

CVS repository is public, nothing hidden
Repository is usually ahead of builds You shall request changes to fit your needs You shall make the changes if you want it

Driver framework promotes high quality benchmarks/workloads Harness provides process automation for
Running the benchmarks Collect results Monitor system stats across rig

Use web interface to view the status, results, and analyze the results

Faban - demo


Sign up to vote on this title
UsefulNot useful