Special Topics

Open Source Tools for Web Application Performance Testing

Dan Downing, VP Testing Services

MENTORA GROUP
Atlanta Boston DC www.mentora.com

Objectives
Present the basics of using OpenSTA for load testing web apps. Things you will learn:
Capabilities of OpenSTA Understand how to interpret its performance data Detect and repair performance bottlenecks

2
Copyright 2006 Mentora Group Inc.

Terms & Concepts

History
Develop by Cyrano, a French software company Released into public domain in 2001

Collector: System resource monitor script Commander: Main UI and test launcher Modeling: Script development SCL: Script Control Language Task Group: Load scenario script or Collector script Test: Set of scripts comprising load scenario and related results Timers: Start-Stop transaction timing markers for actions to be timed Variables: Script Parameters

3
Copyright 2006 Mentora Group Inc.

Capabilities
Applications it will test
Protocols: Standard http/https only (not Oracle 11i, SAP, RMI, web services) Development technologies: D/HTML, javascript, XML payloads, jvm servers (not java applets, latest technologies) Script development: 7/10 (Handles sessionid cookies; UI intuitivess ok; SCL language support good) Test/Debug: 6/10 (Need error-handling and content checks to ensure script functioning properly) Test execution: 7/10 (slow refresh on summary stats Results analysis: 3/10 (built-in graphs limited; need to do much in Excel) Excellent thread model; spawns a new thread only when it needs it to drive more load Sometimes hangs, crashes (must manually clean up processes and sometimes reboot) Conflicts with other load testing tools listener processes (run only one at a time) Launches max of 1664 users per Commander session Watch out for
Trip-ups with mix of http and https pages Too many parameter files can create file open bottleneck, throttling load Code your own error handling of http 400 and 500 errors

Functionality (LoadRunner = 10)

Robustness

Limitations

That said, it works great on many web apps!

Copyright 2006 Mentora Group Inc.

Download & Install

V.1.4.3 12/04 is current release 5.7 MB download Windows installer Runs on XP, W2K, NT SP5 Installer (usually) runs flawlessly

5
Copyright 2006 Mentora Group Inc.

Components & Process

OpenSTA
1

Modeler

Step 1: Develop test scripts

Record, parameterize user-entered data, parameterize dynamic system data, adjust think-times & timers, test/debug

Commander

Step 2: Define & Launch Test

Define load scenario; configure Collectors; launch test; monitor interactively

Name Server*

3 Excel

Step 3: Export, Graph, & Analyze Results

Select & filter results; export; generate graphs; analyze & interpret.
6

* Listener process that communicates between Commander, load drivers, & target system
Copyright 2006 Mentora Group Inc.

Architecture
Remote load driver(s)
SD

Sun

E NT E RP R I SE

40 00

ULTRASPARC
DR IVEN

System Under Test

Local load driver(s) OpenSTA Web Relay Daemon (opt.) Load Driving System Name Server
Copyright 2006 Mentora Group Inc.

Directory Structure
Base directory; location pointed to by Tools/Repository Path
Main applications DLLs More DLLs, exes, ini files More DLLs Test assets (by project)
<EMPTY>

Screen shots <.ALL> Data files <.FVR> Compiled scripts <.tof, .scd> Collector scripts <.NTP> Scripts (source SCL code) <.HTP> Tests Load scenarios <Full_Load_1>
Test results folders <09-06-2006-10.000>

8
Copyright 2006 Mentora Group Inc.

Architecture: Processes
ArchMgrDmn.exe Controls remote load drivers CyrDmn.exe Listener daemon (started by Name Server) DaemonCFG.exe -- Name Server configuration utility OSComnander.exe Main Commander UI TestManager.exe Overall test execution manager TestPlugin,exe Test configuration plug-in TExecuter_htp.exe Runtime engine that runs task groups TExecturer_ntp.exe Runtime engine that runs Collectors TModeller_Web.exe Script Modeler UI

9
Copyright 2006 Mentora Group Inc.

Script Modeler
Menu bar Toolbar

Script editor pane

Html header, body panes

Replay output pane

10
Copyright 2006 Mentora Group Inc.

Modeling Scripts: Cookies

Defined automatically during Record

. . .

Handled automatically during replay

11
Copyright 2006 Mentora Group Inc.

Modeling Scripts:
Parameterizing User-Entered Data
. . .

Define File variable with proper scope Substitute user-entered static data with a variable; do global edit

. . .

Process file values

MUTEX NEXT SET

Display value to ensure its working

LOG
12
Copyright 2006 Mentora Group Inc.

Modeling Scripts:
Capturing System Response Values

. . .

Load Response Header/Body into a buffer (character array) Parse desired value and load into a variable Do global replace so it gets fed back on later urls

13
Copyright 2006 Mentora Group Inc.

Commander
Menu bar Test start/stop Toolbar Test Configuration tab

Repository Tree View pane

Runtime options panel

14
Copyright 2006 Mentora Group Inc.

Commander: Monitoring a Test

Test start/stop Toolbar Monitoring tab

Execution error window

User ramp graph

Execution summary pane

15
Copyright 2006 Mentora Group Inc.

Commander: Exporting Results

Test start/stop Toolbar Results tab

Audit log

http response list Timers list

16
Copyright 2006 Mentora Group Inc.

Exporting Bandwidth
Bandwidth graph

17
Copyright 2006 Mentora Group Inc.

Graphing & Interpreting:

Scalability

All workflows scale well though 60 users, degrading exponentially above that level
18
Copyright 2006 Mentora Group Inc.

Graphing & Interpreting:

Page Times

Sharp degradation of "Display All Studies" indicates this page is the major contributor to RSD process degradation

19
Copyright 2006 Mentora Group Inc.

Graphing & Interpreting:

Bandwidth Utilization
600 500 Load (concurrent users) 400 300 200 100 0

Bandwidth ramps steadily and roughly plateaus characteristically at maximum load

Bandwidth Utilization

Average bandwidth utilization is 6.8 Megabits per second

12 10 8 6 4 2 0 Mbps

Bandwidth lines shape indicates no network throughput bottleneck Average bandwidth utilization is ~7 Mbps

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Test Session Time

20
Copyright 2006 Mentora Group Inc.

Graphing & Interpreting:

Application Errors

Stable range
Load (concurrent users)

HTTP Error Responses

600 500 400 300 200 100 0
0 20 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28

Application unstable above 120 concurrent users

250 200 Errors 150 100 50

30

32

34

36

38

40

42

44

http server errors begin above 120 concurrent users Error spikes indicate unstable application well before target load

Concurrent Users HTTP 5xx Errors

60 120 175 235 295 350 410 453 477 497 495 495 492 488 487 486 485 485 484 479 479 478 1 1 80 94 178 33 134 1 88 46 230 1 1 211 71 158 92 146 88 1 99 19

Test Session Time (min)

21
Copyright 2006 Mentora Group Inc.

Advanced Topics
Development standards
Naming conventions for Scripts, Timers, Tests Results directory structure

Graphing results
Pivot tables Creating & customizing 2 axis charts

Advanced script modeling

Response parsing functions for capturing system dynamic values Parameter file management with ACQUIRE MUTEX Randomizing think times with GENERATE Error handling with ON ERROR Handling state ids in redirects with EXTRACT Debug statements

OpenSTA init settings

Timeout Max Ports
22
Copyright 2006 Mentora Group Inc.

Resources: On-Line Help

Make it your friend! Can get all the basics and more

23
Copyright 2006 Mentora Group Inc.

Resources: FAQs

Many beyond basic how-tos in these FAQs Many forum questions from new users direct you back to the FAQs

24
Copyright 2006 Mentora Group Inc.

Resources: OpenSTA Forum

Very active forum; email-based Quick responses to questions Anyone can sign up, read, submit posts Many OpenSTA experts respond

25
Copyright 2006 Mentora Group Inc.

Resources: Other
OpenSTA Tutorial On-line Course http://mentora.com/training.htm Load testing presentations
http://www.mentora.com/library.htm

Load testing blogs

www.testingreflections.com Antony Marcano www.exploratorytesting.com James Bach

26
Copyright 2006 Mentora Group Inc.

Questions?

ddowning@mentora.com www.mentora.com

27
Copyright 2006 Mentora Group Inc.