FUNDAMENTALS OF LOAD TESTING

Introduction:

Load testing is required at every stage in the life cycle of an application, in particular web
applications. Load tests help identify performance problems during system design and
development stages. For existing applications, the simulation becomes an aid to maintain, fine-
tune and forecast future system needs.

Abstract:

In simple context, Load testing is the simulation of real world loads on computer systems. A load
test is the one that takes into consideration the various # of users that execute tasks on the
system and the various types of software on users machines, including processes, browsers,
connection speeds etc. So, before going into production, load testing is the only way to evaluate
how the system will perform in the real world. There are various ways to proceed with a Load test
and in general load testing is to identify and analyze system load requirements, define a site
map, identify features to load test, define user sets, record tests with single user, randomize
data, select and adjust work load - virtual users (user sets) and load agents, run test and analyze
test.

This write up will give you an overview or fundamentals of Load testing and will not cover any
advanced concepts of load testing. It covers Load testing details of a web application, Benefits of
load testing, Load testing environment, Load test types & variations, Basic work load models,
Quick note on how to select a work load model, Load testing goals and Results and list of well-
known Load testing tools.

A Load Test by definition:

Emulates real-life interaction between users and server systems

Creates a model of what a typical user does within the application and measures the
performance and reliability of the server
Monitors and quantifies the performance of an internet application under expected and
peak user conditions
Measures the stability, responsiveness and throughput of the computer system in a
realistic test environment
Generates different types of traffic expected, in the volumes expected, using actual
computer hardware and software
Brings the entire functionality of the e-business web site under scrutiny in order to identify
the general areas of possible bottleneck occurrences
Measures the performance of the web servers and/or the firewalls
Allows testers to draw useful conclusions from the test results for improving the site's
components
Finally, it is entirely different from functional and regression testing. Functional testing
determines whether the system operates as expected in general - a single user mode
and Regression testing determines whether an upgraded version operates as expected,
but load testing determines whether system operates as expected when used by 100,
200, or a 1000 or more simultaneous users.
Benefits of Load Testing:

Minimize the risk of deploying systems that don't meet quality and performance
requirements
Find the root cause of performance bottlenecks and system failures with ease
Optimize user experience by testing against service levels to ensure that Service Level
Agreements are met in production
Reduce the costs of defects by testing remote components early in the development
cycle
Minimize hardware and software costs by accurately predicting system capacity
Provides performance and capacity statistics
Tests scalability of the site
Allows you to evaluate the bandwidth for the amount of data sent and received
Allows you to verify predicted performance under varying conditions
Verifies the tuning of your system through benchmarking

Typical Load Testing Environment:

Load testing takes place in a client/server environment. The attributes of the client and server
systems will impact your load test design. The basic components of a load testing environment
are

1) Users are individuals who use an application

2) Client is a combination of hardware and software used by the user (browsers,
network, connections, etc)
3) Server refers to system that provide service requests to clients application
servers, database servers

The below diagram shows a load-testing environment for a typical web application and a number
of driver machines, connected together through a local and/or wide area network.

Three conditions must be met before load testing can begin:

1) the load-testing environment the Web application to be tested, the network, and the
load-driving machines needs to be adequately equipped
2) the virtual users must be set up to perform tasks in the same way as real users
3) the Web browser emulation used must be as accurate as possible
Preparation before Load Tests:

First you need to analyze the application under test and determine the factors which affect the
application performance and mark them as your Key features or attributes to measure like
Server side CPU utilization, memory usage, process time, database server/queries and Client
side similar Health monitoring data, network, user profiles and most importantly application data
and transaction points.

Apart from the above and a well-defined Load testing methodology/process has to be determined
to get satisfactory load test results i.e., identify and analyze system load requirements, define a
site map*, identify features to load test, define user Action sets* (test scenarios), identify user
groups, user profiles, form user types, record tests/script load test for single user, randomize
data, form user types* - user profiles* vs. user groups*, adjust work load select work load model
and choose # of users, run test and analyze test.

*Site map A navigational flow chart/map which identifies all of the possible paths that a user can
navigate within an application.
*Action sets defines the actions that typical users perform when using the application, these can
be scripted as functions and best used for Isolation type of load tests
*User profile users distinguished or categorized by the software or hardware configuration they
use like users with IE browser with Modem connection, users with Netscape navigator with High
speed broadband connections etc. Basically they are Browsers type vs. Browser version vs.
Network connection.
*User group These are combined with action sets to form
*User types are formed when User profiles and User groups are multiplied (permutations n
combinations of both)

You can pin-point the exact bottleneck of an Application under load test by choosing the right
combination of a User type and Action set.

NOTE: However predicting sever-side errors is quite tricky, it needs reference to wide range of
result reports to pin-point the exact bottleneck like network behavior reports, database (if any)
server behavior, behavior of each network component like proxy, firewall, switches, router etc.

So, Load testing methodology can be summarized as below:

Derive Gather statistics to get the feature list most accessed by customers
Identify load test scenarios
Generate load test plan for each test scenario
Generate load test script for test plan derived
Try script for single virtual user
Customize session variables and randomized data variables
Prepare required input Test data files
Run script with required workload configuration
Analyze load test reports generated
Basic Types of Load Testing:

To make an accurate diagnosis of a performance problem in an e-business application, it is

usually necessary to run a sequence of tests that progressively focus in on the particular problem.

1. End-to-end Testing
Is generally the first type of testing executed
Consists of a global examination of the entire Web application
Is designed to bring the entire functionality of the Web application under scrutiny in order
to identify the general areas where bottlenecks are
Can use a variety of load test types and workload types depending on the context most
often, stress tests or stability tests, deploying constant or increasing numbers of virtual
users are used

2. Module Testing
Typically follows the end-to-end testing
Separately tests specific modules of an application to accurately pinpoint any problems
Generally uses isolation tests that are designed to bombard the suspect module with
loads large enough to force repeated occurrences of a particular problem
Generate very specific and detailed information, allowing the tester to accurately
diagnose the problem

Load Test Variations:

Load tests are designed in different ways, depending on what they are required to measure. The
most frequently used load tests are stress tests, stability tests, and isolation tests.
1) Stress Test
Determines how much load a system can handle
Determines the maximum number of requests a server can manage
Generates a large and continuous load to stress the system to its limit
Minimizes the time between transactions
Ignores think times
May be set using the profile or by running a try script
2) Stability Test
Determines whether a system will remain stable over a long period of time
May last 24 hours, a week or even longer
Mimics the real traffic a system will encounter
Determines whether response time will remain stable over a long period of time
Determines whether resource consumption will remain stable over a long period
of time (i.e. memory leaks)
Helps identify other technical faults that may compromise the stability of the
system
3) Isolation Test
Helps identify bottlenecks in specific areas of the application
To isolate a troublesome functional area & find flaws by repeated load testing
Often executes specific sets of transactions over and over again
Load Testing Basic Work Load Models:

A work load model In general any load testing tool will provide the following work load models.

Increasing - An increasing workload helps testers to find the limit of a Web applications
work capacity, and is thus crucial for establishing fail-safe conditions. This workload
model is especially useful when you want to find out at which load level your system
crashes or does not respond within acceptable response times

Steady State - In this model, the same number of virtual users is employed throughout
the test, and there is no delay between transactions. This workload model is especially
useful when you want to find out about the behavior of your tested system at a specific
load level. The use of steady-state workload provides exact response time and
throughput measurements for a given number of users. This type of workload is often
employed for stress and stability tests

All Day - You can assign different numbers of virtual users to any interval of the load test.
Each user type can use a different load distribution. Therefore, you can design very
complex workload scenarios such as workday workloads or even weekly workloads. As in
the dynamic workload model, you can adjust the load level during a load test. This
workload model is especially useful when you want to model complex, long lasting
workload scenarios in the most realistic way possible

Queuing - In this model, transactions are scheduled following a prescribed arrival rate.
The load test will finish when all of the virtual users have completed their prescribed
tasks. Note that the test may take longer than the simulation time that was specified,
because of the randomized arrival rates.
 Dynamic - In this model, you can manually change the number of virtual users in the test
while it is being run. The maximum number of virtual users to be run is set; within this
limit; the number can be increased or decreased at any time during the test. No
simulation time is specified, and you must bring the test to an end manually. This
workload model is especially useful when you want to experiment with different load
levels and to have the control over the load level during a load test.

The graph cant be provided since load is manually controlled.

How to identify the workload model & Best practice:

Review your scenario question. Usually, the language in the question helps you determine which
workload model to use.

See the table below:

Language in Scenario/ Question Workload Model

Support Increasing
Stable, Memory Leaks Stead state
Respond to morning/afternoon spikes All Day
Response when 1000 users perform the same Either Stead state with the wait for
action simultaneously function or Queuing.

Best practice: When workload requirements are not given as part of your product specifications,
go blindly with a increasing workload model when you know it will be used by multiple and
simultaneous users. So, use 10 % users as initial user with 10% user increment at 10% of total
test time as the increasing period.
Load Testing Goals & Results:

The goal of any load test should be clearly understood and documented. A load test usually fits
into one of the following categories:

1. Quantification of risk - Determine, through formal testing, the likelihood that system
performance will meet the formal stated performance expectations of stakeholders, such
as response time requirements under given levels of load. This is a traditional Quality
Assurance (QA) type test. Note that load testing does not mitigate risk directly, but
through identification and quantification of risk, presents tuning opportunities and an
impetus for remediation that will mitigate risk.
2. Determination of minimum configuration - Determine, through formal testing, the
minimum configuration that will allow the system to meet the formal stated performance
expectations of stakeholders - so that extraneous hardware, software and the associated
cost of ownership can be minimized. This is a Business Technology Optimization (BTO)
type test

To speak in load testing tool terms, load testing is to

Determine each Transaction* Response time* under specific load

Determine each user Session time* under specific load
Determine Transactions/sec* within Load test simulation time
Determine Errors/sec* within Load test simulation time
Determine application overall Throughput* during Load test simulation time

Finally, analyzing all the above Load test results under various Loads and determine how much
load (maximum # of users) your Application can withstand without its performance getting
degraded. Depending on these results, you can plan how your production environment should
like and scale it up whenever needed.

*Transaction a unique business query/event matching application requirements

*Response time it is the time taken from a user initiating a request to the instant at which the
first part of the response is received at by the application.
*Session time it is the time taken by the application for each user to execute all transactions
specified under load test scenario
*Transactions/sec generally rate of no. of successful transactions per sec. Might differ for each
load test tool, for few it is categorized as TransacationOK/sec which means successful
transactions and Transactions/sec might mean successful and failed transaction rate.
*Errors/sec No. of errors load test generated during simulation time. It need not be only
transaction failures but can be actual errors application has generated (bugs), script errors
mismatched with actual result, network issues etc.
*Throughput it is the overall transaction time taken from a user initiating a request to the instant
at which the last part of the response is received at by the application
Major Load Testing Tools (in the 93% of Market Share)

HPs (earlier MERCURY INTERACTIVE) - LoadRunner [54%]

IBM RATIONALs Performance Tester [11%]
COMPUWAREs QA load [9%]
BORLANDs (earlier SEGUE) - Silk Performer - 7%
EMPIRIXs e-TEST suite [6%]
RADVIEWs WebLoad [3%]

Conclusion:

Load testing is the only way and plays an important role in analyzing the system performance
for a specific load (no. of simultaneous users). Before an application is planned to go public -
typically a client/server application where the no. of users accessing it are larger in number,
unpredictable and random, there is a desperate need to perform load testing and define the
maximum load/simultaneous users it can sustain without its performance getting degraded.
Once this performance measured is defined as a requirement, you can definitely plan to scale
up your production environment to meet whatever additional load of users that keep
bombarding your application.

References:

Fundamentals of Load testing Search links from Google

SilkPerformer Modeling & Implementing Load tests student guide (Segue)