Professional Documents
Culture Documents
1
Load Runner
TABLE OF CONTENTS
Introduction 5
• Definition for Performance testing 5
• Design 7
• Build 7
• Execute 8
• Preparation 13
• Customization 14
• Execution/Analysis 15
Introduction to Load Runner 16
2
Load Runner
• Configure Run time Settings 20
• Scheduling a Scenario 29
Analyzing Results 33
• Summary Report 34
• Transaction Summary 37
Tuning 39
3
Load Runner
Abstract
The incredible pace of change and the explosion of software complexity introduce
tremendous risk into software development process. Rigorous testing is the most
common strategy to quantify and reduce this risk to the business. The question for
developers, QA teams, and management alike is how accurately and thoroughly they
can validate the system’s performance before Go Live – without breaking the Budget.
This paper presents a complete practical approach for performance testing and
covers the best approach which includes the following:
4
Load Runner
o Analyzing Results
o Tuning the application.
Introduction
Load Testing:
To verify that system can handle the expected load, on deploying it under Real Time
conditions.
Stress Testing:
To identify the application’s Break Point, also to measures whether the application’s
environment is configured to handle the expected or potentially unexpected high
transactional volume.
Scalability testing:
Scalability testing integrates very well with performance testing. The purpose of
scalability testing is to determine whether the application automatically scales to
meet the growing user load.
5
Load Runner
Volume testing:
To verify the stability of the system with respect to handling large amounts of data
over extended period of time.
Planning/Design
Build
Execution
6
Load Runner
Analysis and Tuning
Requirements Collection
Design
This is the primary phase where team will be gathering the requirements of the
performance testing. Requirements can be Business, Technical, System and Team
requirements.
4. Transaction List: List of Key activities with in the Business process that
need to be performed in unit time.
2. How many no. of users that the system should support during normal and
peak hours.
3. How many no. of transactions per unit time should be processed by the
system
7
Load Runner
6. How the load balancing will be done during the testing
Build
This phase consists of automating the requirements collected during the design
phase. It includes
1. Scripting: Record the business processes gathered in the design phase into
Automated scripts.
Environment setup consists of assembling Hardware, Software and Data required for
executing a successful real load. This may involve working with the systems, DBA,
Operations and Business Teams.
Execute
Execution will be done in multiple phases. It consists of various types of testing like
Baseline Testing
This testing is done to ensure that both applications’ functionality & environment
setup for performance testing is proper. This will be done with 10 to 20 users.
Performance Tests
This includes running the scripts with Average and Maximum load with specified no
of transactions. We will be simulating the real time environment with different load
conditions using different Scenarios, Think Time, Pace Time and no. of users.
Benchmark Tests
These are designed to measure and compare the performance of each machine type,
environment or build of the application in ideal situation. These tests are run after
8
Load Runner
the system undergoes scalability testing to understand the performance impact of
different architectures.
During the performance testing we will be capturing all the details related to the
system like Response time and System Resources for identifying the major
bottlenecks of the system. After the bottlenecks are identified we have to tune the
system to improve the overall performance.
Performance Optimization
To make any operation successful, planning plays a major role. According to the
80-20 theories, 80% of the time should be spent in planning and 20% in real
time plan execution/implementation. In the similar procedure, software
performance test planning is very crucial as well.
Any Software Performance Test Plan should have the minimum contents as
mentioned below,
9
Load Runner
• Test environment setup requirements. (Hardware Setup,
Software Setup, Os Configurations, Third Party Utilities Setup etc.)
10
Load Runner
Test process, methodology and strategy for any Performance testing will
definitely vary from project to project based on the client’s requirement /
company. By using a methodology a project can make sure that resources are
applied to the problem effectively and the people involved in the work in a
structured way.
11
Load Runner
configurations, Agent Installations, setup in Load Clients, Third Party Monitors on
Clients / Servers etc.
After consolidating all the results, the statistical analysis should be done and the
performance test report should be created such that the development, design-
architecture, database group can get a clear idea about the application’s
performance and all the other factors affecting it. After analysis, based on the
scope tuning like OS tuning, Database Tuning, Memory Up-gradations, CPU
counts in Multi-CPU machine, Code tuning, Application Server cache tuning or
other tunings can be done. Same tests executed earlier should be repeated in the
tuned environment to make sure that there are differences in the results before
and after tuning. These tuning cycles can be repeated until the required
performance is achieved.
12
Load Runner
criterion entirely depends on the scope of testing and the environment. The failed
items may be due to any of the performance test parameters. So the failed item
details should be included in the application’s limitations list. By doing so, the
limitations list can give a clear picture about the performance limitations with
respect to load parameters which in turn will give inputs for the tuning /
enhancements for the application. Based on the test results, tuning cycles will be
repeated but the exit criteria should be pre-defined to avoid unnecessary tuning
life cycles.
This section describes the most common issues that occur during the load testing
and also suggests strategies for addressing those issues.
Preparation
Planning:
The biggest issue with planning is that it is not performed adequately. A good
process is to allow three months for a load test with completion at least a month
before “go live”. Preparation, development, execution/analysis and results
summary each should be allocated one month. Typically the results summary can
be performed in parallel with late-term project tasks.
Business processes:
High volume business processes / transactions should be built into the test.
Choosing too few might leave gaps in the test while choosing many will expand
the script creation time. We can effectively model the most common 80 percent
of the transaction throughput but trying to achieve greater accuracy is difficult
and expensive.
13
Load Runner
Test Environment Setup
Load testing requires a complete, stable and independent environment for testing.
Hardware:
To get the reliable results, the physical components of the test environment must
represent the production environment. This offers the advantage for ongoing
testing. This is particularly useful for maintenance or system expansion.
Software:
In addition to the hardware required for load test, the test bed must also have
fully installed and functioning software. Since Load Runner (Vuser) represents a
real time user (s), the system should successfully support all the user actions.
Network:
Organization’s network infrastructure shared by computers may create significant
amount of “background noise”. While it is probably impossible to accurately
model each and every network access, but it is good to examine the current
network utilization and understand the impact on network traffic.
Geography:
Often the system under test will support the global enterprise. In this
environment tests may often need to be run at remote sites across the WAN.
WAN connectivity need to be emulated in the lab or assumptions must be made.
Customization
Functional Support:
One of the most important factors in script creation productivity is the amount of
functional support provided – access to individuals who understand the
14
Load Runner
application’s functionality. When a test team member encounters a functional
error while scripting, the business Operation doesn’t function properly. The team
member typically has to stop since he or she is not equipped with the skill to
solve the issue. At that point, script creation is temporarily halted until a
functional team member helps to resolve the issue.
System Changes:
The last factor in script development is the frequency of system changes. For
each system revision, test scripts must be evaluated. Each test should be
unaffected, require simple rework, or complete reconstruction. While testing tools
are used to minimize the effect of the system change, which will reduce the
scripting time.
Execution/Analysis
System Changes:
Often there is shift between the systems used for development and execution
analysis. The first step is to execute all the business scenarios with single user.
15
Load Runner
This typically involves working with each test to account for system changes like
difference in data load, configuration, system version etc which may cause
issues. It is recommended to perform a backup of the development environment
and/or database that may be restored for execution.
Volume data:
System performance may vary widely with the volume of the system data. The
system under test should have sufficient data to model the size of the product
database at the time of deployment. It is also good to expand the database size
to model the system performance with more than expected data for one or two
years.
System Support:
Just as the development requires functional support, execution/analysis requires
system support because the test team may not understand the system
architecture and technology. The purpose of the system support is to help in
interpreting the load runner results and performance log files. While Load Runner
will describe the facts based on which the system support will describe the reason
and suggest remedies for the problems. These suggestions can be implemented
and the tests will be rerun to cross verify the new implementation.
16
Load Runner
3. Enterprise Monitoring Support: Load Runner’s real-time performance
monitors provide detailed metrics on all parts of the system under test. This
includes Web servers, application servers, databases, ERP and CRM systems,
firewalls, and load balancers. Load Runner can identify hardware limitations
and software configuration issues that might otherwise go undetected.
7. Ease of Use: Load Runner is built from the ground up for QA users. It
provides a visual scripting language, Data and autocorrelation wizards and
Active Screen technology to make scripting and running of performance tests
as easy as possible.
Scenario: A scenario defines the events that occur during each testing session.
Scenario defines and controls the number of users to emulate, the actions that they
perform and the machines on which they run their emulations.
17
Load Runner
Vusers: In the scenario, Load Runner replaces human users with virtual users or
Vusers. When you run a scenario, Vusers emulate the actions of human users
working with your application.
Vuser Scripts: The actions that a Vuser performs during the scenario are described
in a Vuser script. When you run a scenario, each Vuser executes a Vuser script.
Load generator: When you execute a scenario, the Load Runner Controller
distributes each Vuser in the scenario to a load generator. The load generator is the
machine that executes the Vuser script, enabling the Vuser to emulate the actions of
a human user.
Remote agent
Dispatcher Vuser
Agent Vuser
Controller
Vuser
Load generator
18
Load Runner
Creating Scripts using Load Runner
We have to create the scripts by recording the steps specified in the business
scenario document. Creating the script consists of the flowing steps.
Configure Runtime
Settings
We can use VuGen to develop the Vuser script by recording user performing typical
business processes on a client application. Each Vuser script that you create with
VuGen can directly communicate with a server by executing calls to the server API
without relying on client software. In addition, when a user communicates directly
with the server, system resources are not used on a user interface. This lets you run
large number of Vusers on a single workstation.
19
Load Runner
5. Record all the steps for the specified business scenario
While recording a script or after recording, you can enhance its capabilities by adding
the following functions to the script.
1. General functions
4. Parameterization.
5. Correlation.
General Vuser functions greatly enhance the script functionality of any Vuser script.
You can use general Vuser functions to measure the server performance, control
server load and debugging code or retrieve run-time information about the Vusers
participating in the scenario.
There are several library functions that you can use to enhance your script. For
example LRS functions for Win Socket applications, LRD functions for database
applications etc.
You can enhance your Vuser scripts by adding ANSI C functions to the script. ANSI C
functions allow you to add comments, Control flow statements and conditional
statements so on to Vuser scripts.
Parameterization:
20
Load Runner
When you record a business process VuGen generates a script that contains actual
values used for recording. Suppose you want to perform script actions like query,
Submit using different values from those recorded values. To do this you need to
replace the recorded values with parameters. This is called as parameterization.
Correlation:
Correlation is used to capture the dynamic value for a particular field at run time.
The run time settings will define the way that the script runs. These settings are
stored in the file default.cfg located in the Vuser directory. Runtime settings are
applied to Vusers when you run the scripts in VuGen or controller. We have to adjust
the run time settings based on the no. of transactions/hour. We will be calculating
this using the formula.
21
Load Runner
Action section Execution time + Think time + Pacing Time)
Run Logic
Number of Iterations: Based on the number of iterations set, Load Runner repeats
all the Actions for specified number of times. If you specify scenario duration in the
Controller’s Scheduling settings, the duration setting overrides the Vuser iteration
settings. This means that if the duration is set to one hour, the Vusers will continue
to run as many iterations as possible in one hour, even if the run-time settings set to
one iteration.
Pacing
The Pacing Run-Time settings let you control the time between iterations. The pace
tells the Vuser how long to wait between iterations of your actions.
22
Load Runner
You instruct the Vusers to start each iteration using one of the following
As soon as the previous iteration ends: The new iteration begins as soon as the
previous iteration ends.
After the previous iteration ends with a fixed or random delay of …: Starts
each new iteration, with a delay of specified amount of time after the end of the
previous iteration. Specify either an exact number of seconds or a range of time.
At fixed or random intervals, every … [to …] seconds: You can specify the time
between iteration either a fixed number of seconds / range of seconds. For example,
you can specify to begin a new iteration every 30 seconds, or at a random rate
ranging from 30 to 45 seconds from the beginning of the previous iteration. Each
scheduled iteration will only begin when the previous iteration is complete. The
23
Load Runner
actual amount of time that the Vuser waits between the end of one iteration and the
start of the next one appears in the Execution Log when you run the script.
Log
Enable Logging
This option enables automatic logging during replay. VuGen writes log messages that
you can view in the Execution log. This option only affects automatic logging and log
messages issued through lr_log_message. Messages sent manually, using
lr_message, lr_output_message, and lr_error_message, are still issued. The
Log run-time settings allow you to adjust the logging level depending on your
development stage.
Standard Log
Creates a standard log of functions and messages sent during script execution to use
for debugging. Disable this option for large load testing scenarios. If the logging level
24
Load Runner
is set to Standard, the logging mode is automatically set to JIT logging when adding
it to a scenario.
Extended Log
Creates an extended log, including warnings and other messages. Disable this option
for large load testing scenarios. If logging is disabled or if the level is set to
Extended, adding it to a scenario does not affect the log settings. You can specify
additional information, which should be added to the extended log using the
Extended log options:
Parameter substitution
Select this option to log all parameters assigned to the script along with their values
Select this option to log all of the data returned by the server.
Advanced trace
Select this option to log all of the functions and messages sent by the Vuser during
the session. This option is useful when you debug a Vuser script.
Think Time
25
Load Runner
Vuser think time emulates the time that a real user waits between actions. For
example, when a user receives data from a server, the user may wait several
seconds to review the data before responding. This delay is known as the think time.
VuGen uses lr_think_time functions to record think time values into your Vuser
scripts.
By default, when you run a Vuser script, the Vuser uses the think time values that
were recorded into the script during the recording session. VuGen allows you to use
the recorded think time, ignore it, or use a value related to the recorded time.
Ignore think time: Ignore the recorded think time option replay the script by
ignoring all lr_think_time functions.
Replay the think time: The second set of think time options let you use the
recorded think time.
26
Load Runner
As recorded: During replay, use the argument that appears in the
lr_think_time function. For example, lr_think_time (10) waits ten seconds.
Multiply recorded think time by: During replay, use a multiple of the
recorded think time. This can increase or decrease the think time applied
during playback
Limit think time to: Limit the think time’s maximum value to the specified
value.
27
Load Runner
Multithreading
Vusers support multithread environments. The primary advantage of a multithread
environment is the ability to run more Vusers per load generator. Only thread safe
protocols should be run as threads.
Automatic Transactions
You can instruct Load Runner to handle every step or action in a Vuser script as a
transaction. This is called using automatic transactions. Load Runner assigns the step
or action name as the name of the transaction. By default, automatic transactions
per action are enabled.
• To disable automatic transactions per action, clear the Define each action
as a transaction check box. (enabled by default)
28
Load Runner
• To enable automatic transactions per step, check the Define each step as a
transaction check box. (disabled by default)
Speed Simulation
Using the Speed Simulation settings, you can select a bandwidth that best emulates
the environment under test. The following options are available:
Proxy
29
Load Runner
Proxy runtime settings are used to provide the Proxy server details to the scripts.
No Proxy: For all the Vusers make the connection directly to the internet, without
using Proxy Server.
Obtain the Proxy Server settings from the default Browser: All Vusers use the
proxy settings of the browser on the machine they are running. This is the default
option.
User Custom Proxy: All Vusers connect to the Internet through a Proxy using the
settings you specify.
Use Proxy: Enabling this check box provides the Address of the proxy to use and
port. If the proxy requires authentication information provide this in the proxy
authentication window specified below.
30
Load Runner
Executing Scenario Using Controller
Scenario: A scenario defines the events that occur during each testing session.
Scenario defines and controls the number of users to emulate, the actions that they
perform and the machines on which they run their emulations.
Manual Scenario: Select this method if you want to build a manual scenario. You
build a manual scenario by creating groups and specifying the script, the load
generator, and the number of Vusers included in each group.
Use the Percentage Mode to distribute the Vusers among the scripts:
Select this option if you want to build a manual scenario by specifying a
number of Vusers to be distributed among the selected Vuser scripts.
Goal Oriented Scenario: Select this method to have Load Runner build a scenario
for you. In a goal-oriented scenario, you define the goals you want your test to
achieve, and Load Runner automatically builds a scenario for you, based on these
goals. This scenario is mainly used for stress testing.
Scheduling a Scenario
Ramp up: Specifies the entry criteria for the users into the server.
Load All users Simultaneously: Will load all the users at a time after the
scenario is started.
Start {No. of users} after every {Time}: will load the specified number of
users in to the server in a given time.
Duration:
Run until Completion: Will run the scripts once. If the number of iterations
specified inside the Runtime Settings, Script will be executed for specified
number of iterations for each user.
31
Load Runner
Run for {Time} after ramp up has been completed: Scripts will be run
for specified duration after all the users were ramped up. Script will be
repeated for ‘n’ number of times based on the duration specified. On selecting
this option, Run Logic runtime setting specified in the script will be
overwritten.
Ramp Down: Specifies the exit criteria for the users from the server.
Stop all Vusers Simultaneously: Exits all the users from the server at a
time.
After installing the load runner agent software on the Load Generator machine we
have to configure the Agent for connectivity with the controller machine. Follow the
below steps for configuring the Agent
1. Go to Programs -> Load Runner ->Advance Settings -> Agent Configuration. On
selecting this it will display the following window
Select the Enable Firewall Agent option and click on Settings button. It will
display all the setting related to the agent configuration. Set the following properties
for configuring the agent.
Local Machine Key: Specify the IP Address of the controller machine to establish a
unique connection between the Controller the agent machine.
32
Load Runner
Configuring Load Generators
For configuring the load Generator click on the Generator button in Controller
design window and add the load generator machine by specifying the name of the
machine. Click on Connect button to check whether the connectivity between the
controller machine and Load generator machine can be established. If the
connectivity is failed please check the following.
1. Login into the Load Generator machine ensure that Agent process is launched
and running.
2. Check the network connectivity between controller machines by running the
command Ping machine_name at command prompt of the controller/ agent
machine.
3. Verify the Load Generator Machine name / IP address in the load generator
window.
4. Verify whether the agent machine configured using Agent configuration.
5. Verify whether any firewall exists.
Monitors
system performance data from the Load Runner Controller. We can configure the
monitors in controller to capture these details.
33
Load Runner
3. Drag and drop the monitor into any of the graphs.
4. Right click on the graph and select Add Measurements
5. Click on Add(top) and enter the machine name of the Web/App/DB server
6. Now click on the Add (Down) button and select the required Performance
counters to be measured during the scenario execution.
7. Click on OK button.
If controller is not able to capture the details after configuring please ensure the
following.
1. Verify the connectivity between the Load Runner Controller machine and
Server Machine.
2. Verify whether you are Admin user or you have sufficient privileges to capture
the performance counters from the server machine.
34
Load Runner
Analyzing Results
Once the results are ready, analysis is the most important job in the whole
performance test if the requirements of performance test are not defined. The main
criteria for analysis should be to find out the bottlenecks and we should be able to
track the transactions / users / hits that create the highest level of performance
reduction. Based on the test run, Load Runner will generate various graphs and
reports. This phase contains:
Analysis
Once the load test is completed user can correlate the various graphs to analyze the
results and to identify the business scenarios causing high CPU utilization and
network traffic.
Monitoring
Tuning
Diagnostics
Performance testers have a single, unified view of how individual tiers, components
and SQL statements impact the overall performance of a business process under load
conditions. With diagnostics, performance testers using Load Runner can see all
components touched by an end – user transaction. More over, users can also see
how long each component takes and how many times it is called. With this
information corresponding team can focus on resources to improve the end-user
experience by targeting the most significant area of the web application and
database server bottlenecks.
35
Load Runner
Summary Report
The summary Report provides the information about the scenario execution. It
provides the information like Running Vusers, Throughput, Hits Per second, HTTP
Responses per Second, Transaction Summary and Average Transaction Response
time. 90 percent column indicates the maximum response time for ninety percent of
the Transactions.
36
Load Runner
Running Vusers Graph
Running Vusers graph displays the number of users that executed the scripts and
their status during the execution. This graph is useful to determine the load on the
server at any given moment.
The Hits per Second graph shows the number of HTTP requests made by Vusers to
the Web servers during the scenario execution. This graph helps you to evaluate the
amount of load Vusers generate in terms of number of hits.
37
Load Runner
38
Load Runner
Transaction Summary
39
Load Runner
Average Transaction Response Time
This graph displays the average time taken to perform transactions during each
second of the scenario run.
40
Load Runner
Tuning
For analyzing the results and for identifying the bottlenecks capture the following
information during the Scenario execution.
1. Run the SQL Server Trace file during the Scenario Run.
2. Capture the Performance counters during the run into the Load Runner
controller (if you have access to LR Monitors) otherwise use the windows
performance monitor to capture the same.
3. Run the Diagnosis tools, if you find any Hardware issues.
1. CPU Utilization
2. Avg. Disk Queue Length
3. Network Interface
4. System
5. Server
6. Process
7. Physical Disk
8. Logical Disk
9. Memory
Parameter Tips
None of the CPUs should have utilization of more than
Utilization 80 percent.
Avg. Disk Queue Length Should not be more than 2.
Load Average More than three processes should not wait for the CPU.
Any non-DB, system processes utilizing CPU intensively
CPU Intensive Processes warrant attention.
Swapping Minimal to no swapped-out processes
Paging Minimize page faults
Data Buffer Hit Ratio More than 70 percent
Disk should have less than 50-70 I/Os per second. Also,
High Disk Activity the disks should be less than 60 percent busy.
41
Load Runner
Minimal to no disk request queue. A queue of 2-4
Long Disk Queues warrants attention.
Un-indexed Tables All the tables should have at least one Primary Key.
Measurement Calculations:
The less time per transaction, the greater a system's throughput, as expressed by
the formula derived from Little's Law:
Server Thruput = # of real simultaneous users / ( Average Response Time + Think
Time )
So when no think time is used, the number of virtual users can be calculated by this
formula suggested by Menasce & Almeida:
For example, to simulate 1,000 real users using 2 second response time and 20
second think times:
1,000 x [ 2 / ( 2 + 20 ) ] = 91
System Load
o Peak number per hour or per second is typically used for system
sizing.
o Normal load levels are used to plan on-going support requirements.
o Low load levels are used to plan the percentage of resources which
can be taken off-line for repairs and upgrades.
42
Load Runner
Measures of Variation
Standard Deviation
The number in the 90% column is shown after a run is completed in the
Analysis program, not in the Controller) because it is calculated by sorting
all items by value obtained during the run, then identifying the value
associated the item at the top 10% of all items.
References
1. http://www.aptest.com/resources.html
2. http://www.sql-server-performance.com
3. http://www.mercuryinteractive.com/
4. http://www.qaforums.com
43
Load Runner