Introduction to Performance Testing IVS training

Improved Business experience = High Throughput * Speedy Response

Agenda
• Performance Testing Basics
– What & Why
– Differences
– Types of Tests
– Performance Parameters
• Performance Testing Methodology
– P-Test Methodology
– Test Environment – factors to be considered
• Factors taken into consideration for Estimation
• Execution Models
• Performance Testing Tools

Confidential property of Infosys. For internal circulation only 2

Impact / Importance of Performance focus
Source - Butler Group
Poorly performing IT applications cost the world's For 42% of organizations, the cost of downtime is above $100,
industrialized nations GBP45 billion annually 1
000 per hour
A study by Contingency Planning Association Research found that the average hourly cost of downtime ranges from $28,000 in
the shipping and transportation industry to a staggering $6.4 million for brokerage houses
Source - Forrester Group
Slow in-store apps earn a retail CIO the ire of other executives. Performance problems in live applications disrupt
the business, damage productivity, and result in lost revenues, ultimately damaging the credibility of the entire IT
organization. When complaints about the performance of a $50 billion retailer’s in-store POS apps flooded
headquarters, management laid them in the CIO’s lap. The pressure to resolve the problem and speed up store
operations was ferocious, and the CIO ultimately hired a systems integrator to implement a cradle-to-grave performance
management program.
A health insurance company wastes millions of dollars every year. A $20 billion US health insurance company
calculated that its inattention to performance prior to deployment — notably in design, where more than half of the
problems originated — was resulting in several millions of dollars in avoidable costs every year. Resolving performance
problems before deployment is more cost-efficient by at least one order of magnitude, and ignoring performance in the
short term only racks up long-term costs that IT organizations can ill afford.
A telecom provider spends millions on app support instead of new app development. At one telecom company,
each 15-second timeout in enterprise application integration (EAI) infrastructure resulted in a $4 call to an outsourced
contact center; over the course of six months, this resulted in unanticipated support costs of almost $3 million — funds
that would otherwise have been used for new development efforts. Enterprise IT organizations struggle to drive down
maintenance costs and fund new projects; Forrester’s data indicates that the average IT organization spends 75% of its
software budget on ongoing operations and maintenance.2 IT shops stuck in firefighting mode dedicate a larger portion
of their budgets to maintenance than is necessary, diverting resources from efforts to deliver new value to the business.
Source - Infosys
In one of the corporate banking application, the bank lost more than $ 10 million in the last 4 years on issues related to
Performance

Confidential property of Infosys. For internal circulation only 3

Key Customer Issues
Stressed Do we know How can you
Production When and Batch Jobs
pinpoint and fix
Support Team how taking longer
problems in
and Application Production time to finish?
production before
Development break and its they impact the
Team impact? End – User?
How do the different
components of the
Hardware system affect each
Upgrade does other?
not help
achieving better
Performance!!
Optimized
server
above all...
utilization?
Are users
SLA of clients complaining
are violated about
due to system
Response? Why I am
issues
spending too
much on
maintenance
What are my than on
competitor’s System / Performance
applications not able enhancements
benchmarks? degradation
to support additional ?
happening over
load despite lower a period of time
H/w utilisation but how?

Confidential property of Infosys. For internal circulation only 4

Single user request

“Give me red” Red

Desired Outcome

Confidential property of Infosys. For internal circulation only 5

Concurrent users requests
“Give me red” Blue

“Give me red” Green

Memory?

“Give me red” Yellow

* Network? DISK? *
* *
* *
“Give me red” Orange
CPU?

“Give me red” Pink

Undesirable
outcome from a functionally
tested application

Confidential property of Infosys. For internal circulation only 6

What Performance testing can eliminate
 Would you want to use a cricket web-site
that takes 5 minutes to update the score and tell
you that Sachin Tendulkar was out 10 minutes ago?
 Would u want to use an Airlines web-site to book a ticket and wait for 15 minutes to know that
there are no seats available?
 Payment gateway unavailable after
providing all the details to get tickets at
PVR web-site

Confidential property of Infosys. For internal circulation only 7

What Performance testing can eliminate contd…
 Would u want to use a university site that crashes when your engineering results are out?

 How many of us like to click on a button and see the result – “Error : Page not found”?
None of us would want to face these situations
Neither do the owners of these sites want their
users to face such situations
This is where Performance testing comes into action

Confidential property of Infosys. For internal circulation only 8

Typical scenario..
• Popular Online Music Store (POMS) - Good music download service for the users ; Good Business
for the proprietor
• A new song (say Rap song) uploaded on POMS website - The Song which was not so famous
initially, becomes a huge hit after the video was aired on TV Channels ; Never-before-seen-
popularity
• Just a week after the upload, the Rap song is downloaded by several thousand users, innumerable
times. Essentially, the User Base has shot up about 25 times
• New to such a scenario, the POMS website struggles to handle hundreds of Concurrent downloads
• USP of POMS: Average download time of 5s / 1MB  Gets a beating. Terrible experience for the
users who face a high Response Time by POMS to download the now famous Song  Dissatisfied
Customer
• POMS which boasted a download rate of 800MB/hr  Gets a beating. Throughput dropped down to
< 60MB/hr  Dissatisfied Proprietor

Confidential property of Infosys. For internal circulation only 9

What is Performance Testing?
 Definition : Testing conducted to evaluate the compliance of a system or component with specified
performance requirements. [IEEE]
 Performance Testing is a process by which software is tested and tuned with the intent of realizing
the required Quality of Service requirements
 This process aims to optimize the application performance for key performance indicators like:
Response time
Throughput
System threshold limits

 Performance Tests determine the runtime “behavior” of the application and its supporting
infrastructure, under operational conditions 

Confidential property of Infosys. For internal circulation only 10

Performance Testing – what?

 Performance testing is the discipline concerned

with determining and reporting the current
performance of a software application when
subjected to virtual user load
 Performance testing is a critical component of
an application’s overall success. It prevents
failures by predicting the application behavior Speed
and capacity prior to going live

Measuring Application Performance in terms of

 Speed -> time taken by the application to Scalability Stability

respond to user request

 Scalability -> ability to handle increasing user

load without performance degradation

 Stability -> ability of the application to sustain

prolonged usage without performance
degradation

Confidential property of Infosys. For internal circulation only 11

Performance Testing – why?

Objectives of Performance Testing

 Determine Application Performance at various user loads

 Identify Performance Bottlenecks

 Performance Benchmark

Confidential property of Infosys. For internal circulation only 12

Key Facts - One

Detect Performance Defects

 Software Applications when subjected to User Load expose Performance Defects

 Defects could originate from any part of the application – Application Program, Database
Systems, Application Server, Web Server, Network, Operating System

 Causes for Performance Defects could be one of :

• Poor Programming Practices – Programming Logic, Database Queries
• Inappropriate System Designs – Table/Index Design, Application Design
• Incorrect System Configurations – App/Webserver Configuration, OS parameter

Myth: Performance Bottlenecks occur only at high User Load

Confidential property of Infosys. For internal circulation only 13

Key Facts - Two

Performance Testing is all about

 Application User perspective : Response Time of Business Transactions

 Application Owner perspective : Transaction Throughput

Myth: Performance Issues could be solved by increasing Hardware Resources

Confidential property of Infosys. For internal circulation only 14

Key Facts - Three

Production Like Test Environment

 To simulate real time issues its recommended to conduct performance tests in close-to-real
time test environments

 If not the same, Test Environments should be proportionately scaled down from Production
Environments

Myth: Number of application users is synonymous to Concurrent users

Confidential property of Infosys. For internal circulation only 15

Different from Performance Engineering

Performance Testing is a subset of Performance Engineering

Performance Testing Performance Engineering

Evaluates the application Eliminates issues in the application

Assesses the system Addresses the issues in the system

Finds problematic areas in the SUT Fixes problematic areas in the SUT

Confidential property of Infosys. For internal circulation only 16

Different from Functional Testing

Performance Testing is starkly different from Functional Testing

Performance Testing Functional Testing

Evaluates the system Performance Evaluates the application Functionality

Assesses only Performance Critical Assesses all possible Functional Scenarios

Scenarios

Conducted at Multiple Users Load Conducted at Single User Load

Requirements are provided in numbers Requirements are hard-line specifications

Analytical and mathematical in nature, based Either pass or fail is the test case’s execution
on various theories. result
Classified as partly art and science Quite evolved and best practices baseline for
various methodologies

Confidential property of Infosys. For internal circulation only 17

Types of Performance Tests
Load Test –> Exert constant user load for a relatively lesser duration ; Behaviour @ various loads

Stress Test –> Start with a low user load. Increment the user load with a fixed number of users at a
regular interval ; Break Point

Endurance Test –> Exert constant user load for a prolonged duration ; Memory Leaks

Volume Test –> Exert constant user load for multiple iterations with different database volumes each
time ; Behaviour @ various DB volumes

Scalability Test –> Start with a low user load. Increment the user load with a fixed number of users at a
regular interval ; Max TPS

Confidential property of Infosys. For internal circulation only 18

Performance Test Methodology

Confidential property of Infosys. For internal circulation only 19

P-Test Methodology
Requirement Analysis(NFR) Planning & Design Execution
Recommendations Build Knowledge Repository

Breaking down Test Architecture Validation of

the scope to Environment test environment
Project Design setup done
critical
Kickoff Mining Iteration
Transactions
the tuning
historical data
Test Scenarios
Identification/ Execution of
Interviewing Workload Test scripts
Design
the Modeling & Recommendation
Key Stake Characterization
Business/ and Go No Go
holders
Architecture/ Development decision making
Identification Collection of
Environment of test
of the test results
Assessment Stubs/Scripts
Key Performance Performance
Post deployment
Indicators Test Strategizing/
review
Tool identification Validation/
Scope Dry run Analysis of
Identification of scripts results
and signoff
Strategy Definition& Design

Iterative Design and Development

Change Management
Milestone Recommendation
Requirement Design
Iterative Analysis & tuning
Checkpoint
process Checkpoint Checkpoint
Joint Activity

Confidential property of Infosys. For internal circulation only 20

Methodology – Requirements Analysis(NFR)
System Appreciation –> Understand and appreciate the significance of the various s/w & h/w
components which are a part of the SUT (System Under Test)

Influx Model -> Collect the requirements using Influx Methodology.

Infrastructure Model
Transaction Model
WorkLoad Pattern Model
Growth Model
QoS Model
Data Retention Model

SLA -> Verify whether the SLAs are reasonable and have a concrete basis. Determine In/Out of
Scope activities

Priority -> Given a set of applications, prioritize them to pick up the critical ones

Microsoft Word Microsoft Word Microsoft Word

Document Document Document

Confidential property of Infosys. For internal circulation only 21

 Methodology – Planning & Design
Workload Modeling Determine the usage pattern of the various transactions. Analyze the transactions by
frequency and volume. Resultant is a real time Distribution Mix of the various transactions
Critical Scenarios Identify critical scenarios to be Performance Tested. Business Critical, Resource Critical

Type of Tests Identify the types of Tests to be Performance Tested as driven by the requirement viz.
speed, scalability, stability
Tools Choose the Testing tool, Monitors, Profilers as driven by the requirement. Application
Protocol, Budget are key factors
Stubs/Drivers Determine the need of Stubs/Drivers if tools/monitors don’t satisfy the need

Performance Parameters Identify the monitoring parameters to critically analyze the SUT

POC May be required to critically ascertain whether the desired output is obtained, if the testing
involves a new Tool/Technology
Offshore-ability Not all applications could be Performance Tested offshore. Need to derive an Offshore-
ability criteria and identify applications to be offshore
Test Environment Validate whether the installed software versions & patches reflect the production
environment. Also the Prod Env hardware should be linearly scaled down to the Test Env
hardware to facilitate scalability assessment
Design & Develop Develop Performance Test Scripts, Stubs/Drivers as decided. Validate scripts

Confidential property of Infosys. For internal circulation only 22

Methodology – Test Execution
Execute -> the different types of Tests as per the Test Plan ; Baseline/Benchmark Tests

Monitor -> the counters during test execution. Requirements will dictate what measurements will be
collected and later analyzed
Measurements collected during this activity may include but aren't limited to the following:
Transaction Response Time
TPS
Memory Usage
CPU Usage

Correlate -> the captured output from all the monitors (Testing Tool, Server/DB/Network Monitors) to
identify the problematic component

Iterative Test Execution -> Isolate the problematic components (modules/scenarios/transactions)

and re-iterate the test executions to confirm the findings

Confidential property of Infosys. For internal circulation only 23

Methodology – Recommendations
Analyze -> the results and identify Performance Bottlenecks

Scope -> Not Tuning but providing Tuning Recommendations is a in-scope function of Performance
Testing

Tuning Recommendations -> Instances like:

Altering System Configurations – DB Init/Storage Parameters, AppServer, WebServer
Tweak Programs – Database Queries, Program Design
Modifying Designs – Database Design, Table/Index Design

Experience -> Common observation is that 60% of Performance issues are related to Database –
which includes DB configuration, Indexing and SQL Query structuring

Confidential property of Infosys. For internal circulation only 24

P-Test Environment
Real Time -> Ensuring Test Environment as real as Production Environment is desirable

Scale -> In case of a scaled down Test Env, its recommended to Linearly scale down from the
Production Env, to facilitate scalability assessment

Network -> An Isolated Test Environment ensures the performance results are free from all kinds of
biases

Confidential property of Infosys. For internal circulation only 25

How to go about deciding the Approach?
• What kind of Application is it?
• What is the application architecture?
– In production environment
– In Test environment
• What is the deployment strategy?
• What is the peak load expected?
– Current peak load
– future peak load
• What are the protocols?
– Web/HTTP
– SAP
– Oracle NCA
– Citrix
– Thick client etc

Confidential property of Infosys. For internal circulation only 26

Estimation Considerations
• NFR analysis, planning, strategize, etc.
• Identify the scenarios
• Evaluate the tool that is to be used
• Define or collect the SLA’s
• Identify the Test Environment
• Test Data generation
• Scripting of the identified scenarios
• Dry-Run of the scripts
• Scenarios for the various types of performance testing
• Iterations of execution

Confidential property of Infosys. For internal circulation only 27

Scripting
• Test tools replace human users with virtual users, the actions a Vuser performs are captured
in the script.
• Develop the basic script with the help of the tool for recording script.
• Enhance the basic script by
– Parameterization – It is the input data to simulate the real time scenario e.g. User Id,
Password, entry fields for a transaction
– Inserting Transactions – Transactions are defined to measure the performance of the server. It
measures the time taken by the server to respond to a specified request. It is identified by
marking the beginning & end of the task e.g. to measure the time taken for Login, define a
transaction for the click of Login button
– Correlation – Correlating the data enables us to save the dynamic data & use it throughout the
scenario run e.g. in a web based application a session id is generated whenever a user logs
in. Hence the record time session id needs to be correlated so that the script can be executed.
Similarly the functionality of creating a new Insurance case, the policy no. needs to be
correlated as it is a unique no. generated for each case.
– Inserting check points – Checkpoints are added to a transaction to verify that the scripted
action has received the correct response from the server or not e.g. a particular string
appearing on the page can be checked for in the response
– Inserting check points – Checkpoints are added to a transaction to verify that the scripted
action has received the correct response from the server or not e.g. a particular string
appearing on the page can be checked for in the response
– Adding control-flow structures – Additional logic can be incorporated to the scripts by inserting
if-else, looping structure etc.
– Error Handling – Abnormal termination of the script can be prevented by handling errors in the
response of a transaction and displaying appropriate message or taking an appropriate action

Confidential property of Infosys. For internal circulation only 28

Execution phase
• Iterative execution
• Joint Analysis of test results, logs
• Identify the bottlenecks
• Identify network impact on the end-user perceived response time
• Identify the most expensive call functions
• Identify the database constraints
• Tuning Recommendations
• Release of P-test Summary Recommendation Report

Confidential property of Infosys. For internal circulation only 29

Execution Modes
OSPTC provides the flexibility to performance test applications either in Remote Execution mode,
Script Development mode, Offshore Testing mode or in a combination there of. The mode to be used
for a particular application will be determined based on a set of criteria (detailed in subsequent slides)
Remote Execution Mode
 Performance Test Environment located onsite at Client
 Test tools & load generator machines setup at Client’s onsite test environment
 Infosys Offshore performance test team accesses test tools & test environment over a
remote desktop connection which is secured
 Offshore Team scripts & triggers performance tests from offshore over the dedicated network
Script Development Mode
 Scaled down test environment setup at offshore, for development of test scripts
 Load generator machines setup at Client’s onsite test environment
 Execution of tests trigged on the Client’s onsite test environment

Confidential property of Infosys. For internal circulation only 30

Execution Modes contd…
Offshore Testing Mode
 A scaled down or production like test environment set up at offshore
 Performance tests executed using tools and load generation machines setup at offshore
Infosys locations
 All activities done at offshore
 Prior to application deployment, sanity tests run on production/test environment to validate
results

Confidential property of Infosys. For internal circulation only 31

Reports analyzed for performance issues
• Transaction Summary
– This report summarizes the number of transactions that failed, passed, aborted or ended
in error
• Transaction Response Time – Under Load
– Displays the transaction response time relative to the number of running vusers at any
given point during the scenario
– The increase in response time should not be too high/too less with increasing vuser
• If the difference is too high, then it means the system has problems scaling up
because of queued requests or waiting for server resources to be allocated
• Transaction Performance Summary
– This report displays the min, max and avg. response time for all the transactions in the
scenario
• Percentile Graph
– This graph can be used to find the percentage of transactions that have been completed
with in the acceptable response time
• Network Monitor Graph vs. Running vusers
– This cross comparison provides us the info on the impact of network delay due to
increasing running vusers. More running vusers might cause network congestion and so
the network delay might increase heavily after a point of time. This gives us the
limitation of the network (if any)
• Transactions per second vs Transaction Response time – Average
– This cross comparison lets us know the impact in response time (if any) against the
number of transactions the server is handling at each second during the scenario run
– With increasing transactions, the average response time going too high means a
bottleneck/limitation in the web architecture

Confidential property of Infosys. For internal circulation only 32

Contd…
• Running vusers vs throughput
– This cross comparison lets us know how running vusers increase load to the server and thus increases the
number of hits to the server
• If the number of hits is increasing along with more running vusers then the system is behaving fine
• If the number of hits remains constant with more running vusers, then it means that the system is not
capable of handling so many people. In this case, we should try to find the number of vusers after which
it remains constant
• Running vusers vs hits per second
– This cross comparison tell us how the server throughput increases/decreases against running vusers
increasing/decreasing
• If the throughput is increasing along with more running vusers then the system is behaving fine
• If the throughput is decreasing with more running vusers, this may be because the failures are more after
some time, so the server is sending minimal response for the error messages
• Hits/sec vs Transaction Response time – Average
– This cross comparison tells us how the number of hits affects transaction performance
• CPU Utilization of all tiers
– The benchmark is that CPU utilization going above 80% and staying at that level for a longer interval is not
favorable. If that’s the case then there is some problem. The application is doing lot of processing and making
a heavy load on the CPU
• Memory Utilization of all tiers
– If there is a high memory requirement, then there may be a lot of virtual memory that need to be used and also
this requirement might cause trashing (high disk I/O) which will affect the performance
• Disk I/O on all tiers
– If the I/O is high, then it affects the performance

Confidential property of Infosys. For internal circulation only 33

Various Performance test tools available in the market

Name Protocols Supported Licensing

Mercury Interactive- Http, Winsocket, SOAP, WAP, IMAP, POP3, EJB, Licensed
Loadrunner RMI-Java, COM/DCOM, Tuxedo, Siebel, SAP,
Terminal Emulation
Radview’s Webload Http Licensed
Compuware QALoad HTTP, SSL, SOAP, XML, CORBA, TUXEDO, Licensed
Winsock, Java, ODBC,
SAP R/3, PeopleSoft, VT100-520
Segue’s SilkPerformer HTTP, FTP, Streaming Media, LDAP, TCP/IP, Licensed
UDP, WAP, MMS, XML/SOAP, Microsoft .NET
SOAP Stack, CORBA (IIOP), Oracle OCI,
PeopleSoft
Microsoft Web Http Freeware
Application Stress Tool
(WAS)
OpenSTA Http Freeware , Open
source
Http-Load Http Freeware

Confidential property of Infosys. For internal circulation only 34

Common Tools and protocols
• Some of the commonly used tools
– Testing tool
• HP LoadRunner
• Radview WebLoad
• Borland SilkPerformer
• Opensource JMeter
– Monitoring tool
• Wily Introscope
• Quest Performasure

• Some of protocols used

– Web/Http
– Oracle NCA
– Citrix
– SAP
– MQ etc

Confidential property of Infosys. For internal circulation only 35

Tool’s selection
Borland Silk
Feature HP LoadRunner Radview WebLoad
Performer

Ability to change recording to a

Yes, for some No, Supports only one
different protocol in the middle Yes
protocol protocol (Http)
of a recording session

Actions in a script can be

Simple runtime No, Programming effort Simple runtime
iterated any specified number of
setting change is required setting change
times without programming

Different modem connection

Yes Yes Yes
speed simulation

Cookies and session Ids

No, Programming effort
automatically correlated during Yes Yes
is required
recording

Simulated IP address for virtual

Yes No Yes
users

Ability to provide graphical

results and export them to Yes Yes Yes
common formats

Confidential property of Infosys. For internal circulation only 36

 Any Questions?

Thank you

Confidential property of Infosys. For internal circulation only 37