See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/261260567

Performance testing: Analyzing differences of response time between

performance testing tools

Conference Paper · June 2012

DOI: 10.1109/ICCISci.2012.6297157

CITATIONS READS

4 909

2 authors, including:

Dhiauddin Suffian
MIMOS Technology Solutions Sdn. Bhd.
15 PUBLICATIONS 77 CITATIONS

SEE PROFILE

All content following this page was uploaded by Dhiauddin Suffian on 01 January 2016.

The user has requested enhancement of the downloaded file.

 Performance Testing: Analyzing Differences of
Response Time between Performance Testing Tools
Muhammad Dhiauddin Mohamed Suffian
Faculty of IT & Multimedia Communication
Open University Malaysia
Kuala Lumpur, Malaysia
dhiauddin@oum.edu.my
Abstract— This research focuses on the study and evaluation of
response time differences given by three tools used for
performance testing. The motivation for this research work is to
understand the behavior of various performance testing tools
towards determining the accuracy of the response time result. It
is conducted with the aim of demonstrating and proving that
differences of response time do exist between different tools when
conducting performance test for the same webpage as well as
analyzing the reasons behind that situation. A static HTML
webpage is put under load test for 1, 100, 200, 300, 400 and 500
concurrent users performed by the three tools. The findings
clearly showed that different performance testing tool gave
different response time when conducting load testing on same
webpage. The findings are also supported with the justification
for these differences, which involve architecture and simulation
mechanism of the respective tool. The summary and future work
is presented at the end of the research
Keywords- performance testing; response time; load test; open
source tool; concurrent users
I. INTRODUCTION
Performance testing is one of the test strategies performed
for the software under test, usually at system testing level. By
conducting this test, we could assess the readiness of the
software system in handling users’ loads thus ensure it could
response within an acceptable time range as expected by the
end users. This type of testing becomes more crucial if it
involves heavy integration with other external systems since
performance degradation might take place.
Ignoring performance test means that your system is not fully
tested, especially from the risk and operational profile
perspectives.
The trigger point to embark on this research is driven by
real experiences as independent testing team in dealing with
various issues, difficulties, challenges as well as successes in
doing performance test. These involve conducting performance
test for various type of software: standalone, web services, web
application, mobile application as well as grid and cloud
applications. Suitable and correct approach or strategy of
performance test need to be put in place for software so that
performance defects can be fixed and the test results are
accurate. Thus, the selection and use of right testing tool for
978-1-4673-1938-6/12/$31.00 ©2012 IEEE
Fairul Rizal Fahrurazi
Product Quality & Reliability Engineering
MIMOS Berhad
Kuala Lumpur, Malaysia
fairul.fahrurazi@mimos.my
performance test is very crucial to achieve this target. This is
where the issues start to come into picture.
Several issues have been observed related to tools when
conducting performance testing such as tools compatibility
with the software under test, tools installation, tools setup, tools
flexibility in doing test both for client and server side and also
the one that becomes the focus of this research, which is
response time generated by the tools.
The research problem falls into area on demonstrating and
proving that the response time for conducting performance test
for the same website is different when using different
performance testing tools. In addition, the research also
suggests potential reasons or root cause behind these
differences. This work tries to answer the question on “Why do
different performance testing tools produce different response
time that are not even closed to each other?”
The research discussion is organized into several sections.
Section II discusses the prior related works on webpage
response time and calculation of response time. Section III
outlines the overview of the common features of the testing
tools used for the experiment. Section IV describes the
environment setup while Section V drills down further on
discussing and elaborating the findings of the experiment.
Section VI concludes the overall research together with
recommendation for future works
II. RELATED WORKS
Most previous work on performance testing tools
comparison ignored on different result reported by each tools
[1][2][4][5][7][8][9][10][11][12].VCAA[1] uses pricing and
user friendliness as a criteria to decide which tool to use while
JDS[2] mentions about the ability to emulate a complex
business process and support of unlimited number of
concurrent users.
Testingrefeclections.com [3] concludes that accuracy of
load and response time is something we need to evaluate
against our particular application and not something to
compare when determining the tool to use or buy. But, there is
no work so far to understand why they are different against
each other at the first place. Shall we have a framework or
memorandum of understanding (MOU) about the uniformity of
response time on all performance testing tools? Each tool
claims they are better than the others but none able to justify
the performance testing results against the real world.
Our research effort is proving if there is a difference
response time between different performance testing tools and
potential reasons contribute to it. This is to enlighten the
performance testers that no tool in the world able to fully
replacing human when performing performance testing as the
result from each tools are not consistent, at least for now.
III. OVERVIEW OF PERFORMANCE TESTING TOOLS
Performance tools are used for several kinds
of performance testing including load test, stress test, volume
test and endurance test. Most tools are capable to do this either
open source or proprietary tools. For this research, three
performance testing tools was selected comprise of two open
source tools and one proprietary tool, which their names are not
mentioned and only represented as Tool A, Tool B and Tool C,
respectively. However, for the purpose of understanding the
tools, only their characteristics or features are explained below:
A. Tool A
Tool A is an open source tool that is purely developed on
Java platform. It sits as desktop based tool for performance
testing. It is designed to serve functional, load and stress
testing, in which it is extensible to write own test to suit the
scenario. Tool A can simulate heavy load on the application,
server and even the network. Besides able to give instant visual
feedback, this tool is also capable in performing load and stress
testing via distributed approach. It supports protocols such as
HTTP, JMS, JDBC, FTP, SOAP as well as LDAP. Other
features of this tool are it can be used across platforms,
supports full multithreading framework besides allows caching
and offline analysis with replaying of test results.
B. Tool B
Tool B is an open source load testing tool that is developed
using C++ language. It can perform heavy load tests with
performance measurements using scripted HTTP and HTTPS.
This feature-rich GUI-based web server benchmarking tool can
only runs on Windows-based platform. Its performance scripts
are recorded using own proprietary language that is simple and
can support custom functions, variable scopes, and random or
sequential lists
C. Tool C
Tool C is a proprietary tool and is one of the established
performance testing tools in the market. It is built on Eclipse
and Java. It offers automated performance testing for web and
served based application so that the scalability of those
applications can be validated. This tool can be used across
platforms such as Windows, UNIX and Linux. System
performance bottlenecks and their root cause can be identified
by using this tool. Besides the capability to create code free
tests, this tool is also able to automate test data variation and
enables insertion of custom Java code for flexible test
customization. In terms of operating systems, it supports
operating systems such as Windows, Linux, and z/OS.
IV. THE ENVIRONMENT SETUP
Correct test environment setup for this experiment is very
important to ensure the accuracy of response time from the
performance testing tools. The setup involved two identical
machines. One machine serves as the client with performance
testing tools installed and a server which hosts the static web
pages. These machines are connected via PC-to-PC LAN cable
or commonly referred as cross cable connection. The reason for
this kind of environment setup is to minimize the network
factors from influencing the results of performance test. The
setup is represented in Figure 1 below:
Figure 1. Test setup for comparison of performance testing tools
As presented in the diagram, strict rule for conducting the
test is to make sure both client and server machines must be
rebooted after completing each cycle of test for each tool used,
so that the machine resources such as RAM and CPU are at
their fresh states. Another important element for this test is to
use static webpage in order to maintain the page size for correct
analysis of the results obtained.
While the test was running for a particular tool, the services
for the other remaining tools as well as other unnecessary
services were stopped or disabled. As for the server, it only
hosted static HTML pages with no cache and cookies. Running
processes on this machine were kept to a minimum since
unnecessary processes and server-side logging were disabled.
Load testing was performed from 1 user access and gradually
increased to 100, 200, 300, 400 and 500 concurrent users with
zero ramped-up time.
The hardware specifications for both machines are outlined
as below:
CPU/processor : Intel Pentium D 3.4 GHz
RAM/memory : 2 GB
Hard disk storage : 80 GB
Network Card : Integrated 10/100/1000 Ethernet
As for the software specifications, the details are as
follows:
Client machine
Operating system : Windows XP SP2
Java JDK : JDK 1.6.0 update 21
Tool : Tool A (open source); Tool B
(open source); Tool C (proprietary)
 Server machine C. Result of Tool C
Operating system : Windows Server 2003 Enterprise
Edition SP1 The result of load test for Tool C is presented below in
Java JDK : JDK 1.6.0 update 21 Table III:
Web server : Internet Information Services 6
TABLE III. PERFORMANCE TEST RESULT OF TOOL C
HTML page size : 65.8 KB (Page: 7 KB; Image 1:
25.2 KB; Image 2: 33.6 KB)

V. FINDINGS AND DISCUSSION

The test began by simulating one user accessing the
HTML page at the server machine using Tool A. At the same
time, the CPU, memory and disk I/O utilization for both
client and server machines were recorded. This was repeated
another two times for the same virtual user and also for other
virtual users set until the number reached 500 virtual users.
The same kinds of load test were performed using Tool B
and Tool C. Response time, average CPU utilization,
memory utilization and disk I/O utilization for each set of
virtual users of every test are the metrics recorded and
collected for further analysis.

A. Result of Tool A D. Comparison of Performance Test Results between Tools

The result of load test for Tool A is presented in Table I Response time results obtained from the three tools were
below: grouped and plotted into the graph. Three graphs were produced
TABLE I. PERFORMANCE TEST RESULT OF TOOL A
to represent results for each round of test. The graphs are as
below in Figure 2, Figure 3 and Figure 4:

B. Result of Tool B
The result of load test for Tool B is presented below in Figure 2. Performance (load) test result for round 1
Table II:

TABLE II. PERFORMANCE TEST RESULT OF TOOL B

Figure 3. Performance (load) test result for round 2


Figure 4. Performance (load) test result for round 3
From the graphs plotted, it is clear that different
performance testing tools produced different response time
results. The experiment of running the test using different
tools via load test for the same HTML web page has achieved
the objective on proving that when using different
performance test tools, the response times obtained are not
similar or even close to each other, especially as the
number of concurrent users increases. At the same time, it
also triggers another question “Why do different performance
testing tools give different response times when testing the
same web page?”. Subsequent sub-section tries to provide
possible reasons on this scenario.
E. Potential Reasons for Response Time Differences between
Performance Testing Tools
Some fundamental understandings are already in place to
explain potential reasons behind the differences of response
time between performance testing tools. Among others
are capturing and simulating the load used for the
performance test, the method of calculating metrics
gathered by each tool, the language to develop the tools, the
architecture of the respective tools. Subsequent
paragraphs try t o compare potential reasons between Tool
A and Tool C.
Capturing and simulating the load plays the biggest role
so far that contribute to the variation of response time. In
comparison between Tool A and C reveals there are several
extra items being recorded and simulated when generating
user loads. Tool C by default uses Internet Explorer browser
when recording and it's observed ASHX files recorded in
the list as an additional items compared to Tool A. ASHX
is a web handler file . Tool A however does not do the
same. Apart from that, JavaScript and CSS files seem have
higher response time in Tool C compared to Tool A
around 18 percent. It is observed the remaining file types
consists of images (GIF, JPG) and HTML do have a small
variation between 1 to 4 percent.
The method for calculating metrics gathered by each tool
contributes to the variation of the response time. The
fundamental formula to calculate for Tool A and Tool C is
identical which is based on last byte sent and last byte
received. However, Tool C for an example introduces Inter-
Request-Delay, where some requests may have delays
associated to them. Tool A does not automatically
implement a delay and it is up to the user to manually
configure it based on their preferences. To note, Inter-
Request-Delay or wasted time is a separate entity from Think
Time .All these 3 tools are identical when excluding Think
Time from total response time.
Architecture for each tool differs greatly. For an example,
Tool A and C developed by using Java and they require JVM
to run , the value setting for Java Heap Size plays a role here
to generate the best user load without putting extra burden to
the client and by default after the installation, both tools have
their own settings. As for Tool B, its architecture relies on web
relay daemon facility allowing CORBA-based communication
to be transmitted between machines during executing the
performance test.
VI. CONCLUSION AND RECOMMENDATION
This paper has shown that different performance testing tools
give a different response time and leading us in future for an
opportunity to quantify and prove further for each of the
potential reasons contribute to that. The next critical research is
around capturing and simulating the load by each tools, where
we may need to continue analyzing each of the HTTP request
and response through tool available in the market for an example
Wireshark. With this, we can fully understand at the packet level
what are being transferred and received at each tools and why
they are being included or excluded. Currently, there is no tool
able to tell us if our application is fast enough in term of user
experience in a reality or in other words, response time reported
by testing tools versus one done by naked eye. Hence it is crucial
for performance testers to understand that there is no tool able to
automate and tell us the full picture of the application's
performance going to be in a real world. It is back to human
brain to analyze the information given and performance testing
tools are just one of the tool can be used to achieve that.
REFERENCES
[1] VCAA (2005). Stress, Load, Volume, Performance, Benchmark,
Baseline Testing Tool Evaluation and Comparison
[Online].Available:http://www.vcaa.com/tools/
loadtesttoolevaluationchart-023.pdf
[2] JDS Australia (2009), A comparison of Open Source Load Testing
tools [Online]. Available:http://www.jds.net.au/tech-tips/load- testing-
tool-comparison.
[3] Barber, S. (2006). Choosing Performance Testing [Online].
Testingreflections.com. Available: http://www.testingreflections.com/
node/view/3939.
[4] Vinod, P. (2008). Open Source & Commercial Performance Testing
Tools [Online]. Accenture. Available: http://www.scribd.com/
doc/27765939/Performance-Test-Tools-Comparison.
[5] Nageswaran, S. (2005). Performance Tools Comparison [Online].
Testingreflections.com. Available:http://www.testingreflections.com/
node/view/1756.
[6] IBM Support Portal (2011). RPT HTTP Response Time [9] Subraya, B. M. (2006). Selection of Tools - in Test Preparation Phase
Breakdown [Online]. Available:http://www-01.ibm.com/support/ II:Test Design. IGI Global. United States of America.
docview.wss?uid=swg21391178. [10] Barber, S. (2007). How Fast Does a Website Need To Be. PerfTestPlus
[7] Molyneaux, I. (2009). Choosing the Right Performance Testing Tools - Inc. Florida, United States of America.
in The Art of Application Performance Testing. O’Reilly Media. United [11] Savoia, A. (2001). Web Load Test Planning: Predicting How Your Web
States of America. Site Will Respond to Stress. Google Inc. Mountain View, United States
[8] Microsoft Corporation. (2007). Quantifying End-User Response Time of America.
Goals - in Performance Testing Guidance for Web Applications. [12] Barber, S. (2007). Beyond Performance Testing., Rational Developer
Microsoft Press. United States of America. Network. PerfTestPlus Inc. Florida, United States of America.

View publication stats