LoadUIWeb Pro

by SMARTBEAR
2
LoadUIWeb Pro
by SMARTBEAR
A Guide to Effective
Load Testing
Contents
Performance Maturity Assessment ........................................... 3
Are your business associates planning your Performance Intervention? ....... 3
What Makes a Good Load Test Report .................................... 5
How ready are you for heavy load on your website? ........................................ 8
Transactional testing .................................................................................. 9
Geographic diversity .................................................................................. 9
Secrets of Effective Performance Analysis ............................. 10
Endurance testing .................................................................................... 10
Stress testing ........................................................................................... 10
Time to frst byte ............................................................................................. 12
Response transfer time ................................................................................... 12
Gaps between requests .................................................................................. 12
The Secrets of Proactive Performance Management............. 13
Summary ................................................................................ 16
3
LoadUIWeb Pro
by SMARTBEAR
Performance Maturity Assessment
Are your business associates planning your Performance
Intervention?
We have seen it happen in many organizations at some time or another,
when the blissfully-unaware Performance Manager fnds out that their
current approach to performance monitoring is disappointing their
organization. Slow response time when web pages are trying to load,
broken links and redirects, timeouts, all are perceived by the end-user
as lousy performance, but the internal organization does not have a
handle on where the bottlenecks are occurring.
The Performance Manager is invited into a conference room, and the
dialogue often sounds like:
Help Desk Manager: Why are we always hitting the same issues?
Cant we fx the app to stop doing that?
The Network Manager: It is NOT a frewall issue.
The Development Manager: Its not the application either!
Your Boss: What are you DOING about this?
The Performance Manager, without the beneft of solid performance
analytics, is forced to simply acquiesce to the charges.
Are you in danger of an Intervention? Take our performance quiz to see
if you might be headed for an ambush in a conference room. Consider
the following point system:
1. Clueless These things are happening on my site, but I have no
method to anticipate them, plan for them, or guard against them. My
help desk is my alert system
2. Foggy I have some idea what is hitting us, but I have no way of
compiling different indicators and statistic into something useful. We
4
LoadUIWeb Pro
by SMARTBEAR
use end-of-month issue analysis to decide what will be our focus for
the next month.
3. Handle I have a basic grip on what we see day-to-day, I have a
process I can use to defne Root Cause Analysis on the big issues,
but I am still reacting to these issues based on business impact.
4. Driver I have a system that is collecting useful indicators
automatically, and it is giving me enough data on which to base
strategic decisions.
5. Leader My decisions are second-nature, based on the full
scope of information I am getting. This information and analysis is
at my fngertips in real-time, making it easy to fx issues before my
customers or my peers even see degradation in performance.
Rate your current System Performance with a 1 to 5 on the following
questions.
Question 1: How would you rate your ability to draw meaningful
conclusions on your systems performance based on the information
presented in the reports you currently use?
Question 2: How comfortable are you that your system and its
individual components are certifed against peak loads, and times
of stress on the network? How ready are you for your next peak
interval?
Question 3: How easy is it for you to quickly assess the root cause
of an issue, isolating a problem down to a component or procedure?
Question 4: How are improvements driven on your system, are they
anticipated, planned and implemented based on metrics, or are
they frequently deployed in a particular response to an outage or
customer gripe?
5
LoadUIWeb Pro
by SMARTBEAR
Question 5: How comprehensive a Service Level Agreement are
you willing to offer your customers? On what would you base your
numbers, and how would you argue them when your customer says
you have not achieved them?
Based on the described point system, calculate your score to rate your
approach to performance management.
25-20 There is no way they could surprise you with an intervention,
and they know that even if they tried, you would be able to turn the
tables painfully back on them
19-15 You are in reasonable shape, but something could surprise
you
14-7 Some positive steps in Performance Management could make
this a whole lot easier on you
Under 7 Do you have a minute to talk in this conference room?
And Performance Managers, if you have had this forwarded to you
by someone <gulp>, you might already be headed into a session. Be
prepared.
In this series, we will discuss each of the fve sectors of Performance
Management, and will present you with ways you can raise your score,
make your application more bullet-proof, and hopefully keep you out of
the dreaded conference room intervention.
Load testing can be a complex activity, from defning the tests
accurately to running them effectively. Most conversations about
proper load testing focus on those activities but an often overlooked
part of the discussion is how effective your test reports are. Being able
What Makes a Good Load Test Report
6
LoadUIWeb Pro
by SMARTBEAR
to accurately interpret the results of your load test may help prevent
production problems later.
Performance reporting should follow the same basic rules of reporting
that we learned in elementary school:
Who-What-When-Where-Why-How
A concise opening
statement of theme
And some concrete
argument to back up the
theme statement
In essence, a performance
reporting system should be
able to deliver a high-level overview of how a web site is performing
under load, but it should also contain detailed visibility into the internal
structure of the site and its infrastructure. Reports should illustrate
the context and highlights of performance so clearly that anyone could
interpret them without having to be an expert about the data and metrics
used to reach these indicators.
The system should also have some degree of customizability, allowing
the information to be re-contrasted as needed - often the most
compelling analysis involves only the interaction of two or three key
elements of the data gathered and the report should be able to be
re-formatted to showcase them. The timeliness of this information is
paramount, as near-instant availability of the metrics and the ability to
concentrate on different testing timeframes, will allow the proper context
of an issue to be stressed.
It is also important that the load test reporting system be able to hold
and reproduce test results of multiple passes of a given test, as a
method to baseline results over time and thus gauge improvement.
How would you rate your ability to
draw meaningful conclusions on
your systems performance based
on the information presented in the
test reports you currently use?
7
LoadUIWeb Pro
by SMARTBEAR
A good load test report should be able to show:
Who: A good reporting system identifes and reports on the
Key Performance Indicators (KPIs) of the website youre testing.
For example: Fulfllment numbers are not where we would
like., Shopping cart abandonment is too high, Customers are
complaining of slowness. The reports should focus on KPIs as
success criteria.
When: A good report should also be able to address a discrete
time duration, be it an instant test of a single component, or a
duration test of multiple scenarios on a website happening over a
weekend or longer period. As stated previously, regular testing of a
website might also be compiled as a series of referencable reports,
to baseline performance of the site and its infrastructure.
Where: There are two where components to consider: a physical
version of where that alludes to the infrastructure and hosts of
the website, and a virtual where, indicating at what step of the
application the issues are being seen.
With a product like SmartBears LoadUIWeb Pro, a distributed
platform of load testing can be deployed using LoadUIWeb Pro
Remote Agent Services at multiple locations on your network,
inside and outside of your frewall, to contrast load testing
results, and physically pinpoint where bottlenecking is occurring.
By setting alerts on parameters such as such as Quality of
Service or Content Validation, analysis of the steps in the
application can be graphically represented in the report as
errors, allowing for quick identifcation of logically where in the
application the issue is being seen.
8
LoadUIWeb Pro
by SMARTBEAR
Why: (More correctly Why Not?) Reporting should have some
method to plainly identify the reason why the KPI objectives are
not being met, by indicating an issue on an individual object or
subsystem Hierarchical systems, such as a compiled list of the poor
performers or a graphical representation of the performance of each
request made to the website can quickly determine a course of
action for remediation. SmartBears WebLoadUI offers both a Top
10 list of poor performers, and a waterfall representation of each of
these Top 10, for ease of diagnosis.
With the proper reporting, the overall status and health of the website
and infrastructure is quickly comprehended by all of the stakeholders.
Such a concise view can suggest possible areas to address, but more
importantly gives the intelligence into the system for the Performance
Manager to base their decision. However, the ease of generating a
concise report, pre-formatted with a summary section and compilation of
the relevant data, is of most concern. Using WebLoadUI, this step has
been automated, so that a report is produced at the end of every test
run, using the data contained in the tests unique log fle.
How ready are you for heavy load on your website?
Load Testing is an important subset of any overall Performance
Management strategy. It is a technical investigation done to determine
the scalability, speed, and/or stability characteristics of the system
under test. Load testing is crucial for assessing if the software
solution, including the server equipment on which the solution runs,
the infrastructure network where the servers exist, and the web access
routes and equipment, will react with the necessary speed when
subjected to normal or above-normal amounts of user traffc.
In its infancy, load testing entailed events where multiple people
accessed an application simultaneously from distributed computers.
While this may successfully simulate a stressful event on a system,
it lacks the qualities that are crucial to a true evaluation of a system
9
LoadUIWeb Pro
by SMARTBEAR
under load. Through load test
automation using tools like
LoadUIWeb Pro, we can create
and control populations of virtual
users that can simulate valuable
aspects of load testing:
Transactional testing
To fully test an application, merely accessing the landing page of a
website is not going to give an adequate assessment of how well the
supporting pages of that site are performing. A better test is to simulate
a virtual population running a transaction comprised of a series of
steps, each of these steps possibly consisting of multiple requests
and responses to both your host server as well as third-party sources,
should be able to be performed by the virtual user population. Further,
as different users to a website may follow differing sets of steps on that
site, comprehensive load testing should include running multiple test
scenarios concurrently. These can be further varied by using multiple
browser styles, transmission bandwidths, individual session IDs, and
other variations to the test parameters to most closely simulate a real
world experience.
Geographic diversity
For a website that is meant to be accessed from outside of the frewall,
testing that only occurs on the internal infrastructure will overlook some
important possible bottlenecks, such as single sign-on mechanisms
or other security devices. Even a singular perspective inside of the
frewall, such as initiating all virtual users from just one location, will
show a more myopic result than testing done from multiple separate
locations in the network. It is therefore important to be able to
place remote instances of your virtual user population, but be able
to confgure, control and correlate data from them with one central
management console.
How comfortable are you that your
system and its individual compo-
nents are certied against peak
loads, and times of stress on the
network? How ready are you for
your next peak interval?
10
LoadUIWeb Pro
by SMARTBEAR
Endurance testing
Endurance testing is an assessment lasting many hours or days, and is
a way of testing the durability of the solution and its components. These
exercises can reveal the occurrence of memory leaks or similar system
degradations on the host servers over time. Here, the ability to draw
server-side metrics into the test results can be greatly benefcial to the
assessment of the systems durability in the production environment.
Stress testing
Testing in this phase entails pushing the website and its infrastructure
well past its anticipated normal loads, as a way to assess the upper limit
of system capability, fnding the theoretical capacity for a solution. This
type of testing should be used to not simply knock over the service, but
to look at the elasticity of the solution, and determine if it can recover
from being pushed past infrequent stressful loads. In addition to fnding
that upper limit of performance, stress testing can also be useful for
right-sizing your system, and being able to justify future build-out plans
and timeframes.
Load testing is vital to your Performance Management planning. Having
the answers from these kinds of tests allow the tester to really feel the
capabilities of their infrastructure, and more importantly, to instinctively
recognize the signs of a website undergoing a stressful incident and
make intelligent decisions.
Successful Performance Analysis depends on the ability to identify
bottlenecks in the application or system infrastructure. A bottleneck
could be caused by any object on a page that is taking longer than other
page objects to fully load, or it could be an overloaded segment of the
network or a security process that is delaying the browsers requests
and responses.
Secrets of Effective Performance Analysis
11
LoadUIWeb Pro
by SMARTBEAR
Analysis and tuning is not a one and done type of event, but rather
a cyclical process of evaluation and elimination of performance
bottlenecks, using iterative load
testing of the application. Every
pass of a load test could uncover
new issues in the system, as the
elimination of one larger issue
in a previous pass might unmask other issues. The errors found
should become more granular with each testing cycle and subsequent
remediation, so the results of iterative testing are stronger, faster
applications with fewer performance defects. If we know that an
application has been tuned previously, new degradations to the system
can be easily identifed on the latest run of the test.
Waterfall charting can be a useful tool for identifcation of root cause.
How easy is it for you to get to the
root cause of an issue, isolating a
problem down to a component or
procedure?
12
LoadUIWeb Pro
by SMARTBEAR
Through the graphical representation of each request against time, we
can easily see the objects causing degradation in the test. The different
sub-timers included in the graphed request, such as time to frst
byte and response transfer time, can give us insight into suggested
remediation for the object. Here are some things to look at when
performing any kind of load test analysis.
Time to frst byte
A long frst byte time indicates that the host server is delayed in getting
the beginning of the requested information back to the browser. This is
most frequently caused by an overloaded host server, or a congested
pathway between the server and the browser. Remediation on an
object with a long frst byte value might include having it served from
a different host, moving the serving host to another location in the
infrastructure, or possibly optimizing the object through the use of a
Content Delivery Network (CDN)
Response transfer time
13
LoadUIWeb Pro
by SMARTBEAR
A long transfer time might indicate that the object itself might be
oversized for the application. This could be rectifed by allowing the
application to call down the information contained in this object through
a series of smaller requests, relying on the nature of the browser to
perform these requests in parallel.
Gaps between requests
In the waterfall report, it might appear that there is a short length of time
between the conclusion of one request and the beginning of processing
for the next request. The most frequent reason for this is that the
browser requires extra time to establish a connection to the target web
server. This can be tuned in the application.
The comparison of a recent waterfall chart to a historical version can
give you the necessary detail for your frst assessment of root cause.
Also, by graphically seeing the components and infrastructure used
by the site, you can correlate information he has received from other
sources regarding network activity or recent outages, which might also
suggest root cause.
Reactivity to issues is often an expensive and imprecise method for
dealing with outages or degradations in service. Often your customers
are citing that an issue is chronic, they are demanding to know what
concrete steps are being taken to fx the issue, and the technology
The Secrets of Proactive Performance Management
14
LoadUIWeb Pro
by SMARTBEAR
team feels pressure to provide a quick fx. Reactivity saps the teams
productivity, as triage is no longer based on strategic improvement, but
rather on the seat-of-the-pants, If it bleeds, it leads approach to daily
issues; the team never seems to get ahead of problems. Possibly,
equipment and resources purchased for testing of new solutions are
now being repurposed into the production environment, resulting in
accounting nightmares.
In previous posts, we have discussed the value of stress testing and a
comprehensive reporting system in fnding and reducing bottlenecks,
and in tuning application performance. This testing can have a strategic
purpose as well, as we can use it to determine triggering criteria for
expansion or upgrade of the system. By proactively testing capacity
limits on the system, you can accurately predict the thresholds for your
applications.
Stress testing allows you to feed what if situations like the following to
an application:
What if we experience 300% growth instead of our planned
200% growth?
What if interest in our site goes international?
We are adding another 2 steps to the checkout process, what
will it do to our overall session times?
All of these situations can be modeled with a robust load testing tool,
like SmartBears LoadUIWeb Pro, which allows instances of the testing
platform to be deployed on remote servers inside your network or in the
cloud for testing of your site from anywhere in the world. By allowing
this fexibility, you can test a situation virtually identical to what the
customer will experience on their own browsers in production. Further,
How are improvements driven on your system, are they antici-
pated, planned and implemented based on metrics, or are they
frequently deployed in a particular response to an outage or
customer gripe?
15
LoadUIWeb Pro
by SMARTBEAR
by knowing at what stage the new system will outgrow current capacity,
infrastructure upgrades can be intelligently scheduled to avoid busy
seasons for the application.
Of course, load testing can, and should, also be used as an internal
quality assurance test when system improvements are made. As we
have outlined, load test reporting can provide a baseline if run at regular
intervals. By running a short load test directly after a maintenance
interval has concluded, we can immediately gauge the integrity of the
system as it returns to service, using the user scenarios to fully test the
application and assuring that all features are in a ready status for real
customer traffc. If a key portion of the application is still down, it is far
less impactful to discover it internally than to have a customer open
a trouble ticket regarding a failed transaction. Running an internal
load test can also be benefcial at this time so we can quantitatively
compare the results of the test against the baseline, giving us immediate
knowledge of the performance changes we just experienced.
There are defnitely times when changes on the system must be made
on the fy in response to an outage. However, a system that has a level
of change control in place has a greater chance of stability over the long
term.
The true benchmark of Performance Management success is the
ability to set a Service Level Agreement with your customer. An SLA
not only defnes a contracted level of service quality, but it represents
the confdence that both sides have in their solution, and the value
they place on partnership. To be willing to offer an SLA, you must be
confdent enough in your preparations:
How comprehensive a Service Level Agreement are you willing to
offer your customers? On what would you base your numbers, and
how would you argue them when your customer says you have not
achieved them?
16
LoadUIWeb Pro
by SMARTBEAR
You have suffciently hardened your website through analysis of
the components and infrastructure. The application has been
reviewed down to the object level, and you have optimized your
application through redundancy and load balancing. You have
the baseline data to prove it as well.
You have performed stress testing on your system, and have
documented the warning signs of a network in a stressed
condition. You have shared your fndings with your team and
coached them on these warning signs, and they know the steps
to take when these signs appear. Further, you know when
your busy times are on the website, and have eliminated the
extraneous activity on the network at those peak times to guard
against contention.
Your infrastructure has been right-sized for the traffc you are
expecting to see in the next year. Upgrades to the infrastructure
have already been planned out for that time down the road
when you foresee you will expand. Further, you have a plan for
analysis on the new application code when Development says it
is ready for pre-deployment testing, and your Development team
knows the standards to which you will be holding their work.
You have baselines of the system performance over time and
you can determine by week, month and quarter your averages
for response times, based on data from multiple locations inside
your network, and if appropriate, from outside your network as
well.
What are the key metrics, the issues most important to your customer?
Though there will be some of these metrics that will fall outside of
Performance Management infuence (such as metrics pertaining to
Service Desk ticket performance), the data provided from regular load
testing can be used as a basis for creating a very workable SLA.
17
LoadUIWeb Pro
by SMARTBEAR
We have illustrated how load testing can positively impact each of the
fve sectors of Performance Management, and offered suggestions
on with ways you can raise your performance score by making your
application stronger. The key take-away offered here is that load
testing is not a singular event to be done only once during a software
development cycle. Rather, it is a continuous process that is constantly
employed to fnd bottlenecks on our applications and infrastructure.
Software applications and the networks they run on are constantly
improving and evolving, and we should never lose sight of the value of a
regular performance check-up on them to keep them in top form.
Summary
About SmartBear Software
More than one million developers, testers and operations professionals
use SmartBear tools to ensure the quality and performance of their
APIs, desktop, mobile, Web and cloud-based applications. SmartBear
products are easy to use and deploy, are affordable and available for
trial at the website. Learn more about the companys award-winning
tools or join the active user community at http://www.smartbear.com, on
Facebook or follow us on Twitter @smartbear and Google+.
SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915
+1 978.236.7900 www.smartbear.com
2012 by SmartBear Software, Inc. Specifcations subject to change. SB-AQt-030513-WEB