FUNDAMENTALS OF LOAD TESTING

Introduction: Load testing is required at every stage in the life cycle of an application, in particular web applications. Load tests help identify performance problems during system design and development stages. For existing applications, the simulation becomes an aid to maintain, finetune and forecast future system needs. Abstract: In simple context, Load testing is the simulation of real world loads on computer systems. A load test is the one that takes into consideration the various # of users that execute tasks on the system and the various types of software on users machines, including processes, browsers, connection speeds etc. So, before going into production, load testing is the only way to evaluate how the system will perform in the real world. There are various ways to proceed with a Load test and in general load testing is to identify and analyze system load requirements, define a site map, identify features to load test, define user sets, record tests with single user, randomize data, select and adjust work load - virtual users (user sets) and load agents, run test and analyze test. This write up will give you an overview or fundamentals of Load testing and will not cover any advanced concepts of load testing. It covers Load testing details of a web application, Benefits of load testing, Load testing environment, Load test types & variations, Basic work load models, Quick note on how to select a work load model, Load testing goals and Results and list of wellknown Load testing tools. A Load Test by definition: Emulates real-life interaction between users and server systems Creates a model of what a typical user does within the application and measures the performance and reliability of the server Monitors and quantifies the performance of an internet application under expected and peak user conditions Measures the stability, responsiveness and throughput of the computer system in a realistic test environment Generates different types of traffic expected, in the volumes expected, using actual computer hardware and software Brings the entire functionality of the e-business web site under scrutiny in order to identify the general areas of possible bottleneck occurrences Measures the performance of the web servers and/or the firewalls Allows testers to draw useful conclusions from the test results for improving the site's components Finally, it is entirely different from functional and regression testing. Functional testing determines whether the system operates as expected in general - a single user mode and Regression testing determines whether an upgraded version operates as expected, but load testing determines whether system operates as expected when used by 100, 200, or a 1000 or more simultaneous users.

Benefits of Load Testing: Minimize the risk of deploying systems that don't meet quality and performance requirements Find the root cause of performance bottlenecks and system failures with ease Optimize user experience by testing against service levels to ensure that Service Level Agreements are met in production Reduce the costs of defects by testing remote components early in the development cycle Minimize hardware and software costs by accurately predicting system capacity Provides performance and capacity statistics Tests scalability of the site Allows you to evaluate the bandwidth for the amount of data sent and received Allows you to verify predicted performance under varying conditions Verifies the tuning of your system through benchmarking

Typical Load Testing Environment: Load testing takes place in a client/server environment. The attributes of the client and server systems will impact your load test design. The basic components of a load testing environment are 1) Users are individuals who use an application 2) Client is a combination of hardware and software used by the user (browsers, network, connections, etc) 3) Server refers to system that provide service requests to clients application servers, database servers The below diagram shows a load-testing environment for a typical web application and a number of driver machines, connected together through a local and/or wide area network. Three conditions must be met before load testing can begin: 1) the load-testing environment the Web application to be tested, the network, and the load-driving machines needs to be adequately equipped 2) the virtual users must be set up to perform tasks in the same way as real users 3) the Web browser emulation used must be as accurate as possible

Preparation before Load Tests: First you need to analyze the application under test and determine the factors which affect the application performance and mark them as your Key features or attributes to measure like Server side CPU utilization, memory usage, process time, database server/queries and Client side similar Health monitoring data, network, user profiles and most importantly application data and transaction points. Apart from the above and a well-defined Load testing methodology/process has to be determined to get satisfactory load test results i.e., identify and analyze system load requirements, define a site map*, identify features to load test, define user Action sets* (test scenarios), identify user groups, user profiles, form user types, record tests/script load test for single user, randomize data, form user types* - user profiles* vs. user groups*, adjust work load select work load model and choose # of users, run test and analyze test. *Site map A navigational flow chart/map which identifies all of the possible paths that a user can navigate within an application. *Action sets defines the actions that typical users perform when using the application, these can be scripted as functions and best used for Isolation type of load tests *User profile users distinguished or categorized by the software or hardware configuration they use like users with IE browser with Modem connection, users with Netscape navigator with High speed broadband connections etc. Basically they are Browsers type vs. Browser version vs. Network connection. *User group These are combined with action sets to form *User types are formed when User profiles and User groups are multiplied (permutations n combinations of both) You can pin-point the exact bottleneck of an Application under load test by choosing the right combination of a User type and Action set. NOTE: However predicting sever-side errors is quite tricky, it needs reference to wide range of result reports to pin-point the exact bottleneck like network behavior reports, database (if any) server behavior, behavior of each network component like proxy, firewall, switches, router etc. So, Load testing methodology can be summarized as below: Derive Gather statistics to get the feature list most accessed by customers Identify load test scenarios Generate load test plan for each test scenario Generate load test script for test plan derived Try script for single virtual user Customize session variables and randomized data variables Prepare required input Test data files Run script with required workload configuration Analyze load test reports generated

Basic Types of Load Testing: To make an accurate diagnosis of a performance problem in an e-business application, it is usually necessary to run a sequence of tests that progressively focus in on the particular problem. 1. End-to-end Testing Is generally the first type of testing executed Consists of a global examination of the entire Web application Is designed to bring the entire functionality of the Web application under scrutiny in order to identify the general areas where bottlenecks are Can use a variety of load test types and workload types depending on the context most often, stress tests or stability tests, deploying constant or increasing numbers of virtual users are used 2. Module Testing Typically follows the end-to-end testing Separately tests specific modules of an application to accurately pinpoint any problems Generally uses isolation tests that are designed to bombard the suspect module with loads large enough to force repeated occurrences of a particular problem Generate very specific and detailed information, allowing the tester to accurately diagnose the problem Load Test Variations: Load tests are designed in different ways, depending on what they are required to measure. The most frequently used load tests are stress tests, stability tests, and isolation tests. 1) Stress Test Determines how much load a system can handle Determines the maximum number of requests a server can manage Generates a large and continuous load to stress the system to its limit Minimizes the time between transactions Ignores think times May be set using the profile or by running a try script 2) Stability Test Determines whether a system will remain stable over a long period of time May last 24 hours, a week or even longer Mimics the real traffic a system will encounter Determines whether response time will remain stable over a long period of time Determines whether resource consumption will remain stable over a long period of time (i.e. memory leaks) Helps identify other technical faults that may compromise the stability of the system 3) Isolation Test Helps identify bottlenecks in specific areas of the application To isolate a troublesome functional area & find flaws by repeated load testing Often executes specific sets of transactions over and over again

Load Testing Basic Work Load Models: A work load model In general any load testing tool will provide the following work load models. Increasing - An increasing workload helps testers to find the limit of a Web applications work capacity, and is thus crucial for establishing fail-safe conditions. This workload model is especially useful when you want to find out at which load level your system crashes or does not respond within acceptable response times

Steady State - In this model, the same number of virtual users is employed throughout the test, and there is no delay between transactions. This workload model is especially useful when you want to find out about the behavior of your tested system at a specific load level. The use of steady-state workload provides exact response time and throughput measurements for a given number of users. This type of workload is often employed for stress and stability tests

All Day - You can assign different numbers of virtual users to any interval of the load test. Each user type can use a different load distribution. Therefore, you can design very complex workload scenarios such as workday workloads or even weekly workloads. As in the dynamic workload model, you can adjust the load level during a load test. This workload model is especially useful when you want to model complex, long lasting workload scenarios in the most realistic way possible

Queuing - In this model, transactions are scheduled following a prescribed arrival rate. The load test will finish when all of the virtual users have completed their prescribed tasks. Note that the test may take longer than the simulation time that was specified, because of the randomized arrival rates.

Dynamic - In this model, you can manually change the number of virtual users in the test while it is being run. The maximum number of virtual users to be run is set; within this limit; the number can be increased or decreased at any time during the test. No simulation time is specified, and you must bring the test to an end manually. This workload model is especially useful when you want to experiment with different load levels and to have the control over the load level during a load test. The graph cant be provided since load is manually controlled.

How to identify the workload model & Best practice: Review your scenario question. Usually, the language in the question helps you determine which workload model to use. See the table below: Language in Scenario/ Question Support Stable, Memory Leaks Respond to morning/afternoon spikes Response when 1000 users perform the same action simultaneously Workload Model Increasing Stead state All Day Either Stead state with the wait for function or Queuing.

Best practice: When workload requirements are not given as part of your product specifications, go blindly with a increasing workload model when you know it will be used by multiple and simultaneous users. So, use 10 % users as initial user with 10% user increment at 10% of total test time as the increasing period.

Load Testing Goals & Results: The goal of any load test should be clearly understood and documented. A load test usually fits into one of the following categories: 1. Quantification of risk - Determine, through formal testing, the likelihood that system performance will meet the formal stated performance expectations of stakeholders, such as response time requirements under given levels of load. This is a traditional Quality Assurance (QA) type test. Note that load testing does not mitigate risk directly, but through identification and quantification of risk, presents tuning opportunities and an impetus for remediation that will mitigate risk. 2. Determination of minimum configuration - Determine, through formal testing, the minimum configuration that will allow the system to meet the formal stated performance expectations of stakeholders - so that extraneous hardware, software and the associated cost of ownership can be minimized. This is a Business Technology Optimization (BTO) type test

To speak in load testing tool terms, load testing is to Determine each Transaction* Response time* under specific load Determine each user Session time* under specific load Determine Transactions/sec* within Load test simulation time Determine Errors/sec* within Load test simulation time Determine application overall Throughput* during Load test simulation time

Finally, analyzing all the above Load test results under various Loads and determine how much load (maximum # of users) your Application can withstand without its performance getting degraded. Depending on these results, you can plan how your production environment should like and scale it up whenever needed. *Transaction a unique business query/event matching application requirements *Response time it is the time taken from a user initiating a request to the instant at which the first part of the response is received at by the application. *Session time it is the time taken by the application for each user to execute all transactions specified under load test scenario *Transactions/sec generally rate of no. of successful transactions per sec. Might differ for each load test tool, for few it is categorized as TransacationOK/sec which means successful transactions and Transactions/sec might mean successful and failed transaction rate. *Errors/sec No. of errors load test generated during simulation time. It need not be only transaction failures but can be actual errors application has generated (bugs), script errors mismatched with actual result, network issues etc. *Throughput it is the overall transaction time taken from a user initiating a request to the instant at which the last part of the response is received at by the application

Major Load Testing Tools (in the 93% of Market Share) HPs (earlier MERCURY INTERACTIVE) - LoadRunner [54%] IBM RATIONALs Performance Tester [11%] COMPUWAREs QA load [9%] BORLANDs (earlier SEGUE) - Silk Performer - 7% EMPIRIXs e-TEST suite [6%] RADVIEWs WebLoad [3%]

Conclusion: Load testing is the only way and plays an important role in analyzing the system performance for a specific load (no. of simultaneous users). Before an application is planned to go public typically a client/server application where the no. of users accessing it are larger in number, unpredictable and random, there is a desperate need to perform load testing and define the maximum load/simultaneous users it can sustain without its performance getting degraded. Once this performance measured is defined as a requirement, you can definitely plan to scale up your production environment to meet whatever additional load of users that keep bombarding your application. References: Fundamentals of Load testing Search links from Google SilkPerformer Modeling & Implementing Load tests student guide (Segue)