PEAK PERFORMANCE

Running your first load test with JMeter
Michael Kelly 11.24.2009 Rating: -4.00- (out of 5)
Software quality news and advice Digg This! StumbleUpon Del.icio.us

In this tip, we'll take a look at running your first load test with JMeter. JMeter is a popular open source Java application used for load testing. It supports a variety of technologies like, HTTP/HTTPS, SOAP, JDBC, LDAP, JMS. These are all commonly used by teams looking to introduce load on a server, network, or even specific objects. While it's not as full featured as some commercial tools, it's a great tool with a lot of firepower for it's price -- free.
Mike Kelly

Editor's note: This article was written using JMeter 2.3.4 running on Java 1.5.0_20. The application used in this article is Yahoo Search since it's a publically available application. The example uses a very low load to simply illustrate how JMeter works. If you would like to test with larger loads, do not use Yahoo. Instead, please use your own applications.

We won't be looking at getting JMeter setup and running in this article. That's not because it's difficult - typically all I have to do to get it setup is download and double-click to fire it up. It's because operating systems and versions change, so the best place to get setup advice is on the "Getting Started" section of the JMeter user manual on the JMeter website. First some terminology JMeter tests are built around the idea of a test plan. Within a test plan you can have Thread Groups, controllers, listeners, timers, assertions, and other elements. Each test plan is a performance test scenario - it's the series of steps JMeter will execute when you run the plan. The following table gives a brief overview of the basic elements you can include in a

test plan. Element Thread Group Description As with any load test, execution is multi-threaded. The thread group element is what controls concurrent connection to your application. In end user terms, it's your group of users. A sampler is the basic type of controller. Very simply, samplers tell JMeter to send requests (HTTP, SOAP, etc…) to a server. There are also Logic Controllers, but we won't be using one in this tip. Listeners are what you use to access to the information JMeter gathers while running. They give you the pretty charts Timers are how you set delays in JMeter. Timers fire before each request that a thread makes. Assertions, like in any test tool, allow you to check for specific behavior as your test executes. Assertions provide the standard pass/fail results.

Samplers

Listeners Timers Assertions

For full details on these and other elements, please read the Elements of a Test Plan section of the JMeter user manual. However, this short list is enough for us to get a simple test running. If you have troubles with any particular element when trying to get it to work with your own application, the Component Referencesection of the JMeter user manual is awesome for understanding each element and it's settings. Building your first test

When you first open JMeter, you should be greeted with an empty Test Plan. To add elements to a Test Plan, you can either right-click on it, or with it highlighted select Edit -> Add from the menu. The option of either right-clicking to add elements or using the menu to add elements will be true for all the items we talk about in this article. To reduce confusion, I'll be referencing object by asking you to right-click on them. The first thing we need to get a test running is a Thread Group. To add a Thread Group, right-click on the Test Plan and select Add -> Thread Group. For our example, we don't want to run with a lot of load. Yahoo won't like that. So we're only going to test with 10 threads - or users. In addition, we'll set the ramp-up period to 2 seconds, and we'll have each user search ten times. The only reason I'm having them loop ten times is so the charts you see at the end will actually contain some data in them. Once we have our Thread Group configured, we're ready to add a timer. We'll be using a Constant Timer between transactions to space them apart. To add a timer, right-click on the Thread Group and select Add -> Timer -> Constant Timer. Set the thread delay to three seconds - or 3000 milliseconds. At this point we're ready for our HTTP request. To add a request, right-click on the Thread Group again and this time select Add -> Sampler -> HTTP Request. For the example, I want to have you run a simple search on Yahoo. To do that, set the Server Name to 'www.search.yahoo.com'. That tells JMeter what server you're hitting. Then for the path, I want you to enter '/search? p=testing'. That's the same as if you went to Yahoo's homepage and did a search on the word 'testing'. You can leave everything else on the page set to the defaults. When I'm testing, I always want to know that what I did had the desired effect. So we're going to add an assertion to our test. To do that, right-click on the HTTP Request element and select Add -> Assertion -> Response Assertion.

There are a number of options here that might be confusing. We're not going to look at them in detail, but what we want to do is test to make sure we're getting search results back in our response. To do that, we're going to test the 'Main sample only' and we're going to be looking at the 'Text Response.' Looking at the text response means we're looking at the HTML coming back. Because we want to make sure it has results, we're doing a 'Contains' pattern match on the response. At the time I wrote this article, the top response to the search was the Wikipedia entry for Software Testing. I simply searched for the starting text to that definition which was 'Software Testing is an empirical investigation'. You'll notice that there's a bold HTML tag in the check. That's because that's what you'll see if you do a view source

on the page. Always check for what you see in the source code - not what you see on the screen. Next, we're going to add a couple of listeners so we can see our results. I like the Graph Results and View Results as Table listeners. They both offer intuitive interfaces. To add a Graph Results listener, right-click on the Thread Group and select Add -> Listener -> Graph Results. For this example, you might want to uncheck the Deviation and Throughput checkboxes. Next, add the View Results in Table listener. Right-click on the Test Group and select Add -> Listener -> View Results in Table. You now have a really simple test ready to go. Go ahead and save your Test Plan. Running your test Once you've saved your test, you're ready to run. From the menu, select Run -> Start. It's very possible you won't notice anything happen. Many first time users of JMeter don't. Here are some tips. First, when you don't have a test running, you won't see any active users. In the upper right-hand corner there's a user activity indicator. If you're not running a test, it looks like this: When the test is running, this indicator changes to show you how many of your users are active, like in the following

figure: Second, you can select your listeners to see what's happening during your tests. This information is updated realtime. This is also how you view your results after your test is complete. In the figure below, I've selected the Graph Results view after my test has executed: You can see that in the Graph Results, it's charted each of the individual data points during the test (that's all those black dots scattered around) along with two lines showing the median and average response times. This is a hand chart to notice trends and odd behaviors.

Alternatively, the View Results in Table listener shown below is a more structured view of the data: With this listener you can see stats for each data point, along with a graphic representation of the status of the response. If you wanted to export data, this is likely one of the listeners you'd use to save off your results to import into another tool like Excel.

Recording and running software load tests with JMeter
Mike Kelly 11.21.2009 Rating: --- (out of 5)
Software quality news and advice Digg This! StumbleUpon Del.icio.us

In this tip we will take a look at recording performance tests using JMeter. In a previous tip on "Running your first load test with JMeter, we looked at manually adding a Sampler for Yahoo Search. However, you may find that for more complicated test scenarios, you'll want the ability to record tests. This will help you save time when creating Samplers for your test plans. One easy way is to create multiple Samplers and record with them simultaneously.
Mike Kelly

Author's note: This tip was written using JMeter 2.3.4 running on Java 1.5.0_20. The version of FireFox used in 3.0.14. The application used in this article is Yahoo Search since it's a publically available application. The example uses a very low load to simply illustrate how JMeter works. If you would like to test with larger loads, do not use Yahoo. Instead, please use your own application.

Setting up a JMeter proxy If you're not familiar with setting up a proxy, a quick explanation might be in order. A proxy simply acts as an intermediary for client requests to the various servers from which you've requested resources. By setting up a JMeter proxy, you're allowing JMeter to record the requests made and use the information derived to create a test plan. If you're following along with this example, you'll want to start with a new test plan. To do this, add a proxy by rightclicking on WorkBench. Next click Add -> Non-Test Elements -> HTTP Proxy Server (as shown in figure 1 below).

Once the HTTP Proxy Server is added, select it in the left navigation tree, change the port from "8080" to "9090" and ensure the checkbox labeled "Retrieve All Embedded Resources from the HTML Files" is checked. Your settings should look similar to those shown in figure 2 below.

Next you'll need to set up your browser to point to a proxy server. For FireFox, you can find these settings by opening Preferences, selecting the Advanced icon, then the Network tab, and clicking the "Settings…" button. This should open a proxy configuration screen similar to the one shown in figure 3 below.

Select the "Manual proxy configuration" radio button and enter an HTTP Proxy of localhost on port 9090. When finished, click OK and close FireFox preferences. ((Content component not found.)) Recording information using the proxy You should now be set up to record a test using JMeter. To start recording your tests, you'll want to start the JMeter proxy. To do this, navigate to the HTTP Proxy Server in JMeter and click the Start button at the bottom of the screen. This will start up the server. At this point, JMeter will record anything you do in the browser you configured. Sometimes it's helpful to have two browsers opened. I'll often record tests using FireFox, and then also have an Internet Explorer or Safari browser open on the side so I can look stuff up without recording the transactions in JMeter. For this tip's example, we'll be testing Yahoo Search. Open a FireFox browser and navigate to http://search.yahoo.com/. Once that page loads, search for "software testing." When the search results are returned, navigate back to JMeter and take a look at what's been recorded. As shown in figure 4 below, you should see a number of recorded Samplers under the WorkBench.

This should be enough for our example, so let's stop the proxy server. To do this, navigate to the HTTP Proxy Server in JMeter and click the Stop button at the bottom of the screen. You're no longer recording. If you're completely done, you'll also want to reset the proxy settings in your browser back to what they were before you set them to the manual configuration. Making sense of what was recorded

Previous JMeter tips:
Running your first load test with JMeter

JMeter is a highly recommended tool for load testing open source Java applications, as it supports technologies from HTTPS SOAP. Learn how run your first JMeter test in this tip.
Recording and running software load tests with JMeter

As you look at the list of recorded HTTP requests, you'll want to figure out what you're looking at. At the end of each item name, you'll see a file extension indicating what's being requested. For example, the first few files from Yahoo included a couple of javascript files (.js), a cascading style sheet (.css), and a portable network graphic (.png). These files give you the basic search functionality for the initial page we loaded. After that, you'll see a number of "gossip-us-sayt" requests. The purposes of those requests might not be immediately obvious, but if you dig in just a bit by selecting one of them, you'll see that the server name these requests go to is " sugg.us.search.yahoo.net". That's the server that provides the spelling suggestions while you're typing (as shown below in figure 5).

Recording JMeter load tests can simplify the creation of Samplers needed for your test plans. This tip explains how to record JMeter test results and analyze the data generated.

After that series of requests, you'll find more javascript images and actual search results coming back. Those files represent the search results page. At this point, you can move each of these elements from the WorkBench into your Test Plan. You can take all the Samplers recorded or you can select individual Samplers to move up. This allows you to record a series of Samplers, and then mix and match them as needed to create custom scenarios. As a quick example, add a Thread Group to your test plan and move the first six recorded Samplers up under that Thread Group. Once those have been added, add a View Results Tree Listener as well (as shown in figure 6).

You can now run this test. If you do so (just run one thread for one loop), you'll see six responses come back successfully in the View Results Tree Listener. If navigate those results and look at the response data, you should find the content required to load the Yahoo Search page. The easiest example to recognize is the png file in the third result set shown in figure 7 below.

At this point, you can now customize those Samplers as needed to create the test scenarios necessary to generate your target load.

SOFTWARE TESTING

Tips for debugging your JMeter tests
Mike Kelly 11.19.2009 Rating: --- (out of 5)
Software quality news and advice Digg This! StumbleUpon Del.icio.us

In this tip we'll take a look at some techniques for debugging your JMeter tests. In a previous tip on "Running your first load test with JMeter," we looked at getting JMeter setup and we ran a simple test against Yahoo Search. If you're new to JMeter, you'll quickly discover that once you get past very simple HTTP requests you'll need to be able to get visibility into what's happening at runtime so you can debug and tune your tests. For the examples we'll look at in this article, we'll be using the Yahoo Example Test Plan we created in the first tip in this series. Use listeners to see what's happening during execution As we saw in the previous tip when looking at the Graph Results and View Results listeners, listeners are what you use to access to the information JMeter gathers while running. There are a number of listeners available, and while in the previous article we looked at two listeners that are handy for results reporting, in this article we'll be focused on a couple of listeners more appropriate for debugging. While different listeners are useful in different situations, a common listener that I use when debugging a JMeter test is the Assertion Results listener. The Assertion Results listener logs assertion requests and reports failures of any Assertions within your test plan. When I'm creating a Test Plan, I try to add an assertion to each step for just this reason. Assertions are critical to the debugging process, and even if you take them out later, they let you know your test is doing exactly what you expected it to do when you were creating

Editor's note: This tip was written using JMeter 2.3.4 running on Java 1.5.0_20. The application used in this article is Yahoo Search since it's a publically available application. The example uses a very low load to simply illustrate how JMeter works. If you would like to test with larger loads, do not use Yahoo. Instead, please use your own application.
it.

For example, in the following figure, you can see successful assertions added to the Yahoo Example.

Each HTTP Request log message shown above represents a successful assertion from my Test Plan. In the figure below, you can see what an error looks like.

In this case, JMeter shows not on the HTTP Request, which indicates an assertion was attempted, but also shows that the assertion failed and why it failed. Using the checkboxes at the top of the screen, you can filter this to display only errors if you like. Using the View Results Tree listener There's one particular listener that's a bit more powerful than the others when it comes to debugging: the View Results Tree listener. The View Results Tree shows a tree of all sample responses, allowing you to drill down into the response for any sample. Example data provided for HTTP requests includes thread runtime information, response headers, the request and response data, and response assertion results if there was an assertion associated with the request. The following two figures show examples of response data shown in text and rendered HTML format for the Yahoo Example Test Plan:

From this listener you can not only see the details of the request sent, but you can see all the details of the response that came back. This is important, because if your assertions are off by as much as a single space they will be wrong. Many times I end up going to the text response data and copying the results there that I'm looking for. In addition, if you're testing web services and need to do XPath queries, the rendered XML view of the response data is quite helpful. Checkout the JMeter Component Reference for a bit more detail on some of the subtle nuances of using the View Results Tree. Using Debug Sampler and Debug PostProcessor JMeter has two built in debugging elements: Debug Sampler and Debug PostProcessor. Think of these two elements as helpers for providing more information in the View Results Tree listener. The Debug Sampler generates a sample containing the values of all the JMeter variables and properties at runtime. If you use the Debug Sampler in conjunction with a View Results Tree listener, you'll be able to navigate those values as shown below.

The Debug PostProcessor on the other hand, creates a "subSample" with the details of the previous sampler properties. The Debug PostProcessor shows additional details about the request performed as shown in the figure

below: Both of these can be handy for figuring out what's happening under load. I know that when I'm looking at the "standard" results shown in the View Results Tree listener, I'm often focused on debugging a single transaction. However, when I switch to looking at the data in the Debug Sampler and Debug PostProcessor, I'm often trying to figure out what's happening under load. Save your response data to a file By default, listeners display information graphically in the JMeter user interface. This is a low-information density medium, and one you'll quickly need to move past to get at the details of your test results. One of the first things you'll need to figure out when using JMeter is how to write to files. As you can see in the two figures above, listeners in JMeter have a "Write results to file" field. If you provide a path to file, you can capture detailed information in either XML or CSV format. To configure what JMeter logs for a listener, click on the Configure button next to the file name.

If you save your results in XML, the output will look similar to the following:

By contrast, if you're looking for something that you can easily manipulate in Excel, the CSV format shown below might be more appropriate:

Each format has it's own advantages, but I find myself using CSV more than XML. Each listener saves it's own configuration settings separately. So be careful you don't assume if you change the settings for one listener that it will cascade to all of them. It won't. This allows you to tune different listeners to collect different information. Be sure your load generator has enough resources Another common issue when load testing is figuring out if your load generators have enough resources for the testing required. If your load generation machine(s) can't adequately accommodate the demands your JMeter test is putting on them, you'll skew your execution results.

Previous JMeter tips:
Running your first load test with JMeter

JMeter is a highly recommended tool for load testing open source Java applications, as it supports technologies from HTTPS SOAP. Learn how run your first JMeter test in this tip.

One place to start is to monitor your load generators while your tests are running. If you're using Windows, this likely means just running Perfmon. If you're on another platform, use the Perfmon equivalent for that operating system - they all have one. If you start to notice performance issues on the load generator, here are some things to look at:

Threading: The more threads your load test uses, the harder your load generator will work. This is likely the biggest factor to performance. If you need large-scale load testing, consider running your test across multiple machines. You know your load generator is over burdened when you're pinging the CPU or when you're pushing the upper limits of your memory capacity. Recording and running software • The GUI: JMeter has a "headless" mode that allows you to run load tests with JMeter without the GUI. Rendering all those pretty charts taxes the system. Recording JMeter load tests can To run without the GUI, use jmeter -n -t test.jmx test.jtl from the simplify the creation of command line. Samplers needed for your test • The number of Listeners: Just like the GUI, it takes processing plans. This tip explains how to power to populate listeners. Even if you run without the GUI, you're record JMeter test results and still collection listener data. This is the main reason I consider the analyze the data generated. View Results Tree and Assertion Response listeners debug tools instead of just something you'd always use. That have overhead. You can either remove them from your test, or if you're running from the command line, add a '-l' argument before the test plan name. Next steps In future tips, we'll look a bit more at some other ways you can use JMeter in your load testing. These tips will include recording tests and tips for making sense of all the results. In the meantime, for more on JMeter, you can always checkout the project website at jakarta.apache.org.

Sign up to vote on this title
UsefulNot useful