A Practical Guide to Usability Testing Joseph S. Dumas Janice C. Redish intellect Exeter, ENGLAND Port.anp OR, USA2 Introducing Usability Testing ‘What is Usability Testing? ‘The goal isto improve the usability of a product. ‘The participants represent real users... . ‘The participants do real tasks. ‘Observe and record what the participants do and say. Analyze the data, diagnose the real problems, and recommend changes to fix those problems... ‘The results are used to change the product—and the process. What is Not Required for a Usability Test? ‘When is a Usability Test Appropriate? ‘Questions that Remain in Defining Usability Testing ‘esting Applies to All Types of Products... Testing All Types of interfaces... Testing All Pats of the Product. ‘Testing Diflerent Aspects of the Documentation. Testing with Different Techniques. CordiscoVEFY en Active intervention... ‘Additional Benefits of Usability Testing. ‘Changing people's atitudes about users. ‘Changing the design and development proces... Comparing Usability Testing to Beta Testing... Comparing Usability Testing to Research Studies. Focusing on different goals Using the same laboratory ‘Selecting participants different Controlling fewer variables. ‘Weighing observations more Fs ‘Analyzing and reporting data without inferential statistics.22+ A Practical Gulde to Usability Testing ‘We begin this chapter by defining usability testing, examining both ‘what is and isnot required fora usability test and reiterating the {important point that you should test early and often—not just once’ ‘atthe end of the development process. We then look at how our definition of usability testing applies in a wide variety of situations. We also add two more benefit tothe list we began in Chapter I; these two are benefits that come more from usability testing than from other techniques in the usability engineering approach. Finaly, We end the chapter by comparing usabilty testing with two quite diferent techniques with which itis often confused: beta (or field) testing and research studies What is Usability Testing? ‘While there can be wide variations in where and how you conduct a ‘usability test, every usability test shares these five characteristics: 1. ‘The primary goal is to improve the usability ofa product, For each test, you also have more specific goals and ‘concerns that you articulate when planning the test. ‘The participants represent real users. ‘The participants do real tasks. ‘You observe and record what participants do and say. You analyze the data, diagnose the real problems, and recommend changes to fix those problems. sees The Goal is to Improve the Usability of a Product ‘The primary goal of a usability testis to improve the usability ofthe. "product thats being tested. Another goa, as we will discus in detail” later isto improve the process by which products are designed and — ‘developed, s0 that you avoid having the same problems again in” -other products: This characteristic distinguishes a usablity tes from a research study, in which the ga! isto investigate the existence of some ‘phenomenon. Although the same facility might be used for both, they have different purposes. See the section, “Comparing Usability Tests ‘to Research Studies,” ater inthis chapter.) This characteristic also distinguishes a usability test from a quality assurance or function fest, which has a goal of assessing whether the product works according to its specifications ‘Within the general goal of improving the product, you will have ‘more specific goals and concerns that differ from one test to another. ‘+ You might be particularly concerned about how easy itis for users to navigate through the menus. You could test that Introducing Usbity Testing + 23 ‘concer before coding the product, by creating am interac- tive prototype of the menus, or by giving users paper ‘versions of each screen. (See Chapter 5 for more on these types of tests.) ‘+ You might be particularly concerned about whether the interface that you have developed for novice users will also be easy for and acceptable to experienced users. ‘+ For one test, you might be concerned about how easily the customer representatives who do installations will be able to Install the product. For another test, you might be concerned about how easily the client's nontechnical staff will be able to operate and maintain the product. ‘These more specific goals and concerns help determine which users are appropriate participants for each test and which tasks are appropriate to have them do during the test. (See Chapter 8, “Defining Your Goals and Concerns,” for more on this topic) The Participants Represent Real Users ‘The people who come to test the product must be members of the group of people who now use oF who will use the product. A test that uses programmers when the product is intended for legal secretaries isnot a usability test. ‘The quality assurance people who conduct function tests may also {ind usablty problems, andthe problems they find should not ben Pay ay ‘ignored, but they are not conducting a usability test They arenot So 'nogrepresat ‘eal users—unless itis a product about function testing, They are the real aera you! ‘acting more like expert reviewers. ‘are not seeing Ii the participants are more experienced than actual users, you what will happen ‘may miss problems that wil cause the product to fail in the ‘when the product ‘marketplace. Ifthe participants are ess experienced than actual {ets othe real users, you may be led to make changes that aren't improvements for Users the real users (See Chapter 9, "Deciding Who Should Be Participants,” and Chapter 10, “Recruiting Participants,” for more about making sure thatthe people who come to your lest represent the users.) The Participants Do Real Tasks The tasks that you have users do in the tet must be ones that they ‘will do with the product on their jobs or in their homes. Txis means that you have to understand users jobs andthe tasks for which this product is relevant. “In many usability tess, particularly of functionally rienand complex software products, you can ony test some ofthe many tasks» “that users willbe abe todo withthe produet: In addition to being24+ A Practical Guide to Usblty Testing realistic and relevant for users, the tasks that you include ina test should relate to your goals and concerns and have a high probability | of uncovering a usability problem. (Chapter 11, "Selecting and Orga nizing Tasks to Test,” and Chapter 12, “Creating Task Scenarios,” ve more information about what to have users doin atest.) Observe and Record What the Participants Do and Say ‘na usability test, you usualy have several people come, one ata time to work wit the product. You observe the participant, “recording both performance and comments. ‘opinions about the product: A» ‘You also ask the participant for _ usability test includes both times when participants are doing tasks _ with the product and times when they are filling out questionnaires _ “about the product (For more on deciding what to observe and record ‘while users are working withthe product, see Chapter 19, "Deciding How to Measure Usability” For more on questionaies, see Chapter 14,"Preparing Test Materials) Observing and recording individual participant's behaviors distinguishes a usability test from focus groups, surveys, and beta testing A typical focusigroupiis a discussion among 8 to 10 real users, led by professional moderator. Focus groups provide information sbout ‘users opinion, attitudes, preferences, and their se-eport about their performance, but focus groups donot usually let you see how users, actualy behave with the product (We dacs focus groups in more detail in Chapter 3, “Uncovering Usability Needs Before You Design.”) ‘Surveys)iby telephone or mail, let you collect information about users’ opinions, attitudes, preferences, and their self-report of behavior, but you cannot use a survey to observe and record what ers actualy do witha product A typical betaltest (field test, clinical trial, user acceptance test) is an early release ofa product to afew users. Abeta test has ecological validity, thats, real people are using the product in eal environments todo real tasks. However, beta testing seldom yields any useful information about usablty. Most companies have found beta testing to be too lite, too unsystematic, and mach foo late to be the primary test of usability. (Because some people stil believe that a beta test substitutes fora usability engineering approach and for usability testing during the proces, we discuss beta testing in some etal later inthis chapter) Analyze the Data, Diagnose the Real Problems, and Recommend Changes to Fix Those Problems Collecting the data is necessary, but not sufficient, for a usability test. Alter the test itself, you still ned to analyze the data. You consider Introducing Usability Testing +25 the quantitative and qualitative data from the participants together ‘with your own observations and users’ comments. You use al ofthat to diagnose and document the product’ usability problems and to recommend solutions to those problems. ‘As we will discuss in Chapter 20, “Tabulating and Analyzing Data,” this is nota wivial task. Usablity testing is distinguished trom beta testing by both the quality and quantity of data that you have. The data are systematic, comparable across the participants that you saw, and very rich. The Results Are Used to Change the Product — and the Process ‘We would also add another point. t may not be part ofthe definition ofthe usability test itself, asthe previous five points were, but itis crucial, nonetheless. “A usability testis not successful if itis used only to mark off. “eden onthe evenpmont ech: Asti tot wsscetl > euse e “only if it helps the product that was tested and the process » wail ts. “by which it was developed: As we discuss in Chapter 24, “Changing the Product and the Process,” pat of your task as a usability tester is ‘doing what you can to make sure that the results of the test are used appropriately. Someone must use What Is Not Required for a Usability Test? (Our definition leaves out some features you may have been expecting to 600, uch as: ‘laboratory with one-way mirror datalogging software videotape a formal test report Each of these is useful, but not necessary, for a successful usability test. For example, a memorandum of findings and recommendations ‘or a meeting about the test results, rather than a formal test report, ‘may be appropriate in your situation. Each of these features has advantages in usability testing that we discuss in detail later, but none isan absolute requirement. Through ‘out the book, we discuss methods that you can use when you have only a shoestring budget, limited staf, and limited testing equipment. When is a Usability Test Appropriate? [Nothing in our definition of a usability test its i to a single, summa- tive test atthe end of a project. The five points in our definition are