Perfecting Your Usability Studies 2ndEdition-Group

Perfecting Your Usability Studies
186 Tips and Tricks for Better Usability Testing
2nd Edition
Kim Salazar and Kate Moran

Foreward by Jakob Nielsen
48105 Warm Springs Blvd., Fremont, CA 94539-7498 USA
Copyright © Nielsen Norman Group; All Rights Reserved. To get your own copy, please go to:
https://www.nngroup.com/reports/how-to-conduct-usability-studies/
Order NNG211105-5163-49141
Copyright Notice
Copyright Notice
Please do not post this document to the internet or to publicly available
file-sharing services.
This report required hundreds of hours of planning, recruiting, testing, analyzing, writing and
production. We sell these reports to fund independent, unbiased usability research; we do not
have investors, government funding or research grants that pay for this work.
We kindly request that you not post this document to the internet or to publicly available file-
sharing services. Even when people post documents with a private URL to share only with a
few colleagues or clients, search engines often index the copy anyway. Indexing means that
thousands of people will find the secret copy through searches.
If someone gave you a copy of this report, you can easily remedy the situation by going to
www.nngroup.com/reports and paying for a license.
We charge lower fees for our research reports than most other analyst firms do, because we
want usability research to be widely available and used.
Thank you!
Report Authors: Kim Salazar, Kate Moran
Order NNG211105-5163-49141
© Nielsen Norman Group | www.nngroup.com 2
Table of Contents
Table of Contents
Foreword 5
Introduction 7
GUIDELINES 8
General Attitudes 17
The Politics of Usability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
The Ethics of Usability Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Recruiting Participants 22
Finding Good Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Re-using Participants. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Making Sure Participants Show Up. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Writing Good Tasks 29
How to Perform Tests 33

Planning and Preparation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Running the Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Debriefing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Discussing Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Reporting Results 47
Reporting Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Report Format & Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Testing in a Lean UX or Agile Environment 55
How To Test on a Minimal Budget 59
Order NNG211105-5163-49141
Table of Contents
Remote Testing 62
Remote Moderated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Remote Unmoderated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
Testing with Experienced Users 72
International Usability Testing 75
Assessing the Quality of a UX Research Agency 78
References 80
Basic Textbooks About Usability Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Politics of Usability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Ethical Guidelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
About the Authors 81
Acknowledgements 82
About NN/g 83
Our Experts Do it All. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Publications and Content. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Order NNG211105-5163-49141
Foreword
Foreword
These days, there are many resources available to learn the basics of usability testing
including: techniques for facilitating sessions, how to develop test plans, analyzing data,
and presenting the findings. Compared to when the first edition of this report was published,
information about the basics is very accessible. In fact, there are many excellent textbooks
listed in the References section that are well suited for those who are completely new to the
practice. However, going from the basics to knowing everything about usability testing is
much more difficult. This report is intended for people who already understand the basics
of conducting usability tests and are looking to improve their skills. This report will increase
the effectiveness of already established UX testers. You may know several of the tips in this
report, but you probably don’t know all of them. Nobody is perfect, but we can all improve.
It has become clear over our more than 20 years of work in the field of user experience
that there is a great interest all over the world in finding ways to improve the outcome of
usability testing.
For the last twenty years, we spent an exorbitant amount of time on the very first step of
getting companies to include any usability activities in their development process. We still
need to evangelize this basic concept because many companies are still not enlightened
about the value of usability. Better companies have progressed up the UX maturity curve,
however, and are now at the stage where they need to reflect on their own methodology and
find ways of improving it.
I recommend allocating a few percent of a company’s UX resources to activities that do not

do anything directly for the bottom line but simply improve the effectiveness of the work you
perform the rest of the time. Reading this report is an example of an activity that will help
increase the return on investment of all the tests you conduct. Methodology improvements
can bring huge gains for a relatively small effort. Let’s say that one of the tips results in
making your user tests 1% more effective. In an organization with ten UX professionals, a 1%
improvement corresponds to more than twenty person-days per year — the same as hiring
an external contractor for a month.
This report contains 186 tips. Not all of them will apply to all organizations. Find the ones that
work for you, and I would not be surprised if you end up improving the effectiveness of your
UX team by a good deal more than 1%.
Order NNG211105-5163-49141
Foreword
For those who are entirely new to usability testing, we have a full-day training seminar1 that
will empower you to conduct quick and cheap usability tests throughout the project lifecycle.
This course will teach you the benefits of traditional usability testing techniques and online
testing tools. You will learn how to facilitate sessions, develop test plans, analyze data, and
present the findings. This course is ideal for people who are new to usability testing or have
some usability testing experience, but no formal training.
—Jakob Nielsen
1
See https://www.nngroup.com/courses/usability-testing/
Order NNG211105-5163-49141
Introduction
Introduction
This report describes a number of tips and tricks that we have compiled over the course many
years of practical usability testing. We’ve discovered most of these tips the hard way: By making
mistakes. Other tips have come from watching and listening to other UX professionals to learn
about their practices.
The main criteria for including a tip in this collection are:

• The tip is not included in usability testing textbooks.
• The tip is included, and it is important, but experience shows that it is often disregarded
by UX professionals.
Some of the tips explicitly refer to websites. However, almost all of the tips apply equally well for
testing any product or service.
Order NNG211105-5163-49141
Guidelines
Order NNG211105-5163-49141
Guideline List
Guideline List
General Attitudes 17
1. Don’t test your own design, when you can avoid it............................................................. 17
2. Check the quality of your work processes regularly............................................................. 17
3. Look for ways to maintain and improve your skills.............................................................. 17
4. Avoid making design decisions based solely on user opinions......................................... 17
5. Make usability test results usable for your audience.......................................................... 18
6. Check the usefulness and usability of your work................................................................. 18
7. Avoid personal opinions, and base suggestions on data whenever possible................. 18
8. Sell your ignorance by insisting that only research has the right answers....................... 18
9. Position yourself as an ally, not as an enemy or the police................................................ 18
10. Build alliances.......................................................................................................................... 19
11. Pick your battles; you can’t win them all............................................................................... 19
12. Build trust by being completely open.................................................................................... 19
13. Make the consequences of bad design visible.................................................................... 19
14. Document and sell even small successes to your colleagues and to management........ 19
15. Spend time in management meetings to get visibility and report on UX activities,
as well as to understand the pressures on your stakeholders........................................... 19
16. Familiarize yourself with recognized ethical guidelines...................................................... 20
17. Always ensure you have informed consent.......................................................................... 20
18. Don’t allow managers to watch usability tests if their staff are the participants............. 20
19. Emphasize that you’re testing the product, not the participant........................................ 20
20. Make the first task simple....................................................................................................... 21
21. Observe your own rules for handling video recordings...................................................... 21
Recruiting Participants 22
22. Use online panels if you need people “off the street.”........................................................ 22
23. Use a recruitment agency if you need people who meet a certain
demographic profile................................................................................................................ 22
Order NNG211105-5163-49141
Guideline List
24. Set up a table in a public place.............................................................................................. 22
25. Ask for lists of people who have contacted customer service........................................... 22
26. Ask for help from the sales department................................................................................ 23
27. Intercept users in their natural environment........................................................................ 23
28. Approach internal users through their managers, not directly.......................................... 23
29. Approach organizations (churches, charities, and so forth).............................................. 23
30. Use your existing networks and tools.................................................................................... 23
31. Avoid recruiting your friends and relatives........................................................................... 24
32. Ask practical questions to screen for level of comfort using technology......................... 24
33. Screen out people like you – anyone who works in design,
development, or marketing.................................................................................................... 24
34. Ask participants to suggest other participants.................................................................... 25
35. Be careful in re-using participants from previous tests...................................................... 25
36. Keep a database of previous participants............................................................................ 25
37. Send a thank-you email after the test................................................................................... 25
38. Describe what it’s like to participate in a usability test before the test............................ 26
39. Provide detailed instructions on how to get to the site...................................................... 26
40. Send a confirmation message to each participant immediately after recruiting............. 26
41. Avoid words that make a usability test sound like a scientific experiment...................... 26
42. Check the comprehensibility of the confirmation message................................................ 27
43. Stress that “Your participation is particularly important to us.”........................................ 27
44. Provide a reasonable incentive.............................................................................................. 27
45. When increasing incentives doesn’t work, emphasize the altruism of participation...... 28
Writing Good Tasks 29

46. Focus your test on core tasks, rather than on what’s new, fun or easy to test................. 29
47. Test critical tasks...................................................................................................................... 29
48. Test edge cases........................................................................................................................ 29
49. Write or customize tasks so that they are realistic for each participant........................... 30
50. Write tasks that have context................................................................................................. 30
Order NNG211105-5163-49141
Guideline List
51. Include confirmation or verification emails, authentication messages, and

other similar interactions in your test................................................................................... 31
52. Write tasks that begin with triggering assets or interactions............................................. 31
53. Avoid tasks you think are humorous..................................................................................... 31
54. Avoid hidden clues in task descriptions............................................................................... 31
55. Give the participant a goal but avoid describing the steps................................................ 32
How to Perform Tests 33

56. Spend some time with the product team or client, so that you can identify and
better meet their needs........................................................................................................... 33
57. Work with your product team to define the user profile..................................................... 33
58. Ask your product team what feedback they may already have and where they
got that feedback..................................................................................................................... 34
59. Work with your product team to create the test scenarios................................................. 34
60. Make it as easy as possible for your clients (managers, developers, designers) to
watch the tests......................................................................................................................... 34
61. Ask your product team to observe the tests as a group..................................................... 34
62. Ask the product team members to take notes of what they observe................................ 34
63. Practice everything the day before the test.......................................................................... 35
64. Leave sufficient time between test sessions for key activities........................................... 35
65. Make a list of things to do between tests............................................................................. 35
66. Ask participants to do the paperwork before the test......................................................... 35
67. Provide an observation area where observers can watch tests without restrictions...... 36
68. Train observers in usability test etiquette............................................................................ 36
69. Write the nondisclosure and consent form in plain language — not in legalese............ 37
70. Align on a notetaking strategy or framework....................................................................... 40
71. Avoid nondisclosure agreements whenever possible......................................................... 41
72. Sit next to and slightly behind the participant..................................................................... 41
73. Ask, “What are your expectations?”....................................................................................... 41
74. Start with an open-ended task to allow participants to get comfortable with the
website or interface................................................................................................................. 41
Order NNG211105-5163-49141
Guideline List
75. Let the participant know that she’s participating correctly................................................ 42
76. Be gracious when participants blame themselves.............................................................. 42
77. Wait a few moments before you help.................................................................................... 42
78. Decide in advance when to intervene................................................................................... 43
79. Help immediately if the participant is struggling with something that you
already have sufficient data about........................................................................................ 43
80. Be very cautious what you say when providing help........................................................... 43
81. Watch your body language, facial expressions, and unconscious reactions................... 43
82. Watch and listen to session recordings to improve your facilitation................................ 43
83. Present test tasks as “activities” to your participants......................................................... 44
84. Show your appreciation for any participant suggestions................................................... 44
85. Write down the debriefing questions before the test.......................................................... 44
86. Get questions from the observers.......................................................................................... 44
87. Write down specifics to probe on later.................................................................................. 45
88. Return to key problems that the participant encountered................................................. 45
89. Don’t defend the design.......................................................................................................... 45
90. Ensure that participants leave the test feeling no worse than when they arrived.......... 45
91. Follow the sessions with a findings workshop to build early consensus among
the group................................................................................................................................... 45
92. Downplay your role and opinions.......................................................................................... 46
93. Keep discussions of severity and resources separate......................................................... 46
Reporting Results 47
94. Ask your client about reporting requirements...................................................................... 47
95. Quick and dirty reporting is sometimes needed.................................................................. 47
96. Do a quality and usability check of your test report............................................................ 47
97. Record usability problems in the bug tracking system used by your organization......... 48
98. Get involved in sessions where managers decide what bugs need to be fixed............... 48
99. Let the project team comment on the usability report before you show it to others...... 48
100. Make sure that you know who should get usability reports............................................... 48
Order NNG211105-5163-49141
Guideline List
101. Set up a database of reports or individual findings that is easily accessible.................. 48
102. Provide high-level analysis and executive summaries........................................................ 49
103. Consider creating a report that looks at problems across studies.................................... 50
104. Make it short............................................................................................................................. 50
105. Consider the following usability test report format............................................................. 50
106. Include a short executive summary....................................................................................... 50
107. Classify the severity of problems........................................................................................... 51
108. Include quantitative data........................................................................................................ 51
109. Mention positive findings....................................................................................................... 51
110. Organize findings in a way that is useful to the particular audience. .............................. 52
111. Include sufficient information to reproduce the test. ......................................................... 52
112. Include many screenshots...................................................................................................... 52
113. Develop a consistent report format....................................................................................... 52
114. Be careful with findings that were observed with only one participant............................ 53
115. Consider using a table that indicates which participants had what problems................ 53
116. Provide precise problem descriptions, preferably with examples..................................... 53
117. Use participant quotes extensively........................................................................................ 53
118. Include a recommendation with each problem................................................................... 54
119. Provide brief notes about relevant basic UX or human factors principles........................ 54
Testing in a Lean UX or Agile Environment 55

120. Designate a regular cadence for testing during sprints...................................................... 55
121. Instead of focusing on features, focus on research questions........................................... 55
122. Encourage cross-functional team members to observe and participate in testing......... 55
123. Create logistics templates to make testing plans reusable and easy to repeat............... 56
124. Define lean testing checklists................................................................................................. 56
125. Designate tools for your team to collaborate in................................................................... 57
126. Leverage paper prototypes for quick insights...................................................................... 57
127. Consider the Rapid Iterative Testing and Evaluation (RITE) Method for
experienced researchers......................................................................................................... 57
Order NNG211105-5163-49141
Guideline List
128. Create one-page research findings........................................................................................ 57
129. Bring your findings to daily standup...................................................................................... 58
How To Test on a Minimal Budget 59

130. Run at most six participants with one set of test tasks....................................................... 59
131. Take notes as the session unfolds......................................................................................... 59
132. Explain why you are taking notes.......................................................................................... 59
133. Use one usability professional to run tests.......................................................................... 59
134. Don’t use a market research or usability test lab................................................................ 60
135. Consider a portable discount usability lab........................................................................... 60
136. Talk to people who know users well, such as customer service representatives
(if you’re unable to talk to actual users)............................................................................... 60
137. Consider remote testing.......................................................................................................... 60
Remote Testing 62
138. Avoid recruiting professional testers..................................................................................... 62
139. Ensure that remote testing is the right method for your study.......................................... 62
140. Take advantage of remote tool demos before committing to a testing tool.................... 62
141. Run a pilot test to iron out technical or logistical issues.................................................... 62
142. If the testing tool’s chrome covers the interface, remind users to hide the
chrome while they work.......................................................................................................... 63
143. If you need certain technical requirements, include this in your screener....................... 63
144. If you are testing a live product, notify the entire product team about testing
dates and check that nothing will be changed during the study....................................... 64
145. Prepare scripts for all types of participant communications in advance and save
in a format that allows you to easily copy and paste.......................................................... 64
146. When choosing a tool for remote moderated testing, consider features needed
for observers, the type of meeting setup you need, and the experience you
need for your participants....................................................................................................... 65
147. Consider doing tech check sessions in advance.................................................................. 65
148. Send technical requirements out in advance....................................................................... 66
Order NNG211105-5163-49141
Guideline List
149. Specify upfront how you will provide participants with remuneration. Immediate
payment methods are recommended................................................................................... 66
150. Set your computer to do-not-disturb mode and/or shut off any distracting
notifications or applications that could interrupt the session........................................... 66
151. Determine how you will send and receive consent to record the user.............................. 66
152. Have observers arrive well in advance of the session to ensure their technical
setup and settings are correct before participants arrive................................................... 67
153. Plan to use a specific private channel for any communications between the
facilitator and observers during sessions............................................................................. 67
154. When choosing a tool for unmoderated testing, consider recruiting needs, what
type of data you need, and how your study must be designed......................................... 68
155. Consider using a remote unmoderated testing platform that allows you to
recruit your own participants and invite them to the test.................................................. 69
156. Keep tasks specific and direct................................................................................................ 69
157. Have a teammate proofread tasks and instructions before launching a test.................. 70
158. Remind participants to limit background noise................................................................... 70
159. Try to analyze as soon as possible following the test recordings...................................... 70
160. Don’t just use the stock charts the remote unmoderated platform gives you.
Download an exported spreadsheet, remove outliers, & make your own charts............ 71
Testing with Experienced Users 72

161. Always test some expert users............................................................................................... 72
162. Consider the internal and external marketing effect of testing with
experienced users.................................................................................................................... 72
163. Include both a novice and an expert user in your pilot testing.......................................... 72
164. When recruiting, ask contacts to suggest known experienced users to you.................... 72
165. Ask each participant to bring typical examples of their work related to your
target product........................................................................................................................... 73
166. Ask participants to postpone work they will do with the product or website, so
they can do this work while you are present........................................................................ 73
167. Ask participants about typical and critical tasks................................................................. 73
168. Quiz experts about the size of data sets. ............................................................................ 73
Order NNG211105-5163-49141
Guideline List
169. Consider asking expert users these questions: to get their input..................................... 73
170. Ensure faux data in prototypes is realistic............................................................................ 74
International Usability Testing 75

171. If your budget doesn’t allow for testing in each target country, at least have a
local usability person review the site for potential cultural issues................................... 75
172. Don’t assume that the USA, UK, and Australia are the same just because English
is their primary language........................................................................................................ 75
173. Run the test in the native language of the user................................................................... 75
174. Hire a local usability specialist to run the tests................................................................... 75
175. Arrange for one person to supervise all usability tests across countries......................... 75
176. Consider using simultaneous translation............................................................................. 76
177. Ask about the native language of local providers you seek to help coordinate
your particular test................................................................................................................... 76
178. Ask for task translations and check the quality in advance................................................ 76
179. Prepare a detailed facilitator’s guide that describes how you want the tests
to be run.................................................................................................................................... 76
180. Prevent trivial international usability problems by using appropriate checklists............ 77
Assessing the Quality of a UX Research Agency 78

181. Ask for references and follow up on one or two of them.................................................... 78
182. Request a sample usability testing report............................................................................ 78
183. Notice whether their sample usability report focuses too much on visual design.......... 78
184. Check the recommendations in the sample usability report: Are they detailed
enough to follow for people who were not involved in testing?........................................ 78
185. Inquire about the experience and qualifications of the facilitator who will run
your study.................................................................................................................................. 78
186. Notice whether they have any requirements or questions for you.................................... 79
Order NNG211105-5163-49141
General Attitudes
General Attitudes
1. Don’t test your own design, when you can avoid it.
Ask another person to test products to which you contributed significant design input. Chances
are that you are not sufficiently unbiased to evaluate polite criticism of key design decisions.
Of course, testing your own design is better than no testing at all, if those really are the only
options available to you.
You can test a product that you have already tested after changes have been made, and it can
be very efficient to do so because you’ll already know the product. Often, you will be able to offer
deeper insights. But you should be aware that you might also be biased, because you have a
personal interest in showing that your previous recommendations were good.
2. Check the quality of your work processes regularly.

Learn from other professionals; for example, consider seeking out the advice of an independent
coach or consultant every couple of years to comment on your test practices.
Seek and listen to feedback on your processes from the product team. Remember to model a
willingness to iterate and improve your process design, just as you would ask the product team to
iterate their designs.
Record yourself facilitating interviews and usability tests, and reflect on your facilitation skills with
another colleague.
3. Look for ways to maintain and improve your skills.

Be open to new methods and variations on old methods. Keep an open mind about new ways to
do your work. Keep up with the literature.
4. Avoid making design decisions based solely on user opinions.

“Don’t tell me — show me!” should be the basic point of departure for any usability test. Watch
out particularly for user opinions of the form of, “I know how to do this, but you should redesign
it because most other users would not know how to do it.” Such opinions are often worthless,
because they are pure speculation.
Explain to participants that they were chosen for their specific and unique qualifications and that
they needn’t worry about representing other people. Tell them you want to see them working with
the product as if they were at their office or home. Opinions often differ from performance, and
it’s the latter that’s (usually) more important.
Order NNG211105-5163-49141
General Attitudes
5. Make usability test results usable for your audience.

Keep in mind that the users of your usability test results are the members of the product team,
including developers, product managers, marketing, and so forth. You should take some time up
front to discuss what they are looking for in the output you produce from the test.
6. Check the usefulness and usability of your work.

Seek feedback on how useful and usable your results are for your product teams. This could
be done via a few questions in an email. Iterate until you have a procedure and a presentation
format that is appropriate for your audience.
Any comments about problems in a usability test procedure or in a usability test report should be
taken very seriously.
THE POLITICS OF USABILIT Y
7. Avoid personal opinions, and base suggestions on data

whenever possible.
There is no answer to the question “Why are your opinions better than mine?” Opinions only lead
to opinion wars. You can’t win arguments based on opinions, your recommendations should be
based on findings from research.
8. Sell your ignorance by insisting that only research has the

right answers.
You become a new and interesting player if you continue to insist on your own ignorance: “Gee,
I don’t know what users prefer. And I have learned from experience that my opinions are not
always reliable. Let’s test both of these suggestions! That will tell us which one is working better
for the user.”
9. Position yourself as an ally, not as an enemy or the police.

UX practitioners can easily be seen as the bearer of bad news or the analyst that does nothing but
point out flaws. Creating this sort of reputation will not foster an ongoing interest in conducting
usability tests. Be conscious about how you sell the benefits of usability testing and how you
deliver the results. These engagements should make your partners walk away feeling like you’ve
helped them succeed.
Order NNG211105-5163-49141
General Attitudes
10. Build alliances.

Form alliances with project groups who are interested in usability. Give these groups
preferential treatment.
11. Pick your battles; you can’t win them all.

Even if you could, it would be unwise. Defeated colleagues rarely make good allies.
12. Build trust by being completely open.

There is nothing mysterious about usability testing. Demystify the process. Explain carefully what
you do to anyone who wants to know.
13. Make the consequences of bad design visible.

The most important task for a UX professional is to demonstrate through testing the
consequences of ignoring usability.
• Pull together a short video clip showing multiple users experiencing difficulty with a
design or feature that was not tested.
• Emphasize the cost-benefit analysis for doing testing up-front vs. tracking defects and
fixing them later.
• Prepare UX metrics that describe the negative business impact. For example: “In a
quantitative usability study, 58% of participants failed to complete the onboarding process.”
14. Document and sell even small successes to your colleagues and
to management.
When usability testing helps discover and solve design problems, publicize it. Make sure people
know that those successes are the reasons why UX research is important.
15. Spend time in management meetings to get visibility and report on UX

activities, as well as to understand the pressures on your stakeholders.
Empathize with your stakeholders. Understand why they might resist UX research, and work to
address those concerns.
Order NNG211105-5163-49141
General Attitudes
THE ETHICS OF USABILIT Y TESTING

Users are human. We must ensure that our fellow humans perceive their encounter with UX
professionals as pleasant, without sacrificing the accuracy of our results.
It is essential that participants leave a usability test session feeling no worse (and possibly better)
than when they arrived.
16. Familiarize yourself with recognized ethical guidelines.

There are guidelines produced by professional organizations like the APA2 and the ACM3 about
how UX professionals should behave.
17. Always ensure you have informed consent.

Informed consent means participants understand what their involvement will be, what will
happen in the session, whether recordings will be made, and if so, what kind of recordings, and
what will happen to them. They will have explicitly agreed to this all in a consent form, which
should be given to the participant before the research starts. Consent forms should be written
in plain language and contain all the necessary information a participant needs to make an
informed choice about whether to take part or not.
18. Don’t allow managers to watch usability tests if their staff are
the participants.
If you’re testing an internal product, like an intranet, protect employees’ privacy. Don’t run the test
if managers insist on being present. You want employees to feel safe sharing honest feedback.
Show only completely anonymized test results to managers.
19. Emphasize that you’re testing the product, not the participant.
Make this clear when the participant is recruited. Repeat it in the confirmation email. Say it again
when you greet the participant. Emphasize it once more if the atmosphere in the testing room
becomes tense.
2
APA (https://www.apa.org/)
3
ACM (https://www.acm.org/)
Order NNG211105-5163-49141
General Attitudes
20. Make the first task simple.

No matter how many times you tell participants that you are not testing them, they will still
feel under pressure initially. A quick success relaxes the participants and helps them to feel
comfortable in the surroundings and with thinking aloud.
A simple task might be something like, “Visit ACME.com. What products or services does this
company provide?”
21. Observe your own rules for handling video recordings.

Unfortunately, we have seen situations where unethical researchers have handed testing clips
to other UX professionals outside of the organization, or where “funny” clips from usability tests
were shown to those outside of the product team for entertainment purposes.
Make sure your organization has strict rules for who can watch recordings from usability tests,
where they will be stored, and how they will be shared. Be sure that you are treating the
participants with the respect they deserve.
Order NNG211105-5163-49141
Recruiting Participants
Note: For a much more extensive and in-depth set of advice for getting participants, please
see our separate report Recruiting Participants for User Research: Tips and Tricks. There
is a small amount of overlap between the tips provided in this report and the recruiting report;
however, the recruiting report provides more detailed advice on finding and enticing users to
participate in your studies.
FINDING GOOD PARTICIPANTS
22. Use online panels if you need people “off the street.”
Participants from online recruitment panels are often a good value. However, these panels
typically include very general users, so they aren’t great if you need a very specific profile.
23. Use a recruitment agency if you need people who meet a certain
demographic profile.
Recruitment agencies all vary in their ability to recruit target users. Make sure to ask them how
big their panel of users is, and where they recruit them from.
24. Set up a table in a public place.

If you need “people off the street,” (users that don’t have to meet a strict user profile) see if you
can set up a booth in a local grocery store, library or mall. Always ensure you have permission to
be there. It’s sometimes even possible to conduct short tests in this very informal setting.
25. Ask for lists of people who have contacted customer service.
Often, such people are quite motivated to provide comments. We don’t recommend recruiting
more than 40% of the participants this way, however, because such people are often on the
extremely knowledgeable and motivated end of the continuum.
Watch out for highly dissatisfied customers who wish to use the test session as an outlet for their
complaints — this is not the sort of motivation you want. One or two articulate, critical devil’s
advocates can be quite useful in a test, however.
Order NNG211105-5163-49141
26. Ask for help from the sales department.

Sales people often know a lot of users of the product.
If you need participants who do not yet use the product, and the company has more than one
product, contact users from those other product customer lists. You may also be able to identify
potential future users from lists of sales leads.
27. Intercept users in their natural environment.

If your task is to test a florist website, go to a local florist’s shop and contact customers from
their customer database. If your task is to test a library website, visit a branch of the library and
intercept people as the come or go.
28. Approach internal users through their managers, not directly.

Sometimes you may need users of a particular company’s product. When reaching out directly
to these employees, they can often be unsure if they can or should commit to something without
their boss’ approval. If you approach users through their manager, you get a better chance to
explain how the usability test will benefit the company, which makes it more likely that you will
get a positive answer. This person can also introduce you to her employees and set you up to
recruit them for your study.
29. Approach organizations (churches, charities, and so forth).

Approach organizations that include the population you are seeking to involve in the test. Offer
to donate money to the organization for each participant they recruit. This donation will interest
individuals who would not be interested in the small sum paid to volunteers.
30. Use your existing networks and tools.

If have a fairly general target research audience, utilize your existing social networking tools to
advertise for participants.
• Post a call on social media for participants who fit your criteria and are interested
in participating.
• Link to a form that can gather more information from interested parties to determine if
they fit your recruiting profile.
• Invite your connection to share your post for an even wider reach.
When you use these channels, just be mindful not to recruit people you know too intimately, as
stated in the next guideline.
Order NNG211105-5163-49141
31. Avoid recruiting your friends and relatives.

Friends and relatives often are not representative users. If they aren’t typical users, the test may
result in them feeling foolish — not something you want to do to someone you care about. They
may also be overly nice or critical.
Additionally, if you speak often to your close friends and relatives about design, they be primed to
analyze interfaces a bit more than the average person who has no background in UX.
If your own friends and relatives match the user profile it may be OK to recruit them but ask a
colleague to facilitate those tests so your participants don’t feel like they have to tell you only
good things.
32. Ask practical questions to screen for level of comfort

using technology.
Ask prospective participants to demonstrate their knowledge. Don’t waste time asking them to
provide personal judgments about their level of expertise.
Ask “If you are on a web page, how do you return to the page you just came from?” Don’t ask
“How experienced are you in using the web?”
33. Screen out people like you – anyone who works in design,
development, or marketing.
Unless your product or website is intended for developers, it’s a good idea to screen out people
who claim to know a programming language or to have created a website.
You may want to screen out anyone who currently or previously worked in any of these fields:
• UX or CX design
• Marketing
• Visual or graphic design
• Development or computer engineering
• Quality assurance
• Customer service
Order NNG211105-5163-49141
34. Ask participants to suggest other participants.

Participants often know other prospective participants. Remember to ask.
You might even suggest a deal where everyone benefits: “Usually we pay $120 to a participant. If
you get another qualified participant for us, we’ll pay both of you $180.”
In one case where we successfully applied this method, the participant immediately suggested:
“And if I get you two qualified participants, how much will you pay?”
RE-USING PARTICIPANTS
35. Be careful in re-using participants from previous tests.

If a participant is good at thinking aloud and sparkles with catchy quotes you can re-use that
participant, but don’t use a participant more than twice a year — and not on the same product or
type of product.
When participants are tested too frequently, they may learn too much about the product or UX
research to be an “average” user.
36. Keep a database of previous participants.

After each test, make a note of how expressive and articulate the participants were and the
depth of their comments, so you can target them for future studies. However, if the participant
was recruited from an outside recruitment agency, it’s not appropriate to contact them for future
studies without going through the same recruitment agency.
37. Send a thank-you email after the test.

This courtesy will make it easier for you to recruit the participant for another test. Don’t forget
to ask them for referrals to other people they know who might enjoy participating as much as
they did.
Order NNG211105-5163-49141
MAKING SURE PARTICIPANTS SHOW UP

Sometimes scheduled participants may get cold feet at the last minute or forget about their
scheduled session. It can be costly to have no-show participants. Do the extra work to ensure
your test sessions will be filled.
38. Describe what it’s like to participate in a usability test before the test.
Provide this information both when the participant is recruited and in the confirmation email.
39. Provide detailed instructions on how to get to the site.

Provide a researcher’s phone number to call if a problem arises. Request the participant’s phone
number, so a researcher can follow up if the participant doesn’t arrive on time.
It’s always wise to usability test the directions you intend to provide to make sure they’re clear —
poor directions are a major cause of no-shows. Go beyond providing an address.
• Suggest places for the participant to park if they drive.
• Let them know if they may need to pay for parking.
• Tell them how early to arrive to find parking.
• Send a photo of the parking lot and/or entrance to the facility.
• Tell them what level the test lab is on and how to navigate the building to find it.
• Let them know if they should expect to check in with a receptionist.
40. Send a confirmation message to each participant immediately

after recruiting.
This could be an email, meeting invitation, or text. The confirmation note should tell the
participant when they should expect more communication or details regarding the test.
41. Avoid words that make a usability test sound like a

scientific experiment.
Avoid words and phrases like “lab,” “experiment,” “test” and “test subject” in your description of
a usability test. Use terms like “session” instead.
Order NNG211105-5163-49141
42. Check the comprehensibility of the confirmation message.

Ask two or three people to read the confirmation message out loud and comment on any points
that seem problematic, ambiguous, and so forth. If they hesitate or pause during reading, ask
afterward what caused them to hesitate.
After they have read the message, remove it and ask them to recall a few points from the email.
Iterate on the communication as needed.
43. Stress that “Your participation is particularly important to us.”

Emphasize that each participant was chosen because he or she meets carefully chosen criteria.
If you find participants using a recruitment agency, sometimes those people are used to being
recruited for focus group studies with 5-8 people per session. If the participant doesn’t realize
this is a different type of study, he might think it doesn’t matter much whether or not he shows up
– he doesn’t realize your facilitator will be wasting his time. If you recruit with this method, stress
that the study will be a “one-on-one” session.
44. Provide a reasonable incentive.

When incentives are too low, you run the risk that participants will decide last-minute that
participation isn’t worth their time.
For internal studies on products like intranets, promotional gifts like company coffee mugs
or pens might be enticing. But for external participants, they may be considered worthless.
Ask yourself, “How much am I — the facilitator — getting paid to run this test? Why should a
participant do it for less?”
Although the incentive should not be the driving force behind participation, a substantial gift
shows that you appreciate that the participants have set aside precious time to help you.
In 2020, decent incentives seem to be in the $100–$150 range for a 90-minute test. If you recruit
a rare, busy, or highly-paid professional participant (like doctors or lawyers) you may need to offer
more to attract participants.
Keep in mind that some participants may not be allowed to accept compensation of any kind
(government workers for example).
Order NNG211105-5163-49141
45. When increasing incentives doesn’t work, emphasize the altruism

of participation.
Sometimes, increasing the amount of the incentive doesn’t yield the expected increase in
participation and reduction of no-show participants. In those cases, it can sometimes help to
stress that participation can help people. You might say something like, “Your participation will
help us to improve this product for you and people like you.”
This approach can be particularly effective when testing a nonprofit or charity product.
Order NNG211105-5163-49141
Writing Good Tasks
Writing Good Tasks

Writing the tasks for a usability test is one of the most critical activities of test planning. Poorly
written tasks can invalidate your efforts. There are different ways to choose the tasks for a test:
frequency of use, criticality, new features, customer complaints, and so forth.
46. Focus your test on core tasks, rather than on what’s new, fun or easy
to test.
Focus on the key purpose of a product. Ask yourself: What user goals are crucial to the success of
this product?
If testing a video conferencing tool, for example, focus on tasks like:

• Join a meeting
• Mute yourself
• Show your screen
• Invite others to your meeting
De-emphasize tasks like:

• Change your profile photo
• Add a virtual background
• Send an emoji in the chat
Test non-core tasks only after the basic functionality has been covered completely.
47. Test critical tasks.

Critical tasks are tasks that, if done wrong, could lead to dire consequences, such as
abandonment, complaints or errors, and which in turn could affect key business metrics like
customer retention, sales, satisfaction, and so forth.
48. Test edge cases.

Don’t focus just on the happy-path or common scenarios. Think about how users may actually be using
the product in the real world and try to recreate some of the more unique scenarios. Test behavior in
complicated but realistic scenarios, such as those in which large amounts of data are involved, some
resource limit is almost exceeded, service is denied, system load is heavy, and so forth.
If testing an email client, for example, include test tasks that involve inboxes containing so many
messages that scrolling is required, or very long messages.
Order NNG211105-5163-49141
Writing Good Tasks
49. Write or customize tasks so that they are realistic for each participant.
If it is appropriate, consider using tasks based on the participants’ personal goals, background,
or interests.
For example, in a job search website, you might have a formal task that asks a participant to find
an industrial engineering job in London, England. If the participant works as a receptionist, that
specific job will be irrelevant to him.
The process that participants use for this irrelevant task may be quite different than if you asked
them to describe a job that they would be interested in and then have them locate one that
matches their specific goals.
Consider customizing each task to fit the person who will perform it, and ensuring that recruited
participants would realistically perform the tasks.
50. Write tasks that have context.

Avoid tasks that simply instruct the participant to do certain things with the interface. Tasks
should have some brief but realistic context that will motivate the participant.
Task without context:

“See if you can create a new calendar event for Sunday at 9:00 AM.”
Task with context:

“Your friend invited you to have coffee at her home Sunday morning at 9:00 AM. Add this
coffee date to your calendar so you don’t forget.”
However, be careful not to overdo the scenario writing. Provide only as much context as is needed
to set the stage for the task – no more than that.
A task with too much context:

“Your best friend from college has invited you to have coffee in her home Sunday morning
at 9:00 AM. She lives 20 minutes away from you, and she has a pet cat. Add this coffee
date to your calendar so you don’t forget.”
Order NNG211105-5163-49141
Writing Good Tasks
51. Include confirmation or verification emails, authentication messages,

and other similar interactions in your test.
In many instances, when users sign up for a service, they may be required to authenticate,
or verify an email address. After making a purchase or completing a transaction, users may
receive a confirmation email. You should test these in your usability test, along with the
interface you are designing.
Many usability problems occur because designers have not tested thoroughly the end-to-
end service, but have focused only on one touchpoint of a string of touchpoints. After a user
completes an activity, write a followup scenario for them to look at whatever messages were
generated from that activity to see if they meet expectations set by the prior interaction.
52. Write tasks that begin with triggering assets or interactions.

These days, a digital interaction is rarely truly isolated. A visit to a website is often part of a series
of related interactions that make up a customer’s journey. Ask yourself what drives people to
the digital experience you are trying to test. Perhaps a marketing email, an automated email, an
advertisement. Use these items as the prompts and build them into your test scenarios so you
can ensure the transitions between these touchpoints work well also.
53. Avoid tasks you think are humorous.

Avoid silly names, which aren’t so funny when the participant finds the task difficult. Avoid instructions
like “Send an email to Mickey Mouse with the message, ‘Meet me tonight at 7 — Minnie Mouse.’”
54. Avoid hidden clues in task descriptions.

You do not want to lead users to behave a certain way. For this reason, avoid using specifics from
the interface in your task descriptions. Task descriptions should be in the form of general requests.
Flawed task with clues:

“Look up a person named Lois McClaren in the company directory.” — This task leads the
users to the right place by using the label of the directory that is displayed in site’s navigation.
Better task without clues:

“You need to send an email to a fellow employee named Lois McClaren, how might you
find her email address?” — This task is more general. It gives the person a typical goal and
allows them to determine how to complete it.
Order NNG211105-5163-49141
Writing Good Tasks
55. Give the participant a goal but avoid describing the steps.
Step descriptions often contain hidden clues.
Task with flawed goal:

“Register and send an email to walterk89@gmail.com.” — In this task description,
“Register” is not a typical goal from most users’ point of view. Instead it gives a clue
about how the system works.
Task with better goal:

“Send an email to walterk89@gmail.com” — This task helps us understand whether new
users can figure out that they have to register in order to send an email.
Order NNG211105-5163-49141
How to Perform Tests

A basic model for a usability test is as follows:
1. The facilitator greets the participant and asks him or her to fill out the consent form and
to do any other required paperwork.
2. The facilitator explains how the session will work, and what to expect.
3. The facilitator briefly interviews the participant. If related to the study, the facilitator
might ask about the participant’s background, preferences, opinions, or previous
experiences with the product.
4. The facilitator hands written task descriptions (scenarios) one at a time to the
participant. (In a remote moderated study, the facilitator might do this through a video
conferencing app’s chat feature.)
5. The facilitator interacts with the participant as necessary while the task is in progress.
This interaction might include reminding the participant to think aloud, or asking
follow-up questions.
6. After the participant has completed the test tasks, the facilitator may explore the
participant’s mental model of the product, or conduct any other relevant probing
questions during a short debriefing.
PLANNING AND PREPARATION
56. Spend some time with the product team or client, so that you can
identify and better meet their needs.
Just like you ask your product teams to spend some time with their users, spend time with your
users. Ask your product teams what questions they would most like to ask users. Consider these
questions when you write the test script and when delivering the findings.
57. Work with your product team to define the user profile.
Often your product team will have some knowledge of users, but you may have to help them to
identify and focus on the significant characteristics you must screen for.
Order NNG211105-5163-49141
58. Ask your product team what feedback they may already have and
where they got that feedback.
Help them to recall what they know and what they don’t know about the usability of the product,
and help them to evaluate the quality and source of existing feedback.
Often product teams have user data that they forget to consider before launching into new
research projects. This feedback will often be vague, for example “users have trouble finding
things.” However, it can trigger more detailed research questions.
59. Work with your product team to create the test scenarios.
This exercise — seeing the system through the eyes of a typical users — is often worthwhile
in itself. Members of the product team (developers, product marketing, quality assurance
specialists) can give valuable input and trigger research questions, but test scenarios are best
written by the UX professional writing the test plan. This is because UX professionals are trained
in how to write the tasks to avoid hints and mirror realistic user goals. Ask the product team to
comment on and approve the task scenarios to ensure all research questions will be touched on
during sessions.
60. Make it as easy as possible for your clients (managers, developers,

designers) to watch the tests.
Run the tests where your main target audience is. Use a discount portable lab (see tip 135) to run
the test at the location where your clients are. However, be sure not to have too many people
observing in the room, if more than 1 person would like to observe it would be best to setup a
viewing room nearby.
61. Ask your product team to observe the tests as a group.

Watching a usability test is a group experience. The discussions of common experiences during
the tests are important for building consensus on what needs changing. You’ll get more buy-in
from a group observation than from sending out highlight clips from the recordings.
62. Ask the product team members to take notes of what they observe.
It takes some practice to take good notes so you might want to offer some minimal training on
what to look for while observing.
Order NNG211105-5163-49141
63. Practice everything the day before the test.

Go through and complete the test tasks with the exact setup you plan to use for the tests. Doing
this may uncover unexpected issues you will want to resolve before the actual test.
64. Leave sufficient time between test sessions for key activities.
You should build in at least 30 minutes between sessions to complete the following important
activities before the next participant arrives. You might use this time to quickly discuss with your
team, or to reset the testing device (clearing cookies, for example.) You’ll also want time for your
team to take a restroom break between each session.
This buffer is also useful if people show up a little late (or early). If possible, have a waiting area that
is set apart from the testing area, so people can wait in comfort if there is an unexpected delay.
65. Make a list of things to do between tests.

This list would include things such as:
• Save the recording.
• Delete browsing history and cookies that may impact the next session.
• Clearing the cache and history in a browser.
• Get the prototype ready again, or pointing the browser to the agreed-upon starting page.
• Gather your incentive and consent form for the next session.
• Prepare a glass of water for the next participant.
• Take any restroom breaks.
• Prepare the order of the test tasks.
• Deleting accounts, files, or database records the participant created during the session.
66. Ask participants to do the paperwork before the test.

Send the consent form and any questionnaires you might have to the participants before the test.
Ask them to fill in the forms before the test.
This can save time, but some participants will forget. Have spare copies of the forms ready in case
a participant forgets the completed documents.
Order NNG211105-5163-49141
67. Provide an observation area where observers can watch tests

without restrictions.
It is natural and desirable for people who have a strong interest in the product to observe test
sessions. Participants should never hear any reactions to the test through walls from observing
parties. Make sure that the sound isolation between the room where the participant sits and the
observation room is perfect, possibly by locating the observation area far from the test room.
68. Train observers in usability test etiquette.

Any observers should have (at a minimum) specific training on how to be as neutral as
possible throughout the test and how to ask questions at the end. Untrained people could ask
problematic questions.
Participants may be embarrassed by comments or laughter from the audience, even if it is

sympathetic. You should provide strict instructions to observers, to remain quiet. It’s also
important for observers not to interact with each other.
Even if you think the observation room is perfectly soundproof, hold a pre-test meeting to train
all observers in basic lab etiquette. Summarize it in a one-page etiquette handout and post it in
the lab:
• Speak softly. Noise or laughter can make the participant uncomfortable, and can impact
the study.
• Assume that the participant can hear your comments.
• Don’t slam the doors if you must go in and out.
• Never speak about a participant in the office because you never know when that person
may be around the next corner or come back into the lab to pick up a forgotten item.
Order NNG211105-5163-49141
69. Write the nondisclosure and consent form in plain language — not
in legalese.
Writing consent forms in plain language is important to ensure informed consent. Consider
testing your consent form for comprehension with potential participants.
Excerpt from a nondisclosure form written in legalese:
CONFIDENTIALITY: During and after this contract, Participant agrees to

keep in confidence all proprietary information. “Proprietary information”
means any information about the specific merits or flaws of the product, or
Participant’s evaluations of the product. “Proprietary information” does not
include any information:
which Participant knew before it was disclosed in this study,
which has become publicly known through no wrongful act on Participant’s

part, or
which Participant developed independently of this study, as evidenced by

appropriate documentation.
When the contract terminates, Participant will return all papers and other
materials provided by this study. Participant agrees not to disclose any
“Proprietary information” and to take all reasonable precautions to prevent its
unauthorized dissemination, both during and after the contract. Participant
agrees not to use any “Proprietary information” for Participant’s own benefit or
for the benefit of anyone other than the sponsoring companies.
Participant acknowledges that all “Proprietary information” remains the

property of the sponsoring companies, and that no license or other rights in
this “Proprietary information” are granted hereby.
Signature:
___________________________________________________
Print name:
___________________________________________________
Date: ________________
Order NNG211105-5163-49141
Consider the example below as inspiration for your own plain language consent form. Please
note that you need to work with your team and legal department to ensure that your own
consent form meets all your specific internal standards and legal requirements.
Study Consent
What This Study Is About
This study is being conducted by Nielsen Norman Group (NN/g), a research

company based in the U.S., to learn about how people use websites. Your
participation in this study will help us create guidelines that enable designers
to create easy-to-use applications.
Your Involvement in the Study
In each 60-minute session, we will watch you use various websites or

applications, and ask you questions. With your permission, we will record
you and your screen on audio and video, including any personal information
you provide during the session, and we will take notes and photographs to
document your comments and actions.
Your Participation Is Voluntary
You may take a break or leave the session at any time without giving a reason.
You may also withdraw from the study after the session.
How We Will Use Session Data
Other NN/g researchers may observe your session remotely, or join us in this
room (if in a physical space). They may also view session recordings or notes
in the future.
We may publish research reports, articles or training content that include

your anonymous comments and actions, including session recordings. Except
for the image of you, NN/g will not provide any details with its use of the
information and images resulting from the study which would allow any third
party to identify you, nor will it use this information and images in any way that
can be damaging to you.
Your Consent
I give my consent for (please check all that apply):
o NN/g researchers to observe me during the research. [Required for study

participation.]
o The session to be recorded (audio, video and screen capture).
Order NNG211105-5163-49141
o NN/g to take photographs during the session which may contain me.
o NN/g researchers to view session recordings and documentation containing

my information.
o NN/g to use session data in articles, reports and training content. (I

understand that I will not be identified in the reporting of this research.)
Storage of Personal Information and Session Data
If you want to withdraw your consent in the future, contact the primary
researcher listed below, who will destroy any personal data collected as part of
this research. Otherwise, we will store and process your personal information
and session data in the U.S. until we deem the content no longer necessary for
the research purposes outlined above.
[researcher contact details]
Your Agreement to Participate
I hereby consent to participate in this study and for NN/g to collect and use
data as agreed upon by me and outlined above. I agree with the related
storage of my personal data, including my name, email address and phone
number by NN/g researchers. I am aware that I may withdraw my consent at
any time.
___________________________________________________
Signature Date
___________________________________________________
Print Name
Order NNG211105-5163-49141
70. Align on a notetaking strategy or framework.

For studies with many sessions being conducted by multiple parties it can be incredibly helpful to
set up a specific way you plan to take notes collectively. If all notes follow the same format, this
will make data analysis much easier.
Consider creating a spreadsheet template that notetakers follow. Discuss whether a taxonomy
for tagging insights will be helpful for analysis or locating particular quotes or findings for a
resulting report.
This screenshot shows an Excel spreadsheet with notes from a study on Indian
ecommerce websites. The top of the page (rows 1-4) has high-level information
about the participant and the session. The first column (column A) has an Excel
function that automatically generates a timestamp when a note is made in
the other columns (columns C, D, and E.) This helps when re-finding important
moments in the video recordings during analysis.
Order NNG211105-5163-49141
71. Avoid nondisclosure agreements whenever possible.

Because usability testing is done with a small number of strangers, there’s usually little risk that
they’ll leak your corporate secrets to the world. Having to sign a nondisclosure agreement can be
a barrier to participation — participants have refused to participate in tests for this reason.
Note, however, that the use of nondisclosures often depends on how soon a product will go to
market and on corporate policy. If your organization feels a nondisclosure agreement is necessary,
you can certainly use one.
RUNNING THE TEST
72. Sit next to and slightly behind the participant.

Place yourself so you are out of sight, for example to the right of and slightly behind the
participant as to not distract the participant as they are working.
73. Ask, “What are your expectations?”

Ask this question before the participant sees the home page but after you have briefly introduced
the company or the organization that the website belongs to. Consider asking “What would be
wonderful?” Listen to the user’s terminology.
You might prefer to avoid asking questions about expectations early on if you’re showing the
participant something brand new and he or she has not had time to grasp it yet.
74. Start with an open-ended task to allow participants to get

comfortable with the website or interface.
Before asking participants to do something very specific or complicated, give them a chance to
understand what they’re looking at and familiarize themselves with the website or interface. A
good example of a first task can be something like, “A friend told you about X website. Go to the
website, and see what it has to offer. When you think you have an understanding of what the
website is about, let me know you are finished.”
Providing this type of open-ended task early on is also a great way to get first impressions from
participants. As they think aloud in these broad tasks, they will voice their initial reactions.
Order NNG211105-5163-49141
75. Let the participant know that she’s participating correctly.

People are often very self-conscious in usability tests, particularly if it’s their first time. They can
feel stupid while narrating all their thoughts and actions. Because they aren’t researchers and
don’t know what we’re looking for, they sometimes worry that what they’re doing is boring to us.
After the first task, if the participant has been thinking aloud, we say, “Great, thank you for doing
that activity. By the way, I know this can feel strange, talking out loud like this and having us
watch you quietly. But you’re doing exactly what we need you to do, and this is very helpful and
interesting to us. We really appreciate it.”
Make a point of saying some variation of this reassurance, whether or not the participant is
showing any signs of distress or discomfort.
76. Be gracious when participants blame themselves.

If the participant says, “You must think I’m really dumb” reply:
“Everything you’ve done so far makes perfect sense. Remember that we’re testing
the product, not testing you. This is all just part of the process, it’s normal.”
This feedback works because it directly acknowledges the participant’s cognitive abilities in a way
that the rather lame “Don’t worry, it’s not your fault” can’t. The remark also works well because it’s
not offensive to any product team member who might be watching the test.
77. Wait a few moments before you help.

Provide help only when it is clear that the participant is unable to solve the task alone or when
the participant gives up (for example, says that they would call or go to another website). Even
when a participant asks you for help, you might say, “Imagine I’m not here, what would you do
next if you were really on your own?”
Otherwise, the usability test will become a demonstration of the application. Provide the
minimum help necessary to get the participant to start acting independently again.
Offering help too early may cause you to miss valuable feedback or interesting user behavior.
While learning when to intervene, try this trick: if you think you want to help, count to 60 in your
head before speaking.
Order NNG211105-5163-49141
78. Decide in advance when to intervene.

You may decide to allow a specific number of unsuccessful attempts, a set amount of time, or a
participant’s level of distress as criteria.
Again, don’t offer help too early, but know that users will rarely solve a problem if they have been
stuck for ten minutes. Observing them during this period provides valuable lessons about users’
problem-solving approaches and the design’s ability to communicate the available options.
79. Help immediately if the participant is struggling with something that

you already have sufficient data about.
For example, if three previous usability tests have turned up problems with feature X, it might be
appropriate to step in and help the subsequent participants as soon as they encounter the same
problem. This intervention is justified if there are later tasks about which the product team still
lacks sufficient data.
80. Be very cautious what you say when providing help.

One of the greatest opportunities for giving unintentional clues occurs when the facilitator
starts helping the participant. It is very difficult not to provide any help beyond the absolute
minimum required.
Because of this risk, some facilitators have a firm policy of not providing any help at all during a
test session; if participants run into insurmountable problems, they just ask them to proceed to
the next task.
81. Watch your body language, facial expressions, and

unconscious reactions.
Avoid giving unconscious help. For example, a faint smile or an almost breathless sigh can
provide an important clue to the participant that they’re getting close.
82. Watch and listen to session recordings to improve your facilitation.

Review session video recordings of your facilitation to learn how to improve. Listen to when you
intervene, your tone of voice, and how you respond to different testing situations.
If you’re trying to learn to be an excellent facilitator, consider having an additional camera in the
testing room pointed towards you. That way, you or your colleagues can watch the video and
provide feedback on your body language as you facilitate.
Order NNG211105-5163-49141
83. Present test tasks as “activities” to your participants.

The word task can sound intimidating to a participant. It can indicate that it is a pass/fail test, which
is not the case. Instead of using the word task with participants, present them as “activities.” This
word is less intimidating and does not suggest there is a right or wrong way to proceed.
When moving on to the next task, we often say, “Here’s the next activity. Please remember to read
the instructions out loud.”
84. Show your appreciation for any participant suggestions.

If a participant suggests a different way of doing things, say “Thank you,” and make a note
— even if you consider the suggestion worthless. Avoid comments like “Oh, yeah, we already
thought of that,” or, even worse, “We already thought of that, and it doesn’t work because ... .”
Comments that dismiss the participant’s ideas only make the participant feel stupid, and might
discourage him from speaking up again.
DEBRIEFING
The debriefing portion of the session should be used to elicit any final input, to ask clarifying
questions, and to wrap up the session and put the participant in a good frame of mind.
85. Write down the debriefing questions before the test.

There will likely be questions you want participants to answer after they’ve used the website
or system. Write out these research questions and have them proofed for good interviewing
techniques ahead of the session. This will ensure you use good interview methods, which can be
more difficult to do when you are formulating questions on the fly.
86. Get questions from the observers.

If you have observers, tell the participant that there may be a few questions that your
colleagues have.
If the observers are in a separate observation area, excuse yourself for a moment and get the
questions from the observers. You can make this easy by having your observers write down
questions on note cards and just picking them up at the end of the test session. You could also
have them private message you their questions via email or chat.
Order NNG211105-5163-49141
87. Write down specifics to probe on later.

As the participant is completing the activities, keep a list of items that you want to probe on
during the debrief. This ensures you don’t forget to touch on anything you deemed important
during the test.
88. Return to key problems that the participant encountered.

Ask probing questions to elicit feedback, such as:
• How could we prevent this problem from occurring?
• What would you have liked to see instead?
89. Don’t defend the design.

Defending the design signals to the participant that they did something wrong. For example,
during debriefing avoid remarks like, “I’ll show you a really smart way of accomplishing the
scheduling task you had problems with.” Much better is, “Lets go back to the scheduling task.
How should we change the interface to avoid the problems you had?”
90. Ensure that participants leave the test feeling no worse than when
they arrived.
Your goal should be to never make a participant feel that they failed or disappointed. You are
testing the interface, so no matter what the participant does it is correct. It would be unfortunate
to ruin a person’s day because of a test session.
DISCUSSING RESULTS
91. Follow the sessions with a findings workshop to build early

consensus among the group.
Schedule a meeting immediately after the test series where all product team members, including
developers, designers, managers, and so forth, who have watched one or more tests are invited.
Proceed as follows:
• Brief brainstorm — Ask each observer to write down all usability observations that they
consider important on sticky notes — one note per observation. The facilitator may also
put down their observations. Ask everyone to remain silent; discussions are not allowed
during this brainstorm.
Order NNG211105-5163-49141
• Share findings — Put up the sticky notes on a board or wall. No discussions.

• Secondary brainstorm — Add additional observations inspired by the observations
of others, if any. Let this process continue as long as people want to add findings. Still
no discussions.
• Group findings — Sort the findings in suitable categories.
• Prune — Eliminate duplicate findings. Elimination is allowed only if everyone agrees.
From this point on discussions are allowed.
• Name the categories — Use differently colored index cards for this purpose.
• Prioritize the findings — Give each participant 10 points that can be distributed over the
findings to mark the most important ones. The points can be shown as colored dots on
the stickies describing the findings. Points can be awarded to one finding or distributed
evenly or unevenly among 2–10 findings as desired. This will show which findings have
the most collective priority.
• Take a photo — Photograph the board and or pull down each group of stickies and store
in labeled envelopes. Use these artifacts as the basis for writing the test report. With this
method the report will mainly serve to record what was agreed during the meeting.
It’s best to convene this way directly following test sessions, so the findings are still fresh in
everybody’s mind. At the end of the meeting everyone will have a common understanding of
the most important usability problems in the interface. Corrections of the usability problems
can start immediately.
This method is sometimes called the “KJ–method” after the Japanese ethnologist Kawakita Jiro.
92. Downplay your role and opinions.

During the findings workshop, UX professionals should downplay their own role and opinions. It
is important that the facilitator mainly acts as a catalyst during this consensus building process.
93. Keep discussions of severity and resources separate.

Sometimes UX practitioners and their product teams find it difficult to agree because they have
different and hidden agendas. The point of departure for usability people is “How serious is this
problem for the user?” The point of departure for the product team is “How many resources are
necessary to correct this problem?” Both viewpoints are entirely legitimate, but they are often at
odds with each other and need to be addressed separately.
Order NNG211105-5163-49141
Reporting Results
Reporting Results
The primary purpose of a usability test is to cause beneficial changes to the user interface — not
to write a good test report.
Even the best test report is worthless if it does not cause beneficial changes to the user
interface. Communicating and selling test results to product team members is a crucial task for
usability professionals.
The facilitator must try to convince the product owners how important it is to allocate time in their
busy schedules to correcting the problems found.
REPORTING PROCESS
94. Ask your client about reporting requirements.

Above all, your report needs to meet your client’s needs. Give new clients a sample report and ask
them if the format is what they expect.
95. Quick and dirty reporting is sometimes needed.

In some situations, a simple email summary written within hours might be more effective than a
detailed report written within a week.
Depending on deadlines, you may have to send out initial results before you’ve had time to
completely analyze all of the data. For example, if a specific feature needs to be launched soon,
you might first review all session recordings when participants used that feature, so you can send
the needed findings as quickly as possible.
96. Do a quality and usability check of your test report.

Ask one or two experienced colleagues to read your usability report before you give it to the client.
Pay attention to comments from your users (developers, designers, managers) regarding the
usability and usefulness of your test reports.
Order NNG211105-5163-49141
Reporting Results
97. Record usability problems in the bug tracking system used by

your organization.
Put all usability issues into whatever bug tracking system is used for all other software problems.
This doesn’t guarantee that things will be fixed, but it does ensure that usability issues can be
tracked from a single location.
It also allows you to track the destiny of the usability problems.
98. Get involved in sessions where managers decide what bugs need to
be fixed.
You can lobby for serious usability bugs and increase the visibility of your usability efforts.
99. Let the project team comment on the usability report before you
show it to others.
The first version of a report sometimes contains simple misunderstandings. Make sure that
misunderstandings can be corrected without causing tension. It is unfortunate when project
teams feel bad about having been criticized unjustifiably because of a misunderstanding.
100. Make sure that you know who should get usability reports.
Make this clear with your sponsor. Some companies publish all reports company-wide; others
want only the immediate team to have the report.
101. Set up a database of reports or individual findings that is

easily accessible.
Your research is worthless if it just gets lost a few weeks after the study. Dedicate time and
thought to the best ways to share and democratize research findings. Some organizations have
set up databases of individual findings, so that designers can easily search for specific insights
they need. These databases are sometimes called “research repositories,” or “research insights
repositories.”
Taking this step also avoids redundant research. Maybe an in-depth study of a particular user
group has already happened in another department in your company, and a second study on that
group would be a waste of your time.
Order NNG211105-5163-49141
Reporting Results
102. Provide high-level analysis and executive summaries.

Not all stakeholders will have time to read the fine details of each usability finding in a report.
Reports should always include an executive summary that highlights top-level takeaways,
strengths, and areas for improvement.
However, for long reports with many subsections consider how else you might provide a macro-
level analysis for stakeholders to understand how the product tested without reading each
section in depth.
Example chart indicating how the software performed in specific areas. In the left
column are listed categories about the ease of various activities and the quality of
information provided. The analyst rated these general categories with a scale of
low, medium, and high.
Order NNG211105-5163-49141
Reporting Results
103. Consider creating a report that looks at problems across studies.

This cross-study report might highlight underlying problems like poor performance on registration
activities originating from a common module, or major consistency issues across varying parts of
a system.
REPORT FORMAT & CONTENT
104. Make it short.

Keep reports as brief as possible. The shorter the report, the more likely people are to actually
read it. Exact lengths will vary based on the size and scope of the study.
105. Consider the following usability test report format:

• Executive summary
• Table of contents
• Methodology
• Participant profiles
• Detailed findings, screenshots, quotes, and recommendations
• Appendix: Test script, including test scenarios
• Appendix: Screenshots annotated with key issues.
106. Include a short executive summary.

Describe:
• Top 3-5 successes
• Top 3-5 problems (“Opportunities for Improvement”)
• Your conclusions and recommendations
Focus on major usability issues. Avoid getting too detailed here. Discuss the issues that could
lead to likely product failure or user complaints in the marketplace, like “Three out of six
participants were unable to put products in the shopping cart.” Avoid micro-usability issues like
“Four out of six participants expected the company logo in the upper left corner to be a link to the
home page.”
Order NNG211105-5163-49141
Reporting Results
107. Classify the severity of problems.

The severity of the problems should be based on:
• How severely the user’s task was impacted (did it cause her to fail, or just some
minor inconvenience?)
• How many participants encountered the problem (is every participant running into this
issue, or just one?)
• How severely the company’s goals are impacted (is this a problem that directly impacts
revenue, or is it irrelevant to our goals?)
Distinguish among the following severity categories for problems:

• High severity (“Participants did not want to sign up for the service, because they didn’t
understand its value.”)
• Medium severity (“Participants were unable to easily locate our contact information.”)
• Low severity (“Participants didn’t think the photo on the landing page was relevant.”)
108. Include quantitative data.

Avoid usability reports that are just long lists of problems. Include simple quantitative data like:
• The number of people who successfully completed each task.
• The number of people who experienced a particular problem.
• The longest and shortest amounts of time it took participants to complete a task
In a qualitative usability test (as described here) you cannot make claims about proportions or
averages. It’s fine to say “4 out of 5 participants were successful,” but you should not report that
you found an 80% success rate unless you actually performed a quantitative usability test with
around 30-40 participants.
109. Mention positive findings.

While the emphasis in usability testing is on finding the problems, it is both useful and politically
advantageous to note what worked well.
Ideally there should be one positive finding for each problem. However, it is rare to have more
than one positive finding for each three problems. This ratio seems to be acceptable.
Make sure to mention specifics on positive findings, rather than just generalizing. Specificity
makes the comment more useful and seem more credible. A test report seems insincere if it starts
by saying “Generally, the participants were very happy about this website,” and then lists more
than 30 problems without any positive findings to substantiate the initial claim.
Order NNG211105-5163-49141
Reporting Results
Sometimes positive findings may manifest as an absence of complaints or problems, such as

participants not getting lost. It takes an effort to remember to look for things that didn’t happen,
but sometimes there is significant good news in these non-events.
110. Organize findings in a way that is useful to the particular audience.

Developers may like to see problems sorted by page or screen. Managers may want to see
problems sorted by severity in order to easily identify the worst problems found during the study.
Consider keeping the master list of problems in a spreadsheet so the problems can be sorted
on various columns, for example object, priority, problem category. Then duplicate the table and
sort it according to the particular audience. Furthermore, using one large table allows you to
add a new column for additional sorting. For example, you could add and sort by the projected
resources needed to fix each problem.
111. Include sufficient information to reproduce the test.

In the appendix, always include task descriptions and the instructions that you gave participants.
While most of your stakeholders may not care about those specifics, other researchers trying to
understand your findings might.
112. Include many screenshots.

Screenshots help to illustrate findings and reduce verbosity. They also serve to make the context
clearer to the reader. Screenshots are useful for readers who are not intimately familiar with the
product, as well as for later reference after the design has changed.
Almost every finding should have at least one screenshot paired with it.
When working on websites, make sure to grab screenshots the day you do testing, if possible,
because sites change content and designs often.
113. Develop a consistent report format.

Use a report format — and in particular a title page — that acts as a brand for your work. If you aren’t
comfortable with visual design, ask a visual designer for advice on how to improve the report format.
This isn’t just about branding, but also scannability and ease-of-use for your audience.
Order NNG211105-5163-49141
Reporting Results
114. Be careful with findings that were observed with only

one participant.
With qualitative research, it’s difficult to be confident that problems observed with only one
participant will be prevalent in the entire user population. It’s always possible that the problem
that participant encountered could be a one-off unique issue.
If the participant who encountered the problem didn’t perfectly fit the user profile (for example,
that person was less experienced in the field), then consider whether that participant is truly
representative of your users.
If a problem could lead to severe consequences, however, it still might be good to include it.
115. Consider using a table that indicates which participants had

what problems.
You might notice a trend based on the participant’s background. The table also helps readers of
your report in understanding the amount of data you have for each of your task related comments.
116. Provide precise problem descriptions, preferably with examples.

Avoid vague problem descriptions without examples that would probably not help a design team.
Example:
“Severe problem: Terminology was often confusing, especially when different

terms referred to similar features or the same feature could be accessed by
different terms.”
Two or three examples would have been useful in that example. Which terminology? What did
participants say was confusing about the language?
117. Use participant quotes extensively.

Well-selected quotes make the reader “feel the user’s pain.”
Maintain the integrity of quotes by using quotation marks to indicate the participants’ exact
words. If you paraphrase the participant, do not use quotation marks. This implies that you need
to mark quotes in your notes — after the test, you won’t be able to tell whether you were quoting
or paraphrasing the participant unless you take the time to review the test recording.
Consider slight modifications of the quotes if they have overly provocative language, for example
offensive words. The paraphrases should appear within [editorial brackets] if used inside
quotations, however.
Order NNG211105-5163-49141
Reporting Results
118. Include a recommendation with each problem.

If you are conducting testing for internal or external clients, provide actionable recommendations
for project teams to know what to do with a problem.
Be prescriptive, but also indicate that recommendations are suggestions, as some project
teams prefer to find their own solutions to usability problems, while others insist on getting
your expert advice.
Whether to include recommendations with the problems can be a political issue. Ask your project
team or your client how they would like you to report findings.
119. Provide brief notes about relevant basic UX or human

factors principles.
Help educate your audience about UX as they learn about your findings. Briefly explaining
relevant principles can help readers understand why things went wrong.
Example from a test of Disney.com:
“Four out of six participants did not understand what an ‘I like you Pooh Gram’
is. The detailed product description was not helpful either.
General usability principle: Speak the language of the users. Avoid

marketing speak.”
For a list of the ten most fundamental usability principles, see

https://www.nngroup.com/articles/ten-usability-heuristics/
Order NNG211105-5163-49141
Testing in a Lean UX or Agile Environment

Many teams conduct usability studies in lean and agile environments. In these settings, testing
methodologies will need to be adjusted to reduce overhead and work with the iterative nature of
the team’s sprint cycle.
120. Designate a regular cadence for testing during sprints.

Agile product teams should ideally include user testing within each sprint. If this is not feasible,
testing every other sprint must be a priority. Agile product development is centered around
continuous learning and improvement as the product evolves, so user testing must be a core
function in a team’s methodology. Designate a particular day each week or each sprint to hold
these test sessions, depending on your team’s schedule. When they occur regularly at predictable
intervals, it is easier for people to participate meaningfully and it will take less effort to convince
them to show up.
121. Instead of focusing on features, focus on research questions.

Agile product teams typically work from a product backlog made up of user stories and epics,
which represent upcoming features of the product. When discussing the upcoming backlog items
with your team, focus on writing research questions to capture what you don’t yet know.
For example, if you have Provide offline functionality as an item on your backlog, you’ll want to
do research to determine if users even want to use your product without network access. Write a
research question that captures this and sets you up to conduct research, such as “Will our users
want to use our application offline?”
Once you have a list of questions, you can prioritize them and decide which you’d like to explore
further through research.
122. Encourage cross-functional team members to observe and

participate in testing.
To move quickly in an agile environment, keeping everybody informed is important, so time
is not wasted documenting and communicating research findings. Having team members
available to observe and participate in sessions ensures everybody can walk away with a shared
understanding of how the prototype performed and quickly begin discussing what to do next.
Order NNG211105-5163-49141
123. Create logistics templates to make testing plans reusable and easy
to repeat.
Because you will be conducing test-based research in a regular cadence, enable yourself to
recreate and run these test sessions swiftly as you progress. Creating reusable planning and
notetaking templates (either in standard text documents or in a Kanban board) can help teams
achieve this. Templates can be easily duplicated, allowing team members to fill in the specifics for
that sprint’s test session.4 Consider including the following details in these templates:
• Goals — Realistic research goals for the week
• Links — Quick links to the screener, participant list, interview guide, consent forms,
and prototype
• Participant Details — Scheduled participants for each day of research
• Quotes — Important quotes tagged with the participant who said it
• Insights — Patterns of behavior shared across three or more participants
• Action Items — New features and next steps to take that address user insights
124. Define lean testing checklists.

Many agile teams find themselves sharing the effort of user research and testing amongst all
roles on the product team. To ensure testing is done correctly and consistently, create checklists
for team members to use to set up their own user tests. Consider the following checklists, along
with ideas for what can belong in each:
• Before Research Begins — Determine research goals, create screener, schedule
participants, determine compensation, print interview guides, and delegate roles to
team members
• Before Each Interview — Finalize tech setup, organize forms, welcome the participant,
start session recording (with consent), and brief participant on thinking aloud
• During Each Interview — Listen and guide the conversation, capture quotes and
reactions, and follow up on questions
• Post-Interview — Send incentive, follow up with participant (if necessary), regroup with
observers, and debrief
• After All Interviews — Review all notes, look for patterns across three or more
participants, prioritize insights, assign action items to team members, and share insights
with stakeholders
4
Medium, “Everything you need to plan and share user research with Trello”
(https://medium.com/product-labs/everything-you-need-to-plan-and-share-user-research-with-trello-41e6f101327e)
Order NNG211105-5163-49141
125. Designate tools for your team to collaborate in.

There are a wealth of web-based tools that make agile research collaboration more efficient.
Select a set of tools that support collaboration, cocreation, versioning, and so on. Ensuring each
team member knows where to go and has easy access to the right resources will help teams
remain efficient during sprints.
126. Leverage paper prototypes for quick insights.

In early sprints you may not yet have a digital prototype or product to test with. However, this
does not mean you cannot begin evaluating your ideas and eliciting feedback. You can start
collecting high-level behavioral observations on basic paper-based representations of your ideas.
Paper prototypes are good tools to help you create ideas and get feedback quickly without the
overhead involved in creating high-fidelity digital mockups and prototypes.
127. Consider the Rapid Iterative Testing and Evaluation (RITE) Method
for experienced researchers.
The RITE Method is similar to traditional usability testing, with the main difference being that the
prototype being tested can be changed as soon as a problem is identified, and a clear solution
is determined. In traditional user tests, you would test the same prototype with all of your
participants, but when using the RITE Method, you test with one user, learn about major issues,
iterate on the prototype, and then test again with another participant. You can continue with
additional participants until you feel good about the design.
Using this method allows you to test your improved design and uncover new problems rather
than just wasting time on many users who are likely going to uncover the same issue.
Since the RITE Method relies on identifying glaring usability issues from potentially one user, it
is recommended for experienced researchers who have conducted several user tests and can
identify a glaring issue from a user’s personal preference.
128. Create one-page research findings.

Each sprint’s test scope should be focused on whatever is relevant for that time period. This
means research findings will also be targeted and can be captured in a basic and consumable
format. Agile teams should aim to reduce the creation of documentation, while also keeping the
entire team informed and ready to move forward. Single-page findings documents can easily
serve to communicate insights gathered for the research questions in that sprint.
Order NNG211105-5163-49141
An example of a single-page research findings document.

This example highlights key takeaways — both quantitative
and qualitative — in a brief and easily consumable format.
129. Bring your findings to daily standup.

Your team’s daily standup is a great opportunity to communicate and reiterate important research
findings to your team. Since this time is already dedicated to the sharing of ideas and information,
utilize your time in this meeting to keep teammates aware of recent research or usability studies
on a regular basis.
Order NNG211105-5163-49141
How To Test on a Minimal Budget

Always consider the cost / benefit tradeoff for alternative (cheaper) test solutions.
130. Run at most six participants with one set of test tasks.
If you have resources to run more tests, continue after the most important usability problems
have been fixed to make sure the fixes actually work.
131. Take notes as the session unfolds.

This approach means you’re saving time, by avoiding the second pass through video later on. You’ll
may miss a few details compared to what you would learn from watching a recording after the test,
but chances are good that you’ll have sufficient time to record all problems that really matter.
Take notes even if you are recording. That way you are clearly showing the participants that their
views matter to you, and you have the recording to review later if there is an important moment
you missed or where your notes were not sufficient.
132. Explain why you are taking notes.

At the beginning of the session when you talk about what to expect during the session, be sure
to tell participants that you will be taking notes during the session. Forgetting to do so can
cause participants to feel insecure about their behavior or how they approach the activities. Set
participants at ease. Let them know that you are not taking notes about them personally. Rather,
you are taking notes about the interface.
133. Use one usability professional to run tests.

Choose a lean approach and run sessions with one person who acts as a facilitator and notetaker.
Two usability professionals (one facilitator and one notetaker) will do a better job than one, but
the result will not be twice as good. This approach increases the cost of each session. Using a
single professional, along with a recording of the session will allow that professional to quickly
jump into a video at points in time to catch a missed quote or evaluate a problem a second time.
Order NNG211105-5163-49141
134. Don’t use a market research or usability test lab.

You can run most tests just as efficiently in a meeting room or in the participant’s office — at
a fraction of the cost of a traditional usability or market research lab. If more than one or two
people want to watch the test, you can use a secondary monitor or a television with a long cord
to mirror the participant’s screen. With this inexpensive equipment you can allow project team
members to follow the test from a neighboring room.
135. Consider a portable discount usability lab.

A “portable discount lab” should consist of the following equipment, at minimum.
• An adequately sized laptop PC. We generally suggest using Windows devices, as these
are the most universally used. We suggest a 15” laptop at minimum, to ensure the user
can comfortably use the device and view content.
• A quality external webcam. Your computer may have an internal webcam, but it is best to
have a second webcam as a backup in the event the primary fails.
• A secondary monitor that is connected to the participant’s laptop to mirror the
participant’s screen for observers to view. Viewing on the secondary screen allows you to
observe without being uncomfortably close to the participant.
• A cable to connect the test computer and the secondary monitor.
• An external mouse and mouse pad. Not all users are comfortable using laptop trackpads.
• An external microphone. External microphones typically produce higher quality audio
recordings than the microphone in a webcam or a PC.
136. Talk to people who know users well, such as customer service
representatives (if you’re unable to talk to actual users).
Consider talking quarterly with different customer service representatives. Ask them to highlight
the top ten usability issues that come up in their conversations with customers. You can do a
briefing to the customer support team to explain what you are looking for first.
137. Consider remote testing.

There are many tools available to conduct usability testing remotely, which can often make for a
cheaper study option. When you ask participants to come to a physical lab, you often need to pay
more for this added effort, so testing remotely can save you money on participant incentives. In
addition, if you must rent lab space from a testing lab this can often be expensive. Remote testing
eliminates this expense.
Below we show some basic pricing estimates for both remote moderated and unmoderated research.
Order NNG211105-5163-49141
Low Estimate High Estimate
Video conferencing app $15 / month $30 / month
Recruiting fees $0 / participant $80 / participant
Incentives $80 / participant $250 / participant
Planning and setup 16 hours 24 hours
90 min / participant
Conducting sessions
(60 min sessions)
Analysis 8 hours 16 hours
Costs for 5-participant study $415 + 32 hours $1,680 + 48 hours
Remote moderated usability testing: A five-participant study might cost from $415 to $,1680 and might
require 32–48 researcher hours.
Low Estimate High Estimate
Platform subscription $0 / month $100 / month
Recruiting fees $0 / participant $80 / participant
Incentives $50 / participant $150 / participant
Planning and setup 4 hours 12 hours
30 min / participant
Watch recordings
(20 min sessions)
Analysis 4 hours 12 hours
Remote unmoderated usability testing: A five-participant study might cost from $250 to $1,250 and might
require 11–27 researcher hours.
Order NNG211105-5163-49141
Remote Testing
Remote Testing
138. Avoid recruiting professional testers.
Professional participants are people who participate in too many studies, too frequently. Often,
these people have learned too much UX, user research, or design to be considered to behave as
a “regular” person not in our industry. Or, they may be overly motivated to please the researcher,
or to move quickly through tasks just to get paid.
Try to screen these people out of your study. For example: Ask, “When was the last time you
participated in a study?” and exclude people if they’ve participated within the past 3 months.
139. Ensure that remote testing is the right method for your study.
Remote testing may not be the best method if your study includes:
• Very personal tasks or sensitive data
• A user population with poor internet access or low technical experience
• Children or teenagers
• A product behind a firewall or a paywall
If you have any of these situations and in-person testing is an option, it might be best to avoid
remote testing.
However, you might be able to make remote testing work for these situations if necessary. For
example, if you need to test with American children ages 5-7 years old, and you live in Europe,
an in-person study may not be practical. You just might have to do extra work in recruiting and
preparation to ensure that your participants’ parents will be available to help keep the child
focused on the study.
140. Take advantage of remote tool demos before committing to a

testing tool.
Most remote testing platforms and video conferencing tools offer free trials. Take advantage of
those trials, and experiment with the tool’s features, reliability, and stability before committing.
141. Run a pilot test to iron out technical or logistical issues.

Pilot testing is critical for remote usability testing (moderated or unmoderated). It can help you
identify problems in your technical setup, logical plans, or instruction wording. Take the time to
run 1-2 participants through your study before you begin testing all of your participants.
Order NNG211105-5163-49141
Remote Testing
142. If the testing tool’s chrome covers the interface, remind users to
hide the chrome while they work.
When participants are completing tasks through an unmoderated testing platform, or sharing
their screens through a video conferencing tool, those tools often have little toolbars or instruction
window panes (also called “chrome”). The chrome can cover substantial portions of the user’s
screen, and participants don’t always know that they’re allowed to minimize that window.
Remind participants (verbally in moderated sessions, or in the instructions in unmoderated

sessions) to minimize those toolbars so they don’t block the screen.
The screenshot above shows how the chrome from testing tool Userlytics, is
covering the website, including the navigation.
143. If you need certain technical requirements, include this in

your screener.
Many remote testing tools have specific device or operating system requirements. Make sure you
consider those requirements in your screener, so you don’t end up with a participant who can’t
actually participate because she has the wrong operating system.
Order NNG211105-5163-49141
Remote Testing
144. If you are testing a live product, notify the entire product team
about testing dates and check that nothing will be changed during
the study.
This is particularly important if you will be running the study over several days. Ensure that
nobody will be pushing code changes to the product while you’re testing.
145. Prepare scripts for all types of participant communications

in advance and save in a format that allows you to easily copy
and paste.
Particularly for moderated studies, you may need to communicate several times with your
participants leading up to the study. These may be delivered via platform message, email, or
verbally during the session.
The types of communications you’ll need to plan for may include:

• Confirmation: You’ve been scheduled, here’s how to prepare
• Reminder: Don’t forget, we have a session today
• Join the session: Here’s how to join the session, we’re ready to begin
• Introduction: Welcome, here’s what we’ll do today
• Task instructions and questions: The individual task instructions, along with any follow-
up or interview questions
• Conclusion: Thanks for participating, here’s your thank-you gift
Be sure to think through everything you want to say in each of those instructions, and write out
your scripts in advance to save you time.
Order NNG211105-5163-49141
Remote Testing
REMOTE MODERATED
146. When choosing a tool for remote moderated testing, consider

features needed for observers, the type of meeting setup you need,
and the experience you need for your participants.
For remote moderated testing, you can use a video conferencing tool or a dedicated remote
testing platform. Either way, consider the following features you might need.
Participants:
• Install an app or plugin
• Tech requirements
• Webcam sharing and recording
• Calendar plugins for easy scheduling
• Ability to allow participants to control the facilitator’s screen
• Mobile app support (for mobile moderated testing)
Meeting setup:
• Ability to record the session
• Chat
• Document sharing within the session
Observers:
• Hide attendee list
• Force muting
• Any limitation on the number of attendees or session length
147. Consider doing tech check sessions in advance.

It’s a good idea to schedule a short 15-30 minute session with each participant a day or two
before their actual study session. This approach allows you to check that each participant’s audio,
video, and internet connection are working well, which can save you time in the actual session.
This is particularly useful when you have high-priority observers in the sessions (like a client or
a person in a leadership position). You don’t want to waste their time trying to debug technical
issues that could have been avoided.
Order NNG211105-5163-49141
Remote Testing
148. Send technical requirements out in advance.

In your communications with participants leading up to the session, tell them the technical
requirements (device, operating system, internet connection, hardware). Ask them to download
and install any necessary applications in advance.
149. Specify upfront how you will provide participants with remuneration.
Immediate payment methods are recommended.
If you are conducting moderated testing through a testing platform, participant payments
are often built into the service. However, if you are not using a testing platform, but instead
conducting remote moderated sessions over video conferencing tools, providing payment is
something you will have to plan for and manage on your own. It’s best to select a payment
method that allows you to immediately send payment following the session, using checks sent
via the postal service is inconvenient for users, who must trust that you will follow through with
payment, and there’s a chance the payment will get lost.
There are many digital payment channels available. We recommend choosing 2-3 that you are
willing to use and leave it up to the participant to dictate how they would like to receive payment.
When advertising and screening for participants, be sure to specify which channels you can use
to provide payment.
150. Set your computer to do-not-disturb mode and/or shut off

any distracting notifications or applications that could interrupt
the session.
Consider also using headphones so that any noise or notification from your computer will not be
picked up by your microphone. Make sure your microphone doesn’t pick up the sound of typing
from your keyboard, which can be distracting for participants
151. Determine how you will send and receive consent to record the user.
If you need to record the sessions for any reason, it’s important to ask your participants for their
consent to record their face and voice, due to privacy concerns and regulations. You should
communicate and receive this consent in writing as a way to record this agreement for the safety
of both parties.
We recommend getting user consent before the sessions begin if possible. Doing so during the
session will take up valuable time and requires both parties to juggle multiple technologies.
Order NNG211105-5163-49141
Remote Testing
If you have access to online document signing services, these can be a convenient way to facilitate
a way for users to digitally sign your consent form. However, if you do not it is also possible to ask
users to print a PDF version of the form and return a photo of the signed document.
152. Have observers arrive well in advance of the session to ensure their
technical setup and settings are correct before participants arrive.
You and your colleagues and stakeholders should do a trial run of the sessions before the actual
participant sessions take place. This will allow everyone to get their technical setup correct and
practice how to operate things on the day of the test.
In addition to this trial run, you should also request that all observers arrive at the meeting about
20 minutes prior to the participants. Even with a trial run, there is likely something that will be off
on the day of the session.
Meet ahead of time and ensure everything is ready to go before the session is set to begin. It’s
unprofessional and time consuming to deal with your own staff’s technical issues while you have
your participant online.
If your moderated testing tool has a waiting room feature, you can use it to double-check that
your observers are set before admitting the participant to the meeting session.
A few tips for observer technical setup during remote sessions:

• Observers should have their webcams off.
• If possible, hide observers from the attendee list.
• Ensure all observer’s microphones are set to mute.
• If possible, meeting hosts should change global settings to disallow certain actions
for observers.
153. Plan to use a specific private channel for any communications

between the facilitator and observers during sessions.
There are situations when observers and facilitators may need to communicate during sessions.
Avoid influencing the participant or making them uncomfortable by hiding that communication
from them.
Specify ahead of time how your testing team will communicate with each other outside of the
session. Some testing platforms may have private messaging, but for messaging between more
than one person this may need to be done via text or any other messaging platform.
Order NNG211105-5163-49141
Remote Testing
If you intend to be sharing your screen at all during the session, be cognizant of where your
messaging application is on your desktop and ensure it is not shared. You do not want your
user to see a messaging thread between you and your colleagues. An even safer approach is to
designate a secondary device dedicated to this communication.
REMOTE UNMODERATED
154. When choosing a tool for unmoderated testing, consider

recruiting needs, what type of data you need, and how your study
must be designed.
Every study has unique needs. Many factors such as, audience requirements, data requirements,
and team collaboration can influence what you need in a remote testing platform. When looking
for a testing tool, consider the attributes below and use these to identify a platform that meets
your needs.
Recruiting
• Participant panel needs (size, location, demographics, cost)
• Are there quotas for multiple user types?
• Are custom screening questions allowed, and is there an extra cost?
• Does the platform support multiple languages?
• Can you bring your own users?
• Is there integration with external recruiting panels?
Study Setup
• What testing setup does the platform support? (desktop, mobile, app, prototype)
• Do you need to support persistent access to study instructions?
• Can you customize the welcome screen and final screens?
• Is there skip logic to personalize the task flow?
• Can you randomize task order?
• Are you able to share and collaborate with team members in the project?
Qualitative Data
• Does the platform record the user with a webcam?
• Is note-taking supported?
• Are notes timestamped?
• Can you export notes?
Order NNG211105-5163-49141
Remote Testing
• Can you download the recordings?

• Can you share recordings with URL links?
• Does it facilitate the production of a highlight reel or video compilation?
• Are automatic transcription services available?
Quantitative Data:
• Does it provide support for simple rating questions?
• Can you customize rating questions
• Does it record time on task?
• Does it identify and filter out speeders and cheaters?
• Can you export this data?
155. Consider using a remote unmoderated testing platform that allows

you to recruit your own participants and invite them to the test.
Most remote unmoderated testing platforms have their own panels, which you can use to recruit
participants for your study. However, if you have a very specific user group (for example, orthopedic
surgeons living in North Carolina), you’ll find that those panels may not be very helpful.
It’s a good idea to choose a platform that allows you to bring your own participants. That way you
can recruit through a recruitment agency, within your product, or using your own contacts. Many
platforms offer this option.
156. Keep tasks specific and direct.

Because you won’t be actively facilitating the session, make sure your remote tasks have a very
specific goal. Broad and open-ended tasks can make it difficult for unmoderated participants to
determine when to stop and move to the next activity. They may spend longer amounts of time in
areas that are not particularly interesting to you.
So instead of , “See if the TSA blog gives any helpful advice for traveling over the holidays.” Try
something with a clear endpoint like, “Find out how much a Nest Thermostat costs to purchase
and install.”
Order NNG211105-5163-49141
Remote Testing
157. Have a teammate proofread tasks and instructions before

launching a test.
It’s always good to have another person to look over tasks and instructions before submitting
them. Once you’ve submitted a test, most platforms will not allow you to make changes. Have
colleagues read over the instructions and activities to ensure they make sense to another person
before submitting. Teammates can also provide feedback on task wording, so your task will be as
successful as possible. You don’t want to waste time and resources by finding out about issues
with tasks and instructions after the tests have already been completed.
158. Remind participants to limit background noise.

Conducting unmoderated tests often means you’re relying on recordings of the user’s screen and
think-aloud narrative after they’ve completed the activities. Users may not know how important
it is for you to hear them clearly in the recording, so they may have music playing or other
background noise.
For this reason, always remind users to limit the background noise in any instructions provided at
the beginning of the session.
159. Try to analyze as soon as possible following the test recordings.

With unmoderated testing, you will typically receive recordings of your participants performing
the tasks. It can be easy to put off the analysis of this data because it is safely stored for later.
However, the more time that passes between study kickoff and analysis, the more difficult it is
to refamiliarize yourself with the goals and context of your study. When conducting moderated
remote testing, you have at least observed the sessions as they took place, so it’s simpler to
extract insights later on. With unmoderated, you are removed from the session, so extracting
insights is easier if you do it soon after sessions have finished.
Order NNG211105-5163-49141
Remote Testing
160. Don’t just use the stock charts the remote unmoderated platform
gives you. Download an exported spreadsheet, remove outliers, &
make your own charts.
Many remote unmoderated testing platforms have built-in quantitative analysis tools. While
these are nice, they probably aren’t comprehensive enough to do all of your analysis for your
quantitative remote unmoderated study.
You’ll want to download the spreadsheet version of your data and remove any outliers or cheaters
from your data.
Create your own data visualizations and charts with the cleaned-up data, and make sure you
calculate confidence intervals and statistical significance where necessary.
Order NNG211105-5163-49141
Testing with Experienced Users

There are several kinds of experience:
• experience with a particular version of a product
• experience with other similar products
• general domain experience
This section deals primarily with participant’s experience with the product you are testing.
161. Always test some expert users.

Experts can reveal usability problems that new users are unlikely to experience, especially those
dealing with efficiency.
162. Consider the internal and external marketing effect of testing with
experienced users.
Experienced users and expert users often have the ear of management. In some organizational
settings, it could be a nice bonus if those users suggest further improvement of the product to
management based on their experiences during test sessions.
163. Include both a novice and an expert user in your pilot testing.
Pilot testing is critical for any usability test. It can help you identify problems in your technical
setup, tasks, and prototype if you’re using one. Take the time to run at least 2 participants
through your study before you begin testing all of your participants. Choosing a novice and an
expert ensures you will get varying perspectives on your test plans.
164. When recruiting, ask contacts to suggest known experienced users

to you.
Your client contacts or whoever is familiar with the user audience you are testing should know
of some users that are experienced with the interface you are testing. These suggestions are
valuable to ensure you have this subset in your test sample.
Order NNG211105-5163-49141
165. Ask each participant to bring typical examples of their work related
to your target product.
For some scenarios such as internal software testing, using actual use cases for testing and
observation is ideal, rather than writing faux tasks and asking participants to take action. If
you are testing a live product that people are currently using, ask them to come prepared to
complete that work during the session. This may include bringing documents and printouts or
representative files (as long as they are not proprietary). Ask for a variety of samples; one typical
example, one easy example, and one difficult sample.
166. Ask participants to postpone work they will do with the product or
website, so they can do this work while you are present.
Asking participants to carry out their own tasks during the usability test will garner the most
organic behaviors. Ask participants if they have upcoming work they intend to use the system
for. Ask them if they can wait to do the work until the session, so you are able to observe real
scenarios instead of manufactured ones.
167. Ask participants about typical and critical tasks.

Focus on:
• Typical tasks for the participant. We are not interested in the participant’s perception of
what might be typical tasks for others.
• Critical tasks — tasks that people don’t do every day, but when they do perform them,
they are critical.
• Untraditional applications of the website (unintended or novel uses).
168. Quiz experts about the size of data sets.

A flaw in some usability tests is the use of a sample database that is unrealistically small, which
could affect the validity of the study.
169. Consider asking expert users these questions: to get their input.
• “What are the three most important changes you would like to see?”
• “What are the three things you like most [in this product]?”
• “What small issues [in this product] annoy you?”
Order NNG211105-5163-49141
170. Ensure faux data in prototypes is realistic.

When testing with prototypes you may not be able to use actual data from a database. Instead
many teams choose to populate their prototypes with dummy data for testing purposes. Double
and triple check the data you have in the prototype before testing with users.
If the data in the prototype is not accurate or represents impossible situations, you risk losing the
entire test session. Do not ask users to pretend the data is accurate, as this is very difficult to do.
Many users will get hung up on data that couldn’t possibly exist in a normal context, and this will
derail your session. They may not be able to proceed and provide valuable feedback on what you
are interested in testing.
Order NNG211105-5163-49141
International Usability Testing

171. If your budget doesn’t allow for testing in each target country, at
least have a local usability person review the site for potential
cultural issues.
Testing is better, but it’s not always possible. An experienced, local usability person may pick up
minority cultural viewpoints that a participant, carefully chosen to be typical, may not be aware of.
172. Don’t assume that the USA, UK, and Australia are the same just
because English is their primary language.
These countries have different dialects and expressions.
173. Run the test in the native language of the user.

Just because it may be possible to run usability tests in English doesn’t mean that it’s a good idea.
Although many local users may speak English deceptively well, almost all of them will be more at
ease in their native language. Local users who are fluent in English may be atypical users, also.
When you run tests in English, native users are likely to suppress comments you would consider
important because they can’t easily express themselves in a foreign language. People are more at
ease when expressing tactful criticism in their own language.
Even the test of an English interface should be conducted in the local language. This may mean
you need to hire a local consultant, as discussed in the next guideline.
174. Hire a local usability specialist to run the tests.

If your facilitators cannot speak the local language, you may need to hire a local consultant. It’s
important to try to find someone who can speak the language and is experienced in facilitation.
175. Arrange for one person to supervise all usability tests

across countries.
In most cases it will be necessary for an experienced usability professional to supervise the test
staff in each of the countries where testing will take place. Supervision is required to ensure
similar testing practices and to observe cultural differences. If budget allows, consider having
the supervisor travel to all of the test countries. However, this is a situation where remote
moderated testing will likely be the most practical and cost-effective option, to reduce travel.
Order NNG211105-5163-49141
176. Consider using simultaneous translation.

Professional simultaneous translation is required whenever somebody who watches usability
tests in other countries does not have a working knowledge of the local language.
For example, an American company wants to do usability testing in China. They should
also coordinate the assistance of a translator to observe the test session in Chinese and
simultaneously translate to English for the English observers.
Doing this in real-time is important, because it allows the usability professionals to react and
adjust in the moment if necessary. They can suggest prompts and alternative tasks if necessary.
This would not be possible if the translation was done based on a recording.
Be aware, however, that those translations might be imperfect. The translator might have to speak
quickly to keep up with the pace of the study, and even small mistranslations can impact findings.
For example, consider the difference between a participant saying, “I would use this product” vs.
“I have used this product.”
177. Ask about the native language of local providers you seek to help
coordinate your particular test.
You may need to request the help of a local firm or market research lab you employ to help prepare
the test and recruit participants. Check to see they have staff available who are native speakers of
the target language. Their help will be needed to translate and prepare the tasks for your study.
178. Ask for task translations and check the quality in advance.
You probably will supply the tasks in English to be translated into the local language. Ask for a
copy of the translated tasks. If you don’t understand the local language, have the tasks translated
back into your language by a third party, possibly a colleague.
Then check the twice-translated test tasks for important changes in content and for the usual task
problems such as hidden clues and unclear instructions.
179. Prepare a detailed facilitator’s guide that describes how you want
the tests to be run.
The facilitator’s guide must spell out everything in detail for the hired usability professional,
anticipate all contingencies, and provide a backup plan for potential issues.
Stress an absolute, authoritarian adherence to the tester’s guide. Otherwise some of the testers
may take small liberties with the administration of the test. For example, testers may omit one or
two scenarios because they decide that some of the scenarios do not apply to their situation.
Order NNG211105-5163-49141
180. Prevent trivial international usability problems by using

appropriate checklists.
Use, for example, the extensive checklists in the International Users report in the Nielsen Norman
Group Ecommerce User Experience series5. Most of the findings of that report also apply to
non-ecommerce websites.
Examples of international usability problem areas:

• Formats for addresses and dates
• Currency issues and currency conversion
• Weights and other measures
• Global English
Here’s an example of advice from that report:
On written test tasks, consider showing the month in text form. The only widely used, clear
way to differentiate between day and month is to spell out the month name. For example,
neither 2 Nov 2020 nor 11 Feb 2020 can be misinterpreted. As a bonus, this date format
doesn’t need punctuation.
5
https://www.nngroup.com/reports/ecommerce
Order NNG211105-5163-49141
Assessing the Quality of a UX Research Agency

Sometimes you must use an external research agency to carry out usability testing for you. The
following tips will be useful in assessing potential agencies or consultancies.
181. Ask for references and follow up on one or two of them.

This is the most reliable way to find out if past clients were impressed with the agency’s work.
182. Request a sample usability testing report.

Any reputable research agency should have at least a few prepared case studies that they can
share with you.
183. Notice whether their sample usability report focuses too much on
visual design.
An established and qualified research agency should provide findings on all aspects of an
interface: interaction and content, as well as visual design. If the report focuses too much on
color or branding, they may not have an adequate background or understanding of human-
computer interaction.
184. Check the recommendations in the sample usability report:

Are they detailed enough to follow for people who were not
involved in testing?
You want to see recommendations that are detailed enough to understand, and actionable
enough to give you an idea of what to do next. However, recommendations that are too specific
may be problematic as well – it’s up to your design team to decide exactly how they want to
address the identified problems.
185. Inquire about the experience and qualifications of the facilitator

who will run your study.
An experienced facilitator is critical to the success of a study. It often takes years of practice
before a facilitator is good enough to run a test without inadvertently biasing the results.
Order NNG211105-5163-49141
186. Notice whether they have any requirements or questions for you.
A good research agency will have certain requirements of its clients. If they’re willing to accept
you as a client without any standards whatsoever, it may be a sign that they’re too desperate.
They should also question you carefully about:

• Your specific goals for running a usability test
• The most important features or tasks in the product
• The characteristics and needs of your users
• Your exact reporting requirements
If the agency doesn’t ask about these things, it can be a sign that they don’t completely
understand how important context can be in running a study.
Order NNG211105-5163-49141
References
References
BASIC TEXTBOOKS ABOUT USABILIT Y TESTING

1. Joseph S. Dumas and Janice (“Ginny”) Redish
A Practical Guide to Usability Testing. Intellectbooks 1999.
http://www.intellectbooks.com
ISBN 1–84150–020–8
(US distributor: ISBS, Portland, OR, 1–800–944–6190)
2. Jeff Rubin
Handbook of Usability Testing: Second Edition. Jeffrey Rubin, Dana Chisnell, Wiley, 2008.
ISBN 978-0470185483
3. Jakob Nielsen
Usability Engineering. Academic Press, 1993
ISBN 0–12–518405–0
POLITICS OF USABILIT Y
4. Deborah Mayhew
The Usability Engineering Lifecycle, chapter 18, “Promoting and Implementing the Lifecycle.”
Morgan Kaufman Publishers, 1999
ISBN 1–55860–561–4
5. Deborah Mayhew
Strategic Development of the Usability Engineering Function. interactions, volume VI.5,
September 1999.
ETHICAL GUIDELINES
6. Association for Computing Machinery (ACM). ACM Code of Ethics and Professional Conduct.
https://www.acm.org/code-of-ethics
7. American Psychological Association (APA). Ethical principles of psychologists and code of conduct.
http://www.apa.org/ethics/code.html
Order NNG211105-5163-49141
About the Authors
About the Authors

Kim Salazar is a Senior User Experience Specialist with Nielsen Norman Group. She works with
clients in a variety of industries and presents regularly about webpage UX design, omnichannel
user experience, personas for user-centered design, and creating effective UX deliverables. Kim
combines her background as a developer and education in Computer Science with her user
experience expertise, particularly around complex applications, to bring well-rounded insights to
her work.
Prior to joining NN/g, Salazar worked as a User Experience Designer for Berkley Technology
Services, focusing on design and usability for enterprise software solution for large property
casualty insurance corporation, W.R. Berkley Corporation. Before that, she worked in a similar role
for a digital advertising agency doing research and UX design for clients in a variety of industries
including, food and beverage, financial services, insurance, and telecommunications. She also
previously held a variety of software development roles at Principal Financial Group.
Salazar holds a B.S. in Computer Science from Central College, and master’s degree in Human
Computer Interaction from Iowa State University.
Kate Moran is a Senior User Experience Specialist with Nielsen Norman Group. She conducts
research and leads training seminars to help digital product teams expand and improve their UX
practice.
Kate has extensive experience conducting user research to guide UX strategy for websites and
applications. She provides UX advice to clients from a wide variety of industries, including finance,
healthcare, government agencies, ecommerce, B2B, and nonprofit organizations.
Kate’s recommendations and research findings are informed by her background in information
theory and design, as well as her development experience. Prior to joining NN/g, Kate worked
at IBM, first as a Web Content Manager and then later as a Front-End Web Developer. She also
worked in UX design for a small web design agency.
Kate holds a Master’s degree in Information Science with a specialization in Human-Computer

Interaction from the University of North Carolina at Chapel Hill.
Kate is based in Raleigh, North Carolina.
Order NNG211105-5163-49141
Acknowledgements
Acknowledgements
We would like to acknowledge the following usability professionals for their excellent work and
contributions to the first edition of this report.
• Maria Rosala, for her help in authoring tips for testing ethics
• Rachel Krause, for her help in authoring tips for testing in a lean UX and agile environment
• Rolf Molich, the original author of this report’s first edition. Rolf owns DialogDesign
in Denmark and has been working in usability since 1984. He has previously been a
successful independent usability consultant for more than five years, head of IT for a small
bank, and responsible for development methodology and user interface standards at a
large insurance company. He is currently a visiting lecturer in usability engineering at the
Technical University of Denmark.
Molich is the co-inventor of the heuristic evaluation method (with Jakob Nielsen), and
is the author of the best-selling Danish book User Friendly Computer Systems, which
has sold more than 25,000 copies to date. He also conceived and coordinated the
comparative usability evaluation studies CUE-1, 2, 3, and 4 where multiple professional
usability teams tested the same applications.
Molich was a principal investigator in NN/g’s large-scale usability test of 20 US

ecommerce websites involving more than 60 users. His focus areas in this study were
usable search, trust and international usability.
Molich has an M.S. in Software Engineering from the Technical University of Denmark from 1974.
• Chauncey Wilson
• Marie Tahir
• Carolyn Snyder
• Kara Pernice
• Caroline Jarrett
• The CUE (Comparative Usability Evaluation) teams
• The professional CUE team captains were Scott Butler, Ian Curson and Nigel Bevan, Erika
Kindlund and Dana Miller, Jurek Kirakowski, Barbara Karyukina, Klaus Kaasgaard and Ann
D. Thomsen, Lars Schmidt, Meghan Ede, Wilma van Oel, Joseph Seeley, and Kent Norman.
Order NNG211105-5163-49141
About NN/g
Order NNG211105-5163-49141
A Leading Voice In The Field of User Experience Since 1998
A Leading Voice In The Field of User Experience

Since 1998
• Conducting groundbreaking research. Every week, somewhere around the globe, NN/g
team members are conducting user experience research. Our experts are experienced at
executing dozens of research methodologies. We know when and how to apply each one,
resulting in trusted insights that inform everything we do.
• Evaluating interfaces of all shapes and sizes. We’ve tested thousands of different user
interface designs over the years. Every year we evaluate hundreds of interfaces, adding
to our collective understanding of design and user behavior. Our experts work with users,
interfaces and organizations from all over the world, widening our collective understanding.
• Insights with more depth. We don’t have a narrow focus. Our observations are the
result of the cumulative findings across a large volume and variety of circumstances,
making them highly valid and comprehensive. For everything we publish, we combine our
collective knowledge of longstanding patterns with any new findings we uncover from our
targeted research studies. This allows us to interpret observations through the lens of the
dozens of similar things we’ve seen before to conclude their importance and validity. So
this report is based not only on our most recent research on the topic, but more than 20
years of UX knowledge.
OUR EXPERTS DO IT ALL

Our experts are researchers, educators, and UX Design practitioners all in one. Everything we do
informs the 3 pillars of our business, research, training, and consulting.
• Research — Every person on our team conducts research. We tackle every topic in UX
from the most evergreen challenges like information architecture to the most recent and
emerging trends.
• Training — Our UX Conference training events take place in cities around the world and
are also offered as virtual events that can be attended from anywhere.
Attendees learn from NN/g experts and invited industry peers, leaving with practical skills
that can be applied immediately.
With more than 40 courses available, there’s something for everyone. Courses are
taught in areas such as: research practices, design processes, working with stakeholder
and product teams, among other things.
Order NNG211105-5163-49141
• Certification — Our certification program helps UX professionals quickly gain skills

and credibility. By taking 5 courses and passing related exams, practitioners earn
NN/g UX certification and badge, a recognized credential in the industry.
• In-house Training — Many of our courses can be taught at your location and
customized to fit your unique needs. In-house training is ideal for teams that
want to spread user experience perspective throughout a large group and those
working on large projects that need to kick start the creative process and head in
the right direction.
• Consulting — Our experts are available for custom consulting. Our services include but
are not limited to:
• Design Reviews and Competitive Analysis (starting at $38,000 USD)
• Usability testing (starting at $20,000)
• Benchmarking (starting at $45,000)
PUBLICATIONS AND CONTENT

Over the years, our experts have developed a comprehensive body of knowledge and UX
resources. We’ve taken the results of our decades of research and distilled it down into actionable
guidelines, best practices and proven methodologies.
Articles and Videos

Over the years, we have created one of largest and most comprehensive sources of free user
experience content and research insights.
• Articles — Each week we publish new articles on current UX topics available to
practitioners around the world at no cost.
• Videos — In addition to our free articles, we produce a variety of short topical UX videos
posted to our YouTube channel weekly.
Reports
Our research library contains more than 60 published reports and books addressing a variety of
topics including but not limited to the following:
• Doing UX in Agile Environments
• UX Design for specific audiences (e.g., children, college students, seniors, people with
disabilities)
• Information Architecture
• B2B Websites
Order NNG211105-5163-49141
• Corporate Websites
• Ecommerce UX Design
• Marketing Email and Newsletters
• Intranets
• Non-Profit Websites
• University Website
Shop for reports at https://www.nngroup.com/reports.
Our Free Newsletter

Nielsen Norman Group’s Weekly Newsletter includes summaries of our latest research and
insights published each week. Subscribe at https://www.nngroup.com/articles/subscribe.
Order NNG211105-5163-49141
GROUP LICENSE
For Nielsen Norman Group Reports and Videos
You have purchased a report or video with a group license. This means that you have the
following rights:
This is OK:
• You are allowed to make as many backup copies as you want and store the file(s) on as
many computers and servers as you want, as long as the workforce of the organization
that bought the group license only accesses all of these copies.
• You are allowed to place the file(s) on your intranet, a shared drive, or a password-
protected website as long as access is restricted to the workforce of the organization that
bought the site license.
• You are allowed to print out as many copies of the group license report(s) as you want, as
long as these printouts are only given to the workforce of the organization that bought the
group license.
This is NOT OK:

• You may NOT place the file(s) on an extranet, the public internet, or any other file sharing
or file distribution mechanism where the file(s) can be accessed by users who are not part
of the workforce of the organization that bought the site license.
• You may NOT distribute printed copies or electronic copies to anybody who is not part
of the workforce of the organization that bought the site license. By “the workforce of
the organization,” we mean your employees, on-site contractors, and others who are
essentially treated as employees with access rights to your intranet.
Special rules for universities:

Non-profit universities can treat registered students as part of “the workforce of the organization” for the
purpose of this group license. This means that they can give copies to their students and can place the files
on networks that are closed to the general public but accessible for students. If you take advantage of this
right to distribute the report(s) to your students, please make sure that they are informed that the information
is copyrighted material and that they cannot share their copy with people outside the university.
For-profit educational organizations are NOT allowed to distribute the report(s) to non-employee course
participants without paying an additional fee. If interested, please contact support@nngroup.com for
permission or if you have any questions about the NNG Group License.
Order NNG211105-5163-49141
© Nielsen Norman Group | www.nngroup.com

Perfecting Your Usability Studies 2ndEdition-Group

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Perfecting Your Usability Studies 2ndEdition-Group

Uploaded by

Copyright:

Available Formats

Perfecting Your Usability Studies

186 Tips and Tricks for Better Usability Testing

Kim Salazar and Kate Moran

48105 Warm Springs Blvd., Fremont, CA 94539-7498 USA

Report Authors: Kim Salazar, Kate Moran

Writing Good Tasks 29

How to Perform Tests 33

Testing in a Lean UX or Agile Environment 55

How To Test on a Minimal Budget 59

Testing with Experienced Users 72

International Usability Testing 75

Assessing the Quality of a UX Research Agency 78

About the Authors 81

I recommend allocating a few percent of a company’s UX resources to activities that do not

The main criteria for including a tip in this collection are:

2. Check the quality of your work processes regularly............................................................. 17

3. Look for ways to maintain and improve your skills.............................................................. 17

4. Avoid making design decisions based solely on user opinions......................................... 17

5. Make usability test results usable for your audience.......................................................... 18

6. Check the usefulness and usability of your work................................................................. 18

7. Avoid personal opinions, and base suggestions on data whenever possible................. 18

9. Position yourself as an ally, not as an enemy or the police................................................ 18

10. Build alliances.......................................................................................................................... 19

11. Pick your battles; you can’t win them all............................................................................... 19

12. Build trust by being completely open.................................................................................... 19

13. Make the consequences of bad design visible.................................................................... 19

16. Familiarize yourself with recognized ethical guidelines...................................................... 20

17. Always ensure you have informed consent.......................................................................... 20

20. Make the first task simple....................................................................................................... 21

21. Observe your own rules for handling video recordings...................................................... 21

24. Set up a table in a public place.............................................................................................. 22

26. Ask for help from the sales department................................................................................ 23

27. Intercept users in their natural environment........................................................................ 23

28. Approach internal users through their managers, not directly.......................................... 23

29. Approach organizations (churches, charities, and so forth).............................................. 23

30. Use your existing networks and tools.................................................................................... 23

31. Avoid recruiting your friends and relatives........................................................................... 24

34. Ask participants to suggest other participants.................................................................... 25

35. Be careful in re-using participants from previous tests...................................................... 25

36. Keep a database of previous participants............................................................................ 25

37. Send a thank-you email after the test................................................................................... 25

39. Provide detailed instructions on how to get to the site...................................................... 26

40. Send a confirmation message to each participant immediately after recruiting............. 26

42. Check the comprehensibility of the confirmation message................................................ 27

43. Stress that “Your participation is particularly important to us.”........................................ 27

44. Provide a reasonable incentive.............................................................................................. 27

Writing Good Tasks 29

47. Test critical tasks...................................................................................................................... 29

48. Test edge cases........................................................................................................................ 29

50. Write tasks that have context................................................................................................. 30

51. Include confirmation or verification emails, authentication messages, and

52. Write tasks that begin with triggering assets or interactions............................................. 31

53. Avoid tasks you think are humorous..................................................................................... 31

54. Avoid hidden clues in task descriptions............................................................................... 31

How to Perform Tests 33

61. Ask your product team to observe the tests as a group..................................................... 34

63. Practice everything the day before the test.......................................................................... 35

65. Make a list of things to do between tests............................................................................. 35

66. Ask participants to do the paperwork before the test......................................................... 35

68. Train observers in usability test etiquette............................................................................ 36

70. Align on a notetaking strategy or framework....................................................................... 40

71. Avoid nondisclosure agreements whenever possible......................................................... 41

72. Sit next to and slightly behind the participant..................................................................... 41