Analyzing User Tests!

• 5-7 test sessions!

• 25-50 tasks attempted!
• 4-10 hours of video!
• A pile of logging sheets!
• 5-14 questionnaires!
Remember Your Goals!
•  “Can user group X use system Y to do activity Z?”!
•  Hopefully!
•  You recruited representatives of group X!
•  Your tasks are a good representation of activity Z!

•  Well, how did they do?!

•  Usually it’s not 100% clear!

•  Some users did well on most tasks!
•  Some users did well on some tasks!
•  Some users didn’t do well on most tasks!
Analyzing User Tests!
• Collate baseline statistics!
•  Task success/failure!
•  Errors!
•  Timing!
• Review critical incidents!
•  Where did breakdowns occur? Why?!
• Collate questionnaire results!
• Interpret debrief responses!
Task Success: Simple Case!
8

0
Task 1 Task 2 Task 3 Task 4

Number of par8cipants who succeeded at each task

Task Timing: Simple Case!
200
180
160
140
120
100
80
60
40
20
0
Task 1 Task 2 Task 3 Task 4

Average 8me taken for each task

Task Success: Non-binary Outcomes!
Summarize Subjective Feedback!
•  Questionnaire responses (e.g., SUS scores)!

•  Responses to high level questions!

•  What do you think system does well?!
•  Where does the system needs the most improvement?!
•  Other noteworthy responses—especially repeated responses!
Explaining the outcomes!
•  Review all critical incidents!
•  Group together similar incidents!
•  Did multiple people experience the same problem?!
•  Do problems reoccur in multiple tasks?!

•  Figure out!
•  Why did problems happen? !
•  What usability principles, if any, were violated?!
•  Did multiple problems stem from the same root causes?!
For Each Issue!
•  Assess severity!

•  Prioritize!

•  Identify Key Findings!

For Key Findings!
•  Clearly describe the problem, well enough to fix!

•  Provide evidence!

•  Suggest a course of action!

Assess Severity!
1 = cosmetic problem; no real usability impact!
2 = minor usability problem; fix if there is time!
3 = major usability problem; important to fix!
4 = usability catastrophe; imperative to fix!
Prioritize!
•  Identify 5-10 “key findings”!
•  The most severe problems!

•  Key findings will be highlighted and detailed in your report!

•  Less severe findings will be listed in an appendix!

Describe the Problem!
•  On what screen/page/interaction does it occur?!

•  Are there particular conditions under which it occurs?!

•  Provide a screen shot (or video)!

Provide Evidence!
•  Describe critical incidents!

•  Indicate if problem led to task failure or extended time!

•  Indicate if problem led to user frustration, perhaps linked to

lower subjective scores!

•  Provide quotes from think aloud/debrief if relevant!

Suggest a Course of Action!
•  Possible sources for suggestions:!
•  A best practice seen in other products!
•  A design principle or heuristic!
•  Your own brilliance!

•  Sometimes the best suggestion is more research!

•  An additional study!
•  An iterative design process!
Less Severe Problems!
•  Describe where found, description, and severity!

•  Provide a table/spreadsheet as an “appendix”!

Reporting!
•  Format determined by audience and purpose!

•  Personal project: prioritized list!

•  Internal team, friendly: key findings, and less-severe list!

•  Internal team, skeptical: mostly formal report, evidence is key!

•  External stakeholder: formal report, more emphasis on method !