What to look for!

•  Were people able to complete the tasks?!

•  Did it take them an exceptionally long time?!

•  Did people make errors? Did they know it? Did they recover?!

•  How did people feel about the experience?!

What to look for!
•  Were people able to complete the tasks?!

Why?!
•  Did it take them an exceptionally long time?!

•  Did people make errors? Did they know it? Did they recover?!

•  How did people feel about the experience?!

What to capture!
•  Performance measures!
•  Task completion rate!
•  Task completion time!
•  Error rate!
•  Subjective measures!
•  Perceived usability!
•  Perceived usefulness!
•  Desirability!
What to capture!
•  Performance measures!
•  Task completion rate!
•  Task completion time!
•  Error rate!
Quantitative measures!
•  Subjective measures!
•  Perceived usability!
•  Perceived usefulness!
•  Desirability!
What to capture!
•  Critical incidents!
•  Things that happen along the way
that might explain outcomes!

•  Verbal accounts!
•  Statements that indicate thought
process, attitudes, explanations!
What to capture!
•  Critical incidents!
•  Things that happen along the way
that might explain outcomes!

Qualitative data!
•  Verbal accounts!
•  Statements that indicate thought
process, attitudes, explanations!
What matters more?!
•  Summative tests: quantitative!
•  May not even capture qualitative data!

•  Formative tests: qualitative!

•  Use quantitative data to ground the important stuff!
Critical Incidents!
•  The stuff that happens while users perform tasks!
•  Clicking the wrong button!
•  Ignoring the instructions showing on the screen!
•  Entering the wrong information!
•  Following the wrong path!
•  Misinterpreting a label!
•  Expressing confusion or frustration!
•  Asking for help!
•  Staring at the screen for a long time!
•  Giving up!
•  …!
Verbal accounts!
•  Stuff that users say when performing a tasks!
•  “I’m looking for…”!
•  “I was expecting to see…”!
•  “I wonder what this does…”!
•  “That doesn’t seem right…”!
•  “Wait…what?”!
•  “I think that was right…”!
•  “Oops, I guess not…”!
•  “How do I…?”!
From Data to Actionable Intelligence!
•  Data: “5 out of 7 users failed task 3”!

•  Better: !
•  3 users could not figure out how to enter shipping information!
•  2 users entered shipping information but then cancelled because “it seemed wrong”!

•  Even better: !
•  “Shipping information” dialog does not appear until after clicking “enter payment info”
– this does not match users’ mental model!
•  After entering shipping info, final checkout screen does not show users’ selection –
lack of feedback causes leads to abandonment!
Where the Actionable Intelligence is!
•  Critical incidents!
•  Failure to enter shipping info!
•  Cancellation!
•  Long time spent inspecting checkout pages!

•  Verbal accounts!
•  “I don’t see anywhere to…”!
•  “I would expect to find…”!
•  “I don’t see my selection here…”!
•  “It doesn’t seem like it worked…”!
Data about your users!
•  Technical expertise!
•  Domain expertise!
•  Frequency of relevant behaviors (e.g., shopping)!
•  General demographics (maybe)!
•  Age!
•  Gender!
•  Education level!
•  Country of origin!
•  …!
Relevance of user data!
•  Explain different outcomes!
•  Performance!
•  Attitudes!

•  “3 out of 4 users with low technical proficiency failed task 5; all 3

users with high technical proficiency succeeded”!

•  “While all users succeeded in most tasks, users with higher

domain expertise gave lower perceived usefulness ratings”!
Summary: What to look for!
•  Capture quantitative data as grounding!
•  Performance: How well does the system do what it needs to do?!
•  Subjective: What are users’ perceptions?!

•  Capture qualitative data to explain why!

•  Knowing why tells you how to make it better!

•  Capture data about your users!

•  To explain differences in quantitative and qualitative data!