SOFTWARE DESIGN: SESSION 10

ARE WE THERE YET?

USER TESTING
Alex Cowan

2015 COWAN+
WHERE YOU SHOULD BE NOW
Epic Stories Experiment(s) for
CUSTOMER
Child Stories DISCOVERY &
hypothesis testing
EXPERIMENTS
Supporting Storyboards Usability test suite
?
Parallel Prototypes

E
M
Interactive Prototype

WH
W

AT
S HO
PRODUCT &

IF
PROMOTION

?
Personas, Problem Scenarios
VALUE
including Think- Alternatives PROPOSITIONS &
USER STORIES
ASSUMPTIONS Summary Value
See-Feel-Do Value Propositions & PROTOTYPES
Hypothesis
(Converged) Before & After
MVP Ideas
Storyboards

E?
Assumptions for Value

AL
PIVOT?

SC
Hypothesis
AIDAOR Storyboard
?
AT
WH
PROBLEM Interview Guide
PERSONAS WHO? SCENARIOS &
ALTERNATIVES
2 (or more) Interviews

2015 COWAN+
WHAT WELL FOCUS ON TODAY
CUSTOMER
DISCOVERY &
EXPERIMENTS

?
ME
OW
SH
USER STORIES
& PROTOTYPES

2015 COWAN+
YOUR USABILITY HYPOTHESIS- TESTING IT
USABILITY HYPOTHESIS
QUESTIONS
Is a strong narrative (user story) guiding
your work?
Do the UIs affordance deliver against those
narratives?

TOOLS
User Stories
Comparables
Prototypes/Wireframes
Test Plans

2015 COWAN+
WHAT WELL WORK ON TODAY
Finding the Right Finding the Right
PROBLEM SOLUTION

Value Hypothesis &

Personas Assumptions
alternatives

Problems Experiments on Motivation

Alternatives Product
Hypothesis User Stories
Field (input to)
Discovery Comparables
Prototypes
Usability Tests
Software

divergence convergence divergence convergence

time
source: adapted from The Design of Everyday Things 2015 COWAN+
WHAT ARE WE TESTING?

Motivation

Action : )

Action Line

Inaction : (

Ability (Usability)

source: adapted from BJ Foggs Behavioral Model 2015 COWAN+

WHAT ARE WE TESTING?
DONT RUIN YOUR
RESULTS BY TRYING
TO TEST BOTH AT THE
Motivation SAME TIME
Proposition Testing
(ala Lean Startup,
MVPs) Action : )

Action Line

Inaction : (

Ability (Usability)
Usability Testing

source: adapted from BJ Foggs Behavioral Model 2015 COWAN+

IF USABILITY TESTING, MOTIVATION SUPPLIED

Your software paints

virtual walls. Tell the
subject youd like to see
the wall yellow and
observe what they do.
(If you want to test motivation,
start with interviews, then
MVPs, experiments.)

2015 COWAN+
USABILITY TEST SUITES: A PROGRESSION
Exploratory Assessment Validation

It looks like this The implementation

approach will is sound and ready Ready for
fundamentally work. for tuning. prime time.
2015 COWAN+
TEST-DRIVEN PRODUCT DESIGN
70% of users didnt This change in
get the drag and drop the annotation
in this version was enough so
they got it

2015 COWAN+
A USABILITY TEST DESIGN FOR ANYONE
Objectives & Methods
Product Version
Subjects
Research Composition
Pre-Session Checklist
Session Design

bit.ly/cdhandbook bit.ly/vdt-usability

2015 COWAN+
STEP 2: CREATE SUBJECT SCREENER
Persona
Sam the Small
Business Owner

Screener
How many times last
month did you post to
social media for your
business?

2015 COWAN+
STEP 3: DEFINE FUNCTIONAL SCOPE (USING STORIES)
Made up epics for small business social media automation:

As a small business owner, I want to sign up for the service, so I can

give it a try.
As a small business owner, I want to connect my social media
accounts, so I can create automated posts to them.
As a small business owner, I want to try out a post so I can see how
the service works.
Define the stories (epics
and/or children) that
youd like to test (5 min)
2015 COWAN+
STEP 4: DESIGN RESEARCH ITEMS

Take at least one story

and design a research
item (5 min)
2015 COWAN+
STEP 5: BUTTON UP THE REST OF THE DETAILS
Objectives & Methods
Product Version
Subjects
Research Composition
Pre-Session Checklist
Session Design

bit.ly/cdhandbook bit.ly/vdt-usability

2015 COWAN+
STEP 6: UPDATE, CREATE YOUR PROTOTYPE (OR SW)
Your prototype should
support your research
design, not the other way
around.

2015 COWAN+
STEP 7: PRE-TEST
For most projects, subjects are hard to get.

Try to find some looser approximations and test the research design-
it will usually have at least a few kinks that would waste a subject
you worked hard to arrange.

2015 COWAN+
KEY INTER-RELATIONSHIPS
USABILITY
TESTING

VALUE
PROPOSITIONS
PROTOTYPES &
USER STORIES ASSUMPTIONS
The user stories should
be the nexus of your
ideation, the other
items supporting them.

2015 COWAN+
A) I know the part and I want to
order it so I can figure out next
steps on the repair.

B) I dont know the part and I want

to try to identify it online so I can
move the job forward.

C) I dont know the part and I cant

determine it and I want help so I can
move the job forward.

2015 COWAN+
EXAMPLE TEST ITEM- EXPLORATION + STORY B
# RESEARCH OBJECTIVE NOTES

3 How are we doing on this story: MODERATOR GUIDE
I dont know the part number and I Lets say you were looking at this unit and this was the part you
want to try to identify it online so I needed to replace (offer photos). You dont have the part number.
can move the job forward. How would you identify it?

OUTPUT
Result on whether the subject understands and can complete the
applicable search process.

2015 COWAN+
EXAMPLE: AGILE USER STORIES
EPIC STORY
As Ted the HVAC technician, I want to identify a part that needs replacing so I can decide my next steps.

CHILD STORIES
A) I know the part number and I want to find it on the system so I can figure out next steps on the repair.
B) I dont know the part number and I want to try to identify it online so I can move the job forward.
C) I dont know the part number and I cant determine it and I want help so I can move the job forward.
D) I want to see the cost of the part and time to receive it so I decide on next steps and get agreement from
the customer.

2015 COWAN+
EXAMPLE TEST ITEM- EXPLORATION + STORY C
# RESEARCH OBJECTIVE NOTES

4 How are we doing on this story: MODERATOR GUIDE
I dont know the part number and I Lets say you were looking at this unit and (whether or not you
cant determine it and I want help so actually know what it is) youre just stuck identifying it and you
I can move the job forward. want help from dispatch. Can you show me what you would do?

OUTPUT
Result on whether the subject understands and can complete the
applicable search process.

2015 COWAN+
EXAMPLE: AGILE USER STORIES
EPIC STORY
As Ted the HVAC technician, I want to identify a part that needs replacing so I can decide my next steps.

CHILD STORIES
A) I know the part number and I want to find it on the system so I can figure out next steps on the repair.
B) I dont know the part number and I want to try to identify it online so I can move the job forward.
C) I dont know the part number and I cant determine it and I want help so I can move the job forward.
D) I want to see the cost of the part and time to receive it so I decide on next steps and get agreement from
the customer.

2015 COWAN+
EXAMPLE: AGILE USER STORIES

D) I want to see the

cost of the part and
time to receive it so I
decide on next steps
and get agreement
from the customer.

2015 COWAN+
EXAMPLE TEST ITEM- EXPLORATION + STORY B
# RESEARCH OBJECTIVE NOTES

4 How are we doing on this story: MODERATOR GUIDE
I want to see the cost of the part Tell me what youre seeing here? What the price for this part? How
and time to receive it so I decide on soon can you get it? Given that, what would you tell the customer?
next steps and get agreement from
the customer. OUTPUT
Result on whether the subject understands and can complete the
applicable search process.

2015 COWAN+
TOP 3 REASONS WHY TEAMS DONT TEST MORE
1 Its painful to see how wrong you were about the user.
Even though youll be much better off if you try something new soon.

2 Members are have been turned off by phase-inappropriate testing

(usually too detailed, too soon).
Even though testing early and often avoids waste, pointless
discussion.

Getting in front of real users is kind of a pain.

3 Even though the customer is why were here.

2015 COWAN+
TESTING IS REWARDING

Everyone has an opinion about

usability- until it comes time to test.

Testing improves product and avoids

lots of waste.

2015 COWAN+
ASSIGNMENTS
1. Do/Group: Draft usability TestPlan (30-40 minutes)
2. Do/Group: Update Interactive Prototype (40-60 minutes)
3. Do/Group: Dry Run the Test Plan + Prototype (~30 minutes)

v10 2015 COWAN+