Introduction to Human

Computer Interaction
Lecture 13

Course Instructor: Ms. Farheen Ramzan

Department of Computer Science

University of Engineering and Technology Lahore
 Reference

For the material covered in this lecture,

read
• Chapter 15 – Evaluation Studies: From
Controlled to Natural Settings
• (Section 15.1 – 15.4)
(from Interaction Design: Beyond Human-
Computer Interaction, 5th Edition by Helen
Sharp et al.)
 Evaluation Studies:
From Controlled to Natural
Settings
 Overview
 Usability Testing
 Methods, Tasks, Users, Labs and Equipment
 Case Study: Testing the iPad Usability
 Conducting Experiments
 Hypotheses Testing
 Experimental Design
 Field Studies
 In-the-Wild Studies
 Case Study: A Pain Monitoring Device

www.id-book.com 4
 Goals
• Explain how to do usability testing
• Outline the basics of experimental
design
• Describe how to do field studies

www.id-book.com 5
 Introduction
• Focus on evaluation studies that take place in a spectrum of
settings, from controlled laboratories to natural settings.

– Usability testing
• which takes place in usability labs and other
controlled lab-like settings
– Experiments
• which take place in research labs
– Field studies
• which take place in natural settings, such as
people’s homes, schools, work, and leisure
environments

www.id-book.com 6
 Usability Testing
• The usability of products tested in controlled laboratory
settings.
• Performing usability testing in a laboratory
– enables designers to control what users do
– allows them to control the environmental
– and social influences that might impact the user’s
performance.
• The goal is to test whether
– the product being developed is usable by the intended
users
– in order to achieve the tasks for which it was designed
– and whether users are satisfied with their experience.

www.id-book.com 7
 Usability Testing: Methods, Tasks, and
Users
• Collecting data about users’ performance on predefined
tasks - central to usability testing
– Controlled settings
– Users are observed and timed
– Data is recorded on video, and key presses are
logged
– Think aloud technique
– Performance measures
• time it takes typical users to complete a task
• number of errors that users make
– User satisfaction is evaluated using questionnaires
and interviews
www.id-book.com 8
 Quantitative Performance
Measures
• Number of users successfully completing the task
• Time to complete task
• Time to complete task after time away from task
• Number and type of errors per task
• Number of errors per unit of time
• Number of navigations to online help or manuals
• Number of users making a particular type of error

Source: Wixon and Wilson, 1997

www.id-book.com 9
 Usability Testing Conditions
• Usability lab or other controlled space
• Emphasis on:
 Selecting representative users
 Developing representative tasks
• 5-10 users typically selected
• Tasks usually around 30 minutes
• Test conditions are the same for every
participant
• Informed consent form explains procedures and
deals with ethical issues
www.id-book.com 10
Usability Testing: Labs and Equipment

• Many large companies, such as Microsoft,

Google, and Apple, test their products in custom
built usability labs
– a main testing lab with recording equipment
– an observation room for designers
– a reception area, a storage area and a viewing room

• Wall-mounted video cameras to record the users’

behavior
• Microphones to record users’ comments

www.id-book.com 11
 Usability Lab with Observers
Watching a User and Assistant

A usability laboratory in which designers watch participants on a monitor and

through a one-way mirror

www.id-book.com 12
 Beyond Usability Labs
• Usability labs can be expensive and labor-intensive to run and
maintain.
– Mobile usability and Remote usability
• Mobile usability equipment typically includes video cameras,
laptops, eye-tracking devices, and other measuring equipment that
can be set up temporarily in an office or other space, converting it
into a makeshift usability laboratory.
• Advantages
– the equipment can be taken into work settings enabling testing to be
done on-site
– which makes it less artificial and more convenient for the participants.

• lab-in-a-box or lab-in-a-suitcase

www.id-book.com 13
 Tobii Glasses Mobile
Eye-Tracking System

Source: Dalton et al., 2015, p.3891. Reproduced with permission of ACM Publications.

www.id-book.com 14
Tobii Glasses Mobile Eye-Tracking
System (Cont.)
• In a recent study, eye-tracking glasses were used to record
the eyegaze of people in a shopping mall.
• Study goal
– find out whether shoppers pay attention to large format plasma
screen displays when wandering around a large shopping mall in
London.

• The displays varied in size and contained information about

directions to different parts of the mall or advertisements.
• 22 participants took part in the study
• User carried out a typical shopping task while wearing Tobii
Glasses Mobile Eye Tracking glasses.
• Participants were paid and rewarded

www.id-book.com 15
Case Study: Testing the iPad Usability
• When Apple’s iPad first came onto the market, usability
specialists from the Nielsen Norman Group conducted
user tests to evaluate participants’ interactions with
websites and apps specifically designed for the iPad.
• Tests had to be done quickly, as information was needed
by third-party app developers
• Also needed to be done secretly so that the competition
was not aware of the study before the iPad was
launched
• A second study was carried out, a year later, to examine
some additional usability issues.

www.id-book.com 16
 Case Study: Testing the iPad
Usability (Cont.)
• iPad Usability: First Findings from User
Testing
– Used two usability evaluation methods
– 1. usability testing with think-aloud
– 2. an expert review
• The usability testing was carried out in two cities in the United
States: Fremont, California, and Chicago, Illinois.
• The test sessions were similar: typical usability issues when using
apps and websites.
• Seven participants with over three months experience with iPhones

www.id-book.com 17
Case Study: Testing the iPad Usability
(Cont.)
• Signed an informed consent form explaining:
 What the participant would be asked to do
 The length of time needed for the study
 The compensation that would be offered for participating
 Participants’ right to withdraw from the study at any time
 A promise that the person’s identity would not be disclosed
 An agreement that the data collected would be confidential
and available to only the evaluators
• Participants were asked to explore the iPad
• Next, they were asked to perform randomly-assigned
specified tasks
www.id-book.com 18
 Examples of the Tasks
Used in the iPad Evaluation

Adapted from Budiu and Nielsen, 2010

Source: iPad App and Website Usability Study. Used courtesy of the Neilsen Norman
Group.
www.id-book.com 19
 Portable Equipment for
Use in the Field

Setup used in the Chicago usability testing sessions

Source: iPad App and Website Usability Study. Used courtesy of the
Neilsen Norman Group.

www.id-book.com 20
 Usability Problems and Actions
• Examples of problems detected:
 Accessing the Web was difficult
 Lack of affordance and feedback
 Getting lost in an application
 Knowing where to tap
• Actions by evaluators:
 Reported to developers
 Made available to public on Neilsen Norman
Group.
 Study did not address how iPad would be used in
people’s everyday lives
www.id-book.com 21
 Case Study: Testing the iPad
Usability (Cont.)
• Another study was done a year later to examine this and
other issues that there was insufficient time to address in
the first study
• iPad Usability: Year One
– the first study was done with participants who did not have direct
experience with an iPad.
– A year later, the researchers were able to recruit participants
with at least two months’ experience of using an iPad.

• 16 iPad users, half men and half women

www.id-book.com 22
 Usability Demo
• A video illustrates the usability problems that a
woman had when navigating a website to find the
best deal for renting a car in her neighborhood.
• It illustrates how usability testing can be done in
person by a designer sitting with a participant.
• The video is called Rocket Surgery Made Easy by
Steve Krug, and you can view it here:
https://www.youtube.com/watch?v=QckIzHC99Xc

www.id-book.com 23
 Conducting Experiments
• In research contexts, specific hypotheses are tested
– that make a prediction about the way users will perform with an
interface.
– Example hypothesis: context menus are easier to select as compared to
cascading menus.

• Hypotheses Testing
– Predict the relationship between two or more variables
– Independent variable is manipulated by the researcher (e.g. different
menu types)
– Dependent variable influenced by the independent variable (e.g. time
taken to select an option)
– Validated statistically and replicable
– Analysis of the data using statistical tests (e.g., t-tests)

www.id-book.com 24
 Experimental Design
• To test a hypothesis, the researcher has to
set up the conditions and find ways to
keep other variables constant to prevent
them from influencing the findings. This is
called the experimental design.
• Determine which participants to involve for
which conditions in an experiment.
• Different designs

www.id-book.com 25
 Different Types of Experimental
Designs
• Different participants (between subjects):
• Single group of participants is allocated
randomly to the experimental conditions

• Same participants (within subjects):

• All participants appear in both conditions

• Matched participants (pairwise):

• Participants are matched in pairs, for example,
based on expertise, gender, and so on
www.id-book.com 26
27
28
 Different, Same, Matched
Participant Design

The advantages and disadvantages of different allocations of participants to

conditions

www.id-book.com 29
 Field Studies
• Field studies are done in natural settings
• Qualitative data of people’s behavior and activities
• Data on the frequency and patterns of daily activities
• Data collection by observing and interviewing people
(video, audio, field notes, photos or diaries)
• Deciding whether to tell the people being observed that
they are being studied
– might change the way they behave or disrupt their behavior
• Ensure the privacy of participants, sign an informed
consent form

www.id-book.com 30
 Field Studies: In-the-Wild Studies
• “In the wild” is a term for prototypes being
used freely in natural settings
• Seek to understand what users do naturally
and how technology impacts them
• Field studies are used in product design to:
 Identify opportunities for new technology
 Determine design requirements
 Decide how best to introduce new technology
 Evaluate technology in use
www.id-book.com 31
 Case Study: A Field Study of a
Pain-monitoring Device
• Monitoring patients’ pain is a known challenge for
physicians
• Goal of the study was to evaluate the use of a pain-
monitoring device for use after ambulatory surgery
• Painpad, a pain-monitoring tangible device for
patients to report their pain levels
• It was usability tested extensively in the lab before
brought into two hospitals
• Goal was to understand how Painpad was used in
the natural environment and as part of routines in two
UK hospitals.
• How manual pain-monitoring differed with Painpad
www.id-book.com 32
 Painpad

A tangible device for inpatient self-logging of pain

Source: Price et al., 2018. Reproduced with permission of ACM Publications.
www.id-book.com 33
 Painpad: Data Collection and Participants
• Two studies in two hospitals involving 54 people
– 13 males, 41 females
• Privacy was an important concern
• Hospital stay ranged from 1-7 days, mean and median age 64.6,
64.5
• Patients given Painpad after surgery and prompted to report pain
levels every two hours
– flashing light and sound notifications
• Nurses also collected scores
• Patients in one hospital were given a user-satisfaction survey when
they were discharged from the hospital
– Rated Painpad on a 1-5 Likert scale

www.id-book.com 34
 Painpad: Data Analysis and Presentation

• Three types of data were collected:

 Satisfaction with Painpad was based on questionnaire
responses
 Patients’ compliance with the two-hour routine
 Data collected from Painpad compared with data
collected by nurses
• Data showed:
 Satisfaction with Painpad 4.63 on Likert scale
 Patience compliance was mixed: some liked it while
others disliked or didn’t notice the prompts
 Patients recorded more scores with Painpad than
through the nurses
www.id-book.com 35
 Usability Testing vs. Experiments
• Usability testing is applied experimentation
• Developers check that the system is usable
by the intended user population by collecting
data about participants’ performance on
prescribed tasks
• Experiments test hypotheses to discover
new knowledge by investigating the
relationship between two or more variables

www.id-book.com 36
 Usability Testing and Experiments
Usability Testing
• Improve products
• Few participants
• Results inform design
• Usually not completely
replicable
• Conditions controlled as
much as possible
• Procedure planned
• Results reported to
developers
www.id-book.com
Experiments for Research
• Discover knowledge
• Many participants
• Results validated
statistically
• Must be replicable
• Strongly controlled
conditions
• Experimental design
• Scientific report to
scientific community
37
 How Many Participants Are Needed When
Carrying Out An Evaluation Study?

• The number is a practical issue

• Depends on:
 goal of the study
 type of study
 And the constraints encountered
 Availability of participants
 Cost of running tests

• Some experts argue that testing should

continue until no new insights are gained
www.id-book.com 38
 How Many Participants Are Needed When
Carrying Out An Evaluation Study?
Usability studies
– 5–12 participants are recommended
– Others say that when there is nothing new, it is time to stop.
Experiments
– Depends on the type of experimental design, the number of
dependent variables being examined, and the kinds of statistical
tests that will be used.
– Minimum 15 participants is suggested (Cairns, 2019).
Field studies
– Depends on what is of interest: it may be a family at home, a
software team in an engineering firm, children in a playground, a
whole community in a living lab, or even tens of thousands of
people online.

www.id-book.com 39
Questions?