LAB 05: Usability Testing

Purpose
This lab will make students learn the technique of evaluating interactive system’s design
through user participation where the actual users are asked to interact with the system.

Outcomes
After completing this lab, students will be able to apply this technique to evaluate
interactive system’s design. They will have a good insight about:

 How to evaluate design with users.

 What are the different steps involved?
 When to evaluate design with users.
 Benefits and drawbacks of evaluation with users

Introduction
A popular and effective way to gather information, about actual use of a system, for
evaluation purposes is to observe users interacting with it. In this lab, you are going to
evaluate an interactive system’s design through user participation where the actual users
are asked to complete a set of predetermined tasks and the evaluator watches and records
the users’ actions. Such an approach is termed as empirical in which evaluation is based on
observation or experience rather than theory or pure logic. In this approach, a real
environment is simulated to conduct user tests. Before conducting them, a careful planning
is required to make sure that conducted tests will be free of biases and representative of
future interactions.

When to Evaluate with users

It is very common to perform user testing after evaluation with experts because that helps
us not waste our users on stuff that’s just going to get picked up automatically. We want to
focus the valuable resources of user testing on stuff that other people wouldn’t be able to
pick up on.

Benefits and Drawbacks of Evaluation with Users

Experts walking through your system can generate false positives that might not actually
happen in a real environment while this does not happen in evaluation with users and it is
going to be more accurate.

One of the things that we see is that heuristic evaluation can often be a lot faster — It takes
just an hour or two for an evaluator — while the mechanics of getting a user test up and
running can take longer.

CCS356 –Lab Manual 1

The heuristic evaluation results come pre-interpreted because evaluators are directly
providing us with problems and things to fix while it takes more time to infer from the user
tests what might be the problem or solution.

Lab Task: System to be evaluated

To explain how the evaluation takes place, let us consider Google map application and
apply this technique to evaluate it. Since google map offers a lot of feature so it would not
be possible in this lab to apply this technique to evaluate all of them. We, thus, focus on
following three features:

1- Street View
2- Finding a place of interest and its related information near some location
3- Adding an intermediate destination between source and destination and the
“Avoid tolls” option.

Activity 1: Tasks design

To evaluate the ergonomics of the Google Maps for the aforementioned features, we need to
design tasks for each of them. These tasks (given below) would eventually be given to
actual users.

1- Center the Eiffel Tower in Street View

2- Find out the color of the signage of any Chinese restaurant in/near Orsay, France
3- Find a path from Orsay to Orléans, via Chartres, without paying tolls

Activity 2: Determining Criteria

Criteria refer to those characteristics of the population that can influence the way product
is used. We need to figure out different criteria for evaluating the Google Maps.

Activity 3: Determining Context

Context refers to those characteristics of the environment that can influence the use of
product. We need to come up different contexts that can affect the use of Google Map.

Activity 4: Choosing Participants

A rule of thumb is to have 6 persons per criteria and per context. We must choose our users
carefully. They must be minimal in numbers and at the same time satisfy the rule.

Ideally 18 carefully chosen persons can satisfy that rule for the criteria. The following table
shows the sample of population. We can see that carefully chosen 18 users do satisfy the
rule.

CCS356 –Lab Manual 2

 Table 5.1: Sample of Population (“X” indicates that the user satisfies the criterion)

User Gender Age Expertise in Expertise

ID in Paper
Google Map /Computers Maps
M F <30 30-60 >60 Yes/Yes No/Yes No/No

1 X 20 X

2 X 22 X

3 X 27 X X

4 X 29 X X

5 X 29 X

6 X 24 X

7 X 30 X

8 X 32 X

9 X 35 X X

10 X 43 X

11 X 51 X

12 X 33 X X

13 X 77 X

14 X 80 X X

15 X 61 X

16 X 65 X X

17 X 62 X

18 X 61 X X

Total 10 8 6 6 6 6 6 6 7

In the case of Contexts, it is easy to satisfy that rule because a context is not a characteristic
of the population and it can be controlled by evaluators during the tests.

CCS356 –Lab Manual 3

Table 5.2: Contexts: Sample of Population (“X” indicates that the user satisfies the criterion)

User ID Screen Size Time Stress Mouse with Scroll

Wheel
14” Laptop 5” Mobile
Screen Screen
1 X X

2 X

3 X X X

4 X

5 X X

6 X

7 X X

8 X X

9 X X

10 X X X

11 X X

12 X X

13 X

14 X X

15 X X

16 X

17 X X

18 X X

Total 10 8 9 6

Activity 5: Usability Experiment

Once we have chosen the users, we are ready to perform usability experiment with them.
During the experiment, we should observe the participant, and take notes of interesting
events, such as difficulties encountered. We may also perform video recording so that we
can view it later in case some events get missed by the evaluator.
CCS356 –Lab Manual 4
Activity 6: Post Experiment Investigation
Once the experiment has finished, we can ask user to fill out a questionnaire inquiring
about user’s experience with the system. It also often happens that we want to know about
why user performed a specific action. So, this is the time to inquire this.

Given below is an example of questionnaire to inquire about user’s experience.

Questionnaire
Task 1:

1- How many different ways do you know to see the Google maps in street view? Which
ones? Which one do you prefer all the time?
2- How would you rate Google map in terms of its “ease of use” for achieving this task?
(1-very easy ~ 5-very difficult)

Task 2:

1- Do you know any other way to do this task?

2- How would you rate Google map in terms of its “ease of use” for achieving this task?
(1-very easy ~ 5-very difficult)

Task 3:

1- Did you know about the “Avoid toll” option in Google maps? What is your opinion on
the placement of “Avoid toll” option in Google maps?
2- How would you rate Google map in terms of its “ease of use” for achieving this task?
(1-very easy ~ 5-very difficult)

Activity 7: Analysis based on Observation and User’s

suggestions
In this phase, the evaluator sums up all the activities performed by the user and write
down its observations. He provides valuable suggestions to resolve the difficulties faced
by the users.

CCS356 –Lab Manual 5

Home Activities
Activity 1: Choice of System to be Evaluated
The application, system, or device should be similar, or at least related in some sense, to the
topic your group will select as Semester Project. This will help in identifying design issues
a similar system possesses and lead you to design better product compared to the
competitor.

Activity 2: Tasks Design

Since the selected system may offer a lot of feature so would not be possible to apply this
technique to evaluate all of them. You would therefore identify 3 to 5 features of the system
and design tasks for them. These tasks would eventually be given to the actual users.

Activity 3: Determining Criteria

The users should be representative of the “typical” users of the system or device. You will
need to figure out different criteria for user selection.

Activity 4: Determining Contexts

You will need identify different characteristics of the environment that can influence the
use of product.

Activity 5: Choosing Participants

A rule of thumb is to have 6 persons per criteria and per context. You must choose your
users carefully. They must be minimal in numbers and at the same time satisfy the rule of
thumb.

Activity 6: Usability Experiment

During the experiment, you should observe the participant, and take notes of interesting
events, such as difficulties encountered, frustration and annoyance, or surprise. For
complicated experiments, it is advisable to have an observer and a facilitator; the role of the
latter is to assist the participant if necessary. After the experiment, take some time to write
down observations about the behavior of the participant and properties of the interaction
between participant and device. After the experiment, you can also ask the participants
questions about their experience, or give them a brief survey.

Activity 7: Analysis
Finally, you would recommend the possible improvements based on the observations you
record during experiment and user’s suggestions.

Assignment Deliverables
You need to submit a report containing the following items:

 Small description of the chosen system

CCS356 –Lab Manual 6
  Chosen features and the corresponding tasks designed.
 Criteria, along with sample of population table
 Contexts along with sample of population table
 Observations recorded for each user
 User’s Feedback/Suggestions obtained through interviews/questionnaires
 Recommendations to improve the usability of the system

CCS356 –Lab Manual 7