You are on page 1of 31

Lect ure Topi c 12

Usabi lity Testing - part 1


E-Commerce Fundamentals and Applications, by Henry Chan et al., John
Wiley and Sons Pty Ltd, 2001
Lecture 12 USABILITY
• Usability can be defined as:
“the extend to which end-users can successfully accomplish
required goals in a specified environment subject to various
comfort, effectiveness and satisfaction criteria” (Thimbleby, SIGCHI94)
• Usability is becoming an integral part of the software
development lifecycle.
• Software developers aim at improving their product from
the end-users viewpoint which involves identifying
problems with the interface, and after some careful
analysis of the problems, the interface is redesigned with
the hindering aspects removed.
• The interface design is the main factor affecting usability,
since it is the only part of the system with which the user
interacts with.

• Usability testing usually entails building prototypes of

the system and testing it on users.
• Usability testing is a broad context where the interface
is put under a test, with users critically analysing the
system, evaluating the design layout and flow of
interactions by interacting with the implemented
version of the system.
• The main reasons for adopting this approach are that it
can be used :
– to measure user performance and satisfaction,
– to identify and assess usability defects,
– to provide ideas as to how a system might be
– to educate design teams.
Lecture 12

• Many usability testing strategies have been developed.

Most of these testing can be categorised into two main
– Empirical Testing
– Inspection Testing
Lecture 12

Phases of product usability testing

Phase Purposes Objects

Investigation •
Set usability objectives •
Existing solutions

Discover opportunities
Design and build •
Resolve conflicts •
Early concepts

Answer questions •

Debug •

Measure against •
Partial products
objectives •

Post release •
Get data for •
Finished product
sales/marketing •
Competitive product

Input to next release

Competitive analysis
Lecture 12 Empirical Testing

• Empirical testing involves the user testing the product

under review in a testing laboratory.
• Users are asked to use the system to complete a work
related task that is likely to target problem areas.
• The users’ task results are recorded, often through a
video camera.
• The users’ concerns, which are expressed through the
use of a questionnaire, are also documented.
• The results are analysed by the development team in an
attempt to refine their product.
• The primary focus of empirical testing if the product is
still in a prototype stage is to use the results of the
testing for refining the final version of the product.
Lecture 12 Empirical Testing

• If the product to be tested is in the final stage of

development, this approach is useful for determining
whether one selects a given product.
• There are three types of Empirical testing:
–Fixed Laboratory Usability Testing
–Portable Usability Testing
– Remote Usability Testing
Lecture 12
Empirical Testing : Fixed Laboratory
• Fixed laboratory testing involves the usability testing of
a product in a fixed laboratory. This method is often
referred to as in-house testing, since the users have to
come to the developer to complete the usability test.
• The user interact with the product by completing a
work-related task which is created by the development
• To assist the development team in analysing the results
of the usability test, the tests are recorded using a video
camera installed in the laboratory.
Lecture 12
Empirical Testing : Fixed Laboratory
– All necessary equipment is already in place. This
will make it easier for the development team to
conduct the usability test and record all essential
– Can be very expensive, a fully equipped fixed
laboratory costs between $50,000 to $100,000 and
even more to operate and maintain. A big
proportion of the amount goes towards recruiting
and training of staff.
– The user has to come to the developer to perform
the test. (Explain what consequences this may cause
Lecture 12
Empirical Testing : Fixed Laboratory
• IMPROVED Fixed Laboratory
– The recent version of fixed laboratory is designed
to address the problems of the earlier version of the
– The main emphasis is to design a comfortable,
natural office scenario for the users and to
eliminate the artificial element that is often
associated with usability testing in a fixed
– This improved version fixed laboratory has a
separate observation room, which is linked to the
main laboratory through a one-way glass. This
enables the observer to view the tester without the
tester conscientious of an observer peering over
their shoulder.
Lecture 12
Empirical Testing : Fixed Laboratory
• IMPROVED Fixed Laboratory (ctd.)
– This new layout enables testers to proceed without
interruption as they would while performing the task
in their own work environment.
– Very often, the layout of the observation room is as
similar as possible to the users real working
– However, this improved laboratory still does not
address the problem of high costs.
– Note that this fixed laboratory testing is normally
used for building a new sophisticated software
system or a new operating system whose usability
will not be affected by internet connection, download
time etc. (eg. Microsoft windows operating system
Lecture 12
Empirical Testing: Portable Usability
• Portable usability testing was introduced to address the
problem of high cost found in a fixed laboratory setting.
• Portable usability testing takes the testing laboratory to
the users.
• The product is tested while the users interact with it in
their own work environment.
Lecture 12
Empirical Testing: Portable Usability
• Portable Testing is best suited when the customer base
is widely distributed.
• The developers will test a wide range of users from
each of the customer sites.
• The recording of the task results of the user is best
accomplished with the use of a video camera. However,
an over the shoulder video camera can place undue
stress on the users, which could jeopardise the field test
• The recording equipment must be setup in such a
manner that it does not interfere with the usability
testing process.
Lecture 12
Empirical Testing: Portable Usability
• The SETUP:
• The computer output is directed to two
• The video camera should focus on the second
• The test results can then be recorded without
the video equipment getting in the user’s way.
• Another important element is to encourage the users to
think out loud while they undertake the tasks. These
verbal comments can prove to be valuable for analysis.
Lecture 12
Empirical Testing: Remote Usability
• Remote usability testing is a usability testing where
neither the developers or the users have to travel.
• Users are observed and tested through the aid of
computer networks and modem connections in a
time efficient and cost effective manner.
• Can you think of any limitations of this type of
usability testing?
Lecture 12
Empirical Testing: Summary

• In general, the main advantage of empirical testing

is that it is able to directly monitor the performance
reactions of the user to the particular interface and
one does not have to rely on a usability expert
trying to second guess what a user’s reaction might
• The disadvantage is that it is only useful for testing
an implemented product, and therefore the ability to
correct the problem immediately is greatly reduced.
Lecture 12 Empirical Testing: Test Procedure
• The test layout in an empirical testing consists of:
– the test subject (the user)
– the computer with the software being tested
– the test monitor (the tester)
• The user must perform a set of tasks and answer a set of
questionnaire prepared earlier by the tester. The tasks must
be designed such that:
– they be work related to stimulate the manner in which the
software would actually be used in the work environment
– they cover most of the significant functions of the package
so that all significant and frequently used features of the
interface are tested
– the time of the test session clearly reflects work periods
and is not too short (too artificial) or too long (may cause
Lecture 12
Inspection Testing

• Usability Inspection testing is often referred to as

“discount usability engineering methods”.
• Inspection testing is performed by usability inspectors
who will inspect the design layout of an interface and
comment based on the result of their test and
• Results from inspection methods influence the design
layout of an interface since it captures potential
problems at an early stage in the development life cycle.
• Most commonly used Inspection testing procedures are:
– Cognitive Walkthrough
– Heuristic Evaluation
Lecture 12
Inspection Testing: Cognitive
• People who are evaluating the interface (called the
inspectors) are asked to answer a set of derived
questions for each step of the user task scenario. These
questions are based on the relationships between user’s
goals and the actions available for achieving these.
• This inspection testing can be done during the design
stage especially when the dynamic aspects of the web
site/application is determined.
• The user task scenario can be drawn from the Web
Navigation Analysis Model (lecture 10) by defining a
scenario for each branch of the hierarchical structure in
the Site Navigation Net.
Lecture 12 Inspection Testing: Cognitive

• For example,

we can Prc: sesion active

Prc: sesion active
Prc: sesion active

perform a
Enter Booking Details Search Lot Search Lot
Di: MT3 Di: Di:
Do: Booking details Do: Do:



GD 2
Prc: sesion active Prc: sesion active Prc: sesion active
Poc: Poc: Poc:


GD 1.2
Booking Details Confirm Lot Result List Lot Result List

test on the Di: Booking details

Di: Warehouse ID
Di: Warehouse ID
Do: Lot no.

WB 3

GD 3
‘Warehouse Prc: sesion active
Prc: sesion active
Prc: sesion active

Booking’ Booking Receipt

Enter Transfer Details
Di: Lot no.
Enter Delivery Details
Di: Lot no.

activity as
Do: Do: Do:

GD 4
shown in the Prc: sesion active
Prc: sesion active

Transfer Details Confirm Delivery Details Confirm

left most Di:


branch of the

GD 5

Prc: sesion active Prc: sesion active

SNN. Poc:

Transfer Receipt

Delivery Receipt
Di: Di:
Do: Do:
Lecture 12 Inspection Testing: Cognitive
Cognitive Walkthrough Sheet example
Task description: Warehouse Booking WB1(-MT3) -
Tester name: ………..
Problems identified:
1. Failure to identify a goal
in the booking detail form, there is a ‘+’ sign which
indicates a facility to add a new transporter if it does not
exist in the list. User might fail to identify this goal.
2. Failure to identify the correct flow
after clicking ‘+’ sign and transferred to the ‘Add New
Transporter’ form (MT3), the user might fail to identify
the correct action to come back to the enter booking form.
Lecture 12 Inspection Testing: Cognitive

Cognitive Walkthrough Sheet example (ctd.)

3. Correct action does not match goal

in the booking confirmation form, the user might want
to edit the booking details (goal=edit). The correct
action here is to click on the ‘Cancel’ button, which
does not match to the user’s goal.
Lecture 12
Inspection Testing: Cognitive

• Cognitive walkthrough requires an in depth, step-by-step

evaluation of the interface.
• However, it often requires a substantial investment in
• See also:
Lecture 12 Inspection Testing: Heuristic Evaluation

• In heuristic evaluation, the interface is analysed based

on a set of known heuristic rules.
• In general, those rules can be summarised into the
following nine usability heuristics:
1. Visibility of system status
2. Match between system and real world
3. User control and freedom
look & feel or
4. Consistency and standards functionality?
5. Error prevention
subjective &
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimal design
9. Helping users recognise, diagnose and recover
from errors
Lecture 12
Inspection Testing: Heuristic Evaluation

• Heuristic evaluation can be improved by a group of

inspectors working independently on the same system
interface. The inspectors do their initial evaluations
individually and then collaborate afterwards.
• BENEFITS of Heuristic evaluation:
– intuitive and easy to motivate people to do it
– does not require advanced planning
–can be used early in the development process
Lecture 12 Inspection Testing: Heuristic Evaluation

Example of Heuristic issues:

• Visibility of system status
– Does the system let one know where one is (site
title, page title, window title)?
– Does the system acknowledge that an action
requested has been carried out?
• Consistency and standard with respect to the following:
– Message display methods (prompts, warning, helps)
– Colour use (entry form, menu/submenu,
– Keys definition
– Data entry method
– Navigation, dialogue, window display method
– Search facility
Lecture 12
Inspection Testing: Heuristic Evaluation

Example of Heuristic issues (ctd.):

• Match between system and real world
– Menu hierarchy that is consistent with the real
– Terminology used is the same as in real life of that
• Flexibility and efficiency of use:
– Browser compatibility
– Time taken for task completion
– Unproductive period
– Download time
Lecture 12
Inspection Testing: Heuristic Evaluation

Example of Heuristic issues (ctd.):

• Helping users recognise, diagnose and recover
from errors:
– Does the system explain why an action cannot be
– Does the system give prompts on how to proceed?
– If an action is not allowed, does thet system give the
Lecture 12
Inspection Testing: Heuristic Evaluation

• Heuristic evaluation is one of the most widely used

method at assessing the usability of an interface at the
design stage.
• Heuristic evaluation has been described as the usability
assessment method which identifies the most serious
usability problems with the least amount of effort.
• However, it can only be used to find usability problems.
It cannot assess the performance of the user while they
interact with the system like empirical user testing can.
• Another problem is that heuristic evaluation is biased
towards more experienced inspectors, since they seem
to predict more problems than people without usability
Further reading on Heuristic
• Case studies:

• Background:
Next Lecture ...

Next Lecture:
Usability Testing - part 2