You are on page 1of 31

Lect ure Topi c 12

Usabi lity Testing - part 1

Dr. SENG LOKE

References:
E-Commerce Fundamentals and Applications, by Henry Chan et al., John
Wiley and Sons Pty Ltd, 2001
Lecture 12 USABILITY
• Usability can be defined as:
“the extend to which end-users can successfully accomplish
required goals in a specified environment subject to various
comfort, effectiveness and satisfaction criteria” (Thimbleby, SIGCHI94)
• Usability is becoming an integral part of the software
development lifecycle.
• Software developers aim at improving their product from
the end-users viewpoint which involves identifying
problems with the interface, and after some careful
analysis of the problems, the interface is redesigned with
the hindering aspects removed.
• The interface design is the main factor affecting usability,
since it is the only part of the system with which the user
interacts with.
Lecture 12 USABILITY TESTING

• Usability testing usually entails building prototypes of


the system and testing it on users.
• Usability testing is a broad context where the interface
is put under a test, with users critically analysing the
system, evaluating the design layout and flow of
interactions by interacting with the implemented
version of the system.
• The main reasons for adopting this approach are that it
can be used :
– to measure user performance and satisfaction,
– to identify and assess usability defects,
– to provide ideas as to how a system might be
improved,
– to educate design teams.
Lecture 12
USABILITY TESTING

• Many usability testing strategies have been developed.


Most of these testing can be categorised into two main
categories:
– Empirical Testing
– Inspection Testing
Lecture 12
USABILITY TESTING

Phases of product usability testing


Phase Purposes Objects

Investigation •
Set usability objectives •
Existing solutions

Discover opportunities
Design and build •
Resolve conflicts •
Early concepts

Answer questions •
Simulations

Debug •
Prototypes

Measure against •
Partial products
objectives •
Training

Documentation
Post release •
Get data for •
Finished product
sales/marketing •
Competitive product

Input to next release

Competitive analysis
Lecture 12 Empirical Testing

• Empirical testing involves the user testing the product


under review in a testing laboratory.
• Users are asked to use the system to complete a work
related task that is likely to target problem areas.
• The users’ task results are recorded, often through a
video camera.
• The users’ concerns, which are expressed through the
use of a questionnaire, are also documented.
• The results are analysed by the development team in an
attempt to refine their product.
• The primary focus of empirical testing if the product is
still in a prototype stage is to use the results of the
testing for refining the final version of the product.
Lecture 12 Empirical Testing

• If the product to be tested is in the final stage of


development, this approach is useful for determining
whether one selects a given product.
• There are three types of Empirical testing:
–Fixed Laboratory Usability Testing
–Portable Usability Testing
– Remote Usability Testing
Lecture 12
Empirical Testing : Fixed Laboratory
Testing
• Fixed laboratory testing involves the usability testing of
a product in a fixed laboratory. This method is often
referred to as in-house testing, since the users have to
come to the developer to complete the usability test.
• The user interact with the product by completing a
work-related task which is created by the development
team.
• To assist the development team in analysing the results
of the usability test, the tests are recorded using a video
camera installed in the laboratory.
Lecture 12
Empirical Testing : Fixed Laboratory
Testing
• ADVANTAGE:
– All necessary equipment is already in place. This
will make it easier for the development team to
conduct the usability test and record all essential
data.
• LIMITATION:
– Can be very expensive, a fully equipped fixed
laboratory costs between $50,000 to $100,000 and
even more to operate and maintain. A big
proportion of the amount goes towards recruiting
and training of staff.
– The user has to come to the developer to perform
the test. (Explain what consequences this may cause
?)
Lecture 12
Empirical Testing : Fixed Laboratory
Testing
• IMPROVED Fixed Laboratory
– The recent version of fixed laboratory is designed
to address the problems of the earlier version of the
laboratory.
– The main emphasis is to design a comfortable,
natural office scenario for the users and to
eliminate the artificial element that is often
associated with usability testing in a fixed
laboratory.
– This improved version fixed laboratory has a
separate observation room, which is linked to the
main laboratory through a one-way glass. This
enables the observer to view the tester without the
tester conscientious of an observer peering over
their shoulder.
Lecture 12
Empirical Testing : Fixed Laboratory
Testing
• IMPROVED Fixed Laboratory (ctd.)
– This new layout enables testers to proceed without
interruption as they would while performing the task
in their own work environment.
– Very often, the layout of the observation room is as
similar as possible to the users real working
environment.
– However, this improved laboratory still does not
address the problem of high costs.
– Note that this fixed laboratory testing is normally
used for building a new sophisticated software
system or a new operating system whose usability
will not be affected by internet connection, download
time etc. (eg. Microsoft windows operating system
releases).
Lecture 12
Empirical Testing: Portable Usability
Testing
• Portable usability testing was introduced to address the
problem of high cost found in a fixed laboratory setting.
• Portable usability testing takes the testing laboratory to
the users.
• The product is tested while the users interact with it in
their own work environment.
Lecture 12
Empirical Testing: Portable Usability
Testing
• Portable Testing is best suited when the customer base
is widely distributed.
• The developers will test a wide range of users from
each of the customer sites.
• The recording of the task results of the user is best
accomplished with the use of a video camera. However,
an over the shoulder video camera can place undue
stress on the users, which could jeopardise the field test
results.
• The recording equipment must be setup in such a
manner that it does not interfere with the usability
testing process.
Lecture 12
Empirical Testing: Portable Usability
Testing
• The SETUP:
• The computer output is directed to two
monitors.
• The video camera should focus on the second
monitor.
• The test results can then be recorded without
the video equipment getting in the user’s way.
• Another important element is to encourage the users to
think out loud while they undertake the tasks. These
verbal comments can prove to be valuable for analysis.
Lecture 12
Empirical Testing: Remote Usability
Testing
• Remote usability testing is a usability testing where
neither the developers or the users have to travel.
• Users are observed and tested through the aid of
computer networks and modem connections in a
time efficient and cost effective manner.
• Can you think of any limitations of this type of
usability testing?
Lecture 12
Empirical Testing: Summary

• In general, the main advantage of empirical testing


is that it is able to directly monitor the performance
reactions of the user to the particular interface and
one does not have to rely on a usability expert
trying to second guess what a user’s reaction might
be.
• The disadvantage is that it is only useful for testing
an implemented product, and therefore the ability to
correct the problem immediately is greatly reduced.
Lecture 12 Empirical Testing: Test Procedure
• The test layout in an empirical testing consists of:
– the test subject (the user)
– the computer with the software being tested
– the test monitor (the tester)
• The user must perform a set of tasks and answer a set of
questionnaire prepared earlier by the tester. The tasks must
be designed such that:
– they be work related to stimulate the manner in which the
software would actually be used in the work environment
– they cover most of the significant functions of the package
so that all significant and frequently used features of the
interface are tested
– the time of the test session clearly reflects work periods
and is not too short (too artificial) or too long (may cause
tiredness).
Lecture 12
Inspection Testing

• Usability Inspection testing is often referred to as


“discount usability engineering methods”.
• Inspection testing is performed by usability inspectors
who will inspect the design layout of an interface and
comment based on the result of their test and
experiences.
• Results from inspection methods influence the design
layout of an interface since it captures potential
problems at an early stage in the development life cycle.
• Most commonly used Inspection testing procedures are:
– Cognitive Walkthrough
– Heuristic Evaluation
Lecture 12
Inspection Testing: Cognitive
Walkthrough
• People who are evaluating the interface (called the
inspectors) are asked to answer a set of derived
questions for each step of the user task scenario. These
questions are based on the relationships between user’s
goals and the actions available for achieving these.
• This inspection testing can be done during the design
stage especially when the dynamic aspects of the web
site/application is determined.
• The user task scenario can be drawn from the Web
Navigation Analysis Model (lecture 10) by defining a
scenario for each branch of the hierarchical structure in
the Site Navigation Net.
Lecture 12 Inspection Testing: Cognitive
Walkthrough
SNN2

GT1
• For example,
B1 GD
1
W

we can Prc: sesion active


Poc:
Prc: sesion active
Poc:
Prc: sesion active
Poc:

perform a
Enter Booking Details Search Lot Search Lot
Di: MT3 Di: Di:
Do: Booking details Do: Do:

cognitive

WB2

GD 2
GT2
walkthrough
Prc: sesion active Prc: sesion active Prc: sesion active
Poc: Poc: Poc:

GT1.2

GD 1.2
Booking Details Confirm Lot Result List Lot Result List

test on the Di: Booking details


Do:
Di: Warehouse ID
Do: Lot.no.
Di: Warehouse ID
Do: Lot no.

WB 3

GD 3
GT3
‘Warehouse Prc: sesion active
Poc:
Prc: sesion active
Poc:
Prc: sesion active
Poc:

Booking’ Booking Receipt


Di:
Enter Transfer Details
Di: Lot no.
MC3
Enter Delivery Details
Di: Lot no.
MT3

activity as
Do: Do: Do:

GD 4
GT4
shown in the Prc: sesion active
Poc:
Prc: sesion active
Poc:

Transfer Details Confirm Delivery Details Confirm

left most Di:


Do:
Di:
Do:

branch of the

GD 5
GT5

Prc: sesion active Prc: sesion active

SNN. Poc:

Transfer Receipt
Poc:

Delivery Receipt
Di: Di:
Do: Do:
Lecture 12 Inspection Testing: Cognitive
Walkthrough
Cognitive Walkthrough Sheet example
Task description: Warehouse Booking WB1(-MT3) -
WB2-WB3
Tester name: ………..
Problems identified:
1. Failure to identify a goal
in the booking detail form, there is a ‘+’ sign which
indicates a facility to add a new transporter if it does not
exist in the list. User might fail to identify this goal.
2. Failure to identify the correct flow
after clicking ‘+’ sign and transferred to the ‘Add New
Transporter’ form (MT3), the user might fail to identify
the correct action to come back to the enter booking form.
Lecture 12 Inspection Testing: Cognitive
Walkthrough

Cognitive Walkthrough Sheet example (ctd.)

3. Correct action does not match goal


in the booking confirmation form, the user might want
to edit the booking details (goal=edit). The correct
action here is to click on the ‘Cancel’ button, which
does not match to the user’s goal.
Lecture 12
Inspection Testing: Cognitive
Walkthrough

• Cognitive walkthrough requires an in depth, step-by-step


evaluation of the interface.
• However, it often requires a substantial investment in
training.
• See also:
http://www.infodesign.com.au/usabilityresources/evaluation/conductingwalkthroughs.asp
Lecture 12 Inspection Testing: Heuristic Evaluation

• In heuristic evaluation, the interface is analysed based


on a set of known heuristic rules.
• In general, those rules can be summarised into the
following nine usability heuristics:
1. Visibility of system status
2. Match between system and real world
3. User control and freedom
look & feel or
4. Consistency and standards functionality?
5. Error prevention
subjective &
6. Recognition rather than recall
objective?
7. Flexibility and efficiency of use
8. Aesthetic and minimal design
9. Helping users recognise, diagnose and recover
from errors
Lecture 12
Inspection Testing: Heuristic Evaluation

• Heuristic evaluation can be improved by a group of


inspectors working independently on the same system
interface. The inspectors do their initial evaluations
individually and then collaborate afterwards.
• BENEFITS of Heuristic evaluation:
– intuitive and easy to motivate people to do it
– does not require advanced planning
–can be used early in the development process
Lecture 12 Inspection Testing: Heuristic Evaluation

Example of Heuristic issues:


• Visibility of system status
– Does the system let one know where one is (site
title, page title, window title)?
– Does the system acknowledge that an action
requested has been carried out?
• Consistency and standard with respect to the following:
– Message display methods (prompts, warning, helps)
– Colour use (entry form, menu/submenu,
foreground/background)
– Keys definition
– Data entry method
– Navigation, dialogue, window display method
– Search facility
Lecture 12
Inspection Testing: Heuristic Evaluation

Example of Heuristic issues (ctd.):


• Match between system and real world
– Menu hierarchy that is consistent with the real
world
– Terminology used is the same as in real life of that
domain
• Flexibility and efficiency of use:
– Browser compatibility
– Time taken for task completion
– Unproductive period
– Download time
Lecture 12
Inspection Testing: Heuristic Evaluation

Example of Heuristic issues (ctd.):


• Helping users recognise, diagnose and recover
from errors:
– Does the system explain why an action cannot be
performed?
– Does the system give prompts on how to proceed?
– If an action is not allowed, does thet system give the
reason?
Lecture 12
Inspection Testing: Heuristic Evaluation

• Heuristic evaluation is one of the most widely used


method at assessing the usability of an interface at the
design stage.
• Heuristic evaluation has been described as the usability
assessment method which identifies the most serious
usability problems with the least amount of effort.
• However, it can only be used to find usability problems.
It cannot assess the performance of the user while they
interact with the system like empirical user testing can.
• Another problem is that heuristic evaluation is biased
towards more experienced inspectors, since they seem
to predict more problems than people without usability
expertise.
Further reading on Heuristic
Evaluation
• Case studies:
– http://www.bls.gov/ore/htm_papers/st960160.htm
– http://www.infodesign.com.au/usabilityresources/evaluation/webevaluation.asp

• Background:
– http://www.useit.com/papers/heuristic/
Next Lecture ...

Next Lecture:
Usability Testing - part 2