ITSM University of Computer Studies HCI

Human–Computer Interaction

Chapter 9
Evaluation techniques

Human–Computer Interaction Group

Department of Information Technology Supporting and Maintenance
ITSM Objectives HCI

• To enable users to perform their intended tasks more

easily.
• To identify specific problems with the design.
• To evaluate user experience of interaction.
ITSM Learning Outcomes HCI

• A design can be changed more easily at the beginning of the process than
at the end.
ITSM Outlines HCI

• Introduction
• Evaluation Techniques
• Goals of evaluation
• Evaluation design
• Evaluating through user participation
• Evaluating implementations
• Observational methods
• Choosing an evaluation methods
• Summary
• References
• Next lecture
ITSM Introduction HCI

• Evaluation tests the usability, functionality and acceptability of an

interactive system.
• Evaluation techniques focus on two main themes: expert analysis and
user involvement.
• Evaluation that studies actual use of the system.
 HCI
ITSM
Evaluation Techniques

Evaluation
-tests usability and functionality of system
-occurs in laboratory, field and/or in collaboration with users
-evaluates both design and implementation
-should be considered at all stages in the design life cycle
ITSM Goals of Evaluation HCI

• assess extent of system functionality

• assess effect of interface on user

• identify specific problems

ITSM Evaluating Designs HCI

Cognitive Walkthrough
Heuristic Evaluation
Review-based evaluation
ITSM Cognitive Walkthrough HCI

Proposed by Polson et al.

-The sequence of actions refers to the steps that an interface will
require a user to perform in order to accomplish some known task
-Main focus is to establish how easy a system is to learn
ITSM
To do a walkthrough, you need 4 things HCI

• A specification or prototype of the system. It doesn’t have

to be complete, but it should be fairly detailed.
• A description of the task the user is to perform on the
system.
• A complete, written list of the actions needed to complete
the task with the proposed system.
• An indication of who the users are and what kind of
experience and knowledge the evaluators can assume about
them
ITSM Heuristic Evaluation HCI

• Proposed by Nielsen and Molich.

• Is a guideline or general principle or rule of thumb can

guide a decision or be used to critique a decision that has
already been made.
• Usability criteria are identified

• Heuristic evaluation `debugs' design.

ITSM Nielsen’s ten heuristics HCI

1. Visibility of system state

2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Error Prevention
6. Recognition rather than recall
7. Flexibility and efficiency
8. Aesthetic and minimalist design
9. Help users recognize and recover from
errors
10. Help and documentation
ITSM Review-based evaluation HCI

• Results from the literature used to support or refute

parts of design.

• Care needed to ensure results are transferable to new

design.

• Model-based evaluation

• Cognitive models used to filter design options

• e.g. GOMS prediction of user performance.

• Design rationale can also provide useful evaluation

information
ITSM HCI

Evaluating through user Participation

ITSM HCI
Styles of Evaluation

Two distinct evaluation styles

-Laboratory
-Field
ITSM Laboratory studies HCI

- Users are taken out of their normal work environment to

take part in controlled test, often in a specialist usability
laboratory

• Advantages:
• Specialist equipment available
• uninterrupted environment

• Disadvantages:
• lack of context
• difficult to observe several users cooperating
ITSM Field Studies HCI

- Takes the designer or evaluator out into the user’s work

environment in order to observe the system on action

• Advantages:
• natural environment
• context retained (though observation may alter it)

• Disadvantages:
• distractions
• noise
ITSM Experimental evaluation HCI

• To use a controlled experiment

• evaluator chooses hypothesis to be tested
• a number of experimental conditions are considered which
differ only in the value of some controlled variable.
• changes in behavioural measure are attributed to different
conditions
ITSM Experimental factors HCI

• Subjects
• who – representative, sufficient sample
• Variables
• things to modify and measure
• Hypothesis
• what you’d like to show
• Experimental design
• how you are going to do it
ITSM Variables HCI

• independent variable (IV)

characteristic changed to produce different conditions
e.g. interface style, number of menu items

• dependent variable (DV)

characteristics measured in the experiment
e.g. time taken, number of errors.
ITSM Hypothesis HCI

• prediction of outcome
• framed in terms of IV and DV

e.g. “error rate will increase as font size decreases”

• null hypothesis:
• states no difference between conditions
• aim is to disprove this

e.g. null hyp. = “no change with font size”

ITSM Experimental design HCI

• within groups design

• each subject performs experiment under each condition.
• transfer of learning possible
• less costly and less likely to suffer from user variation.
• between groups design
• each subject performs under only one condition
• no transfer of learning
• more users required
• variation can bias results.
ITSM Analysis of data (statistical Measure) HCI

• Before you start to do any statistics:

• look at data
• save original data

• Choice of statistical technique depends on

• type of data
• information required

• Type of data
• discrete - finite number of values
• continuous - any value
ITSM Observational Methods HCI

Think Aloud
Cooperative evaluation
Protocol analysis
Automated analysis
Post-task walkthroughs
ITSM Query Techniques HCI

Interviews
Questionnaires
ITSM Interviews HCI

• analyst questions user on one-to -one basis

usually based on prepared questions

• Advantages
• can be varied to suit context
• can elicit user views and identify unanticipated
problems
• Disadvantages
• very subjective
• time consuming
ITSM Questionnaires HCI

• Set of fixed questions given to users

• Advantages
-quick and reaches large user group
-can be analyzed more rigorously
• Disadvantages
-less flexible
-less probing
ITSM Questionnaires (ctd) HCI

• Need careful design

-what information is required?
-how are answers to be analyzed?

• Styles of question
-general
-open-ended
-scalar
-multi-choice
-ranked
ITSM
Choosing an Evaluation Method HCI

1.The stage in the cycle at which the evaluation is carried out

2.The style of evaluation
3.The level of subjectivity or objectivity of the technique
4.The type of measures provided
5.The information provided
6.The immediacy of the response
7.The level of interference implied
8.The resources required
ITSM Summary HCI

• Evaluation is part of the design process and should happen

throughout the design process. Its goal is to test the design and find
and fix any problems.
ITSM Reference HCI

• Alan Dix , Janet Finlay, Gregory D. Abowd, Russell Beale ,(2004),

Human–Computer Interaction,(third edition),Pearson Education Limited
ITSM Next Lecture HCI

Chapter10

Universal design