See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/274979188

Eye Movement-Based Interface Evaluation: What can and Cannot be Assessed?

Article in Proceedings of the Human Factors and Ergonomics Society Annual Meeting · July 2000
DOI: 10.1177/154193120004403721

CITATIONS READS
10 330

1 author:

Joseph H. Goldberg
Meta
78 PUBLICATIONS 4,864 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Optimal Product Bundling in Recommender Systems View project

Eye Tracking View project

All content following this page was uploaded by Joseph H. Goldberg on 03 August 2015.

The user has requested enhancement of the downloaded file.

 EYE MOVEMENT-BASED INTERFACE EVALUATION:
WHAT CAN AND CANNOT BE ASSESSED?

Joseph H. Goldberg
Pennsylvania State University
University Park, Pennsylvania

Interface evaluation by eye tracking-derived data is discussed in this review and synthesis
paper. While analysis of eye movements during interface use is becoming more popular,
there is little basis for justification of eye tracking methods. A review of traditional
interface assessment methods and criteria is provided, to establish areas where eye
tracking may potentially impact interface evaluations. Studies are then reviewed, that
have used eye tracking-derived measures for performance assessment as interfaces are
manipulated. A synthesis is then provided, suggesting that eye tracking-based analysis
could have a positive impact in evaluations of consistency, resources, visual clarity, and
flexibility, and should have difficulty in ascertaining interface compatibility and locus of
control.

INTERFACE USABILITY ASSESSMENT AND
CRITERIA
A hardware or software interface should allow
individuals to effectively achieve their goals with
minimal effort and confusion, while achieving a
high level of satisfaction (Jordan, 1998). The
assessment of effectiveness, efficiency, and
satisfaction at an interface thus comprises a large
portion of most usability evaluations. Evaluation
approaches may be based upon theory,
guidelines/standards, experiments, protocols,
surveys/interviews, or walkthroughs. Each method
has its advantages and disadvantages (see Jordan,
1998; Dix et al., 1998). Generally, more subjective
methodologies (e.g., cognitive walkthroughs or
theory-based approaches) are applied to early
concepts, whereas more formalized, objective
approaches (e.g., experiments) are applied during
latter design stages. Many approaches have been
critiqued due to high resource costs, lengthy
implementation, subjectivity, and/or poor access to
users' actual performance and behaviors (Jordan,
1998). Some assessment methods (e.g., task-based
evaluations) can intrude upon or change the nature
of the actual tasks under evaluation. Although users
may complete the objectives for a task (high
effectiveness), they may expend tremendous
resources in this process (poor efficiency).
Usability evaluations are usually conducted with
respect to specific criteria. While many criteria
taxonomies exist, several criteria are universally
found (e.g., Shneiderman, 1998; Jordan, 1998; Dix,
et al., 1998), as listed in Table 1. Usability is
generally enhanced by the degree to which these are
criteria are satisfied.
EYE TRACKING ON INTERFACES
Analysis of eye movements during one's
interaction with an interface can provide otherwise
covert information underlying attention and
decision making strategies. As a methodology, eye
tracking measures the location of one's gazepoint at
relatively rapid frequency (e.g., 60 Hz) and
Table 1. Common Evaluation Criteria for Interface Usability
Evaluation Description
Criteria
Consistency Similar tasks must be performed in
similar ways
Compatibility Interface operation must match user
expectations
Locus of Control Users must feel that they are in control
of the interface
Feedback Every interface event must provide
feedback to the user
Minimum User resource demands (e.g., memory
Resources and attention) must be controlled
Error Handling User errors must be minimized, and
easily handled when they occur
Visual Clarity Visual components must be easily and
rapidly identified
Flexibility Interaction must accommodate broad
range of users
reasonable accuracy (usually within 1° of visual
angle). While one's current attentional focus may
lead or lag the current gaze location, the current
gaze location is usually assumed to reflect what is
on top of the stack of cognitive operations (Just and
Carpenter, 1976; Potter, 1983).
The goal of the present work is to determine
whether eye tracking-derived parameters are
sensitive to some aspects of usability at an interface.
Positive indications in this direction can potentially
allow the development of more rapid and objective
measures of usability than currently exist. A few
published studies have relied upon gazepoint data
from eye tracking methodologies, while
manipulating aspects of interfaces.
Text manipulations during computer interface
search have clearly resulted in eye movement-
related effects. Using fixation-related parameters,
Kolers, Duchnicky, and Ferguson (1981) noted that
reading rates were faster with more densely packed
characters, as they manipulated line spacing,
character density, and text scrolling rate. Yamamoto
and Kuto (1992) found that more text columns in a
display lead to more re-fixations as individuals
searched for target words in a matrix.
Ellis, et al. (1998) evaluated four different
versions of a web page by manipulating the size and
layout of text blocks, inclusion of supporting
graphics, and the percentage of words that were
hyperlink phrases. Based upon eye tracking data,
dense text with wide paragraphs and no graphics
supported the fastest search, but was also the least
preferred of the interfaces. Dense text produced
shorter eye fixations to reject irrelevant targets,
allowing more information per fixation.
Goldberg and Kotval (1998) manipulated the
grouping of graphical tools in a drawing package-
like interface. A large group of interface users and
professional designers rated the subjective usability
of each interface and grouping. When required to
rapidly find a particular tool, eye movement
patterns were strongly related to rated usability, as
well as to timed search performance.
Goldberg, Probart, and Zak (in press) used eye
movement-derived parameters to determine
precisely when desired information is located on
presented nutrition labels. More eye movements
were made in denser information sections of the
labels, and fixation durations were sensitive to
manipulations of lines separating areas of the labels.
Eye tracking as a methodology has also aided
the interpretation of display hardware quality. For
example, Deffner (1995) showed that fixation
locations and durations are sensitive to perceived
visual clarity of images presented on alternative TV
displays.
While the preceding studies have only initiated
research in this area, it is clear that eye movement
derived parameters are sensitive to at least some
manipulations of interface quality. Most of the
preceding tasks have contained a visual search
component, in which users attempt to locate a
memorized or known object on a display. Further
discussion, below, will consider which types of eye
tracking-derived measures should be sensitive to
which aspects of usability on displays.
EYE TRACKING MEASURE CATEGORIES
Although an organized taxonomy of eye
movement-derived parameters has never been
developed, some classification can be attempted
here, in order to provide some generalizations about
eye movement based usability evaluation.
The following categories of measures assume
that samples of gazepoint locations (x, y locations)
have acquired, then broken into saccades and
fixations. Scanpaths are defined by linking
successive fixation locations from an observer's
data record. Summaries of these procedures are
available in Goldberg and Schryver (1995) and
Goldberg and Kotval (1999).
Scanpaths
Consisting of sequences of saccades and
fixations, scanpaths define the serial/temporal
record of attention during interaction with an
interface. While there is a large stochastic element
to scanpath shape, overall cycles can be found in
experienced visual searchers (Goldberg and Kotval,
1998). Aspects of scanpaths, such as mean fixation
duration, ratio of number of saccades to fixation
duration, and number of acute angles between
successive saccades can indicate complexity of
visual processing and salience of visual targets
(Goldberg and Kotval, 1999; Kotval and Goldberg,
1998).
Cumulative Time in Areas of Interest (AOIs)
An interface can be divided specific areas, with
cumulative visiting (or fixation) time measured in
each of these areas. Areas may be defined by a
regular grid, or by irregular individual objects or
features. Referring to defined areas on an interface,
message or interface component salience is
routinely assessed by cumulative time spent within
AOIs. Interference from unintended AOIs may also
be assessed. Because AOIs may differ in size and
shape, these approaches are best suited for relative
comparison of condition manipulations within a
study, as opposed to between-study assessment of
interface usability. Extent and balance of coverage
on an interface can be assessed from these
cumulative times.
Transitions Among AOIs
The number and probability of transitions
among AOIs provides information about search
strategies on an interface. Compact representations,
such as transition matrices and Markov models are
feasible. Emphasis on transitions between AOIs
allows an assessment of search efficiency and
complexity on an interface. Strategies and
novice/expert differences, as in judging the skill of
airplane pilots, can be obtained from analysis of
transitions. Scanpaths are another form of transition
representation. Variance in transitions are also a
meaningful way to judge the usability of an
interface. Comparison of transitions among AOIs
or portions of an interface can be made by
comparing transition probabilities among these
areas.
Similarity among transition records can be done
by direct transition matrix comparisons, or by the
process of string editing, where the number of
deletions and substitutions required to transform
one record to another defines their similarity
(Hacisalihzade, Stark, and Allen, 1992).
SYNTHESIS: EYE MOVEMENT
ASSESSMENT OF EVALUATION CRITERIA
A synthesis among the preceding discussions of
usability assessment, eye movements on displays,
and categories of eye tracking-derived dependent
measures can now be attempted, with the objective
of establishing which portions of a usability
assessment could benefit from an eye tracking
approach.
Nature of Task
Candidate interface tasks clearly require active
eye movements to be sensitive to eye tracking-
derived measures. These include visual search and
visual monitoring tasks, but could also include time
shared tasks, as well as those with some element of
stimulus movement. Long duration tasks, with
some element of visual fatigue, should also be
sensitive to these measures.
Tasks that likely will not provide useful eye-
tracking data include those with little eye
movement, such as pure memory and decision
making tasks, as well as auditory input tasks. Little
is known about the response of eye movements to
variations in output requirements, such as vocal
versus manual output.
Eye Movements and Usability Criteria
At this early stage of research predications of
relationships between eye movement-derived
parameters and usability criteria are admittedly
speculative. Each prediction, however, should
result in testable hypotheses. To consider whether
there is a relationship with each of these criteria,
one can consider whether quality of each criterion
could be ascertained from eye tracking records.
Table 2 lists each of the previously stated usability
criteria, then assess whether a relationship exists
between that criterion and eye tracking-derived
measures. Specific predications are then provided
in the third column.
The preceding discussion has focused on the
role of eye movement-based data within interface
evaluations. Using 'classic' criteria, eye tracking
should provide positive contributions to evaluations
of consistency, cognitive resources, visual clarity,
and flexibility. Limited information may be Clearly, a substantial amount of research is required
achieved for interface feedback and error handling. to enable eye tracking to serve as a valid and
Eye tracking should have difficulty in ascertaining reliable interface evaluation tool.
interface compatibility and locus of control.

Table 2. Estimates of Eye Tracking Data Relationship by Evaluation Criteria

Evaluation Assessment Predictions and Explanations
Criteria Estimate
Consistency Good Consistent interfaces should result in similar cumulative AOI times, as well as display coverage.
However, AOI transitions are not expected to be similar unless users are highly homogeneous and
similarly trained.
Compatibility Poor Compatibility should be difficult to assess, because of its reliance upon a cognitive expectation. In
novel displays, unexpected areas draw visual attention, but most routine usage has significant context
and expectation by the user.
Locus of Poor While pacing and timing of interface interaction can be assessed from eye movements, it will be
Control difficult to determine whether the user or system is controlling the interaction, from these records.
This could become a promising evaluation approach if this issue can be solved.
Feedback Limited Feedback on an interface must be salient, and must be properly interpreted by the user. Eye
movement data can greatly aid the determination of whether feedback was observed, but cannot
significantly aid whether it has been properly interpreted. Auditory feedback also presents a
challenge to eye movement-based evaluation.
Cognitive Good As more complex decisions and greater focused attention with greater memory loads generally result
Resources in longer fixation durations, resources required by an interface can generally be evaluated by an eye
tracking approach.
Error Limited Eye tracking can provide some strategy analysis underlying an improper choice at an interface, but it
Handling cannot provide information concerning system error handling.
Visual Clarity Excellent One of the primary benefits of an eye tracking evaluation approach is in understanding the visual
clarity and salience of displayed objects. High visual clarity in search tasks should result in directed
scanpaths, sparse AOI transition matrices, and little unnecessary coverage of a display.
Flexibility Good In a similar analysis as for consistency, a flexible interface should result in a large variance of
scanpaths and display coverage between and within users, given that poor visual clarity has been
ruled out. These eye movement data should reflect that interface operations should be easily located
and executed in multiple ways.

REFERENCES
Deffner, G. (1995), Eye movements recordings to study determinants
of image quality in new display technology, in Findlay, J.M., et
al. (Eds.), Eye Movement Research, Elsevier Science Publishers.
Dix, A., Finlay, J., Abowd, G., and Beale, R. (1998), Human-
Computer Interaction, 2nd Ed., London: Prentice Hall Europe.
Ellis, S., Candrea, R., Misner, J., Craig, C.S., Lankford, C.P., and
Hutchinson, T.E. (1998), Using eye tracking data to help build
better web pages (abstract), Proc. of the 42nd Annual Meeting of
the Human Factors and Ergonomics Society, p. 1613.
Goldberg, J.H. and Kotval, X.P. (1998), “Eye Movement-Based
Evaluation of the Computer Interface,” in Kumar, S.K. (Ed.),
Advances in Occupational Ergonomics and Safety, Amsterdam:
IOS Press, pp. 529-532.
Goldberg, J.H., and Kotval, X.P. (1999), "Computer Interface
Evaluation Using Eye Movements: Methods and Constructs,"
International Journal of Industrial Ergonomics, 24: 631-645.
Goldberg, J.H., and Schryver, J.C. (1995), “Eye-Gaze Contingent
Control of the Computer Interface: Methodology and Example
for Zoom Detection,” Behavior Research Methods, Instruments
& Computers, 27(3): 338-350.
Goldberg, J.H., Zak, R.E., and Probart, K. (in press), “Visual Search
of Food Nutrition Labels,” Human Factors.
Hacisalihzade, S.S., Stark, L.W., and Allen, J.S. (1992), Visual
perception and sequences of eye movement fixations: A
stochastic modeling approach, IEEE Transactions on Systems,
Man, and Cybernetics, 22(3): 474-481.
Jordan, P.W. (1998), An Introduction to Usability, Bristol, PA:
Taylor & Francis.
Just, M.A., and Carpenter, P.A. (1976), The role of eye-fixation
research in cognitive psychology, Behavior Research Methods
& Instrumentation, 8(2): 139-143.
Kolers, P.A., Duchnicky, R.L., and Ferguson, D.C. (1981), Eye
movement measurement of readability of CRT displays, Human
Factors, 23(5): 517-527.
Kotval, X.P., and Goldberg, J.H. (1998), "Eye Movements and
Interface Components Grouping: An Evaluation Method," Proc.
of the 42nd Annual Meeting of the Human Factors and
Ergonomics Society, Santa Monica: HFES, pp. 486-490.
Potter, M.C. (1983), Representational Buffers: The Eye-Mind
Hypothesis in Picture Perception, Reading, and Visual Search,
Ch. 24 in Rayner, K. (Ed.), Eye Movements in Reading:
Perceptual and Language Processes, Academic Press, New
York.
Shneiderman, B. (1998), Designing the User Interface, 3rd. Ed.,
Reading, MA: Addison Wesley Longman.
Yamamoto, S., and Kuto, Y. (1992), A method of evaluating VDT
screen layout by eye movement analysis, Ergonomics, 35(5/6):
591-606.
View publication stats