Applied Ergonomics 1989, 20.

4,255-260
User-computer interface

Towards a practical method of

user interface evaluation
G.I. Johnson t, C.W. Ctegg and S.J. Ravden*

MRC/ESRC Social & Applied Psychology Unit, University of Sheffield, Sheffield Sl0 2TN, UK
tnow at Nederlandse Philips Bedrijven BV, Applied Ergonomics Group, Corporate Industrial Design,
Eindhoven, The Netherlands
*now at Ciba-Geigy Plastics, Information Management Support Centre, Duxford, Cambridge, UK

This paper describes a practical method for evaluating the usability of human-computer
interfaces. The paper specifies the requirements of such a method, and then outlines our
work in developing a method to meet this specification. The method is based on the
conduct of realistic tasks with an interactive system and the subsequent systematic
elicitation of end-users' and designers' reactions to the interface using a criterion-based
evaluation checklist.
Two practical examples are used to illustrate development of the method: (a) evaluation
of a prototype production scheduling system, and (b) comparative assessment of the
usability of three prototype user interfaces to a public-access database. The paper discusses
some issues raised by the method and considers how it can be further developed.

Keywords. Computers, human-computer interaction, criterion-based evaluation

Introduction the-art). This reflects the growth of interest in this area,

The objectives of this paper are to:
• specify a set of requirements for a practical method for
evaluating the usability of human-computer interfaces;
• discuss the development and application of the method
in two very different evaluation settings;
• examine its success in meeting the stated requirements;
• consider the method's future development.
The paper is organised in six further parts. The first
briefly outlines current issues in user interface design and
evaluation, especially from a human factors perspective,
stressing the significance of 'usability'. The second section
offers a set of requirements for a method for evaluating the
usability of interfaces. The third section describes how the
method was developed and used, drawing on experience of
conducting two practical usability evaluations of very
different computer-based applications. In the following
section, the extent to which the method is believed to meet
the specified requirements is evaluated. Some views
concerning outstanding issues in this area are offered, and
the paper ends with some concluding remarks.
Current issues
There is a burgeoning literature focusing on user inter-
face design and development (e g, Rubin, 1988; Shneiderman,
1987; Sutcliffe, 1988) and human-computer interaction
(see Helander, 1988, for a thorough review of the state-of-
caused by the increasing use of computer-based systems, the
variety of people who are interacting with them, both
within and outside work, and the rapid pace of technol-
ogical advance.
Evaluation can be seen as an intrinsic element within the
process of user-interface design. Ideally, evaluation should
be an iterative process throughout design and development.
However, it more commonly occurs towards the very end
of the design process. The commercial justification for
consideration of the human aspects of interface design and
evaluation is clear. Sutcliffe (1988), for example, points
out that the cost of poor interface design can be:
" l . Increased mistakes in data entry and system operation.
Mistakes cost money to rectify and errors which go
undetected can have very damaging consequences if
decisions are taken on the basis of incorrect data.
"2. User f r u s t r a t i o n . . , may be manifest in low productivity
employee s t r e s s . . , or simple under-utilisation of the
system.
"3. Poor system performance. The system may not handle
the volume of throughput it was designed for, or the
accuracy of output may not agree with the specification.
Poor interface design makes it too cumbersome to use
or too obscure to learn. Extra resources and money
have to be put into the system." (p 3--4)
Similar arguments have been presented by Cohill et al
(1988) who comment on the benefits of user interface

0003 6870/89/04 0255 06 $03.00 © 1989 Butterworth & Co (Publishers) Ltd AppliedErgonomics December 1989 255
User-computer interface
evaluation in terms of increased user satisfaction, increased
sales, decreased development costs, increased productivity,
decreased product returns, and decreased training costs
(p 130-131). There is little doubt that user interface eval-
uation, as a crucial part of that process, warrants attention.
The way in which a simple, practical evaluation can be
conducted, however, remains largely unresolved. As the pace
of technological innovation quickens, and the design of
user interfaces involves more complex interaction techniques,
the need for simple evaluation tools is increasing.
In this paper we have not attempted to review the
existing work on user interface and software evaluation
methods. Rather, we focus on the specification and develop-
ment of a practical method. The reader is referred to Karat
(1988) and Lea (1988) who provide excellent reviews of the
area.
In specifying an evaluation technique we sought to pro-
vide the user of such a technique - for example, a software
engineer or user interface designer - with a method that
could be explained and used without reference to relatively
time-consuming and complex laboratory-based empirical
assessments. The general requirements of such a method are
presented in the next section.
Requirements of an evaluation method
The requirements outlined below are based on our
appreciation of the existing literature on user interface
evaluation, and on our work with systems and software
engineers in collaborative research and development
ventures (such as ESPRIT projects 534 and 1217 (1199)).
We therefore propose that a method for evaluating usability
of user interfaces should exhibit the following nine attributes.
It should be:
1. systematic - i e, it should be capable of being applied in
a relatively straightforward and routine manner;
2. based on existing criteria, that are derived from empirical
and theoretical work in the area;
3. iterative - i e, it should be capable of use within realistic
iterative design processes;
4. general, in that it can be applied in a range of settings on
a range of interfaces;
5. participative, in the sense that it can be used and under-
stood by a range of different 'experts' - for example,
including end-users, designers and human-factors
specialists;
6. sensitive - i e, capable of relatively fine discriminations
between the usability of similar interfaces;
7. simple to use by people who are unfamiliar with usability
issues;
8. face valid and related to realistic usage of the system; and
9. reasonably exhaustive, insofar as it considers most of the
relevant issues that constitute usability - i e, it should
not omit major factors affecting the usability of an
interface.
Ravden and Johnson (1989) describe a method which
attempts to meet these requirements. The method itself was
developed iteratively and refined through a number of
256 Applied Ergonomics December 1989
usability evaluation studies. The next section of the paper
describes this process and the progress towards our goal of
a practical usability evaluation method.
Development of the method
A large part of the literature in this area concerns the use
of design guidelines (see Mosier and Smith, 1986; Smith and
Mosier, 1986). However, there are a number of problems
with the usability and perceived utility of such guidelines
(Johnson et al, 1986; Smith and Mosier, 1984). Whilst based
on sound empirical work within such fields as cognitive
psychology (see, e g, Gardner and Christie, 1987), they are
not easily incorporated into the design and evaluation
process. The underlying principles or criteria on which
most guidelines are based, however, are valid.
The first stage in developing the method described in
this paper was to review the literature and establish a set of
major criteria for 'usable' interfaces (Ravden, 1988). These
criteria, presented in Table 1, were then translated into a
checklist of specific questions. The idea was that a standard
checklist could then be employed in an evaluation setting.
The checklist forms the core of the method, and
comprises specific questions about features of the interface
in relation to particular criteria. For instance, the criterion
of consistency incorporates a number of questions that
Table 1: The usability criteria
Visual clarity: Information displayed on the screen should
be clear, well prganised, unambiguous and easy to read.
Consistency: The way the system looks and works should
be compatible with user conventions and expectations.
Informative feedback: Users should be given clear,
informative feedback on where they are in the system,
what actions they have taken, whether these actions have
been successful and what actions should be taken next.
Explicitness: The way the system works and is structured
should be clear to the user.
Appropriate functionafity: The system shou Id meet the
needs and requirements of users when carrying out tasks.
Flexibility and control: The interface should be sufficiently
flexible in structure, in the way information is presented
and in terms of what the user can do, to suit the needs
and requirements of all users, and to allow them to feet
in control of the system.
Error prevention and control: The system should be designed
to minimise the possibility of user error, with inbuilt
facilities for detecting and handling those which do occur;
users should be able to check their inputs and to correct
errors or potential error situations before the input is
processed.
User guidance and support: Informative, easy-to-useand
relevant guidance and support should be provided, both
on the computer (via an on-line help facility) or in hard
copy document form, to help the user understand and
use the system.

could be asked concerning consistency of formatted text,
menu structure, action sequences, use of icons and abbrevia-
tions, and so on (see Ravden and Johnson (1989) t'or details
of the checklist). Table 2 illustrates a specific example
from the checklist.
The second key feature of this method is that a checklist
of this kind is best completed by people with experience of
undertaking realistic tasks using the system. Therefore,
before completing the checklist, evaluators (system designers,
human factors specilaists or end-users) should first carry out
one or more tasks using the system. These should be rep-
resentative of 'real work' using the system, and should test
as much of the system's functionality as possible.
Carrying out realistic tasks as part of the evaluation has
a number of benefits:
• the most effective way of exploring functionality is to
use the system;
• it enables evaluators to see the interface as part of the
overall system rather than as an artificial series of screens
and actions;
• many problems, difficulties and aspects of usability are
only exposed when carrying out real tasks;
Table 2." Example of the evaluation checklist (Reprinted by
permission of the publishers, Ellis Horwood, from
Ravden and Johnson, 1989)
(bl in the same l a y o u t ?
5. Does the appear in the same i n i t i a l
position on d i s p l a y s of a s i m i l a r type?
6. Is the same item of i n f o r m a t i o n d i s p l a y e d in the
same f o r m a t , w h e r e v e r it appears?
7. Es the format in w h i c h the user s h o u l d e n t e r
p a r t i c u l a r t y p e s of i n f o r m a t i o n on the screen
c o n s i s t e n t t h r o u g h t ~ u t the system!
8. Is the method of e n t e r i n g i n f o r m a t i o n c o n s i s t e n t
throughout the system?
9. Is the ,ict:,Jn r e q u i r e d to move the c u r s o r a r o u n d
the ( o n s l s t ~ r N t h r o u g h o u t the system?
lO. ]s t~e rnetPod of s e e c t i n g c~ptior~s e . g . from a
ITl~rlt ~ corqsisten[ / h r m ~ g h o u t the s~stelrl ~
I]. Whet,: ,i k e v b u a r d is used, Jre the same k e y s
u~sed f?r the' sarm f u n c t i o n s t h r o u g h o u t the
~,, ster,!"
12. Are t h e r e s t a H d a r d p r o c e d u r e s for c a r r y i n g o u t
s l m l a r , rehl',ed o p e r d t i o n s ? ( e , g , b u d d t m g a n d
delt, tlrlg infc~rmath~n ~ t a r t m g and f i n i s h i n g
t r d n s ~ctio,,s I
13. Is the way the system responds to a p a r t i c u l a r
user a c t i o n c o n s i s t e n t at all times?
14. Are t h e r e , m y commer~ts (good a r bad) you w i s h to add r e g a r d i n g
issues '
15. O v e r a l l , how would y o u r a t e the system in terms of c o n s i s t e n c y ?
{please t i c k a p p r o p r i a t e box below)
Ver,, Moderately Neutral Moderately
sati,factorv satisfactory unsatisfactory
User-computer interface
• observing evaluators carrying out tasks and recording
certain aspects of task performance is extremely valuable
in identifying areas of difficulty (and ease) for end-users.
During its development the method was employed in a
number of evaluation settings. In the first of these we used the
method to evaluate a prototype production scheduling system.
Evaluating a prototype scheduling system
This practical study formed part of our work within an
ESPRIT Computer Integrated Manufacturing Project (No 534)
and was concerned with the human factors aspects of flexible
automated assembly. The prototype system in question was
developed to schedule and control the transportation of
pallets carrying components for assembly around a conveyor
network to and from various assembly workstations. The
software ran upon an industry-compatible microcomputer
and was intended to enable the operator of the system to
monitor, load, edit, create and save schedules for the pallet
transportation.
The first stage in evaluating the usability of this user inter-
face was to study the tasks to be undertaken by the (potential)
operator. This basic task analysis provided sufficient inform-
ation with which to create two representative tasks which the
evaluators carried out prior to completion of the checklist.
The 'evatuators' in this case fell into three categories - the
system designer, human factors researchers, and design
engineers unfamiliar with the system.
The procedure for the evaluation was as follows. Each
evatuator was presented with standard instructions detailing
the tasks to be undertaken; then attempted the tasks,
commenting as difficulties were encountered (these were
audio-taped); and completed the usability checklist con-
ferring with the system as necessary.
The evaluation was able to identify certain weaknesses
within the user interface designed for the scheduling system.
Specifically, four areas emerged as requiring attention in the
subsequent design stage. These areas were:
(a) the use of colour, which was excessive and led to some
confusion;
(b) the menu positioning, which was inconsistent through-
out the interface;
(c) the lack of flexibility in the structure of the interface,
making transfer between elements of the system
laborious and irregular; and
(d) the clarity of data entry procedures.
The evaluation method proved to be popular and easily
used, and was shown to be successful in eliciting useful
responses from evaluators with contrasting backgrounds.
In particular, the designer of the system, initially overtly
cynical about the exercise, became very enthusiastic about
the evaluation and the relevance of the results produced. In
the abo~e
particular, he was able to use the findings to look at the next
version of the software with a clear idea of existing problems,
both from a 'user's' perspective and also as a result of focusing
on specific problematic aspects of the interface. During this
evaluation it became clear that the method was performing
Very
unsatisfactory well in its overall assessment of the interface, and that the
people carrying out the evaluation were impressed by the
amount and specificity of the information produced.
Applied Ergonomics December 1989 257
User-computer interface
A number of important areas for improving the method
were revealed by this study. For example, some usability
problems were not adequately tapped by the checklist, the
tasks proved too difficult and lengthy, and limits on the
evaluation were posed because some of the software was not
fully operational.
As a result of this evaluation the checklist needed to
include:
• 'deeper' issues concerning the user's view of the system;
• more detail in the sections of the checklist dealing with
'surface' features (e g, information format, specifically
for graphical and symbolic information); and
• a means of assessing the perceived functionality of the
system, and its appropriateness.
With revisions, in line with the above, made to the coverage
of the checklist we then proceeded to conduct another soft-
ware evaluation exercise, the assessment of alternative user
interfaces to a public-access database.
Comparative evaluation of database interfaces
This evaluation study involved a comparative assessment
of the usability of three prototype interfaces. Each was
developed independently by separate commercial organisa-
tions with the requirement that they be as 'easy to learn
and use' as possible. The intended user population of this
large database system was the general public.
The user interfaces were representative of modern public-
access front-ends, adopting a touch screen or touch-pad input
medium, along with coloured displays using text, icons and
windows. Each system was run on an 'industry-standard'
microcomputer, without the familiar QWERTY keyboard,
each making use of simple, non-voice audio feedback. One
could regard these as similar to those public systems installed
by banks and building societies, the major difference being
the complexity and degree of access to the underlying data-
base. The database of information accessed by these proto-
type interfaces held details of recreational, training and
educational courses. All of the systems were to be used
without any supporting documentatiort (i e, there were no
user guides or manuals).
The users recruited for the evaluation exercise were, as
far as was possible, representative of the general public. Thus,
the sample of over 60 users covered both sexes, a range of
ages (over 16 but under 60), and was drawn from a number
of occupations, including hairdressers, shop assistants,
administrators, redundant mineworkers, students with
learning difficulties, housewives, caretakers, computer
science students, teachers and secretaries. All participants
in the evaluation were unpaid volunteers.
The first part of the evaluation involved the analysis of
tasks carried out using similar (information retrieval)
systems. Having gathered basic information, we devised a
set of tasks to be carried out by 'users' in the evaluation.
These were modified following a pilot study. Specifically,
the degree of task difficulty and the number of tasks to be
undertaken by users were reduced. The pilot study also
permitted the testing of the procedures for recording
observational data. The type of information sought from
user-system interactions was consequently limited to records
258 Applied Ergonomics December 1989
of the time taken to carry out the task, number of times
referring to help facilities, incorrect menu option choices,
and comments made during the task.
Three tasks were developed for the evaluation: general
browsing through the database for information; a general
search (defined by two parameters) with a subsequent
browse of information; and a specific search (again requiring
the user to input particular search constraints), with only
one correct 'answer'. Participants were allocated to cells
within this 3 x 3 (interfaces and tasks) design, each user
attempting one task using one of the three interfaces. No
evaluator was exposed to more than one of the interfaces.
The procedure for each evaluator was as follows. The
evaluator was introduced to the nature of the study, and
shown the interface in question. Evaluators were able to ask
any questions about the evaluation and the task. The evaluator
was informed about the type of system and its purpose, and
presented with a brief questionnaire about his/her back-
ground, experience with computers, etc. The task was carried
out with an observer noting progress. On average, the time to
carry out the task was 25 min. Following this, evaluators
were shown the checklist and asked to complete it for the
interface they had used.
The information gathered in this evaluation was collated
and analysed and found to be invaluable in pinpointing the
strengths and weaknesses of the interfaces under examination.
Furthermore, the utility of the method was demonstrated.
In performing this usability evaluation, we found that
the method was able to discriminate between what were
essentially three 'well-designed' interfaces, and that the
usability checklist was acceptable to a wide range of
evaluators. Furthermore, the results of the evaluation proved
useful to the research sponsors who were seeking to develop
a new interface. The findings were specific enough to allow
them to alter their requirement specification for potential
suppliers. In addition, they were slightly surprised by the
outcomes, since their favoured prototype in practice fared
less well in evaluation than one of the alternatives.
Meeting the requirements - an assessment
In the third section we outlined the requirements of an
evaluation method for interface design. In this section our
method is considered against each of these requirements.
1. ls it systematic? The method requires that evaluators
conduct representative tasks and then work systematic-
ally through a series of questions based on usability
criteria. It can and should be used systematically,
although its use may be constrained by the availability
of users, the time available and other practical consider-
ations. These, of course, are constraints which exist for
any fairly plural and rigorous method.
2. Is it based on existing criteria? Yes, the criteria were
generated from current theoretical and empirical work.
As further research is conducted, new criteria may emerge
and can be included in the format.
3. Is the method iterative? Its use requires an existing proto-
type. It can then be repeated with revised or new versions.
Thus during later stages of design the method can easily

encompass, andindeed it can encourage, iterative working.
In principle there is no reason why the method cannot be
presented to designers working at earlier conceptual
stages as a means of influencing their design ideas, but
it has no formal application at this stage; although the
criteria would remain relevant during conceptual design,
the mode of presentation may need to be significantly
different.
4. Is it general? This issue is unproven. Whilst we have found
the method applicable to several different kinds of inter-
face, it seems likely that different systems may require
the definition and elaboration of new criteria and new
questions. We hope the method is sufficiently flexible to
cope with such system-specific additions and deletions,
and indeed we anticipate that it will be most effectively
used in this way.
5. Is it participative? Thus far we have found the method
does allow for, and indeed encourages, participation. For
example, we have tbund that designers and users have
had little difficulty in using the method. We should add,
however, that the outcomes of the evaluations have
required some input from people with some expertise in
this area. We are' not claiming the method is 'free-standing'.
6. Is it sensitive? Again this is unproven, but in our second
study described above the method seemed well able to
identify specific problems with the three different inter-
faces. It was also able to separate out their relative merits
and demerits, even though each had been specifically
designed to be 'user-friendly'.
7. Is it simple to use? This is more problematic and raises a
number of issues of general relevance to any type of
relatively specialist evaluation. The method is certainly
easy to administer and easy for users to conceptualise.
However, like any other form of evaluation, the inter-
pretation of the results require some expertise.
8. Is it face valid? The users of the method have appreciated
that it relies heavily on the conduct of realistic tasks
which should be chosen both to represent its typical use
and also to explore the functionality of the system. Our
experience is that the method appears face valid.
9. Is it reasonably exhaustive? We would expect that the
method will need editing as it is applied to a wider range
of interfaces. In time we hope that the criteria and check-
list are extended to form a menu from which prospective
users can select relevant evaluation topics.
Our conclusion is that the method lares reasonably well
against the requirements specified earlier in this paper.
Clearly, however, we make no claims that it is a 'finished'
piece of work ready for universal application. In the next
section of this paper we discuss the prospects for the method
and some of the substantive and methodological issues
which arise from our experience with its application.
Discussion
This part of the paper considers some important issues
arising from the development and use of practical methods
of the kind reported in this paper.
User-computer interface
Relative importance of usability issues
One potential criticism of the method is that it fails to
distinguish between major and minor problems. Is poor use
of colour potentially a serious problem? In a comparison
between two competing systems, is it more or less important
than inconsistent menu formatting?
One approach to this difficulty would be to try to weight
the significance of certain criteria, and the particular issues
within them. Our own view, however, is that such an enter-
prise, at least for the forseeable future, is misguided. It
would need to take account of a wide range of variables
such as the nature of the task, the system and the user
population. There would be a real danger that the method
became cumbersome and that more energy was invested in
the abstract weighting of priorities than in debate about user
preferences.
Our view of the method at this stage of development is
that it is a tool for helping users, designers and others
identify the strengths and weaknesses of an interface. They
themselves are in the best position to decide the relative
importance of particular issues, and what action needs to be
taken. The best safeguard that the outcomes of such eval-
uations are useful is not to resort to quasi-mathematical
weighting formulae, but to ensure that end-users are
involved in discussing and interpreting the results.
Quantifying the method
To some people a method of this kind should ideally
result in quantifiable outcomes which allow some com-
parison between interfaces, and perhaps even some notional
scoring of the 'usability' of a particular system (e g, in
percentage terms). For the reasons outlined in the previous
section, we believe this an unrealistic goal. For example, a
single problem concerned with functionality may render a
system virtually unusable to a sub-group of users, even
though it 'scores' highly on all other dimensions. Epistem-
ologically, our position is that different stakeholders
(Mitroff, 1984) interpret and perceive their work and their
IT systems quite differently, and thereby the notion of a
unitary system for 'scoring' usability is misguided.
Relationship to performance metrics
The relationship between self-reported usability as
assessed using this method and objectively measured indices
of system use and performance remains unclear. Ideally,
both forms of evaluation should be included if resources
permit. It remains an important research issue to examine
how and when usability, as assessed in this method, becomes
manifest in performance problems.
Applicability of the method
The two key issues here were discussed briefly earlier.
The first concerns the stages in the design process at which
the method can be applied. Our experience is that it is best
used when operational prototype interfaces are available for
test. We are not especially confident that giving copies of
the checklist to designers who are still conceptualising their
work (i e, they are in the early stages of design) is especially
productive.
Applied Ergonomics December 1989 259
 Uwr-computer interface
The second issue concerns the potential range of appli-
cations or systems. In deliberately attempting to devise a
generalisable method, it is perhaps incumbent upon us to
specify those circumstances where it is, and is not, appropriate.
For example, is it beyond the scope of the method to apply it
to a video-conferencing system which relies heavily on the use
of voice input/output and televisual media? Clearly the
method should now be employed in as many evaluation
settings as possible so as to establish its scope of utility. As we
said earlier, our ambition would be to develop a menu of
criteria and checklists which would cover a wide range of inter.
faces, from which evaluators should select the most relevant.
Concluding remarks
This paper has described the development of a practical
method for evaluating the usability of human-computer
interfaces. The paper has proposed the requirements of such
a method, examined its development with respect to two
case studies, and discussed the method's general suitability.
The method, which employs a typical framework for
user-centred interface evaluation, is based upon a detailed
checklist of criterion-driven questions. The use of the
method depends to a large extent upon those factors that
influence all such evaluation exercises including:
• availability of time within the design and development
cycle;
• availability and willingness of user (and designers) to
participate in the evaluation;
• definition of tasks that are comprehensive and repre-
sentative of the intended use of the application; and
• availability of operational prototypes.
However, the method adopts a different perspective to
many usability evaluation approaches in that it:
• is aimed at being especially usable by evaluators without
experience in the collection and interpretation of
behavioural data;
• requires 'user' involvement and the systematic collection
of their judgements;
• explicitly encompasses both representative tasks and
usability criteria.
One of the most positive attributes of the method is the
degree to which it appears valid to users. Whilst the language
of usability evaluation is often a frustrating hurdle for
system designers and users (e g, cognitive compatibility,
users' conceptual mental models, and the formulae derived
from keystroke data), the checklist offers a straightforward
operational definition of usability and is relatively trans-
parent in how it works. The void that is often apparent
between the production of learning curves, usability plots or
annotated verbal protocols, and their subsequent translation
to recommendations for modification, is markedly reduced.
For example, if the consistency of command/option names
or the informational content of error messages are problems,
then they are quickly identified. Of course, the idea that we
examine the utility of such evaluation techniques from the
perspective of the end-user shoulc( be encouraged with
respect to all 'products' and 'systems' produced by the
human factors community.
260 AppliedErgonomics December 1989
A final more speculative comment should be made. It
seems likely that the role of methods such as this is essenti-
ally to help people to avoid developing interfaces that are
difficult to learn and use. Its function may thus be seen as
rather negative. Whilst it is conceivable that, with regular
use, designers may internalise these criteria and the more
user-centred perspective that lies behind them, and that this
may promote new ideas and approaches, we would not
claim this method encourages innovative and creative
designs. Perhaps other techniques and tools are required
for such purposes?
References
Clegg, C.W., Warr, P.B., Green, T.R.G., Monk, A., Kemp, N,J.,
Allison, G., and Lansdale, M. 1988, 'People and computers
- H o w to evaluate your company's new technology'. Ellis
Horwood, Chichester.
Cohill, A.M., Gilfoil, D.M., Pilitsis, J.V., and Carey, T. 1988,
Measuring the utility of application software. In: Hartson,
H.R. and Hix, D. (Eds), 'Advances in human-computer
interaction, Volume II'. Ablex, Norwood, New Jersey.
Gardner, M.M., and Christie, B. (Eds) 1987, 'Applying
cognitive psychology to user-interface design'. John Wiley
& Sons, Chichester.
Helander, M. (Ed) 1988, 'Handbook of human-computer
interaction'. North-Holland, Amsterdam.
Johnson, G.I., Clegg, C.W., and Ravden, S.J. 1986,
'Human factors in the design of a flexible assembly system:
Design guidelines for design guidelines': In: Oborne, D.J.
(Ed), 'Contemporary Ergonomics 1986'. Taylor and Francis,
London.
Karat, J. 1988, 'Software evaluation methodologies'. In:
Helander, M. (Ed), 'Handbook of human-computer inter-
action'. North-Holland, Amsterdam.
Lea, M. 1988, 'Evaluating user interface designs'. In:
Rubin, T. 'User interface design'. Ellis Horwood, Chichester.
Mitroff, l.I. 1984, 'Stakeholders of the mind'. Jossey-Bass,
San Francisco.
Mosier, J.N., and Smith, S.L. 1986, Behaviour and ln/br
Technol, 5, 39--46. Application of guidelines for designing
user interface software.
Ravden, S.J. 1988, Int J Computer Applications in Technol,
1, 35-42. Ergonomics criteria for design of the software
interface between human and computer in CIM.
Ravden, S.J., and Johnson, G.I. 1989, 'Evaluating usability
of human-computer interfaces: A practical method'. Ellis
Horwood, Chichester.
Rubin, T. 1988, 'User interface design'. Ellis Horwood,
Chichester.
Shneiderman, B. 1987, 'Designing the user interface:
Strategies for effective human-computer interaction'.
Addiscn Wesley, Reading, Mass, USA.
Smith, S.L., and Mosier, J.N. 1984, Behaviour and lnfi~r
Technol, 3(3), 195-203. The user interface to computer-
based systems: A survey of current software design practice.
Smith, S.L., and Mosier, J.N. 1986, 'Guidelines for user-
system interface software'. Technical report ESD-TR-86-
278, MTR 10090. Mitre Corporation, Bedford, Mass.
Sutcliffe, A. 1988, 'Human-computer interface design'.
Macmillan Education, London.