HUMAN FACTORS, 1992,34(4),429-439

Comparison of Four Subjective Workload

Ra ting Scales

SUSAN G. HILL,1 EG&G Idaho, Inc., Idaho Falls, Idaho, HELENE P. lAVECCHIA,
Computer Sciences Corporation, Moorestown, New Jersey, JAMES C. BYERS, EG&G Idaho,
Inc., Idaho Falls, Idaho, ALVAH C. BITTNER, Jr., Battelle Human Affairs Research Center,
Seattle, Washington, ALLEN L. ZAKLAD, CHI Systems, Spring House, Pennsylvania, and
RICHARD E. CHRIST, U.S. Army Research Institute, Fort Bliss, Texas

Four subjective workload scales were compared along four dimensions: sensitivity,
operator acceptance, resource requirements, and special procedures. The scales
were the Modified Cooper-Harper scale, the National Aeronautics and Space Ad-
ministration Task Load Index (TLX), the Overall Workload (OW) scale, and the
Subjective Workload Assessment Technique. Three U.S. Army systems were stud-
ied for potential workload concerns. Data from five different studies on the three
systems were compared along the aforementioned four dimensions. Results indi-
cate that all four scales are acceptable tools and are sensitive to different levels of
workload. However, TLX and OW are consistently superior when considering sen-
sitivity, as measured by factor validity, and operator acceptance. This research is
an example of a systematic approach for examining human factors measurement
tools.

INTRODUCTION ing number of techniques for measuring

In recent years several books and reports
have presented workload studies, theoretical
discussions, and reviews of past research
(e.g., Hancock and Meshkati, 1988; Lysaght et
al., 1989; Moray, 1979; O'Donnell and Egge-
meier, 1986). A growing number of different
approaches and techniques for measuring
workload have been discussed, developed,
and used. Subjective techniques, in particu-
lar, have seemed to proliferate. The increas-
1 Requesls for reprints should be sent to Susan G. Hill,
Human Factors Research Unit, Idaho National Engineer-
ing Laboratory, EG&G Idaho, Inc., P.O. Box 1625, Idaho
Falls, ID 83415.
workload creates both an opportunity and a
problem for human factors practitioners and
researchers. On one hand, tools have been de-
veloped for a wide variety of situations; on
the other hand, human factors specialists
faced with choosing the most appropriate of
these tools may find the information neces-
sary to make an informed choice sparse or
nonexistent.
The present systematic study of workload
techniques was undertaken to provide such
information. The Operator Workload Pro-
gram, sponsored by the U.S. Army Research
Institute, was a three-year basic and applied
research effort specifically directed toward

© 1992, The Human Factors Society, Inc. All rights reserved.

Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
430-August 1992
establishing guidance for the assessment of
operator workload associated with the oper-
ation of army systems. To establish such
guidance, comparisons were made between
and among workload techniques. These com-
parisons were made across a number of army
systems and were used to examine the useful-
ness of various operator workload methodol-
ogies, to draw inferences about their relative
value and usefulness in examining workload,
and to assess workload for the army systems.
A major objective of the Operator Work-
load Program was to examine operator work-
load in a broad army context. Previous work
regarding workload has been conducted pri-
marily in relatively narrow aviation environ-
ments frequently set in laboratories (see
Lysaght et aI., 1989). However, much of the
U.S. Army's interest is in ground-based and
nonaviation systems in more operational set-
tings. This has resulted in workload studies
with characteristics different from those of
traditional aviation studies. For example, the
selection criteria for these operators (e.g., ed-
ucation, verbal skills, and general intelli-
gence) were typically less stringent than
those for pilots. Hence one part of the Oper-
ator Workload Program was concerned with
examining and comparing workload mea-
sures for more diverse populations than those
evaluated in more traditional studies.
It was also deemed critical to test the work-
load measures in field settings to yield im-
portant information about the operators
and systems in these settings and about the
usefulness of current workload methodolo-
gies under more operational conditions.
Studies in controlled environments, such
as laboratories and simulators, can yield im-
portant information and may be the only
alternative for nonexistent or unfielded
systems. However, with a programmatic goal
to look at workload in more realistic, less
controlled operational environments, the
present research differs considerably from
Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
HUMAN FACTORS
previous studies in the breadth of evaluation
environments.
FOUR SUBJECTIVE WORKLOAD SCALES
The focus of the Operator Workload Pro-
gram's empirical research was on operator
workload ratings, often referred to as subjec-
tive rating techniques. This may refer to their
apparently weaker objectivity relative to
other workload measurement approaches
(such as physiological measures; see Lysaght
et aI., 1989, for a critical evaluation of empir-
ical and analytical approaches to measuring
workload). It has been argued, however, that
operator ratings are the most direct indica-
tors of operator workload (Sheridan, 1980). In
addition, operator ratings are among the
least intrusive of all techniques because they
can be administered after the task or mission
is completed without disturbing the operator
during task performance. Further, the tech-
niques are flexible and portable; no equip-
ment or special data collection devices are
needed. Finally, these techniques can be
quick and inexpensive to administer and an-
alyze. These substantial advantages recom-
mended their application in most operator
workload investigations and as a focus of the
Operator Workload Program.
The Operator Workload Program's compre-
hensive review of workload methodologies
stated a number of conclusions concerning
operator rating methods (Lysaght et al.,
1989). Three of the most relevant here are as
follows:
• Operator rating methods are sensitive to key
mission variables and can provide valuable
low-cost information.
• There are both unidimensional and multidi-
mensional techniques, 'with different advan-
tages and disadvantages.
• Three particular techniques-the National
Aeronautics and Space Administration (NASA)
Task Load Index (TLX), the Subjective Work-
load Assessment Technique (SWAT),and the
Modified Cooper-Harper (MCH) scale-were
found to have substantial validated sensitivity.
COMPARISON OF WORKLOAD SCALES
Lysaght et al. (1989) found a number of pub-
lished studies that used subjective rating
techniques. Comparisons have been made be-
tween two techniques (e.g., Vidulich and
Tsang, 1985, compared SWAT and NASA Bi-
polar scales; Warr, Colle, and Reid, 1986,
compared MCH and SWAT). However, these
studies did not statistically compare the effi-
cacy of measures, and no more than two
methods were compared at anyone time.
The review also identified four measures
that were most researched or which had the
greatest promise for field application: MCH
(Wierwille and Casali, 1983), NASA TLX
(Hart and Staveland, 1988), the Overall
Workload (OW) scale (Vidulich and Tsang,
1987), and SWAT (Reid, Shingledecker, and
Eggemeier, 1981). Three of these techniques
(TLX, SWAT, and MCH) were selected be-
cause of the extent of previous validation; the
other scale (OW) was chosen primarily be-
cause of its simplicity and suspected potential.
Modified Cooper-Harper Scale
The Modified Cooper-Harper scale is a 10-
point unidimensional rating scale that results
in a global rating of workload. The rating
scale uses a decision tree to assist the rater in
determining the most appropriate rating to
assign. The MCH was developed for workload
assessment in systems in which the task is
primarily cognitive, rather than motor or
psychomotor, and in which the original
Cooper-Harper scale (Cooper and Harper,
1969) may not be appropriate.
NASA Task Load Index
The NASA Task Load Index uses six dimen-
sions to assess workload: mental demand,
physical demand, temporal demand, perfor-
mance, effort, and frustration. Twenty-step
bipolar scales are used to obtain ratings for
these dimensions. A score from 0 to 100 (as-
Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
August 1992-431
signed to the nearest 5) is obtained on each
scale. A weighting procedure is used to com-
bine the six individual scale ratings into a
global score; this procedure requires a paired
comparison task to be performed prior to the
workload assessments. Paired comparisons
require the operator to choose which dimen-
sion is more relevant to workload for a par-
ticular task across all pairs of the six dimen-
sions. The number of times a dimension is
chosen as more relevant is the weighting of
that dimension scale for a given task for that
operator. A workload score from 0 to 100 is
obtained for each rated task by multiplying
the weight by the individual dimension scale
score, summing across scales, and dividing
by the total weights (i.e., 15 for the 15 paired
comparisons).
Overall Workload Scale
Overall Workload is a rating of the sub-
ject's overall workload on a unidimension-
al scale of 0 to 100, with 0 representing very
low workload and 100 representing very
high workload. A single, 20-step bipolar scale
is used to obtain this global rating. A score
from 0 to 100 (assigned to the nearest 5) is
obtained.
Subjective Workload Assessment Technique
The Subjective Workload Assessment Tech-
nique is a subjective rating technique de-
veloped by the U.S. Air Force Armstrong
Aerospace Medical Research Laboratory at
Wright-Patterson Air Force Base. SWAT uses
three levels-low (1), medium (2), and high
(3}-for each of the three dimensions of time
load, mental effort load, and psychological
stress load to assess workload. It uses con-
joint measurement and scaling techniques to
develop a single, global rating scale with in-
terval properties.
The use of SWAT entails three distinct
steps. The first is called scale development.
All possible combinations of the three levels
432-August 1992 HUMAN FACTORS

of each of the three dimensions are contained further development. A Force Development
in 27 cards. Each operator sorts the cards into Test and Evaluation (FDTE) is a field test on
the rank order that reflects his or her percep- a system already selected for development.
tion of increasing workload. Conjoint scaling Both tests are operationally oriented and use
procedures are used to develop a scale with trained operators in field environments.
interval properties. The second step is the Aquila FDTE
event-scoring-that is, the actual rating of
Workload assessments were conducted of
workload for a given task or mission segment.
four crews of operators of the Aquila Ground
In the third step, each three-dimension rating
Control Station during an FDTE in the last
is converted into numeric scores between 0
quarter of 1987. Four operator workload rat-
and 100 using the interval scale developed in
ing scales were used: MCH, OW, SWAT, and
the first step.
TLX. The objectives of this investigation were
DATA COLLECTION EFFORTS to (1) explore the applicabili.ty of the work-
load scales under the conditions characteriz-
Three U.S. Army systems, two ground-
ing field evaluations (i.e., Army FDTEs) and
based and one aviation, were selected for
to evaluate ground operation workload vari-
study of potential workload concerns (Bittner
ations during the Aquila Remotely Piloted
et aI., 1988): the Aquila Remotely Piloted Ve-
Vehicle FDTE. Subsequent to completion of
hicle; the Line-of-Sight, Forward, Heavy
the FDTE, a rating scales questionnaire was
(LOS-F-H) mobile air defense system; and the
administered which solicited judgments re-
UH-60A Blackhawk helicopter.
garding the procedures and test instruments,
A plan for investigating workload tech-
particularly those used to assess operator
niques was developed in which several spe-
workload.
cific studies were to be conducted for each
Two broad conclusions were drawn from
system (Bittner et aI., 1988). Each of the stud-
the Aquila FDTE evaluation of the use of the
ies had specific objectives in examining oper-
operator workload scales under field test con-
ator workload methodologies. At the same
ditions. First, operator workload measures
time, effort was made to conduct tests in a
may be successfully applied and evaluated
similar manner so that comparison across all
within the stringent field test environment
data collection efforts could be made. Each of
characterizing a U.S. Army FDTE. Second,
the studies is briefly described in the follow-
TLX had both the highest validity and best
ing sections. The studies are summarized by
user acceptance within the limited subject
subjects, objectives, method, and results in
group and conditions of the present investi-
Table 1. The five studies consisted of one of
gation. This study is documented in Byers,
the Aquila, three of the LOS-F-H, and one of
Bittner, Hill, Zaklad, and Christ (1988) and in
the Blackhawk helicopter.
Byers, Hill, Zaklad, and Christ (1989).
Most of the workload studies were part
of previously scheduled U.S. Army test LOS-F-R NDICE
and evaluation efforts. One such type of test Retrospective workload assessments were
and evaluation field exercise was a Non- conducted of six operators of the selected
Developmental Item Candidate Evaluation LOS-F-H candidate system 10 weeks after the
(NDICE), which is an evaluation of two or candidate field evaluation, which was held
more already developed systems that enables during late 1987. Four operator workload
selection of the better performing system for rating scales were used: MCH, OW, SWAT,

Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
COMPARISON OF WORKLOAD SCALES
and TLX. The objectives of this investigation
were (1) to explore the applicability of the
operator workload scales for obtaining retro-
spective workload assessments several weeks
after a system field evaluation (i.e., Army
NDICE); (2) to evaluate the relationship be-
tween system performance and the retrospec-
tive workload assessments of the crew mem-
bers of the selected system; and (3) to
compare the results of the present program-
matic investigation with those from the Aq-
uila study (Byers et aI., 1988). Data on total
system performance for each mission were
provided to the authors by independent eval-
uators.
Substantial and significant multiple corre-
lations between system performance and
workload (as measured by TLX) were found.
In particular, with R = 0.66, the resulting
model relating system performance to TLX
indicated generally decreasing performance
with increases in workload. In addition, ret-
rospective application of operator workload
measures proved successful. This study is
documented in Hill, Zaklad, Bittner, Byers,
and Christ (1988).
LOS-F-H Generic
Generic workload ratings were obtained
from five operators and nine subject matter
experts of the LOS-F-H system during two
separate data collection sessions. Four oper-
ator workload rating scales were used: MCH,
OW, SWAT, and TLX. The objectives of this
investigation were to explore the applicabil-
ity of the workload scales for obtaining ge-
neric estimates from operators and subject
matter experts and to explore operator work-
load in the baseline LOS-F-H, which was
evaluated during the NDICE.
Two broad conclusions were drawn from
this evaluation of the use of operator work-
load scales. First, generic ratings may be used
to assess mission conditions and task seg-
Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
August 1992-433
ments while minimizing differences caused
by specific mission idiosyncrasies. Second, no
systematic differences were found between
generic operator workload ratings made by
subject matter experts and ratings made by
crew members who had operated the system.
This study is documented in Bittner, Byers,
Hill, Zaklad, and Christ (1989).
LOS-F-H FDTE Basic
Workload assessment was conducted of
seven operators during the LOS-F-H FDTE,
which took place at Fort Bliss, Texas, from
May through July 1988. Four workload rating
scales were used: MCH, OW, SWAT, and TLX.
The objectives of this investigation were to
explore the applicability of the operator
workload scales under the conditions char-
acterizing field test evaluations and to eval-
uate operator workload during LOS-F-H
operations.
The results of the subjective ratings of op-
erator workload in the LOS-F-H FDTE indi-
cated the following. Overall workload ratings
were significantly different for crew position.
There were some significant effects of mission
variables on workload. There were differ-
ences in both magnitude and dimensions of
workload among mission segments. This
study is documented in Hill, Byers, Zaklad,
and Christ (1989).
UH-60A Blackhawk Helicopter
Operator workload data were collected
from 10 two-man UH-60A crews during sim-
ulated one-hour resupply missions. Four op-
erator workload rating scales were used:
MCH, OW, SWAT, and TLX. In addition, a
tool was developed to assess peak workload
(the PW scale). The objectives of this investi-
gation were (1) to determine the relationship
between an analytical model's (the Task
Analysis/Workload, or TAWL) prediction of
 co
.::
'"
<:;
:a-
'"
o

~
o ~ III

~
o
to S"'Ul
Qi ·W
'"
C-
c- ~:::!:
O
o Oen
.•..
I'-
<0 It)0l

co
CO
Ql
<J
.•..
Ol

c:
e
.l!!
Ql
ex:

W
l- I
e
u.. u.
.!!!
.-0'
en
':;
Qio
C:...J
C7 Q)~
« CJ

-434--
Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
 c
Ul
o
o
iV
Ul
m
ma. 6~
!~
e
,... 0
0

0-
.- J:
l3LL
In'(J)
We
t-...J
0-
u..

-435-
Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
436-August 1992
workload (Bierbaum, Szabo, and Aldrich,
1987) and the actual workload reported by
the pilot and copilot while flying a simulated
mission; (2) to investigate differences be-
tween workload reported during the mission
and workload recalled following the mission;
(3) to examine operator acceptance of the var-
ious operator workload assessment tech-
niques; and (4) to evaluate the effects of key
mission variables on crew workload as well
as the relationship between performance and
workload.
Four primary conclusions concerning work-
load were drawn from this investigation.
First, the TAWL model has shown an ability
to predict empirical data (r = 0.82 to 0.95, P
< 0.01) and has substantial potential as an
analytical workload estimation technique
that may be applied to predict workload be-
fore system development. Second, empirical
workload assessment techniques can be
readily applied in an army aviation setting
with favorable operator acceptance, with
TLX and OW being the most preferred scales.
Third, workload ratings have shown reason-
able variation with key mission conditions.
Fourth, the PW measure may be a useful ad-
dition to the repertoire of subjective work-
load assessment techniques but requires fur-
ther study and validation. This study is
documented in lavecchia, Linton, Bittner,
and Byers (1989) and lavecchia, Linton, Har-
ris, Zaklad, and Byers (1989).
COMPARISON DIMENSIONS
The four subjective scales used in all stud-
ies-TLX, OW, MCH, and SWAT-were com-
pared with one another along four dimen-
sions: sensitivity (as measured by factor
validity), operator acceptance, resource re-
quirements, and special procedures.
Sensitivity
To examine how each of the four scales was
able to discriminate among different levels of
Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
HUMAN FACTORS
task loading, factor analysis was performed
on the aggregate data. For all studies, a sin-
gle factor emerged which was named the
operator workload (actor. The correlation of
each scale with the operator workload factor
was calculated and is the factor validity for
that scale for that study. Sensitivity, as
measured by the factor validity, is a descrip-
tion of the degree to which the scales measure
workload.
The factor validity analysis for a given
study was conducted in two stages. During
the first stage, a principal component analy-
sis was conducted on the sets of segment rat-
ings collected across subjects and missions
using BMDP4M (Dixon, 1983). Each set in-
cluded overall workload ratings using four
scales: TLX, SWAT, OW, and MCH. In all five
data sets, this analysis revealed a single com-
ponent, the operator workload factor. The re-
sults of this initial analysis supported the
view that the four workload scales essentially
provide assessments of a single common
factor.
Jackknife principal component analyses
were conducted of the workload measures
during the second stage in order to evaluate
the stability of the factor loadings of the four
scales (i.e., correlations with the operator
workload factor). Jackknife analysis, it
should be noted, generally involves succes-
sive analyses (in the present case, principal
component analyses), dropping subjects
one at a time from a data set in order to al-
low analysis of the stability of parameter
estimates (Hinkley, 1983). For example, in
the LOS-F-H NDICE analysis with four fac-
tor loadings and six subjects, a 4 (loadings)
x 6 (subject dropped) matrix was produced
that could be analyzed by conventional
repeated-measures analysis of variance
(ANOVA). ANOVA using BMDP2V (Dixon,
1983) was used to examine significant dif-
ferences among the workload scale factor
loadings.
COMPARISON OF WORKLOAD SCALES August 1992--437

Operator Acceptance RESULTS

Another source of comparative information Sensitivity

on the four rating scales was operators' reac-
tions to the scales. This dimension is of in-
terest because the increased operator ac-
ceptance of a subjective measurement tool
may result in increased willingness to express
a valid opinion that can be taken seriously
and used.
Resource Requirements
TLX has the highest factor validity (i.e., the
greatest correlation with the operator work-
load factor) for each of the five studies, as
seen in Table 2. Subsequent analysis revealed
the orderings of the mean factor loadings for
the other scales, also shown in this table. The
simplest scale, OW, had the second highest
average factor validity after TLX.

For practical purposes, it is important to Operator Acceptance

know the relative resource requirements of
the scales (i.e., how much it costs to use). The
differences among the scales are reflected in
the required time for training, preparation,
completion, data reduction, and analysis.
The data show that TLX was liked best, OW
was the easiest to complete, and TLX was
rated the best in its ability to represent work-
load (Table 3).

Special Procedures Resource Requirements

Requirements of the multidimensional OW took the least time to complete, TLX

scales-TLX and SWAT-include special pro-
cedures, which consume additional time.
Both the TLX and SWAT require sorts, or
judgments, that are designed to uncover the
relative salience of the scales' component di-
mensions to each individual (independent of
the workload ratings themselves).
took the most, and MCH and SWAT were in-
termediate (Table 4). Table 4 can be readily
understood if one looks at the number of
judgments made for each scale: six for TLX,
three for SWAT, and one each for MCH and
OW. Although other times were not system-
atically measured, our experience suggests

TABLE 2

Workload Scale Factor Validities

Study % Total Variance Factor Validities

LOS-F-H 80 TLX (0.935) > OW (0.927) > MCH (0.862) > SWAT (0.860)
NDICE
LOS-F-H 80 TLX (0.924) > OW (0.905) > MCH (0.904) > SWAT (0.778)
generic
LOS-F-H 79 TLX (0.942) > SWAT (0.900) > OW (0.898) > MCH (0.818)
FDTE basic
Aquila FDTE 75 TLX (0.910) > SWAT (0.893) > OW (0.869) > MCH (0.833)
UH-60A 71 TLX (0.899) > OW (0.872) > SWAT (0.805) > MCH (0.799)

Note: Breaks In the underline Indicate statistically significant differences batween factor validities (a = 0.05).
FDTE = Force Development Test and Evaluation; lOS-F-H = line-of-Sight. Forward, Heavy; MCH = Modified Cooper-Harper scale; NDICE
= Non·Developmentalltem Candidate Evaluation; OW = Overall Workload scale; SWAT = Subjective Workload Assessment Technique; TLX
= NASA Task load Index; UH-60A = Blackhawk helicopter.

Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
438-August 1992 HUMAN FACTORS

TABLE3 1987). Our observations suggest that the

problems may be more pronounced for less
Operator Acceptance Ratings
verbal and sophisticated operators.
Rating Scale Attribute TLX OW MCH SWAT
DISCUSSION
Liked best 1.5* 2 3.5 3
Easiest to complete 2 1 4 3 Each of the four subjective workload tech-
Best decription of workload 1 2 3.5 3.5 niques has unique characteristics. All mea-
• Average of mean rankings (1 = best •... 4 = worst) of 37 oper- sure workload, as shown by the moderate to
ators.
TlX = NASA Task load Index: OW = Overall Workload scale: high positive correlations with the operator
MCH = Modified Cooper-Harper scale; SWAT = Subjective Work- workload factor (i.e., the factor validities).
load Assessment Technique.
Two are unidimensional workload scales,
which take less time to complete and less
that OW requires substantially less time for
data reduction time (OW and MCH). Two oth-
training, preparation, and data reduction
ers are multidimensional scales, which tradi-
than do the other scales. Both TLX and SWAT
tionally entail special sorting procedures that
require more time for data reduction and
take more time but which are justified by the
analysis because of the multidimensional na-
additional information they provide (TLX
ture of the scales.
and SWAT). Certainly the multidimensional
Special Procedures scales information may also be used diagnos-
tically to address the question of what is
The additional information gained from causing the workload reported, and it may
the multidimensional representation of work- point toward ways to alleviate excessive
load may justify the cost of the additional workload.
sorting procedures. However, in the case of All four scales are moderately to highly ac-
TLX, our research suggests that its paired- ceptable tools with respect to sensitivity to
comparison sort procedure may be skipped different levels of workload. However, TLX
without compromising the measure (Byers, and OW are consistently superior in terms of
Bittner, and Hill, 1989). SWAT sorts pre- sensitivity and have the strongest operator
sented problems to some of the subjects; 23 of acceptance. With regard to resources and spe-
54 subjects did not initially produce adequate cial procedures, the needs of the study must
SWAT card sorts. That results in a 43% fail- be examined to determine which tool may be
ure rate on the first attempt to perform the most appropriate for the particular circum-
SWAT sort as suggested by the criteria pre- stance. The unidimensional scales, particu-
sented in the SWAT user's guide (Armstrong larly OW, may be useful as a screening tool to
Aerospace Medical Research Laboratory, identify potential chokepoints of workload;
the multidimensional tools, particularly
TABLE4 TLX, may be used to obtain more detailed
and diagnostic data.
Time to Complete Each Rated Workload Situation
(in Seconds) This research is an example of a systematic
approach to examining four subjective work-
TLX OW MCH SWAT
load measurement tools. The comparative
Number of cases 38 33 27 27 approach, attempting to draw inferences
Mean time to complete 51.3 9.8 29.1 33.6 concerning workload techniques from sev-
Standard deviation 29.5 8.4 26.3 24.6
eral studies, has proved to be useful. This
TLX = NASA Task lOad Index; OW = Overall Workload scale; approach has provided information with
MCH = Modified Cooper-Harper scale; SWAT = Subjective Work-
load Assessment Technique. which
Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN to make
DIEGO informed
on January 24, 2016 decisions about se-
COMPARISON OF WORKLOAD SCALES
lection of workload assessment techniques.
Such a systematic approach should be used
to assess many human factors measurement
tools.
ACKNOWLEDGMENTS
This work was supported by the u.s. Army Research
Institute under Contract No. MDA903-86-C-0384 to Analyt-
ics, Inc. The views, opinions, and findings contained in this
paper are those of the authors and should not be construed
as an Official Department of the Army position, policy, or
decision. During the majority of the research program, the
first five authors were affiliated with Analytics, Inc.,
Willow Grove, Pennsylvania. The authors acknowledge
the contribution of other members of the team to the suc-
cess of the Operator Workload Program: John P. Bulger,
Regina M. Harris, Paul M. Linton, Robert J. Lysaght, and
Michelle Sams.
REFERENCES
Armstrong Aerospace Medical Research Laboratory.
(1987). Subjective workload assessment technique
(SWAT): A user's guide. Wright-Patterson Air Force
Base, Author.
Bierbaum, C., Szabo, S., and Aldrich, T. (1987). A compre-
hensive task analysis of the UH-60 mission with crew
workload estimates and preliminary decision rules for de-
veloping a UH-60 workload prediction model (Tech. Re-
port MDA903-87-C-0523). Ft. Rucker, AL: Anaeapa Sci-
ences.
Bittner, A. C., Jr., Byers, J. C., Hill, S. G., Zaklad, A. L., and
Christ, R. E. (1989). Generic workload ratings of a mo-
bile air defense system (LOS-F-H). In Proceedings of the
Human Factors Society 33rd Annual Meeting (pp. l47b-
1480). Santa Monica, CA: Human Factors Society.
Bittner, A. C., Jr., Zaklad, A. L., Dick, A. 0., Wherry, R. J.,
Jr., Herman, E. D., Bulger, J. P., Linton, P. M., Lysaght,
R.J., and Dennison, T. W. (1988). Operator workload
(OWL) assessment program for the army: Validation and
analysis plans for three systems (ATHS, Aquila, WS-F-
H) (Tech. Report 2075-3b). Willow Grove, PA: Analyt-
ics, Inc.
Byers, J. C., Bittner, A. C., Jr., and Hill, S. G. (1989). Tra-
ditional and raw Task Load Index (TLX) correlations:
Are paired comparisons necessary? In A. Mital (Ed.),
Advances in industrial ergonomics and safety I (pp. 481-
485). London: Taylor & Francis.
Byers, J. C., Bittner, A. C., Jr., Hill, S. G., Zaklad, A. L., and
Christ, R. E. (1988). Workload assessment of a re-
motely piloted vehicle (RPV) system. In Proceedings of
the Human Factors Society 32nd Annual Meeting (pp.
1145-1149). Santa Monica, CA: Human Factors Soci-
ety.
Byers, J. C., Hill, S. G., Zaklad, A. L., and Christ, R. E.
(1989). Aquila system report (Tech. Report 2075-4A).
Willow Grove, PA: Analytics, Inc.
Cooper, G. E., and Harper, R. P. (1969). The use of pilot
ratings in the evaluation of aircraft handling qualities
(NASA TN-D-5153). Moffett Field, CA: NASA Ames Re-
search Center.
Dixon, W. J. (Ed.). (1983). BMDP statistical software. Los
Angeles: University of California Press.
Hancock, P. A., and Meshkati, N. (Eds.). (1988). Human
mental workload. Amsterdam: North-Holland.
Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016
August 1992-439
Hart, S. G., and Staveland, L. E. (1988). Development of a
NASA-TLX (Task Load Index): Results of empirical
and theoretical research. In P. A. Hancock and N.
Meshkati (Eds.), Human mental workload (pp. 139-
183). Amsterdam: North-Holland.
Hill, S. G., Byers, J. C., Zaklad, A. L., and Christ, R. E.
(1989). Subjective workload ratings of the LOS-F-H mo-
bile air defense missile system in a field test environment
(Tech. Memo 5). willow Grove, PA: Analytics, Inc.
Hill, S. G., Zaklad, A. L., Bittner, A. C., Jr., Byers, J. C., and
Christ, R. E. (1988). Workload assessment of a mobile
air defense missile system. In Proceedings of the Human
Factors Society 32nd Annual Meeting (pp. 1068-1072).
Santa Monica, CA: Human Factors Society.
Hinkley, D. V. (1983). Jackknife methods. In S. Konz, N. L.
Johnson, and C. B. Read (Eds.), Encyclopedia of statis-
tical sciences (Vol. 4, pp. 280-287). New York: Wiley.
lavecchia, H. P., Linton, P. M., Bittner, A. C., Jr., and By-
ers, J. C. (1989). Operator workload in the UH-60
Blackhawk: Crew results vs. TAWL model predictions.
In Proceedings of the Human Factors Society 33rd An-
nual Meeting (pp. 1481-1485). Santa Monica, CA: Hu-
man Factors Society.
lavecchia, H. P., Linton, P. M., Harris, R. M., Zaklad, A. L.,
and Byers, J. C. (1989). UH-60 system report (Tech. Re-
port 2075-4C). Willow Grove, PA: Analytics, Inc.
Lysaght, R. J., Hill, S. G., Dick, A.O., Plamondon, B. D.,
Wherry, R. 1., Jr., Zaklad, A. L., and Bittner, A. C., Jr.
(1989). Operator workload: Comprehensive review and
evaluation of operator workload methodologies (ARI
Tech. Report 851). Alexandria, VA: U.S. Army Re-
search Institute for the Behavioral and Social Sciences.
Moray, N. (Ed.). (1979). Mental workload: Its theory and
measurement. New York: Plenum.
O'Donnell, R. D., and Eggemeier, F. T. (1986). Workload
assessment methodology. In K. R. Boff, L. Kaufman,
and J. P. Thomas (Eds.), Handbook of perception and
human performance (pp. 42-1-42-49). New York: Wiley.
Reid, G. B., Shingledecker, C. A., and Eggemeier. F. T.
(1981). Application of conjoint measurement to work-
load scale development. In Proceedings of the Human
Factors Society 25th Annual Meeting (pp. 522-525).
Santa Monica, CA: Human Factors Society.
Sheridan, T. (1980). Mental workload-What is it? Why
bother with it? Human Factors Society Bulletin, 23(2),
1-2.
Vidulich, M. A., and Tsang, P. S. (1985). Assessing subjec-
tive workload assessment: A comparison of SWAT and
the NASA-Bipolar methods. In Proceedings of the Hu-
man Factors Society 29th Annual Meeting (pp. 71-75).
Santa Monica, CA: Human Factors Society.
Vidulich, M. A., and Tsang, P. S. (1987). Absolute magni-
tude estimation and relative judgment approaches to
subjective workload assessment. In Proceedings of the
Human Factors Society 31 st Annual Meeting (pp. 1057-
1061). Santa Monica, CA: Human Factors Society.
Warr, D., Colle, H., and Reid, G. (1986). A comparative eval-
uation of two subjective workload measures: The Subjec-
tive Workload Assessment Technique and the Modified
Cooper-Harper Scale. Paper presented at the Sympo-
sium on Psychology in the Department of Defense. U.S.
Air Force Academy, Colorado Springs, CO.
Wierwille, W. W., and Casali, J. G. (1983). A validated
rating scale for global mental workload measurement
application. In Proceedings of the Human Factors Soci-
ety 27th Annual Meeting (pp. 129-133). Santa Monica,
CA: Human Factors Society.
Downloaded from hfs.sagepub.com at UNIV CALIFORNIA SAN DIEGO on January 24, 2016