You are on page 1of 3



Ben Shneiderman

Department of Computer Science

University of Maryland
College Park, MD 20742

INTRODUCTION subjects. Computer science or information

systems majors often encounter questions
This paper describes an undergraduate of experimental design and statistical
human factors course which emphasizes analysis for the first time when taking a
psychologically oriented experimentation course on human factors in programming or
on the human use of computers. The user interface design. Since it is
reductionist principles of the scientific unrealistic to teach a complete course in
method are emphasized throughout the experimental methods, I try to convey only
course: lucid statement of testable what they need to comprehend some of the
hypotheses, alteration of independent research literature and to car[v out their
variables, measurement of dependent own experimental projects.-
variables, selection and assignment of
subjects, control for biasing, and This novelty in course content mirrors
statistical testing. Term-length team the new approaches that researchers in
projects are highly motivating for human factors of computer and information
students and have led to worthwhile systems must learn. The traditional
research contributions. computer science approach of
introspection, mathematical analysis, and
The software psychology or human theorem proving is now being enhanced by
factors approach is increasingly important rigorous controlled experimental testing
since the intuitions of experienced and construction of cognitive models of
computer professionals are no longer human behavior. This rich blend provides
adequate to guide designers of software fresh insights and a clearer understanding
standards or interactive systems. of how to design better user interfaces,
Scientific study of human performance in programming languages, software
programming is necessary when software is development methodologies, and database
a central component of life-critical query facilities.
systems such as intensive care units, air
traffic or nuclear reactor control, police Teaching students these experimental
or fire dispatching, and spacecraft approaches by experience has been
guidance. challenging and rewarding. Within a
%ypical 15-week university level course
Further motivation comes from the students may be required to design,
expanded use of computers by administer, evaluate and write up a
non-technically trained people in office complete experiment. I have found this
automation or personal computing laboratory approach extremely productive
applications. Controlled experimentation in that it makes the issues more realistic
can lead to valuable insights about the and is highly motivating for students.
components of user friendliness: ease in Since the students may not be experienced
learning, speed of performance, error in experimental design, precise guidance
rates, retention, and subjective is essential to ensure success.
I encourage students to work in teams
COURSE OVERVIEW of two, ideally composed of a
computer/information science oriented and
The central difference, in my view, a psychology oriented major. Having a
between a human factors and other partner is useful to split the workload
computer/information science courses is and increase the diversity of skills
the emphasis on controlled psychologically within a team. I allow students to choose
oriented experimentation on human their partners and do allow one-person or
Permission to copy without fee all or part of this material is granted provided that
the copies are not made or distributed for direct commerical advantage, the ACM The teams make their experimental
copyright notice and the title of the publication and its date appear, end notice is proposals and I provide feedback for
given that copying is by permission of the Association for Computing Machinery.
To copy otherwise, or to republish, requires a fee and/or specific permissimL revisions. Then the experimental

materials are submitted for review and 2.1 Subjects (i): Describe subjects,
revision. Next a pilot study with 3-15 their background and assignment to
subjects is required to test out the experimental groups.
experimental materials and to gauge the
difficulty of the task for the intended 2.2 Materials (1-2): Describe the
subjects. Students and professional materials so that the knowledgeable
researchers are notoriously bad in judging reader has a clear picture of what
how long it will take subjects to perform they were like.
a task or how difficult they will find it.
A report on the pilot study describing 2.3 Administration (1-2): Describe
what happened and what changes have been the test conditions, time, protection
made is required. of anonymity.
Administering the experiment turns out 3. Results (2-3): Uninterpreted results
to be a great moment in the course. I with tables, graphs, histograms, etc.
encourage students to make contact with This is a simple report of what happened.
instructors of courses which have students
with the right background to be subjects. 4. Discussion (3-6): Explanations,
If necessary I will contact the instructor conjectures, interpretations, and
to answer questions, but generally suggestions for future experiments.
instructors and students are intrigued by Advice to practitioners and researchers.
the novelty of being involved in an
experiment. Instructors should be shown 5. Conclusions (i): Summary and statement
the experimental materials and told how of most important findings.
much time is needed when they are asked to
permit use of their course. We have found 6. References.
that fixed-time experiments work best, but
subjects should be told that they must 7. Appendix: Complete set of experimental
stay till the end of the experiment. We materials.
have our subjects sign an experimental
consent form in which they affirm their I require a first draft of sections 1
voluntary participation and their right to and 2 of the final writeup to be turned in
quit at any time. Student subjects are early for review and evaluation. When the
usually interested in the experiment, but final reports are turned in, students make
can become unhappy if the task is 5-10 minute presentations about their
completely unrelated to their course work, findings. Several student projects have
extremely difficult or extremely led to a published results or have been
time-consuming. Many student-run combined with other work to form a
experiments are completed in 15 or 20 publishable paper. In other situations
minutes but others have taken 45 minutes student projects have become the basis for
or an hour. When the experiment is larger experiments. In any case, the goal
administered the team members should be for the students is merely to gain
present: to answer questions and experience with experimentation.
demonstrate the importance of the research
effort. In about ten percent of our SCHEDULE
experiments something went wrong during
the administration which required a new I've found it important to have
set of subjects and a new administration. numerous milestones to structure the
student's work. Late submission of these
REPORT FORMAT intermediate stage materials is
discouraged but accepted. No grades are
Project teams should turn in their given for these submissions. I've used
experimental data in compact form and the following schedule:
conduct their statistical analyses by hand
or with available statistical program 2nd week - brief d e s c r i p t i o n of topic area
packages. The format for the final report and team member names
is (the figures in parentheses indicated
estimated length in double-spaced 3rd week - statement of the hypothesis,
typewritten pages): independent and dependent variables,
experimental design and background of
i. Introduction (3-6) subjects
State the area of research and why it
is interesting. 5th week - first draft of experimental
Describe relevant previous research. materials for review and identification of
subjects to be used (contact should have
2. Experimental procedures (1-2)
State the hypotheses and briefly been made with the course instructor or
Briefly the manager if professional subjects are
outline the experiment.
describe the pilot study.

7th week - one page description of pilot ACKNOWLEDGEMENTS
experiment and intended revisions
The University of Maryland Computer
9th week - sample of statistical reports Science Center provided support for
from hypothetical data (familiarizes computer services for this work.
students with statistical techniques and
software packages such as SPSS or SAS) REFERENCES

10th week - after actual administration, IEEE Computer, Volume 12, Number 12,
submit raw data in compact form (December 1979), Special Issue on "Human
Factors in Software Engineering", Edited
llth week - first draft of sections 1 and by John Gannon.
2 of final report
Ergonomics, Volume 23, Number 9 (September
12th week - tables, graphs or histograms 1980), Special Issue on "Man-computer
of results communication: Ergonomics and the design
of computer dialogues."
14th week - final report and class
presentations ACM Computing Surveys, Volume 13, Number
i, (March 1981), Special Issues on "The
Directing a large number of student Psychology of Human-Computer Interaction",
experiments can be time-consuming for the Edited by Tom Moran.
instructor, but it is exciting and
rewarding. The students generally have IBM Systems Journal, Volume 20, Numbers 2
very positive comments about this and 3, (1981), Special Issues on "Human
component of the course work. They can Factors".
genuinely be pursuing state of the art
questions and often become intensely Ledgard, H., Singer, Andrew, Whiteside,
involved in their projects. Several John, Directions in Human Factors for
students have pursued their experiments in Interactive Systems, Sprlnger-Verlag,
succeeding terms as independent study Beriin Heidelberg, Germany (1981).
Martin, J., Des~ of Man-Computer
SUMMARY Dialogues, Prentice-Hall, Inc., Englewood
Cliffs, NJ, (1973).
Undergraduate students can be motivated
to learn about software psychology issues Mehlmann, Marilyn, When People Use
quite effectively by requiring a term Computers: A__n_ Approach to Developing an
length team project. Difficulties may Interface, Prentice-Hall, Inc., Englewood
arise and more than average amounts of CliffS, NJ (1981).
instructor effort are required but the
rewards for the students and the Shneiderman, Ben, Software Psychology:
instructor can be great. Human Factors in Computer and Information
Systems, Winthrop Publishers, Cambridge,
It is gratifying that this course has MA, (1980).
been received with tremendous student
interest. For the Spring 1982 semester Smith, H. and Green T.R.G. (editors),
157 students attempted to pre-register for
the 35 seats. Training undergraduates in Human Interaction with Computers, Academic
the software psychology approach can have Press, N.Y. (1980).
a strong impact on the professional
workplace in the coming years, but only if
sufficient numbers of universities
institute similar courses. I look forward
to the expansion of course offerings on
human factors issues and to the inclusion
of these topics in the undergraduate
computer/information science curriculum.

We must begin to train the next

generation of professionals to be more
aware of the importance of human
performance aspects in software
development and interactive systems
design. Simultaneously, we must nurture a
greater sensitivity and desire to serve
the needs of programmers and interactive
systems' users.