Educational and Psychological

Measurement
http://epm.sagepub.com/

Response Sets and Test Validity

Lee J. Cronbach
Educational and Psychological Measurement 1946 6: 475
DOI: 10.1177/001316444600600405

The online version of this article can be found at:

http://epm.sagepub.com/content/6/4/475

Published by:

http://www.sagepublications.com

Additional services and information for Educational and Psychological Measurement can
be found at:

Email Alerts: http://epm.sagepub.com/cgi/alerts

Subscriptions: http://epm.sagepub.com/subscriptions

Reprints: http://www.sagepub.com/journalsReprints.nav

Permissions: http://www.sagepub.com/journalsPermissions.nav

Citations: http://epm.sagepub.com/content/6/4/475.refs.html

>> Version of Record - Dec 1, 1946

What is This?

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE SETS AND TEST VALIDITY
LEE J. CRONBACH
University of Chicago

A PSYCHOLOGICAL or educational test is constructed, by

choosing items of the desired content, and refining them by
empirical techniques. The assumption is generally made, and
validated as well as possible, ,that what the test measures is
determined by the content oflthe items. Yet the final score
of the person on any test is a composite of effects resulting
from the content of the item and effects resulting from the form
of item used. A test supposedly measuring one variable may
also be measuring another trait which would not influence the
score if another type of item were used. This paper attempts
to demonstrate these influences in a variety of tests, to examine
the effect of these extraneous factors, and to suggest means of
controlling them.
Numerous studies show that scores may be influenced by
variables other than the one sppposedly tested. I n the Min-
nerotn Mziltiphasic Personality Inveatory, for example, it is
expIicitly recognized that a spbject may evade questions by
the excessive use of the response “Cannot Say,” even though
his actual behavior might be properly described by the response
“True” or “False.” This tendency invalidates the test profile
for persons who show an extreme number of “Cannot Say”
responses. Another example is the influence of acquiescence on
true-false test performance (2, 4). Under many conditions,
when separate scores are obtained on the true items and on the
false items of the’typical achievement test, the correlation of
the two scores is near zero even when the test as a whole is re-
liable. I n other words, items in these two forms, the true and
the false, do not measure the same trait. This is attributed to
the tendency of some students to respond “True” when in
4i5

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

476 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

doubt, so that their score on true statements is high, but on

false statements is low. Apparently, the score of a person is
influenced by his set to react t o the items in a certain way,
apart from their content.
Tendencies such as those just described are characterized in
this paper as “response sets.” A response set is defined as any
tendency causing a person consistently t o give different re-
sponses t o test items t h a n he would when the same content is
presented in a different form. This is a theoretical rather than
a practical definition, since it is never possible wholly t o separate
the content of an item from its form. Yet many acquiescent
students who fail a false item would pass the item had it been
presented as a true statement. Subjects who invalidate the
Multiphasic interpretation by the evasive use of “Cannot Say”
responses, could not have done so’if allowed only the “True” or
(6
False” options. I n this definition, “form” includes the form of
statement, the choice of responses offered, and the directions,
since all of these are part of the situation t o which the person
reacts.
A Listing of Response Setr
1. Tendency t o Gamble; Caution uersus Incaution.-If stu-
dents are allowed t o omit test items, some omit more items than
others who .have equal knowledge. It has been established
that there are individual differences in this tendency, that these
differences are reliable within a particular test, and that the
differences are reliable from one test t o another of similar type
(9, 10, 25,26,29, 30). The iendency to “gamble,” t o respond
when doubtful, appears t o be distributed over a continuum,
from the student who answers only when very sure t o the one
who attempts every item. On objective items, the student
usually has better-than-chance probability of choosing the
correct answer, because of his partial knowledge; therefore, the
more he guesses, the higher his score will tend t o be, even
though a correction for chance is used. When knowledge is
held constant, caution will be negatively correlated with score.
Gambling, by increasing the spread of individual differences,
increases reliability in many cases, but by attenuating the re-
lation between knowledge and the criterion, tends to reduce

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE: SETS 477
validity (30). Correction formulas may be made more severe
to penaIize gambIing, but they cannot eliminate the effects of
the variable from individual scores (5).
Similar t o this tendency t o avoid commitment is a tendency
to use the neutral response in tests of attitudes, personality, and
psychophysical judgments. Some students more than others
use the middle response in the “Yes-?-No” option in the Bell
Adjustment Inventory, in the “I,-I-D”. option in the Strong
Vocational Interest Blank, or in the “Agree-Uncertain-Dis-
agree” choice on Likert-type attitude scales. Individual dif-
ferences in the use of the “equality” response in psychophysical
judgments are reported by Fernberger (8) and Woodworth (31,
p, 422).
Evidence of individual differences in caution is provided by
the author’s “Test on the Effe%s of War” ( 3 ) . The student
judged, on a five-point scale, the likelihood of fifty predicted
effects of war. Responses were scored in terms of optimism-
pessimism. Responses of 100 typical Washington high-school
students were analyzed. T h e number of E (equally likely t o
happen or not t o happen) responses ranged from zero t o 22,
with a mean of 7.6 and a sigma of 4.5. A split-half reliability
coefficient for the E score is 0.73 (corrected). This is nearly
as high as the reliability of the Optimism score, weighting all
five choices and considering the content of the items. Use of
E responses by an individual reduces his possible range of
Optimism scores. If this response had not been allowed, stu-
dents who used many E’s probably would have received dif-
ferent Optimism scores.
Another example of this response set came t o light in a mi!i-
tary training situation. Men were required to judge a stimulus
as “too red,” “too yellow,” or “no difference from standard.”’
In actual samples from different dye lots there is probably some
difference between samples, but in practice, samples within a
certain tolerance of the standard are reported as “No dif-
ference.” I n a test to grade men after training, samples were
judged on the three-choice pattern. Some persons reported a
difference nearly always, while others reported no difference
* T h e judgment described is not the actual one. The alteration in detail does
not affect the point under discussion.

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

478 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

unless they were very sure of the direction of the deviation.

It had a t first been thought that men who failed had poor
discrimination ability, but it became apparent that the test
measured both this ability, reflecting the content of the item,
and set t o ignore small differences, introduced by the form of
the judg:nent. I n fact, the training problem was-to teach the
response set, to produce uniform reports, as much as it was t o
train judgment.
2. Definition of Jzcdgment Categories.-Different persons
assign different meanings to the terms used in responding:
“Yes,” “Strongly agree,” etc. This problem overlaps t h a t of
caution, just described. I n P.E.A. Test 8.2, “Interest Index”
(24, p. 338), which calls for an “GI-D” response, two students
enjoying a particular area equally may have different “Per cent
Like” scores, because one defines “Like” as any degree of ac-
ceptance rather than rejection \Ghile the second reserves “Like”
for those things he has a real desire t o do. Simpson (23) and
Mosier (17) have shown individual differences in using words
such as “frequently,” “indifferent,” and “desirable.” Mosier
found these differences to be reliable. Osgood reports that on a
seven-point scale some persons predominantly use position ‘1
and 7, some use 1,4 and 7, while some use the whole scale ( IS).
Using intermediate rather than more extreme scale positions,
is used as a reliable index of behavior, (“caution in drawing
conclusions”) in P.E.A. Test 2.51, “Interpretation of Data”
(24, pp. 62-65). The various writers have attributed these
differences as possibly due tcJ true personality differences in
caution or conservatism, intellectual differences such as critical
thinking, or t o differences in word meaning.
3. IncZusiveness.-In some tests, the student is permitted
to give as many answers as he desires, as in such essay questions
as “Point out differences between” or “List the causes of,” etc.
An open question of this type may elicit an extensive listing of
points from one student, and a short selected list from another.
Which receives the better score depends on the scoring method,
but the score may reflect technique or set in answering as well
as ability. The same possibility occurs in some objective ex-
aminations. I n P E A . Test 1.41, “Social Problems,” the stu-

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE SETS 479

dent is given a problem, asked to select his choice of courses of

action and to check reasons t o support his conclusion (24, pp.
180-190). Some students check many reasons, some few. But
one who checks many reasons is likely t o receive a higher score
in Irrelevancy or Inconsistency than one who lists only the
reasons he considers truly basic t o his opinion. This may be a
basic trait in the student’s reasoning, but it may as easily be a
reflection of the way he interpreted the directions and the in-
tent of the test, While inclusiveness need not confuse interpre-
tation when the pattern of scores is studied as a whole, it does
prevent meaningful treatment of a single score, such as Irrele-
vant Reasons. The P.E.A. Tests of Application of Principles
in Scieace (24, pp. SO-111) also permit inclusiveness to affect
scores, since the student is perm\tJed .to check as many reasons
as he wishes t o support his conclusions. (But cf. 24, p. 117,
where a test was redesigned because inclusiveness interfered
with validity.)
The Thurstone-type attitude scale permits inclusiveness to
affect score. The subject is directed t o check those statements
which reflect his attitude, with no limitation on the number t o
be checked. Some persons check only two or three statements,
while others check several. The score is the median scale value
of the statements checked. There is a tacit assumption that
when one checks additional statements beyond the most ap-
propriate ones, they are balanced around the median of these.
But there is no evidence that these additional statements do
not bring the person’s score neirer t o the group mean, or in
some other way modify it. The same tendency can be found in
checklists of any sort.
4. Bias; acquiescence.-When students are offered two al-
ternatives, as “True” versus “False,” “Like” versus “Dislike,”
“Agree” versus “Disagree,” etc., some use one more than the
other. This effect has been demonstrated in the true-false test
and in personality and attitude tests (2,4, 15, 19). Individual
differences in acquiescence (tendency t o use “Yes” or “True”)
are reliable by split-half and parallel-test-with-elapsed-time
methods. T h e majority of students have an excess number of
“Yes” responses on true-false tests. Since response tendencies
affect an answer only when the student is t o some degree un-

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

480 EDUCATIONAL AND PSYCHOLOGICAL MEASUREhlENT

certain about the content of the item, acquiescence tends t o

make false items,more valid, and true items less valid (2, 4).
The poor student, guessing, tends t o be right on the true item
because of acquiescence, but tends t o be wrong on the false item.
False items alone are often as reliable and valid as the entire
test of double the length. A large amount of acquiescence tends
to reduce the deviation of the student’s score from the mean.
Rundquist found personality test items where “Yes” repre-
sented a negativistic answer more valid than those where “Yes”
was favorable ( 19).
Another instance of individual differences in bias appears
in a pitch-diskrimination test. A recorded experimental test,
roughly similar t o the Seashore Pitch test, was given t o three
psychology classes. Students were directed to mark each item
H or L according as the sdcond.of a pair of tones was higher or
lower than the first. T h e test was very easy, the median num-
ber right being 94. Individual students, particularly those with
poor scores, showed marked tendencies t o overuse one of the
two alternatives. One student had 18 errors of the HmL (read
“higher marked lower”) type b u t only three LmH; another had
16 L m H to 5 HmL. More definite evidence was found on the
last twenty items, which were near the pitch-difference thresh-
old for most students. In general, response sets influence per-
formance most on ambiguous or relatively unstructured items.
This twenty-item test was split into four sets, two containing
five L items, and two containing five H items. The two sets
of L items had equal difficulty, as did the two sets of H items.
Scores were obtained for 133 students. If there is bias in re-
sponding, the number of correct H answers exceeds the number
of correct L answers for the student, or vice versa. This bias
is represented by the formula H-L (where H is number right
on H items).. Out of a possible range, for twenty items, from
1 0 to - 10, actual bias scores ranged from 5 to - 7. The mean
bias was negligibly different from zero. The correlations of
scores for the split tests were as follows:
5 H items x 5 H items, 0.37; corrected, 0.54 (reliability of H score)
5 L items x 5 L items, 0.45; corrected, 0.62 (reliability of L score)
5 H + 5 L items x 5 H + 5 L items, 0.46; corrected, 0.63 (reliability of total)
- -
H L, 10 items x H L, 10 items, 0.33; corrected, 0.49 (reliability of bias score)
-
H + L, 20 items x H L, 20 items, - 0.125. (score x bias)
10 H items X 10 L items, 0.07; corrected, 0.13

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE SETS 48 1
The bias score is definitely reliable (the probable error of an T
of .OO is 0.06 for 133 cases). Bias is nearly independent.of
pitch ability. The reliability of the bias score might be a re-
flection of the fact that the superior student makes few errors,
and so has a reliably low bias score, but when only cases making
five or more errors out of twenty are considered, the corrected
reliability of the bias score is 0.56.
A twenty-item test of H items alone would have a predicted
reliability of 0.70; a test of twenty L items would have a re-
liability of 0.77; yet the total test, with ten of each type, had a
reliability barely as good as the 10 L items alone. The test
obviously measures two factors, pitch and bias. Wyatt has
reported (32, p. 41), and the writer’s experience confirms, that
training for superior discrimination requires deliberate effort
to overcome these biases. Seashore was evidently not suc-
cessful in designing his test t o satisfy his demand that “the
factor under consideration (pitch) must be isolated in order
that we may know what it is that we are measuring” (quoted
by Wyatt, 32, p. 15).
The evidence that the twenty items are not measuring a
single unitary factor is of interest in the light of attempts t o
determine the factor structure of the Seashore test. Guilford’s
data (11) have been questioned by Wherry and Gaylord on
statistical grounds (28), but it is also apparent that he failed
t o detect all the factors in the test because he erroneously as-
sumed that all items having a fixed pitch difference measure
the same trait, whether the hiiher tone is in first or second
position. His data, revealing different factor loadings for items
of different difficulty, are consistent with the hypothesis that
bias becomes more important as difficulty increases.
Bias may result from a deliberate mental set. I n testing
anti-submarine detector personnel for ability t o distinguish be-
tween true indications of a submarine and false indications
yielded by the detector, a test was set up which reproduced
true and false indications, the man being required to report
“Yes” or “No,” according as he believed a submarine t o be
present or not. Even the best operators tended to show a
strong bias toward the “Yes” response which reduced their

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

482 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

scores because they reported many false indications as sub-

marines. When attempts were made to train for higher per-
formance, they defended their errors on the ground that the
only safe course in operation is t o consider all doubtful cases
as submarines until proved otherwise. Despite the fact that
accurate judgment was desirable t o reduce false alarms, it was
found impossible to eliminate this set in order to test discrimi-
nation ability.
5 . Speed v e r ~ u sacc1sracy.-In many tests, speed is an im-
portant element. I n taking such a test, the student has his
choice of responding carefully, or of answering rapidly, achiev-
ing a score through quantity rather than accuracy. Although
empirical scoring devices may compensate for this effect over
a group as a whole, a given individual’s score depends on his
set t o be rapid or t o be accurate: This influence is especially
serious on a test such as the Nels‘on-Denny reading test, which
presents five-choice items scored b y the formula “number
right.” The writer recently reviewed scores of a class of tenth
graders. One student, selected by his teacher as a retarded
reader, and having a test IQ of 69, appeared in the list of scores
a t the 60th percentile on the Nelson-Denny vocabulary test.
He had merely rushed through the test, guessing a t every item;
by chance, he had answered twenty-five out of 100 items cor-
rectly. Even the best correction formula can only estimate how
much a speedy, careless student would earn had he been care-
ful. The criticism offered here on the Nelson-Denny test and
others of like pattern applik primarily t o the meaningfulness
of scores; in a test designed t o predict grades, the scoring
procedure used may be justified empirically in the majority of
cases.
Hall and Robinson (12) made a factor analysis of 25 read-
ing scores, and concluded that the first factor in their battery
was best named “attitude of comprehension accuracy.” This
response set appeared more prominently than the ability
factors.
6. Response sets on esray terts.-Anyone who has taken or
given an essay test is faced with the effect of different sets on
scores. I n addition to inclusiveness, there are probably as

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE SETS 453
many different response sets as there are styles of composition.
T h e student may write a carefully organized response, or he
may produce a stream-of-consciousness answer with no effort
a t organization. Whether he receives as much credit as an-
other student with a different set depends on the method of
grading. If credit is given for organization, the former set is
superior; if credit for number of ideas expressed, the second
may receive the higher score. Some students attempt t o
write smooth essays, while others merely list points in a skeleton
form. Some students elaborate their answers and bring in il-
lustrations, while others present the “bare bones’’ of the re-
sponse. What type of answer is given appears dependent upon
set, and the student’s idea of what is desired. The wise student
tries t o find what type of answer is favored, and adjusts his
procedure accordingly, but this adjustment cannot take place
unless the test situation itself is altkrkd by providing a cue t o
the teacher’s desires. Campbell has discussed individual modes
of reaction t o open questions for public opinion polling (1).
Characteristics of Response SetJ
Individual differences in response sets are reliable. Nu-
merous studies have shown the reliability of differences in the
sets discussed above, by internal-consistency tests. Several
investigators have measured response sets in tests of a given
type by giving severaI tests, sometimes with separation in time;
response set scores have shown substantial correlations.
Response sets have the greatest influence on score in am-
biguous or unstructured situations. If a situation is structured
for the student, so that he knows the answer required, he re-
sponds directly to the content of the item, and response sets
probably are unimportant. If he does not know the answer,
his response is determined by caution, acquiescence, or other
sets. Acquiescence appears on difficult true-false items; bias on
difficult pitch judgments; and evasiveness on attitude judg-
ments where the student has no strong opinion. Ambiguity
may be increased by the test situation or by directions which
leave the student t o judge whether guessing is penalized,
whether speed is more rewarding than carefulness, how many

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

484 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

statements should be checked, or what is meant by “Indiffer-

ent.” The problem is reminiscent of those encountered in
rating scales, where traits and scale positions must be made un-
ambiguous to obtain validity. No “objective” test is truly ob-
jective, so long as any part of the stimulus situation is suffi-
ciently unstructured t o permit individual interpretation.
Degree of structuration varies from the item where the response
is obvious for the group tested, t o the one where the student
has no idea what is wanted.
Response sets affect test reliability. Because they are con-
sistent, response sets may heighten reliability (30). I n other
situations where the response set lowers the correlation between
items, reliability appears t o be lowered. Where a response set,
such as gambling versus caution. increases the spread of scores,
reliability will tend t o rise. Where a response set such as bias
reduces the range of scores, the‘ reliability may be expected t o
decline. The relative reliabilities of the trait under study and
the response set, and the variation of the group in each, must
also‘be considered.
Response sets affect test validity. Since a response set
permits persons with the same knowledge or attitude or ability
t o receive different scores, response sets always lower the logi-
cal validity of a test. Empirical validity, based on ability t o
predict a criterion, may be raised or lowered, depending on the
correlation of the response set with the criterion. Tendency t o
gamble may reflect primarily confidence, in which case the
better student might be l e d cautious when in doubt, and so in-
crease his score; but should willingness t o gamble and ability
have no relation, or a negative relation, empirical validity would
be lowered. I n a test of morale, where the person responds
“Yes” or “No” to pessimistic predictions, acquiescence might be
correlated with everyday morale, because it influences not only
the test performance, but also the readiness to believe rumors;
in this case, the response set could raise correlations with
criteria.
Response sets interfere with inferences from test data. It
becomes difficult to judge learning difficulties from an item
analysis, since response sets influence the peicentage of students

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE SETS 485

passing an item. It is difficult t o evaluate pupil growth when

response sets affect score as a major change of score may reflect,
not growth in knowledge or change in interest, but a set-de-
termined change in inclusiveness, caution, or avoidance of ex-
treme response positions. There is no way of knowing t h a t
such a drastic change in response pattern may not be due t o
temporary moods, or t o increased test-wiseness, rather than t o
basic learning.
The Nature of Response Sets
Response sets are a special case of the learned “work meth-
ods” discussed by R. H. Seashore (21), Jones and Seashore
(13), and Sargent (20). These writers point out that a test
measures, not the subject’s ability, but his performance with
whatever methods he uses; a change of technique might change
the ability score. Seashore comments:
I n measuring individual differences it is not sufficient to
control the instructions or working situation, for the observer’s
previous incidental background may lead him to adopt very
differerit work methods than those expected. It follows that
<L
control” limited to ordinary instructions and demonstrations
is incomplete, and that other unnoticed factors operate t o
modify the work method actually adopted. (21, p. 123.)
Work methods may be temporary sets or may be habitual
techniques of performance.
Response sets may also be compared with constant errors
in psychophysics. I n fact, the bias reported in connection with
pitch has been studied as an erfor in judgment (31, p. 439).
The emphasis, in treating this as a response set, is on the evi-
dence that there are consistent individual differences, though
the error may be “constant” for the individual. Since this
error can be a t least partly overcome by conscious attention t o
it, it does not seem superfluous t o intioduce the concept of
individual differences in set.
Sherif and Cantril (22) describe what we have called re-
sponse sets in terms of frames of reference in their recent re-
view of the psychology of attitudes. They review studies by
Kulpe, Bartlett, Sherif, Durkheim and others, all of which con-
firm the viewpoint that internal conditions of the organism de-
termine response in any partially unstructured.situation. They
state:

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

486 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

.
. . in the absence of an objective scale (frame) and
objective standard (reference point) each individual builds
u p a scale of his own and a standard within that scale. The
range and reference point established by each individual is
’peculiar t o himself when facing the situation alone. Once . ..
a scale is established there is a tendency for the individual t o
preserve this scale in subsequent sessions (within a week in
these experiments). . . .
. .. these frames and points of reference are b y no means
always confined to consciously accepted Instructions. or. ~ m -
posed norms but can become established without an individ-
ual’s realization of it. (22, LII, p. 319; LIII, p. 2:)
Essentially the notion of response sets here developed is an ap-
plication t o testing of the findings reviewed by Sherif and
Cantril.
The crucial question for an understanding of response sets
is the extent to which they are transient or fixed. Is an acqui-
escent person equally acquiescent in a history test, a chemistry
test, and an adjustment inventory? Is the cautious, evasive
person equally so in a grammar test, a personality test, and an
attitude scale? I n experimental studies there seems to be a
consistency of frames of reference from one trial t o another,
and in studies of similar tests given weeks apart, scores in
acquiescence and gambling have been found stable. But at-
tempts t o compare scores in different types of examinations
have shown only negative results, and this would be expected
even if response sets are basic in the personality. For response
sets operate in proportion as a situation is unstructured, and
the student who finds a psychology test unstructured because
of his ignorance, may be able to answer his chemistry test on
the basis of knowledge. Unless degree of structuration could
be equated for all individuals, correlations of “response set
scores” from test t o test are meaningless.
For the present one cannot decide to what extent a reaction
such as evasiveness is specific to the immediate test situation
a t a particular time under particular conditions, t o what ex-
tent it would be expected t o recur in similar situations, or how
much it reflects a basic trait that would appear in any life
situation permitting evasion, if that situation is unstructured.
Probably all three interpretations are valid.
Light is thrown on response sets by the Rorschach test.

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE SETS 487

Unlike the usual test, where the content is crucial and the form
of response is disregarded in the interpretation, the Rorschach
interpretation is based almost entirely on the mode of response,
the work method, or the response set shown by the subject.
The stimulus is almost completely unstructured, and the sub-
ject is allowed to interpret his task as he chooses. T h e “ap-
proach,” or “apperception type” reflects the set of the subject
t o respond t o wholes, subunits, or small details. It is by now
well-validated that the response sets shown in the Rorschach
test are reflections of basic drives and traits, though it is also
recognized that temporary anxiety or desire t o impress the
examiner may also influence scores. Rorschach results suggest
that in any relatively unstructured situation, including “ob-
jective” tests, the response sel of the subject may reflect per-
sonality, as well as learned habits of ’iesponse t o the particular
type of test. It is interesting to speculate on analogies between
the Rorschach signs and the response sets in other tests.
Caution in achievement tests may spring from the same force
that leads to form-accuracy; evasion in personality and attitude
tests may relate t o Rorschach rejection and other withdrawing
or inactivity indicators. Inclusiveness may compare with the
number of responses in the Rorschach; negativism, the opposite
of acquiescence, might have its analog in the white space re-
sponses of the projective test. Essay examinations are always
potential projective situations, and the response sets found:
elaboration, organized sequence of attack, etc., have their exact
counterparts in the inkblot test!
Controlling Response Sets
It is more important to control response sets in some situ-
ations than in others. Where a test is easy and there is a wide
range of ability in the group, response sets have little effect;
dificult items, or a homogeneous group, permit response sets
to have a greater influence. I n some cases, response sets im-
prove reliability and even empirical validity. But respon&esets
always lower the validity with which one measures the trait
defined by the content of the items. Even though the empirical
effect may be small, the writer feels that response sets should
be eliminated where possible. It is only by identifying and

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

488 EDUCATIONAL A N D PSYCHOLOGICAL MEASUREMENT

rooting out, one by one, the factors that dilute validity that
educational and psychological tests can increase their usefulness
as scientific tools.
The first step in controlling response sets is t o identify the
sets possible in a particular test. The list above includes all
the significant sets the writer has been able t o identify, but
others may also exist.
Response sets are reduced by any procedure that increases
the structuration of the test situation. The best procedure ap-
pears to be t o adopt an item form which does not invite re-
sponse sets, wherever that can be done without hampering
measurement. The multiple-choice pattern appears t o be the
only generally useful form that is free from response sets.
This form should be adoped wherever the cqntent permits.
This applies t o both achievimexh tests and to other types. T h e
Kuder Preference Record use’s this form for measuring interests,
as contrasted with the Strong blank, which allows several re-
sponse sets. Where the Bernreuter, Bell, Multiphasic, and
other inventories are open t o evasion and acquiescence, Jurgen-
sen (14)and Viteles (27) recently reported promising attempts
t o obtain more valid answers in personality tests by a multiple-
choice pattern. I n attitude tests, experimentation with a
multiple-choice form in which the student checks the statement
he most agrees with, in each group, seems desirable. Other pat-
terns may be modified t o eliminate opportunities for response
sets. The writer would reduce the five-choice pattern of the
Likert-type scale t o a two-choice judgment; he would discard
the “?,” “Neutral,” and “Indifferent” responses from the three-
choice pattern. This may reduce reliability, which in the past
has been increased by the effect of response sets upon the score.
Eisenberg has suggested that personality tests would be made
less ambiguous by increasing the number of alternatives per
item (7, p. 39), but the writer feels that this places stronger
weight on semantic factors. Woodworth indicates ,that the
three-category scale in psychophysical judgment is neither
better nor worse than the two-category scale (31, p. 425), and
favors retention of the neutral judgment in rating scales (31,
p. 377). His arguments, however, apply to measuring thresh-
olds and differences between rated objects, not t o the present

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE SETS 489

problem of studying individual differences between judges. It

might also improve Likert-type scales t o define the alternatives,
such as “strongly agree,” more objectively, as in the better
rating scales (23).
Directions may be changed t o increase structuration. Dif-
ferences in tendency t o gamble may be eliminated by directing
students t o respond t o every item. Gritten and Johnson (10)
and the writer favor this suggestion, despite all that has been
written against it; encouraging guessing increases the random
errors of measurement, but i t is the only means of eliminating
t h e systematic error resulting from response set. I n .attitude
tests of the Thurstone type, it might be helpful to direct the
student t o check, e.g., the four statements best describing his
beliefs, t o eliminate variation in inclusiveness. Changes in the
test should not be allowed t o interfere with the measurement
intended; it might make a better statistical instrument of the
Mooney Problem Checklist t o limit the number of responses,
but it would make the list less satisfactory for interviewing and
counselling. Directions indicating what is wanted and defining
ambiguous terms may be particularly profitable in essay tests.
Fernberger’s partly successful use of this procedure in psycho-
physical judgments is reported by Woodworth (31, p. 423).
Experimental attempts t o structure the entire test, as by
informing students that just half the items are true, have not
been successful (6, 4). Each response is a separate act of
judgment, and attempts t o increase structuration must be
aimed a t the individual item.
I n many economical and desirable test forms, it will not be
possible t o remove response sets. Other approaches t o cope
with the problem are required. One of the best is increasing the
test-wiseness of the student. By showing the acquiescent stu-
dent how many False-marked-True errors he makes, in com-
parison with True-marked-False, it is often possible t o cause
him to become consciously critical in answering questions. jEn-
couragement may overcome obercaution which is causing a
student t o receive poorer scores than he deserves. Training t o
overcome bias has already been mentioned. It is relatively easy
t o teach mature students what is desired in essay examination
responses.

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

490 EDUCATIONAL A N D PSYCHOLOGICAL RlEASUREMENT

If the tester is conscious of response sets, he can examine

papers t o determine what sets may have affected scores. T h e
device of the Minnesota Multiphasic, which discards as invalid
all tests showing excessive evasiveness, might be useful in other
instruments.
T h e basic problem in response sets is that we are assuming
that a test score measures a single variable. Actually, even if
item content is homogeneous, item form introduces several
variables into a score. The A-U-D pattern for attitude tests
permits two degrees of freedom in the response t o any item.
T h e A-a-u-d-D pattern introduces 4 d.f., which might be named
reaction t o content, neutrality or evasiveness, acquiescence ver-
sus negativism, and tendency t o go t o extremes. T h e simplest
approach t o this problem,'if the response set cannot be elim-
inated, is t o report .as &a.ny scores for each individual as
there are degrees of freedom in the test pattern. T h e scores
for a person can be successfully interpreted as a profile or
pattern. This conforms t o current organismic attempts t o con-
sider the total behavior in the test situation, as used both in the
Rorschach test, and the relatively structured tests 1.3,4.2, and
8.2 of the Progressive Education Association. If statistical
treatment is t o be made, it is important t o retain all degrees of
freedom. Attempts t o reduce L-I-D percentages to a single
score always discard information. By a choice of two scores or
L
functions of scores (L-D, - etc.), meaningful relations may
L+D'
be made clearer. I n an inferest test of the L-I-D type, Livesay
and the writer found t h a t the most meaningful picture was t o
be obtained by plotting scores in a two-space with three homo-
geneous coordinates. Statistical methods for such a space can
be devised which permit considering all variables a t once (16).
One final suggestion is t o weight responses so that in the
majority of cases the response set increases validity. Since the
majority of persons, when in doubt, tend t o judge statements
true, doubt may be penalized by counting false responses more
heavily, or by loading the test with a majority of false state-
ments. This increases the validity of scores, on the whole, but
gives a spuriously high score to the occasional highly critical
individual. This practice underlies such weighted scorings as

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE SETS 491

used by Strong and Bernreuter, which makes their tests reliable

on the whole, however invalid they may be for a person with an
atypical set. I n such sets as gambling versus caution, or speed
versus accuracy, the score is normally weighted t o favor one
particular set.
SZMItma ry ati d Conclusio tts

1. Response sets are defined as any tendency causing a

person consistently t o make different responses to test items
than he would have made had the same content been presented
in a different form.
2. Evidence is presented, or cited from other studies, to
demonstrate the existence of these response sets:
a ) Tendency to gambl?; caution versus incaution. This
is found in usual objecGye examinations, and appears as
evasiveness in personality, interest, and attitude tests.
b ) Definition of judgment categories. Individuals dif-
fer in the meaning assigned to, and the frequency of use
pf, alternatives offered in attitude and personality scales.
c ) Inclusiveness, the tendency t o give many responses
where the number of statements to be checked, or the
like, is unspecified. This appears in certain tests of
reasoning, attitudes, adjustment, etc.
d ) Bias; acquiescence. This appears in true-false tests,
discrimination tests, and some attitude, personality, and
interest tests.
e ) Speed versus accuraFy.
f ) Miscellaneous response sets on essay tests, related
to style of response.
3. Individual differences in response sets are reliable.
4. Response sets have the greatest influence on performance
in ambiguous or unstructured situations.
5. Response sets may raise or lower reliability, and may
raise or lower validity as measured by correlations with criteria.
But because they permit persons with equal knowledge, identi-
cal attitudes, or equal amounts of a personality trait to receive
different scores, response sets always reduce logical validity.
Response sets interfere with interpreting test data t o reveal
difficulty of item content, or growth as a result of training.
6. It is uncertain whether response sets are specific t o a

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

492 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

given type of situation, or whether the student who is acqui-

escent on an achievement test would also be acquiescent in
personality and attitude tests if they were equally unstructured
for him. Temporary variations in mood or set may influence
performance, but retest studies show stability in response sets.
7. The following suggestions are made for eliminating the
effect of response sets upon test validity:
a ) The multiple-choice form, which appears free from
response sets, should be used wherever possible.
b ) The test pattern should be made less ambiguous, by
reducing the number of alternatives for a judgment and
eliminating the neutral response. Alternatives in Likert-
type scales should be objectively defined.
c ) Directions should be changed t o eliminate variations
in response set. Directions t o respond when in doubt
are recommended.
d ) The test-wiseness of the student may be increased by
,an explanation regarding his response sets.
e) Scores of persons revealing strong response sets may
be discarded.
f ) Because most item forms permit more than one de-
gree of freedom in the response, methods of retaining all
of the information are needed. Interpretation of pro-
files or patterns of scores is desirable. Statistical meth-
ods for handling two scores a t once are referred to. .
g) Scores may be wei4hted so that response tendencies
which correlate with lack of knowledge in the majority
of cases are penalized.
There are many poifits in the response-set hypothesis not
supported by direct evidence, but it appears that sufficient
evidence is available t o prove that a real effect is present. It
may seem that the points raised are trivial, in view of the great
service rendered in the past by personality, interest, attitude,
and achievement tests where sets are permitted t o influence
scores. Yet recognition and control of such irrelevant factors
are precisely the improvements needed t o raise mental measure-
ment from its present imperfect level.
Further research, including experimental validation of the
suggestions for controlling sets here offered, is called for. If

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

 RESPONSE SETS 493
methods of study can be found, knowledge is required regarding
the nature and origin of individual differences in response sets.
The only sound procedure for controlling structuration, t o
study response sets unaffected by the person’s knowledge about
item content, is t o use nonsense items where no one has a
knowledge of the content. This is difficult to.use on any large
scale with the retention of normal test attitudes.
REFERENCES
1. Campbell, A. A. “Two Problems in the Use of the Open Ques-
tion.” Journal of Abnormal and Sock! Psychology, XL
(1945), 340-343.
2. Cronbach, L. J. “An Experimental Comparison of the Multiple
True-False and Multiple Multiple-Choice Tests.” Journal
of Educational PJychology, XXXII (1941), 533-543.
3. Cronbach, L. J. “Exploring the Wartime Morale of High School
Youth.” Applied Psychology Monographs, I (1943), No. 1,
4. Cronbach, L. J. “Studies of Acquiescence as a Factor in the
True-False Test .” Journal of Educational Psychology,
XXXIII (1912), 401-415.
5. Cronbach, L. J. “The True-False Test: a Reply t o Count
-Etoxinod.” Education, LXII (1941), 59-61.
6. Dunlap, J. W., DeMello, A., and Cureton, E. E. “The Effects of
Different Directions and Scoring Methods on the Reliability
of a True-False Test.” School and Society, XXX (1929),
3 78-3 82.
7. Eisenberg, P. “Individual Interpretation of Psychometric In-
ventory Items.”‘ Journal of General Psychology, X X V
(1941), 1 9 4 .
8. Fernberger, S. W. “The Use of Equality Judgments in Psycho-
physical Procedures.” Psychological Review, XXXVII
(1930), 106-112.
9. Gilmour, W. A. and Gray,’D. E. “Guessing on True-False
Tests.” Educational Research Bulletin, XXI (1942), 9-12.
10. Gritten, F. and Johnson, D. M. “Individual Differences in
Judging Multiple-Choice Questions.” Journd of Educa-
tional Psychology, XXXII (1911), 423430.
11. Guilford, J. P. “The Difficulty of a Test and Its Factor Com-
position.” Psychometrika, VI (1941), 67-77.
12. Hall, W. E. and Robinson, F. P. “An Analytical Approach to
the Study of Reading Skills.” Journal of Educational Psy-
chology, XXXVI (1945), 429-442. >
13. Jones, H. E. and Seashore, R. H. “The Development of Fine
Motor and Mechanical Abilities.” Adolescence, 43rd Year-
book of the National Society for the Study of Education.
Edited by N. B. Henry. Chicago: University of Chicago
Press, 1944, pp. 123-145.

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014

494 EDUCATIONAL A N D PSYCHOLOGICAL hTEASUREMENT

14. Jurgensen, C. E. “Report on the ‘Classification Inventory,’ a

Personality Test for Industrial Use.” Journal of Applied
Psychology, XXVIII (1944), 445460.
15. Lentz, T. F. “Acquiescence as a Factor in the hleasurement of
Personality.” Psychological Bulletin, X X X V (1938), 659.
16. Livesay, N. and Cronbach, L. J. “Statistical h/lethods for
Closed Systems.” Unpublished.
17. Mosier, C. I. “A Psychometric Study of Meaning.” Journal of
Social Psychology, XI11 (1941), 123-140.
18. Osgood, C. E. ‘‘Ease of Individual Judgment-Processes in Re-
lation to Polarization of Attitudes in the Culture.” Journal
of Social Psyclzology, X I V ( 1941), .403-118.
19. Rundquist, E. A. “Form of Statement in Personality Measure-
ment.” Joutsal of Educational Psychology, XXXI ( 1940),
135-1 47.
20. Sargent, S. S. “How Shall We Study Individual Differences?’,
Psychological Review, XLIX (1912), 170-181.
21. Seashore, R. H. “Work Methods: an Often Neglected Factor
Underlying Individual Diff~rences.” P.ryclzologicd Review,
XLVI (1939). 123-141.
22. Sherif, M. a i d Ca&il, Hadley. “The Psychology of Attitudes.”
Psychological Review, LII ( 1 9 4 9 , 295-319; LIII (1946),
4 n ,
l-L+.
23. Simpson, R. H. “The Specific Meanings of Certain Terms In-
dicating Different Degrees of Frequency.” Quarterly
Journal of Speech, XXX (1944), 328-330.
24. Smith, E. R. and Tyler, R. W. Appraising and Recording Stu-.
de7zt Progress. N. Y.: Harper, 1942. 550 pp.
25. Swineford, F. “Analysis of a Personality Trait.” Journal of
Educational Psychology, XXXII (1941), 438444.
26. Swineford, F. “The Measurement of a Personality Trait.”
Journal of Educational Psychology, XXIX ( 1938), 295-
300.
27. Viteles. M. S. “The Aircrdt Pilot: Five Years of Research.
A ’Summary of Outcomes.” Psychological Bulletin, X L I I
(1945), 489-526.
28. .Wherry, R. J. and Gaylord, R. H. “Factor Pattern of Test
Items and Tests as a Function of the Correlation Co-
efficient.” Psycl~owzetrika,IX ( 1914), 237-244.
29. Wiley, L. N. and Trimble, 0. C. “The Ordinary Objective Test.
as a Possible Criterion of Certain Personality Traits.”
Sckool and Society, XLIII (1936), 446448.
30. Wood, B. D. “Studies of Achievement Tests.” Jozrriial of
Educational Psyclzology, XVII ( 1926), 1-22. 1

31. Woodworth, R. S. Experimental Psycliology. N. Y.: Holt,

1938. 889pp.
32. Wyatt, R. F. “Improvability of Pitch Discrimination.” Psy-
chological Monographs, LVIII (1945), No. 2. 58 pp.

Downloaded from epm.sagepub.com at GEORGE MASON UNIV on June 18, 2014