PROFESSIONAL PSYCHOLOGY: RESEARCH AND PRACTICE
he wrote "Recall too that beyond a surprisingly limited num-
ber of variables or amount of information, additional infor-
mation usually does not increase but may decrease predictive
accuracy" (p. 427). He cited five studies, one of which does
not even involve the presentation of information to clinicians
(Winch & More, 1956). His most recent reference was from
1965. I recently reviewed the incremental validity literature
(Garb. 1984). I found 29 appropriate studies in which clini-
cians were given increasing amounts of information. Validity
generally increases when biographical, Minnesota Multiphasic
Personality Inventory (MMPI), or neuropsychological test
data are added to demographic or psychometric information.
However, the findings for other types of assessment data were
not as positive; for example, the addition of videotapes of
interviews to transcripts of interviews did not lead to an
increase in validity.
Confidence Ratings
Faust wrote. "A number of researchers have found that
individuals who are very confident in their judgments are
often wrong" (p. 423). He cited several studies from the areas
of social and cognitive psychology. In none of the studies did
the researchers use mental health professionals as judges. I
reviewed studies in which clinical psychologists or psychia-
trists made judgments and then rated the degree of confidence
that they had in each of their judgments (Garb, 1986). Eight-
een appropriate studies were found. Confidence ratings were
related positively to the validity of judgments in a number of
studies.
Level of Experience
Faust wrote, "There is limited evidence—in fact, almost
none—that experts or those with exhaustive experience per-
form significantly better than 'regulars' or relative beginners"
(p. 420). Research does indicate that the validity of judgments
is not greater for experienced clinicians than for inexperienced
judges. However, experienced clinicians tend to make more
appropriate confidence ratings than do inexperienced judges
as long as they are given information that is generally consid-
ered to be valid (e.g., neuropsychologica) data, MMPI results;
Garb, 1986). These results were found in six studies.
Reliability of Diagnoses
According to Faust, "Diagnostic reliability and validity are
longstanding problems, and the judgment research may help
clinicians and researchers to understand why this is so" (p.
423). He also wrote. "Clinicians often have difficulty with
broad diagnostic choices, to say nothing of more refined
assessment" (p. 426). Faust cited no studies to support his
claim that reliability and validity are low. Matarazzo (1983)
reviewed the literature on the reliability of psychiatric and
psychological diagnosis. He concluded that "clinician-to-cli-
nician diagnoses have now attained remarkably high levels of
reliability" (p. 103).
549
Final Comment
Faust overlooked many studies on clinical judgment. He
consistently underestimated the success that clinicians have
when making judgments. A comment by Christensen-Szalan-
ski and Beach (1984) seems relevant in this context: "It is our
hypothesis that the widely held belief in the hopelessness of
human judgment and decision performance results in part
from the fact that only evidence to that effect gets much
attention. That is, evidence for poor performance is cited
more frequently than is evidence for good performance" (p.
75).
References
Christensen-Szalanski, J. J. J., & Beach, L. R. (1984). The citation
bias: Fad and fashion in the judgment and decision literature.
American Psychologist, 39, 75-78.
Faust. D. (1986). Research on human judgment and its application
to clinical practice. Professional Psychology: Research and Practice.
77,420-430.
Garb. H. N. (1984). The incremental validity of information used in
personality assessment. Clinical Psychology Review, 4, 641-655.
Garb. H. N. (1986). The appropriateness of confidence ratings in
clinical judgment. Journal of Clinical Psychology. 42. 190-197.
Goldberg, L. R. (1959). The effectiveness of clinicians' judgments:
The diagnosis of organic brain damage from the Bender-Gestalt
Test. Journal of Consulting Psychology. 21, 25-33.
Matarazzo. J. D. (1983). The reliability of psychiatric and psycholog-
ical diagnosis. Clinical Psychology Review, 3, 103-145.
Oskamp, S. (1965). Overconfidence in case-study judgments. Journal
of Consulting Psychology. 29. 261-265.
Winch. R. F.. & More, D. M. (1956). Does TAT add information to
interview? Statistical analysis of the increment. Journal of Clinical
Psychology. 72,316-321.
HOWARD N. GARB
Psychology Service
Highland Drive Veterans Administration Medical Center
Pittsburgh. Pennsylvania •
Response to Garb
Even if one accepts what I will show to be Garb's dubious
criticisms, many core aspects of my original argument remain
untouched. Garb does not dispute, for example, the superi-
ority of actuarial methods, problems in the assessment of
covariation and use of base rates, limits in configural judg-
ment capacities, and so on.
Garb argues that 1 have misrepresented research on a num-
ber of topics. As stated in my article, coverage was necessarily
skeletal and readers were encouraged to review original
sources for themselves. However, I have not misrepresented
overall findings. True, I did not cite some studies showing
greater accuracy than that obtained in the studies I did cite,
but I omitted an equal or greater number showing lower levels
of accuracy (e.g., Temerlin & Trousdale's 1969 study dem-
onstrating a 100% error rate among professionals). Garb
contends that I also misrepresent the literature on incremental
validity or performance in relation to amount of information.
However, these studies show that once one is given access to
550 PROFESSIONAL PSYCHOLOGY: RESEARCH AND PRACTICE
the two or three most valid variables, additional information
produces little or no improvement, or even a decline, in
accuracy. Obviously, if one first provides minimally useful
data and then adds much more useful data, judgment can
hardly help but improve. The pertinent questions are, Does
judgment improve when the clinician starts with the two or
three most valid variables and then receives further informa-
tion? and Given an array of data, do clinicians place greatest
weight on the most useful information'.' Studies on cue utili-
zation, the continuing popularity of human figure drawings,
and disregard of base rates, to cite a few examples, tell an
obvious story. Although many clinicians assume that it is best
to "integrate all of the data," there is absolutely no scientific
support for this policy; the value of exclusion is rarely recog-
nized.
Have 1 misrepresented research on confidence and accu-
racy? In my article 1 merely stated that individuals quite
confident of their conclusions can often be wrong. Garb cites
his own review (1986) on confidence and accuracy, in which
he mainly seems to "count noses" and misses deeper mean-
ings. First, Garb notes that a number of studies show no
relationship between confidence and accuracy, but he argues
that low performance levels precluded significant relation-
ships. However, it should be very obvious that any clinician
who is confident about invalid judgments is overconfident.
Second, these studies show that confidence generally increases
as amount of information increases, and that as such the
relationship between confidence and accuracy will be deter-
mined by the relationship between additional information
and accuracy. Add more information, and confidence will
increase on average. If the additional information is useful
and happens to increase accuracy (see previous paragraph),
the relationship between confidence and accuracy will be
positive. If the additional information does not increase ac-
curacy, or decreases accuracy, there will be no relationship or
a negative one (as well illustrated by Wedding's 1983 study).
Thus these studies provide virtually no evidence that confi-
dence levels are appropriately calibrated.
Garb criticizes my failure to cite studies on the low relia-
bility of psychological diagnosis. I thought that the readership
of Professional Psychology: Research and Practice hardly
required documentation of this claim. Garb counters with an
article by Matarazzo (1983) covering some earlier and seem-
ingly promising results, mainly as pertains to DSM-HI
(American Psychiatric Association, 1980) or related diag-
noses. A more recent review by Kutchins and Kirk (1986)
showed that replication was not attained in later studies.
Indeed, the reliability of DSM-HI may be no better than
DSM-II and, in absolute terms, remains abysmal for many
diagnostic categories. In a more recent article, Matarazzo
(1985) stated that extended research efforts have produced no
evidence that clinicians' interpretations are any more valid
than computerized interpretations, the latter of which he
characterized as having failed to meet even beginning tests of
validity. I find it ironic that Garb repeatedly criticizes my
supposed selectivity, although he himself cites Matarazzo
(1983) but fails to cite subsequent studies that render this
article's claims doubtful at best and even fails to cite Matar-
azzo's own later statements about validity (which have ap-
peared in at least two journals). In regard to my claims about
experience and accuracy, if the strongest counterpoint is that
a few questionable studies show that greater experience may
sometimes be associated with better calibration of confidence
(including better awareness of inadequate performance), but
not with greater diagnostic or predictive accuracy, then what
of substance needs to be modified?
Garb seems to suggest that I view human judgment as
hopeless. I never said such a thing, and I certainly do not
believe it; if I did, I could find better uses of my time than
writing these papers. Obviously, human beings have devel-
oped and are capable of further developing decision aids and
technology. The main problem is convincing most clinicians
of the restrictions in unaided judgment, the pressing need for
supplementary tools, and the means by which current limi-
tations may eventually be overcome. There is no shame in
recognizing the limits of unaided human capacity. After all, I
readily concede the need to fly to Paris rather than to swim
there and do not think less of people because of this. As
eloquently argued by Hayek (1955), it may be among the
most important and most difficult of tasks to grasp the limits
of human intellect.
References
American Psychiatric Association (1980). Diagnostic and statistical
manual of mental disorders (3rd ed.). Washington, DC: Author.
Garb, H. N. (1986). The appropriateness of confidence ratings in
clinical judgment. Journal of Clinical Psychology, 42, 190-197.
Hayek, F. A. (1955). The counter-revolution of science. New York:
Free Press of Glencoe.
Kutchins, H., & Kirk, S. A. (1986. Winter). The reliability of DSM-
III: A critical review. Social Work Research & Abstracts, 3-12.
Matarazzo, J. D. (1983). The reliability of psychiatric and psycholog-
ical diagnosis. Clinical Psychology Review. 3, 103-145.
Matarazzo, J. D. (1985). Clinical psychological test interpretations by
computer: Hardware outpaces software. Computers in Human
Behavior, 1, 235-253.
Temerlin, M. K., & Trousdale, W. W. (1969). The social psychology
of clinical diagnosis. Psychotherapy: Theory, Research, and Prac-
tice, 6, 24-29.
Wedding, D. (1983). Clinical and statistical prediction in neuropsy-
chology. Clinical Neuropsychology. 5, 49-54.
DAVID FAUST
Division of Child and Family Psychiatry
Rhode Island Hospital •
What Do Counseling Psychologists Think About
Their Graduate Training? Some Positive and Negative
Reflections
The issue of how to most effectively train psychologists has
long been important in professional psychology. The impor-
tance of this issue is reflected in the number of training
conferences that have been held over the years (e.g., the
Chicago, Greyston, and Vail conferences), as well as in re-
cently scheduled conferences (e.g., National Conference on
Graduate Education in Psychology, National Conference on
Internship Training in Psychology, and Conference on Spe-