Psychological Assessment Copyright 2001 by the American Psychological Association, Inc.

2001, Vol. 13, No. 4, 452-471 1040-3590/01/S5.00 DOI: 10.1037//1040-3590.13.4.452

The Rorschach: Facts, Fictions, and Future

Donald J. Viglione Mark J. Hilsenroth

Alliant International University Adelphi University

A large body of empirical evidence supports the reliability, validity, and utility of the Rorschach. This
same evidence reveals that the recent criticisms of the Rorschach are largely without merit. This article
systematically addresses several significant Rorschach components: interrater and temporal consistency
reliability, normative data and diversity, methodological issues, specific applications in the evaluation of
thought disorder and suicide, meta-analyses, incremental validity, clinician judgment, patterns of use, and
clinical utility. Strengths and weaknesses of the test are addressed, and research recommendations are
made. This information should give the reader both an appreciation for the substantial, but often
overlooked, research basis for the Rorschach and an appreciation of the challenges that lie ahead.

A large body of empirical evidence supports the reliability,
validity, and clinical utility of the Rorschach, suggesting that the
recent criticisms of the Rorschach are largely without merit. We
address 10 areas in which the Rorschach has recently been chal-
lenged: (a) interrater reliability, (b) temporal consistency reliabil-
ity, (c) normative data and diversity, (d) specific applications in the
evaluation of thought disorder and suicide, (e) meta-analyses, (f)
incremental validity, (g) clinician judgment, (h) implications of
clinical and forensic usage, (i) clinical utility, and (j) methodolog-
ical issues relevant to validity. In each section, we summarize the
empirical findings and their implications, explore the challenges,
and recommend areas for future investigation to enhance the
clinical utility of the Rorschach in personality assessment, so that
we may increase our ability to understand our clients and to
intervene effectively. This review will give the reader an appreci-
ation for the substantial, but often overlooked and discounted,
research basis for the reliability, validity, and utility of the
Rorschach.
Reliability
Interrater Reliability
Individual research studies published over the past 20 years in
peer-reviewed journals (Acklin, McDowell, & Verschell, 2000;
McDowell & Acklin, 1996; Meyer, 1997a, 1997b; Viglione,
1999), in Exner's volumes (Exner, 1993), and in the publication
Donald J. Viglione, California School of Psychology at Alliant Interna-
tional University; Mark J. Hilsenroth, Derner Institute of Advanced Psy-
chological Studies, Adelphi University.
We extend our appreciation to William Perry and Leonard Handler for
their thoughtful comments on an earlier version of this manuscript. Thanks
to our students for their help in collecting materials for this review: Steven
Ackerman, Matthew Baity, Nicole Dizon, Erin Eudell, Alice Laing, Kath-
leen Montemagni, Joseph Obegi, Suzanne O'Brien, Maria Ortiz, and Todd
Pizitz.
Correspondence concerning this article should be addressed to Donald J.
Viglione, California School of Psychology at Alliant International Univer-
sity, 6160 Cornerstone Court, San Diego, California 92121.
process (Meyer et al., in press) provide conclusive empirical
evidence of strong interrater reliability for the great majority
(95%) of Comprehensive System (CS) variables. An extensive
amount of data reveals that well-trained raters can score both high
or low base-rate CS variables with good (K and intraclass corre-
lation [ICC] > .60) to excellent (>.75-.80) reliability (Garb, 1998;
Shrout & Fliess, 1979).1
Conclusion: Empirical Data Support Rorschach Interrater
Reliability
The great majority (95%) of the individual variables are coded
with good or excellent interrater reliability. Interrater reliability for
form quality, individual cognitive special scores, form versus color
dominance, shading subtypes, DOv and DOv/+, and some content
categories could be enhanced. Only a few very low base-rate
variables (e.g., MQ none) occasionally produce poor reliability
coefficients, but they have not yet been analyzed with sufficiently
large samples given their miniscule base rates. As in the past
(Exner, 1974, 1995; Viglione, 1997), Rorschach researchers have
recognized the limitations of the test. Improvement in training for
the coding of these variables is the focus of a new reference book
currently being prepared for publication (Viglione, 2001).
Empirical Facts Regarding Temporal Consistency
Reliability
There is general agreement (Meyer, 1997a; Viglione, 1999) with
Wood and Lilienfeld's (1999) statement that "Exner and his col-
leagues have reported test-retest reliability coefficients and that
these coefficients are generally quite good" (p. 344). The great
majority of the CS variables with such strong temporal consistency
reliability are central interpretive variables (Exner, 1991,1993). In
1
Our review of interrater reliability only addresses CS reliability. Many
other studies have demonstrated good to excellent reliability data for
non-CS variables such as the Mutuality of Autonomy (e.g., Fowler, Hilsen-
roth, & Handler, 1996), Aggressive Content (e.g., Gacono & Meloy, 1994),
and the Rorschach Oral Dependency scales (e.g., Fowler et al., 1996).

452
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE
other words, these variables are listed among the 68 nonredundant
variables on the lower portion, "below the line" in the CS struc-
tural summary blank. In turn, the great majority of variables
without test-retest coefficients (e.g., those identified by Garb,
Wood, Nezworski, Grove, & Stejskal, 2001) are not included
among these central, interpretive ratios, percentages, and deriva-
tions. Thus, the variables supposedly without reliability coeffi-
cients are not as central to interpretation. Of the 51 variables
identified by Garb et al. (2001) as having test-retest coeffi-
cients, 46 (90%) are central, interpretive ratios, percentages, and
derivations. Consequently, the great majority of the most impor-
tant variables have adequate data for temporal consistency reli-
ability evaluation.
Garb et al. (2001) presented two tables addressing the issue of
CS test-retest reliability. Their Table 1 presents the central inter-
pretive variables that have test-retest reliability coefficients.
Among the 30 variables that Garb et al. identified (in their Table
2) as not having test-retest coefficients, there are 3 variables
(FC:CF + C, a:p, EB) that actually do have adequate data.
FC:CF + C, a:p, EB are each ratios of two other variables, and all
six of the component scores are listed by Garb et al. in their
Table 1 as variables for which test-retest coefficients have been
reported. The text in a study by Exner, Armbruster, and Viglione
(1978) clearly reveals that evidence of good reliability for EB,
EA:ep (now called EA:es), and FC:CF + C cannot be attributed to
chance. With respect to other variables that Garb et al. said lacked
retest data, one, EBPer, is actually just a cutoff score of EB, two
of the listed scores do not exist [F-% and H + (Hd):(A) + (Ad)],
and one variable is listed (blends), even though Exner (1999) has
reported on its retest stability. In addition, Garb et al. listed the
An + Xy score twice, listed the W variable twice, first as W and
then as W:M, and listed the variable M (in the W:M comparison),
even though it is also listed in their Table 1 as one of the scores
that has test-retest data. The variables CONTAM, PSV, and CP are
listed, but they have such incredibly low base rates that correla-
tional, parametric statistics are unstable without huge samples.
Among the 30 variables that Garb et al. (2001) alleged to lack
test-detest correlation coefficients, there are only 12 variables that
actually do not have test-retest data.
Table 1 is a more complete listing of the available test-retest
coefficients. These test-retest coefficients compare favorably to
the Wechsler scales, second version of the Minnesota Multiphasic
Personality Inventory (MMPI-2) and other tests. Table 2 presents
data culled from many sources, comparing CS test-retest correla-
tions with those from these other scales. Table 2 presents the recent
meta-analyses that address this issue, an exhaustive selection of
recent MMPI and MMPI-2 data, and data on two other represen-
tative tests, one a performance test and the second a commonly
used research instrument.
Memory effects cannot account for strong temporal consistency
reliability findings. With respect to retest intervals of 6 months or
more, psychometric texts (e.g., Anastasi & Urbina, 1996) mini-
mize the possibility that memory effects act as confounds. It is
certainly not an issue in 3-year retest intervals (Exner et al., 1978).
In addition, Haller and Exner (1985) addressed the memory issue
empirically. They instructed experimental respondents to give
different answers on the second testing, but they gave no such
instructions to the control groups. The experimental group pro-
duced adequate temporal consistency reliability coefficients (mean
453
correlation = .66; Mdn = .71), which were almost identical to
those of the control group (mean correlation = .72; Mdn = .74).
Nonetheless, a review of the responses revealed that experimental
participants gave many more new responses in the second testing
than did control participants. Thus, even though the content of the
responses changed, the formal scores for the central interpretive
variables largely remained the same.
The six actuarial indexes in the CS (i.e., Schizophrenia [SCZI],
depression [DEPI], suicide constellation [S-CON], coping deficit
[GDI], hypervigilance [HVI], and obsessive style [OBS]) lack
definitive temporal consistency data, but available findings support
their stability. The actuarial indexes are derived from 58 vari-
ables, 47 of which possess or adequate test-retest data in the
research reports. Six of the 11 scores without test-retest data (FQo,
FQ-- FQu, Level 2 special scores, and FABCOM2) are subcom-
ponents of other variables (X—%, X+%, Ego Impairment Index
[EII], WSUM6, SUM6, INC + FAB) with good to excellent test-
retest data.
Garb et al. (2001) cited several studies that seemed to demon-
strate less acceptable temporal consistency reliability for the CS,
but they did not report their serious methodological flaws. These
flawed studies include research with a nonstandard form of ad-
ministration (Schwartz, Mebane, & Malony, 1990); a dissertation
involving only 17 older adults with an unspecified test-retest
interval (Erstad, 1996); and a sample of schizophrenic patients
with various illness courses, medications, hospitalizations, psycho-
therapies, and test-retest intervals ranging from 1 to 18 years
(averaging 6.4 years; Adair & Wagner, 1992).
Conclusion and Future Research
The empirical data support the conclusion that the great majority
of CS central interpretive ratios, percentages, and derivations pos-
sess impressive temporal consistency. As demonstrated in Table 2,
the test-retest coefficients for CS Rorschach variables compare
quite favorably with those for other tests. For the limited number
of variables without published retest coefficients, most are of
secondary interpretive significance. Typically, they enter into cal-
culations to produce the central interpretive variables, and the
latter have demonstrated adequate or better temporal consistency.
The CS has more than adequate temporal consistency reliability in
all respects, especially in the context of comparing Rorschach data
to other established methods of assessment.
Nonetheless, more test-detest reliability research should be un-
dertaken. The Rorschach and the CS, like many other psycholog-
ical assessment measures (e.g., MMPI/MMPI-2 and Wechsler
Adult Intelligence Scale—Revised [WAIS-R/WAIS-III]), con-
tinue to evolve; therefore, temporal consistency information
should be updated and include variables for which no data are
available. There were only 22 ratios, percentages, and derivations
in the CS structural summary and approximately a dozen other
variables central to interpretation (Exner, Weiner, & Schuyler,
1976) at the time that the original temporal consistency data were
processed (Exner, Armbruster, & Viglione, 1978). Future research
could utilize short test-retest intervals and ensure minimal change
in psychiatric, environmental, and target constructs. The sample
should include both nonpatients and a variety of patients to avoid
restrictions of range on actuarial indices and pathology indicators.
To analyze data with moderately low base rates, a large sample
Table 1
Test-Retest Correlation Coefficients for Comprehensive System Structural Summary Variables Among Adults

Group8
Variables" 1 2 3 4
N 50 100 35 25 57
Retest period a 1 year z 3 years 3 weeks 3 or 4 days 2 years
Participants Adult nonpatients Adult nonpatients Adult depressed inpatients Adult nonpatients Adolescent nonpatients age 14 to 16
Complexity, style, controls
R .86 .79 .84 .77 .80
Lambda .78 .82 .76 .82 .78
Eff* (M:WSumQ
M .84 .87 .83 .75 .77
WSumC .82 .86 .83 .68 .69
a
EA .83 .85 .84 .71 .71
EM .77 .72 .72 .28 .81
m .26 .39 .34 .84 .06
C .73 .67 .67 .77 .69
V .87 .81 .89 .82 .92
T .91 .87 .96 .86 .91
Y .31 .23 .41 .69 .13
eb (FM + m:Sum Sh)
FM + m .69
SumSh .66 .71 .74 .71
es .64 .72 .59 .69 .73
D-Score .91 .83 .88
Adj esd .82 .85
Affecte
FC .86 .86 .92 .71
CF + Cf .81 .79 .83 .74 .72
C + Cn .56 .51 .59
Afr. .82 .90 .85 .57 .76
S .18
Blend? .74 .71
Ideation
<f .83 .86 .87 .71 .83
p" .72 .75 .85 .51 .78
2AB + Art + Ay (Intell.) .84 .88
Sum6 special scores .81 .88 .81 .84f .68f
WSum6 .86 .86
Mediation
P .83 .73 .81 .76 .82
X+% .86 .80 .87 .82 .85
X-% .92 .80 .88
Xu% .85 .89
Information processing
7f .85 .83 .89 .79
7d .70
Interpersonal
COP .81 .88
AG .82 .86
Isolate/R .84 .83
Self-perception
3r + (2)/R .89 .87 .90 .74 .73
Fr + rF .82 .89
FD .88 .90 .76
MOR .71 .83 .87

" Citations for the groups are as follows: Group 1 = Exner, 1993; Exner, 1999. Group 2 = Exner, Armbruster, & Viglione, 1978; Exner, 1993; Exner &
Weiner,
b
1995. Group 3 = Exner, 1993, 1999; Exner & Weiner, 1995. Group 4 = Mailer & Exner, 1985. Group 5 = Exner, Thomas, & Mason, 1985.
For those variables listed as missing by Garb et al. (2001), and for which we could find no published retest coefficients, we asked Exner to provide the
coefficients. Garb et al. (2001) did not list M—, DQ+, DQv, and Level 2 cognitive special scores as missing. Accordingly, we did not process coefficients
for these variables and do not present mem in the table. CP and Mnone are listed as missing by Garb et al. (2001). They have extremely low base rates.
Only 5 of the approximately 13,392 responses in the nonpatient reference group of 600 adults (Exner, 2001) were assigned a CP and only 4 were assigned
a Mnone. Accordingly, a huge sample is needed to stabilize the retest coefficients for these two variables and values were not requested for them. Test-retest
coefficients are still needed for Ma, Mp, Level 2 cognitive special scores, W. D, Dd, DQ+, DQv, Fd, Pure H, (H) + Hd + (Hd), An + X>, and the new
comprehensive system variable (Exner, 2001).
c
d
EBPer is merely a cutoff score of EB. It is listed as missing by Garb et al. (2001).
c
Adj D is merely a transformation of EA—Adj es, both of which have coefficients.
Listed by Garb et al. (2001) as lacking test-retest coefficients.
f
At the time of these publications (Haller & Exner, 1985; Exner, Thomas, & Mason, 1985), there were only five cognitive special scores.
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 455

Table 2 ified for geographic distribution and partially stratified for socio-
A Comparison of Test-Retest Correlations Among Frequently economic level. The first of these nonpatient participants was
Used and Highly Regarded Tests collected before 1975, followed by various additions and revisions
through 1991 (Exner, 1993). All respondents were volunteers, with
Correlation the majority coming from places of employment. The gender
Tests variables Retest interval M Mdn distribution is balanced, but the sample is rather young; 73% of the
participants were under 36 years of age. Nonetheless, there are
Multiple studies some differences between this adult group and the 15- and 16-
Meta-analysis—Self-report, observer. year-old sample (Exner et al., 2001). The adult group is relatively
and projectivea 1 year .55
.52 — well-educated; the median grade level is over 13 years, and only
5 years
Meta-analysis—8 self-report tests b
.78 —
— 5% had less than 12 years of schooling. Eighteen percent of the
1 year
5 years .58 — respondents were non-Caucasian, which is consistent with other
Meta-analysis—MMPf 5 years .57 — normative samples, although this percentage could be increased.
MMPI—7 studies0 1 year .57 .58
Single studies For example, the minority percentage for the WAIS-III is 21%
MMPI" 1 day to 2 years .68 .68 (The Psychological Corporation, 1997) and for the MMPI-2 it is
MMPI-2 1 week* .80 .83 19% (Hathaway et al., 1989).
4 monthsf .73 .74 Rorschach researchers (Brinderson, 1995; Kates, 1994; Shaffer,
5 years8 .72 .73
.47
Erdberg, & Haroian, 1999; Viglione, Gaudiana, & Gowri, 1997;
California Verbal Learning Test 1.3 years" .43
1 year1 .53 .54 Weiner, 200 Ib) have questioned the continued suitability of these
Attributional Style Questionnaire1 5 weeks' .63 .64 CS norms. For reasons similar to the emergence of concerns about
Rorschach 3 years* .79 .80 the aging of MMPI norms (e.g., Colligan, Osborne, & Offord,
3 weeks1 .80 .85 1984), one might contend that social and cultural shifts, cohort
1 year1 .76 .82
3 weeks' .82 .85 differences, and the gradual evolution of the test justify collection
1 year1 .82 .83 of a new sample.
1 year"1 .75 .82
3 to 4 days .71" .74
3 to 4 days .66° .71 Research Data as Approximations of Nonpatient Sampling
Note. Dashes indicate measure was not applicable. MMPI = Minnesota
Multiphasic Personality Inventory.
To approximate a contemporary nonpatient reference sample,
1
Roberts & DelVecchio (2000). b Schuerger, Zarrella, & Holtz, we reviewed the literature for available adult control and nonpa-
(1989). °Mauger (1972) as cited in Dahlstrom, Welsh, & Dahlstrom tient groups. Shaffer et al. (1999) published the most comprehen-
(1975); Stone (1965) as cited in Dahlstrom et al. (1975); Sines, Silver, & sive and notable effort to produce normative approximation data.
Lucero (1961) as cited in Dahlstrom et al. (1975); Ryan, Dunn, & Paolo To construct Table 3, we selected central interpretive variables that
(1995); Milott, Lira, & Miller (1977). "Hunsley, Hanson, & Parker
(1988). e Butcher, Dahlstrom, Graham, et al. (1989). 'Putnam, Kurtz, appeared to deviate from CS normative expectations in the Shaffer
& Houts (1996). g Spiro, Butcher, Levenson, Aldwin, & Bosse (1993) as et al. study. We then supplemented the Shaffer et al. findings with
cited in Putnam et al. (19%). "Paolo, Troster, & Ryan (1997). 'Delis data from five other published samples known to us and one
et al. (1996) as cited in Paolo et al. j Peterson et al. (1982). kExner, previously presented sample (Viglione et al., 1997). These data
Armbruster, & Viglione (1978). 'Exner (1999). m Exner (1993).
" Mailer & Exner (1985), depressed inpatients. ° Haller & Exner (1985), allowed us to address some issues regarding deviation from CS
depressed inpatients instructed to give different answer to evaluate "mem- reference sample expectations.
ory effects." R. The data in Table 3 suggest that R is quite variable.
Whether or not such variability presents a problem for the Ror-
schach has been debated for many years. The distributional pa-
(N = 80 or more) should be collected. Interpretive temporal rameters (mean is greater than median, a large standard deviation,
consistency based on established interpretive cutoffs should be and a minimum value of 14 with no predetermined maximum
included. Systematically assigning multiple raters to recede value) suggest that a great deal of variability is due to a small
records would allow one to discriminate the effects of temporal proportion of records with excessively high R. From a clinical
inconsistency from interrater disagreement in accounting for utility perspective, long records consume administration and cod-
changes over time. ing time and increase cost without necessarily increasing interpre-
tive yield.
Lambda. The data for Lambda are incomplete, but suggest that
Normative Data and Diversity Issues
median values are close to CS reference values. The Shaffer et al.
The detailed presentation of large reference samples from non- (1999) study provides the only sample that seems to deviate from
patients, various patient groups, and children from ages 5 to 16 has the expected distribution. Nonetheless, on the basis of standard
contributed to a resurgence of the Rorschach over the past 20 years deviations, within each sample there are records that are much
(American Psychological Association, [APA] 1998; Exner, 1974, more form dominant than would be expected. The occasional high
1993; Exner et al., 2001; Wood & Lilienfeld, 1999). To limit our Lambda record does not pose much of a problem for routine
discussion of normative data, we focus here on the adult, nonpa- interpretation. On the other hand, low /f-high Lambda records
tient CS reference sample, consisting of 600 adults, which is a may lack negative predictive power and may reduce interpretive
relatively large normative sample (Exner et al., 2001). It is strat- yields and cost-benefit ratios.
456 VIGLIONE AND HILSENROTH

Table 3
Nonpatient Adults' Data for Some Rorschach Variables

Lambda" x-w WSVM6

N M Mdn SD M Mdn SD M SD M SD M Mdn SD

20" 245 245 93 075 061 051 055 015 020 0 15 90 38 123
20b 17.6 3.7 0.68 0.58 0.52 0.14 0.17 9.7 11.4
35C 26.0 7.3 0.83 0.57 0.52 0.13 0.21 2.8 3.8
25" 1.0 0.80 0.57 0.18
123e 20.8 18 75 1.2 075 1 72 0.51 0 15 021 0.11 66 40 80
35f 20.6 21 7.7 0.16 0.13 2.1 0.0 2.9
83s 22.8 21 7.3 0.58 0.45 0.46 0.55 0.14 0.17 0.17 7.9 5.0 9.3
Summary means across studies
Weighted 21.9 21.5 7.3 0.92 0.63 1.01 0.53 0.15 0.19 0.14 6.4 4.4 7.8
Not weighted 22.1 22.2 7.1 0.84 0.60 0.77 0.54 0.15 0.19 0.14 6.4 4.3 7.9
Exner et al. (2001) reference sample
21.9 22 4.4 0.60 0.53 0.31 0.77 0.09 0.07 0.05 4.5 5.0 1.8

"Netter & Viglione (1994), volunteers—no history of psychological hospitalizations or medication, deny
substance abuse. bFrueh & Kinder (1994), White, male college students. cGreenwald (1991), college
students. dSloan, Arsenault, Hilsenroth, Handler, & Harvill (1996), White, male, military reservists. "Shaf-
fer, Erdberg, & Haroian (1999). f Bums & Viglione (19%), female volunteers with superior interpersonal
relationships. g Viglione, Gaudiana, & Gowri (1997), nonpatient volunteers, without history of psychiatric
treatment, deny substance abuse. h Because of significant problems with skew for this ratio as the denominator,
Pure F, approaches zero, means and standard deviations may be misleading. The large standard deviations with
means greater than medians suggest that skew is present and that the means are artificially elevated by outliers
or skewed high values. ' Distributions are quite normal for X+% and X—%, so that only means and standard
deviations are presented.

Form quality. X+% and X—% results suggest that FQu and
FQ— are more numerous than anticipated, but they are consistent
with some previous observations (Viglione, 1989). Interpretive
ranges for X+%, as a measure of overall conventionality, and
X— %, as a measure of perceptual distortion, need to be modified.
The data in Table 3 may underestimate X+% and overestimate
X—% because one cannot be sure that pathological respondents
were excluded from the samples. Nevertheless, the results do
suggest that a more appropriate cutoff for the "unconventional"
interpretation of X+ % may be approximately X+ % < .50 and that
the cutoff for the "distortion" interpretation of X—% may be closer
to X-% > .25. The data suggest that the cutoffs for children
should also be more liberal (Hamel, Shaffer, & Erdberg, 2000;
Kates, 1994). With improvements in coding FQ, as suggested in
the interrater reliability section in this article, these interpretive
bands might tighten. It is also important to recognize that FQ has
some of the best validity data of any coding CS variable (Dawes,
1999; Viglione, 1999). Cases with elevations in the SCZI produced
solely by form quality should be closely examined to rule out
possible false-positive findings.2
Exner (2000) has made some changes in the form quality
percentages to address these issues. However, if it is true that 20%
of the responses in nonpatient records are distorted (i.e., FQ—),
then we have a problem with the operational definition for the
construct of perceptual distortion. One could justify 5% or 10% of
the responses being distorted, but it does not make sense for us to
call 20% of Rorschach responses among nonpatients distorted. We
may need to narrow the operational definition of FQ—, so that this
code truly represents distorted perception as outlined by Dawes
(1999).
WSUM6. For cognitive special scores in the form of WSUM6,
the large standard deviations suggest that a small proportion of
individuals provide many of the elevated values. Given that these
respondents are not patients (though it is possible that some had
been prior to testing), one cannot be sure whether these results are
associated with poor sampling, idiosyncratic responses, idiosyn-
cratic coding, or overly long, elaborated, and complex records. On
the other hand, central tendencies are consistent with the CS data.
The standard deviations for WSUM6, as well as R and Lambda,
suggest that these samples contain more exceptional or extreme
cases than does the CS reference group.
Cultural and International Implications
These normative approximation data also have implications for
cultural and international issues. Research data on the Rorschach
and diversity issues support the use of this measure with diverse
populations (Viglione, 1999). Large-scale cultural diversity and
international projects have been under way for some time, and
expertise among the project managers around the world is improv-
ing (Erdberg & Schaffer, 1999). These data have the potential of
teaching us a great deal about cultural and geographic issues. In
terms of thinking about cultural differences and expectations from
2
The SCZI is being replaced by a dimensional index of thought disor-
der, the Perceptual Thought Index (Exner, 2001). Such an index more
accurately reflects the available data (Hilsenroth, Fowler, & Padawer,
1998; Kleiger, 1999) and a modem understanding of thought disorder as
existing on a continuum from optimal to most disordered within and across
diagnoses (Viglione, 1999).
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 457

a normative perspective, the data emerging from these studies are 1974). The findings with R, and the possibility of related variabil-
more consistent with the normative approximation data given in ity among the other indices, suggest that constraining R through
Table 3 than they are with the CS reference sample. statistical comparisons or procedurally through minor administra-
Moreover, emerging childhood reference group data from tive modifications deserves some consideration.
United States samples (Hamel et al., 2000; Kates, 1994) and
international samples (Erdberg & Schaffer, 1999) are consistent
with a normative approximation sample of African American Meta-Analyses
children that was collected about 20 years ago (Krall et al., 1983).
Do meta-analytic results support the validity of the Rorschach in
Garb et al. (2001) cited the Krall et al. study in their challenge of
general terms? There are at least six recent and original meta-
the diversity applications of the Rorschach. However, this is quite
analyses that address Rorschach validity (Atkinson, Quarrington,
puzzling because a close examination of the data indicates that the
Alp, & Cyr, 1986; Bornstein, 1996,1999; Hiller, Rosenthal, Born-
only problematic difference between Krall et al.'s findings and the
stein, & Berry, 1999; Meyer & Handler, 1997; Parker, Hanson, &
norms available at that time (Exner & Weiner, 1982) is for a form
Hunsley, 1988). All concluded with empirically based statements,
quality score (i.e., a lower F+%), which is no longer being used in
supporting the validity of the Rorschach. Weiner (200la) summa-
the CS (Exner, 2000). Krall et al. reported a range for F+% of .48
rized these empirical data and offered them as convincing support
to .69, which is consistent with the normative approximation
of the overall validity of the Rorschach. Opposing these five
results presented in Table 3 and the emerging international data.
meta-analyses is one re-analysis of the Parker et al. data (1988) by
Also, differences in popular responses, for example, are as ex-
Garb, Florio, and Grove (1998). The research in this meta-analyses
pected given cultural variations (Mattlar, Carlsson, & Forsander,
was published from 1954 through 1982 and contains only 15
1993). Thus, supposed problematic cultural differences evaporate
Rorschach studies (compared with 75 utilizing the MMPI), only 5
when one considers the emerging international data. This conclu-
of which utilized the CS. As suggested in the original authors'
sion is supported by the very limited differences between nonpa-
(Parker, Hunsley, & Hanson, 1999) reply to the re-analysis of their
tient African American and Caucasian American adults matched
data, the relevance of the Garb et al. (1998) re-analysis is highly
on important demographic variables (Presley, Smith, Hilsenroth, &
questionable and adds little to our contemporary understanding of
Exner, in press). Finally, Meyer (in press) extended these findings
these two clinical instruments. The Rorschach research and tech-
by conducting a series of analyses to explore potential ethnic bias
niques have advanced quite a bit over the last 20 years. Likewise,
in Rorschach CS variables with a patient sample. After matching
the MMPI-2 was not available to be evaluated in either of these
on several salient demographic variables, ethnicity revealed no
two meta-analytic studies.
significant findings, and principal-components analyses revealed
Analysis of moderator variables in the Hiller et al. (1999)
no evidence of ethnic bias in the Rorschach's internal structure.
meta-analysis in this Special Series revealed that the MMPI had
larger validity coefficients than the Rorschach for self-report cri-
A New Normative Sample? teria and psychiatric diagnosis "whereas the Rorschach had higher
validity coefficients in studies using objective outcomes as crite-
The Shaffer et al. (1999) and other normative approximation
rion variables" (Hiller et al., 1999, p. 291). This is consistent with
data are relatively consistent with CS reference group data (Exner
Viglione's (1999) finding that the Rorschach variables were valid
et al., 1995). Adjusting one's expectations for form quality, as
and useful when one emphasizes ecologically valid behavioral
outlined above, should eliminate the possibility of overpathologiz-
criteria, rather than self-report criteria. Thus, as the criterion im-
ing adults.
proves, so do the Rorschach validity coefficients. In contrast, the
An alternative point of view about these reference data with
MMPI does best with criteria that share self-report method vari-
nonpatients is also emerging (Meyer, 2001; Weiner, 2001b). This
ance. Hiller et al. speculated that impression management plays
point of view suggests that there should be a careful consideration
less of a part in the Rorschach than in self-report testing. "In these
of the guidelines used for collecting this Rorschach reference data.
instances, it may be that the Rorschach is more objective than
Figure! in this understanding of differences is the recognition that
so-called objective instruments" (p. 291). Although Hiller,
Exner's (1993) reference sample consists of socially-vocationally
Rosenthal, and their colleagues' methods, data, and conclusions
functioning nonpatients. In addition, it is equally important to note
have been critiqued, the support for the Rorschach from their
that CS data collected from both United States and worldwide
work, as well as from all the other original meta-analyses, are quite
samples are somewhat more healthy than Exner's reference sam-
clear and accurate. As a whole, these meta-analyses reveal that the
ples of people who are starting outpatient psychotherapy or dif-
Rorschach produces valid data to measure many objective, behav-
ferent patient groups. The interested reader is directed to Weiner
ioral criteria.
(2001b), who proposed guidelines for research when obtaining
reference samples from patients and nonpatients. Meyer (2001)
also provided an extensive and in depth review of possible factors Incremental Validity
affecting normative shifts in Rorschach data. Both of these authors
presented rational and empirical data to support their position that Research data from a variety of sources reveals considerable
CS norms do not overpathologize potential examinees. emerging empirical support for the incremental validity of the
Nonetheless, collecting new reference samples, with increased Rorschach. This research is not exhaustive, but it is clear that
attention to cultural and other diversity issues, should improve Rorschach variables do possess incremental validity over other
reliability and validity, and enhance utility. Like the MMPI, the CS well-known tests and collateral information. As such, the evidence
and Rorschach have evolved since they were introduced (Exner, indicates that the Rorschach adds meaningful data to the assess-
458 VIGLIONE AND HILSENROTH

ment process. Accordingly, negative characterizations, such as
"The frequent claim that the Rorschach adds meaningful assess-
ment information to other data has not been supported in any
published study" (Hunsley & Bailey, 1999, p. 271), are false
because they ignore empirical findings.
Empirical Data That Address Incremental Validity
Research presented in Weiner's (2001a) and Viglione's (1999)
reviews of the literature contain empirical data consistent with the
conclusion that Rorschach variables possess incremental validity
over other tests, demographic data, and other information (see
Table 4). These data were not sufficiently recognized by those who
adopted a critical view of the Rorschach in this Special Series. On
the other hand, consistent with Weiner, Garb (1999) did confirm
incremental validity support to the EII, the Rorschach Oral De-
pendency scale, and the Rorschach Prognostic Rating scale (PRS).
Meyer (2000) has addressed incremental validity issues in two
meta-analyses. Within six studies, using both the Rorschach PRS
and the MMPI Ego Strength scale (Es), the Rorschach PRS dem-
onstrated considerable incremental validity for predicting treat-
ment outcome. The uncorrected and corrected effect sizes were
r = .32 and r = .48 (N = 229), respectively. The uncorrected and
corrected effect sizes, after excluding the outlier results from one

Table 4
Studies Included in Viglione (1999) With Findings Consistent With Rorschach Incremental Validity

Author and year Findings

Archer & Gordon (1988)
Archer & Krishnamurthy (1997)
Bornstein et al. (1997)
Cooper et al. (1991)
Hilsenroth et al. (1995)
Holzman et al. (1974)
O'Connell et al. (1989)
Perry & Braff (1994)
Perry & Viglione (1991)
Russ (1980)
Russ (1981)
Shapiro et al. (1990)
Skelton et al. (1995)
Optimal overall correct classification (OCC hit rate) of individual inpatients with schizophrenia for
Rorschach Schizophrenia (SCZI = 5) = .80. Optimal OCC for Minnesota Multiphasic Personality
Inventory (MMPI) Sc scale (Sc s 75) = .76. Utilizing traditional cutoff scores, OCC rates were as
follows: SCZI > 4, OCC = .69; Sc > 65, OCC = .48; and Sc > 70, OCC = .60.
In classification of depression in adolescents with a stepwise discriminant function analysis, Rorschach
variables Vista and Afr added R2 = .05 beyond combined R2 = .14 for MMPI-A scales A-DEP and Ma.
These four variables had highest positive predictive power over any single variable or combination of
variables.
Rorschach Dependency scores significantly correlated with both number of significant interpersonal events
(r = .84) and impact ratings of these events (r = —.38, p < .01). Self-report measure of dependency was
not significantly related to either (r = —.11 and r = .16, respectively).
The Rorschach Defense scales provided unique prediction of outcome GAP (Global Assessment of
Functioning, Health-Sickness) ratings in regression equations beyond initial GAP and borderline
personality self-report scale.
Rorschach variables were able to significantly differentiate (p = .008) those patients prematurely
terminating from psychotherapy vs. those continuing in treatment, whereas the MMPI-2 was unable to do
so (p = .56). Specifically Rorschach scores from the interpersonal-relational cluster had a mean effect
size (ES) of .57, while the MMPI-2 content scale Negative Treatment Indicators had an ES of -.14.
The classification of a recent schizophrenia diagnosis (hospitalized less than 6 months) and deviant eye
tracking was greater (65% accuracy) utilizing Rorschach data alone than a clinical team diagnosis (58%).
Rorschach data (Thought Disorder Index [TDI]) predicted the development of psychotic and psychotic-like
symptoms over a 2-3 year period in a sample of Axis II and affective disorder patients over and above
information from clinical interview on lifetime prevalence of psychotic and psychotic-like symptoms
(combined R2 = .21, TDI-beta = .32, p < .03), schizotypal symptoms (combined R2 = .42, TDI-beta =
.32, p < .009), or schizotypal and borderline symptoms (combined R2 = .51, TDI-beta = .31, p < .006).
Initial TDI scores also demonstrated clinical utility in prediction of psychotic and psychotic-like
symptoms at follow-up.
Human Experience variable component of the Ego Impairment Index (EH) significantly related to
neuropsychological markers of schizophrenia (r = -.42, p < .01; r = -.37, p < .025; r = -.35, p <
.025), whereas thought disorder scales based on clinical interview (Schedule for Positive Symptoms and
Schedule for Negative Symptoms) were not (p > .05).
Rorschach EH predicted outcome Beck Depression Inventory (BDI; p < .0002) and Carroll Rating Scale for
Depression (p < .01) scores in depressed patients treated with tricyclic antidepressants beyond variance
accounted for by gender and baseline scores on BDI and EH. Other demographic variables were also
considered but did not affect outcome.
Rorschach measures of adaptive regression (AR) and defensive effectiveness (DE) were significantly related
to academic achievement, independent of IQ (AR: r = .45, p < .01; DE: r = .40, p < .01, respectively).
Rorschach measure of AR was significantly related to reading and overall academic achievement
independent of IQ (r = .51, p < .001, and r = .47, p < .001). Index AR scores were significantly
predictive of reading achievement 1 year later (r = .29, p < .05).
Rorschach Depression Index significantly differentiated sexually abused African American girls from
controls (p < .005). Children's Depression Inventory scores for sample were not significantly different
from controls (p > .05), consistent with incremental validity of the Rorschach relative to self-reported
depression. The groups did differ on the Intemalization scale of the Child Behavior Checklist (p <
.0001).
The dependent variable was a ratio of Rorschach TDI over a TDI derived from the Wechsler. This ratio
was 2.46 times higher in a group of 25 identity-disordered adolescents than it was among 35 conduct-
disordered and oppositional-defiant adolescents (p < .01).
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE
study (Fiske, Cartwright, & Kirtner, 1964), were r = .40 and r =
.59 (N = 187), respectively. Effect sizes for the Es were essentially
nil (range, -.03 to 03). Clearly, the Rorschach PRS demonstrates
incremental validity over the MMPI Es scale. Nine studies (N =
358) included both the Rorschach PRS and an intelligence test.
Across these studies, the Rorschach PRS demonstrated consider-
able incremental validity over and above intelligence in the pre-
diction of future outcome. After correcting for methodological
artifacts, the unique contribution for the PRS was r = .48; and
without such corrections, r = .36. Blatt's (Blatt et al., 1994)
findings also demonstrate incremental validity of Rorschach scales
over IQ in the prediction of treatment outcome, as do a number of
studies summarized in Viglione's (1999) synthesis of the literature
(see Table 4).
Blais, Hilsenroth, Castlebury, Fowler, and Baity (2001) ex-
plored the incremental validity of the Rorschach in discriminating
Diagnostic and Statistical Manual of Mental Disorders (4th ed.;
DSM-IV; American Psychiatric Association, 1994) personality
disorders (PD). In a re-analysis of existing data, select Rorschach
variables and the MMPI-2, PD scales were used to predict chart
review ratings of DSM-IV antisocial, borderline, histrionic, and
narcissistic personality disorders in a sample of treatment-seeking
outpatients. Hierarchical regression analyses showed that both
MMPI-2 and Rorschach data added incrementally to the prediction
of DSM-IV borderline and narcissistic PD criteria. The findings
were less clear for the incremental value of Rorschach and
MMPI-2 data in predicting the total number of DSM-IV histrionic
PD and antisocial PD criteria. Rorschach data were the best pre-
dictors of DSM-IV histrionic criteria whereas MMPI-2 data were
best at predicting the DSM-IV antisocial criteria.
Dawes's (1999) Results Actually Support the Criterion
and Incremental Validity of the Ell and Rorschach
Earlier in this Rorschach Special Series, Dawes (1999) "pre-
sent[ed] two methods for assessing the incremental validity of a
particular Rorschach variable" (p. 297), namely the EH, with the
expressed purpose of demonstrating how incremental validity of
the Rorschach might be investigated. He investigated two data sets
from: (a) Meyer and Resnick (1996), which includes the Ror-
schach, MMPI, and impairment as approximated by severity de-
fined by diagnosis (DxMax); and (b) Perry, Moore, and Braff
(1995), which contains the Rorschach, measures of social compe-
tence and thought disorder, and another psychiatric rating scale.
His conclusions are quite critical of the Rorschach but do not fit
the data he presents. The analytic strategies he used reduced the
probability of finding support for the Rorschach and EII and biased
the analyses against the Rorschach. Therefore, we take issue with
his methods and conclusions in this article.
Reducing the power of the analyses. To re-evaluate these data
sets, Dawes (a) entered the number of Rorschach responses (R) and
his new form quality measure, XQUAL, before the EII in the
hierarchical regression; and (b) used a unit weighting scheme in
addition to multiple regression. The argument for using the number
of responses and his new measure, XQUAL, is that these Ror-
schach variables are easier to score and involve less projection by
the clinician.3 There is no established body of research that sup-
ports the number of responses to predict impairment; therefore,
there is no a priori reason to justify it as a predictor of impairment
459
in a regression analysis. Also, data from the study do not support
including R as a predictor The number of responses is not signif-
icantly correlated with the criteria (Meyer's DxMax, r = .13, ns
and in the wrong direction; and Perry's Social Competence Index,
r = .16, ns).
Similarly, Dawes's decision to create a new variable, XQUAL
and to require the EH to predict beyond XQUAL in a regression
equation cannot be justified. The EH contains form quality (FQ—)
and R among its subcomponents. XQUAL and R are, thus, struc-
turally redundant with EII and produce multicollinearity problems
within the regression, reducing the power of the analyses. Among
the arguments for the unit weighting system is that a clinician's
intuition does not reliably "capture optimal weighting systems" (p.
299). Its effect in the analysis is to reduce power once again.
Garb et al. (2001) argued that Dawes's (1999) and Meyer and
Resnick's (1996) use of the severity of clinical diagnosis as a
criterion is only a gross approximation of impairment or psycho-
logical disturbance. This is true, as it is for the Social Competence
Index criterion in the Perry, Moore, and Braff (1995) data set.
These measures do not share a great deal of variance with impair-
ment, the intended target construct of the EII. This limited overlap
between measure and target construct stacks the cards against the
Rorschach by limiting maximal effect sizes and the power of the
analyses. Small effect sizes are to be expected, even in the best
circumstances, with such a global criterion. Under such circum-
stances, the maximum multiple Rs of .49 and .35, reported by
Dawes, are quite impressive.
Dawes also introduced DxSum, the sum of the impairment
scores for up to three diagnoses, as a criterion measure.4 It was
used without precedent, supporting data, or a straightforward ra-
tionale. The DxSum makes the sum of mild diagnoses much more
impaired than the most virulent but pure schizophrenia or bipolar
disorder with psychotic features and adds systematic error to
DxMax. We obtained the Meyer and Resnick (1996) data set (N =
187) and found that 36 individuals in the data set had a diagnosis
with moderate severity, namely a DxMax score of 3 or less, but a
DxSum of 6 or more. For these 36 patients, even though they had
been assigned several diagnoses of mild to moderate severity, their
DxSum score made them appear more impaired than individuals
with a single diagnosis of schizophrenia (N = 16). In addition, the
data do not support Dawes's argument: All correlations in Dawes's
Table 1 between meaningful predictors (the MMPI mean eleva-
tion, the MMPI Goldberg index, XQUAL and EII) are higher for
DxMax than for Dawes's DxSum. Clearly, the DxSum variable
adds unacceptable error to the criterion and only functions to bias
the results against the Rorschach.
3
One may presume that some researchers and clinicians might disagree
with the statement that form quality distortions (FQ-) "involve neither
empirical research to establish their validity nor training in the Rorschach
to score" (Dawes, 1999, p. 297). However, it appears that Dawes accepts
that certain FQ— responses can be seen as clear, in vivo problematic
behaviors, consistent with understanding the Rorschach as behavioral
sample of cognitive problem-solving behaviors (Viglione, 1999).
4
Dawes (1999, p. 298) stated that "the authors considered the average
rating for up to three diagnoses and the sum of the ratings for up to three
diagnoses." Meyer (personal communication, March 6, 2000) calculated
the average and sum of diagnoses, but he did not recall considering them
to be valid criterion variables and did not include them in his analyses.
460 VIGLIONE AND HILSENROTH
With respect to the Perry et al. data set, the only relevant statistic
is the univariate correlation relating the EII to the Social Compe-
tence Index (r= .31, p < .05). Also, on the basis of the additional
measures included in the Perry data set, the EII was correlated with
the Schedule for Positive Symptoms global score (r = .367, p =
.007) and with a thought disorder subscale of the Brief Psychiatric
Rating scale (r = .334, p = .014). However, as expected, it was
not correlated with the Schedule for Negative Symptoms global
score (r = -.092, ns).
The Meyer data set allows us to address incremental validity.
For the reasons given above, the only defensible hierarchical
analytic strategy for this purpose is to add the EII and the other
Rorschach variables after the MMPI variables. These analyses are
presented in Table 5. The correlation between EII and DxMax is
.35. Given the limitations of DxMax as a measure of impairment,
the incremental Rs for the EII and the Rorschach are quite impres-
sive and may approach the maximum validity of DxMax as a
measure of impairment.
Conclusion regarding Dawes (1999). Thus, Dawes's (1999)
analyses support the incremental and criterion validity of the
Rorschach. The EII provided significant unique predictive power
over MMPI variables for psychological impairment or disturbance
as grossly estimated by the maximum severity of diagnosis. Ana-
lytic decisions by Dawes reduce the sensitivity of the analyses and,
as a result, minimize the potential to support the Rorschach.
Re-analyses of these data sets provide clear empirical support for
criterion validity and incremental validity of the EII and the
Rorschach.
Challenges to Incremental Validity of the Rorschach
Recent claims (Garb, 1999; Garb et al., 2001; Hunsley & Bailey,
1999) that the Rorschach has poor incremental validity are based
largely on old studies that confound clinical judgment with incre-
mental validity (Garb, 1994; see Garb, 1998, for a reproduction of
these studies), a point we return to later. These studies also do not
have the benefit of the CS scoring system and established inter-
pretive routines, in addition to lessons learned from clinical judg-
ment studies (Exner, 1993; Garb, 1994, 1998; Meyer et al., 1998).
Accordingly, these old experiments now lack ecological validity.
Table 5
Incremental Validity of Ego Impairment Index (EII) and
Rorschach From Dawes (1999) With Meyer and Resnick's
(1996) Data Set Predicting Ratings of Severity of Impairment of
Dx (DxMax)
Incremental r
Predictor R Change in R2 over MMPI only
MMPI .366 .134
MMPI + EII .424 .058 .241
MMPI + Rorschach .492 .108 .329
Note. The Minnesota Multiphasic Personality Inventory (MMPI) and Ror-
schach variables for analysis were those selected by Dawes: For the MMPI,
mean elevation clinical scales and Goldberg Index; and for the Rorschach,
EII, number of responses, and XQUAL.
Incremental Validity and Clinical Utility
The clinical utility of a test emerges within individual cases that
pose idiosyncratic questions and require idiographic decisions
(Kleinmuntz, 1990; Meehl, 1957; Meyer et al., 1998). Another
complicating factor is that the "yield" from other sources of data
and tests varies from case to case. For example, a Rorschach may
be too sparse to adequately address interpersonal perception or
thought disorder. In one case, the MMPI may be too exaggerated
to address depression. For these and other reasons, the usefulness
of a particular test varies greatly from case to case and question to
question. "Field" incremental validity, therefore, only manifests in
actual clinical practice in the form of clinical usefulness and the
impact of different tests in decision making within individual
cases. Weiner (1999) and Strieker and Gold (1999) provided
compelling rationales and case examples that clearly demonstrate
the clinical "field" incremental validity of the Rorschach in indi-
vidual cases.
Future Research
As with all personality measures, more research that demon-
strates incremental validity is needed. More sophisticated research
studies might address complexes of Rorschach variables with real
life behavioral and cognitive criteria. It would also be helpful to
look at response styles, response sets, and contextual moderator
variables that might influence incremental validity. In other words,
Rorschach variables may generalize best to contexts that resemble
the Rorschach problem, such as complex situations in which there
is little external guidance so that the individual has to make his or
her own way. Also, the adequacy of criterion variables and statis-
tical power present technical challenges to such research.
Two Applications of Special Interest to Practitioners
Thought Disorder and Implications for the Course of
Schizophrenic Spectrum Disorder
A variety of Rorschach summary scores have traditionally been
used to assess and conduct research on thought disorder. Early
research in this area used the Delta Index of Thought Disorder
from which the Thought Disorder Index (Johnston & Holzman,
1979) was developed. Both indexes share many features with CS
cognitive special scores and their weighted sum, the WSUM6.
There are abundant data that support the irrefutable conclusion that
these scales, as well as similar summary scores from the CS and
other systems, are related to thinking disturbance, psychoses, and
schizophrenic spectrum disorders (e.g., Acklin, 1999; Cadenhead,
Perry, & Braff, 1996; Exner, 1991, 1993; Hilsenroth, Fowler, &
Padawer, 1998; Kleiger, 1999; Perry & Braff, 1994; Perry, Geyer,
& Braff, 1999; Perry, Moore, & Braff, 1995; Viglione, 1999;
Weiner, 1966: Perry, Viglione, & Braff, 1992). In addition, re-
search summarized below also associates Rorschach measures
with schizophrenic spectrum disorders, genetic and environmental
risk factors, psychophysiological processes in schizophrenia, and
subsequent outcome.
Empirical findings. The Thought Disorder Index is associated
with multiple criteria related to biological aspects of schizophrenia
and psychosis proneness. Two studies associated the Thought
Disorder Index with the Chapman Psychosis Proneness Scales
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE
(Coleman, Levy, Lenzenweger, & Holzman, 1996; Edell & Chap-
man, 1979), a measure of magical thinking, perceptual aberration,
and social anhedonia (Chapman, Chapman, & Kwapil, 1995;
Chapman, Chapman, & Raulin, 1976,1978; Eckblad, Chapman, &
Chapman, 1982). In another study, similar Thought Disorder Index
elevations were produced by unaffected relatives of schizophrenics
(Shenton, Solovay, et al., 1989). The Thought Disorder Index also
predicted the development of psychotic and psychotic-like symp-
toms over a 2 to 3 year period in a sample of Axis II and affective
disorder adult patients over and above information from clinical
interviews (O'Connell, Cooper, Perry, & Hoke 1989). Initial
Thought Disorder Index scores also demonstrated clinical utility to
predict psychotic and psychotic-like symptoms at follow-up. The
Thought Disorder Index was about three times greater among
psychotic children and high-risk children compared with normal
children (Arboleda & Holzman, 1985). In this study, Thought
Disorder Index scores in nonpsychotic, hospitalized children were
not different from those of normal children. In a family adoptive
study, Wahlberg et al. (1997) found that genetic and environmental
factors interacted to produce elevated Thought Disorder Index
scores.
Empirical data also associate Rorschach measures of thought
disorder with more direct biological measures of brain functioning.
For example, Daniels, Shenton, et al. (1988) found that patients
with right hemisphere damage displayed fragmented thinking,
manic patients displayed playful thinking (as measured by sub-
components of the Thought Disorder Index), and schizophrenic
patients used idiosyncratic thinking. A series of studies (Perry &
Braff, in press; Perry & Braff, 1994; Perry et al., 1999) demon-
strates relationships between (a) the EII and its subcomponents and
(b) neurophysiological criterion measures of "gating." (Gating is
the ability to filter out irrelevant information in the environment,
and it is considered to be a deficit in the neural circuitry of
schizophrenia.) In one study, these information-processing crite-
rion variables and Rorschach responses were measured in a near-
simultaneous computer paradigm (Perry et al., 1999). Information-
processing deficits were not correlated with symptom rating scales
but were correlated with form quality minus (X— %) and cognitive
special scores (WSUM6). X—% independently accounted for 57%
(p < .001) of the variance in the neurophysiological measure and
contributed 35% (p < .01) of the unique variance beyond the
clinician-rated Schedule for Positive Symptoms Delusion subscale
score. The Schedule for Positive Symptoms Delusion scale was
selected among five symptom scales, because it had the greatest
correlation with the neurophysiological measure. Thus, X—%
demonstrated incremental validity over a variety of symptom
scales. In this study the EII was not calculated because of the
restriction of the number of responses, due to the demands of the
computer-generated paradigm. Nevertheless, these results support
the hypothesis that the EII and its subcomponents tap into the
cognitive dysfunction that is observed among schizophrenia
patients.
Challenges, conclusions, and future research. Garb et al.
(2001) took exception to Viglione's (1999) claim that Rorschach
variables can be helpful in determining biological vulnerability for
schizophrenia and that these variables have been linked to biolog-
ical processes associated with schizophrenia. To do so, they iso-
lated two individual studies from the mass of findings in the
literature. The research with various Rorschach measures of
461
thought disorder, with and outside of the CS, clearly reveals that
the Rorschach can produce data related to the biological aspects of
thought disorder and psychosis proneness. Although the Thought
Disorder Index administration typically differs from the CS ad-
ministration and conclusive data need to be collected in regard to
the impact of these administrative differences, similar findings
across the many Rorschach summary scores of thought disorder
suggest that these positive findings should generalize to the EII
and other CS-based variables measuring disordered thinking. Of
course, research is needed to confirm this logical expectation.
In addition, further research should be conducted to specify the
association between the Thought Disorder Index and CS summary
scores of thought disturbance, using protocols collected according
to CS administrative procedures. Other research goals might be to
conduct prospective, longitudinal studies to confirm that these
Rorschach variables can predict (a) the emergence or return of
psychoses, (b) negative outcomes in the course of schizophrenic
spectrum disorders, and (c) the development of these disorders
among high-risk children. It would be useful to know whether
these associations are moderated by Rorschach variables and col-
lateral non-Rorschach information. For example, simple or preser-
vative records may be associated with false-negative Rorschach
thought disorder findings, given that the perseveration evident in
Rorschach records is poorly captured by existing scores. Improve-
ments in the measurement of thought disorder within the CS are
possible (e.g., Kleiger, 1999; Perry, Potterat, Auslander, Kaplan, &
Jeste, 1996), including the possibility of incorporating aspects of
the Thought Disorder Index, adding the EII to the CS, developing
more meaningful measures of perseveration, and refining proce-
dures for coding ALOG and DR so that they more selectively
identify problem-solving failures. This research should also ad-
dress the sensitivity of Rorschach variables to problem-solving
limitations in relatively well-functioning individuals.
Suicide
The CS S-CON is an actuarial variable that yields a single score
to assess suicide risk from a composite of 12 variables and ratios.
On the basis of all the available data and the cost-benefit of
false-positive versus false-negative errors, continued use of the
S-CON is in order (Viglione, 1999). It should not be used to rule
out suicide risk, but it can be used to increase awareness about
self-destructive behavior and suicide.
The Exner and Wylie (1977) study still qualifies as one of the
more methodologically rigorous studies of suicide prediction. In
the original study of 59 completed adult suicides, the authors found
a total S-CON score of 8 positive indices correctly identified 75%
of the patients who subsequently died as a result of their suicide
attempt. Fowler, Piers, Hilsenroth, Hold wick, and Padawer (2001)
examined the predictive relationship between the CS S-CON and
lethality of suicide attempts within 60 days following administra-
tion of the Rorschach. This study uses an ecologically valid and
compelling behavioral criterion. Patient records were reliably rated
as nonsuicidal (n = 37), parasuicidal (n = 37), or near lethal (n =
30), based on the presence and lethality of self-destructive acts.
The S-CON differentiated patients who would later engage in
near-lethal suicide attempts from those who would engage in
parasuicidal behavior and from those who did not engage in
self-destructive behaviors. An S-CON score of 7 or more success-
462 VIGLIONE AND HILSENROTH
fully predicted which patients would engage in near lethal suicidal
activity relative to parasuicidal patients (overall correct classifica-
tion [OCC] rate = .79), nonsuicidal inpatients (OCC = .79), and
college students (OCC = .89). Logistic regression analysis re-
vealed that an S-CON score of 7 or more was the sole predictor
(p < .01) of emergency medical transfers due to suicide attempts
from among nine demographic, IQ, psychiatric, and Rorschach
predictors. Some of these predictors included clinician-rated global
psychiatric severity, GAP scores, and the presence of DSM-IV
Axis I and II disorders. Thus, this study demonstrates the incre-
mental validity of this Rorschach scale beyond clinician interview
and clinician ratings. The total S-CON score also predicted which
patients would have a subsequent drug overdose (r = .32, p =
.001) within 60 days of the Rorschach administration.
To challenge the S-CON, Garb et al. (2001) focused on two
studies that used postattempt suicide criteria (Meyer, 1993; Silberg
& Armstrong, 1992) and one presentation that has been described
only in reviews by Eyman and Eyman (1987, 1991, 1992). How-
ever, direct examination of these reviews reveals empirical support
for the efficacy for two single sign suicide predictor variables—
transparency responses and color-shading blends. These scores are
incorporated into shading, form dimension, vista, and color-
shading subcomponents of the S-CON.5 The Rorschachs in the
Eyman and Eyman study were administered in the Rapaport-
Schafer system (Rapaport, Gill, & Schafer, 1946; Schafer, 1954)
but recoded in the CS. The Rapaport-Schafer system does not
contain all the same variables as the CS so that inquiry questions
do not elicit all relevant information. Translation of these protocols
to CS variables on an actuarial basis with specific cutoffs may not
be justified without further research.
As a whole, the empirical data support the continued use of the
S-CON. The decision-making utility of the S-CON may not have
been exhaustively evaluated, but based on substantial positive
results one cannot ignore the S-CON data if the false-negative
error means losing a life. The most enlightened clinical interpre-
tation of an elevated S-CON is to assert that the Rorschach
indicates additional suicide risk that might not be recognized
through other means. Practice ethics and an S-CON of 7, 8, or
more should lead clinicians to consider suicidal risk and to collect
additional information about self-destructive risk. Of course, the
S-CON may produce false positives in some contexts because of
the low base rates of actual suicide (Finn & Kamphuis, 1995). In
real-life clinical decision making, which takes into consideration a
wide range of contextual and collateral information, the cost of this
false-positive error is minimal compared with the cost of false-
negative error (i.e., missing an imminent suicide).
Clinician Judgment
Ecologically Valid Clinical-Actuarial Judgment
The goal of a large portion of the decisions in formulating the
administration procedures of the CS was to minimize examiner
and contextual effects identified in earlier research (Exner, 1974).
Interrater reliability and temporal consistency data summarized
earlier in this article, as well as other research data (e.g., Brinder-
son, 1995; Cheyette, 1992; Perry, Sprock, Schaible, McDougall, et
al., 1995; Viglione & Exner, 1983), suggest that these problems
have been minimized. Similarly, a large portion of the interpretive
tactics and routines, particularly the interpretive search routines,
were developed to minimize problems identified in clinical judg-
ment studies.
The current CS combination of actuarial and clinical methods
(Exner, 1991, 1993; Strieker & Gold, 1999; Weiner, 1999) is
consistent with thinking in the field of judgment and decision
making (e.g., Garb, 1989). This clinical-actuarial method uses
procedures whose goal is to minimize cognitive judgment errors
and distortions.6 These procedures are probably ideal in addressing
unique questions in clinical contexts where no two cases are alike.
In contrast to the dated clinician judgment analogue research, in
which the participant clinician interprets a limited, predefined, and
idiosyncratic set of data, the clinician in the field uses data to
develop, to reformulate, and to refine referral questions. The
clinician also uses expertise to gather or to select accurate data
relevant to those questions. In doing so, the examiner constantly
interacts with emerging data sets, to change existing questions,
probabilities, and base rates (Meyer et al., 1998). Along the way,
the examiner brings extreme base rates nearer to levels in which
testing data would be more efficient in decision making. In that
sense, the single event, low base analogy found in the literature, for
example, "Is this client thought disordered?" as addressed by
Hunsley and Bailey (1999, p. 272), is an oversimplified analogue
to clinical decision making in the field.
In actual clinical practice, decisions with low base rate phenom-
ena rest on the consideration and synthesis of many small data
points. An example of such a situation may be the evaluation of a
schizophrenic-like thought disorder in outpatient or forensic con-
texts. Synthesizing such data as history, symptoms, observations of
thought disorder, and related characteristics can eventually lead to
optimal base rates closer to 50% (Hunsley & Bailey, 1999). In
other words, in a patient without the history and observations that
might lead to a sufficiently high probability of diagnosing a
schizophrenic-like thought disorder, the clinician with positive
evidence of confused thinking on the Rorschach might search for
historical or behavioral information consistent with (a) excessively
elaborated, idiosyncratic, and overly abstract thinking (Franklin &
Cornell, 1997; Gallucci, 1989) or (b) thinking problems associated
with intrusions of traumatic imagery (Holaday & Whittenberg,
1994; Sloan, Arsenault, Hilsenroth, Harvill, & Handler, 1995).
These are alternative conditions that may lead to indices of con-
fused thinking on the Rorschach that have been supported by
research.
Such a clinical-actuarial model is consistent with the threshold
model described by Rogers (1997), which mitigates errors in
evaluating malingering. It also explains why configurational, syn-
thetic approaches, as described by Strieker and Gold (1999) and as
demonstrated by Weiner (1999), are fundamental to the clinical-
actuarial method. In this form, this method minimizes the possi-
bility of a false-positive actuarial interpretation. This clinical-
5
The Eyman and Eyman study was not included in the journals identi-
fied by Meyer (1999) and was not published as a research report. Thus,
Viglione (1999) did not consider it according to the predetermined, objec-
tive inclusion criteria established for his review.
6
There is no a priori reason that this actuarial-clinical—nomothetic-
idiographic approach would have to be confined to CS variables. Other
variables with a sound research basis can certainly be added.
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE
actuarial method with the Rorschach uses an ecologically valid and
informed understanding of interactive probabilities to increase
accuracy of in vivo decision making in applied psychology.
Meehl and Rosen (1955) addressed this issue in the form of the
"successive-hurdles" approach. Their example considered separate
tests for a brain tumor decision, and they described an increase in
hit rates. Meehl and Rosen reported small relative increases in hit
rates over base rates and large relative drops in false-negative
errors with this procedure. For example, successive tests with (a)
90% true positive and 15% false positive for Test 1, followed by
(b) Test 2 with 80% true positive and 10% false positive would
increase hit rates from 90% to 96%. In this example, overall errors
are cut by more than half, and false-negative errors are reduced by
more than three times, from 10% to 3%. The improvement in
clinical utility is a function of the cost of these false-positive and
false-negative errors. In psychological evaluation, this process
contains many more interactive steps. Thus, illness course, symp-
tom report, Rorschach scores for thought disorder, observations of
thought-disordered behavior in interview, and other Rorschach
variables and observations come together. On the basis of each of
these "tests," the examiner chooses to observe more data to con-
tinue to test the hypotheses. Both a yes-no decision and an
idiographic, qualitative description of the problem emerge in the
typical clinical or forensic Rorschach application.
The clinical-actuarial judgment method is further complicated
when one adds the strengths, weaknesses, and predilections of the
individual examiner (Strieker & Gold, 1999). Along these lines,
and in response to criticisms by Wood, Lilienfeld, Garb, and
Nezworski (2000), Garfield (2000) emphasized the skill of the
clinician as an important determinant when using the Rorschach in
applied practice. To study these issues, more creative and ecolog-
ically valid ways of studying the clinician and improving in vivo
applied judgment need to be developed both within the quantita-
tive tradition, but also in terms of qualitative research (e.g.,
Schaich, 1999). This approach is consistent with the APA Psycho-
logical Assessment Work Group report (Meyer et al., 1998.)7
Clinical Judgment: Future Research
Further research on clinical judgment is needed not only for the
Rorschach but also for other types of decision making. It might be
illuminating to conduct new studies with the CS and non-CS
variables, using clinicians who are aware of errors and pitfalls in
clinical judgment. As Garb (1998) stated, "most of the judgment
studies on the use of projective tests are over 20 years old" (p.
232), with virtually no studies with the modern CS and other
Rorschach variables. Such analogue clinical judgment research
would have to meet ecological validity challenges, namely, how to
reproduce the clinical-actuarial method in the lab, how to motivate
clinicians and enforce the real life cost-benefit ratios for accurate
versus inaccurate judgments, and how to make the questions and
decisions sufficiently idiographic, meaningful, and malleable. De-
signs must also incorporate better criterion measures. Reproducing
the cogwheeling interactive approach (Meyer et al., 1998), vicar-
ious functioning of the respondent and interpreter in facing un-
bounded problems (Hammond, 1955; Levy, 1963) in an ecologi-
cally valid way is critically important.
Lack of ecological validity, poor criteria, and the failure to
motivate clinicians all reduce potential effect sizes. If incremental
463
validity is to be addressed quantitatively, an additional challenge is
power. Sample sizes must be very large or multiple clinical deci-
sions must be evaluated to be able to rule out false-negative results.
Utility, Cost-Benefit, and the Rorschach
The Rorschach allows us to study samples of behavior collected
under similar conditions in different native languages around the
world (Erdberg & Schaffer, 1999). There is an efficiency to
sampling behaviors with a single technique to develop one's
understanding and appreciation of developmental changes across
the age span and across all types of disorders and problems.
To make cost-benefit comparisons ecologically valid, one would
have to envision a full range of costs and benefits equivalent to the
Rorschach. The alternative to the Rorschach would have to address
a wide range of clinical, personality, forensic, developmental,
cognitive, and neuropsychological constructs and applications
across the life span and around the world. The Rorschach allows
one to develop a common, experientially based database for the
problem-solving practices and predilections of clients from age 5
through old age.8 One could identify many other single purpose
scales within a specific content-construct area that might compete
well with the Rorschach, but to produce the same benefits, one
would have to master and monitor developments in perhaps 50
alternative instruments. Another cost is the expense and hassle of
purchasing kits, test blanks, and computer programs and paying
per use fees. Also, the Rorschach provides an efficient way to
collect a behavior sample outside of interview behavior. Psychol-
7
In addressing clinical utility, Garb et al. (2001) presented a factually
incorrect and misleading picture of Viglione's (1999) review of the liter-
ature on clinician judgment: "Half of the studies cited by Viglione were
published in the 1960s and 1970s, with the remainder of the studies
published in the 1980s" (p. 437). Viglione cited five research reports,
which met a priori inclusion and exclusion criteria, providing evidence that
clinicians can make valid and useful judgments from Rorschach data
(Bilett, Jones, & Whitaker, 1982; Cerney, 1984; Dana & Back, 1983;
Dudek & Marchand, 1983; Holzman et al., 1974). Viglione cited three
additional works, not as research reports, but merely as descriptions for
scales used in the five cited studies: (a) Urist (1977) for the Mutuality of
Autonomy Scale, (b) Johnston and Holzman (1979) for the Thought
Disorder Index, and (c) Masling, Rabid, and Blondheim (1967) for the
Rorschach Oral Dependency Scale. Garb et al. (2001) argued that Viglione
(1999) consistently ignored negative findings and then listed seven studies
supposedly ignored by Viglione (Albert, Fox, & Kahn, 1980; Bilett et al.,
1982; Cochrane, 1972; Gadol, 1969; Golden, 1964; Perez, 1976; Turner,
1966). Only the first two of the studies were among the 445 studies selected
for review by the editor of this Special Series. These studies addressed the
incremental validity of judgments and the relationship between the validity
of judgments and confidence or training, rather than simply the issue of
judgment. Garb et al. (2001) also asserted that "in fact, positive results have
never been obtained for the Rorschach in studies on clinical judgment and
[italics added] incremental validity, confidence and [italics added] validity,
and training and [italics added] validity" (p. 437). To clarify this statement,
no single Rorschach study simultaneously offers (a) a demonstration of
valid clinician judgment with incremental validity techniques, (b) a dem-
onstration of an association between judges' avowed confidence and ac-
curacy of judgments, and (c) a demonstration of an association between
judges' training and accuracy of judgment.
8
The Rorschach has applications below age 5 for testing developmental
issues (Leichtman, 1996).
464 VIGLIONE AND HILSENROTH
ogists' time is valuable, but so is the client's time; therefore, it is
another cost to be entered into cost-benefit analysis. Mastering one
test, the Rorschach, is efficient because it eliminates the need for
many content specific tests. As a result, we can pick and choose
more judiciously and master select instruments within an assess-
ment battery, rather than misusing a large number of tests for all
the potential purposes that the Rorschach addresses.
Use of the Rorschach: Past, Present, and Future
Training, Research, and Clinical Use
Repeated surveys of psychological test use over the past 40
years have shown a substantial, consistent, and sustained use of the
Rorschach in academic training, research, and clinical settings.
Surveys consistently indicate that over 80% of graduate programs
teach the Rorschach and that students regard this training as
important in developing other clinical skills and in understanding
their patients better, and as useful in their practicum-intemship
training (e.g., since 1995, Camara, Nathan, & Puente, 2000;
Hilsenroth & Handler, 1995; Watkins, Campbell, Nieberding, &
Hallmark, 1995). Although some surveys report that the Rorschach
takes more of the clinician's time than many other tests, they also
report that it is one of the most frequently used (e.g., Camara et al.,
2000).9
Also, 90% of clinical practitioners working in the field ex-
pressed a belief that clinical students should be competent in
Rorschach assessment (Watkins et al., 1995). This attitude was
shared by internship directors (Durand, Blanchard, & Mindell,
1988). In a recent but relatively small survey of 84 (19%) of the
445 APA-approved internship programs in 1997 (Piotrowski &
Belter, 1999), internship directors reported that the MMPI-2/
MMPI-A, WAIS-R/WAIS-III, and Rorschach were the top three
assessment instruments used in internship programs and were
considered essential for practicing psychologists. Twenty-seven
training directors reported decreased emphasis on projective as-
sessment, whereas 12 reported an increased emphasis. Also, hours
of seminar time required by internship programs were virtually
identical for projective and self-report assessment measures. Fi-
nally, the Rorschach was used more than all five of the behavioral
assessment instruments listed by Piotrowski and Belter (1999). A
recent survey of practitioners' test use with adolescents suggests
that testing has declined because of managed care constraints, but
that the Rorschach has retained its status as the second most used
instrument (Archer & Newsom, 2000).
The two most comprehensive recent surveys of predoctoral
internships, one including 329 (Stedman, Hatch, & Schoenfeld,
2000) and another including 324 (Clemence & Handler, 2001)
Association of Psychology Postdoctoral and Internship Centers
sites (most of which were programs accredited by the APA),
revealed that internship training directors greatly value the Ror-
schach as well as integrated test batteries. Again, training directors
reported a desire for incoming interns to have had courses or a
good working knowledge of the Rorschach. Academic depart-
ments that are making decisions about the place of the Rorschach
in their curriculum should closely attend to these data.
The Rorschach is the second most frequently researched per-
sonality assessment instrument. This consistently high rate of
empirical investigation led the authors of a recent review to state
the following: "Whether viewed from the perspective of research
attention or practical usage, the Rorschach inkblot technique con-
tinues to be among the most popular personality assessment meth-
ods and predictions about the technique's demise would appear
both unwarranted and unrealistic" (Butcher & Rouse, 1996, p. 91).
Forensic Use
Weiner, Exner, and Schiara (1996) reviewed responses from 93
practioners who testified in over 4,000 criminal cases, over 3,000
custody cases, and 858 personal injury cases, for a total of almost
8,000 cases. In only 6 of these cases (0.08%) did these respondents
report that the integrity of the test was challenged and in only 1
case (0.01%) was it ruled inadmissible. Although Weiner's sam-
pling technique may have limitations, these data provide a "good
empirical basis for concluding that, contrary to whatever opinion
may be voiced, the Rorschach is welcome in the courtroom" (pp.
423-424).
To address the question of the legal weight, or the effect the
Rorschach has on court judgments, Meloy, Hansen, and Weiner
(1997) reviewed court of appeals citations from 1945 to 1995. In
90% of the 194 cases, "the admissibility and weight of Rorschach
data were not questioned by either the appellant or the respondent
and were important enough to be mentioned and discussed in the
legal ruling by the court of appeal" (p. 60). Typically when
questions arose, the practitioner's interpretation or findings de-
rived from the test, rather than the test itself, were challenged. In
only two cases (0.8%) was the test criticized as a psychological
measure. These data demonstrate that the Rorschach "has author-
ity, or weight, in higher courts of appeal in the United States" (p.
61). hi a systematic, comprehensive legal analysis of the admissi-
bility of the Rorschach, McCann (1998) agreed with the courts. He
concluded that a focus on the structural summary from the CS
"meets professional and legal standards for admissibility of psy-
chometric evidence and expert testimony" (p. 140).
Challenges
In an earlier article in this Special Series, Dawes (1999) referred
to the "deficiency" (p. 301) of the Rorschach in relation to the
criterion of reasonable certainty used in forensic applications. His
argument, concerning the forensic use of the Rorschach, appears to
be based on some elusive juxtaposition of incremental validity
statistical analysis with the forensic notion that expert testimony
should be "incremental" in the forensic sense of aiding the trier of
facts. Contrary to his argument and his own data, the evidence
from the courts suggests that more direct self-report assessment
methods may not be as valuable in forensic adversarial evaluation
contexts; therefore, the incremental value of the Rorschach in
terms of assisting the trier of facts increases. In fact, Dawes's own
findings indicated that whatever deficiency the Rorschach pos-
sessed, the MMPI also possessed because the Rorschach improved
on predictions made by the MMPI.
Given these data, the recent task force report by Division 12
(Clinical) of the APA (1999), concerning the development of a
9
Citations before 1995 that support the assertions in this section are
excluded for the sake of brevity.
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE
model assessment curriculum for graduate training in assessment,
is confusing and misguided. In this report, Rorschach training was
ranked 95th in importance out of the total 105 topics. The available
reliability and validity data, coupled with other contemporary
survey material reported earlier, suggest that there is vast discrep-
ancy between the Division 12 Task Force future model curriculum
for assessment and actual graduate training needs. Should these
changes in graduate assessment training occur, students and future
clients will have to pay for the misconceptions of graduate faculty.
Also Hunsley and Bailey (1999) cited a dated survey (Wade &
Baker, 1977) of practicing clinicians to claim that practitioners
disregard scientific evidence. Actually, there is empirical evidence
to show quite the contrary. Beutler, Williams, Wakefield, and
Entwistle (1995) recently found that clinicians in private practice
value research and put forth effort to understand scientific find-
ings. There is no empirical evidence that indicates that the manner
in which psychologists typically use the Rorschach is different
from the standardized methods in which they were trained, and no
evidence that psychologists who are using the Rorschach are
unaware of its research findings.
Conclusion
The data on the use and perceived value of the Rorschach
clearly demonstrate that psychologists find the Rorschach to be of
value in routine clinical practice. Some psychologists ask why so
many practitioners persist in their use of the Rorschach despite
current time pressures and limited fees by third party payers. This
becomes even more perplexing when self-report questionnaires
and structured interviews may require less of the clinician's time.
Considering the persistent use of the Rorschach in the context of
the available reliability and validity information leads to some
obvious conclusions: One must speculate that clinicians and those
in charge of clinical training find it helpful because of its reliability
and validity relevant to challenges in applied settings. In other
words, clinicians continue to use the test despite its time require-
ments because of its clinical utility.
A substantial body of research demonstrates that individuals
exhibit a defensive bias and manipulate their responses when
asked to self-report aspects of their personality or psychopathol-
ogy. For example, individual differences in self-serving biases
have been demonstrated in relation to attributions, self-
descriptions, inferences, personality traits, and avoidance of neg-
ative affect. Rorschach research supports the view that the Ror-
schach is most useful in (a) contexts in which the respondent may
be unwilling or unable to self-report problems and (b) those in
which clinicians are trying to predict real-life behavior (Hiller et
al., 1999; Viglione, 1999). In addition to routine clinical contexts,
the data suggest that the Rorschach may be especially useful under
conditions that might induce response manipulation in self-report.
These might include employment, forensic, and custody evalua-
tions, and other settings that involve adversarial examiner-
respondent relationship components (Rogers, 1997).
Methodological Challenges to Validity of the Rorschach
Challenges
Rorschach researchers (Exner, 1995; Viglione, 1997) have of-
fered recommendations to improve Rorschach research and, in
465
turn, improve the validity and utility of the test. Others have also
repeatedly offered methodological recommendations and stan-
dards for Rorschach research, and they have used these standards
to reject findings from individual studies supportive of the Ror-
schach (Garb, 1999; Garb et al., 2001; Hunsley & Bailey, 1999;
Wood et al., 2000; Wood, Nezworski, & Stejskal, 1996a, 1996b).
These same standards are not fully applied when evaluating studies
cited as yielding negative evidence for the Rorschach. Also, im-
partiality would require applying the same standards to all tests,
rather than reserving them specifically for the Rorschach (Garfield,
2000). In the first set of publications for this Special Series in
Psychological Assessment, Viglione (1999) met a number of the
challenges posed by Rorschach critics. Nevertheless, the positive
research findings summarized by Viglione (1999) were met with
the same repeated methodological criticisms (Garb et al., 2001).
Similarly, other research reports (e.g., Meyer, 1997a; Hilsenroth,
Fowler, Padawer, & Handler, 1997) have not been recognized as
positive responses to criticisms. In the section below, we examine
Garb, Wood, and their colleagues' applications of these criticisms
about methodology in Rorschach research.
Within and Across Diagnoses
Viglione's (1999) synthesis of the empirical research supports
the conclusion that "the Rorschach is useful in examining many
clinically relevant criteria and behavior within and across diag-
noses" (p. 260). However, in reaction to Strieker and Gold's
(1999) summary of the literature, endorsing the usefulness of the
Rorschach in evaluating depressed, schizophrenic, or acutely sui-
cidal patients, Garb et al. (2001) asserted that "empirical research
suggests that the Rorschach is not useful for all these tasks" (p.
433) and cited only Wood et al. (1996a, 1996b) to bolster this
statement. This claim overlooks a great deal of evidence in support
of the usefulness of the Rorschach in these disorders and associ-
ated phenomena. There are numerous examples of research that
support the validity of the Rorschach applied to various problems
within and across diagnoses (e.g., see Viglione, 1999).
Empirical Data in Peer-Reviewed Journals, Accessible to
All
Some critics have maintained that too much of the empirical
support for the Rorschach contained within studies in Exner's
volumes is not published in peer-reviewed journals and is therefore
not available for review, In response to this criticism, Viglione
(1999) synthesized research findings from peer-reviewed journals
only. He reported considerable empirical support for the test.
Viglione focused on the Methods and Results sections, rather than
on the theorizing of the authors. Thus, a careful review of the
empirical data published in peer-reviewed journals supports the
utility of the Rorschach.
Selection of Studies
Rorschach critics (e.g., Garb et al., 2001; Wood et al., 2000;
Wood, Nezworski, Stejskal, Garven, & West, 1999) have claimed
that Rorschach researchers selectively present findings. Meyer
(1999) requested that the contributors of this Special Series ad-
dress 445 articles in five targeted journals published between 1977
466 VIGLIONE AND HILSENROTH
and 1997. In response, Viglione (1999) applied objective, meta-
analytic type selection criteria to select 128 research reports. In
contrast, Hunsley and Bailey (1999) confined their review to 28
studies, whereas Garb et al. (2001) reviewed only 44 of these same
445 articles. In addition, Viglione's selection criteria resulted in
the exclusion of supportive Rorschach evidence found not only in
Exner's texts, but also in review articles, meta-analyses, and books
(e.g., Blatt et al., 1994; Exner, 1993; Gacono & Meloy, 1994;
Kleiger, 1999; Masling, 1986; Weiner, 1966). Most of the negative
studies selected by Garb, which Viglione was said to have ignored,
were not among those identified by the series editor. On the other
hand, Garb et al. uncovered virtually no research supportive of the
Rorschach, even though they were provided with the same 445
target articles as the other series contributors. Viglione included
negative findings when they occurred.10
Experiment-Wise Alpha Levels
Garb et al. (2001) maintained that the Rorschach literature fails
to control experiment-wise alpha levels. For example, Garb et al.
(2001) viewed Lemer and St. Peter's (1984a, 1984b) positive
findings about borderline personality structure and Rorschach hu-
man representations as a flawed study because of the failure to
control for false-positive findings. Garb et al. (2001) did not report
that 41 of 64 analyses were significant at p < .05, and 33 of 64
were significant at p < .01.n Mathematically, these positive
results cannot be explained by chance. All other analyses were
legitimate, post hoc group contrasts for the 41 significant
comparisons.
Criterion Contamination and Diagnosis
Critics have faulted Rorschach researchers for using clinician
diagnosis as a criterion (Garb et al., 2001; Wood et al., 2000). For
example, Garb (1998) questioned the use of clinicians as criterion
judges: "Historically, the clinicians making the indicator ratings
have been called criterion judges and their ratings criterion ratings,
but these terms are inaccurate because the ratings are fallible" (p.
14). Nonetheless, these same critics have emphasized studies by
Archer and his colleagues (Archer & Gordon, 1988; Archer &
Krishnamurthy, 1997) to argue against the incremental validity of
the Rorschach. The Archer and Krishnamurthy (1997) study used
the clinical diagnosis from just one clinician as the sole basis for
the diagnosis. Moreover, Archer and Gordon (1988) noted that
"psychological test results were frequently available and poten-
tially involved in the diagnostic decisions" (p. 279) so that crite-
rion contamination and bias are possible. Both studies lack reli-
ability statistics for their diagnostic criteria. These studies do not
meet the methodological standards imposed by the critics on
studies with evidence supportive of the Rorschach. Yet, these
studies are cited as negative evidence for the validity of the
Rorschach. This is clearly evidence of a double standard used by
the critics of Rorschach research.
Implications and Conclusions
We have addressed the Rorschach's strengths and weaknesses
and offered research recommendations in 10 focal areas relevant to
the clinical utility of the test. In addressing these issues, it is our
intention to give readers an informed appreciation for the substan-
tial but often overlooked research basis for the reliability, validity,
and utility of the Rorschach. From the broadest point of view, we
have demonstrated that examining the empirical findings can only
lead to the conclusion that the Rorschach is a valuable test (Vigli-
one, 1999; Weiner, 1996). Interrater reliability by well-trained
coders both in the field and in the lab are very strong. Temporal
consistency data are good, as are the currently available normative
data, though both could be improved. Criterion validity for the test
is very strong, and incremental validity and meta-analytic data are
present with more data emerging. The current sweeping criticisms
of the Rorschach are largely without merit, uninformed, and biased
against the test. Thus, the Rorschach, by virtue of the accumulated
research, is now "better prepared to stand the tests of reliability and
validation" (Exner, 1974, p. 16).
Clinicians persist in using the test because it continues to pro-
vide comprehensive and valuable information about their patients.
Thus, these clinicians are living proof of the application of these
research findings. In turn, the research findings mandate that the
test continue to be used. We have broadened the scope of the
cost-benefit issues and utility analysis in the application of the test
to be more consistent with clinical practice. In doing so, we have
identified other obvious benefits to the Rorschach. These include
the use of a single test for many applications and the opportunity
to collect quantifiable data while simultaneously collecting a be-
havioral sample in a context that maximizes individual differences
in problem solving. The Rorschach serves those who are interested
in having a valuable tool to help understand an individual com-
prehensively and to synthesize information across many domains
(Strieker & Gold, 1999). Undoubtedly, the behavior sampling
opportunity that occurs during administration promotes these goals
and provides another foundation to generalize outside of the con-
sulting room.
Acclaim for the Rorschach has elicited criticism (Wood &
Lilienfeld, 1999). Recent critical commentaries of the Rorschach
have identified some areas (e.g., normative data and methodolog-
ical problems in some studies) in which the Rorschach can be
improved. Like other tests, the Rorschach has its weaknesses.
However, these recent criticisms are undermined by methodolog-
ical double standards, confirmatory bias, incomplete coverage of
the literature, failure to integrate positive contributions over the
last 5 years that have specifically addressed earlier criticisms from
these authors, and a lack of any original data to support their
positions. For example, calls for a moratorium on the use of the
Rorschach (Garb, 1999; Garb et al., 2001) clearly contradict em-
pirical findings. Some clinical utility criteria (Hunsley & Bailey,
1999) set such a high and unrealistic standard that implementing
them would result in abandoning all testing, interviews, and ob-
servation for assessment. Useful criticism and reasonable clinical
validity criteria challenge future research rather than mandate our
10
Viglione (1999) cited 19 studies on pages 254-259 with negative
findings.
11
Viglione (1999) is criticized for not including Lerner and St. Peter's
(1984a) first article. To be factual, the earlier article was published in
Psychiatry, a journal excluded from the series editor's targeted list. It was
never considered for inclusion, according to the objective inclusion criteria
adopted in this review.
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE
forsaking psychological and neuropsychological assessment alto-
gether. However, an important contribution from critical reviews
has been to improve Rorschach research (Acklin, 1999), some-
thing that is happening much more quickly than was the case after
Rorschach researchers called for similar improvements a few years
ago (Exner, 1995; Viglione, 1997).
The informed use of the Rorschach will lead to a more sensitive
and helpful understanding of our clients as well as more effective
interventions. Continuing research and development of the test as
well as the provision for sound training and continuing education
in the Rorschach method, as with any assessment measure, are also
essential. Nevertheless, the extant empirical research findings ne-
cessitate that any scholar must accept the fact that the Rorschach
test method, its administration procedures, and its objective rules
for quantifying behavior produce valid data and observations.
References
Acklin, M. W. (1999). Behavioral science foundations of the Rorschach
test: Research and clinical applications. Assessment, 6, 319-324.
Acklin, M. W., McDowell, C. J., & VerscheU, M. S. (2000). Interobserver
agreement, intraobserver reliability, and the Rorschach Comprehensive
System. Journal of Personality Assessment, 74, 15—47.
Adair, H. E., & Wagner, E. E. (1992). Stability of unusual verbalizations
on the Rorschach for outpatients with schizophrenia. Journal of Clinical
Psychology, 48, 250-256.
Albert, S., Fox, H. M., & Kahn, M. W. (1980). Faking psychosis on the
Rorschach: Can expert judges detect malingering? Journal of Person-
ality Assessment, 44, 115-119.
American Psychiatric Association. (1994). Diagnostic and statistal manual
of mental disorders (4th ed.). Washington, DC: Author.
American Psychological Association, Division 12, Presidential Task Force
(1998). Assessment for the twenty-first century: A model curriculum.
The Clinical Psychologist, 52, 10-15.
Anastasi, A., & Urbina, S. (1996). Psychological testing (7th ed.). New
York: Macmillan.
Arboleda, C., & Holzman, P. S. (1985). Thought disorder in children at risk
for psychosis. Archives of General Psychiatry, 42, 1004-1013.
Archer, R. P., & Gordon, R. A. (1988). MMPI and Rorschach indices of
schizophrenic and depressive diagnoses among adolescent inpatients.
Journal of Personality Assessment, 52, 276-287.
Archer, R. P., & Krishnamurthy, R. (1997). MMPI-A and Rorschach
indices related to depression and conduct disorder An evaluation of the
incremental validity hypotheses. Journal of Personality Assessment, 69,
517-533.
Archer, R. P., & Newsom, C. R. (2000). Psychological test usage with
adolescent clients: Survey update. Assessment, 7, 227-235.
Atkinson, L., Quarrington, B., Alp, I. E., & Cyr, J. J. (1986). Rorschach
validity: An empirical approach to the literature. Journal of Clinical
Psychology, 42, 360-362.
Beutler, L. E., Williams, R. E., Wakefield, P. J., & Entwistle, S. R. (1995).
Bridging scientist and practitioner perspectives in clinical psychology.
American Psychologist, 50, 984-994.
Bilett, J. L., Jones, N. F., & Whitaker, L. C. (1982). Exploring schizo-
phrenic thinking in older adolescents with the WAIS, Rorschach and
WISC. Journal of Clinical Psychology, 38, 232-243.
Blais, M., Hilsenroth, M., Castlebury, F., Fowler, C., & Baity, M. (2001).
Predicting DSM-W Cluster B personality disorder criteria from MMPI-2
and Rorschach data: A test of incremental validity. Journal of Person-
ality Assessment, 76, 150-168.
Blatt, S. J., Ford, R. Q., Berman, W. H., Jr., Cook, B., Cramer, P., &
Robins, C. E. (1994). Therapeutic change: An object relations perspec-
tive. New York: Plenum.
467
Bornstein, R. F. (1996). Construct validity of the Rorschach Oral Depen-
dency scale: 1967-1995. Psychological Assessment, 8, 200-205.
Bornstein, R. F. (1999). Criterion validity of objective and projective
dependency tests: A meta-analytic assessment of behavioral prediction.
Psychological Assessment, 11, 48-57.
Bornstein, R. F., Bowers, K. S., & Robinson, K. J. (1997). Differential
relationships of objective and projective dependency scores to self-
reports of interpersonal life events in college student subjects. Journal of
Personality Assessment, 65, 255-269.
Brinderson, C. (1995). Apparent nonpatient psychopathology on the Ror-
schach: Disturbance or disinhibition? Unpublished doctoral disserta-
tion, California School of Professional Psychology, San Diego, CA.
Burns, B., & Viglione, D. J., Jr. (1996). The Rorschach Human Experience
Variable, interpersonal relatedness, and object representation in nonpa-
tients. Psychological Assessment, 8, 92-99.
Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A. M., &
Kaemmer, B. (1989). MMPI-2: Manual for administration and scoring.
Minneapolis: University of Minnesota Press.
Butcher, J. N., & Rouse, S. (1996). Personality: Individual differences and
clinical assessment. Annual Review of Psychology, 47, 87-111.
Cadenhead, K. S., Perry, W., & Braff, D. L. (1996). The relationship of
information-processing deficits and clinical symptoms in schizotypal
personality disorder. Biological Psychiatry, 40, 853-858.
Camara, W., Nathan, J., & Puente, A. (2000). Psychological test usage:
Implications in professional use. Professional Psychology: Research
and Practice, 31, 141-154.
Ceraey, M. S. (1984). One last response to the Rorschach test: A second
chance to reveal oneself. Journal of Personality Assessment, 48, 338-
344.
Chapman, L. J., Chapman, J. P., & Kwapil, T. R. (1995). Scales for the
measurement of schizotype. In A. Raine. T. Lencz, & S. A. Mednick
(Eds.), Schizotypal personality (pp. 79-106). New York: Cambridge
University Press.
Chapman, L. J., Chapman, J. P., & Raulin, M. L. (1976). Scales for
physical and social anhedonia. Journal of Abnormal Psychology, 85,
374-382.
Chapman, L. J., Chapman, J. P., & Raulin, M. L. (1978). Body-image
aberration in schizophrenia. Journal of Abnormal Psychology, 87, 399-
407.
Cheyette, L. (1992). The effects of examiner style on Rorschach test results
of children with borderline psychopathology. Unpublished doctoral dis-
sertation, California School of Professional Psychology, San Diego, CA.
Clemence, A., & Handler, L. (2001). Psychological assessment on intern-
ship: A survey of training directors and their expectations for students.
Journal of Personality Assessment, 76, 18-47.
Cochrane, C. T. (1972). Effects of diagnostic information on empathic
understanding by the therapist in a psychotherapy analogue. Journal of
Consulting and Clinical Psychology, 38, 359-365.
Coleman, M. J., Levy, D. L., Lenzenweger, M. F., & Holzman, P. S.
(1996). Thought disorder, perceptual aberrations, and schizotypy. Jour-
nal of Abnormal Psychology, 105, 469-473.
Colligan, R. C., Osbome, D. M. S. W., & Offord, K. P. (1984). The aging
MMPI: Development of contemporary norms. Mayo Clinic Proceed-
ings, 59, 377-390.
Cooper, S. H., Perry, J. C., & O'Connell, M. (1991). The Rorschach
Defense Scales: II. Longitudinal perspectives. Journal of Personality
Assessment, 56, 191-201.
Dahlstrom, W. G., Welsh, G. S., & Dahlstrom, L. E. (1975). A MMPI
handbook: Vol. 11: Research application. Minneapolis: University of
Minnesota Press.
Dana, R. H., & Back, B. R. (1983). The concurrent validity of child
Rorschach interpretations. Journal of Personality Assessment, 47, 3-6.
Daniels, E. K., Shenton, M. E., et al. (1988). Patterns of thought disorder
468 VIGLIONE AND HILSENROTH
associated with right cortical damage, schizophrenia, and mania. Amer-
ican Journal of Psychiatry, 145, 944-949.
Dawes, R. M. (1999). Two methods for studying incremental validity of a
Rorschach variable. Psychological Assessment, 11, 297-302.
Dudek, S. Z., & Marchand, P. (1983). Artistic style and personality in
creative painters. Journal of Personality Assessment, 47, 139—142.
Durand, V., Blanchard, E., & Mindell, J. (1988). Training in projective
testing: Survey of clinical training directors and internship directors.
Professional Psychology: Research and Practice, 19, 236-238.
Eckblad, M. L., Chapman, L. J., & Chapman, J. P. (1982). The Revised
Social Anhedonia Scale. Unpublished manuscript.
Edell, W. S., & Chapman, L. J. (1979). Anhedonia, perceptual aberration,
and the Rorschach. Journal of Consulting and Clinical Psychology, 47,
377-384.
Erdberg, P., & Schaffer, T. W. (1999, July). International symposium on
Rorschach nonpatient data: Findings from around the world. Paper
presented at the International Congress of Rorschach and Projective
Methods, Amsterdam, the Netherlands.
Erstad, D. (1996). An investigation of older adults' less frequent human
movement and color responses on the Rorschach. Unpublished doctoral
dissertation, Marquette University.
Exner, J. E. (1974). The Rorschach: A Comprehensive System (Vol. 1).
New York: Wiley.
Exner, J. E. (1991). The Rorschach: A Comprehensive System: Vol. 2.
Interpretation (2nd ed.). New York: Wiley.
Exner, J. E. (1993). The Rorschach: A Comprehensive System: Vol. 1.
Basic foundations (3rd ed.). New York: Wiley.
Exner, J. E. (Ed.). (1995). Issues and methods in Rorschach research.
Hillsdale, NJ: Erlbaum.
Exner, J. E. (1999). The Rorschach: Measurement concepts and issues of
validity. In S. E. Embretson & S. L. Hershberger (Eds.), The new rules
of measurement: What every psychologist and educator should know
(pp. 159-183). Mahwah, NJ: Erlbaum.
Exner, J. E. (2000). A primerfor Rorschach interpretation. Asheville, NC:
Rorschach Workshops.
Exner, J. E., Armbruster, G. L., & Viglione, D. (1978). The temporal
stability of some Rorschach features. Journal of Personality Assess-
ment, 42, 474-482.
Exner, J. E., Colligan, S. C., Hillman, L. B., Metis, A. S., Ritzier, B.,
Rogers, K. T., Sciara, A., D., & Viglione, D. J. (2001). A Rorschach
workbook for the Comprehensive System (5th ed.). Asheville, NC: Ror-
schach Workshops.
Exner, J. E., Colligan, S. C., Hillman, L. B., Ritzier, B., Sciara, A. D., &
Viglione, D. J. (1995). A Rorschach workbook for the Comprehensive
System (4th ed.). Asheville, NC: Rorschach Workshops.
Exner, J. E., Thomas, E. A., & Mason, B. J. (1985). Children's Rorschachs:
Description and prediction. Journal of Personality Assessment, 49, 13-
20.
Exner, J. E., & Weiner, I. B. (1982). The Rorschach: A Comprehensive
System: Vol. 3. Assessment of children and adolescents. New York:
Wiley.
Exner, J. E., & Weiner, I. B. (1995). The Rorschach: A Comprehensive
System: Vol. 3. Assessment of children and adolescents (2nd ed.). New
York: Wiley.
Exner, J. E., Weiner, I. B., & Schuyler, W. (1976). A Rorschach workbook
for the Comprehensive System. Bayville, NY: Rorschach Workshops.
Exner, J. E., & Wylie, J. (1977). Some Rohrschach data concerning
suicide. Journal of Personality Assessment, 41, 339-348.
Eyman, J. R., & Eyman, S. K. (1991). Personality assessment in suicide
prediction. Special Issue: Assessment and prediction of suicide. Suicide
and Life Threatening Behavior, 21, 37-55.
Eyman, J. R., & Eyman, S. K. (1992). Personality assessment in suicide
prediction. In R. W. Maris, A. L. Berman, J. T. Maltsberger, & R. I.
Yufit (Eds.), Assessment and prediction of suicide (pp. 183—201). New
York: Guilford Press.
Eyman, S. K., & Eyman, J. R. (1987, August-September). An investigation
of Exner's Suicide Constellation. Paper presented at the 95th Annual
Convention of the American Psychological Association, New York.
Finn, S. E., & Kamphuis, i. H. (1995). What a clinician needs to know
about baserates. In J. N. Butcher (Ed.), Clinical personality assessment
(pp. 224-235). New York: Oxford University Press.
Rske, D. W., Cartwright, D. S., & Kirtner, W. L. (1964). Are psychother-
apeutic changes predictable? Journal of Abnormal and Social Psychol-
ogy, 69, 413-426.
Fowler, C., Hilsenroth, M. J., & Handler, L. (1996). A multimethod
approach to assessing dependency: The early memory dependency
probe. Journal of Personality Assessment, 67, 399-413.
Fowler, C. J., Piers, C. C., Hilsenroth, M. J., Holdwick, D., & Padawer, R.
(2001). Assessing risk factors various degrees of suicidal activity: The
Rorschach Suicide Constellation. Journal of Personality Assessment, 76,
333-351.
Franklin, K. W., & Cornell, D. G. (1997). Rorschach interpretation with
high-ability adolescent females: Psychopathology or creative thinking?
Journal of Personality Assessment, 68, 184-196.
Frueh, B. C., & Kinder, B. N. (1994). The susceptibility of the Rorschach
inkblot test to malingering of combat-related PTSD. Journal of Person-
ality Assessment, 62, 280-298.
Gacono, C. B., & Meloy, J. R. (1994). The Rorschach assessment of
aggressive and psychopathic personalities. Mahwah, NJ: Erlbaum.
Gadol, I. (1969). The incremental and predictive validity of the Rorschach
test in personality assessment of normal, neurotic, and psychotic sub-
jects. Dissertation Abstracts International, 29, 3482-B. (University Mi-
crofilms No. 3469-4469)
Gallucci, N. T. (1989). Personality assessment with children of superior
intelligence: Divergence versus psychopathology. Journal of Personal-
ity Assessment, 53, 749-760.
Garb, H. N. (1989). Clinical judgment, clinical training, and professional
experience. Psychological Bulletin, 105, 387-396.
Garb, H. N. (1994). Cognitive heuristics and biases in personality assess-
ment. In L. Heath, R. S. Tindale, J. Edwards, E. J. Posavac, F. B. Bryant,
E. Henderson-King, Y. Suarez-Balcazar, & J. Myers (Eds.), Applications
of heuristics and biases to social issues. New York: Plenum Press.
Garb, H. N. (1998). Studying the clinician: Judgment research and psy-
chological assessment. Washington, DC: American Psychological
Association.
Garb, H. N. (1999). Call for a moratorium on the use of the Rorschach
Inkblot Test in clinical and forensic settings. Assessment, 6, 313-315.
Garb, H. N., Florio, C. M., & Grove, W. M. (1998). The validity of the
Rorschach and the Minnesota Mulu'phasic Personality Inventory: Re-
sults from meta-analyses. Psychological Science, 9, 402-404.
Garb, H. N., Wood, J. M., Nezworski, M. T., Grove, W. M., & Stejskal,
W. J. (2001). Toward a resolution of the Rorschach controversy. Psy-
chological Assessment, 13, 433-448.
Garfield, S. L. (2000). The Rorschach test in clinical diagnosis—A brief
commentary. Journal of Clinical Psychology, 56, 431-434.
Golden, M. (1964). Some effects of combining psychological tests on
clinical inferences. Journal of Consulting Psychology, 28, 440-446.
Greenwald, D. F. (1991). Personality dimensions reflected by the Ror-
schach and the 16PF. Journal of Clinical Psychology, 47, 708-715.
Haller, N., & Exner, J. E. (1985). The reliability of Rorschach variables for
inpatients presenting symptoms of depression and/or helplessness. Jour-
nal of Personality Assessment, 49, 516-521.
Hamel, M., Shaffer, T. W., & Erdberg, P. (2000). A study of nonpatient
preadolescent Rorschach protocols. Journal of Personality Assess-
ment, 75, 280-294.
Hammond, K. R. (1955). Probabilistic functioning and the clinical method.
Psychological Review, 62, 255-262.
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE
Hathaway, S. R., McKinley, J. C, Butcher, J. N., Dahlstrom, W. G.,
Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). Minnesota Mul-
tiphasic Personality Inventory-2: Manual for administration and scor-
ing. Minneapolis: University of Minnesota Press.
Miller, J. B., Rosenthal, R., Bornstein, R. F., & Berry, D. T. (1999). A
comparative meta-analysis of Rorschach and MMPI validity. Psycho-
logical Assessment, 11, 278-296.
Hilsenroth, M. J., Fowler, C. J., & Padawer, J. R. (1998). The Rorschach
Schizophrenia Index (SCZI): An examination of reliability, validity, and
diagnostic efficiency. Journal of Personality Assessment, 70, 514-534.
Hilsenroth, M. J., Fowler, J. C., Padawer, J. R., & Handler, L. (1997).
Narcissism in the Rorschach revisited: Some reflections on empirical
data. Psychological Assessment, 9, 113-121.
Hilsenroth, M. J., & Handler, L. (1995). A survey of graduate students'
experiences, interests, and attitudes about learning the Rorschach. Jour-
nal of Personality Assessment, 64, 243—257.
Hilsenroth, M. J., Handler, L., Toman, K. M., & Padawar, J. R. (1995).
Rorschach and MMPI-2 indices of early psychotherapy termination.
Journal of Consulting and Clinical Psychology, 63, 956—965.
Holaday, M., & Whittenberg, T. (1994). Rorschach responding in children
and adolescents who have been severely burned. Journal of Personality
Assessment, 62, 269-279.
Holzman, P. S., et al. (1974). Eye-tracking dysfunctions in schizophrenic
patients and their relatives. Archives of General Psychiatry, 31, 143-
151.
Hunsley, J., & Bailey, J. M. (1999). The clinical utility of the Rorschach:
Unfulfilled promises and an uncertain future. Psychological Assess-
ment. 11, 266-277.
Hunsley, J., Hanson, R. K., & Parker, K. C. (1988). A summary of the
reliability and stability of MMPI scales. Journal of Clinical Psychol-
ogy, 44, 44—46.
Johnston, M. H., & Holzman, P. S. (1979). Assessing schizophrenic think-
ing: A clinical and research instrument for measuring thought disorder.
San Francisco: Jossey-Bass.
Kates, J. (1994). A validation study of the Rorschach Ego Impairment
Index. Unpublished doctoral dissertation, California School of Profes-
sional Psychology, San Diego, CA.
Kleiger, J. H. (1999). Disordered thinking and the Rorschach: Theory,
research, and differential diagnosis. Hillsdale, NJ: Analytic Press.
Kleinmuntz, B. (1990). Why we still use our heads instead of formulas:
Toward an integrative approach. Psychological Bulletin, 107, 296-310.
Krall, V., Sachs, H., Langer, B., Rayson, B., & Growe, G. (1983). Ror-
schach norms for inner city children. Journal of Personality Assess-
ment, 47, 155-157.
Leichtman, M. (1996). The Rorschach: A developmental perspective. Hills-
dale, NJ: Analytic Press.
Lerner, H. D., & St. Peter, S. (1984a). Patterns of object relations in
neurotic, borderline and schizophrenic patients. Psychiatry, 47, 77-92.
Lemer, H. D., & St. Peter, S. (1984b). The Rorschach H response and
object relations. Journal of Personality Assessment, 48, 345-350.
Levy, L. H. (1963). Psychological interpretation. New York: Holt, Rine-
hart & Winston.
Masling, J. M. (1986). Empirical studies of psychoanalytic theories. Hills-
dale, NJ: Erlbaum.
Masling, J. M., Rabid, L., & Blondheim, S. H. (1967). Obesity, level of
aspiration, and Rorschach and TAT measures of oral dependence. Jour-
nal of Consulting Psychology, 31, 233-239.
Mattlar, C. E., Carlsson, A., & Forsander, C. (1993). The issue of the
popular response: Definition and universal vs. culture-specific popular
responses. Special Issue: International Rorschach Congress 1993. British
Journal ofProjective Psychology, 38, 53-62.
Mauger, P. A. (1972). The test-retest reliability of persons: An empir-
ical investigation utilizing the MMPI and the Personality Research
469
Form. Unpublished doctoral dissertation, University of Minnesota,
Minneapolis.
McCann, J. T. (1998). Defending the Rorschach in court: An analysis of
admissibility using legal and professional standards. Journal of Person-
ality Assessment, 70, 125-144.
McDowell, C. J., & Acklin, M. W. (1996). Standardizing procedures for
calculating Rorschach interrater reliability: Conceptual and empirical
foundations. Journal of Personality Assessment, 66, 308-320.
Meehl, P. E. (1957). When shall we use our heads instead of the formula?
Journal of Counseling Psychology, 4, 268-273.
Meehl, P. E., & Rosen, A. (1955). Antecedent probability and the effi-
ciency of psychometric signs, patterns, or cutting scores. Psychological
Bulletin, 52, 171-190.
Meloy, J. R., Hansen, T. L., & Weiner, I. B. (1997). Authority of the
Rorschach: Legal citations during the past 50 years. Journal of Person-
ality Assessment, 69, 53—62.
Meyer, G. J. (1993). The impact of response frequency on the Rorschach
constellation indices and on their validity with diagnostic and MMPI-2
criteria. Journal of Personality Assessment, 60, 153-180.
Meyer, G. J. (1997a). Assessing reliability: Critical correlations for a
critical examination of the Rorschach Comprehensive System. Psycho-
logical Assessment, 9, 480-489.
Meyer, G. J. (1997b). Thinking clearly about reliability: More critical
correlations regarding the Rorschach Comprehensive System. Psycho-
logical Assessment, 9. 495-498.
Meyer, G. J. (1999). Introduction to the Special Series on the utility of the
Rorschach for clinical assessment. Psychological Assessment, 11, 235-
239.
Meyer, G. J. (2000). Incremental validity of the Rorschach Prognostic
Rating Scale over the MMPI Ego Strength Scale and IQ. Journal of
Personality Assessment, 74, 356-370.
Meyer, G. J. (2001). Evidence to correct misperceptions about Rorschach
norms. Manuscript submitted for publication.
Meyer, G. J. (in press). Exploring possible ethnic differences and bias in
Rorschach Comprehensive System scores. Journal of Personality As-
sessment.
Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Kubiszyn, T. W.,
Moreland, K. L., Eisman, E. J., & Dies, R. R. (1998). Benefits and costs
of psychological assessment in healthcare delivery: Report of the Board
of Professional Affairs Psychological Assessment Work Group, Part 1.
Washington, DC: American Psychological Association.
Meyer, G. J., & Handler, L. (1997). The ability of the Rorschach to predict
subsequent outcome: A meta-analysis of the Rorschach Prognostic Rat-
ing Scale. Journal of Personality Assessment, 69, 1-38.
Meyer, G. J., Hilsenroth, M. J., Baxter, D., Exner, J. E., Jr., Fowler, C. J.,
Piers, C. C., & Resnick, J. (in press). An examination of interrater
reliability for scoring the Rorschach Comprehensive System in eight
data sets. Journal of Personality Assessment.
Meyer, G. J., & Resnick, G. D. (1996, February). Assessing ego impair-
ment: Do scoring procedures make a difference? Paper presented at the
15th International Research Conference, Boston.
Milott, S. R., Lira, F. T., & Miller, W. C. (1977). Psychological assessment
of the burned patient. Journal of Clinical Psychology, 33, 425-430.
Netter, B. E. C., & Viglione, D. J. (1994). An empirical study of malin-
gering schizophrenia on the Rorschach. Journal of Personality Assess-
ment, 62, 45-57.
O'Connell, M., Cooper, S., Perry, C., & Hoke, L. (1989). The relationship
between thought disorder and psychotic symptoms in borderline person-
ality disorder. Journal of Nervous and Mental Disease, 177, 273-278.
Paolo, A. M., Troster, A. I., & Ryan, J. J. (1997). Test-retest stability of the
California Learning Test in older persons. Neuropsychology, 11, 613-
616.
Parker, K. C., Hanson, R. K., & Hunsley, J. (1988). MMPI, Rorschach, and
470 VIGLIONE AND HILSENROTH
WAIS: A meta-analytic comparison of reliability, stability, and validity.
Psychological Bulletin, 103, 367-373.
Parker, K. C. H., Hunsley, J., & Hanson, R. K. (1999). Old wine from old
skins sometimes taste like vinegar: A response to Garb, Florio, and
Grove. Psychological Science, 10, 367-373.
Perez, F. I. (1976). Behavioral analysis of clinical judgment. Perceptual
and Motor Skills, 43, 711-718.
Perry, W., & Braff, D. L. (1994). Information-processing deficits and
thought disorder in schizophrenia. American Journal of Psychiatry, 151,
363-367.
Perry, W., & Braff, D. L. (in press). The simultaneous measurement of
attention and information processing and disturbed thinking in male
schizophrenic patients. Archives of General Psychiatry.
Perry, W., Geyer, M. A., & Braff, D. L. (1999). Sensorimotor gating and
thought disturtence measured in close temporal proximity in schizo-
phrenic patients. Archives of General Psychiatry, 56, 277—281.
Perry, W., Moore, D., & Braff, D. L. (1995). Gender differences on thought
disturbance measures in schizophrenia. American Journal of Psychiatry,
152, 1298-1301.
Perry, W., Potterat, E. G., Auslander, L., Kaplan, E., & Jeste, D. (19%). A
neuropsychological approach to the Rorschach in patients with dementia
of the Alzheimer type. Assessment, 3, 351-363.
Perry, W., Sprock, J., Schaible, D., McDougall, A., et al. (1995). Amphet-
amine on Rorschach measures in normal subjects. Journal of Personality
Assessment, 64, 456-465.
Perry, W., & Viglione, D. J. (1991). The Ego Impairment Index as a
predictor of outcome in melancholic depressed patients treated with
tricyclic antidepressants. Journal of Personality Assessment, 56, 487-
501.
Perry, W., Viglione, D., & Braff, D. L. (1992). The Ego Impairment Index
and schizophrenia: A validation study. Journal of Personality Assess-
ment, 59, 165-175.
Peterson, C., Samel, A., von Baeyer, C., Abramson, L. H., Metalsky, G. I.,
& Seligman, M. E. P. (1982). The Attributional Style Quesionnaire.
Cognitive Theory and Research, 6, 289-299.
Piotrowski, C., & Belter, R. W. (1999). Internship training in psychological
assessment: Has managed care had an impact? Assessment, 6, 381-389.
Presley, G., Smith, C., Hilsenroth, M., & Exner, J. (in press). Rorschach
validity with African Americans. Journal of Personality Assessment.
The Psychological Corporation. (1997). Wechsler Adult Intelligence Scale,
technical manual (3rd ed.). San Antonio, TX: The Psychological
Corporation.
Putnam, S. H., Kurtz, J. E., & Houts, D. C. (1996). Four-month test-retest
reliability of the MMPI-2 with normal male clergy. Journal of Person-
ality Assessment, 67, 341-353.
Rapaport, D., Gill, M., & Schafer, R. (1946). Diagnostic psychological
testing (Vols. 1 & 2). Chicago: Yearbook Publishers.
Roberts, B. W., & DelVecchio, W. F. (2000). The rank-order consistency
of personality traits from childhood to old age: A quantitative review of
longitudinal studies. Psychological Bulletin, 126, 3—25.
Rogers, R. (Ed.). (1997). Clinical assessment of malingering and deception
(2nd ed.). New York: Guilford Press.
Russ, S. W. (1980). Primary process integration on the Rorschach and
achievement in children. Journal of Personality Assessment, 44, 338-
344.
Russ, S. W. (1981). Primary process integration on the Rorschach and
achievement in children: A follow-up study. Journal of Personality
Assessment, 45, 473-477.
Ryan, J. J., Dunn, G. E., & Paolo, A. M. (1995). Temporal stability of the
MMPI-2 in a substance abuse sample. Psychotherapy in Private Prac-
tice, 14, 33-41.
Schafer, R. (1954). Psychoanalytic interpretation in Rorschach testing.
New York: Grune & Stratton.
Schaich, D. (1999). A synthesis of the conceptual and inferential proce-
dures involved with the assessment of malingering: An applied study.
Unpublished doctoral dissertation, California School of Professional
Psychology, San Diego, CA.
Schuerger, J. M., Zarrella, K. L., & Holtz, A. S. (1989). Factors that
influence the temporal stability of personality by questionnaire. Journal
of Personality and Social Psychology, 56, 777-783.
Schwartz, N. S., Mebane, D. L., & Malony, H. N. (1990). Effects of
alternate modes of administration of Rorschach performance of deaf
adults. Journal of Personality Assessment, 54, 671-683.
Shaffer, T. W., Erdberg, P., & Haroian, J. (1999). Current nonpatient data
for the Rorschach, WAIS-R, and MMPI-2. Journal of Personality
Assessment, 73, 305-316.
Shapiro, J. P., Leifer, M., Martone, M. W., & Kassem, L. (1990). Multi-
method assessment of depression in sexually abused girls. Journal of
Personality Assessment, 55, 234-248.
Shenton, M. E., Solovay, M. R., et al. (1989). Thought disorder in the
relatives of psychotic patients. Archives of General Psychiatry, 46,
897-901.
Shrout, P. E., & Fliess, J. L. (1979). Intraclass correlations: Uses in
assessing rater reliability. Psychological Bulletin, 86, 420-428.
Silberg, J. L., & Armstrong, J. G. (1992). The Rorschach test for predicting
suicide among depressed adolescent inpatients. Journal of Personality
Assessment, 59, 290-303.
Sines, L. K., Silver, R. J., & Lucero, R. J. (1961). The effect of therapeutic
intervention by untrained "therapists." Journal of Clinical Psychol-
ogy, 17, 394-396.
Skelton, M. D., Boik, R. ;., & Madero, J. N. (1995). Thought disorder on
the WAIS-R relative to the Rorschach: Assessing identity disorder
adolescents. Journal of Personality Assessment, 65, 533-549.
Sloan, P., Arsenault, L., Hilsenroth, M., Handler, L., & Harvill, L. (1996).
Rorschach measures of posttraumatic stress in Persian Gulf War veter-
ans. Journal of Personality Assessment, 66, 54-64.
Sloan, P., Arsenault, L., Hilsenroth, M., Harvill, L., & Handler, L. (1995).
Rorschach measures of posttraumatic stress in Persian Gulf War veter-
ans. Journal of Personality Assessment, 64, 397-414.
Spiro, A., Butcher, J. N., Levenson, M. R., Aldwin, C. M., & Bosse, R.
(1993, August). Personality change and stability over five years: The
MMPI-2 in older men. Paper presented at the 101st Annual Convention
of the American Psychological Association, Toronto, Ontario, Canada.
Stedman, J., Hatch, J., & Schoenfeld, L. (2000). Preintemship preparation
in psychological testing and psychotherapy: What internship directors
say they expect. Professional Psychology: Research and Practice, 31,
321-326.
Stone, L. A. (1965). Test-retest stability of the MMPI scales. Psycholog-
ical Reports, 16, 619-620.
Strieker, G., & Gold, J. R. (1999). The Rorschach: Toward a nomotheti-
cally based, idiographically applicable configurational model. Psycho-
logical Assessment, 11, 240-250.
Turner, D. R. (1966). Predictive efficiency as a function of amount of
information and level of professional experience. Journal of Projective
Techniques and Personality Assessment, 30, 4-11. .
Urist, J. (1977). The Rorschach Test and the assessment of object relations.
Journal of Personality Assessment, 41, 3-9.
Viglione, D. J. (1989). Rorschach science and art. (Review of the Ror-
schach: A Comprehensive System, Vol. 1, 2nd ed.). Journal of Person-
ality Assessment, 53, 195-197.
Viglione, D. J. (1997). Problems in Rorschach research and what to do
about them. Journal of Personality Assessment, 68, 590-599.
Viglione, D. J. (1999). A review of recent research addressing the utility of
the Rorschach. Psychological Assessment, 11, 251-265.
Viglione, D. J. (2001). Challenges and solutions in Rorschach coding and
administration. Manuscript in preparation.
Viglione, D. J., & Exner, J. E. (1983). The effects of state anxiety and
 SPECIAL SECTION: RORSCHACH FACTS AND FUTURE 471

limited social-evaluative stress on the Rorschach. Journal of Personality
Assessment, 47, 150-154.
Viglione, D. J., Gaudiana, S., & Gowri, A. (1997, April). Two questions
about Rorschach norms: Cultural issues and divergence. Paper pre-
sented at the Society for Personality Assessment, San Diego, CA.
Wade, T. C., & Baker, T. B. (1977). Opinions and use of psychological
tests: A survey of clinical psychologists. American Psychologist, 32,
874-882.
Wahlberg, K. E., Wynne, L. C., Oja, H., Keskitalo, P., Pykalainen, L.,
Lahti, I., Moring, J., Naarala, M., Sorri, A., Seitamaa, M., Laksy, K.,
Kolassa, J., & Tienari, P. (1997). Gene-environment interaction in
vulnerability to schizophrenia: Findings from the Finnish adoptive fam-
ily study of schizophrenia. American Journal of Psychiatry, 154, 355-
362.
Watkins, C., Campbell, V., Nieberding, R., & Hallmark, R. (1995). Con-
temporary practice of psychological assessment by clinical psycholo-
gists. Professional Psychology: Research and Practice, 26, 54—60.
Weiner, I. B. (1966). Psychodiagnosis in schizophrenia. New York: Wiley.
Weiner, I. B. (1996). Some observations on the validity of the Rorschach
Inkblot Method. Psychological Assessment, 8, 206-213.
Weiner, I. B. (1998). Principles of Rorschach interpretation. Mahwah, NJ:
Erlbaum.
Weiner, I. B. (1999). What the Rorschach can do for you: Incremental
validity in clinical applications. Assessment, 6, 327-338.
Weiner, I. B. (2001a). Advancing the science of psychological assessment:
The Rorschach Inkblot Method as exemplar. Psychological Assess-
ment, 13, 423-432.
Weiner, I. B. (2001b). Considerations in correcting Rorschach reference
date. Journal of Personality Assessment, 77, 122—127.
Weiner, I. B., Exner, J. E., Jr., & Scbiara, A. (1996). Is the Rorschach
welcome in the courtroom? Journal of Personality Assessment, 67,
422-424.
Wood, J. M., & Lilienfeld, S. O. (1999). The Rorschach Inkblot Test: A
case of overstatement? Assessment, 6, 341-349.
Wood, J. M., Lilienfeld, S. O., Garb, H. N., & Nezworski, M. T. (2000).
The Rorschach test in clinical diagnosis: A critical review with a
backward look at Garfield (1947). Journal of Clinical Psychology, 56,
395-430.
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996a). The Compre-
hensive System for the Rorschach: A critical examination. Psychological
Science, 7, 3-10.
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996b). Thinking
critically about the Comprehensive System for the Rorschach: A reply to
Exner. Psychological Science, 7, 14-17.
Wood, J. M., Nezworski, M. T., Stejskal, W. J., Garven, S., & West, S. G.
(1999). Methodological issues in evaluating Rorschach validity: A com-
ment on Burns and Viglione (1996), Weiner (19%), and Ganellen
(1996). Assessment, 6, 115-129.
Received July 27, 2000
Revision received February 26, 2001
Accepted July 3, 2001

Low Publication Prices for APA Members and Affiliates

Keeping you up-to-date. All APA Fellows, Members, Associates, and Student Affiliates
receive—as part of their annual dues—subscriptions to the American Psychologist and
APA Monitor. High School Teacher and International Affiliates receive subscriptions to
the APA Monitor, and they may subscribe to the American Psychologist at a significantly
reduced rate. In addition, all Members and Student Affiliates are eligible for savings of up
to 60% (plus a journal credit) on all other APA journals, as well as significant discounts on
subscriptions from cooperating societies and publishers (e.g., the American Association for
Counseling and Development, Academic Press, and Human Sciences Press).

Essential resources. APA members and affiliates receive special rates for purchases of
APA books, including the Publication Manual of the American Psychological Association,
and on dozens of new topical books each year.

Other benefits of membership. Membership in APA also provides eligibility for

competitive insurance plans, continuing education programs, reduced APA convention fees,
and specialty divisions.

More information. Write to American Psychological Association, Membership Services,

750 First Street, NE, Washington, DC 20002-4242.