See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/8658395

Methodological Issues in Cross-Cultural and Multicultural Rorschach Research

Article  in  Journal of Personality Assessment · May 2004

DOI: 10.1207/s15327752jpa8202_7 · Source: PubMed

CITATIONS READS
39 763

2 authors, including:

James Allen
University of Minnesota Duluth
91 PUBLICATIONS   2,741 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Intervention Research to Improve Native American Health (IRINAH) View project

Acculturaltion and parenting in the context of trauma and exile: Vietnamese refugees after 25 years in exile View project

All content following this page was uploaded by James Allen on 08 February 2022.

The user has requested enhancement of the downloaded file.

 JOURNAL OF PERSONALITY ASSESSMENT, 82(2), 189–206
Copyright © 2004, Lawrence Erlbaum Associates, Inc.

Methodological Issues in Cross-Cultural

and Multicultural Rorschach Research CROSS-CULTURAL
ALLEN
RORSCHACH
AND DANA
METHODOLOGY

James Allen
Department of Psychology
University of Alaska Fairbanks and
Psychosocial Centre for Refugees
University of Oslo

Richard H. Dana
Regional Research Institute for Human Services
Portland State University

Hermann Rorschach researched the utility of his inkblot experiment to understand

psychopathology and cultural differences. Contemporary research with the Rorschach has
evaluated its utility as a test, although it may more properly represent a clinical method with
somewhat different evaluation criteria. Recent controversy regarding the adequacy of the Ror-
schach as a test and the adequacy of its normative data has at times distorted and oversimplified
important methodological issues inherent in the study of cultural difference. Cultural processes
remain a central and inadequately examined variable in Rorschach research; an important
emergent area of inquiry is the Rorschach’s clinical utility as a cross-cultural assessment instru-
ment. We review multicultural and cross-cultural methodological issues intrinsic to contempo-
rary Rorschach research here. Consideration of cultural issues enlarges and enriches the Ror-
schach clinical utility debate and suggests underexplored research strategies that can contribute
to its resolution.

Psychologists in the United States have made extensive use
of the Rorschach in understanding personality and diagnos-
ing psychopathology for more than half a century. The Com-
prehensive System (CS; Exner, 1993) for administration and
scoring transformed the Rorschach into an assessment instru-
ment for which reliability and validity could be empirically
examined using many of the fundamental tenets of classical
psychometric theory. High-inference psychoanalytic inter-
pretation was gradually eschewed in favor of low-inference
interpretation. Widespread acceptance of the CS occurred
within a larger professional objective in clinical psychology
of establishing an empirical basis for practice. The CS em-
phasizes reliable scoring procedures and development of a
body of validity data to support interpretive rules that use
normative data.
The CS has been largely responsible for maintaining the
prominence of the Rorschach in psychological assessment
practice in the United States. In one recent survey
(Watkins, Campbell, Nieberding, & Hallmark, 1995), 80%
of 412 clinical psychologists reported use of the Rorschach
in their work. Other surveys have indicated that 80% of
American Psychological Association approved clinical psy-
chology programs provided Rorschach training and 75% of
students learned the CS (Hilsenroth & Handler, 1995;
Ritzler & Alter, 1986). The Rorschach was the fourth most
widely used instrument in a survey of 1,500 clinical psy-
chologists (Camara, Nathan, & Puente, 2000). Interna-
tionally, the Rorschach enjoys a similar high profile. In a
survey of psychologists in Spain, Portugal, and Latin
America (Muniz, Prieto, Almeida, & Bartram, 1999), the
Rorschach was the third most frequently used assessment
instrument, following translated versions of the Wechsler
Adult Intelligence Scale (Wechsler, 1997) and the Minne-
sota Multiphasic Personality Inventory–2 (MMPI–2;
Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer,
1989). There is significant international clinical use and re-
search with the Rorschach (Butcher, Nezami, & Exner,
1998); the International Rorschach Society has over 2,400
190 ALLEN AND DANA
members in 23 countries (Weiner, 1999), and over 7,000
Rorschach research articles have been published (Butcher
& Rouse, 1996).
CS CONTROVERSY
Despite continued widespread acceptance of the Rorschach
by professional psychologists, as evidenced by this survey
data, controversy concerning its clinical use has existed for
over 50 years. The earliest Rorschach critiques have exam-
ined shortcomings in its psychometric foundations and valid-
ity research (e.g., Cronbach, 1949; Eysenck, 1959; Jensen,
1965; McFarlane, 1942; Mensh, 1950; Munroe, 1945). In his
seminal critique, Cronbach described inadequacies of the
Rorschach as a psychometric instrument. These inadequa-
cies, along with statistical errors of omission and commis-
sion, drew into question the conclusions in over 90% of the
Rorschach studies Cronbach reviewed. Contemporary criti-
cism reiterates many of these same still unresolved issues
(e.g., Dawes, 1999; Garb, Wood, Lilienfeld, & Nezworski,
2002; Hunsley & Bailey, 1999; Lilienfeld, Wood, & Garb,
2000; Wood & Lilienfeld, 1999; Wood, Lilienfeld, Garb, &
Nezworski, 2000a, 2000b; Wood, Lilienfeld, Nezworski, &
Garb, 2001; Wood, Nezworski, Garb, & Lilienfeld, 2001a;
Wood, Nezworski, & Stejskal, 1996; Wood, Nezworski,
Stejskal, & Garven, 2001), and numerous responses have ap-
peared in the literature to address these critiques (Bornstein,
2001; Gacono, Loving, & Bodholdt, 2001; Ganellen, 2001;
Hibbard, 2003; Meyer, 2001; Miller, Rosenthal, Bornstein,
Berry, & Brunell-Neuleib, 1999; Stricker & Gold, 1999;
Viglione, 1999; Weiner, Spielberger, & Abeles, 2002). The
controversy diminishes not only Rorschach credibility; it
threatens acceptance of all standard assessment instruments
among third-party reimbursers and service providers.
Piotrowski (1999) and Stout (1997) both have noted a trend
of diminished use of standard assessment instruments, in par-
ticular in managed care practice settings, despite a recent
meta-analysis using 800 studies that reported validity coeffi-
cients for the Rorschach and other standard psychological
tests that matched or exceeded those of standard medical
tests (Meyer et al., 2001).
In this article, we seek to expand the scope of response to
CS antagonists, along with the scope of its clinical applica-
tion, by (a) reviewing findings from recent multicultural re-
search with the CS, (b) discussing methodological issues in
this work, and (c) describing a research agenda for future
work. We seek to foster Rorschach credibility by enhancing
the methodological quality of cross-cultural and multicul-
1
We refer here to application of the Rorschach across cultural
groups, often internationally, as cross-cultural application. We refer
to application with ethnic minorities within a specific nation as
multicultural application, recognizing the frequent overlap inherent
in cultural and minority group status.
tural clinical utility research with the CS.1 Our goal in this ar-
ticle is to encourage the development of an empirical basis
for cross-cultural and multicultural practice with the Ror-
schach through development of a methodologically sound
body of research testing its cross-cultural and multicultural
clinical utility. We begin our discussion with a review of the
current controversy regarding the cross-cultural adequacy of
CS norms, which has important relevance to Rorschach re-
search and practice.
VALIDITY CONSIDERATIONS RELEVANT
TO CROSS-CULTURAL AND MULTICULTURAL
RORSCHACH RESEARCH AND PRACTICE
A key consideration is typically glossed over by both sides of
the CS debate. Confusion is created when authors question
the validity of the Rorschach or the CS, a point now acknowl-
edged by one group of Rorschach antagonists (Garb, Wood,
Lilienfeld, & Nezworski, 2002). Underlying this question are
two fundamental misunderstandings. First, on multiple in-
dexes instruments such as the Rorschach, the validity status
associated with individual indexes and variables differs. Dis-
cussion of validity requires reference to the interpretations
associated with data from a specific CS variable or index.
Even more important from our perspective, recent contro-
versy has generally ignored a fundamental tenet of validity
theory. Empirical research does not validate a test but instead
validates the interpretations derived from test data
(Cronbach, 1971); validity is not a property of a test or index
of a test.
A point of departure for this article, which concerns itself
with study of the cross-cultural and multicultural clinical
utility of the CS, is a corollary of this basic tenet of validity
theory; validity is not a general characteristic of a test but in-
stead specific to a particular test purpose (Dana, 1962;
Wainer & Braun, 1988). Thus, interpretations from a specific
test can be highly valid for one purpose but not for other pur-
poses; interpretations may be highly valid for one ethnic or
cultural population and not valid for another cultural setting.
Validation of interpretations from CS data in different cul-
tural settings is the core concern of research that seeks to de-
velop an empirical basis for its cross-cultural and
multicultural clinical use.
Cross-Cultural and Multicultural Rorschach
Clinical Application
A recently emerging issue in the Rorschach controversy is al-
leged “shortcomings of the Rorschach for the assessment of
American minority groups and for non-Americans” (Garb,
Wood, Nezworski, Grove, & Stejskal, 2001, p. 436; see also
Lilienfeld et al., 2000). In support of their assertion, Garb et
al. cited findings of 18 authors. In the process, Garb et al.
misconstrued these authors’ intentions and the meaning of
 CROSS-CULTURAL RORSCHACH METHODOLOGY
their findings through oversimplification, omission of rele-
vant content, and an apparent failure to understand the meth-
odologies, intent, and findings of the cross-cultural and
multicultural assessment literature. This provides little in the
way of resolution of the Rorschach controversy as alluded to
in the title of Garb et al.’s article. Of greater concern, the arti-
cle obfuscates the intentions and accomplishments of current
research efforts to develop an empirical basis for
cross-cultural and multicultural Rorschach practice (e.g.,
Dana, 1993, 2000b).
In a specific example, Garb et al. (2001) cited a quote from
Dana (1993, p. 160) indicating that “the Rorschach and the
Exner Comprehensive versions are not recommended for rou-
tine cross-cultural application” (p. 436; italics added). The
complete quotation, however, refers to a preceding table of
comparisons across major projective methods indicating the
status of culture-relevant desiderata including stimuli, scores,
norms, context, and theory for several projective tests/meth-
ods (see Dana, 1993, p. 148, Table 8–1). The complete quota-
tion also indicates that there were “instances with acculturated
assesses and for particular assessment problems where either
the original Rorschach or the Exner Comprehensive Ror-
schach should be useful” (Dana, 1993, pp. 160–161). Each of
the desiderata in the Dana (1993) Table 8–1 indicated an area
in which additional research is needed to document CS valid-
ity.
The new American Psychological Association (2002) Eth-
ics Code includes a number of newly introduced subsections
under the area of assessment that are particularly relevant to
multicultural practice. It calls for use of instruments with es-
tablished reliability and validity among members of the popu-
lation tested or discussion of limitations in the case where they
do not exist, and it mandates test interpretation to consider lin-
guistic and cultural differences. It also asserts assessors may
adapt instruments “in light of the research on or evidence of”
usefulness (American Psychological Association, 2002, p.
1070). However, routine cross-cultural application of any in-
strument, including the Rorschach, follows the establishment
of standards for multicultural and cross-cultural use of the test
in assessment practice. Thus, clinical competency in
cross-cultural application is predicated on adherence to proce-
dures relative to standards (Beutler, Crago, & Arizmendi,
1986). The new American Psychological Association Ethics
Code now appears to recognize cultural adaptations derived
from cultural knowledge and clinical judgment in the adminis-
tration or interpretation of standard tests. However, in the ab-
sence of a consensually accepted research basis for such
adaptation, standards for culturally competent assessment
practice cannot exist. The absence of a body of acceptable
cross-cultural research for major assessment instruments, in-
cluding but not limited to the Rorschach, on the previously
mentioned desiderata point to areas of need for additional re-
search as a basis for practice. In this way, a paucity of equiva-
lence research has been responsible for the absence of research
based, cross-cultural/multicultural assessment standards and
191
specific procedures. This, in part, has led to allegations of
“cultural negligence” on the part of mainstream clinical psy-
chology (Kazarian & Evans, 1998).
Although we believe it is appropriate to criticize the cur-
rent research base of the Rorschach for cross-cultural prac-
tice as deficient, we also believe selective targeting of the
Rorschach (e.g., Wood, Garb, Lilienfeld, & Nezworski,
2002) is unwarranted and ignores the existence of similar de-
ficiencies in the research basis for cross-cultural practice of
most other standard personality assessment instruments.
This state of affairs can only occur within a professional con-
text that possesses limited cross-cultural and multicultural
research to guide practice, limited understanding of the
methodological issues inherent in this type of cross-cultural
research, and limited awareness of the relatively new consen-
sus on multicultural research standards (e.g., Council of Na-
tional Psychological Associations for the Advancement of
Ethnic Minority Interests [CNPAAEMI], 2000).
As Widiger (2001) suggested, “the dispute over the Ror-
schach may be a flashpoint of a wider and more fundamental
dispute within the profession” (p. 374), described further in
Howard et al. (1987). We also view this dispute as a flashpoint
but in a way additional to that envisioned by Howard et al.; it
incorporates the debate over whether culture is properly con-
sidered as a proximal or distal variable in research and prac-
tice. In their review of ethnic minority assessment research,
Okazaki and Sue (1995) organized approaches by the proxim-
ity of cultural variables in study design and interpretation to
central validity considerations;2 as an example of this distinc-
tion, they noted that ethnicity operationalized as a demo-
graphic variable places culture more distally in that ethnicity
as a demographic variable often functions as proxy for the ac-
tual culturally based personality processes of interest that un-
2
The proximal/distal is a theme that appears throughout the multi-
cultural research and practice assessment literature. In their assess-
ment practice recommendations, Sue and Zane (1987) stressed the
importance of proximal linkages that transform cultural knowledge
into concrete operations in practice. Examples of proximal linkages
include use of culture-specific assessment service delivery with re-
spect to cultural norms of helping behavior and social etiquette and
the use of information on acculturation status as moderator variables
in assessment interpretation. Similarly, the Multicultural Assess-
ment-Intervention Process model (Dana, 2000b) provides a number
of specified opportunities to employ cultural knowledge within the
assessment process per se or within a community mental health cen-
ter service delivery system (Dana, Aragon, & Kramer, 2002). Re-
search-relevant examples following Sue and Zane’s recommenda-
tions include Malgady’s (1996) suggestions for reversal of the null
hypothesis of no cultural bias to that of culture bias, thereby revers-
ing Type I and II error with regard to cultural difference based on the
consequences of potential disservice to ethnic minority clients;
Okazaki and Sue’s (1995) assessment methodology considerations
regarding the specificity of the ethnicity variable through adequate
description of the sample, or preferably through direct measure of
personality processes thought to affect test performance for which
ethnicity functions as proxy variable, is yet another example of how
cultural variables become proximal to interpretation of the data.
192 ALLEN AND DANA
derlie ethnicity, which in turn leads to a lack of specificity in
the independent variable. Proximal loci and specificity of the
cultural variable are central methodological considerations in
cross-cultural and multicultural research with the Rorschach
as well as all other assessment instruments. This lack of speci-
ficity in the research has resulted in unsystematic introduction
of numerous variables associated with culture, along with “all
manner of demographic variables such as age, sex, regional lo-
cation, interest, acculturation, [and] socioeconomic status”
(Hibbard, 2003, p. 262) that can potentially mediate observed
relationships and differences. This methodological flaw is
common in much of the current multicultural assessment re-
search, rendering clear conclusions problematic; it has partic-
ular relevance to cross-cultural comparisons of CS scores to
the CS norms.
Adequacy of CS Norms
Wood, Nezworski, Garb, et al. (2001a) assembled a large sam-
ple combining data from 32 Rorschach studies of nonclinical
populations. Wood, Nezworski, Garb, et al.’s (2001a) compar-
isons of the composite means of 14 CS variables from these
combined samples to the current CS norms yielded statisti-
cally significant differences. Based on this data, Wood,
Nezworski, Garb, et al. (2001b) asserted the current CS norms
are flawed, despite later observing, “the numbers in our article
are unsuitable for norms” (p. 398). Meyer (2001) responded by
citing numerous sampling considerations and analytic errors
that limit the interpretability of the analysis of data from
Wood, Nezworski, Garb, et al.’s (2001a) composite sample.
Additional considerations associated with multicultural
sampling concerns inherent in the Wood, Nezworski, Garb,
et al. (2001a) data expand Meyer’s (2001) critique. Contra-
vening basic methodological guidelines for ethnic minority
research (Okazaki & Sue, 1995; CNPAAEMI, 2000), the
Wood, Nezworski, Garb, et al. composite data set was cre-
ated through nonsystematic inclusion of African American
and Hispanic samples without description of their cultural
and ethnic characteristics. Although Meyer’s (2001) inter-
pretation of differences as representative of normative shifts
in the contemporary samples are reasonable, consideration of
the ethnic composition of the Wood, Nezworski, Garb, et al.
samples expands the parameters of the debate regarding the
CS norms to cultural factors. The Wood, Nezworski, Garb, et
al. findings could be due, at least in part, to the effects of their
nonsystematic inclusion of ethnic minority participants.
More broadly, any potential CS normative shifts could be
grounded, in part, in recent cultural and demographic transi-
tions in the U.S. population. Because of their neglect of im-
portant multicultural methodological considerations relevant
to their inclusion of ethnic minority data, the design of the
Wood, Nezworski, Garb, et al. study does not allow the re-
searchers to conclude that any observed differences in CS
scores, when compared to the existing CS norms, are not in-
stead due to cultural differences tapped by CS variables that
underlie the ethnic group status of a number of the
participants. These considerations emphasize an unintended
consequence of Wood, Nezworski, Garb, et al.’s cultural di-
versity critique of the Rorschach. Given the methodological
shortcomings of Wood, Nezworski, Garb, et al.’s study, an
alternative interpretation of their findings highlights unique
positive characteristics of the Rorschach as a cross-cultural
instrument, tapping potentially important cultural differ-
ences in central personality variables among ethnically di-
verse samples. We argue in the following that the
Rorschach’s unique potential as a cross-cultural assessment
instrument offers a powerful rationale for its continued use,
particularly internationally, as well as an avenue of construc-
tive response to CS criticisms. This has important implica-
tions for the study of the clinical utility of the Rorschach and
other assessment instruments.
CROSS-CULTURAL/MULTICULTURAL
CLINICAL UTILITY OF THE RORSCHACH
An important issue in the CS clinical utility debate generally
omitted in the literature is consideration of its cross-cultural
and multicultural applicability. This issue is now frequently
addressed with many other widely used psychological tests
(e.g., Dana, 2000b; Lonner, 1996; Roysircar-Sodowsky &
Kuo, 2001; Sue, Kurasaki, & Srinivaran, 1999). Given cur-
rent demographic transitions in the United States and glob-
ally, consideration of cross-cultural utility in the evaluation
of clinical utility is inevitable (Dana, 2000a; Sanz, Ritzler,
Ivanouw, & Dana, 2001).
Hermann Rorschach’s original 1921 monograph is note-
worthy in its description of the inkblot procedure as a useful
technique for investigation of psychopathology as well as cul-
tural variation in personality (Rorschach, 1921/1942). Ror-
schach contrasted cultural differences in experience type
between Bernese and Appenzeller canton residents and find-
ing it interacted with differences in the kinds and frequencies
of psychopathological disorders and specific
symptomatologies. Rorschachs’s (1921/1942) statement that
“the experience type should be very different in various people
and races” (p. 96) represents the original impetus for studies of
Rorschach cross-cultural/multicultural utility and suggested
hypotheses for factor analytic investigation of the experience
balance (eb) as a bona fide etic or cross-culturally universal
psychological construct (Dana, Hinman, & Bolton, 1977).
Lindzey (1954) noted that Hermann Rorschach provided
this preliminary data on group cultural differences because
he recognized his inkblot test’s unique potential for
cross-cultural transportability given its composition from
task stimuli of neutral, nonpictorial, visual forms requiring
no item translation. This potential remains underutilized to
this day; the CS provides brief standard instructions, reliable
coding, norms, and validity data, but the generalizability of
these CS elements beyond Euro-Americans living in the
United States remains an important empirical question.
 CROSS-CULTURAL RORSCHACH METHODOLOGY
One frequently voiced concern involves a critique of the
projective hypothesis as culture specific (cf. Draguns, 1990).
This critique appears to confuse three distinct issues. First,
the CS conceptualizes Rorschach scoring as primarily as-
sessing a perceptual and associational (or informa-
tion-processing) task (Exner, 1993); projection may or may
not be part of many Rorschach responses, and typical proto-
cols do not necessarily include significant projective materi-
als. Substantive CS interpretative data describing the person
can occur without consideration of projection. Second, these
critiques generally examine inappropriately applied cul-
ture-specific interpretations of projection by the examiner
(Draguns, 1990; Okazaki & Sue, 1995). These critiques re-
duce to legitimate critiques of theoretical and interpretative
formulations regarding projective material that are embed-
ded in worldviews incongruent with the cultural context of
the examinee rather than direct critiques of the projective hy-
pothesis itself. Third, ethnic match may influence projection;
indeed, issues and implications associated with exam-
iner–examinee ethnic match and cultural competence of the
examiner constitute important ramifications of the projective
hypothesis unexamined in the assessment literature.
In summary, attributes of the Rorschach afford it a unique
potential as a truly international assessment instrument dur-
193
ing a time when clinical utility will be increasingly under-
stood in terms of an instrument’s cross-cultural and multicul-
tural applicability. Important methodological issues arise in
research that explores the cross-cultural and multicultural
clinical utility of the Rorschach.
METHODOLOGICAL ISSUES
IN CROSS-CULTURAL AND MULTICULTURAL
RORSCHACH RESEARCH
Core methodological cross-cultural/multicultural issues in
Rorschach research include (a) linguistic equivalence, (b)
setting and instructional set equivalence, (c) interrater coding
reliability, (d) cross-cultural/multicultural norms, (e) accul-
turation status norms, (f) predictor bias, (g)
cross-cultural/multicultural construct equivalence, and (h)
construct validation research. Descriptions and recommen-
dations for each of these methodological issues are summa-
rized in Table 1. Some of these issues are common to cultural
research with other assessment instruments; in such cases,
we provide a brief comparative overview of recent findings,
most notably with the MMPI–2. Other multicultural method-
ological issues are more specific to the Rorschach and not of
direct relevance to self-report instruments.

TABLE 1
Core Cross-Cultural Methodological Issues

Issue Description Recommendation

Linguistic equivalence
Setting/instructional set
equivalence
Interrater coding reliability
Cross-cultural/multicultural
norms
Acculturation status norms
Predictor bias
Screen for cross-cultural
construct equivalence
Cross-cultural construct
validation research
Address language translation and
ethnic/regional differences in English
usage
Cross-cultural appropriateness of
standard administration
Comparability of cross-ethnic and
cross-national samples
Correct use of cross-national/cross-ethnic
comparisons of norms
Establishment of norms for identification
of cultural orientation of assessee
Identification of potential bias in
Rorschach prediction
Hypothesis testing research to disconfirm
null hypothesis of cross-cultural
nonequivalence of Rorschach variables
(e.g., Malgady, 1996)
Cross-cultural construct validity research
Standard procedures to translate instructions including
prompts
Empirical study of administration, interactional style, and
meaning in terms of cultural sociolinguistic conventions
Empirical study of examiner–examinee ethnic match
Implement existing recommendations for research studies
along with careful specification of any differences in
cross-national coding procedures
Use limited to screening for group differences in CS variable
scores; difference may demonstrate either (a)
nonequivalence of the measure or (b) valid measurement of
cultural difference on underlying construct; only
differential validity studies can address meaning of group
difference scores
Minimize pathologization of individuals with traditional
cultural orientations by altering clinician through use of
potentially inappropriate normative data and nonequivalent
constructs
Detection of slope and intercept bias in Rorschach-based
predictions of psychopathology and personality (note
implicit assumption of construct equivalence)
Regression of cultural identity measures on CS variables
Item Response Theory analysis
Confirmatory Factor Analysis
Construct validation studies for CS variables among cultural
groups of interest, including mixed quantitative/qualitative
ethnographic designs
194 ALLEN AND DANA
Linguistic Equivalence
Understanding of the current CS instructions as intended by
the CS developers, with their combination of brief initial in-
structions and often complex prompts, may require a particu-
lar style of abstraction that is itself culture specific. However,
the linguistic equivalence of these instructions has never
been formally studied either for translations (Geisinger,
1994; Van de Vijver & Hambleton, 1996), or equally impor-
tant, for ethnic/regional differences in English usage within
and outside of the United States (for an MMPI–2 Singapore
example, see Wuts, 2002). Standard systematic translation
procedures such as back translation or committee procedures
can be readily applied to the entire CS administration process
including the opening instructions and prompts. Following
this, evaluation through field testing is necessary to examine
several important issues, most prominent the applicability of
translation/adaptation by bilinguals for use with monolingual
populations that may not share the same linguistic habits and
implicit understandings (Nichols, Padilla, & Gomez-
Maqueo, 2000). The issue of linguistic equivalence is rele-
vant to all standard personality instruments; the MMPI–2 has
been translated into more than 150 languages using
back-translation procedures (Butcher, 1996). However, field
testing of these MMPI–2 translations has received limited at-
tention in the literature.
Setting/Instructional Set Equivalence
As is the case with linguistic equivalence, despite the
lengthy, intensive one-on-one examiner–assessee interaction
required in CS administration, there is also little research on
the cultural appropriateness of the standard CS administra-
tion procedure among ethnic minority groups in the United
States and internationally. Culturally appropriate
interactional styles are directed by sociolinguistic conven-
tions. These conventions include such features as appropriate
pacing of communication and modes of back channeling.
Adherence to these conventions is critical in the establish-
ment of rapport and can affect task orientation, defensive-
ness, response frequency, and a myriad of associated CS vari-
ables. This is because the sociolinguistic conventions of each
culture convey meaning and in addition convey a comfort-
able and credible interactional style for the assessee. Finally,
although discussion of its potential importance appears in re-
cent Rorschach literature (e.g., Presley, Smith, Hilsenroth, &
Exner, 2001), the impact of ethnic match between examiner
and examinee during Rorschach administration has not been
explored. Ethnic matching is empirically related to public
sector managed care treatment outcomes in the United States
(Gamst, Dana, Der-Karabetian, & Kramer, 2000, 2001) as
well as to treatment factors and outcome in the psychother-
apy (Atkinson, 1983; Maki, 1999).
Although self-report instruments typically require limited
interaction with the examiner, they involve written instruc-
tions. Difficulties with self-report instruments and their asso-
ciated instructional sets have been described in use with
groups ranging from unacculturated non-Western individu-
als to counselors completing multicultural competence in-
ventories (Pope-Davis, Liu, Toporek, & Brittan-Powell,
2001). Despite this, instructional set equivalence has not yet
been systematically studied with standard self-report assess-
ment instruments such as the MMPI–2.
Interrater Coding Reliability
Interrater reliability does not arise as a concern for self-report
instruments such as the MMPI–2; however, it comprises a
crucial concern in research and clinical application of com-
plex scoring systems for verbal responses such as those pro-
duced by the Rorschach. Introduction of the CS provided ma-
jor improvements in interrater coding reliability among 95%
of scoring variables (see review in Weiner et al., 2002). Con-
temporary Rorschach studies using the CS employ clearer
coding guidelines and use more sophisticated measures of
agreement (Meyer et al., 2002; Viglione & Hilsenroth, 2001)
than pre-CS studies (e.g., Dana, 1955; Voigt & Dana, 1964).
Cross-cultural and especially international research requires
careful specification of CS coding procedures, examiner
training in consensual practice, use of training materials with
high and low base rate coding scoring, and ongoing
evaluative feedback to examiners and scorers.
Informal reports of international variations in inquiry and
coding suggest that local researchers must clearly specify
their procedures. Cultural adaptations may dictate departures
from CS procedures in research studies. However, reports
must (a) identify all such departures from CS procedures, (b)
provide the rationale for departures, (c) discuss potential im-
plications, and (d) rename adapted and nonequivalent scor-
ing variables. Without these cautions, cross-cultural
comparisons become hazardous; nonequivalent and contra-
dictory cross-cultural findings can result from the presence
of undocumented CS alterations that distort scoring vari-
ables. This distortion can lead to artifactual findings of group
norm differences, factorial variance, or differential validity
and erroneously suggest the existing nomological net vali-
dating an underlying construct is not transferable across cul-
tural data sets when instead, changes in scoring procedures
have resulted in variables of interest that are different across
data sets.
Cross-Cultural/Multicultural Norms
There are no generally accepted multicultural norms for the
Rorschach or other standard assessment instruments in the
United States. Instead, in the case of the MMPI–2, ethnicity
is typically considered one of several demographic variables
stratified through sampling procedures; the resulting com-
posite normative data is used in interpretation (Butcher et al.,
1989). In the case of the Rorschach, national CS norms for
 CROSS-CULTURAL RORSCHACH METHODOLOGY
countries outside the United States are not available with the
exception of Portugal (Pires, 2000). However, there are a
number of normative adult samples for the CS from Argen-
tina, Belgium, Denmark, Finland, Israel, Italy, Japan, Peru,
and Spain (Meyer, 2001; T. Schaeffer, personal communica-
tion, November 3, 2002), and comparable samples from Ven-
ezuela and Chile are cited in Vinet (2000). The availability of
this cross-cultural data allows researchers to screen for dif-
ferences between cultural groups on CS variables similar to
the 25 comparative MMPI and MMPI–2 studies in the United
States that permitted a recent meta-analytic investigation of
ethnicity and the MMPI (Hall, Bansal, & Lopez, 1999). The
CS comparative research to date has produced mixed results
regarding cross-cultural differences.
Wood and Lilienfeld (1999) cited 11 studies in which they
asserted U.S. ethnic minorities and non-Americans score dif-
ferently than U.S. Euro-Americans on key Rorschach vari-
ables. However, only 6 of these studies used the CS, and of
these, 1 study contrasted results qualitatively on only three
variables with the CS norms (Sangro, 1997), whereas the re-
maining 5 studies compared targeted minority convenience
samples to the Exner normative data. Elsewhere, Wood,
Nezworski, Stejskal, Garven, and West (1999) argued such
targeted sample comparisons with the Exner norms are meth-
odologically flawed.
Meyer (2001) tested for differences among 69 CS variables
in samples from Argentina, Belgium, Denmark, Finland, Ja-
pan, Peru, Portugal, and Spain when compared to the CS
norms. Meyer found differences cross-culturally; however,
these differences averaged only .4 SDs, which he interpreted
as statistically but not clinically significant. Wood,
Nezworski, Garb, et al. (2001b) reanalyzed this same data set,
restricting their analyses to only 14 of the 69 variables they as-
serted were selected because they were central to CS interpre-
tation. Wood, Nezworski, Garb, et al. reported differences on
these selected variables averaging .8 SDs, which they inter-
preted as clinically meaningful differences.
Finally, Ritsher, Slivko-Kolchik, and Oleichik (2001)
found similar means in 104 of 107 Rorschach variables when
comparing a Russian inpatient sample with depression to
Exner’s (1993) U.S. sample of 315 with depression; how-
ever, Scale 3 (Depression) from the Russian version of the
MMPI but not the CS DEPI scale functioned as a valid indi-
cator of depression in this sample. In summary, Rorschach
proponents and antagonists disagree on the magnitude,
meaning, and importance of findings of group difference on
CS variables cross-culturally.
In the United States, initial between-group comparative
studies of CS norms have been done with African Ameri-
cans. Presley et al. (2001) found only one clinically signifi-
cant difference among 23 core Rorschach variables in a
comparison between 44 African Americans matched on key
demographic variables with 44 Euro-Americans, all se-
lected from the CS nonpatient norm group. Several studies
with the MMPI and MMPI–2 have found African Ameri-
195
cans score higher on MMPI and MMPI–2 clinical scales
(Frueh, Smith, & Libet, 1996; Graham, 1990; Greene,
1987), leading to continued calls for separate ethnic group
norms (Lindsey, 1998). These findings of differences on
the MMPI and MMPI–2 scales have often been small, in-
consistent, and difficult to replicate (e.g., Frueh, Gold, de
Arellano, & Brady, 1997). Hall et al. (1999) found only
small effect sizes for the demographic variable of race/eth-
nicity across the 37 studies used in their meta-analysis. In
summary, the existence and meaning of ethnic group norm
differences on the MMPI/MMPI–2 remains controversial
among MMPI–2 proponents and antagonists.
Resolution to this controversy is found through noting that
problems with these types of between-group comparisons
have been described for years (e.g., Azibo, 1988; Greene,
1987; Okazaki & Sue, 1995; Pritchard & Rosenblatt, 1980;
Roysircar-Sodowsky & Kuo, 2001). The finding of a group
difference on a Rorschach, MMPI, or other personality as-
sessment instrument variable does not permit conclusions re-
garding its meaning. It is unclear whether a between-group
difference is due to (a) a real difference in personality or
psychopathology attributes, (b) nonequivalence of the mea-
sure or its metric properties, or (c) a shift that has occurred
across cultures in the underlying construct tapped by the
measure and its associated nomological network (Allen &
Walsh, 2000). “Accordingly, such obtained differences have
no necessary implications for the validity of using a test with
diverse cultures” (Weiner et al., 2002, p. 8). Research ques-
tions regarding the cross-cultural appropriateness of the Ror-
schach or any psychological test are better addressed through
methodologies, described following, that explore the relation
of test variables among a cultural or ethnic group to (a) a
cross-culturally equivalent criterion variable or (b) the un-
derlying nomological net of the variable through tests of con-
vergent and divergent validity.3
Acculturation Status Norms
Investigation of potential multicultural group differences
raises accompanying questions regarding ethnic group
norms and acculturation status norms. The conventional as-
sessment view militates against establishment of norms for
all ethnic subgroups because of the within-group heterogene-
ity of scores. Because of this, the development of local norms
in the United States has been an infrequent recourse limited
primarily to geographically isolated and unacculturated in-
digenous groups (Dana, 1993). Neither the CS nor MMPI–2
provides ethnic group norms.
3
Moreover, despite Malgady’s (1996) cogent critique advocating
the directionality of the null hypothesis in ethnic minority assess-
ment research be set to assumption of cultural difference, given the
potential deleterious impact on ethnic minority clients of improper
test use, most research on group differences tests the hypothesis of
no group difference as the null.
196 ALLEN AND DANA
The development of acculturation norms is now important
to consider because of the documented importance of accul-
turation status across ethnic minority groups and the con-
founding of traditional acculturation status with
psychopathology (Dana, 2000a).4 Acculturation status repre-
sents both an important source of heterogeneity within ethnic
groups and a specific variable of interest underlying ethnic
group membership. The rationale for acculturation status
norms is supported by empirical research with the
MMPI/MMPI–2 among the four major ethnic/racial minority
groups in the United States (Dana, 2000a; Whatley, Allen, &
Dana, 2003). In this research, relationships between accul-
turation/cultural identity status and MMPI/MMPI–2 clinical
scale scores have been reported in studies with Asian Ameri-
cans (Sue, Keefe, Enomoto, Durvasula, & Chao, 1996), His-
panic Americans (Montgomery & Orozco, 1985; Whitworth
& McBlaine, 1993), American Indians (Hoffmann, Dana, &
Bolton, 1985; Pollack & Shore, 1980), and African Ameri-
cans (Whatley et al., in press). Additionally, acculturation
status emerges as a performance correlate for Hispanics on
standard neuropsychological tests (Arnold, Montgomery,
Castanada, & Longoria, 1994).
The availability of acculturation status norms for the four
major ethnic groups in the United States could minimize
pathologization of traditional individuals. By allowing for
assessment of acculturative status, clinicians can then iden-
tify traditional individuals for whom the existing normative
data for the CS and other psychological assessment instru-
ments such as the MMPI–2 may be inappropriate as well as
other ethnic minority individuals for whom standard inter-
pretations may be relevant. As a global society with increas-
ingly permeable borders, it is now commonplace to find
substantial multicultural populations with individuals who
retain their traditional cultural orientations even as
biculturalism becomes more the norm in the immediate fu-
ture (Birman, 1994), along with increasingly large numbers
of genuinely international persons who possess skills and
knowledge from multiple cultural perspectives. In this way,
development of acculturation status norms would also help
assure the clinical utility of the CS internationally while ad-
vancing its multicultural clinical utility in the United States.
The Multicultural Assessment-Intervention Process model
(Dana, 2000b) procedure provides one model for their use in
clinical practice.
4
Acculturation refers to changes in the traditional culture patterns
of groups resulting from continuous, first-hand contact, whereas ac-
culturation status refers to changes in the individual during this pro-
cess as described by assimilation, biculturalism, marginalization,
and traditional cultural orientations (Dana, 1993). Acculturation sta-
tus roughly indicates the locus of an individual in a personal process
of cultural change using orthogonal measurements of the extent to
which original culture characteristics are retained and new or host
culture characteristics are incorporated (Cuéllar, 2000).
Predictor Bias
Although often described as a technique to identify test bias,
the concept of predictor bias can be understood more broadly
from the perspective of basic validity theory, here referring to
the case in which there exists differential validity for a test
score in the prediction of a criterion variable for individuals
who are members of a particular ethnic group. In the most
comprehensive cross-ethnic study of predictor bias5 in a per-
sonality instrument to date, Meyer (2002) explored the rela-
tionship of ethnicity to Rorschach variables in a sample of
432 Euro-American, African American, Hispanic American,
Asian American, and American Indian psychiatric patients
matched on key demographic variables. Using the approach
of Nunnally and Bernstein (1994), Meyer (2002) found no
evidence of slope bias on 17 key CS variables in comparisons
of Euro-Americans to African Americans and of
Euro-Americans to a combined group that included all four
ethnic minority groups. However, 4 of these CS variables, all
associated with psychotic disorder, displayed moderate inter-
cept bias in the direction of underprediction of
psychopathology among African Americans, and 3 of these 4
variables displayed intercept bias, again in the direction of
underprediction of psychopathology, among the composite
ethnic minority group.
Similar research with the MMPI–2 has been limited to Af-
rican American samples compared to Euro-Americans.
McNulty, Graham, Ben-Porath, and Stein (1997) and
Timbrook and Graham (1994) both failed to find evidence of
slope bias in MMPI–2 based predictions of
psychopathology. Arbisi, Ben-Porath, and McNulty (2002)
found mean differences between clinical and validity scales
5
Nunnally and Bernstein (1994) and Anastasi and Urbina (1997)
have described procedures to identify differential validity of the
meaning of tests scores between groups. The relation between a pre-
dictor variable and a conceptually relevant criterion variable can be
explored for systematic differences between groups in the slope of
the regression line produced by this relationship. In cross-cultural
and ethnic minority research, a significant difference in the correla-
tion coefficients between groups for a CS variable with a criterion
variable across the range of the CS variable scores would provide ev-
idence for a difference in the accuracy of its prediction of the crite-
rion variable scores. This is termed slope bias and can be considered
evidence of differential validity for the variable across groups. How-
ever, a predictor variable can systematically overpredict or
underpredict the criterion variable for an ethnic minority group with-
out displaying a between-group difference in the degree of associa-
tion between the variables. Nunnally and Bernstein (1994) and
Lautenschlager and Mendoza (1986) have described two proce-
dures, moderated regression and step-down hierarchical regression,
for identifying both slope bias and this second type of systematic
measurement error across groups, termed intercept bias. Intercept
bias is an important and less often studied aspect of predictor bias;
Lopez (1989) noted the importance of study of both overprediction
and underprediction of psychopathology, which has received less at-
tention, across groups in cross-cultural and multicultural assessment
research.
 CROSS-CULTURAL RORSCHACH METHODOLOGY
when scores of African Americans were compared to
Euro-Americans and, in the first study of both slope and in-
tercept bias with the MMPI–2 using the procedures of
Lautenschlager and Mendoza (1986) in 65 comparisons,
found important Gender × Ethnicity interactions. Intercept
bias was found in 32 MMPI–2 scale-criterion comparisons
with men and 12 comparisons with women.6 Overprediction
of psychopathology was found in 7 cases, the remaining
cases involved underprediction.
Meyer (2002) described a number of methodological limi-
tations to his study. Most important, Meyer (2002) noted his
interpretation of findings (as with any predictor bias study)
assume the external correlates of behavior are equivalent.
Meyer (2002) included education level, psychotic and de-
pressive disorder diagnoses, maximum impairment associ-
ated with the diagnosis, and composite MMPI–2 scores as
criterion variables; Arbisi et al. (2002) used comprehensive
record review ratings with the additional methodological
strength of checks for interrater reliability, of demographics,
Diagnostic and Statistical Manual of Mental Disorders
(American Psychiatric Association, 1987 [DSM–III], 1994
[DSM–IV]) diagnosis, and 87 mood, affect, cognition, and
organic indicator ratings derived from the charting of the
psychiatrist’s interview. The creation of criterion variables in
these studies was well conceived and methodologically quite
sophisticated; however, the variables rely on clinician re-
cords, and MMPI–2 scores as their data sources, all of which
are the topic of ongoing controversy regarding cross-cultural
equivalence.
For example, the cross-cultural accuracy of clinical judg-
ment and diagnosis by clinicians has been a topic of immense
historical controversy (Dana, 2001). Similarly, diagnosis and
psychopathology can differ in important ways
cross-culturally. In one example, Manson (1995) reviewed
factor analytic research with the Center for Epidemiological
Studies–Depression Scale (Radloff, 1977), a standardized,
widely used composite measure of depressive
symptomatology developed by researchers at the National In-
stitute of Mental Health for epidemiological studies that in-
cludes items from several previously established depression
scales. Important differences emerged in the internal structure
of the instrument across Chinese American, Mexican Ameri-
can, and American Indian groups, which Manson interpreted
as reflective of a change in the underlying patterning of affect
in depression cross-culturally; this renders important facets of
the construct nonequivalent. In addition, regression can not
establish causality, leaving unanswered several intriguing
questions. In the case where intercept bias indicates
underprediction, it remains unclear whether the observed
6
However, these differences, at least in part, were likely related to
the sample size for women in the study (Arbisi et al., 2002) that was
comparatively smaller than the male sample; therefore, a larger ef-
fect size was required to attain significance among women.
197
underprediction of psychopathology reflects differential va-
lidity in the instruments tested, or if the instruments are instead
identifying bias among clinicians in their assignment of diag-
noses (in this case, overpathologizing behavior in ethnic mi-
nority individuals), or if the instruments have identified cases
in which the diagnostic criteria are nonequivalent, or also
likely, that there exists some interaction of these factors.
Several additional methodological considerations are rel-
evant to the interpretation of findings from predictor bias
studies. Meyer (2002) noted that the cases of differential va-
lidity he observed in his study (as in the case on all predictor
bias studies) could be related to one or more unmeasured and
uncontrolled third variable (or variables; Anastasi & Urbina,
1977; Linn & Werts, 1971). This is a key point; we argue un-
measured differences in variables associated with racial and
cultural identity processes could comprise just such an im-
portant third variable in these predictor bias studies.7 All the
existing predictor bias studies in the literature provide lim-
ited description of their samples with regard to accultura-
tion/identity status; this represents an important empirical
question for future predictor bias studies. Nunnally and
Bernstein (1994) described how an analysis of the residuals
from the regression analysis can shed important light on po-
tential third variables and assist with their identification.
None of the existing predictor bias studies have reported re-
sidual analysis, which would be a useful analytic strategy for
future studies. Further complicating the interpretation of re-
gression analyses in predictor bias studies (and analysis of
their residuals), Cuéllar (2000) discussed how culture can
function as both a mediator variable influencing both the pre-
dictor and criterion and a moderator variable altering the
strength and direction of the relationship between predictor
and criterion. Because of the numerous complexities inher-
ent in the interpretation of predictor bias studies, although
they provide important screening tests regarding the
cross-cultural functioning of scales and measures, they inevi-
tably leave unanswered important questions. These unre-
solved questions are best addressed through direct tests of the
7
Typically, controlling for a third variable is done to reduce vari-
ance extraneous to the criterion variable, which is assumed unrelated
to the predictor. Reducing extraneous variance allows a clearer test
of the relationship between criterion and predictor. However, we are
including in our consideration of unmeasured third variable(s) those
personality processes that underlie ethnic and cultural group mem-
bership, such as acculturation and cultural identity processes, or
better still, specific psychological variables that underlie identity. In
this case, we are proposing third variables that both are related to
group status and are hypothesized to measure processes underlying
group status. This is an important point because simply treating an
identity variable as a third variable to be statistically controlled for
(e.g., partialed out) would reduce effects in regression analyses and
disguise its impact as a moderator variable when instead, covariance
with predictor and criterion may be indication that it functions as the
variable of interest underlying group status.
198 ALLEN AND DANA
cross-cultural construct validity and equivalence of the inter-
pretations of data from test variables.
Multicultural and Cross-Cultural Construct
Equivalence and Construct Validation Research
With the Rorschach
As the science of clinical psychology developed and empha-
sized an empirical basis for practice, the Rorschach remained
viable as an assessment instrument through efforts directed to-
ward its adaptation into a test.8 However, Weiner (1994) as-
serted the Rorschach is a method rather than a test. According
to this view, the Rorschach inkblot method (RIM) is composed
of a variety of measures, scales, and indexes that tap numerous
and often overlapping constructs, each supported by varying
degrees of validity data and each possessing different
psychometric properties, operating characteristics, and vary-
ing degrees of conformity to the assumptions of classical
psychometric theory. Therefore, the RIM is best understood as
8
A test, as defined by Anastasi and Urbina (1997), is an objective
measure of a sample of behavior standardized to ensure uniformity
of procedures using norms from a large, representative sample with
demonstrated reliability and validity predicated on independent ex-
ternal criteria. Dahlstrom (1993) provided additional specificity by
defining a psychological test to include the presence of six criteria:
“(a) standardized materials and procedures, (b) optimal motivation,
(c) immediate recording, (d) objective scoring, (e) appropriate
norms, and (f) established validity” (p. 395), although we again must
clarify the important distinction that validity is not a property of a
test but instead the interpretations made from test data, which can
change based on the use intended. The CS makes significant inroads
toward fulfilling both the Anastasi and Urbina (1997) and the
Dahlstrom (1993) criteria for a test; however, numerous CS vari-
ables display violations of assumptions for the application of many
of the commonly used methods of inferential statistics used to test
their validity (e.g., numerous CS variables display non-normal dis-
tributions of scores that are skewed and kurtotic; cf. Viglione, 1995,
1997). In addition, issues fundamental to the Rorschach task system-
atically influence scores, most prominently the confounding influ-
ence of individual differences in response frequencies (Meyer,
1992; Meyer & Archer, 2001). There exists limited detailed exami-
nation of violation of assumptions such as normality and
homoscedasticity for many of the statistical methods most widely
used in validation work with most tests, scales, or items, including
those found on the Rorschach and MMPI–2. For example, recent
work in confirmatory factor analysis and linear structural modeling
provides evidence that many item and scale score distributions on
numerous tests (or more precisely, their error distributions because
the other components are typically constants) are exceedingly
skewed, or peaked/flattened, or both. However, we are also learning
that robust estimation procedures, particularly maximum likelihood,
provide a reasonable remedy for this problem (West, Finch, &
Curran, 1995). In summary, “these [psychometric] methods require
only a definable population of numbers that meets the assumptions
in the particular statistical method” (Nunnally & Bernstein, 1994, p.
23), and we are discovering many of our psychological measures in-
cluding the MMPI–2 share with the Rorschach both numerous viola-
tions of these assumptions and the existence of remedies for them.
Careful examination of these issues for most all scales and items is
still in its early stages.
a set of complex measures of personality structure and dynam-
ics including (a) attention, perception, and cognition; (b) the
experience and expression of emotion; and (c) attitudes toward
self and others. A substantial body of literature from cultural
psychology has now conclusively demonstrated important
ways in which normative personality attributes associated
with cognition, emotion, and self-representation vary widely
across cultural groups in intensity, form, and meaning (Church
& Lonner, 1998; Kitayama & Markus, 1994; Markus &
Kitayama, 1991; Mesquita & Frijda, 1992; Oyserman, Coon,
& Kemmelmeier, 2002; Triandis & Suh, 2002). Some CS vari-
ables that comprise one component of the RIM likely tap cul-
ture specific or emic constructs, other variables probably tap
culturally universal or true etic constructs, and still other vari-
ables may include both emic and etic content (e.g., Populars).
CS variables as well as the underlying constructs they tap may
be universal or culture specific. However, the cultural univer-
sality of specific Rorschach variables remains an empirical
question.
This is not to suggest that research will fail to provide sup-
port for certain CS variables functioning as measures of etic
constructs. For example, the process of introver-
sion–extraversion may represent a culturally universal or etic
construct, perhaps even possessing similar distributions
across groups (Eysenck, 1991). Although Rorschach and
Exner (1993) both highlighted important ways in which
Erlebnistypus (EB) differs from the broader construct of in-
troversion–extraversion, it is reasonable to hypothesize that
the two constructs share linking nomologicals or that EB
may even be a facet of the broader construct. Initial research
is encouraging regarding the potential for cultural universal-
ity of the underlying dimensions tapped by EB (Dana, 1993;
Dana et al., 1977), but further research is needed.
Weiner (2000) proposed that effective clinical applica-
tion of the Rorschach as a method makes use of all relevant
Rorschach data in the formulation of interpretive state-
ments to describe personality processes. Weiner (YEAR)
emphasized a critical second stage in clinical interpretation
with the RIM involving the specification of the levels of
certainty associated with each interpretative statement. This
requires strong assessor background and knowledge base in
the empirical support for the specific Rorschach variables
utilized in each interpretative statement. In contemporary
multicultural Rorschach clinical application, an implication
of this viewpoint is that RIM interpretations are best devel-
oped using nomothetic data along with idiographic data
grounded in the cultural context of the assessee. Use of
nomothetic data requires a knowledge base of the
cross-cultural clinical utility of the CS interpretative vari-
ables in question. Use of idiographic data is only possible
when the assessor is skilled and culturally competent in
work within the assessee’s cultural group. For assessment
science in international and multicultural settings within the
immediate future, nomothetic data sources from objective
tests in conjunction with nomothetic and idiographic data
 CROSS-CULTURAL RORSCHACH METHODOLOGY
sources from methods such as the Rorschach provide a
framework for assessment practice. Such an approach holds
potential to combine instruments possessing culturally uni-
versal (etic) and culture specific (emic) elements guided by
empirical findings from a common cross-cultural/multicul-
tural research rubric (Dana, 2003).
METHODOLOGIES FOR CROSS-
CULTURAL/MULTICULTURAL VALIDATION
RESEARCH WITH THE RORSCHACH
Between-group comparisons of scores on Rorschach vari-
ables provide only a screening test of limited utility in investi-
gations of the hypothesized cultural universality of Rorschach
variables. Within the United States, accompanying accultura-
tion data is necessary for these comparisons to have meaning.
To treat ethnicity as a demographic variable compromises the
internal validity of research designs, as the psychological vari-
able of interest is not measured but assumed to function
through its demographic proxy variable. Cultural identity may
represent a more relevant variable when considering the use of
Rorschach normative data (Presley et al., 2001), and of even
greater interest to personality researchers in both international
cross-cultural and multicultural research are the specific per-
sonality processes underlying cultural identity that are respon-
sible for any observed differences on CS variables. Continued
development of this comparative basis would best be advanced
through national norms on such identity variables for coun-
tries in which significant differences exist on CS variables
compared to the U.S. norms, as with acculturation/cultural
identity status norms for the major multicultural groups within
the United States.
Currently, international CS norms are being complied in
the absence of accompanying equivalence research. We
have described a number of important methodological con-
cerns associated with this absence of equivalence research
in the literature. Most prominent, a finding of difference in
norms can mean several things including the measure or its
underlying construct has changed or the groups differ on
the construct of interest. Similarly, the instance in which
CS normative data matches across cultural groups is not a
direct empirical test of cross-cultural construct equivalence
and is prone to the effects of numerous confounding vari-
ables. Predictor bias testing, although a useful methodolog-
ical advance on comparative studies that focus on
differences between scores across groups, has its own set of
methodological difficulties, most prominent, the assump-
tion that the criterion variable used is equivalent. In light of
these important methodological considerations that impact
the interpretability of even well designed comparative stud-
ies, research is also needed that screens specific CS vari-
ables in a more rigorous manner for potential cross-cultural
construct nonequivalence and then directly tests the
cross-cultural construct validity of these CS variables.
199
Regression of Measures of Cultural Identity
on CS Variables
Cultural identity can be understood as a broad concept that
subsumes the construct of acculturation and comprises an as-
pect of self-concept (Rosenberg, 1979). Cultural identity en-
compasses the totality of thoughts and feelings that relate to
the social and ethnic identities associated with the cultural self.
Broadly stated, measures of cultural identity tap important
facets of these social and ethnic identities. Many of the CS
variables are intended to tap dimensions of psychopathology.
If a CS variable tapping psychopathology, when regressed on a
measure of normative culture identity for members of that cul-
tural group, produces a statistically significant regression
equation, it suggests one of two possibilities: (a) cultural iden-
tity measures are tapping identity development processes as-
sociated with psychopathology, or (b) the variable is instead
pathologizing cultural difference; in this case, normative cul-
tural identity processes and culturally sanctioned modes of be-
havior explain a substantial portion of the variance in what the
CS variable is labeling as psychopathology. This becomes an
important research question. Measures of psychopathology,
when conceptually equivalent within an ethnic minority
group, generally should not covary with measures of norma-
tive group identity processes within the group (Allen & Walsh,
2000).9 Should such association occur, given the potential for
harm inherent in alternative (b), further research is needed to
identify which competing interpretation is accurate, and this
becomes an important additional component of the
cross-cultural validation process for the variable. Routine ad-
ministration of cultural identity variable measures in ethnic
minority research would allow for routine tests of this impor-
tant question. By providing an understanding of the relation of
test variables to specific facets of identity processes and their
associated psychological processes, the routine use of identity
measures would move cross-cultural and ethnic minority as-
sessment research in the direction of providing greater speci-
ficity in the variables of interest, which are the psychological
variables that underlie group membership for which group
membership as a demographic variable serves as proxy
(Okazaki & Sue, 1995). For CS variables that do not tap
psychopathology, such a relationship with a cultural identity
measure could either represent cultural difference or a shift in
the nomological network underlying the construct across cul-
tural groups; in this latter case, the variable is not
cross-culturally transportable.
9
This is not meant to assert that it can never be the case that the
prevalence of psychopathology varies between racial, ethnic, or cul-
tural groups or in the more restricted sense that psychopathology can
never vary with normative cultural identity within a group. Both
cases are possible, however, as we outline in this article, numerous
associated methodological issues and empirical questions must be
addressed before such conclusions can be drawn.
200 ALLEN AND DANA
As is the case with MMPI–2 research, most Rorschach
studies to date with ethnic minorities treat ethnicity as a de-
mographic variable. However, the variable of interest in psy-
chology is instead the specific culturally embedded
psychological variables responsible for an observed differ-
ence on a Rorschach score. An important goal of a
cross-cultural and multicultural Rorschach research program
to guide practice is an understanding of the role of specific
culturally mediated variables that underlie identity and ac-
count for observed group differences in CS variables that tap
cross-culturally equivalent constructs. For example, facets of
the culturally mediated variable of self-concept, such as indi-
vidualism–collectivism (Oyserman et al., 2002), may poten-
tially contribute to an understanding of meaning of the
observed cultural difference in scores on a particular CS vari-
able and in addition, an enhanced cross-cultural understand-
ing of the underlying construct tapped by the variable. Both
of these understandings are goals of cross-cultural and ethnic
minority assessment research as articulated in the method-
ological perspectives offered by Okazaki and Sue (1995).
Factor Analytic and Item Response Theory
Approaches
Much of the controversy in cross-cultural and multicultural
use of the Rorschach and other psychological tests has fo-
cused on evidence or lack of evidence of bias. In multicul-
tural and cross-cultural personality assessment, one way to
conceptualize many of the commonly described types of bias
(e.g., Van de Vijver & Poortinga, 1997) is through the logic
of construct validation. Test bias, from this viewpoint, occurs
when a test does not measure equivalent constructs across
groups; this can be the result of the underlying construct
shifting between cultural groups or the measure in question
displaying different measurement characteristics across
groups, leading interpretations from test data to possess dif-
ferential validity across groups. Viewed from this light, the
central question in multicultural and cross-cultural research
with the Rorschach or any other standard psychological as-
sessment instrument is that of cross-cultural construct valid-
ity. Tests of CS structural validity, most notable confirmatory
factor analysis (CFA), and tests of differential item function-
ing (DIF), most notable item response theory (IRT) analysis
described elsewhere (Allen & Walsh, 2000) along with the
regression of cultural identity measures on CS variables can
provide screening tests to identify promising CS variables for
the more painstaking process of cross-cultural validation.
Meyer (2002) provided an initial structural test of the CS
with a sample of 432 Euro-American, African American, His-
panic American, Asian American, and American Indian psy-
chiatric patients matched on key demographic variables. No
significant differences emerged when the factor structure of
67 key CS variables produced by Euro-Americans was com-
pared to that of a composite sample of non-Euro-Americans
using Jensen’s (1980) method of computation of coefficients
of similarity among principal components loadings adjusted
for family-wise error rate. Because the combination of numer-
ous different groups with limited description of their cultural
and ethnic backgrounds into one non-Euro-American group
potentially obscures important differences between different
ethnic minority groups, replication of these promising find-
ings with separate samples from each of the distinct ethnic mi-
nority groups in the United States is needed. This type of
analysis is also termed the assessment of measurement
invariance across groups. Jöreskog and Sörbom (1989) de-
scribed extensions of CFA for use with multiple samples well
suited to the study of measurement invariance. Multigroup
CFA is currently possible using existing international data de-
scribed by Meyer (2001) and T. Schaeffer (personal commu-
nication, November 3, 2002) and would provide for an
immediate and important test of the factorial invariance of the
CS across multiple international samples.
Thissen, Steinberg, and Wainer (1988) provided a brief dis-
cussion of IRT’s uses in the study of DIF across groups and in-
cluded brief discussion of direct tests of hypotheses regarding
item parameters. Although widely utilized in intelligence test-
ing, IRT can be equally effectively used in personality testing.
However, IRT has only been sporadically applied in research
on personality instruments (Panter, Swygert, Dahlstrom, &
Tanaka, 1997); our review of the literature could locate no IRT
study with the CS. In the only published IRT study with the
MMPI–2, Childs, Dahlstrom, and Panter (2000) discovered
DIF on the MMPI–2 by gender in the measurement of depres-
sion using Scale 2; this same methodology used to test gender
differences by Childs et al. is identically applicable in tests of
cultural differences. IRT holds great promise for study of the
differential functioning for several CS variables because
many of the CS variables are not true scales as articulated by
the tenets of classical psychometric theory (e.g., Nunnally &
Bernstein, 1994). IRT provides an alternative to classical mea-
surement theory, providing an avenue to more rigorous study
of CS variables across cultural and ethnic groups. One draw-
back to the use of IRT has been the requirement for large sam-
ple sizes. However, Meijer (2003) discussed the use of
person-fit statistics that allow one to study the fit of an individ-
ual item pattern to an IRT model. If a significant proportion of
a small cultural or ethnic sample were identified as misfitting
the IRT model produced by the normative sample for a CS
variable, it would suggest a pattern of person misfit to the
model that is associated with group status and in this way pro-
vide a test of item equivalence cross-culturally (see van der
Flier, 1982). However, Allen and Walsh (2000) described four
important limitations to the use of IRT procedures in what is
commonly termed the investigation of DIF across ethnic and
cultural groups.10
10
A number of important limitations exists regarding IRT proce-
dures in tests of item equivalence. Two important assumptions must
be met for the use of IRT with personality instruments. First, the set
of items to be analyzed is assumed to be unidimensional as demon-
 CROSS-CULTURAL RORSCHACH METHODOLOGY
Cross-Cultural Construct Validation
Because of limitations to each of the previously described
methods, the most unequivocal method to establish absence
of bias for the inferences from a CS variable cross-culturally
is to establish evidence of cross-cultural construct equiva-
lence of the variable with the cultural group of interest
through compelling validity argument. Predictor bias stud-
ies, tests of structural equivalence, and IRT analyses all pro-
vide useful screens to identify promising variables to devote
energies toward the more painstaking task of cross-cultural
construct validation. It is quite reasonable to assume that
norms for some CS variables may differ cross-culturally, and
this normative difference may reflect only cultural difference
in quantity of an attribute within an equivalent construct. The
critical question is does the variable measure a
cross-culturally equivalent construct? Returning to the exam-
ple of EB, what is important here is evidence for the construct
equivalence of the underlying introversive–extratensive Ror-
schach dimension, not whether scores on EB differ
cross-culturally. Such research could include cross-cultural
replication of earlier validation studies with EB and its com-
ponents, M and Color responses (e.g., Blatt & Feirstein,
1977; Cocking, Dana, & Dana, 1969; Dana, 1968; Dana &
Cocking, 1968; Dana, Cocking, & Dana, 1970; Exner &
Sanglade, 1992; Weiner & Exner, 1991; and see Exner, 1993;
Molish, 1967; and Singer, 1960 for a review of earlier re-
search literature). Methodological recommendations from
the mainstream Rorschach research literature are essential in
guiding cross-cultural studies (e.g. Exner, 1995; Weiner,
strated empirically through factor analysis. CS variables intended to
measure a particular construct are often probably multidimensional,
and this is often the case with MMPI–2 scales. Whereas Drasgow
and Parsons (1983) suggested IRT is relatively robust to violations
of unidimensionality, others (e.g., Panter et al., 1997) have asserted
that IRT is best used with empirically identified factor scales to en-
sure properties of the IRT model are maintained in the analysis. Sec-
ond, germane to the MMPI–2, local independence must hold for the
item set, that is, item responses must depend on or be based on their
common association with θ, the underlying personality trait or at-
tribute, rather than other external variables related to the items them-
selves such as the potential effects of item order, content carryover
from other items, or use of common stem. Another limitation of IRT
involves the relatively large sample sizes needed for stable parame-
ter estimation, a major disadvantage for researchers who study small
overall population groups. Because of this, adaptation of person-fit
statistics to detect item pattern score misfit among a proportion of
cultural group members holds promise. Finally, IRT yields findings
regarding item equivalence and not conceptual equivalence of the
construct. This is an important distinction. Item equivalence on a test
indicates that individual items function similarly across cultures or
ethnic groups and that the observed score metrics are equivalent.
Item equivalence does not imply construct or conceptual equiva-
lence. IRT does not provide a test of whether it may be appropriate to
adopt the nomological network of the construct of interest to the new
cultural or ethnic group. Further, the previously mentioned applica-
tions of IRT cannot provide information regarding the source of
DIF.
201
1995, 2001a, 2001b). In particular, key issues to be attended
to include interrater reliability (Meyer, 1997a, 1997b; Meyer
et al., 2002; Viglione & Hilsenroth, 2001; Weiner, 1991) and
the use of appropriate statistical methods (Mcguire, Kindler,
Curtiss, & Viglione, 1995; Viglione, 1995).
Mixed Quantitative/Qualitative-Ethnographic
Research Designs With the Rorschach
In their pilot study of the role of clitoridectomy in structuring
the self-concept of West-African immigrant women in Paris,
France from the Soninké culture, Franchi and
Andronikof-Sanglade (1999) used a combined emic–etic ap-
proach that provides one promising model for innovative
cross-cultural validity research with the Rorschach. In
Franchi and Andronikof-Sanglade’s study, self-report mea-
sures of variables associated with their hypotheses were ad-
ministered along with the Rorschach. Data from the quantita-
tive CS measures was examined contextually through the
idiographic narrative data produced by the Rorschach proto-
col. In one example from Franchi and
Andronikof-Sanglade’s approach, alternative inferences re-
garding the positive SCZI scores produced by a majority of
the participants (SCZI was the precursor to the current PTI in
the CS) were developed out of the narrative data and accom-
panying quantitative measures. With no other indication of
psychopathology in their nonpatient Soninké sample,
Franchi and Andronikof-Sanglade inferred a shift had oc-
curred in the nomological net underlying the construct
tapped by SCZI within their sample and accordingly devel-
oped new inferences based in the narratives, the self-report
variables, and interestingly, other Rorschach variables tap-
ping cognitive functioning. Further research with
West-African women is necessary, and not all of the Franchi
and Andronikof-Sanglade’s speculations may receive sup-
port in future studies. Nonetheless, Franchi and
Andronikof-Sanglade’s study highlights the utility of the
multiple quantitative and qualitative data sources produced
by a Rorschach protocol in cross-cultural research.
Many authors (e.g., Brink, 1994; Kleinman, 1988) have
argued that the exclusive reliance on quantitative approaches
can result in research that misses or potentially misunder-
stands important depth and nuance in investigations of cul-
tural factors. Methodologies that integrate qualitative
approaches with nomothetic assessment research are sorely
needed because of their utility in discovery-based research
and initial validation work with psychological constructs.
Qualitative methods in more mature and advanced areas of
cultural research can provide important triangulation and
checks on quantitative findings as well as potentially enrich
and extend them. In the Franchi and Andronikof-Sanglade
(1999) study, qualitative-idiographic data provided a critical
check that prevented erroneous interpretations from the
quantitative data and raised important questions regarding
the construct equivalence of particular variables. Franchi and
202 ALLEN AND DANA
Andronikof-Sanglade’s study demonstrates both the value of
qualitative method design components in cross-cultural per-
sonality assessment research along with the utility of the
Rorschach in cultural research through its applicability to
mixed quantitative–qualitative designs.
CURRENT STATUS OF THE CROSS-
CULTURAL/MULTICULTURAL
RORSCHACH RESEARCH
In their review, Costantino, Flanagan, and Malgady (1995)
described numerous methodological deficiencies in current
cross-cultural and multicultural Rorschach research. At that
time, most research was at the stage of cross-ethnic and
cross-cultural comparisons of variable scores. Costantino et
al. noted the circularity of using the variables themselves as
criteria in the absence of demonstrated universality of the un-
derlying constructs. Recent research with the CS described
previously has begun to move beyond these problematic
comparative methodologies, and we present methodologies
here to advance this research further.
Multicultural and Cross-Cultural Clinical
Utility Research
McGrath (2001) differentiated between clinical utility and
validity research with the Rorschach. Many of McGrath’s
distinctions are useful and relevant to cross-cultural and
multicultural research with the Rorschach. Indeed, much of
the debate regarding the merits of the CS focuses on clini-
cal utility. McGrath discussed the issue of categorical pre-
diction and positive and negative predictive power in rela-
tion to the accuracy of an instrument’s prediction of clinical
categories such as diagnostic categories. Returning yet
again to PTI as example, in cross-cultural research, the re-
search question from the perspective of clinical utility re-
search translates to “once we have established PTI taps an
equivalent construct with a new cultural group, does PTI
display equivalent predictive power for the diagnosis of
schizophrenia within a new cultural group?” However, ex-
perience with SCZI among the Soninké highlights the im-
portance of first establishing conceptual equivalence of a
variable before proceeding to this next stage of analysis of
cross-cultural predictive precision.
Moreover, although McGrath (2001) described clinical
utility as a distinct consideration from conventional validity
theory and research, an alternative perspective is to view
clinical utility as an important and critical component of va-
lidity theory as originally proposed by its primary architects.
Cook and Campbell (1979) described the generalizability as-
pect of construct validity, which concerns itself directly with
its utility within the various settings a measure is used. This
generalizability aspect includes the settings and demands of
actual clinical use of an instrument.
Consequential Aspects of Cross-Cultural
and Multicultural Rorschach Validity Research
During three decades of work, Messick (1965, 1975, 1980,
1995) developed a comprehensive approach to construct vali-
dation in which he has described six aspects of construct valid-
ity. Our discussion of cross-cultural research methodologies
with the CS has thus far been primarily occupied with only five
aspects: content validity through discussion of strategies to in-
vestigate the cross-cultural CS stimulus materials and instruc-
tional set relevance; substantive validity through proposed
study of whether the CS task and variables actually engage
processes tapped by the construct cross-culturally; structural
validity through cross-cultural investigation of the invariance
of the internal structure of CS variables; generalizability,
which includes study both of the generalizability of CS inter-
pretive frames across cultural groups and their cross-cultural
clinical utility; and external validity, which encompasses pro-
cedures to establish cross-cultural and multicultural conver-
gent and discriminant validity of specific CS variables.
Messick (1995) highlighted a final aspect of validity, the
consequential aspect, which elevates a consideration of val-
ues and meaning as integral to the test validation process.
Use of the Rorschach, as with any test or measure, should be
appraised in light of the potential and actual social conse-
quences of test interpretation consistent with other social val-
ues. Important consequential validity considerations arise in
cross-cultural interpretations of test data. Throughout the
past century of psychological research, cultural differences
have been typically interpreted as deficiencies (Jones, 1988).
Because of this, consequential validity considerations point
to the need for development of an empirical basis for multi-
cultural and cross-cultural practice grounded in consensual
research standards. Standards currently exist for U.S. minori-
ties (CNPAAEMI, 2000) and provide a model for interna-
tional cross-cultural research.
CONCLUSIONS
Consequential validity elevates investigations of multicultural
and cross-cultural clinical utility as a key component of con-
temporary assessment instrument validation. The Rorschach
enjoys a long history of cross-cultural application, both as a
clinical instrument and measure of cultural difference, begin-
ning with Herman Rorschach’s early research interests. The
nonverbal nature of its stimulus materials uniquely situates it
as a true international assessment instrument of underutilized
potential. However, development of an empirical basis for
cross-cultural and multicultural Rorschach practice is needed
to realize this promise. We have reviewed here selected meth-
odological considerations for such efforts. Additionally,
cross-cultural and multicultural Rorschach research can offer
an important contribution in the contemporary debate regard-
ing the clinical utility of the CS. The cross-cultural and multi-
cultural potential of the Rorschach provides an important ra-
 CROSS-CULTURAL RORSCHACH METHODOLOGY
tionale for its continued use in the 21st century. Equally
important, continued work with the CS can also provide mod-
els for the development of new, performance-based, interna-
tional personality measures with nonverbal materials de-
signed for cross-cultural transportability.
ACKNOWLEDGMENTS
This research has been supported in part by a grant from the
National Institute of Alcohol Abuse and Alcoholism and the
National Center for Minority Health Disparities
1RO1AA11446–03 (principal investigator: G. Mohatt;
coinvestigators: K. Hazel, J. Allen, & C. Geist). We thank
Mark J. Hilsenroth, James A. Walsh, and four anonymous re-
viewers for their helpful comments on earlier versions of this
manuscript.
REFERENCES
Allen, J., & Walsh, J. A. (2000). A construct-based approach to equivalence:
Methodologies for cross-cultural/multicultural personality assessment re-
search. In R. H. Dana (Ed.), Handbook of cross-cultural and multicultural
personality assessment (pp. 63–85). Mahwah, NJ: Lawrence Erlbaum As-
sociates, Inc.
American Psychiatric Association. (1980). Diagnostic and statistical man-
ual of mental disorders (3rd ed.). Washington, DC: Author.
American Psychiatric Association. (1987). Diagnostic and statistical man-
ual of mental disorders (3rd ed., rev.). Washington, DC: Author.
American Psychiatric Association. (1994). Diagnostic and statistical man-
ual of mental disorders (4th ed.). Washington, DC: Author.
American Psychological Association. (2002). Ethical principles of psychol-
ogists and code of conduct. American Psychologist,57, 1060–1073.
Anastasi, A., & Urbina, S. (1997). Psychological testing (7th ed.). Upper
Saddle River, NJ: Prentice Hall.
Arbisi, P., Ben-Porath, Y., & McNulty, J. (2002). A comparison of MMPI–2
validity in African American and Caucasian psychiatric inpatients. Psy-
chological Assessment, 14, 3–15.
Arnold, B. R., Montgomery, G. T., Castanada, I., & Longoria, R. (1994). Ac-
culturation and performance of Hispanics on selected Halstead–Reitan
neuropsychological tests. Assessment, 1, 239–248.
Atkinson, D. R. (1983). Ethnic similarity in counseling psychology: A re-
view of research. The Counseling Psychologist, 11, 79–92.
Azibo, D. A. (1988). Understanding the proper and improper usage of the
comparative framework. Journal of Black Psychology, 15, 81–91.
Beutler, L. E., Crago, M., & Arizmendi, T. G. (1986). Therapist variables in
psychotherapy process and emotions. In A. E. Bergin & S. L. Garfield
(Eds.), Handbook of psychotherapy and behavior changes (pp. 257–310).
New York: Wiley.
Birman, D. (1994). Acculturation and human diversity in a multicultural soci-
ety. In E. J. Trickett, R. J. Watts, & D. Birman (Eds.), Human diversity: Per-
spectives on people in context (pp. 261–284). San Francisco: Jossey-Bass.
Blatt, S. J., & Feirstein, A. (1977). Cardiac response and personality organi-
zation. Journal of Consulting and Clinical Psychology, 45, 111–123.
Bornstein, R. F. (2001). Clinical utility of the Rorschach inkblot method:
Reframing the debate. Journal of Personality Assessment, 77, 39–47.
Brink, T. L. (1994). The need for qualitative research on mental health of el-
derly Hispanics. International Journal of Aging and Human Develop-
ment, 38, 279–291.
Butcher, J. N. (Ed.). (1996). International adaptations of the MMPI–2: Re-
search and clinical applications. Minneapolis: University of Minnesota
Press.
203
Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A., & Kaemmer, B.
(1989). MMPI–2: Minnesota Multiphasic Personality Inventory–2: Manual
for administration and scoring. Minneapolis: University of Minnesota Press.
Butcher, J. N., Nezami, E., & Exner, J. (1998). Psychological assessment of
people in diverse cultures. In S. S. Kazarian & D. R. Evans (Eds.), Cul-
tural clinical psychology: Theory, research, and practice (pp. 61–105).
New York: Oxford University Press.
Butcher, J. N., & Rouse, S. V. (1996). Personality: Individual differences and
clinical assessment. Annual Review of Psychology, 45, 87–111.
Camara, W., Nathan, J., & Puente, A. (2000). Psychological test usage: Im-
plications in professional use. Professional Psychology: Research and
Practice, 31, 141–154.
Childs, R. A., Dahlstrom, W. G., & Panter, A. T. (2000). Item response the-
ory in personality assessment: A demonstration using the MMPI–2 De-
pression Scale. Assessment, 7, 37.
Church, A. T., & Lonner, W. J. (1998). The cross-cultural perspective in the
study of personality: Rationale and current research. Journal of
Cross-Cultural Psychology, 29, 32–62.
Cocking, R., Dana, J., & Dana, R. H. (1969). Six constructs to define Ror-
schach M: A response. Journal of Projective Techniques & Personality
Assessment, 33, 322–323.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and
analysis issues for field settings. Chicago: Rand McNally.
Costantino, G., Flanagan, R., & Malgady, R. (1995). The history of the Ror-
schach: Overcoming bias in multicultural projective assessment.
Rorschachiana, 20, 148–171.
Council of National Psychological Associations for the Advancement of Eth-
nic Minority Interests. (2000). Guidelines for research in ethnic minority
communities. Washington, DC: American Psychological Association.
Cronbach, L. J. (1949). Statistical methods applied to Rorschach scores: A
review. Psychological Bulletin, 46, 393–429.
Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed.), Educa-
tional measurement (2nd ed., pp. 433–507). Washington, DC: American
Council on Education.
Cuéllar, I. (2000). Acculturation as a moderator of personality and psycho-
logical assessment. In R. H. Dana (Ed.), Handbook of cross-cultural and
multicultural personality assessment (pp. 113–129). Mahwah, NJ: Law-
rence Erlbaum Associates, Inc.
Dahlstrom, W. G. (1993). Tests: Small samples, large consequences. Ameri-
can Psychologist, 48, 393–399.
Dana, R. H. (1955). Rorschach scorer reliability. Journal of Clinical Psy-
chology, 11, 401–403.
Dana, R. H. (1962). The validation of projective tests. Journal of Projective
Techniques, 26, 182–186.
Dana, R. H. (1968). Six constructs to define Rorschach M. Journal of Pro-
jective Techniques and Personality Assessment, 32, 138–145.
Dana, R. H. (1993). Multicultural assessment perspectives for professional
psychology. Boston: Allyn & Bacon.
Dana, R. H. (2000a). Culture and methodology in personality assessment. In
I. Cuéllar & F. Paniagua (Eds.), Handbook of multicultural mental health:
Assessment and treatment of diverse populations (pp. 97–120). San
Diego, CA: Academic.
Dana, R. H. (Ed.). (2000b). Handbook of cross-cultural and multicultural
personality assessment. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Dana, R. H. (2001). Clinical diagnosis of multicultural populations in the
United States. In L. Suzuki, J. Ponterotto, & P. Meller (Eds.), The hand-
book of multicultural assessment (2nd ed., pp. 101–131). San Francisco:
Jossey-Bass.
Dana, R. H. (2003). Assessment training, practice, and research for the new
millennium: Challenges and opportunities for professional psychology.
Ethical Human Sciences and Services, 5, 127–140.
Dana, R. H., Aragon, M., & Kramer, T. (2002). Public sector mental health
services for multicultural populations: Bridging the gap from research to
clinical practice. In M. N. Smyth (Ed.), Health care in transition (Vol. 1,
pp. 15–29). Hauppauge, NY: Nova Science.
Dana, R. H., & Cocking, R. (1968). Cue parameters, cue probabilities, and
clinical judgment. Journal of Clinical Psychology, 24, 475–480.
204 ALLEN AND DANA
Dana, R. H., Cocking, R., & Dana, J. (1970). The effects of experience and
training on accuracy and configural analysis. Journal of Clinical Psychol-
ogy, 26, 28–32.
Dana, R. H., Hinman, S., & Bolton, B. (1977). Dimensions of examinee re-
sponses to the Rorschach: An empirical analysis. Psychological Reports,
40, 1147–1153.
Dawes, R. M. (1999). Two methods for studying the incremental validity of
a Rorschach variable. Psychological Assessment, 11, 297–302.
Draguns, J. G. (1990). Applications of cross-cultural psychology in the field
of mental health. In R. W. Brislin (Ed.), Applied cross-cultural psychology
(pp. 302–324). Newbury Park, CA: Sage.
Drasgow, F., & Parsons, C. K. (1983). Application of unidimensional item
response models to multidimensional data. Applied Psychological Mea-
surement, 7, 189–199.
Exner, J. E. (1993). The Rorschach: A Comprehensive System: Vol. I. Basic
foundations (3rd ed.). New York: Wiley.
Exner, J. E. (Ed.). (1995). Issues and methods in Rorschach research.
Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Exner J. E., & Sanglade, A. A. (1992). Rorschach changes following
brief and short term therapy. Journal of Personality Assessment, 59,
59–71.
Eysenck, H. J. (1959). Review of the Rorschach Inkblot Test. In O. K. Buros
(Ed.), The fifth mental measurements yearbook (pp. 276–278). Highland
Park, NJ: Gryphon.
Eysenck, H. J. (1991). Dimensions of personality: The biosocial approach to
personality. In J. Strelau & A. Angleitner (Eds.), Explorations in tempera-
ment: International perspectives on theory and measurement (pp.
87–103). New York: Plenum.
Flier, H. van der (1982). Deviant response patterns and comparability of test
scores. Journal of Cross-Cultural Psychology, 13, 267–298.
Franchi, V., & Andronikof-Sanglade, A. (1999). Methodological and
epistemological issues raised by the use of the Rorschach comprehensive
system in cross-cultural research. Rorschachiana, 23, 118–134.
Frueh, B. C., Gold, P. B., de Arellano, M. A., & Brady, K. L. (1997). A racial
comparison of combat veterans evaluated for PTSD. Journal of Personal-
ity Assessment, 68, 692–702.
Frueh, B. C., Smith, D. W., & Libet, J. M. (1996). Racial differences on psy-
chological measures in combat veterans seeking treatment for PTSD.
Journal of Personality Assessment, 66, 41–53.
Gacono, C. B., Loving, J. L., & Bodholdt, R. H. (2001). The Rorschach and
psychopathology: Toward a more accurate understanding of the Ror-
schach findings. Journal of Personality Assessment, 77, 16–38.
Gamst, G., Dana, R. H., Der-Karabetian, A., & Kramer, T. (2000). Ethnic
match and client ethnicity effects on global assessment and visitation.
Journal of Community Psychology, 28, 547–564.
Gamst, G., Dana, R. H., Der-Karabatian, A., & Kramer, T. (2001). Asian
American mental health clients: Effects of ethnic match and age on global
assessment and visitation. Journal of Mental Health Counseling, 23,
57–71.
Ganellen, R. J. (2001). Weighting evidence for the Rorschach’s validity: A
response to Wood et al. (1999). Journal of Personality Assessment, 77,
1–15.
Garb, H. N., Wood, J. W., Lilienfeld, S. O., & Nezworski, M. T. (2002). Ef-
fective use of projective techniques in clinical practice: Let the data help
with selection and interpretation. Professional Psychology, 33,
454–463.
Garb, H. N., Wood, J. W., Nezworski, M. T., Grove, W. M., & Stejskal, W. J.
(2001). Toward a resolution of the Rorschach controversy. Psychological
Assessment, 13, 433–448.
Geisinger, K. F. (1994). Cross-cultural normative assessment: Translation
and adaptation issues influencing the normative interpretation of assess-
ment instruments. Psychological Assessment, 6, 304–312.
Graham, J. R. (1990). MMPI–2: Assessing personality and
psychopathology. London: Oxford University Press.
Greene, R. L. (1987). Ethnicity and MMPI performance: A review. Journal
of Consulting & Clinical Psychology, 55, 497–512.
Hall, G. C. N., Bansal, A., & Lopez, I. R. (1999). Ethnicity and
psychopathology: A meta-analytic review of 31 years of comparative
MMPI/MMPI–2 research. Psychological Assessment, 11, 186–197.
Hibbard, S. (2003). A critique of Lilienfeld et al.’s (2000) “The scientific sta-
tus of projective techniques.” Journal of Personality Assessment, 80,
260–271.
Hilsenroth, M. J., & Handler, L. (1995). A survey of graduate students’ expe-
riences, interests, and attitudes about learning the Rorschach. Journal of
Personality Assessment, 64, 243–257.
Hoffmann, T., Dana, R. H., & Bolton, B. (1985). Measured acculturation and
MMPI–168 performance of Native American adults. Journal of
Cross-Cultural Psychology, 16, 243–256.
Howard, A., Poin, G. M., Sechrest, L. B., Cordray, D. S., Kaplan, L., Hall, J.,
et al. (1987). Membership opinions about reorganizing APA. American
Psychologist, 42, 763–779.
Hunsley, J., & Bailey, J. M. (1999). The clinical utility of the Rorschach: Un-
fulfilled promises and an uncertain future. Psychological Assessment, 11,
266–277.
Jensen, A. R. (1965). Review of the Rorschach Inkblot Test. In O. K. Buros
(Ed.), The sixth mental measurements yearbook (pp. 501–509). Highland
Park, NJ: Gryphon.
Jensen, A. R. (1980). Bias in mental testing. New York: Free Press.
Jones, J. M. (1988). Racism in black and white: A bicultural model of reac-
tion and evolution. In P. A. Katz & D. A. Taylor (Eds.), Eliminating rac-
ism: Profiles in controversy (pp. 117–135). New York: Plenum.
Jöreskog, K. G., & Sörbom, D. (1989). LISREL 7: User’s reference guide.
Mooresville, IN: Scientific Software.
Kazarian, S. S., & Evans, D. B. (1998). Cultural clinical psychology. In S. S.
Kazarian & D. R. Evans (Eds.), Cultural clinical psychology: Theory, re-
search, and practice (pp. 1–38). New York: Oxford University Press.
Kitayama, S., & Markus, H. R. (1994). Emotions and culture: Empirical
studies of mutual influence. Washington, DC: American Psychological
Association.
Kleinman, A. (1988). Rethinking psychiatry: From cultural category to per-
sonal experience. New York: Free Press.
Lautenschlager, G. J., & Mendoza, J. L. (1986). A step-down hierarchical
multiple regression analysis for examining hypotheses about test bias in
prediction. Applied Psychological Measurement, 10, 133–139.
Lilienfeld, S. O., Wood, J. M., & Garb, H. N. (2000). The scientific status of
projective techniques. Psychological Science in the Public Interest, 1,
27–66.
Lindsey, M. L. (1998). Culturally competent assessment of African Ameri-
can clients. Journal of Personality Assessment, 70, 43–53.
Lindzey, G. (1954). Projective techniques and cross-cultural research. New
York: Appleton-Century-Crofts.
Linn, R. L., & Werts, C. E. (1971). Considerations for studies of test bias.
Journal of Educational Measurement, 8, 1–4.
Lonner, W. J. (1996). Psychometrics and culture. In Advanced methodological
issues in culturally competent evaluation for substance abuse prevention.
Washington, DC: SAMHSA, Center for Substance Abuse Prevention.
Lopez, S. R. (1989). Patient variable biases in clinical judgment: Conceptual
overview and methodological considerations. Psychological Bulletin,
106, 184–203.
Maki, M. T. (1999). The effects on clinician identification when clinician
and client share a common ethnic minority background. Journal of Multi-
cultural Social Work, 7, 57–72
Malgady, R. G. (1996). The question of cultural bias in assessment and diag-
nosis of ethnic minority clients: Let’s reject the null hypothesis. Profes-
sional Psychology: Research and Practice, 27, 73–77.
Manson, S. M. (1995). Culture and major depression: Current challenges in
the diagnosis of mood disorders. Psychiatric Clinics of North America,
18, 487–501.
Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for
cognition, emotion, and motivation. Psychological Review, 98, 224–253.
McFarlane, J. W. (1942). Problems in validation inherent in projective meth-
ods. American Journal of Orthopsychiatry, 12, 405–410.
 CROSS-CULTURAL RORSCHACH METHODOLOGY
McGrath, R. E. (2001). Toward more clinically relevant assessment re-
search. Journal of Personality Assessment, 77, 122–127.
Mcguire, H., Kindler, B. N., Curtiss, G., & Viglione, D. J. (1995). Some spe-
cial issues in data analysis. In J. Exner (Ed.), Issues and methods in Ror-
schach research (pp. 227–250). Mahwah, NJ: Lawrence Erlbaum Associ-
ates, Inc.
McNulty, J. L., Graham, J. R., Ben-Porath, Y., & Stein, L. A. R. (1997).
Comparative validity of MMPI–2 scores of African American and Cauca-
sian mental health center clients. Psychological Assessment, 9, 464–470.
Meijer, R. R. (2003). Diagnosing item score patterns on a test using item re-
sponse theory-based person-fit statistics. Psychological Methods, 8,
72–87.
Mensh, I. N. (1950). Statistical techniques in present day psychodiagnostics.
Psychological Bulletin, 47, 474–492.
Mesquita, B., & Frijda, N. H. (1992). Cultural variations in emotions: A re-
view. Psychological Bulletin, 112, 179–204.
Messick, S. (1965). Personality measurement and the ethics of assessment.
American Psychologist, 20, 136–142.
Messick, S. (1975). The standard problem: Meaning and values in measure-
ment and education. American Psychologist, 30, 955–966.
Messick, S. (1980). Test validity and the ethics of assessment. American
Psychologist, 35, 1012–1027.
Messick, S. (1995). Validity of psychological assessment: Validation of in-
ferences from persons’ responses and performances as scientific inquiry
into score meaning. American Psychologist, 50, 741–749.
Meyer, G. J. (1992). Response frequency problems in the Rorschach: Clini-
cal and research implications with suggestions for the future. Journal of
Personality Assessment, 58, 231–244.
Meyer, G. J. (1997a). Assessing reliability: Critical corrections for a critical
examination of the Rorschach Comprehensive System. Psychological As-
sessment, 9, 480–489.
Meyer, G. J. (1997b). Thinking clearly about reliability: More critical cor-
rections for a critical examination of the Rorschach Comprehensive Sys-
tem. Psychological Assessment, 9, 495–498.
Meyer, G. J. (2001). Evidence to correct misperceptions about Rorschach
norms. Clinical Psychology: Science and Practice, 8, 389–396.
Meyer, G. J. (2002). Exploring possible ethnic differences and bias in the
Rorschach Comprehensive System. Journal of Personality Assessment,
78, 104–129.
Meyer, G. J., & Archer, R. P. (2001). The hard science of Rorschach re-
search: What do we know and where do we go? Psychological Assess-
ment, 13, 486–502.
Meyer, J. G., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R.
R., et al. (2001). Psychological testing and psychological assessment: A
recent review of evidence and issues. American Psychologist, 56,
128–165.
Meyer, J. G., Hilsenroth, M. J., Baxter, G., Exner, J. E., Jr., Fowler, C. J.,
Piers, C. C., et al. (2002). An examination of interrater reliability for scor-
ing the Rorschach Comprehensive System in eight data sets. Journal of
Personality Assessment, 78, 219–274.
Miller, J. B., Rosenthal, R., Bornstein, R. F., Berry, D. T. R., &
Brunell-Neuleib, S. (1999). A comparative meta-analysis of Rorschach
and MMPI validity. Psychological Assessment, 11, 278–296.
Molish, H. B. (1967). Critique and problems of the Rorschach. A survey. In
S. J. Beck & H. B. Molish (Eds.), Rorschach’s test. II: A variety of person-
ality pictures (2nd ed.). New York: Grune & Stratton.
Montgomery, G. T., & Orozco, S. (1985). Mexican Americans’ performance
on the MMPI as a function of level of acculturation. Journal of Personal-
ity Assessment, 54, 328–342.
Muniz, J., Prieto, G., Almeida, L., & Bartram, D. (1999). Test use in Spain,
Portugal, and Latin American countries. European Journal of Psychologi-
cal Assessment, 15, 151–157.
Munroe, R. L. (1945). Objective methods and the Rorschach blots. Ror-
schach Research Exchange, 9, 59–73.
Nichols, D. S., Padilla, J., & Gomez-Maqueo, E. L. (2000). Issues in the
cross-cultural adaptation and use of the MMPI–2. In R. H. Dana (Ed.),
205
Handbook of cross-cultural and multicultural personality assessment (pp.
247–266). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.).
New York: McGraw-Hill.
Okazaki S., & Sue, S. (1995). Methodological issues in assessment research
with ethnic minorities. Psychological Assessment, 7, 367–375.
Oyserman, D., Coon, H. M., & Kemmelmeier, M. (2002). Rethinking indi-
vidualism and collectivism: Evaluation of theoretical assumptions and
meta-analyses. Psychological Bulletin, 128, 3–72.
Panter, A. T., Swygert, K. A., Dahlstrom, W. G., & Tanaka, J. S. (1997). Fac-
tor analytic approaches to personality item-level data. In J. A. Schinka &
R. L. Greene (Eds.), Emerging issues and methods in personality assess-
ment (pp. 285–307). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Piotrowski, C. (1999), Assessment practices in the era of managed care:
Current status and future directions. Journal of Clinical Psychology, 55,
787–796.
Pires, A. A. (2000). National norms for the Rorschach normative study in
Portugal. In R. H. Dana (Ed.), Handbook of cross-cultural and multicul-
tural personality assessment (pp. 367–392). Mahwah, NJ: Lawrence
Erlbaum Associates, Inc.
Pollack, D., & Shore, J. H. (1980). Validity of the MMPI with Native Ameri-
cans. American Journal of Psychiatry, 137, 946–950.
Pope-Davis, D. B., Liu, W. M., Toporek, R. L., & Brittan-Powell, C. S.
(2001). What’s missing from multicultural competency research: Review,
introspection, and recommendations. Cultural Diversity and Ethnic Mi-
nority Psychology, 7, 121–138.
Pritchard, D. A., & Rosenblatt, A. (1980). Racial bias in the MMPI: A meth-
odological review. Journal of Consulting and Clinical Psychology, 48,
263–267.
Presley, G., Smith, C., Hilsenroth, M., & Exner, J. (2001). Rorschach valid-
ity with African Americans. Journal of Personality Assessment, 77,
491–507.
Radloff, L. S. (1977). A CES–D scale: A self-report scale for research in the
general population. Applied Psychological Measurement, 1, 385–401.
Ritsher, J., Slivko-Kolchik, E., & Oleichik, I. (2001). Assessing depression
in Russian psychiatric patients: Validity of MMPI and Rorschach. Assess-
ment, 8, 373–389.
Ritzler, B., & Alter, B. (1986). Rorschach teaching in APA-approved clinical
graduate programs: Ten years later. Journal of Personality Assessment,
50, 44–49.
Rorschach, H. (1942). Psychodiagnostics: A diagnostic test based on per-
ception (3rd ed., B. Kronenberg & P. Lemkau, Trans.) Berne, Switzerland:
Hans Huber. (Original work published in 1921)
Rosenberg, M. (1979). Conceiving the self. New York: Basic.
Roysircar-Sodowsky, G., & Kuo, P. Y. (2001). Determining cultural validity
of personality assessment. In D. B. Pope-Davis & H. L. K. Coleman
(Eds.), The intersection of race, class, and gender in multicultural coun-
seling (pp. 213–239). Thousand Oaks, CA: Sage.
Sangro, F. L. (1997). Location tables, form quality, and popular responses in
a Spanish sample of 470 sujects. Rorschachiana, 22, 38–66.
Sanz, I. A. E., Ritzler, B. A., Ivanouw, J., & Dana, R. H. (2001). El impacto
de las neuvas technologies en el futuro de las technicas de evaluacion
psicologica clinica [The impact of new technologies on the future of clini-
cal psychological assessment]. Revista Argentina de Clinica Psicologica,
X, 203–225.
Singer, J. L. (1960). The experience type: Some behavioral correlates and
theoretical implications. In M. Rickers-Ovsiankina (Ed.), Rorschach psy-
chology (pp. 325–372). Huntington, NY: Kreiger.
Stout, C. E. (1997). Psychological assessment in managed care. New York:
Wiley.
Stricker, G., & Gold, J. R. (1999). The Rorschach: Toward a nomothetically
based, idiographically applicable configurational model. Psychological
Assessment, 11, 240–250.
Sue, S., Keefe, K., Enomoto, K., Durvasula, R. S., & Chao, R. (1996). Asian
American and White college students’ performance on the MMPI–2. In J.
N. Butcher (Ed.), International adaptations of the MMPI–2: Research
206 ALLEN AND DANA
and clinical applications (pp. 206–218). Minneapolis: University of Min-
nesota Press.
Sue, S., Kurasaki, K. S., & Srinivarsan, S. (1999). Ethnicity, gender, and
cross-cultural issues in clinical research. In P. C. Kendall, J. N. Butcher, &
G. N. Holinbeck (Eds.), Handbook of research methods in clinical psy-
chology (2nd ed., pp. 54–71). New York: Wiley.
Sue, S., & Zane, N. (1987). The role of culture and cultural techniques in
psychotherapy: A critique and reformulation. American Psychologist, 42,
37–45.
Thissen, D., Steinberg, L., & Wainer, H. (1988). Use of item response theory
in the study of group differences in trace lines. In H. Wainer & H. I. Braun
(Eds.), Test validity (pp. 213–238). Hillsdale, NJ: Lawrence Erlbaum As-
sociates, Inc.
Timbrook, R. E., & Graham, J. R. (1994). Ethnic differences on the MMPI?
Psychological Assessment, 6, 212–217.
Triandis, H. C., & Suh, E. M. (2002). Cultural influences on personality. An-
nual Review of Psychology, 53, 133–160.
Van de Vijver, F., & Hambleton, R. K. (1996). Translating tests: Some prac-
tical guidelines. European Psychologist, 1, 89–99.
Van de Vijver, F. J. R., & Poortinga, Y. H. (1997). Towards an integrated
analysis of bias in cross-cultural assessment. European Journal of Psy-
chological Assessment, 13, 29–37.
Viglione, D. J. (1995). Basic considerations regarding data analysis. In J.
Exner (Ed.), Issues and methods in Rorschach research (pp. 227–250).
Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Viglione, D. J. (1997). Problems in Rorschach research and what to do about
them. Journal of Personality Assessment, 68, 590–599.
Viglione, D. J. (1999). A review of recent research addressing the utility of
the Rorschach. Psychological Assessment, 11, 251–265.
Viglione, D. J., & Hilsenroth, M. J. (2001). The Rorschach: Facts, fictions,
and future. Psychological Assessment, 13, 452–471.
Vinet, E. V. (2000). The Rorschach Comprehensive System in Iberoamerica.
In R. H. Dana (Ed.), Handbook of cross-cultural and multicultural per-
sonality assessment (pp. 345–365). Mahwah, NJ: Lawrence Erlbaum As-
sociates, Inc.
Voigt, W., & Dana, R. H. (1964). Inter- and intrascorer reliability. Journal of
Projective Techniques and Personality Assessment, 28, 92–95.
Wainer, H., & Braun, H. I. (Eds.). (1988). Test validity. Hillsdale, NJ: Law-
rence Erlbaum Associates, Inc.
Watkins, C. E., Jr., Campbell, V. I., Nieberding, R., & Hallmark, R. (1995).
Contemporary practice of psychological assessment by clinical psycholo-
gists. Professional Psychology: Research and Practice, 26, 54–60.
Wechsler, D. (1997). Manual for the Wechsler Adult Intelligence Scale III.
New York: The Psychological Corporation.
Weiner, I. B. (1991). Editor’s note: Interscorer agreement in Rorschach re-
search. Journal of Personality Assessment, 56, 1.
Weiner, I. B. (1994). The Rorschach Inkblot Method (RIM) is not a test: Im-
plications for theory and practice. Journal of Personality Assessment, 62,
498–504.
Weiner, I. B. (1995). Methodological considerations in Rorschach research.
Psychological Assessment, 7, 330–337.
Weiner, I. B. (1999). Contemporary perspectives on Rorschach assessment.
European Journal of Personality Assessment, 15, 78–86.
Weiner, I. B. (2000). Making Rorschach interpretation as good as it can be.
Journal of Personality Assessment, 74, 164–174.
Weiner, I. B. (2001a). Advancing the science of psychological assessment:
The inkblot method as exemplar. Psychological Assessment, 13,
423–432.
Weiner, I. B. (2001b). Considerations in collecting Rorschach reference
data. Journal of Personality Assessment, 77, 122–127.
Weiner I. B., & Exner, J. E. (1991). Rorschach changes in long-term and
sort-term psychotherapy. Journal of Personality Assessment, 56,
453–465.
View publication stats
Weiner, I. B., Spielberger, C. D., & Abeles, N. (2002). Scientific psychology
and the Rorschach Inkblot Method. The Clinical Psychologist, 55(4), 7–12.
West, S. G., Finch, J. F., & Curran, P. J. (1995). Structural equation models
with nonnormal variables: Problems and remedies. In R. H. Hoyle (Ed.),
Structural equation modeling: Concepts, issues, and applications (pp.
56–75). Thousand Oaks, CA: Sage.
Whatley, J. P., Allen, J., & Dana, R. H. (2003). Racial identity and the MMPI
in African American male college student. Cultural Diversity and Ethnic
Minority Psychology, 9, 344–352.
Whitworth, R. H., & McBlaine, D. D. (1993). Comparison of the MMPI and
MMPI–2 administered to Anglo-American and Hispanic American uni-
versity students. Journal of Personality Assessment, 61, 19–27.
Widiger, T. A. (2001). The best and worst of us? Clinical Psychology: Sci-
ence and Practice, 8, 374–377.
Wood, J. M., Garb, H. S., Lilienfeld, S. O., & Nezworski, M. T. (2002). Clin-
ical assessment. Annual Review of Psychology, 53, 519–543.
Wood, J. M., & Lilienfeld, S. O. (1999). The Rorschach Inkblot Test: A case
of overstatement? Assessment, 6, 341–351.
Wood, J. M., Lilienfeld, S. O., Garb, H. N., & Nezworski, M. T. (2000a).
Limitations of the Rorschach as diagnostic tool: A reply to Garfield
(2000), Lerner (2000), and Wiener (2000). Journal of Clinical Psychol-
ogy, 56, 441–448.
Wood, J. M., Lilienfeld, S. O., Garb, H. N., & Nezworski, M. T. (2000b). The
Rorschach test in clinical diagnosis: A critical review with a backward look
at Garfield (1947). Journal of Clinical Psychology, 56, 395–430.
Wood, J. M., Lilienfeld, S. O., Nezworski, M. T., & Garb, H. N. (2001).
Coming to grips with the negative evidence for the Comprehensive sys-
tem for the Rorschach: A comment on Gacono, Loving, and Bodholdt;
Ganellen; and Bornstein. Journal of Personality Assessment, 77, 48–70.
Wood, J. M., Nezworski, M. T., Garb, H. N., & Lilienfeld, S. O. (2001a). The
misperception of psychopathology: Problems with the norms of the Com-
prehensive System for the Rorschach. Clinical Psychology: Science and
Practice, 8, 350–373.
Wood, J. M., Nezworski, M. T., Garb, H. N., & Lilienfeld, S. O. (2001b).
Problems with the norms of the Comprehensive System for the Ror-
schach: Methodological and coneptual consideration. Clinical Psychol-
ogy: Science and Practice, 8, 397–402.
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996). The Comprehen-
sive System for the Rorschach: A critical examination. Psychological Sci-
ence, 7, 3–10.
Wood, J. M., Nezworski, M. T., Stejskal, W. J., & Garven, S. (2001). Ad-
vancing scientific discourse in the controversy surrounding the Compre-
hensive Systems for the Rorschach: A rejoinder to Meyer (2000). Journal
of Personality Assessment, 76, 369–378.
Wood, J. M., Nezworski, M. T., Stejskal, W. J., Garven, S., & West, S. G.
(1999). Methodological issues in evaluating Rorschach validity: A com-
ment on Burns and Viglione (1996), Weiner (1996), and Ganellen (1996).
Assessment, 6, 115–129.
Wuts, G. (2002). A preliminary validation study of the psychometric proper-
ties of selected scales of the MMPI–2 with English speaking
Singaporeans. Unpublished doctoral dissertation, California School of
Professional Psychology, Alliant University, San Diego.
James Allen
Department of Psychology
University of Alaska Fairbanks
Fairbanks, AK 99775–6480
E-mail: Jim.Allen@uaf.edu
Received April 23, 2003
Revised September 26, 2003