This article was downloaded by: [University of Dayton]

On: 28 December 2014, At: 02:42

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954
Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,
UK

Journal of Personality
Assessment
Publication details, including instructions for
authors and subscription information:
http://www.tandfonline.com/loi/hjpa20

Weighing Evidence for the

Rorschach's Validity: A
Response to Wood et al. (1999)
Ronald J. Ganellen
Published online: 10 Jun 2010.

To cite this article: Ronald J. Ganellen (2001) Weighing Evidence for the Rorschach's
Validity: A Response to Wood et al. (1999), Journal of Personality Assessment, 77:1,
1-15, DOI: 10.1207/S15327752JPA7701_01

To link to this article: http://dx.doi.org/10.1207/S15327752JPA7701_01

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the
information (the “Content”) contained in the publications on our platform.
However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness,
or suitability for any purpose of the Content. Any opinions and views
expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the
Content should not be relied upon and should be independently verified with
primary sources of information. Taylor and Francis shall not be liable for any
losses, actions, claims, proceedings, demands, costs, expenses, damages,
and other liabilities whatsoever or howsoever caused arising directly or
indirectly in connection with, in relation to or arising out of the use of the
Content.

This article may be used for research, teaching, and private study purposes.
Any substantial or systematic reproduction, redistribution, reselling, loan,
sub-licensing, systematic supply, or distribution in any form to anyone is
 expressly forbidden. Terms & Conditions of access and use can be found at
http://www.tandfonline.com/page/terms-and-conditions
Downloaded by [University of Dayton] at 02:42 28 December 2014
 JOURNAL OF PERSONALITY ASSESSMENT, 77(1), 1–15
Copyright © 2001, Lawrence Erlbaum Associates, Inc.

SPECIAL SERIES:
More Data on the Current Rorschach Controversy

Weighing Evidence for the

Downloaded by [University of Dayton] at 02:42 28 December 2014

Rorschach’s Validity:
A Response to Wood et al. (1999)*
Ronald J. Ganellen
Department of Psychiatry and Behavioral Sciences
Northwestern University Medical School

Wood, Nezworski, Stejskal, Garven, and West (1999) challenged Ganellen’s (1996)
characterization of the revised Rorschach Depression Index (DEPI; Exner, 1991) as a
promising psychometric marker of depression that deserves serious attention by re-
searchers and clinicians. To the contrary, however, a careful examination of existing
studies indicates that no compelling empirical evidence exists indicating that
Ganellen’s conclusions should be modified at the present time, although no firm con-
clusions about the DEPI can be reached until further evidence accumulates. Further-
more, although Wood et al. (1999) suggested that evidence supporting the reliability
and validity of the Rorschach in general is weak, ample evidence exists demonstrating
that the Rorschach can be scored reliably (Meyer, 1997), that Rorschach variables in
general have respectable levels of criterion-related validity (Bornstein, 1996; Hiller,
Rosenthal, Bornstein, Berry, & Brunnel-Neuleib, 1999), and that the Minnesota
Multiphasic Personality Inventory (MMPI; Hathaway & McKinley, 1943) and Ror-
schach have comparable levels of criterion-related validity, with the MMPI outper-
forming the Rorschach in certain respects and the Rorschach outperforming the
MMPI in others (Bornstein, 1999; Hiller et al, 1999).

In a series of articles, Wood and his colleagues have raised a number of criticisms
about the psychometric properties of the Rorschach Comprehensive System (CS;
Exner, 1991, 1993). For instance, Wood, Nezworski, and Stejskal (1996) ques-

*This and the next three articles are included in the special series on the Current
Rorschach Controversy.
 2 GANELLEN

tioned whether the coding system used to score Rorschach responses can be applied
in a reliable manner; they also expressed doubt about the validity of the scores and
indexes the CS uses for clinical interpretation. Wood et al. (1996) warned that be-
cause the reliability and validity of the CS had, in their view, not been adequately
established, it might be inappropriate if not unethical to use the CS in applied clini-
cal settings.
In response to Wood et al.’s (1996) concerns about whether the Rorschach can
be scored reliably, Meyer (1997) analyzed data from 16 published studies that in-
cluded CS interrater reliability statistics. The results of his meta-analysis showed
that CS variables had excellent interrater reliability. Meyer’s (1997) findings con-
Downloaded by [University of Dayton] at 02:42 28 December 2014

tradicted Wood et al.’s (1996) claims that the interrater reliability of the CS had ei-
ther not been adequately demonstrated or did not meet accepted criteria for test
standards. Despite this data, Wood, Nezworski, & Stejskal (1997) argued that
Meyer’s (1997) findings were not conclusive and should not be accepted. For in-
stance, Wood et al. (1997) claimed that although excellent interrater reliability
might be obtained in research contexts, similar results could not be assumed to oc-
cur in clinical practice until the CS was shown to have adequate field reliability. It
should be noted, however, that this standard has never been applied to any other
commonly used psychological assessment instrument that involves multiple-scor-
ing decisions for data that is often complex and ambiguous, such as responses to
the Wechsler Adult Intelligence Scale–III (Wechsler, 1997a) or the Wechsler
Memory Scale–III (Wechsler, 1997b).
More recently, Wood et al. asserted that the Rorschach’s validity has not been
adequately established and cited several different sources to support this claim.
For instance, Wood et al. (1999) referred to Archer and Krishnamurthy’s (1993a,
1993b) review articles examining the relations among a wide variety of Minnesota
Multiphasic Personality Inventory (MMPI; Hathaway & McKinley, 1943) and
Rorschach scores in adult and adolescent samples as evidence that “the Rorschach
has less validity than the Minnesota Multiphasic Personality Inventory” (p. 116).
Note that Archer and Krishnamurthy’s (1993a, 1993b) reviews did not directly
compare the validity of the MMPI and Rorschach and were not intended to do so.
They reviewed studies looking at relations among MMPI and Rorschach variables
and concluded that the existing evidence showed that MMPI and Rorschach vari-
ables were unrelated or, at best, were only weakly correlated with one another. Ex-
amining whether the MMPI and Rorschach are correlated with one another does not
necessarily reflect on the validity of either measure, as it is logically possible for two
instruments measuring conceptually similar constructs to have adequate validity
and yet be unrelated to one another. For instance, two measures of driving compe-
tence may each be valid but uncorrelated with one another if one involves a multiple-
choice test concerning knowledge of the rules of the road, whereas the other involves
a test of driving skill in actual traffic conditions; an individual may “know” the right
thing to do, yet have difficulty putting that knowledge into action. It is misleading to
 WEIGHING EVIDENCE FOR THE RORSCHACH’S VALIDITY 3

characterize the results of Archer and Krishnamurthy’s reviews as providing evi-

dence that the MMPI is more valid than the Rorschach because the studies included
in these reviews were basically correlational in nature and did not directly compare
the strength of each measures’s association with an external criterion.
Wood et al. referred to several other sources to support their claim that the Ror-
schach “has less validity than the Minnesota Multiphasic Personality Inventory”
(p. 116), including the results of Garb, Florio, and Grove’s (1998) recent
reanalysis of studies contained in a previously published meta-analysis (Parker,
Hanson, & Hunsley, 1988). Although Parker et al. concluded that the MMPI and
Rorschach had comparable levels of criterion-related validity based on their meta-
Downloaded by [University of Dayton] at 02:42 28 December 2014

analysis, Garb et al. claimed that Parker et al.’s conclusions may have been errone-
ous. Garb et al. contended that their meta-analysis showed the MMPI had higher
levels of criterion-related validity than the Rorschach.
This assertion, however, has been challenged by the results of a recent, more
comprehensive meta-analysis. Hiller et al. (1999) compared criterion-related va-
lidity evidence for the MMPI and the Rorschach using meta-analytic procedures
and concluded that global validity coefficients for the two measures were compa-
rable, .30 for the MMPI and .29 for the Rorschach. Hiller et al. noted that confi-
dence in both the MMPI and Rorschach should be buttressed by their findings,
which they characterized as being high for measures of personality functioning in
general. They added that, viewed in a broader context, the “validity for these in-
struments is about as good as can be expected for personality tests” (p. 291).
Although the magnitude of the validity coefficients found for the MMPI and
Rorschach were not significantly different, Hiller et al.’s (1999) analyses sug-
gested that the two measures may be differentially related to different types of
criterion variables. For instance, Hiller et al. reported that the MMPI had larger
validity coefficients than the Rorschach for studies using psychiatric diagnoses
and self-report measures as criterion variables. Hiller et al. noted that the stron-
ger association between the MMPI and other self-report measures was not unex-
pected given the shared method variance between the MMPI and other self-
report measures (Campbell & Fiske, 1959). The stronger correspondence be-
tween the MMPI and psychiatric diagnosis was thought to occur either because
of the empirical criterion-keying procedure used to develop the MMPI, or be-
cause psychiatric diagnoses are based to a large extent on self-report data and,
similar to self-report measures, may be related to the MMPI in part because of
substantial shared method variance (Meyer, 1996). In contrast, Hiller et al.
pointed out that the Rorschach had larger validity coefficients than the MMPI
for studies using objective criterion variables, such as the prediction of unambig-
uous outcomes or behavior. Hiller et al.’s conclusions contradict Wood et al.’s
assertions by showing first, that the Rorschach’s criterion-related validity is
quite respectable and, second, that the Rorschach’s validity is comparable to that
of the MMPI.
 4 GANELLEN

GLOBAL VERSUS SPECIFIC VALIDITY

Hiller et al. (1999) urged future research to focus not on comparisons of the global
validity of the MMPI and Rorschach, but to focus more specifically on the validity
of specific scales, scores, and indexes, a position endorsed by Wood et al. (1996,
1999) and others (Hunsley & Bailey, 1999; Weiner, 1996). One example of this ap-
proach is provided by Bornstein (1996) who examined the construct validity of the
Rorschach Oral Dependency scale (ROD; Masling, Rabie, & Blondheim, 1967), a
Rorschach variable developed independently of the CS. Bornstein found that labo-
ratory, clinical, and field studies support the convergent and discriminant validity
Downloaded by [University of Dayton] at 02:42 28 December 2014

of the ROD as a measure of interpersonal dependency. Bornstein concluded that not

only do these studies support the construct validity of the ROD in particular, but
these studies more generally provide evidence that a carefully constructed, empiri-
cally based Rorschach variable can be shown to be reliably associated with external
criteria such as an individual’s interpersonal behavior.
More recently, Bornstein (1999) conducted a meta-analysis examining evidence
for the construct validity of a number of self-report and projective measures of inter-
personal dependency, a dimension of considerable interest because of the associa-
tion between dependency and increased risk for a variety of psychological
disorders—including depression, anxiety, and tobacco addiction (Bornstein,
1995a)—as well as its association with a number of health-related behaviors such as
an elevated risk for becoming ill and compliance with treatment (Bornstein, 1995b).
Included among these measures was the ROD and an MMPI scale developed to mea-
sure interpersonal dependence, the Dependency scale (Navran, 1954). The results of
this meta-analysis showed that the ROD had higher levels of criterion-related valid-
ity than the MMPI Dependency scale, .37 and .20, respectively. In this instance, not
only was strong evidence found for the validity of a specific Rorschach variable, the
ROD, but the meta-analysis demonstrated that the ROD had higher levels of crite-
rion-related validity than a conceptually similar MMPI scale.

IS THE DEPI A VALID MARKER FOR DEPRESSION?

Wood et al. offered an appraisal of the research literature concerning specific Ror-
schach variables and expressed agreement with the tentative conclusion reached by
Ganellen (1996) that the existing evidence shows that some Rorschach variables are
valid measures of thought disorder, psychosis, and schizophrenia, such as the re-
vised Schizophrenia Index (SCZI; Exner, 1991). They also stated that research has
shown an association between other Rorschach variables and intellectual function-
ing. However, Wood et al. quarreled with Ganellen’s (1996) conclusion that the
available evidence indicates the revised Rorschach Depression Index (DEPI; Exner,
1991) functions similarly to MMPI and Millon Clinical Multiaxial Inventory–II
 WEIGHING EVIDENCE FOR THE RORSCHACH’S VALIDITY 5

(MCMI–II; Millon, 1987) depression scales and his statement that the MMPI,
MCMI–II, and DEPI have “comparable ability to correctly identify patients diag-
nosed with depression as being depressed” (Ganellen, 1996, p. 235). Wood et al.
stressed that their disagreement is based on the view that Ganellen’s conclusions re-
lied on problematic research or methodologically unsound reasoning to bolster
claims of Rorschach validity. These objections were made even though Ganellen
qualified these statements, stating that the findings of his review “should be viewed
cautiously, however, until additional data accumulates” (p. 236).
The reasons for the different conclusions reached by Wood et al. and Ganellen
(1996) deserves close examination. As the discussion following shows, these differ-
Downloaded by [University of Dayton] at 02:42 28 December 2014

ences reflect in large part Wood et al.’s decision to treat the original and revised
forms of the DEPI (Exner, 1991) as being the same scale, a decision that obfuscates
rather than clarifies the issue of the validity of the revised DEPI—the version that
has been used in clinical practice since it was introduced by Exner in 1991.

Original Versus Revised DEPI

Wood et al. offered a number of criticisms of Ganellen’s (1996) conclusions con-

cerning the DEPI. For instance, Wood et al. commented that “independent peer-re-
viewed studies of the DEPI, published both before and after Ganellen’s article,
have nearly all found that DEPI scores are unrelated to diagnoses of depression in
either adolescents or adults” (p. 124). They characterize these studies as consis-
tently showing that MMPI scores are related to diagnoses of depression, whereas
DEPI scores are not.
Several of the studies referenced by Wood et al. in support of this claim were
excluded by Ganellen (1996) because the studies used the original version of the
DEPI rather than the more recently revised version of the DEPI (Exner, 1991).
When discussing the criteria used to include or exclude studies in his review,
Ganellen explained that he included only studies using the revised version of the
DEPI for the following reason:

Rorschach studies that used the original DEPI and SCZI were excluded for this review
as the original versions of these scales were revised after they were found to have lim-
ited clinical utility. Instead, only studies using the revised DEPI and SCZI (Exner,
1991) were included. (p. 222)

I note that similar inclusion and exclusion criteria were applied to relevant
MCMI scales:

Studies using the MCMI–I were not included because the scales relevant to detection
of depression and schizophrenia were modified substantially after initial research
found these scales performed poorly. Only studies using the revised and presumably
 6 GANELLEN

improved versions of relevant MCMI–II scales were included. (Ganellen, 1996, p.

222)

It was for this reason that several of the studies mentioned by Wood et al. were ex-
cluded by Ganellen, including studies by Archer and Gordon (1988); Ball, Archer,
Gordon, and French (1991); and Viglione, Brager, and Haller (1988).
Wood et al. criticized this approach by saying the exclusion of these studies
“minimized the importance of existing negative research findings regarding the
DEPI” (p. 124). This statement, however, is inaccurate and may be misleading, as
Wood et al. ignored the fact that the revised version of the DEPI is substantially
Downloaded by [University of Dayton] at 02:42 28 December 2014

different from the original DEPI, the version used in most of the studies cited by
Wood et al. that were excluded from Ganellen’s (1996) review.
As noted earlier, the original DEPI was revised after research and clinical expe-
rience showed it had limited clinical utility (Exner, 1991). Although the original
DEPI and revised DEPI share a name, they differ in a number of important re-
spects. The differences between the original and revised versions of the DEPI can
be appreciated best by comparing the variables contained in each version of the
DEPI, as shown in Table 1.
Comparison of the original and revised forms of the DEPI shows that certain
variables are contained in both, but it also shows clear differences as variables are
included in the revised DEPI that are not part of the original DEPI, such as FD re-
sponses, S responses, the Intellectualization Index, COP responses, and the Isola-
tion Index. Furthermore, the range of scores for one variable contained in both
forms of the DEPI, the Egocentricity Index, was modified for the revised DEPI.
As a result of these differences, some individuals would be identified as being de-
pressed or nondepressed by both versions of the DEPI, whereas an individual
could be identified as being nondepressed using the criteria contained in the origi-
nal DEPI but depressed using the revised DEPI criteria. For instance, an individual
with (a) no Vista responses but three FD responses; (b) no Color-Shading Blend
responses but three S responses; (c) an Egocentricity Index of .45; (d) no C' re-

TABLE 1
Differences Between Original and Revised DEPI

Original DEPI Revised DEPI

1. Sum Vista > 0 Sum Vista > 0 or FD > 2

2. Color-Shading Blends > 0 Color-Shading Blends > 0 or S > 2
3. Egocentricity Index < .30 Egocentricity Index < .33 or > 44
4. Sum C' > 2 Sum C' > 2 or Sum Shading > FM + m
5. Sum MOR > 3 Sum MOR > 2 or Intellectualization Index > 3
6. COP < 2 or Isolation Index > .24

Note. DEPI = Rorschach Depression Index.

 WEIGHING EVIDENCE FOR THE RORSCHACH’S VALIDITY 7

sponses but eight Shading, as opposed to four FM + m, responses; (e) no MOR re-
sponses but a score of 5 on the Intellectualization Index; and (f) no COP responses
would be identified as being depressed on the more recent version of the DEPI but
not the original DEPI.
The implication of the differences between the original and revised DEPI is that
for both research and clinical purposes, the two versions of the DEPI should be
treated independently rather than as interchangeable forms of the same scale.
Stated differently, the original and revised versions of the DEPI are not parallel
forms of the same index but are independent, although related, markers of depres-
sion. Furthermore, the efficiency with which the original and revised forms of the
Downloaded by [University of Dayton] at 02:42 28 December 2014

DEPI correctly identify individuals as being depressed or nondepressed should not

be expected to be comparable given the differences in the composition of the two
versions of the DEPI.
In my opinion, it would be as inappropriate to equate the results of research in-
volving the original DEPI with the results of research addressing the revised DEPI
as it would be to equate the current Chicago Bulls basketball team (minus Michael
Jordan, Scottie Pippin, and Dennis Rodman) with the unbeatable, unstoppable, in-
comparable Chicago Bulls that won six NBA titles; although both teams wore
Bulls’ jerseys, the differences between the teams are (sadly) obvious. Put differ-
ently, the string of losses the current Bulls team racked up during the 1999 to 2000
season does not tarnish the record of the championship Bulls, just as the limitations
of the original DEPI do not provide data one way or the other about the strengths or
weaknesses of the revised DEPI.

ARTICLES NOT CITED BY GANELLEN

Wood et al. cited three articles published after Ganellen’s 1996 review appeared
that they claimed provide further evidence that the DEPI is not validly related to di-
agnoses of depression. One of these, a study by Carter and Dacey (1996), compared
the performance of the original DEPI, MMPI Scale 2, and Beck Depression Inven-
tory (Beck, Ward, Mendelson, Mock, & Erbaugh, 1961) in measuring depression.
The results of Carter and Dacey’s study should not be viewed as reflecting on the
conclusions presented by Ganellen because the original rather than the revised
DEPI was used by Carter and Dacey. For the reasons discussed previously, their re-
sults do not bear directly on the revised DEPI.
A second study to which Wood et al. referred was conducted by Carlson, Kula,
and St. Laurent (1997). In this study, Carlson et al. examined the performance of
the revised DEPI and the Coping Deficit Inventory, a measure developed by Exner
(1991) to identify psychological characteristics associated with a vulnerability to
depression that the DEPI does not tap. However, the Carlson et al. study provided
only limited information about the performance of the revised DEPI given the na-
 8 GANELLEN

ture of the sample used and, for the reasons I discuss later, would not have been in-
cluded in Ganellen’s (1996) review even if their study had been published before
his review appeared.
Ganellen (1996) explicitly stated that a study would be included in his review
only if the study examined the diagnostic performance of the MMPI, MCMI–II, or
Rorschach, or if indexes of diagnostic efficiency could be computed from the in-
formation contained in the study. Unfortunately, these statistics could not be com-
puted because Carlson et al.’s (1997) sample was composed entirely of
participants who met diagnostic criteria for a current episode of major depression.
Given the absence of groups who did not meet diagnostic criteria for an episode of
Downloaded by [University of Dayton] at 02:42 28 December 2014

major depression (e.g., patients with other diagnoses, such as schizophrenia, or

nonpatient controls), it was not possible to compute indexes of the revised DEPI’s
diagnostic efficiency other than its sensitivity. In other words, because of the na-
ture of Carlson et al’s sample, the revised DEPI’s specificity, positive predictive
power, negative predictive power, and overall classification rate could not be com-
puted. However, note that Carlson et al.’s results suggested that the revised DEPI
was relatively insensitive to identifying the presence of a current episode of de-
pression in their sample.
The third study cited by Wood et al. published after Ganellen’s (1996) review
was conducted by Archer and Krishnamurthy (1997), who compared the diagnos-
tic performance of the revised DEPI, the MMPI–A Scale 2 (Butcher et al., 1992),
and the MMPI–A Content Depression scale in a heterogeneous sample of adoles-
cent psychiatric patients. The Archer and Krishnamurthy study is the only one of
the studies cited by Wood et al. that met the criteria Ganellen used to include and
exclude studies for his review, namely examination of the effectiveness of MMPI
depression scales, or the revised DEPI, or both for identifying depression. How-
ever, when Archer and Krishnamurthy’s data are examined more closely, their
findings do not support Wood et al.’s position for reasons that are discussed
following.

Scale 2 Versus DEPI

Wood et al. were correct when they stated that the revised DEPI in Archer and
Krishnamurthy’s (1997) sample was insensitive to detecting the presence of a pri-
mary diagnosis of a mood disorder. However, according to the data presented by
Archer and Krishnamurthy in Table 3, Scale 2, the MMPI–A clinical scale tradi-
tionally associated with depression, was only slightly more sensitive than the re-
vised DEPI at identifying a diagnosis of depression. It is most appropriate to
compare the DEPI and Scale 2 using a cutoff T score of 65 because that is the cutoff
recommended to determine clinically significant levels of depression (Butcher et
al., 1992). Using this cutoff level, Scale 2 correctly identified 22 of the 56 depressed
 WEIGHING EVIDENCE FOR THE RORSCHACH’S VALIDITY 9

TABLE 2
Values Reported by Archer and Krishnamurthy (1997) for Scale 2 and DEPI

Variable True Positives True Negatives Sensitivity Specificity

Scale 2
T ≥ 60 31 73 .39 .84
T ≥ 65 22 81 .56 .76
DEPI ≥ 5 20 68 .36 .71

Note. DEPI = Rorschach Depression Index.

Downloaded by [University of Dayton] at 02:42 28 December 2014

patients in this sample, whereas the DEPI correctly identified 20 of the 56 de-
pressed patients. In other words, Scale 2 correctly identified only 2 more patients
than the revised DEPI.
It is possible that Wood et al.’s claim that the performance of Scale 2 was supe-
rior to the revised DEPI occurred because of an error contained in Archer and
Krishnamurthy’s (1997) Table 3. This error is obvious when the performance of
Scale 2 using a cutoff T score of 60 is compared with performance using a cutoff T
score of 65; according to their Table 3, 31 of 56 depressed patients were correctly
identified as being depressed using a cutoff of 60, whereas 22 of 56 depressed pa-
tients were correctly identified using a cutoff of 65. The values reported by Archer
and Krishnamurthy for Scale 2 and for the DEPI are shown in Table 2.
The findings reported in Archer and Krishnamurthy’s (1997) Table 3 indicated
that more depressed adolescents were correctly identified by Scale 2 as being de-
pressed when a cutoff T score of 60 as opposed to a T score of 65 was used (e.g., 31 vs.
22 of 56 depressed adolescents using these cutoff scores, respectively). Stated dif-
ferently, Scale 2 accurately identified more depressed adolescents as being de-
pressed when a cutoff of 60 rather than 65 was used. Given these values, one would
expect a higher sensitivity value using the cutoff level of 60 rather than 65 because
sensitivity refers to the true positive rate of identification of individuals with a partic-
ular disorder (Kessel & Zimmerman, 1993). However, the diagnostic efficiency sta-
tistics contained in Archer and Krishnamurthy’s Table 3, reproduced previously,
report the opposite pattern. Rather than showing Scale 2 to be more sensitive to the
presence of depression using a cutoff score of 60 as opposed to 65, they report greater
sensitivity for Scale 2 using a cutoff of 65 (sensitivity = .56) than 60 (sensitivity =
.39). This is logically impossible. Closer examination of the values contained in Ar-
cher and Krishnamurthy’s Table 3 reveals that the table contains an error, namely
that the values for Scale 2 were reversed for the different cutoff values.
When the values shown in Archer and Krishnamurthy’s (1997) Table 3 are re-
calculated using a T score of 65 as the cutoff for identification of depression, the
cutoff recommended to identify depression in clinical practice (Butcher et al,
1992), the sensitivity of Scale 2 is .39, only slightly higher than that of the revised
DEPI, which had a sensitivity of .36. In other words, Archer and Krishnamurthy’s
 10 GANELLEN

data document comparable performance for the revised DEPI and Scale 2 rather
than showing a superiority for the MMPI–A, as claimed by Wood et al.
Note that the sensitivity of both Scale 2 and the revised DEPI were disappoint-
ingly low in Archer and Krishnamurthy’s (1997) sample. This may have occurred
for several reasons, including the possibility that the scores produced by the ado-
lescent patients on both the MMPI–A and Rorschach were atypical in some impor-
tant respects, a possibility strongly suggested by scores on several relevant
variables. For example, descriptive statistics for Rorschach variables from this
sample were originally presented by Krishnamurthy, Archer, and House (1996).
Although the adolescent patients in their sample produced an average number of
Downloaded by [University of Dayton] at 02:42 28 December 2014

responses on the Rorschach (R = 22.39), the responses were characterized by a pre-

dominance of Pure Form responses as shown by the unusually high average value
of 1.84 for Lambda. This is quite unusual for adolescents in general, as scores of
1.0 or greater occur in less than 10% of the nonclinical adolescent samples be-
tween the ages of 13 and 16 reported by Exner (1991). The mean scores for
Lambda ranged from .65 to .67 in these samples of nonpatient adolescents. Simi-
larly, the mean number of Pure Form responses in Archer and Krishnamurthy’s
sample was considerably higher than expected (11.66), as the mean score for Pure
Form responses ranged from 6.48 to 6.96 in the adolescent nonpatient samples re-
ported by Exner (1991).
These unusual scores cannot be explained by Archer and Krishnamurthy’s
(1997) sample’s psychiatric status, as the mean score for Lambda in a sample of
100 adolescent psychiatric patients who met Diagnostic and Statistical Manual of
Mental Disorders (DSM–III–R; American Psychiatric Association, 1987) diag-
nostic criteria for depression was .74 and the mean score for Pure Form responses
was 7.03 (Exner & Weiner, 1995). In Exner and Weiner’s sample of 140 adoles-
cents who met DSM–III–R diagnostic criteria for conduct disorder, the mean score
for Lambda and Pure Form responses was 0.91 and 9.61, respectively. The unusual
elevations on Lambda and Pure Form responses in Archer and Krishnamurthy’s
(1997) sample are noteworthy, as it is possible these response tendencies lowered
the values of several of the variables that load on the revised DEPI, such as Vista,
FD, C', Shading, Color-Shading Blends, FM, or m responses.
Not only were the scores on the Rorschach unusual in Archer and
Krishnamurthy’s (1997) sample, but scores on the MMPI–A were also unexpected
in several respects. For example, although the depressed group scored signifi-
cantly higher than the nondepressed groups on Scale 2, the depressed group’s
mean score of 62.8 was lower than a T score of 65, the cutoff used to determine
clinically significant levels of depression (Butcher et al., 1992). Furthermore, the
depressed group scored higher than the conduct disorder group on Scale 4 (64.3 vs.
61.1, respectively), a measure related to acting out behavioral problems, failure to
internalize societal norms, difficulties with impulse control, and abuse of alcohol
or street drugs (Butcher et al., 1992). Although this difference was not statistically
 WEIGHING EVIDENCE FOR THE RORSCHACH’S VALIDITY 11

significant, the fact that depressed patients as a group scored higher than conduct
disordered patients on the MMPI–A clinical scale most directly related to charac-
teristics associated with conduct disorder is puzzling. These findings raise some
questions as to how much confidence can be placed in the findings from Archer
and Krishnamurthy’s sample given what may be anomalous findings for both the
Rorschach and the MMPI–A.

Implications for the Revised DEPI

Although Wood et al. claimed that extensive research conducted by researchers

Downloaded by [University of Dayton] at 02:42 28 December 2014

other than Exner casts doubt on the validity of the DEPI, in actuality only one study
exists that has examined the revised DEPI’s diagnostic performance: the study by
Archer and Krishnamurthy (1997). Rather than demonstrating that the MMPI–A
Scale 2 is more strongly related to depression than the revised DEPI, as claimed by
Wood et al., Archer and Krishnamurthy’s study actually showed comparable per-
formance for the two measures. As discussed previously, the disappointing perfor-
mance of both the MMPI–A and revised DEPI may have occurred because of
anomalies in Archer and Krishnamurthy’s sample.
For the reasons discussed previously, Wood et al.’s characterization of empiri-
cal evidence concerning the DEPI’s validity should be viewed with considerable
caution. Although their presentation of studies at first glance appears to indicate
that considerable negative evidence exists weighing against the revised DEPI, this
impression was created to a large extent by Wood et al.’s failure to distinguish be-
tween the original and revised DEPI, a distinction explicitly pointed out by
Ganellen (1996). Their decision to treat the two versions of the DEPI as being
identical when critiquing Ganellen’s review is therefore puzzling.
One might question how objectively Wood et al. considered these issues in light
of their inaccurate characterization of other studies. For instance, they cited a study
by Meyer (1993) as providing evidence that the original SCZI does not add incre-
mental validity to the prediction of a diagnosis of a psychotic disorder above and
beyond what can be obtained using the MMPI. However, Meyer’s (1993) article
did not address this issue. Instead, the analyses Meyer (1993) conducted examined
the extent to which response productivity on the Rorschach affects MMPI–Ror-
schach interrelations, not correct diagnostic classification using the MMPI and
Rorschach alone or in combination.
Questions about Wood and colleagues’ (Wood et al., 1996; 1997; 1999) ob-
jectivity are also raised by the manner in which they present evidence they claim
shows that the CS’s psychometric properties have not been substantiated and
their suggestion that the use of the Rorschach in applied clinical settings may be
unethical. Calling for a moratorium on the use of an established, widely used
test is a rather extreme proposal, one not advocated even by the harshest critics
of other tests in common usage, such as those who have raised concerns about
 12 GANELLEN

the theoretical, psychometric, and normative foundations of the MMPI–2

(Helmes & Reddon, 1993). As discussed earlier, many of the reasons Wood and
colleagues have given for reaching these conclusions appear to be unfounded
because they inappropriately treated the two versions of the DEPI as one; they
inaccurately characterized Archer and Krishnamurthy’s (1993a; 1993b) reviews
as providing evidence that the MMPI is more valid than the Rorschach; they
suggested that different standards of reliability be applied to the Rorschach than
to other assessment instruments; and they misstated the findings of Meyer’s
(1993) study.
Downloaded by [University of Dayton] at 02:42 28 December 2014

FUTURE RESEARCH DIRECTIONS

Wood et al. noted that Ganellen’s (1996) conclusions were based exclusively on
findings reported by Exner (1991) and pointed out that these conclusions “may
have been based on an atypical data set” (p. 124). These limitations were explicitly
recognized by Ganellen (1996), who emphasized that his findings should be taken
cautiously in large part because of concerns that the statistics for the DEPI and
SCZI were drawn only from Exner’s (1991) samples:

Overall, these factors limit the conclusions that can be drawn concerning the Ror-
schach’s diagnostic efficiency using these methods.… These data were used to draw
preliminary conclusions about the Rorschach’s diagnostic performance, conclusions
that may be modified or altered as additional data are published. The need for further
study of the Rorschach’s clinical utility is clear. (Ganellen, 1996, p. 235)

I enthusiastically agree with Wood et al. that additional research is needed to

examine how effectively the revised DEPI functions to identify depression in
other samples as well as to examine its convergent validity with other measures of
depression. However, one should not conclude that the need for further research in
this area indicates that the data reviewed by Ganellen (1996) should be ignored or
treated as being meaningless.
Wood et al. raised another concern about Ganellen’s (1996) conclusions,
namely that most of the studies included in Ganellen’s review investigated the di-
agnostic efficiency of each test (MMPI, MCMI–II, and Rorschach) separately,
rather than comparing all measures in the same sample. Wood et al. (1999) de-
scribed the former approach as a “between-groups” strategy and the latter strategy
a “within-groups” strategy, and indicated that the between-group strategy is pref-
erable to the within-group strategy when one is interested in comparing the perfor-
mance of two or more measures, such as the MMPI, MCMI–II, and Rorschach.
They criticized Ganellen’s conclusions as he “relied on the problematic between-
groups approach, rather than the within-groups approaches, when he concluded
 WEIGHING EVIDENCE FOR THE RORSCHACH’S VALIDITY 13

that the MMPI and Rorschach are comparable for evaluating diagnoses of depres-
sion” (Wood et al., 1999, p. 124).
As Ganellen (1996) stated, however, he did not identify any published studies in
which the revised DEPI and either or both the MMPI or MCMI–II were adminis-
tered to the same participants. As discussed previously, the only published study
comparing the Rorschach and MMPI available at the time of his review used the
original DEPI (Archer & Gordon, 1988). For the reasons discussed earlier, Ganellen
did not include this study in his review. It should be noted that the Archer and Gordon
study also used the original SCZI, rather than the revised SCZI. Thus, the implica-
tions of Archer and Gordon’s findings for the validity of the revised SCZI are limited
Downloaded by [University of Dayton] at 02:42 28 December 2014

for the same reasons as the results concerning the revised DEPI.
Similar to Wood et al.’s recommendation that comparisons of the diagnostic ef-
ficiency of the Rorschach and MMPI ideally should be based on studies in which
both measures are administered to the same participants, Ganellen (1996) recom-
mended that the optimal way to examine the validity of these measures was to
“compare the rates of correct diagnostic identification for psychological tests indi-
vidually and in combination with one another in a sample of patients who com-
pleted the MMPI, MCMI–II, and Rorschach” (p. 240). These methodological
issues as well as other suggestions for future research examining the convergent
and incremental validity of Rorschach and MMPI variables, are discussed by
Ganellen in detail.
In summary, Wood et al.’s comments may be taken by some as indicating (a)
that evidence supporting the reliability and validity of the Rorschach is either
weak or has not been established and (b) that the available evidence proves that
the revised DEPI does not function as well as claimed by Ganellen (1996). To
the contrary, however, ample evidence exists demonstrating that the Rorschach
can be scored reliably using the criteria of the CS (Meyer, 1997); that Rorschach
variables in general have respectable levels of criterion-related validity
(Bornstein, 1996; Hiller et al., 1999); and that the MMPI and Rorschach have
comparable levels of criterion-related validity with the MMPI outperforming the
Rorschach in certain respects and the Rorschach outperforming the MMPI in
others (Bornstein, 1999; Hiller et al., 1999). Although conclusions concerning
the DEPI must be considered preliminary until further evidence accumulates,
Wood et al. did not provide any reason to modify or reject Ganellen’s character-
ization of the revised DEPI “as a promising psychometric marker of depression”
(p. 236), which deserves serious attention by researchers and clinicians.

REFERENCES

American Psychiatric Association. (1987). Diagnostic and statistical manual of mental disorders (3rd
ed., rev.). Washington, DC: Author.
 14 GANELLEN

Archer, R. P., & Gordon, R. A. (1988). MMPI and Rorschach indices of schizophrenic and depressive
diagnoses among adolescent inpatients. Journal of Personality Assessment, 52, 276–287.
Archer, R. P., & Krishnamurthy, R. (1993a). Combining the Rorschach and the MMPI in the assessment
of adolescents. Journal of Personality Assessment, 60, 132–140.
Archer, R. P., & Krishnamurthy, R. (1993b). A review of MMPI and Rorschach interrelationships in
adult samples. Journal of Personality Assessment, 61, 277–293.
Archer, R. P., & Krishnamurthy, R. (1997). MMPI–A and Rorschach indices related to depression and
conduct disorder: An evaluation of the incremental validity hypothesis. Journal of Personality As-
sessment, 69, 517–533.
Ball, J. D., Archer, R. P., Gordon, R. A., & French, J. (1991). Rorschach depression indices with chil-
dren and adolescents: Concurrent validity findings. Journal of Personality Assessment, 57,
Downloaded by [University of Dayton] at 02:42 28 December 2014

465–476.
Beck, A., Ward, C., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depres-
sion. Archives of General Psychiatry, 4, 53–63.
Bornstein, R. G. (1995a). Comorbidity of dependent personality disorder and other psychological disor-
ders: An integrative review. Journal of Personality Disorders, 9, 286–303.
Bornstein, R. G. (1995b). Interpersonal dependency and physical illness: The mediating roles of stress
and social support. Journal of Social and Clinical Psychology, 14, 225–243.
Bornstein, R. G. (1996). Construct validity of the Rorschach Oral Dependency Scale. Psychological As-
sessment, 8, 200–205.
Bornstein, R. G. (1999). Criterion validity of objective and projective dependency tests: A meta-analytic
assessment of behavioral prediction. Psychological Assessment, 11, 48–57.
Butcher, J. N., Williams, C. L., Graham, J. R., Archer, R. P., Tellegen, A., Ben-Porath, Y. S., &
Kaemmer, B. (1992). Minnesota Multiphasic Personality Inventory–A (MMPI–A): Manual for ad-
ministration, scoring, and interpretation. Minneapolis: University of Minnesota Press.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the
multitrait–multimethod matrix. Psychological Bulletin, 45, 81–105.
Carlson, C. F., Kula, M. L., & St. Laurent, C. M. (1997). Rorschach revised DEPI and CDI with inpatient
major depressives and borderline personality disorder with major depression: Validity issues. Jour-
nal of Clinical Psychology, 53, 51–58.
Carter, C. L., & Dacey, C. (1996). Validity of the Beck Depression Inventory, MMPI, and Rorschach in
assessing adolescent depression. Journal of Adolescence, 19, 223–231.
Exner, J. E., Jr. (1991). The Rorschach: A comprehensive system: Vol. 2. Interpretation (2nd ed.). New
York: Wiley.
Exner, J. E., Jr. (1993). The Rorschach: A comprehensive system: Vol. 1. Basic foundations (3rd ed.).
New York: Wiley.
Exner, J. E., Jr., & Weiner, I. B. (1995). The Rorschach: A comprehensive system: Vol. 3. Assessment of
children and adolescents (2nd ed.). New York: Wiley.
Ganellen, R. J. (1996). Comparing the diagnostic efficiency of the MMPI, MCMI–II, and Rorschach: A
review. Journal of Personality Assessment, 67, 219–243.
Garb, H. N., Florio, C. M., & Grove, W. M. (1988). The validity of the Rorschach and the Minnesota
Multiphasic Personality Inventory: Results from meta-analyses. Psychological Science, 9,
402–404.
Hathaway, S. R., & McKinley, J. C. (1943). Manual for the Minnesota Multiphasic Personality Inven-
tory. New York: Psychological Corporation.
Helmes, E., & Reddon, J. R. (1993). A perspective on developments in assessing psychopathology: A
critical review of the MMPI and MMPI–2. Psychological Bulletin, 113, 453–471.
Hiller, J. B., Rosenthal, R., Bornstein, R. F., Berry, D. T. R., & Brunnel-Neuleib, S. (1999). A compara-
tive meta-analysis of Rorschach and MMPI validity. Psychological Assessment, 11, 278–296.
 WEIGHING EVIDENCE FOR THE RORSCHACH’S VALIDITY 15

Hunsley, J., & Bailey, J. M. (1999). The clinical utility of the Rorschach: Unfulfilled promises and an
uncertain future. Psychological Assessment, 11, 266–277.
Kessel, J. B., & Zimmerman, M. (1993). Reporting errors in studies of diagnostic performance of self-
administered questionnaires: Extent of the problem, recommendations for standardized presenta-
tions of results, and implications for the peer review process. Psychological Assessment, 5,
395–399.
Krishnamurthy, R., Archer, R. P., & House, J. J. (1996). The MMPI–A and Rorschach: A failure to es-
tablish convergent validity. Assessment, 3, 179–191.
Masling, J. M., Rabie, L., & Blondheim, S. H. (1967). Obesity, level of aspiration, and Rorschach and
TAT measures of oral dependence. Journal of Consulting Psychology, 31, 233–239.
Meyer, G. J. (1993). The impact of response frequency on the Rorschach constellation indices and on
Downloaded by [University of Dayton] at 02:42 28 December 2014

their validity with diagnostic and MMPI–2 criteria. Journal of Personality Assessment, 60,
153–180.
Meyer, G. J. (1996). The Rorschach and MMPI: Toward a more scientific understanding of cross-
method assessment. Journal of Personality Assessment, 67, 558–578.
Meyer, G. J. (1997). Assessing reliability: Critical corrections for a critical examination of the Ror-
schach Comprehensive System. Psychological Assessment, 9, 480–489.
Millon, T. (1987). Millon Clinical Multiaxial Inventory–II manual. Minneapolis, MN: National Com-
puter Systems.
Navran, L. (1954). A rationally derived MMPI scale to measure dependence. Journal of Consulting Psy-
chology, 18, 192.
Parker, K. C. J., Hanson, R. K., & Hunsley, J. (1988). MMPI, Rorschach, and WAIS: A meta-analytic
comparison of reliability, stability, and validity. Psychological Bulletin, 103, 367–373.
Viglione, D. J., Brager, R. C., & Haller, N. (1988). Usefulness of structural Rorschach data in identify-
ing inpatients with depressive symptoms: A preliminary study. Journal of Personality Assessment,
52, 524–529.
Wechsler, D. (1997a). WAIS–III administration and scoring manual. San Antonio, TX: Psychological
Corporation.
Wechsler, D. (1997b). WMS–III administration and scoring manual. San Antonio, TX: Psychological
Corporation.
Weiner, I. B. (1996). Some observations on the validity of the Rorschach Inkblot Method. Psychological
Assessment, 8, 206–213.
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996). The Comprehensive System for the Ror-
schach: A critical examination. Psychological Science, 7, 3–10.
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1997). The reliability of the Comprehensive System
for the Rorschach: A comment on Meyer (1997). Psychological Assessment, 9, 490–494.
Wood, J. M., Nezworski, M. T., Stejskal, W. J., Garven, S., & West, S. G. (1999). Methodological issues
in evaluating Rorschach validity: A comment on Burns and Viglione (1996), Weiner (1996), and
Ganellen (1996). Assessment, 6, 115–129.

Ronald J. Ganellen
Suite 2810
405 North Wabash Avenue
Chicago, IL 60611
E-mail: r-ganellen@northwestern.edu

Received November 6, 2000