Journal of Consulting and Ctinica] Psychology Copyright 1989 by the American Psychological Association. Inc.

1989, Vol. 57,No. 1,138-147 0022-006X/89/J00.7!

Power to Detect Differences Between Alternative Treatments

in Comparative Psychotherapy Outcome Research

Alan E. Kazdin and Debra Bass

Western Psychiatric Institute and Clinic
University of Pittsburgh School of Medicine

Comparative studies of psychotherapy often find few or no differences in the outcomes that alterna-
tive treatments produce. Although these findings may well reflect the comparability of alternative
treatments, as a rule, studies are often not sufficiently powerful to detect the sorts of effects sizes
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

likely to be found when two or more treatments are contrasted. The present survey evaluated the
This document is copyrighted by the American Psychological Association or one of its allied publishers.

power of psychotherapy outcome studies to detect differences for contrasts of two or more treatments
and treatment versus no-treatment control conditions. Outcome studies (A' = 85) were drawn from
nine journals over a 3-year period (1984-1986). Data in each article were examined first to provide
estimates of effect sizes and then to evaluate statistical power at posttreatment and follow-up. The
findings indicate that the power of studies to detect differences between treatment and no treatment
is generally quite adequate given the relatively large effect sizes usually evident for this comparison.
On the other hand, the power is relatively weak to detect the small-to-medium effect sizes likely to
be evident when alternative treatments are contrasted with each other. Thus, the equivalent outcomes
that treatments produce (i.e., "no difference") may be due to the relatively weak power of the tests.
The implications for interpreting current outcome studies and for designing future comparative
studies are highlighted.

In psychotherapy research, comparative outcome studies ad-
dress the question of which of two or more techniques is the
most effective for a particular clinical problem and patient pop-
ulation. Often the constituent treatments reflect conflicting
conceptual views about the nature of dysfunction, the focus of
treatment, and the techniques required to produce change.
Consequently, comparative studies generate tremendous inter-
est (see Heimberg& Becker, 1984; Kazdin, 1986; Lambert, Sha-
piro, & Bergjn, 1986; Luborsky, Singer, & Luborsky, 1975;
Rachman & Wilson, 1980; Stiles, Shapiro, & Elliott, 1986).
Outcome evidence on the relative effectiveness of alternative
treatments has been evaluated extensively. Conclusions have
been drawn from individual comparative outcome investiga-
tions (e.g., Sloane, Staples, Cristol, \orkston, & Whipple,
1975), box-score (e.g., Luborsky et al., 1975) and narrative re-
views (e.g., Kazdin & Wilson, 1978; Lambert etal., 1986; Rach-
man & Wilson, 1980), and meta-analyses (see Brown, 1987).
Although individual studies and large-scale reviews occasion-
ally argue for the superiority of one technique over another,
evaluations of the literature usually suggest that treatments tend
not to differ or at least not to differ very much in the outcomes
they produce.
Completion of this article was facilitated by Research Scientist Devel-
opment Award MH00353 and by Grant MH35408 from the National
Institute of Mental Health. We are extremely grateful to Jacob Cohen,
Larry V. Hedges, and Kenneth I. Howard. Special thanks are also ex-
tended to Helena C. Kraemer, who provided comments and guidance
on prior drafts.
Correspondence concerning this article should be addressed to Alan
E. Kazdin, Western Psychiatric Institute and Clinic, 3811 O'Hara
Street, Pittsburgh, Pennsylvania 15213.
The absence of clear outcome differences between alternative
treatments has served as an important point of departure re-
garding the nature of therapy and the processes that account for
change (see Stiles et al., 1986). As a prominent example, Frank
(1982) suggested that therapeutic change results from several
features that are common among different techniques. Thus,
no outcome differences between treatments might be expected
given their common ingredients. This view has been bolstered
in part by the recognition that different techniques often appear
more diverse in theory than they do in clinical practice (e.g.,
Klein, Dittmann, Parloff, & Gill, 1969; Sloane et al., 1975).
Common ingredients, particularly the special relationship be-
tween client and therapist and the provision of support, empa-
thy, and concern, are pervasive among alternative techniques
(Waterhouse & Strupp, 1984).
Current estimates suggest that well over 400 psychotherapy
techniques are in use for adults (Karasu, personal communica-
tion, March 1, 1985) and that over 230 techniques are in use
for children (Kazdin, 1988). If we assume that many of these
techniques are effective in producing therapeutic change, it is
difficult to conceive that they vary in effectiveness or operate
through different therapeutic processes. Yet, this assumption is
not tantamount to stating that the results from viable treatment
contenders for a given clinical problem will be similar. Whether
treatment outcomes differ can only be evaluated empirically.
The finding that treatments do not differ in many, if not most,
tests may mean that treatments are approximately equal in their
effects. Yet, it is important to know if the studies are, as a rule,
designed to detect outcome differences. When two or more ac-
tive interventions are expected to produce change, the investi-
gation must be sufficiently sensitive to detect what could prove
to be relatively small differences.

138
 COMPARATIVE OUTCOME STUDIES AND STATISTICAL POWER
A critical research issue is the extent to which an investiga-
tion can detect differences between groups when differences ex-
ist within the population. This notion is referred to as statistical
power and reflects the probability that the test will lead to rejec-
tion of the null hypothesis.1 Power is a function of the criterion
for statistical significance (alpha), sample size (.V), and the
difference that exists between groups (effect size).
Although power is an issue in virtually all research, it raises
special issues in studies where two or more conditions (groups)
are not significantly different (Fagley, 198S; Freiman, Chal-
mers, Smith, & Kuebler, 1978; Kazdin, 1980). The absence of
significant differences can contribute to knowledge under a va-
riety of circumstances. However, an essential precondition is
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
that the investigation be sufficiently powerful to detect mean-
This document is copyrighted by the American Psychological Association or one of its allied publishers.
ingful differences. In the vast majority of psychotherapy out-
come studies that have contrasted two or more treatments, the
power may have been relatively weak due to small samples sizes.
There are many reasons to suspect that outcome studies as a
general rule provide weak tests. Over 25 years ago, Cohen
(1962) examined clinical research published in the Journal of
Abnormal and Social Psychology for a 1-year period (1960).
Over 2,000 statistical tests were identified (from 70 articles) that
were considered to reflect direct tests of the hypotheses. To eval-
uate power, Cohen examined different effect sizes, that is, the
magnitude of the differences between alternative groups based
on standard deviation units. Cohen distinguished three levels of
effect sizes (small = .25, medium = .50, and large = 1.00) and
evaluated the power of published studies to detect differences at
these levels, assuming alpha = .05 and using nondirectional
(two-tailed) tests.
The results indicated that power was generally weak for de-
tecting differences equivalent to small and medium effect sizes.
For example, the mean power of studies to detect differences
reflecting small and medium effect sizes was. 18 and .48, respec-
tively. This means that, on the average, studies had slightly less
than a 1 in 5 chance to detect small effect sizes and less than a
1 in 2 chance to detect medium effect sizes. These levels are
considerably below the recommended level of power, .80 (4 in 5
chance).2 Cohen concluded that the power of the studies was
weak and that sample sizes in future studies should routinely
be increased (see also Cohen, 1977).
A more recent analysis examined if the situation has im-
proved since Cohen's earlier portrayal. Rossi, Rossi, and Cot-
trill (1984) sampled research from the Journal of Personality
and Social Psychology and the Journal of Abnormal Psychology,
journals that are descendants of the publication Cohen ana-
lyzed. The data were obtained from 142 articles in the 1982
volume of each journal. Rossi et al. found that 3%, 26%, and
69% of the studies in 1982 had power above .80 for detecting
small, medium, and large effects, respectively. This compared
with parallel data from Cohen (1962) of 0%, 9%, and 79%, re-
spectively. Although slight increases in power were evident for
detecting small and medium effects, the vast majority of studies
were quite weak with regard to detecting such effects.
The applicability of these findings to psychotherapy outcome
research might be challenged. The surveys of Cohen (1962) and
Rossi et al. (1984) covered only a 1 -year period, sampled a small
number of journals, and encompassed diverse areas of research
within clinical and social psychology. Perhaps, more to the
139
point, the surveys did not focus on psychotherapy outcome ex-
clusively or primarily. Yet, the characteristics of psychotherapy
outcome studies may make the conclusions even more appli-
cable.
To begin with, small sample sizes may be dictated in part by
the subject matter due to difficulties in recruiting, treating, and
retaining large samples of homogeneous subjects (Kraemer,
1981). In contemporary outcome research, studies typically in-
clude 20 or fewer subjects per group (e.g., Cross, Sheehan, &
Khan, 1982; DiLoreto, 1971; Liberman & Eckman, 1981;
Rush, Beck, Kovacs, & Hollon, 1977). Indeed, it is not difficult
to find studies comparing alternative treatment and control
conditions in which the sample sizes are 10 or fewer cases per
condition (e.g., Forman, 1980; Yu, Harris, Solovitz, & Franklin,
1986). Inadequate levels of power could be a rival explanation
for the absence of treatment differences.
A second problem of psychotherapy research is the loss of
subjects over time. Apart from the possible selection biases that
attrition can introduce, sample sizes (and power) may be mark-
edly reduced by the end of treatment and by the follow-up as-
sessment (Garfield, 1980). The toll of attrition on the design is
often high. In some treatment programs, between 40% and 50%
of subjects may drop out of treatment (e.g., Fleischman, 1981;
Patterson, 1974; Viale-Val, Rosenthal, Curtiss, & Marohn,
1984). In one large-scale comparative outcome study, almost
90% of the cases (396 of 450) that completed treatment were
lost at follow-up 1 year later (Feldman, Caplinger, & Wodarski,
1983). Thus, in psychotherapy outcome studies, sample size
and power may weaken considerably over time. If the conclusion
of no differences between treatments is not evident at posttreat-
ment, it may well be reached at follow-up.
One may be able to discern from psychotherapy research the
approximate effect sizes for classes of comparisons such as
treatment versus no treatment and treatment versus treatment.
Indeed, such effect sizes have often been examined in the con-
text of meta-analysis. Effect sizes obtained after experiments
are completed provide an estimate of population effect sizes
(Cohen, 1973). It is important to examine effect sizes for com-
parisons of alternative therapy techniques, not only to deter-
mine the magnitude of the differences with which we may be
working, but also to interpret studies in which critical tests of
treatment are provided and few or no differences emerge.
From the published research, one can estimate the power of
studies to detect differences of the magnitudes that emerge for
1
Power (1-beta) is the probability of rejecting the null hypothesis
when it is false. Stated differently, power is the likelihood of finding
differences between the treatments when, in fact, the treatments are
truly different in their outcomes.
2
The level of power that is adequate is not easily specified or justified
mathematically. As with the level of confidence (alpha), the decision is
based on convention about the margin of protection one should have
against falsely accepting the null hypothesis (beta). Cohen (1965) rec-
ommended adoption of the convention that beta = .20 and hence power
(l-beta) = .80 when alpha = .05. This translates to the 4 in 5 likelihood
of detecting an effect when a difference exists in the population. Al-
though power a .80 is used as a criterion for discussion in the present
article, a higher level (.90, .95) is often encouraged as the acceptable
criterion (e.g., Freiman, Chalmers, Smith, & Kuebler, 1978; Friedman,
Furberg,&DeMets, 1985).
 140 ALAN E. KAZDIN AND DEBRA BASS
comparisons of alternative treatment conditions. There is, of
course, the bias that derives from consulting only published
studies. Such studies may be more prone to yield significant
effects, whereas those that did not yield such effects may be un-
published and relegated to the investigator's file drawer (Rosen-
thai, 1979). However, comparative outcome studies with few or
no differences are often published (e.g., Sloane et al., 1975). The
reasons include the keen interest in the comparisons in their
own right, the likely differences between the treatments and the
no-treatment or waiting-list control condition, and the evalua-
tion of therapeutic processes common to alternative tech-
niques.
In the present article, we examine the power of treatment
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
outcome studies to detect differences when alternative treat-
This document is copyrighted by the American Psychological Association or one of its allied publishers.
ments are compared. The question is important because the
general absence of differences between two or more treatments
tested in a given study may be viewed differently, depending on
whether the experimental tests have adequate power. It might
well be that the outcome evidence argues generally for the ab-
sence of major treatment differences. To argue this position re-
quires some assurance that the tests comparing different treat-
ments are sufficiently powerful. The question can be addressed
by examining the effect sizes of current outcome studies to de-
tect differences between alternative treatment conditions and
the sample sizes that are used.
In meta-analyses of psychotherapy, the usual goal is to com-
pare treatment against a control group and to draw conclusions
about the magnitude of effect sizes for specific techniques (e.g.,
behavior therapy vs. cognitive therapy). A related goal in the
present article was to examine the power of psychotherapy stud-
ies to detect differences when two or more treatments are in-
cluded. However, the specific techniques and the direction of
their differences in relation to each other were not of interest.
The objective was to examine the power to detect differences
with different types or classes of comparisons.
The power of psychotherapy studies was examined by esti-
mating effect sizes from findings of psychotherapy outcome
studies for the comparisons of interest. Effect sizes and power
were estimated for comparisons at posttreatment and follow-up
because of the changes in sample sizes (attrition) over the course
of therapy studies and the different and occasionally diametri-
cally opposed outcomes that alternative treatments can produce
at different points in time (see Kazdin, 1988). Treatment stud-
ies were surveyed from nine journals over a 3-year period. Sta-
tistical tests were examined from the psychotherapy studies to
estimate their power to detect differences where they exist in
comparing alternative treatment conditions.
Method
Studies of Interest
Psychotherapy outcome studies were selected for the present evalua-
tion. Psychotherapy was defined to include interventions designed to
decrease distress, psychological symptoms, and maladaptive behavior
or designed to improve adaptive and prosocial functioning through the
use of interpersonal interaction, counseling, or activities following a spe-
cific treatment plan (see Garfield, 1980; Walrond-Skinner, 1986). Ex-
cluded from the definition were interventions using medication as a
form of treatment or interventions directed singularly at educational
goals (e.g., reading improvement). Outcome investigations referred to
studies designed to measure some facet of psychological adjustment or
functioning after treatment was completed (posttreatment). At least two
groups or conditions were required for the study to be included. The
groups could include any combination of treatment and control condi-
tions. Although primary interest was in studies comparing two or more
treatments, all psychotherapy outcome studies were included if there
were at least two groups. This inclusion criterion was adopted to permit
evaluation of power for the different comparisons (one treatment vs.
another treatment vs. no-treatment).
To provide a sample of psychotherapy outcome research, nine jour-
nals were studied for a 3-year period (1984-1986). Four journals were
selected because they were the most frequent contributors to meta-anal-
yses of psychotherapy outcome research (Shapiro & Shapiro, 1982;
Smith, Glass, AMillei; 1980). The journals included Ox Journal of Con-
sulting and Clinical Psychology, the Journal of Counseling Psychology,
Behavior Therapy, and Behaviour Research and Therapy. In addition,
five other journals were selected because they reflect disciplines (psychi-
atry) in which treatment outcome research is published, they are often
viewed as publication outlets for clinical trials, or their publication do-
main explicitly delineates psychotherapy research. Thus, these latter
journals were selected largely on the basis of face validity, that is, be-
cause they include therapy research and appear by virtue of publication
domain to be a primary outlet For this group, five journals were se-
lected: Archives of General Psychiatry, American Journal of Psychiatry,
Psychotherapy: Theory, Research and Practice, British Journal of Psy-
chiatry, and British Journal of Clinical Psychology. From the different
journals for a 3-year period, 120 psychotherapy outcome studies were
identified. Of these, 85 (70.8%) provided sufficient statistical informa-
tion to compute effect sizes.3
Comparisons of Interest
To evaluate the statistical power of these psychotherapy studies, three
types of comparisons were made: (a) comparison of each treatment with
the other treatments included in the study, (b) comparison of treatment
with a no-treatment control condition, and (c) comparison of treatment
with an active, nonspecific (attention placebo) control condition. Of pri-
mary interest was the fust comparison that addresses the question of
the effect size and power when alternative treatments are compared. The
comparisons involving treatment and alternative control conditions
provide important baseline data regarding the power for delecting
different types of effects in outcome research.
Treatments consisted of alternative psychotherapy techniques given
the definition noted previously. No-treatment control referred to any
group from which treatment was withheld during the pre- to posttreat-
ment interval for the treatment group. Active control referred to groups
in which an attention placebo or discussion control procedure was used
to control for such factors as meeting with a therapist and attending
sessions.4
3
A list of the studies for the present analyses is available on request.
4
In meta-analyses of psychotherapy, effect sizes obtained by compar-
ing treatment versus alternative types of control groups are occasionally
no different (Casey & Herman, 1985). For the present purposes, the dis-
tinction between no-treatment (including waiting-list) control and at-
tention-placebo control was nevertheless retained. The rationale was
the likelihood that effect size and power would be lower for detecting
differences between treatment and an attention-placebo group than be-
tween treatment and a no-treatment group. The differences could have
important implications for designing studies if sample sizes need to be
planned in accord with different expected effect sizes. Also, evaluations
of alternative treatments have occasionally noted that a strong placebo
is often no more or less effective than a technique considered to be more
 COMPARATIVE OUTCOME STUDIES AND STATISTICAL POWER
Measure and Calculation of Effect Size
Within each study, an effect size was calculated between each pair of
groups on each outcome measure. Effect size (ES) was denned as (m, -
m2)/i where m, and m2 refer to two group means (treatment or control)
and s is the pooled within-group standard deviation.9 Each ES was clas-
sified as coming from a comparison of two treatments, of treatment
versus no treatment, or of treatment versus an active control. Also, each
ES was classified as evaluated at posttreatment or at follow-up. If multi-
ple follow-up assessments were reported, only the last (longest duration)
assessment was considered. Thus, six types of ESs might have been de-
rived from a single study. Within a study, multiple values of each effect
size might have been obtained if there were several outcome measures.
The ESs of each type within a study were averaged, so that each study
contributed no more than one mean ES per comparison of interest.6
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Of central interest was the case in which two or more treatments (T,,
This document is copyrighted by the American Psychological Association or one of its allied publishers.
r2 Tt) were compared. In these cases, T, and F2 were each sepa-
rate treatments (e.g., cognitive therapy, behavior therapy). The specific
conditions that constituted Tt and 7"2 within a given study or across
studies were not of interest. There was no interest, for the present
purposes, in drawing conclusions about the relative merits of alternative
psychotherapy techniques. Across all studies, the identity of the treat-
ment conditions designated as T, or 7"2 varied. The goal was to evaluate
the power of comparative studies to detect differences between alterna-
tive treatments. Consequently, in computing ESs for treatment versus
treatment comparisons, the absolute value of the ES (| m, - m21 /.v) was
calculated. Thus, the mean ES for a study with two treatments would
be the mean of the absolute value differences between each of these
treatments for each of the outcome measures. Similarly, for compara-
tive purposes, the absolute magnitude of the differences was used to
compute ESs between treatment and control conditions.7 Effect sizes
were calculated from the means and standard deviations reported in the
studies. When this information was unavailable, ESs were calculated
from other reported statistics. Techniques for estimating ESs from other
data sources are described elsewhere (Smith et al, 1980).
Results
Descriptive Characteristics
From the nine journals, 85 psychotherapy outcome studies
were used to compute effect sizes and to estimate power. Two of
the journals (American Journal of Psychiatry, Psychotherapy:
Theory, Research and Practice) yielded no studies for the pres-
ent analysis. Effect sizes for the comparisons of interest were
computed for each of the measures within a study. A total of
2,501 ESs were computed: 1,752 at posttreatment and 749 at
follow-up. However, each study could yield only one ES for each
of the comparisons of interest (e.g., treatment vs. no treatment)
at posttreatment and follow-up. Thus, the number of ESs varied
depending on the groups and assessment periods included in
the study.
Table 1 presents the number of studies available from each
journal, the sample sizes for the studies, the group composition,
the types and time points of comparisons, and the study length.
The journals are presented separately in this initial table for
descriptive purposes. Individual journals were not of interest
well-grounded in theory, research, or practice (Kazdin & Wilcoxon,
1976; Prideau, Murdock, & Brody, 1983). Thus, no-treatment and ac-
tive control conditions reflect different substantive issues.
141
and, hence, are combined in subsequent analyses. The table de-
tails information relevant to the evaluation of power but also
portrays several basic characteristics of psychotherapy outcome
studies.
The mean sample size (N) for all studies was 53 subjects
(SD = 39.2); the mean number of groups was 3 (SD = 1.2). All
85 studies included at least one treatment group, given that this
was a selection criterion for inclusion; 75 (88.2%) of the studies
included two or more treatment conditions; 40 (47.1%) of the
studies included a no-treatment or waiting-list control condi-
tion. Only 8 (9.4%) studies included an active control (e.g., at-
tention-placebo) condition. In terms of outcome evaluation all
studies included posttreatment and 67.1% of the studies in-
cluded a follow-up assessment.8 The mean duration of the lon-
gest follow-up in the studies was 8.0 months (SD = 7.5).
Sample Sizes and Estimates of Effect Size
Table 2 summarizes the sample sizes observed from the stud-
ies surveyed. As noted in this table and in the others that follow,
medians, 25th, and 75th percentiles are presented. The median
(50th percentile) is presented as a measure of central tendency
uninfluenced by extreme data points (outliers) that might affect
both the mean and the standard deviation. The 25th and 75th
percentiles are presented as the interval that captures 50% of
the studies about that median. The discussion focuses on means
to make the present comments parallel with other evaluations
5
In computing effect sizes, not all researchers have used the pooled
standard deviation as the estimate of variance. The pooled estimate was
used in the present analyses because it is readily estimated from studies
in which t and F tests are reported but standard deviations for individual
groups are omitted. In addition, the pooled estimate may be a less-bi-
ased estimate than the standard deviation of the control group (Hedges
AOlkin, 1985).
6
Within a given study, more than one outcome measure was likely to
be included. In some previous meta-analyses, separate outcome mea-
sures have been used as separate ESs for the data analyses. The issues
raised with this procedure include the undue weight given to studies
with a large number of outcome measures and the nonindependence of
observations (ESs) for any data analyses. An alternative strategy is to
calculate a mean ES for a given comparison (e.g., treatment vs. non-
treatment) by averaging the ESs from the individual outcome measures.
Thus, for a given study with a treatment and no-treatment control con-
dition, one ES would be generated for that comparison based on the
mean of ESs for all of the outcome measures. Alternative strategies for
handling multiple outcomes within a study have different limitations. A
major limitation of the present method is that it assumes that all out-
come measures in a study should be weighted equally (i.e., are equally
important). There remains no consensus on how to prioritize outcome
measures to resolve this concern (see Brown, 1987).
7
In meta-analyses, comparisons of treatment versus control groups
do not use absolute effect size. In such analyses, the interest is in iden-
tifying positive or negative effect sizes, that is, in whether the treatment
group was better or worse than the control group. Effect size data based
on directional differences of treatment versus control conditions for the
present survey are available from the authors.
' Thisfigureis an underestimate of follow-up evaluations of psycho-
therapy studies because investigators occasionally publish follow-up
data in publications separate from the original article hi which post-
treatment data were presented.
 142 ALAN E. KAZDIN AND DEBRA BASS

Table 1
Psychotherapy Outcome Studies: Sources and Study Characteristics

Variable All journals JCCP JCP BT BRT AGP BJP BJCP

No. of studies 85 38 10 16 16 2 2 1
Sample size
M 53.60 55.63 81.10 38.44 45.31 71.00 42.50 64.00
SD 39.21 31.97 79.45 18.00 29.84 42.43 31.82 0
Range 12-298 12-146 31-298 14-66 17-131 41-101 20-65
No. of groups —
M 3.14 3.05 3.70 2.56 3.31 4.00 2.00 8.00
SD 1.20 .93 1.06 .63 1.35 2.83 0 0
Range 2-8 2-6 2-6 2-4 2-6 2-6
No. of studies with — —
Two or more treatments 75' 34 9 13 15 2 1 1
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

No-treatment control 40" 19 7 9 4 0 1 0

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Active control 8° 5 0 0 3 0 0 0
Follow-up assessment 57" 27 4 11 12 2 1 0
Follow-up length (months)
M 8.03 7.14 5.00 8.73 11.75 3.50 1.00 0
SD 7.55 5.11 4.97 6.23 12.48 3.54 0 0
Range 1-46 1-18 1-12 1-24 2-46 1-6 0 0

Nate. JCCP = Journal oj'ConsultingandClinical Psychology; JCP = Journal of Counseling Psychology; BT = Behavior Therapy; BRT= Behaviour
Research and Therapy; ASP = Archives of General Psychiatry; BJP = British Journal of Psychiatry; BJCP = British Journal of Clinical Psychology.
Dash denotes that the high and low numbers for the range were the same or that only one entry was available.
•88.24%. b 47.06%. "9.41%. "67.06%.

in which effect sizes of psychotherapy research have been exam- Table 3 presents estimated effect sizes for the comparisons of
ined. interest. As a potential guideline for the interpretation of the
As is evident in the table, the sample sizes were quite similar data, it is useful to bear in mind Cohen's (1977) classification
across treatment and control groups. Consideration of the data of small, medium, and large ESs at .20, .SO, and .SO.9 When two
pooled across groups revealed that the mean group sizes at post- or more treatments were compared with each other, the mean
treatment and follow-up were 16.1 and 15.3, respectively. Ex- absolute ESs at posttreatment and follow-up were .50 and .47,
amination of the 75th percentile conveyed that three fourths of respectively. These fall within the range of medium ESs. The
the studies included fewer than 20 subjects per group. mean absolute ES across all outcome studies comparing treat-
ment versus no treatment was .85 at posttreatment and .89 at
follow-up. These ESs fall within the range of large ESs. Few
studies (8 at posttreatment, 5 at follow-up) were available that
Table 2 compared treatment versus active control conditions. The
Sample Sizes (n) Obtained From Surveyed Studies mean ES for this comparison was .38 at posttreatment and .32
Group Posttreatmenl Follow-up at follow-up, both between small and medium ESs.10
The range of ESs obtained for the comparison of interest are
Treatment illustrated in Figure 1. This figure conveys the median ES and
M 16.01(11.12) 14.56 (7.83) the 25th and 75th percentiles, a range that includes 50% of the
Man 12.00 13.00
10.00-19.00
studies for the comparison of interest. The data convey clearly
25th-75th percentiles 10.00-17.00
Range 6-107 3-74 that the comparisons of alternative treatments span the small-
No-treatment control to-medium range. In contrast, the ESs for treatment versus no
M 16.49(16.52) 19.91 (21.70) treatment are in the medium-to-large range.
Mdn 12.00 13.00
25th-75th percentiles 10.00-18.00 9.00-20.00
Range 5-114 7-84 Estimated Power
Active control
If ESs fall within the ranges noted previously, one can esti-
M 16.89(9.74) 20.17(10.28)
Mdn 12.00 16.50 mate the extent to which studies, as currently designed, are
25th-75th percentiles 12.00-24.50 11.50-32.50
Range 5-34 10-34 ' The magnitudes for small, medium, and large effect sizes have been
All revised from those specified originally by Cohen (1962), which were .25,
M 16.12(12.09) 15.25(9.81)
.50, and 1.00. The effect sizes of .20, .50, and .80 reflect the magnitudes
Mdn 12.00 13.50
10.00-19.00 10.00-17.00 in more recent references (Cohen, 1977).
25th-75th percentiles 10
Range 5-114 3-84 Because of the small number of studies, the data for this compari-
son will not be discussed further. However estimates of ES, power, and
Note. Standard deviations are expressed in parentheses. related information are presented in the tables.
 COMPARATIVE OUTCOME STUDIES AND STATISTICAL POWER 143

Table 3 Table 4
Estimated Effect Sizes for Comparisons of Interest Estimated Power to Detect Differences Between Alternative
Treatments and Treatment Versus Control Conditions
Comparison Posttreatment Follow-up
Comparison Posttreatment Follow-up
Treatment vs treatment
M .50 (.31) .47 (.32) Treatment vs. treatment
Mdn .47 .40 Mdn .74 .63
25th-75th percentiles .26-.66 .27-.64 25-75 percentiles .27-97 .38-87
Range .04-1.52 .02-1.76 Range .05-995 .03-995
No. of articles 75 42 No. of articles 75 42
Treatment vs no treatment Treatment vs no treatment
M .85 (.47) .89 (.46) Mdn .995 .995
Mdn .78 .87 25th-75th percentiles .89-995 .92-995
25th-75th percentiles .54-1.05 .52-1.15 Range .37-995 .73-995
Range .30-2.67 .26-1.85 No. of articles 40 10
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

No. of articles 40 10 Treatment vs active control

This document is copyrighted by the American Psychological Association or one of its allied publishers.

Treatment vs active control Mdn .51 .53

M .38 (.17) .32 (.24) 25th-75th percentiles .25-79 .19-84
Mdn .44 .27 Range .14-87 .03-97
25th-75th percentiles .19-53 .13-53 No. of articles 8 5
Range .15-.59 .02-66
No. of articles 8 5

Note. Standard deviations are expressed in parentheses.

likely to be powerful enough to detect significant differences be-
tween alternative types of comparisons. Based on sample sizes
and estimated ESs, power can be estimated from the tables pro-
vided by Cohen (1977). For the purpose of identifying power,
alpha was set at .OS for two-tailed tests of significance. Also, the
mean group size (n) was used in cases in which groups were
unequal in size or individual group size could not be deter-
mined (Cohen, 1977). The use of an arithmetic mean for the
groups facilitates the use of power tables. Assuming equal group
sizes also maximizes the power for the sample size of a study
1.2
1
0.8
Effect
Size 0.6
0.4
0.2
and, consequently, provides an optimal level of power for the
study given the fixed sample size. Power functions are nonlin-
ear. Because the arithmetic average (mean) does not weight the
different levels appropriately, median levels of power are dis-
cussed. The median estimated levels of power for the different
comparisons of interest, accompanied by 25th and 75th percen-
tiles, are presented in Table 4.
The primary question concerned the power to detect differ-
ences when two or more alternative treatments are compared.
As noted in Table 4, the power to detect differences for this type
of comparison yielded a median of .74 at posttreatment and of
.63 at follow-up. Thus, the median chance of a study to detect
a difference at posttreatment and at follow-up is about 7 in 10
and 3 in 5, respectively. For individual studies comparing two
or more treatments, 45.3% (34 of 75) and 28.6% (12 of 42),
respectively, met the criterion of power 2; .80 needed to detect
differences between treatment groups at posttreatment and fol-
low-up. The majority (54.7% and 71.4%) of studies that com-
pared alternative treatments did not meet the recommended
level of power.
The table indicates relatively strong power for the test of treat-
ment versus no treatment. Median power was estimated at .995
at posttreatment and follow-up for this comparison. Using .80
as a criterion for adequate power, outcome studies generally are
sufficiently powerful to detect differences between treatment
versus no treatment. As for the individual outcome studies,
82.5% (33 of 40) and 90.0% (9 of 10) met the criterion of power
& .80 for the comparison of treatment versus no treatment
at posttreatment and follow-up, respectively.

Sample Sizes for Psychotherapy Research

T (P) T (FU) NT (P) NT (FU)
Treatment Compared To...
Figure 1. Effect sizes (ESs) obtained from comparisons of treatment
(T) with another treatment or with no treatment (NT) separately for
posttreatment (P) and follow-up (FU). The column for each compari-
son reflects toe range of ESs from the 25th to the 75th percentiles; the
horizontal line within each column reflects the median ES.
Based on data obtained in the present survey, more specific
comments can be made about requisite samples sizes when al-
ternative forms of psychotherapy are compared. Assume for a
moment that an investigator wishes to compare two treatments
and to retain generally acceptable levels of alpha, beta, and
power (1-beta); in this case, alpha = .05 for a two-tailed test and
 144 ALAN E. KAZDIN AND DEBRA BASS
beta = .20 so that power = .80. For the purpose of illustration,
sample size and ESs will be considered for the different compar-
isons in relation to the posttreatment evaluation of outcome.
In the present survey, the median ES for comparing two treat-
ments was .47 at posttreatment, with a range from .26 to .66
representing the 25th and 75th percentiles. Figure 2 conveys
the corresponding sample sizes needed for these ESs. A sample
size of 71 per group would be needed to retain power at the
desired level for the median ES, with a corresponding range
from 232 to 36 subjects for the 25th and 75th percentiles.11 Ex-
amination of the figure shows the number of subjects per group
actually used. The median number of subjects for studies com-
paring alternative treatments was 12.0, with a range from 10 to
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
19 for the 25th and 75th percentiles. These actual sample sizes
This document is copyrighted by the American Psychological Association or one of its allied publishers.
are far below the number needed to detect treatment differ-
ences.
If the investigator wishes to compare treatment versus no
treatment, the median ES is .78 at posttreatment. This ES is
bounded by ESs of .54 and 1.05, which reflect the 25th and 75th
percentiles and represent 50% of the studies surveyed. Figure 2
plots the corresponding sample sizes for these ESs. A sample
size of 27 per group would be needed for the desired power with
the median ES for this comparison; the corresponding range of
54 to 14 subjects per group would be required for the 25th and
75th percentiles. These numbers are larger but closer to the ac-
tual sample sizes used. Figure 2 shows the actual sample sizes,
with a median of 12 at posttreatment bounded by a range from
10 to 18 for the respective percentiles.
Discussion
Psychotherapy outcome research was examined in nine jour-
nals over a 3-year period (1984-86) to assess the extent to which
studies are sufficiently powerful to detect differences between
alternative treatment and control conditions. Effect sizes were
estimated at posttreatment and follow-up, and these were used
along with sample size data to evaluate power. The results can
be discussed with reference to Cohen's (1977) criteria for small
(.20), medium (.50), and large (.80) effect sizes. The present re-
sults indicate that comparisons of alternative treatments yield
effect sizes close to the medium level (mean ESs = .50 at post-
treatment and .47 at follow-up) and that comparisons of treat-
ment versus no treatment tend to yield relatively large effect
sizes (mean ESs = .85 at posttreatment and .89 at follow-up).
The question of interest is: To what extent are psychotherapy
outcome studies sufficiently powerful to detect differences given
these effect sizes? Using estimated effect sizes and sample sizes
from the studies themselves and adopting an alpha of .05 (for
two-tailed tests), we obtained power estimates. In evaluating the
findings, we considered power 2 .80 a criterion of adequate sen-
sitivity of a test.
Power for comparisons of alternative treatments was below
the recommended level (medians = .74 for posttreatment and
.63 for follow-up). Indeed, for studies comparing two or more
active treatments, fewer than half at posttreatment and fewer
than one third at follow-up (45.3% and 28.6% of the studies)
had power at or above the recommended level. The power to
detect differences for comparisons of treatment versus no treat-
ment was quite adequate (medians = .995 for both posttreat-
ment and follow-up). Also, the majority of studies (82.5% and
90.0%) comparing treatment with a no-treatment control met
or surpassed the power of .80 at posttreatment and follow-up.
A number of implications can be drawn from the present sur-
vey. First, power in psychotherapy outcome studies research is
generally low for detecting small and medium effects. The need
for improved power remains, and several recommendations
made over 20 years ago (Cohen, 1965) continue to be timely.
The concern with weak power in relation to psychotherapy re-
search echoes points voiced in relation to research in clinical,
social, educational, and applied psychology more generally
(e.g., Brewer, 1972; Chase & Chase, 1976; Cohen, 1962; Rossi
et al., 1984). Psychological research, of course, is not alone on
this score. The absence of differences in major clinical trials of
alternative interventions in medicine and public health may, in
many instances, be attributed to insufficient power (see Frei-
manetal., 1978).
A second and more focused implication of the present results
pertains to the special issues regarding comparative psychother-
apy outcome research (Heimberg & Becker, 1984; Kazdin,
1986). In evaluations of alternative psychotherapy techniques,
both in individual studies and in literature reviews, a frequent
interpretation is that there are few or no outcome differences
between alternative techniques. Perhaps most treatment tech-
niques are not very different in the effects they produce, and
the common processes of therapy advanced to explain the null-
hypothesis findings are well-based. However, in light of the pres-
ent findings, it is appropriate and parsimonious to raise another
prospect. Possibly, studies that compare alternative treatments
are not sufficiently powerful to detect differences between treat-
ments unless the effect sizes are large.
Given the complexities of clinical problems and psychother-
apy and the limitations and variability of outcome assessment,
large effects, even if they are evident in the population, might be
difficult to obtain in a given investigation. The present survey
suggests that effect sizes for comparisons of alternative treat-
ments are likely to be in the small-to-medium range. Given the
usual sample sizes that are used, the majority of studies may
not be sufficiently powerful to detect such differences.
There are several limitations of the present survey. To begin
with, the methods of estimating and evaluating effect sizes in
psychotherapy research are not entirely free from controversy.
Significant issues such as the appropriate weighting of alterna-
1
' The requisite sample (N) or group (n) sizes for a given alpha, power,
and effect size of various increments can be obtained from various pub-
lished tables (see Cohen, 1977;Hinkle&Oliver, 1983;Kraemer&Thie-
mann, 1987). For the present purposes, the requisite group sizes, when
we assumed an equal number per group, were obtained by direct calcu-
lation (see Lachin, 1981; Snedecor & Cochran, 1 980). For a two-tailed
test of means from two independent samples of equal group sizes,
For a two-tailed test with alpha = .05 and beta = .20, this translates to
(1.96 + .842)2(2)
" -- JSP
 COMPARATIVE OUTCOME STUDIES AND STATISTICAL POWER
Required ns for power = .80
250-
200-
Group 150-
Size (n)
100- -
50-
e a
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
T (P) T (FU) NT (P)NT (FU)
Treatment Compared To-
Obtained ns from Survey
25-
20-
Group 15
Size (n)
10-
5-
T (P) T (FU) NT (P) NT (FU)
Treatment Compared To...
Figure 2. Sample sizes required for comparisons of treatment (T) with
another treatment or with no treatment (NT) separately for posttreat-
ment (P) and follow-up (FU). (The sample sizes required are based on
alpha = .05 for a two-tailed test, with power set at. 80 and an assumption
of equal ns per group. The column for each comparison reflects the
range of sample sizes from the 25th and 75th percentiles; the horizontal
line within each column corresponds to the median sample size needed.
The upper figure represents the sample sizes required for power - .80
given the effect sizes that are obtained for the different types of compari-
sons; the lower figure conveys the actual sample sizes used in studies of
the present survey.)
live outcome measures, the appropriate standard deviation
unit, the extent to which the sample effect sizes within and
among different studies can be pooled to estimate population
effect sizes, the nonindependence of effect sizes computed for a
given study for the types of comparisons of interest, and the
exclusion of unpublished studies, (among others) have been
carefully discussed, but not entirely resolved, in other second-
ary analyses. The present method of computing effect sizes fol-
lowed several practices used in prior meta-analyses and is sub-
ject to similar issues and concerns. The present survey suffers
from greater interpretive obstacles by combining studies, effect
sizes, and power estimates across broad classes of treatment in
which the identity of the type of treatment, patient, measure,
145
and other study characteristics were ignored. The impact of
these and other characteristics of therapy on effect sizes have
been studied in large-scale meta-analyses (e.g., Shapiro & Sha-
piro, 1982; Smith etal., 1980).
The purpose of the present survey was to examine broad
classes of comparisons to estimate power. The estimates, when
viewed in the range of 25th and 75th percentiles, provide an
interval that suggests reasonable consistency, even though the
distinctions between different techniques and studies were not
made. One might want to examine whether the power of studies
of particular treatments is weaker than that of others and to
make technique distinctions. However, this was beyond the
present goal.
A feature directly related to different characteristics of the
therapy studies was neglected in the present survey and may
have important implications for power. The present analysis fo-
cused on the power to detect differences between treatments.
The analyses examined main effects of a single variable (treat-
ment technique). This focus is tantamount to asking the ques-
tion, does psychotherapy work? The question has long been re-
jected as much too general to warrant serious attention (e.g.,
Edwards & Cronbach, 1952). The global question has been re-
placed by a more specific focus aimed at identifying which type
of treatment works best for which type of client, as adminis-
tered by which type of therapist under which circumstances
(Kiesler, 1966; Paul, 1967). This question focuses on interac-
tions (e.g., Treatment X Patient x Therapist X Setting) rather
than simply on the efficacy of treatment or the relative effec-
tiveness of different treatments. Yet, testing interactions in fac-
torial studies is likely to divide samples into smaller groups than
those used to test main effects of treatment. If power is weak for
testing the main effect of treatment techniques, a fortiori, it is
likely to be weak for testing the interactions. This concern has
been voiced in relation to other areas of psychological research
in which the inadequacy of power to test statistical interactions
is common (Chase & Tucker, 1975; Cohen, 1965).
Another limitation of the present evaluation is the exclusive
focus on power and the limited discussion even within this do-
main. The focus has suggested that power needs to be increased
in psychotherapy outcome studies given the magnitude of effect
sizes usually reported. The discussion has emphasized that
power is a function of alpha, sample size, and effect size. Sample
size seems to be the only variable for the investigator to manipu-
late and improve given that the adherence to alpha (at .05 or
.01) has become a strongly entrenched convention (Cohen,
1977) and that effect size, at first blush, seems merely to reflect
the state of affairs evident in nature, that is, an estimate of true
population differences.
Actually, power entails much more than the factors empha-
sized in this article. Power is a function of effect size, which is
influenced in manifold ways by the care and consistency with
which an investigation is conducted. Stated generally, effect size
can be increased or rather optimized in an investigation by re-
ducing error variance. Methodological features such as select-
ing homogeneous sets of patients, ensuring the integrity of
treatment, standardizing the assessment conditions, carefully
choosing outcome measures, and similar practices increase the
power of an investigation by reducing variability in its execu-
tion. Although the present focus was limited to consideration
 146 ALAN E. KAZDIN AND DEBRA BASS
of power, power itself can be augmented by the careful execu-
tion of the study.
The present survey illustrates the utility of power analyses in
evaluating the results of previously conducted research. Such
analyses can estimate the likelihood that research can detect
differences given sample and effect sizes. Another and more im-
portant use of power analysis is the planning of a study to ensure
that sample sizes can detect an effect of a given magnitude (Co-
hen, 1977; Kraemer & Thiemann, 1987; Meinert, 1986). Psy-
chotherapy outcome research can profit greatly from this use of
power analyses. The usual impediment for incorporating power
in the design of research is hesitancy in estimating effect size.
However, data from meta-analyses of psychotherapy (Brown,
1987) as well as descriptive analyses such as those found in the
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
present survey provide estimates of the approximate effect sizes
for alternative types of comparisons. As guidelines, they may be
used to identify the sample sizes required for adequate power.
It is possible, of course, that large sample sizes needed to pro-
vide adequate comparisons of treatment are not available for
particular clinical problems or settings. These limitations do
not necessarily warrant undertaking a comparison that has
minimal chances of detecting differences. The levels of confi-
dence adopted to protect against Type I error (alpha) and Type
II error (beta) and the resulting power (1 -beta) may need to be
reconsidered. Depending on the specific question, the compari-
son of interest, and the cost and consequences of no differences
between treatments, one might wish to vary alpha or beta
(Meinert, 1986). The issue requires consideration in advance of
the execution of the study. Given the sample sizes usually used
in comparative outcome studies and the likely effect sizes usu-
ally reported, differences are not likely to be detected with ac-
ceptable levels of power.
The neglect of power can have major implications for inter-
preting research, perhaps especially so in the evaluation of psy-
chotherapy. As noted previously, psychotherapy research is an
area in which the absence of differences is often taken to be
quite significant from conceptual and clinical perspectives. It
may well be that treatments are similar in the outcomes they
produce and that findings of no difference reflect the actual
state of the population. However, there remains a plausible al-
ternative hypothesis. The present evaluation suggests that the
power of studies comparing alternative treatments is relatively
weak. The results are not intended to fuel further the search for
the superiority of one treatment over another. More sophisti-
cated questions involving interactions of treatment with vari-
ables or other conditions of interest need to be studied. How-
ever, the points raised in relation to power in the present survey
may need to be considered even further. Unless sample sizes are
substantially increased from current levels, finer grained analy-
ses of treatments are likely to be associated with even lower
power as the division of the sample for the comparisons of inter-
est results in smaller units for analyses.
The present survey evaluated alternative classes of compari-
sons that are made in comparative outcome studies. The im-
plications of weak power for more specific substantive questions
(e.g., Treatment X Patient X Therapist X Setting interactions)
were not addressed. Even so, the general conclusion that might
be drawn is worth underscoring. The absence of differences be-
tween treatments in comparative outcome studies may not be
interpreted unambiguously without improved power. In the
context of psychotherapy research, as opposed to government
and politics, it may be the absence rather than the presence of
power that corrupts.
References
Brewer, J. K. (1972). On the power of statistical tests in the American
Educational Research Journal. American Educational Research
Journal, 9,391-401.
Brown, J. (1987). A review of meta-analyses conducted on psychother-
apy outcome research. Clinical Psychology Review, 7.1-23.
Casey, R. J., & Herman, J. S. (1985). The outcome of psychotherapy
with children. Psychological Bulletin, 98,388-400.
Chase, L. J., & Chase, R. B. (1976). A statistical power analysis of ap-
plied psychological research. Journal of Applied Psychology 61,234-
237.
Chase, L. J., & Tucker, R. K. (1975). A power-analytic examination of
contemporary communication research. Speech Monographs, 42,
29-41.
Cohen, J. (1962). The statistical power of abnormal-social psychological
research: A review. Journal of Abnormal and Social Psychology, 65,
145-153.
Cohen, J. (1965). Some statistical issues in psychological research. In
B. B. Wolman (Ed.), Handbook of'clinicalpsychology (pp. 95-121).
New %rk: McGraw-Hill.
Cohen, 1. (1973). Statistical power analysis and research results. Ameri-
can Education Research Journal, 10, 225-230.
Cohen, J. (1977). Statistical power analysis for the behavioral sciences
(2nd ed.). New York: Academic Press.
Cross, D. G., Sheehan, P. W., & Khan, J. A. (1982). Short- and long-
term follow-up of clients receiving insight-oriented therapy and be-
havior therapy. Journal of Consulting and Clinical Psychology, 50,
103-112.
DiLoreto, A. O. (1971). Comparative psychotherapy: An experimental
analysis. Chicago: Aldine-Atherton.
Edwards, A. L., & Cronbach, L. J. (1952). Experimental design for re-
search in psychotherapy. Journal of Clinical Psychology, 8,51 -59.
Fagley, N. S. (1985). Applied statistical power analysis and the interpre-
tation of nonsignificant results by research consumers. Journal of
Counseling Psychology, 32, 391-396.
Feldman, R. A., Caplinger, T. E., & Wodarski, J. S. (1983). The St. Louis
conundrum: The effective treatment of antisocial youths. Englewood
Clifls, NJ: Prentice-Hall.
Heischman, M. J. (1981). A replication of Patterson's "Intervention
for boys with conduct problems." Journal of Consulting and Clinical
Psychology, 49,342-351.
Fonnan, S. G. (1980). A comparison of cognitive training and response
cost procedures of modifying aggressive behavior of elementary
school children. Behavior Therapy, 11, 594-600.
Frank, J. D. (1982). Therapeutic components shared by all psychothera-
pies. In J. H. Harvey & M. M. Parks (Eds.), Psychotherapy research
and behavior change (Vol. 1, pp. 5-37). Washington, DC: American
Psychological Association.
Freiman, J. A., Chalmers, T. C, Smith, H., & Kuebter, R. R. (1978).
The importance of beta, the Type II error, and sample size in the
design and interpretation of the randomized control trial. New En-
gland Journal of Medicine, 299, 690-694.
Friedman, L. M., Furberg, C. D., & DeMets, D. L. (1985). Fundamen-
tals of clinical trials (2nd ed.). Littleton, MA: PSG Publications.
Garfield, S. L. (1980). Psychotherapy: An eclectic approach. New Tforfc
Wiley.
Hedges, L. V, & Olkin, I. (1985). Statistical methods for meta-analysis.
Orlando, FL: Academic Press.
 COMPARATIVE OUTCOME STUDIES AND STATISTICAL POWER
Heimberg, R. G., & Becker, R. E. (1984). Comparative outcome re-
search. In M. Hersen, L. Michelson, & A. S. Bellack (Eds.), Issues in
psychotherapy research (pp. 251-283). New York: Plenum Press.
Hinkle, D. E., & Oliver, J. D. (1983). How large should the sample be?
A question with no simple answer? Or. . . .EducationalandPsycho-
logical Measurement, 43,1051-1060.
Kazdin, A. E. (1980). Research design in clinical psychology. New York:
Harper & Row.
Kazdin, A. E. (1986). Comparative outcome studies of psychotherapy:
Methodological issues and strategies. Journal of Consulting and Clini-
cal Psychology, 54, 95-105.
Kazdin, A. E. (1988). Child psychotherapy: Developing and identifying
effective treatments. New York: Pergamon Press.
Kazdin, A. E., & Wilcoxon, L. A. (1976). Systematic desensitization
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
and nonspecific treatment effects: A methodological evaluation. Psy-
This document is copyrighted by the American Psychological Association or one of its allied publishers.
chological Bulletin. 83, 729-758.
Kazdin, A. E., & Wilson, G. T. (1978). Evaluation of behavior therapy:
Issues, evidence, and research strategies. Cambridge, MA: Ballinger.
Kiesler, D. J. (1966). Some myths of psychotherapy research and the
search for a paradigm. Psychological Bulletin, 65,110-136.
Klein, M. H., Dittmann, A. T., Parloff, M. B., & Gill, M. M. (1969).
Behavior therapy: Observations and reflections. Journal of Consulting
and Clinical Psychology, 33,259-266.
Kraemer, H. C. (1981). Coping strategies in psychiatric clinical re-
search. Journal of Consulting and Clinical Psychology, 49,309-319.
Kraemer, H. C, & Thiemann, S. (1987). How many subjects?Statistical
power analysis in research. Newbury Park, CA: Sage.
Lachin, J. M. (1981). Introduction to sample size determination and
power analysis for clinical trials. Controlled Clinical Trials, 2, 93-
113.
Lambert, M. J., Shapiro, D. A., & Bergin, A. E. (1986). The effective-
ness of psychotherapy. In S. L. Garfield & A. E. Bergin (Eds.), Hand-
book of psychotherapy and behavior change (3rd ed., pp. 157-211).
New York: Wiley.
Liberman, R. P., & Eckman, T. (1981). Behavior therapy vs. insight-
oriented therapy for repeated suicide attemptors. Archives of General
Psychiatry, 38, 1126-1130.
Luborsky, L., Singer, B., & Luborsky, L. (1975). Comparative studies of
psychotherapies: Is it true that "everyone has won and all must have
prizes?" Archives of General Psychiatry, 32,995-1008.
Meinert, C. L. (1986). Clinical trials: Design, conduct, and analysis.
New York Oxford University Press.
Patterson, G. R. (1974). Interventions for boys with conduct problems:
147
Multiple settings, treatments, and criteria. Journal of Consulting and
Clinical Psychology, 42,471-481.
Paul, G. L. (1967). Outcome research in psychotherapy. Journal of Con-
sulting Psychology, 31,109-118.
Prioleau, L., Murdock, M., & Brody, N. (1983). An analysis of psycho-
therapy versus placebo studies. The Behavioral and Brain Sciences,
8, 275-285.
Rachman, S. J., & Wilson, G. T. (1980). The effects of psychological
therapy (2nd ed.). Oxford, England: Pergamon Press.
Rosenthal, R. (1979). The "file drawer problem" and tolerance for null
results. Psychological Bulletin, 86,638-641.
Rossi, J. S., Rossi, S. R., & Cottrill, S. D. (1984, April). Statistical power
of research in social and abnormal psychology: What have we gained
in 20 years? Paper presented at the meeting of the Eastern Psychologi-
cal Association, Baltimore, MD.
Rush, A. J., Beck, A. X, Kovacs, M., & Hollon, S. (1977). Comparative
efficacy of cognitive therapy and pharmacotherapy in the treatment
of depressed outpatients. Cognitive Therapy and Research, 1,17-38.
Shapiro, D. A., & Shapiro, D. (1982). Meta-analysis of comparative
therapy outcome studies: A replication and refinement. Psychological
Bulletin, 92, 581-604.
Sloane, R. B., Staples, E R., Cristol, A. H., Yorkston, N. J., & Whipple,
K. (1975). Psychotherapy versus behavior therapy. Cambridge, MA:
Harvard University Press.
Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psycho-
therapy. Baltimore, MD: Johns Hopkins University Press.
Snedecor, G. W., & Cochran, W. G. (1980). Statistical methods (7th
ed.). Ames: Iowa State University Press.
Stiles, W. B., Shapiro, D. A., & Elliott, R. (1986). "Are all psychothera-
pies equivalent?" American Psychologist, 41,165-180.
Viale-Val, G., Rosenthal, R. H., Curtiss, G., & Marohn, R. C. (1984).
Dropout from adolescent psychotherapy: A preliminary study. Jour-
nal of the American Academy of Child Psychiatry, 23,562-568.
Walrond-Skinner, S. (1986). Dictionary of psychotherapy London:
Routledge & Kegan Paul.
Waterhouse, G. J., & Strupp, H. H. (1984). The patient-therapist rela-
tionship: Research from the psychodynamic perspective. Clinical
Psychology Review, 4, 77-92.
Yu, P., Harris, G. E., Solovitz, B. L., & Franklin, J. L. (1986). A social
problem-solving intervention for children at high risk for later psy-
chopathology. Journal of Clinical Child Psychology, 15, 30-40.
Received February 1,1988
Revision received May 10,1988
Accepted May 23,1988 •