Clinical Psychology and Psychotherapy, Vol.

3 (4), 249-258 (1996)

The Reliability and Validity of the

Outcome Questionnaire
Michael J. Lambert^
Gary M. Burlingame
Val Umphress
Nathan 6. Hansen
David A, Vermeersch
Glenn C. Clouse
Stephen C. Yanchar
Brigham Young University, Prcwo, USA

With the rise in efforts to evaluate the quality of mental health care and
its outcomes, the measurement of change has become an important
topic. This paper tracks the creation of a new instrument designed to
assess psychotherapy outcome. The Outcome Questionnaire (OQ) was
designed to include items relevant to three domains central to mental
health: subjective discomfort, interpersonal relations, and social role
performance. This study describes the theoretical development and
psychometric properties of the OQ. Psychometric properties were
assessed using clinical, community, and undergraduate samples. The
OQ appears to have high reliability and evidence to suggest good
concurrent and construct validity of the total score. The data presented
show that it distinguishes patient from non-patient samples, is sensitive
to change, and correlates with other measures of patient distress.

INTRODUCTION a reliable means of defining treatment goal criteria

Current trends in health care reform show the
managed health care system to be an increasingly
popular alternative to traditional health care
delivery. As some kind of managed care or national
health coverage eventually becomes a reality, a
growing body of literature indicates that health
policy planners are concerned with the relationship
between mental health care cost containment and
the quality of services rendered (Bloom, 1987;
Richardson and Austad, 1991; Mirin and Namerow,
1991; Brokowski, 1991). In the midst of debate
surroxmding a definition of proper mental health
care delivery lies the issue of determining criteria
for, and assessing the effectiveness of, therapeutic
systems. As indicated elsewhere (Lambert, 1983;
Ahmed and Smith, 1991; Mirin and Namerow, 1991;
Moses-Zirkes, 1993), outcome assessment provides
'Addressee for correspondence: Professor Michael J. Lambert,
Department of Clinical Psychology, Brigham Young Univer-
sity, 284 Taylor Building, PO Box 28626, Provo, Utah 84602-
8626, USA.
and monitoring efficacy of treatments. But in
contrast to traditional scientific studies of the effects
of psychotherapy, much current research is being
undertaken to assess the quality of services and
track patient progress.
Formal outcome research is a manifold enterprise,
ideally incorporating nim\erous measures of
patients subjective discomfort, clinician ratings,
trained observer ratings, physiological indices, and
environmental information (Lambert et al. 1992).
While it is commonly accepted that such a multi-
dimensional approach offers greatly improved
means of charting patient progress in terms of both
scientific rigour and comprehensive assessment,
practical considerations encountered in routine
clinical practice limit a clinical researcher's ability
to conduct comprehensive assessments that inte-
grate the aforementioned criteria. Despite this,
outcome minded third-party payers show a con-
tinued interest in brief measures that tap a variety of
potential outcomes, without the attendant method-
ological complexities of formal outcome research.

CCC 1063-3995/96/040249-10
© 1996 by John WUey & Sons, Ltd.
250
The Outcome (Questionnaire (OQ) is a brief self-
report instrument, designed for repeated measure-
ment of client status through the course of therapy
and at termination. Ease of administration and
scoring, low cost, sensitivity to changes in psycho-
logical distress over short periods of time, and an
ability to tap a wide array of symptomatology and
role ftinctioning may make this instrument useful in
a variety of clinical and counselling applications.
The OQ was formulated in accordance with
Lambert's (1983) organizational scheme for out-
come assessment, suggesting that three dimensions
or content areas (called domains) be evaluated:
intrapersonal (subjective) discomfort or symptom-
atic distress, interpersonal functioning, and social
role performance. Use of this conceptualization
seems justified, in that its breadth affords a
comprehensive review which encompasses inner
life as well as progress in applied situations like
work and school. In addition some items were
included to tap positive states of mental health and
life functioning. It was believed that these items
would not only assess quality of Ufe as perceived by
the client, but increase the range of measurement so
that the test did not suffer from an artificially low
ceiling as is true in tests that only measure the
presence or absence of psychopathology rather than
aspects of healthy functioning.
The primary aim of the present study was to
examine the psychometric properties of the OQ.
Specifically, both internal consistency and test-retest
reliability were assessed with undergraduate and
patient samples. Concurrent and construct validity
were examined using undergraduate, community,
and several clinical samples. After describing the
item selection procedure and the development of the
OQ, the Methods section will address efforts to
acquire normative data and to assess the ability of the
OQ to measure patient change and to distinguish
clinical from non-clinical samples.
METHODS
The selection of all items and domains for the OQ
was based on two methods: (1) rational selection
and (2) empirical data analysis.
Domain: Symptomatic Distress
The choice of items to measure psychological
symptoms was derived from a 1988 National
Institute of Mental Health (NIMH) epidemiological
survey which identified the most prevalent types of
M. /. Lambert et al.
mental disorders across five catchment areas and a
review of the most frequently occurring DSM-III-R
diagnoses in a national managed care corporation
(Human Affairs International). The 1988 epidemio-
logical study of 18571 people across the United
States showed that 15.4% of the population over 18
years of age fulfilled diagnostic criteria for a mental
disorder. Approximately 12% of the total popula-
tion received either an anxiety diagnosis or an
affective disorder classification (Regier et ai, 1988).
This pattern was also present in data provided by
Human Affairs International (HAI). For instance,
examination of the diagnosis of 2145 HAI out-
patients treated in the first quarter of 1992 showed
that nearly 1/3 of all diagnoses involved a form of
affective disorder. An additional 1/3 of the patients
were given some kind of anxiety disorder diag-
noses, including post traumatic stress disorder.
These data suggest that the most prevalent intra-
psychic symptoms are depression and anxiety
based. Therefore, the OQ is heavily loaded with
such items. Following affective and anxiety dis-
orders, substance abuse problems were the next
most frequently occurring diagnoses, and thus
relevant items were included in the symptomatic
distress subscale of the OQ.
Individual items in this domain were selected
according to the following criteria: (1) fit to
DSM-III-R criteria; (2) frequent citation in the litera-
ture as a descriptor or symptom of the selected
psychopathology; and (3) statistical analysis of
individual items (specifically, inter-item correla-
tions). Sample items are: I feel no interest in things;
I tire quickly; I have difficulty concentrating.
Domain: Interpersonal Relations
The inclusion of items for assessing interpersonal
relations was based on research suggesting that
people consider relationships essential to happiness
and life satisfaction (Beiser, 1973; Andrews and
Withey, 1974; Blau, 1977; Veit and Ware, 1983;
Diener, 1984). Other research has shown that the
most common problems addressed in therapy are
interpersonal in nature (Horowitz 1979; Horowitz
et al., 1988). While factors associated with quality of
life vary from study to study, most emphasize the
importance of intimate relationships and their
central contribution to well-being (Diener, 1984).
In addition, interpersonal problems are clearly
related to intrapersonal distress, either as a direct
cause of psychopathology, as a result of psycho-
pathology, or as both a cause and a result (Klerman,
1974; Horowitz et al, 1988). Therefore items dealing
Outcome Questionnaire
with friendships, family, family life, and marriage
were developed. These included attempts to mea-
sure friction, conflict, isolation, inadequacy, and
withdrawal, which are all common complaints
addressed in therapy sessions. These items were
derived from the existent marital and family
relations literature, as well as research on inter-
personal problems most often reported by patients
who are undergoing psychotherapy (Horowitz et al.,
1991). As with the symptomatic distress questions,
item analyses were performed on individual items
aimed at measuring interpersonal functioning to
ensure their suitability. Sample items are: I feel
loved and wanted; I have trouble getting along with
friends and close acquaintances; I am satisfied with
my relationships with others.
Domain: Social Role
Social role performance was assessed by focusing
on patient's level of dissatisfaction, conflict, distress,
and inadequacy in tasks related to their employ-
ment, family roles, and leisure life. Assessment of
social roles suggests that a person's intrapsychic
problems and symptoms can effect their ability to
work, love, and play. This is supported by the
aforementioned quality of life research as well as
the assumption that once people start to develop
sj^nptoms it is common for these symptoms to have
an effect on their personal and work lives (Frisch
et al., 1992). Thus, items were developed that
measure performance in these three areas. Satisfac-
tion in these areas is highly correlated with ratings
of overall life satisfaction (Beiser, 1973; Blau, 1977;
Veit and Ware, 1983; Frisch et al, 1992). Sample
items are: I enjoy my spare time; I feel that I am
doing weU at work/school; I feel angry enough at
work/school to do something I might regret.
Preliminary Data Analysis
After initial development of items for each content
domain preliminary reliability analysis were con-
ducted on the Outcome Questionnaire using 86
undergraduate psychology students. Item analysis
was conducted to determine each item's reliability
by examining its point biserial correlations with the
total test score (Cronbach, 1951). Problematic items
(e.g. low reliability, reverse discrimination) were
either discarded or reworded, and alterations were
made as were needed to improve the item.*
'Data on the original version including reliability data on
each item are available from the first author.
251
Subjects
Subjects included two normal (student and com-
mimity) and three clinical groups. Several student
samples totaling 424 undergraduate students from a
large western university were collected. A commun-
ity sample of 102 subjects was selected at random
from several medium sized cities within a single
western state. The clinical samples were collected
from clinics in Arizona, Florida, Pennsylvania,
Georgia, Maryland, North Carolina, Ohio, Utah
and Washington, DC. Clinical data were pooled for
the large sample of patients who had sought
treatment via their place of employment through
their employee assistance programme (EAP). This
clinical (EAP) sample consisted of 504 patients. AU
the patients in this sample had been given a DSM-
III-R diagnosis by their clinician. Two additional
clinical samples were collected. The community
mental health centre sample (CMH) was composed
of 100 patients seeking treatment at a community
mental health centre serving a rural catchment area.
The outpatient clinic sample (university outpatient
clinic, N = 76) was collected from patients attend-
ing therapy in a public training clinic sponsored by
a large private university. The clinic patients came
from both urban and rural neighbourhoods. Demo-
graphic information on these various samples are
presented in Table 1.
Measures
All samples completed the OQ. The OQ requires
subjects to rate their feeUngs on a five-point Likert
scale ranging from 0 to 4.t
Measures of Symptomatic Distress
The Symptom Checklist 90-Revised (SCL-90-R) is
a 90-item self-report questionnaire that assesses
common psychiatric symptoms in nine areas. The
SCL-90-R has high internal consistency (0.77 to 0.90)
and test-retest reliability (0.78 to 0.90) (Derogatis,
1977). Its validity as a meaure of S5miptomatic
distress has also been supported (Derogatis et al.,
1976). It is frequently used in psychotherapy
outcome research as a measure of improvement
(McRoberts and Lambert, 1993). The typical
tCopies of this test will be made available upx)n request. The
user licensing agreement makes it available to individual
clinicians for a nominal fee of $10, after which it may be
reproduced without additional cost. Larger networks of
providers pay a higher licensing fee, but enjoy the same
privileges. Practically speaking, the OQ costs little more than
the three to five cents it costs to reproduce it.
252 M. J. Lambert et al.

Table 1. Demographic characteristics of the normative samples used to evaluate the Outcome Questionnaire
Sample N Age Gender Marital status Ethnicity H.S.
m SD M F married divorce/ (Zaucasian/other Grad
separated
University students 424 21.13/3.66 38 62 15% 2% 96% 4% 100%
Community sample 102 42.51/10.08 43 57 87% 13% n/a n/a 95%
EAP 504 37.23/10.13 59 41 n/a n/a 82.5% 17.5% n/a
Commtinity mental health 100 39.55/10.04 39 61 n/a n/a n/a n/a n/a
University outpatient clinic 76 29.58/9.69 68 32 48% 35% 97% 3% n/a
Note. In some cases demographic data were not available (denoted by n/a) due to differences in forms used to collect the data at different
localities.

SCL-90-R score used in outcome research is the
General Severity Index (GSI).
The Beck Depression Inventory (BDI) is a 21-item
questionnaire that was developed through clinical
observation of 21 attitudes and symptoms common
to depressed psychiatric clients (Beck et al, 1961,
1988). These symptoms are rated by clients on a
4-point, anchored scale, from 0 to 3. The BDI has
been shown to have good psychometric properties
with coefficient alpha ratings between 0.73 and 0.92
and test-retest coefficients ranging from 0.48 to
0.86, depending on the time intervals for retesting
and sample characteristics (Beck et al, 1988). The
BDI is a frequently used outcome measure for
examining changes in depressive symptomatology
(McRoberts and Lambert, 1993).
The State-Trait Anxiety Inventory (STAI) is a
40-item test for measuring anxiety. The STAI
(form Y) is divided into two 20-item parts; the
Y-1, for assessing 'state' anxiety; and Y-2, for
assessing 'trait' anxiety. The test allows subjects to
answer on a 4-point Likert scale ranging from 1 to 4
(Speilberger, 1983). The internal consistencies for
both scales range from 0.83 to 0.92. Test-retest
coefficients vary by form: state anxiety (0.16 to 0.54);
frait anxiety from 0.76 to 0.86 (Dreger and Brabham,
1987). These scales are frequently used measures of
anxiety (Froyd et al, 1996).
The Zimg Self-Rating Depression Scale (ZSDS) is
a widely-used 20-item questionnaire which assesses
the frequency of depressive symptoms (Lambert,
1983). Specifically, the ZSDS examines three basic
aspects of depression: pervasive affect, physio-
logical S3miptoms, and psychological symptoms
(Zung, 1965). Reliability and validity data support
the value of the ZSDS as a measure of depression
(Zung, 1965, 1972).
The Zung Self-Rating Anxiety Scale (ZSAS) is a
20-item self-report instrument based on diagnostic
criteria for anxiety (Zung, 1971). The ZSAS is
answered using a Likert scale ranging from 1 to 4
which allows a patient to rate the amount of time
they are distressed by a particular symptom.
Reliability and validity of the ZSAS appears to be
quite high (Corcoran and Fischer, 1987).
The Taylor Manifest Anxiety Scale (TMAS)
consists of 50 items that were taken from a pool of
200 anxiety-related items found on the Minnesota
Multiphasic Personality Inventory (MMPI). These
50 items were statistically analysed and found to be
the most indicative of manifest anxiety (Lambert,
1983). The TMAS has excellent psychometric prop-
erties (Spielberger, 1983; Lambert, 1983).
Validity Measure of Interpersonal Relations
The Inventory of Interpersonal Problems (IIP) is a
127-item self-report test that was designed to
measure the type of interpersonal difficulties
patients experience as well as the corresponding
degree of discomfort (Horowitz et. al, 1988).
Patients rate the level of distress experienced for
each problem on a 5-point Likert scale, ranging
from 0 to 4. The IIP demonstrates high internal
consistency (0.82 to 0.93). Test-retest reliability is
also high (0.81 to 0.98). Discriminant validity is also
high (Horowitz et al, 1988). The IIP was included in
this study because it is the most frequently recom-
mended measure of outcome for interpersonal
problems (Horowitz et al, 1994).
Validity Measure of Social Role Functioning
The Social Adjustment Scale (SAS) is a self-report
form based on the Social Adjustment Scale semi-
structured interview. The SAS covers seven role
areas: work—as a worker, homemaker, or student;
social and leisure activities; relationships with
extended family; marital role and parental role;
family unit role; and economic role. Reliability and
validity were both foxmd to be adequate for this
self-report outcome measure (Weissman and
BothweU, 1976).
Outcome Questionnaire 253

Procedures clinic, and community mental health subjects),

The undergraduate samples were tested in a class-
room setting, with a proctor administering the
instruments. Retest administration used the same
procedure 3 weeks following the initial testing
period.
The community sample was first contacted by
phone by choosing each tenth name in the local
phone directory. At this time adults in the house-
hold were asked if they would fill out question-
naires in order to help the researchers better
understand some psychological tests and how
people respond to them. If they consented to
participate they were mailed questionnaires along
with a consent form and a self-addressed stamped
return envelope. Responses were anonymous to
encourage candid reporting. Approximately 55% of
those contacted returned questionnaires.
Data from the clinical samples were typically
collected by clinic receptionists who administered
the OQ prior to the patients first therapy session.
Included in the test packet was information
pertaining to subject confidentiality as well as a
consent form. The OQ was made available to
clinicians on a voluntary basis. The employee
assistance programme patients were drawn from a
data base supplied by Human Affairs International
based on their routine use of the instrument. The
various samples that made up the subject pool were
tested in multiple settings with various testing
conditions, and different samples completed sub-
sets of the entire test battery. In some cases, subjects
completed only the OQ (i.e. the EAP, university
while in other cases test batteries aimed at assessing
the validity of the OQ were also administered (i.e.
college undergraduates). The Results (Table 3)
section dealing with validity specifies which test
batteries were administered to particular normative
samples.
RESULTS
Normative Data
Outcome Questionnaire scores for the five study
groups are presented in Table 2 for OQ total score
and for domain scores. Mean scores were broken
down to display gender differences. For those
groups where gender data were available t-tests
for independent samples were conducted indicating
that no statistically significant differences existed
between the average scores of males and females.
This finding held up across patient as well as non-
patient samples. Tliis result suggests that it is
appropriate to develop norms for all subjects
combined rather than developing separate male/
female interpretive graphs.
Sufficient data were available to analyse the
relationship between age and OQ scores. This was
done using a Pearson product-moment correlation
(Cohen and Cohen, 1983). Within the broad age
range of the EAP database (ages 17-74) no
significant relationship was found between age
and OQ score (N = 504, r = 0.01).

Table 2. Comparison of normative samples on the Outcome Questionnaire

Sample N Total Score Distress Interpersonal Sodal
Mean (SD) Mean (SD) Mean (SD) Mean (SD)
Undergraduate 238 42.33 (16.60) 23.08 (10.53) 8.95 (5.39) 10.37 (3.62)
Male 91 42.73 (15.89) 22.71 (10.07) 9.81 (6.24) 10.43 (3.63)
Female 147 42.10 (17.21) 23.43 (10.89) 8.31 (4.72) 10.35 (3.65)
Community 102 48.16 (18.23) 25.73 (10.26) 10.81 (5.74) 9.81 (3.91)
Male 46 49.20 (17.59) 25.37 (9.70) 11.51 (5.83) 10.43 (3.39)
Female 56 48.43 (18.48) 26.52 (10.85) 10.52 (5.39) 9.48 (3.95)
EAP 504 73.02 (21.05) 41.83 (14.15) 17.13 (6.03) 13.76 (4.83)
Male 198 73.52 (21.87) 41.64 (14.48) 17.49 (6.26) 14.41 (5.04)
Female 306 72.70 (20.70) 41.96 (13.96) 16.90 (5.88) 13.33 (4.64)
University oupatient clinic 76 78.01 (25.71) 42.88 (14.72) 17.25 (6.61) 14.24 (5.72)
Male 23 76.27 (26.53) 40.86 (15.08) 17.86 (6.42) 14.27 (5.75)
Female 53 81.82 (23.58) 45.34 (13.82) 17.80 (6.17) 14.70 (5.62)
Community mental
health centre 100 86.07 (19.33)'
'Gender and subscale information not available.
254
Table 3. Test-retest reliability and internal consistency values for the outcome questionnaire total and domain scores
Test-Retest'
(Student)
Symptom distress 0.78 (N == 157)
Interpersonal 0.80 (N == 157)
Social role 0.82 (N == 157)
OQ total 0.84 (N == 157)
'Pearson product-moment correlation coefficient (Cohen and Cohen, 1983).
^Coefficient alpha (Cronbach, 1951).
'All coefficients significant at the 0.01 level of confidence.
Reliability of the OQ
Both test-retest and internal consistency reliability
were assessed using a subsample of 157 under-
graduate students. Internal consistency was also
calculated on the EAP patients. As is evident in
Table 3, both the total score of the OQ and the
symptom distress subscale demonstrated excellent
internal consistency. More heterogeneity existed in
the interpersonal relations and social role content
domains which is not surprising given the breadth
of functioning that these latter subscales attempt to
assess. The undergraduates retook the OQ under
similar circumstances 21 days after the original
administration. All domain and total scores
appeared to be temporally stable using this 3-week
time period (Table 3).
Validity of the OQ
Concurrent Validity
Concurrent validity as measured with the under-
graduate samples involved computing Pearson
product-moment correlations (Cohen and Cohen,
1983) between the OQ total score (and subscale
scores) and the various criterion tests. The expecta-
tion was that a specific criterion would correlate
most highly with particular domain scores. Spec-
ificaUy it was expected that the IIP and SAS would
correlate most highly with the interpersonal rela-
tions and the social role subscales, respectively.
Since the symptom subscale measured general
intrapsychic distress the specific criterion tests
measuring anxiety and depression as well as the
Symptom Checklist-90R were expected to correlate
most highly with this subscale.
As depicted in Table 4, moderate to high validity
coefficients were found between the total score of
the OQ and all criterion measures. As predicted, the
strongest relationships for the total score (and
symptom distress subscale) were found on the
depression (ZSDS and BDI), anxiety (TMAS,
M. /. Lambert et al.
Internal consistency^
(Student) (Patients)
0.92 (N == 157) 0.91 (N == 298)
0.74 (N == 157) 0.74 (N == 294)
0.70 (N == 157) 0.71 (N == 295)
0.93 (N == 157) 0.93 (N == 289)3
STAI-Y2, and ZSAS), and global distress (GSI)
criteria. The interpersonal relations and social role
subscales yielded lower validity coefficients with
these measures of symptomatic distress, but in
every case, relationships between the OQ scales and
criteria were statistically reliable (p < 0.05).
Pearson product-moment correlations were also
calculated between the OQ subscales and the OQ
total score. These correlations computed on the 504
patients from the EAP population showed signifi-
cant relationships between the total score and the
domains of interpersonal relations (r = 0.71), social
role (r = 0.70), and symptomatic distress (r = 0.94).
The lowest correlation was 0.36 between the
domain of social role and interpersonal relations.
At the mid-point were correlations between symp-
tomatic distress and interpersonal relations
(r = 0.51), and sjmiptomatic distress and social role
(r = 0.54).
Sensitivity to change
The OQ's construct validity depends, in part, on
the ability of the OQ to reflect change following
interventions such as psychotherapy. While retest
scores for individuals are not expected to fluctuate
systematically over time, it is expected that
the scores of patients receiving psychological or
psychopharmcological interventions would become
progressively lower over time. Past psychotherapy
research shows that most patients typically improve
in therapy and a portion improve even in placebo
treatments. The greatest gains are expected to take
place by about the eighth therapy session (Lambert
and Bergin, 1994).
Given the consistent nature of these past findings
the construct validity of the OQ would be
supported if the scores for patients after seven
sessions of therapy were lower than their pre-
therapy levels. This hypothesis was tested by
following a subset of patients in treatment at the
University Outpatient Clinic. Of the 76 patients who
Outcome Questionnaire 255

Table 4. Validity estimates for the Outcome Questionnaire

Criterion OQ total score Symptom Interpersonal Social role
distress relations
0.78 (N = 115)' 0.61 (N = 115)'
0.72 (N = 238) 0.70 (N = 238) 0.50 (N = 238) 0.52 (N = 238)
BDI2 0.80 (N = 115)' 0.63 (N = 115)'
0.62 (N = 238) 0.59 (N = 238) 0.44 (N = 238) 0.47 (N = 238)
2SDS2 0.88 (N = 71) 0.89 (N = 71) 0.67 (N = 71)
ZSAS^ 0.80 (N = 71) 0.81 (N = 71) 0.53 (N = 71) 0.71 (N = 71)
TMAS^ 0.86 (N = 71) 0.88 (N = 71) 0.63 (N = 71) 0.64 (N = 71)
STAI(y-l)2 64 (N = 115)' 0.50 (N = 115)'
STAI(y-2)2 80 (N = 115)' 0.65 (N = 115)'
IIP2 0.60 (N = 71) 0.60 (N = 71) 0.54 (N = 71) 0.47 (N = 71)
0.63 (N = 238) 0.58 (N = 238) 0.50 (N = 238) 0.60 (N = 238)
SAS^ 0.62 (N = 71) 0.56 (N = 71) 0.65 (N = 71) 0.44 (N = 71)
0.60 (N = 238) 0.52 (N = 238) 0.47 (N = 238) 0.41 (N = 238)
'These values were obtained with a preliminary 43 item version of the current 45 item test.
^GSI = General Severity Index; BDI = Beck Depression Inventory; ZSDS = Zung Self Rating Depression Scale; ZSAS = Zung Self Rating
Anxiety Scale; TMAS = Taylor Manifest Anxiety Scale; STAI = State Trait Anxiety Inventory (Y-1 = State Anxiety; Y-2 = Trait Anxiety);
IIP = Inventory of Interpersonal Problems; SAS = Social Adjustment Scale.

Table 5. Pre-test and post-test scores for outpatients following seven sessions of Psychotherapy
OQ score N Pre-test Post-test
Mean (SD) Mean (SD) f-value (df)
Total score 40 84.65 (24.14) 67.18 (27.12) 4.78 (39)'
Symptom distress 40 46.20 (14.42) 36.65 (16.58) 4.26 (39)'
Interpersonal relations 40 18.35 (5.75) 15.67 (6.08) 3.30 (39)'
Social role performance 40 15.83 (6.0) 11.98 (5.68) 4.30 (39)'
'Significant beyond the 0.001 level of confidence.

took the CX2 prior to entering therapy 40 patients
had at least seven therapy sessions. A Mest for
correlated samples was used to test the hypothesis
that OQ scores would decrease following a
sufficient dose of individual psychotherapy. As
expected the f-test between the mean of the patient
pre-test scores and the mean of their post-test scores
revealed statistically significant improvement.
These data are presented in Table 5.
Ability to Discriminate Patient and
Non-Patient Samples
Support for the construct validity of the OQ was
also sought by comparing the three cUnical samples'
scores on the OQ with those of the community and
undergraduate samples. It was assumed that
statistically significant differences between the
means of the clinical and normative samples would
suggest that the OQ could reliably distinguish
between these groups. Further it was expected that
the mean scores for the groups would be ordered
from most pathological to least pathological. We
expected the community mental health group to be
most disturbed followed by the university out-
patient clinic sample, the EAP sample, the commu-
nity sample and the undergraduate samples. A one-
way analysis of variance was conducted to deter-
mine the difference between adjacent sample
means. These results are presented in Table 6. Not
only were the samples ranked in order of most to
least disturbed, but in all but one comparison (EAP
versus university outpatient clinic) these differences
were significant at or beyond the 0.01 level of
confidence.
A final consideration for the OQ 45 is related to
the accuracy of the proposed cut-off scores that
discriminate between the normal (w = 342) and
clinical (« = 744) samples. Sensitivity and specificity
indices were used to examine classification accur-
acy. Sensitivity is the probability of correctly
classifying a person in the clinical population when
they are in fact a member of the clinical sample. The
OQ 45 was shown to have a sensitivity index of
0.85, thus the proposed cut-off scores would
correctly classify 85 clinical subjects out of every
100 persons tested.
Specificity is the probability of correctly classify-
ing a person in the normal population when they
256
Table 6. Comparison of level of psychopathology as measured by the OQ across patient and non-patient samples
Comparison group N Mean (SD)
Undergraduate 238
Community 102
Employee assistance programme 504
Uruversity outpatient clinic 76
Community mental health centre 100
'Significant beyond the 0.001 confidence level.
^All values differ significantly, except between the employee assistance programme sample and the university outpatient clinic sample,
using Tuke/s Honestly Significant Difference test.
are indeed in the normal sample. The OQ 45
produced a specificity index of 0.74. Thus 74 out
of 100 normal subjects are correctly identified by the
proposed cut-off scores. Since specificity reflects the
false positive rate (identifying someone as clinical
when they are a member of the normal sample) one
recent finding should be pointed out. Epidemio-
logical studies have estimated that about 20% of a
'normal' population have clinically notable levels of
psychopathology (Regier et al, 1988). Thus, given
this relatively high percentage, one would expect a
higher false positive rate on the OQ 45 as indicated
by the lower specificity index.
DISCUSSION
This research reports on the development of a brief
questionnaire that can be used to assess change in
patients following psychological interventions. The
goal was to create a cost effective self-report scale
that was both reliable and valid.
Results demonstrated that the overall question-
naire and its three subscales possess adequate
stability, as demonstrated by the test-retest coeffi-
cients. The test-retest reliability of the OQ is
comparable to other brief outcome measures such
as those with which it was compared in this study
(e.g. BDI, SCL-90-R). Future research should focus
on the stability of the OQ with clinical samples prior
to their undergoing therapy and over longer
periods of time with normal controls. The internal
consistency for the vmdergraduate as well as clirucal
samples appears to be very high for both the total
OQ score as well as domain scores. These results
suggest that the test as a whole measures common
variables.
Both undergraduate and clinical samples were
used to assess the validity of the OQ. It is clear from
the correlational data found in this study that the
OQ has a strong relationship with other self-report
measures whether they purport to measure anxiety
M. /. Lambert et al.
f-ratio
42.33 (16.60) 146.09'
48.16 (18.23)
73.02 (21.05)2
78.01 (25.71)2
86.07 (19.33)
(e.g. TMAS, ZSAS), depression (e.g. BDI, ZSDS),
general symptoms (e.g. SCL-90-R), interpersonal
problems (i.e. IIP) or social adjustment (i.e. SAS).
The total OQ score correlated most highly with
mono-sjnnptomatic scales and the general severity
index of the SCL-90-R. All these measures tap
symptoms that are common in psychological dis-
orders.
The total OQ score was least correlated with the
two measures that did not specifically aim to
measure psychological distress per se (IIP and
SAS). These two instruments produced the lowest
correlations with the total OQ score as well as the
two domain scores with which we expected the
highest correlations (Interpersonal Relations and
Social Role Performance). This finding can be
partially explained by the high overall internal
consistency of the OQ, suggesting similarity of
resporwes to items by the subjects. This high
internal consistency might be due to item content
and suggests the need to carefully examine and
possibly revise the Interpersonal and Social Role
subscales. It could also be a result of the homo-
geneous subject sample (i.e. imdergraduates) and
suggests the need to conduct further validity
studies with patient samples since patients can be
expected to have a greater range of role impairment
and interpersonal difficulties.
The high intercorrelations among the subscale
scores suggests a single underlying factor. This may
reflect the fact that both problems in social role
functioning as well as interpersonal problems are
clearly related to internal distress, either as a direct
cause of psychopathology, as a result of this
psychopathology, or as both cause and effect. This
intertwining of problems has been noted elsewhere
(Klerman, 1974; Horowitz et al, 1988). It may be
that these three domains covary so highly that they
cannot be statistically separated. That is, despite
differences in content, subscales may not provide
distinct information. A factor analytic investigation
of a larger data set has been completed that further
Outcome Questionnaire
examines these issues (Meuler, 1995). A related
topic for future research is to examine whether
clients change at the same rate or show differential
response to treatment across the three domains. It is
important to note that the OQ was not designed as a
multitrait test such as the MMPI. The OQ was
instead designed as a single measure of patient
recovery in mental health contexts that draws items
from three theoretically interrelated domains, a
presupposition supported by its high internal
consistency.
It was anticipated that patients completing the
OQ would score significantly higher than those
from community and student samples. Such a
difference was found in this study, strongly sug-
gesting that the OQ measures what it purports to
measure—psychological distress. It is interesting to
note that the OQ could distinguish not only the
community mental health patients from other
clinical groups but that it differentiated student
samples from community volimteers. It is not
common for outcome measures to be able to make
these same distinctions (Gnmdy, Grundy and
Lambert, 1996). It remains to be seen if outpatient
and inpatient samples can also be successfully
distinguished from each other.
Because psychotherapy should serve to decrease
the client's levels of distress, the scores on the OQ
should decline over treatment sessions. Changes in
distress following therapy would therefore bolster
the construct validity of the OQ. That such change
was measurable with the OQ was confirmed
following seven sessions of individual psycho-
therapy. Of course, such change cannot be attrib-
uted to therapy itself since the design of our study
did not include any control groups or additional
measures of change. Future research will address
these shortcomings, but the data presented do
suggest that the items that make up the OQ are
rated differently over time by a majority of
treatment participants. Further analysis of session
by session change as measured by the OQ has been
published by Kadera et al, (1996).
Problems typically noted with brief self-report
tests (Derogatis et al, 1974; Boulet and Boss, 1991)
apply to the OQ as a measure of patient distress and
deserve mention. First, interpretation of OQ scores
typically relies on the assumption that the client will
be accurate in the assessment of their mental or
emotional states. Either because of acquiescence,
carelessness, boredom, lack of understanding, psy-
choticism, or numerous other factors, the client's
responses may not be congruent with how (s)he is
really feeling. The OQ has no control for response
257
sets, of which social desirability is reported to be the
most common and problematic (Edwards, 1957).
While this problem may seem serious, it is a general
problem in the scales that have typically been used
to assess outcome. Research suggests no systematic
bias as a consequence of using such self-report
scales (e.g. Ogles et al., 1995) in outcome studies.
However, it seems imperative to use the OQ only in
settings in which clients are motivated to accurately
report their psychological state.
This study provides evidence for the reliability
and validity of the OQ as a measure of psychological
distress. Specifically, high internal consistencies and
test stability were found for the overall score as well
as for each of the three domains. Evidence for
concurrent validity was also found using vinder-
graduate populations and support for construct
validity was demonstrated by the OQ showing
sensitivity to changes across therapy, and the ability
to discriminate between community, student and
clinical samples. Suggestions for future research
include assessing its concurrent validity with a
clirucal sample with other well-known distress
measures. Although certain limitations presently
exist with the measure, the OQ will provide
therapists with an instrument that is easily admin-
istered, available at low cost, measures general
distress, and is sensitive to change in sjrmptoms
and behaviours over short periods of time, while
maintaining sound psychometric properties.
REFERENCES
Ahmed, T. and Smith, R. (1991). Impacts of Managed Mental
Health Care Employee Assistance Programs On Costs and
Utilization. A report prepared for Aetna Health Plans.
Andrews, F. M. and Withey, S. B. (1974). Developing
measures of perceived life quality: Results from several
national surveys. Social Indicators Research, 1, 1-26.
Beck, A. T., Steer, R. A. and Garbin, M. G. (1988). Psycho-
metric properties of the Beck Depression Inventory:
Twenty-five years later. Clinical Psychology Review, 8,
77-100.
Beck, A. T., Ward, C. H., Mendelson, M., Mock, J. and
Erbaugh, J. (1961). An inventory for measuring depres-
sion. Archives of General Psychology, 4, 53-63.
Beiser, M. (1973). Components and correlates of mental
well-being. Journal of Health and Social Behavior, 15,
320-327.
Blau, T. H. (1977). Quality of life, social interaction and
criteria of change. Professional Psychology, 8, 464-473.
Bloom, A. (1987). Liability concern of utilization review
and quality assurance programs. HMO, 1, 128-133.
Boulet, J. and Boss, M. W. (1991). Reliability and validity
of tiie Brief Symptom Inventory. ]oumal of Consulting
and Clinical Psychology, 3(3), 433-437.
258 M. /. Lambert et al.

Brokowski, A. (1991). Current mental health care envir- relationship. The Journal of Psychotherapy Practice and
onments: Why managed care is necessary. Professional Research, 5, 132-151.
Psychology: Research and Practice, 22, 6-14. Klerman, G. L. (1974). Depression and adaptation. In R. J.
Cohen, J. and Cohen, P. (1983). Applied Multiple Regression/ Friedman and M. M. Katz (Eds), The Psychology of
Correlation Analysis for the Behavioral Sciences, 2nd edn. Depression: Contemporary Theory and Research. Washing-
New Jersey: Lawrence Erlbauin Associates. ton, DC: Winston-Wiley.
Corcoran, K. and Fischer, J. (1987). Measures for Clinical Lambert, M. J. (1983). Introduction to assessment of
Practice: A Sourcebook. New York: The Free Press. psychotherapy outcome: Historical perspective and
Cronbach, L. J. (1951). Coefficient alpha and the internal current issues. In M. J. Lambert, E. R. Christensen and
structure of tests. Psychometrika, 16, 297-334. S. S. DeJuUo (Eds), The Assessment of Psychotherapy
Derogatis, L. R. (1977). The SCL-90 Manual: Scoring, Outcome. New York: John Wiley.
Administration and Procedures for the SCL-90. Baltimore: Lambert, M. J. and Bergin, A. E. (1994). The effectiveness
Johns Hopkins University School of Medicine, Clinical of psychotherapy. In A. E. Bergin and S. L. Garfield
Psychometrics Unit. (Eds), The Handbook of Psychotherapy and Behavior
Derogatis, L. R., Lipman, R. S., Rickels, K., Uhlenhuth, Change, 4th edn. New York: John Wiley.
E. H. and Covi, L. (1974). The Hopkins Symptom Lambert, M. J., Ogles, B. M. and Masters, K. S. (1992).
Checklist (HSCL): A self-report symptom inventory. Choosing outcome assessment devices: An organiza-
Behavioral Science, 19, 1-15. tional and conceptual scheme. Journal of Counseling and
Derogatis, L. R., Rickels, K. and Rock, A. F. (1976). The Development, 70(4), 527-532.
SCL-90 and the MMPI: a step in the validation of a McRoberts, C. H. and Lambert, M. J. (1993). Current
new self-report scale. British Journal of Psychology, 128, trends in psychotherapy outcome measurement. A
280-289. poster presented at the Western Psychological Associa-
Diener, E. (1984). Subjective well-being. Psychological tion, phoenix, Arizona.
Bulletin, 95(3), 542-575. Meuler, R. (1995). An exploratory and confirmatory factor
Dreger, R. M. and Brabham, J. L. (1987). Two clinical analysis of the Outcome Questionnaire. Unpublished
validation studies on the state form and types of doctoral dissertation. Department of Psychology,
reliability of the trait form of the State-Trait Anxiety Brigham Young University: Provo, Utah.
Inventory. Multivariate Experimental Clinical Research, Mirin, S. and Namerow, M. (1991). Why study treatment
8(2), 195-209. outcome? Hospital and Community Psychiatry, 42, 1007-
Edwards, A. L. (1957). The Social Desirability Variable in 1013.
Personality Assessment and Research. New York: Holt. Moses-Zirkes, S. (1993). Outcome research: Everybody
Frisch, M. B., Cornell, J., Villanueva, M. and Retzlaff, P. J. wants it. American Psychological Association Monitor,
(1992). Clinical validation of the quality of life March, p. 7.
inventory: A measure of life satisfaction for use in Ogles, B. M., Lambert, M. J. and Swayer, J. D. (1995).
treatment planning and outcome assessment. Psycho- The clinical significance of the NIMH treatment of
logical Assessment, 4, 92-101. depression collaborative research program data. Journal
Froyd, J. E., Umbert, M. J. and Froyd, J. D. (1996). of Consulting and Clinical Psychology, 63, 317-325.
A survey and critique of psychotherapy outcome Regier, D. A., Boyd, J. H., Burke, Jr. J. D., Rae, D. S.,
measurement. Journal of Mental Health, 5, 11-15. Myers, J. K., Kramer, M., Robins, L. N., George, L. K.,
Grundy, C , Grundy, E. and Lambert, M. J. (1996). Kamo, M. and Locke, B. Z. (1988). One-month
Assessing clinical significance: Application to the prevalence of mental disorders in the United States.
Hamilton Rating Scale for Depression. Journal of Archives of General Psychiatry, 45, 977-986.
Mental Health, 5, 25-33. Richardson, L. M. and Austad, C. S. (1991). Realities
Horowitz, L. M. (1979). On the cognitive structure of of mental health practice in managed care settings.
interpersonal problems treated in psychotherapy. Professional Psychology Research and Practice, 22(1), 52-59.
Journal of Consulting and Clinical Psychology, 47, 5-15. Spielberger, C. D. (1983). Manual for the State-Trait Anxiety
Horowitz, L. M., Locke, K. D., Morse, M. B., Waikar, S. V., Inventory STAI (PormY). Palo Alto, California: Consult-
Dryer, D. C , Tamow, E. and Ghannam, J. (1991). Self- ing Psychologists Press.
derogations and the interpersonal theory. Journal of Veit, C. T. and Ware, J. E. (1983). The structure of psycho-
Personality and Social Psychology, 61, 68-79. logical distress and well-being in general populations.
Horowitz, L. M., Rosenberg, S. E., Baer, B. A., Ureno, G. Journal of Consulting and Clinical Psychology, 51(5),
and VUlasenor, V. S. (1988). Inventory of interpersonal 730-742.
problems: Psychometric properties and clinical Wiessman, M. M. and Bothwell, S. (1976). Assessment of
applications. Journal of Consulting and Clinical Psy- sodal adjustment by patient self-report. Archives of
chology, 56, 885-892. General Psychiatry, 33, 1111-1115.
Horowitz, L., Strupp, H. H. and Lambert, M. J. (1994). Zung, W. W. (1965). A self-rating depression scale.
Report on the APA sponsored core battery confer- Archives of General Psychiatry, 12, 63-70.
ence. March 1994, Vanderbilt University, Nashville, Zung, W. W. (1971). A rating instrument for anxiety
Tennessee. disorders. Psychosomatics, 6, 371-379.
Kadera, S. W., Lambert, M. J. and Andrews, A. A. (1996). Zung, W. W. (1972). The depression status inventory:
How much therapy is really enough: A session-by- An adjunct to the Self-Rating Depression Scale. Journal
session analysis of the psychotherapy dose-effect of Clinical Psychology, 28, 539-543.