PSYCHO-ONCOLOGY

Psycho-Oncology 13: 223–234 (2004)

Published online 27 May 2003 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/pon.715

PSYCHOMETRIC REFINEMENT OF AN
OUTPATIENT, VISIT-SPECIFIC SATISFACTION
WITH DOCTOR QUESTIONNAIRE
D. ANDREW LOBLAWa,b ANDREA BEZJAKa,* P. MONY SINGHc, ANDREW GOTOWIECd, DAVID JOUBERTe,
KENNETH MAHe and GERALD M. DEVINSe
a
Department of Radiation Oncology, Princess Margaret Hospital, University Health Network,
University of Toronto, Canada
b
Department of Radiation Oncology, Toronto Sunnybrook Regional Cancer Centre, University of Toronto, Canada
c
Faculty of Medicine, University of Calgary, Canada
d
Faculty of Medicine, McMaster University, Canada
e
Department of Psychosocial Oncology, Princess Margaret Hospital, University Health Network,
University of Toronto, Canada

SUMMARY

Measuring patient’s satisfaction with their physician is gaining interest but requires a questionnaire that is valid,
reliable and acceptable to patients. We previously published a self-administered visit-specific satisfaction with
physician questionnaire for cancer patients. Eighty outpatients at a Canadian Cancer Center completed the Princess
Margaret Hospital Patient Satisfaction with Doctor Questionnaire and the FACT-G questionnaires along with
demographic information just after clinic visit and again 3–5 days later. Exploratory factor analysis extracted two
factors, labeled ‘physician disengagement’ and ‘perceived support,’ with average coefficient alpha values of 0.93 and
0.90. Test–retest reliability was 0.83 and 0.73, respectively, for the two factors. Confirmatory factor analysis applied
to the data from 174 patients in the original study indicated excellent goodness of fit. PMH/PSQ-MD correlated
moderately with FACT-G (average r=0.37, p50.005). The PMH/PSQ-MD questionnaire is a brief, valid and
reliable questionnaire that taps two complementary facets of patient satisfaction. Copyright # 2003 John Wiley &
Sons, Ltd.

INTRODUCTION (Baider et al., 1995; Baker, 1990; Cohen et al.,

Patient satisfaction is an important facet of quality
of care that is receiving increasing attention. Many
aspects of the medical experience have been
measured from the patient’s perspective including
the quality of care provided by a specific service
(Franklin and McLemore, 1967), the cost and
convenience of various health insurance plans
(Davies and Ware Jr., 1991), general attitudes
towards primary care physicians (Baker, 1996;
Winefield et al., 1995; Zyzanski et al., 1974) and
attitudes about a particular physician–patient visit
*Correspondence to: Department of Radiation Oncology,
Princess Margaret Hospital, 610 University Avenue, Toronto,
Ontario, Canada M5G 2M9. E-mail: andrea.bezjak@rmp.
uhn.on.ca
1996; Cromarty, 1996; Davies and Ware Jr., 1991;
DiBartola and Kanter, 1997; Frederikson, 1995;
Linder-Pelz and Struening, 1985; Smith et al.,
1984; Wiggers et al., 1990; Wolf et al., 1978).
Patients’ satisfaction with the physician is a
critical component of quality of care because the
relationship between doctor and patient entails the
context in which much healing and caring occurs
(Brody, 1992). Empirical findings are consistent
with this assertion. When asked to rank the
components that most significantly influenced
their satisfaction with care, the behavior of doctors
and nurses was ranked first (Pascoe and Attkisson,
1983). In addition, the patient–physician relation-
ship is related to fundamental aspects of health
care. High satisfaction with physician–patient
interaction has been associated with increased
compliance (Haddad et al., 2000; Winefield et al.,

Copyright # 2003 John Wiley & Sons, Ltd. Received 20 December 2001
Accepted 26 October 2002
224 D.A. LOBLAW ET AL.
1995), promptness in seeking help (Ware and
Davies, 1983), superior communication, under-
standing, information retention (Barker et al.,
1996; Ley, 1989; Rubin et al., 1993; Smith et al.,
1984) and continuity of care (Haddad et al., 2000;
Senf and Weiss, 1991; Ware and Davies, 1983).
Confidence in the results obtained from any
outcome measure requires an instrument to be
valid, reliable, and appropriate for that patient
group. In the quality of life (QOL) field, there are
generic (Ware et al., 1992; The EuroQol Group,
1990) and disease-specific QOL instruments (Barry
et al., 1992; Cella et al., 1993; Bellamy et al., 1988)
with the latter at times being better able to detect
specific QOL-related issues. Similarly, the patient
satisfaction field has increasingly been moving
towards disease or condition-specific patient satis-
faction questionnaires (e.g. for diabetes, migraines,
etc). Although disease-specific instruments may be
able to tap aspects of satisfaction that are unique
to that disease, they may suffer from lack of
validation or limited experience in their use and
cannot be compared across disease types (Di Palo,
1997).
To our knowledge, only two published studies
have developed and tested a patient satisfaction
questionnaire in the oncology population (Bredart
et al., 1998, 1999; Loblaw et al. 1999). Only the
Princess Margaret Hospital Patient Satisfaction
with Physician Questionnaire (PSQ-MD) focussed
on patient satisfaction with the physician in an
outpatient setting (Loblaw et al. 1999).
Construct validity is a fundamental psycho-
metric characteristic that involves the extent to
which test scores can be interpreted to represent
the underlying variable (or hypothetical construct)
that the investigator hopes to measure (Cronbach
and Meehl, 1955). Factor analysis provides a
method to evaluate the construct validity of a test
instrument (Anastasi, 1988). If the underlying
factor structure relates in a theoretically mean-
ingful way to the variable(s) a test is believed to
measure, this can be taken as evidence in support
of the instrument’s construct validity. The factor
analysis reported in our previous study indicated
four factors that displayed eigenvalues greater
than 1.0: ‘Interpersonal Skills,’ ‘Quality of Time,’
‘Empathy,’ and ‘Information Exchange’. Only two
of these factors, however, generated eigenvalues
greater than 2.0 (Loblaw et al., 1999). Because the
eigenvalue-greater-than-one criterion can result in
the retention of unstable factors (Cliff, 1988), we
investigated the stability of the PSQ-MD scale’s
Copyright # 2003 John Wiley & Sons, Ltd.
factor structure in a new group of oncology
outpatients.
Discriminant validity is also crucial to effective
measurement (Anastasi, 1988) but is often over-
looked (Campbell and Fiske, 1959). This indicates
the extent to which the measurements produced by
a questionnaire do not overlap with other con-
cepts. In the case of the PSQ-MD, for example, it
is important to establish that scores do not reflect
some other attribute, such as health-related QOL,
in addition to satisfaction with the physician.
Although we found that the PSQ-MD did not
correlate significantly with Spitzer’s QOL ques-
tionnaire (Loblaw et al., 1999), we were concerned
that the range of QOL scores may have been
insufficient to detect a statistically significant
association. In the present study, therefore, we
re-evaluated the discriminant validity of the PSQ-
MD using a more contemporary health-related
Quality of Life Scale (the Functional Assessment
of Cancer Therapy-General, or FACT-G, scale)
(Cella et al., 1993).
Reliability, the consistency with which test
measurements are achieved (Anastasi, 1988), is a
third important psychometric characteristic. In
evaluating the psychometric properties of the
PSQ-MD, we estimated both internal consistency
(the extent to which all items in a scale or subscale
may be interpreted as representing a common
measurement domain) and test–retest reliability
(the extent to which the consistency of measure-
ments is compromised by idiosyncratic errors
associated with a particular measurement occa-
sion) (Anastasi, 1988).
The objectives of this study were thus: (a) to
perform a factor analysis with data from a new
group of oncology outpatients to evaluate con-
struct validity; (b) to examine the correlation
between PSQ-MD scores and the FACT-G, a
cancer-specific health-related QOL measure, to
evaluate discriminant validity; and (c) to estimate
the questionnaire’s reliability (internal consistency
and test–retest reliability).
METHODS
Patient characteristics
Patients were recruited from the outpatient
clinics of a Canadian Tertiary Care Center
(Princess Margaret Hospital) during a 9-week
Psycho-Oncology 13: 223–234 (2004)
 PSYCHOMETRIC REFINEMENT
period in the summer of 1998. Supervising
oncologists were asked whether their patients
could be approached to participate in the study.
A researcher approached patients of consenting
physicians and provided a written letter explaining
the reason for the study, the time commitment
required, and the assurance that all information
would be anonymous and confidential. Oncology
outpatients in follow-up clinic visits were eligible
to enter the study provided they could read
English. New patients and those who had already
entered the study were excluded. Patients who
agreed to participate provided a verbal consent.
The hospital research ethics board approved the
study before we initiated data collection.
Instruments
Respondents completed two questionnaire
packages. Both contained the PSQ-MD and the
FACT-G QOL questionnaire (Cella et al., 1993).
The PSQ-MD is anchored by a four-point Likert-
type response scale (strongly agree, agree, disagree,
strongly disagree) as well as a ‘does not apply
category.’ In its original form, the PSQ-MD
includes 29 items, summed to generate a total
score (this scoring was employed in the present
study whenever PSQ-MD total scores were re-
quired for analyses). The first package (t1) was
completed in the clinic and also included demo-
graphic information (age, gender, education,
marital status, primary cancer site and whether
the patient had immigrated to Canada). Patients
were instructed to complete the second package at
home 3–5 days later and to return it by mail. This
interval was specified a priori to minimize the
effects of intervening events or experiences and to
minimize recall of questions answered in the clinic
at t1.
Statistical analyses
All data were analyzed using the SPSS 9.0 and
AMOS statistical software packages. Negatively
worded items on the PSQ-MD and FACT-G were
scored according to published instructions.
Critical readers may have concerns about the
arbitrary nature of factor analysis used for
determining factor structure in our initial study.
To overcome this, a multi-step process was used to
determine whether a shorter but more meaningful
Copyright # 2003 John Wiley & Sons, Ltd.
225
factor structure could be derived that would be
invariant across t1 and t2. First, an orthogonal
varimax-rotated exploratory factor analysis (EFA)
was performed on the items of the PSQ-MD at t1
and t2. Scree plots showed two factors with
eigenvalues >1.0 at both times. The EFA was
then repeated at both times, constraining the
factor analysis to a two-factor solution. We
specified that items loaded meaningfully on a
factor if its squared loading was greater than 50%
of the communality. According to this criterion, 24
items loaded meaningfully on the same factors at
t1 and t2 in this data set. The resulting factor
solution was evaluated using confirmatory factor
analysis (CFA) applied to our original data set
(Loblaw et al., 1999). In order to address concern
about whether a single-dimensional factor struc-
ture could have equally explained the observed
results in original data set, a CFA was also
performed using the single-item model.
Factorial invariance for both models was
determined using standard goodness-of-fit criteria
(Bentler, 1990; Bollen, 1989; Browne and Cudeck,
1993; Byrne, 1994; Joreskog, 1993; Tanaka, 1993).
The Chi-squared per degree-of-freedom ratio tests
the difference between the specified model and the
observed data. A statistically significant chi2/df
value suggests that the observed data do not fit the
model. (Bentler, 1990; Joreskog, 1993; Tanaka,
1993). Goodness of Fit Index (GFI) assess how
much better a specified model fits the observed
data, relative to a ‘baseline’ of no model at all. The
GFI typically varies between zero and one, with
higher GFI values suggesting better model fit than
lower GFIs. The Adjusted Goodness of Fit (AGFI)
Index indicates goodness of model fit after
correcting for differential effects associated with
sample size and the number of parameters in a
statistical model (Bollen, 1989). In contrast, the
Comparative-Fit-Index (CFI) assesses the fit of the
model of interest with a competing model in which
there are as many factors as there are observed
variables. As with the GFI, higher CFI values
represent better model ‘fit’ than do lower values.
Typically, the CFI varies between zero and one,
with values exceeding 0.90 accepted as suggesting
good fit (Bentler, 1990; Byrne, 1994). The Stan-
dardized Root Mean Square Residual (SRMR)
indicates the average remaining variance in the
data after extracting the systematic variance
explained by the model of interest. Therefore,
lower values are preferred. The Root Mean Squared
Error of Approximation (RMSEA) estimates not
Psycho-Oncology 13: 223–234 (2004)
226 D.A. LOBLAW ET AL.

relative fit, as with the CFI and GFI, but relative
lack of fit. Therefore, lower RMSEA values are
preferred. The RMSEA evaluates lack of model fit
by comparing the observed model with a hypothe-
tical population distribution of parameter values
(i.e. a w2 distribution; Browne and Cudeck, 1993).
Suggested values for the RMSEA are 0.05 to
indicate a close fit between the model and the
observed data, 0.08 to indicate an acceptable fit,
and values greater than 0.10 to imply lack of fit
between the data and model (Browne and Cudeck,
1993).
Discriminant validity, test–retest reliability and
correlations between demographic variables and
PSQ-MD scores (the sum of the original 29 items)
were estimated using Pearson’s correlation coeffi-
cients. Internal consistency reliability was deter-
mined using Cronbach’s (1951) coefficient alpha.
The significance of changes in questionnaire scores
from t1 to t2 was evaluated by paired t-tests.
RESULTS
Patient characteristics
Of the 149 eligible patients we approached,
92 (62%) agreed to participate and 80 (87%)
provided data at both t1 and t2. There were 57
patients who did not participate. The most
commonly cited reasons were that 17 (31%) did
not have enough time and 14 (25%) were not
interested. Table 1 lists the demographic charac-
teristics of those who entered the study. Their
mean age was 60.8 years (range=32–85 years,
S.D.=13.2 years) and 83% were female. The
second questionnaire package was completed a
mean of 4.3 days (S.D.=2.7 days) after the first
questionnaire package; 83% of respondents com-
pleted it within 3–5 days, as per protocol instruc-
tions. There were no significant correlations
between any demographic variable and PSQ-MD
scores at t1 or t2.
Factor analysis
There were 24 items that loaded consistently
on the same factors following EFA at t1 and t2.
Table 2 shows these 24 items, their factor loadings
and the communalities. The factor onto which
each item loaded meaningfully is indicated by
bold-faced type. Thirteen items loaded on the first
factor and 11 items loaded on the second. The
content of each of the factors was examined by
three of the investigators (DAL, AB, GMD) and

Table 1. Patient characteristics of those who participated in the study (n=92); some patients did
not complete the demographic questionnaire. yIncludes skin, brain, bone, testicular cancer and
sarcoma

Demographic variable Proportion %

Primary cancer site Breast 47/85 55.3

Gastrointestinal tumors 4/85 4.7
Gynecologic tumors 8/85 9.4
Head and neck 13/85 15.2
Lung 1/85 1.2
Lymphoma 3/85 2.4
Prostate 0/85 0
Othery 9/85 10.6

Level of education Secondary school 41/85 48.2

Community college 18/85 21.2
University 15/85 17.6
Graduate/professional school 11/85 12.9

Current marital status Married/living with partner 59/86 68.6

Single, never married, not 9/86 10.4
living with partner
Widowed/separated/divorced 18/86 20.9

Born in Canada Yes 51/85 60.0

Copyright # 2003 John Wiley & Sons, Ltd. Psycho-Oncology 13: 223–234 (2004)
 PSYCHOMETRIC REFINEMENT 227

Table 2. Factor loadings and communalities for the 24 items which load consistently on factor 1 (‘physician disengagement’) and
factor 2 (‘perceived support’) at time 1 and 2 for the 29-item two-factor exploratory factor analysis. Bolded factor loading values
represent the dominant factor for each item at t1 and t2. Overall variance explained, internal consistency reliability and average
inter-item correlations for each factor are listed

Items Time 1 Time 2

Loadings Communality Loadings Communality

Factor 1 Factor 2 Factor 1 Factor 2

I will follow the doctor’s 0.09 0.48 0.24
0.13 0.63 0.41
advice because I think s/he
is absolutely right
The doctor did not take my 0.65 0.06 0.42 0.76
0.07 0.59
problems very seriously
The doctor considered my
0.34 0.40 0.27
0.40 0.61 0.54
individual needs when
treating my condition
The doctor did not give me 0.41
0.36 0.30 0.76
0.19 0.61
all the information I thought
I should have been given
The doctor went straight to 0.83
0.05 0.69 0.47
0.17 0.25
my medical problem without
greeting me first
The doctor used words I did 0.57
0.23 0.38 0.34
0.29 0.20
not understand
There was not enough time 0.64
0.47 0.63 0.82
0.24 0.73
to tell the doctor everything
I wanted
I feel the doctor did not 0.75
0.24 0.62 0.89
0.14 0.81
spend enough time with me
It seemed to me that the 0.57
0.23 0.37 0.83
0.22 0.73
doctor was not really interested
in my emotional well-being
I really felt understood by my
0.19 0.69 0.51
0.46 0.66 0.65
doctor
After my last visit with my
0.26 0.70 0.56
0.40 0.73 0.69
doctor, I feel much better
about my concerns
The doctor was not friendly 0.26
0.05 0.07 0.40
0.13 0.17
to me
I understand my illness much
0.09 0.65 0.43
0.19 0.72 0.56
better after seeing this doctor
This doctor was interested in
0.12 0.77 0.61
0.12 0.67 0.46
me as a person and not just
my illness
I feel I understand pretty well
0.23 0.58 0.39
0.17 0.83 0.73
the doctor’s plan for
helping me
I would not recommend this 0.67
0.18 0.48 0.80
0.11 0.65
doctor to a friend
The doctor seemed to brush 0.72
0.25 0.58 0.85
0.22 0.77
off my questions

Copyright # 2003 John Wiley & Sons, Ltd. Psycho-Oncology 13: 223–234 (2004)
228 D.A. LOBLAW ET AL.

Table 2. (continued)

Items Time 1 Time 2

Loadings Communality Loadings Communality

Factor 1 Factor 2 Factor 1 Factor 2

The doctor should have told 0.53
0.38 0.42 0.71
0.24 0.56
me more about how to care
for my condition
After talking with the doctor,
0.36 0.54 0.42
0.13 0.73 0.55
I have a good idea of what
changes to expect in my
health over the next few
weeks and months
The doctor told me to call
0.18 0.51 0.29
0.43 0.61 0.55
back if I had any questions or
problems
I felt the doctor was being
0.24 0.52 0.33
0.40 0.68 0.62
honest with me
The doctor explained the
0.15 0.39 0.17
0.10 0.65 0.43
reason why the treatment
was recommended for me
It seemed to me that the 0.52
0.34 0.39 0.58
0.25 0.39
doctor was not really
interested in my physical
well-being
The doctor should have 0.68
0.34 0.57 0.65
0.40 0.58
shown more interest
Eigenvalue 10.1 2.1 13.1 2.9
Variance explained 34.9% 7.4% 45.1% 9.9%
Reliability: Cronbach’s alpha 0.92 0.85 0.93 0.95
Average inter-item 0.50 0.35 0.53 0.62
correlation

the factors were named ‘physician disengagement’
and ‘perceived support,’ respectively. The eigen-
values at t1 and t2, respectively, were: (a) for
‘physician disengagement,’ 10.1 and 13.1 and (b)
for ‘perceived support,’ 2.1 and 2.9. The variance
explained by the two factors was 42.3% (34.9%
for ‘physician disengagement’ and 7.4% for
‘perceived support’) at t1 and 55.0% (45.1% for
‘physician disengagement’ and 9.9% for ‘perceived
support’) at t2.
The ‘physician disengagement’ and ‘perceived
support’ subscales correlated moderately but sig-
nificantly with each other: Pearson’s correlations
were
0.62 and
0.63 at t1 and t2, respectively
(both p’s50.001). The average inter-item correla-
tion within each factor was 0.50 and 0.52 for the
‘Perceived Support’ and ‘Physician Disen-
gagement’ Scales, respectively, while the average
between inter-item correlation was
0.36. CFA
applied to the 174-patient data set from the
original study (of which 173 patients provided
complete data) corroborated the two-factor solu-
tion. Table 3 reports the statistical fit of the model
factor structure to the observed data. w2/df=2.13,
CFI=0.88 and RMSEA=0.08 when a two-factor
CFA was used. In addition, the one-factor
model resulted in inferior fit to the original data:
w2/df=3.18, CFI=0.77 and RMSEA=0.11.
Discriminant validity
Median FACT-G scores at t1 and t2 were 86.0
and 90.5 and the ranges were 35–107 and 23–108,
respectively. These numbers suggest that the
population had good to excellent QOL, on average

Copyright # 2003 John Wiley & Sons, Ltd. Psycho-Oncology 13: 223–234 (2004)
 PSYCHOMETRIC REFINEMENT
Table 3. Goodness of Fit statistics for two-versus one-dimensional factor structure
of patient satisfaction questionnaire
Statistic
Chi-square/degrees of freedom (df=252)
Comparative Fit Index (CFI)
Robust Comparative Fit Index
Lisrel Goodness of Fit Index (GFI)
Lisrel Adjusted Goodness of Fit Index
Standardized Root Mean Square Residual
Root Mean Sq. Error of Approximation (RMSEA)
n
p50.0001.
(Cella et al., 1993). The average change in FACT-
G total score was
2.3, t (df )=
2.85, p50.006),
indicating a reduction in overall QOL. This change
was due largely to a drop in the emotional
subscale, which decreased 3.6 points, on average,
t (df )=
9.03, p50.001). The PSQ-MD correlated
significantly but not highly with the FACT-G at
t1 and t2 (Pearson r’s=0.38 and 0.36, respectively,
both p’s50.005). Significant correlations were
also observed at t1 and t2 between the FACT-G
and the two subscales but, again, these were
not especially high: (a) ‘physician disengagement’
(Pearson r’s=0.38, p=0.002, and 0.35, p=0.004,
respectively) and (b) ‘perceived support’ subscales
(Pearson r’s=0.24, p=0.05, and 0.37, p=0.002,
respectively).
Reliability
The test–retest reliabilities (as estimated by
Pearson r’s) for the 29-item questionnaire, the
24-item questionnaire, the ‘physician disengage-
ment’ subscale, and ‘perceived support’ subscale
were 0.86, 0.60, 0.79 and 0.76, respectively, all
p’s50.001. The internal consistencies (coefficient
alpha) were 0.80, 0.70, 0.92 and 0.85 at t1 and 0.63,
0.53, 0.93 and 0.95 at t2, respectively. The average
inter-item correlations were 0.50 and 0.53 for
‘physician disengagement’ and 0.35 and 0.62 for
‘perceived support’ at t1 and t2, respectively.
DISCUSSION
Patient satisfaction is an important indicator of
the quality of health care and, in recognition of
this, the concept has received increasing attention
in recent years. This phenomenon is likely due, at
least in part, to increasing pressures from govern-
Copyright # 2003 John Wiley & Sons, Ltd.
229
Two-factor One-factor
structure structure
2.13n 3.18n
0.88 0.77
0.91 0.82
0.79 0.64
0.75 0.57
0.07 0.09
0.08 0.11
ments and third-party payers to maximize health-
care resource efficiencies while preserving a high
standard of quality. Patients’ satisfaction with
their physicians is central to their satisfaction with
health care, overall (Brody, 1992; Pascoe and
Attkisson, 1983). Compared to those who report
comparatively low satisfaction, highly satisfied
individuals are also more likely to display superior
compliance (Haddad et al., 2000; Winefield et al.,
1995), greater promptness in seeking help (Ware
and Davies, 1983) and to retain a higher amount
of information (Barker et al., 1996; Ley, 1989;
Rubin et al., 1993; Smith et al., 1984).
Effectiveness in measuring patients’ satisfaction
with the physician requires an instrument that has
strong support for its validity, high reliability and
is appropriate for the intended population. Be-
cause cancer patients identify different needs and
expectations as compared to people with other
diseases (Wiggers et al., 1990), it can indeed be
argued that it is important when measuring
patient satisfaction in this population to use a
questionnaire that is relevant to cancer patients.
While it is not necessary to develop a question-
naire specifically for the intended population of
study, questionnaires not previously tested in the
study population should at the very least be
evaluated within that context.
Although a number of patient satisfaction
questionnaires include questions about satisfaction
with the physician (Baider et al., 1995; Baker,
1990; Bredart et al., 1998, 1999; Cohen et al., 1996;
DiBartola and Kanter, 1997; Linder-Pelz and
Struening, 1985; Loblaw et al., 1999; Rubin et al.,
1990; Smith et al., 1984; Wiggers et al., 1990; Wolf
et al., 1978), only three are specific to cancer
patients (Bredart et al., 1998, 1999; Loblaw et al.,
1999; Wiggers et al., 1990). For one of them
(Wiggers et al., 1990), there is currently no
Psycho-Oncology 13: 223–234 (2004)
230 D.A. LOBLAW ET AL.
published evidence for its validity or reliability. A
second (Bredart et al., 1998, 1999) reports strong
internal consistency (reliability) and encouraging
initial evidence of validity. The questionnaire is
comprehensive and lengthy, containing 61 items
that tap 11 separate domains of care, including
physicians’ availability. The development and
evaluation of this instrument relied primarily on
hospital inpatients however, and so its relevance to
other patient populations is not established. The
PSQ-MD (Loblaw et al., 1999) is the third-patient
satisfaction questionnaire to be developed specifi-
cally with cancer outpatients in mind. The present
report presents encouraging evidence that builds
on earlier findings supporting its validity and
reliability in a new group of cancer outpatients.
The questionnaire was developed for use with
cancer patients but examination of its items reveals
it is not specific to the condition. Although the
questionnaire should be tested before it is used in a
different population, one may hypothesize that it
would be applicable to patients with other chronic
conditions, such as heart or lung disease.
Construct validity
Exploratory and confirmatory factor analyses
generated evidence that supports construct validity
of the PSQ-MD. Applying a stricter criterion to
maximize the reliability of findings, we observed a
two-factor solution that provided an excellent fit
to the data. The factors corresponded to funda-
mental dimensions of satisfaction with the physi-
cian, one positive and the other negative. The
‘perceived support’ factor taps the extent to which
patients perceive their physicians as concerned
about their welfare, supportive, and aligned with
the patient’s best interests. The ‘physician disen-
gagement’ factor taps the extent to which patients
appraise their physicians as interested only in the
medical aspects of their problems, aloof, and
impatient to move on to the next case. The
goodness of fit of this two-factor solution was
excellent, a conclusion supported by a number of
observations. First, only items that loaded con-
sistently on the same factor across the test and
retest administrations were retained for further
consideration following exploratory principal-
component analysis. Second, confirmatory factor
analysis applied to the data provided by the
original sample upon which the instrument was
developed evidenced a good-to-excellent fit as
Copyright # 2003 John Wiley & Sons, Ltd.
indicated by a number of standard criteria and
superior to the fit explained by a single factor.
Third, although the analysis reported in our
original article yielded a four-factor solution, two
of these factors had comparatively low eigenvalues
(substantially lower than 2.0) and, in fact, the
second factor observed in the present analyses
(physician disengagement) includes three of the
four factors extracted in our previous analyses
(‘interpersonal skills,’ ‘empathy,’ and ‘quality of
time’). With only one exception, all of the items
that loaded on the present ‘perceived support’
factor loaded on the factor we labeled ‘informa-
tion exchange’ in our earlier publication. Table 4
lists the 24 items retained in the two-factor
solution and the factors they loaded on in this
and in our earlier study.
The two subscales identified by factor analysis
demonstrated good test–retest and internal con-
sistency reliability coefficients. In fact, each sub-
scale demonstrated higher internal consistency and
test–retest reliability than did the total score
generated by summing across all 24 items. The
subscales were also moderately negatively corre-
lated, suggesting that although they relate to a
common underlying general construct, the items
within each subscale do not simply tap the same
domain, differentiated solely by the evaluative
phrasing of the items (i.e. positively vs negatively
phrased items). The two-dimensional construct is
further supported by the fact that the average
inter-item correlations within each subscale was
higher than the correlation between the two
subscales. Collectively, these results increase our
confidence in the reliability of the two-factor
solution and indicate that the subscales tap
complementary but distinct facets of patients’
satisfaction with encounters with their physicians.
Given the limited sample of patients in our
exploratory and validation sets, however, we
acknowledge that it will be important to replicate
these results in new patient populations. We are
currently re-examining the factor structure of the
PSQ-MD in a larger group of cancer patients with
approximately equal numbers of men and women.
Discriminant validity
A weak but statistically significant correlation
between the PSQ-MD and the FACT-G QOL
provided evidence in support of the discriminant
validity of our new measure of patient satisfaction.
Psycho-Oncology 13: 223–234 (2004)
 PSYCHOMETRIC REFINEMENT 231

Table 4. The items which loaded significantly at t1 and t2 in the exploratory factor analysis and the factors each of these items
loaded on in the original study.

Item Factor loading

Present study Original study

There was not enough time to tell the doctor everything MD disengagement Quality of time
I wanted
I feel the doctor did not spend enough time with me MD disengagement Quality of time
The doctor went straight to my medical problem without MD disengagement Quality of time
greeting me first
The doctor used words I did not understand MD disengagement Quality of time
The doctor did not take my problems very seriously MD disengagement Interpersonal skills
The doctor did not give me all the information I thought MD disengagement Interpersonal skills
I should have been given
The doctor was not friendly to me MD disengagement Interpersonal skills
I would not recommend this doctor to a friend MD disengagement Interpersonal skills
The doctor seemed to brush off my questions MD disengagement Interpersonal skills
The doctor should have told me more about how to care for MD disengagement Interpersonal skills
my condition
It seemed to me that the doctor was not really interested in my MD disengagement Interpersonal skills
physical well-being
The doctor should have shown more interest MD disengagement Empathy
It seemed to me that the doctor was not really interested in my MD disengagement Empathy
emotional well-being
The doctor considered my individual needs Perceived support Empathy
I will follow the doctor’s advice because I think s/he is Perceived support Information exchange
absolutely right
I really felt understood by my doctor Perceived support Information exchange
After my last visit with my doctor, I feel much better about Perceived support Information exchange
my concerns
I understand my illness much better after seeing this doctor Perceived support Information exchange
This doctor was interested in me as a person and not just my Perceived support Information exchange
illness
I felt I understand pretty well the doctor’s plan for helping me Perceived support Information exchange
After talking with the doctor, I have a good idea of what Perceived support Information exchange
changes to expect in my health over the next few weeks and
months
The doctor told me to call back if I had any questions or Perceived support Information exchange
problems
I felt the doctor was being honest with me Perceived support Information exchange
The doctor explained the reason why the treatment was Perceived support Information exchange
recommended for me
n
MD=Physician.

Although this deviates slightly from our earlier
finding in which Spitzer’s QOL-index did not
correlate significantly with PMH/PSQ-MD scores
(Loblaw et al., 1999), it may be, as we speculated,
that the wider dispersion of the cancer-specific
health-related QOL measure allowed us to detect a
statistically significant correlation. The fact that
the PSQ-MD’s correlation with the FACT-G was
comparatively low nevertheless supports our con-
tention that our scale is not contaminated

Copyright # 2003 John Wiley & Sons, Ltd. Psycho-Oncology 13: 223–234 (2004)
232 D.A. LOBLAW ET AL.
substantially by measurement redundancy. Other
authors have reported results that are of interest in
light of the present finding. Similar to the results
reported here, Andoh and others observed that
their patient satisfaction questionnaire correlated
moderately but significantly with a health-related
QOL questionnaire characterized by a large
response range (the EORTC QLQ-C30) (Andoh
et al., 1997). However, unlike Marshall, Hays and
Mazel’s results from the Medical Outcomes Study
where patient satisfaction correlated with emo-
tional but not physical health (Marshall et al.,
1996), we found that both the emotional and
physical well-being subscales of FACT-G corre-
lated with patient satisfaction.
Reliability
Both the original 29-item PMH/PSQ-MD and
the new two-factor version are highly reliable. All
forms displayed high coefficients indicative of both
internal consistency and test–retest reliability.
These results indicate that the consistency of
PSQ-MD scores is not compromised significantly
by method error associated with either measure-
ment occasions or the domain of item content. The
high level of test–retest reliability should not be
taken to suggest that the PSQ-MD is insensitive
to change. The 3–5 day test–retest interval was
sufficiently short as to rule out valid changes in
patient satisfaction between measurement occa-
sions as a plausible explanation for deviations
from one occasion to the other. Once again, it will
be necessary to evaluate the instrument’s sensitiv-
ity to change directly in future research before it
will be possible to draw conclusions about this
important measurement issue.
Respondent acceptability and appropriateness for
cancer outpatients
We administered the PSQ-MD to a new sample
of 92 oncology outpatients. Almost 90% of the
respondents completed the questionnaire package
on two occasions over a 3–5 day retest interval,
corroborating our earlier observation that the
instrument is acceptable to cancer outpatients.
We also provided space for comments but received
no negative feedback. There were more women in
the present sample (83%) than in our previous
study (59%) but in neither did we observe
Copyright # 2003 John Wiley & Sons, Ltd.
significant correlations between satisfaction with
the physician and a patient’s age, gender, primary
cancer site, education, marital status or whether he
or she had immigrated to Canada. These results
suggest that the questionnaire can be applied to a
broad range of patients with diverse demographic
backgrounds.
Study limitations
There are a number of limitations of our study
and these can be separated into those that are
statistical and clinical. With regard to statistical
limitations, it is possible that the reliability of
parametric estimates may have been lower than
ideal due to a comparatively small sample size. It is
important to note in this regard, however, that the
two-factor structure was cross-validated using
confirmatory factor analysis applied to our origi-
nal 174-patient data set and results indicated
excellent goodness of fit. Additional considera-
tions consistent with the position that our sample
size was adequate to the task are the findings
that we observed high levels of internal consistency
and test–retest reliability and statistically signifi-
cant correlations between patient satisfaction
and QOL. It will be important, nevertheless, to
replicate the observed two-factor structure in an
independent and larger sample.
Clinically, we acknowledge that our convenience
sample is not representative of the larger popula-
tion of patients seen in oncology practice. A high
proportion of our respondents, for example, were
women with breast cancer. In addition, respon-
dents included only those able to read the English
language. Although these considerations relate to
the generalizability of our findings, they do not
compromise the validity of our conclusion that the
PSQ-MD is comprised of two complementary
underlying factors.
Future steps
We continue to evaluate and refine the PSQ-
MD. Research currently in progress involves
administering the questionnaire (along with
others) to large samples of cancer outpatients with
diverse diagnoses, including equal numbers of men
and women (Devins et al., 2002). This will allow
us to continue testing the instrument’s construct
and discriminant validity, its underlying factor
Psycho-Oncology 13: 223–234 (2004)
 PSYCHOMETRIC REFINEMENT
structure, reliability, and to begin accumulating
norms to interpret individual scores. Given the
increasingly pluralistic character of North Amer-
ican society, it will also be important, once
sufficient evidence has been marshaled in support
of the instrument’s psychometric adequacy, to
generate culturally equivalent translations for non-
English speaking patients (Devins, 1999).
ACKNOWLEDGEMENTS
This work was presented in part at the Annual Meeting
of the Canadian Association of Radiation Oncology,
Edmonton, September 22–24, 2000. This research was
supported in part by the Canadian Institutes of Health
Research through a Senior Investigator Award to G.M.
Devins.
REFERENCES
Anastasi A. 1988. Psychological Testing, Macmillan:
New York.
Andoh M, Kobayashi K, Kudoh S, Takeda F. 1997.
Using ‘Care Note’ to measure the level of satisfaction
patients feel with their care, in palliative cancer care,
as a measure of quality of life (Japanese). Nippon Ika
Daigaku Zasshi 64: 538–545.
Baider L, Ever-Hadani P, De-Nour AK. 1995. The
impact of culture on perceptions of patient-physician
satisfaction. Isr J Med Sci 31: 179–185.
Baker R. 1990. Development of a questionnaire to assess
patients’ satisfaction with consultations in general
practice. Br J Gen Pract 40: 487–490.
Baker R. 1996. Characteristics of practices, general
practitioners and patients related to levels of patients’
satisfaction with consultations. Br J Gen Pract 46:
601–605.
Barker DA, Shergill SS, Higginson I, Orrell MW. 1996.
Patients’ views towards care received from psychia-
trists. Br J Psychiatry 168: 641.
Barry MJ, Fowler FJ Jr, O’Leary MP et al. 1992.
Measurement Committee of the American Urological
Association: The American Urological Association
Symptom Index for benign prostatic hyperplasia.
J Urol 148: 1549.
Bellamy N, Goldsmith CH, Campbell J, Stitt LW. 1988.
Validation study of WOMAC: a health status
instrument for measuring clinically important patient
relevant outcomes to antirheumatic drug therapy
in patients with osteoarthritis of the hip or knee.
Rheumatologia 15: 1833–1840.
Bentler PM. 1990. Comparative fit indexes in structural
models. Psychol Bull 107: 238–246.
Copyright # 2003 John Wiley & Sons, Ltd.
233
Bollen KA. 1989. Structural Equations with Latent
Variables. Wiley: New York.
Bredart A, Razavi D, Delvaux N, Goodman V,
Farvacques C, van Heer CA. 1998. A comprehensive
assessment of satisfaction with care for cancer
patients. Support Care Cancer 6: 518–523.
Bredart A, Razavi D, Robertson C, Didier F, Scaffidi E,
de Haes JC. 1999. A comprehensive assessment of
satisfaction with care: preliminary psychometric
analysis in an oncology institute in Italy. Ann Oncol
10: 839–846.
Brody H. 1992. The Healer’s Power. Yale University
Press: New Haven, Conn.
Browne MW, Cudeck R. 1993. Alternative ways of
assessing model fit. In Testing Structural Equation
Models, Bollen KA, Long JS (eds). Sage: Newbury
Park, CA; 136–162.
Byrne B. 1994. Structural Equation Modeling with
EQS and EQS/Windows. Sage: Thousand Oaks,
CA.
Campbell DT, Fiske DW. 1959. Convergent and
discriminant validation by the multitrait-multimethod
matrix. Psychol Bull 56: 81–105.
Cella DF, Tulsky DS, Gray G et al. 1993. The functional
assessment of cancer therapy scale: development and
validation of the general measure. J Clin Oncol 11:
570–579.
Cliff N. 1988. The eigenvalue-greater-than-one rule
and the reliability of components. Psychol Bull 103:
276–279.
Cohen DS, Colliver JA, Marcy MS, Fried ED,
Swartz MH. 1996. Psychometric properties of a standar-
dized-patient checklist and rating-scale used to assess
interpersonal and communication skills. Academ Med
71(Jan. Suppl.): S87–S89.
Cromarty I. 1996. What do patients think about during
their consultations? A qualitative study. Br J Gen
Pract 46: 525–528.
Cronbach LJ. 1951. Coefficient alpha and the
internal factor structure of tests. Psychometrika 16:
297–334.
Cronbach LJ, Meehl PE. 1955. Construct validity in
psychological tests. Psychol Bull 52: 281–302.
Davies AR, Ware Jr, JE. 1991. GHAA’S Consumer
Satisfaction Survey and User’s Manual (2nd edn).
GHAA: Washington, DC.
Devins GM. 1999. Culturally informed psychosomatic
research. J Psychosom Res 46: 519–524.
Devins GM, Bezjak A, Mah K, Loblaw DA, Gotowiec
AP. 2002. Confirmatory factor analysis and clinical
significance of the illness intrusiveness ratings scale in
men and women with six common cancer diagnoses.
Clin Therapeut 24(Suppl. B): 25–26.
DiBartola LM, Kanter SL. 1997. Perceptions of
physicians’ skills. Acad Med 72: 566–567.
Di Palo MT. 1997. Rating satisfaction research: is it
poor, fair, good, very good or excellent? Arthritis Care
Res 10(6): 422–430.
Psycho-Oncology 13: 223–234 (2004)
234 D.A. LOBLAW ET AL.
The EuroQol Group. 1990. EuroQol}a new facility for
the measurement of health-related quality of life.
Health Policy 16: 199–208.
Franklin B, McLemore S. 1967. A scale for measuring
attitudes toward student health services. J Psychol 66:
143–147.
Frederikson LG. 1995. Exploring information-exchange
in consultation: the patients’ view of performance and
outcomes. Patient Educ Counsel 25: 237–246.
Haddad S, Potvin L, Roberge D, Pineault R, Remondin
M. 2000. Patient perception of quality following a
visit to a doctor in a primary care unit. Family Pract
17: 21–29.
Joreskog KG. 1993. Testing structural equation models.
In Testing Structural Equation Models, Bollen KA,
Long JS (eds). Sage: Newbury Park, CA: 294–316.
Ley P. 1989. Communicating with patients: Improving
Communication, Satisfaction, and Compliance. Croom
Helm: London.
Linder-Pelz S, Struening EL. 1985. The multidimen-
sionality of patient satisfaction with a clinic visit.
J Commun Health 10: 42–54.
Loblaw DA, Bezjak A, Bunston T. 1999. Development
and testing of a visit-specific patient satisfaction
questionnaire: the Princess Margaret Hospital Satis-
faction with Doctor Questionnaire (PMH/PSQ-MD).
J Clin Oncol 17: 1931–1938.
Marshall GN, Hays RD, Mazel R. 1996. Health status
and satisfaction with health care: results from the
Medical Outcomes Study. J Consult Clin Psychol 64:
380–390.
Pascoe G, Attkisson C. 1983. The evaluation ranking
scale: a new methodology for assessing satisfaction.
Evaluation Program Plann 6: 347.
Rubin HR, Gandek B, Rogers WH, Kosinski M,
McHorney CA, Ware JE. 1993. Patients’ ratings of
outpatient visits in different practice settings. Results
Copyright # 2003 John Wiley & Sons, Ltd.
from the medical outcomes study. J Am Med Assoc
270: 835–840.
Rubin HR, Ware JEJ, Hays RD. 1990. The PJHQ
questionnaire: exploratory factor analysis and empiri-
cal scale construction. Med Care 28: S22–S29.
Senf JH, Weiss BD. 1991. Patient satisfaction with
health care: intentions and change in plan. Evaluation
Program Plann 14: 299–306.
Smith JK, Falvo D, McKillip J, Pitz G. 1984. Measuring
patient perceptions of the patient-doctor interaction.
Development of the PDIS. Evaluation Health Profess
7: 77–94.
Tanaka JS. 1993. Multifaceted conceptions of fit in
structural equation models. In Testing Structural
Equation Models. Bollen KA, Long JS (eds). Sage:
Newbury Park, CA: 10–39.
Ware JE, Davies AR. 1983. Behavioural consequences
of consumer dissatisfaction with medical care. Evalua-
tion Program Plann 6: 291–297.
Ware JE, Sherbourne CD. 1992. The MOS 36-item
Short-Form Health Survey (SF-36). I. Conceptual
framework and item selection. Med Care 30: 473–483.
Wiggers, JH, Donovan KO, Redman S, Sanson-Fisher
RW. 1990. Cancer patient satisfaction with care.
Cancer 66: 610–616.
Winefield HR, Murrell TG, Clifford J. 1995. Process and
outcomes in general practice consultations: problems
in defining high quality care. Soc Sci Med 41:
969–975.
Wolf MH, Putnam SM, James SA, Stiles WB. 1978. The
medical interview satisfaction scale: development of a
scale to measure patient perceptions of physician
behavior. J Behav Med 1: 391–401.
Zyzanski S, Hulka B, Cassel J. 1974. Scale for the
measurement of satisfaction with medical care:
modifications in content, format and scoring. Med
Care 12: 611–620.
Psycho-Oncology 13: 223–234 (2004)