Copyright # Munksgaard 2001

Acta Psychiatr Scand 2001: 103: 465470

Printed in UK. All rights reserved

ACTA PSYCHIATRICA
SCANDINAVICA
ISSN 0001-690X

A structured interview guide increases Brief

Psychiatric Rating Scale reliability in raters
with low clinical experience
Crippa JAS, Sanches RF, Hallak JEC, Loureiro SR, Zuardi AW. A
structured interview guide increases Brief Psychiatric Rating Scale
reliability in raters with low clinical experience.
Acta Psychiatr Scand 2001: 103: 465470. # Munksgaard 2001.
Objective: To assess the beneficial impact of a structured interview on the
reliability of BPRS ratings in raters with low clinical experience.
Method: Each patient was rated once a week in two separate interviews,
conducted on the same day. The first interview was conducted by a rater
with low clinical experience (recruited from a group of five residents in
psychiatry and one clinical psychologist in training). All second
interviews were conducted by the same highly experienced psychiatrist.
Results: The number of items with full agreement between observers
increased with the use of SIG. The value of intraclass correlation
coefficients for individual items and the total score also increased,
approaching reported studies with experienced raters.
Conclusion: These results suggest that the use of SIG reduces variability
of information gathering in reliability testing of BPRS with less
experienced raters.

Introduction

Since its introduction by Overall and Gorham (1)

in the early 1960s, the Brief Psychiatric Rating
Scale (BPRS) has been widely used to study
changes in psychopathology (2). In a large
number of studies, BPRS showed good reliability
coefficients, both for total score and for individual
items (3). However, most studies used inter-rater
tests and experienced raters, usually in the same
setting. It is difficult to generalize these results to
other conditions, especially when raters with low
clinical experience are used. Indeed, lower reliability coefficients have been reported with less
experienced raters (4, 5).
The testretest method is often used to evaluate
reliability, because it is close to clinical conditions
where ratings from single observers are the rule.
However, the reliability coefficients obtained with
this methodology are usually low, because the
information collected is more variable (6, 7). In

J.A.S. Crippa, R.F. Sanches,

J.E.C. Hallak, S.R. Loureiro,
A.W. Zuardi
Department of Neuropsychiatry and Medical
Psychology, Faculty of Medicine, University of Sao
Paulo, Ribeirao Preto, Sao Paulo, Brazil

Key words: psychiatric status rating scales;

psychometrics; psychopathology; Brief Psychiatric
Rating Scale
J.A.S. Crippa, Departamento de Neurologia, Psiquiatria
e Psicologia Medica, Faculdade de Medicina de
Ribeirao Preto, USP, Av. Nove de Julho, 980, Ribeirao
Preto, Sao Paulo, Brazil
Accepted for publication December 18, 2000

spite of their high feasibility, studies using test

retest methodology to evaluate BPRS are scant
(3, 8).
The traditional procedure for application of the
BPRS has been to fill out the scale soon after a
brief (2030 min), non-structured interview. To
enhance BPRS reliability, versions of the scale
accompanied by operational definitions (anchor
points) of severity levels have been developed
(913).
In addition to the construction of anchor
points, some BPRS versions include guidelines
for questions in order to structure the psychiatric
interview. Structured interviews or guidelines may
be particularly useful when single raters and/or
raters not extensively trained in the recognition
and conceptualization of psychopathology perform the evaluation (14). Unfortunately, few
psychometric studies made use of such modified
versions of the BPRS (15).
465

Crippa et al.
Although reported results have shown that
structured interviews can enhance the reliability
of some rating scales (16, 17), no systematic
studies on the influence of these interviews over
the reliability of the BPRS were accomplished.
The aims of the present work were to assess the
testretest reliability of the Portuguese version
of BPRS with raters having low clinical experience and to ascertain whether the use of a
structured interview guide (SIG) improves reliability coefficients.

Material and methods

Subjects

A total of 52 adults admitted to the Psychiatric

Unit of the University Hospital of the Medical
School in Ribeirao Preto participated in the
study. There were 26 females and 26 males with
average age of 38.56, SDt16.44. They were
diagnosed according to ICD-10 as: bipolar
affective disorders (15), depressive disorders (13),
schizophrenia (12), schizoaffective disorders (5),
acute and transient psychotic disorders (2),
persistent delusional disorders (2), dissociative
(conversion) disorders (1), other specified mental
disorders due to brain damage and dysfunction
and to physical disease (1) and dementia in other
specified diseases classified elsewhere (1).

Instruments

BPRS. The instrument was the Bech et al.s (13)

version of the BPRS, which adds anchor points
and reduces severity to five levels, considering
that additional levels would not have clinical
significance for symptoms such as delusions. The
translation and adaptation to Portuguese of this
BPRS version reproduced the factor structure of
the original scale and added a factor related to
somatic concern. Validity tests of this instrument
performed by experienced raters yielded good
inter-rater reliability coefficients (18).

The Structured Interview Guide. The present

version of SIG was elaborated by two
psychiatrists experienced in the use of BPRS
(J.A.S.C. and J.E.C.H.), based on the guidelines
proposed by Rhoades and Overall (19). Some
questions were modified and some were added,
in order to fit the way doctors usually interview
patients when applying the Portuguese version
of the BPRS. The intention was to include all
items that come from the patients report. The
item disorientation was considered as dependent
on the patients answer to questions formulated
by the interviewer.
The same psychiatrists tested the first series of
questions in a pilot-study with five in-patients,
which had the following diagnoses: schizophrenia
(2), bipolar disorder (2) and depressive disorder
(1). The patients were submitted to two interview
sessions with the same psychiatrist, which were
conducted 1 week apart. Questions that were
found to be difficult to understand were changed.
Finally, a third psychiatrist (A.W.Z.), also experienced in the use of the scale, supervised the
interview. Further changes were made, and a new
pilot-study was performed with the same patients,
resulting in the final version of the instrument
(Appendix I).
466

Procedures

The raters were five residents in psychiatry,

graduated at different medical schools, and one
clinical psychologist. All the raters were undergoing their first year of training, and had no
previous experience with the scale. The participants were trained in the use of BPRS by one of
two of the authors (J.A.S.C. and J.E.C.H.). The
training procedure was as follows:
1)
2)

3)

Discussion on the registration protocol and

definitions of the scale, followed by explanations about their content.
Observation through a unidirectional mirror
of interviews conducted by experienced psychiatrists, followed by application of the
scale. Initially, five interviews were conducted, followed by BPRS filling and further
discussion about doubts and disagreements
among the trainees.
Raters were grouped in pairs, in order to
conduct simultaneous weekly interviews with
in-patients from the psychiatric unit, alternating roles of interviewer and rater. Half
the pairs applied three interviews with
SIG, followed by three interviews without
SIG. The remaining pairs followed the
same procedure in reverse order. After each
evaluation, BPRS was filled independently by
both examiners. There was a between-rater
concordance index above 70% for all individual items of BPRS, allowing the study to
continue.

When the training was successfully completed

the raters were divided into two groups, with
three interviewers each. During a 3-month stay at
the psychiatric unit, each group applied the BPRS
to the patients under care. A single rater
conducted the interview. SIG was given to one

An interview guide for BPRS

Table 1. Enhancing effect of structured interview guidance (SIG) on correlation coefficients of BPRS reliability tests with non-experienced raters
Testretest reliability
Raters without experience
Without SIG

With SIG

Inter-rater reliability
Raters with experience
Zuardi et al. (18)

Items dependent on answers

01. Somatic concern
02. Anxiety (psychic)
05. Self-depreciation and guilt feelings
08. Exaggerated self-esteem
09. Depressive mood
10. Hostility
11. Suspiciousness
12. Hallucinations
15. Unusual thought content
18. Disorientation and confusion

0.58
0.60
0.54
0.73
0.67
0.73
0.64
0.80
0.70
0.72

0.67
0.82
0.72
0.81
0.74
0.70
0.71
0.82
0.78
0.73

0.57
0.70
0.72
0.85
0.72
0.83
0.81
0.89
0.79
0.91

Items dependent on observation

03. Emotional withdrawal
04. Conceptual disorganization
06. Anxiety (somatic)
07. Specific motor disturbances
13. Psychomotor retardation
14. Unco-operativeness
16. Blunted or inappropriate affect
17. Psychomotor agitation

0.62
0.83
0.52
0.60
0.71
0.82
0.69
0.68

0.64
0.78
0.60
0.56
0.75
0.83
0.72
0.60

0.70
0.85
0.61
0.64
0.62
0.62
0.76
0.85

Total score

0.78

0.91

0.93

BPRS items

of the groups, who applied the interviews to the

same patients, once a week for 4 consecutive
weeks, in the mornings. Afterwards, the same
group interviewed other patients for a further 4
consecutive weeks without SIG. Whenever a
patient was discharged, the one admitted to the
same bed was included in the study. The other
group followed the same procedure, but in a
reverse order concerning SIG. All the patients
were interviewed in the afternoon of the same day
by an experienced psychiatrist (J.A.S.C.) using
SIG. BPRS was filled out immediately after each
interview and kept inside individual envelopes for
later disclosure. Following the initial guidance
there was no further discussion among raters, and
no information about the tests was shared until
the end of the experiment. On the whole, the
trainees filled 178 scales (90 with and 88 without
SIG), that were compared with the 178 reference
scales, filled by the experienced psychiatrist. This
allowed assessment of testretest reliability. Since
retest was performed by a different rater, and the
interview between test and retest was short, this
procedure may be viewed as an inter-rater
reliability measure (20).
Analysis of results

The BPRS testretest reliability for total scores

and individual items was assessed through the
intraclass correlation coefficient (ICC). The ICC
for items dependent on answers and for those

dependent on observation, obtained with and

without SIG, were compared through the paired
t-test.
For each patient, the number of items with full
agreement between the morning (inexperienced)
rater and the afternoon (experienced) rater was
recorded. This number was submitted to one-way
analysis of variance (ANOVA) for each group
(SIG first or second). Between-group comparisons, each week, were performed through

Fig. 1. Number of items with full agreement between two

raters (one inexperienced and the other experienced). The m
indicates the period with SIG and J indicates the period
without SIG. Points represent mean and vertical bars the
SEM. The full lines show the group that used SIG in the first
4 weeks, while the interrupted lines show the group that used
it in the last 4 weeks. The asterisk (*) means significant
statistical difference between the two groups (t-test,
Pf0.05).

467

Crippa et al.
Students t-test. The calculation was performed
with the Statigraphic program 2.7 using an IBM/
PC computer. The differences were considered
significant when they reached Pf0.05.

Results

The ICC for individual items and total score

obtained with and without SIG are presented in
Table 1. In this table are also presented the interrater coefficients obtained by Zuardi et al. (18)
with experienced raters without SIG. There was a
significant increase of ICC values with the use of
SIG for items dependent on the patients answers
to questions made by the examiner (paired
t=3.34, df=9, P=0.009). In contrast, for items
directly dependent on observation there was no
significant difference between applications with
and without SIG (paired t=0.4, df=8, P=0.5).
Also in the group using SIG, the correlation
coefficients for individual items approached those
obtained by Zuardi et al. (18) using interobserver
comparisons between experienced raters. In addition, the reliability for the total score of BPRS
was higher.
The number of items with full between-raters
agreement along the weeks of observation is
shown in Fig. 1. At the first, second and fourth
weeks, the group using SIG had scores significantly higher than the group without SIG. In the
group that started with SIG, the number of
concordant items varied significantly along the
time (F7.77=2.125, P=0.0471), with lower numbers when SIG stopped being used. In the group
which started without SIG, a nearly significant
trend towards change (F7.84=1.860, P=0.0865)
occurred, also with higher numbers when SIG was
used. Overall, there was higher agreement among
observers with SIG than without the instrument.

Discussion

The main finding of the present study was that

the use of SIG increased the reliability for
individual and total scores of the BPRS, measured
by the testretest method performed by raters
with no previous experience with the scale. The
obtained scores approached those found in a
previous study in which raters were experienced
and inter-rater comparisons were made (18). This
is remarkable, since the last procedure usually
produces concordance indexes that are higher
than those obtained by using a testretest
approach to reliability assessment (6, 7). As
expected, improvement occurred only in items
that were depended on the patients answer to
468

questions. Therefore, interview guidance seems to

be successful in reducing the variability of
interpretation among different clinicians.
The present results further show that the
number of BPRS items with full agreement
among inexperienced and experienced observers
was greater in the weeks where SIG was used.
Figure 1 shows that the smallest difference
between the two groups that used SIG in opposite
order happened in the fifth week, soon after the
group which started with the SIG stopped using
it, and the group which started without SIG
started doing so. This means that the group that
was making use of SIG retained learning for a
short period, but in the subsequent weeks there
was a decrease. This observation suggests that
continuous use of SIG is important to keep the
ratings consistent along time.
These results are similar to those reported by
Williams (7) with the Hamilton Depression Scale
and by Lindstrom (17) with the Positive and
Negative Syndrome Scale (PANSS). In both cases
there was higher concordance with the use of a
guided interview than without it. This increase in
reliability is likely to be due to standardization of
information gathering during the interviews that
provide the elements for the filling of the scale.
This is particularly important for raters with low
clinical experience. Andersen et al. (4), when
evaluating inter-rater reliability in a group of
psychiatrists with different levels of BPRS training, reported that experienced raters reached
satisfactory intra and inter-rater reliability. The
less experienced showed lower indexes because
their ratings were less consistent. In the present
study, the raters had no previous experience
with BPRS, were just beginning graduate training,
had never worked together in any research
project, had only a short period of training with
the scale, and came from different academic
backgrounds different medical schools and
one psychology course. Therefore, the present
findings support that a structured interview guide
such as SIG can be successful in reducing
variability in information-gathering among less
experienced raters.

Acknowledgments
This work was supported by a grant from FAPESP. A.W.Z. is
the recipient of a CNPQ fellowship. J.A.S.C. was supported
by CAPES (199899) and FAPESP (200001) fellowships.
We also thank Sandra Aparecida Bernardo and Geraldo
Cassio dos Reis for their help in the statistical analysis.
Finally, we thank Frederico G. Graeff for comments and
suggestions.

An interview guide for BPRS

References
1. OVERALL JE, GORHAM DR. The Brief Psychiatric Rating
Scale. Psychol Rep 1962;10:799812.
2. BECH P, MALT VF, DENCKER SJ et al. Scales for assessment of
diagnosis and severity of mental disorders. Acta Psychiatr
Scand 1993;87 (Suppl 372):3540.
3. HEDLUND JL, VIEWEG BW. The Brief Psychiatric Rating
Scale (BPRS): a comprehensive review. J Operat Psychiatry
1980;11:4865.
4. ANDERSEN J, LARSEN JK, SCHULTZ V et al. The Brief
Psychiatric Rating Scale. Dimension of schizophrenia
reliability and construct validity. Psychopathology 1989;
22:168176.
5. RONCONE R, VENTURA J, IMPALLOMENI M et al. Reliability of
an Italian standardized and expanded Brief Psychiatric
Rating Scale (BPRS 4. 0) in raters with high vs. low clinical
experience. Acta Psychiatr Scand 1999;100:229236.
6. SPITZER RL, ENDICOTT J, ROBINS E. Research diagnostic
criteria: rationale and reliability. Arch Gen Psychiatry
1978;35:773782.
7. WILLIAMS JB, GIBBON M, FIRST MB et al. The Structured
Clinical Interview for DSM-III-R (SCID). II. Multisite
testretest reliability. Arch Gen Psychiatry 1992;49:
630636.
8. HAFKENSCHEID A. Psychometric evaluation of a standardized
and expanded Brief Psychiatric Rating Scale. Acta
Psychiatr Scand 1991;84:294300.
9. WOERNER MG, MANNUZZA S, KANE JM. Anchoring the
BPRS: an aid to improved reliability. Psychopharmacol
Bull 1988;24:112117.
10. MCGORRY PD, GOODWIN RJ, STUART GW. The development,
use, and reliability of the Brief Psychiatric Rating Scale
(nursing modification) an assessment procedure for the
nursing team in clinical and research settings. Compr
Psychiatry 1988;29:575587.
11. GABBARD GO, COYNE L, KENNEDY LL et al. Inter-rater
reliability in the use of the Brief Psychiatric Rating Scale.
Bull Menninger Clin 1987;51:519531.
12. LUKOFF D, LIBERMAN RP, NUECHTERLEIN KH. Symptom
monitoring in the rehabilitation of schizophrenic patients.
Schizophr Bull 1986;12:578602.
13. BECH P, KASTRUP M, RAFAELSEN OJ. Mini-compendium of
rating scales for states of anxiety, depression, mania, and
schizophrenia with corresponding DSM-III syndromes.
Acta Psychiatr Scand 1986;83 (Suppl 326):737.
14. RASKIN A. Discussion: recent developments in ascertainment and scaling of the BPRS. Psychopharmacol Bull
1988;24:122124.
15. FAUSTMAN WO. Brief Psychiatric Rating Scale. In: MARUISH
ME, ed. The use of psychological testing for treatment
planning and outcome assessment. New Jersey: Hillsdale,
1994; 371401.
16. WILLIAMS JB. A structured interview guide for the Hamilton
Depression Rating Scale. Arch Gen Psychiatry 1988;45:
742747.
17. LINDSTROM E, WIESELGREN I-M, vON KNORRING L. Interrater
reliability of the Structured Clinical Interview for the
Positive and Negative Syndrome Scale for schizophrenia.
Acta Psychiatr Scand 1994;89:192195.
18. ZUARDI AW, LOUREIRO SR, RODRIGUES CRC, CORREA AJ,
GLOCK SS. A study of the reliability, validity and factor
structure of a Portuguese translation and adaptation of the
modified Brief Psychiatric Rating Scale. Revista ABPAPAL 1994;16:6368.
19. RHOADES HM, OVERALL JE. The semistructured BPRS
interview and rating guide. Psychopharmacol Bull 1988;
24:101104.

20. HAFKENSCHEID A. Psychometric measures of individual

change: an empirical comparison with the Brief
Psychiatric Rating Scale (BPRS). Acta Psychiatr Scand
2000;101:235242.

Appendix I. Structured Interview Guide for the Brief

Psychiatric Rating Scale (SIG BPRS)

The final version of SIG includes a preface listing

the items to be rated taking into account the last
three days (psychic anxiety, hostility, suspiciousness, hallucinations, unusual thought content and
blunted or inadequate affect), and is divided into
three parts. The first, called initial contact, is made
of questions that facilitate the establishment of an
adequate rapport, such as the patients demographic data plus general information about his/
her health. In this part, there are questions to be
used in the first interview, as well as questions
appropriately modified for subsequent interviews.
The second part, named the interview, is made of
more specific questions that refer to every item of
BPRS, in such an order as to warrant a logical
and smooth sequence in the interview. On the left
of each group of guide-questions, the item being
rated is specified. Nevertheless, the rater does not
evaluate individual items on the basis of specific
questions only, as they normally do in structured
interviews of BPRS. Instead, as suggested by
Rhoades and Overall (19), the rater should listen
to and observe the patient, and only after the end
of the interview should evaluate the severity level
of each symptom. Some groups of questions
evaluate more than one item and are marked with
an asterisk. The third and last part of the
interview conclusion gives suggestions on
how to end the contact with the patient in an
empathic way. Usually, conducting the interview
plus filling the scale takes 1030 minutes,
depending on the amount of symptoms, familiarity with the patient and the patients skill to
describe his/her symptoms. At the end of an
interview with SIG, it is possible to obtain
information to assess the severity level of all the
items of the scale. In spite of that, the rater may
pose additional questions to obtain further
information or to better appraise the severity of
certain symptoms.
Structured Interview Guide for the Brief Psychiatric
Rating Scale (SIG BPRS)

Observation: The interviewer should evaluate

the patients condition at the moment of the
interview. Items number 2 (psychic anxiety), 10
(hostility), 11 (suspiciousness), 12 (hallucinations),
15 (unusual thought content) and 16 (blunted or
469

Crippa et al.
inappropriate affect) should be rated in regard to
the patients conditions during the preceding 3
days.
1. Initial contact
BPRS item

Initial interview
Whats your full name? And your age? Where do you live?
Who do you live with? Are you working at the moment?
(Have you worked before? What did you do?)
Somatic concern (01) How do you usually feel (about your physical health)?
How were you last year? Are you worried about any
health problem at the moment?
Orientation (18)
How long have you been here? Tell me why you were
admitted to the hospital. When did it start?
What happened afterwards? Can you tell me which day
it is today (week, month, year)?
Second interview
Screening
How are you feeling since our last talking? Whats most
bothering you these days? (Besides that, what else
bothers you?)
Orientation (18)
Can you tell me which day it is today (week, month, year)?
Can you tell me what you had for dinner yesterday?
Somatic concern (01) Are you worried with any health problem at the moment?

2. Interview
BPRS item

Guiding questions

Orientation (18)

Are you able to concentrate? Whats your memory like?

(If necessary, perform a specific examination.)
Are you worried with something? Have you been feeling
tense or anxious most of the time? (When you feel like
that, do you know why? In which way your anxiety or
worries affect your daily life? Is there anything which
helps you to feel better?)
In the last days have you been feeling you are a
burden to your family or colleagues? Have you
been feeling guilty for something youve done in
the past? Do you believe what is happening now
is some kind of punishment? (Why do you think so?)

Guilty feelings (05)

470

Guiding questions

Depressive mood (09)

How is your mood (happy, sad, irritable)?

Do you believe you can improve?
How does this feeling affect your daily life?
Do you feel other people are talking of or laughing
at you? (How do you know that?) Do you
think there is someone mad at you or trying to
cause you any problems? (Who? Why? How
do you know that?)
In the last days have you been feeling you have a gift
or ability most people do not have? (How do you
know that?) Do you think people envy you? Do you
believe you were born to do some important thing in the
world? In the last days have you been impatient or
irritable with other people? (Were you able to keep cool?
Have you tolerated any provocation? Have you hit
someone or broken any objects?)
Do you believe someone or something outside you is
controlling your thoughts or your actions against your
will? Have you got the feeling that the radio or the
television sends messages to you? Do you feel that
something unusual is happening or is about to happen?
Do you feel that something unusual is happening
in your body or in your head?
Have you had unusual experiences, most of the people
do not have? Have you heard things other people
cannot hear? (Were you awake then? What did
you hear [noise, whispers or voices talking with
you or among them?] How often? Did they interfere with
your daily life?) Have you seen things other people
cannot see? (Were you awake then? What did you
see [lights, shapes, images]? How often?
Did they intefere with your daily life?)

Suspiciousness (11)*

Guiding questions

Orientation (18)

Psychic anxiety (02)

BPRS item

Exaggerated
self-esteem (05)
(hostility) (10)*

Unusual thought
content (15)
(somatic concern)
(01)*

Hallucinations (12)*

* Main item; can rate other items.

3. Closure
Those were the questions I had to ask you. Is
there anything you think is important to mention?
Things I did not ask, or anything you would like to
ask me?