You are on page 1of 8

Early Childhood Education Journal, Vol. 30, No.

4, Summer 2003 ( 2003)

Agreement and Stability of Teacher Rating Scales


for Assessing ADHD in Preschoolers

Sandra B. Loughran1,2

This study investigated the agreement and stability of 3 teacher rating Scales used to assess ADHD
in preschool children: the ADHD Rating Scale, the Child Attention Profile (CAP), and the Conners’
Teacher Rating Scale-28 (CTRS-28). A sample of suburban children (n = 60) was observed and
rated by their teachers and assistant teachers at preschool level (Time 1) and 4 years later at the
elementary school level (Time 2). Agreement among the rating scales and interrater agreement
between teacher and assistant teacher ratings yielded noticeably stronger correlations at Time 2
than at Time 1. Over the 4-year interval of the study, there was a significant change in the number
of children identified as potential ADHD risks. It is probable there were a high number of false-
positive indications in the preschool ADHD screenings. It is also possible that immature behavior
of preschool children may mimic ADHD behavior at the elementary school level.

KEY WORDS: ADHD; preschool; elementary; teacher rating scales.

INTRODUCTION early diagnosis. With the recent clarification of these


mandates (Part B of the Individuals with Disabilities Act
Attention Deficit/Hyperactivitiy Disorder (ADHD)
[IDEA]; Section 504 of the Rehabilitation Act of 1973)
has been recognized as a strong contributor to academic
regarding evaluation, there is an urgent need for reliable
underachievement and behavioral maladaptation (Gold-
educational assessments of children suspected of having
stein & Goldstein, 1993; Lerner, Lowenthal, & Lerner,
ADHD and a need to address these concerns as early
1995; Palfrey, Levine, Walker, & Sullivan, 1985). ADHD
as possible. Additionally, there has been an increasing
is a disorder that is marked by developmentally inappro-
interest in the assessment and treatment of very young
priate levels of inattention, impulsivity, and overactivity.
children since Public Law 99–457 mandated develop-
With a mean onset between 3 and 4 years of age (Ameri-
mental services for this age group (Shelton & Barkley,
can Psychiatric Association, 1994), there is a need to
1993).
clarify the behavioral criteria that put a preschool child
At the same time, excessive diagnosing and overpa-
at risk of this disorder.
thologizing in the assessment of ADHD in preschool
Given the legal mandates that allow ADHD chil-
children in our society continues to be a controversial
dren to be considered eligible for special services by
and ethical concern (American Psychiatric Association,
their school district when the diagnosis is accompanied
1994; Barkley, 1995). Many children between ages 3
by a documented developmental delay (Public Law 94–
and 5 years are inattentive, hyperactive, and impulsive
142), it is important to provide these children with an
at times as part of their normal development. However,
under certain conditions, these same behaviors are also
1
identified as behavioral descriptors of ADHD. The pos-
Elementary Education Department, Dowling College, Oakdale, Long
sibility of a misdiagnosis due to normal developmental
Island, NY.
2
Correspondence should be directed to Sandra B. Loughran, Dowling
variations is cause for concern.
College, Oakdale, Long Island, New York 11769; sandra.loughran A definitive set of criteria sensitive to the develop-
@eudoramail.com. mental level of preschool children is needed, but such

247
1082-3301/03/0600-0247/0  2003 Human Sciences Press, Inc.
248 Loughran

criteria are lacking (Campbell, 1995). As a result, most rately predict the presence of ADHD later in childhood
referrals for ADHD are made after a child’s entrance as defined by DSM-IV (American Psychiatric Associa-
into elementary school. A child who is truly ADHD is tion, 1994). The agreement among these scales with a
almost certain to be ADHD at both preschool and at group of preschool children was examined over an ex-
elementary school level. The problem in identifying that tended period using ratings by the same teachers and
child in preschool is that the symptoms are masked by the assistant teachers at preschool (Time 1) and again 4
much wider range of acceptable behavior in preschool. years later in elementary school (Time 2).
The change in setting from a preschool environment to a One of the most important factors in the use of be-
more structured elementary school environment may re- havioral ratings scales is the raters’perceptions of what
duce this masking effect and be a contributing factor to is appropriate behavior in children. It was considered
the identification of true ADHD. that the same rater would be unlikely to have changed
Teacher rating scales of behavior are considered to significantly in that respect as opposed to the differences
be one of the most important components in the assess- that might be found in two different raters.
ment of ADHD and, as such, are among the most com-
monly used instruments for gathering school informa-
tion. These same rating scales also provide data for METHODOLOGY
developing an intervention plan for the individual child
This study examined the agreement and the stabil-
who has been diagnosed with ADHD.
ity of three teacher rating scales administered at two dif-
The questions that continue to plague the field and
ferent times with a group of children when they were in
that need further investigation are many. How are we
preschool (Time 1) and again, 4 years later, when they
going to detect and adequately assess ADHD in pre-
were in elementary school (Time 2). As indicated, the
school children? What instruments are going to be pre-
same teachers and assistant teachers did the ratings at
dictive of ADHD? Is there agreement among these in-
both Time 1 and Time 2.
struments? Will the children who exhibit symptoms as
At Time 1, each child in the preschool was ob-
preschoolers continue to exhibit those same symptoms
served and rated by his/her teacher and assistant teacher
at the elementary school level? In what ways, if any, is
using the three teacher rating scales. During a reunion
early diagnosis helpful for the preschool population?
at Time 2, the children returned to the preschool, the
And, in what ways, if any, may misdiagnosis be harmful?
same environment in which they were rated at Time 1,
Because teacher rating scales are a primary source
and were assigned to the peer group with whom they
of information, it is important to research their agree-
had attended preschool. It is believed that the study bene-
ment and stability over time when used with preschool
fited from the greater control afforded through bringing
children. There is a continuing need for longitudinal
the same children together in a replicated environment.
studies addressing the use of teacher rating scales as pre-
An additional benefit was to reunite the participants in
dictors of ADHD in preschool children.
an environment that closely approximated that in which
Although most teacher rating scales that assess
they had originally been rated.
ADHD are normed for school-aged children aged 5
years and older, the Conners’ Teacher Rating Scale
(CTRS; Goyette, Conners, & Ulrich, 1978) has been
PARTICIPANTS
normed for children as young as 3 years of age. Re-
search by Miller, Koplewicz, and Klein (1997) found The sample consisted of 60 children, 27 boys (47%)
this instrument to be reliable with the preschool popula- and 33 girls (53%). The children were from a suburban,
tion. upper-middle-class community where each of the chil-
The present research compared the CTRS to two dren had attended the same private preschool. The mean
other commonly used rating scales: The ADHD Rating age at Time 1 was 4.2 years and at Time 2, 8.9 years.
Scale (Du Paul, 1990) and the Child Attention Profile At Time 1, the children came from three different classes,
(CAP), which was derived from the Child Behavior 3-year-olds, 4-year-olds, and 5-year-olds.
Checklist-Teacher Report Form (Edelbrock & Achen- There were three teachers and three assistant teach-
bach, 1984). These comparisons serve two purposes. ers. The teachers had elementary education certification
First, agreement among several teacher rating scales with experience at both the preschool and elementary
could prove greater reliability in assessing ADHD in school levels. The teachers had experience levels of 7,
preschool children. Second, it is necessary to know if 10, and 30 years, respectively. The three assistant teach-
rating scales used with preschool children will accu- ers did not have professional training; however, they did
Assessing ADHD in Preschool 249

have prior experience working in various settings at both of these characteristics on a 3-point scale. This scale
elementary and preschool levels. also has a total score, an inattention score, and an over-
All teachers and assistant teachers had participated activity score. The total score was used for this study.
in a staff development workshop on ADHD prior to The normative cut-off point at the 93 percentile was the
Time 1. The workshop addressed both assessment and threshold used. The only norms available for the CAP
strategies for working with ADHD symptoms in the were for 6–16-year age group. There was no differentia-
classroom. They were also familiarized with the teacher tion for preschool vs. elementary school ages.
rating scales used in this research. The CTRS-28 is a 28-item questionnaire concerning
All of the teachers and assistant teachers engaged various types of child behavior problems and widely
in the study had uninterrupted employment over the 4- used for clinical and research applications with children.
year interim period at the preschool where the study was This scale is the only scale of the three scales normed
conducted. During that time, none of them received any for use with 3-year-olds and with norms for the different
special training in the field of ADHD. However, they age groups in this study. It can be scored to yield four
were in daily contact with a number of children who factors: Conduct Problem, Hyperactivity, Inattention-
evidenced ADHD symptoms, including some who were Passive, and Hyperactivity Index. Each factor is consid-
formally diagnosed by outside agencies before leaving ered a scale and has been normed separately. The factors
the preschool. for this study include the Hyperactivity Index, or total
At Time 1, the children ranged in age from 2.9 to score, the Inattentive-Passive factor, and the Hyperactiv-
6.0 years. None of the children in this sample had been ity factor. The descriptors of behavior are rated by the
diagnosed with ADHD prior to Time 1. observer on a 4-point scale. The Hyperactivity Index
The ages of the children at Time 2 ranged from 7.6 was the scale used in this study, and the appropriate
to 10.9 years. At that time, the children were attending norms were used for each group.
public or private schools in the area, with the exception Written informed consent was obtained from each
of one child who was being schooled at home by a par- child’s parents to participate in this study. Parents were
ent. At Time 2, when the children were in grades 2 questioned as to any ADHD identification done by other
through 5, they were reunited with their same preschool medical or educational professionals.
peer group. Prior to the reunion, two children had been
diagnosed with ADHD at elementary school. Neither
DATA COLLECTION PROCEDURES
child was on medication nor receiving special services
at school. None of the teachers or assistant teachers were The children were rated at Time 1 during the course
informed of these diagnoses. of normal attendance at the preschool. The raters were
cautioned against preconceived notions influencing the
rating of participants. In particular, at Time 2, they were
MEASURES
encouraged to rate behaviors as they observed them at
Three teacher rating scales were used in this study: Time 2 only. The observations for Time 1 were made
the ADHD Rating Scale (DuPaul, 1990), the CAP (Edel- over a longer period of time than observations had in
brock & Achenbach, 1984), and the CTRS-28 (Goyette Time 2. It is believed that the benefits of having the
et al., 1978). The ADHD Rating Scale is a scale using participants reunited in this unique set of circumstances
the 14 items of DSM-III-R for ADHD, which rates the outweighed any limitation that might be associated with
number of symptoms, a total score, an inattention/rest- a shorter period of observation. The Time 2 observations
lessness score, and an impulsivity/hyperactivity score. were conducted over a one week period. During that
Observers rate the severity of symptoms on a 4-point week, the children came in small groups (each with their
scale. The measures used in this study were the number original class) and were observed for a single day. Each
of symptoms presented in the ratings. For the older group, day was 5 hours long.
the 6–12-year norms of 8 or more symptoms were used Every effort was made to structure the reunion and
as the cut-off score. While the recommended threshold the day’s activities to afford the teachers and the assis-
of 10 of 14 symptoms, which was not normed for pre- tant teachers an opportunity to refamiliarize themselves
schoolers but was recommended was used (Barkley, with their former students and to ensure the maximum
1990). observation time. Age-appropriate activities were con-
The CAP is composed of 12 items taken from the ducted, including group-structured activities as well as
Child Behavior Checklist Teacher Report Form (Edel- free-play and individual activities. The activities specifi-
brock & Achenbach, 1984). Observers rate the severity cally addressed each question asked on the three teacher
250 Loughran

rating scales (e.g., a game with specific rules addressed Table I. t-Test Comparison of Scale Ratings Between Time 1
the questions on each of the scales pertaining to atten- and Time 2 by Teacher and by Assistant Teacher
tion as well as ability to get along with peers). Each Time 1 (N = 60) Time 2 (N = 60)
teacher and assistant teacher completed the teacher rat-
ing scales immediately following the reunion each day. Scale Ratings M SD M SD t p

Teacher
RESULTS ADHD 6.71 7.50 4.81 8.04 1.7 ns
CAP 4.52 5.1 3.07 4.69 2.3 .05
The results of comparing the performance of the CTRS-28 45.13 6.37 46.77 9.07 −1.5 ns
three rating scales are shown in Fig. 1. Although the Assistant Teacher
correlation among rating scales was higher at Time 2 ADHD 4.78 6.98 3.08 9.60 1.5 ns
CAP 3.37 5.11 1.80 3.53 2.7 .05
than at Time 1, the ratings were strongly correlated
CTRS-28 44.23 7.66 43.80 6.18 .5 ns
throughout the study with the CAP/CTRS-28 being the
highest (r = .95). Note. df = 59.
With respect to the stability of the individual rating
scales, the results of paired t tests between Time 1 and
Time 2 are shown in Table I. The t tests, when applied tween teachers and assistant teachers at Time 1 and
to the total population (N = 60) showed no significant Time 2 were: .61 vs. .75 for the ADHD Rating Scale,
changes in the ADHD Rating Scale and the CTRS-28 .62 vs. .80 for CAP, and .60 vs. .80 for CTRS-28.
between Time 1 and Time 2. CAP, however, did show Perhaps the most telling results are the number of
a significant change on both teacher ratings (t = 2.3, p = subjects identified by each rating scale as potentially
.05) and assistant teacher ratings (t = 2.7, p = .01). ADHD in preschool and 4 years later in elementary
Because both teachers and assistant teachers were school. Those results are shown in Fig. 3.
used in the study, a comparison was made between their There were nine children who met the criteria for
ratings for each of the rating scales at Time 1 and Time potential ADHD at Time 1 on one or more of the rating
2. The results of that comparison are shown in Fig. 2. scales by either teacher or assistant teacher, that is, at
Results of Pearson product-moment correlations be- Time 1, 15% of the sample was identified by at least

Fig. 1. Correlation between rating scales for teachers and assistant teachers at Time 1 and Time 2.
Assessing ADHD in Preschool 251

Fig. 2. Agreement between teachers and assistant teachers for different Rating scales at Time 1 and Time 2.

one rating. Significantly, only one child in that group lence of ADHD in the school population. Based on the
was later identified as a potential candidate for ADHD. confirmation of the two subjects (by outside diagnosis),
It is also interesting to note that one additional student the overall score for the rating process was; 1 correct, 1
was identified at Time 2 who did not exceed the identifi- missed, and 8 false positives at Time 1; 2 correct, 0
cation threshold on the rating scales at Time 1. There missed, and 0 false positives at Time 2.
had been identification of ADHD for these two children
between Time 1 and Time 2 in their elementary schools,
DISCUSSION
although no special services were provided during that
interval. The parents of the child identified at Time 1 The summary results shown in Fig. 3 might indi-
opted for further observation rather than early interven- cate that CTRS-28 was the most conservative at Time 1
tion. The two children confirmed as meeting the ADHD (least false positives) and was also the most consistent
criteria represent 3% of the study sample, a figure in over the 4-year interval. But the disparity between the
close agreement with the DSM-IV estimate for preva- number of potential ADHD candidates indicated by the

Fig. 3. Summary of students meeting ADHD criteria on rating scales at Time 1 and Time 2.
252 Loughran

different rating scales at Time 1 is not consistent with A limitation of the study may be that all of the
the high degree of correlation between scales when ap- children came from an upper-middle class community
plied to the total population. that was characterized by ethnic and economic homoge-
An important factor that bears on the results is the neity. However, although fluctuations in the rate of ob-
set of norms that were used in classifying a child as served ADHD as a function of socioeconomic status have
potentially ADHD. As indicated earlier, only the CTRS- been reported (Taylor, 1986; Trites, 1979), it should be
28 had different norms for the two different age groups. kept in mind that the present study addressed the rating
In the case of the ADHD Rating Scale, the appropriate scales, not the population.
norms covering the age group at Time 2 were applied; In conclusion, teacher rating scales provide a valu-
at Time 1, the recommended threshold of 10 of 14 able piece of the information needed to evaluate and di-
symptoms of this scale, which was not normed for pre- agnose a child presenting the symptoms of ADHD in the
schoolers, was used (Barkley, 1990). For the CAP, only preschool setting and in the elementary school setting.
one set of norms (male and female) for the age group Agreement of observers on these scales is important
6–16 years was available and was used for both Time 1 (Danforth, Barkley, & Stokes, 1991), because physicians
and Time 2. The norms for all three rating scales differ- often use teacher rating scales in the initial assessment
entiated between male and female. of children in diagnosing ADHD as well as assessing the
Considering that the overall agreement between rat- efficacy of stimulant medication with ADHD children
ing scales was high (Fig. 1), it is unlikely that the dispar- (Barkley, Fischer, Newby, & Breen, 1988). Additionally,
ity in the number of children identified by each of the for a portion of the preschool population, these teacher
rating scales at Time 1 was due to any shortcoming of rating scales may be used for monitoring purposes rather
the individual scales. It is most probable that the high than as diagnostic tools. This is important because as
number of false positives noted in the ADHD Rating this study has indicated, some preschoolers identified as
Scale and the CAP at Time 1 were caused, in part, by at risk for ADHD later demonstrated a lack of sufficient
the lack of norms for preschool children. symptoms to meet the diagnostic criteria.
Because ADHD is a chronic disorder, the children The effect of diagnosis and treatment of all identi-
who met the criteria at Time 1 but not at Time 2, for fied children of preschool age could have had profound
whatever reason, confirm the cautions of previous re- consequences and is clearly out of order. Labeling can
searchers about early labeling. Some children do, in fact, be limiting, if not damaging, to a child who is still devel-
outgrow the symptoms of ADHD that are presented in oping—clearly a quandary for parents and teachers of
preschool years (Campbell, 1995). Research by Fischer preschoolers. However, labeling may be in order if it
and colleagues (Fischer, Barkley, Fletcher, & Smallish, can have a prescriptive benefit, for example, facilitating
1993) indicated that hyperactivity might normalize with the availability of special services.
behavioral adjustment across time. This supports the More research is needed in providing the preschool
maturational lag hypothesis for hyperactivity (Kins- child with a valid diagnosis. An accurate diagnosis is
bourne, 1977) in that behavior comes closer to normal crucial because children who could benefit from services
with age and maturation in some children. This hypothe- rendered to those with ADHD might be denied these
sis, in turn, supports the presence of the behaviors of services if they are not identified. Clear definitions of
inattention, impulsivity, and hyperactivity that may very what constitutes ADHD behavior at the preschool level
well be part of the developmental process of early child- would be of value for young children, their families, and
hood. educators.
All in all, it is believed that the high number of
false positives at the preschool level is attributable, in
the main, to two factors; the lack of preschool age norms REFERENCES
or threshold data in the ADHD Rating Scale and the
American Psychiatric Association (1994). Diagnostic and statistical
CAP, and the wide range of developmental variation in manual of mental disorders (4th ed.). Washington, DC: Author.
a preschool child. All three rating scales were in close Barkley, R. A. (1990). Attention deficit hyperactivity disorder: A
agreement at the elementary level and at the preschool handbook for diagnosis and treatment. New York: Guilford
Press.
level. Given a proper norm for the preschool age, they Barkley, R. A. (1995). Taking charge of ADHD: The complete author-
should all be effective tools for preschool ADHD assess- itative guide for parents. New York: Guilford Press.
ment. The use of multiple rating scales for any assess- Barkley, R. A., Fischer, M., Newby, R., & Breen, M. (1988). Develop-
ment of a multi-method clinical protocol for assessing stimulant
ment might further improve the effectiveness or accu- drug responses in ADHD children. Journal of Clinical Child Psy-
racy of a single assessment tool. chology, 17, 14–24.
Assessing ADHD in Preschool 253

Campbell, S. B. (1995). Behavior problems in preschool children: A Kinsbourne, M. (1977). The mechanism of hyperactivity. In M. Blaw,
review of recent research. Journal of Child Psychology and Psy- I. Rapin, & M. Kinsbourne (Eds.), Topics in child neurology (pp.
chiatry, 36, 113–149. 289–306). New York: Spectrum.
Danforth, J. S., Barkley, R. A., & Stokes, T. F. (1991). Observations Lerner, J. W., Lowenthal, B., & Lerner, S. R. (1995). Attention deficit
of parent-child interactions with hyperactive children: Research disorders: Assessment and teaching. Pacific Grove, CA: Brooks/
and clinical implications. Clinical Psychology Review, 11, 703– Cole.
727. Miller, L. S., Koplewicz, H. S., & Klein, R. G. (1997). Teacher ratings
Du Paul, G. J. (1990). The ADHD Rating Scale: Normative data, reli- of hyperactivity, inattention, and conduct problems in preschool-
ability, and validity. Unpublished manuscript, University of Mas- ers. Journal of Abnormal Child Psychology, 25, 113–119.
sachusetts Medical Center at Worcester. Palfrey, J. S., Levine, M. D., Walker, D. K., & Sullivan, M. (1985).
Edelbrock, C. S., & Achenbach, T. A. (1984). The teacher version of The emergence of attention deficits in early childhood: A pro-
the child behavior profile: Boys aged 6–11. Journal of Consulting spective study. Developmental and Behavorial Pediatrics, 6, 339–
and Clinical Psychology, 52, 207–217. 348.
Fischer, M., Barkley, R. A., Fletcher, K. E., & Smallish, L. (1993). Shelton, T. L., & Barkley, R. A. (1993). Assessment of attention-defi-
The stability of dimensions of behavior in ADHD and normal cit hyperactivity disorder in young children. In J. L. Culbertson &
children over an 8-year follow-up. Journal of Abnormal Child D. J. Willis (Eds.), Testing young children: A reference guide for
Psychology, 21, 315–337. developmental, psychoeducational, and psychosocial assessments
Goldstein, S., & Goldstein, M. (1993). Managing attention disorders (pp. 290–318). Austin, TX: Pro-ed.
in children: A guide for practitioners. New York: Wiley. Taylor, E. A. (1986). Childhood hyperactivity. British Journal of Psy-
Goyette, C. H., Conners, C. K., & Ulrich, R. F. (1978). Normative chiatry, 149, 562–573.
data on revised Conners’ parent and teacher rating scales. Journal Trites, R. L. (1979). Hyperactivity in children: Etiology, measurement,
of Abnormal Child Psychology, 6, 221–238. and treatment implications. Baltimore: University Park Press.