Early Childhood Education Journal, Vol. 30, No.

4, Summer 2003 ( 2003)

Agreement and Stability of Teacher Rating Scales for Assessing ADHD in Preschoolers
Sandra B. Loughran1,2

This study investigated the agreement and stability of 3 teacher rating Scales used to assess ADHD in preschool children: the ADHD Rating Scale, the Child Attention Profile (CAP), and the Conners’ Teacher Rating Scale-28 (CTRS-28). A sample of suburban children (n = 60) was observed and rated by their teachers and assistant teachers at preschool level (Time 1) and 4 years later at the elementary school level (Time 2). Agreement among the rating scales and interrater agreement between teacher and assistant teacher ratings yielded noticeably stronger correlations at Time 2 than at Time 1. Over the 4-year interval of the study, there was a significant change in the number of children identified as potential ADHD risks. It is probable there were a high number of falsepositive indications in the preschool ADHD screenings. It is also possible that immature behavior of preschool children may mimic ADHD behavior at the elementary school level.
KEY WORDS: ADHD; preschool; elementary; teacher rating scales.

INTRODUCTION Attention Deficit/Hyperactivitiy Disorder (ADHD) has been recognized as a strong contributor to academic underachievement and behavioral maladaptation (Goldstein & Goldstein, 1993; Lerner, Lowenthal, & Lerner, 1995; Palfrey, Levine, Walker, & Sullivan, 1985). ADHD is a disorder that is marked by developmentally inappropriate levels of inattention, impulsivity, and overactivity. With a mean onset between 3 and 4 years of age (American Psychiatric Association, 1994), there is a need to clarify the behavioral criteria that put a preschool child at risk of this disorder. Given the legal mandates that allow ADHD children to be considered eligible for special services by their school district when the diagnosis is accompanied by a documented developmental delay (Public Law 94– 142), it is important to provide these children with an

1

Elementary Education Department, Dowling College, Oakdale, Long Island, NY. 2 Correspondence should be directed to Sandra B. Loughran, Dowling College, Oakdale, Long Island, New York 11769; sandra.loughran @eudoramail.com.

early diagnosis. With the recent clarification of these mandates (Part B of the Individuals with Disabilities Act [IDEA]; Section 504 of the Rehabilitation Act of 1973) regarding evaluation, there is an urgent need for reliable educational assessments of children suspected of having ADHD and a need to address these concerns as early as possible. Additionally, there has been an increasing interest in the assessment and treatment of very young children since Public Law 99–457 mandated developmental services for this age group (Shelton & Barkley, 1993). At the same time, excessive diagnosing and overpathologizing in the assessment of ADHD in preschool children in our society continues to be a controversial and ethical concern (American Psychiatric Association, 1994; Barkley, 1995). Many children between ages 3 and 5 years are inattentive, hyperactive, and impulsive at times as part of their normal development. However, under certain conditions, these same behaviors are also identified as behavioral descriptors of ADHD. The possibility of a misdiagnosis due to normal developmental variations is cause for concern. A definitive set of criteria sensitive to the developmental level of preschool children is needed, but such 247
1082-3301/03/0600-0247/0  2003 Human Sciences Press, Inc.

are among the most commonly used instruments for gathering school information. 4-year-olds. 1978) has been normed for children as young as 3 years of age. As a result. Teacher rating scales of behavior are considered to be one of the most important components in the assessment of ADHD and. It was considered that the same rater would be unlikely to have changed significantly in that respect as opposed to the differences that might be found in two different raters. 4 years later. The children were from a suburban.9 years. most referrals for ADHD are made after a child’s entrance into elementary school. 27 boys (47%) and 33 girls (53%). METHODOLOGY This study examined the agreement and the stability of three teacher rating scales administered at two different times with a group of children when they were in preschool (Time 1) and again. Although most teacher rating scales that assess ADHD are normed for school-aged children aged 5 years and older. It is believed that the study benefited from the greater control afforded through bringing the same children together in a replicated environment. 1984). when they were in elementary school (Time 2). 1990) and the Child Attention Profile (CAP). as such. These comparisons serve two purposes. it is important to research their agreement and stability over time when used with preschool children. each child in the preschool was observed and rated by his/her teacher and assistant teacher using the three teacher rating scales. How are we going to detect and adequately assess ADHD in preschool children? What instruments are going to be predictive of ADHD? Is there agreement among these instruments? Will the children who exhibit symptoms as preschoolers continue to exhibit those same symptoms at the elementary school level? In what ways. upper-middle-class community where each of the children had attended the same private preschool. 8. respectively.2 years and at Time 2. As indicated. in what ways. which was derived from the Child Behavior Checklist-Teacher Report Form (Edelbrock & Achenbach. Conners. the same teachers and assistant teachers did the ratings at both Time 1 and Time 2. There were three teachers and three assistant teachers. The teachers had experience levels of 7. may misdiagnosis be harmful? Because teacher rating scales are a primary source of information. 3-year-olds. First. Second.248 criteria are lacking (Campbell. These same rating scales also provide data for developing an intervention plan for the individual child who has been diagnosed with ADHD. The present research compared the CTRS to two other commonly used rating scales: The ADHD Rating Scale (Du Paul. The change in setting from a preschool environment to a more structured elementary school environment may reduce this masking effect and be a contributing factor to the identification of true ADHD. the children came from three different classes. The problem in identifying that child in preschool is that the symptoms are masked by the much wider range of acceptable behavior in preschool. One of the most important factors in the use of behavioral ratings scales is the raters’perceptions of what is appropriate behavior in children. PARTICIPANTS The sample consisted of 60 children. and 5-year-olds. The teachers had elementary education certification with experience at both the preschool and elementary school levels. 10. it is necessary to know if rating scales used with preschool children will accu- Loughran rately predict the presence of ADHD later in childhood as defined by DSM-IV (American Psychiatric Association. and were assigned to the peer group with whom they had attended preschool. The three assistant teachers did not have professional training. 1995). Koplewicz. the Conners’ Teacher Rating Scale (CTRS. & Ulrich. The questions that continue to plague the field and that need further investigation are many. however. if any. There is a continuing need for longitudinal studies addressing the use of teacher rating scales as predictors of ADHD in preschool children. the children returned to the preschool. agreement among several teacher rating scales could prove greater reliability in assessing ADHD in preschool children. At Time 1. The agreement among these scales with a group of preschool children was examined over an extended period using ratings by the same teachers and assistant teachers at preschool (Time 1) and again 4 years later in elementary school (Time 2). An additional benefit was to reunite the participants in an environment that closely approximated that in which they had originally been rated. A child who is truly ADHD is almost certain to be ADHD at both preschool and at elementary school level. At Time 1. and Klein (1997) found this instrument to be reliable with the preschool population. Goyette. 1994). is early diagnosis helpful for the preschool population? And. During a reunion at Time 2. Research by Miller. and 30 years. they did . the same environment in which they were rated at Time 1. if any. The mean age at Time 1 was 4.

The Hyperactivity Index was the scale used in this study. The Time 2 observations were conducted over a one week period. None of the teachers or assistant teachers were informed of these diagnoses. At Time 2. Prior to the reunion. Parents were questioned as to any ADHD identification done by other medical or educational professionals. they were encouraged to rate behaviors as they observed them at Time 2 only. when the children were in grades 2 through 5. 1984). and the CTRS-28 (Goyette et al. the children came in small groups (each with their original class) and were observed for a single day.9 years. Observers rate the severity 249 of these characteristics on a 3-point scale. During that week. None of the children in this sample had been diagnosed with ADHD prior to Time 1. the 6–12-year norms of 8 or more symptoms were used as the cut-off score. and the Hyperactivity factor. The ages of the children at Time 2 ranged from 7. The factors for this study include the Hyperactivity Index. There was no differentiation for preschool vs. For the older group. At Time 1. none of them received any special training in the field of ADHD. and an impulsivity/hyperactivity score. All of the teachers and assistant teachers engaged in the study had uninterrupted employment over the 4year interim period at the preschool where the study was conducted. The ADHD Rating Scale is a scale using the 14 items of DSM-III-R for ADHD. and the appropriate norms were used for each group. 1978). The raters were cautioned against preconceived notions influencing the rating of participants. At that time. It can be scored to yield four factors: Conduct Problem. at Time 2. Observers rate the severity of symptoms on a 4-point scale. The observations for Time 1 were made over a longer period of time than observations had in Time 2. It is believed that the benefits of having the participants reunited in this unique set of circumstances outweighed any limitation that might be associated with a shorter period of observation. This scale also has a total score. Hyperactivity. The only norms available for the CAP were for 6–16-year age group. the children were attending public or private schools in the area.. This scale is the only scale of the three scales normed for use with 3-year-olds and with norms for the different age groups in this study. with the exception of one child who was being schooled at home by a parent. MEASURES Three teacher rating scales were used in this study: the ADHD Rating Scale (DuPaul. While the recommended threshold of 10 of 14 symptoms. including some who were formally diagnosed by outside agencies before leaving the preschool. Each factor is considered a scale and has been normed separately.6 to 10. Written informed consent was obtained from each child’s parents to participate in this study. elementary school ages. The normative cut-off point at the 93 percentile was the threshold used. an inattention score. The workshop addressed both assessment and strategies for working with ADHD symptoms in the classroom. 1984). the CAP (Edelbrock & Achenbach. All teachers and assistant teachers had participated in a staff development workshop on ADHD prior to Time 1. Each day was 5 hours long. InattentionPassive. Age-appropriate activities were conducted.0 years. they were reunited with their same preschool peer group. 1990). The CTRS-28 is a 28-item questionnaire concerning various types of child behavior problems and widely used for clinical and research applications with children.9 to 6. The measures used in this study were the number of symptoms presented in the ratings. Every effort was made to structure the reunion and the day’s activities to afford the teachers and the assistant teachers an opportunity to refamiliarize themselves with their former students and to ensure the maximum observation time. Neither child was on medication nor receiving special services at school. However. or total score. The descriptors of behavior are rated by the observer on a 4-point scale. The total score was used for this study. an inattention/restlessness score. In particular. the Inattentive-Passive factor. The CAP is composed of 12 items taken from the Child Behavior Checklist Teacher Report Form (Edelbrock & Achenbach. and an overactivity score. they were in daily contact with a number of children who evidenced ADHD symptoms. and Hyperactivity Index. a total score. two children had been diagnosed with ADHD at elementary school. including group-structured activities as well as free-play and individual activities. The activities specifically addressed each question asked on the three teacher . the children ranged in age from 2. DATA COLLECTION PROCEDURES The children were rated at Time 1 during the course of normal attendance at the preschool.Assessing ADHD in Preschool have prior experience working in various settings at both elementary and preschool levels. which was not normed for preschoolers but was recommended was used (Barkley. They were also familiarized with the teacher rating scales used in this research. 1990). During that time. which rates the number of symptoms.

Perhaps the most telling results are the number of subjects identified by each rating scale as potentially ADHD in preschool and 4 years later in elementary school.50 5.5 1.66 4.60 vs.01). that is. M SD Time 2 (N = 60) M SD t p 6. . 15% of the sample was identified by at least Fig. p = .3 −1.53 6. 2.69 9.. .07 46.7 .80 43.37 6.80 for CAP.1 6. Those results are shown in Fig.71 4. 1.13 4.7 2.g. t-Test Comparison of Scale Ratings Between Time 1 and Time 2 by Teacher and by Assistant Teacher Time 1 (N = 60) Scale Ratings Teacher ADHD CAP CTRS-28 Assistant Teacher ADHD CAP CTRS-28 Note. a game with specific rules addressed the questions on each of the scales pertaining to attention as well as ability to get along with peers).60 3. the results of paired t tests between Time 1 and Time 2 are shown in Table I.78 3.04 4.05 ns ns . p = .08 1. .80 for CTRS-28.3. Correlation between rating scales for teachers and assistant teachers at Time 1 and Time 2. 3.37 44. The t tests.7.05 ns tween teachers and assistant teachers at Time 1 and Time 2 were: .250 rating scales (e. the ratings were strongly correlated throughout the study with the CAP/CTRS-28 being the highest (r = . There were nine children who met the criteria for potential ADHD at Time 1 on one or more of the rating scales by either teacher or assistant teacher. at Time 1.07 9. and . Results of Pearson product-moment correlations be- Loughran Table I.61 vs.77 3. The results of that comparison are shown in Fig.98 5.5 2.18 1. With respect to the stability of the individual rating scales. when applied to the total population (N = 60) showed no significant changes in the ADHD Rating Scale and the CTRS-28 between Time 1 and Time 2.11 7.05) and assistant teacher ratings (t = 2. RESULTS The results of comparing the performance of the three rating scales are shown in Fig. Although the correlation among rating scales was higher at Time 2 than at Time 1. Because both teachers and assistant teachers were used in the study.95). . . df = 59. 1. however.75 for the ADHD Rating Scale.52 45. CAP. did show a significant change on both teacher ratings (t = 2. a comparison was made between their ratings for each of the rating scales at Time 1 and Time 2.80 8.23 7.5 ns .62 vs.81 3. Each teacher and assistant teacher completed the teacher rating scales immediately following the reunion each day.

a figure in close agreement with the DSM-IV estimate for preva- lence of ADHD in the school population. 1 missed. But the disparity between the number of potential ADHD candidates indicated by the Fig. the overall score for the rating process was. only one child in that group was later identified as a potential candidate for ADHD. 0 missed. 3 might indicate that CTRS-28 was the most conservative at Time 1 (least false positives) and was also the most consistent over the 4-year interval. Based on the confirmation of the two subjects (by outside diagnosis). DISCUSSION The summary results shown in Fig. 2. The parents of the child identified at Time 1 opted for further observation rather than early intervention. and 0 false positives at Time 2. and 8 false positives at Time 1. The two children confirmed as meeting the ADHD criteria represent 3% of the study sample. one rating. 3. There had been identification of ADHD for these two children between Time 1 and Time 2 in their elementary schools.Assessing ADHD in Preschool 251 Fig. 1 correct. Significantly. Summary of students meeting ADHD criteria on rating scales at Time 1 and Time 2. 2 correct. Agreement between teachers and assistant teachers for different Rating scales at Time 1 and Time 2. It is also interesting to note that one additional student was identified at Time 2 who did not exceed the identification threshold on the rating scales at Time 1. . although no special services were provided during that interval.

In the case of the ADHD Rating Scale. it is believed that the high number of false positives at the preschool level is attributable. R. and educators. for whatever reason. However. which was not normed for preschoolers. some preschoolers identified as at risk for ADHD later demonstrated a lack of sufficient symptoms to meet the diagnostic criteria.. 1990). Diagnostic and statistical manual of mental disorders (4th ed. 1988). Additionally.. Attention deficit hyperactivity disorder: A handbook for diagnosis and treatment. to two factors. Research by Fischer and colleagues (Fischer. A. All in all. It is most probable that the high number of false positives noted in the ADHD Rating Scale and the CAP at Time 1 were caused. More research is needed in providing the preschool child with a valid diagnosis. in part. Barkley. (1990). supports the presence of the behaviors of inattention. 1). Barkley. in fact. Loughran A limitation of the study may be that all of the children came from an upper-middle class community that was characterized by ethnic and economic homogeneity. M. only one set of norms (male and female) for the age group 6–16 years was available and was used for both Time 1 and Time 2. the lack of preschool age norms or threshold data in the ADHD Rating Scale and the CAP. Labeling can be limiting. 1986. Given a proper norm for the preschool age. labeling may be in order if it can have a prescriptive benefit. Journal of Clinical Child Psychology. The use of multiple rating scales for any assessment might further improve the effectiveness or accuracy of a single assessment tool. Barkley. confirm the cautions of previous researchers about early labeling. & Breen. Barkley.. An accurate diagnosis is crucial because children who could benefit from services rendered to those with ADHD might be denied these services if they are not identified. the appropriate norms covering the age group at Time 2 were applied. it is unlikely that the disparity in the number of children identified by each of the rating scales at Time 1 was due to any shortcoming of the individual scales. As indicated earlier. However. Agreement of observers on these scales is important (Danforth. M. & Breen. Taking charge of ADHD: The complete authoritative guide for parents. (1988). R. R. The effect of diagnosis and treatment of all identified children of preschool age could have had profound consequences and is clearly out of order. This hypothesis. This supports the maturational lag hypothesis for hyperactivity (Kinsbourne. for a portion of the preschool population. by the lack of norms for preschool children. REFERENCES American Psychiatric Association (1994). the recommended threshold of 10 of 14 symptoms of this scale. Fischer. because physicians often use teacher rating scales in the initial assessment of children in diagnosing ADHD as well as assessing the efficacy of stimulant medication with ADHD children (Barkley. In conclusion. facilitating the availability of special services. All three rating scales were in close agreement at the elementary level and at the preschool level. was used (Barkley. 1979). it should be kept in mind that the present study addressed the rating scales. for example. although fluctuations in the rate of observed ADHD as a function of socioeconomic status have been reported (Taylor. A. they should all be effective tools for preschool ADHD assessment. and hyperactivity that may very well be part of the developmental process of early childhood. if not damaging. Newby. not the population. (1995). & Stokes.252 different rating scales at Time 1 is not consistent with the high degree of correlation between scales when applied to the total population. Fischer. Trites. This is important because as this study has indicated. Some children do. in turn. only the CTRS28 had different norms for the two different age groups. and the wide range of developmental variation in a preschool child. New York: Guilford Press. R. . An important factor that bears on the results is the set of norms that were used in classifying a child as potentially ADHD. A.). 14–24. Because ADHD is a chronic disorder. the children who met the criteria at Time 1 but not at Time 2. Development of a multi-method clinical protocol for assessing stimulant drug responses in ADHD children. teacher rating scales provide a valuable piece of the information needed to evaluate and diagnose a child presenting the symptoms of ADHD in the preschool setting and in the elementary school setting. Newby. Clear definitions of what constitutes ADHD behavior at the preschool level would be of value for young children. outgrow the symptoms of ADHD that are presented in preschool years (Campbell. Considering that the overall agreement between rating scales was high (Fig. Barkley. New York: Guilford Press. impulsivity. 1995). 1993) indicated that hyperactivity might normalize with behavioral adjustment across time. Washington. 1977) in that behavior comes closer to normal with age and maturation in some children. & Smallish. these teacher rating scales may be used for monitoring purposes rather than as diagnostic tools. to a child who is still developing—clearly a quandary for parents and teachers of preschoolers. DC: Author. For the CAP. Fletcher. 17. The norms for all three rating scales differentiated between male and female. in the main. their families. at Time 1. 1991).

Journal of Abnormal Child Psychology. S. and validity. R. K. & Barkley. (1985). 36. Normative data on revised Conners’ parent and teacher rating scales. R. R. Goldstein. R. Du Paul. K. The emergence of attention deficits in early childhood: A prospective study. Assessment of attention-deficit hyperactivity disorder in young children. R. . Managing attention disorders in children: A guide for practitioners. G. S. 11. Conners. and conduct problems in preschoolers. 221–238. Barkley. Clinical Psychology Review. Developmental and Behavorial Pediatrics.. J. CA: Brooks/ Cole.. J. (1997). & Stokes. C. British Journal of Psychiatry. W. (1979). Willis (Eds. Fletcher. Walker. Rapin. L. S. S. H. (1995). Blaw. Kinsbourne (Eds. C. New York: Wiley. 149. E.. A. Attention deficit disorders: Assessment and teaching. D. J.. J. In J. M. Palfrey. D. F. T. University of Massachusetts Medical Center at Worcester. (1993). 339– 348. Koplewicz. E.. (1977). The ADHD Rating Scale: Normative data. 52.. & Goldstein. (1995).. & M. G. Fischer. F. & Achenbach. 315–337. (1990). (1991). and treatment implications. Lerner. Trites. 253 Kinsbourne. L. T. 113–119. L.. Hyperactivity in children: Etiology. (1993). R. C. The stability of dimensions of behavior in ADHD and normal children over an 8-year follow-up. psychoeducational. S. Barkley. 207–217. K. Goyette. 25. Journal of Consulting and Clinical Psychology. Behavior problems in preschool children: A review of recent research. The mechanism of hyperactivity. 703– 727.. & Klein. (1978).). B. (1984). R. 21.. I. measurement. 113–149. L. Unpublished manuscript. L. S. New York: Spectrum. and psychosocial assessments (pp.. B. Levine. S. M. In M. A. Topics in child neurology (pp. Culbertson & D. & Ulrich. TX: Pro-ed. 6. & Lerner.. S.. Journal of Abnormal Child Psychology. Testing young children: A reference guide for developmental. A. 562–573. M.. & Smallish. T. inattention.Assessing ADHD in Preschool Campbell. Danforth. reliability. (1993). Childhood hyperactivity. M. Teacher ratings of hyperactivity. 289–306).. 6. H... Baltimore: University Park Press. A. Pacific Grove. Miller. Journal of Child Psychology and Psychiatry. & Sullivan. The teacher version of the child behavior profile: Boys aged 6–11. (1986). Austin. 290–318). J.). Observations of parent-child interactions with hyperactive children: Research and clinical implications. Edelbrock. Lowenthal. Journal of Abnormal Child Psychology. M. Taylor. A. Shelton.

Sign up to vote on this title
UsefulNot useful