Accelerat ing t he world's research.

Determining an Instructional Level

for Early Writing Skills
Matthew Burns

Cite this paper Downloaded from Academia.edu 

Get the citation in MLA, APA, or Chicago styles

Related papers Download a PDF Pack of t he best relat ed papers 

T ECHNICAL REPORT #7: Technical Feat ures of New and Exist ing CBM Writ ing Measures Wit hi…
Krist en McMast er

Technical Feat ures of Curriculum-Based Measures for Beginning Writ ers

Anna-Lind Pét ursdót t ir

T ECHNICAL REPORT #25: Monit oring Progress of Beginning Writ ers: Technical Feat ures of t he Slope
Krist en McMast er
School Psychology Review,
2011, Volume 40, No. 1, pp. 158 –167

RESEARCH BRIEF

Determining an Instructional Level

for Early Writing Skills

David C. Parker, Kristen L. McMaster, and Matthew K. Burns

University of Minnesota

Abstract. The instructional level is helpful when identifying an intervention for

math or reading, but researchers have yet to investigate whether the instructional-
level concept can be applied to early writing. The purpose of this study was to
replicate and extend previous research by examining technical features of poten-
tial instructional-level criteria for writing. Weekly writing performance was
assessed with 85 first-graders over 12 weeks using Picture-Word and Sentence
Copying prompts. Data from the students with the highest slopes were used to
derive instructional level criteria. Several scoring procedures across Picture-Word
and Sentence Copying prompts produced reliable alternate-form correlations and
statistically significant relationships with a standardized writing assessment. De-
termining an instructional level in writing appears feasible; however, further
research is needed to examine the instructional utility of this approach.

Writing skills are essential for satisfac-
tory academic progress during kindergarten
through twelfth-grade education and for later
vocational success (Graham & Perin, 2007),
but writing problems often go undetected until
late elementary or middle school, when they
become increasingly difficult to remediate
(Baker, Gersten, & Graham, 2003). Early
identification and intervention are critical
for preventing the long-term negative con-
sequences of persistent writing problems
(Berninger, Nielsen, Abbott, Wijsman, & Ras-
kind, 2008). Given that previous research has
established that early writing skills (e.g., tran-
scription skills) are related to compositional
fluency (Graham, Berninger, Abbott, Abbott,
& Whitaker, 1997), the successful develop-
ment of these skills might establish a founda-
tion on which to develop later writing skills,
contributing to the reverse of the national
trend in poor writing outcomes (Salahu-Din,
Daane, & Persky, 2008).

This research was supported in part by Grant H324H030003 awarded to the Institute on Community
Integration and the Department of Educational Psychology, College of Education and Human Develop-
ment, at the University of Minnesota, by the Office of Special Education Programs in the U.S. Department
of Education.
Correspondence regarding this article should be addressed to David Parker, 250 Education Sciences
Building, 56 East River Rd., Minneapolis, MN 55455; e-mail: parke384@umn.edu
Copyright 2011 by the National Association of School Psychologists, ISSN 0279-6015

158

The learning hierarchy (Haring & Eaton,
1978) is an intervention heuristic that could
guide intervention development for writing
problems. In using the learning hierarchy, in-
terventions are identified by matching student
skill with one of four phases of student learn-
ing (Haring & Eaton, 1978). First, interven-
tions focus on skill accuracy (the acquisition
phase) through high modeling and frequent
cueing. Next, interventions are targeted to en-
hance the speed with which the skill is per-
formed (fluency phase) through additional
practice and contingent reinforcement. Once a
student can accurately and fluently exhibit the
skill, efforts can focus on the later phases of
maintenance and generalization.
Although research supports the learning
hierarchy as an intervention heuristic (Burns,
Codding, Boice, & Lukito, 2010), it remains
unknown as to when the intervention focus
should change. The instructional level is a
potential criterion that could be used to iden-
tify whether the intervention should focus on
acquisition, fluency, or maintenance/general-
ization. Gickling and Armstrong (1978) oper-
ationally defined the instructional level for
reading as material in which the student could
read 93%– 97% of the words. Reading less
than 93% of the words represented a frustra-
tion level and exceeding 97% was an indepen-
dent level. Researchers have found that task
completion, task comprehension, and time on
task increased when instructional level mate-
rial was used (Gickling & Armstrong; Trep-
tow, Burns, & McComas, 2007). Moreover,
students provided with an acquisition inter-
vention (i.e., high modeling and cueing) to
facilitate an instructional level in reading ex-
perienced increased and sustained growth over
a period of 15 weeks compared to a randomly
assigned control group (Burns, 2007).
Math intervention research also supports
the instructional level as a decision-making
criterion. Burns, VanDerHeyden, and Jiban
(2006) empirically derived instructional-level
criteria for math by computing slopes of
growth and finding the mean baseline score for
students with the highest growth. Students
who did not experience high growth rates may
have (a) demonstrated proficient skills before
Determining an Instructional Level for Writing
instruction (i.e., an independent level) and had
little room to grow, or (b) started too low (i.e.,
at a frustration level) and the instruction did
not adequately address their learning needs
(Burns et al., 2006). Analysis of a subset of the
data used for validation showed that students
who had initial math scores within the instruc-
tional level made the greatest gains over time
(Burns et al. ). Recent meta-analytic research
that used the Burns et al. (2006) instructional-
level criteria found stronger effects for acqui-
sition interventions (modeling and cueing)
provided for students with frustration-level
skill than for those whose baseline perfor-
mance represented an instructional level
(Burns et al., 2010).
Identifying an instructional level for
early writing skills could help interventionists
determine the type of intervention struggling
writers need (e.g., modeling vs. practice). To
determine a student’s instructional level in
other academic domains, the interventionist
assesses student skill within a specific domain
by sampling the behavior for a short time
period (e.g., 1– 4 min) from a specific instruc-
tional stimulus (e.g., a passage that will be
read as part of instruction, or a single-skill
math probe such as single-digit multiplica-
tion), and then computes either the percentage
of the words read correctly for reading or the
rate at which the skill was completed for math
(e.g., digits correct per minute). The resulting
data are sufficiently reliable for instructional
decisions in reading and math (Burns et al.,
2000, 2006), but less is known about the ad-
equacy of assessments for instructional deci-
sion making in writing.
One assessment approach that has
shown promise for yielding technically sound
data for instructional decision making in writ-
ing is curriculum-based measurement (CBM;
Deno, Mirkin, & Marston, 1982), which might
serve as an approach for identifying students’
instructional level in writing. CBM for writing
typically consists of brief prompts to which
students respond for 3–5 min that are scored
for the number of words written (WW), words
spelled correctly (WSC), or correct word se-
quences (CWS; Videen, Deno, & Marston,
1982). Several CBM options for assessing stu-
159
School Psychology Review, 2011, Volume 40, No. 1
dent progress provide information about early
writing skills that are important for later writ-
ing development (Graham et al., 1997). For
example, McMaster, Du, & Petusdottir (2009)
found CBM using sentence copying or pic-
tures with words produced promising techni-
cal characteristics, and subsequent research
showed these measures were sensitive to
growth over time (McMaster et al., 2011).
Whereas previous research has estab-
lished that early writing CBM is promising for
measuring progress in early writing, less is
known about which measures and assessment
procedures can be used to directly inform in-
tervention. Given that acquisition interven-
tions at the instructional level correspond to
positive learning outcomes in other academic
domains, research is needed to determine
whether an instructional level can be identified
for writing. The purpose of the current study
was to replicate Burns and colleagues’ (2006)
study of instructional level in math by apply-
ing their methods to writing. Specifically, we
examined the reliability and criterion validity
of potential instructional-level estimates for
beginning writing based on different types of
prompts and scoring procedures. The follow-
ing research questions guided the study: (a) To
what extent do different prompts and scoring
procedures affect the reliability of writing as-
sessment data? (b) To what extent do different
prompts and scoring procedures affect esti-
mates of the criterion validity of writing as-
sessment data? (c) What mean initial writing
scores are linked to the highest rates of growth
during weekly progress monitoring of writing?
(d) How reliable are instructional-level cate-
gories based on empirically derived criteria?
(e) How well do instructional-level categories
relate to a standardized measure of writing?
Method
Setting and Participants
Data for this study were drawn from a
larger study of the technical features of slopes
produced from weekly administered early
writing CBM prompts (McMaster et al.,
2011). The larger study was conducted in a
Midwestern urban school district with five
160
first-grade classrooms from two schools that
were selected by convenience sampling. As
described in the larger study, all consenting
students participated, resulting in 85 students
(51% male). Forty-one percent were White,
28% Black, 26% Hispanic, 2% Native Amer-
ican, and 2% Asian American. Fifty-seven
percent were eligible for the federal free or
reduced-price lunch program, 21% were
English learners, and 17% received special
education.
Measures
CBM tasks. Participants in the larger
study completed several CBM tasks on a
weekly basis for 12 weeks. In the present
study, we examined data from two tasks com-
pleted each week that were informed by the
current understanding of early writing devel-
opment (i.e., that transcription skills play a
critical role in writing development; Graham
et al., 1997) and examined for use with first-
grade students (McMaster et al., 2009). Sen-
tence Copying consisted of packets of eight
pages, with three sentences on each page. Par-
ticipants were instructed to copy an example
sentence at the top of the first page (e.g., “We
have one cat.”). Then, they were instructed to
copy the remaining sentences and to stop
after 3 min. Alternate-form reliability on Sen-
tence Copying using the current scoring pro-
cedures ranged from r ⫽ .63 to .80, and
criterion validity with a standardized norm-
referenced writing measure ranged from r ⫽
.23 to .50 (McMaster et al., 2011).
Picture-Word prompts consisted of
words with a picture above each word. Partic-
ipants wrote a sentence using the word pro-
vided. Before the task, the examiner drew a
picture (e.g., a tree) on the board and wrote the
word underneath. Then, the examiner asked
the students to generate sentences using the
word. After allowing the students to practice,
the examiner instructed them to write as many
sentences as they could using the words and
pictures on their probes. After 3 min, the ex-
aminer instructed participants to stop. Alter-
nate-form reliability on Picture Word using
the current scoring procedures ranged from

r ⫽ .70 to .77, and criterion validity ranged
from r ⫽ .23 to .54 (McMaster et al., 2011).
Writing samples were scored using
words written, words spelled correctly, and
correct word sequences. A word was defined
as at least two letters written in sequence, or
single-letter words such as “I” and “a” (Deno,
Mirkin, & Marston, 1980). Words were
judged as spelled correctly by scoring them as
a computer would score them (i.e., syntax and
semantics were not taken into account), and
were computed by subtracting the number of
all words that were judged as being spelled
incorrectly from the total words written. A
correct word sequence was defined as any two
adjacent, correctly spelled words that are ac-
ceptable within the context of the sample to a
native speaker of English (Videen et al.,
1982).
Test of Written Language—3rd Edi-
tion (TOWL-3). The TOWL-3 (Hammill &
Larsen, 1996) is a comprehensive test of writ-
ten language designed for students from 7
years to 17 years 11 months of age. The Spon-
taneous Writing subtest (Form A) was group
administered to all participants. Students were
presented with a picture of astronauts, space
ships, and construction activity; asked to plan
a story about the picture; and then write as
much as they could in 15 min. Writing sam-
ples were scored based on Contextual Conven-
tions (capitalization, punctuation, and spell-
ing), Contextual Language (quality of vocab-
ulary, sentence construction, and grammar),
and Story Construction (e.g., quality of plot,
prose, character development, and interest).
Alternate-form reliabilities for Spontaneous
Writing for 7-year-olds ranged from r ⫽ .60 to
.87, and the measure is reported to correlate
well with other standardized writing measures
(r ⫽ .50; Hammill & Larsen, 1996).
Procedures
All writing prompts were administered
by a trained graduate student during Week 1
and by the classroom teachers thereafter. Fi-
delity observations indicated that measures
were administered with high levels of accu-
racy. Scorers included the first and second
Determining an Instructional Level for Writing
authors, four additional graduate research as-
sistants (all special education or school psy-
chology students), and one special education
teacher. All scorers had scoring experience on
previous CBM projects and received training
for this project. Mean interrater agreement
was 95% for both CBM prompt types and 88%
for the TOWL-3 Spontaneous Writing score.
See McMaster et al. (2011) for a complete
description of prompt administration, scoring,
and interrater agreement.
Data Analyses
The first step was to compute reliability
coefficients for both fluency and accuracy
metrics of writing performance. Fluency met-
rics consisted of the total scores students re-
ceived for the two prompt types using WW,
WSC, and CWS (e.g., 25 CWS on the Picture-
Word prompt). Accuracy metrics were com-
puted for the two prompt types by dividing the
total number of WSC by the total number of
words written, and the total number of CWS
by the total number of word sequences. For
example, a student who produced 25 WSC out
of a total of 30 words on Sentence Copying
would have an accuracy score of 83%. Flu-
ency and accuracy scores for Weeks 2 and 3
were then correlated using Pearson product
moment correlation coefficients (the first week
of data were omitted because of potential task
novelty). Next, data that produced acceptable
reliability coefficients were evaluated for cri-
terion-related validity by correlating Sentence
Copying and Picture-Word scores at Week 2
with the age-based total of the TOWL-3 stan-
dard scores using a Pearson product moment
correlation.
Fluency data were subsequently con-
verted to categories of frustration, instruc-
tional, and independent levels. To make these
conversions, slopes of growth over Weeks
2–12 of progress monitoring were computed
using ordinary least-squares regression, which
represented average weekly growth in WW,
WSC, and CWS for each student. Next, we
identified students whose slopes of growth
equaled or exceeded the 66th percentile of all
the slopes within the sample (as in Burns et al.,
161
School Psychology Review, 2011, Volume 40, No. 1

Table 1
Means, Standard Deviations, and Correlation Coefficients for Fluency and
Accuracy Scores for Sentence Copy and Picture-Word Prompts and
Accompanying Scoring Procedures
Fluency Accuracy

Prompt Probe 2 Probe 3 Probe 2 Probe 3

Procedure M SD M SD r M SD M SD r

Picture-Word
Words written 17.0 8.4 18.4 8.6 .71*
Words spelled correctly 13.4 7.7 15.0 8.6 .67* 76.1 23.4 77.8 24.6 .52*
Correct word sequences 11.9 8.6 13.1 9.1 .67* 54.2 28.6 55.9 26.5 .46*
Sentence Copy
Words written 16.7 7.1 16.8 7.7 .71*
Words spelled correctly 12.8 6.6 13.3 7.1 .74* 74.6 25.1 78.8 19.7 .60*
Correct word sequences 11.9 7.3 12.6 8.1 .70* 59.8 29.6 64.6 26.7 .56*

*p ⬍ .01.

2006). Finally, the mean score on each probe
type among the group of students with high
rates of growth was considered an estimate of
an instructional level for beginning writing.
Categories were subsequently created by es-
tablishing a range for the instructional-level
scores, which was determined by computing
the standard error (SE) of the mean. Scores
that exceeded the mean by two SEs were con-
sidered an independent level and those that fell
more than two SEs below the mean were at a
frustration level. This process was repeated for
the data from Week 3, and kappa coefficients
were computed for agreement between the
data from each week. The criterion-related
validity of the categorical data were then eval-
uated by computing Spearman rho correlation
coefficients between the total standard scores
on the TOWL-3 and the categories produced
by Week 2 data for each of the scoring pro-
cedures and prompts.
Results
To address the first research question
regarding reliability of prompts and scoring
procedures, delayed alternate-form reliability
coefficients for each scoring procedure for
162
each prompt were computed for both fluency
and accuracy metrics, and are reported in Ta-
ble 1. Correlation coefficients for scores on the
fluency metric approached or exceeded .70 for
each scoring procedure under both prompt
types, whereas the coefficients for scores on
the accuracy metrics were at or below .60,
suggesting less reliable scores for accuracy
measures. Reliability coefficients that are at or
above .70 are acceptable for programs of re-
search that are in early stages (Nunnally &
Bernstein, 1994), but coefficients below .70
are less interpretable. For that reason, subse-
quent analyses excluded the accuracy data and
included only the fluency measures.
To address the research question regard-
ing the criterion validity of prompts and scor-
ing procedures, the fluency scores for each
procedure and prompt were correlated with the
TOWL-3 Spontaneous Writing standard score
total. The results are presented in the second
column of Table 2. With the exception of WW
for Sentence Copying prompts, each of the
scoring procedures was significantly corre-
lated, using the more conservative alpha level
of .01, with the TOWL-3 total standard scores.
Significant correlations ranged from r ⫽ .42
 Determining an Instructional Level for Writing

Table 2
Criterion-Related Validity
Coefficients Between Scoring
Procedures for Each Prompt and the
TOWL-3 Total Score
Table 3
Derivation of and Estimates for
Fluency Instructional-Level Criteria
for Scoring Procedures Within
Prompt Types

Correlation with Fluency

TOWL-3 Total Criteria
Prompt
(3-min
Fluency Procedures Mean SD SE Probe)
Prompt Raw Category
Picture-Word
Procedure r ␳ Words written 14.46 8.68 1.64 11–18
Words spelled
Picture-Word correctly 11.43 7.03 1.33 9–14
Words written .32* .36* Correct word
Words spelled correctly .48* .46* sequences 10.93 8.56 1.62 8–14
Correct word sequences .52* .50* Sentence Copy
Sentence Copy Words written 16.25 6.58 1.24 14–19
Words written .26 .21 Words spelled
Words spelled correctly .42* .46* correctly 13.39 6.52 1.23 11–16
Correct word sequences .46* .48* Correct word
sequences 13.32 7.86 1.49 10–16
Note. TOWL-3 ⫽ Test of Written Language—3rd Edi-
tion.
*p ⬍ .01.

CWS, and .90 for WSC. Instructional-level

for WSC on the Sentence Copying prompt to
r ⫽ .52 for CWS on the Picture-Word prompt.
Validity was further explored by examining
concurrent (i.e., within Week 2) and predictive
(i.e., between Weeks 2 and 3) correlations of
the same scoring procedures across Sentence
Copying and Picture-Word prompts. All coef-
ficients for these analyses were significant and
moderate (r ⫽ .42 and r ⫽ .69).
The third research question addressed
whether an instructional level could be esti-
mated for beginning writing. Given the ac-
ceptable technical adequacy of the fluency-
based measures, instructional-level estimates
were computed for each of the scoring proce-
dures and prompts. These estimates are based
on the initial performance of the students
whose slopes were at or above the 66th per-
centile. The slopes that represented the 66th
percentile for the Picture-Word prompts were
.87 for WW, .96 for CWS, and .84 for WSC.
For Sentence Copying, slopes that represented
the 66th percentile were .97 for WW, .96 for
criteria were computed by finding the mean
score from Weeks 2 and 3 and building a two
SE range around those data (see Table 3).
Twenty-eight (34.6%) of the students were
classified as high responders using each scor-
ing procedure for both prompt types.
To answer the fourth research question,
the fluency writing data were converted to the
categories of frustration, instructional, and in-
dependent levels. Reliability of the categorical
data was estimated using Cohen’s (1960)
kappa coefficient for chance agreement. The
number and percentage of students scoring in
the frustration, instructional, and independent
levels of difficulty are presented in Table 4
along with the kappa coefficients. Kappa co-
efficients for the categories using Sentence
Copying data were all .46, and using Picture-
Word data ranged from .37 (WW) to .47
(CWS). This indicates that the categorical
agreements across Weeks 2 and 3 were 46%
above chance for the scoring procedures for
Sentence Copying data and 37% to 47% above
chance for the Picture-Word data.
163
School Psychology Review, 2011, Volume 40, No. 1

Table 4
Number and Percentage of Fluency Scores Categorized as Frustration,
Instructional, and Independent and Kappa Coefficients

Probe 2 Probe 3

Prompt Frustration Instructional Independent Frustration Instructional Independent

Procedure N % N % N % N % N % N % ␬

Picture-Word
Words written 19 23.8 19 23.8 42 52.5 21 25.3 15 18.1 47 56.6 .46*
Words spelled
correctly 24 30.0 18 22.5 38 47.5 23 27.7 12 14.5 48 57.8 .46*
Correct word
sequences 30 37.5 20 25.0 30 37.5 29 34.9 17 20.5 37 44.6 .46*
Sentence Copy
Words written 24 29.6 30 37.0 27 33.3 23 28.8 24 30.0 33 41.2 .37*
Words spelled
correctly 29 35.8 24 29.6 28 34.6 29 36.2 28 35.0 23 28.8 .46*
Correct word
sequences 32 39.5 25 30.9 24 29.6 31 38.8 20 25.0 29 36.2 .47*

*p ⬍ .01.

Finally, to answer the fifth research
question, the categorical data from Week 2
were correlated with the TOWL-3 standard
score total using Spearman’s rho, presented in
the third column of Table 2. As with the
continuous fluency data, the correlations were
all significant ( p ⬍ .01) with CWS resulting in
the correlation coefficients at or near .50 for
the Picture-Word (␳ ⫽ .50), and Sentence
Copying prompts (␳ ⫽ .48).
Discussion
Results of this study suggest that WSC
and CWS for Sentence Copying, as well as all
scoring procedures for the Picture-Word
prompts, were sufficiently reliable and statis-
tically significantly correlated with a standard-
ized measure of writing. Similar to previous
research in math (Burns et al., 2006), accept-
able coefficients were found for fluency but
not accuracy scores. Overall, correlations were
similar to reliability and validity findings from
previous research on CBM for beginning writ-
ers (Coker & Ritchey, 2010; McMaster et al.,
164
2009). Our findings also suggest that these
data can be converted to instructional-level
categories that are reliable and significantly
related to scores on a standardized measure of
writing.
Although these findings provide a
method to assess the instructional level for
early writers, discussion is warranted with re-
gard to how the instructional-level concept
applies to writing. In keeping with the learning
hierarchy (Haring & Eaton, 1978), frustration-
level scores might indicate that a student re-
quires additional modeling in the skill being
taught, but scores within the instructional level
might suggest that the student is ready for
fluency-building activities in the skill. In read-
ing and math, it is possible to manipulate the
difficulty of the material, whereas the task for
writing is to produce an entirely new product
on blank paper. Thus, the instructional level
for writing may be more dependent on the
skills of the learner than the required task
difficulty. This study was the first step in a line
of inquiry designed to understand the instruc-

tional level as related to writing, and addi-
tional instructional implications should be de-
termined with future research. For example, as
currently conceptualized, the instructional
level might serve as a heuristic for determin-
ing at what stage of the instructional hierarchy
(Haring & Eaton, 1978) a student’s skills fall,
which suggests that future research examine
the effectiveness of early writing interventions
in a way that is similar to that done with math
(Burns et al., 2010). Moreover, other types of
CBM prompts, such as spelling (a critical
component of transcription; Berninger & Amt-
mann, 2003), might offer a curricular domain
in which the difficulty of the material could
drive application of the instructional level in a
way similar to how reading material is manip-
ulated to correspond to students’ instructional
levels (Gickling & Armstrong, 1978).
The findings and implications of this
study should be considered in light of the
limitations of the data. First, the current sam-
ple of students came from two schools in a
single district with a relatively high proportion
(21%) of students who were classified as Eng-
lish language learners. Thus, the degree to
which the results would apply to students in
other locations with other characteristics and
curricula is unknown, and the overall rates of
growth from which instructional levels were
computed may have been affected by growth
characteristics that are specific to English lan-
guage learner students. Moreover, this study
included only first-graders, and previous re-
search in math found different instructional
levels across grades (Burns et al., 2006).
A second limitation is the somewhat arbi-
trary designation of high and low growth rates
used to determine instructional level. The current
procedure used the top third of all slopes to
identify students at the instructional level be-
cause that was the approach used in previous
research (Burns et al., 2006). Other approaches
could be selected, such as the median slope (cf.
Vellutino et al., 1996), and might lead to differ-
ent estimates of instructional-level ranges. Fur-
ther research could be conducted to identify cut-
offs that yield the most precise instructional level
estimates. Moreover, different scoring proce-
dures yielded different numbers of students in
Determining an Instructional Level for Writing
each of the instructional-level categories. For
instance, for the Picture-Word prompt types,
WW, WSC, and CWS, resulted in 19, 24, and 30
students, respectively, at the frustration level.
These scoring procedures assess different skills,
and thus future research would be necessary to
identify which of the procedures resulted in the
most useful instructional-level criteria. Related
to this point, the criteria derived in this study
suggest three categories in which students’ skills
can fall, but such a fine-grained evaluation of
student skills may not be necessary for early
writing, and perhaps a dichotomous interpreta-
tion of the instructional level (i.e., below and at
or above) would sufficiently suggest the need for
different interventions.
Third, whereas kappa correlations calcu-
lated to determine the reliability of the cate-
gorical data were statistically significant, co-
efficients were modest. It is possible that stu-
dents who were at the upper limits of the
instructional-level criteria made sufficient
gains between assessments such that they
moved to the independent level; this possibil-
ity could be explored in future studies by
conducting direct evaluations of classification
accuracy. Further research is also needed to
determine whether other indices of student
writing yield more reliable classifications. Re-
lated to this point, whereas criterion-related
validity coefficients were statistically signifi-
cant and similar to criterion-validity coeffi-
cients generally found in the writing literature
(cf. McMaster & Espin, 2007), these coeffi-
cients are not strong and thus prevent firm
conclusions regarding criterion validity. This
problem is not new to writing assessment re-
search, given the complex, multidimensional
nature of writing (cf. Tindal & Hasbrouk,
1991). Because of these limitations in techni-
cal adequacy of writing measures, future re-
searchers may wish to continue to investigate
various writing tasks, durations, and scoring
procedures to identify more precise estimates
of writing skill. One direction for future re-
search might investigate the influence of the
amount of time allowed for completion on
fluency-based assessments on writing, and
whether increasing the duration of writing
165
School Psychology Review, 2011, Volume 40, No. 1
samples would yield more technically sound
and instructionally useful data.
In light of the above limitations, the
current study should primarily be used as a
heuristic for empirically deriving instructional
levels for writing; further work is needed to
determine whether using CBM, or perhaps
other skill-based writing assessments, to iden-
tify instructional levels for writing is an in-
structionally useful approach. To validate the
utility of identifying an instructional level in
writing, instruction or intervention would need
to be manipulated to occur within and outside
of the instructional-level ranges suggested by
this study. Results that indicated the strongest
outcomes for those students taught at their
instructional level would support the applica-
bility of instructional level to writing. The
current design did not permit this analysis, but
the results suggest the value of future efforts to
do so.
References
Baker, S., Gersten, R., & Graham, S. (2003). Teaching
expressive writing to students with learning disabili-
ties: Research-based applications and examples. Jour-
nal of Learning Disabilities, 36, 109 –123.
Berninger, V., & Amtmann, D. (2003). Preventing written
expression disabilities through early and continuing
assessment and intervention for handwriting and/or
spelling problems: Research into practice. In H. L.
Swanson, K. Harris, & S. Graham (Eds.), Handbook of
learning disabilities (pp. 345–363). New York Guil-
ford Press.
Berninger, V. W., Nielsen, K. H., Abbott, R. D., Wijsman,
E., & Raskind, W. (2008). Writing problems in devel-
opmental dyslexia: Under-recognized and under-
treated. Journal of School Psychology, 46, 1–21.
Burns, M. K. (2007). Reading at the instructional level
with children identified as learning disabled: Potential
implications for response-to-intervention. School Psy-
chology Quarterly, 22, 297–313.
Burns, M. K., Codding, R. S., Boice, C. H., & Lukito, G.
(2010). Meta-analysis of acquisition and fluency math
interventions with instructional and frustration level
skills: Evidence for a skill by treatment interaction.
School Psychology Review, 39, 69 – 83.
Burns, M. K., Tucker, J. A., Frame, J., Foley, S., &
Hauser, A. (2000). Interscorer, alternate-form, internal
consistency, and test-retest reliability of Gickling’s
model of curriculum-based assessment for reading.
Journal of Psychoeducational Assessment, 18, 353–
360.
Burns, M. K., VanDerHeyden, A. M., & Jiban, C. L.
(2006). Assessing the instructional level for mathemat-
ics: A comparison of methods. School Psychology Re-
view, 35, 401– 418.
166
Cohen, J. (1960). A coefficient of agreement for nominal
scales. Educational and Psychological Measurement,
20, 37– 46.
Coker, D. L., & Ritchey, K., D. (2010). Curriculum based
measurement of writing in kindergarten and first grade:
An investigation of production and qualitative scores.
Exceptional Children, 76, 175–193.
Deno, S. L., Mirkin, P., & & Marston, D. (1982). Valid
measurement procedures for continuous evaluation of
written expression. Exceptional Children Special Ed-
ucation and Pediatrics: A New Relationship, 48, 368 –
371.
Deno, S. L., Mirkin, P., & Marston, D. (1980). Relation-
ships among simple measures of written expression
and performance on standardized achievement tests
(Vol. IRLD-RR-22, pp. 109). University of Minnesota,
Institute for Research on Learning Disabilities.
Gickling, E. E., & Armstrong, D. L. (1978). Levels of
instructional difficulty as related to on-task behavior,
task completion, and comprehension. Journal of
Learning Disabilities, 11, 559 –566.
Graham, S., Berninger, V. W., Abbott, R. D., Abbott,
S. P., & Whitaker, D. (1997). Role of mechanics in
composing of elementary school students: A new
methodological approach. Journal of Educational Psy-
chology, 89, 170 –182.
Graham, S., & Perin, D. (2007). A meta-analysis of writ-
ing instruction for adolescent students. Journal of Ed-
ucational Psychology, 99(3), 445– 476.
Hammill, D. D., & Larsen, S. C. (1996). Test of Written
Language—Third Edition. Austin, TX: PRO-ED, Inc.
Haring, N. G., & Eaton, M. D. (1978). Systematic instruc-
tional technology: An instructional hierarchy. In N. G.
Haring, T. C. Lovitt, M. D. Eaton, & C. L. Hansen
(Eds.), The fourth R: Research in the classroom (pp.
23– 40). Columbus, OH: Merrill.
McMaster, K. L., Du, X., & Petursdottir, A. (2009). Tech-
nical features of curriculum-based measures for begin-
ning writers. Journal of Learning Disabilities, 42, 41–
60.
McMaster, K. L., Du, X., Yeo, S., Deno, S. L., Parker, D.,
& Ellis, T. (2011). Curriculum-based measures of be-
ginning writing: Technical features of the slope. Ex-
ceptional Children, 77, 185–206.
McMaster, K. L., & Espin, C. A. (2007). Technical fea-
tures of curriculum-based measurement in writing: A
literature review. Journal of Special Education, 41,
68 – 84.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric
theory (3rd ed.). New York: McGraw-Hill.
Salahu-Din, D., Persky, H., & Miller, J. (2008). The
Nation’s Report Card: Writing 2007. Retrieved De-
cember 1, 2009, from http://nces.ed.gov/nationsreport-
card/writing/
Tindal, G., & Hasbrouck, J. (1991). Analyzing student
writing to develop instructional strategies. Learning
Disabilities Research and Practice, 6, 237–245.
Treptow, M. A., Burns, M. K., & McComas, J. J. (2007).
Reading at the frustration, instructional, and indepen-
dent levels: Effects on student time on-task and com-
prehension School Psychology Review, 36, 159 –166.
Vellutino, F. R., Scanlon, D. M., Sipay, E. R., Small,
S. G., Chen, R., Pratt, A., et al. (1996). Cognitive
profiles of difficult-to-remediate and readily remedi-
ated poor readers: Early intervention as a vehicle for
distinguishing between cognitive and experiential def-
 Determining an Instructional Level for Writing

icits as basic causes of specific reading disability. Jour- Date Received: November 12, 2009
nal of Educational Psychology, 88, 601– 638. Date Accepted: October 25, 2010
Videen, J., Deno, S. L., & Marston, D. (1982). Correct
word sequences: A valid indicator of proficiency in Action Editor: Tanya Eckert 䡲
written expression (Vol. IRLD-RR-84, pp. 61). Min-
neapolis: University of Minnesota, Institute for Re-
search on Learning Disabilities. Article accepted by previous Editor.

David C. Parker is a doctoral candidate in the school psychology program within the
Department of Educational Psychology at the University of Minnesota. His research
interests include direct assessment and intervention for reading and writing, particularly
in the primary elementary grades.

Kristen L. McMaster, Ph.D. is an Associate Professor of Special Education in the

Department of Educational Psychology, University of Minnesota. Her research interests
include (a) promoting teachers’ use of data-based decision making and evidence-based
instruction and (b) developing individualized interventions for students at risk for or
identified with reading and writing-related disabilities.

Matthew K. Burns, Ph.D. is a Professor of Educational Psychology, Coordinator of the

School Psychology Program, and Co-Director of the Minnesota Center for Reading
Research at the University of Minnesota. His current research interests include curricu-
lum-based assessment for instructional design, interventions for academic problems,
problem-solving teams, and incorporating all of these into a response-to-intervention
model.

167
Copyright of School Psychology Review is the property of National Association of School Psychologists and its
content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's
express written permission. However, users may print, download, or email articles for individual use.