670637

research-article2016
SEDXXX10.1177/0022466916670637The Journal of Special EducationRomig et al.

Article
The Journal of Special Education

Meta-Analysis of Criterion Validity

2017, Vol. 51(2) 72­–82
© Hammill Institute on Disabilities 2016
Reprints and permissions:
for Curriculum-Based Measurement sagepub.com/journalsPermissions.nav
DOI: 10.1177/0022466916670637

in Written Language journalofspecialeducation.sagepub.com

John Elwood Romig, MEd1, William J. Therrien, PhD1, and

John W. Lloyd, PhD1

Abstract
We used meta-analysis to examine the criterion validity of four scoring procedures used in curriculum-based measurement
of written language. A total of 22 articles representing 21 studies (N = 21) met the inclusion criteria. Results indicated
that two scoring procedures, correct word sequences and correct minus incorrect sequences, have acceptable criterion
validity with commercially developed and state- or locally developed criterion assessments. Results indicated trends for
scoring procedures at each grade level. Implications for researchers and practitioners are discussed.

Keywords
curriculum-based measurement, meta-analysis, assessment, writing, written language

Since the passage of the Education for All Handicapped
Children Act (EAHCA) in 1975, Individual Education Plans
(IEPs) and annual goals for students with disabilities have
been a hallmark of special education. Schools are required
to report progress toward these goals at least annually. The
2004 reauthorization of EAHCA, renamed the Individuals
With Disabilities Education Improvement Act, required that
a plan for monitoring progress toward annual goals be
included in a student’s IEP and that reports of progress be
given to parents at least as often as peers not receiving spe-
cial education services receive reports of their academic
progress (i.e., report cards; Individuals With Disabilities
Education Act, 2004).
Educators monitor many aspects of students’ educational
activities. They may be concerned about progress toward
academic outcomes or social-behavioral goals (Archer &
Hughes, 2010; Epstein, Atkins, Cullinan, Kutash, & Weaver,
2008). Within the academic domain, some teachers employ
curriculum-based assessments (e.g., Venn, 2013), but cur-
riculum-based measurement is probably the most widely
employed method for monitoring progress and making edu-
cational decisions (Lembke, Hampton, & Hendricker,
2013). Curriculum-based measures have been employed to
assess diverse aspects of reading, arithmetic, spelling, and
writing (Hosp, Hosp, & Howell, 2012).
As with any measurement system, curriculum-based
measures should meet standards for technical adequacy.
They should be reliable and valid. In this study, we examine
aspects of the technical adequacy of a progress monitoring
measures for written language. After a brief description of
the various measures used to monitor students’ writing per-
formance, we discuss the research about psychometric qual-
ities of measures of writing.
Written Language
Assessing written language is complicated by the multi-
faceted nature of the skill. Although there is some disagree-
ment as to the exact components in writing, there is general
consensus about five basic elements: handwriting, spelling,
mechanics, usage, and ideation (Taylor, 2006). Often
experts emphasize some of these elements of writing more
than other elements. For example, Graham, Harris, and
Hebert (2011) argued that presentation effects (e.g., spell-
ing, handwriting, and grammar) should not overly factor
into judgments of writing for students with disabilities.
Espin (2014), however, argued that writing instruction and
assessment should focus on effectively communicating a
message to readers, which includes skills such as grammar,
spelling, and punctuation.
Cutler and Graham (2008) conducted a national survey
of primary grade teachers and found that approximately
70% of teachers reported monitoring student writing at least
1
University of Virginia, Charlottesville, USA
Corresponding Author:
John Elwood Romig, Department of Curriculum, Instruction, and Special
Education, University of Virginia, P.O. Box 400273, Charlottesville,
VA 22904, USA.
E-mail: romig@virginia.edu
Romig et al.
weekly. Although it is encouraging that a large majority of
teachers self-reported assessing student writing weekly, it is
concerning that almost a third of teachers are not assessing
student writing on a weekly basis.
The lack of frequency in assessing writing is understand-
able given the time required for assessing writing. Teachers
can spend a significant amount of time assessing writing due
to the fact that for many components of writing, there is not
a “right answer.” Ideation, for example, is largely subjective
and requires the teacher to assess the ideas developed by the
student in relation to the purpose for writing. These judg-
ments take significant time for the teacher to make for one
student, and the time demands are compounded when moni-
toring an entire class. These time demands increase as the
students age and their writing becomes more complex.
Curriculum-Based Measurement
Curriculum-based measurement (CBM) can decrease the
time necessary to monitor writing progress. Deno (1985;
Deno, Mirkin, & Marston, 1980) led the initial development
of CBM with the goal of establishing a set of measures that
were technically adequate (e.g., reliable and valid), efficient
to administer and score, inexpensive, sensitive to growth,
and easily understood. His efforts established procedures
for administering and scoring CBMs in reading, mathemat-
ics, written language, and spelling (Deno, 1985).
CBM for written language (CBM-W) involves adminis-
tration of a series of probes of equivalent difficulty. Each
probe includes a prompt (e.g., picture prompt, story starter,
narrative prompt, essay prompt) and a timed response from
the student (e.g., 3, 5, 7, or 10 min). Dozens of scoring pro-
cedures have been developed and tested by researchers (see
Amato & Watkins, 2011; Gansle, Noell, VanDerHeyden,
Naquin, & Slider, 2002; and Jewell & Malecki, 2005, for a
variety of alternative measures), but four have emerged as
frequently researched and used in the classroom.
Words Written (WW)
WW was one of the CBM-W scoring indices developed by
Deno et al. (1980). In WW, the scorer simply counts the
number of words in the student’s writing sample. Words do
not have to be spelled correctly or used in correct grammati-
cal function. Any two conjoined letters are counted as a
word. Single letter words (e.g., “I” and “a”) are also counted.
Words Spelled Correctly (WSC)
WSC was also originally developed by Deno et al. (1980).
This scoring procedure involves counting WSC in the writ-
ing sample regardless of the appropriateness of usage (much
like a computer spell-checker would assess correctly spelled
words).
73
Correct Word Sequences (CWSs)
CWS was developed by Videen, Deno, and Marston (1982).
This scoring procedure involves counting sequences of cor-
rect words. A CWS is defined as two adjacent words that are
both spelled correctly and used appropriately in a sentence.
The first scored sequence in a sample is from “blank-to-
first-word.” The last scored sequence is from “last-word-to-
blank.” In this scoring procedure, there will always be one
more possible sequence than words in the writing sample.
Correct Minus Incorrect Word Sequences
(CIWSs)
CIWS was developed by Espin et al. (2000). The rules for
this scoring procedure follow the rules for CWS. In addi-
tion, the number of incorrect word sequences are counted
and subtracted from the total CWS. The difference between
CWS and incorrect word sequences is the student’s score
for the writing sample.
Technical Adequacy
As with any worthwhile measure, if CBM-W is to be used
as a progress monitoring tool, it must meet certain technical
requirements of reliability and validity. Previous research in
CBM-W has focused specifically on criterion validity and
alternate form reliability. Criterion validity assesses the
degree to which a measure correlates with another measure
(Taylor, 2006). Although reliability of a test is certainly
important and necessary, a measure that produces accurate
but invalid scores is meaningless.
Establishing criterion validity of a measure in writing is
difficult due to disagreement in the field about how much
weight should be placed on various elements of writing.
Researchers have typically reported lower criterion validity
for writing measures than reading measures. For example,
researchers have reported criterion validity coefficients of .85
or higher for the Woodcock Reading Mastery Test–Revised
with the Woodcock–Johnson Psychoeducational Battery as a
criterion measure (Taylor, 2006). In contrast, researchers have
reported criterion validity coefficients of .30 to .50 for the Test
of Written Language–3 with the Comprehensive Test of
Nonverbal Intelligence as a criterion measure (Taylor, 2006).
For CBM measures in reading, researchers have reported cri-
terion validity coefficients that reach .70 to .80 and even
approach .90 at times (Wayman, Wallace, Wiley, Tichá, &
Espin, 2007). Given the criterion validity of standardized
writing measures in comparison with standardized reading
measures and the difficulty defining good writing, CBM mea-
sures in written language would be expected to have lower
criterion validity than CBM measures in reading.
Another way of determining the validity of a measure is
to consider the consequences of decision making with and
74
without the measure (Messick, 1989). Researchers and
practitioners do not have many options for progress moni-
toring tools in writing that are sensitive to growth, techni-
cally adequate, and efficient to administer and score. A
lack of measures does not justify using measures with a
dearth of evidence supporting their technical properties.
However, researchers and practitioners might accept mea-
sures with lower criterion validity in writing than they
would in reading due to the numbers of available measures
in each of these areas.
Fuchs (2004) described three stages of research neces-
sary for determining the usefulness of CBM. Stage 1 requires
investigating the technical features of a static score. Stage 2
requires examination of technical features of CBM slopes
(i.e., rates of growth over time). Stage 3 requires examina-
tion of CBM’s usefulness in instruction. Although Stages 2
and 3 are important considerations, Stage 1 is foundational
for CBM-W to be used as a progress monitoring tool.
McMaster and Espin (2007) conducted a comprehensive
literature review of CBM-W studies investigating the reli-
ability and validity of these measures (i.e., Stage 1). Their
review included studies across all K–12 grade levels. Their
review was helpful for beginning to answer questions
related to Stage 1; however, the methods of their review led
to highly variable results for validity and reliability. For
example, criterion validity was reported as low as −.24 and
as high as .99 across the studies. As a literature review, they
did not calculate aggregated statistics across studies.
Instead, they presented and summarized the findings of
each study. The most common scoring procedures across
studies were WW, WSC, CWS, and CIWS. They found cri-
terion validity differed for varying grade levels. Furthermore,
they found that studies conducted after the initial Institute
for Research on Learning Disabilities (IRLD) studies,
which developed CBM, had much lower criterion validity
than the IRLD studies did and that studies that reported
results across grades had higher criterion validity than stud-
ies that reported results within grades.
McMaster, Ritchey, and Lembke (2011) summarized the
existing CBM research for early writers in each of Fuchs’s
(2004) three stages. Of the 11 studies they described, all had
answered Stage 1 questions, five answered Stage 2 ques-
tions, and none answered Stage 3 questions. Similar to
McMaster and Espin (2007), McMaster, Ritchey, and
Lembke (2011) provided a narrative description of studies
and identified strengths and weaknesses without calculating
statistics across studies. The most common scoring proce-
dure in these studies was WW, WSC, and CWS.
Rationale and Research Questions
The purpose of this meta-analysis was to extend the work of
McMaster and Espin (2007) and McMaster, Ritchey, and
Lembke (2011). Specifically, we wanted to increase the
The Journal of Special Education 51(2)
specificity of criterion validity determinations. Calculating a
criterion coefficient in a meta-analytic way has the advantage
of providing a more precise result by giving a criterion valid-
ity coefficient with a confidence interval, rather than a range
of the minimum and maximum coefficients found in all stud-
ies. Mean coefficients across several studies are a more stable
indicator of criterion validity than results from individual
studies. Therefore, this review sought to answer four basic
questions regarding the use of the four most common CBM-W
scoring procedures found in the two previous reviews.
Research Question 1: Overall, what is the mean
weighted criterion validity coefficient of CBM-W scor-
ing procedures?
Research Question 2: What is the mean weighted crite-
rion validity coefficient of WW, WSC, CWS, and CIWS?
Research Question 3: Does a given scoring procedure
have stronger criterion validity at varying grade levels?
Research Question 4: Does the criterion validity of
scoring procedures differ when predicting performance
on commercially developed norm-referenced assess-
ments or state-developed norm-referenced assessments?
Method
Literature Review
We adopted search procedures from McMaster and Espin
(2007) as a model and searched the ERIC, PsycINFO, and
Science Citation Index Expanded databases using the search
terms curriculum based measurement, curriculum based
measure, general outcome measure, and progress monitor-
ing. In addition, we searched the ERIC, Academic Search
Complete, and PsycINFO databases using the search terms
written language and curriculum based measurement. In
total, our searches returned 11,802 results. Using RefWorks,
we removed exact duplicates and had 9,771 results remain-
ing. Before searching the results, two authors reviewed 200
titles and abstracts to ensure agreement with the inclusion
criteria. We had 100% agreement after this initial review of
results. Next, we divided the search results between two
authors and used 2,195 results for intercoder agreement. We
calculated kappa at .67 (with three disagreements) based on
the review of these 2,195 search results. After reviewing titles
and abstracts, 22 met the inclusion criteria for our study.
Inclusion Criteria
Focus of study and design. First, to be included, studies had
to report quantitative measurement of criterion validity
for at least one of the four forms of CBM-W under inves-
tigation (WW, WSC, CWS, CIWS) in a peer-reviewed
journal. We included only peer-reviewed journal articles
due to concerns regarding the rigor of non-peer reviewed
Romig et al.
publications. We also reasoned that strong reports and
dissertations would subsequently be published in peer-
reviewed sources. However, including only peer-reviewed
studies opens the possibility of publication bias affecting
results. Therefore, we conducted a test for publication
bias. Criterion validity could be determined by commer-
cially developed achievement measures (e.g., the Test of
Written Language–3, Woodcock–Johnson III) or a state-
or district-developed achievement measure. We did not
require normative data be available for all state- or locally
developed assessments. Studies that used locally devel-
oped measures had to use a test or rubric developed by the
district and not by individual teachers. We excluded stud-
ies that used only holistic ratings, teacher ratings, devel-
opmental sentence scoring, grade point average (GPA), or
classroom grades. To be included, studies had to provide
information necessary to calculate a mean correlation
(correlation coefficient and sample size).
Participants in the study. We included only K–12 studies. We
did not require student disability status to be reported. Most of
the studies we found included students with and without dis-
abilities and did not report results for these groups separately.
Final corpus. Campbell (2010) was excluded because it only
included students learning English as a second language,
and it examined the utility of passage copying tasks for stu-
dents in ninth to 12th grade. Passage copying is not a typical
writing task for the majority of high school writers. Because
the data for D. C. Parker, McMaster, and Burns (2011) were
drawn from McMaster, Du, et al. (2011), we considered
these two articles as one study. After making decisions
about all articles in our search records, we compared our
results with McMaster and Espin’s (2007) to ensure we did
not neglect any studies that met our criteria.
After excluding and combining studies, 21 studies
remained to form the corpus of this analysis. Often, studies
reported correlations for several grades individually. In
addition, many studies reported correlations with several
criterion measures for each grade level and each scoring
procedure. In total, we examined 739 correlations for this
analysis.
Coding
We assigned codes for scoring procedure, grade level, and
criterion assessment type. We found 44 different types of
CBM-W scoring procedures in these studies. By a wide
margin, the three most common scoring procedures were
WW, WSC, and CWS. Of the 22 articles, 95% (n = 21)
reported criterion validity of WW, 73% (n = 16) reported
criterion validity of WSC, 100% (n = 22) reported criterion
validity of CWS, and 45% (n = 10) reported criterion valid-
ity of CIWS.
75
We initially coded grade level as it was reported in each
study. Grade level ranged from K–12. However, there were
too few studies at some grade levels to make analysis at the
individual grade level sensible, so we grouped studies into
four categories: K–2, third to fifth, sixth to eighth, and ninth
to 12th. Two studies did not fit into these grade categories.
Gansle, VanDerHeyden, Noell, Resetar, and Williams (2006)
included students in second to fifth grades and did not report
correlations for the individual grade levels; we coded this
study as a third to fifth study. Cheng and Rose (2009)
reported results for students in seventh to 12th grades col-
lectively; we coded it as a ninth- to 12th-grade study.
We coded two types of assessments. We distinguished
between commercially developed, norm-referenced assess-
ments and state- or district-developed assessments. Studies
could report total or subscale-subtest results for criterion
measures. If we had questions about whether a commer-
cially developed assessment was norm referenced, we
searched the test manual or the Internet for normative infor-
mation for the given assessment. We excluded the Test of
Emerging Academic English used in Campbell, Espin, and
McMaster (2013) because we were interested in criterion
validity using criterion measures that assess a wide range of
students, not English Language Learners only. Two raters
reviewed all articles, and interrater reliability (agreements/
total of agreements and disagreements) was 97.3%. We rec-
onciled the disagreements as a group and used the recon-
ciled codes in the subsequent analyses.
Selecting Correlations for Analysis
Due to the large number of correlations, we selected the
highest reported correlation for each scoring procedure for
each sample. Selecting the highest correlation for each sam-
ple gave each scoring measure the highest possible score to
be included in our calculations. Some studies reported the
criterion validity of CBM-W with a large number of tests
and subtests, some of which had little to do with writing
ability. For example, Espin, Scierka, Skare, and Halverson
(1999) reported criterion validity of CBM-W with subtests
of the California Achievement Test. The subtests included
reading, math, language arts total, language arts expression,
and language arts mechanics. We were only interested in
CBM-W measures as a predictor of performance on crite-
rion measures that assessed writing or language arts ability.
By picking the highest correlations, we avoided diluting the
results with irrelevant criterion measures, such as a math
subtest. Selecting the highest correlation also gave the scor-
ing procedures the best possible criterion validity under the
best administration procedures (i.e., prompt type and dura-
tion) for each study. Many studies found that criterion valid-
ity increased as the duration of writing was extended. The
correlation selection procedures avoided the shorter writing
durations tempering the criterion validity of the scoring
76
procedure for these studies. We treated grade levels that
were reported individually as separate samples and selected
the highest correlation for each scoring procedure in each
grade level. For example, if a study reported second and
third grade separately (e.g., Ritchey & Coker, 2013), we
selected the highest correlation for each scoring procedure
(in this case WW, WSC, and CWS) in each grade level.
Ritchey and Coker used two different prompts (story starter
and picture story), two different writing durations (3 and 5
min), and one criterion measure (Woodcock–Johnson III
Writing Samples subtest). Ritchey and Coker reported a
total of 18 correlations. However, by selecting the highest
correlation for each scoring procedure at each grade level,
we included six for our analysis: the highest correlation for
WW, WSC, and CWS in second and third grade. In total, we
selected 102 correlations across all studies for our analysis.
Calculating Mean Correlations
To determine whether there was enough variability in our
data to permit an exploration of our research questions
beyond reporting an overall mean weighted correlation
value, we calculated Q and I2 values. Cochran’s Q assesses
homogeneity of data points that were used to calculate a
weighted mean (e.g., effect size or mean correlation;
Cochran, 1954). A significant Q statistic indicates heteroge-
neity in the data used to calculate the weighted mean
(Cochran, 1954). Higgins and Thompson (2002) recom-
mended the I2 statistic, which measures the percentage of
homogeneity across the data; they argue that Cochran’s Q is
underpowered to detect heterogeneity when a meta-analysis
includes a small number of studies. All mean weighted cor-
relations and Q statistics were calculated using
Comprehensive Meta-Analysis (Version 2) software.
Using all selected correlations, we calculated a Q statis-
tic of 396.229 (p < .000, I2 = 76.024). The I2 value calcu-
lated here indicates that 76% of the variance is likely not
due to chance or error. We conducted a second Q statistic
calculation using the highest reported correlation for each
study regardless of scoring procedure. This analysis pro-
duced a Q statistic of 89.297 (p < .000, I2 = 68.644). The I2
value indicates that approximately 64% of the variance is
likely not due to chance or error. The Q statistics and I2
statistics for these overall correlations indicate a high level
of heterogeneity within the data. These results indicated
that there was enough heterogeneity within our data set to
explore our research questions.
We checked for publication bias in our results by calcu-
lating Orwin’s failsafe N (Borenstein, 2005). When calcu-
lating Orwin’s failsafe N, we used the mean criterion
validity coefficient found when combining all scoring pro-
cedures. The overall mean weighted correlation was .55.
We assumed that the mean effect of all hidden studies was
.00 and calculated the number of hidden studies necessary
The Journal of Special Education 51(2)
to lower the mean weighted correlation to .30. Orwin’s fail-
safe N indicated that 29 studies would have to be unpub-
lished (or published via non-peer-reviewed sources) with a
mean effect of .00 in the hidden studies.
Typically, the next step in meta-analytic research would
be to conduct inferential statistics to explore sources of het-
erogeneity through our research questions. However, we
did not calculate inferential statistics due to small sample
sizes. Instead, we calculated weighted correlations and con-
fidence intervals, similar to Slavin’s (1986) recommenda-
tions for a Best Evidence Synthesis.
Results
A total of 22 articles (21 studies due to combining D. C.
Parker et al., 2011 and McMaster, Du, et al., 2011) were
included in this meta-analysis. Studies were published
between 1991 and 2015. The data in the studies represented
the written language performance of 3,361 students.
Fourteen studies were published after McMaster and Espin
(2007). See Table 1 for a summary of each study.
To calculate the overall mean criterion validity of all
CBM-W scoring procedures when combined, we selected
only the single highest criterion validity coefficient from
each sample. When selecting only the highest reported cor-
relation for each study, we calculated an overall correlation
of .55 (CI = [.51, .60]). This correlation represents the high-
est possible mean weighted correlation for all four CBM-W
scoring measures combined.
The overall correlations for all four scoring measures are
presented in Table 2. WW had an overall correlation of .37.
WSC had an overall correlation of .44. CWSs had an overall
correlation of .51. CIWSs had an overall correlation of .60.
Mean correlations were calculated for each scoring pro-
cedure in four different age groups. Results are presented in
Table 3. In each grade category, CIWS had the highest cri-
terion validity.
Mean correlations were calculated for each type of crite-
rion assessment by combining all scoring procedures.
Results are presented in Table 4. The overall correlation for
state-developed assessments was .61. The overall correla-
tion for commercially developed assessments was .54.
Discussion
Our meta-analysis addressed questions about what Fuchs
(2004) termed Stage 1 research in CBM. Overall, we found a
mean correlation of r = .55 when combining all CBM-W
scoring indices. This correlation is the highest possible crite-
rion validity coefficient when aggregating the highest corre-
lation coefficients from each study. This correlation includes
all types of scoring procedures for all grade levels. Therefore,
it is difficult to make conclusive statements about CBM-W
based on this one statistic. In general, these findings fall in
Romig et al. 77

Table 1. Peer-Reviewed Studies Examining Criterion Validity of Curriculum-Based Measurement in Written Language.

Sample Writing measure Criterion validity

Time
Study n Grade(s) Type of prompt Scoring procedurea (min) Criterion measure
Parker, Tindal, 54 6–8 Story WW, WSC, CWS 6 TOWL-Vocabulary
and Hasbrouck TOWL-Theme
(1991)b TOWL-Spelling
TOWL-Word Use
TOWL-Style
TOWL-Handwriting
TOWL-Total Quotient
Espin, Scierka, 147 10c Story WW, WSC, CWS 3 CAT-LA Total
Skare, and CAT-LA Exp
Halverson CAT-LA Mech
(1999) CAT-Reading
CAT-Math
Espin et al. 37, 3 min 8 Story WW, WSC, CWS, 3, 5 District writing test
(2000) 34, 5 min Descriptive CIWS
Gansle, Noell, 75, third grade 3–4 Story WW, WSC, CWS 3 ITBS-Total subscale
VanDerHeyden, 96, fourth grade LEAP-Write
Naquin, and Competently
Slider (2002) LEAP-Use Conventions
of Language
Gansle et al. 45 3–4 Story WW, CWS 3 WJ-R-Writing samples
(2004)
Jewell and 87, second grade 2, 4, 6 Story WW, WSC, CWS 3 SAT 9–Language
Malecki (2005) 59, fourth grade SAT 9–Spelling
57, sixth grade
Weissenburger 156, fourth grade, 3 and 4, 8, 10 Story WW, CWS, CIWS 3, 5, 10 WKCE Language Arts
and Espin 5 min WKCE Writing
(2005) 184, fourth grade, 10 min
127, eighth grade, 3 and
5 min
137, eighth grade, 10 min
152, 10th grade, 3 min
151, 10th grade, 5 min
163, 10th grade, 10 min
Gansle, 163, SAT 9 total language 2–5 Story WW, WSC, CWS 3 SAT 9–Total Language
VanDerHeyden, 118, SAT 9 prewriting, SAT 9–Prewriting
Noell, Resetar, composing, and editing SAT 9–Composing
and Williams SAT 9–Editing
(2006)
Espin, Wallace, 183 10 Story WW, WSC, CWS, 3, 5, 7, MBST/MCA-Writing
et al. (2008) CIWS 10 subtest
McMaster and 25, third grade 3, 5, 7 Passage copy WW, WSC, CWS, 3, 5, 7 MCA-Writing subtest
Campbell 43, fifth grade Picture CIWS TOWL-3 Spontaneous
(2008) 55, seventh grade Narrative Writing
Expository
Cheng and Rose 22 7–12 Picture WW, WSC, CWS, 3 TOWl-3 Spontaneous
(2009)b Narrative CIWS Writing
McMaster, 100 1 Sentence WW, WSC, CWS, 3, 5 District rubric
Du, and Copying CIWS TOWL-3 Spontaneous
Pétursdóttir Story prompt Writing
(2009) Picture word
Prompt
Picture Prompt
(continued)
78 The Journal of Special Education 51(2)

Table 1. (continued)
Sample Writing measure Criterion validity

Time
Study n Grade(s) Type of prompt Scoring procedurea (min) Criterion measure
Coker and 75, Kind-TOWL-2 Basic K–1 Sentence WW, WSC, CWS 3 TEWL-2 Basic Writing
Ritchey (2010) Writing TEWL-2 Contextual
69, Kind-TOWL-2 Writing
Contextual Writing WJ-III Spelling
149, first grade, WJ-III WJ-III Writing Samples
spelling and broad WJ-III Broad Writing
writing
154, first grade, WJ-III
writing samples
Amato and 447 8 Story WW, WSC, CWS, 3 TOWL-3 Writing
Watkins (2011) CIWS Quotient
López and 36, sixth grade 6–8 Story CWS 3 AIMS-Writing
Thompson 23, seventh grade assessment
(2011) 24, eighth grade
McMaster, Du, 79 1 Sentence copy WW, WSC, CWS 3, 5 TOWL-3 Contextual
et al. (2011) Picture word Conventions
Story TOWL-3 Contextual
Language
TOWL-3 Story
Construction
TOWL-3 Total
TOWL-3 Total
Combined
Parker, 85 1 Picture word WW, WSC, CWS 3 TOWL-3
McMaster, and Sentence
Burns (2011) copy
Mercer, Martinez, 163 10 Passage copy WW, CWS, CIWS 5 ISTEP Plus EOC-Total
Faust, and Expository ISTEP Plus EOC-
Mitchell (2012) Narrative Reading
ISTEP Plus EOC-
Writing
Campbell, Espin, 23–35 (varied by probe) 10–12 Narrative WW, WSC, CWS, 3, 5, 7 MBST
and McMaster Expository CIWS TOWL-3 Spontaneous
(2013)b Picture Writing
Ritchey and 88, second grade 2–3 Picture story WW, WSC, CWS WJ-III Writing Samples
Coker (2013) 82, third grade Story starter
Ritchey and 150 1 Sentence WW, CWS, CIWS 3, 5 WJ-III Spelling
Coker (2014) writing WJ-III Writing Samples
Picture story WJ-III Broad Writing
Dockrell, 236 3–5 Narrative WW, CWS 5 Wechsler Objective
Connelly, Expository Language Dimensions
Walter, and
Critten (2015)

Note. WW = words written; WSC = words spelled correctly; CWS = correct word sequence; TOWL = Test of Written Language; CAT = California
Achievement Test; CIWS = correct minus incorrect word sequence; ITBS = Iowa Test of Basic Skills; LEAP = Louisiana Educational Assessment Program;
WJ-R = Woodcock–Johnson–Revised; SAT-9= Stanford Achievement Test–9th Edition; WKCE = Wisconsin Knowledge and Concepts Exam; MBST/MCA =
Minnesota Basic Standards Test/Minnesota Comprehensive Assessment; MCA = Minnesota Comprehensive Assessment; TOWL-3 = Test of Written Language–3;
TEWL-2 = Test of Early Written Language–2; WJ-III = Woodcock–Johnson III; AIMS = Arizona Instrument to Measure Standards writing assessment; ISTEP
Plus EOC = Indiana Statewide Testing for Educational Progress Plus End of Course Assessment; IEP = Individual Education Plan.
a
Only scoring procedures related to this study are reported here (WW, WSC, CWS, CIWS). Some of these studies included other scoring
procedures not reported here. bThese studies used specific samples. Parker, Tindal, and Hasbrouck (1991) used a sample of students receiving
IEP services. Cheng and Rose (2009) used a sample of students receiving services for hearing impairments. Campbell et al. (2013) used a sample
of students receiving services as English Language Learners. cCBM-W probes were administered in 10th grade and criterion measures were
administered in 11th grade.
Romig et al. 79

Table 2. Mean Correlations for Scoring Procedures. When examining each scoring procedure individually,
CIWS (r = .60) had the highest criterion validity followed by
Scoring procedure Correlation Confidence interval n
CWS (r = .51), WSC (r = .44), and WW (r = .37). Although
WW .37 [.31, .43] 28 we did not conduct inferential statistics due to the small sam-
WSC .44 [.37, .51] 22 ple size, there was overlap in the confidence intervals of
CWS .51 [.46, .56] 31 CIWS and CWS, CWS and WSC, and WSC and WW. CWS
CIWS .60 [.54, .66] 15 and CIWS appear to substantially outperform WW or WSC.
Note. WW = words written; WSC = words spelled correctly; CWS
Some may find these results surprising. WW had high
= correct word sequence; CIWS = correct minus incorrect word criterion validity at its initial development (Deno et al.,
sequence. 1980), but our results are consistent with the trends
McMaster and Espin (2007) described: After initial IRLD
Table 3. Mean Correlations for CBM Scoring Procedures studies, CBM-W measures tended to have lower criterion
Across Four Grade Categories. validity. CIWS was developed much later than the other
Scoring procedure Correlation Confidence interval n measures and was primarily developed for use with second-
ary students (Espin et al., 2000). Although it has few studies
K–2 examining criterion validity with younger students, it does
WW .46 [.34, .56] 8 appear to be a promising measure in all grade levels.
WSC .52 [.43, .61] 8
When examining CBM-W measures at each grade level,
CWS .56 [.47, .64] 8
we identified trends for each scoring procedure. Specifically,
CIWS .58 [.49, .66] 3
WW had the lowest and CIWS had the highest criterion
3–5
validity at each grade level. The confidence interval for
WW .34 [.26, .42] 8
WSC .34 [.25, .41] 5
WW had little overlap with the confidence interval for
CWS .48 [.42, .54] 8 CIWS indicating that CIWS likely has stronger criterion
CIWS .65 [.54, .73] 2 validity than WW at every grade level. In most grade levels
6–8 (3–5 being the only exception), criterion validity increased
WW .32 [.20, .42] 5 as the complexity of the scoring procedure increase. For all
WSC .39 [.22, .54] 4 other grade levels, CIWS was the highest followed by
CWS .50 [.40, .59] 8 CWS, WSC, and WW.
CIWS .59 [.49, .67] 4 Finally, no major differences were detected between
9–12 using commercially developed (r = .54) and state-devel-
WW .35 [.21, .48] 7 oped assessments (r = .61) as the criterion. CBM-W appears
WSC .47 [.27, .64] 5 to predict performance similarly on both types of criterion
CWS .52 [.37, .65] 7 measures included in this analysis.
CIWS .65 [.49, .77] 6 Based on the results of Orwin’s failsafe N, we believe
these results are robust to publication bias. The failsafe N
Note. WW = words written; WSC = words spelled correctly; CWS =
correct word sequence; CIWS = correct minus incorrect word sequence. test indicated that 29 studies with a mean effect of .00 would
have to be hidden from this search to lower the overall mean
correlation from .55 to .30. We believe it is unlikely that 29
Table 4. Correlations of Combined Scoring Procedures for
State- and Commercially Developed Tests. non-peer-reviewed manuscripts exist that meet the rest of
our criteria, and if 29 studies do exist, we do not believe the
Assessment type Correlations Confidence interval n overall mean effect would be .00.
State-developed .61 [.51, .69] 10 When answering questions about Phase 1 of CBM
tests research, considering context of writing assessments is help-
Commercially .54 [.48, .59] 19 ful. Although lower than what has been reported for CBM in
developed tests reading (Wayman et al., 2007), these mean correlations are
similar to correlations of other standardized writing measures.
In an area such as written language where few progress moni-
the middle of the ranges reported by McMaster and Espin toring measures exist, these correlations could be considered
(2007); however, this result allows researchers to provide a strong enough to influence instructional decision making.
more specific and stable answer to the question of how valid
CBM-W scores are. Based on the study samples included in
this meta-analysis, CBM scoring procedures appear to have
Future Research
moderate criterion validity in relation to commercially and The results from this study have at least four implications for
state- or locally developed criterion measures. future research. First, meta-analytic research should
80
continue answering questions about Stage 1 research in
CBM. One possibility is to examine the criterion validity of
specific administration procedures. For example, McMaster
and Espin (2007) found that longer durations of writing typi-
cally lead to stronger criterion validity. Also, Espin et al.
(2000) hypothesized that expository prompts may have
higher criterion validity with secondary students because
much of the academic writing they do in school is exposi-
tory. Second, researchers should continue examining the cri-
terion validity of CIWS. Our extensive search revealed
relatively few studies examining the criterion validity of this
scoring procedure. Because it had the highest criterion valid-
ity in all grade levels, it merits additional study.
Third, researchers should continue investigating the
technical aspects of static scores as criterion measures
change. The Test of Written Language–3 was commonly
used as a criterion measure in these studies. However, the
Test of Written Language–4 is now available. In addition,
many states are changing state assessments to align with the
Common Core State Standards. As new writing assessments
are developed commercially and at the state level, research-
ers should continue to investigate the criterion validity of
CBM-W measures with these new measures.
Finally, researchers should consider entering Phase 2 of
CBM-W research. Some have begun this work already. For
example, McMaster, Du, et al. (2011) examined the slopes
of first-grade CBM data and determined that eight or nine
data points are needed for reliable and stable slopes. Similar
work needs to be done across grade levels and demographic
groups.
Implications
Based on the results of this analysis, it appears that practi-
tioners should discontinue the use of WW as a measure of
overall writing progress. Of all the CBM-W scoring mea-
sures, it provided the weakest criterion validity. Furthermore,
practitioners should feel comfortable with the CWS and
CIWS scoring procedures as a moderate predictor of overall
writing ability.
Limitations
In addition to the shortage of studies in certain grade levels,
results of this analysis should be viewed in light of at least
three other limitations. First, this review examined criterion
validity of CBM-W measures only when using commer-
cially developed or state-developed writing assessments.
Using holistic ratings, teacher ratings, and GPA may pro-
duce different validity coefficients. However, based on
findings from McMaster and Espin (2007), any differences
are likely to be minimal.
Second, this analysis only included the highest coeffi-
cient for each scoring procedure in each sample. This
The Journal of Special Education 51(2)
selection procedure means that these results are the highest
possible mean weighted criterion validity coefficients that
can be calculated from these studies. Including more coef-
ficients from each study could lower the criterion validity
results reported here.
Finally, this analysis included only studies from peer-
reviewed sources. Orwin’s failsafe N bolsters confidence in
the results, but it cannot eliminate the possibility of publica-
tion bias.
Conclusion
The present meta-analysis was the first known study
which calculated the mean weighted criterion validity of
CBM-W measures. The findings suggest that practitio-
ners should consider incorporating CBM-W into IEP
development and progress monitoring. Researchers
should consider analyzing technical characteristics of
CBM-W slope data.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of this
article
Funding
The author(s) received no financial support for the research,
authorship, and/or publication of this article.
References
References marked with an asterisk indicate studies included in
the meta-analysis.
*Amato, J. M., & Watkins, M. W. (2011). The predictive
validity of CBM writing indices for eighth-grade stu-
dents. The Journal of Special Education, 44, 195–204.
doi:10.1177/0022466909333516
Archer, A. L., & Hughes, C. A. (2010). Explicit instruction:
Effective and efficient teaching. New York, NY: Guilford.
Borenstein, M. (2005). Software for publication bias. In H.
R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.),
Publication bias in meta-analysis: Prevention, assess-
ment, and adjustments (pp. 193–220). West Sussex, UK:
John Wiley.
Campbell, H. M. (2010). The technical adequacy of curriculum-
based measurement passage copying with secondary school
English language learners. Reading & Writing Quarterly, 26,
289–307.
*Campbell, H. M., Espin, C. A., & McMaster, K. (2013). The
technical adequacy of curriculum-based writing measures
with English learners. Reading and Writing, 26, 431–452.
doi:10.1007/s11145-012-9375-6
*Cheng, S. F., & Rose, S. (2009). Investigating the technical ade-
quacy of curriculum-based measurement in written expres-
sion for students who are deaf or hard of hearing. Journal of
Deaf Studies and Deaf Education, 14, 503–515. doi:10.1093/
deafed/enp013
Romig et al.
Cochran, W. G. (1954). The combination of estimates
from different experiments. Biometrics, 10, 101–129.
doi:10.2307/3001666
*Coker, D. L., & Ritchey, K. D. (2010). Curriculum-based mea-
surement of writing in kindergarten and first grade: An inves-
tigation of production and qualitative scores. Exceptional
Children, 76, 175–193. doi:10.1177/001440291007600203
Comprehensive Meta-Analysis (Version 2) [Computer software].
(2005). Englewood, NJ: Biostat.
Cutler, L., & Graham, S. (2008). Primary grade writing instruc-
tion: A national survey. Journal of Educational Psychology,
100, 907–919.
Deno, S. (1985). Curriculum-based measurement: The emerg-
ing alternative. Exceptional Children, 52, 219–232.
doi:10.1177/001440298505200303
Deno, S., Mirkin, P. K., & Marston, D. (1980). Relationships
among simple measures of written expression and perfor-
mance on standardized achievement tests (Report No. 22).
Minneapolis: University of Minnesota Institute for Research
on Learning Disabilities.
*Dockrell, J. E., Connelly, V., Walter, K., & Critten, S. (2015).
Assessing children’s writing products: The role of curricu-
lum based measures. British Education Research Journal, 41,
575–595. doi:10.1002/berj.3162
Epstein, M., Atkins, M., Cullinan, D., Kutash, K., & Weaver, R.
(2008). Reducing behavior problems in the elementary school
classroom: A practice guide (NCEE #2008-012). Washington,
DC: National Center for Education Evaluation and Regional
Assistance, Institute of Education Sciences, U.S. Department
of Education. doi:10.1037/e602222011-001
Espin, C. A. (2014). What are goed righting! (What is good writ-
ing?). Literacy Research and Instruction, 53, 93–95. doi:10.1
080/19388071.2014.869944
*Espin, C. A., Scierka, B. J., Skare, S., & Halverson, N. (1999).
Criterion-related validity of curriculum-based measures in
writing for secondary students. Reading & Writing Quarterly,
15, 5–27. doi:10.1080/105735699278279
*Espin, C. A., Shin, J., Deno, S. L., Skare, S., Robinson,
S., & Benner, B. (2000). Identifying indicators of writ-
ten expression proficiency for middle school stu-
dents. The Journal of Special Education, 34, 140–153.
doi:10.1177/002246690003400303
*Espin, C., Wallace, T., Campbell, H., Lembke, E. S., Long, J.
D., & Ticha, R. (2008). Curriculum-based measurement in
writing: Predicting the success of high-school students on
state standards tests. Exceptional Children, 74, 174–193.
doi:10.1177/001440290807400203
Fuchs, L. S. (2004). The past, present, and future of curriculum-
based measurement research. School Psychology Review, 33,
188–192.
*Gansle, K. A., Noell, G. H., VanDerHeyden, A. M., Naquin,
G. M., & Slider, N. J. (2002). Moving beyond total words
written: The reliability, criterion validity, and time cost of
alternate measures for curriculum-based measurement in
writing. School Psychology Review, 31, 477–497.
*Gansle, K. A., Noell, G. H., VanDerHeyden, A. M., Slider, N. J.,
Hoffpauir, L. D., Whitmarsh, E. L., & Naquin, G. M. (2004).
An examination of the criterion validity and sensitivity to
81
brief intervention of alternate curriculum-based measures
of writing skill. Psychology in the Schools, 41, 291–300.
doi:10.1002/pits.10166
*Gansle, K. A., VanDerHeyden, A. M., Noell, G. H., Resetar, J.
L., & Williams, K. L. (2006). The technical adequacy of cur-
riculum-based and rating-based measures of written expres-
sion for elementary school students. School Psychology
Review, 35, 435–450.
Graham, S., Harris, K. R., & Hebert, M. (2011). It is more than just
the message: Presentation effects in scoring writing. Focus on
Exceptional Children, 44, 1–12.
Higgins, J. P., & Thompson, S. G. (2002). Quantifying hetero-
geneity in a meta-analysis. Statistics in Medicine, 21, 1539–
1558. doi:10.1002/sim.1186
Hosp, M. K., Hosp, J. L., & Howell, K. W. (2012). The ABCs of
CBM: A practical guide to curriculum-based measurement.
New York, NY: Guilford.
Individuals With Disabilities Education Act, 20 U.S.C. § 1400
(2004).
*Jewell, J., & Malecki, C. K. (2005). The utility of CBM written
language indices: An investigation of production-dependent,
production-independent, and accurate-production scores.
School Psychology Review, 34, 27–44.
Lembke, E., Hampton, D., & Hendricker, E. (2013). Data-based
decision-making in academics using curriculum-based mea-
surement. In J. W. Lloyd, T. J. Landrum, B. Cook, & M.
Tankersley (Eds.), Research-based approaches for assess-
ment (pp. 18–31). Boston, MA: Pearson.
*López, F. A., & Thompson, S. S. (2011). The relationship
among measures of written expression using curriculum-
based measurement and the Arizona Instrument to Measure
Skills (AIMS) at the middle school level. Reading & Writing
Quarterly, 27, 129–152. doi:10.1080/10573561003769640
*McMaster, K. L., & Campbell, H. (2008). New and existing
curriculum-based writing measures: Technical features
within and across grades. School Psychology Review, 37,
550–566.
*McMaster, K. L., Du, X., & Pétursdóttir, A. L. (2009).
Technical features of curriculum-based measures for begin-
ning writers. Journal of Learning Disabilities, 42, 41–60.
doi:10.1177/0022219408326212
*McMaster, K. L., Du, X., Yeo, S., Deno, S. L., Parker, D., & Ellis,
T. (2011). Curriculum-based measures of beginning writing:
Technical features of the slope. Exceptional Children, 77, 185–
206. doi:10.1177/001440291107700203
McMaster, K. L., & Espin, C. (2007). Technical features of curric-
ulum-based measurement in writing: A literature review. The
Journal of Special Education, 41, 68–84. doi:10.1177/002246
69070410020301
McMaster, K. L., Ritchey, K. D., & Lembke, E. (2011).
Curriculum-based measurement of elementary students’
writing: Recent developments and future directions. In T. E.
Scruggs & M. A. Mastropieri (Eds.), Assessment and inter-
vention: Advances in learning and behavioral disabilities
(pp. 111–148). Bingley, UK: Emerald.
*Mercer, S. H., Martinez, R. S., Faust, D., & Mitchell, R. R.
(2012). Criterion-related validity of curriculum-based mea-
surement in writing with narrative and expository prompts rel-
82
ative to passage copying speed in 10th grade students. School
Psychology Quarterly, 27, 85–95. doi:10.1037/a0029123
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational
measurement (3rd ed., pp. 13–103). New York, NY:
Macmillan.
*Parker, D. C., McMaster, K. L., & Burns, M. K. (2011).
Determining and instructional level for early writing skills.
School Psychology Review, 40, 158–167.
*Parker, R. I., Tindal, G., & Hasbrouck, J. (1991). Progress moni-
toring with objective measures of writing performance for
students with disabilities. Exceptional Children, 58, 61–73.
doi:10.1177/001440299105800106
*Ritchey, K. D., & Coker, D. L. (2013). An investigation of the
validity and utility of two curriculum-based measurement
writing tasks. Reading & Writing Quarterly, 29, 89–119. doi:
10.1080/10573569.2013.741957
*Ritchey, K. D., & Coker, D. L. (2014). Identifying writing dif-
ficulties in first grade: An investigation of writing and read-
ing measures. Learning Disabilities Research & Practice, 29,
54–65. doi:10.1111/ldrp.12030
The Journal of Special Education 51(2)
Slavin, R. E. (1986). Best-evidence synthesis: An alterna-
tive to meta-analytic and traditional reviews. Educational
Researcher, 15, 5–11. doi:10.3102/0013189x015009005
Taylor, R. L. (2006). Assessment of exceptional students:
Educational and psychological procedures (7th ed.). Boston,
MA: Pearson Education.
Venn, J. J. (2013). Assessing students with special needs. Boston,
MA: Pearson Education.
Videen, J., Deno, S., & Marston, D. (1982). Correct word
sequences: A valid indicator of proficiency in written expres-
sion (Report No. 84). Minneapolis: University of Minnesota
Institute for Research on Learning Disabilities.
Wayman, M. M., Wallace, T., Wiley, H. I., Tichá, R., &
Espin, C. A. (2007). Literature synthesis on curriculum-
based measurement in reading. The Journal of Special
Education, 41, 85–120. doi:10.1177/0022466907041
0020401
*Weissenburger, J. W., & Espin, C. A. (2005). Curriculum-based
measures of writing across grade levels. Journal of School
Psychology, 43, 153–169. doi:10.1016/j.jsp.2005.03.002