See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/321434164

Developing a reading comprehension test for cognitive diagnostic

assessment: A RUM analysis

Article  in  Studies In Educational Evaluation · December 2017

DOI: 10.1016/j.stueduc.2017.10.007

CITATIONS READS

6 329

2 authors, including:

Fatemeh Ranjbaran
Iran University of Medical Sciences
10 PUBLICATIONS   13 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

PhD dissertation View project

All content following this page was uploaded by Fatemeh Ranjbaran on 15 January 2020.

The user has requested enhancement of the downloaded file.

 6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²

Contents lists available at ScienceDirect

Studies in Educational Evaluation

journal homepage: www.elsevier.com/locate/stueduc

Developing a reading comprehension test for cognitive diagnostic 0$5.

assessment: A RUM analysis
⁎
Fatemeh Ranjbaran , Seyyed Mohammed Alavi
Department of Foreign Languages and Literature, University of Tehran, Iran

A R T I C L E I N F O A B S T R A C T

Keywords: A critical issue in cognitive diagnostic assessment (CDA) lies in the dearth of research in developing diagnostic
Attributes tests for cognitive diagnostic purposes. Most research thus far has been mainly carried out on large-scale tests,
CDA e.g., Test of English as a Foreign Language (TOEFL), Michigan English Language Assessment Battery (MELAB),
Formative assessment International English Language Testing System (IELTS), etc. In particular, CDA of formative language assessment
CDM
that aims to inform instruction and to discover strengths and weaknesses of students to guide instruction has not
Fusion model
Q-matrix
been conducted in a foreign (i.e., second) language-learning context. This study explored how developing a
Reading comprehension test reading comprehension test based on a cognitive framework could be used for such diagnostic purposes. To
RUM achieve this, initially, a list of 9 reading attributes was prepared by experts based on the literature, and then the
Second language targeted attributes were used to construct a 20-item reading comprehension test. Second, a tentative “Q-matrix”
that specified the relationships between test items and the target attributes required by each item was developed.
Third, the test was administered to seven language-testing experts who were asked to identify which of the 9
attributes were required by each item of the test. Fourth, on the basis of the overall agreement of the experts’
judgments concerning the choices of attributes, review of the literature and results of student think-aloud
protocols, the tentative Q-Matrix was refined and used for statistical analyses. Finally, the test was administered
to 1986 students of a General English Language Course at the University of Tehran, Iran. To examine the CDA of
the test, the Reparameterized Unified Model (RUM) (also known as the Fusion Model), a type of cognitive
diagnostic measurement model (CDM), was used for further refining the Q-Matrix for future data analyses and,
most importantly, for diagnosing the participants' strengths and weaknesses. Data analysis results confirmed that
the nine proposed reading attributes are involved in the reading comprehension test items. Such diagnostic
information could be helpful for teachers and practitioners to prepare instructional materials that target specific
weaknesses and inform them of the more problematic areas that need to be emphasized in class in order to plan
for better L2 reading instruction. Further, such information could inform individualized student instruction and
produce improved diagnostic tests for future use.

1. Introduction detect student errors, incomplete understandings, and misconceptions

Over recent decades, testing and assessment in general have been
employed to obtain overall, average or individual scores of achieve-
ment for examining student’s acquired knowledge. However, assess-
ment that can function as formative to inform instruction has currently
become the focus of attention for diagnostic purposes and is intended to
assess strengths and weaknesses of students to guide instruction. From
another stance, teacher-made tests and quizzes have recently gained
attention because of their formative assessment role in aiding students’
learning (Huang & Wu, 2013). To make the process of learning easier
and more effective, teachers are expected to be competent in test con-
struction and learning diagnostics in class. As these tests are expected to
during the learning process, the use of diagnostic tests to improve
student’s conceptual understanding has become highly valued and re-
cognized in many fields (Hartman, 2001). Hence, researchers and
practitioners are increasingly focusing on the integration of cognitive
psychology, educational pedagogy, and educational measurement for
the improvement of learning and instruction (Leighton, Gierl, & Hunka,
2004; Mislevy, 2006; Tatsuoka, 1995; Snow & Lohman, 1989).
Reading is a fundamental skill for gaining knowledge in all aca-
demic fields. Therefore, it is necessary to determine the various com-
ponents of reading ability for a better understanding of reading and to
find language learner problems when teaching second language (L2)
reading. If problematic areas of reading proficiency are diagnosed

⁎
Corresponding author.
E-mail address: franjbaran@ut.ac.ir (F. Ranjbaran).

http://dx.doi.org/10.1016/j.stueduc.2017.10.007
Received 17 December 2016; Received in revised form 8 October 2017; Accepted 25 October 2017
;‹(OVHYLHU/WG$OOULJKWVUHVHUYHG
F. Ranjbaran, S.M. Alavi
during the instructional term, sufficient and timely feedback can be
provided to students in order to improve learning and eliminate
weaknesses during the learning process.
Criticism occurs because the main goal of educational tests is
usually to provide quantitative assessment of a student’s general
overall, often unidimensional ability and proficiency as compared to
other students in a normative group. This type of norm-referenced
testing has been used extensively for ranking and selecting students for
various educational decisions. In addition to merely providing general
summarizing and usually unidimensional information about students’
skills and their ability to perform on a test, these assessments are in-
variably incapable of providing necessary detailed information about
students’ strengths and weaknesses that could possibly help them in
improving their skills or that might also assist the teacher in instruc-
tional planning. Recently, scholars have suggested that cognitive di-
agnostic assessment has a key role in improving the informational value
of assessment (Alderson, 2010; de la Torre, 2009; Jang, 2005;
Leighton & Gierl, 2007; Rupp, Templin, & Henson, 2012). In his com-
mentary on “Cognitive Diagnosis and Q-Matrices in Language Assess-
ment,” Alderson (2010) shows his disappointment of there being very
few truly diagnostic tests in existence. In fact, nearly all studies carried
out thus far have been on existing large-scale assessments and profi-
ciency tests, not those developed for low-stakes formative assessment.
He argues that far more studies focus on developing and researching
high-stakes proficiency tests aimed at placement, achievement, or ap-
titude than are specifically constructed for cognitive diagnosis in the
form of classroom-based formative assessments. The most desirable
cognitive diagnostic assessment is the one that is diagnostically de-
signed, constructed, and scored from the initial phase. In such an ap-
proach, cognitive attributes are explicitly defined to be targeted in the
test construction phase. These predetermined attributes should be in
line with the instructional goals. When the attributes are set, the data
are to be analyzed with an appropriate CDM. Afterwards, the scores are
to be reported in a fine-grained diagnostic system. While fine-grained
cognitive diagnostic assessment is intended to inform instructional
settings in this way, diagnostically constructed designs have hardly
been discussed in the literature. A few tests, however, have been de-
signed in order to fulfill the needs of diagnostic analysis (e.g. DIALANG
by Alderson, 2005; Alderson & Huhta, 2005; DELNA (www.delna.
auckland.ac.nz/uoa); and DELTA by Urmston, Raquel, & Tsang, 2013),
while, none have yet provided individualized score reports to enhance
learning and teaching at the classroom level. This study responds to the
call for cognitive diagnostic assessment using a specially constructed
diagnostic test, one that will attempt to provide detailed information
about students’ strengths and weaknesses in L2 reading comprehension,
and perhaps in reading comprehension in general.
2. Literature review
2.1. L2 reading ability
In CDA, the different components of a specific domain (in this case,
L2 reading) are referred to as attributes. Attributes are the divided
components of a general cognitive ability, which can be defined as
“procedures, skills, or knowledge a student must possess in order to
successfully complete the target task” (Birenbaum, Kelly, & Tatsuoka,
1993, p.443). Therefore, L2 reading attributes are composed of dif-
ferent types of language knowledge, and reading strategies, which are
required in comprehending texts (Birenbaum et al., 1993; Templin,
2004). CDA has been introduced as a new method in educational
measurement that can provide fine-grained diagnostic information
about test-takers’ degree of mastery of domain sub-skills (Lee & Sawaki,
2009). Sub-skills are defined as domain-specific knowledge and skills
that are required to indicate mastery in a specific cognitive domain
(Leighton & Gierl, 2007). Taking reading skill as a cognitive domain, it
is necessary to have knowledge of vocabulary, grammar, and making

6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²
inferences in order to fully comprehend a text. These are considered the
sub-skills of the reading domain, which are called attributes as well,
used interchangeably throughout this paper. The most distinct char-
acteristic of this approach is that it is the point where cognitive psy-
chology and psychometric modeling meet within a single framework,
therefore it aims to assess the test-takers' knowledge and underlying
cognitive processing sub-skills (DiBello, Roussos, & Stout, 2006;
Leighton & Gierl, 2007).
In the assessment of reading comprehension in a second or foreign
language, the many underlying cognitive attributes required for reading
ability mastery have made it a complex process. Reading ability is a
fundamental tool for gaining knowledge and improving learning in
everyday academic settings and everyday life in general. Therefore, it
comes as no surprise that the nature of reading ability has been the
focus of research in applied linguistics, education and psychology for
quite some time (Cohen & Upton, 2006). Regardless of the extensive
research on reading ability, there is still some debate as to how second
language reading ability is defined and how its performance should be
evaluated and reported. It seems that teachers, students, and practi-
tioners have not been given diagnostic feedback tools that could be
used for improvements in reading ability, specifically for classroom-
based profile score reporting. These are issues that mostly need con-
sideration in the context of L2 reading assessment in the Iranian con-
text.
At times, there is such emphasis on reading strategies that other
important elements of reading competence such as language knowl-
edge, including pragmatic knowledge and grammatical knowledge,
have been have been given less attention. One aspect of second lan-
guage reading ability is the use of language to understand written text.
Therefore, the knowledge of language components and strategic
reading competence should both be considered in order to master the
written text. While the difficulty of defining the construct of reading
ability is clear, other problems have been seen with regards to how L2
reading performance has been analyzed and reported. L2 reading test
scores are often reported using a general test score without any detailed
information (Goodman & Hambleton, 2004). When an exam provides
only one total score, it can serve the test’s immediate summative pur-
pose; however, it cannot be easily used to improve reading performance
(Stiggins, Alter, & Chappius, 2004). Only providing a total score does
not provide information regarding each student’s specific strengths and
weaknesses (Sheehan & Mislevy, 1990). On the other hand, a detailed
score report of each individual, including their level on each reading
attribute can be used to both improve individual student reading ability
and guide teacher instruction (Snow & Lohman, 1989).
2.2. Frameworks for developing cognitive diagnostic tests
There are two measurement driven approaches that have been
widely used for diagnostic test development. One is Embretson’s
Cognitive Design System (CDS) (Embretson & Gorin, 2001) and the
other is Mislevy’s Evidence-Centered Design (ECD) (Mislevy, 1994;
Mislevy, Steinberg, & Almond, 2002). These two approaches focus on
the use of cognition in the process of item and test development, con-
sidering the issues of construct definition while item writing, and
concluding with validation procedures (Leighton & Gierl, 2007).
CDS and ECD may differ in their emphasis on the various parts of
assessment design and their details, but both share the three principles
of the assessment triangle. The assessment triangle includes three re-
lated elements, which are cognition (theories of learning), observation
(test data) and interpretation (the probabilistic model that relates a
student’s multidimensional latent cognitive learning state to his/her
test response pattern) (Pellegrino, Chudowsky, & Glaser, 2001). This
NRC panel of researchers state that cognition is related to a cognitive
model about how students represent knowledge and how they develop
competence in a certain subject (p.44). A cognitive model provides a
description of what should be assessed, but it is different from cognition
F. Ranjbaran, S.M. Alavi
in a number of ways: According to Leighton and Gierl (2007), a cog-
nitive model specifies the cognitive components and processes that
constitute the construct (such as reading comprehension) being tested.
This leads to more detailed specifications that are more applicable for
instructional feedback. It should be noted that cognitive theory explains
these specifications, meaning that a model about specific cognitive
processes related to the construct being tested is empirically supported.
In this study, the theory of learning underlying the assessment tri-
angle includes the reading theories of second language learning, in-
cluding the information processing theory, constructivism theory of
reading, and Bloom’s Revised Taxonomy. It is clear that when reading,
engagement in processing at the phonological, morphological, syn-
tactic, semantic and discourse levels, as well as in goal setting, text-
summary building, interpretive elaborating from knowledge resources,
monitoring and assessment of goal achievement, making various ad-
justments to enhance comprehension, and making repairs to compre-
hension processing are necessary (Carrell & Grabe, 2002, p. 234). Even
though a great portion of the reading process is somewhat natural, it is
thought to exceed our conscious control. It is believed that readers do
employ a high level of active control over their reading process through
the use of strategies that are deliberate and purposeful conscious pro-
cedures (Urquhart & Weir, 1998). While processes are considered more
automatic than intentional, and could even be considered subconscious,
strategies are more controllable, and used to act upon the processes
(Cohen & Upton, 2006). Insights into reading strategies explain how
readers interact with the text and to what extent strategies used can
influence their reading comprehension rate. Following the use of
learning theories for the cognition element, which are used to construct
test items, the observation element includes gathering data through
administering the test. The interpretation element makes use of the
CDM (in the case of this study) to relate an examinees multidimensional
latent cognitive learning state to his/her test response pattern. These
circulating elements of the assessment triangle intermingle and even-
tually provide teachers with means to an imperfect attempt at trans-
lating the student’s responses into beneficial information.
The core purpose of diagnostic assessment development from an
ECD framework perspective is the development of coherent evidentiary
arguments about students that can serve as assessment of and assess-
ment for learning, depending on the desired primary purpose of a
particular assessment. The structure of the evidentiary arguments that
are used in the assessment narrative can be described with the aid of
terminology first introduced by Toulmin (1958). An evidentiary argu-
ment is constructed through a series of logically connected claims or
propositions that are supported by data through warrants and backing.
Based on Toulmin’s schema for arguments, a claim about an examinee's
knowledge and skill is situated at the top. At the bottom is the ob-
servation of the examinee acting in some situation. An assessor's rea-
soning moves up from what is important in the examinee's actions
(which the examinee produces) and the features of the situation that are
important in eliciting those actions (partly determined by the assess-
ment designer, but also partly determined by the examinees as they
interact with the task). These data support the claim through a warrant,
or rationale, for why examinees with particular capabilities are likely to
act in certain ways in the situation at hand.
In concrete terms, the ECD framework allows one to distinguish the
different structural elements and the required pieces of evidence in
narratives similar to the following: Ali has most likely mastered basic
reading comprehension skills (claim), because he has answered cor-
rectly to items of a reading passage about the theory of education
(data). It is most likely that he did this because he applied all of the
reading skills and strategies correctly (backing) and the task was de-
signed to force him to do that (backing). He may have used his back-
ground knowledge to understand parts of the text (alternative ex-
planation), but that is unlikely since he was new to the topic (refusal)
(Rupp et al., 2012).
While a theory-driven process of analysing and modeling is led by

6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²
the main elements of expertise in a domain, the core elements in the
ECD framework include (1) the student models, which formalize the
postulated proficiency structures for different tasks, (2) the task models,
which formalize which aspects of task performance are coded in what
manner, and (3) the evidence models, which are the psychometric
models linking those two elements. These three core components are
complemented by (4) the assembly model, which formalizes how these
three elements are linked in the assessment, and (5) the presentation
model, which formalizes how the assessment tasks are being presented
(Rupp et al., 2012). Specifically, the student model is motivated by the
learning theory that underlies the diagnostic assessment system. It
specifies the relevant variables or aspects of learning that we want to
assess at a grain size that suits the purpose of the diagnostic assessment.
As many of the characteristics of learning that we want to assess are not
directly observable, the student model provides a probabilistic model
for making claims about the state, structure, and development of a more
complex underlying system.
2.3. Cognitive diagnostic models
Cognitive Diagnostic Models (CDMs) are data analysis probability
models and often associated techniques that are designed to link cog-
nitive theory with the test items’ psychometric (probabilistic) proper-
ties (Leighton & Gierl, 2007). According to Rupp and Templin (2008),
CDMs are:
“probabilistic, confirmatory multidimensional latent-variable
models with a simple or complex loading structure. They are sui-
table for modeling observable categorical response variables and
contain unobservable (i.e., latent) categorical predictor variables.
The predictor variables are combined in compensatory and non-
compensatory ways to generate latent classes”. (Rupp & Templin,
2008, p.226)
Examples of CDMs include Tatsuoka’s Rule Space Model (Tatsuoka,
1995), the Attribute Hierarchy Method (AHM) (Leighton,
Gierl, & Hunka, 2004), an enhancement of the Fusion Model termed the
RUM (Hartz, 2002; Roussos et al., 2007), and the original RUM
(DiBello, Stout, & Roussos, 1995). Most CDMs are IRT-based latent-class
models in which the characteristic of discrete multidimensionality of
the latent space is the most important. In previous unidimensional IRT-
models, examinee ability was modelled by a single general continuous
ability parameter. By contrast, the discrete multidimensionality of
CDMs makes it possible to investigate the mental processes underlying
the student’s test response by breaking the overall targeted ability, e.g.,
reading competency, down into its component parts. The number of
dimensions depends on the number of skill components involved in the
assessment. Here, α is a vector of dichotomous variables. These prob-
abilities can then be used to estimate each αk, that is, to classify an
examinee as either a master (αk = 1) or a non-master (αk = 0). An-
other feature of CDMs is being compensatory or non-compensatory
(DiBello et al., 1995; Roussos et al., 2007). In non-compensatory models
success in an attribute does not compensate for the deficiency in an-
other attribute in order to correctly respond to an item, as is the case
with the RUM in this study. According to Roussos et al., 2007, a non-
compensatory interaction of skills occurs when application of the re-
quired skills is necessary in order to successfully complete the task; that
is, a lack of competency on any one of the skills for the task will result
in a serious hindrance to successfully complete that task. Non-com-
pensatory models have been preferred for cognitive diagnostic analysis,
as they can generate more fine-grained diagnostic information.
The latent variables of CDMs consist of dichotomous, such as mas-
tery or non-mastery, or polytomous levels, such as a rating variable
with values such as excellent, good, fair, poor, etc. The design structure
of a CDM is the Q-matrix, which maps the skills necessary to success-
fully answer each item on the test (Li & Suen, 2013). Most of the CDM-
based diagnostic studies carried out thus far have been mostly limited
F. Ranjbaran, S.M. Alavi
to analysis of existing summative tests, not the development of new
cognitive diagnostically focused formative assessments. By contrast,
this study focuses on a cognitive diagnostic assessment of a test de-
veloped that is based on a cognitive diagnostic modeling framework.
2.4. Fusion model
The Fusion model, also known as the Reparameterized Unified
Model (RUM), is a cognitive diagnostic model that is used to make
inferences about the mastery level of each latent space attribute for
each examinee, based on the examinee’s item responses (Stout, 2008a,
2008b; Stout, 2008a, 2008b). In other words, RUM is an IRT discrete
multidimensional model, that expresses the stochastic relationship be-
tween item responses and underlying skills as follows (DiBello et al.,
1995):
P (Xi, j = 1 αj , θj ) = πi*Πkk = 1 rik*
(1 − αjk ) qik
pci (θj )
in which, P(Xij = 1ǀαj, θj; πi*, rik*, ci) is the probability of person j on
item i scoring a correct response (X = l) instead of an incorrect one
(X = 0), given person abilities – αj, θj – and item parameters πi*, rik*, ci.
The RUM takes into account the degree of incompleteness of a Q-matrix
resulting from incomplete knowledge representations probably due to
rather complex cognitive strategies being assessed (for example,
reading abilities) via two modeling components for examinees ability
parameters, αj and θj. The former refers to the vector of indicators of
attribute mastery versus non-mastery while the latter refers to the
continuous ability that summarizes attributes not specified by the Q-
matrix. Regarding the item parameters, πi refers to the probability of
correctly applying all required Q-matrix skills for item i given αjk = 1,
so it is similar to the item difficulty based on the Q-matrix. It is expected
that if examinees mastered all required skills for item i, they would
probably be able to correctly apply them in solving item i. Another
important parameter is rik which compares the correct item response
probabilities between mastering skill k and lacking skill k. In the last
part of the equation, Pci(θj), ci refers to the amount that correct item
performance depends on α; the required Q-matrix based skills. There-
fore, it is indicative of the degree to which item performance depends
on θj, which is an additional residual ability not specified in the Q-
matrix. The lower ci is, the more the item depends on θj.
3. Developing the Q-Matrix
First, as described below and based on a review of the second lan-
guage reading comprehension literature, the latent space of the im-
portant attributes that are combined to constitute reading comprehen-
sion mastery is chosen. Next and interactively, as also described below,
a set of test items is constructed to assess the chosen attributes. The
initial step in generating the student learning model that links attributes
to item responses is to map each test item onto an item-by-skill table
known as the Q-matrix. It consists of an I × K: matrix of binary in-
formation in ones and zeros, where I is the number of items and K is the
number of attributes. A Q-matrix is a representation of a working hy-
pothesis regarding which skills are necessary to answer each item on
the test (Li & Suen, 2013). Each item will most likely require more than
one skill to be answered correctly. According to Buck and Tatsuoka
(1998), developing a Q-Matrix requires following a certain procedure:
First a list of skills is developed and then each item is coded as a row in
the Q matrix based on what skills are required for the item. This Q-
matrix construction uses the previously referred to attribute develop-
ment literature, content experts’ judgment and student verbal reports
on the underlying skills required for each item which will be discussed
further in the procedure section. The next step is to analyse the test data
produced by an administration of the test using a Cognitive Diagnostic
Model, in this case the RUM, with the developed Q-Matrix. Finally, the
Q-Matrix can be modified based on statistical information about the

6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²
strength of association of each skill/item combination. For example, an
expert proposed skill required for an item could be dropped because
statistical evidence indicates the hypothesized association is lacking.
This inferred Q-matrix is then available for future statistical analyses.
In developing a Q-matrix, a review of the literature indicates that a
number of alternatives exist. One of the less costly but less efficient
approaches is to use existing test specifications, such as provided by the
test’s developer; however, the attributes indicated via the test specifi-
cations are usually too general for diagnostic purposes. According to
Leighton and Gierl (2007), relying on existing test specifications for Q-
matrices is usually unwarranted, so in this study an attempt was made
at developing a reading comprehension test based on a cognitive di-
agnostic framework, with the help of a rigorous Q-matrix construction.
Jang (2009) suggests using data from a small number of student verbal
reports of how they answered each item to construct the Q-matrix. Even
though there are doubts as to the validity of such verbal reports, they
are considered as fairly reliable and hence useful for reading research
(Leighton & Gierl, 2007). Another approach is to use a group of experts
(cognitive, instructional, curricular, etc.) to describe the underlying
cognitive skills needed to answer each question, based on their previous
experience in this realm (Sawaki, Kim & Gentile, 2009). According to
Leighton and Gierl (2007), an underlying problem with this approach is
the higher ability level of the experts compared with the students, re-
sulting in a gap between the skills and processes truly used by the
students and those proposed by the experts. For example, a physics
expert might cast an inclined plane problem more abstractly than a
beginning physics student. Nevertheless, studies on Q-matrix con-
struction have indicated that using content expert judgment in Q-matrix
construction does increase its reliability (Jang, 2005; , 2009; Kim, 2015;
Li and Suen, 2013; Li, 2011; Svetina, Gorin, & Tatsuoka, 2011; Sawaki
et al., 2009).
After developing the initial Q-matrix, large-scale data analysis based
on an administration of the test to a large sample of students can be
used to empirically validate the Q-matrix that was based on the initial
results of cognitive diagnostic modeling. Often, attributes that are si-
milar can thus be combined to reduce the number of attributes. For
example, Jang (2009) refined her initial Q-matrix of the LanguEdge
reading comprehension test by reducing the number of attributes based
on RUM analysis results. Another instance is the Q-matrix construction
done by Kim (2015). This study followed Hartz (2002), in which at-
tributes that were measured by fewer than three items were either
merged with similar attributes or deleted from the Q-matrix. Thus the
empirical analysis can result in modifying the latent space of attributes
and modifying which attributes are hypothesized to influence each
item. This is useful then for future analyses and for drawing cognitive
conclusions.
4. Research questions
The purpose of this study is to develop a reliable and useful second
language reading comprehension test based on an expert-based selec-
tion of the important attributes embedded in a cognitive diagnostic
framework. Then, to link the attribute selection and test construction, a
Q-matrix of the underlying skills necessary to respond correctly to each
of the test items is constructed. The questions are as follows:
1. What skills/attributes are required in second language reading
comprehension mastery, and how do they theoretically and em-
pirically interact in CDA?
2. Do the results of the RUM and ECD based CDM approach in this
study produce reliable and useful diagnostic information about
student mastery/nonmastery of the attributes and about the test’s
measurement characteristics, with an eye towards future applica-
tions, both in L2 reading and other contexts?
F. Ranjbaran, S.M. Alavi 6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²

Table 1 passages. On three of the occasions, the participant declined giving

Think-aloud verbal protocol participants. verbal reports for all three passages, so I consented to only one passage
for these specific cases. Then the recordings were transcribed and think-
Student Gender Language TOEFL/IELTS Duration of stay
Background score abroad aloud reports in Persian were translated to English for analysis pur-
poses.
NE Female 8.5 years of English 8.5 IELTS 7 years (London) The procedure for analysing think-aloud verbal protocols of this
SN Female 11 years – 11 years (Dubai)
study is the text-focused process based on the study by Jang (2005). For
SJ Female 5 years – –
BA Male Self-studied – – the text-focused process, the verbal report transcripts were analyzed to
KP Female 6 years, institute – – search distinct thought processes that involved text-focused processing.
MK Female 7 years, institute – – The existing text-focused categories were identified. Two example ca-
MH Male A few terms, institute – – tegories of the text-focused processing strategies are provided for fur-
SM Male 7 years, private tutor – –
ther illustration of the analysis procedure.
PR Female Self-studied – –
SK Female 11 years, institute – – Example 1. Deducing word meaning from context.
AP Female 8 years, institute – –
AT Female 8 years, institute – – Referring to key words or phrases in the questions in order find the
MP Male Institutes since – –
answer in the passage according to those clues. Then the examinee
childhood
refers to explicitly stated facts and statements that can guide them to
the correct response. This might also include summarizing and scanning
5. Methodology strategies. The following verbal report excerpts indicate this processing
category.
5.1. Participants

Nazanin: context clues, anything, its up to many things, but I go to

1986 students from the University of Tehran took part in the
the sentence after or the one before the word to better
reading comprehension test. They were bachelor’s students of various
understand it.
majors taking part in the General English course, a requirement of the
I find a key word in the question and then seek for the
bachelor’s program at the University of Tehran. Response data from the
word in the passage.
1986 examinees to each of the 20 questions of the developed test were
I have to understand the entire passage to answer the
used for empirical validation through RUM analysis.
questions. Because one of these levels are connected to
Thirteen students were recruited to understand their use of the
the other level, its part of the level, its not a particular
reading skills on test items through a think-aloud verbal protocol. The
level.
participants included nine female and four male B.A. students, as shown
in Table 1. Several factors that might effect participants verbal re-
porting have been considered for the recording session. Based on ob-
servations made and previous research studies (e.g. Jang, 2005), an
important point to be considered for think-aloud verbal protocol ac- Mojtaba: At first I focus on the whole structure of reading very very
tivities was that not all students would have sufficient English speaking swiftly and very smooth and after that I focus on each
potential to verbalize their thoughts during the think-aloud activity. paragraph and I try to pull the main idea out, I think. I
According to Gass and Mackey (2000), forcing students to report in summarize it, and I hypothetically highlight the
English might result in great potential differences between their actual important sentences, like history, main details.
thoughts and their verbalization and hence the inability to reach rich Vocabulary, I put them in the context. If Im suspicious I
verbal data. Since all participants are in an EFL environment, it is an- go to the passage.
ticipated that they have difficulties in expressing ideas in L2, so pro- At first I read the question and I figured out it’s a general
duction rates of verbal comments will be hampered a great deal because question, includes all the reading, so at first I should read
they might tend to refrain from commenting to avoid the difficulty. the whole thing, but since Im short of time, I just scanned
Thus, participants’ oral ability is a crucial factor to ensuring sufficient and guessed one is correct.
production rates of verbal reports. An attempt was made to minimize The question says chimpanzee, so I look for the key word.
this dilemma by allowing participants to report in their native lan- I think A, because the passage described that
guage, Persian, the language they feel most comfortable with. 7 parti- chimpanzees can, at first they put the chimpanzee in a
cipants verbalized in their native language, while the others decided to paragraph that explains about how crows use sticks to pry
report in English. Each of the students had previously taken the reading peanuts out of cracks. This is an example of intelligence
comprehension test and was asked to verbalize their thought processes by showing what a stick can do. So the props are
when answering the test items. This included their reporting of what important in this area, so chimpanzee is an instance of
skills and strategies first came to mind when answering the items. The this subject.
reading passages and items were given again as reference and retro-
spective verbal reports were recorded. The retrospective verbal reports
were conducted individually in a classroom secure from external noise
and disruptions. A think-aloud verbal protocol instrument was prepared Ava: The sentence was explicitly stated in the passage, instinct is…
and used by the interviewer in each session. Each participant was asked
to talk about what he or she was thinking while reading the passage and
responding to the items. After a silence of about 10 s, the researcher
asked questions such as, “What are you thinking now?” After answering
Arezoo: Context clues help me with finding the meaning of words.
the questions related to the specific passage, the researcher asked the
For those questions which require reading the paragraph, I
participant to recount the process he/she used in order to clarify the
found key words in the question to search the paragraph.
verbal reports and had a chance to ask related questions. The duration
Using tools; means using instruments; for ex. I find a word
of time spent for each student was between 25 and 30 min for the three


F. Ranjbaran, S.M. Alavi
which is similar to instinct. I didnt find instinct. Again
finding key words; animal communication in the passage.
The connection between text ideas and key words is evaluated in
terms of success in solving the items. In this example, Arezoo, Nazanin,
Mojtaba and Ava all make use of context clues to help them deduce the
meaning of the text. Sometimes looking for certain words requires
reading the entire passage, or seeking for key words in parts of the
passage.
Example 2. Inferencing
by referring to background knowledge. With this process readers
make inferences about the authors intentions or lexical word meaning
of the text based on their background knowledge and personal experi-
ence. Inferential comprehension is demonstrated by the reader through
the process of referring to personal knowledge, his/her own intuition,
or personal experiences. The following are some excerpts of this process
from the transcribed and translated verbal reports.
Saeideh: I read something and the concept is highlighted in my
mind. I remember it from an incident that happened for
me.
I didnt know the meaning of the word, but since I knew
what the passage was talking about and I had heard it
before, I guessed the meaning.
Because uhh… I know what is happening, I used my own
knowledge.
Behnam: This one is really easy for me because I learned it before.
Moein: It happened for one of my relatives once and I remembered
that day. It was very bad.
Many words I have seen before but I dont know their exact
meaning, but it helps me to guess from the context.
Saman: I read this word exactly yesterday, and I remembered it.
This possibility that an examinee is acquainted with a topic or has
had previous experience with a topic will make the task easier for him/
her and it will be a benefit to those who have background knowledge of
a certain topic. Sometimes there is a bias on reading comprehension
tests because some students are knowledgeable of a topic while others
fall behind on responding because they are clueless and they haven’t
read up on the topic, despite the fact that they are might be knowl-
edgeable on other topics. This was the case with those examinees that
had previous experience of heat stroke and heat exhaustion and were
easily able to respond to the items, while those who hadn’t any back-
ground knowledge fell behind. Nevertheless, this is the nature of any
reading comprehension exam.
In the next phase, a group of content raters served as developers of
the attributes that reflect the main language skills necessary for suc-
cessful performance on each item. This group included 6 PhD students
at the University of Tehran studying Teaching English as a Foreign
Language, 3 females and 3 males, who all had experience in applied
linguistics research and teaching reading comprehension courses. They
reviewed each test item and the previously selected attributes and then
decided whether or not each attribute was necessary for answering the
item correctly. Each expert read the passages and performed the rating
task independently. They identified the skills for each item and also

6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²
made annotations about the evidence on which they based their as-
sessments.
They examined the extent to which the specified attributes are
distinguishable from each other and whether the raters agree with each
other upon the attributes associated with each test item (a type of inter-
rater reliability effort).
With reference to Li (2011) and Jang (2005), a coding scheme was
created based on the cognitive framework and think-aloud data. An
initial Q-matrix was constructed based on evidence from the think-
aloud verbal report, the expert rating, and the coding scheme extracted
from the literature. However, a frequently encountered problem here is
that students’ verbal reports may not agree with expert rating (Jang,
2005; Leighton & Gierl, 2007). When this discrepancy occurred in the
present study, the think-aloud verbal reports were regarded as the
primary evidence, because the verbal reports captured the real-time
reading process to a certain extent and were thus considered more re-
liable and authentic. Nevertheless, the value of expert rating should not
be underestimated because it provides important evidence from a dif-
ferent perspective. Furthermore, when it was difficult to determine
whether a certain skill should be retained for an item, the skill was
usually retained. This is because the resulting RUM calibration provides
evidence concerning the importance of the skill for the item; that is, if
the calibration showed the skill to be insignificant, it could be dropped
at the later stage. Via RUM analyses, the test takers’ (N = 1986) test
performance data were used to detect relevant item characteristics and
then to refine the Q-matrix by the use of statistical analysis through the
RUM.
5.2. L2 reading attributes
The list of reading attributes (as shown in Table 2) was developed
based on previous literature (Birch, 2002; Cohen & Upton, 2006;
Fletcher, 2006; Francis et al., 2006; Jang, 2009; Rupp, Ferne, & Choi,
2006), content expert judgment (6 content raters), and examinees’
think-aloud verbal protocols (13 students). This list of attributes largely
consisted of language knowledge and strategies, required when a
learner's language ability interacted with the written text. Language
knowledge was further categorized into grammatical knowledge (i.e.,
grammatical form and semantic meaning) and pragmatic knowledge,
while strategic competence was composed of metacognitive strategies
(i.e., assessing the situation, monitoring, evaluating) and cognitive
strategies (i.e., comprehension, memory, retrieval). Each of these ca-
tegories consisted of a number of L2 reading attributes (e.g., knowl-
edge, strategies). In addition, language knowledge and strategic com-
petence were presumed to interact with each other (Bachman & Palmer,
1996). Considering that strategic competence manages all language
use, which includes using language knowledge (Bachman & Palmer,
1996), test-takers needed to utilize strategies to invoke their language
knowledge. In this study, the list included those L2 reading attributes
that were considered to be involved in the reading process, comprised
of four knowledge-based attributes (attributes 1–4 in Table 2) and five
reading strategies (attributes 5–9 in Table 2).
Table 2
Attributes of L2 reading ability.
L2 Reading Attributes
A1 determining word meaning from context
A2 determining word meaning out of context
A3 comprehending text-explicit info
A4 comprehending text-implicit info
A5 skimming
A6 summarizing
A7 inferencing
A8 applying background knowledge
A9 inferring major ideas or writers purpose
F. Ranjbaran, S.M. Alavi
6. Procedure
The study was carried out in three stages; 1) Developing the reading
comprehension test based on a cognitive assessment framework that
includes constructing items that target the specified attributes. 2)
Constructing and validating a Q-Matrix of i test items by K reading
attributes; 3) RUM based statistical analysis of data resulting from a
large-scale test administration. The first stage of the study was to carry
out an extensive study on the literature pertaining to cognitive diag-
nostic assessment and test development for the purpose of developing a
L2 reading comprehension test based on a cognitive diagnostic frame-
work. From the review of literature, an initial conceptualization of test
specifications based on Evidence Centered Design put forth by Mislevy
(1996) was put to use in developing the 20 MC-items for the test. The
author spent one term of teaching the general English course to gather
data to create a list of suggested attributes, which where then used to
develop the test items. Focusing on test specification for developing the
test items required an extensive literature survey to better understand
the ECD framework, which aims to gather evidence to support claims
about an individual. As Mislevy (1994) puts forth, this framework is
composed of three models or components, the student model, the evi-
dence model, and the task model. The student model includes the types
of inferences that are made about a student. In this study, after working
on reading comprehension skills and strategies in class, reading pas-
sages were assigned to students in groups and they were asked to re-
spond to reading comprehension questions. Observations were made
about the student’s responses and skills and strategies that were more
frequently used in their responses. Evidence was collected through
group work exercises and feedback from students. This supports the
claim that the evidence model gives evidence to support the inferences
made, and the task model provides tasks that elicit usable pieces of
evidence (Mislevy, 1994).
It should be emphasized that the process of assessment design is not
necessary linear (Roussos et al., 2007); hence, although defining attri-
butes typically precedes task construction, the feasibility of task con-
struction that measures a set of attributes may necessitate going back to
how the attributes have been defined. In this process, task construction
informs attribute definition, which will then inform the next phase of
task construction.
In this study, data gathered from observations of student responses
during the course, evidentiary reasoning, specifically the task model by
using tasks to elicit usable pieces of evidence during the course, were all
used in constructing the reading comprehension test items. After the
test was developed, it was administered to 1986 students in general
English courses at the University of Tehran. Data from this test ad-
ministration was to be used for RUM based statistical analysis. The
second stage was constructing a Q-matrix based on the chosen L2
reading attributes and the 20 constructed items. In order to construct
the Q-matrix, initially a list of L2 reading attributes was specified based
on the previous literature, the 13 participants’ think-aloud verbal re-
ports, and the 6 content experts’ judgment. The third and final phase
consisted of the reading test data being analyzed using the Q-matrix and
the Arpeggio suite software (Stout, 2008a, 2008b; Stout, 2008a,
2008b), which implements the RUM. This also resulted in an empirical
validation of the proposed Q-matrix.
7. Instruments
The test used was the reading comprehension test developed for this
study. The test was developed based on a cognitive diagnostic frame-
work, including three different passages along with 20 multiple-choice
items. Table 3 shows the structure of the developed reading compre-
hension test. The passages varied content-wise (descriptive/in-
formative, causation/process, and comparison/contrast), topic-wise
(animal intelligence, communication satellites, and body temperature),
and by length (323, 318 and 257 words). In addition, among the 20

6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²
Table 3
Structure of the reading test.
Passage Topic Content structure Length Number of
items
Passage 1 Animal Intelligence Description/ 323 words 7
Informative
Passage 2 Communication Causation/ 318 words 7
Satellites Process
Passage 3 Body Temperature Compare/ 257 words 6
Contrast
items, fourteen of the items were related to identifying semantic
meaning, and six items required identifying the pragmatic meaning of
the text. In this study, reading for identifying semantic meaning refers
to reading for literal and intended meaning. Reading for literal meaning
refers to identifying information conveyed in the text through para-
phrasing or translating, and reading for intended meaning refers to
obtaining the meaning of sentences by making connections between
them. Pragmatic meaning refers to contextualized implied meaning,
such as contextual, sociocultural or psychological meanings. Therefore,
reading for pragmatic meaning refers to deriving a deeper under-
standing of the text by combining the information with readers' specific
prior knowledge and experiences (Purpura, 2004).
8. Data analyses
Both qualitative and quantitative analyses were carried out in the
process of test development and Q-matrix construction. Initially, qua-
litative analyses were carried out to specify reading skills assessed by
the reading test. For this, various classifications of reading skills and
strategies in the literature were studied. Then, think-aloud verbal pro-
tocols were analyzed qualitatively to help understand the character-
istics of the cognitive processes and skills used by the students and to
identify primary reading skills. Six content rater’s judgment was also
used to examine to what extent the specified skills are necessary to
correctly answer the test items. Reading test data were analyzed to-
gether with the Q-matrix using the Arpeggio Suite software
(DiBello & Stout, 2008b). The first step in the RUM analysis is the
analysis of the Markov Chain Monte Carlo (MCMC) convergence to
guarantee that model parameters had a stable value (Roussos et al.,
2007). Arpeggio software uses a Bayesian modeling approach with a
Markov Chain Monte Carlo (MCMC) algorithm. “The MCMC estimation
provides a jointly estimated posterior distribution of both the item
parameters and the examinee parameters (given the test data), which
may provide a better understanding of the true (estimation’s) standard
errors involved” (Patz & Junker, 1999). MCMC convergence is mainly
evaluated by visually examining the time–series chain plots and esti-
mated post-burn in posterior probability density plots, which, if con-
vergence has occurred, should be roughly unimodal and roughly bell-
shaped. By using the results from the Arpeggio software, the R statis-
tical package (The R Foundation for Statistical Computing, 2010) pro-
duces both the chain plot (time series plots) and the density plot for
each parameter. The chain plot graphically indicates the degree to
which the chain has converged to a desired value after searching the
space. Ideally, a chain should sample values over the parameter space,
but also around a certain location. That is, a chain should converge to a
specific value across time and reach a stationary state (Johnson, 2006).
Usually the first portion of the chain is used as a burn-in and is dis-
carded. For example, in a chain of 5000, the first 1000 steps will be
used as the burn-in.
In this study, since a total of nine L2 reading attributes were ex-
amined, nine pk values were produced from the RUM analysis and a
chain length of 100,000 was initially used to reach MCMC convergence.
It is generally stated that the more complex the model (as determined
by the number of item parameters—which, for RUM, is related to the
F. Ranjbaran, S.M. Alavi 6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²

number of skills and the average number of skills per item), the longer Table 4
the required chain length (Roussos, Templin, & Henson, 2007, p. 299). Q-matrix of Attributes.
Chain plots and density plots of the nine reading attributes indicated
I/A A1 A2 A3 A4 A5 A6 A7 A8 A9
convergence of the parameter estimates.
Among the different parameters, first the convergence of examinees’ 1 1 0 1 0 1 0 0 0 0
probability of mastery for each attribute (pk) was evaluated overall. 2 1 0 1 0 1 0 0 0 1
3 1 0 1 0 1 0 0 0 0
This estimates for each attribute the proportion of students mastering
4 0 0 1 0 1 0 0 1 0
the attribute, information useful for classroom instructional purposes. 5 0 0 0 1 1 0 1 0 0
Also, three item-based parameters that indicate the item difficulty for 6 1 0 1 0 1 0 0 0 0
item masters (πi*), item attribute k specific discrimination power (rik*), 7 1 0 0 0 1 0 1 0 0
and item completeness (ci) were evaluated. In fact, ci is a measure of 8 1 1 0 0 0 0 0 0 1
9 0 1 1 0 1 0 0 0 0
being inversely proportional to the amount of model-based examinee
10 1 1 0 0 0 0 0 0 0
responding not captured by the pi and r parameters. 11 0 0 1 0 1 0 0 0 0
Here an “item master” is defined to be an examinee that possesses 12 1 1 0 0 0 0 0 1 0
all the attributes (skills) required for the item in question. In this sec- 13 0 0 0 1 1 0 1 0 0
14 0 0 1 1 0 1 1 0 1
tion, examinees’ L2 reading performance on the reading comprehension
15 0 0 1 0 1 0 0 0 0
test was evaluated in terms of their mastery and non-mastery of L2 16 1 1 0 0 0 0 0 1 0
reading attributes. 17 0 0 1 0 1 0 0 0 0
Fit statistics were calculated to evaluate the fit of the model to the 18 0 0 0 1 1 0 1 0 0
data. The two types of fit statistics measured are named FUSIONStats 19 1 0 0 0 1 1 0 0 0
20 0 0 0 1 0 1 1 0 1
and IMStats, or item mastery statistics, both explained below. The first
compares the difference between the observed proportion of examinees
getting the item correct and the predicted proportion of examinees obtained are listed in Table 2. Due to the complicated nature of reading,
getting the item correct. A low difference between the two p-values numerous reading attributes are involved in completing each item
indicates a good fit of the data. IMStats indicate a comparison of the (Alderson, 2000; Urquhart & Weir, 1998), as was the case with this
observed performance of item masters and item non-masters (defined as study. Also, it is not claimed that the choice of the 9 attributes is un-
examinees lacking at least one of the item’s required attributes) at the iquely better than some other choice. However, it seems clear that these
item level. Sizable differences between the performance on most items 9 have been well chosen and further do provide good reliability and as
by item masters versus item non-maters indicates a good fit of the such provide cognitively and instruction-wise useful information.
model and the specified Q-matrix to the data. Importantly, a sizable
difference for a specific item is an indication of the high quality of the
item. Also, the reliability of the RUM was examined by analysing the 9.2. Fusion model analysis
Correct Classification Rate (CCR), which is the consistency of classifi-
cation of examinees into masters versus non-masters of attributes. That To statistically assess the 9 identified attributes in the initial Q-
is, CCR (k) measures the proportion of masters of Attribute K that are matrix and evaluate the 20 written items, RUM analysis was conducted
classified as masters and the proportions of nonmasters classified as using Arpeggio software. The item parameter estimates were analyzed
nonmasters, this at the attribute level for each k, CCR denotes the to evaluate the quality of the reading test items. As discussed above,
average of the CCR(k)s. These are calculated in practice via a simula- three types of item parameter estimates were examined in the current
tion process; for details see Roussos et al. (2007) and DiBello and Stout study: item difficulty (π*), item attribute discrimination (rik), and the
(2008a). In the final step, the population of examinees’ strengths and completeness index (ci). These Fusion Model parameters have im-
weaknesses in L2 reading ability at the attribute level were evaluated portant roles because they not only provide diagnostic information
through probability of mastery for each attribute (pk). about each test taker and item, but also highlight the properties of the
Q-matrix and possible Q-matrix misspecifications observed (Aryadoust,
9. Results 2011).
The πi* parameter is the probability that an examinee, having
9.1. Q-Matrix development mastered all the Q-matrix required skills for item i, will correctly apply
all these skills to solving item i. A high item probability value above 0.6
Results from think-aloud verbal protocols and content rater’s judg- indicates that examinees will have a good chance of correctly re-
ment, in conjunction with consultation of the L2 reading literature, sponding to the item if they have mastered all the necessary attributes
were analyzed to develop the list of reading attributes. This list of (Johnson, 2006), thus having a higher probability of successfully ap-
reading attributes was then used to develop the Q-matrix. In the Q- plying the skills required by that item. The item discrimination index
matrix, the rows represent items and the columns correspond to the (rik) shows how well the item discriminates masters from non-masters.
attributes. 1 means that the particular attribute is required for the A reasonably well defined Q matrix and a reasonably well written
completion of the item, whereas 0 means that it is unnecessary. Since item combine to produce estimates r*ik below the 0.8 cutpoint, in-
the test contained 20 items, the Q-matrix for the current study re- dicating sufficient item discrimination of masters from non-masters, as
presented the relationship between 20 items and their corresponding believed by CDM experts. This is while values below 0.5 are regarded as
attributes. skills highly necessary to answer the question correctly. A low item
As shown in Table 4, the rows of the Q-matrix indicate the 20 items discrimination value below 0.3 indicates that the item discriminates
from the reading comprehension test, and the columns indicate the nine very well between masters and non-masters of the attribute, and hence
reading attributes. the item has a strong dependence on the corresponding attribute
Attributes that three or more of the 6 raters agreed upon were (Johnson, 2006).
considered as essential for the item and were included in the Q-matrix. In this study, the average π obtained was 0.833, indicating that the
According to Hartz (2002), those attributes that were measured by set of identified L2 reading comprehension skills for the items were
fewer than three items do not provide statistically meaningful in- generally adequate and reasonable. Results indicate that three items
formation about these attributes and therefore, can be merged with (i.e., items 2, 5, and 13) had π* values lower than 0.6, which means that
similar attributes or deleted from the Q-matrix. The nine attributes they were too difficult and may need some modification. However,


F. Ranjbaran, S.M. Alavi 6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²

Table 5
Item Parameter Estimates.

Item π* r* 1 r* 2 r* 3 r* 4 r* 5 r* 6 r* 7 r* 8 r* 9 ci

1 0.879 0.895 0 0.827 0 0.568 0 0 0 0 1.97

2 0.583 0.870 0.847 0 0 0 0 0 0 0.678 2.12
3 0.963 0.923 0 0.899 0 0.726 0 0 0 0 2.73
4 0.786 0 0 0.835 0 0.631 0 0 0.877 0 2.43
5 0.376 0 0 0 0.843 0.926 0 0.722 0 0 2.76
6 0.804 0.694 0 0.709 0 0.490 0 0 0 0 2.24
7 0.993 0.860 0 0 0 0.809 0 0.944 0 0 2.61
8 0.950 0.423 0.443 0 0 0 0 0 0 0.811 2.68
9 0.962 0 0.694 0.536 0 0.853 0 0 0 0 2.70
10 0.769 0.531 0.580 0 0 0 0 0 0 0 2.02
11 0.974 0 0 0.594 0 0.759 0 0 0 0 1.60
12 0.789 0.828 0.521 0 0 0 0 0 0.757 0 2.47
13 0.557 0 0 0 0.704 0.883 0 0.730 0 0 2.04
14 0.894 0 0 0.544 0.863 0 0.650 0.822 0 0.786 2.54
15 0.979 0 0 0.812 0 0.713 0 0 0 0 0.87
16 0.940 0.939 0.941 0 0 0 0 0 0.878 0 1.05
17 0.661 0 0 0.887 0 0.821 0 0 0 0 0.65
18 0.934 0 0 0 0.909 0.860 0 0.884 0 0 0.74
19 0.997 0.981 0 0 0 0.938 0.980 0 0 0 1.72
20 0.863 0 0 0 0.738 0 0.804 0.826 0 0.719 0.80

items 2 and 13 with a π* value of 0.58 and 0.56, respectively, were less 19. Interestingly, items 16–18 and 20 do suggest the possibility that the
of a concern than item 5 with a attribute space is incomplete and that one or more attributes could be
π* value of 0.38, which should be replaced. This illustrates that added to the 9 attributes. The above discussion demonstrates that item
estimated item parameters can drive decisions to replace one or more parameter estimates do provide substantially meaningful information
items that, when carried out, will definitely improve the diagnostic regarding the quality of the items. Thus the next iteration of the test can
power of the test. The item parameter estimates produced from the be considerably improved, based on the RUM analysis.
current study are presented in Table 5.
Examining the table, it is observed that all r*s were below 0.9, in-
dicating that all the items were discriminating at least somewhat be- 9.3. Analysis of model fit
tween masters versus non-masters of the attributes. It is observed that
most items except 7, 16, 17, 18 and 19 have at least one r < 0.8, hence In order to examine the fit of the RUM to the data, two types of
discriminating at least one attribute sufficiently. This indicates that the goodness of fit measures were used: (1) FUSIONStats and (2) item
majority of the test-takers, including both masters and non-masters of mastery statistics (IMStats). FUSIONStats compares the difference be-
the attributes, were slightly influenced by the specified attributes. tween the observed item p-value (the proportion of examinees getting
When such a result is obtained, we must come to the conclusion that the item correct) and the estimated item p-value (the model estimated
revising or eliminating/replacing these items on the test with replace- proportion of examinees that should get the item correct) for each item.
ment items that better discriminate between the masters and non- A small difference between the two for most or ideally all of the items
masters of the specified attributes will create a better performing di- suggests a good fit of the data. Table 6 shows the observed item p-
agnostic test. Indeed we have 1/5 of the test items essentially providing values and the estimated item p-values for each item. The absolute
no diagnostic power. Thus, replacing them with effective items would difference between each observed item p-value and the estimated item
make a major difference in the discrimination power of the test.
Therefore, the r* parameter could be used to evaluate the quality of an Table 6
Observed and estimated item p-values.
item in terms of its discriminatory power (Leighton & Gierl, 2007).
The completeness index, ci, shows the degree to which the attributes Item Observed p-value Estimated p-value Absolute observed-estimated p-
specified in the Q-matrix are complete in describing the skills needed to value
successfully respond to the ith item, and ranges from 0 to 3. Values
1 0.660 0.670 0.01
close or even fairly close to zero, say ≤1.25 (somewhat arbitrary) in-
2 0.461 0.498 0.037
dicate that the item has low completeness, that is, other unspecified 3 0.825 0.847 0.022
skills are important for answering item i correctly. ci values somewhat 4 0.615 0.649 0.034
close to 3 indicate that the specified attributes in the Q-matrix suffice to 5 0.314 0.332 0.018
explain examinee responding to that item (von Davier, DiBello, and 6 0.554 0.607 0.053
7 0.872 0.898 0.026
Yamamoto, 2006). From a brief glance at the table it is observed that 8 0.629 0.722 0.093
items 1–14 and 19 have an overall high completeness index ≥1.5 and 9 0.713 0.730 0.017
in all but two cases ≥2, while items 15–18 and 20 show a low com- 10 0.537 0.602 0.065
pleteness index of 0.65 to 1.05. This indicates that the specified attri- 11 0.700 0.737 0.037
12 0.580 0.625 0.045
butes selected for items 15–18 and 20 have attributes besides the spe-
13 0.433 0.456 0.023
cified attributes that seriously impact examinee responding. 14 0.603 0.671 0.068
Interestingly, items 16 and 18 are virtually useless in discriminating 15 0.630 0.673 0.043
among the specified attributes. Item 19 is also useless in discriminating 16 0.678 0.719 0.041
among the specified attributes but it is seemingly not influenced by one 17 0.413 0.439 0.026
18 0.595 0.642 0.047
or more attributes lying outside the specified latent space as its 19 0.863 0.887 0.024
c = 1.72 suggests. The result is that the item is both useless for our 20 0.482 0.570 0.088
purposes and also very easy. Item 7 has almost the same story as Item Mean Absolute difference: 0.040


F. Ranjbaran, S.M. Alavi 6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²

Fig. 1. Observed vs. Estimated Item p-value.

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

p-value for each item should be below the suggested value of 0.05 for attributes on average by 45.2%. This high value implies a good fit be-
all items as put forth in (Roussos et al., 2007). However, in this case this tween the estimated model and the observed data, and moreover in-
value was higher than 0.05 for four items out of twenty, in particular dicates the strong diagnostic power of this L2 reading test. Further, it
item 8 at 0.09, item 10 at 0.06, item 14 at 0.06 and item 20 at 0.08. This indicates the usefulness of the Fusion modeling approach for carrying
suggests these items may have moderate lack of fit problems. However, out cognitive diagnostic studies such as the one reported on herein.
the mean absolute difference between the p-values was low at 0.04, In addition, the reliability of the RUM was examined by evaluating
suggesting good overall fit. Fig. 1 graphically depicts the results of the Correct Classification Rate (CCR) index, which as discussed above is
observed versus estimated item p-values, which suggests that the RUM the diagnostic power to correctly classify examinees. The output file
fits the data well. from the Arpeggio Tabulator, called classfile.csv, reports the estimated
In addition, item mastery statistics (IMStats) were used to compare correct classification rate CCR(k) for each skill k with CCR the average
the observed performance of item masters and item non-masters at the over all k. The CCR theoretically ranges between zero and one. In this
item level. Three different values are considered to evaluate IMStats: data set, the CCR was high at 0.826, indicating a high reliability of the
the phat (m), which refers to the observed proportion correctly re- RUM and in particular the capacity to correctly classify examinees over
sponding to an item among item masters; the phat (nm), which refers to 80% of the time.
the observed proportion correctly responding to an item among item Population mastery probability of each of the reading attributes was
nonmasters; and the pdiff, which indicates the average difference be- also analyzed for the student’s L2 reading performance. For the overall
tween phat (m) and phat (nm) across items. In this study, as shown in test-taking group, the population’s probability of mastery for each at-
Table 7, the average phat (m) across all items was 0.774, which in- tribute (pk) was investigated. Table 8 shows the L2 reading attributes,
dicates that the average observed probability of getting a correct re- and their pk values that range from a high of 0.769 (determining word
sponse to an item by item masters was relatively high at 77.4%. On the meaning from context) to 0.627 (comprehending text-implicit info).
other hand, the average phat (nm) was 0.322, indicating that the This indicates that 76.9% of the students had mastery of determining
average observed probability of having a correct response to an item by vocabulary from context, making it the easiest attribute. This is while
item non-masters was much lower at 32.2%. Thus, the pdiff was 0.452, 62.7% of the students had mastery of comprehending text-implicit info,
indicating that item masters outperformed item non-masters of making it the more difficult attribute. The overall average level of
mastery probability for the nine L2 reading attributes was 0.701. These
population level statistics should prove useful for instructional pur-
Table 7 poses.
Probability of Correctly Responding to an Item.
Among the four knowledge-related attributes (the first four in
Item phat(m) phat(nm) phat(m) − phat(nm) Table 8), there were small differences in the overall groups' attributes
mastery probabilities, ranging from 0.627 for comprehending text im-
Item 1 0.841 0.398 0.443 plicit information (pragmatic meaning) to 0.769 for deducing word
Item 2 0.543 0.398 0.214
meaning from context (lexical meaning). This signified that “prag-
Item 3 0.957 0.609 0.348
Item 4 0.772 0.380 0.392 matic” word meaning was the most difficult attribute to master,
Item 5 0.358 0.197 0.161 whereas “lexical” word meaning was the easiest to master on this test.
Item 6 0.788 0.278 0.51
Item 7 0.982 0.565 0.417
Item 8 0.958 0.181 0.777 Table 8
Item 9 0.960 0.312 0.648 Attribute mastery probability for overall group.
Item 10 0.718 0.209 0.509
Item 11 0.919 0.334 0.585 L2 Reading Attribute Mastery probability (pk)
Item 12 0.780 0.128 0.652
Item 13 0.552 0.169 0.383 deducing word meaning from context 0.769
Item 14 0.864 0.091 0.773 determining word meaning out of context 0.676
Item 15 0.812 0.368 0.444 comprehending text-explicit info 0.662
Item 16 0.790 0.426 0.364 comprehending text-implicit info 0.627
Item 17 0.513 0.301 0.212 skimming 0.722
Item 18 0.751 0.393 0.358 summarizing 0.698
Item 19 0.938 0.753 0.185 inferencing 0.754
Item 20 0.684 0.012 0.672 applying background knowledge 0.666
Average 0.774 0.322 pdiff = 0.452 inferring major ideas or writers purpose 0.737


F. Ranjbaran, S.M. Alavi
Among the five reading strategies, the attribute mastery probability of
strategies ranged from 0.666 (applying background knowledge)to
0.737 (inferring major ideas). In other words, about 67% of examinees
had mastery of applying background knowledge, making it the most
difficult strategy. On the other hand, about 74% of the examinees had
mastery of inferring major ideas, making it the easiest strategy.
10. Discussion and conclusions
Among the four skills of English language proficiency, reading
ability might well be considered the most essential skill for success in
the academic world. Hence, it is crucial to accurately assess learners' L2
reading ability to gauge their overall progress and in particular help
enhance specific reading skills where lack of mastery is indicated. The
main goal of the current study was to develop a L2 reading test using
the ECD framework and based on a cognitive diagnostic model (in our
case the RUM). This is being done in order to diagnose learners'
strengths and weaknesses in L2 reading ability, with the ultimate goal
of providing detailed information that can assist teachers and admin-
istrators for instructional purposes and to thus improve student per-
formance.
In fact, two research questions were carried out in the course of this
study. First was investigating the L2 reading attributes necessary for
successfully completing every item on the reading test, noting that the
20 item test was then carefully designed to assess mastery of each of
these 9 attributes judged to capture the reading comprehension latent
space. Raters identified the various attributes, consisting of both
knowledge and strategy attributes, by referring to the L2 reading lit-
erature and students think-aloud verbal reports. This list of nine reading
attributes and the developed test were then organized into an item-by-
attribute Q-matrix.
Second, test-takers' performance on the reading test was examined
for individual student and group diagnostic purposes. In particular, the
test scores were then analyzed in conjunction with the initial Q-matrix
(see Table 4) using the Fusion model analysis.
Findings of the study suggest that a majority of the items on the test
can successfully discriminate between masters and non-masters and are
therefore appropriate for CDA. In addition, the list of L2 reading at-
tributes can be used as a framework for CDA research in the future.
Regarding the frequency of reading attributes for each item on the test,
the Q-matrix was examined to identify any recurring patterns among
the L2 reading attributes. Since language knowledge and strategic
competence are believed to interact with each other in constituting the
mastery of reading ability (Bachman & Palmer, 1996), it was expected
that each item on the reading test would measure at least one knowl-
edge-related attribute and one reading strategy. In fact, it was indeed
found that almost all items measure one knowledge-related attribute
and a minimum of one reading strategy. Since strategies, by definition,
manage the use of language knowledge (Bachman & Palmer 1996), they
were assumed necessary to reveal mastered knowledge-related attri-
butes.
Examinees’ performances on the attributes existing in reading
comprehension items were evaluated. As Roussos et al. (2007, p. 293)
put forth, “A key issue for mastery/nonmastery of diagnostic models is
whether the proportion of examinees estimated as masters on each skill
is relatively congruent with the user’s expectations.” RUM analysis was
carried out to obtain relationships among the participant’s perfor-
mances on the test items. The Arpeggio suite software provides a
number of output files that give specific information regarding ex-
aminees’ performance on each item of the test. Two of the output files
that help us respond to research question 2 (essentially studying ex-
aminee/test attributes relationships) are the classification file (classfi-
le.csv) and the fit report file (fitreports.csv). The fit report file is an
output file that provides fit statistics for how well the model fits the test
data, noting that a good fit is required for usefulness of the RUM based
statistical analyses. The classification file indicates the consistency of

6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²
classifying the examinees in terms of their mastery or nonmastery of
each attribute (i.e., the correct classification rate: CCR). First of all, the
model fit the data well for several reasons. First, the mean absolute
difference between the observed and estimated item p-values was low
at 0.04 (see Table 6). In addition, the average phat (m) across all items
was 0.774, indicating that the average probability of getting a correct
response to an item by item masters of attributes was relatively high at
77.4%. This suggests that the 9 attributes include much of what is re-
quired for answering the test’s items correctly. The average phat (nm)
was 0.322, so the average probability of having a correct response to an
item by non-masters of attributes was much lower at about 32.2%. This
indicates the items, on average, are “hard” for non-masters. In addition,
the pdiff was 0.452, indicating that the masters of required attributes on
an item outperformed non-masters of required attributes on average by
45.2% across all items. This high value indicated a good fit between the
estimated model and the observed data, and further, suggesting a strong
diagnostic power of the test as modelled by the RUM.
According to Lumley (1993), identifying implicit information
(equivalent to inferencing) and synthesizing to draw a conclusion
(equivalent to summarizing) were difficult reading attributes compared
to vocabulary (similar to identifying word meaning) and identifying
explicit information (similar to finding information and skimming).
This could be attributed to the fact that inferencing and summarizing
are higher-level strategies involving more complex cognitive processing
than the other three strategies, which require lower-level strategies as
was the case in this study. As an example, summarizing requires readers
to first comprehend the overall text and then extract the gist of meaning
from it. Understanding the gist involves numerous components, such as
knowledge of grammar, vocabulary, discourse structure, and various
cognitive processes (Birch, 2002). Thus, the nature of summarizing
seems quite complex. In a similar vein, the strategy of inferencing has
long been believed to be a challenging one (Fletcher, 2006). In order to
make inferences, readers should already have the ability to understand
the literal meaning of the text, which makes inferencing more difficult
to master. However, the results of this study are contrary to this belief,
whereas for the overall group, the attribute of inferencing stands in
second at 0.754 with less difficulty. On the other hand, between
skimming and summarizing, the latter was more difficult with 0.698 as
compared with skimming at 0.722. In fact, as put forth by Urquhart and
Weir (1998), skimming involves quickly understanding the surface-
level propositional meaning of the text and is considered a less chal-
lenging strategy.
Furthermore, comprehending text implicit information, compre-
hending text explicit information and applying background knowledge
were the more difficult attributes with 0.627, 0.662 and 0.666, re-
spectively. Similarly, deducing meaning from context the easiest attri-
bute, which is in accordance with the belief that word recognition,
which is similar to identifying word meaning, involves lower-level
processing (Alderson, 2000). Overall, examinees have performed com-
paratively well on these three attributes (i.e., deducing meaning from
context, inferring major ideas and skimming) due to the nature of the
attribute, which requires relatively less cognitive processing. The de-
tailed classification student level reports made possible by the RUM
analysis of the L2 reading assessment of this study strongly indicate that
L2 reading assessments using this test should be highly beneficial in
facilitating learning on the part of individual students and in teacher
instructional preparation as the course proceeds and finally in curri-
culum development on the part of the teacher. With such detailed re-
ports of test results available, teachers can become aware of students’
problematic areas and focus on them in lesson planning and providing
learning materials. Since the reading test developed here was based on
a cognitive framework followed by RUM analysis, the problematic
items were identified and could be further modified or even replaced.
Hence, an item bank can be developed for cognitive diagnostic devel-
opment items in L2 reading comprehension testing for use during or
after a reading comprehension unit is carried out, and thus for future
F. Ranjbaran, S.M. Alavi
studies.
11. Limitations, strengths and future research
The current study provides a number of implications for teachers
and practitioners, both pedagogically and theoretically. First and fore-
most are the positive pedagogical implications for assessment purposes
implied by the successful attempt at constructing a cognitive diagnostic
test to gauge L2 reading proficiency. The type of diagnostic feedback
that can be provided to teachers includes attribute mastery probabilities
for overall groups and individual students. Teachers can refer to this
information to refine their lesson plans and provide L2 reading material
necessary to meet the needs of various mastery levels.
Developing a test based on a cognitive diagnostic framework re-
quires much focus on the diagnostic components of test design. By using
a RUM statistical analysis, the quality of test items was evaluated. The
item mastery probability difficulty values indicated which items were
too difficult given the mastery of the attributes involved at the item
level, while the item discrimination values indicated which items dis-
criminated between masters and non-masters of a particular attribute,
showing that some items were deficient in this regard. The item com-
pleteness values indicated that for some items the Q-matrix lacks cer-
tain skills that are important for answering those items correctly and
therefore the Q-matrix should be refined either by modifying entries or
perhaps adding a skill or two.
Generally, item parameter estimates could be used for improving
the quality of reading test items. Indeed, in this case, four of the items
need either serious modification or replacement. By providing in-
formation regarding item mastery probability difficulty estimates to
teachers, they can use the detailed information to modify or replace the
problematic items to enhance the quality of reading comprehension test
items. Also, since item discrimination estimates indicate which items
discriminated better between masters and non-masters of certain at-
tributes, items similar to those with high discriminatory power can be
added to the test to perhaps considerably improve it.
As the process of all research faces some limitations, the present
study might also suffer from some limitations. While determining the
attributes that constitute the latent knowledge space of a CDM model
necessitates a deep understanding of the nature of these cognitive skills
and the resulting test items, the complexity of reading comprehension
space does not allow a full understanding of its cognitive processes
(Lee & Sawaki, 2009). In addition, there is a lack of consensus on the
skill components of reading comprehension (Alderson, 2000). Further,
although a great number of attributes may be identified as related to
reading comprehension mastery, not all such attributes can, or should,
be kept in the Q-Matrix for RUM analysis. As a result, the purpose is not
to identify all the attributes that could be involved in correctly/in-
correctly responding to the reading comprehension items, but to find
the major attributes required to successfully complete each item.
Therefore, the reading attributes found are not exhaustive to capture all
of reading comprehension, but are specifically related to the reading
comprehension test, which indicates there may be a need for more at-
tributes, or simply that an item might be measuring inappropriate at-
tributes. Of course, the better the test, the closer the specified test-de-
pendent attributes are to capturing most of the reading comprehension
latent space. Meanwhile, those attributes that were identified for this
study together with their Q-matrix were not effectively discriminated
by all the non-easy items, specifically not for items 15–18, 20. Thus,
exploration of additional attributes to be used for Q-matrix construction
is recommended for future studies of L2 reading test construction for
formative assessment purposes.
In addition, the reading comprehension test that was herein devised
based on a cognitive diagnostic framework is only at relatively early
stages of experimentation. Further research should be carried out to
devise more cognitively based L2 language tests, not only for the
reading skill, but also for writing and listening assessments. Devising

6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²
cognitively-based assessments requires numerous pilot studies and
carrying out further research for one content area such as L2 reading or
more generally other L2 areas can only improve the accuracy of items
developed and thus lead to more accurate formative assessment attri-
bute level classification accuracy.
One of the limitations of the study was the format of the reading
comprehension test, which was in the form of strictly multiple-choice
items. In future studies, other item types such as fill-in-the-blank, short
essay type and open-ended questions should be considered. Moreover,
corresponding modifications in the Fusion modeling approach are thus
called for. Also, another suggestion for future research is to focus on
item distractors to enhance the diagnostic potential of the test. The
proportion of diagnostic information that is obtained from a diagnostic
test greatly depends on the test design and item construction.
Specifically, in a retrofitting approach, many or even most test items
will fail to diagnostically differentiate examinees based on their un-
derlying skill competencies as well as items specifically designed to do
so, as is the case for the test in this study. However, test/item diagnostic
discrimination for either a retrofitted or a specifically cognitive diag-
nosis designed test can be enhanced by observing incorrect response
patterns of examinees. This necessitates developing diagnostically
sensitive distractors in MC items (DiBello et al., 2015; DiBello,
Roussos, & Stout, 2015).
Another limitation of the study is that the Q-matrix, which is es-
sential in the RUM analysis, could be developed by a greater number of
content experts, including their undergoing training sessions prior to
rating. In the current study, six context experts chose the L2 reading
attributes and Q-matrix coded them, which is sufficient for the purpose
of the study, but having a greater number of experts could possibly
make the Q-matrix more effective and the latent space more complete.
Since the quality of the Q-matrix determines the quality of the RUM
analysis, it is important to take extreme precautions in developing it
(Jang, 2005).
While this study was an attempt at diagnostic assessment of L2
reading attributes, it also demonstrates a need for continued research in
the area of cognitive diagnostic assessment in the Iranian context,
particularly with regard to constructing diagnostic tests for different L2
language skills including writing, speaking and listening.
In conclusion, now is the time to focus attention on designing and
developing educational assessments that are based on a CDM frame-
work, perhaps using the Fusion modeling approach, for the Iranian
context, the L2 language acquisition context, and for other learning
contexts, such as the STEM disciplines for instance. Carrying through
such an endeavour necessitates the cooperation of various experts from
different fields (i.e., subject matter, learning sciences, measurement,
pedagogy). By succeeding in such an effort, educational assessments
will become more instructionally oriented and more relevant to the
needs of present day classrooms. This paper establishes that such efforts
should be highly successful, probably in a great many subject areas, not
merely L2 English reading comprehension for first language Farsi col-
lege students.
References
Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press.
Alderson, J. C. (2005). Diagnosing foreign language proficiency: The interface between
learning and assessment. A & C Black.
Alderson, J. C., & Huhta, A. (2005). The development of a suite of computer-based di-
agnostic tests based on the Common European Framework. Language Testing, 22(3),
301–320.
Alderson, J. C. (2010). Cognitive diagnosis and Q-Matrices in language assessment: A com-
mentary. 96–103.
Aryadoust, V. (2011). Cognitive diagnostic assessment as an alternative measurement
model. SHIKEN: JALT Testing & Evaluation SIG Newsletter, 15(1), 2–6.
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. London: Oxford.
Birch, B. M. (2002). English L2 reading: Getting to the bottom. Routledge.
Birenbaum, M., Kelly, A. E., & Tatsuoka, K. K. (1993). Diagnosing knowledge states in
algebra using the rule-space model. Journal of Research in Mathematics Education, 24,
442–459.
F. Ranjbaran, S.M. Alavi
Buck, G., & Tatsuoka, K. (1998). Application of the rule-space procedure to language
testing: Examining attributes of a free response listening test. Language Testing, 15(2),
119–157.
Carrell, P. L., & Grabe, W. (2002). Reading. In N. Schmitt (Ed.). An introduction to applied
linguistics (pp. 233–250). London: Arnold.
Cohen, A. D., & Upton, T. A. (2006). Strategies in responding to the new TOEFL reading tasks.
Princeton, NJ: ETS [TOEFL Monograph No. MS-33].
DiBello, L. V., Stout, W. F., & Roussos, L. (1995). Unified cognitive psychometric as-
sessment likelihood-based classification techniques. In P. D. Nichols, S. F. Chipman, &
R. L. Brennan (Eds.). Cognitively diagnostic assessment (pp. 361–390). Hillsdale, NJ:
Erlbaum.
DiBello, L. V., Roussos, L. A., & Stout, W. (2006). 31a review of cognitively diagnostic
assessment and a summary of psychometric models. Handbook of statistics, 26,
979–1030.
DiBello, L. V., Henson, R. A., & Stout, W. F. (2015). A family of generalized diagnostic
classification models for multiple choice option-based scoring. Applied Psychological
Measurement, 39(1), 62–79.
DiBello, L., & Stout, W. (2008a). Arpeggio documentation and analyst manual. Chicago:
Applied informative assessment research enterprises (AIARE)–LLC.
DiBello, L., & Stout, W. (2008b). Arpeggio suite, version 3.1. 001 [Computer program].
Chicago: Applied informative assessment research enterprises (AIARE)–LLC.
Embretson, S. E., & Gorin, J. (2001). Improving construct validity with cognitive psy-
chology principles. Journal of Educational Measurement, 38(4), 343–368.
Fletcher, J. M. (2006). Measuring reading comprehension. Scientific Studies of Reading,
10(3), 323–330.
Francis, D. J., Snow, C. E., August, D., Carlson, C. D., Miller, J., & Iglesias, A. (2006).
Measures of reading comprehension: A latent variable analysis of the diagnostic as-
sessment of reading comprehension. Scientific Studies of Reading, 10(3), 301–322.
Gass, S. M., & Mackey, A. (2000). Stimulated recall methodology in second language research.
Routledge.
Goodman, D. P., & Hambleton, R. K. (2004). Student test score reports and interpretive
guides: Review of current practices and suggestions for future research. Applied
Measurement in Education, 7(2), 145–220.
Gu, Z. (2011). Maximizing the potential of multiple-choice items for cognitive diagnostic as-
sessment. University of Toronto [Doctoral dissertation].
Hartman, H. J. (2001). Developing students’ metacognitive knowledge and skills. In H. J.
Hartman (Ed.). Metacognition in learning and instruction: Theory, Research and Practice
(pp. 33–68). Dordrecht, the Netherlands: Kluwer.
Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive
abilities: Blending theory with practicality. Dissertation Abstracts International: Section
B: The Sciences and Engineering, 63(2-B), 864.
Huang, T. W., & Wu, P. C. (2013). Classroom-based cognitive diagnostic model for a
teacher-made fraction- decimal test. Educational Technology & Society, 16(3),
347–361.
Jang, E. E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and
learning in the context of NG TOEFL. [Available from ProQuest Dissertations and
Theses database. (AAT 3182288)].
Jang, E. E. (2009). Demystifying a Q-matrix for making diagnostic inferences about L2
reading skills. Language Assessment Quarterly, 6(3), 210–238.
Johnson, J. F. (2006). Diagnosing skill mastery in the national assessment of educational
progress: Applications of the Fusion Model. [Available from ProQuest Dissertations and
Theses database. (AAT 3223309)].
Kim, A. Y. A. (2015). Exploring ways to provide diagnostic feedback with an ESL pla-
cement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing
[0265532214558457].
Lee, Y. W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL
reading and listening assessments. Language Assessment Quarterly, 6(3), 239–263.
Leighton, J. P., & Gierl, M. J. (2007). Cognitive diagnostic assessment for education: Theory
and applications. Cambridge University Press.
Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The attribute hierarchy method for
cognitive assessment: A variation on tatsuoka's rule-Space approach. Journal of
Educational Measurement, 41(3), 205–237.
Li, H., & Suen, H. K. (2013). Constructing and Validating a Q-Matrix for Cognitive
Diagnostic Analyses of a Reading TestConstructing and validating a Q-Matrix for
cognitive diagnostic analyses of a reading test. Educational Assessment, 18(1), 1–25.

View publication stats
6WXGLHVLQ(GXFDWLRQDO(YDOXDWLRQ²
Li, H. (2011). Evaluating language group differences in the subskills of reading using a cognitive
diagnostic modeling and differential skill functioning approach. The Pennsylvania State
University [Doctoral dissertation].
Lumley, T. (1993). The notion of sub-skills in reading comprehension test: An EAP ex-
ample. Language Testing, 10(3), 211–234.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). Design and analysis in task-based
language assessment. Language Testing. Special Issue: Interpretations, Intended Uses, and
Designs in Task-based Language, 19(4), 477–496.
Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometrika,
59, 439–483.
Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33(4),
379–416.
Mislevy, R. J. (2006). Cognitive psychology and educational assessment. Educational
Measurement, 4, 257–305.
Patz, R. J., & Junker, B. W. (1999). Applications and extensions of MCMC in IRT: Multiple
item types, missing data: And rated responses. Journal of Educational and Behavioral
Statistics, 24, 342–366.
Pellegrino, J. W., Chudowsky, N., & Glaser, R. (Eds.). (2001). Knowing what students know:
The science and design of educational assessment (National research council’s committee
on the foundations of assessment. Washington, DC: National Academy Press.
Purpura, J. E. (2004). Assessing grammar. John Wiley Sons, Inc.
R Development CORE TEAM (2010). R: A language and environment for statistical com-
puting. Vienna, Austria: R Foundation for Statistical Computing ISBN
3–900051–07–0, URL: http://www. R-project.org.
Roussos, L. A., DiBello, L. V., Stout, W. F., Hartz, S. M., Henson, R. A., & Templin, J. H.
(2007). The fusion model skills diagnostic system. In J. Leighton, & M. Gierl (Eds.).
Cognitive diagnostic assessment for education: Theory and applications (pp. 275–318).
New York, NY: Cambridge University Press.
Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification
models: A comprehensive review of the current state-of-the-art. Measurement, 6(4),
219–262.
Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with
multiple-choice questions shapes the construct: A cognitive processing perspective.
Language Testing, 23(4), 441–474.
Rupp, A. A., Templin, J., & Henson, R. A. (2012). Diagnostic measurement: Theory, methods,
and applications. Guilford Press.
Sawaki, Y., Kim, H. J., & Gentile, C. (2009). Q-matrix construction: Defining the link
between constructs and test items in large-scale reading and listening comprehension
assessments. Language Assessment Quarterly, 6(3), 190–209.
Sheehan, K., & Mislevy, R. (1990). Integrating cognitive and psychometric models to
measure document literacy. Journal of Educational Measurement, 27, 255–272.
Snow, R. E., & Lohman, D. F. (1989). Implications of cognitive psychology for educational
measurement. American Council on Education.
Stiggins, R., Arter, J., & Chappuis, S. (2004). Classroom assessment for student learning:
Doing it right–using it well. Dover, NH: Assessment Training Institute.
Svetina, D., Gorin, J. S., & Tatsuoka, K. K. (2011). Defining and comparing the reading
comprehension construct: A cognitive-Psychometric modelling approach.
International Journal of Testing, 11(1), 1–23.
Tatsuoka, K. K. (1995). Architecture of knowledge structure and cognitive diagnosis: A
statistical pattern recognition and classification approach. In P. D. Nichols, S. F.
Chipman, & R. L. Brennan (Eds.). Cognitively diagnostic assessment (pp. 327–361).
Hillsdale, NJ: Lawrence Erlbaum Associates.
Templin, L. (1958). Generalized linear mixed proficiency models for cognitive diagnosis.
[Available from ProQuest Dissertations and Theses database. (AAT 3160960)].
Toulmin, S. (1958). The uses of argument. Cambridge, UK: Cambridge University Press.
Urmston, A., Raquel, M., & Tsang, C. (2013). Diagnostic testing of Hong Kong tertiary
students’ English language proficiency: The development and validation of DELTA.
Hong Kong Journal of Applied Linguistics, 14(2), 60–82.
Urquhart, S., & Weir, C. J. (1998). Reading in a second language: Process, product and
practice. New York: Longman.
de la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice
options. Applied Psychological Measurement, 33(3), 163–183.
von Davier, M., DiBello, L., & Yamamoto, K. Y. (2006). Reporting test outcomes with models
for cognitive diagnosis (ETS Research Rep. No. RR-06-28)Princeton, NJ: ETS.