Developing a Reading Comprehension Test for Cognitive

Diagnostic Assessment

Fatemeh Ranjbaran1, Seyyed Mohammed Alavi2

1
PhD in Applied Linguistics, University of Tehran
2
Professor of Applied Linguistics, Faculty of Foreign Languages and Literatures, University of Tehran

Abstract

A critical issue in cognitive diagnostic assessment (CDA) lies in the dearth of

research in developing tests for cognitive diagnostic purposes. CDA of
formative language assessment that aims to inform instruction has not been
conducted in foreign language learning contexts. This study explored how
developing a reading comprehension test based on a cognitive framework
could be used for diagnostic purposes. To validate the developed test items,
a Q-matrix that specified the relationships between test items and target
attributes was theoretically developed. Then, the test battery was
administered to seven language testing experts who identified the attributes
of each item coming to an overall agreement of nine major attributes. On the
basis of the overall agreement of the experts’ judgments concerning the choices of attributes,
review of the literature and results of student think-aloud protocols, the tentative Q-Matrix was
refined and used for statistical analyses. Finally, the test was administered to 1986
students of a GE course at the University of Tehran. RUM, a cognitive
diagnostic model (CDM), was used for refining the Q-Matrix and diagnosing
the participants' strengths and weaknesses. Data analysis results confirmed that the
nine proposed reading attributes are involved in the reading comprehension test items. Such
diagnostic information could be helpful for teachers and practitioners to prepare instructional
materials that target specific weaknesses and inform them of the more problematic areas that need
to be emphasized in class in order to plan for better L2 reading instruction. Further, such
information could inform individualized student instruction and produce improved diagnostic
tests for future use.

Keywords: Attributes; Cognitive Diagnostic Assessment; Cognitive Diagnostic Model; Q-Matrix;

Reparameterized Unified Model (RUM)