Article

pubs.acs.org/jchemeduc

Using Ordered Multiple-Choice Items To Assess Students’

Understanding of the Structure and Composition of Matter
Jan C. Hadenfeldt,*,† Sascha Bernholt,† Xiufeng Liu,‡ Knut Neumann,† and Ilka Parchmann†
†
Leibniz Institute for Science and Mathematics Education (IPN), Kiel, Germany
‡
Department of Learning and Instruction, State University of New York at Buffalo, Buffalo, New York 14260-1000, United States
*
S Supporting Information

ABSTRACT: Helping students develop a sound understanding of scientific

concepts can be a major challenge. Lately, learning progressions have received
increasing attention as a means to support students in developing understanding of
core scientific concepts. At the center of a learning progression is a sequence of
developmental levels reflecting an idealized progression toward understanding a
particular core concept. This sequence is supposed to serve as a basis for designing
instruction that can foster learning as well as assessments that can monitor students’
progression. So-called ordered multiple-choice (OMC) items have recently been
suggested as a simple and effective way of assessing students’ level of understanding
of a core concept. This article details our efforts in developing an instrument for
assessing students’ understanding of the structure and composition of matter based
on OMC items. Ten OMC items were developed and administered to a sample of
N = 294 students in grades 6−12. Rasch analysis was used to investigate instrument
functioning and to determine linear measures of person abilities and item difficulties.
In addition to the OMC items, students were administered corresponding open-ended items in order to investigate the validity of
the results obtained through the OMC items. Our findings suggest assessing students’ understanding of scientific concepts
through OMC items is indeed quite worthwhile and should be subject to further research.
KEYWORDS: Elementary/Middle School Science, High School/Introductory Chemistry, Chemical Education Research,
Testing/Assessment
FEATURE: Chemical Education Research

■ INTRODUCTION
One important goal in science education is fostering students’
understanding of core scientific concepts.1,2 However, science
education research has shown that students are struggling with
developing a deeper understanding of these concepts.3
Recently, learning progressions have been suggested as means
to foster students’ progression in understanding core concepts.4
Learning progressions build on the idea that instruction and
assessment need to be aligned. At the core of a learning
progression is a sequence of levels reflecting an increasingly
sophisticated understanding of the concept. As students’
pathways to scientific understanding are oftentimes not linear
and not the same for every student, this sequence is thought to
describe an idealized pathway.5 And while students are not
expected to all have the same level of understanding or progress
in the same way, each level is considered to mark a major step
toward a deeper scientific understanding. Instructional
components specifically designed for each level of under-
standing are expected to help students progress to the next
level. However, in order to choose the proper instructional
component for their students (or individual groups of
students), teachers need to know about students’ current
level of understanding.6 Hence, assessment instruments are
© 2013 American Chemical Society and
Division of Chemical Education, Inc.
needed that can elicit individual students’ level of under-
standing.
While interviews or open-ended items are considered to
provide a rich view of one student’s understanding, such
methods do not efficiently provide an overview of students’
understanding in class. Although multiple-choice items are
much more efficient for larger groups, they do not elicit
students’ understanding as well as interviews can. Recently,
Briggs, Alonzo, Schwab, and Wilson7 suggested that ordered
multiple-choice (OMC) items might offer deeper insights into
students’ thinking while still being efficient to analyze. In an
OMC item, each response option reflects a particular level of
understanding. In contrast to concept inventories, OMC items
incorporate a developmental perspective. An instrument based
on OMC items will not only provide information about which
alternative conceptions of a scientific concept students hold but
also allow for determining where students are on their way
toward a deeper understanding of the respective concept.
Students who consistently choose response options related to
one level across a set of OMC items can be expected to have
obtained that level of understanding. Thus, OMC items provide
Published: November 5, 2013
1602 dx.doi.org/10.1021/ed3006192 | J. Chem. Educ. 2013, 90, 1602−1608
Journal of Chemical Education
a more detailed picture of students’ understanding compared to
multiple-choice items. Previous research suggests that OMC
items have the potential to assess students’ level of under-
standing almost as well as interviews.8
The purpose of this paper is to detail our efforts in
developing an instrument for assessing students’ understanding
of matter based on OMC items. The specific questions to be
answered by this paper are as follows:
1. How can a successful process of designing an instrument
using OMC items to assess students’ understanding of
matter be characterized?
2. To what extent does an instrument based on OMC items
provide reliable and valid information about the level of
students’ understanding of matter?
■
METHOD
One approach to design assessment instruments has been
suggested by Wilson.9 This approach describes four building
blocks in the process of designing assessment instrument: the
construct map, the items’ design, the outcome space, and the
measurement model (see Figure 1). In the past, Wilson’s
Figure 1. The four building blocks.9
approach has been successfully used to develop assessment
instruments for a variety of different constructs.7,8,10 We will
describe each of the building blocks in greater detail and discuss
how to use them to design an instrument assessing students’
understanding of matter based on OMC items.
Construct Map
The first building block entails the construction of a construct
map. A construct map provides a substantive definition of the
construct and an ordering of qualitatively different levels of
increasing sophistication of the construct.10 That is, construct
map defines what students should understand and how
students progress in developing this understanding.11 In the
case of our instrument, what students should understand is the
concept of matter.
Students’ understanding of matter is well researched.12
Although different conceptions of what students should
understand about matter can be found in the literature,13−15
researchers agree that an understanding of the structure and
composition of matter is central to understanding the concept
of matter as a whole.16 And while there are only few
longitudinal studies on students’ progression,17,18 the extensive
research based on students’ conceptions of matter at different
1603
Article
age levels provides a good picture of how students progress in
their understanding of the structure and composition of
matter.17−21 We synthesized five hierarchically ordered levels
of students’ understanding of the structure and composition of
matter from a review of this research. These levels describe a
progression from an understanding of the macroscopic
properties of matter toward an understanding of the particulate
structure of matter and its relation to macroscopic properties.
The lowest level, level 1, relates to an everyday conception of
matter. At this level students do not use the idea of a particulate
nature of matter when trying to explain phenomena relating to
science. At the next level, level 2, students have learned about
the particulate nature of matter. However, instead of using a
particle model, students’ will explain phenomena using a hybrid
model, in which particles are thought to be embedded in a
substance.17,22 At level 3, students are able to make use of a
simple particle model to explain phenomena. Not having
learned about the substructure of these particles, they believe
that matter is built by particles as the smallest unit.18,23 At level
4, students can make use of a differentiated particle model, in
which each particle (i.e., atom) is made up of smaller particles
(i.e., protons, electrons, and neutrons).14,24 And at level 5,
students finally can explain macroscopic properties of matter as
a result of the properties of the particles the matter is made
from and the complex interactions between these particles.25,26
For a more detailed description of the individual levels, see the
Supporting Information. It is important to note that while
individual students may have a different level of understanding
and how they progress through this sequence of levels may
vary, the sequence represents a reasonable basis for assessing
students’ level of understanding of the structure and
composition of matter.
Items’ Design
The second building block in the process of developing is the
items’ design. This building block relates to specifying the type
of item used to elicit evidence about students’ level of
understanding of the construct with respect to the construct
map.9 Recently, Briggs et al.7 suggested a novel type of items,
so-called ordered multiple-choice items that aim to facilitate
diagnostic assessment linking the response options in multiple-
choice items to different levels of understanding of a construct.
In an OMC item, each response option is assigned one level of
understanding (Figure 2). Similar to regular multiple-choice
Figure 2. Sample ordered multiple-choice item. In order to recognize
option A as the correct answer, an understanding at level 4 is required.
dx.doi.org/10.1021/ed3006192 | J. Chem. Educ. 2013, 90, 1602−1608
Journal of Chemical Education
items, one response option is correct. This option reflects the
highest level of understanding that can be tested with this item.
All other response options are incorrect. These options reflect
(scientifically) incorrect ideas of students at lower levels of
understanding.
To develop OMC items assessing students’ understanding of
the structure and composition of matter, in our study we used
information from both the curriculum and research on
students’ alternative conceptions. Based on the curriculum
and respective textbooks, the item context and question were
formulated.1,27 Once the question and correct response option
were formulated, the response option was assigned a level of
understanding. Subsequently, the response options for lower
levels of understanding were formulated based on typical
alternative conceptions students on the respective levels of
understanding would typically have.28−30 On the one hand, the
aim was to develop response options meaningful to students.
On the other hand, it should not be obvious to students which
answer is correct. A sample item is shown in Figure 3. As can be
Figure 3. An OMC item for the category of structure and
composition.31 Additional OMC items, as well as a more detailed
description of the levels of understanding, can be found in the
Supporting Information.
seen from this item, it is not that all levels of understanding
have to be covered by response options.7 As a matter of fact,
the item shown in Figure 3 can only be used to differentiate
between the first three levels of understanding. The OMC
items were authored and compiled into an instrument by a
team of science educators and science teachers.
Outcome Space
The third building block, outcome space, is about the
relationship between the construct map and the items.9 In
the case of our OMC items, this means how the response
options relate to the levels of understanding of the structure
and composition of matter. In order to investigate the validity
of the assignment of levels to response options, the assignment
was carried out independently by three researchers. A Fleiss κ
measure was used to obtain information about the agreement of
the three researchers. A value of κ = 0.88 suggests the
researchers agreed quite well in their assignment of levels of
understanding to the response options.
Measurement Model
In the fourth and last building block, the measurement model
(i.e., the relation between the scored outcomes and students’
level of understanding as described in the construct map) is
defined.9 For the process of developing assessment instruments,
1604
Article
item response theory (IRT) has proven to be fruitful. The IRT
approach we used to analyze our data set is commonly known
as Rasch analysis.32 For a detailed description of how Rasch
analysis can guide the process of developing measurement
instruments, see for example Liu and Boone.33 The first step in
Rasch analysis is to define a scoring scheme. As all response
options of an OMC item are linked to a specific level of
understanding, each response option was assigned the value
corresponding to the respective level of understanding, based
on the framework described (from 1 to 5). Finally, a
polytomous Rasch model, more specifically the partial credit
Rasch model (PCM), was applied to the data. Analyzing the
data set using the PCM can provide valuable information about
item functioning with respect to the construct map. To detail
this point, the design and findings of a study in which an
instrument based on OMC items has been used will be
reported in the next two sections.
■ DESIGN OF THE STUDY
In order to investigate the functioning of the instrument
assessing students’ understanding of the structure and
composition of matter based on OMC items, we administered
it to a sample of 294 students in grades 6−12 in a German
grammar school. The instrument included a set of 10 OMC
items. Because the school’s science curriculum would not allow
for students to develop an understanding on level 5 yet, only
items assessing an understanding up to level 4 were included in
the instrument. In order to investigate to what extent the OMC
items can be used to assess students’ understanding, open-
ended items corresponding to the OMC items were included in
the instrument. These items were basically open-ended versions
of the OMC items (same question, but no multiple-choice
response format).
The testing was done on a single day. It was up to the
teachers to decide whether a given class would participate in the
study. For the classes that did participate, the instrument was
administered to all students in those classes. (See Table 1 for
Table 1. Distribution of Students across Grades
Grade Levels
Students 6 7 8 9 10 11 12
Number 41 73 40 47 34 29 30
information about the distribution of participating students.)
Due to limited testing time, each student was presented with a
different version of the instrument. Each version included the
10 OMC items and three open-ended items. The different
versions were designed so that each of the 10 open-ended items
was answered by the same number of students. To avoid an
influence of OMC items on the open-ended items, the open-
ended items were given first. Students were allowed to proceed
to the OMC items only after completing the open-ended items
and were not allowed to go back to the open-ended items
afterward. To obtain additional information for item refine-
ment, each student received a questionnaire designed
specifically to collect such information for a subset of three
OMC items. This questionnaire included questions regarding,
for example, words students would not understand, or their
reasoning for choosing a particular option.34 Again, we made
sure we would receive feedback for each item from about the
same number of students.
dx.doi.org/10.1021/ed3006192 | J. Chem. Educ. 2013, 90, 1602−1608
Journal of Chemical Education
■
FINDINGS
In the first step of our analysis, we compared students’ answers
to the open-ended version of the OMC items and the response
option chosen in the corresponding OMC item. Students’
answers to the open-ended items were assigned one level of
understanding. This assignment was independently carried out
by two different researchers. Cohen’s κ was used to determine
inter-rater reliability. A value of κ = 0.91 indicates that the two
researchers agreed in nearly every case. Examples of students’
responses and the respective level of understanding assigned to
these responses are provided in Box 1.
Box 1. Students’ Responses to the Open-Ended Items by
Level of Understanding
In 79% of the cases, the level assigned to a student’s response
to an open-ended item matched the level of understanding
students demonstrated in the corresponding OMC item. (This
is the average of all values reported in the second column of
Table 2). In order to analyze to which extent the level of
understanding identified through OMC and open ended items
is the same, we calculated the following measure of agreement:
The level assigned to a student’s open answer was subtracted
from the level of understanding identified through the
corresponding OMC item (leading to the number of paired
answers listed in column 3 of Table 2). For each item we then
calculated the mean and standard deviation of this discrepancy.
The overall mean across all items was M = 0.1 with a standard
deviation of SD = 0.67. A Wilcoxon signed ranks test revealed
that in four cases the discrepancy is statistically significantly
different from zero (see Table 2). These results indicate that
the majority of the OMC items conform to their open-ended
counterparts regarding the assessment of students’ level of
understanding of the structure and composition of matter.
Table 2. Comparison of OMC Items and Open-Ended Items
Conformity of Students’ Level of Understanding Assessed through Mean Discrepancy
Item OMC and Open-Ended Items, % of Levels
1 89 −0.12
2 79
3 77
4 90
5 84
6 81 −0.10
7 92
8 60
9 67
10 68
a
N = 294.
1605
Article
In the second step of our analysis, we investigated whether
the OMC items test a unidimensional trait (i.e., students’
understanding of the structure and composition of matter).
Rasch analysis, specifically the partial credit Rasch model, was
used to investigate to what extent the individual items fit on a
one-dimensional latent scale on which students in grades 6−12
might progress. For analyzing item quality, the mean square
residual (MNSQ) and the standardized mean square residual
(ZSTD) are two important indicators. They resemble the
difference between what is observed and what is expected by
the Rasch model. Items with a good model fit should have
MNSQ values within the range of 0.8 to 1.2 and ZSTD within
the range of −2.0 to 2.0.35 Analysis of item quality reveals that
the items meet the standards for item quality criteria except for
one, item 8. (See the Supporting Information for detailed item
characteristics.) This item addressed students’ understanding of
particle dynamics, whereas the other items focused mainly on
structure and structure−property ideas. Another indicator for
instrument functioning is the WLE reliability, which can be
interpreted similarly to Cronbach’s κ, in which values above 0.7
are supportive of reliability inferences.36 The WLE reliability
was found to be 0.71. This can be considered as adequate given
the complex nature of the trait and the small number of items.36
In summary, the fit statistics support the assumption that the
partial credit Rasch model fits to the data and that the obtained
data from the students can be analyzed with the Rasch model,
indicating the local stochastic independence of the items and
their alignment to a single unidimensional latent trait.
One of the benefits of using Rasch analysis is that a student’s
ability can be analyzed in relation to item difficulty, as the
values for both item difficulty and student ability are
transformed to the same scale.35 Regarding the current item
design, a student gets a specific score in accordance to his or
her selected answer: The higher the conceptual understanding
represented in the chosen answer in the framework, the higher
the score the student gets. The PCM analysis provides
threshold values (so-called Thurstonian thresholds) that reflect
students’ progression from one level of understanding to the
next. If a student’s ability is above the threshold for a score
category, the probability of achieving that score or higher is
above 50% for that particular student.37 With regard to the
Thurstonian thresholds for the items 1 to 10 (Figure 4), the
items cover a wide range of ability levels. In addition, the
thresholds for all items are in the correct order, that is, it is
more difficult to reach a higher level of understanding than a
lower level. The results displayed in Figure 4 thus indicate that
Number of Standard Deviation of the Wilcoxon p
Paired Answers Discrepancy of Levels Valuesa
59 0.50 0.09
0.21 50 0.55 0.02
0.28 41 0.56 0.01
0.03 59 0.32 0.48
0.12 33 0.49 0.20
33 0.54 0.37
0.03 41 0.28 0.77
0.06 51 1.10 0.57
0.36 36 0.86 0.02
0.56 59 0.96 0.00
dx.doi.org/10.1021/ed3006192 | J. Chem. Educ. 2013, 90, 1602−1608
Journal of Chemical Education
Figure 4. Thresholds for items 1 to 10.
the assumption of ordered multiple-choice options in fact holds
true for the developed set of items. In line with the framework,
a higher level of understanding as represented in the OMC
options was difficult to reach for the students.
An analysis of variance indicates that about 68% of the
variance in the thresholds is explained by the assigned level of
understanding: τ = 0.69, p < 0.001; F(2,22) = 23.23; p < 0.001;
R2 = 0.679. Figure 5 (left) illustrates the distribution of the
Figure 5. Threshold over level of understanding required to solve the
item (left), person ability over grade (right).
threshold parameters for the three levels of understanding.
Accordingly, higher levels of understanding are on average
more difficult than lower levels of understanding. In addition,
the overlap between two adjacent levels is quite small,
supporting the idea of hierarchically ordered levels of
understanding. (However, no perfectly disjunctive hierarchy is
expected.) This suggests that the OMC items can indeed assess
different levels of understanding of the structure and
composition of matter.
In the third and last step of our analysis, we investigated the
extent to which the OMC items were suitable to measure
students’ progression in understanding the structure and
composition of matter. For this purpose, we investigated to
which extent students from higher grades (i.e., students who
had received more instruction on the concept of matter) were
showing higher person ability measures. As shown in Figure 5
(right), person ability measures were indeed found to increase
with increasing grade. Despite the small sample size per grade
1606
Article
level, the average person ability increases constantly. The
boxplot reveals a substantial overlap of person ability between
two adjacent grades; nevertheless, 54% of the ability measure
can be explained by the person’s grade: τ = 0.55, p < 0.001;
F(6,287) = 28.92; p < 0.001; R2 = 0.536. Although the data
were obtained from a cross-sectional design and though no
inferences about individual learning paths can be made, this
result again supports the structural validity of the data.38
■ CONCLUSIONS AND OUTLOOK
In this paper, we described a procedure for successfully
developing an instrument based on ordered multiple-choice
items to assess students’ understanding of the structure and
composition of matter (Question 1). Moreover, we investigated
the extent to which the instrument based on OMC items is
suitable to provide science educators with reliable and valid
information about how students progress in their under-
standing of matter (Question 2).
Our findings suggest that an instrument based on OMC
items can assess students’ understanding of the structure and
composition of matter as well as open-ended items. We found
OMC items to be able to detect different levels of
understanding and to detect students’ progress in under-
standing the structure and composition of matter. Overall,
these findings provide initial evidence that OMC items can be
used to design instruments for a valid assessment of students’
understanding of core concepts in science. Although our
findings create a sound base for further research on how to
assess students’ understanding of the structure and composition
of matter using OMC items, some issues remain in terms of
instrument development. The procedure described in the
Method section resulted in an initial version of an instrument
that already could produce meaningful results. However,
instrument development is an iterative process.9 Based on the
findings, we identified the following points that need to be
addressed in refinement of the instrument.
First, as this instrument was administered to students in
grades 6−12, students’ response behavior needs to be more
closely analyzed in terms of the items’ wording and clarity.
Findings from a modified AAAS questionnaire indicate that
although some students in grades 6 and 7 were not familiar
with all scientific terms provided in the OMC items, about 66%
were sure that only their chosen answer was correct. This
percentage increases with grade. This result indicates that
OMC items need to be carefully developed with respect to the
target population. Without the usage of scientific terms, it is
difficult to assess students’ understanding at higher levels;
nonetheless, including scientific terms may confuse younger
students. To address this problem, research needs to be carried
out, especially with younger students (e.g., using think-aloud
interviews), to identify commonly understood as well as
problematic terms in the items.
Second, as Table 2 shows, some open-ended counterparts of
the administered OMC items reflect students’ response
behavior better than others. This result indicates that not all
of students’ possible answers were covered with the response
options provided in the current version of the OMC items.
Some open-ended answers could not be assigned a level of
understanding. However, these answers can guide the develop-
ment of additional response options for a revised version of the
items. In return, analyzing the distribution of chosen response
options per item might reveal those options that appear
unattractive or implausible and thus rarely chosen by students.
dx.doi.org/10.1021/ed3006192 | J. Chem. Educ. 2013, 90, 1602−1608
Journal of Chemical Education
Third, although the levels linked to the response options of
the OMC have a significant influence on item difficulty (see
Figure 5), item difficulty still overlaps for some items from
different levels. That is, levels of understanding cannot
necessarily be distinguished from each other. Therefore,
response options should be refined in order to better represent
the different levels of understanding. For example, in item 8
(Figure 4), the difference in difficulty between level 2 and level
3 needs to be adjusted. Response options that were not
attractive to students could be revised. Conversely, response
options that reflected quite common misconceptions could be
revised in order to make the right answer more attractive.
Although the items cover the latent trait well, more items need
to be developed. Particularly, more items are needed with
thresholds between 0.7 to 1.7 logits. As it is well-known that
students do not respond consistently to similar problems set in
different contexts,6 more items need to be developed to get
more information about the role of context with respect to the
structure and composition of matter. The clear progression (see
Figure 5) might be due to the fact that most of the OMC items
focused on structure−property ideas. It would be of interest to
determine to what extent the progression in students’
understanding was due to items that focused on single ideas
(i.e., dynamics or structure itself). As the only misfitting item
was the one that addressed dynamics of particles, it might be
assumed that students might progress at different rates along
each idea. This would be another issue to be tackled in further
studies.
The possibility of establishing a link between a proposed
framework (i.e., a construct map) and students’ performance on
the instrument (i.e., students’ scores) is one of the main
advantages of using Rasch analysis. Rasch analysis allowed us to
link students’ performance to the difficulty of (different levels of
understanding in) OMC items. In the case of our instrument, it
seems that students performing well on the instrument have
indeed reached a high level of understanding with respect to
the underlying framework. Students with a lower achievement
were found to prefer response options assigned to lower levels
of understanding. Thus, the instrument enables differentiation
between high- and low-performing students, as well as allowing
interpretation of this performance in terms of qualitatively
different levels of understanding of the composition and
structure of matter. The possibility of this kind of interpretation
makes the instrument interesting for further research and for
classroom practice. Teachers can use the instrument in two
important ways: (i) as a formative test to see which level of
understanding students enter class with and to monitor student
learning throughout instruction; and (ii) as a tool for evaluation
to assess student learning and give feedback to students about
their achievement. Analysis can focus on the distribution of
students’ responses to single items as well as students’
performance on the entire test. While Rasch analysis needs
specialized software (and an analysis with data from a single
classroom is not necessarily meaningful), once instrument
functioning has been fully established, teachers can simply
evaluate their students’ thinking by assigning them the level of
understanding reflected by the chosen response option. Based
on the analysis of both students’ performance on single items
and the whole instrument, teachers may then design instruc-
tional components to specifically address their students’ needs.
In summary, instruments based on OMC items seem to offer
teachers a simple and efficient way to obtain a detailed picture
about students’ understanding of core concepts.
1607
■
Article
ASSOCIATED CONTENT
*
S Supporting Information
Detailed description of the individual levels and of the OMC
items. This material is available via the Internet at http://pubs.
acs.org.
■
AUTHOR INFORMATION
Corresponding Author
*E-mail: hadenfeldt@ipn.uni-kiel.de.
Notes
The authors declare no competing financial interest.
■
ACKNOWLEDGMENTS
We would like to thank the journal reviewers and editors for
their detailed and helpful recommendations. The research
reported here was supported by the German Federal Ministry
of Education and Research.
■ REFERENCES
(1) National Research Council. National Science Education Standards;
National Academy Press: Washington, DC, 1996.
(2) Waddington, D.; Nentwig, P.; Schanze, S. Making It Comparable:
Standards in Science Education; Waxmann: Münster, Germany, 2007.
(3) Vosniadou, S. International Handbook of Research on Conceptual
Change; Lawrence Erlbaum: Mahwah, NJ, 2008.
(4) Duschl, R. A.; Schweingruber, H. A.; Shouse, A. W. Taking science
to school: Learning and teaching science in grades K-8; National Academy
Press: Washington, DC, 2007.
(5) Neumann, K.; Viering, T.; Boone, W. J.; Fischer, H. E. J. Res. Sci.
Teach. 2013, 50, 162−188.
(6) Pellegrino, J. W.; Chudowsky, N.; Glaser, R., Eds. Knowing What
Students Know: The Science and Design of Educational Assessment;
National Academies Press: Washington, DC, 2001.
(7) Briggs, D. C.; Alonzo, A. C.; Schwab, C.; Wilson, M. Educ.
Assessment 2006, 11 (1), 33−63.
(8) Alonzo, A. C.; Steedle, J. T. Sci. Educ. 2009, 93 (3), 389−421.
(9) Wilson, M. Constructing Measures: An Item Response Modeling
Approach; Lawrence Erlbaum: Mahwah, NJ, 2005.
(10) Wilson, M. Res. Sci. Teach. 2009, 46 (6), 716−730.
(11) Wilson, M. Responding to a Challenge That Learning
Progressions Pose To Measurement Practice. In Learning Progressions
in Science, Alonzo, A., Gotwals, A., Eds.; Sense Publishers: Rotterdam,
2012; pp 317−344.
(12) Duit, R. Bibliography for Students’ and Teachers’ Conceptions and
Science Education. http://www.ipn.uni-kiel.de/aktuell/stcse/stcse.html
(accessed Oct 2013).
(13) Smith, C.; Wiser, M.; Anderson, C. W.; Krajcik, J.; Coppola, B.
Measurement 2006, 14 (1, 2), 1−98.
(14) Stevens, S.; Delgado, C.; Krajcik, J. S. J. Res. Sci. Teach. 2010, 47
(6), 687−715.
(15) Liu, X.; Lesniak, K. J. Res. Sci. Teach. 2005, 43 (3), 320−347.
(16) Othman, J.; Treagust, D. F.; Chandrasegaran, A. L. Int. J. Sci.
Educ. 2008, 30 (11), 1531−1550.
(17) Johnson, P. Int. J. Sci. Educ. 2002, 24 (10), 1037−1054.
(18) Löfgren, L.; Helldén, G. Int. J. Sci. Educ. 2009, 31 (12), 1631−
1655.
(19) Liu, X. Int. J. Sci. Educ. 2001, 23 (1), 55−81.
(20) Krnel, D.; Watson, R.; Glazar, S. A. Int. J. Sci. Educ. 1998, 20 (3),
257−289.
(21) Anderson, B. Stud. Sci. Educ. 1990, 18, 53−85.
(22) Renström, L.; Andersson, B.; Marton, F. J. Educ. Psych. 1990, 82
(3), 555−569.
(23) Pfundt, H. Chim. Didakt. 1981, 7, 75−94.
(24) Harrison, A. G.; Treagust, D. F. Sci. Educ. 1996, 80 (5), 509−
534.
dx.doi.org/10.1021/ed3006192 | J. Chem. Educ. 2013, 90, 1602−1608
Journal of Chemical Education Article

(25) Taber, K. S. Sci. Educ. 2005, 89 (1), 94−116.

(26) Ayas, A.; Ö zmen, H.; Ç alik, M. Int. J. Sci. Math. Educ. 2010, 8
(1), 165−184.
(27) Baumert, J.; Bos, W.; Watermann, R. TIMSS/III. Schülerleistun-
gen in Mathematik und den Naturwissenschaften am Ende der
Sekundarstufe II im internationalen Vergleich. Zusammen fassung
deskriptiver Ergebnisse; Max-Planck-Institut für Bildungsforschung:
Berlin, 1998.
(28) Kind, V. Beyond Appearances: Students’ Misconceptions about
Basic Chemical Ideas, 2nd ed.; Durham University: Durham, U.K.,
2004. http://www.rsc.org/images/Misconceptions_update_tcm18-
188603.pdf (accessed Oct 2013).
(29) Taber, K. Chemical MisconceptionsPrevention, Diagnoses and
Cure; Royal Society of Chemistry: London, 2002.
(30) Talanquer, V. Int. J. Sci. Educ. 2009, 31 (15), 2123−2136.
(31) Mullis, I. V. S.; Martin, M. O.; Foy, P. IEA’s TIMSS 2003
International Report on Achievement in the Mathematics Cognitive
Domains: Findings from a Developmental Project; TIMSS & PIRLS
International Study Center, Boston College: Chestnut Hill, MA, 2005.
(32) Rasch, G. Probabilistic Models for Some Intelligence and
Attainment Tests; University of Chicago Press: Chicago, 1980.
(Originally published as Rasch, G. Probabilistic Models for Some
Intelligence and Attainment Tests, Studies in Mathematical Psychology I;
Danmarks Pædagogiske Institut: Copenhagen, 1960.)
(33) Liu, X., Boone, W., Eds. Application of Rasch Measurement in
Science Education; JAM Press: Maple Grove, MN, 2006.
(34) American Association for the Advancement of Science. Getting
Assessment Right: Rigorous Criteria, Student Data Inform Project
2061’s Assessment Design. 2061 Today 2007, 17 (1), 1−5.
(35) Bond, T. G.; Fox, C. M. Applying the Rasch Model: Fundamental
Measurement in the Human Sciences; Lawrence Erlbaum: Mahwah, NJ,
2001.
(36) Field, A. Discovering Statistics Using SPSS; SAGE: London, 2009.
(37) Wu, M.; Adams, R. Applying the Rasch Model to Psycho-Social
Measurement: A Practical Approach; Educational Measurement
Solutions: Melbourne, 2007.
(38) Messick, S. Am. Psychol. 1995, 50 (9), 741−749.

1608 dx.doi.org/10.1021/ed3006192 | J. Chem. Educ. 2013, 90, 1602−1608