You are on page 1of 9

View Article Online / Journal Homepage / Table of Contents for this issue

Chemistry Education Dynamic Article Links

Research and Practice

Cite this: Chem. Educ. Res. Pract., 2012, 13, 3038 PAPER
The application and evaluation of a two-concept diagnostic instrument
with students entering college general chemistryw
Keily Heredia,z Xiaoying Xuz and Jennifer E. Lewis*
Received 30th October 2010, Accepted 23rd October 2011
DOI: 10.1039/c0rp90017f
Published on 07 December 2011. Downloaded on 20/10/2014 10:47:33.

The Particulate Nature of Matter and Chemical Bonding Diagnostic Instrument (Othman J.,
Treagust D. F. and Chandrasegaran A. L., (2008), Int. J. Sci. Educ., 30(11), 15311550) is used to
investigate college students understanding of two chemistry concepts: particulate nature of matter
and chemical bonding. The instrument, originally developed for secondary school students, is a
two-tier diagnostic test. In this study, the instrument was given as a paper and pencil test to
college students enrolled in General Chemistry I during the second week of class in Spring 2010.
General Chemistry I students were divided into three groups for analysis: (A) with preparatory
chemistry, (B) repeating general chemistry (without preparatory chemistry); and (C) rst time in
general chemistry (without preparatory chemistry); Groups B and C are expected to have taken at
least one year of secondary school chemistry. However, Group A had little or no exposure to
high school chemistry. Regardless of their pathway into the course, students in General Chemistry
I are expected to have a fair understanding of the two concepts. Furthermore, students responses
were used to think about the role of a preparatory chemistry course in promoting deep
understanding of chemistry concepts. An ANCOVA analysis was performed. Preparatory
Chemistry was found to have a statistically signicant but small eect on students scores after
controlling for prior math achievement. Conrmatory factor analysis and reliability studies were
performed and are presented here. Analysis also revealed alternative conceptions.

Introduction Costu, 2008), have been widely used as an eective tool to

investigate students thinking and conceptual understanding.
College-level chemistry courses are required for science majors. While these approaches provide rich and detailed information,
Students who are unable to successfully pass introductory they are time consuming, and not easy to use in classroom
college chemistry are prevented from continuing in science- assessment. Multiple-choice test format is convenient, and
oriented programs. Therefore, student performance in intro- therefore, typically used in standardized and in classroom
ductory college-level chemistry courses remains a recognized tests (Examinations Institute, 2011). For example, multiple-
area of concern. Extensive research has been done to predict choice summative tests cover content as broadly as possible
student performance in college chemistry courses (Russell, to reect the student cumulative knowledge, and therefore,
1994; Wagner et al., 2002; Tai et al., 2005; Potgieter et al., provide information about the student content knowledge at
2010). Previous studies have used the California Chemistry the end of a chapter or a semester in a particular subject area.
Diagnostic Exam (McFate and Olmsted III, 1999), the SAT However, the interpretation of each item in summative tests
(Spencer, 1996), as well as logical reasoning instruments cannot specically provide information about the alternative
(Bunce and Hutchinson, 1993; Lewis and Lewis, 2007; Jiang conceptions students have about chemistry concepts.
et al., 2010) to predict student achievement in introductory Chemistry-based diagnostic tests have been developed and
college chemistry. used to measure student alternative conceptions (Treagust, 1986;
Other approaches have been used to assess student learning. Treagust, 1988; Voska and Heikkinen, 2000; Tsai and Chou,
Qualitative approaches such as open-ended responses (Nyachwaya 2002; Treagust et al., 2011). These assessments can diagnose
et al., 2011), and clinical interviews (Ebenezer and Gaskell, 1995; students understanding of concepts that are included in the
introductory college chemistry curriculum. Two important
Department of Chemistry, University of South Florida, concepts, the particulate nature of matter and chemical bonding,
4202 E. Fowler Avenue CHE205A, Tampa, Florida 33620, USA. are included in a diagnostic instrument developed by Othman
E-mail:; Fax: (813) 974-3203; Tel: (813) 974-1286 et al. (2008) for secondary school students. Other instruments are
w Electronic supplementary information (ESI) available. See DOI:
10.1039/c0rp90017f available to measure students alternative conceptions; however,
z The two authors contributed equally to this work. they are often more general (Mulford and Robinson, 2002)

30 Chem. Educ. Res. Pract., 2012, 13, 3038 This journal is c The Royal Society of Chemistry 2012
View Article Online

or more appropriate for later in the curriculum (Villafane and for those who did not take secondary school chemistry.
et al., 2011). Because of its tight focus on two concepts and its The course is intended to prepare students to take General
suitability for students who have little experience with college Chemistry I, and has a history of promoting success in that
chemistry, the two-tier Particulate Nature of Matter and course (Garcia, 2010). According to the syllabus, the emphasis
Chemical Bonding Diagnostic Instrument was chosen for this of Preparatory Chemistry is on providing foundational under-
study (Diagnostic Instrument hereafter). The two concept standing of chemical principles and developing fundamental
instrument is also accessible, short, and easy to administer, processing skills such as critical thinking and learning strategies.
and its two-tier design allows students to provide both an Upon completion of the course, students are expected to
answer and a reason. understand and be able to apply the particulate nature of
matter and the rst law of thermodynamics. More detailed
Study purpose course information is presented in the ESI.w
The purpose of this study is to probe student understanding of
two important topics from secondary school chemistry that will be General chemistry I
covered in greater depth in introductory college-level chemistry, General Chemistry I is the rst of two semester-long introductory-
Published on 07 December 2011. Downloaded on 20/10/2014 10:47:33.

and to compare the understanding of students who enter the level college chemistry courses, required for all science majors.
course via dierent pathways. Within the context of the study, Students who take General Chemistry I without taking Pre-
students who enroll directly in General Chemistry I are expected paratory Chemistry are expected to have at least one year of
to have taken a year of chemistry in secondary school. However, secondary school chemistry and evidence of prior mathematics
some students in the U.S. are able to graduate from secondary achievement, such as an SAT Math score of at least 550 or a
school without taking a course focused on chemistry. For passing grade in a college-level algebra course. Principles and
example, to achieve the three required high school science credits applications of chemistry, including properties of substances
(full-year courses) in the school district in which the university is and reactions, thermo-chemistry, atomic-molecular structure and
located, students may take an introductory integrated science bonding, and periodic properties of elements and compounds
course in which some chemistry topics may play a role, followed are discussed in the course. The structure of the course has
by a biological science course, capped with an elective such as been discussed previously (Lewis and Lewis, 2005, 2008).
ecology. Therefore, upon their arrival at the university, students
without (or with limited exposure to) secondary school chemistry Diagnostic instrument
are recommended to take Preparatory Chemistry prior to
General Chemistry I. Regardless of the pathway into General The Particulate Nature of Matter and Chemical Bonding
Chemistry I, students need to have developed a conceptual Diagnostic Instrument (Othman et al., 2008), was originally
understanding of the particulate nature of matter and chemical developed for secondary school students. It is a two-tier
bonding before enrolling in the course. We believe that the diagnostic used to investigate students conceptions of the
Particulate Nature of Matter and Chemical Bonding Diagnostic particulate nature of matter, and their understanding of
Instrument is suitable for determining the state of their under- chemical bonding. The instrument consists of ten items, ve
standing at the time of course entry. However, one important items for each concept. These items comprised topics such as
question is whether the instrument functions well at the college molecular and macroscopic properties, solutions, conservation of
level. Thus, this study will begin by investigating the factor matter, and phase changes for the concept of particulate nature
structure of the item scores to determine whether the instruments of matter, and electrical conductivity, structure of sodium
designed factors are relevant for the present use. The study will chloride, and intermolecular forces for the concept of chemical
then move to the interpretation of the students responses to bonding. Each item consists of two multiple-choice questions.
examine whether a preparatory chemistry course appears to have The rst tier of each item is a multiple-choice question that
any inuence on student understanding of the two topics. relates to a problem statement. The second tier of each item is
composed of a set of explanations for the answers from the
Research questions rst tier. The second tier (reasoning) can provide information
The following research questions will guide this study: (1) Do about which alternative conceptions students have about the
students entering General Chemistry I having taken Preparatory two chemistry concepts.
Chemistry perform better on the Diagnostic Instrument than The original study conducted by Othman et al. (2008)
students without Preparatory Chemistry? (2) Which alternative reported alternative conceptions about the particulate nature
conceptions, if any, do students entering General Chemistry of matter and chemical bonding held by secondary school
I have? (3) Are the alternative conceptions the same or dierent students in Singapore. More than half of the students in grades
for students entering General Chemistry I with Preparatory 9 (N = 140) and 10 (N = 120) believed that an atom of an
Chemistry as compared to those entering without it? element shares some physical properties as a sample of the
element. Similar alternative conceptions about the particulate
nature of matter have been found in dierent educational
Course context and chosen instrument contexts, and reported in many other studies. For example,
Taber (2001) and Johnson (2005) have explained that
Preparatory chemistry
many students believe the particles in a substance possess
Preparatory Chemistry is a one-semester course that is recom- the same macroscopic properties as the substance and sug-
mended for students with an SAT Math score lower than 550, gested that instructional practices may reinforce this belief.

This journal is c The Royal Society of Chemistry 2012 Chem. Educ. Res. Pract., 2012, 13, 3038 31
View Article Online

Recently, Salta and Tzougraki (2011) found that secondary About 3% of the students were majoring in chemistry. This
school students in Greece failed to successfully answer questions diverse sample is typical of the student population taking
regarding the particulate nature of matter due to their inability introductory college-level chemistry at this institution.
to transfer understanding from the atomic level to the macro- It is common for science students to register for General
scopic level. A common alternative conception about chemical Chemistry I during the fall semester, and register for General
bonding found by Othman et al. (2008) was that sodium Chemistry II during the following spring semester. However,
chloride exists as molecules. About 40% percent of the not every science major follows this path. First, there are often
students in both grades selected this alternative conception. more students wanting to take General Chemistry I in the fall
During one-on-one interviews with college students enrolled in than seats available, and second, as described above, selected
introductory chemistry in the U.S., Teichert et al. (2008) also students are encouraged to take a preparatory chemistry
found that students believe that ionic compounds exist as course in the fall semester. There are also students who are
molecules. Because similar alternative conceptions are so able to enroll in General Chemistry I in the fall, but do not
pervasive, appearing in students of dierent ages, in dierent succeed with a course passing grade of C or above, and
countries, and through dierent methods of investigation, therefore cannot take General Chemistry II without repeating
Published on 07 December 2011. Downloaded on 20/10/2014 10:47:33.

there is reason to check for these ideas among our students General Chemistry I. In order to meet student demand from
entering college level general chemistry. these diverse pathways, General Chemistry I is routinely
oered in the spring semester. Since the participants in this
Diagnostic instrument & preparatory chemistry study are enrolled in the General Chemistry I course during
the spring semester, it makes sense to examine the student
In order to be successful in General Chemistry I, students need population in more detail.
to have a conceptual understanding of the particulate nature of Three distinct groups emerge from this examination: Group
matter and chemical bonding. A closer look at the Preparatory A, the with prep-chem group, 364 students who have taken
Chemistry syllabus, textbook (Tro, 2009), and in-class activities Preparatory Chemistry; Group B, the repeaters, 125 students
revealed that students discussed matter, compounds, solutions, who have not taken Preparatory Chemistry but who have been
and physical changes as early as the second and third week of enrolled in General Chemistry I in a prior semester; and
class. The concepts are then built on this general overview, and Group C, the rst-timers, 236 students who have not taken
become more in-depth in later weeks. By the fourth and fth Preparatory Chemistry but also have never been enrolled in
week of classes, students are challenged with questions about General Chemistry I. Not only are these groups dierent in
ionic and molecular compounds. Various examples using macro terms of their pathway to General Chemistry I in Spring 2010,
and micro representations of NaCl and CO2 are used to explain they are also dierent with respect to demographic information.
the dierence between ionic and molecular compounds. In The prep-chem group is more heavily tipped toward female
week seven, students discuss the concept of solubility. A sodium (68%) than male (32%) students as compared to the other two
chloride solution, represented as NaCl(aq) is provided as an groups (w2 (2, N = 725) = 20.7, p o 0.01) and has a more
example to discuss electrical conductivity. By the thirteenth and diverse population as compared to the other two groups,
fourteenth week of class, students review in further detail the including a sizable number of underrepresented minority
concept of phase changes. A picture with a pot of water is students (w2 (12, N = 725) = 52.2, p o 0.01). In terms of
shown, and an explanation about how the bubbles are formed prior math achievement, the prep-chem group, as expected, has
is provided. Multiple representations of NaCl, CO2, and H2O a lower average Math SAT score than the other two groups
are used throughout each chapter in the book to encourage (F (2, N = 576) = 22.6, p o 0.01). Detailed demographic
student conceptual understanding. At the end of each chapter, information for each group is presented in the ESI.w
students are confronted with mathematical problems, which
assumes that students have learned the concepts. After con- Data analysis
sidering the topics discussed in Preparatory Chemistry, one When data from any instrument is collected, the rst step is to
would expect that this group of students would have a better examine descriptive statistics. SPSS 18.0 was used to obtain
conceptual understanding of the two concepts measured by descriptive statistics for the ten items and total score on the
the Diagnostic Instrument than would students entering from diagnostic instrument, and the results appear in Table 1. The
diverse secondary school chemistry backgrounds. next step is to look for reliability and validity evidence. For
reliability, Cronbachs alpha values were calculated in SAS
Methodology 9.1, and the results compared to the commonly used cuto of
0.70 (Thompson, 2003; Murphy and Davidshofer, 2005). For
factorial validity, a conrmatory factor analysis in Mplus 5.2
This study took place at a large public research university in the estimated how well the 2-concept model for the instrument t
southeastern United States. The instrument was administered the item data obtained with our sample (Crocker and Algina,
to students enrolled in General Chemistry I in Spring 2010, 2006). In addition to recommended cutos for model t
during the second week of class as a paper and pencil test. The statistics (Hu and Bentler, 1999; Brown, 2006), factor correlation
students were given 25 min to answer the 10-item Diagnostic and item loadings are used to evaluate the t.
Instrument. Students in this course were from more than 25 Data analysis then moves to interpretation. To determine
majors, the largest of which were (1) bio-medical sciences whether group membership was associated with a dierence in
26%, (2) biology 17%, and (3) pre-medical sciences 10%. performance on the Diagnostic Instrument, ANCOVA was

32 Chem. Educ. Res. Pract., 2012, 13, 3038 This journal is c The Royal Society of Chemistry 2012
View Article Online

Table 1 Descriptive statistics of item score and total score (N = 679)

With Prep-Chem (N = 348) Without Prep-Chem (N = 331)

Repeaters (N = 108) First-timers (N = 223)
Item A B C
M SD Skew Kurt M SD Skew Kurt M SD Skew Kurt
1 0.35 0.48 0.62 1.63 0.27 0.45 1.06 0.89 0.25 0.43 1.16 0.67
2 0.45 0.50 0.21 1.97 0.47 0.50 0.11 2.03 0.46 0.50 0.17 1.99
3 0.38 0.49 0.51 1.75 0.29 0.45 0.95 1.11 0.30 0.46 0.88 1.24
4 0.39 0.49 0.46 1.80 0.43 0.50 0.30 1.94 0.39 0.49 0.47 1.79
5 0.14 0.35 2.11 2.46 0.24 0.43 1.23 0.50 0.23 0.42 1.30 0.31
6 0.07 0.26 3.25 8.61 0.08 0.28 3.06 7.49 0.05 0.23 3.98 13.98
7 0.07 0.25 3.51 10.37 0.08 0.28 3.06 7.49 0.04 0.19 5.03 23.46
8 0.19 0.39 1.57 0.46 0.20 0.40 1.49 0.23 0.17 0.37 1.81 1.28
9 0.38 0.49 0.51 1.75 0.39 0.49 0.46 1.82 0.37 0.48 0.55 1.71
10 0.46 0.50 0.16 1.99 0.31 0.47 0.81 1.37 0.46 0.50 0.15 1.99
Published on 07 December 2011. Downloaded on 20/10/2014 10:47:33.

Total score 2.87 1.75 0.50 0.28 2.77 1.76 0.55 0.30 2.71 1.75 0.60 0.15

performed in SAS 9.1. Detail information about the ANCOVA Preparatory Chemistry (both repeaters and rst-timers) is
analysis is presented in the ESI.w Finally, chi-square compari- 2.71, which breaks down as 2.77 for repeaters and 2.71 for
sons allowed the identication of two alternative conceptions rst-timers. For students with Preparatory Chemistry the
that are less prevalent in a particular group. mean is 2.87. Overall, these low means suggest that students
have a poor understanding of the two chemistry concepts
Guessing value
regardless of their pathway into the course.
Although the strength of a multiple choice assessment is that it
can be given to a large number of students, as in this setting, a
Reliability and validity
weakness is that is hard to identify whether students have a
real misconception or are just guessing. With smaller groups, The reliability of the Diagnostic Instrument scores was examined
interviews and/or open-response options allow for additional using Cronbachs alpha estimates. A Cronbachs alpha of 0.62
analysis, but with over 500 students in a sample, the workload is obtained when students responses for the 20 questions are
for these two strategies becomes prohibitively large, and clever considered, which is close to the value of 0.66 reported in the
ways of using multiple-choice instruments are needed. literature (Othman et al., 2008) and not too far below the usual
Following Othman et al. (2008), student response patterns benchmark of 0.7 (Murphy and Davidshofer, 2005). Since this
were used to identify prevalent alternative conceptions, but, rather is a two-tier diagnostic instrument, two questions create an
than applying an across-the-board cuto of 10%, a variable item for a total of 10 items. Therefore, one would expect to
percentage based on the potential for guessing associated with calculate a reliability coecient based on 10 item scores rather
a given item serves as the cuto. One way of handling the than on 20 question scores. In that case a Cronbachs alpha of
guessing eect associated with a multiple choice assessment is 0.42 is produced, which is considerably lower than cited in the
to decide that, for a guesser, all options are equally plausible. For literature. Splitting the instrument into its two suggested
an exam question that has four options, the chance of randomly factors results in even lower reliability coecients. Overall,
guessing the correct answer would then be approximately 25%. these reliability investigations suggest that, for this sample, it is
However, for a two-tier item, if each question has four options, prudent to avoid making claims beyond the item level. In other
the chance of guessing the correct answer combination drops words, from a reliability perspective, the instrument appears to
to approximately 6% (0.25  0.25). be functioning similarly to an end-of-chapter test covering
dierent aspects of related topics.
Another way to check for evidence of relationships among
Results and discussion
items is to utilize factor analysis. Othman et al. (2008) proposes
For the Spring 2010 data, 683 responses were returned. Four that the Diagnostic Instrument is divided into two concepts, the
sets of responses could not be veried as from students particulate nature of matter and chemical bonding. Therefore,
enrolled in the course, and were therefore excluded from this since the sample size is suciently large (Hogarty et al., 2005;
analysis for a total of 679 students. For the remaining 679 MacCallum et al., 1999) a conrmatory factor analysis (CFA)
students, the item mean scores range from 0.06 (6%) to 0.46 was performed in Mplus 5.2 on a rst-order model with two
(46%), and standard deviations range from 0.24 to 0.50. latent factors that were allowed to correlate. According to the
Descriptive statistics for each of the three groups are shown proposed model items 1, 2, 3, 4, and 5 were set to load in factor
in Table 1. Items 6 and 7 are extremely dicult for all three Particulate Nature only, and items 6, 7, 8, 9, and 10 were set to
groups, with less than 10% of students answering correctly in load in factor Chemical Bonding only. While model t statistics
each case. The large values (larger than 1) for the skewness and (see ESIw) were within the range for an acceptable t (Brown,
kurtosis for these two items suggest violation of normality, so 2006), the factor loadings were problematic. As shown in Table 2,
interpretations based solely on these items must be cautious. all items except 1 and 3 load only weakly in their proposed factor.
The overall mean (out of 10 items) of students without In addition, the loadings for items 7 and 8 from the Chemical

This journal is c The Royal Society of Chemistry 2012 Chem. Educ. Res. Pract., 2012, 13, 3038 33
View Article Online

Table 2 CFA loadings for the 2-factor solution (N = 679) signicant but small eect. Previous exposure to the General
Chemistry I course had a positive though not signicant eect
Item Loading
on students scores. This result aligns with common sense about
Particulate nature of matter the eect of prior coursework in chemistry, and provides hope
Item 1 0.78 that Preparatory Chemistry is assisting students to some degree.
Item 2 0.29
Item 3 0.88 However, since the second tier of each item provides information
Item 4 0.20 about the student conceptual understanding, a detailed item-
Item 5 0.27 by-item look at the data will determine whether there are
Chemical bonding
Item 6 0.40 any particular alternative conceptions that appear to a lesser
Item 7 0.16a extent for the prep-chem group of students.
Item 8 0.05a
Item 9 0.42
Item 10 0.44 Alternative conceptions
Not signicant. As discussed, the instrument seems to be functioning similarly
Published on 07 December 2011. Downloaded on 20/10/2014 10:47:33.

to a focused content test in this setting. The best approach is to

Bonding concept are not large enough to be signicant. A CFA look at total score, as above, followed by performance on
for a 1-factor model, which places all 10 items in a single factor, individual items. At the question level, students tend to perform
was also performed. The t statistics were slightly worse than for better on the content part of an item (rst tier) than on the
the 2-factor model, and factor loadings were very similar to the reasoning part (second tier), and getting both parts of an item
ones presented for the 2-factor model, so are not shown here. Like correct is dicult. These results are consistent with those
the reliability analysis, these factor analysis results again suggest reported by Othman et al. (2008). Students may know the
the instrument is functioning more like a traditional content test correct answer for the content question being asked but have
in this setting, with student responses for one item on the test not not understood why it is so, indicating a lack of understanding
necessarily related to their responses for a another item. of the concept. These ndings support the idea that it is
appropriate to look at student response patterns within an item
Comparison of students performance with and without
when deciding whether alternative conceptions are present.
preparatory chemistry
How to do so requires some decisions.
At rst glance, although students in Group A (with prep-chem) In this study, the number of response options for each
performed slightly better than students in the other two groups question varies from two to ve, so the chance of guessing
(Table 1) on the Diagnostic Instrument, the dierence is quite the correct answer for both items under the equal plausibility
small, such that ANOVA did not nd a statistically signicant assumption varies as well, but is never higher than 12.5%.
dierence among groups (F (2, N = 679) = 0.63, p = 0.53). Since the means for the three groups in this study, with
However, the three groups are dierent in other ways. The preparatory chemistry (2.87), repeaters (2.77), and rst-timers
students in Group A had signicantly lower prior achievement (2.71), are quite a bit lower than the mean of 5.2 reported by
in math than the other two groups. This is particularly Othman et al. (2008), we were concerned that students may
important since in the study setting, prior math achievement have been guessing in many cases. Given that, in the real world
as measured by SAT Math score has been strongly associated of guessing, not all response options are equally plausible, we
with chemistry content measures (Lewis and Lewis, 2007). chose a cuto value of twice the approximate random guessing
Therefore, to further investigate the student performance on value. In this way, we can be certain that the response patterns
the Diagnostic Instrument, an ANCOVA was performed we identify are very likely to be common alternative conceptions.
(Stevens, 1999). This analysis allowed us to determine whether In other words, if students choose a particular combination of
the inclusion of SAT Math score as a covariate would reveal a response options for the two items at a level that is so much
signicant pathway eect (see the ESIw). The main eect of higher than by chance alone, we can say with condence that
the grouping variable was signicant (F (3, N = 544) = 3.73, there is something attractive about that combination.
p = 0.02), yielding adjusted group means of 2.93, 2.65 The observed percent of students choosing alternative
and 2.45 for Groups A, B, and C respectively. In other words, conceptions is compared with twice the approximate guessing
after controlling for SAT Math, students who took the prep- value in Tables 3 through 6. We will use this comparison to
chem course (Group A) are predicted to score 0.48 points guide us in the next phase of analysis, determining whether
higher on the Diagnostic Instrument than the students who there is any evidence that a Preparatory Chemistry course
simply waited to take General Chemistry I (the rst-timers, or helps to alleviate a prevalent misconception, but a descriptive
Group C). Although the repeaters (Group B) are predicted to look at the patterns is also revealing.
score 0.20 points higher than rst-timers (Group C) in the A total of eleven alternative conceptions related to the topic
course, this dierence is not signicant. A graphical display of of Particulate Nature of Matter were found, and are shown in
the ANCOVA analysis is provided in the ESI.w Tables 3 and 4. Five answer choice combinations were above
In summary, before the inclusion of SAT Math scores as a the cuto for all three groups of students, including, in order
covariate, no statistically signicant dierence among groups of decreasing frequency, the notion that an atom has the
was found. The ANCOVA, however, revealed that the Pre- same properties as an element, that stirring is necessary for
paratory Chemistry course was associated with students higher dissolution to occur, and that iodine weighs less in the vapor
performance on the Diagnostic Instrument, having a statistically phase than in the solid phase. For phase changes in water,

34 Chem. Educ. Res. Pract., 2012, 13, 3038 This journal is c The Royal Society of Chemistry 2012
View Article Online

Table 3 Alternative conceptions about the Particulate Nature of Matter held by all three groups of General Chemistry I students (N = 679)

A: With
Choice Prep-Chem B: Repeaters C: First-timers Twice %
Alternative Conception combination (N = 348) (N = 108) (N = 223) guessing value
Molecular and macroscopic properties
An atom is the smallest particle of an element that has the same Item 5 [A1] 49.4 43.5 42.6 12.5
properties as the element.
A solute only dissolves when stirring causes the crystals to break into Item 4 [B4] 39.1 37.0 44.4 25
smaller particles that can no longer be seen.
Conservation of matter during phase changes
Iodine gas weighs less than solid iodine. Item 2 [A1] 21.8 22.2 22.8 17
Boiling/evaporation of water
The hydrogen and oxygen atoms in water molecules break away from Item 1 [B1] 13.2 23.1 14.0 8
each other to form gaseous oxygen and hydrogen.
Water molecules have escaped into the air, and are not represented Item 3 [C2] 10.34 11.1 11.21 8
Published on 07 December 2011. Downloaded on 20/10/2014 10:47:33.

in a particulate way.

Table 4 Alternative conceptions about Particulate Nature of Matter held by at least one group of General Chemistry I students (N = 679)a

Choice A: With Prep- B: Repeaters C: First-timers Twice %

Alternative Conception combination Chem (N = 348) (N = 108) (N = 223) guessing value
Boiling/evaporation of water
Water molecules have broken free from one another and Item 3 [D4] 9.18 7.41 14.8 8
decomposed into oxygen and hydrogen atoms.
Water molecules have decomposed into diatomic oxygen Item 3 [A3] 8.33 11.1 4.93 8
and hydrogen gas.
Water molecules have decomposed into oxygen atoms and Item 3 [D1] 3.45 9.25 5.83 8
hydrogen atoms.
When the water is heated, the air between the water molecules Item 1 [D2] 9.70 5.56 10.3 8
is released in the form of bubbles.
The hydrogen and oxygen atoms in water molecules break Item 1 [D1] 8.90 7.41 10.8 8
away from each other to form gaseous water vapor.
Heat energy is absorbed by the water and released as bubbles. Item 1 [D3] 6.90 12.0 13.5 8
A bold number is used to indicate being above the cuto in the last column.

there is more diversity. Two answer choice combinations, that cuto in two instances and the prep-chem students are slightly
hydrogen and oxygen atoms in water molecules break apart to below for the third.
form gases and that evaporation can be thought of as water A total of seven alternative conceptions related to the topic
molecules escaping into the air with no particulate representation, of Chemical Bonding were found, and are shown in Tables 5
are above the cuto for all three groups. Six other combinations and 6. Four alternative conceptions were found in all three
are more mixed, with not all groups showing evidence of the groups of students, including that molten calcium uorides
identical alternative conceptions. Choosing the correct parti- free electrons allow it to conduct electricity, and that sodium
culate representation for evaporated water was, however, a chloride contains molecules formed by the donation of a
problem for all three groups, as over 50% of each group got valence electron from sodium to chlorine. The two other
the rst tier of item 3 wrong (see the ESIw). For three cases, in answer combinations have opposite perspectives on whether
which students answered the rst tier of item 1 correctly (that carbon dioxide has low melting and boiling points but the
the bubbles in boiling water contain water vapor) but had commonality of using an empirical fact (carbon dioxide is a
trouble with the reason, the repeaters are slightly below the gas at room temperature) rather than a particulate-level

Table 5 Alternative conceptions about Chemical Bonding held by all three groups of General Chemistry I students (N = 679)

Choice A: With Prep- B: Repeaters C: First-timers Twice %

Alternative Conception combination Chem (N = 348) (N = 108) (N = 223) guessing value
Electrical conductivity of ionic compounds
Calcium uoride is an ionic compound; it has free electrons Item 8 [A3] 42.5 31.5 33.2 25
that enable it to conduct electricity.
Structure of sodium chloride
After donating its valence electron to the chloride, the sodium Item 6 [A2] 39.1 39.8 33.2 25
ions form a molecule with the chloride ion.
Intermolecular and intramolecular forces
Carbon dioxide has low melting and boiling points because Item 7 [A4] 29.9 38.9 41.3 25
is a gas at room temperature.
Carbon dioxide doesnt have low melting and boiling points Item 7 [B4] 28.5 27.7 34.5 25
because is a gas at room temperature.

This journal is c The Royal Society of Chemistry 2012 Chem. Educ. Res. Pract., 2012, 13, 3038 35
View Article Online

Table 6 Alternative conceptions about Chemical Bonding held by at least one group of General Chemistry I students (N = 679)a

Choice A: With Prep- B: Repeaters C: First-timers Twice %

Alternative Conception combination Chem (N = 348) (N = 108) (N = 223) guessing value
Electrical conductivity of ionic compounds
NaCl produces free electrons when dissolved in water, Item 9 [A3] 24.7 19.4 26.0 25
but not when is solid.
Calcium uoride consists of covalent molecules. Item 8 [B4] 18.9 22.2 30.0 25
Structure of sodium chloride
The sodium atom shares a pair of electrons with the Item 6 [A1] 21.6 32.4 33.6 25
chlorine atom to form a simple molecule.
A bold number is used to indicate being above the cuto in the last column.

explanation as a reason. Two alternative conceptions were stirring in dissolution, the nature of the ionic bond in sodium
found among rst-timers only, that sodium chloride produces chloride, and the mechanism by which a molten ionic compound
Published on 07 December 2011. Downloaded on 20/10/2014 10:47:33.

free electrons when dissolved in water and that calcium uoride conducts electricity, in which most of the students in each
is made up of covalent molecules. Finally, the alternative group responded with alternative conceptions.
conception that sodium chloride is made up of covalent Two tier multiple-choice instruments like the one used in
molecules involving a shared pair of electrons is prevalent for this study are good formative assessment tools. The use of a
repeaters and rst-timers, but the students with prep-chem second tier (reasoning) provides instructors with more infor-
were not, as a group, quite as attracted to this idea. mation about student conceptual understanding. The next
Since we are interested specically in the role Preparatory step, once alternative conceptions have been uncovered, is to
Chemistry is supposed to play in helping students understand determine whether instructional strategies can improve student
concepts from secondary school chemistry before they enter understanding. These types of assessments then become useful
General Chemistry I, we focus on the two cases from the as evaluative resources to examine the eectiveness of instruc-
descriptive analysis in which the cuto indicated an alternative tion in addressing student alternative conceptions.
conception for the other two groups, but not for the students In terms of the role of Preparatory Chemistry, the ANCOVA
with prep-chem. The question is whether the percentage of results, although statistically signicant, showed that the
students with the alternative conception is signicantly dierent course eect on the overall Diagnostic Instrument score is
among the three groups, which can be addressed with a chi-square small. In addition, we were able to nd only two alternative
test. For the alternative conception that heat energy is absorbed conceptions that were less attractive to students with prep-
by boiling water and released as bubbles, the percentages were chem as compared to the other two groups. Therefore, it is
found to be signicantly dierent (w2 (2, N = 679) = 7.25, dicult to conclude that exposure to Preparatory Chemistry
p = 0.03), with prep-chem signicantly lower. For the helped students understanding of the particulate nature of
alternative conception that sodium chloride is made up of matter and chemical bonding in a substantive way. The results
covalent molecules involving a shared pair of electrons, of this assessment must be used to rethink the role of that
again the percentages were found to be signicantly dierent course. If the course intention remains to help students to
(w2 (2, N = 679) = 11.00, p = 0.003) with prep-chem understand and apply the particulate nature of matter as
signicantly lower. described in the syllabus, specically targeting the alternative
While these statistical tests provide some evidence that conceptions found in this study can help.
Preparatory Chemistry students lack two alternative ideas that A contribution of this study has been the implementation of
the other two groups of students still have, when the curriculum a cuto derived from a plausible guessing frequency to identify
for Preparatory Chemistry is examined, it is not at all clear by students alternative conceptions. An approximate guessing
what mechanism these two particular alternative conceptions percentage for each item was calculated so that response
would have been corrected while the other alternative concep- patterns that were the result of guessing alone could be
tions persisted. As indicated by all three groups responses to reduced to the level of noise. Using twice the approximate
the Diagnostic Instrument, the dismaying news is that a high guessing value as a cuto, we were able to identify very
percentage of all students entering General Chemistry I in common alternative conceptions in the study population.
Spring 2010 had not yet understood fundamental ideas of the One of the limitations of this study includes using a low
particle theory, even though one group of them took a course stakes exam to evaluate students alternative conceptions.
that, on the surface, contained exactly the right level of content Although the Diagnostic Instrument was given under similar
knowledge for them to be successful on this instrument. conditions as the regular course exams, students knew that it
was a low stakes exam. This can inuence the performance of
the students, reducing the impetus to think hard about subtle
dierences among response options. Another limitation is that
In general, all three groups of students have similar alternative the Diagnostic Instrument was not given again to the students
conceptions, which have also been reported in previous studies at a later point in the term. Therefore, we do not have any data
(Johnson, 1998; Harrison and Treagust, 2002; Mulford and about whether the students were able to overcome or change
Robinson, 2002; Othman et al., 2008). This is particularly true the alternative conceptions described in this paper. This next
for the distinction between an atom and an element, the role of step will be particularly important in the future, to determine

36 Chem. Educ. Res. Pract., 2012, 13, 3038 This journal is c The Royal Society of Chemistry 2012
View Article Online

to what degree the General Chemistry I course instructors are Ebenezer J. V. and Gaskell P. J., (1995), Relational conceptual change
able to use these results to inform their teaching. in solution chemistry, Sci. Educ., 79(1), 117.
Garcia C. A., (2010), Tracking chemistry self-ecacy and achievement
Using this instrument to try to understand the role of in a preparatory chemistry course, Unpublished dissertation.
Preparatory Chemistry has left us with some outstanding Harrison A. G. and Treagust D. F., (2002), The particulate nature
questions for future research. We did identify specic answer of matter: Challenges in understanding the submicroscopic
combinations that were prevalent in all three populations, but, world. In J. K. Gilbert, O. De Jong, R. Justi, D. F. Treagust and
J. H. Van Driel (ed.), Chemical education: Towards a research-based
in the absence of interviews or open response items, we feel practice, (pp. 189212), Dordrecht: Kluwer Academic Publishers.
unsatised about the robustness of these alternative conceptions. Hogarty K. Y., Hine C. V., Kromrey J. D., Ferron J. M. and Mumford
Would students have responded in the same way if the questions K. R., (2005), The quality of factor solutions in exploratory factor
analysis: The inuence of sample size, communality, and over-
were phrased slightly dierently, or if they were asked to
determination, Educ. Psychol. Meas., 65(2), 202226.
explain their thinking? Constrained by a very large chemistry Hu L. and Bentler P. M., (1999), Cuto criteria for t indexes in
course-taking population and responsible to instructors who covariance structure analysis: conventional criteria versus new
value having information about all students, we do need a alternatives, Struct. Equ. Modeling: A Multidisciplinary Journal,
6(1), 155.
multiple-choice approach. While the two-tier Diagnostic Jiang B., Xu X., Garcia C. A. and Lewis J. E., (2010), Comparing two
Published on 07 December 2011. Downloaded on 20/10/2014 10:47:33.

Instrument has a signicant advantage over a single-item test, tests of formal reasoning in a college chemistry context, J. Chem.
we believe there is room for still more renement. The number Educ., 87(12), 14301437.
of items related to each specic concept within the larger topics Johnson P., (1998), Progression in childrens understanding of basic
particle theory: longitudinal study, Int. J. Sci. Educ., 20, 393412.
of Particulate Nature of Matter and Chemical Bonding limited Johnson P., (2005), The development of childrens concept of a
our understanding with regard to the students understanding of substance: A longitudinal study of interaction between curriculum
the specics. Having multiple items designed to probe for the and learning, Res. Sci. Educ., 35(1), 4161.
Lewis S. E. and Lewis J. E., (2005), Departing from lectures: An
same misconception would provide a greater certainty that evaluation of a peer-led guided inquiry alternative, J. Chem. Educ.,
students are answering as they truly think, and are not overly 82(1), 135139.
inuenced by a small detail of question wording or a momentary Lewis S. E. and Lewis J. E., (2007), Predicting at-risk students in
distraction. What this study has highlighted for us is the critical general chemistry: comparing formal thought to a general achieve-
ment measure, Chem. Educ. Res. Pract., 8(1), 3251.
importance of having even shorter, more-tightly-focused instru- Lewis S. E. and Lewis J. E., (2008), Seeking eectiveness and equity in
ments that help instructors probe particular alternative concep- a large college chemistry course: An HLM investigation of peer-led
tions at the appropriate time in the curriculum. guided inquiry, J. Res. Sci. Teach., 45(7), 794811.
MacCallum R. C., Widaman K. F., Hing S. and Zhang S., (1999),
The Diagnostic Instrument was created to assess students
Sample size in factor analysis, Psychol. Methods, 4(1), 8499.
understanding about two broad concepts, particulate nature of McFate C. and Olmsted III J., (1999), Assessing student preparation
matter and chemical bonding. It was also intended to identify through placement tests, J. Chem. Educ., 76, 562565.
whether students understanding about one of the concepts Mulford D. R. and Robinson W. R., (2002), An inventory for
alternate conceptions among rst-semester general chemistry
inuences their understanding of the other concept. Results students, J. Chem. Educ., 79(6), 739744.
from the CFA in this case did not support a two-factor Murphy K. R. and Davidshofer C. O., (2005), Psychological testing:
structure, so we were unable to use the instrument for that Principles and applications, Upper Saddle River, NJ: Prentice Hall.
purpose within this study, and we look forward to others work Nyachwaya J. M., Mohamed A.-R., Roehrig G. H., Wood N. B.,
Kern A. L. and Schneider J. L., (2011), The development of an
on this topic. Regardless, the two-tier Diagnostic Instrument open-ended drawing tool: an alternative diagnostic tool for
provided useful information related to students conceptions assessing students understanding of the particulate nature of
about the particulate nature of matter and chemical bonding matter, Chem. Educ. Res. Pract., 12(2), 121132.
Othman J., Treagust D. F. and Chandrasegaran A. L., (2008), An
that poses challenges for the courses investigated in this study.
investigation into the relationship between students conceptions of
the particulate nature of matter and their understanding of
chemical bonding, Int. J. Sci. Educ., 30(11), 15311550.
Acknowledgements Potgieter M., Ackermann M. and Fletcher L., (2010), Inaccuracy of self-
evaluation as additional variable for prediction of students at risk of
The authors thank Sachel Villafane for her involvement and failing rst-year chemistry, Chem. Educ. Res. Pract., 11, 1724.
support in data analysis and writing, and the course instructors Russell A. A., (1994), A rationally designed general chemistry diagnostic
test, J. Chem. Educ., 71, 314317.
for General Chemistry I for being willing to learn about their Salta K. and Tzougraki C., (2011), Conceptual Versus Algorithmic
incoming students ideas about chemistry. Problem-solving: Focusing on Problems Dealing with Conservation
of Matter in Chemistry, Res. Sci. Educ., 41, 587609.
Spencer H. E., (1996), Mathematical SAT test scores and college
References chemistry grades, J. Chem. Educ., 73, 11501153.
Stevens J. P., (1999), Intermediate statistics: a modern approach,
Brown T. A., (2006), Conrmatory factor analysis for applied Mahwah, NJ: Lawrence Erlbaum Associates.
research, (1st ed.), New York: Guilford Press. Taber K. S., (2001), Building the structural concepts of chemistry:
Bunce D. M. and Hutchinson K. D., (1993), The use of the GALT Some considerations from educational research, Chem. Educ. Res.
(Group Assessment of Logical Thinking) as a predictor of Pract., 2(2), 123158.
academic success in college chemistry, J. Chem. Educ., 70, 183187. Tai R. H., Sadler P. M. and Loehr J. F., (2005), Factor inuencing
Examinations Institute, (2011), American Chemical Society Division success in introductory college chemistry, J. Res. Sci. Teach., 42,
of Chemical Education, Full Year General Chemistry, University 9871012.
of Iowa: Ames, IA. Teichert M. A., Tien L.T., Anthony S. and Rickey D., (2008), Eects
Costu B., (2008), Big bubbles in boiling liquids: students view, of context on students molecular-level ideas, Int. J. Sci. Educ.,
Chem. Educ. Res. Pract., 9, 219224. 30(8), 10951114.
Crocker L. and Algina J., (2006), Introduction to classical and modern Thompson B., (2003), Score reliability: Contemporary thinking on
test theory, Wadsworth Pub. Co. reliability issues, Thousand Oaks, CA: Sage Publication, Inc.

This journal is c The Royal Society of Chemistry 2012 Chem. Educ. Res. Pract., 2012, 13, 3038 37
View Article Online

Treagust D. F., (1986), Evaluating students misconceptions by Tsai C. C. and Chou C., (2002), Diagnosing students alternative
means of diagnostic multiple-choice items, Res. Sci. Educ., 16, conceptions in science, J. Comput. Assist. Learn., 18, 157165.
199207. Villafane S. M., Bailey C. P., Loertscher J., Minderhout V. and
Treagust D. F., (1988), Development and use of diagnostic tests to Lewis J. E., (2011), Development and analysis of an instrument
evaluate students misconceptions in science, Int. J. Sci. Educ., 10, to assess student understanding of foundational concepts before
159170. biochemistry coursework, Biochem. Mol. Biol. Educ., 39(2),
Treagust D. F., Chandrasegaran A. L., Zain A. N. M., Ong E. T., 102109.
Karpudewan M. and Halim L., (2011), Evaluation of an interven- Voska K. W. and Heikkinen H. W., (2000), Identication and
tion instructional program to facilitate understanding of basic analysis of student conceptions used to solve chemical equilibrium
particle concepts among students enrolled in several levels of study, problems, J. Res. Sci. Teach., 37(2), 160176.
Chem. Educ. Res. Pract., 12(2), 251261. Wagner E. P., Sasser H. and DiBiase W.J., (2002), Predicting students
Tro N. J., (2009), Introductory chemistry (3rd ed.). Upper Saddle at risk in general chemistry using pre-semester assessments and
River, NJ: Pearson. demographic information, J. Chem. Educ., 79, 749755.
Published on 07 December 2011. Downloaded on 20/10/2014 10:47:33.

38 Chem. Educ. Res. Pract., 2012, 13, 3038 This journal is c The Royal Society of Chemistry 2012