Article

pubs.acs.org/jchemeduc

Development of the Flame Test Concept Inventory: Measuring

Student Thinking about Atomic Emission
Stacey Lowery Bretz* and Ana Vasquez Murata Mayo
Department of Chemistry and Biochemistry, Miami University, Oxford, Ohio 45056, United States

ABSTRACT: This study reports the development of a 19-item Flame Test

Concept Inventory, an assessment tool to measure students’ understanding of
atomic emission. Fifty-two students enrolled in secondary and postsecondary
chemistry courses were interviewed about atomic emission and explicitly asked to
explain flame test demonstrations and energy level diagrams. Analysis of students’
explanations offered insight into students’ alternative conceptions and was used to
design items and distractors for a 19-item Flame Test Concept Inventory about
atomic emission. Results from a pilot study with first-year university chemistry
and with upper-division chemistry students were analyzed to create a final version
of the inventory that was administered to both secondary students (N = 459) and
first-year university students (N = 100) who had completed formal instruction
and course assessment about atomic emission. Analysis of student responses indi-
cated the inventory generated valid and reliable data. Common alternative
conceptions about atomic emission that remain postinstruction and their prevalence
are discussed.
KEYWORDS: High School/Introductory Chemistry, First-Year Undergraduate/General, Upper-Division Undergraduate,
Analytical Chemistry, Chemical Education Research, Demonstrations, Misconceptions/Discrepant Events, Testing/Assessment,
Atomic Spectroscopy, Atomic Properties/Structure
FEATURE: Chemical Education Research

■ INTRODUCTION
Students can construct coherent understandings of phenomena
that do not match accepted scientific views. These alternative
conceptions, if not challenged through instruction and assess-
ment, can become integrated into students’ cognitive structures
and may interfere with subsequent learning of a new concept.1
One way that instructors can elicit their students’ alternative
conceptions is by administering a concept inventory to char-
acterize students’ understandings of a particular concept compared
to expert thinking.2,3
When developing a concept inventory, one goal is to design
an instrument that is easy to use, can be administered in a short
period of time, and can accurately identify students’ alternative
conceptions by generating reliable and valid data.4 A variety of
methods for developing concept inventories exist, ranging
from the creation of questions, responses, and distractors being
drafted by expert mapping of the content domain5 to gen-
eration of items through clinical interviews with students.6
The National Research Council7 recommends the use of
student interviews in the development of assessment instru-
ments because interviews generate qualitative data that can be
used to formulate attractive distractors written in the natural
language of students, thus ensuring that the test elicits common
student ideas and reasoning difficulties.8 Such distractor-driven
multiple-choice tests combine the richness of qualitative research
with the power of quantitative assessment, thereby creating a tool
© 2017 American Chemical Society and
Division of Chemical Education, Inc.
to measure the prevalence of alternative conceptions with larger
numbers of students.
Concept inventories have been reported in the literature for a
variety of chemistry concepts including particulate nature
of matter,9−11 bonding,12,13 light and heat,14−17 equili-
brium,18,19 inorganic qualitative analysis,20 ionization energy,21
organic acids,22 phase changes and solutions,23,24 quantum
chemistry,25 redox reactions,26 and biochemistry concepts.27,28
A majority of these concept inventories consist of questions
that ask students to interact with either symbolic or particulate
representations of matter. None of these concept inventories,
however, asks students to interact with macroscopic representa-
tions of chemistry such as visual observations in a laboratory
experiment or chemical demonstration. This paper reports the
development of a concept inventory that requires students to
interact with macroscopic observations, namely, an inventory to
measure students’ understandings about atomic emission in the
context of flame test observations.
The flame test is a longstanding demonstration in chemistry
classrooms. From 1928 to 2015, the Journal has published
32 different procedures for conducting flame test demon-
strations.29−60 Flame tests are a colorful, visually interesting
demonstration of the anchoring concept61−63 that matter
Received: August 25, 2017
Revised: October 27, 2017
Published: November 22, 2017
17 DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education
consists of atoms that have an internal structure that dictates
their properties often as part of the evidence that led to the
development of the quantum model. Despite the showy nature
of this demonstration, there is scarce evidence that it promotes
student learning: “students move on, graduate, and [when]
I cross paths with them, they remember that flame test that
I did for them; they may not remember the chemistry, but
they remember the demo.”64 While limited research has been
reported regarding students’ understandings of atomic line
spectra,65 there are no reports published to date of investigations
into student thinking about flame tests and atomic emission.
■
THEORETICAL FRAMEWORKS
In order to conceptualize the interview guide to investigate
students’ understandings about flame tests and atomic
emission, two different theoretical frameworks were employed.
Meaningful Learning
Ausubel and Novak have written extensively about the
construct of meaningful learning which is the process where
students make substantive connections between what they
need to learn and what they already know, i.e., their prior
knowledge.66,67 Meaningful learning stands in contrast to the
process of rote memorization where information is stored
without forming connections to prior knowledge. Ye and Lewis
explained rote learning as involving tasks such as “a first-year
chemistry student [being asked] to recall the color of a par-
ticular metal when it is put in a flame test. As the information
that is being recalled has no meaningful association with
existing content knowledge, the information must be learned
through rote learning.”68 Ausubel and Novak’s construct of
meaningful learning has informed several chemistry education
research studies including research leading to the develop-
ment of concept inventories regarding bonding representa-
tions,12 redox reactions,26 and challenges with learning organic
chemistry.69,70
Johnstone’s Domains
To guide our choices for investigating students’ knowledge
of atomic emission, the interview guide was constructed to
intentionally explore Johnstone’s domains. According to
Johnstone, the chemistry to be learned by students falls across
three domains: the macroscopic domain (the tangible or
visible), the particulate domain (the invisible), and the symbolic
domain (formulas or representations).71 Johnstone has argued
that one reason that science is so hard to learn is that, in formal
science classrooms, students are expected to simultaneously
understand the connections among all three domains whereas
they should learn about dyads between two of the domains
rather than have all three inflicted upon them at once.72
Therefore, the interview guide in this research study that was
constructed to purposefully explore the connections (or lack
thereof) in students’ understandings about atomic emission was
designed using prompts from two domains, namely, energy
level diagrams (symbolic) and flame tests (macroscopic).
■
METHODS
This research employed a mixed-method, sequential design.73
Semistructured interviews74,75 were conducted to elicit student
thinking about atomic emission. These findings were then used
to create items and distractors for a concept inventory that
was then used to quantify the prevalence of these under-
standings with a larger sample of students. Details on each of
these procedures are provided below.
18
Article
Interview Sample
The sample included students in advanced placement chemistry
(AP, N = 14) and secondary students in a first-year chemistry
course (SC, N = 12), with both groups enrolled at a large,
suburban, public secondary school in the midwestern United
States. The university sample consisted of undergraduate
students in first-year chemistry (FYC, N = 14) and under-
graduate students in upper-division chemistry (UD, N = 12),
with both groups enrolled in a large, public and predominantly
undergraduate institution in the midwestern United States.
The UD students were enrolled in either an instrumental
analysis or an analytical chemistry course. All data collection
was approved by the Institutional Review Board with regard
to protection of the rights of human subjects with respect to
informed consent and minimization of risk. Each student was
assigned a pseudonym.
Interviews
To develop a “thick description”76 of students’ understandings
about atomic emission, students were interviewed individually
in an instructional laboratory setting so that flame tests could
be conducted. Secondary student interviews lasted an average
of 60 min and were conducted at their school. University
student interviews lasted an average of 45 min. Interviews were
both audio and video recorded in order to capture student
interactions with the prompts described below.
The interviews consisted of four phases. Phase I asked open-
ended questions about atomic emission, and students were
asked to create representations77 to explain how atoms release
energy. Students were asked to think aloud78 about the topic
and were provided with a periodic table, a digital Livescribe
Pulse Smartpen,79 and paper to draw if desired. Phase II
focused on flame test demonstrations, asking students to
respond to predict−observe−explain80 questions regarding
what would happen to three chloride salts in a flame test,
conducted one salt at a time. Of the 52 students interviewed, 48
reported being familiar with the flame test before it was
conducted in the interview. The flame tests were used to
prompt student thinking about atomic emission in the macro-
scopic domain of Johnstone’s triangle, and to elicit what
connections, if any, that the students identified between the
macroscopic domain and atomic structure. In Phase III,
students were asked to explain the conventions and symbols
found in an energy level diagram for a hydrogen atom such as
the meaning of n (principal quantum number), the horizontal
lines (energy levels), and the numbers (with a maximum of
zero) indicating energy. The energy axis was labeled “Energy ×
1020 (J/atom)”. Phase III explored the students’ understanding
of the symbolic domain and its connections to atomic structure
with regard to electronic transitions. Lastly, in Phase IV, two
additional energy level diagrams were shown to students: one
with an arrow pointing from n = 2 to n = 4 and the other with
an arrow pointing from n = 4 to n = 2. The students were asked
to choose whether one, both, or neither of these diagram(s)
might represent the flame tests they had carried out in Phase II
of the interview. The intent of Phase IV was to elicit con-
ceptions related to connections between the symbolic and the
macroscopic domains.
Data Treatment
All interviews were transcribed verbatim using both the audio
and video recordings. Data management was facilitated using
QSR NVivo8.81 The interview transcripts were analyzed
for alternative conceptions using the Constant Comparison
DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education
Method (CCM) which is an inductive data coding process
used to categorize and compare qualitative data for analysis
purposes.82 In CCM, a unit of data (i.e., a transcript interview)
is analyzed and broken into codes based on emerging themes,
which are then organized into categories that reflect a logical
understanding of the codes. Then, this analyzed unit of data is
systematically compared with another unit of data. Ideally, this
iterative process goes on until theoretical saturation is reached,
which occurs when no new categories or new information
emerge from subsequent interviews. In this study, a purposeful
sample was used to include a diverse range of student expertise,
secondary and postsecondary chemistry students with the goal
to reach theoretical saturation.
A detailed analysis of students’ explanations has been
reported.83 The alternative conceptions that emerged through
the analysis of student interviews formed the basis for creating
questions and distractors for the flame test concept inventory
(FTCI) in the form of multiple-choice items. For example,
Question 8 on the FTCI is shown in Box 1, and the correct
answer is indicated with an asterisk.
Box 1. Question 8 on the Flame Test Concept Inventory
Q8. Copper(II) chloride releases energy in a flame test when...
A. losing valence electrons
B. gaining valence electrons
C. breaking bonds in the compound
*D. valence electrons return to the ground state
All the distractors on the FTCI were inspired by multiple
explanations offered by students during the clinical interviews.
For example, responses A, B, and C in Question 8 were crafted
from students’ explanations about losing/gaining electrons and
breaking bonds during the flame test:
“In the flame test the valence electrons on the substances were
lost to make them more stable that’s why the color of the
flame stopped and when back to normal (referring to initial
color) after a while, so this chart (energy level diagram with
arrow pointing from n = 4 to n = 2) shows that after a while,
after the flame test loses electrons, it became more stable, the
energy decreases.” (Alex, SC)
“When atoms lose energy, (pause), when is releasing energy,
one of them is gaining an electron and one of them is losing
an electron (referring to arrows in energy level diagrams),
just trying to figure out [pause], to go up in energy level
(indicates energy level diagram with arrow going from n = 2
to n = 4), you have to gain an electron, but this is going to
coincide with gaining an electron and this one (points
at energy level diagram with arrow pointing from n = 4 to
n = 2) is going to be losing an electron.” (Rachel, UD)
“When the copper chloride burns (during flame test) there
are bonds breaking, one way the energy’s being released is in
that certain wavelength of light that it is being emitted.”
(Arthur, AP)
“Energy is stored in the bonds between atoms, not in the
actual atoms itself, and when you break the bonds that’s
when energy is released... The energy release by the bonds
breaking through the heat (during flame test), and
depending in what wavelength is the visible light spectrum,
wherever that lands, that’s the color you see.” (Waldo, AP)
“The flame (in flame test) is required to have the reaction
take place and to break the bonds apart and to make the
electrons jump.” (Michael, FYC)
19
Article
Expert Content Validation
A preliminary version of the FTCI was a paper-and-pencil
inventory of 18 multiple-choice items. Each copy of the FTCI
included a handout with color pictures of three chloride salts
and their respective flame tests. Seven faculty members at
the researchers’ institution who were experienced first-year
chemistry and/or analytical chemistry instructors were asked to
review the chemistry content of the FTCI for accuracy.
Specifically, the faculty were asked to answer these three
questions:
1. Which item(s), if any, do you find confusing? Why?
2. Which item(s) best represent atomic emission? Why?
3. Which topic(s) were omitted that you feel need to be
included to best represent atomic emission? Why?
The faculty provided several comments and suggestions to
improve this preliminary version of the FTCI. For example, one
expert suggested that an item that included a student-generated
Bohr atomic model to represent a multielectron substance
be followed by an item that explicitly asked students about the
limitations of that student-generated Bohr model. The item
containing a Bohr atomic model was not deleted because that
model was the one most commonly drawn by secondary
students in their explanations of how atoms release energy.
The expert was agreeable to retaining the item with the model
as long as an additional item was added to explore students’
comprehension of the limitations of that model for multi-
electron atoms. Other experts suggested changes in wording for
some items to more accurately reflect the chemistry that takes
place in flame tests. For example, rather than talk about the ions
in the flame, the word “ions” was changed to “atoms” because
atomization more accurately describes the current scientific
understanding of the multiple processes taking place in a flame.
A careful analysis of expert suggestions, balanced against the
need to keep the essence of the alternative conceptions found
in the interviews, led to revisions in wording and the addition of
two new items, resulting in the pilot-test version of the FTCI
with 20 items.
Concept Inventory Administration
The 20-item version of the FTCI was pilot tested both with
FYC students (N = 222) and with UD students (N = 40)
enrolled in analytical chemistry courses. Seven of these students
(5 FYC, 2 UD) subsequently participated in individual student
validation interviews (described below), resulting in the
deletion of two items, the addition of one item, and revisions
in wording to others. The revised, final version of the FTCI that
consists of 19 items was then administered to secondary high
school chemistry students (SC, N = 308 and AP, N = 151)
from 15 schools in 9 states across the United States, and to an
additional FYC class (N = 100) giving consent for their results
to be used in our research. Three weeks later, three students
from the FYC class of N = 100 participated in validation
interviews; no additional modifications were deemed necessary.
The 19-item FTCI was then administered for a second time in
this same classroom in order to conduct a test−retest reliability
study. After omitting the responses from the three students
who participated in the validation interviews, the responses of
N = 80 students were used for a test−retest reliability study.
In all cases, directions were given to students by their instruc-
tors regarding voluntary participation and student consent for
the use of results in research. Students required 10−15 min to
respond to the FTCI, and the FTCI was administered after
students had been formally assessed by their instructors on the
DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education Article

topic of atomic emission in order to identify the most firmly Table 1. Student Scores for the 20-Item Pilot-Test Version
held alternative conceptions. Excel and SPSS84 were used to and the Final 19-Item Version of the FTCI
analyze the data.
Version Scoring (Students) N Mean SD Median Min Max
Student Validation Interviews
20 items 1-tier (FYC) 222 6.61 2.770 6 1 17
The student validation interviews offered insights as to how to 1-tier (UD analytical) 25 12.56 3.150 13 7 18
further improve the quality of the data generated by the FTCI 1-tier (UD instrumental) 19 12.31 3.130 13 6 18
by identifying problems or ambiguities in the content and 19 items 1-tier (SC) 308 6.01 2.910 6 1 15
clarity of items that could lead to confusion for students.85 2-tier (SC) 4.29 2.344 4 0 11
Students who reported being unfamiliar with the flame test 1-tier (AP) 151 8.87 5.008 7 0 18
agreed that the FTCI handout with color pictures of the flame 2-tier (AP) 6.79 3.888 5 0 14
tests was clear and self-explanatory. The first round of vali- 1-tier (FYC) 100 8.48 4.011 8 1 18
dation interviews, which each lasted approximately 45 min, 2-tier (FYC) 6.09 3.220 6 0 14
were conducted 3 weeks after the administration of the 20-item 19 items 1-tier test (FYC) 80 8.53 4.081 8 1 18
FTCI. This waiting time was chosen because it has been shown test−retest
that explicit memory does not affect responses in a test−retest 1-tier retest (FYC) 8.68 3.871 9 1 17
2-tier test (FYC) 6.08 3.352 6 0 14
situation if it is obtained at a 3 week interval.86 The validation
2-tier retest (FYC) 6.18 2.647 6 0 12
interviews followed a think-aloud protocol where students
reanswered all items, providing information about how well
they understood the items, and which strategies (elimination, p-values exceeding 0.05 indicate normality. Only the FYC
guessing) they used to choose an answer. If a student was not scores were normally distributed, as shown in Table 2. One
guessing and chose a distractor, this indicated the distractor unusual feature of the AP scores in Figure 2 is the prominent
was appealing and/or rooted in the alternative conceptions number of students who scored 17/19. The AP data set
identified in the four-phase flame test interviews.86 includes responses from students with 5 different teachers in

■ RESULTS AND DISCUSSION

Concurrent Validity
The scores for the pilot test of the 20-item FTCI for both FYC
(N = 222) and UD (N = 25 and N = 19) students can be found
in Table 1. Concurrent validity was examined to determine if
the 20-item version of the FTCI could successfully distinguish
between populations.87 The FYC mean (M = 6.61; SD = 2.77)
was significantly lower than the mean for either class of UD
students. The more experienced UD chemistry students scored
higher than the less expert FYC students. Two, one-tailed
t-tests established that the two UD classes were equivalent,88
resulting in the subsequent merger of both courses’ data for
analysis (n = 44; M = 11.5; SD = 2.83). A t-test indicated that
the UD pilot test FTCI scores were significantly higher than the
FYC scores [(t (264) = −11.50, p = 0.01 with η square (η2 =
0.33) showing a large effect accounting for 33% of the variability
between groups], thereby providing a form of concurrent
validity. It bears noting that, even though the UD students
outperformed the FYC students by a statistically significant
amount, the average UD student answered only 11.5 out of
20 (57.5%) items on the 20-item FTCI.
Descriptive Statistics
The final version of the FTCI consists of 19 items, including
four, two-tier answer−reason pairs.5 Student responses were
scored both as a one-tier FTCI with 19 points possible (“1” for
correct, “0” for correct) and again as a two-tier FTCI with 15
points possible (“1” for correct answers to both the answer and
the reason tiers, “0” for all other responses). Table 1 and
Figures 1−3 summarize the final version scores for secondary
students (SC), secondary students enrolled in an AP chemistry
course (AP), and FYC students, including test−retest data.
All SC, AP, and FYC score distributions were skewed to the
left, that is, score below the theoretical median.
The Kolmogorov−Smirnov (K−S) test was used to assess
the normality of the distributions in Figures 1−3 as the student
sample sizes were above 50.89 In the K−S test, the scores of
interest are compared to a normal distribution and, therefore,
20
4 states, but it would be incorrect to presume that the peak at
17 suggests there were only two (or three) questions that were
particularly difficult for the AP students. In fact, there were
six questions (nearly one-third of the items) on the FTCI
for which 30% or more of the AP students chose incorrect
distractors. The discriminatory power of the entire inventory
was measured by Ferguson’s delta (δ) where δ ≥ 0.9 indicates a
broad score distribution and good discrimination among all
students.90 Both one-tier and two-tier scorings of the final
version of FTCI yielded satisfactory results for all students
(Table 2).
Internal Consistency
The internal consistencies of SC, AP, and FYC scores on the
final 19-item version of the FTCI were measured by calculating
a Cronbach α where α ≥ 0.7 values are considered acceptable
and, thus, indicate that all items closely measure the same
construct.91 Kuder−Richardson (KR-20) indices were also
calculated as a measure of internal consistency to examine the
dichotomous scoring, either right or wrong, again with an
accepted criterion value of 0.7 or greater. Both one-tier and
two-tier FTCI scores yielded acceptable α values and KR-20
indices for both the AP and the FYC students (Table 3),
indicating that all items closely measured the same con-
struct.90 The scores of the less experienced SC students,
however, did not exceed either threshold, reflecting a more
fragmented knowledge about the concepts assessed by the
FTCI.92
Internal consistency was also examined by conducting a
test−retest analysis. Scores were available for N = 80 of the
FYC students (see Table 1) and used to calculate a stability
coefficient, as a measure of consistency in responses using
one-tier scoring. While the stability coefficient as measured by
the Pearson correlation resulted in a strong correlation of 0.591,
it did not meet the recommended threshold of 0.7. A lower
correlation, however, is not cause for concern as the appro-
priateness of internal consistency thresholds for CIs and
alternative conceptions have been recently questioned given
the challenges associated with measuring incomplete and
incorrect student understandings.3,22,92 Examining the correlation
DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education Article

Figure 1. FTCI one-tier and two-tier score distributions for SC (N = 308) students.

Figure 2. FTCI one-tier and two-tier score distributions for AP (N = 151) students.

Figure 3. FTCI one-tier and two-tier score distributions for FYC (N = 100) students.

between scores in a test−retest design does not account for
the fact that students with identical scores may not, in fact,
hold the same misconceptions. While internal consistency can
be artificially inflated by asking questions that iterate on the
same concept, when instruments are designed to detect
misconceptions of students that result from fragments of
knowledge and incorrect relationships among concepts, the
21
consistency among students’ responses is likely to be lower.
Most importantly, Streiner93 has cautioned against stringent
application of 0.7 as a threshold given that concept inven-
tories are not unidimensional. The FTCI measures students’
knowledge of flame tests, but the constructs required for
understanding here include electronic structure of the atom and
properties of light.
DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education Article

Table 2. Results for Test of Normality and Discrimination A high p-value (p > 0.8) may indicate that the item was too easy
for FTCI Final Version Scores for the test’s intended population and may not be appropriate
for inclusion in an inventory designed to elicit students’ alter-
Scoring (Student K−S
Categories) Statistic df p-Value Ferguson’s δ native conceptions. Note that a low p-value does not necessarily
1-tier (SC) 0.121 308 0.000 0.94
indicate a malfunctioning item. A valid item could be answered
2-tier (SC) 0.144 308 0.000 0.92
incorrectly by a large number of students because of the very
1-tier (AP) 0.170 151 0.000 0.95
fact that it addresses a deep-rooted alternative conception.
2-tier (AP) 0.181 151 0.000 1.00
Item discrimination was calculated to determine how well
1-tier (FYC) 0.088 100 0.052a 1.00
each item differentiated between students who scored in the
2-tier (FYC) 0.920 100 0.035a 0.95
top 27% and those whose scores were in the bottom 27% of all
a scores.94 Discrimination index values of D > 0.3 are considered
These p-values were found to be normally distributed.
ideal.90 A low item discrimination index can be measured when
an item is either too difficult or too easy because either extreme
Table 3. Internal Consistency for Final Version FTCI Scores of difficulty corresponds to having all students getting an item
Internal Consistency Measures either correct or incorrect.
Student Categories Scoring Cronbach’s α KR-20
Discrimination index versus item difficulty plots for SC, AP,
and FYC students can be found in Figure 4. The majority of the
SC 1-tier 0.55 0.55
items functioned acceptably for all students. A few items were
2-tier 0.50 0.50
answered correctly by fewer than 25% of the SC students,
AP 1-tier 0.87 0.87
indicating that the items were particularly difficult for those
2-tier 0.83 0.83
students and that students either resorted to guessing (25%
FYC 1-tier 0.76 0.77
marks the guessing threshold for a 4-distractor item) or found
2-tier 0.73 0.74
another distractor particularly attractive. Note that the most
difficult items also poorly discriminated, given that students in
Item Analysis both the top and bottom 27% found them difficult.
Classical test theory was used to evaluate item difficulty, Individual item reliability in the form of a point biserial (ρbis)
discrimination, and reliability. Item difficulty, p, is defined was calculated as the correlation between each item’s score
as the proportion of students answering that item correctly. (correct = “1” or incorrect = “0”) and the overall test score.
The acceptable range of item difficulty is 0.30 < p < 0.80.90 Satisfactory values for ρbis equal or exceed 0.2.90 The majority

Figure 4. Discrimination index versus item difficulty with one-tier scoring. (The number next to each dot indicates the FTCI item.)

22 DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education
of the items functioned acceptably for all students. Notably,
items 9 and 10 (Box 2; correct answers are marked with an
asterisk), which asked students about representing absorbance
and emission using a Bohr atomic model and about the
limitations of doing so, both had low ρbis values for both SC
and AP students, indicating the strong appeal of distractors in
both of those items. The ρbis was low for FYC students for item
10, which is the item that was strongly recommended during
expert content validation. Student validation interviews con-
firmed that students were not explicitly guessing the answer,
that they understood the intention of items 6 and 10, but that
they were still strongly attracted to distractors.
Alternative Conceptions as Measured by the FTCI
The FTCI items elicited alternative conceptions about atomic
emission. Each FTCI distractor is directly tied to an alternative
conception revealed through the interviews and falls into one of
these seven categories provided in Table 4.
Not one SC student correctly answered both items 4 and 15
(category b), and only 8% correctly answered items 9, 10, 11,
Table 4. FTCI Items as They Correspond to Alternative
Conceptions about Atomic Emission
Corresponding FTCI
Alternative Conception Categories Item(s)
a Misrepresentations of atomic emission 9, 10, 11, and 12
b Atomic properties affecting atomic absorption 4, 15
and emission
c Breaking and/or forming bonds affect 3, 5
absorption and emission
d Losing and/or gaining electrons 2, 6, 8
e Process related alternative conceptions 7, 13, and 14
f Heuristics 16 and 17; 18 and 19
g Terminology used out of context 1 During the validation interviews, students were focusing on the
23
Article
and 12 (category a). Figure 5 shows that 94% of SC students
held at least six common alternative conceptions about atomic
emission. It bears repeating that these data were collected af ter
students had completed instruction and been tested by their
instructor on these ideas.
Only 4% of AP students correctly answered items 9, 10, 11,
and 12 (category a), while 45% of students correctly answered
both items 4 and 15 (category b). Distractors in category a
were the most commonly selected by AP students, while
distractors in categories b and f were the least selected by these
students. Figure 5 shows that 75% of AP students held at least
four common alternative conceptions about atomic emission.
Only 4% of FYC students correctly answered items 9, 10, 11,
and 12 (category a), while 46% of students correctly answered
both items 4 and 15 (category b). Distractors in category a
were the most commonly selected by FYC students, while
distractors in categories b and f were the least selected by these
students. Figure 5 shows that 85% of FYC students held at least
four common alternative conceptions about atomic emission.
The distractors can be further categorized according to the
students’ difficulties with connecting Johnstone’s domains
(Table 5). For example, the “macroscopic/particulate” refers
to the difficulties that students had with connections between
the macroscopic and particulate domains. Students’ alternative
conceptions regarding the connections between the macro-
scopic and particulate domains are evidenced by their selection
of distractors across five different FTCI items (Table 5).
Students’ alternative conceptions at the “macroscopic/symbolic”
interface are evident by the popularity of distractors in Table 5
regarding the colors of the flame test and their connection to
energy level diagrams. It was particularly challenging for students
to connect the energy level diagrams to a color such as “red”.
DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education Article

Figure 5. Students holding one or more alternative conceptions about atomic emission in the context of a flame test.

Table 5. Comparison of Students Choosing FTCI Distractors about Atomic Emission as They Correspond to Johnstone’s
Domains

direction of arrows and the terms “absorption” and “emission”,
rather than explaining how the size of the gap between energy
levels would result in the generation of different colors being
24
observed. Their confusion about the relationship between the
symbolic/particulate domains was also revealed by their com-
bined responses to a two-tier question, where the first tier asked
DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education
them to choose a symbolic representation of emission and the
second tier asked them to choose a reason for their symbolic
selection.
■
CONCLUSIONS AND IMPLICATIONS FOR
TEACHING
The flame test concept inventory generates reliable and valid
data for instructors’ use in assessing their students’ alternative
conceptions about atomic emission. The FTCI is an easy to
administer CI and requires only 10−15 min, but if an instructor
has limited class time, s/he may select individual items for
in-class discussion or for use as a formative assessment in
classroom response systems, for example, “clicker” questions.
The flame test is a colorful demonstration that easily captures
the attention of students. However, the results from student
interviews and responses to the FTCI suggest that there is a
large gap between the positive affective response garnered by
the flame test and the cognitive understanding of what takes
place and what these observations suggest to chemists about
atomic structure. Instructors could engage students in their
classroom using the flame test demonstration accompanying
the “predict, observe, and explain” tasks used in the interview
protocol and have a class discussion.
Instructors who teach upper-division courses may wish to
formatively assess what prior knowledge their students bring
with them as residual from earlier coursework in chemistry.
The development of FTCI shed light onto alternative con-
ceptions related to challenges with understanding representa-
tions of the Bohr atomic model and energy level diagrams.
Interviews with secondary students indicated their preference
for the Bohr atomic model to explain how atoms release energy.
The FTCI includes both of these representations because
students bring these preferences to their first-year chemistry class-
rooms. Instructors can assess understanding of the limitations to
these models and start classroom discussions. The FTCI may also
be administered in a classroom after formal instruction, similar to
how the data were collected in the current study.
Colleagues interested in obtaining a copy of the FTCI
(including the color handout of the flame tests) for classroom
use or additional research should contact the corresponding
author.
■ AUTHOR INFORMATION
Corresponding Author
*E-mail: bretzsl@miamioh.edu.
ORCID
Stacey Lowery Bretz: 0000-0001-5503-8987
Notes
The authors declare no competing financial interest.
■
ACKNOWLEDGMENTS
This material is based upon work supported by the National
Science Foundation under Grant 0733642. Any opinions,
findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily
reflect the views of the National Science Foundation. We thank
the students and instructors who made this study possible.
■
REFERENCES
(1) Treagust, D. F. Diagnostic Assessment in Science as a Means to
Improving Teaching, Learning and Retention. In Proceedings of the
25
Article
Assessment in Science Teaching and Learning Symposium; The University
of Sydney; UniServe Science: Sydney, Australia, 2006; pp 1−9.
(2) Libarkin, J. C. Concept Inventories in Higher Education Science.
Manuscript prepared for the National Research Council Promising
Practices in Undergraduate STEM Education Workshop 2, Wash-
ington, DC, Oct 13−14, 2008. http://sites.nationalacademies.org/cs/
groups/dbassesite/documents/webpage/dbasse_072624.pdf (accessed
Oct 2017).
(3) Adams, W. K.; Wieman, C. E. Development and Validation of
Instruments To Measure Learning of Expert-Like Thinking. Int. J. Sci.
Educ. 2011, 33 (9), 1289−1312.
(4) Krause, S.; Birk, J.; Bauer, R.; Jenkins, B.; Pavelich, M. J.
Development, Testing, and Application of a Chemistry Concept
Inventory. Paper presented at the 34th ASEE/IEEE Frontiers in
Education Conference, Savannah, GA, Oct 20−23, 2004. Institute of
Electrical and Electronics Engineers; Piscataway, NJ, 2004. DOI:
10.1109/FIE.2004.1408473. http://ieeexplore.ieee.org/stamp/stamp.
jsp?arnumber=1408473 (accessed Oct 2017).
(5) Treagust, D. F. Development and Use of Diagnostic Tests To
Evaluate Students’ Misconceptions in Science. Int. J. Sci. Educ. 1988,
10, 159−169.
(6) Bretz, S. L. Designing Assessment Tools To Measure Students’
Conceptual Knowledge of Chemistry. In Tools of Chemistry Education
Research; Bunce, D. M., Cole, R. S., Eds.; American Chemical Society:
Washington, DC, 2014; pp 155−168.
(7) National Research Council. Knowing What Students Know: The
Science and Design of Educational Assessment; The National Academies
Press: Washington, DC, 2001; pp 1−14. DOI: 10.17226/10019.
https://www.nap.edu/catalog/10019/knowing-what-students-know-
the-science-and-design-of-educational (accessed Oct 2017).
(8) Sadler, P. M. Psychometric Models of Student Conceptions in
Science: Reconciling Qualitative Studies and Distractor-Driven
Assessment Instruments. J. Res. Sci. Teach. 1998, 35 (3), 265−296.
(9) Mulford, D.; Robinson, W. An Inventory for Alternate
Conceptions among First-Semester General Chemistry Students. J.
Chem. Educ. 2002, 79 (6), 739−744.
(10) Othman, J.; Treagust, D. F.; Chandrasegaran, A. L. An
Investigation into the Relationship between Students’ Conceptions
of the Particulate Nature of Matter and Their Understanding of
Chemical Bonding. Int. J. Sci. Educ. 2008, 30 (11), 1531−1550.
(11) Nyachwaya, J. M.; Mohamed, A.-R.; Roehrig, G. H.; Wood, N.
B.; Kern, A. L.; Schneider, J. L. The Development of an Open−Ended
Drawing Tool: An Alternative Diagnostic Tool for Assessing Students’
Understanding of the Particulate Nature of Matter. Chem. Educ. Res.
Pract. 2011, 12, 121−132.
(12) Luxford, C. J.; Bretz, S. L. Development of the Bonding
Representations Inventory To Identify Student Misconceptions about
Covalent and Ionic Bonding Representations. J. Chem. Educ. 2014, 91
(3), 312−320.
(13) Peterson, R. F.; Treagust, D. F.; Garnett, P. Identification of
Secondary Students’ Misconceptions of Covalent Bonding and
Structure Concepts Using a Diagnostic Instrument. Res. Sci. Educ.
1986, 16, 40−48.
(14) Artdej, R.; Ratanaroutai, T.; Coll, R. K.; Thongpanchang, T.
Thai Grade 11 Students’ Alternative Conceptions for Acid−Base
Chemistry. Res. Sci. Technol. Educ. 2010, 28 (2), 167−183.
(15) Chandrasegaran, A. L.; Treagust, D. F.; Mocerino, M. The
Development of a Two-Tier Multiple-Choice Diagnostic Instrument
for Evaluating Secondary School Students’ Ability To Describe and
Explain Chemical Reactions Using Multiple Levels of Representation.
Chem. Educ. Res. Pract. 2007, 8, 293−307.
(16) Linke, R. D.; Venz, M. I. Misconceptions in Physical Science
among Non-Science Background Students: II. Res. Sci. Educ. 1979, 9,
103−109.
(17) Wren, D.; Barbera, J. Gathering Evidence for Validity during the
Design, Development, and Qualitative Evaluation of Thermochemistry
Concept Inventory Items. J. Chem. Educ. 2013, 90 (12), 1590−1601.
(18) Banerjee, A. C. Misconceptions of Students and Teachers in
Chemical Equilibrium. Int. J. Sci. Educ. 1991, 13 (4), 487−494.
DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education
(19) Voska, K. W.; Heikkinen, H. W. Identification and Analysis of
Student Conceptions Used To Solve Chemical Equilibrium Problems.
J. Res. Sci. Teach. 2000, 37 (2), 160−176.
(20) Tan, K. C. D.; Goh, N. K.; Chia, L. S.; Treagust, D. F.
Development and Application of a Two-Tier Multiple-Choice
Diagnostic Instrument To Assess High School Students’ Under-
standing of Inorganic Chemistry Qualitative Analysis. J. Res. Sci. Teach.
2002, 39 (4), 283−301.
(21) Tan, K.-C. D.; Taber, K. S.; Goh, N.-K.; Chia, L.-S. The
Ionization Energy Diagnostic Instrument: A Two-Tier Multiple-
Choice Instrument To Determine High School Students’ Under-
standing of Ionization Energy. Chem. Educ. Res. Pract. 2005, 6, 180−
197.
(22) McClary, L. M.; Bretz, S. L. Development and Assessment of a
Diagnostic Tool To Identify Organic Chemistry Students’ Alternative
Conceptions Related to Acid Strength. Int. J. Sci. Educ. 2012, 34 (15),
2317−2341.
(23) Adadan, E.; Savasci, F. An Analysis of 16−17-Year-Old Students’
Understanding of Solution Chemistry Concepts Using a Two-Tier
Diagnostic Instrument. Int. J. Sci. Educ. 2012, 34 (4), 513−544.
(24) Linke, R. D.; Venz, M. I. Misconceptions in Physical Science
among Non−Science Background Students. Res. Sci. Educ. 1978, 8,
183−193.
(25) Dick-Perez, M.; Luxford, C. J.; Windus, T. L.; Holme, T. A. A
Quantum Chemistry Concept Inventory for Physical Chemistry
Classes. J. Chem. Educ. 2016, 93 (4), 605−612.
(26) Brandriet, A. R.; Bretz, S. L. The Development of the Redox
Concept Inventory as a Measure of Students’ Symbolic and Particulate
Redox Understandings and Confidence. J. Chem. Educ. 2014, 91 (8),
1132−1144.
(27) Villafañe, S.; Heyen, B. J.; Lewis, J. E.; Loertscher, J.;
Minderhout, V.; Murray, T. A. Design and Testing of an Assessment
Instrument to Measure Understanding of Protein Structure and
Enzyme Inhibition in a New Context. Biochem. Mol. Biol. Educ. 2012,
44, 179−190.
(28) Bretz, S. L.; Linenberger, K. J. Development of the Enzyme−
Substrate Interactions Concept Inventory. Biochem. Mol. Biol. Educ.
2012, 40 (4), 229−233.
(29) Oser, J. I. Flame Tests. J. Chem. Educ. 1928, 5 (2), 192.
(30) Clark, A. R. The Test-Tube Method for Flame Testing. J. Chem.
Educ. 1935, 12 (5), 242−243.
(31) Clark, A. R. Test-Tube Flame Test Applied to the Rarer
Elements. J. Chem. Educ. 1936, 13 (8), 383−384.
(32) Kiplinger, C. C. Paper for Paltinum in Flame Tests. J. Chem.
Educ. 1941, 18 (6), 297.
(33) Anderson, H.; Corwin, J. F. A Simple Method of Demonstrating
Flame Tests. J. Chem. Educ. 1947, 24 (9), 443.
(34) Brown, J. A. Lacquer Color Filters for Qualitative Flame Tests. J.
Chem. Educ. 1953, 30 (7), 363−364.
(35) Strong, III F.C. Improving Potassium Flame Tests. J. Chem.
Educ. 1969, 46 (3), 178.
(36) Smith, D. D. Producing Flame Spectra. J. Chem. Educ. 1979, 56
(1), 48.
(37) Pearson, R. S. An Improved Calcium Flame Test. J. Chem. Educ.
1985, 62 (7), 622.
(38) Bouher, J. H. Capillary Tube Flame Test. J. Chem. Educ. 1986,
63 (2), 158.
(39) Ager, D. J.; East, M. B.; Miller, R. A. Vivid Flame Tests. J. Chem.
Educ. 1988, 65 (6), 545−546.
(40) Gouge, E. M. A. Flame Test Demonstration Device. J. Chem.
Educ. 1988, 65 (6), 544−545.
(41) Peyser, J. R.; Luoma, J. R. Flame Colors Demonstration. J.
Chem. Educ. 1988, 65 (5), 452−453.
(42) Mattson, B. M.; Snipp, R. L.; Michels, G. D. Spectacular
Classroom Demonstration of the Flame Test for Metal Ions. J. Chem.
Educ. 1990, 67 (9), 791.
(43) Barnes, Z. K. Alternative Flame Test Procedures. J. Chem. Educ.
1991, 68 (3), 246.
26
Article
(44) Ragsdale, R. O.; Driscoll, J. A. Rediscovering the Wheel: The
Flame Test Revisited. J. Chem. Educ. 1992, 69 (10), 828−829.
(45) Thomas, N. C.; Brown, R. A Spectacular Demonstration of
Flame Tests. J. Chem. Educ. 1992, 69 (4), 326−327.
(46) McRae, R. A.; Jones, R. F. An Inexpensive Flame Test
Technique. J. Chem. Educ. 1994, 71 (1), 68.
(47) Li, J.; Peng, A.-Z. Multiple Burning Heaps Of Color − An
Elegant Variation of a Flame Test. J. Chem. Educ. 1995, 72 (9), 828.
(48) Dalby, D. K. Bigger and Brighter Flame Tests. J. Chem. Educ.
1996, 73 (1), 80−81.
(49) Bare, W. D.; Bradley, T.; Pulliam, E. An Improved Method for
Students’ Flame Tests in Qualitative Analysis. J. Chem. Educ. 1998, 75
(4), 459.
(50) McKelvy, G. M. Flame Tests That Are Portable, Storable, and
Easy To Use. J. Chem. Educ. 1998, 75 (1), 55.
(51) Dragojlovic, V.; Jones, R. F. Flame Tests Using Improvised
Alcohol Burners. J. Chem. Educ. 1999, 76 (7), 929−930.
(52) Johnson, K. A.; Schreiner, R. A Dramatic Flame Test
Demonstration,. J. Chem. Educ. 2001, 78 (5), 640−641.
(53) Sanger, M. J. Flame tests: Which Ion Causes the Color? J. Chem.
Educ. 2004, 81 (12), 1776A−1776B.
(54) Sanger, M. J.; Phelps, A. J.; Banks, C. Simple Flame Test
Techniques Using Cotton Swabs. J. Chem. Educ. 2004, 81 (7), 969−
970.
(55) Mortier, T.; Wellens, A.; Janssens, M.−J. Inexpensive Alcohol
Burners for Flame Tests Using Aluminum Tea Light Candle Holders.
J. Chem. Educ. 2008, 85 (4), 522.
(56) Vitz, E. Demonstration Extensions: Flame Tests and
Electrolysis. J. Chem. Educ. 2008, 85 (4), 522.
(57) Landis, A. M.; Davies, M. I.; Landis, L.; Thomas, N. C. Magic
Eraser” Flame Tests. J. Chem. Educ. 2009, 86 (5), 577−578.
(58) Maines, L. L.; Bruch, M. D. Identification of Unkown Chlroide
Salts Using a Comination of Qualitative Analysis and Titration with
Silver Nitrate: A General Chemistry Laboratory,. J. Chem. Educ. 2012,
89 (7), 933−935.
(59) Neel, B.; Crespo, G. A.; Perret, D.; Cherubini, T.; Bakker, E.
Camping Burner-Based Flame Emission Spectrometer for Classroom
Demonstrations. J. Chem. Educ. 2014, 91 (1), 1655−1660.
(60) Yu, H. L. L.; Domingo, P. N., Jr.; Yanza, E. R. S.; Guidote, A. M.,
Jr. Making a Low-Cost Soda Can Ethanol Burner for Out-of-
Laboratory Flame Test Demonstrations and Experiments. J. Chem.
Educ. 2015, 92 (1), 127−128.
(61) Murphy, K.; Holme, T.; Zenisky, A.; Caruthers, H.; Knaus, K.
Building the ACS Exams Anchoring Concept Content Map for
Undergraduate Chemistry. J. Chem. Educ. 2012, 89 (6), 715−720.
(62) Holme, T.; Murphy, K. The ACS Exams Institute Under-
graduate Chemistry Anchoring Concepts Content Map I: General
Chemistry. J. Chem. Educ. 2012, 89 (6), 721−723.
(63) Holme, T. A.; Luxford, C. J.; Murphy, K. L. Updating the
General Chemistry Anchoring Concepts Content Map. J. Chem. Educ.
2015, 92, 1115−1116.
(64) Price, D. S.; Brooks, D. W. Extensiveness and Perceptions of
Lecture Demonstrations in the High School Chemistry Classroom.
Chem. Educ. Res. Pract. 2012, 13, 420−427.
(65) Körhasan, J. D.; Wang, L. Students’ Mental Models of Atomic
Spectra. Chem. Educ. Res. Pract. 2016, 17, 743−755.
(66) Ausubel, D. Educational Psychology: A Cognitive View; Holt,
Rinehart and Winston: New York, 1968.
(67) Novak, J. D. Human Constructivism: A Unification of
Psychological and Epistemological Phenomena in Meaning Making.
Int. J. Pers. Constr. Psych. 1993, 6 (2), 167−193.
(68) Ye, L.; Lewis, S. E. Looking for Links: Examining Student
Responses in Creative Exercises for Evidence of Linking Chemistry
Concepts. Chem. Educ. Res. Pract. 2014, 15, 576−586.
(69) Bretz, S. L. Human Constructivism and Meaningful Learning. J.
Chem. Educ. 2001, 78 (8), 1107.
(70) Grove, N. P.; Bretz, S. L. A Continuum of Learning: From Rote
Memorization to Meaningful Learning in Organic Chemistry. Chem.
Educ. Res. Pract. 2012, 13, 201−208.
DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27
Journal of Chemical Education Article

(71) Johnstone, A. H. You Can’t Get There from Here. J. Chem.

Educ. 2010, 87 (1), 22−29.
(72) Johnstone, A. H. Why Is Science Difficult To Learn? Things Are
Seldom What They Seem. J. Cmptr. Assist. Lrng. 1991, 7 (2), 75−83.
(73) Towns, M. H. Mixed Methods Designs in Chemical Education
Research. In Nuts and Bolts of Chemical Education Research; Bunce, D.
M., Cole, R. S., Eds.; American Chemical Society: Washington, DC,
2008; pp 135−148.
(74) Bretz, S. L. Qualitative Research Designs in Chemistry
Education Research. In Nuts and Bolts of Chemical Education Research;
Bunce, D. M., Cole, R. S., Eds.; American Chemical Society:
Washington, DC, 2008; pp 79−96.
(75) Phelps, A. J. Qualitative Methodologies in Chemical Education
Research: Challenging Comfortable Paradigms. J. Chem. Educ. 1994,
71 (3), 191−194.
(76) Geertz, C. Thick Description: Toward an Interpretive Theory of
Culture. In The Interpretation of Cultures: Selected Essays; Geertz, C.,
Ed.; Basic Books: New York, 1973; pp 3−30.
(77) Linenberger, K. J.; Bretz, S. L. A Novel Technology To
Investigate Students’ Understanding of Enzyme Representations. J.
Coll. Sci. Teach. 2012, 42 (1), 45−49.
(78) Bowen, C. W. Think-Aloud Methods in Chemistry Education:
Understanding Student Thinking. J. Chem. Educ. 1994, 71 (3), 184−
190.
(79) Livescribe. http://www.livescribe.com/en-us/ (accessed Oct
2017).
(80) White, R.; Gunstone, R. F. Probing Understanding; Falmer:
London, 1992.
(81) QSR International. http://www.qsrinternational.com/product
(accessed Oct 2017).
(82) Creswell, J. W. Qualitative Inquiry & Research Design: Choosing
among Five Approaches; Sage Publications: Thousand Oaks, CA, 2007.
(83) Mayo, A. V. Atomic Emission Misconceptions As Investigated
through Student Interviews and Measured by the Flame Test Concept
Inventory. Doctoral dissertation. Miami University, Oxford, OH, 2012.
https://etd.ohiolink.edu/pg_10?0::NO:10:P10_ACCESSION_
NUM:miami1362754897 (accessed Oct 2017).
(84) SPSS. https://www.ibm.com/analytics/us/en/technology/spss/
(accessed Oct 2017).
(85) Leighton, J. P.; Heffernan, C.; Cor, M. K.; Gokiert, R. J.; Cui, Y.
An Experimental Test of Student Verbal Reports and Teacher
Evaluations as a Source of Validity Evidence for Test Development.
Appl. Measurement in Educ. 2011, 24 (4), 324−348.
(86) McKelvie, S. J. Does Memory Contaminate Test−Retest
Reliability? J. Gen. Psych. 1992, 119 (1), 59−72.
(87) Trochim, W. M. K. Measurement Validity Types. Research
Methods Knowledge Base. http://www.socialresearchmethods.net/kb/
measval.php (accessed Oct 2017).
(88) Lewis, S. E.; Lewis, J. E. The Same or Not the Same:
Equivalence as an Issue in Educational Research. J. Chem. Educ. 2005,
82 (9), 1408−1412.
(89) Razali, N. M.; Wah, Y. B. Power Comparisons of Shapiro-Wilk,
Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. J. Stat.
Model. and Analytics 2011, 2 (1), 21−33.
(90) Ding, L.; Beichner, R. Approaches to Data Analysis of Multiple-
Choice Questions. Phys. Rev. ST Phys. Educ. Res. 2009, 5 (2), 020101−
02010317.
(91) Cronbach, L. J. Coefficient Alpha and the Internal Structure of
Tests. Psychometrika 1951, 16 (3), 297−334.
(92) Bretz, S. L.; McClary, L. M. Students’ Understandings of Acid
Strength: How Meaningful Is Reliabilty When Measuring Alternative
Conceptions? J. Chem. Educ. 2015, 92 (2), 212−219.
(93) Streiner, D. L. Starting at the Beginning: An Introduction to
Coefficient Alpha and Internal Consistency. J. Pers. Assess. 2003, 80
(1), 99−103.
(94) Crocker, L. M.; Algina, J. Introduction to Classical and Modern
Test Theory; Holt, Rinehart, and Winston: New York, 1986.

27 DOI: 10.1021/acs.jchemed.7b00594
J. Chem. Educ. 2018, 95, 17−27