Connecting symbolic difficulties with failure in physics

Eugene T. Torigoe and Gary E. Gladding

Citation: Am. J. Phys. 79, 133 (2011); doi: 10.1119/1.3487941

View online: http://dx.doi.org/10.1119/1.3487941
View Table of Contents: http://ajp.aapt.org/resource/1/AJPIAS/v79/i1
Published by the American Association of Physics Teachers

Additional information on Am. J. Phys.

Journal Homepage: http://ajp.aapt.org/
Journal Information: http://ajp.aapt.org/about/about_the_journal
Top downloads: http://ajp.aapt.org/most_downloaded
Information for Authors: http://ajp.dickinson.edu/Contributors/contGenInfo.html

Downloaded 03 Jun 2013 to 216.47.136.20. Redistribution subject to AAPT license or copyright; see http://ajp.aapt.org/authors/copyright_permission
PHYSICS EDUCATION RESEARCH SECTION
The Physics Education Research Section 共PERS兲 publishes articles describing important results from
the field of physics education research. Manuscripts should be submitted using the web-based system that
can be accessed via the American Journal of Physics home page, http://www.kzoo.edu/ajp/, and will be
forwarded to the PERS editor for consideration.

Connecting symbolic difficulties with failure in physics

Eugene T. Torigoea兲
Department of Physics, Allegheny College, Meadville, Pennsylvania 16335
Gary E. Gladding
Department of Physics, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801
共Received 3 February 2010; accepted 19 August 2010兲
We find that symbolic physics questions are significantly more difficult than their analogous
numerical versions. Very few of the errors are due to manipulation errors of the symbolic equations.
Instead, most errors are due to confusions of symbolic meaning. We also find that performance on
symbolic questions is more highly correlated with the overall performance than performance on
numeric questions. We devised a coding scheme that distinguishes questions based only on the
mathematical structure of the solutions. The coding scheme can be used to identify both difficult and
discriminating physics questions. The questions identified by this coding scheme require an
algebraic representation and discourage problem solving strategies that do not require an
understanding of symbolic equations. Our results suggest that an inability to interpret physics
equations may be a major contributor to student failure in introductory physics. © 2011 American
Association of Physics Teachers.
关DOI: 10.1119/1.3487941兴

I. INTRODUCTION much of this research is on the facilitation of the transition

Our work is motivated by the assumption that poor math-
ematical preparation is a major reason why many students
fail introductory physics. Physics instructors often encounter
students with mathematical difficulties, which hinder their
ability to be successful. In our earlier work, we quantified
mathematical difficulties in introductory physics by compar-
ing the differences in the score on numeric and symbolic
versions of the same question.1,2 We gave a question in
which the numeric version had an average of 95%, and the
otherwise equivalent symbolic version had an average of
45%. Although some errors originated from incorrect alge-
braic manipulations, the great majority of the errors were
related to the confusion of the meaning of the symbols.
We calculated the ratio of the symbolic score to the nu-
meric score for the two numeric-symbolic pairs of questions
we studied 共see Fig. 1兲. A comparison of subgroups of the
class based on the overall course performance showed that
the top 1/4 of students showed little difference in perfor-
mance between the versions, but there was a large difference
in performance for the bottom 1/4 students. Of the latter
group of students who could correctly solve the numeric ver-
sion 共80% of the group兲, the ratio for these questions sug-
gests that only 40% would also be able to solve the symbolic
version.
We believe that the mathematical difficulties underlying
the differential performance on numeric and symbolic ques-
tions reflect students’ algebraic difficulties. As a result, edu-
cation research concerning the teaching and learning of alge-
bra is relevant to the teaching of physics. The emphasis of
between the arithmetic and the algebraic stages of develop-
ment. The difficulties that students have making the transi-
tion from the arithmetic stage to the algebraic stage fore-
shadow the symbolic difficulties that we have observed in
college students. Although the arithmetic stage emphasizes
the calculation of numeric results, the algebraic stage empha-
sizes the setup of symbolic equations.
When students make the transition from arithmetic to al-
gebra, they are often confused by algebraic expressions be-
cause the two subjects are similar enough that students can
recognize the symbols used, but sufficiently different that
students have difficulty understanding the meaning of the
expressions that use those symbols. Kieran3,4 noted that in
arithmetic, mathematical expressions contain procedural in-
formation about how to proceed to the solution. For example,
3 + 5 = ?, contains the symbol + to denote that we must use
the addition to find the solution, and the equal sign, which is
used to assert the solution to the expression. In contrast, al-
gebraic expressions express relations and do not necessarily
yield information about the process to solve for an unknown
in the equation.
Expressions in arithmetic are thought of only as a process
to a numerical solution. In algebra the expressions must be
understood both in terms of the process to solve for a par-
ticular quantity and as an expression of a relation between
quantities. For example, from the algebraic perspective, the
expression a + b is thought of as a process of combining the
values of a and b and also as an object that represents the
sum of a and b.
The transition is especially troublesome for students be-

133 Am. J. Phys. 79 共1兲, January 2011 http://aapt.org/ajp © 2011 American Association of Physics Teachers 133

Downloaded 03 Jun 2013 to 216.47.136.20. Redistribution subject to AAPT license or copyright; see http://ajp.aapt.org/authors/copyright_permission

Fig. 1. Ratio of the symbolic version score to the numeric version score for
different subgroups of the class averaged over two questions analyzed in
Ref. 1. We interpret this ratio to represent the likelihood that the students
who could solve the numeric version correctly would also solve the sym-
bolic version correctly.
cause procedures that successfully solve questions in arith-
metic often fail for questions in algebra. Filloy and Rojano5
showed that although solutions of the form Ax + B = C can be
solved using arithmetic methods, they cannot be used to
solve equations of the form Ax + B = Cx + D. The following
question can be solved with a series of simple calculations,
which makes symbolic representation unnecessary.
Daniel went to visit his grandmother, who gave
him $1.50. Then he bought a book costing $3.20. If
he has $2.30 left, then how much money did he
have before visiting his grandmother?6
If the question is designed to use an equation Ax + B = Cx
+ D, then students must first represent the relation math-
ematically before calculating a number. In the following
question, if the student is unable to reconceptualize the ques-
tion, they must use an algebraic representation.
The Westmont Video shop offers two rental plans.
Then first plan costs $22.50 per year plus $2.00 per
video rented. The second plan offers a free mem-
bership for one year but charges $3.25 per video
rented. For what number of rental videos per year
will these two plans cost exactly the same?6
Similar results have also been observed by Larkin et al.7
while studying novice and expert problem solving proce-
dures in physics. They found that two equation and two un-
known simultaneous-equation questions where both un-
knowns appear in both equations could not be solved by
novice students. The common backward working strategy
employed by novice students was not effective during the
interviews. Similar to algebraic solutions with the form Ax
+ B = Cx + D, simultaneous-equation questions require stu-
dents to setup the algebraic relations before they can calcu-
late a number.
Clement and colleagues8 found that many introductory
physics students have difficulty mathematically representing
relations between quantities. They studied variations of what
is known as the “Students and Professors” problem.
134 Am. J. Phys., Vol. 79, No. 1, January 2011
Downloaded 03 Jun 2013 to 216.47.136.20. Redistribution subject to AAPT license or copyright; see http://ajp.aapt.org/authors/copyright_permission
Write an equation using the variables S and P to
represent the following statement: “There are six
times as many students as professors at this univer-
sity.” Use S for the number of students and P for
the number of professors.
Only 63% of first-year engineering students were able to
correctly answer this question.8 A more recent study involv-
ing introductory physics students found a similar
percentage.9 The most common error was a reversal of the
correct quantitative relation 共6S = P instead of S = 6P兲. Al-
though it might first appear that this error was due to care-
lessness, the authors found that for many students this error
was due to an alternative conception of the meaning of alge-
braic equations.8 During interviews many students who made
this reversal error knew which group was larger, but believed
that the larger number should be placed next to the letter
representing the larger group. In an equation such as 6S = P,
the symbol S is used to signify a single student rather than
the number of students. This use is similar to the use of
labels in a unit conversion such as 100 cm= 1 m, in which
we consider cm to mean a single centimeter rather than the
number of centimeters.
In a related study, Soloway et al.10 showed that when the
question was asked in the context of a computer program, the
likelihood of success was significantly higher 共87% correct兲
than when asked in the context of an algebraic expression.
The authors demonstrated that although many students could
successfully express the relation in the context of a numeric
computation, fewer could correctly produce the relation in
the context of an algebraic representation.
These results suggest that there are many students in in-
troductory physics with major difficulties with algebraic re-
lations. In this paper, we explore the numeric and symbolic
distinction with a greater number of question pairs than in
our previous work1 and determine the correlations between
performance on questions that require symbolic representa-
tion and overall success in introductory physics.
II. EXPANDED STUDY OF QUESTIONS
IN MECHANICS
In our previous study1 we investigated numeric and sym-
bolic versions of two kinematics questions. These questions
both had high numeric version scores and involved one-
dimensional kinematics. To understand different question
properties, we expanded our study of numeric and symbolic
questions to include ten pairs of questions that span many
different introductory mechanics topics and a variety of dif-
ficulty levels.
The 765 students in the study were enrolled in the
calculus-based introductory mechanics course, Physics 211,
at the University of Illinois at Urbana-Champaign in Spring
2007. Students in Physics 211 take three midterm exams and
a multiple-choice cumulative final.11 The students completed
one of the two randomly administered versions of the final
exam. Each version of the final contained the numeric or the
symbolic version of each of the ten questions.
To ensure that any differences between the versions were
not due to systematic differences between the groups, we
compared the average midterm exam score of the students
for each of the final exam versions. The average midterm
E. T. Torigoe and G. E. Gladding 134
Table I. Average and standard error for numeric and symbolic versions of each question in the study. The p-values were calculated using a two-tailed t-test.
The p-value represents the likelihood that such a difference could be observed under the assumption that the null hypothesis is true.

Question 1 Question 2 Question 3 Question 4 Question 5 Question 6 Question 7 Question 8 Question 9 Question 10

Numeric 91.5⫾ 1.4 93.3⫾ 1.3 79.6⫾ 2.1 90.5⫾ 1.5 44.9⫾ 2.5 61.2⫾ 2.4 76.0⫾ 2.1 33.2⫾ 2.3 78.6⫾ 2.0 48.8⫾ 2.3
Symbolic 70.4⫾ 2.3 56.8⫾ 2.5 63.4⫾ 2.4 82.3⫾ 1.9 31.9⫾ 2.3 52.9⫾ 2.5 75.6⫾ 2.1 29.8⫾ 2.2 54.5⫾ 2.5 52.8⫾ 2.4
Difference 21.1⫾ 2.7 36.5⫾ 2.8 16.2⫾ 3.1 8.1⫾ 2.4 13.0⫾ 3.4 8.3⫾ 3.4 0.4⫾ 2.9 3.4⫾ 3.1 24.1⫾ 3.2 −3.9⫾ 3.3
p-value ⬍0.001 ⬍0.001 ⬍0.001 ⬍0.001 ⬍0.001 0.01⬍ p ⬍ 0.05 0.5⬍ p 0.2⬍ p ⬍ 0.4 ⬍0.001 0.2⬍ p ⬍ 0.4

score and standard error for final 1 was 78.9⫾ 0.5 and for
final 2 was 78.6⫾ 0.5, and we conclude that the two groups
were equivalent.
A sample of some question pairs used in the study is found
in Appendix A.12 All but one of the ten paired questions
contained analogous choices for each the numeric and sym-
bolic versions of the question.13 To discourage cheating,
many of the choices were in a different order in the two
versions.
A. Results
Table I shows the score, the standard error, the difference,
and the p-value of the difference for the numeric and sym-
bolic versions of each question.14 Although there exist large
differences between numeric and symbolic scores for some
questions, which is consistent with our earlier findings, other
questions show very little difference in score between ver-
sions. In the following, we describe properties of the ques-
tions that influenced the difference in score between ver-
sions.
An analysis of each of the ten questions, including the
popularity of the correct choices and incorrect choices, and
an analysis of students’ written work were conducted to un-
derstand why some questions showed differences between
the versions, while others did not. Table II shows the differ-
ence in scores between versions and the associated question
properties. Many of the question properties we identified re-
late to the meaningful use of symbolic equations. The fol-
lowing is a list of question properties with explanations.
Table II. Questions ranked by the difference in the score between numeric
and symbolic versions of each question with the associated question prop-
erties. The abbreviations are MultEq 共multiple equations in the solution兲,
MoGE 共manipulation of general equations兲, CompExp 共use of a compound
expression兲, SingEq 共single equation in solution兲, SimulEq 共simultaneous
equations兲, ManipErr 共manipulation error兲, Diff 共difficult question兲.
1. Physics difficulty
The score for the top 1/4 on the numeric version was used
as a measure of each question’s physics difficulty. Questions
5 and 8 were significantly more difficult for the top 1/4 stu-
dents than the other questions, and the difficulty of the phys-
ics content overwhelmingly dominated the mathematical dif-
ficulty of the questions. Consequently, we removed them
from further analysis in this study. This removal is supported
by the fact that the average scores by the bottom 1/4 for the
numeric versions of these questions was 23%, consistent
with the random guessing rate of 20%.
2. Multiple equations
This property distinguishes whether the problem is com-
monly solved with one equation or with multiple equations.
The symbolic questions tend to be more difficult because
symbol confusion often occurs when equations are com-
bined. The presence of multiple equations allows for the pos-
sibility of confusion of the same type of quantity 共for ex-
ample, confusing two different velocities兲. Questions 1–3, 6,
and 9 require multiple equations for their solution. Questions
4 and 7 are the only questions in which the solution can be
found with a single equation.
3. Simultaneous equations
The solution is simultaneous if, for example, there are two
equations and two unknowns and both unknowns are present
in both equations. In contrast, a sequential set of equations
has an equation containing only a single unknown. Although
numeric values cannot be obtained until the equations are
combined in the first case, a numeric value can be obtained
immediately from one of the sequential equations. Question
10 is the only question in our sample that requires simulta-
neous equations for the solution. The numeric and symbolic
versions were equally difficult because students are forced to
use the same procedure to solve both the numeric and the
symbolic versions.

Difference Question Question properties 4. General equation manipulation

36.5⫾ 2.8 2 MultEq, MoGE, CompExp This property signifies whether it is possible to obtain one
24.1⫾ 3.2 9 MultEq, MoGE, CompExp of the incorrect choices by combining general equations or
21.1⫾ 2.7 1 MultEq, MoGE by manipulating a single general equation with minimal
16.2⫾ 3.1 3 MultEq, MoGE changes 共for example, replacing x by d兲. This property en-
13.0⫾ 3.4 5 Diff hances the difference in score between the numeric and the
8.3⫾ 3.4 6 MultEq, CompExp symbolic versions because it penalizes students who do not
8.1⫾ 2.4 4 SingEq, ManipErr meaningfully use the equations.
3.4⫾ 3.1 8 Diff Questions 1–3 and 9 contain incorrect choices correspond-
0.4⫾ 2.9 7 SingEq ing to the blind manipulation of the general equations.
−3.9⫾ 3.3 10 SimulEq This error was the most popular incorrect choice for
Questions 1–3.

135 Am. J. Phys., Vol. 79, No. 1, January 2011 E. T. Torigoe and G. E. Gladding 135

Downloaded 03 Jun 2013 to 216.47.136.20. Redistribution subject to AAPT license or copyright; see http://ajp.aapt.org/authors/copyright_permission

Fig. 2. Ratio of the symbolic version score to the numeric version score for
different subgroups of the class for all ten questions.
5. Use of a compound expression
This property signifies that to reach the correct symbolic
solution students must replace a variable in a general equa-
tion with a compound expression 共for example, replacing the
variable v by a more specific compound expression v / 2兲.
Questions 2, 6, and 9 require that students specify a vari-
able as a compound expression. For question 2, the second
most popular incorrect choice corresponded to using v in-
stead of v / 2 or t instead of t / 2. For question 6, the most
popular incorrect choice corresponded to a failure to specify
the variable m as 2M. Question 9 required that students
specify both the variable m as 3m and the variable v as v / 3.
6. Manipulation error
This property signifies that a common error on this ques-
tion is related to an incorrect manipulation of a symbolic
equation. The mean score for the symbolic version of ques-
tion 4 was 8.2% higher than the numeric version because of
this effect. An analysis of student work found that the differ-
ence in score was in large part due to students who setup the
energy conservation equation correctly, but who choose an
option corresponding to an incorrect manipulation of that
equation.
B. Connection to failure in physics
To study the relation between the specific questions and
the overall course performance, we divided the class into
three subgroups based on the total course points. The groups
are the bottom 1/4, the middle half, and the top 1/4. For each
group and for each question in the study, we calculated the
ratio of the average score on the symbolic version to the
average score on the numeric version 共see Fig. 2兲. This ratio
represents the fraction of the students who solved the nu-
meric version correctly and who would also be able to solve
the symbolic version. Even though no student was given
both versions of a single question, this interpretation is jus-
tified because of the equivalence of the midterm exam aver-
ages of the students given each version of the final exam 共see
Sec. II兲.
In Sec. I we showed that an analysis of data from our
previous study1 revealed that the lowest ratio of symbolic to
the numeric scores was observed for the bottom 1/4 of the
class. Our present results are consistent with that result. For
almost all questions, the smallest ratio was observed for the
bottom 1/4 of the class. In contrast, the top 1/4 rarely showed
136 Am. J. Phys., Vol. 79, No. 1, January 2011
Downloaded 03 Jun 2013 to 216.47.136.20. Redistribution subject to AAPT license or copyright; see http://ajp.aapt.org/authors/copyright_permission
any distinction between the numeric and symbolic versions.
For the top 1/4 only, one question had a ratio lower than
0.93.
A two-tailed t-test was performed to compare the score for
the bottom 1/4 on the numeric and symbolic versions of each
question. Questions 4, 7, 8, and 10 were the only questions
for which p ⬎ 0.05, and thus the hypothesis that the perfor-
mance by the bottom 1/4 on the numeric and symbolic ver-
sions of these questions were equivalent remains tenable.15
The high ratio for questions 8 and 10 is due to an equally
poor performance on both versions of those questions. Ques-
tion 4 and 7 were the only questions that could be solved
with single equations, and the error for question 4 was re-
lated to a manipulation error. The lack of a correlation indi-
cates that those types of question properties do not correlate
well with the overall success in the course. The low ratios for
the bottom 1/4 for the remaining questions, which contain
properties that stress meaningful representation, suggests that
the bottom 1/4’s algebraic difficulties may be related to their
poor performance in physics.
III. CODING QUESTIONS BY MATHEMATICAL
STRUCTURE
Many of the question properties described in Sec. II A
were discovered after the research was performed and were
not independently validated. In this study, we attempt to vali-
date those question properties by analyzing an entire semes-
ter of exam questions. Specifically, we are interested in de-
termining if question properties that stress meaningful
algebraic representation can be used to identify discriminat-
ing exam questions.
We examined the set of exam questions administered in
Spring 2006 in Physics 211 and categorized them by the
mathematical properties of their solutions. The exam ques-
tions were made by faculty members with no knowledge of
the question properties we intended to study. There were 169
unique16 multiple-choice questions administered 共three mid-
terms and two versions of the final exam兲 and 870 students
who completed the course without any missing exam grades.
We analyzed the mathematical structure of the solutions to
all of the questions in our sample and coded for the follow-
ing types of questions: 共1兲 Multiple-equation symbolic ques-
tions; 共2兲 simultaneous-equation numeric questions; and 共3兲
single-equation numeric questions where the target unknown
appears on opposite sides of the equal sign.
This last property is equivalent to the form Ax + B = Cx
+ D, which is a structure that requires students to use alge-
braic methods. An example of a question with a solution of
this form can be found in Appendix B. We referred to ques-
tions with any of these three properties as equation priority
questions. As the name suggests, the solutions to these ques-
tions emphasize the meaningful representation of symbolic
equations. The three question properties require students to
formally represent the equations and prevent simple one
equation at a time numeric sequential solutions. Specifically
omitted were single-equation and sequential-equation nu-
meric questions, both of which were shown in Sec. II B to be
poorly correlated with the overall performance in the course.
Two additional properties identified in Sec. II A were not
included in this coding scheme: General equation manipula-
tion and the use of compound expressions. Even though we
have some evidence that these properties tend to make sym-
bolic problems more difficult, such a distinction is not im-
E. T. Torigoe and G. E. Gladding 136
Table III. Comparison of equation priority and other questions. The error on
the mean difference was calculated using the distribution of question score
differences between groups 共Ref. 18兲.
Equation priority questions
Number 40
Bot. 1/4 score 33.9% ⫾ 2.5%
Top 3/4 score 62.0% ⫾ 2.9%
Mean difference 28.1% ⫾ 1.7%
Pearson r 0.38
portant for this analysis. In this coding we were not inter-
ested in distinguishing one type of symbolic question from
another, but questions whose mathematical structure require
the formal setup of equations from those that do not.
A. Results
After analyzing the solutions to all of the exam questions,
40 out of 169 共24%兲 were coded as equation priority ques-
tions. One author and another experienced physics instructor
agreed on 92% of the question categorizations. Disagree-
ments were due to differences in judgment of what consti-
tuted a single-equation solution. The two raters agreed on the
coding of all of the questions after a discussion of what the
average student would consider to be a single equation. Al-
though most of the solutions of the other, non-equation pri-
ority questions required equations, they did not satisfy our
coding scheme.
To study the ability of these questions to differentiate stu-
dents who failed or were on the verge of failing from the rest
of the class, we compared the performance of the bottom 1/4
and the rest of the class for both equation priority and the
other questions. Table III shows that for both the bottom 1/4
and the rest of the class, the equation priority questions are
significantly more difficult than other questions. This finding
is consistent with our results in Sec. II A in which we found
that questions with these mathematical properties were more
difficult. Also consistent with our earlier findings is that sym-
bolic difficulties are the most pronounced for the bottom 1/4
of students; we find that the difference in score between the
bottom 1/4 and the rest of the class is larger for equation
priority questions than the other questions. The mean differ-
ence for equation priority questions was 28.1% ⫾ 1.7% com-
pared to 20.6% ⫾ 0.9% for other questions.
A common measure of discrimination is the Pearson coef-
ficient r.17 In this context r is a measure of the relation be-
tween the performance on individual questions and the over-
all performance in the course. We find an average value of
r = 0.38 for equation priority coded questions and r = 0.29 for
the other questions. Figure 3 shows a histogram of the values
for r for both sets of questions. Although 50% of equation
priority questions were highly correlated with course score
共r ⬎ 0.4兲, only 17.8% of the other questions were.
B. Analysis of systematic errors: The relation between
difficulty and discrimination
The higher discrimination of equation priority questions
might be due to a positive correlation between difficulty and
discrimination, that is, difficult questions tend to be more
137 Am. J. Phys., Vol. 79, No. 1, January 2011
Downloaded 03 Jun 2013 to 216.47.136.20. Redistribution subject to AAPT license or copyright; see http://ajp.aapt.org/authors/copyright_permission
Other questions
129
56.9% ⫾ 1.8%
77.5% ⫾ 1.5%
20.6% ⫾ 0.9%
0.29
Fig. 3. Histogram of values for the Pearson correlation coefficient r for
equation priority questions and the other questions.
discriminating. To address this concern, we performed an
analysis using three methods for controlling for the effect of
the question difficulty on the discrimination.
First we combined the results for the class to determine
the relation between difficulty and discrimination for the en-
tire class. The size of the bins for the question score was
15%, except the highest bin that spanned the range from 90%
to 100%. The size of the bins was chosen so that there were
an adequate number of questions from each category. We
calculated the fraction of questions that were highly dis-
criminating 共r ⬎ 0.4兲 in each bin for both types of questions.
Figure 4 shows that the fraction in each bin of highly dis-
criminating equation priority questions is equal to or greater
than that of the other questions. A chi-squared test of asso-
ciation was performed in each bin and it was found that the
fraction of questions that were highly discriminating was sig-
nificantly higher for equation priority questions in three bins:
30%–45%, 60%–75%, and 75%–90% 共p ⬍ 0.05兲.
The discrimination is also reflected in the fact that the
mean difference between the bottom 1/4 and the rest of the
class was greater for equation priority questions than for the
other questions. It is possible that this effect was due to the
fact that the equation priority questions were also more dif-
ficult, on average, than the other questions. To control the
Fig. 4. Equation priority and other questions were binned by their difficulty
for the entire class. A histogram was created by calculating the fraction of
highly discriminating questions 共r ⬎ 0.4兲 in each bin.
E. T. Torigoe and G. E. Gladding 137
Table IV. Comparison between the bottom 1/4 and the rest of the class on
groupings of questions based on fixing the score for the top 3/4 of the class.
Other difficult non-equation priority questions were found by omitting easy
questions until the other question score was the same as the equation priority
question score for the top 3/4 of the class. Other matched questions were
created by matching each equation priority question to other non-equation
priority questions by difficulty for the top 3/4 of the class.
Equation priority Other difficult Other matched
Number 40 57 40
Bot. 1/4 score 33.9% ⫾ 2.5% 40.7% ⫾ 1.9% 41.7% ⫾ 2.9%
Top 3/4 score 62.0% ⫾ 2.9% 61.9% ⫾ 1.8% 62.3% ⫾ 2.8%
Mean difference 28.1% ⫾ 1.7% 21.2% ⫾ 1.3% 20.6% ⫾ 1.6%
Pearson r 0.38 0.29 0.29
effect of question difficulty, we used two methods of omit-
ting the other questions so that the difficulty for the top 3/4
of the class on other questions was the same. This procedure
allowed us to compare the performance of the bottom 1/4 on
two sets of questions that were equally difficult for the top
3/4. The first method equalized the difficulty of both ques-
tions for the top 3/4 of the class by omitting the easiest other
non-equation priority questions. The second method
achieved the same result by matching the difficulty of each
equation priority question to another non-equation priority
question for the top 3/4 of the class. Table IV shows that a
gap in the mean difference remains almost unchanged after
these two corrections for question difficulty.
The discriminatory ability between equation priority and
other questions is unchanged even when different methods
are employed to correct for question difficulty. Hence, the
discriminatory ability of equation priority questions is not an
effect of the difficulty of the questions.
IV. DISCUSSION
Our studies show the importance of mathematical struc-
tures in the difficulty and discrimination of questions in in-
troductory physics. Questions whose structure force alge-
braic representation tend to be more difficult and more
discriminating than questions that can be solved by a series
of calculations. Even though these properties are not neces-
sarily related to the physics content of the questions, perfor-
mance on questions with these properties is correlated with
the overall performance in the course.
This relation may exist for a variety of reasons. One rea-
son might be the teaching methods employed in introductory
physics. Because symbolic algebra is vital to the understand-
ing of the expert physicists who teach introductory physics,
it is often a prominent component of instruction. At the Uni-
versity of Illinois symbolic derivations are often used to in-
troduce new concepts, and symbolic example questions are
often shown to demonstrate general problem solving proce-
dures. However, students with algebraic difficulties may be
easily confused by such methods.
The correlations might also reflect that the ability of stu-
dents to solve questions symbolically may increase the effec-
tiveness of practice problems by allowing for a more general
understanding of the solution. The ability to see the symbolic
structure of a problem might improve the probability that
students with this ability will be able to solve similarly struc-
tured questions on later exams.
138 Am. J. Phys., Vol. 79, No. 1, January 2011
Downloaded 03 Jun 2013 to 216.47.136.20. Redistribution subject to AAPT license or copyright; see http://ajp.aapt.org/authors/copyright_permission
Some research suggests that the structure of equations can
assist the acquisition of physical concepts. Schwartz et al.19
found that equations help children develop an understanding
of balance problems. They hypothesize that the structure of
the equations supports the precision of conceptual ideas, al-
leviates working memory load, and allows for the organiza-
tion of multiple parameters. Sloutsky et al.20 found that
learning and transfer can be facilitated when knowledge is
expressed in an abstract generic form. In introductory phys-
ics the ability to meaningfully construct and interpret sym-
bolic equations may be beneficial to learning in all aspects of
the course. Conversely, students who are unable to under-
stand symbolic equations may learn less than those that do.
The question properties we have identified penalize stu-
dents who attempt to solve problems without understanding
the meaning of the symbolic equations they use. Tuminaro21
observed an activity that he called “recursive plug-and-chug”
while studying students working on homework questions in
groups. In this activity students try to match variables to a
list of equations. If they find an equation where the target
quantity is the only unknown, they solve for the target quan-
tity; if it is not, they replay the game until an appropriate
equation is found. Although this procedure might result in
the correct answer for some questions, it does not require an
understanding of the underlying physics, and it is not very
effective for simultaneous-equation and symbolic questions.
Symbolic questions inhibit strategies like recursive plug-
and-chug by allowing for a greater amount of confusion of
meaning. Questions that contain incorrect options corre-
sponding to the manipulation of general equations and the
failure to specify variables as compound expressions penal-
ize students who use strategies that do not understand the
meaning of the symbols and symbolic expressions.
These confusions of meaning might be alleviated by the
use of subscripts on symbols, but few students use them.
Although seemingly trivial, subscripts allow experts to dis-
tinguish variables from specific quantities belonging to spe-
cific objects or individuals. A common error on symbolic
questions was the confusion of two quantities of the same
type. For example, combining two equations by inappropri-
ately equating object A’s velocity with object B’s velocity. If
students were in the practice of immediately writing sub-
scripts after choosing a general equation, they would be
forced to consider the meaning of the symbols that they were
using.
Conceptual exams like the force concept inventory 共FCI兲
have demonstrated that the ability to solve quantitative prob-
lems does not always indicate conceptual understanding.
Mazur22 demonstrated that high average scores for complex
quantitative questions can often be associated with low av-
erage scores on analogous conceptual questions with the
same population of students. We suggest that the question
properties we have identified can bridge the gap between
quantitative and conceptual questions. The question proper-
ties we have described can be used by instructors to produce
quantitative questions that emphasize meaningful symbolic
representation.
The use of these question properties may have the effect of
reorganizing the reward structure in our physics courses. Stu-
dents who were rewarded on exams and homework assign-
ments for using strategies that did not require an understand-
ing of the equations they used would not be rewarded with
the types of questions we have described.
Improving students’ ability to symbolically represent rela-
E. T. Torigoe and G. E. Gladding 138
 Fig. 5. Diagram for question 2.

tions may be an important factor in lowering student failure

in and retention through the introductory physics sequence.

ACKNOWLEDGMENTS
The authors would like to thank the reviewers for their
helpful comments. The authors would also like to thank the
members of the physics education research group at the Uni-
versity of Illinois at Urbana-Champaign. This material is Fig. 7. Diagram for question 10.
based upon work supported by NSF DUE 0088734 and NSF
DUE 0341261.

APPENDIX A: QUESTIONS IN THE PHYSICS 211

SPRING 2007 FINAL EXAM STUDY 共a兲 ␦x = 0.18 m 关4%, 8%, 4%, 1%兴
共b*兲 ␦x = 0.32 m 关79%, 63%, 78%, 94%兴
The following are a sample of the questions in the Physics
共c兲 ␦x = 0.45 m 关6%, 11%, 5%, 2%兴
211 Spring 2007 final exam study. The symbolic questions
共d兲 ␦x = 0.55 m 关11%, 14%, 13%, 4%兴
used the same wording but replaced the numbers with sym-
bols. An asterisk has been placed next to the correct option 共e兲 ␦x = 0.95 m 关1%, 4%, 0%, 0%兴
for each question. Each option is accompanied in brackets by Question 9 (symbolic). A block of mass M slides on a
the percentages of all students, the bottom 1/4, the middle frictionless surface with a velocity V. It strikes a second
half, and the top 1/4 that choose that option. The sum of the block of mass 2M that is at rest and is attached to a long,
percentages do not always add up to 100% due to rounding relaxed ideal 共massless兲 spring of spring constant k. Assume
errors and students who left the question blank. that the blocks stick together after colliding and the collision
Question 2 (numeric). A car can go from 0 to 60 m/s in 8 takes place very quickly. Provided that the spring is stiff
s. At what distance d from the start 共at rest兲 is the car trav- enough to stop the blocks before striking the wall, determine
eling 30 m/s? 关Assume a constant acceleration 共see Fig. 5兲.兴 ␦x, the maximum amount the spring is compressed from its
relaxed length 共see Fig. 6兲.
共a兲 30 m 关1%, 4%, 0%, 1%兴
共b*兲 60 m 关93%, 80%, 96%, 99%兴 共a兲 ␦x = 共3MV2 / k兲1/2 关14%, 33%, 12%, 0%兴
共c兲 120 m 关3%, 10%, 2%, 0%兴 共b兲 ␦x = 共MV2 / k兲1/2 关13%, 17%, 16%, 2%兴
共d兲 240 m 关2%, 4%, 2%, 0%兴 共c兲 ␦x = 共2MV2 / 共3k兲兲1/2 关5%, 9%, 6%, 0%兴
共e兲 480 m 关0%, 1%, 0%, 0%兴 共d*兲 ␦x = 共MV2 / 共3k兲兲1/2 关55%, 29%, 50%, 90%兴
Question 2 (symbolic). A car can go from 0 to v1 in t1
共e兲 ␦x = 共MV2 / 共9k兲兲1/2 关13%, 12%, 15%, 8%兴
seconds. At what distance d from the start 共at rest兲 is the car
traveling 共v1 / 2兲? 关Assume a constant acceleration 共see Fig. Question 10 (numeric). A uniform disk of mass M = 8 kg
5兲.兴 and radius R = 0.5 m has a string wound around its rim. The
disk is free to spin about a pin through the center of the disk.
共a兲 d = v1t1 关1%, 1%, 1%, 1%兴 A mass M = 8 kg 共same mass as the disk兲 is connected to the
共b兲 d = v1t1 / 2 关20%, 37%, 16%, 11%兴 string and is dropped from rest. What is the acceleration a of
共c兲 d = v1t1 / 4 关22%, 28%, 27%, 8%兴 the block? 共See Fig. 7.兲
共d*兲 d = v1t1 / 8 关57%, 35%, 56%, 81%兴
共e兲 d = v1t1 / 16 关0%, 0%, 0%, 0%兴 共a兲 a = 2.45 m / s2 关5%, 8%, 6%, 1%兴
Question 9 (numeric). A block of mass M = 1.5 kg slides 共b兲 a = 3.27 m / s2 关8%, 16%, 7%, 3%兴
on a frictionless surface with a velocity V = 4 m / s. It strikes 共c兲 a = 4.91 m / s2 关30%, 37%, 36%, 11%兴
a second block of mass 2M = 3.0 kg that is at rest and is 共d*兲 a = 6.54 m / s2 关49%, 28%, 43%, 82%兴
attached to a long, relaxed ideal 共massless兲 spring of spring 共e兲 a = 7.36 m / s2 关7%, 11%, 7%, 4%兴
constant k = 80 N / m. Assume that the blocks stick together Question 10 (symbolic). A uniform disk of mass M and
after colliding and the collision takes place very quickly. radius R has a string wound around its rim. The disk is free
Provided that the spring is stiff enough to stop the blocks to spin about a pin through the center of the disk. A mass M
before striking the wall, determine ␦x, the maximum amount 共same mass as the disk兲 is connected to the string and is
the spring is compressed from its relaxed length 共see Fig. 6兲. dropped from rest. What is the acceleration a of the block?
共See Fig. 7.兲
共a兲 a = 共3 / 4兲 ⴱ g 关8%, 10%, 10%, 2%兴
共b*兲 a = 共2 / 3兲 ⴱ g 关53%, 26%, 50%, 84%兴
共c兲 a = 共1 / 2兲 ⴱ g 关32%, 45%, 35%, 15%兴
共d兲 a = 共1 / 3兲 ⴱ g 关3%, 5%, 3%, 0%兴
Fig. 6. Diagram for question 9. 共e兲 a = 共1 / 4兲 ⴱ g 关4%, 11%, 3%, 0%兴

139 Am. J. Phys., Vol. 79, No. 1, January 2011 E. T. Torigoe and G. E. Gladding 139

Downloaded 03 Jun 2013 to 216.47.136.20. Redistribution subject to AAPT license or copyright; see http://ajp.aapt.org/authors/copyright_permission

Fig. 8. An example of a question requiring a single numeric equation where
the target unknown appears on opposite sides of the equal sign.
APPENDIX B: SAMPLE EQUATION PRIORITY
QUESTION
We give an example of a question that requires a single
numeric equation, where the unknown target appears on op-
posite sides of the equal sign. 共Physics 211 Spring 2006
exam 2, question 10.兲
Three boxes are arranged as shown. The middle box has a
mass of 2 kg and accelerates to the right at 5 m / s2 on a
horizontal frictionless table. The boxes to the left and right
hang freely, suspended by strings over massless, frictionless
pulleys. The tension in the left string is T1 = 10 N. What is
the mass of the box M 1 on the left? 共See Fig. 8.兲
共a兲 M 1 = 0.675 kg
共b兲 M 1 = 1.91 kg
共c兲 M 1 = 2.13 kg
共d兲 M 1 = 3.75 kg
共e兲 M 1 = 4.16 kg
a兲
Electronic mail: etorigoe@gmail.com
1
E. Torigoe and G. Gladding, “Same to us, different to them: Numeric
computation versus symbolic representation,” in 2006 Physics Education
Research Conference, edited by L. McCullough et al. 共AIP, New York,
2007兲, pp. 153–156.
2
E. Torigoe, “What kind of math matters? A study of the relationship
between mathematical ability and success in physics,” Ph.D. dissertation,
University of Illinois at Urbana-Champaign, 2008.
3
C. Kieran, “Cognitive processes involved in learning school algebra,” in
Mathematics and Cognition: A Research Synthesis by the International
Group for the Psychology of Mathematics Education, edited by P. Nesher
and K. Kilpatrick 共Cambridge U. P., Cambridge, 1990兲, pp. 96–112.
4
C. Kieran, “The learning and teaching of school algebra,” in Handbook of
Research on Mathematics Learning and Teaching, edited by D. Grouws
共Macmillan, New York, 1992兲, pp. 390–419.
5
E. Filloy and T. Rojano, “Solving equations: The transition from arith-
metic to algebra,” For the Learning of Mathematics 9 共2兲, 19–25 共1989兲.
6
The examples shown are from Ref. 4.
7
J. H. Larkin, J. McDermott, D. P. Simon, and H. A. Simon, “Models of
competence in solving physics problems,” Cogn. Sci. 4 共4兲, 317–345
共1980兲.
8
J. Clement, “Algebra word problem solutions: Thought processes under-
lying a common misconception,” J. Res. Math. Educ. 13 共1兲, 16–30
共1982兲.
9
E. Cohen and S. E. Kanim, “Factors influencing the algebra ‘reversal
error’,” Am. J. Phys. 73 共11兲, 1072–1078 共2005兲.
10
E. Soloway, J. Lochhead, and J. Clement, “Does computer programming
enhance problem solving ability? Some positive evidence on algebra
word problems,” in Computer Literacy, edited by R. J. Seidel, R. E.
Anderson, and B. Hunter 共Academic, Burlington, 1982兲, pp. 171–201.
11
M. Scott, T. Stelzer, and G. Gladding, “Evaluating multiple-choice exams
in large introductory physics courses,” Phys. Rev. ST Phys. Educ. Res. 2
共2兲, 020102 共2006兲.
12
See supplementary material at http://dx.doi.org/10.1119/1.3487941 for all
ten numeric and symbolic pairs of questions used in this study.
13
Question 4 was created by modifying an existing symbolic question.
When numbers were introduced to create the numeric version, one of the
symbolic options corresponded to an imaginary quantity. To ensure the
similarity of all of the options, only the magnitude of this quantity was
displayed in the numeric version. Two of the other five options for this
question do not agree between the versions, but each of these options was
chosen by 2% or less of the students.
14
The p-value represents the likelihood that such a difference can be ob-
served under the assumption that the null hypothesis is true 共see Ref. 15兲.
15
G. V. Glass and K. D. Hopkins, Statistical Methods in Education and
Psychology, 2nd ed. 共Prentice-Hall, Englewood Cliffs, NJ, 1984兲, pp.
229–235.
16
Some questions were common between the two versions of the final
exam.
17
The discrimination of multiple-choice questions is most commonly mea-
sured using the point biserial coefficient of correlation because the result
of a multiple-choice question is most commonly dichotomous. The
multiple-choice questions in this study were analyzed using the Pearson
correlation coefficient r, because students were given partial credit for
multiple selections. As a result a student could receive a score of 0, 0.33,
0.5, or 1 on each question.
18
The error of the mean difference shown in Table III is less than what one
would calculate if the errors for the top and bottom groups were com-
bined in quadrature. To calculate the error shown, we took advantage of
the fact that the difference in score between the top and bottom groups
could be determined for each question. The error in the mean difference
for the equation priority questions was determined by calculating the
variance of the distribution of differences for the 40 equation priority
questions. This process of pairing data is analogous to how one would
calculate gains on the FCI by pairing the each precourse and postcourse
score by student, rather than finding the mean difference between the
average precourse score and the postcourse score for the class as a whole.
19
D. L. Schwartz, T. Martin, and J. Pfaffman, “How mathematics propels
the development of physical knowledge,” Cognit. Dev. 6 共1兲, 65–88
共2005兲.
20
V. M. Sloutsky, J. A. Kaminski, and A. F. Heckler, “The advantage of
simple symbols for learning and transfer,” Psychon. Bull. Rev. 12 共3兲,
508–513 共2005兲.
21
J. Tuminaro and E. F. Redish, “Elements of a cognitive model of physics
problem solving: Epistemic games,” Phys. Rev. ST Phys. Educ. Res. 3
共2兲, 020101 共2007兲.
22
E. Mazur, Peer Instruction: A User’s Manual 共Prentice-Hall, Upper
Saddle River, NJ, 1997兲, pp. 5–7.

140 Am. J. Phys., Vol. 79, No. 1, January 2011 E. T. Torigoe and G. E. Gladding 140

Downloaded 03 Jun 2013 to 216.47.136.20. Redistribution subject to AAPT license or copyright; see http://ajp.aapt.org/authors/copyright_permission