Thinking Skills and Creativity 30 (2018) 64–75

Contents lists available at ScienceDirect

Thinking Skills and Creativity

journal homepage: www.elsevier.com/locate/tsc

The interplay of creativity, failure and learning in generating

T
algebra problems
⁎
Esther Ziegler , Manu Kapur
Institute for Learning Sciences and Higher Education, ETH Zurich, Switzerland

A R T IC LE I N F O ABS TRA CT

Keywords: Typical mathematics instruction starts with a teacher explaining a worked example, followed by
Creativity students’ practicing isomorphic problems. Because such instruction may not optimally help
Failure students notice the critical features of the concept, an alternative way is to have them generate
Algebra learning their own examples across a range of contexts. One hundred and sixty four students participated
Example generation
in three algebra-learning studies, where they completed several worksheets, each targeting a
Problem-solving
specific algebraic manipulation principle. Each worksheet started with worked examples pre-
senting the targeted principle, after which the students had to generate their own examples.
Student-generated examples varied from being only superficially different from the worked-ex-
amples, to creative ones that differed on a structural level. Path analyses revealed significant
indirect and direct effects of structural creativity. The indirect effect revealed that structural
creativity was associated with a higher error rate, which, in turn, negatively influenced im-
mediate learning. Interestingly, in spite of the indirect effect, and what was a relatively simple
and short intervention embedded in a longer instructional program, the direct effect of structural
creativity on delayed posttest outcomes was positive and significant. We situate our findings in
the extant literature on the role of failure in mathematical creativity and learning.

1. Introduction

A major goal of mathematics instruction is to gain flexible problem-solving skills. Problem-solving skills can be basic in nature: for
example, algebraic manipulation and simplification skills. Such skills require not only a simple application of the principles and
procedures, but also the ability to distinguish and select between different types of procedures (Sweller, Van Merriënboer, & Paas,
1998; Ziegler & Stern, 2014). As an analytical contrast, problem-solving skills can also be more complex: for example, finding
solutions to novel problems that require students to attend to the deep structure of the problem and generate multiple representations
and solutions (e.g., Kapur, 2016).
Creativity can be considered an integral part of problem solving, especially when it comes to generating solutions (Chamberlin &
Moon, 2005; Plucker, Beghetto, & Dow, 2004). Good problem-solving skills are characterized by originality and flexibility of
thinking, the same features that characterize creative thinking. Similarly, good manipulation skills may be characterized by a flexible
and original use of skills, also a feature of creative use (Haylock, 1997). Thus, mathematics instruction should not only teach problem
solving and manipulation skills, but do so in a way that develops creativity (Cropley, 2005; Nadjafikhah, Yaftian, & Bakhshalizadeh,
2012; Schoenfeld, 1988).
Typical mathematics instruction expects students to learn correct solution procedures and methods (Nathan, 2012). The teacher

⁎
Corresponding author at: Institute for Learning Sciences and Higher Education, ETH Zurich, Clausiusstrasse 59, 8092 Zurich, Switzerland.
E-mail address: esther.ziegler@gess.ethz.ch (E. Ziegler).

https://doi.org/10.1016/j.tsc.2018.03.009
Received 5 July 2017; Received in revised form 20 March 2018; Accepted 22 March 2018
Available online 26 March 2018
1871-1871/ © 2018 Elsevier Ltd. All rights reserved.
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

uses worked examples and highlights critical features in the process. The students’ task is to follow the teachers’ instruction, and
understand and apply the taught procedures. Typically, worked examples are followed up with problem-solving practice on iso-
morphic problems to develop fluency.
Solving isomorphic problems does not require flexibility, because a superficial variation of a problem (e.g., changing the value of
a parameter, or the label for a variable) is easily noticed. In such instruction, creative approaches are under-emphasized because they
go beyond the scope of to-be-learned solutions, and may lead to errors (Beghetto & Sriraman, 2016; Cropley, 2010; Gralewski &
Karwowski, 2013; Runco, Acar, & Cayirdaga, 2017). Of course, good teachers may naturally highlight critical features of problems
and discuss alternative solutions, but students are rarely given the opportunities to be creative (Cropley, 2005; Nadjafikhah et al.,
2012; Sriraman, 2005). One way to develop creativity and flexibility is to provide students with the opportunity to solve problems on
their own, and generate their own examples, problems and solutions (Kapur, 2016; Watson & Mason, 2002). Thus, the focus should be
not only on teaching correct procedures but also on enabling creative thinking and behavior (Ginsburg, 1996).

1.1. Conceiving creativity

A widely-used definition of creativity, across a number of fields, is simply the number of ideas one can generate, also called
productive fluency (Cropley, 2000; Guilford, 1967). Additionally, these ideas are also assessed for novelty and usefulness (e.g.,
Benedek, Jauk, Sommer, Arendasy, & Neubauer, 2014; Lubart, 1994; Plucker et al., 2004; Runco & Jaeger, 2012). Thus, creativity
refers not only to the production of ideas, but also to the extent to which the ideas are novel and useful.
In the field of mathematics, the measure of novelty and usefulness may apply to professional mathematicians’ creativity.
Mathematicians are expected to produce novel ideas that are applicable; the more novel the idea, the better. However, we need to
reconceive novelty and usefulness if we are to use them for students in the process of learning mathematics. Of course, students
cannot be expected to produce new and original work, or extend the body of existing mathematical knowledge (Ervynck, 1991).
Instead, they should be given the opportunity to produce novel solutions to problems, or formulate new possibilities or questions
(Nadjafikhah et al., 2012; Sriraman, 2005). Therefore, even though students’ ideas may not be novel from the standpoint of the
discipline, they can be conceived as novel from the perspective of the learner and their prior knowledge (Kapur & Bielaczyc, 2012;
Lobato, 2003). Likewise, usefulness can be conceived as whether, and to what extent, these ideas influence students’ learning and
understanding (Schwartz & Bransford, 1998).
In short, creativity can be conceived as the number of ideas students produce, as well as the usefulness of the ideas for learning.
Next, we examine the relationship between creativity and failure: how creativity and failure co-occur when students are given the
opportunity to generate problems and solutions during initial learning.

1.2. Creativity and failure

Failure in mathematics is traditionally seen as an opportunity for teachers and students to receive feedback about the current state
of knowledge and areas for improvement (Hattie & Timperley, 2007). Research has also shown that failure can be a tool for preparing
students to benefit from subsequent instruction (Kapur, 2008; Schwartz & Martin, 2004). The shared idea in both the approaches is
that failure is a constructive part of learning. In the context of learning algebraic manipulation skills, we define failure as errors
student make in the application of these skills.
Performance in mathematics varies depending on individual differences. One of the key predictors of achievement differences is
intelligence (e.g., Mayes, Calhoun, Bixler, & Zimmerman, 2009). High-intelligence students usually make fewer errors when pro-
cessing mathematical material compared to low-intelligence students. As creativity and intelligence share a common cognitive base,
creative students can be expected to outperform less-creative students (Benedek et al., 2014; Nusbaum & Silvia, 2011). High error
rates are often seen as an indicator of low performers and low understanding (Bjork, 1994), which is why there tends to be little
constructive use of failure in traditional western classrooms (e.g., Stevenson & Stigler, 1994).
Apart from individual differences, low learning rates and sub-optimal solutions during initial learning are not necessarily a bad
sign. On the contrary, some learning impediments are even welcome, for example when they raise the level of difficulty in a desirable
way that helps students encode and retrieve information better (Bjork, 1994, Schmidt & Bjork, 1992). Difficulties during exposure to
new material can be said to be desirable when they activate, highlight, or connect pieces of information and, in turn, lead to improved
retention.
An example of a desirable difficulty is contrasted or interleaved processing of two or more principles. Such processing is more
demanding than sequenced or blocked processing of the principles. The former leads to more errors during instruction but higher
gains in retention (Rohrer & Pashler, 2010; Rohrer & Taylor, 2007; Ziegler & Stern, 2014, 2016). Another example is having students
generate information on their own, such as generating answers, explanations, solutions and procedures (Atkinson, Derry, Renkl, &
Wortham, 2000; Bjork & Bjork, 2011; Chi, 2000; Renkl, 1997; Watson & Mason, 2002; Ziegler, Edelsbrunner, & Stern, 2018). Similar
results of how errors can be beneficial in initial learning are also found in productive failure research, where students are prompted to
find multiple solutions to complex problems before learning from instruction (Kapur, 2008, 2013, 2014).
In sum, making errors appears to be pedagogically desirable for specific materials and learning settings, provided students are
engaged in processing that is germane for learning. Characteristics of such processing are combining information, experiencing
variation of material, going beyond presented material, and generating multiple solutions—all characteristics of creative thinking
and behavior. One might therefore conjecture a connection between errors and creativity, and its effect on learning. Integrating
research on errors and mathematical creativity makes it conceivable that students who make more errors in the process of generating

65
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

solutions, when controlled for reasoning ability, may also tend to produce more creative solutions. Conversely, students who tend to
produce more creative solutions may also tend to make more errors. Opportunities for student-generated solutions may increase the
likelihood of forming novel associations and combinations. This may drive creativity, but may also increase the likelihood of making
errors. The current study tests this conjecture on the co-occurrence of creativity and errors, and its influence on learning.

1.3. The current study

The aim of our study is to examine whether and how the co-occurrence of errors and creativity influences learning in the context
of learning algebraic manipulation skills.
Our algebra material consisted of learning two similar algebra principles of addition and multiplication. Students often tend to
confuse the addition and multiplication principles. Algebraic manipulation skills are typically not considered to be complex problem-
solving skills. Instead, they include the ability to distinguish and correctly apply these principles. A correct application can be seen as
an indicator of understanding the syntactic structure of algebra. An understanding of the syntactic structure of algebra is crucial for
learners, as it forms the basis for flexible use in applications (Kirshner & Awtry, 2004; Ottmar, Landy, & Goldstone, 2012). Speci-
fically, we look at the errors students make when given the opportunity to generate worked examples (that is, the problem and the
associated solution) from given worked examples.
We analyzed existing data of three former algebra-learning studies (Ziegler, Edelsbrunner, & Star, under revision; Ziegler & Stern,
2014). Our main goal was to examine how the production of creative worked examples (hereinafter referred to as examples)influ-
ences student learning. We carried out our goal through the following three-step analytical process: First, students’ generated ex-
amples were rated on the basis of superficial and structural creativity (defined in the following section) in comparison to the worked
examples presented in the learning program. Second, we examined the co-occurrence of students’ creativity scores and the generation
errors. Third, we examined the influence of students’ creativity scores and their generation errors on immediate learning and posttest
learning outcomes, controlling for pre-existing differences in reasoning ability.
We hypothesized that variations on a superficial level would neither influence the generation error rate nor posttest outcomes.
However, variations that attempt to change the deep structure may lead to more errors during initial learning. What remains un-
certain is, as argued earlier, whether these generation errors will have a beneficial effect on learning. We conjecture that the nature of
the targeted knowledge could further mediate the effect. Because this study did not focus on complex problem-solving and transfer,
but on basic problem-solving skills, we are unable to robustly predict the effects of creativity and errors on learning of basic algebraic
manipulation skills. Finally, we expected positive correlations of more creative productions with pre-existing differences in reasoning
ability.

2. Methods

2.1. Participants

Our data come from a sample of 164 students participating in three algebra learning studies (Ziegler et al., under revision; Ziegler
& Stern, 2014). In all three studies, contrast and sequential conditions were compared in a self-learning setting. The students in the
contrast condition studied addition and multiplication algebra examples presented to them simultaneously next to each other on a
worksheet (see Fig. 1). The students in the sequential conditions received first all the addition examples followed by all the multi-
plication examples. The results showed a consistent advantage of contrasted learning of the two algebra principles in the medium and
long term. Overall, 393 students participated in the studies. However, because the students in the contrasted learning condition
consistently outperformed those in the sequential learning condition, we only evaluated data of the 164 students in the contrasted
learning condition. The data we used for this study have not been analyzed or reported elsewhere.
The classes were from urban and suburban public schools in the Swiss canton of Zurich. All participants were sixth-graders who
had not yet received formal instruction in algebra. Students with insufficient language comprehension and those with special needs
were not eligible to participate in the study and instead worked on other materials with their teachers in a separate room. The
participating students were volunteers, and their parents provided written consent. Every teacher participating with their class was
rewarded 150 Swiss francs (approx. 150 US dollars), and each student received a small gift.
Data of six students were excluded from assessments and analysis because they missed training days, leaving 158 students in the
final sample with a mean age of 12.48 years (SD = 0.56 years).

2.2. Design and procedure

Each student participated in four 90-min training sessions on four consecutive days, and in three posttests. For the present
purposes, only the first posttest was included.
At the beginning of the first session, the students completed a pretest to assess their prior algebra knowledge. Then, a 5-min slide
presentation was given to explain how to read and write terms with letters (algebraic terms) and how to use the mathematical
expression “raise to the power of”. Subsequently, the students completed nine algebra units in a self-learning mode, which included
instruction using self-study worksheets followed by practice problems. After each unit, they took an end-of-unit test as a measure of
immediate learning. At the end of the experiment, the students completed a reasoning test.
The experiment was carried out by the first author, with support from a research assistant, in groups of 12–15 students in

66
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

A3 xy + xy + xy = 3 · xy M3 xy · xy · xy = x · y · x · y · x · y
= 3xy = x· x· x· y· y·y
= x3 · y3 = x3y3
2b + 2b + 2b + 2b + 2b = 5 · 2b 2b · 2b · 2b · 2b · 2b = 2 · b · 2 · b · 2 · b · 2 · b · 2 · b
= 10b = 2· 2· 2· 2· 2 · b· b· b· b· b
= 32 · b5 = 32b5
3cx + 3cx = 2 · 3cx 3cx · 3cx = 3· c· x· 3· c· x
= 6cx = 3· 3· c· c· x· x
= 9 · c2 · x2 = 9c2x2

A4 c2 + c2 + c2 + c2 = 4 · c2 = 4c2 M4 c2 · c2 · c2 · c2 = c · c · c · c · c · c · c · c = c8
a4 + a4 = 2 · a4 = 2a4 a4 · a4 = a · a · a · a · a · a · a · a = a8
x3 + x3 + x3 = 3· x3 = 3x3 x 3 · x 3 · x 3 = x · x · x · x · x · x · x · x · x = x9

A5 2x + 5x + 2x = 9x M5 2x · 5x · 2x = 2·x·5·x·2·x
= 2·2·5·x·x·x
= 20 · x3 = 20x3
3bc + bc + 6bc = 3bc + 1bc + 6bc 3bc · bc · 6bc = 3·b·c ·b·c·6·b·c
= 10bc = 3·6·b·b·b·c·c·c
= 18 · b3 · c3 = 18b3c3
y3 + 4y3 = 1y3 + 4y3 y3 · 4y3 = y·y·y·4·y·y·y
= 5y3 = 4·y·y·y·y·y·y
= 4 · y6 = 4y6

A6 m+m+a+m+a+m = a+ a+m+m+m+m M6 m · m · a · m · a · m = a·a·m·m·m·m

= 2·a + 4·m = a2 · m4
= 2a + 4m = a2m4
4+x+z+x+4+x = x+ x+ x+ z+ 4+ 4 4·x·z·x·4·x = 4·4·x·x·x·z
= 3·x + 1·z + 8 = 16 · x3 · z1
= 3x + z + 8 = 16x3z

A7 xy + x + xy + x + x
c2 + c + c + c2 + c
= x + x + x + xy + xy M7 xy · x · xy · x · x = x·y·x·x·y·x·x
= 3 · x + 2 · xy = x·x·x·x·x·y·y
= 3x + 2xy = x5 · y2 = x5y2
= c + c + c + c2 + c2 c2 · c · c · c2 · c = c·c·c·c·c·c·c
= 3 · c + 2 · c2 = 3c + 2c2 = c7

Fig. 1. Worked examples: addition learning steps (A3–A7), and multiplication learning steps (M3–M7). Grey-cursive = the intermediate steps to the
solutions that were marked red in the original version; black-bold = the examples and the results.

classrooms at the students’ schools. The students were instructed to work on their own and to ask the instructors directly if they had
questions or problems with the material.

2.3. Study materials

The instructional materials consisted of a paper-and-pencil self-study packet that included nine units with nine worksheets on the
topic of simplifying algebraic expressions. Each of the nine worksheets targeted a specific algebraic manipulation principle (i.e.,
adding and multiplying algebraic terms). Each worksheet consisted of a worked example section, an example generation section, and
a practice section.
In the worked example section, the main section at the top of each worksheet (Table 1), the students were presented with two

Table 1
Scheduling of the units.
Activities 1 unit per lesson 2 units per day

1st lesson: pretest (day 1) or repetition tests (days 2–4) 5’ 10’

2nd lesson: clearing at the end (days 1–4)
worked example section 20’ 40’
example generation section 10’ 20’
practice problems 5’ 10’
end-of-unit test 5’ 10’
Total 45’ 90’

67
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

blocks of 2–3 worked examples, one block of addition and one block of multiplication (for an example, see Fig. 1).
Every learning step exemplified one or two rules of increasing difficulty. The addition and multiplication examples used identical
numbers and letters; thus, the examples were differentiated only in the use of addition and multiplication signs, which naturally
affected the solution steps and results. The students were instructed to self-explain the worked examples by writing down how the
algebra problems were solved, and to ascertain the underlying principle on their own. To facilitate this step, the students were given
prompts to guide them in looking carefully at the worked examples and verbalizing the algebra principles. Their verbal explanations
were assessed by the research assistants. If their explanations were incorrect, the students were required to correct them. If their
explanations were too short, they were asked to write them out in greater detail before continuing to the next section.
In the example generation section of the worksheet, a section of shorter duration than the preceding worked example section (see
Table 1), the students were instructed to generate two examples for each block of worked examples. They were asked to generate
varied examples using other numbers and letters. If the students made an error, the only feedback given was whether the example
was correct or wrong. This feedback was given to all the students who made errors, and they were asked to correct their mistakes as
part of the self-learning instruction. Hence, the correct-incorrect feedback was the mechanism for the students to figure out where the
difficulty in the material was, and this mechanism was available to all the students. There was no additional teaching. Two measures
were used to assess the generated examples: Errors in generating examples (generation errors), and creativity in generating examples
(creativity). It is also important to note that the analyses of errors and creativity reported in this paper was not part of the original
study from which we extracted data for secondary analyses here, and therefore, there was no possibility of providing differential or
additional feedback or help to particular groups of students in situ.
In the practice problems section, at the end of each unit, the students received isomorphic practice problems to apply the self-
learned principles.

2.4. Assessment materials

2.4.1. Errors in generating examples

The generated examples were assessed by the first author. The students received correct-incorrect feedback, and errors had to be
corrected by the students themselves before continuing to the next section. If they needed help, the students were instructed to look at
the corresponding worked example. The number of errors made in generating their own examples was assessed separately for each
unit, and aggregated into an overall score of generation errors. Four examples were generated per unit, thus up to four errors were
possible.

2.4.2. Creativity in generating examples

We expected the student-generated examples to vary in their creativity. To capture the degree of students’ creativity we developed
a coding scheme. Recall that we defined creativity by the number of novel ideas produced (novelty conceived in relation to the
learners’ prior knowledge), and also the usefulness of the ideas (conceived in relation to the influence of the ideas on learning).
Novelty was defined as the degree of deviation from the presented examples, that is, the number of structurally novel elements in
the generated examples. Usefulness was defined as meaningful deviations from the examples, such as those that attend to the deep
structure. Generating novel examples can be done on a superficial level or on a structural level (Gentner, 2010; Holyoak et al., 2005),
which in turn influences how useful they can be for learning. As argued earlier, superficial changes would conceivably be less useful
for learning than deep structure changes.
Thus, our scheme coded the deviations from the presented examples found in the generated examples across two categories:
superficial creativity and structural creativity. Every generated example of units 3–7 was coded separately for its superficial and
structural creativity. For the analyses, we chose only units 3–7, because they were the ones that presented the relevant rules to
distinguish between the two algebra principles of addition and multiplication. Units 1 and 2 were introductory units, while units 8
and 9 extended the rules to more varied examples.
Students’ generated examples could deviate from the presented examples on a superficial level by only changing the letters
(Table 2A), for example from “a + a + a + a =” to “z + z + z + z =”, or from “xy + xy + xy =” to “cd + cd + cd =”. Changing the
letters was what the students were required to do, and was thus within the range of expected deviation. In other words, it can be seen
as superficial creativity. The students varied the letters in their examples to different degrees. We identified four categories of
increasing value: equal letters, displayed letters, familiar letters, and rare letters. For the superficial creativity, 1 point could be given
for each of the 4 examples. Thus, for the five worksheets, a maximum superficial creativity score of 20 points was possible.
Students’ generated examples could also deviate from the presented examples on a structural level by altering the structure of a
worked example beyond simply changing numbers and letters (Table 2B). Such deviations can be seen as structural creativity, for
they make something novel out of a normal task solution. We identified three categories of such creative structural changes: using
terms of a higher degree, using different terms, and extraordinary changes. For the structural creativity, 3 points could be scored for
each of the 4 examples. Thus, for the five worksheets, a maximum structural creativity score of 60 points was possible.

2.4.3. Pre-existing differences

Prior knowledge and reasoning ability were used as control variables. Prior algebra knowledge was measured by a pretest con-
taining eight problems, e.g., “a + a + a + a =”, “7b · 7b =”, “xy + xy + xy + xy=”. Reasoning ability was measured by the L-P-S, a
comprehensive intelligence test (Horn, 1983). The test is based on Thurstone’s primary mental abilities and contains a total of 15
subtests. The subtests can be administered as standalone assessments. The students completed two subtests that targeted inferential

68
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

Table 2
Coding scheme.
A For superficial creativity
Category Description Scoring

Equal letters ○ Not changing the letters at all 0 point

Displayed letters ○ Choosing letters that are used in other examples of the same worksheet ½ pointa
Familiar letters ○ In the case of single letters: Choosing letters that are frequently used in the learning program and in mathematics in general, ½ pointa
e.g., a, b, c, x, y, z
○ In the case of double letters: Choosing letters that are frequently used in the learning program and in mathematics in
general, such as ab, bc, xy, yz
Rare letters ○ Using letters that are neither used in any presented worked examples, nor are familiar letters as described above 1 pointa

B For structural creativity

Category Description Scoring

Terms of higher ○ Not changing the degree of the terms, i.e., retaining double letters xy to ab, or retaining the same exponents c2 to m2 0 point
degree
○ Changing the numbers of terms by more than 2 ½ pointb
○ Using terms with more than the presented number of letters, i.e., changing xy to bmq, or xy to ypqhj, e.g. 1 pointb
flc + s + s + flc, 5qgh + qgh + 4qgh =, 4abcd + 2abcd + 10abcd =
○ Using exponents that do not appear in any of the presented examples, i.e., changing c2 to m6, or c2 to u7, or new
combinations of double letters and powers, e.g., nm3 + 2nm3 =
Different terms ○ Using the same number of different terms, i.e., changing 2 terms “ab, a” to 2 similar terms “xy, x” 0 point
○ Using more different terms, i.e., changing 3 different terms “4, x, y” to 4 different terms “8, z, y, b” or 2 different 1 pointb
terms “ab, a” to 3 different terms “ab, a, m”, or 2 different terms “c2, c” to 3 different terms “m2, m, xm”
Extraordinary Creating examples in an extraordinary way, such as: 1-2 pointsb
changes ○ Using high exponents (≥ 8), e.g., from c2 ∙ c2 ∙ c2 ∙ c2 = to g9 + g9 + g9 =, a30 ∙ a30 =, z10 + z10 + z10 + z10 =,
zw9 + 9zw9 =
○ Using mixed exponents in the same problem, e.g., from x3 ∙ x3 ∙ x3 = toy6 ∙ 3y4 =, x4 ∙ x7 ∙ x2 ∙ x24 =, a + b2 + c3 =,
m ∙ y ∙ x3 ∙ m ∙ m2 ∙ y ∙ y2 =, m2 + x4 + y8 =, e5b5a5y5 ∙ e3b3a3y3 ∙ 3ebay =, a3 + b + b2 + a + a3 + b =,
○ Using high Coefficients (≥ 10), e.g., from 5ab + 3ab = to 5c + 18c + 100000c =, 36 cd + 46 cd + 60cd =,
1327x + 53x =, 3la ∙ 10la ∙ 20la =
○ Applying many repetitions of terms (≥ 4), e.g., from 4cx + 4cx = to 3st + 3st + 3st + 3st + 3st + 3st + 3st + 3st =,
z8 + z8 + z8 + z8 + z8 + z8 + z8 + z8 + z8 + z8 =,
5+a+5+b+a+b+b+a+b+a+a+4+a+5+b+a=
○ Using 2 and more new different terms, e.g., from c2 + c + c2 + c + c2 = to c2 + d + d2 + c + cd + c + c2 + cd =,
a3 + b + b2 + a + a3 + b =
○ Using words with a meaning, e.g., johannes ∙ johannes = j2o2h2a2n2n2e2s2, ebay ∙ ebay = e2b2a2y2,
7xyzjuhe + xyzjuhe + 2xyzjuhe = 10xyzjuhe, beo ∙ beo ∙ a2 ∙ b2 ∙ c2 ∙ ha ∙ ha ∙ e3 ∙ c3 ∙ o3 = b4e5o5a4c5h2, ubs + ubs = 2ubs

Note. amaximum score for superficial creativity per example = 1 point, bmaximum score for structural creativity per example = 3 points.

thinking. In subtest 3, the students had to recognize regularity or irregularity in geometrical figures across 40 items. In subtest 4, they
had to recognize regularity or irregularity in number and letter series across 40 items. Their reasoning ability was an average of the
two sub-scores.

2.4.4. End-of-unit tests

As an indicator of immediate learning, that is, what they had learned during the unit, the students completed a test after each unit.
The end-of-unit tests contained 3–8 algebra items. These items were similar to the ones presented on the worksheet the students had
just completed. The students were not allowed to use the self-study instruction material while attempting these tests.

2.4.5. Posttest
As an indicator of posttest learning outcomes, all the students completed a 58-item algebra posttest that assessed their ability to
simplify expressions targeted in the intervention (e.g., “a2 · a · ay · 4a =”, “5ab + b + 3b + 2ab + 2b =”, “y · y3 · y2 · y =”,
“2 + 5x + 4 + 2x + 3 =”). The posttest was given one day after all the units were completed. Students’ algebra knowledge was
operationalized as the number of completely correct answers. No marks were given for partially correct answers.
Given the focus of the study was on algebraic manipulation skills, these end-of-unit tests as well as the posttest targeted basic
problem-solving skills. Basic problem-solving skills require not only a simple application of the rules and procedures, but also the
ability to distinguish between different types of procedures. However, it should be noted these tests did not contain items that target
complex problem-solving skills or transfer.

3. Results

3.1. Pre-existing differences

Out of eight algebra problems, the students only solved 1.17 (SD = 1.24) on average, and only 6 students out of 158 reached 4
points. This low score on the algebra pretest implies a floor effect and indicates that the students had almost no prior algebra

69
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

knowledge. The reasoning score reached 27.54 (out of 40, SD = 3.95) points. Thus, for the analyses, only reasoning ability was
included as a control variable.

3.2. Generation error score

The generation error rate was generally low. The generation error scores averaged over the nine units ranged from 0.05 to 0.34
(out of a maximal score of 4 per unit) with high standard deviations between 0.25 and 0.70. The generation error score summed up
over the three units 4, 5, and 7 reached only 0.79. The generation error rate was sufficiently manifest only when the error scores of all
nine units were summed up to 2.04 points (SD = 1.99). However, the error sum score of the three units 4, 5, and 7 was highly
correlated to the overall score of all nine units, r = 0.699, p < .001, and also the overall score to the single unit scores. In particular,
for unit 4, r = 0.464, p < .001, for unit 5, r = 0.514, p < .001, and for unit 7, r = 0.383, p < .001.

3.3. Creativity scores

3.3.1. Structural creativity

We coded structural creativity on units 3–7. However, we expected units 4, 5, and 7 to offer greater possibility for creativity and
thus used these units to measure structural creativity. Our justification for using these three units is twofold.
First, on a conceptual basis, a task analysis revealed units 1–3 and 6 to be less appropriate for generating creative examples. Unit 1
and 2 were introductory units with only easy problems, such as a + a + a = 3a, or c + c = 2c. Unit 3 was the first unit with more
complex problems, such as xy + xy + xy = 3xy, or 2b + 2b + 2b + 2b + 2b = 10b. Thus, the students were more focused on pro-
cessing the greater complexity, and not yet ready to use the opportunity for creativity. Unit 6 only used single letters, which made
creativity less probable, for example, m + m + a + m + a + m = 2a + 4m, or 4 + x + z + x + 4 + x = 3x + z + 8.
In contrast, unit 4 exposed the students to three examples that are similar, such as c2 + c2 + c2 + c2 = 4c2, or a4 + a4 = 2a4, thus
affording greater opportunities for creativity. Unit 5 introduced coordination of variables and coefficients, such as
3bc + bc + 6bc = 10bc, offering greater possibility for new combinations. And finally, unit 7 involved the combination of single
variables with double variables, as well as variables with exponents, such as xy + x + xy + x + x = 3x + 2xy, or
c2 + c + c + c2 + c = 3c + 2c2. These new elements afforded greater opportunity for creativity.
Second, we examined the five units for the distribution of the creativity scores (Fig. 2A). The pie charts show no manifestation of
creativity in unit 3 (84.8% scored 0 points), and only sparse manifestation in unit 6 (53.8% scored 0 points). Creativity was suffi-
ciently manifest only in units 4, 5, and 7. Furthermore, the structural creativity sum score for units 4, 5, and 7 was highly correlated
with the overall structural creativity score of all the units, r = 0.929, p = < .001, as well as with the single unit structural creativity
scores for unit 4, r = 0.688, p = < .001, unit 5, r = 0.731, p = < .001, and unit 7, r = 0.761, p = < .001. For this reason, we
decided to include only units 4, 5 and 7 to measure the creativity score.

3.3.2. Superficial creativity

For the superficial score, creativity was similarly distributed among the five units with sufficient manifestation on all the units
(Fig. 2B). However, for parity with structural creativity, we only included the sum score of units 4, 5, and 7. Also in parallel with
structural creativity, the superficial creativity sum score of units 4, 5, and 7 was highly correlated with the overall superficial
creativity score of all the units, r = 0.671, p = < .001, and with the single unit superficial creativity scores for unit 4, r = 0.719,
p = < .001, unit 5, r = 0.845, p = < .001, and unit 7, r = 0.748, p = < .001.

3.4. Relationship between creativity and the outcome measures

The path analyses were run with SPSS Amos 23. The path diagrams (Fig. 3) illustrate the direct and indirect relationships together
with the standardized regression coefficients. We ran separate path analyses for structural creativity and superficial creativity. Both
models showed good fits, for structural creativity, RMSEA = 0.00, RMSEA CI 90 = 0.00 to 0.07, CFI = 0.99, and for superficial
creativity, RMSEA = 0.04, RMSEA CI 90 = 0.00 to 0.22, CFI = 0.99. Statistical significances of the standardized indirect effects were
tested using bootstrapping procedures (resampled 10,000 times and used the bias-corrected method to create the 90% confidence
interval (Arbuckle, 2009)). In addition, we ran path models separately for immediate learning and posttest outcomes. The resulting
patterns and model fits were very similar to our final models (Fig. 3) and are therefore not reported separately.

3.4.1. Structural creativity

As Fig. 3A illustrates, the relationships between all measures were controlled for reasoning ability, which was significantly
correlated with generation errors, β = −0.20, p = .008, immediate learning, β = 0.21, p = .003, and posttest outcomes, β = 0.34,
p < . 001, but not with structural creativity, β = 0.01, p = .925. The standardized direct effect between structural creativity and
generation error was positive and statistically significant β = 0.18, p = .021. The direct effect between generation errors and im-
mediate learning was significantly negative, β = −0.25, p = .021. Also, the relationship between structural creativity and immediate
learning was negative even though not significant, β = −0.05, p = .471. However, this relationship between structural creativity and
immediate learning was mediated by the number of generation errors, β = −0.04, p = .019 (see Table 3). This mediation indicates
that making more errors associated with higher structural creativity significantly diminished the negative effect on immediate learning
[from β = −0.10, p = .203 to β = −0.05, p = .471]. The relationship between structural creativity and posttest outcomes was

70
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

Fig. 2. The pie charts (A) display the distribution of the structural creativity scores on the single units. The legend displays the score ranging from 0
to 11 points. The pie charts (B) display the distribution of the superficial creativity scores on the single units. The legend displays the score ranging
from 0 to 4 points.

significantly positive with no mediation via the indirect path, β = 0.11, p = .035. Overall, even though structural creativity was
associated with increased generation errors indirectly, which negatively influenced immediate learning, it had overall a direct positive
impact on the posttest outcomes.

3.4.2. Superficial creativity

As Fig. 3B illustrates, the relationships between all measures were controlled for reasoning, which was significantly correlated
with generation errors, β = −0.20, p = .010, immediate learning, β = 0.21, p < .001, and posttest outcomes, β = 0.35, p < .001, but
not with superficial creativity, β = 0.09, p = .234, showing a pattern similar to structural creativity. In contrast to structural crea-
tivity, the relationship between superficial creativity and immediate learning was neither significant, β = 0.04, p = .637, nor mediated

71
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

.01
A e1
1

-.20** reasoning ability

generation errors
.21**

.18* -.25* e2
1

-.10 .34***
-.05
structural creativity immediate learning
e3
1
.54***
.11*
posttest outcomes

.09
B e1
1

-.20* reasoning ability

generation errors
.21***

.00 -.26** e2
1

-.08 .35***
.04
superficial creativity immediate learning
e3
1
.54***
-.09
posttest outcomes

Fig. 3. Standardized regression coefficients for the relationship between structural creativity and immediate learning or posttest outcomes (A) and
between superficial creativity and immediate learning or posttest outcomes (B) as mediated by generation errors and controlled by reasoning ability.
*p < .05, **p < .01 ***p < .001.

Table 3
Test of mediation for path analysis model.
Relationship Standardized direct effect Standardized indirect effect Result

A structural → errors → immediate −.05 (ns.) −.04* (p = .019) mediation

structural → errors → immediate → posttest .11* (p = .035) .07 (ns.) no mediation

B superficial → errors → immediate .04 (ns.) .00 (ns.) no mediation

superficial → errors → immediate → posttest −.09 (ns.) .02 (ns.) no mediation

Note. *p < .05. Structural = structural creativity, superficial = superficial creativity, errors = generation errors, immediate = immediate learning,
posttest = posttest outcomes. (A) For the relationship between structural creativity and the outcomes, and (B) between superficial creativity and the
outcomes. The arrows (→) show indirect mediation paths.

by generation errors, β = 0.00, p = .960 (see Table 3). The standardized direct effect between structural creativity and generation
error was not related at all, β = 0.00, p = .964. Thus, there was only a negative direct effect between generation errors and immediate
learning, β = −0.26, p = .009. Also, the relationship between superficial creativity and posttest outcomes showed a negative trend
with no mediation on the indirect path, β = −0.09, p = .090. Overall, superficial creativity did not impact immediate learning or
posttest outcomes.

4. Discussion

This study was designed to examine the interplay of failure, creativity and learning in an algebraic manipulation context. We
analyzed existing data from three former studies implementing an algebra self-learning program that prompted the students to
generate their own examples based on presented worked examples. Our main analytical goal was to examine whether student-
generated examples were correct or incorrect, shared superficial or structural similarity with the worked examples, and relate this
variation to immediate learning and posttest outcomes, controlling for pre-existing differences in reasoning ability.
Path analyses allowed us to examine the direct and indirect effects of the association between creativity and errors, and its effect
on learning. The indirect path showed that creativity was associated with higher error rates, which negatively influenced immediate
learning, but not posttest outcomes. The direct path showed a positive effect of structural creativity (after controlling for reasoning
ability) on posttest outcomes, in spite of it leading to higher error rate in the generation phase, and its negative influence on

72
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

immediate learning. Finally, we also showed that there was neither a direct effect nor an indirect effect for superficial creativity,
further bolstering the case for structural creativity.
As expected, structural creativity in generating examples led to a higher error rate after controlling for pre-existing differences in
reasoning ability, whereas superficial creativity did not influence the error rate. One possible explanation is that structural changes
may result in problems that are more difficult, increasing the likelihood of error when students try to solve them. Because superficial
changes do not attend to deep structure, it is not surprising that such changes do not influence learning.
We also found that higher error generation rates negatively influenced immediate learning. Unexpectedly, however, this negative
influence did not extend to posttest outcomes. At the same time, structural creativity had a significant and positive, direct effect on
posttest outcomes. One interpretation of these findings is that generating examples with structural creativity introduces a desirable
difficulty (Bjork, 1994, Rohrer & Taylor, 2007; Schmidt & Bjork, 1992; Ziegler & Stern, 2014; Ziegler et al., 2018), because such
problems are also harder to solve. This, in turn, may have increased the error rates. In other words, attempts at structural creativity
came at a cost of generating errors, but making these errors was only detrimental for immediate learning, because structural crea-
tivity had a direct effect on posttest outcomes. Even though the direct effect is small, it is important to note that a simple and short
creativity intervention of 20 min embedded in a longer instructional program of 90 min managed to produce this small but significant
effect.
An alternative interpretation of these findings is that the feedback given to the students when they made errors, which they were
asked to correct, may have led to learning from these errors. As a result of such learning, they may perform better on both the
immediate learning and posttest outcomes. However, our data does not provide full empirical support for this interpretation. Our
findings showed that the learning gains became manifest only on the posttest outcomes, but not in immediate learning.
An important contribution of our findings emerges when contrasted with past research on productive failure, which demonstrated
a positive impact on conceptual knowledge and transfer but not on basic procedural knowledge (Kapur, 2014). Our study, although
not designed to be a productive failure demonstration, shows that making errors can help learn basic problem-solving skills and
procedures as well. Thus conceived, our findings can be seen as extending past research and showing advantages of failure in dealing
with even basic algebraic manipulation skills (in combination with creative behavior). That is, failure in initial learning may be useful
for not only novel conceptual material but also for basic problem-solving skills.

4.1. Limitations and future directions

Although the effects reported are significant, they are weak. While this is a limitation, the results are nonetheless interesting and
warrant continued research and exploration. Future research would do well to replicate and test our conjectures experimentally.
Based on the findings concerning students’ creativity in present study, an experimental study could be designed with an explicit
prompting of students to design multiple and creative examples. Specifically, the posttest could be designed to include items that tap
not only basic algebraic manipulation skills targeted in our study, but also more complex problem-solving and transfer. Perhaps then
we may be able to detect a stronger effect of creativity on learning.
Another limitation in our study might be that the students, in general, made only sparse errors in generating examples. This was
an indication that the task of generating examples was easy. Past research on productive failure indicates that there needs to be
sufficient and diverse student production for the failure to be productive (Kapur, 2013, 2014). This was not the case in our study,
where the students engaged with relatively easy algebraic manipulation material that is quite different from typical productive failure
material dealing with more complex conceptual concepts. Prompting students for multiple creative solutions and explaining their
solutions may lead to a broader range of errors.
Related to this point, another limitation might be that the students who generated highly creative examples and struggled with
them during the intervention did not get feedback on their creative examples, other than correct-incorrect feedback. The students had
to correct these errors by themselves, and were not given the opportunity to further elaborate or discuss these problems in class. Note
also that the students processed a whole series of problems over four days. What they learned in one step was overlaid by new
problems in the next step. Thus, applied creativity in one step may have remained isolated instead of it deepening through deliberate
discussion and consolidation. An intervention that addresses these limitations may afford students more opportunities to be creative,
make errors, and learn from them.

4.2. Conclusion

Our study provides a simple but significant demonstration of the co-occurrence of creativity and errors. Even with a short
intervention, we were able to show how creativity is associated with making errors, how these errors have a detrimental effect on
immediate but not longer-term learning, and that creativity positively influences learning in the longer term. In doing so, our study
underscores the importance of designing for creativity and failure in mathematics instruction, while also suggesting several ways in
which future research could build upon our findings.

Acknowledgements

We are grateful to Sara Ziegler for her assistance in the implementation of the Study in the school classes, Adrienne Suter for her
contribution in the development of the creativity coding scheme, and Sara Ziegler and Adrienne Suter for their assistance in the
evaluation of the test materials.

73
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

References

Arbuckle, J. L. (2009). Amos 18 user’s guide. Chicago: Amos Development Corporation.

Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of
Educational Research, 70(2), 181–214. http://dx.doi.org/10.3102/00346543070002181.
Beghetto, R. A., & Sriraman, B. (2016). Creative contradictions in education: Cross disciplinary paradoxes and perspectives. Schweiz: Springer Verlag.
Benedek, M., Jauk, E., Sommer, M., Arendasy, M., & Neubauer, A. C. (2014). Intelligence, creativity, and cognitive control: The common and differential involvement
of executive functions in intelligence and creativity. Intelligence, 46(1), 73–83. http://dx.doi.org/10.1016/j.intell.2014.05.007.
Bjork, E., & Bjork, R. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L.
M. Hough, & J. R. Pomerantz (Eds.). Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56–64). New York: Worth Publishers.
Bjork, R. (1994). Institutional impediments to effective training. In D. Druckman, & R. A. Bjork (Eds.). Learning, remembering, believing: Enhancing human performance
(pp. 295–306). Washington, DC: National Academy Press.
Chamberlin, S. A., & Moon, S. M. (2005). Model-eliciting activities as a tool to develop and identify creatively gifted mathematicians. The Journal of Secondary Gifted
Education, 17(1), 37–47. http://dx.doi.org/10.4219/jsge-2005-393.
Chi, M. T. H. (2000). Self-explaining expository texts: The dual processes of generating inferences and repairing mental models. In R. Glaser (Vol. Ed.), Advances in
instructional psychology: Educational design and cognitive science: Vol. 5, (pp. 161–238). Mahwah, NJ: Erlbaum.
Cropley, A. (2000). Defining and measuring creativity: Are creativity tests worth using? Roeper Review, 23(2), 72–79. http://dx.doi.org/10.1080/
02783190009554069.
Cropley, A. (2005). Creativity and problem-solving: Implications for classroom assessment. Leicester, Great Britain: British Psychological Society.
Cropley, A. (2010). Creativity in the classroom: The dark side. In D. H. Cropley, A. J. Cropley, J. C. Kaufman, & M. A. Runco (Eds.). The dark side of creativity (pp. 297–
316). New York: Cambridge University Press.
Ervynck, G. (1991). Mathematical creativity. In D. Tall (Ed.). Advanced mathematical thinking (pp. 42–52). New York: Kluwer Academic Publishers.
Gentner, D. (2010). Bootstrapping the mind: Analogical processes and symbol systems. Cognitive Science, 34(5), 752–775. http://dx.doi.org/10.1111/j.1551-6709.
2010.01114.x.
Ginsburg, H. P. (1996). Toby‘s math. In R. J. Sternberg, & T. Ben-Zeev (Eds.). The nature of mathematical thinking (pp. 175–202). .
Gralewski, J., & Karwowski, M. (2013). Polite girls and creative boys? Students' gender moderates accuracy of teachers' ratings of creativity. Journal of Creative
Behavior, 47(1), 290–304. http://dx.doi.org/10.1002/jocb.36.
Guilford, J. P. (1967). The nature of human intelligence. New York: McGraw-Hill.
Hattie, J. A., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. http://dx.doi.org/10.3102/003465430298487.
Haylock, D. W. (1997). Recognising mathematical creativity in school children. ZDM Mathematics Education, 29(3), 68–74. http://dx.doi.org/10.1007/s11858-997-
0002-y.
Holyoak, K. J. (2005). Analogy. In K. J. Holyoak, & R. G. Morrison (Eds.). The cambridge handbook of thinking and reasoning (pp. 117–142). Cambridge, UK: Cambridge
University Press.
Horn, W. (1983). Leistungsprüfsystem LPS 2. Auflage. Göttingen, DE: Hogrefe.
Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–424. http://dx.doi.org/10.1080/07370000802212669.
Kapur, M. (2013). Comparing learning from productive failure and vicarious failure. The Journal of the Learning Sciences, 23(4), 651–677. http://dx.doi.org/10.1080/
10508406.2013.819000.
Kapur, M. (2014). Productive failure in learning math. Cognitive Science, 38(5), 1008–1022. http://dx.doi.org/10.1111/cogs.12107.
Kapur, M. (2016). Examining productive failure, productive success, unproductive failure, and unproductive success in learning. Educational Psychologist, 51(2),
289–299. http://dx.doi.org/10.1080/00461520.2016.1155457.
Kapur, M., & Bielaczyc, K. (2012). Designing for productive failure. The Journal of the Learning Sciences, 21(1), 45–83. http://dx.doi.org/10.1080/10508406.2011.
591717.
Kirshner, D., & Awtry, T. (2004). Visual salience of algebraic transformations. Journal for Research in Mathematics Education, 35(4), 224–257. http://dx.doi.org/10.
2307/30034809.
Lobato, J. (2003). How design experiments can inform a rethinking of transfer and vice versa. Educational Researcher, 32(1), 17–20. http://dx.doi.org/10.3102/
0013189X032001017.
Lubart, T. (1994). Creativity. In R. J. Sternberg (Ed.). Thinking and problem solving (pp. 289–332). San Diego: Academic Press.
Mayes, S. D., Calhoun, S. L., Bixler, E. O., & Zimmerman, D. N. (2009). IQ and neuropsychological predictors of academic achievement. Learning and Individual
Differences, 19(2), 238–241. http://dx.doi.org/10.1016/j.lindif.2008.09.001.
Nadjafikhah, M., Yaftian, N., & Bakhshalizadeh, S. (2012). Mathematical creativity: Some definitions and characteristics. Procedia – Social and Behavioral Sciences,
31(1), 285–291. http://dx.doi.org/10.1016/j.sbspro.2011.12.056.
Nathan, M. J. (2012). Rethinking formalisms in formal education. Educational Psychologist, 47(2), 125–148. http://dx.doi.org/10.1080/00461520.2012.667063.
Nusbaum, E. C., & Silvia, P. J. (2011). Are intelligence and creativity really so different? Fluid intelligence, executive processes, and strategy use in divergent thinking.
Intelligence, 39(1), 36–45. http://dx.doi.org/10.1016/j.intell.2010.11.002.
Ottmar, E., Landy, D., & Goldstone, R. L. (2012). Teaching the perceptual structure of algebraic expressions: Preliminary findings from the pushing symbols inter-
vention. Paper Presented at the Thirty-Fourth Annual Conference of the Cognitive Science Society.
Plucker, J. A., Beghetto, R. A., & Dow, G. T. (2004). Why isn’t creativity more important to educational psychologists? Potentials, pitfalls, and future directions in
creativitiy research. Educational Psychologist, 39(2), 83–96. http://dx.doi.org/10.1207/s15326985ep3902_1.
Renkl, A. (1997). Learning from worked-out examples: A study on individual differences. Cognitive Science, 21(1), 1–29. http://dx.doi.org/10.1016/S0364-0213(99)
80017-2.
Rohrer, D., & Pashler, H. (2010). Recent research on human learning challenges conventional instructional strategies. Educational Researcher, 39(5), 406–412. http://
dx.doi.org/10.3102/0013189X10374770.
Rohrer, D., & Taylor, K. (2007). The shuffling of mathematics practice problems boosts learning. Instructional Science, 35(6), 481–498. http://dx.doi.org/10.1007/
s11251-007-9015-8.
Runco, M. A., Acar, S., & Cayirdaga, N. (2017). A closer look at the creativity gap and why students are less creative at school than outside of school. Thinking Skills and
Creativity, 24(1), 242–249. http://dx.doi.org/10.1016/j.tsc.2017.04.003.
Runco, M. A., & Jaeger, G. J. (2012). The standard definition of creativity. Creativity Research Journal, 24(1), 92–96. http://dx.doi.org/10.1080/10400419.2012.
650092.
Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of practice: Common principles in three paradigms suggest new concepts for training. Psychological
Science, 3(4), 207–217. http://dx.doi.org/10.1111/j.1467-9280.1992.tb00029.x.
Schoenfeld, A. H. (1988). When good teaching leads to bad results: The disasters of 'well-taught' mathematics courses. Educational Psychologist, 23(2), 145–166. http://
dx.doi.org/10.1207/s15326985ep2302_5.
Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475–522. http://dx.doi.org/10.1207/s1532690xci1604_4.
Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction.
Cognition and Instruction, 22(2), 129–184. http://dx.doi.org/10.1207/s1532690xci2202_1.
Sriraman, B. (2005). Are giftedness and creativity synonyms in mathematics? The Journal of Secondary Gifted Education, 17(1), 20–36. http://dx.doi.org/10.4219/jsge-
2005-389.
Stevenson, H., & Stigler, J. W. (1994). The learning gap: Why our schools are failing and what we can learn from Japanese and Chinese education. New York: Simon &
Schuster.

74
E. Ziegler, M. Kapur Thinking Skills and Creativity 30 (2018) 64–75

Sweller, J., Van Merriënboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251–296. http://dx.doi.
org/10.1023/A:1022193728205.
Watson, A., & Mason, J. H. (2002). Student-generated examples in the learning of mathematics. Canadian Journal of Science, Mathematics and Technology Education,
2(2), 237–249. http://dx.doi.org/10.1080/14926150209556516.
Ziegler, E., Edelsbrunner, P., & Star, J. R. (under revision). Preventing interference: Reordering complexity in the learning of new concepts.
Ziegler, E., Edelsbrunner, P., & Stern, E. (2018). The relative merits of explicit and implicit learning of contrasted algebra principles. Educational Psychology Review.
http://dx.doi.org/10.1007/s10648-017-9424-4.
Ziegler, E., & Stern, E. (2014). Delayed benefits of learning elementary algebraic transformations through contrasted comparisons. Learning and Instruction, 33, 1–16.
http://dx.doi.org/10.1016/j.learninstruc.2014.04.006.
Ziegler, E., & Stern, E. (2016). Consistent advantages of contrasted comparisons: Algebra learning under direct instruction. Learning and Instruction, 41(1), 41–51.
http://dx.doi.org/10.1016/j.learninstruc.2015.09.006.

75