Computers & Education 75 (2014) 253–262

Contents lists available at ScienceDirect

Computers & Education

journal homepage: www.elsevier.com/locate/compedu

Cognitive and affective implications of persuasive

technology use on mathematics instruction
Mehmet Ersoy a, *, Yavuz Akbulut b
a
Department of Computer Education & Instructional Technology, Eskisehir Osmangazi University, Faculty of Education, 26480, Eskisehir, Turkey
b
Anadolu University, Faculty of Education, Turkey

a r t i c l e i n f o a b s t r a c t

Article history: This study investigated the effects of different computer based interventions on mathematics achieve-
Received 27 December 2013 ment, attitudes towards search engines and Internet self-efficacy. Participants were 105 pre-service
Received in revised form teachers enrolled in a calculus course at a Turkish state university. A 3 (interventions)
2 (pre- and
10 March 2014
post-test) factorial design was implemented. The first group resorted to activities provided by a web-
Accepted 10 March 2014
Available online 19 March 2014
based semantic search engine (i.e. Wolfram Alpha). The same content was provided to the second
group through a computer algebra system (i.e. Mathematica). The third group followed the same content
through linear web pages. The intervention lasted eight weeks. Data were collected through focus-group
Keywords:
Human–computer interface interviews, an achievement test, an attitude scale on the instructional use of search engines and an
Interactive learning environments Internet self-efficacy scale. The quantitative data were analyzed through descriptive statistics followed by
Intelligent tutoring systems univariate and multivariate analyses of variance whereas the qualitative data were examined through
Post-secondary education descriptive analysis. MANOVA results revealed that the main effect of testing time, the main effect of
intervention, which favored the semantic search engine group, and the interaction effect of intervention
by time led to significant differences between students’ achievement scores. However, the intervention
and interaction effects were not significant for attitude and self-efficacy. Qualitative data revealed
strengths and limitations of each intervention.
Ó 2014 Elsevier Ltd. All rights reserved.

1. Introduction

Advances in emerging technologies have a profound impact on contemporary communication and learning experiences. ICTs contribute
to the production and consumption rate of information, which increases the importance of equipping 21st century individuals with relevant
information processing skills. Accordingly, many learning technologies have been adopted in the last century most of which perform
perceivably well at the inception. However, failures or shortcomings also occur while implementing relevant instructional technology
principles with the novel instruments. Then, new applications and alternatives emerge and scholars begin to question the persuasive power
of the previous one in comparison to the most recent one.
Persuasion is an influential construct, which shelters both argumentation and information sharing to change individuals’ beliefs, atti-
tudes and behaviors. Traditionally, it means ‘human communication designed to influence the autonomous judgments and actions of others’
(Simons, Morreale, & Gronbeck, 2001, p. 9). The current study questions the potential persuasive power of different web-based tools in
mathematics instruction since an increasing amount of human communication and instruction is facilitated through web-supported tools.
Even a troubleshooting problem pertaining to the web lead individuals to resort to the web for potential solutions. In this regard, the
persuasive role of these technologies is highly prominent in individuals’ daily experiences whereas the educational persuasiveness is still
under scrutiny.
The evolution in the field of persuasive technology has accelerated in the last decade particularly in terms of applications in human–
computer interaction. These applications had fiscal outcomes such as user-based customization of advertisements or usability hints in
commercial web pages. On the other hand, the reflections of persuasive technologies in instructional design and educational settings have

* Corresponding author. Tel.: þ90 (222) 239 3750x1646; fax: þ90 (222) 229 3124.
E-mail addresses: mehmetersoy@ogu.edu.tr, ersoycimeyil@gmail.com (M. Ersoy).

http://dx.doi.org/10.1016/j.compedu.2014.03.009
0360-1315/Ó 2014 Elsevier Ltd. All rights reserved.
254 M. Ersoy, Y. Akbulut / Computers & Education 75 (2014) 253–262

been observed rarely (Mintz & Aagard, 2012). Thus, the current study aimed to implement relatively novel and potentially persuasive tools
to enhance mathematics instruction. That is, the research design is expected to scrutinize on the persuasive potential of the selected tools for
instructional settings. The following session revisits the theoretical foundations of the study.

2. Theoretical framework

Internet has a prominent role both on daily endeavors of individuals and instructional practices. Through the advances in web-based
technologies, scholars revisit several cognitive and affective variables related to instructional effectiveness such as achievement, attitude
and self-efficacy. This study addressed these variables and resorted to different web-based tools to facilitate effectiveness in mathematics
instruction. The theoretical rationale of the study is summarized under the following headings.

2.1. Persuasive technology use and mathematics instruction

Persuasive technology or ‘captology’ refers to the overlap of technology and persuasion. That is, captology focuses on ‘the design,
research and analysis of interactive computing products created for the purpose of changing people’s attitudes or behaviors’ (Fogg, 2003, p.
5). The theoretical framework of persuasion can be adopted for different instructional applications including virtual reality, educational
games, mobile devices and web-based environments. These tools can be used to address the change in critical learning variables including
attitude, behavior, motivation, perspective and adaptation. Thus, persuasive technologies which are quite common in individuals’ daily lives
are also expected to facilitate instructional endeavors (Ijsselsteijn, de Kort, Midden, Eggen, & van den Hoven, 2006).
Three means of persuasion are proposed in the literature, which are human–human interaction, human–computer interaction, and
computer supported human–human interaction (Oinas-Kukkonen, 2008). The prominent type of persuasiveness in instructional settings is
generally human–computer interaction (Mintz & Aagard, 2012). Accordingly, instructional web-based tools employed in the current design
are instances of human–computer interaction.
Integration of human–computer interaction with the concept of persuasion can be realized through semantic web applications. Studies
which address semantic web and hypertext simultaneously (Cristea, 2004; Papasalouros, Retalis, & Papaspyrou, 2004) maintain that se-
mantic web is generated in a really smart way by Berners-Lee (2003). This is because it requires XML use rather than HTML, which helps it
diagnose meaningful statements of users. Similar applications are used in mathematics instruction through additional markup languages
such as Mathematics Education Markup Language (i.e. MeML, Zou, 2005). If such languages are used in a way to integrate the math language
and content with web contemporary facilities, effectiveness of online mathematics instruction can be improved.
To name an instructional technology as persuasive is not possible within the scope of small-scale empirical works. On the other hand,
interactive and instructional affordances of semantic web tools are promising to claim that they can be persuasive as long as attitude and
behavior changes in the desired direction are observed. For instance, several studies suggested that these applications are effective facil-
itators in mathematics instruction (David, Ginev, Kohlhase, & Corneli, 2010; Jones, Geraniou, & Tiropanis, 2013; Ullrich & Melis, 2010). In this
regard, the current work focuses on a web-based semantic search engine (i.e. Wolfram Alpha), and discusses its instructional affordances on
the basis of the diffusion and acceptance models so that its persuasive potential can be justified theoretically.

2.2. Innovation–Decision process

Almost all adoption and diffusion theories propose that adoption of an innovation is not a single act but a process occurring in a social
system over time (Straub, 2009). Potential users go through stages when interacting with an innovation. These stages are discovery of an
innovation (knowledge), formation of a positive impression (persuasion), adopting or rejecting (decision), actual usage (implementation)
and evaluation of the impact of the innovation to continue or discontinue usage (confirmation) (Rogers, 2003). Accordingly, adopters are
categorized through a five-stage continuum ranging from those who readily adopt (i.e. innovators) through those who resist an innovation
(i.e. laggards). Another contribution of the Innovation–Decision Process model is the concept of perceived key attributes, or the perceptions
of potential users regarding the advantages of an innovation. That is, ‘individuals are more likely to adopt an innovation if it offers them a
better way to do something (relative advantage), is compatible with their values, beliefs and needs (compatibility), is not too complex
(complexity); can be tried out before adoption (trialability); and has observable benefits (observability)’ (Surry & Ely, 2007, p. 106). Based on
this theoretical framework, in order to consider a web-based semantic search engine as an innovation, its perceived key attributes should be
unique or comparatively stronger than a preceding application. For instance, the comparison of a contemporary computer algebra system
(CAS; Mathematica) with a web-based semantic search engine (Wolfram Alpha) is summarized in Table 1.
CAS can be considered as an innovation in comparison to previous instructional applications. However, Wolfram Alpha is relatively new
and looks somewhat advantageous. For instance, it does not require any additional license cost or installation effort. All a user need is the
Internet access. In addition, command use is not as strict as Mathematica. In addition to ready commands, conceptual search queries can

Table 1
Comparison of a sample computer algebra system with a web-based semantic search engine.

System Example Relative advantage Compatibility Complexity Trialability Observability

Computer Mathematica It is necessary to know Manipulation of formulas Prerequisite steps and Specialized Observation of
algebra system how to ask a question through commands commands are needed software is needed individuals who
already use it
Web-based Wolfram Alpha Similar to contemporary Basic understanding of Even simple questions Internet connection Can be shared through
semantic search search engines concepts is sufficient may lead to holistic is sufficient blogs, social networks
knowledge or e-mails

Adapted from Andersen, 2009.

 M. Ersoy, Y. Akbulut / Computers & Education 75 (2014) 253–262 255

lead to sample calculations and solutions. On the other hand, there are also several limitations pertaining to Wolfram Alpha, which can be
eliminated through a subsequent persuasive technology.
The influence of Rogers’s theory is quite strong in the literature as it reveals the factors which influence the choices of individuals
regarding the adoption of an innovation (Straub, 2009). On the other hand, Straub’s comprehensive review on technology adoption suggests
that the giant scope of the theory makes it difficult to frame a single study accordingly. More specifically, the theory tells why adoption
occurs, but does not tell how to facilitate adoption. In this regard, further models should be embraced to understand the nature of
persuasiveness.

2.3. Emerging theories of adoption: TAM and UTAUT

Supplementary viewpoints regarding the adoption processes can be extracted from technology acceptance models. The Technology
Acceptance Model (TAM) (Davis, 1989) and the Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh, Morris, Davis, &
Davis, 2003) are two models which are originally based out of computer science, but have been applied to instructional settings effectively
(Straub, 2009).
TAM provides quantifiable variables to understand individuals’ adoption tendencies (Straub, 2009). Two perceived characteristics of an
innovation are ease of use and usefulness. The former is related to self-efficacy and refers to the usage with less effort, while the latter refers
to perceived improvement in job performance, which influences future use (Davis, 1989). In this regard, one can claim that embracing
relevant design principles to facilitate interface usage and employing motivational design tips to increase perceived usefulness may foster
the adoption.
UTAUT, on the other hand, empirically compares previous theories through a longitudinal validation in different work-based settings
(Straub, 2009). Key determinants of the usage behaviors are performance expectancy, effort expectancy, social influence and facilitating
conditions. In addition, moderators for intention to use were added into the model which are gender, age, experience and voluntariness of
use. Both the key determinants and moderators are expected to shift in importance in different settings. The comprehensiveness of the
model is likely to influence empirical works in new contexts; however, the implementations based on this model are still very few.
Both models helped the current research to determine variables of interest. Relatively lower cost and coding effort warranted by Wolfram
Alpha can be promising in terms of the perceived ease of use. In addition, intention for active and continuous search, which facilitate the
learning process (Choo & Marton, 2003; Wilson, 1997), is originated from current technology acceptance models. In this regard, charac-
teristics of a typical information seeking behavior can be reflected in the Wolfram Alpha context. Finally, if the instructional endeavors lead
to better cognitive and affective outcomes, usefulness and cost-effectiveness can be considered plausible as well. In brief, variables
considered in the current study (i.e. achievement, attitude and self-efficacy) are theoretically grounded through the relevant innovation–
decision and technology acceptance models.

2.4. Constructivism

Constructivism assumes that reality is created by individuals and social groups through their interactions and experiences with the
world. That is, rather than being objectively independent from individuals, knowledge is embodied in human experiences, perceptions,
imaginations and social constructions (Jonassen, Cernusca, & Ionas, 2007). Thus, learners are likely to create knowledge when they are
actively engaged in producing something, which is also personally meaningful and sharable with others. This idea of learning by doing
constituted the first forms of constructivism which was formulated by Piaget and Inhelder (1966), and ameliorated through the
theoretical and empirical contributions of further scholars (Harel & Papert, 1990; Papert, 1980, 1986; Piaget, 1971; Resnick, 1989). Such
a framework has been successfully implemented in the field of instructional technology while teaching cognitive psychology (Gabriele,
Tavernise, & Bertacchini, 2012), automatics and robotics (Jara, Candelas, Puente, & Torres, 2011), and mathematical methods to un-
dergraduate students (Faraco & Gabriele, 2007; Tiernan, 2010). These studies revealed that engaging students, providing them with
appealing interaction experiences, and letting them use different strategies enhance students’ learning efficiency and retention, and
help them acquire a variety of relevant skills. Similarly, affordances of the current web-based semantic search engine are expected to
facilitate a constructivist learning atmosphere where students can learn by manipulating mathematical input (compatibility), accessing
holistic knowledge through simple and meaningful queries (complexity), and sharing their experiences through relevant means
(observability).

2.5. Cognitive Load Theory

While generating instructional activities pertaining to salubrious human–computer interaction, theoretically and empirically sound
approaches should be employed. In this regard, instructional principles originated from the Cognitive Load Theory (Sweller, 2008) can
be plausible. The theory maintains that there are two types of cognitive loads: intrinsic and extraneous. While the former is related to
the complexity of the materials, which cannot be manipulated without losing the originality of the learning tasks, the latter is about
the quality of the instructional design and should be manipulated in a way to increase instructional effectiveness (Sweller, van
Merriënboer, & Paas, 1998). The theory further posits that extraneous cognitive load can be reduced and instructional effectiveness
can be empowered through eliminating redundant materials, employing multiple channels and preventing split attention. Accordingly,
Wolfram Alpha helps the instructor to use relevant materials in a simple manner. In addition, the interface helps the users to see
interacting learning components in a holistic way. That is, the learners are able to see the consequences of their mathematical ma-
nipulations on the interface. Moreover, these manipulations are not strictly conducted through professional commands or complex
formulas. Rather, simple statements by a novice user can be used to initiate the processes. Thus, even materials with high element
interactivity can be reflected easily as any manipulation of the learner is immediately reflected on relevant calculations, equations and
diagrams.
256 M. Ersoy, Y. Akbulut / Computers & Education 75 (2014) 253–262

3. Purpose

The main purpose of the current research was to investigate the effects of different instructional interventions on mathematics
achievement, attitudes towards the use of search engines as an information retrieval tool and self-efficacy regarding educational internet
use. More specifically, three instructional techniques were compared with regard to their cognitive and affective outcomes. These tech-
niques were a semantic web-based search engine (i.e. Wolfram Alpha), a popular CAS (i.e. Mathematica) and use of linear web pages during
instructor-centered mathematics instruction.
The collateral purpose was to contribute to the experimental literature in the field since a serious decline in experimental interventions
has been observed recently (Nolen, 2009; Ross, Morrison, & Lowther, 2010). Moreover, recent empirical work has mainly addressed the
effectiveness of different CAS software, which facilitated the use of visual inputs during mathematics instruction (Aksoy, 2007; Aktumen,
2007; Godarzi, Aminifar, & Bakhshalizadeh, 2009; Tokpah, 2008). On the other hand, such systems require professional software programs,
above-average command knowledge and standard infrastructure. In contrast, the semantic search engines, such as the one investigated in
the current study, require nothing else than an Internet-connected device.
In brief, the following research questions were considered to address the purpose of the study:

1. Are there any differences between the pre- and post-tests scores of the web-based semantic search (WBSS), computer algebra system
(CAS) and web-supported instruction (WSI) groups in terms of mathematics achievement, attitudes towards search engines and
Internet self-efficacy?
2. What are the strengths and limitations of each intervention?

4. Methods and procedures

In order to evaluate the influence of three different treatments on the dependent variables, the current research involved a pre-test/post-
test control group design. Allocation of participants to treatment groups was realized through the random assignment procedure. Thus, the
design was a true experiment (Gravetter & Wallnau, 2000; Ross & Morrison, 1996).

4.1. Participants

The experiment was conducted in 2013 spring with undergraduate students who were enrolled in a pre-service mathematics teacher
education department at a Turkish state university. The study was initiated with 130 students who were registered to a compulsory calculus
course. However, subject loss occurred due to several reasons such as irregular attendance or lack of voluntariness for participation. Thus,
the experiment was completed with 105 participants. The majority of the participants were females (f: 83; 79.05%) while the male pop-
ulation was smaller (f: 22; 20.95%). They were homogeneous with regard to age since the majority were 19–21 year-olds (f: 92; 87.62%)
whereas a few were between the ages of 22 and 24 (f: 13; 12.38%). There were equal number of members in each group (i.e. 35).

4.2. Experimental context

The experiment began with a 3-h- orientation which helped the researchers to equip students with the basics of the technology
they would use. Following the orientation week, the experiment lasted seven weeks and each lesson lasted 3 h. The instructional
content covered limit, continuity, derivative and integral. Levels of the independent variable were web-based semantic search (WBSS),
computer algebra system (CAS) and web-supported instruction (WSI). On the other hand, dependent variables of interest were
mathematics achievement, attitudes towards search engines and Internet self-efficacy. Reliable and valid scales pertaining to
achievement, attitudes and self-efficacy were either developed or adapted by the researchers, which were used both as pre- and post-
implementation measures.
The first experimental group (WBSS) was exposed to Wolfram Alpha as a semantic search engine. The group was not exposed to any
additional materials such as instructional CDs or web pages. The students began each class with sample conceptual search activities
regarding the topic of the week. Students were provided handouts with study tips at the beginning of each course, and they took down their
search results on the handouts. In accordance with the tips, the search behavior proceeded till the learners grasped the topic and took down
all their inferences on the handouts. The program let students generate conceptual search queries through a virtual keyboard in addition to
conventional calculations. Students were also able to conduct partial searches for different components of the equations, or visualize parts of
the equations through selecting specific lines. Through the steps provided in the handouts, students could visualize each learning outcome.
The instructor was the facilitator whereas the center of the instructional activities was Wolfram Alpha.
The second experimental group was exposed to Mathematica which is a conventional computer algebra system (CAS). Similar to the
WBSS group, the content was introduced through handouts. However, students were not provided with any web-based search options. The
intervention involved applying software commands according to the course handouts and taking down the inferences on handouts through
interpreting command results. Each topic in the learning outcomes was visualized through applying relevant command lines. Besides,
learners were allowed to manipulate command lines to see the influence of their modifications on the outputs. They were also able see the
diagrams from different angles in a multi-dimensional environment. Similar to the WBSS, the program was the center of instructional
activities whereas the instructor was the facilitator.
The third group was provided with the same content through instructor-led lectures which were supplemented with linear web
pages. The supplementary materials were mostly static where the calculation tasks were primarily on the shoulders of students.
Moreover, learners were able to manipulate the diagrams to the extent that the Java application let them. The group was provided with
similar instructions; however, rather than taking down their inferences on handouts, they were asked to prepare assignments after the
content was delivered.
 M. Ersoy, Y. Akbulut / Computers & Education 75 (2014) 253–262 257

4.3. Data collection tools

4.3.1. Calculus II achievement test (CAT)

The first dependent variable was achievement scores. The achievement test was comprised of 30 multiple-choice questions which were
prepared through a table of specifications to address limit, continuity, derivative and integral. Responding time was determined as 40 min
for each group.
While preparing the instrument, 40 questions were prepared at the beginning. Piloting of the items was done in 2012 fall with 206
undergraduate students at the same department. The pilot sample did not involve any students from the actual experiment. Similar to the
distribution in experimental groups, the majority of the sample was females (71.35%) and 20 to 22 year-olds (86.89%).
Item facility, item discrimination and reliability analyses were conducted with a software program (i.e. ITEMAN). Based on the sug-
gestions of Ozcelik (1997) items with facility indices above .80 or below .30 were eliminated. In addition, items with discrimination indices
below .30 were also suppressed. The average item facility of the final form was .504. The internal consistency coefficient was .79 which was
plausible (George & Mallery, 2003).

4.3.2. Internet self-efficacy scale (ISES)

The Internet Self-Efficacy Scale was first developed by Sahin (2009) in a similar research context with undergraduate students who were
enrolled in different pre-service teacher education departments. The scale included 28 Likert items where each competency was rated on a
scale ranging from not qualified (1) to completely qualified (5). The scale had a one-factor structure with an internal consistency coefficient
of .96. The current study revealed internal consistency coefficients ranging from .81 to .83 in different implementations, which were all
plausible (George & Mallery, 2003).

4.3.3. Attitudes towards search engines (ATSE)

Attitudes toward search engines were addressed through 15 Likert items developed by Liaw and Huang (2003). The original scale
involved 20 items which were rated ranging from completely disagree (1) through completely agree (7). The scale was translated and
adapted to Turkey in the current study.
After the official permission from the developers of the scale was received, original items were translated to Turkish by two professional
interpreters, and reviewed by five experts in relevant fields. In order to pursue reliability and validity studies, exploratory (EFA) and
confirmatory factor analyses (CFA) were conducted with students from the same college. The EFA sample consisted of 219 participants
whereas the CFA sample involved 403 participants.
The EFA revealed ideal Bartlett (c2(105) ¼ 1899.519, p < .001) and KMO (.91) values, which indicated ideal factorability (Field, 2009;
Pallant, 2007). Principal components analysis was followed by the Varimax rotation. Both the Kaiser criterion (Kaiser, 1974) and Catell’s
Scree test (Catell, 1966) indicated a one-factor solution. Items with loadings less than .40; communalities less than .30 and insignificant
inter-item correlations were eliminated from the final solution. The final form explained 49.52% of the total variance with a one-factor
solution, which was plausible (Kline, 1994). The scale had a high internal consistency coefficient (a ¼ .92) (George & Mallery, 2003).
Item total correlations ranged from .532 through.743; factor loadings ranged from .576 through .807 and communalities ranged from .332
through .652.
In addition to the EFA, the top and bottom 27 percent of the participants were compared through independent samples t-tests for each
item. All comparisons revealed statistically significant probability values which indicated that the item discrimination values of each item
was plausible (p < .001). Moreover, the final form of the adapted scale was administered to 72 students biweekly to investigate test and re-
test reliability. The Pearson correlation of these successive implementations was .79 with a probability value below .01.
The final form was exposed to the CFA with a new sample. None of the error covariance values was problematic enough to remove further
items. In addition, all t values of the items were above 2.56, which indicated that the scale had a sound structure. As provided in Table 2, all fit
values were plausible, which revealed that the scale structure was confirmed with the new sample.

4.3.4. Focus group interviews

In order to obtain in-depth information regarding the treatments, validate quantitative findings and reveal the strengths and weaknesses
of each condition, qualitative data were collected through focus group interviews. The interview form was prepared by the researchers and
reviewed by the dissertation committee. Questions and further discussions in the interview addressed issues such as the instructional
strategies, instructional content and materials, role of the instructor, level of learning, meeting learner expectations, differences of the
treatments from conventional practices and learner suggestions for further implementations.

Table 2
Evaluation of the confirmatory factor analysis.

Index Ideal interval Observed value Rationale

c2 0  c2  2df 152.72 < 164 Kline (2005)
p .05  p  1.00 <.05 Hoyle (1995)
c2/df 0  c2/df  2 1.07 Tabachnick and Fidel (2001)
RMSEA 0  RMSEA  .05 .047 Raykov and Marcoulides (2006)
SRMR 0  SRMR  .05 .047 Sumer (2000)
NFI .95  NFI  1.00 .89 Thompson (2008)
NNFI .95  NNFI  1.00 .93 Raykov and Marcoulides (2006)
CFI .95  CFI  1.00 .95 Thompson (2008)
GFI .95  GFI  1.00 .95 Hu and Bentler (1999)
AGFI .90  AGFI  1.00 .93 Yilmaz and Celik (2009)
Chi-square: 152.72; df: 82
.
258 M. Ersoy, Y. Akbulut / Computers & Education 75 (2014) 253–262

In order to realize focus group meetings which represent everybody in the experiment, a hierarchical cluster sampling was realized. That
is high-, medium- and low-achievers in each group were determined. Then, three independent meetings were held with three intervention
groups to obtain the feedback of learners from all achievement levels.

4.4. Data analysis

Quantitative data were summarized through descriptive statistics first. The first research question required a 3
2 mixed-design
multivariate analysis of variance (MANOVA) since there were three theoretically related dependent variables (achievement, attitude,
self-efficacy), one between-groups independent variable with three levels (WBSS, CAS, WSI) and one within-groups independent variable
with two levels (pre- vs. post-test). In addition, whenever the main effects or interaction effect of the two factors were significant; further
post-hoc analyses were conducted through one-way and two-way ANOVAs. All assumptions of these parametric tests were checked before
the analyses.
To address the second research question, descriptive analysis was conducted with the focus group interview data (Yildirim & Simsek,
2006). The purpose was to define the findings in accordance with pre-determined questions and themes rather than generating undis-
covered themes from the data. Interrater reliability of the thematic coding procedure was done through the formula proposed by Miles and
Huberman (1994) (i.e. Reliability ¼ Number of agreements/[Number of agreements þ Number of disagreements]*100). The formula was
employed three times for each focus group interview and the coefficients ranged from 87 to 92 percent. That is, a concordance between the
researchers and independent raters was sustained. Moreover, member check was implemented. In other words, findings were confirmed by
the participants as well. As an ethical consequence of the written agreement with the participants, real respondent names were concealed
and pseudonyms were used during all analyses.

5. Results

The following sections summarize the quantitative and qualitative findings successively.

5.1. Group comparisons with regard to dependent variables

First of all, descriptive statistics of each group with regard to the dependent variables were examined. Means and standard deviations of
each group are provided in Table 3.
Group means revealed that scores pertaining to all dependent variables increased in the post-test. However, in order to see whether
these differences were statistically and practically significant, a robust parametric test was needed. Moreover, to compare the groups
without conducting a Type I error (Alpha Error), the probability value of .05 was divided by the number of dependent variables (i.e. 3)
through the Bonferroni Adjustment Procedure (Huck, 2012; Pallant, 2007). In essence, the cut-off probability for significance was deter-
mined as .017 in further analyses. Results of the multivariate analysis are provided in Table 4.
The first line of the table revealed that scores on the dependent variables differed significantly with regard to the type of intervention
(Wilks’ L ¼ .732; F(6,198) ¼ 5.571; p < .017). That is, the three groups performed differently with regard to the dependent variables. The
second line of the table involved a significant main effect for time (Wilks’ L ¼ .319; F(3,99) ¼ 70.388; p < .001), which indicated that the
increase from the pre-test through the post-test was a significant one. Finally, the third line of the table involved a significant main effect for
interaction (Wilks’ L ¼ .837; F(6,198) ¼ 3.070; p < .01), which suggested that the increase from the pre-test to post-test was not homogeneous
across the three groups. The eta squared values revealed that the interaction effect of intervention by time had a medium effect size, while
the individual main effects for time and intervention had large effect sizes (Cohen, 1988).
In two-way analyses of variances, a significant interaction effect provides more valuable information than individual main effects of the
independent variables (Huck, 2000, 2008). That is, rather than reporting the isolated influence of a single factor, interpreting the combined

Table 3
Descriptive statistics.

Intervention n X SD
Calculus achievement1 WBSS 35 15.085 3.156
CAS 35 13.657 2.721
WSI 35 14.228 3.068
Calculus achievement2 WBSS 35 20.628 3.581
CAS 35 18.342 2.375
WSI 35 16.942 3.095
Attitudes towards search engines1 WBSS 35 59.828 5.943
CAS 35 62.171 7.018
WSI 35 58.914 8.441
Attitudes towards search engines2 WBSS 35 64.200 5.443
CAS 35 63.685 5.454
WSI 35 61.171 6.080
Internet self efficacy1 WBSS 35 97.600 16.031
CAS 35 87.342 19.663
WSI 35 96.085 21.945
Internet self efficacy2 WBSS 35 106.000 14.540
CAS 35 93.514 16.400
WSI 35 100.371 17.196
1 Pretest; 2 Posttest; WBSS: Web-based semantic search; CAS: Computer algebra system; WSI: Web-supported instruction
.
 M. Ersoy, Y. Akbulut / Computers & Education 75 (2014) 253–262 259

Table 4
Summary table for the 3
2 mixed-design MANOVA.

Effect Wilks’ L F df1 df2 p h2

Intervention (WBSS-CAS-WSI) .732 5.571 6 198 <.017 .143
Time (Pre- vs. Post) .319 70.388 3 99 <.001 .681
Time* Intervention .837 3.070 6 198 <.01 .085
Dependent variables: Calculus achievement, attitudes towards search engines and internet self-efficacy
.

effects of multiple factors is more plausible. Such an analysis requires conducting multiple comparisons to reveal the source of this variation
across groups (Green & Salkind, 2007). Thus, each dependent variable was examined one by one through conducting further 3
2 mixed-
design ANOVAs.
The first ANOVA was conducted for the achievement. The main effect for intervention (F(2,102) ¼ 7.993, p < .017, h2 ¼ .135), the main effect
for time (F(1,102) ¼ 181.989, p < .001, h2 ¼ .641) and the main effect for interaction (F(2,102) ¼ 6.856, p ¼ .002, h2 ¼ .119) were all statistically
significant. More specifically, post-test scores were significantly higher than pre-test scores and this difference was not equal across groups.
Thus, further comparisons across groups were conducted through one-way ANOVAs, which revealed that the groups were equal at the
beginning (F(2,102) ¼ 2.026, p ¼ .137). However, they differed significantly with a large effect size in the end (F(2,102) ¼ 12.954, p < .001,
h2 ¼ .202). Multiple comparisons through a conservative post-hoc test (i.e. Scheffe) revealed that the difference between the WBSS and WSI
groups was statistically significant (p < .001). Furthermore, the difference between the WBSS and CAS groups was statistically significant as
well (p ¼ .009). However, the difference between the CAS and WSI group was not statistically significant (p ¼ .165). This finding revealed that
Wolfram Alpha led to better achievement scores than both Mathematica and web-supported instruction, whereas the difference between
Mathematica and instructor-led web-supported instruction was not statistically significant.
The same procedure was followed for the second dependent variable, i.e. attitudes towards search engines. The two-way ANOVA results
revealed that the main effect for intervention was not statistically significant (F(2,102) ¼ 2.246, p ¼ .111). This finding might be due to the lack
of sufficient statistical power (i.e. .285). Thus, a large sample may work better for such an analysis (Ellis, 2010). With regard to the effect for
time, findings revealed that post-test scores were significantly better than the pre-test scores (F(1,102) ¼ 386.786, p < .001, h2 ¼ .189). Finally,
the interaction effect of time by intervention was not statistically significant (F(2,102) ¼ 2.362, p ¼ .099). That is, all interventions empowered
positive attitudes towards search engines and this trend was homogeneous across experimental conditions.
The final analysis was conducted for Internet self-efficacy. The analysis revealed that the main effect for intervention (F(2,102) ¼ 4.074,
p ¼ .020) and the interaction effect were not statistically significant (F(2,102) ¼ 1.408, p ¼ .249). Note that the critical p value should be
reduced to .017 through the Bonferroni Adjustment, thus the p value of .020 was not considered significant. The lack of sufficient statistical
power (i.e. .55 for intervention, .165 for interaction) could be the reason for these non-significant findings. On the other hand, the main effect
for time was significant with a large effect size (F(1,102) ¼ 39.339, p < .001, h2 ¼ .278). This result was similar to the previous one, which
indicated that all interventions increased Internet self-efficacy.

5.2. Interview findings

The focus group interviews with each group revealed strengths and limitations pertaining to each intervention. The interview with the
WBSS group revealed that using semantic search engine was perceived as an advantage because it allowed users to search through the
names of different geometric diagrams, follow search strategies through visuals and repeat activities whenever needed. The program let
students learn the meanings of different coefficients and equations more effectively. Learners believed that they were more attentive and
the exercises were consistent with the provided content. In addition, they thought that they were able to combine different search strategies
to enhance their learning experiences. They also believed that visuals provided through the intervention were far more effective and richer
than those provided through conventional handouts. They further reported that the coefficients and equations were more concrete;
however, they found the steps of preparing geometric diagrams insufficient. Moreover, they had problems in comprehending and
employing the use of parentheses. They asked for further details regarding the current calculation steps. They also needed further assistance
while conceptualizing the variables through search engines; however, the details they required can only be provided through new de-
velopments in the search engine.
The interview with the CAS group revealed that learners were able to identify and create different commands in addition to the
contribution of these commands on diagrams. The content and exercises were found suitable and visual support was evaluated as an
advantage. Learners believed that diagrams provided by the CAS helped them extend their two-dimensional blackboard practices. More
specifically, they reported that they learned how to produce multi-dimensional diagrams through the program. CAS activities further
facilitated conceptualizing novel diagrams and examining functions more closely. However, they had difficulty in implementing commands
which are too abstract to conceptualize. In addition, they needed native language support and additional implementations to make better
use of the program.
The WSI group found linear web page support positive as they facilitated the transfer of their current language. On the other hand, they
reported difficulty in conceptualizing the novel diagrams. They further reported that they learned how to examine the diagrams better than
those provided in conventional handouts and on the board. They liked seeing additional examples, equations and diagrams through the
computer. However, they considered the transition from in-class activities to web-supported examples as an interruption, and asked for a
more natural environment where transitions are realized smoothly. Finally, they believed that materials in the native language are quite
insufficient.

6. Discussion and conclusion

Findings revealed that all interventions increased achievement, Internet self-efficacy and attitudes towards search engines. Advantages
of each intervention observed in the current study were mostly in line with the previous literature. For example, Simsek (2010) observed
260 M. Ersoy, Y. Akbulut / Computers & Education 75 (2014) 253–262

that web-supported mathematics instruction increased achievement while teaching derivatives. A similar increase in achievement was also
reported by Aktumen (2007) who thought general mathematics, and Aksoy (2007) who taught derivatives through contemporary computer
algebra systems. In addition, the contribution of the semantic search engines to achievement was maintained in a previous work by Zhu,
Chen, Chen, and Chern (2011). A novel finding was that achievement scores varied with regard to the type of intervention. That is, the
semantic search engine group was better than other groups in terms of achievement. This finding refuted the results of a previous study by
Rouet, Ney, Charles, and Lallich-Boidin (2009) who compared linear web-supported applications and web applications involving semantic
search, and found no differences with regard to performance.
Emerging theories of adoption discussed in the theoretical framework underline the importance of self-efficacy and attitudes in
empowering persuasion and adoption (Davis, 1989; Venkatesh et al., 2003). All interventions used in the current study increased the scores
in these variables and no differences were observed among the conditions. This might be due to the insufficient statistical power, which
should lead researchers to replicate the current experiment through a larger sample. On the other hand, findings suggest that if the purpose
is to empower self-efficacy or positive attitudes, it is up to the implementers to choose the condition which suits their unique instructional
context best. However, if the achievement scores are on the foreground, the current analysis suggested that web-supported search engines
can be integrated into the mathematics curricula whenever possible. In general, current findings tend to support the implications of pre-
vious work regarding the facilitating role of semantic search engines in mathematics instruction (David et al., 2010; Jones et al., 2013; Ullrich
& Melis, 2010). Combining this advantage with the potential adoption rate of semantic search engines may lead to promising educational
outcomes. That is, current findings imply that Wolfram Alpha may have an equally good or faster adoption rate in math than other CAS
technology as suggested by Anderson (2009). More specifically, the program is free of charge (relative advantage), quite similar to con-
ventional search engines (compatibility), can be used on any internet-connected PC (trialability), learners are able to get a holistic view
through simple searches (complexity) and sharing results is easier and more compatible with the current online social networks
(observability). Thus, further research with larger samples in different contexts may be helpful to retain these promising implications.
Contemporary mathematics instruction has transformed through the advances in web technologies, which provided users with new
alternatives such as contacting the developers for fine-tuning the activities, providing instant and effective feedback or conceptualizing
mathematical content through more effective materials. In addition, advances in web design and programming languages provide better
versions of the current materials. The current study provided both qualitative and quantitative support for the use of semantic search
engines for mathematics instruction, which required active and cognitively more complicated participation of the learners. That is, active
search strategies employed by the learners seemed to contribute to achievement more than linear web pages or conventional software
support. This implication can be further supported through the theoretical framework (e.g., Piaget, 1971) and recent empirical studies
resorting to ‘learning by doing’ (Faraco & Gabriele, 2007; Gabriele et al., 2012; Jara et al., 2011; Tiernan, 2010), since engaging learners in
interactive activities and learning by doing enhanced students’ learning efficiency and helped them acquire relevant skills better.
Qualitative findings revealed that all interventions had strengths and limitations. Integrating multiple search criteria to instruction
helped learners in the semantic search group to grasp the equations and coefficients better. On the other hand, they still asked for further
details regarding diagram drawing. Similarly, the CAS group saw diagrams through manipulating commands, but had troubles in creating or
selecting the right command. Finally, the WSI group needed further chances to manipulate the diagrams. The majority of such limitations
stem from the nature of the interface. Thus, current findings may help developers to fine-tune the programs in a way that complex materials
are presented through less extraneous cognitive load on learners (Sweller, 2008) and through increasing the user-friendliness of the
interface (Davis, 1989; Venkatesh et al., 2003).
A mutual problem revealed in the qualitative analysis was the lack of sufficient language background to operate the programs effectively.
Thus, either the number of field-related English courses should be increased in the curriculum or the current interfaces should be provided
in learners’ native languages. Further analysis revealed that learners’ English levels were not contaminating the current dependent vari-
ables. Thus it was not included as a covariate. However, qualitative findings underlined the importance of English use in operating the
programs. Moreover, learners complained about the lack of adequate Turkish resources addressing multivariate functions whereas they
believed that there were sufficient resources on univariate functions.
A methodological caution regarding the current study was that the increase in the post-test might have been due to pre-test
sensitization rather than the unique affordances of the intervention. Such a probability was relatively week in terms of the achieve-
ment variable since the semantic search engine use led to significant performance increase in comparison to other interventions even
though all groups were pre-tested. However, the increase in attitude and self-efficacy scores might be explained through the testing
history. In this regard, empowering the current research can be possible through the addition of intervention groups without pretesting
and adding pretested groups to the design without any intervention. That is, an extension of the Solomon four-group design (Solomon,
1949) to current interventions can lead to more robust and generalizable outcomes. Furthermore, replicating similar designs with other
topics or in other courses can help scholars to see the influences of semantic search engines on achievement, attitude and self-efficacy.
Finally, each technology implemented in the current study has mobile versions. Preparing the course curricula in terms of learning
outcomes and transferring relevant contents to web environments may facilitate further implementations and empower fruitful learning
experiences. In addition, rather than providing a command list for all topics, preparation of supplementary handouts for each learning
outcome may increase the use and effectiveness of these programs.

Acknowledgment

This study is based on the first author’s PhD dissertation, which was conducted through the financial support provided by Anadolu
University Research Fund (Project ID: 1210E162).

References
Aksoy, Y. (2007). The effect of computer algebra systems in teaching the concept of derivative. Unpublished Doctoral Dissertation. Ankara: Gazi University Graduate School of
Educational Sciences.
 M. Ersoy, Y. Akbulut / Computers & Education 75 (2014) 253–262 261

Aktumen, M. (2007). The effect of computer algebra systems in teaching the concept of definite integral. Unpublished Doctoral Dissertation. Ankara: Gazi University Graduate
School of Educational Sciences.
Andersen, M. H. (2009). Why WjA will have a faster adoption rate in math than other CAS technology. Retrieved December 17, 2013 from http://teachingcollegemath.com/files/
pdf/rate_of_adoptions.pdf.
Berners-Lee, T. (2003). Semantic Web status and direction. ISWC 2003 Opening Speech. Retrieved December 17, 2013 fromhttp://www.w3.org/2003/Talks/1023-iswc-tbl/
slide26-0.html.
Catell, R. B. (1966). The scree test for number of factors. Multivariate Behavioral Research, 1, 245–276.
Choo, C. W., & Marton, C. (2003). Information seeking on the Web by women in IT professions. Internet Research, 13(4), 267–280.
Cohen, J. W. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Cristea, A. I. (2004). What can the Semantic Web do for adaptive educational hypermedia? Educational Technology & Society, 7(4), 40–58.
David, C., Ginev, D., Kohlhase, M., & Corneli, J. (2010). eMath 3.0: Building blocks for a social and semantic Web for online mathematics & E-Learning, 1st International Workshop
on Mathematics and ICT: Education, Research and Applications. Retrieved August 28, 2013 from http://kwarc.info/kohlhase/papers/malog10.pdf.
Davis, F. D. (1989). A technology acceptance model for empirically testing new end user information systems: theory and results. Cambridge, MA: Sloan School of Management,
Massachusetts Institute of Technology. Unpublished Doctoral Dissertation.
Ellis, P. D. (2010). The essential guide to effect sizes: An introduction to statistical power, meta-analysis and the interpretation of research results. United Kingdom: Cambridge
University Press.
Faraco, G., & Gabriele, L. (2007). Using LabVIEW for applying mathematical models in representing phenomena. Computers & Education, 49, 856–872.
Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: Sage.
Fogg, B. J. (2003). Persuasive technology: Using computers to change what we think and do. San Francisco, CA: Morgan Kaufmann Publications.
Gabriele, L., Tavernise, A., & Bertacchini, F. (2012). Active learning in a robotics laboratory with university students. In C. Wankel, & P. Blessinger (Eds.), Cutting-edge tech-
nologies in higher education (pp. 315–339). UK: Emerald Group Publishing Limited.
George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th ed.). Boston, MA: Allyn & Bacon.
Godarzi, S. Q., Aminifar, E., & Bakhshalizadeh, S. (2009). The impact of using computer algebraic system (CAS) in teaching and learning of “double integral”. In Proceedings of
the Third International Conference on Science and Mathematics Education (CoSMEd), Penang, Malaysia. Retrieved on July 31, 2013 from http://www.recsam.edu.my/cosmed%
202013/cosmed09/AbstractsFullPapers2009/Abstract/Mathematics_Parallel_PDF/Full_Paper/M30.pdf.
Gravetter, F. J., & Wallnau, L. B. (2000). Statistics for the behavior sciences. Belmont, CA: Wadsworth/Thomson.
Green, S. B., & Salkind, N. J. (2007). Using SPSS for Windows and Macintosh: Analyzing and understanding data. Upper Saddle River NJ: Pearson, Prentice Hall.
Harel, I., & Papert, S. (1990). Software design as a learning environment. Interactive Learning Environment, 1(1), 1–32.
Hoyle, R. H. (1995). Structural equation modeling: Concepts, issues and applications (1st ed.). CA: SAGE Publications.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structural analysis: conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–
55.
Huck, S. W. (2000). Reading statistics and research (3rd ed.). New York: Addison Wesley Longman.
Huck, S. W. (2008). Reading statistics and research (5th ed.). Boston: Pearson.
Huck, S. W. (2012). Reading statistics and research (6th ed.). Boston: Pearson.
Ijsselsteijn, W., de Kort, Y., Midden, C., Eggen, B., & van den Hoven, E. (2006). Persuasive technology for human well-being: setting the scene. Persuasive, 2006, 1–4.
Jara, C. A., Candelas, F. A., Puente, S. T., & Torres, F. (2011). Hands-on experiences of undergraduate students in automatics and robotics using a virtual and remote laboratory.
Computers & Education, 57, 2451–2461.
Jonassen, D. H., Cernusca, D., & Ionas, I. G. (2007). Constructivism and instructional design: the emergence of the learning sciences and design research. In R. A. Reiser, &
J. V. Dempsey (Eds.), Trends and issues in instructional design and technology (pp. 45–52). Upper Saddle River, NJ: Pearson Education.
Jones, K., Geraniou, E., & Tiropanis, T. (2013). Patterns of collaboration: towards learning mathematics in the era of the semantic web. In D. Martinovic, V. Freiman, &
Z. Karadag (Eds.), Visual mathematics and cyberlearning (pp. 1–22). London: Springer.
Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39, 31–36.
Kline, P. (1994). An easy guide to factor analysis. London: Routledge.
Kline, R. B. (2005). Principles and practice of structural equation modeling. NY: Guilford Publications.
Liaw, S., & Huang, H. (2003). An investigation of user attitudes toward search engines as an information retrieval tool. Computers in Human Behavior, 19, 751–765.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). California: Sage Publications.
Mintz, J., & Aagard, M. (2012). The application of persuasive technology to educational settings. Educational Technology Research and Development, 60, 483–499.
Nolen, A. L. (2009). The content of educational psychology: an analysis of top ranked journals from 2003 through 2007. Educational Psychology Review, 21, 279–289.
Oinas-Kukkonen, H., & Harjumaa, M. (2008). A systematic framework for designing and evaluating persuasive systems. Proceedings of Third International Persuasive Technology
Conference (Persuasive 2008) (pp. 164–176). Heidelberg: Springer.
Ozcelik, D. A. (1997). Test hazırlama kılavuzu [Test development guide] (3rd ed.). Ankara: ÖSYM Educational Publications.
Pallant, J.. (2007). SPSS survival manual. Maidenhead, PA: Open University Press.
Papasalouros, A., Retalis, S., & Papaspyrou, N. (2004). Semantic description of educational adaptive hypermedia based on a conceptual model. Educational Technology & Society,
7(4), 129–142.
Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York: Basic Books.
Papert, S. (1986). Constructionism: A new opportunity for elementary science education. Cambridge, MA: MIT, Media Laboratory, Epistemology and Learning Group.
Piaget, J. (1971). L’epistemologia genetica. Roma-Bari: Laterza.
Piaget, J., & Inhelder, B. (1966). La psychologie de l’enfant. Paris: P.U.F.
Raykov, T., & Marcoulides, G. A. (2006). A first course in structural equation modeling (2nd ed.). New York: Lawrence Erlbaum Associates.
Resnick, M. (1989). LEGO, logo, and life. In C. Langton (Ed.), Artificial life (pp. 397–406). Redwood City, CA: Addison-Wesley.
Rogers, E. (2003). Diffusion of innovations (5th ed.). New York: Free Press.
Ross, S. M., & Morrison, G. R. (1996). Experimental research methods. In David H. Jonassen (Ed.), Handbook of research for educational communications and technology: A project
of the Association for Educational Communications and Technology (pp. 1213–1245).
Ross, S. M., Morrison, G. R., & Lowther, D. L. (2010). Educational technology research past and present: balancing rigor and relevance to impact school learning. Contemporary
Educational Technology, 1(1), 17–35.
Rouet, M. M., Ney, M., Charles, S., & Lallich-Boidin, G. (2009). Students’ performance and satisfaction with Web vs. paper-based practice quizzes and lecture notes. Computers &
Education, 53, 375–384.
Sahin, I._ (2009). Eg
itsel internet kullanımı öz-yeterlig i inançları ölçeg
inin geçerlig
i ve güvenirlig
i [Validity and reliability of educational Internet use self-efficacy beliefs scale].
Selcuk University Journal of Social Sciences Institute, 21, 461–471.
Simons, H. W., Morreale, J., & Gronbeck, B. (2001). Persuasion in society. Thousand Oaks London New Delhi: Sage Publications.
Simsek, Ö. (2010). The effect of video lectures used in web enhanced mathematics instruction on students’ derivative achievement and views of students about the video lectures.
_
Unpublished Master’s Thesis. Izmir: Ege University, Graduate School of Natural and Applied Sciences.
Solomon, R. L. (1949). An extension of control group design. Psychological Bulletin, 46, 137–150.
Straub, E. T. (2009). Understanding technology adoption: theory and future directions for informal learning. Review of Educational Research, 79(2), 625–649.
Sumer, N. (2000). Yapısal eşitlik modelleri [Structural equation models]. Turkish Psychological Articles, 3(6), 49–74.
Surry, D. W., & Ely, D. P. (2007). Adoption, diffusion, implementation, and institutionalization of instructional innovations. In R. A. Reiser, & J. V. Dempsey (Eds.), Trends and
issues in instructional design and technology (2nd ed.) (pp. 104–111). Upper Saddle River, NJ: Prentice Hall.
Sweller, J. (2008). Human cognitive architecture. In J. M. Spector, Md. D. Merrill, J. V. Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications
and technology (3rd ed.) (pp. 369–381). New York: Lawrence Erlbaum Associates.
Sweller, J., van Merriënboer, J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296.
Tabachnick, B. G., & Fidel, L. S. (2001). Using multivariate statistics (4th ed.). MA: Allyn & Bacon.
Thompson, B. (2008). Exploratory and confirmatory factor analysis. Washington, D.C.: American Psychology Association.
Tiernan, P. (2010). Enhancing the learning experience of undergraduate technology students with LabVIEWÔ software. Computers & Education, 55, 1579–1588.
Tokpah, C. L. (2008). The effects of computer algebra systems on students’ achievement in mathematics. Unpublished Doctoral Dissertation. OH: Kent State University Graduate
School of Education.
262 M. Ersoy, Y. Akbulut / Computers & Education 75 (2014) 253–262

Ullrich, C., & Melis, E. (2010). Complex course generation adapted to pedagogical scenarios and its evaluation. Educational Technology & Society, 13(2), 102–115.
Venkatesh, V., Morris, M., Davis, G., & Davis, F. (2003). User acceptance of information technology: toward a unified view. MIS Quarterly, 27(3), 425–478.
Wilson, T. D. (1997). Information behavior: an interdisciplinary approach. Information Processing and Management, 33(4), 551–572.
Yildirim, A., & Simsek, H. (2006). Sosyal bilimlerde nitel araştırma yöntemleri [Qualitative research methods in social sciences]. Ankara: Seçkin Publications.
Yilmaz, V., & Celik, H. E. (2009). LISREL ile yapısal eşitlik modellemesi-I [Structural equation modeling with LISREL-1]. Ankara: Pegem Academy Publishing.
Zhu, Y., Chen, L., Chen, H., & Chern, C. (2011). How does internet information seeking help academic performance? The moderating and mediating roles of academic self-
efficacy. Computers & Education, 57, 2476–2484.
Zou, X. (2005). Support for online mathematics education: MeML and WME services. In Proceedings of SoutheastCon, IEEE 2005 (pp. 656–662).