Computers and Education: Artificial Intelligence 4 (2023) 100119

Contents lists available at ScienceDirect

Computers and Education: Artificial Intelligence

journal homepage: www.sciencedirect.com/journal/computers-and-education-artificial-intelligence

Pre-service teachers’ perceptions and intentions regarding the use of

chatbots through statistical and lag sequential analysis
Tzu-Chi Yang *, Jian-Hua Chen
Institute of Education, National Yang Ming Chiao Tung University, 1001 University Road, Hsinchu, 300, Taiwan

A R T I C L E I N F O A B S T R A C T

Keywords: Chatbots provide unique interactions with compatible learning system features, improving the limitations of
Educational chatbot current learning systems. Educational chatbots are seen as the future of technology integration in the field of
Pre-service teacher education. The success and usefulness of chatbots in the educational setting are highly dependent on teachers’
Learning behavior analysis
beliefs regarding their efficacy, yet most research focuses on the effects on students’ learning. Only a few studies
Technology enhanced learning
have investigated teachers’ beliefs regarding the use of chatbots, which is considered an important issue. Owning
to teachers’ beliefs having been transformed from their pre-service teacher training, this study used quantitative
(i.e., questionnaires), qualitative (i.e., interview), and evidence-based (i.e., behavioral analysis) methods to
investigate pre-service teachers’ learning perceptions and intentions about using chatbots for learning during
their training phases. The results of this study revealed that learning perceptions did not reflect pre-service
teachers’ propensity to use chatbots, but the behavioral analysis uncovered some specific intentions for using
chatbots. We further discuss these findings to provide recommendations for the future development of chatbots
use in education.

1. Introduction related to student–system interactions (Prior et al., 2016). Students’

The development of digitalization and information networks has
changed almost every mode of interaction, and it has also produced
significant changes in the field of education. As technology advances,
more and more of the complex tasks that used to be part of the teaching
and learning process are being improved (Ng’ambi et al., 2016). For
example, generating real-time feedback gives students the opportunity
to reflect immediately, thus enhancing their learning effectiveness
(Khan & Jawaid, 2020). As today’s educational models evolve with
advances in learning technology, technology plays a critical role in
facilitating teaching and learning practices. Its use to assist teaching and
learning has become one of the most essential issues in the last decade
(Naidu, 2017), and various learning management systems (LMSs) that
facilitate teaching and learning continue to be proposed (Shen & Ho,
2020), including enhancing learning effectiveness, improving the limi­
tations of physical learning, reducing teaching load, or enhancing
existing learning methods (Shen & Ho, 2020; Pardo, Han, & Ellis, 2016).
However, researchers have noted that while we can significantly
improve learning and teaching through learning technologies, different
forms and scales of learning systems will generate a range of issues
attitudes, emotional states, and learning experiences with these learning
technologies directly affect their learning processes and outcomes (Ali,
2020; Kemp et al., 2019). Most LMSs are based on certain rules, pro­
cedures, and functional designs, and usually provide standardized
learning resources and methods of operation, which limit the use and
interaction, making the interaction pattern between students and the
system relatively fixed. This lack of flexibility reduces the frequency of
student use (Shim & Lee, 2020), which is related not only to whether the
learning system delivers the expected benefits (Czerkawski & Lyman,
2016; Tarhini et al., 2017), but also to student learning outcomes
(Dumford & Miller, 2018). Therefore, how to sustain students’ interest
and motivation to interact with the system becomes a new issue (Hsu &
Wang, 2018; Osei et al., 2022; Saltan, 2017).
To address this issue, the development of educational chatbots has
received a lot of attention. This is because, on the one hand, chatbots not
only can deliver learning content, but also provide support for students
alike LMS. On the other hand, chatbots mimic human conversations and
make the system look like a human, providing a more natural way of
interaction (Barak & Assal, 2018; Yin et al., 2021). More importantly,
chatbots are not limited to a fixed process; they can simulate human

* Corresponding author.
E-mail addresses: tcyang.academic@gmail.com (T.-C. Yang), yangsung0859636.ie08g@nctu.edu.tw (J.-H. Chen).

https://doi.org/10.1016/j.caeai.2022.100119
Received 30 June 2022; Received in revised form 18 December 2022; Accepted 18 December 2022
Available online 20 December 2022
2666-920X/© 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
T.-C. Yang and J.-H. Chen
conversations and provide a different learning experience, helping to
overcome the inflexibility of learning systems and improve the percep­
tion of student interaction with the system (Chaves & Gerosa, 2021).
Educational chatbots are gaining attention as one of the key de­
velopments in the future of technology-enhanced learning (Chang et al.,
2022; Mendez et al., 2020). Nevertheless, it should be noted that the
chatbot’s implementation relies on the content and support provided by
domain experts, and the guidelines provided by the experts are directly
related to the effectiveness of chatbots (Kuhail et al., 2022). In educa­
tional settings, teachers are at the heart of the content and learning
activities and are key to constructing educational chatbots (Fidan &
Gencel, 2022). Teachers’ perceptions of chatbots may be reflected in
how they provide content, how engaged, they are, and how they actually
use the technology. If teachers believe that chatbots can improve student
learning, they are likely to build chatbots with learning enhancement as
an expected goal and try to integrate them into their instruction (Van
Katwijk et al., 2021a). Conversely, teachers may not adopt chatbots if
they do not think doing so can help them learn. In other words, teachers’
beliefs about using chatbots are essential to the effectiveness of educa­
tional chatbots.
However, few studies have attempted to investigate teachers’ beliefs
about adopting such a method in their instruction, with the majority
using questionnaires to obtain teachers’ perceptions of chatbots. This
might have made the results biased (Bii et al., 2018; Hsu et al., 2017;
Thurm & Barzel, 2020). Thus, more research is necessary to truly un­
derstand teachers’ beliefs regarding using such emerging instructional
technology (Huang et al., 2022). Investigating pre-service teachers’
beliefs might address this issue because teachers’ familiarity with and
beliefs about instructional technology are largely derived from their
pre-service learning experiences (Clawson, 1996; Van Katwijk et al.,
2021b). Furthermore, pre-service teachers’ notions about the use of
educational technology are reflected in their learning perceptions and
intentions to use educational technology (Bower et al., 2020; Wang,
2002). Understanding how teachers perceive and intend to use chatbots
during their pre-service training may help uncover additional directions
for enhancing the development of educational chatbots (Papadakis
et al., 2021). In this vein, this study aimed to analyze pre-service
teachers’ perceptions and intentions regarding use of chatbots in their
training phases. The specific research questions included:
RQ1: Are preservice teachers’ learning perceptions related to the use of
chatbots?
RQ2: How do pre-service teachers use chatbots during their training
phases?
In order to accomplish this, we conduct an empirical study and
quantitative (i.e., questionnaires), qualitative (i.e., interview), and
evidence-based (i.e., behavioral analysis) data were considered to obtain
a solid evidence of learning behavior analysis to answer the research
questions. The structure of this study is organized as follows. Section 2
reviews the related studies, while Section 3 describes the experimental
design in detail. Section 4 provides the results and discussion, while
Sections 5 and 6 summarize the findings and present the limitations and
future research recommendations, respectively.
2. Literature review
2.1. The roles of chatbots in the educational settings
A chatbot is an information application based on artificial intelli­
gence that can infer the meaning of human language through natural
language processing (NLP) techniques and respond appropriately ac­
cording to the meaning (Reshmi & Balakrishnan, 2016). In recent years,
due to the advancement of NLP technologies, chatbots have evolved
from simple pattern matching to coping with increasingly complex in­
teractions and reasoning, even replacing some of the conversations with
2
Computers and Education: Artificial Intelligence 4 (2023) 100119
real people. Users expressed satisfaction and positive attitudes towards
the functionality, convenience, entertainment, attractiveness, and so­
cialization of chatbots (Shum et al., 2018). The advantages of chatbots
have attracted a lot of attention in the education field. Researchers point
out that most learning systems are mostly fixed-format interfaces, which
limit flexibility and interactivity in use. At the same time, the formatted
operations and responses tend to distract students and lead to lower
engagement (Serrano et al., 2019). Researchers have also found a cor­
relation between the immediacy of teachers’ instruction and students’
emotions about learning (Camacho-Zuñiga et al., 2021; Liu et al., 2019;
Witt et al., 2014). The sense of authenticity and intimacy in verbal and
nonverbal interactions with teachers is difficult to find in interactions
with learning systems (Short et al., 1976). Unlike LMSs, chatbots can be
user- or machine-directed, providing various conversational experi­
ences. Such a natural conversational experience also allows students to
feel social connection (Alotaibi et al., 2020; Kowald & Bruns, 2020).
Even if the vocabulary of chatbots is not as rich, the ease of conversation
and sense of engagement make people interested and willing to talk to
chatbots (Fryer et al., 2019).
Moreover, chatbots can provide interactive messages similar to
conversations with a human to prompt learners to access learning re­
sources (Hope, 2020; Polasek & Javorcik, 2019). This allows chatbots to
act like peers or partners in the learning process of students (de Las
Heras et al., 2020). Many scholars have found that naturalistic conver­
sation with chatbots can reduce the isolation and disconnection that
students experience in the traditional LMS (Wu et al., 2020). Learning
with chatbots is a novelty and interesting, and students often engaged in
learning with chatbots (Alm & Nkomo, 2020; Fryer & Carpenter, 2006).
Some students even enjoy chatting with chatbots more than with
teachers or peers (Fryer et al., 2019; Yin et al., 2021). Chatbots not only
allows students to feel socially engaged but also serves as a medium of
instruction that improves student performance and motivation (Car­
ayannopoulos, 2018; Polasek & Javorcik, 2019; Jeon, 2022).
Thanks to various advantages, more and more studies are consid­
ering chatbots as an LMS to assist students’ learning (Pérez et al., 2020).
For example, Pereira et al. (2019) used chatbots to facilitate student
interactions in online courses, and their findings indicated that chatbots
not only increase the percentage of students completing learning tasks,
but 90% of students wanted to use chatbots in other courses as well. Oh
(2020) used chatbots to promote anti-bullying activities to students and
showed that students’ attitudes changed to a more positive response
when faced with bullying after using chatbots, which was one of the
expected goals. Some scholars have also suggested that the use of
chatbots can yield better results than traditional LMS (Pereira et al.,
2019; Wu et al., 2020). In this vein, the application of chatbots in the
field of education is gaining great attention and is considered an
important development trend (Berger et al., 2019; Tamayo et al., 2020).
2.2. Pre-service teachers’ learning experiences and technology integration
in teaching
Scholars are beginning to note that teachers are an essential medium
for transferring knowledge and skills and are the key to integrating
learning technologies (Saltan & Arslan, 2017). Even though chatbots
have demonstrated many benefits, teachers’ beliefs and perceptions are
the key factor in achieving the expected outcomes of using bots in ed­
ucation that cannot be ignored (Choi et al., 2022). However, the use of
questionnaires to investigate teachers’ beliefs about the use of technol­
ogy has been limited. For example, Ha and Lee (2019) designed ques­
tionnaires to investigate elementary school teachers’ self-efficacy on
their knowledge of technology and their use of technology in the
classroom, suggesting that only using questionnaires to obtain percep­
tions may have potential bias. In related research, van Twillert et al.
(2020) used questionnaires to obtain university teachers’ beliefs about
integrating Web 2.0 into their pedagogy, including belief-based mea­
sures, behavioral intention, attitude, perceived norms, and perceived
T.-C. Yang and J.-H. Chen
behavioral control. The participants in their study reported that the
survey was both difficult and lengthy. More recently, Hoareau et al.
(2021) used a self-developed questionnaire to investigate preschool
teachers’ beliefs about the use of digital technology in teaching. They
found that the results of the questionnaire could not be applied as a
whole, after considering the specificity of the instrument.
Scholars further indicated that teachers’ beliefs about using of
technology are highly relevant to their learning experiences and
perception in pre-service teacher learning (Teo, 2009). Pre-service
teachers’ beliefs and perceptions during their learning phase directly
reflect their performance in future teaching practices in the classroom
(Istenic et al., 2021; Kesicioglu, 2021; Van Katwijk et al., 2021). In
addition, pre-service teachers are at a critical stage for developing skills
and knowledge and realizing how to appropriately integrate learning
technologies in educational settings (Valtonen et al., 2020). Paying
attention to teachers’ perceptions of technology during their pre-service
teacher training (student) phase can not only help pre-service teachers
cope with challenges and difficulties in practicing instruction (Sorge
et al., 2019), but also ensure that pre-service teachers acquire the
adequate literacy of integrating information and technology into
teaching and learning (Dincer, 2018; Ye et al., 2021; Zhang & Chiu,
2021). In today’s environment of rapid expansion of various learning
technologies, it is even more important to focus on pre-service teachers’
education and their technology-assisted learning experiences (Islamoglu
et al., 2021). Even though there is no consensus on whether teachers’
beliefs change, most researchers agree that pre-service teachers’ beliefs
directly predict their beliefs to practice technology integration in
teaching after graduation (Liu, 2011; Ndlovu et al., 2020). Hence, un­
derstanding pre-service teachers’ beliefs of using learning technology
will enable educators to grasp more realistically how to integrate tech­
nology in education to meet future educational expectations for
technology-assisted teaching and learning (Bardakci & Unver, 2020;
Yalman & Basaran, 2021).
Beliefs are part of teachers’ thinking that are conscious/unconscious
thoughts about one’s role in the process and are influential in how in­
dividuals organize their tasks (Thompson, 1992). Although beliefs are
not directly observable, it is generally believed that teachers’ beliefs
about integrating technology into their teaching can be derived from
their perceptions and intentions (Wombacher et al., 2018). Perceptions
are positive or negative evaluations that reflect overall feelings and are
related to a teacher’s decision about whether or not they are willing to
use technology during instruction, which can be evaluated by using a
questionnaire. Intentions are assumed to be an antecedent to behavior
and reflect a teacher’s beliefs, and these can be obtained by observing
their learning behavior, e.g., interacting with an LMS or a chatbot (Borg,
2001; Pierce & Ball, 2009; Summers et al., 2017; Tan & Laswad, 2006).
In this vein, this study attempts to adopt self-reported questionnaires,
interviews, and system usage logs to uncover their learning perceptions
and intentions regarding learning with a chatbot. The detailed study
design is described below.
3. Experimental design
3.1. Participants
The participants of this study were 26 pre-service teachers in the
field of information technology at a national university in northern
Taiwan, and the subjects were also third-year students with information
and technology majors. All participants have certain information skills
that allow them to operate the LMS and chatbot. Moreover, all partici­
pants were enrolled in the same information course during the experi­
ment process, and they were free to use the LMS and chatbot provided by
the study. Based on the collected logs, we classified the participants with
less than 10 interactions with chatbots as those who did not intend to
learn with chatbots (tNbot) and the participants with more than 10 in­
teractions with chatbots as those who intended to learn with chatbots
3
Computers and Education: Artificial Intelligence 4 (2023) 100119
(tTbot). Eventually, 10 participants were in the tNbot group and 16
participants were in the tTbot group.
3.2. The design of LMS and chatbot
Previous studies have shown that LMSs, in addition to serving as a
medium for accessing learning content, provide personal annotation, as
well as information retrieval mechanisms that can further help students
organize information and thus enhance learning effectiveness (Yang
et al., 2018). Therefore, this study provides students with an LMS that
supports learning content access, personal notation, and information
retrieval. In addition, it is also important for students to respond to the
system and provide feedback, such as whether they find the material at
hand difficult or easy, and we take this into account. To summarize, we
adopted Bookroll, a free system developed by Kyoto University in Japan
and used in many universities, as the provided LMS (Shimada et al.,
2018). Fig. 1 shows an example of the provided LMS.
The chatbots, constructed by this study, provide the three learning
supports mentioned above (i.e., learning content access, personal no­
tation, and information retrieval). A main difference with the LMS is that
the chatbot cannot perform some notation such as drawing and book­
marking. We use a navigation menu as an alternative to personal nota­
tion (e.g., to follow weekly progress and lesson highlights). Moreover, to
reinforce the characteristics of chatbots, the provided chatbots respond
not only to course content but also to non-target-oriented conversations
(Bibauw et al., 2022), e.g., everyday conversations, Internet memes,
etc., which will make the chatbot more like learning partners (Boot­
h-Butterfield & Wanzer, 2010, p. 221; Wu et al., 2020). Fig. 2 demon­
strates an example of provide chatbot. This study includes iOS, Android,
and web versions of the chatbots. This provided the same interface and
content for all three platforms so that the students could use their own
devices to interact with the chatbots. Due to the inherent differences, the
LMS includes more robust personal annotation features (for website
support), while the chatbots allow for more natural conversational in­
teractions. Nevertheless, both the LMS and chatbots provide three basic
supports related to learning: learning content access, personal notation,
and information retrieval.
3.3. Research instruments
In order to investigate the participants’ perspectives of learning with
LMS- and chatbot-assisted learning, internal and external motivation,
test anxiety, and self-efficacy, learning attitudes, were considered in this
study. Motivation stimulates behavioral intention, which can be further
classified as extrinsic or intrinsic motivation. Self-efficacy and test
anxiety are generally considered to be common factors in the composi­
tion of motivation (Chang, 2005). The Motivated Strategies for Learning
Questionnaire (MSLQ) considers beliefs, motivations, perceptions, and
the relationship among them, it is often used to investigate students’
overall perceptions of the course, as well as their perceptions of the use
of technology in their learning (Pintrich, 1991; An & Reigeluth, 2011;
Trigwell et al., 2013). Moreover, the MSLQ has been used by numerous
researchers worldwide, due to its high validity (Duncan & McKeachie,
2005). Hence, we accordingly considered the MSLQ as the research in­
strument. Researchers have found that the overall perception of learning
is also related to learning attitudes (Gómez-Carrasco et al., 2019; Lin,
2019). Thus, we adopted the learning attitudes questionnaire from
Hwang et al. (2013), which has been widely used to access students’
learning experiences in a technology-assisted learning environment, due
to its high validity. Briefly, we obtained the participants’ learning per­
ceptions, including internal and external motivation (e.g., “The most
satisfying thing for me in this course is trying to understand the content
as thoroughly as possible” and “If I can, I want to get better grades in this
class than most of the other students”), test anxiety (e.g., “I have an
uneasy, upset feeling when I take an exam”), and self-efficacy (e.g., “I am
certain that I can master the skills being taught in this class”), and
T.-C. Yang and J.-H. Chen
Fig. 1. An example of the provided LMS.
Fig. 2. An example of provided chatbot.
learning attitudes (e.g., “It is important for me to learn the computer
architecture course well”). To objectively assess the participants’
behavioral patterns, we also recorded each click and conversation on the
LMS and chatbots. In addition, interviews were conducted to obtain
qualitative data on all of the participants’ perceptions. Specifically, the
interviews focused on why they did (or did not) use chatbots, how they
typically used chatbots, and their perceptions about continuing to use
such tools in the future. The interviews were limited to 10 min for each
participant.
3.4. Experimental procedure
This study was conducted during a 3-h-per-week course designed to
help students gain an in-depth understanding of computer architecture
and operating principles such as memory swapping, pipelining, and
instruction scheduling. The course described in this study is taught by a
professor who is affiliated with the Center for Teacher Education at a
university. This professor has a background in computer science and 3
4
Computers and Education: Artificial Intelligence 4 (2023) 100119
years of teaching experience.
The entire process lasted for 10 weeks, with the first four weeks using
traditional face-to-face (F2F) instruction without any system provided.
After the F2F instruction, an LMS (i.e., Bookroll) was used for three
weeks, which was the first phase. In this phase, the students could use
the LMS at any time, in addition to attending the F2F lectures. Subse­
quently, a pre-questionnaire was administered to assess the participants’
learning perceptions toward the LMS. A chatbot was also provided to the
participants in the second phase of the learning process for three weeks.
During this phase, in addition to attending the F2F lectures, all of the
participants were free to use either the LMS or chatbot (or both) at any
time. We conducted post-questionnaires and interviews with all of the
participants at the end of this phase.
3.5. Data analysis
In this study, quantitative data on the participants’ perceptions was
obtained from the questionnaires, qualitative data was obtained from
T.-C. Yang and J.-H. Chen Computers and Education: Artificial Intelligence 4 (2023) 100119

the interviews, and evidential data was obtained from system usage
records. Specifically, the quantitative data was statistically examined to
determine whether there was a significant difference in the perceptions
between the students who used chatbots and those who did not. More­
over, the logs were used to analyze the usage intentions by identifying
actual operations patterns, while the qualitative data from the in­
terviews was used to provide more evidence regarding the students’
perceptions and intentions.
4. Results and discussion
In this study, we identified the students who actively used the
chatbot (tTbot) and those who did not (tNbot) in an environment that
already provided an LMS. We also examined their learning perceptions
and behaviors in order to explore what characteristics might be associ­
ated with the desire to use the chatbot and what their intentions were for
using the chatbot. There were 58% (15) of the participants were tNbots,
while 42% (11 participants) were tTbots. Therefore, we compared the
differences in learning perceptions and behavioral patterns between the
groups.
not significant predictor of learners’ intent of learning the chatbot. In
other words, we have not been able to identify any relationship with
chatbot use from the aspect of learning perceptions. A possible reason
for this is that this study was conducted in a face-to-face course. The LMS
and chatbot were additional tools in which all of the participants pri­
marily interacted with the same instructor and peers in the physical
classroom. Hence, the students’ perceptions of learning were more
strongly connected to the physical course (F2F classroom), making their
perceptions about the course as a whole similar whether they decided to
use the chatbot. In other words, it may not be the perceptions of learning
from the course that ultimately motivated them to use the chatbot, but
rather other factors such as having a specific purpose (i.e., intent). Ac­
cording to one of the participants:
It is actually possible to read the learning materials, the content of which is
similar to what the teachers introduced in class. So, I will not use the
special tools, unless I cannot find the topic and need to search for it.
(Participant 10)
4.2. Learning behavior analysis

In terms of the learning behavior analysis, we collected the logs from

4.1. Learning perceptions
the provided LMS and chatbot, and categorized the behavior according
to the purpose of the operations (e.g., whether they were searching for
We compared participants’ learning experiences before and after
keywords in the LMS or chatbot, or whether they intended to obtain
incorporating the chatbot into the course to identify the relationship
information about the keywords). Hence, this behavior was coded as
between learning perceptions and chatbot use. For example, when a
“searching for content,” after which we conducted the coding schema.
student demonstrated high test anxiety in Phase 1 (only the LMS) and
This was also reviewed by a professor with expertise in learning
used the chatbot on his/her own in Phase 2 (LMS + chatbot), we may
behavior analytics. Overall, the coding schema categorized the learning
assume that test anxiety is related to the willingness to use the chatbot. A
behaviors into five codes: activating tools (I), accessing learning content
one-way multivariate analysis of variance (MANOVA) was performed to
(R), organizing understandings (O), searching for content (S), and non-
determine whether learning perceptions varied by the intention of using
course content interaction (T). At the same time, we generated a code
chatbot. First, the results of the Box’s test indicated that the assumption
scheme (Table 2) to objectively code and analyze the learning behavior
of homogeneity is met (F = 1.405, p > .05). The MANOVA results
patterns of participants’ when learning with these two learning assis­
(Table 1) show that there were no significant differences in either the
tance tools during the two phases. Since the participants who did not use
pre-questionnaires (Wilks’ Lambda = 0.646, F = 2.191, p > .05) or the
the chatbot (tNbot) hardly used it in the second phase (<10 in­
post-questionnaires (Wilks’ Lambda = 0.798, F = 1.010, p > .05) for
teractions), the learning behavioral analysis did not include this group’s
each dimension between the tNbot and tTbot groups. The reliability of
behavior patterns.
each dimension of learning perception was measured by Cronbach’s
The results (Table 3) of the behavior frequency analysis indicate that
alpha, while the reliability of the questionnaire ranged from 0.62 to 0.91
tTbot demonstrated a higher frequent interaction than those of tNbot in
(i.e., intrinsic motivation: 0.72; extrinsic motivation: 0.62; learning
both phases 1 and 2. This means that participants who are more willing
attitude: 0.78; self-efficacy: 0.91; and test anxiety: 0.82), which is
to interact with LMS are also likely to be willing to learn with chatbots.
acceptable. This result suggests participants’ learning perceptions were
This result is not surprising, as teachers/students who are willing to use
learning technologies are likely to be willing to try new technologies
Table 1
themselves (Foulger et al., 2021). We also found that the participants’
MANOVA results of participants’ learning perceptions of in each group.
activity allocation on the LMS was almost the same in both phases: the
Dimension Group N Mean S.D. F p main interaction was accessing the learning content (95%), followed by
Phase 1 searching for content. However, the tTbot group not only increased
Internal motivation tNbot 11 4.39 .86 0.27 .610 significantly in the proportion of activating chatbot (from 1.63% to
tTbot 15 4.2 .94
External motivation tNbot 11 4.64 .39 4.27 .050
tTbot 15 4.22 .58 Table 2
Learning attitudes tNbot 11 4.47 .53 0.02 .878 Coding schema.
tTbot 15 4.43 .63
Code Behavior Example of behavior Example of behavior to
Test anxiety tNbot 11 3.65 1.10 2.07 .163
to chatbot the LMS
tTbot 15 4.16 .69
Self-efficacy tNbot 11 3.70 1.05 1.90 .181 I Activating tools Open and talk to the Login to the system and
tTbot 15 4.14 .60 chatbot click on any function
Phase 2 R Accessing learning Click on a handout or Open a learning materiel
Internal motivation tNbot 11 4.41 1.14 0.44 .835 content video from the menu and click on the previous
tTbot 15 4.33 .69 or next page
External motivation tNbot 11 4.64 .68 3.63 .069 O Organizing Click on highlights Add/manage notes,
tTbot 15 4.07 .80 understandings from the menu, ask for bookmarks
Learning attitudes tNbot 11 4.45 .75 0.71 .406 weekly progress
tTbot 15 4.21 .71 S Searching for Search keywords of Search for keywords,
Test anxiety tNbot 11 4.13 1.07 2.20 .151 content content jump to specific page
tTbot 15 3.64 .60 T Non-course Chat with no specific Express emotions (e.g.,
Self-efficacy tNbot 11 4.09 1.19 0.01 .946 content interaction target, giving crying face, smiley face)
tTbot 15 4.07 .62 feedback to LMS, giving feedback

5
T.-C. Yang and J.-H. Chen Computers and Education: Artificial Intelligence 4 (2023) 100119

Table 3 Table 5
The result of behavior frequency analysis. The result of the sequential analysis of behavior demonstrated by tTbot.
I R O S T Given/Target I R O S T
a a a
Phase 1 – only LMS I 6.95 − 26.42 10.56 25.78 4.3a
tNbot 215 10209 102 155 3 (0.03%) R − 61.84 123.99a − 53.29 − 96.1 − 4.7
(2.01%) (95.55%) (0.95%) (1.45%) O 89.21a − 89.01 71.64a − 1.52 − 0.61
tTbot 616 39645 1034 1068 9 (0.02%) S − 1.19 − 85.11 1.12 138.72a − 0.55
(1.45%) (93.56%) (2.44%) (2.52%) T − 0.42 − 2.45 − 0.61 − 0.55 49.37a
Phase 2 – LMS + chatbot a
tNbot - 179 7710 76 (0.94%) 103 15 z > +1.96.
LMS (2.21%) (95.39%) (1.27%) (0.19%)
tTbot – 366 20504 1107 509 0 (0.00%) we conduct the behavioral transitions diagram (Fig. 3.) of the two
LMS (1.63%) (91.19%) (4.92%) (2.26%)
groups of participants at two stages.
tTbot - 243 120 1074 149 46
Chatbot (14.89%) (7.35%) (65.81%) (9.13%) (2.82%) The results show that the behavior patterns of tNbot and tTbot in the
first phase are almost identical – e.g., repeatedly accessing learning
content (R→R) and searching for content knowledge after activating the
14.89%), but also the main interaction on chatbot shifted to organizing tool (I→R). This is understandable, because in the first phase only the
understandings (65.81%) and searching for content (9.13%). This result LMS is provided and, therefore, the behaviors of all participants rely on
suggests that the chatbot may have been perceived by participants as a the functionality provided by the system. As a result, similar behavior
different learning tool because they generated different intentions when patterns are exhibited. However, in the second phase, both LMS and
using the chatbot than when using LMS. chatbot were provided for participants. The tNbot maintains almost the
We can reveal some insights from this result that even if there is no same behavior patterns as in phase 1, while tTbot exhibits many
significant difference in participants’ learning perceptions (via ques­ different behavior patterns. First, instead of non-course interactions
tionnaire), the learning behavior demonstrated by them can still reflect (I→T) in the LMS, tTbot repeated this behavior on chatbot (I→T, T→T).
their intention to use the chatbot. In other words, the behavioral analysis In other words, participants may shift non-course content interactions to
helped us to understand how pre-service teachers perceived these chatbot when the chatbot is available.
technologies in learning. Therefore, we further conducted a lag Such findings echo those of previous studies that chatbot simulated
sequential analysis (LSA) on the participants’ behavioral transition be­ human interactions may make users more willing to interact with them
tween the two stages in an attempt to obtain a deep understanding of the (Huang et al., 2022). Another finding was that tTbot showed some
pre-service teachers’ intention to learn with chatbots. learning behaviors that were not present in the LMS. For example, they
might tend to directly access learning content after activating the tool
(I→R) followed by searching for content (R→S) and possibly repeatedly
4.3. Lag sequential analysis
searching for content (R→R). One possible reason is that chatbot re­
sponds immediately to the user’s conversation without extended oper­
LSA, which is based on the development of a probabilistic theory, can
ations, which accommodates the needs of the participants in an efficient
effectively identify the specific behavioral patterns of participants
way. In other words, the convenience of accessing and searching
(Bakeman & Gottman, 1997). It is also widely used to understand the
learning content may be one of the intents why preservice teachers
differences in learning behavior patterns among learners in a
tended to use the chatbot.
technology-enhanced learning environment (Authors, 2020). Therefore,
The responses of participants from the interview provide evidence to
this study adopts LSA to identify the differences in learning behavior
support the aforementioned arguments. For example,
patterns exhibited by the two groups. We first coded the participants’
behaviors (Table 3). The GSEQ 5 software (Bakeman & Quera, 2011) "The advantage is that it is simple and practical, and I will be replied
was then used as a tool to measure the z-score value of each behavior to when selection option, rather than looking at the slides one by
sequence. A z-score greater than +1.96 indicates that this behavior one" (Participant 17)
sequence is statistically significantly more than the other behavior se­
"When I encounter a problem, it is easier to find the answer, and I can
quences. After identifying all significant behavioral sequences, we can
read the handout while doing a query to understand. I can ask it
generate a behavior transition diagram to compare the learning
questions without looking on the internet, … " (Participant 24)
behavioral patterns demonstrated by different groups. For example,
when a participant activates the tool (I), searches a keyword (S), and "It gives an explanation when you enter a term, but it seems that not
then searches another keyword (S), followed by adding an annotation all explanatory terms can be generated and the answer is not always
(O), this sequence of behaviors is coded as I S S, O. Thereafter, the given. The advantage is that it can reply quickly and it is easy to
z-score is used to examine whether the four behavioral sequences (i.e., review." (Participant 21)
I→S, S→S, S→O) achieve statistical significance. Tables 4 and 5 show the
We also found that the tTbot showed repeated behavior patterns of
z-scores of behavioral sequences for tNbot and tTbot, respectively, with
organizational understandings (O→ O) in both the LMS and chatbot.
significant sequences labeled. For example, both tNbot and tTbot tended
This result may imply that the chatbot may be considered an alternative
to organize their understandings after activating the tool (I→O); both of
to the LMS to some extent, or replace some interactions done using the
them also repeatedly tended to search for content (S→S). Accordingly,
LMS. The responses from participants in the interview provide further
evidence. For example,
Table 4
The result of the sequential analysis of behavior demonstrated by tNbot. "You can use it at odd hours because the learning material is like a
professor [posting on] an LMS that needs to be logged into, and the
Given/Target I R O S T
chatbot can be used at odd hours on the bus or during commuting
I 12.87a − 24.45 16.22a 12.75a 5.4a time, with a cell phone. (Participant 22)
R − 38.48 61.22a − 11.58 − 48.91 − 7.04
O 65.1a − 40.79 − 1.05 − 1.38 − 0.25 "In addition to the key points of the lesson, there are also words that
S − 0.84 − 42.04 2.3a 67.64a − 0.33
will tell you what they mean if you type them directly. It’s easier to
T 2.6a − 5.18 − 0.25 − 0.33 33.99a
a
z > +1.96.

6
T.-C. Yang and J.-H. Chen
Fig. 3. The transition diagram of tNbot and tTbot in two phases.
find the key points, the read-through is from LMS, and the key points
are from the chatbot. (Participant 4)
"I think it’s not bad. I think it’s not bad, it’s better for organizing high­
lights. (Participant 3)
The aforementioned results suggest that, compared to obtaining their
perception via using questionnaires, learning behavioral analysis pro­
vides us with more insight into preservice teachers’ intention to learn
with chatbots. Specifically, preservice teachers may see chatbots as a
tool for "organizing understandings" and "searching for content" at the
student level because of the convenience and immediate response.
Meanwhile, they may also engage in "non-course interactions" with
chatbots because of the novelty.
These findings imply that in the future, they are likely to start with
these intentions when they become teachers to help develop chatbots or
adapt chatbots to teaching. Accordingly, we will conclude our findings
in the following section.
5. Conclusion
The development and implementation of educational chatbots rely
on the content and guidelines provided by teachers, and The effective­
ness of integrating chatbots into instruction is strongly related to
teachers’ perceptions and intentions, because these perceptions and
intentions are transformed through their learning experiences in pre-
service teacher training. It is important and valuable to understand
how pre-service teachers perceive learning with chatbots. To this end,
we conducted an experiment to answer two research questions via
obtaining quantitative (i.e., questionnaires), qualitative (i.e., in­
terviews), and evidence-based (i.e., learning behavior analysis) data.
Regarding RQ1: Are preservice teachers’ learning perceptions related to
the use of chatbots? We found that the perceptions of learning investi­
gated by the questionnaire do not currently reflect the pre-service
teachers’ tendency to use chatbots. Regarding RQ2: How do pre-
service teachers use chatbots during their training phases? The results
of this study indicated that to "access learning content" is the major
intention of the preservice teacher to learn with the LMS. Conversely,
the main intention of preservice teachers in using chatbot during the
7
Computers and Education: Artificial Intelligence 4 (2023) 100119
learning process is to "organize understandings" and "search for content."
This may be due to the convenience and novelty interaction of the
chatbot. At the same time, they also had a higher frequency of activating
chatbots and engaging in "non-course content conversations."
The current study makes several contributions to the understanding
perception and intent of preservice teachers on learning with chatbots.
a) The questionnaire does not fully reflect the pre-service teachers’
propensity to use chatbots. b) Behavioral analysis can effectively reveal
pre-service teachers’ intention to learn with chatbots. c) Organizing
understandings and search content is the main intention of preservice
teachers to use chatbots during the learning phase due to the convenient
interactions of chatbots, and may even replace the activities they would
otherwise do in the LMS. d) As in other studies, learners have a higher
willingness to interact with chatbots, especially for non-course content
(Liu et al., 2022). These findings are worthy of consideration in future
studies, for example, allowing pre-service teachers to better understand
the role of chatbots during their training, to better exploit the potential
of chatbots in teaching and learning or to develop additional issues for
academic research.
6. Limitation and future works
Despite being encouraged by the findings, we must acknowledge
some limitations of this study. First, the sample size of this study was
relatively small, and future studies should include more samples to
strengthen this finding. Second, the participants in this study were pre-
service teachers with certain information skills. Thus, further research is
needed to verify whether the same results are obtained for pre-service
teachers in other fields (e.g., English, physics), or for teachers with
lower information skills. Finally, This study lies in the fact that majority
of scholars who stated that teachers’ beliefs about using technology are
based on their experiences in pre-service teacher learning (Bahcivan
et al., 2019; Papadakis et al., 2021; Watson & Rockinson-Szapkiw,
2021). Moreover, when pre-service teachers develop more beliefs
about or demonstrate intentions to use educational technology, they are
more likely to use these technologies when they become teachers (Lin,
2019; Griful-Freixenet et al., 2021). Although the results of this study
revealed an approach to understanding pre-service teachers’
T.-C. Yang and J.-H. Chen Computers and Education: Artificial Intelligence 4 (2023) 100119

perceptions and intentions regarding the use of chatbots in their British Journal of Educational Technology, 53(1), 171–188. https://doi.org/10.1111/
bjet.13158
respective training courses, pre-service teachers are still not professional
Chaves, A. P., & Gerosa, M. A. (2021). How should my chatbot interact? A survey on
teachers. There is a need for longitudinal research in order to generalize social characteristics in human–chatbot interaction design. International Journal of
whether pre-service teachers who use chatbots will retain the same Human-Computer Interaction, 37(8), 729–758. https://doi.org/10.1080/
perceptions and intentions about using chatbots during their teaching 10447318.2020.1841438
Choi, S., Jang, Y., & Kim, H. (2022). Influence of pedagogical beliefs and perceived trust
careers. on teachers’ acceptance of educational artificial intelligence tools. International
Journal of Human-Computer Interaction, 1–13. https://doi.org/10.1080/
10447318.2022.2049145
Declaration of competing interest
Clawson, J. G. (1996). Mentoring in the information age. Leadership & Organization
Development Journal. https://doi.org/10.1108/01437739610785414
The authors declare that they have no known competing financial Czerkawski, B. C., & Lyman, E. W. (2016). An instructional design framework for
interests or personal relationships that could have appeared to influence fostering student engagement in online learning environments. TechTrends, 60(6),
532–539. https://doi.org/10.1007/s11528-016-0110-z
the work reported in this paper. Dincer, S. (2018). Are preservice teachers really literate enough to integrate technology
in their classroom practice? Determining the technology literacy level of preservice
Acknowledgements teachers. Education and Information Technologies, 23(6), 2699–2718. https://doi.org/
10.1007/s10639-018-9737-z
Dumford, A. D., & Miller, A. L. (2018). Online learning in higher education: Exploring
This study is supported in part by the Ministry of Science and advantages and disadvantages for engagement. Journal of Computing in Higher
Technology of Taiwan under contract number 108-2511-H-009-019- Education, 30(3), 452–465. https://doi.org/10.1007/s12528-018-9179-z
Duncan, T. G., & McKeachie, W. J. (2005). The making of the motivated strategies for
MY2 and 111-2410-H-A49-029. learning questionnaire. Educational Psychologist, 40(2), 117–128. https://doi.org/
10.1207/s15326985ep4002_6
References Fidan, M., & Gencel, N. (2022). Supporting the instructional videos with chatbot and
peer feedback mechanisms in online learning: The effects on learning performance
and intrinsic motivation. Journal of Educational Computing Research, 60(7),
Ali, W. (2020). Online and remote learning in higher education institutes: A necessity in
1716–1741. https://doi.org/10.1177/07356331221077901
light of COVID-19 pandemic. Higher Education, 10(3), 16–25. https://doi.org/
Foulger, T. S., Buss, R. R., & Su, M. (2021). The IT2 survey: Contextual knowledge (XK)
10.5539/hes.v10n3p16
influences on teacher candidates’ intention to integrate technology. Educational
Alm, A., & Nkomo, L. M. (2020). Chatbot experiences of informal language learners: A
Technology Research & Development, 69(5), 2729–2760. https://doi.org/10.1007/
sentiment analysis. International Journal of Computer-Assisted Language Learning and
s11423-021-10033-4
Teaching, 10(4), 51–65. https://doi.org/10.4018/IJCALLT.2020100104
Fryer, L., & Carpenter, R. (2006). Bots as language learning tools. Language, Learning and
Alotaibi, R., Ali, A., Alharthi, H., & Almehamdi, R. (2020). AI chatbot for tourist
Technology, 10(3), 8–14. https://doi.org/10.3390/educsci9040299
recommendations: A case study in the city of jeddah, Saudi arabia. International
Fryer, L. K., Nakao, K., & Thompson, A. (2019). Chatbot learning partners: Connecting
Journal of Interactive Mobile Technologies, 18–30. https://doi.org/10.3991/ijim.
learning experiences, interest and competence. Computers in Human Behavior, 93,
v14i19.1720
279–289. https://doi.org/10.1016/j.chb.2018.12.023
An, Y. J., & Reigeluth, C. (2011). Creating technology-enhanced, learner-centered
Gómez-Carrasco, C. J., Monteagudo-Fernández, J., Moreno-Vera, J. R., & Sainz-
classrooms: K–12 teachers’ beliefs, perceptions, barriers, and support needs. Journal
Gómez, M. (2019). Effects of a gamification and flipped-classroom program for
of Digital Learning in Teacher Education, 28(2), 54–62. https://doi.org/10.1080/
teachers in training on motivation and learning perception. Education Sciences, 9(4),
21532974.2011.10784681
299. https://doi.org/10.1016/j.tate.2021.103503
Bahcivan, E., Gurer, M. D., Yavuzalp, N., & Akayoglu, S. (2019). Investigating the
Griful-Freixenet, J., Struyven, K., & Vantieghem, W. (2021). Exploring pre-service
relations among pre-service teachers’ teaching/learning beliefs and educational
teachers’ beliefs and practices about two inclusive frameworks: Universal Design for
technology integration competencies: A structural equation modeling study. Journal
Learning and differentiated instruction. Teaching and Teacher Education, 107. https://
of Science Education and Technology, 28(5), 579–588. https://doi.org/10.1007/
doi.org/10.1016/j.tate.2021.103503
s10956-019-09788-6
Ha, C., & Lee, S. Y. (2019). Elementary teachers’ beliefs and perspectives related to smart
Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential
learning in South Korea. Smart Learning Environments, 6(1), 1–15. https://doi.org/
analysis. Cambridge university press.
10.1080/02607476.2019.1708628
Bakeman, R., & Quera, V. (2011). Sequential analysis and observational methods for the
Hoareau, L., Thomas, A., Tazouti, Y., Dinet, J., Luxembourger, C., & Jarlégan, A. (2021).
behavioral sciences. Cambridge University Press. https://doi.org/10.1017/
Beliefs about digital technologies and teachers’ acceptance of an educational app for
CBO9781139017343.002
preschoolers. Computers & Education, 172, 104264. https://doi.org/10.1016/j.
Barak, M., & Assal, M. (2018). Robotics and STEM learning: Students’ achievements in
compedu.2021.104264
assignments according to the P3 Task Taxonomy—practice, problem solving, and
Hope, J. (2020). Answer students’ questions anytime, using technology they embrace.
projects. International Journal of Technology and Design Education, 28(1), 121–144.
The Successful Registrar, 20(8), 8–9. https://doi.org/10.1002/tsr.30761
https://doi.org/10.1007/s10798-016-9385-9
Hsu, C. Y., Tsai, M. J., Chang, Y. H., & Liang, J. C. (2017). Surveying in-service teachers’
Bardakci, S., & Unver, T. K. (2020). Preservice ICT teachers’ technology metaphors in the
beliefs about game-based learning and perceptions of technological pedagogical and
margin of technological determinism. Education and Information Technologies, 25(2),
content knowledge of games. Journal of Educational Technology & Society, 20(1),
905–925. https://doi.org/10.1007/s10639-019-09997-x
134–143, 0.1080/1475939X.2020.1752296.
Berger, R., Ebner, M., & Ebner, M. (2019). Conception of a conversational interface to
Hsu, C. C., & Wang, T. I. (2018). Applying game mechanics and student-generated
provide a guided search of study related data. International Journal of Emerging
questions to an online puzzle-based game learning system to promote algorithmic
Technologies in Learning, 14, 37–47. https://doi.org/10.3991/ijet.v14i07.10137, 07.
thinking skills. Computers & Education, 121, 73–88. https://doi.org/10.1016/j.
Bibauw, S., François, T., & Desmet, P. (2022). Dialogue systems for language learning:
compedu.2018.02.002
Chatbots and beyond. In The routledge handbook of second language acquisition and
Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—are they
technology (pp. 121–135). https://doi.org/10.4324/9781351117586-12. Routledge.
really useful? A systematic review of chatbot-supported language learning. Journal of
Bii, P. K., Too, J. K., & Mukwa, C. W. (2018). Teacher attitude towards use of chatbots in
Computer Assisted Learning, 38(1), 237–257. https://doi.org/10.1111/jcal.12610
routine teaching. Universal Journal of Educational Research, 6(7), 1586–1597. https://
Hwang, G.-J., Yang, L.-H., & Wang, S.-Y. (2013). A concept map-embedded educational
doi.org/10.13189/ujer.2018.060719
computer game for improving students’ learning performance in natural science
Booth-Butterfield, M., & Wanzer, M. B. (2010). Humor and communication. The SAGE
courses. Computers & Education, 69, 121–130. https://doi.org/10.1016/j.
Handbook of Communication and Instruction.
compedu.2013.07.008
Borg, M. (2001). Teachers’ beliefs. ELT Journal, 55(2), 186–187. https://doi.org/
Islamoglu, H., Yurdakul, I. K., & Ursavas, O. F. (2021). Pre-service teachers’ acceptance
10.1093/elt/55.2.186
of mobile-technology-supported learning activities. Etr&D-Educational Technology
Bower, M., DeWitt, D., & Lai, J. W. (2020). Reasons associated with preservice teachers’
Research and Development, 69(2), 1025–1054. https://doi.org/10.1007/s11423-021-
intention to use immersive virtual reality in education. British Journal of Educational
09973-8
Technology, 51(6), 2215–2233. https://doi.org/10.1111/bjet.13009
Istenic, A., Bratko, I., & Rosanda, V. (2021). Are pre-service teachers disinclined to utilise
Camacho-Zuñiga, C., Pego, L., Escamilla, J., & Hosseini, S. (2021). The impact of the
embodied humanoid social robots in the classroom? British Journal of Educational
COVID-19 pandemic on students’ feelings at high school, undergraduate, and
Technology, 52(6), 2340–2358. https://doi.org/10.1111/bjet.13144
postgraduate levels. Heliyon, 7(3), Article e06465. https://doi.org/10.1016/j.
Jeon, J. (2022). Exploring a self-directed interactive app for informal EFL learning: A
heliyon.2021.e06465
self-determination theory perspective. Education and Information Technologies, 27(4),
Carayannopoulos, S. (2018). Using chatbots to aid transition. The International Journal of
5767–5787. https://doi.org/10.1007/s10639-021-10839-y
Information and Learning Technology, 35(2), 118–129. https://doi.org/10.1108/
Kemp, A., Palmer, E., & Strelan, P. (2019). A taxonomy of factors affecting attitudes
IJILT-10-2017-0097
towards educational technologies for use with technology acceptance models. British
Chang, M. M. (2005). Applying self-regulated learning strategies in a web-based
Journal of Educational Technology, 50(5), 2394–2413. https://doi.org/10.1111/
instruction—an investigation of motivation perception. Computer Assisted Language
bjet.12833
Learning, 18(3), 217–230. https://doi.org/10.1080/09588220500178939
Chang, C. Y., Hwang, G. J., & Gau, M. L. (2022). Promoting students’ learning
achievement and self-efficacy: A mobile chatbot approach for nursing training.

8
T.-C. Yang and J.-H. Chen
Kesicioglu, O. S. (2021). Opinions of pre-service pre-school teachers on the use of
mathematics activities. South African Journal of Education, 41(1). https://doi.org/
10.15700/saje.v41n1a1813
Khan, R. A., & Jawaid, M. (2020). Technology enhanced assessment (TEA) in COVID 19
pandemic. Pakistan Journal of Medical Sciences, 36(COVID19-S4), S108. https://doi.
org/10.12669/pjms.36.COVID19-S4.2795
Kowald, C., & Bruns, B. (2020). Chatbot kim: A digital tutor on ai. How advanced dialog
design creates better conversational learning experiences. International Journal of
Advanced Corporate Learning, 13(3), 26–34. https://doi.org/10.3991/ijac.
v13i3.17017
Kuhail, M. A., Al Katheeri, H., Negreiros, J., Seffah, A., & Alfandi, O. (2022). Engaging
students with a chatbot-based academic advising system. International Journal of
Human-Computer Interaction, 1–27. https://doi.org/10.1080/
10447318.2022.2074645
de Las Heras, S. C., Jones, M. N., Gernaey, K. V., Kruhne, U., & Mansouri, S. S. (2020). An
E-learning bot for bioprocess systems engineering. Computer Aided Chemical
Engineering, 48, 2023–2028. https://doi.org/10.1016/B978-0-12-823377-1.50338-4
Lin, Y. T. (2019). Impacts of a flipped classroom with a smart learning diagnosis system
on students’ learning performance, perception, and problem solving ability in a
software engineering course. Computers in Human Behavior, 95, 187–196. https://
doi.org/10.1016/j.chb.2018.11.036
Liu, S. H. (2011). Factors related to pedagogical beliefs of teachers and technology
integration. Computers & Education, 56(4), 1012–1022. https://doi.org/10.1016/j.
compedu.2010.12.001
Liu, C. C., Liao, M. G., Chang, C. H., & Lin, H. M. (2022). An analysis of
children’interaction with an AI chatbot and its impact on their interest in reading.
Computers & Education, 189, Article 104576. https://doi.org/10.1016/j.
compedu.2022.104576
Liu, Z., Yang, C., Rüdian, S., Liu, S., Zhao, L., & Wang, T. (2019). Temporal emotion-
aspect modeling for discovering what students are concerned about in online course
forums. Interactive Learning Environments, 27(5–6), 598–627. https://doi.org/
10.1080/10494820.2019.1610449
Mendez, S. L., Johanson, K., Conley, V. M., Gosha, K., Mack, N., Haynes, C., &
Gerhardt, R. (2020). Chatbots: A tool to supplement the future faculty mentoring of
doctoral engineering students. International Journal of Doctoral Studies, 15(1),
373–392. https://doi.org/10.28945/4579
Naidu, S. (2017). Openness and flexibility are the norm, but what are the challenges?
Distance Education, 38(1), 1–4. https://doi.org/10.1080/01587919.2017.1297185
Ndlovu, M., Ramdhany, V., Spangenberg, E. D., & Govender, R. (2020). Preservice
teachers’ beliefs and intentions about integrating mathematics teaching and learning
ICTs in their classrooms. ZDM, 52(7), 1365–1380. https://doi.org/10.1007/s11858-
020-01186-2
Ng’ambi, D., Brown, C., Bozalek, V., Gachago, D., & Wood, D. (2016). Technology
enhanced teaching and learning in South African higher education–A rearview of a
20 year journey. British Journal of Educational Technology, 47(5), 843–858. https://
doi.org/10.1111/bjet.12485
Oh, Y. (2020). Interactive computing technology in anti-bullying education: The effects
of conversation-bot’s role on K-12 students’ attitude change toward bullying
problems. Journal of Educational Computing Research, 58(1), 200–219. https://doi.
org/10.1177/0735633119839177
Osei, H. V., Kwateng, K. O., & Boateng, K. A. (2022). Integration of personality trait,
motivation and UTAUT 2 to understand e-learning adoption in the era of COVID-19
pandemic. Education and Information Technologies, 1–26. https://doi.org/10.1023/A:
1018617431082
Papadakis, S., Vaiopoulou, J., Sifaki, E., Stamovlasis, D., & Kalogiannakis, M. (2021).
Attitudes towards the use of educational robotics: Exploring pre-service and in-
service early childhood teacher profiles. Education Sciences, 11(5), 204. https://doi.
org/10.3390/educsci11050204
Pardo, A., Han, F., & Ellis, R. A. (2016). Combining university student self-regulated
learning indicators and engagement with online learning events to predict academic
performance. IEEE Transactions on Learning Technologies, 10(1), 82–92. https://doi.
org/10.1109/TLT.2016.2639508
Pereira, J., Fernández-Raga, M., Osuna-Acedo, S., Roura-Redondo, M., Almazán-
López, O., & Buldón-Olalla, A. (2019). Promoting learners’ voice productions using
chatbots as a tool for improving the learning process in a MOOC. Technology,
Knowledge and Learning, 24(4), 545–565. https://doi.org/10.1007/s10758-019-
09414-9
Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020). Rediscovering the use of chatbots
in education: A systematic literature review. Computer Applications in Engineering
Education, 28(6), 1549–1565. https://doi.org/10.1002/cae.22326
Pierce, R., & Ball, L. (2009). Perceptions that may affect teachers’ intention to use
technology in secondary mathematics classes. Educational Studies in Mathematics, 71
(3), 299–317. https://doi.org/10.1007/s10649-008-9177-6
Pintrich, P. R. (1991). A manual for the use of the motivated Strategies for learning
questionnaire (MSLQ).
Polasek, R., & Javorcik, T. (2019). Results of pilot study into the application of
MicroLearning in teaching the subject computer architecture and operating system
basics. In 2019 international symposium on educational technology (ISET) (pp.
196–201). IEEE. https://doi.org/10.1109/ISET.2019.00048.
Prior, D. D., Mazanov, J., Meacheam, D., Heaslip, G., & Hanson, J. (2016). Attitude,
digital literacy and self efficacy: Flow-on effects for online learning behavior. The
Internet and Higher Education, 29, 91–97. https://doi.org/10.1016/j.
iheduc.2016.01.001
Reshmi, S., & Balakrishnan, K. (2016). Implementation of an inquisitive chatbot for
database supported knowledge bases. Sādhanā, 41(10), 1173–1178. https://doi.org/
10.1007/s12046-016-0544-1
9
Computers and Education: Artificial Intelligence 4 (2023) 100119
Saltan, F. (2017). Blended learning experience of students participating pedagogical
formation program: Advantages and limitation of blended education. International
Journal of Higher Education, 6(1), 63–73. https://doi.org/10.5430/ijhe.v6n1p63
Saltan, F., & Arslan, K. (2017). A comparison of in-service and pre-service teachers’
technological pedagogical content knowledge self-confidence. Cogent Education, 4
(1), Article 1311501. https://doi.org/10.1080/2331186X.2017.1311501
Serrano, D. R., Dea-Ayuela, M. A., Gonzalez-Burgos, E., Serrano-Gil, A., & Lalatsa, A.
(2019). Technology-enhanced learning in higher education: How to enhance student
engagement through blended learning. European Journal of Education, 54(2),
273–286. https://doi.org/10.1111/ejed.12330
Shen, C. W., & Ho, J. T. (2020). Technology-enhanced learning in higher education: A
bibliometric analysis with latent semantic approach. Computers in Human Behavior,
104, Article 106177. https://doi.org/10.1016/j.chb.2019.106177
Shimada, A., Konomi, S. I., & Ogata, H. (2018). Real-time learning analytics system for
improvement of on-site lectures. Interactive Technology and Smart Education, 15(4),
314–331. https://doi.org/10.1108/ITSE-05-2018-0026
Shim, T. E., & Lee, S. Y. (2020). College students’ experience of emergency remote
teaching due to COVID-19. Children and Youth Services Review, 119, Article 105578.
https://doi.org/10.1016/j.childyouth.2020.105578
Short, J., Williams, E., & Christie, B. (1976). The social psychology of telecommunications.
Toronto; London; New York: Wiley.
Shum, H. Y., He, X. D., & Li, D. (2018). From eliza to XiaoIce: Challenges and
opportunities with social chatbots. Frontiers of Information Technology & Electronic
Engineering, 19(1), 10–26. https://doi.org/10.1631/FITEE.1700826
Sorge, S., Kröger, J., Petersen, S., & Neumann, K. (2019). Structure and development of
pre-service physics teachers’ professional knowledge. International Journal of Science
Education, 41(7), 862–889. https://doi.org/10.1080/09500693.2017.1346326
Summers, J. J., Davis, H. A., & Hoy, A. W. (2017). The effects of teachers’ efficacy beliefs
on students’ perceptions of teacher relationship quality. Learning and Individual
Differences, 53, 17–25. https://doi.org/10.1016/j.lindif.2016.10.004
Tamayo, P. A., Herrero, A., Martín, J., Navarro, C., & Tránchez, J. M. (2020). Design of a
chatbot as a distance learning assistant. Open Praxis, 12(1), 145–153. https://doi.
org/10.5944/openpraxis.12.1.1063
Tan, L. M., & Laswad, F. (2006). Students’ beliefs, attitudes and intentions to major in
accounting. Accounting Education: An International Journal, 15(2), 167–187. https://
doi.org/10.1080/09639280600787194
Tarhini, A., Hone, K., Liu, X., & Tarhini, T. (2017). Examining the moderating effect of
individual-level cultural values on users’ acceptance of E-learning in developing
countries: A structural equation modeling of an extended technology acceptance
model. Interactive Learning Environments, 25(3), 306–328. https://doi.org/10.1080/
10494820.2015.1122635
Teo, T. (2009). Modelling technology acceptance in education: A study of pre-service
teachers. Computers & Education, 52(2), 302–312. https://doi.org/10.1016/j.
compedu.2008.08.006
Thompson, A. G. (1992). Teachers’ beliefs and conceptions: A synthesis of the research.
Journal of Research in Science Teaching, 29(4), 331–359. https://doi.org/10.1002/
tea.3660290404
Thurm, D., & Barzel, B. (2020). Effects of a professional development program for
teaching mathematics with technology on teachers’ beliefs, self-efficacy and
practices. ZDM, 52(7), 1411–1422. https://doi.org/10.1007/s11858-020-01158-6
Trigwell, K., Ashwin, P., & Millan, E. S. (2013). Evoked prior learning experience and
approach to learning as predictors of academic achievement. British Journal of
Educational Psychology, 83(3), 363–378. https://doi.org/10.1111/j.2044-
8279.2012.02066.x
Valtonen, T., Leppanen, U., Hyypia, M., Sointu, E., Smits, A., & Tondeur, J. (2020). Fresh
perspectives on TPACK: Pre-service teachers’ own appraisal of their challenging and
confident TPACK areas. Education and Information Technologies, 25(4), 2823–2842.
https://doi.org/10.1007/s10639-019-10092-4
Van Katwijk, L., Jansen, E., & Van Veen, K. (2021a). Pre-service teacher research: A way
to future-proof teachers? European Journal of Teacher Education. https://doi.org/
10.1080/02619768.2021.1928070
Van Katwijk, L., Jansen, E., & van Veen, K. (2021b). Development of an inquiry stance?
Perceptions of preservice teachers and teacher educators toward preservice teacher
inquiry in Dutch primary teacher education. Journal of Teacher Education, 73(3),
286–300. https://doi.org/10.1177/00224871211013750
van Twillert, A., Kreijns, K., Vermeulen, M., & Evers, A. (2020). Teachers’ beliefs to
integrate Web 2.0 technology in their pedagogy and their influence on attitude,
perceived norms, and perceived behavior control. International Journal of Educational
Research Open, 1, 100014. https://doi.org/10.1016/j.ijedro.2020.100014
Wang, Y. M. (2002). When technology meets beliefs: Preservice teachers’ perception of
the teacher’s role in the classroom with computers. Journal of Research on Technology
in Education, 35(1), 150–161. https://doi.org/10.1080/15391523.2002.10782376
Watson, J. H., & Rockinson-Szapkiw, A. (2021). Predicting preservice teachers’ intention
to use technology-enabled learning. Computers & Education, 168, 10. https://doi.org/
10.1016/j.compedu.2021.104207
Witt, P. L., Schrodt, P., Wheeless, V. E., & Bryand, M. C. (2014). Students’ intent to
persist in college: Moderating the negative effects of receiver apprehension with
instructor credibility and nonverbal immediacy. Communication Studies, 65(3),
330–352. https://doi.org/10.1080/10510974.2013.811428
Wombacher, K., Dai, M., Matig, J. J., & Harrington, N. G. (2018). Using the integrative
model of behavioral prediction to understand college students’ STI testing beliefs,
intentions, and behaviors. Journal of American College Health, 66(7), 674–682.
https://doi.org/10.1080/07448481.2018.1454928
Wu, E. H. K., Lin, C. H., Ou, Y. Y., Liu, C. Z., Wang, W. K., & Chao, C. Y. (2020).
Advantages and constraints of a hybrid model K-12 E-learning assistant chatbot. IEEE
Access, 8, 77788–77801. https://doi.org/10.1109/ACCESS.2020.2988252
T.-C. Yang and J.-H. Chen
Yalman, M., & Basaran, B. (2021). Examining PRESERVICE teachers’ use of
SMARTBOARD and pc tablets in lessons. Education and Information Technologies, 26
(2), 1435–1453. https://doi.org/10.1007/s10639-020-10292-3
Yang, C., Flanagan, B., Akcapinar, G., & Ogata, H. (2018). Maintaining reading
experience continuity across e-book revisions. Research and Practice in Technology
Enhanced Learning, 13(1), 1–16. https://doi.org/10.1186/s41039-018-0093-9
Ye, X., Liu, P. F., Lee, X. Z., Zhang, Y. Q., & Chiu, C. K. (2021). Classroom misbehaviour
management: An SVVR-based training system for preservice teachers. Interactive
10
Computers and Education: Artificial Intelligence 4 (2023) 100119
Learning Environments, 29(1), 112–129. https://doi.org/10.1080/
10494820.2019.1579235
Yin, J., Goh, T. T., Yang, B., & Xiaobin, Y. (2021). Conversation technology with micro-
learning: The impact of chatbot-based learning on students’ learning motivation and
performance. Journal of Educational Computing Research, 59(1), 154–177. https://
doi.org/10.1177/0735633120952067