See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/372487229

Measuring EFL learners' use of ChatGPT in informal digital learning of English

based on the technology acceptance model

Article in Innovation in Language Learning and Teaching · July 2023

DOI: 10.1080/17501229.2023.2240316

CITATIONS READS

34 2,238

2 authors, including:

Guangxiang Leon Liu

The Chinese University of Hong Kong
24 PUBLICATIONS 124 CITATIONS

SEE PROFILE

All content following this page was uploaded by Guangxiang Leon Liu on 21 July 2023.

The user has requested enhancement of the downloaded file.

Innovation in Language Learning and Teaching

Measuring EFL learners’ use of ChatGPT

in informal digital learning of English
based on the technology acceptance model

Guangxiang Leon LIU1*, Department of English, The

Chinese University of Hong Kong
Chaojun MA2, Department of English, The City
University of Hong Kong

This is an author-produced PDF of an empirical

article accepted for publication in Innovation in
Language Learning and Teaching following
peer review. The full, definitive publisher-
authenticated version can be made available
online soon.

To Cite: Liu, G., & Ma, C. (2023, in press). Measuring EFL learners’ use of ChatGPT in informal
digital learning of English based on the technology acceptance model. Innovation in Language
Learning and Teaching, 1-18.

Notes on Contributors

Guangxiang Liu is a Ph.D. candidate in Applied English Linguistics at the Department of English, The
Chinese University of Hong Kong, Hong Kong. His research interests include informal digital learning of
English, digital literacies, identity and investment. He has published in international peer-reviewed
journals like Computer Assisted Language Learning, TESOL Quarterly, and Applied Linguistics.
Email: G.Liu@link.cuhk.edu.hk

Chaojun Ma is currently pursuing her PhD in the English Department, The City University of Hong
Kong. Her research areas include linguistic landscape, language learning beyond the classroom, and
intercultural communication. She has published in international peer-reviewed journals such as Computer
Assisted Language Learning, English Today, and Linguistic Landscape.
Email: C.Ma-21@outlook.com

Corresponding Address:
Department of English
The Chinese University of Hong Kong
Shatin, NT
Hong Kong, China

Data availability statement

The raw data can be made available upon reasonable request.

Acknowledgments
We are thankful to all participants involved in this research. Many thanks also go to the
anonymous reviewers and the editor-in-chief Professor Hayo Reinders for their time and
professional help.

1
Innovation in Language Learning and Teaching

Measuring EFL learners’ use of ChatGPT in informal digital learning of

English based on the technology acceptance model

ABSTRACT

Purpose: The advent of ChatGPT, one of the most advanced chatbots building on OpenAI’s
Large Language Models, may transform the ways in which language is learned, taught, and
assessed in the contemporary world wrought by technological innovation. Against this
backdrop, this study aims to generate empirical insights into the extent to which ChatGPT is
perceived and leveraged by EFL learners beyond the classroom.
Design/Methodology: This quantitative cross-sectional investigation draws upon the
technology acceptance model (TAM) as developed by Davis (1989) to conceptualize EFL
learners’ attitudes, intentions, and actual behaviors of using ChatGPT in their informal digital
learning of English. A total of 405 EFL learners answered the revised TAM questionnaire with
scales including Perceived Ease of Use, Perceived Usefulness, Attitude, Behavioral Intention,
and Actual Use.
Findings: The results of structural equation modeling analyses indicated that while Perceived
Ease of Use fails to predict learners’ Attitude directly, it can leave an impact on Attitude
through the full mediator Perceived Usefulness. It was also found that learners who take
positive attitudes toward the usefulness of ChatGPT tend to demonstrate a higher level of
Behavioral Intention, which positively and strongly predicts their actual use of this latest AI-
powered tool in English learning outside the classroom.
Originality/value: This study provides empirical evidence that supports the potential of
ChatGPT as a powerful language-learning tool that EFL learners should utilize to participate
creatively and productively in the ecological CALL shaped by constantly emerging AI
technologies.

Keywords: ChatGPT; Informal digital learning of English; Technology Acceptance Model;

Structural equation modeling, AI technology

Introduction

In the field of Computer Assisted Language Learning (CALL), Dorothy Chun (2016)
conceptualizes the phase from 2010 onwards as ecological CALL because technological
innovation has expanded language learning contexts by extending the time and space where
language learning and socialization occur beyond the classroom. In the third decade of the
twenty-first century, the world is increasingly being transformed by Artificial Intelligence (AI)
technologies based on a range of advanced Large Language Models (Kiaer & Lee, 2023;
Warschauer & Xu, 2022). Examples comprise ChatGPT-3.5, ChatGPT-4.0 and Bing Chat from
OpenAI, PaLM2 from Google, which are models that can predict, comprehend and generate
texts in a highly accurate and human-like fashion (Tam, 2023). Such expansion of
technological affordances has largely complicated the dynamics of the ecological CALL by
bringing about new possibilities for language learning and teaching.

As one of the most advanced chatbots building on OpenAI’s Large Language Models,
ChatGPT1 can revolutionize the ways in which language is learned, taught, and assessed in the
1
In this study, ChatGPT refers to the chatbot application powered by GPT-3.5 rather than GPT4.0 because all
data were collected before ChatGPT-4.0 was made publicly available on March 14, 2023.

2
Innovation in Language Learning and Teaching

fast-changing world where globalization and technological advancement have dramatically

changed the education landscape. ChatGPT is known for its astounding ability to generate
human-like and seemingly accurate responses in a variety of languages. Scholars have argued
that the advent of ChatGPT may herald the age of artificial general intelligence where AI
exhibits the potential to acquire the level of intelligence that human beings possess by evolving
“from an algorithmic intelligence to linguistic one” (Zhou et al., 2023, p. 5). Against such a
backdrop, despite the concern from policymakers and educators that ChatGPT can incur ethical
issues, wholescale academic misconduct, and even dehumanization (Chomsky, 2023; Rudolph
et al., 2023), a close inspection of public discourse may reveal worldwide interest in the
opportunities that this super-intelligent conversational bot has offered for language learning
(Cai, 2023; Sharma, 2023; Shemesh, 2023). Based on their preliminary interactions with
ChatGPT, some scholars have named a few affordances of this emergent AI technology (Cai,
2023; Rudolph et al., 2023). For example, by engaging students in conversational interactions
and offering immediate feedback, ChatGPT allows language learners to enact new meaning-
making practices and benefit from enhanced personalized learning with creativity and
productivity. In addition, language learners can increase their exposure to the target language
in the seemingly authentic and real-life language use situations simulated by ChatGPT.
Building on her own user experience, Shemesh (2023) even provides an extensive and
accessible guide on improving one’s English communicative competence, vocabulary, and
grammatical knowledge using ChatGPT. These results combine to accentuate the power and
prospect of ChatGPT as a utility AI language-learning technology.

Empirically, there have been a few peer-reviewed articles that examine the opportunities and
challenges of ChatGPT for the general field of education by conducting qualitative case studies
(Tlili et al., 2023) or reviewing preprints of relevant studies, media outlets, and blog posts
(Rudolph et al., 2023). However, thus far, little research rigorously investigates the extent to
which ChatGPT is perceived and accepted by English language learners in the ecological
CALL. A complete understanding of such a reality is necessary because it is the first step in
helping educators and language instructors learn about and utilize transformative AI
technologies for pedagogical purposes. To address such a gap, this quantitative investigation
draws upon the technology acceptance model (TAM) as developed by Davis (1989) to
conceptualize English-as-a-foreign-language (EFL) learners’ attitudes and Behavioral
Intention to use ChatGPT in out-of-class and naturalistic settings. To frame the naturalistic
digital learning context, we build on the notion of informal digital learning of English (IDLE)
which refers to self-motivated autonomous English learning activities in the wider extramural
digitalized learning ecology (Lee & Drajati, 2019; Liu et al., 2023; Zhang & Liu, 2022). We
aim to address one research question: In what ways do EFL learners accept and use ChatGPT
in their IDLE activities?

Literature Review

ChatGPT, AI chatbot, and informal digital learning of English

While we still fall short of empirical evidence that confirms the usefulness of ChatGPT in
English language learning and teaching, research on AI chatbots as conversational partners has
started to proliferate in recent years. Usually situated within the EFL contexts, these studies
focus on dialogue-based automated agents/applications that enable language learners to
practice English through verbal or written interactions with the chatbot (Belda-Medina &
Calvo-Ferrer, 2022; Bibauw et al., 2022; Ebadi & Amini, 2022; Haristiani & Rifa’I, 2020;
Huang et al., 2022; Kim et al., 2019; Sharadgah & Sa’di, 2022). For instance, adopting an

3
Innovation in Language Learning and Teaching

experimental design, Kim (2019) measured two groups of Korean undergraduate EFL learners’
(n = 36 for the chatbot group, n = 34 for the human chat group) perceptions of the effects of AI
chatbots on their English grammar skills. The independent T-test results indicated that chatbot
use could significantly improve students’ English grammar skills. Concentrating on the use of
a chatbot application in an Iranian EFL university context, Ebadi and Amini (2022) collected
quantitative and qualitative data from 256 undergraduates. They found that chatbot accuracy
and human likeness could positively influence students’ confidence and motivation to learn
English. In their meta-analysis article, Bibauw et al. (2022) concluded that engaging language
learners with AI-powered dialogue systems could predict their vocabulary and
morphosyntactic outcomes and increase learners’ holistic proficiency and accuracy in language
use. Their article also highlighted the existing insufficient understanding of chatbots’ impact
on language learners’ reading and listening comprehension.

It should be acknowledged that, in the studies reviewed above, students’ participation in

chatbot-based language learning was usually structured by teachers or researchers. There have
also been a growing number of scholars giving heed to how chatbot platforms constructed
effective and self-directed learning spaces in informal and digitalized contexts. To create a
more personal learning environment for Japanese language learners in Indonesia, Haristiani
and Rifa’I (2020) developed Gengobot, a chatbot-based application based on the popular social
media platform LINE. The study found that the integration of AI conversational partners and
social media was able to cater to personalized language learning needs and preferences.
Similarly, Belda-Medina and Calvo-Ferrer (2022) conducted a mixed-method study to trace
the outcomes of applying AI chatbots (e.g., Kuki, an Avatar-based AI bot designed to simulate
interactions) to students’ and in-service teachers’ autonomous language learning activities
beyond the classroom. Analyses of data from 115 Spanish EFL undergraduates and 61 Polish
EFL students revealed that specific linguistic and technological features of different chatbots
figured prominently in students’ perceived usefulness, acceptance, and actual use of AI
conversational robots for informal language learning.

Although the utilization of AI-powered technologies may lead to students’ gains in language
proficiency and bring them emotional benefits, a more context-specific perspective should be
established to understand how chatbot platforms contribute to their self-directed language
learning in the ecological CALL. The reason is that, with varying design features and language
processing models, AI chatbot tools may develop their different cultures of use, which can in
no small measure shape “language learners’ participation and forms of language development
taking place in computer-mediated contexts” (Thorne & Black, 2007, p. 133). In the case of
ChatGPT, despite still operating in the form of a monomodal chatbot (at least for ChatGPT
3.5), it differs from other AI technologies as it can leave transformative impacts on many
industries (e.g., language education) by demonstrating the ability to learn or even transcend
human intelligence (Zhou et al., 2023).

Technology acceptance model as the framework

To theoretically support the measurement of EFL learners’ acceptance of ChatGPT, this study
draws on the Technology Acceptance Model (TAM) developed by Fred Davis (1986; 1989), a
leading scholar in the field of information management. Over the past three decades, TAM has
been recognized as an influential model that explicates individuals’ acceptance, rejection, and
usage of information technologies in the fields of information management, digital sociology,
education technology, and CALL (see Al-Emran et al., 2018; Alfadda & Mahdi, 2021; Bailey
et al., 2022; Davis, 1986; Sun & Mei, 2022; Venkatesh, 2000; Yang & Wang, 2019).

4
Innovation in Language Learning and Teaching

As a powerful framework that predicts individuals’ adoption of new technology, at the heart of
the TAM model is the Attitude-Intention-Behavior relationship. As Davis (1989) argues,
attitude refers to the extent to which people are interested in and positively assess the use of
specific technologies. Two crucial factors that predict one’s attitude are perceived ease of use
as “the degree to which a person believes that using a particular system would be free of effort”
(p. 320), and perceived usefulness which points to “the degree to which a person believes that
using a particular system would enhance his or her job performance” (Davis, 1989, p. 320).
Attitude can influence people’s behavioral intention to use technology which may finally
predict the actual behaviors of using technology to perform specific tasks (Alfadda & Mahdi,
2021; Davis 1989). It is worth noting that although there have been attempts to incorporate
additional external variables (e.g., subjective norms, computer self-efficacy) to explain
variance in technology usage behaviors (Bailey et al., 2022; Yang & Wang, 2019), the core
TAM components are generally assumed to be Perceived Ease of Use, Perceived Usefulness,
Attitude, Behavioral Intention, and Actual Use.

This TAM-informed study seeks to measure and account for EFL learners’ behavioral usage
of ChatGPT—currently one of the most recognizable and capable chatbots across the world—
in informal and extramural English settings. Table 1 demonstrates the revamped definitions of
TAM components in the present study based on previous literature (Bailey et al., 2022; Davis,
1989; Lee & Drajati, 2023; Shemesh, 2023).

Table 1. Revamped Definition of the Main TAM Constructs

Definitions
Perceived Ease of The extent to which learners perceive that little effort will be required to
Use (PEU) use ChatGPT.
Perceived The extent to which learners perceive that ChatGPT will be very useful
Usefulness (PU) and facilitate their English learning in informal and digital contexts.
Attitude (AT) The extent to which learners are interested in and positively assess the
use of ChatGPT in their IDLE activities.
Behavioral Intention The extent to which learners intend to use ChatGPT in their IDLE
(BI) activities.
Actual Use (AU) The extent to which learners engage in the autonomous use of ChatGPT
in their IDLE activities.

Conceptual model and hypotheses

Based on prior TAM literature (Alfadda & Mahdi, 2021; Bailey et al., 2022; Davis, 1989), this
study proposes the following conceptual model to showcase the hypothesized inter-factor
relationships among TAM component variables (Figure 1).

5
Innovation in Language Learning and Teaching

Figure 1. The Hypothesized Conceptual Model

Figure 1. Alt Text: A hypothesized structural model starting with two exogenous variables PEU
and PU which predict the variables AT, proceeding with the hypothesized path from AT to BI
and finally AU.

Building on this conceptual model, the following six hypotheses were posed:

H1: PEU positively predicts PU.

H2: PU positively predicts AT.
H3: PEU positively predicts AT.
H4: AT positively predicts BI.
H5: BI positively predicts AU.
H6: PU mediates the relationship between PEU and AT.

Method
Context and participants

Adopting the cross-sectional survey design (Paltridge & Phakiti, 2015), this study sought to
collect quantitative data from a relatively large population to examine users’ perceptions and
acceptance of ChatGPT in their informal English language learning. Employing a modified
TAM questionnaire, we gathered online survey responses from ChatGPT users from Chinese
EFL backgrounds. The questionnaire was administered and released online in the form of an
e-poster. Drawing on purposive sampling techniques, we sent out the e-poster to different
online forums or discussion groups on Chinese social media (e.g., Douban, Weibo,
Xiaohongshu, and WeChat) to recruit interested ChatGPT users. The recruitment criteria
included 1) participants should come from Chinese EFL backgrounds; 2) participants should
have used ChatGPT for their informal language learning; 3) participants should be willing to
participate in this study. The only participant exclusion criterion was that the total length of
time that participants spent on ChatGPT in their informal English learning was less than three

6
Innovation in Language Learning and Teaching

weeks. Our intention behind this criterion was to make sure that all participants had used
ChatGPT in their IDLE activities for a relatively long time before filling out the questionnaire.

Data collection lasted from mid-February 2023 to early March 2023. A total of 405 ChatGPT
users (73 males, and 332 females) from Chinese EFL backgrounds answered the survey
questions online. Such an imbalanced gender ratio may be related to the observed phenomenon
that female users in the online ChatGPT-related forums tend to be more active and willing to
respond to our requests and fill out the online questionnaire, while male users are more likely
to remain silent. Of these respondents, the majority (n = 270, 66.67%) were aged between 18
and 24, and 132 participants (32.59%) were beyond 24 years old. Only three (0.74%) were
younger than 18. Also noted is that the 405 EFL learners were at different stages in their life
including working (n = 65, 16.05%), gap year (n = 18, 4.44%), doing undergraduate studies (n
= 206, 50.86%), pursuing master’s degrees (n = 73, 18.02%) and doctoral degrees (n = 43,
10.62%). The participants were primarily based in China mainland (n = 323, 79.75%), while
44 (10.86%) were located in Hong Kong, Macau, and Taiwan as short-term visitors or cross-
border students. There were also 38 (9.38%) respondents who were Chinese undergraduate or
postgraduate students studying abroad in the United States or other Anglophone countries.

The questionnaire

The questionnaire opened with demographic items that adopted a nominal scale to collect
information regarding participants’ gender, age, employment status/education, geographical
location, and experiences of using AI-powered technologies. Using a six-point Likert scale
ranging from “strongly disagree” (1) to “strongly agree” (6), the second section included five
TAM constructs that we modified based on the studies of Davis (1986; 1989) and later TAM
instruments (e.g., Bailey et al., 2022; Sun & Mei, 2022). The five constructs were Perceived
Ease of Use, Perceived Usefulness, Attitude, Behavioral Intention, and Actual Use respectively.
Importantly, we also drew on the scales used within IDLE literature (e.g., Lee & Drajati, 2019;
Lee & Xie, 2022; Liu et al., 2023a; Liu et al., 2023b) to better frame items that focused on how
participants actually used ChatGPT in their self-directed English learning beyond the
classroom. The initial pool of items was later sent to three experts in the field for their scrutiny
to ensure the modified questionnaire’s content validity. Following two rounds of revisions, the
final 22-item TAM questionnaire was generated (see Appendix in the supplementary material).

Research ethics

Ethical issues were meticulously examined in this study. The ethical approval was gained in
the form of an official approval memo. In addition, when collecting responses from participants,
we paid special attention to the achievement of their informed consent. To this end, we have
provided around 200-word background information about the aim of this research at the
beginning of the questionnaire. We also made it explicit that there would be little potential
harm as participants did not need to provide any sensitive personal information (e.g., name,
university, phone number), and the quantitative data that we collected would be stored in
encrypted online files. In addition, we set the questionnaire's first question as “Are you willing
to give your informed consent and participate in this online survey?”, which is a required one.
Participants could access the following items only after they clicked on “Yes”.

Data analysis

7
Innovation in Language Learning and Teaching

As aforementioned, this study collected 405 questionnaire responses. To process the large data
asset, SPSS 26.0 and AMOS 26.0 were utilized to conduct structural equation modeling
analyses in five steps. The first was concerned with data screening. We thoroughly examined
issues including respondents’ misconduct (e.g., insufficient questionnaire completion time),
outliers, errors, and missing values. Meantime, the data’s normal distribution was checked by
calculating the 22 items’ skewness and kurtosis values respectively. Second, we computed
Cronbach’s alpha (α) for each construct to measure the questionnaire's internal consistency. As
indicated in Table 2, the five TAM constructs’ α values were .78 (PEU), .80 (PU), .83 (AT), .86
(BI), and .82 (AU). These values were all higher than the .7 recommended by Kline (2015) and
thus suggested that reliability has been established.

Thirdly, the questionnaire’ construct validity, convergent validity, and discriminant validity
were checked in turn. To this end, the KMO-value (.95) and Bartlett’s Test of Sphericity (χ 2
= 5123.53; df = 231; p < .001) were calculated. Also, as shown in Table 2, the standardized
factor loadings (all higher than .4) and the communality values (all larger than .4) supported
the achievement of construct validity. Convergent validity was examined by deriving the
composite reliability (CR) score and average variance extracted (AVE) score for each factor.
Table 3 demonstrated that the five factors’ CR scores all exceeded the .7 limit, while only the
factor AU’s AVE value (.47) did not surpass the threshold value of .5 advised by Kline (2015).
Given .47 was relatively close to .5 and the high CR value of AU (.81), it could be accepted
that the convergent validity was constructed. The discriminant validity was checked by
comparing the five factors’ square roots of AVE and inter-factor correlation coefficients (see
Table 3). The results indicated the five factors were statistically different from each other.

Fourthly, capitalizing on the maximum likelihood method, the measurement model among the
five factors was examined and modified in AMOS. Finally, building on the acceptable
measurement model, path analysis and mediation analyses were conducted to verify the inter-
factor relationships.

Table 2. Reliability and Construct Validity

Construct Items Sample (n = 405)

Standardized factor Communality Cronbach’s alpha
loading (> .4) (> .4) (α) (>.7)
PEU Q7 .55 .68 .78
Q13 .60 .60
Q22 .84 .75
Q23 .80 .80
PU Q6 .66 .73 .80
Q8 .70 .75
Q16 .78 .64
Q18 .74 .59
Q19 .75 .61
AT Q10 .69 .56 .83
Q12 .69 .58
Q20 .82 .70
Q26 .78 .67
BI Q14 .77 .68 .86

8
Innovation in Language Learning and Teaching

Q15 .83 .74

Q21 .85 .78
AU Q9 .64 .70 .82
Q11 .60 .73
Q17 .69 .55
Q25 .66 .69
Q27 .81 .66
Cronbach’s alpha (α) of .94
the entire questionnaire
(>.7)
KMO (>.6) .95
Bartlett's Test of Approx. Chi-Square 5123.53
Sphericity Degree of Freedom 231
p value (<0.05) 0.000
Note: PEU = Perceived Ease of Use; PU = Perceived Usefulness; AT = Attitude; BI = Behavioral Intention;
AU = Actual Use

Table 3. Convergent Validity and Discriminant Validity

AVE CR The square root of AVE

(>0.5) (>0.7) 1 2 3 4 5
1 PEU 0.51 0.80 0.71
2 PU 0.53 0.85 0.55 0.73
3 AT 0.55 0.83 0.58 0.72 0.74
4 BI 0.66 0.86 0.51 0.69 0.70 0.82
5 AU 0.47 0.81 0.36 0.63 0.62 0.59 0.68

Note: The square root of AVE is demonstrated along the diagonal line in bold. The values below
the diagonal display the correlation coefficients (r) between the constructs.

Note: PEU = Perceived Ease of Use; PU = Perceived Usefulness; AT = Attitude; BI = Behavioral Intention;
AU = Actual Use

Findings

Descriptive statistics

Table 4 displayed the Mean (M), Standard Deviation (SD), Skewness, and Kurtosis for each
item. The 22 items had Skewness values spanning from -0.44 to 0.71 (lower than the
recommended value |3.0|) and Kurtosis values varying between -0.77 and -0.13 (smaller than
the recommended value |8.0|). These suggested a symmetrical and normal distribution of data
(Kline, 2015). The Mean scores for individual items lay in the range of 4.00 to 4.61, with a
corresponding SD fluctuating between 0.87 and 1.19. High Mean values (all above 4.00) and
the low degree of variability in their responses demonstrated participants’ positive perceptions
and attitudes towards the usefulness and functionality of ChatGPT in their informal language
learning. For instance, participants usually used ChatGPT to efficiently collect and integrate
English learning information and resources online (Q24, M = 4.61, SD = 0.98). They might
also leverage ChatGPT to improve their English writing and reading abilities through AI-
powered proofreading (Q17, M = 4.35, SD = 1.08) or to better understand the English language
and culture (Q25, M = 4.18, SD = 1.08).

9
Innovation in Language Learning and Teaching

Table 4. Descriptive Statistics

Constructs Items Mean SD Skewness Kurtosis

Q7 4.45 0.93 0.11 -0.32
Q13 4.51 0.87 0.33 -0.40
PEU Q22 4.36 0.99 -0.44 -0.13
Q23 4.38 0.98 0.17 -0.40
Q6 4.29 0.91 0.53 -0.24
Q8 4.39 0.93 0.10 -0.33
PU Q16 4.52 1.01 0.17 -0.59
Q18 4.51 0.97 -0.06 -0.43
Q19 4.36 1.07 -0.10 -0.48
Q10 4.61 1.01 0.44 -0.59
Q12 4.45 0.96 0.01 -0.29
AT Q20 4.32 1.03 0.19 -0.52
Q26 4.56 1.01 0.71 -0.72
Q14 4.42 1.07 -0.04 -0.48
BI Q15 4.30 1.16 -0.35 -0.50
Q21 4.48 1.09 0.45 -0.77
Q9 4.00 1.11 -0.37 -0.23
Q11 4.02 1.19 -0.09 -0.49
Q17 4.35 1.08 -0.07 -0.43
AU Q24 4.61 0.98 0.23 -0.59
Q25 4.18 1.08 0.17 -0.52
Q27 4.23 1.05 0.07 -0.43
Note: PEU = Perceived Ease of Use; PU = Perceived Usefulness; AT = Attitude; BI = Behavioral Intention;
AU = Actual Use

The measurement model

The measurement model was constructed by examining various indices that assessed model
fitness, including the ratio between χ2 and degree of freedom (χ2/df), comparative fit index
(CFI), normed fit index (NFI), incremental fit index (IFI), root mean square of approximation
(RMSEA), Tucker-Lewis index (TLI), standardized root mean squared residual (SRMR).
According to Table 5, the initial model was deemed unacceptable, as five out of seven indices
were inadequate. In this case, modification indices (MI) were employed to detect items with
high cross-loadings that potentially hindered the statistical construction of a valid model.
Consequently, items Q7 and Q24 were eliminated, and correlations among the errors of Q6 and
Q8, Q9 and Q11, and Q18 and Q19 were established in response to atypical MI values
suggested by AMOS. As illustrated in Table 5, the adjustments led to a well-fitted model that
met the threshold criteria for all indices.

Table 5. Goodness-of-fit Indices of the Measurement Models

χ CFI NFI IFI RMSEA TLI SRMR

2
/df
The initial measurement model 3.57 0.90 0.86 0.90 0.08 0.86 0.07
The revised measurement 2.74 0.94 0.91 0.94 0.07 0.93 0.06
model
Recommended values <3 > 0 .90 > 0.90 > 0.90 < 0.10 > 0.90 ≤ 0.10
(Kline, 2015)

The structural model and mediation analysis

10
Innovation in Language Learning and Teaching

Table 6 indicated the results of testing five hypotheses of which four were confirmed by the
data, while only one failed to be supported. To be specific, PEU exhibited a positive impact on
PU (β = .56, p < .001, t-value = 8.92), but it failed to predict AT (β = .03, p = .50, t-value =
0.67). In contrast, PU could serve as a significant predictor of AT (β = .98, p < .001, t-value =
12.08). Additionally, AT was observed to have a statistically significant effect on BI (β = .98,
p < .001, t-value = 15.29) which plays a positive role in predicting AU (β = .77, p < .001, t-
value = 11.19).

Looking at R2 scores in Figure 2, it became evident that PEU contributed to 31% of the total
variance in PU. PU and PEU combined accounted for a considerable portion of the variance
(99%) in AT. Also noted is that the model explained 95% of the variance of AT and 59% of
the total variance of AU. Additionally, a close examination of the inter-factor correlation
coefficients (r) in Table 3 could find that the four confirmed hypotheses’ r values were all
greater than the .5 recommended by (Cohen, 1988), which revealed that the four statistically
significant paths (i.e., PEUàPU, PUàAT, ATàBI, BIàAU) were in a strong effect size
range.

Table 6. Hypotheses Test Results

Note: p＜0.001***; A critical T-value of 1.96 is recommended by Steven (2009); PEU=Perceived Ease of
Use; PU = Perceived Usefulness; AT = Attitude; BI = Behavioral Intention; AU = Actual Use

While PEU failed to predict AT directly, the paths PEUàPU and PUàAT were supported. To
further clarify the relationship between PEU and AT, we proceeded with a test to examine the
mediating role of PU in the path from PEU to AT. Building on 5000 samples and 95%
confidence intervals, a bootstrapped analysis was performed in AMOS. As shown in Table 6,
the PEU-AT indirect effect was calculated to be .48 and the lower and upper bounds were .35
and .64 respectively, without crossing zero. The statistical significance of this indirect effect
was highly robust (p<0.001). These findings suggested that PEU could positively influence AT
through the full mediator PU.

11
Innovation in Language Learning and Teaching

Figure 2. The Final Structural Model

Note: R2 (PU = 31%; AT = 99%; BI = 95%; AU = 59%); PEU = Perceived Ease of Use; PU = Perceived
Usefulness; AT = Attitude; BI = Behavioral Intention; AU = Actual Use
Figure 2 Alt Text: The final structural model elaborating on the inter-factor relationships
among PEU, PU, AT, BI, and AU, showing that PEU can influence AT indirectly through PU,
while AT can predict BI, and finally BI predicts AU.

Discussion
The present research utilized a modified TAM questionnaire to evaluate the process by which
EFL learners accept ChatGPT in their informal digital learning of English. Five out of the six
hypotheses are confirmed in the conceptual model, which adds credit to the explanatory power
of the TAM model in learners’ acceptance of newly emerging technologies in the field of
CALL (Sun & Mei, 2022; Yang & Wang, 2019). The quantitative results have highlighted five
major findings that should be discussed. The first finding was generated based on descriptive
statistics, while the rest four findings were extracted from path analysis and mediation analysis.

First, although the aim of this study is to validate a model of how EFL learners accept ChatGPT
in their IDLE activities, the descriptive results of the variable Actual Use may add to our
understanding of the diverse ways in which ChatGPT is utilized to facilitate learners’ informal
English language learning. In this study, as the means of all six items in the variable Actual
Use are no less than 4.00 (see Table 4), it can be argued that participants generally agree that
they use ChatGPT to engage in various English learning activities beyond the classroom such
as “Q17 I use ChatGPT to enhance my English reading and writing abilities” (M = 4.35, SD =
1.08), “Q25 I use ChatGPT to help me better understand the English language and culture” (M
= 4.18, SD = 1.08). These results are compatible with the point that as a highly-capable chatbot

12
Innovation in Language Learning and Teaching

that was trained on a large volume of data written in English, ChatGPT can act as a powerful
and authentic language-learning tool for EFL learners (Cai, 2023; Sharma, 2023). In this sense,
although this study does not generate insights into the extent to which using ChatGPT can result
in language learners’ increased proficiency or language confidence, it confirms that ChatGPT
has the potential to be “the guardian angel” to assist learners’ engagement with IDLE activities
creatively and effectively (Tlili et al., 2023, p. 1).

Second, this study finds that while Perceived Ease of Use can influence Perceived Usefulness
positively and significantly, it fails to predict Attitude. The significant path from Perceived
Ease of Use to Perceived Usefulness chimes with the extant TAM literature (Alfadda & Mahdi,
2021; Bailey et al., 2022; Davis, 1986; Yang & Wang, 2019). In addition, the variance of
Perceived Usefulness being explained by Perceived Ease of Use (R2 = 31%) and the correlation
coefficient between Perceived Ease of Use and Perceived Usefulness (r = .55) evince that
Perceived Ease of Use serves as a stable and strong predictor for Perceived Usefulness in this
study. Also noted is the insignificant impact of Perceived Ease of Use on Attitude, which
contradicts the results of prior studies (Alfadda & Mahdi, 2021; Bailey et al., 2022). This
surprisingly unsupported hypothesis may add to our understanding that while learners tend to
relate the easy-to-use feature of ChatGPT to its usefulness in their out-of-class English learning,
the only advantage of easy operation cannot lead to their positive attitudes toward ChatGPT
usage directly.

The third primary finding is that Perceived Usefulness can positively and significantly affect
Attitude and also fully mediate the relationship between Perceived Ease of Use and Attitude.
The correlation coefficient (r = .72) between Perceived Usefulness and Attitude and the R2 of
Attitude (99%) help to reveal the considerably positive relationship between Perceived
Usefulness and Attitude, which is consistent with the recent TAM studies that reiterate the
strong effect size of the significant path from Perceived Usefulness to Attitude (e.g., Al-Emran
et al., 2018; Sun & Mei, 2022). The other important part of the second finding is the mediating
role of Perceived Usefulness in the path from Perceived Ease of Use and Attitude, which
receives relatively less attention within the literature at the intersection of TAM and CALL
(Al-Emran et al., 2018; Bailey et al., 2022). With the empirical evidence for the mediating
effect of Perceived Usefulness, this study can extend the assertion that although EFL learners
with a higher level of Perceived Ease of Use may not hold positive attitudes toward ChatGPT
directly, they may gradually foster optimistic attitudes and interest in ChatGPT while
establishing the perception of the usefulness of this new AI technology.

Fourth, Attitude plays an extremely significant role in predicting learners’ Behavioral Intention.
This suggests that once EFL learners have established positive attitudes toward ChatGPT, they
are more likely to plan to use ChatGPT in IDLE activities. Interestingly, in recent years, there
have been scholars calling for the elimination of the variable Attitude in the TAM model on
the ground that technology acceptance behaviors could be accounted for singly by measuring
the impacts of perceived beliefs on Behavioral Intention (Venkatesh, 2000; Yang & Wang,
2019). However, in this study, the finding refutes such an assumption by providing evidence
that attitude is an indispensable part of the TAM conceptual model, which explains 95% of the
variance in learners’ behavioral intention to use technology. Notably, the fourth finding is also
reminiscent of the study of Belda-Medina and Calvo-Ferrer (2022) which focuses on Spanish
and Polish university EFL learners’ adoption of intelligent conversation agents in English
language learning. They reported that while participants indicated high levels of perceived
usefulness and attitudes toward integrating chatbots into their language learning, students only
demonstrated moderate interest in leveraging AI chatbots in their future learning. The different

13
Innovation in Language Learning and Teaching

explanatory power of the variable Attitude may be related to the functionality of the chatbot
itself. As Belda-Medina and Calvo-Ferrer (2022) cautioned in their research, despite certain
levels of interest, EFL learners might be less willing to wholeheartedly engage with
chatbots/virtual conversation partners who are not as intelligent as real human beings. In
contrast, in the present study, the focused technology ChatGPT appears to be more able to
attract students’ intended behavioral usage due to its unprecedented abilities in the upcoming
age of artificial general intelligence (Zhou et al., 2023).

Finally, it is found that Behavioral Intention can generate a strongly significant and positive
impact on Actual Use with the fact that 59% of the change in Actual Use is accounted for by
Behavioral Intention. Aligning with what has been reported in research that examines learners’
acceptance of educational technology (Al-Emran et al., 2018; Alfadda & Mahdi, 2022; Sun &
Mei, 2022), this finding enables us to make the case that language learners’ actual practices of
using ChatGPT in informal digital learning of English are predicated on their willingness and
intention to use this technology for language learning purposes. Altogether, the five findings
shed light on the ways in which EFL learners recognize and accept ChatGPT to learn English
in the out-of-class context through constructing an explicit structural model supported by the
large quantitative data set.

Conclusion

This quantitative cross-sectional study built on the TAM model to explain EFL learners’
acceptance of ChatGPT in their self-directed English learning beyond the classroom. By
offering the descriptive statistics of items in the variable Actual Use, this study first contributed
to understanding how students utilize ChatGPT for various language learning purposes (e.g.,
collecting English learning information online). By establishing a theoretically supported
model, the study revealed that while Perceived Ease of Use fails to predict learners’ attitudes
toward ChatGPT directly, it leaves an impact on attitudes through the full mediator Perceived
Usefulness. It was also found that learners who take positive attitudes toward the usefulness of
ChatGPT tend to demonstrate a higher level of Behavioral Intention to use, which positively
and strongly predicts their Actual use of this AI-powered tool in English learning outside of
the classroom. Such a modified and validated TAM model that accounts for EFL learners’
recognition of ChatGPT constitutes the major theoretical significance of this study.

In terms of practical significance, the research findings provide a reference for educators and
policymakers in the heated debate over embracing or banning ChatGPT in language education
(Cai, 2023). In this study, the positive attitudes and frequent use of ChatGPT in EFL learners’
IDLE activities highlight that ChatGPT represents more opportunities to learn English
effectively in the era of AI, at least for today’s students who may be able “to use technology
more effortlessly and intuitively than their teachers” (Rudolph et al., 2023, p. 14). In this sense,
although ChatGPT has imposed challenges to assessment, academic integrity, and other aspects
of education (Sharma, 2023), it is important to better train students on how to effectively utilize
the technology rather than dismissing the potential benefits it offers language learners. Also,
as learners in this study are generally willing to draw on the affordances of ChatGPT to engage
in various IDLE activities, we would like to call for more language teachers’ attention to
facilitating and encouraging students’ personalized learning beyond the classroom assisted by
powerful AI technologies. To produce detailed and context-specific implications for
pedagogical practices, there still should be more empirical studies to advance the agenda of
integrating powerful AI technologies into language learning beyond the classroom (Reinders
& Benson, 2017; Reinders et al., 2022).

14
Innovation in Language Learning and Teaching

Methodologically, despite its strengths and the novelty of the topic, the present research has
some limitations. First, this study adopted purposive sampling techniques and recruited
participants on social media platforms based on the consideration of feasibility; however, future
research may follow random sampling or stratified probability sampling (e.g., based on
education level) to generalize results to a larger population. Second, this study only provides a
quick snapshot of the use of ChatGPT in informal digital learning of English. A full
understanding of the affordances and constraints of ChatGPT should build on longitudinal
studies that integrate multiple data sources, especially qualitative in-depth data (see Liu &
Darvin, 2023). Third, to make the online questionnaire respondent-friendly, we did not
incorporate items of external variables (e.g., computer self-efficacy). As such, future attention
may be directed to the degree to which external variables account for users’ acceptance of the
evolving AI technologies. Fourth, dependent variables like perceived language proficiency or
affective gains were not taken on board in this study as we believe that understanding the
impact of ChatGPT on language learners’ linguistic or emotional gains should be based on
users’ long-term engagement with it (e.g., more than one year).

15
Innovation in Language Learning and Teaching

References

Al-Emran, M., Mezhuyev, V., & Kamaludin, A. (2018). Technology Acceptance Model in
M-learning context: A systematic review. Computers & Education, 125, 389–412.
https://doi.org/10.1016/j.compedu.2018.06.008
Alfadda, H. A., & Mahdi, H. S. (2021). Measuring students’ use of zoom application in
language course based on the technology acceptance model (TAM). Journal of
Psycholinguistic Research, 50(4), 883–900. https://doi.org/10.1007/s10936-020-09752-
1
Bailey, D. R., Almusharraf, N., & Almusharraf, A. (2022). Video conferencing in the e-
learning context: Explaining learning outcome with the technology acceptance
model. Education and information technologies, 27(6), 7679–7698.
https://doi.org/10.1007/s10639-022-10949-1
Belda-Medina, J., & Calvo-Ferrer, J. R. (2022). Using chatbots as AI conversational partners
in language learning. Applied Sciences, 12(17), 8427.
https://doi.org/10.3390/app12178427
Bibauw, S., Van den Noortgate, W., François, T., & Desmet, P. (2022). Dialogue systems for
language learning: A meta-analysis. Language Learning & Technology, 26(1), 1–24.
https://hdl.handle.net/10125/73488
Cai, W. (2023, March 10). ChatGPT can be powerful tool for language learning. University
Affairs. https://www.universityaffairs.ca/career-advice/career-advice-article/chatgpt-
can-be-powerful-tool-for-language-learning/
Chun, D. M. (2016). The role of technology in SLA research. Language Learning &
Technology, 20(2), 98–115. http://llt.msu.edu/issues/june2016/chun.pdf
Chomsky, N. (2023, March 11). The false promise of ChatGPT. The New York Times.
https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
Ebadi, S., & Amini, A. (2022). Examining the roles of social presence and human-likeness on
Iranian EFL learners’ motivation using artificial intelligence technology: A case of
CSIEC chatbot. Interactive Learning Environments. Advance online publication.
https://doi.org/10.1080/10494820.2022.2096638
Davis, F. D. (1986). A technology acceptance model for empirically testing new end-user
information systems: Theory and results [Doctoral dissertation, MIT Sloan School of
Management]. MIT Theses and Dissertations Archive.
http://dspace.mit.edu/handle/1721.1/15192
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of
information technology. Management Information System Quarterly, 13(3), 983–1003.
https://doi.org/10.2307/249008
Godwin-Jones, R. (2023). Emerging spaces for language learning: AI bots, ambient
intelligence, and the metaverse. Language Learning & Technology, 27(2), 6–27.
https://hdl.handle.net/10125/73501
Haristiani, N., & Rifa’i, M. M. (2020). Combining chatbot and social media: Enhancing
personal learning environment (PLE) in language learning. Indonesian Journal of
Science and Technology, 5(3), 487–506. https://doi.org/10.17509/ijost.v5i3.28687
Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—Are they
really useful? A systematic review of chatbot‐supported language learning. Journal of
Computer Assisted Learning, 38(1), 237–257. https://doi.org/10.1111/jcal.12610
Kiaer, J., & Lee, J. (Eds.) (2023). Call for papers for a special issue on using ChatGPT in
language learning [Special issue]. Languages.
https://www.mdpi.com/journal/languages/special_issues/K1Z08ODH6V

16
Innovation in Language Learning and Teaching

Kim, N. Y. (2019). A study on the use of artificial intelligence chatbots for improving
English grammar skills. Journal of Digital Convergence, 17(8), 37–46.
https://doi.org/10.14400/JDC.2019.17.8.037
Kim, N. Y., Cha, Y., & Kim, H. S. (2019). Future English learning: Chatbots and artificial
intelligence. Multimedia-Assisted Language Learning, 22(3), 32–53.
https://www.kci.go.kr/kciportal/ci/sereArticleSearch/ciSereArtiView.kci?sereArticleSe
archBean.artiId=ART002505056
Kline, R. B. (2015). Principles and practice of structural equation modeling. Guilford
publications.
Lee, J. S., & Drajati, N. A. (2019). Affective variables and informal digital learning of
English: Keys to willingness to communicate in a second language. Australasian Journal
of Educational Technology, 35(5), 168–182. https://doi.org/10.14742/ajet.5177
Lee, J. S., & Xie, Q. (2022). Profiling the affective characteristics of EFL learners’ digital
informal learning: A person-centered approach. Innovation in Language Learning and
Teaching. Advance online publication. https://doi.org/10.1080/17501229.2022.2085713
Liu, G., & Darvin, R. (2023). From rural China to the digital wilds: Negotiating digital
repertoires to claim the right to speak online. TESOL Quarterly. Advance online
publication. https://doi.org/10.1002/tesq.3233
Liu, G., Ma, C., Bao, J., & Liu, Z. (2023a). Toward a model of informal digital learning of
English and intercultural competence: A large-scale structural equation modeling
approach. Computer Assisted Language Learning. Advance online publication.
https://doi.org/10.1080/09588221.2023.2191652
Liu, G., Zhang, Y., & Zhang, R. (2023b). Bridging imagination and informal digital learning
of English: A mixed-method investigation. Journal of Multilingual and Multicultural
Development. Advance online publication.
https://doi.org/10.1080/01434632.2023.2173214
Paltridge, B., & Phakiti, A. (Eds.). (2015). Research methods in applied linguistics: A
practical resource. Bloomsbury Publishing.
Reinders, H., Lai, C., & Sundqvist, P. (Eds.). (2022). The Routledge handbook of language
learning and teaching beyond the classroom. Routledge.
Reinders, H., & Benson, P. (2017). Research agenda: Language learning beyond the
classroom. Language Teaching, 50(4), 561–578.
https://doi.org/10.1017/S0261444817000192
Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional
assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 1–
22. https://doi.org/10.37074/jalt.2023.6.1.9
Sharadgah, T. A., & Sa’di, R. A. (2022). A systematic review of research on the use of
artificial intelligence in English language teaching and learning (2015-2021): What are
the current effects? Journal of Information Technology Education: Research, 21, 337–
377. https://doi.org/10.28945/4999
Sharma, Y. (2023, March 10). What ChatGPT means for linguistic diversity and language
learning. University World News.
https://www.universityworldnews.com/post.php?story=20230223132734294
Shemesh, H. (2023, March 10). The ultimate guide for using ChatGPT English learning. The
Accent's Way Magazine. https://hadarshemesh.com/magazine/chatgpt-for-learning-
english/
Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.).
Routledge.
Sun, P. P., & Mei, B. (2022). Modeling preservice Chinese-as-a-second/foreign-language
teachers’ adoption of educational technology: A technology acceptance

17
 Innovation in Language Learning and Teaching

perspective. Computer Assisted Language Learning, 35(4), 816–839.

https://doi.org/10.1080/09588221.2020.1750430
Tam, A. (2023, June 22). What are large language models. Machine Learning Mastery.
https://machinelearningmastery.com/what-are-large-language-
models/#:~:text=There%20are%20multiple%20large%20language,language%20and%2
0can%20generate%20text.
Thorne, S. L., & Black, R. W. (2007). Language and literacy development in computer-
mediated contexts and communities. Annual review of applied linguistics, 27, 133–160.
https://doi.org/10.1017/S0267190508070074
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., &
Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study
of using chatbots in education. Smart Learning Environments, 10(1), 1–24.
https://doi.org/10.1186/s40561-023-00237-x
Venkatesh, V. (2000). Determinants of perceived ease of use: Integrating control, intrinsic
motivation, and emotion into the technology acceptance model. Information Systems
Research, 11, 342–365. https://www.jstor.org/stable/23011042
Warschauer, M., & Xu, Y. (2022). Call for papers for a special issue on artificial intelligence
for language learning. Language Learning & Technology, 26(2), 149–150.
http://doi.org/10125/73483
Yang, Y., & Wang, X. (2019). Modeling the intention to use machine translation for student
translators: An extension of Technology Acceptance Model. Computers &
Education, 133, 116–126. https://doi.org/10.1016/j.compedu.2019.01.015
Zhang, Y., & Liu, G. (2022). Revisiting Informal Digital Learning of English (IDLE): a
structural equation modeling approach in a university EFL context. Computer-Assisted
Language Learning. Advance online publication.
http://dx.doi.org/10.1080/09588221.2022.2134424
Zhou, J., Ke, P., Qiu, X., Huang, M., & Zhang, J. (2023). ChatGPT: potential, prospects, and
limitations. Frontiers of Information Technology & Electronic Engineering. Advance
online publication. https://doi.org/10.1631/FITEE.2300089

18

View publication stats