Assessing Writing 57 (2023) 100745

Contents lists available at ScienceDirect

Assessing Writing
journal homepage: www.elsevier.com/locate/asw

Using ChatGPT for second language writing: Pitfalls and potentials

Jessie S. Barrot *
College of Education, Arts and Sciences, National University, Manila, Philippines

A R T I C L E I N F O A B S T R A C T

Keywords: Recent advances in artificial intelligence have given rise to the use of chatbots as a viable tool for
ChatGPT language learning. One such tool is ChatGPT, which engages users in natural and human-like
Chatbots interactive experiences. While ChatGPT has the potential to be an effective tutor and source of
Immersive technology
language input, some academics have expressed concerns about its impact on writing pedagogy
L2 writing
Computer-assisted language learning
and academic integrity. Thus, this tech review aims to explore the potential benefits and chal­
lenges of using ChatGPT for second language (L2) writing. This review concludes with some
recommendations for L2 writing classroom practices.

1. Introduction

Chatbots have garnered considerable attention in recent years as an artificial intelligence (AI) tool that engages users in natural and
human-like interactive experiences through audio or text. They respond contextually to a variety of questions and prompts and provide
differentiated feedback (Huang et al., 2022; Kohnke, 2022). ChatGPT is one such chatbot that has taken the Internet by storm since its
launch in November 2022. This AI-powered chatbot interacts in a conversational way by responding to questions, admitting mistakes,
challenging incorrect premises, and declining inappropriate requests in a dialogue form (OpenAI, 2023). While ChatGPT can be a
powerful L2 writing tool, some academics have expressed concerns about its impact on writing pedagogy and academic integrity
(Thorp, 2023). In fact, some universities temporarily ban students from using ChatGPT and other AI-based tools. Nonetheless, ChatGPT
can potentially address some of the pressing needs of second language (L2) writers, including timely and adaptive feedback, a platform
for practice writing, and a readily available and dependable writing assistant tool. Thus, this technology review aims to explore the
potential benefits and challenges of using ChatGPT for L2 writing, with focus on important writing constructs, such as language form
and style, coherence and cohesion, and writing conventions. The review concludes with some recommendations for L2 writing
classroom practices and assessment.

2. ChatGPT: nature and affordances

ChatGPT, short for Chat Generative Pre-trained Transformer, is an AI-powered chatbot developed by the United States-based
startup OpenAI. It is powered by computing techniques and a vast amount of information to connect words and ideas together and
understand prompts in context. It can be accessed through desktop computers and mobile phones using different web browsers, such as
Safari, Firefox, and Chrome. Powered by a large language model (LLM), the tool is capable of understanding and producing human-like
responses across a wide range of topics. To use ChatGPT, you must register and create an account at https://chat.openai.com/auth/
login. Once you have created an account, verify it through your email or mobile phone. To start using ChatGPT, simply type your

* Correspondence to: National University, 551 MF Jhocson St., Sampaloc, Manila 1008, Philippines.
E-mail address: jessiebarrot@yahoo.com.

https://doi.org/10.1016/j.asw.2023.100745
Received 1 March 2023; Received in revised form 19 May 2023; Accepted 21 May 2023
Available online 25 May 2023
1075-2935/© 2023 Elsevier Inc. All rights reserved.
J.S. Barrot Assessing Writing 57 (2023) 100745

question or prompt in the search box. Fig. 1 displays the ChatGPT dashboard that displays sample prompts, its capabilities and lim­
itations, Clear Conversations tab, Updates and Frequently Asked Questions tab, Upgrade tab, and New Chat tab. You can also find your
search history, such as chatbots in classrooms and complexity of language, can also be found below the New Chat tab.
ChatGPT offers functionalities that serve several purposes. It holds a vast amount of knowledge and retrieves relevant information
to write in a human-like way, making it a rich source of information for language input and language practice anytime and anywhere.
ChatGPT’s knowledge base is derived from diverse domains, genres, and sources, including books, articles, websites, and other online
content. Rather than being stored in a traditional database, the corpus is saved and processed as an extensive collection of statistical
patterns and associations. ChatGPT utilizes this corpus to generate context-specific and appropriate responses. To ensure that this AI
tool has access to the most up-to-date and relevant information, the corpus is continuously updated and expanded. The latest iteration,
GPT-4, was released in March 2023, and has the ability to process 32,000 tokens simultaneously and accept visual queries as well as
possesses 100 trillion parameters. Furthermore, this chatbot remembers previous conversations, correcting itself when errors occur. In
the same way, it allows users to make follow-up corrections and declines any inappropriate requests (OpenAI, 2023). L2 language
learners can utilize these affordances for sustained and coherent conversations without any fear of being judged. Hence, they may feel
less stressed about their simulated language learning experience.
Based on my own engagement with ChatGPT, it appears that the tool considers a wide range of writing constructs, such as
pragmatics, semantics, coherence, cohesion, conventions, language style, format, grammar, and syntax. However, capturing certain
aspects of human writing quality, such as emotional depth, writing voice and identity, and rhetorical flexibility, remains a challenge.
Nonetheless, ChatGPT offers a broad range of writing assistance, from simple to complex tasks. It is a valuable writing tool that can
produce full-length human-like texts. The text it generates is typically coherent and grammatically correct, making it a helpful tool for
users to refine their writing and improve their use of language forms. ChatGPT can also generate a comprehensive list of essay topics
based on user’s area of interest. When prompted to recommend essay topics related to artificial intelligence, it can generate around ten
topics about ethics, privacy, biases, singularity, use in healthcare, impact on education, role in climate change, among others. ChatGPT
can further narrow down these topics based on specific user instructions. This affordance is particularly useful in selecting the most
relevant topics within the interest of the L2 writers. In addition to topic generation, ChatGPT can create an outline in various formats. It
can transform any outline into a sentence, topic, alphanumeric, or decimal system format. Organizing one’s thoughts is a common
challenge for students when writing full-length essays (Barrot, 2018; De Smet et al., 2012). With ChatGPT’s outlining feature, students
are exposed to sample outlines that can serve as a guide and can be used as a springboard for creating a well-designed outline
appropriate for their selected topics.
One important phase in the writing process is written corrective feedback. However, quality feedback is not always available to the

Fig. 1. ChatGPT dashboard.

2
 J.S. Barrot
Table 1
Sample list of references generated by ChatGPT.
Number References Remarks

1 Chen, M. Y., & Cheng, W. (2020). Automated Writing Evaluation and Feedback in L2 Writing. CALICO Journal, 37(3), 233–257. doi: 10.1558/cj.40015 non-existent
2 Crossley, S. A., & McNamara, D. S. (2019). Automatic analysis of text and discourse with Coh-Metrix. Language Testing, 36(3), 405–430. doi: 10.1177/0265532219827265 existing but
published as a
book
3 Dong, Y., & Zhang, Q. (2020). Investigating the effects of AI-assisted writing instruction on L2 writing development. Journal of Second non-existent
Language Writing, 50, 100701. doi: 10.1016/j.jslw.2020.100701
4 Dikli, S. (2016). Automated Writing Evaluation (AWE) Tools for L2 Writing: Teacher and Student Perceptions, Attitudes, and Expectations. non-existent
CALICO Journal, 33(2), 119–146. doi: 10.1558/cj.v33i2.29406
5 Geng, Y., & Lai, C. (2019). The Effect of Automated Writing Evaluation Feedback on L2 Writing Accuracy and Fluency. Language non-existent
Learning & Technology, 23(1), 112–131. doi: 10125/44610
3

6 Lee, J. (2017). Artificial intelligence and its effects on L2 writing: The current state of research and future directions. Journal of Second non-existent
Language Writing, 36, 27–40. doi: 10.1016/j.jslw.2017.02.003
7 Li, X., & He, D. (2019). The Effect of Automated Writing Evaluation on L2 Writing Quality: A Meta-Analysis. Educational Psychology non-existent
Review, 31(1), 1–27. doi: 10.1007/s10648–017–9419–4
8 Lin, Z., & Yang, J. (2020). Investigating the effects of artificial intelligence-assisted writing instruction on L2 learners’ writing performance and attitude. Computer Assisted non-existent
Language Learning, 33(5–6), 500–522. doi: 10.1080/09588221.2019.1617167
9 Lu, X., & Ai, H. (2020). Can artificial intelligence provide better feedback than human teachers for L2 writing? Journal of Second Language non-existent
Writing, 50, 100688. doi: 10.1016/j.jslw.2020.100688
10 Wang, Y., & Li, J. (2019). Automated Writing Evaluation for L2 Learners: A Systematic Review and Future Directions. Journal of Educational non-existent
Technology & Society, 22(3), 191–202. doi: 10.14742/ijcsi.2019.22.03.11

Assessing Writing 57 (2023) 100745

J.S. Barrot Assessing Writing 57 (2023) 100745

students. ChatGPT addresses this need as it offers useful features linked to writing assessment. This AI tool can automatically grade
students’ written work. The system assigns a score to the paper based on predefined criteria and provides specific comments to support
the assigned score. ChatGPT’s decision and interpretation about each predefined criterion is based on contents from a wide range of
online sources processed as a large set of statistical patterns and associations. Similarly, by prompting the tool to provide feedback (e.
g., Provide feedback on this text), it highlights the strengths and weaknesses of the text and offers recommendations on how the paper
can be improved. In addition to assessing stylistics and mechanics, the tool can evaluate the richness and relevance of content, clarity
of audience, clarity of purpose, depth of analysis, focus, and organization. For example, after I prompted ChatGPT to identify the target
audience of a text about the importance of part-time jobs for college students, it responded that “the text has no clear audience” and
“seems to target a general audience.” The tool then suggested possible target audiences based on its content (i.e., students working
part-time). In the same way, this tool provides a score and examines the essay’s focus, specificity of examples, consistency of argument,
clarity of explanation, and organization of ideas when I asked it to assess an essay’s reasoning. When instructed to edit a text (e.g., edit
this text), it offers corrections on grammatical accuracy, syntax, and vocabulary. While ChatGPT can be useful as an L2 writing as­
sistant, adopting it as a supplemental tool for writing essays is highly advisable rather than relying on it as a content creator. A helpful
approach is to encourage students to write their original outputs first and then refine them using ChatGPT. This approach helps
students develop their writing skills while using ChatGPT to improve their written outputs further.

3. Issues and limitations

ChatGPT has great potential as a language learning tool but has some limitations. The developers acknowledged that it may
produce inaccurate and unintelligible responses. For example, when asked about Curriculum 5.0, it only briefly describes 21st-century
curriculum. ChatGPT is also sensitive to minor adjustments in input phrasing and multiple attempts with the same prompt. For
instance, when asked to explain “Project Essay,” ChatGPT responded, “It is unclear what you mean by project essay.” However, when
prompted with the addition of “scoring” at the end of the sentence, ChatGPT provided a clear description of “Project Essay” as an
automated essay scoring tool. Another limitation of ChatGPT is its frequent use of unnecessary statements, such as “As an AI language
model, I don’t have personal opinions.” In the context of writing, ChatGPT may have limited capabilities when checking for potentially
plagiarized texts and adjusting the text to a specific group of audience. A certain level of template rigidity of writing was also observed.
For instance, when given a prompt to write an essay about different concepts, such as smoking and online game addiction, ChatGPT
tends to follow a specific structure. It typically starts with a definition and a brief history of the concept. Then, it mainly discusses the
effects on the body and summary of main points, final thoughts, and call to action as the conclusion. While it can generate a list of
references for a specific topic and format (e.g., American Psychological Association, Chicago Manual of Style, and Modern Language
Association) them according to users’ preferences, the list contains unrelated, inaccurate bibliographic information, or non-existent
articles. Table 1 shows the sources that ChatGPT generates when prompted to provide a list of journal references about the use of
artificial intelligence in L2 writing with doi.
In addition to the limitations mentioned earlier, ChatGPT presents some challenges to writing instruction and academic integrity
(King & chatGPT, 2023; Rudolph et al., 2023; van Dis et al., 2023). ChatGPT allows students to complete a writing assignment without
much effort. This raises concerns about learning loss, especially in developing critical and creative thinking. This will most likely to
arise when students become too reliant on the tool for convenience. Using ChatGPT may also likely decrease students’ creativity and
critical thinking when they are not trained to use it and evaluate the information it offers. ChatGPT also generates original content that
may not be detected by online plagiarism checkers, particularly when the content is rephrased using tools such as QuillBot. As a result,
teachers may have difficulty distinguishing between the students’ work and the text generated by ChatGPT. Moreover, students’
reliance on ChatGPT may undermine some important goals of writing pedagogy, which are to foster creativity, develop critical
thinking, and enhance the ability to articulate ideas with clarity and precision (Yanning, 2017). These skills allow students to connect
their written texts to real-life context and create new understandings to solve real-life problems (Curriculum Planning and Devel­
opment Division, 2020).

4. What L2 writing teachers can do

The availability of ChatGPT presents both challenges and opportunities for L2 writing teachers to recalibrate their classroom
practices. Rather than outright banning ChatGPT, teachers can explore ways to work alongside these AI-based tools and capitalize on
their potential. One strategy to mitigate the potential issues of using ChatGPT is for teachers to emphasize the value of the writing
process. For example, students can document their steps (e.g., topic selection, setting of the context, outlining, drafting, and revising)
to produce their final written output. L2 writing teachers can also teach students to develop a distinctive writing voice and identity,
which ChatGPT would struggle to replicate. Encouraging students to incorporate their personal experiences and share these stories in
their writing can help achieve this objective. Since ChatGPT cannot process more recent information (i.e., October 2021 onwards),
teachers can use more recent and highly localized topics as writing prompts. However, this limitation may be temporary due to the
rapid pace of technological advancements. Lastly, teachers can leverage ChatGPT’s editing capabilities to teach students correct
language forms and style. Students can input the essays into ChatGPT and instruct it to edit their work. Students can then review the
changes made by ChatGPT and decide whether to accept or reject them based on their own assessment.
As an AI tool, ChatGPT may mimic a high degree of human intelligence and excel in many aspects of writing. However, this tool
cannot totally replace humans because it lacks the emotional depth and life experiences that contribute to an individual’s voice,
identity, and distinctiveness in writing. Moreover, the tool is limited in its ability to critically reflect on its own writing and assess the

4
J.S. Barrot Assessing Writing 57 (2023) 100745

accuracy of generated content, particularly on topics with limited consensus. It also lacks a deep understanding of complex concepts
that require higher-order thinking skills, contextualization, common sense, and emotions. Although this tool challenges writing
pedagogy, it also presents opportunities to reassess our writing practices and approaches to teaching writing. As L2 writing teachers,
we can capitalize on ChatGPT’s capabilities while stressing the importance of creativity and human expression. By acknowledging the
limitations of this AI-based tool, we can encourage students to think creatively, treat writing as a means of intellectual and personal
growth, and develop a deeper understanding and appreciation of the writing process.
As an L2 writing scholar and teacher, it is crucial to use ChatGPT purposefully and systematically in writing instruction. To achieve
this, I integrate the tool throughout the writing process to encourage effective writing practices and quality output among students.
Initially, I ask students to select a contemporary topic or issue and brainstorm it with their peers. To enhance their output, I encourage
them to use ChatGPT to generate additional relevant responses. They are then required to source model texts from two credible sources
and examine their structural features. Students may use ChatGPT to get additional model texts. After that, they conduct preliminary
research on their topic and create an outline following the designated format. To avoid restricting students’ creativity and critical
thinking, they may only use ChatGPT to refine their initial outline. Students then begin to write their essays, using reputable sources
and incorporating personal experience, critical reflection, and writing voice. At this stage, the use of ChatGPT is not allowed. When
students reach the self-assessment phase, they may utilize ChatGPT to receive feedback on their first draft regarding content, orga­
nization, and structure. They will then revise their work based on both their self-evaluation and ChatGPT-based feedback. As students
move towards the final stage of writing, they will use ChatGPT’s editing feature to improve the language style, vocabulary, and
grammar of their final draft. However, since ChatGPT’s ability to detect potential plagiarism is limited, students must use other
plagiarism checking tools, such as Grammarly or Turnitin.
To address the limitations and challenges of using ChatGPT as an L2 writing tool, it is essential to provide clear guidelines to
teachers and students. These guidelines should help them understand the purpose of using the tool in L2 writing classrooms and
provide appropriate ways to engage with this kind of technology, including ethical considerations and precautions. In addition to the
guidelines, hands-on training may be provided to offer opportunities for exploration of ChatGPT’s features and elicit feedback on
engagement with the tool. Collaborative efforts should also be encouraged, with teachers sharing best practices and experiences in
workshops, online forums, and in-person meetings. Students can also create a community of learners where they can share their
experiences, challenges, and ethical practices in using ChatGPT. Lastly, ongoing support is crucial for teachers and students as they
adopt ChatGPT in the writing classrooms. This suggested approach hopes to engage teachers and students behaviorally, emotionally,
and cognitively (Fredricks et al., 2004).

5. Some future research directions

To date, research on ChatGPT is in its early stage of development and continues to expand. This line of research holds significant
developmental potential and deserves serious attention from researchers and teachers. Practitioners can explore systematic ways to
integrate ChatGPT into the writing process and their teaching practices. As demonstrated above, ChatGPT can be utilized as a pre-
writing, during-writing, and post-writing tool, with necessary safeguards to ensure the integrity of writing. L2 writing researchers
may also investigate the linguistic features of texts produced by ChatGPT and compare them to those produced by human writers to
identify differences. This way, teachers can be trained to recognize an AI-generated text while assessing writing. The emergence of AI
tools such as ChatGPT has the potential to support fair and equitable writing assessment by providing consistent feedback on students’
writing, free from predetermined assumptions that a teacher may have about them. This technology can also significantly reduce the
workload of L2 writing teachers while providing personalized and timely feedback that enhances student learning. As a result, stu­
dents, regardless of their background, can have equal opportunities to improve their writing skills. As an automated written corrective
feedback tool, future research may explore how ChatGPT can improve students’ writing accuracy and genre awareness. The way
ChatGPT writes is largely influenced by the language patterns present in the input, which is mainly standard written American English.
However, it can also generate text using other varieties of English, such as British English, Canadian English, and Australian English.
That being said, future versions of ChatGPT may fine-tune texts that follow other varieties of English. There are still many things to
learn about this new tool. As it develops, so as our understanding of it and our ability to harness its potential.

Declaration of Competing Interest

None.

Data Availability

No data was used for the research described in the article.

References

Barrot, J. S. (2018). Using the sociocognitive-transformative approach in writing classrooms: Effects on L2 learners’ writing performance. Reading & Writing Quarterly,
34(2), 187–201.
Curriculum Planning and Development Division. (2020). English language syllabus: Secondary express course. Ministry of Education, Singapore. https://www.moe.gov.
sg/-/media/files/secondary/syllabuses/eng/sec_exp-na_els-2020_syllabus-(1).ashx.

5
J.S. Barrot Assessing Writing 57 (2023) 100745

De Smet, M. J., Brand-Gruwel, S., Broekkamp, H., & Kirschner, P. A. (2012). Write between the lines: Electronic outlining and the organization of text ideas. Computers
in Human Behavior, 28(6), 2107–2116.
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1),
59–109.
Huang, W., Hew, K., & Fryer, L. (2022). Chatbots for language learning—Are they really useful? A systematic review of chatbot-supported language learning. Journal
of Computer Assisted Learning, 38(1), 237–257.
King, M. R., & chatGPT. (2023). A conversation on artificial intelligence, chatbots, and plagiarism in higher education. Cellular and Molecular Bioengineering, 16, 1–2.
Kohnke, L. (2022). A pedagogical chatbot: A supplemental language learning tool. RELC Journal Advance Online Publication. https://doi.org/10.1177/
00336882211067054
OpenAI. (2023). ChatGPT: Optimizing language models for dialogue. 〈https://openai.com/blog/chatgpt/〉.
Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6
(1), 1–22.
Thorp, H. (2023). ChatGPT is fun, but not an author. Science, 379(6630), 313.
van Dis, E. A., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023). ChatGPT: five priorities for research. Nature, 614(7947), 224–226.
Yanning, D. (2017). Teaching and assessing critical thinking in second language writing: An infusion approach. Chinese Journal of Applied Linguistics, 40(4), 431–451.

Jessie S. Barrot is a professor and assistant vice president for research and development at the National University (Philippines). He has published papers in peer-
reviewed journals in the area of L2 writing, curriculum and instruction, computer assisted learning, and language teaching.