Computer Assisted Language Learning

ISSN: 0958-8221 (Print) 1744-3210 (Online) Journal homepage: https://www.tandfonline.com/loi/ncal20

The effectiveness of educational technology

applications on adult English language learners’
writing quality: a meta-analysis

Zhihong Xu, Manjari Banerjee, Gilbert Ramirez, Gang Zhu & Kausalai (Kay)
Wijekumar

To cite this article: Zhihong Xu, Manjari Banerjee, Gilbert Ramirez, Gang Zhu & Kausalai (Kay)
Wijekumar (2019) The effectiveness of educational technology applications on adult English
language learners’ writing quality: a meta-analysis, Computer Assisted Language Learning, 32:1-2,
132-162, DOI: 10.1080/09588221.2018.1501069

To link to this article: https://doi.org/10.1080/09588221.2018.1501069

Published online: 11 Dec 2018.

Submit your article to this journal

Article views: 1235

View related articles

View Crossmark data

Citing articles: 2 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=ncal20
COMPUTER ASSISTED LANGUAGE LEARNING
2018, VOL. 32, NOS. 1-2, 132–162
https://doi.org/10.1080/09588221.2018.1501069

The effectiveness of educational technology

applications on adult English language learners’
writing quality: a meta-analysis
Zhihong Xua , Manjari Banerjeea , Gilbert Ramireza , Gang Zhub
and Kausalai (Kay) Wijekumara
a
Texas A&M University, College Station, TX, USA; bInternational Teacher Education at Faculty
of Education, East China Normal University, Shanghai, China

ABSTRACT KEYWORDS
Learning to write well can be a significant contributor to Educational technology;
English language proficiency. The large population of ELLs; writing; meta-analysis
English language learners (ELLs) has highlighted the need
for educational technology as an important support for the
different approaches to writing instruction for ELLs. In this
meta-analysis, we examined the effectiveness of educa-
tional technology applications on the writing quality of
adult ELLs. Twenty-one independent studies from 16 publi-
cations were included in this review. Our findings con-
firmed that technology applications produce a large effect
size (.93 and 1.28, fixed effect model and random effect
model respectively) as compared to non-technology
instructional methods. We conducted sub-group analyses
with six substantive and methodological factors: type of
technology, genre of writing, program duration, program
intensity, measures outcome, and research design, and fol-
lowed the analyses with research and pedagogical
implications.

Most educational practices focus on writing as a small part of instruc-

tion for second language (L2) learners. However, writing is a much
more important component of language learning for L2 learners.
Writing is a meaningful way to learn a foreign language (Harklau,
2002), and it can play an important role in facilitating second language
development (Williams, 2012). In fact, where speech and writing are
compared, writing has a bigger impact on second language acquisition
(Weissberg, 2000). Additionally, print literacy, or the ability to read
and write in English is viewed as an important aspect of adult English
language learners (ELLs) ability to the master the language (Bigelow &
Schwarz, 2010).

CONTACT Zhihong Xu xuzhihong@tamu.edu Department of Teaching, Learning and Culture, Texas

A&M University, College Station, TX 77843-4222, USA.
ß 2018 Informa UK Limited, trading as Taylor & Francis Group
 COMPUTER ASSISTED LANGUAGE LEARNING 133

Educational technology applications have been shown to be effective in

improving student learning outcomes (Schmid et al., 2014; Tamim,
Bernard, Borokhovski, Abrami, & Schmid, 2011). Several other research
studies have explored a number of different aspects of language learning
that are impacted by educational technology, like the effect of technology
on reading (Cheung & Slavin, 2012, Cheung & Slavin, 2013; Golonka,
Bowles, Frank, Richardson, & Freynik, 2014). The importance of technol-
ogy on the writing skills and writing quality of ELLs in particular has
been examined in numerous studies (Abrams, 2016; Azari, 2017; Dizon,
2016; Shadiev, Hwang, & Huang, 2017; Wang, 2015; Zhu, Mark Shum,
Brian Tse, & Liu, 2016; Zou, Wang, & Xing, 2016). However, there is no
synthesis or meta-analysis on this pressing topic.
The current study examines the cumulative quantitative evidence of
the use of educational technology applications for improving writing
quality of adult ELLs. We use the genre of writing and collaborative
learning theory as the cornerstones of this meta-analysis, identifying the
important contributors to the writing quality of adult ELLs.

Review of literature
Commonly used educational technology applications
Educational technology is defined as a variety of electronic tools and appli-
cations that help deliver learning content and support the learning process
(Cheung & Slavin, 2013). Some examples of these technology applications
include computer-assisted instruction (CAI), integrated learning systems
(ILS), and technology-based curricula (Cheung & Slavin, 2012; Cheung &
Slavin, 2013). Educational technology applications may also be in the form
of classroom-based technologies like course management systems (CMS),
interactive white boards, ePortfolios, individual study tools like electronic
dictionaries, electronic glossary or annotations, intelligent tutoring systems,
grammar checkers, automatic speech recognition (ASR) and pronunciation
programs (Golonka et al., 2014). Golonka et al. (2014) also list network-
based social computing in educational technology applications. Some
examples are virtual world or serious game, chat, social network, blog,
internet forum or message board, and Wiki. Mobile and portable devices
like tablet PC or PDA, iPod, cell phone or smartphone are also a part of
technology applications used in education (Golonka et al., 2014).

Technology applications specific to ELLs

Among all of the technology applications for ELLs, some have been spe-
cifically studied for their effects on learning English as a second
134 Z. XU ET AL.

language. For example, computer mediated glosses and their effects on

second language learners were reviewed in detail by Abraham (2008).
Similarly, iPods and iPads have been researched and found effective in
second language learners’ ability to learn language skills (Liu, Navarrete,
& Wivagg, 2014). Distance education technologies and online learning
instructional components for adult ELLs were reviewed by Coryell and
Chlup (2007), and their findings support the use of these components
for ELLs. Much of the previous research in this field therefore directs the
reader’s attention on how the use of these educational technology appli-
cations can have a positive impact on the reading ability of ELLs.

Technology applications for writing for ELLs

Definition of writing quality for second language writing
L2 performance includes multiple components. The primary attributes of
these components can be comprehensively assessed by the constructs of
complexity, accuracy and fluency (Skehan, 1998; Ellis, 2008; Ellis &
Barkhuizen, 2005). Complexity, accuracy, and fluency (CAF) have been
widely employed as performance descriptors for the assessment of ELLs
(Housen & Kuiken, 2009). The focus of most scoring scales for ELLs are
the above three dimensions. Additionally, the rating scales for L2 writing
also comprise of the evaluation for organization and content (Cumming,
Kantor, Powers, Santos, & Taylor, 2000). Therefore, the evaluation crite-
ria for writing include language use and content. Most of the studies
included in this meta-analysis provided the overall writing score, while a
few of them provided specific scores for particular criteria, like accuracy
(Wichadee, 2013). Even though the included studies chose different
approaches to report the writing scores, they evaluated writing quality
for ELLs. We have based our meta-analysis on the reported scores from
the included studies.

Type of technology for ELLs’ writing

More recent attention has focused on the use of technology like weblog-
based processes (Azari, 2017), smartphones, mobile phones, and personal
digital assistants (Shadiev et al., 2017) as effective tools for improving
writing skills in ELLs. Researchers have also found that computer-sup-
ported collaborative writing environments like Facebook (Dizon, 2016)
and Google Docs (Abrams, 2016) also have a positive impact on the
writing skills of ELLs. Similarly, Zou et al. (2016) and Wang (2015) have
found that Wiki-based collaborative writing is also effective in improving
ELLs’ writing skills. Additionally, applications as simple as word process-
ing software have shown a positive impact on ELLs when compared to
 COMPUTER ASSISTED LANGUAGE LEARNING 135

using paper and pencil (Zhu et al., 2016). In the studies included in this
meta-analysis, we examined technology based on collaborative and non-
collaborative applications.

Cornerstones of the meta-analysis

Genre of writing
The common understanding of genre is that it is a system of classifica-
tion, but some scholars claim that this is a trivial way of treating genres
(Devitt, 1993). Hyon (1996) describes genres as “as structured, events
and as staged, social processes, respectively, sensitizing ESL instructors to
links between formal properties of texts that they teach.” (p. 712).
According to Chandler (1997), genres can also be defined as texts that
have similar family characteristics. Genre-based pedagogies aid student
learning by offering students definitive and methodical explanations of
language usage within the social context, since writing is primarily a
social activity (Hyland, 2003; Hayes, 2000).
Genre of writing is critical in facilitating the mastery of knowledge
about writing for different tasks and different purposes (Gillespie,
Olinghouse, & Graham, 2013). To produce well-written passages, skilled
writers have to figure out the nuances (e.g., organization, language out-
put, language styles) between diverse genres (McCutchen, 1986). Hayes
(2011, 1996) claims that a good writer needs to possess content know-
ledge and also the knowledge of how to write. Knowledge of how to
write consists of the ability to differentiate the needs of the target audi-
ence for a specific writing task, genre knowledge, as well as a writer’s lin-
guistic knowledge. Therefore, genre knowledge and the writers’ ability to
recognize the needs of target audiences have great impact in generating
good written texts.
Empirical studies also support the view that proficiency of genre
knowledge has positive impact on students’ writing quality (e.g.,
Olinghouse & Graham, 2009). Similarly, research studies indicate that
teaching the properties of specific genres in the writing classrooms
enhances students’ writing quality (Graham, Kiuhara, McKeown, &
Harris, 2012). Some studies also imply that L2 learners can recognize
similarities in a text as a community, and draw from their experiences to
understand that specific text (Hyland, 2007).
We use genres of writing as our framework for this meta-analysis. We
believe that L2 acquisition; especially L2 writing skills draw heavily from
genres of writing. Genres are generally classified as narrative, persuasive/
argumentative, and informative (Olinghouse & Wilson, 2013; National
Assessment Governor’s Board, 2010). However, for this meta-analysis, we
136 Z. XU ET AL.

adopt a classification that includes descriptive, summary, and academic

writing, along with narrative and argumentative. These are genres that
authors of the included papers describe (e.g. Alharbi, 2015; Wichadee,
2013; Kashani, Mahmud, & Kalajahi, 2013). For those studies that have
not clearly stated what genre they have used, we have used the category
“mixed” (e.g., AbuSeileek, 2004).

Collaborative learning theory

The manner in which people learn is impacted by their surroundings
and there is a “dynamic interdependence of social and cultural forces”
(John-Steiner & Mahn, 1996, p. 192). Vygotsky believed that individual
development has its origins in social sources, and that social and individ-
ual action is mediated by tools and signs. Additionally, these two themes
have been further examined by developmental analysis (John-Steiner &
Mahn, 1996). Among all the constructs used by Vygotsky, mediation and
internalization have been found to be significant contributors to language
learning (Lantolf, 2006). According to Lantolf, Thorne, and Poehner
(2015), when children learn a new language, their learning is influenced
by the social perception and concepts of the words in the language.
Vygotsky also believed that individuals have the ability to use social sym-
bols as tools for learning (Lantolf et al., 2015). Lantolf and Thorne
(2006) found that mediation and regulation are two important constructs
of the sociocultural theories that support second language learning prac-
tices. They describe mediation as higher level cultural tools that help a
child mediate between the individual and socio-material world (p. 199).
Regulation is a subcategory of mediation, which helps reshape biological
perception into cultural perceptions (p. 199). They go on to further clas-
sify the final stage of language development as self-regulation. In this
stage the learner needs minimum external support, and can be said to
have internalized the learning (p. 200). Lantolf and Thorne (2006) also
states that even though a second language learner might have reached
the final stage of internalization, it is not unusual for them to revisit the
earlier phases of learning during their language learning journey. We
believe this process applies to adult language learners, especially in the
learning of writing skills (Kashani et al., 2013).
According to Wertsch (1998), collaborative work is a significant con-
tributor to mediation and internalization. Students gain knowledge and
understanding of the new language through social interaction like class-
room discourse, instructional dialogue, and peer interaction and so on.
In this article, we will focus on how an individual’s language develop-
ment, namely writing skills are influenced by socio-cultural theories of
 COMPUTER ASSISTED LANGUAGE LEARNING 137

Vygotsky, Werstch, and Lantolf, and second language learning as seen in

the context of writing in adult ELLs.
We have categorized technology into two types: collaborative and non-
collaborative. If the technology applications generate interaction or col-
laboration between teachers and students or between students/peers, we
identified them as collaborative technology. For instance, Zou et al.
(2016) defined their technology use as “collaborative learning on Wiki-
spaces” (p. 1005). Some other studies also addressed collaborative learn-
ing in detail, such as Wichadee (2013), and Ferriman (2013). However,
other studies did not specify collaboration. In these cases, we have eval-
uated their technology use to classify it as collaborative or non-collabora-
tive. For example, Alharbi (2015) did not classify the technology he
examined in his paper as collaborative. However, through the use of dis-
cussion boards, wikis, and blogs, there is clear evidence of interactions
between learners.
When the technology use did not produce any interaction or collabor-
ation between learners and teachers, or between learners/peers, we classi-
fied them as non-collaborative. For instance, in AbuSeileek (2013)’s
study, the computer tracks change, identifies and reformulates the error,
and provides corrective feedback to ELLs, but there is no mention of
teacher or peer feedback. Thus, we coded this study as non-collaborative
technology. There are also some technologies which could be categorized
as collaborative in general but were not used in a collaborative manner
in the included studies. For example, even though emails can bring about
interaction in general, in the included studies (Nazari & Niknejad, 2015),
emails were only used to provide background information. So, there was
no observable collaboration. Another example is podcast use. Podcasts
were only used to distribute podcast lectures to ensure flexibility of time
(Bamanger & Alhassan, 2015). No collaboration was observed in the
study so we classified it as non-collaborative. Through the categorization
of collaborative and non-collaborative technology, we aim to find out if
collaborative technology use has a higher impact on adult ELLs writing
quality (See Table 1).

Previous reviews
Meta-analyses of instructional practices on language learning have shown
some interesting results. In a meta-analysis on literacy practices and
reading and writing instruction, Graham et al. (2017) found medium
effect sizes (ES ¼ .47) on writing quality among other outcome variables.
In another meta-analysis, Graham, McKeown, Kiuhara, and Harris
(2012) examined studies that analyzed six different writing interventions.
138 Z. XU ET AL.

Table 1. Collaborative and non-collaborative type of technology.

Characteristics Affordance
Collaborative type of technology
Publications Technology type
Alharbi (2015) Discussion board, Computer mediated com- Discussion board - space
blog, wiki munication environments. for students to publish
Discussion board - online their opinions and respond
forum. Blog -easily main- to others. Blog - insert
tain content and add graphics, multimedia,
dated entries. Wiki - audio, text. Wiki - feeling
create and edit web of responsibility, owner-
pages (p. 112). ship, and authority
(p. 112).
Al-Jarf (2013) Web-based learning - Web-based instruction Attractive, easy to use,
Blackboard based course (p. 52). contains all of the neces-
sary course components
(p. 52).
Arslan and Şahin- Blog Authentic communication Improve content organiza-
Kızıl (2010) that is peer reviewed tion, access larger body of
(p. 185). materials on writing,
improve language use and
vocabulary (p. 186).
Ciftci and Kocoglu Blog Online peer feedback Foster a sense of commu-
(2012) nity in the classroom
(p. 65).
Ferriman (2013) Nicenet- bulletin board Simplicity of use, visual Foster a sense of commu-
clarity and ease of access nity in the classroom (p.
(p. 245). 65), peer editing (p. 64).
Kashani et al. (2013) Blog Review writing of their obtain feedback and
peers, and review teacher rewrite based on the feed-
feedback (p. 207). back (p. 210).
Wichadee (2013) Wiki based environment Web-based facilitation Students engage in collab-
instruction in the form of orative writing, generate
a collective web-site online materials (p. 108).
where a large number of
participants work together
(p. 108).
Yang (2016) Computer supported Student interface and
collaborative teacher interface (p. 686). Student interface - evalu-
learning systems ate improvements in
students’ summary writing,
provide peer feedback.
Teacher interface - monitor
student interactions with
peers (pp. 686–688).
Zaini and Microsoft Word Office, Computer assisted lan- Automatic feedback, pro-
Mazdayasna (2015) WhatsApp, chat guage learning. vide comments on HW
rooms, emails assignments (p. 519).
Zou et al. (2016) Wiki based environment Communicate synchron- Post messages, track
ously and asynchronously changes, induction guide,
(p. 1003) function menu (p. 1005)
Non-collaborative type of technology
Publications Technology type
AbuSeileek (2013) Computer mediated writ- Track changes, word pro- Track change-identify and
ing corrective feed- cessor, combination of reformulate the error.
back techniques both (p. 323) Word processor-corrective
feedback (p. 320)
Ali (2016) Screen cast video Audio and video feedback Students see their work in
feedback (p. 108) multimodal formats,
receive computerized feed-
back similarly (p. 109)
Bamanger and Podcast Audio or video files posted Enriching vocabulary,
Alhassan (2015) as a series (p. 65). reviewing lessons, samples
of real language and
authentic materials (p. 67).
(continued)
 COMPUTER ASSISTED LANGUAGE LEARNING 139

Table 1. Continued.
Characteristics Affordance
Nazari and Computer mediated cor- Asynchronous corrective Receive directions via
Niknejad (2015) rective feedback- feedback (p. 197). email (p. 199). Provide
email writing background information
(p. 196).
Shintani and Computer mediated cor- Synchronous - feedback Synchronous and asyn-
Aubrey (2016) rective feedback-synchron- while writing. chronous correct-
ous and asynchronous Asynchronous - feedback ive feedback.
(Google Docs and after writing is completed
Microsoft SkyDrive) (p. 296).
Yang (2015) Concept map - automatic Automatic scaffolding, Building concept map for
3-layer concept map automatics measurement, central idea, main idea,
immediate feedback supporting idea (p. 273).
(p. 273).

They found several positive effects from these interventions, for example,
prewriting activities (ES ¼ .54), peer assistance when writing (ES ¼ .89),
product goals (ES ¼ .76), and assessing writing (ES ¼ .42) etc. These
meta-analyses provide a very succinct picture of reading and writing
instructions and their effectiveness. However, they do not examine the
use of education technology applications on reading and writing, espe-
cially on ELLs literacy development.
Numerous reviews and meta-analyses have been conducted in the field
of technology and learning. In a second order meta-analysis, Tamim
et al. (2011) found that use of educational technology has a moderate
effect (ES ¼ .35) on student learning outcomes as compared to face-to-
face teaching methods. Similarly, another meta-analysis that examines
the use of technology in education has found that as long as technology
tools are used as cognitive aids, student learning is enhanced (Schmid
et al., 2014). A tertiary meta-analysis of educational technology in the
classroom by Archer et al. (2014) found that implementation factors, like
administrative support and operational use of technology are also
important in the effectiveness of educational technology. Together, these
meta-analyses present a very good review of the effects of educational
technology on student learning in general. They do not, however, focus
on the importance of educational technology and language learning
for ELLs.
Meta-analyses and reviews that examine the importance of learning
language with educational technology applications are also prevalent
among academic journals. For example, according to a review by
Cheung and Slavin (2013), educational technology applications produced
a small effect on reading scores of K-6 struggling learners (ES ¼ .14).
Additionally, it was also found that Computer Mediated Glosses had a
medium effect on second language reading (ES =.73) comprehension and
140 Z. XU ET AL.

a large effect on incidental vocabulary learning (ES =1.3; Abraham,

2008). In a research synthesis on technology supported peer feedback in
ELL classes, Chen (2016) presented several different findings. Of his
findings, the one that is significant for our analysis is that technology
mediated language learning indirectly helps written communication. In a
review of past and current educational technology practices in language
learning, Zhao (2003) found that technology supported language learning
is as effective as teacher led language learning. Golonka et al. (2014)
compared traditional methods of teaching second language and newer
methods of teaching foreign languages. They found that computer sup-
ported pronunciation was most helpful for learners of foreign languages.
However, none of these syntheses focused on the educational technology
applications’ effect on ELLs writing competence.

Research questions
This meta-analysis addresses the relative effectiveness of educational
technology applications on adult ELL’s writing quality and tries to
answer the following research questions:

1. Do educational technology applications improve adult ELL’s writing

quality compared to non-technology control groups?
2. How do the substantive and methodological characteristics of the
studies, such as the types of technology applications, program inten-
sity and duration, genre of writing, research design, and outcome
measures, affect the estimated effects?
3. Specifically, how types of technology applications (collaborative and
non-collaborative) and genre of writing moderate the effect of tech-
nology on the writing quality of adult ELLs?

Method
Study search
In order to find the publications for this meta-analysis, we employed a
two-step process. First, we located hundreds of potential reports through
the electronic databases of our library. Then, we manually examined the
references of each included publication to ensure that we had not
excluded any relevant studies.
To find the studies, we carried out computer searches of databases
from six sources: (a) the Educational Resources Information
Clearinghouse (ERIC), (b) Academic Search Complete, (c) Computer
Source, (d) PsycINFO, (e) Linguistics and Language Behavior Abstracts,
 COMPUTER ASSISTED LANGUAGE LEARNING 141

and (f) Communication Source. Our search concept includes educational

technology applications (as keyword concept and by various types of
educational technology applications), ELLs (including ESL, EFL, second
language learner, etc.), higher education, and literacy and writing. We
searched relevant thesaurus terms for each concept, along with keywords
in the titles and abstracts to find all of the applicable citations. Our
search protocol in ERIC is confined to 2000–2017, and peer reviewed
articles. Then we copied the search process for the other databases.
Finally, we repeated the procedure of manual examination of the referen-
ces and bibliography lists of the selected studies to ensure that our search
included all of the relevant publications.

Criteria for inclusion

Included publications met the following eight criteria.

1. Publications had to be empirical studies of the effects of educational

technology applications on adult ELLs’ writing quality. Any educa-
tional technology applications developed for native speakers are
excluded. We excluded any secondary data analyses and litera-
ture reviews.
2. Publications had to be published or reported during the period from
Jan. 1, 2000, to Mar. 31, 2017, and had to be available in English. We
wanted to focus on the most recent educational technology applica-
tions and in the past 17 years, technology applications on ELLs writ-
ing have shown swift advancement.
3. Publications had to concentrate on adult ELLs, including undergradu-
ate, graduate, and adult programs conducted in the university.
Publications focusing exclusively on K-12 ELL students were excluded.
4. The research studies had to measure the effectiveness of educational
technology applications on writing quality on at least one learning
outcome. Common measurements included standardized test scores,
school grades, or researcher-designed tests.
5. The research studies had to have used an independent comparison
group. Comparison conditions did not use technology as their
intervention.
6. We included only those studies that have used randomized experi-
mental or quasi-experimental designs.
7. In order for us to calculate the effect sizes, included publications had
to provide the necessary quantitative data.
142 Z. XU ET AL.

8. The studies that we included in this meta-analysis had to be pub-

lished in a peer-reviewed journal in order to ensure quality. We
excluded any conference papers or dissertations.

Study coding
In order to set up a systematic study coding and generate the process
of information retrieval, we followed a comprehensive coding proced-
ure. The coding procedure provided the major characteristics and
methodology for each study: (a) the basic features of the study (e.g.,
which year the study was published, coded “publication”), (b) research
design features (e.g., sample sizes, coded “N”; whether the study used a
true experimental or quasi-experimental design, coded “research
design”), (c) the contexts of intervention (e.g., the intensity of the pro-
gram, coded “intensity”, the duration of technology intervention,
coded “duration”, types of technology applications, coded “type of edu-
cational technology”, and genre of writing, coded “genre”), and (d) the
study’s learning outcome measurement (e.g., what and how outcomes
were measured—standardized test or researcher-designed test, coded
“measures outcome”).
Three researchers independently coded the major features of every
study. The three coders checked the agreement of the coding. Percentage
of total agreement was approximately 98%. If there was a disagreement
about a code, the three researchers re-examined the codes and selected
the appropriate change. If an agreement could not be reached, the third
author assisted the three coders to resolve the issue.

Effect size calculation and statistical analysis

We used Hedge’s g to estimate the effect sizes of educational technology
applications on ELLs writing quality for each single study (Lipsey &
Wilson, 2001). When the raw data did not include means and standard
deviations for calculating effect sizes, we used other statistical values,
such as p-value, F-test, or confidence level, to compute an effect size. For
example, in Zou et al. (2016), we used the post-test mean and t-statistics
to obtain the effect size. In Ferriman (2013), we calculated the t-statistics
by F-test first, and then we calculated the effect size by the post-test
mean (estimated from graph) and t-statistics.

Results
Our search located 16 publications that met our inclusion criteria (see
Figure 1). The studies were published between Jan. 2000 and Mar. 2017.
 COMPUTER ASSISTED LANGUAGE LEARNING 143

Figure 1. Search flowchart.

Some of the 16 publications included more than one single study. In

total, there were 21 independent studies in 16 articles. Among 21 studies,
11 were categorized as collaborative type of technology; the remaining 10
were non-collaborative type of technology. Among these studies, only
one publication was published in the 2000s, and 15 publications were
published in the 2010s. Seventeen studies were quasi-experiments,
whereas the other four used randomized experimental designs. The
intensity of the programs varied from 20 to 240 minutes each week. The
duration of the programs extended from four weeks to one entire school
semester. The sample size in all of the studies was small, ranging from
24 to 113. The selected studies and their basic features are compiled in
Table 2.

Overall effects
The overall effect size was estimated using both a fixed effect model and
a random effect model. The fixed effect size estimate was 0.93 (95% CI
[0.80, 1.06]) and the random effects estimate was 1.28 (95% CI [0.85,
1.70]). The effect sizes were statistically heterogeneous (p < 0.001) with
an I2 of 89.7%. The fixed effect and random effects forest plots are pre-
sented at Figure 2. The substantial heterogeneity, when incorporated into
the random effects model, resulted in a larger effect size as a result of
redistribution of a wider range of weights into the fixed effect model
Table 2. Basic features and effect size information for all studies.
144

Study ID Publication Type of educational technology Genre Duration (weeks) Intensity (mins/week) N Measures outcome Research design ES
1 Wichadee (2013) Collaborative Summary writing 14 140 80 Researcher-designed test Quasi-experimental 0.19
2 Ferriman (2013) Collaborative Academic writing 11 120 30 Researcher-designed test Quasi-experimental 0.74
3 Ferriman (2013) Collaborative Academic writing 11 120 30 Researcher-designed test Quasi-experimental 1.31
4 Ferriman (2013) Collaborative Academic writing 11 120 30 Researcher-designed test Quasi-experimental 0.98
5 AbuSeileek (2013) Non-collaborative Mixed 8 25 32 Standardized test Quasi-experimental 2.97
Z. XU ET AL.

6 AbuSeileek (2013) Non-collaborative Mixed 8 25 32 Standardized test Quasi-experimental 1.36

7 AbuSeileek (2013) Non-collaborative Mixed 8 25 32 Standardized test Quasi-experimental 2.37
8 Alharbi (2015) Collaborative Argumentative 16 135 63 Researcher-designed test Quasi-experimental 3.23
9 Bamanger and Non-collaborative Mixed 6 60 55 Researcher researcher-designed True-experimental 0.77
Alhassan (2015) test
10 Yang (2016) Collaborative Academic writing 18 180 24 Researcher-designed test Quasi-experimental 3.14
11 Zou et al. (2016) Collaborative Mixed NO NO 66 Researcher-designed test Quasi-experimental 0
12 Kashani et al. (2013) Collaborative Argumentative 8 60 64 Standardized test True-experimental 0.33
13 Nazari, and Non-collaborative Mixed 4 60 50 staNdardized test True-experimental 2.3
Niknejad (2015)
14 Arslan, and Şahin-Kızıl Collaborative Mixed NO NO 50 Researcher-designed test Quasi-experimental 1.17
(2010)
15 Shintani and Aubrey Non-collaborative Narrative NO NO 43 Researcher-designed test Quasi-experimental 0.84
(2016)
16 Shintani and Aubrey Non-collaborative Narrative NO NO 47 Researcher-designed test Quasi-experimental 1.47
(2016)
17 Al-Jarf (2004) Collaborative Mixed 12 240 111 Researcher Quasi-experimental 0.33
researchresearcher-designed test
18 Ali (2016) Non-collaborative Descriptive 10 120 63 Researcher-designed test Quasi-experimental 2.09
19 Zaini, and Mazdayasna Collaborative Argumentative 20 NO 44 Researcher-designed test True-experimental 2.8
(2015)
20 Ciftci, and Kocoglu. Collaborative mixed 20 NO 30 reseArcher-designed test Quasi-experimental 0.93
(2012)
21 Yang (2015) Non-collaborative Summary writing 8 120 107 Standardized test Quasi-experimental 0.59
Note: When ES <0.2, it is slight; when 0.2 < ES <0.5, it is small; when 0.5 < ES <0.8, it is medium; when ES >0.8, it is large (Cohen, 1988).
 COMPUTER ASSISTED LANGUAGE LEARNING 145

Fixed Effect Model Forest Plot

Study %
ID ES (95% CI) Weight

1 0.19 (-0.24, 0.62) 9.30

2 0.74 (-0.00, 1.48) 3.12
3 1.31 (0.53, 2.09) 2.81
4 -0.98 (-1.74, -0.22) 2.96
5 2.97 (1.97, 3.97) 1.73
6 1.36 (0.60, 2.12) 2.96
7 2.37 (1.47, 3.27) 2.13
8 3.23 (2.47, 3.99) 2.96
9 0.77 (0.22, 1.32) 5.74
10 3.14 (1.94, 4.34) 1.21
11 0.00 (-0.49, 0.49) 7.20
12 0.33 (-0.16, 0.82) 7.20
13 2.30 (1.59, 3.01) 3.47
14 1.17 (0.56, 1.78) 4.69
15 0.84 (0.21, 1.47) 4.40
16 1.47 (0.82, 2.12) 4.13
17 0.33 (-0.04, 0.70) 12.47
18 2.09 (1.48, 2.70) 4.69
19 2.80 (1.96, 3.64) 2.44
20 0.93 (0.19, 1.67) 3.12
21 0.59 (0.20, 0.98) 11.26
Overall (I-squared = 89.7%, p = 0.000) 0.93 (0.80, 1.06) 100.00

-4.34 0 4.34

Random Effects Model Forest Plot

Study %
ID ES (95% CI) Weight

1 0.19 (-0.24, 0.62) 5.18

2 0.74 (-0.00, 1.48) 4.68
3 1.31 (0.53, 2.09) 4.61
4 -0.98 (-1.74, -0.22) 4.64
5 2.97 (1.97, 3.97) 4.19
6 1.36 (0.60, 2.12) 4.64
7 2.37 (1.47, 3.27) 4.38
8 3.23 (2.47, 3.99) 4.64
9 0.77 (0.22, 1.32) 5.02
10 3.14 (1.94, 4.34) 3.80
11 0.00 (-0.49, 0.49) 5.10
12 0.33 (-0.16, 0.82) 5.10
13 2.30 (1.59, 3.01) 4.75
14 1.17 (0.56, 1.78) 4.92
15 0.84 (0.21, 1.47) 4.89
16 1.47 (0.82, 2.12) 4.85
17 0.33 (-0.04, 0.70) 5.26
18 2.09 (1.48, 2.70) 4.92
19 2.80 (1.96, 3.64) 4.50
20 0.93 (0.19, 1.67) 4.68
21 0.59 (0.20, 0.98) 5.23
Overall (I-squared = 89.7%, p = 0.000) 1.28 (0.85, 1.70) 100.00

NOTE: Weights are from random effects analysis

-4.34 0 4.34

Figure 2. Forest plot.

146 Z. XU ET AL.

(weights ranging from 1.21 to 12.47%) to a narrower range of weights in

the random effects model (3.80–5.26%). The effect sizes with the largest
weights in the fixed effect model tended to have effect sizes closer to
zero with five studies having a combined weight of 47.43%; the same five
studies combined weight in the random effects model reduced to only
25.87%, as shown in the table below.
Weight Study ID Effect Size
12.47% 17 0.33 (0.04 to 0.70)
11.26% 21 0.59 (0.20 to 0.98)
9.30% 1 0.19 (0.24 to 0.62)
7.20% 11 0.00 (0.49 to 0.49)
7.20% 12 0.33 (0.16 to 0.82)

Among study effect sizes with substantial heterogeneity, one implica-

tion of a random effects model is a weighted average effect size that
moves closer to a simple average (narrower range of weights).

Examining for publication bias

A funnel plot was created to check the presence of publication bias. In
the funnel plot, standard error is plotted on the y-axis, and effect size is
plotted on the x-axis. We worked with the assumption that if publication
bias exists, the funnel will be asymmetric. However, the funnel plot for
these studies is difficult to interpret with respect to publication bias,
given that half of the studies (12 of 21) lay outside the funnel, reflecting
the significant heterogeneity and potential outliers (see Figure 3).

Funnel plot with pseudo 95% confidence limits

0

17
.2

21
1
11 12
9
SE

15 14 18
16
13
4 2 20 6 8
.4

3
19
7

5
.6

10

-1 0 1 2 3
ES

Figure 3. Funnel plot.

 COMPUTER ASSISTED LANGUAGE LEARNING 147

Sub-group analysis
Next, we tried to explain the heterogeneity by sub-group analysis. Six sub-
stantive and methodological features of the studies were used to model var-
iations in outcomes: type of technology, genre of writing, program duration,
program intensity, measures outcome, and research design (see Table 3). A
fixed effect model was used for the sub-group analysis in order to obtain an
assessment of between group variations using the Stata command “metan.”

Type of technology
We first conducted analysis about the effect on different types of tech-
nology: collaborative type of technology (N ¼ 11), and non-collaborative
type of technology (N ¼ 10). The between-group variation was statistic-
ally significant (p < 0.001). Non-collaborative type of technology gener-
ated a larger effect size than collaborative technology: 1.435 (95% CI
[1.236, 1.635]) compared to 0.537 (95% CI [0.362, 0.712]).

Genre of writing
Examination of effect size by genre of writing revealed a statistically sig-
nificant (p < 0.001) between-group heterogeneity. In the absence of
Table 3. Subgroup analysis.
Between Group
ES [95% Statistical Subgroup
Fixed Effect Model Sub-Groups (n¼) Conf. Interval] Significance Homogeneity
Type of Technology Collaborative (n ¼ 11) .537 [.362, .712] Y P < 0.001 N
Non-collabora- 1.435 [1.236, 1.635] N
tive (n ¼ 10)
Genre of Writing Academic Writing .682 [ .268, 1.096] Y P < 0.001 N
(n ¼ 4 )
Argumentative 1.489 [1.118, 1.859] N
(n ¼ 3)
Descriptive (n ¼ 1) 2.090 [1.482, 2.698] N
Mixed (n ¼ 9) .899 [.700, 1.098] N
Narrative (n ¼ 2) 1.145 [.695, 1.596] Y P ¼ 0.171
Summary .409 [.119, .699] Y P ¼ 0.179
Writing (n ¼ 2)
Research Design True 1.146 [.843, 1.449] N P ¼ 0.117 N
Experimental (n ¼ 4)
Quasi .87 [.731, 1.023] N
Experimental (n ¼ 17)
Intensity Strong .764 [.579, .948] Y p < 0.05 N
Intensity (n ¼ 9)
Weak 1.248 [.975, 1.521] N
Intensity (n ¼ 6)
Missing (n ¼ 6) .961 [.703, 1.219] N
Duration Long .919 [.685, 1.154] N p ¼ 0.333 N
Duration (n ¼ 6)
Short 1.009 [.820, 1.199] N
Duration (n ¼ 11)
Missing (n ¼ 4) .747 [.456, 1.038] N
Measures Outcome Standardized (n ¼ 6) 1.086 [.84, 1.331] N p ¼ 0.134 N
Researcher- .864 [.708, 1.019] N
designed (n ¼ 15)
148 Z. XU ET AL.

formal post-comparison tests in Stata for meta-analysis, we examined the

results visually using overlap of sub-group effect size at 95% confidence
intervals. Likely candidates for this between-group heterogeneity included
academic writing (0.68; 95% CI [.27, 1.10]) and summary writing (0.41;
95% CI [.12, .70]) vs descriptive writing (2.09; 95% CI [1.48, 2.698]), argu-
mentative writing (1.49; 95% CI [1.118, 1.859]), and narrative writing (1.15;
95% CI [0.695, 1.596]). Each of these subgroups had few studies ranging
from 2 to 4. There was only one study that was included under the descrip-
tive subgroup further limiting its results for between-group comparison
assessments. Figure 4 depicts the between-group differences in our study.

Research design
Different research designs might lead to variations in effect sizes (e.g.
Abrami & Bernard, 2006). In this collection of studies used for meta-
analysis, we identified two main categories of research designs: true
experimental (N ¼ 4) and quasi-experimental studies (N ¼ 17). True
experiments were those in which participants were randomly assigned to
either treatment or control groups and the unit of analysis was at the
level of the random assignment. Quasi-experimental studies referred to
those in which even though participants were not randomly assigned to
treatment and control groups, they were matched on key learning out-
come measurements at pretest. The average effect sizes for true experi-
mental studies and quasi-experimental studies were 1.146 (95% CI
[0.843, 1.449]) and 0.87 (95% CI [0.731, 1.304]), respectively. The
between group variation was not statistically significant (p ¼ 0.117).

Program intensity
We followed Cheung and Slavin (2013) criteria for intensity, with a cut-
off of 75 minutes per week. When the technology program was used
3

2.5

1.5

0.5

Figure 4. Between-group difference on genre of writing.

 COMPUTER ASSISTED LANGUAGE LEARNING 149

more than 75 minutes per week, we categorized as strong (n ¼ 9).

Otherwise, it was categorized as weak (n ¼ 6). There are six studies,
which did not provide information about the program intensity (indi-
cated by missing). The effect sizes for strong- and weak-intensity pro-
gram were 0.764 (95% CI [0.579, 0.948) and 1.248 (95% CI [0.975,
1.521]), respectively; the effect size for the six studies that did not pro-
vide intensity information was 0.961 (95% CI [0.703, 1.219]). The
between group variation was statistically significant (p < 0.001) but as
before, which two groups or if all three groups are statistically different
could not be determined. When the studies with missing intensity data
were dropped, the between group difference between “strong intensity”
and “weak intensity” was statistically significant (p ¼ 0.004).

Program duration
We used 3 months because 3 months represent at least half of the school
year with a typical school year of 180 days. When the program lasted
longer than 3 months, we categorized as long duration (N ¼ 6).
Otherwise, we categorized as short duration (N ¼ 11). Four studies
included in the current meta-analysis did not provide relevant informa-
tion on duration (indicated by missing, N ¼ 4). The effect size for long
and short duration program was 0.919 (95% CI [0.685, 1.154]) and 1.009
(95% CI [0.820, 1.199]), respectively; the effect size for the four studies
that did not provide duration information was 0.747 (95% CI [0.456,
1.038]). The between group variation was not statistically signifi-
cant (p ¼ 0.333).

Measures outcome
Six effect sizes were measured using a standardized test with an overall
effect size of 1.086 (95% CI [0.84, 1.33]). The remaining fifteen effect
sizes were measured with a researcher-designed test with an overall effect
size of 0.864 (95% CI [0.708, 1.019]). The between group variation was
not statistically significant (p ¼ 0.134).

Sensitivity analysis: duration and intensity

We chose as part of our analysis strategy to dichotomize “duration” and
“intensity” (described previously) even though these factors were
reported by the original authors as continuous variables. As a sensitivity
analysis, we used meta-regression to analyze the relation between effect
size and duration (as # of weeks) and intensity (as number of minutes
per week) as continuous variables. We also created a new variable “total
150 Z. XU ET AL.

minutes” which was calculated as by multiplying duration (# of weeks)

by intensity (# of minutes per session). In each of this meta-regression,
the variables (duration, intensity, and total minutes) were not statistically
significantly associated with effect size. These findings are consistent
with the earlier stratified analyses except for “intensity” where a statistic-
ally significant between group variations was observed between “strong
intensity” and “weak intensity.”

Discussion
Overall effectiveness of educational technology applications on adult
ELLs’ writing quality
The purpose of this meta-analysis is to investigate the effects of technol-
ogy applications on adult ELLs writing quality. We identified 16 peer-
reviewed publications including 21 independent studies meeting our cri-
teria. Our results indicate that technology applications have a large effect
(.93 and 1.28, fixed effect model and random effect model respectively)
on adult ELLs’ writing quality. This suggests that the use of technology
is more effective than traditional teaching methods without technology
for ELLs’ writing quality. The mean effect size of the present study is
much bigger than the effect sizes reported in the meta-analysis con-
ducted by Graham et al. (2017), who examined the effects of traditional
writing instruction on native students’ writing quality (ES ¼ .47). The
effect size of technology applications on ELLs’ writing quality (ES ¼ .93
and 1.28) is comparable to some other studies which examined the
effects of technology on ELLs’ literacy development (ES= .73 and 1.3,
technology on second language reading comprehension and vocabulary
learning; Abraham, 2008).
Technology applications can improve ELLs writing from the aspect of
language use and content. In this meta-analysis, we categorized the tech-
nology into collaborative and non-collaborative technology. Among the
collaborative technology applications, there were several that were
studied in the research papers. For example, wiki-based environments
are often seen as being very useful in collaborative language learning.
These environments can be used to collectively create and edit web-pages
(Alharbi, 2015), which is beneficial to the content and organization of
writing. These wiki-based environments can also improve writing
through collaborative engagement and increase student involvement
(Wichadee, 2013), and provide online corrections and feedback for stu-
dents by peers (Zou et al., 2016). Online corrections and feedback can
facilitate students’ writing in multiple aspects, such as vocabulary, lan-
guage use and mechanics.
 COMPUTER ASSISTED LANGUAGE LEARNING 151

Another form of writing in the collaborative environment is via blog-

ging. Blogging provides an audience, peer review, and authentic commu-
nication opportunities (Arslan & Şahin-Kızıl, 2010). It also provides easy
publication, rapid writing, and immediate interaction and enhances
exchange between student and instructor (Kashani et al., 2013).
Additionally, blogging allows users to maintain and add content
(Alharbi, 2015). Blogs also allow students to write for larger audiences,
encourage ownership of content, and help the teacher observe and inter-
vene when needed (Ciftci & Kocoglu, 2012).
Other technologies like Nicenet-bulletin board use computer mediated
communication (CMC) for storing and giving information to students
(Ferriman, 2013), therefore helping students generate better content for
writing. Web-based Blackboard courses facilitate the development of
searching, reading, and writing skills and develop vocabulary and gram-
mar knowledge (Al-Jarf, 2004). Zaini and Mazdayasna (2015) study
found that the students who were exposed to computer-based instruction
performed better than their counterparts in terms of using appropriate
articles and tense. Moreover, they produced paragraphs of higher quality.
Among the non-collaborative software, computer mediated corrective
feedback is useful in drawing a student’s attention to errors and provides
immediate feedback while the student is in the process of writing
(Shintani & Aubrey, 2016), which could benefit students on the language
use and mechanics. Another form of computer mediated feedback—
email helps in providing background information, expanding topics
beyond classroom-based themes, extending language learning time and
place, and promoting student-centered language learning. Along with
these technologies, podcasts are also used as authentic input for teaching
students outside of a classroom and enabling their process of self-learn-
ing by listening to podcasts online or downloading them for their con-
venience (Bamanger & Alhassan, 2015). Yang (2015) uses concept maps
to help students organize central idea, main idea, and supporting ideas
in automatic three-layer concept mapping software.

Findings from testing for predictors

We performed six sub-group analyses in this meta-analysis, among
which there were three statistical significant between-group differences:
genre of writing, type of technology, and program intensity. We did not
find any statistical significance on research design, program duration,
and measures outcome.
Genre of Writing Matters. We classified our included studies into six
genres of essay writing namely, academic, argumentative, descriptive,
152 Z. XU ET AL.

narrative, summary, and mixed writing. The highest number of studies

(N ¼ 9) was in the mixed writing category, either because the study was
examined through different types of genres, or because it was hard to
classify based on the given information. When we did the subgroup ana-
lysis for genre of writing, we found that at least two sub-group effect
sizes were different. In other words, genre of writing moderated the
effect of technology on ELLs writing quality.
Even though we need to be cautious when interpreting the statistical
results and comparing the difference between any two groups because
of the small numbers of studies in most of the sub-groups and the
absence of follow-up comparison tests, our findings are consistent with
the previous study (Bigelow & Schwarz, 2010). We found that the effect
size of narrative writing (1.145) and descriptive writing (2.090) is obvi-
ously higher than academic writing (.68) and summary writing (.41).
Some other studies also confirmed that. For instance, academic writing
is generally recognized to be more difficult for many second language
learners (Mohan & Lo, 1985). Hill (1991) also found that writing sum-
mary can promote learning and thinking but it is difficult. Graham
(2006) found that genres play an important role in a student’s writing
ability. In another study, Gillespie et al. (2013) analyzed students’
knowledge about the writing process and whether it would predict their
genre knowledge of story, persuasive, and informational writing. They
found that students performed differently on different genres of writ-
ing. In his review of research on teaching practices for writing, Hillocks
(2005) found that when teaching writing there is generally greater focus
on form than on content or genre. He found that writing teachers felt
general principles of writing to be sufficient for a student to learn to
write effectively. However, focus on content or genre is important for
improving students’ writing abilities. In a similar vein, Tremmel (2011)
found that generalized writing instruction without emphasis on genre
may be harmful to the overall learning of the student. Qin and Uccelli
(2016) conducted a study in which they examined cross-genre writing
abilities of L2 students across the genres—argumentative and narrative.
They found that argumentative essays produced more complex writing
than narrative essays. Yeh (2015) has also found evidence to suggest
that including genres in writing may be important for students writing
achievement.
The evidence from our meta-analysis supports the claim that teaching
genres in a writing classroom may be of vital importance to improve
students’ writing quality since knowledge of one genre cannot be auto-
matically transferred to another genre (e.g. Gillespie et al., 2013) and one
genre might be more difficult than another (e.g. Mohan & Lo, 1985).
 COMPUTER ASSISTED LANGUAGE LEARNING 153

Is Type of Technology Important in Teaching Adult ELLs Writing?

We had hypothesized that collaborative technology would produce larger
effect sizes than non-collaborative. Our meta-analysis generated a contra-
dictory result. In our meta-analysis, non-collaborative technology gener-
ated a larger effect size (1.435) than collaborative technology (.537).
There are several reasons for that. First, the current meta-analysis com-
pared technology with non-technology interventions, but the interven-
tions and contexts may differ. For instance, some studies compared
collaborative type of technology with face-to-face collaboration (e.g.
Wichadee, 2013), while other studies compared collaborative type of
technology with lecture-based instruction without collaboration (e.g. Zou
et al., 2016). In the future, studies could do more nuanced comparisons
between the technology interventions and multiple comparison groups
using more detailed classifications of the control conditions.
The second and more important underlying reason is that writing is
a learning task, which requires skills that exert high cognitive load on
the writers. Cognitive load theory might be able to explain the results
in this case. This theory suggests an existing relationship between infor-
mation received by a learner and the cognitive load of processing such
information (Paas, Renkl, & Sweller, 2003). It may be possible that the
cognitive load associated with collaboration has a negative effect on
writing (e.g. Chandler & Sweller, 1991). Subsequently, the student may
use large amounts of cognitive effort to understand collaboration, and
this may take away from the cognitive effort required to improve writ-
ing quality.
Another potential reason is that students are not trained to collaborate
and they do not know how to complete the collaborative writing tasks
(e.g. how to provide feedback on writing). Our results imply that collab-
oration has a positive impact on ELLs’ writing, but not as big as the
non-collaborative type of technology. Therefore, the results of the current
study also raise an interesting question: does collaboration really matter
in writing, especially on ELLs’ writing?
There is little agreement among researchers on the effect of collabor-
ation on writing for ELLs. Some studies have found that collaborative
learning can have a positive impact on the writing practices of adult
ELLs. For example, Arnold, Ducate, and Kost (2012) examined online
writing and revision behaviors of university students, and found that
most students contribute positively in collaborative writing environ-
ments. Similarly, Chu, Capio, van Aalst, and Cheng (2017) found that
ELL students who wrote collaboratively produced higher quality writing.
Collaborative writing also increases team work satisfaction when meas-
ured according to three collaborative factors namely, Team Dynamics,
154 Z. XU ET AL.

Team Acquaintance, and Instructor Support (Ku, Tseng, &

Akarasriworn, 2013).
However, not all researchers found a positive impact of collaboration
on writing practices. For instance, online collaborative work by university
students was evaluated in a phenomenological study by Limbu and
Markauskaite (2015). They found that students did not regard online col-
laborative exercises as a contributor to their knowledge of building com-
munity. In the same vein, a review of research on collaborative work by
Kreijns, Kirschner, and Jochems (2003) found mixed results on the col-
laborative writing, and that both research and field observations did not
always show positive collaboration among students. Additional evidence
is provided by Kwon, Liu, and Johnson (2014), who looked for patterns
in collaborative work and found that many groups of students in the
sample did not successfully regulate the group process.
Intensity of the Technology Programs. There was a significant differ-
ence between low-intensity and high-intensity educational technology
programs. Low-intensity programs (1.248) tend to have larger effect on
ELLs’ writing quality than high-intensity programs (.764). This result is
contradictory to Cheung and Slavin (2013) findings, which indicated that
high-intensity programs have a bigger impact on struggling readers. This
might point to the difference of effects on distinctive types of learners,
specifically ELLs. Another reason might be that intensified exposure to
educational technology requires heavier cognitive load, and might there-
fore significantly reduce learners’ positive feelings. However, due to the
limited number of studies included in this meta-analysis and the large
number of studies which lacked the relevant information, we need to be
cautious in interpreting the results.

Implications
Research Implications. Although we found that genre of writing has a
significant impact on adult ELLs’ writing quality, our analysis revealed
that previous research has not placed any importance to genres. In the
21 included studies, there were nine which were classified in the mixed
writing category. This should launch a new direction for future research,
in which researchers might try to focus on specific genres and instruc-
tion that supports ELLs writing in those areas.
Our results also indicated that non-collaborative technologies work
better than collaborative ones in improving ELLs’ writing quality.
Previous studies did not reach consistent agreement on which type of
technology use works more effectively. In the future, researchers may
work towards providing a clear picture of the important elements of
 COMPUTER ASSISTED LANGUAGE LEARNING 155

collaborative technologies and gather data about cognitive load associated

with the use of the specific technology application. Future researchers
may want examine which works better for learning—face-to-face or tech-
nology supported collaboration. The inconsistent findings for this factor
open up an important area of research for the field.
Pedagogical Implications. The current meta-analysis indicates that
genre of writing moderates ELLs’ writing quality greatly, which is con-
sistent with the previous studies (Bigelow & Schwarz, 2010). However,
from the included studies, we can see that instructors do not emphasize
any particular writing genre in classrooms. There were a large number of
studies coded as “mixed.” As with previous research, it is important for
ELLs writing instruction to focus on specific genres since the writing
should be tailored to the needs of the genre (Graham, Kiuhara,
McKeown, & Harris, 2012). Before or during the course of the writing
activity, teachers could allocate more time in making the students aware
of the target audience and how the different audiences and different
writing purposes should modify their writing style.
In the ELL writing classroom, the instructors could pay more attention
to writing genre. For example, as an introduction to the topic, the
teacher might conduct a large group discussion, in which he or she
engages the students in a class discussion about genres and talks about
for whom they will write and what they want to accomplish. Following
the explanation of the general goals of the genre, the teacher may then
begin to discuss the schematic/syntax structure, and the vocabulary
choices of the genre. In the exploratory phase of the lesson, the teacher
could also explain and discuss the language choices of the students and
the purposes of those language choices (Maune, 2016).
In our meta-analysis, we have found that non-collaborative technolo-
gies seem to have a bigger impact on writing than collaborative technol-
ogy applications. It is likely collaboration exerts a higher cognitive load
on a learner than non-collaborative tasks. This may lead to a reduction
of focus on the writing task, resulting in lower scores for collaborative
technology applications. Our findings indicate that even though we may
want to use collaborative tasks to engage students, provide feedback and
access to discussions, etc., the collaborative tasks might have detrimental
effects on writing quality. In ELLs writing classrooms, the instructors
should be careful to balance cognitive load for collaborative tasks and
the writing tasks. It may be reasonable to assume that writing instructors
might require additional training to be able to accomplish these tasks in
their classrooms. In addition, many ELLs may not have encountered col-
laborative writing via technology in their previous learning environ-
ments. Instructors need to take this factor into account before adopting
156 Z. XU ET AL.

collaborative technology. Instructors also need to guide their students to

use collaborative technology effectively to generate positive feelings.

Conclusion
As with any meta-analysis, the inclusion/exclusion criteria for the studies,
the coding procedures and the predictor analyses may have affected the
findings of the current study. First, our inclusion criteria might have left
out some relevant studies since we have only examined peer-reviewed
articles. As can be seen in the funnel plot, it is likely that there was some
publication bias since we had excluded studies that had not been pub-
lished. Therefore, studies with small effects or negative effect sizes may
have been missed by our search. Given the above limitations, out meta-
analysis may not be as comprehensive as possible.
Second, we feel that the conclusions offered must be treated provisionally,
as only 21 studies met our standards for inclusion. Among some of the
included studies, we found certain data missing, which we were unable to
account for. For example, when we tried to investigate how genre of writing
moderated the effect of technology on writing quality of ELLs, we had to
rely on the original studies to perform the analysis. The small number of
studies in each category of writing genres, and some missing data placed
limitations on our analysis and interpretation. These sources of missing data
or information might be something that future researchers could address.
Third, meta-analyses are often confined by the inclusion of articles.
For instance, even though the authors of the papers used this in analysis
identified the educational levels of their participants (e.g. graduate, post-
graduate, adult language learners), most of them have not identified the
level of writing competence within the populations (e.g. low, medium,
high). Therefore, that limits our analysis by language proficiency levels.
However, our study has confirmed the effect of technology applica-
tions on ELLs’ writing quality. As the technology use increases, and the
number of ELLs rises, this study addresses important elements of how
technology may be used to improve writing quality of adult ELLs. More
importantly, results from the sub-group analysis have shown us how
such effect sizes can be made larger. We have also proposed critical
directions for future research on the manner in which technology appli-
cations can be used to develop better writing quality for ELLs and have
provided some pedagogical implication for practitioners.

Acknowledgments
We thank the two anonymous reviewers and the editor for their thoughtful comments
on the manuscript.
 COMPUTER ASSISTED LANGUAGE LEARNING 157

Notes on Contributors
Zhihong Xu, a Ph.D. student in Texas A&M University, has years of teaching and
working experience. She has taught English in Chongqing University (Chongqing,
China) for two years and has worked in the Chinese version of Harvard Business
Review (Shanghai, China) for almost ten years. She has published a series of English
textbooks and numerous business articles. A book co-authored by Mrs. Xu—Ice
Breaking, the Marketing Miracle from 300 Million to 1.5 Billion—has been one of
the best-selling business books in 2006 in China. She is currently conducting
research on the effects of the integration of technology on instructional practices,
and how it could enhance reading comprehension and writing competence of
English Language Learners.
Manjari Banerjee earned her Master’s degree from the Department of Teaching,
Learning and Culture in ’09. She began her career as a High School mathematics
teacher. She has also taught at a non-traditional, Montessori-based school, first at the
primary level, and then at the elementary level. She holds a Merit Fellow with the
College of Education and Human Development. She is currently a PhD student in Texas
A&M in the Department of Teaching, Learning and Culture with a focus on Technology
and Teacher Education. Her current teaching experience consists of Math Methods for
Elementary teachers, and Middle grade teachers. Her research interests include educa-
tion technology applications, STEM teacher education, and math literacy.
Dr. Gilbert Ramirez has a public health career spanning over 40 years that includes
service at multiple institutions of higher education and a public health practice career in
the Armed Forces and civilian sectors. He recently served as the Founding Chair of the
Department of Public Health Studies at Texas A&M University and launched the highly
successful BSPH program. His research interests and expertise are in systematic reviews
and meta-analysis. He has co-directed a Cochrane Collaboration Center as well as serv-
ing in a leadership capacity with multiple Cochrane review groups and fields. He has
also served as a member of the CDC Community Preventive Services Taskforce, other-
wise known as the “Community Guide.”
Gang Zhu, PhD, is an Associate Professor of International Teacher Education at
Faculty of Education, East China Normal University, China. Zhu focuses on inter-
national teacher education, urban education, and comparative education with large
data samples. He published a broad array of peer-reviewed articles and book chap-
ters in both English and Chinese. His scholarly outlets include Journal of Education
for Teaching, Asia-Pacific Educational Researcher, and Frontiers of Education in
China, etc.
Dr. Kausalai (Kay) Wijekumar is Professor of Teaching, Learning and Culture at
Texas A&M University and Director for the endowed Center for Urban School
Partnerships (CUSP). She is a passionate advocate for excellent educational experiences
for all children with a focus on those growing in poverty, urban areas, and multi-cul-
tural settings. She has numerous grant funded research projects to improve reading
comprehension and writing with monolingual and bilingual learners using technology
supported solutions for schools. Her recent work has achieved great success and been
noted by US News and World Report as one evidence-based approach to reading com-
prehension. Dr. Wijekumar was elected to serve on the School Board of the Quaker
Valley School District in Western Pennsylvania for two terms. Most recently she has
developed large scale MOOCs to disseminate evidence-based practices to teachers and
school leaders worldwide.
158 Z. XU ET AL.

ORCID
Zhihong Xu http://orcid.org/0000-0002-4769-5597
Manjari Banerjee http://orcid.org/0000-0002-3263-0942
Gilbert Ramirez http://orcid.org/0000-0001-7559-3646
Gang Zhu http://orcid.org/0000-0001-6264-9356

References
References marked with an asterisk indicate studies included in the meta-analysis.
Abraham, L. B. (2008). Computer-mediated glosses in second language reading compre-
hension and vocabulary learning: A meta-analysis. Computer Assisted Language
Learning, 21(3), 199–226.
Abrami, P. C., & Bernard, R. M. (2006). Research on Distance Education: In defense
of field experiments. Distance Education, 27(1), 5–26.
Abrams, Z. (2016). Exploring collaboratively written L2 texts among first-year learners
of German in Google Docs. Computer Assisted Language Learning, 29(8),
1259–1270.
AbuSeileek, A. (2004). The effect of using a computer-based program on students’ writ-
ing ability in English. Unpublished Doctoral Dissertation, Arab University, Amman,
Jordan.
Al-Jarf, R. S. (2004). The effects of Web-Based learning on struggling EFL college writ-
ers. Foreign Language Annals, 37(1), 49–57.
Alharbi, M. (2015). Effects of Blackboard’s discussion boards, blogs and Wikis on
effective integration and development of literacy skills in EFL students. English
Language Teaching, 8(6), 111.
Ali, A. D. (2016). Effectiveness of using Screencast feedback on EFL students’ writing
and perception. English Language Teaching, 9(8), 106.
Archer, K., Savage, R., Sanghera-Sidhu, S., Wood, E., Gottardo, A., & Chen, V. (2014).
Examining the effectiveness of technology use in classrooms: A tertiary meta-analysis.
Computers & Education, 78, 140–149.
Arnold, N., Ducate, L., & Kost, C. (2012). Collaboration or cooperation? Analyzing
group dynamics and revision processes in wikis. Calico Journal, 29(3), 431.
Arslan, R. Ş., & Şahin-Kızıl, A. (2010). How can the use of blog software facilitate the
writing process of English language learners? Computer Assisted Language Learning,
23(3), 183–197.
Azari, M. H. (2017). Effect of weblog-based process approach on EFL learners’ writing
performance and autonomy. Computer Assisted Language Learning, 30, 529–551.
Bamanger, E. M., & Alhassan, R. A. (2015). Exploring Podcasting in English as a for-
eign language learners’ writing performance. Journal of Education and Practice, 6(11),
63–74.
Bigelow, M., & Schwarz, R. L. (2010). Adult English Language Learners with Limited
Literacy. Washington, DC: National Institute for Literacy.
Chandler, D. (1997). An introduction to genre theory. The Media and Communications
Studies Site.
Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction.
Cognition and Instruction, 8(4), 293–332.
 COMPUTER ASSISTED LANGUAGE LEARNING 159

Chen, T. (2016). Technology-supported peer feedback in ESL/EFL writing classes: A

research synthesis. Computer Assisted Language Learning, 29(2), 365–397.
Cheung, A. C., & Slavin, R. E. (2012). How features of educational technology applica-
tions affect student reading outcomes: A meta-analysis. Educational Research Review,
7(3), 198–215.
Cheung, A. C., & Slavin, R. E. (2013). Effects of educational technology applications on
reading outcomes for struggling readers: A best-evidence synthesis. Reading Research
Quarterly, 48(3), 277–299.
Chu, S. K., Capio, C. M., van Aalst, J. C., & Cheng, E. W. (2017). Evaluating the use
of a social media tool for collaborative group writing of a secondary school students
in Hong Kong. Computers & Education, 110, 170–180.
Ciftci, H., & Kocoglu, Z. (2012). Effects of peer e-feedback on Turkish EFL students’
writing performance. Journal of Educational Computing Research, 46(1), 61–84.
Cohen, Jacob (1988) Statistical Power Analysis for the Behavioral Sciences (2nd ed.)
Hillsdale, NJ: Lawrence Erlbaum.
Coryell, J. E., & Chlup, D. T. (2007). Implementing e-learning components with adult
English language learners: Vital factors and lessons learned. Computer Assisted
Language Learning, 20(3), 263–278.
Cumming, A., Kantor, R., Powers, D., Santos, T., & Taylor, C. (2000). TOEFL 2000 writ-
ing framework. In TOEFL-MS-18. Educational Testing Service Princeton, NJ.
Devitt, A. J. (1993). Generalizing about genre: New conceptions of an old concept.
College Composition and Communication, 44(4), 573–586.
Dizon, G. (2016). A comparative study of Facebook vs. paper-and-pencil writing to
improve L2 writing skills. Computer Assisted Language Learning, 29(8), 1249–1258.
Ellis, R. (2008). The study of second language acquisition (2nd ed.). Oxford: Oxford
University Press.
Ellis, R., & Barkhuizen, G. (2005). Analysing learner language. Oxford: Oxford
University Press.
Ferriman, N. (2013). The impact of blended e-learning on undergraduate academic
essay writing in English (L2). Computers & Education, 60(1), 243–253.
Gillespie, A., Olinghouse, N. G., & Graham, S. (2013). Fifth-grade students’ knowledge about
writing process and writing genres. The Elementary School Journal, 113(4), 565–588.
Golonka, E. M., Bowles, A. R., Frank, V. M., Richardson, D. L., & Freynik, S. (2014).
Technologies for foreign language learning: A review of technology types and their
effectiveness. Computer Assisted Language Learning, 27(1), 70–105.
Graham, S. (2006). Writing. In A. Winne (Ed.), Handbook of educational psychology (pp.
457–478). Mahwah, NJ: Erlbaum.
Graham, S., Liu, X., Aitken, A., Ng, C., Bartlett, B., Harris, K. R., & Holzapfel, J. (2017).
Effectiveness of literacy programs balancing reading and writing instruction: A meta-
analysis. Reading Research Quarterly, 53, 279–304.
Graham, S., McKeown, D., Kiuhara, S., & Harris, K. R. (2012). A meta-analysis of writ-
ing instruction for students in the elementary grades. Journal of Educational
Psychology, 104, 879–896.
Harklau, L. (2002). The role of writing in classroom second language acquisition.
Journal of Second Language Writing, 11(4), 329–350.
Hayes, J. (1996). A new framework for understanding cognition and affect in writing. In
Levy, M. & Ransdell, S. (Eds.), The science of writing: Theories, methods, individual
differences, and applications (pp. 1–27). Mahwah, NJ: Erlbaum.
160 Z. XU ET AL.

Hayes, J. R. (2000). A new framework for understanding cognition and affect in writing.
In R. Indrisano & J. R. Squire (Eds.), Perspectives on writing: Research, theory, and
practice (pp. 6–44). Newark, DE: International Reading Association.
Hayes, J. R. (2011). Kinds of knowledge-telling: Modeling early writing development.
Journal of Writing Research, 3(2), 73–92.
Hill, M. (1991). Writing summaries promotes thinking and learning across the curricu-
lum: But why are they so difficult to write? Journal of Reading, 34(7), 536–539.
Hillocks, G. (2005). At last: The focus on form vs. content in teaching writing. Research
in the Teaching of English, 40(2), 238–248.
Housen, A., & Kuiken, F. (2009). Complexity, accuracy, and fluency in second language
acquisition. Applied Linguistics, 30(4), 461–473.
Hyland, K. (2003). Genre-based pedagogies: A social response to process. Journal of
Second Language Writing, 12(1), 17–29.
Hyland, K. (2007). Genre pedagogy: Language, literacy and L2 writing instruction.
Journal of Second Language Writing, 16(3), 148–164.
Hyon, S. (1996). Genre in three traditions: Implications for ESL. TESOL Quarterly,
30(4), 693–722.
John-Steiner, V., & Mahn, H. (1996). Sociocultural approaches to learning and develop-
ment: A Vygotskian framework. Educational Psychologist, 31(3–4), 191–206.
Kashani, H., Mahmud, R. B., & Kalajahi, S. A. R. (2013). Comparing the effect of blog-
ging as well as pen-and-paper on the essay writing performance of Iranian graduate
students. English Language Teaching, 6(10), 202.
Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social
interaction in computer-supported collaborative learning environments: A review of
the research. Computers in Human Behavor, 19(3), 335–353.
Kwon, K., Liu, Y. H., & Johnson, L. P. (2014). Group regulation and social-emotional
interactions observed in computer supported collaborative learning: Comparision
between good vs. poor collaborators. Computers & Edcuation, 78, 185–200.
Ku, H. Y., Tseng, H. W., & Akarasriworn, C. (2013). Collaboration factors, teamwork
satisfaction, and student attitudes toward online collaborative learning. Computers in
Human Behavior, 29(3), 922–929.
Lantolf, J. P. (2006). Sociocultural theory and L2: State of the art. Studies in Second
Language Acquisition, 28(1), 67–109.
Lantolf, J. P, & Thorne, S.L. (2006). Sociocultural theory and the genesis of second lan-
guage development. Oxford, England:Oxford University Press.
Lantolf, J. P., Thorne, S. L., & Poehner, M. E. (2015). Sociocultural theory and second
language development. Theories in second language acquisition: An introduction (pp.
207–226). New York: Routledge.
Limbu, L., & Markauskaite, L. (2015). How do learners experience joint writing:
University students’ conceptions of online collaborative writing tasks and environ-
ments. Computers & Education, 82, 393–408.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA:
Sage.
Liu, M., Navarret, C.C., & Wivagg, J. (2014). Potentials of mobile technology for k-12
education: An investigation of iPod touch use for English language learners in the
United States. Journal of Educational Technology & Society, (17)2.
Maune, M. J. (2016). Toward a genre writing curriculum: Schooling genres in the
Common Core State Standards (Doctoral dissertation). Purdue University.
 COMPUTER ASSISTED LANGUAGE LEARNING 161

McCutchen, D. (1986). Domain knowledge and linguistic knowledge in the development

of writing ability. Journal of Memory and Language, 25, 431–444.
Mohan, B. A., & Lo, W. A. Y. (1985). Academic writing and Chinese students: Transfer
and developmental factors. TESOL Quarterly, 19(3), 515–534.
Nazari, B., & Niknejad, S. (2015). E-mail writing: Providing background information in
the core of computer assisted instruction. Turkish Online Journal of Distance
Education, 16(1), 193–211.
Olinghouse, N. G., & Graham, S. (2009). The relationship between the discourse know-
ledge and the writing performance of elementary-grade students. Journal of
Educational Psychology, 101(1), 37.
Olinghouse, N. G., & Wilson, J. (2013). The relationship between vocabulary and writing
quality in three genres. Reading and Writing, 26(1), 45–65.
Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design:
Recent developments. Educational Psychologist, 38(1), 1–4.
Qin, W., & Uccelli, P. (2016). Same language, different functions: A cross-genre analysis of
Chinese EFL learners’ writing performance. Journal of Second Language Writing, 33, 3–17.
Schmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R. M., Abrami, P. C., Surkes,
M. A., … Woods, J. (2014). The effects of technology use in postsecondary educa-
tion: A meta-analysis of classroom applications. Computers & Education, 72, 271–291.
Shadiev, R., Hwang, W.-Y., & Huang, Y.-M. (2017). Review of research on mobile lan-
guage learning in authentic environments. Computer Assisted Language Learning,
30(3–4), 284–303.
Shintani, N., & Aubrey, S. (2016). The Effectiveness of synchronous and asynchronous
written corrective feedback on grammatical accuracy in a computer-mediated environ-
ment. The Modern Language Journal, 100(1), 296–319.
Skehan, P. 1998. A cognitive approach to language learning. Oxford: Oxford University
Press.
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011).
What forty years of research says about the impact of technology on learning: A
second-order meta-analysis and validation study. Review of Educational Research,
81(1), 4–28.
Tremmel, M. (2011). What to make of the five-paragraph theme: History of the genre
and implications. Teaching English in the Two Year College, 39(1), 29.
Wang, Y.-C. (2015). Promoting collaborative writing through wikis: A new approach for
advancing innovative and active learning in an ESP context. Computer Assisted
Language Learning, 28(6), 499–512.
Weissberg, B. (2000). Developmental relationships in the acquisition of English syntax:
Writing vs. speech. Learning and Instruction, 10(1), 37–53.
Wichadee, S. (2013). Improving students’ summary writing ability through collabor-
ation: A comparison between online Wiki group and conventional face-to-face group.
TOJET: The Turkish Online Journal of Educational Technology, 12(3), 107–116.
Williams, J. (2012). The potential role (s) of writing in second language development.
Journal of Second Language Writing, 21(4), 321–331.
Yang, Y.-F. (2015). Automatic scaffolding and measurement of concept mapping for
EFL students to write summaries. Journal of Educational Technology & Society, 18(4),
273–286.
Yang, Y.-F. (2016). Transforming and constructing academic knowledge through online
peer feedback in summary writing. Computer Assisted Language Learning, 29(4),
683–702.
162 Z. XU ET AL.

Yeh, H.-C. (2015). Facilitating metacognitive processes of academic genre-based writing

using an online writing system. Computer Assisted Language Learning, 28(6), 479–498.
Zaini, A., & Mazdayasna, G. (2015). The impact of computer-based instruction on the
development of EFL learners’ writing skills. Journal of Computer Assisted Learning,
31(6), 516–528.
Zhao, Y. (2003). Recent developments in technology and language learning: A literature
review and meta-analysis. CALICO Journal, 21, 7–27.
Zhu, Y., Mark Shum, S.-K., Brian Tse, S.-K., & Liu, J. J. (2016). Word-processor or pen-
cil-and-paper? A comparison of students’ writing in Chinese as a foreign language.
Computer Assisted Language Learning, 29(3), 596–617.
Zou, B., Wang, D., & Xing, M. (2016). Collaborative tasks in Wiki-based environment
in EFL learning. Computer Assisted Language Learning, 29(5), 1001–1018.