Abraham 2008

This article was downloaded by: [Tulane University]
On: 30 August 2014, At: 20:32

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered
office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Computer Assisted Language Learning

Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/ncal20
Computer-mediated glosses in second

language reading comprehension and
vocabulary learning: A meta-analysis
a
Lee B. Abraham
a
Department of Modern Languages and Literatures , Villanova
University , Villanova, USA
Published online: 29 May 2008.
To cite this article: Lee B. Abraham (2008) Computer-mediated glosses in second language reading
comprehension and vocabulary learning: A meta-analysis, Computer Assisted Language Learning,
21:3, 199-226, DOI: 10.1080/09588220802090246
To link to this article: http://dx.doi.org/10.1080/09588220802090246
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the
“Content”) contained in the publications on our platform. However, Taylor & Francis,
our agents, and our licensors make no representations or warranties whatsoever as to
the accuracy, completeness, or suitability for any purpose of the Content. Any opinions
and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content
should not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions, claims,
proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or
howsoever caused arising directly or indirectly in connection with, in relation to or arising
out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Conditions of access and use can be found at http://www.tandfonline.com/page/terms-
and-conditions
Vol. 21, No. 3, July 2008, 199–226
Computer-mediated glosses in second language reading comprehension and

vocabulary learning: A meta-analysis
Lee B. Abraham*
Department of Modern Languages and Literatures, Villanova University, Villanova, USA

(Received 5 October 2007; final version received 12 May 2008)
Language learners have unprecedented opportunities for developing second language

literacy skills and intercultural understanding by reading authentic texts on the Internet
Downloaded by [Tulane University] at 20:32 30 August 2014
and in multimedia computer-assisted language learning environments. This article

presents findings from a meta-analysis of 11 studies of computer-mediated glosses in
second language reading comprehension and incidental vocabulary learning. Computer-
mediated glosses had an overall medium effect on second language reading
comprehension and a large effect on incidental vocabulary learning. Mean effect sizes
varied from medium to large depending upon the level of instruction, text type, and
assessment tasks. Drawing upon findings of this meta-analysis and the overall
characteristics of gloss studies, we propose recommendations for future research,
including replications and the systematic study of reading variables and learner
individual differences in multimedia learning environments with authentic texts.
Keywords: second language learning; reading comprehension; vocabulary learning;
computer-assisted instruction; computer-assisted language learning; gloss; annotation;
electronic dictionary; internet; World Wide Web; meta-analysis
Introduction
Reading authentic second language (L2) texts has long been recognized as essential for
developing L2 literacy and intercultural knowledge (Gilmore, 2007; Kern, 2003). Research
on the comprehension of authentic texts has examined learners’ cognitive processes and
strategy use as well as the effectiveness of tasks and modifications of texts in L2 print-
based and computer-mediated formats to aid comprehension and retention of new
vocabulary (Brandl, 2002; Murray & McPherson, 2006; Yano, Long, & Ross, 1994;
Young, 1999). CALL has provided an important pedagogical foundation for the design of
L2 computer-mediated reading tasks by empirically studying the effects of modifying texts
to enhance comprehension and incidental vocabulary learning (Chun, 2006).
The design of computer-based lesson templates based on L2 reading research to
facilitate comprehension of authentic texts and retention of vocabulary has been the
subject of much interest, particularly as the use of CALL authoring software increases and
as the availability of authentic materials on the World Wide Web expands exponentially
(Ariew, 2006; Cobb & Stevens, 1996; Lyman-Hager & Davis 1996; LeLoup & Ponterio,
2005; Martı́nez-Lage, 1997). As Lyman-Hager (2000) points out, CALL lessons with
*Email: lee.abraham@villanova.edu
ISSN 0958-8221 print/ISSN 1744-3210 online

Ó 2008 Taylor & Francis
DOI: 10.1080/09588220802090246
http://www.informaworld.com
200 L.B. Abraham
computer-mediated glosses in particular, can provide ‘insights about specific texts and
cultures . . . bridging the gap between language and literature and between the reader and
the text’ (p. 441). In fact, CALL has contributed to understanding the comprehension of
authentic L2 texts by tracking learners’ use of computer-mediated glosses and dictionaries
and, in particular, by studying the role of individual differences (Blake, 1992; Chun &
Plass, 1997; Fischer, 2007; Ganderton, 1999; Lomicka, 1998).
A central concern for CALL reading research is the study of the relationship between
learners’ individual differences (L2 proficiency, visual and verbal abilities) and the effective
design of multimedia environments (advance organizers, use of verbal and visual glosses)
to facilitate L2 text comprehension and vocabulary learning. As Chun & Plass (1997) point
out, it is essential to understand how visual and verbal features of multimedia learning
materials can be effectively designed to support learners’ individual differences (p. 72). For
almost a decade, CALL reading research has investigated the effects of vocabulary
(lexical) glosses on L2 reading and vocabulary in multimedia environments, including
individual differences such as level of instruction and learning styles (Chun, 2006; De
Ridder, 2003; Plass & Jones, 2005).
This article examines the cumulative quantitative evidence of the use of glosses in
multimedia learning environments for enhancing L2 reading comprehension and
incidental vocabulary learning.1 In particular, we conducted a meta-analysis of 11
empirical studies on the overall effect of computer-mediated glosses, and specifically
address the effect of glosses with regard to the following factors:
(1) level of instruction

(2) text type (genre), and
(3) reading comprehension and vocabulary assessments.
The purpose of this meta-analysis is to provide a systematic and replicable statistical

synthesis of the impact of these factors which have been identified as essential
considerations in theoretical models and research in L2 reading and multimedia learning
environments (Bernhardt, 2005; Chun, 2006; Koda, 2005; Plass & Jones, 2005).
Understanding the relationship between glosses, instructional variables, and learner
individual differences has important implications for the effective design of multimedia
environments since authentic texts increasingly have become the curricular focus in
language learning and teaching. Before discussing the present study, we briefly review
research on the importance of L2 vocabulary knowledge for text comprehension in print
and multimedia environments and discuss recent findings from qualitative and
quantitative syntheses of CALL research.
Review of literature
Comprehending authentic L2 texts in computer-mediated and print-based environments
involves the coordination of lower-level processes (L2 word recognition skills, L2
vocabulary knowledge), higher-level processes (background knowledge/topic familiarity,
L2 text structure, syntax), and their relationship with learner-based factors such as
strategy use, reading purpose, motivation, and interest (Chun, 2006, Chun & Plass, 1997,
Grabe & Stoller, 2002, Koda, 2005). Higher- and lower-level processes ‘operate
synchronically, interactively, and synergistically . . . the more word knowledge is devel-
oped, the more it frees up resources to operate on more complex syntactic patterns’
(Bernhardt, 2005, p. 140). Since learners have limited cognitive resources available in
Computer Assisted Language Learning 201
working memory to manage higher- and lower level cognitive processes while reading,
CALL is widely-believed to play an important role by facilitating text comprehension and
retention of vocabulary with immediate and individualized support specifically designed to
free-up these finite cognitive resources (Chun, 2006; Just & Carpenter, 1992; Kintsch,
1998; Martı́nez-Lage, 1997).
L2 vocabulary knowledge, reading comprehension and the role of glosses

L2 vocabulary knowledge has been identified as one of the primary factors in under-
standing authentic texts. Successful comprehension requires automatic recognition and
decoding of 95 to 99% of words in a text (Laufer, 1997; Nation, 2001). Since vocabulary is
essential for L2 text comprehension, studies have examined how learners process authentic
texts and, in particular, how learners approach unfamiliar vocabulary.
When reading authentic texts, learners do not necessarily look up unknown words and
may ignore these words (Paribahkt & Wesche, 1997, Qian, 2004). L2 readers may not
necessarily recognize words that they do not know (Laufer & Yano, 2001) or guess
incorrectly (Frantzen, 2003; Kondo-Brown, 2006). Among the important variables, both
text-based and learner-based, identified for successful guessing while reading are the
following:
(1) learners’ L2 proficiency (Bengeleil & Paribahkt, 2004);

(2) learners’ use of appropriate guessing strategies (Nassaji, 2003);
(3) the number of times unknown words occur in a text (Rott, 1999; Webb, 2007);
(4) learners’ current level of L2 vocabulary knowledge (Laufer, 1997; Nation, 2001;
Nassaji, 2004; Pulido, 2007); and
(5) the density of unknown L2 words in a text (Nation, 2001).
Although significant knowledge of vocabulary is essential for L2 text comprehension,

the research described previously indicates that understanding unknown words in
authentic texts presents significant challenges for L2 readers. As such, CALL reading
studies have increasingly investigated the effects of providing immediate access to the
meaning of words through glosses (Chun, 2006; Plass & Jones, 2005; Van de Poel &
Swanepoel, 2003). ‘Online glossing is thought to provide fast and easy access to the
meaning of unknown words and to compensate for insufficiently automatic lower-
level processes and thus allows the reader to attend to higher level processes’ (Chun, 2006,
p. 70).
Synthesizing CALL research

Recent studies have examined the accumulated CALL empirical research base using
several approaches. In a qualitative synthesis of CALL research, Liu, Moore, Graham and
Lee (2003) suggest that future studies have solid theoretical foundations, focus on the less-
explored skills of speaking and listening, include learner variables such as anxiety and
attitude, and focus more on the effectiveness of CALL technologies in primary and
secondary education. Felix (2005a, 2005b) reviews a range of CALL studies and
recommends meta-analysis, qualitative syntheses and replications of exemplary empirical
studies that include a larger number of participants in different educational contexts.
Among the recommendations for additional research, Hubbard (2005) points out the need
for longitudinal research as well as the importance of reporting data and methodologies in
202 L.B. Abraham
greater detail and of studying learners who already possess extensive experience with
CALL-oriented technologies.
Meta-analysis and CALL

Representing one of several approaches to synthesizing, analyzing, and interpreting
empirical studies accumulated in a research domain, a meta-analysis systematically reviews
and extracts quantitative data from studies which are then, in turn, ‘analyzed using special
adaptations of conventional statistical techniques to investigate and describe the pattern of
findings in the selected set of studies’ (Lipsey & Wilson, 2001, p. 2). As illustrated below,
we describe the magnitude of the effects that have been found in studies of L2 computer-
mediated glosses by extracting quantitative data and calculating a standardized mean
difference effect size referred to as Cohen’s d. The standardized mean difference effect size
indicates the amount of difference between a control group (e.g., no gloss access) and an
experimental group (e.g., gloss access) measured in standard deviation units. We follow
the benchmarks proposed by Cohen (1988) for interpreting the magnitude of the effect of
using computer-mediated glosses for L2 reading comprehension and incidental vocabulary
learning:
(a) small effect (d is less than .20);

(b) medium effect (d is between .21 and .80); and
(c) large effect (d is greater than .80).
Meta-analysis and other approaches to synthesizing studies have become increasingly

important in CALL and L2 learning and teaching for systematically identifying patterns in
research findings which, in turn, provide specific directions for future research and
pedagogical practices (Felix, 2005b; Norris & Ortega, 2006). Zhao’s (2003) meta-analysis
of nine studies finds a large positive effect of technologies on improving learning (weighted
d ¼ 1.12). Taylor (2006) finds that native language (L1) computer-mediated glosses have a
statistically significantly larger effect on L2 reading comprehension (weighted d ¼ 1.09)
than L1 paper-based glosses (weighted d ¼ .39). With increasing integration of authentic
texts in language teaching and the use of software and Web-based technologies for
supporting L2 reading processes and vocabulary learning, the present meta-analysis builds
on Taylor’s (2006) analysis by examining the quantitative CALL reading studies of glosses
in relation to individual differences and pedagogical factors.
Methodology
The meta-analysis described in the sections to follow is guided by the following questions.
(1) What is the effect of computer-mediated glosses on L2 reading comprehension and

incidental vocabulary learning?
(2) What is the relationship between level of instruction and the effects of computer-
mediated glosses on L2 reading comprehension and incidental vocabulary learning?
(3) What is the effect of text type (genre) on L2 reading comprehension and incidental
vocabulary learning with computer-mediated glosses?
(4) What is the effect of type of assessment on L2 reading comprehension and
incidental vocabulary learning with computer-mediated glosses?
The present study was conducted in the following stages:

(1) identification, retrieval, and selection of eligible studies;
(2) determination of study inclusion and exclusion criteria;
(3) coding of study characteristics;
(4) extraction and calculation of effect sizes; and
(5) analysis and interpretation of effect size data (Cooper & Hedges, 1994; Lipsey &
Wilson, 2001).
Search and retrieval of studies
Searches for relevant studies were conducted in electronic databases using the following
keywords with ‘and’ as the Boolean search term: ‘annotation, ‘computer-assisted reading’,
‘computer-assisted language learning’, ‘computer-assisted instruction’, ‘dictionary’,
‘electronic dictionary’, ‘gloss’, ‘glosses’, ‘glossary’, ‘hypertext’, ‘hypermedia’, ‘lexicon’,

‘lexicography’, ‘multimedia’, ‘reading’, ‘reading comprehension’, ‘second language read-
ing’, ‘second language vocabulary learning’, ‘vocabulary’.
The following electronic databases were employed in this study:
(1) ArticleFirst
(2) Arts & Humanities Search
(3) Communication Abstracts
(4) Dissertation Abstracts
(5) Education Abstracts
(6) Education Resources Information Center (ERIC)
(7) Humanities Abstracts
(8) JSTOR [(French Review, 1927–2001; Hispania, 1917–2001; Italica, 1926–2001;
Journals of the Association of Teachers of Japanese, 1972–2000; Unterricht-
spraxis/Teaching German, 1968–2003)]
(9) Linguistics and Language Behavior Abstracts (LLBA)
(10) MLA Bibliography
(11) PsychINFO
(12) Scopus
(13) Social Sciences Abstracts
(14) Social Science Citation Index.
This database search had no beginning cut-off date and any study published up to and
including September 2007 was considered. References from recent reviews of computer-
assisted language learning, second language reading comprehension, and vocabulary
learning were also consulted (Chun, 2006; Plass & Jones, 2005). Bibliographies from
articles, chapters, refereed conference papers, and books retrieved from the electronic
databases mentioned previously were also cross-referenced.
Study eligibility: inclusion and exclusion criteria

The search for eligible studies was conducted in two phases. In phase one, we retrieved 125
books, articles, book chapters, and refereed conference papers using the search strategy
described previously. The purpose of this phase was to provide an overview of the
204 L.B. Abraham
variables examined in computer-mediated L2 reading with glosses. Published and

unpublished sources were eligible for inclusion in this meta-analysis.
In phase two, a second researcher and the author independently read each of the 125
sources. Eligible studies fulfilled one of the following two criteria:
(1) The study consisted of an experimental (treatment) group in which learners had
access to computer-mediated glosses and read an authentic or researcher-created
text in a computer-mediated environment (software or online); or
(2) The study included an experimental (treatment) group in which learners had access
to computer-mediated glosses and read an authentic or researcher-created text in a
paper-based (print) environment.
Appendices A and B present definitions of each of these coding categories as well as the
coding book.
As illustrated in Appendix C, 62 of the 82 studies examined the effects of computer-

mediated glosses with experimental groups only (i.e., the design of these 62 studies did not
include a control group without access to glosses) with the following independent variables:
(1) paper/print glosses versus computer-mediated glosses;

(2) computer-mediated multimedia glosses versus computer-mediated text glosses;
(3) computer-mediated glosses with visible links versus computer-mediated
glosses with invisible links;
(4) L2 proficiency;
(5) visual and verbal ability;
(6) working memory;
(7) multiple choice versus single glosses;
(8) reading task;
(9) gender; and
(10) learning styles.
One commonly held concern with meta-analysis is the problem of including and
comparing effect sizes of dissimilar control/experimental conditions across studies. We
applied a third and final eligibility criterion to these 82 studies in order to be certain that
studies were similar enough for inclusion and not so different from each other that they
could have been considered incommensurable and could have confounded the
comparisons of effect sizes of experimental and control groups (Lipsey & Wilson, 2001,
pp. 7–10; Norris & Ortega, 2006, pp. 16–18).
(3) The study included a control group which did not have access to either print/paper
or computer-mediated glosses nor any other type of aid (e.g., paper dictionary,
thesaurus, textbook). Learners read an authentic or researcher-created text in a
paper-based (print) environment.
After applying the third criterion, 11 individual documents with seven studies examining
both reading comprehension and incidental vocabulary learning and four studies investigat-
ing reading comprehension comprise the present meta-analytic study (Appendix D). Four
studies were excluded since these did not have data available needed to calculate an effect size
(Appendix E). In the references, one asterisk refers to one of the 82 studies coded in phase one
and two asterisks identify one of the 11 studies in the meta-analysis.
Coding of studying categories and intercoder reliability

The author and a second researcher independently read and coded each of the 82 studies
according to a predefined coding book and coding instructions presented in Appendices A
and B. We based our categories on factors identified in the research on L2 reading
comprehension, L2 incidental vocabulary learning, and multimedia learning (Bernhardt,
2005; Grabe & Stoller, 2002; Koda, 2005; Plass & Jones, 2005). Intercoder agreement rates
and Cohen’s Kappa coefficients for independent variables are summarized in Appendix F
(Cohen, 1960; Orwin, 1994). Several of the categories in Appendix B such as ‘L1 reading
ability’, ‘L2 reading ability’, ‘text readability’ and ‘reading/reader purpose’ either could
not be obtained from all 82 studies or could not be reliably coded and thus, were
eliminated from further analysis.
Effect size extraction and calculation

To obtain a statistically independent set of effect sizes for the 82 studies, means,
standard deviations, and sample sizes were extracted for each vocabulary post-test
(immediate and delayed) and reading comprehension test and transformed into a
standardized mean difference between an experimental (treatment) using computer-
mediated text glosses and control groups without access to these glosses (Lipsey &
Wilson, 2001, pp. 112–114). For studies using more than one outcome measure of
either reading comprehension or vocabulary learning, the most common measure
among the 11 studies in the meta-analysis was selected and used in the calculation of
the effect size (Lipsey & Wilson, 2001, p. 113). Cohen’s d, a standardized mean
difference effect size, was obtained for each vocabulary post-test (immediate and
delayed) and reading comprehension test. Cohen’s d was then used to calculate Hedges’
unbiased estimate of the standardized mean difference effect size (Hedges’ g), in order
to account for differences in sample sizes (Hedges & Olkin, 1985, p. 81; Lipsey &
Wilson, 2001, pp. 49, 113–114). To determine the statistical significance of the
aggregated effect sizes, a 95% confidence interval was constructed around each
weighted mean effect size for immediate and delayed vocabulary post-test as well as for
reading comprehension (Lipsey & Wilson, 2001, p. 132). Tables 1, 2 and 3 present
weighted effect sizes for each study.
Homogeneity of variance tests (Q-test) were calculated in order to determine whether
the distribution of weighted effect sizes for studies of reading comprehension and
vocabulary learning was normal and due only to sampling error. A statistically significant
Q-test indicates that variation in weighted effect sizes is in excess of sampling error alone
and may be accounted for by examining moderating variables such as those recorded in
the coding book (Appendix B).
Effect sizes, confidence intervals, homogeneity of variance tests and Orwin’s (1983) fail-
safe N were calculated by using Comprehensive Meta-Analysis (2007) software. Tables 4,
5 and 6 summarize the descriptive statistics and effect sizes for studies in this meta-
analysis. As mentioned previously, we used Cohen’s (1988) criteria for interpreting the
magnitude of the effect of computer-mediated glosses on reading comprehension and
vocabulary learning:
(1) small effect (less than .20);

(2) medium effect (between .21 and .80); and
(3) large effect (greater than .80).
206 L.B. Abraham
Table 1. Individual effect sizes for L2 reading comprehension.
Immediate
post-test
Immediate standard
Sample size post-test mean deviation
Weighted
No gloss Gloss No gloss Gloss No gloss Gloss effect size
Abraham 2007 34 34 2.20 4.30 2.06 2.33 .94

Ben Salem 2007 18 19 9.06 18.89 3.10 2.51 3.42
Bowles 2004 18 14 3.89 7.64 1.84 2.53 1.69
Coriano Velázquez 2001 12 22 7.17 6.59 2.33 2.36 7.24
DeRidder 2003 28 28 32.36 41.60 18.14 22.25 .45
Hayden 1996 20 16 9.40 14.50 4.94 12.80 .54
Knight 1994 51 54 56.65 74.01 23.35 27.29 .68
Lomicka 1998 4 4 19.88 19.14 9.02 9.88 7.07
Plass, Chun, 38 38 9.30 7.90 3.30 4.10 7.37
Mayer & Leutner 2003

Stoehr 1999 29 33 8.03 15.33 4.26 5.31 1.49
Youngs 1994 13 15 7.00 6.80 1.15 1.61 7.14
Table 2. Individual effect sizes for immediate L2 vocabulary post-tests.
Immediate
Immediate post-test standard
Weighted
Abraham 2007 34 34 6.90 9.40 2.77 3.27 .83

Ben Salem 2007 18 19 6.72 11.53 1.81 1.43 2.90
Bowles 2004a 14 18 d-value d-value d-value d-value 1.76
Coriano 12 22 1.67 2.05 1.50 1.33 .27
Velázquez 2001
DeRidder 2003 28 28 .17 .35 .11 .12 1.54
Knight 1994 51 54 8.75 14.56 3.38 4.49 1.45
a
Cohen’s d effect size (d ¼ 1.8) was reported and used to calculate the weighted effect size.
Table 3. Individual effect sizes for delayed L2 vocabulary post-tests.
Delayed
post-test
Delayed standard
Weighted
Ben Salem 2007 11 12 5.18 9.44 .98 2.16 2.41

Bowles 2004a 18 14 d-value d-value d-value d-value .95
Coriano Velázquez 2001 12 22 1.08 1.32 1.31 1.04 .21
DeRidder 2003 28 28 .08 .24 0.06 .09 2.06
Knight 1994 51 54 8.06 12.24 3.61 4.33 1.04
Plass, Chun, 38 38 20.50 24.60 3.70 3.30 1.16
Mayer & Leutner 2003
a
Cohen’s d effect size (d ¼ .97) was reported and used to calculate the weighted effect size.
Table 4. Weighted mean effect sizes for L2 reading comprehension tests.
Weighted effect size 95% confidence interval

n M SD Lower Upper
All studies 11 .73 .26 .22 1.25
Level of Instruction
Beginning 3 .48 .67 7.84 1.79
Intermediate 7 .88 .35 .19 1.57
Advanced 1 .45 .27 7.07 .97
Text Type
Expository 4 .64 .29 .07 1.22
Narrative 6 .91 .45 .03 1.80
Poetry 1 7.07 .61 71.27 1.14
Assessment
Multiple choice 2 2.53 .87 .83 4.22
Multiple choice & true/false 1 7.24 .35 7.93 .45
Questions 1 .45 .27 7.07 .97

Recall 6 .53 .28 7.02 1.08
Think-aloud 1 7.07 .62 71.27 1.14
Note: n ¼ number of empirical studies contributing one effect size. A random effects model was used for this
analysis.
Table 5. Weighted mean effect sizes for immediate L2 vocabulary post-tests.

n M SD Lower Upper
All studies 6 1.40 .29 .83 1.98
Beginning 2 1.00 .74 7.46 2.45
Intermediate 3 1.63 .47 .71 2.56
Advanced 1 1.54 .30 .95 2.13
Text Type
Expository 3 1.52 .16 1.20 1.84
Narrative 3 1.28 .66 7.004 2.57
Assessment
Productive 2 .60 .27 .07 1.12
Receptive 4 1.81 .28 1.27 2.35
analysis.
Results
Characteristics of the studies
The 11 documents included in this meta-analysis yielded 11 studies of reading
comprehension comparing the effects of learners’ access to computer-mediated glosses
to those learners without access to glosses and six studies comparing these same two
treatments (i.e., no gloss vs. gloss access) on measures of L2 incidental vocabulary.2 We
then calculated a weighted effect size, a standardized difference between the means of the
control (no access to glosses) group and experimental (access to glosses) group that takes
208 L.B. Abraham
Table 6. Weighted mean effect sizes for delayed L2 vocabulary post-tests.

n M SD Lower Upper
All studies 6 1.25 .26 .73 1.76
Beginning 2 .57 .37 7.16 1.29
Intermediate 3 1.34 .29 .78 1.91
Advanced 1 2.06 .33 1.42 2.71
Text Type
Expository 3 1.34 .34 .67 2.00
Narrative 3 1.19 .52 .17 2.21
Assessment
Productive 1 .21 .35 7.48 .89
Receptive 5 1.43 .25 .94 1.91
analysis.
into account differences in study samples sizes, for each study of reading comprehension
and vocabulary learning. Selected study characteristics and weighted effect sizes are
presented in Appendix G and Appendix H.
All studies investigated the effects of computer-mediated glosses at the university-
level. More than half of the studies involved intermediate learners (reading
comprehension ¼ 64%, vocabulary ¼ 57%). The median sample size of reading studies
was 37 and for vocabulary learning the median was 47. For reading comprehension
eight studies examined Romance languages (French ¼ 4, Spanish ¼ 4) and three
focused on Germanic languages (English ¼ 1, German ¼ 2) (Appendix G). The six
immediate vocabulary post-test studies include one study of English and French and
four studies of Spanish (Appendix H). For the six delayed vocabulary tests, the meta-
analysis consists of three studies of Spanish and one study each of English, French,
and German (Appendix H).
Overall effect of glosses

The first research question concerned the overall effect of computer-mediated glosses
on reading and vocabulary learning. As indicated in Table 4, we found a medium
overall weighted mean effect size of computer-mediated glosses on reading com-
prehension (m ¼ .73). The moderately wide confidence interval of .22 to 1.25 suggests
that we need additional studies before we can be certain of the overall medium
effect on reading comprehension. Table 5 shows an overall large effect for the
immediate vocabulary post-test (m ¼ 1.40) and Table 6 indicates a similar large
weighted mean effect size for the delayed vocabulary test (m ¼ 1.25) when compared to
control groups without access to this lexical support. The mean weighted effect of
computer-mediated glosses on incidental L2 vocabulary learning remains robust over
time (Tables 5 and 6). The confidence intervals for immediate (.83 to 1.98) and delayed
(.73 to 1.76) post-tests are wide due to the small number of observations available
for the meta-analysis. Additional studies of the long-term retention of vocabulary
with computer-mediated glosses are needed in order to substantiate these tentative
findings.
Variables moderating the effect of glosses

Tests of homogeneity of variance (Q-test) indicated significantly more variation among
effect sizes than one would expect for sampling error for L2 reading comprehension
(Qb ¼ 78.52, df ¼ 10, p ¼ .005, se ¼ .26) as well as for immediate (Qb ¼ 26.43, df ¼ 5,
p ¼ .000, se ¼ .29) and delayed (Qb ¼ 21.08, df ¼ 5, p ¼ .000, se ¼ .26) vocabulary tests.
Therefore, we investigated whether the following variables accounted for some of the
variation in effect sizes:
(1) level of instruction;

(2) text type (genre); and
(3) type of assessment.
Level of instruction
The second research question sought to account for variation in effect sizes by examining
the influence of level of instruction.3 As shown in Table 4, a medium effect size was found
for reading comprehension for both beginning- (m ¼ .48) and advanced-level learners
(m ¼ .45). For intermediate learners a large effect size was observed (m ¼ .88). The
difference between beginning, intermediate, and advanced-level groups was not statistically
significant (Qb ¼ .97, df ¼ 2, p ¼ .62). For performance on immediate L2 vocabulary
post-tests (Table 5), glosses were most effective for intermediate-level (m ¼ 1.63) and least
effective for beginning-level learners (m ¼ 1.00). However, the difference between levels of
instruction was not statistically significant (Qb ¼ .56, df ¼ 2, p ¼ .76). A similar pattern
was also observed for delayed vocabulary tests (Table 6) with beginning learners benefiting
the least (m ¼ .57). Large effects were found for intermediate (m ¼ 1.34) and advanced
learners (m ¼ 2.06), although these should be interpreted with caution as only one study
involved advanced-level learners. Differences between levels of instruction were statistically
significant (Qb ¼ 9.17, df ¼ 2, p ¼ .01) but should be interpreted with caution due to the
relatively small number of studies for each level of instruction.
Text type (genre)

To account for the variation on measures of reading comprehension and vocabulary
learning, our third research question examined the effect of the type of text (genre).
Computer-mediated text glosses had a medium effect on comprehension of expository
readings (m ¼ .64) and a large effect for narrative texts (m ¼ .91) as presented in Table 4.
However, the difference between text types was not statistically significant (Qb ¼ 1.67,
df ¼ 2, p ¼ .43). Similarly large effect sizes for expository (m ¼ 1.52) and narrative
(m ¼ 1.28) texts were found for immediate vocabulary tests but these were not statistically
significant (Qb ¼ 0.13, df ¼ 1, p ¼ .72) (Table 5). For delayed vocabulary tests (Table 6),
the effect was large for expository (m ¼ 1.34) and narrative (m ¼ 1.19) texts and was
lower than the effects sizes observed for immediate vocabulary tests. Differences on the
delayed vocabulary test when reading expository and narrative texts were not statistically
significant (Qb ¼ .06, df ¼ 1, p ¼ .81).
Type of assessment
Our fourth research question investigated the relationship between the type of assessment
and outcomes on reading comprehension and incidental vocabulary learning. The
210 L.B. Abraham
homogeneity of variance test (Q-test) for reading comprehension assessment was

statistically significant (Qb ¼ 10.12, df ¼ 4, p ¼ .04). A large effect size was found for
multiple choice tests (m ¼ 2.53). For studies using recall protocols, a medium effect size
(m ¼ .53) was observed (Table 4). The wide confidence intervals for multiple choice tests
(.83 to 4.22) indicate that more research studies are needed in order to verify this finding.
Results for the remaining assessments of reading comprehension should be interpreted
with caution as each study contributed only one effect size to the analysis. As indicated in
Table 5, glosses had a larger effect on learners’ performance on immediate receptive
vocabulary tests (m ¼ 1.81) than on productive assessments (m ¼ .60) and this difference
was statistically significant (Qb ¼ 9.96, df ¼ 1, p ¼ .002). A statistically significant
difference between learners’ performance on delayed productive (m ¼ .21) and receptive
(m ¼ 1.43) post-tests was found (Qb ¼ 8.10, df ¼ 1, p ¼ .004) (Table 6). However, this
finding should be interpreted cautiously as only one studied consisted of a delayed
productive vocabulary measure.
Discussion
Overall, this meta-analysis indicates that learners who have access to computer-mediated
text glosses perform consistently better on measures of L2 reading comprehension than
learners who do not have such support. Similar findings were reported for incidental L2
vocabulary learning and the large positive effect appears to remain over time for
intermediate-level learners. Therefore, modifications of authentic L2 readings by using
computer-mediated text glosses in this accumulated body of CALL research seem to be
important for enhancing L2 comprehension and incidental vocabulary learning (Table 7).
The present study also analyzed the relationship between glosses and three variables
identified as important moderators of outcomes on L2 reading comprehension and
incidental vocabulary learning:
(a) level of instruction;

(b) text type; and
(c) type of assessment.
However, the following observations should be interpreted as suggestive rather than

definitive due to the potential error which may result from conducting multiple
significance tests on the same small sample of studies.
Our finding that glosses were not as useful for beginning learners is consistent with
studies showing that learners at higher levels of L2 proficiency perform better on measures
of reading comprehension (Pichette, 2005). One possible explanation also may be that
beginning learners not only need vocabulary (lexical) glosses but also other types of glosses
that would help them to monitor and facilitate comprehension while reading authentic L2
texts (Chun, 2006; Overstreet, 2006). In fact, Chun & Plass (1997) recommended that
instructional multimedia materials ‘be designed as adaptive systems to support learners
with different traits, such as learning preferences and cognitive styles, so that different
learners can receive the type of information in the mode they need or prefer’ (p. 73).
The large effect of glosses for all levels of instruction on immediate vocabulary tests
remained robust for intermediate learners on delayed receptive vocabulary tests but
decreased by almost half for beginning learners (Table 7). Intermediate learners may
possess deeper lexical knowledge allowing them to connect vocabulary encountered in the
glosses more easily to a pre-existing semantic system and network of L2 vocabulary than
Table 7. Summary of weighted effect sizes.
Weighted Effect Sizea

Reading Immediate Delayed
comprehensionb vocabulary post-testc vocabulary post-testc
Overall .73 1.40 1.25
Beginning .48 1.00 .57
Intermediate .88 1.63 1.34
Advanced .45 1.54 2.06
Text Type
Expository .64 1.52 1.34
Narrative .91 1.28 1.19
Poetry 7.07 – –
Assessment
Productive – .60 .21
Receptive – 1.81 1.43

Multiple choice 2.53 – –
Multiple choice & true/false 7.24 – –
Questions .45 – –
Recall .53 – –
Think aloud 7.07 – –
Note: A small effect is less than .20, a medium effect is between .21 and .80, and a large effect is greater than .80
(Cohen, 1988). See also Tables 4, 5, and 6.
a
Hedges’ g.
b
n ¼ 11 studies.
c
n ¼ six studies.
beginners who are still developing their vocabulary base. That is, beginning L2 learners
may need to achieve a threshold level of vocabulary knowledge before being able to retain
words presented in glosses (Koda, 2005; Laufer, 1997). Pulido (2003) found that
knowledge of sight vocabulary was significantly correlated with incidental L2 vocabulary
gains from reading and in a recent study, learners with stronger sight vocabularies learned
even more new words (Pulido, 2007).
Computer-mediated glosses had a medium effect on expository texts and a large effect
on narrative texts. Additional empirical studies examining the relationship between text
type, and reading task across a variety of languages could provide a broader research base
from which to investigate this relationship that has attracted attention in L2 reading research
(Brantmeier, 2005; Chu, Swaffar, & Charney, 2002; Horiba, 2000; Pretorius, 2006).
We found a comparatively larger effect for receptive than for productive tests that was
sustained over time (Table 7). This finding is consistent with research showing that L2
learners’ passive (receptive) vocabulary is larger than their active (productive) vocabulary
(Laufer & Paribakht, 1998). Future research on reading and vocabulary assessments using
different tasks that compare learners at different levels of proficiency could provide additional
information for understanding the relationships between depth and breadth of vocabulary
knowledge and incidental learning of lexical items with computer-mediated glosses.
Limitations
The present meta-analysis has several limitations. First, studies in which researchers
observe no statistically significant differences generally are not published (Rosenthal,
212 L.B. Abraham
1979). As such, the inclusion of only published studies and reports with statistically
significant findings might skew the weighted mean effect size patterns reported in meta-
analytic studies. To mitigate the possibility of such publication bias, a thorough search
was conducted for unpublished reports and studies and the present study included
unpublished dissertations (Appendix G and Appendix H). Nevertheless, we may not have
retrieved all unpublished studies of L2 computer-mediated glosses.4 Second, results of this
meta-analysis are not necessarily applicable to primary and secondary educational
contexts since all studies were conducted in university/college settings. Third, this review
was limited to experimental and quasi-experimental studies involving control groups
without access to computer-mediated text glosses and experimental groups with access to
such support. Qualitative studies, mix methods research, studies in which learners act as
their own controls (within-subject designs), and research comparing learners’ reading in
authentic L2 multimedia learning contexts (i.e., those studies that do not have a control
group without access to glosses) all make important and necessary contributions to
understanding conditions for the effective design and use of lexical glosses and other
computer-mediated aides in L2 reading and vocabulary learning.
Conclusions
Implications for future research
This meta-analysis has provided evidence of an overall beneficial role for computer-
mediated text glosses providing lexical support on comprehending authentic L2 readings
and learning vocabulary. We have also examined the influence of level of instruction, text
type, and type of assessment on the effects of glosses. Among the tentative conclusions
which await further empirical studies is the relatively smaller effect of lexical glosses for
reading comprehension and incidental vocabulary learning for beginning learners than for
intermediate and advanced learners.
Continually evolving software and Web-based technologies always present new
opportunities and challenges for research and development of effective CALL reading
tasks. Future studies should continue to investigate proficiency (level of instruction),
genre, and type of assessment as well as other variables moderating comprehension and
vocabulary learning identified in empirical studies and proposed in theoretical models of
L2 reading and multimedia learning (Bernhardt, 2005; Chun & Plass, 1997; Koda, 2005).
Such studies could include the measurement of L2 proficiency, L1 and L2 reading abilities,
the effects of reading task (Hulstijn & Laufer, 2001; Peters, 2006, 2007), the effects of
different types of multimedia glosses in relation to individual differences (Chun & Plass,
1997; Plass & Jones, 2005; Roby, 1999) and consider the role of affective variables such as
interest, motivation, and reading purpose. Replications and longitudinal research would
also strengthen the empirical basis for the design of pedagogically-sound reading tasks not
only with computer-mediated lexical glosses but also with multimedia and emerging
technologies.
Notes
1. Incidental refers to the learning of new vocabulary as a by-product of reading. Thus, incidental
vocabulary learning is not part of a task whose objective is to purposely learn new words
(Hulstijn, 2001, p. 271). For an extensive review of L2 vocabulary learning in CALL, see Van de
Poel & Swanepoel (2003).
2. Studies included in this meta-analysis compared the effect sizes of experimental groups with
access to text glosses (L1 or L2) to control groups (without access to glosses).
3. We selected ‘level of instruction’ since the variable ‘proficiency’ either was measured with
different assessments across studies or studies only reported ‘level of instruction’ (Appendices A
and B).
4. We calculated Orwin’s (1983) fail-safe N in order to investigate the possibility of publication
bias. Orwin’s fail-safe N (1983) is an estimate of the number of unpublished studies
reporting (statistically) non-significant results that would need to be included in order to
reduce the cumulative effect across studies to the point of having ‘no significance’. In order
to reduce the medium and large effect sizes for all studies presented in Tables 4, 5, and 6 to
a small effect size of .10, Orwin’s (1983) fail-safe N indicates that we would need to find
the following number of additional unpublished studies for each dependent variable:
reading comprehension ¼ 57, immediate vocabulary tests ¼ 73, and delayed vocabulary
tests ¼ 66.
Notes on contributor
Lee B. Abraham is Assistant Professor of Spanish in the Department of Modern Languages and
Literatures at Villanova University where he teaches linguistics and instructional technology in
second language acquisition. His research focuses on socio-cognitive approaches to second language
writing, reading, and vocabulary learning with multimedia and the World Wide Web and discourse
analysis.
References
One asterisk indicates that the study was included in the 82 studies coded in phase one and two
asterisks refer to a study that was one of the 11 studies in the meta-analysis in phase two.
**Abraham, L.B. (2007). Second language reading comprehension and vocabulary learning with
multimedia. Hispania, 90, 98–109.
*Aguilar, J.V. (1996). How can multimedia be designed to assist comprehension of the literary text
for foreign language learners? Dissertation Abstracts International, 57(04), 1527. (UMI No.
9627874).
*Al-Seghayer, K. (2001). The effect of multimedia annotation modes on L2 vocabulary acquisition:
A comparative study. Language Learning & Technology, 5(1), 202–232. Retrieved February 13,
2001, from http://llt.msu.edu/vol5num1/alseghayer/.
Ariew, R. (2006). A template to generate hypertext and hypermedia reading materials: Its design and
associated research findings. The Reading Matrix, 6(3), 195–209. Retrieved January 17, 2007,
from http://www.readingmatrix.com/articles/ariew/article.pdf.
*Ariew, R., & Ercetin, G. (2004). Exploring the potential of hypermedia annotations for second
language reading. Computer Assisted Language Learning, 17, 237–259.
*Aust, R., Kelley, M.J., & Roby, W. (1993). The use of hyper-reference and conventional
dictionaries. Educational Technology Research and Development, 41, 63–73.
*Aweiss, S. (1994). Situating learning in technology: The case of computer-mediated reading
supports. Journal of Educational Technology Systems, 23, 63–74.
*Bell, F.L. (2005). Comprehension aids, Internet technologies, and the reading of authentic materials
by adult second language learners. Dissertation Abstracts International, 66(07), 2517. (UMI No.
3183039).
*Bell, F., & LeBlanc, L. (2000). The language of glosses in L2 reading on the computer: Learners’
preferences. Hispania, 83, 274–285.
Bengeleil, N., & Paribakht, T.S. (2004). L2 reading proficiency and lexical inferencing by university
EFL learners. Canadian Modern Language Review, 61, 225–249.
**Ben Salem, E. (2007). The influence of electronic glosses on word retention and reading
comprehension with Spanish language learners. Dissertation Abstracts International, 67(10),
3785. (UMI No. 3239548).
Bernhardt, E.B. (2005). Progress and procrastination in second language reading. Annual Review of
Applied Linguistics, 25, 133–150.
*Bishop, H. (2004). The effect of typographic salience on the look up and comprehension of
unknown formulaic sequences. In N. Schmitt (Ed.), Formulaic sequences: Acquisition, processing
and use (pp. 227–248). Amsterdam: John Benjamins.
214 L.B. Abraham
*Blake, R.J. (1992). Second-language reading on the computer. ADFL Bulletin, 24(1), 17–22.
*Bouvet, E., & Close, E. (2006). Online reading strategy guidance in a foreign language: Five case
studies in French. Australian Review of Applied Linguistics, 29(1), 07.1–07.19.
**Bowles, M.A. (2004). L2 glossing: To CALL or not to CALL. Hispania, 87, 541–552.
Brandl, K. (2002). Integrating Internet-based reading materials into the foreign language curriculum:
From teacher- to student-centered approaches. Language Learning & Technology, 6(3), 87–107.
Retrieved November 23, 2002, from http://llt.msu.edu/vol6num3/brandl/default.html.
Brantmeier, C. (2005). Effects of reader’s knowledge, text type, and test type on L1 and L2 reading
comprehension in Spanish. Modern Language Journal, 89, 37–53.
*Christensen, E., Merrill, P., & Yanchar, S. (2007). Second language vocabulary acquisition using a
diglot reader or a computer-based drill and practice program. Computer Assisted Language
Learning, 20, 67–77.
Chu, H.J., Swaffar, J., & Charney, D.H. (2002). Cultural representations of rhetorical conventions:
The effects of reading recall. TESOL. Quarterly, 36, 511–541.
Chun, D. (2006). CALL technologies for L2 reading. In L. Ducate & N. Arnold (Eds.), Calling on
CALL: From theory and research to new directions in foreign language teaching (pp. 69–98).
San Marcos, TX: CALICO.
*Chun, D. (2001). L2 reading on the Web: Strategies for accessing information in hypermedia.
Computer Assisted Language Learning, 14, 367–403.
*Chun, D., & Payne, J.S. (2004). What makes learners click: Working memory and look-up
behavior. System, 32, 481–503.
*Chun, D., & Plass, J.L. (1996a). Effects of multimedia annotations on vocabulary acquisition.
Modern Language Journal, 80, 183–198.
*Chun, D.M., & Plass, J.L. (1996b). Facilitating reading comprehension with multimedia. System,
24, 503–519.
Chun, D.M., & Plass, J.L. (1997). Research on text comprehension in multimedia environ-
ments. Language Learning & Technology, 1(1), 60–81. Retrieved August 26, 1997, from
http://llt.msu.edu/vol1num1/chun_plass/default.html.
Cobb, T., & Stevens, V. (1996). A principled consideration of computers and reading in a second
language. In M.C. Pennington (Ed.), The power of CALL (pp. 115–217). Houston: Athelstan.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological
Measurement, 20(1), 37–46.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ:
Lawrence Erlbaum.
Comprehensive Meta-Analysis. (2007). (Version 2.2) [Computer software]. Englewood, NJ: Biostat.
*Cooledge, S.L. (2004). L2 reading and hypertext: A study of lexical glosses and comprehension
among intermediate learners of French. Dissertation Abstracts International, 65(12), 4546.
(UMI No. 3158079).
Cooper, H. & Hes, L.V. (Eds.). (1994). The handbook of research synthesis. New York: Russell Sage
Foundation.
**Coriano Velázquez, A. (2001). Vocabulary acquisition through reading: A study of the
effectiveness of different CALL based annotations. Master’s. Abstracts International, 40(03),
561. (UMI No. 1406590).
*Davis, J.N., & Lyman-Hager, M. (1997). Computers and L2 reading: Student performance, student
attitudes. Foreign Language Annals, 30, 58–72.
*De Ridder, I. (2000). Are we conditioned to follow links: Highlights in CALL materials and their
impact on the reading process. Computer Assisted Language Learning, 13, 183–195.
*De Ridder, I. (2002). Visible or invisible links: Does the highlighting of hyperlinks affect incidental
vocabulary learning, text comprehension, and the reading process? Language Learning &
Technology, 6(1), 123–146. Retrieved March 18, 2002, from http://llt.msu.edu/vol6num1/
DERIDDER/default.html.
**De Ridder, I. (2003). Reading from the screen in a second language: Empirical studies on the effect of
marked hyperlinks on incidental vocabulary learning, text comprehension, and the reading process.
Antwerp: Garant.
Dunlap, W.P., Cortina, J.M., Vaslow, J.B., & Burke, M.J. (1996). Meta-analysis of experiments with
matched groups or repeated measures designs. Psychological Methods, 1, 170–177.
*Ercetin, G. (2003). Exploring ESL learners’ use of hypermedia reading glosses. CALICO Journal,
20, 261–283.
Felix, U. (2005a). Analyzing recent CALL effectiveness research: Towards a common agenda.
Computer Assisted Language Learning, 18, 1–32.
Felix, U. (2005b). What do meta-analyses tell us about CALL effectiveness? ReCALL, 17, 269–
288.
Fischer, R. (2007). How do we know what learners are actually doing? Monitoring learners’ behavior
in CALL. Computer Assisted Language Learning, 20, 409–442.
*Frankenberg-Garcia, A. (2005). A peek into what today’s language learners as researchers actually
do. International Journal of Lexicography, 18, 335–355.
Frantzen, D. (2003). Factors affecting how second language Spanish learners derive meaning from
context. Modern Language Journal, 87, 68–199.
Ganderton, R. (1999). Interactivity in L2 web-based reading. In R. Debski & M. Levy (Eds.),
WorldCALL: Global perspectives on computer-assisted language learning (pp. 49–66). Lisse: Swets
& Zeitlinger.
*Gettys, S., Imhof, L.A., & Kautz, J.O. (2001). Computer-assisted reading: The effect of
glossing format on comprehension and vocabulary retention. Foreign Language Annals, 34,
91–106.
Gilmore, A. (2007). Authentic materials and authenticity in foreign language learning. Language
Teaching, 40, 97–118.

*Goyette, E.S. (1996). The effects of dictionary usage on text comprehension. Dissertation Abstracts
International, 57(10), 4264. (UMI No. GNN12375).
Grabe, W., & Stoller, F.L. (2002). Teaching and researching reading. New York: Longman.
*Grace, C. (1998a). Retention of word meanings inferred from context and sentence-level
translations: Implications for the design of beginning-level CALL software. Modern Language
Journal, 82, 533–544.
*Grace, C. (1998b). Personality type, tolerance of ambiguity, and vocabulary retention in CALL.
CALICO Journal, 15, 19–45.
*Grace, C. (2000). Gender differences: Vocabulary retention and access to translations for beginning
language learners in CALL. Modern Language Journal, 84, 214–224.
**Hayden, S.B. (1996). An investigation into the effect and patterns of usage of a computer-
mediated text in reading comprehension in French (hypertext, glossing). Dissertation Abstracts
International, 58(07), 2629. (UMI No. 9802647).
Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic
Press.
*Hegelheimer, V.H. (1998). Effects of textual glosses and sentence-level audio glosses on on-line
reading comprehension and vocabulary recall. Dissertation Abstracts International, 59(11), 4053.
(UMI No. 9912259).
*Hill, M., & Laufer, B. (2003). Type of task, time-on-task and electronic dictionaries in incidental
vocabulary acquisition. International Review of Applied Linguistics in Language Teaching, 41, 87–
106.
*Hoffman, S.Q. (1998). The effect of hypertext environment on the reading comprehension of second
language learners in an intermediate-level college German course. Dissertation Abstracts
International, 59(05), 1547. (UMI No. 9834255).
*Hong, W. (1997). Multimedia computer-assisted reading in business Chinese. Foreign Language
Annals, 30, 335–344.
Horiba, Y. (2000). Reader control in reading: Effects of language competence, text type, and task.
Discourse Processes, 29(3), 223–267.
*Horst, M., Cobb, T., & Nicolae, I. (2005). Expanding academic vocabulary with an interactive on-
line database. Language Learning and Technology, 9, 90–110. Retrieved September 16, 2005,
from http://llt.msu.edu/vol9num2/horst/default.html.
Hubbard, P. (2005). A review of subject characteristics in CALL research. Computer Assisted
Language Learning, 18, 351–368.
*Hulstijn, J.H. (1993). When do foreign-language readers look up the meaning of unfamiliar words?
The influence of task and learner variables. Modern Language Journal, 77, 139–147.
Hulstijn, J.H. (2001). Intentional and incidental second language vocabulary learning: A reappraisal
of elaboration, rehearsal and automaticity. In P. Robinson (Ed.), Cognition and second language
instruction (pp. 258–286). Cambridge: Cambridge University Press.
Hulstijn, J.H., & Laufer, B. (2001). Some empirical evidence for the involvement load hypothesis in
vocabulary acquisition. Language Learning, 51, 539–558.
216 L.B. Abraham
*Hulstijn, J.H., & Trompetter, P. (1998). Incidental learning of second language vocabulary in
computer-assisted reading and writing tasks. In D. Albrechtsen, B. Hendriksen, I.M. Meese &
E. Poulsen (Eds.), Perspectives on foreign and second language pedagogy: Essays presented to
Kirsten Haastrup on the occasion of her sixtieth birthday (pp. 191–200). Odense: Odense
University Press.
*Johnson, A., & Heffernan, N. (2006). The short readings project: A CALL reading activity utilizing
vocabulary recycling. Computer Assisted Language Learning, 19, 63–77.
Just, M.A., & Carpenter, P.A. (1992). A capacity theory of comprehension: Individual differences in
working memory. Psychological Review, 99, 122–149.
*Karp, A.S. (2002). Modification of glosses and its effect on incidental L2 vocabulary learning in
Spanish. Dissertation Abstracts International, 63(05), 1813. (UMI No. 3051537).
Kern, R. (2003). Literacy as a new organizing principle for foreign language education. In
P.C. Patrikis (Ed.), Reading between the lines: Perspectives on foreign language literacy (pp. 40–
59). New Haven, CT: Yale University Press.
Kintsch, W. (1998). Comprehension: A paradigm for cognition. New York: Cambridge University
Press.
**Knight, S. (1994). Dictionary use while reading: The effects on comprehension and vocabu-
lary acquisition for learners of different verbal abilities. Modern Language Journal, 78, 285–
299.
*Kobayashi, C. (2006). The use of pocket electronic dictionaries as compared with printed
dictionaries by Japanese learners of English. Dissertation Abstracts International, 67(07), 2489.
(UMI No. 3226483).
Koda, K. (2005). Insights into second language reading: A cross-linguistic approach. New York:
Cambridge University Press.
Kondo-Brown, K. (2006). How do English L1 learners of advanced Japanese infer unknown kanji
words in authentic texts?. Language Learning, 56, 109–153.
*Koren, S. (1999). Vocabulary instruction through hypertext: Are there advantages over
conventional methods of teaching?. TESL-EJ, 4(1), 1–18. Retrieved May 1, 2007, from
http://tesl-ej.org/ej13/a2.html.
*Koyama, T., & Takeuchi, O. (2007). Does look-up frequency help reading comprehension of
EFL learners? Two empirical studies of electronic dictionaries. CALICO Journal, 25, 110–
125.
Laufer, B. (1997). The lexical plight in second language reading: Words you don’t know, words you think
you know, and words you can’t guess. In J. Coady & T. Huckin (Eds.), Second language vocabulary
acquisition: A rationale for pedagogy (pp. 20–34). New York: Cambridge University Press.
*Laufer, B. (2000). Electronic dictionaries and incidental vocabulary acquisition: Does technology
make a difference? In U. Heid, S. Evert, E. Lehmann & C. Rohrer (Eds.), Proceedings of the 9th
EURALEX International Congress on Lexicography (pp. 849–854). Stuttgart University.
*Laufer, B., & Hill, M. (2000). What lexical information do L2 learners select in a CALL dictionary
and how does it affect word retention? Language Learning & Technology, 3(2), 58–76. Retrieved
January 26, 2000, from http://llt.msu.edu/vol3num2/lauferhill/index.html.
Laufer, B., & Paribakht, T.S. (1998). The relationship between passive and active vocabularies:
Effects of language learning context. Language Learning, 48, 365–391.
Laufer, B., & Yano, Y. (2001). Understanding unfamiliar words in a text: Do L2 learners understand
how much they don’t understand? Reading in a Foreign Language, 13, 549–566.
*Leffa, V. (1992). Making foreign language texts comprehensible for beginners: An experiment with
an electronic glossary. System, 20(1), 63–73.
LeLoup, J.W., & Ponterio, R. (2005). Vocabulary support for independent online reading. Language
Learning and Technology, 9(2), 3–7. Retrieved May 13, 2005, from http://llt.msu.edu/vol9num2/
net/default.html.
*Levine, A., Bejarano, Y., Carrell, P., & Vered, L. (2004). Comparing dictionary definitions and
glosses in hypertext foreign language reading: Facilitating foreign language reading comprehen-
sion and vocabulary acquisition. CATESOL Journal, 16(1), 59–68.
*Lim, K-M., & Shen, H.Z. (2006). Integration of computers into an EFL reading classroom.
ReCALL, 18, 212–229.
*Liou, H.C. (2000). The electronic bilingual dictionary as a reading aid to EFL learners: Research
findings and implications. Computer Assisted Language Learning, 13, 467–476.
Lipsey, M., & Wilson, D. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage Publications.
*Little, L.J. (2001). Multimedia-assisted reading in Spanish and its relationship with the cognitive
control of field dependence and field independence. Dissertation Abstracts International, 62(05),
1707. (UMI No. 3014948).
*Liu, M. (1995). Contextual enrichment through hypermedia technology: Implications for second-
language learning. Computers in Human Behavior, 11, 439–450.
Liu, M., Moore, Z., Graham, L., & Lee, S. (2003). A look at the research on computer-based
technology use in second language learning: A review of the literature from 1990–2000. Journal
of Research on Technology in Education, 34(3), 250–273.
*Liu, M., & Reed, W.M. (1995). The effect of hypermedia-assisted-instruction on second language
learning. Journal of Educational Computing Research, 12(2), 159–175.
**Lomicka, L. (1998). To gloss or not to gloss: An investigation of reading comprehension
online. Language Learning & Technology, 1(2), 41–50. Retrieved August 10, 1998, from
http://llt.msu.edu/vol1num2/article2/default.html.
*Lopez Chiappone, L. (2003). A comparative study of English language learners reading storybooks
in traditional print and digital formats. Dissertation Abstracts International, 65(01), 121.
(UMI No. 3119893).
*Loucky, J.P. (2005). Combining the benefits of electronic and online dictionaries with CALL Web
sites to produce effective and enjoyable vocabulary and language learning lessons. Computer
Assisted Language Learning, 18, 389–416.
Lyman-Hager, M. (2000). Bridging the language-literature gap: Introducing literature electronically
to the undergraduate language student. CALICO Journal, 17, 431–452.
Lyman-Hager, M., & Davis, J.F. (1996). The case for computer-mediated reading: Une vie de boy.
The French Review, 69, 775–790.
Martı́nez-Lage, A. (1997). Hypermedia technology for teaching reading. In M. Bush & R. Terry
(Eds.), Technology enhanced language learning (pp. 121–163). Lincolnwood, IL: National
Textbook Company.
*McDonell, T.B. (2006). Reading plain text and hypertext on the Internet for native and non-native
speakers of English. Dissertation Abstracts International, 67(05), 1624. (UMI No. 3215490).
Murray, D.E., & McPherson, P. (2006). Scaffolding instruction for reading the Web. Language
Teaching Research, 10, 131–156.
*Nagata, N. (1999). The effectiveness of computer-assisted interactive glosses. Foreign Language
Annals, 32, 469–479.
Nassaji, H. (2003). L2 vocabulary learning from context: Strategies, knowledge sources and their
relationship with success in L2 lexical inferencing. TESOL Quarterly, 37, 645–670.
Nassaji, H. (2004). The relationship between depth of vocabulary knowledge and L2 learners’
lexical inferencing strategy use and success. Canadian Modern Language Review, 61, 107–
134.
Nation, I.S.P. (2001). Learning vocabulary in another language. Cambridge: Cambridge University
Press.
*Nesi, H. (2000). On screen or in print? Learners’ use of a learner’s dictionary on CD-ROM and in
book form. In P. Howarth & R. Hetherington (Eds.), EAP learning technologies (pp. 106–114).
Leeds, UK: Leeds University Press.
*Nikolova, O. (2002). Effects of learners’ participation in authoring of multimedia materials on
student acquisition of vocabulary. Language Learning & Technology, 6(1), 100–122. Retrieved
April 13, 2002, from http://llt.msu.edu/vol6num1/NIKOLOVA/default.html.
*Nikolova, O. (2004). Effects of visible and invisible hyperlinks on vocabulary acquisition and
reading comprehension for high-and average-foreign language achievers. Apprentissage des
langues et syste`mes d’information et de communication, 7, 29–53. Retrieved August 1, 2004, from
http://alsic.u-strasbg.fr/v07/nikolova/alsic_v07_05-rec1v2.htm.
*Nikolova, O., & Taylor, G. (2003). The impact of a language learning task on instructional
outcomes in two student populations: High-ability and average-ability learners. Journal of
Secondary Gifted Education, 14(4), 205–217.
Norris, J.M., & Ortega, L. (Eds.). (2006). Synthesizing research on language learning and teaching.
Amsterdam: John Benjamins.
Orwin, R.G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics,
8(2), 157–159.
Orwin, R.G. (1994). Evaluating coding discrepancies. In H. Cooper & L.V. Hedges (Eds.), The
handbook of research synthesis (pp. 139–162). New York: Russell Sage Foundation.
218 L.B. Abraham
*Overstreet, C. (2006). Reading authentic text in the hypermedia environment: The effects of
question glosses on comprehension processes of intermediate learners of German as a foreign
language. Dissertation Abstracts International, 67(06), 2038. (UMI No. 3224599).
Paribakht, T.S., & Wesche, M. (1997). Vocabulary enhancement activities and reading for meaning
in second language vocabulary. In J. Coady & T. Huckin (Eds.), Second language vocabulary
acquisition (pp. 174–200). New York: Cambridge University Press.
*Peters, E. (2006). L2 vocabulary acquisition and reading comprehension: The influence of task
complexity. In M.P. Garcia-Mayo (Ed.), Investigating tasks in formal language learning (pp. 178–
198). Clevedon: Multilingual Matters.
*Peters, E. (2007). Manipulating L2 learners’ online dictionary use and its effect on L2 word
retention. Language Learning & Technology, 11(2), 36–58. Retrieved June 7, 2007, from
http://llt.msu.edu/vol11num2/peters/default.html.
Pichette, F. (2005). Time spent on reading and reading comprehension in second language learning.
Canadian Modern Language Review, 62, 243–262.
Plass, J.L., & Jones, L.C. (2005). Multimedia learning in second language acquisition. In R.E. Mayer
(Ed.), The Cambridge handbook of multimedia learning (pp. 467–488). New York: Cambridge
University Press.
*Plass, J.L., Chun, D.M., Mayer, R.E., & Leutner, D. (1998). Supporting visual and verbal learning
preferences in a second-language multimedia learning environment. Journal of Educational
Psychology, 90, 25–36.
**Plass, J.L., Chun, D.M., Mayer, R.E., & Leutner, D. (2003). Cognitive load in reading a foreign
language text with multimedia aids and the influence of verbal and spatial abilities. Computers in
Human Behavior, 19, 221–243.
Pretorius, E.J. (2006). The comprehension of logical relations in expository texts by learners who
study through the medium of ESL. System, 34, 432–450.
*Proctor, C.P., Dalton, B., & Grisham, D. (2007). Scaffolding English language learners and
struggling readers in a multimedia hypertext environment with embedded strategy instruction
and vocabulary support. Journal of Literacy Research, 39(1), 71–93.
*Pujolà, J.-T. (2002). CALLing for help: Researching language learning strategies using help
facilities in a web-based multimedia program. ReCALL, 14, 235–262.
Pulido, D. (2003). Modeling the role of second language proficiency and topic familiarity in second
language incidental vocabulary learning. Language Learning, 53, 233–284.
Pulido, D. (2007). The effects of topic familiarity and passage sight vocabulary on L2 lexical
inferencing and retention through reading. Applied Linguistics, 28, 66–86.
Qian, D.D. (2004). Second language lexical inferencing: Preferences, perceptions and practices. In
B. Laufer & P. Bogaards (Eds.), Vocabulary in a second language: Selection, acquisition and
testing (pp. 155–169). Amsterdam: John Benjamins.
*Roby, W.B. (1991). Glosses and dictionaries in paper and computer formats as adjunct aids to the
reading of Spanish texts by university learners. Dissertation Abstracts International, 53(09), 3182.
(UMI No. 9238698).
Roby, W.B. (1999). What’s in a gloss? Language Learning & Technology, 2, 94–101. Retrieved
February 26, 1999, from http://llt.msu.edu/vol2num2/roby/index.html.
Rosenthal, R. (1979). The ‘file drawer problem’ and tolerance for null results. Psychological Bulletin,
86, 638–641.
Rott, S. (1999). The effect of exposure frequency on intermediate language learners’ incidental
vocabulary acquisition and retention through reading. Studies in Second Language Acquisition,
21, 589–619.
*Sakar, A., & Ercetin, G. (2004). Effectiveness of hypermedia annotations for foreign language
reading. Journal of Computer Assisted Learning, 21, 28–38.
*Sercu, L., De Wachter, L., Peters, E., Kuiken, F., & Vedder, I. (2006). The effect of task complexity
and task conditions on foreign language development and performance: Three empirical studies.
ITL, International Journal of Applied Linguistics, 152, 55–84.
*Son, J-B. (2003). A hypertext approach to foreign language reading: Student attitudes and
perceptions. Australian Review of Applied Linguistics, 17, 91–110.
*Stakhnevich, J. (2002). Reading on the Web: Implications for ESL professionals. The Reading
Matrix, 2(2). Retrieved June 1, 2003, from http://www.readingmatrix.com/archives/
archives_vol2_no2.html.
**Stoehr, L.E. (1999). The effects of built-in comprehension aids in a CALL program on student-
readers’ understanding of a foreign language literary text. Dissertation Abstracts International,
60(12), 4353. (UMI No. 9956934).
Taylor, A. (2006). The effects of CALL versus traditional L1 glosses on L2 reading comprehension.
CALICO Journal, 23, 309–318.
*Taylor, G., & Nikolova, O. (2004). Influence of gender and academic ability in a computer-based
Spanish reading task. Roeper Review: A Journal on Gifted Education, 27(1), 42–51.
*Terhune, D.R., & Moore, B. (1991). Computer vs. paper: A preliminary study on vocabulary
expansion. CAELL Journal, 2(3), 30–34.
Van de Poel, K., & Swanepoel, P. (2003). Theoretical and methodological pluralism in designing
effective lexical support for CALL. Computer Assisted Language Learning, 16, 173–211.
Webb, S. (2007). The effects of repetition on vocabulary knowledge. Applied Linguistics, 28, 46–65.
Yano, Y., Long, M.H., & Ross, S. (1994). The effects of simplified and elaborated texts on foreign
language reading comprehension. Language Learning, 44, 189–219.
*Yeh, Y., & Wang, C. (2003). Effects of multimedia vocabulary annotations and learning styles on
vocabulary learning. CALICO Journal, 21, 131–144.
*Yoshii, M., & Flaitz, J. (2002). Second language incidental vocabulary retention: The effect of text
and picture annotation types. CALICO Journal, 20, 33–58.

*Yoshii, M. (2006). L1 and L2 glosses: Their effects on incidental vocabulary learning. Language
Learning & Technology, 10(3), 85–101. Retrieved September 8, 2006, from http://llt.msu.edu/
vol10num3/yoshii/default.html.
Young, D. (1999). Linguistic simplification of SL reading material: Effective instructional practice?
Modern Language Journal, 83, 350–366.
**Youngs, B.L.E. (1994). CALL and second language acquisition: The use of the ‘Reader’ computer
program to improve student proficiency in French. Dissertation Abstracts International, 56(03),
899. (UMI No. 9521148).
Zhao, Y. (2003). Recent developments in technology and language learning: A literature review and
meta-analysis. CALICO Journal, 21, 7–27.
220 L.B. Abraham
Appendix A. Definitions of selected study characteristics.
Definition
Control group Participants did not have access to glosses.
Experimental group Participants had access to glosses.
(comparison/
treatment)
Experimental design Random assignment of participants to either a control group or
an experimental group.
Quasi-experimental Participants not randomly assigned to either a control group or
design an experimental group.
Level of instruction Year of instruction of the course (class).
Gloss (annotation) Aide designed to facilitate reading comprehension and/or
incidental vocabulary learning.
Productive vocabulary Learners wrote answers in either the L1 or L2.
test
Receptive vocabulary Participants selected answers in either the L1 or L2.
test
Timing of vocabulary An immediate post-test was given on the same day the study was
post-test carried out. A delayed post-test was conducted either the day
after the immediate post-test or any time up to four weeks later.
Time of immediate to The number of weeks between the immediate and delayed
delayed vocabulary vocabulary post-tests.
post-test
Multiple choice test Participants selected the correct answer.
Questions Learners answered questions about a text.
Recall protocol Learners wrote or typed a summary of a text.
Think-aloud Participants vocalized whatever came to mind as they read a
computer-mediated text, including the types of glosses used,
their rationale for using these glosses, as well as their
understanding of a text.
Expository text Serves to inform, explain, argue, or persuade. Common types of
expository text include description, comparison/contrast, cause/
effect, and problem/solution or a combination of these types.
Narrative text Consists of ‘characters (animate beings exhibiting clearly specified
goals and motives for their achievement), setting (the particular
frame and spatial location where the events are unfolded),
complications (problems or conflicts encountered by main
characters), plots (sequences of events), affect patterns
(elements soliciting emotional intrigue), and values (morals
emphasized in text)’. (Koda, 2005, p. 155)
Appendix B. Coding book.
Publication characteristics
1) Study identification Assign each study a number beginning with 0001
2) Author Report last name(s), first name(s)
3) Year of study Report year; Use letters a, b, c, for same year but assign different
study identification number
4) Source 1 ¼ journal article, 2 ¼ dissertation or Master’s thesis; 3 ¼ book
chapter; 4 ¼ book; 5 ¼ refereed conference paper
Learner characteristics
5) Native language 0 ¼ not reported; 1 ¼ English; 2 ¼ French; 3 ¼ German;
4 ¼ Spanish, 5 ¼ Japanese, 6 ¼ Chinese, 7 ¼ Arabic;
8 ¼ Hebrew
6) L2 of study 1 ¼ English as a Second or Foreign Language, 2 ¼ French,
3 ¼ German, 4 ¼ Spanish, 5 ¼ Japanese, 6 ¼ Chinese,
7 ¼ Arabic
7) Level of instruction 0 ¼ not reported; 1 ¼ first year: all participants were in their
first year of instruction; 2 ¼ second year: all participants

were in their second year of instruction; 3 ¼ more than two
years: all participants were in their third year or more of
instruction; 4 ¼ mixed: two or more: report levels of
instruction
8) L1 Reading ability Report measure of L1 reading ability
9) L2 Reading ability Report measure of L2 reading ability
10) L2 Proficiency Report measure of L2 proficiency
Design characteristics
11) Size of control group Report size of control group
12) Size of treatment group 1 Report size of treatment group for no control group studies
13) Size of treatment group 2 Report size of treatment group for no control group studies
14) Control group mean Report control group mean on immediate vocabulary post-tests;
immediate vocabulary if not reported, type ‘nr’ and see #26 for additional statistics.
post-test
15) Control group standard Report control group on standard deviation immediate
deviation immediate vocabulary post-tests; if not reported, type ‘nr’ and see #26 to
vocabulary post-test record additional statistics.
16) Treatment group means Report treatment group means on all immediate vocabulary post-
immediate vocabulary tests; if not reported, type ‘nr’ and see #26 to record additional
post-tests statistics.
17) Treatment group standard Report treatment group standard deviation(s) on immediate
deviations immediate vocabulary post-tests; if not reported, type ‘nr’ and see #26 to
vocabulary post-tests record additional statistics.
18) Control group mean Report control group mean on delayed vocabulary post-tests; if
delayed vocabulary post- not reported, type ‘nr’ and see #26 to record additional
tests statistics.
19) Control group standard Report control group standard deviation on delayed vocabulary
deviation delayed post-tests; If not reported, type ‘nr’ and see #26 to record
vocabulary post-tests additional statistics.
20) Treatment group means Report treatment group means on delayed vocabulary post-tests;
Delayed vocabulary post- If not reported, type ‘nr’ and see #26 to record additional
tests statistics.
21) Treatment group standard Report treatment group standard deviations on delayed
deviations delayed vocabulary post-tests; if not reported, type ‘nr’ and see #26 to
vocabulary post-tests record additional statistics.
22) Control group mean Report control group mean on reading comprehension measures;
reading comprehension if not reported, type ‘nr’ and see #26 to record additional
statistics.
(continued)
222 L.B. Abraham
Appendix B. (Continued).
23) Control group standard Report control group standard deviation on reading
deviation reading comprehension; if not reported, type ‘nr’ and see #26 to record
comprehension additional statistics.
24) Treatment group means Report treatment group means on reading comprehension
reading comprehension measures; if not reported, type ‘nr’ and see #26 to record
additional statistics.
25) Treatment group standard Report treatment group standard deviations on reading
deviations reading comprehension measures; if not reported, type ‘nr’ and see #26
comprehension below to record additional statistics.
26) Other statistics Report d-value (Cohen’s d) ; p-value; t-value; F-ratio; MSbetween
27) Setting 0 ¼ not reported; 1 ¼ college/university; 2 ¼ high school;
3 ¼ middle school; 4 ¼ elementary school; 5 ¼ other
28) Dependent variable 1 ¼ reading comprehension; 2 ¼ vocabulary; 3 ¼ reading
comprehension and vocabulary
29) Research design I, phase 0 ¼ not reported; 1 ¼ vocabulary pre-test & control;
one studies 2 ¼ vocabulary pre-test & comparison group only;

3 ¼ vocabulary pre-test, no control group, no comparison
group; 4 ¼ no vocabulary pre-test & control; 5 ¼ no
vocabulary pre-test & comparison group only; 6 ¼ no
vocabulary pre-test, no control group, no comparison
group
30) Research design II, phase 1 ¼ true (randomized) experiment: participants were randomly
two studies assigned to either a treatment or control group; 2 ¼ quasi-
experimental: participants were not randomly assigned to a
treatment nor control group; 3 ¼ non-experimental
31) Experimental design 1 ¼ pre-test/post-test with control; 2 ¼ post-test-only with
control
32) Quasi-experimental design 1 ¼ pre-test/post-test with control; 2 ¼ post-test-only with
control; 3 ¼ within-subject
33) Type of reading assessment 1 ¼ written recall; 2 ¼ multiple choice; 3 ¼ questions; 4 ¼ think-
aloud; 5 ¼ true/false; 6 ¼ translation
34) Type of vocabulary 1 ¼ productive: learners wrote (typed) a correct
assessment answer; 2 ¼ receptive: learners selected a correct
answer; 3 ¼ productive & receptive
35) Vocabulary pre-test 0 ¼ not reported; 1 ¼ presence of distractors; 2 ¼ absence of
sensitization distractors
36) Timing of vocabulary post- 1 ¼ immediate; 2 ¼ delayed 1 (first post-test); 3 ¼ delayed 2
test (second post-test)
37) Timing from immediate to 1 ¼ one week after immediate post-test; 2 ¼ two weeks after
first delayed post-test immediate post-test; 3 ¼ three weeks after immediate post-test;
4 ¼ more than three weeks after immediate post-test
38) Type of text 1 ¼ expository; 2 ¼ narrative; 3 ¼ poem
39) Gloss format 1 ¼ paper (print) ; 2 ¼ computer-mediated
40) Gloss authorship (Roby 0 ¼ not reported; 1 ¼ researchers; 2 ¼ instructors; 3 ¼ material
1999) developers; 4 ¼ participants
41) Gloss function (Roby, 0 ¼ not reported; 1 ¼ metacognitive; 2 ¼ linguistic
1999)
42) Gloss language 1 ¼ L1; 2 ¼ L2; 3 ¼ L3; report gloss language
43) Gloss medium 1 ¼ verbal; 2 ¼ visual; 3 ¼ audio
44) Visual gloss 1 ¼ image or photograph; 2 ¼ video with sound; 3 ¼ video
without sound; 4 ¼ animation; 5 ¼ illustration/drawing
45) Number of words in text 0 ¼ not reported; 1 ¼ 0–500; 2 ¼ 501–1000; 3 ¼ 1001–1500;
4 ¼ 1501–2000; 5 ¼ more than 2000
46) Number of words glossed 0 ¼ not reported; 1 ¼ 0–10; 2 ¼ 11–20; 3 ¼ 21–30; 4 ¼ 31–40;
5 ¼ 41–50; 6 ¼ 51–60; 7 ¼ more than 60
(continued)
Appendix B. (Continued).
47) Percent of text glossed 0 ¼ not available; 1 ¼ 1–5%; 2 ¼ 6 –10%; 3 ¼ 11–20%; 4 ¼ 21–
30%; 5 ¼ 31–40%; 6 ¼ 41–50%; 7 ¼ more than 50%
48) Readability formula 0 ¼ not reported; 1 ¼ Fry graph; 2 ¼ Flesch-Kincaid
49) Reader/reading purpose 0 ¼ not reported; 1 ¼ reading to find information; 2 ¼ reading
for general comprehension; 3 ¼ reading to critique; 4 ¼ other:
report other purpose
50) Independent variables 1 ¼ paper/print glosses versus computer-mediated glosses;
2 ¼ computer-mediated multimedia glosses versus computer-
mediated text glosses; 3 ¼ computer-mediated glosses with
visible links versus computer-mediated glosses with invisible
links; 4 ¼ proficiency; 5 ¼ visual/verbal ability; 6 ¼ working
memory; 7 ¼ multiple choice versus single glosses; 8 ¼ reading
task; 9 ¼ gender; 10 ¼ learning styles
Appendix C. Study characteristics, phase one.
Number of studies
Reading Vocabulary Reading comprehension
comprehension learning and vocabulary learning
Vocabulary pre-test
Control group and experimental group – 0 8
Experimental groups only – 12 5
No control group and – 2 0
no experimental group
No vocabulary pre-test
Control group and experimental group 7 0 0
Experimental groups only 28 8 9
No control group and 3 0 0
no experimental group
Note: n ¼ 82 studies. Control group ¼ no access to glosses. Experimental group ¼ access to glosses.
Appendix D. Study characteristics, phase two.
Number of studies
Reading Reading comprehension
Design comprehension and vocabulary learning
Experimental
Pre-test/post-test with control and experimental groups 0 5
Post-test-only with control and experimental groups 3 1
Quasi-experimental
Pre-test/post-test with control and experimental groups 0 1
Post-test-only with control and experimental groups 1 0
Note: n ¼ 11 studies. Experimental studies randomly assigned each participant to either a control or
experimental group. Quasi-experimental studies did not include random assignment.
Control group ¼ no access to glosses. Experimental group ¼ access to glosses.
224 L.B. Abraham
Appendix E. Exclusion criteria.

1. Empirical studies of multimedia software or Web sites that included full-length video as an
advance organizer or presented video clips which were not glosses were excluded in order to
avoid confounding the effect of listening and reading comprehension on measures of incidental
L2 vocabulary learning and reading comprehension.
2. Research on computer-mediated glosses and the use of concordances was excluded to avoid
confounding the effects of both tools on outcome measures.
3. Studies focusing on the use of computer-mediated glosses or electronic dictionaries to translate
sentences were not included.
4. Empirical studies which did not include a control group (no access to computer-mediated
glosses) when comparing paper-based and computer-mediated glosses were excluded.
5. Research reporting on the design of multimedia software or Web sites such as exercises and
templates for reading comprehension but not conducting empirical research on learners’ look-up
behavior or performance on measures of L2 reading comprehension or incidental vocabulary
learning were not included.
6. Studies including a control group (no gloss access) but not having the statistics needed for
calculating an effect size were excluded.
Appendix F. Coding reliability of categories: Cohen’s Kappa (k) and Agreement Rate (AR).
Study categorya k AR (%)

Study design .92 90
Type of reading comprehension test 1 100
Type of vocabulary post-test 1 100
Type of text .87 83
Level of instruction .85 81
Gloss form 1 100
Gloss language 1 100
Note: n ¼ 82 studies. Cohen’s Kappa ( k ) has a possible range of 0 to 1 with values closer to 1 indicating better
intercoder reliability of the categories (Cohen, 1960). The Agreement Rate or AR (also referred to as percent
agreement) has a possible range of 0 to 100% with values closer to 100% indicating better intercoder reliability
(Orwin, 1994).
a
Appendix A and Appendix B define each of these categories.
Appendix G. Selected study characteristics and effect sizes for L2 reading comprehension.
Publication Level of Type of Weighted

N type L1 L2 Independent variable instruction text Type of assessment effect sizea
Abraham 2007 68 A Enb Sp No glosses vs. L1, L2, Intermediate Narr Recall .94
picture, and video glossesc
Ben Salem 2007 37 D En Sp No glosses vs. L1 glosses Intermediate Narr Multiple choice 3.42
Bowles 2004 32 A En Sp No glosses vs. L1 glosses Beginning Exp Multiple choice 1.69
Coriano 34 T Sp En No glosses vs. L1 glosses Beginning Narr Multiple choice & 7.24
Velázquez 2001 true/false
DeRidder 2003 56 C Du Fr No glosses vs. L1 & L2 glosses Advanced Exp Questions .45
Hayden 1996 36 D En Fr No glosses vs. L1 & L2 glossesd Intermediate Narr Recall .54
Multiple choice .19
Knight 1994 105 A En Sp No glosses vs. L1 & L2 glosses Intermediate Exp Recall .68
Lomicka 1998 8 A En Fr No glosses vs. L1 & L2 glosses Beginning Poem Think-aloud 7.07
Plass, Chun, 76 A En Ge No glosses vs. L1 & L2 glosses Intermediate Narr Recall 7.37
Mayer, and
Leutner 2003
Stoehr 1999 62 D En Ge No glosses vs. L1 glosses Intermediate Narr Recall 1.49
Youngs 1994 28 D En Fr No glosses vs. L1 glossese Intermediate Exp Recall 7.14
Note: N ¼ total number of participants in the control (no gloss) and experimental (gloss) groups. For publication type, A ¼ journal article (refereed), C ¼ book chapter,
D ¼ dissertation, T ¼ Master’s thesis. L1 ¼ participants’ native language; L2 ¼ language of study. Du ¼ Dutch, En ¼ English, Fr ¼ French, Ge ¼ German,
Sp ¼ Spanish. For type of text, Exp ¼ expository, Narr ¼ narrative.
a
Hedges’ g. A weighted effect size takes into account differences in sample sizes between studies (Lipsey & Wilson, 2001, pp. 48–50).
b
One native speaker of each of the following languages was reported: Arabic, French, Mandarin (Chinese), and Zuni.
c
The experimental group consulted glosses with the following frequencies: English ¼ 81%, Spanish ¼ 8%, picture ¼ 7%, and video ¼ 4% (p. 104).
d
Included pronunciation, cultural and grammatical explanations, and relationships between pronouns and their referents.
e
Included cultural and grammatical explanations, pre-reading exercises, and comprehension questions.
225
226
Appendix H. Selected study characteristics and effect sizes for L2 vocabulary learning.
Publication Level of Type Type of Weighted

N type L1 L2 Independent variable instruction of text assessment Duration effect sizea
Abraham 2007 68 A Enb Sp No glosses vs. Intermediate Narr Productive Immediate .83
L1, L2, picture, &
video glossesc
L.B. Abraham
Ben Salem 2007 37 (23)d D En Sp No glosses vs. Intermediate Narr Productive Immediate 1.53
L1 glosses Delayed 1.03
Receptivee Immediate 2.90
Delayed 2.41
Bowles 2004 32 A En Sp No glosses vs. Beginning Exp Productive Immediate –f
L1 glosses Delayed –f
Delayed .95
Coriano 34 T Sp En No glosses vs. Beginning Narr Productive Immediate .27
Velázquez 2001 L1 glosses Delayed .21
DeRidder 2003 56 C Du Fr No glosses vs. Advanced Exp Receptive Immediate 1.54
L1 & L2 glosses Delayed 2.06
Knight 1994 105 A En Sp No glosses vs. Intermediate Exp Productive Immediate 1.47
L1 & L2 glosses Delayed .49
Delayed 1.04
Plass, Chun, 76 A En Ge No glosses vs. Intermediate Narr Receptive Delayed 1.16
Mayer, and L1 & L2 glosses
Leutner 2003
Note: N ¼ total number of participants in the control (no gloss) and experimental (gloss) groups. For publication type, A ¼ journal article (refereed), C ¼ book chapter,
D ¼ dissertation, T ¼ Master’s thesis. L1 ¼ participants’ native language; L2 ¼ language of study. Du ¼ Dutch, En ¼ English, Fr ¼ French, Ge ¼ German,
Sp ¼ Spanish. For type of text, Exp ¼ expository, Narr ¼ narrative.
a
Hedges’ g. A weighted effect size takes into account differences in sample sizes between studies (Lipsey & Wilson, 2001, pp. 48–50).
b
One native speaker of each of the following languages was reported: Arabic, French, Mandarin (Chinese), and Zuni.
c
The experimental group consulted glosses with the following frequencies: English ¼ 81%, Spanish ¼ 8%, picture ¼ 7%, and video ¼ 4% (p. 104).
d
23 learners participated in the delayed vocabulary assessment tasks.
e
Results on immediate and delayed receptive vocabulary tests were used to calculate the weighted effect sizes.
f
Data were not available to calculate a weighted effect size from the repeated-measures design (Dunlap, Cortina, Vaslow, & Burke, 1996).

Abraham 2008

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Abraham 2008

Uploaded by

Copyright:

Available Formats

This article was downloaded by: [Tulane University]

On: 30 August 2014, At: 20:32

Computer Assisted Language Learning

Computer-mediated glosses in second

To link to this article: http://dx.doi.org/10.1080/09588220802090246

PLEASE SCROLL DOWN FOR ARTICLE

Computer-mediated glosses in second language reading comprehension and

Department of Modern Languages and Literatures, Villanova University, Villanova, USA

Language learners have unprecedented opportunities for developing second language

and in multimedia computer-assisted language learning environments. This article

ISSN 0958-8221 print/ISSN 1744-3210 online

(1) level of instruction

The purpose of this meta-analysis is to provide a systematic and replicable statistical

L2 vocabulary knowledge, reading comprehension and the role of glosses

(1) learners’ L2 proﬁciency (Bengeleil & Paribahkt, 2004);

Although signiﬁcant knowledge of vocabulary is essential for L2 text comprehension,

Synthesizing CALL research

Meta-analysis and CALL

(a) small eﬀect (d is less than .20);

Meta-analysis and other approaches to synthesizing studies have become increasingly

(1) What is the eﬀect of computer-mediated glosses on L2 reading comprehension and

The present study was conducted in the following stages:

Search and retrieval of studies

‘electronic dictionary’, ‘gloss’, ‘glosses’, ‘glossary’, ‘hypertext’, ‘hypermedia’, ‘lexicon’,

Study eligibility: inclusion and exclusion criteria

variables examined in computer-mediated L2 reading with glosses. Published and

As illustrated in Appendix C, 62 of the 82 studies examined the eﬀects of computer-

(1) paper/print glosses versus computer-mediated glosses;

Coding of studying categories and intercoder reliability

Eﬀect size extraction and calculation

(1) small eﬀect (less than .20);

Table 1. Individual eﬀect sizes for L2 reading comprehension.

Abraham 2007 34 34 2.20 4.30 2.06 2.33 .94

Mayer & Leutner 2003

Table 2. Individual eﬀect sizes for immediate L2 vocabulary post-tests.

Abraham 2007 34 34 6.90 9.40 2.77 3.27 .83

Table 3. Individual eﬀect sizes for delayed L2 vocabulary post-tests.

Ben Salem 2007 11 12 5.18 9.44 .98 2.16 2.41

Table 4. Weighted mean eﬀect sizes for L2 reading comprehension tests.

Weighted eﬀect size 95% conﬁdence interval

Questions 1 .45 .27 7.07 .97

Table 5. Weighted mean eﬀect sizes for immediate L2 vocabulary post-tests.

Weighted eﬀect size 95% conﬁdence interval

Table 6. Weighted mean eﬀect sizes for delayed L2 vocabulary post-tests.

Weighted eﬀect size 95% conﬁdence interval

Overall eﬀect of glosses

Variables moderating the eﬀect of glosses

(1) level of instruction;

Text type (genre)

homogeneity of variance test (Q-test) for reading comprehension assessment was

(a) level of instruction;

However, the following observations should be interpreted as suggestive rather than

Table 7. Summary of weighted eﬀect sizes.

Weighted Eﬀect Sizea

Receptive – 1.81 1.43

Teaching, 40, 97–118.

and picture annotation types. CALICO Journal, 20, 33–58.

Appendix A. Deﬁnitions of selected study characteristics.

Appendix B. Coding book.

ﬁrst year of instruction; 2 ¼ second year: all participants

one studies 2 ¼ vocabulary pre-test & comparison group only;

Appendix C. Study characteristics, phase one.

Appendix D. Study characteristics, phase two.

Appendix E. Exclusion criteria.