See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/268805274

Analysing the effectiveness of textbooks for vocabulary retention.

Article · September 2014

CITATIONS READS

5 1,390

2 authors:

Rachael Ruegg Cherie Brown

Victoria University of Wellington AIU
40 PUBLICATIONS   197 CITATIONS    14 PUBLICATIONS   9 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Lexis and grammar in writing assessment View project

EMI in Japan and China View project

All content following this page was uploaded by Rachael Ruegg on 12 February 2015.

The user has requested enhancement of the downloaded file.

 Vocabulary Learning and Instruction
Volume 3, Issue 2, September 2014
http://vli-journal.org
Article

Analyzing the Effectiveness of Textbooks for

Vocabulary Retention
Rachael Ruegg and Cherie Brown
Akita International University
doi: http://dx.doi.org/10.7820/vli.v03.2.ruegg.brown

Abstract
Although many language educators are aware of factors necessary for
vocabulary acquisition and retention, in many institutions around the
world instructors are required to use textbooks as a basis for instruction,
yet, to date, little has been done to analyze the effectiveness of textbooks
to foster vocabulary acquisition and retention. Therefore, the purpose of
the present study was to analyze the vocabulary content of a range of
textbooks. The number of target words, frequency level of those words
and length of reading texts were analyzed in reading texts and their
associated activities of 20 English as a Second Language (ESL)/English as
a Foreign Language (EFL) textbooks. It was found that each text had as
little as no target words and on average 10, while the average text length
was 639 words. In terms of the frequency level of the target vocabulary,
although the textbooks claimed to be at an intermediate level or above,
the frequency level of the target vocabulary was very inconsistent, with as
many as 60% of target words coming from the first 1000 words or as
many as 100% of target words being low frequency ‘offlist’ words.
Furthermore, it was found that integrated skills textbooks had signifi-
cantly fewer target words and significantly shorter reading texts than
reading textbooks, while they also had a significantly higher percentage of
words from the 1000 word list and a significantly lower percentage from
the Academic Word List (AWL).

1 Background
A great number of studies have examined different factors necessary for
vocabulary acquisition and retention, and many language educators are aware of
these factors. For example, it is widely accepted that learners should begin their
vocabulary acquisition by learning the most common words in the target language,
as these provide greater coverage of written text and spoken language than other
words. In the study of English vocabulary, in particular, the General Service Lists
(West, 1953) of the most frequent 2000 words in the English language are the most
commonly referred to for vocabulary instruction and research. After the first 2000
words have been acquired, it is generally agreed that the next focus should be the
570 academic words in the Academic Word List (AWL) for learners who hope to
use the language for academic purposes (Coxhead, 2000).
However, in many institutions around the world, instructors are required by
their institutions, or education ministries, to use prescribed textbooks as a basis for

Please cite this article as: Ruegg, R., & Brown, C. (2014). Analyzing the effectiveness of textbooks
for vocabulary retention. Vocabulary Learning and Instruction, 3 (2), 11
18. doi: 10.7820/vli.v03.2.
ruegg.brown
12 Ruegg and Brown: Analyzing the Effectiveness of Textbooks

instruction. Indeed, many instructors who are not required to use textbooks also
choose to do so. To date, textbooks have not been analyzed in terms of their
effectiveness to foster vocabulary acquisition and retention. Therefore, the purpose
of the present study was to analyze a range of textbooks which are widely available
and commonly used internationally in the instruction of English as a Foreign
Language (EFL), English as a Second Language (ESL) and English for Academic
Purposes (EAP) with a view to discovering the extent to which they foster
vocabulary acquisition and retention opportunities.
The terms ‘beginner’, ‘elementary’, ‘intermediate’ and ‘advanced’ have often
been used to describe the proficiency level of learners, and these terms continue to
be used in many contexts. Similarly, these terms are often used to describe the levels
of textbooks. However, the terms are subjective and little effort has been made to
determine what they might mean in quantifiable terms. Perhaps the closest the
field of language education has to a consensus on language proficiency levels is
the Common European Framework of Reference for Languages (CEFR; Council
of Europe, 2001), which was created in an attempt to more objectively measure
learners’ language proficiency levels. However, many of the descriptors used in
the CEFR are also difficult to quantify and therefore, in practice, the CEFR
‘rests on the poles of teacher perceptions’ (Hulstijn, 2007). More specifically, in
terms of vocabulary knowledge, there is also considerable range in the amount of
vocabulary known by learners based on their context of study. For example, EFL
learners typically know less vocabulary than their ESL counterparts because of
their relative lack of exposure to the target language.
Although the issue of how much vocabulary learners should be expected to
know at a given proficiency level is somewhat opaque, textbook producers still need
to decide what target vocabulary to focus on in their books. As a rough guide, graded
reader series provide an estimate of how much vocabulary learners may know at
various proficiency levels. The vocabulary levels of three series of graded readers
(Cambridge, Macmillan and National Geographic) were used to determine a rough
guide to the level of vocabulary that should be taught to learners at different levels of
proficiency. From the data in Table 1, which were compiled from the information
offered by the three publishers, it can be seen that beginners to pre-intermediate level
learners should be focusing on the first 1000 words, elementary to intermediate level
learners should be focusing on the second 1000 words and upper-intermediate to
advanced level learners should be focusing on vocabulary beyond the first 2000
words. In terms of the CEFR, learners at the A1 and A2 levels should be focusing on
the first 1000 words, from A2 to B2 should be focusing on the second 1000 words and
from B2 upwards should be focusing on words beyond the first 2000 words.
Not only do learners need to focus deliberately on vocabulary during study,
but they also need to encounter target words repeatedly, in order to remember
them. According to Folse (2004), encountering a new word multiple times enables it
to be moved from short-term to long-term memory and thus retained. Similarly,
Cobb (1999) found that when learners encounter words in a variety of different
contexts, they are better able to retain them. Consequently, not only do the target
words need to be included as many times as possible in a text, but previously learned
words also need to be recycled regularly (Nation, 2001).

Vocabulary Learning and Instruction, 3 (2), 11
18.

 Ruegg and Brown: Analyzing the Effectiveness of Textbooks 13

Table 1. Graded-reader Vocabulary Levels

Publisher Level Words CEFR

Cambridge Beginner/elementary 400 A1
Cambridge Elementary/lower-intermediate 800 A2
Cambridge Lower-intermediate 1300 B1
Cambridge Intermediate 1900 B1
Cambridge Upper-intermediate 2800 B2
Cambridge Advanced 3800 C1
Macmillan Starter 300
Macmillan Beginner 600 A1
Macmillan Elementary 1100
Macmillan Pre-intermediate 1400 A2
B1
Macmillan Intermediate 1600 B1
B2
Macmillan Upper 2200
National Geographic Pre-intermediate 800
1000 A2
National Geographic Intermediate 1300
1600 B1
National Geographic Upper-intermediate 1900
2200 B2
National Geographic Advanced 2600
3000 C1

2 Research Method
This research analyzed the target vocabulary content of 20 textbooks which
are widely available and commonly used internationally in the instruction of EFL,
ESL and EAP. For the purposes of comparison, 10 reading textbooks and 10
integrated skills textbooks were selected and the fifth reading text of each book was
analyzed. The textbooks used in the analyzis are listed in Table 2.
In order to ascertain the target vocabulary, one reading text and all
vocabulary activities related to the text (pre-reading, post-reading, review and
expansion activities) were examined. A word which appeared in the reading text and
at least one activity was considered to be a target vocabulary item. The length of
texts was compared, in order to determine the amount of exposure to text that
learners gain from the textbook because text exposure gives learners the chance to
encounter new words multiple times and also recycles previously learnt words. In
the analyzis of the frequency level of the target vocabulary, the lexical profiling tool
‘Web Vocabprofile’ (Cobb, n.d.; Heatley, Nation, & Coxhead, 2002) was used. The
tool calculates both the number of words at each of four frequency levels (1000
word level, 2000 word level, AWL and offlist) and the text coverage in the form of
percentages.

3 Results

3.1 Preliminary Analyzis

The descriptive statistics for the length of texts (in words) and the number of
target words can be seen in Table 3.

Vocabulary Learning and Instruction, 3 (2), 11
18.

14 Ruegg and Brown: Analyzing the Effectiveness of Textbooks

Table 2. Textbooks analyzed

Title Author(s) Publisher Year Level Type

Aspire Dummett, P., Robb Heinle 2013 Upper- Integrated
Benne, R., Cengage intermediate skills
Crossley, R. Learning B2
College Reading Avery, J., Fellag, L. Thomson 2006 Upper Reading
Heinle
Concepts and Ackhert, P. Heinle 2005 Upper- Reading
comments Lee, L. Cengage intermediate
Learning
Focus on Schmitt, D. Pearson 2011 Upper- Reading
vocabulary 2 Schmitt, N. Longman intermediate
Global Outlook Bushell, B. McGraw Hill 2013 Intermediate Reading
Dyer, B.
Just Right Harmer, J. Heinle 2012 Upper- Integrated
Lethaby, C. Cengage intermediate skills
Learning
Key Concepts 2 Smith-Palinkas, B. Heinle 2010 Intermediate Reading
Croghan-Ford, K. Cengage
Learning
Mosaic 1 Wegmann, B. McGraw Hill 2014 Intermediate
Reading
Knezevic, M. high-
intermediate
New English File Oxenden, C. Oxford 2008 Upper- Integrated
Latham-Koenig, C. intermediate skills
B2
New Headway Soars, L. Oxford 2005 Upper- Integrated
Soars, J. intermediate skills
B2
Outcomes Dellar, H. Heinle 2010 Upper- Integrated
Walkley, A. Cengage intermediate skills
Learning B2
Passages 1 Richards, J. Cambridge 2008 Upper- Integrated
Sandy, C. intermediate skills
Q: Skills for Caplan, N. Oxford 2011 Reading
Success 5 Roy Douglas, S.
Quest 2 Hartmann, P. McGraw Hill 2007 Intermediate
Reading
high-
intermediate
Reading MacIntyre, P. Heinle 2010 Upper- Reading
Explorer 4 Cengage intermediate
Learning C1
Select Readings Lee, L. Oxford 2011 Upper- Reading
Bernard, J. intermediate
B2
World Class 2 Douglas, N. Cengage 2013 Advanced Integrated
Morgan, J. Learning C1 skills
World English 3 Johannsen, K. Heinle 2010 Intermediate Integrated
Tarver Chase, R Cengage B1 skills
Learning
World Link 3 Stempleski, S., Heinle 2011 Intermediate Integrated
Morgan, J., Cengage B2 skills
Douglas, N. Learning

Vocabulary Learning and Instruction, 3 (2), 11
18.

 Ruegg and Brown: Analyzing the Effectiveness of Textbooks 15

Table 2 (Continued )
Title Author(s) Publisher Year Level Type
World Pass Stempleski, S., Thomson 2006 Upper- Integrated
Morgan, J., intermediate skills
Douglas, N.,
Johannsen, K.

The descriptive statistics for the percentage of target words at each frequency
level can be seen in Table 4.
While analyzing the text length, number and frequency level of target words,
there appeared to be clear differences between the integrated skills textbooks and
reading textbooks. For this reason, it was decided to conduct an additional
analyzis, comparing the text length, number and frequency level of target
vocabulary between the two types of textbooks.

Table 3. Descriptive Statistics for Text

Length and Target Words

Variable Mean SD Min Max

Text length 639 313 243 1221
Target words 10 6 0 25

Table 4. Descriptive Statistics for Frequency Levels

of Target Words

Frequency Mean (%) SD Min Max (%)

1000 words 17.17 19.64 0 60.00
2000 words 13.17 16.42 0 66.67
AWL 30.17 31.14 0 100
Offlist 34.50 27.10 0 100

3.2 Additional Analyzis

Table 5 shows the descriptive statistics for the text length (in words) and the
number of target vocabulary items in both types of textbooks.
Independent samples t-tests were conducted to determine whether there were
any significant differences between the two types of textbooks in terms of text
length or number of target words. It was found that there was a significant
difference (at the 0.05 level) between reading textbooks and integrated skills
textbooks both in terms of the length of texts (in words), (t  8.86, df  18, p 
.001) and the number or target words, (t  2.57, df  18, p  .0019).
Table 6 shows the descriptive statistics for the frequency level of the target
words in both types of textbooks.

Vocabulary Learning and Instruction, 3 (2), 11
18.

16 Ruegg and Brown: Analyzing the Effectiveness of Textbooks

Table 5. Descriptive Statistics for Text Length and Target

Words by Text Type

Variable Text type Mean SD Min Max

Text length Reading 860.90 289.89 432 1221
Int. skills 417.00 116.10 243 616
Target words Reading 12.90 6.59 6 25
Int. skills 6.90 3.35 0 12

Table 6. Descriptive Statistics for Frequency Levels of Target

Words by Text Type

Frequency Text type Mean SD Min Max

1000 words Reading 7.82 10.61 0 27.27
Integrated skills 26.51 22.52 0 60
2000 words Reading 14.65 20.80 0 66.67
Integrated skills 11.68 11.48 0 33.33
AWL Reading 48.22 34.17 0 100
Integrated skills 12.12 12.51 0 40
Offlist Reading 29.30 21.09 0 71.43
Integrated skills 39.69 32.34 0 100

T-tests were conducted, using a Bonferroni adjustment, to compare the

proportions of words from each frequency level (1000, 2000, AWL and Off-list)
between the integrated skills textbooks and reading textbooks. It was found that
there was a significant difference between the number of AWL words in the two
types of books (t  3.137, df  18, p B 0.0125), with significantly more AWL
words appearing in the reading textbooks than the integrated skills textbooks. The
proportions of words at the other three frequency levels (1000, 2000 and Off-list)
were not found to be significantly different between the two types of textbooks.

4 Discussion and Conclusion

Due to the lack of any clear, quantifiable proficiency levels or any guidelines
regarding what vocabulary learners at different proficiency levels may be ready to
learn, textbook producers are left with the difficult task of trying to determine what
level of vocabulary learners at different proficiency levels should be focusing on.
However, as shown in Table 1, looking at graded readers may provide a rough guide
as to what levels are likely to be appropriate. It is clear that learners beyond the pre-
intermediate or B1 level no longer need to focus on the first 1000 words as an object
of direct study, yet 7 out of 10 integrated skills textbooks and 2 out of 10 reading
textbooks focused on words from the first 1000 as target words in their textbooks.
Furthermore, two of the integrated skills textbooks (‘Just right’ and ‘Aspire’), which
claimed to be at the upper-intermediate or B2 level, included 50% and 60% of target
words from the first 1000 words, respectively. It seems that if learners need to focus

Vocabulary Learning and Instruction, 3 (2), 11
18.

 Ruegg and Brown: Analyzing the Effectiveness of Textbooks 17

on these words at the highest frequency level, then they cannot be termed upper-
intermediate level learners and, therefore, it is highly likely that the vocabulary level
of these books is pedagogically inappropriate.
Although the claimed proficiency levels of the reading textbooks and the
integrated skills textbooks were almost the same, with two of each type of textbook
claiming to be at an intermediate level and the rest claiming to be at higher levels
(only one book claimed to be at the advanced level and one book did not state any
proficiency level), 6 of the 10 integrated skills textbooks focused predominantly on
high-frequency levels (50% or more of the target vocabulary from within the first
2000 words). On the other hand, not one of the reading textbooks focused
predominantly on high-frequency words (50% or more of the target words from
within the first 2000 words). It seems clear that reading textbooks are expecting
users to know significantly more vocabulary, although they use the same terms to
describe the level of the textbook.
It may appear that reading textbooks and integrated skills textbooks have
different purposes, and this argument may be used to justify the significantly higher
proportion of academic vocabulary in the reading textbooks. However, textbooks
may not always be used for the purposes for which they are intended. General ESL
schools in English-speaking countries tend to use integrated skills textbooks rather
than skill-specific texts. However, they have many learners who aim to be admitted
into a formal tertiary education institution. Furthermore, even in pre-sessional
EAP programs within universities in English-speaking countries, integrated skills
textbooks are often used as the basis of instruction. Although it may be cheap and
practical for learners to buy just one textbook and use it to learn all four skills, it
seems clear in terms of vocabulary learning that institutions that choose integrated
skills textbooks rather than a dedicated reading textbook may be doing a disservice
to their students, as reading textbooks provide more recycling of vocabulary
through longer texts, more focus on vocabulary and target vocabulary at more
suitable frequency levels than integrated skills books.
What was stated by Hulstijn (2007) about the CEFR could also be said of
the terms ‘beginner’, ‘elementary’, ‘intermediate’ and ‘advanced’. We already have
a number of tests which can be given to individual learners to determine
their vocabulary knowledge. However, this information has not been linked to
proficiency levels. At this point, it seems necessary for vocabulary researchers to
find a method of quantifying what learners are expected to know at each level
so that textbook producers can clearly communicate the intended audience for
their textbooks, enabling teachers and learners to select the material that is most
appropriate to their needs.

References
Cobb, T. (1999). Breadth and depth of vocabulary acquisition with hands-on
concordancing. Computer Assisted Language Learning, 12, 345
360.
Cobb, T. (n.d.). Web Vocabprofile. Retrieved from http://www.lextutor.ca/vp/, an
adaptation of Heatley, Nation, & Coxhead’s. (2002). Range.

Vocabulary Learning and Instruction, 3 (2), 11
18.

 18 Ruegg and Brown: Analyzing the Effectiveness of Textbooks

Council of Europe. (2001). Common European framework of reference for languages:

Learning, teaching, assessment. Cambridge: Cambridge University Press.
Coxhead, A. (2000). A new academic word list. TESOL Quarterly, 34, 213
238.
doi:10.2307/3587951
Folse, K. (2004). Vocabulary myths: Applying second language research to classroom
teaching. Ann Arbor, MI: The University of Michigan Press.
Heatley, A., Nation, I. S. P., & Coxhead, A. (2002). RANGE and FREQUENCY
programs. Retrieved from http://www.victoria.ac.nz/lals/staff/paul-nation
Hulstijn, J. (2007). The shaky ground beneath the CEFR: Quantitative and
qualitative dimensions of language proficiency. Modern Language Journal,
91, 663
667. Retrieved from http://www.jstor.org/stable/4626094
Nation, I. S. P. (2001). Learning vocabulary in another language. Cambridge, UK:
Cambridge University Press.
West, M. (1953). A general service list of English words. London, UK: Longman,
Green & Co.

Vocabulary Learning and Instruction, 3 (2), 11
18.

View publication stats