Materials Evaluation

What is
Materials
Evaluation?
• Materials Evaluation is a
procedure that involves
measuring the value ( or
potential value ) of a set of
learning materials.
• It involves making judgments
about the effect of the
materials on the people using
them and it tries to measure
some or of the following list
(Tomlinson, 2013)
• appeal of the materials to the
learners
• credibility of the material to
learners, teachers, and
administrators
• validity of the materials
• reliability of the materials
• ability of the materials to
interest the learners and the
teachers
• ability of the materials to
motivate the learners
• value of materials in terms of
short-term learning
• value of materials in terms of
long-term learning
• learners’ perceptions of the
value of the materials
• teacher’s perceptions of the
value of the materials
• assistance given to the
teachers in terms of
preparation, delivery, and
assessment
• flexibility of the materials (e.
g. the extent to which it is
easy for a teacher to adapt
the materials to suit a
particular context)
• contribution made by the
materials to teacher
development
• match with the administrative
requirements
No two evaluations
can be the same, as
An evaluation can
the needs,
include an analysis or
objectives,
follow from one, but
backgrounds, and
the objectives and
preferred styles of
procedures are
the participants will
different.
differ from context to
context.
An analysis focuses on
It focuses on the users
the materials, and it
of the materials and
aims to provide an
makes judgments
objective analysis of
about their effects.
them.
What is the
difference
between
evaluation and
analysis?
An evaluation
is your opinion
based on a set
of criteria that
you create
supported by
facts.
Gigi de Lana is the better
artist.

Album sales, performance,

attractiveness

More best-selling
albums, singer, cuter
An analysis
examines the
elements of
something by
separating the
parts for a deeper
understanding.
How do they win
in the 2021
Olympics?
To answer this question:

 You would not just only

consider their record.
 You would also consider
the factors of winning.
 Strengths and weaknesses

Coach skills

Motivation

Determination

Love of country

These factors come together

to show a bigger meaning.
When you are asked to
evaluate that means
give your opinion.

When you are asked to

analyze that means
explain the meaning.
 For Example!

Question: ‘Does it provide a

transcript of the listening texts?’

It is an analysis question
which can be answered by either
‘Yes’ or ‘No’
 For
Example!
Question: ‘What does it ask the learners to do
immediately after reading a text?’

It is an analysis question and can be answered

factually.

As a result of answering many such questions, a

description of the materials can be made which
specifies what the materials do and do not
contain.
 For Example!

Question: ‘Are the listening texts

likely to engage the
learner?’

It is an evaluation question and can

be answered on a cline between
‘Highly unlikely’ and ‘Highly likely’
It can also be given a numerical value (e. g. 2 for
‘Unlikely’) and after many such questions have
been asked about the materials, subtotal scores and
total scores can be calculated, and indications can
be derived of the potential value of the materials
and of subsections of them.
Principles in Materials Evaluation
( Brian Tomlinson, 2013)

Materials should:
• achieve impact
• help learners to feel at ease
• help the learners to develop confidence
• require and facilitate learner self-
investment

What is being taught should be perceived

by learners as relevant and useful.
Principles in Materials Evaluation
( Brian Tomlinson, 2013)

• Learners must be ready to

acquire the points being taught
both in terms of linguistic,
developmental readiness, and an of
psychological readiness too.
• Materials should expose the learners to
language in authentic use.
• The learner’s attention should be drawn to
linguistic features of the input.
Principles in Materials Evaluation
( Brian Tomlinson, 2013)

• Materials should provide the learners

with opportunities to use the target
language to achieve communicative
purposes ( in order to automize
existing procedural knowledge, to
check the effectiveness of their
existing hypotheses)
Principles in Materials Evaluation
( Brian Tomlinson, 2013)

Materials should:

• help the learner to develop cultural

awareness and sensitivity
• reflect the reality of language use
• help the learners to learn in ways similar to the
circumstances in which they will have to use the
language
• help to create readiness to learn
• achieve effective engagement
 Principles in Materials Evaluation
( Richards, 2001)

Materials should:
• give learners something they can take
away from the lesson
• teach something learners feel they can use
• give learners a sense of achievement
• practice learning items in an interesting
and novel way
• provide a pleasurable learning experience
 Principles in Materials Evaluation
( Richards, 2001)

Materials should provide

opportunities for:
• individual practice
• personalization
• self-assessment of learning
 Types of Materials Evaluation

• Evaluations differ in purpose, personnel,

formality, and timing.
• In order to conduct an effective
evaluation, you need to apply your
principles of evaluation to the contextual
circumstances of your evaluation in order
to determine the most reliable and
effective procedures.
 Pre-use Evaluation

• involves making predictions about the

potential value of materials for their users
• can context-free, as in a review of
materials for a journal
• can be context influenced as in a review of
draft materials for a publisher with target
users in mind
• can be context-dependent, as when a
teacher selects a course book for use with
his/her particular class
 Frameworks for generating
evaluation criteria

• Makundan and Ahour (2010) proposed that a

framework for generating flexible criteria would
be more useful than detailed and inflexible
checklists.

• McGrath (2002) suggested a procedure involving

materials analysis followed by first glance
evaluation, user feedback and evaluation using
context-specific checklists.
 Frameworks for generating
evaluation criteria

• Riazi (2003) suggested about surveying the

teaching/learning situation, conducting a
neutral analysis, and the carrying out of a
belief-driven evaluation.

• Rubdy ( 2003) suggested a dynamic model

of evaluation in which the categories of
psychological validity, pedagogical
validity, and process and content validity
interact.
 Frameworks for generating
evaluation criteria
• Mukundan (2006) described the use of a
composite framework combining
checklists, reflective journals, and
computer software to evaluate ELT
textbooks in Malaysia.
• McDonough et al. (2013) focused on
developing criteria evaluating the
suitability of materials in relation to
usability, generalizability, adaptability,
and flexibility.
Whilst-use Evaluation

• involves measuring the value of materials

while using them or while observing them
being used.
• can be more objective and reliable than
pre-use evaluation as it makes use of
measurement rather than prediction.
• limited to measuring what is observable
(e. g. ‘Are the instructions clear to the
learners?’) and cannot claim to measure
what is happening in the learners’ brains.
Whilst-use Evaluation

• can measure short-term memory through

observing learner performance on
exercises
• cannot measure durable and effective
learning because of the delayed effect of
instruction
• very useful but dangerous too, as
teachers and observers can be misled by
whether the activities seem to work or
not
What does it measure?

• Clarity of instructions
• Clarity of layout
• Comprehensibility of texts
• Credibility of tasks
• Achievability of tasks
• Achievement of performance
objectives
• Potential for localization
• practicality of the materials
What does it measure?

• teachability of materials
• Flexibility of the materials
• Appeal of the materials
• Motivating power of the materials
• Impact of the materials
• Effectiveness in facilitating short-
term learning
Post-use Evaluation

• the most valuable (but least administered)

type of evaluation as it can measure the
actual effects of the materials on the
users
• Can measure the short-term effect as
regards motivation, impact, achievability,
instant learning, etc. and it can measure
the long-term effect as regards durable
learning and application
Ways of measuring the post-use
effects of materials include:

• test of what has been taught by the

materials
• test of what the students can do
• examinations
• interviews
• questionnaires
• criterion-referenced evaluations by the
users
Ways of measuring the
post-use effects of materials
include:

• post-course diaries
• post-course ‘shadowing’ of the
learners
• post-course reports on the learners by
employers, subject tutors, etc.
• Materials evaluation is initially a time-consuming
and difficult undertaking.

• Approaching it in the principled, systematic, and

suggested rigorous ways cannot only help to
make and record vital discoveries about the
materials being evaluated but can also help the
evaluators to learn a lo about materials, about
learning and teaching, and about themselves.
• Doing evaluations formally and rigorously can
also eventually contribute to the development of
ability to conduct principled informal evaluations
quickly and effectively when the occasion
demands (e. g. when asked for an opinion of a
new book; when deciding which materials to buy
in a bookshop; when editing other people’s
materials.
• Evaluation is demanding but rewarding.