WHAT TO MENTION

 Content validity
 Construct validity
 Face validity
 Test reliability
 Scorer reliability
 Practicality
 Backwash

ADVANTAGES AND DISADAVANTAGES OF TEST TYPES

Test type Advantages and Disadvantages

multiple-choice or alternative-response tests of reading or Advantages:
listening comprehension
 Easy and objective to mark
 Offers excellent scorer reliability
 Learners see it as valid
 Can test a wide range of language and skills e.g. Can be
used for collocations, gerunds and infinitives, lexis,
tenses...

Disadvantages:

 Quite dull
 Does not assess productive ability
 Random guessing can distort results

gap-fill tests Advantages:

 Quite objective to mark

 Learners see it as valid
 Can test productive ability indirectly
 Can be adapted easily to make it harder or easier (e.g., by
giving multiple-choice alternatives)
 Can be used to target a range of language and lexis

Disadvantages:

 Can require judgement if there are multiple possibilities

 Does not directly assess productive ability
 Random guessing can distort results

re-ordering tasks: paragraphs, sentences, phrases, words Advantages:

 Adaptable: can be used with a range of text types

 Learners see it as valid
 Can test receptive ability indirectly
 Can be used to target syntax or discourse

Disadvantages:

 Multiple correct ordering is sometimes possible so

marking is more difficult
 Time consuming to prepare and do
 Does not target productive ability
sentence transformation / skeleton sentences Advantages:

 Quite objective to mark

 Learners see it as valid
 Targets particular structures directly
 Can be used to target a range of language and lexis

Disadvantages:

 Can require judgement if there are multiple possibilities

 Lack of lexical knowledge can interfere with the ability to
do the grammar task

information transfer (e.g., diagram to written text, diagram to Advantages:

spoken text)
 Directly assesses productive ability
 Learners see it as valid
 Can be adapted to make it harder or easier
 Can be used to target a range of language and specific
lexis (e.g., describing trends, making comparisons)

Disadvantages:

 Time consuming to prepare

 Advantages students who access graphical information
easily
 Subjective marking

guided writing or speaking (structured response tests) Advantages:

 Can be more objectively marked than free writing /

speaking
 Learners see it as valid
 Can test productive ability directly
 Can be adapted to make it harder or easier
 Can be used to target a range of communicative language
functions and lexis

Disadvantages:

 Can advantage students familiar with the topic

 Cannot be purely objectively marked
 Requires reading / listening comprehension to produce
accurate responses in speaking or writing

one-to-one or paired interviews (role plays etc.) Advantages:

 A direct test of communicative ability

 Can test productive ability directly
 Can be adapted as it progresses to gather more
information
 Can be designed to target a range of language functions
and lexis

Disadvantages:
  Subjective marking, even with a set of criteria
 Learners may not see them as fair or valid
 Requires both receptive and productive skills
 Stressful for many learners
 Paired learners can influence each other's ability
 Requires careful standardisation of assessors

Matching tasks Disadvantages

 If learners get one wrong they may get more wrong. It's
possible to guess answers.
 Learners may understand a word but not the definition.

Cloze tests Advantages

 Can test a lot of different language areas simultaneously.

Disadvantages

 There may be lots of possible answers.

 May test IQ rather than language.

Error spotting (editing) Advantages

 Replicates a real life task which is also an important

learning strategy.

Disadvantages

 Students need awareness of the kinds of errors they are

looking for.
Joining sentences Advantages

 Useful for focusing on items such as linkers and linking

words.
Odd one out Disadvantages

 Students may have different criteria for what constitutes

"oddness".
Writing Questions for Answers Advantages

 Good for focusing on questions.

Disadvantages

 There may be more than one possible answer.

SPEAKING TEST EXAMPLE

WRITING TEST examples
Positives
Topic matter is technical (to some degree and presented
pseudo-scientifically
Test an IELTS examination task and that adds face validity
The proposed IELTS course will focus on writing skills and
this is a direct test of the students writing
The rubric is clear and not too complicated
It will is a test of writing skills only
The test contains objective data which will appeal to this
learner and is relevant to the IELTS academic writing test
Negatives
No guidance in terms of staging the response text e.g.
topic>similarities>differences>reasons. In SS L1 structure
Application to learner
They can see the relevance. His background as a Science
student should facilitate his understanding. It enhances
face validity and possibly motivation if he thinks it is
representative of the course he wants to follow
Level of challenge is high but may be manageable and will
show a fair picture of his writing ability. The test has
content and predictive validity
This has a clear content and predictive validity
The test should not confuse him; concerning what he
needs to do (he has strong reading skills).
SS will feel relevant to his expectations of an IELTS course
and therefore has good construct and content validity.
The test has face validity and may be motivating
Application to the learner
Task achievement (one of the IELTS criteria) likely to be
very poor. (the student is at B1 level) So content and
may be different
The task is subjectively marked and will need a careful set
of criteria to produce reliable results. No mention has
been made of any criterion referencing but it could be
assumed that it will be marked according to IELTS criteria
The test requires and number of functions to be realized
in the writing, including, describing trends, contrasting,
comparing and noting similarities. It also requires the
subject to handle past tense structures with confidence
for the writing to be successful. It may, therefore, be
simply too hard for a student at this level (B1) The test
will then lack content and predictive validity and be an
unreliable measure of his abilities.
The test is only of one kind of academic writing, which
may be tested (As well as describing charts, IELTS ask
students to describe processes, put forward arguments
and give opinions)
The test focuses on some of the harder aspects of writing
and is arguably too difficult
SPEAKING TEST EXAMPLE
Positives
Topic matter is familiar to many learners at this level of
maturity and may intrigue them.
This is a direct test of the ability to hold a long turn
The rubric is clear and not too complicated but it is rather
dense
predictive validity may be reduced.
The task has no relevance to physical Science so content
validity may be reduced.
At this stage in his learning he may be unacquainted with
the criteria, so won’t know what he is being assessed on.
This could reduce motivation and predictive and face
validity.
If he recognizes that he doesn’t command the level of
structure and organization in English, he may be
demotivated and perform unreliably below his real
ability.
The test lacks content and predictive validity for IELTS
because it only tests a very narrow range. If the SS is
aware of this, he may feel the test is unfair and lose
motivation to perform well, so reducing the face validity
and reliability of the test
A task asking for some expressions of personal ideas, or
concerning astronomy may be more appropriate in terms
of level and personalized for this learner
Application to learner
The learners are being asked to speak about something
personal rather than abstract and theoretical which may
enhance their performance. Because the three topics are
parallel, they can be marked against the same set of
criteria and this reduces the marking load and may
increase reliability
This is something that participants in seminars and
tutorials may well need to do
It is obvious what skill is being tested and how
Some guidance is given concerning what and in what
order to stage the talk
Negatives
All three topics focus on the same kind of response so a
learner who has little introspective ability may be
disadvantaged
it is not evident that the task is anything like one that the
learner will be required to perform on their studies.
The task is subjectively marked and will need a careful set
of criteria to produce reliable results. No mention is made
of any criterion referencing
The test requires a careful handling of tense structures,
moving from past tense narrative of the event, person or
book to present perfect and present tense forms when
considering current effects.
The test is intimidating for many learners to speak for 5
minutes about personal views with a stranger
The prohibition of the use of any notes is a drawback
because it advantages those with a good language
memory and does not reflect speaking tasks in most
university environments
The test is mostly concerned with one way
communication
The vagueness about what the examiner may ask is
slightly disconcerting
GRAMMAR TESTING EXAMPLE
In a sense the test is a structured response and can be
marked on that criterion (among others)
Application to learner
A wider choice of topic would be a fairer and more valid
test
The test has little content and predictive validity
The learner may not be acquainted with the referencing
criterion so won’t know what is being assessed. This
could reduce motivation and face validity
If the test-takers are not alerted to this, the results may
be unpredictable with only the strongest candidates
conforming to the generic structures.
The learner may feel uncomfortable
For many, making notes is an aid to organizing one’s
thought and the prohibition of that is inexplicable.
Test-takers may become flustered if they lose the thread
or may be intimidated by a perceived need to memorize
what to say in 15mins
However, the final interactive part will also provide some
indication of interactional ability
Disconcerted learners will not probably perform to their
potential
 Positives
Tests language in use rather declarative knowledge in a
grammar exercise, it’s an indirect (testing the ability
which underlies the speaking skill e.g. grammar), thus, an
integrative test of language, valid as both formative or
summative assessment
The instructions are clear for the teacher
The prompts will allow the learner to use some of the
language studied (conditionals and articles)
There are familiar topics/prompts to provide ideas
There are opportunities for fresh starts
The task type/format is familiar and recognisable. It’s an
engaging and unpressurised way to test grammar
The level is appropriate for upper-intermediate level
students
Negatives
The learner can avoid using the language that they have
studied. It doesn’t overtly test the language that they’ve
been taught (e.g. past tense, conditionals, articles)
It does not look like a serious/grammar test so it lacks
face validity
Application to learner
Learner reaction: Students will believe that the test has
face validity, will have faith in the test, will feel
motivated to do the test, will feel confident about doing
the test, will engage with the test
The teacher should be able to direct the learners easily so
they know what to do
The teacher will get useful information about the
learners’ progress and abilities and will be able to assess
the learners’ language accuracy
Learners will have something to say, will be able to relate to the
content, will be interested in the topics and will want to discuss
the topics
Learners will not get stuck if they cannot respond to one
of the topics
Learners should know how to approach the task
Students have a chance to demonstrate abilities and be
able to perform to the best of their abilities
Application to learner
Learner reaction: The learners may be demotivated
because this is not how they expect grammar to
be tested / they may not take it seriously / they may not
engage with the topic / they may lack ideas

It would be difficult for the teacher to monitor and assess
all the individuals, therefore, the test lacks practicality
and reliability
Learner output: Some learners may not say much and
their partners may be dominant
The test is wider in focus that the course (tests
speaking/listening skills and other language systems), it
doesn’t only test grammar, it’s an indirect/integrative
test of grammar and a direct test of speaking skills.
The type of interaction, a discussion, does not lend itself
to testing grammatical accuracy. Learners don’t need to
be accurate in this type of discussion/it does not test
written accuracy
LISTENING EXAMPLE TEST (Missing technical references)
/ they may feel uncomfortable in the group / learners
don’t trust results of the test
Insufficient/inaccurate data: The teacher may not get an
accurate idea of their abilities/progress /
learners won’t get a sense of progress
Learner performance: The learners may not show their
full abilities
Test format: The learners may be hindered by the format
of the test / will not be able to participate
Feedback: The teacher cannot give individual students
feedback
ADVANTAGES

DISADVANTAGES

• Time consuming to write and mark (Practicality)

• Snap shot of a learner not a full picture

• Tests may not reflect what is done in class ( oral work in class / test used a vocabulary written exercise)

• Negative backwash... Teaching to suit what's in the test lots of test practice rather than focusing on the
learners longer term needs....