Comparing Quality:: Nick Saville

Comparing Quality:
what teachers need to ask exam providers
Nick Saville
ALTE, Vilnius
November 2007
Vilnius November 2007 Nick Saville
Introduction
Outline of talk
Testing for teachers - questions to ask

Stakeholders as customers
The ALTE Quality “toolkit”
The CEFR and the Manual for Guidance
Conclusions

Testing for Language Teachers
A.Hughes, 2003 (2nd edition), CUP
‘… many language teachers harbour a mistrust of tests and testers.’
‘Too often language tests have a harmful effect on teaching and

learning and fail to measure accurately – whatever it is they are
intended to measure.’ (page 1)
He also explains why teachers need to know more about language

testing – both in order to write their own tests … and also to
evaluate external examinations (the theme of this talk)

In addressing language teachers in particular
Hughes suggests that:
Teachers need to be able to ask appropriate

questions of test providers
They also need to understand the main concepts of

assessment
Otherwise they can’t understand the answers!

Cyril Weir also argues that more people in
education need to understand about testing
He suggests that users of tests – students and

their teachers - are like customers …..
I’ll come back to this idea later

‘When we are buying a new car or a new house we have a whole
list of questions we want to ask of the person selling it. Any failure
to answer a question or an incomplete answer will leave doubts in
our minds about buying. Poor performance in relation to one of
these questions puts doubts in our minds about buying the house
or car.’
C.Weir

So –
What sort of questions should we ask about language

tests?
How can we tell the difference between a good test

and a bad one?

A few suggestions:
e.g. about a test at CEFR B1 level (Threshold Level)
Is this B1 test really a B1 test?

How do you know?
Does it test what you say it does?

Has that claim been validated?
What evidence, if any, is provided to support it?
Are the judgements made by the oral examiners

accurate and reliable?
Are they appropriately trained and monitored?

What about the answers?
How do you know if the answers are adequate?
Or whether they show that the test meets

acceptable standards?

Key points for this talk:
Stakeholders for tests should be treated as clients or
customers (users of products or services)
The concept of quality is based on customer satisfaction
As customers the stakeholders need to:
Know what to look for (for their own purposes)

Be able to ask the right questions
Understand the answers
Be able to find “tools” to help them make decisions

I’d like one of those
digital cameras
It depends…
How much are
they? What features
do you want?
Client Service provider
Transaction
Panasonic Lumix DMC-FX07 - Digital Camera, Silver
Nico chose this one

because of the
following features:
Price (around 200

euros)
The mega pixels
The brand of the
lens
The battery life
Nico’s camera The size (fits in a
pocket)
Other Lumix models …….
Panasonic DMC- FX100EB-K Panasonic DMC- FZ50EB-K

Jon Simon chose this one

Digital cameras: Which? Magazine
The magazine of the UK consumers’ association
Specification: Rating (5-point scale)

price sharpness of picture
mega pixels picture quality (overcast)
lens picture quality (sunny)
battery life overall picture quality
size/weight

Which? Magazine on tests
For example, Eye tests at the optician

Criteria for analysis
Eye tests
accuracy of
prescription/diagnosis
thoroughness
speed of testing
qualifications of test
providers
interpretability of results
cost

Criterial features for language tests:
purpose
construct definition
test method – tasks etc.
content coverage
skills coverage
accuracy of measurement
predictive/diagnostic power
score interpretability
test length
accessibility
cost

Additional criterial features:
degree of specificity – for work, for study ……
currency and recognition – where and how the test is used
relationship to curriculum – washback
impact in the wider world
Other???

How can the work of ALTE help?
What “tools” are available to help

stakeholders make the right choices?

www.alte.org
ALTE Quality

Aims of ALTE – since 1990
To establish common levels of proficiency in order to
promote the transnational recognition of certification in
Europe
To establish common standards for all stages of the
language-testing process
To collaborate on joint projects and in the exchange of
ideas and know-how
Motto: “Attaining standards: sustaining diversity”

Aims of ALTE – since 1990
To establish common levels of proficiency in order to
promote the transnational recognition of certification in
Europe
To establish common standards for all stages of the
To collaborate on joint projects and in the exchange of
ideas and know-how
Motto: “Attaining standards: sustaining diversity”

Quality issues: the ALTE toolkit
a) ALTE Code of Practice
b) Principles of Good Practice
c) Quality Management System - QMS
QMS checklists
d) Minimum standards – 17
Auditing the quality profile
All available on ALTE website in many languages

QMS checklists

a) The ALTE Code of Practice
general/ ethical ALTE Code of
framework Practice, 1994
philosophical
ALTE COP & QMS

principles ALTE Principles of
Good Practice,
1993; 2001
quality ALTE COP & QM
management checklists, 2001
system
specific/
ALTE auditing
practical standards system

…. common standards for all stages of the

1994 - ALTE published its first Code of Practice
Focused on the roles and responsibilities of

stakeholders in striving for fairness

Striving for fairness for stakeholders of the examinations
The Code of Practice identifies the roles of three groups

of stakeholder in the testing process:
the examination developers - e.g. members of ALTE
the examination takers - primary users who take the examinations

by choice, direction or necessity
the examination users - secondary users (such as teachers) who

require the examination for some decision-making or other purpose

Striving for fairness for stakeholders of the examinations
The Code of Practice identifies the roles of three groups

of stakeholder in the testing process:
the examination developers - e.g. members of ALTE
the examination takers - primary users who take the examinations

by choice, direction or necessity
the examination users - secondary users (such as teachers) who

require the examination for some decision-making or other purpose
Clients or customers
The Code of Practice lays down four broad areas of

responsibility:
developing examinations
interpreting examination results
striving for fairness
informing examination takers
Striving for fairness is a shared responsibility involving all

stakeholders

CoP now in e.g. teachers
22 languages

QMS checklists

The Principles of Good Practice set out in more detail

the principles which ALTE members should adopt in
order to achieve high professional standards
It is also important for other stakeholders to understand

some of these principles too – e.g. language teachers

Principles of Good Practice
VRIP Features of the Exams
Four key concepts:
ÖV
ÖR
ÖI
ÖP

Four key concepts:
Ö Validity
ÖR
ÖI
ÖP

Four key concepts:
Ö Validity
Ö Reliability
ÖI
ÖP

Four key concepts:
Ö Validity
Ö Reliability
Ö Impact
ÖP

Four key concepts:
Ö Validity
Ö Reliability
Ö Impact
Ö Practicality

Four key concepts:
Ö Validity
Ö Reliability
Ö Impact
Ö Practicality
See Bachman and Palmer 1996

Validity
Arthur Hughes, 2003
A test is said to be valid if it provides:
“…consistent measures of precisely the abilities we are interested in.”
So as testers we need to specify:

- the abilities to be tested - the constructs
- how those abilities are measures consistently and
precisely - i.e. to achieve reliability

Reliability
Accurate
Consistent
Stable results
Is the same ability level needed to pass at each
administration?
For Speaking and Writing Tests – is there
standardisation of procedures and the assessment?

Impact
Effects, beneficial or otherwise, on candidates:
washback on the classroom;
Wider social, economic or political impact
Meeting learning goals of the test taker
Testing real skills/using authentic materials
Aligned to the CEFR?

Practicality
Costs and availability of resources
Availability of venues and administration dates
Availability of qualified examiners

QMS checklists

c) The Quality Management System
“Putting the principles into practice”
The CoP as self-evaluation checklists
Based on international QM Systems such as ISO

9001:2000
Incorporates the principle of customer satisfaction

Aim
To improve quality to improve

customer satisfaction
In language testing, who are the customers/clients?

Code of Practice
Primary Users
= test takers
Secondary Users
= those who use the
results tests
Achieving fairness - a shared responsibility

The QM checklists are based on 4 aspects of the
Test Development and Administration Cycle:
1. Test Design and Construction

2. Administration
3. Processing (marking, grading, issue of results)
4. Analysis and Review
- based on Model of Test Development

ALTE Quality Checklists - Units 1 to 4

Unit 1 - Test Construction
A. Conceptual phase
B. Test development, test construct & context
C. Communication with External Stakeholders

Example from the Conceptual Phase
Saville and Hargreaves –

This
model question
of speaking ability
usedrelates to the
for Cambridge
examinations
construct validity of
the test
This relates to the
construct validity of the
test
Example from the Development Phase
This
This question
question also
also relates
relates
to
to various
various aspects
aspects of
of
validity:
validity:
Construct-related
Construct-related
Criterion-related
Criterion-related
Content-related
Content-related

Example from the Communication Phase
This
This question
question relates
relates to
to “informing test takers” and thus to the
“informing impact
test takers”
of theand
test (consequential validity)
thus to the impact of the test
(consequential validity)

Unit 2 - Administration/logistics
This question relates to the important area of setting test performance

This question
conditions – asrelates
changesto the important area
in conditions of setting
can affect test performance
performance this can
conditions – as changes
be seen inasconditions can affectissue
a validity/reliability performance this can
be seen as a validity/reliability issue

Unit 3 - Marking, Grading, Results
Examiner recruitment &

training documents/regulations
This question relates to the establishment of procedures to

This question relates to the
ensure establishment
rater reliability of procedures to
ensure rater reliability

Unit 4 - Test analysis and post-exam review
This
This
question
question relates
relates
to the
to the
estimation
estimation
of test
of test
biasbias – a
– a validity
validity
issueissue

QMS checklists

d) Standards and the Quality Profile
Quality standards
“Best Practice Models”
Good practice
Satisfactory
Quality Standard
In need of improvement

Minimum standards have been agreed to establish a

Quality Profile for an exam or suite of exams
ALTE members are now required to explain how their
examinations meet these standards
Evidence to back up the argument is also required
The explanation and the evidence is checked by an
auditing process

The 17 minimum standards are based on the Quality
Checklists and cover:
test construction
administration & logistics
marking & grading
test analysis
communication with stakeholders
in 26 languages
Minimum Standards – example in English
TEST CONSTRUCTION
1 The examination is based on a theoretical construct, e.g. on a model of communicative

competence.
2 You can describe the purpose and context of use of the examination, and the
population for which the examination is appropriate.
3 You provide criteria for selection and training of test constructors and expert
judgement is involved both in test construction, and in the review and revision of the
examinations.
4 Parallel examinations are comparable across different administrations in terms of

content, stability, consistency and grade boundaries.
5 If you make a claim that the examination is linked to an external reference system (e.g.
Common European Framework), then you can provide evidence of alignment to this
system.

Minimum Standards – another example
TESTO RENGIMAS
1 Egzaminas pagrindžiamas teoriškai, teorinis konstruktas gali būti, pavyzdžiui,
komunikacinės kompetencijos modelis.
2 Apibūdintas egzamino tikslas, jo naudojimo sąlygos, kandidatai, kuriems tinka šis

egzaminas.
3 Yra nustatyti testų rengėjų atrankos ir mokymo kriterijai, ir specialistai aptaria ir

sprendžia testų rengimo, egzamino peržiūros ir pertvarkymo klausimus.
4 Skirtingose vietose vykdomi egzaminai gali būti lyginami turinio, pastovumo, darnos
ir įvertinimo ribų požiūriais.
5 Tvirtinimai, kad egzaminai atitinka kokią nors išorinę atskaitos sistemą (pavyzdžiui,
„Bendruosius Europos kalbų metmenis“), pagrindžiami sąsajų su ta sistema
įrodymais.

Minimum standard
Test construction
1 The examination is based on a

theoretical construct, e.g. on a model of
communicative competence.

Minimum standard
Administration & logistics
6 All centres are selected to administer

your examination according to clear,
transparent, established
procedures, and have access to
regulations about how to do so.

Minimum standard
Marking & grading
11 Marking is sufficiently accurate and

reliable for purpose and type of
examination.

Minimum standard
Test analysis
13 You collect and analyse data on an

adequate and representative sample of
candidates and can be confident that their
achievement is a result of the skills
measured in the examination and not
influenced by factors like L1, country of origin,
gender, age and ethnic origin.

Minimum standard
Communication with stakeholders
15 The examination administration system

communicates the results of the
examinations to candidates and to
examination centres (e.g. schools)
promptly and clearly.

But - different ALTE exams need different quality
profiles to take into account the following:
context and purpose of use

target candidature – size, demographics, distribution
currency and recognition
distribution of test centres
type of organisation involved

The Auditing Process
ALTE Members build an argument that the quality profile is sufficient and appropriate
for a particular test or suite of tests
There is no intention to impose a single set of quality profiles
The audit has both a quality control and consultancy function
It aims to establish that minimum standards are met in a way that is appropriate to the
context of the test and to offer recommendations towards best practice
There is always room for improvement !

Teachers can also base their questions to examination
providers on these minimum standards:
Standard 1
.
What theoretical construct is your exam based on?
What model of communicative competence do you use?
How is this reflected in your test tasks and procedures?

Teachers can also base their questions to examination
providers on these minimum standards:
Standard 4
.
How do you ensure that you produce parallel examinations which
are comparable across different administrations?
How do you ensure comparability of content and stability of the

grade boundaries?
What data and analysis do you have to demonstrate this?

Finally how can the CEFR help
testers … and teachers?
….. returning to the question:


Who knows anything about the CEFR?
The Common European Framework of Reference for Languages:
learning, teaching, assessment
Le Cadre Commun Européen pour les Langues: apprendre, enseigner,

évaluer
Der Gemeinsame Europäische Rahmen für Sprachen: lernen, lehren,

beurteilen
etc…..
Already in 36 languages (Lithuanian in preparation)

Most people now know the levels!!
A B C
Basic User Independent User Proficient User
A1 A2 B1 B2 C1 C2
Breakthrough Waystage Threshold Vantage Effective Mastery
Operational
Proficiency
But it is much more too!

or
Is my B1 comparable with your B1?
How can we tell if two different tests can be used for the
same purpose? Are they equivalent or comparable?
Two test of English at B1

A test of English at B1 and a test of French at B1
What do we mean when we talk of ‘equivalence’ or

‘comparability’?

Equivalent – meaning #1
‘Different versions of the same test, which are
regarded as equivalent to each other in that they
are based on the same specifications and
measure the same competence.’
From: ALTE Multilingual Glossary of Testing Terms
Published in the Studies in Language Testing, Vol 6, (1998),

Cambridge ESOL/CUP

‘To meet the strict requirements of equivalence
under classical test theory, different forms of a
test must have the same mean difficulty,
variance, and co-variance, when administered to
the same persons.’
ALTE Multilingual Glossary of Testing Terms, 1998

‘Equivalence is very difficult to achieve in practice.’
ALTE Multilingual Glossary of Testing Terms

‘the relationship between two (different) tests’
‘Strictly speaking, this concept is unjustifiable, since each test is

designed for a different purpose and a different population, and may
view and assess language traits in different ways as well as
describing test-taker performance differently.’
Dictionary of Language Testing Terms, 1999 – Davies et al
Published in the Studies in Language Testing, Vol 7, (1998),

Cambridge ESOL/CUP

‘However, in reality test users may demand

statements of equivalence between different tests
(for example, admissions officers at educational
institutions).’
Dictionary of Language Testing Terms, 1999

Equivalent – meanings #1 and #2
It is common and quite easy to put together a
table which ‘maps’ a range of different exams
against each other (for example using the
CEFR)
But what evidence is there to support the claims

of equivalence or comparability?

Remember….
‘… each test is designed for a different purpose
and a different population, and may view and
assess language traits in different ways as well
as describing test-taker performance differently.’
This is reflected in the approach taken by ALTE

in the QM system and the way the auditing
system works
Council of Europe Manual for Guidance
Preliminary Pilot Version - 2003/4
November 2003 DGIV/EDU/LANG (2003) 10
Relating Language Examinations to the

Common European Framework of Reference for
Languages: Learning, Teaching, Assessment
It can be accessed in English and French at the Council of Europe
webpage: www.coe.int/lang

The 4 parts of the Manual:
Familiarisation:
You must know the CEFR!!
Specification:
You must specify your constructs and content coverage in relation
to the CEFR
Standardisation:
You must be able to compare the test with validated test samples,
and student performances at CEFR levels
Empirical validation:
You need to have the relevant technical know-how to calibrate
your exam materials

The Manual can also help language teachers:
to ask the the right questions
to understand the answers related to:

Specification
Standardisation
Empirical validation
to evaluate claims of comparability and alignment
Teachers should ask test providers to support their claims

using the CEFR and the Manual as points of reference

Conclusions
Test providers can help teachers understand the concepts
and explain the empirical evidence
E.g. using the ALTE toolkit
Comparative frameworks such as the CEFR serve a useful

function for a wide variety of test stakeholders
the CEFR makes it easier to understand the range of options
which are available and help users to make appropriate choices for
their needs

Conclusions
HEALTH WARNINGS
Claims that are made need to be checked – make sure
that there is adequate evidence to back them up
Be careful in using the CEFR and other Frameworks
they encourage oversimplification and misinterpretation
there is a danger that become prescriptive, rather than informative tools
Use the ALTE toolkit and the Council of Europe Manual to

help evaluate claims and understand the evidence

Conclusions
CAVEAT EMPTOR
Thank you for your attention

Comparing Quality:: Nick Saville

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Comparing Quality:: Nick Saville

Uploaded by

Copyright:

Available Formats

Comparing Quality:

what teachers need to ask exam providers

 Testing for teachers - questions to ask

Vilnius November 2007 Nick Saville

‘… many language teachers harbour a mistrust of tests and testers.’

‘Too often language tests have a harmful effect on teaching and

He also explains why teachers need to know more about language

Vilnius November 2007 Nick Saville

 Teachers need to be able to ask appropriate

 They also need to understand the main concepts of

 Otherwise they can’t understand the answers!

Vilnius November 2007 Nick Saville

 He suggests that users of tests – students and

I’ll come back to this idea later

Vilnius November 2007 Nick Saville

Vilnius November 2007 Nick Saville

What sort of questions should we ask about language

How can we tell the difference between a good test

Vilnius November 2007 Nick Saville

 Is this B1 test really a B1 test?

 Does it test what you say it does?

 Are the judgements made by the oral examiners

Vilnius November 2007 Nick Saville

How do you know if the answers are adequate?

Or whether they show that the test meets

Vilnius November 2007 Nick Saville

 The concept of quality is based on customer satisfaction

 As customers the stakeholders need to:

Know what to look for (for their own purposes)

Vilnius November 2007 Nick Saville

Client Service provider

Nico chose this one

Price (around 200

Panasonic DMC- FX100EB-K Panasonic DMC- FZ50EB-K

Vilnius November 2007 Nick Saville

 Specification:  Rating (5-point scale)

Vilnius November 2007 Nick Saville

 For example, Eye tests at the optician

Vilnius November 2007 Nick Saville

Vilnius November 2007 Nick Saville

Vilnius November 2007 Nick Saville

Vilnius November 2007 Nick Saville

What “tools” are available to help

Vilnius November 2007 Nick Saville

Vilnius November 2007 Nick Saville

Motto: “Attaining standards: sustaining diversity”

Vilnius November 2007 Nick Saville

Motto: “Attaining standards: sustaining diversity”

Vilnius November 2007 Nick Saville

All available on ALTE website in many languages

All available on ALTE website in many languages

ALTE COP & QMS

Vilnius November 2007 Nick Saville

…. common standards for all stages of the

1994 - ALTE published its first Code of Practice

Focused on the roles and responsibilities of

Vilnius November 2007 Nick Saville

The Code of Practice identifies the roles of three groups

 the examination developers - e.g. members of ALTE

 the examination takers - primary users who take the examinations

 the examination users - secondary users (such as teachers) who

Vilnius November 2007 Nick Saville

Testing for teachers - questions to ask

Teachers need to be able to ask appropriate

They also need to understand the main concepts of

Otherwise they can’t understand the answers!

He suggests that users of tests – students and

Is this B1 test really a B1 test?

Does it test what you say it does?

Are the judgements made by the oral examiners

The concept of quality is based on customer satisfaction

As customers the stakeholders need to:

Specification: Rating (5-point scale)

For example, Eye tests at the optician

the examination developers - e.g. members of ALTE

the examination takers - primary users who take the examinations

the examination users - secondary users (such as teachers) who

the examination developers - e.g. members of ALTE

the examination takers - primary users who take the examinations

the examination users - secondary users (such as teachers) who

A test is said to be valid if it provides:

So as testers we need to specify:

Minimum standards have been agreed to establish a