GENERATING

VALIDITY EVIDENCE
Cyril J. Weir.2005. Language Testing and Validation. New York:
Palgrave Macmillan
INTRODUCTION
Messic (1992) points out that many test makers admit a
responsibility for providing validity evidence of the instrumental
value of a test but very few of these test makers do the validity.

Ellerton 1997, alderson and Buck 1993 and Alderson et al. 1995
takes this further and sets the failing of many tests in relation to
the provision of validity evidence
RESEARCH METHODOLOGIES
FOR EXPLORING THE VALIDITY
OF A TEST
1. An Introductory note on research.
2. A priori validation: investigating the specification of the construct and
the operationalization of the test.
3. Establishing context validity.
4. Establishing theory – based validity evidence
5. Establishing scoring validity evidence.
6. Establishing evidence on a posteriori validities.
1. An Introductory note on research

There are some criteria that should be required in good research.

1. It should be believable.
2. It should be logical.
3. It should be feasible.
4. It should be important to the person doing it.
5. It should have value/ interest.
6. It should have relevance.
 2. A priori validation: investigating the
specification of the construct and the
operationalization of the test.

Analyzing the
Literature curriculum, Expert
Need analysis
Review syllabus and Judgement
textbooks
3. Establishing context validity.

One example of context validity that developed by

Akmar Saidatul Zainal Abidin in validating a spoken
language test at university level in Malaysia.
Context Validity Case Study 1: Small-scale Research Into Elements
Of The Framework.

The case of speaking

The Case Of Speaking

context validity task processing

consideration condition
Kinds Of Complexities That Can Influence
Task Difficulty (Skehan, 1996)
1. Code complexity
2. Cognitive complexity
3. Communicative stress
Kinds Of Complexities That Can Influence
Task Difficulty.
A number of empirical findings have revealed that task difficulty
has an effect on performance as measured in the three areas of
accuracy, fluency and complexity (Menhert 1998, Skehan 1996,
Skehan and Foster 1997 & 1999, O’sullivan and weir 2002,
Wigglesworth 1997).
Establishing Task Difficulty
Norris et.al. (1998) argue that task
difficulty can be based on subjective
assessment of the components
contribution variable suggested by Skehan
(1996): code complexity, cognitive
complexity, and communicative demand.
Establishing Task Difficulty
The comparison of variable contribution of
identified sources of task difficulty with
the rating of examinee performances
according to the real world criteria should
provide valuable information regarding
the contribution of task difficulty to
perceived task success (Norris 1998)
The Example Of Task
Difficulty Questionnaires
The Example Of Task
Difficulty Questionnaires
Methods And Research
Methodology Design
Hypotheses

Jessica Wu in her research stated some

hypotheses regarding with some possible sources
of variability conditions may affect performance
on test of spoken language.
 Guided pre-task
planning.
test takers
Amount of time
‘familiarity with
allowed for
the propositional
performing a
content in test
task.
tasks.

the complexity of will have no effect on

language of the test task performance,
inter-language
written input to measures of accuracy, Gender of
test tasks in term fluency, lexical range,
and qualitative self-
speaker.
of syntactic report estimates of
range. difficulty.
 will not be
The effect of altered by
these test takers’
variables proficiency
level.
Quantitative phases
1. Development and validation of
alternate test task forms.
2. Development and validation of
post-test instruments
 Qualitative Studies
Based on Norris et.all 1998.

Number of
reformulation false starts
s replacements

Repetitions

Fluency Hesitations

pauses over 1
second
4. Establishing theory – based validity
evidence.
There are 3 stages in form of theory –
based validity
1. Qualitative expert judgement of items
2. Qualitative introspection/retrospection
by test takers
3. Quantitative and qualitative analysis of
test performances
 Establishing theory-based validity: Case
study 2.

Computer Paper
Based Based
Test Test
The Development Of The Cognitive
Processing Questionnaire

Our Cognitive Processing Questionnaire (CPQ) was

designed to reflect the underlying concepts
identified in the framework of theory-based validity
outlined
Which one is better?

Cognitive and psychological aspect

Computer
Paper Based
Based
Test
Test
Establishing theory-based
validity: Case study 3
to investigate the internal cognitive processes
involved in completing different reading tasks, and
thus provide some evidence for the theory-based
validity of an Advanced Level Reading Test, which is
undergoing development.
Thank you