You are on page 1of 39

09/07/11


1
Aligning the TEF (Test d’évaluation de français)
with different language frameworks
Dominique CASANOVA, Chambre de commerce et d’industrie de Paris (France)
Alexandra CRENDAL, Chambre de commerce et d’industrie de Paris (France)
Pr. Marc DEMEUSE, Université de Mons (Belgium)
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

2
1. What is the TEF?
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

3
A High-stakes test
Immigration to French speaking countries
Studies in French speaking countries
Recognizedinternationally
Since J uly 2002 by Citizenship Immigration Canada
Since March 2004 by the French Ministry of Higher Education
Since October 2006 by the « Ministère de l’Immigration et des Communautés culturelles du
Quebec » (adapted format)
Meeting international standards
Since 2005 : certified OSI 9001
Since 2009 : CCIP became an ALTE full member after auditing the TEF
TEF, a high-stakes test
09/07/11

4
Reading Comprehension
50 questions – 60 minutes
Listening Comprehension
60 questions – 40 minutes
Vocabulary and Language Stuctures
40 questions – 30 minutes
Oral Production
2 parts – 35 minutes
Written Production
2 parts – 60 minutes
What does the TEF consist of?
A1
C2
C1
B2
B1
A2
09/07/11

5
2. The NCLC
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

6
2008-2010 : national consultation new version
NCLC & Canadian Language Benchmarks
1996: Canadian Language Benchmarks
09/07/11

7
NCLC & Canadian Language Benchmarks
• a descriptive scale of communicative
proficiency in French as a Second Language
(FSL) expressed as 12 benchmarks or reference
points
• a set of descriptive statements about
successive levels of achievement on the
continuum of FSL performance;
• statements (descriptions) of communicative
competencies and performance tasks in
which the learner demonstrates application of
language knowledge (competence) and skill;
• a framework of reference for learning,
teaching, programming and assessing adult
French as a Second Language in Canada, and
• a national standard for planning second
language curricula for a variety of contexts, and
a common “yardstick” for assessing the
outcomes.
09/07/11

8
3. RelatingTEF to SLCs
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
2 / 4 points 2 points 1 point 0 point
SLC 1 SLC 2 SLC 3 SLC 4 SLC 5 SLC 6 SLC 7 SLC 8 SLC 9 SLC 10 SLC 11 SLC 12
09/07/11

9
7 levels
12 levels
4 cut-points
Specifications - 2002
Standardisation - 2004
RelatingTEF to SLC 2002
09/07/11

10
RelatingTEF to SLCs - Main stages
Stage 1. Analysis of SLC (per skill and per level summaries)
Stage 2. Differentiation of adjacent SLCs at cut points
Stage 3. Correspondence between TEF and SLC scales
Stage 4. Correspondence between TEF specifications and SLCs
Stage 5. Empirical correspondence between TEF items and SLCs
09/07/11

11
3. Correspondance between TEF and SLC scales
09/07/11

12
4. Correspondance between TEF specifications
and SLCs for receptive skills
09/07/11

13
2 / 4 points 2 points 1 point 0 point
5. Empirical correspondence between
TEF items and SLCs (receptive skills)
Step 1. Individual classification of item descriptions
Step 2. Consensus on items reference level regarding their
general description
Step 3. Individual classification of items
Step 4. Agreement on items classification regarding CIC cut-points
SLC 1 SLC 2 SLC 3 SLC 4 SLC 5 SLC 6 SLC 7 SLC 8 SLC 9 SLC 10 SLC 11 SLC 12
Experts judgements (3 experts), 2-days meeting
09/07/11

14
SLC 1
st
step:
individual classification of item descriptions
Unités
types
(CE)
Unités
types
(CO)
Unités
types
(LS)
Unités
types
(CE)
Unités
types
(CO)
Unités
types
(LS)
Unités
types
(CE)
Unités
types
(CO)
Unités
types
(LS)
Expert 1 (room 1) Expert 2 (room 2) Expert 3 (room 3)
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 2
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 3
SLC 1
unclassifiable unclassifiable unclassifiable
SLC 1
SLC 12
09/07/11

15
Expert 1
Expert 2
Expert 3
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
Common
SLC 1
Common classification of reference
for TEF item descriptions
Sharing results and viewpoints :
> Problems faced during classification
> Rationales for classification
> Comparison of the classifications / levels
> Rationales for inclassifiable items descriptions
> Discussion, consensus
SLC 2
nd
step:
Consensus meeting on item descriptions
09/07/11

16
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Unclassifiable
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 2
SLC 1
Unclassifiable
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 1
SLC 1
Expert 3
SLC 1
Unclassifiable
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
Common
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
Common
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
SLC 1
Common
SLC 1
SLC 3
rd
step:
individual classification of items samples
Items
(CE)
Items
(CO)
Items
(LS)
Items
(CE)
Items
(CO)
Items
(LS)
Items
(CE)
Itemss
(CO)
Items
(LS)
Expert 1 (room 1) Expert 2 (room 2) Expert 3 (room 3)
294 items
09/07/11

17
Agreement Frequency % Cumulated
frequency
% cumulated
4 20 27,78 20 27,78
3 24 33,33 44 61,11
2 28 38,89 72 100,00
0 0 0 72 100,00
Agreement for Reading comprehension items
Classification in the 4 main categories
09/07/11

18
Agreement Frequency % Cumulated
frequency
% cumulated
4 32 27,35 32 27,35
3 48 41,03 80 68,38
2 37 31,62 117 100,00
0 0 0 117 100,00
Agreement for Listeningcomprehension items
09/07/11

19
Agreement for Language structures items
Agreement Frequency % Cumulated
frequency
% cumulated
4 28 26,67 28 26,67
3 36 34,29 64 60,96
2 40 38,10 104 99,05
0 1 0,95 105 100,00
09/07/11

20
Feedback
No standardisation of the familiarization phase.
No training for classification of items
But consensus step after the classification of item descriptions.
More than 3 judges would have been safer.
09/07/11

21
4. RelatingTEF to CECR
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
•Specification
•Standardisation
•Empirical study
Coe (2003), Manual for relating Language Examinations to the Common European
Framework of Reference for Languages (CEFR), Pilot version
09/07/11

22
7 levels
6 levels
Specifications : 2001 - 2002
Specifications and
Standardisation : 2008
AligningTEF with CEFR
09/07/11

23
Correspondance between TEF
specifications and CEFR
Global description of the test and the subtests
For each type of item
•Domains
•Situations
•Themes
•Tasks / language activities
•Strategies
•Scales
•Descriptors
•Levels
09/07/11

24
Theoretical framework and TEF items definition Common European Framework of
Reference
Item
codifica-
tion
TEF
level
Itemdescription CEFR
levels
CEFR descriptors
CEAU1 TEF 0
+
Reconnaître les situations élémentaires de la vie
courante:
- identifier la nature et la fonction d’un document très
court de l’environnement quotidien (pancartes,
étiquettes …) ;
- repérer des mots isolés.
- 20 mots maximum.
A1
Peut comprendre des textes très courts et très
simples, phrase par phrase, en relevant des noms,
des mots familiers et des expressions très
élémentaires et en relisant si nécessaire.
CEAU2 TEF 1
Comprendre les situations simples de la vie courante
(sans complication) :
- comprendre les informations essentielles d’un
document court de l’environnement quotidien;
- textes plutôt injonctifs ou explicatifs (publicités,
annonces, faire-part, notices …)
- 50 mots maximum.
A1
Peut comprendre des textes très courts et très
simples, phrase par phrase, en relevant des noms,
des mots familiers et des expressions très
élémentaires et en relisant si nécessaire.
CEAU3 TEF 1
Comprendre les situations simples de la vie courante
(sans complication) :
- comprendre un document court de l’environnement
quotidien;
- repérer les informations essentielles (textes plutôt
injonctifs ou explicatifs: publicités, annonces, faire-part,
notices ….).
- 50 mots maximum.
A1
Peut comprendre des textes très courts et très
simples, phrase par phrase, en relevant des noms,
des mots familiers et des expressions très
élémentaires et en relisant si nécessaire.
Correspondence between TEF
specifications and CEFR
09/07/11

25
Standardisation
•Selection of judges (15)
•Selection of items (185)
•Familiarization 1 day
•Classification 4 x 2 hours
•Consensus 1 day
•Data analysis
Pre-tests Real tests
Item Bank
Empirical level /
specification level
CE CO LS
Same level 31 29 17
1 level of difference 23 24 13
More than 1 level of
difference
18 18 12
09/07/11

26
Familiarization
•Presentation of the TEF, of the standardisation procedure
•Presentation of the CEFR (main concepts)
•Analysis of levels of reference, self-assessment of one’s level in a
foreign language, sorting of CEFR scales descriptors
•Individual classification of illustrative tasks (Coe CD-Rom),
discussions to reach a consensus
•Individual classification of an illustrative sample of TEF items,
discussion to reach a consensus
09/07/11

27
7
0+
Unités
types
(CE)
Unités
types
(CO)
Unités
types
(LS)
Unités
types
(CE)
Unités
types
(CO)
Unités
types
(LS)
Unités
types
(CE)
Unités
types
(CO)
Unités
types
(LS)
Judge 1 (room 1) Judge 2 (room 2) Judge …(room …)
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
Eval 1
SLC 1
Eval 1
CECR A2
Eval 1
CECR A1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 2
CECR A1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval …
CECR A1
CECR A1
CECR C2
CEFR experiment - 1
st
step:
individual classification of item descriptions
Reading: 16 item descriptions
Listening: 20 item descriptions
Languagestructures: 14 item descriptions
09/07/11

28
Judge 1 (room 1) Judge 2 (room 2) Judge …(room …)
Unités
réelles
(CE)
Unités
réelles
(CO)
Unités
réelles
(LS)
Unités
réelles
(CE)
Unités
réelles
(CO)
Unités
réelles
(LS)
Unités
réelles
(CE)
Unités
réelles
(CO)
Unités
réelles
(LS)
7
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
Eval 1
SLC 1
Eval 1
CECR A2
Eval 1
CECR A1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 2
CECR A1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval 1
SLC 1
Eval …
CECR A1
CECR A1
CECR C2
CEFR experiment - 2
nd
step:
individual classification of items samples
Reading: 72 items
Listening: 71 items
Languagestructures: 42 items
09/07/11

29
14 judges, 6 Itemdescriptions, 43 items
Final individual classification of corresponding items and item descriptions
Sharing results and viewpoints
(limitedto the most problematic elements)
Comparison of the classifications / levels
J ustification basedon the CEFR
CEFR experiment - 3
rd
step:
Consensus meeting

09/07/11

30
4. Results
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

31
Item descriptions
Items which empirical level
corresponds to the specification level
80% of agreement 71% of agreement
Panel decisions versus specification levels
09/07/11

32
Average agreement between individual classifications of item descriptions and items with
- the level granted by the panel (median)
- the level granted by the CCIP .
75%
65%
77%
70%
54%
77%
75%
61%
67%
64%
50%
63%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Reading Listening Language Structures
Group ID
CCIP ID
Group items
CCIP items
Average agreement
09/07/11

33
The 43 most problematic items (high disagreement)
• Reading : 11 items
• Listening : 20 items
• Language structures : 12 items
1- New individual classification before discussion
2- Discussion
3- Final individual classification
Consensus meeting
09/07/11

34
Consensus meeting – Enhancement of the agreement
Impact of consensus on judges agreement
(most problematic items)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
E02 E03 E04 E05 E06 E07 E08 E09 E10 E11 E12 E13 E14 E15
Before discussion
After discussion
Mean :
46% 63%
09/07/11

35
Classification of other items
Results of classifications for the different pools of items
54%
23%
58%
58%
68%
37%
57%
8%
47%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Individual / Panel Indifidual / Specifications individual / Empirical
Same level
Slightly different
Strongly diférent
09/07/11

36
Familiarization is an essential stage (common representation,
common understanding, training)
Consensus is also an essential stage (enhancement of the
inter-judges agreement)
A consensus after the first classification would harmonize even
more the representations before the classification of items
A large panel is safer providedthat you spend time enough on
building common representations.
Feedback
09/07/11

37
6. Perspectives
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
09/07/11

38
In prospect
Speaking and Writing
NCLC 2010
Alignement with other tests (BTC)
09/07/11

39
E-mail : dcasanova@ccip.fr
The Impact of Language Frameworks on Assessment,
Learning and Teaching, viewed from the perspectives
of Policies, Procedures and Challenges
L’évaluation des compétences linguistiques des adultes en français langue étrangère
dans une perspective de multiréférentialisation
Demeuse, M., Desroches, F., Crendal, A., Renaud, F., Oster, P. et Leroux X. (2004)
In Actes du 17e colloque international de l’Association pour le Développement des Méthodologies
d’Évaluation en Éducation (ADMEE-Europe). Lisbonne, Portugal, 18-20 novembre 2004.
Validation empirique d’un test de français langue étrangère en regard du Cadre
européen commun de référence pour les langues
Casanova, D., Crendal, A., Demeuse, M., Desroches, F. et Holle, A. (2010),
In Actes du 22e colloque international de l’Association pour le Développement des Méthodologies
d’Évaluation en Éducation (ADMEE-Europe), 881-902. Braga, Portugal, 14-16 janvier 2010.