Lecture Notes about Designing test tasks to measure L2 grammatical ability.

Introduction:

 Test scores can vary as result of the personal attributes of test-takers such as Age, Gender
and Language Background.
 Test Takers can also fluctuate because of their strategy use, motivation and level of anxiety.
 The characteristics of the test is one of the most important factor affecting grammar test
scores.
 Test takers vary in their test performance depending on the type of questions on the test.
 Test task characteristics – set of unique characteristics. These can potentially interact with
the characteristics of examinee.
 It is necessary for test developers to understand the individual characteristics of the tasks
they use and to follow systematic procedures for designing and developing tasks
 Understand both the nature of grammatical ability and nature of the test tasks so that we will
be able to account for the effect of method on how to interpret scores on grammar tests
 Objectives:

1. To know the basics of initial test construction.

2. To define terms related to initial test construction.

3. To determine the basic steps in the initial test construction.

4. To identify and describe the different characteristics of test construction

How does test development begin?

 Begins with a desire to obtain (and often provide) information about how well a student knows
grammar in order to convey meaning in some situation where the target language is used.
 Information obtained from this assessment then forms the basis for decision-making
 Target language use (TLU) situations (Bachman & Palmer, 1996) is the situations in which we
use it for instruction or testing
 Target language use tasks is the tasks or activities requiring language to achieve a
communicative goal. One of many language-use tasks that test-takers might encounter in the
target language use domain.
 Once we identify the characteristics of the TLU tasks, we can then design test tasks.
 Bachman and Palmer (1996) have proposed a single framework that allows us to characterized
both the features of the language -use task and the features of the test task.
 Test authenticity is the correspondence between the features of language-use tasks and those
of test tasks.
 In developing grammar assessments, we first articulate the purpose (s) of the test, consider the
constructs and identify the situational domain(s) in which we would like to make the inferences
about the test-takers' grammatical ability
 Initial steps in test-task development:
 a. Identify the test purpose(s), the use of the test results, and the potential impact of the
test on test- grammatical and meaningful ways (in a chemistry takers and on further
instructions.
b. Identify the target language use domain
c. Identify a range of language use tasks from the target language use domain
d. Select the target language use task(s) for this test
e. Define the constructs to be measured

 Although this process is sequential, identifying the TLU domain and areas of grammatical knowledge
to be measured does not necessarily have to be done in this order because the specification of
language-use tasks and the definition of the constructs we want to measure is recursive and
interactive. What is essential is that we include both in our procedures.
 Most classroom testing contexts, the definition of test constructs is we test what we teach.
 In instances of language use where the areas of grammatical knowledge are not so clearly identified
or described, the job of identifying and defining test constructs becomes much more complicated.
 Needs analysis helps determine the specific characteristics of language use in these contexts. A
needs analysis focusing on the language needed to communicate in some TLU situation involves
both the gathering of data from the target language use and the analysis of these data from a
language-needs perspective spelled out in grammatical.
 The definition of grammatical ability as it relates to the test purposes, the target language use
domain, and the target language use tasks is an essential preliminary step to the specification of
test tasks
 Once the grammatical constructs have been defined, we need to consider what types of evidence
and how much of it examinees would need to supply in order to demonstrate that they have
grammatical ability. To determine this, we could list the claims we want to make in one column and
the types of evidence we would need to see to support each claim in another.
 Once we know the kinds of evidence we need to observe, we can then design, develop and score
the test tasks so as to put the test to operational use
 Operationalization is the process of designing tasks to elicit concrete samples of performance to
reflect the underlying claims about the constructs, and the process of deciding how to measure or
score the responses. It lays out a plan for how a test will actually look, and it specifies the individual
features of each test task.
 Further steps in test task development:
a. Identify the types of evidence students would need to supply to grammatical demonstrate
support for the grammatical form about what relevant test-takers know and/or can do.
b. Identify how much of the evidence is needed to support claims about the constructs at
different levels of ability.
c. Operationalize test tasks supply the evidence needed support claims of grammatical ability
 Specification of test tasks is the initial step in the operationalization of test constructs. A process of
identifying and describing the characteristics of the individual test tasks. It is also a process of
comparing how target language use tasks and test tasks compare.
 What do we mean by task?
 Task' has referred to any activity that requires students to do something for the intent purpose of
learning the target language
 Traditional learning or teaching tasks are characterized as having an intended pedagogical purpose-
they have a set of instructions that control the kind of activity to be performed; they contain input
(e.g. questions); and they elicit a response
 More recently, learning tasks have been characterized more in terms of their communicative goals,
their success in eliciting interaction and negotiation of meaning, and their ability to engage learners
in complex meaning-focused activities
 Rea-Dickins (1991) argued that grammar tasks should minimally provide a realistic situation in which
test-takers need to exchange information in order to complete the task. She specified five design
characteristics that could contribute to the communicative nature of the task. These include (1) the
contextualization of the test items, (2) the identification of a communicative purpose, (3) the
identification of interlocutors, (4) instructions that focus on meaning rather than forms, and (5) the
opportunity for the test-taker to create his/her own messages and to produce grammatical responses
as appropriate to a given context.
 Task-based language instruction and assessment, has taken on a much more specific meaning. Tasks
in this literature refer to activities that elicit communicative performances similar to those occurring
in some non-test or real-life situation. Good performance is then judged according to criteria that
involve the degree to which examinees have achieved the communicative goal of a specific task.

What are the characteristics of grammatical test tasks?

 . To understand the characteristics of test tasks better, we turn to Bachman and Palmer's (1996)
framework for analyzing target language use tasks and test tasks.
 This framework of task characteristics represents the most recent thinking in language assessment of
the potential relationships between task characteristics and test performance. In this framework,
they outline five (5) aspects of tasks, each of which is characterized by a set of distinctive features.
These five aspects describe characteristics of (1) setting, (2) the test rubrics, (3) the input, (4) the
expected response. (5) the relationship between the input and response. This framework can be used
to (1) describe the TLU tasks as basis for designing test tasks; (2) specify the test tasks; and (3)
compare the characteristics of the TLU tasks with the test tasks.

Characteristics of the setting

 Include the physical characteristics, the participants, and the time of the task. Can have a
serious, unexpected effect on performance.

Characteristics of test rubrics

 Include the instructions, the overall structure of the test, the time allotment, and the
method used to score the response. These characteristics can obviously influence test
scores in unexpected ways
 The overall test instructions (when included) introduce test-takers to the entire test
 The test-task instructions introduce test-takers to the task and make explicit the
procedures for completing it. Involve the purpose an introduction of the areas of
grammatical ability being tested, the scoring method, and the recommended time
 allotment. A second characteristic of the test rubrics, the structure, makes explicit the
number of parts or tasks, the salience and sequence of these parts or tasks, their relative
importance in the overall test and the number of tasks or items per part. Another
important characteristic of the test rubric is the time allotted for the candidate to
complete the task. A grammar can be designed to be speeded, where tasks are all at
about the same level of difficulty, and not all test-takers are expected to have the time to
answer all the items. Finally, the test rubric includes the characteristics of the scoring
method or the ways in which responses are evaluated, and numbers assigned to
responses. The scoring method includes the criteria used to judge the correctness of
responses, the procedures for scoring the responses and the explicitness with which
examinees are told how scores are determined.

Characteristics of the input

 Sometimes called the stimulus, are critical features of performance in all test and TLU
tasks. The input is the part of the task that test-takers must process in order to answer
the question. The topical characteristics of the input refer to the range of topics that the
input could encompass and the degree to which these topics tap into the examinees'
topical knowledge

Characteristics of the expected response

 Refers to the type of grammatical performance we want to elicit. The characteristics of

the expected response are also considered in terms of the format and language. Similar
to the input, the expected response of grammar tasks can vary according to channel,
form, language and vehicle.
 The type of expected response is a critical characteristic of test and language-use
 Types of expected responses: (1) selected-response tasks, (2) constructed-response tasks
(3) extended-production tasks.
 The expected response can also be characterized in terms of its language and topical
characteristics.
 The topical characteristics of the expected response refer to the topics, themes or
subject matter that are communicated in grammar use. According to Bachman and
Palmer (1996) the topical knowledge of language users is always involved in language use
and "will always be a factor in test performance'

Describing grammar test tasks

 Holistic task types constitute collections of task characteristics for eliciting performance
that these holistic task types can vary on a number of dimensions. We need to remember
that the tasks we include on tests should strive to match the types of language-use tasks
found in real-life or language instructional domains.
 Many attempts at categorizing the types of tasks found on tests. Some have classified
tasks according to scoring procedure. For example, objective test tasks (e.g. true false
tasks) are those in which no expert judgment is required to evaluate performance with
regard to the criteria for correctness. Subjective test tasks (e.g. essays) are those that
 require expert judgment to interpret and evaluate performance with regard to the
criteria for correctness
 Selected-response tasks
a. Multiple-choice activities
b. True/False activities
c. Matching activities
d. Discrimination activities
e. Lexical list activities
f. Grammaticality judgement activities
g. Noticing activities
 Limited-response tasks
a. Gap-filling activities
b. Cloze activities
c. Short-answer activities
d. Dictation activities
e. Information-transfer
f. Some information-gap activities
g. Dialogue
 Extended-production tasks
a. Summaries, essays
b. Dialogue, interviews
c. Role-plays, simulations
d. Stories, reports
e. Some information gap activities
f. Problem-solving activities
g. Decision making