ASSESSING WRITING

Ricardo Gamboa
Gina Cauich
Roberto Buenfil
Patricia Medina
Jessica Argüelles
ELICIT A VALID SAMPLE OF WRITING ABILITY
1 Set as many separate tasks as is practical

 Greater reliability and validity.

Keep a balance between what is desible and what is practical.

2 Test only writing ability, and nothing else

To keep Validity, we should not set tasks which measure other

abilities. E.g. creativity, imagination, intelligence.

Reading is another ability that might interfer with accurate

measurement of writing ability.

Instructions must be understood by everyone and not too long.

The use of illustrations is one of reducing dependence on the
candidates’ ability to read.
Example: (for science students)
The diagram below shows three types of bee.

Compare and contrast the three bees. Write about three - quarters of a
page.
A series of pictures can be used to elicit a narrative.

Look at the pictures and then tell the story.

Using the data in the graphic, write a paragraph in which you discuss
the relative annual incomes of each city over the period 1980 ~ 2000
Restrict candidates

Writing tasks should be well defined: candidates should know just what
is required of them, and they should not be allowed to go far astray.

Compare the benefits of a university education in English with that of

one in Arabic. Use all of the points given below and come to a conclusión.
Write about one page.

a) Arabic b) English
1. Easier for students 1. Books and Scientific sources
2. Easieer for most teachers mostly in English.
3. Saves a year in most cases 2. English international language.
3. Learning second language part of
education and culture.
 Tasks should not only fit well with the specifications, but they should
also be made as authentic as posible.

When thinking of authenticity, it is important to take into account the

nature of candidates and their relationship with the people to or for
whom the task requires them to write.
 ENSURE VALID AND RELIABLE SCORING

 -Set tasks which can be reliably scored: a number of the

suggestions made to obtain a representative performance will also
facilitate reliable scoring.

 -Set as many tasks as possible: the more scores for each candidate,
the more reliable should be the total score.
RESTRICT CANDIDATES:
The greater the restrictions imposed on the candidates, the more
directly comparable will be the performances of different candidates.

GIVE NO CHOICE OF TASKS:

Making the candidates perform all tasks also makes comparisons
between candidates easier.

ENSURE LONG ENOUGH SAMPLES:

The samples of writing that are elicited have to be long enough for
judgements to be made reliably.
REVIEW OF WRITING SCALES

 This embodies the test (or scale) developer’s notion of what

skills or abilities are being measured by the test. These are
of critical importance for the validity of the assessment.

 Types of scales: can be characterized by two distinctive

features

Whether the scale is intended to be specific to a single

writing task or generalized to a class of tasks.
Whether a single score or multiple scores are given to each
script.
First of all is to determine what type of rating scale will be
used.
Primary trait scale

This is how well Ss can write within a narrowly defined

range of discourse (e.g. persuasion or explanation).

The rating scale is here defined with respect to the specific

writing assignment and essays are judged according to the
degree of success with which the writer has carried out the
assignment.
 For each writing task a scoring rubric is created which
includes:
1. The writing task, a statement of the primary rhetorical
trait (persuasive essay, congratulatory letter) elicited by
the task,
2. A hypothesis about the expected performance on the task,
3. A statement of the relationship between the task and the
primary trait,
4. A rating scale which articulates levels of performance,
5. Sample scripts at each level,
6. Explanations of why each script was cored as it was.
 Primary trait scoring is very time and labor intensive as a
scoring guide must be developed for every writing task.
 There is little information on how primary trait scoring
might be applied in second language writing assessment
and testing.
 However it would be useful for parents to know a
description of what their child is capable of doing with the
language.
 HOLISTIC SCALE
 This is the assigning of a single score to a script based on the overall
impression of the script. Each script is read quickly and then judged against
a rating scale, or scoring rubric, that outlines the scoring criteria.

 Positive features
 It is faster to read a script once and assign a single score than to
read it several times, each time focusing on a different aspect of the
writing.
 It is less expensive.
 It is intended to focus the reader’s attention on the strengths of the
writing not on its deficiencies, so writers are rewarded for what
they do well.
 It is more valid than analytic scoring methods, because it reflects
most closely the authentic, personal reaction of a reader to a text.
 DISADVANTAGES
 A single score does not provide useful diagnostic information
about a person’s writing ability
 It does not allow rates to distinguish between various aspects of
writing such as control syntax, depth of vocabulary,
organization and so on.
 This is problematic for second-language writers: some have
excellent writing skills in terms of content and organization but
may have much lower grammatical control. Others may have
excellent grammar but may not know how to organize their
writing in a logical way.
 Holistic scores are not always easy to interpret because rates not
use the same criteria: a certain script might be given a 4 because
of its rhetorical features (content, organization, development)
while another rate might give the same 4 because of its linguistic
features (grammar and vocabulary).
 Superficial characteristics such as length and handwriting seem
to interfere in the score.
EXAMPLE TOEFL WRITING SCORING GUIDE
 ANALYTIC SCORING
 Scripts are rated on several aspects of writing or criteria rather
than given a single score. Depending on the purpose of the
assessment, scripts might be rated on such features as content,
organization, cohesion, register, vocabulary, grammar, or
mechanics.

 Punctuation
According to Jacobs et al (1981) scripts are rated on 5 aspects of
writing; content (30 points), organization and vocabulary (20 points),
language use (25 points), and mechanics (5 points).
 ADVANTAGES
 It provides more useful about diagnostic information.
 More useful in rater training, as inexperienced raters can
more easily understand and apply the criteria in separate
scales than in holistic scales.

 It is particularly useful for second-language learners, who

are more likely to show a marked or uneven profile across
different aspects of writing (script well developed but have
numerous grammatical errors or control of syntax but no
content).

 Can be more reliable than holistic scoring.

 DISADVANTAGES

 It takes longer than holistic scoring, since readers are

required to make more than one decision for every script.
 If scores on the different scales are combined to make a
composite score, a good deal of the information provided by
the analytic scale is lost.
 Rates who are experienced at using a particular analytic
scoring system may actually rate more holistically than
analytically if scores are combined into a single a score.
 MULTIPLE-TRAIT SCALE
 Many of its characteristics have to do more with
procedures of developing and using the scales, rather than
with the description of the scales themselves.
TYPES OF WRITING TAKS
 The main types of writing according to Brown (2010) are imitative,
intensive, responsive and extensive writing.
 DESIGNING ASSESSMENT TASKS: IMITATIVE WRITING

 -Imitative writing are the rudiments of forming letters, words, and

simple sentences.

Tasks in (Hand-) Writing letters, words and punctuation.

 -Handwriting has the potential of becoming a lost art but remains
a skill of a paramount importance within the larger domain of
language assessment.
 TYPES OF TASKS COMMONLY USED TO ASSES THE ABILITY TO
PRODUCE WRITTEN LETTERS AND SYMBOLS.

 - Copying: Directing a test-taker to copy letters or words.

 - Listening cloze selection tasks: Combine dictation with a written script

that has a relatively frequent deletion ratio (every 4 or 5 words). The

test-taker must select the words missing from a list.

 -Picture-cued tasks: Pictures are displayed and the test-takers are told

to write the word that the picture represents.

 -Form completion tasks: Filling a simple form (registration, application

etc.) that asks for name, phone number etc.

 -Converting numbers and abbreviations to words: Numbers are written

(hours of the day, dates or schedules) and the test-takers are directed to

write out the numbers. (proceed with caution)

 SPELLING TASKS AND DETECTING PHONEME-GRAPHEME
CORRESPONDENCES

 -Spelling tests: The teacher dictates a simple list of words, one at a time
followed by the word used in a sentence.
 -Picture-cued tasks: Displaying pictures with the objective of focusing on
familiar words whose spelling may be unpredictable or word pairs.
 -Multiple-choice techniques: Presenting words and phrases in the form of
a multiple-choice task (they choose the word with the correct spelling
among the options)
 -Matching phonetic symbols: Test takers are shown phonetic symbols
and asked to write the correct word alphabetically.
 DESIGNING ASSESSMENT TASKS: INTENSIVE (CONTROLLED)
WRITING

 A form focused writing or guided writing. Students produce

language to display their competence in grammar, vocabulary or
sentence formation and not necessarily to convey meaning for an
authentic purpose.
 DICTATION AND DICTO-COMP (L, W)

 A paragraph is read at normal speed, two or three times, then the

teacher asks students to rewrite the paragraph from the best of
their recollection. (The teacher can also provide a worksheet with
keywords)
 GRAMMATICAL TRANSFORMATION TASKS (R, W)

 -Change the tenses in a paragraph

 -Change full forms of verbs to reduced forms or contractions

 -Change statements to yes/no or wh- questions

 -Change questions into statements

 -Combine two sentences into one using a relative pronoun

 -Change direct speech to indirect

 -Change from active to passive voice

 PICTURE CUED TASKS

 - Short sentences: Write a brief sentence from a drawing of a

simple action
 - Picture description: A more complex picture may be presented
with a list of words to describe the picture. (prepositions)
 -Picture sequence description: A sequence of 3 to 6 pictures
depicting a story line can provide a suitable stimulus for writing
production.
VOCABULARY ASSESSMENT TASK

 The major techniques used to assess vocabulary are defining and

using a word in a sentence.

ORDERING TASKS:
 Ordering or reordering a scrambled set of words into a correct
sentence. Generates writing performance and may tap
grammatical word-ordering rules.
SHORT-ANSWER AND SENTENCE COMPLETION TASKS

 Items may range from very simple predictable to more elaborate

responses. Reading must be necessary to understand the directions
but is not crucial in creating sentences.
 DESIGNING ASSESSMENT TASKS: RESPONSIVE AND EXTENSIVE
WRITING

 A continuum of possibilities ranging from lower-end tasks whose

complexity exceeds intensive or controlled tasks. They are more
open-ended tasks such as writing short reports, essays, summaries,
and responses to texts of several pages or more.
PARAPHRASING (R,L,W)
 Say something in one’s own words to avoid plagiarizing, to offer
some variety in expression. For of informal and formative
assessment. An opportunity to gain positive washback.

GUIDED QUESTION AND ANSWER

 It is a guided question and answer format in which the test
administrator poses a series of questions that serve as an outline of
the emergent written text. Has the pedagogical benefit of guiding a
learner without dictating the form of the output.
PARAGRAPH CONSTRUCTION TASKS (R, W)

Read an effective paragraph, you analyze the ingredients of its

success, and you emulate it. Assessment of paragraph development
may take the form of:
 - Topic sentence writing

 - Topic development within a paragraph

 - Development of main and supporting ideas across paragraphs

 STRATEGIC OPTIONS
 Aside from strategies of free writing, outlining, drafting and revising,
writers need to be aware of the task that has been demanded and focus
on the genre of writing and the expectations of that genre.

 -Attending to task: Four types of tasks are commonly addressed in

academic writing courses: compare/contrast, problem/solution,
pros/cons, and cause/effect.

 -Attending to genre: Common genres, Reports (lab reports, project

summaries, article/book reports), Summaries of
readings/lectures/videos, Responses to readings/lectures/videos,
Narration, description persuasive/argument, and exposition,
Interpreting statistical, graphic, or tabular data, Library research paper