Assessing Writing

Nylvie Loire Collamat

• State a rationale for assessing writing as a separate skill as well as a skill
that integrates with reading and possibly other skills.
• Discern the overlap between assessing writing as an implicit, unanalyzed
ability and its explicit, form-focused counterpart.
• Incorporate performance-based assessment into your own assessment
instruments.
• Develop assessments that focus on one or several micro- and macroskills of
writing within a specific genre.
• Design assessments that target one or more modes of performance, ranging
from imitative production to extensive writing.
Genres of
Writing
Academic Writing
• Papers and general subject
reports
• Essays and compositions
• Academically focused journals
• Short-answer test responses
• Technical reports – lab reports
• Theses and dissertations
Job-related Writing
• Messages – phone messages
• Letters/emails
• Memos – interoffice
• Reports – job evaluations, project reports
• Schedules, labels and signs
• Advertisements and announcements
• Manuals
Personal Writing
• Test messages, tweets, e-mails, letters, greeting cards, invitations
• Messages and notes
• Calendar entries, shopping lists, and reminders
• Financial documents
• Forms, questionnaires, medical reports, immigration documents
• Diaries and personal journals
• Fiction – short stories and poetry
Types of Writing
Performance
Imitative
• Learners try to master the mechanics of writing – writing
letters, words, punctuation marks, and brief sentences.
• Form is the primary focus.

Intensive (controlled)
• Meaning and context are important in determining the
correctness and appropriateness.
• Assessment tasks are more concerned with a focus on form
and are strictly controlled by the test design.
Responsive
• Learners are required to perform at a limited discourse level
• Connecting sentences into a paragraph
• Creating logically connected sequence of two or three paragraphs.

Extensive
• Successful management of all the processes and strategies of
writing for all purposes
• Focus on achieving a purpose, organizing and developing ideas
logically, using details to support or illustrate ideas, demonstrating
syntactic and lexical variety, engaging in the process of creating
multiple drafts to achieve a final product.
Microskills and
Macroskills of
Writing
Microskills of Writing
1. Produce graphemes and orthographic patterns of English
2. Produce writing at an efficient rate of speed to suit the
purpose
3. Produce acceptable core of words and use appropriate
word order patterns
4. Use acceptable grammatical systems (e.g., tense,
agreement, pluralization), patterns, and rules
5. Express a particular meaning in different grammatical forms
6. Use cohesive devices in written discourse
Macroskills of Writing
1. Use of rhetorical forms and conventions of written discourse
2. Appropriately accomplish the communicative functions of
written tests according to form and purpose
3. Convey links and connections between events and
communicate such relations as main idea, supporting idea,
new information, given information, generalization, and
exemplification
4. Distinguish between literal and implied meanings when
writing
5. Correctly convey culturally specific references in the context
of the written text
Macroskills of Writing
6. Develop and use a battery of writing strategies, such as
accurately assessing the audience’s interpretation, using
prewriting devices, writing with fluency in the first drafts,
using paraphrases and synonyms, soliciting peer and
instructor feedback, and applying feedback when revising
and editing
 Designing Assessment
Tasks: Imitative Writing
 • Copying – there is nothing innovative or
Tasks in(Hand-)writing modern about directing a test-taker to
copy letters or words.
Letters, Words, and
Punctuations
 Tasks in (Hand-)writing Letters,
Words, and Punctuations
• Listening Cloze Selection Tasks –
combination of dictation and a
written script that has a relatively
frequent deletion ratio.
• Purpose: not to test spelling
but give practice writing.
Tasks in (Hand-)writing Letters,
Words, and Punctuations
• Picture-Cued Tasks – Familiar
pictures are displayed, and
test-takers are told to write the
word that the picture
represents.
Tasks in (Hand-)writing Letters,
Words, and Punctuations
• Form Completion Tasks– a
variation on pictures is the use of a
simple form that asks for name,
address, phone number, and other
data.
 • Converting Numbers and Abbreviations to
Words – reasonably reliable method to stimulate
Tasks in (Hand-)writing hand-written English.
Letters, Words, and • Lacks authenticity and more of a reading task
than writing task
Punctuations • Converting abbreviations to words is more
authentic.
Spelling Tasks and Detecting
Phoneme-Grapheme
Correspondence
• Spelling Tests – the teacher
dictates a simple lists of words,
one word at a time; then uses the
word in a sentence and repeats
the sentence.
• Scoring emphasizes – correct
spelling
 Spelling Tasks and Detecting
Choose the correct word for each photo. Phoneme-Grapheme
Encircle your answer. Correspondence
• Picture-Cued Tasks – pictures
are displayed with the
objective of focusing on
familiar words whose spelling
may be unpredictable.
Boot or book read or reed rain or rein
• Opportunity to present
challenging words and
word pairs: boot/book,
read/reed, bit/bite, and so
on.
Spelling Tasks and Detecting
Phoneme-Grapheme
Correspondence
• Multiple Choice Techniques–
presenting words and phrases
in the form of a multiple-choice
task risks crossing over into the
domain of assessing reading.
• Serve as formative
reinforcement of spelling
conventions.
Spelling Tasks and Detecting
Phoneme-Grapheme
Correspondence
• Matching Phonetic Symbols – if
students have become familiar
with the phonetic alphabet, they
could be shown phonetic symbols
and asked to write the correctly
spelled word alphabetically.
• Mixed opinion regarding the value
of using phonetic symbols at the
literacy level.
• Helps students perceive the
relationship between phonemes
and graphemes.
Designing Assessment Tasks:
Intensive (Controlled) Writing
Intensive (Controlled) Writing
• Display writing – students produce language
to display their competence in grammar,
vocabulary, or sentence formation and not
necessarily to convey meaning for an
authentic purpose.
Designing Assessment Tasks:
Intensive (Controlled) Writing
• Dictation and Dicto-Comp
• Dictation – simply the rendition in
writing of what one hears aurally, so
it could be classified as an imitative
type of writing, test-taker’s
performance centers on correct
spelling.
• Dicto-comp – a paragraph is read at
normal speed, usually two or three
times, then the teacher asks
students to rewrite the paragraph
from the best of their recollection.
Designing Assessment Tasks:
Intensive (Controlled) Writing

• Grammatical Transformational
Tasks – the practice of making
grammatical transformations –
orally or in writing
• Devoid of any meaningful
value.
• Easy to administer and
therefore practical
• Taps into a knowledge of
grammatical forms that are
performed through writing.
 • Picture-cued Tasks – a variety of picture-cued
controlled tasks have been used in English
classrooms around the world.
• Main advantage – giving of non-verbal means
to stimulate written responses.
• Short Sentences – the test-taker writes a brief
sentence
• Picture Description – test-takers are asked to
describe the picture. The test is specific on how
to describe the picture. (E.g., use the following
prepositions to describe the picture below: on,
over, under, next to, around)

Designing Assessment Tasks:

Intensive (Controlled) Writing
Designing Assessment Tasks: Intensive
(Controlled) Writing
• Picture Sequence Description – a
sequence of three to six pictures
depicting a story line can provide a
suitable stimulus for written
production.
• Pictures must be simple and
unambiguous
• Simple form of verb must be
written below the picture if the
criterion is the grammatical form
of the verb
Designing Assessment Tasks:
Intensive (Controlled) Writing
• Picture Sequence Description:
• A few different correct
responses can be given
• If criteria in this task are lexical
and grammatical choices – a
rating scale should be designed.
• Most common vehicle for a deliberate
focus on vocabulary is reading.
• Major techniques used to assess
vocabulary:
• Defining
• Using a word in a sentence

Vocabulary
Assessment Read (2000) suggested several types of items to
assess basic knowledge of the meaning of a word,
collocational possibilities, and derived
Tasks morphological forms. The example above focused
on the word interpret.
• It is form-focused
• Collocations and morphological variants
• lexical choice is one of several possible
components of the evaluation of extensive
writing.
Ordering
Tasks • Challenging to test-takers whose learning
styles do not dispose them to logical-
mathematical problem solving.
• Sentences are kept very simple, with perhaps no
more than four or five words.
• Only one possible sentence can emerge.
• Students have practiced the technique in class.
Short-Answer and
Sentence-Completion Tasks
• Heavy participation of reading
performance
• Ranges from very simple and
predictable to somewhat more
elaborate responses
• Reading-writing connection is apparent in the
first three items but has less effect in the last
three.
• Scoring on a scale of 2-1-0 scale – most
appropriate way to avoid self-arguing about
the appropriateness of a response.
Issues in Assessing
Responsive and
Extensive Writing
 • Creates opportunity for test-takers to offer
an array of possible creative responses
Responsive and • Learners can choose vocabulary, grammar,
and discourse but with conditions.
Extensive • Criteria include discourse and rhetorical
Writing conventions, paragraph structure and
connecting paragraphs
• Developing sequence of connected ideas
The ART and SCIENCE and empathizing with audience
of composing –
real writing • Short reports – structured formats and conventions
• Responses to the reading of an article or story
• Summaries of articles or stories
• Brief narratives or descriptions
• Interpretation of graphs, tables, and charts
Issues
Authenticity
• Content and validity should be ensured.
• Established the basic building blocks of writing
• Setting of authentic real-world contexts
• Teacher – facilitator or coach and less of an instructor
• Assessment – formative, not summative, and positive washback
Issues
Scoring
• Attuned – form and function of the test
• Impact and effectiveness
• How are you to score such creative production, some of which is more artistic
than scientific?
Issues
Time
• Not necessarily constrained by time
• Freedom to produce multiple drafts
• Continuous refinement
• Relies on essential drafting process for its ultimate success
Designing Assessment
Tasks: Responsive and
Extensive Writing
Paraphrasing
• Ensure learners understand the importance of paraphrasing
• Criterion: convey similar or the same message; grammar and
vocabulary second
• Informal and formative assessment
• Opportunity for teachers and students to gain positive
washback
Guided Question and Answer
• Series of questions are made that essentially serves as an
outline of the emergent written text

Scoring: analytic or
holistic

Outline: self-created; guides the

learner through a presumably
logical development of ideas that
have been given.
Paragraph Construction Tasks
Good writers are often good readers.
Writing is the art of emulating what one reads.

Topic Sentence Writing – assessment of the effectiveness

of a topic sentence consists of:
• Specifying the writing of a topic sentence
• Scoring points for its presence or absence
• Scoring and/or commenting on its effectiveness in stating the topic
Paragraph Construction Tasks
Good writers are often good readers.
Writing is the art of emulating what one reads.

Topic Development Within a Paragraph – four criteria to

assess the quality of a paragraph:
• Clarity of expression of ideas
• Logic of the sequence and connections
• Cohesiveness or unity of the paragraph
• Overall effectiveness or impact of the paragraph as a whole
Paragraph Construction Tasks
Good writers are often good readers.
Writing is the art of emulating what one reads.

Development of Main and Supporting Ideas Across Paragraphs –

elements to consider when evaluating a multiparagraph essay:
• Addressing the topic, main idea, or principal purpose
• Organizing and developing supporting ideas
• Using appropriate details to undergird supporting ideas
• Showing facility and fluency in the use of language
• Demonstrating syntactic variety
Strategic Options
Developing main and supporting ideas is the goal for the writer
attempting to create an effective text, whether a short one- to two-
paragraph one or an extensive one of several pages.
Attending to Task
• Task has been designed by the teacher; the writer must fulfill the
criterion of the task.
• Compare/contrast
• Problem/solution
• Pros/cons
• Cause/effect
 Strategic Options
Attending to Genre
Extent by which constraints and
opportunities of the genre are exploited
Standardized Tests of Responsive
Writing
• Standardized tests –
TOEFL, MELADB, PTE,
IELTS
• A prompt
• Require the test-taker to
respond within a time
limit
• Scored with a rating scale
Standardized Tests
of Responsive
Writing
• PTE Essay Scoring Guide
• Not intended to mirror the
real world
• They rend to elicit a sample
of writing performance that
will be indicative of a
person’s writing ability in the
real world.
Standardized Tests of Responsive
Writing
• Standardized tests – indicators not a fail-safe, infallible measures
of competence. Secondary measures should be employed for
learners who:
• Are on the threshold of a minimum score
• May be disabled by highly time-constrained or anxiety-producing
situations
• Could be culturally disadvantaged by a topic or situation
• Have had a few opportunities to compose on a computer (in the case of
computer-based writing tests)
Scoring Methods for Responsive
and Extensive Writing
 Advantages
• Fast evaluation
• Relatively high inter-rater reliability
• Scores represent standards easily
interpreted by lay persons
• Applicability to writing across many
different disciplines

Disadvantages
• One score masks differences across
the subskills within each score
• No diagnostic information is available
(no washback potential)
Holistic Scoring • The scale may not apply equally well
to all genres of writing
• Raters need to be extensively trained
to use the scale accurately
• Provides little washback into the writer’s further stages of writing.
• Focuses on the principal function of the text thus offers some feedback potential
• Classroom evaluation of learning is best served - 6 major elements or writing are scored
• Allows learners to home in on weaknesses and capitalize on strengths

Analytic Scoring
• Analytic profile suggested by Jacobs,
Zinkgraf, Wormuth, Hartfiel, and
Hughey (1981)
• The order in which the five categories
are listed may bias the evaluator
toward the greater importance of
organization and logical
development as opposed to
punctuation and style
Analytic Scoring
Analytic Scoring
 Primary-Trait Scoring
• Focuses on “how well students can
write within a narrowly defined
range of discourse” (Weigle, 2002,
p. 110)
• Assigns a score based on the
effectiveness of the text’s
achieving that one goal
• Example: persuasive essay – the
score depends on the
accomplishment of the function.
Primary-trait score assesses:
• Accuracy of the account of the
original (summary)
• Clarity of the steps of the procedure
and the result (lab report)
• Description of the main features of
the graph (graph description)
• Expression of the writer’s opinion
(response to an article)
Beyond Scoring: Responding to Extensive Writing
The rich domain of assessment lies beyond
mathematically calculated scores in which a developing
writer is coached from stage to stage in a process of
building a storehouse of writing skills.
To give maximum benefit of assessment, it is important to
consider:
• Earlier stages (from freewriting to the first draft or two)
• Later stages (revising and finalizing)
• Involvement of self, peers, and teacher at appropriate
steps in the process
Assessing Initial Stages of the Process of
Composing
Assessing Later Stages of the Process of
Composing
1. It is virtually impossible to isolate any one of the four skills,
perhaps with the exception of reading; at least one other mode
of performance will usually be involved. Don’t underestimate the
power of the integration of skills in assessments designed to
target a single skill area.
2. The variety of assessment techniques and item types and tasks is
virtually infinite in that there is always some possibility for
creating a unique variation. Explore those alternatives but with
some caution.
3. May item types that have been presented in these last four
chapters target language forms as one of the possible
assessment criteria. Make sure you are clear about the extent to
which your objective is to assess form or function or both.
• Look at the list of micro- and Macroskills of writing. In pairs,
with each person assigned to a different skill, brainstorm some
tasks that assess those skills. Share your findings to the rest of
the class.