Assignment 2

SCORING INSTRUMENT
ANALYSIS

Students: Natalia Cuevas

Victoria Tapia
Course: EFL Assessment Methods & Approaches.

July 22, 2019, Concepción

I. Analyze the level of coherence between the tasks presented and
the scoring instruments used. Refer to 3 strengths and 3
weaknesses of the scoring instruments and suggest ways to
improve them. Refer to type of scoring instrument and its
appropriateness to assess that ability.

Task 1 Scoring instrument

1.Answer both compulsory
questions and then choose one of
the optional questions. Question 1 2 ____________
1. (Compulsory)
In both “A Beneficiary” and “To Kill a
Mockingbird” the fathers are very
important. Compare and contrast the
three father figures in the two texts. Content /
Provide examples Interpretation
2. (Compulsory) and analysis
Analyse Scout and Jem’s attitude (10)
towards Boo Radley (Do not write a
description, write an interpretation that
reveals themes)
3. (Compulsory)
In “Hills Like White Elephants” and “A Coherence /
Beneficiary” the Girl and Charlie have Use of
to make decisions. Compare the way
Language (5)
they face this process and the
pressures they receive from other
characters.

Use of text (5)

1. Level of Coherence and Appropriateness

The instrument was identified as a type of rubric, an unfinished analytic scale

specifically. In this case, the instrument and the task concur in terms of coherence.

The task demands writing and reading comprehension, where the questions of the

task are mainly based on critical thinking skills; write an interpretation; compare;

contrast. Bearing that in mind, it is considered coherent and appropriate to use a

scoring tool with the same level of complexity of the skills that are being measured.

It would be difficult to evaluate this type of test with a checklist or a rating scale

because their design features are simpler.

Overall, the scoring instrument relates to the requirements of the task, despite its

poor and ambiguous development, which will be analyzed in the following section.
2. Strengths and Weaknesses of the Scoring instrument

A) WEAKNESSES

 There is an absence of the transparency quality because the instrument is

incomplete. As it can be observed, the scoring instrument includes

dimensions but lacks descriptors. Therefore, students would not know in

detail the requirements to attain the top score.

 As previously stated, the instrument has not been designed properly;

consequently, it cannot be considered as a reliable tool. Proof of that is the

third dimension “use of text”, which without explanation or descriptors it is

unlikely to determine with certainty what the dimension is measuring.

Besides that, without the criteria in any of the dimensions, there are more

chances for the rater to be biased towards the students’ work.

 In terms of washback quality, the instrument lacks of it, because students

would not be able to draw a conclusion about their progress through the

results of the rubric.

B) STRENGTHS

 In the preceding section, it was asserted that the scoring instrument was

quite unreliable. However, there is a certain level of reliability regarding the

score given to each dimension, where the one related to the main objective

of the task (content/analysis) has been assigned with a higher score,

compared to the two other dimensions. Hence, the instrument becomes

slightly more objective and fairer for students.

 The scoring instrument shows a level of practicality because is an

inexpensive way of assessment.

 If we assume that the blank spaces of the scoring instrument are for

delivering feedback, it would be possible to indicate that the instrument

presents a degree of washback.

 3. Suggestions

Although the instrument is rather inefficient and unreliable, it is possible to make

improvements without dismissing the whole features of the assessment instrument.

Firstly, it is fundamental to add the corresponding criteria for each dimension,

including the gradual levels of performance as well. Secondly, the dimension “use

of text” could be replaced by a more substantial component such as “supporting

information and examples”, “organization” or “identification of facts”. Thirdly, it

would be more valid to divide the first dimension into two different ones; one would

be “content” and the other would be “analysis/interpretation”. In that manner,

content would focus on whether the student provides relevant information or

wanders from the actual question. Whilst “analysis/interpretation” would take into

account the fulfillment of the analysis.

 Scoring instrument
Task 2
Criteria Excellent Intermediate Insufficient

By the end of the unit 2 points 1 point 0 point

Knows words in Knows all Knows Knows 3
the students will be able
relation with the the words some the or less
to name and describe
topic. studied in words words
animals orally.
relation studied in studied in
with the relation with relation
topic. the topic. with the
topic.
Knows short Knows all Knows Knows
structures in the short some of the only one
relation with the structures short of the
topic. studied en structures short
the class. studied en structures
the class. studied en
the class.
Uses words Generally Sometimes Never
and short uses short uses short uses short
structures in structures structures structures
real situations during during during
of games en games en games en
communication. the class. the class. the class.
Shows Shows a Shows little Do not
motivation in high motivation show
the activities of motivation to motivation
the class to participate to
participate in the class. participate
in the in the
class. class.
 1. Level of Coherence and Appropriateness

In this second task, the instrument is a mix between a holistic and an analytic

scale. It can be claimed that the coherence and appropriateness between both

elements is questionable. Before going into the reason for the previous statement,

it might be important to point out that the task seems to be the final aim of a lesson

plan unit. Continuing with the analysis, the task objective is clearly stated and

demands observable actions from students; name and describe orally. In contrast,

the scoring instrument presents more complexity than the task itself, apart from the

fact that it has some inadequacies. Perhaps, the main problem lays on the

instrument’s format chosen by the rater more than the areas that were intended to

assess.
 2. Strengths and Weaknesses of the Scoring Instrument

A) WEAKNESSES
 There are issues with the instrument’s transparency because there is great

inaccuracy in the way dimensions and criteria were conveyed. Presuming

that the instrument is a type of rubric, the dimensions should be written as

the constructs that the teacher needs to measure, avoiding ambiguous

verbs such as knows, uses and shows, which can be found in the analyzed

tool.

The same problem occurs with descriptors, where the verbs used (knows -

shows - uses) are neither objective nor measurable. As a consequence,

students would not know in clear terms the way they should demonstrate

their learning.

 The instrument is not valid because it holds more components to assess

than the actual requirements of the task. The problem is in the following

parts:

 Uses words and short structures in real situations of communication.

 Shows motivation in the activities of the class.

It can be assumed that the first point refers to pragmatics; nevertheless, the

task or unit aim does not have a defined context (e.g.; conversation at a

certain place, e-mail to friend, etc.) within the instructions. For this reason, it

is thought that it would be wrong to evaluate such component.

 The second point relates to the attitude of students during different classes,

an aspect valid to measure at the end of a unit. Nonetheless, it should be

assessed with other scoring instrument, apart from the second language

performance of the student.

 The reliability of the tool is sort of weak, as a consequence of its low level

of transparency and validity. In the point of transparency as a weakness, it

was mentioned that dimensions and criteria were not developed properly,

using vague verbs. This negative aspect could lead to having problems

concerning the rater reliability.

 B) STRENGTHS

 Despite the instrument being more unsatisfactory than adequate, it is

possible to assert that there is a degree of washback quality. Students

might obtain an idea of their performance with the final results. In addition,

the teacher would also get information about the class’ achievements.

However, those results might be confusing and not very accurate according

to the weaknesses identified.

 Regarding scoring, there is reliability in the way the points were distributed

for each level of performance; excellent (2 points), intermediate (1 point),

insufficient (0 points). A student graded with the lowest level of performance

(insufficient) in every dimension, will obtain 0 points and a very low mark; in

the scenario that the student did not make efforts or did not accomplish what

was asked. If the insufficient level would have had 1 point, that same

student would get points regardless of his or her poor performance, which is

unfair. Therefore, there is congruence on the scoring decision.

 The tool is practical, in the sense that it is easy to score, affordable for the

raters in charge and the information of accomplishments is summarized for

teachers, parents and students.

 3. Suggestions
Considering that the task is the final aim of a unit, the instrument could be turned

into a holistic scale, rearranging the chart with the criteria that the tool already has.

For that, it would be necessary to remove the “criteria” section to vertically display

the descriptors of each level of performance and the addition of one more level.

In order to obtain a more precise measurement, the second option is to redesign

the instrument as an analytic scale. Based on the weaknesses mentioned, the

following dimensions are suggested:

 Use of vocabulary (instead of knows words in relation with the topic)

 Use of structures (instead of knows short structures in relation with the

topic)

 Use of the language in class (instead of uses words and short structures in

real situations of communication)

 Enthusiasm and collaboration with peers (instead of shows motivation in the

activities of the class)

With the previous dimensions, it would be necessary to write the corresponding

criteria using observable actions.

A final idea is to use a rating scale in the context that the assessment is formative.

Rating scales lack precision and information if the evaluation has to be graded with

a mark. However, it is a practical suggestion in order to obtain a general notion of

students’ progress at the end of a unit.

 II. Study the tasks proposed and design or adapt (indicate source)
scoring instrument(s) to assess the ability the tasks elicit.
2. a) This is an end-of-unit writing assessment
Situation
You just received an e-mail message from your friend Amelia in Australia and she
tells about all of the things she must do at school. You answer the e-mail message
right away and tell Amelia what you have to do. Write an e-mail message to Amelia
and tell her what you have to do for school. You may want to mention when,
where, or for whom you do these things, whether you do them with a friend or a
family member, and whether or not you like doing these things.
Task
Be sure to write about the things you have to do using a verb or expression of
obligation. Write a short e-mail message (at least 7 sentences) to Amelia and tell
her what you have to do this weekend for school.

Adapted from DEVELOPING SPEAKING AND WRITING TASKS • MINNESOTA LANGUAGE PROFICIENCY
ASSESSMENTS 2013, Regents of the University of Minnesota
Scoring Instrument

Category 7-6 5-4 3-2 1-0

Grammar & Writer makes Writer makes Writer makes Writer makes
spelling no errors in 1-2 errors in 3-4 errors in more than 4
(conventions) grammar or grammar grammar errors in
spelling. and/or and/or grammar
spelling. spelling. and/or
spelling.

Content The e-mail The e-mail The e-mail The e-mail

focuses totally focuses mainly partially focuses in
on the topic on the topic focuses on the irrelevant
and the and the task’s topic. information for
requirements requirements. the task.
asked for the
task.

Format Uses the Use the Uses the Doesn’t follow

accurate/corre structure for structure for the structure
ct structure for writing an e- writing an e- for writing an
writing an e- mail, but mail but e-mail.
mail. misses one misses some
part. parts.

Length The e-mail is 7 The e-mail is The e-mail is The e-mail is

or more 6-5 sentences. 5-4 sentences. less than 4
sentences. sentences.

Ideas Ideas were Ideas were Ideas were Ideas were not
expressed in a expressed in a somewhat organized. it
clear and pretty clear organized, but was very
organized manner, but were not very difficult to
manner. It was the clear. It took figure out what
easy to figure organization more than one the e-mail was
out what the e- could have reading to about.
mail was been better. figure out what
about. the e-mail was
about.

Adapted from RubiStar.com

2. B) Speaking task
Situation
It’s the first week of school, and when you walk into class, you see the new
exchange student
sitting by herself. You introduce yourself to Christine, who says that she is from the
United States, and that she has just arrived in Chile. You have been studying
English, and you’d like to make her feel welcome.
Task
Probably Christine wonders about you: where you’re from, how old you are, what
you’re studying, and so on. Tell her about yourself.
Thinking time: 15 seconds
Speaking time: 60 seconds
Adapted from DEVELOPING SPEAKING AND WRITING TASKS • MINNESOTA LANGUAGE PROFICIENCY
ASSESSMENTS 2013, Regents of the University of Minnesota
Scoring Instrument

Very good (4) Good (3) Average (2) Poor (1)

Content Student Student Student Student

indicates all indicates 3 of 4 indicates 2 of indicates only
(name, age, (name, age, 4 (name, age, one (name,
career, and career, and career, and age, career,
place of living) place of living) place of living) and place of
living)

Fluency Student Student Student Student is

presents no presents some hesitates extremely
hesitation and hesitation and frequently and hesitant. she/he
she/he does not she/he rarely she/he often has to search
need to search has to search has to search for words most
for words. for words. for words. of the time

Accuracy Student Student Student Student

presents a clear presents a clear presents a presents
pronunciation pronunciation pronunciation unclear
that helps that helps that sometime pronunciation.
comprehension. comprehension. interference She/he makes
very few Some with too many
pronunciations pronunciation comprehension. pronunciation
mistakes are mistakes are Regularly mistakes that
made (1-3 made (4-6 makes interfere with
mistakes) mistakes) pronunciation comprehension
mistakes (7- 10 (11 or more
mistakes) mistakes)

Time Student speaks Student speaks Student speaks Student speaks

for 60 seconds. between 50-40 between 40-30 for less than 30
seconds. seconds. seconds.