PhD

(ELE) RM@LM, Sep 2019

Quantitative Methods in ESL Research
Worksheet Three: Analysing Learner Data with Research Variables

In this worksheet we will look at what constitutes learner language data and what are the
standard ways to analyze them in research on Applied Linguistics and ESL/EFL learning.

1

 PhD (ELE) RM@LM, Sep 2019



Read the following text (a detailed summary of a research paper) and answer the
questions that follow.


SOURCE: Skehan, P., Foster, P. (1997).Task type and task processing conditions as
influences on foreign language performance. Language Teaching Research, 1,
185-211



I. Background of the study
A study by S & F (1997) deals with task based performance, a strategy primarily followed
in CLT classrooms. The study was conducted on a group of foreign language learners of
English studying in a local college in Britain.

Language teachers make use of task based instruction (TBI) to create optimal conditions
for language learning. This method in an application of the CLT approach and is based on
the premise that language is learnt through use. Through specially designed tasks that
simulate real life contexts ESL/EFL learners are provided opportunities to communicate in
the target language.

In this study, S & F used a set of three tasks to address the following research questions:
(i) Do different task properties influence language performance?
(ii) Do conditions under which specific tasks are done have any impact on learner
performance?

While (i-ii) are specific RQs, this study sets out to understand the larger role of tasks in
language classrooms, specifically do tasks provide opportunities to learn a language? If
tasks do promote language learning, then do learners experience any language growth by
participating in tasks?

Evidence for learning language in this study is obtained from analysis of three performance
measures: COMPLEXITY, ACCURACY & FLUENCY (CAF measures). This paper follows from
the cognitive paradigm to language learning and measures language performance using
countable constructs of CAF.


The study
The study examines the extent to which learners’ performance varies on the basis of two
task factors/varaibles: (a) task properties and (b) task conditions.

To test the first factor, three types of tasks were used:

A. A personal task:

2

 PhD (ELE) RM@LM, Sep 2019


In this task, learners had to exchange personal information on a specific topic:
Describe one feature/event that has pleasantly or unpleasantly surprised you about
life in Britain. [For this task, learners could use background knowledge and produce
content and so this task was the least difficult in terms of access to knowledge and
its presentation. So it was cognitively less demanding.]

B. A narrative task:
For this task, each learner pair/dyad were given two comic stripes by using which
they had to tell stories. One learner in each pair was supposed to use his/her stripe
to tell a story and express whether they story was funny or sad and then the other
learner would do the same. [For this task, learners had to creatively use language
and the content had to be recreated from the pictures. So it was more cognitively
demanding than the first task.]

C. A decision task:
In this task, dyads were given three letters to an agony aunt of a magazine. The pairs
had to decide which advice would be most suitable for each candidate (whoever
wrote the letters). [This was very high on cognitive demands as the content had to
made up from learners’ understanding of the letters and negotiation about which
advice would be best suited for each candidate.]

All the three tasks were done in pairs. In all the three tasks, learners had to pay attention
to both content and form.

One of the assumptions of cognitive learning theories is that both content and form cannot
be attended to at the same time because of working memory limitations. Therefore,
learners who are more proficient will devote more time to produce content and form will
be automatic. But learners who are less proficient will struggle with form and will have
very less time to deal with content. Hence, the latter group is likely to make more
grammatical/structural errors and be less fluent. This study also examines the extent to
which the above mentioned assumptions are valid.

To test the second factor, the learners were divided into two groups. One group got a
planning time of 10 minutes before each task and another group got no planning time
before performing on any of the three tasks.

Language performance and growth was recorded for each task and was analysed in terms
of three countable measures – complexity, accuracy and fluency.

Subjects: 40 learners, with diverse L1 backgrounds, aged between 18 and 25 years,
enrolled in a part-time English course in a local college in Britain served as subjects for the
study. Data was collected after two months from the commencement of the course so that
the learners were quite comfortable in participating in tasks under classroom condition.
The learners were divided into four classes – Class A and B got no planning time, Class C

3

 PhD (ELE) RM@LM, Sep 2019


and D got planning time. Each learner dyad had to perform on all the three tasks but not at
the same time. The order of participating in each task was different in each class.


Data analysis:
Learner performance was measured on three aspects:
Fluency: Count the number of words of each learner from each task.

Accuracy: Count the number of error-free clauses in each task (only
errors in syntax, morphology and word-order were
considered; vocabulary errors were not considered unless
nonexistent words in English were used).

(Language) Complexity: Count the length of communication-unit (c-unit) and number
of clauses per c unit. Each c-unit is one utterance consisting of
a simple clause or subordinating clauses with the main clause
(Broke, 1986). All coordinating c-units are counted as separate
c-units.


Let us look at an example of c-unit measurement and Fluency in the following exchange:

Julie: Oh I was supposed to go to the market today! I simply forgot. [2 c units, 2 clauses]

Eve: That’s all right, maybe you could go today after you finish your work here. [2 c units; 1+2 =3 clauses]

Julie: Ah, yes, of course! I could do that! [2 c units, 2 clauses]

Eve: If you go get me some roses, won’t you? [2 c units, 2 clauses]

Julie: Oh yes! Sure! [2 c-units, 2 partial phrases]


Julie: Total No of words: 24; No of c units: 6 MLU: 24/6 = 4
Clauses per c-unit: 6/6 = 1

Eve: Total No of words: 25; No of c units: 4 MLU: 25/4 = 6.25
Clauses per c-unit: 5/4 = 1.25

Other points:
• Calculate all shortened phrases as number of words (That’s = 2 words).
• Count all short phrases as separate c-units based on style of oral discourse (e.g., Oh
yes! = 1 c-unit)




4

 PhD (ELE) RM@LM, Sep 2019


II. Results & discussion
Overall findings
a. When planning time is given, performance improves - it is more fluent,
accurate, and has higher language complexity.

b. Fluency, accuracy and language complexity – all three cannot be achieved at
the same level of excellence at one point of time. These three capabilities
compete with each other in the brain/working memory.

c. Task properties influence performance selectively:

i. Tasks that are cognitively easier (those that have a neat internal structure
like personal task A) if planned results in greater accuracy;

ii. Tasks that are cognitively complex (those that do not have a clear
structure like decision making task C) if planned gives rise to greater
language complexity.

Specific findings (Descriptive Statistics)
Task types Number of error- Clauses per c-
free clauses unit
(Accuracy) (Complexity)
Unplanned 60% 1.29
Planned 70% 1.44
mean 65% 1.36
Unplanned 53% 1.25
Planned 69% 1.32
mean 61% 1.29
Unplanned 58% 1.35
Planned 69% 1.97
mean 64% 1.68
Table 1: Descriptive statistics (adapted from Table 4, p.197 of the original study)


III. Conclusion

In case of performance on all the three tasks, planners have made significantly lesser
number of pauses, have more error-free clauses and improved in using clauses per c-unit.
So, this finding proves the first hypothesis to be true.

Of the three tasks, the third one was the most cognitively challenging task. When given
planning time for this, learners did not improve so much in terms of accuracy but they
gained in terms of language complexity. So they paid more attention to content of the task
and not so much on grammar and syntax. This shows that when cognitive demand of a task
is higher and learners are given more time to process information (or plan), they gain

5

 PhD (ELE) RM@LM, Sep 2019


because they are able to use more complex structures. In contrast, when a task is medium
on cognitive demands like the second task, the narrative task, if given planning time,
learners are able to improve on accuracy aspect and not so much on the complexity feature.


The results of this study show a rather complex relationship between task properties,
conditions of task performance and nature of language growth. Firstly, it shows that though
planning time shows positive effects for fluency, accuracy and complexity – but they are not
equally high for every task. Secondly, it shows that if planning time is given and when
cognitive demands of a task are higher, learners tend to improve on complexity while when
the cognitive demands are lower they tend to make less grammatical errors.


How to transcribe utterances?
Transcribe it as (each are 1 c Count no of clauses in each
If a speaker says:
units) c-unit
I know, it’s so crazy! I know. 1
it’s so crazy. 1
I don’t know, maybe we 1
I don’t know.
should wait until after dinner.
maybe we should wait until 1

after dinner.
you know what, let’s put the 1
you know what?
toys away first.
let’s put the toys away first. 1
there you go, it’s all fixed. there you go. 1
it’s all fixed. 1
you see, all three things are 1
the same color as they are you see?
shades of red.
all three things are the same 2

color as they are shades of red.

6

 PhD (ELE) RM@LM, Sep 2019



Reflection questions to be done in class:

Work in your respective groups.

1. List the different types variables (I.V, D.V, learner, task, performance, method
variables) used in the study.

2. What kind of learner data does the study deal with? Where do you get evidence of
this?

3. How is data measured? Which scales of measurement are used?

4. Look at Table 1.1. Do you observe any change in performance? List what are the
changes/growth related to.

5. Do you think the CAF measures can help you differentiate between different levels
of performers? Why? If you had to give feedback to your students based on these
measures would that be possible for you to do so? Why?

6. From the findings of the study, do you think ESL teachers can use some elements in
the classroom? List some predictions that teachers can make about ESL learners.

7. We generally think that language proficiency of a person is stable at a point of time.
Hence standardized language proficiency tests show us snapshots of students’
proficiency levels. But is this notion challenged in this study? Explain how.

Data analysis task (group task for submission by 30th September: soft copy and hard
copy)

Here is a set of original writing data from four Indian young learners (2 from grade III and
2 from grade V, aged 9 years to 12 years). These are their responses on a picture based
story-writing task as part of ESL writing research.

Analyse each of these four scripts using the following measures:

Fluency: count of total number of words
Complexity: count the number of t-units (in writing we use t-units while in oral discourse
we use c-units. The count of both the concepts are same) and MCL: total word
count/total c-units in each script.
Accuracy: Number of error free c-units per script.


Present your analysis in a tabular format.

7

 PhD (ELE) RM@LM, Sep 2019


Question to respond: If you do measure learner language data like this will it be possible
for you as a researcher to give feedback to the learners? Why?

DATA SET

STUDENT One: S13M

A lady went to a picnic with her pet to a beautiful garden Then it was suddenly raining it was
suddenly raining, then she saw that the water was going into a house. Then she was floating into
a dirty water, then she saw a wodden piece! then she got an Idea of sitting on it and going. After
sometime it became dark, the she saw a Temple on the way. She was happy

STUDENT Two: S23M

A lady went for picnic with his pet to beautiful garden. Suddenly it started raining.because of the
rain the garden was full of water.because of the rain she to floated and fall in the water. She saw
the wood pice was floating near by her she got the idea of that she can go the wooden piece
water to the land after reaching to land was very happy.

STUDENT Three: S35M

It was spring season.

One day a grandma with her cute little cat was going to beautiful garden for picnic. Suddenly the
sky became dark and it was filled with clouds. It started raining. The wind blowed very strongly.
The grandma and her belongings became wet. She ran to her house, but she was shocked to see
that her house was floating in the river which was beside her house. She suddenly slipped and
fell into the river. She tried to save her life, she saw a piece of wood floating in water. She tried
to climb the wood. As soon as she climbed the wood piece, it started moving. It reached a
temple. She was surprised and happy.

STUDENT Four: S45F

One day a Grandma with her cute little cat were going to a beautiful Garden for picnic. Suddenly
the sky became dark and it was filled with clouds. It started raining. The wind blowed very
strongly. The Grandma and her belongings became wet. She ran to her house, but she was
shocked to see that her house was floating in the river which was beside her house. Suddenly
slipped and fell into the river. She tried to save her live, she saw a piece of Wood floating in
water. She tried to climb the wood. As soon as she climber the wood log it started moving in
flood. Suddenly she reached near a temple.

She was surprised and happy to find that her life was saved.