The Experimental

Method
Dadang Sudana
(Source: Nunan, 1992)
The context of experimentation

• “Experiments are carried out in order to explore the

strength of relationships between variables.”
• “A variable is anything which does not remain
constant,” such as language proficiency, aptitude,
motivation, etc.
• var·i·a·ble /ˈverēəb(ə)l/
• Independent variables are those which are expected to influence
the other variables, the dependent variables.
• Dependent variables are the ones upon which the independent
variables are acting.

• Variables can be measured by different scales of measurement:

nominal, ordinal, interval, ratio.
• Variables can also be classified according to the
types of scale on which they are measured.
• Types of Variables Used in Language Research
Type Examples
Nominal L1 background: e.g., Arabic,
Spanish, etc.
Ordinal Rank on a test of grammar: e.g.,
first, second, third, etc.
Interval Numerical score on standardized
language test
• Ratio scales are of little interest in applied linguistics,
because variables such as language proficiency do not exist
as absolute quantities; therefore, ratio scales do not have to
be dealt with further in the discussion.
• Consider a situation in which an experiment might be an
appropriate way of gathering data: you want to find out if
your innovative listening materials for a low level learners
will improve a way better your students’ listening skills
compared with using traditional ones.
• There are several ways to find out and collect evidence if the
innovative listening materials could help learners improve
their listening skills: survey the students through interviews
and questionnaires, and obtain their subjective impressions,
ask a colleague to observe the teaching and make an
ethnographic record, or doing an experimental study.
• If more accurate quantitative results is expected, an experimental
study can provide evidence.
• There are several types of experimental studies, the most rigorous
one is the true experimental study, represented in the following
figure:
E (random): T1 X T2
C (random): T1 O T2

• It is not always possible to randomly assign students to

experimental and control groups, using intact group design
instead. In such a case, a quasi or pre-experiment rather than true
experiment has been implemented.
• You’ve done all those steps mentioned previously to
find out if your innovative listening materials work and
at the end of the term the groups are retested (post-
test).
• You work out the mean, or average, for each group and
obtain the following:
Control group : 58
Experimental group : 62
• Are you in the position to claim that the innovative
materials are superior to the traditional materials? Not
yet. You need to use ‘statistical inference’
The logic of statistical inference
• Several important concepts
• Population: all cases, situations, or individuals who share one
or more characteristics.
• Sample: a subset of individuals or cases from within
population.
• Mean: the average of a set of scores
• Standard deviation: the extend to which a set of scores varies
in relation to the mean
• Variance: it is the square of the standard deviation.
• Normal distribution: a mathematical and ideal concept how
the frequencies of individual scores are distributed in a
population.
 The Principles of Probability

Normal Curve
• When comparing two means, the appropriate test is a t-test.
• When comparing more than means, or more than two
groups, the appropriate test is the F-test or ANOVA.
• When comparing frequency of an event to find out if it
correlates with another event, the chi-square test is the
statistical test to be used.
• Correlation is another important family of statistical test to
estimate the degree of association between two variables.
Questions and Tasks
• Write out your own definitions of the following terms: construct, frequency
table, central tendency, variable, variability, mean, variance, standard
deviation, population, sample.
• Imagine that you are about to carry out a detailed investigation of the foreign
reading skill of a group of high school students. The school has provided you
with a great deal of information on each student, including the following. For
each of the variables listed identify the most likely type of scale it would
represent.
Potential variable Most likely type of scale
a. Sex __________________
b. Age __________________
c. Class rank __________________
d. Nationality __________________
e. First language
f. Scores on proficiency test __________________
g. Rank on class test __________________
h. Amt. of time spent studying French __________________
i. Whether parents are native speakers
of the language __________________
• Which statistical procedures would you employ
for the following situations? Indicate the
appropriate procedure by placing a number in
the bracket.
1. --- t-test (tests differences between means)
2. --- ANOVA (tests differences between multiple means, and
tests for possible interaction between means)
3. --- correlation (tests strength of association between
variables)
4. --- chi-square (tests association between frequency)
a. This study investigated the relationship between scores on
a traditional standardized test of oral proficiency and a new
test of communicative speaking ability. The investigators
wanted to investigate whether who scored well on the
traditional test did well on the innovative test and vice
versa. [ ]
b. This study examined the effect of content familiarity on the
reading comprehension of secondary level second
language learners. Subjects were given two test passages
to read, and their comprehension was tested on a multiple
choice test. The test passages were analyzed using the Lix
readability formula, which showed they were both at the
same level of linguistic difficulties. However, one passage
was on a topic familiar to the students while the other was
on an unfamiliar topic. [ ]
c. In this investigation, the researcher wanted to find out
whether the native-like production of word-final consonants
and consonant clusters varied according to the type of
communicative task in which the learner is engaged. She
collected data from a single subject as he took part in three
different tasks – free conversation, oral reading of continuous
text, and elicited imitation of words and short passages. [
]
d. A study was established to test the hypothesis that field-
dependent learners would benefit more from learning
grammar inductively, while field-independent learners would
benefit from learning deductively. One group of field-
dependent subjects and one group of field-independent
subjects were taught inductively. A second group of field-
dependent subjects and a second group of field-independent
subjects were taught deductively. Pre- and post-treatment
scores on a standardized test of grammar were administered.
[ ]