The colloquium starts with a 5-minute contextualization of replication

studies in applied linguistics. Five 15-minute presentations follow,

with 3 minutes for brief clarifications after each one. The final section
of the colloquium consists of a 20-minute open discussion among all
participants and attendees which will both critically address the
nature of the 5 presentations and connect them to future directions in
the area of replication studie s.

LET ME BEGIN WITH A BIG

AND BOLD STATEMENT ON
RESEARCH AND THE ACT OF
RESEARCHING.
Initially, it is our approach to the
apparent facts we read that defines the
“scientific” way of going about
research.
That way is characterized by a
questioning, “skeptical”, critical
approach to what we are told or what
we read or what we discover.
 
One of the reasons for that essential
skepticism is our own “human
fallibility”. We are imperfect beings,
and so it is natural for our ideas or
interpretations to be wrong, or
incomplete, at times. So it is sensible
to ask questions of what we hear and
read and what we discover. 

Our aims in replicating research in

applied linguistics begin with such
questions.
BUT we don’t start out replicating
research because we somehow don’t
believe what we are reading is true.
We re-visit the study because there is
something about it that we think
needs our attention, that merits further
investigation.

In a typical extension or follow up

study, you do NOT set out to
question, nor re-consider in any
critical way the previous study’s
procedures and outcomes as a result
of your new data. Any comparisons
you make between your new study
and studies that preceded it, or
stimulated it, would be incidental –
but not your objective. 

In a replication study your stimulus

and your aim shifts: your stimulus
(and hopefully your publication) now
comes from one earlier study, in
particular, rather than a whole set of
studies.  

Your aim now also changes.

In a REPLICATION we assume
that you have a critical interest in
what actually went on in that
previous study. 
*You may want to return to it because
it is a study everyone still cites as
important, or particularly
significant for your area of research.

*Or maybe it is a study with results

which somehow don’t seem to “fit
in” with other, similar, studies you
have read. 

*It may be that you question the

generalizability of the study’s
findings to other learners, contexts,
or target structures. 

*Or perhaps you just feel the results

need closer inspection.
Whatever the reason, it has aroused
your curiosity to the point where you
want to go back to it, to re-consider
what emerged from it in a new light,
or just….take another look at it. 

So….whatever our reason is for

returning to the study, that return
is predicated on the idea that no
one piece of research (or
researcher!) can include, or control
for, all the many variables that can
affect an outcome.
****
As well as the errors. And the
limitations.

And that’s just fine: our research –

since it contains unavoidable error
or limitations - SHOULD never be
the final say on something. There
will always be something else that
needs to be clarified, tweaked, or
further investigated as a result
of limitations on our work.

*Maybe the choice of participants

was restricted by the study’s context;
*Maybe the randomness desired in
selection just was not available;
*Maybe a more powerful statistical
procedure would have revealed more
detailed information on the data. 
*Maybe the presence of a
relationship or the effect of an
instructional treatment is clear ---- but
we are unsure of its strength. 
All this in turn encourages us to
believe that there MUST BE
MANY STUDIES OUT THERE
that need to be revisited to iron out
such inconsistencies and limitations
and provide us with a clearer picture
of what is actually going on.

But errors, flaws or limitations won’t

be found unless we go out and look
for them. And in AL research at
least, there is evidence we are just
not looking!

So….maybe it doesn’t matter that

much then?

Well….In our section of that world -

applied linguistics and second
language acquisition – A helluva lot
of TIME, EFFORT and above all,
MONEY is spent on establishing best
practices. YOU REALLY WOULD
have thought it was in everyone’s
interest to re-visit research studies –
particularly those that have been key
to the development of language
learning theory and teaching practice.

Here is where today’s colloquium is

designed to help.
We have 5 invited guests who will
present a number of studies which
they feel need replication and where
the results are potentially useful – and
publishable!!
We hope you’ll want to take up the
challenge of replicating them!

Each speaker will firstly situate and

present the original study, its aims
and significance.
They will then go on to suggest
which aspects merit/need
replicating, AND why.

Then a suggested methodology and

analysis for this will be given.

Finally, our presenters will suggest

potential outcomes and how these
might well encourage us to re-think
the previous results or situate them
differently in their respective areas
of interest.

((Before I let them loose on you,

some definitions.
Each presenter today has been asked
to suggest replications of one of
THREE kinds: CLOSE replications,
APPROXIMATE replications, or
CONCEPTUAL replications.

BY CLOSE we mean when our

replication of the previous study
has one variable of interest
modified, added or removed.

BY APPROXIMATE we mean The

next step on the continuum of
confirmatory power and one which
ideally comes AFTER a set of close
replications. 
Two major variables are modified -
ideally those shown to have exerted
some influence on results in the
previous close replications.

BY CONCEPTUAL we mean when

we are looking to study the relevance
or generalizability of the theory
or hypotheses of the original study
by, for example, using a completely
different database, a related
collection procedure
(e.g., observation vs. self-report),
using a different operational
definition, methodology, or method
of analysis.))

So without more ado, I call first

upon…..

1 Hossein Nassaji:
Investigating the role of recasts versus
prompts: Replications of three interactional
feedback studies
2 Ana Péllicer-Sánchez:
Replication in L2 vocabulary learning and
teaching
3. Andrea Revesz:
Replication in task-based language teaching
research
4. Shawn Loewen:

5 Charlene Polio:
Wrap up:

After so much positivity,

forgive me if I preface our final
section with some words of
warning.

As with much of modern

terminology, currently we
might be guilty of using a term
like “replication” in a rather
ambiguous sense and one
which limits our progress.

Back to Classical Latin,

then…..“replicare”, we are
referring to the idea of “going
back” and “unfolding” the
previous study. Unfortunately,
in the late 19th C. it acquired a
secondary meaning from Late
Latin of “repeat”. I say,
“unfortunately” because it is
that meaning which many
understand by a replication
these days.

To have a successful replication

does NOT mean we have to
find and report the same
outcomes – to repeat them.

But to think replication success

is to be measured only in terms
of getting the same results as
your original study would
misunderstand the aim here.

It is worth remembering now

and again that the science we
all carry out is COMMUNAL.
The research we undertake
now owes much to the theory
and research carried out by our
predecessors. ((Where we are
now in our AL travels is largely
down to the signposts put
down by other linguists
stretching back to Ancient
Greece. ))

We are all participating in a

science that builds both on the
successes AND lack of success
of those before us.

It follows that undertaking the

kind of work we encourage you
to do today does not have as
its aim, therefore, to somehow
get the same or similar results
as the previous researcher did.
Luckily, we aren’t pure
scientists. Our job is not
eventually to seek out and
present our community with
facts. Our aim by looking
backwards to previous work is
to better understand its
inevitable uncertainties.

By doing this we understand

that RESULTS are there to be
considered, understood within
their contexts and limitations,
confirmed, disputed but
inevitably refined by the work
which follows them.

It is therefore always in all our

interests as applied scientists
to have our work re-visited –
and it is well to remember that
results which do not seem to
fit in with other work are just
as fascinating – and worthy of
publication - as those which
do. Such outcomes need
explaining but can generate
just as much excitement for
the potentially new
phenomena they exhibit and
the breakthrough they might
represent.
From the discovery of
Superglue in the second World
War and, more recently, the
strange side effects
researchers observed when
using a new heart drug,
eventually known as Viagra,
unsuccessful replications can
lead to just as interesting
effects.

Our colloquium today fits to

perfection with this year’s
AAAL Challenge: LOOKING
BACK; MOVING AHEAD.
Replicating key research is
designed to do exactly this.
But, you know, after today’s
session, we hope to have
convinced you that we need
to do a bit more looking back
before moving ahead too
quickly.