EXPERIMENTS

© LOUIS COHEN, LAWRENCE MANION AND

KEITH MORRISON
 STRUCTURE OF THE CHAPTER
• Randomized controlled trials
• Designs in educational experiments
• True experimental designs
• Quasi-experimental designs
• Single-case ABAB design
• Procedures in conducting experimental research
• Threats to internal and external validity in
experiments
• The timing of the pre-test and the post-test
• The design experiment
• Internet-based experiments
• Ex post facto research
 CAUSALITY
Experiments are held up to be able to identify causality
through control and manipulation of variables.

Examine the effect of an independent variable on a

dependent variable.

Identifying the effects of causes by implementing

interventions in a controlled environment.

Held up to be able to offer explanations for outcomes.

 INDEPENDENT AND
DEPENDENT VARIABLES

Parents
Development and Teaching and
planning community learning

School
Effectiveness

Professional Culture and

Management Leadership
development climate
 RANDOMIZATION
• Random sampling of a population and random allocation
to either a control or an experimental group.
• Randomization allows for the many additional
uncontrolled and, hence, unmeasured variables that may
be part of the make-up of the groups in question.
• Randomization operates the ceteris paribus condition (all
other things being equal), assuming that the distribution
of extraneous variables is more or less even and perhaps
of little significance.
• Randomization strives to address Holland’s (1986)
‘fundamental problem of causal inference’, which is that
a person may not be in both a control group and an
experimental group simultaneously.
 CONCERNS IN EXPERIMENTS
• It may not be possible or desirable to isolate, control and
manipulate variables and people under laboratory conditions.
• The ‘real’, social world is not the antiseptic, sealed, artificial
world of the laboratory. The social world is far more complex.
• Cannot assume that a single cause produces a single effect.
• Measurements are of averages, overlooking distributions,
outliers, intervention-response differences, within-group
differences, between-group differences and sub-sample
differences.
• The setting affects the outcomes.
• Limited generalizability in practice.
 HOW TO JUDGE ‘WHAT WORKS’

• Null hypothesis significance testing (which is

problematical)
• Effect size
• Statistical power
• The subtraction approach
• Considering rival explanations
• The contingency approach
• The ethical dimension
BLIND AND DOUBLE-BLIND EXPERIMENTS

Blind experiment: participants do not know to which

group they are assigned.

Double blind experiment: neither the researcher nor

the participants know to which group the participants
are assigned.
 KINDS OF EXPERIMENT
Laboratory experiments (controlled, artificial conditions):
• Pretest-post-test control and experimental group
• Two control groups and one experimental group pretest-post-test
• Post-test control and experimental group
• Post-test two experimental groups
• Pretest-post-test two treatment
• Matched pairs;
• Factorial design;
• Parametric design;
• Repeated measures design;

Field experiments (controlled conditions in the ‘real world’):

• one-group pretest-post-test;
• non-equivalent control group design;
• time series

Natural experiments (no control over real world conditions)

 FEATURES OF A TRUE EXPERIMENT
• Random selection of sample.
• Random allocation of sample to control or experimental groups.
• Identification and isolation of key variables.
• Control of the key variables.
• Exclusion of any other variables.
• Special treatment (the intervention) given to the experimental
group (i.e. manipulating the independent variable) whilst holding
every other variable constant for the two groups.
• Ensuring that the two groups are entirely separate throughout the
experiment (non-contamination).
• Final measurement of outcomes to compare control and
experimental groups and for differences from the pre-test results
(the post-test).
• Comparison of one group with another.
 Stages in an
Randomly assign subjects
experiment
to two matched groups:
control and experimental group

Conduct pre-test

Isolate and control variables,

exclude other variables

Administer intervention to
experimental group

Conduct post-test and compare

control and experimental groups
 ‘TRUE’ EXPERIMENTAL DESIGN

EXPERIMENT Intervention EXPERIMENT

PLUS
Matched on
pre-test
Isolate, Post-test
Random group control and
assignment manipulate
variables

CONTROL CONTROL
 MEASURING EFFECTS

Average causal effect (A):

(A) = (E1E2)  (C1C2)

where:
– E1 = post-test for experimental group;
– E2 = pre-test for experimental group;
– C1 = post-test for control group;
– C2 = pre-test for control group.
 CAMPBELL’S AND STANLEY’S NOTATION

• X represents the exposure of a group to an experimental

variable or event, the effects of which are to be measured.
• O refers to the process of observation or measurement.
• Xs and Os in a given row are applied to the same persons.
• Left to right order indicates temporal sequence.
• Xs and Os vertical to one another are simultaneous.
• R indicates random assignment to separate treatment
groups.
• Parallel rows unseparated by dashes represent comparison
groups equated by randomization, while those separated by
a dashed line represent groups not equated by random
assignment.
 CAMPBELL’S AND STANLEY’S SYMBOLIC
REPRESENTATION OF ‘TRUE’ EXPERIMENTS

RO1 X O2

RO3 O4
(Campbell, D. T. and Stanley, J. (1963)
Experimental and Quasi-experimental Designs for
Research on Teaching. Boston: Houghton Mifflin
Co.)
TWO CONTROL GROUPS AND ONE EXPERIMENTAL
GROUP PRE-TEST–POST-TEST DESIGN

Experimental RO1 X RO2

Control1 RO3 RO4
Control2 X RO5
 THE POST-TEST CONTROL AND
EXPERIMENTAL GROUP DESIGN
Experimental R1 X O1
Control R2 O2
 THE POST-TEST TWO EXPERIMENTAL
GROUPS DESIGN
Experimental1 R1 X1 O1
Experimental2 R2 X2 O2
THE PRE-TEST–POST-TEST TWO TREATMENT
DESIGN
Experimental1 RO1 X1 O1
Experimental2 RO3 X2 O4
THE TRUE EXPERIMENT ONE CONTROL AND
TWO EXPERIMENTAL GROUPS

Experimental1 RO1 X1 O1
Experimental2 RO3 X2 O4
Control RO5 O6
 THE PRE-TEST TWO TREATMENT DESIGN

Experimental1 RO1 X1 O1
Experimental2 RO3 X2 O4
 MATCHED PAIRS DESIGN
• Step One: Measure the dependent variable.
Stage 1

• Step Two: Assign participants to matched pairs, based on the scores and measures
established from Step One.
Stage 2.

• Step Three: Randomly assign one person from each pair to the control group and the
other to the experimental group
Stage 3.

• Step Four: Administer the intervention to the experimental group and, if appropriate, a
placebo to the control group. Ensure that the control group is not subject to the
Stage 4 intervention.

• Step Five: Carry out a measure of the dependent variable with both groups and
compare/measure them in order to determine the effect and its size on the dependent
Stage 5 variable.
 FACTORIAL DESIGN
Performance in an examination may depend on availability of
resources and motivation for the subject studied
INDEPENDENT LEVEL LEVEL LEVEL
VARIABLE ONE TWO THREE

Availability limited moderate high

of resources availability (1) availability (2) availability (3)

Motivation little moderate high

for the motivation (4) motivation (5) motivation (6)
subject
studied
9 combinations: 1+4; 1+5; 1+6; 2+4; 2+5; 2+6; 3+4; 3+5; 3+6
Motivation for mathematics
100

80

60 Males
40 Females

20 Factorial designs
must address the
0 interaction of the
15 16 17 18 independent
variables.
Age
Difference for motivation in mathematics is not constant between
males and females, but varies according to age of participants: an
interaction effect (age and sex).
 PARAMETRIC DESIGN

• Participants are randomly assigned to groups whose

parameters are fixed in terms of the levels of the
independent variable that each receives.

• Parametric designs are useful if an independent

variable has different levels or a range of values
which may have a bearing on the outcome
(confirmatory research) or if the researcher wishes to
discover whether different levels of an independent
variable have an effect on the outcome (exploratory
research).
 REPEATED MEASURES
• Participants in the experimental groups are tested
under two or more experimental conditions.
• The order in which the interventions are sequenced
may have an effect on the outcome (e.g. the first
intervention may have an influence – a carry-over
effect – on the second, and the second intervention
may have an influence on the third).
• Early interventions may have a greater effect than
later interventions.
• Repeated measures designs are useful if it is
considered that order effects are either unimportant
or unlikely.
 REPEATED MEASURES
(two groups receiving both conditions)
Group 1 Group 1
With no With
intervention intervention

Matched on pre-test
Post-test
Random allocation to
groups

Group 2 Group 2
With With no
intervention intervention
 Noise condition No noise condition

Independent      
groups Sara Rob Peter Jane Jack Jim
     
Joan Susan John Lyn Sally Alan
 Noise condition No noise condition

     
Sara Rob Peter Jane Jack Jim

Repeated      
measures Joan Susan John Lyn Sally Alan
     
Jane Jack Jim Sara Rob Peter

     
Lyn Sally Alan Joan Susan John
QUASI-EXPERIMENTS: NON-EQUIVALENT
CONTROL GROUP DESIGN
• Pre-experimental design: the one-group pre-test–
post-test
Experimental O1 X O2
• Pre-experimental design: the one-group post-test
only design
Experimental O1
• The post-tests only non-equivalent groups design
Experimental O1
----------
Control O2
QUASI-EXPERIMENTS: NON-EQUIVALENT
CONTROL GROUP DESIGN
• The pre-test–post-test non-equivalent
group design
Experimental O1 X O2
----------
Control O3 O4
PROCEDURES IN CONDUCTING EXPERIMENTS
1 • Identify research problems

2 • Formulate hypotheses

3 • Select appropriate levels at which to test the independent variables

4 • Decide which kind of experiment to adopt

5 • Decide population and sampling

6 • Select instruments for measurement

7 • Decide how the data will be analyzed

8 • Pilot experimental procedures

9 • Carry out the refined procedures

10 • Analyze results

11 • Report the results

A TEN-STEP MODEL FOR CONDUCTING EXPERIMENTS
Step 1 • Identify the purpose of the experiment.

Step 2
• Select the relevant variables.

Step 3
• Specify the level(s) of the intervention (e.g. low, medium high intervention).

Step 4 • Control the experimental conditions and environment.

Step 5 • Select appropriate experimental design.

Step 6 • Administer the pretest.

Step 7 • Assign the participants to the group(s).

Step 8 • Conduct the intervention.

Step 9 • Conduct the post-test.

Step • Analyze the results.
10
 PROCEDURES IN CONDUCTING EXPERIMENTS:
HYPOTHESES
• Null hypothesis (H1).
• Alternative hypothesis (H0).
• Direction of hypothesis: states the kind of difference
or relationship between two conditions or two
groups of participants.
• One-tailed (directional): e.g. ‘people who study in
silent surroundings achieve better than those who
study in noisy surroundings’.
• Two-tailed (no direction): e.g. ‘there is a difference
between people who study in silent surroundings
and those who study in noisy surroundings’.
 OPERATIONALIZING HYPOTHESES
• Hypothesis: ‘people who study in quiet surroundings
achieve better than those who study in noisy
surroundings’.
• What do ‘work better’, ‘quiet’ and ‘noisy’ mean?
Define the operations:
– ‘work better’ = obtain a higher score on the
Wechsler Adult Intelligence Scale
– ‘quiet’ = silence
– ‘noisy’ = CD music playing
• Operationalized hypothesis: ‘people who study in
silence achieve a higher score on the Wechsler Adult
Intelligence Scale than those who study with CD
music playing’.
DIRECTIONAL AND NON-DIRECTIONAL HYPOTHESES

• People who do homework

Directional without the TV on produce
(one-tailed) better results than those who
do homework with the TV on.

Non- • There is a difference

directional between work produced in
(two-tailed) noisy or silent conditions.
 HISTORY MATURATION
TESTING

DIRECTION
OF CAUSALITY

INSTRUMENT-
THREATS TO ATION
TYPE I AND
VALIDITY AND
TYPE II
ERRORS RELIABILITY

EXPERIMENTAL
MORTALITY
OPERATIONAL-
IZATION
CONTAMIN-
ATION REACTIVITY
 TIMING OF PRE-TEST AND POST-TEST
Pre-test
• As close to the start of the experiment as possible (to avoid contamination
of other variables).

Post-test
• As close to the end of the intervention as possible.

Too soon a post-test

• Misses long-term/delayed effect and only measures short-term gain (which
may be lost later).

Too long a time lapse before a post-test

• Becomes impossible to determine whether it was a particular independent
variable that caused a particular effect, or whether other factors have
intervened since the intervention, to produce the effect.
 ADVANTAGES OF INTERNET-BASED
EXPERIMENTS
Attraction over laboratory and conventional
experiments

Greater generalizability because of their wider

sampling

Demonstrate greater ecological validity as they

are often conducted in settings which are familiar
to the participants and at times suitable to the
participants

Have a high degree of voluntariness

FOUR TYPES OF INTERNET-BASED EXPERIMENTS

Those that present static printed materials

(e.g. printed text or graphics)

Those that make use of non-printed

materials (e.g. video or sound)

Reaction-time experiments

Experiments that involve some form of

interpersonal interaction
 INTERNET-BASED EXPERIMENTS

Check download speeds and time, anticipate problems

of different browsers and platforms.

Can experience greater problems of dropout than

conventional experiments.
 EX POST FACTO RESEARCH
• Co-relational and criterion groups designs
• Characteristics of ex post facto research
• Occasions when appropriate
• Advantages and disadvantages of ex post facto
research
• Designing an ex post facto investigation
• Procedures in ex post facto research

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
TWO APPROACHES TO EX POST FACTO RESEARCH

• Commence with subjects who differ on an

independent variable, for example, their years of
study in mathematics, and then study how they
differ on the dependent variable, for example, a
mathematics test.
• Commence with subjects who differ on the
dependent variable (e.g. their performance in a
mathematics test) and discover how they differ on a
range of independent variables, for example, their
years of study, their liking for the subject, the
amount of homework they do in mathematics.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 EX POST FACTO RESEARCH
AND INDEPENDENT VARIABLES
Differing on the independent variable:
Presence of independent variable
Absence of independent variable
Degrees of independent variable

Investigate Effect on the dependent variable

Same on the independent variable(s)

Investigate Effect on the dependent variable

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 EX POST FACTO RESEARCH AND DEPENDENT
VARIABLES
Differing on the dependent variable

Differing on independent variables:

Presence of independent variables
Investigate Absence of independent variables
Degrees of independent variables

Same on the dependent variable

Differing on independent variables:

Presence of independent variables
Investigate Absence of independent variables
Degrees of independent variables
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 CO-RELATIONAL AND CRITERION GROUP STUDY

Co-relational study (causal research)

• to identify the antecedents of a present condition.
• collect two sets of data, one of which will be
retrospective, with a view to determining the
relationship between them.
Criterion group study (causal-comparative research)
• to discover possible causes for a phenomenon
being studied.
• Compare the subjects in which the variable is
present with similar subjects in whom it is absent.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
TWO CAUSES AND TWO EFFECTS IN
CRITERION GROUP STUDY
EFFECT POSSIBLE CAUSE

Effective Presence of collegial

teaching curriculum planning

Ineffective Absence of collegial

teaching curriculum planning
Two criterion groups:
(a) Presence of collegial planning
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
(b) Absence of collegial planning
CHARACTERISTICS OF EX POST FACTO RESEARCH

In ex post facto research the researcher takes the

effect (or dependent variable) and examines the data
retrospectively to establish causes, relationships or
associations, and their meanings.

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
EX POST FACTO RESEARCH IS USEFUL WHEN . . .

. . . it is impossible, impractical, costly or unethical to

conduct an experiment;
. . . it is not possible to select, control and manipulate
the factors necessary to study cause-and-effect
relationships directly;
. . . the control of all variables except a single
independent variable may be unrealistic and artificial;
. . . the independent variable lies outside the
researcher’s control.

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
ADVANTAGES OF EX POST FACTO RESEARCH
• Useful where the more rigorous experimental
approach is not possible.
• Useful to study what goes with what and under
what conditions.
• Useful where the setting up of the latter would
introduce a note of artificiality into research
proceedings.
• Useful where simple cause-and-effect relationships
are being explored.
• It can give a sense of direction and provide a source
of hypotheses that can subsequently be tested by
the© 2018
more rigorous experimental method.
Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 DIFFICULTIES IN EX POST FACTO RESEARCH
• Direction of causality difficult to establish: what caused
what.
• Lack of control of the independent variable or variables.
• Impossible to isolate and control every possible variable,
or to know with absolute certainty which are the most
crucial variables.
• Randomization impossible.
• Can provide support for any number of different, even
contradictory, hypotheses.
• Correlation does not equal cause.
• Lack of control: the researcher cannot manipulate the
independent variable or randomize her subjects.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
DISADVANTAGES OF EX POST FACTO RESEARCH
• One cannot know for certain whether the causative
factor has been included or even identified.
• It may be that no single factor is the cause.
• A particular outcome may result from different causes on
different occasions.
• It is not possible to disconfirm a hypothesis.
• Classifying into dichotomous groups can be problematic.
• As the researcher attempts to match groups on key
variables, this leads to shrinkage of sample.
• Conclusions may be based on too limited a sample or
number of occurrences.
• It may fail to single out the really significant factor(s).
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 DESIGN AND PROCEDURES IN AN EX
POST FACTO INVESTIGATION
Identify the problem area to be investigated.
Formulate a hypothesis to be tested or questions to be answered.

Make explicit the assumptions on which the hypothesis and

subsequent procedures will be based.

Review of the research literature will follow to ascertain the kinds of

issues, problems, obstacles and findings disclosed by previous
studies in the area.
Plan the actual investigation:
• identify the population and samples;
• select and construct techniques for collecting data;
• establish categories for classifying the data.

Describe, analyse and interpret the findings.

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 DESIGN AND PROCEDURES IN AN EX POST
FACTO INVESTIGATION
• Define the problem and survey the literature
Stage 1

• State the hypotheses and the assumptions or premises on

Stage 2 which the hypotheses and research procedures are based

• Select the subjects (sampling) and identify the methods for

Stage 3 collecting the data

• Identify the criteria and categories for classifying the data

Stage 4 to fit the purposes of the study

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 DESIGN AND PROCEDURES IN AN EX POST
FACTO INVESTIGATION
• Gather data on those factors which are always present in which the
given outcome occurs, and discard the data in which those factors
Stage 5 are not always present

• Gather data on those factors which are always present in which the
given outcome does not occur
Stage 6

• Compare the two sets of data (i.e. subtract the former (Stage Five)
from the latter (Stage Six), in order to infer the causes that are
Stage 7 responsible for the occurrence or non-occurrence of the outcome

• Analyze, interpret and report findings

Stage 8
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors
 CONTROLS IN EX POST FACTO RESEARCH
Match the subjects in the experimental and control groups
where the design is causal-comparative.

Build the extraneous independent variables into the design

and then use an analysis of variance technique.

Select samples that are as homogeneous as possible on a

given variable.

State and test alternative hypotheses that might be plausible

explanations for the empirical outcomes of the study.

© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors