5.

Experimental research designs

1) Formulation of the problem 5) Collecting data

2) Theoretical framework 6) Analyzing data
3) Research questions/statements 7) Interpretation of the data
and hypotheses: about elements (feedback to theory)
and variables
8) Making a report
4) Research plan/design
(predictions)

Research elements → sampling mechanisms

Variables → measuring variables, measurement instruments and quality of an
instrument: validity and reliability.
Research design → two categories in quantitative research: experimental versus
correlational research designs.

It depends on the type of research question:

Descriptive/relational Causal/evaluative
Correlational research Experimental research

1) Relation between cause (IV) and consequence (DV). Both designs allow for a
relation between IV and DV.
2) Cause precedes the consequence in time. A change in IV causes a change in
DV (after some time) and it can take a long time (e.g., lung cancer caused by
smoking) or effect can be instantaneous (e.g., pushing a light button). It´s occurs
in experimental and, depends on the type of data collection, correlational
research.
The type of data could be:
- Cross-sectional (collecting all data at the same moment), that´s not possible
to determine whether cause precedes the consequence.
- Panel data (collecting data at different moments in time), where it can be
determined but nuisance variables should be cancelled out.

40
 QUANTITATIVE RESEARCH METHODS

3) No nuisance variables that may explain the relation between cause and
consequence. We have to exclude effect of/control for nuisance variables it’s
very important in experimental research. The researcher can also control for
nuisance variables not specified in advance by matching. It is to some extent
possible in correlational research, but only for nuisance variables known in
advance (you have to measure them in order to control for them).

The goal of an experiment is to discover a change in DV that is only caused by a change

in IV. We can do this manipulating IV and measuring his effect on DV.
“DV only caused by IV” if we control nuissance variables by standardization or
randomization (+ matching).

5.1. MANIPULATION OF THE CAUSE (IV)

There are two values (conditions) of IV at least:

• Experimental condition: pill with working substance.
• Control condition: pill without working substance.
Subjects are assigned at random to one of the conditions.
Control: no pill or pill without working substance. Sometimes effect has been
observed, it is not clear what caused it (pill (cf. placebo effect) or working substance).
The difference between control and experimental condition should only be one factor
(‘no pill’ shows the difference between the experimental and control condition becomes
larger (pill + working substance)).
Manipulation checks to see if manipulation worked as intended. For example, looking
if subjects take their pill by fear induction (Schachter experiment).

5.2. STANDARDIZATION

His goal is to control nuisance variables. They may influence the IV and/or DV.
Manipulation is the only difference between experimental and control condition.
Change in DV only can be attributed to a change in IV (no alternative explanations for
the change in DV).
There are three aspects that should be controlled (kept constant):
1) Circumstances. Place of experiment, time of the day, amount of sleep.
2) Manipulation/intervention. Should be the same for all subjects (e.g.,
instructions of experimenter) and only the intervention may influence the DV,
not the way an intervention is implemented.
3) Measurement instruments. DV should be measured in the same way in both
conditions and differences in measurement instruments may not be the cause
for the change in DV. For example, in Schachter experiment: the same
questionnaire is administered in both groups.

41
 5.3. RANDOMIZATION OF THE SUBJECTS

Over conditions (between-subject design)

Random assignment
of subjects
Over sequences (within-subject design)

BETWEEN-SUBJECTS DESIGNS

Condition 1 (experimental)
Random assignment to
conditions
Condition 2 (control)

WITHIN-SUBJECTS DESIGNS

Condition 1 (experimental) Condition 2 (control)

Random assignment
to sequences
Condition 2 (control) Condition 2 (control)

Between-subjects designs

Randomization of the subjects over conditions (sequences of running through the

conditions) should be at random.
The consequence is that nuisance factors that may influence IV and/or DV are equally
distributed over control and experimental group, so they cannot explain the change in
DV. For example, “controlling” for IQ by random assignment there should be no
difference in IQ between both groups/sequences.
It works for known and unknown nuisance variables correlational research only known
nuisance variables.
Assumption of equal distribution over conditions may be violated for a particular
randomization. For example, all smart people assigned to experimental group. It occurs
especially with a small sample size (n).
Solutions could be making a pretest which: measure nuisance variables at the start of
the experiment, check whether they are equally distributed across conditions or should
be known in advance (important to incorporate them in your theory).
Also, matching (systematically equating), which guarantee that conditions are equally
regarding one or more features (e.g., gender, IQ). It´s very important to know which
matching variable(s) we should use → importance of theory.
What is a good matching variable? One that is related to IV and/or DV. The value of DV
at the beginning of the experiment (before the manipulation) should guarantee that
groups do not differ with regard to the DV before the manipulation.
It has implications for data analysis (statistics).

42
 QUANTITATIVE RESEARCH METHODS

a) Posttest only

Condition 1 (experimental) Posttest

Random
assignment to
conditions
Condition 2 (control) Posttest

30
Difference between groups for the posttest says if
the groups were equal at the beginning (for DV).

math score
20
Randomization tries to make groups equal at the old method
beginning, but this can go wrong. The solution is 10 new method
pretest-posttest design (controlling for
differences in DV at the start). 0
posttest

b) Pretest / posttest

Pretest Condition 1 (experimental) Posttest

Random
assignment to
conditions
Pretest Condition 2 (control) Posttest

One advantage is related with the difference between groups for the posttest.
Randomization worked when at beginning groups were equal (for DV). Differences in DV
at posttest cannot be caused by differences in DV at pretest.
The other one is related with the change in scores
30 from pretest to posttest, which can be determined
25 (progression in DV caused by the intervention).
20
When intervention is effective, progression in DV
math score

old method should be larger for experimental than for control

15
group.
10 new method

5 There may also be progression in the control

group, learning effect when same test is
0
pretest posttest administered twice. Cannot be determined in a
posttest only design.

No difference between groups at posttest, but

experimental group was lower at pretest. In 90
math score

conclusion: intervention is effective.

70
Why no pretest: can cause a sensitization effect.
Pretest influences the posttest because the pretest 50
may alert participants as to the purpose of the pretest posttest
experiment and consequently influence their new method old method
behaviour. More a problem for attitudes and
opinions than for cognitive abilities.

Example: campaign against prejudices

Manipulation: watching a DVD about prejudices or other DVD
AV: test about prejudices (pretest and posttest)
Result: larger decrease in prejudices in experimental group
Alternative explanation: social desirability

43
 • During intervention experimental group can guess the research topic (prejudices)
• This may influence their posttest score: pretest sensitizated the subjects
• No effect of DVD
Solution: Solomon four group design to determine the effect of the pretest: compare 4
groups. Such studies are scarce
Condition 1
Group 1 Pretest Posttest
pretest / (experimental)
posttest
Condition 2
Group 2 Pretest Posttest
(control)
Randomization
Condition 1
Group 3 Posttest
(experimental)
posttest only:
no sensitization Condition 2
Group 4 Posttest
(control)

Other solutions could be:

• Try to disguise the pre-test by embedding it in some other task carrying it out in
a different context.
• We can increase the length of the interval between the pre-test and the
manipulation: pre-test is less likely to have an effect on the post-test and a big
interval of time will/may result in a reduced practice effect.
• If the effects of the manipulation were relatively short-lived, give the ‘pre-test’
after the post-test. For example, the effects of alcohol on errors test the
participants a couple of hours later, so effects of alcohol away.

Within subject design

Condition 1 Condition 2
Posttest Posttest
(experimental) (control)
Randomization
Condition 2 Condition 1
Posttest Posttest
(control) (experimental)

Advantages Disadvantages
No influence of differences between Nuissance variables (that have nothing to
conditions. All (the same) subjects in all do with your manipulation) may influence
conditions: both conditions are equal. the second posttest.
Because group differences cannot explain
Fatigue effects. Participants may become
the effect of the manipulation and cannot
progressively more tired/bored with task.
be guaranteed with between-subject
So number of mistakes may be greater in
designs.
the second than in the first condition
Larger number of subjects in each because they are tired.
condition. Twice as large for the same n
Learning effects. Participants may
(number of subjects) as compared to
become better at the task. So, number of
between-subject design. More powerful
mistakes may be less in the second than
experiment, one can better demonstrate
in the first condition, because they have
the difference between conditions (cf.
learnt to respond more accurately.
power of statistical tests). Within-subject
designs are in general more powerful and
efficient than between-subject designs.

44
 QUANTITATIVE RESEARCH METHODS

Solution to disadvantages is contrabalancing.

Total experimental: n=10 Total control: n=10

Experimental: n=10
Between subjects Randon assignment
design to conditions
n=20 Control: n=10

Total experimental: n=20 Total control: n=20

Experimental: n=10
Control: n=10
Randon n=10
Within subjects
assignment to
design
sequences Experimental:
Control: n=10
n=10 n=10
n=20

Number of Number of DV Number of IV

conditions
Goal Compare multiple Effect on different Interactions among
experimental or dependent iv
control conditions variables
Design Iv with 3 or more 2 or more dvs 2 or more iv
levels subject variable(s)
Factorial design

0% Posttest
8% Posttest
Randomization
16% Posttest
20% Posttest

45
 6.1. MULTIPLE IV - FACTORIAL DESIGNS

E.g.: effect of punishment/reward on aggression → four groups (2 x 2 design).

Punishment Agression

Randonization Reward Agression

n=40 +
matching Punishment + reward (combined
Agression
effect)

No punishment/no reward Agression

Advantages (over performing separate experiments)

• More efficient: smaller sample size to detect same effect
• Better generalization: multiple factors operate at same time
• Possibility of interaction effects. Effect of iv on dv depends on the levels of
another iv.
• Possibility of interaction effects. A main effect is the influence of a variable acting
on its own – not in combination with any other variable. Interactions can only
occur when there are two or more independent variables. An interaction is
basically a combination of levels of two (or more) variables which produces effects
on the dependent variable which cannot be accounted for by the separate effects
of the variables in question.
E.g.: Effect of alcohol and time on learning performance

Main effect of alcohol? Main effect of time? Interaction between two variable?

8% 16% MEAN ERRORS

Morning 20 40 30
Evening 10 30 20
15 35

50
• Main effect of Time: more errors in the
40
morning.
30
• Main effect of Alcohol: more mistakes with
stronger alcohol. 20

• No interaction. Interactions may be most 10

easily grasped in terms of a graph and 0
there are no interactions when the two Morning Evening
lines are more or less parallel.
6% 16%

46
 QUANTITATIVE RESEARCH METHODS

8% 16% MEAN ERRORS

Morning 20 40 30
Evening 10 30 20
15 35

50
• Main effect of Time: more errors in the
40
morning.
30
• Main effect of Alcohol: more mistakes with 20
stronger alcohol.
10
• No interaction. Interactions may be most 0
easily grasped in terms of a graph. No Morning Evening
interactions ≈ the two lines are more or less
6% 16%
parallel.

8% 16% MEAN ERRORS

Morning 20 60 40
Evening 10 20 15
15 40

80
• Main effect of Time: more errors in the
morning. 60

40
• Main effect of Alcohol: more mistakes with
stronger alcohol. 20

• Interaction: lines not parallel. Effect of 0

strenght of alcohol is much bigger in the Morning Evening
morning than in the evening.
6% 16%

8% 16% MEAN ERRORS

Morning 20 40 30
Evening 40 20 30
30 30

50
• No main effect of Time.
40
• No main effect of Alcohol. 30
• But Interaction!!!! More mistakes with 20
stronger alcohol in the morning and less 10
with stronger alcohol in the evening. 0
Morning Evening

6% 16%

47
 8% 16% MEAN ERRORS
Morning 20 40 30
Evening 40 20 30
30 30

50
• No main effect of Time.
40
• No main effect of Learning tool.
30
• But Interaction!!!! More mistakes with
textbook in the morning and with Ipad in 20
the evening. 10
0
Morning Evening

ipad textbook

8% 16% MEAN ERRORS

Morning 30 30 30
Evening 60 30 45
45 30

80
• Main effect of Time.
60
• Main effect of Learning tool. 40
20
• But Interaction!!!! In morning no difference
between two learning tools and in evening 0
big difference between two learning tools. Morning Evening

ipad textbook

6.2. MULTIPLE DV

48
 QUANTITATIVE RESEARCH METHODS

It is important for designing and evaluating experiments and it´s related with internal
validity: are the conclusions valid? In experiment is (change in) IV causal factor for
change in DV. Controlling nuissance variables to exclude alternative explanations.
When not controlling for nuissance variables, which can explain the effect of IV on dv
away (in reality there is no effect of IV).
Controlling nuissance variables by using a control group and standardization.
Nuissance factors that may play a role (especially in pretest-posttest designs), taking
the same test learning effect may be present or getting bored.
Watch out for differential group composition (especially for between-subject designs &
with a small sample size) and for events that are not part of the experiment (but may
occur during the experiment). E.g.: when children with reading problems in a reading
group (= intervention), parents may send child to speech therapist, and we want to know
if their progress due to the intervention. Also, selective attrition of the sample (= loss of
research participants).

• Placebo effect. Treatment consists of some external characteristic and an active

aspect (= IV). In this case, pill with a working substance.
• External characteristic may influence DV. In this case, getting a pill or not,
there is a difference between conditions becomes larger when controls get no pill.
The solution could be making external characteristics equal across conditions
(double blind procedure). Participants are blind for the manipulation they
receive, and experimenter is blind (cf. Experimentator effects).
• Experimentator effects (Clever Hans). Unconsciously the experimenter may
influence the participants by his/her expectations about the behaviour of the
participants. The solution could be making the experimenter blind for the
conditions.
• Demand characteristics. Participants wants to help the researcher
(unconscious). They want to be good participants. Therefore, they confirm
hypothesis of the researcher. The solution could be covering story + debriefing
after experiment. “Did you know the goal of the experiment?”.

The regression to the mean is a sstatistical regression, reversion to the mean and
reversion to mediocrity.
An extreme score on 1e measurement will tend to be closer to the mean on the 2e
measurement and vice versa.
“Explanation”:
20 questions, y/n: very difficult, hence, everyone is guessing. Mean will be 10/20, some
will have good results (because they were lucky).
Those who perform well on this test, will perform worse on second, but similar test. Best
prediction for this student will be 10/20.
Test normally luck and knowledge. Knowledge more or less constant, but “luck” will
cause regression to the mean.

49