Data Mining

Experimental Design

04.08.2021
Experiment
• The prime method of inquiry in science is the experiment.
• The key features of experiment are: control over variables, careful
measurement, and establishing cause and effect relationships.
• An experiment is an investigation in which a hypothesis is scientifically
tested.
• In an experiment, an independent variable (the cause) is manipulated and the
dependent variable (the effect) is measured; any extraneous variables are
controlled.
• An experiments should be objective.
• The views and opinions of the researcher should not affect the results of a
study. This is good as it makes the data more valid, and less biased.
Experiment

Experiment: Any process that

generates a set of data.

A series of tests in which

purposeful changes are
made to the input variable of
a process or system so that
changes in the output
responses can be observed
and identified.
Objectives of conducting an experiment

1. Determining which variable (Input), X, are most influential on

the response (Output), Y, in a study
2. Determining where to set the influential X’s so that Y is near
the nominal requirement.
3. Determining where to set the influential X’s so that variability
in Y is small
4. Determining where to set the influential X’s so that the effect
of uncontrollable variables Z are minimized.
 Terminologies
Response: A measurable outcome of interest e.g. yield, strength, etc.
Factors: Controllable variables that are deliberately manipulated to determine their individual and joint
effects on the response. (Factors are those quantities that affect the outcome of an experiment)
e.g. temperature, time etc
Levels: Levels refer to the values of factors for which the data is gathered, “ values that factor will take in an
experiment” e.g.
Level-1 for time = 2 hours
Level-2 for time = 3 hours
Treatment:
A set of specified factor levels for an experiment run. E.g.
Treatment-1: Time = 2hours and temperature = 175 degree C
Treatment-2: Time = 3hours and temperature = 225 degree C
Noise: Variable that affect product / process performance, whose values cannot be controlled or not
controlled for economic reasons.
Replication: Replication is a systematic duplication of series of experimental runs. It provides the means of
measuring precision by calculating the experimental error.
Examples
Example-1:
• In a Machining Process:-
• Response: Surface finish ‘Y’
• Factors(control variables): Speed of Machine “Xa” and Depth of Cut
“Xb”
• Levels ( of factors) : High & Low
Factorial Experiments
Factorial experiments is the CORRECT and MOST EFFICIENT type of experiment in dealing with
several factors involved in a study; Factors are varied together instead of one at a time.
Three basic Principles of experimental design are:
Replication:
• Allow us to obtain an estimate of Experimental error which provide a basic unit of measurement for
determining whether observed differences in the data are really Statistically different
• If sample mean is used to estimate the effect of a factor, then replication allows a more precise
estimate of the effect.
Randomization:
• By randomization, both the allocation of the experimental; material and the order of individual runs
or trials can be performed randomly.
• As statistical methods required observations be independent distributed, randomization makes the
assumption valid.
Blocking:
• An experiment is arranging the runs of the experiment in groups “Blocks” so that runs within each
block have as much minor variation in common with each other as possible.
e.g. Runs using material from the same plot., Runs carried out within a short time frame
 Types of experiments
Lab Experiment

• A laboratory experiment is an experiment conducted under highly controlled conditions (not necessarily a
laboratory), where accurate measurements are possible.

• The researcher decides where the experiment will take place, at what time, with which participants, in what
circumstances and using a standardized procedure. Participants are randomly allocated to each
independent variable group.

• Strength: It is easier to replicate (i.e. copy) a laboratory experiment. This is because a standardized
procedure is used.

• Strength: They allow for precise control of extraneous and independent variables. This allows a cause and
effect relationship to be established.

• Limitation: The artificiality of the setting may produce unnatural behaviour that does not reflect real life,
i.e. low ecological validity. This means it would not be possible to generalize the findings to a real life
setting.

• Limitation: Demand characteristics or experimenter effects may bias the results and become confounding
variables.
 Field Experiment
Field experiments are done in the everyday (i.e. real life) environment of the
participants. The experimenter still manipulates the independent variable, but in
a real-life setting (so cannot really control extraneous variables).

• Strength: behaviour in a field experiment is more likely to reflect real life

because of its natural setting, i.e. higher ecological validity than a lab
experiment.

• Strength: There is less likelihood of demand characteristics affecting the

results, as participants may not know they are being studied. This occurs
when the study is covert- (e.g. secret police operation).

• Limitation: There is less control over extraneous variables that might bias the
results. This makes it difficult for another researcher to replicate the study in
exactly the same way.
Natural Experiment
Natural experiments are conducted in the everyday (i.e. real life) environment of the participants, but here
the experimenter has no control over the independent variable as it occurs naturally in real life.
• For example, Comparison of the long term development of children who have been adopted, fostered or
returned to their mothers with a control group of children who had spent all their lives in their biological
families.

• Strength: behaviour in a natural experiment is more likely to reflect real life because of its natural
setting, i.e. very high ecological validity.

• Strength: There is less likelihood of demand characteristics affecting the results, as participants may
not know they are being studied.

• Strength: Can be used in situations in which it would be ethically unacceptable to manipulate the
independent variable, e.g. researching stress.

• Limitation: They may be more expensive and time consuming than lab experiments.

• Limitation: There is no control over extraneous variables that might bias the results. This makes it
difficult for another researcher to replicate the study in exactly the same way.
Experimental Design
An experimental design must deal with four issues:
1. The people who participate in the experiment.

2. The independent variable (IV) or variables, which are also called the treatment variables. These
are the variables the researchers manipulate during the experiment.

3. The dependent variable (DV), or the effect that the researchers measure.

4. The plan for controlling extraneous variables (EV).

Types of Experimental Research Designs
Pre-Experimental Design
• Pre-Experimental Designs are the simplest form of experimental research
designs.
• Pre-experimental designs have little or no control over extraneous variables.
• These designs do not randomly assign subjects to different treatments.
• As a consequence, the results of a test using a pre-experimental design are
difficult to interpret.
• Used in testing television commercials because they are simple and relatively
inexpensive.

Types of pre-experimental designs:

One-Shot Case Studies,
One Group Pre-Test - Post-Test, and
Static Group tests
One-Shot Case Studies
With a one-shot case study, test units—(people / test markets, etc.)—are
exposed to a treatment. ---- (Directly ask students to write test and observe
output)
“X“ is Treatment- (intervention/experiment), and
“O1” Measurement Dependent variable (DV)
No random assignment of test subjects, No Control.

This research design has two significant flaws:

• there is no pre-test and there is no control group.
• A control group would, in this case, be a group that did not receive the
treatment. Without these restraints, this research design cannot establish
internal or external validity.
• Despite these limitations, market researchers often use this design for testing
new-to-the-market products.
Pre-Experimental Design
The One-Short Case Study (initially student average was 70, after
Experimentation (i.e. mock test) the average becomes 85.
Is this increase was because of experiment? We are not sure because it may
be possible that the sample may be from bright god students. The student have
acquired that level of maturity required for that class.
We are just assuming that because off experiment the result improved but can
not say firmly that this effect is because of experiment.
X : Experiment (Students given test for preparation) O : Observation
• One group receives intervention
• Then, observations/measures are taken
• Least adequate design
• Provide weakest basis for generalization
• Should not be used
 One-Group Pre-Test-Post-Test
With this research design the test unit is measured twice, one before the test and once after the
test.
There is still no control group; which is to say, a group not receiving the treatment.

Marketing researchers often use this design to test changes in the marketing plan for established
products.
This allows the researcher to estimate the treatment effect by subtracting the pre-test measure from
the post-test measure.
Due to No control, the validity of the conclusions are questionable.
Extraneous variables like history can affect the results because the observed changes in the
dependent variable might be due to factors outside the research design.
And, maturation can also be a problem as the observed changes to the dependent variable might
be due to changes in the test subjects that are not related to the treatment.
 O1 X O2

Pre-test : Experiment : Post-test

• One group : Give pre-test : Give intervention (experimentation/training..): Give post-test

• No comparison with control group.

• Consider one course which is run in only one school. And we have only one group of 15/20/40 students. We are
experimenting on that one complete group. We can not create are do not have another -control group. So we
can not confirm or study whether the same effect is observed in the control group or not.

• Ex- we have conducted pre test of the student and say the average marks after pre test were 55. Then we
conducted experiment i.e. trained students say using some specific technic say audio video etc. Then
conducted post-test and the average becomes 84.

• Though the average has increased but we can not guarantee that the average has increased because of the
experimentation because there is no control group.

• The increase may be because of the fact tat when the students appear for pre-test their maturity has increased
in the time duration from O1 to O2. May be they have understood or cleared the basic concepts of the methods
after appearing the pre-test.
 Static Group Design
With the Static Group design there is a Control Group (CG) in addition to the Experimental Group
(EG).
The experimental group is exposed to the treatment while the control group is not.
Test units, however, are not randomly assigned to the control or experimental groups.

Measurements for both groups are made after the treatment is administered to the experimental
group.
The treatment effect is measured as O1 - O2.
Weaknesses of this research design stem from the fact that test units are not randomly assigned to
the experimental or control groups and there are no pre-test measurements taken.
The Static-Group Comparison Deign
Experiment Group X (A/V) O (86)

Control Group C (Normal Teaching) O (90)

• We have two groups and comparison is done between these groups.

• And try to find the that whether the improvement in performance is due to experiment or not.

• Experiment group is taught using audio vide technic and control group is taught in conventional
method.

• For example if the average marks of control group is more than Experiment group then: can we
say that conventional method is better than A/V method? Whether Experiment Group and
control Group were at same base line? May be students form control group may be high
achievers. And experimental groups were low achievers. That is why the achievement of control
group is better. It may be reveres as well.
•These are the limitations of this system. This is used when we can not conduct pre-test.
•Comparison with control group is possible.
•No provision for establishing the equivalence of the experimental and the control group.
True Experiment Designs
True Experimental Designs are where the market researchers assign test
units to treatments at random.

There are three basic types of True Experimental Designs:

Post-Test Only Control Group Design,
Pre-Test Post-Test Control Group Design, and
Solomon Four Group Design.
Post-Test Only Control Group Design
With this research design, test units are randomly assigned to the experimental and control
groups. The experimental group is exposed to the treatment and then both the experimental
and control groups are measured.

• The effect of the treatment is calculated as O1 - O2.

• The advantage of this research design is that the random assignment of the test units
should produce roughly equal control and experimental groups before the treatment is
administered.
• The mortality for the control and experimental groups should be similar.
In true experimental design the equivalence of the experimental and control group is
provided by random assignment of subjects.

1. The Post-test-Only, Equivalence-Group Design

Experimental R X (A/v) O1
Control R C (regular lecture) O2

• Treatment Post-test

• Along with random selection of the groups researchers try to create similar conditions, by
controlling some factors, during the treatment to equalize the groups.

• (Factors: Time of the day-morning-afternoon-evening, Treatment length-hours-days in

time, Teacher, Weather) (not possible in social research, as it is not possible to operate
control conditions.)

• Only the independent variable is manipulated.

Pre-Test - Post-Test Control Group Design
With this research design, test units are randomly assigned to experimental and control
groups.
A pre-test measure is taken from both groups.

Selection bias is controlled by the randomized assignments of test units.

Mortality can be a problem if it is not relatively equal between the experimental and control
groups.
History can also be an issue if these factors effect the experimental and control groups
unequally.
The treatment effect or TE is measured by (O2 O1) - (O4 O3).
1. The Pre-test Post-tests, Equivalent-Groups Design
• Experimental R O1 X (A/v) O3
• Control R O2 C (lect) O4
• Pre-test Treatment Post-tests
Randomly assign people to two groups: one group receives treatment, the other group does not
Before treatment begins give both groups some measure
• Treatment is administered
• Measures are taken again.
 Solomon Four Group Design
The Solomon Four Group Design is a research design that assesses the impact of pre-testing on
subsequent measures.
It is used when the researcher suspects that earlier tests influence the results of later tests.
With this research design, test units are randomly allocated to two experimental groups and two
control groups.
One of the experimental groups and one of the control groups is measured.
Both experimental groups are then exposed to a treatment.
Afterwards, both experimental and control groups are measured.
A total of six measurements are taken.
The design aims to account for pre-testing bias and pre-test manipulation interaction bias.
1. The Soloman Four-Groups Design
• Experimental R O1 X (A/v) O2 Pre-Post test
• Control R O3 C (lect) O4 Pre-Post test
• Experimental R X (A/v) O5 Post test
• Control R C (lect) O6 Post test
• Pre-test Treatment Post-test

• Take the Pre-test - Post-tests Control group design and add two more groups
• Third group does not get pre-test but does get the treatment and the post-test
• Fourth group only gets post-test.
• This is the combination of the earlier two methods. So that the limitations of these are minimised.
Quasi-Experimental Designs
Quasi-Experimental Designs are used when the researcher creates an
artificial environment to control for extraneous variables.

With quasi-experimental designs, the research lacks control over when the
treatment is administered or assigns test units to the experimental and
control groups in a non-random fashion.

There are two basic types of quasi-experimental designs:

Time Series and
Multiple Time Series.
Quasi Experimental Design:

In the previous design we discussed The Pre-test - Post test, Equivalent Group design.

1. The Pre-test -Post test Non-Equivalent-Groups Design: (NEGD)

• In this there is no-randomization. (Possibly there were groups decided earlier on some basis and not random)

• Experimental O1 X (A/v) O3
• Control O2 C (lect) O4
• Pre-test Treatment Post test
In some of the literature it is represented as below:

• Experimental N O1 X (A/v) O3
• Control N O2 O4
• Pre-test Treatment Post-test
• N= Non equivalent (Not random). No random assignment.
• The two groups already existed…for example, two classrooms, two support groups etc…
Time Series
There is no randomization of the test units to the treatments.
The timing of the treatment presentation as well as which test unites are exposed to the treatment
may not be within the researcher's control.
Consumer Attitude & Usage panels are an example of quasi-experimental designs using Time
Series.

The advantages of Time Series are that it is easier to interpret the results than a One Group Pre-
Test - Post-Test design because of the many measures it takes.
The multiple measures help determine underlying trends.
Time Series design has two weaknesses:
• First, researchers cannot control history.
• Second, given the repeated measures there is a testing effect on the subjects. Subjects
may become more aware of their shopping habits, which could influence the results of
the study.
1. The Time Series design
• O1 O2 O3 O4 X O5 O6 O7 O8
• We can also have following:
• O1 O2 X O3 O4
• Similar to NEGD but have extra pre-test
• Person was give some treatment in some fixed time interval (O1 to O4) then we give some treatment (X) and then
take observation( O5 to O8). Here the main gain will be between O5 and O4 and will tell what is the effect of the
treatment. If no difference in O1 to O4 and O5 to O8 but there is difference in O4 and O5 then we can say that there is
the effect of treatment and the treatment is effective. Not necessary to have eight observations we can increase or
decrease these observations depending on the need or requirement of our research.
• At periodic interval the observations are applied
• Gain/Change from observation 4 to 5 would indicate that the treatment had an effect.
• No control over extraneous variables, history etc.
• If no change between O1 and O2 or O3 and O4 but a significant difference between O2 and O3 will found only then it
can be said that treatment is effective.
1. The Equivalent Time-Sample Design:
• O1 X1 O2 C O3 X1 O4 C O5

• We can also have following possibilities:

• O1 X1 O2 C1 O3 X1 O4 C1 O5
•
• O1 X1 O2 C1 O3 X2 O4 C2 O5

• Single group is treated as Experimental and Control group both.

• The experimental conditions X1 is present between some observations and not between others. There we
apply control conditions.
1. The Equivalent materials, Pre-test, Post test Design:
• O1 X_MA O2 time gap O3 X_MB O4
• Teaching Method A Teaching Method B
•
• Essential to this design is the learning material may be different but as nearly as possible in
interest to the students and in difficulty of comprehension. If this method is to be effective then
there should not be difference between O2 and O3.
• Find out the mean Gain between O2 & O1 and O4 & O3. If mean Gain of O4 is more than O2
then we can say that second method (B) is better than first method (A) or vice a versa.
• Limitations:
• We can not say that whether there is any effect of maturity or not. We are not sure that when we
taught in method A all student understood and they continued and easily understood method B
easily and that is why their performance is better for method B.
• Some of the limitations of the equivalent-materials single-group, pretest-postetst design can be
partially minimized by series of replications.
• I II
• O1 X O2 O3 C O4
• III IV
• O5 X O6 O7 C O8

• There are four replications.

• Two times treatment and two times Control Conditions are provided…Two times A/V teaching and
Two times Regular teaching.
• Switching Replications Quasi-experimental design:

•N O X O O
•N O O X O
• Switching: groups switch roles over course of the study.
• Replications: the treatment is repeated/replicated
Multiple Time Series
With the Multiple Time Series, the researchers add a control group to the research design.
The addition of a control group enhances the researchers' ability to discern the treatment
effect.
completely randomized design
matched-pair design
randomized block design
Comparison

Pre-Experimental True Experimental Quasi Experimental

Design Design Design
Least effective design Most effective design Less satisfactory
design
No control group Control groups are Control groups are
present present
No way of equating of Equivalence of group is Equivalence of group is
groups possible possible

Randomization may Randomization Used when

be possible randomization is not
possible
Schematic of pre-experimental research designs.
Schematic of a quasi-experimental research design
The randomized controlled study design
Additional Material
Additional Material
Example flowchart for selecting the appropriate statistical test. ANOVA = analysis of variance