You are on page 1of 93

Experimental Designs

CHAPTER 9

1
Chapter Objectives
Distinguish between causal and
correlational analysis
Explain the difference between lab and
field experiments.
Explain the following terms: nuisance
variables, manipulation, experimental
and control groups, treatment effect,
matching and randomization.

2
Chapter Objectives
Discuss internal and external validity in
experimental designs.
Discuss the seven possible threats to
internal validity in experimental designs.
Describe the different types of
experimental designs.
Apply what has been learned to class
assignments and exams.

3
Experimental Designs
Experimental designs fall into two
categories:
Experiments done in an artificial or
contrived environment, known as lab
experiments, and
Experiments done in the natural
environment in which activities regularly
take place, known as the field experiment.

4
Experimental Designs
Experimental designs are set up to
examine possible cause and effect
relationships among variables.
Correlational studies examine the
relationships among variables without
necessarily trying to establish if one
variable causes another.

5
Experimental Designs
To establish that variable X causes
variable Y, all three of the following
conditions should be met:
1. Both X and Y should covary (when one
goes up, the other should also go up or
down).
2. X (the presumed causal factor) should
precede Y. In other words, there must be a
time sequence in which the two occur.
3. No other factor should possibly cause the
change in the dependent variable Y.
6
Experimental Designs
To establish causal relationships
between two variables, several
variables that might covary with the
dependent variable have to be
controlled.
This control would allow us to say that
variable X alone causes the dependent
variable Y.
7
Experimental Designs
establishing cause-and-effect
relationships is not easy because:
Several other variables that covary with the
dependent variable have to be controlled.
It is not always possible to control all the
variables while manipulating the causal
factor (the independent variable that is
causing the dependent variable) in
organizational settings, where events flow
naturally and normally.
8
Experimental Designs
It is possible to first isolate the effects
of a variable in a tightly controlled
artificial setting (the lab setting), and
After testing and establishing the cause-
and-effect relationship under these
tightly controlled conditions, see how
generalizable such relationships
are to the field setting.
9
Example 1
Suppose a manager believes that staffing the
accounting department completely with
personnel with M.Acc (Master of
Accountancy) degrees will increase its
productivity.
Thus, he wants to examine the hypothesis
that possession of a M.Acc degree would
cause increases in productivity.

10
Example 1 (Cont.)
The hypothesis can be tested in an
artificially created setting (not at the
regular workplace) because if it was tested at
the regular workplace, the manager should
transfer all those without the M.Acc
degree currently in the department to other
departments and recruit fresh M.Acc degree
holders to take their place. Such action will
disrupt the work of the entire
organization.

11
Example 1 (Cont.)
The work of the organization will be
disrupted because of the following
factors:
The new people will have to be trained.
Employees will get upset.
Work will slow down.

12
Example 1 (Cont.)
The hypothesis that possession of a
M.Acc degree would cause increases
in productivity can be tested not at
the regular workplace but in an
artificial created setting in which an
accounting job can be given to three
groups of people.

13
Example 1 (Cont.)
An artificial created setting in which an
accounting job can be given to three groups
of people:
The first group contains those with a M.Acc
degree.
The second group contains those without a
M.Acc degree.
The third group contains a mixed group of
those with and without a M.Acc degree (as in
the case in the present work setting)

14
Example 1 (Cont.)
If the first group performs exceedingly
well, the second group poorly, and
the third group falls somewhere in
the middle, there will be evidence to
indicate that the M.Acc degree
qualification might indeed cause
productivity to rise.

15
Example 1 (Cont.)
If such evidence is found, then planned
and systematic efforts can be initiated to
gradually transfer those without the M.Acc
degree in the accounting department to other
departments and recruit others with this
degree to this department.
It is then possible to see to what extent
productivity does, in fact, go up in the
department because all the staff members
are M.Acc degree holders.
16
The Lab Experiment
When a cause-and-effect relationship between
an independent and a dependent variable of interest
is to be clearly established, then all other variables
that might contaminate or confound the relationship
have to be tightly controlled so that the actual
causal effects of the investigated independent
variable on the dependent variable can be
determined.
It is also necessary to manipulate the independent
variable so that the extent of its causal effects can be
established.

17
The Lab Experiment
The controls and manipulations are best
done in an artificial setting (the laboratory),
where the causal effects can be tested.
When controls and manipulations are
introduced to establish cause-and-effect
relationships in an artificial setting, we have
laboratory experimental designs or lab
experiments.

18
Control of the Dependent
Variable
When we assume cause-and-effect
relationships between two variables X and Y, it is
possible that some other factor, say A, might also
influence the dependent variable Y.
In such a case, it will not be possible to determine
the extent to which Y occurred only because of X,
since we do not know how much of the total
variation of Y was caused by the presence of the
other factor A.
So we have to control the contaminating factor, A.

19
Example 2
A Human Resource Development manager might
arrange for special training to a set of newly
recruited secretaries in creating web pages to prove
that such training would cause them to
function more effectively.
However, some of the secretaries might function
more effectively than others because they have had
previous experience with the web.
In this case, the manager cannot prove that the
special training alone caused greater
effectiveness, since the previous experience of
some secretaries with the web is a
contaminating factor.

20
Example 2 (Cont.)
If the true effect of the training on
learning is to be assessed, then the
learners previous experience has
to be controlled.
This might be done by not including in
the experiment those who already have
had some experience with the web.

21
Manipulation of the Independent
Variable

In order to examine the causal


effects of an independent variable on a
dependent variable, certain
manipulations need to be tried.
Manipulation means that we create
different levels of the independent
variable to assess the impact on the
dependent variable.
22
Example 3
If we want to test the theory that
depth of knowledge of various
manufacturing technologies is caused
by rotating the employees on all the
jobs on the production line and in the
design department, over a 4-week
period.

23
Example 3 (Cont.)
To test this theory, we can manipulate the
independent variable, rotation the employees, by:
- rotating one group of production workers and
exposing them to all the systems during the 4-week
period.
- rotating the second group of workers and exposing
them to only half of the manufacturing technologies
during the 4-weeks.
- leaving the third group to continue to do what they
are currently doing, without any special rotation.

24
Example 3 (Cont.)
By measuring the depth of knowledge
of these groups both before and
after the manipulation (also known
as treatment), it would be possible to
assess the extent to which the
treatment caused the effect, after
controlling the contaminating factors.

25
Example 3 (Cont.)
If deep knowledge is indeed caused by
rotation and exposure, the results would
show that:
- the third group had the lowest increase in
depth of knowledge.
- the second group had some significant
increase, and
- the first group had the greatest gains.

26
Example 4
We want to test the effects of lighting
on worker production levels among
sewing machine operators.
To establish cause-and-effect
relationship, we must follow the
following steps:

27
Example 4 (Cont.)
First measure the production levels of all the
operators (60 operators) over a 15-day period with
the usual amount of light they work with- say 60
watt lamps.
Split the operators into three groups of 20
members each:
1. Allowing one subgroup to continue to work under
the same conditions as before (60-watt lambs)
2. Manipulate the intensity of the light for the second
subgroup by working with 75-watt lambs, and
3. Manipulate the intensity of the light for the third
subgroup by working with 100-watt lambs.

28
Example 4 (Cont.)
After the different groups have worked with
these varying degrees of light exposure for
15 days, each groups total production for
these 15 days may be analyzed to see if the
difference between the pre-experimental
and the post-experimental productions
among the groups is directly related to the
intensity of the light to which they have
bees exposed.

29
Example 4 (Cont.)
If the hypothesis that better lighting
increases the production levels is
correct, then the subgroup that did not have
any change in the lighting (called the
control group), should have no increase
in production and the other two groups
should show increases:
- the group with the 100-watt lambs showing
the greatest increase, and
- the group with the 75-watt lambs showing
increase lower than the 100-watt group.
30
Example 4 (Cont.)
The independent variable (lighting)
has been manipulated by exposing
different groups to different degrees of
changes in light.
This manipulation of the independent
variable is also known as the
treatment, and the results of the
treatment are called treatment effects.
31
Example 7.1: Using both Controlling
and Manipulation in a Lab Setting

The owner of a toy shop is disappointed


with the number of imitation Ninja
turtles (which is greatly in demand)
produced by his workers, who are paid
wages at an hourly rate. He might
wonder whether paying them piece
rates would increase their
production levels.
32
Example 7.1: Using both Controlling
and Manipulation in a Lab Setting
Before implementing the piece-rate
system, he would want to make sure that
switching over to the new system would
indeed achieve the objective.
The researcher might first want to test the
causal relationships in a lab setting,
and if the results are encouraging, conduct
the experiment later in a field setting.

33
Example 7.1: Designing the Lab
Experiment
The researcher should first think of possible
factors that would affect the production level of the
workers, and then try to control these factors.
The factors that would influence the
production levels of the employees, other than
piece rates, are:
1. Previous job experience.
2. Gender differences.
3. Age
The researcher needs to control these three
variables.

34
Example 7.1: Designing the Lab
Experiment
To control these three variables, the
researcher intends to set up four
groups of 15 people each, for the lab
experiment.
one to be used as the control
group
the other three subjected to three
different pay manipulations.

35
Example 7.1: Designing the Lab
Experiment
The variables that may impact on
the cause-and-effect relationship can
be controlled in two different ways:
1. Either by matching the groups or
2. Through randomization.

36
Controlling the Contaminating
Exogenous or Nuisance Variables
Matching Groups
Is done by matching the various groups by
picking the confounding characteristics and
deliberately spreading them across groups.
In our example, if there are 20 women
among the 60 members, then each group will
be assigned 5 women. Likewise, age and
experience factors can be matched across the
four groups, such that each group has a
similar mix of individuals in terms of
gender, age , and experience.
37
Controlling the Contaminating
Exogenous or Nuisance Variables

Because the suspected contaminating


factors are matched across the groups,
we may take comfort in saying that
variable X alone causes variable Y.
But here, we are not sure that we
have controlled all the nuisance factors,
since we may not be aware of them all.

38
Controlling the Contaminating
Exogenous or Nuisance Variables
Randomization
Another way of controlling the contaminating
variables is to assign the 60 members randomly to
the four groups.
Every member would have a known and equal
chance of being assigned to any of these four
groups. We might throw the names of all the 60
members into a box, and draw their names. The first
15 names drawn may be assigned to the first group,
the second 15 to the second group, and so on.

39
Controlling the Contaminating
Exogenous or Nuisance Variables
In randomization the process by which
individuals are drawn and their assignment
to any particular group are both random.
Thus, the confounding variables, age, sex,
and previous experience (the controlled
variables) will have an equal probability
of being distributed among the groups.

40
Controlling the Contaminating
Exogenous or Nuisance Variables

Randomization would ensure that all


variables that have effects (known or
unknown factors) on the dependent
variable will be distributed equally
among all groups. Any causal effects
found would be over and above the
effects of the confounding variables.

41
The Difference between Matching
and Randomization
We expect that the process of
randomization would distribute the
inequalities among the groups, based on the
laws of normal distribution. Thus, we
need not be concerned about any known or
unknown confounding factors.
In matching groups, individuals are
deliberately and consciously matched to
control the differences among group
members.
42
The Difference between Matching
and Randomization
Matching might be less effective, since we
may not know all the factors that could
possibly contaminate the cause-and-effect
relationship in any given situation, and hence
fail to match some critical factors across all
groups while conducting an experiment.
Randomization will take care of this, since
all the contaminating factors will be spread
across all groups.
43
The Difference between Matching
and Randomization
Even if we know the confounding variables,
we may not be able to find a match for all
such variables. For instance, if we have
only 2 women in a four-group experimental
design, we will not be able to match all the
groups with respect to gender.
Thus, lab experimental designs involve
control of the contaminating variables
through the process of either matching or
randomization, and the manipulation of the
treatment.

44
Table 7.1: Cause and Effect
Relationship after Randomization
Treatment effect (% Treatment Groups
increase in production
over pre-piece rate
system)
10 $1.00 per piece Experimental group 1

15 $1.50 per piece Experimental group 2

20 $2.00 per piece Experimental group 3

0 Control group (no


treatment)
45
Table 7.1 (Cont.)
Note that because the effects of experience, sex,
and age have been controlled in all the four
groups by randomly assigning the members to them,
and the control group had no increase in
productivity, it can be concluded from the result
that the percentage increases in production
are a result of the piece rate (treatment
effect).
Here we have a high internal validity or confidence
in the cause-and-effect relationship.

46
Example 5

47
48
49
50
Internal Validity
Internal validity refers to the
confidence we place in the cause-and-
effect relationship.
Internal validity addresses the
question, To what extent does the
research design permit us to say that
the independent variable A causes a
change in the dependent variable B?
51
Internal Validity
In research with high internal validity, we
are relatively better able to argue that the
relationship is causal, whereas in studies
with low internal validity, causality can not
be inferred at all.
In lab experiments where cause-and-effect
relationships are substantiated, internal
validity can be said to be high.

52
External Validity or Generalizability
or Lab Experiments

If we do find a cause-and-effect
relationship after conducting a lab
experiment, can we then confidently
say that the same cause-and-effect
relationship will also hold true in
the organizational setting?
The answer is NO.

53
External Validity or Generalizability
of Lab Experiments
The tasks in organizational settings are
far more complex, and there might be
several confounding variables that
cannot be controlled. Under such
circumstances, we cannot be sure
that the cause-and-effect relationship
found in the lab experiment is
necessarily likely to hold true in the
field setting.
54
The Field Experiment
The field experiment is an experiment
done in the natural environment in which
work goes on as usual, but treatments are
given to one or more groups.
In the field experiment, even though it
may not be possible to control all the
nuisance variables because members cannot
be either randomly assigned to groups, or
matched, the treatment can still be
manipulated.
55
The Field Experiment
If there are three different shifts in a
production plant, and the effects of the
piece-rate system are to be studied,
one of the shifts can be used as the
control group, and the two other
shifts given two different treatments or
the same treatment,that is, different
piece rates or the same piece rate.
56
The Field Experiment
Any cause-and-effect relationship found
under these conditions would have wider
generalizability to other similar
production settings, even though we may
not be sure to what extent the piece rates
alone were the cause of the increase in
productivity, because some of the other
confounding variables could not be
controlled.

57
External Validity
External validity refers to the extent
of generalizability of the results of a
causal study to other settings, people,
or events.
Internal validity refers to the degree
of our confidence in the causal effect
(that variable X causes variable Y).

58
External Validity
Field experiments have more external
validity (the results are more generalizable to
other similar organizational settings), but less
internal validity (we cannot be certain of the
extent to which variable X alone causes
variable Y).
In the lab experiment, the reverse is
true. The internal validity is high but the
external validity is rather law.
59
Trade-Off Between Internal and
External Validity
If we want high internal validity, we
should be willing to settle for lower
external validity and vice versa.
To ensure both types of validity,
researchers usually try first to test the
causal relationships in a tightly controlled
artificial or lab setting and once the
relationship has been established, they try to
test the causal relationship in a field
experiment.
60
Factors Affecting Internal Validity
Lab experiments could be influenced
by factors that might affect the
internal validity.
These possible confounding factors
pose a threat to internal validity.

61
Factors Affecting Internal Validity

62
History Effects
Certain events or factors that would
have an impact on the independent
variable-dependent variable relationship
might unexpectedly occur while the
experiment is in progress, and this
history of events would confound the
cause-and-effect relationship between
the two variables, thus affecting the
internal validity.
63
Example 7
Let us say that the manager of a Dairy
Products Division wants to test the effects of
the buy one, get one free sales promotion
on the sale of the company-owned brand of
packaged cheese, for a week.
The manager carefully records the sales of
the packaged cheese during the previous 2
weeks to assess the effect of the promotion.

64
Example 7 (Cont.)

On the first day the sales promotion


goes into effect, the Dairy Farmers
Association unexpectedly launches a
multimedia advertisement on the
benefits of consuming dairy products,
especially cheese. The sales of all dairy
products, including cheese, go up in all
the stores.
65
Example 7 (Cont.)
Here, because of unexpected advertisement, one
cannot be sure how much of the increase in sales of
the packaged cheese in question was due to the
sales promotion and how much to the advertisement
of the Dairy Farmers Association.
The effects of history have reduced the internal
validity or the faith that can be placed on the
conclusion that sales promotion caused the increase
in sales.
The history effects in this case are illustrated in
Figure 7.1

66
Figure 7.1

67
Maturation Effects
Other uncontrollable variable is the passage
of time which called maturation effect.
The maturation effects are a function of
the processes operating within the
respondents as a result of the passage of
time.
Examples of maturation processes could
include growing older, getting tired, feeling
hungry, and getting bored.
68
Example 8
Let us say that an R & D director contends that
increases in the efficiency of workers would result
within 3 months time if advanced technology is
introduced in the work setting. If at the end of the 3
months increased efficiency is indeed found, it will be
difficult to claim that the advanced technology (and it
alone) increased the efficiency of workers, because
with the passage of time, employees would also have
gained experience, resulting in better job
performance and therefore in improved efficiency.

69
Example 8 (Cont.)
Thus, the internal validity also gets
reduced owing to the effects of
maturation inasmuch as it is difficult to
pinpoint how much of the increase is
attributable to the introduction of the
enhanced technology alone.
Figure 7.2 illustrates the maturation
effects in the example.
70
Figure 7.2

71
Testing Effects
The respondents were exposed to the
pretest might influence their responses on
the posttest, which would adversely impact
on internal validity.
For example, if a challenging job is
expected to cause increases in job
satisfaction, and a pretest on job satisfaction
is administered asking for employees level of
satisfaction with their current jobs.
72
Testing Effects
When a challenging job is introduced and a
further job satisfaction questionnaire
administered subsequently, the respondents
might now react and respond to the posttest
with a different frame of reference than if
they had not originally been sensitized to the
issue of job satisfaction through the pretest..
This kind of sensitization through previous
testing is called the testing effect, which
affects the internal validity of experimental
designs.

73
Instrumentation Effects
The instrumentation effects might arise
because of a change in the measuring
instrument between pretest and posttest.
In organizations, instrumentation effects in
experimental designs are possible when the
pretest is done by the experimenter,
treatments are given to the experimental
groups, and the posttest on measures such
as performance is done by different
managers.
74
Instrumentation Effects
One manager might measure
performance by the final units of
output, a second manager might take
into account the number of rejects
as well, and a third manager might also
take into consideration the amount of
resources expended in getting the job
done.
75
Selection Bias Effects
It comes from improper or unmatched
selection of subjects for the experimental
and control groups.
For example, if a lab experiment is set up to
assess the impact of working environment on
employees attitudes toward work, and if one
of the experimental conditions is to have a
group of subjects work for about 2 hours in a
room with some high temperature.

76
Selection Bias Effects
If the researcher select a group of
volunteers who are poor and unemployed,
those will be quite different from the other
workers whom are not poor or unemployed,
and their responses to the treatment might
be quite different. Such bias in the
selection of the subjects might contaminate
the cause-and-effect relationships and pose a
threat to internal validity.

77
Statistical Regression
The effect of statistical regression are
brought about when the members chosen for
the experimental group have extreme scores
on the dependent variable to begin with.
For example, if a researcher wants to test the
understanding of students for Research
Methods classes, he should not choose
those with extremely low or extremely high
ability students for the experiment.

78
Statistical Regression
This is because we know from the laws of
probability that those with very low scores on a
variable have a greater probability of showing
improvement and scoring closer to the mean on
the posttest after being exposed to the treatment.
Likewise, those with very high abilities would also
have a greater tendency to regress toward the mean-
they would score lower on the posttest than on the
pretest.
Thus, those who are at either end of the
continuum with respect to a variable would not
truly reflect the cause-and-effect
relationship.

79
Mortality
The mortality of the members in the
experimental or control group or both, is
another confounding factor on the cause-and-
effect relationship.
When the group composition changes over
time, comparison between the groups
becomes difficult, because those who
dropped out of the experiment may confound
the results.
80
Factors Affecting External Validity
External validity raises issues about the
generalizability of the findings to other
settings. The reasons are:
- the effects of the treatment in lab
experiments are not the same in the field.
- the selection of the subjects in lab setting
could be very different from the types of
subjects selected by the organizations.

81
Example 10

Students in a university might be given


a task that could be manipulated to
study the effects on their performance.
The findings from this experiment
cannot be generalized to the real world
of work, where the employees and the
nature of the jobs would both be quite
different.
82
Factors Affecting External Validity

Maximum external validity can be


obtained by ensuring that the lab
experimental conditions are as close
to and compatible with the real-world
situation.
Thus, field experiments have greater
external validity than lab experiments.

83
Simulation
Simulation is an alternative to lab
and field experimentation.
Simulation uses a model-building
technique to determine the effects of
changes.
Computer-based simulations are
becoming popular in business research.

84
Simulation
A simulation is an experiment conducted in
a specially created setting that very closely
represents the natural environment in which
activities are usually carried on.
The simulation lies somewhere between a
lab and a field experiment since the
environment is artificially created but not far
different from reality.

85
Simulation
Two types of simulations can be done:
One in which the nature and timing of
simulated events are totally determined by
the researcher (called experimental
simulation).
The other where the course of activities is at
least partly governed by the reaction of the
participants to the various stimuli as they
interact among themselves (called free
simulation).
86
Simulation
Experimental and free simulations are
both expensive, since creating real-
world conditions in an artificial setting
and collecting data over extended
periods of time involve a high costs.

87
Simulation
Causal relationships can be tested
since both manipulation and control
are possible in simulations.
Cause-and-effect relationships are
better established in experimental
simulations where the researcher
exercises greater control.

88
Areas Where Simulation can be
Used
The effectiveness of various analytic review
procedures in detecting errors in account
balances has been tested through
simulations.
In the finance area, risk management has
been studied through simulations.
Simulations have also been used to
understand the complex relationships in the
financing of pension plans and making
important investment decisions.
89
Simulation
Simulation has also been used by many
companies to test the robustness and efficacy
of various products.
It is quite likely that we will see simulation
being used as a managerial tool to enhance
motivation, leadership, and the like, in the
future.
Simulation can also be applied as a
problem-solving managerial tool in other
behavioral and administrative areas.
90
Ethical Issues in Experimental
Design Research
The following practices are considered
unethical:
Putting pressure on individuals to participate
in experiments through coercion, or applying
social pressure.
Giving menial tasks and asking demeaning
questions that diminish their self respect.
Deceiving subjects by deliberately misleading
them as to the true purpose of the research.

91
Ethical Issues in Experimental
Design Research
Exposing participants to physical or mental
stress.
Not allowing subjects to withdraw from the
research when they want to.
Using the research results to disadvantage
the participants, or for purposes not to their
liking.
Not explaining the procedures to be followed
in the experiment.

92
Ethical Issues in Experimental
Design Research
Exposing respondents to hazardous and
unsafe environments.
Not debriefing participants fully and
accurately after the experiment is over.
Not preserving the privacy and
confidentiality of the information given
by the participants.

93