EXPERIMEN

TS
 Experimental Designs
 Experimental designs are set up to examine possible cause-
and-effect relationships in contrast to correlational designs.

 Conditions of a Causal Relationship (X Causes Y):

1. Both X and Y should covary (when one goes up, the other should
also simultaneously go up or down). This is called concomitant
variation.
2. X (causal factor) should precede Y. There must be a time sequence.
3. No other factor should possibly cause the change in the dependent
variable Y. Other variables are controlled (isolated) in artificial lab
settings.
4. A logical explanation (a theory) is needed to explain/justify why the
independent variable affects the dependent variable.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.227.
Example: The Importance of M.Acc.
Degree on Department Performance
Does having a M.Acc increase efficiency in the Accounting Department?
 Suppose a manager believes that staffing the accounting department
completely with personnel with M.Acc. (Master of Accountancy)
degrees will increase its productivity. It is impossible to transfer all
those without the M.Acc.degree currently in the department to other
departments and recruit fresh M.Acc. Degree holders to take their
place. Such a course of action is bound to disrupt the work
environment in the entire organization.

However, the hypothesis that possession of an M.Acc. Degree would

cause increases in productivity can be tested in an artificially created
setting in which various accounting problems can be given to three
groups of people: those with an M.Acc. Degree, those without an
M.Acc degree, and a mixed group of those with and without an
M.Acc degree (as is the case in the present work setting).

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.228.
 Example: The Importance of M.Acc.
Degree on Department Performance
 Ifthe first group performs exceedingly well,
 the second group poorly,
 and the third group falls somewhere in the middle…

…then this will be evidence that M.Acc.degree really increases

productivity (effectiveness).

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.228.
 Lab Experiments
 To investigate a causal relationship between two variables, all other
variables that might “contaminate” the relationship have to be
tightly controlled. And then, the independent variable will have to
be manipulated.

 Control: If we cannot control other variables that might influence

Y, we cannot be sure that all of the variation in Y is caused by X.

◦ Secretaries get computer training and this makes them function more
effectively. What if previous skills and experience are also influential
factors? Then, we have to control them and and include only those who
have equal skills and experience level in this experiment.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.229.
 Lab Experiments
 Manipulation (Treatment): creating different levels of the independent
variable to assess its impact on the dependent variable.

◦ Y: depth of various manufacturing technologies

◦ X: rotating employees on the production line

◦ Group 1: rotated on the total line

◦ Group 2: rotated on half of the line
◦ Group 3: no rotation

If all contaminating factors are controlled and three groups show

reasonable differences, we can say X causes Y.
Read Example about lighting.
This manipulation of the independent variable is also called “treatment”
and the resulting differences are called the “treatment effect”.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, pp.229-230.
 Controlling Contaminating or
“Nuisance” Variables
 We have 60 people for an experimental study. We want to divide them into
four groups to test the causal effect.

 Method 1: Matching Groups: spread different levels of contaminating

variables among groups

 If we think gender is a contaminating factor, and we have 60 people with

20 females – 40 males, then each group can be assigned 5 women and 10
men.

 Problems:
◦ Are we sure we controlled “all” nuisance variables?
◦ Can the groups be spread so fluently?

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.231.
 Controlling Contaminating or
“Nuisance” Variables
 Method 2: Randomization: Assign all members randomly
without any predetermination such that every member has a
known and equal chance of being assigned to any of the four
groups. Selection process and group assignment process are
both random.

 Contaminating factors have an equal probability of being

distributed among groups.

 Any errors or biases caused by age, sex, and previous

experience are now distributed equally among all four groups.

 Both problems of Method 1 are solved with randomization. Now

we can measure the results.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, pp.231-2.
 Cause and Effect Relationship After
Randomization
 EG 1: $1/piece
 EG 2: $1,50/piece
 EG 3: $2/piece
 CG (no treatment): Old hourly rate

 If % increases in production are, 10, 15, 20, and 0

respectively, we can say that there is a causal relationship
because through randomization we can now assume that we
controlled for all the contaminating variables.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.232.
 Internal vs. External Validity
 Internal Validity: How confident are we in the cause-and-effect
relationship?

In lab experiments, internal validity is higher so our confidence

in the causality of the relationship is higher.

 External Validity: To what extent are the results found in the

lab setting generalizable to the real environment?

Field Experiments are done for this purpose. We may not

control all nuisance variables but the treatment can still be
manipulated.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.233.
 The Field Experiment
 An experiment done in the natural environment in which treatments are
still given to one or more groups
 Nuisance variables cannot be controlled but manipulation can still be
done.
 “Lower internal validity” but “Higher external validity”

Trade-off between internal and external validity:

Researchers who do not want to give up both have to run both
experiments.

 If there are three different shifts in a production plant, the effects of the
piece rate system can be studied on them with a field experiment. One
of the shifts can be the control group; the other two shifts can be given
two different levels of treatment and we may look at the results. But,
have we controlled all the factors?

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.234-5.
 Factors Affecting the Validity of
Experiments
• History Effects: Certain events or factors that might affect the
independent-dependent variable relationship might unexpectedly occur
during the experiment and this “history” of events would confound the
cause-and-effect relationship between the two variables.

Example: The manager of a Dairy Products Company wants to test the

effects of the “buy one, get one free” sales promotion on the sale of the
company-owned brand of packaged cheese for a week. She carefully
records the sales of the packaged cheese during the previous two weeks
to assess the effect of the promotion. However, on the day that her
sales promotion goes into effect, the Dairy Farmers’ Association
unexpectedly launches a multimedia advertisement on the benefits of
consuming dairy products, especially cheese. The sales of all dairy
products, including cheese, go up in all the stores, including the one
where the experiment had been in progress.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, pp.235-6.
 Illustration of the History Effect

Time:
t1 t2 t3

Sales Sales
promotion
Independent Dependent
variable variable
Dairy
farmers’
advertisement
Uncontrolled
variable
• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.236.
 Other Examples
 Testing the effect of vitamin intake on children’s health, first
measurements are taken, right before measurement of the 30-
day effect, a flu virus hits the city.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.236.
 Factors Affecting the Validity of
Experiments
• Testing Effect: In experiments, to test the effects of a treatment,
subjects are usually given a “pretest”. When a pretest-experiment-
posttest design is used, the difference between the posttest and the
pretest scores is attributed to the treatment. However, participants’
exposure to the pretest may affect both the internal and external
validity of the findings.

• Main testing effect (threatening internal validity): This occurs

when the prior observation (pretest) affects the later observation
(posttest). Subjects respond in the same way to the pretest and the
posttest because they want to be consistent.

• Example: We are testing the effect of a television commercial (the

treatment) on attitudes toward the brand using a pretest and a
posttest. Suppose that no significant difference is found. We may
conclude that the commercial was ineffective. However, an
alternative explanation is that participants wanted to be
consistent and answered parallel to the pretest.
• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.237.
 Factors Affecting the Validity of
Experiments
• Interactive testing effect (threatening external validity): This
occurs when the pretest affects the participant’s reaction to the
treatment (the independent variable).

• Example: Because of the pretest, participants may watch the TV

commercial more closely than consumers that do not take part in
the experiment. So any effects that are found may not necessarily be
generalizable to the population.

Pretest Treatment Posttest

interactive
testing effect

main testing
effect

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.238.
 Factors Affecting the Validity of
Experiments

• Maturation Effect: Cause-and-effect inferences can be contaminated by

the effects of passage of time, i.e. getting tired, hungry, bored, etc.
Ex: testing food ad effectiveness – hunger

• Mortality: Group composition changes over time.

Ex: For example, in a Web-based instruction project entitled Eruditio, it
started with 161 subjects and only 95 of them completed the entire
module. Those who stayed in the project all the way to end may be more
motivated to learn and thus achieved higher performance.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.236-9.
 Factors Affecting the Validity of
Experiments
• Statistical Regression: If the members chosen for the experimental
group have extreme scores on the dependent variable to begin with.
Those with very high or very low scores will have a higher
probability of “regressing toward the mean” so we will not have a
true effect but an effect with regression error.

The impact of sales training on very ineffective or very effective

salespeople is an example.

Suppose we measure the output of all workers in a department for a

few days before an experiment and then conduct the experiment
with only those workers whose productivity scores are in the top
25% and bottom 25%. No matter what is done between the first
measurement and the latter measurement, there is a strong
tendency for the average of the high scores to decline and for the
low scores to increase. This is called regressing toward the mean.

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.239.
 Factors Affecting the Validity of
Experiments
• Instrumentation Effect: If the posttest-pretest difference
stems partly or totally from the change in the measurement
instrument (type of measurement used, person doing the
measurement at the pretest and posttest, etc.), the
instrumentation effect takes place.

• An observer who is involved in observing a particular pattern

of behavior in respondents before a treatment might start
concentrating on a different set of behaviors after the
treatment. Thus, in a sense the measurement instrument has
changed.

• In an experiment done in an organization, pretests can be

done by a different person, treatments may be given to
experimental groups, and posttest measures may be done by
different people. This might cause a problem especially if the
dependent variable (i.e. performance is not operationalized).

• Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, pp.239-240.
 Factors Affecting the Validity of
Experiments
• Selection Bias Effect: This is about the improper selection of
respondents or the selection of a non-representative group of
respondents.
• In the experiment, the subjects selected may be different from the real
population and may not be representative. Thus, findings from the
experiment cannot be generalized.
• Example: A lab experiment is set up to assess the impact of the
working environment on employees’ attitudes toward work and
the experimental condition is to have a group of subjects work for
about two hours in a room with some mild stench. You inform
participants about this situation beforehand. Volunteers may be
attracted by offering a $70 incentive. Those who will volunteer
will be coming from an environment of deprivation and their
responses to the treatment might be different. Since the control
group and the experimental group will mismatch, this will create a
problem.

• Example: inform respondents that you will watch two hours of

commercial
•Sekaran, U. & R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.238.
•Cooper, D.R. & P.S. Schindler, Business Research Methods, McGraw-Hill Irwin, 9th ed., 2006, pp.283-4.
 Experimental Designs
 Definition of Symbols:
◦ O = Observation (Measurement)
◦ X = Exposure to the independent variable (treatment)
◦ EG: Experimental Group
◦ CG: Control Group

•Malhotra, N.K., Marketing Research, Pearson Education, 6th ed., 2010, p.254.
•Aaker, D.A.; V.Kumar & G.S. Day, Marketing Research, Wiley, 8th ed., 2003, p.346.
 Experimental Designs
 One-Group After-Only Design:

 Design:
EG: X 01
 Example: Coke might bring new versions of the product to the
Turkish market and run a one-month intensive promotional
campaign for the new products in Istanbul during June. They, then,
might measure the sales of the product. A high amount of sales
would serve to justify the advertising expenditure and expand to
other cities.*(example adapted from Aaker (2003))
 The major problem with this design is not being able to isolate the
effect of promotions on the amount of sales. History effects might
be very influential here as well as maturation, and selection bias.

•Aaker, D.A.; V.Kumar & G.S. Day, Marketing Research, Wiley, 8th ed., 2003, p.347-8.

•Malhotra, N.K., Marketing Research, Pearson Education, 6th ed., 2010, p.259.
 Experimental Designs
 Post-test Only Control Group Design: EG: X O1
CG: O2
 Treatment Effect: (02-01)
 Example: To measure the effectiveness of a test commercial for a
department store, the posttest-only control group design would be
implemented as follows. A sample of respondents would be selected at
random. The sample would be randomly split, with half the subjects
forming the experimental group and the other half constituting the control
group. Only the respondents in the experimental group would be exposed
to the TV program containing the test (Sears) commercial. Then, a
questionnaire would be administered to both groups to obtain posttest
measures on attitudes toward the department store (Sears). The difference
in the attitudes of the experimental group and the control group is the
treatment effect.

•Cooper, D.R. & P.S. Schindler, Business Research Methods, McGraw-Hill Irwin, 9th ed., 2006, p.289.
•Malhotra, N.K., Marketing Research, Pearson Education, 6th ed., 2010, p.261.
Experimental Designs
 One-Group Pretest-Posttest Design:
Named as “One-Group Before-After Design” in Aaker (2003)
 Design: EG: 01 X 02
 Treatment Effect: 02-01
 Example: A one group pretest-posttest design to measure the
effectiveness of a test commercial for a department store, for
example, Sears, would be implemented as follows. Respondents are
recruited to central theater locations in different test cities. At the
central location, respondents are first administered a personal
interview to measure attitudes toward the store, Sears (01). Then,
they watch a TV program containing the test commercial (X). After
viewing the TV program, the respondents are again administered a
personal interview to measure attitudes toward the store (02). The
effectiveness of the test commercial is measured as 02-01.
 Experimental Designs
 The most important differentiating factor between preexperimental
and true experimental designs is that in true experimental designs,
subjects are randomly assigned to experimental and control groups.
 Pretest-posttest Control Group Design:
Named as Two-Group Before-After Design in Aaker (2003).
 Design: EG: 01 X 02
CG: 03 04
 Treatment Effect: (02-01)-(04-03)

•Aaker, D.A.; V.Kumar & G.S. Day, Marketing Research, Wiley, 8th ed., 2003, p.351.

•Cooper, D.R. & P.S. Schindler, Business Research Methods, McGraw-Hill Irwin, 9th ed., 2006, p.288-9.
•Malhotra, N.K., Marketing Research, Pearson Education, 6th ed., 2010, pp.260-1.
 Time Series Versions of
Experimental Designs
 This type of design offers more control to the researcher than preexperimental
designs but still it is weaker than true experimental designs.
 Time-Series Design: This is similar to the one group before-after design except
that a series of measurements is employed during which an experimental treatment
occurs.
EG: O1 O2 O3 O4 05 X O6 O7 O8 O9 O10
Treatment effect is seen when there is a significant change in the average of the
measurement taken after the treatment compared to those before the treatment.
See Handout about Time Series Designs.
 Multiple Time Series Design:

EG: O1 O2 O3 O4 O5 X O6 O7 O8 O9 O10
CG: O11 O12 O13 O14 O15 016 017 018 019 020

• Sekaran, U.& R.Bougie, Research Methods for Business, John Wiley and Sons Inc., 2010, p.244.
•Malhotra, N.K., Marketing Research, Pearson Education, 6th ed., 2010, p.263.