39-2 Experiment Upld

EXPERIMENTAL AND
SURVEY DATA
DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

AARHUS UNIVERSITY
MANAGEMENT SEPTEMBER 2019 ASSISTANT PROFESSOR
AGENDA
2x 45 min- 15 min break
• EXPERIMENT- CAUSAL INFERENCE

• THE LOGIC BEHIND RANDOM TREATMENT AND SELECTION
• BETWEEN AND WITHIN SUBJECT DESINGS
• HOW DO WE PREPARE EXPERIMENTS?
• DATA ANALYSIS
• WHICH TEST?
• ANOVA WITH C
• TYPES OF EXPERIMENTS
• FIELD EXPERIMENT
• VIGNETTE EXPERIMENT
• QUASI-EXPERIMENTS

AARHUS UNIVERSITY
EXPERIMENT THE GOLD STANDARD
Experiments can determine causal relations through randomize trials
• Causality – real understanding of a phenomenon
What is the effect of x on y?
Independent (x) vs. Dependent (y) variable
How do we do that?
• Make a specific prediction- hypothesis
• Causal Hypothesis: A change in X will lead to a change in Y
• Correlational Hypothesis: A change in X will be associated with a change in
Y
• Change x (only and ONLY x) and observe whether predictions were correct
• Experimental control- prove that if x not changed- no effect

AARHUS UNIVERSITY
CHANGE X AND X ONLY?
• Real experiment MUST manipulate the independent variable and control any extraneous variables
• Manipulation = the independent variable is outside of the control of subjects but controlled by the researchers
• Control means ruling out other causes
• Complete control means exact same groups subjected to different manipulations/treatments

• Effect of healthcare insurance on health
Twins, same gender, same appearance, with the exact same
abilities, raised by same parents, one has been forced to have health care
insurance and the other one does not. They have same jobs, same lifestyle,
eat the same food, smoke exact same amount of cigarettes each day….
…often social experiments like this are IMPOSSIBLE
The problem of manipulation- unethical? and problem with trying to change x and x only- we cannot clone people and
conditions

AARHUS UNIVERSITY
RANDOM TREATMENT
But can’t we just compare people with insurance to people without insurance- NO!!!!!!!!
• Do they have to be freakishly same twins?
People with and without insurance- apples and oranges – WHY?

• People with insurance are usually more educated- don’t engage in risky behaviour as much
• They have stable employment (insurance paid by employer)- better psychological health
• They have higher income- better diet
• SELECTION BIAS
So what? Clone people without insurance?

• The correct counterfactual are comparing people without insurance to people without insurance that all of sudden won
state lottery and are granted some level of insurance for free
• Lottery- random external event that is not related with their income, employment, education etc.
Random application of treatment allows us to use the correct counterfactual- comparing apples with apples without cloning

AARHUS UNIVERSITY
RANDOM SELECTION
The logic of RANDOMIZED TRIALS
• Draw two random samples from the same population
The beauty of random sample= has the same properties as population
Law of big numbers= as the number of subjects increases, the average of a sample approaches the average of
the population  the bigger the sample the more closely it is copying the population
• Averages out the effect of nuisance variables across cases- should include approximately same demographic mix
as the population
• Randomly drawing 2 large enough samples from population will create the same mix of rich and poor, employed
and unemployed, women and men etc. = COMPARABLE GROUPS
• Random samples = cloned twins

AARHUS UNIVERSITY
BETWEEN AND WITHIN SUBJECTS
DESIGNS Well if cloning people and conditions would be the perfect scenario why not just use the same people?
• The same person subjected to a treatment is not necessarily “the same person” again
• Often simply not possible in one lifetime- the same person taking an education and not taking education and getting the same job?
• Time
The effect of education on performance- children mature, its not ONLY because they receive education they perform better
Same person doing the same task might get tired, bored…
• Learning effects
The same person carrying out the same task without training and with training has learned from the first attempt
• Placebo effect
The same person is very likely to figure out what is the experiment about- mindset change
Change in expectations- placebo can change outcome

AARHUS UNIVERSITY
EXPERIMENT EXAMPLE
How should we convey messages about climate change? – story vs. facts
Who should convey messages about climate change?- politicians vs. scientists
This is based on PhD thesis by Brandi Morris- Can Stories Make Climate Change Communication More Effective? Empirical Evidence
/
You can download the whole thesis here- http://pure.au.dk/portal/en/persons/id(70d99f52-a1d8-48d7-bb3b-b5c2fca989f7).html

AARHUS UNIVERSITY
0
DESIGNS
BETWEEN
Hypothesis: Information conveyed in story
Experimental Experimental Control format rather than informational format has
Condition I Condition II Condition greater impact on people in terms of
behavioral change.
Independent Informational
Story format Neutral text
Variable format
Several changes in behavior after completing the

Dependent
experiment (turning the light off, recycling paper used,
Variable
9
cleaning up…)

AARHUS UNIVERSITY
0
DESIGNS Hypothesis: Subjects will identify more with
politicians than scientists trying to spread
messages about climate change. WITHIN Randomize Order of Conditions
Independent Variable Politician video
Treatment I
Dependent Variable Self-identification
Independent Variable Scientist video

9
Treatment II
/
Dependent Variable Self-identification

AARHUS UNIVERSITY
0
EXTENDED DESIGNS: 2X2 AND MORE?
Potential DVs- self-identification, risk-perception, intentions to change behavior…
Message format
Story format Informational format Neutral message
Politician ++ +- +0
Messenger
Scientist -+ -- -0
2 Full Factorial Design: Collect data on each combination of your independent variables – 6 conditions
 More conditions require more participants (or more repeated measures)!
9
Realize where is the control group and from which comparisons can you see the effect of each IV

AARHUS UNIVERSITY
0
SUITABILITY OF EXPERIMENTS AND
EXAMPLES
Why are experiments rare in management journals?
 Experiments might not be well suited for complex, multicomponent, nonlinear phenomena
 Often not applicable to long-term, macro-level phenomena
 Small parts of the entire complex model/process
Where do we see more experiments/examples

 individual decision-making and attitudes – can be recreated in lab-conditions
Recruitment processes (HRM)
Marketing- consumer behaviour and advertising
International management- cross-cultural psychology – decision to expatriate and influence of training
Diversity management- HRM/IB
Management of small teams and groups- response to incentives/ability to self-manage
Leadership- perception by followers

AARHUS UNIVERSITY
HOW DO WE PREPARE EXPERIMENTS?
The process
• Gain access
By far the most difficult part about experiments is gaining access to the respondents
• Prepare resources
Time-consuming and expensive method (f.e. paying participants)
• Ethical considerations
Harm to participants
Could participants be physically or mentally harmed by participating in my experiment?
Informed Consent
Do my participants have enough information about the nature of the study prior to starting the study, so they can make an
informed judgment about whether they want to participate?
Participants must have the possibility to stop the experiment at any point without negative consequences to them

AARHUS UNIVERSITY
The process continued
• Deception- to prevent demand and social desirability effects
Do I deceive my participants? Directly tell them lies about the purpose of the experiment
If so: Can I argue that the deception will not harm participants, and that the benefits of the deception are significant?

• Deception or just good design?

Deception is often not allowed but good design can hide the true purpose of the experiment without deception
• Subtle priming
• Good story
• Distraction

AARHUS UNIVERSITY
• Plan for handing personal data
Do I really need it? If so how do I store it? Delete immediately after use!
Sensitive Personal Data General personal data

• Race and ethnic background (not nationality) • A person’s name
• Religious or philosophical persuasion • A person’s address
• Political affiliation • A photo of an employee on the employee portal
• Trade union membership • Recordings of phone calls with customers
• Health information • An account number
• Sexual orientation (not registered partnership) • A salary number
• Processing of genetic or biometric data for unique • IP addresses allocated to private customers and employees
identification

AARHUS UNIVERSITY
• Pre-registration of the script
Official registration (pre-commitment)
of hypotheses, method, and data analysis strategies, before data collection
• Appropriate stimuli and manipulation checks

Aim is to create stimulus that is interpreted as intended, does not manipulate more than what is intended, but also
does not raise suspicion- representative for the context
• Use one or more stimuli- f.e. manipulating race- picture, name, citizenship?
• Manipulation check- did the respondents notice the manipulation at all?
• Did the manipulation work as intended?
• Manipulation check can be also done as pilot testing

AARHUS UNIVERSITY
Always have your “shopping list” when you’re shopping
• Protocol- the researcher collecting the data must always keep the same procedure
Important to keep all else constant/consistent!
• Receive the participant
• Give them (written) instructions
• Get their informed consent
• Let them do the task
• Debriefing
‘running sheet’ for taking notes of any observations during and after the experiment

AARHUS UNIVERSITY
DEALING WITH THE DATA

AARHUS UNIVERSITY
EXPERIMENTAL DATA- HOW DO WE ANALYSE
THEM?
First of all refresh your understanding of different types of variables

AARHUS UNIVERSITY
EXPERIMENTAL DATA- HOW DO WE ANALYSE
THEM?
Continuous
dependent variable
REGRESSION
• can deal with both multiple
categorical and interval
variables
Just one One or more
independent independent • Vocabulary of
variable variables ANOVA/MANOVA is
usually associated with
experiments and regression
with survey-type of data
Binary IV and two
independent Binary IV and Only categorical Categorical and
random samples- same sample- IVs- factors interval IVs
between design within design
MANOVA/MANCOVA-
Independent t-test Paired sample t-test ANCOVA like ANOVA/ANCOVA but
with multiple dependent
Only one More than one variables
categorical variable categorical variable
ONE WAY ANOVA N-WAY ANOVA

AARHUS UNIVERSITY
ANOVA WITH C

AARHUS UNIVERSITY
ANOVA
What does it do?
Like any other statistical tests it helps us to decide whether our “model” is good
Whether the differences we observe between treatment and control groups are big enough
Our “model” in this case has only one component- treatment variable Come my way
variance, I shall slice
you up… I am the
ANOVA zoroooo
Decomposes the total variance in the data to variance between control and treatment groups and within
• Remember that variance within control and treatment groups should be just normal variation within
samples- we are not interested in this (at least not yet)
• Is the variance BETWEEN GROUPS large enough- any difference should be just due to the
treatment
• H0= the difference between the group means is equal to 0- we want to reject this
• Statistical test used to reject H0- F-test

AARHUS UNIVERSITY
F-TEST
F-test= Mean Sum of Squares model / Mean Sum of Squares residual
between groups variance/ within group variance
variance explained by model / error variance
ANOVA is just decomposition of variance in the data!

In the 3 groups we have 3 group means and one overall mean
SS total= difference between each observation and grand mean
SS model= difference between each group mean and grand mean
SS residual= difference between each observation and group mean

AARHUS UNIVERSITY
F-TEST
SS total =
SS model =
SS residual =
i = individual
k = group
SS total= SS model + SS residual

If we have 5 individuals in each group then SS total was based on 15 scores and overall mean, SS model on 3 groups, SS
residual on 15 scores and 3 group means
For each SS we summed different number of values so to be fair

Þ Average of each SS = divide by degrees of freedom
Mean SS total has N-1 = 15-1= 14 df
Mean SS model has k-1= 3-1= 2 df
Mean SS residual has 14df-2-df= 12df

AARHUS UNIVERSITY
ANOVA- EXAMPLE
ANOVA EXAMPLE – EXPLANATION OF THE SET UP
What is the effect of attending lectures and studying?
30 students selected to 3 groups

Group 1= ordered not to study or go to lectures the entire semester
Group 2= ordered to study but not go to lectures the entire semester
Group 3= ordered to study and go to every lecture the entire semester
Each student in every group has a study-buddy (mentor from a class of students a year above that already passed the
exam last year) to help with final preparation for exam just one week beforehand
DV- continuous- test score a student receives at the exam (varies from 2-9)
IV- categorical – one control and two treatment groups (1,2,3)
Potential covariate- quality of the study-buddy measured by the test score the study-buddy received at the same exam last
year

AARHUS UNIVERSITY
ANOVA- EXAMPLE
ANOVA RESULTS
SS model (between groups)= because 3 groups -1= 2df
SS total=30 individuals -1= 29df How do we tell whether F-stat is significant or not?
SS residual (within groups)- 29-2= 27df P-values are probabilities for observing as high
results as we get in the statistical test if the H0=0.
Value of F- statistics= 8,422 / 3,486 = 2,416 which is insignificant We want low values to reject H0
It seems that studying and lectures do not make a difference! Less then 0.05 so 5%
=> Do not panic we will return to this

AARHUS UNIVERSITY
F-TEST IN ANOVA
F-test by hand
Group 1 mean = 3.22
Group 2 mean = 4.88
Group 3 mean = 4.85
Overall mean = 4.37
SS model (between groups)= 9*(3.22 – 4.37)^2 + 8*(3.88 – 4.37)^2 + 13*(4.85 – 4.37)^2 = 16,8
Average SS model = divide by degreed of freedom = 16,8 / 2 = 8,4
You can also do the significance decision “old school”

Value of F- statistics 2,416 (df 27, df 2)
You can check critical value in tables – 3,35 at 5% - our F-stat is smaller than that so not significant at 5%
SPSS gives you p-value- 0.108- so confirms NOT significant

AARHUS UNIVERSITY
AN(C)OVA
We assume that our randomly selected groups are equal in all perspectives, as random samples from populations should be
BUT what if there is a variable that is causing large variance within these groups (assumed error variance)
• If the variable is observed- good- we can add it to the analysis and “control” it- correct the error it is causing
• The variable should be independent of the treatment!
• The study-buddy influence is causing too much within group variance, the good student who passed the exam once
can help too much
• This makes the teacher look stupid….
…something must be done!
ANCOVA!

AARHUS UNIVERSITY
AN(C)OVA
Remember the model of within and between variance?
• Imagine I would “take out” the variance caused by the buddy
• When we take account of the buddy (take the variance out of DV) the
within group variance is smaller and the between group variance
automatically larger
• Because SS total = SS between (model) + SS within (error)
• Control variables in ANCOVA “take out” the within group variance caused
by them

AARHUS UNIVERSITY
AN(C)OVA
ANCOVA results are read pretty much in the same way
as ANOVA after the adjustment
• After controlling for adjustment (which is significant
predictor) F-test is significant
• Studying and attending lectures helps to improve test
score
• We can even look at the “purefied” means

• We see there are larger differences between them in
comparison to ANOVA
• F.e. treatment group 3 has mean 5.151, in the
simple ANOVA analysis it was 4.85

AARHUS UNIVERSITY
ANOVA- ASSUMPTIONS
Every analytical method stands on certain assumptions about data
Assumptions are important- if not fulfilled our results might be compromised
Some important assumptions for AN(C)OVA
• Homogeneity of variance between groups
• Independence of observations
These are similar to linear regression assumptions, which we will talk about later
• For ANCOVA to work (on experimental data) we assume covariate is not related to the experimental treatment- is independent of
experimental treatment (the grouping variable)- only then does covariate reduce the within group (error) variance
The problem with the F-test is that it only tells us whether the means are different
• We are usually interested in HOW are the means different from each other
• Usually following the analysis with analysing contrasts- comparing the means to each other (to control group f.e.)

AARHUS UNIVERSITY
REMEMBER
Way of thinking about significance
Quantitative researchers ask themselves only one question (repeatedly) IS MY MODEL SINGIFICANT??????
• Answer to that question lies in calculating test statistics
• TEST STATISTICS= variance explained by the model / variance NOT explained by the model (error)
• How do we know my test statistics is large enough?
• P-value has the answer (or critical value)
• If under 0.05 I reject H0 and accept alternative- significance
• Significance means my model is explaining variance in the dependent variable- that is good
Your job- determine the model- collect data- find the right test- was your model significant?

AARHUS UNIVERSITY
EXPERIMENTAL JUNGLE
BETWEEN EXPERIMENTS
AND SURVEYS

AARHUS UNIVERSITY
FIELD EXPERIMENT- SWEET SPOT?
If laboratory experiments are not ecologically valid and survey studies are not causal how should we study human behaviour?
FIELD EXPERIMENTS
• Maximize precision of measurement, realism of context, generalizability to population
• Manipulate behavioural variables in a field setting
• Uses random assignment but manipulation has to be relevant to working adult participants
• Manipulation should affect genuine tasks and genuine outcomes
CATEGORIES OF EXPERIMENTS
• Conventional lab experiment- student subject pool and abstract framing
• Artefactual field experiment- same as lab experiment but real employees as subject pool
• Framed field experiment- field context is in the task
• Natural field experiment- subjects naturally undertake tasks and do not know they are part of the experiment

AARHUS UNIVERSITY
VIGNETTE EXPERIMENT
• Creating a story with a character that is described by 5 pieces of information
• Randomly assigning options for each descriptor f.e. 3 × 3 × 3 × 2 × 2 = 108 different options
• Creating 108 profiles that are randomly assigned to respondents- 108 groups of people see different profile
• the aim- identify and assess the importance of those vignette factors which causally affect individual responses to the
contextualized but hypothetical vignette settings (improving internal validity, keeping the external validity high)
ADDITIONAL BENEFITS
• Can combine the survey data (age, gender, occupation etc. of the respondent) with the vignette experimental data- causally
established attitudes
• Who has stronger/weaker attitudes
• Which factors are the main causes of attitudes

AARHUS UNIVERSITY
VIGNETTE EXPERIMENT
From (Steiner & Atzmüller, 2006)
“Should this applicant become an Austrian citizen?”
EXAMPLE Description by very specific combination of five factors
(A) country of origin (Nigeria, Iran, Hungary)
In this experiment different survey respondents see a (B) marital status (wife from Austria, wife from his country of
different Mr. Miko- for some he is from Hungary, can speak origin, single)
very good German, is an unskilled worker and has an Austrian (C) official accusation (no complaints, failure of obligatory
wife…. for others he is from Iran, can speak only broken registration with the police, slight bodily assault)
German, but is a skilled employee, has also a wife from (D) language proficiency (very good German, broken German)
Austria…. (E) occupational profession (unskilled worker, employee) The
question to be answered by the respondents was: Students
• SHOULD MR. MIKO BECOME AN AUSTRIAN and old aged pensioners had to answer this question from
CITIZEN? their own perspective and the other group’s perspective
(i.e., what they think that students or pensioners would
say).

AARHUS UNIVERSITY
QUASI-EXPERIMENT
Quasi-experimental design
Regression discontinuity design
• As close to random assignment as possible- looking at cases
that are very close to a pre-set cut off point, so there are
minimum innate differences between them
• Basically comparing “what would the regression look like if

treatment was not received”
• Often combined with secondary data/survey data collected
before and after the “intervention”
From: http://www.socialresearchmethods.net/kb/quasird.php

AARHUS UNIVERSITY
QUASI-EXPERIMENT EXAMPLE
Example- Flammer (2015)

• RQ: Does corporate social responsibility (CSR) lead to superior
corporate financial performance (CFP)?
• Design- comparing companies that have passed the CSR proposals

by a very thin margin (close call CSR proposals)
• “Intuitively there should be no systematic difference between
companies that have passed the proposal by 50.1% of votes
and those that rejected the proposals by 49.9%”
• Minor difference in votes => completely different outcomes
=> discrete change (yes/no) = discontinuity
• Results- the abnormal returns on the day of vote (stock price) are
higher for the companies that have passed the CSR initiative

AARHUS UNIVERSITY
QUASI-EXPERIMENTS POST-HOC MATCHING
Matching
• As close to random assignment as possible- having the group that has already received the “treatment” the control
group is produced backwards by matching each individual with non-treated case that is as similar as possible
• Propensity scoring- predicting which individuals would have equal chance to be in the treatment group
• Good subjects for matching- did not get accepted to the treatment by “accident” “coincidence” so the reason for not
receiving treatment is very important
• Job redesign treatment in an organization- after first session management decides to outsource
Notice how vignette, regression discontinuity design and matching all mimic randomization in experiments
• Adopt experimental thinking to improve your research design

AARHUS UNIVERSITY
DEPARTMENT OF MANAGEMENT
AARHUS UNIVERSITY

39-2 Experiment Upld

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

39-2 Experiment Upld

Uploaded by

Copyright:

Available Formats

EXPERIMENTAL AND

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

2x 45 min- 15 min break

• EXPERIMENT- CAUSAL INFERENCE

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

• Complete control means exact same groups subjected to different manipulations/treatments

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

People with and without insurance- apples and oranges – WHY?

So what? Clone people without insurance?

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

​You can download the whole thesis here- http://pure.au.dk/portal/en/persons/id(70d99f52-a1d8-48d7-bb3b-b5c2fca989f7).html

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

Several changes in behavior after completing the

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

Independent Variable Politician video

Dependent Variable Self-identification

Independent Variable Scientist video

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

Story format Informational format Neutral message

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

Where do we see more experiments/examples

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

• Deception or just good design?

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

Sensitive Personal Data General personal data

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

• Appropriate stimuli and manipulation checks

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

ONE WAY ANOVA N-WAY ANOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

ANOVA is just decomposition of variance in the data!

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

SS total= SS model + SS residual

For each SS we summed different number of values so to be fair

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

30 students selected to 3 groups

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

You can also do the significance decision “old school”

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

• We can even look at the “purefied” means

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

• Basically comparing “what would the regression look like if

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

Example- Flammer (2015)

• Design- comparing companies that have passed the CSR proposals

DEPARTMENT OF ANNAMARIA KUBOVCIKOVA

You can download the whole thesis here- http://pure.au.dk/portal/en/persons/id(70d99f52-a1d8-48d7-bb3b-b5c2fca989f7).html