Experiments in

Surveys
Jon A. Krosnick

Stanford University
 Today
• Many examples of experiments to study the
psychology of influence and decision-making.

• If you are building an advertisement, you

want to test whether it works.

• Experiments are the best way to do that.

 The Logic of Experiments

• Begin with a sample of people.

• Separate the people into groups by randomly
assign each person to one “experimental
condition”.
• Administer treatments.
• Measure an “outcome variable”
• Examine whether people in the different
conditions differ from one another on the
measure.
 An Example

• A group of people who get severe headaches.

• Randomly assign each person to:
– Control group: No drug
– Treatment group 1: Small dose of the drug
– Treatment group 2: Large dose of the drug
• Measure severity and length of headaches.
• Examine whether people who got more drug
experienced less pain and shorter headaches.
 An alternative?
• Measure recall?

• Hastie and Park (1986) Psychological Review

 Another alternative?
• Ask people?
– Did that ad change your mind?
– Would the ad change your mind?

• Nisbett and Wilson (1977) Psychological

Review
 Between-Subjects Experiments
• Solid evidence of causal influence.

• Avoid experimental “artifacts”.

 Part I:

Subliminal Advertising?
 An Experiment

- Participants are brought into a lab.

- Form an impression of a person.
- See 9 photographs of the person.
 Subliminal Flashes
Each respondent was randomly
assigned to see:

Pleasant images
 Subliminal Flashes
Each respondent was randomly
assigned to see:

Pleasant images

or

Unpleasant images
 Questionnaire Measuring:

-Liking of the person

-Personality (friendly, honest)
-Physical attractiveness
 Liking of the Person

0
Pleasant Photographs Unpleasant Photographs
 Liking of the Person

5.8
5.7
5.6
5.5
5.4
5.3
5.2
5.1
5
4.9
4.8
4.7
Pleasant Photographs Unpleasant Photographs

p<.005
 Perceptions of Personality

5.6
5.4
5.2
5
4.8
4.6
4.4
4.2
Pleasant Photographs Unpleasant Photographs

p<.009
 Physical Attractiveness

4.3
4.2
4.1
4
3.9
3.8
3.7
3.6
3.5
3.4
Pleasant Photographs Unpleasant Photographs

p<.06
How do we know the images
were subliminal?

- “Did you see any flashes of

other slides?”

- Guess whether a word or photo

was flashed
 Part II:

Do Messages Persuade People?

Cigarette Smoking
 The Question

Are messages from two sources with

contradictory incentives more persuasive
than a message from one source?
 Hypothesis

Message that smoking cigarettes is

dangerous for human health:

Surgeon General of the U.S.

Tobacco Companies
 Experiment Participants:

Current and Former Smokers

 Experiment Design

I’m going to read you a series of quotes and

ask you to tell me for each one, do you think
you probably did hear or read it before
today, or that you probably did not hear or
read it before today?
 Random Assignment To:

• Control group
– No quotes
• SG group
– Surgeon General quotes only
• TB group
– Tobacco executive quotes
• SGTB group
– Surgeon general and tobacco executive quotes
 Measure:

How much do you wish you

didn’t smoke?
 Motivation to Quit Smoking

55%

50%

45%

40%

35%

30%
Control SG TB SGTB
 Motivation to Quit Smoking

55%

50%

45%

40%

35%

30%
Control SG TB SGTB
 Motivation to Quit Smoking

55%

50%

45%

40%

35%

30%
Control SG TB SGTB
 Motivation to Quit Smoking

55%

50%

45%

40%

35%

30%
Control SG TB SGTB
 Implications
Surgeon General Warnings:
“Smoking is Dangerous”

Messages from Multiple Sources with

Competing Motivations are Especially
Motivating
 Part III:

Does Price Influence

Product Choice?
 Procedure
• Describe government policies to reduce
emissions in detail.

• Provide the price to be paid by the

respondent if the policy is implemented.

• Ask the respondent to vote in favor/against.

Between-Subjects Manipulation of Price

Randomly Assign Each Respondent to a Price

Low

Moderate

High
• To reduce future global warming, the government
could require each oil company to change the way
they make fuel.

• By 2020, 30% of the fuel each company sells

would have to be a type that produces less
greenhouse gases than gasoline. Oil companies
can do this by mixing gasoline with ethanol,
which is a kind of alcohol usually made from
plants.

• This requirement would cause fuel prices to go up,

because oil companies would pass on to their
customers the cost of the ethanol, and the cost of
setting up their factories to mix the new fuel.
• The increased price of fuel would also
cause people to drive less and/or to buy
cars that use less fuel.

• Compared to current gasoline prices the

price of fuel will go up about [10/30/70]
cents per gallon each year until it reaches
about [$4/$7/$15] per gallon in 2020.

• This will reduce the amount of greenhouse

gases put out by the U.S. in 2020 by 5%.

• If an election were being held today, would

you vote in favor of this policy or would
you vote against it?
Are People Less Favorable Toward
More Expensive Policies?
 Vehicle Fuel Electricity

60
Low-
70
Carbon
40
Standard 50
20
0 30
40 60
Tax 40
20
20
0 0
40 60
40
Cap-and-
Trade 20
20
0 0
Low Med High Low Med High
 Part IV:

Can small changes in

word choice change
opinions?
 Language Matters?
• Frank Luntz (Republican)

• George Lakoff (Democrat)

 Luntz
“It’s time for us to start talking about ‘climate
change’ instead of global warming.

‘Climate change’ is less frightening than ‘global

warming.’

While global warming has catastrophic

connotations attached to it, climate change
suggests a more controllable and less emotional
challenge.”
 Design of the Experiment
• If nothing is done to reduce global warming
in the future, how serious of a problem do you
think it will be?

• If nothing is done to reduce climate change in

the future, how serious of a problem do you
think it will be?
 Extremely Serious
40%
38%
36%
34%
32%
30%
28%
26%
24%
22%
20%
Global Warming Climate Change
 Extremely Serious
40%
38%
36%
34%
32%
30%
28%
26%
24%
22%
20%
Global Warming Climate Change
 Part V:

Does Threat Inspire People

to Take Action?
 Policy Change Threat
• The danger that an undesirable change in
government policy may take place.

• Hypothesis: People only take action to

influence government when they perceive
threat.
 If Threat Is Motivating,
Who Is Most Motivated?

• People who consider the issue to be

personally important?

• People with the necessary resources

(money)?
 A Survey Experiment

• Each participant was randomly assigned to read

either:

– Information describing policy change threat.

– No information.
• “Now, I’d like to switch to a different issue.
• Members of the U.S. Congress often
introduce bills to change existing laws or
create new laws. We are interested in what
you know about Congress members’
activities these days with regard to the issue
of abortion.
• I’m going to read you a statement that
briefly describes what members of Congress
are currently doing about specific pieces of
abortion legislation and ask you some
questions about it.”
• “Many members of the U.S. Congress favor
enacting the following two laws.
• One would ban all abortions, except when
there is a threat to a woman’s life caused by
the pregnancy, and the other would prevent
women in the U.S. military from being able
to use their own money to pay for abortion
services abroad.
• And many members of Congress are
currently working hard to pass these two
laws.”
 Measures of Willingness to Pay
• How willing would you be to give money to an
organization that you agreed with and that was
concerned with the issue of abortion?
– not at all willing, slightly willing, somewhat willing,
very willing, or extremely willing?

• [If at least slightly willing] What is the most

amount of money that you would be willing to
contribute during the next year?
 Average Amount of Contribution to Lobbying Group
______________________________________________________________________________

High Low High Low

All Income Income Importance Importance
Policy Change Threat Respondents Respondents Respondents Respondents Respondents
______________________________________________________________________________

Low Threat $17.52 $17.98 $17.47 $27.72 $15.73

High Threat $22.52 $38.75 $20.65 $40.90 $19.40

Difference $5.00 $20.77 $3.18 $13.18 $3.67

(High Threat – Low Threat)
_____________________________________________________________________________
N = 2052
 Part VI:

The Impact of Counter-Arguments

in an Ad
 News Stories
• Norm among journalists:
– A “balanced” point-counterpoint news story is:
• More interesting
• More journalistically responsible
• Potential side-effects:
A “balanced” point-counterpoint news story may:
- be harder to understand?
- reduce confidence?
- reduce acceptance of the “point”?
 Building an Online Experiment
• Find a television news story with:
– A natural scientist
– A skeptic

• Edit the story:

– Remove the skeptic from one version
 Random Assignment
Treatment Group 1:
Natural Scientist
Treatment Group 2:
Natural Scientist Skeptic
Control Group:
No news story
 Participants
• Non-scientific sample of American adults
who:
– Volunteered to do surveys
– Have a computer and internet access
– Can watch videos on their computers
– N = about 2,600 per experiment

• Procedure
– Watch video
– Answer questions
 Predictions
• Does the skeptic:
– make the story more interesting?
– make the story more difficult to understand?
– increase perceived disagreement among
scientists?
– reduce confidence in the existence of global
warming?
– make temperature increase seem less likely to
be human-caused?
– Make temperature increase seem less bad?
 How interesting?
3.25

3.20

3.15

3.10

3.05

3.00

2.95
No Skeptic p<.01 Skeptic
 How difficult to understand?
1.50
1.45
1.40
1.35
1.30
1.25
1.20
1.15
1.10
1.05
1.00
No Skeptic n.s. Skeptic
Opinion Change?
 Agreement Among Scientists?
55%

50%

45%
12%
40%

35%

30%

25%
No Story No Skeptic Skeptic

p<.001
 Sure Global Warming is Happening

59%

57%

55%

53%

51%

49%

47%

45%
No Story No Skeptic Skeptic

p < .01
 Global Warming Will Be Bad

68%

66%

64%

62%

60%

58%

56%
No Story No Skeptic Skeptic

p=.10 p<.001
 The Government Should Do More
to Deal with Global Warming
66%

65%

64%

63%

62%

61%

60%

59%
No Story No Skeptic Skeptic

p < .08 p<.05

 Support Cap and Trade

72%
71%
70%
69%
68%
67%
66%
65%
64%
63%
No Story No Skeptic Skeptic

p<.05
 Covariance Structure Model
Belief that
Global
-.19+
Warming’s
Consequences
1.20*** Will Be Very
Perceived Serious
-.55***
News Story with Agreement
Skeptic Among
Scientists
1.17***
Belief that
-.22*
Global Warming
Will Be Bad
 Trust in the News Media

• Not everyone trusts the media to be accurate

and fair.

• Are people who trust the media more more

influenced by news stories?
Scientists Agree with One Another
80%
75%
70%
18%
65%
60%
55%
50%
45%
40%
9%
35%
30%
No Video Skeptic No Video Skeptic
Low Trust High Trust
Controversy:

Mediation
 Covariance Structure Model
Belief that
Global
-.19+
Warming’s
Consequences
1.20*** Will Be Very
Perceived Serious
-.55***
News Story with Agreement
Skeptic Among
Scientists
1.17***
Belief that
-.22*
Global Warming
Will Be Bad
Information on Mediation

Kosuke Imai
http://imai.princeton.edu/

Kris Preacher
http://quantpsy.org/
Conclusions