The UK’s European university

Lecture 4:
Evidence: Matching Observations to
Questions and Collecting Evidence

PO326: INTRODUCTION TO POLITICAL SCIENCE

Erik Gahner Larsen
 Context

1. Introduction: Politics and political science

2. Research Question and Theories in Political Science
3. Causation and Correlation
4. Evidence: Matching Observations to Questions and Collecting
Evidence
5. (Independent Study Week)
6. Experiments and Ethics
7. Observational (Comparative) Methods
8. Numerical Evidence: Surveys and Other Quantitative Data
9. Quantitative Methods: Evidence and Analysis
10. Qualitative Methods: Evidence and Analysis
11. Objectivity and Values
12. Recapitulation and Reflections
Page 2
 Today

• Evidence and facts

• Evidence and biases
• Internal and external validity
• Concepts, operationalisation and measurement
• Research report 1

Page 3
 Evidence? Why facts?

Page 4
 Evidence

Matching observations to questions and collecting evidence

Matching observations to questions
Internal/external validity
Operationalisation and measurement
How does the data match the theory/how
would you know if you were wrong?
Different approaches for collecting evidence
Overview of approaches
Matching approaches to questions – in
practice
Reading (Halperin & Heath ch. 7)
Seminar/workshop (measuring
attitudes)
Lecture
Reading (especially second half)
Seminar (ideology and
participation)

Page 5
 Evidence

• We like systematic evidence/empirics

• In contrast to:
– No evidence
– Anecdotes
• “In God we trust; all others must bring data.”

• But we need to evaluate the evidence:

• “Statistics make officials, and officials make statistics.”
• Some evidence is more relevant for a theory/research
question

Page 6
 Theory & observation: Beware of biases
• Humans are ”pattern recognition machines”
• We tend to see patterns where none exist

Face on Mars (NASA Viking Mission) Face on Mars (NASA Mars Global
Surveyor)
 Cognitive bias (in this case Pareidolia)
Page 7
Page 8
 Context is important for what patterns we see

Page 9
 Context is important for what patterns we see

Page 10
 Biases particularly important to us

• Confirmation bias
• We seek out and accept information that confirms
existing beliefs
• We think about ways to confirm our theory

• Negativity bias
• We put emphasis on negative information
• Focus on negative events (political scandals, bad
policies, corruption, etc.)

Page 11
 Cognitive biases and the search for facts

• Thinking requires conscientious effort

• Difference between 2+2 and 50×149

• We (people) do not have easy access to our

own mental processes

• Experts overestimate ability to control personal

biases
• More than non-experts!

Page 12
 Cognitive biases and the search for truth

• Our brain is not made for establishing truths

• Made for winning arguments

• Bias is a crucial part of the fabric of cognition

and can operate unconsciously!

• Mercier and Sperber (2011): Why do humans

reason? Arguments for an argumentative
theory, Behavioral and Brain Sciences 34, 2:
57–74.

Page 13
Page 14
Page 15 https://www.washingtonpost.com/graphics/2017/politics/cognitive-biases/
 Validity: Internal and external Validity

• Internal validity: how sure are we to draw

conclusions within the study at hand?
• How sure are we that a correlation is causal?

• External validity: how far do the results from

the study generalise more broadly?
• How sure are we that the correlation will be the same
in other contexts?

• Example: political behaviour experiments in

the lab
• Good internal validity
• Questionable external validity

Page 16
 Characteristics: Validity and reliability

• Two important criteria for judging good and bad data

• Validity: are we measuring the concept we want to measure?
• Reliability: are measures consistent across studies and
time?

Page 17
 Concepts, operationalisation and measurement

Conceptualisation
• What are you talking about?
• Define your concepts!

Operationalisation
• How would you know when you saw it?

Measurement
• ‘Scoring’ in a comparable way

Page 18
 Point of departure

• Theoretical concept Empirical operationalisation

• Often we have multiple ways to operationalise a

concept!
• Methodological choices

• Measurement validity
• “Valid measurement is achieved when scores (including
the results of qualitative classification) meaningfully
capture the ideas contained in the corresponding
concept.” (Adcock and Collier 2001, 530)

Page 19
 Operationalisation

Operationalization of Free and Fair Judiciary by Nezzadar 

Page 20
 Measurement
Variable Operational Measurement(s)
definition Content of constitutions:
yes / no
Does the court have Content of constitutions:
the legal authority to yes / sometimes / rarely /
make decisions? never
Legitimacy
Administrative data:
what share of decisions
Does the court have are appealed?
the support of the
people it rules over? Survey item: how much
would you say you trust
the courts? Five point
scale from ‘not at all’ to ‘a
great deal’

Page 21
 What counts as evidence?

• Consider the causal claim: “democracy within

countries causally contributes to peace
between countries, because non-democracies
are more likely to initiate conflict”
• What kind of observations about the world
would be
• Evidence in favour?
• Evidence against?

• You can consider concrete events, or wider

relationships that might be amenable to
analysis

Page 22
 What to ask your evidence

Example: Money in politics

Theory: The financial sector can dominate politics

because it has the resources to fund election
campaigns and make sure the candidates it likes
win

Evidence: average political donation from

financial sector to US Senators elected in 2014:
$500,000

Page 23
 What to ask your evidence

Evidence that financial sector buys political victory?

Donation from High Low

financial sector

Winning candidates

Losing candidates

Thinking about our counterfactual ideas about causation:

this evidence is not enough

Page 24
 What to ask your evidence

Evidence that financial sector buys political victory?

Donation from High Low

financial sector

Winning candidates

Losing candidates

Thinking about our counterfactual ideas about causation: we

want to see evidence that low donation candidates lose; and that
there are not any high-donation losers, or low-donation winners

Page 25
 How could you be wrong?
Donation from High Low
financial sector

Winning candidates

Losing candidates

Finance donates a lot of money but it doesn’t buy success

Page 26
 How does the theory match the evidence?

• Theory usually tells us not just what to expect:

• High financial sector contributions for victorious
candidates
• But also what not to expect:
• Victories for low contributions candidates

• Our tendency to see confirming patterns, and to

fail to see disconfirming ones, makes it crucial
to be systematic about all the relevant
implications

Page 27
 What can we measure? Temperature?

Page 28
 Conclusion

• Political science as an approach to studying politics: a

process of collective discovery
• Theory as a way of simplifying what we’re saying to make
it clear, and creating abstractions to enable generalisation
• Causation as a particularly important kind of relationship
• Evidence: observations from the real world that are
relevant to our research questions and theories

• Next five weeks: practical strategies for gathering

evidence

Page 29
 Research report 1

• Research Report questions up on Moodle

• Pick one:
– Does the age of voters affect the voting behaviour of
British citizens?
– Has Western intervention in the Middle East caused a
radicalisation of the local population?
– Is immigration into EU countries a factor behind the rise
of nationalism?
• You can answer the same question in both
Research Reports
• Indicate your choice in the Moodle Quiz due on
Monday 21 October.
• Research report due Monday 28 October, 3:30pm
(Week 6)
• See the guidelines for the first report on Moodle!

Page 30
 Quiz 1: Online now!

• First quiz assignment up on Moodle

• Deadline: Monday 21st October, 3:30pm (Week

5)

Page 31
 Next week

Independent
Study
Week
Page 32
 Open Forum

• Open Forum: a weekly meeting in which staff

and students of all levels (i.e. BA, MA, PhD)
come together for a range of activities.

• Today:
• How to write academic essays @ university
• Dr Stefan Rossbach
• 12.00-13.00, Rutherford Lecture Theatre 1

• Full programme:
https://www.kent.ac.uk/politics/news-events/ope
n-forum.html

Page 33