FIN4715

Week 6
Impact Measurement
 Learning objectives
To understand:
• Why we need to measure impact
• How to properly measure impact

5
 Social impact creation cycle
What to
invest?
What
How to problem to
increase address?
impact?

How to
measure What steps
success? to take?

Source: Epstein, M. J., and Yuthas, K. (2014). Measuring and improving social impacts: A guide for nonprofits, 6
companies, and impact investors. San Francisco, ca: Berrett-Koehler.
 Why measure?

Target audience: internal

Learning Reasons:
- To understand
- Tests assumptions

Why?
Actions Accountability
Target audience: internal Target audience: external
& external Reasons:
Reasons: - Reports performance
- Guides future behavior - Strengthen willingness
- Makes adjustments to collaborate
- Gets buy-ins 7
 Defining the success criteria
• Base it on the intended impact objectives
– Mission statements Intention

– Framework
• Baseline
After-intervention
Additionality
Baseline?

Baseline?
• Benchmark
9
 Indicator of success
• Come up with the metric (indicator) to measure
success
• Based on the evidence
– Often linked to scientific testing
– Has the intervention (statistically) significantly changed
the behavior/condition?
• Need reliable data:
– Where do you get your data?
– How do you collect your data?
– How often do you collect your data?
– How do you analyze your data? 10
 Data collection methods
1. Primary data collection:
– Data collected directly from main sources
– Examples: surveys, focus groups, interviews,
randomized control trials, natural experiments

2. Secondary data collection:

– Data collected not directly from the main sources

11
 Before-and-after analysis
Simple difference
If the after measurement is different than the before measurement, then the
intervention created a change.

Example: In December 1955, Governor Ribicoff announced a tougher law on

speeding to reduce traffic fatalities. Data shows that traffic fatalities in Connecticut
are lower than in 1955. Was the intervention successful?
Campbell, D. T., & Ross, H. L. (1968). The Connecticut crackdown on speeding: Time-series data in quasi-experimental analysis. Law and Society
Review, 33-53.

12
 Potential problems

1. Mixing of effects
2. Does the dataset represent the entire
population (of beneficiaries)
3. Is the result caused by the intervention?

13
 Mixing of effects

Gross Other Program

outcome impacts impact

Changes Changes
Observed
attributable to attributable to
changes
factors other the
after
than the intervention
intervention
intervention

14
 Mixing of effects - example
Intervention Observed outcome

Higher score
Exposure to on
arts standardized
test

Interventions
Mentoring Observed outcome
Tutoring
Higher score
on
standardized
Exposure
to arts
test
15
 Obtaining the effect of the intervention

Difference-in-difference (DID) method can be utilized

to remove the effects of other factors from the gross
outcome.
Parallel trend assumption:
• The treated and control (non-treated) groups are
comparable

Watch a video on DID:

https://www.youtube.com/watch?v=eiffOVbYvNc

16
 Difference-in-difference (DID)
Treatment application
8

6 The effect of the

intervention
5
Outcome

0
Pre-treatment Post-treatment
Time

Treated Control

17
 Difference-in-difference (DID)
8 B
7
6 The effect of the intervention
Outcome

5
4
3 A D
2
1
0 C
Pre-treatment Post-treatment
Time

Treated Control

18
Table source: https://www.researchgate.net/figure/1-The-Difference-in-Differences-Method_tbl4_280098184
 Statistical matching
• At least 2 groups:
– Treated group: receive treatment
– Control group: do not receive treatment

• The 2 groups must be statistically comparable

– Statistical matching (also called propensity matching) is used to find
statistically comparable groups
8 8
A B
6 6

4 4

2 2

0 0
Pre-treatment Post-treatment Pre-treatment Post-treatment
Treated Control Treated Control

X 19
 Random sampling
In most cases, measurement is done on a subset of the entire
population (of beneficiaries)
– Data should be representative of the entire population
– Random sampling: takes a small, random portion of the entire
population to be part of the dataset
• Simple random sampling
– Subjects are chosen randomly from the entire population
• Stratified random sampling
– Divides the population into smaller groups based on shared
characteristics, then subjects are chosen randomly from each
group

If the data set does not represent the population, we may run into
sampling/selection bias
20
 Sampling bias example

Should mechanical ventilators be used?

Source: https://rtmagazine.com/disorders-diseases/critical-care/icu-ventilation/early-data-on-ventilated-covid-19-patients-reveals-
severe-mortality/ 21
 Sampling bias (cont.)

Patients with No mechanical

>>6%
non-severe ventilators
needed survive
symptoms

Different baseline
survival rate
Patients Mechanical
6%
with severe ventilation
survive
symptoms used

What’s the survival rate for severe

patients if no mechanical
ventilation applied? 22
 Randomized control trial (RCT)
Performed in controlled environment
– Treated & control groups
– Control group is subjected to the same factors
as the group receiving treatment except for the
treatment
– Group assignment is blind & randomized

25
 RCT (cont.)
Advantages
– Show causal relationship
– Could potentially show a more reliable evidence

Disadvantages
– Costly
– Requires strict control over the entire experiment
– Small samples – impact may not materialize
– Generalizability
• People behave differently during experiment
• Specific setting, small samples
26
 Summary
• We measure to monitor and to evaluate
• Impact measurement:
– Before and after analysis
– Potential biases
– Methods to reduce the biases

27
Arts and Economic Prosperity 5

2015
 American for the Arts
Mission

To build recognition and support for the extraordinary

and dynamic value of the arts and to lead, serve, and
advance the diverse networks of organizations and
individuals who cultivate the arts in America.

29