You are on page 1of 15

Module 8

Monitoring Observed Policy Outcomes

Key Terms and Concepts
Social systems accounting Regression discontinuity Social experimentation Social auditing Research and practice synthesis Threats to validity Interrupted time-series Current Euros Lorenz curve Gini Index Purchasing power Random innovation Quasi-experimentation Evaluability assessment Internal validity External validity Control-series Constant Euros

POLICY PERFORMANCE Evaluation problem structuring Forecasting




problem structuring Monitoring PREFERRED POLICIES Recommendation

The Importance of Time
What happened? Was it worthwhile?

What will happen? Will it be worthwhile? Forecasting



Monitoring POLICIES

Why Monitoring Is Important It is not that we have so many wellwell-designed policies. Rather, we have more well-designed policies wellthan we have ways to monitor them. Without monitoring, we cannot know a good policy from a bad one²or whether the policy is a one² policy at all.

An Unmonitored Policy May Conceal A Disabled Vehicle
Failing to monitor the outcomes of a policy is like counting the amount of gasoline a car has consumed without seeing how far it has traveled.

Four Functions of Monitoring 

Compliance²Are laws on local government Compliance² consistent with EU requirements? Auditing²Are revenues intended for local Auditing² communities reaching them? Accounting²Are policies on educational Accounting² reform producing qualified students? Explanation²Are outcomes of a policy Explanation² caused by the policy, or by other factors?   

Approaches to Monitoring 

Social Systems Accounting Social Auditing Research and Practice Synthesis Policy Experimentation

Social Systems Accounting 

Housing² Housing²Area per person (square meters) Average Life Expectancy Quality Adjusted Life Years Income Distribution (Gini Index) Air Pollution Index (parts per million) Lead Concentration Index (blood concentration) Persons in Mental Hospitals Persons Below Poverty Line

Social Auditing With User Surveys     

PolicyPolicy-Program Specification²What goals, Specification² objectives, and resources constitute the policy? Collection of Available Information²What Information² information is available on inputs, processes, outputs and impacts? Policy Modeling²What causal mechanisms Modeling² link inputs and processes to outputs and impacts? Evaluability Assessment²Is the policy clear Assessment² enough and unambiguous to know what to monitor? Collection of New Information²What new Information² information needs to be collected?

Research and Practice Synthesis   

Synthesis of research on planned change, communication of innovations, social marketing strategies (journals and books) Synthesis of published and unpublished policy documents (memos, reports, statistics) Synthesis of reported cases of change, innovation, and reform (case survey analysis)

Policy Experimentation  

Randomized Policy Experiments²Like Experiments² randomized clinical trrials in medicine, randomized policy experiments involve the direct manipulation of an intervention and random selection of participants and random assignment of participants to an intervention and control group. Natural Policy Experiments (also called ³quasi-experiments´² ³quasi-experiments´²Random selection and assignment are not possible or ethical, but there are intervention and control groups.

Threats to Validity (Rival Hypotheses) When Conducting Policy Experiments 

Statistical Conclusion Validity Internal Validity External Validity Construct Validity Context Validity

Interrupted Time-Series Time16 14 Probable and 12 10 8 Durable Probable and Non-Durable

Policy Intervention
6 4 2 0 1 2 3 4 5 6 7 8 9

Improbable Regression to Mean Improbable Constant Change Improbale Non-Linear Change

Sequence number

Some Major Threats to Validity 

History Maturation Instability Instrumentation Testing 


Mortality Selection Regression toward the mean Violated assumptions of statistical tests