Professor: All right, now that we

have gone through the different steps that

are needed for a good impact evaluation,
it is time to go on a journey with me of an actual impact
evaluation.
This is in our city that I conducted in Karnataka state
in India, with Professor Rima Khanna at the Harvard Kennedy
School.
And the National Health Mission of government of Karnataka,
with support from USAID's development innovations
venture.
The links to the paper, as well as a J-PAL summary and policy
insight are included in the resources for this lecture.
So what was the challenge, or the problem
that policymakers were confronting?
Despite high healthcare spending,
health outcomes in India are unsatisfactory
with very slow improvements over time.
When you talk to policymakers, one reason
they will often highlight is that health staff absenteeism
is widespread, leading to suboptimal care.
For example, surveys have shown that healthcare workers
in India are absent 43% of the time they should be at work.
We found that the numbers were about 40% absenteeism
in Karnataka state.
While another study found that in Udaipur,
in the state of Rajasthan, health centers
were closed 56% of the time that they should have been open.
What is causing the doctors not to show up for work?
And how can we improve their attendance?
If we address this, will it improve the quality of care,
and therefore the health outcomes?
So what are possible contributing factors
to the problem?
Potentially, there is not enough health staff.
Or it could be that the health staff
has low intrinsic motivation.
So they don't want to show up to work.
It could also be that they are discouraged from showing up
due to a lack of basic facilities and medicines
at the clinics.
Or it could be that information about attendance
is not properly collected.
Or that there is a lack of enforcement
against those absent leading to furthermore absenteeism.
Depending on what the underlying causes,
there are different possible solutions.
If you think there is not enough health personnel,
you can hire additional staff.
If instead you believe intrinsic motivation is
the main contributing factor, you
may want to remind health staff of the importance
of their work.
If the lack of basic facilities and medicines
may discourage workers from coming to the clinics
you could invest in the infrastructure and medicine
availability.
And finally, if the underlying source
is the lack of information on attendance
and poor enforcement, you could make attendance data
available to the supervisors.
To decide which of these options is best suited to our context,
we need to pair a good understanding of the problem
and knowledge of local conditions
with rigorous evidence from around the world.
In this case, the government decided
to improve attendance data via a tracking system
that would collect fingerprints from present staff,
and transmit the data in real time using cell phones.
It is important to note, that this was a program
that the government itself conceived, based
on a similar model that had worked very well for government
staff working in the state capital of Bangalore, now
Bengaluru.
They were going to expand it to all primary health
centers in the state.
And were in the process of securing funding to do so.
But before doing that, they invited us to work with them
and to hone down the theory of change.
And add in an impact evaluation of a pilot rollout
to understand if it really works or not,
before they scale it up.
So kudos to the state government of Karnataka
for their foresight, as most government programs
in my experience, just go straight from an idea,
to a grand announcement, to an ad scale rollout.
With consequences that we are all
well aware of in terms of bad implementation,
and a lack of impact.
Next, we collaboratively develop the program's theory of change.
That is, the envisioned change process from the identified
problem or need.
What are the direct input and outputs
to the desired intermediate, and to the final outcome.
In our example, the government had the theory
that the tracking machines will improve attendance and thereby,
lead to better care and health outcomes.
Let's go through the theory of change step by step.
As discussed, the identified need
is the low attendance of health staff and poor health outcomes.
We have also already talked about the suggested program
or input.
Namely, the installment of a biometric attendance tracking
system in primary health care centers
to transmit attendance data in real time.
The output of this program is that data
is available in real time and is being used.
Sometimes it can feel a bit artificial
to distinguish between inputs and outputs,
but differentiating the two is really important.
Imagine the program is not implemented as planned.
You will be able to see this because you will not
observe the outputs.
So it is useful to define those in advance.
What are the outcomes of interest?
Note, that intermediate outcomes often
refer to a change in attitudes or behavior.
Our theory was, that the program would
lead to improved attendance of staff.
And that staff would spend more time with patients.
Or more patients would show up to be examined.
Finally, we hope that these intermediate outcomes
would lead to better standards of care and better health
outcomes.
Note, that for there to actually be a change in health outcomes,
a number of other assumptions have to hold.
For example, visits with doctors have
to result in better care due to accurate diagnosis.
Given the focus of this lecture, we
will not go into all those details today.