Observations on Making

Evaluations Effective
Jared Hardner
Hardner & Gullison Associates, LLC
 Introduction
• We assume:
– Evaluations are performed for a purpose. If that
purpose is to inform decision making, we want to
be effective in answering the key questions of
decision makers.
– We, as evaluators, know something about what is
effective. Better to ask the consumers of
evaluations (but I happened to be available to give
a talk.)
 Main Themes
• Be relevant
• Work with the data you can get
• Involve stakeholders
• Deliver actionable recommendations
 Be relevant
• Who will use the evaluation?
– Are they placed to make decisions? What types?
• Did they ask for it?
– Who, if anyone, asked for the evaluation?
• How do they want to use it?
– Inform a decision process
– Summarize progress
– Fulfill accountability requirements
 Work with data you can get
• Quantitative rigor wherever possible
• Place yourself on “Research--Evaluation Continuum”
• Practical solutions: 80% of answer with 20% of data
 Percentage of Sites Employing
Specific

Work with data that exists

Elements of Adaptive Management
100%

80%
Not used

60% Partially or not yet

used
Used but needs
improvement
40%
Well developed and
fully used
Percentage of Sites
20%

0%

Control sites data

Baseline
Regular analysis
Monitoring System
Performance metrics Feedback mechanism
Management objectives Description of activities
Documentation of results
Adaptive Management Element
 Grantee Activity on Limiting Factors

Long term financing

Short term financing

Economic pressure Grantee

Other Moore
Institutional capacity
Grantee
Other
Enforcement
Not
Legal control of target addressed

Enabling policy framework

Scientific basis for mgmt

0% 20% 40% 60% 80% 100%

Percentage of Portfolio
 Involve stakeholders
• Design of evaluation questions and framework
• Review of initial findings
• Review of final evaluation
Involve stakeholders

Step #1
Client Question
What have been the site-level
conservation results of the program?
 Involve Stakeholders
Step #1
Client Question
What have been the site-level
conservation results of the program?
Step #2
Evaluee Questions
What have been the challenges faced at
the site level and on which have site
managers made the most progress?
What are the landscape-scale
conservation planning approaches used
by GCP partners?
What influence has GCP had in their
development?
What are the adaptive management
approaches used by GCP partners?
What influence has GCP had in their
development?
 Involve Stakeholders
Step #1 Step #2 Step #3
Client Question Evaluee Questions Data Analysis
What have been the site- What have been the Site-level survey Compare rating of
level conservation results challenges faced at the GCP-Rep survey & challenges at outset of
of the program? site level and on which interviews GCP funding with today
have site managers Site visits (or end of GCP funding).
made the most GCP PMP indicators Themes that should be
progress? and other partner addressed include:
reports to USAID -Project design
-Spatial scale
-Management system
-Policy & legislation
-Institutional capacity
-Economic factors
-Financial sustainability
-Other

Short case studies,

noteworthy successes and
challenges at site level.

Relate results of above

analysis to GCP PMP-
level indicators as
reported by GCP partners
 Deliver actionable
recommendations
• Strike a balance:
– Decision Makers: Limited number of actionable
recommendations
– Managers/Evaluees: Details of findings, relevant
info for implementing change
 Summary
• Our 4 observations
– Be relevant
– Work with the data that exists
– Involve stakeholders
– Deliver actionable recommendations

• Multitude of other factors

– Ideology, institutions, interests, information