Program Evaluation

Lecture Overview
 Program evaluation and program
development
 Logic of program evaluation (Program
theory)
 Four-Step Model
 Comprehensive Quality Programming
 Empowerment Evaluation
 Group Activity
Program Development
 refers to the planning and implementation
of a program determining its purposes and
setting goals and outcomes, evaluating
program process, and evaluating attainment
of desired program outcomes and wider
impacts.
History of Program Evaluation
 1930’s human service field grew (post depression)
 WWII programs were launched to meet the needs
for urban development and housing, technological
and cultural education, occupation training, and
preventive health activities.
 By the end of the 1950’s program evaluation was
commonplace
 1960’s increased papers on evaluation research
 1970’s, evaluation was a distinct field
 1980’s benefits and costs of programs
 1990’s new social problems and booming
economy
Program Evaluation
 Refers to the collection of evidence to judge
the effectiveness of a community programs
or other community interventions (com
psych definition)
 The use of social research procedures to
systematically investigate the effectiveness
of social intervention programs (research
definition).
Project Drug Abuse Resistance
Education D.A.R.E.

 Most popular school based intervention

 Little evidence on its effectiveness
 Limited positive influence on psychological
factors (self-esteem)
 How can this lack of measurable outcomes
in a popular intervention be explained?
Reasons Why Programs May Be
Unsuccessful
 Theory Failure: concerns program theory
which is the rationale for why a particular
intervention is considered appropriate for a
particular problem with a specific target
population
 Implementation Failure: concerns quality
of program implementation (ie: program
may not be generalizable)
Effects do not occur when…….
 The underlying assumptions of program
theory are incorrect.
 The program, even if implemented well,
doesn’t affect the variables specified by
program theory.
 Or the activity or program is not
implemented adequately
Logic Model
 Graphic representation of the program.
 Shows the logical connections between the
conditions that contribute to the need for a
program in a community, the activities
aimed at addressing these conditions, and
the outcomes and impacts expected to result
from the activities.
Evaluation “Terminology”
 Stakeholders
 Fiduciary
 Funding sources
 Need Assessment
 Evaluation Instruments
Four Step Model of Program
Evaluation (Lindsay & Wandersman, 1991)
 Step 1: Identify Goals and Desired
Outcomes
 Step 2: Process Evaluation
 Step 3 Outcome Evaluation
 Step 4: Impact Evaluation:
STEP 1) Identifying Goals and
Desired Outcomes
 What is your project striving for?
 Goals set a framework for specific
outcomes.
Program Developers MUST describe:
 The primary goals of the program
 Target population
 Outcomes desired
STEP 2) Process Evaluation
 Monitoring program activities helps organize
program efforts
 Provides information to help manage the program
accountability that the program is conducting what
was done and who was reached
 After an evaluation of outcomes and impacts , the
evaluation can provide information about why the
program worked or did not work
 Can help decide whether or not you are ready to
assess the effects of your program
 What were the intended and actual activities of the
program/ Did you learn anything.
STEP 3) Outcome/Impact
Evaluation
 Assess the immediate effects of a program
“the bottom line”
 Looks at the desired outcomes defined in
Step 1 and seeks evidence regarding the
extent to which those outcomes were
achieved
 Should be closely linked to goals, but more
specific
STEP 4) Impact Evaluation
 Concerned with the ultimate effects of the
program
 Impacts are ultimate or longer-term effects
of the program
Comprehensive Quality
Programming: 9 Strategies
 Why is the intervention or program needed?
 What are the program’s goals, target populations and
desired outcomes?
 How does your program use the scientific knowledge and
best practice of what works?
 How will this new program fit in with existing programs
 How will you carry out the program
 How well was the program carried out
 How did the program work?
 What can you do to improve the program the next time you
do it?
 If the program or parts of the program were effective, what
are you doing to continue institutionalize the program
Accountability Questions &
Corresponding CQP Strategies
 Why is the intervention or program Needs and assets assessments
needed?
 What are the program’s goals, target Identifying goals and desired outcomes
populations and desired outcomes?
 How does your program use the Review research literature and best
scientific knowledge and best practices
practice of what works?
 How will this new program fit in Community feedback and planning
with existing programs Planning, implementation
 How will you carry out the program
Process evaluation
 How well was the program carried
Outcome and impact evaluations
out
 How did the program work?
Continuous program improvement
 What can you do to improve the
program the next time you do it? Institutionalization strategies
 If the program or parts of the
program were effective, what are you
doing to continue institutionalize the
program
Empowerment Evaluation
 Program planners and developers learn the
basics of program evaluation in ongoing
program improvement.
 Help program developers and staff to
achieve their program goals, by providing
the with the tools for assessing and
improving the planning, implementation,
and results of their program.
Limitations of Evaluation
 Use of an outside evaluator
 Evaluation feedback is provided at the end
of the program
 Evaluation findings are perceived as being
too complex
Conclusions
 Program evaluation is necessary to find out
whether a program is working
 Evaluation can be used for program
planning and program implementation
 As society becomes more concerned about
accountability and results , evaluation will
become more necessary.
Group Activity
 Apply 4 step model in terms of your group
project.
 Answer CQP questions