PROGRAM

EVALUATION
Expressing and Assessing Program Theory
Introduction
• The focus in this chapter is on the second of the five
evaluation domains in the RFL evaluation framework or
model (needs assessment, program theory
assessment, implementation assessment, impact
assessment, and efficiency assessment).
Introduction
• You must figure out exactly what the program theory is—
that is, you must articulate the program theory; you must
make it explicit, so that it can be examined and evaluated.
• Typically, a program theory will be implicit, at best. That’s
why the evaluator must understand how the program is
intended to operate.
• Program theory (like needs assessment) is very
important, because it is foundational for later types of
evaluation in the RFL model (i.e., implementation
assessment, impact assessment, and efficiency
assessment).
 Introduction

Overview of Program Theory

Service Utilization Plan
Target population Target interaction with delivery system

Impact Theory
Service Proximal Distal
Arena Program - target service interactions Outcomes Outcomes

Program Program facilities, personnel, activities

Program's Organizational Plan
What is Program Theory
• A Program Theory is the assumption that your program’s
design, activities, and execution will lead to the
achievement of the outcomes you intend for your clients.

• A clear and concise Program Theory lays out a logical

description of why the activities you provide will lead to
the results or benefits you intend. Your Program Theory
is the backbone of the success of your program.
How do you define your own Program
Theory?
• You can often capture your Program Theory with a series of if-
then statements; meaning that if something is done for program
clients, then something should happen or change for these
clients. Using our example above of the employment program,
you can create a series of if-then statements leading to an
outline of your Program Theory, such as:

• If a certain amount of inputs (resources such as staff, funding,

etc) are available, then our program can provide job readiness
skills classes for clients.

• If our clients attend these job readiness skills classes, then

they will learn the skills necessary to search for jobs.
• Note: A well-developed theory ensures that there is a well-
established connection between, or at least credible
evidence for, the underlying assumptions.
COMPREHENSIVE CANCER
CONTROL FAMILY OF MODELS
The Evaluability Assessment
Perspective
• Evaluability assessment
• Explores the objectives, expectations, and information needs of program managers
and policy-makers;
• explores program reality;
• assesses the likelihood that program activities will achieve measurable progress
toward program objectives;
• and assesses the extent to which evaluation information is likely to be used by
program management.

• The products of evaluability assessment are:

• (1) a set of agreed-on program objectives, important side-effects, and performance
indicators on which the program can realistically be held accountable;
• (2) a set of evaluation/management options which represent ways in which
management can change program activities, objectives, or uses of information in
ways likely to improve program performance.
The Evaluability Assessment Rules
• 1.    Work with management or others who are likely to
use evaluation results, and determine their
informational needs. 
The Evaluability Assessment Rules
• 2.    Do not evaluate a program if it is not ready to be evaluated.  
• Are the objectives realistic given program conceptualization and
resources? If not, tell your client the program is not ready to be
evaluated. 
• Conduct an evaluation only when someone is going to use the
evaluation results (especially managers who have influence over the
development and operation of the program). 
• 3.    Articulate the program theory. When the intended program
theory is carefully examined and combined with needs
assessment data, you will understand whether the program has a
chance of working. If a program is based on a faulty
conceptualization, it is going to fail no matter how vigorously you
implement it.

• we called this problem theory failure.  

The Evaluability Assessment Rules
• 4.    Make sure that measurable performance
indicators can be obtained if you are going to
recommend that a program be evaluated. 
The Evaluability Assessment Rules
• 5.    Different program circumstances call for different levels or types of
evaluation. Wholey calls this the sequential purchase of information. Here
are the three key types of evaluation discussed by Wholey (not covered
in RFL):
• 1) Rapid feedback evaluation which is a quick assessment of program
performance in terms of agreed-upon objectives and indicators; it also
provides designs for more valid, reliable, full-scale evaluation.  
• 2) Performance monitoring which is the establishment of an ongoing
process and outcome program monitoring system.
• 3) Intensive evaluation which is a rigorous experimental evaluation to test
the validity of causal assumptions linking program activities to outcomes.
• 6.    During the evaluability assessment you should determine whether
there should be (a) no program evaluation, (b) a rapid feedback
evaluation, (c) performance monitoring, or (d) an intensive evaluation. 
Eliciting and Expressing Program
Theory
• An articulated program theory is an explicitly stated
version of program theory that is spelled out in some
detail as part of a program’s documentation and identity or
as a result of efforts by the evaluator and stakeholders to
formulate the theory (RFL’s definition).
Eliciting and Expressing Program
Theory
• Typically a program will not be fully articulated.
What will be present will be an implicit program
theory, which is a largely unstated theory that is
implicit in the program assumptions and
operations. RFL define implicit program theory as
the set of “assumptions and expectations inherent
in a program’s services and practices that have
not been fully articulated and recorded.”
Eliciting and Expressing Program
Theory
• First, you need to come up with a clear definition of
the program, its objectives, and its boundaries that is
negotiated and agreed upon by the evaluator and the
primary stakeholders.
Eliciting and Expressing Program
Theory
• Next, you can use some or all of the following
procedures for articulating or explicating or
describing the program theory.
• Review program documents and other secondary or
extant data.
• Interview program personnel and other
stakeholders.
• Make site visits and observe the program.
Eliciting and Expressing Program
Theory
• Topics to explore.
• Determine the program goals and objectives.
• Determine the program functions, components, and
activities.
• Determine the logic or sequence that links the
program functions, activities, and components.
• Corroborate the theory description that you have
now developed with the primary stakeholders
Assessing Program Theory
• First, assess the program theory in relation to the
identified problem and social needs.
• In assessing the impact theory (i.e., the causal theory), compare
the specifics of the impact theory to what the needs assessment
data indicate is required to improve the local conditions and
problem.
• In assessing the process theory (i.e., the theory of how the
program is maintained and implemented), compare the
assumptions associated with the service utilization and
organizational plans with the needs assessment data that relate to
the target population’s opportunities to obtain services and how
they are likely to react to the program.
Assessing Program Theory
• Second, assess the program theory by using logic and by determining the
plausibility of the theory assumptions, the logic of the various theory
components, and the theory’s overall logic.
• Is the program theory well defined?
• Is the program theory reasonable?
• Question 1. Are the program goals and objectives well defined, and are they measurable?
• Question 2. Are the program goals and objectives feasible, and is it realistic to assume that they can
actually be attained as a result of program action?
• Question 3. Is the change process presumed in the program theory (especially the impact theory)
plausible? For example, do you agree with the cause-and-effect chain?
• Question 4. Are the program procedures for identifying members of the target population, delivering service
to them, and sustaining that service through completion well defined and sufficient?
• Question 5. Are the components, activities, and functions of the program well defined and sufficient to
attain the intended program goals and objectives?
• Question 6. Are the resources allocated to the program and its various components and activities
adequate?
Assessing Program Theory
• Third, assess the program theory by comparing it
with evidence from research and practice.
• Is the program theory compatible with research evidence (both
applied research and basic research) and practical experience of
others?
Assessing Program Theory
• Fourth, assess the program theory by comparing it
with what you observe when you examine the
program in operation.
Conclusion
• The outcome of the process described in this chapter is a description
of the program theory and an evaluation (i.e., judgment of the worth
and merit) of that program theory.
• The evaluator can also provide formative evaluative information by
pointing out what components or specific program parts or activities
need to be revised or reconceptualized.
• It is important that the stakeholders participate each step of the
process outlined in this chapter because they must “buy into” the
program theory, they must implement the program theory, and they
are the ones that you hope will use the results of your theory
assessment.
• Thank You!