Program Evaluation

Types

FORMATIVE

Formative evaluation is an on-going process that allows for feedback to be implemented during a
program cycle. Formative evaluations (Boulmetis & Dutwin, 2005):

Concentrate on examining and changing processes as they occur

Provide timely feedback about program services
Allow you to make program adjustments “on the fly” to help achieve program goals

COMMON TYPES OF FORMATIVE EVALUATION

Needs assessment determines who needs the program, how great the need is, and what might
work to meet the need.
Structured conceptualization helps stakeholders define the program, the target population, and
the possible outcomes.
Implementation evaluation monitors the fidelity of the program delivery.
Process evaluation investigates the process of delivering the program, including alternative
delivery procedures.

SUMMATIVE

Summative evaluation occurs at the end of a program cycle and provides an overall description
of program effectiveness. Summative evaluation examines program outcomes to determine
overall program effectiveness. Summative evaluation is a method for answering some of the
following questions:

Were your program objectives met?

Will you need to improve and modify the overall structure of the program?
What is the overall impact of the program?
What resources will you need to address the program’s weaknesses?
Summative evaluation will enable you to make decisions regarding specific services and the
future direction of the program that cannot be made during the middle of a program cycle.
Summative evaluations should be provided to funders and constituents with an interest in the
program.

COMMON TYPES OF SUMMATIVE EVALUATION

Goal-based evaluation determines if the intended goals of a program were achieved. Has my
program accomplished its goals?
Outcome evaluation investigate whether the program caused demonstrable effects on specifically
defined target outcomes. What effect does program participation have on students?
Impact evaluation is broader and assesses the overall or net effects — intended or unintended —
of the program. What impact does this program have on the larger organization (e.g., high school
or college), community, or system?
Cost-effectiveness and cost-benefit analysis address questions of efficiency by standardizing
outcomes in terms of their dollar costs and values. How efficient is my program with respect to
cost?

Tools

Choosing How to Evaluate Your Project

Evaluation can be a simple, do-it-yourself process, or a full-scale, professional study. The choice
of how to evaluate a project or program is usually determined by:

The scope of the project: A tiny one-day event doesn’t warrant big-time evaluation.
The funding source: Some sources, particularly large foundations, will tell you what kind of
evaluation they want to see.
Your needs: Do you need to prove a concept? Show outcomes to raise money? Or just satisfy
your curiosity?
Your resources: Do you have an evaluator on staff, or the funds to hire someone out of house?
Or are you a shoestring operation?
If evaluation is built into the funding for your project, you’re all set. All you need to do is ask for
proposals, pick the one you like best, and pay the bills. Your evaluator does the rest.

Most project evaluations, however, are do it yourself projects that require some careful planning
and thinking. As project evaluator, you’ll need to understand the tools used to conduct
evaluations, so that you can select the most appropriate tools for your particular project.

Quantitative Tools
Tools come in two groups; the best known are quantitative tools, which measure how many, how
much, how big, and so forth. Quantitative tools allow you to say things like “500 people attended
our event,” “200 people got jobs as a result of our program,” or “grades improved by 20%
because of our tutoring services.”

In some cases, quantitative tools are all you need, because your project has simple, measurable
goals, and your nonprofit CRM has got all the. numbers you need.

Typically, these are the straightforward projects that aim to lower or increase something easily
measured. For example: increase grades; decrease addiction; increase employment; decrease
homelessness.

Did it work? To find out, just measure rates within your target audience before and after your
program was implemented. Quantitative tools include basic, well known methods such as:

Head count: How many people attended?

Testing: Pre and post tests to see how many more correct answers attendees could get after
participating in your program.
Data analysis: What percent of people who attended your program graduated, increased grades,
became employed, etc.
Comparison: How many came to the events before your outreach project; how many came after
your outreach project.

Qualitative Tools
More interesting and complex are Qualitative tools, which allow evaluators to measure
intangible things like awareness, attitude, and appreciation.

These measures are used to evaluate projects with goals like “participants will appreciate the
importance of textiles as a tool for exploring culture and history” or “participants will become
more fully aware of the importance of diet and exercise in maintaining good health.”

While it may seem impossible to actually measure such intangible outcomes, there are tools for
doing just that kind of evaluation. You’ve probably used or at least heard of all of them. They
include:

Surveys. Typically, surveys are carefully crafted tools that allow you to take the pulse of a group
of people before the start of a project, and then again after the project is completed. Surveys can
measure almost anything, from prior and post knowledge of content you’re teaching to attitudes,
preferences, achievements, self-esteem… you name it. When you survey your intended audience,
you’re setting the bar for success. If your post-program survey shows improvement, you’ve done
what you set out to do. Surveys can be conducted in one of several ways: electronically (using
online systems like SurveyMonkey), with paper and pencil, or through a person-to-person
interview. Electronic surveys offer the great benefit of being easy to distribute and easy to
tabulate, but users may share limited information. Paper and pencil surveys are a good way to
start a program that requires participants to show up, sit down, and engage in a classroom like
situation.
Observation. How do you know that youngsters are more interested in fine art after a workshop
than they were before the workshop? One way to find out is to observe the group prior to and
after the workshop. Using a stopwatch, you can compare the amount of time they spend in front
of individual works. Listening to conversation, you can hear which words they use to discuss and
describe the art. You can also note body language: are they zipping past the works? Stopping to
look? Pointing and discussing? All of these observations are data points to help you assess
outcomes.

Case Studies. Your nonprofit is running a program that prepares unemployed individuals for job
placement by crafting resumes, building interview skills, honing business skills, and providing
career-appropriate clothing. Each week, dozens of people go through the program. You know
that X number are getting jobs – but you don’t really know which part of your program is most
useful, or how clients feel about their experience. One way to find out is to conduct a series of
case studies, in which you choose representative individuals and study them in depth. In the end,
you’ll come out with transcripts of interviews, pre- and post-test results, and other data to help
you tell the story of how several unique individuals arrived at your door, experienced your
program, and either succeeded or failed in reaching their goals.

Focus Groups. Your youth development program helps teens improve grades, build self esteem,
get involved with community service projects, and learn study and workplace skills. You have
lots of participants, but you really don’t know how those participants’ attitudes and abilities
changed as a result of their experience. To find out, you might run a focus group. A focus group
usually consists of 3-5 individuals who, together, represent a cross section of your target
audience. Through directed, open-ended questions, you can learn a great deal about how they
perceive and are impacted by your program before it begins, as it runs, and after it ends. Focus
groups are typically “facilitated” by someone with specific experience, recorded, and transcribed.

Interviews. Interviews are versatile and flexible, and may gather a good deal of information, but
they are time consuming and expensive. Have you ever walked into a museum or zoo and been
accosted by a person carrying a clipboard who asks whether you will answer a few questions? If
so, you’ve seen the interview process in action. Typically, interviews are used to gather
marketing information (who is coming, why are they coming, what are they coming for, are they
satisfied, etc.) – but interviews can also be used to assess knowledge, interest, and so forth. For
example, a museum might interview visitors to find out what they already know about the
Impressionists, what they might like to know, which Impressionists they prefer, and so forth.
That sort of information can provide a baseline for later comparison.

Whatever your project or your budget, there are tools available to evaluate your level of success.
When you use those tools, you improve your ability to make a case for support, write a
compelling annual report, or sell donors on your capital campaign. You also gain critical
information for refining your program, and you build a database of knowledge for developing
new programs and projects for the future.