Making the Case for Rigorous Evaluation

Embed Doc
Copy Link
Readcast
Collections

CommentGo Back

Download

Practice:

Develop a theory of action for your program

Key Action:

Develop relevant evaluation questions based on your

logic model

1

VIGNETTE: Making the Case for Rigorous Evaluation

Purpose:

Staff may not see the value of engaging in a rigorous evaluation
in addition to the regular evaluation required by the federal
Magnet Schools Assistance Program grant. Reading and
discussing this fictitious conversation between an evaluator and
a program manager will help staff reflect on how much additional
information a rigorous evaluation can generate about the impact
of their magnet program.

Source:

Excerpted from the Human Resources and Social Development
evaluation website for the Government of Canada (1998,
January). Last accessed on December 16, 2008, at

http://www.hrsdc.gc.ca/en/cs/sp/sdc/evaluation/sp-

ah053e/page04.shtml

Discussion Questions

1. What assumptions did the program manager make about the efficacy of his

program?

2. What questions about the program could not be answered without a

rigorous evaluation?

3. How would a rigorous evaluation address these questions? Specifically,
how would the use of a comparison group provide more useful information
for the program manager?

4. How might the use of a comparison group add value to your program

evaluation?

Practice:

Develop a theory of action for your program

Key Action:

Develop relevant evaluation questions based on your

logic model

2

Vignette: Making the Case for Rigorous Evaluation

Background: As an introduction to quasi-experimental evaluation, the Canadian

government\u2019s evaluation website for Human Resources and Social Development includes this
fictional conversation between an evaluator and a program manager. The evaluator pushes
the manager to think about various possible interpretations of his program\u2019s results, and
describes the benefits of a quasi-experimental evaluation design that includes a comparison
group.

* * *

Program manager: Why do we need to evaluate our program? We have a good handle on

what\u2019s going on with our program and our clients, and we know we are very successful.

Evaluator: Because you never know if it was your program or something else that produced

the success you are claiming.

Program manager: Of course we know it\u2019s our program. What else would cause all these

people to find jobs so quickly?

Evaluator: Maybe the rapid economic expansion that we are now enjoying.

Program manager: That\u2019s nonsense. Anyway, we know we have to have our program

evaluated. But why do we need to go through the trouble of finding a comparison group to do

an evaluation?

Evaluator: Let\u2019s say that six months after finishing your training program, 70% of the trainees

are working. Would you consider that proof your program is a success?

Program manager: I\u2019d say so, yes. We\u2019d like to do better than 70%, in fact, we believe we

are doing better than that but I\u2019d say that having 70% of our trainees working would make our
program look very good, especially given the barriers many of our clients face when they
come to us.

Evaluator: What proportion of these individuals might be working now if they hadn\u2019t gone

through your training program?

Program manager: I\u2019m not sure, but it wouldn\u2019t be as high as 70%, I can tell you that.

Evaluator: Well, you don\u2019t really know that, though. For all you know, 80% might be working

now if they hadn\u2019t taken training.

Program manager: No way. You don\u2019t know how many obstacles our clients face when they

come to us. We are providing a valuable service and are really helping our clients.

of 00

Leave a Comment