You are on page 1of 206

Toolkit

for Outcome Measurement

U N I TE D WAY O F G R E AT E R TO RO NTO

A Toolkit for Outcome Measurement:


Building on Logic Models

United Way gratefully acknowledges the financial support of the Jackman Foundation
and the Ontario Trillium Foundation, an agency of the Ministry of Tourism, Culture and
Recreation.The Foundation receives annually $100 million in government funding
generated through Ontario's charity casino initiative. It provides grants to eligible
charitable and not-for-profit organizations in the arts, culture, sports, recreation,
environment and social service sectors.
Copyright. United Way of Greater Toronto. Permission is granted for reproduction of this document
with attribution to the United Way of Greater Toronto.

Available from United Way of Greater Toronto


26 Wellington Street E., 11th Fl.,Toronto ON Canada M5E 1W9
Telephone: (416) 777-2001 l Fax: (416) 777-0962
Web site: www.unitedwaytoronto.org
United Way of Greater Toronto, 2004

TTY: (416) 359-2083

ISBN # 0-921669-32-1

Acknowledgements
The Outcome Measurement Toolkit was developed in collaboration with many ndividuals and
organizations. United Way wishes to thank Dr. Arnold Love, author of the toolkit, as well as Catharine
deLeeuw and Liane Greenberg, project managers. United Way also gratefully acknowledges the local
community agencies that agreed to meet with us, describe their experiences with outcome
measurement, and share their data collection tools and techniques so that others may learn from and
build upon their efforts. Without these special people, this toolkit would not have been possible. Thank
you to the following organizations:
Abrigo Centre
Canadian Mental Health Association Metro
Canadian National Institute for the Blind
Canadian Red Cross (Toronto Region)
Community Information Toronto
COSTI
Davenport-Perth Neighbourhood Centre
Delta Family Resource Centre
Distress Centre of Toronto
Evas Phoenix
Family Day Care Services
Jewish Family and Child Service of Greater Toronto
JobStart
Partners for Access and Identification (PAID)
The Redwood Shelter
St. Christopher House
St. Stephens Community House
Street Health
Times Change Womens Employment Service
VHA Home Health Care
West Scarborough Neighbourhood Community Centre

Thank you also to the volunteers who helped ensure United Ways Program Effectiveness Organizational Development (PEOD)
initiative was relevant and practical for not-for-profit organizations. We extend our appreciation to all of the following: Anne
Babcock, John Bagnall, Debra Grobstein Campbell, Ed Castro,Yosi Derman, Julius Deutsch, Debbie Douglas, Maureen Fair, Ed
Graca, Maria Harlick, Rob Howarth, Alfred Jean-Baptiste, Jennifer Lau, Cliff Ledwos, Helen Leung, Ursula Lipski, Margarita Mendez,
Arlene Perly Rae, Jennifer Reynolds, B.J. Richmond, Sarah Rix, Trish Stovel and Gayle Waxman.

Table of Contents
Acknowledgements
1. A Toolkit for Outcome Measurement........................................................1
2. Building Evaluation Frameworks ..............................................................8
3. Sampling ..................................................................................................26
4. Analyzing and Communicating Outcome Information ..............................38

Data Collection Tools


5. Developing Effective Questionnaires.........................................................55
6. Conducting Effective Interviews ...............................................................90
7. Developing Effective Focus Groups ..........................................................116
8. Making the Most of Direct Observation .................................................137
9. Putting Program Records to Work ............................................................156
10. Applying Case Studies Method ..............................................................171

Appendices
Glossary of Selected Terms .........................................................................186

Worksheets:
Model Worksheet, Evaluation Framework ...................................................191
References ..................................................................................................193

Chapter 1:
A Toolkit for Outcome
Measurement

Building on Logic Models


When a community worker hears that her involvement helped someone,
she is motivated. When an agency watches how people change and grow
through a program they provide, they get energized. When an agency
follows changes over time and sees how the community has improved, it
feels empowered.
Outcomes can be powerful. Every day, non-profit organizations apply their
resources and skills to make life better for individuals, families, and
communities. Some work to ensure basic life necessities are accessible.
Some work to ease transitions in life, be it arrival in a new country, securing
a home or job, or coping with a personal crisis. Some seek to ensure a safe
and engaging place to live, with rich opportunities for all ages. Whatever the
mission of a community organization, believing that they are making a
difference is what draws them together and keeps them moving forward.
But how can we tell if we are making a difference? How do we know that
we are having an impact? Or, what if we only think we are helping but
nothing really changes? How will we know? What should we change?
Nothing is more compelling than a personal testimonial. Community
agencies can readily demonstrate program effectiveness with personal
stories. But how do we know these individual successes are commonplace;
that they are the rule rather than the exception in a program?

Outcome Challenges
The answers to these questions lie in program outcomes. Monitoring and
evaluating changes and benefits of social service programs provide
information to help agencies answer these questions. For themselves to
celebrate successes and respond in a timely way to opportunities for
improvement. For the community to recognize and support the work of
effective organizations.
Program outcomes are often difficult to evaluate. Consider these examples.
How do we measure improvement in self-esteem? How do we know if a
school readiness program helped children succeed in the early grades of
elementary school? How do we know if our home support program
improves the well-being of seniors?
These real challenges were the motivations behind Program Effectiveness
Organizational Development (PEOD), an initiative of the United Way of
Greater Toronto (with generous support from the Ontario Trillium Foundation)

Forward

launched in the fall of 2000. PEOD is a training and support program aimed
at building the capacity of community agencies in program evaluation. PEOD
introduces to community agencies the ways and means of monitoring and
measuring outcomes and helps them to demonstrate achievement of
program outcomes and facilitate continuous program improvement.
PEOD was a response to significantly changing accountability demands
pressing on the non-profit sector from government and charitable supporters
alike. Increasingly, funders were expecting non-profit agencies to
demonstrate the good effects the impact they were achieving with the
individuals, families and communities they served. Accounting for the
amount of service delivered and to whom it was delivered was no longer
sufficient. Yet, by and large, community agencies were not given the
support and training to meet these new reporting requirements.

Training in Outcome Measurement


Through PEOD, the United Way offered one-day workshops in outcome
measurement and program logic models as the first step in raising
organizational capacity in evaluation.
Over three years, nearly 450 staff and volunteers from 200 communitybased agencies have been trained, from senior managers through to frontline personnel. In addition, United Way offered specialized training for
members of Boards of Directors, attended by over 100 volunteers from 45
community agencies.
Agencies have confronted the challenges presented by outcome
measurement in many ways. Agencies have provided additional training to
their staff, organized staff and committees to coordinate implementation,
applied logic modeling in program areas, and developed or adapted
evaluation instruments to collect outcome information from the people and
communities they serve.

Logic Models: A Beginning


PEOD training began with program logic models. Logic models are
extremely useful tools for all types of evaluation. Logic models are a means
of expressing how a program is expected to work, how the activities are
expected to benefit and improve life for the individuals, families, and
communities involved. In a logic model, all of the components of a program
are identified the inputs (resources), activities, outputs, and outcomes in
a way that makes clear how they combine to stimulate and support the
desired changes (see below for an example of a logic model).
Planning and undertaking an evaluation is easier if a logic model has been
created for the program. It helps focus attention on questions to be
answered. It helps identify what information is needed to obtain the
answers. Some common evaluation questions include:
l

Did we implement the program according to plan? What worked well,


what didnt work as expected? (Look at program records of inputs,
activities, and outputs.)

A Toolkit for Outcome Measurement

Did we reach the intended target population? (Examine program records or


survey program participants.)
Was the program effective did it achieve short-term outcomes? (Obtain
feedback from program participants, observe changes, review records,
etc.).

This outcome measurement toolkit assumes the reader has a working


knowledge of logic models and is ready for the next step. Many print and
web-based resources are available to explain the value and means of
developing logic models. Readers may wish to use the references and
glossary of selected terms included at the end of this manual.
Example of a Logic Model Employment Support Program
Inputs

Activities

Outputs

Short-term
Outcomes

Long-term
Outcomes

1
3

Forward

Purpose of the Outcome Measurement


Toolkit
This outcome measurement toolkit is all about helping agencies build upon a
program logic model to plan and implement an outcome-focused evaluation
realistically and practically. Rarely is outcome measurement (or any type of
evaluation for that matter) in a community organization uncomplicated. Each
agency must adapt textbook advice and examples to their unique
programs, settings and resources. This toolkit aims to help agencies figure
out several key questions such as: How do we measure our program
outcomes? and Do we have the resources to do it?
Outcome measurement usually does not replace an agencys existing
evaluation activities, such as needs assessments, implementation
monitoring and assessing client satisfaction. These remain necessary for
program management. However, implementing outcome measurement does
not necessarily mean agencies have to develop all new evaluation methods
and data collection tools. Some deliberate modification of existing records or
data gathering instruments may be all that is required. It may mean adding
or replacing a few questions on an existing client satisfaction survey,
exerting some administrative discipline to record observations routinely, or
modifying client record processes to ensure that relevant information is
documented formally.
This toolkit will help you choose what measurement tools would be most
practical for your program and community and will provide step-by-step
instructions to enable you to implement the tool in a way that assures
validity and reliability of results to the greatest degree possible.

Evaluation Frameworks: A Plan


Evaluations progress more smoothly when an organization has first created
an evaluation framework. This is a critical step that will enable you to
approach any evaluation exercise with a clear purpose, whether the focus is
on program outcomes or program activities.
An evaluation framework is a work plan. It is a concise document that
outlines the purpose of an evaluation, the questions that will be answered,
and the information required to answer these questions. It can be thought of
as the guide to conducting your evaluation. It details the specifics about the
type of information to be collected (i.e., indicators), how the information will
be collected, when, how often, and by whom, as well as who is responsible
for compiling and analyzing data and communicating the results to various
internal and external audiences. Evaluation frameworks provide a bridge
between a program logic model and the methods of data collection.
Our discussion of evaluation frameworks will explore how to create and use
them to answer key outcome measurement questions: Is what were doing
making a difference? Are the people and communities we serve in a better
position now than before? In what ways? How might we do better?

1
A Toolkit for Outcome Measurement

How to Use this Toolkit


Readers of this toolkit will bring different interests and priorities to their
search for practical and manageable ways of demonstrating program
effectiveness. In this toolkit are explanations of how to collect information
about programs using a variety of tools. All of the data collection methods
described herein may be used for many types of program evaluation, but our
focus and examples will be on outcome measurement. This toolkit has been
organized into a series of chapters. Some of you may choose to read all of
them in sequence, but more likely, readers will select the chapters that
relate to their chosen data collection methods.

Essential for Every Toolkit User: Chapters 2, 3 and 4


We recommend that all readers of this toolkit begin by reviewing thoroughly
Chapter 2: Building Evaluation Frameworks. The first step to a successful
and relevant evaluation exercise is always the preparation of an evaluation
plan! Creating an evaluation framework enables you to focus the evaluation
questions, identify the information you need to collect to answer these
questions, and lay out the process to collect, analyze, and utilize the
information.
Two other chapters offer guidance on general evaluation issues and should
be reviewed in conjunction with others that discuss specific tools. Chapter
3: Sampling addresses the question of how to select who or what to
include in the evaluation process, to make it manageable and still ensure
credibility of the results. Chapter 4: Analyzing and Communicating
Outcome Information tackles how to find meaning in the data and how to
use it internally, for continuous program improvement, and externally, for
community and funder relations. Even though analysis occurs only after the
data is collected, we have chosen to place this chapter near the beginning of
the toolkit because thinking ahead about how the information will be used,
and who will use it, will help in planning the evaluation. Read these three
chapters as you begin to develop and use outcome measurement tools.

Data Collection Methods: Chapters 5 through 10


There are six chapters that describe data collection methods that are
workable in community organizations.
Chapter 5: Developing Effective Questionnaires instructs on how to write
and organize questions related to program and community outcomes. Begin
with this chapter if your organization uses a data collection tool that relies on
asking people questions.
Chapter 6: Conducting Effective Interviews and Chapter 7: Developing
Effective Focus Groups explain how to ask people questions face-to-face or
over the telephone, individually and in groups.

1
5

Forward

Chapter 8: Making the Most of Direct Observation explores how what


you see and hear can be captured as part of an organized evaluation process
to measure program outcomes.
Chapter 9: Putting Program Records to Work describes how to maximize
existing agency information to support outcome measurement. This involves
extracting information (data mining) but may also involve creating or
modifying program records.
Chapter 10: Applying Case Studies Method explains how to gather and
utilize outcome information from case studies about programs and program
users.

Examples of Successful Outcome Measurement


This outcome measurement toolkit is different from other evaluation
resources in that it draws upon existing outcome measurement tools used
by local social services agencies. We wanted to reinforce the message that
outcome measurement is indeed doable, even under difficult
circumstances. Through interviews with local agencies, we uncovered data
collection tools and techniques that are creative, practical and highly
workable. Many are not textbook in approach and yet they present valid
and reliable approaches to collecting outcome information under challenging
real-life situations. These agency examples anchor the description and
usefulness of different tools and processes in each of the toolkits chapters.
Look for the Agency in Action inserted in the introduction of each chapter
to discover how one organization applied the tool successfully in their
community. As details of tool development and implementation are
explained, additional examples are provided to bring clarity and meaning to
each step of the process. Be sure to look for Case in Point, a brief
demonstration of a key point by a local agency.
Examples of outcome measurement are assembled at the end of each
chapter on the various data collection tools are drawn from local agencies
that provide programs and services to diverse communities. These tools
were selected because they represent good practice-success in
accommodating the reality within the agency and community with the
demands of implementing the chosen tool. We hope you will draw
inspiration, confidence and even concrete elements to construct tools for
your own use.

Appendices
A Glossary Of Selected Terms and Worksheets distributed by United Way
during its training on outcome measurement and logic models is appended
to the toolkit. A list of references to facilitate further exploration of
evaluation and outcome measurement is included as well. We hope that this
outcome measurement toolkit provides you with sufficient understanding
and confidence in using evaluation within your organization meaningfully
that you will be assured in using and discussing the concepts, vocabulary
and techniques of evaluation within your organization and in your community
relations.

1
A Toolkit for Outcome Measurement

At the same time, we hope that you will bring your experience and
judgment to the application of outcome measurement, to adapt tools and
techniques sensibly to suit your unique settings, so that your organization
can gain valuable information that will support its work in bringing positive
changes to the members of its community.

1
7

Forward

Chapter 2:
Building Evaluation Frameworks

What Is An Evaluation Framework?


An evaluation framework is a tool for planning an evaluation. It makes the
evaluation process manageable by bridging the gap between the evaluation
questions that you want answered about your program and the methods you
will use to collect the data. An evaluation framework also enhances the
relevance of evaluation for decision-making by identifying your key
information needs, helping to set evaluation priorities and assisting you in
making the best use of scarce evaluation resources. An evaluation
framework builds support for the evaluation by showing clearly the rationale
for the evaluation. Finally, an evaluation framework increases the likelihood
that the evaluation findings will be disseminated to key stakeholders and
used by them to strengthen programs by helping you plan how you will
communicate your findings in the best way.

Agency in Action
The Canadian Red Cross (Toronto Region) develops evaluation frameworks as part of
their Quality Plan process. For each program, an evaluation framework summarizes the
specific data needed to demonstrate that client needs are being met and that clients are
developing the capacity to live the life that they want to live, the sources of the data and
the data collection method.
In this chapter, we will use an example from the Meals on Wheels program of the
Canadian Red Cross (Toronto Region) to illustrate the key components of an evaluation
framework.

Why Is an Evaluation Framework


Important for Outcome Measurement?

Some of the most difficult challenges faced by agencies when measuring


outcomes is deciding what to evaluate, what measurement tools are needed,
and how to organize the evaluation process especially if the agencies are
evaluating more than one program. One of the most effective ways of deciding
what to evaluate is by working with program stakeholders to discover the specific
evaluation questions that they want answered. For outcome measurement, the
evaluation framework is a practical tool that focuses the evaluation on important
questions related to program outcomes. Starting with the key evaluation
questions, the evaluation framework then provides an easy-to-apply format for
spotting the sources of data and evaluation methods needed to measure
outcomes. Moreover, the evaluation framework is a convenient way of planning
an evaluation and the strategies you will use to communicate your findings.

A Toolkit for Outcome Measurement

What Are the Strengths of an


Evaluation Framework?
l

Identifies program stakeholders. The process of building an evaluation


framework starts with identifying those with a stake or investment in
the program. Involving stakeholders in planning the evaluation helps to
build ownership for the evaluation, reduce possible evaluation resistance
and improve the utilization of the evaluation findings.

Facilitates participation in the design of the evaluation. The


process of building an evaluation framework opens the way to
participation. Stakeholders who have a deep interest in the program
engage in discussions about the program and its outcomes, identify
important questions to answer and develop consensus about data
gathering methods that are relevant and useful for them.

Focuses the evaluation. An evaluation framework encourages


stakeholders to focus their evaluations on key evaluation questions. In
turn, this process makes the best use of time and resources by
concentrating them on only answering essential questions and the
evaluation tools needed to gather high-priority data.

Links program logic models directly to the evaluation process.


An evaluation framework develops easily when based on a program logic
model. The logic model gives a comprehensive overview of the program
its strategy and its outcomes and purpose.

Case in Point
In preparing for an evaluation of its Workplace Mentorship Program, staff at Evas Phoenix
developed a logic model to establish a clear overview of the program.The logic model
clarified key project client outcomes, a critical step in defining appropriate outcome
indicators.The logic model also helped to guide the development of a questionnaire that
allowed each youth to give confidential feedback about the impact the workplace mentor
had on their job placement.

What Are the Limitations of an


Evaluation Framework?
Generally speaking, an evaluation framework works very well to plan
evaluations and presents few problems in most situations. It is worthwhile,
however, consider whether the following potential limitations apply to your
situation:
l

Focuses evaluation on a limited set of questions. The use of an evaluation


framework assumes that you understand your program fairly well,
including its outcomes, resources, activities and the theory of how your
program activities produce outcomes. An evaluation framework also
assumes that the stakeholders can identify the types of evaluative
questions that they want to answer. If these conditions are met, then an
evaluation framework works very well to focus the evaluation. You may be
in a situation, however, where you know little about a program or where
you are exploring different program options in such cases, a broad

Evaluation Frameworks

descriptive approach, one that employs multiple methods to capture a


wide variety of information about your program, may be more
appropriate.
l

Makes planning an evaluation a linear process. Some individuals complain


that building a logic model and evaluation framework assume that the
program model is stable, and that an evaluation steering committee can
identify its information priorities and design the evaluation in a step-bystep fashion. They argue that evaluation plans are not linear at all, but
require a cut-and-try process that gradually builds an evaluation plan
from program experiences over time.
Adopts a practical approach that might compromise the technical
adequacy of the evaluation. An evaluation framework is developed
through a participatory group process. The group may be strongly
influenced by practical concerns, such as workload and the cost of
evaluation. The give-and-take of the group process may compromise the
technical adequacy (validity, reliability) of the evaluation by seeking to
meet the diverse needs and viewpoints of the group members.

Main Strengths and Limitations of Evaluation Frameworks

Strengths
l

Limitations
l

Focuses evaluation on a limited


set of questions

Identifies program stakeholders


l

Makes planning an evaluation


a linear process

Facilitates participation in the design


of the evaluation
l

Focuses the evaluation


Links program logic models
directly to the evaluation process

Adopts a practical approach that


might compromise the technical
adequacy of the evaluation

Ten Steps for Building an Evaluation


Framework
The sections below outline the major steps for building an evaluation
framework. It concentrates on the key components of the evaluation
framework, as well as the use of a participatory process to focus the
evaluation, set priorities, select evaluation methods and draft evaluation
and communication plans.

Step 1. Establish an Evaluation Work Group

To help you draft the evaluation framework, set up a work group. You
should select members who represent the major stakeholders in the
program. In this way, you can ensure that their views are heard and that
the evaluation is designed in a manner that meets their information needs.
Having the work group plan and oversee the evaluation increases
cooperation ensures that the evaluation is focused on relevant

A Toolkit for Outcome Measurement

10

TEN STEPS FOR BUILDING


AN EVALUATION FRAMEWORK

M
s

issues/questions and encourages the use of evaluation findings and


recommendations in program decisions. A work group involves a
participatory approach, increasing the ownership of all stakeholders in the
evaluation process. Without ownership, agency managers, staff and clients
are more likely to show resistance to the evaluation process.
As shown in the following table, evaluation work group members can
include managers, staff, clients, service partners, consultants and others
who have a strong interest in the program.
The work group can also play an important role in managing the evaluation,
reviewing evaluation findings and recommendaations, facilitating the
communication of evaluation results and recommendations and formulating
action plans based on the evaluation findings.
Members of Evaluation Work Group and Their Roles

Step 2. Develop a Program Profile


The next step is to develop a brief but accurate picture of the program,
called a program profile. A program profile answers the following questions
in one or two pages:

11

Evaluation Frameworks

What is the name of the program?

Who are the clients (target group) of the program?

What are the major goals of the program?

What outcomes (benefits for clients) does the program want to achieve?

What is the model of service delivery (how do program activities produce


outcomes for clients)?

What seems to be working well and what is not?

What are the important evaluation issues/questions to be answered now?

This should not be a big job. Usually this basic information is readily found in
program descriptions, agency plans, annual reports, or funding proposals
and reports to funders. It is simply a matter of organizing the existing
information under the questions listed above. Other sources of information
include:
l

program documents (plans, reports, files, legislation, needs assessments)


literature reviews

program data (program files and databases)

discussions with managers, staff and clients

observations of program/staff in action

Step 3. Develop a Logic Model


Now that you have constructed a program profile, it is a good time to
develop a logic model. Program logic models are a way of expressing how a
program is expected to work how the key activities are expected to benefit
and improve life for the individuals, families and communities involved.
This section provides only highlights about logic models. Use some of the
references listed at the back of this toolkit if you need more information
about creating program logic models. Figure 1 presents the basic elements
of a logic model.
Figure 1: Basic Elements of a Logic Model
You develop a logic model with your work group, using the Logic Model
Worksheet found in Figure 2. Begin by identifying the outcomes for
program participants the benefits participants receive from the program,
such as new knowledge and skills, higher motivation, improved attitudes,

2
A Toolkit for Outcome Measurement

12

changes in behaviour, or improved conditions. Next, identify the activities


that the program uses to create those outcomes, the resources needed to
support those activities and the service efforts or outputs that are generated
by the activities to achieve the intended outcomes.
Figure 2: Logic Model Worksheet
Consider also the assumptions underlying the program strategy what
conditions do you require to make success happen? Also, identify contextual
factors that may influence the desired outcomes, positively or negatively.
List these factors in the situation box. They will help you assess whether
your outcome expectations are realistic. They will also be important later
when you analyze and interpret your evaluation findings.

Step 4. Identify Key Evaluation Questions


Use the program logic model to help you focus attention on key questions
that you need answered. The work group should discuss what information
they need most about the program. What do they know already? What do
managers need to know? When? What do Boards and funders need to
know? Many things may be interesting - but some things are critical to
know before decisions are made.
How do you determine the key evaluation questions? Figure 3 shows that
programs typically go through a developmental cycle that begins with
assessing the need for the program, moves next to the planning stage, then
progresses to program implementation, and finally after a period of time, to
the point where it is generating outcomes and results.

13

Evaluation Frameworks

Figure 3: Program Developmental Cycle


Specific evaluation questions are commonly asked at each developmental
stage. For example, at Stage 1, important evaluation questions include:
l

Who needs the program?

What kinds of services does the target group need?

Are these services available to the target group right now?

Some important evaluation questions at Stage 2, when a program is being


planned and designed, include:
l

What are the major goals of the program?

What program activities are needed to achieve the goals?

What resources are required for this program?

Once the program is being implemented, Stage 3 evaluation questions


include:

Is the program being implemented as planned?

Who is actually using the program?

Are participants satisfied with the program?

How can program operations be improved?

For outcome evaluation, most of the questions relate to Stage 4. Once the
program has attained sufficient maturity to deliver outcomes, key evaluation
questions include:

A Toolkit for Outcome Measurement

14

Is the program producing the intended outcomes for participants?

What unintended outcomes or consequences is the program producing?

Is the program meeting its service delivery standards?

Are the program outcomes worth their cost (cost-effectiveness)?

Are there more effective or less costly ways of delivering the services?

Through a process of discussion, the work group sets evaluation information


priorities and identifies the most important question(s) to answer.
At this point, make use of a worksheet to organize your thinking and planning
of the evaluation. The Evaluation Framework Worksheet is a series of columns,
each representing an issue that needs consideration and a decision-making
point. Enter the evaluation questions for which you are seeking answers in the
first column.
Lets take an example from the Meals on Wheels program of the Canadian
Red Cross ( Toronto Region). The program logic model showed that the shortterm outcomes were participants maintain or improve their health and
remain functionally independent. The work group wanted to know if the
program produced these benefits, so it asked: What were the effects of the
program on participants health and independence?
Figure 4: Example of Evaluation Question for Meals on Wheels Program

Step 5. Identify Indicators


The next step is for the evaluation work group to agree on the specific measures
needed to answer the evaluation question(s) you identified in Step 4.
Indicators are the evidence that you observe or measure to determine what
you wish to know. They are the specific measures you will use to answer
your evaluation questions. They provide you with some information that will
help you judge whether the changes you expected occurred.
Indicators can be quantitative (numbers) or qualitative (narrative) or a
combination of both. For most community agencies that deliver human
services, a mix of quantitative and qualitative indicators serves to give a
more accurate picture of the programs performance.
Select indicators that can measure the types of changes or improvements
that are needed to answer your evaluation question(s). Usually, indicators
include measures such as:

15

Evaluation Frameworks

l
l

Number of...(number of participants who exercised)


Percent of...(percent of participants in your program who live
independently)
Difference in...(difference in health status before and after program;
difference in feelings of loneliness before and after program)

Proportion of...(proportion of participants who exercise 3 times a week)

Incidence of...(incidence of depression in program participants)

Increase in...(increase in mobility of program participants; increase in


sense of well-being)
Decrease in...(decrease in falls by program participants; decrease in
feelings of isolation)

Try to identify between one and three indicators for each question you want to
answer. You may need more than one indicator to answer your evaluation question
adequately but you also want to avoid becoming bewildered by an unwieldy
number. This is a guideline only; you may select more or less indicators depending
on how easy and practical it is to collect the data and how credible your
stakeholders will consider them.
Double check that your indicators meet these criteria. A good indicator is:
l

Clear and unambiguous

Observable and measurable

Relevant

Reliable

Practical

Figure 5: Examples of Indicators for Meals on Wheels Program

Step 6. Identify Sources of Data

At this point, the work group should examine each indicator and determine
the best source of data. There are three main sources of data: people,
records and observations.

A Toolkit for Outcome Measurement

16

The Meals on Wheels Program involves volunteers, leaders, trainers, staff


and participants in the program. Participants may also have family members,
caregivers, physicians and other service providers who are able to assess
changes in the indicators. The work group must decide who is in the best
position to provide the desired data and how accessible they are.
Data sources are not always people. Very commonly, data to measure
indicators, comes from program or agency records. Data also may come
from external sources, such as the records of other agencies, professionals
(e.g., health care providers), government records and population surveys.
The work group must assess carefully the feasibility of obtaining information
from these sources.
Observations by staff, volunteers, or others may provide valuable
information. Observations are possible for behaviour and non-verbal
communication, relationships and interactions between people, new skills
and physical artifacts, such as safety plans, rsums, posters, newsletters,
or videos.
The Meals on Wheels Program decided that the participants themselves
were the best source of data for measuring the indicators.
Figure 6: Example of Sources of Data for Meals on Wheels Program

Step 7. Select Data Collection Tools


Six data collection tools are discussed in this toolkit: questionnaires,
interviews, focus groups, program records, direct observation and case
studies. Tools can gather either quantitative data information expressed in
numbers or qualitative data information expressed in words, often
involving attitudes, perceptions and intentions.
Quantitative data collection tools are used to gather information from a large
number of persons, provide statistical summaries, or mine factual data from
existing databases. The most frequently used tools for collecting quantitative
data in non-profit organizations includes questionnaires administered in
person, by mail or through telephone interviews, data extracted from

17

Evaluation Frameworks

program records and direct observation of program activities. Qualitative


data tools collect in-depth information from individuals or groups of program
participants. Qualitative data is most commonly obtained from open-ended
questions, interviews administered in person or by telephone, focus groups,
direct observation and case studies.
Once your work group has an idea about possible tools, it should now select
the tools it will use to collect the necessary data by considering the
following factors:
l

Credibility Is the method subject to bias and manipulation?

Validity Does it measure what it is supposed to measure?

Reliability Is the measurement accurate and consistent?

l
l

Transparency Can the stakeholders and the public understand the


method and the data it produces?
Cost How expensive is this method?
Feasibility How realistic is this method, given non-financial issues or
constraints (e.g., difficulties in accessing data, obtaining informed
consent, finding transportation for focus group participants)?

Figure 7: Data Collection Tools for Meals on Wheels Program

Step 8. Decide Who Will Collect the Data

Once the working group has selected the data collection tools, now it must
decide who will collect the data. The first decision is whether to use persons
within your organization or external to your organization. Whether staff or
volunteers, persons within your organization have the advantage of knowing
your programs and clients. They are likely to be less threatening than
persons outside of the organization and cost less too. On the other hand,
there is always the risk of bias when someone who is employed by an
organization or deeply involved in it collects evaluation data. External people
often have specialized training in data collection and can work under contract
to complete the data collection in a timely fashion. They are usually
perceived as more credible, but they tend to cost more.

A Toolkit for Outcome Measurement

18

Community agencies often use the following types of people to collect data:
l

Research assistants who are on staff

Staff members from programs other than the one being evaluated

Staff members from other agencies (e.g., lead focus groups)

Volunteers trained to collect data

Students on placements who have research experience

l
l

Students who complete the evaluation data collection as part of an


assignment
External consultants
External survey or evaluation firms (e.g., conduct large-scale surveys or
complex data collection)

The Meals on Wheels Program decided to use trained volunteers to collect


data through a telephone interview.
Figure 8: Collects the Data for Meals on Wheels Program
Data
Sources

Step 9. Determine When to Collect the Data


The issue of when to collect data is related to the evaluation question, choice
of data collection method, and the following factors:
l

Stage of program development

Timing of data collection

Frequency of data collection

Stage of Program Development. We have already seen that programs tend


to have life cycles. In the early formative stages, when programs are being
planned, they require feedback about the adequacy of their program model to
produce the intended outcomes, information about the persons who are
participating in the program and accurate descriptions about program activities.

19

Evaluation Frameworks

As the program moves into implementation, managers and staff want


information about whether the program is being implemented as planned,
strengths and weaknesses of the program model in the real-world setting
and information for improving program performance. Once the program is
fully implemented, program senior managers, funders and the public usually
want summative information about the outcomes of the program and how
well it uses its resources.
Timing of Data Collection. Evaluations may gather data at different times,
depending on the evaluation questions being asked. Some common times
for data collection include:
l

Before the program begins, to gather baseline data

During pilot-testing of the program, to validate the program model

During program implementation, to monitor and improve program


performance

After clients have completed the program, to measure outcomes

Before-and-after clients have completed the program, to measure change

At follow-up, usually several months or more after clients have completed


the program, to measure intermediate and longer-term effects of the
program
Five years or more after clients have completed the program, to measure
program impact

Frequency of Data Collection. Depending on the evaluation questions that


need to be answered, you may collect data at different times. For the
purposes of measuring outcomes, the following are common frequencies for
data collection:
l

Only once, usually after clients have received services

Twice, usually before and after clients have received services

Several times, usually before, after and one or two follow-up periods

Periodically during the time clients are receiving services, usually for
children or for long-term clients (monthly, quarterly, annually)
Ongoing, each time a client receives service, usually if the client is
anonymous and service duration is short with a measurable outcome (each
call to information service, hot line, distress centre)

The work group should make the decision when to collect data in light of
these factors and the evaluation question that needs to be answered, the
data collection tool chosen and available resources.

2
A Toolkit for Outcome Measurement

20

Figure 9: When to Collect the Data for Meals on Wheels Program

Step 10. Develop Analysis and Communications Strategy


The final step in building an evaluation framework is developing an analysis
and communications strategy. Different stakeholders (managers, staff,
Board, partners, funders, public) require outcome data to be analyzed and
presented in specific ways before they use the information fully. It is
important to consider how results will be used and communicated before
you begin data collection as part of the process of selecting indicators and
data collection tools. If the audience that wants the data cannot understand
an indicator, or if a data collection tool cannot produce data when it is
needed, then the evaluation findings will not be used. Planning at an early
stage on how the data will be used will help you to analyze outcome data
and present your outcome information in a way that contributes to
organizational learning, decision-making and action.
Start by identifying who will use the evaluation findings. If you formed a
work group that reflects the major stakeholders in the program as we saw in
Step 1, then you probably already identified your major audiences for the
data. Consider adding additional audiences to these stakeholders, such as
the Board of Directors or a funding organization. It may be important to
communicate your findings to these audiences, even if they are not
represented on your evaluation work group.
Once you have identified your audiences, the analysis and communications
strategy provides a synopsis of the following for each audience:
l

Name of audience

Audience membership who is included in this audience

Scope of communication specific information required by the audience

Timing of communication quarterly, monthly, annually, Board meeting

Mode of communication.

21

Evaluation Frameworks

Regarding this last point, you will notice that the word communication is
used, rather than reporting. The reason is that reporting is associated with a
lengthy written report and formal presentation. Current evaluation practice
emphasizes the choice of communication strategy matched to the needs of
the audience. For example, the Board may want a one-page summary and
the opportunity to ask the evaluator questions about the findings. Managers
may prefer monthly statistical summaries. Front-line staff may want detailed
information about clients comments. In other words, the communication
plan should identify what information each audience needs and the best
ways to communicate it.
The Canadian Red Cross (Toronto Region) includes program evaluation as
one of several practices to improve continuously the quality of services and
achievement of outcomes. The Quality Management Plan specifies that the
findings and recommendations from the annual client surveys of the Meals
on Wheels Program will be communicated in project reports to managers.
The findings are also communicated in quarterly status/update meetings
with program funders. They are also communicated to the Program Audit
Team bi-annually. Figure 10 shows how this is entered in the Evaluation
Framework Worksheet.
Figure 10: Communication Plan for Meals on Wheels Program

2
A Toolkit for Outcome Measurement

22

Examples
Building Evaluation
Frameworks
2A Program Review Framework Canadian Mental Health Association - Metro
2B Logic Model for Workplace Mentorship Program Evas Phoenix
Note:
the formats of the following outcome measurement tools have been altered to some degree
to correspond with the design and space restrictions of this toolkit. We hope you will draw
inspirations, confidence, and even concrete elements for your own use, drawing upon the
steps and guidelines set out in these pages.

2
23

Evaluation Frameworks

Example 2A
Canadian Mental Health Association-Metro
Program Review Framework
Goals For Individual Program Reviews

To determine if the program is operating as expected in terms of who is served, how the service is provided,
and incorporation of best practices and outcomes.
To collaborate with program staff in discussing options/actions in relation to key issues.

1.

Introductory Overview
Staff will provide a description of the program, its staffing compliment and a summary of changes that
have occurred since the last review.

2.
(i)

The Four Questions


Who is the program serving? Quality Indicators:
Number of people served during reporting period
Is the program meeting its mandate to serve people with serious and persistent mental health problems?
Is the program serving its target populations? i.e. diagnosis, presenting problems, etc.
Is the program accessible? i.e. offering portable intake, assertive outreach, and reflective of the community at
large?

(ii) What is the program model and are services delivered consistent with the model?
Case studies demonstrate flexible, individualized service delivery

Staff/client ratios are consistent with program model

Identify staff training received

Percentage of staff time spent in direct/indirect service

Waiting list

Average cost per client

Staff turnover

Quality Indicators:

(iii) Are program services in keeping with best practices? Quality Indicators:
Continuity of care is provided and is a program priority

Recovery is promoted and demonstrated in case studies provided

Services to clients are voluntary

Program demonstrated partnerships and collaboration

There is demonstrated evidence of client input into program planning

(iv) What outcomes is the program measuring? Describe and present them. Quality Indicators:
Outcomes related to program/client objectives i.e. accessing housing, jobs etc.

Hospitalizations

3.

Discussion
with staff, committee members and relevant others (clients, family/significant others, etc)
about specific issues. Staff will identify program strengths, areas for improvement and present a work plan for
discussion. People will identify options and actions together.

4.

Summary
We also identified some goals for each individual program review.

2
A Toolkit for Outcome Measurement

24

Example 2B
Evas Phoenix
Workplace Mentorship Program Logic Model
Situation: (e.g. problem/ need opportunity/ mandate/ goal)
Providing youth with Mentors at their work placement to enhance their chances of a succes.

How?

What We Want

INPUTS

ACTIVITIES

Employee of
organization
willing to be
a mentor

At least 3 evaluations
Meeting/Phone
calls w/workplace completed during
the placement
mentor

Employer
Suport
Worker

Coach both
Weekly conversations Youth and mentor form
Mentor and youth with both youth and
supportive partnership,
mentor for at least the
first month of
placement and then as
needed for the rest of
the placement

Employer

PROJECT:
Workplace Mentorship

Evaluations

OUTPUTS

SHORT-TERM
OUTCOMES
Youth feels
comfortable in
the job placement

At least 3
Youth sees the mentor as
evaluations completed someone they can confide in
during the placement and/or seek assistance from
on the job site

TIMEFRAME:
1 Year

Why
MEDIUM-TERM
OUTCOMES

LONG-TERM
OUTCOMES

Youth completes the


placement and feels that they are
better prepared to find and keep employment
Youth participates in
mentorship programs
at Evas Phoenix

Youth has a positive


work experience

Youth fits in with the rest of


the team on the job site

Youth feels that they were an important,


contributing member of the organization

Employer expresses satisfaction


with the youths performance

Employer is happy with the youths


work and either employees the youth
or is willing to be a reference

Youth has
sustained
employment in
the career they
have chosen.

2
25

Evaluatiion Frameworks

Chapter 3:
Sampling

What is a sample?
Program evaluation and outcome measurement involve collecting information from
all or some members of a population. Population usually refers to a collection of
people, but it can also refer to a collection of households, parks in a neighbourhood,
schools, agencies, records, observations, telephone calls or other items/units of
interest. A sample is a portion of a population and sampling is the process used to
select that portion.
When a population is small, sampling is not always necessary. Instead, it may be
possible to collect information from all members of the population. Whether or not
you sample will be a decision you make based on the size of the population, what
you want to know and the resources you have available.

Why Sample?
l

It is often cheaper to gather data from a sample rather than the


whole population, especially when the population is very large. From a
relatively small random sample, it is possible to obtain accurate estimates of the
actual characteristics of a much larger population, but at a lower cost than
conducting a full census. For example, a sample of as few as 1,500 persons can
yield accurate estimates of the characteristics of 30 million Canadians.

Sampling provides for the quick production of data. By sampling, the


agency staff can reduce data gathering and analysis time and can produce results
in a timely manner for decision-making. For example, a needs assessment survey
may be needed to identify emerging needs by an agency after a crisis situation or
an evaluation survey may be required to measure changes in attitude and
behaviour following a public awareness campaign.

Sampling may improve validity of data. Collecting data from a smaller


number of persons (or units) may free time and resources to reduce other
potential sources of error in the data collection process. For example, more time
can be given to careful survey planning, questionnaire development, training staff
for accurate data gathering and data coding and follow-up with non-respondents.

Sampling may improve accuracy of data. A policy of carefully sampling


clients is respectful of peoples time and can improve response rates and
accuracy of responses as it may limit respondent fatigue, a situation that
develops when people tire of answering questions and providing feedback. In an
age when market researchers, political pollsters and retailers demand personal
data and our views on an endless variety of topics, the burden is even greater for
those who receive services from government and non-profit organizations.

A Toolkit for Outcome Measurement

26

Special Considerations When Sampling


l

Small numbers. Many non-profits operate programs that serve


relatively small numbers of participants. In these situations, data
collection from everyone (or every unit) in the population a census
may be preferable and feasible. Sampling would yield little savings in time
or money and the chance of error in the results would be greater.

Case in Point
VHA Home Health Care, an agency providing a range of home support services to seniors
living in the community, offers an Extreme Cleaning Service for persons who need intensive
cleaning of their living places. As part of the program evaluation, all 28 referral sources were
contacted and asked to complete a telephone interview or questionnaire sent by mail or fax.
The majority chose the telephone interview, which took on average only 10 minutes to
complete.The response rate to this census approach was 88 percent.
l

Many sub-groups in the population. Sampling may not be a useful


strategy when detailed information is needed from multiple sub-groups in
the population of interest. Numbers in each sub-group may not be great
enough to allow for reliable sampling. For example, Davenport-Perth
Community Centre operates several after-school and summer camp
programs. Staff may be interested in knowing if and how satisfaction
with these programs varies by client age and client participation in
specific activities. Obtaining accurate data may require asking all
participants and their parents/guardians to complete a client satisfaction
form, rather than just a sample.
Non-response/Self Selection. Some people decline to give information
when asked for feedback. It is not unusual to have responses from only
10% to 15% for mail surveys in other words, between 85% and 90%
do not respond. When a large proportion of those approached for
information do not respond, your sample is biased; it represents only the
individuals who self-selected to respond. In this type of situation, the
data would not support useful estimates of the whole population. Even
worse, the results may lead to erroneous conclusions about the
population of interest. There are some strategies to minimize nonresponse. For example, you may decide that it is more important to
select a smaller sample and then use resources to increase response
rate, perhaps by providing incentives. You may decide to have intensive
telephone follow-up or personalized follow-up to produce higher response
rates. Or you may elect to offer an incentive, such as a small honorarium
or free movie tickets.

Case in Point
Dealing with low response rates can be difficult. Over the years, Jewish Family and Child
Service of Greater Toronto (JF&CS) tried many strategies to improve response rates for the
Family Services Outcomes Measure, which involved completion of a questionnaire before and
after service. Even with a call from volunteers to remind clients, the response rates hovered
below 50%. After much experimentation, JF&CS implemented a new approach staff
distribute and collect the questionnaires during sessions with clients. Now, the before-service
response rate is 95% and the after-service rate is over 60%.

3
27

Sampling

Cannot define the population. To ensure unbiased information, you must


identify on a list the members (units) of a population, before selecting a
sample. Often, this is not possible for non-profit organizations; for
example, programs which provide services on a drop-in basis, many of
which neither register their participants nor expect to serve attendees
regularly. In such situations, organizations must balance the need for
unbiased information against the reality of access to the population. Most
often, organizations choose to collect some information rather than no
information at all. There are two choices. For instance, you could redefine
your population such that the population represents homeless individuals
who use the program over a short period of time (rather than all homeless
individuals in the area). Because some homeless people go to shelters or
drop-in centres, a list may be created based on homeless people in a
shelter on a specific day, and then a sample drawn from that list. Results
from the evaluation would be helpful in generalizing to a limited population
of the homeless. Alternatively, you could develop a non-representative
sampling strategy and analyze the data cautiously, mindful of the potential
gaps or misrepresentations in the information.

Two Approaches to Sampling


Over the years, strategies have been developed for taking maximum
advantage of the strengths of sampling and avoiding its limitations. There are
two standard ways to draw samples from a population.
Probability sampling is used when the purpose of the evaluation is to
generalize from a sample to the entire population.
Non-probability sampling is used when the purpose is to learn about
individuals in the population or when probability sampling is not feasible. This
chapter will help you to better understand which type of sampling is
appropriate for your evaluation.

Defining the Population


Defining the population of interest is the essential first step in selecting a
sample. Consider what factors describe the population clearly. This process
includes three parts:
1. Identifying the group of interest
2. Naming the geographic area (or extent) where the group exists
3. Deciding the time period of interest.
Taken together, these three parts form a well-defined set of eligibility criteria
that every member of the population must meet.

Probability Sampling

Probability sampling is necessary when you want to generalize from the


sample to the whole population. That is, when you want to estimate the
characteristics of the whole population based upon what you learn about the
sample. Probability sampling makes use of statistical methods or random
selection to draw a sample.

A Toolkit for Outcome Measurement

28

It is unbiased and does not depend on personal judgment. Random selection


ensures that each unit of the population has an equal and independent
chance of being selected.
The random selection process has independence because the selection of
one person (or unit) does not have an effect on the selection of other
persons (or units).

Define the Population


1. Define the basic elements or characteristics
of the group of interest.
2. Describe the geographic boundaries of the
population.
3. Define time.

Program Example
Teen mothers 13-19 years of age.
Teen mothers living within the catchment
area of the agency.
Teen mothers who gave birth in

The word random can cause confusion. Evaluators often describe a sample
as random because they chose 15 people at random. This is not what is
meant by random sample picking people who happened to be available is a
convenience sample. Although this is one way to get feedback and it is
better than no feedback at all, the results from this form of pseudorandom sample will not represent the population. (See non-probability
sampling below).

Sample Size and Error


Whenever a random sample is drawn, only that part of the population that is
included in the sample is measured, and this is used to obtain estimates
regarding the entire population. Consequently, there is always some error in
the data. The larger the sample size, the more the population is represented,
and therefore, the more accurate the estimate.
No two samples will produce exactly the same results. Probability sampling
allows you to estimate sampling errors, also known as margins of error. In
general, you can reduce sampling error by increasing sample size. If the
population is small, the sample size needs to be fairly large to minimize error
and improve the confidence level that the results do in fact represent the
whole population. On the other hand, when the population is very large,
sample sizes can be proportionately smaller. For instance, for a population of
1000, a sample of 280 would be sufficient to yield representative results
with a small (5%) margin of error. A population of 2000 requires a sample of
approximately 335. Even though the population is twice as big, the sample
size increases only by 55 to achieve the same margin of error an indication
that sampling can make good use of resources when you have a large
population to survey.
On the other hand, a small population of 250 needs a sample of 155 to
produce the same level of confidence that the results accurately represent
the target population (i.e., that results can be generalized to the target
population).

29

Sampling

As you can see, the advantages of probability sampling diminish as


population size decreases. At some point, it may be just as easy to conduct
a census and collect information from every unit in the population.
Detailed direction about sample size can be found elsewhere in evaluation
texts, on the internet or consult with an expert evaluator.

Sampling Frames
Probability sampling requires that you design a sampling frame. A sampling
frame lists each individual member of the population and is used to
eliminate bias and reduce error when drawing a sample from the population.
A sampling frame must meet several criteria to avoid coverage error. It must
be: complete, current, accurate, free of duplicate names, and absent of any
pattern in the way the names/members are listed (i.e., alphabetical).
Proper coverage is achieved when a sampling frame includes all (and only) the
members of the population. Error is associated with under-coverage or overcoverage and can result in patterns that skew results. The more different the
population members excluded from the list are from those included on the
sampling frame, the greater the coverage error.
For instance, under-coverage may occur if you were to construct a sampling
frame of your clients based on their home addresses. The sampling frame
would consist only of those clients who had fixed addresses, not those who
are transient or homeless. That is, it might be an incomplete list because
clients without a fixed address would not be on the list. The error would
occur because clients without fixed addresses are more likely to be poorer,
have insecure shelter and food and experience more risk factors than other
clients. An evaluation that did not include these clients may give an incorrect
estimate of the type and severity of the needs of all clients.
Over-coverage occurs when people who should not be included in the
sampling frame are included. For example, some clients on your lists may
have moved from the geographic area and should therefore not be included
in the sampling frame.
For most non-profits, sampling frames usually come from administrative and
program files. A simple example: if you were interested in finding out the
average education level of volunteers in your agency, the sampling frame
would be the list of all volunteers.

Strategies for Probability Sampling


If you decide that probability sampling is the right approach and is workable
in your situation, then you need to choose a specific sampling method for
use with the sampling frame that you have prepared. The following section
outlines the four major probability sampling designs.
l

Simple Random Sampling

Stratified Random Sampling

Systematic Sampling

Cluster Sampling

3
A Toolkit for Outcome Measurement

30

Simple Random Sampling


Simple random sampling is the most common way of making sure that you
have a truly representative sample. Selecting winners in a lottery is a familiar
example of simple random selection.

To take a simple random sample:


1. Assign a number or identification code to each person (unit) in your
sampling frame.
2. Generate a sequence of random numbers or codes by using a list of
random numbers or a random number computer program. There are lists
of random numbers available in evaluation textbooks or on the internet.
3. Draw a sample by applying the random numbers or codes to the list,
ensuring that you take as large a sample as workable to reduce sampling
error.
If the population is small, numbers or names can be drawn from a hat.
Record the number or name that is drawn. To ensure that each unit has an
equal and independent chance of being selected, put the drawn number or
name back into the hat after recording it. If it is drawn again, simply
disregard it and continue.

Systematic Sampling
If you have a fairly large population, you can save time by using systematic
sampling rather than simple random sampling. Systematic sampling is easier
because only one random number is needed to start the process. It can also
be more accurate because it selects the sample from the population in an
even way. However, it works best if there is no pattern in the sampling
frame, such as alphabetical listing of participants or grouping by program
time. Randomize or shuffle the sampling frame before starting.
1. Decide the sample size for the evaluation.
2.Divide the population by the sample size to calculate the sampling
interval. Round to the nearest whole number. For example, if you need a
sample of 100 from a population of 500, divide the population size (500)
by the sample size (100) to calculate the sampling interval (500/100=5).
3. Select a number randomly between 1 and the sampling interval. This
number identifies the first selection for the sample from the sampling
frame.
4. Draw a sample by using the sampling interval (calculated in step 2) from
that starting point (identified in step 3) from the sampling frame. For
example, with a sampling interval of 5, select every fifth participant until
the total sample has been identified.

Stratified Random Sampling


Stratified random sampling uses known information about the population to
ensure that the sample is representative. It involves dividing the population
into subgroups based on known characteristics and then randomly sampling

3
31

Sampling

within each subgroup. This technique is used when there is an assumption


that the different subgroups might have different experiences or outcomes.
An agency with multiple locations, for example, might prefer stratified
random sampling to ensure proper representation of its inner city and its
suburban clients.
1. Divide (or stratify) the population into subgroups (or strata) based on
characteristics that convey meaningful differences and are likely to
influence their experience or outcomes in a program (i.e., age,
neighbourhood, education level, etc.).
2. Take a simple random sample from each stratum (as described above).

Case in Point
Jewish Family and Child Service of Greater Toronto (JF&CS) used stratified random sampling
to select 30 cases for their Case File Utilization Review Audits. A sampling frame of 248
clients was created from all active cases in which non-mandatory services were provided to
families with children under 16 years of age. As a first step, staff divided all possible cases into
categories (strata) according to the use of non-mandated services: counseling, financial
assistance, and other. Sample selection rules were set so that no case was repeated in the
sample. From within each stratum, a statistical program was used to select cases randomly.

Cluster Sampling
Cluster sampling differs from stratified random sampling in that it makes use
of the populations pre-existing subgroups or clusters.
In this type of sampling, a sampling frame of the known clusters in the
population is created and cases are selected through simple random
sampling; then members of those clusters are then sampled for evaluation.
For example, if we wanted to know the attitude towards reading of grade 5
students, it would be difficult and costly to visit each grade 5 class in the city
to collect this data. Instead, we could randomly select 10 schools (or
clusters) and draw a random sample of grade 5 students from each of the
randomly selected schools. An alternative is to randomly select 10 schools
and then collect data from all of the grade 5 students (census) within each of
the chosen schools.
1. Create a sampling frame of the known clusters in the population and
assign a number or code to each.
2.Use simple random sampling to identify a sample of clusters for use in
the evaluation.
3. Create a sampling frame within each cluster of the units in the population.
4.Use simple random sampling to select a sample of units within each
cluster. Alternatively, select all units (census) within the chosen clusters.

Strategies for Non-probability Sampling


Non-probability sampling relies upon personal judgment rather than
statistical methods to select members from the population. The person or
group selecting the sample chooses each unit of the population deliberately.
The chance of each unit being selected is not known and therefore, it is not

3
A Toolkit for Outcome Measurement

32

possible to generalize the results from the sample to the population. Be


careful when interpreting results from non-random samples. There are no
general rules that govern sample size for non-probability sampling. Sample
size depends upon what you want to know, what method of data collection
you use, what will be seen as credible and perhaps most significantly, what
can be undertaken given the time and resources available.
Choose the sample carefully, in a way that gives you the information you are
seeking. Make sure that you can explain and justify your sampling decisions
so that those who will be using the data will understand your rationale and
will be aware of the limitations of the results.
Non-probability sampling may be a good choice for agencies for several
reasons:
l

When you want in-depth information regarding a particular


program or situation. If you want to understand the experience of
typical or unusual cases, such as outstanding successes or failures,
selection of the sample would be highly deliberate.

When you need to respond to special requirements, such as


language and culture. In some situations, access to the population may
be challenging due to language barriers, culture, transportation, etc. In
these situations, a non-random method of sample selection may be the
most efficient way of accommodating these realities and obtaining
information.

When a sampling frame cannot be created for the population.


Non-probability sampling may be the only possible method in this
situation.

When there is no great concern for numerical precision and


only a general sense of situation is needed. For example, you may
want to do a pilot study of a questionnaire to give you a better
understanding of the range of opinions on an issue. For these purposes,
you could use a sample of people who are available (convenience
sample) to provide this information quickly.

When you are interested only in the members of the


population that comprise the sample, rather than the
population. A non-probability sample is appropriate if you need
information related only to the sample to make a decision or understand
a situation. For example, you may be interested in knowing why people
who attend a drop-in program in one location are not participating
regularly and you do not need similar information from other program
locations.

3
33

Sampling

Agency in Action
In a study of a project to assist homeless and marginally housed individuals their personal
identification was obtained, a before - and - after data collection plan was developed in an
effort to understand better the situation facing the target population and to find out how
individuals benefited (if at all), Information such as gaining access to services previously denied
to themwas collected.The aim was to interview 150 clients from across the projects seven
areas of operation, a number estimated to be manageable and large enough to yield credible
information.The number of clients to be interviewed in each area was determined in
proportion to each areas client volume. Participants were recruited for interviews according
to their position in the queue for service at the clinics.The first client was approached and
every third client thereafter was also asked to participate in the interviews. If one declined,
the next client was asked until one agreed to participate. From that point, the following third
client would be approached, reestablishing the sampling interval.

Non-profit organizations typically use five types of non-probability sampling:


l

Quota Sampling

Volunteer sampling

Purposive sampling

Snowball sampling

Convenience sampling

Quota sampling
A quota sample is created by dividing the population into subgroups based
on some specific characteristics (i.e., age, gender, income) and then
selecting members (units) from within those subgroups in a way that
represents their occurrence in the population.
For instance, you might want to include men and women in your sample in
the same proportion that they participate in the program. Individual men and
women are chosen for the sample until the quota (proportion) is achieved.
Quota sampling may appear to be similar to stratified random sampling but
the results from a quota sample do not generalize to the whole population
because all members (units) of a population are not identified in a sampling
frame and so they do not have an equal chance of being included in the
sample.
Purposive sampling
Purposive sampling occurs when members (units) of a population are
chosen deliberately. Generally, purposive sampling is used when you want
rich information from which to learn. There are many ways to choose a
purposive sample, some relying on special characteristics and others on
ease of implementation.
For example, an evaluation of a new parenting program may select a
purposive sample that includes working mothers, stay-at-home mothers,
single mothers, teen mothers and mothers whose extended family are
raising their children. This purposive sample would ensure that the evaluation
included the viewpoints of these different types of participants.

3
A Toolkit for Outcome Measurement

34

Selection strategies may target any of the following:


l

Sub-groups based on particular characteristics

Highly unusual or extreme cases

A wide range of variation on dimensions of interest

Typical or average situations

Politically sensitive cases

Easily accessible cases

Convenience Sampling
As the name suggests, the key aspect of convenience sampling is that
participants are selected because they are available. This method is
inexpensive and straightforward, to be sure, but it has limited credibility.
Be careful how you interpret the results from a convenience sample.
Volunteer Sampling
This form of sampling makes use of participants who volunteer for the
study. It shares the same drawbacks as convenience sampling, but it is
likely to have even more bias because the respondents select themselves.
For example, some seniors having lunch in a Day Program may volunteer to
complete a survey questionnaire. There is a good chance that the seniors
who volunteer are not representative of all seniors in the Day Program in
terms of cognitive functioning, health status, literacy levels and attitude
towards the program and staff.

Case in Point
A community agency that provides pre-school programs wants to identify best practices for
their playgroup program. Staff identify five experts from the other similar community
programs and one from a local community college to interview. In turn, each of the persons
interviewed recommends a few other people.The sample snowballed to 23 respondents.
Each key informant should be selected for his or her specialized knowledge and unique
perspectives on the topic. To avoid bias, it is very important to select informants with
various points of view. Selection consist of two tasks:
l Identify groups and organizations from which key informants should be drawn. If possible,
include all major stakeholders so that divergent interests and perceptions can be captured.
l

l
l

Control Bias. Although key informant interviews are a powerful and convenient way to
obtain in-depth information, there is the hidden danger that a group of experts has limited
and perhaps, biased views.There are a few wise precautions you can take to control or
reduce possible bias.
Check whether the key informants are representatives of different subgroups. Review the
key informant list with a few other people to ensure no significant groups were overlooked.
Assess the reliability of the key informants. If possible, review your notes and discuss the
informants with other interviewers. Assess each key informants knowledge of the topics,
credibility, objectivity and the willingness to see the pros and cons of their own position.
Place greater weight on the information you receive from the informants you assess as
being more reliable.

35

Sampling

Snowball Sampling
Participants in a snowball sample often are selected deliberately because of their
expertise in a survey topic. The sample grows as respondents suggest other possible
individuals for inclusion in the sample. This process is repeated until there are no
new persons nominated. Typically 15 to 20 persons are included in a snowball
sample. This is a common technique for key informant surveys.

Summary
No one approach to sampling works in all situations. Sampling techniques must be
chosen to suit the circumstance of the evaluation exercise. Decisions should be
based on what you want to find out, how the information will be used and by whom,
and what resources (time, money, people) are available. Most importantly, sampling
decisions should be made thoughtfully so that the strengths and weaknesses of the
sampling procedures can be described and the data can be assessed properly. To
manage the sampling process well and minimize apprehension, develop a sampling
plan to organize the implementation and record your decisions and actions.

3
A Toolkit for Outcome Measurement

36

3
37

Sampling

Chapter 4:
Analyzing and Communicating
Outcome Information

What Is Data Analysis?


Analysis occurs after data has been collected. It is the process of looking at
the information and asking (and answering) the question So What? The first
step in data analysis involves organizing the data and examining it in detail.
Different methods may be used to analyze the data depending on whether it
is quantitative (numerical) or qualitative (narrative) but each method allows you
to look at the data as a whole and divide it into meaningful parts. It entails
asking questions of the data and letting the data raise questions about the
program. (Sometimes, the most interesting results are unexpected!) Analysis
involves bringing your grasp and insight of the environmental context
surrounding the program to bear on the data, to interpret the information and
find meaning in it.
Comparisons are an integral part of data analysis. Comparisons are often the
basis for interpreting and making sense of the evaluation results. You may
compare current results with expectations, or with results obtained from
earlier evaluations. You may be comparing experiences over time that is,
before, during and after participation in the program. You may be comparing
the experience of subgroups within the same program, defined by differences
in demographics, length of service or service location. Or you may be
comparing results in one program with that of another similar program.
As you can see, there are many different ways of looking at outcome
measurement data and the choices you make should relate to the specific
questions you are seeking to answer. Your choices will be influenced by
opportunities you have to collect information at different points in time, or
from other relevant sources external to the program. Some types of analysis
will require information that describes program participants, so you may need
to gather demographic details at the time of outcome data collection.
Analysis should be methodical. It should be orderly and thoughtful, but
analysis does not often require advanced statistical techniques. Rather, it
typically involves simple statistical summaries especially frequency counts
(i.e., number of clients who learned a new skill) and percentages (i.e.,
percentage of clients who said that the program helped them) and content
analysis of qualitative data. Undertaken with care, analysis of outcome data
will promote understanding of your program and help you decide what you
can do to improve your program (and your evaluation).

4
A Toolkit for Outcome Measurement

38

Agency in Action
COSTI is a large community agency that provides a range of services in many neighbourhoods
across Toronto to enable immigrants and refugees to attain self-sufficiency, including settlement,
counselling and education and training programs.They invite participants in all programs to
complete a confidential questionnaire to gauge satisfaction with services received and benefits
experienced as a result of those services. Key outcome questions included are:

1. Did you receive the kind of services you wanted? (Please check one)
c No, definitely not

c No, not really

c Yes, generally c

Yes, definitely

2. Have the services you received helped you deal more effectively with your problems?
c Yes, a great deal

c Yes, somewhat

c No, not really

c No, things got worse

3. How satisfied are you with the service you received?


c Very Satisfied

c Mostly satisfied

c Mildly dissatisfied

c Very dissatisfied

Answers to the questions are combined (aggregrated) across all survey respondents to give
an overall picture of outcome achievement and general satisfaction. Some judgements can be
made about these results by looking at how they compare with those of previous years.
In order to make the analysis even more helpful, COSTI also asks respondents how many
times they have used any of their services. In addition, the staff that distribute the
questionnaires record from what agency site and from which program it was collected. This
additional information allows COSTI to separate the data into meaningful groups by amount
of service, type of service and location of service delivery. In thid way they can examine
whether satisfaction and outcome achievement vary by any of these factors. Any variations in
results would lead to a number of questions and a search for explanations. If one program
location, for instance, were to register lower satisfaction and lower outcome gains, COSTI
would begin to explore what is different about that site Has the target population changed?
Are sufficient resources available for program delivery? Is there something unique in the
neighbourhood that is influencing the program? Answers to these questions would help
COSTI interpret the questionnaire findings and develop constructive responses to improve
the program experience.

4
39

Analyzing Data

Ten Steps for Data Analysis


and Communicating Results
Once readied, the data can be analyzed in several ways. Select the ways that
will help you to understand your program experience best.

Preparing for Data Analysis


Step 1. Process the Data
Step 2. Tabulate the Data

Approaches to Analysis
Step 3. Examine Before/After Experience
Step 4. Analyze Outcomes by Subgroups
Step 5. Compare Actual Outcomes Against Targets
Step 6. Investigate Exceptions
Step 7. Examine Multiple Outcomes Together
Step 8. Examine External Influences
Step 9. Consider Attribution

Disseminating Results
Step 10. Communicate a Performance Story

Step 1. Process the Data


Before any analysis can begin, the information collected through various tools
must be processed organized, formatted and cleaned to facilitate
summaries and comparisons. Data processing involves moving the
information collected with various tools to either a paper-based or electronicbased summary form (spreadsheet). Often this involves the development of a
codebook. This is not a complicated task but it is necessary to allow you to
organize, group, reduce or aggregate the data simply and easily.
Whether you create a summary sheet by hand or with a computer, you will be
creating a table with many rows and columns. The rows typically represent
each tool completed (e.g., each questionnaire completed by a respondent) and
the columns represent the different items of information collected by the tool
(e.g., each question on the questionnaire).

The layout of your summary form (spreadsheet) is shaped by the structure of


the information collected, the questions asked and the scope of answers
given. In general, the transfer of information generated by closed-ended
questions (multiple choice, checklists, yes/no, rating scales) will require fewer
steps and less modification than results from open-ended questions or other
narrative data.

A Toolkit for Outcome Measurement

40

Even with closed-ended questions, there are some formats that require more
steps than others to process the data accurately. This is the case for questions
with multiple answers, such as those that ask respondents to check all that
apply. In effect, this response format actually incorporates a series of yes/no
questions and each of these sub-questions can be represented by its own
column in the summary form.

1.1 Codebooks
Quantitative Data Numerical Information and Closed-Ended Questions
The codebook is a tool that you create to assist with transferring the data
(collected from questionnaires, interviews, program records or direct
observation) into a summary form (spreadsheet or data file). Start by assigning
a distinct identification code (number) for each source of information, such as
a respondent to a questionnaire or a program record.
Some information collected through outcome measurement will be in a
numeric form that is ready for processing without modification. For example,
when participants are asked, How many times a month do you use a food
bank? responses are numerical and can be transcribed directly from the
original source to the summary form. Two visits to the food bank would be
entered as 2, five visits as 5.
At other times, information collected must be modified before entry into the
summary form. Language-based responses that are categorical can be
transferred into number codes to simplify analysis. Consider the following
examples of demographic information:
0=Male
0=Single

1=Female
2=Married

1=Divorced/Separated

3=Widowed

In another example, responses to the following question represent discrete


categories that can be assigned different codes:
Did you receive the kind of services you wanted? (Please check one)
c

No, definitely not

c Yes,

generally

Code=1

Code=3

c Yes,

No, not really Code=2


definitely Code=4

Note that number codes are arbitrary and therefore have no quantifiable value.
They are used only to distinguish one response from another. You can assign
any number to represent any category.

4
41

Analyzing Data

Qualitative Data Narrative Information and Open-Ended Questions


Open-ended questions ask respondents to supply their own answers that you
will need to organize into meaningful categories. This is content analysis. As
you read through responses, you will discover that they fit into general
categories. When developing categories, it is generally a good idea to create
more categories rather than fewer because it is always easier to combine
them later rather than break them apart. A minority of responses will likely not
fit into any category and so you may wish to create an other category to
capture this additional data. Assign numeric codes to all categories.
The following responses might be given for the question How might we
improve our services? Open longer in the evenings; No voice mail; Tamil
speaking staff and volunteers; I had to wait too long to get service; It was hard
to get to your office location; I thought everything was great; I couldnt get an
appointment for 3 weeks. You have many choices on how to categorize this
information. For instance, you could organize them timeliness/
location/language/other. Assign numeric codes to your categories.
A full description of how to undertake content analysis is available in Chapter
6: Conducting Effective Interviews. Variations on content analysis are also
described in other chapters about data collection tools.

1.2 Data Entry


Using the codebook you developed, enter data into the summary form. As you
enter the data, you may encounter irregularities in respondents answers.
Participants do not always respond to every question. Sometimes this occurs
at random, for instance when respondents are fatigued or have not received
proper instructions. At other times, there may be a pattern in the missing data,
such as when responses are withheld due to offensive or invasive questions.
Set up guidelines to deal with all irregularities and missing data. Consider the
following choices:
l

Assign a special code for missing data, i.e., 9

Use the average value to replace the missing data

Exclude all data from the source if 50% or more of the responses are
missing.

Other irregularities in your summary form may stem from human error or
mistakes made during data entry. It is good practice to enter the data twice, if
possible, so that it can be checked for mistakes. If a complete double entry
cannot be done, you can double enter a portion of the data and check for
mistakes. Discovery of data entry errors should prompt a check of the procedures
and codebook used, followed by another attempt at data entry.

Step 2. Tabulate the Data


Begin your data analysis with simple descriptive statistics. Some basic
techniques include:

4
A Toolkit for Outcome Measurement

42

Count the number of sources supplying information (e.g, respondents,


records, observations, etc.)
Frequencies (e.g., how often a behaviour or activity was observed, number
of participants, number achieving an outcome, etc.)
Percentages (e.g., percent of youth observed playing board games
together; percent of ratings that were satisfactory)
Averages means, medians, modes (e.g., average rating of physical
mobility)
Range (e.g., lowest and highest rating, diversity of observations, etc.).

In addition, you may decide that grouping results will help simplify and clarify
the data. For example, in trying to understand different patterns of use of a
service, it might be helpful, in addition to finding the average number of visits
per month, to group respondents into categories based on the frequency of
their visits. Respondents who use an agency service only one time would be
grouped together (and assigned a code of 1 on the summary form) as would
those using a program from 2 to 5 times per month (assigned a code 2), those
using a program 6 to 10 times (assigned a code 3) and those using a program
11 or more times (assigned a code 4).

Step 3. Examine Before/After Experience


Collecting data at different times allows you to track the progress made by
individuals, families and communities on important outcomes. It is very
difficult to interpret the meaning of outcome data that has been collected only
once. Try to incorporate into your data collection plan more than one
opportunity for outcome measurement.
You can begin to form a reference point for making sound judgments about
program outcomes by collecting initial (or baseline) outcome data upon entry
to a program (pre-test). If possible, select some outcome indicators that can
assess client functioning before participation in a program for example,
health status, literacy level, feelings of isolation, job status, etc. Then you can
compare data after participating in your program (post-test) with these initial
or baseline scores and, if possible, again at a later point in time (follow-up
test).
If it is not possible to measure functioning at the time clients enter the
program, then the second suggestion is to treat the first round of data
collection after program participation as baseline data (pre-test). In this
situation, a follow-up with participants sometime after the program would be
valuable (post-test).
If you have only one opportunity to collect outcome data immediately after
participation in the program analysis over time is limited to comparisons
between data collected in successive years/program periods. In effect, past
performance in the program is used to assess current experience and to
detect trends and other significant changes.

4
43

Analyzing Data

Step 4. Analyze Outcomes by Subgroups


Most programs collect outcome data from program participants or sites and
combine, or aggregate, the data for analysis. Another common practice is to
break the data into subgroups, that is, to disaggregate the data. This
technique, referred to as crosstabulation, helps you examine the relationships
between various aspects of the program, such as the relationship between
participant characteristics, program resources, activities and outcomes.
Understanding these relationships can help your program consistently deliver
positive outcomes.
For example, you may want to know how outcomes vary with the
demographic characteristics of program participants, such as age, gender and
level of education, or how outcomes vary based on other differences, such as
length of service.
Some common ways of disaggregating the data include examining:
l

Client characteristics (age, gender, education)

Level of vulnerability/risk and assets of participants

Type of service (differences in service delivery)

Service intensity (amount of service received)

Service duration (length of time in program)

Program factors (staffing levels, resource levels, location).

Aggregate data can be misleading. Breakout analysis helps you to identify


whether subgroups are experiencing different degrees of success in the
program and allows you to explore possible reasons and program
modifications to address the gaps.
For example, in a program to help street youth gain employment, aggregate
outcome results may suggest 60% achieve that change. After breaking out
participants by level of education, the data shows that 75% of street youth
who do have a high school diploma gain full-time employment after
completing the program whereas only 25% of street youth who do not have
a high school diploma find full-time employment after completing the program.
As the mix of participants becomes better known, outcome expectations can
become more realistic for each subgroup and, more importantly, program
enhancements to address the different needs can be developed.
Think ahead to the types of breakout analyses that you will need so that the
final data collection plan includes all necessary information. Trying to assemble
missing data from program records, for example, after the outcome
measurement is completed is usually inefficient and time consuming.

Step 5. Compare Actual Outcomes Against Targets

One of the difficulties with data analysis is to move beyond description and
determine whether or not a program is working well. If 70% of the
participants say that they are benefiting from a program, does this mean that
the program is performing well or not? If you intend to make an evaluative
judgment, it is important to decide on the criteria for success before you
complete your analyses.

A Toolkit for Outcome Measurement

44

A seemingly simple way of analyzing outcome data is to compare outcomes


achieved (actual) against planned outcomes (target). In practice, however, it can
be difficult to set reasonable outcome targets, especially when an outcome
indicator is new and an agency has no previous experience with the specific
indicators of success.
If this is your situation, you are not alone. Setting targets and measuring
outcomes is an evolutionary process that you should not rush it often takes
experience with outcome data over two or three fiscal years before you have
confidence that you are setting reasonable, yet challenging, targets.

5.1 Strategies for Setting Targets


Here are some popular approaches for you to consider for defining success
and setting outcome targets:
l

Treat the first round of outcome data collection as the baseline. This will give
you an initial benchmark against which you can compare future program
performance. Modify the target as you gain experience. Past program
performance informs success targets.
Refer to your strategic plan, budget documents and contracts to find
reference points for setting targets.
If you have been collecting outcome data, examine it for trends (e.g., client
satisfaction ratings have been increasing steadily, complaints have been
dropping) and then set targets based on the trends.
Interview key informants from similar organizations about their outcomes and
outcome targets. Consider this information in light of your own resources,
client characteristics and your local situation to set targets.
If you find an organization with a similar program performing better than
yours, use their performance as a benchmark to set your own targets.
When making comparisons, collect general information about client
characteristics, details about program operations and general information
about the service delivery environment. Otherwise you run the danger of
comparing programs that sound similar but differ on important factors.
Use standards set by your professional association or accreditation body to
set targets in important program areas.
Obtain feedback from clients to gauge the achievement of outcomes and
then set stretch targets for improved performance.

5.2 Setting Targets When Outcomes Are Very Uncertain


In general, these target-setting strategies work well for most agencies. The
outcomes for some agencies, however, are highly uncertain because they are
subject to shifting client needs, caseload and resource fluctuation. These
agencies are justifiably reluctant to set firm program targets under unstable
conditions, especially for their longer-term outcomes.
There are two options used to set targets in this situation. The target range option
encourages the setting of targets based on a range, such as, Between 30% and
45% of developmentally disabled clients with mental health problems will be
living independently 12 months after completing the program.

45

Analyzing Data

In situations where outcomes are affected strongly by participant


characteristics for instance, where some participants experience more
challenges or risks and therefore require more support than others the
variable target option permits setting different targets for each category.
Based on your understanding of subgroups, you set different targets for each
(as in the example above relating to street youth with and without high school
education).

Step 6. Investigate Exceptions


An important part of data analysis is recognizing unusually high or low
outcome results. Investigate why and how these situations occurred. Then
take action to correct any problems or make the most of a favourable
situation depending on the nature of the exception.
There is a wide variety of information that may be useful to clarify the reasons
for the exceptions. Collect a combination of qualitative data, such as staff and
client observations and opinions, and factual data about program operations,
budgets, human resources and external conditions.

Step 7. Examine Multiple Outcomes Together


Most programs will have multiple outcomes and multiple indicators to
measure each outcome. Examining the data for each indicator and each
outcome separately is a necessary part of analysis, but a broader
understanding of program performance may emerge when you examine a
number of outcome indicators collectively.
Look at all indicators for an outcome at the same time. For instance, you may
use questionnaires at the end of a training session along with trained
observers to gauge active learning. Considering the indicators collectively
provides you with a stronger basis for making judgments.
Look at indicator data for multiple program outcomes simultaneously for
gaining an overall perspective of the program. By considering the data from all
of these sources concurrently, the agency can build a better picture of the
overall program performance, any patterns and relationships between
program components, spot areas of program weakness and take action to
improve performance.

Step 8. Examine External Influences


Analysis that reveals unusual improvements, declines or any unexpected
outcome results should trigger a search for the reasons why the changes have
occurred. Numerous unrelated factors are at constant play that could potentially
affect achievement of program outcomes. Give thought to how potential
intervening factors in the lives of program participants, within your
organization and in the larger external environment might help you to
understand your data. Consider factors such as:

Characteristics of the community or target population

Staff changes

Change in resources

Modification of service delivery process

A Toolkit for Outcome Measurement

46

Economic fluctuations

Legislative or policy changes, etc.

Special events.

Step 9. Consider Attribution


The issue of cause-and-effect continues to stir debate in discussions about
outcome measurement and program evaluation generally. Although outcome
indicators may show that outcomes are being achieved, they will not tell you
what caused the changes. How do you address the issue of attribution?
This is not easy to do. Even with carefully designed, well-resourced evaluation
studies led by skilled evaluators, it is safe to say that it is not possible to
determine definitively the extent to which a program contributed to an
outcome. This is because of the complexity and mix of potential influencing
factors in the lives of individuals, families and communities.
It is possible, however, to collect credible, valid information that will
demonstrate to a reasonable person that, given the program activities and
the intended outcomes, the program made a meaningful contribution. You can
almost always gather enough data and information to increase understanding
about a program and its outcomes, even if you cannot prove a cause-andeffect relationship.
There are several strategies you can use to help demonstrate that your
program has contributed meaningfully to program outcomes.
l

A program logic model provides a concise depiction of how your program is


supposed to operate, revealing the influential relationships between program
activities, outputs and the different levels of outcomes. Program logic
models may also include external factors that are likely to affect program
outcomes, offering a portrayal of the program that is more complete,
honest and in its proper context. Use the program logic model when
communicating results to outline exactly what has been measured, what
the major assumptions are regarding the contribution of the program and
what and how external factors are also at play.
Ask clients to provide feedback about how much your program contributed
to their outcomes. For example, you might already ask clients whether
they have experienced an improvement in the problem that brought them
to your agency. You can also ask them how much your program contributed to their
improvement, using either a rating scale or open-ended question. This
additional information allows you to state not only the number and
percentage of clients who improved but also the number and
percentage who improved and who said that the program contributed
greatly to their improvement. This second statement is much more
powerful because it addresses the attribution issue directly. You may elect
to analyze this feedback according to subgroup breakdowns, such as client
characteristics, program components received and program duration (see
above for more information).

4
47

Analyzing Data

Step 10. Communicate a Performance Story


After all the outcome measurement and data analysis is completed, the
results need to be communicated shared, discussed, debated and reported
and used for organizational learning, decision-making, continuous
improvement and accountability. This final step is what makes the data
collection efforts worthwhile.
There are many different and effective ways to share evaluation results, from
formal written reports to informal verbal presentations at meetings. The
manner and style of communicating results should be determined by who the
audience is and the purpose of the information exchange.
Internal audiences include program managers, staff, volunteers, client
representatives and persons from other programs within an organization that
will use the outcome information to make important program decisions (and
celebrate successes). Internal audiences benefit from information about a
relatively large number of indicators.
External audiences, on the other hand, such as funding bodies, will want data
from comparatively fewer indicators that are carefully selected to meet their
information needs. A common mistake is presenting too little outcome
information to internal audiences and too detailed information to external
audiences.
To gain maximum benefit from the outcome measurement process, present
the results and analysis to program managers and staff/volunteers first. This
step helps with understanding and interpreting the findings. It allows you to
validate the findings, spot any errors, obtain explanations and increase
utilization by providing rapid feedback to those most able to improve service
delivery and program outcomes. Program managers should follow up with
meetings to develop action plans that correct problems or take advantage of
opportunities.
Then, develop summaries of the findings that include explanations and action
plans for broader distribution to external audiences by means of reports,
newsletters, websites and annual reports.

10.1 Summarize Your Findings and Provide Explanations


Present the data in more than one way use charts, graphs and narrative
summaries. A variety of presentations is appealing and acknowledges the fact
that certain formats appeal to some people more than others.
Communicating outcome information begins with a summary of your findings.
The summary should draw attention to the most important findings both
positive and negative. It is important to adopt a balanced approach to
communicating your findings and not to jeopardize your credibility by
emphasizing only the positive findings.

Often the summary is more effective if you illustrate numerical findings with
a brief story or case vignette based on client experiences while taking care
not to identify real clients. You could also emphasize points by including
verbatim comments based on interviews, focus groups, open-ended

A Toolkit for Outcome Measurement

48

questions or other qualitative data gathered from clients, key informants or


staff and volunteers. Qualitative data provides the depth and human interest
usually absent in statistical summaries alone.
Another important aspect of communicating your outcome indicators data is
explaining the reasons for below average (or above average) results and
significant differences discovered in comparisons, such as differences
between sites or for different groups of clients. Explanations will be more
credible if you can supply data from different sources that point to the same
explanation. These sources can be from inside your organization, outside it or
both.
Finally, no outcomes report is complete without including a concise
description of actions that your program plans to take or has already taken to
deal with results surprises, disappointments and triumphs. Dont shy away
from reporting negative findings all results are good if they help to inform
understanding and decision-making. Over time, this creates both high-impact
programs and a storehouse of organizational learning about what works and
what does not. Thus, explaining why the outcomes did not meet expectations
and describing the corrective actions taken are equally important aspects of
outcomes reporting.

10.2 Tables and Graphs


Tables and graphs are among the most common ways of communicating
outcome information. Invite your audiences to suggest ways they would like
to receive information to maximize understanding and use of the information.
Visual presentations of data can help create an instant impression. The chart
below shows the results from a telephone interview with clients from the
transportation program offered by the Canadian Red Cross (Toronto Region).
Transportation Interviews with 100 Clients Canadian Red Cross, 2001
Has our service made a difference in your ability to access doctor or other
appointments?

4
49

Analyzing Data

10.3 Comparison of Actual Versus Target Over Annual Cycle


This table format compares outcome indicators for both the current period and
over the year against the annual target. This type of format supplies early
warning information about the program outcomes.

Current Period

Cumulative for Year

Target
%

Actual
%

Target
%

Actual
%

Target for
Year %

% clients reporting that community dining


program led to reduced sense of isolation

70%

55%

75%

70%

75%

% clients reporting that community dining


program helped them cope with hunger

80%

87%

80%

85%

80%

Outcome Indicator

10.4 Comparison of Actual Versus Target for Previous and


Current Periods
Whereas the previous table tracked outcomes with reference to an annual target,
this table shows performance for the previous period and the current period. It
has the advantage, however, of including a column that clearly shows the
difference (or variance) between targets and actuals, which makes exceptions
easy to spot.

Previous Period
Outcome Indicator

Current Period

Target Actual Difference Target Actual Difference

% participants receiving
skills certificate

60%

45%

-15%

60%

55%

-5%

% participants accepted
in job placement

50%

55%

+5%

50%

60%

+10%

10.5 Comparison of Outcomes According to Program


or Participant Characteristics
The next table format allows you to select important program characteristics,
such as program location, so that you can compare differences in outcomes.
This permits managers to discover important differences in outcomes,
investigate them quickly and then make the necessary changes to ensure that
all programs are on track. This format also provides program managers with
important feedback about outcomes against the average for all programs, so
that high, average and low performance can be identified.

4
A Toolkit for Outcome Measurement

50

Program Sites
Outcome Indicator

Site1

Site2

Site3

Site4

Average

% participants receiving
skills certificate

65%

50%

60%

60%

55%

% participants accepted
in job placement

50%

70%

65%

65%

60%

You can use the same format to examine outcomes according to participant
characteristics that you think may be important, such as gender, age, income level,
or risk

Outcome Acheivement Job Placement


Site 1

Site 2

Site 3

Site 4

Average

High Risk
(Difficult to Serve)

15%

13%

25%

18%

17%

Moderate Risk
(Average to Serve)

31%

28%

42%

25%

31%

Low Risk
(Easy to Serve)

50%

55%

62%

53%

55%

Participant Risk Level

The Performance Story


One of the recent trends in communicating outcome information is to view it
as part of a process called telling a performance story. It is an opportunity to
relate more than just findings from a monitoring strategy. It draws together
into a big, comprehensive picture many elements descriptions of program
resources, activities, outputs, outcomes, evaluation choices and methods,
environmental contexts, etc. (all parts of well-prepared logic models and
evaluation frameworks).
The most common way of telling a performance story is to write a narrative
(one or two pages), including exhibits, charts and tables where appropriate.
Key elements of telling a performance story include:
l

Describing the context of your program (key characteristics of the target


population, external and internal influences, etc.)

Stating what longer-term outcomes guide your program

Listing program resources for the relevant period (budget and actual)

Identifying key activities and partnerships

Showing outputs your program achieved relative to what you expected

4
51

Analyzing Data

Explaining the outcome data collection strategy used and the steps taken
to ensure data quality

Outlining key outcomes accomplished

Identifying any unintended results, either positive or negative

Conveying lessons learned, explanatory information, conclusions drawn to


date and changes planned as a result of the findings
Outlining how the findings support the contribution of your program to
outcome achievement. (Be clear regarding expectations about longer-term
outcomes).

4
A Toolkit for Outcome Measurement

52

Examples
Analyzing and Communicating
Outcome Information

4A Client Satisfaction Questionnaire - COSTI


Note:
the formats of the following outcome measurement tools have been altered
to some degree to correspond with the design and space restrictions of this
toolkit. We hope you will draw inspirations, confidence, and even concrete
elements for your own use, drawing upon the steps and guidelines set out
in these pages.

1
53

Analyzing Data

Example 4 A
COSTI

4
A Toolkit for Outcome Measurement

54

Chapter 5:
Developing Effective
Questionnaires

What Are Questionnaires?


A questionnaire is a series of questions organized to obtain information
directly from people, called respondents. People are the most common
source of information in evaluations in non-profit organizations. Agencies seek
feedback from program participants and beneficiaries, community residents,
staff and volunteers, key informants, funders, partner organizations, etc.
Questionnaires are used for written and electronic surveys and are also the
foundation for interviews, focus groups and client satisfaction feedback.
As simple as it sounds, a good questionnaire requires careful forethought so
that the questions being asked are relevant, clear and easy to understand by
all of the respondents. Questionnaires are often a popular and practical tool of
choice because they are a relatively low-cost and rapid way of obtaining
accurate information from a large number of people.
Questionnaires are multi-purpose and as such, can be designed to collect
information that sheds light on how well a program is functioning and also on
whether a program is contributing to changes in, and benefits for, the clients
and communities it serves.

Agency in Action
As part of its Multicultural Women's Wellness Program, the Canadian Mental Health
Association (Toronto Region) offers support groups for women who are socially isolated and
experiencing cultural and linguistic barriers, and/or are at risk of mental health problems due
to difficult life circumstances.The goal of these support groups is to promote the mental
health of women within a supportive group environment. Women meet weekly for about 46
weeks during the year, discussing in their own languages topics they feel are important to
their wellness.
At the end of the year, the changes in group participants are measured using a one-page
paper and pencil questionnaire. On a five point rating scale (where 1 = strongly agree and 5
= strongly disagree), women are asked to indicate their level of agreement to a series of
outcome-related statements such as, I am coping better with family matters and I am
managing things better at home. Questionnaires are translated or administered orally
depending on literacy levels and language of choice of the women.

Recently, the program has begun using an intake questionnaire, which collects
demographic information about program participants (e.g., age, birthplace,
number of years in Canada, etc.). By choosing to use this information, the
agency is able to see whether achievement of outcomes varies by
demographic characteristics of the women.

5
55

Questionnaires

When Are Questionnaires Useful


for Measuring Outcomes?
l

When you want feedback to understand the changes to knowledge, beliefs,


attitudes, behaviours and needs of clients. Through a carefully constructed
questionnaire you can ask clients about many different outcomes that only they know
best. Questionnaires can help you obtain information about what people know, think,
believe, feel, want, have and do. For example, a health promotion program might want
to understand if it is having an impact on teens attitudes towards smoking or to
measure the attainment of outcomes by having the teens report their actual smoking
behaviour in terms of the number of cigarettes smoked in a day.
When you need an overall picture of the program experience. Many non-profits
have limited funds for evaluation. They may resort to unstructured, informal methods,
such as speaking to a few clients about program outcomes. Although this may give
program staff some sense of clients reactions, there is the very real danger that this
informal feedback does not accurately portray the views of all participants in other
words, it is biased. Questionnaires can be used to collect uniform information from a
large number of people and the responses can be tabulated to generate quantitative
(numeric) information. In some circumstances, non-profits use questionnaires with all
program participants but at other times they may approach only a sample of program
participants. If the sampling is done thoughtfully, questionnaires generate an accurate
picture of the outcomes achieved, enabling conclusions to be made confidently.
When you want before and after information. Questionnaires are often used to
collect information from clients before they begin a program. At intake, clients are
asked to describe who they are, where they come from, what services they need and
other questions may be added regarding the things clients want to change or improve
(i.e., stress levels, financial stability, housing conditions, etc.) Questionnaires can be
used at different points in service delivery, even as a support to service delivery, to
record progress on goals or improvements, revealing outcomes achieved in the
program.
When you need information about client attributes who they are and what they
have. Questionnaires are an efficient way to obtain factual (often called demographic)
information from your clients, such as their age, gender, income levels and prior
service history. This data helps in analyzing and interpreting the results by enabling
comparisons made of outcomes by characteristics, amount of service received, etc.

Main Strengths and Limitations of Questionnaires

Strengths
Questionnaires produce accurate data

Questionnaires need to be brief and ask


relatively simple questions

Questionnaires can be completed quickly

Question wording and order can have a major


effect on answers

and are relatively low-cost

Limitations

Questionnaires are good for collecting


information from a large group of people

Interpretation of question may vary between


respondents

A Toolkit for Outcome Measurement

56

What Are the Main


Strengths of Questionnaires?
l

A well-constructed questionnaire generates accurate data, enabling


conclusions to be made with confidence. A good questionnaire follows a
carefully planned process of development and pilot testing. By following the guidelines
for writing question and answer categories and for question order and layout, you can
ensure that each question is clear, easy-to-understand and is valid because it measures
what it is supposed to measure.

Questionnaires can be completed quickly. Most questionnaires used by nonprofit organizations for outcome measurement are short (1 or 2 pages) and are
completed by a client or staff member in just a few minutes.

Questionnaires are relatively low cost. Because the questionnaires used by nonprofits are usually developed and administered by staff, they can be one of the least
expensive ways of gathering information.

Questionnaires are an essential tool for collecting valid and reliable


information from large numbers of people.

What Are the Limitations of Questionnaires?


l

Questionnaires need to be brief. Although there is considerable dissent


about the right length for questionnaires, there is clear agreement that they
must be brief unless you want to see them ignored or crumpled in the trash. This
limits the number of questions that can be asked, while still having a good rate of return.

Questionnaires ask relatively simple questions. Good data comes from writing
questions that can be easily understood by each person who reads the questionnaire.
Writing clear questions means avoiding complex, ambiguous or confusing questions in
favour of simpler ones.

Question wording and order can have a major effect on answers. The words
used in a question or the order in which it appears on the form, may influence how
someone answers it and affect accuracy or validity of the data. During the design phase,
you should give careful attention to question wording and to placing questions in a
logical order that contributes to question clarity. Adapt wording to the vocabulary and
reading skills of the respondents. Be specific and avoid vague terms that may mean
different things to different people, such as regularly or youth.

Cannot correct wrong interpretation of questions. When a person completes a


questionnaire alone, there is no way of knowing whether the person is interpreting the
questions in the intended way. Misinterpretation of questions can lead to inaccurate
answers. On the other hand, in person interviews or even group interviews, such as
focus groups, can identify and correct wrong interpretations of questions.

5
57

Questionnaires

Eleven Steps for Developing


Effective Questionnaires
Although questionnaire design appears easy, effective questionnaires are the
product of systematic planning and development. A systematic approach
helps ensure that you will create a tool that engages respondents and obtains
valid and reliable outcome information from them. A well-designed
questionnaire also makes collection and analysis of responses easier. Whether
you are collecting information about client outcomes, client satisfaction or
program delivery, these same steps can be applied to the development of
your questionnaire.

ELEVEN STEPS FOR DEVELOPING


EFFECTIVE QUESTIONAIRES
Step 1. Specify The Major
Questionnaire Topics
Step 2. Draft Questions For Each Topic
Step 3. Draft Possible Answer
Categories For Each Question
Step 4. Decide Upon Question Structure
Step 5. Decide Upon Response Format
And Wording
Step 6. Determine Question Order
Step 7. Check The Questions
Step 8. Design Questionnaire Layout
Step 9. Pilot Test And Revise
The Questionnaire
Step 10. Implement The Survey
Step 11. Analyze The Data

Step 1. Specify The Major Questionnaire Topics


Questionnaires are organized according to topics that relate to the major
evaluation information needs of the stakeholders. An important step in
developing effective questionnaires is understanding the key topics and issues
from the viewpoint of the respondents the target group who will complete
the questionnaire. There is the danger that in designing the questionnaire you
write questions seen as important only to program staff and that you have a
blind spot to the perspectives of the respondents, especially those who
come from different backgrounds or ethno-cultural groups.

How do you acquire this knowledge? At this point, you should arrange
personal interviews or a small group discussion with several persons who are
actual members of the target group. Describe the purpose for the evaluation
and the major information needs of the stakeholders. Through the interviews
or group discussion, ask the members of the target group for their views
about key topics and issues that the questionnaire should address. If possible,
also try to identify some important questions under each topic.

A Toolkit for Outcome Measurement

58

Case in Point
The West Scarborough Neighbourhood Community Centre wanted to evaluate outcomes for
the Stay-In-School homework/tutoring program offered at their Boys and Girls Club. During the
design of outcome-based questionnaires, senior managers met with parent representatives and
program staff separately to find out what they expected from the program and what they
wanted to know from the evaluation.This feedback helped to shape the format and content of
the questionnaires.

This step is arguably the most important one in the questionnaire planning
process. Good practice demands you check that the topics and key questions
are relevant and meaningful to the target group. No matter how well you feel
that you know the target group, there is no substitute for verifying this
understanding with representatives of the target group directly.

Step 2. Draft Questions For Each Topic


For each topic or issue identified in Step 1, write possible questions. Expect to
draft more questions than are necessary under each topic, including alternative
wording for the same question.
Besides discussing potential outcomes and key questions with members of the
target group, you can use a literature search to identify published material or
information which is relevant to your particular questionnaire topics. This
information can be used to help you map the content of questions. In addition,
you can seek examples from other like agencies, use the internet or seek
assistance at your local library.
It is good practice to check the content of each question by asking yourself:
l

Is this question easy to understand?

Does this question supply the information needed?

Is this question necessary?

Guidelines for Writing Good Questions


Good questions should be easy to understand and limit possible
misinterpretations of their meaning. In addition to content, review the wording
of each question to be sure that it meets the following guidelines:
l

It is written in plain language.

It avoids jargon or technical words.

It is clear and specific (avoid abstract or vague words i.e., older adults,frequently,
etc.).
It is short.

Step 3. Draft Possible Answer Categories For Each


Question
Through discussions with individuals from the target group (see Step 1), you get
a better sense not only about potential questions, but also about the range of
possible answers. It is important to remember that questionnaire development
requires both question and answer development.

59

Questionnaires

To reduce completion time, keep costs low and make analysis easier, consider
using closed questions. Closed questions ask respondents to select an
answer from a list, whereas open questions ask respondents to supply their
own answers.
To know whether a closed or open question format is better, you must have a
very good idea about the kinds of answers respondents are likely to give for
each question. For example, if the possible range of answers is very large,
you may need to make it an open question or you could consider breaking one
question into several. The process of drafting answer categories allows you to
consider both the range of possible answers and the best type of question to
ask.

Step 4. Decide Upon Question Structure


Now you are ready to decide upon the question structure. Typical forms of
question structure include the following:
l

Yes/No questions

Multiple choice questions

Checklists

Rating scales

Open questions

Rank order questions

4.1 Yes/No questions


This structure permits only two answers (e.g., Yes or No). These questions
are easy to use and score. They work well when the answers fall into only two
clear and distinct categories (e.g., Satisfied or Dissatisfied) but can be
confusing if the respondent does not agree with either response. Some other
two-answer response categories include Fair or Unfair, Agree or
Disagree, and True or False.
Example: Does the transportation program meet your needs in terms of
destination?
c

Yes

No

Example: Did you make new friends through the Friendship Group?
c

Yes

No

4.2 Multiple choice questions


This format offers a list of answers and the respondent is asked to select the
one that most closely matches his or her own view. Usually the respondent is
asked to check only one answer from the list. Multiple choice is a
popular format for questions that have several possible answers because the
choice of responses is clear and easy to record.

5
A Toolkit for Outcome Measurement

60

Example: Please complete the following statement with the choice that most
reflects your eating habits: Since attending the information session on The
Health Benefits of Fruit and Vegetable Consumption, I have been incorporating
fruits and vegetables into my diet

q On a daily basis

q Several times a week


q Once a week

q Once every two weeks


Example: How would you rate the overall content of the workshop (select or
check one)?

q Too basic

q Good balance

q Too technical

4.3 Checklists
This question structure presents respondents with a list of items and the
respondents can select all of the answers that apply to them. Checklists are
very useful when there are many possible answers. The lists provide a good
memory jog, so respondents are not forced to remember a long number of
options. Usually the checklist ends with Other please specify so someone
can add their own answer at the end.
Example: Have you experienced LESS of any of these feelings since
participating in the Support Group for Bereaved Partners? Please check all
that apply.
c

Less trouble concentrating, focusing

Less trouble sleeping

Less feeling scattered

Less difficulty eating

Less trouble relating to family members

Less need for time off work

Example: Please check all of the programs that you used at least once during
the last month at the Community Centre?
c

The Meeting Room

Adult Education

Drop-in

Information and Referral

Woman Abuse Program

Settlement and Integration Services

The Homework Club

Greek-Speaking Mothers Program

61 Questionnaires

4.4 Rating scales


Rating scales have a range of answers that represent the way people feel from very
positive to very negative. Although a rating scale can be from two-points to tenpoints, the most common versions are the five-point scale and the four-point scale.
Example: How have family interactions been at home since you and your
partner participated in our group therapy sessions? Would you say they have ...
l

Gotten much better

Gotten somewhat better

Stayed the same

Gotten somewhat worse

Gotten much worse

This format is called a balanced scale because it has an equal number of response
categories on either side of a central or neutral point. In this case, the central or
neutral point is stayed the same. Another popular version of the five-point rating
scale is to couple response categories with a numerical scale.
Example: How well do you understand the dynamics of domestic abuse?
Please circle one number.
I understand the
dynamics very well

I dont understand
the dynamics at all
1

One problem with the balanced approach is that there is a tendency for some
respondents to select the middle response. Critics of the balanced approach prefer
forcing a choice by dropping the central response and using a four-point scale,
creating an unbalanced scale. In the above example, the scale would be written
as:
l

Gotten much better

Gotten somewhat better

Gotten somewhat worse

Gotten much worse

Case in Point
The Canadian Mental Health Association (Toronto Region) uses a balanced rating scale to assess
changes in participants of their Multicultural Womens Wellness program.The Coordinator for this
program noticed that one of the womens group was consistently circling the neutral response on
the scales. She visited this group to determine the reason that is, were the women
misunderstanding the statements or were they simply exhibiting the tendency to select the
neutral response? She discovered it was the latter reason.To remedy this, they are considering
using a four-point scale instead.

5
A Toolkit for Outcome Measurement

62

4.5 Open questions


Research and experience show that respondents are more likely to complete
closed questions. Even so, open questions are an important part of an
evaluation questionnaire. As we have already seen, open questions are useful
when you do not know the range of possible answers to a question or when
a very large number of responses is likely.
Open questions encourage detailed responses. They show more clearly what
respondents are thinking. Carefully chosen open questions also help maintain
interest in the questionnaire by the offering opportunity for in-depth
comments from the respondents, stimulating thought and inviting creative
suggestions.
Use open questions sparingly. Because they require more effort on the part of
the respondent, they may be left blank. This may result in bias because not
everyone responds to the question. In addition, open questions may be the
easiest way to ask for information, but they are the most difficult to analyze:
responses need to be categorized and summarized which is time consuming,
especially if you have a large number of respondents.

4.6 Rank order questions


In some situations, you might be interested in knowing how respondents rate
the items on the list according to some important factor. For example, instead
of just checking the services that a client used, the question might ask the
client to rank the services according to their usefulness. The client would be
asked to place a 1 next to the most useful service, a 2 next to the second
most useful service, and a 3 next to the third most useful service.
As you can quickly see from this example below, rank order questions are
difficult to answer.
Example: Rank the following programs for seniors on a scale of 1 to 5
according to how useful they were to help you feel less lonely.
To answer this question, respondents must have a good knowledge of each
service and then think carefully about how useful it was in comparison with
the others. Even if they can do this, you still do not know whether the
difference between services was great or small.
In general, you should avoid rank order questions or use them very carefully
(especially in questionnaires administered over the telephone). You can also
consider rephrasing a rank order question so that it uses a different structure
that is easier for individuals to answer. In the above example, you could list
each of the five services separately each having its own rating scale.

5
63 Questionnaires

Agency in Action
Evas Phoenix is an innovative shelter for homeless youth 16-24 years of age in Toronto that
requires residents to participate in either a training or employment program. An important
component of the Evas Phoenix model is matching every youth with a workplace mentor.
The workplace mentor helps youth feel comfortable in the job placement and provides
confidential advice and assistance on the job site. Evas Phoenix believes that by participating
in the workplace mentor program, youth are more likely to complete their job placement,
have a positive work experience and be better prepared to find and keep employment.
Youth in the program are asked to complete a confidential questionnaire to give feedback
about the impact that the workplace mentor had on their job placement. Note the mixture
of question structures, including a dichotomous question, several different rating scale
questions and an open-ended question.
1. How much support has your Workplace Mentor been to you? (Please check one)
c A very big support

c A good support

c Not much support

c No support at all

2. Did you feel that you could approach your Workplace Mentor at any time with concerns
or questions you had?
c Yes

c No

3. How strongly do you agree with this statement: The relationship between my Workplace
Mentor and me was a supportive partnership.
c Strongly agree

c Somewhat agree

c Disagree

c Strongly disagree

4. Do you see your Workplace Mentor as someone you can confide in and/or seek assistance
from on the job site?
c Yes, definitely

c Sometimes

c No, never

5. In what area do you think your Workplace Mentor has been the most help to you in your
job placement?

Step 5. Decide Upon Response Format and Wording


Question wording is important since participants may not understand a
question, may misinterpret the question, may tell you what you want to hear
or may get tired. Research clearly shows that questionnaire wording influences
responses.
Writing good questions is considered more of an art than a science. Even
highly experienced evaluators struggle to write good questions that meet the
unique information needs of key stakeholders and diverse client groups. Dont
be discouraged if it takes you several attempts to write a good set of
questions. As you begin drafting questions, you should be aware of several
common types of wording problems and ways of dealing with them.

5.1 Vague questions produce inaccurate responses


Research shows that individuals have difficulty answering vague questions,
especially questions related to frequency (How often do you...) or
comparison (How much improvement...).

The remedy is to define your terms to make the standard clear or phrase the
question differently.

A Toolkit for Outcome Measurement

64

Poor Wording: After participating in Fit for Life, do you exercise often? The term
often may be taken by one person to be every day and by another person to
mean once a week or less.
Better Wording: After participating in Fit for Life, do you exercise:
c every day

c a few days a week

c a few times a month

c less than once a month?

This phrasing replaces often with a clear set of responses.

5.2 Responses that require prior knowledge


Tend to be biased because the answer depends on several other factors,
including memory of the event, prior knowledge and prior experience.
Poor Wording: Are you satisfied with the location of our community kitchens?
To answer this question, a person must know the location of the kitchens first.
A respondent who does not know the locations may feel pressured to guess.
Better Wording: Our community kitchens are located at First Street, Second
Street and Third Street. Are these locations convenient for you?
Rather than depend on memory, this format clearly lists the locations before
asking the questions.

5.3 Compound or double barrelled questions


These questions combine more than one outcome in a single question.
Poor Wording: Has using the community kitchen enabled you to eat more
nutritious meals and make new friends?
This type of question would be confusing to someone who has eaten more
nutritious meals but who has not made any new friends. The person might not
answer the question outright or put down an answer that is meaningless.
Better Wording: It is better to avoid these problems by breaking a compound
question into two separate questions:
Has using the community kitchen enabled you to eat more nutritious
meals?and,
Has using the community kitchen enabled you to make new friends?

5.4 Leading questions


Leading questions are worded in such a way that the correct or socially
desirable answer is clear to the respondent. The respondent feels pressure to
give the acceptable answer that is implied in the question.

5
65 Questionnaires

Poor Wording: Research shows that pre-natal nutrition programs, such as our
Big Baby Fictional Program, achieve positive results. Do you agree? exerts
not-too-subtle pressure on the respondent to agree with the views of
research.
Better Wording: Did the Big Baby Fictional Program help you to eat more
nutritious meals during your pregnancy?

5.5 Loaded questions


Questions that have emotional wording that make the socially desirable response
obvious, do not yield credible data. Take care to avoid emotionally charged
words and phrases.
Poor Wording: Has our employment program enabled you to become a
contributing member of society?
If your program pertains to sensitive topics that are taboo or to topics that
respondents are prone to give socially acceptable responses, then you need to
exercise particular care to avoid loaded questions. You do this by avoiding
emotionally charged words or phrases.
Better Wording: Has our employment program enabled you to find work?

5.6 Negative questions


Respondents may become confused when a question is posed negatively. It
is hard to tell what it means to agree or disagree with the first statement.
Poor Wording: Do you agree or disagree with the following statement: Lack of
loneliness is not a problem for me since I enrolled in the Friendly Visitors
Program?
Better Wording: Drop one of the negatives Do you agree or disagree with the
following statement: Loneliness is not a problem for me since I enrolled in the
Friendly Visitors Program?

5.7 Hypothetical questions


Asking respondents to react to What...if? scenarios by offering their
opinions and views. Although this might seem innocent enough, the danger
of hypothetical questions is that instead of providing valid and accurate data,
they encourage respondents to speculate wildly. This is a favourite type of
question for media reporters because it leads to dramatic headlines, but those
experienced with the media avoid answering them...for good reason!
Poor Wording: Where would you be right now if this program didnt exist?
Better Wording: Please give one or two examples of a difference that this
program has made for you.

5.8 Implicit assumptions

Questions that make the error of assuming that respondents have the
necessary background information to answer a question. By not including the
essential information, the question does not generate valid results.

A Toolkit for Outcome Measurement

66

Poor Wording: Would you recommend this program to a friend? This wording
makes it impossible to know the benchmark the respondent is using to
answer the question or what type of friend the respondent has in mind.
Better Wording: Based on the results you have achieved, would you recommend
this program to a friend with a similar problem? makes the assumptions clear
by setting a standard for making a judgment and making clear the type of
friend.
A different set of assumptions could change the question greatly. The lesson
is clear review your questions for implicit assumptions and then clarify them.

5.9 Technical terms and jargon


Avoid using technical terms or jargon as they confuse respondents.
Poor Wording: Did your L-2 receptive ability increase significantly as a result of
completing our ESL program?
Better Wording: Can you understand English better as a result of completing the English as
a Second Language program?
If you must use a technical term, professional jargon, or acronyms (e.g.,
MOHLTC), be sure to define the term or acronym the first time you use it. The
general rule is to avoid these types of terms unless absolutely necessary and
replace them with plain language.

Step 6. Determine Question Order


Another important factor in questionnaire design is the order of the questions.
You should begin the questionnaire with simple, non-threatening questions
and arrange the order of questions to keep peoples interests.
As a general rule, place potentially sensitive, personal, or intrusive questions
at the end of the questionnaire. This includes the usual questions about the
status or characteristics of the respondent, such as gender, age, employment
status and education level.
It is wise to introduce the section that asks for these personal characteristics
with a statement that tells the respondent the purpose for asking these
personal questions. Usually a statement such as this information will be used
to help us better understand the responses.
The reason for placing personal or demographic questions at the end of the
questionnaire is that even if the respondent decides not to answer these
questions, most of the questionnaire is already completed. On the other hand,
if they are placed at the beginning and the respondent finds that they are too
intrusive or detailed, the respondent may decide not to answer the other
questions either.
It is important to arrange questions in a logical order and cluster questions on
the same topic together. Group questions on the same topic from general to
specific. Examine the sequence of questions carefully to see if answers to
earlier questions will influence or contaminate the responses to questions
following. Usually specific questions influence responses to general
questions.

67

Questionnaires

For example, asking first the specific question, Did you feel that the
counselor was caring? tends to influence responses to the more general
question, In an overall sense, how satisfied were you with the employment
program?.
On the other hand, asking general questions first does not have the same
effect on specific questions. For example, first ask the general question, How
useful have you found this manual? before asking specific questions, such
as, Have you had the opportunity to use the manual for a reference?

Step 7. Check The Questions


At this point it is essential that you confirm that the wording of the questions,
including their answer categories, is easy to understand and presented in a
logical manner. Although you might exercise great care in writing questions
and potential answers, it is impossible to know if the questions carry the
intended meaning or are clear to different respondents until you test them.
Checking the questions does not need to take a long time, but it does require
some planning and it follows a definite sequence. Checking the questions is
different from pilot testing the entire questionnaire, which comes in Step 9.
First, assemble the draft questions, including any alternative versions of the
questions, in a logical order according to topic. Arrange the questions within
each topic from general to specific. At this point, you will have more questions
than you need.
Now, have a few persons who are not members of the target group review
the questions. Ask them to spot any confusing terms and questions that are
not clear. Then revise the questions based on their feedback.
Next, recruit several members from the target group who will be given the
questionnaire. Ask them to review the wording to be sure it is clear, but also
ask them for advice about what questions should be changed, dropped or
added. Invite them to offer suggestions for improving the questions and
answer categories.
Finally, eliminate those questions that were unclear or not needed. You need
to be ruthless at this stage. Only the essential questions should remain.

Benefits of an Attractive Questionnaire Layout

Easier to follow topics and questions

Higher response rate for individual questions

Reduced errors in completing the questionnaire

More questionnaires completed

Step 8. Design Questionnaire Layout

Throughout this guide, we have seen that respondents understanding of


questions and their subsequent answers, are strongly influenced by the
context such as the logical arrangement of questions into topics and the flow
of questions from general to specific. The overall visual layout of the
questionnaire also influences the understanding of questions and the
willingness of the respondent to complete the questionnaire.

A Toolkit for Outcome Measurement

68

Perhaps the most important questionnaire design decision is the length of the
questionnaire. An overall guideline for questionnaires is that they should be
kept as short as possible no more than 4-6 pages in length. Another
recommendation is that a questionnaire should have no more than 5
questions per topic area. This can be accomplished by selecting questions that
link directly to the highest-priority outcome measurement information needs
of the stakeholders.
The cover letter is another important feature of a questionnaires physical
layout. It can be used to build trust and encourage completion of the
questionnaire. Think carefully about the cover letters title and introductory
statements, because it is at this point that people will decide whether the
questionnaire is worthwhile enough to merit further reading (and completion).
A title for the questionnaire should state its purpose clearly, such as
IMPROVING SERVICES TO CHILDREN AND FAMILIES.
The title is followed by introductory statements that describe:
l

The purpose of the questionnaire

How you will use the information

The confidential nature of the questionnaire

What you want the respondent to do (clear instructions for completion)

When you want the questionnaire returned

How you want the questionnaire returned

Who to contact for more information about the questionnaire.

After the cover letter, it is the layout of the questions themselves that
encourage respondents to complete the questionnaire. A clear layout with
plenty of white space and enough room for the questions are the important
elements. A common complaint about questionnaires, especially those for
older persons, is that they are difficult to read, with a poor choice of jumbled
and seemingly microscopic type.
Good questionnaires also employ a mix of different question structures (see
Step 4). Although closed questions are easy to complete, they can become
boring unless the types are mixed and an open question is added occasionally
to engage the respondents attention.
Be sure to use typefaces that are easy to read and large enough to see
without great effort.
With a clean layout and ample white space, usually you can fit 6-8 questions
to a page. Avoid cramming too many questions to a page or having a question
straddle across two pages. If you find that your questionnaire is too long once
you have completed the layout, it is better to trim the number of questions or
the number of topics, rather than crowd the layout.

5
69

Questionnaires

Case in Point
The following cover letter from West Scarborough Neighbourhood Community Centre
invites teacher feedback as part of the evaluation of the Boys and Girls Stay-in-School
program. Note how the statement emphasizes the purpose of the program, ensures
confidentiality and offers persuasive reasons why the teacher should take the time to
complete the questionnaire.
Dear Teacher,
Were evaluating a Stay-In-School homework/tutoring program at the West Scarborough
Boys & Girls Club. We would like your cooperation in evaluating the academic
performance of some of the students in our program.
Your answers to the following questions will help us determine the effectiveness of the StayIn-School program which is aimed at increasing students involvement in productive afterschool learning activities. It will also serve as a useful tool to evaluate programs and services
at the Club.
Please be as honest as possible. Everything you say is confidential. Neither you, your school,
nor the student will be mentioned in any of our reports.
The West Scarborough Boys & Girls Club Stay-In-School program focuses on several areas
that relate to the student's performance.To evaluate the program, we are going to ask you
about each students performance in various areas.
Thank you in advance for your time and cooperation.

Step 9. Pilot Test And Revise The Questionnaire


The final stage of developing effective questionnaires is to test the
questionnaire with a sample of respondents to identify any problems in
questionnaire layout.
The final pilot test does not take much time, but it is an essential part of the
process that should not be overlooked. The best pilot test simulates the use of
the questionnaire with your real target group. Select a small sample of persons,
usually between five and fifteen persons and try to make the sample a fair
representation of your overall target group.
After the questionnaires are completed, review the questionnaire with the pilot
test participants. Take detailed notes and list the problems for each question.
Usually the changes are obvious, but if you have any questions, then discuss
them with your pilot test group.

Checklist for Reviewing the Final Questionnaire

Title and purpose of the questionnaire are clear

Each question is easy to answer

Instructions are easy to follow

Topics and questions flow logically

Layout is attractive and not crowded

Questionnaire is the right length

Meaning of each question is clear

Questionnaire design engages and maintains interest

A Toolkit for Outcome Measurement

70

Step 10. Implement the Survey: Hand Out, Mail Out and E-mail
There are several ways to administer questionnaires, each has its strengths and
limitations. The purpose of this section is to help you choose the best method for your
situation. Consider some of the following questions before choosing an administration
method: What is the literacy level of your respondents? What is the most
convenient way to reach them? Are they more likely to respond to a survey if its
administered by mail?

10.1 Hand-Out Surveys


Handout surveys involve personal contact with respondents. People are more likely to
agree to complete questionnaires when the request is made in person. As such,
handout surveys typically achieve high response rates. Another benefit to handout
surveys is that the administrator can address, on the spot, respondents concerns
about how to answer a given question. Handout surveys are practical to use at the end
of training sessions, workshops, conferences or other time-limited programs as a
means of gathering immediate evaluative feedback. These surveys take advantage of
having program participants all in one place so that data can be gathered quickly and easily.

10.2 Mail Out Surveys


Mail surveys can be a relatively quick and inexpensive way of collecting data from
respondents. They give respondents the opportunity to carefully consider each
question and allow adequate time for detailed comments to open questions. Mail
surveys also give respondents a greater sense of confidentiality than handout surveys.
Mail surveys can be vulnerable to low return rates. You can use the following techniques
to offset this vulnerability and achieve higher return rates of 60% or more:
l

Provide an option to fax the completed survey, with fax number included in the
instructions

Include a stamped, self-addressed envelope

Create a follow-up process that includes a mailed reminder

Send a second mailed reminder with another copy of the questionnaire.

10.3 E-mail Surveys


The term e-mail survey describes questionnaires sent electronically to individuals as
well questionnaires posted on a web-site (more accurately web-based survey).
Once people become comfortable with e-mail, they usually prefer e-mail surveys to
mail out surveys because they are easier to use. Completion of questionnaires by email reduces errors and costs for data entry.
A multiple-channel (mixed mode) approach may be needed to conduct a
comprehensive survey because some populations cannot be reached by e-mail. In
these situations, e-mail surveys are sent to individuals with e-mail addresses and hard
copies are mailed to those without e-mail. Researchers have found that multiplechannel surveys provide similar response rates to mail surveys but with faster
response times and lower costs.

5
71

Questionnaires

Step 11. Analyze The Data


Closed-ended questions from questionnaires can be analyzed using basic
statistical procedures that are descriptive in nature. These techniques will help
you and your audience better understand the data. Some basic techniques will
be highlighted here.
l

Frequencies (i.e., how often a behaviour or activity was observed,


number of participants)
Percentages (i.e., percent of youth observed playing board games
together; percent of ratings that were satisfactory)

Averages (i.e., average rating of physical mobility)

Range (i.e., lowest and highest rating, diversity of observations, etc.).

You can use all of the above techniques to present the data as a whole
(aggregated) or when you break the data down into subgroups (disaggregate).
This technique is also referred to as cross tabulation. For example, you may
want to know how client outcomes vary with the demographic characteristics
of program participants, such as age, gender or level of education. Some
common ways of disaggregating the data include examining:
l

Client characteristics (age, gender, education)

Type of service (differences in service delivery)

Service intensity (amount of service received)

Service duration (length of time in program)

Program factors (staffing levels, resource levels, location).

Because your questionnaire may include both closed and open questions, it
will yield quantitative and qualitative data. For interpreting qualitative data,
content analysis should be undertaken. Refer to Chapter 6: Conducting
Effective Interviews for instructions.

5
A Toolkit for Outcome Measurement

72

Examples
Developing Effective Questionnaires
5A Outcome Measurement Training Workshop
United Way of Greater Toronto
5B Some Common Concerns of Women Seeking Work
Times Change Womens Employment Service
5C Times Change Womens Employment Service
Program Evaluation
5D Job Search Workshop Evaluation Form
Times Change Womens Employment Service
5E Lets Discuss It! Multicultural Womens Wellness Group Intake Form
Canadian Mental Health Association Metro
5F Lets Discuss It! Multicultural Womens Wellness Group Evaluation
Canadian Mental Health Association Metro
5G West Scarborough Boys and Girls Club
What do you do after school?
5H Teacher Survey
West Scarborough Boys and Girls Club
5I Childrens Homework Club Satisfaction Questionnaire
Family Day Care Services
5J Parent Evaluation of The Homework Club
Family Day Care Services
5K Teacher Evaluation of The Homework Club
Family Day Care Services
5L Workplace Mentor Evaluation
Evas Phoenix
Note:
the formats of the following outcome measurement tools have been altered to some degree
to correspond with the design and space restrictions of this toolkit. We hope you will draw
inspirations, confidence, and even concrete elements for your own use, drawing upon the
steps and guidelines set out in these pages.

5
73

Questionnaires

Example 5 A
United Way of Greater Toronto
Program Effectiveness Organizational Development Feedback on Outcome
Measurement Training Workshop
Date:_________________
Please respond to the statements below by circling the appropriate number to the right of each
strongly
agree

strongly
disagree

statement.

1. The information provided in the workshop increased


my knowledge of United Ways PEOD project................................................1
2. The information from this workshop gave me ideas
for using evaluation in my organization.........................................................1
3. I understand the difference between outcomes
and other program components.........................................................................1
4. I have been taught how to identify appropriate program outcomes..... ....1
5. I understand how outcome measurement contributes
new information for program development......................................................1
1
6. The benefits and limits of outcome measurement are clear to me
7. As a result of attending this workshop, I feel confident in my
ability to help staff at my agency understand program logic models.
1
8. There was not enough time for me to fully grasp all of the concepts. 1
9. I am clear about what United Way wants agencies to learn and
do during the PEOD project................................................................................1
10. The information provided has adequately prepared
me for developing program logic models.................................................1

2
2

3
3

4
4

5
5

2
2

3
3

4
4

5
5

2
2

3
3

4
4

5
5

For the following statements, please rate how you feel about the workshop overall. (circle one response for each question)

11. How would you rate the workshop overall?..... poor


12. How would you rate the overall content of the workshop?

fair
too basic

good

very good

excellent

good balance too technical

13. Would you recommend this workshop to someone in your position?. yes probably

maybe

no

Thank you for taking the time to respond to these questions.

5
A Toolkit for Outcome Measurement

74

Example 5 B
Times Change Womens Employment Service
Some Common Concerns Of Women Seeking Work
The women we see at Times Change have made us aware of the many obstacles which women face in seeking
work. When a woman says, I need a job, or I want to change occupations, thats usually just the tip of the
iceberg; there are often underlying issues.
We have developed this handout to guide you in defining some of your underlying needs. This will enable you
to help you figure out how to get started. The examples given are to assist you in defining your own issues.
Additional space is provided to add other concerns. This handout is for you to keep.i
SOME OF MY CONCERNS INCLUDE:
AREA 1 CAREER PLANNING

YES

NO

YES

NO

I dont know what I can do.


I dont believe I have any skills.
I am no longer able to do the work I once did. I want to explore other options.
I want information on the labour market or areas where there will be job vacancies in the future.
I want information about different occupations before I develop new skills.
Is there a market for the kind of job I prefer?
How do I research occupation choices?
What is the effect of new technologies on women?
I want to be clearer about my interests.
I dont know whether Im job ready.
I need to build up my self-confidence.
Im not sure that I want to work.
AREA 2 EDUCATION AND TRAINING
I need help with the application forms for college/university.
How can I finance my education?
I need information about training through government programs.
How can I get qualifications from my own country evaluated here?
Where can I get upgrading?
What are training possibilities?
I am experiencing language difficulties.

5
75

Questionnaires

Example 5 B
Times Change Womens Employment Service
Some Common Concerns Of Women Seeking Work (continued)
AREA 3 JOB SEARCH WORKSHOP

YES

NO

YES

NO

I need money.
I am experiencing cultural barriers.
I want to know about the hidden job market.
I have not held a paid job for a while (anywhere from 6 months to 20 years).
I am getting discouraged looking for work.
Illness of a family member is affecting my job search.
I am a key (or the only) wage earner in my family and Im feeling pressure.
I think I have the skills but I dont know how to market them.
How do I write a better resume?
What is a covering letter?
I need to learn to handle interviews.
I dont know how to make contacts with potential employers.
I am experiencing discrimination.
My experience in my country is not accepted here.
I dont know how to find job vacancies.
AREA 3 - FAMILY OR PERSONAL
I am going through major life changes.
I have conflicts about working outside the house while my children are young.
Attitudes within my family create difficulties for me.
I am going through a major change in my family situation; examples might be: recently
widowed, separated, divorced, self-supporting, new mother, empty nest mother.
I want to know more about: Unemployment Insurance; Day Care; Family Benefits; Welfare;
Canada Employment Centres; Immigration;Workers Compensation Board; Human Rights
Commission; Employment Standards; Unions.
I have questions about my status as a landed immigrant, as a Canadian citizen.

5
A Toolkit for Outcome Measurement

76

Example 5 C
Times Change Womens Employment Service
Womens Employment Service Program Evaluation
Date:_________________
1.

How helpful was Times Changes educational counselling service for you? (please circle one)
very helpful...... helpful...... somewhat helpful...... not very helpful...... not at all helpful

2.

What did you find most helpful?

3.

What did you find least helpful?

4.

What information would you like to see further covered?

5.

6.

7.

Has educational counselling helped you: (please circle one answer)


clarify your training needs?..................................................................................Yes..............................No
l become more knowledgeable about available training options?..................Yes .............................No
l develop a plan of action to meet your education/training goals?..............Yes..............................No
l

After the initial appointment, did you feel comfortable asking for further assistance from the
counsellor? (please circle one answer)...............................................................Yes ..........................No

How satisfied were you with the counsellors performance on the following:
sensitivity to individual needs (please circle one)
very satisfied.......satisfied.........somewhat satisfied......dissatisfied........very dissatisfied

8.

providing information (please circle one)


very satisfied.......satisfied.....somewhat satisfied..........dissatisfied........very dissatisfied
Are there any additional comments on the counsellor or on the service you would like to add?

________________________________________________________________

77 Questionnaires

Example 5 D
Times Change Womens Employment Service
Job Search Workshop Evaluation Form
This evaluation form is intended to rate your satisfaction with the service you received in this workshop.
We appreciate your cooperation in answering the questions.
1. In general, how helpful did you find this workshop? Circle one.
very helpful ...................................... moderately helpful ..................................... not helpful
.....................................
2. What did you find MOST HELPFUL about this workshop? Why?

3. What did you find LEAST HELPFUL about this workshop? Why?
4. Place a

ain the appropriate column:


YES

Did you learn:

NO

Any new methods of job search?


How to find the hidden job market?
How to write a resume?
How to write an effective covering letter?
To present yourself effectively in an interview?
Other (please specify)
5. If you have answered NO to any of the above questions, what further assistance do you need?

6. Have you any suggestions as to how we could improve the WORKSHOP?

7. Which of the HANDOUTS did you find most useful? Please specify.

8. Do you have feedback for the COUNSELLOR?

5
A Toolkit for Outcome Measurement

78

Example 5 E
Canadian Mental Health Association Metro
Lets Discuss It! Multicultural Womens Wellness Group Intake Form
The following information will help us make sure that the groups are meeting your needs.
D D / M M /Y Y

,
First name

First letter of last name

Date started group

Please circle the correct answer, and fill in any information where asked.
1. Age
18-24 .......

25-34 .......

35-44 ......

45-54 ......

55-64 ......

65 or over ......

2a. Birthplace
I was born in Canada
I was born in (name of country)______________________________________________________
2b. If you were born in another country, How long have you lived in Canada?
Less than 2 years .......... 2-5 years .......... 6-10 years........... More than 10 years............

3. Marital/relationship status
Single ...............

Married ................

Separated ................

Divorced ...............

Domestic relationship (same sex/common-law partner) ................ Widowed ................


4. Children
a. How many children do you have who are under the age of 16 and living with you?
None .......... One .......... Two.......... Three.......... Four........... Five or more...........
b. How many children do you have who are 16 and over and living with you?
None .......... One .......... Two.......... Three.......... Four........... Five or more...........
5. How did you find out about this group?

From a friend............ From a settlement worker.......... From a local newspaper ..........


From (other please specify)............
_______________________________________________________________________________________

79

Questionnaires

Example 5 F
Canadian Mental Health Association Metro

Lets Discuss It! Multicultural Womens Wellness Group Evaluation


Current date:

Date started group:


(dd/mm/yy)

(mm/yy):

Group location:
Your feedback is important to us as it will help us make sure that the services we provide are meeting your needs.
Please circle the number that is closest to how you feel.
strongly
agree

strongly
disagree

1.

I cope better with family matters........................................................1

2.

I manage things better at home...........................................................1

3.

I feel better about myself.......................................................................1

4.

I feel I have more support from other people.................................1

5.

I know more about what is available to me......................................1

6.

Being able to talk with other women has been helpful..................1

7.

Overall, I feel much better.....................................................................1

8.

I would recommend these groups to other women like myself...............1

9.

I understand more about the Canadian way of life.........................1

Please complete the following sentances:


In the past year, the topics of most interest to me were:
__________________________________________________________________________________________
__________________________________________________________________________________________
What I like most about groups is:
__________________________________________________________________________________________
__________________________________________________________________________________________
What I think would make these groups better is:
__________________________________________________________________________________________

__________________________________________________________________________________________

A Toolkit for Outcome Measurement

80

Example 5 G
West Scarborough Boys and Girls Club
CODE:
0300

What do you do after school?


How old are you?________ Are you a girl or boy?________
Instructions:These sentences are about what you do after school.
Read each sentence and circle the number that tells what you do after school.
1. I watch TV

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

2. I read books

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

3. I read magazines

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

4. I talk to people about what I read

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

5. I read out loud with other people

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

6. I get help in doing my homework

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

7. I help others to do their homework

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

8. I write on my own time

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

9. I play with other kids at the club

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

10. I have friends at the club

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

11. I make and keep new friends

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

12. I feel good about myself

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

13. I work out problems without getting into a fight

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

14. I like to decide what games to play with my


friends

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

15. I like it better when others decide what I should


play

All the Time 5

A Lot 4

Sometimes 3

Once in a While 2

Never 1

16. My three favorite things to do are:


a)
b)
c)
17. My three least favorite things to do are:
a)
b)
c)
18.When I grow up I would like to be:

81

Questionnaires

Exhibit 5H
West Scarborough Boys and Girls Club
CODE:
0200

Teachers Questionaire

Name of student: ____________________________GRADE:________ SCHOOL:_______________________


Is the student outgoing or quiet?

Is the student enthusiastic about learning?

In which subjects does the student show the most promise?

In which subjects does he/she struggle?

Using a scale of 1 to 10 (where 10 is good and 1 is poor), answer the following questions according to how well
the student performs each activity.
1. How would you describe the students overall performance in school?
10
9
8
7
6
5
4
3
2

Do Not Know

2. How does the students academic performance compare to other students you have known throughout your career?
10
9
8
7
6
5
4
3
2
1
Do Not Know

3. How well does the student read books and magazines?


10
9
8
7
6
5
4

Do Not Know

4. How well does he/she write?


10
9
8
7

Do Not Know

5. How well does the student know math ?


7
6
5
8
10
9

Do Not Know

5
A Toolkit for Outcome Measurement

82

Exhibit 5H
West Scarborough Boys and Girls Club
Teachers Questionaire (continued)
Using a scale of 1 to 10 (where 10 is good and 1 is poor), answer the following questions according to how
frequently the student performs each activity.
Does the student show interest in class material?
10
9
8
7
6
5

Do Not Know

Does the student come to you with questions?


10
9
8
7
6
5

Do Not Know

Does he/she talk about what he/she is reading ?


10
9
8
7
6
5

Do Not Know

Does he/she read aloud with others ?


10
9
8
7
6

Do Not Know

Does the student do his/her homework?


10
9
8
7
6

Do Not Know

Does he/she get help with his/her homework?


10
9
8
7
6
5

Do Not Know

Does the student write notes to other students?


10
9
8
7
6
5

Do Not Know

Does the student play constructive games with other students?


10
9
8
7
6
5
4
3

Do Not Know

Is the student helpful in class?


10
9
8
7

Do Not Know

Would you like to add anything that might be relevant to our evaluation?

Thank you for your time. This will be of great assistance to us as we complete our evaluation!

83

Questionnaires

Example 5 I
Family Day Care Services
Childrens Homework Club Satisfaction Questionnaire

1. Have you enjoyed coming to the homework club?

2. Has the homework club helped you with your school work? In what ways?

3. How often do you finish your homework now? (Circle one)


ALWAYS ............................................... MOST OFTEN ........................................... SOMETIMES NEVER

4. Are you doing better in school now?

5.What other activities did you enjoy at the homework club?

5
A Toolkit for Outcome Measurement

84

Example 5 J
Family Day Care Services
Parent Evaluation of the Homework Club
Please take a few moments to complete this survey. We would appreciate your participation. Thank you!
1. Did your child like coming to the homework club?

2. Did your child benefit from coming to the homework club?

3. Did your child improve academically?

4.Was there an increase in your childs self-confidence?

5. Has there been any improvement in your childs social skills? (Getting along with others, joining in,
assertiveness, ability to resolve conflicts more appropriately)

5
85

Questionnaires

Example 5 K
Family Day Care Services
Teacher Evaluation of the Homework Club
Please complete this evaluation and put in daycare mailbox. Thank you.
1. State the reason that the child was referred to the homework club.

2. Before starting the homework club, how often was the child completing homework?
ALWAYS ......................................................... MOST OFTEN ...................................................... RARELY
Now, how often is the child completing homework?
ALWAYS ......................................................... MOST OFTEN ...................................................... RARELY
3. How has the program helped the child academically?

4. Has there been any increase in the childs self-confidence?

5. Has there been any increase in the childs level of commitment to learning?

6. Has there been any development in the childs social skills and a reduction of anti-social behaviour?

5
A Toolkit for Outcome Measurement

86

Example 5 L
Evas Phoenix
Workplace Mentor Evaluation
Introduction
Below you will find a series of questions relating to your Workplace Mentor or buddy. The purpose of
this questionnaire is to allow you the opportunity to give feedback as to the impact your workplace
mentor had on your employment placement. Please feel free to ask any questions you have.
Participant Information:
Participant Name

Mentors Name

Employment Start Date

Employment Support Worker

Employer Information
Company Name

Supervisors Name

Site Location

Contact Name

NOTE: All responses to the questions on this form will be kept confidential.

Questionnaire NOTE: Place a

a in the appropriate box where applicable.

A. Comfort level in job placement.


1. How comfortable would you say you are in your job placement?
c Very Comfortable c Somewhat Comfortable c Not comfortable at all
2. If you answered very or somewhat comfortable, how soon after you started did you begin to feel
comfortable at your job placement?
c Less than a month c 1-2 months
c 3 or more months
Any Comments?

5
87

Questionnaires

Example 5 L
Evas Phoenix
Workplace Mentor Evaluation (continued)
B.Youth and Mentor partnership.
1. How much support has your Workplace Mentor been to you?
c A very big support

c A good support

c Not much support

c No support at all

2. Did you feel that you could approach your Workplace Mentor at any time with concerns or
questions you had?
c Yes
c No
3. How strongly do you agree with this statement?
The relationship with my Workplace Mentor and myself was a supportive
partnership.
c Strongly agree

c Somewhat agree

c Disagree

c Strongly disagree

4. Do you see your Workplace Mentor as someone you can confide in and/or seek assistance
from on the job site?
c Yes, definitely

c Sometimes

c No, never

5. In what area do you think your Workplace Mentor has been the most help to
you in your job placement?
Any Comments?

C. Familiarity with employers policies and procedures.


1. Does the company you are working for have written Policies and Procedures?
c Yes

c No

2. If Yes, have you read it? c Yes

c I dont know

No

3. How familiar are you with the unwritten rules? E.g., where to smoke, when to
take a break, when does the coffee truck arrive, how to approach your boss etc.
c Very Familiar

c Somewhat Familiar

Not familiar at all

Any Comments?
4. How satisfied do you feel your employer is with your work?
o Very satisfied o Somewhat satisfied o Slightly dissatisfied

o Not satisfied

Any Comments?

5
A Toolkit for Outcome Measurement

88

Example 5 L
Evas Phoenix
Workplace Mentor Evaluation (continued)
D. Relationship with employer and fellow employees.
1. How well do you feel you fit in with the rest of the employees at your work
site?
c Very well

c Good

c Still trying to fit in

Dont fit in at all

2. How do you get along with your boss/supervisor?


c Very well

c Good

c Sometimes dont
get along

c Dont get along at


all

c Slightly dissatisfied

c Not satisfied

3. How satisfied are you with your work?


c Very satisfied

c Somewhat
satisfied

4. How satisfied do you feel your employer is with your work?


c Very satisfied

c Somewhat
satisfied

c Slightly dissatisfied

c Not satisfied

Any Comments?

E. Miscellaneous.
1. At this point in your placement do you feel ?
c Much better prepared to find and keep employment
c Somewhat better prepared to find and keep employment
c Not any better prepared to find and keep employment
c Not prepared at all to find and keep employment
2. Overall has this placement been a positive work experience?
c Yes
c No
Please feel free to give any other feedback here:

1
5
89

Questionnaires

Chapter 6:
Conducting Effective Interviews

What Are Interviews?


An interview is a way of gathering in-depth information about a program
through a discussion between a person, called a respondent, and an
interviewer. Interviews can be conducted face-to-face or over the telephone.
Although a good interview may look like a conversation between friends, at
the heart of the evaluation interview is a carefully constructed set of
questions and a carefully developed interview process guided by a skilled
interviewer. Even so, interviews are more flexible and less structured than a
written questionnaire.
Interviews, like focus groups, are a way of obtaining information directly
from program participants about the kinds of outcomes and benefits they
experienced during or after their involvement in a program. Questions allow
for in-depth exploration about how and why client outcomes were achieved,
the strengths and limitations of programs, and suggestions for program
improvement. When managed properly, individual interviews can be a rapid
and relatively inexpensive way of collecting credible data about program
outcomes and other aspects of program performance. Interviews are often
used in combination with other evaluation methods, especially program
statistics and questionnaires, as a way of interpreting the findings and
providing reasons for the reported outputs and outcome results.
There are two main types of individual interviews. Personal interviews are
conducted face-to-face between an individual respondent and an interviewer.
Telephone interviews require the respondent and the interviewer to interact
over the telephone, presenting unique opportunities and challenges.
A third type, called key informant interviews, is unique because of the way
respondents are selected, usually through a technique called snowball
sampling(refer to Chapter 3: Sampling). Conducted in person or by
telephone, key informant interviews gather information and opinions from a
relatively small number of recognized experts or leaders. A discussion of
this special interview option concludes this chapter.

1
A Toolkit for Outcome Measurement

90

Agency in Action
Community Information Torontos (CIT) Community Helpline provides information to callers
about social, health and government services that are available to them.This information,
which is offered in more than a dozen languages, helps Torontonians connect to the services
that they need.
After having their questions answered by a phone counsellor, callers are asked to participate
in a brief on-the-spot phone interview.This interview was designed to assess demographic
characteristics of callers (e.g., age, first language, level of income, etc.) and give CIT a better
understanding of who they are serving. At the interviews conclusion, callers are asked if they
would be willing to participate in a follow-up phone interview at a later date. If participants
agree, they are asked for their name, contact information, and best times for them to be
reached. CITs approach to recruiting participants for the follow-up interview is realistic and
innovative given the anonymous, short-term nature of providing information and referral to
individuals over the phone. It takes only a moment to ask clients if they would be willing to
be interviewed at a future date.
CIT uses the follow-up telephone interview to gauge client satisfaction with services received
at the time of their initial call to the Community Helpline.This feedback helps reveal strengths
and weaknesses of program functioning as perceived by callers.The follow-up interview also
collects information about client outcomes (e.g., did callers follow up on referrals they were
given?, did callers get help from the organizations they were referred to?). By combining the
information collected from the first interview and the follow-up interview, CIT can assess
whether achieved outcomes vary according to demographic characteristics and information
needs of callers.

When Are Interviews Useful


for Measuring Outcomes?
l

When you want detailed feedback about program experiences.


Interviews, either in person or by telephone, are one of the best ways to
obtain detailed information about program outcomes and program
satisfaction. Often, interviews are used to investigate survey findings in
more depth or to follow up on complaints.

When you need to be sensitive to and accommodate the


respondents. Interviews are useful when you need to collect
information from people who speak different languages, who have
special needs such as not being able to read, or whose cultural practices
or other life circumstances make other data collection methods difficult
or inappropriate.

When you need to investigate sensitive topics or issues.


Confidential interviews are the best way to obtain information about
sensitive topics or issues, such as sources of income, lifestyle behaviour
and opinions about staff. Interviews are also useful to obtain feedback
from individuals who may suffer from discrimination or prejudice because
of some factor, such as being HIV-positive, a former prisoner, illegal
immigrant or intravenous drug user.

6
91

Interviews

When quantitative data collected through other methods needs


to be interpreted. Interviews can help you understand the statistical data
provided by other evaluation tools, such as program records or
questionnaire surveys.

When basic information is needed to design a comprehensive


evaluation. Interviews can help you discover key evaluation questions
and issues that need to be addressed in an evaluation.

When the main purpose is to generate recommendations.


Interviews with knowledgeable persons, whether program clients or
program experts, can help you formulate solid recommendations for
improving program performance.

What are the Main Strengths of Personal


Interviews?
l

Permits clarification and more complete responses. The interview


situation permits clarification if the respondent does not understand a
question or if the interviewer is unclear about an answer. In addition, the
interviewer can ask for more details that may make the response more
complete.

Interview process builds trust. Individual interviews permit an


interviewer to establish a climate of empathy and trust that encourages
respondents to be open and honest about their true feelings regarding the
topics discussed. In personal interviews, interviewers can establish rapport
by using their appearance, empathy, personality, and persuasive skills to
build trust. This is more difficult for a telephone interviewer who typically
has less time and personal influence to win over the respondent.

Suitable for persons with low literacy levels. Individual interviews


do not require reading or writing. Questions and answers can be repeated.
The interview can be conducted in a variety of languages.

Response rate usually higher than for mail surveys. The response
rate for properly conducted personal and telephone interviews is higher
than for mail surveys. Busy individuals usually are more willing to schedule
an interview than to read and complete a questionnaire. The interviewing
process permits the interviewer to make numerous call-backs until the
respondent is reached and a suitable time is arranged.

Main Strengths and Limitations of Interviews

Strengths

Permits clarification and elaboration of responses

Requires skilled interviewers

Interview process builds trust


Useful for populations with low literacy levels

Can be labour intensive and expensive


(usually costs more than mail surveys)

Greater completion rate than paper


and pencil surveys

Coding and analyzing interview data


can be time-consuming

Limitations

A Toolkit for Outcome Measurement

92

What are the Limitations of Individual


Interviews?
l

Requires a skilled interviewer. The success of an interview is highly


dependent on the skills and personal qualities of the interviewer to obtain
accurate information and probe for deeper meanings of respondents
answers.

Interviews can be labour intensive. Interviews are labour-intensive


because the interviewer must make the initial contact, often involving
multiple follow-up calls to schedule interviews, and then conduct the
interviews. Furthermore, interviews involving travel can take as much as
one-half day for each interview.

Considerable time is required to code notes and analyze


interview data. In-depth interviews can generate large volumes of
notes that must be coded and analyzed. This can be a time-consuming
process.

Selecting Personal or Telephone


Interviews
Personal interviews and telephone interviews have many strengths in
common. There are some situations, however, where conducting an
individual interview over the phone may be more appropriate than doing one
in person (and vice versa). Consider the following points to help you decide
if a face-to-face or telephone interview would be preferable:
Choose Personal ifDifficult-to-reach populations. Personal interviews
may be the only practical way to gather data from people who are highly
transient or homeless, without telephones or who live in the urban shadow
world and grey economy such as prostitutes, illegal immigrants or exploited
workers.
Choose Personal ifPeople will not be comfortable discussing sensitive
topics on the phone. Some people may be reluctant to discuss personal
issues or sensitive topics unless they can verify the interviewers identity
and the confidentiality of the call. During personal interviews, the
interviewer can offer written and verbal guarantees of confidentiality, answer
questions about the purpose of the survey and how the information will be
used.
Choose Telephone ifYou need rapid results. Telephone interviews
produce results much faster than personal interviews (or mailed
questionnaires).
Choose Telephone ifCost and Time. Telephone interviews do not involve
the travel costs of personal interviews. If you conduct telephone interviews
regularly, the process can be partially automated through computer-assisted
telephone interviewing, reducing costs per interview further.
Choose Telephone ifSafety. In some situations, it may be unsafe to
conduct personal interviews in certain locations or at certain times of day or
night. The telephone interview provides a secure and flexible alternative.

93

Interviews

Case in Point
The Canadian Red Cross (Toronto Region) interviews seniors over the phone instead of
mailing out questionnaires because:
l

Interviews allow them to ask more probing questions and gather more relevant
information
l They can obtain feedback from a larger sample of clients
l They can use trained volunteers to conduct interviews, thereby freeing staff time
l Clients prefer and enjoy sharing their views over the phone.

Eight Steps for Developing Effective


Interviews
We will examine the major principles of interviewing. Personal and
telephone interviews will be discussed together, but the unique aspects of
each method will be highlighted wherever relevant.
EIGHT STEPS FOR DEVELOPING
EFFECTIVE INTERVIEWS
Step 1. Prepare a questionnaire
Step 2. Design the interviewer script
Step 3. Choose an Interviewer
Step 4. Train the Interviewer
Step 5. Recruit Respondents
Step 6. Conduct the Interview
Step 7. Record Interview data
Step 8. Code and analyze results

Step 1. Prepare a Questionnaire


An interview involves asking questions of people. Before you begin to plan
the interview process, take time to prepare a series of questions that will
elicit the information you need. Make a list of what you want to find. Be
clear about what you will do with the information. Eliminate questions that
give information that may be nice to know but which you will not likely
use.
Follow the guidelines set out in Chapter 5: Developing Effective
Questionnaires. You can use both open-ended and closed-ended questions,
such as two-option,
multiple-choice or rating scales. Open-ended questions are often the
simplest way to ask for information and probe for detail but narrative
responses are harder to record accurately and analyze so consider using
some different formats to help you collect and organize the information you
need in the most efficient manner.

6
A Toolkit for Outcome Measurement

94

Step 2. Design the Interviewer Script


A clear format and an orderly process are essential elements of successful
interviewing. Unless you gather the data in a systematic way from all
respondents, interview data will not be valid.
Because each interview reflects the unique chemistry of respondent and
interviewer, it is impossible to give a fixed set of rules to follow under all
circumstances. There is general agreement, however, that good preparation is
the key to successful interviewing.
You will first need to write a script that is based on the set of questions to be
asked during the interview. Scripts guide interviewers so that all of the
interviews are conducted as consistently as possible. Scripts help pace
interviews, keeping them on track, and help the interviewer avoid adding
comments that might bias the views of respondents. It is good practice to
pilot-test the script on a small sample from the target group of respondents.

Outline for an Interviewers Script

Instruction to Interviewers

Introductory Remarks

Interview Questions

Wrap-Up

Interview Completion

An interview script should include the following elements:


l

Instructions to Interviewers. Briefly remind interviewers of the data


collection process. Ask interviewers to read the questions over before
beginning each interview. Give guidelines for answering respondents
questions or complaints about the interview. Identify whom the interviewer
should contact when there are problems that cant be settled. Consider
including frequently asked questions as a preparatory tool, providing
standard answers to typical questions that a respondent may ask about the
interview. If the interview is done by phone, then include instructions on
how to place and receive calls, including long-distance calls, and how to
place calls on hold, use a hands-free headset, and how to record calls, if
relevant. Remind interviewers how you want calls tracked, so that you have
an accurate record of who was called, how many times, at what numbers,
and what action needs to be taken, for example, Call Tuesday at 3 pm at
416-999-9999 extension 99. Ask for DiscMaster B.
Introductory Remarks. This part of the script is an opening statement that
the interviewer reads to respondents at the start of the interview. It should
include an overview of the purpose and importance of the evaluation and an
estimate of how long the interview is likely to take. Include comments to
reassure respondents about the confidentiality of the information and how it
will be used. At this point, instruct interviewers to engage respondents in a
friendly manner by asking them if they are comfortable or if they have any
questions about the interview process.

95

Interviews

Interview Questions. This section of the script presents the series of


questions that the interviewer will ask respondents. Give directions on
how respondents are to answer each question. It is better to repeat
directions too often than not enough. This is especially true for closedended questions that use multiple choice or rating scales because
participants may not remember the answer categories from which they are
to select a response (or rank a response). Include transitional comments to
enhance the flow of the interview such as when a new topic is introduced.
If some questions may not apply to all respondents, provide clear
instructions for interviewers to follow. For open-ended questions, include
probes that interviewers may use to ask for more details.
Wrap-Up. This part of the script gives respondents a signal that the
interview will end soon, with phrases such as, Weve discussed a great
deal. I really appreciate your help and your time. Id like to finish by asking
you, .?
Interview Completion. The final section of the script reminds the
interviewer to complete housekeeping, such as logging the finish time of
the call, review the interview notes and alert the supervisor or evaluator
about any problems.

Step 3. Choose an Interviewer


Selecting good interviewers is very important. Screen potential interviewers
for their ability to listen and their willingness to follow interview guidelines.
The character and skills of an interviewer are crucial aspects of the interview
process.
Consider having a written agreement for interviewers (both paid and
volunteer). A written agreement with clear job responsibilities also helps to
screen out persons who do not work well in interview situations. The
agreement should state the purpose of the interviews, spell out the
interviewers responsibilities and emphasize the need for confidentiality.

Outline for an Interviewers Script

Instruction to Interviewers

Introductory Remarks

Interview Questions

Wrap-Up

Interview Completion

Step 4. Train the Interviewer

Training is essential for interviewers, whether they are agency staff or


volunteers. Training supplies general information about the interview
process, as well as details about the specific evaluation study. This helps
interviewers to answer respondents questions and build rapport. Training
emphasizes good interviewer behaviour. Role-plays are an effective training
tool. Role-play various interview situations to reinforce and evaluate
knowledge and skills of the trainees.

A Toolkit for Outcome Measurement

96

The specific information you give trainees about the particular evaluation
should include:
l

Discussion of the purpose and general design of the evaluation


interviews with all referral sources, for example and answers to
common questions about the evaluation.

Definitions of any key or unusual terms used in the questions.

Issues and topics you want to examine in the evaluation.

Expectations regarding interviewer behaviour.

l
l

Review of lists of respondents, addresses and telephone numbers


(as needed by type of survey).
Amount of effort that interviewers should spend contacting
respondents before giving up.
Review of interview script or interview questions.
Expectations about note taking, such as review immediately after
each interview and analysis of interview data.
Procedures for supervising the interviews and ensuring the quality of
data.

Deadline for completing all interviews.

Payment for time, transportation and other expenses.

Controlling Interviewer Bias


Interviews are vulnerable to interviewer bias that can distort evaluation
findings and lead to inappropriate recommendations. One way to control
interviewer bias is to ask a group of trainees to brainstorm potential
biases. The brainstorming process generates ideas without requiring the
trainees to own the bias personally. This creates a problem-solving climate
rather than a climate of blame and guilt. Discuss with trainees the potential
sources of interviewer bias, especially the tendency to:
l

Focus on comments that confirm the interviewers own views

See a pattern in the findings too early

Minimize comments inconsistent with comments from earlier interviews

Give greater weight to the opinions of experts.

Step 5. Recruit Respondents


To increase the response rate, it is good practice to call respondents in
advance or send them a letter (or e-mail) stressing the importance of the
interview, letting them know that they will be contacted to schedule a faceto-face or telephone interview and providing a general description of the
issues to be covered. If the situation warrants, guarantees of confidentiality
and anonymity may be appropriate at this time. If a respondent is difficult to
reach, plan for at least three follow-up calls at different times. Refer to
Chapter 2: Building Evaluation Frameworks.

97

Interviews

Case in Point
An evaluation of the Partners for Access and Identification (PAID) program collected data
from program participants at several different stages of time.To increase the
participation rate for stage two (an interview), respondents at stage one (a questionnaire)
were read the following script by the questionnaire administrator, and offered a small
incentive for their participation:
Also, we would like to do a follow-up interview with you.That interview will happen
approximately two months after you receive your identification. Will you be available two
months after you receive your identification? If yes, Would you like to participate? We are
paying people for participating: $10 with this questionnaire and $10 after the second
interview.

Step 6. Conduct the Interview


Although you may have a good script and prepare respondents well, you
must exercise control over the interview process, whether it is done face-toface or by telephone. The interviews will not be valid unless you gather the
data in a systematic way from all respondents. Here are some key
guidelines for conducting effective interviews.
Begin the interview with the scripted introduction. Briefly review the
purpose and importance of the evaluation and the time required for the
interview. Build rapport by reassuring the respondent about the
confidentiality of the information. Engage the respondent in a friendly
manner by asking them if theyre comfortable or if they have any questions
about the interview process.
If you would like to tape record the interview, ask for permission. Tape
recording will help you record correct information, refresh your memory and
supplement your notes. Always obtain permission. Not to do so is unethical
and may jeopardize the evaluation.
Actively listen to the respondent. Stay alert to the respondents patterns
of speaking, especially changes in tone. Occasionally summarize what the
respondent has said, particularly if the interview is by phone. For example, I
hear you saying that coming to the centre helped you to make friends. Is
that right? If the interview is in-person, make good eye contact and lean
slightly towards the respondent
Pacing is important do not rush the interview. Remember, the
respondent must do most of the talking. If not, usually it is because the
questions are not written well, or the respondent is anxious. In these cases,
rely more on open-ended questions (What is your opinion about...) to give
respondents the chance to voice their own views. Keep the interview
moving along in a pleasant way by letting the respondent know that you are
following a script. For example, you could say, I would like to hear more
about the food served for lunch if we have time later, but first there are
several other questions that we need to discuss. Conclude the interview
with an invitation to add any final comments.

Sequence questions. Begin with factual questions. Questions requiring


opinions and judgment should follow. Address important topics early rather
than later in the interview. In general, begin with the present and then move
to questions about the past or future.

A Toolkit for Outcome Measurement

98

Phrase questions carefully to obtain in-depth information. When


interviewing, the interviewer should state each question clearly and, for
closed-ended questions, give each response category completely this is
important because participants cannot read the possible answers. When
necessary, clarify unclear answers by repeating what you understood. For
example, Am I right that you feel your daughter learned to read in this
program? or When you say all of the time, how often is that?
Use probes. Encourage respondents to give details that help explain their
answers by using probes questions that search for specific information. For
example, a respondents comment, The food supplement program has really
been great, can be probed for more details such as, What changes have you
noticed because of the food supplement program? or How has it made a
difference for you? or Can you give me some specific examples of why you
say the food supplement program has really been great?
Remain neutral. Interviewers should be good listeners and avoid giving the
impression of having strong views on the subject under discussion. Remain
neutral by not agreeing or disagreeing with what the respondent says.
Remember that some respondents, to be polite, will say what they think the
interviewer wants to hear.
Encourage. Deal with refusals to answer questions and reluctant respondents
by restating the importance of the interview and probing gently. If the
respondent appears anxious or agitated, then move to the next question.
Minimize translation difficulties. Translation can affect the dynamics of the
interview and the accuracy of data. You can reduce these problems by briefing
translators on the purposes of the study to reduce misunderstandings,
reviewing interview questions in advance with the translators and by having
translators repeat the comments of the respondents as exactly as possible.
Respect time. One of the cardinal rules is to keep interviews as short as
possible. Refusal rates rise sharply for telephone interviews requiring more
than 10-15 minutes.

Step 7. Record Data


Writing down answers during the interview, even if you use a tape recorder, is
good practice. Take accurate notes. Record answers by using respondents
own words wherever possible. Avoid using abbreviations that you wont
remember when you are reviewing your notes later. Your notes provide a
useful way to locate sections of audiotapes that contain comments that you
may want to review or quote later. Some interviewers ask respondents to
review their notes for accuracy.
Some interviewers are worried that taking notes will interrupt the flow of
conversation and reduce rapport. They prefer to write their notes of the
session from memory right after the interview. You should be aware of the
potential for bias if you use this approach there is the human tendency to
remember those comments that agree with your own viewpoint.

6
99

Interviews

Step 8. Analysis of Interview Data


Depending on the types of questions posed in interviews, you may need to
use several different methods of organizing and analyzing the data.
For closed-ended questions, such as multiple choice and rating scales, you
can use counts to tell how many times an answer was given (frequencies)
and percentages to express the information as a proportion of all responses
given.
Open-ended questions produce narrative responses that require content
analysis, a process that is used to identify major themes, cluster responses
and summarize the data in a way that relates to the evaluation questions.
Good content analysis requires that you know your data well. It can be fairly
labour-intensive, depending on the amount of data collected (especially if
you transcribe verbatim from tape recordings).
Several suggestions are given below on how to approach content analysis.
Think Ahead. Immediately following each interview while memory is still
fresh, interviewers can prepare summary sheets. Interviewers should
identify major comments made by the respondent, the importance of those
comments and add their own ideas about themes discussed, main issues
and recommendations. These summary sheets will be useful later for coding
the data.
Get Ready. Read and re-read the responses given in each interview to
become familiar with the data. This begins the process of making sense of
the information. An early start on this process, even before all the
interviewing is completed, allows you to anticipate major patterns and key
findings. Continue to review the responses in multiple rounds as revisiting
the data often will reveal to you distinctions among the responses and
connections between themes, yielding a deeper understanding of the
material. Content analysis is a multi-stage process.
Group Responses. For each interview question, group answers of all
respondents together. This is much like cut and paste. You can do this with
hardcopies of data or electronic records. Assign each respondent an
identification number and make sure that each remark is associated with its
proper source.
Create Categories. The essence of content analysis is grouping responses
that reflect the same concept or theme together into categories. This
process can be approached in two ways using themes that emerge or
using predetermined themes. To identify emergent themes, look for
recurring topics in the data. This ensures all important ideas whether
anticipated or not are incorporated into the analysis. For predetermined
categories, you look through the data for responses that match your
preselected themes. Be aware that this latter approach may not draw into
the analysis themes that you did not foresee. As you begin to code the data, you
may identify additional themes that serve as sub-categories. Continue until you
have identified and labelled all relevant themes.

6
A Toolkit for Outcome Measurement

100

For every category or sub-category, be clear about what you include and what
you exclude. These rules will guide you and other coders in making judgments
about responses in a consistent manner. Develop a descriptive code to identify
each category. For example, improved reading skills may be coded as read.
Code Responses. The next step is to read and examine the responses and
assign the appropriate category code to each. On paper, do this by writing the
codes in the margins next to the responses or cut relevant sections from their
interview notes and paste them onto index cards according to the coding
scheme. If using excel spreadsheets or word processing, the code can be
marked in a column next to the responses. Move the responses together into
their categories. (Specialized software for content analysis is an option for very
large amounts of data).
Interpret Data. Once the data is reconfigured and condensed into categories and
sub-categories, you need to reflect on the importance and meaning of each, and
their relationships, if any, to each other. This is the process of finding meaning in
the data. A good place to start is by counting how many responses fall into each
category. This will suggest which categories are more important than others.
Consider questions such as: What seems most important, least important?
What seems to be unusual or extreme? What themes occur together?
What differences and similarities appear across respondents?
Data analysis is improved if you adopt a cautious approach and are watchful for
data that challenges assumptions about the program or preliminary findings. An
active search for any counterfactual evidence may identify issues that may have
been missed initially.

Key Informant Interviews


Key informant interviews are in-depth interviews of between 15 to 25
people selected for their first-hand knowledge about a specific program or
topic area. The interviews can be conducted either in-person or by
telephone. The key informants are often selected through a special form of
sampling, called a snowball sample (see Chapter 3: Sampling for a full
discussion of this technique).
Key informant interviews take advantage of the years of study, knowledge
and practice experience of experts to give thoughtful and incisive answers to
questions and solutions to problems. The primary danger in using a key
informant approach is that it relies on a small number of experts who may
share a similar set of views. As such, other valid points of view may be
ignored. You can control for bias in two ways by ensuring that the key
informants represent different sub-groups and by assessing their reliability in
terms of knowledge of the topics, credibility, objectivity and the willingness
to see the pros and cons of their own position. Place greater weight on the
information you receive from the informants you assess as being more
reliable.

6
101

Interviews

Agency in Action
VHA Home HealthCare offers an Extreme Cleaning Service for persons who need intensive
cleaning of their living places. As part of the evaluation of this service,VHA Home
HealthCare contacted all 28 referral sources to explain the evaluation and invite them to
participate in the study.The referral sources had the option of a brief telephone interview or
questionnaire sent by mail or fax.The vast majority of referral sources chose the telephone
interview.The interview took only 10 minutes to complete on average.
Interview questions related to outcomes, including how well the Extreme Cleaning Service
met the referral sources expectations and the expectations of their clients.There were also
questions about satisfaction with the reporting by the Extreme Cleaning Service Coordinating
Staff during and after the job, ability of Coordinating Staff to meet referral source information
needs, quality of the service, and views about what the referral sources liked best and least
about the Extreme Cleaning Service.

6
A Toolkit for Outcome Measurement

102

Examples
Conducting Effective Interviews

6A Community Helpline Statistics


Community Information Toronto

6B Community Helpline Survey Follow-up Interviews


Community Information Toronto

6C Key Interview Guide


Davenport-Perth Neighbourhood Centre

6D Evaluation of Early Intervention Program Key Informant Interview Guides


l
l

Early Intervention Specialists


Parents

Canadian National Institute for the Blind


6E Structured Protocol for Client Interviews: Early Intervention Evaluation
Jewish Family and Child Service of Greater Toronto

6F Interview Quetionnaire
Partners for Action and Identification (PAID)
Note:
the formats of the following outcome measurement tools have been altered to some degree
to correspond with the design and space restrictions of this toolkit. We hope you will draw
inspirations, confidence, and even concrete elements for your own use, drawing upon the
steps and guidelines set out in these pages.

6
103

Interviews

Example 6A
Community Information Toronto
Community Helpline Statistics
Counsellor:
Date:
To serve you better we would like to know more about the people who call our
service. Would you mind if I asked a few questions?

ABOUT THE CALLER OR PERSON SERVICE IS FOR


1. How did you find out about our service?
o phone book
o friend

o agency

o other ________________
8. If you are providing service in a language
other than English, please indicate which
language.

2. What number did you dial?


o 397-4636

7. What is the first language of the person


service is for?

o 392-0505

3. Are you calling for yourself, from an


agency or for a relative/friend, other?
o self

o agency

9. Are you a previous caller?

o relative/friend o other________________
4. Does client have children?
No

o Yes

5. What is age of group of client?


o child (1-12)
o adult

o youth ( 13-21)

o senior

o No

10. If yes, on your last call, did you get the


information you were looking for?
o Yes

o No

11. What are the first 3 letters of your postal


code?
o M________________________________

6. Gender?
o male

o Yes

o female

o transgendered

o L_________________________________
o Other (country)____________________

6
A Toolkit for Outcome Measurement

104

Example 6 A
Community Information Toronto
Community Helpline Statistics (continued)
DO YOU MIND IF I ASK WHAT YOUR SOURCE OF
INCOME IS?

. Was the call:

o information only
o emergency food

If employed, are you:


o full time
o part time
o temporary
.

o other

If on financial assistance, is it:


o OW
o ODSP
o OSAP (student)
o Pension

INFORMATION AND REFERRAL

o financial

o EI
o No Income

o other

o information and referral


o shelter/housing
o violence/abuse

. SERVICES OR ORGANIZATIONS REFERRED TO:

SUMMARY OF SERVICE PROVIDED


o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o

personal and family adjustment


childrens services
general community services
consumer/commercial
documents
education
employment/training
environment
financial
government
health
home services
housing
immigration/citizenship
community information centres
legal
recreation
transportation
volunteerism

INTERNET USE
16. To help caller did you use the Internet:
o Yes

o No

17. Did you make any additional calls on behalf of the client?:
o Yes

o No

Would you mind if someone from our agency called you back in a few weeks to ask about the
service you received If yes:
CLIENT NAME:

TEL:

BEST TIME TO CALL:

CALLER REFUSED TO PROVIDE INFORMATION OR CALLER NOT ASKED

6
105

Interviews

Example 6 B
Community Information Toronto
Community Helpline Survey Follow-up Interviews
Call-back Dates: (1) ___________________ (2) ____________________ (3) __________________

Hi! Im calling from Community Information Toronto. A few weeks ago you participated in a survey
we conducted and you agreed to have someone from our agency call you back to ask you about the
service you received.
Would you mind if I ask you a few questions today?

On a scale of 1 to 10 (where 1= low score, 10 = high score):


How professional was the counsellor you spoke with?

_________

How well did the counsellor listen to you?

_________

How clear was the counsellors understanding of your situation?

_________

How well did you understand the options presented to you?

_________

How clearly did the counsellor describe the services you needed?

_________

How comfortable were you in calling the numbers you were given?

_________

How many telephone numbers were you given?

_________

Circle one response for each question (yes or no):


Have you followed up on the referrals you were given?

If yes, were those organizations able to give you the help you needed?

Would you call our Community Helpline again?

When you called, were you encouraged to call back?

Would you recommend the service to family/friends?

6
A Toolkit for Outcome Measurement

106

Example 6 C
Davenport-Perth Neighbourhood Centre

Key Interview Guide


Person Interviewed:

Date of Interview:

Name of Interviewer:

Introduction: Introduce yourself and the purpose of your call.


Purpose of Call: Davenport-Perth Neighbourhood Centre (DPNC) is evaluating its Community Dining Program.
As part of the evaluation process, I am calling to ask your views about the future directions of the Program.

Brief Background: The Community Dining Program began in December, 1996 to provide hot nutritious meals to
the community, as well as information and referral to other community services.The Community Dining program
serves a wide variety of persons, including low income families with children, single men and women, homeless
persons and seniors. In other words, Community Dining participants are multigenerational.
Since Davenport-Perth is a Neighbourhood Centre, we have seen our role as doing more than providing meals alone.
By participating in Community Dining, we strive to:
l
l
l
l

Train volunteers to run the Community


Dining program
l Work in partnership with local agencies
and churches

Reduce isolation among participants


Link participants to other community services
Educate participants about social justice issues
Provide information

Question 1: What do you think the role of the DPNC Community Dining Program should be?
l Concentrate on meeting need for hot, nutritious meals?
l Reduce isolation?
Prompts:
l

How does community dining fit into the spectrum of action to prevent and reduce hunger?

Question 2: In your opinion, should we focus our services on a specific target group?
l What is the real need?
l Homeless persons?
l Low income families?
Prompts:

Isolated persons?

Question 3: What activities should we include in our Community Dining Program?


l Entertainment? l Recreation and education programs? l Rapid access to health and social services?
Prompts:
l

Info and referral to community services?


community development?

Job training for volunteers?

Education about

Question 4: Do you know of any very good Community Dining programs or models that you think DPNC should
contact?

Question 5: Do you have any other comments about Community Dining programs?
Conclusion: Thank the person for taking the time to speak with you and for their valuable opinions.

6
107

Interviews

Example 6 D
Canadian National Institute for the Blind
Evaluation of Early Intervention Program Key Informant Interview Guides
Early Intervention Specialists
Goss Gilroy Inc. has been commissioned by the Canadian National Institute for the Blind to conduct an
evaluation of the Early Intervention Program in Ontario. With this interview, we would like to obtain
your impressions on the relevance, effective implementation and success of the program. This
interview should last approximately 75 minutes.

Background Information
1. Could you please describe your role and responsibilities in delivering the Early Intervention Program?

Service Delivery
2. This program was designed to deliver services around four components: a) Direct Service;
b) Information/Resources; c) Parent/Peer Support; d) Advocacy.
Could you please describe the activities that you undertake under each of these four components?
3. What are the community service providers that collaborate in delivering this program? What are their
roles and responsibilities?
4. In your opinion, are the appropriate partners being involved in delivering this program? Please
explain.
5. What activities are undertaken to make this program known to parents, to other service providers
and to the larger community? Are these activities effective?
6. What are the mechanisms in place to ensure referrals of new cases to this program? Are these
mechanisms effective?
7. What are the procedures put in place to track and report the results of this program? How effective
are these procedures? (case management, filing of case information, data collected, reports)
8. In your opinion, do staff members receive sufficient training and support to deliver this program?
9. How satisfied are you with the quality of the services and information given by this program to blind
and visually impaired children and their parents?
10. Are you satisfied with the timeliness of referrals from specialists? Are you satisfied with the
timeliness of your response to new referrals? What improvements could be made to the timeliness
of this programs services?
11. What are the main challenges you encountered in delivering this program? What are possible
solutions to these challenges?

6
A Toolkit for Outcome Measurement

108

Example 6 D
Canadian National Institute for the Blind
Evaluation of Early Intervention Program Key
Informant Interview Guides Early Intervention
Specialists (continued)
Results
12. In your opinion, how successful is the program in transferring knowledge
to the community on how to work with blind and visually impaired
children?
13. In your opinion, how successful is the program in preparing blind and
visually impaired children for the public education system? What are the
key accomplishments?
14. To date, what positive impacts of this program have you observed for the
parents? For the children? Please specify.
15. Does this program have other impacts? Please specify.

Best Practices and Lessons Learned


16. What successful practices do you think should be repeated or continued
in delivering this program?
17. What unsuccessful practices should be avoided in the future?
18. Do you have any suggestions to improve this program?

Relevance
19. In your opinion, how relevant is this program to the needs of blind and
visually impaired children and their parents? Please explain.
20. What other services could parents use as an alternative to this program?
Does this program duplicate existing services?

6
109

Interviews

Example 6 D
Canadian National Institute for the Blind
Evaluation of Early Intervention Program Key Informant Interview Guides Parents
Goss Gilroy Inc. has been commissioned by the Canadian National Institute for the Blind to conduct an
evaluation of the Early Intervention Program in Ontario. This program is designed to provide specialized
services, information and resources to blind and visually impaired preschool children and their parents,
as well as create support groups for parents and teach them advocacy skills. With this interview, we
would like to obtain your impressions on the relevance, effective implementation and success of the
program. This interview should last approximately 60 minutes.

Service Delivery
1. Could you please tell us how and when you were first introduced to the CNIBs Early Intervention
Program?
2. Were you given sufficient information about the programs services?
3. Please describe the key services and information you received from the Early Intervention Program (probe
for: referrals; diagnosis and explanation; counseling; assessment of readiness for school; specialized services;
support groups, workshops; community resources; books, tapes, etc.; access to CNIB services and library;
public funding)
4. What services or information from this program were the most useful to you? Are there services or
information that were not useful? Please explain.
5. How responsive is this program to your needs and those of your child? Please explain.
6. How satisfied are you with the quality of the services and information provided by this program?
7. How satisfied are you with the timeliness of referrals and services provided by this program?

Results
8. What are the challenges you encountered as the parent of a blind or visually impaired child? In what way did
the program help you overcome these challenges?
9. To what extent do you feel better informed of existing services and resources as a result of this program?
10. To what extent did this program increase you capacity to advocate in favour of your child? Please explain.
11. To what extend did this program provide you access to the support of peers and other parents? Please
explain.

6
A Toolkit for Outcome Measurement

110

Example 6 D
Canadian National Institute for the Blind
Evaluation of Early Intervention Program Key Informant Interview Guides
Parents (continued)
12. In what way has the program facilitated your childs development and integration?
13. In what way has the program contributed to your childs future integration in the public education system?
14. In your opinion, how successful is the program in transferring knowledge to the community on how to
work with blind and visually impaired children?

15. Does this program have other impacts? Please specify.

Best Practices and Lessons Learned


16. Do you have any suggestions to improve this program?

Relevance
17. How relevant is this program to your needs and those of your child? Please explain.
18. What other services could you use as an alternative to this program? How does this program compare to other
services?

6
111

Interviews

Example 6 E
Jewish Family and Child Service of Greater Toronto
Structured Protocol for Client Interviews: Early Intervention Evaluation
STANDARD COMMITMENT TO CONFIDENTIALITY, INFO BEING RELAYED IN A NON-IDENTIFYING
WAY
1. How did you find out about JF&CS? What was the reason that you came to JF&CS? Whose idea was it?
How long have you been receiving service at JF&CS (since last opening)?

2. Tell me about which of our services or programs youve participated in. To which other community services
were you referred? Was the social worker helpful in getting these outside services?
Check all services provided since last opening:
c

Financial Assistance (monthly cheque)

JFCS therapy group

Special requisitions

Case management
School program contact

Just-a-Second

Advocacy

Referred to outside therapy/group

Camp

Individual counseling/therapy

Dental

Couple counseling/therapy
Family counseling/therapy

Pharmacy

Volunteer tutor

Monitoring parenting

Volunteer driver

Abuse/neglect investigation

Supportive/intermittent counseling

Woman Abuse services

PALS group for social skills

Other (specify): ________________________

JFLW/PUTP

Comments:

6
A Toolkit for Outcome Measurement

112

Example 6 E
Jewish Family and Child Service of Greater Toronto
Structured Protocol for Client Interviews: Early Intervention Evaluation (contd)
3. Comparing your situation now with how it was when you came for service at JF&CS _____years/months
ago, how would you rate it? (Include in your rating your whole situation, including how you feel about it/are
dealing or coping with it, etc..For instance, it might be better if you are coping better with the problem,
even if the problem has not gone away.)
Circle one:

1 Much better

2 Better

3 Same

4 Worse

5 Much worse

Comments:

4. What changes have you noticed in your life since coming to JF&CS? What do you notice is different about
the family now (compared with the beginning of service) that impacts on the child and that could be
reasonably attributed to Agency services provided (exclude things like windfall from Aunt Minnies will)?
Check all that apply:
c

1. Improved child social competency

7. Improved coping skills

2. Improvement in child well-being

8. Improved parenting

3. Adult completed further education

9. Improved partner relationship

4. Improvement in woman abuse situation

10. Stabilized housing

5. Improved emotional state-adult

11. Less social isolation

6. Increased financial independence

12. Other (specify) : _____________________

If worse, write in the relevant numbers here, e.g., if childs well-being has deteriorated,
write2" HERE:________________________________________________________________________
5. How has your coming to JF&CS been helpful in bringing about these changes? (Suggestion: Feeling
supported; feeling less lonely; learned parenting techniques or information about my childs development;
learned to communicate better with child or spouse)

6. What is the importance to you of the fact that this is a Jewish agency?

7.

For this question, we are asking you to speculate: Suppose that JF&CS service had not been available to
you____, when you first came to JF&CS. What do you think would have happened with you and your
child/children? (Suggestion: Nothing; child abuse; child taken into care; would have been still looking for
help; ...)

6
113

Interviews

Example 6 F
Partners for Action and Identification (PAID)
Interview Questionnaire
ID#

Date:

Hello! Thank you for joining me to complete the follow-up clinic evaluation questionnaire. We will use the
results of this questionnaire to see if having ones ID makes a significant impact on what services, activities, etc.
people are able to do and access in society. We also want to see if there are issues regarding keeping ones ID
safe. Your name will not go on the questionnaire and all information will be kept confidential. There are around
20 questions and should take around 10 minutes. Would you like to continue? If yes, Great! Also, you will be
paid $10 at the end of the questionnaire to compensate you for your time and information.
1. What ID did you receive since the first interview?
c OHIP
c SIN
c
c

BC

State of live birth

ROL

Delayed state of birth

c
c

Indian Status Card


Convention Refugee
Letter

Refugee Eligibility
Document

2. a) How much income do you receive now compared to the first interview?
c More
c Less
c Same
b) If you have income, what is it?
c F/T Job
c OW
c Panhandling
c Other(specify):______
c P/T Job
cODSP
c PNA
____________________
c Casual/Temp Work
c CPP/OAS
c Yes c No c N/A
3. Did receiving your ID help you gain income?
How/Why (or explain N/A)?___________________________________________________________
4. Did receiving your ID help you acquire employment? c Yes c No c N/A
How/Why (or explain N/A)?______________________________________________________________
5. Are you currently homeless?

Yes

No

6. Are you currently staying in a hostel or shelter?

Yes

No

7. If not homeless or in a hostel/shelter, where are you staying?


c
c

Private lodgings
Non-profit/Co-op

Friend/Relative

Rooming House

____________________

8. Are you living with an extended family?

Other (specify)______
c

Yes

9. If you are housed, do you consider your living situation stable?

No

Yes

No

10. Do you consider your current housing more or less suitable than when we last saw you?

Yes

No

N/A

11. Have you been homeless, in the hostel/shelter or in your current living situation for the entire time since we
c Yes c No
last saw you?
How/Why?_______________________________________________________________________________
12. Did receiving your ID help change your living situation? c Yes c No c N/A
How/Why (or explain N/A)?______________________________________________________________

13. Are you a single parent?

Yes

No

A Toolkit for Outcome Measurement

114

Example 6 F
Partners for Action and Identification (PAID)
Interview Questionnaire (continued)
14. What have you been able to do since receiving your ID?
c Job search/Take job
c See doctor/Use health services
c Get housing
c Apply for social assistance
c Use hostel/foodbank
c Apply for social services
c Open bank account
(specify) __________________
15. What have you still not been able to do since receiving your ID?
c Job search/Take job
c See doctor/Use health services
c Get housing
c Apply for social assistance
c Use hostel/foodbank
c Apply for social services
c Open bank account
(specify)___________________

Apply for training or education


program

Other (specify)_____________
___________________________

Apply for training or education


program

Other (specify)_____________
___________________________

16. Has receiving your ID helped you do anything you could not do before? c Yes c No
If yes, what? How did the ID help?_________________________________________________________
If not, why?____________________________________________________________________________
What were you hoping the ID would help you with?___________________________________________
c

17. Do you still have the ID you received two months ago?

Yes

No

Some

c Yes
c No
18. Did you have any problems picking up your ID?
If yes, what?___________________________________________________________________________
c

19. Have you been able to keep your ID safe?

Yes

No

c Yes c No
20. Have you had problems keeping your ID safe?
If yes, what?__________________________________________________________________________

21. What ID is missing?


c OHIP
c BC
c ROL

SIN

State of live birth

Delayed state of birth

22. If you do not have the ID, what happened to it?


c Lost by self
c Taken by police
c Stolen at a hostel/shelter
c Stolen
23. Where do/did you keep your ID?
c On myself
c Where

I live

c With
c

a relative

In a safe box

Indian Status Card

Convention Refugee
Letter
c

Refugee Eligibility
Document

Lost/destroyed at a hospital

Other (specify) _____________


___________________________

24) Do you have any comments about the effects receiving your ID has or has not had on your life?

6
115

Interviews

Chapter 7:
Developing Effective Focus Groups

What Are Focus Groups?


A focus group is an interview conducted with a small group of people rather
than with an individual. A moderator directs the group discussion using a set
of structured questions while another person, called a recorder, takes notes
of the session. On average, focus groups last about 90 minutes.
With careful planning, focus groups can be an inexpensive and rapid way of
obtaining outcome information directly from program participants. Probing
questions permit in-depth exploration of issues that can offer valuable
insights into how and why client outcomes were achieved. When managed
well, the group process can balance viewpoints of individual participants,
stimulating creativity and suggestions for program improvements.
Focus groups provide a means of gathering information about many aspects
of a program or service. You can ask questions about how satisfied
participants were with service delivery, how they benefited and changed
from the program, and what they would recommend to improve the
program overall.

Agency in Action
Delta Family Resource Centre in North York provides many different forms of caregiver and
child programs, multicultural drop-in programs, ESL classes, food supplement programs, and
job search workshops for new immigrants and refugees.The Centre uses focus groups to
measure client outcomes such as new skills acquired, community resources used, jobs
obtained and improved networking with others and the community. In addition to measuring
outcomes, focus groups give feedback about possible program improvements, new program,
and suggestions for strengthening the community. Delta Family Resource Centre also uses
focus groups to complement other evaluation tools, such as client satisfaction surveys and
informal personal feedback.

When Are Focus Groups Useful


for Measuring Outcomes?
l

When you want in-depth information about program experiences.


Focus groups enable you to ask program participants questions that help
you to understand why they chose to participate in your program, what
changes they experienced and how the program influenced that, what
outcomes are most important to them, and how well your program met
their expectations. In this way, focus groups can help you interpret and
validate the findings provided by other evaluation tools, such as program
records, direct observation or questionnaires.

A Toolkit for Outcome Measurement

116

When literacy and communication barriers exist. Although


questionnaires can be translated into different languages, this may not be
sufficient when serious literacy or communication barriers exist. Because
focus groups do not rely on written questions, they can overcome
language or cultural barriers. Some questions do not always translate well
into other languages. It is useful, therefore, to have present a skilled
moderator assistant moderator, or translator who understands the
language and culture of the group.

Case in Point
Davenport-Perth Neighbourhood Centre finds focus groups a practical way of obtaining
relevant, reliable and detailed information from participants, especially those with limited
English and low literacy. Focus groups can be held in different languages. If the focus group
moderator notices that someone does not understand the question, the moderator can
phrase the question another way. Participants working in focus groups stimulate each other
and generate excellent ideas about programming.
l

When you need a low-cost method to hear the views and


recommendations of diverse stakeholders, including program clients,
staff and service delivery partners. Focus groups can be useful when you
are planning program activities and need to understand clients and other
stakeholders attitudes, preferences or needs. They are also an important
tool when you are experiencing major service delivery problems and you
cannot explain why desired outcomes are not being produced. As such,
focus groups are useful for checking the feasibility of various options for
making program changes.

Main Strengths and Limitations of Focus Groups

Strengths
Relatively inexpensive
Fairly rapid
Identifies unanticipated issues
Helps explain quantitative findings

Limitations
Participants are not representative of the population
Requires skilled moderator and recorder
Group pressures may distort views
Only a limited number of topics can be discussed
Analysis can be time consuming

What Are the Main Strengths of Focus


Groups?
l

Focus groups are a rapid and cost-effective way of collecting


evaluation data. Focus groups are a relatively rapid data collection
option because data can be gathered in the space of several hours rather
than weeks or months. Costs can be kept to a minimum by making use
of experienced staff or volunteers, simple surroundings, reimbursement
of basic expenses and basic methods of analysis.
Focus groups are able to identify unintended outcomes and
unanticipated issues. Because focus group encourage an open sharing
of experiences and perceptions, they are more likely to reveal outcomes

117

Focus Groups

(positive and negative) that may not have been anticipated by the
program planners as well as other unforeseen issues.
Focus groups complement quantitative data. Focus groups provide
rich data that can balance, and help explain or interpret, the data
collected from program records, questionnaires or other quantitative
methods.

What Are the Limitations of Focus


Groups?
l

Participants may not be representative of the population. Focus


groups depend on a very deliberate selection of individuals who have
views about the topics that will be discussed and who can express
themselves in a group setting. Because of the deliberate (purposive)
selection of participants, the findings represent only the views of the
focus group participants.
Requires a skilled moderator and recorder. The focus group moderator
must have the interpersonal and group skills to guide the group in an indepth discussion of the chosen set of questions, manage group
interaction and complete the process within the time permitted. The
recorder must have the skills of observation and analysis to track both
the substance of the discussion and the group process. A moderator
without the right skills can limit the free expression of ideas and bias the
group process, impairing the validity and reliability of the data.
Group process can be undermined by a few individuals. If the
moderator is not able to control group interaction, a few individuals can
dominate or sabotage the group process. These individuals can exert
pressure on the other group members to accept their views or intimidate
members from voicing their own opinions.
Focus groups address only a few topics at a time. The focus group
process trades off the number and variety of questions being asked for
an in-depth discussion focused on only a few selected question areas.
Analysis may be time consuming. Focus groups generate a great
volume of detailed qualitative (narrative) data. To use the results, a
content analysis is required which organizes and meaningfully condenses
the data for the sake of manageability. This involves transcription of
recordings, identifying common themes and exceptions, coding
comments according to theme, and interpretation of the findings.
Sometimes this is completed by more than one analyst and their results
are compared.

7
A Toolkit for Outcome Measurement

118

Ten Steps for Developing Effective


Focus Groups
A good focus group appears to be a relaxed and informal discussion
between friends. This casual outward appearance conceals the strong
structure built by careful planning and preparation. When coupled with the
interpersonal skills of the moderator, the invisible structure gives the focus
group both its focus and easy flow. Taken together, they contribute to
the gathering of credible data.

TEN STEPS FOR DEVELOPING


EFECTIVE FOCUS GROUPS
Step 1. Specify major topics
Step 2. Design the moderator guide
Step 3. Choose a moderator and recorder
Step 4. Develop criteria for
selecting participants
Step 5. Recruit focus group participants
Step 6. Schedule the focus
group sessions
Step 7. Conduct the focus group sessions
Step 8. Record focus group data
Step 9.

Code and analyze results

Step 10. Report focus group results

Step 1. Specify Major Topics


Questions to be asked in a focus group should reflect the major evaluation
questions and indicators outlined in the evaluation framework. In the case
of outcome measurement, these questions would address possible
benefits for and changes in clients skills, knowledge, attitudes and values,
status, condition or behaviour. Did clients gain or improve as expected? In
which ways? Why or why not? How could the program be changed to
support outcomes better?
Of course, focus groups represent an effective method for other types of
evaluation and agencies often add in questions that are important for
quality assurance, client satisfaction and needs assessment. Bear in mind
that time is limited. When very specific information is needed, a focus
group may be as short as 40 minutes.

Step 2. Design the Moderators Guide


In the space of a brief 90 minutes or less, a moderator must elicit needed
answers from a group of strangers and at the same time maintain an

119

Focus Groups

open and friendly atmosphere. To accomplish these tasks smoothly and


efficiently, the moderator must be equipped with clear aims, a clear agenda
and a carefully developed series of questions.
A Moderators Guide is an outline for the moderator to follow. It lists the
ground rules for the session, provides introductions for each topic and lists
in sequential order the questions to be posed to the group as well as
possible probes to be used to elicit additional information. A good Guide will
allow sufficient time for exploration of unexpected issues that may arise
during the session.
If the evaluation design requires several focus groups, perhaps led by
different moderators, then the Moderators Guide ensures you are asking a
consistent set of questions with each group. Without the structure of a
Moderators Guide, each group could be asked different questions, thereby
reducing the credibility of findings.

OUTLINE FOR A FOCUSGROUP


MODERATORS GUIDE
Introduction (welcome, purpose,
introduce moderator)
Ground Rules (open discussion,
confidentiality, recording)
Warm-up (group introductions,
discuss warm-up questions)
General questions for the first topic
Specific questions for the first topic
Probes (open- and closed-ended probes)
Transition to next topic
General, specific, and probe questions
for the remaining topics
Wrap-up

Step 3. Choose a Moderator and a Recorder

A moderator should be comfortable speaking with groups, but also be a good


listener who can establish rapport with individuals. A moderator should
possess the skills to guide a group through a series of prepared questions
while controlling the focus, pace and tone of the discussion and facilitating
interaction and the free exchange of ideas. This requires knowing the subject
matter well enough to be able to lead a convincing discussion of it, and to
follow-up on promising leads. Something to bear in mind is that moderators
must be neutral careful not to inject their own points of view into the
discussion to avoid biasing group interactions and individual responses to
questions. For example, one type of potential bias could occur if a moderator
encourages participants with certain views to speak but discourages others
from presenting their points of view. To be a credible tool for evaluation and to
provide valid data, focus groups must be led by skilled and impartial
moderators.

A Toolkit for Outcome Measurement

120

Non-profits often use staff from other programs, staff from partner
agencies or volunteers as moderators. Although these types of
moderators are at arms-length from the program being evaluated, it is
important for them to have some familiarity with participants and
programs similar to those being evaluated. For example, moderators for
a session to evaluate a recreation program for seniors should have
some experience working with seniors in similar types of programs. This
experience helps the moderators to establish rapport with the
participants, understand special requirements and
limitations, and follow up on relevant issues that are raised during the
session.

Case in Point
Davenpor t-Per th Neighbourhood Centre conducts focus groups using staff and
volunteer facilitators who have good facilitation and analytic skills, who are sensitive to
par ticipants and who have an understanding of the program area. Staff who are
experienced in running focus groups also help by coaching staff who will be facilitating a
focus group for the first time.

If focus group participants speak a language other than English, then


the ideal moderator is one who is a native speaker with a good
understanding of the participants culture. This helps to put the group at
ease. In multi-cultural Toronto, non-profits use other strategies when
native speakers are not available to moderate a session. An effective yet
low-cost strategy is to have the moderator present a question to the
focus group and to have a volunteer native speaker sit with the group
and interpret the questions and take notes. In practice, this strategy
works well.

Step 4. Develop Criteria for Selecting Participants


For an effective focus group, selecting the right participants is as
important as selecting the right moderator. This step ensures that you
have participants who will be able to answer the questions, feel
comfortable speaking in a group and represent varying points of view.
Focus groups function better when there are no sharp divisions
between participants and they share some common features. The first
step, then, is to distinguish the types of participants that should be
represented in the focus groups. For the most part, this will be
determined by the information needs that you identified in Step 1.
Consider the range of factors that may be relevant to the outcome. In
the case of a respite care program for seniors, you may decide that you
need two focus groups to represent different views of the program: one
focus group for the seniors and a second group for the primary caregivers of the seniors. You may want to make focus groups similar in
other ways, such as having one group for new users of a program and
another for long-term participants because you feel that each group
might have a different experience of the programs services.

7
121

Focus Groups

Case in Point
The Canadian National Institute for the Blind (Ontario Division) provides several services to
blind or visually impaired children and their families through its Early Intervention Program.
Specialized rehabilitation services give children skills to enhance their access to lifetime
opportunities, while the provision of information and support helps parents better
understand and meet their childrens developmental needs. As part of an overall evaluation of
the Early Intervention Program, focus groups were used to gather information about program
implementation and program success.Two focus groups, one with service providers and one
with parents, were conducted in order to capture and represent their views separately.The
majority of focus group questions for the two groups were similar, such as questions about
benefits for clients and responsiveness/timeliness of service delivery. Only service providers,
however, were asked about specific implementation issues, such as whether the program
succeeded in developing effective collaborations with service delivery partners.

While focus group members should be similar to each other, they should not
include close friends or relatives. Participants are more likely to give their
honest views if they do not know other members. To prevent this, check
addresses and participant lists carefully during the selection process.
Avoid selecting experts or persons with other characteristics who may
create the perception of a hierarchy in the group. You should avoid mixing
managers and staff or clients and staff in the same group because some
staff or clients may not feel free to voice their true feelings. Depending on
the ethnic and religious composition of participants, it may be necessary to
have separate groups for males and females to ensure free interaction.

Step 5. Recruit Focus Group Participants


Each focus group should range in size from between 8 and 12 participants to
allow for a smooth flow of conversation and diversity of opinions. If the
participants are clients, you may need to invite twice as many to obtain the
desired number of participants.
Figuring out how many focus groups are needed requires balancing
information needs with cost and other practical matters. A rule of thumb is
to hold two focus groups for each factor considered relevant to the result
(i.e., length of time in program, age, income, etc.).
One of the best ways to identify suitable participants for each of your focus
groups is to consult program staff or volunteers who know the program
participants. To reduce the possibility of bias and the influence of personal
preferences, however, it is best to consult with several people. In this
technique, you select focus group participants deliberately for their
knowledge of the topic areas, their diversity of opinions, and their
willingness to share their opinions in a group setting.
Sometimes, though less commonly, focus group members are selected at
random from a list of individuals who have participated in your program or
who are on a waiting list. Depending on the type of focus group, you may
even recruit through advertisements in community newspapers, cable
television, notices posted in agencies and community centres, and flyers
distributed door-to-door.

7
A Toolkit for Outcome Measurement

122

Case in Point
Davenport-Perth Neighbourhood Centre (DPNC) finds it easier to recruit a sufficient
number of focus group members by asking program participants at the start of the
program whether they would like to take part in a focus group that is scheduled for a later
date. Because DPNC might get a pool of 30 willing participants using this approach, they then
select at random a smaller number of people for the focus group.To control for potential
bias in the focus groups, DPNC occasionally conducts individual telephone interviews with a
sample of program participants who were not part of the focus group and compares the
responses.

Focus group participants usually are recruited through personal invitation.


This gives you the chance to answer questions about the session or to listen
to potential problems, such as child care issues that you may be able to
resolve.
Incentives are not essential but they can help boost participation rates by
making it easier for people to attend a session. In addition to selecting
convenient times and locations for your focus groups, you may want to
consider covering all reasonable costs to attend the group (e.g.,
transportation and child-care); adding an attractive feature to the event (light
refreshments, a follow-up movie or presentation); and offering some
incentive (e.g., donated movie passes raffle prizes, and small amounts of
money). To ensure good attendance, remind participants in advance by mail
or telephone.

Case in Point
Delta Family Resource Centre uses the following strategies to recruit focus group
participants:
l Provide child care
l Supply transportation or TTC tickets
l Schedule the focus group at a suitable time usually late afternoon.

As with other methods of data collection, individuals may be concerned


about the anonymity and confidentiality of their comments, and as a result,
may be reticent to participate in the focus group. If it is your intention to
protect participants confidentiality, you must stress at the time you are
recruiting that their comments during the session will in no way be linked
with their names. For some evaluations, it may be important to have
participants names linked with direct quotes; if this is the case, individuals
must be advised of this and consent to it before they participate.

Step 6. Schedule the Focus Group Sessions


Plan on scheduling between one and half to two hours for each focus group.
To keep the discussion moving, many organizers prefer serving
refreshments before and after the focus group, rather than taking a break.
Others prefer scheduling the focus group for two hours and then taking a
20-30 minute break in the middle. Either way, dont schedule focus groups
one right after another, especially if you are using the same moderator and
recorder. Allow time after each focus group for the moderator and recorder
to discuss the session and write/review their notes.
The setting of a focus group is important. The room should be quiet and free
from interruptions. Check that it has adequate ventilation and a comfortable

123

Focus Groups

temperature. If possible, hold the focus group in an accessible and familiar


location because it puts participants at ease and promotes interaction. If you
must use an unfamiliar location, check it in advance to be sure it meets your
needs. Make a note of parking and transit stops and include this information
together with the address and room number in the information you give to
participants.
To keep the focus group atmosphere comfortable, advise the participants to
dress casually. The moderator should dress casually too. The exception to
this rule might be focus groups that involve members of the business
community or other groups where more formal attire is the norm.
You will need a table that seats the participants easily and comfortable
chairs. Participants should be able to see each other. The moderator usually
stands or sits at the head of the group. If the group has a comoderator/translator, then both moderators sit next to each other. The
recorder, who takes notes about the session, sits separately from the group.
The recorder needs a clear vantage point to monitor both the verbal and
non-verbal interaction.
Some non-profits use an audio-tape as a memory aid for the recorder. Be
sure that all participants know about the audio-taping and its purpose when
you are recruiting them. Then remind them again just before the session
starts. It is good practice to have participants sign a consent form that gives
written permission to record the session.
Focus Group Room Layout

REFRESHMENTS

MODERATOR
Dominant
participant

TABLE

RECORDER
Observes and
takes notes

Step 7. Conduct the Focus Group Sessions


Here are the specific activities required to conduct the focus group session.
Usually these activities are the responsibility of the moderator but
sometimes a staff member or assistant will help with the set-up functions.

7
A Toolkit for Outcome Measurement

124

Before the Session Begins Greet and welcome participants, indicate


where they can find refreshments and the necessary materials (e.g.
nametags, writing implements). Ask participants to complete any
confidentiality or consent forms, if applicable.
Introductory Remarks At start time, invite participants to sit and then
review the purpose of the focus group. Keep introductory remarks about
the topics of the session brief because too much information might
influence the group. Review housekeeping items such as location of
washrooms, the time the session will end, break times (if any),
refreshments and reminder to turn off cell phones and pagers. Review
confidentiality safeguards. If you choose to have a written agreement,
then the confidentiality form should spell out promise of confidentiality
and participants responsibility to keep the comments they hear in the
session confidential. If the session is recorded, then describe why you
want to record the session (to aid your memory and notes), how it will be
recorded and how personal identity will be protected. Describe what will
happen to the original recordings they are usually erased after the
analysis is complete.

Ground Rules for Focus Groups


Each member of the focus group has the right to be heard
Each members views are valuable
There are no right or wrong answers
Keep the discussion to the topic
Give your views honestly in a climate of trust
Focus group members will keep all comments confidential
l

Establish Ground Rules The ground rules, stated in the moderators


guide, clarify the moderators expectations for how the group will work
together. Participants should be told the discussion is informal, everyone
is expected to participate and different views are welcome. The following
ground rules are a good starting point and you should modify them to fit
your situation. It is good practice to write these down on a flip chart
before the session and, after reviewing them with the group, post them
on a wall as a reminder.
Guiding the Focus Group Process The moderator now begins guiding
the focus group interview process. It is called a funnel process
because it begins with general questions on each topic, then funnels
down to more specific questions, and finally funnels further by using
probing questions to uncover more details.

7.1 Focus Group Funnel Process


l. Warm-Up Question. Begin the session by asking a general question
from the moderators guide about a relevant topic that will be easy for
the group to discuss, such as, What first interested you in sending your
child to the March Break Camp program?

125

Focus Groups

2.General Questions. Proceed to the first general question in the


moderators guide. Ask mostly open-ended questions because they
encourage participants to tell their views in their own words and they
stimulate discussion. Avoid certain types of questions that dont work
well in focus groups. For example, yes or no questions do not
encourage discussions and why questions tend to generate socially
acceptable answers.
3.Specific Questions. The moderator obtains more detailed information by
asking additional, specific questions related to the general question.
4.Probing Questions. The moderator continues to obtain more details by
using probing questions. When participants give incomplete or irrelevant
answers, the facilitator can probe in several ways:
l

l
l

Repeating the question: Please listen to the question again. Are there
any other examples that you can add?
Asking for specific details: How did our program teach you that skill?
Repeating a participants answer: Keiko told us that the ESL program
gave her more confidence to speak with other mothers. How did the
program help someone else in the group?

Use special probing questions to signal the end of one focus group topic and
to close the discussion. For example, you may seek to confirm your
understanding of the groups feedback in this way:
Id like to summarize what I heard you say about the program activities that
we should continue, stop or start doing in our English as a Second Language
program next year. You would like to continue the small group discussions,
stop doing the crossword puzzles and start having some role-plays. Did I get
that right?
Once the group indicates agreement, you are ready to move on to the next
topic or to end the session.

Case in Point
Delta Family Resource Centre asks the following open-ended questions in focus groups
evaluating the outcomes of settlement programs:
General Question: Have you developed new skills after attending our programs?
Specific Question: Please explain what kinds of skills you have developed.
Probe: Please tell us how you apply that skill in your life.
General Question: How useful have you found the information provided in the programs?
Specific Question: In which ways is the information useful or not useful?
Probe: Please give an example of how the information was useful.

7.2 Managing Focus Group Dynamics


Because focus groups involve a fairly large number of strangers who are
deliberately selected for their diverse views, you may expect some problems
and difficulties with group dynamics from time to time. A skilled moderator
has the knowledge and experience to handle these problems without
disrupting the flow of the focus group. This section offers suggestions for

7
A Toolkit for Outcome Measurement

126

managing focus group dynamics when you encounter three common types
of participants:
l

Dominant participants

Shy participants

Talkative participants

An experienced moderator tries to identify a dominant or shy participant


before the session begins and will then arrange the seating so that the
dominant participant sits next to the moderator and the shy person opposite
the moderator. See the diagram for an illustration of recommended seating
arrangements.
Stating clear Ground Rules at the start of the session often can eliminate the
possibility of problem situations arising. The Ground Rules state that
everyone will get a turn to speak, which helps to control dominant and
talkative participants, and that everyones view is valuable, which helps
encourage shy participants.

Recommended Seating Arrangements for Focus Groups

REFRESHMENTS

MODERATOR
Dominant
participant

TABLE

RECORDER
Observes and
takes notes

Shy
participant
Dominant Participants. Usually a dominant participant wants attention and
recognition from the group. Sitting next to the moderator can satisfy this
need without disrupting the group process. If the dominant person becomes
disruptive, this seating allows you to intervene immediately. Ask that others
be given a turn or, if the situation persists, ask the disruptive person to take
notes about what others are saying.
Shy Participants. You should encourage quiet persons to talk by asking for
their opinions, giving them encouraging nods and sitting them opposite you
so you can use body language and eye contact to coax responses from
them. It is important not to push but to give gentle encouragements,
addressing easy questions to individuals who are reluctant to talk, inviting
them to participate (I believe that Pepe has something to say...), and by

127

Focus Groups

making a general invitation to speak (Is there anyone else who would like
to give their views before we go to the next topic?).
Talkative Participants. To manage talkative participants, establish ground
rules that everyone should participate and everyone deserves to be heard
equally. During a lengthy response, take advantage of any pause and say,
That is an interesting point, but perhaps we can discuss it in a separate
session. Does anyone else have a comment about this question? If the
talkative participant begins to give a rambling response or go off topic, then
intervene, politely summarize the point, then refocus the discussion.
Another strategy is to give the talkative participant nonverbal cues, such as
looking in another direction or stopping taking notes when an individual talks
for an extended period of time.

7.3 Closing the Focus Group Session


There are different ways of ending a focus group. Usually you will make
some closing remarks praising the value of the discussion and thanking the
participants for their time. If you have promised some incentive for
participation, such as movie tickets or cash, distribute them now.
Another approach is to summarize the key points of the session and then
check with the group:
l

Does this accurately represent what the group felt?

Did I miss something?

Is there something else that you wish to add?

Usually the response to your summary will let you know what the group
considers to be the most important aspects of the discussion and how
much the group supports your conclusions. Knowing this information will
help you when you are analyzing your data and writing your report and
recommendations.
It is good form to send personalized thank you letters to all participants.
Many times, participants are interested in the results of the focus group, so
you might include a one-page executive summary of the findings with your
letter.

Step 8. Record Focus Group Data


This step takes place as the session is being held. Accurate recording
provides the raw data that will be analyzed following the focus group. Focus
groups employ a wide range of recording techniques, ranging from paperand-pen to high tech digital recording. Most use a combination of techniques
to ensure all information will be captured should one method fail. Taking
written notes together with audio-taping as an aid to memory is the simplest
and least expensive combination.
In the lowest budget focus groups, the moderator is responsible for taking
down written notes and also handling the audio-taping of the session.

7
A Toolkit for Outcome Measurement

128

A better way is to have a person, called the recorder, take notes about what
happens during the session, help with the audio-taping, and carry primary
responsibility for the analysis of the data. If you have an evaluator on staff,
usually the evaluator will take the recorders role.
The recorder also observes the group process and takes notes about the
interaction. Notes should be extensive and reflect the content of the
discussion as well as nonverbal behaviour (facial expressions and hand
movements). In advance of the session, provide the recorder with a copy of
the questions from moderators guide with a blank space beneath each one
for writing notes during and after the session. Some recorders may prefer
using a laptop computer with the questions from the moderators guide
loaded in advance. Be aware that keyboards can make distracting noise, so
check the laptop beforehand to ensure that it has quiet typing.
The moderator also records observations at the end of the session. The
moderator and recorder supplement these notes with the audio tape or
video tape, especially for analyzing verbatim remarks. Shortly after each
focus group, the moderator and recorder should summarize the findings,
their impressions about the session and implications of the findings for
answering the evaluation questions.

Step 9. Code and Analyze Focus Group Data


Focus groups yield qualitative data which is different from quantitative data
in that it consists of words and observations, not numbers. Focus group
data requires content analysis, an approach that involves summarizing
information, searching for persistent themes and looking for new insights.
Simple Summary
A simple summary of the discussion may be sufficient when focus groups
are used to explore a new program or to obtain detailed suggestions about
an existing program from its participants. This form of simple analysis is also
appropriate when focus group results are obvious and when decisions must
be made rapidly.
A simple analysis takes only a few minutes after each session. The
moderator and recorder go over notes for each question and discuss their
observations. These observations and notes about group interaction and
nonverbal responses will also help you to draw conclusions.
Usually you do not need a formal transcript of the focus group for simple
analysis. You may want to play back some portions of the tape to verify
impressions or to draw upon some quotes that summarize a finding in the
participants own words.
Content Analysis
An in-depth analysis of focus group content will provide more detailed
feedback than a simple summary about what types of outcomes were
achieved, to what degree and how the program contributed. A content

7
129

Focus Groups

analysis of focus group data can be conducted on the notes and


observations recorded or, in situations where focus groups are taperecorded, on the complete verbatim transcripts of participants responses.
As discussed in Chapter 6: Conducting Effective Interviews, content analysis
is used to uncover themes and summarize data that is narrative (or
qualitative). The process of doing a content analysis, whether for interview
data or focus group data, is very similar. There are, however, a few special
considerations for analyzing data from focus groups because several
individuals are being interviewed at the same time. As noted in Step 7
above, focus groups gather together diverse individuals and group dynamics
may influence responses. It is important to keep the following points in mind
when analyzing focus group data.
l

Content. Weigh the meaning of words participants used. Can specific


words and phrases be used to categorize similar responses?
Context. Consider the circumstances in which a comment was made,
such as the subject of previous discussions, tone and intensity of the
comment.
Consistency of Responses. Think about whether shifts in opinions during
the discussion were caused by group pressure.
Accuracy of Responses. Decide which responses were based on
personal experience and give them greater weight than those based on
vague personal impressions.
The Big Picture. Pinpoint major ideas. Take time to step back and reflect
on major findings.

To guard against bias, two separate individuals can summarize and interpret
the focus group notes independently, and then they can compare their
findings. For large amounts of data, you may consider using computerassisted content analysis software.

7
A Toolkit for Outcome Measurement

130

Examples
Developing Effective Focus Groups

7A Questions for Focus Group Resource Programs


Delta Family Resource Centre

7B Evaluation of Early Intervention Program Focus Group Guide


l

Parents

Service Providers / Community Partners

Canadian National Institute for the Blind

7C Focus Group Discussion


Partners for Action and Identification (PAID)

7D Focus Group Moderators Guide


Authors Sample
Note:
the formats of the following outcome measurement tools have been altered to some degree
to correspond with the design and space restrictions of this toolkit. We hope you will draw
inspirations, confidence, and even concrete elements for your own use, drawing upon the
steps and guidelines set out in these pages.

7
131

Focus Groups

Example 7 A
Delta Family Resource Centre

Questions for Focus Group-Resource Programs


Introduction: Review Program Goals and Objectives
1. How did you find out about our programs?
2. What did you learn through attending this program? Are there any other types of information you would
like to learn?
3. Have you developed new skills after attending these programs? If so, please explain. What kind and how do
you apply them? If not, would you like to devlop new skills. Please explain the type of skills.
4. After coming to this program, are you aware of existing community resources? Please explain which ones
(elicit example if needed).
5. Since you have attended this program have you tried to access and use other community programs or
community activities? Please explain how this happened describe (examples can be used)
6. Has the program helped you to connect with others and with your community? If so, please tell us how this
has happened, if not, how else can we help you? Suggestions in how to make this happen?
7. What can we do to let the community know about our programs?
8. To be helpful to you and your families, we need to know what activities we should continue, stop or start
doing Exercise of traffic light and priorities
9. Other comments.

7
A Toolkit for Outcome Measurement

132

Example 7 B
Canadian National Institute for the Blind

Evaluation of Early Intervention Program Focus Group Guide - Parents


Introduction
l

Who I am
Reason for the focus group
l Reason for evaluation
l Evaluation and methodology
l Why and how they were chosen for focus group
l

Breaks, washrooms, etc.


Confidentiality
l Invitation to speak freely, in both official languages
l Questions
l

1. Lets start by finding out a little about you. Id like to ask each of you to introduce yourselves and tell us
how many children you have and how old they are. (5 min.)
2. Could you please tell us how and when you were first introduced to the CNIBs Early Intervention Program?
(5 min.)
3. Could you please tell us about the challenges you encountered as parents of a blind or visually impaired
child? (15 min.)
4. How has the program helped you overcome these challenges? (Probe for: better informed of existing
resources; increased capacity to advocate, access to peer support; access to specialized services.) (15 min.)
5. Of all the services and information you received from the program, which ones did you find the most useful
and why? (10 min.)
6. How has the program contributed to the development and integration of your child? Please provide specific
examples. (15 min.)
7. I would now like to ask you about your satisfaction with the way the program is administered. (Probe for:
communications, responsiveness, timeliness.) (10 min.)
8. How could the program improve its services to best meet your needs? (10 min.)
9. Summary and wrap up by facilitator. ( 5 min.)
Total length: 1 hour 30 minutes

7
133

Focus Groups

Example 7 B
Canadian National Institute for the Blind

Evaluation of Early Intervention Program Focus Group Guide - Service Providers


and Community Partners
Introduction
l

Who I am
l Reason for the focus group
l Reason for evaluation
l Evaluation and methodology
l Why and how they were chosen for focus group

Breaks, washrooms, etc.


l Confidentiality
l Invitation to speak freely, in both official languages
l Questions

1. Lets start by finding out a little about you. Id like to ask each of you to introduce yourselves and tell us the
name and type or organization you work for. (5 min.)
2. Could you please describe your role and responsibilities related to the Early Intervention Program?
(10 min.)
3. In your opinion, to what extent has the program succeeded in developing effective collaborations with
service delivery and community partners? Are the appropriate partners being involved in delivering this
program? (10 min.)
4. What is your opinion on the delivery and administration of this program? (Probe for: communications,
responsiveness, timeliness, expertise of CNIB staff.)
5. What are the main challenges you encountered in your participation in this program? What are possible
solutions to these challenges? (15 min.)
6. To date, what positive impacts of this program have you observed for the parents? For the children?
(15 min.)
7.What have been the specific impacts of this program on your practices? Please provide specific examples.
(15 min.)
8. Do you have suggestions on how to improve this program? (10 min.)
9. Summary and wrap up by facilitator. ( 10 min.)
Total length: 1 hour 45 minutes

7
A Toolkit for Outcome Measurement

134

Example 7 C
Partners for Action and Identification (PAID)
Focus Group Discussion
Focus Groups are being held to gather information regarding the personal experiences of clientele surrounding
issues of identification. In particular, feedback from the focus groups will illustrate the difficulties encountered
without proper identification. The following clinics will be visited to collect data:
l

Second Community Centre


l Second Base Community Centre
l Newtonbrook Community Centre
l The Stop Food Bank

l
l
l
l

WoodGreen Community Centre


Innercity
Start Resource Centre
Glen Rhodes Food Bank

1. Can you tell us how you heard of the identification clinics?


2. Could you tell us how not having identification has had an effect on your life?
3. What problems have you had because you dont have ID?
4. What can change in your life once you get your entire ID?
5. Besides getting your ID, has PAID helped with anything else? (e.g.,Toronto social services/welfare, housing,
food bank info etc.)
6. Have you learned about any additional programs/services by coming to the clinic,
seeing ______________________?
7. What other information could the PAID project provide that would be useful to you?
8. Comments regarding the importance of the PAID project.

7
135

Focus Groups

Example 7 D
Authors Sample
Sample Focus Group Moderators Guide
Introduction
We are here today to discuss the Community Leaders Project.The Project has been operating for a year. Now
we need to hear from you, the volunteer facilitators, who bring with you your wisdom and experience with
the Project.We need your views and honest opinions about what difference the Project has made for you.

Ground Rules
l Todays

session will last about 90 minutes.We will


finish on time at 4:30.
l I have asked (name of person) to take notes during
the session to help me remember what was said. However,
in my report, I will not identify anyone by name.What you
say in here will be kept confidential and no one will be able
to connect what anyone says with who said it.
l I'd like to hear from everyone, but you dont have to
answer every question.
l Please discuss each question I ask among yourselves.
There is no need to address your comments to me.

l This

is an open discussion. Please state your opinions


and views honestly.Try not to be swayed by the
group.There are no right or wrong answers.
l Please avoid side conversations with your neighbour.
l Please take a minute now to check that your cell
phones or pagers are turned off.
l Before we begin, please take a few minutes to help
yourself to coffee, tea or a soft drink and a snack. If
you need to use the rest rooms, they are located in
the corridor to your right.

Warm-Up QuestionsIn your own words, discuss what the Project means to you.
General Question #1 In an overall sense, how satisfied are you with the outcomes of the Project?
Specific Questions #1 After the group addresses the general questions, then the moderator asks specific
questions:
In what way has the Project increased your access to social and health services?
Please give examples of how the Project has increased your contact with women of your own culture and
other cultures.
Has the Project been able to give you more social and emotional supports?
General Question #2
What do you think are the skills needed to be an effective community leader?
Specific Questions #2
What are some concrete examples of personal skills you have developed as a result of the Project?
What are some concrete examples of professional skills you have developed as a result of the Project?
Please give examples of how you have been able to initiate group activities outside the group.

Ending the Focus Group


l

Thank everyone for their contributions to the focus group.


State that you found the comments very interesting and helpful.
l Distribute the small gift to show our gratitude for participating in the focus group.
l

7
A Toolkit for Outcome Measurement

136

Chapter 8:
Making the Most of Direct Observation

What Is Direct Observation?


Direct observation a method of collecting data by watching and listening
can help you acquire a deeper understanding of your program and its
participants than you may get by using questionnaires, interviews or focus
groups. What makes observation distinct from other methods of data
collection is that information can be collected without depending on a
persons willingness or ability to answer questions.
The majority of people who work in social service agencies, whether they
are staff, volunteers or managers, report that they rely on observations in
carrying out their day-to-day work for example, when interacting with
clients, making decisions about improving services or responding to client
needs. With respect to evaluation, however, direct observation remains
underutilized as a method for collecting information. Observations and
recording of information are often two ingredients of the evaluations
undertaken by non-profits, but they are usually done informally without much
thought as to the consistency and quality of data collection. If direct
observation is applied with carefully designed data collection tools (e.g.,
rating scales, pre and post checklists), it can be a very accessible approach
that produces valid and credible information for evaluation.
Direct observation is a very flexible approach that can be used to assess a
variety of client outcomes such as: participants skills, behaviour and
knowledge; interactions between groups or individuals; nonverbal behaviours
(e.g., facial expressions, gestures), and physical surroundings (e.g., weight of
newborns). In fact, with certain types of program participants for example,
toddlers or clients experiencing cognitive impairment the only practical
method for obtaining outcomes information may be through direct
observation.

Agency in Action
Abrigo Centre offers a wide range of programs to low-income families in Toronto who are
vulnerable or in distress and who experience cultural and linguistic barriers to service.To
measure program outcomes, Abrigo created a process to document staff observations of
client outcomes. Collectively, program staff developed outcome statements that capture the
changes they see happening in their clients as they work with them changes such as
understand dynamics of abuse, improved stress management skills, safety plan in place,
improved parenting skills and improved self-esteem. To ensure that all caseworkers
understand the outcomes in a similar way, they identified specific indicators for each. Staff
record what outcomes have been achieved and observed in the client in the computerized
data management system at the conclusion of service. At regular intervals, Abrigo calculates
statistics on their client outcomes for example, how many women achieved a specific
outcome, what percentage of women achieved a specific outcome and what types of
outcomes were achieved.

137 Direct Observation

When Is Direct Observation


Useful for Measuring Outcomes?
l

When there is physical evidence or outcomes can be readily seen or


heard. Sometimes outcomes can be readily seen or heard. For instance,
you can observe the snacks parents choose to bring to drop-in programs
for their pre-schoolers. You can observe whether living conditions in a
seniors home are safe and clean. You can watch youth in after-school
programs complete homework assignments.

Case in Point
The United Way of Greater Toronto wanted to monitor the effectiveness of their training in
developing program logic models and used program records to demonstrate acquisition of
new knowledge and skill by workshop participants. As part of day-long sessions in outcome
measurement, small groups of participants created program logic models on flip charts and
reported back their work to the entire group.These flip charts were retained by workshop
organizers, transcribed and then counted and analyzed with regard to completeness in order
to demonstrate active learning and application of new skills.
l

When other data collection methods (i.e., surveys, interviews) seem


inappropriate. In some situations, program participants may not be able
to provide data accurately because they are hesitant or are unable to
provide information due to existing barriers (e.g., language, cultural).
When the program is not achieving its planned client outcomes and
implementation problems are suspected. Unexpectedly poor
outcomes may be caused by implementation problems. The program may
be attracting unintended clients, the program theory may fail in practice,
staff may not be properly trained or supervised, or the program may fail
to engage clients long enough to produce anticipated outcomes. Direct
observation is one of the best ways to determine quickly whether the
necessary components and resources are in place for proper program
implementation.

Main Strengths and Limitations of Direct Observation

Strengths

Limitations

Permits study of program activities


in the actual locations

Requires skilled, unbiased, and trained


observers to make reliable observations

Allows study over a period of time

Observation can be intrusive and disruptive


in some situations

Facilitates comparisons between program sites


Experienced observers can bring their
insights to the analysis of the data

8
A Toolkit for Outcome Measurement

138

What Are the Strengths of Direct


Observation?
l

Allows you to study program activities and outcomes within the


program setting. This affords you a rich understanding of program
operations and the relationship between program activities and
outcomes. How changes and benefits develop may be revealed if you
observe the staff, participants and service delivery processes in action.
Supports evaluation over a period of time. Direct observation is
usually sustained over a period of time, giving you a view of the program
that is more like a moving picture than a snapshot one that captures
program dynamics.
Facilitates comparisons across program sites. Direct observation
provides a systematic way of assessing program implementation across
program sites. This is useful for testing how well a program design works
in different locations, ensuring high standards across all sites, and for
continuous quality improvement.
Encourages observers to bring their insights to the analysis of the
data. Skilled observers can bring insights from experience in other
settings to produce penetrating and compelling interpretations of the
data. This feature of direct observation often makes it the method of
choice for high-stakes evaluations in diverse fields, such as reviews of
funding applications, ombudsman investigations, program audits and
accreditation reviews.

What Are the Limitations of Direct


Observation?
l

Direct observation is susceptible to observer bias. The observer is at


the centre of the measurement process. If the observer is biased, then it
compromises the validity and accuracy of the data. Bias can be offset
through careful selection of skilled observers, training and systematic
ways of recording and analyzing data.
Observation can alter the behaviour being studied. Most of us are
aware that people behave differently when they know that they are being
observed or evaluated. Direct observation depends on the observer being
unobtrusive to the point that program staff and participants behave as if
the observer was not there.

Seven Steps For Making the Most of


Direct Observation
The primary issue for direct observation is quality of the data gathered. By
preparing and employing a systematic process for conducting, coding and
analyzing your observations, you will not only be better able to maintain the
quality of data but also save time.

139 Direct Observation

SEVEN STEPS FOR MAKING THE MOST


OF DIRECT OBSERVATIONS

Step 1.

Decide what will be observed

Step 2.

Decide on type of observations

Step 3.

Create observation recording forms

Step 4.

Determine where and when to observe

Step 5.

Select and Train the Observers

Step 6.

Conduct Field Observations and Complete Forms

Step 7.

Analyze the Data

Step 1. Decide What Will Be Observed


With limited resources, it is impossible for non-profits to observe all program
components all of the time. As such, direct observation can be applied
purposefully by observing only a small sample of program activities over a
limited period of time. These observations must be planned carefully to
avoid observng a set of activities that are not representative or that do not
answer the major evaluation questions being asked.
Based on the major evaluation questions, narrow the focus of observations
to a limited number of indicators likely to generate the most useful
information and insights.

Case in Point
To assess the developmental progress (outcomes) of children, preschool programs can apply
direct observation by both trained program staff and parents (or caregivers) using a
standardized data collection instrument The High/Scope Child Observation Record (COR)
for Infants and Toddlers.The COR can help staff monitor each childs developmental progress,
pinpoint the strengths and weaknesses of their educational program, and communicate
effectively with parents. High/Scopes assessment materials focus on childrens strengths - on a
broad range of cognitive, social-emotional and physical abilities. Information for completing
the COR is gathered by those who know the children best their caregivers and parents.
Adults record COR observations as they care for, play with and attend to children.
Observations of child behaviour in the following six categories of development are recorded:
l
l
l
l
l
l

Initiative (expressing choices, engaging in complex play)


Social relations (making friends)
Creative representation (making, pretending)
Music and movement (exhibiting body coordination, initiating movements to a song)
Language and literacy (showing interest in reading, beginning reading, beginning writing)
Logic and mathematics (sorting, counting objects, describing time sequences)

Ref: www.highscope.org

8
A Toolkit for Outcome Measurement

140

Step 2. Decide on Type of Observations


The next step is deciding on the type of observations you want for your
evaluation. Direct observations can range from formal and structured to
informal and loosely structured.
Structured Observations
At the formal end of the spectrum, data are gathered by following highly
structured protocols and data collection instruments. Observers are looking
for certain things. Recording sheets may provide a place to indicate whether
the identified behaviour or activity was observed, when and how often, or
observers may be asked to rate what was observed on some predetermined
scale. Structured observations provide for a numerical summary, including
frequency counts, rankings and ratings.
Unstructured Observations
In an informal, unstructured approach to direct observation, an observer looks
at something, recording whatever they see or hear that relates to the
evaluation questions. The observer may follow guidelines but is not overly
confined by a pre-set list of things to observe and record. This style allows
observers to record unexpected occurrences and produces qualitative
information.

Case in Point
Davenport-Perth Neighbourhood Centre, a multi-service centre in west Toronto, wanted to
evaluate its Community Dining Program to assess whether the program helped to combat the
effects of hunger and loneliness, build a caring community and restore hope. As well, they
wanted to review the program climate and the process of service delivery. An observer
external to the program visited the community dining program in operation to observe and
record the interaction between staff and program participants, the interaction among
participants, the physical setting and the meal preparation and distribution processes.The
program coordinator told individuals attending the dining program at the time of the
observations that a visitor was present to see how the program operated.

Step 3. Create Observation Recording Forms


To be useful in evaluation, observations must be recorded. Depending on the
situation, sometimes observers record observations on the spot while at
other times observations are recorded later. The date, time and location of the
observation and who recorded them should be part of every observation
record.
There are numerous ways of measuring both formal and informal observations
including observation record forms (checklists, tally sheets, rating scales),
observation guides, field notes and photographs to name a few. Your
selection will depend on the evaluation questions being asked, how structured
the observations are and the nature of the program you are evaluating.
In general, observation record forms help reduce errors in the observation
process and ensure that all important items are being observed. They also
facilitate the organization of data gathered from multiple sites or by several
different observers.

141 Direct Observation

Inter-rater reliability is an expression used to indicate how much agreement exists


in observation records between different observers. Strong inter-rater agreement
on observations is desirable as it indicates that they have seen and heard the
same things and interpreted them in the same way. If different observers
produce different records of the same events, look to clarify in specific terms
what is to be observed and how to record them. Make sure your observers have
enough training.

3.1 Structured Observation Forms


Structured observation record forms should list the items you want observed and
provide space to report observations:
l

Checklists track whether specific activities take place. If an activity or


behaviour is witnessed, a mark is placed on the sheet to record it. This is the
simplest type of observation form.

Case in Point
The Canadian National Institute for the Blind (Toronto District) uses a checklist in the Orientation
and Mobility program. As clients master new skills that allow for independent and safe travel, the
instructors record this on a checklist that is specific to the type of cane used. Skills include (among
others):
l Indoor

mobility
skills
l Street crossing
l Commercial travel and TTC
l Routes learned.
l Cane

Tally sheets record how often the activities occur. Every time an activity is
witnessed, the observer places a mark on the sheet to record it. Tally sheets
give frequency counts.

Case in Point
A drop-in day shelter offers shower and laundry facilities to homeless men in Toronto. A tally sheet
allows staff to record the number of showers taken and loads of laundry done to indicate the
benefits of personal care and hygiene gained by clients of the community centre.
l

Rating scales measure how well something is being done. These forms
look similar to survey questionnaires but observers record what they see and
hear instead of respondents' answers. For example, if you wanted to measure
how well a community worker facilitated a meeting, you could use a simple
scale to score performance of key activities such as setting ground rules,
facilitating discussion, ensuring everyone has a chance to speak, maintaining
focus on priorities and staying on schedule.

Case in Point
St. Christopher House, a multi-service organization for all ages in west central Toronto, uses a rating
scale in monitoring the quality and outcomes of its home support program for seniors living in the
community. On a visit to the seniors home after six months of service, a staff member records
observations of the condition of the home and personal care given to the client on a 3-point
scale: S= satisfactory, NI= needs improvement, NS= not satisfactory.

8
A Toolkit for Outcome Measurement

142

There are two aspects of rating scales for direct observation worth
mentioning at this time. Each level of a rating, such as the rating Needs
Improvement, must be defined clearly and with enough detail that a trained
observer can make an accurate rating. Next, to ensure consistency and
accuracy, rating scales for direct observation usually are kept to three or four
levels, such as, little or no difficulty, some difficulty, serious difficulty.
You will find more information about rating scales in the chapter Developing
Effective Questionnaires.

3.2 Unstructured Observations


There are two approaches to recording informal, unstructured observations:
l

Observation Guides are printed forms that provide space to record


observations. Some instructions to observers are included. For instance,
observers may be asked: Write down actions you see and comments
you hear under the following headings: physical contact between parent
and child; parental praise; disciplinary action by parent.
Field notes are the least structured observation record. Observers record
in a narrative, descriptive style whatever they see or hear, often using a
notebook.When preparing direct observation forms, consider the
following guidelines:

Case in Point
Family Day Care Services, a multi-service organization providing licensed home child care and
centre-based family support programs across Toronto, requires that staff complete a report
for each home visit.The report identifies which staff conducted the visit, the date and time of
the visit, whether it was scheduled in advance or not, and includes a narrative account of
observations and discussions during the visit. Space is also included to record any follow-up
taken, if warranted. Staff and the child care provider both sign the report.

1. Identify in advance the possible response categories for each item on a


structured observation form. Listing the response categories helps to
reduce observer variability and improve the quality of data by simplifying
the recording process.
2. Limit the number of activities to be observed. Try to limit the observation
form to one or two pages. If necessary, it is better to use several shorter
forms that a single long one that runs several pages.
3. Allow adequate space to record observations that you were not able to
identify in advance. If your observation form covers several topics, be sure
to allow space for additional observations after each topic.
4. Test the observation forms by visiting an actual program site and making a
practice set of recordings, using more than one observer if possible.
Once this is finished, you should discuss any problems with the
observation process and differences in ratings between observers. You
should make every effort to identify and correct any bias or errors in
ratings.

8
143 Direct Observation

3.3 Standardized Assesment Tools


In some situations, community service providers may be able to access and
utilize standardized assessment tools when observing changes and benefits
experienced by program participants. These instruments are carefully
constructed and tested to ensure high levels of validity and reliability and
must be administered by trained observers.

Agency in Action
There is a growing trend towards the use of technical equipment, such as computers,
videotaping and photography, to record observations. Here is an example of an evaluation
that blends high-tech and more traditional approaches to direct observation.
Community Information Toronto (CIT) provides Toronto area residents with information and
referral to human services through three public access points: telephone, web and print. In
June 2002, it launched its 211 services, enabling access to the phone centre through a highly
visible and easy to remember telephone number. CIT used direct observation to evaluate the
usability of the updated 211 web site. CIT set up an on-site testing lab equipped with a
computer to observe users interacting with the site. Each user was given navigation tasks to
perform. Users were asked to voice their thoughts as they attempted to complete tasks.The
time it took to complete tasks and the users comments were noted. Direct observation was
followed by an in-depth interview that asked users to share their overall impressions, make
suggestions and give feedback about new ideas for the site.They also were given a
questionnaire to rate their experience.The entire evaluation process took about 90 minutes
to complete.

Step 4. Determine Where and When To Observe


The next step is to decide on where you will need to record observations
and the timing of your observations: both are critical to reducing the chances
of collecting skewed or distorted data.

Timing
People and programs follow routines associated with different times. Involve
program staff and client representatives as part of your evaluation team and
examine the best times for making outcome observations. Often, agencies
select what is convenient. This is understandable and may be unavoidable
but make sure that when you analyze you acknowledge that the information
does not represent fully the whole experience in the program.
For example, consider an outcome measurement exercise in a parenting
program for young children. Scheduling a one-week period for observation
of activities and childrens progress during the week before summer break
may be appealing in terms of timing but this week is often not
representative of the program field trips, special events, parties and oneon-one sessions with parents in anticipation of the long vacation are
common additions or substitutions. Observations of the program during a
week of typical activities would yield a truer picture of the program and the
progress of the children.

Location

If a program runs at many different sites, select sites based on the


evaluation questions and the nature of the observations. As a general rule,

A Toolkit for Outcome Measurement

144

make observations at several locations to obtain a good picture of the


situation. Locations that appear to be either typical (or unique) often prove to
be otherwise on closer inspection. For example, if you are evaluating an after
school mentoring program that takes place in seven schools, you can select
locations based on program statistics or you may decide to select three
program sites based on the criterion of outcomes achieved: one program
that is producing above-average outcomes, one program with average
outcomes, and one that has below-average outcomes. To help you make the
selection and reduce selection bias, consult with more than one person - the
program co-ordinator, key volunteers and other staff members who are
familiar with the program at the different sites.
In general, use observations at a single location very cautiously if the
program operates in multiple locations. There are times, however, when it
may be sufficient to observe only one of the program locations. If a single
location is unique from the others for example, if only one site has high
dropout rates, you may observe it (and not the others) to discover the
source of the problem.

Step 5. Select and Train the Observers


Virtually anyone can be trained to be a good observer. To reduce costs, nonprofits can train staff members or volunteers, preferably from programs
other than the one being assessed, and colleagues from other organizations
who have experience with similar programs may also be trained. In terms of
skills, the best observers have an eye for detail, are careful in their work and
are able to sustain attention over a period of time.
If you have a small number of observations to make, then it is best to select
and train one observer who is not involved with the program being evaluated
to ensure consistency of ratings.
For a large number of observations, the evaluation will have more credibility
if you use two or more observers. To reduce possible bias, if resources
permit, then at least one of these observers should not be directly involved
with the program you are evaluating.
Once you have recruited your observers, training is essential to ensure the
accuracy, consistency and credibility of the ratings. All observers need to
see and hear and record as similarly as possible. Observers should receive a
description of the evaluation, copies of the observation forms, a careful
review of the definitions for each observation, and a discussion of recording
procedures and rating scales, if any. It is very important that you give
observers examples of the full range of observations they will make along
with each of their associated scale ratings. You should also give the
observers guidelines for writing summary notes about their observation
visits, including any problems they encounter or unusual events that affect
their observations.
The next step is to visit an actual program site to make a practice set of
observations. Once this is finished, you should discuss any problems about
the observation process and differences between observers. You should
make every effort to identify and correct any bias or errors.

145 Direct Observation

Step 6. Conduct Field Observations and Complete Forms


Observations may be done openly, where participants are made aware of
the observer, or secretly, where participants are unaware. It depends on the
situation and purpose of the evaluation as to whether people need to be
informed but in all cases the ethics of the evaluation must be considered.
The presence of outside observers, especially if known to be part of an
evaluation exercise, may cause suspicion and result in altered behaviour.
Explaining in advance that the purpose of the observation is to improve the
overall program and not to report an individuals performance can help
put clients at ease. Even when there is no suspicion about the purpose of
the evaluation, you will find that people tend to behave differently in the
presence of observers. Allowing sufficient time to make your observations is
important for collecting accurate data; over time, people become less selfconscious and behave more naturally. The golden rule is: The longer you
observe, the more you see.
The observer should complete the observation forms and make notes
without disturbing the program staff and participants. The best time for
recording is during the observation itself. This is not always possible because
having someone write or type may make some people self-conscious or
disrupt the program. For example, in a drop-in centre it might be disturbing
to have an unfamiliar person sit in a corner watching the others and taking
notes. Because people come and go in a drop-in centre, it is not possible to
let everyone know in advance about the evaluation process. In these cases,
recording should take place as soon as possible after observation.
Because direct observation is vulnerable to observer biases and inaccurate
ratings, quality assurance procedures are an essential aspect of the method.
These procedures include the following:
l
l

Check a small sample of observation forms regularly for accuracy


Double check very high or very low ratings, because these can have the
most impact on evaluation findings
Review the completed observation forms for errors, missing data and
consistency
If any problems are identified, they should be discussed with the
observer right away and the observer should receive additional training.

Step 7. Analyze the Data


Most data from structured observation forms (e.g., checklists, tally sheets,
rating scales) can be analyzed using basic statistical procedures, such as
frequency counts and percentages. For example, analysis can show the
average number of clients who attended a drop-in program or the
percentage of clients who achieved expected outcomes. Smaller programs
usually tabulate these data by hand or with spreadsheet programs using the
following statistics:

8
A Toolkit for Outcome Measurement

146

l
l

Frequencies (number of participants)


Percentages (percent of youth observed playing board games together;
percent of ratings that were satisfactory)

Averages (average rating of physical mobility)

Range (lowest and highest number of participants)

Analyses of direct observation data is often are broken down (or crosstabulated) according to client characteristics, geographic and program
factors.
l
l

Outcomes achieved according to education level or age of students


Participation rates by programs located in the city core and those in the
suburbs
Percent of ratings that were satisfactory for Day Program, Residential
Program and Home Visitor Programs

Refer to Chapter 5: Developing Effective Questionnaires for guidance on


basic statistical analysis.
To analyze information from your unstructured observations that has been
recorded in observation guides and field notes, use content analysis to find
patterns and themes in the results. A full discussion of content analysis is
found in Chapter 6: Conducting Effective Interviews.
Take care in interpreting what was observed. Include others in the
interpretation or use other data to help you make sense of observational
data. It may be important to validate observations. You may choose to repeat
the observational process, make the sample size larger or record
observations with audiotape or videotape.
When reporting results of direct observation, include details about the
evaluation design, the process used to construct observation forms and
response categories (if applicable), training given to observers, and recording
procedures. If you used more than one observer, make a note of how similar
recordings of the same event were between them (a measure of inter-rater
reliability). Describing how the direct observation was implemented will help
assure audiences about the reliability and credibility of the data.

8
147 Direct Observation

Examples
Making the Most of Direct Observation
8A Orientation and Mobility Progress Checklist
Canadian National Institute for the Blind

8B Home Child Care Programs Coordinator/Caregiver Home Visit Report


Family Day Care Services

8C Direct Observation Rating Scale Home Support Services Evaluation


St. Christopher House

8D Outcomes Definitions Used for Direct Observation Client Services Program


Abrigo Centre
Note:
the formats of the following outcome measurement tools have been altered to some degree
to correspond with the design and space restrictions of this toolkit. We hope you will draw
inspirations, confidence, and even concrete elements for your own use, drawing upon the
steps and guidelines set out in these pages.

8
A Toolkit for Outcome Measurement

148

Example 8 A
Canadian National Institute for the Blind
Orientation and Mobility Progress Checklist - Identification cane
Client name

Phone: (

LESSONS (DATE AND LENGTH):


1

10

GOALS:
1

R = REPORTED NO NEED FOR SKILL

D = DEMONSTRATED MASTERY OF SKILL BEFORE INSTRUCTION

N/A = NOT APPLICABLE TO CLIENTS SKILLS/GOALS

GOALS ACHIEVED BY MASTERING:


1. INDOOR MOBILITY:
c Protective techniques
c Trailing
c Searching
c Basic sighted guide

c Sighted guide doors


c Sighted guide stairs
c Locating mailbox
c Elevator use

c Telephone
c Fire escape plan
c Accept and refusing aid

Observations/recommendations:

2. ID CANE USED: Length__________Manufacture ___________Paid for by:


c Folding procedure
c Diagonal technique
c Stairs(ascend/descend)

c Trailing
c Shore lining
c Examining Objects

ADP

CNIB

Client

c Parked car avoidance

Observations/recommendations:

8
149 Direct Observation

Example 8 A
Canadian National Institute for the Blind

Orientation and Mobility Progress Checklist - Identification cane (continued)


3. ALIGNMENT:

c Parallel with sound


c Perpendicular with sound

c Indented alignment
c Straight-line travel

c Squaring off to object


(side/back)

Observations/recommendations:

4. STREET CROSSING AND RECOVERING FROM VEERING:

c Asking for assistance


c Locating curbs graduated/drop off
c Residential crossings controlled
c Commercial Street controlled

c Standard traffic lights 4-way


c Advanced green
c Pedestrian crossings
c Traffic islands

Observations/recommendations:

5. COMMERCIAL TRAVEL SKILLS:

c Escalators

c Elevators

c Revolving doors

Observations/recommendations:

6. PUBLIC TRANSPORTATION:

c Locating Bus Stop


c Asking for assistance
c Boarding bus
c Seating position

c Mapping and recovering


c Subway travel
c Locating DWA (explain services)
c Approaching platform

c Boarding
c Exiting
c Streetcars (boarding/exit)
c Wheel trans

Observations/recommendations:

8
A Toolkit for Outcome Measurement

150

Example 8 A
Canadian National Institute for the Blind

Orientation and Mobility Progress Checklist - Identification cane (continued)


7.VISION TRAINING:

c Tracking skills
c Tracing skills
c Scanning skills

c Focusing skills
c Use of sunglasses
c Use of remaining vision to cross

c Use of remaining vision to


complete other skills

intersections
Observations/recommendations:

8. AUDITORY TRAINING:

c Use of hearing cues to cross intersections c Sound discrimination c Hearing aids


Observations/recommendations:

9. ROUTES MASTERED:
1

Observations/recommendations:

FINAL OBSERVATIONS/RECOMMENDATIONS:

8
151 Direct Observation

Example 8 B
Family Day Care Services
Home Child Care Programs Coordinator/Caregiver Home Visit Report
Caregivers
Date of Visit
Name:__________________________________(yy/mm/dd):______/______/___
Name
(List Names of Children Enrolled)

Family Day
Placed

Private CGs own Children


under 6 yrs in care

Reason of Visit:
q Scheduled
q Unscheduled
q Other:___________________________
In care
during visit

In
School

Absent
State reason

OBSERVATIONS AND DISCUSSIONS DURING VISIT:

Follow-up

Caregivers signature

Coordinators Signature

Time
From:

To:

A Toolkit for Outcome Measurement

152

Example 8 C
St. Christopher House
Home Support Services Evaluation
CLIENTS NAME: ________________________________________________PHONE NUMBER_________________________________
ADDRESS: ______________________________________________________________________________________________________
NAME OF WOKER: ______________________________________________________________________________________________
LAST SERVICE DATE: _______________________________DATE OF HOME VISIT: __________________________________________
Please assist us in evaluating the home support services.This evaluation is used to implement changes in service delivery and to monitor
the quality of the work provided by St. Christopher House.
Key: S = Satisfied NI = Needs Improvement NS = Not Satisfied

Meal Preparation

NI

NS

Personal/Attendant Care

Meals

Dressing

Snacks

Bathing

Special Diets

Toileting

Clean Up After Meals

Bedding Up/Down

Feeding

Teeth Brushing

Encourage Eating

Wash Hair
S

Medication Assistance

NI

NS

Skin Care/Mouth Care

Pick Up

Incontinence
S

NI

NS

Activation/Ambulating

Sweeping

Encourage Self Care

Ironing

Encourage Social Interaction

Laundry

Reading,, listening to music, games

Bed Making

Assist with Exercises

Vacuuming

Go for a Walk

Washing Floors

Assist with Transfer

Cleaning Bathrooms

Companionship

Cleaning Kitchen

Escort

Washing Dishes

Shopping, Banking,
Doctors Appointment

Caregiver Relief

NI

NS

NS

NI

NS

Supervision of Self Care

Dusting

Other

NI

Shaving

Remind
Housekeeping

Safety Check

Client Comments:
Supervisor Comments:
Follow up Report:
Program Worker:

153 Direct Observation

Date:

Coordinators Signature Date:

Example 8 D
Abrigo Centre
Outcomes Definitions Used for Direct Observation Client Services Program
OBSERVATION DEFINITION
CATEGORY
Access to essential
government and/or
social program(s)
achieved

Client is able to access government and/or social programs after counsellor intervention. Such intervention is
not limited to advocacy but could simply include counselling or translation i.e., housing, Legal Aid, legal advice
services, education, employment, heath services and health insurance (does NOT include income-related
programs.)

Dynamics of abuse
understood

Client reports or demonstrates enough understanding of the dynamics of abuse, whereby the
counsellor is able to observe the client making use of this understanding.

Emergency shortterm needs met

Clients emergency needs are met as a direct result from counsellor intervention where action is
required and provided on the same day. Emergency needs include: shelter, food, physical safety
(from assault, child abuse or suicide) and emergency funds.

Employment
secured

Client finds a job, part-time, full-time, temporary or permanent after staff intervention regarding
any employment-related issue

Income
maintenance
needs achieved

Client secures any essential income, except paid work, after staff intervention, i.e., :
l Ontario Works (OW) l Ontario Disability Support Plan (ODSP) l Canada Pension Plan
(CPP)
l Employment Insurance (EI) l Retirement and Disability l Old Age Security (OAS)
l Guaranteed Income Supplement, Federal (GIS)
l Ontario Guaranteed Annual Income System for Seniors (GAINS)

Informed decisions
made

As a result of staff intervention, client sufficiently understands options available to him/her with
respect to clients presenting issue or question. Client indicates to staff that s/he has information
required to make a decision, i.e., this would include translation of such information, such as mailed
advertising, resulting in client making a decision.

Interpersonal skills
improved

Clients interpersonal skills improved as a direct result of Abrigo Centre intervention. Examples
include, but are not limited to, situations where client demonstrates: Ability to interact more
positively with others
l A better understanding of impact of own behaviour on others. Improved ability to decipher
between controllable and non-controllable in ones life. Improved communication skills: verbal,
written, body language, trust
l Improved ability to understand varying points of view. Improved negotiation skills. Improved
assertiveness skills.
l A better understanding of own emotional needs and emotional safety.

Isolation decreased

Clients social network is improved, both formal and/or informal.

8
A Toolkit for Outcome Measurement

154

Example 8 D
Abrigo Centre
Outcomes Definitions Used for Direct Observation Client Services Program (continued)
OBSERVATION DEFINITION
CATEGORY
Living free of abuse
in the home

Violence Against Women program: Woman reports that she is no longer living in a relationship
that is abusive or assaultive to her and/or her children. Men in Transition program: Man has
stopped abuse behaviour as reported by the victim and/or partner.

Other

Counsellor is able to determine other specifically accomplished outcomes that will then be
documented in the Notes and in the Service Log Notes entry.

Parenting skills
improved

Clients parenting skills are improved as a direct result of Abrigo Centre intervention. Examples
include but are not limited to clients demonstration of the following points: Improved genuine
empathy for ones children.
l Normalization of child behaviour; l Improved understanding of child development;
l Alternatives to corporal punishment and other forms of negative reinforcement;
l Use of positive reinforcement; l Respectful behaviour towards ones children; l Ability to
provide
compromises between parent and children; l Information regarding child safety; l Impact of

Perpetrator
acknowledges
abusive behaviour

As a direct result of counselling intervention client acknowledges abusive behaviour (beyond court
convictions) directly to counsellor.The client is therefore able to discuss and provide examples of
his improved behaviour in the following areas: The man ceases blaming others for his actions of
violence and abuse; Decrease denial of abusive behaviour; Develop empathy for the victim and/or
children; Identifying impact of ones abusive behaviour on self.

Safety plan in place

Client has a copy of a safety plan or is able to discuss a safety plan that includes:
l Awareness of dangers client and/or children are in; l Formal emergency resources available to
client, e.g., 911, police, shelter, hospital; Informal emergency resources available to client such as
friends or family; l Emergency suitcase with spare cash, jewelry, special toys and keepsakes
l Planned emergency exits; Rehearsal of steps to take in emergency situation with client; Children
have been rehearsed in calling 911.

Self-esteem
improved

Client is improved in the following areas as a direct result of Abrigo Centre intervention:
l Client able to articulate positives in self and in own life; l Client pursues personal goals;
l Client able to validate own feelings and emotional reactions; l Confidence to expose
vulnerabilities.

Stress management
skills improved

Unknown

As a direct result of Abrigo Centre intervention the client is able to manage stress, i.e.: l
Relaxation, physical and mental; l Understanding own sources of stress; l Understanding own
unhealthy means of managing stress; l Client now has healthy means to manage stress;
l Recognition of own physical and emotional signs of stress.
Where conclusion needs to be put to file but counsellor not able to determine what outcomes
were achieved.This is a last resort outcome selection.

155 Direct Observation

Chapter 9:
Putting Program Records to Work

What Are Program Records?


Program records are a combination of official paper and computerized documents
including intake and referral forms, assessments, contact sheets, case files, financial
records, statistical reports and other documents.
Extracting information from program records is a relatively easy and inexpensive way
to collect data about program performance and achievement of outcomes because
you are drawing on a source of information that already exists. What is important to
recognize, however, is that data harnessed from program records can only be as good
as the information that was documented in the first place. Accordingly, if you are
looking to gather outcome data from program records, that information must be
documented in one or more records. Planning ahead is helpful to ensure that all
required information is anticipated, especially outcome information, and a process or
form developed to include it in the program records.
Program records, often gathered as part of the program implementation process,
typically contain data about client needs and characteristics, participation in program
activities, type and intensity of services, resources used and, in some cases,
achievement of short-term outcomes.The purpose of this chapter is to help you
harness data that you have by putting program records to work.
Agency in Action
The Distress Centres of Toronto operates a telephone helpline 24 hours a day, 7
days a week for callers who may be experiencing problems related to domestic
violence, social isolation, suicide, addictions, mental or physical health, or they may
simply be feeling sad or lonely. Counsellors offer callers emotional support, crisis
intervention and linkage to emergency help when necessary. Outcome measurement
is particularly challenging for this agency given that callers are anonymous and service
ends when the call ends. Even so, their one-page Call Report Form, completed for
every caller to the helpline, includes information about short-term benefits.
Counsellors document outcomes achieved by category, such as: (1) Provided
Information; (2) Exploration of Options; (3) Intervention Initiated (4) Referrals Made.
As well, written remarks recorded by counsellors often include qualitative outcomes
such as de-escalation of client emotions and client feels connected. Compiled
daily, this information can be aggregated at regular intervals along with other
information critical to program implementation and planning, such as demographics
of callers, presenting problems and referral organizations.

9
A Toolkit for Outcome Measurement

156

When Are Program Records Useful for


Measuring Outcomes?
l

When ongoing feedback is needed or a program is new. All programs


require ongoing monitoring of their day-to-day operations as a way of ensuring the
right clients are being served, the right services are being delivered in the right way
and at the right time. Program records usually hold important information to
monitor program implementation. In many programs, records can also document
outcome indicators to enable ongoing tracking of program effectiveness. Program
records are especially useful for new programs in which links between program
activities and client outcomes have not yet been established. If a quick check on
records reveals that clients are not progressing as anticipated, the program model
and service delivery process can be strengthened and improved in a timely
manner.
When the program establishes a relationship over time with the
client. Records of assessed needs, action plans, progress made and achievement
of personal goals are often maintained in programs that entail a relationship with
participants over time, such as case management or counselling services. Outcome
information may already be routinely recorded, or documentation procedures may
be modified to ensure changes and benefits experienced by program participants
are noted.

Case in Point
St. Stephens Community House operates with an open door philosophy serving
24,000 clients per year, some very briefly (one or two sessions) and others
intensively (every day) in 9 program areas. Given the informal, drop-in nature of
many of their programs and their limited resources, they concentrate on monitoring
activities and demonstrating only one longer-term outcome for each program area.
Most of this information comes from program records. In the Corner Drop-In for
homeless adults, for example, staff can, on a daily basis, keep track of such things as
how many people attend, the number of meals distributed, and how many referrals
are made and to which organizations. Because the majority of clients (over 80%)
return regularly, staff work continuously over time with clients to find and maintain
housing. When this outcome is achieved, it is documented as part of routine case
management.
l

When the program is not achieving its planned outcomes and


implementation problems are suspected. Programs can experience
implementation problems and program records can help you compare how your
current program looks against the original program plan. Program records may
reveal that the program is not being delivered to the intended target groups, that
there are problems in service delivery or they may help you understand better the
relationship between your program design and program outcomes.

9
157

Program Records

Main Strengths and Limitations of program Records


Strengths

Low cost

Information is accessible

Ongoing monitoring of program operations


over time

Early warning of problems

Facilitates comparisons between program sites

Limitations

Quality of data entered into


records may be poor

May be difficult to conceptualize


and implement

Requires good understanding of


program design

What Are the Strengths of Program


Records?
l

Information is available at low cost. Because records are created for and
maintained as part of program implementation, fewer resources are needed to
harness data from them than is required for other methods of evaluation.
Information is available and accessible. Program information is at your
disposal when you need it, supporting timely decision-making and sound program
management.
Ongoing monitoring of program operations over time. Well-organized and
accessible program records allow you to monitor the operations of your program
over a period of time.
Early warning of problems. Program records can give you early warning of
problems so you can make the changes needed to ensure that outcomes are
achieved. For example, monthly reports may show that not enough clients from the
target group are entering the program and new recruiting methods are in order.
Facilitates comparisons between program sites. Records provide the means
for assessing program implementation and outcome measurement across program
sites.This is useful for testing how well a program design works in different locations,
ensuring high standards across all sites and for continuous quality improvement.

9
A Toolkit for Outcome Measurement

158

What Are the Limitations of Program


Records?
l

Quality of data entered into records may be poor. It is fairly commonplace to find
that data in program records is, at times, incomplete, entered incorrectly or missing
altogether.This is sometimes the result of time constraints or confusion among
recorders about definitions and procedures. Some information needed for
evaluation and outcome measurement may not be available relevant information
may not have been documented because it wasnt anticipated when the record
system was developed. If you intend to use program records in evaluation, these
problems have to be addressed because the integrity of the evaluation relies on
the quality of the data in the records.You may find it necessary to go back to the
original sources of data to fill in the missing pieces and to make corrections. In
some situations, you may need to collect new data to supplement existing
documents. If you find that the program records have serious problems, you may
need to assess whether you should abandon them as a source of evaluative data.
May be difficult to conceptualize and implement. Most non-profit organizations do
not have a single information system that brings program record data together in
one spot. On the contrary, data is scattered throughout the agency in a variety of
record-keeping systems. Even if these systems are computerized, they rarely speak
with each other.These realities mean that it may take some effort to understand
your agencys record keeping systems, conceptualize a strategy for gathering the
information you need for the evaluation and then implementing the data collection
process without disrupting other agency activities.
Requires good understanding of program design. Full benefit from program records
can be obtained only if you have a good understanding of your program design
and how the various components are supposed to fit together and that you
recognize opportunities for recording meaningful information regularly.This means
investing in developing program logic models (or another framework) and making
continuous effort to understand the relationship between your program theory,
program activities and client outcomes.

Seven Steps for Putting Program


Records to Work
SEVEN STEPS FOR PUTING
PROGRAM RECORDS TO WORK
Step 1. Identify Items Of Data Needed
And Locate Records
Step 2. Check Quality Of Recorded Data
Step 3. Specify New Data Required
Step 4. Design Data Extraction Forms
Step 5. Organize Data Extraction Process
Step 6. Check For Data Transfer Errors
Step 7. Analyze The Data

159

Program Records

Step 1. Identify Items of Data Needed and Locate


Records
Create a list of specific information from your records that you will need to answer
your evaluation question. From an outcome perspective, this question would be What
effects has this program had on program participants? You will be looking for
information about benefits and changes experienced, including new knowledge, skills,
attitudes, values, behaviour, status or condition.
Are the data you need contained in the existing program records?

Examples of program records used by agencies


that may contain outcome information

Client intake and referral forms

Assessments

Service activity forms


Records of goal attainment and

client progress

Financial records

Minutes of meetings

Service statistics reports

Quality reports

Weekly and monthly variance reports

Quarterly and annual reports

Strategic and operational plans

Using your list of the items of data you need, examine your existing program records
to see if the items are available. Pay careful attention to the definition of the items you
need. Sometimes, where and how information is recorded changes over time without
anyone realizing it. For example, estimates of the percentage of clients speaking English
might be possible, firstly, based on answers to an intake question about language
spoken at home, but subsequently from an assessment of English language proficiency.
The two sources are likely to produce different statistics this can lead to unreliable
data and incorrect conclusions.
Note whether the data available in the program records can be used without any
changes. If not, note the changes that are required, such as adding missing data or
transforming the format i.e., months into days (see next step for more about
cleaning data).
If the data is missing or incomplete, you may consider gathering it specifically for the
evaluation using some other methodology or you may decide to omit it from the
evaluation entirely.You may also decide to modify your program records and
procedures to avoid this situation in the future.

9
A Toolkit for Outcome Measurement

160

Step 2: Check Quality of Recorded Data


Now that you know what data you need and have located the pertinent program
records, you are ready to assess the quality of the data.
Question 1: Is the data accurate?
Agency records often have problems and it is a general rule that the records should
be examined carefully before using the data in an evaluation.The most common
problems with both client records and service delivery records are missing and
incomplete data.The first step is to ask program managers and staff, especially data
entry staff, their views about the quality of the data.They may be able to pinpoint
potential problem areas.
The next step is to examine a sample of program records and determine what
proportion has missing, incomplete or inaccurate data. If the proportion is low, you
should judge the feasibility of your options, such as dropping missing and incomplete
cases or replacing missing and incomplete data with an average value or some other
estimate. If the proportion is high, you will need to consult with program managers
and staff to see if it is feasible and affordable to obtain the missing data.
Question 2: Is the data accessible?
You know that the data is in the program records and that it is accurate, but is it
accessible? A common problem is that data about individuals is summarized regularly
and only aggregate data is available. For your purposes, you may want to determine
whether the outcomes achieved by program participants vary with their demographic
characteristics gender, first language, education, work experience, level of income,
etc. If only summarized or aggregate data is accessible, you will be unable to perform
cross-tabulations necessary to uncover variations in outcome achievement. A related
problem is that the way data is organized and summarized varies from program to
program and cannot be integrated or compared.
Other difficulties include definitions of data or sources of data that changed over time,
so that the program records contain a variable mix of data under one item.You may
need to link data from different programs or locations, only to find that client
identifiers are not the same and links cannot be made.
When measuring outcomes, it is common to find many different ways of calculating
the percentage of outcomes achieved or to measure outcomes at different intervals
after program completion.These inconsistencies make using records challenging. For
example, you may need to extract information about how long it took program
participants to secure employment after being trained in job search techniques. In
some records, this outcome may be recorded in months and other times in days. In
these situations, some transformation of data will be required.
Finally, there is the important consideration of confidentiality. Usually agencies treat
program records as part of the administrative system that is covered under the
blanket permission signed by clients at intake. Staff undertaking an internal evaluation
within the agency usually have little problem accessing the program records. Agencies
often require explicit informed consent before records can be accessed by external
evaluators and this can be a time-consuming and expensive process.

9
161

Program Records

Step 3. Specify New Data Required


Once you have assessed the data available from program records, including its quality
and accessibility, you are in a good position to identify data that is not in the records
and so must be collected directly using other evaluation methods.You will need to list
each item of data you need from primary sources, a concise but clear definition of
each item, the sources of the data and what process you would use to collect it. Refer
to other chapters in this toolkit for guidance in using other data collection methods.

Step 4. Design Data Extraction Forms


Design and pilot-test the forms you will use to organize and compile data extracted
from program records. Make sure that you include a space in the forms to identify the
program record from which you are drawing data (i.e., name of client or assign a
number code) as well as the name of the reviewer and the date when the form was
completed. In the event of mistakes being made during data extraction, you will have a
way to go back and locate the original records.

4.1 Format
How you choose to document the information extracted from program records will
depend, in part, on what form it currently exists in. For instance, if the data is a
number, you might transcribe it directly (i.e., number of meals eaten, number of
referrals made).
Sometimes data may be presented in a checklist format, in which case you can
transcribe the information directly. For instance, you may have a checklist in the
program record regarding sources of income, employment status, languages spoken in
the home or type of accommodation.
If it is a written record, you may do a couple of things.You may do a content analysis
on the narrative.This requires that you have clear definitions of outcomes or other
issues that you want to extract. After reading the program record, the reviewer would
note, by word, checkmark or other notation on a prepared list, which outcomes or
issues were present.
Alternatively, you may prepare your reviewers to read the narrative record and make
a judgment about it, using a rating scale to portray the contents of the record (i.e.,
Rate how much progress the client has made in achieving his/her personal goals? 1= A
lot; 2= Some; 3= Little; 4= None).
In some cases, the data in the program records is recorded in different formats and
you must clean it to make it uniform. For instance, you may need to extract length of
service that is sometimes recorded in months and other times in weeks. Provide clear
instructions to reviewers on how data is to be transformed for use in the evaluation.

Case in Point
Community Information Toronto (CIT), when preparing for its new 211 telephone
access number, created a computerized system to monitor the amount and type of
service provided to callers. Software automatically monitors the number of calls
received, answered, abandoned and average wait times. In addition, for each call, staff
documents in the computerized record the number of referrals made, the type and
amount of advocacy calls implemented, as well as caller demographics. On a monthly
basis, CIT extracts information that supports day-to-day management of the program
and also shows how their assistance facilitated access to needed human services.

A Toolkit for Outcome Measurement

162

4.2 Unit of Analysis


Be aware of whether the data in the program records reflects the experiences of an
individual, family or community. Who or what is benefiting and changing from the
program? Do you want information at one level or multiple levels? Make sure the unit
of analysis is consistently applied as you extract the information and transfer it to the
data collection form.

4.3 Contextual Data


Look to create a form that allows you to assemble relevant information that will assist
you in analyzing and drawing conclusions about the outcome data you are using in the
evaluation. For instance, you may be interested in how outcomes vary depending
upon client characteristics such as age, gender, income level, neighbourhood, amount of
services used or language spoken at home.

4.4 Range of Acceptable Values


In preparing the forms for extracting information from program records, give some
thought to the range of possible values for each item of data.This will help you to
identify errors in the original files and again when verifying the accuracy of the
extracted data. For instance, if you are recording sources of income, consider the
possibilities.The range may be readily anticipated, including no income, wages,
government programs, pension, personal investments, family or friends. In another
situation, if you are looking to record the number of hours of service received in the
last month, you may be able to estimate an average amount and anything far in excess
or meagre may be cause for checking for errors. For example, if you figure clients
attend on average once per week for 45 minutes, and one record reveals that a client
attended every day for two weeks for 90 minutes, you would be inclined to verify that
with the staff person who made the entries originally.

4.5 Confidentiality and Privacy


The need to protect the privacy of personal information is a matter all organizations
must consider seriously. In extracting information, a level of protection may be given
to clients by assigning number codes to the extraction records rather than using
names. If you decide to offer this protection, make sure that you make a separate
record of client names and assigned codes to ensure that you have a way of checking
back in the event of mistakes in extracting information from the original program
record.
Client confidentiality can also be protected by making sure that identifying information
is not used in the analysis and reporting of the evaluation results.
Case in Point
Jewish Family and Child Service of Greater Toronto used existing program records to
explore whether good outcomes for children were achieved with an early
intervention, multi-service approach to support for families in distress.The following
is the template created for extracting required information from existing case files.
Note the different formats of data, the use of content analysis to code outcomes
achieved, the use of a rating scale to capture the reviewers judgment about family
progress, the differing units of analysis (family, parents, children) and the addition of
descriptive characteristics of the family situation and use of services to inform the
analysis.The date and identity of the file reviewer are also noted on this form.

163

Program Records

JEWISH FAMILY AND CHILD SERVICE OF GREATER TORONTO OUTCOME EVALUATION


File#:
Name :
File Social Worker:
Reviewer:
Main reason: (Check ONE ONLY) for service (last opening)
c 1 Child academic
c 4 Parent-child
c 2 Child social
c 5 Couple
c 3 Financial need
c 6 Family

Date:
c
c

7 Parenting
8 Separation/divorce

Other (specify):

DATE OF LAST OPENING:


REFERRAL SOURCE: :
c 1 IFRS
c 4 School
c 7 Other (specify):
c 2 Self
c 5 Report from community
c 3 JIAS
c 6 Outside professional
SOURCE OF INCOME: (check as many as apply, write F for Female, M for Male over your choices):
c 1 Wages
c 4 CPP
c 7 Other(specify) ___________
c 2 Investments
c 5 JF&CS Assistance
___________________________
c 3 OW/ODSP
c 6 Family contributions
c 8 Unknown
ACCOMMODATION:
c 1 House
c 1 Owned
FIRST LANGUAGE:
c
c
c

1 English
2 Russian
3 Caucasian language

c
c

2 Apartment
2 Commercial rent

c
c

3 Shelter
3 Subsidized

4 Other

c7 Other (specify) ___________

4 French
5 Spanish
c 6 Arabic
c

ENGLISH SKILLS:
c

c 2 Excellent
c 3 Good
c
c5
Poor
1 First language
4 Fair
ANY FAMILY OR COMMUNITY SOCIAL SUPPORTS (e.g. grandparents, neighbours, friends; check all that apply):
c
c
c
c

1
2
3
4

Parents
Grandparents
Other family
Friends

c
c
c

5 JACS
6 Other community self-help group
7 Community drop-in center

SERVICES PROVIDED SINCE LAST OPENING:


c
c
c
c
c
c
c
c

1 Financial Assistance
(monthly cheque)
2 Special requisitions
3 Just-a-Second
4 Advocacy
5 Camp
6 Dental
7 Pharmacy
8 Volunteer tutor

c
c
c
c
c
c
c
c

9 Volunteer driver
10 Supportive/intermittent
counselling
11 Group for social skills
13 JFLW/PUTP
13 JFCS therapy group
14 Case management
15 School program contact
16 Referred to outside therapy/group

CHECK ALL

8 Outside professionals independently

accessed

c
c
c
c
c
c
c

9 Other (specify):_____________

17 Individual counselling
18 Couple counselling/therapy
19 Family counselling
20 Monitoring parenting
21 Abuse/neglect investigation
22 Woman Abuse services
23 Other (specify):

From your reading of the file (including RFS; Service Plans and their ratings; other recordings and notes) how would you rate familys
progress to date since last file opening? Check ONE:
c 1 Much better
c 2 Better
c 3 Same
c 4 Worse
c 5 Much worse
What do you notice is different about the family now (compared with the beginning of service) that impacts on the child and that could
be reasonably attributed to Agency services provided (exclude things like windfall from Aunt Minnies will)? Check ALL that apply:
c
c
c
c

1 Improved child social


competency
2 Improvement in child wellbeing
3 Adult completed further
education
4 Improvement in woman
abuse situation

c
c
c
c
c

5 Improved emotional stateadult


6 Increased financial
independence
7 Improved coping skills
8 Improved parenting
9 Improved partner
relationship

c
c
c

10 Stabilized housing
11 Less social isolation
12 Other (Specify):
___________________________

If worse, write in the relevant numbers here, e.g. if childs well-being has deteriorated, write 2"

HERE:_________________________________________________

A Toolkit for Outcome Measurement

164

9
1
165

Program Records

Examples
Putting Program Records to Work

9A Distress Centre record of call


Distress Centres of Toronto
Note:
the formats of the following outcome measurement tools have been altered to some degree
to correspond with the design and space restrictions of this toolkit. We hope you will draw
inspirations, confidence, and even concrete elements for your own use, drawing upon the
steps and guidelines set out in these pages.

9
A Toolkit for Outcome Measurement

166

Example 9 A
Distress Centres of Toronto
Distress Centre Record of Call
CALL REPORT FORM

Call Report Number #:

Volunteers Name :

Number/Alias:

Leaders Name:

CALLER PROFILE

Date:

_____ /_____ /______


MM

YY

Callers Name ____________________


Gender:

Male

Female

Time (24hr Clock): Start:

___________

End:

___________

DD

_______________________________________________________ Municipality Code

Unsure

Age Group:

c<15

15-24

25-44

45-64

65-84

85+

____________

Unsure

(PLEASE CHECK APPROPRIATE CODES)


MARITAL STATUS
(max.2 codes)
c 1 Single
c 2 Separated/Divorced
c 3 Widowed
c 4 Married/Partnered
c 9 Unsure

ECONOMIC STATUS
c 1 Working Full Time
c 2 Working PartTime
c 3 Unemployed on EI
c 4 Unemployed on SA
c 5 Disabled on SA
c 6 Retired Pensioner
c 7 House Person

8 Student
9 Other
c 10 Homeless
c 19 Unsure

LIVING ALONE
c 1 Yes
c 2 No
c 9 Unsure

IN TREATMENT
(max.2 codes)
1 Mental Health
c 2 Physical Health
c 3 None
c 4 Support Group
c 9 Unsure

CALLER STATUS
c 1 New
c 2 Occasional
c 3 Continuous
Support
c 9 Unsure

ACTION (PLEASE CHECK APPROPRIATE CODES)


CONTENT OF CALL
(max.2 codes)
c 01 Information
c 02 Support
c 03 Distress
c 04 Crisis
c 05 Prank Call/Wrong #
c 06 Phone Device for
Deaf
c 07 Thank-you Call
c 08 Third Person Call
c 09 Update from Caller
c 10 Admin/Potential Vol.

PROBLEMS

19 Other

RESPONSE TO CALL
c 01 Intervention Initiated
c 02 Emotional Support
c 03 Exploration of
Options
c 04 Call not Completed
c 05 Provide Information
c 09 Other
CALLERS REACTION
c01 Appreciates Service

02 Dissatisfied
03 No perceived
difference
c 04 Perceived
improvement
c 05 Not applicable
c

REFERRAL FROM
c 01 Professional
c 02 Advertising
c 03 Phone Book
c 04 Police
c 05 Community Agencies

06 Friends/Relatives
07 Previous Caller
c 08 Self Referral
c 09 Not Applicable
c 10 Bell Canada
c 19 Other/Unsure
c

REFERRAL TO
(max.2 codes)
c 01 Medical
c 02 Counselling
c 03 Primary Service
c 04 Youth Service

05 Seniors Services
06 Self Help Group
07 Legal
c 08 Agencies
c 09 Emergency Housing
c 10 Police
c 11 None Appropriate
c 12 No Referral Required
c 13 Crisis Support
c 19 Other
c 21-39 For Centre Use
c
c

(PLEASE CHECK APPROPRIATE CODES MAXIMUM SEVEN PROBLEM CODES)

PHYSICAL HEALTH
(max. 2 codes)
c 01 Short Term Physical
c 02 Long Term Physical
c 03 Pregnancy
c 04 Physical Disability
c 05 Communicable
Diseases
c 06 Sleep Disorders
c 09 Other
SUICIDE
c 10 Significant Other
c11 Attempt in Progress
c 12 High Risk
c 13 Ideation
c 19 Other

MENTAL HEALTH
c 20 Self Esteem/Self Worth
c 21 Diagnosed Mental
Illness
c 22 Developmental
Disability
c 23 Mental
Health/Emotional
c 24 Inconsitencies/Unsure
c 25 Phobias
c 29 Other
INTERPERSONAL
c 30 Parent/Child
c 31 Family
c 32 Relationships
c 33 Caregiver/Helper
c 34 Married/Partnered
c 35 Bereavement

c
c
c
c
c

36
37
38
39
40

Isolation
Loneliness
Personal Adjustment
Social Maintenance
Other

SEXUALITY
c 45 Sexual Dysfunction
c 46 Sexual Orientation
c 47 Sexual Issues
c 49 Other
OCCUPATIONAL/
FINANCIAL
c 50 Unemployment
c 51 Employment
c 52 Financial
c 53 Education/Training
c 54 Basic Training

c
c
c

55 Accommodation
56 Childcare
59 Other

ABUSE/VIOLENCE
c 60 Physical Abuse
c 61 Sexual Abuse
c 62 Verbal/Emotional
Abuse
c
63 Violence Toward Others
c
64 Abuser
c
65 Self Mutilation
c
66 Childhood Abuse
c
69 Other
SUBSTANCE ABUSE
AND ADDICTIONS
c 70 Alcohol
c 71 Drugs

See other side for Narrative, Volunteers Comments and Feedback

SUPPORT FOR THE CALLER COMES FIRST; DATA SECOND.

167

Program Records

c UPDATED

c
c
c
c
c
c

72 Medications
73 Gambling
74 Smoking
75 Seeking Sexual
Stimulation
76 Eating Disorders
79 Other

GENERAL
c 80 Legal
c 81 Police/Criminal Activities
c 82 Immigration
c 83 Spiritual/Religious
c 84 Abusive/Angry Caller
c 85 Caller Complaints
c 86 Multicultural
c 87 Government
c 88 Future Use
c 89 Other

FOR CENTRE USE:


95 96 97 98 99

c DATA

Example 9 A
Distress Centre Inc.
ENTERED

Distress Centre Record of Call (continued)


CALL REPORT FORM

Call Report Number #:

Volunteers Name :

Number/Alias:

Leaders Name:

CALLER PROFILE

Date:

_____ /_____ /______

Callers Name ____________________

____________
Gender:

Male

Female

Time (24hr Clock): Start:

___________

End:

___________

_______________________________________________________ Municipality Code

Unsure

Age Group:

c<15

15-24

25-44

45-64

65-84

85+

Unsure

NARRATIVE:

VOLUNTEERS COMMENTS:

FEEDBACK:

9
A Toolkit for Outcome Measurement

168

Chapter 10
Applying Case Study Methods

What Are Case Study Methods?


Case study methods combine information from a variety of sources to
produce a rich, in-depth analysis of specific situations. Useful for providing
detailed descriptions of program implementation over time, exploring the
feasibility of program models and explaining how and why a program is
producing results (or not), a case study is an excellent way to present a
realistic portrait of a program and its ability to achieve intended outcomes.
Unlike random sample surveys, case studies are not representative of entire
populations, nor do they claim to be. In case studies, one is generalizing to a
theory the program logic model based on cases selected to represent
aspects of that theory.
During the last decade, the uses and approaches to case study methods for
evaluation has expanded greatly, each method providing a broad overview of
a complex situation and offering insights into how the program operated and
why things happened as they did.

Agency in Action
Jewish Family and Child Service of Greater Toronto(JF&CS) undertook an evaluation of its
Early Intervention (EI) program using a case study approach.The purpose of the study was to
evaluate the success of the EI program in fostering better outcomes for children whose
families were receiving non-mandated services from JF&CS.The primary hypothesis of the
evaluation was that a multi-service approach to helping families with children leads to
successful outcomes for the children.
The methodology included selecting a random sample of 30 case files from the population of
248 non-mandatory cases.To gain a better understanding of how the EI program had been
helpful to them, confidential interviews were conducted with 27 of the same 30 clients, with
three clients being ruled out of the interview on clinical grounds. JF&CS developed a file
study protocol to guide the collection of data from the 30 case files. Sections of this protocol
were incorporated into the interview protocol to allow for comparisons between data
collected from case files and data collected through interviews.This data collection strategy
strengthened the evaluation findings of the EI program.

10
1
169 Case Studies

When Are Case Study Methods Useful


for Measuring Outcomes?
l

When you want to examine the contribution of program


components to outcomes. An in-depth case study can provide valuable
insight into the relationship between program components and program
outcomes, and may help to identify the role of other intervening factors
as well. Case studies help test aspects of the theory of change
underlying a program. Findings from several case studies can be brought
together to answer credibly cause-and-effect evaluation questions.
When you want to tell a holistic human story. Case studies uniquely
allow you to see the client in the program within the larger environment.
Case studies also illustrate how important and valuable the changes and
benefits are to the client, however small they may seem out of context.
When a program is new. Demands for tangible proof of outcomes often
begins within months after a program begins. Senior managers, Board
members and funding sources often want assurances of program
effectiveness. One strategy to cope with these pressures is to conduct
several case studies. In the earliest months of a program, you could
conduct case studies of clients who are progressing as expected, to
document the ability of the program to deliver positive outcomes under
favourable conditions. You could add case studies of clients who,
according to their own feedback and staff perceptions, are participating
well but who are not achieving the expected outcomes. This provides an
opportunity to consider, at an early stage, flaws in program design and
theory, timely corrective action, if warranted, and development of realistic
outcome expectations.

Main Strengths and Limitations of Case Study Methods

Strengths

"Rich" information on specific cases

Explains program effects in typical situations

Can answer cause-and-effect questions


with careful design

Limitations

May be seen as unreliable and biased

Case studies are complex

Can be time-consuming and expensive

What Are the Strengths of Case Study


Methods?
l

10

Makes the program come alive. Case study methods bring together
personal, program and contextual information that gives the reader of
evaluation reports a human and first-hand glimpse of what it is like to

A Toolkit for Outcome Measurement

170

participate in the program and to experience the types of outcomes


achieved. Multiple sources of evidence help to build the credibility and
validity of findings and to place the case in its proper context.
l

Explains program effects in typical situations. One valuable feature of


case study methods is that they portray a program and its effects within
a real-world context. Descriptions and explanations based on thoughtfully
designed case studies have a distinctive air of authenticity and persuasive
ability not easily found in other evaluation methods. Cases can be
selected to show a variety of typical situations, such as problems of
service delivery without adequate resources, the challenges of delivery
to difficult-to-serve populations, or the effects of contextual factors on
program delivery and outcomes.
With careful design, answers cause-and-effect evaluation questions.
In recent years, case study methods have emerged as a workable
alternative to experimental and quasi-experimental designs to answer
questions about the cause-and-effect relationship between program
activities and the observed outcomes. Although case studies do not exert
the type of control over clients and programs found in experimental
designs, they do allow the logic of experiments to be used by framing
hypotheses/ theoretical propositions and carefully choosing cases to rule
out alternative explanations.

What Are the Limitations of Case


Study Methods?
l

Misunderstandings about case study methods. Case studies often are


associated with highly selective personal testimonials that are utilized for
program promotion and justification. Case studies may be viewed as
unscientific, unreliable or so unique that results cannot be generalized.
Although these concerns can be addressed, it is important to recognize
that some prejudice does exist against case studies when you are
selecting your evaluation methods.
Case studies are complex, using multiple methods. Case studies
seem easy a few well-chosen quotes taken from client satisfaction
surveys or client interviews. On the contrary, good case study methods
go to great lengths to control bias and use multiple methods to present a
balanced view of both positive and negative aspects of the program. You
may want to consult an evaluator to help design a systematic and
rigorous approach to design study protocols and minimize bias.
Case studies can be time consuming and expensive. Some forms of
case studies draw on data collection practices, such as participantobservation methods, that require time in the field and generate
extensive observational and narrative data that takes time to analyze.
Depending on the evaluation questions you want to answer, it is possible
to do high quality yet relatively rapid, inexpensive case studies by using,
for example, existing program records and telephone interviews.

10
171

Case Studies

Six Steps for Applying Case Study


Methods
Case study methods can produce valid and reliable evaluation findings if they are
designed and implemented systematically. The following sections outline the key
steps in designing and applying case study methods.
SIX STEPS FOR APPLYING
CASE STUDY METHODS
Step 1. Determine the Purpose
of the Case Study
Step 2. Select Unit of Analysis
Step 3. Determine Case Study Design
Step 4. Select Data Collection Methods
Step 5. Develop Protocols and Collect Data
Step 6. Develop Data Analysis Plan

Step 1. Determine the Purpose of the Case Study


Case study methods are a flexible approach to outcome measurement. They can
provide qualitative information about the benefits experienced by clients but they
can also examine program processes. They can illuminate, for example, how
and why a program produces, or fails to produce, desired outcomes.
There are three major types of case studies, each having a different purpose.

Descriptive Case Study. Descriptive case studies are used to make the
unfamiliar familiar, to give various stakeholders a common way of understanding
a program as clients experience it. They can illustrate in detail the outcomes
achieved by clients, variations in programs implemented in different locations or
with different populations. Outcome information from descriptive case studies
may be used to substantiate or elaborate on evaluation findings from other
sources such as questionnaires and interviews. Descriptive case studies can
also demonstrate how a program is truly being implemented. This type of case
study usually is normative it compares planned vs. actual, or actual vs.
standards.
Exploratory Case Study. The exploratory case study helps to sharpen
program theory, inform program delivery, confirm or change the program logic
model and generate hypotheses, indicators and cause-and-effect propositions
about a program. These studies may be used to provide early verification that a
program is effective and may also be used to prepare for a later evaluation of the
program.

10
A Toolkit for Outcome Measurement

172

Case in Point
To explore the possibility of expanding a program, an agency reviews the range of program
costs and outcomes for different types of clients. By pulling together data from case files,
direct observation and interviews with clients and staff, the agency constructs three different
case scenarios:
Case 1: Easy-to-serve client, requiring little service, such as information and referral
Case 2: Average client, requiring the usual range of services
Case 3: Difficult-to-serve client, requiring intensive services that are not typical
The three cases show how the program responds to different needs. Program statistics help
estimate the proportion of clients that fall into each scenario.The case study gives a good
picture of the potential benefits and costs of expanding the program.

Explanatory Case Study. This type of case study examines the causeand-effect relationships between the program activities, the local context for
the program, the general environment of the program, and the program
outcomes. Its purpose is to provide evidence that the experience of the
program participant matches the theoretical expectations, thereby
demonstrating that the program contributed to the observed changes and
benefits. It is also useful when you suspect that factors outside of the
program design may affect program outcomes.

Step 2. Select Unit of Analysis


Case study methods are flexible enough to be used for many different units
l

Individuals

Organization

Family/Group

Neighbourhood/Community

Program

Province/Nation

of analysis. Case studies can reveal the experience of:


If you have difficulty identifying the unit of analysis, it may be because you
have too many evaluation questions or that the questions are too vague. To
clarify the unit of analysis, make a list of your evaluation questions and then
think through how the proposed specific case or group of cases will help to
answer these questions.

Step 3. Determine Case Study Design


Case studies can be either single or multiple-case designs. Single-case
designs require careful investigation to avoid misrepresentation. Case
studies can represent a single point in time, two points in time (before and
after an intervention), or multiple points in time (longitudinal). You can use
any of these designs for descriptive, exploratory or explanatory case studies,
and with any unit of analysis (individual, family, community, program). Choice
of design will be influenced by the resources available, what you need to
know, and access to information.

10
173 Case Studies

3.1 Single Case Designs


Single case designs apply when you are documenting one situation one
client, one program, one community, one instance, one event. Consider the
following examples (this is not an exhaustive list):
l

Representative Case A case provides a description of the average


client or program or some aspect of the program.
Extreme or Unique Case A case describes an unusual situation to
document that the extreme situation is a possibility and/or to learn
lessons about coping with similar situations, should they arise again.

Case in Point
Every month, Community Information Toronto(CIT) asks each staff member to write a brief
description of one average Day-in-the Life and one Exceptionalcall to its 211 information
and referral telephone service.The descriptions are posted on the bulletin board for staff to
read. Some are selected for inclusion in the Annual Report to give a representative picture of
CIT clients.The first example is an average call, the second one an exceptional call.
Having just located to downtown Toronto from Oakville and with a new job starting in a
month, a working mom needed child care for her two infant children. After asking questions
about her needs and determining where she lives and works, the 211 Information and
Referral Specialist was able to suggest a range of child care options and a local parent
support group where she could meet other families in the neighbourhood.
A distressed mother of three young children called because she needed some financial
assistance. She was relieved to speak to an Information and Referral Specialist in her own
language, Punjabi.They talked about her situation, and the Specialist was able to help her
identify that, in addition to financial assistance, she also needs legal advice and counselling. She
was grateful to be referred to supportive family service organizations and a community legal
clinic in her area.
l
l

Test Case A case confirms, challenges, or extends program theory.


New Program A case documents a new program, or program
enhancement to assess its feasibility, correct any problems in program
implementation or check against plans or standards.

3.2 Multiple Case Designs


Multiple case designs involve either (a) the use of single cases to explore
several different aspects of a program, i.e., different locations, different
target populations, etc., or (b) the use of several cases to examine one
situation, looking for corroborating data and patterns in the findings. When
you have one of the following conditions, then multiple case designs are
usually the right choice:
l

10

Representative Cases Cases are selected because they represent an


average for each important variation in the program. For example, an
agency that operates multiple drop-ins may select for case study one
program that represents a youth drop-in, another drop-in for adults and
another drop-in for seniors. The case studies would show the similarities
and differences in each of these representative cases.

A Toolkit for Outcome Measurement

174

Bracketing Cases In this situation, you want to know what is


happening at the extremes of the program, so that you understand the
range of, and explain the variation in, program outcomes.
Best / Worst Cases In this scenario, you study several cases that
have shown strong (or poor) progress in the program. This can help you
understand what aspects of the program, or what intervening factors,
contribute to (or hinder) positive outcomes.
Special Interest Cases If your organization is involved in responding to
a special situation or unique crisis, you may undertake a case study,
drawing on several different experiences, to understand how the program
operated in these circumstances, the results achieved and the lessons
learned.
Cumulative Case Study The cumulative case study design brings
together the findings of several case studies in an overall evaluation. The
cumulative case study is often used in complex programs or situations,
for example, to answer How are our community programs making a
difference in reducing child poverty? To answer this evaluation question
and related ones may require several case studies of programs located in
different neighbourhoods that span a number of years.
Cluster of Cases A cluster analysis usually involves several programs
that are similar and vary in certain aspects of their program model. For
example, one employment program may provide job preparation and job
search skills. Another employment program may add career counselling
to job preparation and job search skills. A third program may add
mentoring. The case studies let you know how each of the programs
compare with each other in terms of their outcomes, but also on other
pertinent factors, such as clients acceptance of the program, drop-out
rate and regained sense of self-esteem.

Step 4. Select Data Collection Methods


A defining feature of the case study approach is the use of multiple
sources of evidence. For example, analysis of case files combined with
interviews and direct observation. The following methods are used most
often in case studies.
Review of Program Records. Program documents contain a great deal of
information about a programs history, values, goals, program plans,
budget, management and staffing, and changes in leadership and
operation over time. The most commonly used types of program records
(paper and electronic) include:
l

Client records that show the number of clients served and their
demographic characteristics
Service delivery records that show the type and frequency of service
activities
Program plans and budgets

175 Case Studies

10
1

Human resource records, including job descriptions, organizational


charts, personnel lists, volunteer records, etc.

Minutes of meetings, correspondence, reports, etc.

Survey and evaluation data.

The main drawback of program records is that they can be incomplete,


inaccurate or biased. For these reasons, program records are seldom the
primary source of case study data. Instead, it is used to check information
from other sources and to supply important program details. For details,
see Chapter 9: Putting Program Records to Work.
Interviews. Interviews are one of the most important sources of data for
case studies because they provide in-depth information that may not be
available from other sources. Interview data is essential in situations
where program documentation or records may not exist or are incomplete.
In general, case study interviews tend to be less structured and more
open-ended than other forms of interviews. It is not unusual for
interviewers to ask informants to clarify factual information, provide their
view of events, suggest explanations of why events took place or
outcomes occurred, and suggest other sources of information and persons
to interview. In the case study environment, an interviewer must be well
trained, have a keen sense of ethical standards and be able to retain
objectivity. For more information about interviews, see Chapter 6:
Conducting Effective Interviews.

A TIP FOR CONDUCTING THE CASE


STUDY INTERVIEW
One of the main reasons for using case studies is to
ask why questions. When conducting an
interview, it is important to recognize that
interviewees may feel defensive when they are
asked why questions, such as Why did you
decide to include only recent immigrants in your
program (although others needed these services
too)? The general rule is to replace why
questions how questions that are less threatening.
For example, the question How did you select
participants for your program evokes less negative
reaction than Why did you decide to include only
recent immigrants in your program?

10
A Toolkit for Outcome Measurement

176

Direct Observation. Case studies can make use of different forms of


observation direct, participant and artifact. Direct observation can be
informal, such as recording general impressions during visits to a program
site. On the other end of the spectrum, direct observation can be formal
and follow data collection protocols. See Chapter 8: Making the Most of
Direct Observation.
Participant-observation requires the observer to take an active role within
the situation, including participation in some of the events being studied.
This technique provides some unusual opportunities for collecting data, but
could face some major problems as well: The researcher could lose
objectivity or alter the course of events by being part of the group.
In some situations, observing physical artifacts can be very useful. For
example, in a case study of a youth mentoring program, it may be possible
to study the physical products of the mentoring activities, in the form of
videos and music CDs produced. Each video or CD could be assessed for
its quality.

Step 5. Develop Protocols and Collect Data


The case study protocol, a set of brief instructions, helps guide your data
collection activities and improves reliability of the data, especially when
doing multiple case studies and using more than one data collector. The
protocol keeps you focused by defining the purpose and key evaluation
questions. It guides data collection by linking sources of information to
the evaluation questions. It can help organize data so that analysis will be
easier. Finally, it assists you to plan ahead, anticipate problems and set
priorities by presenting the outline for the report.
Overview of the Case Study. The overview contains the purpose and
sponsors of the evaluation, a synopsis of relevant background information
and important issues, and the rationale for using case studies. Write the
overview so you can give it to anyone who needs to know about the
evaluation, including people you plan on interviewing. You may choose to
list the key evaluation questions you are seeking to answer.
Data Collection Guide. For each evaluation question, develop a list of
sources of data that you can use to answer the question. Depending upon
the methods chosen, develop necessary forms and procedures for
gathering data from interviews, program records, focus groups or direct
observation.
Data Analysis Guide. Case studies can generate large amounts of data
from multiple sources, both quantitative and qualitative, and become
difficult to organize and analyze. Plan ahead to avoid being overwhelmed
by the volume of information. Because data collection and analysis often
happen simultaneously, it is important to draft data analysis plans and tools
to organize the data. For example, you may choose to develop simple
matrices to show visually what data has been assembled to answer each
evaluation question.

10
1
177 Case Studies

Outline of the Case Study Report. To help manage and organize


findings in a way that will meet stakeholders information needs, you
should develop an outline of the case study report. The outline of the
report helps maintain focus on the study priorities, understand the
intended format for data presentation, and ensure that all of the data
needed for the case study report is collected.

Step 6. Develop Data Analysis Plan


Unlike other evaluation methods, there are few guides for the analysis
of case study data. Thinking and reflecting about the findings, often
over a short period of time, will help you to interpret and bring meaning
to the data collected. The key is to develop a general strategy in
advance. This will help you control bias, help rule out various
interpretations of the data and strengthen your evaluation conclusions.
The merit of the case study approach, based as it is on select
situations, lies in the opportunity to test the theory of change upon
which the program is based. Case study analysis helps to refine the
program logic model, illuminating relationships between program
activities, external factors and program outcomes.
You are looking to find experiences in the cases that match the ones
predicted by the program logic model. This provides evidence that helps
prove that your program contributed to the changes and benefits that
occurred.
You should also actively look for evidence that factors other than the
program caused the observed outcomes. By eliminating rival
explanations, you are in a stronger position to argue that program
interventions played a critical role. The rival explanations that are likely
most applicable in community services include:
l

Another program or policy is producing the results (e.g., support


group vs. your program; income tax credits vs. your program)
Environmental forces overwhelm program effects (e.g., personal
situations, community conditions, economy, etc.)
Another theory explains the results better (e.g., encouragement by
staff build self-esteem and teaching skills are not that important).

10
A Toolkit for Outcome Measurement

178

You must examine the data, formulate and test hypotheses, apply rigorous
logic to rule out hypotheses, collect supplemental information as
warranted, and then, through this interactive process of logic and data
gathering, draw conclusions.
Be sure to demonstrate that your analysis, and the conclusions you draw
from it, are based on all relevant data gathered. And address in your
conclusions the most significant aspect of the case study.

10
179

Case Studies

Examples
Applying Case Study Methods

10A Excerpts from Executive Summary Early Intervention (EI) Study


Jewish Family and Child Service
Note:
the formats of the following outcome measurement tools have been altered to some degree
to correspond with the design and space restrictions of this toolkit. We hope you will draw
inspirations, confidence, and even concrete elements for your own use, drawing upon the
steps and guidelines set out in these pages.

Editors Note: The unit of analysis in this case study evaluation is a program.

10
A Toolkit for Outcome Measurement

180

Example 10 A
Jewish Family and Child Service
Excerpts from Executive Summary Early Intervention (EI) Study
March 2003
Background
Jewish Family and Child Service in 2001 undertook an evaluation of the Early Intervention (EI,
previously known as Prevention) program . The EI Study was to augment the extensive
literature search and limited file study of its predecessor by reviewing a larger number of
randomly-chosen files and by interviewing their subjects.
In the fall of 2001, a staff advisory committee was struck to provide oversight and support for
the effort, and to review the files chosen for the study. The study was designed to evaluate the
success of the EI program in fostering better outcomes for children whose families were
receiving non-mandated services with JF&CS. For the purpose of this study, Early Intervention
cases are those including a child below the age of 16 in the home, especially non-mandatory
child protection cases.
Hypothesis That a multi-service approach to helping families with children leads to successful
outcomes for children.

Methods
Outcomes were to be measured via: 1) Reading of 30 randomly chosen files (from amongst the
categories outlined above); 2) Interviews with the same 30 clients to gather their understanding
of how the Agency had been helpful to them.

File study
l
l
l

A stratified random sample of 30 files was selected from among 248 non-mandatory, i.e. EI,
child cases (child under 16 in family) in three categories.
A protocol for the file study was designed and piloted. It was then employed in gathering
the required data from the 30 study files.
Sections of the protocol were incorporated into the interview protocol to facilitate
comparison.

Interviews
l

l
l
l

Twenty-seven of the 30 randomly chosen clients were deemed by their social workers to be
appropriate (on clinical grounds) for the personal interviews, and 24 (90%) of those were
actually interviewed.
An independent consultant was employed to perform the interviews and to write a report on
her findings.
Respondents were guaranteed confidentiality in order to encourage candor on their part, and
to enhance the validity of any findings from the interviews.
Quantitative data was entered anonymously into a data file.

10
181

Case Studies

Example 10 A
Jewish Family and Child Service
Excerpts from Executive Summary Early Intervention (EI) Study
March 2003 (continued)
Discussion
Conclusions
l

Within the limitations of this study, both file study data and interview data indicate that
clients situations had improved over the course of their contact with JF&CS.
Not only did clients situations improve, they improved in ways that are known to
correlate with positive outcomes for children.
The validity of the above conclusions is bolstered by the :
l

Random selection of the 30 study cases

Substantive congruence of file data with client data, thus mutually supporting each
other

Direct testimony of clients via anonymous interviews with an outside consultant,


especially their overwhelming perception that they were indeed helped by JF&CS.

The EI Study supports JF&CS Strategic Priority 1: To reaffirm the Agencys focus on
prevention and wellness. It further affirms the Agencys present position in that
direction, and opens possibilities for further studies in this area.
The Study can also contribute to JF&CS operational excellence. As pointed out above,
the role of direct service volunteers is inadequately recorded in Agency clinical files.
Since such volunteers have made, and still make, such an important contribution to the
families served, their activities must be better chronicled in case recordings.

Clients in Their Own Words


Without JF&CS help?
l

There is a good chance [without JF&CS intervention] that Id be a single parent and my child would have
missed something amazing...
...bad things would happen if JF & CS wasn't here....my kids might be F students....I walked in feeling I had
insurmountable problems and JF & CS gave me another way/voice to look at it....JF & CS keeps me
anchored somehow and it's not just about the money.

10
A Toolkit for Outcome Measurement

182

Example 10 A
Jewish Family and Child Service
Excerpts from Executive Summary Early Intervention (EI) Study March
2003 (continued)
l

I was suicidal, so I don't know.....Not many things in my life are perfect, this agency is one of the
perfect things in my life.

The experience at JF&CS, how it helped


l

...she [the social worker] doesnt do magic tricksits just the talking...

I feel better equipped to deal with problems, my worker has been a tremendous help.

I've had cancer, I was alone before with a child and no one was helping me, and then I got a worker and she is so
helpful.
...that I can talk about my financial situation and not feel like I'm groveling and to maintain my dignity as a human
being is just amazing.

JF&CS as a Jewish agency


l

They understand the need to maintain a Jewish lifestyle -- the need for Jewish education, to keep Shabbat or to attend
a Jewish camp, no one else would understand that.

10
183

A Toolkit for Outcome Measurement

Appendix

Glossary of Selected Terms


Terms Related to Outcome Measurement and the Program Logic Model
Outcome Measurement is a process of regular monitoring of the results of a
program for its participants or organization against agreed-upon goals and
objectives.
A Program Logic Model is a tool that supports program planning and outcome
measurement design by helping organizations recognize the relationship
between program activities and changes expected to occur as a result of these
activities. The relationship is typically illustrated by a flow chart. Components of
the program logic model include inputs, activities, outputs, outcomes
(immediate, intermediate, long-term) and indicators.
Inputs are resources dedicated to achieving program outcomes and are used in
the delivery of a service. Examples include: dollars, staff and staff time,
volunteers and volunteer time, facilities, equipment, supplies, ideas and
innovation.
Activities describe what an agency does (with its inputs), or what services it
provides to run a program that will help the agency meet its mission. Examples
include sheltering and feeding homeless families, educating the public about
child abuse, training, counselling and research.
Outputs are the direct products of program activities, usually measured in terms
of the volume of work accomplished (i.e., units of service). A program's outputs
should produce desired outcomes for the program's participants. Examples
include the number of meals provided, classes taught, brochures distributed or
participants served.
Outcomes are the benefits or changes for participants (or communities) during
or after their involvement with a program or service. Examples include new
knowledge, new skills, changed attitudes or values, improved condition, altered
status or modified behaviour. For a particular program, there can be various
levels of outcomes, with initial outcomes leading to longer-term ones. For
example, students who participate in an after-school homework support program
might initially complete a greater number of homework assignments, which can
lead to improved attitudes toward school, which can lead to grade level
performance or higher.

A Toolkit for Outcome Measurement

184

l
l

Immediate (Short-Term) Outcomes are the first benefits or changes


that participants experience.
Intermediate Outcomes link initial changes to longer-term outcomes.
Long-Term Outcomes are the ultimate outcomes desired for
participants; also referred to as impact.

Indicators are specific items of information that track a program's success.


They describe observable, measurable characteristics or changes that
represent achievement of an outcome and they help to define what
information must be collected to answer evaluation questions. For example,
if the desired outcome is that participants pursue a healthy lifestyle, healthy
lifestyle could be defined as: not smoking; maintaining a recommended
weight, blood pressure, and cholesterol level; getting at least two hours of
exercise each week; and wearing seat belts consistently. The number and
percent of program participants who demonstrate these behaviours are
indicators of how well the program is doing with respect to the desired
outcome.
Enablers and Constraints are environmental factors within a given situation
that may positively or negatively influence a program and its participants.
These factors can be found inside an agency as well as at a family,
community, national and even international level. Early identification of these
factors enhances an agencys ability to respond to, and manage them.
Understanding the environmental context of a program is important when
interpreting outcome measurement findings.
Assumptions and Risks. Assumptions refer to conditions that are
necessary for a program to succeed. Risks relate to the chance that these
conditions will not exist. The calculation of risk is often an intuitive exercise
based on good understanding of constraints and enablers.

Other Program Evaluation Language


as it Relates to Outcome Measurement
A Goal expresses the vision or purpose (long-term outcome) of a program,
stated as the intended benefits or changes for participants or the community
as a whole. A goal should specify one long-term outcome that is
measurable, and ideally should state a timeframe and target level for its
achievement. Example: Displaced workers will be employed in full-time
jobs that pay at least 90% of previous wage levels one year after they
complete career training.
Objectives define specific shorter-term outcomes that contribute to reaching
the goals (long-term outcomes) of a program. They can be conceptualized as
markers along the way to a goal. While goals are broad and may be achieved
over one or more years, objectives are clear, measurable and can be
achieved in much shorter time periods. Objectives may relate to service or
outcomes. Example: Displaced workers will complete up-to-date rsums
by six months into career training.

185 Appendix

Service objectives define the operational results expected from program


activities (i.e., outputs, units of service, service quality). Example: The
number of career training sessions offered to displaced workers during a
one-year period of time.
Outcome objectives define the specific benefits or improvements to be
expected by clients as a result of participating in a program. An outcome
objective, using the above example, is that displaced workers will be
employed on a full-time basis.

Performance Targets are projected numerical objectives for achievement of


program outcomes and are estimated by the agency based on past
experience or experience in a similar program. In the Rensellaerville
approach to program evaluation, performance targets are conceptualized as
being connected to outcomes, for example, less truancy in school may be a
specified performance target but higher test scores in school is the desired
outcome (where less truancy is directly linked to higher scores on tests).
l

Milestones are critical interim points that mark progression towards


achieving a long-term outcome or performance target. A series of
milestones are the essential steps that define a participants interaction
with a program. The Rensellaerville model of evaluation defines a
program as a progression of separable steps that must be achieved for a
performance target to be reached.

General Evaluation Terms


Benchmarks are performance data that are used for comparative purposes.
A program can use its own data as a baseline benchmark against which to
compare future performance or it can use data from another similar
program.
Effectiveness vs. Efficiency. Effectiveness refers to how competent a
program/agency is in achieving its mission or goals while efficiency relates to
the measurement of how much input is necessary to achieve the output.
An Evaluation encompasses activities that enable judgement of the worth
or merit of an evaluation object such as programs, policies, organizations,
products or individuals.
l

Formative Evaluations help with both the development of a project and


adjustments and improvements to the project as it is implemented.
Formative evaluations provide information related to how well the project
is designed, planned and operating. Process evaluation, an example of
formative evaluation, is used to determine if a program is reaching the
desired target group and if it is being delivered in the way it was planned.
Summative evaluation provides summary information about the
outcomes or results of a project after activities or services have been
operating for some time. A summative evaluation involves judgments of
cause and effect, in particular the effectiveness and worth of a projects
activities in improving the problems being addressed.

A Program is a specific service (or set of services) provided by an


organization. It includes all of the work necessary to bring about the
service(s).

A Toolkit for Outcome Measurement

186

Tracking is a process for counting inputs and outputs and determining the
relationship between them. For example, for every training workshop an
agency runs (input), they get 20 people participating (output).

Data Collection Terms


Case Studies are used to collect in-depth information about the experiences
of a program on a single participant or site. A case study can be simply the
story of one person's experience related to a program.
Focus Groups or group discussions are used for encouraging a group of
people to share their feelings and perceptions. Although led by a facilitator
with a discussion guide, these conversations often flow naturally and explore
unexpected or unintended areas. The qualitative analysis should focus on the
areas where participants' responses converged and the tone or feeling that
accompanied those responses. For instance, if participants felt very strongly
about an issue or if only some felt strongly, then this should be noted in the
analysis.
Questionnaires and Surveys: A questionnaire is a uniform written tool
completed by a group of respondents that is useful in gathering focused,
limited information (e.g., participants attitudes, behaviours, beliefs, and
activities). This information can be used to generalize about a population or
to get a total count of the aspects concerning a group of people. Questions
can range from being close-ended (for limited and easy-to-analyze
responses), to open-ended (with room for a variety and explanation of
responses). Questionnaires are sometimes referred to as surveys but
surveys are not necessarily completed by respondents in written form: A
survey can be in the form of an interview. Interviews are telephone or faceto-face discussions with an individual respondent or a group of people. They
provide the opportunity to probe and explore responses given, and can
provide a better quality of responses, though they contain less objective
information. Interviews are useful in collecting sensitive information that
respondents may not wish to write down. Focus groups are a type of group
interview.
Tests are useful for measuring knowledge or skills gained through
participation in a program. Program participants can be tested at the end, or
at the beginning and end, of a program to measure the change in their
knowledge or skills.
A Sample is the number of people in a population from whom you should
collect information in order to generalize about that population with some
degree of accuracy.
Random Sampling is a process by which the people in a sample are
chosen at random from a given population. For example, everyone in the
population of 100 people can be assigned a unique number, then the
numbers are put in a hat, and 40 numbers are drawn to choose 40 people to
be in that sample. In a random sample, all of the people in the population
have an equal chance of being chosen.
Qualitative Data record a thought, observation, opinion or words and

187

Appendix

typically come from asking open-ended questions to which the answers are
not limited by a set of choices or a scale. Examples of qualitative data
include answers to questions like How can the program be improved? or
What did you like best about your experience? but only if the user is not
restricted by a pre-selected set of answers. Qualitative data is best used for
questions that produce too many possible answers to list them all, or that
you would like people to answer in their own words. Qualitative data is more
time-consuming to analyze than quantitative data.
Quantitative Data captures information that is numeric and include
variables like personal income, amount of time or a rating of an opinion on a
scale from 15. Variables that we do not think of as quantitative, like
feelings, can be collected using numbers if we create scales to measure
them. Quantitative data are used with closed-ended questions where users
are given a limited set of possible answers. It is best for responses that fall
into a relatively narrow range of possible answers.
Participatory Evaluation allows for participants (subjects in traditional
evaluation and research conceptualizations) of programs to contribute
actively in all stages of the evaluation planning and implementation,
including determination of research objectives, research questions,
methodology, data collection, data analysis and reporting.

A Toolkit for Outcome Measurement

188

189

Appendix

A Toolkit for Outcome Measurement

190

References
Logic Models
Cooksy, L.J., Gill, O. and Kelly, P.A. (2001). The Program Logic Model as an
Integrative Framework for a Multimethod Evaluation. Evaluation and Program
Planning, 24, 119-128.
Den Heyer, M. (2002). The Temporal Logic Model Concept. Canadian Journal
of Program Evaluation, 17(2), 27-47.
Millar, A., Simeone, R.S. and Carnevale, J.T. (2001). Logic Models: A
Systems Tool for Performance Management. Evaluation and Program Planning,
24, 73-81.
Montague, S. (1997). The Three Rs of Performance. Ottawa: Performance
Management Network.
Poister, T.H. (2003). Measuring Performance in Public and Nonprofit Organizations. San
Francisco: Jossey-Bass.
United Way of America. (1996). Measuring Program Outcomes: A Practical Approach.
Arlington, VA: United Way of America.
W.K. Kellogg Foundation. (2001). Logic Model Development Guide.
[http://www.wkkf.org/pubs/tools/evaluation/pub3669.pdf]

Evaluation Frameworks
Porteous, N.L., Sheldrick, B. J. and Stewart, P. J. (1997). Program Evaluation Tool
Kit: A Blueprint for Public Health Management. Ottawa: Ottawa-Carleton Health
Department.

Planning and Managing Evaluations


Bell, J. (2004). "Managing Evaluation Projects." In J. S. Wholey, H. P. Hatry,
and K. E. Newcomer (Eds.), Handbook of Practical Program Evaluation (2nd ed.). San
Francisco: John Wiley, pp. 571-603.
Berk, R. A. and Rossi, P.H. (1998).Thinking about Program Evaluation (2nd ed)
Thousand Oaks, CA: Sage.
Hatry, H. P. (1999). Performance measurement: Getting results. Washington, DC: The
Urban Institute Press.
Lampkin, L. M. and Hatry, H. P. (2003). Key steps in outcome management.
Washington, DC: The Urban Institute Press.
Love, A. J. (1991). Internal Evaluation: Building Organizations from Within. Thousand
Oaks, CA: Sage.
Mayne, J. (2001). Addressing attribution through contribution analysis: Using
performance measures sensibly. The Canadian Journal of Program Evaluation 16
(1): 1-24.

191

References

Owen, J.M. and Rogers, P. (1999). Program Evaluation: Forms and Approaches
Thousand Oaks, CA: Sage.
Patton, M.Q. (1997). Utilization-Focused Evaluation: The New Century Text. Thousand
Oaks, CA: Sage.
Robson, C. (2000). Small-Scale Evaluation: Principles and Practice. Thousand Oaks,
CA: Sage.
Wong-Rieger, D. and David, L. (1994). A hands-on guide to planning and evaluation:
How to plan and evaluate programs in community-based organizations. Ottawa,ON:
Canadian Hemophilia Society

Questionnaires and Surveys


Aldridge, A. E. and Levine, K. (2001). Surveying the Social World : Principles and
Practice in Survey Research. Philadelphia, PA: Open University Press, 2001.
Behlin, O. and Law, K. S. (2000). Translating Questionnaires and Other Research
Instruments: Problems and Solutions. Thousand Oaks, CA: Sage.
Braverman, M. C. and Slater, J. K. (Eds.) (1996). Advances in Survey Research.
San Francisco: Jossey-Bass.
Dillman, D. (2000). Mail and Telephone Surveys: The Tailored Design Method. New
York: Wiley.
Fowler, F. J. (2001). Survey Research Methods (3nd ed.). Thousand Oaks, CA:
Sage.
Gillham, B. (2000). Developing a Questionnaire. New York: Continuum.
Groves, R. M., Dillman, D., Eltinge, J. L. and R. J. A. Little (Eds.) (2002).
Survey Nonresponse. New York: Wiley.
Hayes, B. E. (1998). Measuring Customer Satisfaction: Survey Design, Use, and Statistical
Analysis Methods (2nd ed.). Milwaukee, WI: ASQC Quality Press.
Lavrakas, P. J. (1993). Telephone Survey Methods: Sampling, Selection, and Supervision
(2nd ed.). Thousand Oaks, CA: Sage.
Menon, G. M. (Ed.) (2002). Using The Internet as a Research Tool for Social Work and
Human Services. New York: Haworth Press.
Porter, S. R. (Ed.) (2004). Overcoming Survey Research Problems. San Francisco:
Jossey-Bass.
Presser S. (Ed.) (2004). Methods for Testing and Evaluating Survey Questionnaires.
Hoboken, NJ: Wiley.
Punch, K. F. (2003). Survey Research:The Basics. London: Sage.
Statistics Canada (2003). Survey Methods and Practices. Ottawa: Statistics
Canada, Social Survey Methods Division.

A Toolkit for Outcome Measurement

192

Interviews
Gubrium, J. F. and Holstein, J. A. (Eds.) (2002). Handbook of Interview Research:
Context and Method. Thousand Oaks, CA: Sage.
Hersen M. and Turner, S. M. (2003). Diagnostic Interviewing (3rd ed.). New York:
Kluwer.
Holstein, J. A. and Gubrium, J. F. (Eds). (2003). Inside Interviewing: New Lenses,
New Concerns. Thousand Oaks, CA: Sage.
Houtkoop-Steenstra, H. (2000). Interaction and the Standardized Survey Interview: The
Living Questionnaire. Cambridge: Cambridge University Press.
Minichiello, V. (1995). In-Depth Interviewing: Principles, Techniques, Analysis (2nd ed.).
Melbourne: Longman.
Oppenheim, A. N. (2000). Questionnaire Design, Interviewing and Attitude
Measurement (new edition). New York: Continuum.
White, A. (2003). Interview Styles and Strategies. Cincinnati, OH: Thomson
Learning.

Focus Groups
Fern, E. F. (2001). Advanced Focus Group Research. Thousand Oaks, CA: Sage.
Goldenkoff, R. (2004). "Using Focus Groups." In J. S. Wholey, H. P. Hatry, and
K. E. Newcomer (Eds.), Handbook of Practical Program Evaluation (2nd ed.). San
Francisco: John Wiley, pp. 571-603.
Langer, J. (2001). The Mirrored Window : Focus Groups from a Moderator's Point of
View. Ithaca, NY: PMP.
Litosseliti, L. (2003). Using Focus Groups in Research. New York: Continuum.
Morgan, D. L. (1997). The Focus Group Kit (Vols. 1-6). Thousand Oaks, CA: Sage.
Morgan, D. L. (1997). Focus Groups as Qualitative Research (2nd ed.). Thousand
Oaks, CA: Sage.
Morgan, D. L. (Ed.) (1993). Successful Focus Groups: Advancing the State of the Art.
Thousand Oaks, CA: Sage.
Stewart, D.W. and Shamdasani, P.N. (1990). Focus Groups: Theory and Practice.
Thousand Oaks, CA: Sage.

Direct Observation
Ammons, D.N. (2001). Municipal Benchmarks (2nd ed.). Thousand Oaks, CA:
Sage.
Cohn, B.J. (1999). "Public-Minded Measurement." New Public Innovator.
Spring/Summer.
deMarrais, K. and Lapan S. D. (Eds.) (2004). Foundations for Research: Methods of
Inquiry in Education and the Social Sciences. Mahwah, NJ: Lawrence Erlbaum.

193

References

Feldman, M. S., Bell, J., Berger, M. T. (2003). Gaining Access: A Practical and
Theoretical Guide for Qualitative Researchers. Walnut Creek, CA : Altamira Press.
Galavotti, M. C. (Ed.) (2003). Observation and Experiment in the Natural and Social
Sciences. Boston: Kluwer.
Hatry, H.P. and others (1992). How Effective Are Your Community Services? Procedures
for Measuring Their Quality (2nd ed.). Washington, D.C.: Urban Institute.
Hutt, S. J. and Hutt, C. (1970). Direct Observation and Measurement of Behavior.
Springfield, IL: C. C. Thomas.
Janesick, V. J. (2004). "Stretching" Exercises for Qualitative Researchers (2nd ed.).
Thousand Oaks, CA: Sage.
Marion, M., (2003). Using Observation in Early Childhood Education. Upper Saddle
River, NJ: Pearson/Merrill-Prentice Hall.
Webb, E. J., Campbell, D. T., Schwartz, R. D., Sechrest, L. (1999). Unobtrusive
Measures (new edition). Thousand Oaks, CA: Sage.

Using Agency and Program Records


Elder, G.H., Jr., Pavalko, E.K. and Clipp, E.C. (1993). Working with Archival Data:
Studying Lives. Thousand Oaks, CA: Sage.
Love, A. J. (1991). Internal Evaluation: Building Organizations from Within. Thousand
Oaks, CA: Sage.
Rossi, P.H., Freeman, H.E. and Lipsey, M.W. (2004). Evaluation: A Systematic
Approach (7th ed.). Thousand Oaks, CA: Sage.
Stewart D. W. and Kamins M. A. (1992). Secondary Research: Information Sources
and Methods (2nd ed.). Thousand Oaks, CA: Sage.

Case Studies
Gummesson, E. (2000). Qualitative Methods In Management Research (2nd ed.).
Mertens, D. M. (1998). Research Methods in Education and Psychology: Integrating
Diversity with Quantitative and Qualitative Approaches. Thousand Oaks, CA: Sage.
Stake, R. E. (1995). The Art of Case Study Research. Thousand Oaks, CA: Sage.
Thomas, A. B. (2004). Research Skills for Management Studies. New York:
Routledge.
Yin, R. K. (2003). Case Study Research: Design and Methods (3rd ed.). Thousand
Oaks, CA: Sage.
Yin, R. K. (2002). Applications of Case Study Research (2nd ed.). Thousand Oaks,
CA: Sage.

A Toolkit for Outcome Measurement

194

Analyzing Data and Presenting Findings


Adelheid, A., Nicol, A. M. and Pexman, P. M. (1999). Presenting Your Findings: A
Practical Guide for Creating Tables. Washington, DC : American Psychological
Association.
Bernstein, I. H. and Rowe, N. A. (2001). Statistical Data Analysis Using Your Personal
Computer. Thousand Oaks, CA: Sage.
Branch, R.C. (1996). Graphing Data: Techniques for Display and Analysis.
Evaluation and Program Planning,19 (1), pp. 103-105
Henry, G. T. (1997). Creating Effective Graphs: Solutions for a Variety of
Evaluation Data. New Directions for Evaluation. no. 73. San Francisco: JosseyBass.
Levin, H. M., McEwan, P. J. (2001). Cost-Effectiveness Analysis (2nd ed.). Thousand
Oaks, CA: Sage.
Lipsey, M. W. and Wilson, D. B. (2000). Practical Meta-Analysis. Thousand Oaks,
CA: Sage.
Patton, M. Q. (2001). Qualitative Research and Evaluation Methods (3rd ed.).
Thousand Oaks, CA: Sage.
Poister, T.H. (2003). Measuring Performance in Public and Nonprofit
Organizations. San Francisco: Jossey-Bass.
Schalock, R. L. and Bonham, G. S. (2003). Measuring Outcomes and
Managing for Results. Evaluation and Program Planning. 26 (3), pp. 229-235.
Torres, R. T., Preskill, H. and Piontek, M. E. (2004). Evaluation Strategies for
Communicating and Reporting: Enhancing Learning in Organizations (2nd ed.). Thousand
Oaks, CA: Sage.

195

References

A Toolkit for Outcome Measurement

196

United Way of Greater Toronto


26 Wellington St E 11th Floor
Toronto ON M5E 1W9

T: (416) 777-2001
F: (416) 777-0962
www.unitedwaytoronto.com