Logic Models & Performance

Measures: Telling Your Performance

Story

Sustainable Communities Conference

March 11, 2009 1

Presented by:
Yvonne M. Watson
U.S. Environmental Protection Agency
National Center for Environmental Innovation
Evaluation Support Division
You Will Learn How to:

• Develop a logic model of your program/project

• Use the logic model to develop a results

oriented work plan

2
• Identify performance measures

• Use the logic model to respond to

Environmental Results Order 5700.7
How does it all fit together?

Logic models
Environmental Results Order

Work plans Measures: Outputs

& Outcomes
Environmental Results Order 5700.7

 Environmental Results Order (EPA

Order 5700.7)
• Requires EPA grant officers and
grant recipients to identify outputs
and outcomes from grants and 4

connect them to EPA‘s Strategic

Plan.
• Learning and continuous program
improvement
EPA Order 5700.7

 Addresses environmental results in 3

stages of the assistance process:
• Funding Announcements (application/
funding process)

• Work Plan Development (containing 5

anticipated outputs & outcomes)

• Performance Reporting (including results

& performance measures)
Performance Management Tools

PERFORMANCE MANAGEMENT
Performance management includes activities to ensure that goals are consistently
being met in an effective and efficient manner. Performance management tools
include logic models, performance measurement and program evaluation.

Logic Model Performance Program 6

Measurement Evaluation
Tool/framework that
helps identify the Helps you Helps you
program/project understand what understand and
resources, level of explain why you‘re
activities, outputs performance is seeing the
customers, and achieved by the program/project
outcomes. program/project. results.
 Module 1:
Picturing Your Program: Developing a
Logic Model

7
8
 What is a logic model?

A picture of your program. Graphic and text

that illustrates the relationship between
your program’s activities and its intended
outcomes and results.
9

To So that these
We use produce customers can Which leads Leading
these For these to these to these
these change their
resources… activities… outcomes…
outputs… ways… results!
 Logic Model

HOW WHY

Longer term
Resources/ Activities Outputs Customers Short term Intermediate outcome
Inputs outcome outcome (STRATEGIC AIM)

10

PROGRAM RESULTS FROM

PROGRAM

EXTERNAL CONDITIONS
INFLUENCING PERFORMANCE (+/-)
Courtney and Bailey Peter’s Model:
A Safe Place to Play
 Lead A Great Life
Program Areas Activities and processes Short-term goals Long-term goals
(Impact Measures) (Outcome Measures)
• VACUUM AT LEAST ONCE A
BY 2009 IMPROVED
MONTH
REDUCED # OF ASTHMA SELF-ESTEM
• FIND A PLACE TO PUT MY ATTACKS FROM DUST MITES (PSYCHOLOGICAL
PERSONAL JUNK TEST)
• GO GROCERY SHOPPING DECREASED
WEEKLY DISTRACTEDNESS
INCREASE # OF DAYS WHEN (PSYCHOLOGICAL
• PLAN MEALS 5 FRUITS AND VEGETABLES TEST)
• MAINTAIN A FOOD DIARY ARE CONSUMED

Lead
• BRUSH AND FLOSS AFTER
a
EVERY MEAL
INTER- • USE OIL OF OLAY
BY 2009, INCREASE BY 200%
THE NUMBER OF DATES PER
BY 2013 HAVE
BIRTH
great
CERTIFICATES
PERSONAL • SKIN TREATMENTS YEAR
FOR TWO life
• IDENTIFY NEW WAYS TO CHILDREN
MEET PEOPLE

• TAKE GUITAR LESSONS

AND PRACTICE DAILY BY 2013, INCREASE BY 300% BY 2025 BECOME A
THE NUMBER OF FEATURED FAMOUS ROCK
• WRITE 2 NEW SONGS PER
HOBBIES PERFORMANCES PER YEAR STAR (WIN A
MONTH
GRAMMY)
• FORM A BAND
 Elements of the Logic Model

HOW WHY
Outcomes

Resources/ Activities Outputs Customer Short-term Inter- Long-

Inputs: mediate term

Programmatic Things you do– Product or User of the Changes in Changes in Change in
investments activities you service delivery/ products/ learning, behavior, condition.
available to plan to conduct implementation services. Target knowledge, practice or
support the in your program. targets you aim audience the attitude, skills, decisions.
program. to produce. program is understanding.
designed to Behavior Condition
reach. Attitudes

PROGRAM RESULTS FROM PROGRAM

External Influences
Factors outside of your control (positive or negative) that may influence the
outcome and impact of your program/project.
 Outputs and Outcomes

 Output: Products and services provided as a

direct result of program/proposal activities.

 Outcome: Changes or benefits resulting

from activities and outputs. Accomplishment
of program goals and objectives. 14

• short-term (Change in knowledge, skills,

understanding, attitude)
• intermediate outcomes (Change in behavior)
• long-term outcomes—impacts (Change in the
environment)
 CASE STUDY: Green Taxis Program

The goal of the Green Taxi Program is to facilitate replacement of existing taxis
with very low emitting vehicles. The program will also provide education on the
additional benefits of green taxis and the suitability of these vehicles to meet
industry needs.
 Given their high mileage  Define replacement vehicle
(estimated between 60,000 and requirements (required emission
80,000 miles per year) and the standard, technology, etc.)
congested driving conditions in
which they operate, the local fleet  Provide a rental car opportunity to
of over 2,000 taxis is a significant lower cost of replacement vehicle
source of potential emission 15
reductions  Develop non-financial incentives
(such as preferential queuing) to
 Quantifiable emission reductions encourage purchase of cleaner
will be achieved by sequential vehicles
replacement of local taxis over the
three-year implementation period  Develop marketing and outreach
program
 Develop public-private
relationships with taxi companies,  Analyze program efficacy and
rental car companies and financial emissions reductions and prepare a
institutions to successfully final report
implement the program
 Exercise 1: Types of Program
Elements
Example Type of Program Element
1. Increased awareness of
incentives and eligibility 1. _________________
criteria

2. Develop marketing and 2. _________________

outreach plan

3. City of Dallas, EPA Region 6, 3. _________________

16
TCEQ Staff

4. Green Taxi Brand 4. _________________

5. Improved air quality

5. __________________
What are Logic Models Used For?

 Planning tool

 Communication tool

 Implementation tool 17

 Measurement design

 Evaluation design
What are the Benefits of Logic Models?
 Illustrates the logic or theory of the
program or project.
 Focuses attention on the most important
connections between actions and results.
 Builds a common understanding among
staff and with stakeholders. 18

 Helps staff ―manage for results‖ and

informs program design.
 Finds ―gaps‖ in the logic of a program and
work to resolve them.
How Do You Develop a Logic
Model?
1. Clarify the program goal and define the
elements of the program in a table.

2. Verify the logic table with stakeholders.

3. Develop a diagram and text describing

logical relationships.
19
4. Verify the Logic Model with stakeholders.

Then use the Logic Model to identify and confirm

performance measures and in planning and evaluation.
Step 1. Clarify the program goal and define the
elements of the program in a table
WHAT and WHY
- HOW - WHO Outcomes
Resources/ Customers Short-term Intermediate Long-term
Inputs Activities Outputs
reached (change in attitude) (Change in behavior) (change in condition)

20
Program
outcomes
related to
factor(s)

External Influences:
 Green Taxi Program: Outreach and
Marketing
Resources Activities Outputs Customer Short-term Intermediate Long-term
reached Outcome Outcome Outcomes
FTE Solicit 5 – 6 Green taxis Taxi Increased Pilot program Improved air
taxis for pilot in operation companies visibility of operational quality
Dallas project green tax
Logic Model Elements

Local Rental Car program Increased Reduction in

EPA R6 Develop corporate agencies market fleet fuel use
initial commitments Improved penetration of
OAQPS Local
commitments to using corporate green taxis
green taxis corporations attitudes
OTAQ
Identify about using Accelerated
marketing Green Taxi Public fleet turnover
TCEQ green taxis
campaign Program interest
$$ brand groups Awareness
Develop of financial
materials Marketing & Airport Police 21
& non-
outreach plan financial
Develop Hotels
incentives,
Green taxi
Local media eligibility
brand
criteria
General
Identify
public
outreach
opportunities

Develop
marketing &
outreach plan
External Influences: Lack of interest from taxi companies.
Step 2. Verify the logic with stakeholders

 Seek review from stakeholders.

 Check the logic

• How-Why Questions. Start with Outcomes and ask

―How?‖ Start at Activities, ask ―Why?‖
• If-Then Questions. Start at Activities and move along
to Outcomes asking ―If this, then that?‖ 22

 Compare to what units in the organization do and

define their contributions to the outcomes.

 Check the logic by checking it against reality.

 Step 3. Develop a diagram and text
describing logical relationships

 Draw arrows to indicate/link the causal

relationships between the logic model
elements.
23

We use To produce So that these Which leads Leading

For these
these these customers can to these to these
activities…
resources… outputs… change their ways… outcomes… results!
Example Logic Models…

 There are many different forms of logic

models. Can you see the logic?
 Key questions to ask:
• Are the program‘s outcomes described?
24
• Are the program‘s customers described?
• Are the program‘s major resources, activities and
outputs described and do they make sense?
• Are there things/issues that might influence the
program‘s ability to achieve its goal?
 GREEN TAXIS PROGRAM

Program Goal: The goal of the Green Taxi Program is to facilitate replacement of existing taxis with very low emitting vehicles. The program will
also provide education on the additional benefits of green taxis and the suitability of these vehicles to meet industry needs.

Outcomes

Resources Activities Outputs Customers Short-term Intermediate Long-term

•Dallas
Solicit 5 – 6 Taxis Several Green •Taxi Increased Pilot
FTE •Reduction
for pilot project Taxis in operation companies visibility of program
•EPA green taxi operational in fleet fuel
•Rental car use
Region 6 program
agencies
FTE
•Local •Improved
•OAQPS Develop initial Local corporate air quality
corporations Increased
FTE commitments commitments to Improved
using Green •Public corporate
market
•OTAQ penetration
Taxis interest attitudes
FTE of green
groups about using
•TCEQ taxis
•Airport Police green taxis
FTE Marketing Strategy •Green Taxis •Hotels
•Develop marketing brand
materials •Local media
•Marketing and Awareness Accelerated
•Develop Green outreach plan •General of financial & fleet
Taxi brand public non-financial turnover
incentives,
•Identify outreach eligibility
opportunities criteria
•Develop marketing
and outreach plan
 Oregon’s Toxic Use Waste Reduction Assistance Program
Goals: Protect public health, safety, and the environment from the risks associated with using toxic chemicals and generating
hazardous waste; improve regulatory compliance; and reduce management and disposal costs.

Knowledge Behavioral Environmental

Activities Outputs Customer Outcomes
Outcomes Outcomes
• Assist with • Site visits and • Large Quantity • Regulated • Regulated entities • Regulated
preparation and follow-up Generators of entities increase implement entities use
implementation visits hazardous understanding of Reduction Plan fewer toxic
of Reduction waste regulations and recommendations chemicals and
Plans and • Recommen- what it will take generate less
• Small Quantity • Coordination hazardous
meeting dations to achieve increases between
Generators of waste
regulatory regulatory trade associations,
hazardous
requirements • Training compliance local colleges, • Regulated
waste
through site seminars and entities safely
• Regulated regulated entities,
visits and workshops • Conditionally- and local dispose of toxic
entities increase
follow-up visits, Exempts governments chemicals and
understanding of
training • Published Generators of hazardous
―beyond • Regulated entities
seminars and information hazardous waste
compliance‖ more safely
workshops, and outreach waste
options manage hazardous • Regulated
fielded phone materials
• Reporters to waste and toxic entities reduce
calls,
the Toxic chemicals during the severity of
educational • Fielded
Release storage and toxic and
materials, etc. phone calls
Inventory transportation hazardous
• Public spills
• Regulated entities
Resources (FY meetings improve
03) compliance and
• 6.6 FTE ―beyond
compliance‖ rates
• $ 0.52 million
 EPA REGION VII SUSTAINABILITY STRATEGIC PRIORITY
Goal #1: The Region VII staff and on-site grantees, contactors, and volunteers will embrace the principles, and engage in the practices of, environmental stewardship and
sustainability (as depicted in our EMS) at organizational and personal levels.
Outcomes

Resources Activities Outputs Customers Short-term Intermediate Long-term

Sustainability • Establish/Deliver Personal Personal Sustainability Knowledge of Region VII Staff Only E85 Fuel is
& Sustainability Action Plan Action Plan Sustainability and On-Site used in Equipped
Stewardship Practices and Grantees Increase Government
• Establish/Deliver
Team and Sustainability Standards Principles Increases Use of E85 Fuel in Vehicles
Sustainability Standards for
other Staff for External Programs Equipped
External Programs
Government
Senior Staff
• Provide Input to EMS Vehicles
(Approval) Consultation Documents Region VII Staff and
Sustainability EMP
On-site Grantee, Region VII Staff Zero Discharge of
Contractor, and and On-site Recyclable and/or
Volunteer Grantees, Reusable Materials
• Develop/Provide Training Personal Sustainability Region VII Staff, Understanding of Contractors, and to Landfills
on Personal Sustainability Training Program On-site Grantees, How Sustainability Volunteers Increase
Contractors, and Applies to Region use of P2/
• Develop/Provide Training Volunteers VII Activities Reuse/Recycling
on Sustainability Standards Sustainability Standards Increases Opportunities
for External Programs Training Program
Region VII Staff and Region VII Staff 100% of
On-site Grantee, Increase EPP Appropriate
Contractor, and Purchases as a Purchases are EPP
TO GOAL #2 Volunteer Percentage of All
Understanding of Purchases
EMS stablish ad hoc
How Sustainability
Sustainment Sustainability Teams
Applies to Personal Region VII Staff; 50% Reduction in
Team • Alt Fuel Vehicles
Ad hoc Teams Decisions and On-site Grantees, Water and
• Energy Conservation
Region VII Established Activities Increases Contractors, and Electricity Use at
• Water Conservation
Staff Volunteers; and Each Region VII
• Env Pref Purchasing
Senior Staff • P2/Recycling/Reuse EMS External Region VII Facility
(Approval) • Commuting Alternatives External EMS Stakeholder Facilities Use Less
Stakeholders Understanding of Water and
(Contractors; How Sustainability Electricity
Establish EMS Local Citizens; Applies to Their
Sustainability Objectives & Sustainability Objectives Local Federal Activities Increases Region VII Staff 50% of Region VII
Targets & Targets Agencies; HQ and On-site Staff and On-Site
EPA; City, State Grantees, Grantees,
& Tribal Contractors, and Contractors, and
Governments) Volunteers Increase Volunteers Car
Establish EMS TO GOAL #2 Use of Car Pooling Pool or Use Mass
Sustainability
Sustainability EMP and Mass Transit Transit for
Environmental Management
Program Commuting
 RCRA Recycling Regulations that Foster Recycling of Hazardous Waste
Resources Activities Outputs Customers

Staff Promulgate Rules

Rulemaking Generators
$ Recyclers
Regions States
Policy Guidance
States
Interpretation Letters
Trade Associations
Individual Companies DSW Network
OGC/ OECA Outreach States/Regions
Calls & Website

Outcomes

Short-Term Long-Term

Exclusion is Awareness
available Regions by industry
Awareness authorize Less haz Better
Relevant states Increased
of Rule states waste env’t
pick up rule Favorable recycling
(states & for rule &
change in (desire to
industry) healthy
recycling recycle and
Firm(s) lobby Conserve
costs capacity) public
state to pick resources
up rule
Change in
stigma
External: market forces (T&D costs), number of recyclers, markets for recycled products, technology,
bureaucratic process, politics, state budgets, PR, public goodwill
 ESD TRAINING LOGIC MODEL

ESD Training Goal: To provide training to enable our EPA partners to more effectively conduct and manage program evaluations and analyses and develop performance
measures that can be used to improve their programs and demonstrate environmental results.
Outcomes

Resources Activities Outputs Customers Short- Intermediate Long- Strategic

term term Plan
NCEI Staff
IAC Staff Knowledge of PE
PE training materials. increased/ • PE skills are used Quality of Customers use
ESD Staff: PEN by customers in the evaluations program
Customers complete improved.
Y. Watson work environment managed and evaluation
training. PEC Customers
Develop and design PE, conducted is regularly and
M. Mandolia Winners equipped with • # of evaluations
PM, IA and Logic Model improved. systematically
skills to manage conducted and
J. curriculum and exercises. HQ/ to improve
PM training materials. and conduct managed increased.
Heffelfinger Regional environmental
Customers complete managers & evaluations. programs in
M. Keene Deliver PE, PM, IA and training. staff terms of:
B. Johnson Logic Model training. - environmental
Knowledge of • PM skills are Quality of & health
Access to: NCEI Staff PM increased/ used by customers measures outcomes
John IA training materials. SIG improved. in the work developed and - reduced costs
McLaughlin Recipients environment. reported is - cost effective-
Customers complete Customers
improved. ness
$65,000 training. HQ/ Regional equipped with • # of staff
- EJ Benefits
managers & staff skills to develop developing
-Public
measures. measures is
Logic Model training States/Tribes Involvement
increased.
materials. - Efficiency
SBAP
Customers complete Knowledge of
CARE Logic modeling
training.
increased/ Customers use
logic models to Environ-
Provide technical assistance improved.
help conduct mental
for workshop/ training Technical assistance Customers programs
delivered. NCEI Staff evaluations and
attendees. equipped with more
develop measures.
SIG skills to develop effectively
Recipients logic models of achieve
States their programs. their
Partners • Customers Partners deliver
strategic
Facilitate Train the trainer Partners complete SBAP goals.
understanding of PE, PM and Logic
sessions for PE, PM and Logic training. CARE their programs is model training to
OCFO Modeling. improved. their clients/
OW customers.
OSWER OCFO, OW, OSWER, ORD, OARM
ORD
Provide guidance for
OARM Environmental Results Grants EPA Project EPA POs & GMs
Environmental Results Officers & recognize outputs/
Training. Grants Training materials. Grant outcomes in grant
Managers proposals
What is ―Z‖ Logic?

A Outputs

Resources Action A

B
Outputs
Resources Action B
30

C Strategic
Outcomes Program
Resources Action C Results
 Energy R,D,&D Program Using ‗Z‘ Logic
For Short-term Intermediate Long-term
Resources Activities Outputs Customers outcomes outcomes outcomes

Ideas for Leads to Potential for

Program Perform For industry
technology applications technology
$, Staff research researchers
change in energy change
technologies documented

Lab Leads to Technology

Added Develop For users and
prototype commercial available for
resources technology Manufacturers
report prototype commercialization

Leads to
Policies, knowledge, Early adopters 31
Added Deploy For users and
incentives, less risk express desire
resources technology Manufacturers
information perceived to buy
(Shared
responsibility)
Produce Manufacture Leads to Consequences Competitive
technology the For buyers technology of use- Lower economy,
Commercial
& educate technology in of that accepted, energy costs cleaner
$, Staff
market market technology purchased and emissions environment

External Influences: Price of oil and electricity, economic growth in industry and in general, perception of
risk of global climate change and need for national energy security, market and technology assumptions.

Source: McLaughlin and Jordan, 1999

 Two Important Rules to Follow

 For every action identified in the Logic Model,

the must be an output that connects to an
outcome through a specific customer.

OR 32

 An action must produce an output that

becomes a key input to another activity.

THINK CONNECTIONS!
 Exercise 2: Logic Modeling

Developing your own logic model

33
 Module 2:

Logic Models and Work Plan

Development
34
Mapping out Your Work Plan

 What is the problem or need you are addressing?

• What are your planned activities to address this need?

• What resources will you need to do these activities?

35
• What are your anticipated accomplishments/outputs of
your activities?

Resources/ Activities Outputs

Inputs
Mapping out your project plan

 Who do you expect to act as a result?

 What do you expect them to do?

36
 What benefits (e.g., environmental, human
health) do you expect to result from these
actions?

Short term Intermediate Longer term

Customers outcome
outcome outcome
Short-term Outcomes
Example IGAP Workplan Component 1
Intermediate Outcomes

Long-term Outcomes

Activities

Resources/Inputs
Outputs
 The Logic Model and Work Plan
Development: Example IGAP Workplan

HOW WHY
Outcomes

Resources/ Activities Outputs Customer Short-term Inter- Long-

Inputs: mediate term
Maintain Research in Staff Increase staff,
environmental response to Council, & Increase in Increase in
$15,000 staff, Council
office on behalf Tribal Council & Tribal Council community Tribal
of Tribal Council Tribal members‘ knowledge of & community program-
Environmental actions matic
Coordinator w/ FT environmental Community environmental capacity to
Coordinator & questions and issues and protective of
(EC) the effectively
PT concerns protection address
Tribal Environmental environment environ-
Technician & mental 38
Administrator issues
Staff

PROGRAM RESULTS FROM PROGRAM

External Influences
Factors outside of your control (positive or negative) that may influence the
outcome and impact of your program/project.
 Module 2:
Identifying and Developing
Performance Measures 39
Performance Management Tools

PERFORMANCE MANAGEMENT
Performance management includes activities to ensure that goals are consistently
being met in an effective and efficient manner. Performance management tools
include logic models, performance measurement and program evaluation.

Logic Model Performance Program 40

Measurement Evaluation
Tool/framework that
helps identify the Helps you Helps you
program/project understand what understand and
resources, level of explain why you‘re
activities, outputs performance is seeing the
customers, and achieved by the program/project
outcomes. program/project. results.
Definitions:

Performance Measurement:
The ongoing monitoring and reporting of program
progress and accomplishments, using pre-selected
performance measures.

Performance Measure:
A metric used to gauge program or project 41
performance.

Indicators:
Measures, usually quantitative, that provide
information on program performance and evidence
of a change in the ―state or condition‖ in the
system.
Performance Measurement
Questions
 What are they?
• Questions designed to assess
progress/ accomplishments of various
aspects of a program/project.
42

• Performance measurement questions

ask/tell you what your program is doing.
 Performance Questions Across the
Performance Spectrum
PROGRAM Resources Activities/ Target Short term Intermediate Long-Term
ELEMENTS: Customer Outcome Outcome Outcome
(We use Outputs
these) (For these (To change (So they can (Which leads
(To do these people) them in these do these to these
things) ways) things) outcomes)
PERFORMANCE Do we have Are we Are we Did the Are What
QUESTIONS: enough, doing things reaching customer‘s customers changes in
the way we the attitude, using the condition
The right, say we customers knowledge, change as have
should? targeted? skills or expected? occurred?
The
understanding With what
necessary Are we Are we Did the
change? results?
level, producing reaching program
products and the Are achieve its
The 43
services at anticipated customers goals and
consistency?
the levels numbers? served objectives?
anticipated? changing in
Are they the expected
According satisfied? direction and
to level?
anticipated
quality If so, what
indicators did we
measures? (others) do
to cause the
change?

EXTERNAL What factors might influence my program‘s success?

INFLUENCES:
Measures Across the Logic Model
Spectrum
Element Definition Example Measure
Resources/ Measure of resources consumed by Amount of funds, # of FTE, materials,
Inputs the organization. equipment, supplies (etc.).

Activities Measure of work performed that # of training classes offered as

directly produces the core products designed; Hours of technical
and services. assistance training for staff.

Outputs Measure of products and services # of technical assistance requests

provided as a direct result of program responded to; # of compliance
activities. workbooks developed/delivered.

Customer Measure of target population receiving % of target population trained; # of 44

Reached outputs. target population receiving technical
assistance.

Customer Measure of satisfaction with outputs. % of customers dissatisfied with

Satisfaction training; % of customers ―very
satisfied‖ with assistance received.

Outcomes Accomplishment of program goals % increase in industry‘s understanding

and objectives (short-term and of regulatory recycling exclusion; # of
intermediate outcomes, long-term sectors that adopt regulatory recycling
outcomes--impacts). exclusion; % increase in materials
recycled.
Work Load/Quality Measures
Category Definition Examples
Efficiency Measure that relates outputs to Cost per workbook
costs. produced; cost per
inspection conducted.

Productivity Measure of the rate of production Number of enforcement

per some specific unit of resource cases investigated per
(e.g., staff or employee). The focus inspector.
is on labor productivity.

Cost Measure that relates outcomes to Cost per pounds of 45

costs. pollutants reduced; cost
Effectiveness
per mile of beach cleaned.

Service Measure of the quality of products Percent of technical

and services produced. assistance requests
Quality
responded to within one
week.
 Exercise 3: Types of Performance
Measures
Example Type of Measure
1. Total $$ invested per year 1. ________________
2. Pounds of air emissions
reduced
2. ________________
3. Number of taxis participating
in the pilot program

4. Number of hits and brochure 3. ________________ 46

downloads on Green Taxis
Website

5. Number and percent of Taxi 4. ________________

companies aware of Green
Taxis Program
5. _________________
Steps for Developing Measures

 Step 1: Identify Potential Measures

 Step 2: Assess Each Measure

47

 Step 3: Choose the Best Measures

 Step 4: Identify Baseline, Target, Timeline and

Reporting Schedule
Key Steps in Identifying Potential
Measures

STEP 1: Identify the information needed and the

audience
 Identify measures in existing documents
 Review the logic model and select the
appropriate logic model element
48
 Express the logic model element as a
performance measure
 Determine if the measure clearly relates to the
program/project goal or objective
STEP 1: Identify the information
needed and the audience
 Review the performance measurement
questions developed earlier

 Consider what information is needed to

assess whether your program/project is
meeting its goals and objectives.
49

Ask yourself:

 Who needs to know what about the

program, why, and in what format?
Identify Measures in Existing
Documents
 Review measures specified in:
• Program/Project Mission, Goals, Objectives,
Service standards

• Legislation, Strategic plans (GPRA), Court

Orders, PART, Regional Plans, National
Program Management Guidance, Regional 50
Priority Commitments

• Previous evaluations and research reports

• Consider other sources

Review the Logic Model

 Review the logic model –

• Identify the aspects of performance that are
most important to measure (resources,
activities, outputs, outcomes)

• Identify contextual factors that could influence 51

the program either positively or negatively and

generate measures for them as appropriate
 GREEN TAXIS PROGRAM

Program Goal: The goal of the Green Taxi Program is to facilitate replacement of existing taxis with very low emitting vehicles. The program will
also provide education on the additional benefits of green taxis and the suitability of these vehicles to meet industry needs.

Outcomes

Resources Activities Outputs Customers Short-term Intermediate Long-term

•Dallas
Solicit 5 – 6 Taxis Several Green •Taxi Increased Pilot
FTE •Reduction
for pilot project Taxis in operation companies visibility of program
•EPA green taxi operational in fleet fuel
•Rental car use
Region 6 program
agencies
FTE
•Local •Improved
•OAQPS Develop initial Local corporate air quality
corporations Increased
FTE commitments commitments to Improved
using Green •Public corporate
market
•OTAQ penetration
Taxis interest attitudes
FTE of green
groups about using
•TCEQ taxis
•Airport Police green taxis
FTE Marketing Strategy •Green Taxis •Hotels
•Develop marketing brand
materials •Local media
•Marketing and Awareness Accelerated
•Develop Green outreach plan •General of financial & fleet
Taxi brand public non-financial turnover
incentives,
•Identify outreach eligibility
opportunities criteria
•Develop marketing
and outreach plan
Express the Logic Model element
as a performance measure
 Consider how to express the measure in
terms of:
• Data:
– Raw Numbers (tons of VOCs reduced)
– Averages (mean tons of VOCs reduced)
– Percentages (% of dry cleaners reporting 53
VOC reduction)
– Ratios (Cost per ton of VOCs reduced)
– Rates (tons of VOCs reduced per 100 dry
cleaners)
• Unit of Measure:
– Is it appropriate to the measure?
 Green Taxi Program: Performance
Measures
Resources Activities Outputs Customer Short-term Intermediate Long-term
reached Outcome Outcome Outcomes
FTE Solicit 5 – 6 Green taxis in Taxi Increased Pilot Improved air
taxis for pilot operation companies visibility of program quality
Dallas project green tax operational
Logic Model Elements

Local Rental Car program Reduction in

EPA R6 Develop initial corporate agencies Increased fleet fuel use
commitments commitments Improved market
OAQPS Local
to using green corporate penetration of
Identify taxis corporations attitudes about green taxis
OTAQ
marketing using green
campaign Green Taxi Public interest Accelerated
TCEQ taxis
Program brand groups fleet turnover
$$ Develop Awareness of
materials Marketing & Airport Police
financial & non-
outreach plan financial
Develop Hotels
Green taxi
incentives, 54
Local media eligibility
brand
criteria
General public

 # Staff Delivery # taxis in Actual vs. #/, % with # of green  Pounds of air
Example Measures

timetable on pilot program desired increased taxis emissions

$$ Invested schedule participation awareness of reduced
# local by customer program # of green
corporate rental cars % reduction in
commitments fuel
#local consumption
corporations
participating
Determine whether the measures
clearly relate to the mission/goal
 Review the program/project mission and or goal
• What key activities, outputs or outcomes are specified in the
mission or goal?

 Review the list of potential measures developed

• Will the data collected from the measures developed
55
clearly demonstrate that the mission and or goal was
accomplished?
Determine whether the measures
clearly relate to the mission/goal
Green Taxis Program Goal: The goal of the Green Taxi
Program is to facilitate replacement of existing taxis with very
low emitting vehicles. The program will also provide education
on the additional benefits of green taxis and the suitability of
these vehicles to meet industry needs.

Performance Measures:
 Number of staff
 Total dollars invested
 Delivery timetable on schedule
56
 Number of taxis in pilot program
 Number of local corporate commitments
 Actual vs. desired participation by customer
 Number and percent with increased awareness of program
 Number of green taxis and green rental cars
 Number of local corporations participating
 Pounds of air emissions reduced
 Percent reduction in fuel consumption
Step 2: Assess the Measures

 Assess the value of the measures in

relation to goals and objectives
 Assess the feasibility of the measure in
terms of:
• Data collection (availability, implementation 57
cost, baselining)
• Data quality (reliability, validity, objectivity)
• Analysis
• Reporting (how to report, to whom to report,
frequency of reporting, meaningfulness to
audiences)
Step 3: Choose the Best Measures

 Assess the value of the measures in relation to

the goals and objectives of the program.
• Required
• Important
• Interesting

 Select final list of measures – you won‘t be able

58
to collect data for all measures.

 Check in with managers and stakeholders.

 Identify a priority list of measures

Step 4: Identify a Standard

For each performance measure

develop a:
1. Baseline – current state
2. Target – desired level of performance
3. Timeline – date when performance 59

will be achieved
Tips for Choosing the Best
Measures
For each measure ask…
 Does the measure clearly relate to the project goal
and objective?
 Is the measure important to management and
stakeholders?
 Is it possible to collect accurate and reliable data for
the measure? 60

 Taken together, do the measures accurately reflect

the key results of the program, activity or service?
 Is there more than one measure for each goal or
objective?
 Are your measures primarily outcome, efficiency, or
quality measures?
Exercise 4: Application of
Performance Measure
Development
61

Developing Your Own Measures

Questions and Answers?

62
Contact:

Yvonne M. Watson
U.S. Environmental Protection Agency
watson.yvonne@epa.gov
(202) 566-2239
63