Data Analysis for Evaluation

Eric Graig, Ph.D.

 Purpose of this Training

 To increase your skills in analysis and

interpretation of the information you collect
 To increase your ability to report your findings
to a variety of audiences
 To learn how to make evaluation results
actionable

Innovation Network, Inc. Slide 2

 Three Steps to Data Analysis

 Analyze Results
 Communicate Findings
 Use Findings for Program Improvement

Innovation Network, Inc. Slide 3

 Keep it Simple

 Aim for a systematic effort

 Keep your audience in mind
 Pay attention to the usability of your
evaluation report

Innovation Network, Inc. Slide 4

 Two Kinds of Data

 Quantitative
 Qualitative

Innovation Network, Inc. Slide 5

 Quantitative Data

 Data that is numerical, counted, or

compared on a scale
 Demographic data
 Answers to closed-ended survey items
 Attendance data
 Scores on standardized instruments

Innovation Network, Inc. Slide 6

 Qualitative Data

 Narratives, logs, experience

 Focus groups
 Interviews

 Open-ended survey items

 Diaries and journals

 Notes from observations

Innovation Network, Inc. Slide 7

 Summary Statistics

 Measures of Central Tendency

 Mean

 Median

 Range

 Frequency distributions
 Percentage distributions

Innovation Network, Inc. Slide 9

 Frequency Distributions

To what extent did you increase your skills in

putting together a household budget?
A lot Some A little Not at all

Women (N=30) 14 9 5 2

Innovation Network, Inc. Slide 10

 Percentage Distributions

To what extent did you increase your skills in

putting together a household budget?
A lot Some A little Not at all

Women (N=30) 46% 30% 17% 7%

Innovation Network, Inc. Slide 11

 Graphing Frequency Data
How did you first hear about the web-site?

Court Referral
Social Worker
Friend or Acquaintance
Librarian
Web Search Engine
Newspaper Story
Other

Innovation Network, Inc. Slide 12

 Means and Medians

Math
History 98
95
English
English 96
Mean = 87
History
Biology 95
93
Music
Latin 94
92
Median = 94
Biology
Math 93
98
Latin
Music 92
94
Gym
Gym 40

Innovation Network, Inc. Slide 13

 Some Subtleties

40 50 55 94 100 100 100 Mean = 81

40 92 93 94 95 96 98 Mean = 87

Innovation Network, Inc. Slide 14

 Histograms
7 6

6
5

5
4
Frequency

Frequency
4
3
3

2
2

1 1

0 0
60 70 80 90 100 60 70 80 90 100

Innovation Network, Inc. Slide 15

 Cross Tabulations
Program Type Area of Inquiry Outcome
Web site Employment law Satisfied
I & R Line Family law Not satisfied
Law clinic Immigration Pending Count of Outcome Outcome
Web site Immigration Satisfied Program Type Not satisfied Pending Satisfied Grand Total
I & R Line Immigration Satisfied I & R Line 7 5 12
I & R Line Family law Not satisfied Law clinic 1 3 7 11
Web site Employment law Not satisfied Web site 6 5 11
Law clinic Other Satisfied Grand Total 14 3 17 34
I & R Line Other Not satisfied
I & R Line Other Satisfied
Law clinic Employment law Satisfied Count of Outcome Outcome
Web site Family law Satisfied Program Type Not satisfied Pending Satisfied Grand Total
I & R Line 58% 0% 42% 100%
Law clinic Family law Satisfied
Law clinic 9% 27% 64% 100%
Web site Immigration Not satisfied
Web site 55% 0% 45% 100%
Law clinic Immigration Not satisfied Grand Total 41.18% 8.82% 50.00% 100.00%
I & R Line Family law Satisfied
I & R Line Immigration Not satisfied
I & R Line Employment law Not satisfied
Law clinic Other Pending

Innovation Network, Inc. Slide 16

 Graphing comparisons
Satisfaction with Services
40

35
Satisfaction Score

30

25

20

15

10

0
A B C D E
Clinic Name

Innovation Network, Inc. Slide 17

 Some subtleties
Satisfaction with Services
16

14
Satisfaction Score

12

10
Staff
8 Advice
Facility
6

0
A B C D E
Clinic
Innovation Network, Inc. Slide 18
 Some more subtleties
Satisfaction with Services
16

14
Satisfaction Score

12
A
10
B
8 C
D
6
E
4

0
Staff Advice Facility
Satisfaction Component
Innovation Network, Inc. Slide 19
 A Note on Quantitative Data Entry

 Several approaches
 Paper and pencil tally
 Word processing table

 Spreadsheet

 Custom database

Innovation Network, Inc. Slide 20

 Qualitative Data
 Textual data
 Interview transcripts
 Case notes/ clinical notes

 Open-ended survey questions

 Photographs
 Video recordings

Innovation Network, Inc. Slide 21

 Qualitative Analysis

 Identify themes or patterns

 Pre-set themes
 Determine as you go
 Tag text segments with themes

Innovation Network, Inc. Slide 22

 Sample data sheet

Site
Interview Write Up Sheet

Naming Standard: Siteinterviewtype#

Cluster ND Futures
Site Oakville
Interviewee Diane Janssen, Elementary School Principal
Date of Interview
Tape # 02-1234-PCM
Length 31
December 2004
Interviewee
Interview Types:

Focus
Com
Coach
Sup
Date
Plan
Other

Site Intervie Questio Text

Tape number
Name w Type/# n ?#

Length
Ellendal Plan PC.1 She states that she came from to the town to attend
e the local Bible College, stayed to become a teacher in
the local school and then took a job teaching at the
college before becoming the principal of the
elementary school. She states that she first got
involved with the program after the large community
meeting that took place in April of 2004. The school
leadership (she doesn't indicate precisely who) felt it
Question number
was important that someone from the school get
involved in the planning committee so she volunteered
since she was interested in getting involved anyway.
She mentions both a desire to bring to the committee
the perspective of the school while at the same time to
Innovation Network, Inc. take back to the school whatever knowledge she Slide 23
acquired through participation.

PC.2 She talks a little bit about the school. It is the K-12
school with about 350 children. She oversees the K-6
program while another principal supervises grades 7
through 12. There are 157 students in the elementary
 Exercise

 Break into small groups

 Using the sheet provided search for snippets
of text that relate to the codes provided
 Identify any other themes that may be
important that are not provided on the sheet

Innovation Network, Inc. Slide 24

 Communicating Findings
“Information that is not effectively
shared with others will not be
effectively used.”

Source: Building a Successful Evaluation

Center for Substance Abuse Prevention

Innovation Network, Inc. Slide 25

 Audience

 Staff  Partners
 Board  Other agencies
 Funders  Public

Innovation Network, Inc. Slide 26

 Reporting Formats
 Written report
 Data book
 Flyer
 Formal oral presentation
 Round table with partners/ stakeholders
 Poster
 Web site

Innovation Network, Inc. Slide 27

 Whatever strategy you choose:

 Link findings to the program’s outputs

and desired outcomes
 Include successes and challenges
 Support your claims with data
 Acknowledge knowledge gaps

Innovation Network, Inc. Slide 28

 Organizing Findings
 Overview
 Describe program/ describe evaluation
 Program Implementation
 What did you do, and how well did you do it?
 Program Outcomes:
 What were the changes that occurred?
 Lessons learned
 Sub-group differences
Innovation Network, Inc. Slide 29
 Overview Section

 Your organization
 Your program
 Purpose of the evaluation
 Evaluation design
 Structure of the report

Innovation Network, Inc. Slide 30

 Program Implementation
 Characteristics of program participants
 Demographics
 Outputs
 Number of sessions held
 Number of participants served
 Partnerships formed
 Satisfaction with program

Innovation Network, Inc. Slide 31

 Program Outcomes

 What has changed?

 What do participants now know?
 What are they now able to do?
 What do they actually do?
 How are there lives different?

Innovation Network, Inc. Slide 32

 Lessons Learned
 Which of the program’s assumptions were validated,
which need to be reconsidered?
 Does the program affect different kinds of people
differently?
 Are results better with some groups?
 How will findings be addressed?
 What new hypotheses did the evaluation generate?
 How did the evaluation work?

Innovation Network, Inc. Slide 33

 Additional Resources

 One-on-one technical assistance

 Synchronous course
 Asynchronous material
 InnoNet website www.innonet.org

Innovation Network, Inc. Slide 34