EVALUATING STUDENT SUCCESS INITIATIVES
LACCD Student Success & 3CSN Summit
Making Sure Things Work Before We Scale Them Up
Center for Applied Research at CPCC 2013
PURPOSE
We want to take some time to discuss common misconceptions and issues experienced by colleges around the subject of evaluation. We want to understand the differences between evaluation and research. We want to know how to develop and implement a good evaluation for an intervention or program.
Center for Applied Research at CPCC 2013
PROGRAM EVALUATION
What is evaluation?
Evaluation is a profession composed of persons with varying interests, potentially encompassing but not limited to the evaluation of programs, products, personnel, policy, performance, proposals, technology, research, theory and even of evaluation itself.
Go to: http://www.eval.org At the bottom of the homepage there is a link to a free training package and facilitators guide for teaching the Guiding Principles for Evaluator Training
Center for Applied Research at CPCC 2013
MORE ON EVALUATION
As defined by the American Evaluation Association, evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness.
Evaluation is the systematic collection and analysis of data needed to make decisions, a process in which most well-run programs engage from the outset. Here are just some of the evaluation activities that are already likely to be incorporated into many programs or that can be added easily: Pinpointing the services needed for example, finding out what knowledge, skills, attitudes, or behaviors a program should address
Center for Applied Research at CPCC 2013
CONTINUED
Establishing program objectives and deciding the particular evidence (such as the specific knowledge, attitudes, or behavior) that will demonstrate that the objectives have been met. A key to successful evaluation is a set of clear, measurable, and realistic program objectives. If objectives are unrealistically optimistic or are not measurable, the program may not be able to demonstrate that it has been successful even if it has done a good job Developing or selecting from among alternative program approaches for example, trying different curricula or policies and determining which ones best achieve the goals
Center for Applied Research at CPCC 2013
CONTINUED
Tracking program objectives for example, setting up a system that shows who gets services, how much service is delivered, how participants rate the services they receive, and which approaches are most readily adopted by staff Trying out and assessing new program designs determining the extent to which a particular approach is being implemented faithfully by school or agency person
Center for Applied Research at CPCC 2013
PROGRAM EVALUATION
Purpose
To establish better products, personnel, programs, organizations, governments, consumers and the public interest; to contribute to informed decision making and more enlightened change; precipitating needed change; empowering all stakeholders by collecting data from them and engaging them in the evaluation process; and experiencing the excitement of new insights. Evaluators aspire to construct and provide the best possible information that might bear on the value of whatever is being evaluated.
Center for Applied Research at CPCC 2013
Definition of Evaluation
Study designed and conducted to assist some audience to assess an objects merit and worth.
(Stufflebeam, 1999)
Identification of defensible criteria to determine an evaluation objects value (worth or merit), quality, utility, effectiveness, or significance in relation to those criteria. (Fitzpatrick, Sanders & Worthen, 2004)
Center for Applied Research at CPCC 2013
Definition of Evaluation
Goal 1 Determine the merit or worth of an evaluand.
(Scriven 1991)
Goal 2 Provide answers to significant evaluative questions that are posed It is a value judgment based on defensible criteria
Center for Applied Research at CPCC 2013
Evaluation Questions
Provide the direction and foundation for the evaluation (without them the evaluation will lack focus) The evaluations focus will determine the questions asked.
Need Assessment Questions? Process Evaluation Questions?
Center for Applied Research at CPCC 2013
Outcomes Evaluation Questions?
T YPES OF EVALUATION
Process evaluation determines if the processes are happening according to the plan The processes of a program are the nitty gritty details or the dosage students, patients or clients receive the activities It is the who is going to do what and when It answers the question Is this program being delivered as it was intended.
Center for Applied Research at CPCC 2013
T YPES OF EVALUATION
Outcome evaluation (most critical piece for accreditation) determines how participants do on shortrange, mid-range or long-range outcomes Usually involves setting program goals and outcome objectives Answers the question is this program working and/or are participants accomplishing what we intended for them to accomplish
Center for Applied Research at CPCC 2013
T YPES OF EVALUATION
Impact evaluation How did the results impact the student group, college, community, family (larger group over time) Answers the question Is this program having the impact it was intended to have (so you must start with intentions)?
Center for Applied Research at CPCC 2013
TWO MAJOR T YPES OF EVALUATION
Formative Evaluation Summative Evaluation
Process
Needs Assessment
Short/ Intermediate Outcomes
Did it Work
Long-range Outcomes
Impact
Center for Applied Research at CPCC 2013
IR DEPARTMENTS
The Good News Is..
You are all data people
The Bad News Is.
You are all data people Sometimes have difficulty realizing this is not research and demands more than data from your student system
Center for Applied Research at CPCC 2013
Evaluation
Use intended for use use is the rationale the decision-maker, not evaluator, comes up with the questions to study. compares what is with what should be does it meet established criteria action setting/priority is to the program, not the evaluation friction among evaluators roles and program givers roles because of judgmental qualities of evaluator.
Research
produces knowledge lets the natural process determine use the researcher determines the questions studies what is
Questions
Judgment
Setting
priority is to the research, not what is being studied not the friction; research vs. funder no friction
Roles
Center for Applied Research at CPCC 2013
ISSUES WITH EVALUATION
In Community Colleges
Center for Applied Research at CPCC 2013
INTERVENTIONS HAVE QUESTIONABLE SUCCESS
The evaluated dont take into consideration all factors including methodology and quality of implementation College needs to have a realistic/courageous conversation on standards of evidence, statistical significance and expectations Spend most of the time planning the interventions, not on how to evaluate it Never define what success should look like, reasonable target
Center for Applied Research at CPCC 2013
INTERVENTIONS ARE OFTEN TOO COMPLICATED
Multiple layers of independent variables College lacks the staff, software or ability to carry it out. Groups keep getting smaller and smaller (for sample or comparison groups). Dont really know what worked. Expansion happens too quickly.
Center for Applied Research at CPCC 2013
INTERVENTIONS HAVE QUESTIONABLE ABILIT Y TO BE ADAPTED ON A LARGE SCALE
Not enough consideration of the costs of scaling Dont want to cancel plans involving un scalable interventions (someones pet project) Develop culture where it is ok to take risk and learn from mistakes
Center for Applied Research at CPCC 2013
THE COLLEGE SKEPTIC
The one who wants everything to be statistically significant The faculty group who wants to talk about confidence intervals or power Fear that things wont work We tried that before They confuse evaluation with research.
Center for Applied Research at CPCC 2013
LIMITED ABILITY TO EVALUATE.
Whole concept is new to many. Funders forces us to begin the process. May be no one at the institution to lead them through it (health faculty are the best place to start). Dont know what resources are out there?
Center for Applied Research at CPCC 2013
ANALYSIS PARALYSIS Lets splice and dice the data more and more and more. Too much data to analyze Dont know what it tells them How do we make a decision about priorities and strategies from 200 pages of data tables?
Center for Applied Research at CPCC 2013
THE SUMMER HIATUS
Faculty leave in June and never give the initiative a thought until August 20 th. No interventions are in place when fall term begins No evaluation tools are in place. Baseline data cannot be collected. August 20-31 they are mostly concerned with preparing for fall classes (as they should).
Center for Applied Research at CPCC 2013
NO WORKABLE EVALUATION TIMELINES
Creating a timeline. Identifying all the detail. Getting a team to actually follow it. Who is responsible for each piece. Where do completed surveys/assessments go who scores them who analyzes them who makes decisions based on them?
Center for Applied Research at CPCC 2013
What does a logic model look like?
Graphic display of boxes and
arrows; vertical or horizontal
Relationships, linkages.
Any shape possible Level of detail
Simple Complex
Circular, dynamic, Cultural adaptations, storyboards.
Multiple models
Source / Adapted from UW-Extension: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html Center for Applied Research at CPCC 2013
Where are you going? How will you get there? What will tell you that youve arrived?
A logic model is your program ROAD MAP
Center for Applied Research at CPCC 2013
Example: Every day logic model Family Vacation
Family Members Budget Car Camping Equipment Drive to state park Set up camp Cook, play, talk, laugh, hike Family members learn about each other; family bonds; family has a good time
Source: E Taylor-Powell, University of Wisconsin- Extension-Cooperative Extension
Center for Applied Research at CPCC 2013
Example: Financial management program
Situation: Individuals with limited knowledge and skills in basic financial management are unable to meet their financial goals and manage money to meet their needs.
INPUTS
Extension invests time and resources
OUTPUTS
We conduct a variety of educational activities targeted to individuals who participate
OUTCOMES
Participants gain knowledge, change practices and have improved financial well-being
WHAT WE INVEST
WHAT WE DO
WHAT RESULTS
Source: E Taylor-Powell, University of Wisconsin- Extension-Cooperative Extension Center for Applied Research at CPCC 2013
Example: One component of a comprehensive parent education and support initiative
Situation:
During a county needs assessment, majority of parents reported that they were having difficulty parenting and felt stressed as a result
INPUTS
Staff Develop parent ed curriculum
OUTPUTS
OUTCOMES
Parents increase knowledge of child dev
Parents identify appropriate actions to take
Money
Partners Research
Deliver series of interactives essions Facilitate support groups
Targeted parents attend
Parents better understanding their own parenting style Parents gain skills in effective parenting practices
Improved childparent relations
Parents use effective parenting practices
Strong families
Assumptions:
External factors:
Center for Applied Research at CPCC 2013
Example: Smoke free worksites
Situation: Secondhand smoke is responsible for lung cancer, respiratory symptoms, cardiovascular disease, and worsens asthma. Public policy change that creates smoke free environments is the best known way to reduce and prevent smoking. Inputs
Assess worksite tobacco policies and practices Develop community support for SF worksites Organize and implement strategy for targeted worksites
Outputs
Increased awareness of importance of SF worksites
Outcomes
Demonstrations of public support for SF worksites SF worksites SF worksites policies drafted
Coalition Time Dollars Partners Including youth
Worksite owners, managers
Unions
Workers; union members Public
Increased knowledge of SF worksite benefits & options Increased commitment, support and demand for SF worksites
SF worksite policies passed
Adherence to smoke-free policies
Source: E Taylor-Powell, University of Wisconsin- ExtensionCenter for Applied Research at CPCC 2013 Cooperative Extension
Need Assessment Questions? INPUT What resources are needed for starting this intervention strategy?
Process Evaluation Questions? PROCESS
Is the intervention strategy being implemented as intended?
Outcomes Evaluation Questions? OUTCOMES
To what extent are desired changes occurring? For whom? Is the intervention strategy making a difference? What seems to work? Not work?
How many staff members are needed?
Are participants being reached as intended?
Source: R. Rincones-Gomez, 2009 Center for Applied Research at CPCC 2013
CHAIN OF OUTCOMES
SHORT
Seniors increase knowledge of food contamination risks Participants increase knowledge and skills in financial management Community increases understanding of childcare needs
MEDIUM
Practice safe cooling of food; food preparation guidelines Establish financial goals, use spending plan
LONG-TERM
Lowered incidence of food borne illness Reduced debt and increased savings
Residents and employers discuss options and implement a plan
Child care needs are met
Empty inner city parking Youth and adults learn Money saved, nutrition lot converted to gardening skills, nutrition, improved, residents enjoy community garden food preparation and mgt. greater sense of community
for Applied Research at CPCC 2013 Source: E Taylor-Powell, University ofCenter WisconsinExtension-Cooperative Extension
WHAT ARE THE SUMMATIVE AND FORMATIVE OUTCOME INDICATORS
Supplemental Instruction Learning Communities Required Orientation Academic Success Course Minority Male Mentoring Developmental Math Redesign Peer Tutoring Accelerated English
Center for Applied Research at CPCC 2013
AT YOUR TABLES .
Select an ATD student success initiative at your college that you plan to evaluate before you make the decision to scale it up. (if you
cant think of one use the online learning one in your handouts)
Use this program for each activity.
Center for Applied Research at CPCC 2013
1. BRING TOGETHER THE PROGRAM DEVELOPERS
Ask them to answer these question:
1. Why did you develop this program with these program characteristics? 2. What do you think students (or participants) will get out of this program (what changes)? 3. How do you tie specific program content to specific expected changes or improvements in participants.
Center for Applied Research at CPCC 2013
2. ORIENT AN EVALUATION TEAM
Who should be on it? What skills do you need at the table (what staff members have those?) What should be their charge?
Center for Applied Research at CPCC 2013
3. GATHER INFORMATION ON POTENTIAL OUTCOMES.
What are potential sources for outcomes?
Center for Applied Research at CPCC 2013
4. WRITE OUTCOME STATEMENTS
Sometime these are already written (from grants) Make them clear Dont draw a number out of a hat Test it out Create a logic model
Center for Applied Research at CPCC 2013
5. CREATE OUTCOME INDICATORS
Outcome Indicator. - Usually referred to as a key performance indicator, this is the data, or set of statistics that best verifies the accomplishment of a specific outcome. An outcome indicator for college readiness might be an SAT score of 1100 or above. It is typically the accomplishment of a specific skill or assessment at a certain level that indicates an outcome is met.
What data can you access? What assessments need to be selected?
Center for Applied Research at CPCC 2013
6. CREATE OUTCOME TARGETS
Outcome Target the benchmark set as a performance indicator for a given outcome. An example would be that 80% of students would score a 75% or above on a reading assessment. The outcome target would be 80% of students.
How would you create these targets or benchmark? Do you need a comparison group? What is an acceptable level of improvement or change?
Center for Applied Research at CPCC 2013
7. CREATE ALL TOOLS
You will probably need:
Demographic sheets Attendance or participation log Formative evaluation tools
Will they be online or pencil/paper tools (benefits of each) When do they need to be ready? Who needs copies? Create evaluation timeline.
Center for Applied Research at CPCC 2013
8. PILOT TEST THE PROCESS
Make sure it works Give a small group of student or faculty/staff the assessments to make sure they are clear Work out all the detail
Who Who Who Who Who distributes it collects it scores it puts it in the spreadsheet keeps up with the post-test dates, etc.
Center for Applied Research at CPCC 2013
9. IMPLEMENT THE EVALUATION
Follow your plan
Center for Applied Research at CPCC 2013
10. ANALYZE RESULTS
Sometimes just numbers and percents Sometimes statistical tests are needed If students dont meet the summative evaluation benchmarks, analyze the formative evaluation
Center for Applied Research at CPCC 2013
11. IMPROVE YOUR PROCESS AND PROGRAM
Takes several years to have good data. Discuss how the evaluation can be improved Discuss how the program can be improved
Center for Applied Research at CPCC 2013
CLOSING
Establish your plan Follow your plan Assign responsibility for it Expect big things Use results to improve what you do (close the loop)
Center for Applied Research at CPCC 2013
SUPPORT AND CONTACT INFO:
Terri Manning, Ed.D. terri.manning@cpcc.edu (704) 330-6592
