Presentation Overview

 QIC-EC Evaluation Design  Preliminary 12-month Outcome Data  Social Innovation and Developmental Evaluation  Participatory Evaluation and Planning

QIC-EC Evaluation Design
 Quasi-experimental design comparing outcomes in

treatment vs. comparison neighborhoods  Examines the effects of concentrating services and increasing engagement in high-poverty neighborhoods on:

Decreasing risk factors Increasing protective factors Reducing child maltreatment

QIC-EC Evaluation Design
 Service tracking: Outreach activities, enrollment,

retention, community events, home visits, and parenting education
 Process data: Parent involvement and satisfaction,

home visitor/parent educator insights, collaborative functioning
 Outcome data: risk and protective factors, child

maltreatment rates

QIC-EC Evaluation Design
Marion County
Swegle Elementary (Tx Group) Washington Elementary (Tx Group)

Yamhill County
Sue Buel Elementary (Tx Group)

Hallman Elementary (Comp. Group)

Hoover Elementary (Comp. Group)

Edwards Elementary (Comp. Group)

Preliminary Outcomes
 Repeated Measures ANOVA  Comparison of intake and 12-month mean scores  Current sample size: n=42 treatment, n=41 comparison
 Results

are preliminary  Current sample represents approximately 60% of the anticipated 60-80 participants in each group by the end of the study period.

Parenting Stress Index
 Profile of perceived child and parent stress.

Parent Domain: Competence, isolation, attachment, health, feeling of role restriction, depression, and spousal support. Child Domain: Distractibility/hyperactivity, adaptability, reinforcement of the parenting experience, demandingness, mood, and acceptability. Total Stress: Level of stress in the parent-child relationship, comprised of the child and parent domains.

Parenting Stress Index
 Decreases in score values demonstrate improvement
Parenting Stress Index : Intake & One Year

20 15 8.62


5 0 -5 -10 -15 -20 -9 0.24


Treatment Change Score
Comparision Change Score


Child Domain (n=48, n=41)

Parent Domain ** (n=48, n=41)

Total Stress * (n=48, n=41)

Adult Adolescent Parenting Inventory
 Index of risk of child maltreatment in five specific

parenting and child rearing behaviors.

Expectations of Children: Lack of knowledge of the needs and capabilities of children at various stages of growth and development. Parental Lack of Empathy Toward Children’s Needs: Parents lacking sufficient levels of empathy find children’s needs and wants irritating and overwhelming. Strong Parental Belief in the Use of Corporal Punishment: Abusive parents not only consider physical punishment a proper disciplinary measure, but also strongly defend their right to use physical force. Reversing Parent-Child Family Roles: Children are expected to be sensitive to and responsible for much of the happiness of their parents. Oppressing Children’s Power and Independence: Obedience and compliance to parental authority are demanded. Breeds powerlessness, inadequacy, rebelliousness, unhealthy compliance, and “followers”, not leaders.

Adult Adolescent Parenting Inventory
 Increases in scores demonstrate improvement
Adult Adolescent Parenting Inventory : Intake & 1 Year
0.74 0.80 0.60 0.40 0.20 0.00 -0.20 -0.40 -0.60 -0.25 -0.44 -0.07 -0.26 0.54 0.76 0.47 0.34 0.18

Treatment Change Score Comparison Change Score

Expectations of Empathy Toward Children Children's Needs (n=51, n= 43) (n=51, n= 43) Corporal Punishment (n=51, n= 43) Parent-Child Roles (n=51, n= 43) Children's Power/ Independence (n=51, n= 43)

Self-Report Family Inventory
 Respondents’ perception of his/her family’s


Health/competence: family affect, parental coalitions, problem-solving abilities, autonomy and individuality, optimistic vs. pessimistic views, acceptance of family members. Conflict: arguing, blaming, fighting openly, acceptance of personal responsibility, unresolved conflict, negative feeling tone. Cohesion: family togetherness, satisfaction received from inside vs. outside the family, spending time together. Emotional Expressiveness: verbal and nonverbal expression of warmth, caring and closeness.

Self-Report Family Inventory

Decreases in score values demonstrate improvement.
Family Inventory: Intake & 1 Year

0.8 0.6 0.4 0.2 0 0.15 Treatment Change Score Comparison Change Score





-0.2 -0.4




Health/ Competence (n=42, n=41)

Conflict (n=42, n=41)

Cohesion (n=42, n=41)

Expressiveness (n=42, n=41)

Caregivers’ Assessment of Protective Factors
 Assesses the presence of protective factors that have been

linked to lower rates of child maltreatment.

Nurturing and Attachment: Responding appropriately, warmly, and consistently to the basic needs of infants and young children; fostering a strong and secure parentchild relationship Social Connections: Having friends, family members, neighbors, and others who provide emotional support, concrete assistance, serve as a resource for parenting information, provide an opportunity to engage with others in a positive manner, help solve problems, or reduce feelings of isolation Concrete Support in Times of Need: Having financial security to cover basic needs and unexpected costs; being aware of and accessing formal supports and crisis services Resilience 1 Resilience 2

Caregivers’ Assessment of Protective Factors

Increases in score values demonstrate improvement.
Caregivers' Assessment of Protective Factors: Intake & 1 Year

0.80 0.60 0.57 0.52 0.25 0.06 -0.04 -0.03

0.20 0.00 -0.20 -0.40 -0.60 -0.80 Nuturing (n=42, n=41)


0.06 0.03



Treatment Change Score Comparison Change Score

Social Connections (n=42, n=41)

Concrete Support (n=42, n=41)

Resilience 1 (n=42, n=41)

Resilience 2 (n=42, n=41)

Total score (n=42, n=41)

A Complex Adaptive System:
o Non-linearity – small change can produce BIG effects (“Butterfly Effect”) o Emergence – innovators can’t determine in advance what will happen, so evaluators can’t determine in advance what to measure o Adaptation – interacting elements respond and adapt to each other o Uncertainty – uncertain interventions and unpredictable outcomes can intersect with and be affected by larger societal uncertainties o Dynamical Systems Change – patterns of change are unpredictable, turbulent, nonlinear, and complex o Coevolution –the social innovator and the developmental evaluator co-create the innovation and the evaluation

 Theory-based innovation to identify “Best Practices”  Standardization and implementation fidelity are key  Formative and Summative Evaluation  Replication of vetted models

A Developmental Model:
Children’s Bureau Center for the Study of Social Policy Harvard Center on the Developing Child

o “Ready, Fire, Aim” – learning by doing o Social innovation- the program is emergent, evolving, and adapting o Seizes new opportunities and adjusts to changing conditions

Developmental Evaluation:
o A dynamic partnership between innovators and evaluators, informed by complexity theory:  Brings evaluative thinking to the development process  Tracks developments  Facilitates interpretation of developments

The “Global Village” (Adaptive Middle) where “top-down” and “bottom-up” collide in the Muddled Middle  Where Systems Change Occurs  Where Complexity Abounds

Fostering Hope

Developmental Methods:
Community Based Orgs Healthcare Families Schools Faith

 Grassroots innovation: Adapting others’ ideas to meet local needs based on local knowledge  Local ownership, relevance, and buyin are key  Context-specific outcomes evaluation  Innovation spreads locally as people see effectiveness for themselves

o Data are used to inform development, not to test a static model o Accepts turbulence as the way that innovation unfolds o Moves away from experimental research o Flexible, creative methodology

Participatory Evaluation and Planning
 A real-time, issues-focused data sharing partnership

between evaluation and collaborative partner agency representatives
  

Utilization-focused Participatory Developmental

Participatory Evaluation and Planning
 Generating research questions
 Planning the evaluation design  Selecting appropriate measures and data collection

methods  Gathering and analyzing data  Reaching consensus about findings, conclusions, and recommendations  Disseminating results and preparing action plans to improve performance

The Shewhart Cycle

PEP Team Members
 Leadership: Executive Director
 Project staff: Project Director, Data Liaison, Home

Visitors/Parent Educators, Supervisors  “Key” partner organization representatives  Evaluation staff

Example 1: Enrollment and Retention in the Edwards Neighborhood
 Purpose: Meet enrollment targets through

collaborative outreach and engagement efforts

Number of participants in each neighborhood
 Marion

County = 40/neighborhood  Yamhill County = 20/neighborhood

Enrollment/Retention: Edwards
Oct 2010-Feb 2011
25 23.46



Number of Participants


Target, accounting for 83% retention rate Target, simple value

Actual, enrolled and maintained 10
8.75 Projected, enrolled and maintained


1 0

Edwards Neighborhood Enrollment Strategies
 Shewhart Cycle Action Steps: March - June, 2011  Send fliers to TANF families  Recruit from community events  Neighborhood canvassing  Post flyers in the community  Visit local food box distributor  Meet with school staff

Enrollment/Retention: Edwards
October 2010-June 2011
25 23.46


20.00 20.00

Number of Participants


Target, accounting for 83% retention rate 11.00 Target, simple value Actual, enrolled and maintained Projected, enrolled and maintained




Edwards Neighborhood Enrollment
 Program and contextual changes: July - Sept., 2011  FHI outreach staff turnover  Summer holiday  August PEP meeting cancelled

Enrollment/Retention: Edwards
Oct/Nov 2010-Sept 2011




Number of Participants

15 Target, accounting for 83% retention rate

Target, simple value
10 11.00 Actual, enrolled and maintained Projected, enrolled and maintained 5


Response to Enrollment Plateau
 Shewhart Cycle Action Steps : October to present  Point-person assigned to lead recruitment  “Friend and Family” referrals  Intern residing in the neighborhood was hired for outreach  Recruitment through WIC  Staff attendance at neighborhood events (e.g. community health fair for families with young children)  Fliers distributed through the elementary school

Enrollment/Retention: Edwards
October 2010-March 2012



24.14 20.00 19.00 17.00 Target, accounting for 83% retention rate

Number of participants



14.00 11.00 Target, simple value



Actual, enrolled and maintained


Example 2: Collaboration Survey
 Online survey was live

for 4 weeks (6/21/11—7/15/11)  33% response rate (n=34)  26 different organizations were represented

Collaboration Survey: Organizational Involvement by County

% Responding "High" or "Moderate" Involvement

90% 80% 81.5% 70% 60% 50% 40% Marion County (n=27) Yamhill County (n=10) 55.5%

20% 20.0% 10% 0% Planning stage Time of survey 30.0%

Collaboration SurveyOrganizational Satisfaction by County
Organizational Satisfaction at Time of Survey

Marion County (n=27)



30.0% Yamhill County (n=10)














Very satisfied

Somewhat satisfied

Not at all satisfied

Qualitative Data Findings
 Yamhill County respondent concerns:  Initiative’s expense  Cross-county service provision  Execution of plans

Collaborative Member Insights
 County Differences

Marion County
 Urban/suburban


presence since 1930’s

Yamhill County
 Rural

 Project

started later  Providers are “highly integrated” and value autonomy and independence  CCS is viewed as “coming in from the outside”

Response to Collaboration Survey Findings
 Shewhart Cycle Action Steps  Join established efforts in Yamhill County to identify opportunities for shared roles
 Discontinue

FHI-centric meetings in Yamhill County  Attend Early Childhood Coordinating Council meetings at Yamhill County Public Health  Request permission to share about FHI at Yamhill County agency meetings

 The approach has also been applied during FHI

expansion to 3 new neighborhoods in other rural counties

Lessons Learned
 Leadership is Essential  Executive Director facilitates most PEP meetings  Project Director turnover – confusion, delays  If possible, partner decision-makers should attend PEP meetings  Expertise in Research also Vital  Sampling, measurement error, power analysis, statistical significance, etc.  Provide “big picture” perspective when needed

Lessons Learned
 Group Membership Matters  Range of perspectives provides context for data interpretation/action  Accountability issues – PEP group members are not employees of CCS  Timing is Key  Error towards meeting more often  Action takes longer than you would expect

Sign up to vote on this title
UsefulNot useful