HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

Healthy Relationships & LGBTQ Domestic Violence

Program Evaluation Plan

Prepared by:

Brandon Ignatowski and Loren Landeros

California State University, San Bernardino

05/22/17

Page 1
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

1. INTRODUCTION AND STAKEHOLDER ENGAGEMENT

Stakeholders for Healthy Relationships Program is California State University, San
Bernardino (CSUSB), Dr. Verissimo, CSUSB Health Center and CSUSB Health Science
Department.

______________________________________________________________________

Evaluation Purpose
The purpose is to evaluate the effectiveness of the presentation of the Healthy
Relationship & LGBTQ Domestic Violence PSA and Zine to the students
attending the Second Annual Run Like a Mother 5K.
Determine how the program is affecting CSUSB students by demonstrating the
programs long-term goals.
Influence CSUSB students by working as a channel for change in autonomy.
Were all 200 students reached during the 5K run?
Did we secure additional resources for the CSUSB LGBTQ community to
access?
Was training started for Police Officers in San Bernardino?
Was training started for local shelters?

The findings from the evaluation will be used to assess achievement of

objectives, develop and increase the shelter locations where assistance is
provided, organize support within the community, and emphasize resources on
the CSUSB community.

The evaluation fits with the overall goal of the CSUSB Student Health Center by
providing student awareness of the CSUSB Student Health Center as a resource
to provide aid in abusive situations and promoting health and well being in the
student population. Also, data will be available to determine the need for
additional resources at the CSUSB Student Health Center to deliver quality and
efficient care.
______________________________________________________________________

Stakeholders
The groups and individuals that have an interest in the outcomes of the
evaluation are CSUSB Health Center, Dr. Verissimo, CSUSB Students, San
Bernardino Police Department, local shelters, Brandon Ignatowski, and Loren
Landeros
The stakeholders listed above are primarily interested in the effectiveness of the
program versus the cost angle.
The roles they will play in developing or implementing this evaluation plan
includes:

Page 2
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

The San Bernardino Police Department will collect data to confirm the number of
cases reported and the decrease in domestic violence in the LGBTQ community.
Local Shelters will collect data to identify how many CSUSB LGBTQ students
used their resources. In addition to determining if the cases were handled
appropriately.
Brandon Ignatowski and Loren Landeros will serve in the planning team,
interpret findings, receive results, and collect data.
Dr. Verissimo and CSUSB Health Center will serve as external reviewers.
.
Table 1. Stakeholder Assessment and Engagement Plan

Stakeholder Name Interest or Perspective Role in the Evaluation

{May be an individual or a group}
Brandon Ignatowski
Loren Landeros
San Bernardino Police Department
Local Shelters
Dr. Verissimo
CSUSB Health Center
{program participant, staff, etc.}
Staff
Staff
Program Participant
Program Participant
Program Participant
Program Participant
{planning team, external
reviewer, etc.}
Planning team, data
collector, interpret
findings, receive results
Planning team, data
collector, interpret
findings, receive results
Collect data
Collect data
Internal reviewer
Internal reviewer

Health Science Department Chair Staff External Reviewer

Cultural Competence
We will engage stakeholders who reflect the diversity of those who may be
affected by the evaluations findings by listening to input they have on the
program, and if there is anything that sounds like it would be a good idea to
incorporate into the program it will be considered, as long as it benefits the target
population.
We will ensure to successfully elicit and incorporate the various perspectives by
providing stakeholders with forms they can fill out to voice their opinions. These
opinions will then be voted on by the rest of the stakeholders to see if they are
something that can be incorporated in the program and be effective. It is
important to get a consensus of the stakeholders to ensure that each idea is
heard and seriously considered for future use.
In order to account for the influence of context and culture in the evaluation,
design, implementation, and reporting of the program, surveys will be provided to
sample groups on areas that affect them. These will include the LGBTQ
community and the straight community of CSUSB to ensure that the program is
addressing both groups appropriately.

Page 3
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

2. DESCRIPTION OF EVALUATION OF THE EFFECTIVENESS OF THE HEALTHY

RELATIONSHIP AND LGBTQ DOMESTIC VIOLENCE PSA AND ZINE TO CSUSB
STUDENTS.

Need
This program is designed to meet the needs of CSUSB students, no matter their
sexual orientation, by providing them with resources that can enable them to
leave abusive relationships.

It will also address these students needs with support groups so they can share
their experiences with others that can relate to them.

It will address student needs by providing counseling from trained CSUSB Health
Center staff.

Student needs will be addressed long after they graduate and are in their
community by training local police officers on how to assist on abusive
relationship calls and through local shelters increasing their support for the
community.

______________________________________________________________________

Context
Cultural factors that may affect the implementation and effectiveness of this
program is the stigma attached to abusive relationships. CSUSB students that
are in an abusive relationship may be hesitant to utilize the program for fear of
their cohort learning about it.

The program may not be as effective for the LGBTQ community because some
of these students may neglect the resources provided by the program for fear of
being outed to their fellow students.

______________________________________________________________________

Population Addressed
The population for whom activities are intended are all Students of CSUSB
including the LGBTQ community within the campus.

Stage of Development
The program is in the evaluation stage.

Page 4
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

Resources/Inputs
The resources that are available to support the program currently include primary
data on abusive relationships, Brandon Ignatowski, Loren Landeros, a PSA, and
a Zine.

______________________________________________________________________

Activities
Specific activities that are conducted (or planned) to achieve the programs
outcomes include:
Educational activities on domestic violence awareness as well as positive
behaviors in healthy relationships.
Education on resource availability for domestic violence.
Demonstrating the PSA and Zine.
Providing student participants with participation incentives.
Providing training to the San Bernardino Police Department and local shelters.
______________________________________________________________________

Outputs
The activities will produce:
Zines that are printed and distributed.
Training sessions for the Police Department and local shelters.
Training sessions for volunteers.
Education sessions for participants.
Incentive purchases.

______________________________________________________________________

Outcomes
The programs intended outcomes include:
CSUSB student awareness on prevalence of LGBTQ domestic violence.
Increasing CSUSB students awareness on healthy relationship behaviors.
50 CSUSB 5K student participants will list 2 domestic violence behaviors each.
50 CSUSB 5K student participants will list 2 healthy relationship behaviors each.
100 CSUSB 5K student participants will recall 1 resource for domestic violence
each.
Intermediate Outcome: Families and individuals that applied for assistance and
resources will receive information on support group locations.

Page 5
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

Change that we ultimately want as a result of the activities include:

o Families and individuals who received assistance will continue

with their support group sessions to sustain themselves without
the abusive partner in their lives.
o What occurs between your activities and the point at which you
see these ultimate outcomes includes: CSUSB students will be
directed to outside support groups, the CSUSB Pride Center,
and other outlets that will enable them to continue their progress
toward a life outside of an abusive relationship.

______________________________________________________________________

Logic Model
See attached document appendix A.

______________________________________________________________________

3. EVALUATION DESIGN
Stakeholder Needs

Evaluation findings will be used by CSUSB Health Science Department, CSUSB

Student Health Center, Dr. Verissimo, Brandon Ignatowski, Loren Landeros, and
Dr. Tamalis.
The stakeholders need to learn who the target audience is, how to reach this
audience, how much will the program cost, were the program intervention
components delivered in an efficient manner, and how much CSUSB students
know about healthy/abusive relationships.

The intended users (CSUSB students) view data on domestic violence among
their peers, within their community, and among the LGBTQ community as
credible information. They also view resources for them to utilize if they or
someone they know is in an abusive relationship. This can include their local
shelters, police department, the Student Health Center, and the Pride Center.

These findings will be used to setup CSUSB Student Health Center with ways to
provide information and resources to the students in a way that keeps their
confidentiality. It will also be used to reach out to the San Bernardino Police
Department and local shelters to ensure that proper training is taking place in
concert with CSUSB staff and program participants.
The evaluation capacity that will need to be built to engage the stakeholders
throughout the evaluation will be conferences, coaching, evaluation policies,
learning circles, mentoring, and a community of circle. These will develop an

Page 6
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

evaluation culture of valuing the evidence, questioning, and evaluative thinking. It

will lay out the definitions, concepts, roles, and use of the evaluation within
CSUSBs stakeholders. It will provide supportive training so they can reach their
goals, and it will allow them to share their common interests and concerns with
each other so they can learn together.

______________________________________________________________________

Evaluation Questions
1) To what extent did content get delivered to the CSUSB community? 2) Was
this program aimed at all students of CSUSB? 3) How much does the CSUSB
community and students know about healthy and abusive relationships? 4) Were
all the program intervention components delivered in an efficient manner (e.g.
LGBTQ resource center information)?

______________________________________________________________________

Evaluation Design
The design for this evaluation is a pre-post test.
The rationale for using this design is that pre-post tests are useful for the
evaluation because it will allow us to ask CSUSB students at the 5K run how
much they know about healthy and abusive relationships at the beginning of the
program. Then we will be able to ask them the same questions again after our
presentation. This will allow us to gauge how much students know about this
topic and what they still need to learn at the end of the design.
__________________________________________________________

4. Gather Credible Evidence

Data Collection Methods
New (primary) data and old (secondary) data will both be used to collect
information to answer the evaluation questions. This is because some data will
be difficult to obtain due to the sensitivity of the topic. Some of the old data that
exists can be used (e.g. how many students face domestic violence issues).
Although, some of the data will be primary data because it is a newer program
(e.g. How to keep confidentiality of students and LGBTQ community). Secondary
data will allow us to discover how much CSUSB students know about healthy
and abusive relationships and if the program was delivered in an efficient
manner.
Surveillance systems will be used to acquire data. Data that is gathered by
surveillance systems are extremely useful for program evaluations and to
measure performance. Qualitative methods, like focus groups and interviews, will
also be used to collect the data to answer the evaluation questions. Interviews

Page 7
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

will be used to find out if the survey and pre-post test was used in an efficient
manner for the students to learn.
A sample will be used with volunteer CSUSB students. There will be two sample
groups used to identify strengths and weaknesses of the program (e.g. One
group of students that are familiar with abusive relationships and one group that
is not familiar with abusive relationships). This can help provide data on the
program that is not just biased information.
Data collection instruments will be created by providing questionnaires for
sample participants. If the sample participants feel comfortable, an interview will
be given to them so they can elaborate on the program effectiveness.
The data collection instruments will be tested for reliability by having Brandon
Ignatowski and Loren Landeros both randomly interview students and checking
each other's responses for any errors. In addition, pilot testing will be done for the
data collection methods before the 5K run and after the presentation, so any
failures can be addressed and rectified before implementing the program. This
can help address if the program will be culturally appropriate (especially when
dealing with the LGBTQ community). This will allow us to see if the resources
that are being offered to them are reliable so they can access them when
needed.
The quality and utility of existing data will be determined by evaluating the
accessibility of existing datasets. If the data is accessible then it is important to
evaluate the compatibility of the sampling approach and techniques used. If
these methods are compatible with the program, then the format and the
condition of the data will be evaluated. It is also important to ensure that the data
are current as well as relevant to the location it is being used (San Bernardino).
Primary data will be collected from the collection instruments (questionnaires,
interviews, pre-post tests, and surveys) that are applied to the CSUSB students
at the 5K run. Secondary data will be collected from online databases like
Ebscohost, Psychinfo, and through the Pfau Library on abusive relationships and
the target population.

Table 3: Evaluation Questions and Associated Data Collection Methods

Evaluation Question
1. Is the program keeping
confidential records of program
participants?
Data Collection Method Source of Data
Logs Primary Data Collection
Diaries Primary Data Collection

2. Is the program reaching all Surveys Primary Data Collection

CSUSB students, including the
LGBTQ students?
Observation Primary Data Collection

Document Review Primary Data Collection

Page 8
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

5. DATA ANALYSIS AND INTERPRETATION

Indicators and Standards
Some measurable or observable elements that can serve as markers of the
programs performance will include reaching approximately 200 students of
CSUSB during the 5K run. This means that the students will be interviewed and
tested on their knowledge of the topic during the 5K run. The pre-post test will
serve as a marker of the programs performance if the students are able to
increase their scores/knowledge after they complete the post test.
The program will be successful if at least 50% of the 200 CSUSB 5K runners are
able to recall one or more resource for domestic violence by completion of a 15-
minute awareness session. Second, if 25% of the 200 CSUSB 5K runners can
list 2 or more domestic violence behaviors after completion of a 15-minute
awareness session.

Table 4. Indicators and Success

Evaluation Question
1. Is the program raising
awareness for LGBTQ
domestic violence for the 5K
run participants?
Criteria or Indicator
On Oct 21, 2017, 50% of
the 200 CSUSB 5K
runners will recall one or
more resource for
domestic violence after a
15-minute awareness
session.
Standards
(What Constitutes
Success?)
Utility- Success will be
measured by ensuring that
at least half of the 5K
runners will be able to recall
1 or more resource for
domestic violence offered
by the program.

2. Are CSUSB students learning
how to spot signs of domestic
violence behaviors?
On Oct 21, 2017, 25% of
the 200 CSUSB 5K
runners will be able to list
two or more domestic
violence behaviors after
completing a 15 minute
awareness session.
Utility- Success will be
determined if students are
able to increase their
knowledge on the signs of
domestic abuse.

Page 9
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

Analysis
Mixed methods can be used to analyze the data of the program evaluation.
Quantitative analysis can be used by comparing the scores of the pre- and post-
tests given to the CSUSB students. This data can also be compared to the
surrounding San Bernardino community and other college campuses to see how
much CSUSB students know about this topic in comparison to others. Qualitative
analysis can also be used to analyze the data. This can be accomplished by
taking the answers that were provided in the pre-post tests and surveys to see if
the program was effective.

______________________________________________________________________

Interpretation
Those that will be involved in drawing, interpreting, and justifying conclusions
include Brandon Ignatowski, Loren Landeros, Dr. Verissimo, CSUSB Student
Health Center, CSUSB Pride Center, and CSUSB Health Science Department.
They will be involved because the conclusions and results drawn from this
program/presentation will affect all of them. All involved will be able to provide
insight and can help interpret the conclusions. Students at the 5K run will also be
included so they can elaborate on their own answers they provide in interviews
and questionnaires.
Plans to involve them in the process include a focus group that can involve the
stakeholders and program participants (e.g. students, CSUSB Health Center,
and Pride Center) so they can share their insights and observations on the
conclusions that were drawn from the program/presentation at the 5K run.

______________________________________________________________________

6. Use and Communication of Evaluation Findings

Use
Evaluation findings will be used to adjust the program prior to implementation.
For instance, more pamphlets on domestic violence will be provided to the
CSUSB Health Center to disseminate to students, and the Pride Center will
provide more educational material for the LGBTQ community to make these
students feel safe coming forward if they are in an abusive relationship. These
findings will also be used to reach out to the local police department and shelters
so a working relationship can begin with them.

Data will not be determined for the 5k event. Findings will be determined and
analyzed before the end of fall quarter. Data on intermediate and long-term goals
will be continuously collected through September of 2018.

Brandon Ignatowski and Loren Landeros are the main individuals responsible for
creating and monitoring an action plan that guides the implementation of

Page 10
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

evaluation recommendations. Other individuals that are responsible include Dr.

Verissimo and the CSUSB Health Science Department, even though they will
have a smaller role in the implementation process. These individuals will take the
information from the evaluation (e.g. Questionnaires, surveys, focus group, and
pre-post test information) and implement new changes to the program that will
accommodate the CSUSB students seeking domestic violence assistance.
CSUSB Health Center will be responsible to continue required recommendations
from October through the end of the program.

The lessons learned should include answers that are provided by CSUSB 5K
student participants from surveys, pre-post tests, questionnaires, and focus
groups. Also, information about what resources students utilize on-campus, or
off-campus, when in need of relationship advice and counseling. This information
will be documented online so the information is not lost. It will also be
documented through written documents in case the system crashes.

______________________________________________________________________

Communication
We will meet with Dr. Verissimo, CSUSB Health Science Department, and the
CSUSB Student Health center to talk about any flaws they see in the program,
ways to improve upon the existing program design, and to improve and update
the program based on the feedback given by the CSUSB students at the 5K run.
The methods used to communicate with evaluation stakeholders will be through
weekly group meetings, open telephone communications, online, social media
apps, and one-on-one meetings.
These methods of communication are appropriate for the specific evaluation
stakeholders because of the sensitivity of the program. Dealing with people in
abusive relationships is a round-the-clock problem that requires all stakeholders
involved the ability to communicate with each other at all times. These methods
will allow for evaluation stakeholders to meet with each other and share
information at their own convenience.
______________________________________________________________________

7. EVALUATION MANAGEMENT
Evaluation Team

Brandon Ignatowski and Loren Landeros will manage and implement evaluation
at least one month after the 5K run. CSUSB will continue to manage the
program from November 2017 through termination.

Brandon Ignatowski and Loren Landeros will need to consider alternatives for the
data being considered. They must also assess the limitation of remedies for
domestic violence. There may be possible constraints for the program that the
evaluation reveals and need to be considered for remediation.

Page 11
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

Dr. Tamalis will provide external review of the evaluation plan to provide
feedback. He will have knowledge of public health issues and is experienced in
statistics.

Table 5. Roles and Responsibilities of the Evaluation Team Members

Individual Title or Role Responsibilities
Brandon Ignatowski Program Creator Create, Evaluate, Implement, and
Oversee program
Loren Landeros Program Creator Create, Evaluate, Implement, and
Oversee program
Dr. Tamalis External Reviewer Reviews evaluation plan

______________________________________________________________________

Data Collection Management

Data that will be collected includes if the program is reaching the CSUSB LGBTQ
community and how confidential records are being kept.
Activities needed to carry out the data collection successfully include 15-minute
educational sessions at the 5K run and training volunteers on proper record
handling. These activities should be completed on Oct. 21, 2017.
Brandon Ignatowski and Loren Landeros are responsible for training volunteers
on proper record handling, and volunteers are responsible for the 15-minute
sessions.
Brandon Ignatowski and Loren Landeros will oversee the conduct of the
evaluation to assure appropriate implementation.

Table 6. Data Collection Plan

Data
Evaluation Activities Person(s)
Collection Due Date
Question Needed Responsible
Method
1. Is the Surveys Providing Volunteers 10/21/2017
program surveys to
reaching the CSUSB
CSUSB students
LGBTQ
community?

2. Is the Logs Providing Brandon 10/21/2017

program training to Ignatowski
keeping volunteers and Loren
confidential on log Landeros
records of management
program
participants?

Page 12
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

______________________________________________________________________

Data Analysis Management and Interpretation

The data will be secured by providing numbers for student participants instead of
using names, using lock box storage of information, not leaving documents
unattended, and ensuring proper training for volunteers on how to handle
sensitive materials.
The pre-post test will be analyzed by Dr. Tamalis within one month of the CSUSB
5K run.
Dr. Tamalis is responsible for conducting the analysis.
Brandon Ignatowski and Loren Landeros will conduct a presentation with a Q&A
session to engage the stakeholders in confirming analyses results and
interpreting them.

Table 7. Data Analysis Plan

Data to Be Person(s)
Analysis to Be Performed Due Date
Analyzed Responsible
Qualitative Knowledge of Brandon 11/15/2017
domestic Ignatowski and
violence Loren
resources Landeros
Descriptive Identification of Brandon 11/30/2017
domestic Ignatowski and
violence signs Loren
Landeros

______________________________________________________________________

Communicating and Reporting Management

The CSUSB Student Health Center and Dr. Verissimo are the audiences for
reporting the progress made on the evaluation and/or evaluation findings.
The purpose of the communications with this audience is to implement the
healthy relationships LGBTQ domestic violence awareness program.
The most appropriate type of communication methods to use with this audience
are face-to-face communication, or anonymous telephone communication due to
the sensitivity of this topic.
The communication will take place throughout the evaluation process.

______________________________________________________________________

Page 13
HSCI 614 Program Evaluation Plan Adapted from CDC Asthma Program Evaluation Guide

Timeline
The planning and administrative tasks will occur throughout the evaluation
process.
Training for data collectors will occur one month prior to data collection at the 5K
run.
Pilot testing data collection instruments will occur three weeks prior to the 5K run.
Formal data collection will occur at the 5K run on 10/21/2017, analyses will occur
on 11/30/2017, and the interpretation tasks will occur by 12/15/2017.
After interpretation tasks occur on 12/15/2017 the information will be
disseminated on 01/15/2018.
The only issue may be if there are any problems with the pilot testing. There may
be limited time to remake our data collection instruments if this happens. All other
tasks are given a one month period to ensure enough time so bottlenecking or
sequencing issues do not occur.

______________________________________________________________________

Wrapping Up

At the end of the evaluation we will acknowledge the contributions of the planning
team members and others who contributed to the successful implementation of
the plan by having an award ceremony and dinner banquet.

Lessons learned in the course of implementing the evaluation will be

documented by creating an online journal of our records.

Relevant documents, instruments, and data will be archived in computer files and
in the Student Health Center.

Page 14