You are on page 1of 62

Running head: PROGRAM EVALUATION NORTHWESTERN 1

Program Evaluation for the Peer Inclusion Educators Program at Northwestern University

Anthony Sis and Lorrena Johnson

Loyola University Chicago


PROGRAM EVALUATION NORTHWESTERN 2

Table of Contents
INTRODUCTION ...................................................................................................................................................... 3
PROGRAM HISTORY .............................................................................................................................................. 3
PROGRAM OVERVIEW .......................................................................................................................................... 4
PROGRAM CONTEXT ............................................................................................................................................. 5
PROGRAM DESCRIPTION .................................................................................................................................... 6
SELECTION OF PEER FACILITATORS ............................................................................................................... 7
PROGRAM PURPOSE ............................................................................................................................................. 8
PROGRAM STAKEHOLDERS ............................................................................................................................... 9
LOGIC MODEL OVERVIEW ................................................................................................................................ 10
PREVIOUS EVALUATION................................................................................................................................... 11
GENERAL EVALUATION APPROACH ............................................................................................................. 12
QUANTITATIVE APPROACH ............................................................................................................................ 13
PARTICIPANTS ............................................................................................................................ 13
SURVEY DESIGN ........................................................................................................................ 14
INSTRUMENT DESCRIPTION ........................................................................................................ 15
PILOT TESTING........................................................................................................................... 17
IMPLEMENTATION/ADMINISTRATION ......................................................................................... 17
STATISTICAL ANALYSIS/ RESULTS PRESENTATION .................................................................... 18
QUALITATIVE APPROACH ............................................................................................................................... 19
QUALITATIVE RATIONALE ......................................................................................................... 19
ANALYTICAL APPROACH ........................................................................................................... 20
FOCUS GROUP PLAN .................................................................................................................. 21
FOCUS GROUP IMPLEMENTATION .............................................................................................. 23
FOCUS GROUP ANALYSIS ........................................................................................................... 23
LIMITATIONS .............................................................................................................................. 24
RESULT PRESENTATION ............................................................................................................. 25
NEXT STEPS ............................................................................................................................... 25
REFERENCES ......................................................................................................................................................... 26
APPENDICES ......................................................................................................................................................... 27
APPENDIX A: PIE PROGRAM LOGIC MODEL .............................................................................. 27
APPENDIX B: PREVIOUS EVALUATION ....................................................................................... 30
APPENDIX C: ABBREVIATED SOCIAL JUSTICE KEY CONCEPTS .................................................. 33
APPENDIX D: INTERGROUP RELATIONS (IGR) LANGUAGE ACTIVITY ........................................ 34
APPENDIX E: AGENDA FOR INTERSECTIONAL LANGUAGE ACTIVITY ......................................... 36
APPENDIX F (LANGUAGE ACTIVITY SURVEY) ........................................................................... 37
APPENDIX G (SOCIAL IDENTITY SURVEY) ................................................................................. 41
APPENDIX H (INTERSECTIONALITY SURVEY) ............................................................................. 45
APPENDIX I: SURVEY ITEM MATRIX .......................................................................................... 48
APPENDIX J: FOCUS GROUP PROTOCOL FOR THE LANGUAGE ACTIVITY .................................... 50
APPENDIX K: FOCUS GROUP PROTOCOL FOR SOCIAL IDENTITY ................................................ 53
APPENDIX L: FOCUS GROUP PROTOCOL FOR INTERSECTIONALITY ............................................ 56
APPENDIX M: CONSENT FORM................................................................................................... 59
APPENDIX N: PROTOCOL INVITATIONS ...................................................................................... 61
APPENDIX O: PROTOCOL INVITATIONS (CONT.) ......................................................................... 62
PROGRAM EVALUATION NORTHWESTERN 3

Introduction
Northwestern University is an academic driven institution. The mission statement for the

university states that Northwestern University is committed to excellent teaching, innovative

research, and the personal and intellectual growth of its students in a diverse academic

community (About: Northwestern university, 2016). Within the university, there is the office of

Campus Inclusion and Community (CIC). In this office, there are three major departments that

promote the mission of CIC, which is to to work collaboratively with the university community

to cultivate inclusive learning environments through the intentional engagement with difference

(Campus Inclusion & Community, 2016, p. 4). These three departments are Multicultural

Student Affairs (MSA), Student Enrichment Services (SES), and Social Justice Education (SJE).

An evaluation and assessment will be created for one particular program within CIC, Peer

Inclusion Educators (PIE), which is housed under the Department of Social Justice Education.

Program History
Campus Inclusion and Community (CIC) began as an office in the Division of Student

Affairs five years ago. In 2012, Executive Director Lesley Ann Brown-Henderson was brought

in by the Vice President of Student Affairs to create this office. This office was created due to

student activists that demanded and protested for a university led effort to create a more

equitable and just campus culture (Campus Diversity & Inclusion, 2016, p. 15). CIC was

initially a one-woman office led by Lesley Ann Brown-Henderson. She began recruiting

students who would be brave enough to engage in dialogue surrounding topics of social justice.

This program became known as Sustained Dialogue. In 2014, Campus Inclusion and

Community was restructured due to the sizeable growth of the Sustained Dialogue program,

which is how the Social Justice Education Department was established. Noor Ali was selected to

be the Assistant Director of the Social Justice Education Department and the Peer Inclusion
PROGRAM EVALUATION NORTHWESTERN 4

Educators Program was created to help alleviate the administrative tasks within CIC and the

Sustained Dialogue program.

Peer Inclusion Educators has only been established for two years. During this time, one

graduate assistant was hired to oversee another program within SJE, but they additionally

volunteered to also be a Peer Inclusion Educator. After each workshop the participants are given

a survey that asks the students if they felt the workshop was beneficial, if the material they were

learning was new, and they are also asked to evaluate the facilitators. To date, no formal

assessment has been completed based off all of the surveys that were collected thus far.

Program Overview
There are several conceptual frameworks that guide the work of CIC, SJE, and PIE.

CICs strategic theme is to advance social justice. CIC has claimed to, work authentically and

ethically to create an inclusive and socially just learning environment for all members of the

Northwestern University community. The model of CIC is adapted from the tri-sector model of

cultural practice by Jenkins & Walton (2008). The tri-sector model is commonly used in student

affairs it provides a framework based on three components education, enrichment, and

engagement. CICs structured its office by using the tri-sector model effectively. The Social

Justice Education department was created for the purposes of education described in the tri-

sector model above. The Social Justice Education department creates co-curricular educational

opportunities in partnership with the student community to foster conversations across difference

and support actions that create social change on campus (p. 9). The Student Enrichment Services

department was created for the purpose of enrichment described in the tri-sector model. The

Student Enrichment Services department works with low-income first generation students to

enhance their academic success, personal development, and professional growth (p. 9). The

Multicultural Student Affairs department was created for the purpose of engagement described in
PROGRAM EVALUATION NORTHWESTERN 5

the tri-sector model. Multicultural Student Affairs enriches the cultural experience of

Northwestern through leadership and education programming providing opportunities for

community engagement (p. 8). This formative assessment will focus specifically on SJE and an

evaluation and assessment will focus primarily on just one of the programs that SJE offers which

is PIE. SJE and its other programs are highlighted in the next paragraph.

SJE operates under a director, two assistant directors, and one graduate assistant. The

specific programming that SJE is responsible for is Sustained Dialogue, Analyzing Whiteness,

Step Up Bystander Intervention, and Peer Inclusion Educators. SJEs programming is mostly

funded through private grants and donors. The Assistant Director is hoping to hire a graduate

assistant whose primary focus will be to work on the enhancement of the PIE program. The

funds will be used to put a three-year plan in motion. In previous years, the department did not

receive any funding for Peer Inclusion Educators and all of the peer facilitators were working on

a voluntarily basis. The peer facilitators still work on a voluntary basis to date, but this is

expected to change over the next three years. The effectiveness of voluntary peer educators will

be included in our formal assessment and evaluation of the PIE program.

Program Context
Peer Inclusion Educators (PIE) is, a program started by students, for students. PIE

dedicates its workshops to focus on social justice, social identities, and dialogue surrounding the

Northwestern community (Social justice education: Peer inclusion educators, 2016). Training is

held for all students who would like to become a peer facilitator. During this training, the

Assistant Director trains the peer facilitators to understand their social identities, meaning

gender, race, ethnicity, sex, religion, socioeconomic status, etc; students are able to explore their

identities and begin to understand what identities give them power and privilege in the United

States and what identities are oppressed. The workshops also put an emphasis on the
PROGRAM EVALUATION NORTHWESTERN 6

intersectionality of each participant privileged and oppressed identities. This is completed by

having the facilitators complete assignments such as the social identity grid, followed by

discussions about social justice terms such as target, agent, inclusion, collusion, privilege,

prejudice, discrimination, and oppression. After the facilitators spend time reflecting on their

own identities they then complete mock facilitations to prepare them to engage in dialogue with

student groups on Northwesterns campus.

Once facilitators have completed their training student groups will have access to

complete requests for one time workshops by visiting the SJE website, which has a separate link

for programming requests under the CICs website. Students can then design their workshop by

requesting activities such as the language activity, social identity grid, or both. The language

activity looks at terms such as Target, Agent, Inclusion, Collusion, Privilege, Prejudice,

Discrimination, Oppression, and Social Justice. The group is then asked to come up with a

definition of each term collectively to begin the process of dialogue. The Social Identity Grid is

an activity that lists social identities such as ethnicity, ability, gender, sex, and race. The idea

with these activities is to have students think about their most salient identities and the identities

that they think about the least. For most people, the identities that you think about least are often

the identities that give you power and privilege. When students can identify parts of their

identities that gives them power and privilege, a rich dialogue is able to happen that plays a

crucial role to the relevance and importance of a program such as PIE to exist on a campus like

Northwestern.

Program Description
The Peer Inclusion Educators (PIE) program has three different workshop topics for

which student groups such as residence assistants, peer mentors, sororities, and fraternities can

request. These workshop topics include: general inclusion facilitation, social justice 101, and
PROGRAM EVALUATION NORTHWESTERN 7

inclusive language. Student groups also have the privilege of designing their workshops to fit

the needs of their group. For example, the general inclusion facilitation workshop outlines the

basics of inclusion and allows for the opportunity for your group to brainstorm ways to be even

more inclusive (Social justice education: Peer inclusion educators, 2016). The social justice 101

workshop focuses on tailoring conversations about social justice by having the group engage in

an intimate dialogue. An intimate dialogue is able to happen because workshops will only take

place with a group no larger than 20 students. Peer Inclusion Educators need to be able to

engage in conversations with group. Often times when a group is too large deep conversations

are unable to happen. The assistant director found that it is easier for facilitators to control the

space in an intimate setting (Social justice education: Peer inclusion educators, 2016). The final

workshop option is inclusive language. This facilitation workshop focuses on implementing

activities such as the language activity by putting those terms within the context of the culture

specifically at Northwestern University. Student groups have the opportunity to create a

workshop engaging all three of these topics, or only one topic, depending on the needs of the

person or group requesting a PIE workshop. All participants are expected to engage in dialogue

and come up with ways in which the language that they use on a daily basis can be more

inclusive by using a critical lens to analyze the meaning of social justice terms. A critical lens is

a tool to analyze works of literature. During the language activity facilitators are asking the

group to define social justice terms and then put them into the context of their lives as a

Northwestern student which is what is meant by using a critical lens when engaging in the

language activity.

Selection of Peer Facilitators


Peer Facilitators are chosen through a selective application process. The Assistant

Director of SJE, Noor Ali, has created a live application on the website
PROGRAM EVALUATION NORTHWESTERN 8

http://www.northwestern.edu/socialjustice. The questions on the website include why an

applicant would like to become a peer inclusion educator, their facilitation experience,

challenges they anticipate in this position, and how the applicant brings fun into the program.

Once the facilitators have been selected, they are required to attend two-day training from 9 AM-

5 PM on Northwesterns campus.

The trainings typically launch in November for new recruits. During the beginning of the

quarter (September through October), previously trained peer facilitators are encouraged to sign

up for the workshop requests that have been submitted to SJE. Noor Ali sends an email to all

facilitators with a description of the group that has completed a request along with dates and

times of the request. Peer facilitators then sign up for workshops that will work around their

schedule. The peer facilitators also have monthly meetings. During the monthly meetings, the

facilitators discuss what challenges they have faced and the group provides suggestions on how

they can improve.

Program Purpose
PIE has four learning outcomes (Social justice education: Peer inclusion educators

(2016), which include:

Educate campus peers on how to best allow for the growth of an inclusive community.

Allow for reflection on ones self , backgrounds, and biases.

Identify issues on campus and address them through active and passive programming.

Encourage dialogue around issues of difference and to develop competent mechanisms

for addressing these issues.

This program has been identified as an introductory sister program to the Sustained Dialogues

program, which encourages in-depth conversations once a week for 90 minutes throughout the

quarter.
PROGRAM EVALUATION NORTHWESTERN 9

Program Stakeholders
There are direct and indirect primary stakeholders in the PIE program which includes the

university at large, students, the vice president of student affairs, the director and assistant

director of SJE and community partners. The students of Northwestern University are

stakeholders because the office of SJE was created by students for students. Students actively

seek out workshops. Student facilitators are trained so they are learning life skills that they can

take with them and sharing knowledge with their peers. Assessment on the effectiveness of PIE

is determined by the participants (students, faculty, and staff) of the workshops as they are given

an evaluation after each workshop. The evaluation measures the effectiveness of the content and

measures the effectiveness of the facilitators. Faculty and staff seek out workshops as well

which would make the university at large indirect primary stakeholders. After each workshop all

participants complete surveys and their feedback is critical to how PIE will continue to operate.

PIE also trains a number of students to become active facilitators of PIE. The students

engagement with this program is what keeps it running. The vice president of student affairs is a

primary direct stakeholder as they are the chief personnel officer of the university and they

directly advise the president of all non-academic activities on campus including PIE (About:

Northwestern university, 2016). The director of SJE is a primary direct stakeholder. The

director is responsible for communicating with alumni and possible donors to fund PIE. The

assistant director of PIE is a direct stakeholder. The assistant director trains all student

facilitators and works collaboratively with the SJE staff to make sure PIE is being run

effectively. This includes recruiting student facilitators, training all participants, and facilitating

workshops.
PROGRAM EVALUATION NORTHWESTERN 10

Logic Model Overview


The PIE program logic model (See Appendix A) focuses on the operations of each of the

three PIE workshops. According to the W.K. Kellogg Foundation (2004), a logic model is a

systematic and visual representation of the relationships and resources provided to create the

intended outcomes of a workshop. We start assessing the logic model by looking at the

following themes: situations, inputs, outputs (activities and participation), followed by outcomes

and impact (short-term, medium-term, and long-term outcomes).

The situation begins with an acknowledgment that there has been no consistent data

produced to assess the effectiveness of each PIE workshop. As previously mentioned, thousands

of people have participated in a PIE workshop but due to capacity and other restraints, the need

for consistent assessments has not been present until now. The presence of a new direction in

SJE not only contributes to the situation, but is also identified as an input in order to operate the

program. The input category also includes two assistant directors, one graduate assistant, one

director for CIC, and the volunteer student facilitators. Another contributing input is a newly-

awarded $15,000 grant for five years for the continuation of the PIE program, which can

influence the effectiveness and implementation of assessments throughout all three PIE

workshops.

Activities in the outputs category are mostly attributed to student facilitator training and

continued dialogues about their experiences as PIE facilitators. Their facilitation expands not to

just training students but also student leaders in various departments such as residence life and

student activities. By offering spaces where the student facilitators can be vocal about their

experiences, SJE is able to check-in with them and enhance their facilitation skills by providing

opportunities for self-reflection and growth. It is noteworthy to mention that not all of the

participants are undergraduate students, but also graduate/professional students, faculty, and/or
PROGRAM EVALUATION NORTHWESTERN 11

staff. We did not identify any major significance in creating two different assessments for both

populations, so one comprehensive assessment will be used for both undergraduate and

graduate/professional students, faculty, and staff who attend a PIE workshop.

The three workshops that are currently offered (Language, Social Identity, and

Inclusivity) all have newly developed learning outcomes. The short-term, medium-term, and

long-term goals of each workshop vary primarily by their topics. These outcomes were

developed based off of the curriculum of each workshop. Although there is some overlap with

terminology, it became very evident during this process that the learning outcomes for each of

the three workshops were slightly different and needed to be addressed through individual

assessments.

Additionally, the three workshop templates are often combined throughout the year,

depending upon the request of the student group or organization. For example, a student

organization can request a workshop on language and social identity, or they can request one

workshop incorporating all three workshops. In these situations, students will receive more than

one assessment immediately after the workshop to assess the effectiveness of these outcomes.

Previous Evaluation
Currently, SJE has no data regarding effectiveness of the workshop content. There is one

evaluation (see Appendix B) that was to be distributed after each PIE workshop. Due to a low

staff in SJE last year which only consisted of an assistant director and one graduate assistant at

the time no data was able to effectively be assesses. PIE is also a fairly new program that has

only been around less than five years. Now that SJE has hired a Director more attention will be

given to the PIE program and assessments are planning to happen with the completion of this

evaluation. The new staff of SJE has hired an undergraduate student who will focus on
PROGRAM EVALUATION NORTHWESTERN 12

collecting data and inputting the survey items into SPSS for the 2017-2018 academic school

years.

The number of workshops and participants who attended a PIE workshop skyrocketed

between 2014-2015 and 2015-2016. According to the assistant director, there were a total of 25

workshops that reached 538 participants in 2014-2015, while in 2015-2016 there were 79

workshops that reached a total of 2,518 participants. It is unclear why there was such a drastic

shift in the amount of workshops and participants within one year but it is clear that an

assessment and evaluation is needed.

General Evaluation Approach


Completed assessments for PIE are very limited. As previously mentioned, there is only

one assessment in their database that was distributed over the course of one year. Since the

initial onset of the PIE program, no formal assessment has been consistently created for the

student facilitators or the participants. Given the high number of participants during the 2015-

2016 academic years, we decided to focus our general evaluation approach to assessing the

participants of each workshop and evaluating the effectiveness of the learning outcomes to each

workshop topic.

Our general evaluation approach will be primarily outcomes based. An evaluation is

considered to be formative if the primary purpose is to provide information for program

improvement (Fitzpatrick et al). The evaluations we created will focus on the effectiveness of

the PIE workshop and contain elements that address ways in which the workshop can be

improved for future purposes on the topics of general inclusion facilitation, social justice 101,

and inclusive language as well as providing critical feedback to the student facilitators so they

can improve their skills.


PROGRAM EVALUATION NORTHWESTERN 13

The learning outcomes associated to PIE are inclusive to all three major topics that are

offered through the PIE program. For this reason, we created an assessment that can effectively

assess all three of the major workshop topics offered by PIE facilitators. We would like to

effectively measure to what extent the goals that PIE is intended to create are achieved during

these one-time workshops. Ideally, after each workshop the participants will be given a survey

asking questions related to the effectiveness of each workshop based off of content and the

facilitators engagement.

As evaluators we decided a formative assessment on PIE would be beneficial to the data

we are trying to collect, analyze, and assess. All data that is collected during this assessment will

be continuously discussed with the stakeholders, specifically in regards to this process, and

throughout this implementation of this evaluation. This evaluation is a processed based

assessment that will analyze short-term, medium-term, and long-term outcomes based on survey

items that participants will complete.

Quantitative Approach
Participants
Participants of PIE workshops vary across the campus of Northwestern University. PIE

facilitators do not go out and recruit student participants. Instead, student groups can request a

PIE workshop (Social justice education: Peer inclusion educators, 2016). Student groups are

defined as any student organization on campus which includes but is not limited to; resident

halls, fraternities, sororities, peer advisors, student government, etc. Many student affairs staff

members will request PIE workshops for their student groups for which they supervise. Staff

members also participate in the PIE workshops so the data will include some surveys completed

by staff members. All of the questions on the PIE workshops post-surveys does include questions
PROGRAM EVALUATION NORTHWESTERN 14

that would be targeted for a staff member or student within the Northwestern community (see

Appendices F-H).

Survey Design
An indirect assessment will be conducted to evaluate the short-term, medium-term, and

long-term outcomes of each of the three workshops: the Language Activity, the Social Identity

Activity, and the Intersectionality Activity. A longitudinal approach will be employed because

an initial, six month, and one year post-survey related to the workshop that they have

participated in will be distributed to participants after each workshop. The groups of students

who participate in PIE workshops are learning social justice key terms right after participating in

a PIE workshop. The three workshops also ask the participants to reflect on the knowledge they

have just learned and how it can apply to spaces outside of the workshop. All of the students are

given individual post-surveys after each workshop, which assesses their knowledge of the social

justice key terms they have learned before participating in a PIE workshop(s) and their

knowledge of the same social justice key terms they learn after participating in the workshop(s).

Our method includes convenience sampling because every participant will complete an

assessment after each workshop (Banta & Palmoba, 2015). Our approach will also include a

longitudinal method of sampling because we ask the participants to complete an initial, six

month, and one year post-survey related to the workshop that they have completed. The survey

response process is complex! This involves comprehension, retrieval, judgment, and response

processes (Tourangeau et al., 2000). We are expecting a lot from the participants by assessing

their comprehension of the material that was presented to them after each workshop. We then

must retrieve the surveys and assess the responses hoping they will provide feedback on how we

can improve the workshops. Not only are we asking for feedback to improve the workshops but

also we are asking the participants to assess how well the facilitators were in each workshop.
PROGRAM EVALUATION NORTHWESTERN 15

The process is complex and we must take this process seriously to ensure funding will continue

for the program.

Instrument Description
PIE has three different workshops; Language Activity, Social Identity Activity, and the

Intersectionality Activity. All three workshops have slightly different surveys that are

administered to each participant after the workshop (see Appendix F-I). The Language Activity

post-survey includes a total of 23 questions (see Appendix F). The first 18 questions are

measured using the Likert Scale (strongly agree, agree, neutral, disagree, and strongly disagree).

The first 12 questions of the survey assess their knowledge of social justice key terms before and

after the workshop. The first six questions gauges their knowledge with the following terms

before participating in the Language Activity workshop these social justice key terms include;

target/agent identities, privilege, power, oppression, discrimination, and collusion. The next six

questions of the survey gauges their knowledge of the social justice key terms mentioned above

after the workshop. The third section of the survey asks participants how they would apply the

social justice key terms learned during this workshop at Northwestern University. This means

defining what discrimination may look like within their community at Northwestern. The fourth

section of the survey ask participants to measure the skills of the facilitators by asking them a

series of questions that measures the environment provided, approachability, timeliness, and

engagement during the workshop. The last section of the survey asks for demographic

information including age, citizenship/nation of origin, class standing, gender, and race.

The Social Identity Activity post-survey includes a total number of 24 questions (see

Appendix G). The first 19 questions are measured using the Likert Scale (strongly agree, agree,

neutral, disagree, and strongly disagree). The first ten questions of the survey assess their

knowledge of social justice key terms before and after the workshop. The first five questions
PROGRAM EVALUATION NORTHWESTERN 16

gauges their knowledge of social justice key terms before the workshop which includes;

target/agent identities, privileged identities as it relates to their own social identity, salient

identity, defining their own salient identities, and how their salient identities are connected to the

terms privilege and oppression. The next five questions assess their understanding of the social

justice key terms mentioned above after the workshop. The third section asks participants to

apply social justice terms discussed during this workshop at Northwestern University. This

means to define what it would look like for them as a student to understand their salient identities

and how it may give them power and privilege within the community at their school. The fourth

section of the survey ask participants to measure the skills of the facilitators by asking them a

series of questions that measures the environment provided, approachability, timeliness, and

engagement during the workshop. The last section of the survey asks for demographic

information including age, citizenship/nation of origin, class standing, gender, and race.

The Intersectionality Activity post survey includes a total of 20 questions (see Appendix

H). The first 15 questions are measured using the Likert Scale (strongly agree, agree, neutral,

disagree, and strongly disagree). The first ten questions of the survey assess their knowledge of

social justice key terms before and after the workshop. The first five questions gauges their

knowledge with the following terms before participating in the Intersectionality workshop key

terms include; privilege, power, oppression, intersectionality, and solidarity. The next five

questions of the survey gauges their knowledge of the social justice key terms mentioned above

after the workshop. The third section of the survey asks participants how they would apply the

social justice key terms learned during this workshop at Northwestern University. This means

defining what solidarity may look like within their community at Northwestern. The fourth

section of the survey ask participants to measure the skills of the facilitators by asking them a
PROGRAM EVALUATION NORTHWESTERN 17

series of questions that measures the environment provided, approachability, timeliness, and

engagement during the workshop. The last section of the survey asks for demographic

information including age, citizenship/nation of origin, class standing, gender, and race.

For all of the post workshop surveys, we are collecting unique ID markers such as their

Northwestern Student ID so that we can make sure to follow up with them during our

administration of our six-month and one year post-surveys. Although all surveys are confidential

they are not anonymous because each participant indicates their student ID number on all of the

surveys. All surveys that students will complete will ask the same questions from the initial

post-surveys (see Appendices F-I).

Pilot Testing
We will pilot test the first two student groups who request a PIE workshop during the fall

quarter. We will ask the two student groups if they could participate in a series of all three

workshops. After each workshop we will immediately collect the initial post-surveys. We will

begin to measure how effective all three of the workshops were by benchmarking and comparing

the before and after survey responses of the social justice terms reviewed in each workshop. We

will assess and evaluate if the facilitators were engaged, on time, created a brave space, were

approachable, and promoted full participation throughout each workshop. All surveys will be

analyzed using SPSS. We will also provide feedback to all participating facilitators so that we

can make sure the workshops are improving. The purpose of pilot testing is to make sure our

survey items are clear, to see how long the surveys take to complete, and to see if the

flow/instructions are clear to the respondents.

Implementation/Administration
Administrations of the post-surveys for each workshop will be handled by the facilitators

present in the space. All facilitators are expected to administer these surveys during the last 10-
PROGRAM EVALUATION NORTHWESTERN 18

15 minutes of their workshop. The expectation for the student groups should have been set

before they participated in the workshop. The facilitators are expected to let all participants know

that a post- workshop survey needs to be completed during the last 10-15 minutes of the

facilitation. The facilitators are then expected to collect the surveys from all participants. All

facilitators will give the surveys to the assistant director, Noor Ali who will then give us the

surveys. We will work in collaboration with the Student Assessment Center to evaluate the post-

surveys for all three workshops. We will only assess the post- workshop surveys from all of the

requested workshops from the fall, winter, and spring quarters. The dates during this time are

September 18, 2017- June 11, 2018. Assessing the data collected during this time will help us

provide immediate feedback to the PIE team so we can improve the quality of the workshops

based on our short term outcomes for the program.

Statistical Analysis/ Results Presentation


All survey data from all three workshops will be organized, described, and collected for

statistical analysis. All information will be coded and analyzed using SPSS. The categorical

demographics asked in all three surveys such as the race and student class standing are nominal

variables which can help provide an in depth evaluation about who are participating in the

workshops. The findings will help us gauge who are our target audience. This will help us

analyze if there is only certain student groups, nationalities, and age ranges requesting

workshops.

As previously mentioned, almost all of the questions on all three workshop surveys

provide interval data with the use of the Likert scale which helps measure the dependent

outcome variables. The information can be treated continuously and can provide useful

information because of the running of means as a descriptive phase for each survey. The mean

scores can be used to gauge the knowledge of social justice key terms before and after the
PROGRAM EVALUATION NORTHWESTERN 19

workshops. The mean scores can also measure the effectiveness of facilitators who are not paid

but are volunteering their time. The mean scores can also measure social justice application of

the social justice key terms reviewed in each workshop before and after student groups and staff

participate (see Appendices F-I). An ANOVA will be used to compare these three means with

each other to see if they significantly differ from one another. This will be helpful as it will help

us determine between which workshops are the most impactful to our participants and if not how

we can improve the content so they will have a better understanding of social justice concepts.

We will also want to conduct comparable analyses of all three workshops to see which workshop

students found to be more beneficial or if participants found all workshops to be equally

beneficial.

Next we will want to analyze how the responses change over time. This can be completed

by using a sample t-test that will provide us with inferential statistics that can gauge if participant

responses will change over time. We will compare results from the initial post-survey, to the six-

month survey, and the one-year survey. This is a longitudinal method that will help us analyze if

the PIE workshops are creating change for the betterment of the campus at Northwestern

University.

Qualitative Approach
Qualitative Rationale
The qualitative approach of the evaluation of the PIE program will include focus groups.

Focus groups are implemented to understand participants diverse experiences throughout the

three workshops that PIE offers, such as the Language Activity, Social Identity Activity, and the

Intersectionality Activity, to help provide a deeper meaning of the data collected during the

quantitative analysis (Schuh, 2009). The purpose of the focus groups is to use the findings to

better explain the extent to which student outcomes are met. The outcomes of the workshops are
PROGRAM EVALUATION NORTHWESTERN 20

assessing the effectiveness of the content that the participants are engaged with as well as the

effectiveness of the student facilitators ability to engage students with the content. We also

want to provide students with a space to provide a deeper meaning of the social justice key terms

they have learned. We hope participants will provide reflections and feedback on the facilitators

abilities to provide a brave space (a space where students felt they could be vulnerable and safe

to do so), their timeliness, and the effort each facilitator put into the workshops in which they

have participated as well. The focus group protocols provide probes and questions that ask the

participants why they signed up for the workshops. We want to assess if students are signing up

because they were forced to do so or if they genuinely were interested. The in depth responses

will help us analyze this question. Another probe seeks to understand the type of student that is

signing up for workshops by assessing which student organization or group they are a part of and

how they got involved to participate in the workshops. The last probe seeks to understand what

information they have learned from the workshops based on the conversation that will take place

because of the goals for each workshop (see Appendices J-L).

The program stakeholders desire to launch the focus groups for each PIE workshop

because no evaluation or assessment has been complete to date. The stakeholders are anxious to

collect data and receive feedback on each individual workshop so improvements can be made for

the following 2018-2019 school year

Analytical Approach
Participants who completed the cross-sectional surveys (see Quantitative section) will all

have the opportunity to participate in a focus group. The last question of each survey asked

participants if they would be interested in participating in a focus group for the workshop(s) they

have participated in. The cross-sectional survey asked participants for their Northwestern ID

number. Based on their response their name will be looked up in the system. An email will be
PROGRAM EVALUATION NORTHWESTERN 21

sent to all participants that indicated they would like to participate in a focus group (s) (please

see Appendices N-O). Program evaluators will use maximum variation sampling because of the

expansive and diverse nature of workshop participants. This strategy recognizes general themes

or patterns where different opinions exist (Schuh, 2009). This is important because if you

deliberately try to interview a very different selection of people, their aggregate answers can be

close to the whole population.

Convenience sampling was used throughout the quantitative surveys but evaluators

recognize the importance of varying sampling when analyzing qualitative data. Depending on

the variance of participants for the focus groups, we are reaching out to 30 participants all of

whom will be selected randomly based on their responses. We are hoping to have 5-10

participants sign up for a focus group for one of the following workshops that they have

participated. We understand that some participants may have experience with multiple

workshops but we will ask that they only participate in one of the focus groups of their choice.

Initially an email invitation will be sent out on April 11, 2018 (see Appendix N) to all

participants that expressed interest in participating in the focus groups. Participants will have two

weeks to decide if they would like to participate. A reminder email will be sent out on April 25,

2018 (see Appendix O). Once participants confirm that they will be attending a confirmation

email will be sent out immediately after their response. The emails will confirm the date, time,

and location of where each focus group will take place. We will also make sure to highlight that

participants are only to choose one focus group for one workshop that they have participated in

even if they want to participate in multiple workshops.

Focus Group Plan


There are three different focus group protocols for each workshop. According to the

focus group protocol for the Language Activity (see Appendix J) participants will begin with an
PROGRAM EVALUATION NORTHWESTERN 22

introduction from the graduate assistant, assistant director, or director of SJE who begins the

focus group with a simple thank you to the participants for dedicating their time to the activity.

Participants are then explained that the focus group activity will take 60 minutes. It is then

explained to participants that the information gathered today will be shared with SJE staff and be

used to improve the quality of PIE language workshop for future students. The feedback that the

participants provide will be recorded by note takers and a tape recorder so we can be sure to

accurately capture everyones responses. The facilitators make sure to reiterate that the

information will be private and they will not be specifically identified as the participants

afterwards. The facilitators of the focus group will then pass out a consent form to every

participant (see Appendix M). It is explained that the consent forms will protect their

confidentiality while participating in the workshops. After, the consent forms have been

collected the introductory question will be asked to begin the dialogue amongst the focus group.

The first question simply asks participants why they signed up for the workshop. Probes are then

asked so that participants can engage in the dialogue in a deeper way, providing more nuance

related to their experiences. Each probe tries to get participants to answer the question in a way

that reflects the goals of each workshop. This process will be completed in all three focus group

workshops. See Appendices J-L to see the learning outcomes and goals for each focus group

based on which workshop participants attended. After the focus group is complete, facilitators

will ask the participants for any feedback or recommendations to improve the workshops. This

question is not limited to the content of workshop but also the ability of the facilitators for the

workshop in which they participated. Again, this process is completed for each individual

workshop with different participants in each focus group.


PROGRAM EVALUATION NORTHWESTERN 23

Focus Group Implementation


The focus groups will be implemented in late April 2017, which is approximately six

months after student/staff have participated in a PIE workshop during the fall quarter. A pilot test

will not be completed on the focus groups beforehand due to a small timeframe and budget costs.

The office of SJE also has a small staff that may not be able to prepare for a pilot focus group

and plan for the focus groups that will be used to assess the workshops. The graduate assistant,

assistant director and director of SJE will be facilitating the focus groups. The assistant director

(Noor Ali) and Director (Robert Brown) are key stakeholders in this evaluation. It will be great

for them to hear firsthand what the participants have learned and to hear feedback about the PIE

facilitators that they select for these roles. The focus groups will be conducted on campus for the

convenience of the student/staff. Light refreshments will be provided in each focus group as

incentives. We recognize that the office of SJE only have three staff members. More staff

members from CIC will be asked in advance to hold focus groups on campus on the same dates

in different rooms across campus.

Focus Group Analysis


Focus group analysis will occur through a process of transcription, coding schemes, and

data analysis. The three focus group sessions will be transcribed. Once the transcripts are

finalized, the evaluators will listen to the audio recordings, review the notes taken by the

moderator and assistant moderators, and read the transcripts. The two evaluators will code each

transcript separately along with the assistance of the Student Assessment Center. A priori codes

and any additional ones that emerged through the coding process, the program goals and learning

outcomes could be explained to stakeholders. Inductive coding will be used as we take from the

findings to theorize the themes. We feel findings will emerge as we are assessing the surveys.
PROGRAM EVALUATION NORTHWESTERN 24

The cutting and sorting technique will be used when analyzing data because it allows the

evaluators to analyze responses to create themes amongst the responses. This method is useful

because it allow us to recognize and pull sub-themes while reading through the evaluations to

pull themes that seem to be important. It is also useful to cut and sort through themes while

laying them in envelopes to create neat piles of the information. The data will also be

triangulated by using the quantitative and qualitative results in order to gain more holistic

interpretations of participants responses from the cross sectional surveys and the focus groups.

Limitations
Due to the structure of CIC and SJE, all of the moderators are very familiar with the PIE

program and the learning outcomes for each workshop. This knowledge that the moderators have

of the program can spark biases when asking participants what they have learned during the

workshop. We will have to recruit a volunteer moderator since there are only two evaluators to

facilitate the third focus group. We also recognize the power dynamics in the room from the

professional staff to students. We know the students may be intimidated by staff so it is even

more important that the evaluators are present in the rooms to ensure the space is as comfortable

as it can be for participants.

Another limitation is acknowledged in our ability to create an effective long-term

outcomes-based assessment. We envisioned this assessment to be distributed via email to all of

the participants of a PIE workshop. The questions for these assessments would be created in

collaboration with the stakeholders and will be influenced by the responses gathered from the

short-term and medium-term focus groups. The structure would be very similar to the

assessment distributed immediately after participating in a PIE workshop. Due to this intentional

collaborative effort on behalf of the stakeholders and SJE staff, in addition to not being

redundant on the types of questions we asked in the final post-year assessment, we did not move
PROGRAM EVALUATION NORTHWESTERN 25

forward with creating what a long-term outcomes-based assessment could look like for each of

the PIE workshops.

Result Presentation
Qualitative data will be presented to key stakeholders in several ways. The responses

from the focus group will be presented as themes to provide feedback on the workshops and the

facilitators abilities. The themes will report, negative, neutral, or positive responses. The themes

will be collected by drawing from direct quotations from the transcription of the notes and

recordings of the focus groups. Bar graphs can also be used to show the major themes, for

example the frequency that each theme emerged in the focus groups.

Next Steps
The PIE program at Northwestern University is a relatively new program. It has been in

existence for five years. The staff of SJE has never had the capacity to complete a formal

assessment. This evaluation and assessment will be very beneficial especially for a relatively

new program. We want to provide feedback on how PIE can improve by analyzing the cross

sectional surveys for the quantitative data and the responses of the focus groups for the

qualitative data. Our next steps are to ensure the SJE staff will receive results and feedback

efficiently so the PIE program can begin implementing changes for the 2018-2019 academic

years.
PROGRAM EVALUATION NORTHWESTERN 26

References

About: Northwestern university. (2016). Retrieved from http://www.northwestern.edu/.

Banta, T.W., &Palomba, C.A. (2015). Assessment essentials: Planning, implementing, and

improving assessment in higher education (2nd ed.). San Francisco, CA: Jossey-Bass

Campus Inclusion & Community. (2016). Division, Campus & Departmental

Information.Unpublished internal document.

Fitzpatrick, J., Sanders, J., &Worthen, B. (2011). Program evaluation: Alternative approaches

and practical guidelines (4th ed.) New York, NY: Longman.

Jacobs, J. (Ed.). (2015, August 24). University halts reorganization of Black House.

Jenkins, T. S. (2010). Viewing cultural practice through a lens of innovation and intentionality.

Culture centers in higher education: Perspectives on identity, theory, and practice, 137-

156.

Multicultural Center space.The Daily Northwestern. Retrieved from

http://dailynorthwestern.com/.

Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA:

Jossey-Bass.

Social justice education: Peer inclusion educators. (2016). Retrieved from

http://www.northwestern.edu/socialjustice/programs-and-events/peer-inclusion-

educators.

Tourangeau, R., Rips, L. J., &Rasinski, K. (2000). The psychology of survey

response.Cambridge University Press.

W.K. Kellogg Foundation. (2004). Using logic models to bring together planning, evaluation,

and action: Logic model development guide. Battle Creek, MI: Author. (pp. 1-48).
PROGRAM EVALUATION NORTHWESTERN 27

Appendices
Appendix A: PIE Program Logic Model

OUTPUTS
SITUATION INPUTS
Activities Participation

No consistent 2 Assistant Train student Undergraduate


assessment to Directors (only 1 facilitators who and Graduate
measure the directly oversees volunteer to Students
participants PIE) participate in PIE
No data that Host monthly Student Leaders
demonstrates the 1 Graduate meetings for
effectiveness of Assistant facilitators to Faculty and staff
these workshops. extend dialogue at Northwestern
The Assistant 1 Director of SJE on identity, power, University
Director who and privilege
oversees PIE and 1 Director of CIC Conduct Multiple campus
SJE is now workshops partners including
working with a Volunteer student surrounded on Residence Life,
new director of facilitators who three main Greek Life, and
SJE. get trained for two themes: Language, Athletics
days (9 AM- 5 Social Identity,
PM) before the and Full-time Staff
academic year Intersectionality from SJE and CIC
starts Present to campus and one Graduate
partners such as Assistant from
Budget from a Residence Life to SJE
newly received expand the scope
private donor for of their workshops
$15,000/per year
for the next 5
years until 2021
PROGRAM EVALUATION NORTHWESTERN 28

OUTCOMES + IMPACT
Short-term Medium-term Long-term
(Learning/Awareness) (Action) (Conditions)
Language: Language: Language:
Participants will be able to Participants will become Participants will feel
define key social justice more familiar with these compelled to engage in
terms terms and concepts more dialogues and
Participants will be able to Participants will recognize discussion of these terms
apply key social justice when certain terms and and concepts
terms to concepts related to concepts are at-play on Participants will apply
power, privilege, and campus these terms and concepts
oppression through self-reflection and
Participants will be able to Social Identity: interactions with other
identify examples of how Participants will apply students on campus
these terms and concepts these key terms and
are experienced on campus concepts to their own social Social Identity:
identities Participants will be able to
Social Identity: Participants will not clearly articulate their ideas
Participants are able to reinforce notions of power, and thoughts on social
reflect on how they view privilege, and oppression justice in relation to their
key terms and concepts on campus social identities
related to social justice Participants will become
relate to their own and Intersectionality: empowered to engage in
others social identities Participants will apply the difficult conversations
Participants can define the definition of target and agent related to social identities
meaning of salient identities to the greater campus on campus
identities community
Participants can Participants will actively engage Intersectionality:
acknowledge how salient in self-reflection in order to Participants will use self-
identities may be deconstruct notions of power, reflection of their target and
intertwined with privilege privilege, and oppression on agent identities as tools to
and oppression in a group campus combat power, privilege, and
setting oppression on campus
Participants will seek further
Intersectionality: dialogues and discussion on
Participants will be able to topics and concepts related to
define both target and agent social justice on campus,
identities, including their particularly through SJE
own.
Participants will gain
knowledge of how
privilege, power, and
oppression intersect with
their social identities
Participants will be able to
define solidarity
Participants will be able to
identify practices that
exhibit solidarity with
social identities other than
their own
PROGRAM EVALUATION NORTHWESTERN 29

Assumptions:
All participants will students come to the program willing to discuss their identities and
awareness of other social identities
All participants students are willing to work through their discomfort in talking about
identities and difference
All participants are going to respect each others learning process throughout the
workshop
All participants will ask questions when they seek clarification on a particular idea or
topic
All participants will be able to clearly articulate their views and opinions on topics related
to social justice and/or their social identities
All participants will engage in the three core topics though some action item post-
workshop

Environment:
New SJE director may want to continue, or reshape, the PIE program
The effect of current events such as #BlackLivesMatter on student engagement around
social justice issues and critical self-reflection
Prevalence of bias incidents on campus among students, faculty, and staff
An academically-driven institution environment may impact the quantity and quality of
engagement among participants in a PIE workshop

(Adapted from The University of Arizona, College of Agriculture & Life Sciences: https://extension.arizona.edu/evaluation/content/logic-
model-worksheets)
PROGRAM EVALUATION NORTHWESTERN 30

Appendix B: Previous Evaluation


SJE Workshop Evaluation

Workshop topic: Program/Organization:

Facilitator(s): Date:

Your feedback is an important element in our ability to evaluate the effectiveness of our workshops and to develop
sessions that meet your needs and expectations. Please complete this evaluation and return it at the end of the
workshop.

RATINGS:
KNOWLEDGE:
Vaguely Completely
1. Before this session, I felt like I understood the workshops content:
1 2 3 4 5
Vaguely Completely
2.After this session, I feel like I understand the workshops content:
1 2 3 4 5
Disagree Agree
3. The content presented increased my level of self awareness:
1 2 3 4 5
4. This workshop increased my understanding of perspectives other than Disagree Agree
my own: 1 2 3 4 5

Comments:

ENVIRONMENT:
Disagree Agree
5. I feel this was a safe environment for my participation:
1 2 3 4 5
Disagree Agree
6. This workshop allowed me to step outside my comfort zone:
1 2 3 4 5

Comments:

APPLICATION:
7. There is an increase in my ability to apply the workshop content to my Disagree Agree
daily life: 1 2 3 45
8. There is an increase in my desire to continue conversations related to Disagree Agree
thecontent presented: 1 2 3 4 5
9. Because of this workshop, I am more comfortable interacting with people Disagree Agree
of different identities from my own: 1 2 3 4 5
Disagree Agree
10. I would be interested in attending future related sessions:
1 2 3 4 5

Comments:

FACILITATORS:
11. The facilitators were effective in their delivery of the workshops Disagree Agree
content: 1 2 3 4 5
PROGRAM EVALUATION NORTHWESTERN 31

Comments:

OVERALL:
Not Beneficial Beneficial
12. My attendance at the workshop was:
1 2 3 4 5

FEEDBACK:
13. Have you attended CIC Workshops before? (Circle one) Yes No

If yes, how many and what topics?

14.Which component of the workshop was most beneficial to you? Why?

15.Which component of the workshop was least beneficial to you? Why?

16.What information are you still unclear about with regard to the content presented?

17. What topics would you suggest for additional sessions?

18.Other comments:

DEMOGRAPHICS:

Age: ____________ Citizenship/Nation of Origin: ________________________________________

Class Standing:First year SophomoreJunior Senior Graduate/Professional Student


Staff Faculty Member Not already listed: ________________________

Gender: Woman Man Transgender Not already listed _______________________________


PROGRAM EVALUATION NORTHWESTERN 32

Race (check all that apply):Native American /Alaskan Native Black or African American
Asian/Asian American Native Hawaiian/Pacific Islander
Latino(a) White/Caucasian
Middle Eastern/North African
Not already listed: ______________________________

YOUR FEEDBACK IS VERY MUCH APPRECIATED. THANK YOU!


Please return this evaluation to the facilitators upon completion.
PROGRAM EVALUATION NORTHWESTERN 33

Appendix C: Abbreviated Social Justice Key Concepts

Racism: The systematic subordination of members of targeted racial groups who have
relatively little social power in the United States (Blacks, Latino/as, Native Americans, and
Asians), by the members of the agent racial group who have relatively more social power
(Whites). This subordination is supported by the actions of individuals, cultural norms and
values, and the institutional structures and practices of society.

Discriminatory Behaviors: The differential allocation of goods, resources, and services,


and the limitation of access to full participation in society based on individual membership
in a particular social group.

Social Justice: Includes a vision of society in which the distribution of resources is


equitable and all members are empowered and are physically and psychologically safe and
secure.

Power: Access to resources that enhance one's chances of getting what one needs in order
to lead a safe, comfortable, and productive life.

Prejudice: A set of negative personal beliefs about a social group that leads individuals to
prejudge people from that group or the group in general, regardless of individual
differences among members of that group.

Privilege: Access to something of value solely because of ones agent social identity
membership.

Individual Oppression: Actions that reflect prejudice against a social group.


Societal Oppression: Policies and individual behaviors within institutions that have a
differential and/or harmful impact on target groups, whether or not they are so intended.

Cultural Oppression: A set of social norms, economic and political arrangements, roles,
rituals, and arts that reflect and reinforce the power of a single agent group or belief
system.

Collusion: Thinking, feeling, and acting in ways that accept the dominant groups ideology
about ones own target group and accepting a definition of oneself that is hurtful and
limiting. This involves demonstrating the devaluation of ones own group and of oneself as
a member of that group. Note: In contrast to an internalized ism, collusion involves the
target group member actively supporting oppression, as opposed to a passive
internalization.
PROGRAM EVALUATION NORTHWESTERN 34

Appendix D: Intergroup Relations (IGR) Language Activity

Purpose: To introduce key IGR concepts to a group of participants. This activity also serves
to gauge the participants levels of understanding regarding these concepts.

Concepts Used: Agent, Target, Privilege, Power, Oppression, Discrimination, and Collusion

Set Up: Write each individual concept at the top of large, separate sheets of paper. Place the
sheets of paper together dyadically as follows: AgentTarget, PrivilegePower,
OppressionDiscrimination. Collusion is not paired with any other concept.

Activity:

Part One: Each participant is to place a check mark on the individual sheets as a
reflection of how well they understand each concept. The sheet represents a
continuum of understanding, with one side representing fully knowledgeable about
said concept and the other side representing not-so-knowledgeable about said
concept. The use of a check mark means that persons do not have to publicly
express their knowledge or lack thereof. Participants are free to discuss where they
placed their checks if so desired.

Part Two: After check marks have been made on all sheets, the facilitator(s)
explains that the purpose is build clarity on what these concepts mean. They note
that there are not clear definitions, but that the conversation should bring to light
some key aspects of each concept.

The facilitator then begins a discussion around each concept, starting with either
agent or target, and then adding the other, seeing what interpretations are held by
participants. Follow-up questions probe for examples and distinctions between
other words. Clarification is also sought if participant uses another concept word
within their interpretation, to see if they can explain it in more simple words.
Steadily, the facilitator adds each new word. After each word, the facilitator may
choose to point out the explanation on the key concept list, highlighting again that
these are not meant to be definitions.

Challenges:
Facilitator needs to have a good sense of what each concept is.
It is challenging to affirm participants when they offer interpretations that are very
different from what IGR views to be the more fitting conceptualization, while
pushing them to understand the IGR perspective.
It is helpful for the facilitator to have examples that help illustrate these concepts
readily available, in case the group doesnt generate enough.
This can be a lengthy exercise, and therefore challenging to keep peoples focus for
the entirety. The duration completely depends on the number of group and prior
understanding of each group member.
PROGRAM EVALUATION NORTHWESTERN 35

Reminding participants that there is no clean definition is challenging because their


instinct is to search for an answer. These conceptualizations are far more complex
than that.
It is challenging to pull out the key points from what participants are saying,
because a lot of times (unless they have practiced articulating these
conceptualizations) they dont have concise thoughts.
PROGRAM EVALUATION NORTHWESTERN 36

Appendix E: Agenda for Intersectional Language Activity

Agenda Facilitator to lead


Welcome- Introductions of facilitators

Icebreaker- (check and see which ones were done before)


Rock, paper, scissors challenge
Look up, Look down
Shake it off
Review Guidelines
Define Intersectionality
Review Language
Power
Privilege
Oppression
Target and Agent
Intersectionality Activity
Divide into small groups of 4 (10 groups) ~ 3:50-3:55pm
Choose an identity you are most comfortable with (aka
salient) and share with group. Do you hold privilege in this
identity? Are you oppressed? How? ~3:55-4:03 (6 min per
round, 1.5 min per person)
3 Other identities (6 min round

Share out- 10min- 4:20-4:30pm (10)


How was this for you?
What surprised you?
Which identity was the easiest for you to interact with?
Which identity was the most difficult for you to interact
with?
What commonalities did you find?
Back into small groups
Solidarity- 10min 4:30-4:40 (10)
How does this impact how you might interact with one
another? How do you relate with others?
How does this help you to relate with others who have
different identities than you hold? (We have different
identities but we all are oppressed and hold privilege----
how does this allow us to move past individual identities?)
How can you show solidarity/create coalitions with these
things in mind?
4:40-4:55pm (15 min)
Come back to large group debrief
Closure- One word Whip
PROGRAM EVALUATION NORTHWESTERN 37

Appendix F (Language Activity Survey)

Peer Inclusion Educators Post Workshop Survey


Workshop: Language Activity
Name of Peer Facilitator(s):
Date:
Northwestern Student ID Number: 000
Please complete the following post-workshop survey. Please complete the following survey
about your experiences in todays PIE workshop. All of your answers are confidential but we
are asking that you provide your Northwestern Student ID Number so that we can follow up with
a 6-month post survey of the content that you have learned in our workshop. Your participation
is appreciated. If you have any questions please feel free to contact the assistant director of the
Social Justice Education Department Noor Ali, noor.ali@northwestern.edu. After you have
completed this survey the Peer Facilitators will collect them. Thank you!
Social Justice Key Terms Knowledge
The following questions are centered on knowledge of the social justice terms you have learned
and or been introduced to by engaging in this workshop. Please rate your overall agreement on
a scale from Strongly Agree to Strongly Disagree.

1. As a result of participating in this workshop, I can provide a deeper definition of


target/agent identities.

Strongly Agree Agree Neutral Disagree Strongly Disagree

2. As a result of participating in this workshop, I can provide a deeper definition of


privilege.

Strongly Agree Agree Neutral Disagree Strongly Disagree

3. As a result of participating in this workshop, I can provide a deeper definition of power.

Strongly Agree Agree Neutral Disagree Strongly Disagree

4. As a result of participating in this workshop, I can provide a deeper definition of


oppression.

Strongly Agree Agree Neutral Disagree Strongly Disagree

5. As a result of participating in this workshop, I can provide a deeper definition of


discrimination.

Strongly Agree Agree Neutral Disagree Strongly Disagree

6. As a result of participating in this workshop, I can provide a deeper definition of


collusion.
PROGRAM EVALUATION NORTHWESTERN 38

Strongly Agree Agree Neutral Disagree Strongly Disagree

Social Justice Key Terms Application


The following questions are centered on the application of the social justice terms you have
learned and or been introduced to by engaging in this workshop. Please rate your overall
agreement on a scale from Strongly Agree to Strongly Disagree.

1. I can identify target/agent identities in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

2. I can identify privilege in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

3. I can identify power in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

4. I can identify oppression in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

5. I can recognize when others are discriminated against in various settings at Northwestern
University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

6. I can recognize when I am discriminated against in various settings at Northwestern


University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

7. I can differentiate between individual and systematic discrimination in various settings at


Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

8. I can identify collusion in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree


PROGRAM EVALUATION NORTHWESTERN 39

Peer Inclusion Educators


The following questions are centered on the facilitation skills of the peer facilitators who
completed this workshop. Please rate your overall agreement on a scale from Strongly Agree to
Strongly Disagree.

1. The facilitators provided ground rules prior to the start of the workshop to engage in
dialogue around social justice key terms.

Strongly Agree Agree Neutral Disagree Strongly Disagree

2. The facilitators encouraged vulnerability on behalf of the participants throughout the


workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

3. The facilitators shared their own experiences with the language activity throughout the
workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

4. The facilitators promoted verbal participation throughout the workshop via large and
small-group discussions.

Strongly Agree Agree Neutral Disagree Strongly Disagree

5. The facilitators were on time and set up the space prior to the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

6. The facilitators were positive throughout the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

7. The facilitators were engaging throughout the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

8. The facilitators were open to questions throughout the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree


PROGRAM EVALUATION NORTHWESTERN 40

DEMOGRAPHICS:

Age: ____________ Citizenship/Nation of Origin: ________________________________________

Class Standing: First-Year Undergraduate Sophomore Junior Senior Graduate/Professional Student


Staff Faculty Member Not already listed: ________________________
Gender: Woman Man Transgender Not already listed _______________________________
Race (check all that apply): Native American/Alaskan Native Black or African American
Asian/Asian American Native Hawaiian/Pacific Islander
Latino(a) or Latinx White/Caucasian
Middle Eastern/North African
Not already listed: ______________________________

YOUR FEEDBACK IS VERY MUCH APPRECIATED. THANK YOU!

Please return this evaluation to the facilitators upon completion.


PROGRAM EVALUATION NORTHWESTERN 41

Appendix G (Social Identity Survey)

Peer Inclusion Educators Post Workshop Survey


Workshop: Social Identity
Name(s) of Peer Facilitators:
Date:
Northwestern Student ID Number: 000
Please complete the following post-workshop survey. Please complete the following survey
about your experiences in todays PIE workshop. All of your answers are confidential but we
are asking that you provide your Northwestern Student ID Number so that we can follow up with
a 6-month post survey of the content that you have learned in our workshop. Your participation
is appreciated. If you have any questions please feel free to contact the assistant director of the
Social Justice Education Department Noor Ali, noor.ali@northwestern.edu. After you have
completed this survey the Peer Facilitators will collect them. Thank you!
Social Identity Key Terms Knowledge
The following questions are centered on knowledge of the social justice terms you have learned
and or been introduced as it relates to your social identities by engaging in this workshop.
Please rate your overall agreement on a scale from Strongly Agree to Strongly Disagree.

1. As a result of participating in this workshop, I can provide a deeper definition of


target/agent identities.

Strongly Agree Agree Neutral Disagree Strongly Disagree

2. As a result of participating in this workshop, I can provide a deeper definition of privilege


as it relates to my social identity.

Strongly Agree Agree Neutral Disagree Strongly Disagree

3. As a result of participating in this workshop, I can provide a deeper definition of salient


identities.

Strongly Agree Agree Neutral Disagree Strongly Disagree

4. As a result of participating in this workshop, I can provide a deeper definition of my own


salient identities.

Strongly Agree Agree Neutral Disagree Strongly Disagree

5. As a result of participating in this workshop, I can provide a deeper definition of how my


salient identities are connected to the terms privilege and oppression.

Strongly Agree Agree Neutral Disagree Strongly Disagree


PROGRAM EVALUATION NORTHWESTERN 42

Social Identity Key Terms Application


The following questions are centered on the application of the social identity key terms you have
learned and or been introduced to by engaging in this workshop. Please rate your overall
agreement on a scale from Strongly Agree to Strongly Disagree.

1. I can identify target/agent identities in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

2. I can identify privilege in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

3. I can identify the term salient identity in relation to the campus community at
Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

4. I can identify my salient identities as being a part of a privileged or oppressed group at


Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

Peer Inclusion Educators


The following questions are centered on the facilitation skills of the peer facilitators who
completed this workshop. Please rate your overall agreement on a scale from Strongly Agree to
Strongly Disagree.

1. The facilitators provided ground rules prior to the start of the workshop to engage in
dialogue around social justice key terms.

Strongly Agree Agree Neutral Disagree Strongly Disagree

2. The facilitators encouraged vulnerability on behalf of the participants throughout the


workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

3. The facilitators shared their own experiences with the language activity throughout the
workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree


PROGRAM EVALUATION NORTHWESTERN 43

4. The facilitators promoted verbal participation throughout the workshop via large and
small-group discussions.

Strongly Agree Agree Neutral Disagree Strongly Disagree

5. The facilitators were on time and set up the space prior to the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

6. The facilitators were positive throughout the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

7. The facilitators were engaging throughout the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

8. The facilitators were open to questions throughout the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

DEMOGRAPHICS:

Age: ____________ Citizenship/Nation of Origin: ________________________________________

Class Standing: First-Year Undergraduate Sophomore Junior Senior Graduate/Professional Student


Staff Faculty Member Not already listed: ________________________
Gender: Woman Man Transgender Not already listed _______________________________
Race (check all that apply): Native American/Alaskan Native Black or African American
Asian/Asian American Native Hawaiian/Pacific Islander
Latino(a) or Latinx White/Caucasian
Middle Eastern/North African
Not already listed: ______________________________

YOUR
YOURFEEDBACK
FEEDBACKIS
ISVERY
VERYMUCH
MUCHAPPRECIATED.
APPRECIATED.THANK
THANKYOU!
YOU!

Please return this evaluation to the facilitators upon completion.


PROGRAM EVALUATION NORTHWESTERN 44
PROGRAM EVALUATION NORTHWESTERN 45

Appendix H (Intersectionality Survey)

Peer Inclusion Educators Post Workshop Survey


Workshop: Intersectionality Activity
Name(s) of Peer Facilitators:
Date:
Northwestern Student ID Number: 000
Please complete the following post-workshop survey. Please complete the following survey
about your experiences in todays PIE workshop. All of your answers are confidential but we
are asking that you provide your Northwestern Student ID Number so that we can follow up with
a 6 month post survey of the content that you have learned in our workshop. Your participation is
appreciated. If you have any questions please feel free to contact the assistant director of the
Social Justice Education Department Noor Ali, noor.ali@northwestern.edu. After you have
completed this survey the Peer Facilitators will collect them. Thank you!
Social Justice Key Terms Knowledge
The following questions are centered on knowledge of the social justice terms you have learned
and or been introduced to by engaging in this workshop. Please rate your overall agreement on
a scale from Strongly Agree to Strongly Disagree.

1. As a result of participating in this workshop, I can provide a deeper definition of


privilege.

Strongly Agree Agree Neutral Disagree Strongly Disagree

2. As a result of participating in this workshop, I can provide a deeper definition of power.

Strongly Agree Agree Neutral Disagree Strongly Disagree

3. As a result of participating in this workshop, I can provide a deeper definition of


oppression.

Strongly Agree Agree Neutral Disagree Strongly Disagree

4. As a result of participating in this workshop, I can provide a deeper definition of


intersectionality.

Strongly Agree Agree Neutral Disagree Strongly Disagree

5. As a result of participating in this workshop, I can provide a deeper definition of


solidarity.

Strongly Agree Agree Neutral Disagree Strongly Disagree


PROGRAM EVALUATION NORTHWESTERN 46

Social Justice Key Terms Application


The following questions are centered on the application of the social justice terms you have
learned and or been introduced to by engaging in this workshop. Please rate your overall
agreement on a scale from Strongly Agree to Strongly Disagree.

1. I can identify my own privilege(s) in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

2. I can identify my own power in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

3. I can identify oppression in regards to my social identities in various settings at


Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

4. I can identify intersectionality in regards to my own social identities in various settings at


Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

5. I can identify solidarity in various settings at Northwestern University.

Strongly Agree Agree Neutral Disagree Strongly Disagree

Peer Inclusion Educators


The following questions are centered on the facilitation skills of the peer facilitators who
completed this workshop. Please rate your overall agreement on a scale from Strongly Agree to
Strongly Disagree.

1. The facilitators provided ground rules prior to the start of the workshop to engage in
dialogue around social justice key terms.

Strongly Agree Agree Neutral Disagree Strongly Disagree

2. The facilitators encouraged vulnerability on behalf of the participants throughout the


workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

3. The facilitators shared their own experiences with the language activity throughout the
workshop.
PROGRAM EVALUATION NORTHWESTERN 47

Strongly Agree Agree Neutral Disagree Strongly Disagree

4. The facilitators promoted verbal participation throughout the workshop via large and
small-group discussions.

Strongly Agree Agree Neutral Disagree Strongly Disagree

5. The facilitators were on time and set up the space prior to the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

6. The facilitators were positive throughout the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

7. The facilitators were engaging throughout the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

8. The facilitators were open to questions throughout the workshop.

Strongly Agree Agree Neutral Disagree Strongly Disagree

DEMOGRAPHICS:

Age: ____________ Citizenship/Nation of Origin: ________________________________________

Class Standing: First-Year Undergraduate Sophomore Junior Senior Graduate/Professional Student


Staff Faculty Member Not already listed: ________________________
Gender: Woman Man Transgender Not already listed _______________________________
Race (check all that apply): Native American/Alaskan Native Black or African American
Asian/Asian American Native Hawaiian/Pacific Islander
Latino(a) or Latinx White/Caucasian
Middle Eastern/North African
Not already listed: ______________________________

YOUR FEEDBACK IS VERY MUCH APPRECIATED. THANK YOU!


YOUR FEEDBACK IS VERY MUCH APPRECIATED. THANK YOU!

Please return this evaluation to the facilitators upon completion.


PROGRAM EVALUATION NORTHWESTERN 48

Appendix I: Survey Item Matrix


PROGRAM EVALUATION NORTHWESTERN 49
PROGRAM EVALUATION NORTHWESTERN 50

Appendix J: Focus Group Protocol for the Language Activity

Post-PIE Workshop Attendees

Overarching Objective of the Language Workshop Focus Group:


What effect, if any, does the language workshop have on its participants immediately after the
workshop? What effect do the facilitators have on the participants immediately after the
workshop?

Learning Outcomes:

Participants will feel compelled to interact with these terms and concepts in other spaces
on campus
Participants will find groups or organizations on campus that engage with social justice
concepts
Participants will initiate dialogues and discussion of these terms and concepts on campus
Participants will partake in an ongoing self-reflection regarding social justice key
concepts with other students on campus

Focus Group Script


Introduction:
Thank you for joining us today to participate in this focus group. My name is (Graduate
Assistant, Assistant Director, or Director of SJE). Our goal through this focus group is to hear
about your experiences in PIEs language activity workshop and how those experiences have
related to your experience at Northwestern University over the last six months. By taking time
to speak with us about your experiences on campus and those you had during your language
workshop, we hope to add a greater depth to the feedback that was provided during the post-
workshop surveys. We greatly appreciate your participation thus far. This will be the last time
that we ask for your involvement in our evaluation.

The discussion should take no longer than 60 minutes. Please note that you are free to skip any
questions that are raised if at any point you feel uncomfortable. Please feel free to keep eating
snacks, refill on drinks, and use the restroom, etc., throughout our discussion.

The information gathered today will be shared with SJE staff and used to improve the quality of
PIE language workshop for future students. The feedback that you provide during this
discussion will be recorded by [state name(s) of notetaker(s)] and a tape recorder so we can be
sure to accurately capture everyones responses. This information will be confidential and will
not be specifically identified to you afterwards. Of course, there are no right or wrong answers,
and wed like to hear from everyone if possible.

At this time I ask that you review our interview consent form to make sure that you understand
the details of what will occur today. By signing the form you consent to participate in this
interview. Once we have everyones consent forms, we will begin the discussion.
PROGRAM EVALUATION NORTHWESTERN 51

Introductory Question:
To begin, wed like to hear about your reasons for completing this particular PIE
workshop
Probe: What were some of the reasons that led you to sign up for a language
workshop through PIE?
Probe: Was it a mandatory part of your training for a student group/organization?
Probe: Did you sign up yourself through a student group or organization?

Goal Area 1: Participants are able to describe a deeper meaning of the following social justice
terms: target/agent identities, privilege, power, oppression, discrimination, and collusion.
Question 1.1: Since completing the workshop, in what ways have you been able to
reflect on how the terms target/agent identities, privilege, power, oppression,
discrimination, and/or collusion are present on campus?
o Probe: When have you found it easier to do this?
o Probe: Are there particular situations when this is more difficult? For what
reasons?
o Probe: Is there a strategy or technique you learned during the language
workshop that has proven useful in addressing these situations?
Question 1.2: Can you identify examples of how these terms have appeared in various
settings through your own personal experience?

Question 1.3: How did you address the situation/context that you identified when one
or more of the terms were present on campus?
o Probe: What prompted you to address the situation as you did?
o Probe: How did you feel after you did this?

Goal Area 2: Participants can recognize systems of privilege at Northwestern University and
can identify power relations in various settings on campus.
Question 2.1: Can you describe whether youve been able to recognize systematic
privilege on Northwesterns campus since your participation in the workshop?
o Probe: What do you think contributed toward your ability to recognize
systematic privilege at NU?
o Probe: How did you feel when you recognized this type of privilege at NU?
o Probe: Did you process or reflect on that experience with anyone? If so, how
was that?
o Probe: Were any of the factors associated with one of your privileged identity
groups?
o Probe: If you could change anything about the how you addressed the
situation, what would it be and why?

Goal Area 3: Participants can recognize systems of oppression at Northwestern University and
can identify when themselves or students different from them have been discriminated against
based on the system in place.
Question 3.1: Can you describe whether youve been able to recognize systematic
oppression on Northwesterns campus since your participation in the workshop?
PROGRAM EVALUATION NORTHWESTERN 52

o Probe: How did you identify the situation/context with a system of


oppression?
o Probe: Since the PIE language workshop, have you proactively fostered a
space, or spaces, for others to be vulnerable and share their personal stories?
o Probe: Since the PIE language workshop, have you proactively fostered a
group or student organization that engages with one or more social justice
term?

Goal Area 4: Participants are able to provide critical feedback on the facilitators abilities to
provide a brave space for students to be vulnerable and share their personal stories, timeliness,
and effort into providing a learning environment.
Questions 4.1: The idea of a brave space is something that comes up in our area a lot.
Based of your experience on campus, how would you define brave space?
o Probe: Based on our definition of brave space, did the facilitator(s) provide
this during your workshop?
o Probe: Did the facilitator or facilitators provide a space where you were able
to have a voice in the conversation?
o Probe: What did the facilitator(s) do to create this space?
o Probe: Was there anything different the facilitator(s) could have done to create
a brave space for you?
Question 4.2: Were there any positive actions or statements made by the facilitator, or
facilitators, that still resonate with you?
Question 4.3: Were there any negative actions or statements made by the facilitator,
or facilitators, that you still remember from the workshop?

Closing Segment: Improvement and Recommendations

Before we end, we would like to take this opportunity to ask you if there are any improvements
or changes that you would make to the language workshop.

Is there anything else you would like to add before we end our discussion? If not, then this
concludes our time together. We appreciate you sharing your feedback with us today. The
success of the PIE program is dependent upon the valuable feedback that you all have shared
during todays discussion. Thank you for taking the time today to participate in this discussion!
PROGRAM EVALUATION NORTHWESTERN 53

Appendix K: Focus Group Protocol for Social Identity

Post-PIE Workshop Attendees

Overarching Objective of the Social Identity Workshop Focus Group:


What effect, if any, does the social identity workshop have on its participants immediately after
the workshop? What effect do the facilitators have on the participants immediately after the
workshop?

Learning Outcomes:
Participants will apply key social justice terms and concepts to their own social identities
Participants will be self-aware of their salient identities in regards to power, privilege,
and oppression on campus
Participants will be able to clearly articulate their social identities in the context of power,
privilege, and oppression
Participants will be able to engage in difficult conversations related to social identities on
campus

Focus Group Script


Introduction:
Thank you for joining us today to participate in this interview. My name is (Graduate Assistant,
Assistant Director, or Director of SJE). Our goal through this focus group is to hear about your
experiences in PIEs social identity workshop and how those experiences have related to your
experience at Northwestern University over the last six months. By taking time to speak with us
about your experiences on campus and those you had during your language workshop, we hope
to add a greater depth to the feedback that was provided during the post-workshop surveys. We
greatly appreciate your participation thus far. This will be the last time that we ask for your
involvement in our evaluation.

The discussion should take no longer than 60 minutes. Please note that you are free to skip any
questions that are raised if at any point you feel uncomfortable. Please feel free to keep eating
snacks, refill on drinks, and use the restroom, etc., throughout our discussion.

The information gathered today will be shared with SJE staff and be used to improve the quality
of PIE language workshop for future students. The feedback that you provide during this
discussion will be recorded by [state name(s) of notetaker(s)] and a tape recorder so we can be
sure to accurately capture everyones responses. This information will be private and will not be
specifically identified to you afterwards. Of course, there are no right or wrong answers, and
wed like to hear from everyone if possible.

At this time I ask that you review our interview consent form to make sure that you understand
the details of what will occur today. By signing the form you consent to participate in this
interview. Once we have everyones consent forms, we will begin the discussion.
PROGRAM EVALUATION NORTHWESTERN 54

Introductory Question:

To begin, wed like to hear about your reasons for completing this particular PIE
workshop
Probe: What were some of the reasons that led you to sign up for a social identity
workshop through PIE?
Probe: Was it a mandatory part of your training for a student group/organization?
Probe: Did you sign up yourself through a student group or organization?
Probe: What were some meaningful takeaways from the workshop?

Goal Area 1: Participants are able to define what identities are oppressed and what identities
gives them power and privilege in our US society.
Question 1.1: Have you been able to reflect on your own oppressed and privileged
identities?
Question 1.2: Have you been able to reflect on how your oppressed and privileged
identities are present on campus?
o Probe: Have you felt ostracized due to your oppressed and/or privileged identities
through various settings on campus?

Goal Area 2: Participants are able to recognize and state their salient identities, meaning the
identities that they identify with the most within US context such as their race, ethnicity, religion,
sexual orientation, gender, etc.
Question 2.1: Can you elaborate on how you describe your salient identities to others
in relation to power, privilege, and oppression?
o Probe: Did participation in the social identity workshop influence or change how
you describe your salient identities to others?
o Probe: What are some of the challenges that have come with the self-reflection of
your salient identities?

Goal Area 3: Participants are able to provide critical feedback on the facilitators abilities to
provide a brave space for students to be vulnerable and share their personal stories, timeliness,
and effort into providing a learning environment.
Questions 3.1: The idea of a brave space is something that comes up in our area a lot.
Based of your experience on campus, how would you define brave space?
o Probe: Based on our definition of brave space, did the facilitator(s) provide
this during your workshop?
o Probe: Did the facilitator, or facilitators, provide a space where you were able
to have a voice in the conversation?
o Probe: What did the facilitator(s) do to create this space?
o Probe: Was there anything different the facilitator(s) could have done to create
a brave space for you?
Question 3.2: Were there any positive actions or statements made by the facilitator, or
facilitators, that still resonate with you?
Question 3.3: Were there any negative actions or statements made by the facilitator,
or facilitators, that you still remember from the workshop?
PROGRAM EVALUATION NORTHWESTERN 55

Closing Segment: Improvement and Recommendations

Before we end, we would like to take this opportunity to ask you if there are any improvements
or changes that you would make to the social identity workshop.

Is there anything else you would like to add before we end our discussion? If not, then this
concludes our time together. We appreciate you sharing your feedback with us today. The
success of the PIE program is dependent upon the valuable feedback that you all have shared
during todays discussion. Thank you for taking the time today to participate in this discussion!
PROGRAM EVALUATION NORTHWESTERN 56

Appendix L: Focus Group Protocol for Intersectionality

Post-PIE Workshop Attendees

Overarching Objectives of the Intersectionality Workshop Focus Group:


What effect, if any, does the intersectionality workshop have on its participants immediately
after the workshop? What effect do the facilitators have on the participants immediately after the
workshop?

Learning Outcomes:
Participants will apply the definition of target and agent identities to the greater campus
community
Participants will engage in self-reflection by being exposed to notions of power,
privilege, and oppression on campus
Participants will use self-reflection of their target and agent identities as tools to combat
power, privilege, and oppression on campus
Participants will seek further dialogues and discussion on topics and concepts related to
social justice on campus, particularly through SJE

Interview Protocol

Introduction:
Thank you for joining us today to participate in this interview. My name is (Graduate Assistant,
Assistant Director, or Director of SJE). Our goal through this focus group is to hear about your
experiences in PIEs intersectionality workshop and how those experiences have related to your
experience at Northwestern University over the last six months. By taking time to speak with us
about your experiences on campus and those you had during your language workshop, we hope
to add a greater depth to the feedback that was provided during the post-workshop surveys. We
greatly appreciate your participation thus far. This will be the last time that we ask for your
involvement in our evaluation.

The discussion should take no longer than 60 minutes. Please note that you are free to skip any
questions that are raised if at any point you feel uncomfortable. Please feel free to keep eating
snacks, refill on drinks, and use the restroom, etc., throughout our discussion.

The information gathered today will be shared with SJE staff and be used to improve the quality
of PIE language workshop for future students. The feedback that you provide during this
discussion will be recorded by [state name(s) of notetaker(s)] and a tape recorder so we can be
sure to accurately capture everyones responses. This information will be private and will not be
specifically identified to you afterwards. Of course, there are no right or wrong answers, and
wed like to hear from everyone if possible.

At this time I ask that you review our interview consent form to make sure that you understand
the details of what will occur today. By signing the form you consent to participate in this
interview. Once we have everyones consent forms, we will begin the discussion.
PROGRAM EVALUATION NORTHWESTERN 57

Introductory Question:

To begin, wed like to hear about your reasons for completing this particular PIE
workshop
Probe: What were some of the reasons that led you to sign up for an
intersectionality workshop through PIE?
Probe: Was it a mandatory part of your training for a student group/organization?
Probe: Did you sign up yourself through a student group or organization?
Probe: What were some meaningful takeaways from the workshop?

Goal Area 1: Participants are able to define the intersectionality of their target/ agent identities,
meaning defining what identities are oppressed and what identities gives them power and
privilege in our US society.
Question 1.1: Since completing the PIE workshop six months ago, have you been
able to reflect on your own target/agent identities in relation to the definition of
intersectionality?
o Probe: Did participation in the intersectionalty workshop influence or change
how you describe your target/agent identities to others?
o Probe: What are some of the challenges that have come with the self-
reflection of your target/agent identities in relation to intersectionality?

Goal Area 2: Participants are able to recognize that their target identities do not outweigh the
power of their privilege identities, but that all of their identities intersect together.
Question 2.1: Over the last six months, have you used your privileged identities as a
tool to combat systems of oppression on campus?
o Probe: Are there times when this has been easier to do?
o Probe: What have been some of the challenges in utilizing your privileged
identities to raise awareness about the marginalization of target identity
groups?

Goal Area 3: Participants are able to provide detailed descriptions of what solidarity would look
like amongst their peers at Northwestern University.
Question 3.1: How have you demonstrated allyship with other marginalized identity
groups on campus?
o Probe: Were you able to better define allyship as a result of the intersectionality
workshop?
o Probe: What spaces on campus, if any, have allowed you to practice allyship?

Question 3.2: What are some techniques or strategies related to solidarity that you
have implemented since your participation in the intersectionality workshop?
o Probe: Did you learn this from the intersectionality workshop? Or from a
different experience?

Goal Area 4: Participants are able to provide critical feedback on the facilitators abilities to
provide a brave space for students to be vulnerable and share their personal stories, timeliness,
and effort into providing a learning environment.
PROGRAM EVALUATION NORTHWESTERN 58

Questions 4.1: The idea of a brave space is something that comes up in our area a lot.
Based of your experience on campus, how would you define brave space?
o Probe: Based on our definition of brave space, did the facilitator(s) provide
this during your workshop?
o Probe: Did the facilitator, or facilitators, provide a space where you were able
to have a voice in the conversation?
o Probe: What did the facilitator(s) do to create this space?
o Probe: Was there anything different the facilitator(s) could have done to create
a brave space for you?
Question 4.2: Were there any positive actions or statements made by the facilitator, or
facilitators, that still resonate with you?
Question 4.3: Were there any negative actions or statements made by the facilitator,
or facilitators, that you still remember from the workshop?

Closing Segment: Improvement and Recommendations

Before we end, we would like to take this opportunity to ask you if there are any improvements
or changes that you would make to the intersectionality workshop.

Is there anything else you would like to add before we end our discussion? If not, then this
concludes our time together. We appreciate you sharing your feedback with us today. The
success of the PIE program is dependent upon the valuable feedback that you all have shared
during todays discussion. Thank you for taking the time today to participate in this discussion!
PROGRAM EVALUATION NORTHWESTERN 59

Appendix M: Consent Form

CONSENT TO PARTICIPATE IN PEER INCLUSION EDUCATORS (PIE) IMPACT


FOCUS GROUP

Project Title: Peer Inclusion Educators (PIE) Impact Focus Group

Evaluators: Anthony Sis and Lorrena Johnson

Introduction:
You are being asked to participate in a focus group regarding your experience outside of the Peer
Inclusion Educators (PIE) workshop. Because you completed a post-workshop survey and
expressed interest in following up regarding your experience on campus, we are interested in
hearing more about your thoughts and experience.

Please read this form carefully and ask any questions you may have as you decide to participate
in this interview.

Purpose:
The purpose of this interview is to gather information about students post-PIE workshop
experience at Northwestern University to better understand the effectiveness of the program.
The information generated from this interview will be used to improve the content of the three
PIE workshops offered (Social Justice Key Concepts, Social Identity, Intersectionality).

Risks/Benefits:
There are no foreseeable risks involved in participating in this research beyond those
experienced in everyday life. A key benefits will be the potential improvement of the PIE
program for future participants.

Confidentiality:
There will be no identifying information attached to your responses in the interview. A report
will be compiled of basic themes that emerge from the interviews. The information will be
shared with the Social Justice Education (SJE) staff and staff from Campus Inclusion and
Community (CIC).

Voluntary Participation:
Participation in this interview is voluntary. If you do not want to take part, simply decline
participation. If at any time you feel uncomfortable answering a question, you may remove
yourself and/or decide not to answer the question.

Contacts/Questions:
If you have any questions about the interview, please contact Noor Ali at 847-491-8063 or
Noor.Ali@Northwestern.edumailto:Noor.Ali@Northwestern.edu
mailto:Noor.Ali@Northwestern.edu
PROGRAM EVALUATION NORTHWESTERN 60

Statement of Consent:
Your signature below indicates that you have read and understood the information provided in
this document, have had an opportunity to ask questions, and agree to participate in this
interview.

You will be given a copy of this form for your records.

_______________________________________________ ___________________
Participants Signature Date

_______________________________________________ ___________________
Evaluators Signature Date
PROGRAM EVALUATION NORTHWESTERN 61

Appendix N: Protocol Invitations

Invitation request if at least ten people have not expressed interest in focus group

Re: Invitation to Join a PIE Impact Research Discussion

Thank you for attending a PIE workshop this academic year. Were interested in learning about
the outcomes of the PIE workshop you attended on (Language Activity/ Social Identity/
Intersectionality). Because of your participation, we are looking to hear back from you about
your experience outside of PIE workshop on campus. We hope that you will want to provide
your thoughts.

The focus groups will be conducted over the next few weeks by a graduate assistant, assistant
director, or director of Social Justice Education (SJE) with additional support from Center for
Inclusion and Community (CIC) staff. The format of the focus group is to include no more than
ten previous student participants and should take no longer than ninety minutes to complete.
They will be scheduled from 12:00 1:00 P.M. or from 4:00- 5:00 P.M., depending on the
availability of the participants. Light snacks and beverages will be provided.

Please let me know if you are interested in joining this interview by replying to this invitation. I
will work with those who have agreed to participate to schedule a time to meet for the focus
group over the next few weeks.

Thank you in advance for agreeing to participate in our interview. If you have any questions,
please call or email me.

Noor A. Ali, M.S.W. (she, her, hers)


Assistant Director, Social Justice Education
Campus Inclusion and Community
Noor.Ali@Northwestern.edu
847-491-8063

Scott Hall 046 (Lower Level)


601 University Place
Evanston, IL, 60208
PROGRAM EVALUATION NORTHWESTERN 62

Appendix O: Protocol Invitations (cont.)

Confirmation Email

Subj: Confirmation: PIE Impact Research Discussion on [insert date/time]

Hello [insert name of previous participant],

This email confirms that you have agreed to attend the Peer Inclusion Educators (PIE) impact
research focus group on [date/time/location]. Light snacks and beverages will be provided.
Please reply to this email if you are no longer able to attend. We look forward to your
participation!

Noor A. Ali, M.S.W. (she, her, hers)


Assistant Director, Social Justice Education
Campus Inclusion and Community
Noor.Ali@Northwestern.edu
847-491-8063

Scott Hall 046 (Lower Level)


601 University Place
Evanston, IL, 60208

Reminder Email

Subj: Reminder: PIE Impact Research Discussion on [insert date/time]

Hello [insert name of previous participant],

This email is a reminder that you have agreed to attend the Peer Inclusion Educators (PIE)
impact research focus group on [date/time/location]. Light snacks and beverages will be
provided. Please reply to this email if you are no longer able to attend. We look forward to your
participation!

Noor A. Ali, M.S.W. (she, her, hers)


Assistant Director, Social Justice Education
Campus Inclusion and Community
Noor.Ali@Northwestern.edu
847-491-8063

Scott Hall 046 (Lower Level)


601 University Place
Evanston, IL, 60208