You are on page 1of 61
Running head: PROGRAM EVALUATION NORTHWESTERN 1
Running head: PROGRAM EVALUATION NORTHWESTERN
1

Program Evaluation for the Peer Inclusion Educators Program at Northwestern University Anthony Sis and Lorrena Johnson Loyola University Chicago

Running head: PROGRAM EVALUATION NORTHWESTERN 1 Program Evaluation for the Peer Inclusion Educators Program at Northwestern

PROGRAM EVALUATION NORTHWESTERN

2

Table of Contents

INTRODUCTION

3

PROGRAM

HISTORY

3

PROGRAM

OVERVIEW

4

PROGRAM

CONTEXT

5

PROGRAM DESCRIPTION

6

SELECTION OF PEER FACILITATORS

7

PROGRAM PURPOSE

8

PROGRAM STAKEHOLDERS

9

LOGIC MODEL OVERVIEW

10

PREVIOUS EVALUATION ...................................................................................................................................

11

GENERAL EVALUATION APPROACH

12

QUANTITATIVE APPROACH

13

PARTICIPANTS

13

SURVEY DESIGN

14

INSTRUMENT DESCRIPTION

15

PILOT TESTING

17

IMPLEMENTATION/ADMINISTRATION

17

STATISTICAL ANALYSIS/ RESULTS PRESENTATION

18

 

QUALITATIVE APPROACH

19

QUALITATIVE RATIONALE

19

ANALYTICAL APPROACH

20

FOCUS GROUP PLAN

21

FOCUS GROUP IMPLEMENTATION

22

FOCUS GROUP ANALYSIS

23

LIMITATIONS ..............................................................................................................................

24

RESULT PRESENTATION

 

25

NEXT STEPS

25

REFERENCES

26

APPENDICES

27

APPENDIX A: PIE PROGRAM LOGIC MODEL

27

APPENDIX B: PREVIOUS EVALUATION

30

APPENDIX C: ABBREVIATED SOCIAL JUSTICE KEY CONCEPTS

33

APPENDIX D: INTERGROUP RELATIONS (IGR) LANGUAGE ACTIVITY

34

APPENDIX E: AGENDA FOR INTERSECTIONAL LANGUAGE ACTIVITY

36

APPENDIX F (LANGUAGE ACTIVITY SURVEY)

37

APPENDIX G (SOCIAL IDENTITY SURVEY)

41

APPENDIX

H

44

APPENDIX I: SURVEY ITEM MATRIX

47

APPENDIX J: FOCUS GROUP PROTOCOL FOR THE LANGUAGE ACTIVITY

49

APPENDIX K: FOCUS GROUP PROTOCOL FOR SOCIAL IDENTITY

52

APPENDIX L: FOCUS GROUP PROTOCOL FOR INTERSECTIONALITY

55

APPENDIX

M: CONSENT FORM ...................................................................................................

58

APPENDIX N: PROTOCOL INVITATIONS

60

APPENDIX O: PROTOCOL INVITATIONS (CONT.)

61

PROGRAM EVALUATION NORTHWESTERN

3

Introduction

Northwestern University is an academic driven institution. The mission statement for the

university states that Northwestern University is “committed to excellent teaching, innovative

research, and the personal and intellectual growth of its students in a diverse academic

community” (About: Northwestern university, 2016). Within the university, there is the office of

Campus Inclusion and Community (CIC). In this office, there are three major departments that

promote the mission of CIC, which is to “to work collaboratively with the university community

to cultivate inclusive learning environments through the intentional engagement with difference”

(Campus Inclusion & Community, 2016, p. 4). These three departments are Multicultural

Student Affairs (MSA), Student Enrichment Services (SES), and Social Justice Education

(SJE). An evaluation and assessment will be created for one particular program within CIC, Peer

Inclusion Educators (PIE), which is housed under the Department of Social Justice Education.

Program History

Campus Inclusion and Community (CIC) began as an office in the Division of Student

Affairs five years ago. In 2012, Executive Director Lesley Ann Brown-Henderson was brought

in by the Vice President of Student Affairs to create this office. This office was created due to

student activists that demanded and protested for a university led effort to create a more

equitable and just campus culture (Campus Diversity & Inclusion, 2016, p. 15). CIC was

initially a one-woman office led by Lesley Ann Brown-Henderson. She began recruiting

students who would be brave enough to engage in dialogue surrounding topics of social justice.

This program became known as Sustained Dialogue. In 2014, Campus Inclusion and

Community was restructured due to the sizeable growth of the Sustained Dialogue program,

which is how the Social Justice Education Department was established. Noor Ali was selected to

be the Assistant Director of the Social Justice Education Department and the Peer Inclusion

PROGRAM EVALUATION NORTHWESTERN

4

Educators Program was created to help alleviate the administrative tasks within CIC and the

Sustained Dialogue program.

Peer Inclusion Educators has only been established for two years. During this time, one

graduate assistant was hired to oversee another program within SJE, but they additionally

volunteered to also be a Peer Inclusion Educator. After each workshop the participants are given

a survey that asks the students if they felt the workshop was beneficial, if the material they were

learning was new, and they are also asked to evaluate the facilitators. To date, no formal

assessment has been completed based off all of the surveys that were collected thus far.

Program Overview

There are several conceptual frameworks that guide the work of CIC, SJE, and PIE.

CIC’s strategic theme is to advance social justice. CIC has claimed to, work authentically and

ethically to create an inclusive and socially just learning environment for all members of the

Northwestern University community. The model of CIC is adapted from the tri-sector model of

cultural practice by Jenkins & Walton (2008). The tri-sector model is commonly used in student

affairs it provides a framework based on three components education, enrichment, and

engagement. CIC’s structured its office by using the tri-sector model effectively. The Social

Justice Education department was created for the purposes of education described in the tri-

sector model above. The Social Justice Education department creates co-curricular educational

opportunities in partnership with the student community to foster conversations across difference

and support actions that create social change on campus (p. 9). The Student Enrichment Services

department was created for the purpose of enrichment described in the tri-sector model. The

Student Enrichment Services department works with low-income first generation students to

enhance their academic success, personal development, and professional growth (p. 9). The

Multicultural Student Affairs department was created for the purpose of engagement described in

PROGRAM EVALUATION NORTHWESTERN

5

the tri-sector model. Multicultural Student Affairs enriches the cultural experience of

Northwestern through leadership and education programming providing opportunities for

community engagement (p. 8). This formative assessment will focus specifically on SJE and an

evaluation and assessment will focus primarily on just one of the programs that SJE offers which

is PIE. SJE and its other programs are highlighted in the next paragraph.

SJE operates under a director, two assistant directors, and one graduate assistant. The

specific programming that SJE is responsible for is Sustained Dialogue, Analyzing Whiteness,

Step Up Bystander Intervention, and Peer Inclusion Educators. SJE’s programming is mostly

funded through private grants and donors. The Assistant Director is hoping to hire a graduate

assistant whose primary focus will be to work on the enhancement of the PIE program. The

funds will be used to put a three-year plan in motion. In previous years, the department did not

receive any funding for Peer Inclusion Educators and all of the peer facilitators were working on

a voluntarily basis. The peer facilitators still work on a voluntary basis to date, but this is

expected to change over the next three years. The effectiveness of voluntary peer educators will

be included in our formal assessment and evaluation of the PIE program.

Program Context

Peer Inclusion Educators (PIE) is, a program started by students, for students. PIE

dedicates its workshops to focus on social justice, social identities, and dialogue surrounding the

Northwestern community (Social justice education: Peer inclusion educators, 2016). Training is

held for all students who would like to become a peer facilitator. During this training, the

Assistant Director trains the peer facilitators to understand their social identities, meaning

gender, race, ethnicity, sex, religion, socioeconomic status, etc; students are able to explore their

identities and begin to understand what identities give them power and privilege in the United

States and what identities are oppressed. The workshops also put an emphasis on the

PROGRAM EVALUATION NORTHWESTERN

6

intersectionality of each participant privileged and oppressed identities. This is completed by

having the facilitators’ complete assignments such as the social identity grid, followed by

discussions about social justice terms such as target, agent, inclusion, collusion, privilege,

prejudice, discrimination, and oppression. After the facilitators spend time reflecting on their

own identities they then complete mock facilitations to prepare them to engage in dialogue with

student groups on Northwestern’s campus.

Once facilitators have completed their training student groups will have access to

complete requests for one time workshops by visiting the SJE website, which has a separate link

for programming requests under the CIC’s website. Students can then design their workshop by

requesting activities such as the language activity, social identity grid, or both. The language

activity looks at terms such as Target, Agent, Inclusion, Collusion, Privilege, Prejudice,

Discrimination, Oppression, and Social Justice. The group is then asked to come up with a

definition of each term collectively to begin the process of dialogue. The Social Identity Grid is

an activity that lists social identities such as ethnicity, ability, gender, sex, and race. The idea

with these activities is to have students think about their most salient identities and the identities

that they think about the least. For most people, the identities that you think about least are often

the identities that give you power and privilege. When students can identify parts of their

identities that gives them power and privilege, a rich dialogue is able to happen that plays a

crucial role to the relevance and importance of a program such as PIE to exist on a campus like

Northwestern.

Program Description

The Peer Inclusion Educators (PIE) program has three different workshop topics for

which student groups such as residence assistants, peer mentors, sororities, and fraternities can

request. These workshop topics include: general inclusion facilitation, social justice 101, and

PROGRAM EVALUATION NORTHWESTERN

7

inclusive language. Student groups also have the privilege of designing their workshops to fit

the needs of their group. For example, the general inclusion facilitation workshop outlines the

basics of inclusion and allows for the opportunity for your group to brainstorm ways to be even

more inclusive (Social justice education: Peer inclusion educators, 2016). The social justice 101

workshop focuses on tailoring conversations about social justice by having the group engage in

an intimate dialogue. An intimate dialogue is able to happen because workshops will only take

place with a group no larger than 20 students. Peer Inclusion Educators need to be able to

engage in conversations with group. Often times when a group is too large deep conversations

are unable to happen. The assistant director found that it is easier for facilitators to control the

space in an intimate setting (Social justice education: Peer inclusion educators, 2016). The final

workshop option is inclusive language. This facilitation workshop focuses on implementing

activities such as the language activity by putting those terms within the context of the culture

specifically at Northwestern University. Student groups have the opportunity to create a

workshop engaging all three of these topics, or only one topic, depending on the needs of the

person or group requesting a PIE workshop. All participants are expected to engage in dialogue

and come up with ways in which the language that they use on a daily basis can be more

inclusive by using a critical lens to analyze the meaning of social justice terms. A critical lens is

a tool to analyze works of literature. During the language activity facilitators are asking the

group to define social justice terms and then put them into the context of their lives as a

Northwestern student which is what is meant by using a critical lens when engaging in the

language activity.

Selection of Peer Facilitators

Peer Facilitators are chosen through a selective application process. The Assistant

Director of SJE, Noor Ali, has created a live application on the website

PROGRAM EVALUATION NORTHWESTERN

8

http://www.northwestern.edu/socialjustice. The questions on the website include why an

applicant would like to become a peer inclusion educator, their facilitation experience,

challenges they anticipate in this position, and how the applicant brings fun into the program.

Once the facilitators have been selected, they are required to attend two-day training from 9 AM-

5 PM on Northwestern’s campus.

The trainings typically launch in November for new recruits. During the beginning of the

quarter (September through October), previously trained peer facilitators are encouraged to sign

up for the workshop requests that have been submitted to SJE. Noor Ali sends an email to all

facilitators with a description of the group that has completed a request along with dates and

times of the request. Peer facilitators then sign up for workshops that will work around their

schedule. The peer facilitators also have monthly meetings. During the monthly meetings, the

facilitators discuss what challenges they have faced and the group provides suggestions on how

they can improve.

Program Purpose PIE has four learning outcomes (Social justice education: Peer inclusion educators

(2016), which include:

Educate campus peers on how to best allow for the growth of an inclusive community.

Allow for reflection on one’s self, backgrounds, and biases.

Identify issues on campus and address them through active and passive programming.

Encourage dialogue around issues of difference and to develop competent mechanisms

for addressing these issues.

This program has been identified as an introductory sister program to the Sustained Dialogues

program, which encourages in-depth conversations once a week for 90 minutes throughout the

quarter.

PROGRAM EVALUATION NORTHWESTERN

9

Program Stakeholders

There are direct and indirect primary stakeholders in the PIE program which includes the

university at large, students, the vice president of student affairs, the director and assistant

director of SJE and community partners. The students of Northwestern University are

stakeholders because the office of SJE was created by students for students. Students actively

seek out workshops. Student facilitators are trained so they are learning life skills that they can

take with them and sharing knowledge with their peers. Assessment on the effectiveness of PIE

is determined by the participants (students, faculty, and staff) of the workshops as they are given

an evaluation after each workshop. The evaluation measures the effectiveness of the content and

measures the effectiveness of the facilitators. Faculty and staff seek out workshops as well

which would make the university at large indirect primary stakeholders. After each workshop all

participants complete surveys and their feedback is critical to how PIE will continue to operate.

PIE also trains a number of students to become active facilitators of PIE. The students’

engagement with this program is what keeps it running. The vice president of student affairs is a

primary direct stakeholder as they are the chief personnel officer of the university and they

directly advise the president of all non-academic activities on campus including PIE (About:

Northwestern university, 2016). The director of SJE is a primary direct stakeholder. The

director is responsible for communicating with alumni and possible donors to fund PIE.

The

assistant director of PIE is a direct stakeholder. The assistant director trains all student

facilitators and works collaboratively with the SJE staff to make sure PIE is being run

effectively. This includes recruiting student facilitators, training all participants, and facilitating

workshops.

PROGRAM EVALUATION NORTHWESTERN

10

Logic Model Overview

The PIE program logic model (See Appendix A) focuses on the operations of each of the

three PIE workshops. According to the W.K. Kellogg Foundation (2004), a logic model is a

systematic and visual representation of the relationships and resources provided to create the

intended outcomes of a workshop. We start assessing the logic model by looking at the

following themes: situations, inputs, outputs (activities and participation), followed by outcomes

and impact (short-term, medium-term, and long-term outcomes).

The situation begins with an acknowledgment that there has been no consistent data

produced to assess the effectiveness of each PIE workshop. As previously mentioned, thousands

of people have participated in a PIE workshop but due to capacity and other restraints, the need

for consistent assessments has not been present until now. The presence of a new direction in

SJE not only contributes to the situation, but is also identified as an input in order to operate the

program. The input category also includes two assistant directors, one graduate assistant, one

director for CIC, and the volunteer student facilitators. Another contributing input is a newly-

awarded $15,000 grant for five years for the continuation of the PIE program, which can

influence the effectiveness and implementation of assessments throughout all three PIE

workshops.

Activities in the outputs category are mostly attributed to student facilitator training and

continued dialogues about their experiences as PIE facilitators. Their facilitation expands not to

just training students but also student leaders in various departments such as residence life and

student activities. By offering spaces where the student facilitators can be vocal about their

experiences, SJE is able to check-in with them and enhance their facilitation skills by providing

opportunities for self-reflection and growth. It is noteworthy to mention that not all of the

participants are undergraduate students, but also graduate/professional students, faculty, and/or

PROGRAM EVALUATION NORTHWESTERN

11

staff. We did not identify any major significance in creating two different assessments for both

populations, so one comprehensive assessment will be used for both undergraduate and

graduate/professional students, faculty, and staff who attend a PIE workshop.

The three workshops that are currently offered (Language, Social Identity, and

Inclusivity) all have newly developed learning outcomes. The short-term, medium-term, and

long-term goals of each workshop vary primarily by their topics. These outcomes were

developed based off of the curriculum of each workshop. Although there is some overlap with

terminology, it became very evident during this process that the learning outcomes for each of

the three workshops were slightly different and needed to be addressed through individual

assessments.

Additionally, the three workshop templates are often combined throughout the year,

depending upon the request of the student group or organization. For example, a student

organization can request a workshop on language and social identity, or they can request one

workshop incorporating all three workshops. In these situations, students will receive more than

one assessment immediately after the workshop to assess the effectiveness of these outcomes.

Previous Evaluation

Currently, SJE has no data regarding effectiveness of the workshop content. There is one

evaluation (see Appendix B) that was to be distributed after each PIE workshop. Due to a low

staff in SJE last year which only consisted of an assistant director and one graduate assistant at

the time no data was able to effectively be assesses. PIE is also a fairly new program that has

only been around less than five years. Now that SJE has hired a Director more attention will be

given to the PIE program and assessments are planning to happen with the completion of this

evaluation. The new staff of SJE has hired an undergraduate student who will focus on

PROGRAM EVALUATION NORTHWESTERN

12

collecting data and inputting the survey items into SPSS for the 2017-2018 academic school

years.

The number of workshops and participants who attended a PIE workshop skyrocketed

between 2014-2015 and 2015-2016. According to the assistant director, there were a total of 0

workshops in 2014-2015, while in 2015-2016 there were 23 workshops that reached a total of

324 participants. It is unclear why there was such a drastic shift in the amount of workshops and

participants within one year but it is clear that an assessment and evaluation is needed.

General Evaluation Approach

Completed assessments for PIE are very limited. As previously mentioned, there is only

one assessment in their database that was distributed over the course of one year. Since the

initial onset of the PIE program, no formal assessment has been consistently created for the

student facilitators or the participants. Given the high number of participants during the 2015-

2016 academic years, we decided to focus our general evaluation approach to assessing the

participants of each workshop and evaluating the effectiveness of the learning outcomes to each

workshop topic.

Our general evaluation approach will be primarily outcomes based. “An evaluation is

considered to be formative if the primary purpose is to provide information for program

improvement (Fitzpatrick et al)”. The evaluations we created will focus on the effectiveness of

the PIE workshop and contain elements that address ways in which the workshop can be

improved for future purposes on the topics of general inclusion facilitation, social justice 101,

and inclusive language as well as providing critical feedback to the student facilitators so they

can improve their skills.

The learning outcomes associated to PIE are inclusive to all three major topics that are

offered through the PIE program. For this reason, we created an assessment that can effectively

PROGRAM EVALUATION NORTHWESTERN

13

assess all three of the major workshop topics offered by PIE facilitators. We would like to

effectively measure to what extent the goals that PIE is intended to create are achieved during

these one-time workshops. Ideally, after each workshop the participants will be given a survey

asking questions related to the effectiveness of each workshop based off of content and the

facilitator’s engagement.

As evaluators we decided a formative assessment on PIE would be beneficial to the data

we are trying to collect, analyze, and assess. All data that is collected during this assessment will

be continuously discussed with the stakeholders, specifically in regards to this process, and

throughout this implementation of this evaluation. This evaluation is a processed based

assessment that will analyze short-term, medium-term, and long-term outcomes based on survey

items that participants will complete.

Quantitative Approach

Participants

Participants of PIE workshops vary across the campus of Northwestern University. PIE

facilitators do not go out and recruit student participants. Instead, student groups can request a

PIE workshop (Social justice education: Peer inclusion educators, 2016). Student groups are

defined as any student organization on campus which includes but is not limited to; resident

halls, fraternities, sororities, peer advisors, student government, etc. Many student affairs staff

members will request PIE workshops for their student groups for which they supervise. Staff

members also participate in the PIE workshops so the data will include some surveys completed

by staff members. All of the questions on the PIE workshops post-surveys does include questions

that would be targeted for a staff member or student within the Northwestern community (see

Appendices F-H).

PROGRAM EVALUATION NORTHWESTERN

14

Survey Design

An indirect assessment will be conducted to evaluate the short-term, medium-term, and

long-term outcomes of each of the three workshops: the Language Activity, the Social Identity

Activity, and the Intersectionality Activity. A longitudinal approach will be employed because

an initial, six month, and one year post-survey related to the workshop that they have

participated in will be distributed to participants after each workshop. The groups of students

who participate in PIE workshops are learning social justice key terms right after participating in

a PIE workshop. The three workshops also ask the participants to reflect on the knowledge they

have just learned and how it can apply to spaces outside of the workshop. All of the students are

given individual post-surveys after each workshop, which assesses their knowledge of the social

justice key terms they have learned before participating in a PIE workshop(s) and their

knowledge of the same social justice key terms they learn after participating in the workshop(s).

Our method includes convenience sampling because every participant will complete an

assessment after each workshop (Banta & Palmoba, 2015). Our approach will also include a

longitudinal method of sampling because we ask the participants to complete an initial, six

month, and one year post-survey related to the workshop that they have completed. The survey

response process is complex! This involves comprehension, retrieval, judgment, and response

processes (Tourangeau et al., 2000). We are expecting a lot from the participants by assessing

their comprehension of the material that was presented to them after each workshop. We then

must retrieve the surveys and assess the responses hoping they will provide feedback on how we

can improve the workshops. Not only are we asking for feedback to improve the workshops but

also we are asking the participants to assess how well the facilitators were in each workshop.

The process is complex and we must take this process seriously to ensure funding will continue

for the program.

PROGRAM EVALUATION NORTHWESTERN

15

Instrument Description

PIE has three different workshops; Language Activity, Social Identity Activity, and the

Intersectionality Activity. All three workshops have slightly different surveys that are

administered to each participant after the workshop (see Appendix F-I). The Language Activity

post-survey includes a total of 23 questions (see Appendix F). The first 18 questions are

measured using the Likert Scale (strongly agree, agree, neutral, disagree, and strongly disagree).

The first 12 questions of the survey assess their knowledge of social justice key terms before and

after the workshop. The first six questions gauges their knowledge with the following terms

before participating in the Language Activity workshop these social justice key terms include;

target/agent identities, privilege, power, oppression, discrimination, and collusion. The next six

questions of the survey gauges their knowledge of the social justice key terms mentioned above

after the workshop. The third section of the survey asks participants how they would apply the

social justice key terms learned during this workshop at Northwestern University. This means

defining what discrimination may look like within their community at Northwestern. The fourth

section of the survey ask participants to measure the skills of the facilitators by asking them a

series of questions that measures the environment provided, approachability, timeliness, and

engagement during the workshop. The last section of the survey asks for demographic

information including age, citizenship/nation of origin, class standing, gender, and race.

The Social Identity Activity post-survey includes a total number of 24 questions (see

Appendix G). The first 19 questions are measured using the Likert Scale (strongly agree, agree,

neutral, disagree, and strongly disagree). The first ten questions of the survey assess their

knowledge of social justice key terms before and after the workshop. The first five questions

gauges their knowledge of social justice key terms before the workshop which includes;

target/agent identities, privileged identities as it relates to their own social identity, salient

PROGRAM EVALUATION NORTHWESTERN

16

identity, defining their own salient identities, and how their salient identities are connected to the

terms privilege and oppression. The next five questions assess their understanding of the social

justice key terms mentioned above after the workshop. The third section asks participants to

apply social justice terms discussed during this workshop at Northwestern University. This

means to define what it would look like for them as a student to understand their salient identities

and how it may give them power and privilege within the community at their school. The fourth

section of the survey ask participants to measure the skills of the facilitators by asking them a

series of questions that measures the environment provided, approachability, timeliness, and

engagement during the workshop. The last section of the survey asks for demographic

information including age, citizenship/nation of origin, class standing, gender, and race.

The Intersectionality Activity post survey includes a total of 20 questions (see Appendix

H). The first 15 questions are measured using the Likert Scale (strongly agree, agree, neutral,

disagree, and strongly disagree). The first ten questions of the survey assess their knowledge of

social justice key terms before and after the workshop. The first five questions gauges their

knowledge with the following terms before participating in the Intersectionality workshop key

terms include; privilege, power, oppression, intersectionality, and solidarity. The next five

questions of the survey gauges their knowledge of the social justice key terms mentioned above

after the workshop. The third section of the survey asks participants how they would apply the

social justice key terms learned during this workshop at Northwestern University. This means

defining what solidarity may look like within their community at Northwestern. The fourth

section of the survey ask participants to measure the skills of the facilitators by asking them a

series of questions that measures the environment provided, approachability, timeliness, and

PROGRAM EVALUATION NORTHWESTERN

17

engagement during the workshop. The last section of the survey asks for demographic

information including age, citizenship/nation of origin, class standing, gender, and race.

For all of the post workshop surveys, we are collecting unique ID markers such as their

Northwestern Student ID so that we can make sure to follow up with them during our

administration of our six-month and one year post-surveys. Although all surveys are confidential

they are not anonymous because each participant indicates their student ID number on all of the

surveys. All surveys that students will complete will ask the same questions from the initial

post-surveys (see Appendices F-I).

Pilot Testing

We will pilot test the first two student groups who request a PIE workshop during the fall

quarter. We will ask the two student groups if they could participate in a series of all three

workshops. After each workshop we will immediately collect the initial post-surveys. We will

begin to measure how effective all three of the workshops were by benchmarking and comparing

the before and after survey responses of the social justice terms reviewed in each workshop. We

will assess and evaluate if the facilitators were engaged, on time, created a brave space, were

approachable, and promoted full participation throughout each workshop. All surveys will be

analyzed using SPSS. We will also provide feedback to all participating facilitators so that we

can make sure the workshops are improving. The purpose of pilot testing is to make sure our

survey items are clear, to see how long the surveys take to complete, and to see if the

flow/instructions are clear to the respondents.

Implementation/Administration

Administrations of the post-surveys for each workshop will be handled by the facilitators

present in the space. All facilitators are expected to administer these surveys during the last 10-

15 minutes of their workshop. The expectation for the student groups should have been set

PROGRAM EVALUATION NORTHWESTERN

18

before they participated in the workshop. The facilitators are expected to let all participants know

that a post- workshop survey needs to be completed during the last 10-15 minutes of the

facilitation. The facilitators are then expected to collect the surveys from all participants. All

facilitators will give the surveys to the assistant director, Noor Ali who will then give us the

surveys. We will work in collaboration with the Student Assessment Center to evaluate the post-

surveys for all three workshops. We will only assess the post- workshop surveys from all of the

requested workshops from the fall, winter, and spring quarters. The dates during this time are

September 18, 2017- June 11, 2018. Assessing the data collected during this time will help us

provide immediate feedback to the PIE team so we can improve the quality of the workshops

based on our short term outcomes for the program.

Statistical Analysis/ Results Presentation

All survey data from all three workshops will be organized, described, and collected for

statistical analysis. All information will be coded and analyzed using SPSS. The categorical

demographics asked in all three surveys such as the race and student class standing are nominal

variables which can help provide an in depth evaluation about who are participating in the

workshops. The findings will help us gauge who are our target audience. This will help us

analyze if there is only certain student groups, nationalities, and age ranges requesting

workshops.

As previously mentioned, almost all of the questions on all three workshop surveys

provide interval data with the use of the Likert scale which helps measure the dependent

outcome variables. The information can be treated continuously and can provide useful

information because of the running of means as a descriptive phase for each survey. The mean

scores can be used to gauge the knowledge of social justice key terms before and after the

workshops. The mean scores can also measure the effectiveness of facilitators who are not paid

PROGRAM EVALUATION NORTHWESTERN

19

but are volunteering their time. The mean scores can also measure social justice application of

the social justice key terms reviewed in each workshop before and after student groups and staff

participate (see Appendices F-I). An ANOVA will be used to compare these three means with

each other to see if they significantly differ from one another. This will be helpful as it will help

us determine between which workshops are the most impactful to our participants and if not how

we can improve the content so they will have a better understanding of social justice concepts.

We will also want to conduct comparable analyses of all three workshops to see which workshop

students found to be more beneficial or if participants found all workshops to be equally

beneficial.

Next we will want to analyze how the responses change over time. This can be completed

by using a sample t-test that will provide us with inferential statistics that can gauge if participant

responses will change over time. We will compare results from the initial post-survey, to the six-

month survey, and the one-year survey. This is a longitudinal method that will help us analyze if

the PIE workshops are creating change for the betterment of the campus at Northwestern

University.

Qualitative Rationale

Qualitative Approach

The qualitative approach of the evaluation of the PIE program will include focus groups.

Focus groups are implemented to understand participants’ diverse experiences throughout the

three workshops that PIE offers, such as the Language Activity, Social Identity Activity, and the

Intersectionality Activity, to help provide a deeper meaning of the data collected during the

quantitative analysis (Schuh, 2009). The purpose of the focus groups is to use the findings to

better explain the extent to which student outcomes are met. The outcomes of the workshops are

assessing the effectiveness of the content that the participants are engaged with as well as the

PROGRAM EVALUATION NORTHWESTERN

20

effectiveness of the student facilitators’ ability to engage students with the content. We also

want to provide students with a space to provide a deeper meaning of the social justice key terms

they have learned. We hope participants will provide reflections and feedback on the facilitators’

abilities to provide a brave space (a space where students felt they could be vulnerable and safe

to do so), their timeliness, and the effort each facilitator put into the workshops in which they

have participated as well. The focus group protocols provide probes and questions that ask the

participants why they signed up for the workshops. We want to assess if students are signing up

because they were forced to do so or if they genuinely were interested. The in depth responses

will help us analyze this question. Another probe seeks to understand the type of student that is

signing up for workshops by assessing which student organization or group they are a part of and

how they got involved to participate in the workshops. The last probe seeks to understand what

information they have learned from the workshops based on the conversation that will take place

because of the goals for each workshop (see Appendices J-L).

The program stakeholders desire to launch the focus groups for each PIE workshop

because no evaluation or assessment has been complete to date. The stakeholders are anxious to

collect data and receive feedback on each individual workshop so improvements can be made for

the following 2018-2019 school year

Analytical Approach

Participants who completed the cross-sectional surveys (see Quantitative section) will all

have the opportunity to participate in a focus group. The last question of each survey asked

participants if they would be interested in participating in a focus group for the workshop(s) they

have participated in. The cross-sectional survey asked participants for their Northwestern ID

number. Based on their response their name will be looked up in the system. An email will be

sent to all participants that indicated they would like to participate in a focus group (s) (please

PROGRAM EVALUATION NORTHWESTERN

21

see Appendices N-O). Program evaluators will use maximum variation sampling because of the

expansive and diverse nature of workshop participants. This strategy recognizes general themes

or patterns where different opinions exist (Schuh, 2009). This is important because if you

deliberately try to interview a very different selection of people, their aggregate answers can be

close to the whole population.

Convenience sampling was used throughout the quantitative surveys but evaluators

recognize the importance of varying sampling when analyzing qualitative data. Depending on

the variance of participants for the focus groups, we are reaching out to 30 participants all of

whom will be selected randomly based on their responses. We are hoping to have 5-10

participants’ sign up for a focus group for one of the following workshops that they have

participated. We understand that some participants may have experience with multiple

workshops but we will ask that they only participate in one of the focus groups of their choice.

Initially an email invitation will be sent out on April 11, 2018 (see Appendix N) to all

participants that expressed interest in participating in the focus groups. Participants will have two

weeks to decide if they would like to participate. A reminder email will be sent out on April 25,

2018 (see Appendix O). Once participants confirm that they will be attending a confirmation

email will be sent out immediately after their response. The emails will confirm the date, time,

and location of where each focus group will take place. We will also make sure to highlight that

participants are only to choose one focus group for one workshop that they have participated in

even if they want to participate in multiple workshops.

Focus Group Plan

There are three different focus group protocols for each workshop. According to the

focus group protocol for the Language Activity (see Appendix J) participants will begin with an

introduction from the graduate assistant, assistant director, or director of SJE who begins the

PROGRAM EVALUATION NORTHWESTERN

22

focus group with a simple thank you to the participants for dedicating their time to the activity.

Participants are then explained that the focus group activity will take 60 minutes. It is then

explained to participants that the information gathered today will be shared with SJE staff and be

used to improve the quality of PIE language workshop for future students. The feedback that the

participants provide will be recorded by note takers and a tape recorder so we can be sure to

accurately capture everyone’s responses. The facilitators make sure to reiterate that the

information will be private and they will not be specifically identified as the participants

afterwards. The facilitators of the focus group will then pass out a consent form to every

participant (see Appendix M). It is explained that the consent forms will protect their

confidentiality while participating in the workshops. After, the consent forms have been

collected the introductory question will be asked to begin the dialogue amongst the focus group.

The first question simply asks participants why they signed up for the workshop. Probes are then

asked so that participants can engage in the dialogue in a deeper way, providing more nuance

related to their experiences. Each probe tries to get participants to answer the question in a way

that reflects the goals of each workshop. This process will be completed in all three focus group

workshops. See Appendices J-L to see the learning outcomes and goals for each focus group

based on which workshop participants attended. After the focus group is complete, facilitators

will ask the participants for any feedback or recommendations to improve the workshops. This

question is not limited to the content of workshop but also the ability of the facilitators for the

workshop in which they participated. Again, this process is completed for each individual

workshop with different participants in each focus group.

Focus Group Implementation

The focus groups will be implemented in late April 2017, which is approximately six

months after student/staff have participated in a PIE workshop during the fall quarter. A pilot test

PROGRAM EVALUATION NORTHWESTERN

23

will not be completed on the focus groups beforehand due to a small timeframe and budget costs.

The office of SJE also has a small staff that may not be able to prepare for a pilot focus group

and plan for the focus groups that will be used to assess the workshops. The graduate assistant,

assistant director and director of SJE will be facilitating the focus groups. The assistant director

(Noor Ali) and Director (Robert Brown) are key stakeholders in this evaluation. It will be great

for them to hear firsthand what the participants have learned and to hear feedback about the PIE

facilitators that they select for these roles. The focus groups will be conducted on campus for the

convenience of the student/staff. Light refreshments will be provided in each focus group as

incentives. We recognize that the office of SJE only have three staff members. More staff

members from CIC will be asked in advance to hold focus groups on campus on the same dates

in different rooms across campus.

Focus Group Analysis

Focus group analysis will occur through a process of transcription, coding schemes, and

data analysis. The three focus group sessions will be transcribed. Once the transcripts are

finalized, the evaluators will listen to the audio recordings, review the notes taken by the

moderator and assistant moderators, and read the transcripts. The two evaluators will code each

transcript separately along with the assistance of the Student Assessment Center. A priori codes

and any additional ones that emerged through the coding process, the program goals and learning

outcomes could be explained to stakeholders. Inductive coding will be used as we take from the

findings to theorize the themes. We feel findings will emerge as we are assessing the surveys.

The cutting and sorting technique will be used when analyzing data because it allows the

evaluators to analyze responses to create themes amongst the responses. This method is useful

because it allow us to recognize and pull sub-themes while reading through the evaluations to

pull themes that seem to be important. It is also useful to cut and sort through themes while

PROGRAM EVALUATION NORTHWESTERN

24

laying them in envelopes to create neat piles of the information. The data will also be

triangulated by using the quantitative and qualitative results in order to gain more holistic

interpretations of participants responses from the cross sectional surveys and the focus groups.

Limitations

Due to the structure of CIC and SJE, all of the moderators are very familiar with the PIE

program and the learning outcomes for each workshop. This knowledge that the moderators have

of the program can spark biases when asking participants what they have learned during the

workshop. We will have to recruit a volunteer moderator since there are only two evaluators to

facilitate the third focus group. We also recognize the power dynamics in the room from the

professional staff to students. We know the students may be intimidated by staff so it is even

more important that the evaluators are present in the rooms to ensure the space is as comfortable

as it can be for participants.

Another limitation is acknowledged in our ability to create an effective long-term

outcomes-based assessment. We envisioned this assessment to be distributed via email to all of

the participants of a PIE workshop. The questions for these assessments would be created in

collaboration with the stakeholders and will be influenced by the responses gathered from the

short-term and medium-term focus groups. The structure would be very similar to the

assessment distributed immediately after participating in a PIE workshop. Due to this intentional

collaborative effort on behalf of the stakeholders and SJE staff, in addition to not being

redundant on the types of questions we asked in the final post-year assessment, we did not move

forward with creating what a long-term outcomes-based assessment could look like for each of

the PIE workshops.

PROGRAM EVALUATION NORTHWESTERN

25

Result Presentation

Qualitative data will be presented to key stakeholders in several ways. The responses

from the focus group will be presented as themes to provide feedback on the workshops and the

facilitators’ abilities. The themes will report, negative, neutral, or positive responses. The themes

will be collected by drawing from direct quotations from the transcription of the notes and

recordings of the focus groups. Bar graphs can also be used to show the major themes, for

example the frequency that each theme emerged in the focus groups.

Next Steps

The PIE program at Northwestern University is a relatively new program. It has been in

existence for five years. The staff of SJE has never had the capacity to complete a formal

assessment. This evaluation and assessment will be very beneficial especially for a relatively

new program. We want to provide feedback on how PIE can improve by analyzing the cross

sectional surveys for the quantitative data and the responses of the focus groups for the

qualitative data. Our next steps are to ensure the SJE staff will receive results and feedback

efficiently so the PIE program can begin implementing changes for the 2018-2019 academic

years.

PROGRAM EVALUATION NORTHWESTERN

26

References

About: Northwestern university. (2016). Retrieved from http://www.northwestern.edu/.

Banta, T.W., &Palomba, C.A. (2015). Assessment essentials: Planning, implementing, and

improving assessment in higher education (2 nd ed.). San Francisco, CA: Jossey-Bass

Campus Inclusion & Community. (2016). Division, Campus & Departmental

Information.Unpublished internal document.

Fitzpatrick, J., Sanders, J., &Worthen, B. (2011). Program evaluation: Alternative approaches

and practical guidelines (4th ed.) New York, NY: Longman.

Jacobs, J. (Ed.). (2015, August 24). University halts reorganization of Black House.

Jenkins, T. S. (2010). Viewing cultural practice through a lens of innovation and intentionality.

Culture centers in higher education: Perspectives on identity, theory, and practice, 137-

156.

Multicultural Center space.The Daily Northwestern. Retrieved from

Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA:

Jossey-Bass.

Social justice education: Peer inclusion educators. (2016). Retrieved from

Tourangeau, R., Rips, L. J., &Rasinski, K. (2000). The psychology of survey

response.Cambridge University Press.

W.K. Kellogg Foundation. (2004). Using logic models to bring together planning, evaluation,

PROGRAM EVALUATION NORTHWESTERN

27

Appendices Appendix A: PIE Program Logic Model

 

SITUATION

 

INPUTS

 

OUTPUTS

 
 

Activities

 

Participation

No consistent

2 Assistant

Train student

Undergraduate

assessment to

Directors (only 1

facilitators who

and Graduate

measure the participants

directly oversees PIE)

volunteer to participate in PIE

Students

No data that

 

Host monthly

Student Leaders

demonstrates the

1 Graduate

meetings for

 

effectiveness of these workshops.

Assistant

facilitators to extend dialogue

Faculty and staff at Northwestern

The Assistant Director who

1 Director of SJE

on identity, power, and privilege

University

oversees PIE and

1 Director of CIC

Conduct

Multiple campus

SJE is now working with a new director of SJE.

Volunteer student facilitators who get trained for two days (9 AM- 5

workshops surrounded on three main themes: Language, Social Identity,

partners including Residence Life, Greek Life, and Athletics

 

PM) before the academic year starts

and Intersectionality Present to campus

Full-time Staff from SJE and CIC and one Graduate

Budget from a newly received private donor for $15,000/per year for the next 5 years until 2021

partners such as Residence Life to expand the scope of their workshops

Assistant from SJE

PROGRAM EVALUATION NORTHWESTERN

28

 

OUTCOMES + IMPACT

 
 

Short-term

 

Medium-term

 

Long-term

(Learning/Awareness)

(Action)

(Conditions)

Language:

Language:

 

Language:

 

Participants will be able to

 

Participants will become

 

Participants will feel

define key social justice terms Participants will be able to

more familiar with these terms and concepts Participants will recognize

 

compelled to engage in more dialogues and discussion of these terms

apply key social justice terms to concepts related to power, privilege, and oppression

 

when certain terms and concepts are at-play on campus

 

and concepts Participants will apply these terms and concepts through self-reflection and

Participants will be able to

Social Identity:

 

interactions with other

identify examples of how

 

Participants will apply

students on campus

these terms and concepts are experienced on campus

 

these key terms and concepts to their own social

Social Identity:

 

identities

 

Participants will be able to

Social Identity:

 

Participants will not

 

clearly articulate their ideas

Participants are able to

 

reinforce notions of power,

and thoughts on social

reflect on how they view key terms and concepts related to social justice

privilege, and oppression on campus

 

justice in relation to their social identities Participants will become

relate to their own and

Intersectionality:

 

empowered to engage in

others’ social identities

Participants will apply the

difficult conversations

Participants can define the meaning of salient identities

definition of target and agent identities to the greater campus community

related to social identities on campus

Participants can

Participants will actively engage

Intersectionality:

acknowledge how salient

in self-reflection in order to

Participants will use self-

identities may be intertwined with privilege and oppression in a group setting

deconstruct notions of power, privilege, and oppression on campus

reflection of their target and agent identities as tools to combat power, privilege, and oppression on campus Participants will seek further

Intersectionality:

 

dialogues and discussion on

Participants will be able to

topics and concepts related to

define both target and agent identities, including their own. Participants will gain

social justice on campus, particularly through SJE

knowledge of how privilege, power, and oppression intersect with their social identities Participants will be able to

 

define solidarity Participants will be able to identify practices that exhibit solidarity with social identities other than their own

PROGRAM EVALUATION NORTHWESTERN

29

Assumptions:

  • All participants will students come to the program willing to discuss their identities and awareness of other social identities

  • All participants students are willing to work through their discomfort in talking about identities and difference

  • All participants are going to respect each other’s learning process throughout the workshop

  • All participants will ask questions when they seek clarification on a particular idea or topic

  • All participants will be able to clearly articulate their views and opinions on topics related to social justice and/or their social identities

  • All participants will engage in the three core topics though some action item post- workshop

Environment:

  • New SJE director may want to continue, or reshape, the PIE program

  • The effect of current events such as #BlackLivesMatter on student engagement around

social justice issues and critical self-reflection

  • Prevalence of bias incidents on campus among students, faculty, and staff

  • An academically-driven institution environment may impact the quantity and quality of engagement among participants in a PIE workshop

(Adapted from The University of Arizona, College of Agriculture & Life Sciences: https://extension.arizona.edu/evaluation/content/logic- model-worksheets)

PROGRAM EVALUATION NORTHWESTERN

30

Appendix B: Previous Evaluation

SJE Workshop Evaluation

Workshop topic:

Program/Organization:

Facilitator(s):

Date:

Your feedback is an important element in our ability to evaluate the effectiveness of our workshops and to develop sessions that meet your needs and expectations. Please complete this evaluation and return it at the end of the workshop.

RATINGS:

KNOWLEDGE:

 
  • 1. Before this session, I felt like I understood the workshop’s content:

Vaguely

 

Completely

1

2

3

4

5

2.After this session, I feel like I understand the workshop’s content:

Vaguely

 

Completely

1

2

3

4

5

  • 3. The content presented increased my level of self awareness:

Disagree

Agree

1

2

3

4

5

  • 4. This workshop increased my understanding of perspectives other than

Disagree

Agree

my own:

1

2

3

4

5

Comments:

ENVIRONMENT:

 
  • 5. I feel this was a safe environment for my participation:

Disagree

 

Agree

1

2

3

4

5

  • 6. This workshop allowed me to step outside my comfort zone:

Disagree

 

Agree

1

2

3

4

5

Comments:

APPLICATION:

 
  • 7. There is an increase in my ability to apply the workshop content to my

Disagree

 

Agree

daily life:

1

2

3

45

  • 8. There is an increase in my desire to continue conversations related to

Disagree

 

Agree

thecontent presented:

1

2

3

4

5

  • 9. Because of this workshop, I am more comfortable interacting with people

Disagree

 

Agree

of different identities from my own:

1

2

3

4

5

10. I would be interested in attending future related sessions:

Disagree

Agree

 

1

2

3

4

5

Comments:

FACILITATORS:

 

11. The facilitators were effective in their delivery of the workshop’s

Disagree

 

Agree

content:

1

2

3

4

5

PROGRAM EVALUATION NORTHWESTERN

31

 

Comments:

 

OVERALL:

 

12.

My attendance at the workshop was:

 

Not Beneficial

 

Beneficial

1

2

3

4

5

FEEDBACK:

 

13. Have you attended CIC Workshops before? (Circle one) If yes, how many and what topics?

Yes

No

14.Which component of the workshop was most beneficial to you? Why?

15.Which component of the workshop was least beneficial to you? Why?

16.What information are you still unclear about with regard to the content presented?

  • 17. What topics would you suggest for additional sessions?

18.Other comments:

DEMOGRAPHICS:

Age: ____________

Citizenship/Nation of Origin: ________________________________________

Class Standing:First year

SophomoreJunior

SeniorGraduate/Professional Student

Staff

Faculty Member

Not already listed: ________________________

Gender:Woman

Man

Transgender

Not already listed _______________________________

PROGRAM EVALUATION NORTHWESTERN

32

Race (check all that apply):Native American Asian/Asian American Latino(a) White/Caucasian

/Alaskan Native Black or African American

Native Hawaiian/Pacific Islander

Middle Eastern/North African Not already listed: ______________________________

YOUR FEEDBACK IS VERY MUCH APPRECIATED. THANK YOU!

Please return this evaluation to the facilitators upon completion.

PROGRAM EVALUATION NORTHWESTERN

33

Appendix C: Abbreviated Social Justice Key Concepts

Racism: The systematic subordination of members of targeted racial groups who have relatively little social power in the United States (Blacks, Latino/as, Native Americans, and Asians), by the members of the agent racial group who have relatively more social power (Whites). This subordination is supported by the actions of individuals, cultural norms and values, and the institutional structures and practices of society.

Discriminatory Behaviors: The differential allocation of goods, resources, and services, and the limitation of access to full participation in society based on individual membership in a particular social group.

Social Justice: Includes a vision of society in which the distribution of resources is equitable and all members are empowered and are physically and psychologically safe and secure.

Power: Access to resources that enhance one's chances of getting what one needs in order to lead a safe, comfortable, and productive life.

Prejudice: A set of negative personal beliefs about a social group that leads individuals to prejudge people from that group or the group in general, regardless of individual differences among members of that group.

Privilege: Access to something of value solely because of one’s agent social identity membership.

Individual Oppression: Actions that reflect prejudice against a social group. Societal Oppression: Policies and individual behaviors within institutions that have a differential and/or harmful impact on target groups, whether or not they are so intended.

Cultural Oppression: A set of social norms, economic and political arrangements, roles, rituals, and arts that reflect and reinforce the power of a single agent group or belief system.

Collusion: Thinking, feeling, and acting in ways that accept the dominant group’s ideology about one’s own target group and accepting a definition of oneself that is hurtful and limiting. This involves demonstrating the devaluation of one’s own group and of oneself as a member of that group. Note: In contrast to an internalized “ism”, collusion involves the

target group member actively supporting oppression, as opposed to a passive internalization.

PROGRAM EVALUATION NORTHWESTERN

34

Appendix D: Intergroup Relations (IGR) Language Activity

Purpose: To introduce key IGR concepts to a group of participants. This activity also serves to gauge the participants’ levels of understanding regarding these concepts.

Concepts Used: Agent, Target, Privilege, Power, Oppression, Discrimination, and Collusion

Set Up: Write each individual concept at the top of large, separate sheets of paper. Place the sheets of paper together dyadically as follows: AgentTarget, PrivilegePower, OppressionDiscrimination. Collusion is not paired with any other concept.

Activity:

Part One: Each participant is to place a check mark on the individual sheets as a reflection of how well they understand each concept. The sheet represents a continuum of understanding, with one side representing fully knowledgeable about said concept and the other side representing not-so-knowledgeable about said concept. The use of a check mark means that persons do not have to publicly express their knowledge or lack thereof. Participants are free to discuss where they placed their checks if so desired.

Part Two: After check marks have been made on all sheets, the facilitator(s) explains that the purpose is build clarity on what these concepts mean. They note that there are not clear definitions, but that the conversation should bring to light some key aspects of each concept.

The facilitator then begins a discussion around each concept, starting with either agent or target, and then adding the other, seeing what interpretations are held by participants. Follow-up questions probe for examples and distinctions between other words. Clarification is also sought if participant uses another concept word within their interpretation, to see if they can explain it in more simple words. Steadily, the facilitator adds each new word. After each word, the facilitator may choose to point out the explanation on the key concept list, highlighting again that these are not meant to be definitions.

Challenges:

Facilitator needs to have a good sense of what each concept is.

It is challenging to affirm participants when they offer interpretations that are very

different from what IGR views to be the more fitting conceptualization, while pushing them to understand the IGR perspective. It is helpful for the facilitator to have examples that help illustrate these concepts

readily available, in case the group doesn’t generate enough. This can be a lengthy exercise, and therefore challenging to keep people’s focus for

the entirety. The duration completely depends on the number of group and prior understanding of each group member.

PROGRAM EVALUATION NORTHWESTERN

35

Reminding participants that there is no clean definition is challenging because their

instinct is to search for an “answer”. These conceptualizations are far more complex

than that.

It is challenging to pull out the key points from what participants are saying, because a lot of times (unless they have practiced articulating these

conceptualizations) they don’t have concise thoughts.

PROGRAM EVALUATION NORTHWESTERN

36

Appendix E: Agenda for Intersectional Language Activity

 

Facilitator to lead

Agenda Welcome- Introductions of facilitators

 

Icebreaker- (check and see which ones were done before) Rock, paper, scissors challenge Look up, Look down Shake it off

 

Review Guidelines

 

Define Intersectionality Review Language Power

 

Privilege

Oppression

Target and Agent

Intersectionality Activity Divide into small groups of 4 (10 groups) ~ 3:50-3:55pm

 

Choose an identity you are most comfortable with (aka

salient) and share with group. Do you hold privilege in this identity? Are you oppressed? How? ~3:55-4:03 (6 min per round, 1.5 min per person) 3 Other identities (6 min round

Share out- 10min- 4:20-4:30pm (10) How was this for you?

 

What surprised you?

Which identity was the easiest for you to interact with?

Which identity was the most difficult for you to interact

with? What commonalities did you find?

Back into small groups Solidarity- 10min 4:30-4:40 (10) How does this impact how you might interact with one

another? How do you relate with others? How does this help you to relate with others who have

different identities than you hold? (We have different identities but we all are oppressed and hold privilege---- how does this allow us to move past individual identities?) How can you show solidarity/create coalitions with these things in mind? 4:40-4:55pm (15 min)

Come back to large group debrief Closure- One word Whip

 

PROGRAM EVALUATION NORTHWESTERN

37

Appendix F (Language Activity Survey)

Peer Inclusion Educator’s Post Workshop Survey

Workshop: Language Activity Name of Peer Facilitator(s):

Date:

Northwestern Student ID Number: 000 Please complete the following post-workshop survey. Please complete the following survey about your experiences in today’s PIE workshop. All of your answers are confidential but we are asking that you provide your Northwestern Student ID Number so that we can follow up with a 6-month post survey of the content that you have learned in our workshop. Your participation is appreciated. If you have any questions please feel free to contact the assistant director of the Social Justice Education Department Noor Ali, noor.ali@northwestern.edu. After you have completed this survey the Peer Facilitators will collect them. Thank you!

Social Justice Key Terms Knowledge

The following questions are centered on knowledge of the social justice terms you have learned and or been introduced to by engaging in this workshop. Please rate your overall agreement on a scale from Strongly Agree to Strongly Disagree.

  • 1. As a result of participating in this workshop, target/agent identities.

I can provide a deeper definition of

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 2. As a result of participating in this workshop, I can provide a deeper definition of privilege. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 3. As a result of participating in this workshop, I can provide a deeper definition of power. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 4. As a result of participating in this workshop, I can provide a deeper definition of oppression. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 5. As a result of participating in this workshop, I can provide a deeper definition of discrimination. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

PROGRAM EVALUATION NORTHWESTERN

38

Strongly Agree

Agree

Neutral

Social Justice Key Terms Application

Disagree

Strongly Disagree

The following questions are centered on the application of the social justice terms you have

learned and or been introduced to by engaging in this workshop. Please rate your overall agreement on a scale from Strongly Agree to Strongly Disagree.

  • 1. I can identify target/agent identities in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 2. I can identify privilege in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 3. I can identify power in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 4. I can identify oppression in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 5. I can recognize when others are discriminated against in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 6. I can recognize when I am discriminated against in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 7. I can differentiate between individual and systematic discrimination in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 8. I can identify collusion in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

PROGRAM EVALUATION NORTHWESTERN

39

Peer Inclusion Educators

The following questions are centered on the facilitation skills of the peer facilitators who completed this workshop. Please rate your overall agreement on a scale from Strongly Agree to Strongly Disagree.

  • 1. The facilitators provided ground rules prior to the start of the workshop to engage in dialogue around social justice key terms. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 2. The facilitators encouraged vulnerability on behalf of the participants throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 3. The facilitators shared their own experiences with the language activity throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 4. The facilitators promoted verbal participation throughout the workshop via large and small-group discussions. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 5. The facilitators were on time and set up the space prior to the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 6. The facilitators were positive throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 7. The facilitators were engaging throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 8. The facilitators were open to questions throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

PROGRAM EVALUATION NORTHWESTERN

40

DEMOGRAPHICS:

Age: ____________

Citizenship/Nation of Origin: ________________________________________

Class Standing: First-Year Undergraduate

Sophomore Junior

SeniorGraduate/Professional Student

Staff

Faculty Member

Not already listed: ________________________

Gender: Woman

Man

Transgender

Not already listed _______________________________

Race (check all that apply): Native American/Alaskan Native Asian/Asian American Latino(a) or Latinx Middle Eastern/North African Not already listed: ______________________________

Black or African American Native Hawaiian/Pacific Islander White/Caucasian

YOUR FEEDBACK IS VERY MUCH APPRECIATED. THANK YOU!

PROGRAM EVALUATION NORTHWESTERN

41

Appendix G (Social Identity Survey)

Peer Inclusion Educator’s Post Workshop Survey

Workshop: Social Identity Name(s) of Peer Facilitators:

Date:

Northwestern Student ID Number: 000 Please complete the following post-workshop survey. Please complete the following survey about your experiences in today’s PIE workshop. All of your answers are confidential but we are asking that you provide your Northwestern Student ID Number so that we can follow up with a 6-month post survey of the content that you have learned in our workshop. Your participation is appreciated. If you have any questions please feel free to contact the assistant director of the Social Justice Education Department Noor Ali, noor.ali@northwestern.edu. After you have completed this survey the Peer Facilitators will collect them. Thank you!

Social Identity Key Terms Knowledge

The following questions are centered on knowledge of the social justice terms you have learned and or been introduced as it relates to your social identities by engaging in this workshop. Please rate your overall agreement on a scale from Strongly Agree to Strongly Disagree.

  • 1. As a result of participating in this workshop, target/agent identities.

I can provide a deeper definition of

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 2. As a result of participating in this workshop, I can provide a deeper definition of privilege as it relates to my social identity. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 3. As a result of participating in this workshop, I can provide a deeper definition of salient identities. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 4. As a result of participating in this workshop, I can provide a deeper definition of my own salient identities. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 5. As a result of participating in this workshop, I can provide a deeper definition of how my salient identities are connected to the terms privilege and oppression. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

PROGRAM EVALUATION NORTHWESTERN

42

Social Identity Key Terms Application

The following questions are centered on the application of the social identity key terms you have

learned and or been introduced to by engaging in this workshop. Please rate your overall agreement on a scale from Strongly Agree to Strongly Disagree.

  • 1. I can identify target/agent identities in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 2. I can identify privilege in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 3. I can identify the term salient identity in relation to the campus community at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 4. I can identify my salient identities as being a part of a privileged or oppressed group at Northwestern University.

Strongly Agree

Agree

Peer Inclusion Educators

Neutral

Disagree

Strongly Disagree

The following questions are centered on the facilitation skills of the peer facilitators who completed this workshop. Please rate your overall agreement on a scale from Strongly Agree to Strongly Disagree.

  • 1. The facilitators provided ground rules prior to the start of the workshop to engage in dialogue around social justice key terms. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 2. The facilitators encouraged vulnerability on behalf of the participants throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 3. The facilitators shared their own experiences with the language activity throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

PROGRAM EVALUATION NORTHWESTERN

43

  • 4. The facilitators promoted verbal participation throughout the workshop via large and small-group discussions. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 5. The facilitators were on time and set up the space prior to the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 6. The facilitators were positive throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 7. The facilitators were engaging throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 8. The facilitators were open to questions throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

DEMOGRAPHICS:

Age: ____________

Citizenship/Nation of Origin: ________________________________________

Class Standing: First-Year Undergraduate Sophomore Junior Senior Graduate/Professional Student Staff Faculty Member Not already listed: ________________________

Gender: Woman Man Transgender Not already listed _______________________________

Race (check all that apply): Native American/Alaskan Native Asian/Asian American Latino(a) or Latinx Middle Eastern/North African Not already listed: ______________________________

Black or African American Native Hawaiian/Pacific Islander White/Caucasian

YOUR YOUR FEEDBACK FEEDBACK IS IS VERY VERY MUCH MUCH APPRECIATED. APPRECIATED. THANK THANK YOU! YOU!

PROGRAM EVALUATION NORTHWESTERN

44

Appendix H (Intersectionality Survey)

Peer Inclusion Educator’s Post Workshop Survey

Workshop: Intersectionality Activity

Name(s) of Peer Facilitators:

Date:

Northwestern Student ID Number: 000

Please complete the following post-workshop survey. Please complete the following survey

about your experiences in today’s PIE workshop. All of your answers are confidential but we

are asking that you provide your Northwestern Student ID Number so that we can follow up with

a 6 month post survey of the content that you have learned in our workshop. Your participation is

appreciated. If you have any questions please feel free to contact the assistant director of the

Social Justice Education Department Noor Ali, noor.ali@northwestern.edu. After you have

completed this survey the Peer Facilitators will collect them. Thank you!

Social Justice Key Terms Knowledge

The following questions are centered on knowledge of the social justice terms you have learned

and or been introduced to by engaging in this workshop. Please rate your overall agreement on

a scale from Strongly Agree to Strongly Disagree.

  • 1. As a result of participating in this workshop, I can provide a deeper definition of

privilege.

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 2. As a result of participating in this workshop, I can provide a deeper definition of power.

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 3. As a result of participating in this workshop, I can provide a deeper definition of

oppression.

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 4. As a result of participating in this workshop, I can provide a deeper definition of

intersectionality.

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 5. As a result of participating in this workshop, I can provide a deeper definition of

solidarity.

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

PROGRAM EVALUATION NORTHWESTERN

45

Social Justice Key Terms Application

The following questions are centered on the application of the social justice terms you have

learned and or been introduced to by engaging in this workshop. Please rate your overall

agreement on a scale from Strongly Agree to Strongly Disagree.

  • 1. I can identify my own privilege(s) in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 2. I can identify my own power in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 3. I can identify oppression in regards to my social identities in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 4. I can identify intersectionality in regards to my own social identities in various settings at Northwestern University. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 5. I can identify solidarity in various settings at Northwestern University. Strongly Agree

Agree

Peer Inclusion Educators

Neutral

Disagree

Strongly Disagree

The following questions are centered on the facilitation skills of the peer facilitators who

completed this workshop. Please rate your overall agreement on a scale from Strongly Agree to

Strongly Disagree.

  • 1. The facilitators provided ground rules prior to the start of the workshop to engage in dialogue around social justice key terms. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 2. The facilitators encouraged vulnerability on behalf of the participants throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 3. The facilitators shared their own experiences with the language activity throughout the workshop.

PROGRAM EVALUATION NORTHWESTERN

46

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 4. The facilitators promoted verbal participation throughout the workshop via large and small-group discussions. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 5. The facilitators were on time and set up the space prior to the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 6. The facilitators were positive throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 7. The facilitators were engaging throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

  • 8. The facilitators were open to questions throughout the workshop. Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

DEMOGRAPHICS:

Age: ____________

Citizenship/Nation of Origin: ________________________________________

Class Standing: First-Year Undergraduate Sophomore Junior Senior Graduate/Professional Student Staff Faculty Member Not already listed: ________________________

Gender: Woman Man Transgender Not already listed _______________________________

Race (check all that apply): Native American/Alaskan Native Asian/Asian American Latino(a) or Latinx Middle Eastern/North African Not already listed: ______________________________

Black or African American Native Hawaiian/Pacific Islander White/Caucasian

YOUR FEEDBACK IS VERY MUCH APPRECIATED. THANK YOU!

YOUR FEEDBACK IS VERY MUCH APPRECIATED. THANK YOU!

PROGRAM EVALUATION NORTHWESTERN

Appendix I: Survey Item Matrix

47

PROGRAM EVALUATION NORTHWESTERN Appendix I: Survey Item Matrix 47

PROGRAM EVALUATION NORTHWESTERN

48

PROGRAM EVALUATION NORTHWESTERN 48

PROGRAM EVALUATION NORTHWESTERN

49

Appendix J: Focus Group Protocol for the Language Activity

Post-PIE Workshop Attendees

Overarching Objective of the Language Workshop Focus Group:

What effect, if any, does the language workshop have on its participants immediately after the

workshop? What effect do the facilitators have on the participants immediately after the

workshop?

Learning Outcomes:

Participants will feel compelled to interact with these terms and concepts in other spaces

on campus

Participants will find groups or organizations on campus that engage with social justice

concepts

Participants will initiate dialogues and discussion of these terms and concepts on campus

Participants will partake in an ongoing self-reflection regarding social justice key

concepts with other students on campus

Introduction:

Focus Group Script

Thank you for joining us today to participate in this focus group. My name is (Graduate

Assistant, Assistant Director, or Director of SJE). Our goal through this focus group is to hear

about your experiences in PIE’s language activity workshop and how those experiences have

related to your experience at Northwestern University over the last six months. By taking time

to speak with us about your experiences on campus and those you had during your language

workshop, we hope to add a greater depth to the feedback that was provided during the post-

workshop surveys. We greatly appreciate your participation thus far. This will be the last time

that we ask for your involvement in our evaluation.

The discussion should take no longer than 60 minutes. Please note that you are free to skip any

questions that are raised if at any point you feel uncomfortable. Please feel free to keep eating

snacks, refill on drinks, and use the restroom, etc., throughout our discussion.

The information gathered today will be shared with SJE staff and used to improve the quality of

PIE language workshop for future students. The feedback that you provide during this

discussion will be recorded by [state name(s) of notetaker(s)] and a tape recorder so we can be

sure to accurately capture everyone’s responses. This information will be confidential and will

not be specifically identified to you afterwards. Of course, there are no right or wrong answers,

and we’d like to hear from everyone if possible.

At this time I ask that you review our interview consent form to make sure that you understand

the details of what will occur today. By signing the form you consent to participate in this

interview. Once we have everyone’s consent forms, we will begin the discussion.

PROGRAM EVALUATION NORTHWESTERN

50

Introductory Question:

To begin, we’d like to hear about your reasons for completing this particular PIE

workshop…

Probe: What were some of the reasons that led you to sign up for a language

workshop through PIE?

Probe: Was it a mandatory part of your training for a student group/organization?

Probe: Did you sign up yourself through a student group or organization?

Goal Area 1: Participants are able to describe a deeper meaning of the following social justice

terms: target/agent identities, privilege, power, oppression, discrimination, and collusion.

Question 1.1: Since completing the workshop, in what ways have you been able to

reflect on how the terms target/agent identities, privilege, power, oppression,

discrimination, and/or collusion are present on campus?

o

Probe: When have you found it easier to do this?

o

Probe: Are there particular situations when this is more difficult? For what

 

reasons?

 

o

Probe: Is there a strategy or technique you learned during the language

 

workshop that has proven useful in addressing these situations?

Question 1.2: Can you identify examples of how these terms have appeared in various

settings through your own personal experience?

Question 1.3: How did you address the situation/context that you identified when one

or more of the terms were present on campus?

o

Probe: What prompted you to address the situation as you did?

o

Probe: How did you feel after you did this?

Goal Area 2: Participants can recognize systems of privilege at Northwestern University and

can identify power relations in various settings on campus.

Question 2.1: Can you describe whether you’ve been able to recognize systematic

privilege on Northwestern’s campus since your participation in the workshop?

o

Probe: What do you think contributed toward your ability to recognize

systematic privilege at NU?

o

Probe: How did you feel when you recognized this type of privilege at NU?

o

Probe: Did you process or reflect on that experience with anyone? If so, how

was that?

o

Probe: Were any of the factors associated with one of your privileged identity

groups?

o

Probe: If you could change anything about the how you addressed the

situation, what would it be and why?

Goal Area 3: Participants can recognize systems of oppression at Northwestern University and

can identify when themselves or students different from them have been discriminated against

based on the system in place.

Question 3.1: Can you describe whether you’ve been able to recognize systematic

oppression on Northwestern’s campus since your participation in the workshop?

PROGRAM EVALUATION NORTHWESTERN

51

o

Probe: How did you identify the situation/context with a system of

oppression?

o

Probe: Since the PIE language workshop, have you proactively fostered a

space, or spaces, for others to be vulnerable and share their personal stories?

o

Probe: Since the PIE language workshop, have you proactively fostered a

group or student organization that engages with one or more social justice

term?

Goal Area 4: Participants are able to provide critical feedback on the facilitators’ abilities to

provide a brave space for students to be vulnerable and share their personal stories, timeliness,

and effort into providing a learning environment.

Questions 4.1: The idea of a brave space is something that comes up in our area a lot.

Based of your experience on campus, how would you define brave space?

o

Probe: Based on our definition of brave space, did the facilitator(s) provide

 

this during your workshop?

 

o

Probe: Did the facilitator or facilitators provide a space where you were able

 

to have a voice in the conversation?

 

o

Probe: What did the facilitator(s) do to create this space?

o

Probe: Was there anything different the facilitator(s) could have done to create

 

a brave space for you?

Question 4.2: Were there any positive actions or statements made by the facilitator, or

facilitators, that still resonate with you?

Question 4.3: Were there any negative actions or statements made by the facilitator,

or facilitators, that you still remember from the workshop?

Closing Segment: Improvement and Recommendations

Before we end, we would like to take this opportunity to ask you if there are any improvements

or changes that you would make to the language workshop.

Is there anything else you would like to add before we end our discussion? If not, then this

concludes our time together. We appreciate you sharing your feedback with us today. The

success of the PIE program is dependent upon the valuable feedback that you all have shared

during today’s discussion. Thank you for taking the time today to participate in this discussion!

PROGRAM EVALUATION NORTHWESTERN

52

Appendix K: Focus Group Protocol for Social Identity

Post-PIE Workshop Attendees

Overarching Objective of the Social Identity Workshop Focus Group:

What effect, if any, does the social identity workshop have on its participants immediately after

the workshop? What effect do the facilitators have on the participants immediately after the

workshop?

Learning Outcomes:

Participants will apply key social justice terms and concepts to their own social identities

Participants will be self-aware of their salient identities in regards to power, privilege,

and oppression on campus

Participants will be able to clearly articulate their social identities in the context of power,

privilege, and oppression

Participants will be able to engage in difficult conversations related to social identities on

campus

Focus Group Script

Introduction:

Thank you for joining us today to participate in this interview. My name is (Graduate Assistant,

Assistant Director, or Director of SJE). Our goal through this focus group is to hear about your

experiences in PIE’s social identity workshop and how those experiences have related to your

experience at Northwestern University over the last six months. By taking time to speak with us

about your experiences on campus and those you had during your language workshop, we hope

to add a greater depth to the feedback that was provided during the post-workshop surveys. We

greatly appreciate your participation thus far. This will be the last time that we ask for your

involvement in our evaluation.

The discussion should take no longer than 60 minutes. Please note that you are free to skip any

questions that are raised if at any point you feel uncomfortable. Please feel free to keep eating

snacks, refill on drinks, and use the restroom, etc., throughout our discussion.

The information gathered today will be shared with SJE staff and be used to improve the quality

of PIE language workshop for future students. The feedback that you provide during this

discussion will be recorded by [state name(s) of notetaker(s)] and a tape recorder so we can be

sure to accurately capture everyone’s responses. This information will be private and will not be

specifically identified to you afterwards. Of course, there are no right or wrong answers, and

we’d like to hear from everyone if possible.

At this time I ask that you review our interview consent form to make sure that you understand

the details of what will occur today. By signing the form you consent to participate in this

interview. Once we have everyone’s consent forms, we will begin the discussion.

PROGRAM EVALUATION NORTHWESTERN

53

Introductory Question:

To begin, we’d like to hear about your reasons for completing this particular PIE

workshop…

Probe: What were some of the reasons that led you to sign up for a social identity

workshop through PIE?

Probe: Was it a mandatory part of your training for a student group/organization?

Probe: Did you sign up yourself through a student group or organization?

Probe: What were some meaningful takeaways from the workshop?

Goal Area 1: Participants are able to define what identities are oppressed and what identities

gives them power and privilege in our US society.

Question 1.1: Have you been able to reflect on your own oppressed and privileged

identities?

Question 1.2: Have you been able to reflect on how your oppressed and privileged

identities are present on campus?

o

Probe: Have you felt ostracized due to your oppressed and/or privileged identities

through various settings on campus?

Goal Area 2: Participants are able to recognize and state their salient identities, meaning the

identities that they identify with the most within US context such as their race, ethnicity, religion,

sexual orientation, gender, etc.

Question 2.1: Can you elaborate on how you describe your salient identities to others

in relation to power, privilege, and oppression?

o

Probe: Did participation in the social identity workshop influence or change how

you describe your salient identities to others?

o

Probe: What are some of the challenges that have come with the self-reflection of

your salient identities?

Goal Area 3: Participants are able to provide critical feedback on the facilitators’ abilities to

provide a brave space for students to be vulnerable and share their personal stories, timeliness,

and effort into providing a learning environment.

Questions 3.1: The idea of a brave space is something that comes up in our area a lot.

Based of your experience on campus, how would you define brave space?

o

Probe: Based on our definition of brave space, did the facilitator(s) provide

 

this during your workshop?

 

o

Probe: Did the facilitator, or facilitators, provide a space where you were able

 

to have a voice in the conversation?

 

o

Probe: What did the facilitator(s) do to create this space?

o

Probe: Was there anything different the facilitator(s) could have done to create

 

a brave space for you?

Question 3.2: Were there any positive actions or statements made by the facilitator, or

facilitators, that still resonate with you?

Question 3.3: Were there any negative actions or statements made by the facilitator,

or facilitators, that you still remember from the workshop?

PROGRAM EVALUATION NORTHWESTERN

54

Closing Segment: Improvement and Recommendations

Before we end, we would like to take this opportunity to ask you if there are any improvements

or changes that you would make to the social identity workshop.

Is there anything else you would like to add before we end our discussion? If not, then this

concludes our time together. We appreciate you sharing your feedback with us today. The

success of the PIE program is dependent upon the valuable feedback that you all have shared

during today’s discussion. Thank you for taking the time today to participate in this discussion!

PROGRAM EVALUATION NORTHWESTERN

55

Appendix L: Focus Group Protocol for Intersectionality

Post-PIE Workshop Attendees

Overarching Objectives of the Intersectionality Workshop Focus Group:

What effect, if any, does the intersectionality workshop have on its participants immediately

after the workshop? What effect do the facilitators have on the participants immediately after the

workshop?

Learning Outcomes:

Participants will apply the definition of target and agent identities to the greater campus

community

Participants will engage in self-reflection by being exposed to notions of power,

privilege, and oppression on campus

Participants will use self-reflection of their target and agent identities as tools to combat

power, privilege, and oppression on campus

Participants will seek further dialogues and discussion on topics and concepts related to

social justice on campus, particularly through SJE

Introduction:

Interview Protocol

Thank you for joining us today to participate in this interview. My name is (Graduate Assistant,

Assistant Director, or Director of SJE). Our goal through this focus group is to hear about your

experiences in PIE’s intersectionality workshop and how those experiences have related to your

experience at Northwestern University over the last six months. By taking time to speak with us

about your experiences on campus and those you had during your language workshop, we hope

to add a greater depth to the feedback that was provided during the post-workshop surveys. We

greatly appreciate your participation thus far. This will be the last time that we ask for your

involvement in our evaluation.

The discussion should take no longer than 60 minutes. Please note that you are free to skip any

questions that are raised if at any point you feel uncomfortable. Please feel free to keep eating

snacks, refill on drinks, and use the restroom, etc., throughout our discussion.

The information gathered today will be shared with SJE staff and be used to improve the quality

of PIE language workshop for future students. The feedback that you provide during this

discussion will be recorded by [state name(s) of notetaker(s)] and a tape recorder so we can be

sure to accurately capture everyone’s responses. This information will be private and will not be

specifically identified to you afterwards. Of course, there are no right or wrong answers, and

we’d like to hear from everyone if possible.

At this time I ask that you review our interview consent form to make sure that you understand

the details of what will occur today. By signing the form you consent to participate in this

interview. Once we have everyone’s consent forms, we will begin the discussion.

PROGRAM EVALUATION NORTHWESTERN

56

Introductory Question:

To begin, we’d like to hear about your reasons for completing this particular PIE

workshop…

Probe: What were some of the reasons that led you to sign up for an

intersectionality workshop through PIE?

Probe: Was it a mandatory part of your training for a student group/organization?

Probe: Did you sign up yourself through a student group or organization?

Probe: What were some meaningful takeaways from the workshop?

Goal Area 1: Participants are able to define the intersectionality of their target/ agent identities,

meaning defining what identities are oppressed and what identities gives them power and

privilege in our US society.

Question 1.1: Since completing the PIE workshop six months ago, have you been

able to reflect on your own target/agent identities in relation to the definition of

intersectionality?

o

Probe: Did participation in the intersectionalty workshop influence or change

how you describe your target/agent identities to others?

o

Probe: What are some of the challenges that have come with the self-

reflection of your target/agent identities in relation to intersectionality?

Goal Area 2: Participants are able to recognize that their target identities do not outweigh the

power of their privilege identities, but that all of their identities intersect together.

Question 2.1: Over the last six months, have you used your privileged identities as a

tool to combat systems of oppression on campus?

o

Probe: Are there times when this has been easier to do?

o

Probe: What have been some of the challenges in utilizing your privileged

identities to raise awareness about the marginalization of target identity

groups?

Goal Area 3: Participants are able to provide detailed descriptions of what solidarity would look

like amongst their peers at Northwestern University.

Question 3.1: How have you demonstrated allyship with other marginalized identity

 

groups on campus?

 

o

Probe: Were you able to better define allyship as a result of the intersectionality

 

workshop?

 

o

Probe: What spaces on campus, if any, have allowed you to practice allyship?

Question 3.2: What are some techniques or strategies related to solidarity that you

have implemented since your participation in the intersectionality workshop?

o

Probe: Did you learn this from the intersectionality workshop? Or from a

different experience?

Goal Area 4: Participants are able to provide critical feedback on the facilitators’ abilities to

provide a brave space for students to be vulnerable and share their personal stories, timeliness,

and effort into providing a learning environment.

PROGRAM EVALUATION NORTHWESTERN

57

Questions 4.1: The idea of a brave space is something that comes up in our area a lot.

Based of your experience on campus, how would you define brave space?

o

Probe: Based on our definition of brave space, did the facilitator(s) provide

 

this during your workshop?

 

o

Probe: Did the facilitator, or facilitators, provide a space where you were able

 

to have a voice in the conversation?

 

o

Probe: What did the facilitator(s) do to create this space?

o

Probe: Was there anything different the facilitator(s) could have done to create

 

a brave space for you?

Question 4.2: Were there any positive actions or statements made by the facilitator, or

facilitators, that still resonate with you?

Question 4.3: Were there any negative actions or statements made by the facilitator,

or facilitators, that you still remember from the workshop?

Closing Segment: Improvement and Recommendations

Before we end, we would like to take this opportunity to ask you if there are any improvements

or changes that you would make to the intersectionality workshop.

Is there anything else you would like to add before we end our discussion? If not, then this

concludes our time together. We appreciate you sharing your feedback with us today. The

success of the PIE program is dependent upon the valuable feedback that you all have shared

during today’s discussion. Thank you for taking the time today to participate in this discussion!

PROGRAM EVALUATION NORTHWESTERN

58

Appendix M: Consent Form

CONSENT TO PARTICIPATE IN PEER INCLUSION EDUCATORS (PIE) IMPACT

FOCUS GROUP

Project Title: Peer Inclusion Educators (PIE) Impact Focus Group

Evaluators: Anthony Sis and Lorrena Johnson

Introduction:

You are being asked to participate in a focus group regarding your experience outside of the Peer

Inclusion Educators (PIE) workshop. Because you completed a post-workshop survey and

expressed interest in following up regarding your experience on campus, we are interested in

hearing more about your thoughts and experience.

Please read this form carefully and ask any questions you may have as you decide to participate

in this interview.

Purpose:

The purpose of this interview is to gather information about students’ post-PIE workshop

experience at Northwestern University to better understand the effectiveness of the program.

The information generated from this interview will be used to improve the content of the three

PIE workshops offered (Social Justice Key Concepts, Social Identity, Intersectionality).

Risks/Benefits:

There are no foreseeable risks involved in participating in this research beyond those

experienced in everyday life. A key benefits will be the potential improvement of the PIE

program for future participants.

Confidentiality:

There will be no identifying information attached to your responses in the interview. A report

will be compiled of basic themes that emerge from the interviews. The information will be

shared with the Social Justice Education (SJE) staff and staff from Campus Inclusion and

Community (CIC).

Voluntary Participation:

Participation in this interview is voluntary. If you do not want to take part, simply decline

participation. If at any time you feel uncomfortable answering a question, you may remove

yourself and/or decide not to answer the question.

Contacts/Questions:

If you have any questions about the interview, please contact Noor Ali at 847-491-8063 or

PROGRAM EVALUATION NORTHWESTERN

59

Statement of Consent:

Your signature below indicates that you have read and understood the information provided in

this document, have had an opportunity to ask questions, and agree to participate in this

interview.

You will be given a copy of this form for your records.

_______________________________________________

Participant’s Signature

___________________

Date

_______________________________________________

Evaluator’s Signature

Date

___________________

PROGRAM EVALUATION NORTHWESTERN

60

Appendix N: Protocol Invitations

Invitation request if at least ten people have not expressed interest in focus group

Re: Invitation to Join a PIE Impact Research Discussion

Thank you for attending a PIE workshop this academic year. We’re interested in learning about

the outcomes of the PIE workshop you attended on (Language Activity/ Social Identity/

Intersectionality). Because of your participation, we are looking to hear back from you about

your experience outside of PIE workshop on campus. We hope that you will want to provide

your thoughts.

The focus groups will be conducted over the next few weeks by a graduate assistant, assistant

director, or director of Social Justice Education (SJE) with additional support from Center for

Inclusion and Community (CIC) staff. The format of the focus group is to include no more than

ten previous student participants and should take no longer than ninety minutes to complete.

They will be scheduled from 12:00 1:00 P.M. or from 4:00- 5:00 P.M., depending on the

availability of the participants. Light snacks and beverages will be provided.

Please let me know if you are interested in joi