1

Understanding Implicit Bias:

Evaluating a Course Designed to Encourage Racial Equity

Julia Heatherwick and Kristin Wolf

Instructional Science and Technology, California State University, Monterey Bay

IST 622: Assessment and Evaluation

Dr. Bude Su

July 27, 2021

 2

Table of Contents
Understanding Implicit Bias: Evaluating a Course Designed to Encourage Racial Equity............3
Methodology....................................................................................................................................3

Instructional Product....................................................................................................................3
Pilot Learners...............................................................................................................................4
Learners.......................................................................................................................................5
Tryout Conditions........................................................................................................................6
Tryout Process.............................................................................................................................6
Usability Observation..................................................................................................................6
Pre-Test........................................................................................................................................6
Post-Test......................................................................................................................................7

Results..............................................................................................................................................7

Entry Conditions..........................................................................................................................7
Instruction....................................................................................................................................7
Outcomes.....................................................................................................................................8
Data Summary.............................................................................................................................9
Recommendations......................................................................................................................16
Summary....................................................................................................................................17

References......................................................................................................................................19
Appendices....................................................................................................................................20
 3

Understanding Implicit Bias: Evaluating a Course Designed to Encourage Racial Equity

The Implicit Bias Module Series is an online course designed by the Kirwan Institute for

the Study of Race and Ethnicity. The instruction addresses the unconscious, cognitive processes

that are associated with implicit bias. The Kirwan Institute, located at Ohio State University, has

a mission to provide training that can aid in the “creation of a just and inclusive society, where

all people and communities have the opportunity to succeed” (Kirwan Institute for the Study of

Race and Ethnicity, 2021).

The Implicit Bias Module Series consists of four learning modules targeted toward

professionals working in child welfare settings or a closely related field (education, social work,

criminal justice, immigration law). Each module is divided into a series of short, video lessons.

Modules 1, 3, and 4 are designed to be generalizable to a broad, adult audience.

Methodology

Instructional Product

The Implicit Bias Module Series consists of lessons grouped by category into four,

separate learning modules. Each of the four learning modules is made up of approximately 15

minutes of short, video instruction. The instructional content is intended to raise awareness

regarding implicit bias, while simultaneously supporting racial equity and inclusion. The videos

primarily use voice-over narration accompanied by visual elements such as graphics and

diagrams. On-camera presenters appear in some lessons, providing instructional content as well

as relevant, personal anecdotes.

For this instructional product evaluation, learning modules 1 and 4 were selected for

presentation to learners. Module 1 consists of three short videos and provides an overview of the

phenomena of implicit bias. Module 4 has four training videos and discusses strategies to
 4

identify one’s own implicit biases, ways to resist employing unconscious bias while making

certain decisions, and the organizational implications of implicit bias.

Learners should be English language fluent and have a familiarity with the basic concepts

of socio-cognitive psychology.

At the conclusion of each module, learners are presented with a short, written summary

of key takeaways, displayed on screen in white text against a black background. A “check-in”

(quiz) of 4-5 questions concludes each module--learners can download and print their scored

quizzes for future reference. In addition, learners may print a transcript of the entire presentation.

The instructional product does not provide learning objectives. It does, however, provide

an overview of what will be covered in each module:

Module 1: Understanding Implicit Bias

● Lesson 1: What is Implicit Bias?

● Lesson 2: Implicit Bias in Action

● Lesson 3: Origins of Our Bias

Module 4: Mitigating Unwanted Biases

● Lesson 1: Our Brains are Malleable

● Lesson 2: Identifying Susceptibility to Unwanted Bias

● Lesson 3: Individual Interventions

● Lesson 4: Institutional Interventions

Pilot Learners

For this analysis, a pilot group of two learners was recruited to view the instructional

product modules 1 and 4, and participate in a 10-question, first draft, Pre- and Post-Test. The two

pilot learners had no prior exposure to the term “implicit bias”. The learners were observed in-
 5

person and participated simultaneously, in a quiet office, on separate monitors. The learners

navigated through the lessons with relative ease and enthusiastically participated in the speak-

aloud activities within the learning modules. Concluding instruction, the learners agreed that

module 1 “flowed” logically into module 4. Learner feedback regarding the (first draft) Pre- and

Post-Test questions included the statements “this is too easy” or “I’m confused”, regarding

specific questions. This feedback led to the modification and rewriting of most of the Pre and

Post-Test questions.

Learners

The Implicit Bias Module Series is designed for child welfare professionals but learning

modules 1and 4 are highly generalized and accessible to broader audiences. For the purposes of

this study, ten new learners were “conveniently” recruited to comprise a Test Group. The group

consists of adult learners possessing the computer skills and learning ability to meet the

navigational tasks’ demands and maintain sustained interest during the learning modules’

instruction. The ten learners are adults, ranging in age from 22 to 75-years-old. The occupations

of the learners include college student, educator, counselor, learning specialist, organizational

development professionals, retired professional, and general contractor.

Some of the learners chose to participate remotely, following instructions according to

an email. Others opted to test in person, using the exact email instructions, but making

themselves available for in-person observation. All learners took the Pre-Test, viewed the

instructional product learning modules, participated in activities within the learning modules, and

completed the in-module quiz elements. Following the instruction, the learners completed the

Post-Test and responded verbally to questions about their learning experience. (Appendix A).
 6

Tryout Conditions

Tryout conditions included some learners working at home on their computers. Learners

were advised to find a distraction-free environment and allow one hour to complete all elements,

including tasks and instruction.

In-person tryout conditions included a learner on a computer, in a distraction-free

environment, with a silent observer present.

Tryout Process

The ten learners were each sent an email with information, instructions, and links to the

Pre-Test, the link to The Kirwan Institute Implicit Bias Series instructional modules 1 and 4, the

Post-Test, and were then verbally prompted to share their navigation and learning experiences.

Usability Observation

The six test learners selected for the usability observation were sent an email with

instructions (Appendix B). The test learners were provided verbal instructions during the set-up

of the observation. All usability observations were done in person. The observations were done

in silence unless the test learner had questions. The Usability Observation Evaluation (Appendix

C) was used to take notes. A verbal interview was conducted at the end of the observation using

the questions identified on the Usability Observation Evaluation.

Pre-Test

An electronic Pre-Test was created using the Google Survey application (Appendix D).

The first five questions focus on content from Module 1, and the remaining five questions focus

on information in Module 4. There was one instance in which the test learner had trouble

accessing both Google Surveys. The learner was an in-person tester and was provided a paper

copy of the Pre-Test and Post-Test. The learner's test responses were manually transferred from
 7

the paper quiz sheets to the Google form survey(s) to facilitate data gathering, and the learner’s

email address was used to match the Pre-Test and Post-Test results.

Post-Test

The Post-Test consisted of the same ten questions as the Pre-Test (Appendix E). It was

also created using the Google Survey application. A different background color was used for the

Post-Test survey to provide a visual reference differentiating the two tests. The questions were

asked in the same order as the Pre-Test.

Results

Entry Conditions

The entry conditions are the same for the test learners and the intended learners. All test

learners are proficient or advanced in accessing and completing online surveys and learning. The

observed learners accessed the instructional product in an office setting or on their personal

computers. The instruction and surveys took learners approximately 45 minutes to complete. All

learners were able to complete the learning and testing within the time allotted.

Instruction

The Implicit Bias Module Series is an open course, available through the Kirwan Institute

for the Study of Race and Ethnicity website and located under Training and Resources. It is

intended to be completed by individual users, starting with module 1, and concluding with

module 4. The test learners accessed the course through a direct web link. For this study, the test

learners were asked to complete only module 1 and module 4.

 8

Outcomes

This product does not provide learning objectives. However, as indicated in the

Methodology summary, the product provided the following module descriptions:

Module 1: Understanding Implicit Bias

● Lesson 1: What is Implicit Bias?

● Lesson 2: Implicit Bias in Action

● Lesson 3: Origins of Our Bias

Module 4: Mitigating Unwanted Biases

● Lesson 1: Our Brains are Malleable

● Lesson 2: Identifying Susceptibility to Unwanted Bias

● Lesson 3: Individual Interventions

● Lesson 4: Institutional Interventions

Based on the content of the course modules, the following learner-friendly objectives

were created to assist the analysis team in developing the Pre-Test and Post-Test questions:

1. Define implicit bias

2. Differentiate between implicit bias and explicit bias

3. Identify behaviors associated with implicit bias

4. Identify strategies one could take to mitigate implicit bias

The alternative hypothesis of this study is that test learners will improve their knowledge

of implicit bias and strategies to mitigate bias. The null hypothesis is that the instructional

instrument will not lead to improvement of knowledge on implicit bias or strategies to mitigate

implicit bias. A paired t-test was conducted, and the null hypothesis was rejected. The test
 9

learners demonstrated measurable improvement between Pre-Test and Post-Test responses.

Cohen’s d indicated the effect size was large.

Data Summary

The dependent Pre-Test and Post-Test scores (Table 1, Table 2, and Table 3) were

analyzed using a paired t-test (Table 4). The t Stat at 2.74 was larger than the t Critical one-tail at

1.83, indicating statistical significance. Therefore, the null hypothesis was rejected. Furthermore,

the Post-Test mean of 9.4, minus the Pre-Test mean of 8.4, divided by the pooled standard

deviation of 1.05, results in Cohen’s d = .95, exceeding the .8 minimum criteria for a large effect

size.

In Table 2, the blue line shows the high degree of variability amongst Pre-Test

respondents’ answers. The orange line indicates the more stable, Post-Test accuracy rate of

responses, indicating a minimum of 80 percent correct responses (8:10) for all ten learners.
 10

Table 1:
 11

Table 2:
 12

Table 3
 13

Table 4:
 14

Learning Gains

The Pre-Test and Post-Test are identical in content, containing the same ten questions in

the same order, and were delivered to learners as a Google Form Survey. Care was taken to align

the survey questions with the instructional content of learning modules 1 and 4, by making use of

a complete transcript of the lesson modules for reference. The questions aim to measure the

learning of key terms and concepts, as well as learning outcomes targeted within the instruction.

An analysis of the Pre-Test and Post-Test question responses was conducted (Table 4).

The most significant learning gain is evidenced in Post-Test responses to Question 3 (Table 3),

which asks learners to identify a characteristic of implicit bias. This is a somewhat nuanced

question and requires highly accurate recall of the instructional episode (Module 1). On the Pre-

Test, 5 of 10 learners answered this question correctly and 5 of 10 learners answered incorrectly.

On the Post-Test, the five learners who answered incorrectly each changed their responses. Three

of the five selected a new, correct, Post-Test answer, demonstrating a 60% gain in learning

within the group. Two of the five selected a new, incorrect answer, demonstrating no gain. Post-

Test cumulative results show that 80% of all learners (8 of 10), provided a correct response to

Post-Test Question 3.

Questions 6 and 7 each garnered a 100% correct response rate (10:10) on the Pre-Test.

However, on the Post Test, each of these questions dropped to a 9:10 ratio. Although the sample

of 10 learners is small, a review of these two test questions (which each reflect a Post-Test

learning loss of 10%) is in order. The 100% Pre-Test accuracy rate suggests that learner

confidence may be attributable to prior knowledge. It also suggests that confidence in answering

these questions may have declined because of participating in the instruction. Because both
 15

questions had 100% accuracy in the Pre-Test, they are probably not relevant for evaluating or

measuring learning that occurs because of the instructional product.

Overall, there was a significant gain in correct responses between the Pre- and Post-Tests,

indicating that learning occurred in almost all participants. One participant did score 10:10 on

both the Pre- and Post-Test. This participant is a college student who has no memory of specific,

previous exposure to the content but states she’s “good at taking tests”. Amongst the remaining 9

testers, an aggregate of 12 correct responses was gained following course instruction. There is

also a small degree of “loss”, indicated by an aggregate of 3 correct responses on Pre-Test

questions that were changed to incorrect responses on Post-Test questions. Additionally, there is

a 3:3 ratio of incorrect Pre-Test responses changed to new, incorrect Post-Test responses.

Learner Reactions

During Usability Testing, learners unanimously expressed positive reactions to the

Instructional Product. Learners reported aloud that the modules were, overall, user friendly.

Three obstacles to automaticity were observed in all users: 1) Following the introduction video

for Learning Module 1, the user must locate and press a rectangular, red “Next” button, which is

in the lower right-hand corner of the user’s computer monitor. The button might be more

intuitively located directly under the video box. (On larger screens/monitors, locating the “Next”

button can take several seconds). 2) Navigating out of the Module 1 quiz to return to the

Learning Module Menu requires users to search for an exit device. Users must locate an “x” in

the user tab bar. Perhaps a more intuitive location would be just outside of the center of the

screen, within the user’s peripheral field of vision. In both circumstances, users were able to

independently navigate forward without too much delay or frustration. 3) All observed users
 16

commented aloud that the end of module summary pages--which appear as white font set against

a black background--are difficult to read.

All observed users seemed to enjoy participating in three, separate activities, which

prompt users to speak out loud. In addition, all observed users seemed engaged during the

“check-in” quiz elements of Learning Modules 1 and 4. One user downloaded the questions and

responses stating an intention to print the results.

Recommendations

1. Address navigation issues discovered during the Usability Survey and described in

Learner Reactions (location of “Next button following Module 1 Intro; location of exit,

“x” on quiz; select a more “readable” font/background for module summaries). In

addition, navigation between modules could be reviewed for ease of use.

2. Provide clear learning objectives at the beginning of instructional modules.

3. Use metaphors or analogies very carefully when describing examples of “implicit bias”.

For example, in Module 1, Lesson one, the narrator expresses a “preference” for the color

red over blue, then states that in preferring red, her “evaluation of blue is therefore

negative”. All observed learners and one (self-reporting) learner expressed dissatisfaction

with this portion of the instruction. One observed learner paused, played the video again,

and stated aloud, “liking red does not result in not liking blue--that’s a logical fallacy.”

4. In Module 1, Lesson two, the scientifically documented phenomena of automatically

“filling in” incomplete stimuli that appear within one’s visual field is used as a metaphor

for unconscious bias. One of our observed learners, who specializes in brain science and

memory, stopped the video to say aloud, “‘filling in’ occurs through the selective
 17

activation of neurons in the visual cortex...it has been observed in laboratory animals. It is

automatic--not exactly an unconscious preference or attitude.”

5. In Module 1, Lesson three, an on-camera presenter describes growing awareness of

personally held implicit biases and states specific actions that will be taken by the

presenter to mitigate these biases. By modeling an increase in self-awareness and the

intention to modify existing attitudes and behaviors, this presenter provides an example

for learners of how biases can be identified and how changes might occur. Our observed

learners, unanimously, expressed a very positive reaction to this speaker.

6. Create better navigation between modules. Following each video, learners must scan the

page for what they need to do next. For example, it would have been helpful to have

instructions in the video indicating what the learner should do next or written instructions

under the video.

7. Update the heavy text-based video in module 1 with an on-screen narrator. The on-screen

narration received the most favorable feedback from the test learners.

Summary

The instructional product provided engaging and effective instruction with measurable

learning results in the test group. The instruction received positive feedback from the learners.

The difference between Pre- and Post-instruction scores demonstrates that learning occurred. The

navigation was user-friendly, with minor suggestions for ease of use. The product proved to be

interesting to all test learners. The content was a review for some test learners; however, they

reported feeling more confident with the terminology. Two test learners said they learned

information that was new to them. The Usability Test offered minor suggestions to improve the

navigation such as locating buttons or finding an exit within the viewer’s field of vision or
 18

providing a way to move to the next module without searching the left navigation bar.

Additionally, the readability of the recap slides proved to be difficult for most observed test

learners. The data shows the instruction presented in the Learning Modules 1 and 4 created

learning that was measurable and both statistically significant and practically significant.
 19

References

Kirwan Institute for the Study of Race and Ethnicity. (2021, July 25). About the Kirwan

Institute. https://kirwaninstitute.osu.edu/about
 20

Appendices

Appendix A

Implicit Bias Module Series product: http://kirwaninstitute.osu.edu/implicit-bias-training/

 21

Appendix B

Test-Learner Email Instruction:

Thank you for volunteering to be a tester for my statistics class term paper.

I am evaluating an eLearning module on implicit bias. The purpose of my evaluation is to review

the eLearning module for effectiveness and not your performance on the pre-test and post-test.
Please answer the pre-test questions based on your current knowledge. Then answer the post-test
questions based on information provided from the eLearning module.

Your personal information will not be shared or used in the final report.

Testing Process:
1. Complete the Pre-Test before taking the eLearning course: Pre-Test Survey
2. Take Module 1 and Module 4 only of the eLearning course: Implicit Bias 101
3. Complete the Post-Test after taking the eLearning course: Post-Test Survey

It should take you about 45 minutes to complete all three steps.

I'm happy to answer any questions.

 22

Appendix C

Usability Observation Checklist:

Navigation
Describe how the user navigates the
eLearning module 1 and module 4.

Was the module easy to navigate?

Accessibility
Was the eLearning module accessible to
the user? What, if any, issues were
encountered by the user?

What portions, if any, of the eLearning

module caused confusion or questions from
the user?

Interactivity

Describe the user’s engagement in the

learning material?

Content
Was the user able to demonstrate
learning/knowledge of the content based on
the self-correcting quiz in each module?

Learning Guidance and Support

What comments or questions did the user
share while participating in the testing?

Visual Design
How does the design of the module support
the learning?
 23

Exit Questions

What is your overall impression of the

eLearning module?

What did you like best about the eLearning

module?

What did you like least about the eLearning

module?

Do you have any other final comments or

questions?

Would you describe the learning modules

as "user friendly"?

Is the content interesting and engaging?

 24

Appendix D

Link to Pre-Test Survey: https://forms.gle/zzeTwye2asFTYwTS9

25
26
27
 28

Appendix E

Link to Post-Test Survey: https://forms.gle/m3Cu3cVDZLLxhpES7

29
30