Computers in Human Behavior Reports 3 (2021) 100082

Contents lists available at ScienceDirect

Computers in Human Behavior Reports

journal homepage: www.journals.elsevier.com/computers-in-human-behavior-reports

Development and validation of the Perceived Online Racism Scale short

form (15 items) and very brief (six items)
Brian TaeHyuk Keum, Ph.D. *
Department of Social Welfare, University of California Los Angeles, 337 Charles E. Young Drive, Los Angeles, CA, 90095, USA

A R T I C L E I N F O A B S T R A C T

Keywords: The Perceived Online Racism Scale (PORS) provides important contemporary information in better understanding
Online racism the experiences of racism in today’s digital society. However, the utility of the 30-item PORS may require lengthy
Cyber-racism addition to study surveys. Especially for large survey batteries (e.g., epidemiological studies), researchers may
Cyberbullying
prefer short measures to avoid participant fatigue that can affect the validity of the responses. Thus, the aim of the
Online victimization
Measure development
current study was to optimize the utility of PORS by developing brief versions that still maintain the psychometric
Brief measure properties. With multi-sample archival data (total N ¼ 947), we developed the 15-item three-factor PORS-SF
(short-form) based on factor analytic procedures and the six-item unidimensional PORS-VB (very brief) based on
bifactor modeling and tested whether both measures performed as well as the original PORS. With additional data
collection (N ¼ 454), we further validated the factor structures and expanded the construct validity of the
shortened measures. Across all samples, the PORS-SF and PORS-VB demonstrated good model fit, adequate in-
ternal consistency estimates, and showed construct validity evidence similar to the original PORS. Measurement
invariance across racial/ethnic groups (Black, Asian, Latinx, Multiracial) and gender (women and men) was also
replicated. Overall, results suggest that PORS-SF captures the same three domains of the PORS and PORS-VB
captures the same general factor of the PORS with much shorter sets of items. Implications for research and
practice are discussed.

Racism is an everyday reality resulting in maltreatment, unjust
burden, and discrimination for people of color in the United States at the
individual (e.g., interpersonal racial discrimination), cultural (e.g., White
supremacy and cultural devaluation of people of color), and systemic
levels (e.g., policies and structures that disadvantage people of color
systematically). In today’s digital era, racism persists more visibly,
commonly, and explicitly on the internet (Keum and Miller, 2018a; Tynes
et al., 2018). Mechanisms such as online anonymity and beliefs in “digital
freedom of speech” have allowed individuals to openly share their racist
ideologies without feeling restricted to social norms and accountability
(Keum and Miller, 2018a). Emerging research has demonstrated that
online racism is significantly linked to poorer well-being among racia-
l/ethnic minority adolescents and emerging adults (Keum & Miller,
2017; Tynes et al., 2008; Umana-Taylor et al., 2015).
To advance empirical research on online racism, Keum and Miller
(2017) developed the Perceived Online Racism Scale (PORS). The
30-item PORS is the first and only measure that assesses perceived online
racism experiences among racial/ethnic minority adults. Based on the
measure, Keum and Miller (2017) provided empirical evidence that
online racism can be encountered in three ways: (a) via personal en-
counters of racial cyber-aggression from others on the internet, (b)
vicariously witnessing racial cyber-aggression inflicted against others on
the Internet, and (c) via consumption of online content that exposes
systemic racism and group level practices of racial injustices in society
(e.g., information on various systemic racial inequalities, online media
dehumanizing entire cultures of racial/ethnic minority groups). The
three domains capture the contemporary online racial victimization
processes for racial/ethnic minority adults who are exposed to racist
interactions and online content via a wide array of social media and
digital communication platforms (Keum, 2017).
Despite the importance of assessing online racism as the risk factor of
poorer well-being among racial/ethnic minority adults, the utility of the
30-item PORS may require lengthy addition to the study surveys. Espe-
cially for large survey batteries (e.g., nationally representative assess-
ments), researchers may prefer brief and short measures to avoid
participant fatigue from affecting the validity of the responses (Lee et al.,
2004). Thus, the aim of the current study was to further optimize the
PORS for utility by developing brief versions that still maintain the

* corresponding author.
E-mail address: briankeum@luskin.ucla.edu.

https://doi.org/10.1016/j.chbr.2021.100082
Received 26 August 2020; Received in revised form 1 December 2020; Accepted 29 March 2021
Available online 17 April 2021
2451-9588/© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
B.T. Keum
psychometric properties of the original PORS.
Perceived Online Racism Scale (PORS)
The PORS is a 30-item measure designed to assess people’s experi-
ences of racist online interactions and exposure to racist online content
and information (Keum & Miller, 2017). Participants rate how often they
have experienced each of the 30 items on the Internet in the past six
months on a five-point Likert-type scale ranging from 1 (Never) to 5 (All
the time). The PORS was modeled as a correlated three-factor structure.
The Personal Experience of Racial Cyber-Aggression subscale (PERCA)
contains 14 items (scores ranging from 14 to 70) and represents the direct
online racial aggression that individuals can face in their online in-
teractions with others. A sample item reads, “I have received racist insults
regarding my online profile (e.g., profile pictures, user ID).” The Vicar-
ious Exposure to Racial Cyber-Aggression subscale (VERCA) contains five
items (scores ranging from five to 25) and represents an observation of
racial aggression experienced by other racial/ethnic minority users in
their online interactions. A sample item reads, “I have seen other
racial/minority users being treated like a second-class citizen.” The
Online-Mediated Exposure to Racist Reality subscale (OMERR) contains
11 items (scores ranging from 11 to 55) and characterizes people’s
exposure to online content (e.g., racist incidents happening in another
location or online information illuminating various systemic racial in-
equalities) through which they may realize and witness the apparent
reality of racism in society. A sample item reads, “I have been informed
about a viral/trending racist event happening elsewhere (e.g., in a
different location).” Examination of the bifactor structure suggested the
utility of the three subscales, as unique variances were partitioned be-
tween the general factor sharing variance in all of the items (scores
ranging from 30 to 150) and the three group factors with unique variance
represented by the respective items in each domain.
Initial psychometric properties for the PORS were promising (Keum &
Miller, 2017). Internal consistency estimates ranged from 0.90 to 0.95
across the subscales. For construct validity, the PORS scores were
significantly and positively correlated to an existing measure of offline
racism and racism-related stress for convergent validity evidence and
with psychological distress and unjust views of society for
criterion-related validity evidence. The scores also significantly predicted
psychological distress for predictive validity evidence. The authors also
tested the incremental validity by examining whether the scores of PORS
would predict unique variance in psychological distress over and above
offline racism. The VERCA subscale scores were found to predict unique
variance in psychological distress beyond offline racism. In the original
development study, the correlations across the three subscales ranged
from 0.49 to 0.70. Measurement invariance was also demonstrated across
the four racial/ethnic groups (Black, Asian, Latinx, Multiracial), and
between women and men (Keum & Miller, 2017, 2018b), indicating that
score differences among these groups reflect true results rather than
response bias.
The present study
To facilitate greater utility of the PORS while preserving the rigorous
psychometric properties, the goal of this study was to develop two brief
versions of the original PORS. First, a shorter version (PORS-Short Form;
PORS-SF) that aligns with the content validity and the psychometric
properties of the original correlated three-factor structure with five items
per subscale was developed using factor analytic procedures. The PORS-
SF was developed for researchers who want to use a brief version of PORS
but would also like to capture the breadth of the online racism experi-
ences represented by the three factors. Second, based on the bifactor
model indices of the original PORS and its indication of a unidimensional
representation, we developed a very brief unidimensional version (PORS-
VB). The PORS-VB was developed for researchers who would like to
maximize brevity (e.g., large surveys with participant fatigue and length
2
Computers in Human Behavior Reports 3 (2021) 100082
concerns). For PORS-SF, we hypothesized a three-factor structure that
would produce an acceptable fit to the data, demonstrate adequate in-
ternal consistency, and show similar construct validity as the original
PORS. For PORS-VB, we hypothesized an acceptable unidimensional fit
and demonstration of similar construct validity and internal consistency
as the bifactor-derived general factor of the PORS.
Method
Participants and procedure
All data sets in the current study were collected in compliance with
the institutional review board. There were three samples involved in this
study. Participants for the present study were obtained from a combi-
nation of archival data that was reanalyzed to further optimize and
shorten the original PORS (samples 1 and 2; Total N ¼ 947; Keum &
Miller, 2017) and new data collection (sample 3; N ¼ 454). Please refer to
the original development paper (Keum & Miller, 2017) for procedures
and demographic information of samples 1 and 2.
Participants for sample 3 were collected by advertising recruitment
messages on social media platforms (e.g., Facebook, Reddit). Participants
were invited to participate in an online survey consisting of study vari-
able measures and demographic items hosted by Qualtrics. The inclusion
criteria for the study were: (1) 18 years old or older, (2) self-identify as a
racial/ethnic minority, and (3) live in the U.S. Participants had a chance
to enter a raffle to win 1 of 2 $50 Amazon gift cards.
There was a total of 457 participants in sample 3. The average age of
the participants was 33.88 (SD ¼ 10.81) and ranged from 18 to 66. About
57% identified as woman, 41% as man, and 2% transgender. Approxi-
mately 40% identified as Black/African American, 23% Asian/Asian
American, 20% Hispanic/Latinx American, 9% Multiracial/Multiethnic,
5% Native American, 2% Middle Eastern, and 1% Native Hawaiian/Pa-
cific Islander. About 84% identified as heterosexual, 9% bisexual, 2%
lesbian, 2% gay, 2% asexual, and 1% queer.
Measures
Mental Health Inventory-5. Participants’ level of psychological
distress was measured using the Mental Health Inventory-5 (MHI-5; Veit
& Ware, 1983). The MHI-5 contains five items with higher scores indi-
cating higher levels of psychological well-being and lower scores indi-
cating higher levels of psychological distress. For the current study, we
reverse scored the items so that higher scores indicate higher levels of
psychological distress. Participants report on the frequency of the feel-
ings related to mental health over the last month (e.g., “Have you felt
downhearted and blue?”). Responses are rated on a six-point scale
ranging from 1 (all of the time) to 6 (none of the time). The responses are
summed and range from 5 to 30. MHI-5 has been linked with stressful life
events and decreased social support and life satisfaction. Reliability co-
efficients for racially diverse populations have ranged upwards of .84
(Fischer & Bolton Holz, 2010). The Cronbach’s alpha in the current study
was 0.85.
Perceived stress. We used the 10-item Perceived Stress Scale (PSS-
10) to assess the extent to which situations in life are perceived as
stressful (Cohen et al., 1983). The PSS-10 was designed to assess how
unpredictable, uncontrollable, and overloading the life situations are for
the participants over the last month. Participants rate their exposure to
stressful situations on a five-point scale ranging from 0 (never) to 4 (very
often). A sample item reads, “How often have you been angered because
of things that were outside of your control?” The responses are summed
(ranging from 0 to 40), with higher total scores indicating greater
perceived stress. Internal reliabilities for the PSS-10 ranged from 0.78 to
0.91 in a racially/ethnically diverse nationally representative sample
(Cohen & Janicki-Deverts, 2012). Perceived stress has been significantly
and positively correlated with negative affect (e.g., anxiety), depression,
and coping behaviors. The Cronbach’s alpha for the current study was
B.T. Keum
0.86.
Perceived racism (offline). The Perceived Ethnic Discrimination
Questionnaire- Community Version Brief (PEDQ-CVB; Brondolo et al.,
2005) is a 17-item measure of general perceived racial discrimination
with the following subscales: exclusion/rejection (4 items; e.g., “Have
others ignored you or not paid attention to you?”), stigmatiza-
tion/disvaluation (4 items; e.g., “Have people not trusted you?”),
work/school Discrimination (4 items; e.g., “Have you been treated un-
fairly by coworkers or classmates?”), treatment/aggression (4 items; e.g.,
“Have others actually hurt you or tried to hurt you?”), and Police (1
item). PEDQ-CVB was selected for our study as it was designed as a
general measure that can be used with adults in the community across
racial/ethnic groups, and educational backgrounds. The full scale was
used for our study. Participants rate their perceived exposure to
discrimination items on a five-point scale ranging from 1 (almost never) to
5 (almost always). Responses are summed and averaged. Higher scores
represent higher levels of perceived racial/ethnic discrimination. Addi-
tional instruction was introduced stating “Please think about your offline
experiences (not online) of racism.” The PEDQ-CVB has been shown to
converge with other racism measures and linked to negative mental
health (Brondolo et al., 2005). Full-scale reliability estimates have
ranged upwards of .90 in multiple racial/ethnic groups (Brondolo et al.,
2005; Keum et al., 2018). The Cronbach’s alpha for the current study was
0.94.
Belief in an unjust world. The 5-item Unjust Views Scale (UVS;
Lench & Chang, 2007) assesses both personal (“The awful things that
happen to me are unfair”) and general beliefs (“People who do evil things
get away with it”) in an unfair world. Participant responses can range
from 1 (strongly disagree) to 5 (strongly agree). Item scores are summed
and averaged. Higher scores indicate a greater belief in an unfair world.
The measure demonstrated discriminant validity with belief in a just
world and positively correlated with the use of denial, anger, pessimism,
and other disengagement coping strategies and the reliability estimates
with racially diverse samples have ranged from 0.72 to 0.80 (Lench &
Chang, 2007; Liang & Borders, 2012). The Cronbach’s alpha for the
current study was 0.65.
Alcohol Use. The 10-item Alcohol Use Disorders Identification Test
(AUDIT; Saunders et al., 1993) was used to assess participants’ risky or
harmful alcohol consumption as well as alcohol dependence and abuse.
The AUDIT items represent alcohol consumption (items 1–3), drinking
behavior/dependence (items 4–7), and alcohol-related problems or
consequences (items 8–10), and is scored by summing up all of the items.
The first eight items are scored on a 5-point Liker scale ranging from 0 to
4, and the last two are scored on 3-point Likert-scale with values of 0, 2,
and 4. The total scores range from 0 to 40, with higher scores indicating
more severe alcohol problems. The AUDIT cutoff score for harmful use of
alcohol is generally recommended as 8, although for women a lower
cutoff score of 5 or 6 is suggested (Reinert & Allen, 2007). The mean
AUDIT score in our sample was 6.66 (SD ¼ 7.96; Range 0–34), with
nearly 30% (n ¼ 121) endorsing harmful use of alcohol (score  8). The
total score Cronbach’s alpha for the current study was 0.93.
Kessler-6. We used the Kessler-6 (K6) to test for general psycholog-
ical distress and has been previously used to identify those who are
experiencing severe impairment in their lives (Kessler et al., 2003). The
measure includes six questions and asks the frequency of psychological
distress symptoms such as depression and anxiety in the past 30 days
from 0 (none of the time) to 5 (all of the time). Items included how often
they felt hopeless, so depressed that nothing could cheer them up,
worthless, nervous, restless, or fidgety, and that everything was an effort.
Higher scores indicate higher levels of psychological distress symptoms
and scores could range from 6 to 30. The reliability estimate was very
high for the sample (α ¼ 0.91) and the scale has been validated with
racial/ethnic minorities (Byrd, 2012). The Cronbach’s alpha for the
current study was 0.92.
Loneliness. The three-item Loneliness Scale (Hughes et al., 2004)
was used to assess participants’ perceived loneliness. The three items
3
Computers in Human Behavior Reports 3 (2021) 100082
assess how often participants lack companionship, feel left out, and feel
isolated, measured on a 1 to 3 scale (where 1 ¼ hardly ever, 2 ¼ some of
the time, and 3 ¼ often). The average score across the three items is
calculated, with higher scores indicating greater loneliness. Hughes et al.
(2004) reported that the scale had acceptable internal reliability (α ¼
0.72) in older adult samples and was highly correlated with scores on the
original R-UCLA scale (r ¼ 0.82, p < .001) and with scores on
loneliness-related items in the Center for Epidemiologic Stud-
ies–Depression Scale demonstrating convergent validity. The Cronbach’s
alpha for the current study was 0.86.
Social Media-related Stress. In order to avoid creating lengthy on-
line surveys, a single item (“Social media brings additional stress to my
life.”) was used to assess participants’ perceived stress due to social
media. Participants responded to the item on a six-point Likert-type scale
(1-Strongly Disagree to 6-Strongly Agree). Although single-item measures
of well-being lack rigorous psychometric properties, they are often used
as part of large nationally representative studies with good predictability
(Hayes et al., 2015).
Data analysis
Please refer to the original development paper (Keum & Miller, 2017)
for data inspection and missing data information for samples 1 and 2. No
missing data were present in sample 3. Mardia’s multivariate test sug-
gested non-normal multivariate skew ¼ 36.96, p < .001, and kurtosis ¼
454.85, p < .001, for sample 3. Thus, we employed a maximum likeli-
hood estimation with standard errors and chi-square test statistics that
are robust to non-normality. All analyses were conducted in Mplus 7.11.
The original development sample (N ¼ 947; Keum & Miller, 2017) was
randomly assigned into two samples for Exploratory Factor Analysis
(EFA; sample 1; n ¼ 476) and Confirmatory Factor Analysis (CFA; sample
2; n ¼ 471). Initial construct validity and measurement invariance
(racial/ethnic and gender groups) of the revised measures were exam-
ined by combining samples 1 and 2. Additional psychometric properties
(construct validity and factor structures) were further validated using
new data (sample 3; N ¼ 454).
PORS-SF development
To develop the PORS-SF, we conducted EFAs on sample 1 with the
original 30 items. Our goal was to reduce the number of items for the
PERCA and OMERR factors to 5 items each, to match the 5 items of the
VERCA factor. Removal of items was conducted via sequential EFAs to
ensure that the model fit was consistently robust throughout the process.
The shortened PORS- SF was then cross-validated (CFA) with sample 2.
PORS-VB development
First, to identify items for the PORS-Very Brief (PORS-VB), we
calculated and used the individual explained common variance (IECV)
values for the PORS derived from a bifactor model factor loadings in the
original development paper (Keum & Miller, 2017). Based on the rec-
ommendations of Stucky and Edelen (2014) and procedures in McDer-
mott et al. (2019), we retained items in which the majority of their
variances were explained via variations on the general PORS factor (i.e.,
close to 70% or greater) and had strong unidimensional factor loadings
(>0.50). We then conducted an EFA with the retained items by fitting a
unidimensional model using sample 1. Subsequently, we cross-validated
(CFA) this unidimensional model with sample 2.
Construct validity, measurement invariance, and further validation
With the combined sample (1 and 2), we examined the construct
validity of PORS-SF and PORS-VB using the same nomological network
used to test the original PORS. In line with the original PORS, we tested
measurement invariance across racial/ethnic and gender groups. With
sample 3, we further validated the factor structures of both measures and
the construct validity based on theory- and empirically-driven relation-
ships with additional criterion variables.
B.T. Keum Computers in Human Behavior Reports 3 (2021) 100082

Model fit assessment Table 1

Across all models, model fit was assessed with the scaled χ 2 test sta- Unidimensional factor loadings, IECV values, and item descriptives of the 30-
tistic (a nonsignificant value indicates a good fit) and the following fit item perceived online racism scale (PORS).
indices (Hu & Bentler, 1999; Kline, 2016): (a) comparative fit index (CFI; Original PORS Items Unidimensional IECV Item Item
> 0.95 for good fit; 0.92 to 0.94 for adequate fit), (b) the standardized Factor Mean Median
root mean square residual (SRMR; close to < 0.08 for acceptable fit), (c) Loadings

the root mean square error of approximation (RMSEA; close to < 0.08 for 1. Received racist insults .75 .49 1.53 1
acceptable fit), and Tucker-Lewis Index (TLI; close to 0.95 or greater for a regarding my online profile
(e.g., profile pictures, user
good fit). All nested models were examined using a scaled χ 2 difference
ID).
test, with a nonsignificant χ 2 indicating the model with more degrees of 2. Been kicked out of an online .63 .30 1.37 1
freedom (i.e., more parsimonious model) was equivalent to a model with social group because I talked
fewer degrees of freedom (i.e., less parsimonious model). Non-nested about race/ethnicity.
3. Been intentionally invited to .52 .18 1.31 1
model comparisons were examined with the Bayesian information cri-
join a racist online social/
terion (BIC), with lower values indicating better model fit and higher hate group.
values of more than 10 units suggesting lack of empirical support for the 4. Received replies/posts .71 .42 1.47 1
goodness of fit (Kline, 2016). suggesting that I should avoid
Invariance testing was conducted via comparison of a series of models connecting online with
friends from my own racial/
with increasing constraints: baseline configural model (no constraints), ethnic group.
metric model (factor loadings constrained), and scalar model (factor 5. Received racist insults about .68 .34 1.47 1
loadings and item intercepts constrained). Model fit was assessed by the how I write online.
same fit index used in our CFA (Hu & Bentler, 1999). Evaluation of the 6. Been threatened of being .68 .32 1.41 1
harmed or killed due to my
invariance was conducted by S-B chi-square tests and assessment of
race/ethnicity.
changes in the fit index. A change in CFI (ΔCFI) less than 0.01, change in 7. Received replies/posts .66 .59 1.65 1
RMSEA (ΔRMSEA) less than 0.015, and change in SRMR (ΔSRMR) less hinting that my success is
than 0.03 suggests no significant decrease in model fit and supports surprising for a person of my
measurement invariance (Chen, 2007). Given that chi-square tests are race/ethnicity.
8. Received a message with a .67 .45 1.53 1
known to be sensitive to sample size and even a small difference may be racist acronym such as FOB
found to be significant with increasing sample sizes (Cheung & Rensvold, (Fresh Off the Boat) or PIBBY
2002), we weighted our invariance assessment more with fit index (Put In Black’s BackYard).
changes where appropriate. 9. Been harassed by someone .77 .59 1.68 1
(e.g., troll) who started a
racist argument about me for
Results no reason.
10. Received a racist meme .72 .59 1.63 1
Development of PORS-SF and PORS-VB (Sample 1) (e.g., racist catchphrases,
captioned photos, #hashtags
etc.).
PORS-SF 11. Been tagged in (or shared) .75 .61 1.65 1
To shorten the 30-item three-factor PORS (Table 1) into the 15-item racist content (e.g., websites,
three-factor PORS- SF (5 items per subscale), we conducted EFAs based photos, videos, posts)
on best practice guidelines with sample 1 (DeVellis, 2016). EFA was insulting my race/ethnicity.
12. Received posts with racist .72 .69 1.75 1
conducted using principal axis factoring extraction and oblique promax
comments.
rotation in Mplus 7.11 consistent with the original development and as 13. Received replies/posts .67 .54 1.58 1
we expected the three factors to correlate with each other as part of the hinting that what I share
online racism experience. We sequentially removed items from the online cannot be trusted due
OMERR and PERCA factors, examining the fit statistics and loadings to my race/ethnicity.
14. Been unfriended/lost online .74 .77 1.6 1
(ensuring no major changes in loadings after each removal) after each ties because I disagreed with
round of item removal on a case-by-case basis guided by the interpret- racist posts.
ability of the domains (DeVellis, 2016). Based on the literature (DeVellis, Factor 2: Online-Mediated
2016; Kline, 2016), our factor analytic reduction process aimed to retain Exposure to Racist Reality
15. Been informed about a .52 .45 2.77 3
5 items for each factor with the following criteria: (a) highly endorsed by
viral/trending racist event
participants (higher mean and median values), (b) highest loadings, (c) happening elsewhere (e.g., in
lowest cross-loadings, (d) contribute most to the internal consistency of a different location).
the scale scores, (e) not correlated highly with other items, and (d) 16. Been informed about .534 .52 2.45 2
represent distinct content domains to ensure breadth of online racism unfairness in healthcare for
racial/ethnic minorities (e.g.,
experiences. Of note, we prioritized items that are highly endorsed by the
biased quality of treatment,
participants and represented distinct content domains given that most insurance issues).
items had robust factor loadings. Thus, even if some of the items were 17. Seen online videos (e.g., .64 .81 2.54 3
stronger candidates to be retained statistically, we decided to remove YouTube) that portray my
racial/ethnic group
them if they were highly correlated with other items, overlapped with the
negatively.
content of other items, and were not endorsed as high by the participants. 18. Encountered online .55 .62 2.76 3
Inter-item correlations also guided the removal of items that correlated resources (e.g., Urban
significantly high (>0.60) with items that were retained as they likely did Dictionary) promoting
not contribute unique variance to the factor. negative racial/ethnic
stereotypes as if they are true.
For the PERCA factor, we retained items 1, 9, 10, 11, and 12. Items 2,
19. Been informed about .49 .42 2.95 3
3, 4, 5, 6, 8, and 13 were removed as their item means indicated lower unfairness in financial gains
endorsement by participants. Items 14 and 7 were removed as they had (continued on next page)

4
B.T. Keum Computers in Human Behavior Reports 3 (2021) 100082

Table 1 (continued ) Table 2

Original PORS Items Unidimensional IECV Item Item
Item factor loadings (Pattern matrix coefficients) for the perceived online.
Factor Mean Median Racism scale short form and very brief.
Loadings PORS-SF Items EFA CFA
for racial/ethnic minorities PERCA VERCA OMERR PERCA VERCA OMERR
(e.g., earning less money than
Whites for doing the same PORS12 .66 .16 -.04 .82
work, unfair housing and PORS11 .79 -.08 .09 .80
loan opportunities). PORS10 .80 -.03 .01 .75
20. Been informed about .53 .52 2.81 3 PORS9 .76 .05 -.01 .76
unfairness in education for PORS1 .83 .01 -.02 .80
racial/ethnic minorities (e.g., PORS30 .09 .60 .11 .82
higher suspension rates for PORS28 .02 .65 .10 .80
racial/ethnic minority PORS29 .19 .65 .01 .75
students). PORS27 .00 .85 .00 .76
21. Been informed about a .59 .77 2.68 3 PORS26 -.08 .78 .15 .80
viral/trending racist event PORS19 -.06 .15 .59 .82
that I was not aware of. PORS24 -.07 -.01 .77 .80
22. Seen online news articles .52 .59 2.48 2 PORS15 -.01 -.03 .77 .75
that describe my racial/ PORS18 .08 .09 .58 .76
ethnic group negatively. PORS17 .18 .01 .63 .80
23. Seen photos that portray my .58 .75 2.62 3 PORS-VB Items EFA CFA
racial/ethnic group
negatively. PORS26 .64 .55
24. Encountered a viral/ .53 .60 2.95 3 PORS25 .64 .71
trending online racist content PORS23 .77 .75
(e.g., many likes, stars). PORS21 .59 .60
25. Encountered online hate .63 .76 2.44 2 PORS17 .78 .78
groups/communities against PORS12 .50 .47
non-White racial/ethnic
Note. PORS ¼ Perceived Online Racism Scale; SF ¼ Short Form; VB ¼ Very Brief;
groups.
Factor 3: Vicarious Exposure to
EFA ¼ Exploratory Factor Analysis; CFA ¼ Confirmatory Factor Analysis; PERCA
Racial Cyber-Aggression ¼ Personal Experience of Racial Cyber-Aggression; VERCA ¼ Vicarious Exposure
26. Seen other racial/minority .63 .76 2.64 3 to Racial Cyber-Aggression; OMERR ¼ Online-Mediated Exposure to Racist
users receive racist Reality.
comments.
27. Seen other racial/minority .66 .69 2.44 2
users being treated like a
PORS-VB
second-class citizen. Although Keum and Miller (2017) concluded that a bifactor repre-
28. Seen other racial/minority .60 .64 2.45 2 sentation of the original PORS is warranted given the magnitude of the
users being treated like a significant factor loadings for the general factor and the group factor
criminal.
loadings were significant and comparatively high even when the general
29. Seen other racial/minority .70 .80 2.11 2
users receive racist insults factor variance is parceled out of the group factors, we calculated the
regarding their online profile model-based indices for the bifactor model of the original PORS to ensure
(e.g., profile pictures, user that a strong general factor is present and merits consideration of a
ID). unidimensional representation based on which the items for PORS-VB
30. Seen other racial/minority .59 .59 2.22 2
users being threatened to be
can be extracted. The OmegaH of the general factor was high at .75,
harmed or killed. suggesting that the total scale scores can be generally attributed to in-
dividual differences on the general factor (Reise et al., 2012). The sys-
Note. IECV ¼ Individual Explained Common Variance; Numbers in bold have met
tematic variance of the subscale scores after partitioning out the
the criteria of (a) being close to or greater than 0.70 IECV value suggesting that
around 70% of the variance is explained by the general factor, and (b) strong
variability due to the general factor was low at 0.48, 0.35, 0.27 for
unidimensional factor loadings (close to or > 0.60). PERCA, OMERR, and VERCA, respectively. Relative Omega suggested
that 77% of the reliable variance in the multidimensional composite was
due to the general factor, compared to the 51% (PERCA), 35% (OMERR),
the lowest factor loadings and were highly correlated with several other
and 30% (VERCA) reliable subscale variance independent of the general
retained items. For the OMERR factor, we retained items 15, 17, 18, 19,
factor (Reise et al., 2012). The Percent of Uncontaminated Correlations
and 24. Items 16, 22, and 25 were removed due to low endorsement. Item
(PUC) was 0.64, and Explained Common Variance (ECV) of the general
23 was removed as it was highly correlated with item 17 and had lower
factor was 0.58. According to Reise et al. (2012), PUC values lower than
factor loading. Item 20 was removed as it was highly correlated with item
0.80, ECV values close to 0.60 or greater, and OmegaH of the general
19 and had lower factor loading. Item 21 was removed as it was highly
factor greater than 0.70 suggest that the presence of multidimensionality
correlated with item 15 and had lower factor loading. The process
is not strong enough to disqualify a primarily unidimensional represen-
resulted in five items for OMERR and five items for PERCA that were
tation of the measure. Hence, taken together, the model-based estimates
retained robustly and consistently throughout the process and captured
suggested a strong general factor of the PORS that can be interpreted as
distinct online racism experiences for each domain (Table 2). The fit of
unidimensional (total scale score) with an expectation of minimal bias in
the resulting structure was good, RMSEA ¼ 0.061 [0.051, 0.072], SRMR
factor loadings (Rodriguez et al., 2016). This suggested the possible
¼ 0.024. The internal consistency estimates were 0.89, 0.81, and 0.89,
extraction of a small set of representative items measuring the same
factor determinacies were 0.95, 0.95, 0.92, and variances accounted for
PORS general factor as in the original measure (Stucky & Edelen, 2014).
were 45%, 10%, and 5%, for the PERCA, OMERR, and VERCA factors
Table 1 lists the IECV values of the original 30 item PORS. Eight items
respectively. The factors were correlated moderately high with each
(in bold) met our criteria of (a) being close to or greater than 0.70 IECV
other: rPERCA-OMERR ¼ 0.455, p < .001; rPERCA-VERCA ¼ 0.562, p < .001;
value suggesting that around 70% of the variance is explained by the
rOMERR-VERCA ¼ 0.637, p < .001.
general factor, and (b) strong unidimensional factor loadings (close to or
> 0.60). With these eight items, we conducted single factor EFAs using

5
B.T. Keum Computers in Human Behavior Reports 3 (2021) 100082

sample 1. Initial fit of the unidimensional model was less than adequate,
CFI ¼ 0.845, TL ¼ 0.801, RMSEA ¼ 0.126 [0.113, 0.139], SRMR ¼ 0.062.
We proceeded to assess the items for removal based on item correlations,
factor loadings, and IECV values, while striving to exclude items that
overlap in content domains. Modification indices suggested item 29
yielded very high item correlations with item 26, as well as with items
22, and 12. Although item 29 had a slightly higher IECV value than item
26, we removed item 29 as it had lower EFA factor loading than item 16,
correlated highly with two other items, was endorsed lower than item 26,
and since item 26 broadly covered the VERCA experience represented by
item 29. We also removed item 14 as it correlated very highly with item
12. We elected to retain item 12 as it was the most endorsed item in the
PERCA domain and covered the general experience of receiving racist
comments. This process resulted in a six-item PORS-VB (items 26, 25, 23,
21, 17, and 12; Table 2) with a good fit to the data, CFI ¼ 0.963, TLI ¼
0.939, RMSEA ¼ 0.078 [0.052, 0.106], SRMR ¼ 0.032, and significant
factor loadings ranging from 0.50 to 78. Internal consistency estimate
was adequate at 0.85, factor determinacy was high at 0.92, and the
PORS-VB accounted for 45% of the variance.
Cross-validation of the PORS-SF and PORS-VB (Sample 2)
With sample 2, we conducted CFAs to cross-validate the oblique
three-factor structure of the PORS-SF and the unidimensional PORS-VB.
For PORS-SF, we also examined the fit of the single factor and bifactor
models in line with the comparisons in the original development.
Descriptives and Cronbach’s alphas for both scales are listed in
Table 4.
PORS-SF
As seen in Table 3, the oblique three-factor and bifactor models of
PORS-SF both had a good fit to the data with very slight differences in the
fit indices. All items loaded significantly (p < .001) on the hypothesized
latent factors. As expected, the one-factor model had a poor fit to the
data. The Bayesian Information Criteria (BIC) values suggested that there
were no significant differences between the two models and in fact, the
three-factor model (18077.26) had a lower BIC than the bifactor model
(18078.73). Given that the aim was to develop PORS-SF to facilitate
utility, we retained and continued our psychometric testing with the
three-factor model given its simpler utility over bifactor modeled mea-
sures in practice. We also examined the fit of the second-order model
with a second-order factor (total score) representing the three first-order
factors (Figure 1; PORS-SF). We expected the fit of the second-order
model to be similar to the first-order model given that the second-
order model with three first-order factors is just-identified. In other
words, with three indicators of the higher-order factor, there are just as
many estimates being made as there are unique observed values available
(number of free parameters exactly equals the number of known values;
Kline, 2016). As expected, the fit was replicated for the second-order
model and suggested that the second-order factor represented the util-
ity of a total scale score.
PORS-VB
For PORS-VB, the single factor model had a good fit to the data, CFI ¼
0.965, TLI ¼ 0.935 SRMR ¼ 0.029, RMSEA ¼ 0.079 [0.052, 0.109]. All
items loaded significantly (p < .001) on the hypothesized latent factors
(Figure 1).

Table 3
Goodness-of-Fit Indicators for Structural Equation Modeling Analyses.
Models/Samples df χ2 RMSEA 90% CI CFI TLI SRMR BIC

PORS-B
Three-factor (sample 2) 87 194.342** .052 [.043, .062] .967 .960 .038 18077.26
Bifactor (sample 2) 75 175.343** .053 [.043, .064] .970 .958 .032 18078.73
One-factor (sample 2) 90 976.986** .145 [.137, .153] .733 .689 .099 19020.02
Three-factor (sample 3) 87 263.069** .067 [.058, .076] .944 .932 .062 17695.20
Configural (R/E) 348 647.950** .062 [.055, .070] .958 .949 .047 34602.04
Metric (R/E) 384 714.900** .062 [.055, .069] .954 .949 .058 34538.92
Scalar (R/E) 420 795.240** .063 [.057, .070] .948 .948 .058 34489.19
Configural (Gen) 174 458.209** .059 [.053, .066] .963 .956 .036 35943.46
Metric (Gen) 186 468.545** .057 [.051, .064] .963 .959 .038 35909.91
Scalar (Gen) 198 503.972** .058 [.051, .064] .960 .958 .039 35901.45
PORS-VB
One-factor (sample 2) 8 31.796** .079 [.052, .109] .965 .935 .029 8004.06
One-factor (sample 3) 8 41.639** .086 [.059, .116] .962 .928 .036 7530.33
Configural (R/E) 20 53.223** .087 [.059, .115] .971 .943 .036 12704.95
Metric (R/E) 32 65.356** .069 [.044, .092] .971 .964 .050 12673.73
Scalar (R/E) 44 85.752** .065 [.044, .086] .964 .967 .054 12650.77
Configural (Gen) 14 64.163** .088 [.067, .110] .970 .936 .031 15840.40
Metric (Gen) 19 70.258** .076 [.058, .096] .970 .952 .039 15828.21
Scalar (Gen) 24 90.635** .077 [.061, .095] .961 .951 .042 15830.30

MI Model Comparison Δχ 2 (df) p ΔCFI ΔRMSEA ΔSRMR

PORS-SF
Configural vs. Metric (R/E) 66.95(36)* .001 -.004 0 .015
Metric vs. Scalar (R/E) 80.34(36)** <.001 -.006 .001 0
Scalar vs. Configural (R/E) 147.29(72)** <.001 .010 .001 .011
Configural vs. Metric (Gen) 10.34(12) .587 0 .002 .002
Metric vs. Scalar (Gen) 35.43(12)* <.001 -.003 .001 .001
Scalar vs. Configural (Gen) 45.76(24)** .005 -.003 -.001 .003
PORS-VB
Configural vs. Metric (R/E) 12.13(12) .435 0 -.018 .014
Metric vs. Scalar (R/E) 20.40(12) .114 -.007 -.004 .004
Scalar vs. Configural (R/E) 32.53(24) .060 -.007 -.022 .018
Configural vs. Metric (Gen) 6.10(5) .297 0 -.012 .008
Metric vs. Scalar (Gen) 20.38(5)* .003 -.009 .001 .003
Scalar vs. Configural (Gen) 26.47(10)* .001 -.009 -.011 .011

Note. RMSEA ¼ root-mean-square error of approximation; CI ¼ confidence interval for RMSEA; CFI ¼ comparative fit index; SRMR ¼ standardized root-mean-square
residual; BIC ¼ Bayesian information criterion; MI ¼ measurement invariance; R/E ¼ Racial/Ethnic Groups; Gen ¼ Gender Groups.
*p < .05. **p < .01.

6
B.T. Keum Computers in Human Behavior Reports 3 (2021) 100082

Table 4
Factor means, standard deviations, and Cronbach’s alphas.
PORS-SF PORS-VB

PERCA VERCA OMERR Total

Samples M SD α M SD α M SD α M SD α M SD α
Sample 1 8.22 4.21 .89 11.95 5.26 .90 13.78 12.05 .86 33.94 12.05 .92 14.43 5.05 .82
Sample 2 8.27 4.17 .89 11.78 4.93 .88 14.17 4.74 .83 34.22 11.60 .91 14.29 4.95 .82
Sample 3 10.29 4.96 .91 13.31 4.97 .89 13.16 4.64 .83 36.76 12.79 .93 15.56 5.47 .87
Black 0 1 .89 0 1 .89 0 1 .86 0 1 .91 0 1 .79
Asian -.40** .66 .88 -.47** .85 .88 -.45** .83 .82 -.38** .57 .90 -.63** .79 .80
Latinx -.24* .71 .85 -.32* .88 .88 -.27* .95 .85 -.24** .63 .92 -.35** .90 .80
Multiracial -.19 .85 .90 -.20 .90 .91 -.20 .84 .82 -.17* .66 .93 -.28* .82 .81
Women 0 1 .89 0 1 .89 0 1 .84 0 1 .92 0 1 .81
Men .10 .82 .89 -.26** .85 .89 -.34** .89 .84 -.14* .66 .93 -.26** .88 .83

Note. PERCA ¼ Personal Experience of Racial Cyber-Aggression; VERCA ¼ Vicarious Exposure to Racial Cyber-Aggression; OMERR ¼ Online-Mediated Exposure to
Racist Reality; PORS ¼ Perceived Online Racism Scale; SF ¼ Short Form; VB ¼ Very Brief; SD ¼ standard deviation; Italicized values indicate latent means and standard
deviations; Sample 1 (N ¼ 476); Sample 2 (N ¼ 471); Sample 3 (N ¼ 454); Black (N ¼ 306); Asian (N ¼ 310); Latinx (N ¼ 163); Multiracial (N ¼ 109); Women (N ¼ 556);
Men (N ¼ 372). **p < .01, *p < .05

Figure 1. Factor structure of the Perceived Online Racism Scale-Short Form and the Perceived Online Racism Scale-Very Brief.Note. PERCA ¼ personal experience of racial
cyber-aggression; VERCA ¼ vicarious exposure to racial cyber-aggression; OMERR ¼ online-mediated exposure to racist reality.

Construct validity (Samples 1 and 2 combined)
With the combined sample (samples 1 and 2), we aimed to test
whether the construct validity of the original PORS can be replicated for
PORS-SF and PORS-VB. For convergent validity, we hypothesized that
the three factors of PORS-SF and the unidimensional PORS-VB would be
positively correlated with scores from an offline perceived racism mea-
sure (PEDQ-CVB) as they together assess racism that can be experienced
concurrently in online and offline settings (Lozada et al., 2020). For
criterion-related validity, we hypothesized positive correlations with
psychological distress (MHI-5) and perceived stress (PSS) given the
negative impact of racism on mental health (Pascoe & Smart Richman,
2009), and positive correlations with unjust views of society (UVS) given
that perceived racism has been linked to beliefs in an unjust world (Liang
& Borders, 2012). For incremental validity, we examined whether the
revised measures significantly predicted unique variance in psychologi-
cal distress (MHI-5, PSS) after accounting for offline perceived racism
(PEDQ-CVB).
PORS-SF
The measurement model with all of the variables and PORS-SF was
adequate, CFI ¼ 0.905, RMSEA ¼ 0.044 [0.042, 0.045], and SRMR
0.057. Convergent validity evidence was demonstrated via significantly
positive correlations between PEDQ-CVB and PORS-SF Total (r ¼ 0.75, p
< .001), PERCA-Brief (r ¼ 0.70, p < .001), VERCA-Brief (r ¼ 0.58, p <
.001), and OMERR-Brief (r ¼ 0.48, p < .001). As with the original PORS,
correlations suggested greater distinction VERCA and OMERR subscales
compared to perceived offline racism.
Criterion-related validity evidence was demonstrated via significantly
positive correlations. MHI-5 was positively correlated with PORS-SF
Total (r ¼ 0.33, p < .001), PERCA-Brief (r ¼ 0.27, p < .001), VERCA-
Brief (r ¼ 0.31, p < .001), and OMERR-Brief (r ¼ 0.24, p < .001). PSS
was positively correlated with PORS-SF Total (r ¼ 0.36, p < .001),
PERCA-Brief (r ¼ 0.25, p < .001), VERCA-Brief (r ¼ 0.34, p < .001), and
OMERR-Brief (r ¼ 0.30, p < .001). UVS was positively correlated with
PORS-SF Total (r ¼ 0.37, p < .001), PERCA-Brief (r ¼ 0.33, p < .001),
VERCA-Brief (r ¼ 0.35, p < .001), and OMERR-Brief (r ¼ 0.24, p < .001).
For incremental validity, as with the original PORS, only VERCA-Brief
predicted significant unique variance in MHI-5 beyond PEDQ-CVB, (b ¼
0.09, SE ¼ 0.05, p ¼ .048, R2 ¼ 0.04). However, after accounting for
PEDQ-CVB, PSS was significantly predicted by OMERR-Brief (b ¼ 0.14,
SE ¼ 0.05, p ¼ .002, R2 ¼ 0.05), and VERCA-Brief (b ¼ 0.17, SE ¼ 0.05, p
< .001, R2 ¼ 0.06).

7
B.T. Keum
PORS-VB
The measurement model with all of the variables and PORS-VB was
adequate, CFI ¼ 0.900, RMSEA ¼ 0.048 [0.046, 0.050], and SRMR
0.061. PEDQ-CVB was significantly and positively correlated with PORS-
VB (r ¼ 0.59, p < .001). PORS-VB was significantly and positively
correlated with MHI-5 (r ¼ 0.26, p < .001), UVS (r ¼ 0.31, p < .001),
and PSS (r ¼ 0.33, p < .001). For incremental validity, PORS-VB
significantly predicted PSS beyond PEDQ-CVB, (b ¼ 0.22, SE ¼ 0.05, p
< .001, R2 ¼ 0.03).
Measurement invariance
Consistent with the original PORS, we conducted multigroup CFAs to
evaluate whether the revised measures also operated in an equivalent
manner across the major racial/ethnic groups (Black, Asian, Latinx,
Multiracial) and across gender (women and men) in the combined
sample (samples 1 and 2). Fit statistics for invariance tests are displayed
in Table 3. The sample sizes for the Black, Asian, Latinx, and Multiracial
groups were 306, 310, 163, and 109, respectively. The sample sizes for
men and women were 372 and 556, respectively. Table 4 lists the latent
means and Cronbach’s alphas across the groups (Black and women as
referent groups).
PORS-SF
For PORS-SF, the fit indices of the configural, metric, and scalar
models indicated no meaningful decrement in fit from configural, to
metric, to scalar models for both racial/ethnic and gender groups, sug-
gesting evidence of measurement invariance across the four racial/ethnic
groups and between men and women (Chen, 2007; Cheung & Rensvold,
2002).
PORS-VB
For PORS-VB, across the four racial/ethnic groups, the fit indices of
the configural metric, and scalar models indicated no meaningful
decrement in fit from configural, to metric, to scalar models, except for
the change in RMSEA (-.018) that was slightly above the recommended
threshold (0.015) between the configural and metric models. However,
the chi-square difference test between the two models was not signifi-
cant. The chi-square difference test between the metric and scalar models
was also not significant. Between men and women, the chi-square dif-
ference test was not significant in comparing the configural and metric
model, and the changes in fit indices from configural to the metric model
did not indicate a significant decrease. The fit indices also did not change
meaningfully from metric to scalar model. Taken together, the results
suggested evidence of measurement invariance of PORS-VB across the
four racial/ethnic groups and between men and women (Chen, 2007;
Cheung & Rensvold, 2002).
Further validation of the PORS-SF and PORS-VB (Sample 3)
Tables 3 and 4 list the model fit indices and all of the descriptive
statistics and Cronbach’s alphas. With data collected from a separate
sample (sample 3; N ¼ 454) we further cross-validated the three-factor
and unidimensional structures of PORS-SF and PORS-VB and examined
additional construct validity evidence with theory-driven relationships.
We hypothesized that the three factors of the PORS-SF and PORS-VB
would be positively associated with additional variables of psychologi-
cal deficits (Keum & Miller, 2017) including psychological distress (K-6)
and loneliness (Priest et al., 2017). Additionally, given that online racism
represents stress mediated via social media (Lim et al., 2020), we hy-
pothesized that PORS-VB would significantly predict greater social
media-related stress. We also hypothesized that PORS-VB would signif-
icantly predict alcohol use. Despite the health risks, studies have found
that people may use alcohol to cope with psychological difficulties,
including racism-related stress (Grekin, 2012).
8
Computers in Human Behavior Reports 3 (2021) 100082
PORS-SF
For PORS-SF, the three-factor structure had a good fit to the data, CFI
¼ 0.944, TLI ¼ 0.932, RMSEA ¼ 0.067 [0.058, 0.076], SRMS ¼ 0.062. All
items loaded significantly (p < .001) on the hypothesized latent factors.
The measurement model with all of the criterion variables and the
three subscales of.
PORS-SF had a good fit to the data, CFI ¼ 935, TLI ¼ 0.929, RMSEA ¼
0.048 [0.044, 0.052], SRMR ¼ 0.056. Criterion-related validity evidence
was demonstrated via significantly positive correlations for PORS-SF
with validity variables. K-6 was positively correlated with PORS-SF
Total (r ¼ 0.44, p < .001), PERCA-Brief (r ¼ 0.55, p < .001), VERCA-
Brief (r ¼ 0.36, p < .001), and OMERR-Brief (r ¼ 0.39, p < .001).
Loneliness was positively correlated with PORS-SF Total (r ¼ 0.25, p <
.001), PERCA-Brief (r ¼ 0.32, p < .001), VERCA-Brief (r ¼ 0.22, p <
.001), and OMERR-Brief (r ¼ 0.27, p < .001). Social media related stress
was positively correlated with PORS-SF Total (r ¼ 0.22, p < .001),
PERCA-Brief (r ¼ 0.34, p < .001), VERCA-Brief (r ¼ 0.33, p < .001), and
OMERR-Brief (r ¼ 0.35, p < .001). Demonstrating predictive validity
evidence, alcohol use was predicted by PORS-SF Total (b ¼ 0.51, p ¼ .05,
R2 ¼ 0.26), PERCA-Brief (b ¼ 0.55, SE ¼ 0.04, p < .001, R2 ¼ 0.42),
VERCA-Brief (b ¼ 0.34, SE ¼ 0.05, p < .001, R2 ¼ 0.11), and OMERR-
Brief (b ¼ 0.39, SE ¼ 0.05, p < .001, R2 ¼ 0.13).
PORS-VB
For PORS-VB, the single factor model had an adequate fit to the data,
CFI ¼ 0.962, TLI ¼ 0.928, RMSEA ¼ 0.086 [0.059, 0.116], SRMR ¼
0.036.
The measurement model with all of the criterion variables and the
three subscales of PORS-VB had a good fit to the data, CFI ¼ 935, TLI ¼
0.926, RMSEA ¼ 0.055 [0.050, 0.061], SRMR ¼ 0.065. Criterion-related
validity evidence was demonstrated via significantly positive correla-
tions for PORS-VB with validity variables. PORS-VB was positively
correlated with K-6 (r ¼ 0.30, p < .001), loneliness (r ¼ 0.18, p ¼ .006),
and social media related stress (r ¼ 0.23, p < .001). Demonstrating
predictive validity evidence, PORS-VB significantly predicted alcohol use
(b ¼ 0.35, SE ¼ 0.06, p < .001, R2 ¼ 0.12).
Discussion
The current study developed two brief versions of the Perceived
Online Racism Scale (PORS) to facilitate greater utility. The PORS-SF is a
15-item short form of the original PORS that assesses the same online
racism construct via the three factors assessing personal victimization,
vicarious exposure, and online-mediated exposure (see Appendix). The
PORS-VB is a six-item very brief version based on the PORS general factor
identified through bifactor modeling (see Appendix). Our hypotheses in
developing these two measures were supported, as both PORS-SF and
PORS-VB had an adequate fit to the data, yielded adequate internal
consistency estimates, demonstrated similar correlation and regression
coefficients regarding the construct validity relationships, and showed
similar level of measurement invariance between women and men, and
across the four racial/ethnic groups (Black, Asian, Latinx, Multiracial).
Further validation with separate data collection also provided additional
evidence of the factor structures and theory-driven construct validity
evidence. Collectively, these results suggest that the PORS-SF and PORS-
VB both measure the same construct as the PORS subscales and the
general factor of the PORS, respectively.
With the original PORS (Keum & Miller, 2017), both the correlated
three-factor and bifactor models provided a good fit. However, the
bifactor model was used to demonstrate construct validity evidence in
order to assess subscale utility. In this study, we established greater ev-
idence for use of the PORS-SF subscales as part of a correlated
three-factor model. We also established the utility of a total scale score
given that the second-order model replicated the good fit as a
just-identified model. Thus, rather than needing to model a latent
bifactor structure using structural equation modeling, researchers can
B.T. Keum
use observed scores to calculate the individual susbscale scores by sum-
ming up the scores of the respective subscale items. Researchers can also
consider a total scale score by summing up all of the items (or summing
up the subscale scores).
We were able to develop a very brief PORS-VB by curating items that
map onto the general factor of the original PORS. The IECV values pro-
vided empirical criteria upon which we were able to select and analyze
items that likely contribute greater unique variance to the general uni-
dimensional factor than the group factors (Reise et al., 2012). This was
especially the case since we observed a strong general factor of the PORS.
In essence, the PORS-VB does not measure the three distinct content
domains of the PORS. Rather, it is a measure of a unidimensional
construct for which all PORS items contribute unique common variance,
exclusive of the unique subscale variance of the items. Hence, although
the items no longer load onto a multidimensional structure, since the
common variances are extracted from items that also represent their
respective group factors, the PORS-VB essentially captures a condensed
unidimensional version of the original PORS that should retain similar
psychometric properties. Indeed, the results suggested that the PORS-VB
mirrored the convergent and criterion-related validity evidence of the
PORS, as well as the measurement invariance across racial/ethnic and
gender groups. Thus, the six-item PORS-VB assessed the same latent
general factor.
In developing the shortened measures, the current study also
contributed some notable validity evidence that demonstrated the psy-
chological deficits of online racism as a public health concern for racial/
ethnic minority adults. Higher scores on both measures were signifi-
cantly associated with greater perceived social media-related stress and
feelings of loneliness. These findings are concerning given how they may
negatively contribute to health outcomes among racial/ethnic minority
adults (Keum & Ahn, 2021; Keum & Cano). For example, research sug-
gests that social media use can have negative health effects (e.g., sleep
difficulties; van den Schuur et al., 2019) via social media-related stress.
Compared to White adults, social media-related stress associated with
online racism may be a unique risk factor that exacerbates such detri-
mental outcomes for racial/ethnic minority adults.
Additionally, social media use has been found to evoke greater feel-
ings of loneliness depending on how it is used (e.g., cyber aggression
victimization) and the type of content (e.g., violent media) consumed
(Lim et al., 2020). As a socio-environmental risk factor, persistent
exposure to online racism is likely a trigger of loneliness. Repeated
exposure to online violence has been associated with a person’s tendency
to adopt a hostile attribution bias, or perceptual tendency to view ambig-
uous social situations as hostile (Gentile & Bushman, 2012). Applying the
hostile attribution bias theory, it is likely that explicit and persistent
exposure to online racism promotes an anticipatory perceptual and
behavioral tendency toward future racial discrimination and violence in
social settings, which can have socially isolating implications. Last, in
line with the literature (Grekin, 2012), alcohol use was predicted by both
measures, suggesting that alcohol use may be a risky coping behavior
associated with the psychological deficits of online racism.
Limitations and directions for future research
The current study has several limitations that should be considered
when interpreting the findings. First, although we do not draw any
conclusions of causality in the findings, results were based on cross-
sectional self-report data. The predictability of PORS-SF and PORS-VB
should be assessed with longitudinal data, which would be particularly
helpful to identify directional mechanisms in expanding the nomological
network of the online racism construct. For example, as discussed, it
would be important to examine the mechanisms on how perceived online
racism may ultimately lead to greater feelings of loneliness and alcohol
use among racial/ethnic minority adults to add additional validity evi-
dence and to also examine potential mediators and moderators that can
inform interventions.
9
Computers in Human Behavior Reports 3 (2021) 100082
Second, we used archival data that was used to develop the original
PORS was reanalyzed (with re-splitting the data) to conduct our factor
analytic procedures. Although such practice is conducted for conve-
nience sampling purposes (e.g., McDermott et al., 2019), independent
sampling with new data would be have been most favorable for short-
ening the scales. Thus, there are sample-dependent limitations within the
original dataset that are carried over into our findings. Even though our
expectation was that the PORS-SF and PORS-VB validity will be similar to
the original 30-item PORS, the validity results should be interpreted with
caution. On the other hand, we did provide supplemental
cross-validation evidence of the two scales with new data (sample 3)
which helps support the use of the scales. Regardless, additional vali-
dation of the psychometric properties should be conducted in future
studies. In doing so, the scales should be validated with other constructs
using psychometrically rigorous measures given that some of the
construct validity measures were short (e.g., 1-item social media-related
stress assessment) and the Unjust Views Scale had relatively low reli-
ability (0.65).
Second, although we examined measurement invariance across race/
ethnicity and gender, the current measures and the findings are limited in
understanding the intersectionality of the online racism experience. For
example, how Black women perceive online racism compared to Black
men would be different in the context of compounding oppression of
gender and race (Lewis et al., 2017). This is also within the context that
Black individuals and women reported significantly higher online racism
compared to other groups in the current study. Thus, future research
should focus on extending the psychometric properties of the measures
for group-specific experiences using an intersectionality framework.
Third, although the PORS-SF still covers the three domains of
perceived online racism, it should be noted that PORS-VB only provides a
total score. Thus, researchers who elect to use the PORS-VB should be
reminded that they would not be able to examine the nuanced differences
apparent across the three subscales. For example, given the similarity
with interpersonal racism experiences, the PERCA factor has not been
found to contribute unique significance in psychological distress beyond
offline perceived racism, whereas VERCA and OMERR seem to be the
driver of unique outcomes related to online racism. More research would
be needed to continue building the psychometric properties of PORS-VB
and identify the drawbacks of using the unidimensional structure in
comparison to the three-factor model. For instance, although still
adequate (>.80), we observed that the internal consistency estimates of
PORS-VB were consistently lower than the total score of the PORS-SF and
its subscales.
Implications for practice
The original PORS (Keum & Miller, 2017) allowed mental health
professionals to use the scale as a clinical tool to assess the level of online
racism experienced by their clients. The shortened measures now reduce
the burden on survey-takers (e.g., fatigue, boredom), making it more
accessible as a self-report assessment. The shortened measures could be
helpful for working with racial/ethnic minority adults whose exposure to
online racism seems to present as an ongoing stressor (Tynes et al., 2018).
For a more nuanced individual assessment across the three domains, the
PORS-SF would be preferable, whereas the PORS-VB would allow a quick
general assessment that may be particularly useful for inclusion in sur-
veys geared towards larger populations (e.g., incoming students surveys
at colleges, large epidemiological study batteries).
The brevity of both measures also allows greater facilitation of
expanding the utility with other populations for which online racism is a
salient issue. Notably, online racism coincides with the cyberbullying
that racial/ethnic minority adolescents may experience (Tynes et al.,
2018). Exposure to online racism likely impacts their racial identity
development and social psychological navigation of the racial dynamics
in today’s digitally-driven society. Understanding these mechanisms is an
essential step for developing coping interventions, and ameliorative
B.T. Keum
advocacy and policy efforts. Thus, the shorter measures, with adequate
validation of psychometric properties, would allow researchers, practi-
tioners, and professionals to expand their tool in understanding the racial
socialization and racism experiences of racial/ethnic minority youths and
adolescents who are increasingly victimized in the online world.
Author note
Brian TaeHyuk Keum, Ph.D., Department of Social Welfare, Univer-
sity of California Los Angeles.
Declaration of interest
None.
Declaration of competing interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
Appendix
The Perceived Online Racism Scale-Short Form and -Very Brief
INSTRUCTION: We are interested in your personal experiences of
racism in online settings as you interact with others and surf the Internet.
As you answer the questions below, please think about your online ex-
periences in the past 6 months.
Please rate your responses based on the following options: 1 ¼ Never,
2 ¼ Rarely, 3 ¼ Sometimes, 4 ¼ Often, 5 ¼ Always.
In the past 6 months, I have …
Short Form
1. Received posts with racist comments.
2. Been tagged in (or shared) racist content (e.g., websites, photos,
videos, posts) insulting my race/ethnicity.
3. Received a racist meme (e.g., racist catchphrases, captioned
photos, #hashtags etc.).
4. Been harassed by someone (e.g., troll) who started a racist argu-
ment about me for no reason.
5. Received racist insults regarding my online profile (e.g., profile
pictures, user ID).
6. Seen other racial/minority users being threatened to be harmed or
killed.
7. Seen other racial/minority users being treated like a criminal.
8. Seen other racial/minority users receive racist insults regarding
their online profile (e.g., profile pictures, user ID).
9. Seen other racial/minority users being treated like a second-class
citizen.
10. Seen other racial/minority users receive racist comments.
11. Been informed about unfairness in financial gains for racial/ethnic
minorities (e.g., earning less money than Whites for doing the
same work, unfair housing and loan opportunities).
12. Encountered a viral/trending online racist content (e.g., many
likes, stars).
13. Been informed about a viral/trending racist event happening
elsewhere (e.g., in a different location).
14. Encountered online resources (e.g., Urban Dictionary) promoting
negative racial/ethnic stereotypes as if they are true.
15. Seen online videos (e.g., YouTube) that portray my racial/ethnic
group negatively.
10
Computers in Human Behavior Reports 3 (2021) 100082
Very Brief
1. Seen other racial/minority users receive racist comments.
2. Encountered online hate groups/communities against non-White
racial/ethnic groups.
3. Seen photos that portray my racial/ethnic group negatively.
4. Been informed about a viral/trending racist event that I was not
aware of.
5. Seen online videos (e.g., YouTube) that portray my racial/ethnic
group negatively.
6. Received posts with racist comments.
References
Brondolo, E., Kelly, K. P., Coakley, V., Gordon, T., Thompson, S., Levy, E., &
Contrada, R. J. (2005). The perceived ethnic discrimination Questionnaire:
Development and preliminary validation of a community Version1. Journal of Applied
Social Psychology, 35(2), 335–365. https://doi.org/10.1111/j.1559-1816.2005.tb021
24.x.
Byrd, D. R. (2012). Race/ethnicity and self-reported levels of discrimination and
psychological distress, California, 2005. Preventing Chronic Disease, 9, Article E156.
https://doi.org/10.5888/pcd9.120042
Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement
invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/
10705510701301834
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing
measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.o
rg/10.1207/S15328007SEM0902_5.
Cohen, S., & Janicki-Deverts, D. (2012). Who’s stressed? Distributions of psychological
stress in the United States in probability samples from 1983, 2006, and 20091.
Journal of Applied Social Psychology, 42(6), 1320–1334. https://doi.org/10.1111
/j.1559-1816.2012.00900.x.
Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress.
Journal of Health and Social Behavior, 24(4), 385–396. https://doi.org/10.2307/
2136404
DeVellis, R. F. (2016). Scale development: Theory and applications, 26. Sage publications.
Fischer, A. R., & Bolton Holz, K. (2010). Testing a model of women’s personal sense of
justice, control, well-being, and distress in the context of sexist discrimination.
Psychology of Women Quarterly, 34(3), 297–310. https://doi.org/10.1111/j.1471
-6402.2010.01576.x.
Gentile, D. A., & Bushman, B. J. (2012). Reassessing media violence effects using a risk
and resilience approach to understanding aggression. Psychology of Popular Media
Culture, 1(3), 138–151. https://doi.org/10.1037/a0028481
Grekin, E. R. (2012). Perceived racism and alcohol consequences among African
American and Caucasian college students. Psychology of Addictive Behaviors, 26(4),
924–930. https://doi.org/10.1037/a0029593
Hays, R. D., Spritzer, K. L., Thompson, W. W., & Cella, D. (2015). US general population
estimate for “excellent” to “poor” self-rated health item. Journal of General Internal
Medicine, 30(10), 1511–1516.
Hughes, M. E., Waite, L. J., Hawkley, L. C., & Cacioppo, J. T. (2004). A short scale for
measuring loneliness in large surveys: Results from two population-based studies.
Research on Aging, 26(6), 655–672. https://doi.org/10.1177/0164027504268574
Kessler, R. C., Barker, P. R., Colpe, L. J., Epstein, J. F., Gfroerer, J. C., Hiripi, E., &
Zaslavsky, A. M. (2003). Screening for serious mental illness in the general
population. Archives of General Psychiatry, 60(2), 184–189. https://doi.org/10.1001/
archpsyc.60.2.184
Keum, B. T. (2017). Qualitative examination on the influences of the internet on racism
and its online manifestation. International Journal of Cyber Behavior, Psychology and
Learning, 7(3), 13–23. https://doi.org/10.4018/IJCBPL.2017070102
Keum, B. T., & Ahn, L. H. (2021). Impact of online racism on psychological distress and
alcohol use: Test of ethnic-racial socialization and silence about race as moderators.
Computers in Human Behaviors. https://doi.org/10.1016/j.chb.2021.106773.
Keum, B.T., & Cano, M.A. (n.d.). Online racism, psychological distress, and alcohol use
among racial minority women and men: A multi-group mediation analysis. American
Journal of Orthopsychiatry. In press.
Keum, B. T., & Miller, M. J. (2017). Racism in digital era: Development and initial
validation of the perceived online racism scale (PORS v1.0). Journal of Counseling
Psychology, 64(3), 310–324. https://doi.org/10.1037/cou0000205
Keum, B. T., & Miller, M. J. (2018a). Racism on the internet: Conceptualization and
recommendations for research. Psychology of Violence, 8(6), 782–791. https://
doi.org/10.1037/vio0000201
Keum, B. T., & Miller, M. J. (2018b). Measurement invariance of the perceived online
racism scale across age and gender. Cyberpsychology: Journal of Psychosocial Research
on Cyberspace, 12(3), 3. https://doi.org/10.5817/CP2018-3-3
Keum, B. T., Thai, C. J., Truong, N. N., Ahn, H. L., & Lu, Y. (2018). Factor structure and
measurement invariance of the perceived ethnic discrimination Questionnaire
community version brief across race and gender. International Journal of Culture and
Mental Health, 11(4), 498–512. https://doi.org/10.1080/17542863.2018.1436578
Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). New
York, NY: Guilford Press.
B.T. Keum
Lee, E., Hu, M. Y., & Toh, R. S. (2004). Respondent non-cooperation in surveys and
diaries: An analysis of item non-response and panel attrition. International Journal of
Market Research, 46, 311–326. https://doi.org/10.1177/147078530404600306
Lench, H. C., & Chang, E. S. (2007). Belief in an unjust world: When beliefs in a just world
fail. Journal of Personality Assessment, 89(2), 126–135. https://doi.org/10.1080/00
223890701468477.
Lewis, J. A., Williams, M. G., Peppers, E. J., & Gadson, C. A. (2017). Applying
intersectionality to explore the relations between gendered racism and health among
Black women. Journal of Counseling Psychology, 64(5), 475–486. https://doi.org/
10.1037/cou0000231
Liang, C. T., & Borders, A. (2012). Beliefs in an unjust world mediate the associations
between perceived ethnic discrimination and psychological functioning. Personality
and Individual Differences, 53(4), 528–533. https://doi.org/10.1016/j.paid.2012.04.0
22.
Lim, M. H., Eres, R., & Vasan, S. (2020). Understanding loneliness in the twenty-first
century: An update on correlates, risk factors, and potential solutions. Social
Psychiatry and Psychiatric Epidemiology, 1–18. https://doi.org/10.1007/s00127-020-
01889-7
Lozada, F. T., Seaton, E. K., Williams, C. D., & Tynes, B. M. (2020). Exploration of
bidirectionality in African American and Latinx adolescents’ offline and online
ethnic-racial discrimination. Cultural Diversity and Ethnic Minority Psychology. https://
doi.org/10.1037/cdp0000355
McDermott, R. C., Levant, R. F., Hammer, J. H., Borgogna, N. C., & McKelvey, D. K.
(2019). Development and validation of a five-item male role norms inventory using
bifactor modeling. Psychology of Men & Masculinities, 20(4), 467–477. https://
doi.org/10.1037/men0000178
Priest, N., Perry, R., Ferdinand, A., Kelaher, M., & Paradies, Y. (2017). Effects over time of
self- reported direct and vicarious racial discrimination on depressive symptoms and
loneliness among Australian school students. BMC Psychiatry, 17(1), 50–61. https://
doi.org/10.1186/s12888-017-1216-3
Reinert, D. F., & Allen, J. P. (2007). The alcohol use disorders identification test: An
update of research findings. Alcoholism: Clinical and Experimental Research, 31(2),
185–199. https://doi.org/10.1111/j.1530-0277.2006.00295.x
11
Computers in Human Behavior Reports 3 (2021) 100082
Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate
Behavioral Research, 47(5), 667–696. https://doi.org/10.1080/00273171.20
12.715555.
Saunders, J. B., Aasland, O. G., Babor, T. F., De la Fuente, J. R., & Grant, M. (1993).
Development of the alcohol use disorders identification test (AUDIT): WHO
collaborative project on early detection of persons with harmful alcohol
consumption-II. Addiction, 88(6), 791–804. https://doi.org/10.1111/j.1360-
0443.1993.tb02093.x
van der Schuur, W. A., Baumgartner, S. E., & Sumter, S. R. (2019). Social media use, social
media stress, and sleep: Examining cross-sectional and longitudinal relationships in
adolescents. Health Communication, 34(5), 552–559. https://doi.org/10.1080/
10410236.2017.1422101
Stucky, B. D., & Edelen, M. O. (2014). Using hierarchical IRT models to create
unidimensional measures from multidimensional data. In S. P. Reise, & D. A. Revicki
(Eds.), Handbook of item response theory modeling: Applications to typical performance
assessment (pp. 183–206). New York, NY: Routledge/Taylor & Francis Group.
Tynes, B. M., Giang, M. T., Williams, D. R., & Thompson, G. N. (2008). Online racial
discrimination and psychological adjustment among adolescents. Journal of
Adolescent Health, 43, 565–569. https://doi.org/10.1016/j.jadohealth.2008.08.021
Tynes, B. M., Lozada, F. T., Smith, N. A., & Stewart, A. M. (2018). From racial
microaggressions to hate crimes: A model of online racism based on the lived
experiences of adolescents of color. In Microaggression theory: Influence and
implications (pp. 194–212). John Wiley & Sons, Inc. Uma~ na-Taylor, A. J., Tynes, B. M.
Uma~ na-Taylor, A. J., Tynes, B. M., Toomey, R. B., Williams, D. R., & Mitchell, K. J. (2015).
Latino adolescents’ perceived discrimination in online and offline settings: An
examination of cultural risk and protective factors. Developmental Psychology, 51(1),
87–100. https://doi.org/10.1037/a0038432
Veit, C. T., & Ware, J. E. (1983). The structure of psychological distress and well-being in
general populations. Journal of Consulting and Clinical Psychology, 51(5), 730–742.
https://doi.org/10.1037/0022-006X.51.5.730