Professional Documents
Culture Documents
Critical Review of Bias in NLP
Critical Review of Bias in NLP
H. Samy Alim, John R. Rickford, and Arnetha F. Ball, John Baugh. 2018. Linguistics in Pursuit of Justice.
editors. 2016. Raciolinguistics: How Language Cambridge University Press.
Shapes Our Ideas About Race. Oxford University
Press. Emily M. Bender. 2019. A typology of ethical risks
in language technology with an eye towards where
Sandeep Attree. 2019. Gendered ambiguous pronouns transparent documentation can help. Presented at
shared task: Boosting model confidence by evidence The Future of Artificial Intelligence: Language,
pooling. In Proceedings of the Workshop on Gen- Ethics, Technology Workshop. https://bit.ly/
der Bias in Natural Language Processing, Florence, 2P9t9M6.
Italy.
Ruha Benjamin. 2019. Race After Technology: Aboli-
Pinkesh Badjatiya, Manish Gupta, and Vasudeva tionist Tools for the New Jim Code. John Wiley &
Varma. 2019. Stereotypical bias removal for hate Sons.
speech detection task using knowledge-based gen-
eralizations. In Proceedings of the International Ruha Benjamin. 2020. 2020 Vision: Reimagining the
World Wide Web Conference, pages 49–59, San Fran- Default Settings of Technology & Society. Keynote
cisco, CA. at ICLR.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Cynthia L. Bennett and Os Keyes. 2019. What is the
Shmatikov. 2019. Differential Privacy Has Dis- Point of Fairness? Disability, AI, and The Com-
parate Impact on Model Accuracy. In Proceedings plexity of Justice. In Proceedings of the ASSETS
of the Conference on Neural Information Processing Workshop on AI Fairness for People with Disabili-
Systems, Vancouver, Canada. ties, Pittsburgh, PA.
April Baker-Bell. 2019. Dismantling anti-black lin- Camiel J. Beukeboom and Christian Burgers. 2019.
guistic racism in English language arts classrooms: How Stereotypes Are Shared Through Language: A
Toward an anti-racist black language pedagogy. The- Review and Introduction of the Social Categories
ory Into Practice. and Stereotypes Communication (SCSC) Frame-
work. Review of Communication Research, 7:1–37.
David Bamman, Sejal Popat, and Sheng Shen. 2019.
An annotated dataset of literary entities. In Proceed- Shruti Bhargava and David Forsyth. 2019. Expos-
ings of the North American Association for Com- ing and Correcting the Gender Bias in Image
putational Linguistics (NAACL), pages 2138–2144, Captioning Datasets and Models. arXiv preprint
Minneapolis, MN. arXiv:1912.00578.
Xingce Bao and Qianqian Qiao. 2019. Transfer Learn- Jayadev Bhaskaran and Isha Bhallamudi. 2019. Good
ing from Pre-trained BERT for Pronoun Resolution. Secretaries, Bad Truck Drivers? Occupational Gen-
In Proceedings of the Workshop on Gender Bias der Stereotypes in Sentiment Analysis. In Proceed-
in Natural Language Processing, pages 82–88, Flo- ings of the Workshop on Gender Bias in Natural Lan-
rence, Italy. guage Processing, pages 62–68, Florence, Italy.
Su Lin Blodgett, Lisa Green, and Brendan O’Connor. Student Researchers as Linguistic Experts. Lan-
2016. Demographic Dialectal Variation in Social guage and Linguistics Compass, 8:144–157.
Media: A Case Study of African-American English.
In Proceedings of Empirical Methods in Natural Kaylee Burns, Lisa Anne Hendricks, Kate Saenko,
Language Processing (EMNLP), pages 1119–1130, Trevor Darrell, and Anna Rohrbach. 2018. Women
Austin, TX. also Snowboard: Overcoming Bias in Captioning
Models. In Procedings of the European Conference
Su Lin Blodgett and Brendan O’Connor. 2017. Racial on Computer Vision (ECCV), pages 793–811, Mu-
Disparity in Natural Language Processing: A Case nich, Germany.
Study of Social Media African-American English.
In Proceedings of the Workshop on Fairness, Ac- Aylin Caliskan, Joanna J. Bryson, and Arvind
countability, and Transparency in Machine Learning Narayanan. 2017. Semantics derived automatically
(FAT/ML), Halifax, Canada. from language corpora contain human-like biases.
Science, 356(6334).
Su Lin Blodgett, Johnny Wei, and Brendan O’Connor.
2018. Twitter Universal Dependency Parsing for Kathryn Campbell-Kibler. 2009. The nature of so-
African-American and Mainstream American En- ciolinguistic perception. Language Variation and
glish. In Proceedings of the Association for Compu- Change, 21(1):135–156.
tational Linguistics (ACL), pages 1415–1425, Mel-
bourne, Australia. Yang Trista Cao and Hal Daumé, III. 2019. To-
ward gender-inclusive coreference resolution. arXiv
Tolga Bolukbasi, Kai-Wei Chang, James Zou, preprint arXiv:1910.13913.
Venkatesh Saligrama, and Adam Kalai. 2016a.
Man is to Computer Programmer as Woman is to Rakesh Chada. 2019. Gendered pronoun resolution us-
Homemaker? Debiasing Word Embeddings. In Pro- ing bert and an extractive question answering formu-
ceedings of the Conference on Neural Information lation. In Proceedings of the Workshop on Gender
Processing Systems, pages 4349–4357, Barcelona, Bias in Natural Language Processing, pages 126–
Spain. 133, Florence, Italy.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Kaytlin Chaloner and Alfredo Maldonado. 2019. Mea-
Venkatesh Saligrama, and Adam Kalai. 2016b. suring Gender Bias in Word Embedding across Do-
Quantifying and reducing stereotypes in word mains and Discovering New Gender Bias Word Cat-
embeddings. In Proceedings of the ICML Workshop egories. In Proceedings of the Workshop on Gender
on #Data4Good: Machine Learning in Social Good Bias in Natural Language Processing, pages 25–32,
Applications, pages 41–45, New York, NY. Florence, Italy.
Shikha Bordia and Samuel R. Bowman. 2019. Identify-
ing and reducing gender bias in word-level language Anne H. Charity Hudley. 2017. Language and Racial-
models. In Proceedings of the NAACL Student Re- ization. In Ofelia García, Nelson Flores, and Mas-
search Workshop, pages 7–15, Minneapolis, MN. similiano Spotti, editors, The Oxford Handbook of
Language and Society. Oxford University Press.
Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ash-
ton Anderson, and Richard Zemel. 2019. Under- Won Ik Cho, Ji Won Kim, Seok Min Kim, and
standing the Origins of Bias in Word Embeddings. Nam Soo Kim. 2019. On measuring gender bias in
In Proceedings of the International Conference on translation of gender-neutral pronouns. In Proceed-
Machine Learning, pages 803–811, Long Beach, ings of the Workshop on Gender Bias in Natural Lan-
CA. guage Processing, pages 173–181, Florence, Italy.
Mary Bucholtz, Dolores Inés Casillas, and Jin Sook Shivang Chopra, Ramit Sawhney, Puneet Mathur, and
Lee. 2016. Beyond Empowerment: Accompani- Rajiv Ratn Shah. 2020. Hindi-English Hate Speech
ment and Sociolinguistic Justice in a Youth Research Detection: Author Profiling, Debiasing, and Practi-
Program. In Robert Lawson and Dave Sayers, edi- cal Perspectives. In Proceedings of the AAAI Con-
tors, Sociolinguistic Research: Application and Im- ference on Artificial Intelligence (AAAI), New York,
pact, pages 25–44. Routledge. NY.
Mary Bucholtz, Dolores Inés Casillas, and Jin Sook Marika Cifor, Patricia Garcia, T.L. Cowan, Jasmine
Lee. 2019. California Latinx Youth as Agents of Rault, Tonia Sutherland, Anita Say Chan, Jennifer
Sociolinguistic Justice. In Netta Avineri, Laura R. Rode, Anna Lauren Hoffmann, Niloufar Salehi, and
Graham, Eric J. Johnson, Robin Conley Riner, and Lisa Nakamura. 2019. Feminist Data Manifest-
Jonathan Rosa, editors, Language and Social Justice No. Retrieved from https://www.manifestno.
in Practice, pages 166–175. Routledge. com/.
Mary Bucholtz, Audrey Lopez, Allina Mojarro, Elena Patricia Hill Collins. 2000. Black Feminist Thought:
Skapoulli, Chris VanderStouwe, and Shawn Warner- Knowledge, Consciousness, and the Politics of Em-
Garcia. 2014. Sociolinguistic Justice in the Schools: powerment. Routledge.
Justin T. Craft, Kelly E. Wright, Rachel Elizabeth Robertson, editors, Routledge International Hand-
Weissler, and Robin M. Queen. 2020. Language book of Participatory Design, pages 182–209. Rout-
and Discrimination: Generating Meaning, Perceiv- ledge.
ing Identities, and Discriminating Outcomes. An-
nual Review of Linguistics, 6(1). Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain,
and Lucy Vasserman. 2018. Measuring and mitigat-
Kate Crawford. 2017. The Trouble with Bias. Keynote ing unintended bias in text classification. In Pro-
at NeurIPS. ceedings of the Conference on Artificial Intelligence,
Ethics, and Society (AIES), New Orleans, LA.
Kimberle Crenshaw. 1989. Demarginalizing the Inter-
section of Race and Sex: A Black Feminist Critique
Jacob Eisenstein. 2013. What to do about bad lan-
of Antidiscrmination Doctrine, Feminist Theory and
guage on the Internet. In Proceedings of the North
Antiracist Politics. University of Chicago Legal Fo-
American Association for Computational Linguistics
rum.
(NAACL), pages 359–369.
Amanda Cercas Curry and Verena Rieser. 2018.
#MeToo: How Conversational Systems Respond to Kawin Ethayarajh. 2020. Is Your Classifier Actually
Sexual Harassment. In Proceedings of the Workshop Biased? Measuring Fairness under Uncertainty with
on Ethics in Natural Language Processing, pages 7– Bernstein Bounds. In Proceedings of the Associa-
14, New Orleans, LA. tion for Computational Linguistics (ACL).
Karan Dabas, Nishtha Madaan, Gautam Singh, Vi- Kawin Ethayarajh, David Duvenaud, and Graeme Hirst.
jay Arya, Sameep Mehta, and Tanmoy Chakraborty. 2019. Understanding Undesirable Word Embedding
2020. Fair Transfer of Multiple Style Attributes in Assocations. In Proceedings of the Association for
Text. arXiv preprint arXiv:2001.06693. Computational Linguistics (ACL), pages 1696–1705,
Florence, Italy.
Thomas Davidson, Debasmita Bhattacharya, and Ing-
mar Weber. 2019. Racial bias in hate speech and Joseph Fisher. 2019. Measuring social bias in
abusive language detection datasets. In Proceedings knowledge graph embeddings. arXiv preprint
of the Workshop on Abusive Language Online, pages arXiv:1912.02761.
25–35, Florence, Italy.
Maria De-Arteaga, Alexey Romanov, Hanna Wal- Nelson Flores and Sofia Chaparro. 2018. What counts
lach, Jennifer Chayes, Christian Borgs, Alexandra as language education policy? Developing a materi-
Chouldechova, Sahin Geyik, Krishnaram Kentha- alist Anti-racist approach to language activism. Lan-
padi, and Adam Tauman Kalai. 2019. Bias in bios: guage Policy, 17(3):365–384.
A case study of semantic representation bias in a
high-stakes setting. In Proceedings of the Confer- Omar U. Florez. 2019. On the Unintended Social Bias
ence on Fairness, Accountability, and Transparency, of Training Language Generation Models with Data
pages 120–128, Atlanta, GA. from Local Media. In Proceedings of the NeurIPS
Workshop on Human-Centric Machine Learning,
Sunipa Dev, Tao Li, Jeff Phillips, and Vivek Sriku- Vancouver, Canada.
mar. 2019. On Measuring and Mitigating Biased
Inferences of Word Embeddings. arXiv preprint Joel Escudé Font and Marta R. Costa-jussà. 2019.
arXiv:1908.09369. Equalizing gender biases in neural machine trans-
lation with word embeddings techniques. In Pro-
Sunipa Dev and Jeff Phillips. 2019. Attenuating Bias in ceedings of the Workshop on Gender Bias for Natu-
Word Vectors. In Proceedings of the International ral Language Processing, pages 147–154, Florence,
Conference on Artificial Intelligence and Statistics, Italy.
pages 879–887, Naha, Japan.
Batya Friedman and David G. Hendry. 2019. Value
Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Sensitive Design: Shaping Technology with Moral
Piper, and Darren Gergle. 2018. Addressing age- Imagination. MIT Press.
related bias in sentiment analysis. In Proceedings
of the Conference on Human Factors in Computing
Batya Friedman, Peter H. Kahn Jr., and Alan Borning.
Systems (CHI), Montréal, Canada.
2006. Value Sensitive Design and Information Sys-
Emily Dinan, Angela Fan, Adina Williams, Jack Ur- tems. In Dennis Galletta and Ping Zhang, editors,
banek, Douwe Kiela, and Jason Weston. 2019. Human-Computer Interaction in Management Infor-
Queens are Powerful too: Mitigating Gender mation Systems: Foundations, pages 348–372. M.E.
Bias in Dialogue Generation. arXiv preprint Sharpe.
arXiv:1911.03842.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and
Carl DiSalvo, Andrew Clement, and Volkmar Pipek. James Zou. 2018. Word Embeddings Quantify 100
2013. Communities: Participatory Design for, with Years of Gender and Ethnic Stereotypes. Proceed-
and by communities. In Jesper Simonsen and Toni ings of the National Academy of Sciences, 115(16).
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Enoch Opanin Gyamfi, Yunbo Rao, Miao Gou, and
Taly, Ed H. Chi, and Alex Beutel. 2019. Counter- Yanhua Shao. 2020. deb2viz: Debiasing gender in
factual fairness in text classification through robust- word embedding data using subspace visualization.
ness. In Proceedings of the Conference on Artificial In Proceedings of the International Conference on
Intelligence, Ethics, and Society (AIES), Honolulu, Graphics and Image Processing.
HI.
Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M.
Aparna Garimella, Carmen Banea, Dirk Hovy, and Branham. 2018. Gender Recognition or Gender Re-
Rada Mihalcea. 2019. Women’s syntactic resilience ductionism? The Social Implications of Automatic
and men’s grammatical luck: Gender bias in part-of- Gender Recognition Systems. In Proceedings of the
speech tagging and dependency parsing data. In Pro- Conference on Human Factors in Computing Sys-
ceedings of the Association for Computational Lin- tems (CHI), Montréal, Canada.
guistics (ACL), pages 3493–3498, Florence, Italy.
Alex Hanna, Emily Denton, Andrew Smart, and Jamila
Andrew Gaut, Tony Sun, Shirlyn Tang, Yuxin Huang, Smith-Loud. 2020. Towards a Critical Race Method-
Jing Qian, Mai ElSherief, Jieyu Zhao, Diba ology in Algorithmic Fairness. In Proceedings of the
Mirza, Elizabeth Belding, Kai-Wei Chang, and Conference on Fairness, Accountability, and Trans-
William Yang Wang. 2020. Towards Understand- parency, pages 501–512, Barcelona, Spain.
ing Gender Bias in Relation Extraction. In Proceed-
ings of the Association for Computational Linguis- Madeline E. Heilman, Aaaron S. Wallen, Daniella
tics (ACL). Fuchs, and Melinda M. Tamkins. 2004. Penalties
for Success: Reactions to Women Who Succeed at
R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Male Gender-Typed Tasks. Journal of Applied Psy-
Jie Qiu, Rebekah Tang, and Jenny Huang. 2020. chology, 89(3):416–427.
Garbage In, Garbage Out? Do Machine Learn-
ing Application Papers in Social Computing Report Jane H. Hill. 2008. The Everyday Language of White
Where Human-Labeled Training Data Comes From? Racism. Wiley-Blackwell.
In Proceedings of the Conference on Fairness, Ac-
Dirk Hovy, Federico Bianchi, and Tommaso Fornaciari.
countability, and Transparency, pages 325–336.
2020. Can You Translate that into Man? Commer-
cial Machine Translation Systems Include Stylistic
Oguzhan Gencoglu. 2020. Cyberbullying Detec-
Biases. In Proceedings of the Association for Com-
tion with Fairness Constraints. arXiv preprint
putational Linguistics (ACL).
arXiv:2005.06625.
Dirk Hovy and Anders Søgaard. 2015. Tagging Per-
Alexandra Reeve Givens and Meredith Ringel Morris. formance Correlates with Author Age. In Proceed-
2020. Centering Disability Perspecives in Algorith- ings of the Association for Computational Linguis-
mic Fairness, Accountability, and Transparency. In tics and the International Joint Conference on Nat-
Proceedings of the Conference on Fairness, Account- ural Language Processing, pages 483–488, Beijing,
ability, and Transparency, Barcelona, Spain. China.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a Dirk Hovy and Shannon L. Spruit. 2016. The social
Pig: Debiasing Methods Cover up Systematic Gen- impact of natural language processing. In Proceed-
der Biases in Word Embeddings But do not Remove ings of the Association for Computational Linguis-
Them. In Proceedings of the North American As- tics (ACL), pages 591–598, Berlin, Germany.
sociation for Computational Linguistics (NAACL),
pages 609–614, Minneapolis, MN. Po-Sen Huang, Huan Zhang, Ray Jiang, Robert
Stanforth, Johannes Welbl, Jack W. Rae, Vishal
Hila Gonen and Kellie Webster. 2020. Auto- Maini, Dani Yogatama, and Pushmeet Kohli. 2019.
matically Identifying Gender Issues in Machine Reducing Sentiment Bias in Language Models
Translation using Perturbations. arXiv preprint via Counterfactual Evaluation. arXiv preprint
arXiv:2004.14065. arXiv:1911.03064.
Ben Green. 2019. “Good” isn’t good enough. In Pro- Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and
ceedings of the AI for Social Good Workshop, Van- Michael J. Paul. 2020. Multilingual Twitter Cor-
couver, Canada. pus and Baselines for Evaluating Demographic Bias
in Hate Speech Recognition. In Proceedings of
Lisa J. Green. 2002. African American English: A Lin- the Language Resources and Evaluation Conference
guistic Introduction. Cambridge University Press. (LREC), Marseille, France.
Anthony G. Greenwald, Debbie E. McGhee, and Jor- Christoph Hube, Maximilian Idahl, and Besnik Fetahu.
dan L.K. Schwartz. 1998. Measuring individual dif- 2020. Debiasing Word Embeddings from Sentiment
ferences in implicit cognition: The implicit associa- Associations in Names. In Proceedings of the Inter-
tion test. Journal of Personality and Social Psychol- national Conference on Web Search and Data Min-
ogy, 74(6):1464–1480. ing, pages 259–267, Houston, TX.
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Masahiro Kaneko and Danushka Bollegala. 2019.
Denton, Kellie Webster, Yu Zhong, and Stephen De- Gender-preserving debiasing for pre-trained word
nuyl. 2020. Social Biases in NLP Models as Barriers embeddings. In Proceedings of the Association for
for Persons with Disabilities. In Proceedings of the Computational Linguistics (ACL), pages 1641–1650,
Association for Computational Linguistics (ACL). Florence, Italy.
Matei Ionita, Yury Kashnitsky, Ken Krige, Vladimir Saket Karve, Lyle Ungar, and João Sedoc. 2019. Con-
Larin, Dennis Logvinenko, and Atanas Atanasov. ceptor debiasing of word representations evaluated
2019. Resolving gendered ambiguous pronouns on WEAT. In Proceedings of the Workshop on Gen-
with BERT. In Proceedings of the Workshop on der Bias in Natural Language Processing, pages 40–
Gender Bias in Natural Language Processing, pages 48, Florence, Italy.
113–119, Florence, Italy.
Michael Katell, Meg Young, Dharma Dailey, Bernease
Hailey James-Sorenson and David Alvarez-Melis. Herman, Vivian Guetler, Aaron Tam, Corinne Bintz,
2019. Probabilistic Bias Mitigation in Word Embed- Danielle Raz, and P.M. Krafft. 2020. Toward sit-
dings. In Proceedings of the Workshop on Human- uated interventions for algorithmic equity: lessons
Centric Machine Learning, Vancouver, Canada. from the field. In Proceedings of the Conference on
Shengyu Jia, Tao Meng, Jieyu Zhao, and Kai-Wei Fairness, Accountability, and Transparency, pages
Chang. 2020. Mitigating Gender Bias Amplification 45–55, Barcelona, Spain.
in Distribution by Posterior Regularization. In Pro-
ceedings of the Association for Computational Lin- Stephen Kemmis. 2006. Participatory action research
guistics (ACL). and the public sphere. Educational Action Research,
14(4):459–476.
Taylor Jones, Jessica Rose Kalbfeld, Ryan Hancock,
and Robin Clark. 2019. Testifying while black: An Os Keyes. 2018. The Misgendering Machines:
experimental study of court reporter accuracy in tran- Trans/HCI Implications of Automatic Gender
scription of African American English. Language, Recognition. Proceedings of the ACM on Human-
95(2). Computer Interaction, 2(CSCW).
Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2015. Os Keyes, Josephine Hoy, and Margaret Drouhard.
Challenges of studying and processing dialects in 2019. Human-Computer Insurrection: Notes on an
social media. In Proceedings of the Workshop on Anarchist HCI. In Proceedings of the Conference on
Noisy User-Generated Text, pages 9–18, Beijing, Human Factors in Computing Systems (CHI), Glas-
China. gow, Scotland, UK.
Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2016. Jae Yeon Kim, Carlos Ortiz, Sarah Nam, Sarah Santi-
Learning a POS tagger for AAVE-like language. In ago, and Vivek Datta. 2020. Intersectional Bias in
Proceedings of the North American Association for Hate Speech and Abusive Language Datasets. In
Computational Linguistics (NAACL), pages 1115– Proceedings of the Association for Computational
1120, San Diego, CA. Linguistics (ACL).
Pratik Joshi, Sebastian Santy, Amar Budhiraja, Kalika Svetlana Kiritchenko and Saif M. Mohammad. 2018.
Bali, and Monojit Choudhury. 2020. The State and Examining Gender and Race Bias in Two Hundred
Fate of Linguistic Diversity and Inclusion in the Sentiment Analysis Systems. In Proceedings of the
NLP World. In Proceedings of the Association for Joint Conference on Lexical and Computational Se-
Computational Linguistics (ACL). mantics, pages 43–53, New Orleans, LA.
Jaap Jumelet, Willem Zuidema, and Dieuwke Hupkes.
2019. Analysing Neural Language Models: Contex- Moshe Koppel, Shlomo Argamon, and Anat Rachel
tual Decomposition Reveals Default Reasoning in Shimoni. 2002. Automatically Categorizing Writ-
Number and Gender Assignment. In Proceedings ten Texts by Author Gender. Literary and Linguistic
of the Conference on Natural Language Learning, Computing, 17(4):401–412.
Hong Kong, China.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W.
Marie-Odile Junker. 2018. Participatory action re- Black, and Yulia Tsvetkov. 2019. Measuring bias
search for Indigenous linguistics in the digital age. in contextualized word representations. In Proceed-
In Shannon T. Bischoff and Carmen Jany, editors, ings of the Workshop on Gender Bias for Natu-
Insights from Practices in Community-Based Re- ral Language Processing, pages 166–172, Florence,
search, pages 164–175. De Gruyter Mouton. Italy.
David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Sonja L. Lanehart and Ayesha M. Malik. 2018. Black
Incorporating Dialectal Variability for Socially Equi- Is, Black Isn’t: Perceptions of Language and Black-
table Language Identification. In Proceedings of the ness. In Jeffrey Reaser, Eric Wilbanks, Karissa Woj-
Association for Computational Linguistics (ACL), cik, and Walt Wolfram, editors, Language Variety in
pages 51–57, Vancouver, Canada. the New South. University of North Carolina Press.
Brian N. Larson. 2017. Gender as a variable in natural- Anastassia Loukina, Nitin Madnani, and Klaus Zech-
language processing: Ethical considerations. In Pro- ner. 2019. The many dimensions of algorithmic fair-
ceedings of the Workshop on Ethics in Natural Lan- ness in educational applications. In Proceedings of
guage Processing, pages 30–40, Valencia, Spain. the Workshop on Innovative Use of NLP for Build-
ing Educational Applications, pages 1–10, Florence,
Anne Lauscher and Goran Glavaš. 2019. Are We Con- Italy.
sistently Biased? Multidimensional Analysis of Bi-
ases in Distributional Word Vectors. In Proceedings Kaiji Lu, Peter Mardziel, Fangjing Wu, Preetam Aman-
of the Joint Conference on Lexical and Computa- charla, and Anupam Datta. 2018. Gender bias in
tional Semantics, pages 85–91, Minneapolis, MN. neural natural language processing. arXiv preprint
arXiv:1807.11714.
Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto,
and Ivan Vulić. 2019. A General Framework for Im- Anne Maass. 1999. Linguistic intergroup bias: Stereo-
plicit and Explicit Debiasing of Distributional Word type perpetuation through language. Advances in
Vector Spaces. arXiv preprint arXiv:1909.06092. Experimental Social Psychology, 31:79–121.
Christopher A. Le Dantec, Erika Shehan Poole, and Su- Nitin Madnani, Anastassia Loukina, Alina von Davier,
san P. Wyche. 2009. Values as Lived Experience: Jill Burstein, and Aoife Cahill. 2017. Building Bet-
Evolving Value Sensitive Design in Support of Value ter Open-Source Tools to Support Fairness in Auto-
Discovery. In Proceedings of the Conference on Hu- mated Scoring. In Proceedings of the Workshop on
man Factors in Computing Systems (CHI), Boston, Ethics in Natural Language Processing, pages 41–
MA. 52, Valencia, Spain.
Nayeon Lee, Andrea Madotto, and Pascale Fung. 2019. Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and
Exploring Social Bias in Chatbots using Stereotype Alan W. Black. 2019. Black is to Criminal as Cau-
Knowledge. In Proceedings of the Workshop on casian is to Police: Detecting and Removing Multi-
Widening NLP, pages 177–180, Florence, Italy. class Bias in Word Embeddings. In Proceedings of
the North American Association for Computational
Wesley Y. Leonard. 2012. Reframing language recla- Linguistics (NAACL), pages 801–809, Minneapolis,
mation programmes for everybody’s empowerment. MN.
Gender and Language, 6(2):339–367. Ramón Antonio Martínez and Alexander Feliciano
Mejía. 2019. Looking closely and listening care-
Paul Pu Liang, Irene Li, Emily Zheng, Yao Chong Lim, fully: A sociocultural approach to understanding
Ruslan Salakhutdinov, and Louis-Philippe Morency. the complexity of Latina/o/x students’ everyday lan-
2019. Towards Debiasing Sentence Representations. guage. Theory Into Practice.
In Proceedings of the NeurIPS Workshop on Human-
Centric Machine Learning, Vancouver, Canada. Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell,
and Simone Teufel. 2019. It’s All in the Name: Mit-
Rosina Lippi-Green. 2012. English with an Ac- igating Gender Bias with Name-Based Counterfac-
cent: Language, Ideology, and Discrimination in the tual Data Substitution. In Proceedings of Empirical
United States. Routledge. Methods in Natural Language Processing (EMNLP),
pages 5270–5278, Hong Kong, China.
Bo Liu. 2019. Anonymized BERT: An Augmentation
Approach to the Gendered Pronoun Resolution Chal- Chandler May, Alex Wang, Shikha Bordia, Samuel R.
lenge. In Proceedings of the Workshop on Gender Bowman, and Rachel Rudinger. 2019. On Measur-
Bias in Natural Language Processing, pages 120– ing Social Biases in Sentence Encoders. In Proceed-
125, Florence, Italy. ings of the North American Association for Compu-
tational Linguistics (NAACL), pages 629–634, Min-
Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zi- neapolis, MN.
tao Liu, and Jiliang Tang. 2019. Does Gender Mat-
ter? Towards Fairness in Dialogue Systems. arXiv Elijah Mayfield, Michael Madaio, Shrimai Prab-
preprint arXiv:1910.10486. humoye, David Gerritsen, Brittany McLaughlin,
Ezekiel Dixon-Roman, and Alan W. Black. 2019.
Felipe Alfaro Lois, José A.R. Fonollosa, and Costa-jà. Equity Beyond Bias in Language Technologies for
2019. BERT Masked Language Modeling for Co- Education. In Proceedings of the Workshop on Inno-
reference Resolution. In Proceedings of the Work- vative Use of NLP for Building Educational Appli-
shop on Gender Bias in Natural Language Process- cations, Florence, Italy.
ing, pages 76–81, Florence, Italy.
Katherine McCurdy and Oğuz Serbetçi. 2017. Gram-
Brandon C. Loudermilk. 2015. Implicit attitudes and matical gender associations outweigh topical gender
the perception of sociolinguistic variation. In Alexei bias in crosslinguistic word embeddings. In Pro-
Prikhodkine and Dennis R. Preston, editors, Re- ceedings of the Workshop for Women & Underrepre-
sponses to Language Varieties: Variability, pro- sented Minorities in Natural Language Processing,
cesses and outcomes, pages 137–156. Vancouver, Canada.
Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent
Nanyun Peng, and Aram Galstyan. 2019. Man is to Disagreements in Human Textual Inferences. Trans-
Person as Woman is to Location: Measuring Gender actions of the Association for Computational Lin-
Bias in Named Entity Recognition. arXiv preprint guistics, 7:677–694.
arXiv:1910.10872.
Xiangyu Peng, Siyan Li, Spencer Frazier, and Mark
Michela Menegatti and Monica Rubini. 2017. Gender Riedl. 2020. Fine-Tuning a Transformer-Based Lan-
bias and sexism in language. In Oxford Research guage Model to Avoid Generating Non-Normative
Encyclopedia of Communication. Oxford University Text. arXiv preprint arXiv:2001.08764.
Press.
Radomir Popović, Florian Lemmerich, and Markus
Inom Mirzaev, Anthony Schulte, Michael Conover, and Strohmaier. 2020. Joint Multiclass Debiasing of
Sam Shah. 2019. Considerations for the interpreta- Word Embeddings. In Proceedings of the Interna-
tion of bias measures of word embeddings. arXiv tional Symposium on Intelligent Systems, Graz, Aus-
preprint arXiv:1906.08379. tria.
Salikoko S. Mufwene, Guy Bailey, and John R. Rick-
Vinodkumar Prabhakaran, Ben Hutchinson, and Mar-
ford, editors. 1998. African-American English:
garet Mitchell. 2019. Perturbation Sensitivity Anal-
Structure, History, and Use. Routledge.
ysis to Detect Unintended Model Biases. In Pro-
Michael J. Muller. 2007. Participatory Design: The ceedings of Empirical Methods in Natural Language
Third Space in HCI. In The Human-Computer Inter- Processing (EMNLP), pages 5744–5749, Hong
action Handbook, pages 1087–1108. CRC Press. Kong, China.
Moin Nadeem, Anna Bethke, and Siva Reddy. Shrimai Prabhumoye, Elijah Mayfield, and Alan W.
2020. StereoSet: Measuring stereotypical bias Black. 2019. Principled Frameworks for Evaluating
in pretrained language models. arXiv preprint Ethics in NLP Systems. In Proceedings of the Work-
arXiv:2004.09456. shop on Innovative Use of NLP for Building Educa-
tional Applications, Florence, Italy.
Dong Nguyen, Rilana Gravel, Dolf Trieschnigg, and
Theo Meder. 2013. “How Old Do You Think I Marcelo Prates, Pedro Avelar, and Luis C. Lamb. 2019.
Am?”: A Study of Language and Age in Twitter. In Assessing gender bias in machine translation: A
Proceedings of the Conference on Web and Social case study with google translate. Neural Computing
Media (ICWSM), pages 439–448, Boston, MA. and Applications.
Malvina Nissim, Rik van Noord, and Rob van der Goot. Rasmus Précenth. 2019. Word embeddings and gender
2020. Fair is better than sensational: Man is to doc- stereotypes in Swedish and English. Master’s thesis,
tor as woman is to doctor. Computational Linguis- Uppsala University.
tics.
Dennis R. Preston. 2009. Are you really smart (or
Debora Nozza, Claudia Volpetti, and Elisabetta Fersini. stupid, or cute, or ugly, or cool)? Or do you just talk
2019. Unintended Bias in Misogyny Detection. In that way? Language attitudes, standardization and
Proceedings of the Conference on Web Intelligence, language change. Oslo: Novus forlag, pages 105–
pages 149–155. 129.
Alexandra Olteanu, Carlos Castillo, Fernando Diaz,
Flavien Prost, Nithum Thain, and Tolga Bolukbasi.
and Emre Kıcıman. 2019. Social Data: Biases,
2019. Debiasing Embeddings for Reduced Gender
Methodological Pitfalls, and Ethical Boundaries.
Bias in Text Classification. In Proceedings of the
Frontiers in Big Data, 2.
Workshop on Gender Bias in Natural Language Pro-
Alexandra Olteanu, Kartik Talamadupula, and Kush R. cessing, pages 69–75, Florence, Italy.
Varshney. 2017. The Limits of Abstract Evaluation
Metrics: The Case of Hate Speech Detection. In Reid Pryzant, Richard Diehl Martinez, Nathan Dass,
Proceedings of the ACM Web Science Conference, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang.
Troy, NY. 2020. Automatically Neutralizing Subjective Bias
in Text. In Proceedings of the AAAI Conference on
Orestis Papakyriakopoulos, Simon Hegelich, Juan Car- Artificial Intelligence (AAAI), New York, NY.
los Medina Serrano, and Fabienne Marco. 2020.
Bias in word embeddings. In Proceedings of the Arun K. Pujari, Ansh Mittal, Anshuman Padhi, An-
Conference on Fairness, Accountability, and Trans- shul Jain, Mukesh Jadon, and Vikas Kumar. 2019.
parency, pages 446–457, Barcelona, Spain. Debiasing Gender biased Hindi Words with Word-
embedding. In Proceedings of the International
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Re- Conference on Algorithms, Computing and Artificial
ducing Gender Bias in Abusive Language Detection. Intelligence, pages 450–456.
In Proceedings of Empirical Methods in Natural
Language Processing (EMNLP), pages 2799–2804, Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won
Brussels, Belgium. Hyun. 2019. Reducing gender bias in word-level
language models with a gender-equalizing loss func- David Rozado. 2020. Wide range screening of algo-
tion. In Proceedings of the ACL Student Research rithmic bias in word embedding models using large
Workshop, pages 223–228, Florence, Italy. sentiment lexicons reveals underreported bias types.
PLOS One.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael
Twiton, and Yoav Goldberg. 2020. Null It Out: Elayne Ruane, Abeba Birhane, and Anthony Ven-
Guarding Protected Attributes by Iterative Nullspace tresque. 2019. Conversational AI: Social and Ethi-
Projection. In Proceedings of the Association for cal Considerations. In Proceedings of the Irish Con-
Computational Linguistics (ACL). ference on Artificial Intelligence and Cognitive Sci-
John R. Rickford and Sharese King. 2016. Language ence, Galway, Ireland.
and linguistics on trial: Hearing Rachel Jeantel (and
other vernacular speakers) in the courtroom and be- Rachel Rudinger, Chandler May, and Benjamin
yond. Language, 92(4):948–988. Van Durme. 2017. Social bias in elicited natural lan-
guage inferences. In Proceedings of the Workshop
Anthony Rios. 2020. FuzzE: Fuzzy Fairness Evalua- on Ethics in Natural Language Processing, pages
tion of Offensive Language Classifiers on African- 74–79, Valencia, Spain.
American English. In Proceedings of the AAAI Con-
ference on Artificial Intelligence (AAAI), New York, Rachel Rudinger, Jason Naradowsky, Brian Leonard,
NY. and Benjamin Van Durme. 2018. Gender Bias
in Coreference Resolution. In Proceedings of the
Gerald Roche. 2019. Articulating language oppres- North American Association for Computational Lin-
sion: colonialism, coloniality and the erasure of Ti- guistics (NAACL), pages 8–14, New Orleans, LA.
betâĂŹs minority languages. Patterns of Prejudice.
Elizabeth B.N. Sanders. 2002. From user-centered to
Alexey Romanov, Maria De-Arteaga, Hanna Wal- participatory design approaches. In Jorge Frascara,
lach, Jennifer Chayes, Christian Borgs, Alexandra editor, Design and the Social Sciences: Making Con-
Chouldechova, Sahin Geyik, Krishnaram Kentha- nections, pages 18–25. CRC Press.
padi, Anna Rumshisky, and Adam Tauman Kalai.
2019. What’s in a Name? Reducing Bias in Bios Brenda Salenave Santana, Vinicius Woloszyn, and Le-
without Access to Protected Attributes. In Proceed- andro Krug Wives. 2018. Is there gender bias and
ings of the North American Association for Com- stereotype in Portuguese word embeddings? In
putational Linguistics (NAACL), pages 4187–4195, Proceedings of the International Conference on the
Minneapolis, MN. Computational Processing of Portuguese Student Re-
Jonathan Rosa. 2019. Contesting Representations of search Workshop, Canela, Brazil.
Migrant “Illegality” through the Drop the I-Word
Campaign: Rethinking Language Change and So- Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi,
cial Change. In Netta Avineri, Laura R. Graham, and Noah A. Smith. 2019. The risk of racial bias in
Eric J. Johnson, Robin Conley Riner, and Jonathan hate speech detection. In Proceedings of the Asso-
Rosa, editors, Language and Social Justice in Prac- ciation for Computational Linguistics (ACL), pages
tice. Routledge. 1668–1678, Florence, Italy.
Jonathan Rosa and Christa Burdick. 2017. Language Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf-
Ideologies. In Ofelia García, Nelson Flores, and sky, Noah A. Smith, and Yejin Choi. 2020. Social
Massimiliano Spotti, editors, The Oxford Handbook Bias Frames: Reasoning about Social and Power Im-
of Language and Society. Oxford University Press. plications of Language. In Proceedings of the Asso-
ciation for Computational Linguistics (ACL).
Jonathan Rosa and Nelson Flores. 2017. Unsettling
race and language: Toward a raciolinguistic perspec- Hanna Sassaman, Jennifer Lee, Jenessa Irvine, and
tive. Language in Society, 46:621–647. Shankar Narayan. 2020. Creating Community-
Based Tech Policy: Case Studies, Lessons Learned,
Sara Rosenthal and Kathleen McKeown. 2011. Age and What Technologists and Communities Can Do
Prediction in Blogs: A Study of Style, Content, and Together. In Proceedings of the Conference on Fair-
Online Behavior in Pre- and Post-Social Media Gen- ness, Accountability, and Transparency, Barcelona,
erations. In Proceedings of the North American As- Spain.
sociation for Computational Linguistics (NAACL),
pages 763–772, Portland, OR. Danielle Saunders and Bill Byrne. 2020. Reducing
Candace Ross, Boris Katz, and Andrei Barbu. Gender Bias in Neural Machine Translation as a Do-
2020. Measuring Social Biases in Grounded Vi- main Adaptation Problem. In Proceedings of the As-
sion and Language Embeddings. arXiv preprint sociation for Computational Linguistics (ACL).
arXiv:2002.08911.
Tyler Schnoebelen. 2017. Goal-Oriented Design for
Richard Rothstein. 2017. The Color of Law: A For- Ethical Machine Learning and NLP. In Proceedings
gotten History of How Our Government Segregated of the Workshop on Ethics in Natural Language Pro-
America. Liveright Publishing. cessing, pages 88–93, Valencia, Spain.
Sabine Sczesny, Magda Formanowicz, and Franziska Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang,
Moser. 2016. Can gender-fair language reduce gen- Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth
der stereotyping and discrimination? Frontiers in Belding, Kai-Wei Chang, and William Yang Wang.
Psychology, 7. 2019. Mitigating Gender Bias in Natural Lan-
guage Processing: Literature Review. In Proceed-
João Sedoc and Lyle Ungar. 2019. The Role of Pro- ings of the Association for Computational Linguis-
tected Class Word Lists in Bias Identification of Con- tics (ACL), pages 1630–1640, Florence, Italy.
textualized Word Representations. In Proceedings
of the Workshop on Gender Bias in Natural Lan- Adam Sutton, Thomas Lansdall-Welfare, and Nello
guage Processing, pages 55–61, Florence, Italy. Cristianini. 2018. Biased embeddings from wild
data: Measuring, understanding and removing.
Procheta Sen and Debasis Ganguly. 2020. Towards So- In Proceedings of the International Symposium
cially Responsible AI: Cognitive Bias-Aware Multi- on Intelligent Data Analysis, pages 328–339, ’s-
Objective Learning. In Proceedings of the AAAI Hertogenbosch, Netherlands.
Conference on Artificial Intelligence (AAAI), New Chris Sweeney and Maryam Najafian. 2019. A Trans-
York, NY. parent Framework for Evaluating Unintended De-
mographic Bias in Word Embeddings. In Proceed-
Deven Shah, H. Andrew Schwartz, and Dirk Hovy. ings of the Association for Computational Linguis-
2020. Predictive Biases in Natural Language Pro- tics (ACL), pages 1662–1667, Florence, Italy.
cessing Models: A Conceptual Framework and
Overview. In Proceedings of the Association for Chris Sweeney and Maryam Najafian. 2020. Reduc-
Computational Linguistics (ACL). ing sentiment polarity for demographic attributes
in word embeddings using adversarial learning. In
Judy Hanwen Shen, Lauren Fratamico, Iyad Rahwan, Proceedings of the Conference on Fairness, Ac-
and Alexander M. Rush. 2018. Darling or Baby- countability, and Transparency, pages 359–368,
girl? Investigating Stylistic Bias in Sentiment Anal- Barcelona, Spain.
ysis. In Proceedings of the Workshop on Fairness,
Accountability, and Transparency (FAT/ML), Stock- Nathaniel Swinger, Maria De-Arteaga, Neil Thomas
holm, Sweden. Heffernan, Mark D.M. Leiserson, and Adam Tau-
man Kalai. 2019. What are the biases in my word
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, embedding? In Proceedings of the Conference on
and Nanyun Peng. 2019. The Woman Worked as Artificial Intelligence, Ethics, and Society (AIES),
a Babysitter: On Biases in Language Generation. Honolulu, HI.
In Proceedings of Empirical Methods in Natural
Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard
Language Processing (EMNLP), pages 3398–3403,
Socher. 2020. It’s Morphin’ Time! Combating
Hong Kong, China.
Linguistic Discrimination with Inflectional Perturba-
tions. In Proceedings of the Association for Compu-
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, tational Linguistics (ACL).
and Nanyun Peng. 2020. Towards Controllable
Biases in Language Generation. arXiv preprint Yi Chern Tan and L. Elisa Celis. 2019. Assessing
arXiv:2005.00268. Social and Intersectional Biases in Contextualized
Word Representations. In Proceedings of the Con-
Seungjae Shin, Kyungwoo Song, JoonHo Jang, Hyemi ference on Neural Information Processing Systems,
Kim, Weonyoung Joo, and Il-Chul Moon. 2020. Vancouver, Canada.
Neutralizing Gender Bias in Word Embedding with
Latent Disentanglement and Counterfactual Genera- J. Michael Terry, Randall Hendrick, Evangelos Evan-
tion. arXiv preprint arXiv:2004.03133. gelou, and Richard L. Smith. 2010. Variable
dialect switching among African American chil-
Jesper Simonsen and Toni Robertson, editors. 2013. dren: Inferences about working memory. Lingua,
Routledge International Handbook of Participatory 120(10):2463–2475.
Design. Routledge.
Joel Tetreault, Daniel Blanchard, and Aoife Cahill.
2013. A Report on the First Native Language Iden-
Gabriel Stanovsky, Noah A. Smith, and Luke Zettle-
tification Shared Task. In Proceedings of the Work-
moyer. 2019. Evaluating gender bias in machine
shop on Innovative Use of NLP for Building Educa-
translation. In Proceedings of the Association for
tional Applications, pages 48–57, Atlanta, GA.
Computational Linguistics (ACL), pages 1679–1684,
Florence, Italy. Mike Thelwall. 2018. Gender Bias in Sentiment Anal-
ysis. Online Information Review, 42(1):45–57.
Yolande Strengers, Lizhe Qu, Qiongkai Xu, and Jarrod
Knibbe. 2020. Adhering, Steering, and Queering: Kristen Vaccaro, Karrie Karahalios, Deirdre K. Mul-
Treatment of Gender in Natural Language Genera- ligan, Daniel Kluttz, and Tad Hirsch. 2019. Con-
tion. In Proceedings of the Conference on Human testability in Algorithmic Systems. In Conference
Factors in Computing Systems (CHI), Honolulu, HI. Companion Publication of the 2019 on Computer
Supported Cooperative Work and Social Computing, Kai-Chou Yang, Timothy Niven, Tzu-Hsuan Chou, and
pages 523–527, Austin, TX. Hung-Yu Kao. 2019. Fill the GAP: Exploiting
BERT for Pronoun Resolution. In Proceedings of
Ameya Vaidya, Feng Mai, and Yue Ning. 2019. Em- the Workshop on Gender Bias in Natural Language
pirical Analysis of Multi-Task Learning for Reduc- Processing, pages 102–106, Florence, Italy.
ing Model Bias in Toxic Comment Detection. arXiv
preprint arXiv:1909.09758v2. Zekun Yang and Juan Feng. 2020. A Causal Inference
Method for Reducing Gender Bias in Word Embed-
Eva Vanmassenhove, Christian Hardmeier, and Andy ding Relations. In Proceedings of the AAAI Con-
Way. 2018. Getting Gender Right in Neural Ma- ference on Artificial Intelligence (AAAI), New York,
chine Translation. In Proceedings of Empirical NY.
Methods in Natural Language Processing (EMNLP),
pages 3003–3008, Brussels, Belgium. Daisy Yoo, Anya Ernest, Sofia Serholt, Eva Eriksson,
and Peter Dalsgaard. 2019. Service Design in HCI
Research: The Extended Value Co-creation Model.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov,
In Proceedings of the Halfway to the Future Sympo-
Sharon Qian, Daniel Nevo, Yaron Singer, and Stu-
sium, Nottingham, United Kingdom.
art Shieber. 2020. Causal Mediation Analysis for
Interpreting Neural NLP: The Case of Gender Bias. Brian Hu Zhang, Blake Lemoine, and Margaret
arXiv preprint arXiv:2004.12265. Mitchell. 2018. Mitigating unwanted biases with ad-
versarial learning. In Proceedings of the Conference
Tianlu Wang, Xi Victoria Lin, Nazneen Fatema Ra- on Artificial Intelligence, Ethics, and Society (AIES),
jani, Bryan McCann, Vicente Ordonez, and Caim- New Orleans, LA.
ing Xiong. 2020. Double-Hard Debias: Tailoring
Word Embeddings for Gender Bias Mitigation. In Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Con-
Proceedings of the Association for Computational ghui Zhu, and Tiejun Zhao. 2020a. Demographics
Linguistics (ACL). Should Not Be the Reason of Toxicity: Mitigating
Discrimination in Text Classifications with Instance
Zili Wang. 2019. MSnet: A BERT-based Network for Weighting. In Proceedings of the Association for
Gendered Pronoun Resolution. In Proceedings of Computational Linguistics (ACL).
the Workshop on Gender Bias in Natural Language
Processing, pages 89–95, Florence, Italy. Haoran Zhang, Amy X. Lu, Mohamed Abdalla,
Matthew McDermott, and Marzyeh Ghassemi.
Kellie Webster, Marta R. Costa-jussà, Christian Hard- 2020b. Hurtful Words: Quantifying Biases in Clin-
meier, and Will Radford. 2019. Gendered Ambigu- ical Contextual Word Embeddings. In Proceedings
ous Pronoun (GAP) Shared Task at the Gender Bias of the ACM Conference on Health, Inference, and
in NLP Workshop 2019. In Proceedings of the Work- Learning.
shop on Gender Bias in Natural Language Process-
ing, pages 1–7, Florence, Italy. Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini,
Kai-Wei Chang, and Ahmed Hassan Awadallah.
Kellie Webster, Marta Recasens, Vera Axelrod, and Ja- 2020. Gender Bias in Multilingual Embeddings and
son Baldridge. 2018. Mind the GAP: A balanced Cross-Lingual Transfer. In Proceedings of the Asso-
corpus of gendered ambiguous pronouns. Transac- ciation for Computational Linguistics (ACL).
tions of the Association for Computational Linguis-
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cot-
tics, 6:605–618.
terell, Vicente Ordonez, and Kai-Wei Chang. 2019.
Gender Bias in Contextualized Word Embeddings.
Walt Wolfram and Natalie Schilling. 2015. American In Proceedings of the North American Association
English: Dialects and Variation, 3 edition. Wiley for Computational Linguistics (NAACL), pages 629–
Blackwell. 634, Minneapolis, MN.
Austin P. Wright, Omar Shaikh, Haekyu Park, Will Ep- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or-
person, Muhammed Ahmed, Stephane Pinel, Diyi donez, and Kai-Wei Chang. 2017. Men also like
Yang, and Duen Horng (Polo) Chau. 2020. RE- shopping: Reducing gender bias amplification us-
CAST: Interactive Auditing of Automatic Toxicity ing corpus-level constraints. In Proceedings of
Detection Models. In Proceedings of the Con- Empirical Methods in Natural Language Process-
ference on Human Factors in Computing Systems ing (EMNLP), pages 2979–2989, Copenhagen, Den-
(CHI), Honolulu, HI. mark.
Yinchuan Xu and Junlin Yang. 2019. Look again at Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or-
the syntax: Relational graph convolutional network donez, and Kai-Wei Chang. 2018a. Gender Bias
for gendered ambiguous pronoun resolution. In Pro- in Coreference Resolution: Evaluation and Debias-
ceedings of the Workshop on Gender Bias in Natu- ing Methods. In Proceedings of the North American
ral Language Processing, pages 96–101, Florence, Association for Computational Linguistics (NAACL),
Italy. pages 15–20, New Orleans, LA.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai- do not. As a result, we excluded them from our
Wei Chang. 2018b. Learning Gender-Neutral Word counts for techniques as well. We cite the papers
Embeddings. In Proceedings of Empirical Methods
here; most propose techniques we would have cate-
in Natural Language Processing (EMNLP), pages
4847–4853, Brussels, Belgium. gorized as “Questionable correlations,” with a few
as “Other representational harms” (Abzaliev, 2019;
Alina Zhiltsova, Simon Caton, and Catherine Mulwa. Attree, 2019; Bao and Qiao, 2019; Chada, 2019;
2019. Mitigation of Unintended Biases against Non-
Native English Texts in Sentiment Analysis. In Pro-
Ionita et al., 2019; Liu, 2019; Lois et al., 2019;
ceedings of the Irish Conference on Artificial Intelli- Wang, 2019; Xu and Yang, 2019; Yang et al., 2019).
gence and Cognitive Science, Galway, Ireland. We excluded Dabas et al. (2020) from our survey
because we could not determine what this paper’s
Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang,
Muhao Chen, and Kai-Wei Chang. 2019. Examin- user study on fairness was actually measuring.
ing gender bias in languages with grammatical gen- Finally, we actually categorized the motivation
ders. In Proceedings of Empirical Methods in Nat- for Liu et al. (2019) (i.e., the last row in Table 3) as
ural Language Processing (EMNLP), pages 5279– “Questionable correlations” due to a sentence else-
5287, Hong Kong, China.
where in the paper; had the paragraph we quoted
Ran Zmigrod, S. J. Mielke, Hanna Wallach, and Ryan been presented without more detail, we would have
Cotterell. 2019. Counterfactual data augmentation categorized the motivation as “Vague/unstated.”
for mitigating gender stereotypes in languages with
rich morphology. In Proceedings of the Association A.2 Full categorization: Motivations
for Computational Linguistics (ACL), pages 1651–
1661, Florence, Italy. Allocational harms Hovy and Spruit (2016);
Caliskan et al. (2017); Madnani et al. (2017);
A Appendix Dixon et al. (2018); Kiritchenko and Mohammad
(2018); Shen et al. (2018); Zhao et al. (2018b);
In Table 3, we provide examples of the papers’ mo-
Bhaskaran and Bhallamudi (2019); Bordia and
tivations and techniques across several NLP tasks.
Bowman (2019); Brunet et al. (2019); Chaloner
A.1 Categorization details and Maldonado (2019); De-Arteaga et al. (2019);
Dev and Phillips (2019); Font and Costa-jussà
In this section, we provide some additional details
(2019); James-Sorenson and Alvarez-Melis (2019);
about our method—specifically, our categorization.
Kurita et al. (2019); Mayfield et al. (2019); Pu-
What counts as being covered by an NLP task? jari et al. (2019); Romanov et al. (2019); Ruane
We considered a paper to cover a given NLP task if et al. (2019); Sedoc and Ungar (2019); Sun et al.
it analyzed “bias” with respect to that task, but not (2019); Zmigrod et al. (2019); Hutchinson et al.
if it only evaluated overall performance on that task. (2020); Papakyriakopoulos et al. (2020); Ravfo-
For example, a paper examining the impact of miti- gel et al. (2020); Strengers et al. (2020); Sweeney
gating “bias” in word embeddings on “bias” in sen- and Najafian (2020); Tan et al. (2020); Zhang et al.
timent analysis would be counted as covering both (2020b).
NLP tasks. In contrast, a paper assessing whether
performance on sentiment analysis degraded after Stereotyping Bolukbasi et al. (2016a,b);
mitigating “bias” in word embeddings would be Caliskan et al. (2017); McCurdy and Serbetçi
counted only as focusing on embeddings. (2017); Rudinger et al. (2017); Zhao et al. (2017);
Curry and Rieser (2018); Díaz et al. (2018);
What counts as a motivation? We considered a Santana et al. (2018); Sutton et al. (2018); Zhao
motivation to include any description of the prob- et al. (2018a,b); Agarwal et al. (2019); Basta et al.
lem that motivated the paper or proposed quantita- (2019); Bhaskaran and Bhallamudi (2019); Bordia
tive technique, including any normative reasoning. and Bowman (2019); Brunet et al. (2019); Cao
We excluded from the “Vague/unstated” cate- and Daumé (2019); Chaloner and Maldonado
gory of motivations the papers that participated in (2019); Cho et al. (2019); Dev and Phillips (2019);
the Gendered Ambiguous Pronoun (GAP) Shared Font and Costa-jussà (2019); Gonen and Goldberg
Task at the First ACL Workshop on Gender Bias in (2019); James-Sorenson and Alvarez-Melis (2019);
NLP. In an ideal world, shared task papers would Kaneko and Bollegala (2019); Karve et al. (2019);
engage with “bias” more critically, but given the Kurita et al. (2019); Lauscher and Glavaš (2019);
nature of shared tasks it is understandable that they Lee et al. (2019); Manzini et al. (2019); Mayfield
Categories
NLP task Stated motivation Motivations Techniques
Language “Existing biases in data can be amplified by models and the Allocational Questionable
modeling resulting output consumed by the public can influence them, en- harms, correlations
(Bordia and courage and reinforce harmful stereotypes, or distort the truth. stereotyping
Bowman, Automated systems that depend on these models can take prob-
2019) lematic actions based on biased profiling of individuals.”
Sentiment “Other biases can be inappropriate and result in negative ex- Allocational Questionable
analysis periences for some groups of people. Examples include, loan harms, other correlations
(Kiritchenko eligibility and crime recidivism prediction systems...and resumé representational (differences in
and sorting systems that believe that men are more qualified to be harms (system sentiment
Mohammad, programmers than women (Bolukbasi et al., 2016). Similarly, performance intensity scores
2018) sentiment and emotion analysis systems can also perpetuate and differences w.r.t. w.r.t. text about
accentuate inappropriate human biases, e.g., systems that consider text written by different social
utterances from one race or gender to be less positive simply be- different social groups)
cause of their race or gender, or customer support systems that groups)
prioritize a call from an angry male over a call from the equally
angry female.”
Machine “[MT training] may incur an association of gender-specified pro- Questionable Questionable
translation nouns (in the target) and gender-neutral ones (in the source) for correlations, correlations
(Cho et al., lexicon pairs that frequently collocate in the corpora. We claim other
2019) that this kind of phenomenon seriously threatens the fairness of a representational
translation system, in the sense that it lacks generality and inserts harms
social bias to the inference. Moreover, the input is not fully cor-
rect (considering gender-neutrality) and might offend the users
who expect fairer representations.”
Machine “Learned models exhibit social bias when their training data Stereotyping, Stereotyping,
translation encode stereotypes not relevant for the task, but the correlations questionable other
(Stanovsky are picked up anyway.” correlations representational
et al., 2019) harms (system
performance
differences),
questionable
correlations
Type-level “However, embeddings trained on human-generated corpora have Allocational Stereotyping
embeddings been demonstrated to inherit strong gender stereotypes that re- harms,
(Zhao et al., flect social constructs....Such a bias substantially affects down- stereotyping,
2018b) stream applications....This concerns the practitioners who use other
the embedding model to build gender-sensitive applications such representational
as a resume filtering system or a job recommendation system as harms
the automated system may discriminate candidates based on their
gender, as reflected by their name. Besides, biased embeddings
may implicitly affect downstream applications used in our daily
lives. For example, when searching for ‘computer scientist’ using
a search engine...a search algorithm using an embedding model in
the backbone tends to rank male scientists higher than females’
[sic], hindering women from being recognized and further exac-
erbating the gender inequality in the community.”
Type-level “[P]rominent word embeddings such as word2vec (Mikolov et Vague Stereotyping
and contextu- al., 2013) and GloVe (Pennington et al., 2014) encode systematic
alized biases against women and black people (Bolukbasi et al., 2016;
embeddings Garg et al., 2018), implicating many NLP systems in scaling up
(May et al., social injustice.”
2019)
Dialogue “Since the goal of dialogue systems is to talk with users...if the Vague/unstated Stereotyping,
generation systems show discriminatory behaviors in the interactions, the other
(Liu et al., user experience will be adversely affected. Moreover, public com- representational
2019) mercial chatbots can get resisted for their improper speech.” harms,
questionable
correlations
Table 3: Examples of the categories into which the papers’ motivations and proposed quantitative techniques for
measuring or mitigating “bias” fall. Bold text in the quotes denotes the content that yields our categorizations.
et al. (2019); Précenth (2019); Pujari et al. (2019); Chopra et al. (2020); Gonen and Webster (2020);
Ruane et al. (2019); Stanovsky et al. (2019); Gyamfi et al. (2020); Hube et al. (2020); Ravfogel
Sun et al. (2019); Tan and Celis (2019); Webster et al. (2020); Rios (2020); Ross et al. (2020); Saun-
et al. (2019); Zmigrod et al. (2019); Gyamfi et al. ders and Byrne (2020); Sen and Ganguly (2020);
(2020); Hube et al. (2020); Hutchinson et al. Shah et al. (2020); Sweeney and Najafian (2020);
(2020); Kim et al. (2020); Nadeem et al. (2020); Yang and Feng (2020); Zhang et al. (2020a).
Papakyriakopoulos et al. (2020); Ravfogel et al.
Vague/unstated Rudinger et al. (2018); Webster
(2020); Rozado (2020); Sen and Ganguly (2020);
et al. (2018); Dinan et al. (2019); Florez (2019);
Shin et al. (2020); Strengers et al. (2020).
Jumelet et al. (2019); Lauscher et al. (2019); Liang
Other representational harms Hovy and Sø- et al. (2019); Maudslay et al. (2019); May et al.
gaard (2015); Blodgett et al. (2016); Bolukbasi (2019); Prates et al. (2019); Prost et al. (2019);
et al. (2016b); Hovy and Spruit (2016); Blodgett Qian et al. (2019); Swinger et al. (2019); Zhao
and O’Connor (2017); Larson (2017); Schnoebelen et al. (2019); Zhou et al. (2019); Ethayarajh (2020);
(2017); Blodgett et al. (2018); Curry and Rieser Huang et al. (2020); Jia et al. (2020); Popović et al.
(2018); Díaz et al. (2018); Dixon et al. (2018); Kir- (2020); Pryzant et al. (2020); Vig et al. (2020);
itchenko and Mohammad (2018); Park et al. (2018); Wang et al. (2020); Zhao et al. (2020).
Shen et al. (2018); Thelwall (2018); Zhao et al. Surveys, frameworks, and meta-analyses
(2018b); Badjatiya et al. (2019); Bagdasaryan et al. Hovy and Spruit (2016); Larson (2017); McCurdy
(2019); Bamman et al. (2019); Cao and Daumé and Serbetçi (2017); Schnoebelen (2017); Basta
(2019); Chaloner and Maldonado (2019); Cho et al. et al. (2019); Ethayarajh et al. (2019); Gonen and
(2019); Davidson et al. (2019); De-Arteaga et al. Goldberg (2019); Lauscher and Glavaš (2019);
(2019); Fisher (2019); Font and Costa-jussà (2019); Loukina et al. (2019); Mayfield et al. (2019);
Garimella et al. (2019); Loukina et al. (2019); May- Mirzaev et al. (2019); Prabhumoye et al. (2019);
field et al. (2019); Mehrabi et al. (2019); Nozza Ruane et al. (2019); Sedoc and Ungar (2019); Sun
et al. (2019); Prabhakaran et al. (2019); Romanov et al. (2019); Nissim et al. (2020); Rozado (2020);
et al. (2019); Ruane et al. (2019); Sap et al. (2019); Shah et al. (2020); Strengers et al. (2020); Wright
Sheng et al. (2019); Sun et al. (2019); Sweeney et al. (2020).
and Najafian (2019); Vaidya et al. (2019); Gaut
et al. (2020); Gencoglu (2020); Hovy et al. (2020); B Full categorization: Techniques
Hutchinson et al. (2020); Kim et al. (2020); Peng Allocational harms De-Arteaga et al. (2019);
et al. (2020); Rios (2020); Sap et al. (2020); Shah Prost et al. (2019); Romanov et al. (2019); Zhao
et al. (2020); Sheng et al. (2020); Tan et al. (2020); et al. (2020).
Zhang et al. (2020a,b).
Stereotyping Bolukbasi et al. (2016a,b);
Questionable correlations Jørgensen et al. Caliskan et al. (2017); McCurdy and Serbetçi
(2015); Hovy and Spruit (2016); Madnani et al. (2017); Díaz et al. (2018); Santana et al. (2018);
(2017); Rudinger et al. (2017); Zhao et al. (2017); Sutton et al. (2018); Zhang et al. (2018); Zhao
Burns et al. (2018); Dixon et al. (2018); Kir- et al. (2018a,b); Agarwal et al. (2019); Basta et al.
itchenko and Mohammad (2018); Lu et al. (2018); (2019); Bhaskaran and Bhallamudi (2019); Brunet
Park et al. (2018); Shen et al. (2018); Zhang et al. (2019); Cao and Daumé (2019); Chaloner
et al. (2018); Badjatiya et al. (2019); Bhargava and Maldonado (2019); Dev and Phillips (2019);
and Forsyth (2019); Cao and Daumé (2019); Cho Ethayarajh et al. (2019); Gonen and Goldberg
et al. (2019); Davidson et al. (2019); Dev et al. (2019); James-Sorenson and Alvarez-Melis (2019);
(2019); Garimella et al. (2019); Garg et al. (2019); Jumelet et al. (2019); Kaneko and Bollegala
Huang et al. (2019); James-Sorenson and Alvarez- (2019); Karve et al. (2019); Kurita et al. (2019);
Melis (2019); Kaneko and Bollegala (2019); Liu Lauscher and Glavaš (2019); Lauscher et al.
et al. (2019); Karve et al. (2019); Nozza et al. (2019); Lee et al. (2019); Liang et al. (2019); Liu
(2019); Prabhakaran et al. (2019); Romanov et al. et al. (2019); Manzini et al. (2019); Maudslay et al.
(2019); Sap et al. (2019); Sedoc and Ungar (2019); (2019); May et al. (2019); Mirzaev et al. (2019);
Stanovsky et al. (2019); Sweeney and Najafian Prates et al. (2019); Précenth (2019); Prost et al.
(2019); Vaidya et al. (2019); Zhiltsova et al. (2019); (2019); Pujari et al. (2019); Qian et al. (2019);
Sedoc and Ungar (2019); Stanovsky et al. (2019); Surveys, frameworks, and meta-analyses
Tan and Celis (2019); Zhao et al. (2019); Zhou Hovy and Spruit (2016); Larson (2017); McCurdy
et al. (2019); Chopra et al. (2020); Gyamfi et al. and Serbetçi (2017); Schnoebelen (2017); Basta
(2020); Nadeem et al. (2020); Nissim et al. (2020); et al. (2019); Ethayarajh et al. (2019); Gonen and
Papakyriakopoulos et al. (2020); Popović et al. Goldberg (2019); Lauscher and Glavaš (2019);
(2020); Ravfogel et al. (2020); Ross et al. (2020); Loukina et al. (2019); Mayfield et al. (2019);
Rozado (2020); Saunders and Byrne (2020); Shin Mirzaev et al. (2019); Prabhumoye et al. (2019);
et al. (2020); Vig et al. (2020); Wang et al. (2020); Ruane et al. (2019); Sedoc and Ungar (2019); Sun
Yang and Feng (2020); Zhao et al. (2020). et al. (2019); Nissim et al. (2020); Rozado (2020);
Shah et al. (2020); Strengers et al. (2020); Wright
Other representational harms Jørgensen et al. et al. (2020).
(2015); Hovy and Søgaard (2015); Blodgett et al.
(2016); Blodgett and O’Connor (2017); Blodgett
et al. (2018); Curry and Rieser (2018); Dixon et al.
(2018); Park et al. (2018); Thelwall (2018); Web-
ster et al. (2018); Badjatiya et al. (2019); Bag-
dasaryan et al. (2019); Bamman et al. (2019); Bhar-
gava and Forsyth (2019); Cao and Daumé (2019);
Font and Costa-jussà (2019); Garg et al. (2019);
Garimella et al. (2019); Liu et al. (2019); Louk-
ina et al. (2019); Mehrabi et al. (2019); Nozza
et al. (2019); Sap et al. (2019); Sheng et al. (2019);
Stanovsky et al. (2019); Vaidya et al. (2019);
Webster et al. (2019); Ethayarajh (2020); Gaut
et al. (2020); Gencoglu (2020); Hovy et al. (2020);
Huang et al. (2020); Kim et al. (2020); Peng et al.
(2020); Ravfogel et al. (2020); Rios (2020); Sap
et al. (2020); Saunders and Byrne (2020); Sheng
et al. (2020); Sweeney and Najafian (2020); Tan
et al. (2020); Zhang et al. (2020a,b).
Vague/unstated None.