i

The Association for Computing Machinery

1601 Broadway, 10th Floor
New York, New York 10019, USA

ACM COPYRIGHT NOTICE. Copyright © 2022 by the Association for Computing Machinery,
Inc. Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this notice and the
full citation on the first page. Copyrights for components of this work owned by others
than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to
republish, to post on servers, or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from Publications Dept., ACM, Inc., fax +1 (212) 869-0481,
or permissions@acm.org.

For other copying of articles that carry a code at the bottom of the first or last page,
copying is permitted provided that the per-copy fee indicated in the code is paid
through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923,
+1-978-750-8400, +1-978-750-4470 (fax).

ACM ISBN: 978-1-4503-9258-7

ii
ii
 ASSETS 2022 Foreword

We are pleased to welcome you to the 24th International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS 2022). ASSETS is the premier computing research
conference exploring the design, evaluation, and use of computing and information technologies
with and for people with disabilities and older adults.

With the continuing COVID-19 pandemic, we have made changes to the conference format. We
are gathering in-person for the first time since 2019 in Athens, Greece, with an additional light-
weight online component for virtual conference attendees. This conference was organized to
support accessibility and inclusivity, given the technical and time-zone constraints of the
conference location. In anticipation of fewer infrastructural resources to support fully streamed
accessible content, and to acknowledge the reality of time-zone differences, we made available
pre-recorded, captioned videos for all presentations to both virtual and in-person conference
attendees. There are also additional creative opportunities for hybrid connections and socializing
across modalities, including online trivia and roundtable social networking events. We
acknowledge that future ASSETS organizing committees will need to make their own decisions
about the length and format of the conference, and so the decision to have an in-person with
light-weight online component is not a permanent decision.

This year’s opening keynote will be given by lawyer and disability rights advocate Haben Girma,
titled “I’m human, too: the biases driving accessibility solutions.” The second day of the
conference features a keynote by SIGACCESS Outstanding Contributions awardee, Dr. Clayton
Lewis.

This year’s technical program features 35 papers, selected by the program committee from 132
submissions (acceptance rate of 26.5%). The program also features six papers published in
TACCESS in the last year. These papers present work from the leading edge of accessible
computing research, including novel tools and interaction methods, advances in VR and AR for
accessibility, accessible technologies for independent living in everyday life, and the perception
and representation of disabilities by diverse groups.

This year was the first year of the Workshops track, led by Kyle Montague and Sowmya
Somanath, which featured five workshops held virtually in the weeks preceding the conference.
These five workshops were selected after a juried review process where researchers and
practitioners who work in the areas of accessibility, disability and computing were invited to
comment and discuss the workshop proposal submissions.

iii
iii
The Posters and Demos track, chaired by Taslima Akter and Hugo Nicolau, received 73
submissions. The program committee selected 43 posters and demos for presentation, resulting in
an acceptance rate of 59%. The accepted posters present late-breaking work from the research
and professional community. The accepted demos provide an opportunity for conference
attendees to experience new advances in accessible technology in-person and virtually.

The Experience Reports track documents authors’ personal or stakeholder experiences related to
the creation, use, and deployment of accessible technologies. Chaired by Katherine Ringland and
Garreth Tigwell, this track received 27 submissions, of which seven reports were selected for
presentation (acceptance rate of 26%).

ACM SIGACCESS is committed to developing the next generation of researchers in the field of
accessible computing. This year’s ASSETS continues the tradition of supporting and showcasing
the work of student researchers. The Doctoral Consortium, chaired by Aqueasha Martin-
Hammond and Katta Spiel, brought together 12 doctoral students to discuss their work with a
panel of established researchers in a one-day workshop on the Saturday before the main
conference. A special edition of the SIGACCESS newsletter will feature extended abstracts from
these doctoral students.

The ACM Student Research Competition (SRC), chaired by Mingming Fan and Roshan Peiris,
offers a unique forum for undergraduate and graduate students to present their original research
before a panel of judges and receive direct feedback on their work from experts. This year, three
entries were selected as SRC finalists to give poster presentations. All participants will give oral
presentations at the conference. Winners from the ASSETS 2022 SRC will go on to compete in
the ACM-wide grand finals.

For three consecutive years, the ASSETS Conference has highlighted submissions that include
components accessible to the public through the Best Artifact Award. The Best Artifact Award is
chaired this year by Dragan Ahmetovic.

The ASSETS 2022 program is organized as follows:

On Monday (Day 1), the program starts with a keynote presentation by Haben Girma. The
keynote will be followed by the first Poster Session, which will feature virtual Student Research
Competition poster presentations. Next, the first technical paper session is convened under the
theme of “AR, VR and Games”. After the lunch break, the second technical paper session
features topics within the theme of “Representation and Inclusion”. The afternoon session
continues with a continuation of the morning Poster session, followed by the technical paper

iv
iv
session “Data for Modeling and Recognition”. Day 1 concludes with an evening reception and
demo session.

On Tuesday (Day 2), the SIGACCESS Outstanding Contributions Keynote address is presented
by Clayton Lewis. The break and second Poster Session is then followed by the technical paper
session on “Composition in Music, Programming and Design”. In the afternoon, a session is
dedicated to the Student Research Competition Finalists presentation. The fifth technical paper
session presents work in the area of “Communication”. After the afternoon break with
continuation of the second Poster Session, the final technical paper session for Day 2 is on
“Social Media and Media”. The SIGACCESS Town Hall will take place at the end of Day 2,
followed by an off-site reception.

On Wednesday (Day 3), we start with a technical session on “Tactile and Haptics”. After a short
break, the session on “Accessibility in Daily Living” is presented. After lunch, the final technical
paper session is dedicated to “Safety, Rehabilitation, and Transportation”. The closing plenary
closes out Day 3.

During the closing session, the ASSETS 2022 awards will be presented, and we will introduce
next year's chairs to make an announcement about ASSETS 2023.

The success of the ASSETS 2022 conference required a tremendous amount of work from
authors, reviewers, committee members, and many others. We thank the authors of all papers,
posters, demos, and experience reports, as well as the applicants to the Doctoral Consortium and
Student Research Competition. We thank the program committee and additional reviewers Sri
Kurniawan and Clayton Lewis for reviewing submissions and providing authors with helpful
feedback.

We also thank the members of the organizing committee: Treasurer and Registration Chair
Martez Mott, Hybrid Experience Chair Christian Vogler, Virtual Chairs Emma McDonnell,
Kelly Mack, Benjamin Tannert, Laurianne Sitbon, Accessibility Chairs J. Bern Jordan and
Lei Shi, Equity and Belonging Chairs Lou Anne Boyd, John A. Guerra Gomez, Proceedings
Chairs Raja Kushalnagar and Sergio Mascetti, Posters and Demos Chairs Taslima Akter and
Hugo Nicolau, Doctoral Consortium Chairs Aqueasha Martin-Hammond and Katta Spiel,
Student Research Competition Chairs Mingming Fan and Roshan Peiris, Experience Reports
Chairs Katherine Ringland and Garreth Tigwell, Web Design Chairs Liang He, Junhan (Judy)
Kong, Jaylin Herskovitz, Jason Wu, Mentoring Chairs Sayan Sarcar and Kirsten Ellis, Publicity
Chairs Alexa Siu and Arthur Theill, Student Volunteer Chairs Franklin Mingzhe Li and Maryam
Banduka, TACCESS Special Issue Chair Maria Wolters, and Best Artifact Award Chairs Dragan
Ahmetovic and Hugo Nicolau.

v
v
We thank the Best Paper Award committee Anthony Hornof, Amy Hurst, Cynthia Putnam,
Sayan Sarcar, and Gerhard Weber. We thank the SIGACCESS Steering Committee: Matt
Huenerfauth (Rochester Institute of Technology, USA) - Chair; Jeffrey Bigham (Carnegie
Mellon University, USA), Tiago Guerreiro (University of Lisbon, Portugal), Jonathan Lazar
(University of Maryland, USA); Kathleen McCoy (University of Delaware, USA), Karyn
Moffatt (McGill University, Canada) for their support and guidance.

Finally, we thank our Champion sponsors Fable and Google, our Gold sponsorsAdobe, Amazon,
Apple, Meta, Microsoft, UW CREATE, our Silver sponsors National Institute on Disability,
Independent Living and Rehabilitation Research (NIDILRR), Twitter, Volkswagen, our Bronze
supporters 2G3R, Intuit, Miraikan, and Studio Pacifica, our In-Kind supporter AIRA, and our
Doctoral Consortium sponsor NSF, and the ACM and SIGACCESS for their very generous
support.

Welcome to ASSETS 2022!

Jon Froehlich Kristen Shinohara Stephanie Ludi

ASSETS 2022 General Chair ASSETS 2022 Program Co-Chair ASSETS 2022 Program Co-Chair
University of Washington, USA Rochester Institute of University of North Texas, USA
Technology, USA

vi
vi
 Table of Contents

ASSETS 2022 Conference Organization����������������������������������������������� xix

ASSETS 2022 Sponsors and Supporters���������������������������������������������� xxi

SIGACCESS Outstanding Contributions Keynote

● Challenges and Opportunities in Technology for Inclusion���������������� Article 1
Clayton Lewis

Paper Session 1: AR, VR and Games

● Access on Demand: Real-time, Multi-Modal Accessibility for the
Deaf and Hard-of-Hearing Based on Augmented-Reality��������������������� Article 2
Roshan Mathew, Brian Mak, Wendy Dannels

● VRBubble: Enhancing Peripheral Awareness of Avatars for

People with Visual Impairments in Social Virtual Reality�������������������� Article 3
Tiger Ji, Brianna R. Cochran, Yuhang Zhao

● “It’s Just Part of Me:” Understanding Avatar Diversity and

Self-Presentation of People with Disabilities in Social Virtual
Reality������������������������������������������������������������������������������������������������������� Article 4
Kexin Zhang, Elmira Deldari, Zhicong Lu, Yaxing Yao, Yuhang Zhao

● SoundVizVR: Sound Indicators for Accessible Sounds in Virtual

Reality for Deaf or Hard-of-Hearing Users��������������������������������������������� Article 5
Ziming Li, Shannon Connell, Wendy Dannels, Roshan Peiris

● Uncovering Visually Impaired Gamers’ Preferences for Spatial

Awareness Tools Within Video Games�������������������������������������������������� Article 6
Vishnu Nair, Shao-en Ma, Ricardo E. Gonzalez Penuela, Yicheng He,
Karen Lin, Mason Hayes, Hannah Huddleston, Matthew Donnelly,
Brian A. Smith

Paper Session 2: Representation and Inclusion

● Expressive Bodies – Engaging with Embodied Disability Cultures
for Collaborative Design Critiques��������������������������������������������������������� Article 7
Katta Spiel, Robin Angelini

vii
 ● Data Representativeness in Accessibility Datasets:
A Meta-Analysis��������������������������������������������������������������������������������������� Article 8
Rie Kamikubo, Lining Wang, Crystal Marte, Amnah Mahmood, Hernisa Kacorri

● Chronically Under-Addressed: Considerations for HCI Accessibility

Practice with Chronically Ill People������������������������������������������������������� Article 9
Kelly Mack, Emma J. McDonnell, Leah Findlater, Heather D. Evans

● Should I Say “Disabled People”, or “People with Disabilities”?

Language Preferences of Disabled People Between Identity- and
Person-First Language�������������������������������������������������������������������������� Article 10
Ather Sharif, Aedan L. McCall, Kianna R. Bolante

● Assistive or Artistic Technologies? Exploring the Connections

between Art, Disability and Wheelchair Use����������������������������������������Article 11
Giulia Barbareschi, Masa Inakage

● “Just Like Meeting in Person” – Examination of Interdependencies

in Dementia-Friendly Virtual Activities������������������������������������������������ Article 12
Elaine Czech, Paul Marshall, Oussama Metatla

Paper Session 3: Data for Modeling and Recognition

● Performing Qualitative Data Analysis as a Blind Researcher:
Challenges, Workarounds and Design Recommendations��������������� Article 13
O. Aishwarya

● Blind Users Accessing Their Training Images in Teachable

Object Recognizers������������������������������������������������������������������������������� Article 14
Jonggi Hong, Jaina Gandhi, Ernest Essuah Mensah, Farnaz Zamiri Zeraati,
Ebrima Haddy Jarjue, Kyungjun Lee, Hernisa Kacorri

● Challenging and Improving Current Evaluation Methods for

Colour Identification Aids��������������������������������������������������������������������� Article 15
Connor Geddes, David R. Flatla

● ASL Wiki: An Exploratory Interface for Crowdsourcing ASL

Translations�������������������������������������������������������������������������������������������� Article 16
Abraham Glasser, Fyodor Minakov, Danielle Bragg

viii
Paper Session 4: Composition in Music, Programming and Design
● Empowering Blind Musicians to Compose and Notate Music
with SoundCells������������������������������������������������������������������������������������� Article 17
William Payne, Fabiha Ahmed, Michael Zachor, Michael Gardell,
Isabel Huey, R. Luke DuBois, Amy Hurst

● Designing Gestures for Digital Musical Instruments: Gesture

Elicitation Study with Deaf and Hard of Hearing People�������������������� Article 18
Ryo Iijima, Akihisa Shitara, Yoichi Ochiai

● Accessible Blockly: An Accessible Block-Based Programming

Library for People with Visual Impairments���������������������������������������� Article 19
Aboubakar Mountapmbeme, Obianuju Okafor, Stephanie Ludi

● CodeWalk: Facilitating Shared Awareness in Mixed-Ability

Collaborative Software Development�������������������������������������������������� Article 20
Venkatesh Potluri, Maulishree Pandey, Andrew Begel, Michael Barnett,
Scott Reitherman

● Designing a Customizable Picture-Based Augmented Reality

Application for Therapists and Educational Professionals
Working in Autistic Contexts���������������������������������������������������������������� Article 21
Tooba Ahsen, Christina Yu, Amanda O’Brien, Ralf W. Schlosser,
Howard C. Shane, Dylan Oesch-Emmel, Eileen T. Crehan, Fahad Dogar

Paper Session 5: Communication

● State of the Art in AAC: A Systematic Review and Taxonomy���������� Article 22
Humphrey Curtis, Timothy Neate, Carlota Vazquez Gonzalez

● AAC with Automated Vocabulary from Photographs: Insights

from School and Speech-Language Therapy Settings���������������������� Article 23
Mauricio Fontana de Vargas, Jiamin Dai, Karyn Moffatt

● LaMPost: Design and Evaluation of an AI-Assisted Email Writing

Prototype for Adults with Dyslexia������������������������������������������������������� Article 24
Steven M. Goodman, Erin Buehler, Patrick Clary, Andy Coenen,
Aaron Donsbach, Tiffanie N. Horne, Michal Lahav, Robert MacDonald,
Rain Breaw Michaels, Ajit Narayanan, Mahima Pushkarna, Joel Riley,
Alex Santana, Lei Shi, Rachel Sweeney, Phil Weaver, Ann Yuan,
Meredith Ringel Morris

ix
 ● Exploring Smart Speaker User Experience for People Who
Stammer�������������������������������������������������������������������������������������������������� Article 25
Anna Bleakley, Daniel Rough, Abi Roper, Stephen Lindsay,
Martin Porcheron, Minha Lee, Stuart Nicholson, Benjamin Cowan,
Leigh Clark

● Beyond Subtitles: Captioning and Visualizing Non-Speech

Sounds to Improve Accessibility of User-Generated Videos������������ Article 26
Oliver Alonzo, Hijung Valentina Shin, Dingzeyu Li

Paper Session 6: Social Media and Media

● Nothing Micro About It: Examining Ableist Microaggressions
on Social Media�������������������������������������������������������������������������������������� Article 27
Sharon Heung, Mahika Phutane, Shiri Azenkot, Megh Marathe,
Aditya Vashistha

● Authoring Accessible Media Content on Social Networks���������������� Article 28

Letícia Seixas Pereira, José Coelho, André Rodrigues, João Guerreiro,
Tiago Guerreiro, Carlos Duarte

● Support in the Moment: Benefits and use of Video-Span Selection

and Search for Sign-Language Video Comprehension Among
ASL Learners������������������������������������������������������������������������������������������ Article 29
Saad Hassan, Akhter Al Amin, Caluã de Lacerda Pataca, Diego Navarro,
Alexis Gordon, Sooyeon Lee, Matt Huenerfauth

● A Dataset of Alt Texts from HCI Publications: Analyses and Uses

Towards Producing More Descriptive Alt Texts of Data
Visualizations in Scientific Papers������������������������������������������������������� Article 30
Sanjana Chintalapati, Jonathan Bragg, Lucy Lu Wang

Paper Session 7: Tactile and Haptics

● Low-Cost Tactile Coloring Page Fabrication on a Cutting
Machine: Assembly and User Experiences of Cardstock-Layered
Tangible Pictures����������������������������������������������������������������������������������� Article 31
Nicole E. Johnson, Tom Yeh, Ann Cunningham

x
 ● Animations at Your Fingertips: Using a Refreshable Tactile
Display to Convey Motion Graphics for People who are Blind or
have Low Vision������������������������������������������������������������������������������������� Article 32
Leona Holloway, Swamy Ananthanarayan, Matthew Butler,
Madhuka De Silva, Kirsten Ellis, Cagatay Goncu, Kate Stephens,
Kim Marriott

● Quantifying Touch: New Metrics for Characterizing What Happens

During a Touch��������������������������������������������������������������������������������������� Article 33
Junhan Kong, Mingyuan Zhong, James Fogarty, Jacob O. Wobbrock

● Creating 3D Printed Assistive Technology Through Design

Shortcuts: Leveraging Digital Fabrication Services to Incorporate
3D Printing into the Physical Therapy Classroom������������������������������ Article 34
Erin L. Higgins, William Easley, Karen Gordes, Amy Hurst, Foad Hamidi

● BentoMuseum: 3D and Layered Interactive Museum Map for

Blind Visitors������������������������������������������������������������������������������������������ Article 35
Xiyue Wang, Seita Kayukawa, Hironobu Takagi, Chieko Asakawa

Paper Session 8: Accessibility in Daily Living

● Depending on Independence – An Autoethnographic Account
of Daily Use of Assistive Technologies����������������������������������������������� Article 36
Felix Fussenegger, Katta Spiel

● “I Used To Carry A Wallet, Now I Just Need To Carry My Phone”:

Understanding Current Banking Practices and Challenges
Among Older Adults in China��������������������������������������������������������������� Article 37
Xiaofu Jin, Mingming Fan

● Mobile Phone Use by People with Mild to Moderate Dementia:

Uncovering Challenges and Identifying Opportunities���������������������� Article 38
Emma Dixon, Rain Michaels, Xiang Xiao, Yu Zhong, Patrick Clary,
Ajit Narayanan, Robin Brewer, Amanda Lazar

● Freedom to Choose: Understanding Input Modality Preferences

of People with Upper-Body Motor Impairments for Activities of
Daily Living��������������������������������������������������������������������������������������������� Article 39
Franklin Mingzhe Li, Michael Xieyang Liu, Yang Zhang, Patrick Carrington

xi
Paper Session 9: Safety, Rehabilitation, and Transportation
● It’s Enactment Time!: High-Fidelity Enactment Stage for Accessible
Automated Driving System Technology Research����������������������������� Article 40
Aaron Gluck, Hannah Solini, Julian Brinkley

● Where Are You Taking Me? Reflections from Observing

Ridesharing Use By People with Visual Impairments������������������������ Article 41
Earl W. Huff Jr., Robin N. Brewer, Julian Brinkley

● A Collaborative Approach to Support Medication Management

in Older Adults with Mild Cognitive Impairment Using
Conversational Assistants (CAs)��������������������������������������������������������� Article 42
Niharika Mathur, Kunal Dhodapkar, Tamara Zubatiy, Jiachen Li,
Brian D. Jones, Elizabeth D. Mynatt

● Designing Post-Trauma Self-Regulation Apps for People with

Intellectual and Developmental Disabilities���������������������������������������� Article 43
Krishna Venkatasubramanian, Tina-Marie Ranalli

Posters and Demos

● “I Should Feel Like I’m In Control”: Understanding Expectations,
Concerns, and Motivations for the Use of Autonomous Navigation
on Wheelchairs�������������������������������������������������������������������������������������� Article 44
JiWoong Jang, Yunzhi Li, Patrick Carrington

● “What Makes Sonification User-Friendly?” Exploring Usability and

User-Friendliness of Sonified Responses������������������������������������������� Article 45
Ather Sharif, Olivia H. Wang, Alida T. Muongchan

● “What’s Going on in Accessibility Research?” Frequencies

and Trends of Disability Categories and Research Domains
in Publications at ASSETS�������������������������������������������������������������������� Article 46
Ather Sharif, Ploypilin Pruekcharoen, Thrisha Ramesh, Ruoxi Shang,
Spencer Williams, Gary Hsieh

● A Participatory Design Approach to Explore Design Directions for

Enhancing Videoconferencing Experience for Non-Signing Deaf
and Hard of Hearing Users�������������������������������������������������������������������� Article 47
Yeon Soo Kim, Sunok Lee, Sangsu Lee

xii
● Co-designing a Bespoken Wearable Display for People with
Dissociative Identity Disorder�������������������������������������������������������������� Article 48
Patricia Piedade, Nikoletta Matsur, Catarina Rodrigues, Francisco Cecilio,
Afonso Marques, Rings of Saturn, Isabel Neto, Hugo Nicolau

● Co-designing the Automation of Theatre Touch Tours���������������������� Article 49

Alexandra Tzanidou, Sami Abosaleh, Vasilis Vlachokyriakos

● Creating Personas for Signing User Populations: An Ability-Based

Approach to User Modelling in HCI������������������������������������������������������ Article 50
Amelie Nolte, Karolin Lueneburg, Dieter Wallach, Nicole Jochems

● Designing a Data Visualization Dashboard for Pre-Screening

Hong Kong Students with Specific Learning Disabilities������������������ Article 51
Ka Yan Fung, Zikai Alex Wen, Haotian Li, Xingbo Wang, Shenghui Song,
Huamin Qu

● Digital Accessibility in Iran: An Investigation Focusing on Iran’s

National Policies on Accessibility and Disability Support����������������� Article 52
Laleh Nourian, Kristen Shinohara, Garreth W. Tigwell

● Exploring Accessibility Features and Plug-ins for Digital

Prototyping Tools���������������������������������������������������������������������������������� Article 53
Urvashi Kokate, Kristen Shinohara, Garreth W. Tigwell

● Does XR Introduce Experience Asymmetry in an

Intergenerational Setting?�������������������������������������������������������������������� Article 54
Vibhav Nanda, Hanuma Teja Maddali, Amanda Lazar

● Flexible Activity Tracking for Older Adults Using Mobility Aids — An

Exploratory Study on Automatically Identifying Movement
Modality�������������������������������������������������������������������������������������������������� Article 55
Dimitri Vargemidis, Kathrin Gerling, Luc Geurts, Vero Vanden Abeele

● Improving Image Accessibility by Combining Haptic and

Auditory Feedback��������������������������������������������������������������������������������� Article 56
Mallak Alkhathlan, ML Tlachac, Lane Harrison, Elke Rundensteiner

● Inter-rater Reliability of Command-Line Web Accessibility

Evaluation Tools������������������������������������������������������������������������������������ Article 57
Eryn Rachael Kelsey-Adkins, Robert Thompson

● Investigating How People with Disabilities Disclose Difficulties

on YouTube��������������������������������������������������������������������������������������������� Article 58
Shuo Niu, Jaime Garcia, Summayah Waseem, Li Liu

xiii
● ProAesthetics: Changing How We View Prosthetic Function����������� Article 59
Susanna Abler, Foad Hamidi

● Towards Visualization of Time–Series Ecological Momentary

Assessment (EMA) Data on Standalone Voice–First Virtual
Assistants����������������������������������������������������������������������������������������������� Article 60
Yichen Han, Christopher Bo Han, Chen Chen, Peng Wei Lee,
Michael Hogarth, Alison A. Moore, Nadir Weibel, Emilia Farcas

● Understanding and Improving Information Extraction From Online

Geospatial Data Visualizations for Screen-Reader Users ����������������� Article 61
Ather Sharif, Andrew M. Zhang, Anna Shih, Jacob O. Wobbrock,
Katharina Reinecke

● Understanding Design Preferences for Sensory-Sensitive

Earcons with Neurodivergent Individuals������������������������������������������� Article 62
Lauren Race, Kia El-Amin, Sarah Anoke, Andrew Hayward, Amber James,
Amy Hurst, Audrey Davis, Theresa Mershon

● Understanding How People with Visual Impairments

Take Selfies: Experiences and Challenges����������������������������������������� Article 63
Ricardo E. Gonzalez Penuela, Paul Vermette, Zihan Yan, Cheng Zhang,
Keith Vertanen, Shiri Azenkot

● Voice-Enabled Blockly: Usability Impressions of a Speech-Driven

Block-Based Programming System����������������������������������������������������� Article 64
Obianuju Okafor, Stephanie Ludi

● The Landscape of Accessibility Skill Set in the Software Industry

Positions������������������������������������������������������������������������������������������������� Article 65
Lilu Martin, Catherine M. Baker, Kristen Shinohara, Yasmine N. Elglaly

● How People who are Deaf, Deaf, and Hard of Hearing use
Technology in Creative Sound Activities��������������������������������������������� Article 66
Keita Ohshiro, Mark Cartwright

● Personalizable Alternative Mouse and Keyboard Interface for

People with Motor Disabilities of Neuromuscular Origin������������������� Article 67
Daniel Andreas, Hannah Six, Adna Bliek, Philipp Beckerle

● PigScape: An Embodied Video Game for Cognitive Peer-Training

of Impulse and Behavior Control in Children with ADHD������������������ Article 68
Yulia Gizatdinova, Vera Remizova, Antti Sand, Sumita Sharma,
Kati Rantanen, Terhi Helminen, Anneli Kylliäinen

xiv
● Serenity: Exploring Audio-Based Gaming for Arm-Hand
Rehabilitation after Stroke�������������������������������������������������������������������� Article 69
Yijing Jiang, Daniel Tetteroo

● UnlockedMaps: Visualizing Real-Time Accessibility of Urban

Rail Transit Using a Web-Based Map��������������������������������������������������� Article 70
Ather Sharif, Aneesha Ramesh, Trung-Anh H. Nguyen, Luna Chen,
Kent R. Zeng, Lanqing Hou, Xuhai Xu

● Website Builders Still Contribute to Inaccessible Web Design��������� Article 71

Athira Pillai, Kristen Shinohara, Garreth W. Tigwell

● An Accessible Smart Kitchen Cupboard��������������������������������������������� Article 72

Marios Gavaletakis, Asterios Leonidis, Nikolaos-Menelaos Stivaktakis,
Maria Korozi, Michalis Roulios, Constantine Stephanidis

● Social Access and Representation for Autistic Adult

Livestreamers����������������������������������������������������������������������������������������� Article 73
Terrance Mok, Anthony Tang, Adam McCrimmon, Lora Oehlberg

● Co-Designing Systems to Support Blind and Low Vision

Audio Description Writers��������������������������������������������������������������������� Article 74
Lucy Jiang, Richard Ladner

● College Students’ and Campus Counselors’ Attitudes Toward

Teletherapy and Adopting Virtual Reality (Preliminary Exploration)
for Counseling Services������������������������������������������������������������������������ Article 75
Vanny T. Chao, Roshan L. Peiris

● Context-Responsive ASL Recommendation for Parent-Child

Interaction����������������������������������������������������������������������������������������������� Article 76
Ekram Hossain, Merritt Cahoon, Yao Liu, Chigusa Kurumada, Zhen Bai

● Designing A Game for Pre-Screening Students with Specific

Learning Disabilities in Chinese����������������������������������������������������������� Article 77
Ka Yan Fung, SIN Kuen Fung, Zikai Alex Wen, Lik-Hang Lee,
Shenghui Song, Huamin Qu

● Exploring Motor-Impaired Programmers’ Use of Speech

Recognition�������������������������������������������������������������������������������������������� Article 78
Sadia Nowrin, Patricia Ordóñez, Keith Vertanen

xv
 ● Investigating Accessibility Challenges and Opportunities for
Users with Low Vision Disabilities in Customer-to-Customer
(C2C) Marketplaces�������������������������������������������������������������������������������� Article 79
Bektur Ryskeldiev, Kotaro Hara, Mariko Kobayashi, Koki Kusano

● Object Recognition from Two-Dimensional Tactile Graphics: What

Factors Lead to Successful Identification Through Touch?������������� Article 80
Anchal Sharma, Srinivasan V., P.V.M. Rao

● Overcoming Barriers to an Accessible e-Learning Ecosystem for

People on the Autism Spectrum: A Preliminary Design��������������������� Article 81
Yussy Chinchay, Javier Gomez, Germán Montoro

● Scaling Crowd+AI Sidewalk Accessibility Assessments: Initial

Experiments Examining Label Quality and Cross-City Training
on Performance�������������������������������������������������������������������������������������� Article 82
Michael Duan, Shosuke Kiami, Logan Milandin, Johnson Kuang,
Michael Saugstad, Maryam Hosseini, Jon E. Froehlich

● SpineCurer: An Inertial Measurement Unit Based Scoliosis

Training System������������������������������������������������������������������������������������� Article 83
Eryuan Mai, Dahua Hu, Jiaming Li, Zhuo Yang

● Systematic Literature Review on Making and Accessibility�������������� Article 84

Saquib Sarwar, David Wilson

● Understanding ASL Learners’ Preferences for a Sign Language

Recording and Automatic Feedback System to Support
Self-Study����������������������������������������������������������������������������������������������� Article 85
Saad Hassan, Sooyeon Lee, Dimitris Metaxas, Carol Neidle,
Matt Huenerfauth

● Vibrotactile Navigation for Visually Impaired People������������������������� Article 86

Stephan Huber, Anastasia Alieva, Aaron Lutz

Student Research Competition Abstracts

● Challenges and Opportunities in Creating An Accessible Web
Application for Learning Organic Chemistry�������������������������������������� Article 87
Allyson Yu

● FootUI: Designing and Detecting Foot Gestures to Assist People

with Upper Body Motor Impairments to Use Smartphones
on the Bed���������������������������������������������������������������������������������������������� Article 88
Xiaozhu Hu, Jiting Wang, Weiwei Gao, Yongquan Hu

xvi
 ● Investigating Sign Language Interpreter Rendering and Guiding
Methods in Virtual Reality 360-Degree Content���������������������������������� Article 89
Craig Anderton

Doctoral Consortium Abstracts

● Accessible PDFs: Applying Artificial Intelligence for Automated
Remediation of STEM PDFs������������������������������������������������������������������ Article 90
Felix M. Schmitt-Koopmann, Prof. Dr. Elaine M. Huang,
Prof. Dr. Alireza Darvishy

● Accessible Communication and Materials in Higher Education������� Article 91

Kelly Mack

● Exploring Collective Medical Knowledge and Tensions with

Medical Systems in Online ADHD Communities��������������������������������� Article 92
Tessa Eagle

● Understanding the Role of Socio-Technical Infrastructures on the

Organization of Access for the Mixed-Ability Collaborators������������� Article 93
Zeynep Şölen Yıldız

● Supporting Physical Activity in Later Life: Perspectives from

Older Adults�������������������������������������������������������������������������������������������� Article 94
Muhe Yang

● Applying Technology in a Hybrid-Fashion to Create

Dementia-Inclusive Community Spaces���������������������������������������������� Article 95
Elaine Czech

● Learning Music Blind: Understanding the Application of

Technology to Support BLV Music Learning��������������������������������������� Article 96
Leon Lu

● Evaluating Haptic Technology in Accessibility of Digital Audio

Workstations for Visually Impaired Creatives������������������������������������� Article 97
Christina Karpodini

● Lessons Learned from Designing, Deploying and Testing an

Accessible BLE Beacon-Based Wayfinding System in a Multi-Floor
Indoor Environment������������������������������������������������������������������������������� Article 98
Ajay Abraham

xvii
 ● Socially Connecting Adults with Intellectual Disabilities Through
Inclusive Co-Design of Tangible and Visual Technology������������������� Article 99
Manesha Andradi

● Understanding Social and Environmental Factors to Enable

Collective Access Approaches to the Design of Captioning
Technology������������������������������������������������������������������������������������������� Article 100
Emma J. McDonnell

● Improving Web and Mobile Accessibility Resources for Iranian

Designers���������������������������������������������������������������������������������������������� Article 101
Laleh Nourian

Workshop Abstracts
● The Future of Urban Accessibility for People with Disabilities:
Data Collection, Analytics, Policy, and Tools������������������������������������ Article 102
Jon E. Froehlich, Yochai Eisenberg, Maryam Hosseini, Fabio Miranda,
Marc Adams, Anat Caspi, Holger Dieterich, Heather Feldner, Aldo Gonzalez,
Claudina de Gyves, Joy Hammel, Reuben Kirkham, Melanie Kneisel,
Delphine Labbé, Steve J. Mooney, Victor Pineda, Cláudia Fonseca Pinhão,
Ana Rodríguez, Manaswi Saha, Michael Saugstad, Judy Shanley, Ather Sharif,
Qing Shen, Cláudio T. Silva, Maarten Sukel, Eric K. Tokuda, Sebastian Felix Zappe,
Anna Zivarts

● A Workshop on Disability Inclusive Remote Co-Design������������������ Article 103

Maryam Bandukda, Giulia Barbareschi, Aneesha Singh, Dhruv Jain,
Maitraye Das, Tamanna Motahar, Jason Wiese, Lynn Cockburn,
Amit Prakash, David Frohlich, Catherine Holloway

● Including Accessibility in Computer Science Education����������������� Article 104

Catherine M. Baker, Yasmine N. Elglaly, Anne Spencer Ross,
Kristen Shinohara

● Multidisciplinary Perspectives on Designing Accessible Systems

for Users with Multiple Impairments: Grand Challenges and
Opportunities for Future Research���������������������������������������������������� Article 105
Arthur Theil, Chris Creed, Mohammed Shaqura, Nasrine Olson,
Raymond Holt, Sayan Sarcar, Stuart Murray

● Designing with and for People with Intellectual Disabilities����������� Article 106
Leandro Soares Guedes, Ryan Colin Gibson, Kirsten Ellis, Laurianne Sitbon,
Monica Landoni

xviii
 ASSETS 2022 Conference Organization

General Chair: Jon E. Froehlich (University of Washington, USA)

Technical Program Chairs: Kristen Shinohara (Rochester Institute of Technology, USA)

Stephanie Ludi (University of North Texas, USA)

Proceedings Chair: Raja Kushalnagar (Gallaudet University, USA)

Sergio Mascetti (University of Milan, Italy)

Treasurer/Registration Kyle Rector (University of Iowa, USA)

Chair: Martez Mott (Microsoft Research, USA)

Hybrid Experiences Chair: Christian Vogler (Gallaudet University, USA)

Local Chairs: Alexandros Pino (National and Kapodistrian University of

Athens, Greece)
Georgios Kouroupetroglou (National and Kapodistrian
University of Athens, Greece)

Virtual Chairs: Emma McDonnell (University of Washington, USA)

Kelly Mack (University of Washington, USA)
Benjamin Tannert (City University of Applied Science
Bremen, Germany)
Laurianne Sitbon (Queensland University of Technology,
Australia)

Workshop Chairs: Kyle Montague (Northumbria University, England)

Sowmya Somanath (University of Victoria, Canada)

Posters and Demos Chairs: Hugo Nicolau (University of Lisbon, Portugal)

Taslima Akter (Indiana University - Bloomington, USA)

Doctoral Consortium Chairs: Katta Spiel (Vienna University of Technology, Austria)

Aqueasha Martin-Hammond (Indiana University - Purdue
University Indianapolis, USA)

Student Research Mingming Fan (The Hong Kong University of Science and
Competition: Technology, China)
Roshan Peiris (Rochester Institute of Technology, USA)

Experience Reports Chairs: Garreth Tigwell (Rochester Institute of Technology, USA)

Kathryn (Kate) Ringland (University of California - Santa
Cruz, USA)

Mentoring Chair: Sayan Sarcar (Birmingham City University, England)

Kirsten Ellis (Monash University, Australia)

Publicity Chairs: Alexa Siu (Adobe, USA)

Arthur Theil (Birmingham City University, England)

Web and Graphics Design Liang He (University of Washington, USA)

xix
xix
 Chairs Junhan (Judy) Kong (University of Washington, USA)
Jaylin Herskovitz (University of Michigan, USA)
Jason Wu (Carnegie Mellon University, USA)

Student Volunteer Chairs: Franklin Mingzhe Li (Carnegie Mellon University, USA)

Maryam Banduka (University College London, England)

Local Accessibility Chairs: Paraskevi Riga (National and Kapodistrian University of

Athens, Greece)
Ariadni Velissaropoulou (National and Kapodistrian
University of Athens, Greece)

Accessibility Chairs: J. Bern Jordan (University of Maryland, USA)

Lei Shi (Google, USA)

ACM Partnerships Chair: Stacy Branham (University of California - Irvine, USA)

Global Outreach Chairs: Uran Oh (Ewha Womans University, Korea)

Yuhang Zhao (University of Wisconsin-Madison, USA)
Manohar Swaminathan (Microsoft Research, India)
Pin Sym Foong National University of Singapore,
Singapore)

Education Outreach Chairs: Anne Spencer Ross (Bucknell University, USA)

Catie Baker (Creighton University, USA)

Equity and Belonging Chair: LouAnne Boyd (Chapman University, USA)

John Guerra Gomez (Northeastern University - Bay Area,
USA)

TACCESS Special Issue Maria Wolters (University of Edinburgh, Scotland)

Chair:

Best Artifact Chair: Dragan Ahmetovic (University of Milan, Italy)

xx
xx
 ASSETS 2022 Sponsors & Supporters

Sponsor:

Champion:

Gold Level:

xxi
xxi
 Gold Level
(continues):

Silver level:

Bronze level:

xxii
xxii
 Doctoral
Consortium
Sponsors:

NSF Award ID 2228013

In-kind Sponsors

xxiii
xxiii
 Challenges and opportunities in technology for inclusion
Clayton Lewis
University of Colorado, Boulder, CO, USA
clayton@colorado.edu
ABSTRACT
What will we be working on in the coming decade? This talk will
consider a wide range of possibilities, ranging from predictable
impacts of new technologies to difcult matters of perspective and
policy that infuence the development and deployment of tech-
nology, to quixotic ambitions. The opinions will be those of the
presenter, if even he endorses them.
CCS CONCEPTS
• Social and professional topics; • People with disabilities;
KEYWORDS
Inclusion, accessible technology
ACM Reference Format:
Clayton Lewis. 2022. Challenges and opportunities in technology for inclu-
sion. In The 24th International ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM,
New York, NY, USA, 1 page. https://doi.org/10.1145/3517428.3566096
1 BIO
Clayton Lewis is Emeritus Professor of Computer Science at the
University of Colorado Boulder. Lewis served previously as Co-
Director for Technology for the Coleman Institute for Cognitive
Disabilities, and Fellow of the Institute of Cognitive Science, at CU,
and as technology advisor to the director of the National Institute
for Disability and Rehabilitation Research, US Department of Edu-
cation. He is well known for his research on evaluation methods in
user interface design. Two methods to which he and his colleagues
have contributed, the thinking aloud method and the cognitive
walkthrough, are in regular use in software development organi-
zations around the world. He has also contributed to cognitive
assistive technology, to programming language design, to educa-
tional technology, and to cognitive theory in causal attribution and
learning.
Before joining the University of Colorado, Lewis was Manager of
Human Factors at IBM’s Watson Research Center, where he was a
member of the research staf from 1970 to 1973 and 1979 to 1984. He
holds degrees from Princeton, MIT, and the University of Michigan.
He has been honored by appointment to the ACM SIGCHI Academy,
by the SIGCHI Social Impact Award, and by the Strache Leadership
Award (CSUN Assistive Technology Conference).
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3566096
 Access on Demand: Real-time, Multi-modal Accessibility for the
Deaf and Hard-of-Hearing based on Augmented Reality
Roshan Mathew
School of Information, Rochester College Of Liberal Arts, Rochester
Institute of Technology, Rochester, Institute of Technology, Rochester,
New York, United States New York, United States
rm1299@rit.edu bm1531@rit.edu
ABSTRACT
In this experience report, two deaf researchers with varying ex-
pertise, communication preferences, and technological skills doc-
ument their experiences using Access on Demand (AoD), an Aug-
mented Reality (AR) based accessibility application that provides
on-demand real-time captioning and sign language interpretation
services using the Vuzix Blade AR smart glasses. The researchers re-
port their observations regarding using remote real-time American
Sign Language (ASL) interpreting, captioning, and auto-captions
ofered by the AoD platform. The authors discuss the benefts and
limitations of using AoD as an assistive technology device and how
it would beneft the deaf community from the perspective of Deaf
and Hard-of-Hearing (DHH) users.
CCS CONCEPTS
• Human-centered computing → Accessibility; Accessibility
technologies; Accessibility; Accessibility design and evaluation
methods.
KEYWORDS
Augmented Reality, Smart glasses, DHH
ACM Reference Format:
Roshan Mathew, Brian Mak, and Wendy Dannels. 2022. Access on Demand:
Real-time, Multi-modal Accessibility for the Deaf and Hard-of-Hearing
based on Augmented Reality. In The 24th International ACM SIGACCESS
Conference on Computers and Accessibility (ASSETS ’22), October 23–26, 2022,
Athens, Greece. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/
3517428.3551352
1 INTRODUCTION
There is a great need for the Deaf and Hard-of-Hearing (DHH) com-
munity to receive access services whenever possible to understand
others who do not know or use American Sign Language (ASL) and
only use spoken English. However, it is not always possible to avail
of such services according to the needs and convenience of DHH
individuals because there is insufcient existing infrastructure and
This work is licensed under a Creative Commons Attribution-NonCommercial
International 4.0 License.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3551352
Brian Mak Wendy Dannels
NTID Center on Culture and
Language, Rochester Institute of
Technology, Rochester, New York,
United States
w.dannels@rit.edu
processes to support the demand. Providing accessibility accom-
modations is either too costly, or there are too few interpreters or
captioners available [1].
An example would be the case of a DHH student sitting in a
classroom far away from the lecturer, unable to see what the inter-
preter is signing clearly. If the DHH student prefers captioning, the
captioning will either be displayed on a projector display up front
or on their laptop or phone. This situation is exacerbated when the
lecturer is always on the move or demonstrating something, and
the DHH student has to switch their focus between the lecturer
and the interpreter/captions. It would be benefcial for the DHH
student to see the interpreter closer or view the captions while also
being able to focus on the lecturer. Undoubtedly, not all classes will
or can always provide interpreting or captioning support, and the
burden of getting that accessibility support rests on the DHH per-
son. Moreover, there is always a possibility that the interpreter or
captioner they requested does not show up. On-demand accessibil-
ity services using Augmented Reality (AR) smart glasses eliminate
this inconvenience and help the DHH user get the accommodations
they need reliably in situations where they are not provided one.
Augmented Reality is an interactive experience that provides a
digital modifcation of the real world. There is a huge potential for
AR smart glasses to be immensely useful for the DHH community
in terms of accessibility to view captions or interpreting. There is
a study that looked at the use of smart glasses to improve lecture
comprehension in the classroom for DHH students, initially using
Google Glass and then fully using EPSON Moverio BT-200 [5].
Another research evaluated a custom-made AR prototype with four
modes that included detecting environmental sounds and emotion
indicators, but their prototype was not tested by a DHH user [7].
In this experience report, we will discuss our evaluation of using
the $999 Vuzix Blade smart glasses with a university-made software
called Access on Demand (AoD) installed to enable interpreting
and captioning in diferent real-world situations such as watching a
niche stand-up comedian show in person or listening in to a speech
during a wedding where the organizers have not considered provid-
ing access services. This experience report includes observations
made by two DHH researchers with unique perspectives and vary-
ing experiences while using the Vuzix Blade AR smart glasses with
AoD installed and shows that a platform/app that ofers on-demand
multi-modal accessibility tools should be able to meet diferent
users’ needs by providing options to select access services based
on their preferences (interpreting or captioning).
ASSETS ’22, October 23–26, 2022, Athens, Greece
2 BACKGROUND
The next subsections provide an overview of the authors’ back-
grounds, information on the processes and tools used, and the
signifcant features ofered by the tool and the platform.
2.1 Team Members
Mak: It should be noted that I am a hard-of-hearing individual
whose main communication method is ASL, with some halting
ability of spoken English here and there. I wear a set of Bluetooth-
capable hearing aids, but I am not profoundly deaf. I was an alpha
tester with the Vuzix Blade when AoD was still being developed
two years ago. I am an undergraduate journalism major. I only have
surface knowledge of Augmented Reality, Virtual Reality, Mixed
Reality, and head-mounted displays. Now I am testing out the AoD
app already installed in the smart glasses.
Mathew: I am deaf and use a cochlear implant (CI) on one ear and
a hearing aid (HA) on the other. I am late-deafened, can speak, and
have a good command of the English language. I attended main-
stream schools and was exposed to sign language only relatively
recently. Therefore, I rely on real-time captioning using human
captioners or automatic speech recognition (ASR) apps for daily
communication accessibility. As a graduate student majoring in
a computing program, I have prior experience using Augmented
Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) glasses
and head-mounted displays (HMDs).
Dannels: I have profound hearing loss, do not use any assistive
technology, and sign ASL. I am the director of XR Accessibility
Solutions Laboratory (XR-ASL), homing in the Rochester Institute
of Technology/National Technical Institute for the Deaf’s Research
Center on Culture and Language (RIT/NTID CCL) [9]. For the
month of June 2022, I am mentoring two university students in
their summer experience as research assistants. I provided them
with the resources allowing them to write their refective paper.
2.2 Process
The XR-ASL team has developed an app that enables remote real-
time captioning and ASL interpreting to be displayed on smart
glasses. This innovative solution enables deaf individuals to look
directly at people, especially during demonstrations, to review in-
structions and acquire important information instead of looking at
diferent individuals and/or screens. Wendy invited two summer
student participants to experiment with smart glasses. As a part of
research and development, the Access on Demand platform and app
were developed at the university to serve as a web portal for several
purposes. Diferent users will have the ability to sign in and utilize
the features. Sign-in will allow users to use diferent tools developed
for their needs. Diferent sets of users are: deaf/hard-of-hearing,
captioners, interpreters, and presenters. This paper focuses on DHH
users’ perspectives only. Selecting two students with completely
diferent backgrounds and experiences will provide the readers
with two polarizing perspectives.
2.3 Tools
Vuzix Blade is currently the only United States made commercially
available pair of AR smart glasses that are not cable linked to a
smartphone, not tethered to a battery pack, and gives access to a
Roshan Mathew et al.
Software Development Kit (SDK), which allows the developer to
program and code an app for its use [11]. During their experience
using Vuzix Blade, DHH users were not given the battery pack.
2.4 Access on Demand
This section describes the main features and functions of the Access
on Demand (AoD) application.
2.4.1 Initial Set up. After you power on your Blade smart glasses,
swipe through the app list and tap to launch into the Access on
Demand app, then swipe down to enable QR scanning mode, and
aim your smart glasses at the specifc QR code on either the session
page on the AoD website or in printed paper form. You will know
that you are in once you see the session ID on the screen. All three
modes below follow this initial setup.
2.4.2 Interpreting. After joining the session, you will see the in-
terpreter on screen in the smart glasses when they get in the same
session as you. If not, then you will get a message saying, ‘No in-
terpreter is available right now.’ Now the next step is getting the
interpreter to hear the outside world for them to interpret for you
inside the glasses. AoD does not use the embedded microphone
inside the smart glasses because it does not capture the outside
audio well. Also, the stereo speakers embedded in the Vuzix Blade
are not quite loud enough for everyday situations. Thus, you need
to use your phone or any other external device with more powerful
microphones and speakers. So, you need to log in to the AoD web-
site from a phone or another external device to access ‘presenter’
mode. This mode enables you to use the microphone and camera
on another device so the interpreter can listen to what the other
person you are talking or listening to is saying so they can interpret.
Finally, you enter the same session ID as the one you scanned the
QR code for.
2.4.3 Real-time Captioning. Access on Demand has two modes for
captioning: real-time captioning with a remote human captioner
and auto-captioning using automatic speech recognition. After
joining a session as described under the initial setup, a one-fnger
forward swipe gesture will toggle the interpreting portal with the
captioning portal. To relay the audio to the captioner, the DHH user
will have to log in to the AoD website using their phone or laptop to
access the presenter mode. The presenter mode enables the camera
and microphone on the user’s device to let the captioner listen
to the conversation and view the event. AoD uses C-Print [10] to
provide real-time captioning services, a speech-to-text technology
service developed at RIT/NTID. The captioner assigned for the
specifc event connects their copy of C-Print software with the
AoD platform and joins the same session. Simultaneously, they also
need to log in to the AoD website to access the audio and video
streams from the DHH user’s end. The captioner can then start
listening to the conversation and provide captioning, which will
be displayed through the Blade smart glasses. Whenever there is a
pause in the conversation, and the captioner is not typing anything,
the Blade will display a status message, "Captioner on standby."
2.4.4 Auto-captioning. The process for using auto-captions is quite
similar to that of real-time captions on the Blade. The only diference
is that instead of using the presenter mode on a phone or laptop,
Access on Demand: Real-time, Multi-modal Accessibility for the Deaf and Hard-of-Hearing based on Augmented Reality
the DHH user needs to use diferent login credentials to access
the user interface for auto-captions on the AoD website. This page
contains a display section for captions, a feld to enter the session
ID, and buttons to enable the microphone and auto-captions. Once
the microphone and auto-captions are enabled, the captions will
display both on the phone (or laptop) and the Blade. AoD uses
Web Speech for automatic speech recognition. Auto-captioning
was tested using the Blade and a laptop as there were technical
glitches with using a phone.
2.4.5 Customization Features. The AoD application ofers the fol-
lowing customization features through a discoverable horizontal
menu bar: Brightness, Font Size, Font Style, and Font color. The
QR Code Scanner described earlier is also accessed using the same
menu. The brightness can be adjusted to three diferent levels. There
are three diferent font sizes and four diferent font styles avail-
able for personalization, but the application does not display the
names of the fonts and font styles. Font colors can be customized
using six colors – white, yellow, green, blue, red, and purple. A
one-fnger tap gesture is used to access a customization feature,
followed by another one-fnger tap gesture to cycle through the
available customization options.
3 DISCUSSION
In this section, the authors describe their experiences and observa-
tions of using AoD based on their individual communication needs
and preferences for communication accessibility.
3.1 Real-time Interpreting - Mak – Journalist
3.1.1 Access on Demand. My experience with using the software
inside the smart glasses is a mixed bag though I applauded that
the software is developed by a tiny team consisting only of Deaf
developers who are students working part-time while studying at
RIT. I was assigned to test out the live interpreting feature from the
perspective of a DHH client. Overall, the primary purpose of seeing
live interpreting in the smart glasses worked though I encountered
some usability problems, especially with presenter mode. It should
be noted that the presenter mode is a recently added feature to the
AoD platform. The purpose of this mode is for the interpreter to
be able to hear the external audio with your phone’s microphone
and enable them to do their job in real time as you observe inside
the video feed. Thus, there are a few bugs to be expected, although
the main function of this mode is to use an external microphone
such as a phone that works well. The latency I experienced while
feld-testing the live interpretation is close to zero. I can clearly see
the interpreter’s movements smoothly and understand them; at the
same time, they can hear the external audio captured in presenter
mode.
Limitations: From the perspective of a DHH user, however, it
does not necessarily improve the communication right now as the
presenter mode does not show the interpreter, and it also cannot
emit audio for others to hear the interpreter. It only improves and
enhances the DHH user’s ability to understand and learn without
moving their head. Also, you limit yourself to signing only with one
hand while the other hand is holding your phone in presenter mode.
Depending on the situation or person, you might or might not give
your phone to the other person you are talking to for a chance of
ASSETS ’22, October 23–26, 2022, Athens, Greece
them stealing it away from you, thus the need to hold on to your
phone in presenter mode while it is unlocked. The presenter mode’s
use of the camera is not helpful as it uses only the selfe camera of
your phone; thus, you see only yourself, and also there is no bottom
video feed to show the interpreter for the other person you are
talking to for them to understand what they are speaking to from
the phone. You cannot use your ASL skill when you give your phone
away for the other person to speak into your phone’s microphone,
and there will be moments when you cannot place your phone
vertically and at some distance away from you for the interpreter
to see what you are signing. Doing two-way communication is
impossible unless you can put your phone down vertically or hold
your phone in a way that lets the other person speak into the
microphone. Thus, for now, the main purpose for live interpreting
in AoD is for you to be more attentive to the presentation given by
someone else without getting distracted or having to move your
gaze back and forth, as Miller et al. demonstrated in their research
back in 2017 [5].
3.1.2 Vuzix Blade. The Vuzix Blade is an exciting device to test
AoD. Currently, it is not an electronic device aimed at consumers.
Rather it is aimed toward business enterprises such as health-
care or manufacturing. I am surprised at its bulkiness, though
I can understand why the design is the way it is based on the
hardware features it has. It has the following features: noise-
canceling microphones, a touchpad, High Defnition (HD) cam-
era (8MP), and microSD storage. It is a device that can be turned
on and of and charged via USB. There is a slight learning curve
in navigating the user interface with one or two fngers with
taps and swipes. I learned rather quickly, but the touchpad on
the right temple side is very sensitive, so you will make many
accidental inputs while using the built-in or custom-installed
apps.
Problems: The issues I have in using the Vuzix Blade are that
it has a short battery life of about 1 hour because AoD requires
a lot of live streaming, does get sustainably hot on both temple
sides, is Wi-Fi dependent, and requires a nose clip if a person is
already wearing prescription glasses (note there is a process where
you can get and install the exact prescription lens from the Vuzix
Blade. I did not have the luxury to do it). Also, what I am seeing
in the smart glasses in terms of ‘augmented reality’ disappointed
me, even if it is still in development. Watching live interpretation
with the Vuzix Blade really consisted of a small 2D video feed on
the right side of the smart glasses with no setting to change it
except by brightness. My original expectation of the AR enabled by
the Vuzix Blade was that it could provide a level of 3D interaction
in the real world that is adjusted on the fy, as shown by Yi-Hao
Peng et al. with their SpeechBubbles prototype [6]. However, the
AR elements here can only be seen in a static feed and cannot be
moved.
Conclusion: Overall, even though personally it is cool to experi-
ment with this device and see how it works with Access on Demand,
in real-world usage, I do not think it can be reliable when you can-
not communicate with hearing people while you are dealing with
the short battery life and connectivity issues. I do not think DHH
people should be using the Vuzix Blade monocular version for live
ASSETS ’22, October 23–26, 2022, Athens, Greece
interpretation specifcally because it does not provide a diferent ex-
perience compared to seeing interpreters on Zoom on a fat laptop
screen nor provides additional benefts other than it being portable.
There still is a lot of coordination involved in getting an interpreter
on board, and the biggest factor that can negatively afect this expe-
rience is Wi-Fi connectivity. Seeing an interpreter on your AR smart
glasses is a novel idea, but in the near future, it still needs some
improvements, especially on the hardware and software categories,
such as making it 5G capable or able to use mobile data via your
SIM card.
3.2 Real-time Captioning and Auto Captioning -
Mathew – Geek
3.2.1 Method. Both captioning modes (real-time captioning and
auto-captioning) were tested in separate scenarios where I use
captioning daily for communication access. Real-time captioning
was tested at one of the campus dining places to interact with
the staf while buying lunch. Real-time captioning was preferred
for this scenario because dining locations are usually noisy, and
auto-captioning does not typically perform well in such situations
because of the background noise. I have also used real-time cap-
tioning previously during trials at a local planetarium to watch
their shows. Auto-captioning was tested in an ofce environment
during a conversation with another individual. Auto-captioning
was chosen for this scenario as it was a relatively quieter environ-
ment where automatic speech recognition would work well. I took
detailed notes documenting these experiences on the same day of
the testing sessions.
3.2.2 Findings. Overall, real-time captioning and auto-captions
work as intended. Real-time captions ofer the advantage of having
the captions provided by a human captioner. Human captioners of-
fer the beneft of better accuracy, the ability to understand diferent
speech accents, and the capability to comprehend speech better in
noisy environments. On the other hand, auto- captions ofer more
privacy when you do not prefer to have a third-party listen to your
conversation and when there is not enough turn-around time to
request and schedule a captioner. I would prefer one over the other
depending on my specifc accessibility needs and preferences. Fol-
lowing are some key observations regarding real-time captioning
and auto-captions through the AoD platform.
Glanceability: One of the signifcant benefts of using AoD is that
I do not have to constantly switch my focus between the speaker and
another screen displaying captions, such as on a phone, laptop, or
tablet. I rely on speech reading while communicating, and viewing
the captions on the Blade allowed me to maintain my gaze on the
speaker’s face to access non-verbal cues and facial expressions
that I would have otherwise missed. My conversation partners
also pointed out that they thought the conversation was quite
natural as I looked directly at them instead of my phone or laptop.
Glanceability would be particularly benefcial in classrooms during
lectures where instructors speak while referring to a presentation
or demonstration, in a doctor’s ofce when they are explaining a
diagnosis while sharing a report, in a museum viewing an exhibit
while listening to an audio description, or while engaging in a
hands-on activity listening to verbal instructions.
Roshan Mathew et al.
Scrollability: The captions are displayed in multiple lines within
the display. The number of lines and characters displayed will
depend on the font size and the font style chosen. When testing
using real-time captions, the contents did not automatically scroll
to the latest input once it flled up the display container, and there
was no gesture defned to let me manually scroll to the latest text.
Consequently, the captioner had to continuously delete the previous
content as soon as it reached the last line on the display so I could
view the latest transcribed text. This issue has since then been
rectifed. However, with auto-captions, this was not a concern as I
could view the latest text as the last line within the display.
Positioning: The captions are displayed to the left of the Blade’s
right lens such that the captions are overlayed in the center of
the three-dimensional space in binocular vision. When I look di-
rectly at the speaker, the captions are right in front of their face.
While Blade’s native device settings allow you to move the dis-
play vertically, it does not ofer the option to move it horizontally
or diagonally. I prefer to have the captions displayed next to the
speaker, on either side, so the ability to position the captions in 3D
space, depending on my preferences, would have been an excellent
functionality. As a workaround, I looked a few degrees to the left
or right so that the captions did not block my view of the speaker’s
face.
Captioning Fidelity and Accuracy: With real-time captioning, the
captioning fdelity and accuracy depend on the captioner. Some
captioners may transcribe speech exactly, while others may choose
to provide a summary of the conversation. With auto-captioning,
the speech is transcribed verbatim, and the accuracy depends on
the clarity of speech, accents, background noise, etc. No formal
tests were conducted to measure the Word Error Rates (WER) for
real-time captioning or auto-captioning. Auto-captioning worked
almost in real-time, but a slight delay was noticed occasionally,
which might have been due to an unstable internet connection.
Auto-captions failed to generate captions when the speaker’s rate
of speech was faster than usual and also when the speaker was
more than six feet away from the device with the microphone.
Auto-captioning was found to be accurate with uncommon words.
Contextual Information: For real-time captions, the captioner
could provide contextual information such as speaker names and
other background noises. However, for auto-captions, additional
contextual information is not available. AoD does not ofer speaker
identifcation or sound recognition. The captions are displayed
continuously without line breaks or punctuations, making it quite
challenging to use auto-captions for group conversations.
Learnability of Gestures: It takes some efort to learn all the pre-
defned gestures for the Blade, so until I familiarized myself with
them, it was easy to use the wrong gesture and perform an undesired
action. The Blade also does not ofer many customizable gestures.
As a result, some of the custom gestures for AoD are not intuitive.
For example, I need to use a one-fnger swipe down gesture to access
the horizontal menu at the bottom. However, this lacks external
consistency with other applications where such a gesture is used
to pull down a menu at the top of the screen.
Social Considerations: The Blade is bulky and noticeable com-
pared to regular eyeglasses, making the wearer stand out, especially
indoors. In addition, the status lights and the AR display’s backlight
make others curious if they are being recorded. While testing the
Access on Demand: Real-time, Multi-modal Accessibility for the Deaf and Hard-of-Hearing based on Augmented Reality
AoD in public spaces such as dining locations, I was asked whether
I was capturing others on video or audio and had to reassure them
that the platform does not save any data.
Wearability: As stated earlier, I wear a behind-the-ear (BTE) CI
and a BTE HA. While the Blade’s frame is compact when compared
to other AR HMDs, its frame is still quite thick and is inconvenient
to wear along with my CI and HA. Often, tilting or turning my
head caused the Blade to knock my CI and HA out of position, so I
had to keep adjusting them so that everything stayed in its place
constantly. The Blade also heats up with constant use, although it
does not get too hot. I also noticed some eye strain with continual
use for over an hour.
4 BENEFITS, LIMITATIONS, AND FUTURE
DIRECTIONS
The AoD prototype is currently designed as an accessibility tool
for situations where there will be a primary presenter, such as
attending a lecture in a large classroom, a presentation or a talk
at an event, or watching live shows at a museum. It enhances
learning and comprehension as the DHH users do not have to
constantly turn their heads between the speaker and the interpreter
or a captioning screen (on a phone or tablet) that would derail
their train of thought. However, due to hardware and software
limitations, the use of AoD in situations beyond scenarios described
in this paper is currently restricted, both for DHH users who prefer
interpreting or captioning.
Mathew and Mak observed that having a microphone with better
capabilities on the Blade would help avoid using a microphone on
personal devices such as phones or laptops. In addition, an alterna-
tive method of capturing and relaying the DHH users’ signs to the
interpreters along with better stereo speakers on the Blade would
help interpreters voice for the DHH user, resulting in possibilities
for two-way communication. It should be noted that Mak could not
really engage in a two-way conversation due to the limitations of
holding the phone, as described earlier, while Mathew was able to
use AoD during a two-way conversation with another individual
because he could voice for himself. Should a DHH person who
prefers captioning chooses not to voice for themselves, they would
need to write on a piece of paper or type on their phone to com-
municate to the other individual as AoD can only be used to view
the captions of the conversation partner. Doing away with pre-
senter mode on personal devices such as phones or laptops would
make the whole experience hands-free, which would be benefcial
in many situations. Some other areas that need signifcant improve-
ments would be Blade’s battery life, its ability to provide stable
Wi-Fi connectivity, and its form factor. For both interpreting and
captioning, the ability to interact with the interpreter and caption
containers in 3D space to reposition or customize them, as some
researchers demonstrated using the Microsoft HoloLens [3], would
also enhance the overall user experience.
In addition to improvements described earlier, future work
should also include streamlining the process of requesting inter-
preters or captioners before testing out the software rigorous in
more real-world situations such as during an in-person conference
meeting or a panel at a convention. As AR technology and hardware
ASSETS ’22, October 23–26, 2022, Athens, Greece
mature, there is potential that AR smart glasses can be used as an
assistive technology tool.
5 CONCLUSION
Our experiences with Access on Demand demonstrate that this
prototype concept is promising and has the potential to fulfll the
accessibility needs of people with hearing loss in contexts where
the services of an in-person human captioner or interpreter are
not available or feasible. However, there are several avenues for
possible improvements, both with the AoD platform and the AR
smart glasses. Other research articles provided a few paths we could
take to improve our platform, especially on the caption components
[2, 3, 5–8]. There is already a technological-social model developed
by Cesar Lozano and Roci Maciel [4] that we can use to determine
how efective the AoD platform on the Vuzix Blade will be once
it has been fully developed and open to the general public and
measure the impact of social inclusiveness it will provide to the
DHH community via the three modes. Of course, there will be some
more feld testing to iron out the bugs and glitches in the software,
and the next generation model of the Vuzix smart glasses called the
Shield™ might have the hardware to fully realize the main goal of
the AoD with its less bulky design and a better microphone [12],
that is to provide accessibility everywhere and anytime.
Overall, it was a rewarding experience to test the latest version
of Access on Demand with the three assistive components and
document our unique experiences in this report. We are looking
forward to the future when this technology becomes mainstream.
ACKNOWLEDGMENTS
This material is based upon work supported by the National Science
Foundation under Award No. 1811509. Any opinions, fndings and
conclusions, or recommendations expressed in this material are of
the author(s) and do not necessarily refect the views of the National
Science Foundation.
REFERENCES
[1] Hawa Allarakhia. 2022. Consider stafng options for ASL interpreters. Disability
Compliance for Higher Education 27, 7 (2022), 1–5. DOI:https://doi.org/10.1002/
dhe.31218
[2] Abraham Glasser. 2019. Automatic Speech Recognition Services: Deaf and Hard-
of-Hearing Usability. In Extended Abstracts of the 2019 CHI Conference on Human
Factors in Computing Systems (CHI EA ’19), Association for Computing Machinery,
New York, NY, USA, 1–6. DOI:https://doi.org/10.1145/3290607.3308461
[3] Dhruv Jain, Bonnie Chinh, Leah Findlater, Raja Kushalnagar, and Jon Froehlich.
2018. Exploring Augmented Reality Approaches to Real-Time Captioning: A
Preliminary Autoethnographic Study. In Proceedings of the 2018 ACM Conference
Companion Publication on Designing Interactive Systems (DIS ’18 Companion),
Association for Computing Machinery, New York, NY, USA, 7–11. DOI:https:
//doi.org/10.1145/3197391.3205404
[4] Cesar Lozano and Rocio Maciel. 2017. Technological-Social Model: Based in
augmented reality platforms for the inclusion of deaf people in the university
classroom and the cities. In Proceedings of the 8th Latin American Conference on
Human-Computer Interaction (CLIHC ’17), Association for Computing Machinery,
New York, NY, USA, 1–4. DOI:https://doi.org/10.1145/3151470.3156648
[5] Ashley Miller, Joan Malasig, Brenda Castro, Vicki L. Hanson, Hugo Nicolau,
and Alessandra Brandão. 2017. The Use of Smart Glasses for Lecture Compre-
hension by Deaf and Hard of Hearing Students. In Proceedings of the 2017 CHI
Conference Extended Abstracts on Human Factors in Computing Systems (CHI
EA ’17), Association for Computing Machinery, New York, NY, USA, 1909–1915.
DOI:https://doi.org/10.1145/3027063.3053117
[6] Yi-Hao Peng, Ming-Wei Hsi, Paul Taele, Ting-Yu Lin, Po-En Lai, Leon Hsu, Tzu-
chuan Chen, Te-Yen Wu, Yu-An Chen, Hsien-Hui Tang, and Mike Y. Chen. 2018.
SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-of-Hearing
People in Group Conversations. In Proceedings of the 2018 CHI Conference on
ASSETS ’22, October 23–26, 2022, Athens, Greece
Human Factors in Computing Systems (CHI ’18), Association for Computing Ma-
chinery, New York, NY, USA, 1–10. DOI:https://doi.org/10.1145/3173574.3173867
[7] Ali Mohammed Ridha and Wessam Shehieb. 2021. Assistive Technology for
Hearing-Impaired and Deaf Students Utilizing Augmented Reality. In 2021 IEEE
Canadian Conference on Electrical and Computer Engineering (CCECE), 1–5.
DOI:https://doi.org/10.1109/CCECE53047.2021.9569193
[8] Phillip Ward, Ye Wang, Peter Paul, and Mardi Loeterman. 2007. Near-Verbatim
Captioning Versus Edited Captioning for Students Who Are Deaf or Hard of
Hearing: A Preliminary Investigation of Efects on Comprehension. American
Roshan Mathew et al.
Annals of the Deaf 152, 1 (2007), 20–28. DOI:https://doi.org/10.1353/aad.2007.0015
[9] NTID Research Center on Culture and Language | National Technical Institute
for the Deaf | RIT. Retrieved August 5, 2022 from https://www.rit.edu/ntid/nccl
[10] The C-Print®System | C-Print. Retrieved August 5, 2022 from https://www.rit.
edu/ntid/cprint/
[11] Vuzix Blade Upgraded Smart Glasses. Vuzix. Retrieved August 5, 2022 from
https://www.vuzix.com/products/vuzix-blade-smart-glasses-upgraded
[12] Vuzix ShieldTM. Vuzix. Retrieved August 5, 2022 from https://www.vuzix.com/
pages/vuzix-shield
VRBubble: Enhancing Peripheral Awareness of Avatars for People
with Visual Impairments in Social Virtual Reality
Tiger Ji Brianna R. Cochran
tfji@wisc.edu bcochran2@wisc.edu
University of Wisconsin-Madison University of Wisconsin-Madison
Madison, Wisconsin, USA Madison, Wisconsin, USA
ABSTRACT
Social Virtual Reality (VR) is growing for remote socialization and
collaboration. However, current social VR applications are not ac-
cessible to people with visual impairments (PVI) due to their focus
on visual experiences. We aim to facilitate social VR accessibility
by enhancing PVI’s peripheral awareness of surrounding avatar
dynamics. We designed VRBubble, an audio-based VR technique
that provides surrounding avatar information based on social dis-
tances. Based on Hall’s proxemic theory, VRBubble divides the
social space with three Bubbles—Intimate, Conversation, and Social
Bubble—generating spatial audio feedback to distinguish avatars
in diferent bubbles and provide suitable avatar information. We
provide three audio alternatives: earcons, verbal notifcations, and
real-world sound efects. PVI can select and combine their preferred
feedback alternatives for diferent avatars, bubbles, and social con-
texts. We evaluated VRBubble and an audio beacon baseline with
12 PVI in a navigation and a conversation context. We found that
VRBubble signifcantly enhanced participants’ avatar awareness
during navigation and enabled avatar identifcation in both con-
texts. However, VRBubble was shown to be more distracting in
crowded environments.
CCS CONCEPTS
• Human-centered computing → Virtual reality; Accessibility
technologies.
KEYWORDS
visual impairments, social virtual reality, proxemics, audio feedback
ACM Reference Format:
Tiger Ji, Brianna R. Cochran, and Yuhang Zhao. 2022. VRBubble: Enhancing
Peripheral Awareness of Avatars for People with Visual Impairments in
Social Virtual Reality. In The 24th International ACM SIGACCESS Conference
on Computers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens,
Greece. ACM, New York, NY, USA, 17 pages. https://doi.org/10.1145/3517428.
3544821
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544821
Yuhang Zhao
yuhang.zhao@cs.wisc.edu
University of Wisconsin-Madison
Madison, Wisconsin, USA
1 INTRODUCTION
Social virtual reality (VR) refers to VR platforms that allow users to
socialize with each other in the form of avatars in a virtual space
[41]. More and more social VR platforms have been deployed to
the market, such as VRChat, Rec Room, and Altspace.[22, 43, 78]
Compared to 2D video conferencing systems that cause “Zoom fa-
tigue” [47], social VR ofers an immersive and engaging experience
that can enhance interpersonal interaction and boost productivity.
As a result, social VR has attracted increasing attention in recent
years. The VR market was valued at $7.81 billion as of 2020, and
is expected to grow 28.2% annually from 2021 to 2028[54]. Meta,
formerly known as Facebook, also re-branded in 2021 alongside
their full commitment to producing a Metaverse, envisioning social
VR to be the next generation of Internet that connects everyone.
Unfortunately, similar to most mainstream VR applications, cur-
rent social VR mainly targets sighted users by providing various
visual avatar designs and supporting non-verbal social interac-
tions, which poses barriers to people with visual impairments (PVI).
With more than two billion people experiencing visual impair-
ments worldwide [53], it is vital to provide PVI equal access to the
emerging social VR as virtual collaboration and gathering increases,
especially during the COVID-19 pandemic [50].
Researchers have started tackling the VR accessibility problems
for PVI by enabling them to navigate and perceive VR scenes. Some
research leveraged or created additional devices (e.g., PHANToM,
game controller thumbsticks) to enable VR navigation by providing
haptic feedback and/or audio feedback [7, 24, 30, 46, 49, 68, 85]. Oth-
ers focused on software solutions, designing accessible interactions
based on existing VR setups, such as keyboard-based interactions
with spatial audio feedback for a virtual world [52, 59, 72, 74], and
more intuitive interactions based on of-the-shelf VR controllers
and headsets [86]. However, prior work mainly focused on basic VR
tasks such as navigation and object perception. It does not address
the unique barriers caused by the dynamic and multiplayer nature
of social VR.
Diferent from a static VR scene with only system-generated ob-
jects, social VR is more complex and challenging—human controlled
avatars constantly move in the environment and interact with other
avatars and objects. All the avatar dynamics are mainly presented
visually in social VR and not accessible to PVI. To our knowledge,
no existing techniques have focused on the avatar dynamics to
support accessible social VR experience for PVI.
We aim to fll this gap by enhancing PVI’s awareness of sur-
rounding avatars in social VR. Unlike prior work that required PVI
to actively explore and query information from the environment
[49, 85], we focus on peripheral awareness—the innate ability to un-
consciously “maintain and constantly update a sense of our social
ASSETS ’22, October 23–26, 2022, Athens, Greece
and physical context” [56]. People can usually maintain peripheral
awareness efortlessly without being distracted from their main
focus [6]. This ability is especially important in social and collabora-
tive environments since it provides more context for one’s activity.
For example, when navigating to a specifc location, a sighted per-
son can easily notice who has passed by to decide whether to greet
that person or start a quick conversation; when in the middle of a
conversation, they can stay aware of who just joined the conversa-
tion or who is close enough to overhear the conversation.
We seek to facilitate the peripheral awareness of avatars for
PVI in social VR. Via an iterative design with six PVI, we designed
VRBubble, an audio-based VR technique that provides surrounding
avatar information based on social distances. Following Hall’s prox-
emic theory [18], we split the virtual space with three “bubbles”
centered on the user—the Intimate Bubble, Conversation Bubble,
and Social Bubble—to represent diferent social spaces that are suit-
able for diferent social interactions. VRBubble then provides three
spatial audio alternatives (i.e., earcons, real-world sound efects,
and verbal notifcations) to convey the avatar information, such as
names, relationships with the PVI (friends or strangers), and their
interactions with the bubbles (entering or leaving). PVI can also
fexibly select and combine their preferred audio alternatives for
diferent bubbles and avatars to maintain awareness of the avatar
dynamics.
We evaluated VRBubble with 12 participants with visual impair-
ments in the context of a navigation task and a conversation task
in social VR, with an audio beacon attached to each avatar as the
baseline. Our study showed that VRBubble enhanced user avatar
awareness while navigating and was efective at providing users
with previously inaccessible identifying information about avatars.
We also found that PVI generally favor receiving verbal descrip-
tions while navigating, and more brief and intuitive sounds while
conversing.
2 RELATED WORK
2.1 Accessibility of Virtual Environments
2.1.1 Audio Techniques for VR accessibility. There has been exten-
sive work that assisted PVI in exploring virtual environments via
audio feedback. We summarize diferent types of audio techniques
in prior work below.
Audio beacons. Audio beacons have been used to convey object
positions [13, 35, 36, 79]. For example, Walker and Lindsay’s [79]
study utilized three diferent audio beacons in navigation guidance.
They observed the impacts on PVI’s navigation performance as
they changed various parameters, such as timbre and distance to
a waypoint to trigger the audio. Maidenbaum et al. [36] provided
a beeping sound based on the distance between the PVI’s avatar
and the virtual object in front of them to facilitate navigation in
a virtual space. As the avatar got closer to the object, the beeping
rose in frequency of beeps. Blind Swordsman [13] was a VR game
on mobile devices, where a blind user can hear the spatial audio
beacon from the enemies, physically turn to that direction, and tap
the touchscreen to swing his sword in the direction he is facing.
Object Sonifcation. Prior work has also used audio to identify
objects within the virtual space [3, 10, 19, 42, 55, 57, 63–65]. For
example, de Oliveira et al. [10] recreated a virtual stage and placed
T. Ji et al.
instruments in the environment that generated spacial music. Par-
ticipants then listened to the music tracks to identify and locate
instruments on the stage. Heuten et al. [19] presented a sonifcation
interface to virtual maps for PVI. PVI can listen to the spatialized
earcons specifc to various geographic objects and landmarks on
the map to construct a mental model of a geographical area. Au-
dioDoom [63, 64] was an acoustic virtual environment designed for
blind children. Virtual objects and events generated spatial sounds
to help users identify objects, navigate the space, and improve
cognitive skills.
Echolocation. Virtual echolocation has been emulated through
signals and audio refections [2, 80, 83]. For example, Andrade et
al. [2] enabled PVI to use echolocation to navigate a desktop-based
virtual world, where the user’s avatar can produce mouth-click or
clap sounds by pressing a key on a keyboard and hear the sound
refected in the environment. Waters and Abulula [80] presented
a sonar system based on the refectivity of ultrasound used in the
echolocation of bats. The audio used was scaled within human
hearing ranges, so that PVI can utilize the audio to navigate a VR
environment. However, the echolocation method was only used by
a small amount of blind people.
User Queried Verbal Descriptions Some works allowed users
to select objects in the environment via a list or a grid and hear
additional descriptions about the selected object [37, 49, 74, 82]. For
instance, Terraformers [82] was a virtual world game designed to
be accessible to PVI. It provided a menu for nearby objects. A user
can thus navigate the menu and hear audio descriptions of selected
objects. Nair et al. [49] also compared similar menu systems with
their novel game controller-based navigation technique, allowing
PVI to look around a virtual world by scrubbing the thumbstick on a
game controller to diferent directions; the system then announced
what was in that direction via spatial verbal descriptions.
2.1.2 Haptic Solutions for VR accessibility. Prior work has also
enhanced the accessibility of virtual environments for PVI by gen-
erating haptic feedback or creating haptic controllers [25, 33, 66–
68, 75, 76, 81, 85]. For example, Jansson et al. [25] enabled PVI to use
the stylus on a Phantom Premium device to “touch” a virtual space
and receive force feedback to perceive diferent virtual surfaces and
objects. Tzovaras et al. [76] leveraged the CyberGrasp haptic gloves
to generate force feedback to a blind user’s fngers, providing the
illusion that they were navigating the virtual space with a cane.
In the same vein, Zhao et al. created Canetroller [85], a wearable
haptic VR controller that simulated white cane interaction for blind
people in virtual reality. When a user swept the controller and hit
a virtual object, a physical resistance and spatial audio sound efect
were generated to simulate the feedback of a white cane hitting a
real object. A follow-up work by Siu et al. [68] further improved
the design of the controller by providing three dimensional force
feedback. Recently, Wedof et al. designed a virtual reality game,
called Virtual Showdown [81]. PVI played the game by hitting a
virtual ball into the opponent’ goal across a virtual table using a bat.
In this game, a Kinect was used to track the user’s movement. A
visually impaired user can then hold a Nintendo Switch controller
as the bat and receive both audio and vibration feedback to perceive
the relative position of the ball from his bat.
VRBubble
Prior research on VR accessibility focuses on space navigation
and object perception. However, social VR introduces additional
complications with the dynamic and non-uniform avatars that
present social implications. No research has addressed the accessi-
bility of avatars in social VR. Our research aims to fll this gap by
facilitating PVI’s awareness of avatars in social VR via customizable
audio techniques.
2.2 Accessibility of the Real World
As with virtual environments, a myriad of prior work has designed
audio techniques to sonify real world environments, including using
audio beacons to mark waypoints [20, 84], informing users about
nearby objects and landmarks through verbal descriptions [15, 66],
generating auditory icons or earcons to identify points of interest
[39, 58], and providing echolocation or sonar systems to enable the
exploration of surrounding environments [21, 77]. Similar to audio
techniques for virtual worlds, these solutions also do not address
the dynamic complexity of avatars in a social VR context.
2.2.1 Technologies to Facilitate Real-World Social Activities. More
relevant research has been done to assist PVI in real-world social
activities by enhancing their awareness of the conversational part-
ners and their non-verbal behaviors. Various wearable or handheld
assistive technologies have been developed. These technologies
came in a variety of forms, including smartphone applications [87],
belts[5, 40], gloves [31, 32], headband [45, 60], glasses [1, 73], and
wearable cameras [12, 73]. Some research focused on haptic feed-
back. For example, Krishna et al. [32] created a haptic glove called
VibroGlove that allowed PVI to understand facial expressions of
a conversation partner. The glove consisted of several vibration
motors mounted on the back of each fnger that was used to present
six diferent emotions (i.e., anger, disgust, happiness, fear, surprise,
and sadness) through vibration patterns. These patterns were de-
signed based around the shape of the mouth and eye area of each
facial expression. Another example was Tactile Ban, a wearable
headband prototyped by Qiu et al. [60], which used tactile feedback
to provide the feeling of other people’s gazes onto the PVI. The
band had two vibration patterns based on if a person glanced at the
PVI or if their gaze was fxated on them currently.
Some work focused on audio feedback. For example, Anam et
al. [1] used Google glasses to track faces and communicate social
signals (i.e., facial features, behavioral expressions, body postures,
and emotions) of surrounding people to the PVI verbally. Recently,
Morrison et al. [45] designed PeopleLens, a head-mounted device
that provided spatial identifying audio to assist blind children with
gaze direction and mental mapping of surrounding people. Bump
and woodblock sounds were used to guide the user’s gaze to center
on a face, while names would be read out for identifed people in
the surroundings as the user’s gaze passed over them.
Prior work has focused on enhancing the primary social tasks,
conveying information about the conversational partners. Besides
the primary task, peripheral awareness (as a secondary task) is also
important in social activities to unconsciously sense the surrounding
dynamics and make ad hoc social decisions. However, this ability
remains understudied for PVI.
ASSETS ’22, October 23–26, 2022, Athens, Greece
2.2.2 Assistive Peripheral Awareness Technologies. There has been
some prior work that facilitated peripheral awareness during col-
laborative tasks through visual [6, 61] or audio [9, 27, 29, 34] cues.
For example, Cadiz et al. [6] created the Sideshow interface to
support the user’s peripheral awareness of information on their
computer. The interface remained on the user’s primary display and
presented personalized information through visual notifcations
and summaries, such as number of unread emails or number of
friends online. Sakashita et al. [61] enabled remote collaboration
involving physical artifacts through the use of video conferencing
and motion tracking. They placed video devices to represent each
collaborator and automatically oriented devices to emulate the gaze
direction of the collaborator. This supported the user’s peripheral
awareness of what part of the physical artifact a collaborator was
focusing on. However, these works focus on sighted people by
providing visual feedback.
Some work designed audio feedback to enhance PVI’s peripheral
awareness in a work collaboration scenarios [9, 27, 34]. For example,
Lee et al. [34] designed the CollabAlly browser extension, which
provided blind users with audio feedback to support the navigation
of content or comment changes. Earcons were utilized to convey
which part of the document was being edited and diferent voices
were used for text-to-speech to contextualize which collaborator
was editing. Jung [27] used ambient music to convey notifcations
to individuals within a physical work space. Speakers were placed
throughout the space to enable spatial audio, then unique musical
notifcations were played at an individual’s location when events
relevant to that individual occurred, such as the reception of an
email.
However, this work does not design for the unique challenges
posed by a social context. Compared to collaborative tasks which
involve a small number of collaborators and allow for asynchronous
interactions, a social context can involve a large number of moving
avatars, generating more peripheral information and distraction.
Thus, a diferent design is needed to adequately facilitate peripheral
awareness for the social context.
3 DESIGN OF VRBUBBLE
We designed VRBubble, an audio-based VR interaction technique
to enhance the peripheral awareness of avatars in social VR for
PVI. Our design followed the method of user-centered design [51].
We frst formulated a set of general design considerations based
on prior literature to design an initial prototype. With a formative
study with six PVI, we further iterated and improved our design
[26]. We describe our design process and the fnal design below.
3.1 General Design Considerations
We formulated the following design considerations (C1-C3) based
on insights from prior literature [17, 87].
(C1) Leverage mainstream platforms. We focus on desktop VR,
where a user can see the virtual environment on the desktop screen,
hear the spatial audio feedback via earphones or speakers, and
interact with it via keyboard and mouse. While VR headsets are
emerging, most people don’t own a headset due to its cost [70],
let alone PVI who cannot beneft from the visual feedback from
the headset. Instead, desktop computers are more widely used and
ASSETS ’22, October 23–26, 2022, Athens, Greece
many social VR platforms (e.g., VRChat, AltSpace) support desktop
access. Moreover, following the VR ofce concept by Grubert et al.
[17], stationary VR in front of a desk that can be controlled by a
keyboard can support more comfortable and efcient interaction
for longer use and higher productivity in social and collaborative
context.
(C2) Convey proper information about avatar dynamics. Prior
research has explored PVI’s needs for information in real-world
social activities and indicated that the top two important informa-
tion include the identity of surrounding people and their relative
location [87]. We thus translate such needs to the virtual world and
provide the corresponding avatar information to support PVI in
the social VR context.
(C3) Avoid intrusiveness and distraction. Since peripheral aware-
ness is supposed to be efortless [6, 38], we seek to minimize the
distraction of our design as well as the user’s conscious efort. We
thus consider short but intuitive audio feedback at suitable tim-
ing to convey avatar information to PVI in an unobtrusive manner.
Moreover, instead of having users to actively query information, we
focus on proactive notifcations to reduce their interaction efort.
3.2 VRBubble based on Hall’s Proxemic Theory
To convey surrounding avatars’ location information (C2) to PVI
without overwhelming them (C3), our design followed Hall’s prox-
emic theory [18] to divide the virtual environment into diferent
social spaces. Hall’s proxemic theory correlated physical distances
with social interactions that typically happen within that distance,
such as the distance for intimate interactions versus the distance
for friendly conversation. Three distances were defned: intimate
distance (1 foot) where people usually feel distress when this space
is encroached upon unwillingly, personal distance (4 feet) where
people interact with familiar people, and social distance (12 feet)
where conversation with less familiar acquaintances or group hap-
pens. The space outside of the social distance is considered to be
public space.
Figure 1: Conceptual diagram of bubbles.
We defned three virtual bubbles centered on a user based on the
distance thresholds defned by Hall; the Intimate Bubble, Conver-
sation Bubble, and Social Bubbles (Figure 1). We describe the social
indication of each bubble:
Intimate Bubble defnes the space within the intimate distance.
Avatars in this bubble signify that they are about to collide with
the user’s avatar. Since PVI cannot visually perceive whether they
are too close to others or whether other people are invading their
T. Ji et al.
intimate space, it is important to alert them when avatars enter this
bubble (C2).
Conversation Bubble represents the space between the intimate
and personal distance. Avatars in this space are close enough to
start a conversation with. Moreover, the user may need to pay
attention to these avatars since they are close to overhear one’s
ongoing conversation. We thus generate audio feedback to notify
the user if any avatar enters or exits this space (C2).
Social Bubble defnes the space outside of the personal distance,
but still within social distance. Avatars in this space are potential
conversational partners, but would not be in the immediate distance
to start the conversation. The user could then decide whether she is
interested in approaching this person for a conversation. Compared
to the Intimate and Conversation Bubble, we expect that avatar
information in the Social Bubble would be less important. We thus
design more subtle audio feedback to inform the user of the avatars
in this bubble (C3).
We do not consider avatars outside of the the social distance (i.e.,
public space in Hall’s theory) since they are much less relevant to
the users’ current social context (C3).
3.3 Audio Alternative Design via Iterations
Based on the three bubbles, we designed spatial audio feedback to
convey the surrounding avatars information, including the avatar
identity and motion dynamics between these bubbles (C2), thus
enabling PVI to build suitable social interactions upon sufcient
avatar awareness. We also sought to make our audio feedback as
least distracting and invasive as possible (C3). To achieve these goals,
we designed and iterated on diferent audio feedback alternatives
via a formative study.
3.3.1 Formative Study. Following the method of user-centered
design [51], we conducted a formative study [26] with six PVI (three
male, two female, and one who preferred not to say) whose ages
ranged from 22 to 58 (���� = 44.167, �� = 13.348). All participants
were legally blind.
Initial Design. In the formative study, we presented our initial
design of VRBubble with the three bubbles described before. When
an avatar entered or exited a bubble, the user heard an earcon with a
verbal description of the avatar’s information. Earcons were utilized
as they are brief, abstract, and distinctive (C3) sounds that encodes
particular information [4]. We used diferent earcons to represents
an avatar’s moving dynamics between diferent bubbles. A two-
beat earcon with increasing tone (or decreasing tone ) was
used to represent an avatar entering (or leaving) the Social Bubble;
similar two-beat earcons with a diferent timbre were used for the
Conversation Bubble; and an “bumping” sound earcon was used to
indicate an avatar in the Intimate Bubble. All earcons were accom-
panied with a more informative verbal description (C2), reporting
the avatar’s name, relationship with the user (friend or stranger),
and the relative position (“nearby” for Conversation Bubble, “In
vicinity” for Social Bubble). For example, the user heard “Friend
Alice nearby (or no longer nearby)” if Alice’s avatar entered (or left)
the Conversation Bubble. All audio feedback were spatial audio
rendered from the avatar’s position. We prototyped VRBubble using
web-based VR (C1, details in Section 4.1.2).
VRBubble
Method and Findings. The formative study was hosted virtually
through zoom and took roughly 2 hours for each participant to
complete. Participants were given access to our VR environment
through a url. Participants were introduced to the initial VRBubble
design through a short tutorial. We then asked them to experience
VRBubble in two tasks: a navigation task, and a conversation task.
Both tasks were used to simulate common social contexts they
could encounter in VR. During the prototype experiencing process,
participants thought aloud, describing whether they like this feature
or not and why. By the end, participants discussed how they wanted
to improve VRBubble in diferent social context.
We summarize our major fndings from the formative study. (1)
While most participants found the earcon design useful, understand-
ing the abstract earcons could create a steep learning curve for PVI.
Some participants suggested more intuitive sound efects to present
avatar information. (2) Verbal description is clear and easy to un-
derstand, but it could be distracting especially in a conversation
context. Participants suggested shortening the verbal description
to reduce distraction. (3) Participants had diferent preferences for
audio feedback and valued various avatar information diferently.
Our design should provide the fexibility for users to customize
their audio experience for diferent avatars and social contexts.
3.3.2 Three Audio Alternatives. Based on the fndings in the
formative study, we designed three spatial audio feedback alterna-
tives: earcons, verbal notifcations, and real-world sound efects.
Each alternative presented similar avatar information, including
avatar identity (name and/or relationship with the PVI) and motion
dynamics between bubbles (C2). We describe the design of three
audio alternatives for each bubble:
Figure 2: Three audio alternatives for diferent bubbles: (a)
earcons, (b) verbal notifcations, (c) real-world sound efects.
Earcon. Given that earcon is abstract and has a high learning
curve, we used earcons to present simple information, such as dis-
tinguishing diferent bubbles and diferent avatars (friend avatar
vs. stranger avatar). Instead of completely diferent earcons, we de-
signed associated earcons with distinctions to minimize the learning
curve (C3). We used a two-beat earcon with the tone of the last beat
increased 1 (or decreased 2 ) to indicate an avatar entering
(or leaving) the Social Bubble. For the Conversation Bubble, since
1 Friend entering Social Bubble earcon: https://cdn.glitch.global/11db17e0-58dc-4f24-
8390-bb2e597d1475/f_soc_in.mp3?v=1639640993569
2 Friend leaving Social Bubble earcon: https://cdn.glitch.global/11db17e0-58dc-4f24-
8390-bb2e597d1475/f_soc_out.mp3?v=1639640998113
ASSETS ’22, October 23–26, 2022, Athens, Greece
avatars in this bubble probably required more immediate attention,
we used four-beat earcons with the tone of last beat increased
3 or decreased 4 to represent an avatar entering or leaving
this bubble. To distinguish friend and stranger avatars, we adjusted
the pitch and speed of the earcons, so that higher pitched, faster
earcons indicated friends, while normal pitch and speed indicated
strangers 5 6 7 8 . We used a game-like bump earcon 9 to signify
when an avatar was in the Intimate Bubble.
Verbal Notifcations. We provided clear and short (C3) verbal
notifcations to present avatar information. To reduce distraction,
we informed avatar identity by only reporting the name. We no
longer provided information on if the avatar was a friend or a
stranger since in the real world a person would be able to know
whether someone was a friend by name. We also used verbal no-
tifcation to convey the avatar’s general position based on their
interaction with the bubbles. Specifcally, we used “in the area” 10
and “left the area” 11 to indicate when an avatar entered or left
the Social Bubble, and “nearby” 12 and “no longer nearby” 13 for
entering or leaving the Conversation Bubble. If an avatar, Alice, en-
tered the Conversation Bubble, a user would hear “Alice nearby.” To
signify that an avatar was in the Intimate Bubble, we used “collided
with” 14 followed by the avatar name.
Real-world Sound Efect. This design alternative focused on
providing realistic real-world sounds to intuitively support periph-
eral awareness and immersion, thus reducing the amount of efort
needed to comprehend the audio notifcation (C3). We used crowd
sound efects as the background sound, with diferent densities to
represent the total number of avatars in all three bubbles within the
social distance (12 feet). Two levels of crowd sound efects with in-
creasing volume and densities indicate two levels of stranger avatar
amount: 1-5 15 and more than 5 16 strangers. Two levels of cheer-
ing crowd sounds were also used to highlight friend avatars due
to their higher importance, indicating two levels of friend avatar
3 Friend entering Conversation Bubble earcon: https://cdn.glitch.global/11db17e0-58dc-
4f24-8390-bb2e597d1475/f_convo_in.mp3?v=1643983723059
4 Friend leaving Conversation Bubble earcon: https://cdn.glitch.global/11db17e0-58dc-
4f24-8390-bb2e597d1475/f_convo_out.mp3?v=1643983757550
5 Stranger entering Social Bubble earcon: https://cdn.glitch.global/11db17e0-58dc-4f24-
8390-bb2e597d1475/s_soc_in.mp3?v=1649899925021
6 Stranger leaving Social Bubble earcon: https://cdn.glitch.global/11db17e0-58dc-4f24-
8390-bb2e597d1475/s_soc_out.mp3?v=1639641013488
7 Stranger entering Conversation Bubble earcon: https://cdn.glitch.global/11db17e0-
58dc-4f24-8390-bb2e597d1475/s_convo_in.mp3?v=1649900021917
8 Stranger leaving Conversation Bubble earcon: https://cdn.glitch.global/11db17e0-
58dc-4f24-8390-bb2e597d1475/s_convo_out.mp3?v=1643983842245
9 Intimate Bubble earcon: https://cdn.glitch.me/11db17e0-58dc-4f24-8390-
bb2e597d1475%2F483602__raclure__game-bump.mp3?v=1636388046675
10 Entering Social Bubble verbal: https://cdn.glitch.global/11db17e0-58dc-4f24-8390-
bb2e597d1475/area.mp3?v=1643807674537
11 Leaving Social Bubble verbal: https://cdn.glitch.global/11db17e0-58dc-4f24-8390-
bb2e597d1475/noarea.mp3?v=1643807963569
12 Entering Conversation Bubble verbal: https://cdn.glitch.global/11db17e0-58dc-4f24-
8390-bb2e597d1475/nearby.mp3?v=1643807938153
13 Leaving Conversation Bubble verbal: https://cdn.glitch.global/11db17e0-58dc-4f24-
8390-bb2e597d1475/nonearby.mp3?v=1643807989565
14 Intimate Bubble verbal: https://cdn.glitch.global/11db17e0-58dc-4f24-8390-
bb2e597d1475/collidedwith.mp3?v=1643983652403
15 Low ambient stranger crowd: https://cdn.glitch.me/11db17e0-58dc-4f24-8390-
bb2e597d1475/stranger_crowd_small.wav?v=1639761138064
16 High ambient stranger crowd: https://cdn.glitch.me/11db17e0-58dc-4f24-8390-
bb2e597d1475/stranger_crowd_medium.wav?v=1639761063563
ASSETS ’22, October 23–26, 2022, Athens, Greece T. Ji et al.

amount: 1-5 17 and more than 5 18 friends. Moreover, we put more Table 1: Demographic information of 12 participants.
emphasis on the avatars in the Conversation Bubble since the user
was more likely to converse with them. A spatial footstep sound ID Age/Sex Prior VR Experience
efect was regularly played for every avatar within the Conversa-
P1 31/M Vision therapy
tion Bubble. We generated diferent footsteps sounds to distinguish
P2 57/M 3D virtual home design game
friend avatars from strangers, with the friend avatars presenting
P3 37/M Audio-based VR games; VR studies
a higher pitch heel sound 19 and the stranger avatars presenting
P4 68/M VR environments; VR studies
a boot stomp 20 . We also generated a realistic bump sound 21 to
P5 44/F Simulated street navigation VR app
indicate an avatar entering the Intimate Bubble.
P6 34/F VR headset games
P7 49/F Oculus Quest accessibility testing; VR games
3.4 Customization P8 25/F None
Customization was key to our design since diferent PVI have dif- P9 33/F None
ferent audio preferences and prioritize diferent information and P10 46/M PVI accessible audio games
goals. Instead of generating all audio feedback together (as we did P11 23/M Navigational VR study
in the initial design), we allowed users to select and combine dif- P12 20/M None
ferent audio alternatives for diferent bubbles, avatars (e.g., friend
vs. stranger), and social context to customize their experiences. For 4.1.2 Apparatus. We built a custom VR environment with avatars
example, a user could choose verbal notifcations for friends but and implemented both VRBubble and a baseline feature in the VR
earcons for strangers in the Social Bubble, and combine both au- environment. To evaluate VRBubble in diferent contexts, we imple-
dio alternatives for both friends and strangers in the Conversation mented two VR scenarios, a navigation scenario and a conversation
Bubble. The user was also allowed to select none audio feedback scenario. We describe the design and implementation of our study
for a specifc type of avatar or bubble (e.g., stranger avatars in the environment.
Social Bubble). Baseline. We compared VRBubble’s performance to a baseline
feature. Our baseline was an audio beacon that consisted of a spatial
4 EVALUATION beeping sound, in intervals, attached to each avatar (except for the
We evaluated VRBubble with 12 PVI in diferent social contexts. We participant avatar) in the environment. We chose this baseline since
aimed to understand: (1) How efective is VRBubble in enhancing audio beacon was an efective and common way of conveying the
PVI’s peripheral awareness of avatars in diferent social contexts? presence of people and objects for PVI in the real world [11]. Some
(2) How distracting is VRBubble? (3) How do PVI customize their current assistive technology for PVI (e.g., Microsoft Soundscape
audio experience for diferent avatars, bubbles, and social contexts? [44]) also uses audio beacon to label target destinations for PVI.
VR environment and Two Scenarios. We built a custom VR
4.1 Method environment with two key VR scenarios: a navigation scenario and
a conversation scenario. In the navigation scenario, we generated a
4.1.1 Participants. We recruited 12 participants with visual im-
VR space plan with eight similar navigation routes (Figure 3(a)). All
pairments (7 male, 5 female) whose ages ranged from 20 to 68
routes were approximately 60 ft with one turn (either turn left or
(���� = 38.92, �� = 13.79, Table 1). No participants were in the
turn right). All routes were considered to be the same according to
formative study. The participants were recruited through the Na-
Ishikawa et al.’s study design [23] since they had same length and
tional Federation of the Blind. We used a survey to check the par-
same number of turns. Avatars were rendered at along the routes
ticipants’ eligibility. Participants were eligible for our study if they
at various distances (Figure 3(a)).
were at least 18 years old, legally blind, and capable of independent
In the conversation scenario, we rendered one avatar in front
consent. We also asked about participants’ VR experience in the
of the user, conversing with the user. Other avatars were rendered
survey. We prioritized participants who had experience with VR
and moving around in the virtual space during the conversation
or virtual environments. Among the 12 participants, nine had VR
(Figure 3(b)).
experience, such as audio-based VR games, VR for vision therapy,
Features for Basic Movements To enable participants to navi-
and VR environment exploration. However, no participants had
gate in the virtual environment and experience the avatar aware-
experience with social VR. Participants were compensated at the
ness features, we added two fundamental features to allow PVI to
rate of $20 per hour.
move and turn in a virtual environment: (1) Movement: PVI can
control the movement of their own avatar via arrow keys, for ex-
17 Low ample, up arrow to move forward and left arrow to move left. Each
ambient friend crowd: https://cdn.glitch.me/11db17e0-58dc-4f24-8390-
bb2e597d1475/friend_crowd_small.wav?v=1639761145228 key press would move the PVI a foot in the virtual space and play a
18 High ambient friend crowd: https://cdn.glitch.me/11db17e0-58dc-4f24-8390- footstep sound to notify the PVI. This footstep sound 22 is centered
bb2e597d1475/friend_crowd_medium.mp3?v=1639761151401 on the user’s avatar and difers from the footsteps used for VRBub-
19 Friend footstep: https://cdn.glitch.global/11db17e0-58dc-4f24-8390-bb2e597d1475/
fstep.mp3?v=1643754264523 ble by pitch. (2) Turning. A user can turn her avatar to the left (or
20 Stranger footstep: https://cdn.glitch.global/11db17e0-58dc-4f24-8390-bb2e597d1475/
right) by pressing the left (or right) arrow keys while holding the
sstep.mp3?v=1643754271035
21 Collision: https://cdn.glitch.global/11db17e0-58dc-4f24-8390-bb2e597d1475/mixkit- 22 User footstep: https://cdn.glitch.me/11db17e0-58dc-4f24-8390-bb2e597d1475%
quest-game-heavy-stomp-v-3049.wav?v=1644177861935 2F571698__rainial-co__cotton-thud-7.wav?v=1636411987805
VRBubble
Figure 3: Two tasks: (a) navigation, (b) conversation.
shift key. A ticking sound 23 would play for each 90 degree turn,
providing a sense of how much the PVI had turned.
VR Environment Implementation and Setup. We implemented
the VR environment and all the features with A-Frame [71], a web
framework that generates 3D spaces via HTML and Javascript. We
hosted our prototype on Glitch [16], a browser-based platform that
allows users to host and share applications as well as share the
source code, to enable a remote setup for the study. Participants
and researchers can go to the same web address to join the same VR
environment. Once connected, researchers can input unique com-
mand sequences via a keyboard to adjust the participant’s client,
such as adjusting the audio alternatives in VRBubble or moving
the participant’s avatar to a diferent VR scenario. Participants’
behavioral data (e.g., their positions at diferent times) was then
logged and output into the researchers’ browser console.
We asked participants to share their screen with audio via Zoom
to ensure the commands from the researchers were properly re-
fected on the participants’ end and to confrm that the audio feed-
back was properly generated at the participants’ end.
4.1.3 Procedure. Our study consisted of a single session that
lasted 2 hours for each participant. We conducted the study re-
motely via a Zoom video call. An initial interview was frst con-
ducted to ask about participants’ prior experience with VR, espe-
cially social VR. Then we asked about their social experiences in
real life.
We then asked participants to experience our VR environment
with the awareness features. Participants were requested to use
their earphones for the study, since we generate spatial audio. We
sent a url to our VR environment through email before the study. If
any participant had difculty accessing their email, we sent the link
again through the Zoom chat or read the link out-loud to them.
After the participant successfully joined the VR environment, we
evaluated participants’ experience and performance in two context:
(1) a navigation context and (2) a conversation context. In each
context, participants completed navigation or conversation tasks
with two awareness features—VRBubble and the baseline (i.e., audio
beacon). We counterbalanced the awareness feature and the context
across all participants. We elaborate the details of the remaining
study in three phases: tutorial, navigation, an conversation.
Tutorial: Introducing the Features. We conducted a tutorial
session to teach participants how to use our VR prototypes. We
frst introduced the interactions for basic movements (arrows keys
to move and shift + left/right arrow key to turn, as explained in
Section 4.1.2). We then rendered an empty VR space and encour-
aged participants to move around until they were familiar with the
23 Turn audio: https://cdn.glitch.me/11db17e0-58dc-4f24-8390-bb2e597d1475%
2F268108__nenadsimic__button-tick.wav?v=1636008594574
ASSETS ’22, October 23–26, 2022, Athens, Greece
controls and the audio feedback produced by the avatar movement.
We then introduced VRBubble and the baseline.
In the tutorial for VRBubble, we introduced each audio alterna-
tive one by one. For each alternative, we demonstrated the audio
feedback of both friend avatar and stranger avatar for each Bubble.
We then rendered avatars in the VR space and asked participants
to freely explore the environment with this audio alternative until
they felt fully familiar with it. During the exploration, participants
were asked to think aloud, talking about their experiences with this
audio design, including whether they like this design or not and
the reason. We counterbalanced the order of the audio alternatives
using Latin Square [48]. We used the similar method to introduce
the baseline.
Navigation. We evaluated the efectiveness of VRBubble in
the navigation context. Each participant conducted 15 trials of
navigation tasks along the routes in Figure 3a in diferent conditions.
We randomly selected a route for each trial. To enable participants
to follow the route and arrive at the destination, we generated audio-
based verbal instructions. Participants heard “turn left (or right)”
when they needed to turn, and heard “arrived destination” when
they reached the end of the route. For each trial, the participant
was frst moved to the start point of a route. When the research
announced “Start”, they would navigate by following the verbal
instruction until arriving at the destination. We asked participants
to complete each navigation task as quickly as possible.
Participants frst conducted three trials of basic navigation tasks
without any avatars around. These trials helped participants get
familiar with the navigation tasks. We then asked participants to
customize their audio feedback in VRBubble based on the navi-
gation task. To guide participants through the customization, we
asked them to choose and combine audio feedback for the fve
following situations: (1) when a friend enters or leaves the Social
Bubble; (2) when a stranger enters or leaves the Social Bubble; (3)
when a stranger enters or leaves the Conversation Bubble; (4) when
a friend enters or leaves the Conversation Bubble; (5) when an
avatar collides with the user in the Intimate Bubble. For each situa-
tion, participants could select any combination of audio feedback
ranging from multiple to none at all (Figure 7). During the cus-
tomization process, the researcher adjusted the audio alternatives
in real time for the participant to experience the selected feedback
and make immediate corrections. After customizing VRBubble, we
asked about the reason behind their choices and any suggestions
they have for customizing. Once the customization phase was com-
pleted, the researcher adjusted the audio feedback in VRBubble to
the participant’s preferred combination.
After the customization, participants continued to perform an-
other 12 trials of navigation tasks with avatars around. Participants
were asked to complete two tasks at the same time: a primary navi-
gation task (i.e., navigating to the destination following the verbal
instructions) and a secondary avatar awareness task (i.e., memo-
rizing how many avatars they’ve passed by and who they are). We
asked participants to reach the destination as quickly as possible,
but also try their best to perceive and memorize the surrounding
avatars. Before the navigation task, we informed the participants
ahead that we would asked them to report the avatar number and
names they remembered after each trial. However, if the task was
ASSETS ’22, October 23–26, 2022, Athens, Greece
too difcult, we asked participants to prioritize the primary naviga-
tion task.
Participants completed above tasks in two awareness conditions:
using VRBubble with the customized audio alternatives, and using
the baseline. Each condition included six trials of navigation. To
further understand the efectiveness of VRBubble in situations with
diferent avatar complexity, we generated avatars in two amount
levels: Low Amount (1-5 avatars) and High Amount (6-10 avatars).
We decided whether a trial had low or high amount of avatars
randomly but ensured that participants experienced each avatar
amount level for three trials per awareness condition. For each
trial, we randomly generated avatars within the corresponding
avatar amount range, and the avatars were randomly distributed
along the navigation route. We guaranteed that all avatars were
within the social distance (12 feet) to the route, so that all avatars
were detectable by VRBubble. After each navigation trial, we asked
participants to report their perceived avatar information, including
the total avatar number, the number of friend avatars, the number
of stranger avatars, and the avatar names they heard. We timed
participants’ navigation time for all trials.
By the end, we asked about participants’ general experience with
VRBubble and the baseline. Participants rated the efectiveness,
distraction, and immersion of both features via 5-point likert scale
scores. Participants also discussed their feedback and suggestions
for improving VRBubble in the navigation context.
Conversation. The procedure of the conversation task was sim-
ilar to the navigation task. This phase of the study was conducted
in the conversation scenario (Figure 3b). Participants conducted
three trials of conversation task in diferent conditions. For each
trial, the participant faced an avatar who asked 10 yes/no questions.
Participants were asked to answer all questions as quickly and
accurately as possible. A researcher on the team acted as the con-
versation avatar to ask the questions. The next question was asked
immediately after the participant answered the prior question.
Participants frst conducted one trial of “dry-run” task, answering
10 questions without any distraction to get familiar with the task.
We then asked them to customize the VRBubble feedback again for
the conversation context. Participants then continued to perform
two trials of conversation tasks with avatars around. One trial
used the customized VRBubble and the other used the baseline.
For each trial, participants conducted two tasks simultaneously:
a primary conversation task (i.e., answering 10 yes/no questions)
and a secondary awareness task (i.e., perceiving and memorizing
surrounding avatars). The primary task was prioritized if the task
was too difcult. After each trial, we asked participants to recall
the avatar information, including total avatar number, the number
of friends and strangers, and the avatar names.
To balance the avatar complexity in each trial, we pre-defned
two similar groups of avatar moving dynamics. Both groups had
eight avatars spawn in and move around the user over the duration
of 70 seconds during the conversation process. Each group had
four friend avatars and four stranger avatars. However, the order
of avatars and the timing they spawn in were diferent between
the two groups. We counterbalanced the avatar groups with the
awareness feature used across participants.
T. Ji et al.
We pre-planned 30 yes/no questions. All questions were easy-to-
answer questions that asked about a specifc fact about the partici-
pants to minimize the variances caused by participants’ literacy and
cognitive ability. Some example questions included “Have you ever
been to Italy?”, “Are you a cofee drinker?” We randomly pooled
10 questions for each trial and also made sure not to repeat any
questions across the study for each participant. We measured par-
ticipants’ response time for each question.
After all trials, we asked all 30 yes/no questions again without
any distractions and collected participants’ answers as the correct
answers. Participants then assessed the efectiveness, distraction,
and immersion of VRBubble and the baseline for the conversation
context via 5-point likert scale scores. Finally, participants talked
about their qualitative feedback and suggestions for VRBubble in
the conversation context.
We ended the study by discussing with the participants about
their general experiences with VRBubble compared to the baseline,
as well as their desired awareness technology for diferent contexts
in social VR in the future.
4.1.4 Analysis. We detail our analysis methods for both the quan-
titative and qualitative data in our study.
Navigation Performance We frst analyzed the impact of VR-
Bubble on participants’ performance in the navigation task, includ-
ing navigation time and the error rate when recalling the number
of avatars they passed by. We had three within-subject factors:
Feature (two levels: VRBubble, Baseline) that defned the avatar
awareness feature participants used, AvatarAmount (Low: 1-5, High:
6-10) that specifed the amount of avatars in the space, and Trial
(1-3) that defned each navigation task in a specifc condition. We
had two measures: NavigationTime—the time taken in seconds by
the participant to reach the end of the navigation path, and Avatar-
ErrorRate—the ratio of the diference between the reported avatar
number and the actual number to the actual number of avatars.
We also added a between-subject factor, Order (VRBubble-Baseline,
Baseline-VRBubble) to our model for counterbalance validation.
The Shapiro-Wilk test showed that both NavigationTime (� =
.867, � < .001) and AvatarErrorRate (� = .898, � < .001) devi-
ated signifcantly from a normal distribution, so we used Aligned
Rank Transform for nonparametric factorial ANOVAs (ART) [28]
24 to model our data. We found no signifcant efect of Order
(� 1,10 = 2.296, � = .161) on the navigation time and no signif-
cant efect of the interaction between Order and other factors on
navigation time. Similarly, we also found no signifcant efects of
Order (� 1,10 = .0005, � = .982) on avatar error rate and no signif-
cant efect of the interaction between Order and other factors on
error rate.
Conversation Performance We then analyzed the impact of
VRBubble on participants’ performance in the conversation task,
including the response time per question, accuracy of answering
questions, and the error rate when recalling the number of passed-
by avatars during the conversation. We had two within-subject
factors: Feature (VRBubble, Baseline), and AvatarGroup (Group1,
Group2) that indicated which pre-defned set of avatar dynamics
was used during each conversation trial. We had three measures:
AvatarErrorRate, ResponseTime—the mean response time for the
24 ART is a non-parametric approach to factorial ANOVA or linear mixed efect model
VRBubble
participant to answer a question, and IncorrectAnswers—the number
of questions that participants answered inconsistently compared
to the standard answers or questions that needed to be repeated.
A between-subject factor, Order (VRBubble-Baseline, Baseline-
VRBubble), was added to valid our counterbalancing. The Shapiro-
Wilk test showed that ResponseTime (� = .886, � = .011) and Incor-
rectAnswers (� = .654, � < .001) were non-normally distributed ,
while AvatarErrorRate (� = .972, � = .708) followed a normal distri-
bution. We thus used ART to model ResponseTime and IncorrectAn-
swers, and used ANOVA [14] for AvatarErrorRate. We found no sig-
nifcant efect of Order on the response time (� 1,8 = .293, � = .603)
or questions answered incorrectly (� 1,8 = .802, � = .397) and no sig-
nifcant efect of the interactions between Order and other factors
on response time or questions answered incorrectly. Similarly, we
found no signifcant efects of Order (� 1,8 = .702, � = .426) or its
interactions on the error rate of recalling avatars.
Moreover, we found no signifcant efect of AvatarGroup or its
interactions on ResponseTime (� 1,8 = .026, � = .876), IncorrectAn-
swers (� 1,8 = .82, � = .392), and AvatarErrorRate (� 1,8 = .157, � =
.097). These results confrmed that the two pre-defned sets of
avatars had similar complexity.
Post hoc Analysis. Based on the results of the ART and ANOVA
models, we conducted post hoc tests to further investigate the rela-
tionship between the diferent levels of signifcant factors. Specif-
ically, we used paired t-test for normally distributed data, and
used the pairwise Wilcoxon signed-rank test for non-normally dis-
tributed data. We corrected the p-value threshold with Bonferroni
Correction.
Qualitative Analysis. We video recorded the whole study and
transcribed participants’ verbal feedback and responses to the in-
terview questions with an automatic transcription service, Otter.ai.
Researchers on the team also read through the transcripts and cor-
rected the auto transcription errors in the transcripts. We analyzed
the transcripts using the the qualitative analysis method described
by Saldaña [62]. Two researchers frst independently coded the frst
four participants’ transcripts. They then discussed and generated a
codebook upon agreement. One researcher then followed the code-
book and coded the rest of the data. When a new code emerged,
the two researchers discussed again and updated the codebook.
The codes were then organized and categorized into themes using
afnity diagram.
4.2 Results
We report the impact of VRBubble on participants’ performance in
diferent context, including both a navigation and a conversation
task. We also report participants’ experiences with diferent audio
alternatives and their preferences when customizing VRBubble for
diferent context.
4.2.1 VRBubble in Navigation. For the navigation task, we found
that VRBubble provided the user with more detailed and accurate
information about surrounding avatars, compared to the baseline.
However, we also noticed that VRBubble could be more distracting
than the baseline. We report the detailed results below.
Avatar Amount Estimation. We analyzed the efectiveness of
VRBubble on enhancing PVI’s awareness of the amount of avatars
ASSETS ’22, October 23–26, 2022, Athens, Greece
nearby in a navigation context. An ART analysis showed a signif-
cant efect of the awareness feature on participants’ error rate when
estimating the amount of avatars they passed by (� 1,110 = 33.54, � <
.001). We then conducted a post hoc paired Wilcoxon signed-rank
test and found that participants had a signifcantly lower error
rate when using VRBubble than the baseline (� = 1566.5, � <
.001, error rate with VRBubble: ���� = .239, �� = .26, baseline:
���� = .465, �� = .323). This result demonstrated that VRBubble
signifcantly improved PVI’s awareness of the surrounding avatar
amount.
We examined the error rate more closely for environments with
diferent avatar complexity (AvatarAmount): low amount of avatars
vs high amount of avatars. A post hoc Wilcoxon signed-rank test
showed that VRBubble signifcantly decreased participants’ error
rate of avatar number estimation regardless the avatar amount
was high (� = 585, � < .001) or low (� = 259, � = .01 < .025
with Bonferroni Correction). This indicated that the efectiveness
of VRBubble was consistent across environments with diferent
avatar complexity.
Participants’ qualitative feedback also explained the efective-
ness of VRBubble. Ten participants expressed positive sentiment
towards VRBubble, emphasizing its ability to provide distinguish-
able audio feedback for each avatar, with seven noting VRBubble
created a better sense of distance for avatars via the diferent bub-
bles. In contrast, participants found the audio beacon baseline hard
to use since all avatars shared the same beacon sound and it was
difcult to distinguish diferent avatars, especially when avatars
were close together. P6 “had a hard time diferentiating if it was the
beacon for the additional avatar or the same avatar. I had a hard time
distinguishing how many people or avatars there were.”
Avatar Identifcation. Compared to the baseline, VRBubble
enabled participants to collect more detailed information about
surrounding avatars, including avatar names and the relationship
to the user.
When customizing VRBubble, nine participants selected verbal
notifcation that reported avatar names. With this feature, partici-
pants in general recalled the names of passed-by avatars accurately
with a mean accuracy of 0.715 (�� = .305). Six participants ex-
pressly liked VRBubble’s capacity to provide information about
each individual avatar. For example, P3 emphasized the importance
of name identifcation in engaging in social interactions: “[The
name] would potentially determine whether or not I would want to go
and speak to them ... or even somebody I want to avoid ... I can see that
having been named there is a useful feature.” However, we noticed
that the avatar identifcation accuracy of some participants dropped
to below 0.5 in more crowded avatar environments. This was be-
cause the verbal notifcations from diferent avatars would overlap
and become confusing when the avatar reached high amounts. As
P5 noted, “If it’s not too many people happening at once then I can
feel it out, but then when so many people are happening, then I can’t
really fgure it out.”
Ten participants’ VRBubble customization supported friend iden-
tifcation (i.e., earcon or real-world sound efect for Conversation
Bubble). Most participants distinguished friends and strangers ac-
curately with a mean accuracy of 0.658 for friends (�� = .367)
and 0.656 for strangers (�� = .376). Ten participants agreed that
discerning between friends and strangers was important for them
ASSETS ’22, October 23–26, 2022, Athens, Greece
to decide what social actions to take. As P2 said, “I might want to
ask a stranger what time it is. But if I hear friendly footsteps coming
towards me, I want to know who’s approaching and I can strike up con-
versation.” However, P4 was an outlier who consistently reported
many more avatars than the actual friend and stranger numbers
in the space. This was because P4 used the abstract earcons for
the Conversation Bubble and found it difcult to discern the pitch
changes between friends and strangers.
The two participants (P5, P10) chose verbal notifcation for the
Conversation Bubble and thus could not distinguish friends and
strangers. They customized VRBubble based on their real-life ex-
periences, as in real-life they would recognize if someone was a
friend or stranger by name. We envisioned that users would be able
to distinguish friends and strangers via verbally reported names in
a real social VR setting after long-term use.
Impact on Navigation Time. We compared the efects of aware-
ness feature on participants’ navigation time. With an ART model,
we found a signifcant efect of awareness feature on navigation
time (� 1 = 36.708, � < .001). A post hoc paired Wilcoxon signed-
rank test with correction showed that participants navigated sig-
nifcantly slower when using VRBubble than using the beacon
baseline (� = 482, � < .001, navigation time with VRBubble:
���� = 36.507, �� = 14.897, baseline: ���� = 31.293, �� = 11.033).
The result indicated that VRBubble distracted participants more
than the baseline in the navigation task.
We further investigated the efect of VRBubble under diferent
avatar complexity (low amount of avatars vs. high amount). We
found a signifcant efect of the interaction between Feature and
AvatarAmount on participants’ navigation time (� 1 = 13.676, � <
.001). Using a post hoc Wilcoxon signed-rank test with Bonferroni
Correction, we found that, there’s no signifcant diference in time
between VRBubble and baseline when the avatar number was low
(� = 209, � = .052), but participants walked signifcantly slower
with VRBubble than the baseline when the avatar number was high
(� = 66, � < .001). This indicated that VRBubble was only more
distracting when the environments became crowded.
The distraction of VRBubble could be caused by the more di-
verse audio feedback provided by VRBubble, where participants
spent longer time to associate the sounds with diferent events. Six
participants mentioned being distracted and slowed down since
they tried to remember the meaning of the diferent sounds.As P5
explained, “If I go too fast, I might miss it ... I have to pay attention
on top of all the other avatars leaving, coming in, and coming out.”
Another reason that may cause a slower navigation was par-
ticipants’ interest. Since VRBubble enabled participants to receive
more information about the surrounding avatars, it aroused partici-
pants’ interest to explore the environment during the navigation.
In our study, while we emphasized that navigation was the pri-
mary task, we still observed that some participants (e.g., P2, P5,
P12) stopped, looked around, and even moved their avatar left and
right to explore nearby avatars with VRBubble. In contrast, since
the baseline did not provide much information, participants mostly
focused on the navigation.
4.2.2 VRBubble in Conversation. Unlike in the navigation task,
VRBubble generally did not perform signifcantly diferent from
T. Ji et al.
the baseline in the conversation context. We report the detailed
measures below.
Avatar Amount Estimation. We analyzed the efectiveness of
VRBubble on enhancing PVI’s awareness of avatar amount nearby
in a conversation context. An ANOVA analysis showed no signif-
cant efect of the awareness feature on participants’ error rate when
estimating the amount of avatar they passed by (� 1,8 = 1.431, � =
.266). Compared to the navigation task, we found that participants
made more errors when estimating the avatar number with VRBub-
ble in the conversation task (���� = .305, �� = .191). The error
rates went especially high when there are more avatars around.
Interestingly, we noticed that the reported avatar number by par-
ticipants never exceeded fve across all trials, even though there
were usually more avatars present. Specifcally, eight participants
encountered more than fve avatars in the conversation task, and
the highest avatar number was eight. This might indicate a limit in
how much audio feedback can be peripherally processed (or a upper
bound of avatar number that can be perceived) while focusing on
an audio-centered task such as conversation. As P4 noted, “Because
you had to answer the question, so you couldn’t put all your brain-
power into [the surrounding avatars]. I just don’t think the human
brain is segmented enough to handle both those tasks at the same
time.”
Avatar Identifcation. Similar to the navigation task, partici-
pants appreciated VRBubble’s ability to identify surrounding avatars
in real-time since it enabled them to dynamically adapt to the so-
cial environments. Ten participants indicated that knowing who
was in the immediate surroundings had an impact on their social
behaviours and what they would say during a conversation.
Seven participants customized VRBubble to allow for name iden-
tifcation. However, participants’ avatar recognition accuracy were
lower (���� = .655, �� = .282) compared to the navigation task
(���� = .715, �� = .305). Moreover, ten participants customized
VRBubble to distinguish friends from strangers. Similarly, they were
less accurate compared to those in the navigation task, both when
identifying friends (���� = .567, �� = .323) and when identifying
strangers (���� = .467, �� = .302). The lower accuracy could be
attributed to the more attention-demanding nature of the conver-
sation task, and the direct confict between verbal notifcations and
the conversation.
Impact on Conversation. We investigated the impact of VR-
Bubble on participants’ ability to answer questions in a conver-
sation. With an ART model, we found no signifcant efect of the
awareness feature on the number of questions answered incorrectly
(� 1,8 = .647, � = .444). Most participants answered all questions
correctly with both features. Four participants (P2, P4, P5, P12)
presented incorrect answers while using VRBubble, and four par-
ticipants (P4, P7, P10, P11) did so with the baseline. P2 was the
only participant who answered two questions incorrectly, while
everyone else answered one question incorrectly.
Moreover, we compared the efects of VRBubble and the baseline
on participants’ response time to questions in the conversation.
With an ART model, we found no signifcant efect of Feature on
response time (� 1,8 = .206, � = .662). These results showed that
VRBubble and baseline were generally at the same distraction level
in the conversation task.
VRBubble
Figure 4: Likert scale score averages for each task: navigation
Task (nav) and conversation task (conv).
4.2.3 Perceived Experience with VRBubble. We compared partici-
pants subjective experience with VRBubble and the baseline from
three dimensions, including efectiveness, distraction, and immer-
sion. Figure 4 shows the mean scores given by participants for each
dimension. Participants perceived VRBubble (���� = 3.917, �� =
.9) to be more efective than the baseline (���� = 2.667, �� = 1.231)
at enhancing avatar awareness in the navigation task, while per-
ceiving VRBubble (���� = 3.167, �� = 1.403) to be slightly less
efective than the baseline (���� = 3.25, �� = 1.485) for the conver-
sation task. These scores matched our expectations given that the
conversation task was more attention-demanding and participants
had less cognitive capacity to use VRBubble.
Participants perceived more immersion in the virtual space while
using VRBubble (���� = 4.333, �� = .888) than the baseline
(���� = 2.667, �� = 1.557) during the navigation task. They also
felt more immersed during the conversation task when using VR-
Bubble (���� = 3.583, �� = 1.443) than when using the baseline
(���� = 3.083, �� = 1.505). Six participants found the audio bea-
con baseline to be annoying or distracting, and thus diminishing
their immersive experience. In contrast, VRBubble provided a better
sense of presence by dividing the social space and associating the
avatar dynamics with social indications. As P3 mentioned, “The
bubble system was much more immersive, just because it gives you
a couple of clear lines between when you’re entering and leaving
someone’s space. The beacon is just super impersonal.”
Interestingly, participants perceived VRBubble (���� = 1.75, �� =
.754) to be less distracting than the baseline (���� = 2.583, �� =
1.379) for the navigation task, despite taking signifcantly more time
to complete the task with VRBubble. This further confrmed that
the slower navigation could be caused by participant’s increased
interest in exploring the surrounding with VRBubble instead of
distraction. For the conversation task, participants perceived VR-
Bubble (���� = 3.167, �� = 1.467) to be slightly more distracting
than the baseline (���� = 3.083, �� = 1.443). Five participants
expressed that the amount of information from VRBubble caused
distractions during conversation, suggesting simpler or reduced
audio was preferred for audio-intensive tasks.
4.2.4 Experience with VRBubble and its Audio Alternatives. All par-
ticipants agreed that the Bubble concept based on social distances
ASSETS ’22, October 23–26, 2022, Athens, Greece
was novel and efective, and expressed favor that VRBubble pro-
vided them with more information than the baseline did. Seven
participants liked that VRBubble separated the space and conveyed
a sense of distance to the user. P3 noted, “I like the three diferent
distances. If I walk into a social situation, I wish I had a bubble radar
just to know when people are in my immediate vicinity versus people
a little ways away. It is just nice to have a concept of relative [dis-
tance].” Participants also liked the sense of avatar presence and
movement that was conveyed by the feedback from the bubbles.
We then elaborate participants’ experiences with diferent audio
alternatives below.
Earcon. Four participants liked the brevity and non-intrusiveness
of the earcons. Unlike the continuous audio feedback in the baseline,
participants liked that the earcons discretely notifed them every
time an avatar entered a bubble. For example, P8 had difculty
with discerning spatial audio. She reported, “I like how concrete
[the earcon] is. I have a really hard time with spatial awareness. So,
without necessarily having to keep track of the space because there
are those diferent [earcons], it’s very helpful.” Three participants
noted that earcons for each avatar event could be easily noticed and
distinguished, which could be used to reduce audio overloading in
crowded environments. As P4 indicated, “To signify the person is
leaving, I’d want the [earcon] sound that shows the person’s going
out. Because then you don’t have too many verbal cues, you know, to
not listen too much.”
The main deterrent of earcons was the deep learning curve. Six
participants found it hard to recognize diferent earcon sounds. P4
pointed out that the meanings of earcons would have to become sec-
ond nature in order to be efective in a dynamic social environment:
“You don’t want to have to think about it, you want to immediately
react.” Five participants had issues with discerning the pitch difer-
ence between friend and stranger earcons. As P1 said, “there were
more and more unexpected [earcons] that threw me of and, being not
as familiar with the tones, second guessing if [the earcon] was higher
[pitched] or lower, entering or exiting.” P4 suggested that instead of
using diferent pitches, completely diferent earcons should be used
for diferent types of avatars.
Verbal Notifcation. Nine participants liked that the verbal noti-
fcations were informative and easy to understand. They expressed
a desire for sufcient information about their surroundings to ad-
just their behaviors, especially in the context of conversation. P3
emphasized that distinguishing between friends and strangers can
help protect his privacy, “there may be something that you’re willing
to share with a friend, but not necessarily with a complete stranger.”
The drawback of verbal notifcations was its verbosity, which
caused more audio overlapping between avatars. Nine participants
mentioned this problem in the study. Four participants also noted
that hearing a verbal notifcation immediately drew their attention,
which could be distracting in certain scenarios, such as conversation.
Due to these issues, three participants cared only about verbal
notifcations for important people, such as friends, rather than
being verbally informed about every avatar.
Real-world Sound Efect. Six participants liked the footstep
and crowd noises since they were ambient and non-distracting.
The audio had no sudden alerts and was quiet enough to tune
out when focusing on the primary task. The crowd sounds were
also immersive and helped convey a general sense of the social
ASSETS ’22, October 23–26, 2022, Athens, Greece T. Ji et al.

environment, such as if the user was at an informal party or a

business meeting. For example, P4 preferred using real-world sound
efects in the conversation task for an immersive experience. As
he expressed, “I was not gearing [my customization] toward the
best [avatar recognition] results but [was] gearing it toward getting
a real feel for it.” P3 even suggested providing “diferent types of
ambient background noises that meet the theme of whatever rooms.”
Two participants (P3, P6) also felt that the two levels of ambient
crowd sound with diferent loudness was sufcient, and any further
levels may be too loud or unable to contribute any additional useful
information. Moreover, two participants (P1, P2) liked the spatial
footstep audio in the Conversation Bubble, which helped them
constantly track important avatars in the conversational distance.
Figure 6: VRBubble customization for the conversation task:
However, six participants disliked the constant real-world sound
side-by-side comparison between amount of times audio al-
efects since it was hard to for them to distinguish each avatar
ternative is selected for friends and strangers, at each bubble.
via the general crowd sounds. P6 thus suggested combining the
Intimate bubble is shared between friends and strangers.
crowd sound with an overall description of the avatar number.
Two participants (P7, P10) had difculty distinguishing friend and
stranger avatars via the diferent footstep sounds, especially with
other louder audio in the environment.
4.2.5 Customization of Audio Alternatives. All participants appre-
Figure 5 and Figure 6. While seven participants used the same
ciated the fexibility of customizing the audio feedback in VRBubble.
combination for both navigation and conversation tasks, fve par-
For example, P8 had an auditory processing disorder that caused
ticipants changed their customization. We found that participants
large quantities of sound to be difcult to process. As she men-
tended to simplify their audio combination in the conversation task,
tioned, “[I like] the idea that you can customize how you’re going
and all changes happened in Social Bubble. The changes included
to handle an event or have part of an event made accessible to you
removing an audio alternative (P4, P11 removed verbal notifcation,
via sound. It is not something most people think about and I really
P5 removed real-world sound efect for strangers, and P7 removed
appreciate it.”
earcons for friends), and changing verbal notifcations to more brief
Participants customized the audio alternatives for diferent so-
earcons (P3 for strangers, P11 for friends). This pattern suggested
cial tasks, avatar roles, and bubbles (Figure 7). Figure 5 shows the
that users preferred shorter and non-verbal feedback during audio-
distribution of audio alternatives chosen for the navigation task,
focused tasks like conversation, whereas more informative feedback
and Figure 6 shows the distribution for the conversation task. While
was preferred for exploration-based tasks like navigation.
no participants selected the exactly same combinations, we saw
Friends vs Strangers. Participants had divergent preferences
some patterns across participants that we report below.
when distinguishing friends and strangers. Some participants (P2,
P5, P7, P8) chose more detailed information for friends and felt
that stranger information was less important especially in the So-
cial Bubble. P2 and P7 even did not want any audio feedback for
strangers in the Social Bubble. On the other hand, some participants
(P3, P11) wanted to know more about strangers: P3 selected earcons
for friends and more informative verbal notifcation for strangers in
the Social Bubble in the navigation task, while P11 added earcons
for the strangers to alert himself. They explained that in a real
use case, users were likely to already know enough information
about friends and would only need more detailed information about
strangers. As P3 described, “The verbal would be useful because that
might indicate to me it’s a stranger I don’t know. And this is their
name.”
Figure 5: VRBubble customization for navigation task: Side- Paterns across Bubbles. Most participants preferred the real-
by-side comparison between amount of times that each audio world sound efects for the Intimate Bubble, while some participants
alternative was selected for friends and strangers, at each bub- selected the abstract earcons. Participants mostly made the decision
ble. Intimate bubble is shared between friends and strangers. based on which audio feedback they found more pleasant. Interest-
ing, P5 was the only participant who expressed interest in knowing
Navigation vs. Conversation Task. In general, we found that who they bumped into, thus choosing verbal notifcation. This may
verbal notifcation was most popular for the navigation task, while suggest that in social VR, avatar collision was an event that PVI
earcons and real-world sound efects were more popular in the wanted to avoid but not important enough for PVI to care about
conversation task (except for Intimate Bubble), as shown between the details.
VRBubble
Figure 7: Chart of participants’ VRBubble customization in navigation and conversation tasks.
For the Social and Conversation Bubbles, we found that partici-
pants had diferent preferences for diferent bubbles without clear
patterns, and only P10 used verbal notifcation for both bubbles.
Audio Combinations. The most popular audio combination was
between verbal notifcations and real-world sound efects, which
was adopted by four participants (P4, P5, and P9 used the combina-
tion for the Social Bubble, P2 used it at the Conversation Bubble).
Two participants (P7, P11) also combined earcons with verbal noti-
fcations. Participants used such combinations to add immersion
to the detailed information provided by verbal notifcations. As P5
described, “Because that will add more reality to what I’m doing, in-
stead of just a boring computer thing.” Only P12 used a combination
of earcons and real-world footsteps. As he explained, “The abstract
[earcon] is my favorite and that’s why I chose most of the options
here. But I thought it would be helpful to have the footsteps as well
just so it’s a little bit diferent [from just the earcon].”
5 DISCUSSION
With VRBubble, we contributed the frst VR technique that lever-
aged the social distances (i.e., Bubbles) in Hall’s proxemic theory
[18] to enhance PVI’s peripheral awareness of avatars in a complex
and dynamic social VR context. VRBubble enabled user customiza-
tion by providing three spatial audio alternatives—earcons, verbal
notifcations, and real-world sound efects—and allowing users to
select and combine their preferred audio feedback for diferent
bubbles, avatars, and social contexts.
Our evaluation with 12 PVI demonstrated the efectiveness of
VRBubble: It signifcantly enhanced participants’ awareness of the
amount of avatars they passed by in a navigation task and enabled
participants to identify most avatars, including their names and
relationship to the users in both navigation and conversation tasks.
However, we found that VRBubble could be distracting especially in
crowded environments with high amount of avatars. Our research
was the frst step towards the general accessibility of the dynamic
social VR. Based on our exploration, we discuss the takeaways,
design implications, technology generalizability, and the lessons
we learnt from our remote study.
ASSETS ’22, October 23–26, 2022, Athens, Greece
5.1 Audio Preferences for Peripheral Awareness
Besides seeing VRBubble as a whole VR technique, our research
also explored PVI’s experiences and preferences of perceiving pe-
ripheral information via diferent audio modalities. Unlike prior
work that studied the use of diferent audio feedback in various
primary tasks (e.g., [1, 45, 87]), we focused on using audio to convey
peripheral dynamics and investigated the feasibility of diferent au-
dio modalities in two representative social contexts: navigation and
conversation. Our study showed that compared to a fxed design
(e.g., nonadjustable audio beacons), fexible audio customization
allows users to efectively control distraction from a given primary
task. In our study, fve participants changed their audio selection
after experiencing a diferent social task (Table 1), highlighting the
need for customization depending on the social context.
Our research identifed PVI’s customization patterns. First, par-
ticipants’ audio preferences were driven by the cognitive load of
the primary task. They preferred more detailed and descriptive au-
dio modality (e.g., verbal notifcation) in less-attention-demanding
tasks, but could only handle brief and distinct audio feedback (e.g.,
earcons, sound efects) in more attention-demanding and audio-
focused tasks. Second, the importance of peripheral information
was another factor that infuenced PVI’s audio selection. Specif-
cally, the more detailed verbal descriptions or multiple audio alter-
native combinations were selected for more important avatars.
Our results also suggested (but need further investigation) that
people may have a upper limit of capacity to perceive peripheral
information in attention-demanding tasks (e.g., perceiving fve
avatars at most in a conversation task), so that they had to give
up on less important peripheral information, such as information
on strangers in the Social Bubble. While our research focused on
the social VR context, the preferences of audio modalities by PVI
to receive peripheral information could be applied to broader use
cases, such as situational awareness systems for drivers.
ASSETS ’22, October 23–26, 2022, Athens, Greece
5.2 Design Implications
Based on participants’ experiences with VRBubble, we distill design
implications to inspire more accessible and immersive social VR
experience for PVI.
Context-Adaptive Feedback. Our study indicated that partici-
pants preferred diferent audio modalities for diferent social con-
texts. However, social activity is a highly dynamic process where
users’ context and primary task can change quickly. For example, a
user could be navigating, but stop to chat with a passing friend in
the next moment. Thus, it could be difcult for PVI to manually ad-
just their audio combinations every time to match the fast-changing
social contexts. Some participants suggested pre-defning multiple
sets of audio combinations for diferent contexts and designing easy
ways to toggle between them. As P9 suggested, “[You can have] a
walking around mode, and a talking conversation mode. If you exit a
conversation and you’re walking around, switch back to names.” To
reduce the efort of active user control, researchers could consider
context-adaptive feedback mechanism, which recognizes users’ so-
cial context with AI technology and automatically switches to users’
preferred feedback based on the current context.
Balance Automatic and Manually-queried Feedback. VRBub-
ble focused on conveying peripheral information, thus providing
proactive feedback that was automatically triggered by environ-
mental changes to reduce users’ interaction eforts. However, in our
study, several participants indicated the desire for actively query-
ing avatar information since they may have missed some prior
information or need more detailed information about particular
avatars of interest. We acknowledge that both types of feedback
have their values. For example, our study showed that while proac-
tive feedback were suitable for important and urgent information,
too much feedback can distract and overwhelm the users. Mean-
while, manually-queried feedback can be a complement to provide
detailed information on demand and reduce feedback overload. It is
thus important to consider how to balance proactive and manually-
queried feedback to optimize users’ experience in social VR, espe-
cially in crowded virtual environments or cognitively heavy tasks.
For example, proactive notifcations can be used for important and
nearby avatars, and manually-queried feedback can serve to provide
additional information based on users’ interest. Additionally, users
should also have the option to manually repeat any information
provided by proactive feedback.
Enable Third-Party Audio Design. While providing difer-
ent audio alternatives, VRBubble provided only pre-defned audio
efects, especially for earcons and real-world sound efects. Inter-
estingly, our study indicated that participants had distinct hearing
abilities, audio preferences, and audio aesthetic. For example, some
participants (P2, P8) found the real-world footstep more pleasant
to hear, while some (P4, P9) preferred the more abstract earcons.
Even for the same audio modality, participants had diferent pref-
erences. Some (P3, P6, P11, P12) liked the current earcon design,
while some (P1, P4, P5, P9, P10) found them difcult to distinguish.
To better fulfll the users’ needs, one solution could be allowing
users to design and upload their own audio source to represent
particular avatars. However, designing and generating audio source
could be technically challenging, especially for people with visual
impairments. It is thus important to consider the trade-of between
T. Ji et al.
the users’ efort and satisfaction. Online communities for avatar
sonifcation could also be encouraged to provide PVI a place to
search and fnd their preferred audio source, similar to Thingiverse
[69] which serves as an online community that provides 3D model
resources for makers.
5.3 Generalizability
We discuss the generalizability of our technique from both the plat-
form and the users’ perspectives. While implemented and evaluated
on a desktop-based VR platform, VRBubble can be easily transferred
to HMD-based VR as it focuses on proactive audio design that is
not device or platform specifc. It can be adapted to other hardware
through combination with suitable input techniques (e.g., input via
controllers or body/head movements).
Beyond PVI, VRBubble could also be applied for sighted users in
social VR. Various real-world presence cues, such as breathing or
wind movement are not conveyed through visual information. Thus
providing sighted users with additional peripheral information via
audio feedback could enhance immersion in social VR, as well as
ensuring users’ privacy (i.e., being aware of potential out of sight
eavesdroppers).
5.4 Evaluation via a Remote Study
We conducted our study remotely due to the restriction of the pan-
demic. We summarize our experience and the lessons we learnt
from the remote study. First, we found it difcult for participants to
setup all the software and hardware required by our study indepen-
dently. To guide participants through the whole setup process, we
connected with the participants 30 minutes ahead of the study via
Zoom, sent them the url to our VR environment, and asked them to
share screen with us during the setup process. Participants gener-
ally appreciated our assistance via Zoom and felt the use of Zoom
sharing function was efective for the remote study. As P7 noted,
“You’ve just proven a feature not in virtual reality right now—feature
sharing. You just showed how important that is for someone with a
disability because accessibility [functionality] doesn’t always work.”
Second, it is hard to guarantee the consistency of participants’
experience in a remote study, depending on each individual’s de-
vices and environments. For example, they may receive diferent
audio quality due to the diferent headphones, and some of them
may experience feedback delay due to unstable Internet access. For
this study, we tried to ensure a certain level of consistency by check-
ing that participants had the pre-requisite equipment, including a
keyboard, a pair of headphones that supports spatial audio, and a
Microsoft Edge browser installed to access our VR environment.
Participants shared their screen and audio via Zoom, so that we can
check whether our feature was functioning properly in real time.
However, the audio share feature in Zoom did not support spatial
audio, which made it impossible to fully confrm users’ experience.
In the future, how to setup a remote study platform and enable
researchers to easily control and confrm users’ experience is a vital
research direction to explore.
5.5 Limitations and Future Work
Our research has limitations. Since our study focused on the design
and evaluation of the audio feedback (i.e., the output) in VRBubble,
VRBubble
we used a Wizard-of-Oz [8] setup to enable participants to cus-
tomize the audio alternatives. Thus, how to enable PVI to efciently
and easily control and customize VRBubble’s audio modalities (i.e.,
the input) still mains unaddressed. A future work of interest would
be to explore efective input modalities to support fast interaction
for PVI in social VR.
Additionally, our current evaluation used mock-up social VR
environments with system-generated avatars, which may not fully
refect the avatar dynamics in the real social VR environments. For
example, avatars in social VR may be conversing rather than stand-
ing silently, so that they might be located or identifed by their voice.
This can potentially change users’ experience with VRBubble. Fu-
ture research should evaluate VRBubble in more realistic social VR
scenarios and adapt the design to diferent situations; for example,
a context-aware technique that adjusts the audio feedback based
on avatars’ voice activity (e.g., muting or reducing the volume of
audio notifcations for a speaking avatar).
6 CONCLUSION
We presented VRBubble, a social VR technique to augment pe-
ripheral awareness by utilizing Hall’s proxemic theory. VRBubble
provided three audio alternatives—earcons, verbal notifcation, and
real-world sound efects—for PVI to select and combine to maintain
aware of diferent avatars at diferent social distances. We compared
VRBubble to a standard audio beacon baseline via a user study with
12 participants with visual impairments. We assessed the efect
of VRBubble on participants’ ability to identify avatars as well as
its distraction in diferent primary social tasks. Participants found
VRBubble efective at providing previously inaccessible avatar in-
formation, and expressed the need for customizing audio feedback
in diferent social VR contexts. Our study contributed a novel acces-
sibility technique to inform the social VR dynamics and provided
implications for the future design of accessible VR for people with
visual impairments.
ACKNOWLEDGMENTS
We thank the National Federation of the Blind for helping us recruit
for our study, as well as the anonymous participants that provided
their perspective.
REFERENCES
[1] ASM Iftekhar Anam, Shahinur Alam, and Mohammed Yeasin. 2014. Expression:
A dyadic conversation aid using Google Glass for people who are blind or visually
impaired. In 6th International Conference on Mobile Computing, Applications and
Services. IEEE, 57–64.
[2] Ronny Andrade, Steven Baker, Jenny Waycott, and Frank Vetere. 2018. Echo-
house: Exploring a virtual environment by using echolocation. In Proceedings of
the 30th Australian Conference on Computer-Human Interaction. 278–289.
[3] Matthew T Atkinson, Sabahattin Gucukoglu, Colin HC Machin, and Adrian E
Lawrence. 2006. Making the mainstream accessible: redefning the game. In
Proceedings of the 2006 ACM SIGGRAPH Symposium on Videogames. 21–28.
[4] Stephen A Brewster, Peter C Wright, and Alistair DN Edwards. 1993. An evalua-
tion of earcons for use in auditory human-computer interfaces. In Proceedings of
the INTERACT’93 and CHI’93 conference on Human factors in computing systems.
222–227.
[5] Hendrik P Buimer, Marian Bittner, Tjerk Kostelijk, Thea M Van Der Geest, Ab-
dellatif Nemri, Richard JA Van Wezel, and Yan Zhao. 2018. Conveying facial
expressions to blind and visually impaired persons through a wearable vibrotac-
tile device. PloS one 13, 3 (2018), e0194737.
[6] J.J. Cadiz, Gina Venolia, Gavin Jancke, and Anoop Gupta. 2002. Designing and
deploying an information awareness interface. CSCW 02 (November 2002).
https://doi.org/10.1145/587078.587122
ASSETS ’22, October 23–26, 2022, Athens, Greece
[7] Chetz Colwell, Helen Petrie, Diana Kornbrot, Andrew Hardwick, and Stephen
Furner. 1998. Haptic virtual reality for blind computer users. ASSETS 98 (January
1998). https://doi.org/10.1145/274497.274515
[8] Nils Dahlback, Arne Jonsson, and Lars Ahrenberg. 1993. WIZARD OF OZ STUD-
IES — WHY AND HOW. Intelligent User Interfaces (1993).
[9] Maitraye Das, Thomas Barlow McHugh, Anne Marie Piper, and Darren Gergle.
2022. Co11ab: Augmenting Accessibility in Synchronous Collaborative Writing
for People with Vision Impairments. In Proceedings of the 2022 CHI Conference
on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22).
Association for Computing Machinery, New York, NY, USA, Article 196, 18 pages.
https://doi.org/10.1145/3491102.3501918
[10] Patricia A. de Oliveira, Erich P. Lotto, Ana Grasielle D. Correa, Luis G. G. Taboada,
Laisa C. P. Costa, and Roseli D. Lopes. 2015. Virtual Stage: An Immersive Musical
Game for People with Visual Impairment. In 2015 14th Brazilian Symposium on
Computer Games and Digital Entertainment (SBGames). 135–141. https://doi.org/
10.1109/SBGames.2015.26
[11] Euan Freeman, Graham Wilson, Stephen Brewster, Gabriel Baud-Bovy, Charlotte
Magnusson, and Hector Caltenco. 2017. Audible Beacons and Wearables in
Schools: Helping Young Visually Impaired Children Play and Move Independently.
CHI 17 (January 2017). https://doi.org/10.1145/3025453.3025518
[12] Lakshmi Gade, Sreekar Krishna, and Sethuraman Panchanathan. 2009. Person
localization using a wearable camera towards enhancing social interactions
for individuals with visual impairment. In Proceedings of the 1st ACM SIGMM
international workshop on Media studies and implementations that help improving
access to disabled users. 53–62.
[13] Evil Dog Games. 2014. Blind Swordsman. https://devpost.com/software/blind-
swordsman last accessed 18 Jan 2014.
[14] Ellen R Girden. 1992. ANOVA: Repeated measures. Number 84. Sage.
[15] Cole Gleason, Alexander J. Fiannaca, Melanie Kneisel, Edward Cutrell, and
Meredith Ringel Morris. 2018. FootNotes: Geo-Referenced Audio Annotations
for Nonvisual Exploration. 2, 3, Article 109 (sep 2018), 24 pages. https:
//doi.org/10.1145/3264919
[16] Glitch. 2000. Glitch. https://glitch.com last accessed 12 January 2022.
[17] Jens Grubert, Eyal Ofek, Michel Pahud, and Per Ola Kristensson. 2018. The
ofce of the future: Virtual, portable, and global. IEEE computer graphics and
applications 38, 6 (2018), 125–133.
[18] Edward T Hall. 1966. The Hidden Dimension. Doubleday.
[19] Wilko Heuten, Daniel Wichmann, and Susanne Boll. 2006. Interactive 3D sonif-
cation for the exploration of city maps. Nordic conference on Human-computer
interaction: changing roles (2006), 155–164.
[20] Simon Holland, David Morse, and Henrik Gedenryd. 2002. AudioGPS: Spatial
Audio Navigation with a Minimal Attention Interface. Personal and Ubiquitous
Computing 6 (09 2002). https://doi.org/10.1007/s007790200025
[21] Seyedeh Maryam Fakhr Hosseini, Andreas Riener, Rahul Bose, and Myounghoon
Jeon. 2014. “Listen2dRoom”: Helping visually impaired people navigate indoor
environments using an ultrasonic sensor-based orientation aid.
[22] Rec Room Inc. 2016. Rec Room. Oculus Quest, Microsoft Windows, PlayStation,
iOS, Android, Xbox.
[23] Toru Ishikawa, Hiromichi Fujiwara, Osamu Imai, and Atsuyuki Okabe. 2008.
Wayfnding with a GPS-based mobile navigation system: A comparison with
maps and direct experience. Journal of environmental psychology 28, 1 (2008),
74–82.
[24] Gunnar Jansson. 1999. Can a Haptic Display Rendering of Virtual Three-
Dimensional Objects be useful for People with Visual Impairments? 93, 7 (July
1999). https://doi.org/10.1177/0145482X9909300707
[25] Gunnar Jansson, Helen Petrie, Chetz Colwell, Diana Kornbrot, J Fänger, H König,
Katarina Billberger, Andrew Hardwick, and Stephen Furner. 1999. Haptic vir-
tual environments for blind people: Exploratory experiments with two devices.
International journal of virtual reality 4, 1 (1999), 8–17.
[26] Tiger F. Ji, Brianna R Cochran, and Yuhang Zhao. 2022. Demonstration of VRBub-
ble: Enhancing Peripheral Avatar Awareness for People with Visual Impairments
in Social Virtual Reality (CHI EA ’22). Association for Computing Machinery, New
York, NY, USA, Article 401, 6 pages. https://doi.org/10.1145/3491101.3519657
[27] Ralf Jung. 2008. Smart sound environments: merging intentional soundscapes,
nonspeech audio cues and ambient intelligence. The Journal of the Acoustical
Society of America 123, 5 (2008), 3935–3935. https://doi.org/10.1121/1.2936002
arXiv:https://doi.org/10.1121/1.2936002
[28] Matthew Kay, Lisa A. Elkin, James J. Higgins, and Jacob O. Wobbrock. 2021.
ARTool: Aligned Rank Transform for Nonparametric Factorial ANOVAs. https:
//doi.org/10.5281/zenodo.594511 R package version 0.11.1.
[29] Fredrik Kilander and Pekka Loennqvist. 2002. A whisper in the woods - an
ambient soundscape for peripheral awareness of remote processes.
[30] Julian Kreimeier and Timo Gotzelmann. 2019. First Steps Towards Walk-In-Place
Locomotion and Haptic Feedback in Virtual Reality for Visually Impaired. CHI
EA 19 (May 2019). https://doi.org/10.1145/3290607.3312944
[31] Sreekar Krishna, Shantanu Bala, Troy McDaniel, Stephen McGuire, and Sethura-
man Panchanathan. 2010. VibroGlove: an assistive technology aid for conveying
facial expressions. In CHI’10 Extended Abstracts on Human Factors in Computing
ASSETS ’22, October 23–26, 2022, Athens, Greece
Systems. 3637–3642.
[32] Sreekar Krishna, Dirk Colbry, John Black, Vineeth Balasubramanian, and Sethura-
man Panchanathan. 2008. A systematic requirements analysis and development
of an assistive device to enhance the social interaction of people who are blind or
visually impaired. In Workshop on Computer Vision Applications for the Visually
Impaired.
[33] Orly Lahav and David Mioduser. 2004. Exploration of unknown spaces by people
who are blind using a multi-sensory virtual environment. Journal of Special
Education Technology 19, 3 (2004), 15–23.
[34] Cheuk Yin Phipson Lee, Zhuohao Zhang, Jaylin Herskovitz, JooYoung Seo, and
Anhong Guo. 2022. CollabAlly: Accessible Collaboration Awareness in Document
Editing. In Proceedings of the 2022 CHI Conference on Human Factors in Computing
Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery,
New York, NY, USA, Article 596, 17 pages. https://doi.org/10.1145/3491102.
3517635
[35] Tapio Lokki and Matti Grohn. 2005. Navigation with auditory cues in a virtual
environment. IEEE MultiMedia 12, 2 (2005), 80–86.
[36] Shachar Maidenbaum, Shelly Levy-Tzedek, Daniel-Robert Chebat, and Amir
Amedi. 2013. Increasing accessibility to the blind of virtual environments, using
a virtual mobility aid based on the" EyeCane": Feasibility study. PloS one 8, 8
(2013), e72555.
[37] Masaki Matsuo, Takahiro Miura, Masatsugu Sakajiri, Junji Onishi, and Tsukasa
Ono. 2016. Audible Mapper & ShadowRine: Development of Map Editor Using
only Sound in Accessible Game for Blind Users, and Accessible Action RPG for
Visually Impaired Gamers. In International Conference on Computers Helping
People with Special Needs. Springer, 537–544.
[38] Tara Matthews and Jennifer Mankof. 2005. A toolkit for evaluating peripheral
awareness displays. In From the Awareness Systems Workshop at CHI.
[39] Keenan R May, Brianna J Tomlinson, Xiaomeng Ma, Phillip Roberts, and Bruce N
Walker. 2020. Spotlights and soundscapes: On the design of mixed reality auditory
environments for persons with visual impairment. ACM Transactions on Accessible
Computing (TACCESS) 13, 2 (2020), 1–47.
[40] Troy McDaniel, Sreekar Krishna, Vineeth Balasubramanian, Dirk Colbry, and
Sethuraman Panchanathan. 2008. Using a haptic belt to convey non-verbal
communication cues during social interactions to individuals who are blind. In
2008 IEEE international workshop on haptic audio visual environments and games.
IEEE, 13–18.
[41] Joshua McVeigh-Schultz, Anya Kolesnichenko, and Katherine Isbister. 2019. Shap-
ing pro-social interaction in VR: an emerging design framework. In Proceedings
of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
[42] Lotf Merabet and Jaime Sanchez. 2009. Audio-based navigation using virtual
environments: combining technology and neuroscience. AER Journal: Research
and Practice in Visual Impairment and Blindness 2, 3 (2009), 128–137.
[43] Microsoft. 2013. Altspace. Oculus Quest, Microsoft Windows.
[44] Microsoft. 2022. Microsoft Soundscape. Microsoft Windows.
[45] Cecily Morrison, Ed Cutrell, Martin Grayson, Geert Roumen, Rita Faia Marques,
Anja Thieme, Alex Taylor, and Abigail Sellen. 2021. PeopleLens. Interactions 28,
3 (2021), 10–13.
[46] Konstantinos Moustakas, Georgios Nikolakis, Konstantinos Kostopoulos, Dim-
itrious Tzovaras, and Michael G. Strintzis. 2007. Haptic Rendering of Visual Data
for the Visually Impaired. IEEE (January 2007). https://doi.org/10.1145/274497.
274515
[47] Robby Nadler. 2020. Understanding “Zoom fatigue”: Theorizing spatial dynamics
as third skins in computer-mediated communication. Computers and Composition
58 (October 2020). https://doi.org/10.1016/j.compcom.2020.102613
[48] Gerhard Nahler. 2009. Latin square. Springer Vienna, Vienna, 105–105. https:
//doi.org/10.1007/978-3-211-89836-9_776
[49] Vishnu Nair, Jay L Karp, Samuel Silverman, Mohar Kalra, Hollis Lehv, Faizan
Jamil, and Brian A Smith. 2021. NavStick: Making Video Games Blind-Accessible
via the Ability to Look Around. In The 34th Annual ACM Symposium on User
Interface Software and Technology. 538–551.
[50] Research Nester. 2021. Virtual Collaboration Market Segmentation by Tools
(Instant Communication, Project Management Tools, Cloud-Based Tools, and
Others); and by End-User (IT & Telecom, BFSI, Retail, Healthcare, Logistics
& Transportation, Education, Manufacturing, and Others) – Global Demand
Analysis and Opportunity Outlook 2021-2029. https://www.researchnester.com/
reports/virtual-collaboration-market/2994 last accessed 10 January 2022.
[51] Donald A. Norman. 1986. User Centered System Design; New Perspectives on
Human-Computer Interaction. L. Erlbaum Associates Inc.
[52] Bugra Oktay and Eelke Folmer. 2010. TextSL: a screen reader accessible interface
for second life. W4A 10 (April 2010). https://doi.org/10.1145/1805986.1806017
[53] World Health Organization. 2021. Blindness and vision impairment.
https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-
impairment#:~:text=Key%20facts,has%20yet%20to%20be%20addressed. last
accessed 26 Jan 2022.
T. Ji et al.
[54] Andrew Osterland. 2021. Virtual Reality Headset Market Size, Share & Trends
Analysis Report By End-device (Low-end, High-end), By Product Type (Stan-
dalone, Smartphone-enabled), By Application (Gaming, Education), And Seg-
ments Forecasts, 2021 - 2028. https://www.grandviewresearch.com/industry-
analysis/virtual-reality-vr-headset-market last accessed 26 Jan 2022.
[55] Jamie Pauls. 2020. Vintage Games Series, Part 4: Immerse Yourself in the World
of Shades of Doom. https://www.afb.org/aw/21/12/17336 last accessed 18 Jan
2022.
[56] Elin R. Pedersen and Tomas Sokoler. 1997. AROMA: abstract representation of
presence supporting mutual awareness. CHI 97 (March 1997). https://doi.org/10.
1145/258549.258584
[57] Aaron Preece. 2018. A Review of A Hero’s Call, an Accessible Role Playing Game
from Out of Sight Games. https://www.afb.org/aw/19/3/15113 last accessed 18
Jan 2022.
[58] Giorgio Presti, Dragan Ahmetovic, Mattia Ducci, Cristian Bernareggi, Luca Lu-
dovico, Adriano Baratè, Federico Avanzini, and Sergio Mascetti. 2019. WatchOut:
Obstacle Sonifcation for People with Visual Impairment or Blindness. In The
21st International ACM SIGACCESS Conference on Computers and Accessibility
(Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New
York, NY, USA, 402–413. https://doi.org/10.1145/3308561.3353779
[59] Radegast Project. 2009. Radegast. Microsoft Windows, Mac, Linux.
[60] Shi Qiu, Matthias Rauterberg, and Jun Hu. 2016. Designing and evaluating a
wearable device for accessing gaze signals from the sighted. In International
Conference on Universal Access in Human-Computer Interaction. Springer, 454–
464.
[61] Mose Sakashita, E. Andy Ricci, Jatin Arora, and François Guimbretière. 2022. Re-
moteCoDe: Robotic Embodiment for Enhancing Peripheral Awareness in Remote
Collaboration Tasks. Proc. ACM Hum.-Comput. Interact. 6, CSCW1, Article 63
(apr 2022), 22 pages. https://doi.org/10.1145/3512910
[62] Johnny Saldaña. 2021. The coding manual for qualitative researchers. sage.
[63] Jaime Sanchez and Mauricio Lumbreras. 1997. Hyperstories: Interactive narrative
in virtual worlds. Hypertextes et hypermédias (1997), 329–338.
[64] Jaime Sánchez and Mauricio Lumbreras. 1999. Virtual environment interaction
through 3D audio by blind children. CyberPsychology & Behavior 2, 2 (1999),
101–111.
[65] JH Sánchez and MA Sáenz. 2006. Assisting the mobilization through subway
networks by users with visual disabilities. Virtual Reality & Assoc. Tech (2006).
[66] Daisuke Sato, Uran Oh, João Guerreiro, Dragan Ahmetovic, Kakuya Naito,
Hironobu Takagi, Kris M. Kitani, and Chieko Asakawa. 2019. NavCog3 in
the Wild: Large-Scale Blind Indoor Navigation Assistant with Semantic Fea-
tures. ACM Trans. Access. Comput. 12, 3, Article 14 (aug 2019), 30 pages.
https://doi.org/10.1145/3340319
[67] David W Schloerb, Orly Lahav, Joseph G Desloge, and Mandayam A Srinivasan.
2010. BlindAid: Virtual environment system for self-reliant trip planning and
orientation and mobility training. In 2010 IEEE Haptics Symposium. IEEE, 363–
370.
[68] Alexa F Siu, Mike Sinclair, Robert Kovacs, Eyal Ofek, Christian Holz, and Edward
Cutrell. 2020. Virtual reality without vision: A haptic and auditory white cane
to navigate complex virtual worlds. In Proceedings of the 2020 CHI conference on
human factors in computing systems. 1–13.
[69] Zach Smith and Bre Pettis. 2008. Thingiverse. https://www.thingiverse.com/
last accessed 12 April 2022.
[70] Statista. 2019. Leading barriers to mass adoption of VR according to XR pro-
fessionals worldwide as of the 3rd quarter of 2019. https://www.statista.com/
statistics/1099109/barriers-to-mass-consumer-adoption-of-vr/ last accessed 7
July 2015.
[71] Supermedium. 2015. A-Frame. Cross-platform.
[72] Manohar Swaminathan, Sujeath Pareddy, Tanuja S. Sawant, and Shubi Agarwal.
2018. Video Gaming for the Vision Impaired. ASSETS 18 (October 2018). https:
//doi.org/10.1145/3234695.3241025
[73] Juan R Terven, Joaquin Salas, and Bogdan Raducanu. 2014. Robust head gestures
recognition for assistive technology. In Mexican Conference on Pattern Recognition.
Springer, 152–161.
[74] Shari Trewin, Vicki L Hanson, Mark R Laf, and Anna Cavender. 2008. PowerUp:
an accessible virtual world. In Proceedings of the 10th international ACM SIGAC-
CESS conference on Computers and accessibility. 177–184.
[75] Dimitrios Tzovaras, Konstantinos Moustakas, Georgios Nikolakis, and Michael G
Strintzis. 2009. Interactive mixed reality white cane simulation for the training
of the blind and the visually impaired. Personal and Ubiquitous Computing 13, 1
(2009), 51–58.
[76] Dimitrios Tzovaras, Georgios Nikolakis, George Fergadis, Stratos Malasiotis, and
Modestos Stavrakis. 2002. Design and implementation of virtual environments
training of the visually impaired. In Proceedings of the ffth international ACM
conference on Assistive technologies. 41–48.
[77] Pablo Vera, Daniel Zenteno, and Joaquín Salas. 2014. A Smartphone-Based Virtual
White Cane. Pattern Anal. Appl. 17, 3 (aug 2014), 623–632. https://doi.org/10.
1007/s10044-013-0328-8
[78] Inc VRChat. 2017. VRChat. Oculus Quest, Microsoft Windows.
VRBubble
[79] Bruce N. Walker and Jefrey Lindsay. 2006. Navigation Performance With a
Virtual Auditory Display: Efects of Beacon Sound, Capture Radius, and Practice.
Human Factors 48, 2 (2006), 265–278.
[80] Dean A. Waters and Husam H. Abulula. 2001. The virtual bat: Echolocation in
virtual reality.
[81] Ryan Wedof, Lindsay Ball, Amelia Wang, Yi Xuan Khoo, Lauren Lieberman,
and Kyle Rector. 2019. Virtual showdown: An accessible virtual reality game
with scafolds for youth with visual impairments. In Proceedings of the 2019 CHI
conference on human factors in computing systems. 1–15.
[82] Thomas Westin. 2004. Game accessibility case study: Terraformers–a real-time
3D graphic game. In Proceedings of the 5th International Conference on Disability,
Virtual Reality and Associated Technologies, ICDVRAT.
[83] Matt Wilkerson, Amanda Koenig, and James Daniel. 2010. Does a Sonar System
Make a Blind Maze Navigation Computer Game More "Fun"?. In Proceedings of
the 12th International ACM SIGACCESS Conference on Computers and Accessibility
(Orlando, Florida, USA) (ASSETS ’10). Association for Computing Machinery,
New York, NY, USA, 309–310. https://doi.org/10.1145/1878803.1878886
ASSETS ’22, October 23–26, 2022, Athens, Greece
[84] Jef Wilson, Bruce N. Walker, Jefrey Lindsay, Craig Cambias, and Frank Dellaert.
2007. SWAN: System for Wearable Audio Navigation. In 2007 11th IEEE Interna-
tional Symposium on Wearable Computers. 91–98. https://doi.org/10.1109/ISWC.
2007.4373786
[85] Yuhang Zhao, Cynthia L Bennett, Hrvoje Benko, Edward Cutrell, Christian Holz,
Meredith Ringel Morris, and Mike Sinclair. 2018. Enabling people with visual
impairments to navigate virtual reality with a haptic and auditory cane simulation.
In Proceedings of the 2018 CHI conference on human factors in computing systems.
1–14.
[86] Yuhang Zhao, Edward Cutrell, Christian Holz, Meredith Ringel Morris, Eyal Ofek,
and Andrew D Wilson. 2019. Seeingvr: A set of tools to make virtual reality more
accessible to people with low vision. In Proceedings of the 2019 CHI conference on
human factors in computing systems. 1–14.
[87] Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2018. A face
recognition application for people with visual impairments: Understanding use
beyond the lab. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems. 1–14.
 “It’s Just Part of Me:” Understanding Avatar Diversity and
Self-presentation of People with Disabilities in Social Virtual
Reality
Kexin Zhang Elmira Deldari
University of Wisconsin-Madison Univ. of Maryland, Baltimore County
Madison, Wisconsin, USA Baltimore County, Maryland, USA
kzhang284@wisc.edu edeldar1@umbc.edu
Yaxing Yao
Univ. of Maryland, Baltimore County
Baltimore County, Maryland, USA
yaxingyao@umbc.edu
ABSTRACT
In social Virtual Reality (VR), users are embodied in avatars and
interact with other users in a face-to-face manner using avatars as
the medium. With the advent of social VR, people with disabilities
(PWD) have shown an increasing presence on this new social me-
dia. With their unique disability identity, it is not clear how PWD
perceive their avatars and whether and how they prefer to disclose
their disability when presenting themselves in social VR. We fll this
gap by exploring PWD’s avatar perception and disability disclosure
preferences in social VR. Our study involved two steps. We frst con-
ducted a systematic review of ffteen popular social VR applications
to evaluate their avatar diversity and accessibility support. We then
conducted an in-depth interview study with 19 participants who
had diferent disabilities to understand their avatar experiences.
Our research revealed a number of disability disclosure preferences
and strategies adopted by PWD (e.g., refect selective disabilities,
present a capable self). We also identifed several challenges faced
by PWD during their avatar customization process. We discuss the
design implications to promote avatar accessibility and diversity
for future social VR platforms.
CCS CONCEPTS
• Human-centered computing → Virtual reality; Accessibility
technologies.
KEYWORDS
Social VR, avatar, self-perception, disability disclosure, visual im-
pairments, d/Deaf and heard of hearing
ACM Reference Format:
Kexin Zhang, Elmira Deldari, Zhicong Lu, Yaxing Yao, and Yuhang Zhao.
2022. “It’s Just Part of Me:” Understanding Avatar Diversity and Self-presentation
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544829
Zhicong Lu
City University of Hong Kong
Kowloon, Hong Kong, China
zhicong.lu@cityu.edu.hk
Yuhang Zhao
University of Wisconsin-Madison
Madison, Wisconsin, USA
yuhang.zhao@cs.wisc.edu
of People with Disabilities in Social Virtual Reality. In The 24th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22),
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 16 pages.
https://doi.org/10.1145/3517428.3544829
1 INTRODUCTION
With the rising popularity of virtual reality (VR), social VR is be-
coming the mainstream of the online social eco-system. Social VR
refers to VR platforms where people communicate and socialize in
the form of avatars [26]. One iconic characteristic of social VR is the
embodied “face-to-face” interaction from a frst-person perspective
through avatars. By customizing their avatars, users can craft and
maintain any characters they prefer in a virtual world.
Due to the deep embodiment of a user with their avatars, to
some degree, a customized avatar can be considered as a proxy
of the user themselves [56]. A myriad of research has explored
how people use avatars to shape their self-images in virtual social
spaces, including both PC-based virtual worlds [41] and the more
immersive social VR [25, 26]. Research has found that while many
people designed their avatars to refect their physical appearance
in the real world [57], some people leveraged the opportunity to
experiment with diferent aspects of their personalities [18, 24]
and even explored completely diferent identities [81]. However,
most prior works focused on the self-presentation preferences of
people without disabilities. People with disabilities (PWD), given
their unique disability identity, may have diferent perceptions and
preferences when constructing self-images via avatars in social VR.
Research on PWD’s self-presentation in social VR is in its in-
fancy. On traditional social media platforms, PWD have a strong
presence and use various strategies to construct their online images
as others do [72, 85]. Unlike people with typical abilities, disabil-
ity disclosure is a unique perspective that PWD need to consider
during self-presentation. Many PWD are cautious about disclos-
ing their disability-related vulnerabilities, and some intentionally
hide their disability to avoid potential risks, such as loss of job
opportunities [90] and cyber-bullying [46]. In contrast, some proac-
tively disclose their disabilities, especially on dating platforms, to
build trust and flter potential partners [65]. Unlike the 2D text-
or image-based social media, social VR provides a more embod-
ied and immersive experience via avatars, which can potentially
ASSETS ’22, October 23–26, 2022, Athens, Greece
change PWD’s self-presentation preferences and their willingness
to disclose their disabilities. To promote safe and inclusive social
VR experiences for PWD, the accessibility and VR communities are
in need of a thorough understanding of PWD’s avatar perceptions
and disability disclosure preferences on the emerging social VR
platforms.
Our research aims to fll the gap by investigating how PWD de-
sign and use their avatars for disability disclosure and self-presentation
in social VR. Given that social VR is an emerging but premature
medium that lacks sufcient accessibility support, we also explore
the status quo of avatar diversity and accessibility on commercial
social VR platforms to reveal the barriers faced by PWD in the
avatar customization process. Specifcally, we aim to answer the
following research questions:
• RQ1: Whether and how is avatar diversity supported on the
mainstream commercial social VR platforms?
• RQ2: Whether and how do PWD disclose their disabilities
when presenting themselves via avatars?
• RQ3: What challenges do PWD face during the avatar design
and creation process?
To answer these research questions, we frst systematically re-
viewed 15 commercial social VR applications to understand what
avatar features are supported for disability representation. We then
conducted in-depth semi-structured interviews with eight visually
impaired people, nine d/Deaf and hard of hearing people, and two
people with multiple disabilities to explore their avatar perceptions
and customization experiences in the social VR context. Our fnd-
ings revealed a spectrum of disability disclosure strategies adopted
by PWD for self-presentation, from accurately disclosing one’s dis-
ability, to selectively presenting a particular aspect of the disability,
presenting the changes in one’s ability, to hiding one’s disability to
construct a diferent self. Our study highlighted the lack of avatar
diversity for disability representation and emphasized PWD’s needs
to fexibly control their disability disclosure in diferent social con-
texts. We also identifed the avatar accessibility barriers faced by
PWD and suggested design implications.
This paper makes three contributions. First, to the best of our
knowledge, this is the frst research that investigates social VR
avatar diversity and self-presentation from the lens of disability.
Second, our research presents rich and in-depth data to uncover
PWD’s diferent disability disclosure strategies and preferences.
Third, design implications are derived to inspire a more accessible
and inclusive avatar experience for PWD in social VR.
2 RELATED WORK
2.1 Self-presentation in Virtual Worlds
Self-presentation, also known as impression management, refers
to the ways that a person “conveys an impression to others which
is in his interests to convey” [30]. Via self-presentation, people
can selectively craft, present, and maintain specifc facets of their
identities based on diferent audiences and social settings [1, 22,
43, 63]. A myriad of research has explored how people manage
their identities on social media (e.g., Facebook, Twitter), revealing
the complexity of self-presentation from various aspects, such as
beautifed real self on anonymous social platforms [29, 87, 88],
Zhang et al.
multiple online identities for diferent audience groups [17], and
the stereotypical representation of genders [3].
With the advances in computer graphic technology and hard-
ware support, social media expands from 2D to 3D avatar-based
systems [25]. Instead of using text and images to construct self-
image, people can directly create and personalize an avatar as their
virtual representation and manage the impression from others via
the avatar. Many researchers explored users’ digital representations
via avatars in PC-based virtual worlds, including both the game
worlds (e.g., World of Warcraft [21]) and the social worlds (e.g., Sec-
ond Life [48]). In the virtual game worlds where a limited number
of pre-defned avatars were provided, many users tended to select
avatars that stood out, followed a trend [18], or served the role they
planned to play in the game [27]. In contrast, social worlds gave
users more fexibility to design their own avatars, for example, by
adjusting and customizing diferent body features (e.g., hair, eyes,
skin color). Without restrictions from the game theme, many users
chose to create avatars that resembled themselves in the real world
[18, 71, 82]. For example, Ducheneaut et al. [18] explored users’
avatar customization across three virtual worlds (i.e., Maple Story,
World of Warcraft, and Second Life), and found that more users
preferred reproducing some of their physical characteristics in the
social virtual worlds than in the game worlds. Koles et al. [44] also
showed that the majority of users in Second Life tended to use
their physical selves as the starting point to create their virtual
representation.
With the high fexibility in avatar customization, people also
have the freedom to create avatars that are diferent from their
physical appearances, thus better shaping their online identities.
As a result, some people used avatars to experiment with diferent
aspects of their personalities [42, 78, 79]. For example, Kafai et
al. [42] surveyed 438 tween players in a virtual social community
named Whyville and found that most tweens did not construct
their avatar appearances based on their real selves. Instead, they
designed avatars to achieve recognition from others or to reveal
specifc aspects of their “real” selves, including those they desired
but could not present in real life. Via avatar creation, users can even
create and explore a completely diferent identity, such as an avatar
with a diferent gender [37] or a diferent race [62], which may help
them discover their true selves and increase their self-esteem [5].
Most prior work on avatar perception focused on the majority
group. Limited attention has been paid to the self-presentation
and identity disclosure of the under-representative user groups,
especially people with disabilities.
2.2 Self-disclosure by People with Disabilities in
Virtual Worlds
Diferent from self-presentation where people can construct and
present any images that they prefer to the audience, self-disclosure
refers to the act of revealing any messages about oneself to others
[14, 29, 31]. While self-disclosure is involved in self-presentation
strategies to help build closer relationships [70], it also poses po-
tential risks of exposing one’s vulnerabilities, especially to the
unknown or anonymous audience online [53]. Prior research has
Avatar Diversity and the Self-presentation of People with Disabilities in Social VR ASSETS ’22, October 23–26, 2022, Athens, Greece

explored the online self-disclosure experiences of various under- 2.3 Avatars Design and Self-presentation in
representative groups, such as racial minority [36, 49, 50] and gen- Embodied Social VR
der minority [9, 11, 20, 60]. In this section, however, we focus on
In recent years, social VR has gained increasing attention. Unlike
the work for people with disabilities (PWD).
the PC-based virtual worlds where users see avatars on a 2D screen
Disability disclosure has always been an important topic for
and can only control the avatars via a keyboard and a mouse, social
PWD even before social media platforms emerged [38, 64, 68, 80].
VR incorporates full-body tracking, providing a more embodied
Prior research focused primarily on the physical working context,
frst-person avatar experience as well as richer and more immer-
showing that disability disclosure could potentially assure appro-
sive social interactions [25]. As a result, social VR provides new
priate workplace accommodations and increase workplace diver-
afordances for avatar design and interaction [34, 45, 59, 83]. For
sity and inclusiveness for PWD [80]. However, it may also lead
instance, Kolesnichenko et al. [45] interviewed industry experts
to negative employment consequences for PWD, such as lowered
who work for diferent commercial social VR platforms and uncov-
supervisor expectations, isolation from colleagues, and increased
ered the current avatar design practices from diferent aspects, such
possibility of termination [38, 64]. Because of these, in working or
as locomotion, avatar aesthetics, and avatar’s relation to one’s vir-
daily living contexts, disability disclosure is highly contextualized,
tual identity. Menon [59] also researched the relationship between
and infuenced by many factors such as employers, managers, and
avatar customization and embodiment in a virtual job interview
workplace climate [64, 80].
context, highlighting the needs for basic avatar customization.
The advent of online social platforms (e.g., social media, social
Some research specifcally focused on avatar-based self-presentation
virtual worlds) afords new forms of social interactions and activi-
in social VR and how it afected users’ behaviors and interactions
ties, along with new social relationships. These online platforms
[25, 26, 55]. For example, Freeman et al. [25] interviewed 30 people
bring PWD more opportunities to interact with others and get ac-
about their avatar perception and social interaction experiences
cess to resources and communities [74, 76]. For example, Boellstorf
in social VR. They found that most people considered the avatars
[6] built “Ethnographia”, a virtual island in Second Life, to investi-
to be themselves and strived to make their avatars similar to their
gate how digital spaces infuenced PWD’s experiences and found
physical appearances. This research also revealed how avatar gen-
that building virtual worlds enhanced PWD’s sense of ability and
ders and race afected people’s experience. For example, female
helped them reconstruct identities. With these online social plat-
avatars received better frst impression and nicer treatment but
forms, PWD were able to manage their disability disclosure, thus
also led to more harassment, and non-white avatars could bring
achieving a “levelling ground” where they could be treated on their
certain social stigma. Moreover, Maloney and Freeman [55] studied
merits as a person without being shadowed by their disabilities
how and why people disclose their information in social VR. They
[7]. However, disability disclosure in online communities can also
argued that creating avatars that look like one’s physical self can
bring risks to PWD [69, 74]. For example, Ringland [69] conducted
disclose important personal information such as gender, race, and
a 200-hour observation in Autcraft, an online community for chil-
appearance. While this helped build close connection with others,
dren with autism, and found that the autistic identity in virtual
it can potentially lead to privacy risks.
worlds could be both a source of empowerment and a source of
With more realistic avatars and more embodied interactions,
harassment and violence.
social VR can also afect PWD’s self-presentation and disability dis-
Researchers have explored PWD’s self-disclosure strategies and
closure preferences. Some researchers have noticed the importance
preferences in diferent online communities [6, 28, 40, 65, 69]. For
of this research direction. Boellstorf [6] emphasized the importance
example, Porter et al. surveyed 91 adults with and without disabil-
of distinguishing “virtual world” and “virtual reality” for under-
ities to understand the needs for disability disclosure on online
standing PWD’s avatar experience. Mott et al. [61] also stressed
dating platforms. They found a higher expectation of the disclosure
the needs for supporting more diverse avatar representations for
of visible disabilities than invisible disabilities [65]. By designing a
PWD in social VR. However, to our knowledge, no research has
movement-based virtual game for young people using wheelchair
thoroughly explored PWD’s avatar design experiences in the em-
and exploring their avatar preferences, Gerling et al. found that
bodied social VR. Our research contributes to this line of research
while 6 out of 8 participants depicted wheelchairs as indispensable
by developing a deep understanding of the current avatar diver-
parts of their self-images, only two participants were willing to use
sity practices, as well as PWD’s self-presentation challenges and
avatars with mobility disabilities in the game [28]. Recently, Davis
strategies in social VR.
and Stanovsek conducted a 3-year ethnography study to explore
how PWD customize their avatars in Second Life and found that 3 STUDY I: APPLICATION REVIEW: AVATAR
many participants used non-human avatars to free themselves from
their visible disabilities [16].
DIVERSITY ON COMMERCIAL SOCIAL VR
Compared to the conventional social media and virtual worlds, PLATFORMS
the emerging social VR brings unique embodied experiences, which The goal of this study is to build a comprehensive understanding of
may potentially afect PWD’s willingness of disability disclosure. the avatar diversity and accessibility practices on current commer-
However, little work has investigated the infuence of social VR on cial social VR platforms (RQ1). Specifcally, we aim to uncover 1) the
PWD’s self-presentation and disability disclosure preferences. general avatar design practices: what types of avatars are supported
and how users can select or customize an avatar on diferent social
VR platforms; 2) avatar diversity support: what avatar features are
provided to enable disability disclosure for PWD; and 3) avatar
ASSETS ’22, October 23–26, 2022, Athens, Greece
accessibility: what accessibility features are provided for PWD to
perceive, use, and customize avatars.
3.1 Method
We conducted a systematic review of ffteen popular commercial
social VR applications. To determine which social VR applications
to review, we conducted an exhaustive search on three mainstream
VR application stores, including Oculus, Viveport, and Steam. Our
search focused on applications available in the United States in
November and December 2021. We frst searched the keyword “so-
cial” in these stores and identifed a total of 133 VR applications:
47 from Oculus, 54 from Viveport, and 50 from Steam (18 apps
were available across multiple stores). To further narrow down
the scope, we fltered the applications by checking the application
descriptions and only focused on the applications whose descrip-
tions clearly indicated a social or collaborative nature.Finally, we
excluded VR applications that showed an intense gaming nature
(e.g., sports, shooting games), which can distract users from socializ-
ing, such as EchoVR, StarTrek: Bridge Crew, and SpaceteamVR. We
also consulted other related work and online articles about popular
social VR applications to ensure not missing any mainstream social
VR platforms [2, 52, 86]. This process resulted in ffteen social VR
applications with strong social nature, including Rec Room, VR-
Chat, Horizon Worlds, vTime XR, AltspaceVR, Bigscreen, Alcove,
Half + Half, Horizon Venues, Villa, Arthur, ENGAGE, Multiverse,
PokestarVR, and Spatial (details can be found in Table 2 in Appen-
dix).
Two researchers on the team reviewed all applications indepen-
dently. We adopted a depth-frst traverse strategy, clicking all avail-
able buttons and menu items in the avatar customization process.
During the review, researchers video-recorded and took notes of
all avatar options and the interaction process. The two researchers
then discussed their results to ensure the reliability of the review.
Following the same strategy, we reviewed the general setting in
each social VR application to examine what accessibility features
were supported. Any accessibility features that could infuence or
be applied to the avatar selection and customization process were
documented. All applications were reviewed with Oculus Quest 2.
3.2 Review Results
3.2.1 Avatar Creation Methods. We identifed four ways to cre-
ate and customize avatars in social VR: (1) Full Avatar Selection: se-
lecting from a set of pre-determined avatars provided by the system,
(2) Avatar Feature Customization: customizing diferent components
of a human avatar (e.g., eyes, hair styles), (3) Photo-based Avatar
Generation: uploading a photo to the system and generating a cor-
responding avatar automatically, and (4) Third-party Avatar Import:
designing an avatar using a third-party platform and uploading it
to the social VR application.
Most social VR platforms (12 out of 15) supported Avatar Feature
Customization, allowing users to customize the avatar’s physical
features (e.g., skin tone, body shape, eyes), clothing (e.g., outfts,
accessories), or both. Three platforms (VRChat, Half+Half, and Mul-
tiverse) employed the Full Avatar Selection model. Notably, VRChat
ofered a set of 80 pre-defned avatars for users to select and sup-
ported various avatar types, including both humanoid avatars and
Zhang et al.
non-humanoid avatars, such as cartoons, robotic fgures, animals,
and objects. VRChat was also the only platform that enabled users
to design and import their own avatars from third-party platforms.
Additionally, three platforms (Villa, Arthur, and Spatial) employed
Photo-based Avatar Generation. Table 2 in Appendix listed the
avatar customization methods supported by diferent platforms.
Beyond individual applications, the Oculus system provided its
own avatar system (i.e., Meta Avatars) that employed the Avatar
Feature Customization model. When frst logged in to an Oculus
device, users were automatically directed to Horizon Home1 to
customize their avatars for the whole VR system. Three platforms–
Horizon Worlds, Horizon Venues, Alcove–directly adopted the Meta
Avatars, so that a user could continue using the same avatar from
the Oculus system.
Our review of the avatar customization process confrmed the
results from Kolesnichenko et al.’s work [45] and further extended
the scope to more widely-used social VR applications.
3.2.2 Avatar Realism. Most social VR platforms provided only
humanoid avatars. We investigated the avatar realism by examining
the completeness of the humanoid avatars. We found that only
three platforms–VRChat, vTime XR, and ENGAGE–provided full
body avatars, including head, neck, torso, arms, hands, legs, and
feet. All other 12 platforms focused on rendering the upper body
of the avatars, presenting at least the avatars’ head and hands.
Specifcally, fve platforms did not show the avatars’ arms but only
presented fowing hands; avatars in Rec Room and Bigscreen did
not have necks; and Villa’s avatars did not even have a torso. In
terms of avatar rendering details, except for Rec Room, all social
VR platforms we reviewed rendered avatar fngers and tracked the
users’ fnger movement via the VR controllers. Table 2 in Appendix
provides details of the avatar realism on all platforms.
3.2.3 Disability Representation in Avatars. Our review uncov-
ered the limited avatar diversity support for PWD. We found that
most commercial social VR platforms did not ofer any disability-
related avatar features. Meta Avatars was the only avatar system
that provided hearing device features for people who are d/Deaf
or hard of hearing. Two types of hearing devices were supported:
cochlear implants and hearing aids. A user can put the hearing de-
vice on the avatar via three options: left ear only, right ear only, and
both ears (Fig 1A). This feature was provided under the category of
“Clothing.” However, no other assistive devices or disabilities were
supported. Additionally, we found some “near disability-related”
features that may be used to indicate a disability. Specifcally, Bige-
screen provided an eye patch feature under “Glasses” category,
which users can add onto their avatars to present their visual im-
pairments or eye injuries. However, we acknowledge that this fea-
ture might be designed as an eye decoration (e.g., to imitate a pirate)
instead of a disability feature.
Besides disabilities, some applications allowed users to customize
wrinkles and face lines on avatars to refect age and represent older
adults. For example, Meta Avatars provided fve levels of face lines
with diferent depth and number of wrinkles on avatar’s face to
1 A virtual home in Oculus Quest 2, which functions like the homepage of 2D platforms
and enables certain social activities [66].
Avatar Diversity and the Self-presentation of People with Disabilities in Social VR
demonstrate diferent ages. vTime XR ofered four sagging levels
for avatars’ facial skin to present age (Fig 1B).
Figure 1: Avatar Diversity: A. Hearing devices in Meta
Avatars; B. Sagging facial skin in vTime XR Avatar.
3.2.4 Accessibility Features for Avatar Customization. The
accessibility features on social VR platforms were limited. While
some platforms supported standard accessibility features that were
commonly used on computer and smartphone interfaces (e.g., mag-
nifcation, color correction), no features were designed specifcally
for avatar customization. Specially, AltspaceVR allowed users to
adjust the scale of the avatar interface. The Oculus system also
provided visual augmentations (i.e., color correction, text size ad-
justment) that can be applied to any applications on Oculus devices.
Moreover, most social VR platforms allowed adjustment to the
audio and haptic feedback, which could be helpful for PWD.
By reviewing mainstream social VR platforms, we developed a
comprehensive understanding of the current practices of the avatar
design, diversity, and accessibility. The results of this study inspired
and served as a solid grounding for our following interview study
(Study II). First, the review identifed existing disability features
for avatars and the social VR platforms that supported such fea-
tures, which enabled us to better understand participants’ social VR
experiences and disability disclosure choices in an in-depth inter-
view. Second, the limited disability features (i.e., hearing aids, eye
patches) identifed in this study helped us narrow down the focus
to participants who were visually impaired and deaf or hard of
hearing, since they were the only disability groups that had avatar
representations in social VR.
4 STUDY II: INTERVIEWS:
SELF-PRESENTATION OF PEOPLE WITH
DISABILITIES VIA AVATARS
In this study, we investigated how PWD design and craft their
avatars to disclose their disabilities and present themselves in social
VR (RQ2) as well as their challenges and needs during the avatar
creation and customization process (RQ3).
4.1 Method
We conducted semi-structured interviews with 19 PWD via Zoom
in February and March 2022. The study was approved by the Insti-
tutional Review Board (IRB).
4.1.1 Participants. Our recruitment focused on two types of sen-
sory disabilities, (1) d/Deaf or Hard of Hearings (DHH) people and
(2) visually impaired (VI) people, since they are common and visible
ASSETS ’22, October 23–26, 2022, Athens, Greece
disabilities (or disabilities that can be indicated by visible assis-
tive technologies). Both disabilities can be easily noticed in the
real world and thus can be refected via avatar design. For exam-
ple, blind people may use a white cane, and d/Deaf people may
wear hearing aids or may sign. We recruited 19 participants (11
female, 7 male, and 1 transgender). Their ages ranged from 20 to 58
(���� = 33, �� = 11.57). Among the participants, eight had visual
impairments (V-P1 to V-P8), nine were DHH (H-P1 to H-P9), and
two had multiple disabilities (M-P1, M-P2). M-P1 was both blind
and DHH, while M-P2 was blind and had a prosthetic leg. Table 1
shows participants’ detailed information.
We spread our recruitment information via (1) the mailing lists
of non-proft disability organizations, such as the National Feder-
ation of the Blind and National Association of the Deaf, and (2)
mainstream social media platforms, such as Facebook groups, Twit-
ter, and Instagram. Interested participants could fll in a screening
survey with their age, disability conditions, and general avatar ex-
periences. Participants were eligible if they were over 18 and had
visual impairments or were d/Deaf or hard of hearing. We limited
the recruitment to individuals who spoke English. If selected to
participate, participants were asked to sign a consent form prior
to the interview. Upon completion, participants received $15 as
compensation.
4.1.2 Interview protocol. The interview included three phases. The
frst phase focused on participants’ background information, in-
cluding demographics (age and gender), self-reported disability, and
experiences with VR and social VR.
The second phase focused on participants’ self-presentation and
disability disclosure via avatars. We frst asked participants to send
photos of their avatars or screen share their avatars with us to
demonstrate their avatar design. We asked about how they de-
signed their avatars, why they customized their avatars in this
way, and whether their avatars involved any features to disclose
their disabilities. If the participants disclosed their disabilities via
their avatars, we further asked how they disclosed their disabilities,
why they wanted to do so, and their experience in social VR after
they disclosed disabilities via avatars. If participants’ avatars did
not indicate their disabilities, we asked about their willingness of
disclosing their disabilities via avatars and the reason. Additionally,
we asked participants whether and how they had disclosed their
disabilities through any other means or on any other online social
platforms and the rationales. Participants also discussed whether
and how they wanted the social VR platforms to support disability
representations via avatars.
The last phase of the interview focused on avatar creation acces-
sibility. We asked about participants’ experiences, difculties, and
strategies when creating and customizing avatars and the types
of assistance they needed to complete the avatar creation process.
We also asked about their suggestions for a more accessible avatar
experience.
4.1.3 Data recording and analysis . Upon participants’ consent,
all 19 interviews were recorded and auto-transcribed by Zoom.
Participants had the option to turn of their cameras during the
interview, although it was not required. Two researchers manually
cleaned all transcripts by checking the recorded interviews.
ASSETS ’22, October 23–26, 2022, Athens, Greece
We conducted thematic analysis [8, 13] to identify repetitive
patterns and themes in the interviews. First, we manually organized
all qualitative data in an Excel sheet and selected fve representative
transcripts (2 DHH participants, 2 VI participants, and 1 participant
with multiple disabilities) as samples. Three researchers coded all
samples independently at the sentence level with open coding.
Then, they discussed and reconciled their codes to resolve any
diferences, and developed an initial codebook upon agreement.
Next, two researchers divided the rest of the transcripts based on the
participants’ disabilities. Specifcally, one researcher analyzed all
VI participants’ data (including the two participants with multiple
disabilities) and the other researcher analyzed all DHH participants’
data. During this process, the two researchers regularly checked
each other’s codes and discussed as needed to ensure consistency.
New codes were added to the codebook based on the agreement
between the two researchers. In the meantime, the third research
oversaw all these activities to ensure a high-level agreement. The
fnal codebook contained over 120 codes. We categorized all the
codes into high-level themes using afnity diagram and achieved
twenty themes.
4.2 Findings
4.2.1 Experience with Avatars and Social VR Platforms. Six-
teen out of 19 participants had social VR experiences. In general, we
found that DHH participants had a richer experience with social VR
avatars than VI participants. Except for H-P3, all DHH participants
had used social VR multiple times. Five DHH participants were
frequent users with experiences of more than a year. In contrast, VI
participants had limited social VR experience, with most of them
only trying it a few times. Three VI participants did not have so-
cial VR experience, but they used avatars on conventional social
media, such as Snapchat Bitmoji and Meta Avatars for Instagram.
Several VI participants mentioned that they attempted to customize
their avatars on social VR several times but were blocked when
setting up an account. We report the accessibility challenges faced
by participants in Section 4.2.5.
The two most commonly used social VR platforms by our par-
ticipants were VRChat (7 participants) and Rec Room (5). Most
participants used social VR via head-mounted VR devices and the
most commonly used devices were Oculus Quest, Oculus Rift, and
Valve Index. However, three participants (H-P1, V-P2, V-P6) pre-
ferred using desktop-based social VR due to the accessibility issues
of VR devices.
Participants used social VR for multiple reasons. Most partici-
pants used social VR to communicate with friends (7 participants)
and play games (6). H-P4 specifcally wanted to know other DHH
people and connect with DHH community in social VR. Six par-
ticipants used social platforms for professional purposes, such as
hosting VR meetups (H-P1), promoting rights for PWDs as accessi-
bility activists (H-P1), and using social VR as an education platform
(e.g., M-P1, V-P7, H-P4). Notably, three participants (H-P4, H-P6,
H-P8) used social VR for American Sign Language (ASL) educa-
tion. They learned or taught ASL in a VRChat community called
“Helping Hands” [84], which was a community for DHH people
to communicate via sign language (see details about ASL in VR in
Section 4.2.4).
Zhang et al.
4.2.2 Disability Disclosure via Avatars. Our study indicated
that disability disclosure via avatars was an essential strategy of
self-presentation for PWD. We identifed participants’ diferent
disability disclosure preferences in social VR.
Refect one’s physical self. The majority of our participants (17
out of 19) designed avatars to refect their physical appearances in
real life, including both facial features and outfts. Some participants
even hoped to craft the fne details (e.g., makeup, accessories) of
the avatars to show their daily styles, habits, and values in real life.
As part of their physical appearance, eight participants (e.g.,
H-P3, H-P7, V-P1) expressed their willingness to disclose their
disability via their avatars since they believed that the disability “is
part of me” (H-P3). As H-P7 indicated, “I have [a cochlear implant]
on [my avatar] all the time really, just because that’s what I do in real
life. I like my avatar to represent me as realistic as possible or as close
to [myself], so if I have a cochlear implant I’m not ashamed of it.” Two
participants (M-P1, H-P8) specifcally emphasized that when they
disclosed their disabilities on social platforms, they never aimed
to highlight their disability since the disability was just like other
physical features, such as hairstyle, skin color, and gender. As M-P1
stated, “Because that’s who I am. I don’t see any reason to not disclose
[my disability]. It’s a part of who I am. And I wouldn’t like try to
pretend that I was a diferent race or try to pretend that I was not
female cisgender kind of person, so I wouldn’t pretend that I didn’t
have a disability.”
Interestingly, although most participants wanted their avatars to
refect their physical selves as much as possible, H-P2 did not want
the details to be completely the same: “I just tried to make it look
kind of like me, but not too much like me, because it’s kind of creepy
when they look like twins. [The avatars] don’t ever look completely
like you, especially the hair, you cannot replicate someone’s hair [with
the current VR technology].” With the limitation of current avatar
realism and the Uncanny Valley efect2 , she discussed the boundary
between the physical self in real life and the virtual image presented
by avatars in the virtual world: “I don’t want [avatars] to be too much
like me, because it’s not real life. Don’t try to make it be real life when
it’s not like [real life].”
Refect one’s physical self with selective disabilities. Unlike
most participants who preferred disclosing their disabilities entirely,
M-P1 selectively disclosed her disability. M-P1 experienced both
visual and hearing loss, and she decided which disability to disclose
based on the visibility of the disabilities. She used both a white cane
and hearing aids in the real life. However, she would only add a
white cane to her avatar to signify her visual impairments but not
disclose her hearing loss. This is because hearing aids were not vis-
ible to people in the real world in most cases. Interestingly, M-P1’s
consideration of disability visibility was to fulfll the expectation of
the audience, specifcally people who knew her in real life. As she
explained,
“I do use hearing aids, but I don’t think that they are visible partic-
ularly... so I was like, well I am not going to add a hearing aid, because
a lot of people probably don’t even know that I use [hearing aids],
because I don’t think they’re that visible, whereas people would know
that I use a cane. I was trying to make [my avatar] look like me, and
2 Uncanny Valley efect refers to viewer’s increased eerie feelings when an entity looks
highly human-like [35].
Avatar Diversity and the Self-presentation of People with Disabilities in Social VR ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Participants’ demographics and social VR experiences. * indicates avatar platforms demonstrated by our participants.

ID Age/ Self-reported Disability Assistive Tech Used Social VR Social VR Platforms Used
Sex Experience
H-P1 52/F Profound deaf since birth Cochlear implant 12 months VRChat, AltspaceVR, Spatial, Mozilla Hubs, Rec
Room, Meta Avatars*
H-P2 20/F Severe deaf in left ear Hearing aid Multiple times A social VR game, Snapchat*

H-P3 20/F 25% hearing loss in right ear Hearing aid Tried once VR Exhibition at a museum, Snapchat*, Meta
Avatars*, Sims*, Memoji*
H-P4 23/Trans Moderate hearing loss Hearing aids 24 months VRChat*

H-P5 29/F Moderate hearing loss Hearing aids Multiple times VRChat, Rec Room, Snapchat*, Meta Avatars*,
Roblox*
H-P6 25/M Profound deaf N/A 12 months VRChat*, SteamVR

H-P7 30/M Profound deaf Cochlear implant 12 months VRChat, Rec Room*, Lost Horizon, Meta
Avatars*
H-P8 25/M Deaf Hearing aids 24 months VRChat*, SteamVR

H-P9 29/F 80% hearing loss Hearing aids Multiple times A social VR app to play games with others,
Snapchat*
V-P1 28/M Blind White cane Few times Horizon Series, Meta Avatars*

V-P2 31/F Residual vision in one eye N/A 2 months Cardboard, Meta Avatars*

V-P3 26/M Loss of side vision N/A 24 months BeanVR, Snapchat*

V-P4 38/M Blind White cane, guide dog Multiple times VRChat, Rec Room, Oculus Venues, Xbox*

V-P5 32/F Only have vision in one eye White cane, guide dog N/A Snapchat*

V-P6 25/F Blind White cane N/A Twitter avatar*

V-P7 49/F Blind White cane Multiple times A social VR app with educational purpose, sev-
eral social VR games, Meta Avatars*
V-P8 53/F peripheral vision loss White cane, guide dog N/A Meta Avatars*

M-P1 58/F Blind since birth; hearing dis- Hearing aids, white Few months A social VR app for education, Meta Avatars*
ability since recently cane, guide dog
M-P2 38/M Blind; has a prosthetic leg White cane Multiple times Rec Room*

I think that if I had hearing aids, a lot of people would be like, why
do you have hearing aids, so why did you do that.”
Moreover, M-P1’s experience with diferent disabilities also infu-
enced her choice of disability disclosure. She preferred to disclose
only the “dominant disability” that can represent her most: “I have
had the hearing disability for a lot less time than the [blindness], I
mean I’ve been blind all my life, so pretty like used to [blindness].
I think that maybe if I had grown up hard of hearing or deaf that
would be more important to represent for me.”
Refect changes of the physical self. Notably, participants
who experienced acquired and progressive disabilities mentioned
that they wanted to use their avatars to refect their physical changes
and inform people about their current abilities. For example, H-P2,
who suddenly experienced one-side hearing loss at the age of 20,
emphasized her need to inform people of her current hearing ability
via her avatar: “Because people don’t know [my acquired disability],
especially when [my hearing loss] happens so suddenly to me. Liter-
ally people are like ‘Oh, you can still hear out with one ear, you’re
just fne.’ But it’s like, I have no ability to locate a noise. When there’s
any sort of background noise, I can’t hear anything that you say. The
whole world is just like quieter. People don’t realize that.”
Present a capable self. Uniquely for PWD, some participants
wanted to present a capable self via avatars, demonstrating that
they were as capable as people without disabilities in accomplish-
ing tasks. However, participants employed divergent strategies to
present their capabilities.
Four participants (H-P4, V-P3, V-P4, M-P2) chose to hide their
disabilities in avatars, so that their disabilities would not bias others
and overshadow their capability in social interactions. For example,
V-P3, who played multiplayer games in BeanVR, preferred not to
disclos his visual disability, because he was afraid of being judged
as a weak player. As V-P3 described, “Sometimes there are challenges,
when you are trying to make friends, people may like disregard your
friend request because [your] visual impairment. Sometimes you have
to keep it a secret. I was like looking for teammate on BeanVR, so that
you can join a game, nobody wanted to be my teammates at the end
because maybe my weakness.”
ASSETS ’22, October 23–26, 2022, Athens, Greece
Reversely, some participants (V-P1,V-P5) crafted their avatars
to show their disabilities and used them as a way to prove their
capability and independence in using this new technology. As V-
P1 said: “I see that avatar can be a symbol of hope, a symbol of I
can do this. For me, disability is not the end. I like to overcome my
disability and show others that I can overcome it. I’ve thought of ways
to overcome every obstacle. So, by [customizing my avatars], [I show
that] every person with disability can become independent.”
Present a professional self for disability education and
awareness. Nine participants (4 DHH and 4 VI participants, and M-
P1) disclosed their disabilities to trigger conversations and educate
the public about disability and equity. As H-P7 expressed, “I think
if people notice [the disability feature of my avatar], it can spark
an interesting discussion about oh look they’re wearing a cochlear
implant, what is that.” H-P1 further emphasized the importance of
educating people about DHH people: “Avatars [with disabilities]
were kind of cool, because that’s me educating people that deafness is
a spectrum, we don’t ofer in buckets.”
V-P7 shared her disability to advocate for social justice. As V-P7
said, “Part of the reason [of disclosing my disability] is that, I am an
advocate for social justice and all marginalized communities, and so
I want people to know that I am an ally, and I am here to support
and work for equitable opportunities for all individuals regardless of
what their diferent diversity might be, and so having my disability
out there on social media gives people a little more insight into me as
a person.”
Disclose disability selectively based on social contexts.
Some participants decided whether to disclose their disabilities
based on specifc social circumstances. Seven participants (three
DHH and four VI participants) considered the audience. Some pre-
ferred disclosing their disabilities only to the audience who they
knew in real life. For example, H-P1 did not want to disclose her
disability to random strangers in social VR: “I think the only situa-
tion I can think of where I may not disclose is when there are other
strangers, too many strangers. In a setting where you can just show up,
and a stranger can come run after you, I would not want to [disclose
my disability].”
V-P2 emphasized that disability was her privacy. She would not
share information about her disability on every occasion. Instead,
she would assess the situation and audience, then make the decision
accordingly. As she indicated, “Well, in general, I am a private person.
I don’t like too many people knowing about my disability, because
they haven’t been through what I’ve been through, they judge before
they even know. I am a really sensitive person, so somebody says that
to me may not be hurtful to them, [but hurtful to me].”
Present a diferent self that is not defned by disability.
While the majority of participants were willing to disclose their
disabilities via avatars, three participants (H-P4, V-P4, and M-P2)
held the opposite attitudes. H-P4 preferred not disclosing her dis-
ability at all because she did not want her disability to become her
representative identity and to overshadow her personality when
socializing with others: “I don’t like to disclose [my disability], be-
cause I don’t want that to be the initial impression that people have,
that this person is deaf. I just don’t feel a need to say anything unless
I have to. Because it might just be a personal thing, and I just don’t
want that to be characteristically associated. It’s not a bad thing. But
Zhang et al.
I’ve been that way (not disclosing disability unless necessary) all my
life, not just in VR.”
Moreover, some participants (H-P4, V-P4, and M-P2) viewed
social VR as a world where they can express themselves freely
and explore an ideal self that was diferent from the real world. As
V-P4 said, “Because I feel like in virtual reality, your disability really
shouldn’t matter because it’s virtual reality. And virtual reality you’re
not hampered or hindered or shouldn’t be any way. So my case of
blindness, plus a prosthetic leg, should not hold me back in virtual
reality.”
4.2.3 Avatar Customization for Disability Disclosure. While
most participants indicated a strong desire to be able to show their
disabilities via avatar design, they experienced various difculties
in practicing disability disclosure via avatars. In this section, we
reveal participants’ challenges in disability disclosure via avatars,
the strategies they adopted, and their desired features to support
disability representation.
Challenges in disclosing disability via avatars. We identifed
four main challenges that participants encountered when disclosing
disability via avatars.
First, almost all participants (18 out of 19) complained about the
lack of disability representations in avatar design. Seven partici-
pants mentioned not knowing or not being able to fnd any avatar
features that represent disabilities on the social platforms they used.
While some platforms support limited disability features (e.g., hear-
ing aids and cochlear implants in Meta Avatars), the choices and
customization options were limited. For example, H-P2 mentioned
that Snapchat only provided hearing aids for avatars, but people
who used cochlear implants did not want to use them to represent
their disabilities. Due to the lack of disability features, V-P5 used
regular avatar decorations (i.e., sunglasses) to present her low vi-
sion. However, it cannot clearly signify her disability since people
without disabilities also use it. As V-P5 said, “My avatar has sun-
glasses [since I’m light sensitive], but this doesn’t really make sense
because anybody can wear sunglasses. I honestly do wish that there is
more representation of people with disabilities.”
V-P4 further emphasized the importance of avatar diversity for
disabilities by comparing it to other minority groups, “I think it’s a
very good idea to have [disability-related] features, I might even use
them. Everyone’s talking about diversity and it’s all about race and
gender. But one common thing that gets left out is disability. I think
that you should have the option to represent, whether you want to use
that or not it’s up to you, but it never hurts to have.”
Second, current disability signifers for avatars were not always
compatible with other avatar features. For example, H-P1 tried to
put a cochlear implant on her avatar but ended up not using it since
it was blocked by the avatar’s hair: “The [Meta Avatars] do have a
cochlear implant you can add. But you can’t see it, because my hair
kind of covered it. I tried to spare [the hair] in a pony tail to see if
you could see the cochlear implants. But you couldn’t, so I put it back
down.”
Moreover, the unrealistic size of the virtual hearing devices also
prevented participants from applying them to their avatars. H-P5
explained her experience in Roblox: “The hearing aids are larger
than the character, so if you try to put them on, it’s seriously like
Avatar Diversity and the Self-presentation of People with Disabilities in Social VR
trying to put Barbie clothes on like a Little Tikes doll. They’re just
wrong, they’re just foating.”
Methods of disclosing disabilities via avatars. Participants
mostly disclosed their disabilities by accessorizing their avatars with
the assistive devices they used in real life. Three DHH participants
added cochlear implants or hearing aids to their avatars (H-P1 and
H-P7 on Meta Avatars, H-P3 on Snapchat). For VI participants, there
were no avatar features designed to represent visual impairments
on current social VR platforms. Two VI participants (V-P2 and V-
P5) thus put dark glasses on their avatars to signify their visual
impairments. “My avatar has sunglasses because when I go out, I
always have sunglasses with me, because I’m light sensitive. So that
is why my avatar is wearing sunglasses.” (V-P5).
Moreover, P-H6 added an ID badge and a mini-status to his
avatars to indicate his disability. As he described, “You can make
your own ID badge to explain that you’re Deaf or/and Hard of Hearing
and have it on your avatar so you can let anyone read it and learn
about you. And plus I have a little mini status that would show above
my avatar head [if you enabled it in the menu], it’ll say ‘I’m Deaf
:D.”’ (Fig. 2A)
Figure 2: A. H-P6’s mini-status in VRChat; B. H-P6’s sign
avatar with black-outlined hands and ID Badge
Alternative approaches to disclosing disabilities. Besides
avatars, participants (nine VI and seven DHH participants) used
other methods, such as bio, posts, photos, emoji, and YouTube
videos, to present their disabilities on social media.
Compared to the 2D social platforms, we found that many par-
ticipants (six DHH and two VI) were more willing to disclose their
disabilities via 3D avatars in social VR. The embodied experience
made participants feel more comfortable disclosing their disabili-
ties. In contrast, some participants (H-P4, H-P5) felt that expressing
their disabilities in a less embodied way (e.g., via text on social
media) was highlighting their disabilities rather than representing
themselves. As H-P5 noted, “The hearing aids are physical reality
for me, so I just have them on [my avatar]. I wouldn’t want to put
it in text. I wouldn’t want it to be [highlighted]. While [the hearing
aid] is visible it feels more subtle [than disclosing my disability in
text].” Meanwhile, some participants felt that the visual presenta-
tion of avatars was more noticeable than in other mediums. For
example, H-P2 claimed that avatar would be "the frst thing" that
people see, and H-P7 added that “it’s the easiest way of [disclosing
my disability]. I don’t think I would put it in like text information
like oh I’m deaf. Because I think most of the time well, probably no
one reads it.”
ASSETS ’22, October 23–26, 2022, Athens, Greece
In addition, H-P7 considered social VR to be a safer place to
disclose disability since he could easily move away from toxic con-
versations. As he indicated, “the freedom of moving around [with
avatars] makes you get the power to walk away [from ofensive com-
ments], you have a bit of control that you wouldn’t have in social
media.”
However, some VI participants preferred using text-based meth-
ods to disclose their disabilities other than avatars since avatars
were visual and not accessible. For instance, V-P7 wrote posts or ar-
ticles to reveal her disability to the public. Five VI participants also
used emojis (e.g., ) in their posts to present their disabilities.
For instance, M-P1 utilized disability-related emojis in her everyday
communications. As she mentioned, “Well, I didn’t create [the emoji],
but I like that [they] are already on the iPhone. There is a person with
a white cane or whatever. I have used those before to represent me, if
I’m writing something, posting a message, or something.”
Desired Disability Representations in Avatar Design. Par-
ticipants suggested various ways to facilitate disability represen-
tation in avatars. Most participants hoped that avatar interfaces
could include more assistive devices for PWD (e.g., prosthetic limbs,
wheelchairs, white cane). While some platforms already provided
hearing aids and cochlear implants, DHH participants wanted more
options that present diferent details. For example, they wanted
hearing aids with diferent colors, brands, and wearing methods
(e.g., one side and both sides), so that users can freely choose their
preferred appearance of assistive devices and feel more realistic.
For VI participants, two main desired features were the white cane
and guide dog, which were commonly used by VI people in real
life. Beyond simply incorporating assistive devices, participants
also expressed the need to communicate their personalities through
the assistive devices they used. Two participants (V-P1 and V-P6)
desired to decorate their avatars’ canes. As V-P6 explained, “[The
cane] not only ties in with my disability, but like my person-hood as
well around and through that disability.” Moreover, M-P1 who used
Braille wished to have jewelry with Braille on it, which represented
“her culture as a person with a disability.”
Besides accessories, H-P1 emphasized the need for a customiz-
able avatar body to accurately refect PWD’s physical appearance,
including their disabilities. She used the Helen Keller doll as an
example, “When the Helen Keller doll came out, Haben Girma, who’s
a very famous blind person, was not happy with that because [the
doll’s] eyes were proportional. They were the same. But Helen Keller in
real life, her eyes are not. I mean, even Haben’s eyes are not perfectly
proportioned. So maybe that would be something worth improving,
is to give people the ability to customize the eyes diferently. Some
people may want to be as accurate as possible.”
However, one participant (H-P8) did not support providing dis-
ability features for avatars. He was concerned that these features
could be abused by people without disabilities and suggested ver-
ifying the authenticity of one’s disability before a user achieves
access to these features. As he described, “From my experience in
VRChat, there are a lot of people who have faked disabilities. Do not
trust anyone even your new online friends that you have met recently.
I don’t think it’s necessary to show any disabilities on the avatars.
ASSETS ’22, October 23–26, 2022, Athens, Greece Zhang et al.

People have to show proof that they truly have disabilities. For exam- (Fig. 3D). Members in Helping Hands can choose sign avatars from
ple, I won’t hold back and sign fast as possible to fake deaf people. his warehouse.
They will suddenly show their ugly true colors.” Unique disability disclosure via sign avatars. All three par-
ticipants who used VR-ASL preferred not using any hearing devices
4.2.4 Specialized avatars and VR-ASL for the Deaf commu-
on their avatars, because these assistive devices can not distinguish
nity. Notably, three people in the Deaf community3 (H-P4, H-P6,
Deaf people from deaf or hard of hearings people. Instead, the best
H-P8) reported using specialized avatars to sign in social VR, specif-
way to present the Deaf culture was to sign via their avatars. As
ically in VRChat (Fig. 2B). All three participants were members
such, the sign avatars and the use of VR-ASL became a unique way
of the “Helping Hands” community in VRChat, where they were
of disability disclosure for Deaf people. As H-P6 explained, “I can
either learning or teaching American Sign Language (ASL). Instead
convey and gesture [my disability] to anyone I meet, ‘Sorry, me deaf,
of typical ASL, they used VR-ASL, a simplifed version of ASL in
me sign’ and they get the gist just fne anyway.”
VR (Fig. 3). We report participants’ experiences with the specialized
avatars and VR-ASL.
VR-ASL. ASL in real life involved rich movements of fngers,
4.2.5 Avatar Creation and Accessibility. Avatar creation and
body gestures, and facial expressions. However, not all signs can
customization posed barriers to some participants, especially those
be recognized in VR due to its limited tracking capabilities. Thus,
with visual impairments. We describe the diferent challenges DHH
Deaf users simplifed or adjusted some signs so that they could be
and VI participants faced.
captured by the current VR controllers, which resulted in VR-ASL
Barriers to DHH people. For DHH participants, the avatar
(Fig. 3A). It is worth noting that all three participants who used
customization interface in social VR was accessible and easy to use.
VR-ASL adopted the Valve Index Knuckles Controllers, which can
Six DHH participants refected that they did not encounter any
capture more hand gestures and fnger movements than other VR
difculties because all avatar features and customization steps were
controllers [39].
visual and rarely involved any audio information.
Specialized sign avatars. To enable people to better perceive
However, the major challenge occurred when creating the spe-
the signs, participants used specialized sign avatars in VRChat. The
cialized sign avatars with third-party platforms since it required
sign avatars had three unique characteristics. First, their hands
programming skills (Section 4.2.4). Both H-P4 and H-P8 reported
usually had a high contrast skin color to the avatars’ clothes, which
experiencing technical issues and highlighted the lack of support
increased the visibility of the hand gestures. As H-P4 mentioned,
in solving the issues. As H-P4 emphasized, “Oh so many times, you
“Oftentimes, people who use sign language, if their [avatar] skin is
don’t even want to believe, it’s always difcult because when the issue
white, it’s easier to see and contrast [with dark shirts color], so that
is about avatar creation, as of now is that there isn’t a lot of resources
helps people visually understand what you’re saying.” Second, many
about it. There isn’t a lot of tutorials that are either very relevant or
sign avatars had black outlined hands to further enhance their visi-
high quality for creating avatars. So that has a lot to be progressed.
bility (Fig. 2B). Last, sign avatars were re-programmed to support
That’s like my biggest issue. I don’t know how to do something.”
richer gestures, so that users can conduct more sophisticated signs
Although some third-party platforms ofered tutorials for avatar
in social VR (see the creation of specialized avatars in Section 4.2.4).
customization, most tutorials focused on audio instructions and
For example, H-P4’s avatar can perform the hand gesture that sig-
were not accessible to DHH users. As H-P6 complained, “I think
nifed the letter “y” in ASL (Fig. 3B), while with the same controller
there were some visual examples and others, which was very helpful
input, typical avatars would only pose a “rock and roll” gesture (Fig.
to learn about, but for most of it was about audio stuf which I, un-
3C) that did not represent any sign letters.
fortunately, can’t hear them. I feel like they haven’t thought this out
Creation of specialized sign avatars. Participants used dif-
very well.”
ferent ways to acquire, create, and customize sign avatars. H-P4
Barriers to VI people. Unlike DHH participants, VI participants
usually imported the publicly available avatars in VRChat into
faced more signifcant challenges when creating avatars and using
Unity and re-programmed them to support more hand gestures.
VR platforms in general. Without the support of a screen reader, VR
H-P4 explained the detailed process of her avatar augmentation
was not accessible at all to blind users. For example, M-P2 purchased
in Unity: “I was using custom animations, and I was able to modify
an Oculus Quest but decided to abandon it due to accessibility issues.
the controller inputs to play animations, like I might be able to do ‘e’
As he explained, “I have tried, the Oculus Quest, totally inaccessible,
hand or fat ‘o’ hand, otherwise you won’t be able to have it. Here
because it does not speak. I see it has no screen reading functionality in
is a little chart with all the diferent combinations you can do (Fig.
it. Because I have absolutely no use for this thing unless Meta includes
4A), and I was able to modify the game to which I can have access to
screen reading technology and text-to-speech technology in this. Still,
even more [gestures] with the combination.” H-P8 also modifed the
it’s sitting in my closet in its original box, it’s never even been opened.”
avatars in VRChat, but he asked his friends to do it for him due to
For low vision participants, similar to Zhao et al.’s results [89],
technical barriers (see Section 4.2.5).
we found that information readability was a big challenge. The text
Instead of creating his own avatars, as a teacher who taught
on some avatar interfaces was too small. Moreover, some interfaces
VR-ASL in Helping Hands, H-P6 received his avatars as a gift from
appeared at a fxed distance, preventing users from getting close
“A9,” a professional creator of sign avatars. A9 owned a warehouse
to see. As V-P4 said, “For avatar creation, they’ll use that sort of fat
in VRChat, showcasing all the sign avatars he created for VR-ASL
screen style interface, but then the texts are not large enough, well
3 The Deaf community “views themselves as a unique cultural and linguistic minority then you either guess, or you just don’t interact with it. So that’s where
who use sign language as their primary language” [12]. the problem is.”
Avatar Diversity and the Self-presentation of People with Disabilities in Social VR
Figure 3: A. Signing the alphabet using ASL vs. VR-ASL; B. H-P4’s avatar signing “y” hand; C. the “Rock and Roll” gesture; D.
Warehouse with sign avatars created by “A9”.
Figure 4: Creation of specialized sign avatars: A. Oculus Hand Chart; B. Unity interface in modifying hand gestures
Avatar creation on social media. Compared to VR, avatar
creation on conventional social media was more accessible to VI
participants since participants can access the interface with their
phone or computer that incorporated a screen reader. Seven VI
participants had experience creating avatars on social media, such
as Instagram and SnapChat. However, VI participants still faced
various challenges.
The biggest issue was the lack of sufcient descriptions across the
whole avatar design process. When designing avatars, participants
(e.g., V-P1, V-P7) had difculty understanding each avatar feature,
tracking their design progress, and confrming the fnal outcome.
Without suitable descriptions and notifcations, the avatars created
by VI participants (e.g., V-P7, V-P8) did not match their desired
avatar appearances. As V-P7 indicated, “Meta has a little bit of
description, but not everything has a description. So the frst [avatar]
I made, [my friends] said it was terrible, didn’t look anything like me.
I didn’t choose the right face, I didn’t choose the right color skin tone.
People told me the outft I chose was ugly. I didn’t get a lot of positive
feedback.” Since participants could not see and confrm the fnal
look of their avatars, they did not build strong connections with
the avatars, thus not caring much about their avatars: “When I was
ASSETS ’22, October 23–26, 2022, Athens, Greece
designing [my avatar], I did have to have my daughter help. It wasn’t
very accessible and so really it doesn’t mean a lot to me because I’m
not completely invested in it, because I can’t confrm that I like it”
(V-P7).
Moreover, the avatar customization interfaces usually ofered a
long list of options, which was difcult for VI users to navigate with
screen readers. As V-P6 said, “I think there were so many buttons to
fick through, for example, I would choose my hair color and it will
be like 60 of them. Those processes [pose] accessibility hazards. I am
good with technology, but I think that someone who was not so good
with technology would get really overwhelmed with that.”
5 DISCUSSION
Our research has contributed the frst exploration of avatar diver-
sity and PWD’s self-presentation in social VR. We answer the three
research questions proposed in the Introduction. Firstly, our sys-
tematic review in Study I highlighted the lack of disability represen-
tation in avatar design on the mainstream social VR platforms. Only
the Meta Avatars (and the platforms that adopted the Meta Avatars)
provided disability-related features, but the features were restricted
to DHH people (RQ1). Secondly, our in-depth interviews in Study
ASSETS ’22, October 23–26, 2022, Athens, Greece
II indicated that PWD employed a spectrum of self-presentation
strategies via avatars in social VR (see details in Section 5.1), and
the majority of them showed strong preferences in disclosing their
disabilities via avatars by adding the assistive tools they used in real
life (RQ2). We also uncovered the major diferences between DHH
and VI participants in avatar perception (Section 5.2). Lastly, the
avatar creation and customization process posed barriers to PWD,
especially for VI participants (RQ3). In this section, we discuss the
unique self-presentation strategies of PWD from the lens of disabil-
ity disclosure, the avatar perception diferences between DHH and
VI users, and the design considerations we derive to inspire more
accessible and inclusive avatar design.
5.1 Disability Disclosure via Avatars in Social
VR
Our research identifed some similar self-presentation patterns
observed in people without disabilities. We confrmed that most
people regarded the embodied avatars as themselves and designed
avatars to refect their physical selves [25], and people may want
their avatars to reveal a diferent self in online games other than
their physical selves [42].
Beyond the insights from prior research, our research uncovered
PWD’s unique avatar perceptions and self-presentation strategies
in social VR. We found that PWD managed their disability dis-
closure to craft their self-images in social VR. Instead of a binary
switch between disclosing and not disclosing [16], PWD adopted
a spectrum of strategies to determine to what extent and from
what aspect they would like to disclose their disability to shape
their avatar fgures. Some participants saw disability as part of
their physical selves and wanted to refect their disability as accu-
rately as possible; some selectively disclosed a certain aspect of
their disabilities (e.g., the more visible disability, or the disability
with stronger personal attachment) to signify their major disability
identity; and some participants with acquired or progressive dis-
abilities employed avatars as a nuanced way to indicate their ability
changes. Moreover, PWD disclosed their disabilities via avatars
to convey certain signals to the audience. For example, some par-
ticipants designed avatars with disability features to demonstrate
their capability and independence, while some participants used
their avatars to increase disability awareness and advocate for di-
versity and equity. These patterns demonstrated the importance
of disability disclosure for PWD in self-presentation, suggesting
the necessity of granting PWD sufcient fexibility to control their
disability disclosure in the avatar creation process.
Our research also highlighted PWD’s diferent disability disclo-
sure preferences from other online social platforms (e.g., social
media, virtual worlds). While avatars were also supported in some
2D/desktop-based social platforms (e.g., Instagram, Second Life),
PWD generally did not feel attached to their avatars due to the lack
of embodiment, and many tended to hide their disabilities by using
non-human avatars [16, 75]. In contrast, our study suggested that
the embodied nature of social VR enabled people to build strong
attachment with their avatars, making them more willing to refect
their disabilities in their avatars. Some participants (H-P4, H-P5)
mentioned that they did not want to disclose their disabilities on
social media since it felt like “showing of” their disabilities, but
Zhang et al.
displaying their disabilities via avatars in social VR were more nat-
ural. The support of embodied interaction further enhanced PWD’s
engagement and attachment. Our Deaf participants can thus com-
municate with their preferred manner in real life—ASL, and they
even created VR-ASL to accommodate to the limited capability of
current VR technology. Our fndings echoed the Embodied Social
Presence Theory [58] that the embodied avatars and the shared vir-
tual space and activities can afect user perception and bring them
to a higher engagement level, and further expanded this theory by
providing evidences from the disability perspective.
5.2 Avatar Perception of People with Diferent
Disabilities
Our fndings uncovered DHH and VI people’s diferent avatar ex-
perience and perception. Given the visual-driven nature of current
VR technology, it is not surprising that VI people face tremen-
dously more challenges than DHH people when designing and
using avatars. Our study showed that DHH users had much more
substantive avatar experiences than VI users. DHH users already
started forming communities in social VR (e.g.,“Helping Hands”),
whereas the majority of VI participants only tried social VR for
a few times. Due to the distinct avatar experiences, DHH and VI
people perceived avatars diferently. Since VI users cannot see or
even imagine the appearances of their avatars, they had weak at-
tachment with their avatars and some ended up not caring about
their avatars at all in social VR. As V-P4 said, “[Avatar design] really
is just for fun, I mean it’s just kind of seeing what types of appear-
ances, or what types of features that the avatars have. I really don’t
necessarily care about avatars all that much.” In contrast, DHH peo-
ple showed stronger attachment with their avatars and spent more
efort customizing, specializing, and even re-programming their
avatars for better self-presentation and communication.
While this research focused only on DHH and VI users, our
fndings indicated that people with diferent disabilities may have
diferent avatar perception and self-presentation preferences. For
example, compared to people who have visible disabilities (e.g., a
person with an amputation) or who use apparent assistive tech-
nologies (e.g., a VI person who has a cane), people with invisible
conditions and using invisible assistive technologies (e.g., a person
who experiences chronic pain and uses a trigger tracker) may be
more reluctant to disclose their disabilities in social VR. Future
research should consider other disabilities and explore diferent
factors that may afect people’s disability disclosure decisions, such
as visibility of disability and visibility of assistive technology [23].
5.3 Design Implications for Avatar Diversity
and Accessibility
5.3.1 Avatar Diversity. We drew three design implications to
promote disability representations for avatars.
Ofer more assistive device representations. Most partici-
pants saw their assistive devices as the key signifer of their disabili-
ties and preferred adding these devices to their avatars for disability
representation. Participants suggested various commonly-used as-
sistive technologies that can represent their disabilities, including
a white cane and guide dog for VI people, cochlear implants and
hearing devices for DHH people, and prosthetic limbs, wheelchair,
Avatar Diversity and the Self-presentation of People with Disabilities in Social VR
and walking aids for people with motor disabilities. Besides the
basic assistive technology representations, participants wanted to
further personalize their virtual assistive devices to refect their
aesthetics and personalities, for example, customizing the color and
the brand. Adding these features in the mainstream avatar systems
would empower PWD to express and present themselves freely and
equally in social VR. Designers should involve PWD throughout
the whole avatar design process to ensure the suitability of the
disability representations.
Support representations for people with invisible disabil-
ities. Compared to visible disabilities, some invisible disabilities
could not be easily expressed, especially when no apparent assis-
tive technologies are used to signify the disability. In our study,
participants suggested using accessories, such as a necklace with
Brailles or an arm badge with their community logo, to present
their disabilities. We suggest that designers should consider using
these indirect mediums, such as outfts with disability signifers
(e.g., a T-Shirt with the Autism Awareness Puzzle logo), to enable
people with invisible disabilities to better manage their disability
visibility in social VR.
Guarantee appropriate use of diversity features. Adding
more avatar diversity features may also pose potential risks. Our
participants indicated their concerns on the inappropriate usage of
these features by people without disabilities. The abuse of diversity
features can lead to cyber-bullying and increase the misconception
of disability from the public. One participant (H-P8) further sug-
gested conducting strict verifcation on the authenticity of one’s
disability before they can access the diversity features. How to
guarantee the proper use of the diversity features is a vital issue
that should be considered by the commercial social VR platforms
from both the design and the policy perspectives. Researchers and
designers should investigate how to set up suitable rules to support
interaction freedom without marginalizing and harming vulnerable
populations on these new and emerging social platforms.
5.3.2 Avatar Accessibility. Our fndings highlighted the dif-
culties faced by VI and DHH people when designing avatars. This
echoed the general VR accessibility issues revealed by prior re-
search [10, 47, 61, 89], and also expanded the problems from a new
avatar creation perspective. We drew three design implications to
enhance avatar accessibility.
Combine avatar automation with fne-grained customiza-
tion. Avatar creation is not accessible to VI participants due to
its heavily visual-driven nature and the complex steps to navigate.
Our blind participants usually needed external human assistance
in customizing their avatars. Instead of crafting the avatars from
scratch, some participants suggested automating the avatar genera-
tion process (M-P1, V-P5): the avatar system should automatically
generate a baseline avatar based on the user’s photo, and allow
the user to adjust details based on their preferences. While not
new, this feature is supported by very limited number of social VR
platforms. In our application review (Study I), only three out of 15
commercial social VR platforms support avatar automation. We
suggest that designers should consider enabling multiple avatar
creation methods—both auto generation and manual adjustment—
to enhance the accessibility of avatar creation and customisation
for people with diverse abilities.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Convey avatar design outcomes for VI users. VI users rely
on alternative text to perceive graphic information. Our research
highlighted the lack of and the need for alternative text and screen
reader support for avatars in social VR. While the avatar design
system for conventional social media (e.g., Facebook, SnapChat)
could be accessed by VI users via alternative text and embedded
screen readers, participants reported challenges in understanding
their customization progress and fnal outcomes, which led to their
emotional detachment from the avatars. Artifcial intelligence (AI)
technology could be considered to recognize the avatars and gener-
ate semantic confrmations about the holistic avatar customization
outcomes, such as whether the avatar looks like the user. However,
this could be technically and ethically challenging since catego-
rizing and recognizing human facial features is difcult in both
technology [15, 67, 73] and sociology felds [19, 33]. For example,
an algorithm may mis-recognize particular avatar or user faces and
report inappropriate results. Moreover, how to suitably label and
describe human appearance, especially the marginalized groups, is
another important question to consider [4, 33, 40]. Besides refning
the algorithms and datasets from the computer vision perspective
[77], HCI solutions could also be adopted, for example, indicating
the potential inaccuracy of the recognized results to users [54, 91],
or leveraging human-AI collaboration to achieve more reliable re-
sults [32, 51].
Add closed caption for all audio information. Although
the avatar creation and customization process was visual-driven,
adding closed caption would still be valuable, especially for DHH
users. We believe that with proper caption positioning, closed cap-
tion will have an signifcantly positive impact on the social VR
ecosystem.
6 LIMITATIONS AND FUTURE WORK
In this paper, we studied the avatar experience and perception of
PWD on social VR platforms. We focused on two disability groups—
DHH and VI people. Future work should take into account more
diverse types of disabilities to get a more comprehensive under-
standing of PWD’s disability disclosure preferences, especially for
people with invisible disabilities. Moreover, our application review
study (Study I) focused on the Oculus Quest 2 platform. Although
social VR applications mostly support the same avatar design and
accessibility features across VR devices, we acknowledge that dif-
ferent VR devices may ofer their own system-level avatars (e.g.,
Oculus ofers the Meta Avatars), which may bring nuances to the
review results. Thus, other mainstream VR devices should also be
considered to achieve a more comprehensive results. Finally, future
work may complement our qualitative fndings with quantitative
analysis to investigate what factors may impact PWD’s disability
disclosure behaviors, such as gender, age, and disability visibility.
ACKNOWLEDGMENTS
We thank the National Federation of the Blind for helping us recruit
for our study, as well as the anonymous participants that provided
their perspective.
REFERENCES
[1] Dominic Abrams and Michael A Hogg. 2004. Metatheory: Lessons from social
identity research. Personality and social psychology review 8, 2 (2004), 98–106.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[2] Alvirmin. 2021. 7 Best Social Networking Apps in VR for Oculus Quest
2. https://allvirtualreality.com/review/best-social-networking-apps-vr-oculus-
quest.html last accessed 5 July 2022.
[3] Jane Bailey, Valerie Steeves, Jacquelyn Burkell, and Priscilla Regan. 2013. Negoti-
ating with gender stereotypes on social networking sites: From “bicycle face” to
Facebook. Journal of Communication Inquiry 37, 2 (2013), 91–112.
[4] Cynthia L Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jefrey P Bigham,
Anhong Guo, and Alexandra To. 2021. “It’s Complicated”: Negotiating Accessibil-
ity and (Mis) Representation in Image Descriptions of Race, Gender, and Disability.
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.
1–19.
[5] Katherine Bessière, A Fleming Seay, and Sara Kiesler. 2007. The ideal elf: Identity
exploration in World of Warcraft. Cyberpsychology & behavior 10, 4 (2007),
530–535.
[6] Tom Boellstorf. 2019. The Ability of Place: Digital Topographies of the Virtual
Human on Ethnographia Island. Current Anthropology 61 (08 2019), S109–S122.
[7] Natilene Bowker and Keith Tufn. 2002. Disability discourses for online identities.
Disability & Society 17, 3 (2002), 327–344.
[8] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative research in psychology 3, 2 (2006), 77–101.
[9] Justin Buss, Hayden Le, and Oliver L Haimson. 2022. Transgender
identity management across social media platforms. Media, Culture &
Society 44, 1 (2022), 22–38. https://doi.org/10.1177/01634437211027106
arXiv:https://doi.org/10.1177/01634437211027106
[10] Diane Carr. 2010. Constructing disability in online worlds: conceptualising
disability in online research. London Review of Education (03 2010), 51–61.
[11] Justine Cassell. 1998. Chess for girls?: Feminism and computer games. (1998).
[12] National Deaf Center. 2021. The Deaf Community: An Introduction.
[13] Victoria Clarke, Virginia Braun, and Nikki Hayfeld. 2015. Thematic analysis.
Qualitative psychology: A practical guide to research methods 222 (2015), 248.
[14] Paul C Cozby. 1973. Efects of density, activity, and personality on environmental
preferences. Journal of Research in Personality 7, 1 (1973), 45–60.
[15] Chirag Dalvi, Manish Rathod, Shruti Patil, Shilpa Gite, and Ketan Kotecha. 2021.
A Survey of AI-Based Facial Emotion Recognition: Features, ML & DL Techniques,
Age-Wise Datasets and Future Directions. IEEE Access 9 (2021), 165806–165840.
[16] Donna Z Davis and Shelby Stanovsek. 2021. The machine as an extension of the
body: When identity, immersion, and interactive design serve as both resource
and limitation for the disabled. Human-Machine Communication 2 (2021), 121–
135.
[17] Joan Morris DiMicco and David R Millen. 2007. Identity management: multiple
presentations of self in facebook. In Proceedings of the 2007 international ACM
conference on Supporting group work. 383–386.
[18] Nicolas Ducheneaut, Ming-Hui Wen, Nicholas Yee, and Greg Wadley. 2009. Body
and mind: a study of avatar personalization in three virtual worlds. In Proceedings
of the SIGCHI conference on human factors in computing systems. 1151–1160.
[19] Yarrow Dunham, Elena V. Stepanova, Ron Dotsch, and Alexander Todorov. 2015.
The development of race-based perceptual categorization: skin color dominates
early category judgments. Developmental Science 18, 3 (2015), 469–483.
[20] Allison Eden, Erin Maloney, and Nicholas David Bowman. 2010. Gender attribu-
tion in online video games. Journal of Media Psychology: Theories, Methods, and
Applications 22, 3 (2010), 114.
[21] Blizzard Entertainment. 2021. World of Warcraft. https://worldofwarcraft.com/
en-us/ last accessed 5 July 2022.
[22] Shelly D Farnham and Elizabeth F Churchill. 2011. Faceted identity, faceted lives:
social and technical issues with being yourself online. In Proceedings of the ACM
2011 conference on Computer supported cooperative work. 359–368.
[23] Heather A. Faucett, Kate E. Ringland, Amanda L. L. Cullen, and Gillian R. Hayes.
2017. (In)Visibility in Disability and Assistive Technology. 10, 4 (2017), 17 pages.
[24] Katrina Fong and Raymond A Mar. 2015. What does my avatar say about me?
Inferring personality from avatars. Personality and Social Psychology Bulletin 41,
2 (2015), 237–249.
[25] Guo Freeman and Divine Maloney. 2021. Body, avatar, and me: The presentation
and perception of self in social virtual reality. Proceedings of the ACM on Human-
Computer Interaction 4, CSCW3 (2021), 1–27.
[26] Guo Freeman, Samaneh Zamanifard, Divine Maloney, and Alexandra Adkins.
2020. My body, my avatar: How people perceive their avatars in social virtual
reality. In Extended Abstracts of the 2020 CHI Conference on Human Factors in
Computing Systems. 1–8.
[27] James Paul Gee. 2003. What video games have to teach us about learning and
literacy. Computers in entertainment (CIE) 1, 1 (2003), 20–20.
[28] Kathrin Gerling, Kieran Hicks, Michael Kalyn, Adam Evans, and Conor Linehan.
2016. Designing Movement-Based Play With Young People Using Powered
Wheelchairs. In Proceedings of the 2016 CHI Conference on Human Factors in
Computing Systems. 4447–4458.
[29] Jennifer L Gibbs, Nicole B Ellison, and Rebecca D Heino. 2006. Self-presentation
in online personals: The role of anticipated future interaction, self-disclosure,
and perceived success in Internet dating. Communication research 33, 2 (2006),
Zhang et al.
152–177.
[30] Erving Gofman. 1959. The presentation of self in everyday life. Bantam Doubleday
Dell Publishing group.
[31] Kathryn Greene, Valerian J Derlega, and Alicia Mathews. 2006. Self-disclosure
in personal relationships. The Cambridge handbook of personal relationships 409
(2006), 427.
[32] Anhong Guo, Anuraag Jain, Shomiron Ghose, Gierad Laput, Chris Harrison, and
Jefrey P Bigham. 2018. Crowd-ai camera sensing in the real world. Proceedings of
the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018),
1–20.
[33] Margot Hanley, Solon Barocas, Karen Levy, Shiri Azenkot, and Helen Nissenbaum.
2021. Computer Vision and Conficting Values: Describing People with Auto-
mated Alt Text. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics,
and Society. 543–554.
[34] Daniel Hepperle, Christian Felix Purps, Jonas Deuchler, and Matthias Wölfel.
2021. Aspects of visual avatar appearance: self-representation, display type, and
uncanny valley. The Visual Computer (2021), 1–18.
[35] Daniel Hepperle, Hannah Ödell, and Matthias Wölfel. 2020. Diferences in
the Uncanny Valley between Head-Mounted Displays and Monitors. In 2020
International Conference on Cyberworlds (CW). 41–48. https://doi.org/10.1109/
CW49994.2020.00014
[36] Tanner Higgin. 2009. Blackless fantasy: The disappearance of race in massively
multiplayer online role-playing games. Games and Culture 4, 1 (2009), 3–26.
[37] Zaheer Hussain and Mark D Grifths. 2008. Gender swapping and socializing
in cyberspace: An exploratory study. CyberPsychology & Behavior 11, 1 (2008),
47–53.
[38] Nora F Huvelle, Milton Budof, and Deidre Arnholz. 1984. To tell or not to
tell: Disability disclosure and the job interview. Journal of Visual Impairment &
Blindness 78, 6 (1984), 241–244.
[39] VALVE INDEX. 2022. Controllers. https://www.valvesoftware.com/en/index/
controllers last accessed 11 April 2022.
[40] Emory James Edwards, Kyle Lewis Polster, Isabel Tuason, Emily Blank, Michael
Gilbert, and Stacy Branham. 2021. "That’s in the Eye of the Beholder": Layers of
Interpretation in Image Descriptions for Fictional Representations of People with
Disabilities. In The 23rd International ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS ’21). Article 19, 14 pages.
[41] Yasmin B Kafai, Deborah A Fields, and Melissa Cook. 2007. Your second selves:
avatar designs and identity play in a teen virtual world. In Proceedings of DIGRA,
Vol. 2007.
[42] Yasmin B Kafai, Deborah A Fields, and Melissa S Cook. 2010. Your second selves:
Player-designed avatars. Games and culture 5, 1 (2010), 23–42.
[43] Sanjay Kairam, Mike Brzozowski, David Hufaker, and Ed Chi. 2012. Talking in
circles: selective sharing in google+. In Proceedings of the SIGCHI conference on
human factors in computing systems. 1065–1074.
[44] Bernadett Koles and Peter Nagy. 2012. Who is portrayed in Second Life: Dr. Jekyll
or Mr. Hyde? The extent of congruence between real life and virtual identity.
Journal For Virtual Worlds Research 5, 1 (2012).
[45] Anya Kolesnichenko, Joshua McVeigh-Schultz, and Katherine Isbister. 2019. Un-
derstanding emerging design practices for avatar systems in the commercial
social vr ecology. In Proceedings of the 2019 on Designing Interactive Systems
Conference. 241–252.
[46] Robin M Kowalski, Chad A Morgan, Kelan Drake-Lavelle, and Brooke Allison.
2016. Cyberbullying among college students with disabilities. Computers in
Human Behavior 57 (2016), 416–427.
[47] Rachel L. Franz, Sasa Junuzovic, and Martez Mott. 2021. Nearmi: A Framework
for Designing Point of Interest Techniques for VR Users with Limited Mobility. In
The 23rd International ACM SIGACCESS Conference on Computers and Accessibility.
1–14.
[48] Linden Lab. 2013. 10 Years of Second Life. https://www.lindenlab.com/releases/
infographic-10-years-of-second-life last accessed 4 Feb 2022.
[49] Jong-Eun Roselyn Lee. 2014. Does virtual diversity matter?: Efects of avatar-
based diversity representation on willingness to express ofine racial identity
and avatar customization. Computers in Human Behavior 36 (2014), 190–197.
[50] Jong-Eun Roselyn Lee and Sung Gwan Park. 2011. “Whose second life is this?”
How avatar-based racial cues shape ethno-racial minorities’ perception of virtual
worlds. Cyberpsychology, Behavior, and Social Networking 14, 11 (2011), 637–642.
[51] Sooyeon Lee, Rui Yu, Jingyi Xie, Syed Masum Billah, and John M Carroll. 2022.
Opportunities for human-AI collaboration in remote sighted assistance. In 27th
International Conference on Intelligent User Interfaces. 63–78.
[52] Qiaoxi Liu and Anthony Steed. 2021. Social Virtual Reality Platform Comparison
and Evaluation Using a Guided Group Walkthrough Method. Frontiers In Virtual
Reality (2021), 1–15.
[53] Xiao Ma, Jef Hancock, and Mor Naaman. 2016. Anonymity, Intimacy and Self-
Disclosure in Social Media. In Proceedings of the 2016 CHI Conference on Human
Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association
for Computing Machinery, New York, NY, USA, 3857–3869. https://doi.org/10.
1145/2858036.2858414
Avatar Diversity and the Self-presentation of People with Disabilities in Social VR
[54] Haley MacLeod, Cynthia L Bennett, Meredith Ringel Morris, and Edward Cutrell.
2017. Understanding blind people’s experiences with computer-generated cap-
tions of social media images. In Proceedings of the 2017 CHI Conference on Human
Factors in Computing Systems. 5988–5999.
[55] Divine Maloney, Samaneh Zamanifard, and Guo Freeman. 2020. Anonymity
vs. familiarity: Self-disclosure and privacy in social virtual reality. In 26th ACM
Symposium on Virtual Reality Software and Technology. 1–9.
[56] Tony Manninen and Tomi Kujanpää. 2007. The value of virtual assets: the role of
game characters in MMOGs. International Journal of Business Science & Applied
Management (IJBSAM) 2, 1 (2007), 21–33.
[57] Rosa Mikeal Martey and Mia Consalvo. 2011. Performing the looking-glass self:
Avatar appearance and group identity in Second Life. Popular Communication 9,
3 (2011), 165–180.
[58] Brian E. Mennecke, Janea L. Triplett, Lesya M. Hassall, and Zayira Jordan Conde.
2010. Embodied Social Presence Theory. In 2010 43rd Hawaii International Con-
ference on System Sciences. 1–10.
[59] Akash Menon. 2021. The Role of Avatar Creation and Embodied Presence in
Virtual Reality Job Interviews.
[60] Helen Morgan, Amanda O’donovan, Renita Almeida, Ashleigh Lin, and Yael
Perry. 2020. The Role of the Avatar in Gaming for Trans and Gender Diverse
Young People. International journal of environmental research and public health
17, 22 (2020), 8617.
[61] Martez Mott, Edward Cutrell, Mar Gonzalez Franco, Christian Holz, Eyal Ofek,
Richard Stoakley, and Meredith Ringel Morris. 2019. Accessible by Design: An
Opportunity for Virtual Reality. In 2019 IEEE International Symposium on Mixed
and Augmented Reality Adjunct (ISMAR-Adjunct). 451–454.
[62] Lisa Nakamura. 1995. Race in/for cyberspace: Identity tourism and racial passing
on the Internet. Works and Days 13, 1-2 (1995), 181–193.
[63] Carman Neustaedter and Elena Fedorovskaya. 2009. Presenting Identity in a Vir-
tual World through Avatar Appearances. In Proceedings of Graphics Interface 2009
(Kelowna, British Columbia, Canada) (GI ’09). Canadian Information Processing
Society, CAN, 183–190.
[64] Veronica Pearson, Frances Ip, Heidi Hui, Nelson Yip, et al. 2003. To tell or not to
tell; disability disclosure and job application outcomes. Journal of Rehabilitation
69, 4 (2003), 35.
[65] John R Porter, Kiley Sobel, Sarah E Fox, Cynthia L Bennett, and Julie A Kientz.
2017. Filtered out: Disability disclosure practices in online dating communities.
Proceedings of the ACM on Human-Computer interaction 1, CSCW (2017), 1–13.
[66] Mark Rabkin. 2021. Connect 2021 Recap: Horizon Home, the future of work, presence
platforms, and more. https://www.oculus.com/blog/connect-2021-recap-horizon-
home-the-future-of-work-presence-platform-and-more/ last accessed 13 April
2022.
[67] Inioluwa Deborah Raji, Timnit Gebru, Margaret Mitchell, Joy Buolamwini, Joon-
seok Lee, and Emily Denton. 2020. Saving Face: Investigating the Ethical Concerns
of Facial Recognition Auditing. Association for Computing Machinery, New York,
NY, USA, 145–151. https://doi.org/10.1145/3375627.3375820
[68] Sheila Riddell and Elisabet Weedon. 2014. Disabled students in higher education:
Discourses of disability and the negotiation of identity. International Journal of
Educational Research 63 (2014), 38–46.
[69] Kathryn E Ringland. 2019. “Autsome”: Fostering an Autistic Identity in an Online
Minecraft Community for Youth with Autism. (2019), 142–143.
[70] Ann E Schlosser. 2020. Self-disclosure versus self-presentation on social media.
Current Opinion in Psychology 31 (2020), 1–6. https://doi.org/10.1016/j.copsyc.
2019.06.025 Privacy and Disclosure, Online and in Social Interactions.
[71] Zicodas Serapis. 2008. Coming of age in second life: an anthropologist explores
the virtually human.
[72] Carmit-Noa Shpigelman and Carol J Gill. 2014. Facebook use by persons with
disabilities. Journal of Computer-Mediated Communication 19, 3 (2014), 610–624.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[73] Luke Stark. 2019. Facial Recognition is the Plutonium of AI. XRDS 25, 3 (apr
2019), 50–55.
[74] Karen Stendal. 2012. How do People with Disability Use and Experience Virtual
Worlds and ICT: A Literature Review. Journal of Virtual Worlds Research (05
2012), 1–17.
[75] Karen Stendal, Judith Molka-Danielsen, Bjørn Munkvold, and Susan Balandin.
2012. Virtual worlds and people with lifelong disability: Exploring the relationship
with virtual self and others. ECIS (01 2012), 156–179.
[76] Stephanie Stewart, Terri S. Hansen, and Timothy A. Carey. 2010. Opportunities
for People with Disabilities in the Virtual World of Second Life. Rehabilitation
Nursing Journal 35 (2010), 254–259.
[77] Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. 2014. DeepFace:
Closing the Gap to Human-Level Performance in Face Verifcation. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[78] Sherry Turkle. 1999. Life on the Screen: Identity in the Age of the Internet. The
Psychohistory Review 27, 2 (1999), 113.
[79] Sherry Turkle. 2005. The second self: Computers and the human spirit. Mit Press.
[80] Sarah Von Schrader, Valerie Malzer, and Susanne Bruyère. 2014. Perspectives
on disability disclosure: the importance of employer practices and workplace
climate. Employee Responsibilities and Rights Journal 26, 4 (2014), 237–255.
[81] Zach Waggoner. 2009. My avatar, my self: Identity in video role-playing games.
McFarland.
[82] Paul Wallace and James Maryott. 2009. The impact of avatar self-representation
on collaboration in virtual worlds. Innovate: Journal of Online Education 5, 5
(2009).
[83] Thomas Waltemate, Dominik Gall, Daniel Roth, Mario Botsch, and Marc Erich
Latoschik. 2018. The impact of avatar personalization and immersion on vir-
tual body ownership, presence, and emotional response. IEEE transactions on
visualization and computer graphics 24, 4 (2018), 1643–1652.
[84] VRChat Legends Wiki. 2022. Helping Hands. https://vrchat-legends.fandom.
com/wiki/Helping_Hands last accessed 9 April 2022.
[85] Shaomei Wu and Lada A Adamic. 2014. Visually impaired users on an online
social network. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems. 3133–3142.
[86] XRTODAY. 2022. The Best Social Apps in VR. https://www.xrtoday.com/virtual-
reality/the-best-social-apps-in-vr/ last accessed 5 July 2022.
[87] Jennifer Yurchisin, Kittichai Watchravesringkan, and Deborah Brown McCabe.
2005. An exploration of identity re-creation in the context of internet dating.
Social Behavior and Personality: an international journal 33, 8 (2005), 735–750.
[88] Shanyang Zhao, Sherri Grasmuck, and Jason Martin. 2008. Identity construction
on Facebook: Digital empowerment in anchored relationships. Computers in
human behavior 24, 5 (2008), 1816–1836.
[89] Yuhang Zhao, Edward Cutrell, Christian Holz, Meredith Ringel Morris, Eyal Ofek,
and Andrew D Wilson. 2019. SeeingVR: A set of tools to make virtual reality more
accessible to people with low vision. In Proceedings of the 2019 CHI conference on
human factors in computing systems. 1–14.
[90] Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2017. The efect
of computer-generated descriptions on photo-sharing experiences of people with
visual impairments. Proceedings of the ACM on Human-Computer Interaction 1,
CSCW (2017), 1–22.
[91] Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2018. A face
recognition application for people with visual impairments: Understanding use
beyond the lab. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems. 1–14.
A APPENDIX
ASSETS ’22, October 23–26, 2022, Athens, Greece Zhang et al.

Table 2: Overview of social VR platforms and their avatar creation options. Description of the icons : humanoid avatar= ,
robot avatar= , cartoon avatar= , animal avatar= , object avatar= , abstract avatar= , full body avatar includes head,
upper body, arm hand, leg= , partial body avatar includes head, upper body, arm, hand = , foating head without neck= ,
foating upper body without arm = , foating hand= , foating fngerless hand = , full avatar selection = , avatar feature
customization = , photo-based avatar generation = , third-party avatar import =

VR Platforms Avatar type Avatar Customization Avatar Realism Disability Representation

Rec Room N/A

VRChat Any uploaded 3rd-party disability features

Horizon Worlds Hearing aids; cochlear implants

vTime XR N/A

AltspaceVR N/A

Bigscreen Eye patch

Alcove Hearing aids; cochlear implants

Half+Half N/A

Horizon Venues Hearing aids; cochlear implants

Villa N/A

Arthur N/A

ENGAGE N/A

Multiverse N/A

PokerStars VR Hearing aids; cochlear implants

Spatial N/A
 SoundVizVR: Sound Indicators for Accessible Sounds in Virtual
Reality for Deaf or Hard-of-Hearing Users
Ziming Li Shannon Connell
zl1398@rit.edu sdcnai@rit.edu
School of Information, National Technical Institute for the Deaf,
Rochester Institute of Technology Rochester Institute of Technology
Rochester, New York, USA Rochester, New York, USA

Wendy Dannels Roshan Peiris

w.dannels@rit.edu roshan.peiris@rit.edu
Center on Culture and Language, School of Information,
National Technical Institute for the Deaf, Rochester Institute of Technology
Rochester Institute of Technology Rochester, New York, USA
Rochester, New York, USA
ABSTRACT ACM Reference Format:
Sounds provide vital information such as spatial and interaction Ziming Li, Shannon Connell, Wendy Dannels, and Roshan Peiris. 2022.
SoundVizVR: Sound Indicators for Accessible Sounds in Virtual Reality for
cues in virtual reality (VR) applications to convey more immersive
Deaf or Hard-of-Hearing Users. In The 24th International ACM SIGACCESS
experiences to VR users. However, it may be a challenge for deaf or Conference on Computers and Accessibility (ASSETS ’22), October 23–26, 2022,
hard-of-hearing (DHH) VR users to access the information given Athens, Greece. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/
by sounds, which could limit their VR experience. To address this 3517428.3544817
limitation, we present “SoundVizVR”, which explores visualizing
sound characteristics and sound types for several types of sounds in 1 INTRODUCTION
VR experience. SoundVizVR uses Sound-Characteristic Indicators
to visualize loudness, duration, and location of sound sources in
VR and Sound-Type Indicators to present more information about
the type of the sound. First, we examined three types of Sound-
Characteristic Indicators (On-Object Indicators, Full Mini-Maps
and Partial Mini-Maps) and their combinations in a study with 11
DHH participants. We identifed that the combination of Full Mini-
Map technique and On-Object Indicator was the most preferred
visualization and performed best at locating sound sources in VR.
Next, we explored presenting more information about the sounds
using text and icons as Sound-Type Indicators. A second study
with 14 DHH participants found that all Sound-Type Indicator
combinations were successful at locating sound sources.
Figure 1: SoundVizVR method that uses Mini-Maps and
On-Object Indicators to present sound source characteris-
CCS CONCEPTS tics and sound type information. The Mini-Map visualizes
• Human-centered computing → Accessibility technologies; sounds in the VR environment while the On-Object Indica-
User studies. tor displays the sound originating from an object (“a text
that says ‘Bark’ is shown”). This example shows using icons
KEYWORDS in the Full Mini-Map and text on the object to describe the
virtual reality, audio visualization, deaf and hard-of-hearing, acces- sound type.
sibility
Virtual reality (VR) technologies have the capability to deliver
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed a completely immersive experience to the user. This experience
for proft or commercial advantage and that copies bear this notice and the full citation allows the user to completely engulf or immerse their senses in
on the frst page. Copyrights for components of this work owned by others than the the content and the interactions. Immersive VR experiences are
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission typically dependent on the quality of the visual, sound, and interac-
and/or a fee. Request permissions from permissions@acm.org. tion dimensions of the experience [35, 38]. As such, sounds in VR
ASSETS ’22, October 23–26, 2022, Athens, Greece that enhance and complement the interactions are a crucial part
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00 of the “immersive” virtual reality experience. VR utilizes sounds
https://doi.org/10.1145/3517428.3544817 in many forms, such as spatial audio, voice, interaction sounds,
ASSETS ’22, October 23–26, 2022, Athens, Greece
rhythmic interaction, background sounds/music, etc., to enhance
the interaction experience [18].
However, although sounds in VR play an essential role in creat-
ing the immersive experience, sounds could also be a limiting factor
for deaf or hard-of-hearing (DHH) individuals to experience a fully
immersive experience in VR due to the limited sound accessibil-
ity [19, 26]. To address similar limitations in everyday situations
(in the non-VR space), visualization-based [17, 20, 24] or haptics-
based [15, 29, 30] methods have been proposed to make everyday
sounds accessible for DHH persons. However, adapting the above
practices to the VR space presents a set of unique challenges due
to the fctional nature of the visual world and novel interaction
possibilities VR presents [25]. With these challenges in mind, a few
pioneering research has already begun exploring making sounds
in virtual reality more accessible for DHH users. Here, Jain et al.
defned a “Taxonomy of Sounds in Virtual Reality” to help future
VR designers design accessible sounds in VR [18]. In addition, Jain
et al. also presented an exploration of the design space for accessible
VR sounds using visual and haptic-based method [19] and another
pioneering work, EarVR [26], presented a haptic-based method for
presenting spatial sounds for DHH VR users.
Inspired by the above approaches, we present SoundVizVR that
explores visualizing VR sounds via sound indicators. Specifcally,
SoundVizVR uses Mini-Map Indicators and On-Object Indicator
style interfaces as Sound-Characteristic Indicators and uses text and
icons as Sound-Type Indicators to increase the ability to localize
sound sources and visualize sound characteristics in VR (Figure 1).
Our proposed sound visualization methods aim to enable users
to visualize several types of sounds that originate from the sound
sources within the VR environments [18], by identifying sound
characteristics (e.g., loudness, duration) and sound types (e.g., foot-
steps, gunshots). Infuenced by Stockburger [42], we use the term
diegetic sound for the sound that comes from the object in the VR
world (e.g., a phone ringing). For the current scope of this work,
we focus only on several diegetic sounds in VR [18, 42]: localized
speech, inanimate objects, animate objects, and point ambience,
as they are critical for the experience in VR [18]. We present our
prototype as a generalizable and customizable plugin1 that can
be integrated into any VR software developed in the Unity game
engine 2 . Using this Unity plugin, we envision that VR designers
and developers can promptly visualize sound source information
to increase the sound accessibility for DHH VR users.
We conducted two user studies to explore the usability of frstly,
the Sound-Characteristic Indicator visualization method and sec-
ondly, the Sound-Type Indicator visualization method. In the frst
study, we conducted a preliminary evaluation with 11 DHH partici-
pants to identify the best and the most preferred Sound-Characteristic
Indicator visualization technique to localize sound sources from six
design combinations of the Mini-Map based and the On-Object In-
dicator based sound visualization method. These methods assisted
DHH VR users in locating the sound sources and visualizing other
characteristics of the sounds, such as the loudness and duration.
The best technique of the frst study was selected for the second
study based on the performance data and participants’ feedback.
1 https://github.com/ZimingLiii/SoundVizVR-Plugin
2 https://unity.com/
Li, Connell, Dannels, and Peiris
In the second study, we integrated the chosen visualization tech-
nique into a VR game scene to explore the best and most preferred
type of indicator for presenting the sound types (e.g., footsteps,
gunshots, etc.). We implemented combinations of icons and text
representation methods into the visualization technique to present
the diferent sound types. We conducted a user experiment with
14 DHH participants to collect user performance data and user
feedback to evaluate these combinations.
In summary, our research contributions are as follows:
• A VR sound visualization prototype software that can be
used as a Unity platform plugin to improve sound accessibil-
ity for DHH users in VR projects
• A study with 11 DHH participants that identifes charac-
teristics of the combinations of Mini-Map Indicators and
On-Object Indicators to represent sound characteristics in
VR, and
• A study with 14 DHH participants that discusses the prefer-
ences of presenting sound types through icons and text in
the Full Mini-Map and on virtual objects.
2 RELATED WORK
2.1 Sound Visualization
Prior work has mainly explored visual and haptic methods, aiming
to make sounds more accessible for DHH individuals. Here, we
focus on prior work closely related to our work that applied visual-
based sound accessibility methods.
Sound visualization techniques used in music can convey music
information (e.g., pitch, tempo, etc.) in real-time with animated
images [41]. Music visualization methods are widely supported
by media player software, such as Windows Media Player, Foo-
bar2000, and iTunes. They visualize music characteristics using
spectrum-like or waveform-like 2D displays. There is also research
on music visualization techniques conducted among DHH commu-
nities to evaluate DHH people’s experience with these methods.
For example, S. Nanayakkara et al. implemented a system combin-
ing a vibrated chair with a visual display to provide an enhanced
musical experience to DHH people [29]. In addition, J. Mori and
D. I. Fels conducted research with DHH people to investigate their
emotional reaction to a song with diferent animated lyrics [28],
which indicated that the animated text could provide entertainment
value of the music without losing the readability of lyrics.
Similarly, sounds also have vital contributions to the experience
designs in games. There are video games that look into using sound
visualization methods to make their content more accessible to
DHH people. For example, a sandbox video game called “Minecraft”
[27] features “Subtitles” that use text labels with arrows to indicate
the sound types and the directions of in-game sounds (e.g., rain falls,
zombie groans, etc.) near the player avatar [47]. And a battle royale
game called “Fortnite” [7] uses a radar-like interface to assist DHH
players in accurately locating the vital sound efects in the game
environment, like footsteps and gunfres. However, although with
caption supported [23], the playing experience is still signifcantly
reduced in many commercial video games once sounds are disabled.
This limitation may be due to the lacking sound accessibility designs
in their major game events (e.g., notifcations of shootings from an
enemy in a frst-person shooter [FPS] game) [3, 48].
SoundVizVR: Accessible VR Sounds for DHH Users with Sound Indicators ASSETS ’22, October 23–26, 2022, Athens, Greece

Inspired by these prior works, we are exploring implementing 3.1 Sound-Characteristic Indicator System
sound visualization methods that can provide sound characteristic Design
information. Our proposed methods adopt the interface compo-
nents of Mini-Maps and environmental indicators, infuenced by
the accessibility designs of existing video games.

2.2 XR Sound Accessibility

Previous research primarily addressed accessibility for people with
visual impairments [40, 46, 50] or mobility impairments [9, 10, 32]
using XR technologies (including virtual reality, augmented reality,
etc.). For DHH users, researchers mainly investigated captioning,
sound awareness, description of sounds, and the ability to locate
sounds in XR environments.
Prior work on sound accessibility in augmented reality (AR)
explores speech captioning for talking people using wearable AR
devices [14, 31, 34]. For example, D. Jain et al. [14] evaluated real-
time captioning in an AR approach, which provided insights on
the UI design of speech captioning systems in AR head-mounted
displays (HMDs) and outlined the benefts of AR head-mounted
display captioning. Another work from Y. Peng et al. [34] proposed
an AR captioning system called “SpeechBubbles” to address prob-
lems in the group conversation scenario, such as the out-of-view
captions and the speaker association. Also, prior research in AR
accessibility addressed sound awareness [16] and sound detection
and localization [11] which provide glanceable sound information
of the environment to DHH people.
In terms of VR, prior work mainly explores providing accessible
acoustic clues using subtitles [1, 13, 36, 44]. Several other research
examines VR in the context of story telling [4] for DHH individuals
and learning sign language for DHH children [33]. And studies
like EarVR [26] also looked into conveying sound information in
VR through haptics. Some VR games in the market adopt sound
accessibility interfaces other than captions, for example, the enemy
sound direction indicator in “The Persistence” [22].
Our work is infuenced by prior work from D. Jain et al., which
proposed a sound taxonomy in VR [18]. And also, our study is
inspired by another research that presented an evaluation of their
prototypes developed using visual-based and haptic-based repre-
sentation methods for VR sounds [19]. However, to the best of our
knowledge, limited research examines sound visualization meth-
ods in VR in detail. Therefore, it would be benefcial to conduct
user experiments to help gain qualitative and quantitative insights,
such as performance, usability, and user experience, of the sound
visualization method designs for VR.
Figure 2: The design of the Sound-Characteristic Indica-
tor. (Study 1 game scene is shown.) The smaller red object
is the Sound-Characteristic Indicator which appears when
the sound source object makes a sound and dynamically
changes its size during the sound. The waveform in the
speech bubble shows a sound being played, and the circles
above indicate how the indicator changes the size based on
the loudness characteristics of the sound wave.
The Sound-Characteristic Indicator was designed as a customiz-
able 3D object (a Unity plugin). It is attached as a part of any sound
source object in the VR design. The indicator appears in a Mini-Map
and on the sounding object when the object starts making sounds
and disappears at the end of the sound. As shown in Figure 2, it can
visualize the current loudness of the sound by dynamically chang-
ing its size - when the sound gets louder, the indicator expands;
when the sound is lower, the indicator shrinks. In this way, the
dynamic changes in the object’s size indicate the sound’s loudness,
and the appearance and disappearance of the object relate to the
duration of the sound.
In addition, to indicate the location of the sounding objects and
inform the loudness of the sounds, we designed two components
for our proposed sound visualization system: Mini-Maps and On-
Object Indicator.

3 SOUNDVIZVR SOUND VISUALIZATION

SYSTEM
The scope of this work mainly focuses on diegetic types of sounds,
specifcally localized speech, inanimate objects, animate objects,
and point ambience [18]. To visualize these sounds in VR, we fo-
cused on two aspects of the sound indicators - 1) Sound-Characteristic
Indicator and 2) Sound-Type Indicator. The Sound-Characteristic Figure 3: Three types of Sound-Characteristic Indicators: (a)
Indicator was implemented to present information such as the On-Object Indicators; (b) Full Mini-Map Indicator; (c) Partial
location, loudness, and the duration of sounds. The Sound-Type Mini-Map Indicator
Indicator was used to convey more meaningful information about
the sounds, such as a footstep sound from a person walking nearby.
ASSETS ’22, October 23–26, 2022, Athens, Greece
3.1.1 On-Object Indicator. We wanted the user to be aware of the
sound playing from a sounding object while looking at the object,
which is an intuitive way to identify a sounding object. Hence, as
seen in Figure 3a, we placed the Sound-Characteristic Indicator
object hovering near the top of the object.
The appearance of the indicator is customizable. To make its
design simple and visually apparent, we designed the appearance
of the On-Object Indicator as a red-colored sphere for Study 1.
However, designers may customize the indicator using any 3D
objects or images based on their requirements.
3.1.2 Mini-Maps. The Mini-Map systems are commonly used in
many video games [49]. The Mini-Map design can present play-
ers with information about their surroundings. So the player can
keep track of the notable updates around them, especially those
in their blind spots. Inspired by such map designs, we integrated
a circular Mini-Map, which is called a “Full Mini-Map”, into our
proposed system to help users locate the sound source (Figure 3b).
Through trial and error and feedback from pilot demos, we placed
the Mini-Map on the screen’s left side, similar to many video games.
Furthermore, the same Sound-Characteristic Indicator described
above is visualized to present sound source information on the Full
Mini-Map.
In addition to the Full Mini-Map, we also present a Partial Mini-
Map (Figure 3c). Here, the front sector of the Partial Mini-Map,
which represents the user’s current feld of view, is hidden. We
explored this design so that the user is only presented with sound
information that is not in their feld of view. In addition, this design
aims to encourage users to pay more attention to the environment
in front of them and observe potentially sounding objects instead
of over-relying on the information shown on the Mini-Maps.
3.2 Sound-Type Indicator
Figure 4: The design of Sound-Type Indicator. (a) The Sound-
Type Indicator with text on the Full Mini-Map and an icon
on the object. Here, in the player’s view, the plushie rabbit
makes a “talking sound”, and its corresponding text and icon
are shown. (b) Icons used for the presented sounds. (c) Text
descriptions of the sounds.
Besides knowing the sound characteristics, it is essential to un-
derstand what types of sounds are presented in a scenario. Thus,
to assist the DHH users in identifying the types of the sounds in
the VR environment, we explored customizing the sound indicator
in the On-Object Indicator and Mini-Maps with icons and text (Fig-
ure 4a). The size of the icon (Figure 4b) and text (Figure 4c) changes
along with the loudness of the indicator’s corresponding sound. In
this way, a sound indicator can inform the sound characteristics to
Li, Connell, Dannels, and Peiris
the user and further notify the sound type via its iconic or textual
label at the same time.
3.3 System Implementation
Figure 5: The implementation of a game scene developed us-
ing the Unity game engine.
We developed two task-based VR game scenes with our proposed
system integrated using the Unity game engine version 2021.1.15f1.
Both scenes can be deployed to Oculus’s PC-powered VR headsets.
In our case, an Oculus Rift S with dual VR controllers is used [6].
To implement the sound indicator, frst, we need to obtain the
sample data from the audio source of the current time frame. Then,
we retrieve the average absolute value from the sample data and
map it to the scale value of the sound indicator object of each frame.
As a result, we can see the indicator’s size changes according to the
current loudness of the audio source (Figure 2) during the game
engine’s run-time.
In terms of the Mini-Maps’ implementation, we use an additional
virtual camera in the Unity game engine that is attached to the
player at the ceiling level (Figure 5). This virtual camera always
faces the ground. It ignores all other objects in the view except for
the sound indicators. By rendering this virtual camera’s view into
the Mini-Map’s texture, the Mini-Maps can refect sound indicators
around the player. After importing the plugin to the Unity editor, the
designer could drag and drop the add-on prefab onto a sound-source
object. Next, the designer would be able to customize the 3D objects,
icons, and text based on the preferences or design requirements.
As mentioned above, the appearance of the sound indicator in the
On-Object Indicator and Mini-Maps is customizable. Our prototype
allows the designers to replace the default red sphere object with
a preferred icon object or a descriptive text object for each sound
source to further visualize the sound type information in a game
scene.
4 STUDY 1: EVALUATING THE
SOUND-CHARACTERISTIC INDICATOR
The main goal of the frst study is to evaluate the performance and
the user experience of our proposed Sound-Characteristic Indicator
visualization methods on locating the sound source and visualizing
sound characteristics like duration and loudness. For this purpose,
we conducted a user experiment in which users were required to
localize sound sources in VR as the main task to identify the best
performing and the most preferred Sound-Characteristic Indica-
tor method from the possible design combinations. Both studies
SoundVizVR: Accessible VR Sounds for DHH Users with Sound Indicators
presented in the article were approved by the institution’s ethics
review board.
4.1 Task Design
Figure 6: Study 1 task design - (a) VR environment of Study
1; (b) The participant’s view of the scene while performing
a task using the FM-OI visualization method.
Similar to the task design in EarVR [26], as the main task, the
participant was required to locate the object (presented as spheres)
from which the sound originated (Figure 6). In the experimental
scene, we placed the player in the center of the stage with eight
identical spheres evenly placed around the player in a circle. The
spheres were labeled from 0 to 7 in counterclockwise order. The
player could look around or lead to any direction but was not
required to walk in the scene.
In the experiment, one of the eight spheres served as a sound
source and started to play a random sound clip selected from an
audio clip set. The audio clip set was formed based on four cate-
gories chosen from the taxonomy of sounds in VR proposed by Jain
et al. that represented diegetic sounds [18]: Localized speech, Inan-
imate objects, Animate objects, and Point ambience. The selected
audio clips were organized into two duration scopes: short duration
sounds (a sound clip’s length was shorter than 3 seconds) and long
duration sounds (a sound clip’s length was over 15 seconds). In
total, eight sound clips were included in the set. The audio clip set
used in the experiment is shown in Table 1. It was selected from
the royalty-free sound clip website “SoundBible” 3 to represent a
wide range of sounds from VR games. In addition, short sound clips
were explored to identify the impact of the visualization techniques
on locating quick sounds that may appear and disappear in the
visualization.
Table 1: Study 1 sound selection
Category Short Duration Long Duration
Localize Speech Old Man Laugh Baby Talk
Inanimate Objects Gun Shot Phone Ringing
Animate Objects Footstep Barking Dog
Point Ambience Fire Burning Waterfall
In the task, the participant was asked to select the sounding
sphere using a VR controller as a pointer. If the participant could
not identify the sounding sphere after 7 seconds from the start of
the sound, they were allowed to press a “skip” button on the VR
controller to skip to the next task. As the localization accuracy rate,
3 https://soundbible.com/
ASSETS ’22, October 23–26, 2022, Athens, Greece
Table 2: Study 1 conditions
On-Object Indicators
Independent Variable
Without With
Without NON OI
Mini-Map Full FM FM-OI
Partial PM PM-OI
we calculated how many times a participant accurately selected
a correct sounding sphere. The next random sphere started play-
ing another random sound clip in 3 seconds after the participant
chose a sphere or pressed the “skip” button. One repetition of a
condition consisted of eight such tasks (eight diferent sounds from
eight spheres) and each condition was repeated three times. The
participant had the right to stop the experiment at any point of
time.
4.2 Study Design
Study 1 used a within-subject evaluation design that consisted
of two independent variables: On-Object Indicator (Without On-
Object Indicator or With On-Object Indicator) and Mini-Map Type
(Full Mini-Map, Partial Mini-Map, or Without Mini-Map). The de-
pendent variables were sound source localization accuracy and the
time of completion of each task. In addition, we recorded additional
data, such as the head rotation angles. In total, there were six condi-
tions (Table 2): NON (no sound visualization technique was used),
OI (On-Object Indicators only-Fig. 3a), FM (Full Mini-Map technique
only-Fig. 3b), FM-OI (a combination of Full Mini-Map technique
and On-Object sound indicator-Fig. 4a), PM (Partial Mini-Map tech-
nique only-Fig. 3c), and PM-OI (a combination of Partial Mini-Map
technique and On-Object sound indicator). The NON condition was
included as the baseline condition in which no Sound-Characteristic
Indicator was presented, similar to existing VR experiences. Each
task was repeated 3 times. Each participant faced a total of 144
tasks during the experiment (8 tasks x 3 repetitions x 6 conditions).
4.3 Participants
We recruited 11 DHH participants for the study from the authors’
institution (6 males, 2 females, 3 non-binary people; ages 18-45,
Mean = 27.36, SD = 8.82). The participant group consisted of 4 deaf
and 7 hard-of-hearing participants. Five of the participants had used
VR devices before. The participants were recruited through fyers
and word-of-mouth advertising in the institution. Each participant
was paid $15 after completing the user experiment.
4.4 Procedure
After signing the informed consent form, the participant was given
an introduction to the system and asked to fll out a demographic
questionnaire. The information was provided to the participant
through text and slides. However, a hard-of-hearing research team
member used sign language to discuss additional details if and when
necessary. Also, the participant was asked to take of the hearing
aid, if there was one, to ensure the visualization was the focus in this
controlled study. Next, the participant put on the VR headset and
held a VR controller. Before a condition started, the participant was
ASSETS ’22, October 23–26, 2022, Athens, Greece
given sufcient time to get familiar with the VR device’s control, the
VR game scene, and the Sound-Characteristic Indicator interface
based on the condition. As the condition started, the participant was
required to perform the tasks with the corresponding combination
of the Sound-Characteristic Indicator visualization technique of
the condition. The order of the condition for each participant was
assigned randomly. After completing a condition, the participant
was asked to fll out a post-condition questionnaire. Then, the
participant could take a 3-minute break if needed. After completing
all six conditions, the participant was asked to fll out a post-test
questionnaire. The experiment took approximately 75 minutes for
each participant.
4.5 Results of Study 1
Figure 7: Results of Study 1 - (a) The average localization ac-
curacy rate; (b) The average completion time (Sec) of each
task; (c) The average localization accuracy rate on two types
of sounds of Study 1. Error bars denote the standard devia-
tion.
4.5.1 Localization accuracy and completion time. Figure 7a shows
the localization accuracy rate of Study 1. The results were further an-
alyzed using repeated measures ANOVA with Greenhouse-Geisser
correction. A signifcant main efect was found on Mini-Map Type
(F1.53,15.37 = 41.618, p < 0.001), and on On-Object Indicator (F1,10
= 40.436, p < 0.001). Post-hoc comparison between each pairs of
Mini-Map Type showed NON, FM: p < 0.001; NON, PM: p < 0.001;
FM, PM: p < 0.001. Post-hoc comparison between each pairs of On-
Object Indicator showed NON, OI: p < 0.001. Post-hoc comparison
between each combination showed signifcant diference between
each pairs, except FM, OI: p = 0.540; FM, FM-OI: p = 1.000; FM,
PM-OI: p = 1.000; OI, FM-OI: 0.258; OI, PM-OI: 0.540; FM-OI, PM-OI:
p = 1.000.
Figure 7b shows the task completion time of Study 1. The re-
sults were further analyzed using repeated measures ANOVA with
Li, Connell, Dannels, and Peiris
Greenhouse-Geisser correction. A signifcant main efect was found
on Mini-Map Type (F1.591,15.912 = 27.453, p < 0.001), and on On-
Object Indicator (F1,10 = 31.983, p < 0.001). Post-hoc comparison
between each pairs of Mini-Map Type showed NON, FM: p < 0.001;
NON, PM: p < 0.001; FM, PM: p = 0.029. Post-hoc comparison be-
tween each pairs of On-Object Indicator showed NON, OI: p < 0.001.
Post-hoc comparison between each combination showed signifcant
diference between each pairs, except FM, OI: p = 0.363; FM, FM-OI:
p = 1.000; FM, PM-OI: p = 1.000; PM, OI: p = 0.843; OI, FM-OI: p =
0.085; OI, PM-OI: p = 0.146; FM-OI, PM-OI: p = 1.000.
Figure 7c shows the localization accuracy rate on two types of
sound duration of Study 1. The results were further analyzed using
paired samples t-test. A signifcant main efect was found on all six
conditions - NON: p = 0.003; OI: p < 0.001; FM: p = 0.006; FM-OI: p
= 0.003; PM: p = 0.006; PM-OI: p < 0.001.
Table 3: The System Usability Scale Scores and Adjective Rat-
ings of Study 1
Condition SUS Score (SD) Adjective Rating
NON 52.05 (21.12) Poor
OI 71.14 (18.45) Good
FM 84.55 (15.24) Excellent
FM-OI 84.77 (16.03) Excellent
PM 51.82 (23.69) Poor
PM-OI 76.36 (24.91) Good
4.5.2 System usability and subjective mental workload. Table 3
shows the System Usability Scale (SUS) results of Study 1. The
SUS adjective rating is obtained from the 7-point adjective scale
proposed by Bangor et al. [2].
Figure 8: NASA TLX scores of Study 1. The lower ratings in-
dicate lower task loads.
Figure 8 shows the subjective mental workload results collected
from NASA Task Load Index (NASA TLX) questionnaires [12]
across six dimensions: mental demand, physical demand, temporal
demand, performance, efort, and frustration.
4.6 Discussion of Study 1
Overall, our results indicated that all the conditions with a Sound-
Characteristic Indicator performed signifcantly better than the
NON condition with FM-OI, FM, PM-OI achieving higher than 90%
accuracy for sound source localization. The NON condition, which
has no integrated sound visualization methods similar to the exist-
ing VR experiences, achieved the lowest localization accuracy (M:
SoundVizVR: Accessible VR Sounds for DHH Users with Sound Indicators
24.24%, SD: 0.29) and the longest task completion time (M: 8.93s,
SD: 2.21). During further analysis of the NON condition, we found
that the correct localization was achieved by a few hard-of-hearing
participants who, although, indicated that it was difcult for them
to confdently locate the sound source without the assistance of
visual cues. For example, one of the hard-of-hearing participants
(F02) wrote in the feedback: “I could not identify any of the sources
of sound securely, I could maybe localize the sound to 3 spheres [three
spheres that present in the view], but past that it’s beyond me.” Simi-
larly, the deaf participants reported that they could not either tell
the location of sounds or the existence of a sound without any visu-
alization. The NASA TLX ratings for the NON condition reported
the highest ratings (higher ratings indicate more cognitive load
requirements) for all categories except for the temporal demand
category (Figure 8).
In terms of the FM condition, which introduced a Full Mini-Map
Sound-Characteristic Indicator component compared to the NON
condition, its localization accuracy reached 93.94% (SD: 0.09), and
task completion reduced to 4.73s (SD: 0.89). It also outperformed
the SUS score (with an “Excellent” adjective rating) and showed
relatively low subjective mental workloads across all NASA TLX
dimensions. It might indicate that the participants were able to
identify the sound locations and perform tasks with the assistance of
this method. The participants’ feedback supported this observation.
For example, F12 wrote: “the Full Mini-Map helps me fnd the sound
location more accurately.” Those who had a wide experience in
playing games indicated that they might get used to the Full Mini-
Map method quickly since the Full Mini-Map was commonly used
in games [49]. For example, F03, who played games 4-6 times a
week, wrote: “I think that because I play video games with full circle
Mini-Map on the screen, it is very easy for me to engage in this
system.” However, some participants reported they had to rely on
the map to perform tasks, even without paying attention to the
environment. They also mentioned that, with the Full Mini-Map
alone, they could fail to pinpoint the sound source from two close
objects. For example, F02 added: “I’m not entirely sure how accurate
I was because I often had to choose between two of the spheres.”
The FM-OI condition, which was developed based on FM, com-
bined a Full Mini-Map component with On-Object Sound-Characteristic
Indicators that also supported visualizing the sound of an object
in the environment. FM-OI had similar yet marginally better per-
formance compared to FM on localization accuracy (M: 94.70%,
SD: 0.07) and task completion time metrics (M: 4.41s, SD: 0.93), al-
though not statistically signifcantly diferent. Similar to FM, FM-OI
achieved an “Excellent” adjective rating in the SUS score and rela-
tively low mental workloads across all six NASA TLX dimensions.
The participants’ comments reported that FM-OI enabled them to
pay more attention to their surroundings with the assistance of
the On-Object Indicators. “Since the notifcation appears on both
map and game, I was able to focus on the game and use the map as a
support.” (F01) They reported that the Full Mini-Map component
was useful to assist them in identifying the sound direction, and the
On-Object Indicator component was helpful when identifying the
specifc sounding sphere. “If there are two objects that are quite close
to each other, knowing just the direction of the sound may not always
be helpful. In such cases, having an environment sound indicator to
ASSETS ’22, October 23–26, 2022, Athens, Greece
diferentiate between two objects in proximity would be an added
advantage.” (F14)
In addition, we looked into OI to better understand the efect of
the On-Object Indicator alone. Although there are no statistically
signifcant diferences when comparing its localization accuracy
(M: 82.58%, SD: 0.15) and completion time (M: 5.71s, SD: 1.41) with
FM and FM-OI, OI achieved signifcantly high subjective mental
workloads in NASA TLX scores among all six dimensions compared
to FM and FM-OI. The participant reported that they had to scan
through all the objects to fnd the audio cues due to the lacking
indicators of the sound direction. “The problem with this system is
although it does indicate the sound, it doesn’t indicate the direction of
the sound which could waste time in a high pressure gameplay.” (F07)
We looked into the results of PM and PM-OI to investigate if
the Partial Mini-Map worked in reducing the information and en-
abled participants to focus on the environment as we intended. PM
achieved the lowest localization accuracy (M: 60.23%, SD: 0.26) and
longest task completion time (M: 6.38s, SD: 2.08) among the fve
proposed methods. In addition, the participants reported that it was
difcult to identify the direction of the sounds with this method.
“If the sound was in front of me, I had no idea which of the 2 spheres
it could be unless I looked to the side to see the dot on the map.” (F02)
These kinds of feedback may back up PM’s low SUS adjective rating
(“Poor”) and relatively high mental workload performance, espe-
cially in mental demand, temporal demand, efort, and frustration
dimensions. In terms of the PM-OI condition, with an On-Object
Sound Indicator component integrated, PM-OI achieved similar
performance in localization accuracy (M: 91.67%, SD: 0.09) and task
time completion (M: 4.55s, SD: 0.87) metrics compared to FM and
FM-OI, which showed no statistically signifcant diference. The
participants reported that the On-Object Sound Indicator enabled
them to locate the sound source within multiple spheres on their
front, which overcame the disadvantages of PM alone. “The sound
indicators help me fnd the sound location better when I can’t fnd
[it] with the Partial Mini-Map. If I [was] unable to locate the sounds
with the front cut up, the sound indicators helped me fnd it faster
and accurately.” (F12) Also, the participants reported that PM-OI
could assist them in focusing more on the game itself. For example,
“The dots over the top made things much easier, and
having the radar showed what’s around me made it
clear when to turn. This is by far the easiest way to
achieve the tasks. And yet the only time I missed the
target. I learned there is no replacement, meaning that
after one came and went, it would not be there again.
That led me to reorient to the unselected options to
get a ‘jump’ on them. Fun combination.” (F08)
However, some participants indicated that the sound localization
experience on PM-OI was ambiguous. As F07 said: “I can’t tell if I’m
looking directly at the source of the noise in the Mini-Map with the
notch and due to the nature of VR, I don’t have the same feld of vision
as I would have in real life. So I have to put in a little extra efort to
identify the sphere.” This may support PM-OI’s higher subjective
mental workload result of the Performance dimension in the NASA
TLX ratings, especially when compared to FM and FM-OI.
When considering the duration of the clips used, sound localiza-
tion in long-duration sounds had a signifcantly better localization
ASSETS ’22, October 23–26, 2022, Athens, Greece
accuracy than in short-duration sounds in all conditions. This result
was expected here as we identifed that if the participant was not in
the view of a short duration sound, it had a higher chance of being
missed in a majority of the conditions. Here, F01 indicated: “For
sounds that disappeared too quickly, it’s difcult to notice if there were
any sound at all. This is especially true for ‘no map with indicator’,
‘Partial Mini-Map without indicator’, and ‘Partial Mini-Map with
indicator’.” About the OI condition, F14 discussed: “For this task, I
had to scan through all the spheres until I found an audio indicator
on top of one of the spheres. Sometimes, for short audio, I guess the
audio indicator disappeared by the time I scanned through the spheres,
which was not helpful in identifying the target... However, having
an audio indicator was a lot better than having no visual cues at
all!” Also, it should be noted that we used diferent sounds for the
diferent durations of the same type of sounds (we did not use a
short and long version of the same clip) to represent a larger variety
of sounds.
4.7 Summary of Study 1
Based on the analysis and observation of the results, we selected the
FM-OI as the Sound-Characteristic Indicator visualization method
for Study 2. We chose FM-OI over the six conditions primarily
because it had the best performance data in both localization ac-
curacy and task completion time. Similarly, the results from SUS,
NASA TLX questionnaires, and participants’ comments of Study 1
reported above supported our decision. Furthermore, in the post-
test questionnaire, most participants (7/11) indicated that FM-OI
was the sound visualization method they liked most among the fve
proposed methods, while preferences over the other methods were
diverse (PM-OI: 2/11, OI: 1/11, FM: 1/11).
5 STUDY 2: EVALUATING THE SOUND-TYPE
INDICATOR
Figure 9: VR environment of Study 2 - (a) An overview of the
Study 2’s scene. A participant stands at the location marked
by the blue arrow; (b) The participant’s view of the scene
before a task. Before the task, the participant was shown the
sound source objects with their labels displayed.
Based on the results from Study 1, we selected the FM-OI Sound-
Characteristic Indicator visualization method for the following
studies. As such, the main goal of Study 2 is to evaluate the perfor-
mance and user experience of the diferent Sound-Type Indicator
visualization methods (texts and icons).
5.1 Task Design
To present a more realistic VR scene for this experiment, we used
the Kid’s Room model package [5] and several additional object
models [43, 45] from the Unity Asset Store to build our VR game
Li, Connell, Dannels, and Peiris
scene for the study (Figure 9a). The participant was placed near
center of the scene and the objects that serve as speakers were
distributed in the scene. Similar to Study 1, we did not require the
participants to move in the game scene. However, the participants
could still rotate physically in place with minor lateral movements
(rotate their bodies or lean towards a target) while wearing the VR
headsets. Before a condition began, the participant was allowed to
familiarize with the sound source objects that were indicated using
a label showing the name (category) of the object (Figure 9b). In
addition, we used the same set of sound clips from the previous
study and used the same text label shown in Figure 4c. The used
icons (Figure 4b) were selected from a free online icon database
website called “Flaticon” 4 . To prevent any biases (due to preferences
and/or color blindness), we used black and white icons.
Table 4: The task list of Study 2
No. Task Instruction
1 Find the radio making the fowing water sound
2 Search for the item making a sound
3 Find and identify the ringing telephone
4 Find the object making the baby talking sound
5 Find and identify the sounding plush toy
6 Find the radio making the gunshot sound
7 Find the object making the campfre sound
8 Find the plushie making the Santa Claus laughter sound
Figure 10: Study 2 task design - (a) Task instruction panels;
(b) The participant’s view of the scene when performing a
task. The TM-IO condition is shown.
As the main task, the participant was presented with a task from
a list of task instructions (Figure 10a) as described in Table 4. It was
presented in the main view for 7 seconds to ensure the participant
had enough time to read it, and afterward, it was maintained in the
upper region of the participant’s VR view. Next, the VR scene played
three diferent sounds from at most three sound source objects at
the same time (Figure 10b). The participant was required to select
the sound source object that played the sound specifed in the
instruction. We designed such a task to encourage the participant
to focus more on the sound-type labels during the task. For example,
during a given task, all three radios in the VR scene might make
sounds, but if Instruction 1 was presented, the participant was
required to identify the correct radio by looking for the “water
fowing” icon or text label.
4 https://www.faticon.com/
SoundVizVR: Accessible VR Sounds for DHH Users with Sound Indicators
The recording of task completion time began only at the point
in time when the correct sound clip started to play from an object.
As the localization accuracy rate, we calculated how many times a
participant accurately selected a correct sounding object after the
correct sound clip started to play.
Similar to Study 1, the participant was able to skip the task if
they could not identify the correct task after 10 seconds by pressing
the skip button. The participant also had the right to stop the
experiment at any point of time. Eight such tasks (eight diferent
sounds from eight objects) were included in each repetition of a
condition. And each condition was repeated twice for a participant.
5.2 Study Design
The experiment of Study 2 used a within-subjects evaluation design.
It consisted of two independent variables: Sound-Type Indicator on
Objects (icon or text) and Sound-Type Indicator on the Full Mini-
Map (icon or text). The dependent variables were sound source
localization accuracy and the task completion time while searching
for the correct sound source. Similar to Study 1, we recorded ad-
ditional data such as the head rotation angles. In total, there were
four conditions as shown in Table 5: IM-IO (Icon indicators on the
Full Mini-Map + Icon indicators on Objects), IM-TO (Icon indicators
on the Full Mini-Map + Text indicators on Objects), TM-IO (Text
indicators on the Full Mini-Map + Icon indicators on Objects), and
TM-TO (Text indicators on the Full Mini-Map + Text indicators
on Objects). Each participant faced 64 tasks in total (8 tasks x 2
repetitions x 4 conditions).
Table 5: Study 2 conditions
Indicators on Objects
Independent Variable
Icon Text
Icon IM-IO IM-TO
Indicators on Full Mini-Map
Text TM-IO TM-TO
5.3 Participants
We recruited 14 DHH participants for the study from the authors’
institution (8 males, 4 females, 2 non-binary people; ages 18-45,
Mean = 26.21, SD = 8.31). The participant group consisted of 6 deaf
and 8 hard-of-hearing participants. Six of the participants had used
VR devices before. Ten of the participants had previously taken
part in Study 1. However, it should be noted that the two research
studies were conducted separately. The participants were recruited
through advertising in the institution by fyers and word of mouth.
Each participant was paid $25 after completing the user experiment.
5.4 Procedure
Similar to Study 1, after signing up for the informed consent form,
the participant was given an introduction to the system and asked
to fll out a demographic questionnaire. Next, after the participant
put on the VR system, they were given sufcient time to familiarize
themselves with the VR device’s control and the VR game scene. In
addition, the participants were allowed to familiarize themselves
with the instructions and the name (category) labels and locations of
the sound source objects. As the condition started, the participant
ASSETS ’22, October 23–26, 2022, Athens, Greece
was required to perform the tasks with the sound visualization
technique of the condition. The order of the condition for each
participant was assigned randomly. After completing a condition,
the participant was asked to fll out a post-condition questionnaire.
Then, the participant could take a 3-minute break if needed. After
completing all four conditions, the participant was asked to fll out
a post-test questionnaire. The experiment took approximately 45
minutes for each participant.
5.5 Results of Study 2
Figure 11: Results of Study 2 - (a) The average localization
accuracy rate of each condition; (b) The average task com-
pletion time (Sec) of each condition. Error bars denote the
standard deviation.
5.5.1 Localization accuracy and task completion time. Figure 11a
shows the localization accuracy rate of Study 2. Repeated measures
ANOVA revealed no signifcant diference on Indicators on Full
Mini-Map (F1,13 = 0.198, p = 0.664) and on Indicators on Objects
(F1,13 = 2.021, p = 0.179).
Figure 11b shows the task completion time of Study 2. Repeated
measures ANOVA revealed no signifcant diference on Indicators
on Full Mini-Map (F1,13 = 6.596e − 4, p = 0.980) and on Indicators
on Objects (F1,13 = 0.409, p = 0.553).
Table 6: The System Usability Scale Scores and Adjective Rat-
ings of Study 2
SUS Score (SD) Adjective Rating
IM-IO 81.96 (14.01) Excellent
IM-TO 73.39 (18.78) Good
TM-IO 75.00 (15.81) Good
TM-TO 76.25 (18.47) Good
5.5.2 System usability and subjective mental workload. Table 6
shows the SUS results of Study 2. Figure 12 reveals the subjec-
tive mental workload results of Study 2 collected from NASA TLX
questionnaires across six dimensions: mental demand, physical
demand, temporal demand, performance, efort, and frustration.
5.6 Discussion of Study 2
5.6.1 Sound Type Identification. Overall, the quantitative results
indicate that DHH participants can perform the tasks with the
assistance of the four evaluated Sound-Type Indicator visualization
methods and achieved high localization accuracy at around 90%.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 12: NASA TLX scores of Study 2. The lower ratings
indicate lower task loads.
Based on the analysis of the results, there is no signifcant diference
among all tested Sound-Type Indicator visualization methods on
both localization accuracy and task completion time metrics of
Study 2. It indicates that the overall performance of the four tested
Sound-Type Indicator visualization methods are similar.
The icons used in Study 2 were selected based on what a designer
would pick as icons for the user interface applying a nomic sound
mapping method [8, 21]. Prior to the study, we did not present
the icons used in the experiment and their descriptive text to the
participants. It aimed to explore if the participants could identify
the unfamiliar icons shown during the experiment. In the post-test
questionnaire, we presented two fve-point Likert scale questions to
respectively investigate if participants could understand the mean-
ings of icons and text shown in the indicator. The results show
that the icons (M: 3.93, SD: 0.83) and text (M: 3.93, SD: 0.92) have
similar ratings, which indicates the participants can understand
the meanings of both the icons and text. The results of the system
usability evaluation can support this conclusion. The usability re-
sults revealed that IM-IO, which used icons in both Full Mini-Map
and On-Object Indicators, achieved the “Excellent” adjective rating.
In the subjective mental workload results (Figure 12), IM-IO out-
performed fve of the six dimensions: physical demand, temporal
demand, performance, efort, and frustration. And it also achieved
a relatively low mental demand load. In the participants’ feedback,
G06 said: “Iconic shows me a lot which allows me to identity easier
and quicker.” G02 added: “It [the iconic representative method] is
something to get use for a bit, but it is easy to understand.”
In terms of using descriptive text as Sound-Type Indicators in
Study 2, TM-TO achieved the lowest mental demand in the subjec-
tive mental workload scale and a “Good” adjective system usability
rating. The participants’ feedback supported that the text represen-
tation method is easier to understand in most cases. For example,
G02 said: “Texts on the environment work well as it describes the
sounds well good enough like phones ringing or waterfall.” “[Text has]
Nothing lost in translation.” G14 added. However, when it came to a
situation where the descriptive text in the Sound-Type Indicators
did not exactly match the instruction, the text Sound-Type Indicator
visualization may cause a higher cognitive load. For example, the
instruction of task No.7 is “Find the object making the campfre
sound” (Table 4), while the sound type label of the correct sound
source shows “Crackling Fire” (Figure 4c). G16 noted: “Inconsistency
in the wording of objects (prior to task vs during task) made it more
difcult to identify the appropriate object.”
Li, Connell, Dannels, and Peiris
5.6.2 Sound Localization. Unlike Study 1, in which the sound
sources were evenly organized around the player in a circle, we ran-
domly distributed the sound sources in the VR environment in Study
2 to mimic the sound design of an actual VR game scene. There
were sound sources that stayed close to each other, and a sound
source was put at position right above another sound source (same
azimuth angle, but diferent altitude). Here, although the Mini-Map
Indicator only indicated the directions of the sound sources in a 2D
plane and did not show the altitude information of the sound source
in a 3D space, we believe that the On-Object Indicator overcame
this issue. Unfortunately however, no comment on this aspect was
received from the participants.
In Study 2, some tasks had multiple (at most three) sounds hap-
pening simultaneously. The tasks of Study 2 with multiple sounding
objects were designed to explore if the DHH participants could
localize the correct sound source among many with the help of
the visualization methods. Based on the analysis of the qualitative
results, most of the participants (11/14) indicated that the visualiza-
tion methods worked well in assisting them to identify the sound
source when multiple sounds were happening in the scene. For
example, when talking about the TM-TO, which used text to indi-
cate sound types, G10 said: “It [TM-TO] works well in cases where
there are multiple diferent types of noise and we need to identify a
specifc sound. This is dependent on how descriptive the text is how-
ever.” However, participants also mentioned when the simultaneous
sounds contained short duration sounds, they would still struggle
to localize the correct sound source, even with the assistance of the
visualization methods. For example, G05 noted: “Works well when
there are various sounds going on. Doesn’t work well when it quickly
fashed and went away when I was trying to read multiple texts to
identify the object making the noise.”
5.6.3 Preferences. In terms of the procedure of searching for the
sound source using the visualization methods, most of the partic-
ipants (10/14) indicated that they frst used the Full Mini-Map to
identify the direction of the correct sound clip, and then use the On-
Object Indicator to locate the specifc sound source. For example,
G06 said: “I looked at the map where the icon is, then I search around
the room to fnd the source of the sound where the icon marked on
the map.” This primarily followed the similar pattern of locating
the sound source using FM-OI as indicated by the participants in
Study 1.
In addition, we investigated the participants’ preferences on the
four tested Sound-Type Indicator visualization methods, especially
preferences on using text or icons to indicate the sound types. For
the overall preferred sound visualization method of Study 2, the
participants provided diverse feedback - IM-IO: 4/14, IM-TO: 3/14,
TM-IO: 3/14, TM-TO: 4/14. The participants who preferred icons
indicated that icons had lower cognitive load, allowing them to
focus on the game content. For example, G16 mentioned: “I thought
that iconic indicators are easier/quicker to understand than texts. I
do not prefer reading a lot when I play VR games.” The participants
who tended to use text as Sound-Type Indicators voiced that text is
more noticeable in the scene and can clearly convey its meaning.
As noted by G10, “Text is easier for me to notice on the map, and also
to identify the specifc type of noise.”
SoundVizVR: Accessible VR Sounds for DHH Users with Sound Indicators
While we could not come to a consensus on what might be
the most preferred method for indicating sound types, this also
indicates that the future designs of Sound-Type Indicators may
allow the user to customize what to be displayed based on their
own preferences.
6 DISCUSSION, LIMITATIONS AND FUTURE
WORK
The results of Study 1 show that FM-OI had the best performance in
localization accuracy and task completion time. The results from the
questionnaires reveal that FM-OI has a good system usability and
less subjective mental workloads. Also, the participants’ feedback
shows that FM-OI can allow them to pay more attention to the game
content while spending less efort in locating the sound source.
The results of Study 2 show that the four tested Sound-Type
Indicators have similar performance on localization accuracy and
task completion time. The participants can identify the sound type
of a sound source and complete the tasks with the assistance of
the tested sound visualization methods. Moreover, the participants
have diverse preferences on the four tested Sound-Type Indica-
tors in terms of using text or icons to indicate the sound types in
the visualization methods, thus indicating a customizability of the
Sound-Type Indicators based on the individual preferences.
There are several limitations we wish to address.
6.1 Limitations
Our current study addressed four sound categories from the sound
taxonomy in VR proposed by Dhruv Jain et al. [18], while another
fve sound categories were not explored here. These unaddressed
categories were Non-localized speech, Notifcation sounds, Inter-
action sounds, Surrounding ambience, and Music. To provide full
immersive and accessible sound experience, our future studies will
explore on these unaddressed categories.
We evaluated our sound visualization methods in a scenario
with at most three synchronous sound sources in our Study 2. The
performance of the sound visualization method in scenarios with
more than three concurrent sound sources (e.g., a large crowd of
talking people) was untested. In that case, fltering strategies of
sounding objects might need to be investigated.
The Full Mini-Map component in our proposed sound visualiza-
tion methods indicated sounds within its circular range. For the
sounds that were outside the range, DHH users can only identify
those, that were within their feld of view, with the assistance of On-
Object Sound Indicators. We aim to explore visualizing the distant
sounds on the Full Mini-Map in our future study.
In our second study, as a starting point, we selected the sound
type icons and text descriptions (and their display parameters such
as the size, black and white color, etc.) based on trial and error with
a few pilot studies. However, such content can be changed based on
user preferences such as the icon types, shorter/longer descriptions,
text and icon sizes/colors, etc. Therefore, these parameters should
be investigated in the future from a user preference perspective as
well as a content-designer’s perspective.
ASSETS ’22, October 23–26, 2022, Athens, Greece
6.2 Future Works
In our future studies, we aim to explore integrating SoundVizVR into
diferent genres of VR applications, especially VR games, to further
evaluate its performance. In addition, we aim to explore Sound-
VizVR in the 3D space [39], how it will afect the immersive ex-
perience and also explore other visualization techniques. Options
that enable customization to the integrated SoundVizVR compo-
nents (e.g., changing the position, size, opacity, or background of
the Mini-Maps, or changing the size or color of the icon and text
in the Sound-Type Indicator) will also be explored in our future
studies.
We also intend to explore using the SoundVizVR plugin with VR
content designers. This would enable us to determine any prefer-
ences of the method and the plugin from a game designer’s perspec-
tive such as the how easy it is to add this plugin to the workfow,
how customizable the experiences are, etc.
In addition, to have a better understanding of SoundVizVR, we
are looking forward to applying eye-tracking devices in our user
experiments to see how participants perceive information from the
user interface. Also, we are looking forward to evaluating Sound-
VizVR in other user groups (e.g., hearing people) to explore if it
could address other accessibility issues, such as situational impair-
ments among the hearing people during their using VR applications
[37]. This knowledge may help further improve our designs.
7 CONCLUSION
In this paper, we proposed SoundVizVR that aimed to advance
sound accessibility in VR environments for DHH users. We con-
ducted a user experiment with 11 DHH participants to identify the
best performing and most preferred Sound-Characteristic Indicator
method from six design combinations. Furthermore, to evaluate
the performance and user experience of four diferent Sound-Type
Indicator visualization methods, we conducted another user study
with 14 DHH participants. Participants’ task performance and feed-
back indicated that SoundVizVR can assist them in locating sound
sources, identifying sound characteristics, and identifying the sound
types of in-game sound efects in VR environments.
REFERENCES
[1] Belén Agulló, Mario Montagud, and Isaac Fraile. 2019. Making interaction with
virtual reality accessible: rendering and guiding methods for subtitles. Artifcial
Intelligence for Engineering Design, Analysis and Manufacturing 33 (11 2019), 1–13.
https://doi.org/10.1017/S0890060419000362
[2] Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining What Indi-
vidual SUS Scores Mean: Adding an Adjective Rating Scale. J. Usability Studies 4,
3 (may 2009), 114–123.
[3] Fl’vio Coutinho, Raquel O. Prates, and Luiz Chaimowicz. 2011. An Analysis
of Information Conveyed through Audio in an FPS Game and Its Impact on
Deaf Players Experience. In 2011 Brazilian Symposium on Games and Digital
Entertainment. 53–62. https://doi.org/10.1109/SBGAMES.2011.16
[4] Sigal Eden and Sara Ingber. 2014. Enhancing Storytelling Ability with Virtual
Environment among Deaf and Hard-of-Hearing Children. In Computers Helping
People with Special Needs, Klaus Miesenberger, Deborah Fels, Dominique Archam-
bault, Petr Peňáz, and Wolfgang Zagler (Eds.). Springer International Publishing,
Cham, 386–392.
[5] 3D Everything. 2021. Kids Room | 3D Interior | Unity Asset Store. Retrieved
April 13, 2022 from https://assetstore.unity.com/packages/3d/props/interior/kids-
room-48596
[6] Facebook. 2022. Oculus Rift S: PC-Powered VR Gaming Headset | Oculus. Re-
trieved April 13, 2022 from https://www.oculus.com/rift-s/
[7] Epic Games. 2022. Fortnite – a free-to-play Battle Royale game and more. Re-
trieved April 12, 2022 from https://www.epicgames.com/fortnite/en-US/home
ASSETS ’22, October 23–26, 2022, Athens, Greece
[8] William W. Gaver. 1987. Auditory Icons: Using Sound in Computer Interfaces.
SIGCHI Bull. 19, 1 (jul 1987), 74. https://doi.org/10.1145/28189.1044809
[9] Kathrin Gerling, Patrick Dickinson, Kieran Hicks, Liam Mason, Adalberto L.
Simeone, and Katta Spiel. 2020. Virtual Reality Games for People Using Wheelchairs.
Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.
org/10.1145/3313831.3376265
[10] Kathrin Gerling and Katta Spiel. 2021. A Critical Examination of Virtual Reality
Technology in the Context of the Minority Body. In Proceedings of the 2021 CHI
Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21).
Association for Computing Machinery, New York, NY, USA, Article 599, 14 pages.
https://doi.org/10.1145/3411764.3445196
[11] Ru Guo, Yiru Yang, Johnson Kuang, Xue Bin, Dhruv Jain, Steven Goodman, Leah
Findlater, and Jon Froehlich. 2020. HoloSound: Combining Speech and Sound
Identifcation for Deaf or Hard of Hearing Users on a Head-Mounted Display. In
The 22nd International ACM SIGACCESS Conference on Computers and Accessibility
(Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New
York, NY, USA, Article 71, 4 pages. https://doi.org/10.1145/3373625.3418031
[12] S. G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings
of the Human Factors and Ergonomics Society Annual Meeting 50 (2006), 904 –
908.
[13] Chris J. Hughes, Marta B. Zapata, Matthew Johnston, and Pilar Orero. 2020.
Immersive Captioning: Developing a framework for evaluating user needs. In
2020 IEEE International Conference on Artifcial Intelligence and Virtual Reality
(AIVR). 313–318. https://doi.org/10.1109/AIVR50618.2020.00063
[14] Dhruv Jain, Bonnie Chinh, Leah Findlater, Raja Kushalnagar, and Jon Froehlich.
2018. Exploring Augmented Reality Approaches to Real-Time Captioning: A
Preliminary Autoethnographic Study. In Proceedings of the 2018 ACM Conference
Companion Publication on Designing Interactive Systems (Hong Kong, China) (DIS
’18 Companion). Association for Computing Machinery, New York, NY, USA, 7–11.
https://doi.org/10.1145/3197391.3205404
[15] Dhruv Jain, Brendon Chiu, Steven Goodman, Chris Schmandt, Leah Findlater, and
Jon E. Froehlich. 2020. Field Study of a Tactile Sound Awareness Device for Deaf
Users. In Proceedings of the 2020 International Symposium on Wearable Computers
(Virtual Event, Mexico) (ISWC ’20). Association for Computing Machinery, New
York, NY, USA, 55–57. https://doi.org/10.1145/3410531.3414291
[16] Dhruv Jain, Leah Findlater, Jamie Gilkeson, Benjamin Holland, Ramani Du-
raiswami, Dmitry Zotkin, Christian Vogler, and Jon E. Froehlich. 2015. Head-
Mounted Display Visualizations to Support Sound Awareness for the Deaf and
Hard of Hearing. In Proceedings of the 33rd Annual ACM Conference on Hu-
man Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). As-
sociation for Computing Machinery, New York, NY, USA, 241–250. https:
//doi.org/10.1145/2702123.2702393
[17] Dhruv Jain, Khoa Huynh Anh Nguyen, Steven M. Goodman, Rachel Grossman-
Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon
E. Froehlich. 2022. ProtoSound: A Personalized and Scalable Sound Recognition
System for Deaf and Hard-of-Hearing Users. In Proceedings of the 2022 CHI
Conference on Human Factors in Computing Systems (New Orleans, LA, USA)
(CHI ’22). Association for Computing Machinery, New York, NY, USA, Article
305, 16 pages. https://doi.org/10.1145/3491102.3502020
[18] Dhruv Jain, Sasa Junuzovic, Eyal Ofek, Mike Sinclair, John Porter, Chris Yoon,
Swetha Machanavajhala, and Meredith Ringel Morris. 2021. A Taxonomy of
Sounds in Virtual Reality. In Designing Interactive Systems Conference 2021 (Virtual
Event, USA) (DIS ’21). Association for Computing Machinery, New York, NY,
USA, 160–170. https://doi.org/10.1145/3461778.3462106
[19] Dhruv Jain, Sasa Junuzovic, Eyal Ofek, Mike Sinclair, John R. Porter, Chris Yoon,
Swetha Machanavajhala, and Meredith Ringel Morris. 2021. Towards Sound
Accessibility in Virtual Reality. In Proceedings of the 2021 International Conference
on Multimodal Interaction (Montréal, QC, Canada) (ICMI ’21). Association for
Computing Machinery, New York, NY, USA, 80–91. https://doi.org/10.1145/
3462244.3479946
[20] Dhruv Jain, Kelly Mack, Akli Amrous, Matt Wright, Steven Goodman, Leah
Findlater, and Jon E. Froehlich. 2020. HomeSound: An Iterative Field Deployment of
an In-Home Sound Awareness System for Deaf or Hard of Hearing Users. Association
for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/
3313831.3376758
[21] Peter Keller and Catherine Stevens. 2004. Meaning From Environmental Sounds:
Types of Signal-Referent Relations and Their Efect on Recognizing Auditory
Icons. Journal of experimental psychology. Applied 10 (04 2004), 3–12. https:
//doi.org/10.1037/1076-898X.10.1.3
[22] Firesprite Limited. 2021. The Persistence - Games | PlayStation®. Retrieved
April 12, 2022 from https://www.playstation.com/en-us/games/the-persistence/
[23] Carme Mangiron. 2016. Reception of game subtitles: an empirical study. The
Translator 22, 1 (2016), 72–93. https://doi.org/10.1080/13556509.2015.1110000
arXiv:https://doi.org/10.1080/13556509.2015.1110000
[24] Tara Matthews, Janette Fong, F Wai-Ling Ho-Ching, and Jennifer Mankof. 2006.
Evaluating non-speech sound visualizations for the deaf. Behaviour & Information
Technology 25, 4 (2006), 333–351.
Li, Connell, Dannels, and Peiris
[25] Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual
displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321–
1329.
[26] Mohammadreza Mirzaei, Peter Kán, and Hannes Kaufmann. 2020. EarVR: Us-
ing Ear Haptics in Virtual Reality for Deaf and Hard-of-Hearing People. IEEE
Transactions on Visualization and Computer Graphics 26, 5 (2020), 2084–2093.
https://doi.org/10.1109/TVCG.2020.2973441
[27] Mojang. 2022. Minecraft Ofcial Site | Minecraft. Retrieved April 13, 2022 from
https://www.minecraft.net/en-us
[28] Jorge Mori and Deborah I. Fels. 2009. Seeing the music can animated lyrics
provide access to the emotional content in music for people who are deaf or hard
of hearing?. In 2009 IEEE Toronto International Conference Science and Technology
for Humanity (TIC-STH). 951–956. https://doi.org/10.1109/TIC-STH.2009.5444362
[29] Suranga Nanayakkara, Elizabeth Taylor, Lonce Wyse, and S H. Ong. 2009. An
Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music
Display and a Haptic Chair. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for
Computing Machinery, New York, NY, USA, 337–346. https://doi.org/10.1145/
1518701.1518756
[30] Suranga Chandima Nanayakkara, Lonce Wyse, S. H. Ong, and Eliza-
beth A. Taylor. 2013. Enhancing Musical Experience for the Hearing-
Impaired Using Visual and Haptic Displays. Human–Computer Interac-
tion 28, 2 (2013), 115–160. https://doi.org/10.1080/07370024.2012.697006
arXiv:https://www.tandfonline.com/doi/pdf/10.1080/07370024.2012.697006
[31] De Xing Ong, Kai Xiang Chia, Yi Yi Huang, Jasper Teck Siong Teo, Jezamine Tan,
Melissa Lim, Dongyu Qiu, Xinxing Xia, and Frank Yunqing Guan. 2021. Smart
Captions: A Novel Solution for Closed Captioning in Theatre Settings with AR
Glasses. In 2021 IEEE International Conference on Service Operations and Logistics,
and Informatics (SOLI). 1–5. https://doi.org/10.1109/SOLI54607.2021.9672391
[32] Shanmugam Muruga Palaniappan, Ting Zhang, and Bradley S. Duerstock. 2019.
Identifying Comfort Areas in 3D Space for Persons with Upper Extremity Mobility
Impairments Using Virtual Reality. In The 21st International ACM SIGACCESS
Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19).
Association for Computing Machinery, New York, NY, USA, 495–499. https:
//doi.org/10.1145/3308561.3353810
[33] David Passig and Sigal Eden. 2001. Virtual reality as a tool for improving spatial
rotation among deaf and hard-of-hearing children. CyberPsychology & Behavior
4, 6 (2001), 681–686.
[34] Yi-Hao Peng, Ming-Wei Hsi, Paul Taele, Ting-Yu Lin, Po-En Lai, Leon Hsu,
Tzu-chuan Chen, Te-Yen Wu, Yu-An Chen, Hsien-Hui Tang, and Mike Y. Chen.
2018. SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-
of-Hearing People in Group Conversations. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems (Montreal QC, Canada)
(CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–10.
https://doi.org/10.1145/3173574.3173867
[35] Sandra Poeschl, Konstantin Wall, and Nicola Doering. 2013. Integration of spatial
sound in immersive virtual environments an experimental study on efects of
spatial sound on presence. In 2013 IEEE Virtual Reality (VR). 129–130. https:
//doi.org/10.1109/VR.2013.6549396
[36] Sylvia Rothe, Kim Tran, and Heinrich Hussmann. 2018. Positioning of Subtitles
in Cinematic Virtual Reality. https://doi.org/10.2312/egve.20181307
[37] Zhanna Sarsenbayeva, Niels van Berkel, Chu Luo, Vassilis Kostakos, and Jorge
Goncalves. 2017. Challenges of Situational Impairments during Interaction with
Mobile Devices. In Proceedings of the 29th Australian Conference on Computer-
Human Interaction (Brisbane, Queensland, Australia) (OZCHI ’17). Association
for Computing Machinery, New York, NY, USA, 477–481. https://doi.org/10.
1145/3152771.3156161
[38] Stefania Serafn and Giovanni Serafn. 2004. Sound Design to Enhance Presence
in Photorealistic Virtual Reality. In ICAD.
[39] Lichao Shen, MHD Yamen Saraiji, Kai Kunze, Kouta Minamizawa, and
Roshan Lalintha Peiris. 2020. Visuomotor Infuence of Attached Robotic Neck
Augmentation. In Symposium on Spatial User Interaction (Virtual Event, Canada)
(SUI ’20). Association for Computing Machinery, New York, NY, USA, Article 14,
10 pages. https://doi.org/10.1145/3385959.3418460
[40] Alexa F. Siu, Mike Sinclair, Robert Kovacs, Eyal Ofek, Christian Holz, and Edward
Cutrell. 2020. Virtual Reality Without Vision: A Haptic and Auditory White Cane
to Navigate Complex Virtual Worlds. Association for Computing Machinery, New
York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376353
[41] Sean M. Smith and Glen N. Williams. 1997. A Visualization of Music. In Proceed-
ings of the 8th Conference on Visualization ’97 (Phoenix, Arizona, USA) (VIS ’97).
IEEE Computer Society Press, Washington, DC, USA, 499–f.
[42] Axel Stockburger. 2003. The game environment from an auditive perspective.
Level Up (2003), 4–6.
[43] Valday Team. 2016. Vintage 50s Radio and Phone | 3D Electronics | Unity Asset
Store. Retrieved April 13, 2022 from https://assetstore.unity.com/packages/3d/
props/electronics/vintage-50s-radio-and-phone-71657
[44] Mauro Teóflo, Alvaro Lourenço, Juliana Postal, and Vicente Lucena Jr. 2018.
Exploring Virtual Reality to Enable Deaf or Hard of Hearing Accessibility in Live
SoundVizVR: Accessible VR Sounds for DHH Users with Sound Indicators
Theaters: A Case Study. 132–148. https://doi.org/10.1007/978-3-319-92052-8_11
[45] vUv. 2021. Speakers PBR | 3D Electronics | Unity Asset Store. Retrieved April 13,
2022 from https://assetstore.unity.com/packages/3d/props/electronics/speakers-
pbr-111606
[46] Ryan Wedof, Lindsay Ball, Amelia Wang, Yi Xuan Khoo, Lauren Lieberman,
and Kyle Rector. 2019. Virtual Showdown: An Accessible Virtual Reality Game
with Scafolds for Youth with Visual Impairments. In Proceedings of the 2019
CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland
Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–15.
https://doi.org/10.1145/3290605.3300371
[47] Minecraft Wiki. 2022. Subtitles – Minecraft Wiki. Retrieved April 12, 2022 from
https://minecraft.fandom.com/wiki/Subtitles
ASSETS ’22, October 23–26, 2022, Athens, Greece
[48] Bei Yuan, Eelke Folmer, and Frederick C. Harris. 2011. Game Accessibility: A
Survey. Univers. Access Inf. Soc. 10, 1 (mar 2011), 81–100. https://doi.org/10.1007/
s10209-010-0189-5
[49] Veronica Zammitto. 2008. VISUALIZATION TECHNIQUES IN VIDEO GAMES.
https://doi.org/10.14236/ewic/EVA2008.30
[50] Yuhang Zhao, Edward Cutrell, Christian Holz, Meredith Ringel Morris, Eyal
Ofek, and Andrew D. Wilson. 2019. SeeingVR: A Set of Tools to Make Virtual
Reality More Accessible to People with Low Vision. In Proceedings of the 2019
CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland
Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14.
https://doi.org/10.1145/3290605.3300341
 Uncovering Visually Impaired Gamers’ Preferences for Spatial
Awareness Tools Within Video Games
Vishnu Nair Shao-en Ma Ricardo E. Gonzalez Penuela
Columbia University Columbia University Cornell University, Cornell Tech
New York, New York, USA New York, New York, USA New York, New York, USA

Yicheng He Karen Lin Mason Hayes

Columbia University Columbia University Rochester Institute of Technology
New York, New York, USA New York, New York, USA Rochester, New York, USA

Hannah Huddleston Matthew Donnelly Brian A. Smith

Stanford University Bowdoin College Columbia University
Stanford, California, USA Brunswick, Maine, USA New York, New York, USA

Figure 1: Illustrations of the four spatial awareness tools we implemented within Dungeon Escape. These approaches — the
smartphone map, the whole-room shockwave, the directional scanner, and the simple audio menu — represent a broad range
of designs for facilitating spatial awareness for VIPs. However, we still do not yet understand what the relative merits and
limitations of each approach are.
ABSTRACT on SATs to gain spatial awareness, especially in complex envi-
Sighted players gain spatial awareness within video games through ronments where using rich ambient sound design alone may be
sight and spatial awareness tools (SATs) such as minimaps. Vi- insufcient. Researchers have developed many SATs for facilitating
sually impaired players (VIPs), however, must often rely heavily spatial awareness within VIPs. Yet this abundance disguises a gap
in our understanding about how exactly these approaches assist
VIPs in gaining spatial awareness and what their relative merits
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed and limitations are. To address this, we investigate four leading
for proft or commercial advantage and that copies bear this notice and the full citation approaches to facilitating spatial awareness for VIPs within a 3D
on the frst page. Copyrights for components of this work owned by others than ACM video game context. Our fndings uncover new insights into SATs
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a for VIPs within video games, including that VIPs value position
fee. Request permissions from permissions@acm.org. and orientation information the most from an SAT; that none of
ASSETS ’22, October 23–26, 2022, Athens, Greece the approaches we investigated convey position and orientation
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00 efectively; and that VIPs highly value the ability to customize SATs.
https://doi.org/10.1145/3517428.3544802
ASSETS ’22, October 23–26, 2022, Athens, Greece
CCS CONCEPTS
• Human-centered computing → Auditory feedback; Accessi-
bility technologies; Accessibility systems and tools.
KEYWORDS
Audio navigation tools; spatial awareness tools; blind-accessible
games; visual impairments
ACM Reference Format:
Vishnu Nair, Shao-en Ma, Ricardo E. Gonzalez Penuela, Yicheng He, Karen
Lin, Mason Hayes, Hannah Huddleston, Matthew Donnelly, and Brian
A. Smith. 2022. Uncovering Visually Impaired Gamers’ Preferences for
Spatial Awareness Tools Within Video Games. In The 24th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22),
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 16 pages.
https://doi.org/10.1145/3517428.3544802
1 INTRODUCTION
Mainstream 3D video games are largely inaccessible to visually-
impaired players (VIPs) because they often lack crucial accessi-
bility tools [5, 46]. Although some recent mainstream games [16]
have made strides in making certain in-game abilities accessible
to VIPs, many crucial abilities remain inaccessible. Among these is
the ability for players to gain spatial awareness of their surround-
ings, which prior work has established is crucial for granting VIPs
with an enhanced sense of space and presence within the game
world [2, 4].
Sighted players gain spatial awareness using a combination of
vision and auxiliary tools such as minimaps and world maps [7, 13,
14, 58, 67], which we will refer to as spatial awareness tools (SATs).
VIPs, however, are not able to beneft from vision in the same way
as sighted players. Although rich environmental sound design can
ensure some level of accessibility and spatial awareness for VIPs,
these elements can prove to be insufcient, especially in complex
virtual environments. As such, SATs are often an indispensable
view of the game world for VIPs, having an immense impact on
their experience within a game.
Researchers have developed several types of SATs which repre-
sent very diferent approaches for facilitating a representation of the
game world for VIPs. These include touchscreen maps, shockwave-
like systems, directional scanners, and audio-based menus. This
relative abundance, however, disguises a gap in our understanding
about how exactly these approaches assist VIPs in gaining spatial
awareness and what the relative merits and limitations of each
approach are. It is not yet clear to developers, for example, whether
it is best to use a shockwave-like system like The Last of Us Part 2
did [16] or audio-based menus like Terraformers did [45, 63], and
blind gamers are ultimately the ones who sufer. As such, we take
a step back and ask two important research questions:
RQ1: What aspects of spatial awareness do VIPs fnd important
within games?
RQ2: How well do today’s difering SAT approaches facilitate
each aspect of spatial awareness, and why?
In this work, we investigate RQ1 and RQ2 by implementing four
leading approaches to facilitating spatial awareness for VIPs and
investigating their merits and limitations. The four tools, illustrated
in Figure 1, are a smartphone map, a whole-room shockwave, a
Nair et al.
directional scanner, and a simple audio menu of points-of-interest.
Together, they represent a broad range of design choices, including
touchscreen-based vs. game controller interaction and "all-at-once"
(collective) overviews vs. pointer-based scanning.
In order to investigate these research questions, we conducted
a user study in which nine visually impaired participants played
multiple levels of a 3D adventure video game using our tools.
For RQ1, we evaluated how important VIPs consider six dif-
ferent types of spatial awareness — that have been found to be
important in physical world settings [20, 22, 24, 29, 31, 34, 49, 65]
— to be in comparison to each other within a video game context.
Section 2.1 identifes the six types. We observed that participants
considered position and orientation to be the most important type
of spatial awareness and that they considered the scale and shape
of an area to be the least important aspects of spatial awareness.
These particular fndings refect existing work within the physical
world that highlights the importance of position and orientation for
VIPs [24, 31, 49], but difer from existing work within the physical
world that has also found knowledge of area scale and shape to be
an important component of better understanding an area, especially
when freely exploring it [4, 30].
With respect to RQ2, we observed that each tool had its own
strength for VIPs: The directional scanner communicated the ar-
rangement of items very well; the simple audio menu communicated
the presence of items very well; the smartphone map communicated
the shape of an area very well; and the whole-room shockwave
communicated the scale of the area well. Importantly, however, we
also discovered signifcant defciencies in today’s spatial awareness
tools. No tool excelled across the board, and in particular, none of
the tools communicated position and orientation very well despite
our fnding from RQ1 that position and orientation is the most
important type of spatial awareness to VIPs within games. Further-
more, we found issues with the tools that infuenced how efectively
they communicated spatial awareness to players, including that
some of the tools provided too much information.
Together, our fndings from RQ1 and RQ2 reveal important de-
sign implications for future spatial awareness tools for VIPs within
video games, and we present these in our Discussion. We also dis-
cuss the potential for developing purpose-built hardware for spatial
awareness and how our fndings within virtual worlds can inspire
further research in physical world navigation and exploration.
2 BACKGROUND AND RELATED WORK
Our work is built upon a rich history of prior work on facilitating
spatial awareness for VIPs, both within the physical world and
within video games.
We begin this section by explaining what we mean by “spatial
awareness” — in particular, by reviewing aspects of spatial aware-
ness that are known to be important to VIPs within physical world
contexts (Section 2.1). RQ1 will investigate the relative importance
of these aspects for VIPs within games. We then review existing
techniques for facilitating spatial awareness for VIPs, both in the
physical world and within video game environments — and review
the tradeofs inherent within the design of these tools (Section 2.2).
Through RQ2, we take a step back and investigate the relative
merits and limitations of these approaches.
Uncovering Visually Impaired Gamers’ Preferences for SATs Within Video Games
2.1 What do we mean by “spatial awareness”?
Spatial awareness, as used in this work, refers to a user’s awareness
of their surrounding environment and of their own state within the
environment [35, 64]. Past literature within physical world contexts
has shown spatial awareness to be multifaceted. Thus, in this work,
we investigate RQ1 and RQ2 with respect to six distinct aspects of
spatial awareness we identifed through prior work. Specifcally, we
looked through existing research in cognitive map formation and
spatial awareness for VIPs within the physical world and looked
for explicit information on what aspects of spatial awareness are
most important to VIPs. We chose to investigate the following six
types of spatial awareness since they were mentioned as important
across a breadth of prior research [20, 22, 24, 29, 31, 34, 49, 65]:
Types 1 & 2: Scale and shape of the area. Prior work —
mainly in tactile maps [30, 49] and echolocation [4] — has
found area shape to be important to VIPs in obtaining a
general impression of the area, which can be especially cru-
cial when exploring and trying to learn about the environment.
Type 3: Position & orientation. Researchers have found that
understanding where one is within a mental map of the area
(for example, their Cartesian coordinates or their heading
direction in degrees) — that is, within an allocentric [34],
map-like mental representation of the environment — is vital
to continuously updating their own current state within the
environment and thus efectively move through it [20, 22, 34].
Yet, prior work in physical world contexts [24] has shown that
obtaining this understanding is especially demanding for VIPs.
Types 4 & 5: Presence and arrangement of items.
Researchers have emphasized that providing VIPs with the
information necessary to perceive the locations of objects can
allow them to infer spatial relationships between objects and
can lead to increased spatial awareness and more accurate
cognitive maps [24, 29].
Type 6: Areas adjacent to the player’s current area. Prior
work with physical world tactile maps [49] and mobile-based
spatial tactile feedback for communicating geographical
information [65] have underscored the importance of
understanding the global structure of the world — general
overviews of an area and spatial relationships between
multiple areas — for VIPs, which can help them plan out
routes and backtrack as needed.
Although prior work has determined these six aspects of spatial
awareness to be important to VIPs in the physical world, video
games are very diferent from the physical world. Within the phys-
ical world, practicality and physical safety are extremely important
factors [6], while in video games, agency and pleasure (fun) are
very important, and VIPs’ in-game “safety” may not always be a
major concern. It is possible that, due to these diferences, VIPs may
fnd certain aspects of spatial awareness more or less important
within games when compared to the physical world. We, thus, use
RQ1 to explore these preferences.
ASSETS ’22, October 23–26, 2022, Athens, Greece
2.2 How do games supplement spatial
awareness?
Games made for VIPs often use ambient signals to provide implicit
spatial awareness to players. These ambient signals usually take the
form of environmental audio cues that communicate information
about the player’s immediate environment. For example, hearing
running water may indicate that there is a waterfall or stream near
the player. When environmental sounds reverberate, the player
may realize that they are inside a cave or tunnel, and the extent of
the reverberation can indicate the size of the cave or tunnel. The use
of 3D sound can additionally communicate the relative direction
that the source of sound is in with respect to the player.
Although ambient signals may be sufcient for simple envi-
ronments, they can become less useful to players as environments
become more complex, as is typical for many mainstream 3D games.
Ambient cues can become overwhelming when there are too many
items in the environment, and they may also be vague, giving
players little information about what the sounds they are hearing
actually represent. As a result, using ambient signals alone as a
means to facilitate spatial awareness for players limits the com-
plexity of games that accessible game designers are able to make.
Accessible game designers, thus, face a tradeof between designing
environments that are interesting and designing games that are
still accessible and playable by VIPs [2, 52].
Given the limitations of implicit forms of spatial awareness, ac-
cessible game designers often turn to creating tools that explicitly
communicate spatial awareness information to players. These spa-
tial awareness tools (or SATs) — which include (but are not limited
to) tactile maps, radar systems, and grid systems — supplement
implicit spatial awareness cues by clarifying environmental infor-
mation and afording players greater control over what information
they hear and when they hear it.
Table 1 shows an overview of SATs from prior work. Below we
review some explicit approaches for facilitating spatial awareness
in games and in the physical world.
2.2.1 Facilitating spatial awareness for VIPs within video games.
Tools that explicitly communicate spatial awareness information to
VIPs are not commonplace within mainstream video games. Most
examples, instead, come from “audio games” (audio-based games
created for VIPs), which generally provide players with spatial
awareness by presenting environments in the form of lists and
grids that players can query. This technique is employed by many
well-known audiogames, including Terraformers [45, 63], A Hero’s
Call [44], and ShadowRine [39]. These representations may com-
municate the presence and arrangement (Types 4 & 5 from Section
2.1) of items and points-of-interest and are sometimes further sup-
plemented by additional tools such as radars and compasses.
Several examples of SATs have come from the research commu-
nity as well. A notable example is NavStick [41, 42], which repur-
poses a game controller’s right thumbstick to allow VIPs to “look
around” their in-game surroundings via line-of-sight. A directional
scanning system like NavStick could allow VIPs to determine the
presence and spatial arrangement of objects around them (Types
4 & 5) as well as their relative position and orientation within the
game world (Type 3).
ASSETS ’22, October 23–26, 2022, Athens, Greece Nair et al.

Artifact Ambient spatial awareness cues Explicit spatial awareness tools (SATs) SAT(s) in our study
ShadowRine Audio cues in environment.
Tactile display showing top-down view of player’s location. Smartphone map
(virtual) [39] (e.g., enemy & object sounds)
The Last of Us Part 2 Audio cues in environment.
"Enhanced listen mode" (shockwave-like tool). Whole-room shockwave
(virtual) [16] (with “audio cue glossary”)
NavStick Audio cues in environment.
"NavStick" (direction-based object scanner). Directional scanner
(virtual) [41, 42] (e.g., enemy & checkpoint sounds)
A Hero’s Call
Audio cues in environment. Menu of nearby points of interest. Simple audio menu
(virtual) [44]
Swamp Audio cues in environment. “Radar” (beeps based upon empty space or solid walls). Whole-room shockwave
(virtual) [32] (e.g., zombie growls) Audio menu (with compass- & tile-based guidance). Simple audio menu
Terraformers “Sonar” (provides distance to object in current facing direction). Directional scanner
Audio cues in environment.
(virtual) [45, 63] “GPS” (audio-based menu of nearby objects & positions). Simple audio menu
SmartTactMaps
Sounds from physical environment. Smartphone-based augmentation of physical tactile map. Smartphone map
(physical) [28]
Timbremap
Sounds from physical environment. Touchscreen-based 2D map exploration. Smartphone map
(physical) [56]
Echolocation
Sounds from physical environment. Behavior of refected sounds within the environment. Whole-room shockwave
(physical) [33, 43, 57]
Talking Points 3
Sounds from physical environment. “Directional Finder” (direction-based landmark scanner). Directional scanner
(physical) [64]
MS Soundscape Selection of points of interest from a menu.
Sounds from physical environment. Simple audio menu
(physical) [40] Notifcations about nearby landmarks using 3D sound.
SpaceSense Vibration cues indicating the direction of a location
Sounds from physical environment. Simple audio menu
(physical) [65] selected from a menu.
NavCog3
Sounds from physical environment. Audio notifcations of immediate surroundings. N/A
(physical) [50]
Table 1: An overview of prior work in communicating spatial awareness to VIPs within both virtual and physical contexts.
These artifacts represent a variety of ideas ranging from audio-based solutions to tactile solutions. For each, we present the
ambient/implicit spatial awareness cues that it provides, the explicit spatial awareness tools (SATs) it introduces, and the
corresponding SAT(s) in our study. The four SATs we implement collectively represent a signifcant portion of prior work.

A notable exception to the lack of SATs in mainstream games
is The Last Of Us Part 2, a 3D action-adventure game released in
2020 [16], which introduced an "enhanced listen mode" for VIPs.
The enhanced listen mode provides spatial awareness to players by
placing 3D audio beacons at the locations of nearby enemies and
other points-of-interest on the press of a button. The beacons may
give players a sense of the spatial arrangement of items in the area
(Type 5) as well as a sense of the surrounding area’s scale (Type 1).
2.2.2 Facilitating spatial awareness for VIPs in the physical world.
Some audio-based tools within the physical world have features
that explicitly provide VIPs with spatial awareness information and
can thus inform the design of SATs for game worlds. NavCog3 [50],
a turn-by-turn indoor navigation system for VIPs, for example,
emits notifcations about nearby landmarks and points-of-interest
to promote awareness in the user of their presence (Type 4). Simi-
larly, Microsoft Soundscape [40], an audio-based wayfnding system
that can be used by VIPs, uses 3D sound to communicate the pres-
ence and relative direction (i.e., arrangement, Type 5) of nearby
landmarks. The spatial awareness that these systems provide is,
however, limited. For example, they do not provide any information
about the area’s shape and size (Types 1 & 2).
Tactile-based systems provide spatial awareness by providing
overviews of areas [30, 49], which may include the scale and shape
of an area (Types 1 & 2), the presence and arrangement of landmarks
and other points-of-interest (Types 4 & 5), and even what areas
may be adjacent to a given area (Type 6). These not only include
physical tactile maps but also mobile-based tactile systems, such as
Timbremap [56] and SmartTactMaps [28], which can allow VIPs to
survey the area they are in using a commodity smartphone.
Echolocation, which has been explored for both physical [33, 43,
57] and virtual [2, 3] environments, is another technique that VIPs
may use to gain spatial awareness within environments. Using the
acoustic properties of the environment can allow individuals to
learn about the structure of the area they are in, including the scale
and shape of the area (Types 1 & 2), as well as the presence and
arrangement of nearby objects (Types 4 & 5) [33, 57].
3 FOUR SPATIAL AWARENESS TOOLS
In order to investigate RQ1 and RQ2, we implemented four existing
approaches for giving VIPs spatial awareness of their surroundings
that represent a wide range of possible designs. Figure 1 depicts
the four approaches. They include a smartphone map, a whole-
room shockwave, a directional scanner, and a simple audio menu.
We limited our exploration to just four tools to avoid fatiguing
our user study participants while still efectively evaluating the
tools. Regardless, Table 1 shows that these four tools collectively
represent a signifcant portion of approaches from prior work on
explicitly communicating spatial awareness information to VIPs.
We were able to replicate two of the tools (the directional scanner
and the simple audio menu) concretely from existing work [41, 42,
Uncovering Visually Impaired Gamers’ Preferences for SATs Within Video Games
45, 63]. For the smartphone map and the whole-room shockwave,
however, we went through multiple design iterations because their
implementation included many open design decisions that were
not fully specifed by prior work. In our design iterations, which we
describe in Sections 3.1 and 3.2, our focus was to polish the tools’
designs and ensure that they showcased the potential of the two
SAT approaches in the best possible way so that our results would
not be confounded by a potentially bad design.
In order to ensure that our tools most accurately represented
current approaches and to ensure that our study procedure was
sound, we conducted pilot tests with two visually impaired and
eight sighted-but-blindfolded people. We intended for the testing
phase that included the sighted-but-blindfolded participants to be
a naïve-yet-useful way of catching any low-hanging fruit with
respect to procedural, game-related, or tool-related issues before
piloting with our visually impaired team members. Our visually
impaired team members — whom we hosted as part of the research
team during the project — then provided feedback that was critical
to developing the fnal designs of the tools.
In the following subsections, we describe the design and imple-
mentation process of the four SATs. We direct readers to the accom-
panying video fgure for a demonstration of all four tools. We cre-
ated all four tools using the Unity game engine (v2020.3.16f1) [59].
3.1 Smartphone Map
The smartphone map interface, shown in the upper-left corner of
Figure 1, uses a smartphone-based touchscreen map that works in
tandem with the game. The player can use their fnger to survey
the map. As the player moves through the level, the map will auto-
matically pan and rotate in real-time, allowing users to explicitly
keep track of their own position and orientation, respectively.
The smartphone map interface represents prior work in tactile-
based maps to support spatial awareness. Tactile maps in the phys-
ical world have been shown to support spatial awareness in VIPs
by providing general overviews of spaces and landmarks [30, 49].
Our work with video games necessitates a digital solution; as such,
the smartphone map interface we implemented also derives from
prior work in touchscreen-based accessible graphics, particularly
in presenting foor plans and other maps to VIPs [23, 25, 26, 28, 56].
When a player places their fnger on the screen, they will begin
surveying at their position, regardless of where on the screen they
are touching. As they move their fnger, they will survey the map
relative to their position, with the app announcing anything that
the player touches. The app will announce all items in the world
(as well as the player’s position) using sound efects and/or text-
to-speech. The app only reacts to touches within the current room
that the player is in — if the player drags their fnger outside the
room, a continuous warning tone will play.
In the frst version of this tool, players started surveying at the
portion of the map where their fnger touched the screen; however,
our visually impaired pilot participants ended up spending large
amounts of time searching for their current position, which frus-
trated them. As a result, our second and fnal version registers a
player’s initial touch at their current position.
ASSETS ’22, October 23–26, 2022, Athens, Greece
3.2 Whole-Room Shockwave
The whole-room shockwave, depicted in the upper-right corner
of Figure 1, uses an acoustic shockwave that the player triggers
to communicate information about their surroundings. When the
shockwave hits anything in the room, an announcement and/or
sound efect emanates from that object via 3D sound. The shock-
wave corresponds to real-world physics in that closer objects will
emanate their sounds back to the player before objects that are
further away. If the player moves while the shockwave is active,
the rate of expansion will match the player’s speed.
The whole-room shockwave originated from our explorations in
echolocation, which has been shown to promote spatial awareness
in VIPs by communicating physical properties of the room and
nearby objects [33, 43, 57]. Our initial echolocation prototype had
players press a button on their game controller to emit a click sound
originating from the player’s position, similar to how some VIPs use
echolocation within the physical world [33, 36]. Our echolocation
prototype was similar to virtual echolocation techniques used in
prior work [2] that used Steam Audio’s built-in head-related transfer
function [62] to generate sound refections based on the physical
structure of each room.
In our pilot tests, however, we found that echolocation by itself
was not equivalent to the other tools. While echolocation commu-
nicates the raw layout of an area, the other tools can communicate
raw layout in addition to specifc object information through sound
efects and text-to-speech. Furthermore, our visually impaired pilot
participants were not at all experienced in echolocation and did
not know how to decode and interpret the sound echoes in our
game environment; they could only interpret broad qualities of the
area such as how large it was. Although users could learn to use
echolocation, prior work has indicated that it may take weeks for
users to learn how to use click-based echolocation efectively [43].
We, thus, made modifcations to the initial echolocation design
and created the whole-room shockwave — a refned and more com-
prehensible version of echolocation. In its frst iteration, the shock-
wave announced every item that it hit, which proved to be auditorily
overwhelming. Furthermore, both visually impaired participants
found the shockwave to be too fast. As a result, our second and
fnal version halved the speed of the shockwave and implemented
a fltering mechanism. Players can press the right button on the
D-pad to cycle through four fltering options — all objects, mission-
critical points-of-interest, non-mission-critical (decorative) objects,
and walls only. Only items within the selected category will emit
sounds during a shockwave.
3.3 Directional Scanner
The directional scanner, illustrated in the lower-left corner of Figure
1, allows players to survey in any direction using the right thumb-
stick. Players use the tool by tilting the thumbstick in any direction.
This triggers an announcement naming the frst object that lies in
that direction via line-of-sight with respect to the player’s current
position and orientation. The announcement is made via 3D sound
from the point of the object in space. If the frst object in a direction
being pointed at is not an object of interest (i.e., a wall or other
generic obstruction), the scanner will emit a 440 Hz sine tone from
the direction of the obstruction.
ASSETS ’22, October 23–26, 2022, Athens, Greece
This tool represents prior work that has sought to replicate
the act of “looking around” (or directionally scanning an area) to
promote spatial awareness for VIPs. We take particular inspiration
from NavStick [41, 42], which introduced the concept of directional
scanning within game worlds and showed how VIPs enjoyed the
ability to survey their game environments directly by “looking
around.” Some prior work with directional scanning also exists in
the physical world. Talking Points 3 [64] is one such example: It
features a “Directional Finder” that provides a list of landmarks that
lie in the general direction that a VIP points their mobile device.
Our implementation of the directional scanner was derived from
NavStick, and we did not implement any major changes to it as a
result of our pilot tests.
3.4 Simple Audio Menu
The simple audio menu, shown in the lower-right corner of Figure
1, represents the idea of using a list to promote spatial awareness
— in particular, by allowing VIPs to learn about the contents of
the area they are currently in. Many audio games made for VIPs,
such as Terraformers [45, 63] and A Hero’s Call [44], use list- and
grid-based representations to present the world to VIPs.
The simple audio menu we implemented exposes an audio-based
list of points-of-interest (POIs). Players use the tool by pressing
the left bumper button to open a list of POIs within the room
they are currently in. As the player scrolls through the list using
the D-pad, they will hear each item’s associated sound efect and
text-to-speech announcement. The simple audio menu is modeled
after list interfaces used in some audio games as well as prior
research [41, 45, 63] in that it employs an alphabetical ordering
of items. Previous research has suggested that, for a linear menu,
a stable alphabetical ordering is less confusing than a proximity-
based or direction-based ordering, both of which can change as the
player moves [41].
Similar to the directional scanner, we did not implement any
major changes to the simple audio menu as a result of our pilots.
4 USER STUDY
We performed a user study to investigate two important research
questions about SATs within video games for VIPs:
RQ1: What aspects of spatial awareness do VIPs fnd important
within games?
RQ2: How well do today’s difering SAT approaches — as repre-
sented by the four tools we implemented — facilitate each
aspect of spatial awareness, and why?
We created a 3D adventure game called Dungeon Escape to in-
vestigate these two research questions. We included the four tools
within Dungeon Escape and used the game to run a user study with
VIPs. In this section, we describe Dungeon Escape and our user
study.
4.1 Game: Dungeon Escape
Dungeon Escape is a 3D third-person adventure game set in a fan-
tasy world, in which the player must escape small dungeons by
fnding objects that allow them to clear obstacles. We chose to
create Dungeon Escape to address RQ1 and RQ2 because the game
requires players to use the tools they are given to search for and
Nair et al.
Figure 2: Overhead views of Dungeon Escape’s trial level and
four main levels. All levels have a common set of points-of-
interest: a start point, a key, an obstacle that the key afords
passage through, and a goal checkpoint. Each level, however,
possesses a unique layout, allowing us to evaluate the four
spatial awareness tools within a variety of layouts.
understand where objects are located and how the rooms in each
level are laid out in order to succeed — thus testing how well they
are able to gain spatial awareness using those tools.
We created Dungeon Escape using the Unity game engine [59],
and we designed the dungeon’s layout using the Dungeon Architect
Unity asset [12]. Figures 1 and 3 show views from Dungeon Escape.
Dungeon Escape consists of four levels (small dungeons), which
allowed us to study the four SATs within separate dungeon layouts.
Figure 2 shows overhead views of all four main levels and the
trial level. In each main level, the player must reach a goal area by
gaining passage through an obstacle: either a locked door, a cracked
wooden door, a spider web, or a dog blocking the exit. To do so, the
player must fnd a relevant object in another room: a key, an axe, a
burning torch, or a bone, respectively. Each level consists of several
rooms scattered with decorative objects such as crates and barrels.
We generated the four level layouts by deriving them from a
single Dungeon Architect “grid fow.” This grid fow defned basic
parameters from which Dungeon Architect would generate levels.
Each level consisted of:
• A “start room” within which the player frst spawns.
• An “obstacle room” containing the obstacle.
• A “key room” containing the object that clears the obstacle.
• A “main hall” connecting the start, key, and obstacle rooms.
• A “fnal hall” containing the goal checkpoint.
We then fed random seed values into this grid fow to generate
the fnal layouts. This allowed us to have unique level layouts while
keeping them equivalent in terms of difculty and structure. The
trial level followed a similar conceptual structure but was much
smaller, consisting of a start room, a combined key-and-obstacle
room, and a fnal hall with the checkpoint.
Players move the main character with the left thumbstick. Tilting
it forward and backward will move the character forward and
backward. Tilting it left and right will rotate the character left and
right. This control scheme refects controls found in mainstream
3D games such as Tomb Raider [15], Resident Evil 1-5 [1, 8–11],
Metroid Prime 1-3 [53–55], Heavy Rain [17], and Silent Hill [51],
which use a fxed over-the-shoulder camera and use left/right on
the left thumbstick to rotate the character. The right thumbstick
Uncovering Visually Impaired Gamers’ Preferences for SATs Within Video Games
is used by the directional scanner condition; thus, to eliminate a
confound, we removed right thumbstick controls from all other
conditions. Players can press the bottom face button to pick up an
object or to use an object to remove the relevant obstacle.
Players hear a scraping sound if they physically hit an obstruc-
tion; the sound will be situated in the direction of contact. Keys,
obstacles, and checkpoints play a relevant sound once the player
is within two meters of the object. Players will hear the name of
the room (for example, “Start Room” or “Key Room”) announced
on entry, and they can also press the right face button to hear the
room name on-demand. We integrated these sounds to allow VIPs
to be informed of these events — i.e., hitting a wall or entering a
room — when they occur. Sighted players can perceive these events
solely via sight, but VIPs require notifcations via other means.
We also implemented a “rotation indicator” utility that helps
players understand how much they are rotating when they turn
left or right using the left thumbstick. As the player rotates, a click
sound will be played at 15° increments via 3D sound only in the
direction of the player’s objective (i.e., the obstacle that must be
cleared). The rotation indicator mimics snap rotation controls found
in many games created for VIPs [32, 45, 63], which allow players
to snap to pre-defned angle increments. In order to bring Dun-
geon Escape’s controls closer to the free movement of mainstream
3D games, we gave players full analog control via the left thumb-
stick but maintained the feedback aforded by snap rotation via the
rotation indicator. The rotation indicator was available across all
four tools and pointed in the direction of the objective regardless
of any intervening obstacles. Similar utilities have also been im-
plemented in prior work that has investigated navigation by VIPs
within virtual environments [2, 4, 41].
Additionally, players could place looping audio beacons on ob-
jects of interest so that they could lock onto and keep targets within
their “feld of view.” Once placed, these beacons emit a looping
sound, which players can use to orient themselves and move to-
wards the target. With NavStick, players point at a target with the
right stick and press the right bumper button to place the beacon.
With the simple audio menu, players scroll to a target and press the
left bumper. With the smartphone map, players tap on the upper
one-ffth of the screen to place a beacon at the last announced
target. There was no mechanism for beacon placement with the
whole-room shockwave. We added the beacons exclusively for guid-
ance purposes to speed up the process of walking toward a target
— players still need to use an SAT to fnd objects and other targets
before they can place a beacon at that object/target.
4.2 Participants
We recruited nine participants for this study. In our pre-study ques-
tionnaire, eight described themselves as being completely blind and
one (P1) described themselves as having light perception only. All
participants were male and have had their vision impairments from
birth. Six participants were 18–25 years old; two (P5 & P9) were
26–35 years old; and one (P3) was 36–45 years old. In addition to
having vision impairments, two participants (P3 & P6) reported
having slight hearing loss in one of their ears.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 3: Remote study session with a participant and two
facilitators. The participant is currently sharing their screen.
Within the game, a blue key is situated to the participant’s
right. The participant will need to collect that key to progress
through the level. (Faces obscured to protect anonymity.)
We recruited participants through posts on the AudioGames.net
Forum,1 an online discussion board that centers around audio-
based games and is frequented by VIPs. Six of our participants
reported themselves as being very experienced with video and
other electronic games (4+ on a 5-point Likert scale), while the
other three (P2, P6, & P8) reported themselves as being moderately
experienced with games (3 on a 5-point Likert scale).
4.3 Technical Setup & COVID-19 Challenges
We performed this study remotely due to the COVID-19 pandemic
and the difculties that VIPs may face in travelling to our institution.
We sent each participant an executable of our game for them to
download to their computer before their study appointment. The
game included all of the SATs except for the smartphone map. We
distributed that tool as both iOS and Android apps using the Google
Firebase App Distribution service [27]. We designed both Dungeon
Escape and the smartphone map to connect with a cloud backend,
which allowed both components to synchronize with each other,
and allowed us to remotely observe and control the runtime state
of participants’ games using a custom-built control panel.
We held the study appointments over Zoom and asked partici-
pants to share their computer audio (and, optionally, video) with
us. Although there was no way for us to see the smartphone map
during the study, most participants’ microphones picked up the
sound from the app. Figure 3 shows a study session in progress.
The study and our data collection eforts were approved by the
Columbia University Institutional Review Board (IRB).
4.4 Procedure
To address RQ1, we began the session by administering a two-
part pre-study questionnaire. The frst part requested demographic
information alongside information about participants’ existing ex-
perience with video games and physical world navigation. The
second part directly asked participants about how important they
fnd each of the six types of spatial awareness — that we identifed in
Section 2.1 — within a video game context. For each type, responses
were given on a 5-point unipolar Likert scale where 1 indicated
1 https://forum.audiogames.net/
ASSETS ’22, October 23–26, 2022, Athens, Greece Nair et al.

that the type of spatial awareness was not-at-all important and 5

indicated that it was extremely important. Afterwards, we placed
participants in a room within the game where we introduced basic
movement and interaction controls.
For each tool, we frst placed participants into the trial level. We
explained how to use the tool and afterwards allowed participants
to traverse the trial level at their own leisure. The trial level was the
same across all tools. After the trial level, we placed participants
into one of the four main levels. Although all participants played
the four levels in the same order, we counterbalanced the order
of the tools themselves via a Latin square design to reduce any
variations caused by order efects. Figure 4: The importance of the six aspects of spatial aware-
In order to address RQ2, we administered a two-part post-level ness within games for VIPs (RQ1). Responses were given on
questionnaire; we did this after participants traversed a level with a fve-point unipolar Likert scale. Red lines indicate median
a tool. In the frst part, we asked participants to elaborate on their ratings within this box plot. Rankings based on median rat-
impressions of the tool, what they think is missing, and in what ings are shown to the right.
game situations they might use the tool. In the second part, we
gauged how well participants thought the tool satisfed each of the
six types of spatial awareness. Responses were given on fve-point
scales, where 1 indicated that the tool facilitated that type not-at-all Then, all fve coders iterated on the codes together until there was
well and 5 indicated that it facilitated that type extremely well. unanimous agreement that they could not iterate further.
Participants were encouraged to elaborate on all questions.
After completing all four levels, we administered a two-part 5 RQ1 RESULTS: ASPECTS OF SPATIAL
post-study questionnaire. In the frst part, we asked participants AWARENESS IMPORTANT TO VISUALLY
to consider a scenario where they were able to play the levels in IMPAIRED PLAYERS
Dungeon Escape using more than one tool at once; we did this
In this section, we report our fndings regarding our frst research
in order to determine if using multiple tools at once could have
question (RQ1): learning what aspects of spatial awareness VIPs
improved participants’ spatial awareness in any way. As part of this
fnd important within games. We captured these opinions in the
section, participants were asked to provide two combinations of
second questionnaire that we administered as part of our pre-study
two tools each that they would have liked to use if they were given
procedure.
the chance to do so. (We should note that Dungeon Escape is capable
Figure 4 shows box plots of participants’ importance ratings for
of activating two tools at once; however, in our initial pilot tests,
the six types of spatial awareness. Note that the median rating for
including additional game levels to test these combinations made
each measure is highlighted in red in each type’s box plot.
study sessions well exceed our limit of two hours.) In the second
The data suggest that the six types of spatial awareness can be
part, we asked participants how likely they were to recommend
divided into three levels of importance for VIPs. The most important
each individual tool to a friend or colleague, assuming they had the
aspect of spatial awareness to VIPs within games is position and ori-
same visual impairments as the participant. Responses were given
entation (Type 3) awareness, which received a median importance
on a 10-point net promoter score scale [48], where 1 indicated they
rating of 5 on a 5-point unipolar Likert scale. Below that, three
were very unlikely to recommend it and 10 was very likely.
spatial awareness aspects — presence of items (Type 4), arrange-
ment of items (Type 5), and adjacent areas (Type 6) — tied each
4.5 Data Collection & Analysis other with a median importance rating of 4. The least important
aspects of spatial awareness to VIPs within games are scale (Type
We administered all questionnaires by having the facilitator read out
1) and shape (Type 2) awareness, both of which received median
each question and input the participant’s response into an internal
importance ratings of 3.
Google Form. For all choice- and rating-based questions, we asked
The following subsections dive deeper into participants’ rea-
for participants’ open-ended opinions via the questionnaire itself by
soning behind how important they rated each aspect of spatial
explicitly following up on their responses. The facilitator was also
awareness to be within games. All quotes come from participants’
encouraged to follow up on any other points they found interesting
open-ended responses while completing this questionnaire. In Sec-
throughout the session — though they were not allowed to disturb
tion 7, we discuss how these fndings and the fndings from RQ2
the participant while a game level was in progress. We have included
(in Section 6) collectively reveal new design considerations and
the questionnaires as part of our supplementary material.
research opportunities for spatial awareness tools.
We recorded all sessions with participants’ permission for tran-
scription purposes. We also obtained raw data of participants’ ac-
tions within the game by capturing in-game logs. 5.1 Rank 1: Position and Orientation [Type 3]
To analyze sessions, we followed an inductive coding process Participants found position and orientation to be the most impor-
that involved fve members of the research team. Individual coders tant aspect of spatial awareness within a video game context. Six
went through session transcripts and coded quotes and other events. participants explicitly afrmed this aspect of spatial awareness as
Uncovering Visually Impaired Gamers’ Preferences for SATs Within Video Games
the most important because it was crucial to determining their
current state within the game world:
“You have an idea of how fast you’re turning and in
what direction. I would say it’s the most important
thing.” — P3
Another participant who echoed this sentiment, P9, recounted
extensive experience with shooter audio games, such as Swamp [32],
that require players to move through a complex environment. P9
afrmed position & orientation awareness — and thus, awareness
of their current state — as extremely important to helping them
plan out future actions, which is a crucial aspect of shooter-type
games:
“You have to know where you are at and where you are
oriented to in order to know where to go and what to do
next.” — P9
These opinions refect work within the physical world that has
found position and orientation to be important to VIPs [22, 24]. They
also establish that SATs for VIPs within video games must satisfy
a high bar in terms of communicating position and orientation
information. In Section 6, we determine if any of the four tools we
implement for this study satisfy this high bar.
5.2 Rank 2 (three-way tie): Presence,
Arrangement, and Adjacent Areas [Types 4,
5, and 6]
After position and orientation, participants found the next most
important aspects of spatial awareness to be the presence and ar-
rangement of items within the space (Types 4 & 5) and information
about areas adjacent to their current area (Type 6).
These aspects are all important to participants in certain con-
texts, but not in all situations like position and orientation is. As P3
and P9 implied in their quotes in the previous subsection, position
and orientation awareness grants players with a sense of their cur-
rent state within the world. Prior work in the physical world has
found ascertaining this knowledge to be cognitively demanding
for VIPs as they move through an environment [24, 38]. This in-
creased cognitive load can interfere with VIPs’ ability to understand
other aspects of spatial awareness, making position and orientation
awareness essential.
Seven participants found presence to be very important for spa-
tial awareness because if an SAT did well at communicating pres-
ence, then they could be confdent that they would not miss fnding
anything within the game:
“If you hear [a familiar object], you know you are rel-
atively in the right place and you can search the area
specifcally.” — P7
Five participants clarifed why the importance of knowing the
arrangement of items is heavily context-dependent. For example,
when faced with objectives that involve fnding a specifc item,
participants believed that knowing the arrangement of items was
very helpful because it would help them fgure out where to go frst.
However, participants noted that having knowledge about items’
arrangement may be detrimental in less restrictive, exploration-
oriented tasks since that knowledge may reveal too much infor-
mation and rob players the enjoyment of discovering items for
themselves:
ASSETS ’22, October 23–26, 2022, Athens, Greece
“[Knowing arrangement] depends on what the task is [at
hand]. It’s especially [important] if it involves triggering
certain things in certain orders, fnding an item then
fnding a person, or facing of against a challenger then
fnding an NPC.” — P7
Six participants felt that having an SAT communicate which
regions are adjacent to the one they are currently in would be
benefcial as it would make navigation through the game world
easier:
“To be honest, I’ll say [having an SAT communicate
adjacencies is] extremely important because it makes
it much easier for the player to move from one area to
another without moving through the whole map.” — P5
However, three others feared that having this information pre-
sented outright may make exploration and discovery less fun. One
such participant was P4 who was a fan of games that required a
high level of strategy. P4 asserted that the game should preserve
a level of challenge and instead convey connections to other nav-
igable places using plot and contextual cues such as dialogue or
readable signs in the world itself:
“If it’s one of those strategy games where you have to
discover it on your own, it’s not important. Let’s say it’s
a hidden area, [...] it should stay hidden.” — P4
The other two participants, P2 and P6, echoed similar sentiments.
This fnding is quite surprising: These three participants thought
that an SAT did not necessarily need to communicate adjacencies
despite us identifying this as a basic aspect of spatial awareness.
This points to the importance that participants place in their ex-
perience within the game over the actual information they receive,
implying that VIPs may be willing to sacrifce receiving some pieces
of information for the sake of a more interesting gaming experience.
5.3 Rank 5 (two-way tie): Shape and Scale
[Types 1 & 2]
Participants generally found scale and shape information about an
area to be the least important aspects of spatial awareness. VIPs’
opinions generally revolved around the sentiment that, unlike the
other types of spatial awareness, scale and shape information may
be outright unnecessary much of the time.
Seven participants stated that having a sense of the room’s scale
was not important to them and that SATs should focus on conveying
information about the presence and location of nearby objects
instead:
“I feel when you are navigating in games you don’t
really need to know how big the area is as long as you
know where the objects in that area are.” — P1
Seven participants thought that an SAT should only convey in-
formation about the surrounding area’s shape when absolutely
necessary and that communicating shape information is too much
for an SAT to do, possibly resulting in information overload. How-
ever, these participants also thought that knowing shape informa-
tion in some situations may make navigation more efcient — for
example, in a situation where the room does not have a circular or
rectangular shape:
“If the room is an odd shape — every time I play a game,
I assume the room is like a square, but that’s not always
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 5: Participant ratings for how well each SAT approach
facilitates each type of spatial awareness. Large text indicates
median values, and small text indicates mean plus/minus
standard deviation. Responses were given on a fve-point
unipolar Likert scale. Blue and red cells indicate the best and
worst performing SAT approaches for each spatial awareness
type (column), respectively. The importance rankings from
our RQ1 analysis are shown again at the bottom.
the case, sometimes rooms may have [many] sides, parts
that jut out — so I believe it’s a consideration.” — P3
In a situation where a room is not rectangular, knowing the
room’s shape could help players plan out their movements more
carefully. Otherwise, players may resort to hugging walls to traverse
the room and search for doors, which may become frustrating.
6 RQ2 RESULTS: COMPARISON OF SPATIAL
AWARENESS TOOLS FOR VIRTUAL WORLDS
In this section, we report our results regarding our second research
question (RQ2): determining how well the four tools we imple-
mented facilitate the various aspects of spatial awareness. Com-
bined with our RQ1 results, these results will shed light on how
SATs should be designed to best facilitate the aspects of spatial
awareness that are most important to VIPs.
Figure 5 shows an overview of the “winners” and “losers” in
terms of participants’ post-level responses on how well each tool
facilitated the various aspects of spatial awareness. We determined
these by looking at the median ratings for all four tools for a given
spatial awareness aspect. A tool “wins” an aspect of spatial aware-
ness if it has the highest median rating out of all the tools; a tools
“loses“ an aspect of spatial awareness if it has the lowest median
rating. We broke any ties using the mean rating.
We see that the directional scanner scored the highest in terms
of facilitating three aspects of spatial awareness (scale, position &
orientation, and arrangement), meaning that participants thought
it was the best tool for facilitating those aspects. The simple audio
menu scored the highest in two types (presence and adjacencies),
and the smartphone interface and whole-room shockwave scored
Nair et al.
the highest in one type each (area shape and tied with the directional
scanner on area scale, respectively). The directional scanner did
not score the lowest on any type.
The results presented in this section are organized thematically,
with each theme representing opinions shared by a majority of our
participants.
6.1 Since participants could trace out the
contours of a room with their fnger, the
smartphone map communicated the shape
of an area better than other tools.
Figure 5 shows that participants’ ratings on how the smartphone
map communicated the shape (Type 2) of the room were generally
the highest out of all four tools. Five participants resonated with
the following sentiment:
"[The smartphone map] gives you a general idea of
where openings and spaces are. It [also] gives you an
idea of how far each edge is from your center point
which tells you, ‘OK, [the wall] angles a bit.’" — P6
One participant even used this ability to their advantage. For
example, in the irregularly-shaped fnal room of Level 3, P8 used
the smartphone interface to trace out the walls of the room and
determine that they needed to turn a corner to reach the goal
checkpoint:
“Having that memory of ‘Oh, I know a little bit more
about the shape of the room than I had previously with
the other tools.’ — that really helped me get a better sense
of exactly where I needed to go in terms of [knowing
that] I have to round a corner instead of trying to run
directly forward for the target.” — P8
6.2 The whole-room shockwave allowed
participants to quickly ascertain a general
overview of an area, especially with respect
to its scale.
Figure 5 shows that the whole-room shockwave tied the directional
scanner for being the best tool at communicating an area’s scale
(Type 1). Participants found that the distance-based volume atten-
uation aforded by Dungeon Escape’s 3D sound system and the
delayed timing of objects’ sounds during a shockwave helped them
approximate how far away objects were and, thus, how big the
room was.
P8 was one such participant; they described themselves as "not-
at-all experienced" with echolocation techniques in the pre-study
questionnaire. (Recall from Section 3.2 that we derived the whole-
room shockwave from our explorations in echolocation.) Yet, they
relayed the following positive sentiment which was shared by many
other participants despite their inexperience with echolocation:
“Even though doors and objects were further away from
me, I was still able to know that they are still in fact
there. [The shockwave] helped me quickly gauge ’OK,
cool. I know I’m in a corridor [...] and I know there is a
door in the far end, and so this helps me determine on a
higher level how big the room might be.’” — P8
Uncovering Visually Impaired Gamers’ Preferences for SATs Within Video Games
The quick nature of the shockwave particularly advantaged par-
ticipants within areas with many items. One such area within Dun-
geon Escape was the irregularly shaped room mentioned in Section
6.1, which contained obstacles in the form of barrels and crates.
Participants who used the whole-room shockwave hit the check-
point in that room much faster (� = 19 sec., �� = 2.8 sec.) than
those who used the directional scanner (� = 51 sec., �� = 17.7
sec.), simple audio menu (� = 123 sec., �� = 5.0 sec.), and smart-
phone map (� = 135 sec., �� = 15.5 sec.). The shockwave provided
participants with an almost-instant overview of what obstacles
were in the room. However, those who used the other tools spent
a much longer time searching for these very obstacles — scrolling
through each item (using the menu), trying to point at them (using
the directional scanner), or trying to fnd them on a map (using the
smartphone).
6.3 Participants made extensive use of the
whole-room shockwave’s flters.
All nine participants made use of the whole-room shockwave’s
fltering mechanism. Six of them explicitly mentioned that this
ability was extremely important to them and that they highly valued
this ability even in applications outside of games. One participant
invoked the customizability of screen readers as an example:
“Everybody has diferent needs and wants so I really
believe in allowing information to be fltered in such
a way where you just get the information you need as
you need it like in the shockwave. [...] Screen readers
have settings like this for a reason.” — P3
6.4 The physical use of the right stick in the
directional scanner meant that participants
could obtain a clear idea of how items were
arranged around themselves.
Our fndings with the directional scanner provided insights into
how physically moving a joystick to survey an environment might
provide players with an enhanced sense of its layout. Five partici-
pants explicitly mentioned that moving the joystick to “look around”
allowed them to understand how objects were arranged:
“Because of where I had to put the stick to see stuf
around me, it really helped. It was easier to tell what
was behind me, what was in front of me, or what was
in any other direction because I knew where my stick
position was.” — P6
P6 went on to say that surveying with the joystick “felt natural”
and compared the directional scanner to a camera which they could
use to “look” for objects. This sentiment is further refected in Fig-
ure 5, which shows that the directional scanner scored the highest
out of all four tools in terms of communicating the arrangement of
items (Type 5).
ASSETS ’22, October 23–26, 2022, Athens, Greece
6.5 The simple audio menu’s straightforward
presentation of items meant that
participants received information about the
presence of items extremely well.
Participants found that the simple audio menu clearly communi-
cated the presence of every item in the room. As seen in Figure 5, the
simple audio menu received an average/median rating of 5.0 with a
standard deviation of zero on communicating the presence of items
(Type 4) — every participant gave the menu the maximum possible
score on this aspect of spatial awareness. Participants unanimously
agreed that they were able to obtain a clear idea of what was in the
room because of its straightforward presentation:
“You know everything that’s there because it’s in a
menu. There’s nothing hidden. It doesn’t matter if you’re
far away from it. If you’re in the same room as it, it’s
on that list. That’s something I really like.” — P3
Yet, some participants complained that the simple audio menu
provided too much information. Within games, a certain degree of
surprise and exploration — that is, the ability to “discover” aspects
of the game world — are core elements for making a game fun for
players [37]. Knowing the presence of objects so easily can remove
this aspect of discovery from the game. Indeed, fve participants
felt that the simple audio menu did not promote exploration and
made the game less enjoyable:
“I thought that I was using a shortcut. [...] I like that
it’s faster, but it takes something out of the game expe-
rience.” — P2
Within Dungeon Escape itself, participants tended to avoid de-
viating from the task at hand while using the simple audio menu,
going directly to POIs they needed to go to. Figure 6 plots paths
taken by participants within the key room in Level 2. Note how P3
went straight to the key when using the simple audio menu in Level
2, while participants who used other tools in the same level roamed
around the room in an efort to survey their surroundings more
thoroughly. This behavior is also visible in the raw time data we
collected within the room: Those who used the menu collected the
key much faster (� = 17 sec., �� = 4.2 sec.) than those who used the
shockwave (� = 84 sec., �� = 6.5 sec.), smartphone (� = 93 sec.,
�� = 17.5 sec.), and directional scanner (� = 105 sec., �� = 5 sec.).
Although players completed the levels with the simple audio menu
(and often did so quickly), it remains an open question whether
players’ increased focus on objectives and lack of exploration is a
net positive for the game experience or not.
6.6 No tool excelled at communicating position
and orientation.
As we found in Section 5, participants rated position and orientation
(Type 3) as the most important aspect of spatial awareness to them.
However, post-level ratings indicate that participants perceived all
four tools to be mediocre at facilitating position and orientation
information. As Figure 5 shows, the average score that each SAT
received in terms of afording position and orientation information
was a low-to-mid three (“moderately well”). Our results indicate
that these four tools may not meet the high bar that these tools
need to meet for such an important aspect of spatial awareness.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 6: Illustration of paths taken by participants within the Level 2 key room. The subfgures depict four diferent participants
traversing the same room with diferent tools; the key is located in the lower left corner, and participants enter through the
door at the bottom. Paths are divided into segments — the end of a segment represents a point where the participant paused to
survey using the tool.
6.7 Participants disliked having to juggle
multiple pieces of hardware when using the
smartphone map.
Five participants mentioned that they found the smartphone map
cumbersome to use. Participants often found themselves needing
to physically switch between their controller and their smartphone
when they wanted to explore the map. Furthermore, at least six
participants used noise-cancelling headphones during their sessions
and had to adjust them as well to hear the smartphone’s audio
output. These experiences annoyed some participants:
"I have mixed emotions [about the smartphone map]
because I have to do one thing on one device and then
move with the other device [...] That made things a bit
confusing and annoying." - P2
6.8 The simple audio menu did not
communicate scale and shape well.
As Figure 5 shows, participants thought that the simple audio menu
communicated the scale (Type 1) and shape (Type 2) of areas quite
poorly with average ratings of around 2 out of 5. The simple audio
menu did not explicitly communicate boundaries or other char-
acteristics of the room itself. As such, many participants could
not defnitively determine the structure (scale and shape) of the
surrounding area using the menu:
"I could probably use [the simple audio menu’s] 3D
sounds to assume that, say, a bunch of items were
against a wall if they’re coming from the same general
side relative to me [...] but that’s an educated guess." —
P6
6.9 Participants found the whole-room
shockwave to be overwhelming, which
negatively afected their spatial awareness.
Five participants felt that despite communicating scale relatively
well, the whole-room shockwave provided too much information,
which negatively afected their sense of spatial awareness. We were
surprised by this fnding since we designed the shockwave to emit
Nair et al.
slowly for better intelligibility, and every participant used the flters
to make the shockwave easier to understand.
Yet, despite these improvements, participants still felt that the
shockwave was too information-dense, making it difcult for them
to ascertain information about their environment. Our conver-
sations with them yielded insights into how VIPs view similar
echolocation-inspired tools within other games. P3, who described
themselves as having played "everything" when asked about their
gaming experience during the pre-study, was especially vocal:
“Everybody thinks you can just send out a sonar ping
and get information about an environment. [...] Echolo-
cation is very overwhelming, especially in a game. Try-
ing to hone in on an item that is far away and being
masked by another item is ludicrous.” — P3
6.10 Participants preferred combinations of
tools that excelled across multiple spatial
awareness aspects.
Figure 5 gives us an interesting perspective on how combinations
of tools can best facilitate multiple aspects of spatial awareness
jointly. As stated in Section 4.4, we asked participants to state their
two most preferred combinations of tools. The (directional scanner
+ simple audio menu) combination was one of the most selected
combinations, with fve participants selecting it. This combination
"wins" in four out of the six aspects of spatial awareness: position
& orientation, presence of items, arrangement of items, and com-
municating adjacent areas. Furthermore, this combination is "tied"
with the whole-room shockwave at being the best at conveying
area scale. Five participants also selected the (directional scanner +
whole-room shockwave) combination, which wins at three of the
six aspects of spatial awareness. We discuss these selections further
in Section 7.
7 DISCUSSION: TAKEAWAYS FROM RQ1 AND
RQ2 TOGETHER
In Sections 5 and 6, we reported our fndings about what aspects of
spatial awareness VIPs fnd important (RQ1) and how well current
SAT approaches facilitate the various aspects of spatial awareness
(RQ2). In this section, we now synthesize these fndings together to
Uncovering Visually Impaired Gamers’ Preferences for SATs Within Video Games ASSETS ’22, October 23–26, 2022, Athens, Greece

form broader takeaways for how SATs should be designed. We hope

that game designers can use these takeaways to decide which SAT
is best for them to incorporate into their game to make it accessible,
if they only have the resources to implement one or two of them.

7.1 Position and orientation is the most

important type of spatial awareness for
VIPs, yet is not well-served by current tools.
Our results indicate that communicating position and orientation
well is a crucial challenge that must be addressed when designing fu-
ture SATs. As we reported in Section 6.6, participants rated position
and orientation as the most important aspect of spatial awareness Figure 7: Box plot of net promoter score responses for all four
to them. Yet, they also felt that all four tools were mediocre at fa- tools. Red lines indicate median. The whole-room shockwave
cilitating position and orientation information. Surprisingly, this received some of the lowest scores out of all four tools.
includes the smartphone map, which was the tool that most ex-
plicitly communicated position and orientation information, as we
described in Section 3.1. This indicates that communicating position
and orientation information to VIPs is harder than researchers as-
sume and that a major opportunity for future research is to develop
better indicators for VIPs’ position and orientation. Design Implication #2: Of today’s SATs, the combination of
Previous research has shown that VIPs rely heavily on land- the directional scanner and simple audio menu gives VIPs the
marks and other environmental features to determine their position greatest spatial awareness.
and orientation, which, in turn, allows them to navigate through
environments [21, 66]. These landmarks include walls and other
boundaries dictating the area’s scale and shape (Types 1 and 2) as
well as the layout of items within the space (Type 5). From our 7.3 VIPs highly value the ability to customize
fndings, however, we see that VIPs do not fnd it suitable to merely
infer their position and orientation from these other cues and would
SATs.
rather beneft from having it communicated more explicitly. In addition to the (directional scanner + simple audio menu) com-
bination, fve participants also picked the (directional scanner +
whole-room shockwave) combination as one of their favorite combi-
Design Implication #1: VIPs will beneft greatly from a nations. Unlike the former combination, however, the latter combi-
purpose-built tool for communicating position and orientation nation only “wins” at three of the six types of spatial awareness (i.e.,
in real time. the two tools cover winning values for three columns in Figure 5).
Additionally, Figure 7 shows that the shockwave had some of the
lowest net promoter scores out of all of the tools, and participants
7.2 The four most important aspects of spatial even complained about the tool being overwhelming. This fnding
awareness are covered by two tools. implies that there exists a consideration that VIPs may fnd even
If we consider the four most important aspects of spatial awareness more important than raw spatial awareness.
from our RQ1 fndings — position and orientation (Type 3), item One possible explanation lies in the fact that the whole-room
presence (Type 4), item arrangement (Type 5), and adjacent areas shockwave was the only tool that participants could change the
(Type 6) — we can see that the combination of the directional behavior of — in this case, selecting the type of information they
scanner with the simple audio menu “wins” at communicating all wanted to hear. Participants’ enthusiasm for customizable tools
four of these types. We do not consider area scale (Type 1) and — especially evident in their comparisons with other tools such
area shape (Type 2) because participants found them to be the least as screen readers — shows that SATs should implement similar
important; however, we can also see that the directional scanner is capabilities, allowing VIPs to take control of what they hear.
tied for “winning” scale as well. From a theoretical standpoint, this
implies that VIPs would most gravitate toward this combination,
Design Implication #3: SATs should embrace customizabil-
and indeed, we saw precisely this during our study.
ity, allowing VIPs to customize and flter the information com-
The fact that participants are excited about the (directional scan-
municated.
ner + simple audio menu) combination makes sense. It seems that,
with this combination of tools, participants gravitated toward a
combination that facilitates the greatest number of spatial aware-
ness aspects well. 8 FUTURE WORK
The fndings from our study revealed several avenues for future
work, which we propose in this section.
ASSETS ’22, October 23–26, 2022, Athens, Greece
8.1 Toward optimally communicating each
spatial awareness type.
Our fndings showed that communicating some aspect of spatial
awareness well is not simply about doing so to the maximum extent
possible. For example, when it comes to conveying the presence
of items (Type 4), the simple audio menu facilitated it perfectly
(receiving perfect Likert scores), but many participants disliked
how it listed every item within the room they were currently in.
They thought that the menu communicated too much information
— enough to afect how much fun they had playing the game.
These fndings indicate that — particularly within video games —
communicating a specifc type of spatial awareness optimally does
not necessarily mean communicating it at the maximum possible
level. Future work should address what "optimal" really means
in terms of communicating each type of spatial awareness. For
example, in the case of item presence information (Type 4): What
is the proper level of item presence that should be communicated
to the player, and what factors — such as game objectives — may
infuence the level of item presence a tool should communicate?
Similar questions can be extended to the other types as well.
8.2 Toward purpose-built hardware.
Participants revealed that they disliked juggling multiple pieces
of hardware while using the smartphone map. Future touch-based
SATs should reduce the number of devices required. One possibility
involves using touchpads found on game controllers such as the Du-
alShock 4 [18], DualSense [19], and Steam Controller [60]. Hybrid
touchscreen controller devices, such as the Nintendo Switch [47]
and the Steam Deck [61], are also promising alternatives.
8.3 Applications for physical world navigation.
In Section 5, we addressed RQ1 by reporting participants’ prefer-
ences for the six types of spatial awareness within a video game
context. Future work could explore VIPs’ preferences within the
physical world and see how they difer from their preferences within
video games. We found that some of our results resemble prior work
in the physical world: Participants generally agreed that position
& orientation was extremely important — in our study, they collec-
tively saw it as the most important type of spatial awareness. In a
similar vein, much physical world work for both visually impaired
and sighted people has found position & orientation awareness
to be very important [20, 22, 24, 31, 34]. Participants also gener-
ally found knowledge of item presence, item arrangement, and
adjacent areas to be relatively important as well — refecting prior
work that has echoed the importance of inter-object and inter-area
relationships in promoting spatial awareness [24, 29, 49, 65].
We were surprised, however, when we found that participants
did not fnd scale and shape awareness to be very important within
video games. This difers from much prior work from physical world
contexts — especially in the realm of tactile maps and echolocation
— that has found general overviews of spaces, including information
such as scale and shape, to be crucial for spatial awareness [30, 49].
An interesting direction for future work may involve repeating
the study presented in this paper, but in the physical world, to enable
a direct comparison. The physical world presents its own challenges
and circumstances. SATs’ accuracy within physical environments
Nair et al.
and VIPs’ physical safety considerations [6], for example, could
infuence how important VIPs fnd the various types of spatial
awareness and even how they wayfnd and explore using the tools.
A direct comparison can help the community establish a hier-
archy from the spatial information that we know is important to
VIPs. It can also help the community establish formal principles
for prioritizing the display of diferent types of information during
physical world navigation.
9 LIMITATIONS
As with many studies that involve people with visual impairments,
we had a low number of participants. The preferences for spatial
awareness that we found are based on the perspective of our nine
participants and may difer for other VIPs. Although the four SATs
we implemented covered a broad range of design possibilities, there
may be other designs that we did not consider that could reveal
more insights into what VIPs value in a spatial awareness tool for
virtual worlds. Furthermore, while we are grateful to the nine VIPs
who participated in our study, we regret not being able to recruit
a more diverse group of participants. Finally, our work focused
on 3D adventure video games with large worlds that players can
traverse, and our testbed did not feature any moving objects. As
such, additional work is needed to investigate SATs that may assist
VIPs within other types of video games, especially those that feature
moving objects (for example, enemies and projectiles).
10 CONCLUSION
In this work, we explore the merits and limitations of existing
approaches to facilitating spatial awareness for VIPs within video
game worlds in order to allow accessible game designers to have
a better understanding of which spatial awareness tools are best
to include in games to make them accessible to VIPs. Through a
user study, we investigated four leading approaches to facilitating
spatial awareness for VIPs in an efort to understand what aspects
of spatial awareness VIPs fnd important within games (RQ1), and
to determine how well today’s difering SAT approaches facilitate
the various aspects of spatial awareness (RQ2).
Regarding the frst question, we found that participants consid-
ered position and orientation to be the most important aspect of
spatial awareness, and that scale and shape are the least important.
Regarding the second question, participants found the directional
scanner to communicate the arrangement of items very well, the
simple audio menu to communicate the presence of items very well,
the smartphone map to communicate the shape of areas very well,
and the whole-room shockwave to communicate the scale of the
area well. Our fndings also revealed defciencies in current SAT
approaches, including that some tools tend to provide too much
information and that no tool excels at communicating position and
orientation information — despite it being the most important type
of spatial awareness to participants.
We hope that better understanding VIPs’ preferences for spatial
awareness as well as how today’s SATs work can open up access to
more mainstream 3D video games, granting VIPs the same gaming
experiences that sighted players are so often aforded.
Uncovering Visually Impaired Gamers’ Preferences for SATs Within Video Games
ACKNOWLEDGMENTS
We would like to thank Michael Malcolm and Sebastián Mercado
for their assistance during our pilot tests. We would also like to
extend our sincere gratitude toward our study participants for their
participation and to the anonymous reviewers for their helpful feed-
back. Mason Hayes, Hannah Huddleston, and Matthew Donnelly
were funded by National Science Foundation Grants 2051053 and
2051060. The opinions, fndings, conclusions, and/or recommen-
dations expressed are those of the authors and do not necessarily
refect the views of the National Science Foundation.
REFERENCES
[1] Capcom Production Studio 4. 2005. Resident Evil 4. Capcom, Osaka, Japan.
[2] Ronny Andrade, Steven Baker, Jenny Waycott, and Frank Vetere. 2018. Echo-
house: exploring a virtual environment by using echolocation. In Proceedings
of the 30th Australian Conference on Computer-Human Interaction (OzCHI ’18).
Association for Computing Machinery, 278–289. https://doi.org/10.1145/3292147.
3292163
[3] Ronny Andrade, Melissa J. Rogerson, Jenny Waycott, Steven Baker, and Frank
Vetere. 2020. Introducing the Gamer Information-Control Framework: Enabling
Access to Digital Games for People with Visual Impairment. In Proceedings
of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20).
Association for Computing Machinery, 1–14. https://doi.org/10.1145/3313831.
3376211
[4] Ronny Andrade, Jenny Waycott, Steven Baker, and Frank Vetere. 2021. Echolo-
cation as a Means for People with Visual Impairment (PVI) to Acquire Spatial
Knowledge of Virtual Space. ACM Transactions on Accessible Computing 14, 1
(Mar 2021), 4:1–4:25. https://doi.org/10.1145/3448273
[5] Dominique Archambault, Thomas Gaudy, Klaus Miesenberger, Stéphane Natkin,
and Rolland Ossmann. 2008. Towards Generalised Accessibility of Computer
Games. Lecture Notes in Computer Science, Vol. 5093. Springer Berlin Heidelberg,
518–527. https://doi.org/10.1007/978-3-540-69736-7_55
[6] Nikola Banovic, Rachel L. Franz, Khai N. Truong, Jennifer Mankof, and Anind K.
Dey. 2013. Uncovering information needs for independent spatial learning for
users who are visually impaired. In Proceedings of the 15th International ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS ’13). Association
for Computing Machinery, 1–8. https://doi.org/10.1145/2513383.2513445
[7] Felix Bork, Ulrich Eck, and Nassir Navab. 2019. Birds vs. Fish: Visualizing Out-of-
View Objects in Augmented Reality using 3D Minimaps. In 2019 IEEE International
Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 285–286.
https://doi.org/10.1109/ISMAR-Adjunct.2019.00-28
[8] Capcom. 1996. Resident Evil. Capcom, Osaka, Japan.
[9] Capcom. 1998. Resident Evil 2. Capcom, Osaka, Japan.
[10] Capcom. 1999. Resident Evil 3: Nemesis. Capcom, Osaka, Japan.
[11] Capcom. 2009. Resident Evil 5. Capcom, Osaka, Japan.
[12] Code Respawn. [n.d.]. Dungeon Architect. https://assetstore.unity.com/
packages/tools/utilities/dungeon-architect-53895
[13] Sebastiano M. Cossu. 2019. Designing Platformers. Apress, 367–380. https:
//doi.org/10.1007/978-1-4842-5010-5_10
[14] Sebastiano M. Cossu. 2019. Metroidvania (Part 2). Apress, 437–497. https:
//doi.org/10.1007/978-1-4842-5010-5_12
[15] Core Design. 1996. Tomb Raider. Eidos Interactive, London, England.
[16] Naughty Dog. 2020. The Last of Us Part II. Sony Interactive Entertainment.
[17] Quantic Dream. 2010. Heavy Rain. Sony Computer Entertainment, Tokyo, Japan.
[18] Sony Interactive Entertainment. 2013. DualShock 4. Sony Corporation.
[19] Sony Interactive Entertainment. 2020. DualSense. Sony Corporation.
[20] Russell A. Epstein, Eva Zita Patai, Joshua B. Julian, and Hugo J. Spiers. 2017. The
cognitive map in humans: Spatial navigation and beyond. Nature neuroscience
20, 11 (Oct 2017), 1504–1513. https://doi.org/10.1038/nn.4656
[21] Navid Fallah, Ilias Apostolopoulos, Kostas Bekris, and Eelke Folmer. 2012. The
user as a sensor: navigating users with visual impairments in indoor spaces
using tactile landmarks. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems. Association for Computing Machinery, 425–432.
https://doi.org/10.1145/2207676.2207735
[22] Nicholas A. Giudice, Benjamin A. Guenther, Nicholas A. Jensen, and Kaitlyn N.
Haase. 2020. Cognitive Mapping Without Vision: Comparing Wayfnding Per-
formance After Learning From Digital Touchscreen-Based Multimodal Maps
vs. Embossed Tactile Overlays. Frontiers in Human Neuroscience 14 (2020), 87.
https://doi.org/10.3389/fnhum.2020.00087
[23] Nicholas A. Giudice, Benjamin A. Guenther, Nicholas A. Jensen, and Kaitlyn N.
Haase. 2020. Cognitive Mapping Without Vision: Comparing Wayfnding Per-
formance After Learning From Digital Touchscreen-Based Multimodal Maps
vs. Embossed Tactile Overlays. Frontiers in Human Neuroscience 14 (2020).
ASSETS ’22, October 23–26, 2022, Athens, Greece
https://doi.org/10.3389/fnhum.2020.00087
[24] Nicholas A. Giudice and Gordon E. Legge. 2008. Blind Navigation and the Role
of Technology. John Wiley and Sons, Ltd, 479–500. https://doi.org/10.1002/
9780470379424.ch25
[25] Cagatay Goncu, Anuradha Madugalla, Simone Marinai, and Kim Marriott. 2015.
Accessible On-Line Floor Plans. In Proceedings of the 24th International Conference
on World Wide Web (WWW ’15). International World Wide Web Conferences
Steering Committee, 388–398. https://doi.org/10.1145/2736277.2741660
[26] Cagatay Goncu and Kim Marriott. 2011. GraVVITAS: Generic Multi-touch Presen-
tation of Accessible Graphics. In Human-Computer Interaction – INTERACT 2011
(Lecture Notes in Computer Science), Pedro Campos, Nicholas Graham, Joaquim
Jorge, Nuno Nunes, Philippe Palanque, and Marco Winckler (Eds.). Springer,
30–48. https://doi.org/10.1007/978-3-642-23774-4_5
[27] Google. [n.d.]. Firebase App Distribution. https://frebase.google.com/docs/app-
distribution
[28] Timo Götzelmann and Klaus Winkler. 2015. SmartTactMaps: a smartphone-
based approach to support blind persons in exploring tactile maps. In Proceedings
of the 8th ACM International Conference on PErvasive Technologies Related to
Assistive Environments (PETRA ’15). Association for Computing Machinery, 1–8.
https://doi.org/10.1145/2769493.2769497
[29] E.W. Hill, J.J. Rieser, M.-M. Hill, M. Hill, J. Halpin, and R. Halpin. 1993. How Per-
sons with Visual Impairments Explore Novel Spaces: Strategies of Good and Poor
Performers. 87 (Oct 1993), 295–301. https://doi.org/10.1177/0145482X9308700805
[30] Emily Holmes and Aries Arditi. 1998. Wall versus Path Tactile Maps for Route
Planning in Buildings. Journal of Visual Impairment & Blindness 92, 7 (Jul 1998),
531–534. https://doi.org/10.1177/0145482X9809200713
[31] Hernisa Kacorri, Sergio Mascetti, Andrea Gerino, Dragan Ahmetovic, Hironobu
Takagi, and Chieko Asakawa. 2016. Supporting Orientation of People with
Visual Impairment: Analysis of Large Scale Usage Data. In Proceedings of the
18th International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’16). Association for Computing Machinery, 151–159. https://doi.org/
10.1145/2982142.2982178
[32] Jeremy Kaldobsky. 2011. Swamp. https://www.kaldobsky.com/ssl/audiogames.
php
[33] Daniel Kish. 2009. Human echolocation: How to “see” like a bat. New Scientist
202, 2703 (Apr 2009), 31–33. https://doi.org/10.1016/S0262-4079(09)60997-0
[34] Roberta L. Klatzky. 1998. Allocentric and Egocentric Spatial Representations:
Defnitions, Distinctions, and Interconnections. Springer, 1–17. https://doi.org/10.
1007/3-540-69342-4_1
[35] Alexander Klippel, Stephen Hirtle, and Clare Davies. 2010. You-Are-Here
Maps: Creating Spatial Awareness through Map-like Representations. Spatial
Cognition & Computation 10, 2–3 (Jun 2010), 83–93. https://doi.org/10.1080/
13875861003770625
[36] Andrew J. Kolarik, Silvia Cirstea, Shahina Pardhan, and Brian C. J. Moore. 2014.
A summary of research investigating echolocation abilities of blind and sighted
humans. Hearing Research 310 (Apr 2014), 60–68. https://doi.org/10.1016/j.heares.
2014.01.010
[37] Marc LeBlanc. 2008. The collected game design rants of Marc LeBlanc. http:
//algorithmancy.8kindsofun.com/
[38] Laura Lewis, Sarah Sharples, Ed Chandler, and John Worsfold. 2015. Hearing the
way: Requirements and preferences for technology-supported navigation aids.
Applied Ergonomics 48 (May 2015), 56–69. https://doi.org/10.1016/j.apergo.2014.
11.004
[39] Masaki Matsuo, Takahiro Miura, Masatsugu Sakajiri, Junji Onishi, and Tsukasa
Ono. 2016. Audible Mapper & ShadowRine: Development of Map Editor Using
only Sound in Accessible Game for Blind Users, and Accessible Action RPG for
Visually Impaired Gamers. In Computers Helping People with Special Needs (Lecture
Notes in Computer Science), Klaus Miesenberger, Christian Bühler, and Petr Penaz
(Eds.). Springer International Publishing, 537–544. https://doi.org/10.1007/978-
3-319-41264-1_73
[40] Microsoft Research. 2018. Microsoft Soundscape - Microsoft Research.
https://www.microsoft.com/en-us/research/product/soundscape/.
[41] Vishnu Nair, Jay L Karp, Samuel Silverman, Mohar Kalra, Hollis Lehv, Faizan
Jamil, and Brian A. Smith. 2021. NavStick: Making Video Games Blind-Accessible
via the Ability to Look Around. In The 34th Annual ACM Symposium on User Inter-
face Software and Technology (UIST ’21). Association for Computing Machinery,
538–551. https://doi.org/10.1145/3472749.3474768
[42] Vishnu Nair and Brian A. Smith. 2020. Toward Self-Directed Navigation for People
with Visual Impairments. In Adjunct Publication of the 33rd Annual ACM Sympo-
sium on User Interface Software and Technology (UIST ’20 Adjunct). Association
for Computing Machinery, 139–141. https://doi.org/10.1145/3379350.3416156
[43] Liam J. Norman, Caitlin Dodsworth, Denise Foresteire, and Lore Thaler. 2021.
Human click-based echolocation: Efects of blindness and age, and real-life im-
plications in a 10-week training program. PLOS ONE 16, 6 (Jun 2021), e0252330.
https://doi.org/10.1371/journal.pone.0252330
[44] Out of Sight Games. 2017. A Hero’s Call.
[45] Pin Interactive. 2003. Terraformers.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[46] John R. Porter and Julie A. Kientz. 2013. An empirical study of issues and barriers
to mainstream video game accessibility. In Proceedings of the 15th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’13). Associ-
ation for Computing Machinery, 1–8. https://doi.org/10.1145/2513383.2513444
[47] Nintendo PTD. 2017. Nintendo Switch. Nintendo.
[48] Frederick F. Reichheld. 2003. The one number you need to grow. Harvard Business
Review 81, 12 (2003), 46–55.
[49] Jonathan Rowell and Simon Ungar. 2005. Feeling our way: tactile map user
requirements-a survey. In International Cartographic Conference.
[50] Daisuke Sato, Uran Oh, Kakuya Naito, Hironobu Takagi, Kris Kitani, and Chieko
Asakawa. 2017. NavCog3: An Evaluation of a Smartphone-Based Blind Indoor
Navigation Assistant with Semantic Features in a Large-Scale Environment. In
Proceedings of the 19th International ACM SIGACCESS Conference on Computers
and Accessibility - ASSETS ’17. ACM Press, Baltimore, Maryland, USA, 270–279.
https://doi.org/10.1145/3132525.3132535
[51] Team Silent and Konami Computer Entertainment Tokyo. 1999. Silent Hill.
Konami, Tokyo, Japan.
[52] Brian A. Smith and Shree K. Nayar. 2018. The RAD: Making Racing Games
Equivalently Accessible to People Who Are Blind. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems. Association for Computing
Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3174090
[53] Retro Studios. 2002. Metroid Prime. Nintendo, Kyoto, Japan.
[54] Retro Studios. 2004. Metroid Prime 2: Echoes. Nintendo, Kyoto, Japan.
[55] Retro Studios. 2007. Metroid Prime 3: Corruption. Nintendo, Kyoto, Japan.
[56] Jing Su, Alyssa Rosenzweig, Ashvin Goel, Eyal de Lara, and Khai N. Truong. 2010.
Timbremap: Enabling the Visually-Impaired to Use Maps on Touch-Enabled
Devices. In Proceedings of the 12th International Conference on Human Computer
Interaction with Mobile Devices and Services (Lisbon, Portugal) (MobileHCI ’10).
Association for Computing Machinery, New York, NY, USA, 17–26. https://doi.
org/10.1145/1851600.1851606
Nair et al.
[57] Lore Thaler and Melvyn A. Goodale. 2016. Echolocation in humans: an overview.
WIREs Cognitive Science 7, 6 (2016), 382–393. https://doi.org/10.1002/wcs.1408
[58] Ross Thorn. 2018. How to Play With Maps. Ph.D. Dissertation. https://minds.
wisconsin.edu/handle/1793/78913 Accepted: 2019-01-17T20:40:13Z.
[59] Unity Technologies. 2020. Unity. Unity Technologies.
[60] Valve. 2015. Steam Controller. Valve Corporation.
[61] Valve. 2022. Steam Deck. Valve Corporation.
[62] Valve Software. 2019. Steam Audio. https://valvesoftware.github.io/steam-audio/
[63] T. Westin. 2004. Game Accessibility Case Study: Terraformers – a Real-Time 3d
Graphic Game. In In Proc. of the The Fifth International Conference on Disability,
Virtual Reality and Associated Technologies. 95–100.
[64] Rayoung Yang, Sangmi Park, Sonali R. Mishra, Zhenan Hong, Clint Newsom,
Hyeon Joo, Erik Hofer, and Mark W. Newman. 2011. Supporting spatial awareness
and independent wayfnding for pedestrians with visual impairments. In The
proceedings of the 13th international ACM SIGACCESS conference on Computers
and accessibility (ASSETS ’11). Association for Computing Machinery, 27–34.
https://doi.org/10.1145/2049536.2049544
[65] Koji Yatani, Nikola Banovic, and Khai Truong. 2012. SpaceSense: represent-
ing geographical information to visually impaired people using spatial tac-
tile feedback. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems. Association for Computing Machinery, 415–424. https:
//doi.org/10.1145/2207676.2207734
[66] Xunyi Yu and Aura Ganz. 2012. Audible vision for the blind and visually im-
paired in indoor open spaces. In 2012 Annual International Conference of the IEEE
Engineering in Medicine and Biology Society. 5110–5113. https://doi.org/10.1109/
EMBC.2012.6347143
[67] Krzysztof Zagata, Jacek Gulij, Łukasz Halik, and Beata Medyńska-Gulij. 2021.
Mini-Map for Gamers Who Walk and Teleport in a Virtual Stronghold. ISPRS
International Journal of Geo-Information 10, 22 (Feb 2021), 96. https://doi.org/10.
3390/ijgi10020096
 Expressive Bodies
Engaging with Embodied Disability Cultures for Collaborative
Design Critiques
Katta Spiel
Robin Angelini
katta.spiel@tuwien.ac.at
robin.angelini@student.tuwien.ac.at
HCI Group – TU Wien
Vienna, Austria
ABSTRACT 1 BRINGING BODIES BACK INTO CRITIQUE
In our experience as researchers engaging with non-academic audi- Methods involving people when evaluating, testing and assessing
ences, we observed that it remains a challenge to receive direct and technologies in Human-Computer Interaction (HCI) come with a
critical feedback from participants. This is particularly amplifed range of expectations as to which skills these people have to bring to
in the context of disabilities even if the researchers identify them- the interaction to be considered as suitable partners. This becomes
selves as disabled given that the interaction is governed by social particularly relevant when designing and developing technologies
status and material power dimensions to say the least. To work with and for disabled people [23], although the notion that bodies
productively with these power dynamics, we explored embodied are relevant to the technological research we conduct has potentials
approaches to articulating critique acknowledging the diferent reaching beyond these populations [40]. Here, the suggestion is
ways of knowing stemming from diferent bodyminds. Here, we often to adapt methods to make them and the overall research envi-
line out two exploratory cases illustrating how physical bodies can ronment, be it virtual or physical, accessible to participants, while
be directly attended to to express critiques in more direct ways leaving the methods themselves untouched [23]. Such adaptations
than participants might be used to on a language based level (spo- become more and more common to be reported on, such as work
ken or signed). We show how communication and critique can by Dingman et al. on adapting interview practices for Deaf and
take on many forms encouraging us to broaden our methodologi- Hard of Hearing (DHH) populations [7]. How easily methods can be
cal toolset to incorporate practices common in disability cultures. adapted to specifc populations might also explain why some disabil-
Our experiences show that we need to embrace crip approaches ities are catered more to in technological accessibility research than
to knowledge production to receive more actionable and useful others [25]. Overall, within communities researching technologies
feedback in developing technologies with disabled communities. for and with disabled people, we further notice an emerging trend
of moving towards ‘cultivating access’ [26]. However, in general
CCS CONCEPTS purpose technological contexts, the ableist paradigms of Western
• Social and professional topics → People with disabilities; • societies [3] seep into the design and development as well as the
Human-centered computing → Accessibility design and eval- methods for assessment allowing us to understand their potentially
uation methods. exclusive character [41], which has been illustrated in detail for
virtual reality technologies [12].
In thinking with disability cultures [36], crip theories [30] and
KEYWORDS
notions of disability justice [33], we draw on an understanding
disability cultures, Deaf cultures, Autism, Neurodivergence, crip of situated knowledges [14] to probe how we might think about
methodologies, embodied critique, critical bodyminds not only making existing methods accessible, but in a notion of
ACM Reference Format: cripistemologies [16] develop complementary methods arriving at
Katta Spiel and Robin Angelini. 2022. Expressive Bodies Engaging with diferent kinds of knowledges. Doing so we aim at acknowledging
Embodied Disability Cultures for Collaborative Design Critiques. In The 24th how bodyminds in their particularities [4] aid us in honouring a
International ACM SIGACCESS Conference on Computers and Accessibility range of ways of knowing about and understanding technologies.
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, A frst step here is to involve disabled people in research about
6 pages. https://doi.org/10.1145/3517428.3551350 them due to their individual and relevant expertise [32], which has
been argued for HCI specifcally prominently in previous publi-
cations [28, 44]. However, a recent survey by Sarsenbayeva et al.
illustrated that 32% of articles discussing motor disabilities and
This work is licensed under a Creative Commons technologies are still published without indicating the involvement
Attribution-NonCommercial-ShareAlike International 4.0 License.
of disabled participants at all [38]. Similarly, even in games, we
ASSETS ’22, October 23–26, 2022, Athens, Greece observe a dominance of the medical, i.e., individualising model of
© 2022 Copyright held by the owner/author(s). disability governing research questions, technological design and
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3551350
ASSETS ’22, October 23–26, 2022, Athens, Greece
development as well as assessment [43]. To attend to the particu-
larities of disabled knowledges, recent years have seen an uptake
of frst person research methods, so far mostly in the context of
autoethnographies detailing travel experiences of hard of hearing
[15] or blind travellers [46]. Regardless, the expertise of disabled
researchers remains, at least partially, subject to epistemic violence
[49]. This makes it difcult to consider methodological approaches
with even less foundation in more classical understandings of what
methods can and should do in HCI and what principles they should
be anchored on.
To explore how this might look like, we refect on two exploratory
case studies that illustrate the potentials of understanding critical
feedback not just in ways of direct language based engagements or
detached sensor measurements (see also, [2]). In both, we collabo-
rated with disabled people in diferent settings at diferent times
with a focus on what we may achieve by attending to what we
call expressive bodies, i.e. the use of the body-based expressiveness
of a range of diferent bodyminds beyond direct language based
communication – spoken or signed. With these, we hope to show
possible steps of how we might go beyond the cultivation of access
by specifcally positioning our research not just in the context of
disabilities but disability cultures and the situated practices and re-
lations therein as they pertain to the construction and negotiations
of diferent knowledges [16].
In this experience report, we start by probing the two difer-
ent ways of embodying critique, we observed, frst in the context
of participatory design with autistic children then in the context
of observing actors in the production of a short movie by Deaf
flmmakers. We show initial steps as to how bodies can express
themselves in diferent ways with diferent functions of what cri-
tique might be as contextually required. In refecting on these, we
discuss and speculate on the potentials of expressive bodies as a
way of moving towards critical crip methodologies in HCI [48]
more broadly. We choose the form of an experience report as a
non-anonymous venue to openly and transparently engage with
our own positionalities and the particularities that come with our
research contexts and inquiries.
2 TWO WAYS OF EMBODYING CRITIQUE
In our initial probing of an understanding of embodied critique as
relevant to the contexts of disabilities and technologies, we draw
on two case studies that allow us to speculate on the potentials of
such approaches. Both lie multiple years apart, the frst one occur-
ring in 2015 whereas the latter happened during the end of 2021.
Furthermore, both of these situations come with entirely diferent
positionalities of participants and, due to the large temporal gap
between them, partly also the researcher involved in both. Our in-
tent is to show how embodied critique can have diferent functions
even though there are shared aspects as to what can be the base
of a conversation stemming from attending to expressive bodies in
addition to language based communication.
In that we need to acknowledge the epistemological limitations
of our own embodiments. Katta is a hearing, neurodivergent, non-
binary, white researcher with chronic illnesses from central Europe
and the main person conducting the research inquiries presented
here, which all occurred in Austria. They have taken classes in
Kata Spiel and Robin Angelini
Austrian Sign Language (Österreichische Gebärdensprache – ÖGS)
regularly during the past two years. Robin is a Deaf, cis male, white
graduate student also from central Europe studying in Austria. He
is a native signer of German Sign Language (Deutsche Gebärden-
sprache – DGS1 ). Both of them have collaborated on this work
by drawing on their researched and lived experiences across and
within their respective communities. However, given the specifcity
of our own embodiments as well as those of our partners, we can
only ofer the start of a broader conversation towards critical crip
methodologies. We do not stake the claim to ofer any fnalised
insights on these matters, but rather intend to present an additional
“articulation towards Crip HCI” [48], an invitation to a conversa-
tion about how we might consider methods grounded in disability
cultures to understand technologies with disabled communities
diferently.
2.1 Taking on Emotions like Capes
The frst project we draw on was concerned with the participatory
design of technologies for the holistic wellbeing of autistic chil-
dren, called OutsideTheBox [10]. As part of this project, which ran
from 2014 through 2017, Katta was involved in eight diferent case
studies where a team of designer-researchers developed eight func-
tional prototypes based on year-long participatory engagements
with individual (or, in one case a pair of) autistic children. Our over-
all methodological approaches and practices have been described
elsewhere (e.g., [ibid]).
Here, we report on our collaboration with Dean2 , specifcally
from a session during which we aimed at evaluating the technologi-
cal artefact we previously designed and built together after a longer
break from meeting each other due to the summer holidays. While
we later had developed a participatory form of evaluation [45], this
was one of the sessions, that inspired us to do so, occurring in 2015,
when Dean was about eight years old. In building up a collaborative
relationship with Dean in the months prior, we had to frst fgure
out how to build up an environment of trust in which Dean would
feel comfortable voicing his perspective instead of trying to fgure
out what might be the specifc thing we would like to hear at a
specifc moment.
This was particularly pronounced for Dean, which we deem
likely to be based in the use of Applied Behavioural Analysis (ABA)
in his education and family. Subsequently, Dean’s parent implied
that they expected us to follow the structural approach of ABA. The
approach requires a child, to be under a near-constant therapeutic
setting (‘intense’ treatments expect 36 hours per week [8]). At the
time of our collaboration, it was not yet academically discussed, but
studies showed later that an exposure to ABA in early education po-
tentially leads to a development of Post Traumatic Stress Disorders
for individuals [21] and is predominantly assessed as “detrimental”
later in life [29]. This has lead to quite a movement of profession-
als leaving the feld of ABA [22], with some psychologists even
going as far as characterising the involved practices, particularly
those around operand conditioning, as abuse [37]. Subsequently,
we did not feel comfortable complying with the parent’s request.
1 ÖGS and DGS are, in contrast to the shared spoken language between Austria and
Germany, entirely diferent languages, even from diferent language families.
2 We have altered his name to protect his privacy in this publication.
Expressive Bodies
Yet, the daily use of ABA still infuenced the communication be-
tween design-researchers and Dean, requiring a base of trust and
a constant, fairly explicit encouragement towards being critical
or even just silly. In addition with the long break between prior
engagements and the diferent context of use of Dean’s technology
away from interactions with us towards use in a family context, we
were worried about how we could encourage Dean to voice critical
feedback, where appropriate.
Figure 1: Interaction with Dean during the evaluation ses-
sion while talking about the technology with the artefact
(to the left) and while playing out emotional responses to
the artefact
Hence, about a month before the session, we had organised a
social outing during which Dean and Katta visited a movie theatre
to watch the movie “Inside Out” (Pixar, 2015). Inspired by the fve
emotions in the movie (Joy, Sadness, Anger, Fear, Disgust), we
supplied fve chairs with fve coloured cloths as props. We further
provided three diferent scenarios that were familiar to Dean. He
could pick any emotion for each scenario and show us how he
would interact with his object in that context. Through that, we
could identify core emotions afecting the experiences Dean had
with the artefact.
In embodying these emotions, Dean became much more expres-
sive and direct in his assessment of the technology. While during an
early conversation in the session, the comments Dean made were
somewhat descriptive and his motions more illustrative of what you
could see (cf. Figure 1 on the left), but less assessing or critiquing
the object or the interactions he had with it. Using the chairs and
using his body to express diferent aspects the interaction could
take on, matching specifc scenarios (i.e., family, school, friends)
to distinct emotions and playing out their reactions made Dean
loosen up and share more insights into his nuanced assessment
of his artefact, that was context dependent and layered. Whereas
previously, he presented the object in a matter-of-fact way as global
“good”, taking on the emotional capes, in a way, encouraged him
to be more expressive with his body in articulating these critiques.
This is further illustrated in the close and animated interaction
between Dean and Katta shown to the right in Figure 1).
ASSETS ’22, October 23–26, 2022, Athens, Greece
This physical closeness was accompanied and potentially even
fuelled by the simultaneously established aesthetic distance through
acting things out [19]. Taking on a personifed embodiment of an
emotion and playing it through creates this distance and, with it,
creates a space of plausible deniability. In light of insecure power
positions and the need for re-establishing previously held relation-
ships which had a precarious nature at best, Dean could enter a
space where he could express critique through his body while also
having the fallback option of declaring this ‘just play’ or ‘a joke’ in
case we would respond negatively to his articulations.
Conceptually, this is supported by the notion of a surrogate body
position [42], one where Dean’s body becomes the expression of the
emotive potential. This means, the self and the embodied persona
(in this case, the specifc emotion with their known characteristics)
create a new melange of critical potential that can be abandoned
at a moment’s notice in case this becomes necessary due to the
social circumstances turning out to be precarious or unsafe for
whatever reason. Through that, this procedure ofers an option
for establishing the trust and safety net that is necessary when
power dimensions are complex and fraught. Or, phrased a bit more
bluntly: if you are used to critique being discouraged, taking it up
again needs to happen in a structural form that allows for plausi-
ble deniability to at least partially remove the stress this form of
communication tends to bring along in your everyday experiences.
2.2 Acting out Technology
Our second case revolves around the notion of sign language avatars
and the critique from Deaf individuals. Particularly, native signers
are already used to utilising their bodies more in language based
interactions, including literally embodying diferent perspectives
in constructed dialogues (i.e., the reporting of a dialogical situation)
[31]. However, this is an implicit language feature that is applied
semi-automatically and with little explicit refection on the topic
discussed (akin to Schön’s concept of refection-in-action). By de-
liberately thinking through making diferent choices in expressing
critical perspectives through one’s body, signers enter a state of
actively refecting on their critique beyond the direct relationality
to language (akin to refection-on-action [39]).
The topic of sign language avatars is a highly controversial one
within Deaf communities globally [6] as well as locally [11]. Deaf
representatives argue that sign language avatars reduce the com-
plexity of both written and signed languages, potentially contribut-
ing to language deprivation for younger signers or those acquiring
sign languages later in life. Further, they are less likely to understand
the register of communication required for specifc audiences and
only operate one-way from written to signed content and should
be carefully used in specifc contexts only [20]. Recent research by
Quandt et al. illustrates further that native signers require a higher
degree of quality regarding the overall motion capabilities of sign
language avatars to be sufciently acceptable for them [34]. Addi-
tional research suggests, that Deaf populations prefer if language
parameters for avatars difer from human signing in some aspects,
for example signing speed and timing [1].
Methodologically, within HCI, there are specifc recommenda-
tions as how to include Deaf communities in research on sign
language avatars. Kipp et al. suggest focus group interviews and
ASSETS ’22, October 23–26, 2022, Athens, Greece
online surveys [18]. However, group dynamics in these research
settings and the often heavily text-focused modes of online surveys
come with their own problems and exclusions, privileging people
who feel comfortable to provide feedback in groups. Surveys, even
with questions signed, come with the issue of providing limited op-
tions for answers, which often still have to be provided as text, and,
subsequently, critical nuance. Open questions are only available to
those who feel confdent in using written English (or, as is the case
with our reference, German), which to many native signers is a
language acquired later in life or those with sufcient technological
savvy to provide a link to a video responding in sign. Hence, these
approaches only present a starting point on including Deaf people
as experts, but they do not orient themselves on Deaf cultures and
linguistic styles, rather they nominally adapt existing methods (fol-
lowing a hearing logic) [47]. Further, there is often an increased
strain on organisational capacities and budgets due to the hiring
of interpreters necessary if not all participants are appropriately
fuid in a shared sign language – though Mack and Tian suggest
that researchers working with Deaf communities need to acquire
profciency in ASL (or rather, the local sign language) to adequately
understand cultural diferences and nuances) [27]. Similarly, while
we know that diferent people and diferent embodiments result in
diferent assessments [17], we need to go beyond just aggregating
those and aiming at a general view but allowing for methods at-
tending to the particular and specifc – as driven by the situated
interests in how critique towards technological artefacts might be
articulated by disabled communities themselves.
For this case study, Katta did not plan any sessions or invite
participants, rather they have been invited by a Deaf flmmaker to
observe practice sessions and the shooting of a short flm which
was conceptualised entirely in Austrian Sign Language. The story
of the movie concerns itself with the technical hubris and glitches
involved if there would be a car navigation application that included
a sign language avatar to replace spoken instructions. The flm
pokes fun at technology developers, startup cultures and the limited
capabilities of sign language avatars. Instead of using an actual
avatar, the director decided to have the avatar being embodied by a
Deaf actor.
We report here from the practice session in November 20213
during which Katta closely observed how the actor, a native signer,
actively worked on fguring out their embodiment of the avatar
along with the director’s instructions. These observations were
recorded as notes, which illustrated diferent styles of signing com-
paring the actor’s general conversational style with how they went
about embodying a sign language avatar. They then discussed their
notes with the actor (in Austrian Sign Language), which prompted
corrections and additional emphasis on certain aspects from the
actor but also aided them in refecting how their acting is perceived
and what it communicates, making subtle changes for the second
practice run, after which we discussed additional observations and
refections. That way, both Katta and the actor could proft from
the interaction with their respective interests, be they observing
embodied critique or refning the performance for the movie.
3 Everyone present was recently PCR tested (as was freely available to everyone in
Austria at the time) and fully vaccinated against COVID-19.
Kata Spiel and Robin Angelini
Figure 2: The actor signing as the avatar during a practice
session (left) and in the fnal movie (right).
From these observations and refective conversations, we iden-
tifed a strong emphasis of the perspective of both the flmmaker
and the actor to be centred around sign language avatars as they
experience them in their environment being ridiculously absurd to
some extent. The director kept on instructing the actor to reduce
their facial expressions even further until they essentially removed
this language feature, which serves both afective and grammatical
functions [35], entirely. At the same time, the actor overemphasised
mouth actions (or visemes, which are used to diferent degrees in
diferent sign languages [5]) to a point of them becoming meaning-
less and void of any information. Additionally, the actor pointed
out during the frst refection that he deliberately kept his shoul-
ders almost entirely motionless, which additionally restricted the
expressiveness of his hands and overall use of the upper body in
communication (see also, the additional stifness on the right of Fig-
ure 2 in the fnal movie compared to initial practice runs depicted
to the right). During the practice run it became clear that three-
dimensional characteristics of Austrian Sign Language, particularly
as they pertained the use of classifers [9], ended up being difcult
to translate to a two-dimensional plane especially in the context
of providing directional information. This difculty could explain
why sign language avatars rarely use classifer constructions.
Essentially, the perspective of the director and the actor to-
wards sign language avatars as they expressed them collaboratively
through embodiment and instruction as well as active linguistic
refection illustrates similar critical aspects as Krausnecker and
Schügerl identifed in their research based on diferent focus groups
with Deaf and hearing participants separately [20]. “In all focus
groups with deaf participants, it was noted that the avatar “closely
follows the German syntax”, which was described as unpleasant,
tiring, not mature, as a “gimmick”, “nice experiment” and even as
a “botch-up”” [20, p.5]. Hence, critique is available in other, more
classical settings as well and content wise, our approach comes
to similar conclusions. However, methodologically, our approach
is oriented on mutual exchange and presents a collaborative pro-
cess instead of one shaped solely by researchers. Even if it is more
Expressive Bodies
difcult to systematise what we found by doing so and make repro-
ducible for other contexts, together with Dean’s case study, we fnd
that there is potential for this concept of expressive bodies to be use-
ful in involving disabled participants by honouring their respective
cultural, personal and communicative styles and preferences.
3 EXPRESSIVE BODIES – TOWARDS A
CRITICAL CRIP METHOD(OLOGY)
Across these two case studies, we could see that turning to ex-
pressive bodies allows us to understand more about the mental
models that people hold about technologies. We aimed at illus-
trating that the embodiment of critique as a mode of attending
to disability cultures allows us to 1) feetingly create safer spaces
when the interactions with researchers might be unclear regard-
ing their implications for power dimensions; and 2) engage in a
mutually refective dialogue that has an explicitly reciprocal charac-
ter compared to the more extractive tendencies occurring in more
traditional methods for knowledge acquisition.
Turning to expressive bodies does not mean abandoning lan-
guage based methods, but using bodies actively as a way to attend
to the particular and the situated assessment of disabled peoples
along their lines of preferred communication and cultural conven-
tions. That way, they complement existing methods and present a
way of drawing on individual experiential knowledge, such as is al-
ready done by autoethnographic and frst-person research methods,
in a relational and collaborative way. We deem these particularly
useful to understand situated nuances of marginalised perspectives
on technologies in a mutually respectful manner.
In a light analogy to the distinction between expressive and in-
strumental technologies in a queering approach to HCI [24], we
suggest that there might be a distinction to make between expres-
sive and instrumental methods in the ways we assess technologies
in HCI more generally and in disability contexts specifcally. Orient-
ing ourselves towards the expressiveness of diferent bodyminds
and the associated cultural aspects means using the research en-
deavours not in an instrumental way with the intent to answer
specifc research questions that are predominantly shaped by re-
searchers and their institutional contexts but instead appreciating
and cherishing the relationships and interactions that might arise
in collaboration.
Hence, the approach is based on practices of cripping. “Cripping
spins mainstream representations or practices to reveal able-bodied
assumptions and exclusionary efects. Both queering and cripping
expose the arbitrary delineation between normal and defective and
the negative social ramifcations of attempts to homogenize human-
ity, and both disarm what is painful with wicked humor, including
camp” [4]. Our second case study illustrates how these practices
might look like, by actively making fun of and exaggerating the
embodiment of the sign language avatar, and how they lead to the
construction of insights and situated knowledges. Subsequently,
we position our approach within cripistemologies [16] and crip
technoscience [13] and hope that the concept of expressive bodies
can function as an additional “articulation” towards what might at
some point be called Crip HCI [48].
ASSETS ’22, October 23–26, 2022, Athens, Greece
ACKNOWLEDGMENTS
This work would not have been possible without the support of
a lot of people. Among those are the original project team mem-
bers of the OutsideTheBox project, Christopher Frauenberger and
Julia Makhaeva as well as, of course, Dean, who showed us how
tender research relationships can be. Further, Katta needs to thank
Christoph Kopal, Brato Avramovic and Joanna Kinberger for let-
ting them join their movie endeavour and being patient with their
limited signing capabilities. Kathrin Gerling provided invaluable
feedback on earlier drafts of this manuscript. Finally, this work
received fnancial support from the Austrian Science Funds (FWF)
through a Hertha-Firnberg Scholarship T 1146-G.
REFERENCES
[1] Sedeeq Al-khazraji, Becca Dingman, Sooyeon Lee, and Matt Huenerfauth. 2021.
At a Diferent Pace: Evaluating Whether Users Prefer Timing Parameters in
American Sign Language Animations to Difer from Human Signers’ Timing. In
The 23rd International ACM SIGACCESS Conference on Computers and Accessibility
(Virtual Event, USA) (ASSETS ’21). Association for Computing Machinery, New
York, NY, USA, Article 40, 12 pages. https://doi.org/10.1145/3441852.3471214
[2] Christine E Ashby. 2011. Whose "voice" is it anyway?: Giving voice and qualitative
research involving individuals that type to communicate. Disability Studies
Quarterly 31, 4 (2011).
[3] Fiona Campbell. 2009. Contours of ableism: The production of disability and
abledness. Springer.
[4] Eli Clare. 2015. Exile and pride: Disability, queerness, and liberation. Duke Univer-
sity Press.
[5] Onno A Crasborn, Els Van Der Kooij, Dafydd Waters, Bencie Woll, and Johanna
Mesch. 2008. Frequency distribution and spreading behavior of diferent types of
mouth actions in three sign languages. Sign Language & Linguistics 11, 1 (2008),
45–67.
[6] World Federation of the Deaf and World Association of Sign Language In-
terpreters. 2019. WFD and WASLI Statement on Use of Signing Avatars.
http://wfdeaf.org/news/resources/wfd-wasli-statement-use-signing-avatars/
[7] Becca Dingman, Garreth W. Tigwell, and Kristen Shinohara. 2021. Interview and
Think Aloud Accessibility for Deaf and Hard of Hearing Participants in Design
Research. In The 23rd International ACM SIGACCESS Conference on Computers
and Accessibility (Virtual Event, USA) (ASSETS ’21). Association for Computing
Machinery, New York, NY, USA, Article 71, 3 pages. https://doi.org/10.1145/
3441852.3476526
[8] Sigmund Eldevik, Richard P. Hastings, J. Carl Hughes, Erik Jahr, Svein Eikeseth,
and Scott Cross. 2010. Using Participant Data to Extend the Evidence Base
for Intensive Behavioral Intervention for Children With Autism. American
Journal on Intellectual and Developmental Disabilities 115, 5 (2010), 381–405.
https://doi.org/10.1352/1944-7558-115.5.381 arXiv:https://doi.org/10.1352/1944-
7558-115.5.381 PMID: 20687823.
[9] Karen Emmorey. 2003. Perspectives on classifer constructions in sign languages.
Psychology Press.
[10] Christopher Frauenberger, Katta Spiel, and Julia Makhaeva. 2019. Thinking
outsideTheBox-designing smart things with autistic children. International Jour-
nal of Human–Computer Interaction 35, 8 (2019), 666–678.
[11] Österreichischer Gehörlosenbund. 2019. Stellungnahme zum Thema
Gebärdensprach-Avatare. https://www.oeglb.at/wp-content/uploads/2021/05/
Avatare_OeGLBOeGSDV_Stellungnahme-2019.pdf
[12] Kathrin Gerling and Katta Spiel. 2021. A Critical Examination of Virtual Reality
Technology in the Context of the Minority Body. In Proceedings of the 2021 CHI
Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21).
Association for Computing Machinery, New York, NY, USA, Article 599, 14 pages.
https://doi.org/10.1145/3411764.3445196
[13] Aimi Hamraie and Kelly Fritsch. 2019. Crip technoscience manifesto. Catalyst:
Feminism, Theory, Technoscience 5, 1 (2019), 1–33.
[14] Donna Haraway. 1988. Situated Knowledges: The Science Question in Feminism
and the Privilege of Partial Perspective. Feminist Studies 14, 3 (1988), 575–599.
[15] Dhruv Jain, Audrey Desjardins, Leah Findlater, and Jon E. Froehlich. 2019. Au-
toethnography of a Hard of Hearing Traveler. In The 21st International ACM
SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (AS-
SETS ’19). Association for Computing Machinery, New York, NY, USA, 236–248.
https://doi.org/10.1145/3308561.3353800
[16] Merri Lisa Johnson and Robert McRuer. 2014. Cripistemologies: introduction.
Journal of Literary & Cultural Disability Studies 8, 2 (2014), 127–147.
[17] Hernisa Kacorri, Matt Huenerfauth, Sarah Ebling, Kasmira Patel, and Mackenzie
Willard. 2015. Demographic and Experiential Factors Infuencing Acceptance of
ASSETS ’22, October 23–26, 2022, Athens, Greece
Sign Language Animation by Deaf Users. In Proceedings of the 17th International
ACM SIGACCESS Conference on Computers & Accessibility (Lisbon, Portugal)
(ASSETS ’15). Association for Computing Machinery, New York, NY, USA, 147–154.
https://doi.org/10.1145/2700648.2809860
[18] Michael Kipp, Quan Nguyen, Alexis Heloir, and Silke Matthes. 2011. Assessing
the Deaf User Perspective on Sign Language Avatars. In The Proceedings of the
13th International ACM SIGACCESS Conference on Computers and Accessibility
(Dundee, Scotland, UK) (ASSETS ’11). Association for Computing Machinery,
New York, NY, USA, 107–114. https://doi.org/10.1145/2049536.2049557
[19] Jeanne Klein and Shifra Schonmann. 2009. Theorizing aesthetic transactions
from children’s criterial values in theatre for young audiences. Youth Theatre
Journal 23, 1 (2009), 60–74.
[20] Verena Krausnecker and Sandra Schügerl. 2021. Best Practice Protocol on the Use
of Sign Language Avatars. https://avatar-bestpractice.univie.ac.at/en/english/
[21] Henny Kupferstein. 2018. Evidence of increased PTSD symptoms in autistics
exposed to applied behavior analysis. Advances in Autism (2018).
[22] Henny Kupferstein. 2019. Why caregivers discontinue applied behavior analysis
(ABA) and choose communication-based autism interventions. Advances in
Autism (2019).
[23] Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods
in human-computer interaction. Morgan Kaufmann.
[24] Ann Light. 2011. HCI as heterodoxy: Technologies of identity and the queering
of interaction with computers. Interacting with computers 23, 5 (2011), 430–438.
[25] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich,
and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? A
Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019.
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York,
NY, USA, Article 371, 18 pages. https://doi.org/10.1145/3411764.3445412
[26] Kelly Mack, Emma McDonnell, Venkatesh Potluri, Maggie Xu, Jailyn Zabala,
Jefrey Bigham, Jennifer Mankof, and Cynthia Bennett. 2022. Anticipate and
Adjust: Cultivating Access in Human-Centered Methods. In CHI Conference
on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22).
Association for Computing Machinery, New York, NY, USA, Article 603, 18 pages.
https://doi.org/10.1145/3491102.3501882
[27] Kelly Mack and Sophie Tian. 2020. Why Researchers Working with the Deaf
Community Should Learn ASL. CHI 2020 Workshop "Nothing About Us Without
Us – Investigating the Role of Critical Disability Studies in HCI" (2020).
[28] Jennifer Mankof, Gillian R. Hayes, and Devva Kasnitz. 2010. Disability Studies as
a Source of Critical Inquiry for the Field of Assistive Technology. In Proceedings of
the 12th International ACM SIGACCESS Conference on Computers and Accessibility
(Orlando, Florida, USA) (ASSETS ’10). Association for Computing Machinery,
New York, NY, USA, 3–10. https://doi.org/10.1145/1878803.1878807
[29] Owen McGill and Anna Robinson. 2020. “Recalling hidden harms”: autistic
experiences of childhood applied behavioural analysis (ABA). Advances in Autism
(2020).
[30] Robert McRuer. 2006. Crip theory: Cultural signs of queerness and disability. NYU
press.
[31] Melanie Metzger. 1995. Action in American Sign Language. Sociolinguistics in
deaf communities 1 (1995), 255.
[32] Damian EM Milton. 2014. Autistic expertise: A critical refection on the production
of knowledge in autism studies. Autism 18, 7 (2014), 794–802.
[33] Leah Lakshmi Piepzna-Samarasinha. 2018. Care work: Dreaming disability justice.
arsenal pulp press Vancouver.
[34] Lorna C Quandt, Athena Willis, Melody Schwenk, Kaitlyn Weeks, and Ruthie
Ferster. 2022. Attitudes Toward Signing Avatars Vary Depending on Hearing
Status, Age of Signed Language Acquisition, and Avatar Type. Frontiers in
Kata Spiel and Robin Angelini
psychology 13 (2022), 730917. https://doi.org/10.3389/fpsyg.2022.730917
[35] Judy Reilly and Diane Anderson. 2002. The acquisition of non-manual morphol-
ogy in ASL. Directions in sign language acquisition (2002), 159–182.
[36] Sheila Riddell and Nick Watson. 2014. Disability, culture and identity. Routledge.
[37] Aileen Herlinda Sandoval-Norton and Gary Shkedy. 2019. How much compliance
is too much compliance: Is long-term ABA therapy abuse? Cogent Psychology 6,
1 (2019), 1641258.
[38] Zhanna Sarsenbayeva, Niels van Berkel, Eduardo Velloso, Jorge Goncalves, and
Vassilis Kostakos. 2022. Methodological Standards in Accessibility Research
on Motor Impairments: A Survey. ACM Comput. Surv. (may 2022). https:
//doi.org/10.1145/3543509 Just Accepted.
[39] Donald A Schön. 1986. The Refective Practitioner: How Professionals Think in
Action. Taylor & Francis.
[40] Katta Spiel. 2021. The Bodies of TEI – Investigating Norms and Assumptions in
the Design of Embodied Interaction. In Proceedings of the Fifteenth International
Conference on Tangible, Embedded, and Embodied Interaction (Salzburg, Austria)
(TEI ’21). Association for Computing Machinery, New York, NY, USA, Article 32,
19 pages. https://doi.org/10.1145/3430524.3440651
[41] Katta Spiel. 2022. Transreal tracing: Queer-feminist speculations on disabled
technologies. Feminist Theory 23, 2 (2022), 247–265.
[42] Katta Spiel and Kathrin Gerling. 2019. The Surrogate Body in Play. In Proceedings
of the Annual Symposium on Computer-Human Interaction in Play (Barcelona,
Spain) (CHI PLAY ’19). Association for Computing Machinery, New York, NY,
USA, 397–411. https://doi.org/10.1145/3311350.3347189
[43] Katta Spiel and Kathrin Gerling. 2021. The Purpose of Play: How HCI Games
Research Fails Neurodivergent Populations. ACM Trans. Comput.-Hum. Interact.
28, 2, Article 11 (apr 2021), 40 pages. https://doi.org/10.1145/3432245
[44] Katta Spiel, Kathrin Gerling, Cynthia L. Bennett, Emeline Brulé, Rua M. Williams,
Jennifer Rode, and Jennifer Mankof. 2020. Nothing About Us Without Us:
Investigating the Role of Critical Disability Studies in HCI. In Extended Abstracts
of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu,
HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY,
USA, 1–8. https://doi.org/10.1145/3334480.3375150
[45] Katta Spiel, Laura Malinverni, Judith Good, and Christopher Frauenberger. 2017.
Participatory Evaluation with Autistic Children. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems (Denver, Colorado, USA)
(CHI ’17). Association for Computing Machinery, New York, NY, USA, 5755–5766.
https://doi.org/10.1145/3025453.3025851
[46] Kate Stephens, Matthew Butler, Leona M Holloway, Cagatay Goncu, and Kim
Marriott. 2020. Smooth Sailing? Autoethnography of Recreational Travel by a
Blind Person. In The 22nd International ACM SIGACCESS Conference on Computers
and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing
Machinery, New York, NY, USA, Article 26, 12 pages. https://doi.org/10.1145/
3373625.3417011
[47] Amelie Unger, Dieter P. Wallach, and Nicole Jochems. 2021. Lost in Translation:
Challenges and Barriers to Sign Language-Accessible User Research. In The 23rd
International ACM SIGACCESS Conference on Computers and Accessibility (Virtual
Event, USA) (ASSETS ’21). Association for Computing Machinery, New York, NY,
USA, Article 37, 5 pages. https://doi.org/10.1145/3441852.3476473
[48] Rua M Williams, Kathryn Ringland, Amelia Gibson, Mahender Mandala, Arne
Maibaum, and Tiago Guerreiro. 2021. Articulations toward a crip HCI. Interactions
28, 3 (2021), 28–37.
[49] Anon Ymous, Katta Spiel, Os Keyes, Rua M. Williams, Judith Good, Eva Hornecker,
and Cynthia L. Bennett. 2020. "I Am Just Terrifed of My Future" – Epistemic
Violence in Disability Related Technology Research. In Extended Abstracts of
the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI,
USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA,
1–16. https://doi.org/10.1145/3334480.3381828
 Data Representativeness in Accessibility Datasets:
A Meta-Analysis
Rie Kamikubo Lining Wang
College of Information Studies Department of Computer Science
University of Maryland, College Park University of Maryland, College Park
United States United States
rkamikub@umd.edu lwang0@umd.edu
Amnah Mahmood
Department of Mathematics
University of Maryland, College Park
United States
amahmoo1@umd.edu
ABSTRACT
As data-driven systems are increasingly deployed at scale, ethical
concerns have arisen around unfair and discriminatory outcomes
for historically marginalized groups that are underrepresented in
training data. In response, work around AI fairness and inclusion
has called for datasets that are representative of various demo-
graphic groups. In this paper, we contribute an analysis of the
representativeness of age, gender, and race & ethnicity in accessi-
bility datasets–datasets sourced from people with disabilities and
older adults—that can potentially play an important role in miti-
gating bias for inclusive AI-infused applications. We examine the
current state of representation within datasets sourced by people
with disabilities by reviewing publicly-available information of 190
datasets, we call these accessibility datasets. We fnd that acces-
sibility datasets represent diverse ages, but have gender and race
representation gaps. Additionally, we investigate how the sensitive
and complex nature of demographic variables makes classifcation
difcult and inconsistent (e.g., gender, race & ethnicity), with the
source of labeling often unknown. By refecting on the current
challenges and opportunities for representation of disabled data
contributors, we hope our efort expands the space of possibility
for greater inclusion of marginalized communities in AI-infused
systems.
CCS CONCEPTS
• Human-centered computing → Human computer interac-
tion (HCI); Accessibility; • Social and professional topics →
People with disabilities; Age; Gender; Race and ethnicity.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544826
Crystal Marte
College of Information Studies
University of Maryland, College Park
United States
cmarte@umd.edu
Hernisa Kacorri
College of Information Studies
University of Maryland, College Park
United States
hernisa@umd.edu
KEYWORDS
AI FATE; datasets; inclusion; diversity, representation; accessibility;
aging
ACM Reference Format:
Rie Kamikubo, Lining Wang, Crystal Marte, Amnah Mahmood, and Hernisa
Kacorri. 2022. Data Representativeness in Accessibility Datasets: A Meta-
Analysis. In The 24th International ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM,
New York, NY, USA, 15 pages. https://doi.org/10.1145/3517428.3544826
1 INTRODUCTION
As AI-infused systems1 become ubiquitous, ensuring that they
work for a diversity of groups is vital [29, 56, 108]. Performance
disparities in these systems could lead to unfair or discriminatory
outcomes for historically and culturally marginalized groups, such
as on the basis of gender, race, or disability [12, 18, 44, 149, 162, 172].
One fundamental source of disparities is the lack of representation
in datasets used to train machine learning models and benchmark
their performance [108, 162, 179]. A notable example comes from
Treviranus [166], where during a simulation, she found that ma-
chine learning models for autonomous vehicles would run over
someone who propels themselves backward in a wheelchair. Merely
adding training examples of people using wheelchairs did not have
the intended efect in this case; the algorithm failed with a higher
confdence [166]. Treviranus suspected ‘backward propelling’ was
still an outlier.
In this important discussion on AI fairness and inclusion, ten-
sions around data representativeness involving disability [60, 79,
118] have also arisen. Data sourced from accessibility datasets can
help AI-infused systems work better when deployed in real-world
scenarios, both for assistive and general-purpose contexts [29, 75,
169]. However, privacy and ethical concerns are especially pro-
nounced in this community, as disclosure of disability can pose
risks associated with re-identifcation and further discrimination
e.g., for one’s healthcare and employment [169, 179]. People who
have distinct data patterns, like in the case of disability, are also
more susceptible to data abuse and misuse [1, 60, 167]. In addition,
1A term used by Amershi et al. , 2019 [4] to indicate “systems that have features
harnessing AI capabilities that are directly exposed to the end user.”
ASSETS ’22, October 23–26, 2022, Athens, Greece
even if AI-infused systems are trained with diverse data, this does
not inherently challenge the power structures in which these sys-
tems are embedded, which may be the actual source of harm and
marginalization for disabled people [7]. For example, a more equi-
table AI-infused system for diagnosing autism does not necessarily
correspond to greater well-being of autistic people, because it may
cement the power that medical institutions have to diagnose and
gatekeep [7].
We contribute to these discussions via our exploration of repre-
sentation in accessibility datasets, which reveal nuanced patterns
of representation and marginalization along intersectional lines. In
this work, we conducted a metadata analysis of existing accessibil-
ity datasets (1984-2021, N=190) spanning multiple communities of
focus and data types to understand the representation and report-
ing of demographic attributes including age, gender, and race &
ethnicity of data contributors. We used the publicly available docu-
mentation and resources of these datasets to explore the potential
opportunities and limitations for increasing data representative-
ness.
Our analysis shows mixed results for diverse representation of
age, gender, and race & ethnicity. For age, we found that older adults
are particularly well-represented, but this did not apply across all
communities of focus (with Autism, Developmental, and Learn-
ing communities being notable exceptions). Gender representa-
tion skewed towards men/boys being more represented overall but
varied widely by community of focus. We also found that well-
documented structural marginalization in certain communities are
refected in accessibility datasets. For example, women/girls are
underrepresented in Autism datasets, corresponding to existing
diagnosis gaps [55, 130]. Marginalization is further embedded on a
meta level, such as the case of binary categories for gender classif-
cation in the collection and reporting of gender data within datasets.
Furthermore, we did not fnd consistent norms for reporting data,
with the lack of standardized documentation, evolving practices,
and variability of categories used across age, gender and race &
ethnicity.
The contributions of this work are 1) a systematic examination of
whether those sourcing data from the disability community are suc-
ceeding in representing diverse demographics, via an intersectional
analysis along the axes of age, gender, and race & ethnicity as well
as a meta-analysis of reporting methods; 2) codes of 190 existing
accessibility datasets annotated with demographic metadata 2 ; and
3) connections to larger conversations about the implications of
representation, data stewardship, and epistemological challenges
of data collection. We contend that data representativeness must
be analyzed contextually using a critical lens, to accurately assess
the potential and implications of greater inclusion of marginalized
communities in AI-infused systems.
2 RELATED WORK
Sociocultural diversity has received attention in a wide range of
disciplines, such as encouraging gender or ethnic diversity in teams
or communities [21, 41, 74], with diferent concepts of diversity
applied in research and applications [159]. More so, AI research has
2 Data codes available at https://www.openicpsr.org/openicpsr/project/174761/version/
V1/view.
Kamikubo et al.
adopted diversity considerations deeply in the ongoing challenge
of responsible and ethical AI [24, 42, 113]. Much conversation has
been associated with the concepts around balanced representation
of sub-groups (e.g., equal participation of racial sub-groups within a
focal group) [47]. A growing number of studies have explored bias
and performance disparities of AI systems concerning representa-
tion [38, 108], especially infuenced by demographic attributes like
age [36, 97, 124], gender [18, 83, 142, 162], race [18, 96], socioeco-
nomic status [34], and disability status [56, 179]. Often such eval-
uations found the source of concerns as the under-representation
of certain demographic groups in the training data underlying pre-
dictive and inferential algorithm [108, 162, 179], calling for action
to create more balanced datasets across diferent demographics. In
response, we have seen eforts like constructing image datasets
balanced in race, gender, and age (FairFace dataset [80]) or text
corpora with gender-balanced labels (GAP [175]).
In support of the current discourse around diversity in AI data,
researchers have argued that datasets sourced from people with
disabilities and older adults can play an important role [75, 79, 118]
such as improving speech recognition with stammering data [40]
and object recognition with photos taken by blind people [75].
Calls for action from this community often center around includ-
ing disability in AI fairness discussions as it pertains to model
performance, data excellence, and privacy [48, 77, 126, 168]. In-
creasing disability representation, however, is complex; there are
myriads of challenges in collecting and sharing datasets from this
group [1, 143]. Consent and disclosure can be problematic regard-
ing sensitive disability status. Ethical concerns also arise given that
datasets collected to mitigate AI bias for people with disabilities
can be used against them by detecting their disabilities, leading
to further discrimination risks [118]. There are also existing so-
cial biases and stereotypes refected in data representing disability
(e.g., [63, 70]), which may produce AI-infused systems that reinforce
greater harms and marginalization of people with disabilities [7].
Eforts aiming to increase inclusion thus need to be carefully con-
sidered [163].
To recognize the opportunities and limitations of accessibility
datasets in the conversation of diversity in broader AI, we frst
need to understand the current status of representation in acces-
sibility datasets. Prior work investigating issues associated with
diversity in AI datasets has mostly focused on examining diferences
in model performance across pre-defned demographic attributes
to draw implications for diversity [18, 34, 162]. This often leaves
inquiries about the benefts and appropriate implementation of
diversity in data unanswered [47], except for a few exceptions (as
shown in Table 1) that explicitly analyzed datasets or issues related
to datasets in terms of demographic representation like gender
and other sociocultural attributes (e.g., language) to explore the
root causes of bias and misrepresentation. These studies concluded
that such AI datasets (often image datasets) are skewed towards
certain demographics, uncovering under-representation of older
adults [109, 128], darker-skin, and females [109, 185], and lack of
geographical diversity [148].
While representation has been discussed broadly across HCI and
accessibility [1, 100] or within specifc communities [114, 138], we
have only seen a few studies analyzing representation and charac-
teristics pertained to AI training datasets in related work [15, 82].
Data Representativeness in Accessibility Datasets: A Meta-Analysis ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Prior work on analysis of broader AI and accessibility datasets with varying sample sizes.

Data # of Datasets Age Gender Race Skin Color Geography Sociocultural

Accessibility
Bragg et al. [15] Sign Language Datasets n=NA •
Kaushal et al. [82] Clinical Image Datasets n=74 •
Broader AI
Dodge et al. [39] C4 Webtext Corpora n=1 • •
Merler et al. [109] Face Image Datasets n=7-8 • • •
Park et al. [128] Face Image Datasets n=92 •
Scheuerman et al. [142] Face Image Datasets n=92 • •
Shankar et al. [148] Open Images, ImageNet n=2 •
Yang et al. [185] ImageNet n=1 • • •

They are yet constrained to very specifc tasks and applications. Ad-
ditionally, discussions of biases against people with disabilities are
found to be manifested in complex ways that require intersectional
attention [63, 150]. This research complements prior work, by an-
alyzing existing accessibility datasets across the communities, to
encourage holistic, societal implications for data representativeness
including people with disabilities and older adults.

3 METHOD
Our aim is to conduct a broad investigation of what and how de-
mographic attributes are represented in accessibility datasets—not
only in terms of disability representation but also age, gender, and
race. To this end, we leverage a recently compiled collection of
accessibility datasets, sourced from people with disabilities and
older adults. We analyze any available information on the data
contributors’ demographics in associated academic publications,
sharing sites, and documentation. Here, we discuss the dataset col-
lections, explain our coding and analysis approach, and refect on
our method and limitations. Refecting on author positionality, we
note that this research was conducted by Asian, Afro-Latina, and
white scholars, four of whom identifed as women, one identifed as
non-binary, and two identifed as disabled. Research in accessibility
ranged from frst year grad students to a professor who has been
publishing accessibility research for about thirteen years.

3.1 Accessibility Datasets in Our Collection

Recently, Kacorri et al. (2020) launched a data surfacing repository,
called IncluSet, as a result of putting together a collection of datasets
sourced from people with disabilities and older adults that were
manually located over a multi-year period [76]. An underlying
promise of these datasets is their potential for training, testing, or
benchmarking machine learning models. The work was later ex-
tended to investigate the risks and benefts of collecting, reporting,
and sharing accessibility datasets, analyzed in terms of 10 commu- Figure 1: Distribution of accessibility dataset count across all
nities of focus, 7 data formats, and 3 data access methods [79]. We communities of focus (a) and data types (b).
leveraged the accessibility datasets (1984-2021, N=190) included in
the existing collection of IncluSet and their groupings (i.e., commu-
nities of focus) as the basis for our investigation. Figure 1a illustrates objects taken by blind people [88], eye-tracking data from autistic
the distribution of the datasets across the communities of focus. children [43], and activity data from older adults [91].
The datasets, including their annotations, are of diferent data types, Identifying publicly available documentation for these datasets
as shown in Figure 1b. For example, there are voice recordings of often depended on how they were shared. Out of 190 datasets,
people with speech impairments [25], video recordings of Deaf about 84 can be downloaded directly and 41 can be accessed upon
signers [69], text written by people with dyslexia [134], stroke ges- request—e.g., through a webpage from the dataset creators or an
tures by people with motor impairments [171], photos of everyday online repository with a summary of the dataset. Summaries vary
ASSETS ’22, October 23–26, 2022, Athens, Greece
highly from a few lines to detailed descriptions of the contents
of the dataset and how it was collected. Even though none of the
datasets had explicitly adopted standardized documentation such
as datasheets for datasets [54], some followed a systematic docu-
mentation dictated by the platforms where the datasets were stored
such as Synapse.org. Associated academic publications were often
referred to in the web documentation to link more detailed informa-
tion about the data collected, though these sources did not always
come with consistent information such as the number of data con-
tributors, which could be easily updated on the web documentation.
Dataset downloads sometimes came with relevant summary fles,
including a spreadsheet listing demographic information about
people represented in the data. The remaining 65 datasets in the
collection did not include any sharing intent with no sources avail-
able other than their academic publications. We still include these
datasets in our analysis, in accordance with prior work analyzing
accessibility datasets [77, 79].
3.2 Manual Coding and Analysis
We conducted an exploratory analysis where our formulation of
what-to-code was based on (a) whether demographic information
about the data contributors is available, (b) how is it collected and
reported, and (c) how are accessibility datasets distributed among
demographic groups within communities of focus.
Specifcally, beyond the existing codes in Kamikubo et al. [79],
we extracted information related to demographic attributes follow-
ing prior surveys on datasets and studies in accessibility and AI
that examined diversity and representation (summarized in Sec-
tion 2.3). A total of three annotators (a PhD student in Information
Studies, a Masters’ student in HCI, and an undergraduate student
in Math) were involved in the process, where at least two reviewed
the documentation for each dataset and discussed to correct any
disagreement and error. They had diferent levels of familiarity with
accessibility and AI. We extracted the following diversity-related
information from the documentation, when available:
Age. We note how any age-related information is obtained (e.g.,
self-reported, inferred, or unknown), reported (e.g., individual
level, year of birth, age bins, and/or aggregate statistics), and
shared (e.g., a separate fle). We only calculate aggregated sta-
tistics from individual-level data when reporting fndings and
plotting distributions.
Gender. We note the labels used (e.g., sex, gender), if any; the
categories used; the number of data contributors that belong to
the categories used; and how metadata was obtained (e.g., self-
reported or inferred) and shared (e.g., spreadsheet or publication).
In response to concerns raised by trans and information science
scholars that the sex/gender distinction can invalidate trans and
intersex identities while veiling the socially constructed nature
of sex categories, for this paper we use the term “gender” to refer
to discussions of characteristics of data contributors (that may
be labeled by researchers as either gender or sex) [46, 142, 146].
Race and ethnicity. Race is a multidimensional and complex
concept, not a singular, biological construct with distinct limits
into which people can be classifed. Alone, race and ethnicity, do
not reveal much about an individual’s experiences. As race and
ethnicity can be viewed through multiple socially constructed
Kamikubo et al.
lenses [17], we started with broad coding techniques to identify
any information that pertains to these demographic attributes,
including potential ethnic and cultural descriptors like geogra-
phy and language. Manly [104] suggests that these attributes are
proxies for or interrelated with unexamined variables, such as
education and socioeconomic status. To better our understand-
ing of race/ethnicity, it is central to deconstruct and examine
the confounding infuences of ethno-racial factors. We note any
categories used to refer to data contributors’ racial groups, such
as those defned in the census [19] and group ethnic and cultural
metadata like nationality, geography, and language under other
sociocultural information. Based on the metadata identifed, we
update the annotation scheme by specifcally going over how
this information is obtained and shared. Metadata related to
education included information in terms of how it is obtained,
reported, and shared; language included information on dialect
and skills earned which may interact with education; geography
included information on data contributors’ birthplaces and the
recruitment location; and other information such as nationality
or socioeconomic status when available.
3.3 Refections on Limitations
Annotation consistency. Annotation tasks are notably difcult, espe-
cially if they involve manual inspection of large data requiring par-
ticular skills and knowledge. Given that we inspected both dataset
documentations and scholarly articles from various publication
venues across many research disciplines and sub-disciplines (e.g.,
Linguistics, Acoustics, Physiology, Computer Vision, HCI, Accessi-
bility), it was unavoidable to go through a messy process to correct
errors and disagreement in our codes. The annotators’ varying
levels of familiarity with accessibility and AI were also sources of
difculty. This is not a surprise. Even similar annotation tasks that
were more limited in scope (i.e. within the feld of accessibility),
were characterized as “challenging and efortful” [100]. To address
the challenges, as the coding process initially started with two an-
notators (PhD and undergraduate level), we invited a third member
(Master’s level) to have a detailed pass. The PhD student took a
fnal pass to ensure that the annotations were agreed upon at least
by two annotators.
We also experienced difculty in programmatically extracting
demographic-related metadata. This often created disparities among
the annotators in identifying the relevant information from the doc-
umentation. We did not fnd a consistent, standardized method. For
example, some methods we used included manually reviewing web
documentation that provided summary statistics in writing [135]
or table [2] formats; downloading fles containing participants’
demographic data (e.g., age, gender) together with collected data
points [164] or a separate csv fle on participant demographics [6];
or extracting metadata from flenames [65]. Without standardized
documentation and evolving practices, whether datasets contained
demographic-related metadata was often unknown prior to down-
loads. In addition, without proper explanation of the labels used for
demographic categories, such as in one dataset [6] that provided
a supplementary spreadsheet with a label ’1’ under the Race col-
umn for each participant, we could not fnd the meaning of this
information.
Data Representativeness in Accessibility Datasets: A Meta-Analysis
Lack of documentation. As discussed in the Results, information
on age, gender and race/ethnicity was in many cases sparse. When
available, it was often unclear how the demographic-related meta-
data was obtained. Thus, we could not verify the source of classifca-
tions (such as for gender). Few datasets explicitly documented that
the reported information was e.g., “according to self-reports” [191].
Even fewer made inferences on these demographics e.g., “using
proprietary classifers” [177] or “based on visual inspection” [151];
typically these inferences were employed on data collected over
the web. Specifcally, we observed that three datasets indicate esti-
mations on data contributors’ age; all three are solicited from user
interactions with a web search engine with users’ age reported
being “over the age of 40 years inferred from their date of birth as
reported at registration to Bing” [189] or “inferred using proprietary
Bing classifers”[177, 178].
White et al. [177, 178] employed a similar approach for gender.
Whereas Shi et al. [151, 152] determine the gender of individuals
by visually inspecting sign language videos from YouTube and
the signers’ social media; they used the code “Other” for videos
including people whose gender was deemed unknown or where
there were multiple signers. While we have included the codes for
these datasets in our collection as a reference for future researchers,
we don’t include them in our analysis of ’reported’ demograph-
ics; inferences can be inaccurate, perpetuate bias, and perpetuate
exclusion (e.g. via binary classifcation of nonbinary individuals).
None of the datasets in the collection inferred or estimated de-
mographics that pertain to race/ethnicity or other metadata related
to nationality, geography, language, and education. Yet, this part of
our analysis is the weakest one as it solely relies on a small num-
ber of datasets where the race/ethnicity information was specif-
ically ‘reported’; the majority (8) came from US institutions and
one from UK even though the institutions of data stewards in the
collection spanned across 42 countries from Asia, Africa, North
America, South America, Europe, and Australia. Thus, our analysis
of this demographic is inherently limited. Only limited reporting of
race/ethnicity may be due to a number of factors, such as diferences
in census reporting among Western and non-Western countries, a
prevailing consensus that racial designations do not identify geneti-
cally distinct populations, and the likelihood of misuse (e.g., privacy
risks for disabled people) [84, 122, 147]. Cooper et al. suggest that
“the correlation between the use of unsupported genetic inferences
and the social standing of a group is glaring evidence of bias and
demonstrates how race is used both to categorize and to rank order
subpopulations.” [31]. However, since federal and state legislation
in the US have established evident discriminatory practices against
African Americans, Hispanics, Asians, and other groups, racial cat-
egorization can be utilized to refect intersectional gaps that are
a product of racial stratifcation practices. Thus, considering the
sociocultural and political contexts of diferent regions to further
understand the decision to utilize racial categories is critical. We
did not see within the scope of this paper a systematic way to re-
port the somewhat sparse metadata across codes related to data
contributors’ nationality, geography, language, and education and
tie them to sociocultural and political contexts of diferent regions.
Nonetheless, we include these codes in our annotations for future
reference.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Non exhaustive collection. One of the main limitations of this
work remains the fact that the list of datasets in the collection is not
exhaustive. While somewhat systematic, the identifcation of these
samples is itself noisy and prone to cascading biased decisions from
the researchers collecting them and those that opt/know to include
their datasets in the IncluSet repository. The lack of inclusion crite-
ria related to when these datasets were introduced or whether they
are currently in use and to what extent, could lead to systematic
misalignment between current eforts and past trends. This is exac-
erbated by the fact that many datasets that are actually employed
currently in commercial AI-infused products are not accessible
for this type of analysis; representation of diferent demographic
groups could be perhaps deduced via biased performance results
(e.g., [18]) but that is beyond the scope of this work. Thus, any
insights from our analysis may not be generalizable beyond the
research community.
4 RESULTS
Of 190 datasets whose publication and documentation we reviewed,
the most commonly found types of demographic-related metadata
are age (46.8%) and gender (54.2%), followed by few datasets re-
porting race (4.7%) and education (12.1%). We fnd that 71 datasets
(37.4%) did not include any information related to the aforemen-
tioned types of metadata. These numbers difer from publications
that also focus on health, wellness, accessibility, and aging, where
few share data; when looking at 792 HCI studies, Abbott et al. (2019)
found a distribution of 69.7%, 67.3% and 6.6% on age, gender, and
ethnicity, respectively [1]. This diference could be due to tensions
inherent in collecting “sensitive attribute data” [1, 11, 16] and con-
cerns related to participant consent and re-identifcation risks [1]. A
similar trend is seen among available metadata with respect to how
others can access the datasets. Among those that are not publicly
shared, 69.2% reported at least one of the demographics, compared
to 57.1% for publicly shared and 53.7% for shared upon request.
In this section, we present our fndings surrounding such “sensi-
tive attribute data” in accessibility datasets across communities of
focus (Figure 2). To better understand the current status in terms of
reporting and including diferent demographic groups and variables,
we focus on the following demographics: age, gender, and race and
ethnicity. In our analysis, we compare with existing categories used
to represent demographic variables in social data collection (e.g.,
racial categories in census [174]), and investigate representative-
ness within accessibility datasets.
4.1 Age
A total of 6050 people within the communities of focus contributed
data to the 89 datasets whose information on age was included.
Their weighted average age was 43.6 (std=26.3). For the remaining
of the report, statistics are reported at the dataset level (i.e. sam-
pling distribution of the mean) even though the sample size across
datasets varies highly from 1 to 990 people (mean=66.8, std=144.5).
Data on age from control groups are not included in the analysis.
4.1.1 What Is Reported. Datasets mostly reported such informa-
tion in aggregate though some (36.0%) reported age at an individual
level. Aggregate information includes minimum age (1.1%), range
(15.7%), median (1.1%), average (20.2%), or a combination (25.8%).
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 2: Proportion of accessibility datasets across all communities including metadata related to the age, gender, race,
education, or other sociocultural factors about their data contributors. Many datasets (e.g., in the Hearing group) did not contain
any metadata.
Typically, age was reported separately for target (i.e., disability)
and control groups (e.g., [45]), contributors’ gender (e.g., [170]), and
dataset purpose (e.g., training versus validation [86]). Few report on
all groups together (e.g., [22]). Data anonymization is a core com-
ponent of data management to minimize risk of disclosure while
preserving its utility for analysis [81]. However, we fnd that a
majority of the datasets did not incorporate these strategies. For ex-
ample, bucketing by age groups (e.g., 18-30, 31-45, 46-60 years [107])
was only found in 7 datasets (7.9%).
Only 5 datasets reported median and 3 datasets reported both
mean and median. More than half (58.4%) indicate standard devia-
tion, including those reporting age at the individual level for which
it can be calculated. All three, mean, standard deviation, and range,
can be found for less than half (42.7%) of the datasets (e.g., “The mean
age of the subjects was 54.9 ± 13.4 (SD) yr (range 36–70 yr)” [64]).
Meanwhile, some documentation noted only the minimum (e.g.,
“participants aged 50 or older” [180]) or the age requirement for
participation (e.g., “18 or older” [13]).
4.1.2 Why Is It Reported. Most often datasets did not specify why
the ages were obtained and reported. It could be an efect of per-
ceived norms and standards for questionnaires within the research
community, which often include age questions [68, 161]. Age is an
established variable that helps understand the general characteris-
tics of participants. Its distribution may refect the quality of data
collection and analysis [5]; not accounting for age can threaten the
generalizability of the work especially when there is a treatment ef-
fect heterogeneity in age or other factors that may covary with age
(e.g., [121]). Some datasets mention eforts to match age between
target and control groups (e.g., [26, 160]) or note age matching
as not feasible (e.g., [111]). Others mention age as a confounding
variable e.g., for early detection of Parkinson’s disease based on
touchscreen typing patterns [72]. Some datasets mentioned the
goal of including data from diverse age groups to assess age-related
decline of cognitive or mobility performance [91, 116]. For example,
in a dataset acquiring age-related pen-based performance [116],
participants were grouped based on cognition changes (’young’
for 18-55, ’pre-old’ for 56-75, and ’old’ for 75+). Grouping varies
across communities; in an attempt to build a diverse sign language
corpus, researchers binned groups as 18-35 years, 36-50 years, 51-
64 years, and 65+, rationalizing their decision based on language
transmission variability within the Deaf community [141].
Kamikubo et al.
4.1.3 Representation Across Communities of Focus. Figure 3 illus-
trates with violin plots the sampling distribution of mean age in
datasets across communities, where the white dot represents the
median, the thick gray bar in the center indicates the interquartile
range, and the thin gray line shows the rest of the distribution, ex-
cept for points that are determined to be “outliers.” Kernel density
estimations on each side of the gray lines show the distribution
shape. Wider sections indicate a higher probability that datasets
will have a mean age of the given value; the skinnier sections indi-
cate a lower probability. We note that datasets vary in their sample
size, which is not accounted for by this visualization.
We fnd that mean age in datasets difers across communities,
with some communities particularly inclining towards samples with
a certain target age (e.g., children, older adults). To better under-
stand the age representation exhibited in accessibility datasets, the
remainder of the section follows age groups discussed or referred
to in prior literature in terms of technology (e.g., ‘older adults’ as
65+, ‘oldest-old adults’ as 85+) [128], disability-related policies (e.g.,
‘children’ between 3 to 21 covered in IDEA [94]), and the commu-
nities of focus (e.g., ‘toddlers’ of 18 to 36 months in developmental
assessment [30]). Of course, variations exist across studies [154] as
there is no rigid defnition for these groupings.
Figure 3: Sampling distribution of ’reported’ mean age, which
difers across communities. Means are calculated on varying
sample sizes.
Older adults. Many accessibility datasets represent older adults.
Among the datasets that contained some form of age-related in-
formation, 48.3% included at least one older adult (65+), and 6.7%
Data Representativeness in Accessibility Datasets: A Meta-Analysis
at least one oldest-old adult (85+). The highest proportion of older
adults was in the Cognitive and Health groups, reporting at least
one older adult in 83.8% and 73.3% of their datasets, respectively.
This may not be surprising, as these groups focus on cognitive and
physical decline that can relate to age—e.g., the risk of onset of
dementia (e.g., Alzheimer’s disease) increases with older age [131].
Specifcally, the Cognitive group had datasets with the highest
mean of mean age (mean=61.7, std=12.4) which were often cross-
listed with the Mobility and Speech groups including speech or
motion data of patients with Parkinson’s disease (e.g., [71, 140]).
The oldest participant, aged 89, was reported in the Cognitive and
Health groups in the image dataset capturing daily activities of
those with episodic memory impairment [89]. Communities that
lack older adult representation are Autism, Developmental, and
Learning, refecting a broader gap in research pertaining to these
groups [66, 73, 130, 139]. This can be due to many factors; for ex-
ample, many autistic older adults experienced a severely delayed
diagnosis [102]. Many adults with learning disabilities live in in-
stitutions such as nursing and residential homes, in which they
arrive “before their 65th birthday” with “few opportunities to get
out” [165].
Children and youth. Children and youth are also represented
in accessibility datasets; about a quarter (24.7%) of the datasets
whose information on age was included contained data sourced by
at least one person younger than 18 years old. It increases to 33.7%
when including those 21 or younger, as the age criteria for study par-
ticipation is often noted as 18 or older [13, 45]. Perhaps this refects
some of the ethical challenges in collecting data from children [32]
as the process for obtaining consent, assent, or parental permission
is more complex for those under the legal age [112]. While overall
there are few datasets sourced from youth, they tend to concentrate
in the Developmental (85.7% of datasets in this group include at
least one person <18) and Learning (100.0%) groups. Datasets in the
Learning group often focus on dyslexia (e.g., [53, 115]), where diag-
nosis is critical at early ages. Data from toddlers (18 to 36 months
old) are typically seen in the Development group for the purpose of
developmental assessment (e.g., [30]). They mostly involve speech
data, sourced by stuttering children [58, 182] or late talkers [120].
The youngest reported age across all the accessibility datasets was
16 months, in a dataset sourced from autistic children [181], though
not many (33.3%) datasets reporting age in the Autism group in-
cluded those under the age of 18. The groups that lack data from
children and youth are Vision, Hearing, and Mobility. We suspect
that this is refective of the most common purpose for collecting
data such as image and video from this age group, which is to better
assess and diagnose; disabilities related to one’s vision, hearing,
and mobility have long established methods and instruments that
might not require such datasets.
Younger and middle-aged adults. When looking at younger
adults (over 18), we fnd that surprisingly, many (9) datasets with
mean age in the Autism group tend to include people between the
age of 18 and 44, with an overall mean of mean age 24.0 (std=13.8).
This is in striking contrast with the broader research on autism,
where the majority (94%) tends to focus on infants, toddlers, chil-
dren, and adolescents [73] due to a focus on early diagnosis and
intervention [117, 127]. Datasets including younger adults in this
group were often collected in the context of assistive technologies
ASSETS ’22, October 23–26, 2022, Athens, Greece
(e.g., evaluating text readability and comprehensibility via gaze fx-
ations [45, 183, 184].)) Looking further at datasets skewed towards
younger and middle-aged adults, the age range of Hearing and
Vision groups was limited, even though visual and hearing impair-
ments could be associated with older age [14, 95]. The datasets in
the Hearing and Vision groups that reported age have an overall
mean of mean age 28.3 (std=4.2) and 48.7 (std=3.6), respectively.
This can be partially explained by how these datasets were col-
lected. For example, the majority (66.7%) of datasets in the Vision
group did not include any age information; they were collected
from thousands of users via real-world applications (e.g., [57, 78]),
where user demographics may not be available or omitted due to
privacy concerns. Similarly, in the Hearing group the majority of
datasets do not include age information; they tend to collect sign
language from online sources (e.g., [92, 151]).
Diverse ages. We observe that the Language group has the
largest age variability. Among others, they include data sourced
from children with epilepsy (e.g., [160]), adolescents with language
impairment (e.g., [176]), and older adults with aphasia (e.g., [3, 35]).
Often datasets in this group come from clinical settings such as
the FluencyBank found in TalkBank [101], a shared database estab-
lished in 2002 for studying human communication. Perhaps this
collaborative efort among a wide range of disciplines could explain
the variability of datasets spanning across diferent communities
over the years. Datasets in Speech also capture diferent age groups.
Some can be found in TalkBank, including spoken phrases of older
adults with Alzeimer’s disease [105] as well as children [182] and
adults [187] who stutter.
4.2 Gender
A total of 5598 people within the communities of focus contributed
data to the 103 datasets whose information on gender was included.
Again, we include information at a dataset level even though the
sample size across datasets varies highly from 1 to 818 (mean=59.6,
std=106.6). Data on gender for the control groups are not included
in the analysis.
4.2.1 What Is Reported. Gender metadata was commonly reported
with the number of data contributors in the form of writing (e.g.,
“10 blind participants (5 female) ranging in age from 18 to 63 years
old” [9]) or table (e.g., a M/F column [6]). Of datasets reporting
such metadata, we observed that a binary classifcation was used
(female/male, women/men, girls/boys), with only one dataset in our
collection reporting data on the “other” category [49]. However,
it is difcult to draw conclusions from this alone, as few datasets
reported their method of gendering contributors. Without this, we
cannot distinguish between self-identifcation (e.g., as part of a
demographics questionnaire), or an external inference infuenced
by implicit assumptions (e.g., by the study designers or validators).
Furthermore, if participants were asked to self-identify, they may
have been limited to choosing from binary options.
4.2.2 Why Is It Reported. Similar to age being asked in standard de-
mographic questions [68], datasets often included gender informa-
tion as part of the data distribution, without specifcally describing
the goal of collecting such information.
ASSETS ’22, October 23–26, 2022, Athens, Greece Kamikubo et al.

Nonetheless, we can attempt to extrapolate the reasoning for

some datasets, especially when they contain particular data formats.
The highest presence of gender information was in datasets that col-
lected audio (66%) compared to video (27%) or image (32%). Perhaps,
this is refective of an assumption of the infuence of gender among
those working with speech data. Datasets that capture motion e.g.,
gait of Parkinson’s disease patients [170], also attempt (about 50% of
them) to account for physical measurement diferences represented
in data by using gender as a proxy.
In order to keep the study design as “unbiased” as possible, some
datasets reported that gender (and/or age) was “balanced” in the
test group (e.g., “roughly balanced for gender of the 249 participants,
52% (n= 129) were women” [141]), but eforts to balance distribu-
tion between target and control groups were much more common
Figure 4: Sampling distribution of gender representation
(e.g., [170], [125]).
across accessibility datasets. The representation gap is more
4.2.3 Representation Across Communities of Focus. Gender demo- prominent in some communities than others.
graphics vary across the world, with most countries having a fe-
male3 share of the population between 49% and 51% [137]. However,
overall, accessibility datasets that include gender information tend skew towards women has been identifed by researchers in this
to be imbalanced with men and boys (60.1%) who are more rep- community as possibly attributable to diferences in life expectancy
resented on average4 than women and girls (39.9%). This is also by gender in addition to increased risk of visual impairments with
evident in Figure 4a, which illustrates with violin plots the sampling age (e.g., macular degeneration) [59], which women are noted to
distribution of gender representation in datasets across communi- be at higher risk of than men [156].
ties of focus, where the vertical dash lines indicate the quartiles
and each side of the distribution shows kernel density estimations
for ‘women/girls’ and ‘men/boys’. This illustration also highlights
4.3 Race & Ethnicity
how the gap is more prominent in some communities than others. Race is a complex and sensitive demographic variable [52, 145].
Specifcally, we see a clear imbalance in the representation of data Only 9 (5%) accessibility datasets reported metadata on contrib-
contributors in the Autism and Developmental groups; on average, utors associated with racial or ethnic groups, typically captured
33.1% (std=8.1) and 27.9% (std=9.8) are women and girls, respectively. by demographic surveys (e.g., [19]). Modern racial classifcation
Such highly skewed representation has been actively discussed in systems construct race using both observable physical features (e.g.,
the evaluation and diagnosis of autistic children, given that boys skin color) and nonobservable characteristics such as culture and
constituted 81% of the sample of children [55]. One widely cited language [27]. Thus, ‘other’ related demographic information we
male-to-female diagnosis ratio is approximately 4:1 [51]. However, found could perhaps be utilized to draw some connections and
when the ASD participants are controlled for cognitive impairments, inferences about race, including the place of birth [23], native lan-
this number changes [85, 98, 103, 106, 138]. About 50-55% of autistic guage [72], or dialect [188]. However, in past studies they have
children are estimated to be intellectually disabled (ID) [98]. Among led to issues of forced classifcation and error [11, 123]. Therefore,
ID autistic children, the male-to-female ratio is signifcantly smaller, in this section we don’t make that connection. We report only on
at 2:1 [67]. In autistic children labeled as “high functioning”, the datasets with explicit racial and ethnic information.
existing literature points to a higher male-to-female ratio, about
4.3.1 What Is Reported. The categories we found delineating racial
6:1. Researchers have theorized an explanation for this relationship
composition were mostly ‘White’ and ‘Black’ [144], with variations
could be the tendency of (so-called) “high-functioning” autistic
of reporting them as ‘White-Caucasian’ or ‘Caucasian’ and ‘African-
females to “mask” or “camoufage” core autistic traits [90, 133]. A
American’ [160, 182, 191]. For other racial groups, data were ambigu-
growing body of evidence suggests that current diagnostic criteria
ously grouped together (e.g., “62% Caucasian, 30% African-American
for ASD may fail to account for these phenomena and the subtleties
and 10% other” [160]) or can be extrapolated by subtracting what
in behavior, leading to misdiagnosis and late-diagnosis for minority
was reported as the proportion of the ‘white’ category only [190].
gender groups (e.g., women, girls, non-binary) [87].
The use of these terms also highlight the limitations of the taxonom-
While many communities of focus portray gender disparity in
ical racial categories; ‘Caucasian’, for example, is rather discussed
their represented samples, it is not seen in the Vision group, with
as outdated and disproved [119].
the average of 50.2% (std=3.2) consisting of women per dataset.
Similar to age and gender, race was reported separately for target
According to 2018 U.S. disability statistics [186], 45.3% of visually
and control groups (e.g., [190]). Notably, one speech dataset sourced
disabled people were male, and 54.7 % were female. The slight
from stuttering children aimed at a race-matched (as well as age-
3When referring to data sourced from external collections, we follow the terminology and gender-matched) cohort of children [132]—here, both stuttering
used in their reports. and non-stuttering groups had 2 African American children and 1
4With both gender-related and sex-related categories used in our collection of datasets,
we report data for ‘women/girls’ or ‘men/boys’ combined with data for e.g., ‘female’ child of mixed racial ancestry. This was also the only dataset in the
or ‘male’. collection reporting about mixed race, although we saw an attempt
Data Representativeness in Accessibility Datasets: A Meta-Analysis
to collect data on race, including ‘Mixed’, from a demographic
questionnaire in a study on Parkinson’s disease [13].
4.3.2 Why Is It Reported. Looking at datasets whose data on race
was collected and/or reported, they are often related to medical
research associated with studies on specifc disorders. Specifcally,
they include speech samples collected from people with apha-
sia [144], Parkinson’s disease[190], Alzheimer’s disease [6], and
epilepsy [160] to study early detection of impairments underly-
ing cognitive disturbance. In medical research domains, there are
controversies around collecting data on race, raising both benefts
and risks given disparities in health outcomes established for racial
minorities [50, 62]. Concerns also lie in the taxonomy of the cate-
gories used, which have brought eforts to standardize and improve
methods of obtaining and reporting data on race [8, 50]. Recent
guidelines [50] suggest including an explanation of who identifed
participant race & ethnicity and reasons for collecting the data. We
did not fnd disclosure of the source of the classifcations among the
datasets included (e.g., self-report, observation), nor a justifcation
of why it was collected.
4.3.3 Representation Across Communities of Focus. It was hard to
distinguish the data between race and ethnicity or other sociocul-
tural information, especially when the data spans multiple concepts
and forms of classifcation (e.g., “129 of Caucasian, 14 of African
American, 2 of Hispanic, and 2 of Asian origin” [182]). For exam-
ple, in US, guidelines that inform data collection for census note
that the concept of race is separate from the concept of Hispanic
origin [173].
For the few datasets that reported data contributors’ race and eth-
nicity, the norms of how to report were highly inconsistent. Thus,
with high variability and a small sample, we could not leverage stan-
dardized methods to analyze racial group composition among the
communities of focus. The categories we saw (often in Cognitive
and Language) were associated with ‘white’ or ‘non-white’, por-
traying one group as primary over another. Mixed race was rarely
indicated, which is problematic given changes in racial categories
(e.g., in the US census) refecting racial mixture [20].
5 DISCUSSION
Our overarching goal lies in understanding the current state of
representativeness of marginalized groups in AI datasets (along the
axes of age, gender, and race & ethnicity) with a specifc focus on
disabled data contributors. This is relevant to the greater discourse
around AI, ethics, and fairness, as marginalized communities tend
to be under-represented in data [47], perpetuating cycles of ex-
clusion as technology advances even for technologies that meant
to promote inclusion such as assistive technology. We contribute
to this important ongoing discussion through our analysis of 190
accessibility datasets. Specifcally, we examine representation gaps
and trends that can potentially lead down the road to further harm
for the people who stand to be adversely afected by emerging,
potentially ubiquitous technology. In this section, we recap and
discuss the challenges and opportunities for representation while
considering directions the accessibility feld could take to carefully
include marginalized communities in AI-infused systems.
ASSETS ’22, October 23–26, 2022, Athens, Greece
5.1 Addressing Challenges and Seizing
Opportunities for Representation
Our analysis revealed unique challenges in ensuring representa-
tion of intersecting demographics in accessibility datasets. Some
representation gaps are attributable to societal and cultural norms
and biases that operate intersectionally. For example, communities
lacking older adult representation are Autism, Developmental, and
Learning. This refects not only a broader research gap on these
groups [66, 73, 130, 139] but also discrimination at the intersection
of disability and age; e.g., many autistic older adults live without
an accurate diagnosis [102]. Similarly, looking at the intersection
of disability and gender, we observe a gap for Autism, Develop-
mental, and Learning groups, where men and boys were often
over-represented. These cases can have pernicious implications
characterized not only by the communities of focus but also long
established research frameworks that propagate existing societal
marginalization, highlighting the importance of making gender-
specifc changes (e.g., diagnostic criteria for autism [37, 87]).
In annotating accessibility datasets, we also surfaced how so-
cially constructed identity categories such as race and gender are
reproduced. Similar to Scheuerman’s meta-analysis of gender in
face datasets [142], by analyzing information such as reasons for
reporting/data collection and labels used for metadata categories,
we contribute a sociological meta-examination through which the
research and data collection process itself can be analyzed for
bias. For example, we found that the notion of a gender or sex
binary was not explicitly challenged in our collection; only one
dataset reported data on the “other” gender category. This may
have downstream efects in shaping machine learning model de-
sign and subsequent problems/contexts—for example, in binary
gender classifcation, which may harm nonbinary communities
through technology-enabled misgendering [61].
We also found that there is very little reporting of how identity
labels were associated with data contributors, whether through self-
identifcation or external assumption (e.g., via preformed binary
categories). We recommend greater transparency in disclosing these
aspects of the data collection process, and for gender in particular, to
include nonbinary, self-describe, and prefer not to disclose options,
as recommended in the related literature [158].
At the same time, we acknowledge the implementation chal-
lenges that may need to be addressed to support transparency—e.g.,
how to produce a set of questions which do not elicit informa-
tion leading to unintentional misuse or unwanted societal biases
for data contributors. We emphasize that careful refection on this
process is needed on the part of researchers who are collecting
and reporting contributor data, including implications of use (e.g.
surveillance) and any potential harms enacted by power structures
through the systems we build. Aligning with recent research [110],
we recommend an examination and contextualization of data rep-
resentativeness grounded in political, economic, and socio-cultural
lenses, integrating insights from scholars in felds such as critical
disability studies [28], trans/gender studies [157], and histories of
social movements [136] into an analysis of power relations. As an
example, one could draw from recent work by disability studies
scholars examining the context the data is collected in (i.e., for
AI systems vs for visibility and activism) and how representation
impacts are also context-dependent [93].
ASSETS ’22, October 23–26, 2022, Athens, Greece
5.2 Developing Participatory Approaches to
Data Stewardship
This challenge of partitioning the pool of accessibility datasets into
sub-communities was very real in our analysis, as the groupings
that we opted for may not necessarily refect the identities of in-
dividual data contributors. Recent work exploring challenges for
collecting disability data suggests the voices of contributors to be
refected and provides best practices to ask about disability sta-
tus [10]. Perhaps, to mitigate harms experienced by those from
marginalized communities who are misclassifed, we can extend
this approach to other categories such as race and gender. Specif-
ically, we urge researchers to come up with approaches for more
meaningful engagement of data contributors in the data steward-
ing process. Echoing Shneiderman’s motto [153], we recommend
“researchers in the loop, disabled contributors in the group”.
One way we could go about this is to employ participatory ap-
proaches to the data collection lifecycle in which users have the
opportunity to enact their values in how their data is collected,
maintained, shared, and interpreted in and out [33, 99]. Of course,
this would require careful consideration of the many moving pieces
in the Fairness, Accountability, Transparency, and Ethics (FATE)
landscape both in terms of parties involved as well as exchange and
access mechanisms; Bragg et al. [15] provide a wonderful starting
point for this discussion in the context of the Deaf community. For
example, to avoid inadvertently extractive approaches, and aligning
with recent literature, we recommend meaningfully compensating
participants for their work as data contributors [155]. In this vein,
we also recommend developing long term relationships with data
contributors and their communities (where possible) to facilitate
sustainable and mutually benefcial collaboration, especially when
designing and evaluating AI-infused systems that use contributor
data [155, 163]. Disability community-led initiatives can help con-
centrate research eforts on those most likely to have a positive
impact; the idea generation phase may be particularly fruitful when
rooted in frst person lived experience (e.g. as provided in [129]).
5.3 Addressing Epistemological Implications in
Future Work
We encountered epistemological limitations at various stages in the
annotation and analysis process. One such limitation is the extent to
which strong claims can be made about overall representativeness,
due to the lack of reporting and global statistics for disability, age,
gender, and race. In addition, our fndings are intrinsically linked
to existing sociocultural contexts and hierarchies. Our analysis of
accessibility datasets showcases these epistemological limitations.
By acknowledging these limitations, we hope to spark conversa-
tions on the inclusion of marginalized communities in AI-infused
systems and its myriad challenges. In future eforts, we recommend
the following for broader research implications:
Exploration of disabled people’s concerns around representation.
Increasing representativeness may not always be benefcial; it may
perpetuate injustice as extensions of existing systems of oppression
and power. As explored in the previous section, it is vital to include
frst person disabled perspectives on representativeness and inclu-
sion, as well as data collection and sharing practices. Future work
remains in exploring contributor concerns such as privacy [60, 77]
Kamikubo et al.
and surveillance [7], especially for multiple marginalized contribu-
tors.
Analyzing other sociocultural factors. A more in-depth analysis of
the sociocultural contexts in which datasets were produced, not just
what was reported, could lead to interesting insights. A quick in-
spection of our datasets revealed that when data involves children,
specifcally in studies of developmental disability, we sometimes
fnd family information, such as socioeconomic status [132] or
parental education [58, 160]. Future work could explore represen-
tation along axes of level of education, language, nationality, and
socioeconomic status of the data contributors, as well as intersec-
tions between them. It would also be interesting to explore the
infuence of dataset origin (i.e. from the HCI vs medical research
community) on demographic representation as they may opt for
diferent models of disability.
Accounting for dataset impact. Our analysis of the implications
of representation is complicated by the fact that datasets vary in
research impact. Potential indicators of impact include the number
of citations, the models they are used to train or benchmark, the
venues in which they are published, and whether they originate
from academia or industry. Future work remains in investigating
and defning impact indicators and metrics, and weaving those
insights into discussions of representativeness.
Beyond accessibility datasets. While any insights from our analy-
sis may not be generalizable beyond the research community, our
fndings present an opportunity for broader AI communities to
strive towards more representativeness—along disability and other
dimensions—by including accessibility datasets in their training
data. For example, AI datasets have been critiqued for being heav-
ily skewed towards younger adults, and under-representing older
adults [128]. In contrast, accessibility datasets yield a wide vari-
ability of age groups. In future research, we strive to connect our
discussions of representation gaps with larger trends for broader
AI datasets and investigate whether accessibility can be used as a
lens to diversify representation for the broader AI community.
6 CONCLUSION
We conducted a detailed analysis of data representativeness among
190 accessibility datasets, with an emphasis on the intersections of
disability with age, gender, and race & ethnicity. While we found
diverse representation of age in accessibility datasets, we identifed
gaps in gender and race & ethnicity representation among these
datasets. Our fndings illustrate the implications of historical and
social contexts. Although we acknowledge there are limitations
when collecting these demographic variables, going forward, we
propose a participatory approach when collaborating with disabled
contributors and encourage transparency regarding data collection
purpose and maintenance throughout the process. We hope our
efort elucidates the current challenges in representation among the
accessibility community while expanding the space of possibility
for greater inclusion of marginalized communities in AI-infused
systems more broadly. Finally, we hope that our eforts provoke
conversations on data representativeness through a critical and
epistemological lens.
Data Representativeness in Accessibility Datasets: A Meta-Analysis
ACKNOWLEDGMENTS
We thank Hal Daumé III for providing valuable feedback on our pre-
liminary work. We also thank our anonymous reviewers for further
strengthening this paper. This work is supported by National Insti-
tute on Disability, Independent Living, and Rehabilitation Research
(NIDILRR), ACL, HHS (#90REGE0008).
REFERENCES
[1] Jacob Abbott, Haley MacLeod, Novia Nurain, Gustave Ekobe, and Sameer Patil.
2019. Local Standards for Anonymization Practices in Health, Wellness, Acces-
sibility, and Aging Research at CHI. In Proceedings of the 2019 CHI Conference
on Human Factors in Computing Systems (CHI ’19). Association for Computing
Machinery (ACM), 1–14. https://doi.org/10.1145/3290605.3300692
[2] Gaurav Aggarwal and Latika Singh. 2018. Evaluation of Supervised Learning
Algorithms Based on Speech Features as Predictors to the Diagnosis of Mild to
Moderate Intellectual Disability. 3D Research 9, 4 (2018), 55. https://doi.org/10.
1007/s13319-018-0207-6
[3] Meghan Allen, Joanna McGrenere, and Barbara Purves. 2007. The Design and
Field Evaluation of PhotoTalk: A Digital Image Communication Application for
People with Aphasia. In Proceedings of the 9th International ACM SIGACCESS
Conference on Computers and Accessibility (ASSETS ’07). Association for Com-
puting Machinery (ACM), 187–194. https://doi.org/10.1145/1296843.1296876
[4] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira
Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen,
Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI
Interaction. Association for Computing Machinery, New York, NY, USA, 1–13.
https://doi.org/10.1145/3290605.3300233
[5] Frank M Andrews and A Regula Herzog. 1986. The quality of survey data as
related to age of respondent. J. Amer. Statist. Assoc. 81, 394 (1986), 403–410.
[6] James T Becker, François Boiler, Oscar L Lopez, Judith Saxton, and Karen L
McGonigle. 1994. The natural history of Alzheimer’s disease: description of
study cohort and accuracy of diagnosis. Archives of neurology 51, 6 (1994),
585–594. https://doi.org/10.1001/archneur.1994.00540180063015
[7] Cynthia L. Bennett and Os Keyes. 2020. What is the Point of Fairness? Disability,
AI and the Complexity of Justice. 125, Article 5 (March 2020), 1 pages. https:
//doi.org/10.1145/3386296.3386301
[8] Rohit Bhalla, Brandon G Yongue, and Brian P Currie. 2012. Standardizing
race, ethnicity, and preferred language data collection in hospital information
systems: results and implications for healthcare delivery and policy. Journal for
Healthcare Quality 34, 2 (2012), 44–52.
[9] Jefrey P. Bigham, Anna C. Cavender, Jeremy T. Brudvik, Jacob O. Wobbrock,
and Richard E Ladner. 2007. WebinSitu: A Comparative Analysis of Blind
and Sighted Browsing Behavior. In Proceedings of the 9th International ACM
SIGACCESS Conference on Computers and Accessibility (Assets ’07). Association
for Computing Machinery (ACM), 51–58. https://doi.org/10.1145/1296843.
1296854
[10] Brianna Blaser and Richard E Ladner. 2020. Why is Data on Disability so Hard
to Collect and Understand?. In 2020 Research on Equity and Sustained Partici-
pation in Engineering, Computing, and Technology (RESPECT), Vol. 1. IEEE, 1–
8. https://www.washington.edu/doit/sites/default/fles/atoms/fles/RESPECT_
2020_DisabilityData.pdf
[11] Miranda Bogen, Aaron Rieke, and Shazeda Ahmed. 2020. Awareness in Prac-
tice: Tensions in Access to Sensitive Attribute Data for Antidiscrimination. In
Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
(Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York,
NY, USA, 492–500. https://doi.org/10.1145/3351095.3372877
[12] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T
Kalai. 2016. Man is to computer programmer as woman is to homemaker?
debiasing word embeddings. Advances in neural information processing systems
29 (2016), 4349–4357.
[13] Brian M. Bot, Christine Suver, Elias Chaibub Neto, Michael Kellen, Arno Klein,
Christopher Bare, Megan Doerr, Abhishek Pratap, John Wilbanks, E. Ray Dorsey,
Stephen H. Friend, and Andrew D Trister. 2016. The mPower study, Parkinson
disease mobile data collected using ResearchKit. Scientifc Data 3 (March 2016),
160011. https://doi.org/10.1038/sdata.2016.11
[14] Michael R Bowl and Sally J Dawson. 2019. Age-related hearing loss. Cold Spring
Harbor perspectives in medicine 9, 8 (2019), a033217.
[15] Danielle Bragg, Naomi Caselli, Julie A. Hochgesang, Matt Huenerfauth, Leah
Katz-Hernandez, Oscar Koller, Raja Kushalnagar, Christian Vogler, and Richard E.
Ladner. 2021. The FATE Landscape of Sign Language AI Datasets: An In-
terdisciplinary Perspective. 14, 2, Article 7 (July 2021), 45 pages. https:
//doi.org/10.1145/3436996
[16] Lundy Braun, Anne Fausto-Sterling, Duana Fullwiley, Evelynn M Hammonds,
Alondra Nelson, William Quivers, Susan M Reverby, and Alexandra E Shields.
2007. Racial categories in medical practice: how useful are they? PLoS medicine
ASSETS ’22, October 23–26, 2022, Athens, Greece
4, 9 (2007), e271.
[17] The Editors of Encyclopaedia Britannica. 2021. critical race theory. https:
//www.britannica.com/topic/critical-race-theory.
[18] Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accu-
racy Disparities in Commercial Gender Classifcation. In Proceedings of the 1st
Conference on Fairness, Accountability and Transparency (Proceedings of Machine
Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR,
77–91. https://proceedings.mlr.press/v81/buolamwini18a.html
[19] United States Census Bureau. 2021. QuickFacts United States. https://www.
census.gov/quickfacts/fact/table/US/PST045221. Accessed: 2022-01-03.
[20] Linda Burhansstipanov and Delight E Satter. 2000. Ofce of Management and
Budget racial categories and implications for American Indians and Alaska
Natives. American Journal of Public Health 90, 11 (2000), 1720.
[21] Lesley G Campbell, Siya Mehtani, Mary E Dozier, and Janice Rinehart. 2013.
Gender-heterogeneous working groups produce higher quality science. PloS
one 8, 10 (2013), e79147.
[22] Romuald Carette, Mahmoud Elbattah, Federica Cilia, Gilles Dequen, Jean-Luc
Guerin, and Jérôme Bosche. 2019. Learning to Predict Autism Spectrum Disorder
based on the Visual Patterns of Eye-tracking Scanpaths. In Proceedings of the
12th International Conference on Health Informatics. 103–112. https://doi.org/10.
5220/0007402601030112
[23] Naomi K Caselli, Zed Sevcikova Sehyr, Ariel M Cohen-Goldberg, and Karen
Emmorey. 2017. ASL-LEX: A lexical database of American Sign Language.
Behavior research methods 49, 2 (2017), 784–801.
[24] L. Elisa Celis, Amit Deshpande, Tarun Kathuria, and Nisheeth K. Vishnoi. 2016.
How to be Fair and Diverse? arXiv:1610.07183 [cs.LG]
[25] Ugo Cesari, Giuseppe De Pietro, Elio Marciano, Ciro Niri, Giovanna Sannino,
and Laura Verde. 2018. A new database of healthy and pathological voices.
Computers & Electrical Engineering 68 (May 2018), 310–321. https://doi.org/10.
1016/j.compeleceng.2018.04.008
[26] Shi Chen and Qi Zhao. 2019. Attention-based autism spectrum disorder screen-
ing with privileged modality. In Proceedings of the IEEE/CVF International Con-
ference on Computer Vision. 1181–1190.
[27] Vivian Chou. 2017. How science and genetics are reshaping the race debate of
the 21st century. Science in the News 17 (2017).
[28] Eli Clare. 2015. Exile and pride. In Exile and Pride. Duke University Press.
[29] Leigh Clark, Benjamin R. Cowan, Abi Roper, Stephen Lindsay, and Owen Sheers.
2020. Speech Diversity and Speech Interfaces: Considering an Inclusive Future
through Stammering. Association for Computing Machinery, New York, NY,
USA. https://doi.org/10.1145/3405755.3406139
[30] Jantina Rochelle Cliford. 2005. An evaluation of the technical adequacy of a
parent-completed inventory of developmental skills. (2005).
[31] Richard S Cooper. 2003. Race and genomics. The New England journal of medicine
348, 12 (2003), 1166.
[32] Imelda T Coyne. 1998. Researching children: some methodological and ethical
considerations. Journal of Clinical Nursing 7, 5 (1998), 409–416.
[33] Jennifer L Davidson and Carlos Jensen. 2013. What health topics older adults
want to track: a participatory design study. In Proceedings of the 15th International
ACM SIGACCESS Conference on Computers and Accessibility. 1–8.
[34] Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten.
2019. Does object recognition work for everyone?. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition Workshops. 52–59.
[35] Roxanne DePaul. 2016. DementiaBank English PPA Corpus. https://doi.org/10.
21415/T5ZH5T
[36] Mark Diaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle.
2018. Addressing Age-Related Bias in Sentiment Analysis. (2018). https:
//doi.org/10.1145/3173574.3173986
[37] Erin Digitale. 2022. Study fnds diferences between brains of girls, boys with
autism. https://med.stanford.edu/news/all-news/2022/02/autism-brain-sex-
diferences.html/
[38] Lucas Dixon, John Li, Jefrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.
Measuring and mitigating unintended bias in text classifcation. In Proceedings
of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 67–73.
[39] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco,
Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting
Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus.
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language
Processing. 1286–1305.
[40] Rohan Doshi, Youzheng Chen, Liyang Jiang, Xia Zhang, Fadi Biadsy, Bhu-
vana Ramabhadran, Fang Chu, Andrew Rosenberg, and Pedro J. Moreno. 2021.
Extending Parrotron: An End-to-End, Speech Conversion and Speech Recog-
nition Model for Atypical Speech. In ICASSP 2021 - 2021 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP). 6988–6992.
https://doi.org/10.1109/ICASSP39728.2021.9414644
[41] Susan M Dray, Anicia N Peters, Anke M Brock, Andrea Peer, Allison Druin,
Shikoh Gitau, Janaki Kumar, and Dianne Murray. 2013. Leveraging the progress
of women in the HCI feld to address the diversity chasm. In CHI’13 Extended
Abstracts on Human Factors in Computing Systems. 2399–2406.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[42] Marina Drosou, HV Jagadish, Evaggelia Pitoura, and Julia Stoyanovich. 2017.
Diversity in big data: A review. Big data 5, 2 (2017), 73–84.
[43] Huiyu Duan, Guangtao Zhai, Xiongkuo Min, Zhaohui Che, Yi Fang, Xiaokang
Yang, Jesús Gutiérrez, and Patrick Le Callet. 2019. A Dataset of Eye Movements
for the Children with Autism Spectrum Disorder. In Proceedings of the 10th
ACM Multimedia Systems Conference (Amherst, Massachusetts) (MMSys ’19).
Association for Computing Machinery (ACM), New York, NY, USA, 255–260.
https://doi.org/10.1145/3304109.3325818
[44] A Engler. 2019. For some employment algorithms, disability discrimination
by default. https://www.brookings.edu/blog/techtank/2019/10/31/for-some-
employment-algorithms-disability-discrimination-by-default/
[45] Sukru Eraslan, Victoria Yaneva, Yeliz Yesilada, and Simon Harper. 2019. Web
users with autism: eye tracking evidence for diferences. Behaviour & Information
Technology 38, 7 (2019), 678–700. https://doi.org/10.1080/0144929X.2018.1551933
[46] Anne Fausto-Sterling. 2000. Sexing the body: Gender politics and the construction
of sexuality. Basic Books.
[47] Sina Fazelpour and Maria De-Arteaga. 2021. Diversity in Sociotechnical Machine
Learning Systems. CoRR abs/2107.09163 (2021). arXiv:2107.09163 https://arxiv.
org/abs/2107.09163
[48] Leah Findlater, Steven Goodman, Yuhang Zhao, Shiri Azenkot, and Margot
Hanley. 2020. Fairness Issues in AI Systems That Augment Sensory Abilities.
125, Article 8 (mar 2020), 1 pages. https://doi.org/10.1145/3386296.3386304
[49] Leah Findlater and Lotus Zhang. 2020. Input Accessibility: A Large Dataset
and Summary Analysis of Age, Motor Ability and Input Performance. In The
22nd International ACM SIGACCESS Conference on Computers and Accessibility
(Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery,
New York, NY, USA, Article 17, 6 pages. https://doi.org/10.1145/3373625.3417031
[50] Annette Flanagin, Tracy Frey, Stacy L Christiansen, AMA Manual of Style Com-
mittee, et al. 2021. Updated guidance on the reporting of race and ethnicity in
medical and science journals. JAMA 326, 7 (2021), 621–627.
[51] Eric Fombonne. 2009. Epidemiology of pervasive developmental disorders.
Pediatric research 65, 6 (2009), 591–598.
[52] Marvella E Ford and P Adam Kelly. 2005. Conceptualizing and categorizing race
and ethnicity in health services research. Health services research 40, 5p2 (2005),
1658–1675.
[53] Núria Gala, Anaïs Tack, Ludivine Javourey-Drevet, Thomas François, and Jo-
hannes C Ziegler. 2020. Alector: A parallel corpus of simplifed French texts with
alignments of misreadings by poor and dyslexic readers. In Language Resources
and Evaluation for Language Technologies (LREC).
[54] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman
Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets
for datasets. Commun. ACM 64, 12 (2021), 86–92.
[55] Ellen Giarelli, Lisa D Wiggins, Catherine E Rice, Susan E Levy, Russell S Kirby,
Jennifer Pinto-Martin, and David Mandell. 2010. Sex diferences in the evaluation
and diagnosis of autism spectrum disorders among children. Disability and
health journal 3, 2 (2010), 107–116.
[56] Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, and
Meredith Ringel Morris. 2020. Toward Fairness in AI for People with Disabilities:
A Research Roadmap. SIGACCESS Accessible Computing 125, Article 2 (March
2020), 1 pages. https://doi.org/10.1145/3386296.3386298
[57] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman,
Jiebo Luo, and Jefrey P Bigham. 2018. VizWiz Grand Challenge: Answering
Visual Questions from Blind People. Proceedings of the 2018 IEEE/CVF Conference
on Computer Vision and Pattern Recognition (Jun 2018). https://doi.org/10.1109/
cvpr.2018.00380
[58] Haya Berman Hakim and Nan Bernstein Ratner. 2004. Nonword repetition
abilities of children who stutter: An exploratory study. Journal of fuency
disorders 29, 3 (2004), 179–199.
[59] Ali G Hamedani, Brian L VanderBeek, and Allison W Willis. 2019. Blindness
and visual impairment in the medicare population: disparities and association
with hip fracture and neuropsychiatric outcomes. Ophthalmic epidemiology 26,
4 (2019), 279–285.
[60] Foad Hamidi, Kellie Poneres, Aaron Massey, and Amy Hurst. 2018. Who
Should Have Access to My Pointing Data? Privacy Tradeofs of Adaptive
Assistive Technologies. In Proceedings of the 20th International ACM SIGAC-
CESS Conference on Computers and Accessibility (Galway, Ireland) (ASSETS
’18). Association for Computing Machinery, New York, NY, USA, 203–216.
https://doi.org/10.1145/3234695.3239331
[61] Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M. Branham. 2018. Gen-
der Recognition or Gender Reductionism? The Social Implications of Embed-
ded Gender Recognition Systems. In Proceedings of the 2018 CHI Conference
on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18).
Association for Computing Machinery, New York, NY, USA, 1–13. https:
//doi.org/10.1145/3173574.3173582
[62] Romana Hasnain-Wynia and David W Baker. 2006. Obtaining data on patient
race, ethnicity, and primary language in health care organizations: current
challenges and proposed solutions. Health services research 41, 4p1 (2006),
1501–1518.
Kamikubo et al.
[63] Saad Hassan, Matt Huenerfauth, and Cecilia Ovesdotter Alm. 2021. Unpacking
the Interdependent Systems of Discrimination: Ableist Bias in NLP Systems
through an Intersectional Lens. CoRR abs/2110.00521 (2021). arXiv:2110.00521
https://arxiv.org/abs/2110.00521
[64] Jefrey M Hausdorf, Apinya Lertratanakul, Merit E Cudkowicz, Amie L Peterson,
David Kaliton, and Ary L Goldberger. 2000. Dynamic markers of altered gait
rhythm in amyotrophic lateral sclerosis. Journal of applied physiology (2000).
[65] Jefrey M Hausdorf, Susan L Mitchell, Renee Firtion, Chung-Kang Peng, Merit E
Cudkowicz, Jeanne Y Wei, and Ary L Goldberger. 1997. Altered fractal dynamics
of gait: reduced stride-interval correlations with aging and Huntington’s disease.
Journal of applied physiology 82, 1 (1997), 262–269. https://doi.org/10.1152/
jappl.1997.82.1.262
[66] Tamar Heller, P Staford, LA Davis, L Sedlezky, and V Gaylord. 2010. People with
intellectual and developmental disabilities growing old: An overview. Impact:
Feature Issue on Aging and People with Intellectual and Developmental Disabilities
23, 1 (2010), 2–3.
[67] Martin Holtmann, Sven Bölte, and Fritz Poustka. 2007. Autism spectrum disor-
ders: Sex diferences in autistic behaviour domains and coexisting psychopathol-
ogy. Developmental Medicine & Child Neurology 49, 5 (2007), 361–366.
[68] Lindsay M Howden, Julie A Meyer, et al. 2011. Age and sex composition: 2010.
[69] Matt Huenerfauth and Hernisa Kacorri. 2014. Release of experimental stimuli
and questions for evaluating facial expressions in animations of American Sign
Language. In Proceedings of the 6th Workshop on the Representation and Processing
of Sign Languages: Beyond the Manual Channel, The 9th International Conference
on Language Resources and Evaluation (LREC ’14). http://dx.doi.org/10.1007/978-
3-642-39188-0_55
[70] Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu
Zhong, and Stephen Denuyl. 2020. Social Biases in NLP Models as Barriers
for Persons with Disabilities. CoRR abs/2005.00813 (2020). arXiv:2005.00813
https://arxiv.org/abs/2005.00813
[71] Dimitrios Iakovakis, Stelios Hadjidimitriou, Vasileios Charisis, Sevasti Bostan-
tjopoulou, Zoe Katsarou, Lisa Klingelhoefer, Heinz Reichmann, Sofa B Dias,
José A Diniz, Dhaval Trivedi, et al. 2018. Motor impairment estimates via
touchscreen typing dynamics toward Parkinson’s disease detection from data
harvested in-the-wild. Frontiers in ICT 5 (2018), 28.
[72] Dimitrios Iakovakis, Stelios Hadjidimitriou, Vasileios Charisis, Sevasti Bostant-
zopoulou, Zoe Katsarou, and Leontios J Hadjileontiadis. 2018. Touchscreen
typing-pattern analysis for detecting fne motor skills decline in early-stage
Parkinson’s disease. Scientifc reports 8, 1 (2018), 1–13.
[73] Jina Jang, Johnny L Matson, Hilary L Adams, Matt J Konst, Paige E Cervantes,
and Rachel L Goldin. 2014. What are the ages of persons studied in autism
research: A 20-year review. Research in Autism Spectrum Disorders 8, 12 (2014),
1756–1760.
[74] Aparna Joshi and Hyuntak Roh. 2009. The role of context in work team diversity
research: A meta-analytic review. Academy of management journal 52, 3 (2009),
599–627.
[75] Hernisa Kacorri. 2017. Teachable Machines for Accessibility. SIGACCESS -
Accessible Computing 119, 10–18. https://doi.org/10.1145/3167902.3167904
[76] Hernisa Kacorri, Utkarsh Dwivedi, Sravya Amancherla, Mayanka Jha, and Riya
Chanduka. 2020. IncluSet: A Data Surfacing Repository for Accessibility Datasets.
Association for Computing Machinery (ACM). https://doi.org/10.1145/3373625.
3418026
[77] Hernisa Kacorri, Utkarsh Dwivedi, and Rie Kamikubo. 2020. Data Sharing in
Wellness, Accessibility, and Aging. (2020).
[78] Hernisa Kacorri, Sergio Mascetti, Andrea Gerino, Dragan Ahmetovic, Hironobu
Takagi, and Chieko Asakawa. 2016. Supporting Orientation of People with
Visual Impairment: Analysis of Large Scale Usage Data. In Proceedings of the
18th International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’16). Association for Computing Machinery (ACM), 151–159. https:
//doi.org/10.1145/2982142.2982178
[79] Rie Kamikubo, Utkarsh Dwivedi, and Hernisa Kacorri. 2021. Sharing Practices
for Datasets Related to Accessibility and Aging. In The 23rd International ACM
SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA)
(ASSETS ’21). Association for Computing Machinery, New York, NY, USA, Article
28, 16 pages. https://doi.org/10.1145/3441852.3471208
[80] Kimmo Kärkkäinen and Jungseock Joo. 2019. FairFace: Face Attribute Dataset for
Balanced Race, Gender, and Age. CoRR abs/1908.04913 (2019). arXiv:1908.04913
http://arxiv.org/abs/1908.04913
[81] Preet Chandan Kaur, Tushar Ghorpade, and Vanita Mane. 2016. Analysis of
data security by using anonymization techniques. In 2016 6th International
Conference-Cloud System and Big Data Engineering (Confuence). IEEE, 287–293.
[82] Amit Kaushal, Russ Altman, and Curt Langlotz. 2020. Geographic distribution
of US cohorts used to train deep learning algorithms. Jama 324, 12 (2020),
1212–1213.
[83] Matthew Kay, Cynthia Matuszek, and Sean A. Munson. 2015. Unequal Rep-
resentation and Gender Stereotypes in Image Search Results for Occupa-
tions. Association for Computing Machinery, New York, NY, USA. https:
//doi.org/10.1145/2702123.2702520
Data Representativeness in Accessibility Datasets: A Meta-Analysis
[84] David Kertzer and Dominique Arel. [n.d.]. Census and identity. ([n. d.]).
[85] Melissa Kirkovski, Peter G Enticott, and Paul B Fitzgerald. 2013. A review of
the role of female gender in autism spectrum disorders. Journal of autism and
developmental disorders 43, 11 (2013), 2584–2603.
[86] Jochen Klucken, Jens Barth, Patrick Kugler, Johannes Schlachetzki, Thore Henze,
Franz Marxreiter, Zacharias Kohl, Ralph Steidl, Joachim Hornegger, Bjoern
Eskofer, et al. 2013. Unbiased and mobile gait analysis detects motor impairment
in Parkinson’s disease. PloS one 8, 2 (2013), e56956. https://doi.org/10.1371/
journal.pone.0056956
[87] Meng-Chuan Lai, Michael V Lombardo, Bonnie Auyeung, Bhismadev
Chakrabarti, and Simon Baron-Cohen. 2015. Sex/gender diferences and autism:
setting the scene for future research. Journal of the American Academy of Child
& Adolescent Psychiatry 54, 1 (2015), 11–24.
[88] Kyungjun Lee and Hernisa Kacorri. 2019. Hands Holding Clues for Object
Recognition in Teachable Machines. In Proceedings of the 2019 CHI Conference
on Human Factors in Computing Systems (CHI ’19). Association for Computing
Machinery (ACM), 1–12. https://doi.org/10.1145/3290605.3300566
[89] Matthew L. Lee and Anind K Dey. 2007. Providing Good Memory Cues for People
with Episodic Memory Impairment. In Proceedings of the 9th International ACM
SIGACCESS Conference on Computers and Accessibility (Assets ’07). Association
for Computing Machinery (ACM), 131–138. https://doi.org/10.1145/1296843.
1296867
[90] Fritz-Georg Lehnhardt, Christine Michaela Falter, Astrid Gawronski, Kathleen
Pfeifer, Ralf Tepest, Jeremy Franklin, and Kai Vogeley. 2016. Sex-related cogni-
tive profle in autism spectrum disorders diagnosed late in life: implications for
the female autistic phenotype. Journal of Autism and Developmental Disorders
46, 1 (2016), 139–154.
[91] Daniel Leightley, Moi Hoon Yap, Jessica Coulson, Yoann Barnouin, and Jamie S
McPhee. 2015. Benchmarking human motion analysis using kinect one: An open
source dataset. In Proceedings of the 2015 Asia-Pacifc Signal and Information
Processing Association Annual Summit and Conference (APSIPA ’15). IEEE, 1–7.
https://doi.org/10.1109/APSIPA.2015.7415438
[92] DONGXU LI, Cristian Rodriguez, Xin Yu, and HONGDONG LI. 2020. Word-
level Deep Sign Language Recognition from Video: A New Large-scale Dataset
and Methods Comparison. In Proceedings of the IEEE/CVF Winter Conference on
Applications of Computer Vision (WACV).
[93] Megan Marie Quaglia Linton. 2021. The Institutional Remains: Transinstitution-
alization of Disability & Sexuality. (2021).
[94] Paul H Lipkin, Jefrey Okamoto, Kenneth W Norwood, Richard C Adams, Timo-
thy J Brei, Robert T Burke, Beth Ellen Davis, Sandra L Friedman, Amy J Houtrow,
Susan L Hyman, et al. 2015. The Individuals with Disabilities Education Act
(IDEA) for children with special educational needs. Pediatrics 136, 6 (2015),
e1650–e1662.
[95] Keng Yin Loh and J Ogle. 2004. Age related visual impairment in the elderly.
The Medical journal of Malaysia 59, 4 (2004), 562–8.
[96] Steve Lohr. 2018. Facial recognition is accurate, if you’re a white guy. New York
Times 9, 8 (2018), 283.
[97] Daria Loi and Thomas Lodato. 2020. On empathy and empiricism: addressing
stereotypes about older adults in technology. Interactions 28, 1 (2020), 23–25.
[98] Rachel Loomes, Laura Hull, and William Polmear Locke Mandy. 2017. What
is the male-to-female ratio in autism spectrum disorder? A systematic review
and meta-analysis. Journal of the American Academy of Child & Adolescent
Psychiatry 56, 6 (2017), 466–474.
[99] Deborah Lupton. 2017. Digital health now and in the future: Findings
from a participatory design stakeholder workshop. Digital health 3 (2017),
2055207617740018.
[100] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich,
and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? A
Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019.
, Article 371 (2021), 18 pages. https://doi.org/10.1145/3411764.3445412
[101] Brian MacWhinney, Steven Bird, Christopher Cieri, and Craig Martell. 2004.
TalkBank: Building an open unifed multimodal database of communicative inter-
action. In Proceedings of the 4th International Conference on Language Resources
and Evaluation (LREC ’04). Evaluations and Language resources Distribution
Agency, 525–528. http://www.lrec-conf.org/proceedings/lrec2004/pdf/392.pdf
[102] David S Mandell, Lindsay J Lawer, Kira Branch, Edward S Brodkin,
Kristin Healey, Robert Witalec, Donielle N Johnson, and Raquel E Gur.
2012. Prevalence and correlates of autism in a state psychiatric hospital.
Autism 16, 6 (2012), 557–567. https://doi.org/10.1177/1362361311412058
arXiv:https://doi.org/10.1177/1362361311412058 PMID: 21846667.
[103] William Mandy, Rebecca Chilvers, Uttom Chowdhury, Gemma Salter, Anna
Seigal, and David Skuse. 2012. Sex diferences in autism spectrum disorder:
evidence from a large sample of children and adolescents. Journal of autism and
developmental disorders 42, 7 (2012), 1304–1313.
[104] Jennifer J. Manly. 2006. Deconstructing Race and Ethnicity: Implications for
Measurement of Health Outcomes. Medical Care 44, 11 (2006), S10–S16. http:
//www.jstor.org/stable/41219499
ASSETS ’22, October 23–26, 2022, Athens, Greece
[105] José Luis Pérez Mantero. 2014. Interacción y predictividad: Los intercambios
conversacionales con hablantes con demencia tipo alzhéimer. revista de investi-
gación Lingüística 17 (2014), 97–118.
[106] Maya Matheis, Johnny L Matson, Esther Hong, and Paige E Cervantes. 2019.
Gender diferences and similarities: Autism symptomatology and developmental
functioning in young children. Journal of autism and developmental disorders
49, 3 (2019), 1219–1231.
[107] Silke Matthes, Thomas Hanke, Anja Regen, Jakob Storz, Satu Worseck, Eleni
Efthimiou, Athanasia-Lida Dimou, Annelies Brafort, John Glauert, and Eva Safar.
2012. Dicta-Sign–building a multilingual sign language corpus. In Proceedings
of the 5th Workshop on the Representation and Processing of Sign Languages:
Interactions between Corpus and Lexicon (LREC ’12). https://www.sign-lang.uni-
hamburg.de/lrec/lrec/pubs/12016.pdf
[108] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram
Galstyan. 2021. A Survey on Bias and Fairness in Machine Learning. ACM
Comput. Surv. 54, 6, Article 115 (July 2021), 35 pages. https://doi.org/10.1145/
3457607
[109] Michele Merler, Nalini Ratha, Rogerio S. Feris, and John R. Smith. 2019. Diversity
in Faces. arXiv:1901.10436 [cs.CV]
[110] Milagros Miceli, Julian Posada, and Tianling Yang. 2021. Studying Up
Machine Learning Data: Why Talk About Bias When We Mean Power?
arXiv:2109.08131 [cs.HC]
[111] Roman Čmejla-Hana Růžičková Jiří Klempíř Michal Novotný, Jan Rusz and
Evžen Růžička. 2016. Hypernasality associated with basal ganglia dysfunction:
evidence from Parkinson’s disease and Huntington’s disease. PeerJ 4 (2016),
e2530. https://dx.doi.org/10.7717%2Fpeerj.2530
[112] Emancipated Minors and Self-Sufcient Minors. 2017. Guidance and Procedures:
Child Assent and Permission by Parents or Guardians. https://ora.research.ucla.
edu/OHRPP/Documents/Policy/9/ChildAssent_ParentPerm.pdf. (2017).
[113] Margaret Mitchell, Dylan Baker, Nyalleng Moorosi, Emily Denton, Ben Hutchin-
son, Alex Hanna, Timnit Gebru, and Jamie Morgenstern. 2020. Diversity
and Inclusion Metrics in Subset Selection. In Proceedings of the AAAI/ACM
Conference on AI, Ethics, and Society (New York, NY, USA) (AIES ’20). As-
sociation for Computing Machinery, New York, NY, USA, 117–123. https:
//doi.org/10.1145/3375627.3375832
[114] Ross E Mitchell, Travas A Young, Bellamie Bachelda, and Michael A Karchmer.
2006. How many people use ASL in the United States? Why estimates need
updating. Sign Language Studies 6, 3 (2006), 306–335.
[115] Masooda Modak, Ketan Ghotane, V Siddhanth, Nachiket Kelkar, and Prachi G
Aravind Iyer. 2019. Detection of Dyslexia using Eye Tracking Measures. Inter-
national Journal of Innovative Technology and Exploring Engineering (IJITEE) 8
(2019), 1011–1014.
[116] Karyn Anne Mofatt. 2010. Addressing age-related pen-based target ac-
quisition difculties. Ph.D. Dissertation. University of British Columbia.
http://www.sigaccess.org/2010/01/addressing-age-related-pen-based-target-
acquisition-difculties/
[117] Vanessa Moore and Sally Goodson. 2003. How well does early diagnosis of
autism stand the test of time? Follow-up study of children assessed for autism at
age 2 and development of an early diagnostic service. Autism 7, 1 (2003), 47–63.
[118] Meredith Ringel Morris. 2020. AI and accessibility. Commun. ACM 63, 6 (2020),
35–37.
[119] Yolanda Moses. 2017. Why Do We Keep Using the Word “Caucasian”? https:
//www.sapiens.org/column/race/caucasian-terminology-origin/
[120] Maura Jones Moyle, Susan Ellis Weismer, Julia L Evans, and Mary J Lindstrom.
2007. Longitudinal relationships between lexical and grammatical development
in typical and late-talking children. (2007).
[121] Kevin Munger, Ishita Gopal, Jonathan Nagler, and Joshua A. Tucker. 2021. Acces-
sibility and generalizability: Are social media efects moderated by age or digital
literacy? Research & Politics 8, 2 (2021), 20531680211016968. https://doi.org/10.
1177/20531680211016968 arXiv:https://doi.org/10.1177/20531680211016968
[122] Karama C Neal et al. 2008. Use and misuse of ‘race’in biomedical research.
Journal of Health Ethics 5, 1 (2008), 8.
[123] David R Nerenz, Bernadette McFadden, Cheryl Ulmer, et al. 2009. Race, ethnicity,
and language data: standardization for health care quality improvement. (2009).
[124] Antony Nicol, Chris Casey, and Stuart MacFarlane. 2002. Children are ready for
speech technology-but is the technology ready for them. Interaction Design and
Children, Eindhoven, The Netherlands (2002).
[125] Spiros Nikolopoulos, Kostas Georgiadis, Fotis Kalaganis, Georgios Liaros, Iouli-
etta Lazarou, Katerina Adam, Papazoglou-Chalikias Anastasios, Elisavet Chatzi-
lari, P. Vangelis Oikonomou, C. Panagiotis Petrantonakis, I. Kompatsiaris, Chan-
dan Kumar, Raphael Menges, Stefen Staab, Daniel Müller, Korok Sengupta, Sev-
asti Bostantjopoulou, Zoe Katsarou, Gabi Zeilig, Meir Plotnin, Amihai Gottlieb,
Sofa Fountoukidou, Jaap Ham, Dimitrios Athanasiou, Agnes Mariakaki, Dario
Comanducci, Eduardo Sabatini, Walter Nistico, and Markus Plank. 2017. The
MAMEM Project - A dataset for multimodal human-computer interaction using
biosignals and eye tracking information. https://doi.org/10.5281/zenodo.834154
[126] World Institute on Disability. 2021. AI and Accessibility. https://wid.org/2019/
06/12/ai-and-accessibility/
ASSETS ’22, October 23–26, 2022, Athens, Greece
[127] Sally Ozonof, Beth L Goodlin-Jones, and Marjorie Solomon. 2005. Evidence-
based assessment of autism spectrum disorders in children and adolescents.
Journal of Clinical Child and Adolescent Psychology 34, 3 (2005), 523–540.
[128] Joon Sung Park, Michael S. Bernstein, Robin N. Brewer, Ece Kamar, and Mered-
ith Ringel Morris. 2021. Understanding the Representation and Representative-
ness of Age in AI Data Sets. CoRR abs/2103.09058 (2021). arXiv:2103.09058
https://arxiv.org/abs/2103.09058
[129] Joon Sung Park, Danielle Bragg, Ece Kamar, and Meredith Ringel Morris. 2021.
Designing an online infrastructure for collecting AI data from people with
disabilities. In Proceedings of the 2021 ACM Conference on Fairness, Accountability,
and Transparency. 52–63.
[130] Joseph Piven, Peter Rabins, and on behalf of the Autism-in Older Adults
Working Group. 2011. Autism Spectrum Disorders in Older Adults: Toward
Defning a Research Agenda. Journal of the American Geriatrics Society
59, 11 (2011), 2151–2155. https://doi.org/10.1111/j.1532-5415.2011.03632.x
arXiv:https://agsjournals.onlinelibrary.wiley.com/doi/pdf/10.1111/j.1532-
5415.2011.03632.x
[131] Martin Prince, Martin Knapp, Maelenn Guerchet, Paul McCrone, Matthew Prina,
A Comas-Herrera, Raphael Wittenberg, Bayo Adelaja, Bo Hu, Derek King, et al.
2014. Dementia UK: -overview. (2014).
[132] Nan Bernstein Ratner and Stacy Silverman. 2000. Parental perceptions of
children’s communicative development at stuttering onset. Journal of Speech,
Language, and Hearing Research 43, 5 (2000), 1252–1263.
[133] Allison B Ratto, Lauren Kenworthy, Benjamin E Yerys, Julia Bascom, An-
drea Trubanova Wieckowski, Susan W White, Gregory L Wallace, Cara Pugliese,
Robert T Schultz, Thomas H Ollendick, et al. 2018. What about the girls? Sex-
based diferences in autistic traits and adaptive skills. Journal of autism and
developmental disorders 48, 5 (2018), 1698–1711.
[134] Luz Rello, Ricardo Baeza-Yates, and Joaquim Llisterri. 2014. DysList: An An-
notated Resource of Dyslexic Errors. In Proceedings of the 9th International
Conference on Language Resources and Evaluation (LREC ’14). European Lan-
guages Resources Association (ELRA), 1289–1296. http://www.lrec-conf.org/
proceedings/lrec2014/pdf/612_Paper.pdf
[135] Luz Rello and Miguel Ballesteros. 2015. Detecting Readers with Dyslexia Using
Machine Learning with Eye Tracking Measures. In Proceedings of the 12th Web
for All Conference (W4A ’15). Association for Computing Machinery (ACM),
Article 16, 8 pages. https://doi.org/10.1145/2745555.2746644
[136] Michael Rembis. 2021. Crip Camp: A Disability Revolution. Jour-
nal of American History 108, 3 (12 2021), 667–669. https://doi.
org/10.1093/jahist/jaab339 arXiv:https://academic.oup.com/jah/article-
pdf/108/3/667/41938029/jaab339.pdf
[137] Hannah Ritchie and Max Roser. 2019. Gender Ratio. Our World in Data (2019).
https://ourworldindata.org/gender-ratio.
[138] Tessa Taylor Rivet and Johnny L Matson. 2011. Review of gender diferences
in core symptomatology in autism spectrum disorders. Research in Autism
Spectrum Disorders 5, 3 (2011), 957–976.
[139] Amanda Roestorf, Dermot M Bowler, Marie K Deserno, Patricia Howlin, Laura
Klinger, Helen McConachie, Jeremy R Parr, Patrick Powell, Barbara FC Van Hei-
jst, and Hilde M Geurts. 2019. “Older Adults with ASD: The Consequences of
Aging.” Insights from a series of special interest group meetings held at the In-
ternational Society for Autism Research 2016–2017. Research in autism spectrum
disorders 63 (2019), 3–12.
[140] Betul Erdogdu Sakar, M Erdem Isenkul, C Okan Sakar, Ahmet Sertbas, Fikret
Gurgen, Sakir Delil, Hulya Apaydin, and Olcay Kursun. 2013. Collection and
analysis of a Parkinson speech dataset with multiple types of sound recordings.
IEEE Journal of Biomedical and Health Informatics 17, 4 (2013), 828–834. https:
//doi.org/10.1109/JBHI.2013.2245674
[141] Adam Schembri, Jordan Fenlon, Ramas Rentelis, Sally Reynolds, and Kearsy
Cormier. 2013. Building the British sign language corpus. Language Documen-
tation & Conservation 7 (2013), 136–154.
[142] Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, and Jed R Brubaker.
2020. How We’ve Taught Algorithms to See Identity: Constructing Race and
Gender in Image Databases for Facial Analysis. Proceedings of the ACM on
Human-Computer Interaction 4, CSCW1 (2020), 1–35.
[143] Andrew Sears and Vicki Hanson. 2011. Representing Users in Accessibility Re-
search. In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machin-
ery, New York, NY, USA, 2235–2238. https://doi.org/10.1145/1978942.1979268
[144] Rajani Sebastian, Carol B Thompson, Nae-Yuh Wang, Amy Wright, Aaron Meyer,
Rhonda B Friedman, Argye E Hillis, and Donna C Tippett. 2018. Patterns of
decline in naming and semantic knowledge in primary progressive aphasia.
Aphasiology 32, 9 (2018), 1010–1030.
[145] Maya Sen and Omar Wasow. 2016. Race as a Bundle of Sticks: Designs that
Estimate Efects of Seemingly Immutable Characteristics. Annual Review of
Political Science 19, 1 (2016), 499–522. https://doi.org/10.1146/annurev-polisci-
032015-010015
[146] Julia Serano. 2013. Excluded: Making feminist and queer movements more inclusive.
Seal Press.
Kamikubo et al.
[147] David Serre and Svante Pääbo. 2004. Evidence for gradients of human genetic
diversity within and among continents. Genome research 14, 9 (2004), 1679–1685.
[148] Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D.
Sculley. 2017. No Classifcation without Representation: Assessing Geodiversity
Issues in Open Data Sets for the Developing World. arXiv:1711.08536 [stat.ML]
[149] Shanya Sharma, Manan Dey, and Koustuv Sinha. 2021. Evaluating Gender Bias
in Natural Language Inference. arXiv preprint arXiv:2105.05541 (2021).
[150] Linda R Shaw, Fong Chan, and Brian T McMahon. 2012. Intersectionality and
disability harassment: The interactive efects of disability, race, age, and gender.
Rehabilitation Counseling Bulletin 55, 2 (2012), 82–91.
[151] Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Jonathan Michaux, Diane
Brentari, Greg Shakhnarovich, and Karen Livescu. 2018. American Sign Lan-
guage Fingerspelling Recognition in the Wild. In 2018 IEEE Spoken Language
Technology Workshop (SLT). 145–152. https://doi.org/10.1109/SLT.2018.8639639
[152] Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Diane Brentari, Greg
Shakhnarovich, and Karen Livescu. 2019. Fingerspelling Recognition in the Wild
With Iterative Visual Attention. In Proceedings of the IEEE/CVF International
Conference on Computer Vision (ICCV).
[153] Ben Shneiderman. 2020. Human-centered artifcial intelligence: three fresh
ideas. AIS Transactions on Human-Computer Interaction 12, 3 (2020), 109–124.
[154] Ajay Singh, Chia Jung Yeh, and Sheresa Boone Blanchard. 2017. Ages and stages
questionnaire: a global screening scale. Boletín Médico Del Hospital Infantil de
México (English Edition) 74, 1 (2017), 5–12.
[155] Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2020. Partic-
ipation is not a design fx for machine learning. arXiv preprint arXiv:2007.02423
(2020).
[156] W Smith, P Mitchell, and JJ Wang. 1997. Gender, oestrogen, hormone replace-
ment and age-related macular degeneration: Results from the Blue Mountains
Eye Study. Australian and New Zealand journal of ophthalmology 25, 4 (1997),
13–15.
[157] Dean Spade. 2009. Trans law and Politics on a Neoliberal landscape. Trans Law
and Politics on a Neoliberal Landscape (June 26, 2009). Temple Political & Civil
Rights Law Review 18 (2009), 09–05.
[158] Katta Spiel, Oliver L. Haimson, and Danielle Lottridge. 2019. How to Do Better
with Gender on Surveys: A Guide for HCI Researchers. Interactions 26, 4 (jun
2019), 62–65. https://doi.org/10.1145/3338283
[159] Daniel Steel, Sina Fazelpour, Kinley Gillette, Bianca Crewe, and Michael Burgess.
2018. Multiple diversity concepts and their ethical-epistemic implications.
European journal for philosophy of science 8, 3 (2018), 761–780.
[160] Amy Strekas, Nan Bernstein Ratner, Madison Berl, and William D Gaillard. 2013.
Narrative abilities of children with epilepsy. International journal of language &
communication disorders 48, 2 (2013), 207–219.
[161] SurveyMonkey. [n.d.]. Gathering demographic information from surveys.
https://www.surveymonkey.com/mp/gathering-demographic-information-
from-surveys/. Accessed: 2022-01-03.
[162] Rachael Tatman. 2017. Gender and Dialect Bias in YouTube’s Automatic Cap-
tions. In Proceedings of the First ACL Workshop on Ethics in Natural Language
Processing. Association for Computational Linguistics, Valencia, Spain, 53–59.
https://doi.org/10.18653/v1/W17-1606
[163] Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily
Morrison, Edward Cutrell, Matthew Tobias Harris, and Katja Hofmann. 2021.
Disability-First Dataset Creation: Lessons from Constructing a Dataset for Teach-
able Object Recognition with Blind and Low Vision Data Collectors. In The
23rd International ACM SIGACCESS Conference on Computers and Accessibility
(Virtual Event, USA) (ASSETS ’21). Association for Computing Machinery, New
York, NY, USA, Article 27, 12 pages. https://doi.org/10.1145/3441852.3471225
[164] Krishna Thiyagarajan. 2016. Parkinson’s Disease Observations: Vari-
ables Regarding Parkinson’s Disease. https://www.kaggle.com/krisht/
parkinsonsdisease.
[165] David Thompson. 2002. Misplaced and forgotten:. Housing, Care and Support 5,
1 (2022/01/13 2002), 19–22. https://doi.org/10.1108/14608790200200006
[166] Jutta Treviranus. 2018. Sidewalk Toronto and Why Smarter is Not Bet-
ter*. https://medium.datadriveninvestor.com/sidewalk-toronto-and-why-
smarter-is-not-better-b233058d01c8
[167] Jutta Treviranus. 2019. The Value of Being Diferent. In Proceedings of the 16th
Web For All 2019 Personalization - Personalizing the Web (W4A ’19). Association
for Computing Machinery (ACM), Article 1, 7 pages. https://doi.org/10.1145/
3315002.3332429
[168] Shari Trewin. 2018. AI Fairness for People with Disabilities: Point of View. CoRR
abs/1811.10670 (2018). arXiv:1811.10670 http://arxiv.org/abs/1811.10670
[169] Shari Trewin, Sara Basson, Michael Muller, Stacy Branham, Jutta Treviranus,
Daniel Gruen, Daniel Hebert, Natalia Lyckowski, and Erich Manser. 2019. Con-
siderations for AI Fairness for People with Disabilities. AI Matters 5, 3 (Dec.
2019), 40–63. https://doi.org/10.1145/3362077.3362086
[170] Juan Camilo Vásquez-Correa, Tomas Arias-Vergara, Juan Rafael Orozco-
Arroyave, Björn Eskofer, Jochen Klucken, and Elmar Nöth. 2018. Multi-
modal assessment of Parkinson’s disease: a deep learning approach. IEEE
journal of biomedical and health informatics 23, 4 (2018), 1618–1630. https:
Data Representativeness in Accessibility Datasets: A Meta-Analysis
//doi.org/10.1109/jbhi.2018.2866873
[171] Radu-Daniel Vatavu and Ovidiu-Ciprian Ungurean. 2019. Stroke-Gesture Input
for People with Motor Impairments: Empirical Results & Research Roadmap.
In Proceedings of the 2019 CHI Conference on Human Factors in Computing
Systems (CHI ’19). Association for Computing Machinery (ACM), 1–14. https:
//doi.org/10.1145/3290605.3300445
[172] Darshali A Vyas, Leo G Eisenstein, and David S Jones. 2020. Hidden in plain
sight—reconsidering the use of race correction in clinical algorithms. , 874–
882 pages.
[173] Katherine K Wallman. 1998. Data on race and ethnicity: Revising the federal
standard. The American Statistician 52, 1 (1998), 31–33.
[174] Katherine K Wallman, Suzann Evinger, and Susan Schechter. 2000. Measuring
our nation’s diversity: developing a common language for data on race/ethnicity.
American Journal of Public Health 90, 11 (2000), 1704.
[175] Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind
the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of
the Association for Computational Linguistics 6 (2018), 605–617.
[176] Danielle Wetherell, Nicola Botting, and Gina Conti-Ramsden. 2007. Narrative
skills in adolescents with a history of SLI in relation to non-verbal IQ scores.
Child Language Teaching and Therapy 23, 1 (2007), 95–113. https://doi.org/10.
1177/0265659007072322
[177] Ryen W White, P Murali Doraiswamy, and Eric Horvitz. 2018. Detecting neu-
rodegenerative disorders from web search signals. NPJ digital medicine 1, 1
(2018), 1–4. https://doi.org/10.1038/s41746-018-0016-6
[178] Ryen W White and Eric Horvitz. 2019. Population-scale hand tremor analysis
via anonymized mouse cursor signals. NPJ digital medicine 2, 1 (2019), 1–7.
https://doi.org/10.1038/s41746-019-0171-4
[179] Meredith Whittaker, Meryl Alper, Cynthia L Bennett, Sara Hendren, Liz Kazi-
unas, Mara Mills, Meredith Ringel Morris, Joy Rankin, Emily Rogers, Marcel
Salas, et al. 2019. Disability, Bias, and AI. AI Now Institute, November (2019).
https://wecount.inclusivedesign.ca/uploads/Disability-bias-AI.pdf
[180] Maria K Wolters, Jonathan Kilgour, Sarah E MacPherson, Myroslava Dzikovska,
and Johanna D Moore. 2015. The CADENCE corpus: a new resource for inclusive
voice interface design. In Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems. 3963–3966.
[181] Dongxin Xu, Jefrey A. Richards, Jill Gilkerson, Umit Yapanel, Sharmistha Gray,
and John Hansen. 2009. Automatic Childhood Autism Detection by Vocaliza-
tion Decomposition with Phone-like Units. In Proceedings of the 2nd Workshop
on Child, Computer and Interaction (WOCCI ’09). Association for Computing
Machinery (ACM), Article 5, 7 pages. https://doi.org/10.1145/1640377.1640382
[182] Ehud Yairi and Nicoline Grinager Ambrose. 1999. Early childhood stuttering
I: Persistency and recovery rates. Journal of Speech, Language, and Hearing
ASSETS ’22, October 23–26, 2022, Athens, Greece
Research 42, 5 (1999), 1097–1112.
[183] Victoria Yaneva, Irina Temnikova, and Ruslan Mitkov. 2015. Accessible Texts
for Autism: An Eye-Tracking Study. In Proceedings of the 17th International ACM
SIGACCESS Conference on Computers & Accessibility (ASSETS ’15). Association
for Computing Machinery (ACM), 49–57. https://doi.org/10.1145/2700648.
2809852
[184] Victoria Yaneva, Irina Temnikova, and Ruslan Mitkov. 2016. A Corpus of Text
Data and Gaze Fixations from Autistic and Non-Autistic Adults. In Proceedings
of the 10th International Conference on Language Resources and Evaluation (LREC
’16). European Language Resources Association (ELRA). https://aclanthology.
org/L16-1077
[185] Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. 2020.
Towards Fairer Datasets: Filtering and Balancing the Distribution of the Peo-
ple Subtree in the ImageNet Hierarchy. In Proceedings of the 2020 Confer-
ence on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT*
’20). Association for Computing Machinery, New York, NY, USA, 547–558.
https://doi.org/10.1145/3351095.3375709
[186] K Lisa Yang and Hock E Tan. 2019. Disability statistics: Online resource for US
disability statistics. Accessed: 2022-01-12.
[187] J Scott Yaruss and Robert W Quesal. 2006. Overall Assessment of the Speaker’s
Experience of Stuttering (OASES): Documenting multiple outcomes in stuttering
treatment. Journal of fuency disorders 31, 2 (2006), 90–115.
[188] Emre Yilmaz, MS Ganzeboom, LJ Beijer, Catia Cucchiarini, and Helmer Strik.
2016. A Dutch dysarthric speech database for individualized speech therapy
research. (2016).
[189] Brit Youngmann, Liron Allerhand, Ora Paltiel, Elad Yom-Tov, and David Arkadir.
2019. A machine learning algorithm successfully screens for Parkinson’s in
web users. Annals of clinical and translational neurology 6, 12 (2019), 2503–2509.
https://doi.org/10.1002/acn3.50945
[190] Hanbin Zhang, Chen Song, Aosen Wang, Chenhan Xu, Dongmei Li, and Wenyao
Xu. 2019. PDVocal: Towards Privacy-Preserving Parkinson’s Disease Detection
Using Non-Speech Body Sounds. In Proceedings of the 25th Annual International
Conference on Mobile Computing and Networking (MobiCom ’19). Association for
Computing Machinery, Article 16, 16 pages. https://doi.org/10.1145/3300061.
3300125
[191] Hui Zheng, Pattiya Mahapasuthanon, Yujing Chen, Huzefa Rangwala, Anya S
Evmenova, and Vivian Genaro Motti. 2021. WLA4ND: A Wearable Dataset
of Learning Activities for Young Adults with Neurodiversity to Provide Support
in Education. Association for Computing Machinery, New York, NY, USA.
https://doi.org/10.1145/3441852.3471220
 Chronically Under-Addressed: Considerations for HCI
Accessibility Practice with Chronically Ill People
Kelly Mack∗
University of Washington
Seattle, Washington
kmack3@uw.edu
Leah Findlater
University of Washington
Seattle, Washington
leahkf@uw.edu
ABSTRACT
Accessible design and technology could support the large and grow-
ing group of people with chronic illnesses. However, human com-
puter interactions (HCI) has largely approached people with chronic
illnesses through a lens of medical tracking or treatment rather
than accessibility. We describe and demonstrate a framework for
designing technology in ways that center the chronically ill expe-
rience. First, we identify guiding tenets: 1) treating chronically ill
people not as patients but as people with access needs and exper-
tise, 2) recognizing the way that variable ability shapes accessibil-
ity considerations, and 3) adopting a theoretical understanding of
chronic illness that attends to the body. We then illustrate these
tenets through autoethnographic case studies of two chronically
ill authors using technology. Finally, we discuss implications for
technology design, including designing for consequence-based ac-
cessibility, considering how to engage care communities, and how
HCI research can engage chronically ill participants in research.
KEYWORDS
chronic illness, accessibility, disability studies
ACM Reference Format:
Kelly Mack, Emma J. McDonnell, Leah Findlater, and Heather D. Evans. 2022.
Chronically Under-Addressed: Considerations for HCI Accessibility Practice
with Chronically Ill People. In The 24th International ACM SIGACCESS
Conference on Computers and Accessibility (ASSETS ’22), October 23–26, 2022,
Athens, Greece. ACM, New York, NY, USA, 15 pages. https://doi.org/10.1145/
3517428.3544803
1 INTRODUCTION
Billions of people around the world [18, 53] are diagnosed with
chronic illness, broadly defned as a range of conditions and diag-
noses that impact functioning and are not expected to go away or
be immediately fatal [34, 49, 115, 122, 124, 149]. Many chronically
∗ Both authors contributed equally to this research.
This work is licensed under a Creative Commons Attribution International
4.0 License.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3544803
Emma J. McDonnell∗
University of Washington
Seattle, Washington
ejm249@uw.edu
Heather D. Evans
University of Washington
Seattle, Washington
hdevans@uw.edu
ill people have symptoms that alter their daily lives, and disability
activism [65, 117] and academic theorizing [42, 72, 148] are begin-
ning to integrate chronic illness into their approaches. However,
despite an active focus on chronic illness in human-computer inter-
action (HCI) health research (e.g., [54, 61, 81, 89]), chronic illness
remains conspicuously underrepresented in HCI accessibility work
[86] (for a few exceptions, see e.g., [19, 57, 69, 85]). We identify
an opportunity for HCI accessibility practitioners to understand
the access needs of chronically ill people and to create technology-
based solutions that are not rooted in medicalized views of chronic
illness.
In this paper, we articulate an opportunity for future HCI acces-
sibility research to work with and support chronically ill people.
To do so, we present three tenets to guide researchers’ approaches
to chronic illness: 1) move beyond medical framings to understand
people with chronic illness as having access needs and valuable
expertise, 2) consider that the variability of ability that many chron-
ically ill people experience presents unique accessibility needs, and
3) adopt a theoretical approach to chronic illness that attends to
bodily and sociocultural experiences. We then apply these tenets
to three autoethnographic case studies about the authors’ own ex-
periences with technology use, demonstrating how our tenets can
be used to surface design considerations for chronically ill users.
By placing disability studies, HCI, and our lived experience as
chronically ill technology users in conversation, we introduce a new
paradigm for designing accessible technology. This shift includes
viewing access as produced by both a user’s innate abilities and the
physiological consequences doing an action causes, which we call
a consequence-based approach to accessibility. Further, it encour-
ages researchers to consider technology design for community use
and alter traditional HCI methods to better match chronically ill
participants’ access needs. We also emphasize that approaches to
technology design for chronically ill people need to be grounded in
community knowledge and can be contextualized within disability
studies and activism.
In summary, with respect to designing technology for chronically
ill people, we contribute 1) three core tenets to guide research, 2) a
in-depth authethnographic exploration of how these tenets reveal
opportunities to understand and design technology for chronically
ill people, and 3) considerations for HCI accessibility practice when
engaging chronically ill people.
ASSETS ’22, October 23–26, 2022, Athens, Greece
2 BACKGROUND AND RELATED WORK
To contextualize our work, we situate our defnition of chronic
illness, explore how HCI has approached work with chronically ill
people, and introduce guiding concepts from disability studies.
2.1 What We Mean by Chronic Illness
In this paper, we draw on work from disability studies scholarship
to broadly defne a chronically ill person as one who has a condition
that: impacts functioning, is not expected to go away or be immedi-
ately fatal, may be ameliorated through treatment and, particularly
when left untreated, can be life-limiting [34, 49, 115, 122, 124, 149].
Conversations around chronic illness and disability often overlap
- indeed many people identify as both chronically ill and disabled
[119]. While we do not intend to take on questions of chronic illness
or disability identity formation, this overlap guides us to engage
with disability studies as a source of useful guiding theory and to
see chronic illness as relevant to HCI accessibility work. However,
we also highlight areas where chronically ill people’s experiences
diverge from mainstream conceptualizations of disability [119],
motivating the need for a chronic illness specifc approach to tech-
nology design. Billions of people globally are diagnosed with at
least one chronic health condition [18, 53] (a growing group in the
wake of COVID-19 [142]), leading to a wide range of experiences
and varied identifcation with chronic illness. We, however, are
primarily interested in how shared functional aspects of chronic
illness could be better considered within accessibility. While our
framework may resonate more strongly with people who identify
as chronically ill, it may also be relevant to many others.
2.2 Chronic Illness and HCI
Chronic illness has received uneven attention across subfelds of
HCI - it is a signifcant topic in HCI health and online communities
research but is scarcely engaged in accessibility contexts. Specif-
ically, the primary foci of current HCI health work on chronic
illness include: exploring the diferent information needs and prac-
tices of patients and providers [15, 37, 81, 112, 127, 131, 141], how
people talk to their support networks [5, 14, 89, 95, 114] how to
support pediatric patients, their parents, and providers in efective
communication [54, 59–62, 75], and how patients gain the knowl-
edge to manage their conditions [20, 24, 63, 120, 150]. This body
of work considers chronically ill people in relation to the medical
care they pursue, often primarily referring to them as ‘patients’,
and therefore proposes technology solutions within this medical
context. An additional focus of HCI work with chronically ill people
is self-tracking, developing tools to track symptoms [76, 101, 116],
treatments [7, 21, 158], and medically-necessary lifestyle changes
[84, 136]. This work often includes medical professionals and is
geared toward helping patients comply with prescribed treatment.
Another avenue of HCI research with chronically ill people outside
of accessibility frameworks focuses on how chronically ill people
connect with and support others via social media communities
[41, 43, 66, 82, 90, 92, 129, 144, 159]. This body of research consid-
ers and ofers solutions for many areas where chronically ill people
can be better supported when seeking medical care, but rarely cen-
ters access needs that are not immediately connected to the clinic,
like accommodations for work or social life.
Mack and McDonnell et al.
There is a small, but growing body of work that situates chronic
illness in relation to accessibility and the broader disability com-
munity. ASSETS has not historically published much work that
engages chronic illness - only a small set of papers include par-
ticipants with chronic illnesses, often focusing on older adults or
rehabilitative technologies (e.g., [3, 17, 25, 26, 73, 109, 157, 160]).
Some ASSETS publications have considered chronically ill people
as having access needs, exploring how people negotiate access at
work [85], considering how to make disability activism accessi-
ble to chronically ill people [19], and centering chronic illness in
theorizing around the role disability studies ought to play in fu-
ture HCI accessibility research [57, 91]. Outside of ASSETS, HCI
work that considers chronic illness within the contexts of disabil-
ity and accessibility remains sparse. Research on disability-related
activist movements has included chronically ill people’s perspec-
tives [1, 80], recent conference workshops on accessibility research
have explicitly considered chronic illness [4, 140], and researchers
have considered the particular access needs of chronically ill peo-
ple on dating apps [121], in the workplace [47], during research
studies [87] and in public places [69]. Considering the prevalence
of chronic illness, this body of work is notably underdeveloped
relative to other foci of accessibility research [86, 152]. Though this
handful of papers examine the access needs of chronically ill people
in specifc contexts, no work yet theorizes about the broader design
considerations needed to make technology for this group. We seek
to grow this body of work by articulating a set of tenets to guide
future accessibility research with chronically ill people.
2.3 Core Concepts from Disability Studies
Within the feld of disability studies and activist communities, schol-
arship by disabled and chronically ill people provides a critical lens
and crucial background to our thinking on how HCI accessibility
practitioners should approach chronic illness. Our work applies
disability studies topics to HCI scenarios, building on the tradition
set by Mankof et al.’s 2010 invitation to engage with models of
disability [91], Bennett et al.’s integration of interdependence into
HCI thinking [12], Ringland et al.’s call to see mental-ill health as
a form of psychosocial disability that can be addressed outside of
clinical contexts, [126], Williams et al.’s framing of crip HCI [151],
and Hofmann et al.’s exploration of crip time [57]. Here we provide
a brief overview of core disability studies concepts that we will
utilize when considering how to build non-medicalized technology
for chronically ill people, namely: social and medical models of
disability, interdependence, access intimacy, and crip time.
One of disability studies’ central pursuits is to name and ana-
lyze the efects of two dominant frameworks for understanding
disability: the social and medical models [113]. The medical model
of disability characterizes deviation from physical and/or intellec-
tual norms as undesirable defects that medical intervention can
eliminate, augment, or cure, motivated by the belief that a better
future is one without disability [30, 133]. The classical counterpoint
to the medical model is the social model of disability, which names
disability as a natural and vibrant part of human diversity and as
a basis for historic and current systemic oppression. Rather than
focusing on cure, proponents of the social model call for changes
to external factors that produce disability (e.g., buildings without
Chronically Under-Addressed: Considerations for HCI Accessibility Practice with Chronically Ill People
ramps, discriminatory policies) [113, 133]. Notably, early disability
studies scholarship, which articulated the social and medical mod-
els, did not center chronic illness in its analysis. Initial adherents
of the social model drew a distinction between impairments (i.e.,
diferences in functioning) and disability (i.e., context-specifc and
social dynamics that create barriers for participation by diferently
functioning individuals) to help combat what Joel Reynolds calls
the “the ablest confation” of the concept of disability with “pain,
sufering, hardship, disadvantage, morbidity, and mortality” [124].
More recent scholarship within critical disability studies and
activist theorizing provide key concepts to analyze and understand
chronic illness. While the historic push for disability rights often
focused on independence [104], disability studies scholars have
begun to theorize about the role of dependence in disabled people’s
lives [77]. Activist scholars have adopted the framing of interde-
pendence as “the state of being dependent upon each other” [65]
which emphasizes the networks of connections and care that peo-
ple provide each other, rather than positioning one person as the
sole recipient or provider of care. Disability justice activists have
explored how interdependence reveals the ways sick and disabled
people form care networks to provide support and access for each
other [99, 117, 118]. A related concept, coined by Mia Mingus, is
that of access intimacy, or “that elusive, hard to describe feeling
when someone else ‘gets’ your access needs” [98]. Access intimacy is
a meaningful dimension to add to considerations of accessibility
as it helps make visible the relationships and shared context that
shape how access is felt and received.
Disabled scholars developed another key concept to our work,
“crip time,” which theorizes about the diferent temporalities (or,
experiences of the passing of time) in which sick and disabled peo-
ple operate. Allison Kafer, in frst formalizing crip time, imagines
its power: “rather than bend disabled bodies and minds to meet the
clock, crip time bends the clock to meet disabled bodies and minds”
[72]. Ellen Samuels, a chronically ill disability studies scholar, calls
attention to the ways that crip time simultaneously provides tools
to imagine a more accessible future while highlighting that rigid,
normative expectations of life paces can be sites of painful inaccessi-
bility [128]. These concepts provide avenues for thinking critically
about the ways that disabled and chronically ill people often ar-
range their lives diferently from nondisabled people and shape our
analysis.
2.4 Positionality
This work was deeply infuenced by authors’ experience with
chronic illness, interactions with medical systems, and (for some)
moving through the world with a non-normatively functioning
body. Three of the four authors identify as chronically ill and all
identify as white, cisgender, women.
3 DESIGN TENETS FOR CREATING
TECHNOLOGY FOR PEOPLE WITH
CHRONIC ILLNESSES
We present three tenets which outline necessary perspectives to
shape technology design for people with chronic illnesses. First,
technology designers must view people with chronic illnesses as
having access needs and valuable expertise rather than only as
ASSETS ’22, October 23–26, 2022, Athens, Greece
patients. Second, chronic illness causes high variability in ability,
which is crucial to consider when designing technology to meet
chronically ill people’s access needs. Third, this work must be done
using a model of disability that accounts for both physical and
mental experiences of impairment while also recognizing disabling
socio-political factors.
3.1 Tenet 1: Beyond Patients
We must view people with chronic illnesses as more than
medical patients, but rather people with valuable expertise
and non-medical access needs.
Much of the existing body of HCI scholarship around chronic
illness adopts a health, rather than accessibility, framework (see 2.2).
Under a medical lens, chronically ill people are primarily viewed
as patients with technology needs defned by medical care and
symptom management. However, we call for HCI practitioners to
contest the dominance of medicalization and emphasis on patient-
hood when designing technology for people with chronic illnesses.
Longstanding critique by feminist [31, 83], and queer [110] activists
calls attention to the ways that labelling people as “patients” takes
away their agency and imposes a set of assumptions around what
patients ought to want, do, and need. The label “patient” also estab-
lishes a clear power hierarchy, implying a subordinate relationship
to a more knowledgeable and powerful clinician [39]. Viewing peo-
ple with chronic illnesses primarily as patients suggests that they
can be best understood in a medical context and situates them as
recipients and dependents of medical practitioners’ expertise. On
the other hand, approaching people with chronic illnesses with an
accessibility lens views them as people with access needs and cre-
ates room to center individuals’ agency and knowledge, countering
epistemic violence [156].
A medical, patient-centric approach often obscures the deeply
contentious relationship many people with chronic illnesses have
with the medical feld. While medical treatments, testing, and guid-
ance can be critical to chronically ill people’s quality of life, the
medical feld is often simultaneously hostile to chronically ill people
[10, 51, 52]. For example, it frequently takes years to get formal
diagnoses for many chronic illnesses [39], patients are routinely
not believed by medical professionals [78, 94, 102], and complex
medical care is often prohibitively expensive [48, 56]. These experi-
ences are exacerbated when people with chronic illnesses are other-
wise marginalized because medical racism, sexism, anti-queerness,
ableism, classism, fatphobia, and other biases harm people’s ability
to access care and be treated with dignity [32, 70, 111, 147]. Future
HCI work must understand that while medical care and assessment
is crucial for many chronically ill people, it can also be a primary
site of trauma, discrimination, and disbelief. Discussion of and en-
gagement with medical systems must be done with caution and
recognize this fraught history. This motivates our focus on non-
medical access needs that remain under-considered within HCI
research.
At the same time, the knowledge shared outside of medical con-
texts makes clear that, individually and in community, chronically
ill people hold vast expertise derived from both their embodied expe-
riences and navigating the world with a chronic illness. Diagnosis-
specifc and general chronic illness social media communities are
ASSETS ’22, October 23–26, 2022, Athens, Greece
abundant (e.g., [11, 41, 129]), and they provide a place to share in-
depth knowledge about living with a chronic illness. While much
discussion centers on how to live with and acquire care for illness
(e.g., symptom and fare identifcation and management, possible
diagnoses, how to navigate the medical system), people also share
information and advice to meet non-medical access needs (e.g., de-
veloping horizontal workstations, suggesting how to disclose access
needs on a date, preparing meals that don’t trigger dietary restric-
tions) [11]. Indeed, there are myriad individual and group examples
that demonstrate the sophistication of this expertise, including a
recent reconsideration of graded exercise therapy as a standard of
care for myalgic encephalitis/chronic fatigue syndrome (ME/CFS)
after ME/CFS advocacy groups demonstrated that it is a harmful
practice [8, 143]. HCI researchers should value this individual and
community-based knowledge.
To conduct HCI accessibility work on chronic illness, researchers
must go beyond patient framings to view chronically ill people as
having access needs and valuable expertise to shape accessible
technology design work. This view challenges the assumption that
medical providers should always be consulted as subject matter ex-
perts, while frmly centering chronically ill people as the relevant,
necessary experts that can guide the development of accessible
technology. This reframing is also necessary to re-render HCI ac-
cessibility work as relevant to chronically ill people. If researchers
perceive chronically ill people as primarily patients, primarily med-
ical technologies emerge as relevant supports. However, if we view
chronically ill people as having a wide range of access needs that
are not well-met in their daily lives, HCI practitioners are well-
positioned to create non-medical tools to improve accessibility. For
example, access-need driven HCI work with chronically ill peo-
ple may explore how technology could adapt to a user’s varied
cognitive abilities, develop research practices that better support
someone with fuctuating capacities, and examine how existing
accessible technologies could be customized to meet chronically ill
users’ needs.
3.2 Tenet 2: Variability of Ability
The experience of chronic illness is diverse and inconsistent,
even for an individual from day to day; consequently, we
have to view access not only in terms of capability to com-
plete an action, but also in terms of its repercussions (e.g.,
consequences such as worsening symptoms).
Chronic illness often causes varying levels of ability, afecting
how chronically ill people move about the world, including what
technology they use. When people’s bodies have vastly diferent
abilities over time, it takes creativity and planning to go about ev-
eryday life [9, 38, 106, 117]. For example, someone with fuctuating
fatigue may use mobility or other technology aids some days or
times in the day, but not others [9, 106]. Like other disabilities,
chronically ill people’s abilities and access needs vary between
people. However, what is especially critical in examining the expe-
rience of chronic illness is understanding the variability of abilities
within an individual. We break down this phenomenon by, frst,
categorizing factors that determine (and vary) ability into two main
categories: individual baseline fuctuations and action-determined
variability. Then, we propose a view of accessibility that is key for
Mack and McDonnell et al.
designing technology for people with fuctuating abilities: design-
ing for the consequences of actions rather than solely for static
capabilities.
Individual Baseline Variability. Beyond interpersonal varia-
tion in experience with chronic illnesses, a single individual can
experience internal fuctuations in ability. Many people’s chronic
illness experiences include “fares”, or an overall exacerbation of
symptoms for an extended period of time [16, 88, 138, 154]. These
fares, as well as shorter periods of fuctuation (e.g., a bad symptom
day) can be triggered by unknown or hard-to-control circumstances
[13, 51, 97, 154]. For example, environmental factors (e.g., a heat
wave, pollen, pollution/smog, season changes) or other physical
experiences (catching the fu, menstruation) can trigger an overall
higher level of disruptive symptoms and, consequently, a lower
level of capability to perform daily tasks [40, 96, 97, 154]. These
baseline fuctuations can occur rapidly, and therefore technology
that is designed for people with chronic illnesses must be usable at
a variety of ability levels to meet the user’s current access needs.
Action-determined variability. A chronically ill person’s abil-
ity levels frequently change after performing actions. While ar-
guably, every person enters a diferent state after performing an
action (e.g., after a run, a person might feel more tired) this dif-
ference in abilities/state can be extreme for people with chronic
illnesses (e.g., post-exertion malaise [22]). For example, a person
without a chronic illness might take a shower and detect no no-
ticeable diference in state. On the other hand, the challenges of
showering with a chronic illness are thoroughly discussed (and
even meme’d) among chronic illness communities because they
often result in extreme fatigue, overall malaise, or other symptoms
[2, 50, 155]. Since actions may trigger lasting, negative symptoms,
this can result in a cumulative efect that leaves chronically ill
people with disruptive symptoms and a low capacity to perform
tasks by the end of a day. Therefore, technology design needs to
consider not just the abilities a person begins with, but the abili-
ties they might have after performing an action, with or without
technological support.
The efects of variability: designing for consequences. Be-
cause signifcant fuctuations of ability pervade many chronically
ill people’s lives, they often have to map out their days based on
the expected consequences of each action they plan to take [100].
A common metaphor used within the chronic illness community
for this form of variable consequence management is the “Spoon
Theory” [29, 100]. This metaphor for understanding chronic illness,
coined by Christine Miserandino, represents capacity or energy
with “spoons” and explains that people have to carefully plan what
they spend their spoons on in a day, since they are often in short
and inconsistent supply; because of baseline variations, like a fare,
the amount of spoons you can spend in a day may be diferent on
Monday than Tuesday [100]. Due to the variety in abilities and
symptoms an individual can experience, the number of spoons an
action takes cannot be perfectly estimated. Even for the same per-
son, the impact of the same action can vary drastically from 10 AM
to 10 PM, though the consequences they incur for that action may
be perpetually higher than they are for non-chronically ill people.
In deciding how to allocate spoons, chronically ill people perform a
complex cost-beneft analysis, informed by the time they’ve spent
living with a condition, to predict the likely costs (e.g., symptoms)
Chronically Under-Addressed: Considerations for HCI Accessibility Practice with Chronically Ill People
of performing a task compared to the benefts they will receive.
Therefore, we argue that to understand accessibility in the context
of chronic illness, we must account for the consequences an action
causes. Under this approach, we frame the accessibility of a task as
not solely in terms of an individual’s capability to perform a task,
but rather the ability to perform a task and remain in an “acceptable”
state afterwards.
3.3 Tenet 3: Include the Body
Research with people with chronic illnesses must be done
using a model of disability that accounts for both the physi-
ological and sociopolitical barriers they face.
Numerous disability studies scholars have explored the ways
in which those living in non-normative bodyminds may experi-
ence limitations from both physiological impairments and socially
constructed dynamics of exclusion [28, 34, 115, 119, 122, 124, 149].
In her foundational essay, Susan Wendell argues that for the “un-
healthy disabled”—or people who are chronically ill and experience
frequent pain, fatigue or other forms of discomfort—a social model
view of disability focused on curing ableism disregards a core part
of their disability experience. She highlights the fact that many
people “experience physical or psychological burdens that no amount
of social justice can eliminate,” and calls for an approach to disability
that does not seek to avoid the realities of physiological impair-
ment [148]. Motivated by Wendell, we argue that researchers must
move beyond the currently discussed social and medical models of
disability toward an approach that attends to both embodied and
sociopolitical aspects of chronic illness.
Disability scholars have critiqued the social model [52, 132, 133]
and developed new ways of thinking about disability that cen-
ter the interplay between individual experiences of impairment
and broader society and disability politics. For example, the politi-
cal/relational model proposed by Alison Kafer “neither opposes nor
valorizes” medical care, but makes space for ‘‘the possibility of si-
multaneously desiring to be cured of chronic pain and to be identifed
and allied with disabled people” [72]. This model makes space to
see chronically ill people as political subjects while not needing to
cast aside the bodily realities of impairment that have historically
been ignored under social model politics.
Other scholars have explored ways that living with diferences in
functioning can generate deep, visceral forms of knowledge avail-
able only to others who share the same experience. Tobin Siebers
explains that embodied knowledge arises when “situated knowl-
edge adheres in embodiment. The disposition of the body determines
perspectives, but it also spices these perspectives with phenomenolog-
ical knowledge–lifeworld experience–that afects the interpretation
of perspective” [134]. In essence, knowledge does not solely come
from a social location, but from the particular, physical experiences
of living in a body. Siebers calls for disability theory that engages
embodied expertise, not only as an object of analysis but as a concep-
tual tool that can strengthen design practices and enrich analytical
capacity [135].
We combine these ideas to articulate a theoretical approach to
understanding the experience of people with chronic illnesses. To
adopt a more nuanced and comprehensive approach to chronic
ASSETS ’22, October 23–26, 2022, Athens, Greece
illness, we believe researchers should center the embodied experi-
ences that often characterize chronic illness. This framing balances
the tensions between seeking care for unwanted symptoms and
valuing disability as part of human expression. We hope this epis-
temological shift away from a purely social model approach makes
space for HCI accessibility work that can account for people’s bodily
realities without defaulting to a medicalized approach.
4 CASE STUDY APPLICATIONS OF
DESIGNING FOR PEOPLE WITH CHRONIC
ILLNESSES
In the following section, we present three autoethnographic case
studies from two chronically ill authors to show how diferent
types of technology can be used to negotiate access in diferent
social and work scenarios. We then demonstrate how our tenets
can make sense of these experiences and, following that analysis,
highlight potential directions and considerations for technology
development.
4.1 Background and Methods
4.1.1 Collaborative Autoethnography. Collaborative autoethnog-
raphy is a methodological approach using autobiographical data
as the subject of ethnographic analysis conducted by a group of
researchers [23]. Autoethnography is a well-established method
among disability studies scholars who have both experienced and
studied the social and structural dynamics of chronic illness, dis-
ability, or long term pain [45, 148, 161, 162] and continues to be
employed by similarly situated researchers today [103, 107, 125].
Conducting autoethnography in collaboration produces rich anal-
yses from multiple ‘insiders’ perspectives of complex health pro-
cesses [23, 105, 108, 137, 145]. Inspired by this history of chronically
ill people’s engagement with autoethnography, we present three
case studies derived from our own autoethnographic refection.
We focus on our own experiences and interactions as two chroni-
cally ill people to avoid being extractive of broader communities
throughout our cases [44].
4.1.2 Background. The frst authors, McDonnell and Mack, recog-
nized rich examples in their lives around utilizing technology to
navigate their own fuctuating access needs associated with chronic
illness symptoms. They therefore chose to employ collaborative
autoethnography to critically analyze their everyday experiences
and refect on them as case studies.
McDonnell and Mack started their doctoral studies at the Univer-
sity of Washington in the same year. While they met as colleagues
collaborating on accessibility research, the experiences they recount
here are primarily shaped by a deep friendship that developed over
the course of that collaboration. Mack was diagnosed in 2016 with
a chronic illness that results in fuctuating symptoms that include
motion sickness in the form of dizziness and nausea that can be
triggered by physical movements as well as visual stimuli. She
has overall malaise that varies, sometimes feeling perfectly fne,
other times feeling ill upon waking up. Although McDonnell has
navigated signifcant dietary restrictions since she was young, her
identifcation with chronic illness shifted in 2019 after what she
thought was post-surgical recovery became unexplained symptoms
ASSETS ’22, October 23–26, 2022, Athens, Greece
that took years to diagnose. McDonnell’s symptoms include unpre-
dictable malaise, an inconsistent ability to be upright and active,
brain fog, heat intolerance, and fatigue.
To develop their cases McDonnell and Mack frst independently
generated a set of scenarios where technology was either inacces-
sible and/or allowed them to meet their access needs, identifying
individual examples and refecting on instances where they used
technology to support each other. They then met and discussed
areas of overlap between their scenarios, ultimately selecting three
cases for their variance, rich engagement with technology, and
interaction between the two authors. Throughout this process they
referenced their shared messaging history to provide more details
about interactions. The full set of authors reviewed these refections,
probing for more details and explanations when needed. Collec-
tively the authors engaged in iterative discussions and analyses,
producing the results presented here.
4.2 Case 1: TikTok Sharing and Consumption
Our frst case examines Mack and McDonnell collaboratively cre-
ating access to social media content that is inaccessible to Mack.
Their experience demonstrates a community-based solution to a
social (nonmedical) access issue and highlights consequence-based
accessibility.
4.2.1 The Scenario. McDonnell is both deeply hooked on the social
media platform TikTok and fond of sharing videos that she fnds
amusing. However, Mack cannot watch all TikToks because shaky
camera movement makes her sick. Therefore, when McDonnell
wants to share a TikTok with Mack, she pauses to assess how much
motion is in it, before copying the link over to Facebook Messenger
and writing up a motion description of the video [123]. Although
they have since discussed how to best craft motion descriptions,
McDonnell began providing descriptions without prompting when
she began sharing TikToks, paralleling how she shares image de-
scriptions of visual memes with a blind friend. This description
explains how much motion is in the video and may also include
consumption guidelines (e.g., wait until someone says “get my cof-
fee” to look at the video) to make semi-accessible videos watchable
or context on why Mack might watch it (e.g., it is one of her inter-
ests, or McDonnell thinks it’s hilarious, required viewing). Example
motion descriptions McDonnell has shared with Mack are:
“This seems very up your alley though has a lot of
motion. The camera is steady when it’s still but moves
side to side to track the dancers (in fts and starts
though, like it moves, stays, they move out of frame
and it then follows, not continuous tracking), and
they’re dancing at a reasonably close zoom so all the
spins constitute motion on the screen. I will audio
describe and pause for you tomorrow if you want “
“Steady cam tho with a lot of jump cuts- you can look
at a still at the start and then away for the rest of the
video and get 95% of it”
When Mack gets that message, she reads the motion description
and decides if it is something she wants and can aford to watch at
that moment. For example, when she receives the frst description,
Mack decides that since she usually feels best in the morning, she
Mack and McDonnell et al.
can risk watching this video after she wakes, particularly because
she knows she’ll enjoy it. It does make her feel slightly ill. Mack
then opens the second video and fnds that McDonnell was more
cautious than she needed to be, and she was able to watch the
TikTok video without triggering symptoms.
4.2.2 Applying the tenets. Whether or not Mack watches a spe-
cifc video at a given time cannot be determined by either party
alone: McDonnell and Mack both provide key information to in-
form these decisions. This process is deeply interdependent and
social in nature.
Because her symptoms fuctuate hourly (Tenet 2), Mack’s deci-
sion to watch a TikTok at any given time needs to consider sev-
eral interwoven factors. She essentially performs a risk-assessment
where her current symptoms and the described amount of motion
determines the risk, the level of novelty of the content determines
the potential beneft, and missing events or feeling very ill later in
the day determines the potential cost. Mack can make this calcu-
lation reasonably accurately, thanks to her expertise derived from
years of lived experience around what will and won’t trigger her
symptoms.
What videos to send and what to describe in these videos is
highly situated in understanding how elements of the social en-
vironment impact Mack’s physical abilities and symptoms (Tenet
3). To make motion descriptions efective, Mack and McDonnell
rely on building a shared understanding of Mack’s symptoms and
how to categorize motion. They work together to defne a shared
vocabulary to consistently describe elements of the social environ-
ment (e.g., what is “unsteady camerawork”) so that Mack can best
predict her physical response to the stimuli. This process involved
demonstrative examples and considerable trial and error; to this
day, there are still elements of guessing at what kinds of visual
stimuli would be accessible and how to convey risk.
Mack’s deep, embodied knowledge and McDonnell’s eforts to
learn her access needs are crucial to this process (Tenet 1). Mack
gained an understanding of how her symptoms change in response
to diferent stimuli over years of self-refection and trial and error,
sometimes accidentally triggering negative symptoms. This self-
knowledge is crucial to craft social accommodations. As McDonnell
is her only friend who regularly sends motion-described TikToks,
it is the only way Mack has to safely access this content – there is
no existing external mechanism to learn about the motion stimuli
in a TikTok in advance. However, McDonnell is not acting as a
visual/motion interpreter for Mack - rather, it is an act of friendship
that deeply considers access . In practice, McDonnell develops the
knowledge of what to include in motion descriptions from being
a curious friend who watches a lot of content with Mack, and
also from Mack being very open with sharing her physiological
reactions. They rely on access intimacy and interdependence to
translate Mack’s expertise into a social workfow that meets her
access needs.
4.2.3 Potential for Technology Support: Contextual Awareness and
Customization. This example highlights opportunities and consid-
erations around how technology could better support people with
fuctuating symptoms that can be triggered by external stimuli
like motion. First, we found that context (i.e., Mack’s current state
and her future plans) was critical in determining what content she
Chronically Under-Addressed: Considerations for HCI Accessibility Practice with Chronically Ill People
would consume and when. Therefore, gathering users’ contextual
information through smart device sensors (e.g., microphones, pulse
sensors) or other information logged in personal devices (e.g., cal-
endar events)1 could be a promising application for future work
[27, 74, 130]. However, Mack’s example demonstrates the sophis-
tication of expertise needed to identify a user’s current state and
predict the impact of content consumption, suggesting that so-
lutions may need to include a human-in-the-loop to ensure that
sensed contextual information is adequately interpreted. Finally,
Mack’s experience of chronic illness is also unique from others, and
the heuristics she has for what videos may be accessible are partic-
ular to her life and body, indicating that personalization would be
key if designing accessible technology for this scenario.
Further, this case raises questions of how one might develop
machine learning models to increase access when target users can’t
label training data and face signifcant consequences when using
inaccurate models. While recent developments in customizable ma-
chine learning models (e.g., few-shot learning [146]), may seem
well-suited to the questions of describing motion stimuli or identify-
ing accessible videos, this example challenges several core machine
learning practices. Even few-shot learning requires that users pro-
vide a training set and then give feedback to iteratively improve
models. For Mack to independently curate a dataset of inaccessible
TikToks or types of motion, she would likely have to slowly submit
examples of videos that trigger her symptoms throughout daily life
or undertake video labeling sessions that are all-but guaranteed to
make her sick. An alternative model of data-labeling could explore
a communal approach, where others (in this example, McDonnell)
could curate a training set for Mack. This is not a panacea, as it
takes time, transparency, and trust to train proxy data labelers and
adds uncertainty to training data - though McDonnell can often
make reasonable calls about what is clearly accessible to Mack, she
does not live with Mack’s symptoms. Additionally, the process of
data labeling may also be a socially-untenable ask to make of others.
Further, assessing model performance and providing feedback to
improve a model poses signifcant risk, as the tolerable error rate
is very low. Finally, since Mack’s decision to watch or not watch a
video depends on many interwoven, nuanced factors, this case raise
interesting questions for machine learning around how to collect
detailed feedback from a user without burdening them.
4.3 Case 2: Hacking Text-to-Speech Technology
We now examine a case around McDonnell and Mack’s use of
text-to-speech (TTS) technology to improve access during their
graduate studies. This common tool had the fexibility required to
support two diferent sets of access needs in performing the same
task: allowing McDonnell and Mack to continue reading while
symptomatic.
4.3.1 The Scenario. Mack’s dizziness varies day to day. She fnds
that 10 minutes of uninterrupted reading consistently makes her
dizzy. Consequently, she started using Text-to-Speech (TTS) tech-
nologies to read without triggering dizziness. Sometimes, Mack
uses a screen reader as a TTS engine, since it has very fast reading
speeds, is easily turned on and of, and doesn’t require internet
1 Note that a heavily sensing-based solution cannot be built without careful considera-
tion for user privacy.
ASSETS ’22, October 23–26, 2022, Athens, Greece
access. However, Mack is not a “traditional” screen reader user: she
uses her eyes to identify the paragraph of interest, highlights the
text with the mouse, and then activates the screen reader. Other
times she uses an online TTS tool, NaturalReader2 , designed for
sighted users. Some of the visual interactions are useful to her, such
as clicking where in a document to start reading, but she has to turn
of others due to her motion sensitivity, such as highlighting each
word as it is read. While this tool limits listening speed, it works
well on PDFs, which are often not fully screen reader accessible.
Context often determines which tool Mack uses. For example,
one day, while attending a meeting, her peer sent an abstract for
her to read. Since an abstract is short, she chose to read this with
her eyes, and it only made her slightly dizzy. However, she was
then asked to review an interview protocol draft. This document
spanned multiple pages, and given her existing symptoms Mack
chose to consume the content with a screen reader. Since it was a
group meeting with multiple people, she pulled out earbuds and put
one earbud in, leaving the other ear uncovered to ensure she could
still hear her colleagues while listening to the protocol. She felt
comfortable using the earbuds without judgment or explanation
since the meeting attendees knew about her chronic illness.
Meanwhile, McDonnell does not experience any uptick in symp-
toms directly related to reading. However, her fuctuating fatigue,
malaise, and brain fog can make both the physical efort of sitting
upright enough to read a PDF on her computer and the cognitive ef-
fort of staying focused on a document prohibitively difcult. When
McDonnell mentioned that she was struggling to balance work and
fatigue, Mack recommended NaturalReader. Initially McDonnell
used this tool sporadically, but eventually it became her default
reading method. As someone who is not a skilled screen reader
user and can consistently navigate interfaces visually, a TTS tool
alone serves as a signifcant access tool.
Through more consistent use, McDonnell has discovered that
she uses the tool diferently when she needs to get reading done
while feeling so physically unwell that she can’t be upright than
when symptoms are impacting her ability to sustain focus. For ex-
ample, McDonnell was taking a graduate seminar, which included
dense readings, during an academic term where she was experi-
encing frequent symptom fares. If she was feeling unable to work
from her desk, she would pivot to uploading the week’s reading
to NaturalReader, putting in her headphones, pressing play, and
laying down on the foor or couch to listen. This allowed her to
continue working and increased the likelihood that she’d be able to
complete other work later in the day. If she was instead trying to
complete seminar readings while dealing with brain fog, she would
load the reading into NaturalReader and then simultaneously lis-
ten to audio output while using the tool’s sentence-highlighting
feature. Multimodal output and consistent pace allowed her to get
through a heavy reading load while brain fog made staying focused
on reading difcult.
4.3.2 Applying the Tenets. This case demonstrates ways that Mack
and McDonnell address the access needs that arise from their
chronic illnesses in nonmedical contexts (Tenet 1). Seeing McDon-
nell and Mack as having access needs de-medicalizes their issues
and allows widely available, nonmedical tools to be a part of the
2 https://www.naturalreaders.com/online/
ASSETS ’22, October 23–26, 2022, Athens, Greece
solution. Further, McDonnell learning about NaturalReader from
Mack exemplifes the common practice of communities creating
and sharing valuable expertise with each other about navigating
through all areas of life with a chronic illness [11, 92, 117]. Addi-
tionally, McDonnell’s case provides insight into how individual
expertise of chronically ill people evolves: she learned about her
own access needs and how to manage them through months of
feeling sick, trying new workfows (i.e., using NaturalReader when
she was sick, lying on the foor), and recognizing where else they
could be useful in her life (i.e., using NaturalReader as a focusing
mechanism). Both experiences exemplify the creative workfows
and rich insights that can be generated by disabled or chronically
ill people hacking technology [55, 139].
Mack and McDonnell use TTS in response to the reality of their
varied abilities (Tenet 2), despite neither of them being the “typical”
target users of TTS systems (e.g., people who are blind, people with
dyslexia or other common print-related disabilities). For Mack, TTS
works to prevent and manage symptoms. For McDonnell, TTS is
a more accessible option than visual reading when she is sympto-
matic, though she often reads visually without consequence when
she is non-symptomatic. In fact, both authors’ experiences high-
light an interesting perspective on achieving access for chronically
ill people: they both can physically read with their eyes, but fnd
reading with TTS to be more accessible. Therefore, for McDonnell
and Mack, oftentimes access is about utilizing modalities that lead
to less friction during or after the activity more so than working
around an inability to perform an activity.
On top of the physical variability that determines technology
use, Mack and McDonnell’s social and environmental contexts are
also key. By default, Mack prefers to read any text longer than a
brief email via TTS as it greatly reduces the risk of long-lasting
symptoms. However, social context occasionally causes her to be
more willing to risk reading with her eyes than to take on the social
stigma of using headphones during a meeting. When working with
established colleagues who understand her chronic illness, Mack’s
use of TTS and headphones is unremarkable, but when meeting new
collaborators she risks seeming unprofessional or having to disclose
full details of her disability to do so. Additionally, McDonnell’s use
of TTS is highly shaped by her environment - she is far less likely
to work from the foor, requiring TTS, when in the ofce, but will
readily do so in her apartment. Both internal and external context
are key determiners of what technology support is most useful at a
given time.
This case highlights the importance of viewing disability from
both a social and physical lens (Tenet 3). For both Mack and Mc-
Donnell, TTS meets access needs that are not fundamentally social
in nature - they are seeking ways to limit or live with physiological
symptoms. This is diferent from many social model approaches
to accessibility which seek to identify and change discriminatory
social and environmental factors. However, it is also not a medi-
cal model approach - TTS use is by no means a cure or treatment
for underlying symptoms, nor does it seek to normalize them to
a nondisabled ideal. Further, having access to TTS does, in many
ways, meet Mack and McDonnell’s access needs in that it allows
them to continue their work where chronic illness may have oth-
erwise prevented it. However, they do not reach some ideal state
where they are no longer experiencing disability or impairment
Mack and McDonnell et al.
when these access needs are met - in fact TTS is often most nec-
essary when they are especially symptomatic. By recognizing the
social factors at play while also leaving space for physiological
experiences of symptoms, we can better understand the goals of
these chronically ill technology users and the role HCI technologies
can play in achieving them.
4.3.3 Potential for Technology Support: Broader User Bases and
Contexts. The fact that McDonnell and Mack are not the “tradi-
tional” users of TTS technology raises interesting design questions
around how to describe, and market to, technology users. Often
in accessibility research, “people who are blind” and “people who
use screen readers” are used synonymously. This case is a demon-
strative example: not all screen reader users are blind or have low
vision. When narrowly conceiving of who the users of accessible
technologies are, this purportedly inclusive design ends up exclud-
ing people with chronic illnesses and anyone else who designers
failed to imagine might have a use for an accessible technology.
Categorizations around who is the “intended user” for a technology
can lead to increased resistance from institutions (e.g., insurance
companies), social stigma, and even denials of requests (e.g., people
with fatigue who can physically walk face resistance requesting
wheelchairs [9]). Chronically ill people’s access needs often overlap
with those that are more comprehensively understood by accessibil-
ity practitioners, but a lack of attention to chronic illness within the
feld means that these unique use cases are not considered in design.
Future work in HCI accessibility should consider the multiplicity of
ways people could meet the same access need and the multiplicity
of access needs that can be met by the same technology, moving
towards future tools with a wide range of customizable options.
Looking beyond the individual, the broader social and physi-
cal environments afect technology use, and therefore need to be
considered in when and how to adapt technology for chronically
ill people. Bennett et al. proposed a model of interdependence for
viewing a disabled persons’ interactions with their environment
and assistive technologies where technology use is infuenced by
factors outside of the disabled person and their technology [12]. In
this case, we see examples where social dynamics and the need to
disclose and explain her disability afected Mack’s choice to use
screen readers. In other cases, technology supports may become
less critical when a trusted ally can provide the same care.
Finally, this case, mirroring trends in online chronic illness fo-
rums [11, 90], demonstrates the crucial role that shared expertise in
managing illness has in communities. McDonnell started imagining
the ways TTS technologies could beneft her after watching Mack
adapt screen reader technologies to her own needs. Since then, Mc-
Donnell and Mack have recommended the technology to numerous
other people who fnd benefts in consuming content auditorily.
This discovery process could be an area to engage occupational
therapists, who frequently focus on creative way to use existing
tools to support people in expanding function [58], though we note
that occupational therapy is often not ofered or available to many
chronically ill people. Future access technologies might consider
1) how they market their capabilities and customizability, and 2)
how to share settings so that current users could introduce others
with similar access needs to their use of a tool. This feature could
Chronically Under-Addressed: Considerations for HCI Accessibility Practice with Chronically Ill People ASSETS ’22, October 23–26, 2022, Athens, Greece

reduce the onboarding and learning cost, especially among people the meeting when a person was screen sharing graphs she had to
with less comfort using new technologies. examine, they kept scrolling the screen which made Mack acutely
nauseous. Since that experience, she often starts meetings by es-
4.4 Case 3: Remote work tablishing group norms: asking people to share links to documents
Finally, we explore the ways that remote access allows McDonnell with her rather than screen sharing and to keep their devices on a
and Mack to more easily meet access needs that emerge throughout stable surface or turn their cameras of if they are moving.
their days. This case highlights the importance of viewing some 4.4.2 Applying the Tenets. Mack and McDonnell’s experiences
access barriers and remedies as social and others as based in the with remote work demonstrate the need to move beyond tradi-
body (Tenet 3), and it introduces the idea of internal access conficts. tional models of disability (Tenet 3). While there are some changes
to the built environment that could lessen the burden of in-person
4.4.1 The Scenario. For both Mack and McDonnell, attending meet-
work for the McDonnell and Mack (e.g., access to a place to lay
ings and classes virtually allow them to more easily and efec-
down as needed), this social model thinking has its limits because
tively manage symptoms and participate in otherwise physically-
it is not solely the built environment that is disabling in their cases.
inaccessible events. Both experience symptoms that can be triggered
Their access needs arise from their bodies, highlighting that it is
by activity, such as walking or commuting to campus. Mack fnds
critical to include bodily realities of impairment in theoretical ap-
that she cannot easily attend classes or meetings in the morning
proaches to chronic illness. We do not suggest that isolation by
(and sometimes all day) without feeling debilitatingly sick. For Mc-
way of an inaccessible environment is justifable for those who
Donnell, the COVID-precipitated shift to work-from-home made
are currently prohibited from being able to participate in physical
it so that she no longer has to leave the house at set times. She
daily life. Instead we consider that sometimes the most accessible
prioritizes going on walks or completing errands after her daily
or preferable option is to provide the opportunity for multiple en-
obligations are met, thus lowering the cost if activity triggers symp-
vironments, rather than one universally accessible space, following
toms. On top of remote work enabling Mack and McDonnell to
Dolmage’s invitation to approach the universal design of spaces as
arrange their days to better control symptoms, it also makes it eas-
“multiple and in-process” [36].
ier for them to manage symptoms as they arise. Both fnd that the
The variability of McDonnell and Mack’s symptoms and abilities
work of managing symptoms requires myriad resources, meaning
requires considerable, burdensome preparation (Tenet 2). While
that leaving the house may require packing beverages, snacks that
their homes house a variety of tools to prevent or manage symptoms
meet their dietary restrictions, medication, or mobility aids. Addi-
(e.g., food, medication), these supplies are not usually by-default
tionally, aspects of their environment can impact symptoms, and
available in all work environments or in transit. Consequently, they
when working in shared spaces, having control over temperature
pack their bags with potentially helpful or needed supplies when
or a place to lay down is not guaranteed. Mack and McDonnell do
they leave home to prepare for whatever symptoms might arise.
sometimes choose to go into campus when feeling well or to see
While some days the extra preparation might be fully unnecessary,
specifc people, but by default choose to work from home.
both fnd the uncomfortable, sometimes life-threatening, conse-
McDonnell found further benefts from the ability to disguise
quences of being unprepared outweighs the cost. This cost is not
how sick she was feeling or her access hacks during virtual meetings.
negligible, however. The process of bringing all the tools to feel
Because of video conferencing’s limited view, she could discreetly
prepared takes time, adds stress to their days if they forget an item,
make adaptations that reduced symptoms. For example, during
and adds physical weight to already fatiguing walks. In McDonnell
one evening class session, McDonnell was feeling particularly un-
and Mack’s cases, the preparation required for the variability of
well - her temperature was dysregulated and she was experiencing
their abilities is not insurmountable, but the ease provided by a way
malaise from having been upright all day. She attempted to limit
to remotely engage in work is often preferable.
her symptoms by grabbing Gatorade from her fridge and opening
In this case, we see the benefts and efects on the solutions of
the window next to her desk to cool down. McDonnell was also
viewing people with chronic illnesses as having access, rather than
able to recline somewhat by putting her feet on her windowsill and
solely medical, needs (Tenet 1). If viewing Mack and McDonnell
leaning back in her chair while still appearing attentive in class
as patients, the most obvious tools to address the inaccessibility
with her camera on. However, as time wore on, she continued to
they face while working in person are medical treatments. While
feel worse, so she turned of her camera, grabbed her computer,
both Mack and McDonnell are actively pursuing the medical care
and fnished class while lying on the foor.
that may make the broader world easier to navigate, understanding
Remote work, while still Mack’s overall preference, is not a per-
them as people with access needs makes visible ways they can be
fect solution. Though remote attendance eliminates the need to
better supported holistically. Therefore, the afordances provided
walk early in the day and trigger symptoms, the shaky video feeds of
by remote work (e.g., the ability to go on and of camera or to block
her peers or professors can trigger her motion sickness. Unlike con-
nauseating motion) become legible as accommodations.
suming TikToks, Mack often needs and is expected to pay attention
to visual content in work contexts. Over months of remote work, 4.4.3 Potential for Technology Support: Consider Internal Access
Mack found a variety of hacks to avoid getting nauseous during Conflicts. While prior work has considered access conficts between
video calls. For example, one day, when a meeting attendee started disabled people [35, 57, 85, 87], this case explores how technology
walking around with their laptop, causing signifcant motion, Mack can create internal access conficts, a phenomenon sometimes dis-
opened a Notepad window on her computer and positioned it so cussed within chronic illness communities [93]. While attending
that it blocked only that person’s camera feed. However, later in class remotely alleviated early-morning symptoms for Mack, it
ASSETS ’22, October 23–26, 2022, Athens, Greece
created a confict by causing her motion sickness. Particularly in
the case of chronic illness, individuals can have diferent access
needs that overlap, confict and synergize in ways that lead to
unique technology use. For example, a technology whose interface
is very visual may lessen cognitive load for users but also lessen the
ability to use the tool non-visually. Technology designers should
consider that users may have internally conficting access needs
and therefore pay attention to implications of all design decisions
and maximize opportunities for customization.
5 DISCUSSION
We have identifed three tenets for future accessibility research with
chronically ill people (Beyond Patients, Variability of Ability, In-
clude the Body), and demonstrated the ways that they can highlight
new opportunities for technology design throughout our autoethno-
graphic cases. We now discuss additional considerations that follow
from our reframing; the need to account for consequence-based
accessibility, approaching design for community use, and method-
ological changes for working with chronically ill people.
5.1 Consequence-Based Accessibility
In this paper, we present a paradigm shift in how we defne ac-
cessibility based on a more dynamic understanding of access. The
traditional, binary approach to technology that cleaves access needs
into “I can” versus “I cannot” fails to encompass the fuctuating
needs of people with chronic illnesses. People with chronic illnesses
can often technically perform an action that is practicably inaccessi-
ble to them because inaccessibility can arise from the repercussions
of doing that action. To demonstrate this diference: handwritten
text might be pervasively inaccessible to a blind person without
the support of technology or sighted companions. However, for
someone with a chronic illness that impacts digestion (e.g., ulcera-
tive colitis) they can technically eat all foods, but face severe and
debilitating reactions to certain foods, rendering those foods practi-
cably inaccessible. We, therefore, present a paradigm of designing
for consequence-based accessibility, which encompasses the conse-
quences an action causes, rather than solely the innate in/ability to
perform a task.
Viewing technology design through the lens of consequence-
based accessibility acknowledges that many chronically ill people
have the choice to incur consequences, even if those consequences
cause discomfort or more access needs. Perhaps Thanksgiving din-
ner is worth a fare in gastrointestinal symptoms, a cute summer
outft without compression socks may be worth later unsteadiness,
and running to catch a kid falling of a playground structure might
take precedence over the later malaise these actions could trigger.
Chronically ill people learn to live in their bodies, and perform a
complex calculus to determine which consequences to avoid and
which to weather, shaped by variables such as current symptoms,
future plans, environments, urgency, social context, availability of
accessible options, resources, desire, and many more.
As designers of accessible technology, we need to reconsider
what “accessibility” means to people who have the option to partake
in an activity, but with varying costs. While chronically ill people
often also deal with more “traditional” access barriers (e.g., low fne
motor control in their fngers may make it so they can’t open a
Mack and McDonnell et al.
jar), other access needs arise when performing an action. When we
consider the impact this framing has on technology, we see areas
for innovation. First, technology can collect and provide easy access
to the information that chronically ill people need to make well-
informed decisions (e.g., a snapshot of what they have planned for
the day, recent heart rate trends). Providing the right information
at apt times poses interesting technical challenges. Second, systems
can consider how to best adapt their interfaces and operations to
meet their users’ needs after an action (within or outside of the
system) triggers symptoms. For example, symptomatic users may
beneft from a lower-cognition interaction mode or shifting from
visual to auditory content output. Future research avenues could
focus on learning what these levels of accessible modes of operation
are and when to enable them.
To operationalize fuctuating access needs, technology designers
must recognize that chronically ill people constantly defne and
redefne what constitutes “unacceptably impaired,” and therefore
inaccessibility. Individuals determine what is inaccessible to them
at a given time based on deeply personal and contextual factors,
performing a situational “consequence calculus” to determine if
an activity is worth its consequences. Having the ability to adjust
their defnition of “accessible” to their current context can aford
chronically ill people greater agency, but also introduces internal
and external doubt around the validity of people’s access needs.
In thinking about the technological consequences of redefning
accessibility, we see that supporting user agency and contextual
adaptations is key.
Finally, approaching accessibility through a consequence-based
lens that centers the underrepresented experience of chronically
ill people creates potential to better meet the individualized, con-
textual needs that many disabled people have when using acces-
sible technology [46, 64]. Future accessibility work done using a
consequences-based model could consider that, for example, many
blind and low-vision people’s vision changes based on the time
of the day, or could account for the optical and mental strain that
speechreading for long periods of time has on d/Deaf or hard of
hearing people, or better match the needs of people with mental
health disabilities that are cyclic in nature (e.g., bipolar disorder).
Further, we hope that our interrogation of what designers assume
when we think about “accessibility” serves as a useful starting place
for future researchers to interrogate the paradigms in which we
work.
5.2 Designing for Communities
Our cases provide examples of two chronically ill people sharing
access support (Case 1) and knowledge (Case 2); these themes of
using care networks or other chronic illness communities to make
sense of one’s condition and create access hacks in day-to-day life
is documented within HCI (e.g., [43, 66, 82, 90, 92, 129, 144, 159])
and among disability community activists [6, 98, 117, 118, 153].
When working with a group of people who have already built,
engaged in, and found joy within [98] a community, we propose that
interdependence may be a more appropriate goal for technology
design than independence, following Bennett et al’s framework
[12]. Indeed, the act of being cared for, like receiving aid from
a care network, might provide emotional benefts that outweigh
Chronically Under-Addressed: Considerations for HCI Accessibility Practice with Chronically Ill People
the benefts of independence provided by purely technological
solutions.
Designing for interdependence involves building for transparency
with others and oftentimes giving other users capacity to take ac-
tion. For example, consider a system where trusted friends could
monitor the biological levels (e.g., heart rate, blood sugar) of a per-
son with a chronic illness and be alerted to intervene or provide
more support in symptomatic times. The people given these priv-
ileges might be trusted members of care networks. However, we
must also resist a naively optimistic view of care, and consider
how to build systems in a way that could protect and grant agency
to a chronically ill person in an abusive or unsafe care arrange-
ment [71]. Further, though interdependence can take the form of a
nondisabled person supporting a disabled person, our case studies
and examples from communities (e.g., [11, 92, 117]) demonstrate
support maintained fully within chronically ill spheres. Thus, any
systems designed to support chronically ill people must avoid as-
suming a distinction between support giver and support recipient -
chronically ill people are often already both.
5.3 Doing Research with Chronically Ill People:
Efects on Methodology
As we propose an approach to HCI accessibility research with chron-
ically ill people, we also refect on how research methods may need
to change to be accessible to this population. Prior work describes
how to plan accessible studies for people with disabilities, including
accommodations for varying fatigue or incorporating notions of
crip time [33, 79, 87]. Mack et al. describe ways to allow for more
fexibility like allowing interviews to take place over multiple ses-
sions, building in breaks, and adjusting the space to be comfortable
for participants’ bodies [87]. Centering a chronically ill perspec-
tive, we add allowing access to food and drink, prioritizing remote
facilitation options, explicitly providing the option to participate
from nontypical locations (e.g., the foor), and considering potential
sensory sensitivity triggers (e.g., motion, light, loud noises).
There are other methods which may be challenging to run
with strong internal validity while prioritizing participant benef-
icence. Consider, for example, within-subjects controlled experi-
ments, which rely on the assumption that an individual’s capacities
are an experimental constant. How might a testing instrument ac-
count for the reality that someone may begin a study reporting a
2/10 on a pain scale but end it at an 8/10 (perhaps directly due to
their participation in the study)? Further, what is the procedure if a
participant with fuctuating symptoms shows up to a study without
the access need for which they were recruited (e.g., someone with
fuctuating brain fog has no brain fog on the study date)? This per-
ceived “threat” to internal validity may be appeased if symptoms
could be triggered consistently, though we argue that this is unrea-
sonable to ask of participants (e.g., triggering a migraine can have
hour or day long impacts). One solution may be to perform data
collection in-situ when the necessary conditions occur naturally
rather than engineering a symptom increase; while this may lessen
internal validity, it increases ecological validity and prioritizes par-
ticipant benefcence. In general, we suggest strategies of planning
studies that prioritize the access needs and comfort of participants,
even if it means being more creative in the study design. Because
ASSETS ’22, October 23–26, 2022, Athens, Greece
chronically ill people’s access needs often manifest diferently than
HCI anticipates, researchers must pay careful attention to method-
ological in/accessibility when working with people with chronic
illnesses.
6 LIMITATIONS AND ETHICS
Our work has limitations and necessary ethical considerations. First,
autoethnographic methods are not designed for broad generaliz-
ability, and the examples we provide in this paper come from the
experiences of two people with similar demographic backgrounds.
We do not intend our work to serve as a survey of chronic illness ex-
periences, but future research could explore how our tenets operate
when applied to a wider range of experiences. Additionally, as we
outline ways of engaging with a large, broadly defned community,
our scope is wide. There are open questions around how chronic
illness and other forms of bodymind diference (e.g., mental health
disabilities) overlap and diverge, and we encourage future work to
explore this nuance. Additionally, we are not able to speak to the
wide range of ways that people identify with chronic illness and/or
disability, a promising area for future work. Finally, while we be-
lieve that HCI accessibility work that includes chronically ill people
could serve under-considered populations, we are also cognizant of
the harm that technical intervention can cause. We encourage de-
signers and researchers to adopt a critical eye around whether their
work is needed and useful, or another disability dongle [67, 68].
7 CONCLUSION
In this work, we present three core tenets for HCI community
members to consider when designing technology for people with
chronic illnesses. First, we must look beyond patienthood to see
chronically ill people as having access needs and expertise. Second,
we highlight that variable ability requires us to consider accessibil-
ity in terms of the consequences actions cause. Finally, we provide
a theoretical approach to chronic illness that highlights both bodily
and socioenvironmental factors. We demonstrate the utility of these
tenets through the analysis of three autoethnographic refections
on the technology use of two chronically ill authors, noting impli-
cations for technology design. Finally, we discuss the implications
of consequence-based accessibility and what researchers should
consider when designing technology for and conducting research
with chronically ill participants. We hope that this work spurs more
work in the HCI community that focuses on the access needs of
this growing population.
ACKNOWLEDGMENTS
This work was supported by the National Science Foundation Grad-
uate Research Fellowship under Grant No. DGE-2140004, by NSF
2009977 and 1836813; and by the University of Washington Center
for Research and Education on Accessible Technology and Expe-
riences. We would also like to thank Jennifer Mankof, Cynthia
Bennett, Megan Hofmann, Abigale Stangl, and Taylor Schenone for
their support in performing this work.
REFERENCES
[1] Brooke E. Auxier, Cody L. Buntain, Paul Jaeger, Jennifer Golbeck, and Hernisa
Kacorri. 2019. #HandsOfMyADA: A Twitter Response to the ADA Education
and Reform Act. In Proceedings of the 2019 CHI Conference on Human Factors
ASSETS ’22, October 23–26, 2022, Athens, Greece
in Computing Systems (CHI ’19). Association for Computing Machinery, 1–12.
https://doi.org/10.1145/3290605.3300757
[2] Shayla Ayn. 2018. The ’Disaster’ That Is Showering With a Chronic
Illness. https://themighty.com/2018/08/fbromyalgia-postural-orthostatic-
tachycardia-syndrome-shower/
[3] Ron Baecker, Kate Sellen, Sarah Crosskey, Veronique Boscart, and Barbara
Barbosa Neves. 2014. Technology to reduce social isolation and loneliness. In
Proceedings of the 16th international ACM SIGACCESS conference on Computers &
accessibility (ASSETS ’14). Association for Computing Machinery, 27–34. https:
//doi.org/10.1145/2661334.2661375
[4] Maryam Bandukda, Aneesha Singh, Catherine Holloway, Nadia Berthouze,
Emeline Brulé, Ana Tajadura-Jiménez, Oussama Metatla, Ana Javornik, and
Anja Thieme. 2021. Rethinking the Senses: A Workshop on Multisensory Embodied
Experiences and Disability Interactions. Association for Computing Machinery,
1–5. http://doi.org/10.1145/3411763.3441356
[5] Andrea Barbarin, Tifany C. Veinot, and Predrag Klasnja. 2015. Taking our
Time: Chronic Illness and Time-Based Objects in Families. In Proceedings of
the 18th ACM Conference on Computer Supported Cooperative Work & Social
Computing (CSCW ’15). Association for Computing Machinery, 288–301. https:
//doi.org/10.1145/2675133.2675200
[6] Imani Barbarin. 2022. The Pandemic Tried to Break Me, but I Know My Black Dis-
abled Life Is Worthy. https://www.cosmopolitan.com/entertainment/a39355245/
imani-barbarin-black-disabled-activist-self-love/
[7] Ereny Bassilious, Aaron DeChamplain, Ian McCabe, Matt Stephan, Bill Kapralos,
Farid H. Mahmud, and Adam Dubrowski. 2012. Power defense: a serious game for
improving diabetes numeracy. In CHI ’12 Extended Abstracts on Human Factors
in Computing Systems (CHI EA ’12). Association for Computing Machinery,
1327–1332. https://doi.org/10.1145/2212776.2212449
[8] BBC. 2021. Chronic fatigue syndrome advice scraps exercise therapy. https:
//www.bbc.com/news/health-59080007
[9] Brianne Benness. 2019. My Disability Is Dynamic. https://medium.com/age-of-
awareness/my-disability-is-dynamic-bc2a619fcc1
[10] Brianne Benness. 2020. Disease Begins Before Diagnosis. https://www.ted.com/
talks/brianne_benness_disease_begins_before_diagnosis
[11] Brianne Benness. 2020. What does #NEISVoid mean? https://noendinsight.co/
neisvoid-explained/
[12] Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence
as a Frame for Assistive Technology Research and Design. In Proceedings of the
20th International ACM SIGACCESS Conference on Computers and Accessibility
(Galway, Ireland) (ASSETS ’18). Association for Computing Machinery, New
York, NY, USA, 161–173. https://doi.org/10.1145/3234695.3236348
[13] Toni Bernhard. 2019. 7 Ways to Survive a Flare When You’re Chronically
Ill. https://www.psychologytoday.com/us/blog/turning-straw-gold/201901/7-
ways-survive-fare-when-you-re-chronically-ill
[14] Andrew B. L. Berry, Catherine Lim, Andrea L. Hartzler, Tad Hirsch, Edward H.
Wagner, Evette Ludman, and James D. Ralston. 2017. How Values Shape Col-
laboration Between Patients with Multiple Chronic Conditions and Spousal
Caregivers. In Proceedings of the 2017 CHI Conference on Human Factors in Com-
puting Systems (CHI ’17). Association for Computing Machinery, 5257–5270.
https://doi.org/10.1145/3025453.3025923
[15] Andrew B. L. Berry, Catherine Y. Lim, Tad Hirsch, Andrea L. Hartzler, Linda M.
Kiel, Zoë A. Bermet, and James D. Ralston. 2019. Supporting Communica-
tion About Values Between People with Multiple Chronic Conditions and
their Providers. In Proceedings of the 2019 CHI Conference on Human Factors
in Computing Systems (CHI ’19). Association for Computing Machinery, 1–14.
https://doi.org/10.1145/3290605.3300700
[16] Jean-Marie Berthelot, Michel De Bandt, Jacques Morel, Fatima Benatig, Ar-
naud Constantin, Philippe Gaudin, Xavier Le Loet, Jean-Francis Maillefert,
Olivier Meyer, Thao Pham, et al. 2012. A tool to identify recent or present
rheumatoid arthritis fare from both patient and physician perspectives: the
‘FLARE’instrument. Annals of the rheumatic diseases 71, 7 (2012), 1110–1116.
[17] Amritpal Singh Bhachu, Nicolas Hine, and John Arnott. 2008. Technology
devices for older adults to aid self management of chronic health conditions. In
Proceedings of the 10th international ACM SIGACCESS conference on Computers
and accessibility (Assets ’08). Association for Computing Machinery, 59–66.
https://doi.org/10.1145/1414471.1414484
[18] Peter Boersma, Lindsey I. Black, and Brian Ward. 2020. Prevalence of Multiple
Chronic Conditions Among US Adults, 2018. https://www.cdc.gov/pcd/issues/
2020/20_0130.htm
[19] Disha Bora, Hanlin Li, Sagar Salvi, and Erin Brady. 2017. ActVirtual: Making
Public Activism Accessible. In Proceedings of the 19th International ACM SIGAC-
CESS Conference on Computers and Accessibility (ASSETS ’17). Association for
Computing Machinery, 307–308. https://doi.org/10.1145/3132525.3134815
[20] Eleanor R. Burgess, Madhu C. Reddy, Andrew Davenport, Paul Laboi, and Ann
Blandford. 2019. “Tricky to get your head around”: Information Work of People
Managing Chronic Kidney Disease in the UK. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems (CHI ’19). Association for
Computing Machinery, 1–17. https://doi.org/10.1145/3290605.3300895
Mack and McDonnell et al.
[21] Gabrielle S. Cantor. 2018. Designing Technological Interventions for Pa-
tients with Discordant Chronic Comorbidities and Type-2 Diabetes. In Ex-
tended Abstracts of the 2018 CHI Conference on Human Factors in Computing
Systems (CHI EA ’18). Association for Computing Machinery, 1–6. https:
//doi.org/10.1145/3170427.3180304
[22] CDC. 2021. Treating the Most Disruptive Symptoms First and Preventing Worsen-
ing of Symptoms. https://www.cdc.gov/me-cfs/healthcare-providers/clinical-
care-patients-mecfs/treating-most-disruptive-symptoms.html#:~:
text=Post%2Dexertional%20malaise%20(PEM)%20is%20the%20worsening%
20of%20symptoms,by%20activity%20management%20(pacing)
[23] Heewon Chang, Faith Ngunjiri, and Kathy-Ann C Hernandez. 2016. Collaborative
autoethnography. Routledge.
[24] Beenish M. Chaudhry, Christopher Schaefbauer, Ben Jelen, Katie A. Siek, and
Kay Connelly. 2016. Evaluation of a Food Portion Size Estimation Interface
for a Varying Literacy Population. In Proceedings of the 2016 CHI Conference
on Human Factors in Computing Systems (CHI ’16). Association for Computing
Machinery, 5645–5657. https://doi.org/10.1145/2858036.2858554
[25] Chen Chen, Janet G Johnson, Kemeberly Charles, Alice Lee, Ella T Lifset, Michael
Hogarth, Alison A Moore, Emilia Farcas, and Nadir Weibel. 2021. Understanding
Barriers and Design Opportunities to Improve Healthcare and QOL for Older
Adults through Voice Assistants. In The 23rd International ACM SIGACCESS Con-
ference on Computers and Accessibility (ASSETS ’21). Association for Computing
Machinery, 1–16. https://doi.org/10.1145/3441852.3471218
[26] Claudia Chen, Robert Wu, Hashim Khan, Khai Truong, and Fanny Chevalier.
2021. VIDDE: Visualizations for Helping People with COPD Interpret Dyspnea
During Exercise. In The 23rd International ACM SIGACCESS Conference on Com-
puters and Accessibility (ASSETS ’21). Association for Computing Machinery,
1–14. https://doi.org/10.1145/3441852.3471204
[27] Youngjun Cho, Simon J Julier, and Nadia Bianchi-Berthouze. 2019. Instant stress:
detection of perceived mental stress through smartphone photoplethysmogra-
phy and thermal imaging. JMIR mental health 6, 4 (2019), e10140.
[28] Eli Clare. 2001. Stolen bodies, reclaimed bodies: Disability and queerness. Public
Culture 13, 3 (2001), 359–365.
[29] Cleveland Clinic. 2021. What Is the Spoon Theory Metaphor for Chronic Illness?
https://health.clevelandclinic.org/spoon-theory-chronic-illness/
[30] David Cobley. 2018. Disability and international development: A guide for students
and practitioners. Routledge.
[31] Boston Women’s Health Book Collective. 1973. Our Bodies, Ourselves. Simon
and Schuster. Google-Books-ID: cWxqAAAAMAAJ.
[32] Tressie McMillan Cottom. 2019. I Was Pregnant and in Crisis. All the Doctors and
Nurses Saw Was an Incompetent Black Woman. https://time.com/5494404/tressie-
mcmillan-cottom-thick-pregnancy-competent/
[33] Kara Pernice Coyne and Jakob Nielsen. 2001. How to conduct usability evaluations
for accessibility: Methodology guidelines for testing websites and intranets with
users who use assistive technology. Nielsen Norman Group.
[34] Liz Crow. 1996. Including all of our lives: Renewing the social model of disability.
Exploring the divide 55 (1996), 58.
[35] Maitraye Das, John Tang, Kathryn E. Ringland, and Anne Marie Piper. 2021.
Towards Accessible Remote Work: Understanding Work-from-Home Practices
of Neurodivergent Professionals. Proc. ACM Hum.-Comput. Interact. 5, CSCW1,
Article 183 (apr 2021), 30 pages. https://doi.org/10.1145/3449282
[36] Jay Dolmage. 2015. Universal design: Places to start. Disability Studies Quarterly
35, 2 (2015).
[37] Melanie Duckert and Louise Barkhuus. 2022. Protecting Personal Health Data
through Privacy Awareness: A study of perceived data privacy among people
with chronic or long-term illness. Proceedings of the ACM on Human-Computer
Interaction 6, GROUP (Jan 2022), 11:1–11:22. https://doi.org/10.1145/3492830
[38] Adriana E. and Brianne Benness. 2020. No End In Sight: 69 – Adriana. https:
//noendinsight.co/2020/11/29/episode-69-adriana/
[39] Laurie Edwards. 2014. In the Kingdom of the Sick: A Social History of Chronic
Illness in America. Bloomsbury Publishing USA. Google-Books-ID: bemn-
BQAAQBAJ.
[40] Chronic Eileen. 2020. Rheumatoid Arthritis and My Period. https://chroniceileen.
com/2020/06/29/rheumatoid-arthritis-and-my-period/
[41] Jordan Eschler and Wanda Pratt. 2017. " I’m so glad I met you" Designing Dy-
namic Collaborative Support for Young Adult Cancer Survivors. In Proceedings
of the 2017 ACM Conference on Computer Supported Cooperative Work and Social
Computing. 1763–1774.
[42] Heather Evans. 2017. Uncovering: Making disability identity legible. Disability
Studies Quarterly 37, 1 (2017).
[43] Shelly Farnham, Lili Cheng, Linda Stone, Melora Zaner-Godsey, Christopher
Hibbeln, Karen Syrjala, Ann Marie Clark, and Janet Abrams. 2002. HutchWorld:
clinical study of computer-mediated social support for cancer patients and
their caregivers. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’02). Association for Computing Machinery, 375–382.
https://doi.org/10.1145/503376.503444
[44] Colleen Flaherty. 2022. ‘Retract or Attack?’ Two white Africanists publish an
article on centering the scholar’s personal experience to help “decolonize” African
Chronically Under-Addressed: Considerations for HCI Accessibility Practice with Chronically Ill People
studies. A call for retraction follows. https://www.insidehighered.com/news/
2022/05/24/black-scholars-demand-retraction-autoethnography-article
[45] Arthur W Frank. 1993. The rhetoric of self-change: Illness experience as narra-
tive. Sociological quarterly 34, 1 (1993), 39–52.
[46] Rachel L Franz, Jacob O Wobbrock, Yi Cheng, and Leah Findlater. 2019. Percep-
tion and Adoption of Mobile Accessibility Features by Older Adults Experienc-
ing Ability Changes. In The 21st International ACM SIGACCESS Conference on
Computers and Accessibility. 267–278.
[47] Kausalya Ganesh and Amanda Lazar. 2021. The Work of Workplace Disclosure:
Invisible Chronic Conditions and Opportunities for Design. Proceedings of
the ACM on Human-Computer Interaction 5, CSCW1 (Apr 2021), 73:1–73:26.
https://doi.org/10.1145/3449147
[48] Kara Gavin. 2017. Study: Health Plan Deductibles Hit Patients with Chronic
Illness Harder. https://labblog.uofmhealth.org/industry-dx/study-health-plan-
deductibles-hit-patients-chronic-illness-harder
[49] Sara Goering. 2015. Rethinking disability: the social model of disability and
chronic disease. Current reviews in musculoskeletal medicine 8, 2 (2015), 134–138.
[50] Chronically Grey. 2017. Showering and Chronic Illness. https://chronicallygrey.
wordpress.com/2017/11/23/showering-and-chronic-illness/
[51] Alex Haagaard. 2019. 2 years of biohacking. https://alexhaagaard.medium.com/2-
years-of-biohacking-8e99b32bb350
[52] Alex Haagaard. 2022. Complicating Disability: On the Invisibilization of
Chronic Illness throughout History. https://blog.castac.org/2022/02/complicating-
disability-on-the-invisibilization-of-chronic-illness-throughout-history/
[53] Cother Hajat and Emma Stein. 2018. The global burden of multiple chronic
conditions: a narrative review. Preventive medicine reports 12 (2018), 284–293.
[54] Shefali Haldar, Sonali R. Mishra, Maher Khelif, Ari H. Pollack, and Wanda
Pratt. 2017. Opportunities and Design Considerations for Peer Support in a
Hospital Setting. In Proceedings of the 2017 CHI Conference on Human Factors in
Computing Systems (CHI ’17). Association for Computing Machinery, 867–879.
https://doi.org/10.1145/3025453.3026040
[55] Aimi Hamraie and Kelly Fritsch. 2019. Crip technoscience manifesto. Catalyst:
Feminism, Theory, Technoscience 5, 1 (2019), 1–33.
[56] Tara O’Neill Hayes and Serena Gillian. 2020. Chronic Disease in
the United States: A Worsening Health and Economic Crisis. https:
//www.americanactionforum.org/research/chronic-disease-in-the-united-
states-a-worsening-health-and-economic-crisis/
[57] Megan Hofmann, Devva Kasnitz, Jennifer Mankof, and Cynthia L Bennett. 2020.
Living Disability Theory: Refections on Access, Research, and Design. In The
22nd International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’20). Association for Computing Machinery, 1–13. https://doi.org/10.
1145/3373625.3416996
[58] Megan Hofmann, Kristin Williams, Toni Kaplan, Stephanie Valencia, Gabriella
Hann, Scott E. Hudson, Jennifer Mankof, and Patrick Carrington. 2019. "Occu-
pational Therapy is Making": Clinical Rapid Prototyping and Digital Fabrication.
In Proceedings of the 2019 CHI Conference on Human Factors in Computing Sys-
tems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery,
New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300544
[59] Matthew K. Hong, Udaya Lakshmi, Kimberly Do, Sampath Prahalad, Thomas Ol-
son, Rosa I. Arriaga, and Lauren Wilcox. 2020. Using Diaries to Probe the Illness
Experiences of Adolescent Patients and Parental Caregivers. In Proceedings of the
2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Associa-
tion for Computing Machinery, 1–16. https://doi.org/10.1145/3313831.3376426
[60] Matthew K. Hong, Udaya Lakshmi, Thomas A. Olson, and Lauren Wilcox. 2018.
Visual ODLs: Co-Designing Patient-Generated Observations of Daily Living
to Support Data-Driven Conversations in Pediatric Care. In Proceedings of the
2018 CHI Conference on Human Factors in Computing Systems (Montreal QC,
Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA,
1–13. https://doi.org/10.1145/3173574.3174050
[61] Matthew K. Hong, Lauren Wilcox, Daniel Machado, Thomas A. Olson, and
Stephen F. Simoneaux. 2016. Care Partnerships: Toward Technology to Sup-
port Teens’ Participation in Their Health Care. In Proceedings of the 2016 CHI
Conference on Human Factors in Computing Systems (CHI ’16). Association for
Computing Machinery, 5337–5349. https://doi.org/10.1145/2858036.2858508
[62] Juan Pablo Hourcade, Martha Driessnack, and Kelsey E. Huebner. 2012. Support-
ing face-to-face communication between clinicians and children with chronic
headaches through a zoomable multi-touch app. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI ’12). Association for
Computing Machinery, 2609–2618. https://doi.org/10.1145/2207676.2208651
[63] Jina Huh and Mark S. Ackerman. 2012. Collaborative help in chronic disease man-
agement: supporting individualized problems. In Proceedings of the ACM 2012
conference on Computer Supported Cooperative Work (CSCW ’12). Association
for Computing Machinery, 853–862. https://doi.org/10.1145/2145204.2145331
[64] Amy Hurst, Scott E Hudson, Jennifer Mankof, and Shari Trewin. 2013. Dis-
tinguishing users by pointing performance in laboratory and real-world tasks.
ACM Transactions on Accessible Computing (TACCESS) 5, 2 (2013), 1–27.
[65] Sins Invalid. 2019. Skin Tooth and Bone: The Basis of Movement is Our People, a
Disability Justice Primer (2nd ed.). Sins Invalid.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[66] Nwakego Isika, Antonette Mendoza, and Rachelle Bosua. 2020. “I need to com-
partmentalize myself”: Appropriation of Instagram for chronic illness manage-
ment. In Proceedings of the Australasian Computer Science Week Multiconference.
ACM, 1–9. https://doi.org/10.1145/3373017.3373040
[67] Liz Jackson. 2019. Disability Dongle. https://twitter.com/elizejackson/status/
1110629818234818570?lang=en
[68] Liz Jackson, Alex Haagaard, and Rua Williams. 2022. Disability Dongle. https:
//blog.castac.org/2022/04/disability-dongle/
[69] Sylvia Janicki, Matt Ziegler, and Jennifer Mankof. 2021. Navigating Illness,
Finding Place: Enhancing the Experience of Place for People Living with Chronic
Illness. In ACM SIGCAS Conference on Computing and Sustainable Societies. 173–
187.
[70] Heidi L Janz. 2019. Ableism: the undiagnosed malady aficting medicine. CMAJ
191, 17 (2019), E478–E479.
[71] Criminal Justice. N.d.. Caregiver Violence against People with Disabil-
ities. http://criminal-justice.iresearchnet.com/crime/domestic-violence/
caregiver-violence-against-people-with-disabilities/
[72] Alison Kafer. 2013. Feminist, queer, crip. Indiana University Press.
[73] Yamini Karanam, Andrew Miller, and Erin Brady. 2017. Needs and Challenges
of Post-Acute Brain Injury Patients in Understanding Personal Recovery. In
Proceedings of the 19th International ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS ’17). Association for Computing Machinery, 381–382.
https://doi.org/10.1145/3132525.3134794
[74] Ravi Karkar, Jessica Schroeder, Daniel A Epstein, Laura R Pina, Jefrey Scofeld,
James Fogarty, Julie A Kientz, Sean A Munson, Roger Vilardaga, and Jasmine
Zia. 2017. Tummytrials: a feasibility study of using self-experimentation to
detect individualized food triggers. In Proceedings of the 2017 CHI conference on
human factors in computing systems. 6850–6863.
[75] Elizabeth Kaziunas, Mark S. Ackerman, Silvia Lindtner, and Joyce M. Lee. 2017.
Caring through Data: Attending to the Social and Emotional Experiences of
Health Datafcation. In Proceedings of the 2017 ACM Conference on Computer
Supported Cooperative Work and Social Computing (CSCW ’17). Association for
Computing Machinery, 2260–2272. https://doi.org/10.1145/2998181.2998303
[76] Christina Kelley, Bongshin Lee, and Lauren Wilcox. 2017. Self-tracking for
Mental Wellness: Understanding Expert Perspectives and Student Experiences.
In Proceedings of the 2017 CHI Conference on Human Factors in Computing
Systems (CHI ’17). Association for Computing Machinery, 629–641. https:
//doi.org/10.1145/3025453.3025750
[77] Eva Feder Kittay. 2011. The Ethics of Care, Dependence, and Disability*. Ratio
Juris 24, 1 (2011), 49–58. https://doi.org/10.1111/j.1467-9337.2010.00473.x
[78] Fortesa Latif. 2021. I’ve spent a lifetime trying to get doctors to believe my pain.
It’s all too common for women. https://www.thelily.com/ive-spent-a-lifetime-
trying-to-get-doctors-to-believe-my-pain-its-all-too-common-for-women/
[79] Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Working with
research participants with disabilities. In Research Methods in Human-Computer
Interaction. Morgan Kaufmann, 493–522.
[80] Hanlin Li, Disha Bora, Sagar Salvi, and Erin Brady. 2018. Slacktivists or Activists?
Identity Work in the Virtual Disability March. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems (Montreal QC, Canada)
(CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13.
https://doi.org/10.1145/3173574.3173799
[81] Catherine Y. Lim, Andrew B.L. Berry, Andrea L. Hartzler, Tad Hirsch, David S.
Carrell, Zoë A. Bermet, and James D. Ralston. 2019. Facilitating Self-refection
about Values and Self-care Among Individuals with Chronic Conditions. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
(CHI ’19). Association for Computing Machinery, 1–12. https://doi.org/10.1145/
3290605.3300885
[82] Leslie S. Liu, Jina Huh, Tina Neogi, Kori Inkpen, and Wanda Pratt. 2013. Health
vlogger-viewer interaction in chronic illness management. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Associa-
tion for Computing Machinery, 49–58. https://doi.org/10.1145/2470654.2470663
[83] Audre Lorde. 1980. The Cancer Journals. Spinsters, Ink. Google-Books-ID:
HSweAQAAIAAJ.
[84] Yuhan Luo, Peiyi Liu, and Eun Kyoung Choe. 2019. Co-Designing Food Track-
ers with Dietitians: Identifying Design Opportunities for Food Tracker Cus-
tomization. In Proceedings of the 2019 CHI Conference on Human Factors in
Computing Systems (CHI ’19). Association for Computing Machinery, 1–13.
https://doi.org/10.1145/3290605.3300822
[85] Kelly Mack, Maitraye Das, Dhruv Jain, Danielle Bragg, John Tang, Andrew
Begel, Erin Beneteau, Josh Urban Davis, Abraham Glasser, Joon Sung Park,
and Venkatesh Potluri. 2021. Mixed Abilities and Varied Experiences: a group
autoethnography of a virtual summer internship. In The 23rd International ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS ’21). Association
for Computing Machinery, 1–13. https://doi.org/10.1145/3441852.3471199
[86] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich,
and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? A
Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to
2019. In Proceedings of the 2021 CHI Conference on Human Factors in Computing
ASSETS ’22, October 23–26, 2022, Athens, Greece
Systems. 1–18.
[87] Kelly Mack, Emma McDonnell, Venkatesh Potluri, Maggie Xu, Jailyn Zabala,
Jefrey P. Bigham, Jennifer Mankof, and Cynthia L Bennett. 2022. Anticipate
and Adjust: Cultivating Access in Human-Centered Methods. In Proceedings of
the 2022 CHI Conference on Human Factors in Computing Systems. 1–18.
[88] Angus Mackay. 2019. A neuro-infammatory model can explain the onset,
symptoms and fare-ups of myalgic encephalomyelitis/chronic fatigue syndrome.
Journal of Primary Health Care 11, 4 (2019), 300–307.
[89] Haley MacLeod, Grace Bastin, Leslie S. Liu, Katie Siek, and Kay Connelly. 2017.
“Be Grateful You Don’t Have a Real Disease”: Understanding Rare Disease
Relationships. In Proceedings of the 2017 CHI Conference on Human Factors in
Computing Systems (CHI ’17). Association for Computing Machinery, 1660–1673.
https://doi.org/10.1145/3025453.3025796
[90] Haley MacLeod, Kim Oakes, Danika Geisler, Kay Connelly, and Katie Siek. 2015.
Rare World: Towards Technology for Rare Diseases. In Proceedings of the 33rd
Annual ACM Conference on Human Factors in Computing Systems (CHI ’15).
Association for Computing Machinery, 1145–1154. https://doi.org/10.1145/
2702123.2702494
[91] Jennifer Mankof, Gillian R. Hayes, and Devva Kasnitz. 2010. Disability studies as
a source of critical inquiry for the feld of assistive technology. In Proceedings of
the 12th international ACM SIGACCESS conference on Computers and accessibility
(ASSETS ’10). Association for Computing Machinery, 3–10. https://doi.org/10.
1145/1878803.1878807
[92] Jennifer Mankof, Kateryna Kuksenok, Sara Kiesler, Jennifer A. Rode, and Kelly
Waldman. 2011. Competing online viewpoints and models of chronic illness. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
(CHI ’11). Association for Computing Machinery, 589–598. https://doi.org/10.
1145/1978942.1979027
[93] Caroline McDonagh. 2022. Have you ever had your access needs directly con-
fict with someone else’s access needs? https://twitter.com/CazMcDo/status/
1538840632076148736
[94] Stephanie McManimen, Damani McClellan, Jamie Stoothof, Kristen Gleason,
and Leonard A. Jason. 2019. Dismissing chronic illness: A qualitative analysis
of negative health care experiences. Health care for women international 40, 3
(Mar 2019), 241–258. https://doi.org/10.1080/07399332.2018.1521811
[95] James Milewski and Hector Parra. 2011. Gathering requirements for a personal
health management system. In CHI ’11 Extended Abstracts on Human Factors
in Computing Systems (CHI EA ’11). Association for Computing Machinery,
2377–2382. https://doi.org/10.1145/1979742.1979881
[96] Claudia S Miller, Raymond F Palmer, Tania T Dempsey, Nicholas A Ashford,
and Lawrence B Afrin. 2021. Mast cell activation may explain many cases of
chemical intolerance. Environmental Sciences Europe 33, 1 (2021), 1–15.
[97] Julia Milligan. 2022. 5 Things That Can Trigger a Multiple Sclerosis Re-
lapse. https://themighty.com/2022/01/multiple-sclerosis-relapse-triggers/
?msclkid=a921cfa9b69111ecb40da23e69d8f9ea
[98] Mia Mingus. 2011. Access Intimacy: The Missing Link. https://leavingevidence.
wordpress.com/2011/05/05/access-intimacy-the-missing-link/
[99] Mia Mingus. 2011. Changing the Framework: Disability Justice.
https://leavingevidence.wordpress.com/2011/02/12/changing-the-framework-
disability-justice/
[100] Christine Miserandino. 2003. The Spoon Theory written by Christine
Miserandino. https://butyoudontlooksick.com/articles/written-by-christine/
the-spoon-theory/
[101] Sonali R. Mishra, Predrag Klasnja, John MacDufe Woodburn, Eric B. Hekler,
Larsson Omberg, Michael Kellen, and Lara Mangravite. 2019. Supporting Coping
with Parkinson’s Disease Through Self Tracking. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems (CHI ’19). Association for
Computing Machinery, 1–16. https://doi.org/10.1145/3290605.3300337
[102] Liz Moore. 2019. How Can I Convince Doctors I’m an Informed Patient? https:
//www.healthline.com/health/doctors-listen-to-patients
[103] Ann Neville-Jan. 2003. Encounters in a world of pain: An autoethnography. The
American journal of occupational therapy 57, 1 (2003), 88–98.
[104] Kim E Nielsen. 2012. A disability history of the United States. Vol. 2. Beacon
Press.
[105] Joe Norris, Richard D Sawyer, and Darren Lund. 2012. Duoethnography: Dialogic
methods for social, health, and educational research. Vol. 7. Left Coast Press.
[106] Valerie Novak and Brianne Benness. 2020. No End In Sight: 67 – Valerie. https:
//noendinsight.co/2020/05/16/episode-67-valerie/
[107] Alexandra CH Nowakowski. 2016. You Poor Thing: A Retrospective Autoethnog-
raphy of Visible Chronic Illness as a Symbolic Vanishing Act. Qualitative Report
21, 10 (2016).
[108] Alexandra CH Nowakowski and JE Sumerau. 2019. Reframing health and
illness: a collaborative autoethnography on the experience of health and illness
transformations in the life course. Sociology of Health & Illness 41, 4 (2019),
723–739.
[109] Francisco Nunes, Maureen Kerwin, and Paula Alexandra Silva. 2012. Design
recommendations for tv user interfaces for older adults: fndings from the
Mack and McDonnell et al.
eCAALYX project. In Proceedings of the 14th international ACM SIGACCESS con-
ference on Computers and accessibility (ASSETS ’12). Association for Computing
Machinery, 41–48. https://doi.org/10.1145/2384916.2384924
[110] ACT UP Advisory Committee of the People with AIDS. 1983. The Denver
Principles. https://actupny.org/documents/Denver.html
[111] Lily O’hara and Jane Taylor. 2018. What’s wrong with the ‘war on obesity?’A
narrative review of the weight-centered health paradigm and development of
the 3C framework to build critical competency for a paradigm shift. Sage Open
8, 2 (2018), 2158244018772888.
[112] Aisling Ann O’Kane and Helena Mentis. 2012. Sharing medical data vs. health
knowledge in chronic illness care. In CHI ’12 Extended Abstracts on Human Fac-
tors in Computing Systems (CHI EA ’12). Association for Computing Machinery,
2417–2422. https://doi.org/10.1145/2212776.2223812
[113] Mike Oliver. 2013. The social model of disability: Thirty years on. Disability &
society 28, 7 (2013), 1024–1026.
[114] Carolyn E. Pang, Carman Neustaedter, Bernhard E. Riecke, Erick Oduor, and
Serena Hillman. 2013. Technology preferences and routines for sharing health in-
formation during the treatment of a chronic illness. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI ’13). Association for
Computing Machinery, 1759–1768. https://doi.org/10.1145/2470654.2466232
[115] Alyson Patsavas. 2014. Recovering a cripistemology of pain: Leaky bodies,
connective tissue, and feeling discourse. Journal of Literary & Cultural Disability
Studies 8, 2 (2014), 203–218.
[116] Adrienne Pichon, Kayla Schifer, Emma Horan, Bria Massey, Suzanne Bakken,
Lena Mamykina, and Noemie Elhadad. 2021. Divided We Stand: The Collabora-
tive Work of Patients and Providers in an Enigmatic Chronic Disease. Proceedings
of the ACM on Human-Computer Interaction 4, CSCW3 (Jan 2021), 261:1–261:24.
https://doi.org/10.1145/3434170
[117] Leah Lakshmi Piepzna-Samarasinha. 2018. Care work: Dreaming disability justice.
Arsenal Pulp Press Vancouver.
[118] Leah Lakshmi Piepzna-Samarasinha. 2021. How Disabled Mutual Aid Is Diferent
Than Abled Mutual Aid. https://disabilityvisibilityproject.com/2021/10/03/how-
disabled-mutual-aid-is-diferent-than-abled-mutual-aid/#site-content
[119] Ruth Pinder. 1996. Sick-but-ft or ft-but-sick? Ambiguity and identity at the
workplace. Exploring the divide (1996), 135–156.
[120] Ari H. Pollack, Uba Backonja, Andrew D. Miller, Sonali R. Mishra, Maher Khelif,
Logan Kendall, and Wanda Pratt. 2016. Closing the Gap: Supporting Patients’
Transition to Self-Management after Hospitalization. In Proceedings of the 2016
CHI Conference on Human Factors in Computing Systems (CHI ’16). Association
for Computing Machinery, 5324–5336. https://doi.org/10.1145/2858036.2858240
[121] John R. Porter, Kiley Sobel, Sarah E. Fox, Cynthia L. Bennett, and Julie A. Kientz.
2017. Filtered Out: Disability Disclosure Practices in Online Dating Communities.
Proceedings of the ACM on Human-Computer Interaction 1, CSCW (Dec 2017),
87:1–87:13. https://doi.org/10.1145/3134722
[122] Margaret Price. 2015. The bodymind problem and the possibilities of pain.
Hypatia 30, 1 (2015), 268–284.
[123] Lauren Race, Amber James, Andrew Hayward, Kia El-Amin, Maya Gold Pat-
terson, and Theresa Mershon. 2021. Designing Sensory and Social Tools for
Neurodivergent Individuals in Social Media Environments. In The 23rd Interna-
tional ACM SIGACCESS Conference on Computers and Accessibility. 1–5.
[124] Joel Michael Reynolds. 2017. “I’d rather be dead than disabled”—the ableist
confation and the meanings of disability. Review of Communication 17, 3 (2017),
149–163.
[125] Rose Richards. 2008. Writing the othered self: Autoethnography and the problem
of objectifcation in writing about illness and disability. Qualitative health
research 18, 12 (2008), 1717–1728.
[126] Kathryn E Ringland, Jennifer Nicholas, Rachel Kornfeld, Emily G Lattie, David C
Mohr, and Madhu Reddy. 2019. Understanding mental ill-health as psychosocial
disability: Implications for assistive technology. In The 21st International ACM
SIGACCESS Conference on Computers and Accessibility. 156–170.
[127] Yasmin Salamah, Rahma Dany Asyifa, and Auzi Asfarian. 2021. Improving The
Usability of Personal Health Record in Mobile Health Application for People
with Autoimmune Disease. In Asian CHI Symposium 2021 (Asian CHI Symposium
2021). Association for Computing Machinery, 180–188. https://doi.org/10.1145/
3429360.3468207
[128] Ellen Samuels. 2017. Six Ways of Looking at Crip Time. Disability studies
quarterly 37, 3 (2017).
[129] Shruti Sannon, Elizabeth L. Murnane, Natalya N. Bazarova, and Geri Gay. 2019.
“I was really, really nervous posting it”: Communicating about Invisible Chronic
Illnesses across Social Media Platforms. In Proceedings of the 2019 CHI Conference
on Human Factors in Computing Systems (CHI ’19). Association for Computing
Machinery, 1–13. https://doi.org/10.1145/3290605.3300583
[130] Jessica Schroeder, Chia-Fang Chung, Daniel A Epstein, Ravi Karkar, Adele
Parsons, Natalia Murinova, James Fogarty, and Sean A Munson. 2018. Examining
self-tracking by people with migraine: goals, needs, and opportunities in a
chronic health condition. In Proceedings of the 2018 designing interactive systems
conference. 135–148.
Chronically Under-Addressed: Considerations for HCI Accessibility Practice with Chronically Ill People
[131] Jessica Schroeder, Jane Hofswell, Chia-Fang Chung, James Fogarty, Sean Mun-
son, and Jasmine Zia. 2017. Supporting Patient-Provider Collaboration to Iden-
tify Individual Triggers using Food and Symptom Journals. In Proceedings of
the 2017 ACM Conference on Computer Supported Cooperative Work and So-
cial Computing (CSCW ’17). Association for Computing Machinery, 1726–1739.
https://doi.org/10.1145/2998181.2998276
[132] Tom Shakespeare. 2013. Disability rights and wrongs revisited. Routledge.
[133] Tom Shakespeare et al. 2006. The social model of disability. The disability studies
reader 2 (2006), 197–204.
[134] Tobin Siebers. 2008. Disability theory. University of Michigan Press.
[135] Tobin Siebers. 2019. Returning the social to the social model. The matter of
disability: Materiality, biopolitics, crip afect (2019), 39–47.
[136] Katie A. Siek, Kay H. Connelly, and Yvonne Rogers. 2006. Pride and prejudice:
learning how chronically ill people think about food. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI ’06). Association for
Computing Machinery, 947–950. https://doi.org/10.1145/1124772.1124912
[137] Alicia Smith-Tran and Tifany Tien Hang. 2021. Professor–Student Interaction
in the Midst of Illness: A Collaborative Autoethnography. Humanity & Society
(2021), 0160597621991547.
[138] National MS Society. 2022. Managing Relapses. https://www.
nationalmssociety.org/Treating-MS/Managing-Relapses?msclkid=
d4196474b69111ec945819a6ee5e9ca1
[139] Ben Spatz. 2015. What a body can do. Routledge.
[140] Katta Spiel, Kathrin Gerling, Cynthia L. Bennett, Emeline Brulé, Rua M. Williams,
Jennifer Rode, and Jennifer Mankof. 2020. Nothing About Us Without Us:
Investigating the Role of Critical Disability Studies in HCI. In Extended Abstracts
of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA ’20).
Association for Computing Machinery, 1–8. https://doi.org/10.1145/3334480.
3375150
[141] Si Sun, Xiaomu Zhou, Joshua C. Denny, Trent S. Rosenbloom, and Hua Xu. 2013.
Messaging to your doctors: understanding patient-provider communications
via a portal system. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’13). Association for Computing Machinery, 1739–1748.
https://doi.org/10.1145/2470654.2466230
[142] Maxime Taquet, Quentin Dercon, Sierra Luciano, John R Geddes, Masud Husain,
and Paul J Harrison. 2021. Incidence, co-occurrence, and evolution of long-
COVID features: A 6-month retrospective cohort study of 273,618 survivors of
COVID-19. PLoS medicine 18, 9 (2021), e1003773.
[143] Ingrid Torjesen. 2020. NICE backtracks on graded exercise therapy and CBT in
draft revision to CFS guidance.
[144] Tatiana A. Vlahovic, Yi-Chia Wang, Robert E. Kraut, and John M. Levine. 2014.
Support matching and satisfaction in an online breast cancer support community.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
(CHI ’14). Association for Computing Machinery, 1625–1634. https://doi.org/
10.1145/2556288.2557108
[145] Eleanor Rose Walker, Sebastian Charles Keith Shaw, and John Leeds Anderson.
2020. Dyspraxia in medical education: A collaborative autoethnography. The
Qualitative Report 25, 11 (2020), 4072–4093.
[146] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Gener-
alizing from a few examples: A survey on few-shot learning. ACM computing
surveys (csur) 53, 3 (2020), 1–34.
[147] Harriet A Washington. 2006. Medical apartheid: The dark history of medical
experimentation on Black Americans from colonial times to the present. Doubleday
Books.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[148] Susan Wendell. 2001. Unhealthy disabled: Treating chronic illnesses as disabili-
ties. Hypatia 16, 4 (2001), 17–33.
[149] Susan Wendell. 2013. The rejected body: Feminist philosophical refections on
disability. Routledge.
[150] Samantha A. Whitman, Kathleen H. Pine, Bjorg Thorsteinsdottir, Paige Organick-
Lee, Anjali Thota, Nataly R. Espinoza Suarez, Erik W. Johnston, and Kasey R.
Boehmer. 2021. Bodily Experiences of Illness and Treatment as Information
Work: The Case of Chronic Kidney Disease. Proceedings of the ACM on Human-
Computer Interaction 5, CSCW2 (Oct 2021), 383:1–383:28. https://doi.org/10.
1145/3479527
[151] Rua M. Williams, Kathryn Ringland, Amelia Gibson, Mahender Mandala, Arne
Maibaum, and Tiago Guerreiro. 2021. Articulations toward a Crip HCI. Interac-
tions 28, 3 (apr 2021), 28–37. https://doi.org/10.1145/3458453
[152] Maria Wolters. 2019. Accessibility and Stigma: Designing for Users with Invis-
ible Disabilities. https://aaate2019.eu/ 15th International Conference of the
Association for the Advancement of Assistive Technology in Europe, AAATE
2019 ; Conference date: 28-08-2019 Through 30-08-2019.
[153] Alice Wong. 2020. I’m disabled and need a ventilator to live. Am I expendable
during this pandemic? https://www.vox.com/frst-person/2020/4/4/21204261/
coronavirus-covid-19-disabled-people-disabilities-triage
[154] Paige Wyant. 2018. 14 ’Triggers’ That Can Cause a Fibromyalgia
Flare. https://themighty.com/2018/06/fbromyalgia-triggers-fare-causes/
?msclkid=d419aaa5b69111ec98057bf006611ed3
[155] Paige Wyant. 2019. If Your Illness Makes Showering a Struggle, These 16 Memes Are
for You. https://themighty.com/2019/03/showering-chronic-illness-depression-
memes-funny/
[156] Anon Ymous, Katta Spiel, Os Keyes, Rua M. Williams, Judith Good, Eva Hor-
necker, and Cynthia L. Bennett. 2020. “I Am Just Terrifed of My Future” Epis-
temic Violence in Disability Related Technology Research. In Extended Abstracts
of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu,
HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY,
USA, 1–16. https://doi.org/10.1145/3334480.3381828
[157] Daihua X. Yu, Bambang Parmanto, Brad E. Dicianno, Valerie J. Watzlaf, and
Katherine D. Seelman. 2014. Accessible mHealth for patients with dexterity
impairments. In Proceedings of the 16th international ACM SIGACCESS conference
on Computers & accessibility (ASSETS ’14). Association for Computing Machinery,
235–236. https://doi.org/10.1145/2661334.2661402
[158] Tae-Jung Yun and Rosa I. Arriaga. 2013. A text message a day keeps the pul-
monologist away. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’13). Association for Computing Machinery, 1769–1778.
https://doi.org/10.1145/2470654.2466233
[159] Xiaomu Zhou, Si Sun, and Jiang Yang. 2014. Sweet Home: understanding diabetes
management via a chinese online community. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI ’14). Association for
Computing Machinery, 3997–4006. https://doi.org/10.1145/2556288.2557344
[160] Wei Zhu, Boyd Anderson, Shenggao Zhu, and Ye Wang. 2016. A Computer
Vision-Based System for Stride Length Estimation using a Mobile Phone Camera.
In Proceedings of the 18th International ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS ’16). Association for Computing Machinery, 121–130.
https://doi.org/10.1145/2982142.2982156
[161] Irving Kenneth Zola. 1972. Medicine as an institution of social control. The
sociological review 20, 4 (1972), 487–504.
[162] Irving Kenneth Zola. 1982. Missing pieces: A chronicle of living with a disability.
Temple University Press.
 Should I Say “Disabled People” or “People with Disabilities”?
Language Preferences of Disabled People Between Identity- and
Person-First Language
Ather Sharif Aedan L. McCall Kianna R. Bolante
asharif@cs.washington.edu aedanmc@cs.washington.edu kbolan@cs.washington.edu
Paul G. Allen School of Computer Paul G. Allen School of Computer Paul G. Allen School of Computer
Science & Engineering | DUB Group, Science & Engineering, Science & Engineering,
University of Washington University of Washington University of Washington
Seattle, Washington, USA Seattle, Washington, USA Seattle, Washington, USA

Figure 1: Language preferences of disabled people. (a) shows the overall preferences for United States, United Kingdom, and
Canada combined (N =519); (b) shows the preferences of disabled people from the United States (N =366); (c) shows the prefer-
ences of disabled people from the United Kingdom (N =112); and (d) shows the preferences of disabled people from Canada
(N =14). I F L stands for “Identity-First Language,” PF L stands for “Person-First Language,” and N P stands for “No Preference.”

ABSTRACT people may have multiple or no preferences. To make our survey

The usage of identity- (e.g., “disabled people”) versus person-frst
language (e.g., “people with disabilities”) to refer to disabled people
has been an active and ongoing discussion. However, it remains
unclear which semantic language should be used, especially for
diferent disability categories within the overall demographics of
disabled people. To gather and examine the language preferences of
disabled people, we surveyed 519 disabled people from 23 countries.
Our results show that 49% of disabled people preferred identity-frst
language whereas 33% preferred person-frst language and 18% had
no preference. Additionally, we explore the intra-sectionality and
intersectionality of disability categories, gender identifcations, age
groups, and countries on language preferences, fnding that lan-
guage preferences vary within and across each of these factors. Our
qualitative assessment of the survey responses shows that disabled
This work is licensed under a Creative Commons Attribution International
4.0 License.
data publicly available, we created an interactive and accessible
live web platform, enabling users to perform intersectional explo-
ration of language preferences. In a secondary investigation, using
part-of-speech (POS) tagging, we analyzed the abstracts of 11,536
publications at ACM ASSETS (N =1,564) and ACM CHI (N =9,972),
assessing their adoption of identity- and person-frst language. We
present the results from our analysis and ofer recommendations
for authors and researchers in choosing the appropriate language
to refer to disabled people.
CCS CONCEPTS
• Social and professional topics → People with disabilities; •
General and reference → Surveys and overviews; Empirical stud-
ies; • Human-centered computing → Empirical studies in acces-
sibility; Web-based interaction; • Computing methodologies
→ Information extraction.

ASSETS ’22, October 23–26, 2022, Athens, Greece

© 2022 Copyright held by the owner/author(s). KEYWORDS
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3544813 identity-frst, person-frst, language, disability, preferences, survey
ASSETS ’22, October 23–26, 2022, Athens, Greece
ACM Reference Format:
Ather Sharif, Aedan L. McCall, and Kianna R. Bolante. 2022. Should I Say
“Disabled People” or “People with Disabilities”? Language Preferences of
Disabled People Between Identity- and Person-First Language. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
18 pages. https://doi.org/10.1145/3517428.3544813
1 INTRODUCTION
Words have power. They refect attitudes that speakers want to
exchange [23]. They also shed light on the sensitivity to matters
involving social justice and cultural awareness, especially for under-
represented and marginalized groups [13], such as disabled people.
Several terms, such as “retarded,” are now considered outdated as
they assert negative connotations on disabled people [26, 73] and
some pejorative terms, such as “crippled” and “gimp,” have been
reclaimed by the disability community [1, 59]. Similarly, the debate
between identity- (using identity frst; e.g., “disabled people”) and
person-frst language (using people frst; e.g., “people with disabili-
ties”) has been an active and ongoing discussion [2, 17, 18, 27].
The American Psychological Association (APA), the American
Medical Association, the American Psychiatric Association, the
American Speech-Language-Hearing Association, and the Associ-
ated Press advocate for person-frst language [2, 17, 27]. However,
a recent inquiry from Vivanti [70] regarding language usage for
autistic people shows that well-intentioned scholars are still unsure
of the “right” language to use. Furthermore, Vivanti’s inquiry is par-
ticularly relevant to our exploration as the autistic community has
long advocated for identity-frst language for themselves, contrary
to the general recommendation from the APA, which explicitly
asks the writers to “put the person frst” [3, 27]. Hence, the prefer-
ences of the disability categories within the disability community
should direct the language used to refer to that particular group
[9, 26, 43, 51]. Additionally, as societies evolve, these preferences
may become obsolete over time. (For example, the National Federa-
tion of the Blind (NFB) [44] has, in recent years, started advocating
for “blind and low-vision” in place of “visually impaired.”)
To understand the preferences of disabled people in the usage
of language that refers to them, we designed and distributed a
survey globally. Specifcally, the survey recorded the respondents’
preference between identity-frst language, person-frst language,
and no preference. Additionally, we asked the respondents for the
reasons behind their preferences. We recorded the timestamps of
the survey responses to track the language preferences over time.
As of the date of writing this paper, our survey had responses from
519 disabled people, representing nine disability categories, six age
groups, and 23 countries. Our fndings show that, overall, disabled
people prefer identity-frst language (49.0%) compared to person-
frst language (33.0%). We also explored the intra-sectionality and
intersectionality of disability categories, gender identity, age group,
and country on language preferences. We found that language pref-
erences vary both within and across these factors. For example,
people with mobility disabilities prefer person-frst language (46.2%)
over identity-frst language (39.4%). Our qualitative assessment of
the responses shows that disabled people may have multiple or
no language preferences. To make our survey data available to
the public, we created an interactive and accessible web platform
Ather Sharif, Aedan L. McCall, and Kianna R. Bolante
displaying live results from the survey. Furthermore, the web plat-
form enables users to flter data by any combination of disability
categories, gender identities, age groups, countries, and years to
support a granular information extraction.
As a secondary exploration, to shed light on the language adop-
tion at academic venues, we analyzed the abstracts from 11,536
publications at ACM SIGACCESS Conference on Computers and
Accessibility (ASSETS; N =1,564) and ACM Conference on Human
Factors in Computing Systems (CHI; N =9,972) from 2000- to 2021
(20 years, excluding 2001 and 2003). Our results show that combined
and separately, both the conferences employed a higher usage of
person-frst language (54.4% combined; 52.6% for ASSETS and 57.1%
for CHI) compared to identity-frst language (45.6% combined; 47.4%
for ASSETS and 42.9% for CHI). Altogether, our fndings indicate
that person-frst language is used more frequently despite disabled
people showing a higher preference for identity-frst language.
The main contributions of our work are as follows:
(1) Empirical results from a survey of 519 disabled people from
23 countries, showing their preferences between identity-
and person-frst language. Overall, 49% of disabled people
preferred identity-frst language, 33% of them were in favor
of person-frst language, and 18% had no preference.
(2) Empirical results from analyzing the abstracts of 11,536 aca-
demic publications published at ACM ASSETS (N =1,564) and
ACM CHI (N =9,972) from the past 20 years, showing the
total count of identity- and person-frst terminologies used
per year. Overall, the publications used person-frst language
9.7% and 24.9% more than identity-frst language at ASSETS
and CHI, respectively.
(3) Accessible web platform, showing live survey results. The
web platform enables users to flter their language prefer-
ences by any combination of disability categories, age groups,
gender identities, countries, and years. We present the de-
sign, functionality, and implementation of our system. Ad-
ditionally, we publish our web platform, making it publicly
available at https://disabilityterminology.athersharif.com/.
2 BACKGROUND AND RELATED WORK
We review literature from Disability Studies on the discussion of us-
ing identity- versus person-frst language to refer to disabled people.
Identity-frst language (IFL) places the identity frst, acknowledging
the disability of a person as their defning characteristic (e.g., “dis-
abled person”). In contrast, person-frst language (PFL) emphasizes
the person frst and then their disability (e.g., “person with a disabil-
ity”). We also review prior work on disability language preference
surveys and analysis of accessibility-related academic publications.
2.1 Identity- Versus Person-First Language in
Disability Studies
Disability Studies is an interdisciplinary feld that explores the
political, intellectual, and cultural dimensions of disability in soci-
ety [21, 29, 33]. Several scholars and researchers have contributed
to the discussion of using identity- versus person-frst language
to refer to people with disabilities. As of the date of this writing,
Semantic Scholar [65], a search engine for academic publications,
Identity- Versus Person-First Language
shows 556 results for a search query containing the terms “person-
frst,” “identity-frst,” and “disability” (all the words appearing at
least once anywhere in the publication text). We only explored the
publications relevant to our work.
Identity-frst language (IFL), which emphasizes the disability
identity of the person, can help reclaim once pejorative terms (such
as “crippled”) used for disabled people [1, 59] and can be instrumen-
tal in driving social change [26, 39]. For example, most recently,
Netfix [45], a popular streaming service, released a documentary ti-
tled “Crip Camp” [36, 54], which is an indispensable flm that sheds
light on the history of the disability rights movement [24, 25], high-
lighting the inequities disabled people face in society. The National
Federation for the Blind [44], which is a national advocacy orga-
nization representing blind and low-vision (BLV) people, elected
to use IFL to refer to BLV individuals in its 1993 resolution [46],
stating that person-frst language “implies shame instead of true
equality” [16, 17]. Similarly, the autistic community is a strong pro-
ponent of using IFL for autistic people [8, 10, 34, 35, 50], expressing
that IFL encourages society to acknowledge and celebrate them as
autistic individuals [35, 50, 52]. Additionally, IFL can help increase
public visibility into the stigma disabled people experience as an
underrepresented minority group and assist in reducing that stigma
to build a more inclusive society [8].
On the other hand, person-frst language (PFL), which recognizes
the person before their diagnostic label, is the most widely-used
language style [2, 17, 27, 49, 66]. One of the frst scholarly works to
advocate for PFL was by Wright [75], who suggested that the person
should be the primary focus in language choices to eradicate the
dehumanizing language used to describe disabled people through-
out the twentieth century [2]. Since then, people and organizations
have widely used PFL with well-intended goals of attenuating the
stigma associated with disabilities [27, 66]. (However, several schol-
ars argue that, although well-intended in its original proposition,
PFL may have overcorrected to accentuate this stigma, particularly
in scholarly writing [27, 31, 66].) PFL’s wide adoption includes
its usage in the Americans with Disabilities Act (ADA) [67, 74]
and recommendations of use in academic writing by numerous
style guides, including the American Psychology Association (APA),
American Medical Association, American Psychiatric Association,
American Speech-Language Hearing Association, and Associated
Press [2, 17, 27].
While PFL is employed widely by several organizations to avoid
daily discourse [17], disability rights advocates and activists pro-
mote IFL [2, 40]. However, several scholars and researchers have
identifed the need to utilize both appropriately, claiming that a
singular linguistic model is non-representative of the entire dis-
ability community [15, 34, 43, 51, 61]. Therefore, disability groups
should direct the language that describes their respective com-
munities [9, 26, 43, 51]. We are sensitive to this discussion and
acknowledge that each individual may have a unique language
preference, which may or may not align with the consensus among
the members of their respective disability category. We also note
that at least one of the authors of this work identifes as a disabled
person. In our work, we seek to gather insights into the language
preferences of disabled people, paying close attention to avoid any
of our personal biases and beliefs on the matter.
ASSETS ’22, October 23–26, 2022, Athens, Greece
2.2 Disability Language Preference Surveys
Several researchers have administered surveys with disabled people
to draw inferences about their experiences and language prefer-
ences [6, 22, 37, 38, 48, 60]. Lister et al. [38] conducted surveys with
723 disabled students to investigate their language preferences for
communication in higher educational institutions, fnding that stu-
dents were uncomfortable with terminologies addressing them as
“disabled” and that language preferences diverged across contexts
and demographics of students. They recommended exploring dif-
ferential and inclusive approaches to fnd the appropriate language
rather than focusing on a single model. Levy et al. [37] surveyed
63 disabled people to investigate the respectfulness of the termi-
nologies used in ASSETS publications to refer to disabled people,
reporting that their respondents found the terms “disabled peo-
ple” and “diferently-abled” the most respectful and disrespectful,
respectively. In a similar exploration, Fernald [22] explored the
diferences between American disability terminologies and the ter-
minologies used in other English-speaking countries by surveying
26 disability-related professional and advocacy organizations, dis-
covering disparities in the preference of disability language between
diferent countries.
Additionally, Bickford et al. [6] explored the preferences of 100
blind and low-vision people on disability terminology, albeit they
used interviews to record their participants’ preferences. They
found that 37% of the individuals interviewed did not have a pref-
erence, and among those who did, 76% favored identity-frst termi-
nologies.
Our work draws inspiration from Lister et al.’s [38] recommen-
dation of utilizing diverse approaches to determine the appropri-
ate language to refer to disabled people. Additionally, we follow
Bickford et al. [6] in exploring a polychotomous classifcation of
preferences (including “no preference” as an option), as opposed
to using the dichotomy of choices between IFL and PFL. However,
in contrast to these surveys, our survey is the frst scholarly work
to perform all of the following in combination: (1) Identify the lan-
guage preferences (IFL, PFL, no preference, or multiple preferences)
of disabled people representing at least one of several disability
categories; (2) explore intersectionality and intra-sectionality of
disability category, gender identity, age, country, and year in deter-
mining language preferences; (3) examine the temporal evolution
of language preferences; and (4) display live results from the sur-
vey through an interactive website that enables users to flter their
query by any combination of the factors mentioned above.
2.3 Analyzing Accessibility-Related Academic
Publications
Researchers have analyzed text from academic publications to in-
vestigate matters in accessibility research [5, 12, 37, 42, 62]. Most
recently, Levy et al. [37] conducted a qualitative literature review
of 106 papers published at ASSETS from 2018- to 2020 (3 years) to
understand the terminology used to refer to disabled people. They
found that authors used PFL terms more than IFL terms. However,
as they stated, their exploration was only preliminary and contained
a small, non-representative sample size. Mack et al. [42] analyzed
835 technical papers published at ASSETS and CHI, refecting on
the growth and history of the feld of accessibility. Their results
ASSETS ’22, October 23–26, 2022, Athens, Greece Ather Sharif, Aedan L. McCall, and Kianna R. Bolante

Table 1: Overview of demographics for our N = 491 participants (after exclusion) per country, further classifed by disability
category, gender identity, and age group. N is the total number of participants and % is the percentage compared to the total
number of participants. For the “overall” column, the % is shown as “-,” naturally assuming it to be 100.

Overall United States United Kingdom Canada

N % N % N % N %
Overall 491 - 365 74.3% 112 22.8% 14 2.9%
By Disability Category (DSB)
Mobility 104 - 59 56.7% 42 40.4% 3 2.9%
Visual 248 - 235 94.8% 7 2.8% 6 2.4%
Cognitive 112 - 72 64.3% 34 30.4% 6 5.4%
Learning 36 - 24 66.7% 11 30.6% 1 2.8%
Neurological 53 - 30 56.6% 22 41.5% 1 1.9%
Auditory 50 - 39 78.0% 8 16.0% 3 6.0%
Chronic Illness 149 - 57 38.3% 90 60.4% 2 1.3%
Mental Health Related 122 - 72 59.0% 44 36.1% 6 4.9%
Other 4 - 3 75.0% 1 25.0% 0 0.0%
By Gender Identity (GND)
Woman 305 - 211 69.2% 89 29.2% 5 1.6%
Man 137 - 121 88.3% 10 7.3% 6 4.4%
Non-binary 49 - 31 63.3% 14 28.6% 4 8.2%
Prefer not to disclose 7 - 6 85.7% 1 14.3% 0 0.0%
By Age Group
18-25 61 - 52 85.2% 6 9.8% 3 4.9%
26-35 125 - 96 76.8% 25 20.0% 4 3.2%
36-50 150 - 106 70.7% 40 26.7% 4 2.7%
51-60 88 - 58 65.9% 30 34.1% 0 0.0%
61-70 49 - 38 77.6% 9 18.4% 2 4.1%
71 and older 18 - 15 83.3% 2 11.1% 1 5.6%

show that accessibility research focuses disproportionately on the 3.1 Method

blind and low-vision (BLV) community. Although their methodol-
ogy and sample size are plausible, their work did not explore the
language used to refer to disabled people.
Our work examines the language used to refer to disabled peo-
ple by analyzing the abstracts from 11,536 publications (including
poster papers and extended abstracts) published at ASSETS and
CHI from 2000- to 2021 (20 years, excluding 2001 and 20031 ).
3 PREFERENCES SURVEY
To gather and understand the language preferences between identity-
and person-frst language, we surveyed 519 disabled people globally.
We present our methodology to conduct the survey and results from
analyzing the survey responses.
1 ASSETS skipped publications in 2001 and 2003
We administered an online survey to assess the language prefer-
ences of disabled people using a mixed-methods approach. Specif-
ically, we investigated the diference in preferences based on dis-
ability category, gender identity, and country of residence using
quantitative methods. Additionally, we evaluated the reasons be-
hind their choice of preferences using qualitative methods.
3.1.1 Procedure. Participants took part in our survey online, with-
out supervision. The survey comprised three steps. In the frst step,
the survey showed the purpose of our study, eligibility criteria,
defnitions and examples of identity- and person-frst language,
and data anonymity clause. We collected demographic information
from our participants in step two, including their gender identity,
pronouns, age, country, disability category, diagnosis, and age of di-
agnosis. We selected the disability categories by contacting several
Identity- Versus Person-First Language ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 2: Overview of language preferences from N = 895 observations, further classifed by disability category, gender iden-
tity, country, and age group. N is the total number of participants and % is the percentage compared to the total number of
participants. For the “overall” column, the % is shown as “-,” naturally assuming it to be 100.

Overall United States

IFL PFL NP IFL PFL NP
T N % N % N % T N % N % N %
Overall 895 435 48.6 295 33.0 165 18.4 598 290 48.5 172 28.8 136 22.7
By Disability Category (DSB)
Mobility 104 41 39.4 48 46.2 15 14.4 59 24 40.7 24 40.7 11 18.6
Visual 248 99 39.9 77 31.0 72 29.0 235 94 40.0 70 29.8 71 30.2
Cognitive 112 80 71.4 18 16.1 14 12.5 72 49 68.1 11 15.3 12 16.7
Learning 36 22 61.1 9 25.0 5 13.9 24 11 45.8 9 37.5 4 16.7
Neurological 53 22 41.5 25 47.2 6 11.3 30 12 40.0 14 46.7 4 13.3
Auditory 50 29 58.0 11 22.0 10 20.0 39 21 53.8 8 20.5 10 25.6
Chronic Illness 149 65 43.6 69 46.3 15 10.1 57 33 57.9 16 28.1 8 14.0
Mental Health Related 122 70 57.4 34 27.9 18 14.8 72 41 56.9 17 23.6 14 19.4
Other 4 1 25.0 3 75.0 0 0.0 3 1 33.3 2 66.7 0 0.0
By Gender Identity (GND)
Woman 305 114 37.4 136 44.6 55 18.0 211 85 40.3 78 37.0 48 22.7
Man 137 54 39.4 45 32.8 38 27.7 121 47 38.8 38 31.4 36 29.8
Non-binary 49 40 81.6 3 6.1 6 12.2 31 24 77.4 3 9.7 4 12.9
Prefer not to disclose 7 3 42.9 2 28.6 2 28.6 6 3 50.0 1 16.7 2 33.3
By Country
United States 365 157 43.0 119 32.6 89 24.4 - - - - - - -
United Kingdom 112 45 40.2 60 53.6 7 6.3 - - - - - - -
Canada 14 6 42.9 6 42.9 2 14.3 - - - - - - -
By Age Group
18-25 61 37 60.7 13 21.3 11 18.0 52 32 61.5 10 19.2 10 19.2
26-35 125 65 52.0 38 30.4 22 17.6 96 51 53.1 24 25.0 21 21.9
36-50 150 60 40.0 63 42.0 27 18.0 106 44 41.5 39 36.8 23 21.7
51-60 88 28 31.8 41 46.6 19 21.6 58 17 29.3 23 39.7 18 31.0
61-70 49 15 30.6 20 40.8 14 28.6 38 10 26.3 16 42.1 12 31.6
71 and older 18 3 16.7 10 55.6 5 27.8 15 3 20.0 7 46.7 5 33.3

disability-related advocacy organizations (for transparency, at least
one of the authors is disabled and is a member of some of these
organizations). Participants were allowed to select multiple disabil-
ity categories. To appropriately ask our participants which gender
identities they relate to, we followed guidelines from [63]. Similar
to the disability categories, we enabled our participants to choose
multiple options from choices including “women,” “men,” “non-
binary,” “prefer not to disclose,” and “prefer to self-describe.” Our
survey displayed an additional text feld if participants preferred to
self-describe their gender identit(y/ies).
In the fnal step, we asked the participants their preference be-
tween identity- and person-frst language, providing them with the
ASSETS ’22, October 23–26, 2022, Athens, Greece Ather Sharif, Aedan L. McCall, and Kianna R. Bolante

Figure 2: Percentage of total count for language preferences by (a) disability categories, (b) countries, (c) age groups, and
(d) gender identities. (a), (c), and (d) are further classifed by regions “overall” (United States, United Kingdom, and Canada)
and “United States.” I F L stands for “Identity-First Language,” PF L stands for “Person-First Language,” and N P stands for “No
Preference.”

additional choice of “no preference.” Finally, we asked the partic-
ipants to state the reason for their preference in detail. To obtain
further contextual insights, we inquired if the participants were
familiar with the terms “identity-frst” and “person-frst” before
taking this survey and the language style they encounter the most
in their everyday lives. (We provide our survey responses [col-
lected as of the time of this writing—before March 16th , 2022] in
the supplementary materials.)
3.1.2 Participants. Our survey respondents (“participants”) vol-
untarily took part in our online survey, advertised through word-
of-mouth, snowball sampling, social media channels (Facebook
and Twitter), and email distribution lists for disabled people. We
contacted several local, national, and global disability-related or-
ganizations (e.g., The National Federation of the Blind [44]) via
contact forms and email addresses mentioned on their websites for
advertising the survey. Altogether, 519 participants (M=42.7 years,
SD=14.9) from 23 countries responded to our survey. We excluded
the responses from countries that had a total count of fewer than
10 responses.
After exclusion, our participant pool comprised 491 participants
(M=42.8 years, SD=14.9) from three countries: (1) United States
(N =365); (2) United Kingdom (N =112); and (3) Canada (N =14).
Three-hundred-and-fve participants identifed as women, 137 as
men, and 45 as non-binary. Eleven participants described their gen-
der identity themselves, and seven did not disclose their gender
Identity- Versus Person-First Language
Table 3: Summary of statistical results from N =895 overall (CTY =“United States,” “United Kingdom,” and “Canada”) and N =598
specifc (CTY =“United States”) observations. “DSB” is the disability category, “GND” is the gender identity, and “CTY” is the
country. Cramer’s V is a measure of efect size [20].
Overall
N χ2 p Cramer’s V
DSB 895 33.95 < .05 .20
GN D 895 61.26 < .001 .26
CTY 895 16.99 < .05 .14
Aдe 895 26.17 < .001 .17
identity. Table 1 shows a demographic breakdown of participants
across the United States, United Kingdom, and Canada.
As we noted in Section 2.1 of this paper, at least one of the
authors identifes as a disabled person. To avoid our personal biases
and beliefs, we did not partake in our survey.
3.2 Quantitative Evaluation
We used the following factors and levels:
• Disability Category (DSB), within-Ss.: {Mobility, Visual, Cog-
nitive, Learning, Neurological, Auditory, Chronic Illness,
Mental-Health Related}
• Gender Identity (GN D), within-Ss.: {Woman, Man, Non-Binary}
• Country (CTY ), within-Ss.: {United States, United Kingdom,
Canada}
Our dependent variables were Language Preference (PRF) and
Language Commonly Encountered (LCE). To analyze PRF , we used
a polychotomous representation (0 for “identity-frst language,”
1 for “person-frst-language,” and 2 for “no preference”) and a
multinomial logistic regression model [4, 69] with the above fac-
tors and a covariate to control for Age. Our statistical model was:
PRF ← DSB + GN D + CTY + Aдe. We did not include interactions
between our factors as our research exploration centered around
investigating the main efects of these factors with PRF .
We analyzed LCE classifying it dichotomously (0 for “identity-
frst language” and 1 for “person-frst language”), using a mixed
logistic regression model [28] with the above factors and a covariate
to control for Age. Our statistical model was the same as for PRF ,
and we only explored the main efects of these factors with LCE.
Additionally, we performed a separate analysis for CTY =United
States as a large majority (N =74.3%) of our survey respondents
were from the United States. We used the same above-stated model,
naturally removing CTY from the list of terms. We present our
quantitative results for Language Preference (PRF) and Language
Commonly Encountered (LCE).
3.2.1 Language Preference (PRF). Disability Category (DSB) had a
signifcant main efect on PRF (χ 2 (16, N =895)=33.95, p<.05, Cramer’s
V =.20), indicating that PRF difers signifcantly between the nine
disability categories. (Cramer’s V is a measure of efect size for χ 2
tests, ranging from 0 to 1. Values greater than .6 demonstrate large
efect, between .2 and .6 demonstrate medium efect, and less than
ASSETS ’22, October 23–26, 2022, Athens, Greece
United States
N χ2 p Cramer’s V
598 21.47 .161 .19
598 13.19 < .05 .15
- - - -
598 37.50 < .001 .25
.2 demonstrate small efect.) Disability categories including Visual,
Cognitive, Learning, and Auditory preferred IFL (39.9%, 71.4%, 61.1%,
and 58.0%, respectively) whereas categories Mobility, Neurological,
and Chronic Illness preferred PFL (46.2%, 47.2%, and 46.3%, respec-
tively). Figure 2 and Table 2 show the PRF percentages across all
independent variables used in our analysis. For participants from
the United States, DSB did not have a statistically signifcant efect
on PRF (p ≈.161).
We also found a signifcant main efect of Gender Identity (GND)
on PRF (χ 2 (6, N =895)=61.26, p<.001, Cramer’s V =.26). This result
indicates that PRF difers signifcantly between diferent gender
identities. People who identifed as Non-binary and as Man pre-
ferred IFL (81.6% and 39.4%, respectively) and people who iden-
tifed as Woman preferred PFL (44.6%). For participants from the
United States, GN D also had a signifcant main efect on PRF (χ 2 (6,
N =598)=13.19, p<.05, Cramer’s V =.15), with all gender categories
preferring IFL over PFL.
The factor Country (CTY) also had a signifcant main efect on
PRF (χ 2 (4, N =895)=16.99, p<.05, Cramer’s V =.14). Specifcally, peo-
ple in the United States preferred IFL (43.0%), people in the United
Kingdom preferred PFL (53.6%), and people in Canada demonstrated
an equal preference.
We investigated the efects of Age on PRF . Age had a signifcant
efect on PRF overall (χ 2 (2, N =895)=26.17, p<.001, Cramer’s V =.17)
and for participants from the United States ((χ 2 (16, N =598)=37.50,
p<.001, Cramer’s V =.25), indicating that PRF difered signifcantly
across the ages of our participants. Participants 35 or older preferred
PFL (54.0%), whereas participants under 35 preferred IFL (43.9%). For
participants from the US, the trend was similar, but for participants
50 years or older (41.4%). Those under 50 preferred IFL (50.0%).
Table 3 shows the statistical results from all of our analyses.
Bickford et al. [6] did not fnd signifcant main efects of Gender
and Age on language preferences of blind and low-vision individuals
(N =100). As they conducted their study 18 years ago, we performed
a second analysis, examining the current efects of GN D and Age on
PRF for our visually-disabled participants. We found that Age had
a signifcant main efect on PRF both overall (χ 2 (2, N =250)=13.52,
p<.05, Cramer’s V =.23) and for participants from the US ((χ 2 (16,
N =237)=10.76, p<.05, Cramer’s V =.21). GN D, however, did not have
a signifcant main efect on PRF . Hence, our results only partially
ASSETS ’22, October 23–26, 2022, Athens, Greece
agreed with those from Bickford et al.’s analyses, showing that lan-
guage preferences signifcantly varied among our visually-disabled
participants.
3.2.2 Language Commonly Encountered (LCE). The factor Disabil-
ity Category (DSB) had a signifcant main efect on LCE (χ 2 (8,
N =895)=21.98, p<.05, Cramer’s V =.16). This result indicates that
LCE difers signifcantly between the nine disability categories.
Specifcally, disability categories including Visual, Cognitive, Neuro-
logical, Auditory, Chronic Illness, and Mental-Health Related com-
monly encounter IFL (62.9%, 58.0%, 60.4%, 56.0%, 55.0%, and 57.4%,
respectively) whereas category Mobility encounter PFL more com-
monly (56.7%). Category Learning had an equal percentage of lan-
guage encounter (50%).
Similarly, Country (CTY) also had signifcant main efects on LCE
(χ 2 (2, N =895)=16.23, p<.001, Cramer’s V =.14), showing that LCE
difers signifcantly between the three countries used in our analysis.
Our participants from the United States and United Kingdom more
commonly encountered IFL (56.2% and 56.3%, respectively), whereas
participants from Canada encountered PFL more commonly (71.4%).
The factors GN D and Aдe did not have a statistically signifcant
efect on LCE. Additionally, none of the factors were statistically
signifcant for participants from the United States.
3.3 Qualitative Evaluation
To qualitatively assess the language preferences of disabled peo-
ple, we analyzed their free-form survey responses. Specifcally, we
examined the reasons for their choice of preference. We used stan-
dard semantic thematic analysis processes [11, 47] to analyze the
responses. Our fnal analysis revealed three themes: (1) one size
does not ft all; (2) not everyone has a preference; and (3) people can
have multiple preferences. We discuss these below, in turn.
3.3.1 One Size Does Not Fit All. Our frst theme shows that the
language preferences of disabled people can vary between diferent
disability categories. For example, as shown in Table 2, the blind and
low-vision community prefers IFL, whereas people with mobility
disabilities prefer PFL. Therefore, using one-size-fts-all language
might not be appropriate for all disabled people. P26, who is autistic,
and P20, who is blind, had emphatic preferences for IFL:
Person-first language implies that being disabled
is a bad thing, which we should continue to
stigmatize. I’m a disabled, autistic person,
not a person with autism or a person with a
disability---you can’t separate those experiences
out from the rest of me. (P26)
I’m not ashamed of my disability. I am who
I am, there’s no point in denying that.
‘‘Person with blindness’’ just... sounds wrong
to me. I feel like person-first language
tries to hide our disabilities. It communicates
the message that, even though you’re disabled,
you’re still a person, treating disability
as something wrong and something to be ashamed
of, which is not the way I feel about it. (P20)
Ather Sharif, Aedan L. McCall, and Kianna R. Bolante
Similarly, P220, who identifes as a person with a mobility dis-
ability, and P120, who is a person with a chronic illness, expressed
clear preferences for PFL:
It’s not a deep loathing, but I just don’t
like it when ‘‘disabled’’ is literally the
first way a person learns of me, as in, ‘‘A
disabled woman I work with.’’ There, I’m
disabled before literally anything else, and
it’s really not one of the most interesting
things about me. (P220)
It is important to me that a person is said
first. We are humans. We have feelings and
deserve to be recognized as a person before
a disabled person. (P120)
Overall, in line with our quantitative results, our frst theme
shows that language preferences vary between disability categories.
3.3.2 Not Everyone Has a Preference. Our fndings show that 18.4%
and 22.7% survey respondents overall and in the United States,
respectively, had no preference between identity- and person-frst
language. For example, P308, P419, and P200 shared their opinions:
All I care about is that people know I’m
blind. If someone wants to say I’m an individual
who is blind, or that I’m a blind person, it
doesn’t make a difference to me. (P308)
I do not have a preference because I prefer
others to feel comfortable and am secure
enough in who I am as a person to not become
overly offended or upset by the ways people
go about communication. Far too often this
issue of person vs identity-first language
becomes unnecessarily heated and seems to
cause more anxiety than is needed, healthy,
and helpful. (P419)
The order of the person and the disability
doesn’t change the end result. Whether the
person or the disability comes first the
disability is still present and, in my opinion,
does not really modify any sort of context.
(P200)
Our second theme shows that disabled people may not have a
preference as long as their disability is “seen” and acknowledged.
3.3.3 People Can Have Multiple Preferences. Our third theme re-
veals that disabled people can have multiple language preferences.
We found that disabled people with multiple disabilities may have
diferent preferences for each disability category. For example, P447,
who is autistic and has PTSD, had diferent preferences for each
disability category:
I use both. I often use identity-first language
when relaying the fact that I am autistic,
and will use person-first language to explain
that I have PTSD. I do this as it seems to be
the preferential consensus of self-advocates
that I know, and it helps people feel comfortable
if I use the language they prefer. I don’t
Identity- Versus Person-First Language
Figure 3: The user interface of the accessible web platform showing fltering options for intersectional explorations using
multi-select dropdown for age group, country, disability category, gender identity, and month/year. Tabs for these factors are
also shown for intra-sectional exploration.
feel very strongly about either one over the
other personally. I think individuals have
the right to choose to use whichever is most
comfortable for them. (P447)
Similarly, P393, who is autistic and has anxiety and ADHD,
shared her opinions:
Depends. With Autism I prefer identity first.
Mostly because I do not see my autism as a
condition. I don’t want it treated like a
disease. With my anxiety I prefer person
first. The same is true of ADHD. (P393)
Additionally, disabled people may have multiple preferences de-
pending on the context. P329 explained their preference diferences
based on professional and informal environments:
I code switch between the two depending on
my audience. When speaking informally, or to
groups of people with disabilities, I use
blind person, but for professional settings
I use person with a disability. (P329)
In our third theme, we found that disabled people may have
multiple preferences. Our fndings show that their language pref-
erences could vary for each disability category they represent or
based on the context or environment. To make our survey data
publicly available, we created an accessible web platform, which
we present in the section below.
ASSETS ’22, October 23–26, 2022, Athens, Greece
4 ACCESSIBLE WEB PLATFORM
To provide transparency and comprehensive means of obtaining up-
to-date language preferences of disabled people, we developed an
accessible web platform that shows the live results from our survey
(updated immediately after a participant flls out the survey). We
present our web platform’s design considerations, functionalities,
and implementation details.
4.1 Design Considerations
In developing the accessible web platform, our goal was to create a
straightforward interface that allows users to explore and extract
information efectively and granularly. Our user interface had two
sections: (1) Sidebar Navigation; and (2) Content Area.
4.1.1 Sidebar Navigation. The sidebar navigation contained fve
options (as shown in Figure 3): (1) Home; (2) Identity-First Language;
(3) Person-First Language; (4) Share Your Preference; and (5) Contact.
Home page displays the survey results and is the entrypoint for the
website. Identity-First Language and Person-First Language pages
show defnitions and examples for identity- and person-frst lan-
guage, respectively. Clicking on Share Your Preference navigates the
user to the survey, whereas the Contact page displays the names
and email addresses for the members of the research team.
4.1.2 Content Area. We displayed the contents for each page in the
content area of our website (as shown in Figure 3). For the Home
page, the contents involved tabs for each independent variable and
a visualization displaying the distribution of language preferences
between IFL (identity-frst language), PFL (person-frst language),
ASSETS ’22, October 23–26, 2022, Athens, Greece
and NP (no preference). All the other pages contained appropriate
text organized using headings and paragraphs.
4.2 Intersectional & Intra-sectional
Exploration
We recorded the disability category, gender identity, age, and coun-
try of the survey respondents to enable users to examine the lan-
guage preferences of disabled people both separately for each inde-
pendent variable (intra-sectionality) and in combination (intersec-
tionality). We implemented “tabs” (or subsections) to support users
in exploring the intersectionality and intra-sectionality of these
factors. The Overall tab allows users to examine the intersectional-
ity of these factors, whereas the other subsections enable users to
perform intra-sectional exploration. Additionally, we added the By
Year tab to track the evolution of language preferences over time.
As we started collecting the survey results only a few months back,
the data under this subsection, at present, may not be signifcant.
However, we hope that tracking the survey data over time will
reveal patterns and results worth exploring.
In the Overall tab (Figure 3), users can flter the results using the
multi-select dropdowns in our web-interface to perform a granular
intersectional exploration as per their needs and curiosity. For
example, users can select “United States” and “United Kingdom”
from the countries dropdown and “2021” and “2022” to display the
language preferences collected from participants from these two
countries in the past two years. For simplicity, we chose a pie chart
to display the results, showing the total count and percentage for
each type of language preference. By Disability (see Appendix A,
Figure 5), By Age Group (see Appendix B, Figure 6), By Gender (see
Appendix C, Figure 7), and By Country (see Appendix D, Figure 8)
subsections display the results using a bar chart, whereas By Year
(see Appendix E, Figure 9) shows the results using a smoothed line
graph. We created our graphs using the D3 visualization library [7].
4.3 Accessibility
We used VoxLens [58], an open-source JavaScript plug-in that im-
proves the accessibility of online data visualizations using a multi-
modal approach. However, VoxLens is currently only applicable
to visualizations created using two-dimensional single-series data.
Therefore, we only used it with our pie charts. For the other visual-
izations containing multi-series data, we followed the recommen-
dations from prior work [41, 56, 57] to generate the alternative text
(“alt-text”) dynamically. Specifcally, we used Accessible Rich Inter-
net Applications (ARIA) attributes [71] to add alt-text to our graphs.
Additionally, we added a tabular representation of data only visible
to screen readers, similar to the accessibility measures employed by
Google Charts [14, 56]. For example, the alt-text for our bar chart
showing the language preferences by disability category was:
Bar chart showing counts for identity-frst, person-frst,
and no preference per disability. People with mobility
disabilities prefer person-frst language, visually dis-
abled people prefer identity-frst language, cognitively
disabled people prefer identity-frst language, learning
disabled people prefer identity-frst language, people
Ather Sharif, Aedan L. McCall, and Kianna R. Bolante
with neurological disabilities prefer person-frst lan-
guage, deaf and hard of hearing people prefer identity-
frst language, people with chronic illnesses disabilities
prefer person-frst language, mental-health-related dis-
abled prefer identity-frst language. The data table is
presented below.
We used appropriate colors for Color Vision Defciency (CVD)
in our data visualizations. We also checked the contrast ratio to be
at least 3:1, using the WebAIM Contrast Checker tool [72]. Finally,
we tested our platform for accessibility with and without several
screen readers. However, we did not conduct studies with screen-
reader users to test the accessibility of our platform. We plan on
continually improving our web platform’s accessibility and usability
by conducting formative studies with diverse groups of users.
4.4 Implementation Details
In developing our accessible web platform, we generated 119,402
lines of developed code through 19 commits, excluding comments.
We used the React [32] framework to build our platform. Therefore,
JavaScript was naturally our choice of programming language. Ad-
ditionally, we used EcmaScript [30], employing modern JavaScript
features. We implemented our survey using Google Forms, auto-
collecting the responses in a Google Sheets document. We used
Metis [55]—a React plug-in that allows the usage of Google Sheets
as a database—to display live results on our web platform. Currently,
our data is not downloadable; we have started the development
work to support exporting the data as CSV and JSON fles.
5 LANGUAGE USED IN PUBLICATIONS AT
ASSETS AND CHI
In a secondary exploration, we analyzed 11,536 abstracts from pub-
lications at ASSETS and CHI to assess the language usage in publi-
cations at these academic venues. In this section, we present our
methodology, analysis, and results.
5.1 Method
Our goal was to investigate the adoption of identity- and person-
frst language in publications at ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS) and ACM Conference on Hu-
man Factors in Computing Systems (CHI). Therefore, we analyzed
11,536 abstracts from papers at these conferences.
5.1.1 Data Set. Similar to prior work [37, 42], we queried the ACM
Digital Library for all papers published since the year 2000 (inclu-
sive) at ASSETS and CHI, collecting their abstracts. Unlike prior
work [37, 42], our data set was not solely limited to technical papers
and comprised all the publications, including extended abstracts
and poster papers, resulting in a total of 1,564 (ASSETS) + 9,972
(CHI)=11,536 abstracts.
Our goal was to determine the identity- and person-frst language
from the collected abstracts using an automated Natural Language
Processing (NLP) approach. Therefore, we utilized “SpaCy” [19, 68]—
a widely-used NLP library—to generate Part-of-Speech (POS) tags.
(POS tags are the grammatical tags that identify the part of speech
of words in text based on both their defnitions and contexts.) First,
we compiled a list of all nouns appearing in the 11,536 abstracts,
Identity- Versus Person-First Language ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 4: Percentage of total count for IFL terms versus PFL terms per year, published at ACM ASSETS and ACM CHI since the
year 2000 (excluding 2001 and 2003). Total count for terms is shown in parenthesis.

resulting in 466 unique nouns. Then, we manually fltered the list,
retaining only the most commonly appearing nouns that identify
a person (“person-identifers”). Our fnal list included 17 person-
identifers: people, individual, user, student, person, participant, adult,
children, researcher, subject, practitioner, learner, developer, designer,
population, activist, and faculty. In addition to collecting the person-
identifers, based on prior work [37, 42], we composed a list of six
IFL terms and their equivalent PFL terms that people commonly
use to refer to disabled people. The IFL/PFL terms are as follows:
disabled/disabilit(ies), impaired/impairment, wheelchair/wheelchair,
blind/blindness, deaf/deafness, and autistic/autism.
Then, for each sentence in each abstract, we searched for IFL
and PFL terms following the criteria below:
(1) IFL: One of the six IFL terms followed one of the 17 person-
identifers (e.g., disabled user).
(2) PFL: One of the 17 person-identifers, followed by an adpo-
sition (e.g., with) and one of the six PFL terms (e.g., student
with autism).
To increase the efciency of our approach, we removed the deter-
miners (e.g., a or an) from the sentences. We designed our algorithm
to include plurals (e.g., disability and disabilities) in search queries.
Additionally, our algorithm included compound nouns (e.g., peo-
ple with physical impairments) when searching for PFL terms. In
addition to extracting the total count of the terms, we recorded all
the sentences containing those terms. We manually tested every
sentence to check for false positives, making algorithmic adjust-
ments wherever necessary. The supplementary materials comprise
ASSETS ’22, October 23–26, 2022, Athens, Greece
our collected data, including all recorded sentences containing the
IFL and PFL terms for ASSETS and CHI for each publication year.
5.1.2 Analysis. In addition to calculating the total count for IFL
and PFL terms in publications at ASSETS and CHI, we explored the
diference in adoption of these terms between the two venues. We
used Conference (CNF) as our independent variable with the follow-
ing levels: {ASSETS, CHI}. Our dependent variable was Language
Preference (PRF). We calculated PRF as the ratio between the total
count for IFL terms and the total for PFL terms in a given publi-
cation year. To analyze PRF , we used Independent-samples t-test
[53, 64] to determine signifcance. As ASSETS skipped publications
in 2001 and 2003, we excluded the data for CHI from these years in
our analysis.
5.2 Results
We investigated the efects of Conference (CNF) on Language Pref-
erences (PRF) but did not fnd a signifcant main efect (p ≈.497).
For each conference separately and combined, PFL counts were
higher (54.4% combined; 52.6% for ASSETS and 57.1% for CHI) than
IFL counts (45.6% combined; 47.4% for ASSETS and 42.9% for CHI).
We also found this trend persistent for publications in the last fve
years (2017-2021). Figure 4 shows the percentage of total count for
IFL and PFL language used across ASSETS and CHI since 2000.
6 DISCUSSION
To provide insights into the language preferences for and by dis-
abled people, we surveyed 519 disabled people from 23 countries
representing at least one of nine disability categories. Our results
show that 49% of disabled people preferred identity-frst language,
33% favored person-frst language, and 18% did not have a pref-
erence. Additionally, we explored the intra-sectionality and inter-
sectionality of disability categories, gender identities, age groups,
and countries on language preferences of our survey respondents,
fnding that language preferences vary within and across these
factors. We also investigated language usage at ASSETS and CHI
by analyzing 11,536 publication abstracts, fnding that PFL’s usage
was 16.1% more than identity-frst language at these venues.
6.1 Diversity Within Disability
In the discussions involving the usage of identity- or person-frst
language to refer to disabled people, fndings and recommenda-
tions are usually generalized for anyone with a disability, irrespec-
tive of the disability categories they represent. Our survey results
showed that although 48.6% of disabled people preferred identity-
frst language overall, compared to 33.0% who favored person-frst
language, these preferences varied across disability categories. For
example, people with mobility disabilities, neurological disorders,
or chronic illnesses preferred person-frst language to identity-frst
language (as shown in Table 2), highlighting the diversity within the
disabled community. However, interestingly, Disability Category
(DSB) did not have a signifcant main efect on language prefer-
ences for participants from the United States, with only people with
neurological disorders having a higher preference for person-frst
language. Our work did not explore this disparity. We invite schol-
ars and researchers to utilize our publicly-available survey data to
investigate the underlying factors contributing to the diference in
Ather Sharif, Aedan L. McCall, and Kianna R. Bolante
language preferences within disability categories for participants
within and outside the United States.
6.2 Intersectionality Matters
Similarly, we found that language preferences varied within and
across gender identities, age groups, and countries. For example,
women 36 years or older had a dominant preference for person-frst
language (53% vs. 29%; 18% had no preference). In contrast, men
in the same age group had no prominent diference in language
preferences (37% for identity-frst, 35% for person-frst, and 28%
for no preference). Likewise, men and women in the United States
had almost identical preferences between identity- and person-frst
language (38% and 33% for men; 39% and 39% for women, respec-
tively). However, men and women in the United Kingdom had
prominent and opposite preferences for identity- (60% for men and
30% for women) and person-frst language (30% for men and 63%
for women). For non-binary participants, the preference was signif-
icantly higher for identity-frst language in all of our explorations.
These fndings indicate that intersectionality can play a pivotal role
in determining the language preferences of disabled people.
Additionally, our results only partially agreed with those from
Bickford et al.’s [6] analysis, in which they studied the intersection-
ality of gender and age on the language preferences of blind and
low-vision people. Their results showed no diference in the pref-
erences for both gender and age. In contrast, our results showed a
statistically signifcant efect for age but not for gender for visually-
disabled survey respondents. We attribute this contrast to the evo-
lution of preferences over time, as their exploration dates back to
2004, about 18 years ago. This fnding accentuates that language
preferences can drastically evolve, therefore, presenting a necessity
for up-to-date data on the language preferences of disabled people.
To keep our data on language preferences up-to-date, we intend to
redistribute the survey every quarter. We also built the functionality
on our accessible web platform to track the preferences over time
to understand and explore the evolution of language preferences.
6.3 Language Adoption in Academia
Our results show that abstracts of published papers at ACM ASSETS
and ACM CHI, since 2000, have used person-frst language 16.1%
more than identity-frst language. We did not fnd a statistically
signifcant diference in language adoption between the two venues.
Although our results align with fndings from prior work [37, 42],
it is worth noting that our exploration was holistic—we did not in-
vestigate the relationship between the language used and disability
categories, gender identities, age groups, or countries. We also only
used the abstracts for our analysis, similar to prior work [37, 42], as
ACM’s laws prohibit the extraction of full texts from publications.
However, our survey results and the analysis of the language used
in publications at ASSETS and CHI, taken together, indicate that al-
though disabled people prefer identity-frst language, the language
used to refer to them is more commonly person-frst. Future work
can study the nuances in language adoption more comprehensively
in publications at scholarly venues, including conferences other
than ACM ASSETS and ACM CHI, to identify adoption patterns.
Identity- Versus Person-First Language
6.4 Recommendations
Based on our fndings, we ofer three recommendations for authors
and researchers when choosing between identity- and person-frst
language to refer to disabled people:
• First and foremost, we recommend authors and researchers
respectfully ask individual users for their language prefer-
ence (e.g., during pre-study or demographic questionnaires)
whenever and wherever possible. To convey to readers that
the language used refects the individual’s preference, au-
thors and researchers can clearly state their processes at
the beginning of the article or in-line when referring to the
individual (e.g., “P0, who preferred identity-frst language...”).
While such additions can increase the word count of the text,
they contribute toward inclusivity and cultural awareness
around disability-related matters.
• Second, we recommend referring to the intended demo-
graphic group using their self-identifed language prefer-
ences, employing intersectionality using their disability cat-
egory, gender identity, age group, and country. When re-
ferring to disabled people as a group, the overall language
preference (e.g., identity-frst language, at present) may be
the most appropriate. Authors and researchers can use our
web platform to stay up-to-date with the language prefer-
ences of disabled people.
• Finally, we recommend that authors and researchers keep
themselves up-to-date with the latest language preferences
of disabled people, considering that preferences may change
over time. We intend to distribute our survey every quarter
and keep our data publicly available through our live website
to assist authors, researchers, and interested individuals in
staying up-to-date with the latest preferences of disabled
people.
7 LIMITATIONS & FUTURE WORK
We did not collect our participants’ race and ethnicity in our sur-
vey; including these factors in intersectional and intra-sectional
explorations may produce noteworthy results. We designed our
survey using Google Forms, enabling future work to include these
factors in our survey and analyze their efects on the language
preferences of disabled people. Our analysis of the language used
in publications at ASSETS and CHI was holistic. While our results
reafrm the fndings from prior work, future work can employ
more rigorous and nuanced hybrid (human + AI) methodologies to
examine language adoption in academic publications.
Additionally, utilizing our fndings, future work could build sys-
tems (e.g., browser extensions) that automatically update the lan-
guage used on web pages to match the individualized preferences of
users and study its efects on users. Users could specify these pref-
erences using a centralized confguration system. Similarly, future
work could replicate “language checker” plug-ins that proofread
the text for authors and researchers based on our fndings, assist-
ing them by ensuring appropriate language usage and providing
suggestions wherever applicable.
ASSETS ’22, October 23–26, 2022, Athens, Greece
8 CONCLUSION
We surveyed disabled people globally to collect their language pref-
erences between identity- and person-frst language and recorded
the reasons behind their preferences. Additionally, we explored the
intersectionality and intra-sectionality of their preferences with
their disability category, gender identifcation, age group, and coun-
try. Our results show that although disabled people prefer identity-
frst language overall, their preferences vary across disability cat-
egories, gender identities, age groups, and countries of residence.
We made our data publicly available through an interactive and
accessible web platform that enables users to granularly extract in-
formation by fltering language preferences using any combination
of disability category, gender identifcation, age group, and country.
We also investigated language usage in papers at ACM ASSETS
and ACM CHI, fnding a higher usage of person-frst language than
identity-frst language at both conferences.
Our fndings, taken together, indicate that although disabled
people prefer identity-frst language, person-frst language is more
commonly used. We provided recommendations for authors and
researchers in choosing the appropriate language. By releasing our
survey data on language preferences through an accessible web
platform, we hope our work will guide people in using appropriate
language to refer to disabled people and investigate the intersec-
tional diferences in language preferences.
ACKNOWLEDGMENTS
We extend our gratitude to the organizations, groups, and individ-
uals who helped us distribute our survey. In particular, we would
like to thank the National Federation of the Blind, Accelerated Cure
Project, Association of Late Deafened Adults, and Fibromyalgia
Awareness UK for their immense support and assistance. We thank
the anonymous reviewers for their helpful comments and sugges-
tions. Finally, we thank and remember our recently-departed team
member Zoey for her feline support, without which the purrusal of
this work would not have been as efective. May she live forever
and in peace in cat heaven.
REFERENCES
[1] Erin E Andrews. 2019. Disability as diversity: Developing cultural competence.
Oxford University Press, New York, NY, USA.
[2] Erin E Andrews, Anjali J Forber-Pratt, Linda R Mona, Emily M Lund, Carrie R
Pilarski, and Rochelle Balter. 2019. # SaytheWord: A disability culture commentary
on the erasure of “disability”. Rehabilitation psychology 64, 2 (2019), 111.
[3] American Psychological Association et al. 2019. Publication Manual of the Ameri-
can Psychological Association, (2020). American Psychological Association, Wash-
ington, DC, US.
[4] Colin B Becg and Robert Gray. 1984. Calculation of polychotomous logistic
regression parameters using individualized regressions. Biometrika 71, 1 (1984),
11–18.
[5] Alexy Bhowmick and Shyamanta M Hazarika. 2017. An insight into assistive
technology for the visually impaired and blind people: state-of-the-art and future
trends. Journal on Multimodal User Interfaces 11, 2 (2017), 149–172.
[6] James O Bickford. 2004. Preferences of individuals with visual impairments for
the use of person-frst language. RE: view 36, 3 (2004), 120.
[7] Michael Bostock, Vadim Ogievetsky, and Jefrey Heer. 2011. D3 data-driven
documents. IEEE transactions on visualization and computer graphics 17, 12 (2011),
2301–2309.
[8] Monique Botha and David M Frost. 2020. Extending the minority stress model
to understand mental health problems experienced by the autistic population.
Society and mental health 10, 1 (2020), 20–34.
[9] Monique Botha, Jacqueline Hanlon, and Gemma Louise Williams. 2021. Does lan-
guage matter? Identity-frst versus person-frst language use in autism research:
A response to Vivanti. , 9 pages.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[10] Kristen Bottema-Beutel, Steven K Kapp, Jessica Nina Lester, Noah J Sasson,
and Brittany N Hand. 2021. Avoiding ableist language: Suggestions for autism
researchers. Autism in Adulthood 3, 1 (2021), 18–29.
[11] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative research in psychology 3, 2 (2006), 77–101.
[12] Emeline Brulé, Brianna J. Tomlinson, Oussama Metatla, Christophe Joufrais,
and Marcos Serrano. 2020. Review of Quantitative Empirical Evaluations of
Technology for People with Visual Impairments. In Proceedings of the 2020 CHI
Conference on Human Factors in Computing Systems. Association for Computing
Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376749
[13] Taylor Cox. 1994. A comment on the language of diversity. Organization 1, 1
(1994), 51–58.
[14] Google Developers. 2014. Charts. https://developers.google.com/chart/
[15] Thomas P Dirth and Nyla R Branscombe. 2018. The social identity approach to
disability: Bridging disability studies and psychological science. Psychological
bulletin 144, 12 (2018), 1300.
[16] Ellary A Draper. 2018. Navigating the labels: Appropriate terminology for stu-
dents with disabilities. General Music Today 32, 1 (2018), 30–32.
[17] Dana S Dunn and Erin E Andrews. 2015. Person-frst and identity-frst lan-
guage: Developing psychologists’ cultural competence using disability language.
American Psychologist 70, 3 (2015), 255.
[18] Patrick Dwyer. 2022. Stigma, Incommensurability, or Both? Pathology-First,
Person-First, and Identity-First Language and the Challenges of Discourse in
Divided Autism Communities. Journal of developmental and behavioral pediatrics:
JDBP 43, 2 (2022), 111–113.
[19] Explosion. n.d.. spaCy · Industrial-strength Natural Language Processing in
Python. https://spacy.io/. (Accessed on 03/18/2022).
[20] Christopher J. Ferguson. 2016. An efect size primer: A guide for clinicians and
researchers. In Methodological issues and strategies in clinical research, A.E. Kazdin
(Ed.). American Psychological Association, Washington, DC, USA, 301––310.
[21] Philip M Ferguson and Emily Nusbaum. 2012. Disability studies: What is it
and what diference does it make? Research and Practice for Persons with Severe
Disabilities 37, 2 (2012), 70–80.
[22] Charles D Fernald. 1995. When in London.: Diferences in Disability Language
Preferences Among English-Speaking Countries. Mental Retardation 33, 2 (1995),
99.
[23] Phillip Ferrigon and Kevin Tucker. 2019. Person-First Language vs. Identity-First
Language: An examination of the gains and drawbacks of Disability Language
in society. In Journal of Teaching Disability Studies. CUNY Academic Commons,
New York. NY, USA, Online.
[24] Vic Finkelstein. 2007. The ‘social model of disability’and the disability movement.
The Disability Studies Archive UK 1 (2007), 15.
[25] Doris Fleischer and Fleischer Doris Zames. 2012. The disability rights movement:
From charity to confrontation. Temple University Press, Philadelphia, PA.
[26] Patrick Flink. 2021. Person-frst & identity-frst language: Supporting students
with disabilities on campus. Community College Journal of Research and Practice
45, 2 (2021), 79–85.
[27] Morton Ann Gernsbacher. 2017. Editorial perspective: The use of person-frst
language in scholarly writing may accentuate stigma. Journal of Child Psychology
and Psychiatry 58, 7 (2017), 859–861.
[28] Arthur Gilmour, Robert D. Anderson, and Alexander L. Rae. 1985. The analysis
of binomial data by a generalized linear mixed model. Biometrika 72, 3 (1985),
593–599.
[29] Dan Goodley. 2016. Disability studies: An interdisciplinary introduction. Sage, Los
Angeles, CA.
[30] Jordan Harband, Shu-yu Guo, Michael Ficarra, and Kevin Gibbons. 1999. Standard
ecma-262.
[31] Holly Hofman, Marie Hengesbach, and Shana Trotter. 2020. Perspectives on
Person-First Language: A Focus on College Students. Journal of Postsecondary
Education and Disability 33, 1 (2020), 39–48.
[32] Facebook Inc. n.d.. React – A JavaScript library for building user interfaces.
https://reactjs.org/. (Accessed on 08/08/2021).
[33] David Johnstone. 2012. An introduction to disability studies. Routledge, Abingdon,
Oxfordshire, UK.
[34] Lorcan Kenny, Caroline Hattersley, Bonnie Molins, Carole Buckley, Carol Povey,
and Elizabeth Pellicano. 2016. Which terms should be used to describe autism?
Perspectives from the UK autism community. Autism 20, 4 (2016), 442–462.
[35] Emily Ladau. 2014. What should you call me? I get to decide: Why I’ll never
identify with person-frst language. In Criptiques. May Day Publishing, Seattle,
WA, USA, 47–56.
[36] James LeBrecht and Nicole Newnham. n.d.. Crip Camp | A Disability Revolution.
https://cripcamp.com/. (Accessed on 04/05/2022).
[37] Lior Levy, Qisheng Li, Ather Sharif, and Katharina Reinecke. 2021. Respectful
Language as Perceived by People with Disabilities. In The 23rd International
ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA)
(ASSETS ’21). Association for Computing Machinery, New York, NY, USA, Article
83, 4 pages. https://doi.org/10.1145/3441852.3476534
Ather Sharif, Aedan L. McCall, and Kianna R. Bolante
[38] Kate Lister, Tim Coughlan, and Nathaniel Owen. 2020. Disability’or ‘Additional
study needs’? Identifying students’ language preferences in disability-related
communications. European Journal of Special Needs Education 35, 5 (2020), 620–
635.
[39] Rebecca-Eli M Long and Albert Stabler. 2021. “This is NOT okay:” Building
a creative collective against academic ableism. In Journal of Curriculum and
Pedagogy. Taylor & Francis, Oxfordshire, UK, 1–27.
[40] Paul K Longmore. 1985. A note on language and the social identity of disabled
people. American Behavioral Scientist 28, 3 (1985), 419–423.
[41] Alan Lundgard, Crystal Lee, and Arvind Satyanarayan. 2019. Sociotechnical
considerations for accessible visualization design. In 2019 IEEE Visualization
Conference (VIS). IEEE, IEEE, New York, NY, USA, 16–20.
[42] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich,
and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? A
Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019.
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York,
NY, USA, Article 371, 18 pages. https://doi.org/10.1145/3411764.3445412
[43] Brett Ranon Nachman, Ryan A Miller, and Edlyn Vallejo Peña. 2020. “Whose
Liability Is It Anyway?” Cultivating an Inclusive College Climate for Autistic
LGBTQ Students. Journal of Cases in Educational Leadership 23, 2 (2020), 98–111.
[44] National Federation of the Blind. n.d.. Homepage | National Federation of the
Blind. https://nfb.org/. (Accessed on 03/12/2022).
[45] Netfix, Inc. n.d.. Netfix - Watch TV Shows Online, Watch Movies Online.
https://www.netfix.com/. (Accessed on 04/05/2022).
[46] National Federation of the Blind. 1993. Convention Resolutions ’93. https://nfb.
org/sites/nfb.org/fles/images/nfb/publications/convent/resol93.htm. (Accessed
on 04/05/2022).
[47] Michael Quinn Patton. 1990. Qualitative evaluation and research methods. SAGE
Publications, Inc., Thousand Oaks, CA, USA.
[48] Victoria Pearson, Katharine Lister, Elaine McPherson, Anne-Marie Gallen, Gareth
Davies, Chetz Colwell, Kate Bradshaw, N St J Braithwaite, and Trevor Collins.
2019. Embedding and Sustaining Inclusive Practice to Support Disabled Students
in Online and Blended Learning. Journal of Interactive Media in Education 2019,
1 (June 2019), 4.
[49] Danielle Peers, Nancy Spencer-Cavaliere, and Lindsay Eales. 2014. Say what you
mean: Rethinking disability language in Adapted Physical Activity Quarterly.
Adapted Physical Activity Quarterly 31, 3 (2014), 265–282.
[50] Beth Pickard, Grace Thompson, Maren Metell, Efrat Roginsky, and Cochavit
Elefant. 2020. It’s Not What’s Done, But Why It’s Done. Voices: a world forum for
music therapy 20, 3 (2020), 19.
[51] Jesse Rathgeber. 2019. Troubling Disability: Experiences of Disability in, through,
and around Music. Arizona State University, Tempe, AZ.
[52] Lidia Ripamonti. 2016. Disability, diversity, and autism: Philosophical perspectives
on health. The New Bioethics 22, 1 (2016), 56–70.
[53] Amanda Ross and Victor L Willson. 2017. Independent samples T-test. In Basic
and advanced statistical tests. SensePublishers, "Rotterdam", 13–16.
[54] Marrok Sedgwick. 2021. Review of Crip Camp co-directed by James LeBrecht
and Nicole Newnham. Disability Studies Quarterly 41, 1 (2021), Online.
[55] Ather Sharif. n.d.. athersharif/metis: React HOC to use Google Sheets as
a Makeshift Database. https://github.com/athersharif/metis. (Accessed on
03/30/2022).
[56] Ather Sharif, Sanjana Shivani Chintalapati, Jacob O. Wobbrock, and Katharina
Reinecke. 2021. Understanding Screen-Reader Users’ Experiences with Online
Data Visualizations. In The 23rd International ACM SIGACCESS Conference on
Computers and Accessibility (Virtual Event, USA) (ASSETS ’21). Association for
Computing Machinery, New York, NY, USA, Article 14, 16 pages. https://doi.
org/10.1145/3441852.3471202
[57] Ather Sharif and Babak Forouraghi. 2018. evoGraphs — A jQuery plugin to create
web accessible graphs. In 2018 15th IEEE Annual Consumer Communications
Networking Conference (CCNC). IEEE, Las Vegas, NV, USA, 1–4. https://doi.org/
10.1109/CCNC.2018.8319239
[58] Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and
Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible
with an Interactive JavaScript Plug-In. In CHI Conference on Human Factors in
Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing
Machinery, New York, NY, USA, Article 478, 19 pages. https://doi.org/10.1145/
3491102.3517431
[59] Mark Sherry*. 2004. Overlaps and contradictions between queer theory and
disability studies. Disability & Society 19, 7 (2004), 769–783.
[60] Carmit-Noa Shpigelman and Carol J Gill. 2014. How do adults with intellectual
disabilities use Facebook? Disability & Society 29, 10 (2014), 1601–1616.
[61] John Sommers-Flanagan and Rita Sommers-Flanagan. 2012. Clinical interviewing:
2012-2013 update. John Wiley & Sons, Hoboken, NJ, USA.
[62] Katta Spiel, Christopher Frauenberger, Os Keyes, and Geraldine Fitzpatrick. 2019.
Agency of autistic children in technology research—A critical literature review.
ACM Transactions on Computer-Human Interaction (TOCHI) 26, 6 (2019), 1–40.
Identity- Versus Person-First Language ASSETS ’22, October 23–26, 2022, Athens, Greece

[63] Katta Spiel, Oliver L Haimson, and Danielle Lottridge. 2019. How to do better
with gender on surveys: a guide for HCI researchers. Interactions 26, 4 (2019),
62–65.
[64] Student. 1908. The probable error of a mean. Biometrika 6, 1 (1908), 1–25.
[65] The Allen Institute for Artifcial Intelligence. n.d.. Semantic Scholar | About Us.
https://www.semanticscholar.org/. (Accessed on 04/05/2022).
[66] Tanya Titchkosky. 2001. Disability: A rose by any other name?“People-First”
language in canadian society. Canadian Review of Sociology/Revue canadienne de
sociologie 38, 2 (2001), 125–140.
[67] U.S. Department of Justice. n.d.. Americans with Disabilities Act of 1990,AS
AMENDED with ADA Amendments Act of 2008. https://www.ada.gov/pubs/
adastatute08.htm. (Accessed on 04/05/2022).
[68] Yuli Vasiliev. 2020. Natural Language Processing with Python and SpaCy: A
Practical Introduction. No Starch Press, San Francisco, CA.
[69] W. N Venables and B. D Ripley. 2013. Modern Applied Statistics with S-Plus.
Springer, New York, NY.
[70] Giacomo Vivanti. 2020. Ask the editor: What is the most appropriate way
to talk about individuals with a diagnosis of autism? Journal of Autism and
Developmental Disorders 50, 2 (2020), 691–693.
[71] W3C. n.d.. WAI-ARIA Overview | Web Accessibility Initiative (WAI) | W3C.
https://www.w3.org/WAI/standards-guidelines/aria/. (Accessed on 04/11/2021).
[72] WebAIM (Web Accessibility in Mind). n.d.. WebAIM: Contrast Checker. https:
//webaim.org/resources/contrastchecker/. (Accessed on 03/30/2022).
[73] Elizabeth A West, Darlene E Perner, Linda Laz, Nikki L Murdick, and Barbara C
Gartin. 2015. People-First and Competence-Oriented Language. International
Journal of Whole Schooling 11, 2 (2015), 16–28.
[74] Americans with Disabilities Act (ADA). 1990. Americans with Disabilities Act of
1990, Pub. L. No. 101–336, 104 Stat. 328.
[75] Beatrice A Wright. 1960. Physical disability–A psychological approach. Harper &
Row Publishers, New York, NY, USA.
ASSETS ’22, October 23–26, 2022, Athens, Greece Ather Sharif, Aedan L. McCall, and Kianna R. Bolante

A LANGUAGE PREFERENCES BY DISABILITY

Figure 5: Screen capture from our accessible web platform showing a bar chart that displays the language preferences of dis-
abled people between identity-frst language (IFL), person-frst language (PFL), and no preference (NP) by disability categories.

B LANGUAGE PREFERENCES BY AGE GROUP

Figure 6: Screen capture from our accessible web platform showing a bar chart that displays the language preferences of
disabled people between identity-frst language (IFL), person-frst language (PFL), and no preference (NP) by age group.
Identity- Versus Person-First Language ASSETS ’22, October 23–26, 2022, Athens, Greece

C LANGUAGE PREFERENCES BY GENDER IDENTITY

Figure 7: Screen capture from our accessible web platform showing a bar chart that displays the language preferences of
disabled people between identity-frst language (IFL), person-frst language (PFL), and no preference (NP) by gender identity.

D LANGUAGE PREFERENCES BY COUNTRY

Figure 8: Screen capture from our accessible web platform showing a bar chart that displays the language preferences of
disabled people between identity-frst language (IFL), person-frst language (PFL), and no preference (NP) by country.
ASSETS ’22, October 23–26, 2022, Athens, Greece Ather Sharif, Aedan L. McCall, and Kianna R. Bolante

E LANGUAGE PREFERENCES OVER TIME

Figure 9: Screen capture from our accessible web platform showing a line chart that displays the language preferences of
disabled people between identity-frst language (IFL), person-frst language (PFL), and no preference (NP) over time. (As we
started collecting the survey results only a few months back, the data, at present, may not be signifcant to draw any conclusion.
However, we hope that tracking the survey data over time will reveal patterns and results worth exploring.)
 Assistive or Artistic Technologies? Exploring the Connections
between Art, Disability and Wheelchair Use
Giulia Barbareschi
Keio Graduate School of Media Design, Yokohama, Japan
barbareschi@kmd.keio.ac.jp
ABSTRACT
Art has deep connections with both disability and HCI research.
From disabled bodies becoming avatars of novel forms of expression,
to artistic work being created as an act of resistance, art has been a
powerful tool to subvert ableist narratives. Artistic practices have
also helped to inspire, innovate and push the boundaries of HCI,
giving rise to new technologies and interaction possibilities. Our
paper presents the exploration of the experiences and practices of
17 artists who used wheelchairs for mobility. Through the thematic
analysis of interviews, we conceptualize three themes: (1) Personal
journeys through art and disability; (2) Social encounters through
art, (3) Skills and technology in art making. From these themes,
we refect on how art can help HCI researchers to capture the
complexity of the experiences of disability and assistive technology
use and how collaboration with disabled artists could help to rethink
the design of disruptive artistic technologies.
CCS CONCEPTS
• Human Centered Computing; • Accessibility; • Accessibil-
ity theory, concepts and paradigms;
KEYWORDS
Disability, Art, Wheelchairs, Creativity
ACM Reference Format:
Giulia Barbareschi and Masa Inakage. 2022. Assistive or Artistic Technolo-
gies? Exploring the Connections between Art, Disability and Wheelchair
Use. In The 24th International ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM,
New York, NY, USA, 14 pages. https://doi.org/10.1145/3517428.3544799
1 INTRODUCTION
The relationship between art and disability is powerful and complex
[56]. Depending the artist, and the socio-historical context, art has
helped the voice of people with disabilities to be heard or silenced,
disabled bodies to be seen as aesthetically beautiful or ugly, and
Crip narratives to be stereotyped or reclaimed [34, 48, 49, 70, 76].
Art is also deeply interconnected with HCI research [41, 43, 73,
81]. The pictorial by Sturdee et al 2021 [81] shows how many HCI
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544799
Masa Inakage
Keio Graduate School of Media Design, Yokohama, Japan
inakage@kmd.keio.ac.jp
researchers have a “second life” as artists that enables them to cre-
ate deeper connections with stakeholders, and promotes a more
instinctive forms of knowledge. In a similar fashion, Andersen et al
2018 [2] have explored the role of “disruptive improvisations”, ma-
terial and procedural explorations used to question and creatively
problematize mainstream and habitual thinking patterns, to fos-
ter novel insights. More generally, the majority of speculative and
provocative design futuring practices used by researchers in HCI
encompasses methods with strong roots in diferent forms of art
[46].
Despite these overlaps, within the context of assistive technology
and accessibility, researchers have only begun to investigate how
the use of assistive and accessible technologies contributes to the
creative expressions of disabled artists. Similarly, it is unclear how
disability art could disrupt the design of assistive and accessible
technologies leveraging provocative narratives that reshape the
perception of disability. A number of accessibility researchers have
investigated the barriers that creative professionals and artists with
disabilities encounter in their work [1, 4, 21, 69]. Others have lever-
aged novel technologies to create radically diferent interaction
modalities that make artistic creation processes more accessible
to people with diferent capabilities [17, 21, 39]. These lines of en-
quiry are essential to increase the accessibility of the art world,
enabling disabled artists to overcome existing barriers and allowing
for more intuitive and adaptable ways of expressing themselves
regardless of one’s capabilities. However, they fall short of high-
lighting what is unique and disruptive about the existing practices
of disabled artists, what role do their assistive technologies play
in their artistic expressions, and how these learnings could serve
to push the boundaries of current research around assistive and
accessible technologies.
In this paper we propose a diferent approach to re-think ac-
cessibility and assistive technology research in HCI through the
lens of art. Through a series of semi-structured interviews with 17
artists who use wheelchairs for mobility, we examine their prac-
tice across the domains of visual art, music, dance, crafting and
performance focusing on the infuences that their experiences of
disabilities and wheelchair use have on their art. Specifcally, we
framed our research around three research questions:
• How do embodied and social experiences of disability afect
the practices of wheelchair-using artists?
• What role do wheelchairs and other assistive technology
play in their practice of art?
• How do the practice of wheelchair-using artists be leveraged
to foster new paradigms for accessibility research?
Through refexive thematic analysis we conceptualize three dif-
ferent themes:
ASSETS ’22, October 23–26, 2022, Athens, Greece
• Personal journeys through art and disability: art, dis-
ability, and wheelchair use are embodied experiences that
continually interact and afect each other, shaping internal
meaning and external manifestations. The result is a deeper
personal connection with one’s body and a richer artistic
practice.
• Social encounters through art: social aspects of artistic
practices range from enriching professional collaborations
and interactions with supportive communities, to navigating
access barriers, and being faced with ableist discrimination.
Resulting acts of resistance include advocacy, mentoring,
and outreach within and beyond the art world.
• Skills and technology in art making: diferent dynamics
of movement that emerge in connection to disability shape
the process through which artists create and perform. The
use of technology, the development of specifc competencies,
and the integration of wheelchair experiences are weaved
together to shape unique creative strategies.
Based on these fndings we elaborate on how interacting with
disabled artists can help HCI researchers to develop a more nu-
anced understanding of the many intersectional factors shaping
the experience of disability, access, and assistive technology use
[18, 35, 74, 84, 86, 88]. Finally, we invite researchers to move beyond
dichotomies of impairment-based and ability-based technology and
engage with artists towards the creation of artistic technologies that
enhance the unique competencies, interaction modalities and cre-
ative possibilities that emerge as a result of disability and assistive
technology use [13, 14, 42, 67, 79].
In summary, our contributions in this paper include: (1) a frst
exploration of how disabled artists integrate their personal experi-
ences of disability and wheelchair user in various artistic practices
including visual art, dance, music, crafting and performance; (2) a
series of refections on how the work of disabled artists can fos-
ter a deeper understanding of embodied and social experiences of
disability, and assistive technology use; (3) insights on leveraging
work with disabled artists to challenge current assumptions around
the role of accessibility and assistive technology, and re-imagine
technology that enhances the uniqueness of disabled users.
2 RELATED WORK
In the following sections we provide a brief review of research
examining the practices of disabled artists, focusing on both frst-
hand accounts of creative processes and the broader social role of
art in questioning disabled narratives. We also provide a summary
of HCI literature that specifcally looked at the use of assistive
and accessible technology in the context of art. Although these
short reviews are not exhaustive, they serve to provide a contextual
understanding of our study involving wheelchair using artists and
help to frame the refections and implications presented in the
paper.
It is important to highlight that throughout the paper we in-
terchangeably use identity-frst and person-frst language. This is
to recognize the diferent preferences of individuals and disability
communities and that, while certain artists strongly identify as
disabled and see their work as being part of artistic movements
Giulia Barbareschi and Masa Inakage
specifc to disability, others do not, preferring to position their work
away from Disability Culture [9, 25, 82].
2.1 Disability and art practice between social
and creative narrative
Solvang 2018, argues that the intersections between art and disabil-
ity can be examined using diferent frames: art therapy, outsider
art, disability art, and disability aesthetics [78]. These diferent dis-
courses are not mutually exclusive and often coexist, but each of
them carries a number of specifc implications.
When disabled people’s artistic activities are seen through the
lens of therapy, the focus is on understanding the value of the prac-
tice to the individual, or specifc group of disabled people engaged
in the creative process [22, 78]. In a variety of contexts, individ-
uals with disabilities have been shown to beneft from engaging
in projects that seek to leverage art to improve one’s physical and
psychological wellbeing, or to increase self-esteem and promote
community empowerment [40]. However, the art therapy frame-
work is connected to a predominantly medicalized view of disability,
belittling the artistic value of the work produced by disabled people
and favoring ableist discrimination within the mainstream art com-
munity [77, 78]. In contrast, the framing of outsider art seeks to
highlight how the perceived “dysfunctionality of artist” (primarily
in connection to their mental and social status as outsiders) can
produce bolder, more intuitive and powerful art expressions that
are unique to these outcast groups [47, 77]. However, outsider art
has been criticized due to its tendencies to reinforce othering of
disabled artist and promote exploitation from curators and art deal-
ers who have a vested interest in maintain the status of artists as
outsiders [47, 62].
Disability Art is defned by Solvang 2018 as encompassing the
work of disabled artists that is specifcally informed by their expe-
rience of disability, and is often rooted in the social dynamics of
identity, disability culture, and the struggle for disability justice and
equality [55, 77]. The close connections of disability art with the
disability rights and disability justice movements have led to the
creation of powerful works of art that seek to reframe the disability
discourse and question ableist narratives [25, 30, 37, 72, 78]. Yet, the
specifcity of disability art has also had the counter efect of sidelin-
ing the work of disabled artists to socio-political activism, resulting
in challenges to receive adequate recognition from the mainstream
artistic world [77, 82, 87]. Finally, the discourse related to disability
aesthetics can help to reclaim the visibility of disability in main-
stream art, in particular visual and performance arts, through the
depiction of disabled bodies as both beautiful and inspiring [48, 78].
Disability aesthetics does not necessarily carry a social meaning
of inclusion, but leverages the uniqueness of disabled bodies as a
means to re-imagine artistic expressions in bold and innovative
ways [48, 76, 78].
Much of the literature on art and disability is focused on broader
societal discourse. However, more recently, researchers have dis-
played a growing interest towards the experiences of disabled artists
and have attempted to unpick the complex relationship between
their overlapping identities [5, 22, 38, 82, 85]. For example, Bang
and Kim 2015 [5] conducted questionnaires and interviews with 12
Korean disabled artists, including writers, visual artists, musicians,
Assistive or Artistic Technologies? Exploring the Connections between Art, Disability and Wheelchair Use
and performance artists to understand their motivations, creative
processes, and the barriers experienced throughout their careers.
Results show how their journeys to become professionals were
riddled with difculties ranging from personal struggles with pain
or functional impairments, to barriers accessing formal art educa-
tion, and prejudice and discrimination that negatively infuenced
their confdence as individuals and professional artists [5]. Consid-
erations around disability stigma drove certain artists to highlight
social components of their work, situating their artistic endeavors
in the Disability Art movement, whereas it led others to purpose-
fully choose to not disclose their disability status, in order to not
be labelled as a disabled artist [5].
On the other hand, the interviews with 47 young artists con-
ducted by Sulewski et al 2012 show a more nuanced picture [82].
Identities as artists were shaped by a variety of factors from the
presence of role models in the family, the support and encourage-
ment of teachers, or the inspiration that emerged when interacting
with the work of famous artists. Similarly, their disability identity
was shaped by personal experiences facing social and environmen-
tal challenges or navigating their own impairments, as well as their
pride in the broader disability culture [82]. The intersection of these
two identities was often dynamic and complex with art being used
as a way to claim and reinforce their own identities as disabled, or
promote disability pride. Some participants explained how art rep-
resented a therapeutic outlet that allowed them to better elaborate
personal experiences of emotional and physical sufering [82]. In
some cases, young artists felt that these two identities had little
overlap, and simply saw themselves as artists who happened to
have a disability [82].
In their recent review on the social role of disabled artists and
their work, Wolbring & Jamal Al-Deen 2021 attempted to capture
an additional dimension of the art disability relationship by specif-
cally looking at the intersections with technology and science [85].
Their fndings showed that there is anecdotal evidence of artists
leveraging diferent technologies to create and perform, enhancing
their artistic expressions and providing better ways to connect with
the audience [85]. However, the authors highlighted how, despite
the proliferation of artistic work showcasing synergies and ten-
sions between art, disability and technology use, there is still little
academic research examining how the use of technology helps to
shape the work of disabled artist and how in turn, disabled artists
could help to shape the development of disruptive new technologies
[85].
Although studies focusing specifcally on the experiences of
artist wheelchair users are rare, some poignant examples are pro-
vided by both Linton 2021 [50] and García-Santesmases Fernández
& Arenas Conejo 2017 [27]. The former highlights how despite the
widespread existence of numerous physical barriers in the world of
theatre, the greatest obstacles faced by wheeling actors are the mis-
guided negative attitudes of producers, and fellow actors. Regard-
less of barriers, the author presents several examples of extremely
successful performances of wheeling actors who were cast for roles
that were not necessarily written to include a wheelchair, without
the need for any change in the original script [50]. The latter study
describes examples of how the creative performance of a disabled
actor using a wheelchair to immobilise an non-disabled actor has
ASSETS ’22, October 23–26, 2022, Athens, Greece
been leveraged to subvert expectations around mobility, able and
disable bodies [27].
2.2 HCI literature on assistive and accessible
technology use in the art
Within the domain of assistive technology and accessibility re-
search in HCI, there are several examples of studies focusing on
artistic practice. Many of these studies are specifcally focused on
the domain of visual arts and mainly investigate the access barriers
encountered by artists with disabilities and the role of technology
in addressing, or reinforcing them [4, 19, 20, 60]. For example, both
Perera et al 2007 and Creed 2018 engaged with physically impaired
artists to understand how they leveraged digital and non-digital
technologies to produce their work, and what difculties emerged
though these interactions [20, 60]. Findings showed how artists
often used diferent technologies to address limitations connected
to their own impairments, but existing interaction modalities often
forced them to perform cumbersome repetitive tasks and unnatural
movements that could generate fatigue and injuries [20, 60]. Tech-
nology limitations also limited their independence which could
create unnecessary limitations about the time and modality of work
depending on the availability of help [20, 60].
In a similar fashion, the recent study by Payne et al 2020 investi-
gated the practices of visually impaired composers, music produc-
ers, and songwriters to understand the accessibility of currently
available technologies [59]. Results highlighted how, despite their
facilitating role, technologies for composition and production still
featured signifcant access barriers that required participants to use
custom made scripts, and develop DIY solutions and workarounds
to successfully accomplish their goals. These limitations created
artifcial ceilings that hampered creativity, complicated collabora-
tions between musicians with and without sights, and restricted
one’s operational choices [59].
Rather that investigating the barriers faced by disabled artists, a
number of researchers have focused on developing technological
solutions that could make artistic practices more accessible, often
leveraging embodied interaction modalities [3, 4, 17, 19, 32, 39, 45,
66]. Some of these designs were developed to address difculties
linked to specifc tasks, such as the HapticEQ by Karp & Pardo
2017 that allowed visually impaired music producers to create and
modify equalization curves in an intuitive manner [45]. Others,
such as the AirSticks by Ilsar & Kenning 2020, aimed to be more
fexible and adaptable, enabling users with diferent disabilities to
engage in a variety of diferent musical improvisation according to
their own taste and preferred interaction modalities [39].
Most of the accessible technologies for art developed by HCI re-
searchers are targeted towards novices with an interest in art, rather
than professional artists [19, 39, 58, 66]. For example, Ragone 2020
developed OSMoSIS an interactive system that leverages motion
tracking technologies to allow autistic children to create diferent
sounds through body movements [66]. However, technologies such
as the voice-controlled system for creative object positioning de-
veloped by Aziz et al 2021 have been created and tested with the
needs of both novice and expert graphic and interface designers
with physical impairments in mind [4].
ASSETS ’22, October 23–26, 2022, Athens, Greece
Examples of creative interactions between disabled individuals
and technology have also been explored by HCI scholars focusing
on crafting practices. The experience report by Hawthorn & Ash-
brook 2017 [33] illustrates how the creative act of developing one’s
own prosthetic hand turns the body itself into a living prototype
and the artist into a self-making cyborg. Even when the diferences
generated by one’s unique creative interactions might be impossi-
ble to perceive by others, the visually impaired weavers who took
part in the study conducted by Das et al. 2020 [23] explained how
individual making practices leave a personal signature that will
always be recognizable by the artist.
Although there are exceptions to this, much of the existing re-
search around the use and creation of accessible and assistive tech-
nology for artistic expressions is primarily centered around visual
art, making, and music. This is interesting especially if we consider
the strong embodied connections of many aspects of the experi-
ence of disability and assistive technology use, often described
as an extension of the body, which might be more prominent in
dance and performance art [11, 24, 54, 68, 79]. Moreover, existing
studies largely focus on the functional aspects of the interaction
between the artist and the technology, often focusing on the neg-
ative connotations of access barriers, without looking at how the
use of technology by disabled artists might create unique artistic
expression and generate social implications around the perception
of disability more generally [31, 35, 36, 61].
3 METHODS
3.1 Participants
In total, 17 artists who used wheelchairs for mobility were recruited
for the study. The choice of focusing our research on artists that
used wheelchairs for mobility was driven by a combination of fac-
tors. Firstly, we assumed that using a wheelchair for mobility would
have afected several aspects of artistic creation, regardless of the
particular feld of the artist, which we believed could enable indi-
viduals to experiment and explore with diferent creative strategies.
Secondly, wheelchairs users are an extremely diverse group that
encompass individuals who can have extremely diferent capabili-
ties, experience diferent limitations, and have diferent preference.
While this is true for most “sub-groups” of disabled people, we
felt that the additional diversity created by the wide variety of
wheelchair types available across diferent countries could further
contribute to the development of unique creative paradigms. Fi-
nally, the choice was also motivated by the positionality of the
frst author, who has previously worked as a physiotherapist and
has extensive experience conducting HCI research with wheelchair
users. Increased understanding of how wheelchair-using bodies
move enabled us to explore in more depth participants’ artistic
practices.
Artists were recruited through a variety of strategies ranging
from social media posts, direct approach through personal websites,
introduction from the personal and professional network of the
authors, approach through national and international disability
art organizations, and word of mouth. Recruitment criteria were
self-identifcation as artist, age above 18, and use of a wheelchair
in daily life. No restrictions were introduced concerning the feld
of art practiced, or the professional status of the artist. Similarly,
Giulia Barbareschi and Masa Inakage
no limitation on the type of wheelchair used, modality, or length
of use was introduced in the inclusion criteria. Seven participants
identifed as having multiple disabilities (P1, P2, P5, P6, P11, P14,
P15), and nine stated they used multiple assistive technologies,
beyond the wheelchair, in their everyday lives (P1, P2, P5, P6, P8,
P9, P11, P14, P15). A summary of participants’ characteristics is
provided in Table 1
Potential participants were approached by the frst author and
informed about the purpose of the research. If interested in taking
part in the study, the research team provided them with the consent
form and invited them to participate in an interview. According
to the preferences and access needs of participants, the interview
could take place synchronously (though phone or video call using
MS Teams or Zoom), or asynchronously (in a written format). All
participants but one, P14, chose to take part in a synchronous
interview. Another participant, P11 who communicates nonverbally,
chose to participate in the interview alongside his father, who was
also his artistic collaborator.
3.2 Materials & Procedure
Most participants preferred to be interviewed in English, but P11
asked for the interview to be conducted in Italian, and P9 in Japan-
ese. The frst author who conducted the interviews is fuent in
Italian and the second author is fuent in Japanese, which made it
possible to easily conduct the interview in the language requested
by the artists. Ahead of the interview participants were provided
with a consent form for the study in their preferred language and
written or verbal consent was obtained before the start of the in-
terview. The duration of synchronous interviews varied between
40-96 minutes. For the asynchronous interview, P14 was emailed
the questions by the frst author, and provided the answers in writ-
ten format. Clarifcations were requested as appropriate to ensure
that the experience of the artist could be adequately captured. The
interview guide was structured across four diferent areas, with the
goal to capture the complexity of the personal and creative experi-
ences of artists: details of the artistic practice, use of a wheelchair in
everyday life, integration of the wheelchair and other technologies
in the artistic practice, public perceptions of wheelchair use and
art. A full copy of the interview guide is provided as supplemental
material. Participants were also asked to share any audio/visual/or
written material that could help illustrate their personal experiences
as disabled artists and the particular characteristics of their artistic
practice. This material was often part of the personal portfolio of
the artists and, in accordance with participants’ wishes, it will not
be made available in connection to this publication to protect their
privacy.
3.3 Analysis
Synchronous interviews were audio recorded using a portable
recorder and transcribed verbatim by the frst author. The Japanese
and Italian interview were frst translated in their original language
and then translated into English. The corpus of data included inter-
views transcripts (or written response to the interview questions in
the case of P14), supplemented by digital photographs and videos
provided by participants, feld notes taken by the frst author. The
data was analyzed thematically using an inductive approach with
Assistive or Artistic Technologies? Exploring the Connections between Art, Disability and Wheelchair Use ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Summary of Participants

Participant Art feld Country Primary Wheelchair Daily Disability Artist before
wheelchair experience wheelchair Onset wheelchair
(y) use use (Y/N)
1 Digital UK Electric and power 42 Part-time Acquired No
Visual Art assisted wheels
2 Sculpture UK Electric and 21 Full-time Acquired Yes
and attendant propelled
installation manual
3 Baking and Canada Manual self-propelled 14 Full-time Acquired No
Cake
Design
4 Dance US Manual self-propelled 28 Full-time Acquired No
5 Dance and Australia Manual both self and 5 Part-time Acquired Yes
Perfor- attendant propelled
mance
Art
6 Dance and UK Manual self-propelled 3 Full-time Acquired Yes
Music
7 Dance and US Manual self-propelled 7 Full-time Acquired Yes
Music
8 Dance UK Manual self-propelled 9 Part-time Acquired Yes
9 Dance and Japan Manual self-propelled 30 Full-time Congenital No
Circus Art
10 Music Canada Electric 46 Full-time Acquired Yes
11 Painting Italy Electric 30 Full-time Congenital No
12 Music Canada Manual self-propelled 33 Full-time Acquired Yes
13 Painting US Manual self-propelled 35 Full-time Acquired No
14 Combined UK Manual and electric 23 Part-time Acquired Yes
Arts
15 Dance US Manual self-propelled 34 Full-time Acquired No
16 DJ-ing and India Electric and Manual 27 Full-time Congenital No
Music
17 DJ-ing India Manual 5 Full-time Acquired No

the initial coding carried out by the frst author and discussed
between both authors to defne themes [16]. During the concep-
tualization of themes, the researchers consulted with the artists
on multiple occasions, to ensure that the interpretation of the data
(by a non-disabled researcher with no experience as a professional
artist and a second non-disabled researcher with experience as an
interactive artist) captured the meaning of their own experiences.
Once the analysis was completed, the researchers provided a sum-
mary of results to the artists to verify the representativeness of the
themes. Although participants did not receive additional compen-
sation for their feedback, most artists stated that the possibility to
contribute in the analysis had been empowering as it gave them
more agency on the outcomes of the study. Suggested changes were
discussed between the researchers and artists to develop a joint
interpretation of the themes that could refect the complexity of
the experience of diferent artists.
4 RESULTS
As explained in the methods section, the following themes were
conceptualized through an iterative collaborative process between
the researchers and the artists. The goal was to provide a compre-
hensive and shared understanding of the complex relationships
between creative expressions and experience of disability, paying
attention to the specifc role of assistive and accessible technologies.
4.1 Personal journeys through art and
disability
Although the individual journeys that led each participant to the
decisions to pursue an artistic practice and claim their identities as
wheelchair users and/or people with disabilities were deeply rooted
in their own personal experiences, we identifed several common
elements. Identities as artists and disabled people and/or wheelchair
users were generally formed separately. However, these identities
were also deeply interconnected. For many, their combined iden-
tities as wheelchair users and disabled individuals signifcantly
ASSETS ’22, October 23–26, 2022, Athens, Greece
afected the artistic practice in ways that were not necessarily con-
scious or immediately visible. Ultimately, the shared belief was
that disability and wheelchair use led an artist to deeper, albeit
often challenging, life experiences that created a more complex
connection to the body and a richer creative practice.
4.1.1 Becoming an artist. Most participants spoke of their decisions
to become an artist as something that occurred at a very young
age. This decision was sometimes inspired by a member of their
social circle, who was an artist. Other participants shared stories
about how interacting with works of famous artists captured their
imagination to the extent that motivated them to pursue an artistic
practice themselves (“When I was very, very young, I think about
fve years old I watched a ballet performance in [name of city in the
UK]. And I went to see Swan Lake. And apparently, I couldn’t take my
eyes of the stage. So I think that’s kind of kind of where it started.”
– P8 Dancer). Although the initial interest towards art was strong
for all participants, the decision to become a professional artist
often matured over the years. Moreover, the interest towards their
chosen feld of art was often accompanied by multifaceted passion
for other creative endeavors which were pursued for leisure.
Interestingly, both P1 and P5, described their artistic journey as
a personal calling rather than a conscious decision. The strength of
their imagination, or the ways in which their personal neurocog-
nitive patterns were articulated, made art the only logical way to
express themselves. In a way, both participants felt that they had
been born artists rather than having become one.
“I think fundamentally, there’s something about the way
that my brain is wired, the way my neurodiversity exists,
the way I sensorially exist in the world, that meant that
art felt like a really natural form of expression for me,
and natural way of connecting and communicating and
it’s also where I felt the most myself in the most alive”
– P5 Dancer and Performance Artist
Although the majority of participants had an interest towards
art from a relatively young age, others did not. Both P9 and P15 had
never thought about pursuing a career as a dancer until they started
to take classes for leisure. On the other hand, P11 and his father
begun painting as a playful family bonding activity that became
more serious as they gained more recognition and felt that the
nature of their work was becoming more deliberate and focused.
Finally, P14 stated that they did not initially recognize themselves
as artists, but that some of her previous research and documentary
work had been labeled as such by others.
4.1.2 Becoming Disabled. The large majority of participants, see
Table 1, had acquired their disability throughout their life as a
result of an accident, or an illness. Regardless of the individual
circumstances, the onset of an acquired disability was described
as a traumatic event that completely changed one’s life trajectory.
This sudden transition was often associated with shock and grief
as one struggled to understand how to resume their life.
“When I got into this injury, I will not run from it. But for
six to eight months, I was into too much of a depressive
state. I was only 23 years old and just lying on the bed
and trying to understand what exactly had happened.
Giulia Barbareschi and Masa Inakage
What is the cause of this, why I’m not able to move my
legs? I just started my life.” – P17 DJ
While some of the participants had a traumatic accident or a
sudden illness that led them to become wheelchair users, others
started to use a wheelchair as a result of a progressive condition. The
transition from walking, with or without other kinds of mobility
aids, to using a wheelchair was often resisted, largely due to self-
stigma. However, as they begun to use a wheelchair, participants
felt that they had found a tool of freedom, something that could
enable them to claim back not only their mobility, but also their
life aspirations.
“My chair enables me to live. My chair is... So, I fun-
damentally I’m a dancer. That’s my core. That’s my
essence, it’s who I am. So my chair enables me to live
my purpose as a dancer” – P6 Dancer and musician
Interestingly, while most participants identifed themselves both
as wheelchair users and disabled, others did not. P14 for example
described themselves as a disabled person (using the social model
of disability), and only as wheelchair-using artist in the context of
dance performance. In contrast, P12 described himself as someone
who had a spinal cord injury and used a wheelchair but did not
identify as disabled. For some participants claiming their identity as
disabled and wheelchair users had been complicated by internalized
notions of ableism (“I had all these notions about disability at frst,
and it was super scary. I didn’t want to be a part of that world”- P4
Dancer). However, for others it was made difcult by prejudices
within the disabled community and the difculties that emerged as
a result of other intersectional identities.
“I once went to a really big event, where there was sort of
celebrating disability, but I felt really misplaced, because
I’d never chained myself to a bus or anything like that.
But there were like, a hierarchy of people that had. And
if you weren’t one of those people, you hadn’t really
tried hard enough. And I thought, well, that’s wrong.
Because, you know, I came from this background where
opportunities were really difcult” – P1 Digital Artist
In contrast to other participants, P9, P11 and P16 who acquired
their disability at birth spoke of their life journeys through disability
as more linear. This included their transition to using a wheelchair,
which was seen as a natural process, as much as learning to walk
could be for a child without a motor impairment.
4.1.3 Interconnections between art and disability. Although per-
sonal journeys that led participants to claim their identities as artists
and disabled people/wheelchair users were seen as being gener-
ally separated, they were also deeply intertwined. After the onset
of acquired disabilities, the connections with previously existing
artistic practices could be used with an almost therapeutic goal. P4
for example spoke about art being integrated in her occupational
therapy to regain strength after a spinal cord injury. On the other
hand, P7 leveraged her own artistic practice as a way to process
trauma and grief:
“In the beginning of my injury, I was processing a lot of
grief, a lot of trauma. I also was not very mobile. So a
lot of the ways that I passed the time and was able to
Assistive or Artistic Technologies? Exploring the Connections between Art, Disability and Wheelchair Use
sort of just emotionally processing thing was through
art.” – P7 Dancer and Musician
Some of the participants who had an artistic practice before
they acquired their disability recalled having a complete rejection
of art for a certain period of time. This creative hiatus was often
sustained by the fear that their functional impairments would make
it impossible to resume practicing art in the same way they did
before (“I tried getting my drums back going, but I didn’t imagine that
you could play with less than three limbs” – P12 Musician). Resuming
their artistic practice was often challenging, with individuals facing
barriers that were both personal and projected by others. Ultimately,
strong personal motivations, signifcant resilience, and substantial
trial and error enabled artist to fnd ways to return to their practice.
The experience of acquiring or being born with a disability had a
signifcant impact on one’s life, and artists found a range of creative
ways to integrate this experience in the way they danced, made
music, or created visual art. For example, P9, who was born with
spina bifda, explained:
“When the doctor took out a piece of vertebra after a
surgery, he told me that the fat spread after taking out
the vertebra looked like a blooming fower. Inspired by
this, I came up with an action like making fowers bloom
with my hands behind my back. There are two versions
of this work one is to crawl on the ground with only
hands, and the other is to perform on the wheelchair
with exactly the same choreography.” – P9 Dancer and
Circus Artist
The experience of disability and wheelchair use could both af-
fect the modality in which one’s created art, and the meaning
underneath one artistic expression. Themes around disability and
assistive technologies would fnd ways to enter artists’ work in
both subtle and deliberate ways. This could be done in provocative
ways with the aim to challenge stereotypes and other times simply
to promote diverse representation.
“I’m just more aware of the fact that like inclusivity is
important to me in terms of disability. So, you know, if
I’m doing something that has numerous people in it, I’ll
make sure to include a silhouette of somebody in a chair
as well or similar” – P3 Baker and Cake Designer
Regardless of the specifc ways in which artists felt that their
disability and their use of wheelchairs afected their artistic prac-
tice, all of them described this infuence in positive terms. For some,
acquiring a disability led them to commit to their career as artists.
Others spoke of the ways in which mobility impairments that lim-
ited their possibilities drove them to explore in more depth the
movement and techniques which were accessible to them. In con-
trast, others became more instinctive in their practice which led
them to create art that could connect with their audience in more
meaningful ways.
“Does my wheelchair infuence my painting style? Yes,
absolutely. Am i a better painter in the wheelchair, then
if i was walking? Absolutely” – P13 Painter
ASSETS ’22, October 23–26, 2022, Athens, Greece
4.2 Social encounters through art
Participants described how their artistic practices were strongly
infuenced by social aspects. Some participants had active collabora-
tions where they practiced, performed or experimented with other
artists. Social connections occurred both within the disabled and the
art community and beyond. Depending on the context, these con-
nections could have either positive and empowering connotations,
or be riddled with ableist discriminations.
4.2.1 Connections and collaborations. While some artists worked
primarily alone, many collaborated with others on a regular basis.
For example, P10 and P12 both played the drums in multiple bands.
As part of his training as a DJ and electronic music composer, P16
had frequent interactions with a mentor that supported his eforts
in music production. P4, P6, and P15 were regular members of in-
clusive dance troupes, whereas P5, P9, and P14 had performed with
other artists and dancers as part of specifc projects. The deepest
example of collaboration was probably the one between P11 and
his father. Together they described themselves as a symbiotic pair
of artists. P11 is a painter who creates his work by driving his
wheelchair, to which brushes dipped in diferent colors of paint
have been attached. His father has worked in the textile industry
for a number of years and, creating and mixing colors used on dif-
ferent items of clothing. Besides helping his son switching between
diferent brushes during the painting process, attaching them and
removing them to the modifed wheelchair as required, P11’s father
was also the one who applied the resin locking the colors in place
as they worked through the layers of their joint artistic creations.
While P11 is the hand, or the wheels, drawing the lines which make
up the paintings, he often leverages his father professional expertise
in choosing colors:
“When we work on paintings I ask him to choose which
colours he wants to use next, because our taste can be
diferent. But he often just looks at me and points at the
colours. He makes me understand that choosing colours
is my everyday job, so I should be the one to do it” -
Father and artistic collaborator of P11, Painter
Collaborations established by participants could involve artists
with or without disabilities. For example, P6 was member of an
inclusive dance troupe, but also did extensive work with other
non-disabled dancers. P6 explained how establishing artistic part-
nerships between dancers who had a signifcantly diferent prac-
tices and bodies could be challenging and often required signifcant
“translation efort” when devising choreographies. However, suc-
cessful collaborations could lead to transformative discoveries that
enriched everyone’s understanding of movement.
Many collaborations were largely professional, but in several
cases, artistic collaborations could also become meaningful friend-
ships. In particular, long-standing collaborations such as the ones
amongst dancers in a troupe or musicians in a band, could become
an integral part of one’s social support structure. Often, personal
and professional aspects of these relationships could positively
enforce each other, ultimately increasing creative drive.
“So our dance company can feel like a little bit like
a family, so it is one reason I stuck with dancing. It
also opened opportunities to engage in other things like
ASSETS ’22, October 23–26, 2022, Athens, Greece
acting. The more I feel confdent with them the more I
am willing to try things which are out of my comfort
zone” – P15 Dancer
4.2.2 Barriers and prejudice. Although many of the social interac-
tions described by artists had positive connotations, others did not.
All participants reported having faced signifcant barriers through-
out their careers, largely in connection with prejudice and ableism.
Challenges and discrimination were particularly common at the
beginning of one’s artistic journey, when trying to obtain training
and support to develop skills. Being able to get the proverbial “foot
in the door” required signifcant resilience from the artist. Moreover,
even after being able to access training, many participants were
expected to have to prove themselves to show that they deserved
to be given access to opportunities that would be easily granted to
someone who did not have a disability.
“When I started to apply for pastry schools, the frst
school I contacted, was an accessible building, but said
they weren’t willing to accommodate somebody in a
wheelchair. So just like the inconvenience, I guess, of
having me that they weren’t up for. The second place, I
tried to take some classes with said that I’d have to pay
for two spots, because I’d take up double the amount of
room of an able bodied person” – P3 Baker and Cake
Designer
Ableism could manifest in diferent ways. One of the most com-
mon ones was mistrust of one’s ability. For example, P11’s father
recalled being asked in numerous occasions if he “touched-up” his
son’s paintings as they looked “too beautiful to be painted with a
wheelchair”. Patronizing comments and inspiration porn were also
common with artist being complimented for their “courage” rather
than their skills. The continuous questioning of one’s ability could
signifcantly hamper confdence both personally and professionally.
Stereotypical beliefs were seen as a problematic not only for the
individual, but for disabled artists and their outputs more in general,
with disability art often being declassed as a social efort rather
than art in its own right.
“If I ever, like say that I’m a dancer and someone doesn’t
know me, or has never seen anything I’ve done they’re
like, Oh, okay. . .. Like, in their brain, they’re like, how
did that person dance? Or they think I’m just going to
do it for fun. I think that perception is like super limited.”
– P4 Dancer
As they became more established, artists would often face less
ableist prejudice, but many participants stated that it would never
entirely disappear. Although most artists had developed their own
strategies and could sometimes rely on the support of close allies
to challenge other’s misguided ableist beliefs, the need to con-
stantly maintain confdent and confrontational personalities was
described as exhausting, especially when one was already feeling
“knackered because of medication and stuf ” (P1). Some of the artists
disclosed their disability on their personal websites and art portfo-
lios, whereas others chose not to as they did not want to have their
work to be necessarily associated to the theme of disability. The
audience’s unawareness of the artist disability status could allow
for a more open-minded interpretation of the work and, in certain
Giulia Barbareschi and Masa Inakage
circumstances, enabled participants to solicit unfltered comments
from their audience:
“No one suspects the person in the wheelchair to be an
artist. So whenever I went an art gallery that features
my work into their exhibitions, I just go around, go up
to people and just sort of listening to what they say, and
ask them what they think. Because they would never
imagine is mine, I can actually get honest feedback” –
P2 Sculptor and installation artist
4.2.3 Advocacy, mentoring, and supporting the community. As so-
cial barriers and ableist prejudices represented some of the most
signifcant frustrations and difculties encountered by participants
in their personal and artistic lives, acts of resistance aimed at de-
bunking stereotypes and promoting inclusions were seen as ex-
tremely important. For some artists such as P1, P2, P5, P9, and P14
the advocacy intent could sometimes manifest directly in artistic
work inspired by outrage and protest, combined with the drive to
specifcally shock or surprise their audience:
“I don’t always get inspired by disability - ideas just
come - but sometimes my work is driven by outrage.
That’s a powerful energising force when you have
chronic pain and fatigue.” – P14 Combined art artist
Moreover, artists such as P1, P2, P5, P6, P7, P8, P10 and P14 were
regularly engaged in various eforts to increase the accessibility
of the art world to diferent marginalised groups within the dis-
ability community. For example, P1 and P2 had been engaged in a
number of collaborative art projects involving young people with
profound learning disabilities. P8 instead, worked with diferent
professional ballet organizations to promote the inclusion of people
with disabilities. P5, P6, and P7 had studied extensively to develop
ways to make dancing instruction more inclusive towards people
with diferent bodies and capabilities and were engaged in train-
ing others to make their dance classes more accessible. Together
with another wheelchair musicians and through the support of a
number of organizations in Canada, P10 had set up and maintained
an accessible music recording studio which has been used by both
amateur and professional musicians with disabilities for over 30
years.
“When we set up the music studio, I don’t know that
we were setting out to prove something but I think we
wanted to do something about it. Because we were kind
of looking at this difculty with music practice and
thinking that there must be lots of people with disabili-
ties who want to get into music and the arts and, you
know, and they weren’t just able to do it. I mean it was
self-serving, but it was also for a larger community.” –
P10 Musician
Finally, both P4 and P15 engaged in broader community outreach
programmes targeted towards children alongside other members
of their dance troupes. As part of these programmes and events
they would perform, deliver fun inclusive dance classes, and enable
children to ask questions about disability, with the goal of chal-
lenging stereotypical beliefs and promoting empowering images.
These programmes where described as important for disabled and
Assistive or Artistic Technologies? Exploring the Connections between Art, Disability and Wheelchair Use
non-disabled children alike, as they would help to tackle both public
and self-stigma.
4.3 Skills and technology in art making
Both individual impairments and the use of wheelchair for mobility
had a signifcant impact on participants’ movement. Although some
functional limitations could create challenges in one’s artistic prac-
tice, artists experimented with diferent technologies and their own
movement strategies to fnd new and exciting creative possibilities.
4.3.1 A continuous innovative process. Mobility impairments and
other embodied aspects of disability and wheelchair use could be
linked to signifcant challenges in one’s artistic practice. Dancers
pointed out how jumps and leaps could be difcult to perform on
a wheelchair, musicians explain how limited movement or pos-
tural issues could hinder their ability to play an instrument, and
visual artists with reduced trunk and upper limb mobility would
struggle to prepare a canvas or set up an installation. However,
participants explained that these limitations only prevented one to
make art in the “conventional way”. As persons with disabilities who
had become somehow profcient in navigating challenges through
creativity in their everyday life, participants employed the same
innovative and experimental approach in their artistic practices.
Innovating often begun by questioning the established ways
to create art and starting to look beyond the “how” art is conven-
tionally made using a certain approach, towards investigating the
meaning one wishes to transmit and develop a way to achieve that
goal. This was particularly relevant in the context of dance where
choreographies were often created using a particular vocabulary
of movement based on the features and abilities of standing and
walking bodies. Dancers and performance artists such as P4, P5,
P6, P7, P8, P9, P14, and P15 illustrated how they didn’t simply rely
on mimicking the movements of a standing dancers while sitting
in a wheelchair, but looked at the purpose of each movement and
created a completely diferent vocabulary of movement that would
leverage the unique expressive qualities of a wheelchair using body.
“In the Lecoq pedagogy, there’s a lot of animal work and
looking at how that creates certain movement within
the body. So when looking at this we found for example
that the kind of shapes that are made by the arm when
using a wheelchair, mirrors the shapes that are made by
a bird’s wingtip when it’s fying. So using a wheelchair
is actually quite analogous to the way birds fy in one
sense” – P5 Dancer and performance artist
These experimentations around movement and its meaning were
further enriched by exploring combined dynamics of movements
involving both multiple wheelchair users who had diferent capabil-
ities, and other disabled dancers with non-normative bodies [29], to
disrupt the existing expressive barriers and codifed ways of being
creative.
“In [name of the dance company], we often partnered
with each other. So a lot of lifting each other a lot of
supporting each other leaning. So when you’re doing
that with someone with an entirely diferent body than
yours, what are the really creative things you can do?
So it opened up a lot of like, creative opportunities that
ASSETS ’22, October 23–26, 2022, Athens, Greece
I guess I didn’t consider before I was disabled.” – P7
Dancer and Musician
Creative movement experimentation was not solely restricted to
dance. For example, P2’s body produced involuntary movements
through painful spasms that afected both her arms and legs. In the
past, she had leveraged these movements to create art by letting the
spasms draw the initial lines and progressively adding smoother
lines as the spasms slowly subsided.
“I’ve literally just held my pencil on the on a piece of
paper and just let my spasms draw. And it felt quite
upsetting. So it’s like. . . ‘what is it doing? I can’t control
this’ but then it was like. . . I looked at it and like it so I
continued to work on it” – P2 Sculptor and installation
artist
4.3.2 Combining personal and technological mediations. Although
in certain circumstances participants explained that they were di-
rectly able to take advantage of how their disabled bodied moved to
create unique artistic expressions, in other cases this was achieved
through the development of personalised skills and techniques or
through the use of technology.
To make up for his inability to play the kick drum using his foot,
in 1990 P12 developed an electric pedal with a solenoid that could
be activated using a mouth switch. The solution proved widely
inappropriate due to a signifcant activation delay and the inconve-
nience of featuring exposed copper wire that caused him to “mildly
electrocute” his face during numerous tests. As technology proved
to be so unsuccessful, he opted to focus on modifying his technique
and adapting the set-up of his drum kit to accommodate his new
style, which enabled him to be able to play the drums without hav-
ing to rely on additional equipment that could interfere with his
experience as a musician.
“The biggest diference is that instead of playing with
four limbs, I just play with two hands. So instead of
having your left hand, let’s say playing the snare drum
and your foot playing the kick drum, I played both
with my hands. I just had to change my set up” – P12
Musician
In contrast, P1 found that his biggest creative advantage was
fully realised as he “embraced the pixel” and started exploring the
boundaries of what he could create by integrating diferent forms
of technologies in his artwork.
“I was trained to be a painter, but once I left that be-
hind and decided to really look at art and technology,
I became able to do anything. I could make a piece of
artwork that’s as big as a block of fats, or I could mass
produce little toys that are kind of crazy little versions
of diferent wheelchairs” – P1 Digital artist
As a DJ, P16 made extensive use of technology. However, through
the experience gained navigating inaccessible set-ups where he had
to deal with consoles placed at an excessive height or DJ-ing stations
that did not enable him to have a view of the dancing crowd, he
developed better memory and listening skills. The frst ones would
be essential for him to be able to remember the location of various
controls in the interface that he might be unable to see. The latter,
ASSETS ’22, October 23–26, 2022, Athens, Greece
described as a key skill for any DJ, would enable him to improvise
and more quickly adjust to the reactions of the crowd:
“As a DJ you have to maintain the crowd. Keep their
energy up, you don’t let the crowd go out. By listening I
can know what we beat or the BPM of the song going
on is, but also paying attention and not letting of the
crowd. All these things are really important and now I
can keep them in mind when I and doing my set” – P16
DJ
4.3.3 Integrating wheelchairs from life to art. Through their experi-
ences in everyday life, participants developed intimate bonds with
their own wheelchairs. The wheelchair was described by many as
an extension of their body, but also as a trusted partner that enabled
participants to achieve their goals in both art and life. This embod-
ied connection was largely instinctive. As a result, participants did
not actually think about their wheelchair during their daily life,
until external barriers forced them to do so.
“The wheelchair is like my legs. I can control it without
thinking about it. But if there is a staircase in front of
me, the wheelchair suddenly becomes a trouble because
I cannot use it for climbing the stairs. I can move freely
at normal times, only when a barrier is in front of me I
suddenly realize the existence of my wheelchair.” – P9
Dancer and circus artist
Within the context of art, the connection between the person
and the wheelchair became both more playful and more deliberate.
Artistic practices allowed individuals to test and expand on the
existing boundaries of how they normally used their wheelchair.
At the same time, using their wheelchairs within their art practice
enabled them to fully leverage its distinctive features making their
performance unique and exciting. However, current wheelchair
design also limited these possibilities and many artists felt that
more should be done to allow them to truly explore the boundaries
of their creativity.
“I love to explore how diferent wheelchair confgura-
tions afect my dancing, it’s almost like having a new
body to dance with. But I need to get out of my chair,
get tools and. . . It takes time. I can’t do that in a show
in between routines. I want to be able to make those
types of modifcations very, very simply and quickly” –
P15 Dancer
Although participants felt that the way in which their wheelchair
afected their movement and body image was visible, the ways in
which their movements and their skills changed the movement of
the chair was often overlooked. Similarly, artists explained how
the fact that their own bodies and the wheelchair existed both in
connection with each other, and as separate entities was rarely
understood by most people without lived experience. Dancers such
as P4, P6, P8 and P9 tried to make this relationship visible by inte-
grating choreographies that saw them dancing with the wheelchair,
away from it, and using the wheelchair as a dance prop on which
movement was enacted. Finally, P1, P5, P14 and P17 suggested that
creating ways in which the wheelchair itself could become a work
of art, or an object with a specifc sense of presence on the stage,
Giulia Barbareschi and Masa Inakage
could help to challenge people’s idea of what a wheelchair could
be.
“But if you could make a wheelchair that in the middle
of a set starts to project light or something synced to the
music which you are playing, it’s going to be a diferent
kind of feeling for the person who is playing and for the
for the people who are seeing. Because nobody thinks it
is possible” – P17 DJ
5 DISCUSSION
In this section we discuss how the learnings from this study high-
light tensions and synergies between disability art and HCI research
around accessible and assistive technologies. Throughout these re-
fections we look at diferent ways in which collaborating with
disabled artists could help HCI researchers to overcome the limi-
tations that lead to oversimplifed interpretations of disability and
assistive technology use. Moreover, we explore some of the possi-
bilities that could arise if the design of assistive technologies was
re-imagined through the lens of art.
5.1 Embracing complexity and challenging
oversimplifcation
Over the last decade, thanks to the work of disabled scholars and
allies within the community, the feld of accessibility and assistive
technology research has begun to change [10, 35, 53, 67, 80]. From
greater integration of disability studies, to increased recognition
to the hidden work of people with disabilities, and acknowledge-
ment of the embodied and biographical relationships between dis-
abled users and their assistive technologies, we have developed a
more complex understanding of the role of assistive technologies
within the experience of disability [6, 8, 10, 12, 35, 75]. However,
accessibility and assistive technology research is still prone to use
rigid dichotomies such as disability vs ability, personal vs social
dimensions, access vs exclusion, and use vs nonuse of assistive
technologies [7, 35, 64, 74, 75].
The experiences of disabled artists have shown unique features
that bypass these dichotomies and allow us to explore complex nar-
ratives in more powerful ways observing how interactions between
multiple identities, embodied aspects, social encounters, and as-
sistive technology explorations are reciprocally connected to each
other. Journeys that led participants to become artists could be
sparked by external events, embedded into one’s way of experi-
encing the world, or brought to light by the recognition of others.
In a similar fashion, one’s experience of disability could be rooted
in the occurrence of a particular traumatic event, be part of one’s
identity since birth, or situated in a social context afected by the
perceptions of wheelchair using bodies.
When looking at the interconnections between these two ex-
periences, art positively infuenced the experience of disability by
providing a meaningful way to process physical pain and emo-
tional grief, or express outrage and frustration. However, art was
never described as being solely a form of therapy, or a tool for
individual refection, but maintained its focus on technical quality
and expressive power [40]. Similarly, disability could deepen one’s
artistic practice by providing a more complex and multifaceted
lived experience, directly inspiring an artistic performance or a
Assistive or Artistic Technologies? Exploring the Connections between Art, Disability and Wheelchair Use
piece of artwork, or promoting a more inclusive and meaningful
artistic practice. Yet, the work of disabled artists often transcended
the label of “Disability art” in the same way as one’s identity as a
disabled person constituted an important part of their self-identity,
but it was not the only defning characteristics of an individual
[82]. These tensions show how multiple strong identities, don’t just
coexist, but continually infuence each other enriching experiences
and creating dynamics that can converge and diverge at the same
time.
Embodied and internalized experiences as artists, disabled in-
dividuals and wheelchair users were also shaped by the social en-
counters of participants which could incorporate both positive and
negative connotations. Even when challenging, empowering collab-
orations could see the artists either receiving or providing support
to others, co-creating artistic expressions, or promoting more in-
clusive ways of working. At the same time, interacting with others
within and beyond the artistic and disabled communities meant
having to consistently face ableist and discriminatory attitudes.
Both the power of collaboration and allyship, and the negative
impact of ableism and prejudice have been explored in the context
of disability and assistive technology use [10, 28, 35, 51]. However,
we argue that art allows to explore these dynamics in a more pro-
found way. First of all, the work of disabled artists can become a
politically situated act of resistance towards predominant ableist
narrative incorporating both the individual experience of the artist
and broader social considerations [18, 85]. While disabled artists
did not claim that they could act as representative for the whole
disabled community, they felt that art gave them unique tools to
challenge existing narratives, inviting their audiences to refect
on diferent representations of concepts of disability and assistive
technologies.
Secondly, art allowed for the opportunity to move beyond the
limitations of an inaccessible environment that often equates dis-
ability with disadvantage [65]. Within their practices, many artists
did not focus on what they could not do as a result of disability
or assistive technology use. Nor they simply looked at what they
could do despite their disability or assistive technologies. Instead,
most artists emphasized the uniqueness of their creative expres-
sions by looking at what they could do because of their disability
and assistive technology use, which was often inaccessible to them
before they acquired a disability or begun to use their assistive
technologies. Removing these artifcially created barriers means
creating better opportunities to challenge ability-based hierarchies
and highlight the work of disabled individuals both in connection
with and beyond the use of assistive and accessible technologies
[10, 12, 15].
When considering these refections is important to notice that
the current research has been carried out specifcally with artists
who used wheelchairs for mobility. Several artists stated that they
experienced multiple limitations and that they used a variety of
assistive and accessible technologies, within art practice and ev-
eryday life, there was a certain degree of similarities between their
experiences as a result of our methodological choice. Although we
argue that many of the implications presented are generalizable,
further exploration with artists who use diferent types of assistive
technologies should be carried out to embrace the complexity of
experiences even further.
ASSETS ’22, October 23–26, 2022, Athens, Greece
5.2 The value of exploring artistic technologies
Engaging with disabled artists enabled us to investigate how
wheelchairs could be used in unique and unconstrained ways as
part of artistic expressions and observe how artistic expressions
could become bolder and more creative as a result of the integra-
tion of wheelchairs. The embodied experience of wheelchair use
in everyday life was largely instinctive, with participants natu-
rally integrating the image of the wheelchair in their own bodies
without necessarily having to refect on it [8, 11]. On the other
hand, the use of wheelchairs as part of one’s artistic work involved
thoughtful exploration and a much deeper understanding of the
unique features associated with the combined movement of body
and wheelchair. For example, the translational work of wheelchair
dancers on challenging the notion of the existence of ideal ways to
dance, required signifcant work on deconstructing the expressive
qualities of movement. Moreover, P5’s work on developing a new
vocabulary of movement for non-standing dancers was grounded in
the reciprocal analysis of how the movement of the dancer afected
the movement of the wheelchair and vice versa, promoting an idea
of bi-directional partnership rather than top-down control of user
over device.
Interestingly the connections between the artist and their
wheelchair seemed to go beyond direct interactions and continued
to exist even when the artists and the wheelchair were not physi-
cally in contact with each other. P1, highlighted how for example it
was often possible to identify the wheelchair of a particular person,
even if the user was not actually sitting on it. Previous work by
Profta et al 2016 and 2018 analyzed how cochlear implant users
engage in purposeful eforts to personalize their devices in order
to refect their preferences and match their personality [63, 64].
Furthermore, the work on the biographical prototypes by Bennet
e al 2019 showed how meaningful connections between disabled
users and the devices that they use are built through sustained
interactions that take place over long periods of time [12].
Exploring these dynamics through art can ofer some unique
opportunities to reimagine how the connections between a disabled
user and their assistive technology is reciprocal and features both
intentional and non-deliberate aspects. While substantial work in
recent years had investigated how the use of assistive technolo-
gies afected the way in which people with disabilities perceive
themselves and are perceived by others, the implications surround-
ing how the actions of users afect the perception of assistive and
accessible technologies is much less understood [7, 26, 75].
When examining the use of wheelchairs of other assistive and
accessible technologies in everyday life, as HCI researchers we
often focus on the identifcation of frictions and access barriers
that can limit one’s ability to use their devices to achieve specifc
goals [52]. However, examining the practices of disabled artist
and the ways in which they manage to integrate assistive and
accessible technologies into their art can shift our point of view from
closing gaps and solving problems to leverage strengths and take
advantage of unique features. Painting with a wheelchair created
more instinctive lines than the one’s drawn by hand, getting used
to DJ-ing without being able to see the crowd made one more
responsive, and dancing with a wheelchair created opportunities
for more complex choreographies.
ASSETS ’22, October 23–26, 2022, Athens, Greece
The studies by Braggs 2018 and 2021 and the competency frame-
work by Reyes Cruz et al 2020, have highlighted how as a result of
their disability and assistive technology use, people with disabili-
ties have developed unique sets of skills that should be leveraged
to build on one’s unique strengths [13, 14, 67]. In the examples
shared by our participants we could identify ways to further ex-
pand this by considering what is uniquely possible through the
use of a wheelchair or other assistive devices and how these fea-
tures could be augmented and expanded to facilitate novel creative
expressions in art and everyday life.
Wheeling dancers experimenting with how moving the centre of
mass and the position of the wheels afects their style, represent a
poignant example of self-driven body reconfguration. This ability
is primarily available to disabled-cyborg bodies, but currently held
hostage by technology designers [84]. What would happen if rather
than enabling wheeling dancers to merely change the position of
the axle, we would instead create amorphous mobility technologies
that can be completely adapted to one’s body and preferences?
Explorations rooted in disabled people skills, art and technology
can also extend beyond wheelchairs. For example, studies have
shown that Deaf and Hard of Hearing (DHH) individuals displayed
increased ability to interpret haptic and vibro-tactile stimuli [83].
Although this ability has been leveraged to produce music chairs
and vests that allow DHH individuals to feel music [17, 44, 71], how
could it be harnessed to enable DHH artists to create revolutionary
musical experiences for all? Could a haptic disco become an activity
that leads us to redefne what music is, and stimulate new debates
about the accessibility of tactile versus auditory communication?
Would this lead to the development of “feeling aids” that would
replace conventional hearing aids? Another example can be seen
when looking at the practice of the blind artist Victor Tan Wee
Tar who waves wire to create sculptures, similarly to how blind
weavers interviewed by Das et al 2020 created unique patterns on
cloth [23]. The wire sculptures by Tan Wee Tar are somehow akin
to the wireframe 3D prints by Mueller et al 2014 [57], but could
weaving be used as an input to digital fabrication as well? Could
wove patterns based in braille cells be read as speech or music
opening the door to new creative endeavors? Ultimately, exploring
the deliberate and daring practices of disabled artists opens the
door to reimagining the way we leverage technology in the context
of art and accessibility.
6 CONCLUSIONS
Disability shares a powerful connection with the world of art. Simi-
larly, the interactions of art with HCI have given rise to new meth-
ods, questioned assumptions and fostered novel insights. How-
ever, within the context of accessibility and assistive technology
research, it has rarely been explored how the work of disabled
artists help to overcome limitations of existing research approaches
and spur disruptive thinking about ways to explore the unique
creative possibilities that emerge in connection to disability and
assistive technology use. In this paper we present the results from
collaboratively analyzed interviews that examined the practices
of seventeen artists who used wheelchair for mobility to explore
the reciprocal infuences between art, disability and assistive tech-
nology use. The insights shared by participants showed how their
Giulia Barbareschi and Masa Inakage
diferent identities as disabled individuals, artists, and wheelchair
users were somehow separate, yet deeply interconnected. Moreover,
their experiences as disabled people and artists had been shaped
by embodiment as much as by social interactions. Finally, both
their creative and everyday lives were afected by a complex bal-
ance between their own capabilities, alongside the limitations and
advantages of the technologies they used, both assistive and non.
ACKNOWLEDGMENTS
We wish to sincerely thanks all the artists who have shared their
knowledge, wisdom and creative journeys with us. This research
would have never seen the light without their support. We are also
grateful for the funding from the Japan Society for Promotion of
Science which supports the frst author (JSPS International Research
Fellow PE 207228).
REFERENCES
[1] Fabiha Ahmed, Dennis Kuzminer, Michael Zachor, Lisa Ye, Rachel Josepho,
William Christopher Payne, and Amy Hurst. 2021. Sound Cells: Rendering Visual
and Braille Music in the Browser. In The 23rd International ACM SIGACCESS
Conference on Computers and Accessibility (ASSETS ’21), Association for Comput-
ing Machinery, New York, NY, USA, 1–4. DOI:https://doi.org/10.1145/3441852.
3476555
[2] Kristina Andersen, Laura Devendorf, James Pierce, Ron Wakkary, and Daniela
K. Rosner. 2018. Disruptive improvisations: 2018 CHI Conference on Human
Factors in Computing Systems, CHI 2018. CHI 2018 - Extended Abstracts of the
2018 CHI Conference on Human Factors in Computing Systems (April 2018), 1–8.
DOI:https://doi.org/10.1145/3170427.3170630
[3] Ryoichi Ando, Isao Uebayashi, Hayato Sato, Hayato Ohbayashi, Shota Katagiri,
Shuhei Hayakawa, and Kouta Minamizawa. 2021. Research on the transcendence
of bodily diferences, using sport and human augmentation medium. In Aug-
mented Humans Conference 2021 (AHs’21), Association for Computing Machinery,
New York, NY, USA, 31–39. DOI:https://doi.org/10.1145/3458709.3458981
[4] Farkhandah Aziz, Chris Creed, Maite Frutos-Pascual, and Ian Williams. 2021.
Inclusive Voice Interaction Techniques for Creative Object Positioning. In Pro-
ceedings of the 2021 International Conference on Multimodal Interaction (ICMI
’21), Association for Computing Machinery, New York, NY, USA, 461–469.
DOI:https://doi.org/10.1145/3462244.3479937
[5] Gui Hee Bang and Kyung Mee Kim. 2015. Korean disabled artists’ experiences
of creativity and the environmental barriers they face. Disability & Society 30, 4
(April 2015), 543–555. DOI:https://doi.org/10.1080/09687599.2015.1030065
[6] Giulia Barbareschi, Catherine Holloway, Katherine Arnold, Grace Magomere,
Wyclife Ambeyi Wetende, Gabriel Ngare, and Joyce Olenja. 2020. The Social
Network: How People with Visual Impairment use Mobile Phones in Kibera,
Kenya. In Proceedings of the 2020 CHI Conference on Human Factors in Computing
Systems (CHI ’20), Association for Computing Machinery, Honolulu, HI, USA,
1–15. DOI:https://doi.org/10.1145/3313831.3376658
[7] Giulia Barbareschi, Norah Shitawa Kopi, Ben Oldfrey, and Catherine Holloway.
2021. What diference does tech make? Conceptualizations of Disability and
Assistive Technology among Kenyan Youth: Conceptualizations of Disability
and AT. In The 23rd International ACM SIGACCESS Conference on Computers and
Accessibility (ASSETS ’21), Association for Computing Machinery, New York, NY,
USA, 1–13. DOI:https://doi.org/10.1145/3441852.3471226
[8] Giulia Barbareschi, Sibylle Daymond, Jake Honeywill, Dominic Noble, Nancy
N Mbugua, Ian Harris, Victoria Austin, and Catherine Holloway. 2020. Value
beyond function: analyzing the perception of wheelchair innovations in Kenya.
In proceedings of the 22nd International ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS ’20), Association for Computing Machinery, New York,
NY, USA.
[9] Colin Barnes. 2003. Efecting change: Disability, culture and art. 31.
[10] Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence
as a Frame for Assistive Technology Research and Design. In Proceedings of the
20th International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’18), Association for Computing Machinery, Galway, Ireland, 161–173.
DOI:https://doi.org/10.1145/3234695.3236348
[11] Cynthia L. Bennett, Keting Cen, Katherine M. Steele, and Daniela K. Rosner.
2016. An Intimate Laboratory?: Prostheses As a Tool for Experimenting with
Identity and Normalcy. In Proceedings of the 2016 CHI Conference on Human
Factors in Computing Systems (CHI ’16), ACM, New York, NY, USA, 1745–1756.
DOI:https://doi.org/10.1145/2858036.2858564
Assistive or Artistic Technologies? Exploring the Connections between Art, Disability and Wheelchair Use
[12] Cynthia L. Bennett, Burren Peil, and Daniela K. Rosner. 2019. Biographical Pro-
totypes: Reimagining Recognition and Disability in Design. In Proceedings of
the 2019 on Designing Interactive Systems Conference (DIS ’19), Association for
Computing Machinery, San Diego, CA, USA, 35–47. DOI:https://doi.org/10.1145/
3322276.3322376
[13] Danielle Bragg, Cynthia Bennett, Katharina Reinecke, and Richard Ladner. 2018.
A Large Inclusive Study of Human Listening Rates. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems (CHI ’18), Association for
Computing Machinery, New York, NY, USA, 1–12. DOI:https://doi.org/10.1145/
3173574.3174018
[14] Danielle Bragg, Katharina Reinecke, and Richard E. Ladner. 2021. Expanding a
Large Inclusive Study of Human Listening Rates. ACM Trans. Access. Comput. 14,
3 (July 2021), 12:1-12:26. DOI:https://doi.org/10.1145/3461700
[15] Stacy M. Branham and Shaun K. Kane. 2015. Collaborative Accessibility: How
Blind and Sighted Companions Co-Create Accessible Home Spaces. In Proceedings
of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI
’15), Association for Computing Machinery, Seoul, Republic of Korea, 2373–2382.
DOI:https://doi.org/10.1145/2702123.2702511
[16] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative Research in Psychology 3, 2 (January 2006), 77–101. DOI:https://doi.
org/10.1191/1478088706qp063oa
[17] Doga Cavdir and Ge Wang. 2020. Felt sound: A shared musical experience for
the deaf and hard of hearing.
[18] Eliza Chandler, Nadine Changfoot, Carla Rice, Andrea LaMarre, and Roxanne
Mykitiuk. 2018. Cultivating disability arts in Ontario. Review of Education, Peda-
gogy, and Cultural Studies 40, 3 (May 2018), 249–264. DOI:https://doi.org/10.1080/
10714413.2018.1472482
[19] Chris Creed. 2016. Assistive tools for disability arts: collaborative experiences in
working with disabled artists and stakeholders. Journal of Assistive Technologies
10, 2 (January 2016), 121–129. DOI:https://doi.org/10.1108/JAT-12-2015-0034
[20] Chris Creed. 2018. Assistive technology for disabled visual artists: exploring
the impact of digital technologies on artistic practice. Disability & Society 33, 7
(August 2018), 1103–1119. DOI:https://doi.org/10.1080/09687599.2018.1469400
[21] Chris Creed, Maite Frutos-Pascual, and Ian Williams. 2020. Multimodal Gaze
Interaction for Creative Design. In Proceedings of the 2020 CHI Conference on
Human Factors in Computing Systems. Association for Computing Machinery,
New York, NY, USA, 1–13. Retrieved April 12, 2022 from https://doi.org/10.1145/
3313831.3376196
[22] Simon Darcy, Hazel Maxwell, Simone Grabowski, and Jenny Onyx. 2022. Artistic
Impact: From Casual and Serious Leisure to Professional Career Development in
Disability Arts. Leisure Sciences 44, 4 (May 2022), 514–533. DOI:https://doi.org/10.
1080/01490400.2019.1613461
[23] Maitraye Das, Katya Borgos-Rodriguez, and Anne Marie Piper. 2020. Weaving
by Touch: A Case Analysis of Accessible Making. In Proceedings of the 2020 CHI
Conference on Human Factors in Computing Systems. Association for Computing
Machinery, New York, NY, USA, 1–15. Retrieved July 5, 2022 from https://doi.
org/10.1145/3313831.3376477
[24] Hazel Dixon. 2021. Immersive performance and inclusion through a lens of
the social model of disability. interactions 28, 3 (April 2021), 70–72. DOI:https:
//doi.org/10.1145/3460111
[25] Jennifer Eisenhauer. 2007. Just Looking and Staring Back: Challenging Ableism
through Disability Performance Art. Studies in Art Education 49, 1 (October 2007),
7–22. DOI:https://doi.org/10.1080/00393541.2007.11518721
[26] Heather A. Faucett, Kate E. Ringland, Amanda L. L. Cullen, and Gillian R. Hayes.
2017. (In)Visibility in Disability and Assistive Technology. ACM Trans. Access.
Comput. 10, 4 (October 2017), 14:1-14:17. DOI:https://doi.org/10.1145/3132040
[27] Andrea García-Santesmases Fernández and Miriam Arenas Conejo. 2017. Playing
crip: the politics of disabled artists’ performances in Spain. Research in Drama
Education: The Journal of Applied Theatre and Performance 22, 3 (July 2017),
345–351. DOI:https://doi.org/10.1080/13569783.2017.1327804
[28] Giulia Barbareschi, Mark T. Carew, Norah Kopi, and Catherine Holloway. 2021.
When they see a wheelchair, they’ve not even seen me”- Factors shaping the
experience of disability stigma and discrimination in Kenya. International Journal
of Environmental Research and Public Health (in press 2021).
[29] Dan Goodley and Katherine Runswick-Cole. 2013. The body as disability and
possability: theorizing the ‘leaking, lacking and excessive’ bodies of disabled
children. Scandinavian Journal of Disability Research 15, 1 (March 2013), 1–19.
DOI:https://doi.org/10.1080/15017419.2011.640410
[30] Aimi Hamraie. 2022. (Ir)resistible Stairs: Public Health, Desiring Practices, and
Material-Symbolic Ableism. Journal of Architectural Education 76, 1 (January
2022), 49–59. DOI:https://doi.org/10.1080/10464883.2022.2017691
[31] Aimi Hamraie and Kelly Fritsch. 2019. Crip technoscience manifesto. Catalyst:
Feminism, Theory, Technoscience 5, 1 (2019), 1–33.
[32] Kaito Hatakeyama, MHD Yamen Saraiji, and Kouta Minamizawa. 2019. MusiArm:
Extending Prosthesis to Musical Expression. In Proceedings of the 10th Augmented
Human International Conference 2019 (AH2019), Association for Computing Ma-
chinery, New York, NY, USA, 1–8. DOI:https://doi.org/10.1145/3311823.3311873
[33] Peregrine Hawthorn and Daniel Ashbrook. 2017. Cyborg Pride: Self-Design in
e-NABLE. In Proceedings of the 19th International ACM SIGACCESS Conference on
ASSETS ’22, October 23–26, 2022, Athens, Greece
Computers and Accessibility (ASSETS ’17), Association for Computing Machinery,
New York, NY, USA, 422–426. DOI:https://doi.org/10.1145/3132525.3134780
[34] Anna Catherine Hickey-Moody. 2009. Unimaginable bodies: Intellectual disability,
performance and becomings. In Unimaginable Bodies. Brill.
[35] Megan Hofmann, Devva Kasnitz, Jennifer Mankof, and Cynthia L Bennett. 2020.
Living Disability Theory: Refections on Access, Research, and Design. In The
22nd International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’20), Association for Computing Machinery, New York, NY, USA, 1–13.
DOI:https://doi.org/10.1145/3373625.3416996
[36] Catherine Holloway and Giulia Barbareschi. 2021. Disability Interactions:
Creating Inclusive Innovations. Synthesis Lectures on Human-Centered
Informatics 14, 6 (December 2021), i–198. DOI:https://doi.org/10.2200/
S01141ED1V01Y202111HCI053
[37] Rhiann Holloway. 2021. Artists as Defenders: Disability Art as Means to Mobilise
Human Rights. (2021).
[38] Erin J. Hoppe. 2017. Perspectives of Young Artists with Disabilities: Negotiating
Identity. In Handbook of Arts Education and Special Education. Routledge.
[39] Alon Ilsar and Gail Kenning. 2020. Inclusive improvisation through sound and
movement mapping: from DMI to ADMI. In The 22nd International ACM SIGAC-
CESS Conference on Computers and Accessibility (ASSETS ’20), Association for
Computing Machinery, New York, NY, USA, 1–8. DOI:https://doi.org/10.1145/
3373625.3416988
[40] Jens Ineland and Lennart Sauer. 2007. Institutional Environments and Sub-
Cultural Belonging: Theatre and Intellectual Disabilities. Scandinavian Journal
of Disability Research 9, 1 (January 2007), 46–57. DOI:https://doi.org/10.1080/
15017410601029770
[41] Rachel Jacobs, Steve Benford, and Ewa Luger. 2015. Behind The Scenes at HCI’s
Turn to the Arts. In Proceedings of the 33rd Annual ACM Conference Extended
Abstracts on Human Factors in Computing Systems (CHI EA ’15), Association
for Computing Machinery, New York, NY, USA, 567–578. DOI:https://doi.org/10.
1145/2702613.2732513
[42] Mohit Jain, Nirmalendu Diwakar, and Manohar Swaminathan. 2021. Smartphone
Usage by Expert Blind Users. In Proceedings of the 2021 CHI Conference on Human
Factors in Computing Systems (CHI ’21), Association for Computing Machinery,
New York, NY, USA, 1–15. DOI:https://doi.org/10.1145/3411764.3445074
[43] Myounghoon Jeon, Rebecca Fiebrink, Ernest A. Edmonds, and Damith Herath.
2019. From rituals to magic: Interactive art and HCI of the past, present, and
future. International Journal of Human-Computer Studies 131, (November 2019),
108–119. DOI:https://doi.org/10.1016/j.ijhcs.2019.06.005
[44] Maria Karam, Carmen Branje, Gabe Nespoli, Norma Thompson, Frank A. Russo,
and Deborah I. Fels. 2010. The emoti-chair: an interactive tactile music exhibit.
In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (CHI EA
’10), Association for Computing Machinery, New York, NY, USA, 3069–3074.
DOI:https://doi.org/10.1145/1753846.1753919
[45] Aaron Karp and Bryan Pardo. 2017. HaptEQ: A Collaborative Tool For Visually
Impaired Audio Producers. In Proceedings of the 12th International Audio Mostly
Conference on Augmented and Participatory Sound and Music Experiences (AM
’17), Association for Computing Machinery, New York, NY, USA, 1–4. DOI:https:
//doi.org/10.1145/3123514.3123531
[46] [46] Sandjar Kozubaev, Chris Elsden, Noura Howell, Marie Louise Juul Søn-
dergaard, Nick Merrill, Britta Schulte, and Richmond Y. Wong. 2020. Expand-
ing Modes of Refection in Design Futuring. In Proceedings of the 2020 CHI
Conference on Human Factors in Computing Systems. Association for Com-
puting Machinery, New York, NY, USA, 1–15. Retrieved April 12, 2022 from
https://doi.org/10.1145/3313831.3376526
[47] Petra Kuppers. 2016. Diversity: Disability. Art Journal 75, 1 (January 2016), 93–97.
DOI:https://doi.org/10.1080/00043249.2016.1171549
[48] Mike Levin. 2010. The art of disability: An interview with Tobin Siebers. Disability
Studies Quarterly 30, 2 (2010).
[49] Susan Levy and Hannah Young. 2020. Arts, Disability and Crip Theory: Temporal
Re-Imagining in Social Care for People with Profound and Multiple Learning
Disabilities. Scandinavian Journal of Disability Research 22, 1 (March 2020), 68–79.
DOI:https://doi.org/10.16993/sjdr.620
[50] Regan Linton. 2021. Acting Training and Instruction for Wheelchair-Using Artists.
Routledge. DOI:https://doi.org/10.4324/9781003125808-2
[51] Kevin M. Storer and Stacy M. Branham. 2021. Deinstitutionalizing Independence:
Discourses of Disability and Housing in Accessible Computing. In The 23rd
International ACM SIGACCESS Conference on Computers and Accessibility (AS-
SETS ’21), Association for Computing Machinery, New York, NY, USA, 1–14.
DOI:https://doi.org/10.1145/3441852.3471213
[52] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich,
and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? A
Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019.
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems,
1–18.
[53] Jennifer Mankof, Gillian R. Hayes, and Devva Kasnitz. 2010. Disability Studies As
a Source of Critical Inquiry for the Field of Assistive Technology. In Proceedings of
ASSETS ’22, October 23–26, 2022, Athens, Greece
the 12th International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’10), ACM, New York, NY, USA, 3–10. DOI:https://doi.org/10.1145/
1878803.1878807
[54] Neil Marcus, Devva Kasnitz, and Pamela Block. 2016. If Disability Is a Dance,
Who Is the Choreographer? A Conversation About Life Occupations, Art,
Movement. In Occupying Disability: Critical Approaches to Community, Jus-
tice, and Decolonizing Disability, Pamela Block, Devva Kasnitz, Akemi Nishida
and Nick Pollard (eds.). Springer Netherlands, Dordrecht, 347–358. DOI:https:
//doi.org/10.1007/978-94-017-9984-3_24
[55] Paddy Masefeld. 2006. Strength: Broadsides from disability on the arts. Trentham
Books Limited.
[56] Ann Millett-Gallant, Elizabeth Howie, and Ann Millett-Gallant. 2017. Disability
and art history. Routledge London and New York.
[57] Stefanie Mueller, Sangha Im, Serafma Gurevich, Alexander Teibrich, Lisa Pfs-
terer, François Guimbretière, and Patrick Baudisch. 2014. WirePrint: 3D printed
previews for fast prototyping. In Proceedings of the 27th annual ACM symposium
on User interface software and technology (UIST ’14), Association for Computing
Machinery, New York, NY, USA, 273–280. DOI:https://doi.org/10.1145/2642918.
2647359
[58] Timothy Neate, Abi Roper, Stephanie Wilson, Jane Marshall, and Madeline Cruice.
2020. CreaTable Content and Tangible Interaction in Aphasia. In Proceedings
of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20),
Association for Computing Machinery, Honolulu, HI, USA, 1–14. DOI:https:
//doi.org/10.1145/3313831.3376490
[59] William Christopher Payne, Alex Yixuan Xu, Fabiha Ahmed, Lisa Ye, and
Amy Hurst. 2020. How Blind and Visually Impaired Composers, Producers,
and Songwriters Leverage and Adapt Music Technology. In The 22nd Interna-
tional ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’20),
Association for Computing Machinery, New York, NY, USA, 1–12. DOI:https:
//doi.org/10.1145/3373625.3417002
[60] Dharani Priyahansika Perera, Jim R. T. Eales, and Kathy Blashki. 2007. The drive
to create: an investigation of tools to support disabled artists. In Proceedings of
the 6th ACM SIGCHI conference on Creativity & cognition (C&C ’07), Association
for Computing Machinery, New York, NY, USA, 147–152. DOI:https://doi.org/10.
1145/1254960.1254981
[61] Helen Polson. 2013. “ The Dance is in Your Body and Not in Your Crutches”:
Technique, Technology, and Agency in Disability Movement Performance. (2013).
[62] Jesse Prinz. 2017. Against outsider art. Journal of Social Philosophy 48, 3 (2017),
250–272.
[63] Halley P. Profta, Abigale Stangl, Laura Matuszewska, Sigrunn Sky, and Shaun K.
Kane. 2016. Nothing to Hide: Aesthetic Customization of Hearing Aids and
Cochlear Implants in an Online Community. In Proceedings of the 18th In-
ternational ACM SIGACCESS Conference on Computers and Accessibility (AS-
SETS ’16), Association for Computing Machinery, Reno, Nevada, USA, 219–227.
DOI:https://doi.org/10.1145/2982142.2982159
[64] Halley P. Profta, Abigale Stangl, Laura Matuszewska, Sigrunn Sky, Raja Kushal-
nagar, and Shaun K. Kane. 2018. “Wear It Loud”: How and Why Hearing Aid and
Cochlear Implant Users Customize Their Devices. ACM Trans. Access. Comput.
11, 3 (September 2018), 13:1-13:32. DOI:https://doi.org/10.1145/3214382
[65] Graham Pullin. 2009. Design Meets Disability. MIT Press.
[66] Grazia Ragone. 2020. Designing Embodied Musical Interaction for Children with
Autism. In The 22nd International ACM SIGACCESS Conference on Computers and
Accessibility (ASSETS ’20), Association for Computing Machinery, New York, NY,
USA, 1–4. DOI:https://doi.org/10.1145/3373625.3417077
[67] Gisela Reyes-Cruz, Joel E. Fischer, and Stuart Reeves. 2020. Reframing Disability
as Competency: Unpacking Everyday Technology Practices of People with Visual
Impairments. In Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems (CHI ’20), Association for Computing Machinery, Honolulu,
HI, USA, 1–13. DOI:https://doi.org/10.1145/3313831.3376767
[68] Kathryn E. Ringland. 2019. A Place to Play: The (Dis)Abled Embodied Experience
for Autistic Children in Online Spaces. In Proceedings of the 2019 CHI Conference
on Human Factors in Computing Systems (CHI ’19), Association for Computing Ma-
chinery, New York, NY, USA, 1–14. DOI:https://doi.org/10.1145/3290605.3300518
[69] Abir Saha and Anne Marie Piper. 2020. Understanding Audio Production Practices
of People with Vision Impairments. In The 22nd International ACM SIGACCESS
Conference on Computers and Accessibility (ASSETS ’20), Association for Comput-
ing Machinery, New York, NY, USA, 1–13. DOI:https://doi.org/10.1145/3373625.
3416993
[70] Richard Sandell, Jocelyn Dodd, and Rosemarie Garland-Thomson. 2010. Re-
presenting disability. Activism and agency in the museum. London (2010).
Giulia Barbareschi and Masa Inakage
[71] Anastasia Schmitz, Catherine Holloway, and Youngjun Cho. 2020. Hearing
through Vibrations: Perception of Musical Emotions by Profoundly Deaf People.
DOI:https://doi.org/10.48550/arXiv.2012.13265
[72] Caitlin Ostrow Seidler. 2011. Fighting Disability Stereotypes with Comics. Art
Education 64, 6 (November 2011), 20–23.
[73] Phoebe Sengers and Chris Csikszentmihályi. 2003. HCI and the arts: a conficted
convergence? In CHI ’03 Extended Abstracts on Human Factors in Computing
Systems (CHI EA ’03), Association for Computing Machinery, New York, NY,
USA, 876–877. DOI:https://doi.org/10.1145/765891.766044
[74] Kristen Shinohara and Jacob O. Wobbrock. 2011. In the Shadow of Misperception:
Assistive Technology Use and Social Interactions. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (CHI ’11), ACM, New York,
NY, USA, 705–714. DOI:https://doi.org/10.1145/1978942.1979044
[75] Kristen Shinohara and Jacob O. Wobbrock. 2016. Self-conscious or self-confdent?
A diary study conceptualizing the social accessibility of assistive technology.
ACM Transactions on Accessible Computing (TACCESS) 8, 2 (2016), 1–31.
[76] Tobin Siebers. 2013. Disability and the theory of complex embodiment—for
identity politics in a new register. The disability studies reader 4, (2013), 278–297.
[77] Per Koren Solvang. 2012. From identity politics to dismodernism? Changes in
the social meaning of disability art. Alter 6, 3 (July 2012), 178–187. DOI:https:
//doi.org/10.1016/j.alter.2012.05.002
[78] Per Koren Solvang. 2018. Between art therapy and disability aesthetics: a so-
ciological approach for understanding the intersection between art practice
and disability discourse. Disability & Society 33, 2 (February 2018), 238–253.
DOI:https://doi.org/10.1080/09687599.2017.1392929
[79] Katta Spiel. 2021. The Bodies of TEI – Investigating Norms and Assumptions in
the Design of Embodied Interaction. In Proceedings of the Fifteenth International
Conference on Tangible, Embedded, and Embodied Interaction (TEI ’21), Association
for Computing Machinery, New York, NY, USA, 1–19. DOI:https://doi.org/10.
1145/3430524.3440651
[80] Katta Spiel, Kathrin Gerling, Cynthia L. Bennett, Emeline Brulé, Rua M. Williams,
Jennifer Rode, and Jennifer Mankof. 2020. Nothing About Us Without Us:
Investigating the Role of Critical Disability Studies in HCI. In Extended Ab-
stracts of the 2020 CHI Conference on Human Factors in Computing Systems
(CHI EA ’20), Association for Computing Machinery, New York, NY, USA, 1–8.
DOI:https://doi.org/10.1145/3334480.3375150
[81] Miriam Sturdee, Makayla Lewis, Angelika Strohmayer, Katta Spiel, Nantia
Koulidou, Sarah Fdili Alaoui, and Josh Urban Davis. 2021. A Plurality of Prac-
tices: Artistic Narratives in HCI Research. In Creativity and Cognition (C&C
’21), Association for Computing Machinery, New York, NY, USA, 1. DOI:https:
//doi.org/10.1145/3450741.3466771
[82] Jennifer Sullivan Sulewski, Heike Boeltzig, and Rooshey Hasnain. 2012. Art and
Disability: Intersecting Identities Among Young Artists with Disabilities. DSQ 32,
1 (January 2012). DOI:https://doi.org/10.18061/dsq.v32i1.3034
[83] Pauline Tranchant, Martha M. Shiell, Marcello Giordano, Alexis Nadeau, Isabelle
Peretz, and Robert J. Zatorre. 2017. Feeling the Beat: Bouncing Synchronization
to Vibrotactile Music in Hearing and Early Deaf People. Frontiers in Neuroscience
11, (2017). Retrieved July 6, 2022 from https://www.frontiersin.org/articles/10.
3389/fnins.2017.00507
[84] Rua M. Williams and Juan E. Gilbert. 2019. Cyborg Perspectives on Computing
Research Reform. In Extended Abstracts of the 2019 CHI Conference on Human
Factors in Computing Systems (CHI EA ’19), Association for Computing Machinery,
New York, NY, USA, 1–11. DOI:https://doi.org/10.1145/3290607.3310421
[85] Gregor Wolbring and Fatima Jamal Al-Deen. 2021. Social Role Narrative of Dis-
abled Artists and Both Their Work in General and in Relation to Science and
Technology. Societies 11, 3 (September 2021), 102. DOI:https://doi.org/10.3390/
soc11030102
[86] Anon Ymous, Katta Spiel, Os Keyes, Rua M. Williams, Judith Good, Eva Hornecker,
and Cynthia L. Bennett. 2020. “I am just terrifed of my future” — Epistemic
Violence in Disability Related Technology Research. In Extended Abstracts of
the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA ’20),
Association for Computing Machinery, New York, NY, USA, 1–16. DOI:https:
//doi.org/10.1145/3334480.3381828
[87] Jung Hyoung Yoon, Caroline Ellison, and Peggy Essl. 2021. Shifting the perspec-
tive from ‘incapable’ to ‘capable’ for artists with cognitive disability; case studies
in Australia and South Korea. Disability & Society 36, 3 (January 2021), 443–467.
DOI:https://doi.org/10.1080/09687599.2020.1751079
[88] Najma Al Zidjaly. 2011. Managing Social Exclusion through Technology: An
Example of Art as Mediated Action. Disability Studies Quarterly 31, 4 (2011).
“Just like meeting in person” - Examination of interdependencies
in dementia-friendly virtual activities.
Elaine Czech Paul Marshall
elaine.czech@bristol.ac.uk p.marshall@bristol.ac.uk
University of Bristol University of Bristol
Bristol, UK Bristol, UK
ABSTRACT
Many dementia-friendly social programs were adapted to online
delivery due to the COVID pandemic. Hasty adaptations make it
unclear how to design these programs to capture the benefts of
online delivery and face-to-face interactions. To understand the
complexities of program delivery, we interviewed program coor-
dinators and held focus groups with people living with dementia
(PLWD) and their informal carers. We applied an interdependence
framework to examine how the relationships between individuals
afect program benefts. We found that interdependencies within an
organization related to fnances and networking are key and that
organizational and individual interdependencies converge during
program delivery. Our fndings suggest these two interdependen-
cies could infuence one another more efectively if technology,
like video conferencing, were designed to account for it. We dis-
cuss how an expanded notion of interdependency for the design of
technology helps expand inclusivity in accessible social programs.
CCS CONCEPTS
• Social and professional topics → Seniors; • Human-centered
computing → Accessibility technologies; Accessibility theory,
concepts and paradigms.
KEYWORDS
interdependency, dementia, socialization, community
ACM Reference Format:
Elaine Czech, Paul Marshall, and Oussama Metatla. 2022. “Just like meeting
in person” - Examination of interdependencies in dementia-friendly virtual
activities. . In The 24th International ACM SIGACCESS Conference on Comput-
ers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM,
New York, NY, USA, 15 pages. https://doi.org/10.1145/3517428.3544815
1 INTRODUCTION
Due to the COVID-19 pandemic, social organizations which ofered
face-to-face programs for people living with dementia (PLWD) and
their informal carers considered alternative ways of delivering their
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544815
Oussama Metatla
o.metatla@bristol.ac.uk
University of Bristol
Bristol, UK
Figure 1: [Left] Interdependence frame[7]. [Right] Our fnd-
ings suggest that the environment consists of organizational
interdependencies. The intervention (program’s method of
delivery or for Bennett et al. assistive technology) bridges
the interdependency types
services1 . Many of these organizations chose to adapt their pro-
grams to use technologies like videoconferencing. However, factors
such as an organization’s fnances, service users’ willingness to
engage with technology, and staf members’ technology familiarity
impacted program adaptation. These factors caused some programs
to become unfamiliar or unreliable to people living with dementia,
leading to decreased participation and increased social isolation
[16, 57, 78, 80]. Since stability is key for these programs, fexibility
to sudden or drastic changes is limited.
Nevertheless, fexibility is essential for social programs pre-
scribed to help a person maintain their overall wellbeing, as having
these resources suddenly unavailable can be detrimental [32]. So-
cial prescribing is a non-medical intervention that links "clinical
practice with activities and support services within [a] community"
[26]. It can help to reduce social isolation and depression for PLWD
[2]. However, quick adaptations not tailored to those living with de-
mentia can make accessing these programs impossible, eliminating
any benefts.
Although the term "social prescription" is relatively new to HCI
[90] there has been extensive research into strengthening social
connections for PLWD [30, 36, 40, 59, 83, 87]. For example, solu-
tions that create opportunities for sharing social experiences within
families [36, 59, 87] and developing activities for care homes resi-
dents at varying stages of dementia [30, 40]. However, there is little
1 Forthis research, "social organizations" are those which provide a cultural, historical,
or social service to people with dementia, such as art and history museums, public
zoos, and dementia-specifc social services. "Programs" refers to activities hosted by
these organizations such as art creation, tours of their collections, and planned social
gatherings.
ASSETS ’22, October 23–26, 2022, Athens, Greece
research on how technology can assist those living with mild cog-
nitive impairment or early stage dementia to pursue wider social
engagements.
With other groups, there has been a shift towards exploring
social accessibility and how environmental factors afect the design
of technology [34, 66, 75]. Academic literature outside of HCI advo-
cates for developing "dementia-friendly" communities to improve
social connectedness and inclusivity [67, 79]. In turn, HCI and de-
mentia research has also started to broaden its understanding of
community engagement and PLWD [19, 24, 48]. This research has
highlighted that the degree of societal inclusion infuences how
PLWD engage with the world. Therefore, there is potential in under-
standing how organizations that provide social prescriptive services
might improve their social inclusion eforts through technology.
However, when organizations assume a one-to-one adaptation of
an in-person program to any readily online platform will have
similar positive efects this potentially limits social interactive for
PLWD [38]. Such adaptations do not consider that readily available
platforms are designed for a hyper-cognitive society [80].
In parallel with this shift in perspective towards broader so-
cietal interactions, within accessibility and assistive technology
research there has been a change from an independence framework
to an interdependence framework [7]. Interdependence repositions
research to focus on being in the world rather than just micro-
interaction with immediate social circles. However, this shift as it
applies to the engagement of PLWD with social programs has not
been explored yet [21, 24, 25]. In this paper, we considered how
technology can improve dementia-friendly programs. We believe
that designing these programs to support interdependency will, in
the future, allow social programs to be more inclusive.
To achieve our aim, we consider interdependencies within dementia-
friendly programs and how technology can strengthen them. We
report fndings from semi-structured interviews with 15 program
coordinators who manage and design dementia-friendly program-
ming. We also report fndings from focus groups conducted with six
participants living with dementia and three of their informal carers.
We interviewed key stakeholders deliberately as we believe that
realizing dementia-inclusivity means understanding and meeting
the needs of those directly impacted. Our fndings suggest that
an expanded application of Bennett et al.’s [7] interdependency
framework is needed to outline future social-dementia technology
research (Fig. 1). Bennett et al. discussed that interdependence could
lead to individuals working as a collective. However, our analysis
reveals that individuals working as part of a collective within an
organization creates a diferent layer of interdependence not yet
considered. We outline how organizational interdependencies can
afect individual interdependencies. Technology used in program
delivery creates an opportunity that allows these two networks
of interdependencies to converge and then infuence each other.
Our paper makes three primary contributions to the ASSETS com-
munity: 1) identifying and highlighting the complexities related to
designing social programs for PLWD; 2) grounding and expanding
the interdependency framework in this context; and 3) providing a
design checklist to assist coordinators and the ASSETS community
with developing technology for facilitating social programs.
Czech, et al.
2 RELATED WORKS
2.1 Accessing Social Spaces and Dementia
Social participation by all is considered vital for a community to
be considered age-friendly [65]. Social connection is essential for
overall community members’ health and wellbeing [65, 81]. Social
prescription, which encourages integration into one’s community, is
becoming more common. When used in conjunction with a medical
prescription, it allows a person’s health to be managed holistically
[2]. Since dementia is progressive and there is no current cure,
social prescribing can help slow progression and reduce medical
costs [26].
Often it involves art appreciation or making/crafting through
an institution such as a museum or care home, although Kelson
et al. [44] found that enjoyment of public art was also benefcial
for PLWD. Their research suggests that social prescribing does not
need to occur within an institutional setting and can be informal.
However, Thomson et al. [81] found that museum-based interven-
tions allowed participants to interact with a range of people not
typically encountered. Compared to informal interactions, inter-
ventions occurring within a museum increased the likelihood of
short-term or long-term bonding between participants.
While HCI has explored leveraging technology to make museums
more accessible and interactive [13, 39, 82], there is little research
on how to support the social aspects of visiting a museum for PLWD.
Ryding [72] designed a two-person interactive app for museums
and found that this social dynamic allowed participants to create a
more bespoke experience and explore the museum more playfully.
In contrast, Clarke et al.’s [17] research on interactive exhibition
design explored distributed control, fnding enforced cooperation
allowed participants to have more enriched social interactions by
getting them to focus on others.
Beyond museums, there is not enough research on the appli-
cation of technology to improve the social accessibility of public
spaces, not only for PLWD but for older adults in general [69].
Hodge et al. [37] examined how PLWD and their loved ones created
new memories on public outings. However, they focused on en-
riching the relationships between the person living with dementia
and their loved ones and not the interactions between participants
and the public. Therefore, more research is needed to examine how
technology can improve opportunities for social connectivity in
various public settings for PLWD.
2.2 Dementia-Specifc Interventions
Overall, dementia and HCI research has helped us understand what
type of interactions and designs are appropriate for those afected
by dementia [29, 30, 49, 56, 85, 87], advocating for approaches that
allow the person living with dementia to have agency and express
their personhood [10, 47, 49]. Still, since the focus of these solutions
is dementia, often it seems the fndings and solutions do not appeal
to those not living by dementia.
In the past few years, researchers in the dementia and HCI feld
have recognized this limitation and have begun to focus on en-
compassing a broader frame of reference. Although still within
the care home setting, Dixon and Lazar [24] looked at the efects
of social perspective and dementia care. Their research analyzed
the various infuences that determine the care practitioner’s care
“Just like meeting in person” - Examination of interdependencies in dementia-friendly virtual activities.
style when interacting with a person with dementia. An example
is carer wages: Since carer positions are often underpaid, carers
may save time with pragmatic approaches that attribute behav-
iors to dementia instead of looking for meaning in the person’s
behavior [24]. Similarly, Dai and Mofatt’s [19] research looked at
how social perspective comes into play in a community setting by
examining how technology could support a reminiscent activity
in a public library for carers and their loved ones with dementia.
Social interaction in a public space provided them with "a sense of
normality." This normalcy is extended to the community as it can
help destigmatize the stereotypical notions of living with dementia
[19]. Progressing the boundaries of contextual infuences further,
Dixon, Piper, and Lazar [25] outlined everyday self-management
strategies used by PLWD. Infuenced by critical realism theory and
Bennett et al.’s interdependence framework [7], Dixon, Piper, and
Lazar [25] suggest that societal pressures afect how PLWD navi-
gate interpersonal relationships through the use of technology to
manage their dementia [25]. Overall, these papers explore how so-
cial perspectives infuence technology use, how people view those
living with dementia, and how PLWD view themselves.
Despite progress in the feld of HCI and dementia, the COVID
pandemic forced many social interactions to be conducted at a
distance. As a result of the pandemic, it became important to un-
derstand how readily available technologies could be adapted to
provide support to those living with dementia and their carers
[16, 60, 80]. For instance Mynatt et al. [60] researched how an
in-person empowerment program for people with mild cognitive
impairment was adapted to asynchronously online delivery. While
they added synchronous video calls over Zoom, they found that
participants still required more facilitation to have meaningful in-
teractions. However, facilitators found that the online format made
scafolding activities difcult. As of writing, HCI research related
to the impact of the pandemic on online social interactions for not
only those living with dementia, but also older adults in general[76],
is limited. This research builds upon the fndings of Mynatt et al.
[60] by considering the contextual infuences that afect the devel-
opment of meaningful interactions for PLWD online.
2.3 Interdependence to Promote Inclusivity
Models for creating societal inclusion have tended to focus on
helping those with disabilities achieve independence. However,
emphasizing self-sufciency ignores that all people form interde-
pendencies by contributing and relying on community support
and assistance [7]. The prevalence of social connections suggests
that strengthening interdependencies between people of varying
abilities can improve societal inclusion. According to White et al.
[88], supporting interdependence increases social capital, helping a
society function more efciently.
Therefore, the interdependence framework proposed by Bennett
et al. [7] encourages "tangible accommodations", such as assistive
technologies or a wheelchair ramp, to be assessed and designed
within their context of use. They defne context to include people
who might not interact with the accommodation in addition to the
overall environment. However, they focus on how the accommoda-
tion afects interpersonal relationships. One example their paper
provides is how two visually impaired people navigated out of a
ASSETS ’22, October 23–26, 2022, Athens, Greece
store through environmental cues and relying on each other. This
example highlights how anyone can use an accommodation, even
if it was not intentionally designed for them.
Das et al. [21] further examine the social and structural aspects
of collaboration that afect task completion. They highlight how
ableism and power dynamics in the workplace make completing
tasks inaccessible to those with visual impairments, preventing
access to jobs and promotions. While not implicitly stated, their
fndings suggest that completing a task is interdependent with the
workplace’s structure. Thus, understanding structural infuences
can shed light on the more signifcant issues that need to be ad-
dressed before implementing a technology solution. A limitation
of this research is that it did not include sighted collaborators,
who might have shed light on how interdependency then afects
colleagues’ interpersonal relationships. Moreover, design research
by Baldwin et al. [3] applied the interdependence framework to
community design and was able to increase public attentiveness
to inclusivity and gain community interest in their work. Increas-
ing public awareness of dementia is crucial for destigmatizing it
and building community inclusivity. As these researchers have ap-
plied it, the interdependence frame suggests that it can holistically
examine an experience making clear appropriate solutions.
Overall, there is a lack of research on social programs associated
with social prescription services for PLWD in HCI. The public na-
ture of these programs means an intermix of abilities infuenced by
societal structures will occur. Therefore, there is a need to frst iden-
tify and characterize the interdependencies involved within these
programs to then be able to improve the social support participants
receive. There is an opportunity for technology to improve the qual-
ity of the programs and lead to greater inclusivity. In this paper, we
use a critical realist approach to the interdependence framework to
examine how high-level or contextual factors afected individuals
[7, 31].
3 STUDY OF DEMENTIA-FRIENDLY SOCIAL
PROGRAMS
To identify and characterize the interdependencies that assist with
program delivery, we interviewed program coordinators and held
two focus groups: one with PLWD and one with carers. We applied
a grounded theory [15], critical realist [12, 63] approach to gain a
thorough understanding of social programs. Since programs are
typically designed and then experienced, we aimed to mirror this in
our data collection and analysis procedures. First, we collected and
examined data from the coordinators to identify the key structural
infuences and understand the development to delivery process
of programs. Once the structure and development process was
understood, we could then identify the primary interdependencies.
We applied the resulting interdependency structure deductively to
examine the perspectives of PLWD and their carers to determine
whether social interdependencies were supported.
3.1 Participants
The participants were divided into two groups: program coordina-
tors, and PLWD and their carers. An information sheet and consent
form were given to those interested and signed before the study
began.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Six participants who self-identifed as living with dementia (ages
55-90, mean 72, 4 male) and three of their carers (ages 56-65, mean
62, all female) were recruited through an Alzheimer’s Social Club
(where the frst author volunteers, Tab.1). Participation in the Social
Club requires an early-to-mid-stage dementia diagnosis; therefore,
we did not require participants to share their specifc diagnoses.
They were asked to participate in a group interview: one for only
carers and one for those living with dementia. These were con-
ducted separately so that the voices of those living with dementia
would not potentially be dismissed or overshadowed [20]. The par-
ticipants were interviewed as a group since they were comfortable
and felt safe discussing various topics with each other and encour-
aged each other to share. The group interview was also chosen so
that it would mimic the feel of a Social Club gathering. Participants
living with dementia were not tech-averse and had a basic under-
standing of technological devices (such as smartphones or voice
controlled home devices). Although all, with the exception of Evan,
needed some form of carer assistance with using Zoom, ranging
from reminders to use it to being entirely dependent on the carer
to get them online.
Table 1: List of pseudonyms for participants living with de-
mentia and their informal carers.
Person Living with Dementia Carer (relation)
Adam Alice, daughter
Beth
Chris Christine, spouse
Daniel Dana, spouse
Evan
Additionally, program coordinators from various social orga-
nizations were recruited to be interviewed (Tab.2). Coordinators
were recruited from the US and UK due to having similar social
services available for PLWD. Coordinators in the US were recruited
via word of mouth, while those in the UK were found via groups
listed on the Dementia Engagement and Empowerment Project UK
[43], and were recruited via email invite. Fifteen semi-structured
interviews were conducted with seven program coordinators from
the UK and eight from the US. While each coordinator’s job role
varied slightly, most oversaw managing and designing accessibility
programming.
3.2 Procedures
We obtained approval to conduct this research from our university
Ethics Committee. Before the start of each individual or group in-
terview, we collected verbal consent. All interviews and both group
interviews were performed remotely via Zoom, with all participants
joining individually from their homes. Interviews were one-to-one
(except for Michael and Nancy, who interviewed together) with the
frst author and lasted 30-60 minutes. The frst group interview was
conducted with six participants living with dementia and was sched-
uled for 1 hour but ended up lasting 2 hours since the participants
wanted to keep socializing. While the second group interview, with
the three informal carers, lasted 1.5 hours. Detailed notes and au-
dio recordings were taken for the group and individual interviews.
Czech, et al.
These notes and transcriptions of the audio-fles were used for
analysis. Pseudonyms were used, and identifable information was
removed from the transcription. We followed a semi-structured for-
mat to allow participants to talk freely about their experiences with
social programs. For the interviews, coordinators were asked about
their programming in general, how programming has changed and
adapted, and how they envision programming to be in the future2 .
For the group with participants living with dementia, they were
asked how they would want to change the programs, who they
would want to invite to the programs, and how dementia has af-
fected their lives. For the group of carers, we asked how they got
involved with programming, how the pandemic is afecting their
loved one’s dementia, how dementia has afected their lives, and
how programming could be improved.
Program Coordinator Interviews
Grounded Theory
1. Open Coding: Influenced by person-centered care model
2. Axial Coding: institution features, programming creation and design, and working with others
Theme development: Reliance and Support
Deeper analysis with Critical Realism
Abduction
Develop hypotheses for research question: What allows programs to develop and be successful?
Determined interdependencies as most plausible hypothesis and that two networks are at play:
Contextual Social
Occur before Occur during
program delivery or after delivery
Retroduction
Review data with specific question:
What are the causal mechanisms behind these interdependency networks?
Once causal mechanisms determined, review other data confirm these
People living with dementia Focus Group Carer Focus Group
Confirm causal mechanisms are correct
Determine where and when these mechanisms either support or deter
interdependency
Figure 2: Diagram of our process with grounded theory criti-
cal realist analysis.
3.3 Data Analysis
We took a critical realist3 approach to our data (Fig. 2)[14, 28, 31, 63].
To begin, the frst author approached the data with a constructivist
grounded theory approach [14] to focus on the empirical and ac-
tual [12]; this provided a concrete framework to determine demi-
regularities [28, 63]. The frst author began by open coding the data
from the program coordinator interviews [15]. Then, following
2 For example:How do you design your programs to be accessible? How has your
programming changed? What is the future of programming?
3 Critical realist ontology stratifes reality into three layers: the empirical, the actual, and
the real. Analysis thus moves from what we experience, to events that occur whether
we experience them or not, to fnally the causal mechanisms of an experience[28].
Causal mechanisms are the "inherent properties in an object or structure that act as
causal forces to produce events" [28]. The key steps of critical realism are identifcation
of demi-regularities, abduction, and retroduction [28, 63]. Since the world is not always
orderly, critical realists believe that there are only tendencies or demi-regularities to
occurrences [28]. Through abduction, possible theoretical explanations are considered
for these demi-regularities. Hypotheses are framed for each possible explanation and
empirically checked by examining the data until the most plausible explanation is
selected to be pursued [14, 63]. Finally, through retroduction, a process like abduction
but in which the researcher looks at the data with a specifc question [28], the most
plausible explanation is explored.
“Just like meeting in person” - Examination of interdependencies in dementia-friendly virtual activities. ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 2: List of pseudonyms for coordinators.

Coordinator’s Name Gender Organization Type Location

Barb female Dementia Social Service Midwestern US
Elizabeth female History Museum Midwestern US
Francis female Dementia Social Service Eastern US
Gretchen female History Museum Western US
Isabella female Dementia Social Service Southeast England
Joan female Art Museum Southwest England
Kevin male History Museum Northern England
Laura female History Museum Northern England
Michael male Dramatic Arts Organization Southwest England
Nancy female Dramatic Arts Organization Southwest England
Polly female History Museum Midwestern US
Quentin male Art Museum Midwestern US
Rachel female Older Adults Social Services Southwest England
Theresa female Botanical Garden Eastern US
Ursula female Zoo Midwestern US

constructivist grounded theory methods, we refected on our per-
spectives and recognized our approach to the data was infuenced
by the person-centered care model perspective [15]. Throughout
our analysis, the frst author wrote analytic memos and engaged
in a process of constant comparison of data to data. These codes
were then reviewed collectively by all the authors. The codes made
clear the hierarchy that went into program development, and all
the authors again validated these emerging concepts. The codes
were categorized into institution features, programming creation and
design, and working with others through axial coding.
In an initial analysis, the themes of Reliance and Support were
identifed in all levels of program development. Organizations and
program coordinators relied on and were supported by other or-
ganizations for funding, as sources of knowledge and comparison,
and as a means of improving community outreach. Participants
relied on and were supported by their carers, other participants, and
program coordinators to supply consistent programming. Organiza-
tions and participants were brought together through the delivery
of the program. Informed by our earlier interviews, we adjusted our
interview questions to understand the complexity of the interde-
pendencies better and asked coordinators how their organizations
engaged with their community and what are the challenges they
face with online.
However, with critical realism, researchers do not commit to
their initial theory as it is recognized that deeper analysis is needed
to develop a "more accurate explanation of reality" [28]. Next, the
frst author engaged with the process of abduction and examined
interdependency as one plausible explanation for the causes behind
a program’s development and success [63]. Finally, we retroduc-
tively analyzed the codes to identify the causal mechanisms behind
the interdependencies [7, 12, 31, 63]. We distinguished codes re-
lated to things that occurred before a program was delivered as
part of the organization. In contrast, the codes that occurred during
or after a program was delivered were tied to social interactions.
We then deductively analyzed our focus group data to determine
when and how interdependencies are or are not supporting those
living with dementia and their carers. This data allowed us to un-
derstand better the current strengths and weaknesses of the social
interdependencies which occur because of a program.
4 FINDINGS
We determined two main networks of interdependency which infu-
ence the various aspects of social programming. The frst includes
the interdependencies which occur on a contextual level before the
program among the organizations involved with it (such as inter-
/intra-organization networking and fnances/resources). The other
main network of interdependencies occurs during and after the
program on the social level. These are the interactions among par-
ticipants and determine the amount of social fulfllment provided
by a program (such as interactions between staf, PLWD, carers,
etc.). Then, during program delivery, these two networks have an
opportunity to converge.
4.1 Organizational Interdependencies
Organizational interdependencies occur between organizations pro-
viding the context that determines the nature of a program and
whether a program exists. Based on the interviews with the pro-
gram coordinators, we derived that two key mechanisms, fnances
and networking, infuenced the reliability and supportive strength
of the interdependency between organizations.
4.1.1 Financial Interdependency. While accessible and inclusive
programming was important to all the organizations we inter-
viewed, what each organization provided depended on their f-
nancial stability. Stable fnances often meant an organization could
hire enough staf to accommodate community needs. For Polly,
COVID meant her team was downsized: “I am the manager in our
department and then I have two part time staf members [. . . ] before
COVID, we had a quite a few more people involved [. . . ] the downsiz-
ing has defnitely, hurt all of us.” Fewer staf mean that organizations
have less human capacity to design and provide programs.
Loss of funding could also mean, for larger organizations like the
zoo, reshufing of staf. During the COVID lockdowns, Ursela had
ASSETS ’22, October 23–26, 2022, Athens, Greece
to once again take over running the dementia programming since
the program lead was laid of: “[I had taken] a step back from the
[dementia programs . . . ] And one of my colleagues took over, and then
when COVID happened, he unfortunately got laid of. And so I was
able to step back into the program, and kind of take over and get back
acquainted with our individuals.” Having to get reacquainted with
a group can mean that program quality sufers since participants
have built up a rapport and expectation of the program based on
the previous coordinator.
Since the 2020 pandemic caused loss of funding for many or-
ganizations, afordable solutions had to be creatively confgured.
In general, the coordinators interviewed found it cheaper or more
straightforward to adapt programs online. The cost efciency meant
that Rachel’s organization has kept a signifcant portion of its pro-
grams online or via telephone. "We don’t have to pay for the venue
[... or] getting people to locations." Before the COVID lockdowns
in the UK, Rachel’s organization did not focus on using technol-
ogy. However, technology has become more critical now that the
organization sees fnancial and even logistical benefts.
Online solutions increased participation accessibility previously
prevented by physical barriers. Barb’s organization could save costs
by not needing to set up multiple locations since, through online
programming, they could, for instance, reach one participant living
more than 400 miles away. Without online programming, Barb’s
organization would not provide an opportunity for that person to
socialize with others living with dementia. "I think virtually there’s
that impact that we can reach more people, right? Especially, you
know we have [participants on the other side of the State attending
. . . ] I think doing it virtually it just really shows how much more could
be done." By providing virtual programs, Barb realized how limited
the reach of in-person programs has been since people living too
far away could never attend thems.
Finances in another sense afected programming since coordi-
nators also had to be aware of the fnancial limitations of their
participants. For Ursula “[accessibility] was a conversation [our orga-
nization] had a lot, at the time, we weren’t thinking about seniors, we
were thinking about kids. So these kids who don’t have internet, who
don’t have computers, how are we going to access them?” Despite the
digital divide in the US, all of the organizations we interviewed in
the States had opted to do online programming. However, in the
UK, organizations we talked with seemed more likely to opt for
more low tech solutions (such as mailing items or phone commu-
nication) since this would not limit their program’s availability to
older participants. Yet, beyond this distinction, coordinators in the
two countries shared similar experiences. Finances then decide if
there will be a program, what type of program it will be, who it
will be for, and how it will be delivered. Finances can either help
to strengthen or weaken an interdependency between a program
and an organization. Thus, while fnances determine how inter-
dependencies at the organizational level come about; networking
determines the types of interdependencies.
4.1.2 Networking Interdependency. Networking was another key
mechanism for determining whether an inter-organizational inter-
dependency occurs. The benefts to an organization in establishing
professional partnerships include: opening funding possibilities,
Czech, et al.
exchanging ideas, expanding community outreach, and develop-
ing new programs. For well-known and larger organizations, like
Quentin’s art museum or Ursula’s zoo, the organization’s promi-
nence made fnding partners easy: “Our recurring partners have
been a result of somebody at [one of our main partner organization’s]
saying, you know you really need to talk to so and so 10 miles up the
road or [this other organization’s] got a program there that would
really beneft [from working with your organization]” (Quentin).
For organizations still establishing themselves in their communi-
ties, they joined or formed a group of local organizations. Around
2015, Elizabeth seeing a need for accessibility services in museums
pioneered an “Alliance of Cultural Accessibility” so that organiza-
tions in her State could have relevant conversations around being
and becoming more accessible. Similarly, in the UK, Kevin’s orga-
nization became “a member of a local collaboration of community
groups [. . . which] enabled us to engage with groups, develop new
partnerships, and [has] supported us in our ongoing program.”
Because of its critical role in the functioning of social organiza-
tions and even its possibility to afect fnances, the ebb and fow
of networking, despite the COVID-19 pandemic, functioned con-
sistently. In Polly’s case, the pandemic seemed to help with her
ability to network efectively. "I have a lot of organizations that have
helped us out in like again, the last fve, six years that I have been in
this role, and I haven’t been able to be as reciprocal in the past. And
thankfully, the pandemic has provided me a little bit of fexibility to
do that [. . . ] I’ve got probably about seven or eight partnerships, right
now (laughs)." By reciprocating her partnering organizations, Polly
can strengthen her organization’s inter-organizational interdepen-
dencies. Strengthening existing partnerships also can improve an
organization’s reputation.
However, the efort and reward of reciprocity had to remain
equal for partnerships to occur or remain. Rachel has found that
"you have to be giving something extra that the activity coordinator
can’t give for it to be worth their while, [if it is] something that
they probably can do themselves in the home then there’s not much
of a draw for them." Which for social organization programming,
often revolves around providing space for socializing, virtual or
telephone activities could sometimes prove difcult to recruit for.
Without something appealing, partnerships are unable to form
or be successful. Alternatively, in the case of Michael and Nancy,
they dissolved some of their care home partnerships because staf
turnover made coordination difcult.
Often, interdependencies through networking occur because of
the organization’s reputation or due to staf members changing
felds (i.e., an educator moving into museum education). These
traits help organizations expand their community outreach. For
instance, Francis’ organization expanded community outreach by
relying on her established relationships from her previous job. Fran-
cis was formerly a museum educator and was hired to develop the
cultural programming for a dementia organization: "In my role, my
experience as a long-time museum educator allowed me to establish
rapport with almost everyone, I’ve done education programs with
and invite them to [join the dementia organization’s] programs and
services." Organizations will hire specifc staf to absorb their con-
nections and make themselves more qualifed to develop programs
that will interest those connections. At the organizational level, the
“Just like meeting in person” - Examination of interdependencies in dementia-friendly virtual activities.
reliance and support, or lack thereof, determines what programs
are developed and for who they are developed.
4.2 Social Interdependencies
Person-centered strategies within dementia-specifc programs led
to social interdependence between the participants and program
coordinators. We derived four strategies which develop social in-
terdependence and lead to feelings of social inclusion: creating a
socially safe atmosphere, supporting social needs, co-leading pro-
grams, and building lasting connections.
4.2.1 Socially Safe Atmosphere to Support Creation of Interdepen-
dencies. The atmosphere is crucial for making a program into some-
thing socially fulflling and must be negotiated by the staf and
participants. The difculty of socializing that PLWD face was re-
ported by Dana, wife and carer of Daniel, "When [my husband]
is with other people that don’t have any issues, he truly clams up
and takes himself away from the group [. . . with the Social Club
though] that’s usually the time when I hear him laugh." In social
situations outside of the Club, Daniel is uncomfortable; however,
being around those with a similar diagnosis allows him to relax and
have fun. Presentation of self with dementia might infuence this
impression[33]. Participants of the Social Club would use humor
to provide empathy and serve as a distraction when someone had
some difculty due to their cognitive impairment. In this exchange
Adam had asked Beth how she had found out about their group:
Beth: I don’t remember, psssh. I really don’t.
Adam: (Laughs) That’s why you’re in the group! (col-
lective laughter)
Laughter was observed to help relieve tension in the group and
ease the social anxiety associated with a dementia diagnosis. This
relaxed and playful atmosphere allowed participants to have a
positive experience while engaging in socialization. Dementia co-
ordinators, like Barb, know their programs create "a safe space
[for PLWD] to still socially interact." So, unsurprisingly, we found
that often those working at dementia services were the ones who
reached out to the cultural organizations we interviewed to set up
activities. By partnering with other organizations, the dementia
service providers could bring that socially safe atmosphere with
them to other spaces.
Coordinators also tried to be mindful of how they structured
their programs by trying to avoid singling out a participant. Laura
for instance avoids using the terms “reminiscence” or “remember.”
“I don’t want to sit down with someone and be like, remember this,
remember this [. . . so] we call the boxes [with museum items] in-
spiration boxes rather than reminiscence boxes [. . . for example] an
object that me and you don’t remember, we can’t reminisce about it,
but we can be inspired by it.” Coordinators at the various organiza-
tions would then focus on the use of senses in the present. At the
botanical garden, Theresa encourages participants to focus on their
senses: “[You can] talk to them about a color, a sense. Something that
they ate, something that they you know can recall, or something that
they’re feeling at the moment [. . . ] It’s about enjoying the moment,
being in nature, connecting with plants, no judgment, nothing to learn,
nothing to know, just enjoying.” To focus on the present (and not
past or future) coordinators need to work interdependently with
ASSETS ’22, October 23–26, 2022, Athens, Greece
participants to co-discover what is going on or being inspired by
the moment.
However, over virtual platforms coordinators and participants
found it difcult to maintain the same level of social atmosphere.
Christine noted that she did not think the virtual presentation
was as benefcial as in-person had been for her husband. "Since
COVID, Chris just sits back. You know he doesn’t participate." Despite
Christine feeling that attending the programs overall is positive,
decreased engagement has her concerned about the current virtual
format.
Hastily adapting their programs online allowed coordinators to
continue to nurture the bonds already developed by participants
who had frequently engaged with their programs. Being able to
extend this engagement has also allowed Ursela to speculate about
how they can add to their programs: “I like the combination of both.
I think there’s a lot of value in being able to connect with people
frst [virtually] before they come to the zoo.” Virtually connecting
then not only allows for maintaining connections but potentially
for building them in advance of coming into a new space. There-
fore, how participants interact and how coordinators design their
program makes a space socially safe for PLWD.
4.2.2 Supporting Unique Social Needs through Interdependence. Co-
ordinators further try to reduce anxiety and social pressures for
those living with dementia and their carers. As dementia can afect
social interactions [11, 19, 49, 87], this ends up afecting a person’s
closest relations the most (i.e., their spouse/informal caregiver).
Carers are then not only relied on for daily living needs but also
social needs. "[Daniel was becoming] more and more isolated and he,
you know, basically any friends, acquaintances whatever they’ve kind
of fallen by the wayside, sooo it was me entertaining [him by myself]
and I (laughs) I needed help with that." For that reason, Dana had
her spouse attending various programs for PLWD. Social organi-
zations recognize this reliance and thus design their programs for
the person with dementia and their carer. Coordinators then work
with carers within the program setting to support the person living
with dementia to engage socially. By sharing this responsibility, the
coordinator and the carer can avoid the pressure and other emo-
tions that can come with being the sole source of another person’s
social fulfllment.
This reliance between the informal carer and the coordinator
became more apparent to coordinators when the informal carer
was not present. Quentin’s museum, for instance, designed their
in-person art appreciation and creation program with this pair in
mind. However, with the sudden switch to virtual, he was surprised
to fnd that often just the person with dementia attended. Because
the person with dementia was alone, Quentin found it challenging
to engage with the participants and felt the result of the program
was not as socially fulflling. "We see in the in-person version a lot of
appreciation at the end. [. . . The participants] get to know each other
through the years and form an extra level of support [. . . ] we see from
time to time, a level of appreciation, sometimes within [the] pairs."
Additionally, working in-person allowed coordinators to provide
individual support to participants helping them remain engaged.
However, when programs moved online, participants like Adam
had to get online on his own, making attending programs more
inconvenient for his daughter Alice. Alice not only did not receive
ASSETS ’22, October 23–26, 2022, Athens, Greece
the social benefts of attending the programs with her father, but
she also had to do more work on her own to help coordinate Adam’s
ability to engage with a program. Alice would have to organize
all the links and be ready to do over-the-phone troubleshooting.
"There’s a lot of accommodation, a lot of things that I do [. . . ] I make
the spreadsheet for him [. . . and] I FaceTime him, so I watch every
step he does." Therefore, for the informal carers who did not live
with the person living with dementia, supporting their social needs
became more difcult with the hasty switch to online.
Aware of the technical limitations that not only those with de-
mentia face, but older adults in general deal with, in the UK, organi-
zations tended to avoid virtual programs. As Francis noted, “The US
is all Zoom. Assuming everyone has access [. . . which is] especially
difcult for older adults who are not familiar or comfortable with
technology." By not ofering programs that meet the technological
limitations of the participants, programs are unable to meet their
social needs. Which, as Rachel found with her participants in the
UK, “it’s the conversation, regardless of how it’s delivered [. . . ] they’re
happy just to chat on the phone.” In contrast, those we interviewed
who are living with dementia felt that video conferencing was im-
portant for their experience. Daniel felt that virtual was: “just like
meeting in person, we see each other, and we can see if we’re happy or
sad or, then we can talk about experiences, [Zoom] is a very way, good
way [sic] to handle it.” Therefore, negotiating the engagement efort
interdependently so that it would meet the needs of participants
and coordinators leads to the greatest sense of social fulfllment for
all involved.
4.2.3 Co-leading Programs to Build Interdependence. When design-
ing programs for people with dementia, organizations try to cater
to their participants’ interests while keeping the topics or themes
inclusive. These design strategies tended to revolve around a tangi-
ble item that was used as inspiration to allow the participants to tell
others about themselves and control the direction of the activity.
Designing programs in this way also allows participants to teach
organizations about items in their collections. This opportunity
creates an interdependency based on knowledge exchange and pro-
vides a sense of social fulfllment by teaching the person a sense
of self-worth. Laura’s museum developed with a local dementia
services organization their "Inspiration Box" program. During this
program, informal carers and their loved ones with dementia would
gather at the museum and use the items in the box as social inspira-
tion. Laura then provided us with an instance in which she gained
a new appreciation for an object because the participants shared
their knowledge of how to use the object:
"When I frst came across [the scrub brush] and put it in the box I
was looking at, I was like, ’eww’, like it just looks a bit scrufy, but it
ended up in the box. And do you know what that is? My star object
[...] I get all these older people who then show us, like, get down on
their hands and knees and start scrubbing (laughter)." -Laura
Stepping into the role as teacher and demonstrating how they
would use the scrub brush allowed participants to have agency over
the program material. It also provided the organization staf with
information about the scrub brush and its relation to their local
culture. Moreover, the opportunity to fnd the trigger that would
allow the person living with dementia to communicate their life
story provides both staf and the participants with social benefts.
Czech, et al.
For PLWD, social situations can become difcult to follow. There-
fore, unless clear opportunities are presented, they can become
withdrawn[47]. Applied in other contexts with other objects not
necessarily part of a museum collection could mean that staf does
not need to design their program structure extensively. Instead,
they can rely on their interdependency with their participants to
enrich program content. For instance, during the focus group for
members of the Social Club, the participants ofered each other
advice. Adam told a story about his difculties remembering names
and numbers, a situation with which Francine empathized. Adam
then told Francine one of his tricks, "[On Zoom] the banner, if you
just take a picture of the screen, you get everybody’s picture, with a
name on it." He then shared with the group his printed-out image,
and the other members afrmed it was a good idea. Sharing advice
and coping strategies provide members with an opportunity to
build up empathy and express agency.
Co-leading did not always involve teaching as often participants
in the Social Club shared items or anecdotes based on what others
shared. During the focus group, Francine changed the discussion
topic when she shared some of her stufed animals with the group.
The topic change caused Beth, despite forgetting some words, to
talk about her stufed animal: "It’s just a little ball like this (makes a
circle with her hands) and it’s, it looks like a dog". By switching to a
topic that interested Beth, Francine provided Beth an opportunity
to share. Then, Beth’s use of body language helped her illustrate
what her words could not communicate to the group. By directing
the conversation, participants can then express their personhood
by sharing items or stories that are important to them.
4.2.4 Using the Program to Develop Lasting Interdependencies. How-
ever, participants acting as teachers for more personal topics helped
create a more lasting connection between staf and participants, for
instance, when signifcant but commonly experienced life changes
occurred in either a staf member’s or participant’s life. For Polly,
participants remember her being pregnant when they last met
in person. When they see her now in their virtual sessions, even
though she has had the baby, they ask for updates:
"They’re like, ’oh my gosh Polly did you have your baby yet?’ And
they always ask about my son and it’s just uber cute. And every time
I tell a story, with things that I’m learning and motherhood, they’re
like ’oh yeah we remember that.’" -Polly
The mutual experience shared between Polly and the partici-
pants created an opportunity to exchange knowledge about raising
children. Yet, instead of adding to the organization’s knowledge, the
information provides personal gain and a tool for connectedness.
Furthermore, the process of raising a child is ongoing, encouraging
a lasting connection.
Lasting connections also manifested in other ways. For Theresa,
during her programs, they would often have participants pot plants
to take home:
"[Participants can] look at that plant, the next day or the next week
and say ’Oh, we went to the botanic garden. Look at this plant is still
doing great.’ Well, that’s another thing! I have some people who come
on the [Zoom] call who say, "remember, we potted these last year?"
and I’m like "holy Christmas you still have that?" -Theresa
“Just like meeting in person” - Examination of interdependencies in dementia-friendly virtual activities.
The physical presence of the plant serves as, and supports, a
lasting memory of the interaction between the staf and the par-
ticipants. Keeping the plant keeps the participant’s connection to
Theresa and her organization intact. Seeing the plant again in the
participant’s home via her online program also reafrms Theresa’s
work. The strategies supporting social interdependencies seemed
to add to the amount of fulfllment achieved at the social level.
4.3 Opportunity for convergence
The opportunity for convergence is the intervention that allows
organizational interdependencies to intermingle with social interde-
pendencies. This convergence allows for the networks to infuence
one another. Programs can be designed to encourage participants to
be interdependently supportive (organizational infuencing social),
or participants who feel supported by the program invite more
people to join, increasing the organization’s outreach (social level
infuencing the organizational level). Overall, the method of de-
livery can afect the organizational by determining the program
design and the social by determining how people will interact.
While many social organizations are primed to be pillars of
their communities due to their location or prominence of their
building, paradoxically, they struggle to break through to the social
level. Despite obstacles, museums such as Kevin’s and Quentin’s
have used their prime physical space to have public exhibitions of
community members’ artworks alongside famous artists’ artworks:
"It’s just a sweet moment at the museum for building awareness
[. . . ] I think that the casual museum visitor really enjoys seeing the
community represented in that way because their expectation [of
coming] into our big marble building is to see the Van Gogh or the
Monet ... so to see beautiful clay works made by veterans or cognitively
challenged adults or people in recovery that is a really nice meaningful
thing for the casual visitor too" -Quentin
For Quentin, such displays of the community members’ art were
a way to bring the community together within their space and
elevate the community’s image of itself. Thus, making the organi-
zation’s space more accessible by allowing it to be an extension
of the community. This displays how the social interdependencies,
such as building that safe atmosphere, can infuence organizational
interdependencies and help expand an organization’s network.
As Gwen put it, "the goal [of the programs]... is to encourage this
audience of people to see the museum as a social setting and a safe
space for them to enjoy their time with [others]." By opening up
their space to the community, the organizational interdependen-
cies, those that have gone into the development and design of the
organization’s physical space, infuence the social interdependen-
cies by revealing that the space is safe, supportive, and wants to
create lasting connections with the community.
In some instances, developing a specifc program was unneces-
sary as the building’s atmosphere lent itself to informal gatherings.
Barb’s two-hour social groups for only those living with dementia
spawned an informal social group of carers since the organization’s
physical space aforded them privacy and opportunity for social
engagement. Such informal programs proved to be as benefcial for
informal carers as the organization’s more formal programs:
ASSETS ’22, October 23–26, 2022, Athens, Greece
Christine: In the Atrium we had those two hours that
we felt that we could share. Or, I have a conversation
with somebody else besides [Chris].
Dana: Yea it was a very relaxed atmosphere [...] It was
like sharing with a friend.
Dana and Christine expressed that attending informal social
gatherings aforded them similar benefts to formal programs and
created a safe atmosphere to develop lasting connections. Thus, how
the two networks converge is not rigidly bound to the organization
designing a program but relies more on the organization and social
context by which informal engagements can happen.
With the shift to online, carers felt that such informal activities
were not supported. This lack of support occurred since coordina-
tors were pressured to adapt online hastily and could not consider
all ramifcations. Thus, when one of the Social Club members passed
away, Christine and Chris were afected. "I felt alienated cuz I didn’t
know what happened to [the member. . . ] it was such a shock [. . . ] I felt
like I couldn’t even send [the spouse of the deceased member] a [card]
I felt sad, you know." While organizations are trying to respect their
member’s privacy by not sharing personal information, they were
the gatekeeper of this social connection and inhibited Christine
from informally reaching out to share her condolences. Therefore,
accommodating participants online still needs more consideration.
However, with in-person programs, when bringing participants
to the organization’s space was not feasible, social organizations
would host programs locally within a targeted community. For ex-
ample, Joan’s organization, pre-pandemic, held drawing workshops
at various community centers, and during the pandemic, Rachel’s
organization continued to manage and devise programming for
diferent community gardens. Rachel’s organization also supported
the shift to online by providing teleconferencing social spaces for
those living with dementia. By using these adapted methods of de-
livery, organizations can extend their social network through their
programs. "The community garden is so nice. It’s not ghettoizing old
people it’s having everybody there". As Rachel points out, when or-
ganizations make changes to the method of delivery, this can allow
the organization to diversify its participant reach. The larger the
spectrum of participants an organization can reach, the more signif-
icantly it could increase the depth of the social interdependencies
they infuence. Therefore, deliberately designing the opportunities
for convergence efectively to better support the organizational and
social levels in expanding their infuence is crucial for moving pro-
gramming into a realm of developing inclusive programs and not
just those for specifc accessibility needs, such as dementia-specifc
programming.
5 DISCUSSION
Unlike in-person/onsite programming, virtual/ofsite programming
for PLWD has only been around for many organizations since
the start of the COVID pandemic. Thus, the advantages that such
programming can provide require further exploration. Our fnd-
ings examined the program development and execution of a small
subset of organizations, leading us to expand Bennett et al.’s [7])
interdependency framework to include the relationships between
organizations in addition to the social interactions happening be-
tween individuals. Additionally, when organizational and social
ASSETS ’22, October 23–26, 2022, Athens, Greece Czech, et al.

interdependencies converge, they infuence how the other will react Table 4: Checklist for designing virtual spaces for PLWD.
and adapt (Fig. 3). As this infuence increases, the outreach of these
interdependencies expands, which could allow for more inclusivity. Action to be Done Accomplishing this Action
Organizations design programs to be socially inclusive of those Participant Share Participants engage with the
living with dementia (organizational infuencing social) and a di- their Individual space/objects around them and
verse range of participants feel invited to join in (social infuencing Sensory Experience their senses
organization). Thus, we found that changing the opportunity for Objects and/or Objects are transported between
convergence afects cross-infuence between the two interdepen- Technology locations creating a tangible
dencies. Improving interdependencies potentially can increase the Connect Diferent memory or potentially
social prescriptive benefts of such programs for PLWD. Locations asynchronous technologies create a
Consequently, the design and application of technology afects link
the amount of cross-infuence of the interdependencies. Most blended Social Agendas that Focus on empowering PLWD
programming currently involves thinking of virtual as a replace- Appropriately during virtual and support bonding
ment or equivalent to in-person. We argue that future programming Match the and inclusion of informal carers
for PLWD would beneft from complementary virtual and in-person Locations when in-person
activities since this is a promising method for building and sustain-
ing interdependencies. We also believe that understanding how
virtual spaces function is crucial to design virtual programs that
5.1 Designing for Blended Programming for
are more appealing to PLWD beyond when circumstance necessity
their use. In Tables 3 and 4, we provide a checklist for coordinators People Living with Dementia
and technologists to assist them with the design of a program’s Programs Should be Designed to Highlight the Strengths of Where
structure or technologies that will support social programs. They Will Occur: Blended programming often means that the con-
tent in person is as close as possible to the online content. This
model for blended programming is frequently applied to educational
Table 3: Checklist for designing blended programs. settings as it is successful at conveying information to students
[54]. However, students in fully virtual courses gain few social
benefts [86]. Thus, current, readily available technologies cannot
Action to be Questions to Keep Accomplishing
allow either type of site to imitate the other, nor is it efective to
Done in Mind this Action
try such imitation [38]. Since the primary purpose of the programs
Highlight location is the program - create greater
examined in this paper is providing social engagement to PLWD,
strengths in-person/onsite? social engagement
the application of blended programming in this way is likely to be
- make it a sensory
inappropriate.
experience
- raise awareness • The strengths of in-person are that they encourage greater
about the social engagement, sensory experience, and awareness of
organization by the organization within the community. In-person programs
bringing people in thus allow for social interdependencies to be infuenced by
or virtual/online? - personalize organizational interdependencies. For example, not having
- customize the ability to have informal meetings with other carers in
- and localize the organization’s atrium made participants feel alienated
content and socially disconnected when another participant passed
Complementary does the program - participants away.
Program incorporate should not be • In comparison, virtual programs require an organization to
continuous use of constrained by personalize, customize, and localize content for participants.
in-person and mobility, distance, These adjustments mean social interdependencies have more
virtual? or a lack of access infuence over the organizational interdependencies as they
to technology cause organizations to cater to their participants’ needs. For
- program ofering instance, when organizations hire certain staf or approach
and accessibility specifc organizations to form partnerships, their motivation
should be reliable comes from the need to serve their community.
is there an - celebrate
overarching repetition by While Boyd et al. [8] suggested that individuals’ roles when
program narrative? building it into the jointly using assistive technology need fexibility, our work suggests
program’s that technology should also have role fexibility. Preparation of
narrative technology to be fexible would allow program coordinators to
quickly adapt a participant to, for instance, a suitable asynchronous
activity when there are connection issues or switching from in-
person to virtual should another pandemic occur.
“Just like meeting in person” - Examination of interdependencies in dementia-friendly virtual activities.
Figure 3: [Left] We represent organizational interdependencies on top of our diagram since they determine whether a program
even exists. The opportunity for interdependency networks to converge is the method by which a program is delivered. [Middle]
When an organization’s programs to cater to a community’s needs, the infuence shifts so that the organizational is more
infuenced by the social. [Right] Ideally, the amount of infuence should be balanced, helping programs be more socially
inclusive.
Programs Should Consist of Complementary Components: The
complementary design of virtual and in-person programs will allow
more fexibility for engagement since some participants will only
be able to attend in-person or virtually. Keeping the programs
complementary then permits those who can attend in-person and
virtually to feel encouraged to attend without potentially getting
bored with repetitive content [58].
• Overlap allows participants to gain access to the content
without being constrained by mobility, distance, or a lack of
access to technology issues. As Francis pointed out, virtual
participation will not be accessible to everyone, but online
adaptations, for those that could use them, mediated more
social interactions by giving participants the ability to use
and express with body language [40, 44, 49, 55, 62, 76]. Body
language allows a participant to demonstrate a forgotten
word or phrase to communicate and remain socially active.
Other participants can also show support and interdepen-
dency when they assist with word or phrase recall.
• A complementary program builds opportunities for repeti-
tion, celebrating the individual’s life story instead of focusing
on repetition as a stigmatized symptom of dementia [84].
Furthermore, having some repeat in the content creates fa-
miliarity, which could inspire recollection of new or diferent
stories from participants [22, 40, 47, 70, 84]. Much like when
the one coordinator was pregnant, this encouraged partici-
pants to ask for updates and share or recollect stories related
to whatever stage of pregnancy she was in.
• Complementary programs should build upon a program’s
overarching narrative. Always ofering both types of pro-
grams will allow for stability of program ofering and acces-
sibility. Participants that attend both programs gain from
seeing familiar faces and build upon those social connections
while also gaining new knowledge – important for cogni-
tive stimulation and slowing the progression of dementia
[1, 6, 19, 20, 42, 51, 81, 89]. Technology could further assist
with reinforcing social ties between virtual and in-person
attendees through the design of an asynchronous platform
that congregates key points brought up in the programs.
ASSETS ’22, October 23–26, 2022, Athens, Greece
5.2 Design of Virtual Spaces for Programs for
People Living with Dementia
Virtual spaces are an organization’s online space or platform for
delivering programming. Virtual spaces allow the organization to
enter the participant’s home, which allows the participant to con-
trol the environment [25, 60] while remaining in a location that
is familiar and presumably considered a safe space for them. An
instant beneft of virtual programs is that concerns about preparing
the in-person atmosphere to be primed as a safe space are unneces-
sary. The organizations we interviewed used video conferencing
for their virtual spaces. Participants Should Share the Sensory Expe-
rience of their Individual Environments: A major issue with virtual
spaces is the lack of sensory engagement. In-person, coordinators
could easily create and share sensory experiences by encouraging
participants to describe what they were sensing or by engaging in
tangible activities such as art-making. As other HCI researchers
have shown, jointly sharing multi-sensory and tangible interactions
can add to the quality of engagement for those living with dementia
[9, 30, 41, 49, 70, 73, 81]. This joint engagement is lost in virtual
spaces since each person’s individual space is diferent.
• Coordinators can embrace location diferences by asking
participants to share and describe personal items to create
unique sensory experiences. Additionally, by having to de-
scribe their items, participants take on the role of program
leader since no one else can smell or physically touch the
item they are sharing.
Objects and/or Technology Can Be Used to Create Links Between
Diferent Program Locations: Typically, when participants enter an
organization’s space, they add value to objects by sharing their sto-
ries with staf. Although this was not the coordinator’s intention,
participants shared that the plants they had grown at the organiza-
tion were still alive and well in their homes in the virtual program.
This sharing served to re-link the plant back to the organization. In
contrast, a beneft of the virtual context is that staf can instead add
value to participants’ objects by linking them to their organization.
• Bonding an interaction through objects, when applied to
a blended program, can be built over time, potentially de-
veloping associations across virtual and in-person contexts.
Such associations through diferent contexts and sensory
ASSETS ’22, October 23–26, 2022, Athens, Greece
stimuli can help with memory recall and communication
[18, 27, 74, 77]. As one coordinator alluded to, an initial as-
sociation could be created in the virtual space as it allows
organizations, such as museums, to share rarely viewed ob-
jects. Coordinators can weave a narrative between items in
museum storage with items participants have in their home
or recall using. A further association can occur during the on-
site program, as a coordinator can connect the items shared
virtually with those that are available in the organization’s
space. For example, after having participants play with the
organization’s scrubbing brush, in the next virtual program,
participants could share and compare the brushes they use.
Having a program fow in this way blends the organization’s space
with the participant’s space. This blending of spaces by researchers
is usually considered in terms of asynchronous interactions, such
as collaborative work [46] or synchronous activities, such as family
meals [4]. We propose developing this link between spaces over time
to help form a lasting bond between individual staf and participants.
This bond could help strengthen organizational interdependencies
by promoting the organization to understand the participants’ lives
and needs. It could also improve social interdependency by allowing
empathetic bonds to be created through developing a collective
narrative throughout the blended program [18, 47, 49, 85, 87].
The Program’s Social Agenda Should Match with Where the Pro-
gram will Occur: Finally, for the carers, programs provided virtually
allow them to have individual respite since they know their loved
ones will be engaged and enjoying themselves. While coordinators
see this as discouraging participation, the carer not being present
can allow the person with dementia to have greater ownership over
the group. By requiring carers to attend virtual programs, coordina-
tors could negatively afect the carers’ wellbeing and be taking away
some of the participants’ sense of agency [27, 30, 35, 45, 47, 53].
Additionally, since organizations act as gatekeepers, some carers’
agencies are also lost by not being provided an informal virtual
space to gather.
• As a blended program, coordinators can ensure that they
provide opportunities that allow participants to rely on each
other more during virtual. Then when in-person, they can
instead encourage bonding and inclusion of the carer. By de-
signing the program this way, it can encourage and enforce
social interdependencies between participants as each space
changes who is relied on for social satisfaction. Blended
programs then allow for fuctuations in the relationship dy-
namics between the carer and person living with dementia
[45]. Moreover, blended programs can change how organi-
zational interdependencies develop and support programs
based on what is best for the participants. Virtual spaces
can be more efective when designed explicitly as a part of a
blended program and not as a separate component.
5.3 Refections on Critical Realism,
Interdependencies, and Dementia
Programming
There are numerous eforts within HCI to promote empowerment
of those living with dementia through technology [16, 18–20, 25,
29, 49, 56, 85]. However, as Storer and Branham unpacked with
Czech, et al.
their examination of accessible computing [52], we believe that
unless approached diferently, the power dynamics and structures
underlying activities such as dementia programming will continue
to be present. Barros Pena et al. [5], for instance, used a life course
perspective to “challenge reductive understandings of technology
design.” Our research illustrates another method researchers can
apply to help change the existing narrative. Instead of using critical
realism to point out the existing power dynamics, we used it to fnd
interdependencies since interdependency as a concept promotes
mutual access[7]. We found that within organizational structures,
which are normally entrenched with power imbalances, there are
interdependencies at play that help create accessible programs
and applications of technology. We found that reinterpreting the
context to focus on interdependencies allowed us to consider how
that afects the design of technology—for instance, considering how
organizational shifts, like hiring or fring of staf, can afect what
technology is used and how it is used with participants.
By looking at social programming and not focusing on a specifc
location, our research also expanded the types of contextual set-
tings typically explored in dementia and HCI research. Surprisingly,
despite recruiting coordinators from several types of organizations,
there were only minor diferences in program presentation for those
that ofered virtual presentations. The zoo, botanical garden, art
museums, and history museums centered their program around
a PowerPoint presentation. Deviations did occur, such as the zoo
showing videos of the animals or the history museums using online
maps to show current locations. The major diferences occurred
based on geographical location, with the organizations we inter-
viewed in the US relying more on virtual programs while those in
the UK tended to try less technologically advanced options frst.
Since the frst author was already acquainted with volunteering vir-
tually, we chose to pursue examining that space. Virtual dementia
programming allowed us to think about how diferent coordinators
would approach a group of participants that already have some
familiarity with each other. For this reason, we felt that the group
interview of the carers and those living with dementia were useful
since they mimicked the atmosphere of a supportive social environ-
ment. This setup allowed us to see how a group of somewhat typical
attendees interacted with each other virtually when not attending a
program. By interviewing this way, we could also approach the one
group interview with a diferent context. The interview session was
then able to last well beyond what was expected since participants
were enjoying their time together and, even when given the option,
did not want to leave. Thus, we found value in recontextualizing
how we conducted our interview with PLWD.
6 LIMITATIONS AND FUTURE WORKS
For this study, the programs we looked at targeted middle to upper-
class Caucasian PLWD. Additionally, of the coordinators who of-
fered to be interviewed, 13 of 15 were female and Caucasian, with
no visible disabilities. Also, we only interviewed coordinators in the
US and UK. Therefore, this research does not consider the variety
of interdependencies that arise for those who do not ft this descrip-
tion or those locations. This means we did not explore the depth of
systemic racism that has led to the underdiagnosis of dementia in
“Just like meeting in person” - Examination of interdependencies in dementia-friendly virtual activities.
people of color nor how it afects who interacts with social organi-
zations [23, 68]. However, we still feel that our research provides a
baseline for the types of interdependencies that can occur in public
situations for those living with dementia and how technology can
support them. In the future, we hope to apply these baseline ideas
to work with more diverse community groups.
Our hope with this work is to move towards inclusivity; how-
ever, we realize that further studies on achieving that are necessary.
Even if our fndings applied to programming, this would not au-
tomatically make them dementia inclusive. The inability of our
fndings to be applied to any program (those not designed to be
dementia-friendly) was made more apparent to us when an informal
carer mentioned how her spouse withdraws from even close friends
and family. Therefore, study into mixed ability interactions and
interdependencies is required. We are currently researching tech-
nological interventions that we hope can be used during program
delivery that make it possible for organizations to design programs
that support social interdependencies and infuence organizational
interdependencies.
Finally, while we did observe did observe several of the common
virtual communication issues (i.e., latency, eye contact, and issues
with internet) [50, 71], in the context of PLWD, these were unique
due to the cost associated with conversation repair [64]. Similar
issues were explored in research conducted by Neate et al. [61]
with people living with aphasia. For instance, latency issues could
cause participants to talk over each other causing confusion and
various participants to lose track of the conversation. However, in
this study we did not look at this issue in close detail, thus further
exploration into conversation repair is needed.
7 CONCLUSION
As a result of the COVID pandemic, many social organizations
were forced to adapt their dementia-friendly programs to virtual
platforms hastily. This shift allowed these organizations to see var-
ious benefts of online and most of them plan to continue virtual
programming to some extent in the future. However, the quick
development of virtual programs means there is still a lack of un-
derstanding of how to design programs efectively for PLWD. To
understand this, we chose to examine the interdependencies within
dementia-friendly social programs. Through our examination, we
recognized that organizational interdependencies play a key role
in infuencing the social interdependencies which occur between
individuals. We also found that cross-infuence between organi-
zational and social interdependencies happens when there is an
opportunity for convergence. This convergence is linked to the
method of program delivery, and thus, technology could act as an
intervention to create such an opportunity. However, programs
developed to be completely in-person or online can create some
alienating efects for some participants both with and without de-
mentia which negatively afects social interdependencies. There-
fore, programs designed to be delivered in a hybrid fashion with
overlapping and complementary content are a feasible way of de-
veloping dementia-inclusive programs. Such a blended approach
also encourages the building and reinforcing of interdependencies
over time. Strengthening interdependencies could help expand a
social programs’ outreach, allowing them to be more inclusive.
ASSETS ’22, October 23–26, 2022, Athens, Greece
REFERENCES
[1] Elisa Aguirre, Robert T. Woods, Aimee Spector, and Martin Orrell. 2013. Cognitive
stimulation for dementia: A systematic review of the evidence of efectiveness
from randomised controlled trials. Ageing Research Reviews 12, 1 (2013), 253–262.
https://doi.org/10.1016/j.arr.2012.07.001
[2] Keith Baker and Adele Irving. 2015. Co-producing Approaches to the Manage-
ment of Dementia through Social Prescribing. Social Policy and Administration
50, 3 (16 March 2015), 379–397. https://doi.org/10.1111/spol.12127
[3] Mark S. Baldwin, Sen H. Hirano, Jennifer Mankof, and Gillian R. Hayes. 2019.
Design in the Public Square: Supporting Assistive Technology Design Through
Public Mixed-Ability Cooperation. Proc. ACM Hum.-Comput. Interact. 3, CSCW,
Article 155 (Nov. 2019), 22 pages. https://doi.org/10.1145/3359257
[4] Pollie Barden, Rob Comber, David Green, Daniel Jackson, Cassim Ladha, Tom
Bartindale, Nick Bryan-Kinns, Tony Stockman, and Patrick Olivier. 2012. Telem-
atic Dinner Party: Designing for Togetherness through Play and Performance.
In Proceedings of the Designing Interactive Systems Conference (Newcastle Upon
Tyne, United Kingdom) (DIS ’12). Association for Computing Machinery, New
York, NY, USA, 38–47. https://doi.org/10.1145/2317956.2317964
[5] Belén Barros Pena, Rachel E Clarke, Lars Erik Holmquist, and John Vines. 2021.
Circumspect Users: Older Adults as Critical Adopters and Resistors of Technology.
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York,
NY, USA, Article 84, 14 pages. https://doi.org/10.1145/3411764.3445128
[6] Mark R Beauchamp, D Ph, Serena Mccutcheon, B Sc, Oliver Harper, and B Sc. 2007.
Older Adults ’ Preferences for Exercising Alone Versus in Groups : Considering
Contextual Congruence. Annals of Behavioral Medicine 33, 2 (2007), 200–206.
[7] Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence
as a Frame for Assistive Technology Research and Design. In Proceedings of the
20th International ACM SIGACCESS Conference on Computers and Accessibility
(Galway, Ireland) (ASSETS ’18). Association for Computing Machinery, New York,
NY, USA, 161–173. https://doi.org/10.1145/3234695.3236348
[8] LouAnne E. Boyd, Kyle Rector, Halley Profta, Abigale J. Stangl, Annuska Zolyomi,
Shaun K. Kane, and Gillian R. Hayes. 2017. Understanding the Role Fluidity of
Stakeholders During Assistive Technology Research "In the Wild". In Proceedings
of the 2017 CHI Conference on Human Factors in Computing Systems (Denver,
Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY,
USA, 6147–6158. https://doi.org/10.1145/3025453.3025493
[9] Margot Brereton, Alessandro Soro, Kate Vaisutis, and Paul Roe. 2015. The Mes-
saging Kettle : Prototyping Connection over a Distance between Adult Children
and Older Parents. In CHI ’15: Proceedings of the 33rd Annual ACM Conference
on Human Factors in Computing Systems. Seoul, Republic of Korea, 713–716.
https://doi.org/10.1145/2702123.2702462
[10] Dawn Brooker. 2005. Dementia Care Mapping: A Review of the Research
Literature. The Gerontologist 45 (10 2005), 11–18. https://doi.org/10.
1093/geront/45.suppl_1.11 arXiv:https://academic.oup.com/gerontologist/article-
pdf/45/suppl_1/11/1470527/11.pdf
[11] Jessica Budgett, Anna Brown, Stephanie Daley, Thomas E Page, Sube Banerjee,
Gill Livingston, and Andrew Sommerlad. 2019. The social functioning in dementia
scale ( SF-DEM ): Exploratory factor analysis and psychometric properties in mild
, moderate , and severe dementia. Alzheimer’s & Dementia: Diagnosis, Assessment
& Disease Monitoring 11 (2019), 45–52. https://doi.org/10.1016/j.dadm.2018.11.001
[12] Sarah Bunt. 2016. Critical realism and grounded theory: Analysing the adoption
outcomes for disabled children using the retroduction framework. Qualitative
Social Work 17 (09 2016). https://doi.org/10.1177/1473325016664572
[13] Francesco Cafaro, Leilah Lyons, Joshua Radinsky, and Jessica Roberts. 2010. RFID
Localization for Tangible and Embodied Multi-User Interaction with Museum
Exhibits. In Proceedings of the 12th ACM International Conference Adjunct Papers on
Ubiquitous Computing - Adjunct (Copenhagen, Denmark) (UbiComp ’10 Adjunct).
Association for Computing Machinery, New York, NY, USA, 397–398. https:
//doi.org/10.1145/1864431.1864455
[14] Kathy Chamaz. 2017. Constructionism and the Grounded Theory Method. In
Handbook of Contructionist Research, James A. Holstein and Jaber F. Gubrium
(Eds.). The Guilford Press, New York, Chapter 20, 397–412. https://doi.org/10.
4135/9781526402196.n2
[15] K. Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through
Qualitative Analysis. SAGE Publications. https://books.google.co.uk/books?id=
2ThdBAAAQBAJ
[16] Gary Cheung and Kathryn Peri. 2020. Challenges to dementia care during COVID-
19: Innovations in remote delivery of group Cognitive Stimulation Therapy.
Aging and Mental Health 25, 6 (2020), 1–3. https://doi.org/10.1080/13607863.2020.
1789945
[17] Loraine Clarke, Eva Hornecker, and Ian Ruthven. 2021. Fighting Fires and
Powering Steam Locomotives: Distribution of Control and Its Role in Social
Interaction at Tangible Interactive Museum Exhibits. In Proceedings of the 2021
CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI
’21). Association for Computing Machinery, New York, NY, USA, Article 344,
17 pages. https://doi.org/10.1145/3411764.3445534
ASSETS ’22, October 23–26, 2022, Athens, Greece
[18] Elaine Czech, Mina Shibasaki, Keitaro Tsuchiya, Roshan L. Peiris, and Kouta
Minamizawa. 2020. Discovering Narratives: Multi-Sensory Approach Towards
Designing with People with Dementia. In Extended Abstracts of the 2020 CHI
Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI
EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https:
//doi.org/10.1145/3334480.3375209
[19] Jiamin Dai and Karyn Mofatt. 2020. Making Space for Social Sharing: Insights
from a Community-Based Social Group for People with Dementia. In Proceedings
of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu,
HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA,
1–13. https://doi.org/10.1145/3313831.3376133
[20] Jiamin Dai and Karyn Mofatt. 2021. Surfacing the Voices of People with Dementia:
Strategies for Efective Inclusion of Proxy Stakeholders in Qualitative Research.
Association for Computing Machinery, New York, NY, USA. https://doi.org/10.
1145/3411764.3445756
[21] Maitraye Das, Darren Gergle, and Anne Marie Piper. 2019. "It Doesn’t Win You
Friends": Understanding Accessibility in Collaborative Writing for People with
Vision Impairments. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 191
(Nov. 2019), 26 pages. https://doi.org/10.1145/3359293
[22] Sabeth Diks, Timothy Hendrik Coen Muyrers, Guangyu Chen, Tzu-Jou Huang,
Myrte Thoolen, and Rens Brankaert. 2021. CoasterChat: Exploring Digital Commu-
nication between People with Early Stage Dementia and Family Members Embedded
in a Daily Routine. Association for Computing Machinery, New York, NY, USA.
https://doi.org/10.1145/3411763.3451635
[23] Peggye Dilworth-Anderson and Brent E. Gibson. 2002. The cultural infuence of
values, norms, meanings, and perceptions in understanding dementia in ethnic
minorities. Alzheimer Disease & Associated Disorders 16 (2002). https://doi.org/
10.1097/00002093-200200002-00005
[24] Emma Dixon and Amanda Lazar. 2020. Approach Matters: Linking Practitioner
Approaches to Technology Design for People with Dementia. In Proceedings of
the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI,
USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–15.
https://doi.org/10.1145/3313831.3376432
[25] Emma Dixon, Anne Marie Piper, and Amanda Lazar. 2021. “Taking Care of
Myself as Long as I Can”: How People with Dementia Confgure Self-Management
Systems. Association for Computing Machinery, New York, NY, USA. https:
//doi.org/10.1145/3411764.3445225
[26] Chris Drinkwater, Josephine Wildman, and Suzanne Mofatt. 2019. So-
cial prescribing. BMJ 364 (2019). https://doi.org/10.1136/bmj.l1285
arXiv:https://www.bmj.com/content/364/bmj.l1285.full.pdf
[27] James Edmeads and Oussama Metatla. 2019. Designing for Reminiscence with
People with Dementia. In Extended Abstracts of the 2019 CHI Conference on Human
Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association
for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/
3290607.3313059
[28] Amber Fletcher. 2016. Applying critical realism in qualitative research: Method-
ology meets method. International Journal of Social Research Methodology 19 (02
2016), 1–14. https://doi.org/10.1080/13645579.2016.1144401
[29] Sarah Foley, John McCarthy, and Nadia Pantidi. 2019. The Struggle for Recog-
nition in Advanced Dementia: Implications for Experience-Centered Design.
ACM Trans. Comput.-Hum. Interact. 26, 6, Article 40 (Nov. 2019), 29 pages.
https://doi.org/10.1145/3359594
[30] Sarah Foley, Daniel Welsh, Nadia Pantidi, Kellie Morrissey, Tom Nappey, and
John McCarthy. 2019. Printer Pals: Experience-Centered Design to Support Agency
for People with Dementia. Association for Computing Machinery, New York, NY,
USA, 1–13. https://doi.org/10.1145/3290605.3300634
[31] Christopher Frauenberger. 2015. Disability and Technology: A Critical Realist
Perspective. In Proceedings of the 17th International ACM SIGACCESS Conference
on Computers & Accessibility (Lisbon, Portugal) (ASSETS ’15). Association for
Computing Machinery, New York, NY, USA, 89–96. https://doi.org/10.1145/
2700648.2809851
[32] Sophie N. Gaber, Louise Nygård, Anna Brorsson, Anders Kottorp, Georgina
Charlesworth, Sarah Wallcook, and Camilla Malinowsky. 2020. Social participa-
tion in relation to technology use and social Deprivation: A mixed METHODS
study among older people with and Without Dementia. International Jour-
nal of Environmental Research and Public Health 17, 11 (2020), 4022. https:
//doi.org/10.3390/ijerph17114022
[33] Erving Gofman. 1959. The presentation of self in everyday life. Bantam Doubleday
Dell Publishing Group.
[34] Jenni Greig, Sabih-Ur Rehman, Anwaar Ul-Haq, Greg Dresser, and Oliver K.
Burmeister. 2019. Transforming Ageing in Community: Addressing Global Age-
ing Vulnerabilities through Smart Communities. In Proceedings of the 9th Inter-
national Conference on Communities & Technologies - Transforming Communities
(Vienna, Austria) (C&T ’19). Association for Computing Machinery, New York,
NY, USA, 228–238. https://doi.org/10.1145/3328320.3328380
[35] Connie Guan, Anya Bouzida, Ramzy M. Oncy-avila, Sanika Moharana, and Lau-
rel D. Riek. 2021. Taking an (Embodied) Cue From Community Health: Designing
Dementia Caregiver Support Technology to Advance Health Equity. Association
Czech, et al.
for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.
3445559
[36] James Hodge, Madeline Balaam, Sandra Hastings, and Kellie Morrissey. 2018.
Exploring the Design of Tailored Virtual Reality Experiences for People with De-
mentia. In Proceedings of the 2018 CHI Conference on Human Factors in Computing
Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery,
New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3174088
[37] James Hodge, Kyle Montague, Sandra Hastings, and Kellie Morrissey. 2019. Ex-
ploring Media Capture of Meaningful Experiences to Support Families Living with
Dementia. Association for Computing Machinery, New York, NY, USA, 1–14.
https://doi.org/10.1145/3290605.3300653
[38] Jim Hollan and Scott Stornetta. 1992. Beyond Being There. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems (Monterey, California,
USA) (CHI ’92). Association for Computing Machinery, New York, NY, USA,
119–125. https://doi.org/10.1145/142750.142769
[39] Leona Holloway, Kim Marriott, Matthew Butler, and Alan Borning. 2019. Making
Sense of Art: Access for Gallery Visitors with Vision Impairments. In Proceedings
of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19).
Association for Computing Machinery, New York, NY, USA, 12 pages. https:
//doi.org/10.1145/3290605.3300250
[40] Maarten Houben, Rens Brankaert, Saskia Bakker, Gail Kenning, Inge Bongers,
and Berry Eggen. 2020. The Role of Everyday Sounds in Advanced Dementia
Care. In Proceedings of the 2020 CHI Conference on Human Factors in Computing
Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery,
New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376577
[41] Agnes Houston and Julie Christie. 2018. Talking Sense: Living with sensory changes
and dementia (1st ed.). HammondCare, Sydney, Australia. 1–63 pages.
[42] J. D. Huntley, R. L. Gould, K. Liu, M. Smith, and R. J. Howard. 2015. Do cognitive
interventions improve general cognition in dementia? A meta-analysis and meta-
regression. BMJ Open 5, 4 (2015). https://doi.org/10.1136/bmjopen-2014-005247
[43] Innovations in Dementia. 2021. DEEP: https://www.dementiavoices.org.uk/. Last
accessed September 2021.
[44] Elizabeth Kelson, Alison Phinney, and Glen Lowry. 2017. Social citizenship,
public art and dementia: Walking the urban waterfront with Paul’s Club. Cogent
Arts & Humanities 4 (08 2017). https://doi.org/10.1080/23311983.2017.1354527
[45] Debbie Kinsey, Iain Lang, Noreen Orr, Rob Anderson, and Daisy Parker. 2021. The
impact of including carers in museum programmes for people with dementia: a
realist review. Arts & Health 13, 1 (2021), 1–19. https://doi.org/10.1080/17533015.
2019.1700536 PMID: 33538657.
[46] Aniket Kittur, Bryant Lee, and Robert E. Kraut. 2009. Coordination in Collective
Intelligence: The Role of Team Structure and Task Interdependence. In Proceedings
of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA,
USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA,
1495–1504. https://doi.org/10.1145/1518701.1518928
[47] Tom Kitwood. 1993. Towards a Theory of Dementia Care: The Interpersonal
Process. Ageing and Society 13, 1 (1993), 51–67. https://doi.org/10.1017/
S0144686X00000647
[48] Amanda Lazar and Emma E. Dixon. 2019. Safe Enough to Share: Setting the
Dementia Agenda Online. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article
85 (Nov. 2019), 23 pages. https://doi.org/10.1145/3359187
[49] Amanda Lazar, Caroline Edasis, and Anne Marie Piper. 2017. A Critical Lens
on Dementia and Design in HCI. In Proceedings of the 2017 CHI Conference
on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17).
Association for Computing Machinery, New York, NY, USA, 2175–2188. https:
//doi.org/10.1145/3025453.3025522
[50] Minha Lee, Wonyoung Park, Sunok Lee, and Sangsu Lee. 2022. Distracting
Moments in Videoconferencing: A Look Back at the Pandemic Period. In CHI
Conference on Human Factors in Computing Systems (New Orleans, LA, USA)
(CHI ’22). Association for Computing Machinery, New York, NY, USA, Article
141, 21 pages. https://doi.org/10.1145/3491102.3517545
[51] Else Lykkeslet, Eva Gjengedal, Torill Skrondal, and May-britt Storjord. 2014.
Sensory stimulation A way of creating mutual relations in dementia care. Inter-
national Journal of Qualitative Studies on Health and Well-being 1 (2014), 1–11.
[52] Kevin M. Storer and Stacy M. Branham. 2021. Deinstitutionalizing Independence:
Discourses of Disability and Housing in Accessible Computing. Association for
Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3441852.
3471213
[53] Galina Madjarof and Helena M Mentis. 2017. Narratives of Older Adults with
Mild Cognitive Impairment and Their Caregivers. In ASSETS. Baltimore, MD,
USA, 140–149.
[54] Malissa Maria Mahmud, Marisha Barth Ubrani, and Wong Shiau Foong. 2020. A
Meta-Analysis of Blended Learning Trends. In Proceedings of the 2020 11th Inter-
national Conference on E-Education, E-Business, E-Management, and E-Learning
(Osaka, Japan) (IC4E 2020). Association for Computing Machinery, New York, NY,
USA, 30–36. https://doi.org/10.1145/3377571.3379439
[55] Jane McKeown, Amanda Clarke, Christine Ingleton, Tony Ryan, and Julie Repper.
2010. The use of life story work with people with dementia to enhance person-
centred care. International Journal of Older People Nursing 5, 2 (2010), 148–158.
“Just like meeting in person” - Examination of interdependencies in dementia-friendly virtual activities.
https://doi.org/10.1111/j.1748-3743.2010.00219.x
[56] Kellie Morrissey, John McCarthy, and Nadia Pantidi. 2017. The Value of
Experience-Centred Design Approaches in Dementia Research Contexts. In Pro-
ceedings of the 2017 CHI Conference on Human Factors in Computing Systems
(Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New
York, NY, USA, 1326–1338. https://doi.org/10.1145/3025453.3025527
[57] Thomas Morton, Dawn Brooker, G. Wong, Teresa Atkinson, and Shirley Evans.
2021. Keeping community groups and activities going - Sustainable Community
Interventions for people afected by Dementia: Recommendations for practice from
the SCI-Dem Project. University of Worcester. https://eprints.worc.ac.uk/10225/
7/SCI-Dem%20booklet%20PRACTICE%20%28v4.2%29.pdf
[58] Wendy Moyle, Cindy Jones, Toni Dwan, and Tanya Petrovich. 2018. E ectiveness
of a Virtual Reality Forest on People With Dementia : A Mixed Methods Pilot
Study. The Gerontologist 58, 3 (2018), 478–487.
[59] Diego Muñoz, Stu Favilla, Sonja Pedell, Andrew Murphy, Jeanie Beh, and Tanya
Petrovich. 2021. Evaluating an App to Promote a Better Visit Through Shared
Activities for People Living with Dementia and Their Families. Association for
Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.
3445764
[60] Elizabeth D. Mynatt, Kayci Vickers, Salimah LaForce, Sarah K. Farmer, Jeremy M.
Johnson, Matthew Doiron, Aparna Ramesh, Walter Bradley Fain, Tamara Zubatiy,
and Amy D. Rodriguez. 2022. Pivoting an MCI Empowerment Program to Online
Engagement. Proc. ACM Hum.-Comput. Interact. 6, GROUP, Article 32 (jan 2022),
26 pages. https://doi.org/10.1145/3492851
[61] Timothy Neate, Vasiliki Kladouchou, Stephanie Wilson, and Shehzmani Shams.
2022. “Just Not Together”: The Experience of Videoconferencing for People
with Aphasia during the Covid-19 Pandemic. In Proceedings of the 2022 CHI
Conference on Human Factors in Computing Systems (New Orleans, LA, USA)
(CHI ’22). Association for Computing Machinery, New York, NY, USA, Article
606, 16 pages. https://doi.org/10.1145/3491102.3502017
[62] Mmachi God’sglory Obiorah, Anne Marie Marie Piper, and Michael Horn. 2021.
Designing AACs for People with Aphasia Dining in Restaurants. Association for
Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.
3445280
[63] Carolyn Oliver. 2011. Critical Realist Grounded Theory: A New Approach for
Social Work Research. British Journal of Social Work 42 (03 2011). https://doi.
org/10.1093/bjsw/bcr064
[64] J. B. Orange, Rosemary B. Lubinski, and D. Jefery Higginbotham. 1996. Con-
versational repair by individuals with dementia of the alzheimer’s type. Journal
of Speech, Language, and Hearing Research 39, 4 (Aug 1996), 881–895. https:
//doi.org/10.1044/jshr.3904.881
[65] World Health Organization. 2007. Global age-friendly cities : a guide. Re-
trieved April 18, 2021 from http://apps.who.int/iris/bitstream/handle/10665/
43755/9789241547307_eng.pdf;sequence=1
[66] Andrea Grimes Parker. 2013. Designing for Health Activism. Interactions 20, 2
(March 2013), 22–25. https://doi.org/10.1145/2427076.2427082
[67] Shibley Rahman and Kate Swafer. 2018. Assets-based approaches and dementia-
friendly communities. Dementia 17, 2 (2018), 131–137. https://doi.org/10.1177/
1471301217751533
[68] Jemma L. Regan. 2013. Redefning dementia care barriers for ethnic minorities:
The religion–culture distinction. Mental Health, Religion & Culture 17, 4 (2013),
345–353. https://doi.org/10.1080/13674676.2013.805404
[69] Olivia K. Richards, Gabriela Marcu, and Robin N. Brewer. 2021. Hugs, Bible
Study, and Speakeasies: Designing for Older Adults’ Multimodal Connectedness.
In Designing Interactive Systems Conference 2021 (Virtual Event, USA) (DIS ’21).
Association for Computing Machinery, New York, NY, USA, 815–831. https:
//doi.org/10.1145/3461778.3462075
[70] Helma Van Rijn, Joost Van Hoof, and Pieter Jan Stappers. 2010. Designing Leisure
Products for People With Dementia : Developing “ the Chitchatters ” Game.
American Journal of Alzheimer’s Disease & Other Dementias 25, 1 (2010), 74–89.
[71] E. Sean Rintel. 2010. Conversational Management of Network Trouble Pertur-
bations in Personal Videoconferencing. In Proceedings of the 22nd Conference of
the Computer-Human Interaction Special Interest Group of Australia on Computer-
Human Interaction (Brisbane, Australia) (OZCHI ’10). Association for Comput-
ing Machinery, New York, NY, USA, 304–311. https://doi.org/10.1145/1952222.
1952288
[72] Karin Ryding. 2020. The Silent Conversation: Designing for Introspection and
Social Play in Art Museums. In Proceedings of the 2020 CHI Conference on Human
Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for
Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/
3313831.3376357
[73] Corina Sas. 2018. Exploring Self-Defning Memories in Old Age and Their Digital
Cues. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong
Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY,
USA, 149–161. https://doi.org/10.1145/3196709.3196767
[74] Corina Sas, Nigel Davies, Sarah Clinch, Peter Shaw, Mateusz Mikusz, Madeleine
Steeds, and Lukas Nohrer. 2020. Supporting Stimulation Needs in Dementia Care
through Wall-Sized Displays. Association for Computing Machinery, New York,
ASSETS ’22, October 23–26, 2022, Athens, Greece
NY, USA, 1–16. https://doi.org/10.1145/3313831.3376361
[75] Kristen Shinohara, Nayeri Jacobo, Wanda Pratt, and Jacob O. Wobbrock. 2020.
Design for Social Accessibility Method Cards: Engaging Users and Refecting on
Social Scenarios for Accessible Design. ACM Trans. Access. Comput. 12, 4, Article
17 (Jan. 2020), 33 pages. https://doi.org/10.1145/3369903
[76] Frances Sin, Sophie Berger, Ig-Jae Kim, and Dongwook Yoon. 2021. Digital Social
Interaction in Older Adults During the COVID-19 Pandemic. Proc. ACM Hum.-
Comput. Interact. 5, CSCW2, Article 380 (oct 2021), 20 pages. https://doi.org/10.
1145/3479524
[77] Steven M. Smith, Arthur Glenberg, and Robert A. Bjork. 1978. Environmental
context and human memory. Memory & Cognition 6, 4 (1978), 342–353. https:
//doi.org/10.3758/BF03197465
[78] Alzheimer’s Society. 2020. The impact of COVID-19 on People Afected by Dementia.
Technical Report.
[79] Kate Swafer. 2014. Dementia: Stigma, Language, and Dementia-friendly. De-
mentia 13, 6 (2014), 709–716. https://doi.org/10.1177/1471301214548143
[80] Catherine V Talbot and Pamela Briggs. 2021. The use of digital technologies
by people with mild to moderate dementia during the COVID-19 pandemic: A
positive technology perspective. https://doi.org/10.31234/osf.io/d4qv8
[81] Linda Thomson, Bridget Lockyer, Paul Camic, and Helen Chatterjee. 2017. Efects
of a museum-based social prescription intervention on quantitative measures
of psychological wellbeing in older adults. Perspectives in Public Health 138 (11
2017), 175791391773756. https://doi.org/10.1177/1757913917737563
[82] Milka Trajkova, A’aeshah Alhakamy, Francesco Cafaro, Rashmi Mallappa, and
Sreekanth R. Kankara. 2020. Move Your Body: Engaging Museum Visitors with
Human-Data Interaction. In Proceedings of the 2020 CHI Conference on Human
Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for
Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/
3313831.3376186
[83] David Unbehaun, Konstantin Aal, Daryoush Daniel Vaziri, Peter David Tolmie,
Rainer Wieching, David Randall, and Volker Wulf. 2020. Social Technology
Appropriation in Dementia: Investigating the Role of Caregivers in Engaging
People with Dementia with a Videogame-Based Training System. In Proceedings
of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu,
HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA,
1–15. https://doi.org/10.1145/3313831.3376648
[84] Feliciano Villar, Rodrigo Serrat, and Stephany Bravo-Segal. 2019. Giving them
a voice: Challenges to narrative agency in people with dementia. Geriatrics
(Switzerland) 4, 1 (2019). https://doi.org/10.3390/geriatrics4010020
[85] Jayne Wallace, Peter C. Wright, John McCarthy, David Philip Green, James
Thomas, and Patrick Olivier. 2013. A Design-Led Inquiry into Personhood in
Dementia. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems
(Paris, France) (CHI EA ’13). Association for Computing Machinery, New York,
NY, USA, 2883–2884. https://doi.org/10.1145/2468356.2479560
[86] Wei Wang, Lihuan Guo, Ling He, and Yenchun Jim Wu. 2019. Efects of social-
interactive engagement on the dropout ratio in online learning: insights from
MOOC. Behaviour & Information Technology 38, 6 (2019), 621–636. https:
//doi.org/10.1080/0144929X.2018.1549595
[87] Daniel Welsh, Kellie Morrissey, Sarah Foley, Roisin McNaney, Christos Salis,
John McCarthy, and John Vines. 2018. Ticket to Talk: Supporting Conversation
between Young People and People with Dementia through Digital Media. In
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
(Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New
York, NY, USA, 1–14. https://doi.org/10.1145/3173574.3173949
[88] Glen White, Jamie Simpson, Chiaki Gonda, Craig Ravesloot, and Zach Coble.
2010. Moving From Independence to Interdependence: A Conceptual Model
for Better Understanding Community Participation of Centers for Independent
Living Consumers. Journal of Disability Policy Studies 20 (03 2010), 233–240.
https://doi.org/10.1177/1044207309350561
[89] Bob Woods, Elisa Aguirre, Aimee E Spector, and Martin Orrell. 2012. Cog-
nitive stimulation to improve cognitive functioning in people with dementia.
Cochrane Database of Systematic Reviews (2012). https://doi.org/10.1002/14651858.
cd005562.pub2
[90] Özge Nilay Yalçın, Sylvain Moreno, and Steve DiPaola. 2020. Social Prescribing
Across the Lifespan with Virtual Humans. In Proceedings of the 20th ACM Inter-
national Conference on Intelligent Virtual Agents (Virtual Event, Scotland, UK)
(IVA ’20). Association for Computing Machinery, New York, NY, USA, Article 56,
3 pages. https://doi.org/10.1145/3383652.3423897
 Performing Qalitative Data Analysis as a Blind Researcher:
Challenges, Workarounds and Design Recommendations
O. Aishwarya
International Institute of Information Technology
Bangaluru, Karnataka, India
O.Aishwarya@iiitb.ac.in
ABSTRACT
Over the last 2 decades, the way in which disabled body minds are
regarded in research has evolved considerably. From an understand-
ing that disability research involves research or enquiry on or of
disabled people, there is now a shift in thinking that regards disabil-
ity research as research done by and with disabled people as well,
thus challenging traditional researcher-participant relationships.
This has lead to a larger number of people with disabilities aspiring
to enter academia. However, several barriers to full inclusion of
people with disabilities still remain. This paper talks about one
such barrier, that is, the lack of accessibility in software packages
used for qualitative data analysis. Through the author’s experi-
ences in hunting for accessible ways to perform qualitative data
analysis, the paper reviews existing software packages, discusses
possible workarounds, and considers the feature requirements for
an accessible qualitative data analysis tool.
CCS CONCEPTS
• Human-centered computing → Human computer interaction
(HCI); Accessibility.
KEYWORDS
Accessibility, Qualitative Data Analysis, Human Centered Design
ACM Reference Format:
O. Aishwarya. 2022. Performing Qualitative Data Analysis as a Blind Re-
searcher: Challenges, Workarounds and Design Recommendations. In The
24th International ACM SIGACCESS Conference on Computers and Accessibil-
ity (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY,
USA, 4 pages. https://doi.org/10.1145/3517428.3551356
1 INTRODUCTION
The WHO estimates suggest that the total global number of people
with disabilities has already surpassed one billion. About 15 per-
cent of the world’s population lives with some form of disability, of
whom 2-4 percent experience signifcant difculties in functioning.
A majority of these, around 80 percent, live in the global South.
Population ageing, rapid spread of chronic disabilities and a bet-
terment in the methodologies used to measure disability have all
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3551356
given rise to a considerable increase in disability prevalence around
the world [18]. Of the 285 million people in the world who are
visually impaired, 90 percent are in developing countries. While
39 million people are blind, 246 million have severe or moderate
visual impairments [15].
Over the last 2 decades, the way in which disabled body minds
are regarded in research has evolved considerably. From an under-
standing that disability research involves research or enquiry on or
of disabled people, there is now a shift in thinking that regards dis-
ability research as research done by and with disabled people as well,
thus challenging traditional researcher-participant relationships
[13]. This has been a signifcant paradigm shift. From regarding
people with disabilities as passive subjects to be researched on,
research methodology has evolved to a point where it recognises
people with disabilities as people with agency and Choice, capable
not only of being research subjects, but also of being the ones to
tell their own stories. Consequently, the number of people with
disabilities who regard academia as a possible inclusive space, and
research as a tool to create genuine positive societal changes, has
increased. There has been an increasing interest in participatory
and co-design methodologies, frst-person accounts of disability,
autoethnographies, etc [1, 3, 8, 9, 11, 19] Advocates of participatory
research emphasise the right of all people to be actively involved as
researchers in matters relevant to their own lives and communities
[17]. The United Nations Convention on the Rights of Persons with
Disabilities (UNCRPD), supports this principle, articulating that
people with disabilities have the right to participate in all levels of
society [16], which clearly includes disability research.
Several studies in recent years talk about the need for integrating
disabled scholars into disability studies research, and give concrete
ideas about diferent ways of doing so. Disabled researchers have
pointed out some barriers to there full and complete participation in
academia, such as ableism in the form of denial of accommodations,
the burden of always having to educate the academic community
around them, axis work always being considered the responsibility
of individual disabled people,infrastructural accessibilities etc [1, 2,
4–6, 10–12, 14].
This paper focuses on one such barrier holding disabled peo-
ple, particularly blind and visually impaired people, back from full
participation in academia: the inaccessibility of software used for
qualitative data analysis. By reccounting and analysing my own
experiences of trying to fnd a tool to analyse the qualitative data I
collect as an early career research scholar, I highlight the need for
greater accessibility in this area, discuss some alternative methods
that can be used to perform qualitative data analysis, and provide
design recommendations and feature requirements for an accessible
qualitative data analysis package.
ASSETS ’22, October 23–26, 2022, Athens, Greece
2 EXPERIMENTING WITH EXISTING
SOLUTIONS
Some of the popular existing software packages that enable re-
searchers to collect, clean, and analyse the qualitative data they
collect includeAtlas.Ti, MaxQDA, Nvivo and RQDA, among others.
However, all of them have signifcant accessibility issues. Some of
them have interfaces that are completely silent when accessed with
a screen reader, some have no accessible ways for adding or viewing
codes, and yet others have completely visual ways of displaying
codes and code categories.
Realizing that no ready-made solution exists to fullfll my needs,
I turned to word processors and other mainstream software pack-
ages which had already proven to be accessible, such as Microsoft
Word, Adobe Reader, and Microsoft Excel. Some of these tools were
moderately helpful for performing qualitative data analysis, but
some were completely unusable.
The frst software tool I tried to use for data analysis was Mi-
crosoft Word. I saved each interview transcript as a separate Word
fle, and then tried to use the “Comments” feature of word to add
each Code in. However, there were several problems with this
method: I had no way of easily moving from one assigned code
to another; highlighting text snippets was of no use because high-
lights are not detected by a screen reader; exporting my codes
and snippets to a new document for analysis could be done only
manually;double checking the codes I have assigned to a particular
snippet became difcult as typical screen leader behaviour while
dealing with comments is to only announce “has comment” and an
extra keystroke needs to be performed to read the comment itself..
Thus, I determined that using Microsoft Word as a qualitative data
analysis tool was not feasible, due to the nature of both the screen
reader and the software tool itself
Using Adobe Reader was equally unsuccessful. Although the
highlights feature on this software program is actually accessible
via the screen reader, the behaviour of the “comments” feature with
a screen reader was similar in many ways to my experiences with
Microsoft Word. Once again, I could not navigate easily between
the codes I assigned, nor export them to a new document for further
analysis.
3 POSSIBLE WORKAROUNDS
In this section, I would like to discuss some possible workarounds
that I have used in my process of qualitative data analysis., with
specifc emphasis on features that make these solutions ideal for
my needs.
Rather than transcribe audio data such as interviews into text
and then struggle to fnd a tool to help me analyse the text data
so collected, I now prefer to trim and clean up my audio record-
ings a little, and analyse them using an open source audio editing
tool called Audacity. My reasoning behind adopting such an un-
conventional method is that, as a screen reader user, I am always
listening to the interview, as even the transcripts are read out to
me using the screen reader. Moreover, accessible ways of coding
audio qualitative Data do exist. I typically listen to the audio twice
to familiarise myself with the content. I then go through the audio
once more, this time at twice the normal speed, pausing multiple
O. Aishwarya
times to select snippets and attach labels to the selected audio. Au-
dacity allows unlimited labels on an audio fle. I therefore add one
label per Code. Once I am done, I can export my labels to a new
text fle. Using keyboard shortcuts, I can attach and remove labels
easily, and also move from one label to the next. I then use the text
fle generated, to create code categories, choose likely quotations,
and perform thematic analysis. Once I got past the strangeness of
coding without a transcript, this method worked the best for me.
However, not all data collected during a study can be in audio
format. I do occasionally come across textual Data that needs to be
analysed. For this, I use several workarounds.
One such way is to use Microsoft Excel. First, I collect my text
data in Microsoft word. Second, I make sure that each codeable
segment, such as a sentence or a paragraph, is enclosed by blank
lines. Next, I create a new blank workbook or sheet in Microsoft
Excel and create my columns (which are typically ID, Data, and
Notes). I then copy my data from Microsoft word, and paste into the
column named data. My data is now ready for coding, with one cell
per text segment. I use the ID column to assign numbers, starting
with 1, for each text segment, which makes sorting and fltering
easier later on while performing analysis. I then read through all
of my data, cell by cell. To code a particular text segment, I create a
new column to the right of my data for each Code I assign. Now,
as I read, to mark a code as being present in my data, I put an ‘X’
in its column.’. As and when new codes occur to me, I create new
columns for them to the right of the data column. I can also have
primary and secondary themes/codes for my data. I can indicate
primary and secondary themes by using X for a primary code and Z
for a secondary code. I then use the “sort by” and “flter by” options
to view the diferent segments according to The codes assigned
to them. This negates the need to export coded data for further
analysis. This method is also not without drawbacks. When I have
a large amount of data to Code, and consequently have a large
number of codes as well, it becomes a little difcult to keep track
of all these diferent codes, as each code has a column of its own.
In such a case, I end up with so many columns that navigating
through the spreadsheet becomes difcult. One way of handling
extremely Virti or text heavy data is too create a new sheet for each
interview transcript, or feld notes segment.
One other method I have used is to code data using Markdown.
Markdown is a lightweight markup language that can be used to
add formatting elements to plain text text documents. Created by
John Gruber in 2004, Markdown is now one of the world’s most
popular markup languages [7]. As opposed to selecting text and
applying formatting through menus and tool bars, markdown uses
text based ways of indicating formatting. Mark down, as a text-
based system, ofer certain advantages to people who are blind and
visually impaired. Since Markdown uses markdown syntax, typi-
cally consisting of special characters such as *, #, and @ to indicate
formatting elements such as bold, italics, underline, headings, etc,
formatting elements become uniquely accessible to people who are
blind and visually impaired. Using markdown, screen reader users
can go through documents and understand the way they are for-
matted, without using special keystrokes to check the formatting of
any given text. While using word processors, formatting elements
such as font, size, bold, italics, etc are typically not automatically
Performing Qalitative Data Analysis as a Blind Researcher
announced by a screen reader, and have to be accessed using a
keystroke.
It is the ability to add in-line footnotes To mark down text docu-
ments that is of particular signifcance in the context of analysing
qualitative data. Using this feature, I add my codes to my qualitative
data as in-line footnotes. Each snippet is enclosed in to mark them
as bold, before attaching a footnote. As I re-read through this doc-
ument, I can read through the formatting using my screen reader,
and understand the codes I have assigned to each segment. This
is particularly helpful while dealing with a large amount of data,
as it negates the need to cycle through a large number of columns
in excel. However, there is no easy way of exporting text snippets,
or the codes attached to them, to a new fle. This has to be done
manually, as opposed to the sorting and fltering options in Excel
that make it easier to process coded data.
Analysing and coding videos and images are even more chal-
lenging as a blind researcher, as these dataformats are generally
rather inaccessible to people who are visually disabled. I generally
refrain from using video as a method of documentation. Rather than
recording videos when I am in the feld, I tend to textually record
my observations. I prefer to make transcripts of conversations and
record textual and audio feld notes.
However, in certain situations, collecting data in the form of
videos and images is unavoidable. Projects that involve data col-
lected from social media platforms, for instance, require me to
engage with data in the form of Screenshots, pictures, and videos.
In such situations, I must incorporate a preliminary step into my
process of data analysis: a step where I convert my pictures and
videos into descriptions of what happens in them. I can analyse this
extra data in the same way that I would analyse any other piece of
textual data.
What is evident from this analysis of possible workarounds to
performing qualitative data analysis using unconventional methods
is that, at this point in time, a one size fts all solution to analysing
qualitative data in a screen reader accessible way does not exist.
Every method I have outlined above is usable only in specifc situa-
tions or with specifc kinds of data. Moreover, typically, a research
project does involve dealing with multiple types of data such as
feld notes, interview recordings, pictures, etc, making it necessary
to use more than one method in analysing this data. It is in the
complexities like these that blind and visually impaired researchers
end up spending more than the usual amount of time to produce
work that is comparable in every way, including methodological
rigour, to that produced by sighted researchers.
4 DESIGN RECOMMENDATIONS FOR AN
ACCESSIBLE QUALITATIVE DATA ANALYSIS
PACKAGE
It is evident from the methods and problems I have outlined in
the previous section that developing a tool that enables Blind and
visually impaired researchers to accessibly analyse qualitative data
is a necessity. In this section I therefore in numerate the feature
requirements for an accessible qualitative data analysis tool, from
my experiences of using diferent tools and workarounds to analyse
diferent kinds of data.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Firstly, the tool should have an easy way of Assigning a partic-
ular code to a certain text snippet. To reduce the number of steps
necessary to select text and then go through menus to fnd the nec-
essary options, there must be a keyboard shortcut for this feature.
There must, of course, also be a way of identifying a particular text
snippet as coded or not coded.
Second, information such as whether a particular snippet has
been coded or not, and what code has been assigned to a particular
snippet should be announced by a screen reader as the user goes
through the document, without any additional keystroke. This will
go a long way in having accessible ways of re-reading and re-coding
documents. These attributes must also be visually indicated, for the
use of researchers who have low vision, or sighted collaborators.
Third, the tool must have an easy way to move from one as-
signed code to another. Screen reader behaviour such as the ability
to navigate by headings using a keystroke, can be taken as an exam-
ple. Just as a user can move through headings by pressing the ‘H’
key repeatedly on the web page, the tool should ideally allow for
navigation to the next and previous codes in the document using a
keystroke. This could possibly be control+c or something similar.
Next, the tool should also have the ability to deal with and process
multiple types of data such as audio, video, text, and images. This
is especially important since research projects typically do require
the researcher to collect multiple types of data in the form of feld
notes, interviews, photographs etc. Therefore a researcher who is
blind and visually impaired must have the ability to input all these
diferent types of data into the analysis tool, and analyse them using
this tool as well.
Finally, the tool should also have a way to perform further anal-
ysis on this data. This can possibly be facilitated in two ways. The
frst is to have an “analysis mode” or something similar which
displays quotes one after the other with each assigned text seg-
ment displayed below. This mode can also include a feature to
collapse multiple codes into one category or theme. This will facili-
tate easier thematic analysis. The option to create sub-themes or
sub categories would also be welcome. This information can per-
haps be displayed using a treeview control, which a screen reader
typically understands well. An alternative way of doing this is to
provide the ability to export codes and text snippets assigned to
them into a spreadsheet-based data format such as XLS orCSV for
further analysis.
5 CONCLUSION
It is evident that new and accessible ways of doing qualitative
research and analysing qualitative data are necessary for the Blind
and visually impaired researcher community. As more and more
blind researchers stepped into academia it becomes necessary not
only to avoid social and societal ableism, but also create a safe and
technologically accessible place within academia.
This is important not just for the sake of accessibility, but also to
promote more research about disability by the disabled community
themselves. If more and more researchers are to participate in
research on an equal footing, and if more research using co-design
and participatory design as methodologies must be conducted, it
is necessary to strip away the inaccessibilities in the platforms
and processes used for conducting research. Creating and fnding
ASSETS ’22, October 23–26, 2022, Athens, Greece
accessible ways of performing qualitative research is but one step
forward in this direction.
ACKNOWLEDGMENTS
I am extremely grateful to all the disabled researchers who have
come before me, whose footsteps into the world of academia have
paved the way for mine. I would also like to extend my gratitude
to my advisor, Dr. Amit Prakash, whose constant support for my
experiments into conducting qualitative data analysis has made
this paper possible.
REFERENCES
[1] Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence
as a Frame for Assistive Technology Research and Design. In Proceedings of the
20th International ACM SIGACCESS Conference on Computers and Accessibility
(Galway, Ireland) (ASSETS ’18). Association for Computing Machinery, New York,
NY, USA, 161–173. https://doi.org/10.1145/3234695.3236348
[2] Nicole Brown and Jennifer Leigh. 2018. Ableism in academia: Where are the
disabled and ill academics? Disability & Society 33, 6 (2018), 985–989.
[3] Fiona Kumari Campbell. 2009. Internalised ableism: The tyranny within. In
Contours of Ableism. Springer, 16–29.
[4] Mark Anthony Castrodale and Daniel Zingaro. 2015. You’re such a good friend:
A woven autoethnographic narrative discussion of disability and friendship in
Higher Education. Disability Studies Quarterly 35, 1 (2015), 1–27.
[5] Kristie Dotson. 2011. Tracking epistemic violence, tracking practices of silencing.
Hypatia 26, 2 (2011), 236–257.
[6] Hannah S Facknitz and Danielle E Lorenz. 2021. Refections on disability and
(dis)rupture in pandemic learning. http://activehistory.ca/2021/10/refections-
on-disability-and-disrupture-in-pandemic-learning/
[7] Keiran Healy. 2018. The Plain Person’s Guide to Plain Text Social Science.
https://plain-text.co/index.html
[8] Megan Hofmann, Devva Kasnitz, Jennifer Mankof, and Cynthia L Bennett. 2020.
Living Disability Theory: Refections on Access, Research, and Design. In The
22nd International ACM SIGACCESS Conference on Computers and Accessibility
O. Aishwarya
(Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery, New
York, NY, USA, Article 4, 13 pages. https://doi.org/10.1145/3373625.3416996
[9] Dhruv Jain, Audrey Desjardins, Leah Findlater, and Jon E Froehlich. 2019. Au-
toethnography of a hard of hearing traveler. In The 21st International ACM SIGAC-
CESS Conference on Computers and Accessibility. 236–248.
[10] Dhruv Jain, Venkatesh Potluri, and Ather Sharif. 2020. Navigating Graduate
School with a Disability. In The 22nd International ACM SIGACCESS Conference
on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association
for Computing Machinery, New York, NY, USA, Article 8, 11 pages. https:
//doi.org/10.1145/3373625.3416986
[11] Deborah Marks. 1999. Dimensions of oppression: Theorising the embodied
subject. Disability & Society 14, 5 (1999), 611–626.
[12] Robert McRuer and Merri Lisa Johnson. 2014. Proliferating cripistemologies: A
virtual roundtable. Journal of Literary & Cultural Disability Studies 8, 2 (2014),
149–169.
[13] Jenny Morris. 2014. Pride against prejudice: Transforming attitudes to disability.
The Women’s Press.
[14] Jackie Leach Scully. 2018. From “she would say that, wouldn’t she?” to “does she
take sugar?” epistemic injustice and disability. IJFAB: International Journal of
Feminist Approaches to Bioethics 11, 1 (2018), 106–124.
[15] India Today. 2017. World sight day 2017: Statistics and facts about visual
impairment and tips to Protect your eyes. https://www.indiatoday.in/education-
today/gk-current-afairs/story/world-sight-day-2017-facts-and-fgures-
1063009-2017-10-12
[16] United Nations. 2006. Convention on the Rights of Persons with Disabilities.
Treaty Series 2515 (Dec. 2006), 3.
[17] Maja van der Velden and Christina Mörtberg. 2021. Participatory Design and
Design for Values. Springer Netherlands, Dordrecht, 1–22. https://doi.org/10.
1007/978-94-007-6994-6_33-1
[18] WorldHealthOrganization. 2018. World Report on Disability. https:
//www.who.int/teams/noncommunicable-diseases/sensory-functions-
disability-and-rehabilitation/world-report-on-disability
[19] Anon Ymous, Katta Spiel, Os Keyes, Rua M. Williams, Judith Good, Eva Hornecker,
and Cynthia L. Bennett. 2020. "I Am Just Terrifed of My Future" — Epistemic
Violence in Disability Related Technology Research. In Extended Abstracts of
the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI,
USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA,
1–16. https://doi.org/10.1145/3334480.3381828
Blind Users Accessing Their Training Images in Teachable Object
Recognizers
Jonggi Hong Jaina Gandhi Ernest Essuah Mensah
Smith-Kettlewell Eye Research University of Maryland, College Park University of Maryland, College Park
Institute United States United States
San Francisco, United States jaina.gandhi@gmail.com eessuahm@umd.edu
jhong@ski.org

Farnaz Zamiri Zeraati Ebrima Haddy Jarjue Kyungjun Lee

University of Maryland, College Park University of Maryland, College Park University of Maryland, College Park
United States United States United States
farnaz@umd.edu ebjarjue@terpmail.umd.edu kjlee@cs.umd.edu

Hernisa Kacorri
University of Maryland, College Park
United States
hernisa@umd.edu

Figure 1: A blind participant in our study training the MYCam app in their homes to recognize Lays with real-time descriptors.
A dual video conferencing captures participant’s activities via a laptop camera and smart glasses worn by the participant.
ABSTRACT
Teachable object recognizers provide a solution for a very practical
need for blind people – instance level object recognition. They
assume one can visually inspect the photos they provide for training,
a critical and inaccessible step for those who are blind. In this work,
we engineer data descriptors that address this challenge. They
indicate in real time whether the object in the photo is cropped or
too small, a hand is included, the photos is blurred, and how much
photos vary from each other. Our descriptors are built into open
source testbed iOS app, called MYCam. In a remote user study in
(� = 12) blind participants’ homes, we show how descriptors, even
when error-prone, support experimentation and have a positive
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544824
impact in the quality of training set that can translate to model
performance though this gain is not uniform. Participants found
the app simple to use indicating that they could efectively train
it and that the descriptors were useful. However, many found the
training being tedious, opening discussions around the need for
balance between information, time, and cognitive load.
CCS CONCEPTS
• Human-centered computing → Accessibility; Ubiquitous
and mobile computing; Empirical studies in HCI.
KEYWORDS
blind, visual impairment, object recognition, machine teaching,
participatory machine learning
ACM Reference Format:
Jonggi Hong, Jaina Gandhi, Ernest Essuah Mensah, Farnaz Zamiri Zeraati,
Ebrima Haddy Jarjue, Kyungjun Lee, and Hernisa Kacorri. 2022. Blind Users
Accessing Their Training Images in Teachable Object Recognizers. In The
24th International ACM SIGACCESS Conference on Computers and Accessibil-
ity (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY,
USA, 18 pages. https://doi.org/10.1145/3517428.3544824
ASSETS ’22, October 23–26, 2022, Athens, Greece
1 INTRODUCTION
Echoing the end-user programming paradigm, the idea of hav-
ing end-users consciously provide training examples in AI-infused
applications has recently gained traction along with advances in
neural networks. By leveraging prior work in transfer, meta, and
few-shot learning (e.g., [7, 11, 70, 76, 80]), we are now able to build
teachable applications, where end-users can train models of their
own. These applications facilitate personalization as they promise
a better ft for real-world scenarios by signifcantly constraining
the machine learning task to a specifc user and their environ-
ment. Thus, it is no surprise to see early mentions of the term
“teachable” in accessibility research (e.g. [55]), where data is sparse
and there is a high diversity even for a given disability [32, 33].
A more recent example (and the focus of this work) is teachable
object recognizers [3, 34, 41, 42, 47, 68], where blind users train
their camera-equipped devices such as mobile phones to recognize
everyday objects by providing a few photos as training examples.
Why is it important to access one’s training examples? In
teachable applications such as teachable object recognizers, users
are called to interact with the machine learning model and improve
its performance by accessing and controlling their examples [83].
Personalization is often the ultimate goal. However, the interactive
nature of these applications can also help people uncover basic
machine learning concepts and gain familiarity with AI (e.g., [14,
17, 25, 27, 56]). Thus, they can also contribute to the larger goal
of “making the process of teaching machines easy, fast and above
all, universally accessible” [65]. An underlying assumption for both
improving a model and uncovering concepts via experimentation is
that users can inspect their data and iterate the training and testing.
By doing so, they could build an intuition about what works and
what doesn’t and perhaps why. However, this assumption does not
often hold for assistive teachable applications. Inspecting training
examples typically requires similar skills to those the technology
aims to fulfll [20, 31]; thus, it is often inaccessible. For example,
teachable object recognizers, where users teach the model to recognize
objects on their behalf, assume that they can see the training images
they are providing, which is almost never the case with blind users.
Sure enough, blind participants in prior studies with this technology
wanted to know more about their training examples [31] with one
of them stating “the most challenging and most fun is training the
person”.
Existing approaches for real-time ‘alt text’ for individual images
and ‘scene description’ for a series of images are not suitable for this
task; they do not capture fne-grained diferences across otherwise
similar images (e.g., see Figure 1). In this paper, we explore this
challenge of accessing one’s training data. Within the context
of teachable object recognizers for the blind, we study the poten-
tial and limitations of real-time ‘data descriptors’ that can capture
users’ training examples with photo- and set-level attributes. Specif-
ically, we investigate whether these descriptors could be derived
from visual attributes used to code training photos from sighted
(e.g., [26, 27]) and blind (e.g., [34, 41]) people. To this end, we en-
gineer photo-level descriptors that communicate to the user in
real-time information about the photo they just took such as blur-
riness, presence of their hand, object visibility, and framing. We
also engineer set-level descriptors that communicate information
one would get from glancing over a group of training photos such
Hong, et al.
as variation in object background, distance, and perspective; all
factors that can afect model performance.
Through a remote user study with 12 blind participants (with the
setup shown in Figure 1), we demonstrate that our data descriptors
support blind users in reducing photos with cropped objects in their
training sets and increase variation. Many participants chose to it-
erate after inspecting their training sets and refected by improving
many photo attributes, which resulted in models that generalize
better to photos from others, even though they reduced variation in
background. Aligned with prior studies, we also observe challenges
among participants in crafting good testing examples that could
further promote experimentation. Still, their models perform better
when tested on their own photos compared to both aggregated test
sets from all 12 blind participants in our remote study and from 9
blind participants in an in-lab study [41]. Observations from our
analysis of the photos and model performance are also confrmed by
participants’ subjective feedback. Responses support the potential
of descriptors, with blind participants indicating that they were
able to tell their meaning by looking at relative changes in values
and fnding them useful. However, errors in descriptors afected the
reliability of the app for some. More so, some considered training
being tedious referring both to time and cognitive load (e.g., opti-
mize for multiple variables). Many made design recommendations
that could further improve the efectiveness of the descriptors and
the training process.
To the best of our knowledge, this is the frst work to propose
non-visual access to training data and to provide empirical results
with blind participants on automatically estimating and incorpo-
rating descriptors for data inspection in teachable computer vision
applications. Our analysis focuses on object recognizers, where
‘learning to train’ is deemed as one of the main challenges among
blind users [31, 34]. However, we see how the underlying methods
for extracting meaningful instance- and set-level descriptors can
be adopted for other teachable applications both in assistive and
informal AI learning contexts. Perhaps, they can also serve towards
more accessible approaches for explainable AI interfaces, where
there is an underlying assumption on people’s ability to visually
inspect explanations [62, 66, 69].
2 RELATED WORK
There is a rich literature exploring how computer vision can beneft
people with disabilities (e.g., [10, 13, 30, 36, 46, 58, 71]). This is
especially the case with assistive technologies for the blind, where
computer vision is employed on smartphones (e.g., [2, 24, 38, 60,
78, 82]), smart glasses (e.g., [18, 43, 67, 81]), and smart suitcases
(e.g., [23, 35]). A common challenge we share with prior work is
that aiming the camera and inspecting recognition errors typically
requires similar skills to those the technology aims to fulfll (i.e.,
sight), even though the majority of prior work employs AI-infused
systems pre-trained by engineers, not fne-tuned by the end-user.
Thus, it is not a surprise to fnd that recognition errors afect blind
users’ experience [51]; sometimes, to a degree where it can not be
corrected even by human clarifcation [61]. In fact, blind users may
depend on the recognition especially when it is difcult to verify
its predictions. They may overtrust the predictions even when they
know they can be error-prone [41, 45] though, errors are especially
Blind Users Accessing Their Training Images in Teachable Object Recognizers
non-acceptable when they can adversely afect interactions with
others [1, 43]. Aligned with prior eforts aiming to support users’
recovery from errors [6, 28, 52], we explore how to make training
and resolving errors in teachable object recognizers more accessible
to blind users. Below, we focus on prior work that closely relates
to ours and contrast it to our study.
2.1 Teachable Object Recognizers
Looking at prior work on teachable object recognizers, we see di-
versity in research aims. Some, similar to this work, focus on the
blind community. They explore the potential of teachable object
recognizers as an assistive technology for blind users [34, 68], build
feedback mechanisms for better camera aiming [3, 41], and collect
benchmarking datasets for evaluating approaches in transfer learn-
ing and meta learning [72]. Our work is orthogonal and highly
complementary to these eforts – our shared goal is to improve
blind users’ experience with teachable object recognizers.
We also see studies involving sighted people both adults
(e.g., [27]) and children (e.g., [17, 75]). They aim to better understand
the potential of teachable machines for enabling non-experts to un-
cover basic machine learning concepts as well as better understand
common AI misconceptions they may have. Insights from these
studies are very informative for our eforts in making the ‘learning
to train’ challenge more accessible to blind adults and perhaps in
the future to blind children that may want to participate in similar
informal learning activities as in Dwivedi et al. [17].
Table 1 provides a more detailed overview from a sample of these
prior studies over the past fve years (2017-2021) with the number of
participants being typically smaller for in-person studies with blind
people and sighted children. As the performance of teachable object
recognizers and users’ behavior in taking photos can be afected
by environmental factors such as background, light condition, and
selection of objects, many studies collected inputs from participants’
environments to incorporate these factors [3, 27, 34, 72]. We also
opted for this approach in our study; the study was conducted in
the homes of blind participants while we control for factors such
as study procedure and object stimuli.
The majority of prior work on teachable object recognizers facil-
itates training through photos [3, 14, 27, 34, 41], except for one [72],
where blind users are called to use short videos. In our study, we
also used photos so that the outcomes of our study could be ap-
plicable (and comparable) to the majority of existing approaches.
More so, collecting videos may increase the burden on the user,
especially when they are given several instructions and tasks to
do [72] as in the case of our study. In addition, video-based assistive
technology can pose a greater privacy risk for blind users [5] as it
is more likely to capture unwanted objects and unnecessary infor-
mation in a video. Perhaps, live photos [53], could be the middle
ground between the two. We further refect on the potential of this
approach in the discussion section.
In their early explorations, Kacorri et al. [34] highlighted some
of the main challenges that blind users may face when training
a teachable object recognizer and testing its performance. They
revolve around camera framing (i.e., adjusting the distance between
the camera and an object and centering the object), capturing the
side of the object with the most distinctive visual features (i.e.,
ASSETS ’22, October 23–26, 2022, Athens, Greece
product logos), and reviewing the training photos after taking them
(i.e., quality and characteristics). Lee et al. [41] and Ahmetovic et
al. [3] aimed to resolve the camera framing challenge by developing
real-time audio/haptic feedback that helps blind users estimate the
proper distance and position of the object in the image frame [3, 41].
However, the challenge of reviewing photos for iteration has not
been addressed yet.
Typically, studies included a training and testing step for explor-
ing participants’ interactions with the teachable interfaces. Very
few of them [17, 27], though, allowed people to refect and iterate
giving them access to their training data for review. We believe iter-
ation is a critical step for understanding the potential of descriptors
for making the review process more accessible. Thus, in our study,
we also provide blind participants with an opportunity to refect
and the option to iterate after reviewing their images with the de-
scriptors. After all, our goal is to examine how data descriptors that
provide non-visual access to training photos, either individually or
as a set, can be helpful during the iterative process of training and
testing, as well as how blind users may interact with them.
3 MYCAM: A TESTBED TEACHABLE OBJECT
RECOGNIZER WITH DESCRIPTORS
To explore the potential and limitations of real-time descriptors
derived from visual attributes for accessing one’s training data, we
build MYCam. MYCam serves as a testbed for deploying descriptors
in a teachable object recognizer. In the background, it sends users’
photos to a server, where an image recognition model is being fne-
tuned by the user. While privacy is one of the promises of teachable
object recognizers [31] (i.e., by processing photos entirely on the
user’s mobile device), we fnd that the state-of-the-art on-device
training is not there yet. As a screen-reader accessible iOS mobile
app, MYCam enables remote studies with blind participants. This
was critical for us; due to the pandemic, we had to move our study
from the lab to blind participants’ homes. By open sourcing both
the MYCam app (available at https://iamlabumd.github.io/MYCam-
Mobile/) and our proof-of-concept implementation of the descrip-
tors (available at https://iamlabumd.github.io/MYCam-Server/), we
are hoping that others can contribute to further advance this work.
3.1 Design Rationales
DR1: Prioritize Blind Users. Both the form factor and interaction
modalities of MYCam are informed by prior work with blind users
and teachable object recognizers as well as broader real-world object
recognition applications. We opted for an iOS app since prior work
in the United States, the location of our study participants, suggests
that blind smartphone users overwhelmingly favor the iPhone [50]
though the actual numbers may be changing these past years [49].
When users open the app, they enter the main screen (Figure 2a),
which shows a camera preview. We opted for the default camera
app in iOS maximizing both compatibility with VoiceOver and user
familiarity with it. The recognition mode for MYCam was modeled
after existing real-world applications, such as Seeing AI [63], where
users can immediately ask the app to recognize what is captured by
the camera with a double-tap; the Scan button is activated by default.
In this case, the app takes a photo, sends it to the personalized object
recognition model in the server, and indicates the predicted label
ASSETS ’22, October 23–26, 2022, Athens, Greece Hong, et al.

Table 1: Characteristics of related studies on teachable object recognizers juxtaposed with ours.

[34] [68] [41] [27] [3] [75] [17] [72] This study
Blind Adults 8 14 9 10 52 12
Sighted Adults 2 10 2 100
Participants
Sighted Children 6 14
Real-world • • • • • •
Setting
Controlled • • • • • •
Photo • • • • • • •
Input
Video • •
Train • • • • • • • •
Tasks Test • • • • • • • • •
Iterate • • • •
Framing • •
Access
Review • • • •

Figure 2: The user fow of MyCam. MyCam has three main parts: Recognizing an object in the camera view (purple thread),
reviewing and editing the information of the objects (red thread), and teaching an object to the model (green thread).

both via speech and visually (Figure 2b). To mitigate potential errors
that can’t be verifed non-visually, the app says “Don’t know” when
uncertain (approach for uncertainty is discussed in Implementation).
DR2: Simplify the Machine Teaching Flow. Users can add a
new object to the recognition model via the Teach button on the
Home screen. The app displays the (rear-facing) camera preview
with the shutter button at the bottom center and a thumbnail image
of the last photo in the lower-left corner (Figure 2f). Users are asked
to take 30 photos with the count indicated in real-time via speech
and visually (Figure 2g); in Kacorri et al. [34], blind participants
indicated that they would like to obtain feedback from the camera
on the number of photos taken. The number of training examples
(i.e., 30) is also informed by the same study [34] with blind partici-
pants spending on average 65 seconds (SD=35.2) to take 30 photos
and often providing variation in their training examples. More so,
�-shot learning results in the literature are often reported for � = 1,
5, or 20. Thus, 30 examples could allow for bootstrap estimates
for future comparisons in this feld. As discussed in Related Work,
the majority of prior work in teachable object recognizers opt for
photos rather than videos – we followed this approach in hope that
photos provide blind users with more control over their training
examples in terms of both conscious variation incorporated and pri-
vacy concerns mitigated (e.g., presence of their hand, bystanders, or
Blind Users Accessing Their Training Images in Teachable Object Recognizers
Figure 3: The architecture of the MYCam system indicating approaches for estimating the descriptors and recognizing the
object.
surroundings in the camera frame). In the one study where videos
were used, blind users had to be explicitly trained to record videos
and follow specifc flming techniques [72]. Given the emphasis of
this study on the descriptors, we decided to simplify the machine
teaching and not require any explicit training steps for the users.
After the training examples, a dialogue box with a text feld shows
up prompting users to enter the name of the object (Figure 2i). More
so, in this screen users can opt to add an audio description. (Both
object name and description can be edited at a later time, as shown
in Figures 2d and 2e.) Once this step is completed, the app notifes
the user with a “Training in progress” message (Figure 2j). At this
point Scan and Teach buttons are made inactive. They are activated
once training on the server is complete and the user is notifed.
DR3: Enable Access to Training Data with Descriptors. Some
of the main concerns of blind participants about teachable object
recognizers in Kacorri et al. [34] were: “knowing whether the photos
were good, knowing the area of a package where the label or distin-
guishing information resides,.., and deciding on the distance between
the object and camera lens.” We observe that this information wanted
by the participants can be provided both at a photo level and at
a higher level across a set of photos. Thus, we devise two types
of descriptors, shown in Table 2. These are all derived from visual
attributes used to code training photos from sighted (e.g., [26, 27])
and blind (e.g., [34, 41]) people. Photo-level descriptors are binary,
they indicate whether the object is too small or partially included
in the frame (cropped), whether the photo is blurred, and if user’s
hand is included in the frame. Set-level descriptors are indicated as
a percentage. They draw from parallels to how humans recognize
objects independent of size, viewpoint, and location [54].
As shown in Figure 2f, users access the photo-level descriptors
after every photo that they take so that they can identify problems
in the photo (e.g., object being cropped) right away; since this gets
repetitive, a photo-level descriptor is communicated only when true.
Users can access this detailed information also later, when review-
ing their trained objects (Figure 2e). Photo-level descriptors are also
provided in aggregate together with the set-level descriptors (e.g.,
photo blurred in 50% of the training examples for an object). Users
ASSETS ’22, October 23–26, 2022, Athens, Greece
can access these aggregates along with the set-level descriptors at
the end of a training session (Figure 2h), where they are called to
select either OK to proceed or Retrain to retake the photos from
scratch. Both photo-level and set-level descriptors can be accessed
at a later time when reviewing and editing trained objects, as shown
in Figure 2d.
3.2 Implementation
We built the MYCam testbed on Apple iPhone 8 with the object
recognition models and descriptor estimators running on our server
on an NVIDIA GeForce GTX 1080 Ti GPU; the two communicate
through HTTP. The architecture of the system, indicating how both
descriptors and recognition predictions are obtained, is illustrated
in Figure 3. The estimation of the descriptors in the current im-
plementation of MYCam is error-prone; our approaches merely
serve as proof of concept. Prior to making these approaches more
robust, we wanted to examine whether blind users can leverage
such descriptors in the frst place for accessing their training data
and experimenting with the model.
3.2.1 Descriptors. In all previous studies that informed our descrip-
tors, researchers coded the attributes of photos manually through
visual inspection of the photos from participants. Given that this is
a time-consuming process, methods like Wizard of Oz do not deem
appropriate in this early exploration of descriptors for facilitating
accessible non-visual experimentation. Thus, we opt for methods
that attempt to automatically estimate them, even though, develop-
ing techniques for more accurate estimations is beyond the focus
of this paper and is briefy discussed in Section 6. Specifcally, we
employ state-of-the-art computer vision techniques such as world
tracking in ARKit, a YOLOv3 object detection model [57], and hand
segmentation models [42] to estimate the descriptors. To speed
up the calculations for real-time interactions, the object detection,
hand segmentation, and edge detection run on our server.
• Small object: Given a bounding box of an object in an image
from YOLOv3 [57], the object is considered too small if the
ASSETS ’22, October 23–26, 2022, Athens, Greece Hong, et al.

Table 2: Photo-level and set-level descriptors. The descriptors are informed by prior studies with sighted and blind people who
have no machine learning expertise looking at the way they synthesize their data for training [17, 26, 27, 34, 41].

Photo-level descriptors
Small object The bounding box of the object is smaller than 1/8 (12.5%) of the image.
Cropped object The object is partially included in the image.
Blurry photo The photo is too blurry to recognize textures or texts.
Hand in photo A user’s hand is visible in the image.
Set-level descriptors
Variation in size A set of images shows objects with diferent sizes.
Variation in perspective A set of images shows diferent sides of objects.
Variation in background A set of images show backgrounds with diferent textures or items.

size of its bounding box is smaller than 1/8 (12.5%) of the
image.
• Cropped object: If the YOLOv3 [57] bounding box is at the
edge of the image, the object is considered cropped.
• Blurry photo: The original RGB image is converted to
grayscale (pixels values range: 0-255). We use Laplacian edge
detection [37] to produce an image with the edges in the
grayscale image. In this last image, we then calculate vari-
ance in pixel values to quantify blurriness. If the variance is
lower than a threshold, the photo is considered blurry. In this
study, we set the threshold at 3.0; we found it classifes the
blurriness most accurately when tested on photos collected
in a prior study with blind participants [41].
• Hand in photo: The server detects the pixels from a hand via
a hand segmentation model that has been previously tested
with blind participants [42]. If the proportion of pixels of a
hand(s) in an image is greater than a threshold, it considers
the photo to show a hand. The threshold is 0.3%, which
detected photos with hands most accurately when tested
with the photos collected in a prior study by Lee et al. [41]
• Variation in size: When users take a photo, we detect the
position of the smartphone with ARKit. As the size of the
object depends on the distance between the phone and the
object, we used the standard deviation of the diferences
between the phone positions (����� ) to measure the vari-
ation in size indirectly. We set the maximum value of the
variation as 0.15 (����� ) that we could observe with the
photos collected by a sighted person in our research team
through an internal test. The app presents the variation in
size as percentage (����� /����� ∗ 100).
• Variation in perspective: We detect the sides of an object
using ARKit. For this, we pre-trained the 3D object detection
model in ARKit with the three object stimuli in our study. The
model provides an enclosing bounding box of an object with
six sides in 3D space when it detects the object regardless
of the object shape. The model fnds the main side of the
bounding box based on the object orientation. We calculate
the variation in perspective based on the number of object
sides shown in a training set. For consistency with other
descriptors, we present the variation in perspective as a
percentage with scaling (� ∗ 15% where � is the number of
sides in photos).
• Variation in background: Assuming that the backgrounds
captured in photos can vary as a user moves the camera to
diferent places or changes its orientation, we used the loca-
tion and orientation of the camera to measure the variation
in the background. We calculate the standard deviations of
diferences in both orientation (using 1-cosine similarity)
and the location of the camera in the 3D coordinate system
in ARKit. The greater value of the two standard deviations is
selected as a variation in background. Like variation in size,
we set the maximum value as 0.15 through an internal test.
We present the variation in background as a percentage.
3.2.2 Object Recognition Model. The base model for object recog-
nition is Inception V3 pre-trained on ImageNet [15]. When users
train the app, it fne-tunes the last layer of the base model using
transfer learning with photos taken by the users. The transfer learn-
ing works with a gradient descent algorithm with 500 iterations
and a 0.01 learning rate. The training takes around 80 seconds with
90 photos of three objects. When users recognize an object with a
personalized model, the time from taking a photo to notifying the
recognition result is around 100 milliseconds. To make the model
distinguish the objects in a user’s training set and tell the diference
from other objects that it has not been trained on, we employed an
approach of quantifying the confdence level of the discriminability
based on the entropy of confdence scores [79]. Specifcally, when
the entropy value is greater than 2.0 or the confdence score is lower
than 0.4, the app says "Don’t know" in synthesized speech instead
of the label predicted by the model. We decided the thresholds of
the entropy and confdence score through internal tests such that
the app could diferentiate the three objects for the user study in
the Section 4 from other items (e.g., pen, keyboard, mouse, keys)
with the thresholds most accurately.
4 USER STUDY
To explore the potential and limitations of descriptors in the context
of a teachable object recognizer, we conducted a remote user study
with blind participants. The study took place in participants’ homes
to minimize safety concerns during the COVID-19 pandemic. The
study was approved by the Institutional Review Board at the Uni-
versity of Maryland, College Park (IRB #1255427-1). In designing
this remote study, we came across many challenges, including how
to provide remote guidance and observe participants’ interactions
with MYCam and their objects. We quickly found that having just
the third-person camera view from the laptop was not enough.
Blind Users Accessing Their Training Images in Teachable Object Recognizers
Thus, as shown in Figure 1, we added a frst-person view with
smart glasses. We iterated via several pilot tests that involved blind
and sighted researchers in our team to anticipate the logistics (i.e.,
study equipment delivery) and communication methods (i.e., laptop
and smart glasses) required for this remote study. Lessons learned
from accessing blind participants’ interactions via smart glasses
(with this study serving as part of a larger case study) are discussed
in depth in Lee et al. [40].
4.1 Participants
We recruited 12 blind participants (6 women, 6 men, 0 nonbinary)
from campus email lists and local organizations. As shown in Ta-
ble 3, their ages ranged from 32 to 70 (� = 54.3, �� = 15.2). Three
participants reported being totally blind, fve having some light
perception, and four being legally blind. P1 and P2 reported having
an “auditory processing disorder” and difculty in hearing “very high
sound”, respectively. All participants reported using smartphones
several times a day and taking a photo or recording a video at least
once a month. As for their familiarity with machine learning, two
participants reported being somewhat familiar, eight being slightly
familiar, and two being not familiar at all—we used a 4-point scale
for this question: (1) not familiar at all (have never heard of ma-
chine learning), (2) slightly familiar (have heard of it but don’t
know what it does), (3) somewhat familiar (have a broad under-
standing of what it is and what it does), (4) extremely familiar (have
extensive knowledge on machine learning). While all participants
had experience taking photos before, many indicated that they had
challenges related to image framing (9), focusing (2), holding a
camera steadily (2), and controlling the lighting (2). Many partici-
pants indicated prior experience with other camera-based assistive
mobile applications such as Aira [4], Be My Eyes [8], Google Look-
out [21], Microsoft Seeing AI [63], Mediate Labs Supersense [48],
Super Lidar [29], and Voice Dream Scanner [16].
4.2 Procedure
Participants communicated with the experimenter remotely via
dual Zoom video conferencing [84] connected both via a laptop
and a pair of Vuzix Blade smart glasses [77] that we delivered prior
to their study sessions (see Lee et al. [40]). At the beginning of
the study, we briefy explained the concept of a teachable object
recognizer. Here, we provided a minimal description of how to
take photos to train or test the app to mitigate priming in photo-
taking strategies for training and testing an object recognizer. The
description given at the beginning of the study reads as follows:
“The idea behind the app is that you can teach it to
recognize objects by giving it a few photos of them,
their names, and if you wish, audio descriptions. Once
you’ve trained the app and it has them in its memory,
you can point it to an object, take a photo, and it will
tell you what it is. You can always go back and manage
its memory.”
Then, participants were asked to perform three tasks: (1) train
the app with their own photos and labels of three snacks that
served as object stimuli shown in Figure 4, (2) use the app again
to recognize those objects later i.e. to test the performance of the
app, and (3) review and edit the information of the already trained
ASSETS ’22, October 23–26, 2022, Athens, Greece
objects. For the frst task, the order of objects for training was fully
counterbalanced between participants. When participants trained
the app with the frst object, the experimenter provided step-by-step
instructions on the MYCam user interface (e.g., the position and
functionality of buttons as well as the audio feedback that indicates
the steps of training). Then, participants trained the app with the
second and third objects and asked the experimenter for help when
necessary. When participants were testing the app for the frst time,
the experimenter also gave detailed instructions on the MYCam
interface for testing. After that, participants were free to test their
models for as long as they wished (taking any number of photos).
When reviewing their trained objects in the third task, participants
could access both information related to the descriptors as well as
their own object labels and any recorded audio descriptions.
After reviewing a training set with the descriptors, participants
decided whether they would collect the photos again or not for
that object. We made retraining optional for two reasons: (1) to
avoid collecting data from participants who are not motivated to
experiment by retraining a model (as this could add a confounding
factor in our analysis) and (2) to be able to contrast the attributes
of training sets for who decided to retrain their models and those
who did not.
Throughout the study, we encouraged participants to think out
loud and to ask questions at any time. After each task, participants
were asked to answer questions related to their experience with
the descriptors and MYCam and questions captioning usability
satisfaction [44]. All questions in this study were either open-ended
or on a 5-point Likert scale (i.e., strongly disagree, disagree, neutral,
agree, strongly agree).
4.3 Object Stimuli
Accounting for blind people’s need for recognizing objects with
similar sizes, weights, and textures with fne-grained labels [34, 68],
we selected three snacks, shown in Figure 4, with the same size,
texture, and nearly identical weights for our user study. As prior
work shows that end-users’ strategies of collecting training photos
are often inconsistent between objects [27], we expect that the
choice of three similar objects allows us to observe blind people’s
teaching strategies in the context of fne-grained object recognition.
With these snacks, we simulated a scenario in which a blind user
interacts with the app to recognize diferent objects that the blind
user may feel difcult to distinguish using only the tactile sensation.
It was engineered to be a challenging scenario for machine learn-
ing models since these objects were similarly shaped and colored,
had refective surfaces, and were deformable. Unique and personal
objects without logos or texts on them (e.g., key, mug cup) can be
potentially used with a teachable object recognizer and perhaps
could be ft for a more realistic scenario. However, for this study,
we included only commercial products to allow for comparison
and replicability similar to prior studies regarding teachable object
recognizers [34, 41, 68].
5 RESULTS
Participants spend on average 143.8 seconds (�� = 72.4) taking
30 photos of an object. Five out of 12 participants re-train the ob-
ject recognizer after inspecting their training sets with descriptors.
ASSETS ’22, October 23–26, 2022, Athens, Greece Hong, et al.

Table 3: Participants’ demographics and experience with machine learning, photo taking, and camera-based assistive apps.

ID Age Gender Level of vision Onset Machine learning Photo taking Experience with assistive apps
P1 39 Woman Light perception Birth Not familiar at all Once a day Aira, Be My Eyes, Seeing AI
P2 67 Man Legally blind 55 Slightly familiar Once a month Seeing AI
P3 62 Woman Totally blind Birth Somewhat familiar Several times a month Seeing AI, Be My Eyes
P4 32 Man Legally blind 20 Slightly familiar Several times a day None
P5 66 Man Light perception 46 Slightly familiar Once a week Seeing AI, Supersense, Super Lidar
P6 61 Man Light perception 41 Somewhat familiar Several times a week Seeing AI
P7 70 Man Legally blind Birth Slightly familiar Several times a week None
P8 50 Woman Legally blind 45 Slightly familiar Several times a week Seeing AI
P9 69 Woman Totally blind 55 Not familiar at all Several times a day VD Scanner, Be My Eyes, Seeing AI
P10 66 Woman Light perception Birth Slightly familiar Several times a week None
P11 33 Woman Light perception Birth Slightly familiar Once a month Seeing AI, VD Scanner
P12 36 Man Totally blind Birth Slightly familiar Several times a day Seeing AI, VD Scanner, Lookout

variation in size, small object), the researcher annotated the bound-

Figure 4: Object stimuli in the study chosen for a challenging
fne-grained classifcation task: Fritos, Cheetos, and Lays.
Examples of the photo-collection attempts and their annotated
attributes (i.e., ground-truth attributes annotated by a researcher
through visual inspection) are shown in Figure 5. Through the
analysis of the participants’ photos and the performance of the
personalized object recognition models, we show how descriptors
may relate to the participants’ strategies for collecting training
photos when they decided to retrain their models. We also show
the impact of these changes in training photos on the performance
of the models. We observe promising trends in the characteristics
of photos (i.e., adding more variations and reducing problematic
photos) over time and iterations. Participants’ subjective feedback
also indicate that our descriptors can be a promising approach for
providing access to one’s training data in this context.
5.1 Correlation Between Estimated Descriptors
and Annotated Attributes
We report the performance of our approach in estimating descrip-
tors as it is a critical context for interpreting the remainder of the
results. More so, it can provide a glimpse at future eforts for esti-
mating such descriptors in a real-world context. Here we measure
performance by computing the correlation between the estimated
descriptors and annotated attributes. Given that prior work indi-
cates high inter-rater agreement for the annotation of these at-
tributes [27], we had a single researcher in our team performing
this task. To quantify the variation of background and perspective,
the researcher grouped the photos within a set based on their simi-
larity in terms of background and object side. We used the groups
to calculate the Shannon-Wiener Diversity Index [64], a measure
of variation in background and perspective. The researcher also
coded the photos with a cropped object, participants’ hands, and
blurriness. For the attributes related to the size of the object (i.e.,
ing boxes of the objects. The variation in size was considered as
the standard deviation of the proportions that the bounding boxes
occupy in photos. The proportions range from 0.0 (i.e., the object is
not captured) to 1.0 (i.e., a bounding box covers the entire photo).
A photo with a small object is defned as one having a bounding
box smaller than 12.5% of the photo.
As shown in Figure 6, the correlation coefcients between es-
timated descriptors and annotated attributes ranged from 0.23 to
0.57, highlighting that this is a challenging task. The correlation
for "small object" is not shown since only three of all photos had
small objects that are not detected by our descriptor estimator. Even
though we employed naive approaches for estimating the descrip-
tors as a proof of concept, all pairs had positive correlations. This
indicates that even with partial access there can be an opportunity
for refection and experimentation i.e., if participants considered
relative changes rather than absolute values. Below, we see some
empirical evidence in support of this premise.
5.2 Changes in Annotated Descriptors For
Participants who Choose to Retrain
Five participants (P1, P3, P5, P8, P10) decided to retrain with a new
set of photos for an object after reviewing their initial training sets;
one of them (P3) trained the same object three times, each time
with a new set of photos. A participant (P10) retrained with new
sets of photos for two of the three objects. No participant retrained
all three objects.
As shown in Figure 7, we contrast the estimated descriptors
for initial attempts to those during retraining attempts. When the
participants decided to retrain, their new training sets had fewer
photos with cropped objects, no hands included, almost no blurred
photos, and higher variation in perspective and size on average
compared to their initial photos. This is a promising trend providing
some evidence on participants’ attempt to respond and adhere to the
descriptors though it may have come at the cost of lower variation
for background.
Specifcally, the average numbers of photos with cropped objects
and users’ hands were fewer at 15.83 (�� = 13.41) and 0.00 (�� =
0.00) in their new training photos versus the initial at 19.50 (�� =
12.52) and 0.33 (�� = 0.81), respectively. The number of blurry
photos was 0.00 (�� = 0.40) and 0.17 (�� = 0.41) in retrained and
Blind Users Accessing Their Training Images in Teachable Object Recognizers ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 5: Photos from P10 and manually annotated attributes to be compared with automatically estimated descriptors.

Figure 6: Scatter plots indicating correlations between manual annotations (x-axis) and estimations (y-axis) for each descriptor.
ASSETS ’22, October 23–26, 2022, Athens, Greece
(a) The aggregate of photo-level descriptors.
(b) Set-level descriptors.
Figure 7: Contrasting descriptor values in initial attempts
to retraining attempts for P1, P3, P5, P8, and P10. Red dots
indicate means.
initial, respectively. The system did not detect any photos with a
too-small object in either set. As for variation, mean variation in
perspective and size in retrained were 0.37 (�� = 0.33) and 0.12
(�� = 0.09), respectively, which is higher compared to those in the
initial sets at 0.20 (�� = 0.32) and 0.11 (�� = 0.07). However, this
trend was reversed for variation in background. This descriptor
was on average lower in retrained at 0.19 (�� = 0.28) compared to
the initial at 0.26 (�� = 0.28).
5.3 Changes in Annotated Attributes for All
Participants Over Time
Many participants chose not to retrain. Perhaps the interactive
nature of the descriptors created opportunities for early refection
and experimentation; not just at the end of training. To explore
this, we measured trends over time at diferent levels of granularity;
for this analysis, we use the manually annotated attributes, which
serve as the ground truth, rather than the estimated descriptors.
5.3.1 Fine-grained Changes Across 90 Training Photos. With photo-
level descriptors participants’ could gauge potential image quality
issues right away; MYCam indicates them immediately after a photo
is taken. As shown in Figure 8a, we observe a dropping trend in the
number of images where the object was cropped as participants
progressed in the study. This is promising for a descriptor that
merely provides binary feedback (i.e., whether the object is cropped
or not) instead of directional guidance on how to move a camera to
fully capture an object (e.g., Lee et al. [41]). The proportion of photos
with cropped objects was around 0.56 at the beginning (1st photo
in 1st training), decreasing to 0.37 by the last photo (30th photo in
3rd training). Whereas the proportion of training examples with
Hong, et al.
(a) Cropped object.
(b) Blurry photo, hand in photo, too small object.
Figure 8: The average values of annotated photo-level at-
tributes for individual photos among 12 participants. The
charts include photos of the frst three training sets (1-30:
frst set, 31-60: second set, 61-90: third set). The lines are ftted
to dots using LOWESS smoothing.
participants’ hands included, objects too small, or blurry photos
were nearly zero throughout the study (Figure 8b).
5.3.2 Coarse-grained Changes Across 3 Training Sets. With photo-
level descriptors and set-level aggregates participants’ could gauge
potential issues related to their teaching strategies or image quality
at the end of each training attempt; MYCam shows them imme-
diately after 30 photos are taken. Participants may or may not
choose to go back and retrain. But they may also choose to refect
when training the next object, especially since our object stimuli
were engineered to be very similar. As shown in Figure 9, partici-
pants increased the variation among their training examples and
reduced the number of photos with cropped objects. A one-way
repeated-measure ANOVA indicate a signifcant efect of order of
sets on variation in background (� (2, 22) = 4.59, � = 0.022, partial
� 2 = 0.18) and in perspective (� (2, 22) = 3.61, � = 0.044, partial
� 2 = 0.05). We did not observe a statistically signifcant efect of the
other attributes. However, we do observe a tendency for an increase
in the number of photos that were blurry or where the participant’s
hand was included. Perhaps these descriptors were not deemed as
that problematic or they were ranked lower in priority as teaching
strategies evolved. Participants’ feedback below can shed a bit more
light on these observations.
5.4 Performance of Participants’ Object
Recognition Models
After fnalizing their training for all objects, participants were called
to test the performance of their models; we explicitly did not allow
Blind Users Accessing Their Training Images in Teachable Object Recognizers
(a) Set-level descriptors.
(b) Cropped object.
(c) Blurry photo, hand in photo, small object.
Figure 9: The average annotated values of set-level attributes
and the annotated number of photos with photo-level at-
tributes for all 12 participants across three training sets (a
training set per object).
for intermediate train-test iterations in an attempt to limit interfer-
ence from that type of experimentation in the observed behaviors.
For the purpose of our analysis, we report model performance not
only on participants’ fnal training sets but also dive deeper and
look at their photos chosen to test their models and how well their
model generalizes e.g., if tested with photos taken by others.
5.4.1 Model Performance with Testing Images from Self. We found
that participants used a very small number of photos (� = 3.7,
�� = 3.2) to check if their models were working properly. Some
(4 out of 12) included photos where the object was more than half
cropped. Others (4 out of 12) captured multiple objects in the frame.
Some of these observations could be perhaps explained by our study
setup (e.g., participants were done with taking photos for the day or
objects were in close proximity due to study setup). However, prior
work in teachable object recognizers employing diferent study
designs also indicates that model testing and evaluation can be
challenging for end users [17, 27]. These challenges are critical
as perceived and actual performance may be diferent when the
models are actually used after testing.
ASSETS ’22, October 23–26, 2022, Athens, Greece
(a) Object recognition accuracy on one’s own testing images.
(b) Accuracy against participant satisfaction with performance.
Figure 10: When testing their models, participants’ experi-
ences varied (a), which seems to be refected in their satisfac-
tion scores (b).
Overall, we fnd that when testing on one’s own data the av-
erage accuracy (i.e., the number of correct predictions divided by
total test images) of the models was 0.65 (�� = 0.24) with a break-
down across participants shown in Figure 10a. These results may
seem surprisingly low for a 3-way classifcation task. However,
beyond being a fne-grained classifcation, the task can be partic-
ularly challenging with objects of deformable shapes, same-size,
refective-surface, and similar colors that can be hard to distinguish.
Among the high-performing models are those of P1 and P8 who
choose to iterate on their training (they tested the models with 3
and 7 photos, respectively); though the same is not to be said for the
models of P3, P5, and P10 who also iterated on their training (they
tested the models with 12, 28, and 10 photos, respectively). When
juxtaposing model performance with participants’ subjective re-
sponses on the satisfaction of their models (Figure 10b), we fnd that
those whose models did not perform well disagree with this state-
ment and those whose models perform better agree. This alignment,
however, did not hold for those on the edges (strongly disagree and
strongly agree). Participants’ feedback in the next section, provides
a potential explanation.
5.4.2 Model Performance with Testing Images from Others. One of
the promises of a teachable object recognizer is that it works well
for each individual since the training and test sets are collected by
the same person and it is highly likely that they are going to exhibit
similar patterns [34, 68]. This was also the case in our study. As
shown in Figure 11, for 9 out of 12 participants, the accuracy of the
model was higher when tested with an individual participant’s test
set than an aggregated test set from all participants in our study
and photos from another study with blind participants [41] on the
ASSETS ’22, October 23–26, 2022, Athens, Greece
(a) Accuracy per participant.
(b) Summary statistics of accuracy.
Figure 11: Model accuracy when tested on individual test
images, aggregated test images from all 12 blind participants
in this remote study, and aggregated test images from all 9
blind participants in a prior in-lab study [41].
same objects. The accuracy of the model with individual test sets
was 0.65 (�� = 0.24). The accuracy was lower at 0.51 (�� = 0.14)
and 0.52 (�� = 0.09) when pooling test sets across all participants
in the current study and testing photos from a prior study [41],
where nine blind participants trained and tested a teachable object
recognizer, respectively. However, we observed that the iteration
can make the models generalize better. Among the fve participants
who did retraining, four and three participants had higher accuracy
after retraining when their models were tested with the aggregated
test set and the set from the prior study [41], respectively.
5.5 Subjective Feedback from Participants
5.5.1 Overall Experience. To provide more context on participants’
feedback for the descriptors, we illustrate in Figure 12 their re-
sponses related to the MYCam testbed. Overall, participants agreed
that they could train the object recognition model efectively with
MYCam and disagreed on training being difcult, though they were
divided on whether it could be done quickly. This is promising.
Specifcally, ten participants agreed or strongly agreed that they
could train their models efectively with some pointing to the need
for onboarding. P1 and P10, for example, who are not familiar at
all and slightly familiar with machine learning said “after a while, I
learned that I could train it” and “It’s pretty easy. You have to teach
me though. But if you teach me then it’s pretty easy to follow in-
structions and fnish the process.” respectively. On the other hand,
Hong, et al.
P11 and P12 were neutral. P11 mentioned that taking 30 photos
is time-consuming, saying “I don’t really feel like I was all that ef-
fective because it takes a while to train for each one.” The errors in
descriptors afected the reliability of the app, making a participant
think the training process was less efective even though the two
models work independently of each other. P12 said "I don’t think
that the app is correct, especially when I know, for example, that my
hand was not in the photo...I don’t have a lot of confdence in the app’s
accuracy.”
When asked whether they could train the app quickly, fve par-
ticipants agreed, four disagreed, and three were neutral. Seven
participants indicated that taking 30 photos is tedious. For exam-
ple, P10 said, "The process is pretty straightforward. But I have to
spend, like, quite long time to train the three objects." When asked
about the difculty of the training task, all but one who remained
neutral, disagreed or strongly disagreed that the task was difcult.
P11 who was neutral, found it not difcult but tedious. Surprisingly,
this sentiment of the task being tedious was not presented in the
initial study with teachable object recognizers [34] even though the
number of training photos was identical. We suspect this diference
refects more on our implementation of the descriptors in the MY-
Cam testbed rather than the process of training itself. In MYCam
users could not opt in or out of the photo-level descriptors during
training leading to higher training times; specifcally, the time for
taking photos for training an object was doubled from 65 seconds
(�� = 35.2) reported in that frst study [34] to 143.8 seconds on
average (�� = 72.4) in our study.
5.5.2 Descriptors. As shown in Figure 13, all but one participant
(P1, who was neutral) agreed or strongly agreed that the descrip-
tors were easy to understand. P6 said “I understood what it was
telling me. I didn’t have questions about what I was supposed to do.”
Participants’ responses indicate user refection based on descrip-
tor changes across multiple attempts, strengthening some of our
observations in the previous sections. P2 elaborated “It gives you
directions. The explanation (descriptors) afterwards, in the analysis,
told me that my photographs were not always good. So I have to learn
to take better photographs.” Some participants found it difcult to
understand the absolute values of provided in the descriptors and
were wondering whether they should have a specifc value as a
goal. For example, the values of descriptors were somewhat am-
biguous to P1 who said, “I guess just knowing exactly what they’re
referring to what numbers are really preferable.” P4 also mentioned
the challenge in understanding the values of descriptors, but then
mentioned that over repeated data collection during the study, he
fgured out their purpose. P4 said “I wasn’t aware of any of those
felds when we did the frst object [...] For the second and third objects,
I could take a little bit more variation in the photos or to better train
the application.” This is interesting feedback as the descriptors are
there merely to provide access to what one could infer via a visual
inspection not per se dictate optimal characteristics for the training
set. The diference of course is that when a sighted person glances
over their training photos they may or may not make an inference
on potentially problematic photos or lack of variation (see Hong et
al. [27]), but a blind person always hears the descriptors. This ex-
plicit presence of the descriptors calls for the need for more context.
Blind Users Accessing Their Training Images in Teachable Object Recognizers ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 12: Participants’ feedback on training with the MYCam testbed.

Figure 13: Participants’ feedback on the descriptors.

While “ideal values” are use-case depended, during onboarding for the training.” This is an intriguing approach, one that we aim to
users could perhaps be provided with some rationale or examples. explore.
Ten participants agreed or strongly agreed that the descriptors
5.5.3 Model Performance. When we asked participants if they were
were useful. P10 (who was neutral on this) and P11 thought de-
satisfed with the performance of the object recognizer, opinions
scriptors helped them understand how to collect training examples
were divided; fve participants agreed or strongly agreed, six par-
for the object recognizer. P10 said, “(I agree) because I know the
ticipants disagreed or strongly disagreed, and one participant was
quality of the photos, the diferent aspects of the photos that I take.”
neutral, as shown in Figure 14. When accounting for the perfor-
P11 said, “It helped me understand what the camera needed in order to
mance of their model in their subjective responses (Figure 10b),
recognize the objects.” Participants also mentioned that descriptors
we observe that participants were not satisfed if the accuracy was
were useful to identify problems in their training sets. P10 elabo-
lower than 0.6. However, it did not all come down to model per-
rated “you have to get feedback or you’re not going to improve [...] it
formance. Open-ended feedback indicates that satisfaction is also
helps you to understand what you’re doing wrong.” P2 had a similar
related to efort. P11 remained neutral even though she did not ob-
idea: “the explanation (descriptors) afterward, in the analysis, told
serve any recognition errors (her model had the highest accuracy)
me that my photographs were not always good, so I have to learn to
but attributed this to the training task being tedious (see Section
take better photographs.” P12 thought they were not useful because
5.5.1). P11 said, “Because it took so much work to get that small
they were error-prone. P12 said, “I don’t think that the app is correct,
amount of performance.”
especially when I know, for example, that my hand was not in the
P7 and P10 agreed that they were satisfed with the performance
photo, or that the object is not cropped because the previous objects
though their model accuracy was only 0.6 and 0.4, respectively. As
were cropped.” This feedback highlights the need for improving the
legally blind P7, expressed that the performance is good enough
estimation of descriptors in future work.
to supplement his vision. P10 believed that she just did not train
Participant feedback suggests that it would have been helpful
the app properly. She said, “I think it recognized objects, but if you
to include more explicit guidance on how to improve the training
don’t train it properly, then it’s not going to recognize anything [...]
photos. For example, P7 suggested similar feedback to Lee et al. [41]
the Fritos bag was the one that didn’t work out, but that was probably
and Ahmetovic et al. [3] along the descriptors, elaborating “Cropped,
my fault.”
it did not help me know what to do diferently. If it said, maybe move
While the majority (9 out of 12) of participants observed recogni-
up, move down and move camera left, move the camera, right. That
tion errors during testing, many could not explain why. Six partici-
would have been more useful.” Our current implementation of this
pants were neutral or disagreed with the statement that they have a
photo-level descriptor actually can be re-purposed to provide such
good sense of why the recognition errors occurred. Their responses
feedback. More so, P6 mentioned that the interface for replacing
were simply “I have no idea.” or “I don’t know.” Though P7 and P10
problematic photos in a training set would improve the app. He
strongly agreed and agreed, respectively, their rationale was vague.
said “I would assume the training process can self-evaluate itself and
P10 said “I think it was my fault. I think it was my training. Other
it should sum that up for me and tell me what photos I should replace.
than that, I don’t know.” P9 strongly agreed and contributed the
[...] you need to replace those bad pictures unless you don’t need them
recognition errors to imperfect descriptors in training, elaborating
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 14: Participants’ feedback on the performance of their object recognition models.
“The reason is because I was teaching it, and I wasn’t 100% sure that it
was 100% accurate. It makes sense that while I was teaching it, I was
a little bit of, so its recognition was a little bit of. It kept telling me
that the hand was in the photos.” We believe that these observations
motivate the need for accessible computer vision explanations.
6 DISCUSSIONS
Our user study, exploratory in nature, shows both promising results
and future research directions for supporting blind users’ interac-
tions with teachable machines. In this section, we frst refect on
lessons learned, while discussing implications for designing descrip-
tors to access one’s training data in teachable object recognizers
and broader teachable applications either assistive or educational.
We then discuss limitations in our study that may afect the general-
izability of our fndings as well as future work for better estimating
such descriptors and exploring their potential for explainability.
6.1 Implications
Our study provides evidence that descriptors derived from visual
attributes used to code training photos in teachable object recog-
nizers, can provide blind users with a means to inspect their data,
iterate, and improve their training examples. Challenges often in-
volve onboarding, time needed for training, as well as descriptor
accuracy and interpretation.
Insights from this work are complementary to prior studies
exploring the feasibility of training [34, 47, 68] and camera aim-
ing [3, 42, 72] in teachable object recognizers for the blind. More so,
the underlying methods for extracting meaningful descriptors, i.e.,
instance- and set-level characteristics that can be coded by quickly
inspecting the training data and that point to noise and variation,
respectively, can be adopted for other teachable applications. This
is especially critical for those assistive applications where training
typically requires similar skills to those the technology aims to
fulfll. For example, teachable sound detectors for Deaf/deaf and
hard of hearing people [9, 20] could beneft from visualizations
of the sound examples in a way that allows users to quickly in-
spect potential noise in a training example and variation across
the training set (e.g., better start and end of a recording, multiple
sound sources, variation, and other characteristics that hearing
users could leverage for experimentation just by listening to the
audio). Indeed, Goodman et al. [20] observed that Deaf/deaf users
Hong, et al.
collected similar-sounding examples during training and thus, could
beneft from an interface that visualizes features of their training
sets; data descriptors could fll that need.
Accessing one’s training data is also critical for making informal
learning activities that typically employ teachable machines with
children more inclusive. Learning objectives for AI education in
K12 (e.g. [73]) highlight the use of interactive systems for exposing
children to AI prior to using those that leverage block-based pro-
gramming [74]. Dwivedi et al. [17] suggest that future teachable
interfaces for such activities beneft from classifcation tasks that
allow children to quickly inspect the data and uncover patterns.
Thus, it is not a surprise to see many learning activities for exposing
children to AI often leverage teachable image classifcation appli-
cations [14, 17, 22, 39, 75]. However, in these initial explorations,
none of these applications are inclusive of blind children. Our data
descriptors could help increase their accessibility e.g., by leverag-
ing our shared code for MYCam and the descriptors. Further, we
see how researchers working in teachable object recognizers and
broader contexts, could beneft from the following insights:
Balancing with demand on time and cognitive load.
MYCam’s set-level descriptors are given at the end of
training but image-level descriptors are played every time
the user takes a training photo (they can also be accessed
when reviewing at a later time). Participants’ feedback
indicates that, although the image-level descriptors are
informative, they add to the training time and cognitive load.
Indeed, if we were to compare our study with the times
reported in Kacorri et al. [34] training with descriptors (4.8
seconds per photo on average) took more than double the
time without them (2.2 seconds on average), respectively.
Still, this was much less when compared to another study,
where blind users took photos with real-time camera aiming
guidance (i.e., audio and haptic feedback for camera aiming);
there, they spent on average 10.3 seconds per photo [41]
but did not refect much on the time needed to train. The
diference between the two is: MYCam feedback when
taking photos is passive and requires listening to a list
of descriptors and optimizing simultaneously multiple
variables whereas the feedback in Lee et al. [41] is interactive
and requires listening to an audio cue or sensing vibration
and optimizing a single variable (including the object in the
Blind Users Accessing Their Training Images in Teachable Object Recognizers
frame). This challenge of maximizing information while
minimizing cognitive load is not new and calls for better
interactivity with the descriptors via opt in/out mechanisms
(e.g., play descriptors via press and hold), verbosity controls,
or audio haptic feedback. For example, P10 expressed that
while descriptors provide hints to problems, they do not
directly instruct users on how to solve them. For example,
when an object is cropped in a photo, participants did not
get feedback on in which direction the camera should move
even though this information can be made available from
the current implementation. We expect that combining the
descriptors with camera guidance (e.g., [3, 41]) could be
helpful.
Balancing descriptors with instructions. There is a rich lit-
erature on the value of tutorials, instructions, and in-context
interactive assistance for supporting users with technology;
a comprehensive review for blind users and smartphone
devices can be found in Rodrigues et al. [59]. Some prior
studies with blind users have shown that real-time descrip-
tions can lead to better accuracy and confdence compared to
instructions at the start of a task (e.g., [19]). While we did not
compare the two, participants’ feedback indicate that data
descriptors would be complementary and not a substitute
for tutorials and instructions. In addition to the real-time
feedback, participants call for support in navigating the app
and interpreting descriptor values. In our study, the exper-
imenter provided some of this information. For example,
for the set-level descriptors the experimenter said: “you can
check how much variation your photos have. For example,
a 10% variation in the background means that most of your
photos have similar backgrounds.” However, participants
mentioned that they could better understand the absolute
values of descriptors after experimenting. We suspect that
the level of understanding for these values would afect both
the quality of the training sets as well as how reliable the
system is perceived by the users.
Editing a training set based on descriptors. The current
design of MYCam focuses on informing the users of the
attributes of their training sets rather than instructing how
to spot potential issues in their training sets or making
the data collection process efcient. Participants had to
diagnose problems for themselves based on descriptors
and replace the entire training set with new photos if they
wanted to fx something. Participants suggested adding
functions to edit (e.g., delete) at a photo level e.g., right after
taking a photo that is deemed noisy or while reviewing the
training set at the end to make the iterations more efcient.
For example, P8 suggested having an interface that flters
out bad images based on descriptors or enables users to
replace them instead of starting from scratch. This opens up
interesting venues for approaches such as active learning
and data valuation.
6.2 Limitations and Future Work
There are many limitations that could impact the generalizability
of our fndings. Our observations come from a small sample even
ASSETS ’22, October 23–26, 2022, Athens, Greece
though � = 12 is most common in human-computer interaction
studies [12]. Our study is remote, yet participants are recruited
from a relatively small area in the US in proximity to the authors’
institution. The study is conducted in participants’ homes, yet it
shares more characteristics with a controlled in-lab study rather
than a real-world deployment: object stimuli were predefned and
small in number, the duration of the study was relatively short,
MYCam was deployed on one of our devices, participants were
being observed and had real-time support from the experimenter,
and they were somewhat confned in terms of space.
Specifcally, participants were asked to wear smart glasses and
communicate with the experimenter through a laptop computer
in front of them. Though these devices were necessary for com-
munication and data analysis, they limit participants’ behavior e.g.,
in walking around with the phone and taking photos in diferent
locations and illuminations. For example, when participants wanted
to vary the backgrounds in photos, they took pictures with diferent
parts of a table. However, if they could move around outside the
user study setup, perhaps they would choose completely diferent
locations for background variation. More so, using MYCam on one
of our iPhone 8 devices instead of their own mobile devices could
have afected our observations. All but one participant owned an
iPhone; most participants were familiar with iOS apps. However,
the diference between their personal phones and our device (e.g.,
in terms of size and camera location) could have afected the quality
of photos and overall perception of the descriptors. We expect that
the use of MYCam in a real-world scenario would have resulted
in a richer set of contexts in users’ photos (typically a table in our
study). Though we limited the objects to three snacks with similar
textures and weights, blind people may choose to train on personal
objects that may not be products in the market with a larger num-
ber of object instances. As the performance of an object recognizer
depends on the number of classes and visual diference between
the objects, these diferences could have afected the performance
of a personalized model and blind users’ experiences with it.
Our experimental setup was in part restricted by our implemen-
tation of the descriptors, which is meant to serve as a proof of
concept and is somewhat tied to a predefned set of object stimuli
(e.g., for the ARKit to work and for establishing diferent thresholds).
Although the estimated descriptors had a positive correlation with
the manually annotated attributes and enabled participants to in-
spect their training sets, they were error-prone. When some of the
participants noticed the errors in descriptors, they deemed them as
well as the object recognition model unreliable. This suggests that
it is imperative to further advance approaches related to descriptor
estimation for a better user experience.
Due to the lack of datasets for benchmarking our descriptor-
based approach, we had to manually create our own dataset for
comparison. As in other AI-based systems evaluation, having bench-
mark datasets is useful to assess systems for generating descriptors
in a more widely accepted way. One potential step in this direction
would be to invite blind data contributors, who can inspect their per-
sonal training data and agree to data sharing, to contribute to such
benchmark datasets employing approaches similar to Theodorou
et al. [72].
Last, in this study, images were used for the purpose of training.
This approach can provide more control for the blind users over
ASSETS ’22, October 23–26, 2022, Athens, Greece
their training sets regarding both incorporating variations and miti-
gating privacy risk concerns [5] as it would be less likely for a blind
user to capture unnecessary information in an image. For example,
blind users usually use their hands as a reference to center the
object in the camera frame, but they are often not willing to include
their hands in the fnal photo to preserve user anonymity [41]. Also,
in the one study where videos were used, blind users had to be
trained to follow some instructions and flming techniques [72].
On the other hand, video increases the number of collected images
since it is a collection of frames. Also, the use of video increases the
chance of the object being in the frame at some point [72]. Perhaps
a way to get the best of the two worlds could be live photos as they
are easy to capture (like photos), and they include multiple frames
over one to three seconds [53].
7 CONCLUSION
In this work, we examined the challenge of accessing one’s training
examples in teachable object recognizers, where visual inspection
of training photos is not accessible to blind users with the ultimate
goal of making machine teaching more inclusive. To this end, we
engineered real-time descriptors that indicate to the blind user
whether the photo they just took is blurry, if their hand is in it,
if the object is cropped, and whether their photos overall vary
in object background, distance, and perspective; all factors that
can afect model performance. We built MYCam, an accessible and
open-source teachable object recognizer iOS app with descriptors.
We shared our fndings, observations, and lessons learned from a
remote study with 12 blind participants who trained MYCam in
their homes to recognize three distinct but visually similar objects.
Our results showed that participants who choose to iterate their
training for an object, were able to provide fewer photos where
the object was cropped, included no hand in their photos, and
had slightly less blurry photos that overall had more variation in
terms of object perspective and size but less in terms of background.
Overall, participants increased the variation among their training
examples and reduced the number of photos with cropped objects
as they moved in training from one object to the next. Some of these
changes are refected in their model performance that somewhat
relate to their satisfaction scores. However, errors in descriptor
estimates seem to afect overall participants’ perception and trust
of model performance. Participants’ responses indicate that even
though it was difcult to gauge the meaning of absolute values for
some of the descriptors (e.g., variation), they could infer it based
on relative changes. However, many found the training being te-
dious, opening discussions around the need for balance between
information, time, and cognitive load. These results, taken together,
indicate that our novel data descriptors, realized in MYCam, hold
potential for facilitating quick inspection of training photos among
blind individuals. Going forward, we are excited to continue our
endeavors towards building more inclusive participatory machine
learning experiences both for blind youth and adults.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their thoughtful comments
on an earlier draft of this paper. This work is supported by NSF
(#1816380). Kyungjun Lee is supported by NIDILRR (#90REGE0008).
Hong, et al.
REFERENCES
[1] Ali Abdolrahmani, William Easley, Michele Williams, Stacy Branham, and Amy
Hurst. 2017. Embracing Errors: Examining How Context of Use Impacts Blind
Individuals’ Acceptance of Navigation Aid Errors. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems (Denver, Colorado, USA)
(CHI ’17). Association for Computing Machinery, New York, NY, USA, 4158–4169.
https://doi.org/10.1145/3025453.3025528
[2] Dragan Ahmetovic, Cristian Bernareggi, Andrea Gerino, and Sergio Mascetti.
2014. ZebraRecognizer: Efcient and Precise Localization of Pedestrian Crossings.
In 2014 22nd International Conference on Pattern Recognition. 2566–2571. https:
//doi.org/10.1109/ICPR.2014.443
[3] Dragan Ahmetovic, Daisuke Sato, Uran Oh, Tatsuya Ishihara, Kris Kitani, and
Chieko Asakawa. 2020. ReCog: Supporting Blind People in Recognizing Personal
Objects. Association for Computing Machinery, New York, NY, USA, 1–12. https:
//doi.org/10.1145/3313831.3376143
[4] Aira. 2017. Your Life, Your Schedule, Right Now. https://aira.io
[5] Taslima Akter, Bryan Dosono, Tousif Ahmed, Apu Kapadia, and Bryan Semaan.
2020. "I am uncomfortable sharing what I can’t see": Privacy Concerns of the
Visually Impaired with Camera Based Assistive Applications. In 29th USENIX
Security Symposium (USENIX Security 20). USENIX Association, 1929–1948. https:
//www.usenix.org/conference/usenixsecurity20/presentation/akter
[6] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi,
Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, and et
al. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk)
(CHI ’19). Association for Computing Machinery, New York, NY, USA, Article 3,
13 pages. https://doi.org/10.1145/3290605.3300233
[7] Matthias Bauer, Mateo Rojas-Carulla, Jakub Bartłomiej Świątkowski, Bernhard
Schölkopf, and Richard E Turner. 2017. Discriminative k-shot learning using
probabilistic models. arXiv preprint arXiv:1706.00326 (2017). https://doi.org/10.
48550/ARXIV.1706.00326
[8] BeMyEyes. 2016. Lend you eyes to the blind. http://www.bemyeyes.org/
[9] Danielle Bragg, Nicholas Huynh, and Richard E. Ladner. 2016. A Personalizable
Mobile Sound Detector App Design for Deaf and Hard-of-Hearing Users. In
Proceedings of the 18th International ACM SIGACCESS Conference on Computers
and Accessibility (Reno, Nevada, USA) (ASSETS ’16). Association for Computing
Machinery, New York, NY, USA, 3–13. https://doi.org/10.1145/2982142.2982171
[10] Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault,
Annelies Brafort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa
Verhoef, Christian Vogler, and Meredith Ringel Morris. 2019. Sign Language
Recognition, Generation, and Translation: An Interdisciplinary Perspective. In
The 21st International ACM SIGACCESS Conference on Computers and Accessibility
(Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New
York, NY, USA, 16–31. https://doi.org/10.1145/3308561.3353774
[11] John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and
Richard Turner. 2020. TaskNorm: Rethinking Batch Normalization for Meta-
Learning. In Proceedings of the 37th International Conference on Machine Learning
(Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh
(Eds.). PMLR, 1153–1164. https://proceedings.mlr.press/v119/bronskill20a.html
[12] Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of
the 2016 CHI Conference on Human Factors in Computing Systems (San Jose,
California, USA) (CHI ’16). Association for Computing Machinery, New York, NY,
USA, 981–992. https://doi.org/10.1145/2858036.2858498
[13] Kathleen Campbell, Kimberly LH Carpenter, Jordan Hashemi, Steven Espinosa,
Samuel Marsan, Jana Schaich Borg, Zhuoqing Chang, Qiang Qiu, Saritha Vermeer,
Elizabeth Adler, Mariano Tepper, Helen L Egger, Jefery P Baker, Guillermo
Sapiro, and Geraldine Dawson. 2019. Computer vision analysis captures atypical
attention in toddlers with autism. Autism 23, 3 (2019), 619–628. https://doi.
org/10.1177/1362361318766247 arXiv:https://doi.org/10.1177/1362361318766247
PMID: 29595333.
[14] Michelle Carney, Barron Webster, Irene Alvarado, Kyle Phillips, Noura Howell,
Jordan Grifth, Jonas Jongejan, Amit Pitaru, and Alexander Chen. 2020. Teachable
Machine: Approachable Web-Based Tool for Exploring Machine Learning Classi-
fcation. In Extended Abstracts of the 2020 CHI Conference on Human Factors in
Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing
Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3382839
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Im-
ageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on
Computer Vision and Pattern Recognition. 248–255. https://doi.org/10.1109/CVPR.
2009.5206848
[16] Voice Dream. 2022. Scanner – Voice Dream. https://www.voicedream.com/
scanner/
[17] Utkarsh Dwivedi, Jaina Gandhi, Raj Parikh, Merijke Coenraad, Elizabeth Bon-
signore, and Hernisa Kacorri. 2021. Exploring Machine Teaching with Children.
In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing
(VL/HCC). 1–11. https://doi.org/10.1109/VL/HCC51201.2021.9576171
Blind Users Accessing Their Training Images in Teachable Object Recognizers
[18] Alexander Fiannaca, Ilias Apostolopoulous, and Eelke Folmer. 2014. Headlock:
A Wearable Navigation Aid That Helps Blind Cane Users Traverse Large Open
Spaces. In Proceedings of the 16th International ACM SIGACCESS Conference on
Computers & Accessibility (Rochester, New York, USA) (ASSETS ’14). Association
for Computing Machinery, New York, NY, USA, 19–26. https://doi.org/10.1145/
2661334.2661453
[19] Nicholas A. Giudice, Benjamin A. Guenther, Toni M. Kaplan, Shane M. Anderson,
Robert J. Knuesel, and Joseph F. Ciof. 2020. Use of an Indoor Navigation System
by Sighted and Blind Travelers: Performance Similarities across Visual Status
and Age. ACM Trans. Access. Comput. 13, 3, Article 11 (aug 2020), 27 pages.
https://doi.org/10.1145/3407191
[20] Steven M. Goodman, Ping Liu, Dhruv Jain, Emma J. McDonnell, Jon E. Froehlich,
and Leah Findlater. 2021. Toward User-Driven Sound Recognizer Personalization
with People Who Are d/Deaf or Hard of Hearing. Proc. ACM Interact. Mob.
Wearable Ubiquitous Technol. 5, 2, Article 63 (jun 2021), 23 pages. https://doi.org/
10.1145/3463501
[21] Google. 2022. Lookout - Assisted vision - Apps on Google Play.
https://play.google.com/store/apps/details?id=com.google.android.apps.
accessibility.reveal&hl=en_US&gl=US
[22] Google. 2022. Teachable Machine. https://teachablemachine.withgoogle.com/.
[23] João Guerreiro, Daisuke Sato, Saki Asakawa, Huixu Dong, Kris M. Kitani, and
Chieko Asakawa. 2019. CaBot: Designing and Evaluating an Autonomous
Navigation Robot for Blind People. In The 21st International ACM SIGAC-
CESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (AS-
SETS ’19). Association for Computing Machinery, New York, NY, USA, 68–82.
https://doi.org/10.1145/3308561.3353771
[24] Anhong Guo, Xiang ’Anthony’ Chen, Haoran Qi, Samuel White, Suman Ghosh,
Chieko Asakawa, and Jefrey P. Bigham. 2016. VizLens: A Robust and Interactive
Screen Reader for Interfaces in the Real World. In Proceedings of the 29th Annual
Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16).
Association for Computing Machinery, New York, NY, USA, 651–664. https:
//doi.org/10.1145/2984511.2984518
[25] Tom Hitron, Yoav Orlev, Iddo Wald, Ariel Shamir, Hadas Erel, and Oren Zuck-
erman. 2019. Can Children Understand Machine Learning Concepts? The
Efect of Uncovering Black Boxes. In Proceedings of the 2019 CHI Conference
on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19).
Association for Computing Machinery, New York, NY, USA, 1–11. https:
//doi.org/10.1145/3290605.3300645
[26] Jonggi Hong, Kyungjun Lee, June Xu, and Hernisa Kacorri. 2019. Exploring
Machine Teaching for Object Recognition with the Crowd. In Extended Abstracts
of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow,
Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY,
USA, 1–6. https://doi.org/10.1145/3290607.3312873
[27] Jonggi Hong, Kyungjun Lee, June Xu, and Hernisa Kacorri. 2020. Crowdsourcing
the Perception of Machine Teaching. Association for Computing Machinery, New
York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376428
[28] Kristina Höök. 2000. Steps to take before intelligent user interfaces become real.
Interacting with computers 12, 4 (2000), 409–426. https://doi.org/10.1016/S0953-
5438(99)00006-5
[29] Virtual Collaboration Research Inc. 2022. Super Lidar - Lidar for Blind. https:
//apps.apple.com/us/app/super-lidar-lidar-for-blind/id1543706309
[30] Hairong Jiang, Ting Zhang, Juan P Wachs, and Bradley S Duerstock. 2016. En-
hanced control of a wheelchair-mounted robotic manipulator using 3-D vision
and multimodal interaction. Computer Vision and Image Understanding 149 (2016),
21–31. https://doi.org/10.1016/j.cviu.2016.03.015
[31] Hernisa Kacorri. 2017. Teachable Machines for Accessibility. SIGACCESS Access.
Comput. 119 (nov 2017), 10–18. https://doi.org/10.1145/3167902.3167904
[32] Hernisa Kacorri, Utkarsh Dwivedi, Sravya Amancherla, Mayanka Jha, and Riya
Chanduka. 2020. IncluSet: A Data Surfacing Repository for Accessibility Datasets.
In The 22nd International ACM SIGACCESS Conference on Computers and Accessi-
bility (Virtual Event, Greece) (ASSETS ’20). Association for Computing Machinery,
New York, NY, USA, Article 72, 4 pages. https://doi.org/10.1145/3373625.3418026
[33] Hernisa Kacorri, Utkarsh Dwivedi, and Rie Kamikubo. 2020. Data Sharing in
Wellness, Accessibility, and Aging. NeurIPS 2020 Workshop on Dataset Curation
and Security (2020).
[34] Hernisa Kacorri, Kris M. Kitani, Jefrey P. Bigham, and Chieko Asakawa. 2017.
People with Visual Impairment Training Personal Object Recognizers: Feasibility
and Challenges. In Proceedings of the 2017 CHI Conference on Human Factors in
Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing
Machinery, New York, NY, USA, 5839–5849. https://doi.org/10.1145/3025453.
3025899
[35] Seita Kayukawa, Keita Higuchi, João Guerreiro, Shigeo Morishima, Yoichi Sato,
Kris Kitani, and Chieko Asakawa. 2019. BBeep: A Sonic Collision Avoidance
System for Blind Travellers and Nearby Pedestrians. In Proceedings of the 2019
CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland
Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12.
https://doi.org/10.1145/3290605.3300282
[36] Taha Khan, Dag Nyholm, Jerker Westin, and Mark Dougherty. 2014. A computer
vision framework for fnger-tapping evaluation in Parkinson’s disease. Artifcial
ASSETS ’22, October 23–26, 2022, Athens, Greece
intelligence in medicine 60, 1 (2014), 27–40. https://doi.org/10.1016/j.artmed.2013.
11.004
[37] Ron Kimmel and Alfred M Bruckstein. 2003. Regularized Laplacian zero crossings
as optimal edge integrators. International Journal of Computer Vision 53, 3 (2003),
225–243. https://doi.org/10.1023/A:1023030907417
[38] Masaki Kuribayashi, Seita Kayukawa, Hironobu Takagi, Chieko Asakawa, and
Shigeo Morishima. 2021. LineChaser: A Smartphone-Based Navigation System
for Blind People to Stand in Lines. In Proceedings of the 2021 CHI Conference on
Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association
for Computing Machinery, New York, NY, USA, Article 33, 13 pages. https:
//doi.org/10.1145/3411764.3445451
[39] Google Creative Lab. 2017. Teachable Machine. https://teachablemachine.
withgoogle.com/v1/.
[40] Kyungjun Lee, Jonggi Hong, Ebrima Jarjue, Ernest Essuah Mensah, and Hernisa
Kacorri. 2022. From the Lab to People’s Home: Lessons from Accessing Blind
Participants’ Interactions via Smart Glasses in Remote Studies. In Proceedings of
the 19th International Web for All Conference (Lyon, France) (W4A ’22). Association
for Computing Machinery, New York, NY, USA, Article 24, 11 pages. https:
//doi.org/10.1145/3493612.3520448
[41] Kyungjun Lee, Jonggi Hong, Simone Pimento, Ebrima Jarjue, and Hernisa Ka-
corri. 2019. Revisiting Blind Photography in the Context of Teachable Object
Recognizers. In The 21st International ACM SIGACCESS Conference on Computers
and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing
Machinery, New York, NY, USA, 83–95. https://doi.org/10.1145/3308561.3353799
[42] Kyungjun Lee and Hernisa Kacorri. 2019. Hands Holding Clues for Object
Recognition in Teachable Machines. In Proceedings of the 2019 CHI Confer-
ence on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI
’19). Association for Computing Machinery, New York, NY, USA, 1–12. https:
//doi.org/10.1145/3290605.3300566
[43] Kyungjun Lee, Daisuke Sato, Saki Asakawa, Hernisa Kacorri, and Chieko
Asakawa. 2020. Pedestrian Detection with Wearable Cameras for the Blind: A
Two-Way Perspective. Association for Computing Machinery, New York, NY, USA,
1–12. https://doi.org/10.1145/3313831.3376398
[44] James R Lewis. 1995. IBM computer usability satisfaction questionnaires:
psychometric evaluation and instructions for use. International Journal of
Human-Computer Interaction 7, 1 (1995), 57–78. https://doi.org/10.1080/
10447319509526110
[45] Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell.
2017. Understanding Blind People’s Experiences with Computer-Generated
Captions of Social Media Images. In Proceedings of the 2017 CHI Conference
on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17).
Association for Computing Machinery, New York, NY, USA, 5988–5999. https:
//doi.org/10.1145/3025453.3025814
[46] Cristina Manresa-Yee, Javier Varona, Francisco J Perales, and Iosune Salinas.
2014. Design recommendations for camera-based head-controlled interfaces that
replace the mouse for motion-impaired users. Universal access in the information
society 13, 4 (2014), 471–482. https://doi.org/10.1007/s10209-013-0326-z
[47] Daniela Massiceti, Lida Theodorou, Luisa Zintgraf, Matthew Tobias Harris, Si-
mone Stumpf, Cecily Morrison, Edward Cutrell, and Katja Hofmann. 2021. ORBIT:
A real-world few-shot dataset for teachable object recognition collected from
people who are blind or low vision. https://doi.org/10.25383/CITY.14294597
[48] Mediate. 2022. Supersense - AI for Blind / Scan text, money and objects. https:
//www.supersense.app/
[49] Carrie Morales. 2019. What’s Better for the Blind and Low Vision? Android or
iPhone? https://liveaccessible.com/2019/03/03/whats-better-for-the-blind-and-
low-vision-android-or-iphone/
[50] John Morris and James Mueller. 2014. Blind and deaf consumer preferences for
android and iOS smartphones. In Inclusive designing. Springer, 69–79. https:
//doi.org/10.1007/978-3-319-05095-9_7
[51] Meredith Ringel Morris. 2020. AI and Accessibility. Commun. ACM 63, 6 (2020),
35–37. https://doi.org/10.1145/3356727
[52] Donald A Norman. 1994. How might people interact with agents. Commun. ACM
37, 7 (1994), 68–71. https://doi.org/10.1145/176789.176796
[53] Lauren Olson, Chandra Kambhamettu, and Kathleen McCoy. 2021. Towards Using
Live Photos to Mitigate Image Quality Issues In VQA Photography. Association for
Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3441852.
3476541
[54] Thomas J. Palmeri and Isabel Gauthier. 2004. Visual object understanding. Nature
Reviews Neuroscience 5, 4 (2004), 291–303. https://doi.org/10.1038/nrn1364
[55] Rupal Patel and Deb Roy. 1998. Teachable interfaces for individuals with
dysarthric speech and severe physical disabilities. In Proceedings of the AAAI
Workshop on Integrating Artifcial Intelligence and Assistive Technology. Citeseer,
40–47.
[56] Rubens Lacerda Queiroz, Fábio Ferrentini Sampaio, Cabral Lima, and Priscila
Machado Vieira Lima. 2020. AI from concrete to abstract: demystifying artifcial
intelligence to the general public. https://doi.org/10.1007/s00146-021-01151-x
arXiv:2006.04013 [cs.CY]
ASSETS ’22, October 23–26, 2022, Athens, Greece
[57] Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement.
https://doi.org/10.48550/ARXIV.1804.02767
[58] Alejandro Reyes-Amaro, Yanet Fadraga-González, Oscar Luis Vera-Pérez, Eliz-
abeth Domínguez-Campillo, Jenny Nodarse-Ravelo, Alejandro Mesejo-Chiong,
Biel Moyà-Alcover, and Antoni Jaume-i Capó. 2012. Rehabilitation of patients
with motor disabilities using computer vision based techniques. Journal of acces-
sibility and design for all 2, 1 (2012), 62–70. https://doi.org/10.17411/jacces.v2i1.87
[59] André Rodrigues, André Santos, Kyle Montague, Hugo Nicolau, and Tiago Guer-
reiro. 2019. Understanding the Authoring and Playthrough of Nonvisual Smart-
phone Tutorials. In Human-Computer Interaction – INTERACT 2019, David Lamas,
Fernando Loizides, Lennart Nacke, Helen Petrie, Marco Winckler, and Panayiotis
Zaphiris (Eds.). Springer International Publishing, Cham, 42–62.
[60] Manaswi Saha, Alexander J. Fiannaca, Melanie Kneisel, Edward Cutrell, and
Meredith Ringel Morris. 2019. Closing the Gap: Designing for the Last-Few-
Meters Wayfnding Problem for People with Visual Impairments. In The 21st
International ACM SIGACCESS Conference on Computers and Accessibility (Pitts-
burgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York,
NY, USA, 222–235. https://doi.org/10.1145/3308561.3353776
[61] Elliot Salisbury, Ece Kamar, and Meredith Ringel Morris. 2017. Toward Scalable
Social Alt Text: Conversational Crowdsourcing as a Tool for Refning Vision-to-
Language Technology for the Blind. In HCOMP.
[62] Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. 2017. Explainable
Artifcial Intelligence: Understanding, Visualizing and Interpreting Deep Learning
Models. CoRR abs/1708.08296 (2017). arXiv:1708.08296 http://arxiv.org/abs/1708.
08296
[63] SeeingAI. 2017. An app for visually impaired people that narrates the world around
you. https://www.microsoft.com/en-us/seeing-ai
[64] Claude Elwood Shannon. 2001. A mathematical theory of communication. ACM
SIGMOBILE mobile computing and communications review 5, 1 (2001), 3–55.
[65] Patrice Y. Simard, Saleema Amershi, David M. Chickering, Alicia Edelman Pelton,
Soroush Ghorashi, Christopher Meek, Gonzalo Ramos, Jina Suh, Johan Verwey,
Mo Wang, and John Wernsing. 2017. Machine Teaching: A New Paradigm for
Building Machine Learning Systems. https://doi.org/10.48550/ARXIV.1707.06742
[66] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep Inside
Convolutional Networks: Visualising Image Classifcation Models and Saliency
Maps. https://doi.org/10.48550/ARXIV.1312.6034
[67] Hojun Son, Divya Krishnagiri, V. Swetha Jeganathan, and James Weiland. 2020.
Crosswalk Guidance System for the Blind. In 2020 42nd Annual International
Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). 3327–
3330. https://doi.org/10.1109/EMBC44109.2020.9176623
[68] Joan Sosa-García and Francesca Odone. 2017. “Hands On” Visual Recognition
for Visually Impaired Users. ACM Trans. Access. Comput. 10, 3, Article 8 (Aug.
2017), 30 pages. https://doi.org/10.1145/3060056
[69] Pierre Stock and Moustapha Cisse. 2018. ConvNets and ImageNet Beyond Accu-
racy: Understanding Mistakes and Uncovering Biases. In The European Conference
on Computer Vision (ECCV).
[70] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. 2019. Meta-transfer
learning for few-shot learning. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition. 403–412.
[71] Ruxandra Tapu, Bogdan Mocanu, and Titus Zaharia. 2019. DEEP-HEAR: A
Multimodal Subtitle Positioning System Dedicated to Deaf and Hearing-Impaired
People. IEEE Access 7 (2019), 88150–88162. https://doi.org/10.1109/ACCESS.2019.
2925806
Hong, et al.
[72] Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily Morri-
son, Edward Cutrell, Matthew Tobias Harris, and Katja Hofmann. 2021. Disability-
First Dataset Creation: Lessons from Constructing a Dataset for Teachable Object
Recognition with Blind and Low Vision Data Collectors. In The 23rd International
ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA)
(ASSETS ’21). Association for Computing Machinery, New York, NY, USA, Article
27, 12 pages. https://doi.org/10.1145/3441852.3471225
[73] David Touretzky. 2019. The AI4K12 Initiative: Developing National Guidelines
for Teaching AI In K-12. https://github.com/touretzkyds/ai4k12/blob/master/
documents/CSTA_2019_How_To_Teach_AI_Across_K-12.pdf.
[74] David Touretzky. 2020. The AI4K12 Initiative: Developing National Guidelines
for Teaching AI In K-12. https://raw.githubusercontent.com/touretzkyds/ai4k12/
master/documents/GlobalSWEdu2020_Touretzky.pdf.
[75] Henriikka Vartiainen, Matti Tedre, and Teemu Valtonen. 2020. Learning machine
learning with very young children: Who is teaching whom? International Journal
of Child-Computer Interaction 25 (Sept. 2020), 1–11. https://linkinghub.elsevier.
com/retrieve/pii/S2212868920300155
[76] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan
Wierstra. 2016. Matching Networks for One Shot Learning. In Advances in
Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg,
I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 3630–3638. http://papers.
nips.cc/paper/6385-matching-networks-for-one-shot-learning.pdf
[77] Vuzix. 2021. Vuzix Blade Smart Glasses. https://www.vuzix.com/products/blade-
smart-glasses-upgraded
[78] Yutaro Yamanaka, Seita Kayukawa, Hironobu Takagi, Yuichi Nagaoka, Yoshimune
Hiratsuka, and Satoshi Kurihara. 2022. One-Shot Wayfnding Method for Blind
People via OCR and Arrow Analysis with a 360-Degree Smartphone Camera. In
Mobile and Ubiquitous Systems: Computing, Networking and Services, Takahiro
Hara and Hirozumi Yamaguchi (Eds.). Springer International Publishing, Cham,
150–168. https://doi.org/10.1007/978-3-030-94822-1_9
[79] Guangxiao Zhang, Zhuolin Jiang, and Larry S Davis. 2012. Online semi-supervised
discriminative dictionary learning for sparse representation. In Asian conference
on computer vision. Springer, 259–273. https://doi.org/10.1007/978-3-642-37331-
2_20
[80] Gongjie Zhang, Zhipeng Luo, Kaiwen Cui, and Shijian Lu. 2021. Meta-detr:
Few-shot object detection via unifed image-level meta-learning. arXiv preprint
arXiv:2103.11731 (2021).
[81] Yuhang Zhao, Elizabeth Kupferstein, Brenda Veronica Castro, Steven Feiner, and
Shiri Azenkot. 2019. Designing AR Visualizations to Facilitate Stair Navigation
for People with Low Vision. In Proceedings of the 32nd Annual ACM Symposium
on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19).
Association for Computing Machinery, New York, NY, USA, 387–402. https:
//doi.org/10.1145/3332165.3347906
[82] Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2018. A Face
Recognition Application for People with Visual Impairments: Understanding Use
Beyond the Lab. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing
Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3173574.3173789
[83] Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna N. Raferty. 2018. An Overview
of Machine Teaching. CoRR abs/1801.05927 (2018). arXiv:1801.05927 http:
//arxiv.org/abs/1801.05927
[84] Zoom. 2022. Video Conferencing, Cloud Phones, Webinars, Chat, Virtual Events.
https://zoom.us/
 Challenging and Improving Current Evaluation Methods for
Colour Identification Aids
Connor Geddes
geddesc@uoguelph.ca
School of Computer Science
University of Guelph
Guelph, Ontario, Canada
ABSTRACT
Identifcation of and discrimination between colours is an impor-
tant task in everyday life, but for the 5% of the population who
have Colour Vision Defciency (CVD), correctly identifying or dis-
criminating between colours can be difcult or impossible. Colour
Identifcation (or CVD) Aids have been developed to assist people
with CVD, however, the methods used to evaluate them are often
limited and many use CVD simulations instead of participants with
CVD. To address this, we propose two new CVD Aid evaluation
tasks and show that they can assist in providing a more thorough
evaluation of potential CVD aids. In addition, we evaluate the efec-
tiveness of CVD simulations used by non-CVD people in providing
results similar to those for people with CVD, and found that both
the results and participant behaviour often difered. Our results
indicate that greater care is needed when evaluating CVD Aids.
CCS CONCEPTS
• Human-centered computing → Accessibility.
KEYWORDS
Colour Vision Defciency, Colour Vision Defciency Simulation
ACM Reference Format:
Connor Geddes and David R. Flatla. 2022. Challenging and Improving Cur-
rent Evaluation Methods for Colour Identifcation Aids. In The 24th Interna-
tional ACM SIGACCESS Conference on Computers and Accessibility (ASSETS
’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 12 pages.
https://doi.org/10.1145/3517428.3544818
1 INTRODUCTION
Colour Vision Defciency (CVD) afects an individual’s ability to
accurately diferentiate and name colours. Failure to correctly deter-
mine colours can result in frustrating or dangerous consequences.
For example, taking the wrong route on a train/subway by misin-
terpreting a colour encoding or missing the dangerous naturally
colour coded medical symptom of blood in the stool can both result
from an inability to correctly distinguish colours.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544818
David R. Flatla
dfatla@uoguelph.ca
School of Computer Science
University of Guelph
Guelph, Ontario, Canada
Colour Identifcation (or CVD) Aids have been developed that
look to assist people with CVD, however, determining the potential
efectiveness of these aids is commonly performed via diagnostic
tests like the Ishihara plate test [29] or the FM-100 hue test [15].
Other tasks such as judging natural photos with the aid applied
to them, the Colour Matching Task [16], and the Odd one out
Colour Task [17] have been developed to be used to verify colour
identifcation aids. However, these methods of evaluation all fall
short in that they only cover one type (Comparative Judgement)
of the four types of colour judgement that individuals with CVD
struggle with as defned by Cole [9]: Comparative, Connotative,
Denotative, and Aesthetic.
Three recently developed colour identifcation tasks were de-
signed to address more of Cole’s types of colour judgement: Selec-
tion Task, Transition Task, and Sorting Task [20]. However, these
new tasks do not evaluate connotative judgement of colour (match-
ing colour to a meaning/category). To address this, we developed
a new connotative colour identifcation task called Category Task.
In addition, we also created another task (called Distinction Task),
which looks to identify the number of distinct colours an aid allows
an individual with CVD to correctly identify.
In addition, several previous evaluations of CVD Aids have used
CVD simulations with non-CVD participants (or with the authors
or readers serving as the only evaluators) as stand-ins for individ-
uals with CVD [1, 3, 7, 8, 13, 25–27, 30, 41, 43, 53, 54], a technique
that has come to be referred to as ‘indirect’ evaluation [57]. Further-
more, most research that looks to develop colour identifcation aids
uses CVD simulations as the core component of their respective
evaluations and rarely include people with CVD in the evaluation at
all, and then often only tangentially (e.g., asking for preferences on
which CVD Aid flter looks the best, but not determining efective-
ness in realistic or abstract tasks). Of the hundreds of publications
in this research area, very few studies [16, 17, 20, 24, 48] have gone
through the efort to develop tasks for a comprehensive user study
that fully evaluates colour identifcation aids while simultaneously
involving those with CVD as crucial contributors towards deter-
mining the efectiveness of the aid.
To determine how efective ‘indirect’ evaluations are at mimick-
ing genuine issues for those with CVD, we evaluated the use of
CVD simulations with non-CVD participants on the three previ-
ously developed tasks: Selection Task, Transition Task, and Sorting
Task, as well as our two new tasks: Distinction Task and Category
Task. As it is often unclear how much information non-CVD partic-
ipants are provided regarding the vision of individuals with CVD
during ’indirect evaluations’, we also provided three diferent levels
of simulation information: No Information, Minimum Information
(where they were informed that they are under a CVD simulation
ASSETS ’22, October 23–26, 2022, Athens, Greece
and colour will appear diferent), and Maximum Information (where
they were provided details of exactly which colours individuals with
CVD tend to confuse and why). Our ground truth data was obtained
from actual people with a variety of types and severities of CVD.
We employed the Machado CVD simulation method [38], due
to its rising popularity in industry (both Chrome and Firefox have
integrated it into their browser devtools [21, 22, 42]), its popularity
in research studies [35, 58], and its ability to simulate a range of
CVD severities. As people with CVD are often diagnosed into one
of three categories of severity, we opted to include three simulation
severity conditions in order to not unfairly compare a more severe
CVD simulation to a ‘mild’ CVD participant. To do this, we used
the 0.3 severity for mild CVD, the 0.6 severity for moderate CVD,
and the 1.0 severity for strong CVD (dichromacy) in the Machado
simulation.
Our fndings indicate signifcant diferences between the results
for all simulation conditions and the results for CVD participants.
We also observed no diference or very minor diferences between
the diferent simulation information conditions. These fndings
lead us to conclude that CVD simulations – even with the addition
of detailed information – cannot hope to match years of genuine
experience living with CVD.
This paper makes two important contributions: 1) Two new tasks
for evaluating CVD Aids, with empirical evidence that suggests that
they should help generalize current evaluation techniques to better
match the diversity of challenges experienced by people with CVD;
and 2) Clear empirical evidence that we (as a discipline) should NOT
be evaluating CVD Aids by using non-CVD participants under CVD
simulations, no matter the severity, type, or amount of contextual
information provided.
2 RELATED WORK
2.1 Colour Vision Defciency
To identify colour, we use retinal cone cells (or cones) that are sensi-
tive to diferent wavelengths of visible light. Human eyes typically
contain long (Red), medium (Green), and short (Blue) wavelength
sensitive cone types [10], hence are described as ‘trichromatic’.
When one or more of the cone types are either missing, damaged,
or have variant sensitivity, it results in Colour Vision Defciency
(CVD) [5]. Three main classes of CVD exist, each depending on how
the cones are afected: 1) Anomalous Trichromacy (one variant-
sensitivity cone - approximately 3/4 of all cases of inherited CVD),
2) Dichromacy (one cone missing - approximately 1/4 of all cases of
inherited CVD), and 3) Monochromacy (two or three cones missing -
very rare). Anomalous Trichromacy and Dichromacy have three dif-
ferent subtypes depending on the cone that is afected: Protanomaly
and Protanopia for the long wavelength sensitive cone, Deutera-
nomaly and Deuteranopia for the medium wavelength sensitive
cone, and Tritanomaly and Tritanopia for the short wavelength
sensitive cone. CVD can be either inherited or acquired. Inherited
CVD most commonly afects the long (1/4 of cases) and medium
wavelength (3/4 of cases) cones resulting in diminished colour dis-
tinguishability along the red-green axis, leading to the common
misnomer ‘red-green colourblindness’. Acquired CVD most com-
monly afects the short wavelength cone, and can be the result of
Connor Geddes and David R. Flatla
age [12, 28, 31], chemical exposure [14, 36], brain injury [56], and
disease [19, 45].
a b
c d
Figure 1: Illustrating Machado’s Colour Vision Defciency
simulation (deutan): a) no simulation, b) severity = 0.3, c)
severity = 0.6; d) severity = 1.0 (or dichromatic).
2.2 Simulating Colour Vision Defciency
CVD simulations have been designed to help non-CVD people
understand how people with CVD see the world, and are often
used to help identify accessibility concerns that could afect people
with CVD. In order to simulate CVD, the established approach is to
frst convert an RGB colour into LMS colour space (corresponding
to long, medium, and short wavelength cone stimulation levels).
Next, the data for the cone corresponding to the type of CVD to
be simulated is replaced with the data from another cone (e.g.,
the middle-wavelength cone data is overwritten with the long-
wavelength cone data for Deuteranopia). Finally, the modifed LMS
colour is converted back to RGB.
A variety of CVD simulations have been developed, ranging from
academic research projects to personal-interest projects. Meyer and
Greenberg [40] proposed the frst algorithm for CVD simulation,
in which they simulated CVD perception in CIE XYZ color space
after making the appropriate adjustments to account for the vision
of individuals with dichromatic vision in LMS space.
Brettel et al. [6] developed a simulation to model the appearance
of colours for people with dichromacy. They based their simulation
on previous work that identifed colours that are perceived identi-
cally by people with unilateral dichromacy (dichromatic vision in
one eye and trichromatic vision in the other) [23]. For protanopia
and deuteranopia, their simulation projects all colours onto two
half-planes that stretch from the achromatic (greyscale) axis to
either a 475nm (blue) or a 575nm (yellow) anchor. Vienot et al. [52]
simplifed the simulation process for Brettel’s method.
Kondo [32] developed an early simulation method that mod-
elled the vision of anomalous trichromacy based on established
Challenging and Improving Current Evaluation Methods for Colour Identification Aids
dichromacy simulation methods. Yang et al. [55] also attempted to
simulate anomalous trichromacy by attempting LMS space adjust-
ments at the photoreceptor level.
Machado et al. [38] based their simulation method on physiolog-
ical principles around how cones shift in sensitivity in anomalous
trichromacy, taking into account a novel ‘stage theory’ of human
colour vision. Their model simulates the vision of various nanome-
ter shifts in the sensitivity of the anomalous cones, allowing the
simulation of a variety of severities of anomalous trichromacy. Fig-
ure 1 contains a demonstration of Machado’s deutan CVD simula-
tion with increasing severity level. Flatla & Gutwin [18] developed
a personalized simulation adapted to an individual via a calibra-
tion process, and later favourably compared their simulation to
Machado’s adjustable simulation [37], however their personalized
simulation is not tunable to arbitrary severities.
The ‘ColourMatrix’ implementation is a method that roughly
simulates CVD, which, despite its questionable implementation and
having little scientifc foundation [2], has gained in popularity. The
HCIRN simulation 1 is another method that simulates CVD based on
the Meyer and Greenberg method [40] and is the method employed
by the popular Coblis online simulator 2 . However, similar to the
‘ColourMatrix’ method, the HCIRN method has been noted to have
issues with validity [39, 44].
2.3 Criticisms of Using Disability Simulations
Other disability simulation methods have been present in the acces-
sibility literature (e.g., using a blindfold to approximate blindness
during design, development, or evaluation), with the goal being
more inclusive designs, and what could be characterized as a ‘better-
than-nothing’ attitude. Sears and Hanson [49] argue that it might
be acceptable to do pilot testing with non-representative (i.e., sim-
ulated) users, however, in studies that target archival publication,
representative users must be recruited.
More recently, Bennett & Rosner [4] discuss how empathy build-
ing through simulations might threaten inclusive design because it
can lead to the wholesale exclusion of individuals with impairments
from the design work because the team already has an (artifcially in-
fated) ‘understanding’ of those individuals through the simulation.
Likewise, Tigwell [51] found that very few designers understand
the problems that can arise from over-reliance on disability simula-
tions, and that designers tend to believe that disability simulations
provide ‘sufcient validation’ for their fnal design.
In summary, Bennett & Rosner [4] and Tigwell [51] both recom-
mend that collaboration between individuals with disabilities and
designers is necessary for the creation of designs and interfaces that
better refect the true life experiences of people with disabilities.
2.4 Colour Identifcation Aids and the Methods
used to Evaluate them
Colour Identifcation Aids have been developed to assist people with
CVD. There are three categories for CVD aids: haptic aids, visually
modifying aids, and sound based aids. Haptic colour identifcation
aids generally use patterns of vibration and push feedback in order
signify a specifc colour. Visual aids either: 1) recolour the view
1 http://colorlab.wickline.org/colorblind/colorlab/docs/acknowledgments.html
2 https://www.color-blindness.com/coblis-color-blindness-simulator/
ASSETS ’22, October 23–26, 2022, Athens, Greece
in order to clarify colours of confusion and increase contrast, 2)
overlay patterns to signify specifc colours or signify a diference, or
3) highlight a specifc colour. Sound based aids use diferent sounds
to signify specifc colours.
The common tasks for evaluating aids to determine whether
they are benefcial to individuals with CVD or if they improve on
previous methods are: 1) Ishihara plate test [3, 8, 13, 27, 29, 30, 33,
34, 41, 43, 46, 50, 53, 57, 58], 2) FM-100 Hue test [15, 57], 3) Colour
matching test [16], 4) Image ranking [3, 13, 25, 26, 33, 43, 50, 54,
57, 58], 5) Odd-one-out colour task [17], 6) Selection Task [20], 7)
Transition Task [20], and 8) Sorting Task [20].
Most tasks that are used in the evaluation of CVD aids do not
adequately address all four types of colour challenges that people
with CVD face, as defned by Cole [9]. These four challenges are:
(1) Comparative: Where colours must be compared to fnd simi-
larities or diferences (e.g., matching two fabrics).
(2) Denotative: Where colour is defned or referenced only by
name (e.g., "Mine is the red car.").
(3) Connotative: Where colour can have a meaning or be as-
signed a meaning (e.g., Yellow bananas are ripe).
(4) Aesthetic: Where colour can be used to convey beauty (e.g.,
in art or fashion).
Most CVD Aid evaluation tasks tend to only cover comparative
colour challenges, however, some more recent tasks (Selection and
Sorting) also cover denotative uses of colour.
However, a task that covers connotative uses of colour has not
been developed yet, nor has a task that allows for an evaluation
of the number of simultaneous variations of specifc colour names
that can be accurately distinguished. The beneft of having a task
that assesses the number of distinct colours that an aid enables
to be distinguished gives a clear metric that not only allows for
comparison of aids but also to determine how efective the aids can
be in the most extreme of circumstances.
Finally, the common approach to evaluating colour identifcation
aids is the ‘indirect’ evaluation [57], which bypasses recruiting par-
ticipants with CVD by employing CVD simulations and showing
them to non-CVD participants (sometimes just the research team
or reader of the paper). This problem is very pervasive – ‘indirect’
evaluation form a core component of almost every paper looking
to assist people with CVD. However, some papers do employ CVD
participants, but often only by treating CVD participants as tokens
for verifying the results of a tool rather than for determining ef-
fectiveness or obtaining feedback from the target population. Such
tokenizing is employed in the most commonly used evaluation
method when CVD participants are involved, the Image Ranking
Task. The Image Ranking Task has CVD participants rank output
images from several visual aids (usually recolouring) simply based
on which looks the ‘best’. We share the concerns expressed else-
where in the accessibility literature (see Section 2.3) regarding this
approach.
In this paper, we look to address the lack of a connotative CVD
Aid evaluation task, plus the lack of an extreme task that allows for
the determination of the number of colours that can be simultane-
ously distinguished. In addition, we also look to address whether
‘indirect’ evaluations are useful and what their limitations are. We
ASSETS ’22, October 23–26, 2022, Athens, Greece Connor Geddes and David R. Flatla

make use of recently developed colour identifcation tasks – Selec- 3.2 Distinction Task
tion, Transition and Sorting – to help us with this and we make use
of the Machado CVD simulation method [38] as it is the state-of-the-
art. We chose to utilize the three previous methods from [20] over
the Ishihara Test as although the Ishiraha Test garners far more
use in the related work, its value as a CVD Aid evaluation task is
extremely limited due to the very fne-tuned colours used in the
Ishihara Test (precisely chosen to identify diferent types of CVD) –
essentially any chromatic modifcation of an Ishihara plate will ren-
der its hidden number visible to someone with CVD. This presents
such a low barrier to ‘success’ that it leaves the Ishihara Test almost
useless as a CVD Aid evaluation task. We did not employ the Image
Ranking Task due to: a lack of standardized set of images (previous
studies vary widely on the images used), the ‘entire-image’ nature
of this task (which prevents assessing component diferences, such
as colour naming performance), and the highly subjective nature
of this task (e.g., no clear defnition of ‘best’ exists for this task).
Figure 3: Example of Distinction Task shown with nine total
3 PROPOSED CVD EVALUATION METHODS ‘distinct’ hues. In the above grid the hues of ‘red’, ‘orange’,
‘yellow’, ‘green’, ‘blue’, ‘purple’, ‘pink’, ‘grey’ and ‘brown’ are
3.1 Category Task shown.

We designed the Distinction Task to create a task that allowed for

Figure 2: Example of Category Task. In this example, the
correct category is Category 6 for the colour ‘purple’, which
matches the slightly diferent purple in the top-right.
The category task was designed to address the lack of a task
that featured connotative judgement of colour [9]. We designed
the category task to function very similar to that of a pH strip test,
where a target category colour (indicated at the top in the task) must
be matched to a corresponding coloured category in a coloured
category grid. A key aspect of this task is that the colour that is
required to be matched to a category will never exactly match the
correct corresponding category. Figure 2 illustrates this idea where
the colour to match is a slightly more vibrant ‘purple’ than the
corresponding ’purple’ in the correct category (Category 6).
3.1.1 Category Task Scoring. We scored the Category Task simply
based on the index of the selected category and the index of the
answer for the task. This resulted in every trial being either correct
or incorrect.
the number of distinct colour names/hue categories an aid enabled
an individual with CVD to distinguish to be assessed. This allows
a unique opportunity for potential aids to compare the number of
relative colour names they accurately allow individuals with CVD
to distinguish simultaneously. Research looking to improve upon
recolouring aids often uses subjective judgement and/or contrast
measurements to determine the relative success of a novel technique
over existing methods. While we believe the ability of an aid to
improve the contrast between two colours is very important, we also
believe there is great beneft in creating this task which enables an
assessment of the number of distinct colours that can be accurately
named by an individual with CVD to be equally important. The
design of the Distinction Task is inspired by the Selection Task,
where the Distinction Task features a large number of coloured
swatches in grid of colours. The bottom features a grid of check
boxes indicating the distinct colour names to be identifed from the
the grid.
3.2.1 Distinction Task Scoring. We scored the Distinction Task
very similarly to how the Selection Task was reported to be scored
in [20], due to both the Selection Task and the Distinction Task
allowing participants to select more distinct colours than the correct
answer. If the participant selects a total number of colour names
equal to or less than the number of ‘true’ answers, then accuracy is
calculated by dividing the total number of distinct colour names
the participant got correct by the total correct number of distinct
colours. However, if the participant selected a number of distinct
colours greater than the number of ‘true’ colours, than the accuracy
is calculated by dividing the number of distinct colours a participant
got correct by the total number of distinct colours selected.
4 ONLINE EVALUATION
We conducted an online evaluation in order to test the efectiveness
of our new methods as potential techniques that could be utilized for
Challenging and Improving Current Evaluation Methods for Colour Identification Aids
evaluating CVD Aids by comparing the performance of participants
with and without CVD on a collection of tasks. Simultaneously,
we also evaluated the efectiveness of ‘indirect’ evaluations by
obtaining results from non-CVD participants under a variety of
CVD simulation severities to compare to the CVD participant data.
4.1 Participants
We conducted a remote online evaluation with a total of 284 partic-
ipants recruited from Reddit and Facebook. Of our 284 participants,
26 had CVD, consisting of 16 deutan (mild = 4, moderate = 10, strong
= 2) CVD participants and 10 protan (mild = 2, moderate = 4, strong
= 4) CVD participants. CVD type and severity was self-reported
by participants, as we recruited our participants from social me-
dia groups with members that are typically very knowledgeable
about their colour vision. With this in mind, we sub-sampled the
remaining 258 non-CVD participants (most of whom did the study
under CVD simulation) to ensure similar distributions in CVD type
and severity between the CVD group and the non-CVD simula-
tion groups. We did this to avoid unfairly biasing our study for or
against CVD simulations (e.g., by comparing performance under
dichromatic simulations to the performance of participants with
mild CVD). In particular, in each of the three level-of-information
conditions, we ensured we chose a total of 26 (non-CVD under
CVD simulation) participants such that 16 had used a deutan simu-
lation with 4 using mild (0.3), 10 moderate (0.6) and 2 strong (1.0)
simulations, and 10 used the protan simulation with 2 having mild
(0.3), 4 having moderate (0.6), and 4 having strong (1.0) simulations.
We further tried to maintain consistency by attempting to match
participant reported ages, the device used, environment (indoor
vs. outdoor) and lighting (natural vs. artifcial).
After the subsampling, we ended up with 130 participants total
(5 groups of 26 each: non-CVD control, CVD, plus three level-of-
information CVD simulation groups). Self-reported sex was roughly
balanced (male = 59, female = 61, and prefer not to disclose = 10),
although the CVD group was likely predominantly biologically
male due to inherited CVD being sex-linked. We had 55 participants
between the ages of 18-24, 57 between the ages of 25-34, 10 between
the ages of 35-44 and 8 between the ages of 45-64. Participants used
a mix of devices to complete the study with 76 using mobile devices,
27 using laptops, 25 using a desktop or monitor and 2 using a tablet.
Finally, all participants completed the study indoors, with 84 doing
so under artifcial lighting, 41 doing so under natural lighting and
5 who reported they did not know.
4.2 Task, Stimuli & Apparatus
Our study consisted of fve colour-related tasks that tested various
ways of interpreting and determining colours. The task order was
fxed so that participants were always given tasks in the order of
1) Selection Task, 2) Transition Task, 3) Sorting Task, 4) Distinc-
tion Task and 5) Category Task. The specifc implementations and
colours used for each task are described in each subsection below.
To evaluate the efectiveness of ‘indirect’ evaluations we utilized
the Machado CVD simulation method [38], specifcally we used
both the Deutan and Protan simulation. We decided on using the
Machado simulation method as the Machado simulation has gained
alot of attention due to its severity specifcation potentially allowing
ASSETS ’22, October 23–26, 2022, Athens, Greece
Red Orange Yellow Green Blue
Purple Pink Brown Grey Black
Figure 4: Colours generated for the tasks
for non-CVD people to better see how individuals with anomalous
trichomacy see colours. We decided to add three diferent levels of
severity of the Machado CVD simulation with roughly equal steps:
1) 0.3 - mild CVD, 2) 0.6 - moderate CVD, and 3) 1.0 - strong CVD.
To test the potential conditions for how simulations could be
employed in studies utilizing ’indirect’ evaluation we decided on
three levels of living-with-CVD information provided to the par-
ticipant using the simulation: 1) None, 2) Minimal, 3) Maximum
(Images for the Minimal and Maximum conditions are available
in the supplementary material). In the condition where no infor-
mation was provided, participants would not be informed of any
simulation that occurred within the study. In the minimal condition,
participants were simply informed that they were under a CVD
simulation and that colours may not appear as they normally do.
Finally, in the maximum condition, participants were informed that
they were under a simulation and were informed what colours
that individuals with CVD tend to confuse (e.g., Purples lose their
‘redness’, so are often seen as shades of blue).
4.2.1 Colours Generated. First, we modifed the ‘ColourIconizer_
Props’ colour name dictionary published in the supplementary
material of [20] by splitting all ‘grey’ proportions into ‘black’,
‘white’, and ‘grey’. We did this in recognition that those with
red-green CVD can confuse reds and blacks, plus whites and
cyans/aquamarines/teals. As such, we opted to include ‘black’ and
‘white’ to improve the generalizability of our evaluation tasks.
Next, we subsampled the dictionary resulting from the above
step for RGB colours that had a proportion value of >= 0.90 for
their dominant colour term. The resulting set thus contained only
colours that people without CVD should easily and consistently
name (i.e., at least 90% of the population should agree that it is the
same name). Unfortunately, this eliminated ‘white’ and ‘teal’ as
there were no RGB colours that had either of these terms above
0.90.
ASSETS ’22, October 23–26, 2022, Athens, Greece
As we wanted a set of colours that were easily and consistently
named by people without CVD (which we now had) but also exhib-
ited as much variation as possible to challenge people with CVD,
we further subsampled our colour set by selecting the 21 most visu-
ally distinct colours (defned using Euclidean distance in CIE LUV
space) for each colour term. Only eight RGB colours had ‘black’
>= 0.90 as their dominant colour term, so we included them all.
These colours are shown in Figure 4.
Finally, to generate pairs of colours that are named diferently
by people without CVD but likely confused by people with CVD,
we followed a process similar to that used in [17] by defning a
CIE LUV 3D line between each Figure 4 colour and the protan, and
deutan copunctal points (as defned in [47]). Any Figure 4 colours
with a diferent dominant colour term but within 5.0 CIE LUV units
of the 3D lines defned above were considered candidates for CVD
confusion for the type of CVD for that line’s copunctal point.
4.2.2 Selection Task. The Selection Task was designed to assess
a combination of both denotative and comparative colour judge-
ment [20]. Our implementation of the Selection Task consists of
a grid of 80 colour swatches. To choose colours for our task, we
sampled from our generated colours (described above). We gener-
ated two distinct colour sets, one each for Protans and Deutans.
Our Protan colour set consisted of Red & Brown, Orange & Green,
and Blue & Purple. Our Deutan colour set consisted of Green &
Brown, Pink & Grey, and Blue & Purple. Similar to the original
implementation [20], all colours (for either Protan or Deutan) were
presented in the grid even if they were not the target pair. Finally,
in all cases participants completed each colour pair with each of 0-3
target colours. This resulted in the Selection Task being completed
a total of 12 times.
4.2.3 Transition Task. We implemented the Transition Task as de-
scribed in [20], with a progression of ten colour swatches presented
horizontally. The extreme colours (far left and far right) were fxed
and used as anchors to order the eight randomly-placed intermedi-
ary colours between them. To complete the Transition task, only
comparative colour judgement is required. We generated two dis-
tinct sets of colours, one each for Protan and Deutan CVD. Our
Protan set consisted of ‘yellow’ to ‘green’, ‘red’ to ‘green’, and ‘blue’
to ‘pink’ transitions. Our Deutan set consisted of ‘yellow’ to ‘green’,
‘red’ to ‘brown’, and ‘grey’ to ‘pink’ transitions.
4.2.4 Sorting Task. The Sorting Task was designed to require only
denotative judgement of colour by showing a single swatch of
colour that is named by the participant, thereby removing any
comparative colour judgement. We implemented the Sorting Task
identically to the original implementation [20], with a single colour
swatch presented at the bottom of the screen that needed to be
sorted into one of ten colour names presented in a grid above the
colour swatch. However, in this implementation (unlike the original)
we removed ‘teal’ and added ‘black’. We did this as we had difculty
generating high confdence teal colours when we generated our
test colours, so we replaced it with ‘black’ to maintain a consistent
colour grid size (5x2). The colours we used for the Sorting Task
were generated from our set of colours. We randomly chose fve
each of ‘red’, ‘orange’, ‘yellow’, ‘green’, ‘blue’, ‘purple’, ‘pink’, ‘grey’,
and ‘brown’, resulting in 45 colours that needed to be sorted. To
Connor Geddes and David R. Flatla
make sure we had comparable results between the results of each
run of the Sorting Task, we used the identical set of colours for
each participant.
4.2.5 Distinction Task. To complete the Distinction Task, partic-
ipants need to use a combination of denotative and comparative
colour judgement, similar to the Selection Task, however, in a more
demanding task (more colours are presented). Also, similar to the
Selection Task, we implemented the Distinction task to have a grid
size of 80 colour swatches, but with a grid of 10 (5x2) checkboxes
with colour name labels below. We specifcally ensured that we
generated distinct colour sets of one to nine colours each once,
resulting in a total of nine repetitions of the Distinction Task (one
for each set of colours). We randomized the layout of the colours
for each participant and the order of the specifc distinct colour
sets. With this we got the following nine colour distinction sets:
(1) blue
(2) orange and yellow
(3) red, blue, and grey
(4) red, orange, yellow, and blue
(5) red, orange, green, pink and grey
(6) red, orange, green, blue, pink and brown
(7) orange, yellow, green, blue, pink, grey and brown
(8) orange, yellow, green, blue, purple, pink, grey, and brown
(9) red, orange, yellow, green, blue, purple, pink, grey, and brown
4.2.6 Category Task. The Category Task served as our connota-
tive colour task, requiring users to match a colour swatch to its
respective colour category (also represented as a diferent colour
swatch). In the Category Task, a target colour swatch was presented
in a prompt at the top of the screen, with a grid of 10 (5x2) colour
swatches presented below. Importantly for this task, there was
never an exact match for the target colour in the lower swatches.
To ensure that each colour name was encountered throughout the
study (similar to the Distinction Task) we ensured that every colour
name of ‘red’, ‘orange’, ‘yellow’, ‘green’, ‘blue’, ‘purple’, ‘pink’, and
‘brown’ each served once as the target category colour. This resulted
in a total of eight total trials of the Category Task.
4.2.7 Apparatus. To create our study, we used the cross-
platform SDK Flutter (version 2.2.0) and we used Firebase to
host our website online. The study is available here https://
colourmethodevaluation.web.app, and the code of the study can be
accessed here https://github.com/geddesc/ASSETS22-Challenging-
and-Improving-Evaluation-Methods.
4.3 Procedure & Design
At the start of the study participants completed a demographics
questionnaire. If the participants were not colour blind and were as-
signed one of the simulation conditions with information, next they
were then supplied with information telling them about the simula-
tion. Next participants completed the tasks in the following order:
1) Selection Task, 2) Transition Task, 3) Sorting Task, 4) Distinction
Task, and 5) Category Task. Between every task participants were
asked if they struggled on the task and if so to optionally provide
more information.
Challenging and Improving Current Evaluation Methods for Colour Identification Aids
4.4 Results
For all the tasks except Transition Task, we measured accuracy
(higher is better) and completion time (lower is better). For the
Transition Task, we also measured completion time but measured
error score (lower is better) instead of accuracy.
4.4.1 Data Analysis. To start, we checked our data for normality
using the Lilliefors test. We found that of 50 data distributions (5
conditions x 5 tasks x 2 dependent measures [accuracy/error score
+ completion time]), 22 rejected the null hypothesis that the data
was normally distributed. As such, we used the non-parametric
Kruskal-Wallis test for our analysis with Dunn’s Test (which uses
Bonferroni’s correction) for post-hoc pairwise comparisons.
Figure 5: Selection Task accuracy & completion time results.
4.4.2 Selection Task Results. Analyzing accuracy of the Selec-
tion Task with Kruskal-Wallis, indicated a signifcant main efect
(H ′ (4) = 32.614, p < .0001), with posthoc tests indicating signif-
icant pairwise diferences between the non-CVD condition and
all other conditions. Notably we did not observe a diference be-
tween the CVD condition and any of the simulation conditions.
We believe this is in part due to the difculty the task posed; the
variable number of target colours presented seems to have equally
challenged both CVD participants and participants under a CVD
simulation. However, we did not explore deeper to see if there were
any patterns in which colours were chosen (or not) between these
groups, and plan to do this in future work. Specifcally, we expect
there could be consistency in the colours that CVD participants
tend to confuse but we expect the simulation condition to be highly
variable, likely indicating random guessing with little justifcation
or thought behind it. Selection Task results are shown in Figure 5.
For completion time, analysis with Kruskal-Wallis indicated a sig-
nifcant main efect (H ′ (4) = 29.785, p < .0001), with posthoc tests
indicating signifcant diferences between the non-CVD condition
and the CVD condition, the non-CVD condition and the Simulation
condition with no information provided, the CVD condition and
all three simulation conditions, and the no information simulation
condition and the other two simulation conditions. We believe in
the case of the diferences observed between the CVD condition
and all simulation conditions, this is telling that CVD participants
clearly behave diferently in a colour task such as this, taking more
time to complete the task as a result of likely comparing a set of
colours they distinctly believe to be the target colour. Whereas non-
CVD participants under a simulation (regardless of the information
condition) were more likely to spend far less time as they do not
ASSETS ’22, October 23–26, 2022, Athens, Greece
have the lifelong knowledge of being colour blind that they can
apply to a task like this to assist them in knowing which shades of
the colours presented before them are the most likely candidates
to match the target colour name. This also provides more justifca-
tion for our suspicion that a future investigation into the specifc
colours chosen in a task such as this could reveal more evidence of
a diference in this specifc task.
Figure 6: Transition Task accuracy & completion time re-
sults.
4.4.3 Transition Task Results. Our analysis with Kruskal-Wallis for
error score indicates a signifcant main efect (H ′ (4) = 39.472, p <
.0001), with posthoc tests indicating pairwise diferences between
the non-CVD condition and all other conditions, and between the
CVD condition and all simulation conditions. The diferences be-
tween the CVD condition and all the simulations show how the
simulations struggled to emulate the challenge embodied in the
colour distinction task, which uses very fne colour diferences.
This result does not mean that simulations cannot provide some
useful insights, but they just cannot match the difculty of the task
as experienced by participants with CVD, which could also belie a
weakness in the simulation algorithm itself. Transition Task results
are shown in Figure 6.
For completion time, our analysis with Kruskal-Wallis did not
indicate a signifcant main efect (H ′ (4) = 4.706, p = .3188), per-
haps because all groups used a similar technique to complete the
Transition Task (comparing individual swatches to each other one
at a time).
Figure 7: Sorting Task accuracy & completion time results.
4.4.4 Sorting Task Results. Our analysis with Kruskal-Wallis for
accuracy indicates a signifcant main efect (H ′ (4) = 53.557, p <
.0001), with posthoc tests indicating pairwise diferences between
ASSETS ’22, October 23–26, 2022, Athens, Greece
the non-CVD condition and all other conditions, and between
the CVD condition and all simulation conditions. The diference
found with the non-CVD condition despite not specifcally choosing
colours along confusion lines for this task shows the efectiveness of
this task in being utilized to test potential colour identifcation aids’
ability in assisting individuals with CVD. The diferences between
the CVD condition and all simulation conditions in the Sorting Task
raises the idea that CVD participants have greater knowledge for
how colours in their specifc perception are named; they may not
always be correct, however, they are most certainly always more
correct than non-CVD participants under a simulation (regardless
of information provided). This raises the idea that CVD simulations
– regardless of information provided to participants on how CVD
people confuse colours – cannot hope to overcome the years of
(sometimes socially painful) experience that an individual with
CVD has in naming and distinguishing colours by name. Further,
we can also see by the lack of diference in any of the simulation
conditions as well as the relatively stable accuracy data as seen
in Figure 7 that the idea that the amount of information given to
non-CVD participants under a simulation does not matter much at
all as they do not have the life experience to utilize the knowledge
properly.
For completion time, our analysis with Kruskal-Wallis for com-
pletion time did not indicate a signifcant main efect (H ′ (4) =
4.449, p = .3487). This is interesting in that it probably refects
similarity in how each group completed the Sorting Task, which is
surprising given that only the non-CVD participants were expected
to not struggle with this task (and hence be able to complete it
quickly).
Figure 8: Distinction Task accuracy & completion time re-
sults.
4.4.5 Distinction Task Results. Our analysis with Kruskal-Wallis
for accuracy indicates a signifcant main efect (H ′ (4) = 36.351, p <
.0001), with posthoc tests indicating pairwise diferences between
the non-CVD condition and all but the CVD condition, and the CVD
condition and all Simulation conditions. First, notably CVD partici-
pant results did not difer too much from the non-CVD participants,
we believe this is due to the selection of colours for the task not
being chosen to specifcally fall along a confusion line. For future
studies implementing this task it would be benefcial to ensure
that colour name categories (e.g., red, blue) contain corresponding
pairs of colours that are confused. Finally, the diference between
the CVD condition and the simulation conditions is noteworthy
as this once again reinforces the idea that not only can non-CVD
Connor Geddes and David R. Flatla
participants under a simulation not name colours in the same way
as (or even make mistakes similar to) CVD participants, but also
that increasingly descriptive information regarding the experience
of people with CVD does not help at all in overcoming the gap.
For completion time, our analysis with Kruskal-Wallis indicates a
signifcant main efect (H ′ (4) = 9.755, p = .0448), with posthoc tests
indicating pairwise diferences between the non-CVD condition
and the simulation condition with minimum information, between
the CVD condition and all simulation conditions, the simulation
condition with no information and the simulation condition with
minimum information, and between the simulation condition with
minimum information and the simulation condition with maximum
information. These results are somewhat confusing, but might be
due to the very high variance in the ‘No Info’ and ‘Max Info’ condi-
tions. We are planning deeper analysis into these diferences in the
future.
Figure 9: Category Task accuracy & completion time results.
4.4.6 Category Task Results. Our analysis with Kruskal-Wallis for
accuracy indicates a signifcant main efect (H ′ (4) = 21.984, p <
.0002), with posthoc tests indicating pairwise diferences between
the non-CVD condition and all other groups. This result actually
provides some support for CVD simulations achieving similar ac-
curacy performance as participants with CVD.
However, our analysis with Kruskal-Wallis for completion time
indicates a signifcant main efect (H ′ (4) = 10.797, p = .0289), with
posthoc tests indicating pairwise diferences between the non-CVD
condition and the CVD condition, the non-CVD and the simulation
condition with minimum information, and between the CVD condi-
tion and all simulation conditions. By and large, these results indi-
cate that CVD participants behaved diferently than non-CVD and
CVD simulation groups. This not only underlines that simulations
do not allow non-CVD participants to act like CVD participants, but
it also underlies that the non-CVD participants act very similarly,
whether under simulation or not.
4.5 Diferences in How Colours were Named
To get a better understanding of the diferences that a lifetime of
living with CVD can provide in colour naming compared to that of
a non-CVD person using a CVD simulation, we looked at how these
two groups of participants difered in how they named colours. To
do this, we dug deeper into the individual-colour results for the
Sorting and Distinction Tasks as these tasks featured the widest
Challenging and Improving Current Evaluation Methods for Colour Identification Aids ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 10: Sorting Task naming for each colour term for each participant group, showing for each ‘correct’ colour name (x-axis)
how participants ended up naming it during the sorting task (stacked column). ‘red’, ‘orange’, ‘green’ and ‘purple’ illustrate
clear diferences between CVD and simulation participants.

Figure 11: Distinction Task colours participants selected for the one-distinct-colour condition (left) and the fve-distinct-colour
condition (right). The one-distinct-colour was ‘blue’. The fve-distinct-colours were ‘red’, ‘orange’, ‘green’, ‘pink’ and ‘grey’.

variation of colours, required participants to identify colours by
name, and used a consistent set of colours for every participant.
We found two major diferences in the data between CVD and
non-CVD participants under simulations: 1) CVD participants were
more consistent and correct when naming colours, and 2) the
colours that were incorrectly named difered wildly between CVD
participants and non-CVD participants under simulations (regard-
less of the amount of contextual information provided).
Figure 11 illustrates that in the ‘fve distinct colours’ condition of
the Distinction Task, CVD participants more frequently chose the
correct colours for ‘red’, ‘orange’, ‘green’, ‘pink’ and ‘grey’. These
fndings again reinforce that CVD participants have a lifetime of
experience seeing and naming colours in largely a non-CVD world.
While it might be attractive and easier to evaluate CVD Aids using
simulations, ‘indirect’ evaluations simply will never replicate the
knowledge rooted in the lived experience of people with CVD.

4.5.1 CVD Participants More Accurately Named Colours. In the

Sorting and Distinction Tasks, CVD participants in most cases were
more correct at naming colours than non-CVD participants under a
simulation. This makes perfect sense as CVD participants are most
familiar with their perception of colour and have been living with
and self-accommodating their CVD their entire lives. To directly
see how CVD participants outperformed non-CVD participants
under simulations, see Figure 10, specifcally the ‘red’, ‘orange’,
‘green’, and ‘purple’ data, which illustrates the diferences in CVD
participants naming the colours far more accurately than non-
CVD participants under simulations. In the case of how the ‘reds’
were sorted, we can see that they were sorted to mostly ‘red’ with
very little to ‘brown’ for the CVD participants. However, for the
participants under simulations, ‘reds’ were largely sorted to ‘brown’
with the remainder to a combination of ‘red’, ‘orange’, ‘yellow’ and
only in the case of the maximum information condition ‘green’.
4.5.2 Incorrectly Named Colours Difer Wildly Between CVD and
Simulations. When colours were incorrectly named in the Sorting
and Distinction Tasks, we again found that there were large difer-
ences between CVD and non-CVD participants under simulations.
To illustrate this, we will discuss the errors made in the Sorting Task
for ‘orange’ as can be seen in Figure 10. CVD participants mostly
correctly sorted all the ‘orange’ colours in the Sorting Task, however,
when they did incorrectly sort ‘oranges’, they named them mostly
‘red’ or ‘green’. Looking at the results for the non-CVD participants
under simulations, we can see that they did much worse overall and
that they confused colours diferently. In all simulation conditions
the major colours non-CVD participants incorrectly sorted ‘orange’
into were ‘yellow’ and ‘brown’. There was a slight improvement in
the maximum information condition, however ‘yellow’ and ‘brown’
still remained the most common confused colours.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Turning to the Distinction Task and the ‘one distinct colour’ data
in Figure 11, we can see that all participants across all conditions
chose ‘blue’ at least once – errors in this condition are represented
by choosing anything other than ‘blue’. CVD participants largely
did so with ‘purple’, however non-CVD simulation participants
rarely chose ‘purple’ but instead opted for ‘black’.
Both the Sorting Task and Distinction Task examples outlined
above show clear diferences in how when naming colours incor-
rectly CVD participants confused colours diferently than non-CVD
participants, once again afrming our conclusion that CVD simu-
lations cannot replace a lifetime of living with CVD. Furthermore,
we can also see that the provided CVD contextual information did
not help non-CVD participants name colours in a similar fashion
as CVD participants.
5 DISCUSSION
5.1 Summary & Explanation of Results
To summarize, our fndings indicate that the use of simulations
by individuals without CVD does not produce results equivalent
to those obtained from people with CVD. Further, even providing
more contextual information for the simulation does not make a
diference, as non-CVD people cannot simply adapt to a perception
of colour that is so foreign to them. What this means is that CVD
simulations and information appended to those simulations are not
equivalent to the life experiences that an individual with CVD has
in their colour perception.
The most notable illustration of these results are the tasks that
involved selecting distinct colours (Distinction Task) or sorting
colours (Sorting Task) by name. There was a clear diference in
these conditions that highlighted the diference between someone
with CVD who has been identifying colours by name or distin-
guishing separate-named colours their entire lives, and someone
who is experiencing a CVD simulation (with or without contextual
information) for the frst time.
Finally, our results, specifcally those from the Transition Task,
put into question the recommendations provided in a recent Nature
Communications paper [11], which justifes specifc continuous
colour maps as accessible for those with various forms of CVD
based on their appearance under CVD simulations. In our results,
we saw that when non-CVD participants used a CVD simulation
to complete the Transition Task, their results were always more
accurate than those who had CVD. This means that using CVD
simulations to evaluate if a given continuous colour map is suitable
for people with CVD does not actually assist in understanding
how people with CVD will interact with any visualization that
uses such a colour map. Unfortunately, using CVD simulations to
justify accessible colour maps has already been adopted by both
Matplotlib 3 and R 4 .
5.2 What this Means for CVD Simulation Use?
We do NOT recommend that CVD simulations be used to evaluate
the efectiveness of CVD aids or in any case to substitute for the
lived colour perception experiences of those with CVD. If carefully
employed, CVD simulations can provide some insights for non-CVD
3 https://matplotlib.org/3.5.2/tutorials/colors/colormaps.html
4 https://cran.r-project.org/web/packages/viridis/vignettes/intro-to-viridis.html
Connor Geddes and David R. Flatla
people regarding how the world might look for those with CVD, for
example to help build understanding between deeply-connected
people [18] (e.g., spouses, siblings) with diferent colour perception.
However, our results show that CVD simulations do not capture
the lived experience gained from actually having CVD, and that
no amount of contextual information provided can compensate
for those years of experience. Our fndings agree with the advice
from Tigwell [51] and Bennett & Rosner [4], where they argue that
disabled people should be incorporated into the design team and
that simulations simply cannot match the life experiences of those
with disabilities.
Even if CVD simulations can allow an individual without CVD
to obtain diagnostic results to ‘classify’ as an individual with CVD.
This does not allow for that individual to understand the deep
contextual cues that people with CVD have developed over their
entire lives to be able to better discriminate and interpret colours.
We assert that studies that look to assist people with CVD must
understand and assist them not the tools or simulations that look
to approximate them.
6 LIMITATIONS AND FUTURE WORK
The Sorting, Distinction and Category Tasks all featured colour sets
that were randomly generated without specifcally choosing pairs
along confusion lines. This had a side efect where it appeared as
though the colours were not always difcult for individuals with
CVD. Future implementations of these tasks should consider this
and perhaps take an approach similar to what we did for both the
Selection and Transition Tasks and ensure that there are pairs of
colours that lie along confusion lines.
Cole’s [9] four categories of colour judgement that individuals
with CVD tend to struggle with also includes aesthetic judgement
of colour. We have not yet implemented an aesthetic task as there
is very little understanding of what makes specifc colour com-
binations aesthetically pleasing to individuals with CVD. Future
work should look to understand what makes colours aesthetically
pleasing to people with colour vision defciency and should develop
a task that assesses potential aesthetic colour judgement of people
with CVD based on that understanding.
We conducted our study online, and thus had no contact with
our participants. The results we observed indicated diferences in
behaviour when completing our colour-related tasks between CVD
and non-CVD participants under simulations. However, future work
should look to ground and verify this in a controlled lab setting.
We had the tasks ordered specifcally so that it was always Selec-
tion, Transition, Sorting, Distinction and fnally Category task. This
may have had a learning efect on the overall results even though
our tasks feature diferent types of colour judgement. Future studies
should be cautious of the potential efect ordering could have on
the results of their study when utilizing these tasks.
Finally, there are also perhaps deeper diferences that may be
found on the basis of which specifc colours individuals with CVD
confused or struggled with for specifc tasks (e.g., Selection Task).
We next plan to examine our data more deeply to see if more fne-
grained patterns emerge.
Challenging and Improving Current Evaluation Methods for Colour Identification Aids
7 CONCLUSION
This study illustrated that there were signifcant diferences be-
tween the results of CVD participants and non-CVD participants
under a CVD simulation in tasks relating to colour identifcation,
comparison, and distinction. People with CVD have had years of
experience and understand how specifc colours can look to them,
they are not always correct, but they are certainly always more
correct than a non-CVD participant under a simulation. Further, the
amount of CVD-context-information provided with the simulation
seems to make no diference; we saw that no matter how much
information we provided – none, minimal or a maximum amount –
there was no clear diference between the results for those three
conditions. Even assuming that non-CVD participants using CVD
simulations ‘see’ what participants with CVD see, they cannot du-
plicate their behaviour and cannot fake the deep knowledge that
individual with CVD have developed over their lives.
ACKNOWLEDGMENTS
We thank our participants for taking part in our research. We would
also like to thank our anonymous reviewers for their time and efort
in the peer review of this paper.
REFERENCES
[1] Katrin Angerbauer, Nils Rodrigues, Rene Cutura, Seyda Öney, Nelusa Path-
manathan, Cristina Morariu, Daniel Weiskopf, and Michael Sedlmair. 2022. Ac-
cessibility for Color Vision Defciencies: Challenges and Findings of a Large
Scale Study on Paper Figures. In Proceedings of the 2022 CHI Conference on
Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). As-
sociation for Computing Machinery, New York, NY, USA, Article 134, 23 pages.
https://doi.org/10.1145/3491102.3502133
[2] Anonymous. 2008. Inkscape - Color Blindness Simulation. https://web.
archive.org/web/20111123145815/http://kaioa.com/node/75#comment-247. Ac-
cessed: 2022-4-14.
[3] Jibin Bao, Yuanyuan Wang, Yu Ma, and Xiaodong Gu. 2008. Re-coloring im-
ages for dichromats based on an improved adaptive mapping algorithm. In 2008
International Conference on Audio, Language and Image Processing. IEEE, 152–156.
[4] Cynthia L. Bennett and Daniela K. Rosner. 2019. The Promise of Empathy: Design,
Disability, and Knowing the "Other". In Proceedings of the 2019 CHI Conference
on Human Factors in Computing Systems. Association for Computing Machinery,
New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300528
[5] Jennifer Birch and Jennifer Birch. 2001. Diagnosis of defective colour vision.
(2001).
[6] Hans Brettel, Françoise Viénot, and John D Mollon. 1997. Computerized simula-
tion of color appearance for dichromats. Josa a 14, 10 (1997), 2647–2655.
[7] Olivier Burggraaf, Sanjana Panchgnula, and Frans Snik. 2021. Citizen science
with colour blindness: A case study on the Forel-Ule scale. PLoS ONE 16(4):
e0249755. (2021). https://doi.org/10.1371/journal.pone.0249755
[8] Yu-Chieh Chen and Tai-Shan Liao. 2011. Hardware digital color enhancement
for color vision defciencies. ETRI Journal 33, 1 (2011), 71–77.
[9] Barry L Cole. 2004. The handicap of abnormal colour vision. Clinical and
Experimental Optometry 87, 4-5 (2004), 258–275.
[10] Bevil R Conway. 2009. Color vision, cones, and color-coding in the cortex. The
neuroscientist 15, 3 (2009), 274–290. https://doi.org/10.1177/1073858408331369
[11] Fabio Crameri, Grace E. Shephard, and Philip J. Heron. 2020. The misuse of
colour in science communication. Nature Communications 11, 5444 (2020). https:
//doi.org/10.1038/s41467-020-19160-7
[12] Peter B Delahunt, Michael A Webster, Lei Ma, and John S Werner. 2004. Long-term
renormalization of chromatic mechanisms following cataract surgery. Visual
neuroscience 21, 03 (2004), 301–307.
[13] Yinhui Deng, Yuanyuan Wang, Yu Ma, Jibin Bao, and Xiaodong Gu. 2007. A fxed
transformation of color images for dichromats based on similarity matrices. In
International Conference on Intelligent Computing. Springer, 1018–1028.
[14] F Dick, S Semple, R Chen, and A Seaton. 2000. Neurological defcits in solvent-
exposed painters: a syndrome including impaired colour vision, cognitive defects,
tremor and loss of vibration sensation. QJM 93, 10 (2000), 655–661.
[15] Dean Farnsworth. 1943. The Farnsworth-Munsell 100-Hue and Dichotomous
Tests for Color Vision∗. J. Opt. Soc. Am. 33, 10 (Oct 1943), 568–578. https:
//doi.org/10.1364/JOSA.33.000568
ASSETS ’22, October 23–26, 2022, Athens, Greece
[16] David Flatla and Carl Gutwin. 2012. SSMRecolor: Improving Recoloring Tools
with Situation-Specifc Models of Color Diferentiation. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA)
(CHI ’12). Association for Computing Machinery, New York, NY, USA, 2297–2306.
https://doi.org/10.1145/2207676.2208388
[17] David R. Flatla, Alan R. Andrade, Ross D. Teviotdale, Dylan L. Knowles, and
Craig Stewart. 2015. ColourID: Improving Colour Identifcation for People with
Impaired Colour Vision. In Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems. Association for Computing Machinery,
New York, NY, USA, 3543–3552. https://doi.org/10.1145/2702123.2702578
[18] David R. Flatla and Carl Gutwin. 2012. "So That’s What You See": Building Under-
standing with Personalized Simulations of Colour Vision Defciency. In Proceed-
ings of the 14th International ACM SIGACCESS Conference on Computers and Ac-
cessibility (Boulder, Colorado, USA) (ASSETS ’12). Association for Computing Ma-
chinery, New York, NY, USA, 167–174. https://doi.org/10.1145/2384916.2384946
[19] Donald S Fong, Franca B Barton, George H Bresnick, and Early Treatment Diabetic
Retinopathy Study Research Group. 1999. Impaired color vision associated with
diabetic retinopathy: early treatment diabetic retinopathy study report no. 15.
American journal of ophthalmology 128, 5 (1999), 612–617.
[20] Connor Geddes, David R. Flatla, Garreth W. Tigwell, and Roshan L Peiris. 2022.
Improving Colour Patterns to Assist People with Colour Vision Defciency. In
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
(New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New
York, NY, USA, Article 479, 17 pages. https://doi.org/10.1145/3491102.3502024
[21] Google. 2020. Improve vision defciency color matrices per research paper. https://
chromium-review.googlesource.com/c/chromium/src/+/2124500 Accessed: 2022-
4-14.
[22] Google. 2020. What’s New In DevTools (Chrome 83). https://developer.chrome.
com/blog/new-in-devtools-83/#vision-defciencies. Accessed: 2022-4-14.
[23] C. H. Graham and Yun Hsia. 1959. Studies of Color Blindness: A
Unilaterally Dichromatic Subject. Proceedings of the National Acad-
emy of Sciences 45, 1 (1959), 96–99. https://doi.org/10.1073/pnas.45.1.96
arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.45.1.96
[24] Soon Hau Chua, Haimo Zhang, Muhammad Hammad, Shengdong Zhao, Sahil
Goyal, and Karan Singh. 2015. ColorBless: Augmenting Visual Information
for Colorblind People with Binocular Luster Efect. ACM Trans. Comput.-Hum.
Interact. 21, 6, Article 32 (Jan. 2015), 20 pages. https://doi.org/10.1145/2687923
[25] Chun-Rong Huang, Kuo-Chuan Chiu, and Chu-Song Chen. 2010. Key color
priority based image recoloring for dichromats. In Pacifc-Rim Conference on
Multimedia. Springer, 637–647.
[26] Jia-Bin Huang, Yu-Cheng Tseng, Se-In Wu, and Sheng-Jyh Wang. 2007. Infor-
mation preserving color transformation for protanopia and deuteranopia. IEEE
Signal Processing Letters 14, 10 (2007), 711–714.
[27] Jia-Bin Huang, Sih-Ying Wu, and Chu-Song Chen. 2008. Enhancing color represen-
tation for the color vision impaired. In Workshop on Computer Vision Applications
for the Visually Impaired.
[28] K. Ichikawa, S. Yokoyama, Yoshiki Tanaka, Hideki Nakamura, R. T. Smith, and
S. Tanabe. 2020. The Change in Color Vision with Normal Aging Evaluated on
Standard Pseudoisochromatic Plates Part-3. Current Eye Research 46 (2020), 1038
– 1046.
[29] Shinobu Ishihara. 1987. Test for colour-blindness. Kanehara Tokyo, Japan.
[30] Luke Jeferson and Richard Harvey. 2007. An Interface to Support Color Blind
Computer Users. In Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems (San Jose, California, USA) (CHI ’07). Association for
Computing Machinery, New York, NY, USA, 1535–1538. https://doi.org/10.1145/
1240624.1240855
[31] Kenneth Knoblauch, François Vital-Durand, and John L Barbur. 2001. Variation
of chromatic sensitivity across the life span. Vision research 41, 1 (2001), 23–36.
[32] Shoji Kondo. 1990. A computer simulation of anomalous color vision. Color
Vision Defciencies (1990), 145–159.
[33] Tobias Langlotz, Jonathan Sutton, Stefanie Zollmann, Yuta Itoh, and Holger
Regenbrecht. 2018. ChromaGlasses: Computational Glasses for Compensating
Colour Blindness. Association for Computing Machinery, New York, NY, USA,
1–12. https://doi.org/10.1145/3173574.3173964
[34] Jinmi Lee and Wellington Pinheiro dos Santos. 2011. An adaptive fuzzy-based sys-
tem to simulate, quantify and compensate color blindness. Integrated Computer-
Aided Engineering 18, 1 (2011), 29–40.
[35] Huei-Yung Lin, Li-Qi Chen, and Min-Liang Wang. 2019. Improving Discrimi-
nation in Color Vision Defciency by Image Re-Coloring. Sensors 19, 10 (2019).
https://doi.org/10.3390/s19102250
[36] Richard B Lomax, Peter Ridgway, and Maureen Meldrum. 2004. Does occupational
exposure to organic solvents afect colour discrimination? Toxicological reviews
23, 2 (2004), 91–121.
[37] Rhouri MacAlpine and David R. Flatla. 2016. Real-Time Mobile Personalized
Simulations of Impaired Colour Vision. In Proceedings of the 18th International
ACM SIGACCESS Conference on Computers and Accessibility (Reno, Nevada, USA)
(ASSETS ’16). Association for Computing Machinery, New York, NY, USA, 181–189.
https://doi.org/10.1145/2982142.2982170
ASSETS ’22, October 23–26, 2022, Athens, Greece
[38] Gustavo M. Machado, Manuel M. Oliveira, and Leandro A. F. Fernandes. 2009.
A Physiologically-Based Model for Simulation of Color Vision Defciency. IEEE
Transactions on Visualization and Computer Graphics 15, 6 (nov 2009), 1291–1298.
https://doi.org/10.1109/TVCG.2009.113
[39] MaPePeR. 2015. jsColorblindSimulator. https://github.com/MaPePeR/
jsColorblindSimulator. Accessed: 2022-4-14.
[40] Gary W Meyer and Donald P Greenberg. 1988. Color-defective vision and
computer graphics displays. IEEE Computer Graphics and Applications 8, 5 (1988),
28–40.
[41] Neda Milić, Miklós Hofmann, Tibor Tómács, Dragoljub Novaković, and Branko
Milosavljević. 2015. A content-dependent naturalness-preserving daltonization
method for dichromatic and anomalous trichromatic color vision defciencies.
Journal of Imaging Science and Technology 59, 1 (2015), 10504–1.
[42] Mozilla. 2020. Color vision simulation. https://frefox-source-docs.mozilla.org/
devtools-user/accessibility_inspector/simulation/index.html. Accessed: 2022-4-
14.
[43] Shigeki Nakauchi and Tatsuya Onouchi. 2008. Detection and modifcation of
confusing color combinations for red-green dichromats to achieve a color univer-
sal design. Color Research & Application: Endorsed by Inter-Society Color Council,
The Colour Group (Great Britain), Canadian Society for Color, Color Science Associ-
ation of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre
Foundation, Colour Society of Australia, Centre Français de la Couleur 33, 3 (2008),
203–211.
[44] nburrus. 2021. Let’s collectively determine the best color blindness simula-
tion method! https://www.reddit.com/r/ColorBlind/comments/qzkl7h/lets_
collectively_determine_the_best_color/. Accessed: 2022-4-14.
[45] M Pacheco-Cutillas, DF Edgar, and A Sahraie. 1999. Acquired colour vision
defects in glaucoma – their detection and clinical signifcance. British Journal of
Ophthalmology 83, 12 (1999), 1396–1402. https://doi.org/10.1136/bjo.83.12.1396
[46] S Poret, RD Dony, and S Gregori. 2009. Image processing for colour blindness
correction. In 2009 IEEE Toronto International Conference Science and Technology
for Humanity (TIC-STH). IEEE, 539–544.
[47] B.C. Regan, J.P. Refn, and J.D. Mollon. 1994. Luminance noise and the rapid
determination of discrimination ellipses in colour defciency. Vision Research 34,
10 (1994), 1279–1299. https://doi.org/10.1016/0042-6989(94)90203-8
[48] Behzad Sajadi, Aditi Majumder, Manuel M. Oliveira, Rosalia G. Schneider, and
Ramesh Raskar. 2013. Using Patterns to Encode Color Information for Dichro-
mats. IEEE Transactions on Visualization and Computer Graphics 19, 1 (Jan. 2013),
Connor Geddes and David R. Flatla
118–129. https://doi.org/10.1109/TVCG.2012.93
[49] Andrew Sears and Vicki Hanson. 2011. Representing users in accessibility re-
search. In Proceedings of the SIGCHI conference on Human factors in computing
systems. 2235–2238.
[50] Jonathan Sutton, Tobias Langlotz, and Alexander Plopski. 2022. Seeing Colours:
Addressing Colour Vision Defciency with Vision Augmentations Using Compu-
tational Glasses. ACM Trans. Comput.-Hum. Interact. 29, 3, Article 26 (jan 2022),
53 pages. https://doi.org/10.1145/3486899
[51] Garreth W. Tigwell. 2021. Nuanced Perspectives Toward Disability Simulations
from Digital Designers, Blind, Low Vision, and Color Blind People. In Proceedings
of the 2021 CHI Conference on Human Factors in Computing Systems. Association
for Computing Machinery, New York, NY, USA, Article 378, 15 pages. https:
//doi.org/10.1145/3411764.3445620
[52] Françoise Viénot, Hans Brettel, and John D Mollon. 1999. Digital video
colourmaps for checking the legibility of displays by dichromats. Color Research
& Application 24, 4 (1999), 243–252.
[53] Ken Wakita and Kenta Shimamura. 2005. SmartColor: Disambiguation Frame-
work for the Colorblind. In Proceedings of the 7th International ACM SIGAC-
CESS Conference on Computers and Accessibility (Baltimore, MD, USA) (As-
sets ’05). Association for Computing Machinery, New York, NY, USA, 158–165.
https://doi.org/10.1145/1090785.1090815
[54] Alexander Wong and William Bishop. 2008. Perceptually-adaptive color enhance-
ment of still images for individuals with dichromacy. In 2008 Canadian Conference
on Electrical and Computer Engineering. IEEE, 002027–002032.
[55] Seungji Yang, YongMan Ro, EdwardK Wong, and Jin-Hak Lee. 2008. Quantifca-
tion and standardized description of color vision defciency caused by anomalous
trichromats—part I: simulation and measurement. EURASIP Journal on Image
and Video Processing 2008 (2008), 1–9.
[56] Semir Zeki. 1990. A century of cerebral achromatopsia. Brain 113, 6 (1990),
1721–1777.
[57] Zhenyang Zhu and Xiaoyang Mao. 2021. Image recoloring for color vision
defciency compensation: a survey. The Visual Computer 37 (12 2021), 1–20.
https://doi.org/10.1007/s00371-021-02240-0
[58] Zhenyang Zhu, Masahiro Toyoura, Kentaro Go, Kenji Kashiwagi, Issei Fujishiro,
Tien-Tsin Wong, and Xiaoyang Mao. 2021. Personalized Image Recoloring for
Color Vision Defciency Compensation. IEEE Transactions on Multimedia PP (03
2021), 1–1. https://doi.org/10.1109/TMM.2021.3070108
 ASL Wiki: An Exploratory Interface for Crowdsourcing ASL
Translations
Abraham Glasser Fyodor Minakov
atg2036@rit.edu fyodorominakov@gmail.com
Computing and Information Sciences Microsoft Research
Rochester Institute of Technology Cambridge, MA, USA
Rochester, New York, USA
ABSTRACT
The Deaf and Hard-of-hearing (DHH) community faces a lack of
information in American Sign Language (ASL) and other signed
languages. Most informational resources are text-based (e.g. books,
encyclopedias, newspapers, magazines, etc.). Because DHH signers
typically prefer ASL and are often less fuent in written English,
text is often insufcient. At the same time, there is also a lack of
large continuous sign language datasets from representative sign-
ers, which are essential to advancing sign langauge research and
technology. In this work, we explore the possibility of crowdsourc-
ing English-to-ASL translations to help address these barriers. To
do this, we present a novel bilingual interface that enables the com-
munity to both contribute and consume translations. To shed light
on the user experience with such an interface, we present a user
study with 19 participants using the interface to both generate and
consume content. To better understand the potential impact of the
interface on translation quality, we also present a preliminary trans-
lation quality analysis. Our results suggest that DHH community
members fnd real-world value in the interface, that the quality of
translations is comparable to those created with state-of-the-art
setups, and shed light on future research avenues.
CCS CONCEPTS
• Human-centered computing → Accessibility systems and
tools; Accessibility technologies; Empirical studies in col-
laborative and social computing; • Information systems →
Collaborative and social computing systems and tools; Web
interfaces; Web searching and information discovery; • Applied
computing → Digital libraries and archives; E-learning.
KEYWORDS
Deaf and Hard-of-Hearing, Sign Language, Bilingual, Interface,
Education, Corpus, Crowdsourcing, ASL data collection
ACM Reference Format:
Abraham Glasser, Fyodor Minakov, and Danielle Bragg. 2022. ASL Wiki:
An Exploratory Interface for Crowdsourcing ASL Translations. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544827
Danielle Bragg
Danielle.Bragg@microsoft.com
Microsoft Research
Cambridge, MA, USA
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
13 pages. https://doi.org/10.1145/3517428.3544827
1 INTRODUCTION
Approximately 1 in 6 adults in the U.S. is Deaf or Hard-of-Hearing
(DHH), and prior literacy research shows that over 17% of deaf
adults have "low literacy" [1]. Signed languages are the primary
languages of Deaf communities worldwide, and they are completely
distinct from local spoken/written languages. For example, Ameri-
can Sign Language (ASL) is the primary signed language used in
the U.S., but it is a completely diferent language from English – not
a one-to-one mapping. As a result, if a person is fuent in ASL, they
are not necessarily fuent in English reading and writing. ASL is of-
ten DHH signers’ primary language, and they typically prefer ASL
over English, are more comfortable with, and understand content
better in ASL [21]. Among this bilingual community, there is a wide
range of literacy levels (e.g. studies have found fourth-grade reading
levels among DHH high-school graduates [39]). Research has found
that as a result there are lower educational outcomes among DHH
individuals and lower rates of employment (and salaries) among
DHH adults [2].
A major obstacle facing DHH signers is a lack of educational
resources in sign language. Many educational resources are avail-
able in text (e.g. textbooks, literature books, online encyclopedias,
etc.), but not in a signed language. As there is no standardized
written form of ASL and sign language is typically in video form,
these text-based interfaces do not adequately support users who
prefer a signed language. Because of this lack of ASL content, DHH
users often have to look up individual English words on a sepa-
rate website or interface (e.g. English-to-ASL dictionaries) [10], and
re-read the English content they are trying to consume [3]. Even
though individual words can be looked up when necessary, this is
not efcient, does not help to understand English grammar, and
may be insufcient for DHH signers trying to understand English
text. It would be helpful if an ASL version of the target content
was available – having the entire sentence/article signed might be
preferred by a DHH ASL signer rather than looking up individual
words and/or re-reading multiple times.
At the same time, advancing sign language research and technol-
ogy is currently impeded by lack of sign language data [8]. Existing
ASL datasets typically ofer a set of individual ASL signs, with their
respective English meanings, and/or ASL glosses. They do not have
sufciently representative and diverse signers – they often consist
of homogeneous sign language interpreters, small sets of signers,
and poorly labeled videos of unverifed quality (listed in [6]). In
order to more fully understand and model the language, labelled
continuous signing (i.e. complete sentences with annotations) from
ASSETS ’22, October 23–26, 2022, Athens, Greece
diverse signers is needed. However, creating such a dataset is ex-
tremely difcult. It is not only expensive to produce, but it also
requires a massive amount of human labeling and annotation, since
there is no automated system to do so. It is also hard to enable a
large pool of contributors, since most in-person data collection ef-
forts are limited to individuals who live close-by within commuting
distance, and have time in their schedules to contribute. How to en-
able everyday signers to efciently contribute labelled continuous
content, and how DHH users might respond to crowd-generated
content remain open questions.
In this work, we present a novel interface that addresses two
needs at once: 1) it provides a bilingual information resource and 2)
it simultaneously generates a continuous labelled signing dataset
that could be used by artifcial intelligence researchers, ASL lin-
guists, ASL learners, DHH ASL signers, and others. Our interface
provides a side-by-side ASL (video) and English (text) synchronized
interface, where users are able to read/view articles simultaneously
in both languages. Users can also use this platform to contribute
ASL translations of existing English texts in the communal database.
For this exploratory work, we seed the interface with popular Eng-
lish Wikipedia articles, which are translated into ASL, and refer to
this prototype system as "ASL Wiki". However, the same interface
could be seeded with any long text, and could be used with any
pair of written and signed languages. In terms of dataset creation,
by enabling contributors to record segments of English text with
known contents, the interface eliminates the need for humans to
later segment and align the text and video. Such intensive labelling
work is commonly done in creating parallel corpora containing
signed language and spoken/written text (e.g. [17]).
To help understand the efectiveness of such an interface, we
ran two exploratory studies. First, to better understand the user
experience with the interface, we conducted an exploratory user
study where 19 participants used the interface to consume and
generate content, and shared feedback. Our results suggest that
DHH individuals fnd real-world value in our interface, thought
it was easy and intuitive to use, and were excited to see further
development and identifed several target audiences they would
recommend the site to. Second, we also conducted an exploration
into the quality of translations that can be generated through our
interface. Results suggest that the translation quality is comparable
to the quality of translations created through state-of-the-art setups
for sign language translation. We conclude by discussing future
work that this initial exploration introduces.
2 RELATED WORK
In this section, we focus on work relevant to our two motivations:
supporting bilingual content, and supporting sign language data
collection eforts.
2.1 Sign Language Educational Resources
Existing resources that make information available in a signed
language compromise a small number of dictionaries ([11], [18]),
educational materials ([14], [12], [20], [42]), lexical databases ([37],
[19]), and mobile vocabulary apps ([27], [28]). Several examples
of these are listed in [6]. The landscape of existing sign language
resources is very small compared to the resources available for
Abraham Glasser, Fyodor Minakov, and Danielle Bragg
spoken and written language users, who are typically considered
by default. There have been limited attempts to create browser
tools that provide signed translations of written content, to create
signing avatars, and to more generally create recognition and trans-
lation systems [6]. These tools and resources are not viable due
to the very limited amount of labeled signing videos with diverse,
well-representative signers. There are some DHH content creators
that strive to support accessibility of information and support the
DHH community, such as the Daily Moth – a group who “deliver
news in video using American Sign Language” [29]. However, as
these eforts are sponsored and often composed of a small group of
people, they are limited in the amount of content they can create,
and often have to selectively ofer a handful of content options –
for instance, the Daily Moth says “the deaf host, Alex Abenchuchan,
covers trending news stories and deaf topics on new shows Monday-
Fridays”. Many people in the DHH community praise the Daily
Moth due to the level of access it provides, being a bilingual in-
formation resource for selected news happening around the world
[33, 41].
Our interface would enable crowdsourcing to address the prob-
lems of large-scale sign language data collection and diversity, nat-
uralness, all while serving as a bilingual educational resource.
2.2 ASL and English Bilingualism
Prior work suggests that bilingual resources are useful for DHH fu-
ent signers, rather than having any negative information-overload
efects. Psychology researchers have established that it is not costly
to switch from single to dual lexical retrieval (using two languages
at once), and revealed a signifcant cost to turning of a language,
which bilingual DHH users might do while trying to understand
English text alone [16]. This suggests that an ASL and English bilin-
gual interface, such as the one we have developed in this work,
could be benefcial to DHH fuent signers by providing greater
accessibility than English text alone.
The value of bilingualism in ASL and English has been further
substantiated by Deaf-led organizations. The National Association
of the Deaf (NAD), a nonproft organization whose mission “is to
preserve, protect and promote the civil, human and linguistic rights
of deaf and hard of hearing people in the United States of America.”
Internationally, NAD represents the U.S. to the World Federation of
the Deaf (WFD), an international human rights organization. NAD
supports bilingualism, using ASL and English, in the home and
educational environment for DHH individuals. They advocate that
bilingualism is important and efective because it fosters “positive
self-esteem, confdence, resilience, and identity, factors necessary
for lifelong learning and success” [30].
Despite the value of bilingual ASL/English resources, few exist.
The Deaf Studies Digital Journal (DSDJ) “is the world’s frst peer-
reviewed journal dedicated to advancing the cultural, creative, and
critical output of published work in and about sign languages and
Deaf culture” [22]. It is a bilingual and bimodal publication primarily
presented in both ASL and English. It features academic work in
other sign languages, and ofers scholarly articles, commentary,
literature, visual arts, flm/video, interviews, reviews, and archival
history footage and commentary. To date, there have been 5 issues
(spanning 2009-2020) with about 150 articles total. In the most
ASL Wiki
recent issue, each article has a split side-by-side view showing ASL
(or other sign language) on the left, and English text on the right.
The content is synchronized so that the English sentence being
signed is highlighted. The video has controls so that the user can
control playback of the signed video. Our interface builds on this,
similarly providing side-by-side views in both languages.
To the best of our knowledge, there has been only one past
attempt to systematically provide sign language translations of
existing text. Signly1 is a recent commercial efort to add “synchro-
nous, in-vision, sign language translations on any webpage for any
deaf sign language user”. They enable website visitors to select
English text they would like translated into British Sign Language
(BSL), which is sent to a professional interpreter for translation.
Once the translation video is created, website visitors can click
on the English text to trigger a pop-up translation video at the
bottom-right corner. While this company helps make English texts
online accessible, users have to request translations, and website
creators have to contact and pay Signly to incorporate and maintain
their services. Scale is also limited, as the translations are done by
the Signly team. Our interface is similarly motivated to provide
access to English text online. However, we enable crowdsourcing
translations to streamline and scale data collection, and to enable
a more diverse and representative group to contribute. We also
display the text and video side-by-side in a more bilingual manner.
2.3 Sign Language Technologies Need Data
A recent paper [8] summarized the state of sign language processing.
They hosted an interdisciplinary workshop with 39 domain experts
with diverse backgrounds, where they reviewed the state-of-the-art,
and listed calls to action for the research community. These calls
included but were not limited to focusing on real-world applications
and creating larger, more representative, public video datasets. They
emphasized the current lack of data, cited as the biggest obstacle
in sign language technology research. Data collection is difcult
and costly, yet “without sufcient data, system performance will
be limited and unlikely to meet the Deaf community’s standards”.
Despite these challenges, groups have worked on sign language
data creation and curation. Datasets exist for many signed lan-
guages, including but not limited to German2 , American3 , Argen-
tinean4 , Turkish5 (more listed in [8, 13, 31, 32], etc.). The main pa-
rameters of sign language datasets include the number of subjects,
samples, language level, type, and annotations/labels. As explained
in [8], existing sign language datasets greatly limit the robustness
of systems trained on them. Current datasets are not sufciently
large – typically containing fewer than 100,000 articulated signs.
Also, many existing datasets contain individual signs, which
may not be as useful for real-world use cases of sign language
processing. For real-world applications, there needs to be natural
conversation with complete sentences, i.e. "continuous" sign lan-
guage. Continuous sign language recognition and translation are
challenging due to epenthesis efects (insertion of extra features
into signs) and co-articulation (ending of a sign afecting the start
1 https://signly.co/
2 https://www.phonetik.uni-muenchen.de/forschung/Bas/SIGNUM/
3 https://github.com/YAYAYru/sign-lanuage-datasets
4 http://sedici.unlp.edu.ar/handle/10915/56764
5 https://www.cmpe.boun.edu.tr/pilab/BosphorusSign/home_en.html
ASSETS ’22, October 23–26, 2022, Athens, Greece
of the next), among other difculties. Solving these challenges re-
quires large amounts of continuous sign language data to learn
from. Continuous signed sentences would also be useful for DHH
individuals trying to understand content, especially new concepts,
as it is natural and comfortable for them. There are some continuous
signing datasets, such as [15], which help fll this void. However,
these datasets are typically small and recorded in a laboratory or
studio environment, rather than a natural setting, which makes
generalizing to diverse users and real-world environments difcult.
Currently, the process of producing large ASL datasets is pro-
hibitively expensive, due to the equipment needed and the time it
takes to collect and label/annotate ASL data. Crowdsourcing is a
more afordable alternative to traditional in-lab collection, and has
been successfully used in other accessibility domains. For example,
research has explored collecting images/videos and questions from
blind and low-vision users, with answers provided by the crowd
[4, 5]. Other accessibility crowdsourcing research has explored pro-
viding image alt-text [35, 36], transportation information [34, 38],
and live captions [23, 24]. Our current work adds to this body of
work, by providing an initial exploration into crowdsourcing con-
tinuous sign language data.
Along with the lack of bilingual information resources, lack of
data motivates our “ASL Wiki” interface.
3 ASL WIKI PROTOTYPE
We have created “ASL Wiki” – a prototype site where people can
crowdsource ASL translations of English articles, providing a com-
munity resource that supports accessibility as a bilingual infor-
mation resource, while also tackling the lack of continuous ASL
datasets with English labels. In this section, we describe our proto-
type and design process.
3.1 Design Process and Criteria
We engaged in an iterative design process to arrive at our “ASL
Wiki” website design. We frst identifed design criteria the plat-
form needed to meet (e.g. that the text used is available for use
on the platform and in a dataset, that participants can contribute
remotely without specialized hardware, and that translations are
segmented and labelled). With these identifed, we started with
drawn designs which were iteratively refned and implemented.
Throughout the process, we continued to meet with stakeholders
consisting of a group of interdisciplinary Deaf and hearing individ-
uals who have deep ties with the DHH community and incorporate
their input. These stakeholders tried out the evolving prototype,
and also discussed the project and provided guidance.
Through our meetings, we chose to explore creating and reading
bilingual versions of Wikipedia articles, rather than play scripts,
books, or other resources. We decided on Wikipedia articles be-
cause they are generally neutral, publicly available, and popular
informational resources. There also exist other parallel corpora of
Wikipedia content which have been useful for natural language
processing and artifcial intelligence/machine learning.
Our iterative design process uncovered specifc user require-
ments of our interface. We found that the interface needed to show
ASL and English at the same time, so that users could see both
and easily look at one or the other as they wished. Our interface
ASSETS ’22, October 23–26, 2022, Athens, Greece
also needed to show which English portion is being signed in the
current ASL video, so that users can keep track of their position
in both the video timeline and the English article. Users who are
recording their videos should have an efcient, streamlined way to
record sentences, meaning that the interface should not pose unnec-
essary overhead. It should be allowed for multiple people to submit
recordings for the same English sentence, as diferent people might
sign diferently (e.g. regional accents or varied interpretations), or
have preferred signs for specifc English words.
3.2 "ASL Wiki" Design
3.2.1 Homepage. We took inspiration from the idea that Wikipedia
is a "free content, multilingual online encyclopedia written and
maintained by a community of volunteers" [40]. On the homepage
of the ASL Wiki site, on the left hand side, is a checkbox list of
featured categories. Users can use these checkboxes to bring up
relevant articles, which appear in the middle with fractions indi-
cating how many sentences there are in the article, and how many
of these sentences have been recorded by at least one user. Being
clickable, the rows of article titles also display a "Record" button
that takes you to the reading/recording interface (discussed further
in subsequent section).
On the top of the homepage is an introductory title and para-
graph, along with an ASL video of someone signing this text. Once
you are logged in, on the top right of the page is a button that allows
you to view and edit your profle, or sign out. Next to this button is
a gamifying trophy icon displaying the number of sentences the
logged-in user has recorded. This was added as it is a common
element of social media sites that display the number of "posts" an
user has submitted. It potentially incentivizes the user by showing
them how much sentences they have recorded.
In the middle of the page, between the top banner and article
table of contents, is a numbered instruction summary to remind
users how to navigate and use the interface. Especially as users are
able to leave the site and come back later, and since they navigate
into and out of specifc articles, they may need a persistent reminder
of how to use the site, which is why we added this. A screenshot
of the homepage is shown in Figure 1. Once the user selects an
article, they are taken to the reading/recording view. This view has
a toggle on the top to switch between the recording view and the
reading view.
3.2.2 Recording and Reading View. In both recording view and
reading view, the main layout is the same: it is a split, side-by-side
bilingual interface. On the left is a placeholder for an ASL video.
On the right is the article in English.
If in record view, once the user selects a sentence on the article,
the ASL video placeholder becomes a self-view of the user’s web-
cam, so that they can see themselves. Their self-view is overlaid
with a head and body guide to encourage users to center themselves
in the recording. A 3-second countdown commences, and then the
user would sign the English sentence in ASL. While they are record-
ing, the according English sentence is highlighted, to mark and keep
track of their place in the article. When they are fnished, clicking
on a stop button underneath their self-view stops the recording,
and displays their recorded video for playback. If the user approves,
clicking "Keep" will submit the video, and auto-progress to the next
Abraham Glasser, Fyodor Minakov, and Danielle Bragg
sentence in the article, or clicking "Redo" will prompt them to redo
the recording. The English sentences that have been recorded will
show a video camera icon. There is also a guide on the top, above
the English article, to remind users how to use the interface. Also,
underneath the ASL video placeholder is a picture demonstrating
a good recording setup and a bad example, to remind users that
they should be sure to position their webcams so it captures their
upper body and that their arms/hands do not go out of frame while
signing. There is also an upvote/downvote button where users can
give feedback on the ASL video.
In reading view, the site enables users to access parallel con-
tent in ASL and English. After toggling to reading view, the same
English article is kept, and now shows a "play" icon next to the
sentences that have been recorded. Clicking on a sentence will
highlight that sentence, and play the respective ASL video. Once
the video completes, it auto-progresses to the next sentence. There
is a playback control underneath the video so that the user can go
back, forward, redo, pause/play, and control the playback speed.
There is also a toggle to turn on or of the auto-progression. It is
possible that multiple users would sign the same English sentence,
so underneath the ASL video is a list of the users who submitted
videos for the currently selected English sentence. The user has the
option to switch between signers if they desire. A screenshot of a
sample reading view is shown in Figure 2.
4 USER STUDY
To explore the usability of our ASL Wiki site design, we ran a remote
user study, with Institutional Review Board (IRB) approval. In this
user study, participants answered survey questions, tried out the
reading and recording views, and discussed interview questions
about their experience.
4.1 Participants
4.1.1 Recruitment. Participants were recruited via mailing lists,
social media posts, and snowball sampling. The recruitment criteria
was that they use ASL, are 18 or above years of age, and have a
computer with a webcam. 19 participants were recruited in total.
The sessions ran for about 1 hour, and participants were given a
$30 (USD) Amazon gift card for their participation.
4.1.2 Demographics. Out of the 19 participants, 15 identifed as
Deaf, 3 deaf, and 1 Hard-of-Hearing. 11 identifed as female, and
8 male. The average age of all participants was 26.1 with standard
deviation 2.2.
Participants self-reported their ASL fuency on a scale from 1 (I
do not use ASL) to 7 (I am fuent). The average fuency was 6.4 with
a standard deviation of 1. Generally, all participants were educated,
with only 3 out of 19 not having a bachelor’s degree yet at the time
of participation. Participants were diverse, with 12 self-identifying
as White (e.g. German, Irish, English, Italian, Polish, French, etc), 5
as Asian (e.g. Chinese, Filipino, Asian Indian, Vietnamese, Korean,
Japanese, etc.), 1 as Black or African American (e.g. African Ameri-
can, Jamaican, Haitian, Nigerian, Ethiopian, Somalian, etc), and 1 as
Middle Eastern or North African (e.g. Lebanese, Iranian, Egyptian,
Syrian, Moroccan, Algerian, etc). The 19 participants came from 8
diferent U.S. states.
ASL Wiki ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 1: Screenshot of ASL Wiki homepage.

Figure 2: Screenshot of reading view of article "Caramel".

4.1.3 Prior Experience with ASL and English. All participants re-
ported that they read English text online daily (n=5) or multiple
times a day (n=14). It was reported in the demographics survey that
participants read English text via websites, books, articles, video
transcripts, and social media posts. Along with these 5 options, we
had also listed podcasts (and "other") as the answer-choices on this
survey question, but nobody selected that.
Participants were asked the question "How often have you en-
countered websites you wish provided ASL videos instead of or
in addition to English text?" 3 answered "multiple times a day", 5
"daily", 5 "weekly", 2 "monthly", 2 "less than once a year", and 2
"never".
All participants except one said that they watch ASL videos
online frequently (1 said "yearly", 3 "monthly", 5 "weekly", 5 "daily",
and 5 "multiple times a day"), typically through video blog (vlog)
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 3: Screenshot of recording view of article "Agriculture".
posts, YouTube videos and other social media videos. Participants
commented that they have seen content on various social media
platforms where someone is signing in ASL, and there are English
captions visible, so they have seen bilingual/bimodal content before,
and are comfortable with it. 10 out of 19 participants said that they
have at least once created content like this that had both ASL and
English, and that they created the ASL video frst and added English
subtitles afterwards. 9 out of these 10 did this to post on social media,
where they have both DHH and hearing friends, and 1 said they
only did it for a homework assignment or class project in college.
4.2 Procedure
An online form walked participants through the study procedures
while a DHH fuent ASL signer was on a video call with the partic-
ipant. Each participant was scheduled for their own session, and
the entire procedures took approximately 1 hour. The procedures
were as follows:
(1) Consent: Participants engaged in a consent process with
IRB-approved language through the online form. The re-
searcher on the video call checked whether the participant
needed any portion of the consent language signed in ASL
so that it was fully understood.
(2) Background: Through the online form, participants were
asked multiple-choice questions about their prior experience
with using English and ASL online.
(3) Reading: After this, they followed instructions on how to
access the ASL Wiki site and sign in, and were directed to the
“Caramel” article which had been entirely pre-recorded by a
DHH ASL signer from our research team. They engaged with
the interface to read this article until they were satisfed.
(4) Recording: Next, they were instructed to select any arti-
cle of their choice and record themselves signing. Since we
wanted to closely match a real-world experience of using
our site, participants were given the fexibility to record as
much (or as little) as they wanted to, but were told to use
the recording interface until they were confdent that they
Abraham Glasser, Fyodor Minakov, and Danielle Bragg
got and understood the full experience of recording and con-
tributing to the site, and were told they would discuss their
experience afterwards.
(5) Semi-structured interview: While the fuent DHH ASL
signer continued to be on a video call with them, they en-
gaged in a semi-structured interview with guiding questions
spanning short answer, long answer, and Likert-scaled ques-
tion items. The interview focused frst on the reading view,
asking about their experience and understandability using
the interface, and then were asked questions about their
experience and challenges (if any) while recording. Lastly,
questions were asked about the overall concept of the site,
what they liked and disliked, and whether they would recom-
mend the site to others. Appendix A.1 provides our interview
questions.
(6) Demographics: After the interview portion, participants re-
turned to the online form where they flled out demographics
and compensation information.
5 USER STUDY RESULTS
We discussed with each of the 19 DHH participants during the ex-
periment to gauge their reactions and experiences with the reading
and recording views of the interface. We evaluated how they used
the site, to understand their motivations, challenges and strategies,
and the benefts they took away from the site. We thematically ana-
lyzed the interview responses and performed statistical analysis of
their responses to the questionnaire. We also collected feedback and
identifed several target audiences who the users would recommend
the interface to.
5.1 Reading View
5.1.1 ASL vs. English. Participants valued having both English and
ASL versions of articles available for consumption. On average,
participants self-reported that, while reading the "Caramel" article
that our research team had entirely pre-recorded, they looked at
the English part 65% of the time, and the ASL video 35% of the
ASL Wiki
time. Participants explained that the English part is faster to read,
with P1 saying "...it’s faster to read and skim through. It’s more of a
habit because I’m used to reading English articles". P8, who reported
looking at the English part 50% of the time, said "I like ASL. [It] is
more visual and I can visualize it better, but for English I can read it
faster. If I just want to consume the content and save time, I would look
at the English 100%. If I wanted to fully understand, learn, visualize,
maybe 50/50 – I’d also be curious what it looks like in ASL".
Participants were asked to indicate how understandable the ASL
content and the English content in the "Caramel" article they viewed
was, on a scale from 1 (very difcult) to 5. For the ASL content,
the average was 4.6 (s.d. .7), and 4.8 (s.d. .4) for the English part.
Even though participants said that the English part was very easy
to understand, all 19 participants answered "Yes" to the question
"Was it helpful to view the content in both English and ASL?" P11
explained "Yes, I can imagine how it would be helpful for the general.
It’s a nice tool for me to use, and I would like having it even if I don’t
use it much".
5.1.2 Interface Usability. On a scale from 1 (very difcult) to 5
(very easy), participants said that the interface was very easy to
use, giving it an average of 4.5. Most of the difculty came from not
having prior experience and not knowing what to expect with the
interface, e.g. P10 saying "I didn’t think there was any information
overload – in the beginning I wasn’t fully sure what to do. Maybe the
frst sentence could be highlighted with the video, that would make it
more clear there is ASL there". P2 commented "at frst when I opened
it, I wasn’t sure what to do – my eyes caught the English part frst,
and I ignored the left half – and then it took me a while to realize
that the left side was empty until I clicked on some text, and the video
player showed up. [...] I think there should have been some kind of
tutor/illustrations with directions of how to use this site before I went
ahead and looked at an actual article".
Most participants (12 out of 19) did not use the upvote/downvote
button that was available to them while viewing the "Caramel"
article. Some participants said that they did not see it, while some
did but decided not to use it. P12 said that they do not use it in
general, such as upvoting/downvoting on Reddit, liking/disliking on
Facebook or YouTube. P9 said "I didn’t know about the feature until
I arrived at this question. I normally skim through contents", and P14
said "I wasn’t focusing on providing feedback on performance". Those
who did use it generally said that they wanted to give feedback,
with P4 saying "I wanted to give feedback on the video, so I clicked yes
– I noticed the signing was clear and matched the English so I went
ahead and clicked yes". Some participants such as P6 emphasized
it was important: "I think it’s important to use, yes I would use it, it
gives feedback to other people and I can help this website advance
and develop in the future and make sure it has good content", and P16
suggested it would prevent misunderstandings, saying "I don’t want
some signers to use wrong signs or say it in the wrong way which will
make viewers misunderstand. We want to avoid that", and P11 made
an analogy to real-world applications they’ve seen, reminiscing
"Yes, it’s the same as FAQs or other articles that say “was this article
helpful?” – this is the same situation".
We collected some feedback about the interface, to understand
how our interface could be improved and help inform future work
on such bilingual interfaces. These feedbacks typically consisted
ASSETS ’22, October 23–26, 2022, Athens, Greece
of user interface preferences and suggestions, such as coloring
and layout styling. There were also some suggestions about the
fundamental system. P7 suggested a diferent layout: "For me, I
would prefer top and bottom rather than side by side, so it’s kind of like
captions. It was a little challenging for me to have it side by side". P13
suggested making the recorded videos easier to fnd: "One suggestion
I have is that it might be nice to have a separate scrolling bar other
than the browser one where it’ll indicate the recorded statement bits.
E.g. code changes in a code review". Besides these feedbacks, users
also complimented the interface, P4 said "I liked the clarity, green
highlight, follow each other, I liked the time/playback, matching", P13
"What I liked about the interface is that each statement and section is
reasonably spaced out which makes it easier to read and I like how
there’s a clear indicator whether if there’s recordings for it or not".
5.2 Recording View
A total of 202 sentences were recorded from our 19 participants. On
average, participants recorded 11 sentences. Participants recorded
in 25 total articles from the Entertainment, Deaf Culture, Sports,
Books, Mathematics, Technology, Food, Geography, Art, and Poli-
tics categories.
5.2.1 Challenges and Strategies. Participants were asked if they
found any content challenging to record. They reported that they
generally selected articles from topics they thought they were the
most familiar and comfortable with. For instance, P11 said "I picked
the content I was most comfortable with, and it was straightforward
and just facts, so it wasn’t challenging. I can imagine if I picked a
STEM article or something complicated it would be challenging". Some
participants commented that they felt it was challenging to actually
translate the English content into ASL, because they were not sure
how to sign some words, or were not sure how to make it so it
wasn’t a word-for-word English-to-ASL translation, but rather a
concept-to-concept translation – P18 said "it can be a bit challenging
to keep it simple and brief yet informative", P16 refected that there
were "some words that I’m not sure if they have signs for them", and
P3 summarized "sometimes I have to reread and think about how I
will sign it to try not to be too English".
We asked participants if they had any strategies they employed
while recording content. Most said they did not – they commented
about trying not to be too "English" in their signing, with P1 saying
"I would read frst, and then think about my understanding of it, and
try my best to explain it in ASL. I wanted to avoid one to one or exact
English translations" and P7 commenting "I tried to fnd simpler
sentences, but most sentences required a lot of fngerspelling. It was
challenging to use it, I didn’t really think through it, I just read the
English part a couple times and then tried my best". We noticed that
not all participants started at the top of their selected articles. It
seems that some participants selected sentences throughout their
articles and did not always record consecutively.
5.2.2 Interface Usability. To ensure whether the interface itself
caused any signifcant issues for participants trying to record them-
selves, we asked them to rate, on a scale from 1 (very difcult) to
5 (very easy), how easy the interface was to use. For the 19 par-
ticipants, the average was 4.6. It did not appear that the interface
caused any further challenges to the recording experience, with
ASSETS ’22, October 23–26, 2022, Athens, Greece
P1 saying "I thought it was straightforward and simple", P3 "...liked
Redo/keep, add playback/review to watch it before deciding". Some
participants had suggestions about the interface to make their expe-
rience better, such as the seemingly abrupt countdown that started
as soon as they clicked on an English sentence to record, but there
were conficting responses as some participants said they disliked
it, e.g. P7 suggesting "maybe instead of auto countdown, I felt more
pressure, I would rather click on the sentence and then have a record
button" and P13 who said "I dislike that I can’t manually start record-
ing", while some liked it, e.g. P11 "... I liked the countdown, 3-2-1".
Participants also made some suggestions for extra features, such as
being able to trim the video before submitting, moving the place-
ment of the self-view, an explicit way to "skip" (rather than "redo"
or leaving the page).
5.3 General Experience
5.3.1 Enthusiasm. After participants had tried out our bilingual in-
terface, we asked whether they "wish more content online provided
both English and ASL?" from a scale from 1 (strongly disagree) to
5 (strongly agree). The average response was 4.6 (s.d. .76). Partici-
pants gave examples of where they have wished they had access
to both English and ASL. These examples included but were not
limited to news, podcasts, articles, social media and entertainment.
Participants mentioned the Daily Moth, where they have seen both
ASL and English captions or transcripts, but they mentioned that
these are selected specifc news, and they wish they had access to a
more broad, general selection of news around the world. Some par-
ticipants mentioned that they wish they had this kind of bilingual
resource when they were learning about things for their classes,
projects, and homework. These fndings suggest that people may
want to use a tool, similar to our novel interface, in the real-world.
Despite the 19 participants’ desire of having more content online
provided in both English and ASL, they were not as interested in
generating this sort of content themselves. When they were asked
"Would you be interested in generating content available in both
English and ASL? (1-5: Strongly disagree– Strongly agree)", their
average response was 3.6 (s.d. 1.6). Some of their rationale included
not wanting to record themselves and/or posting publicly, with P1
saying "I personally would not, because I personally don’t like record-
ing myself and posting online publicly", P10 "No, because I feel like I’m
signing wrong, or feel that people would judge my signing for being
English, etc", P17 "No, I’m a camera shy". Participants who indicated
that they are interested were inclined to do so because they felt
they would be giving back to the community, and supporting this
concept of accessibility, e.g. P2 "Yes, I wouldn’t mind – because I feel
like there is a lot of ASL content out there that is not neutral, where
the people who are signing are biased, or give biased information.
This would be nice and I would like to help increase access while still
keeping neutral and spreading information in a neutral manner" and
P3 "Yes, because if I can get access like this, why not I give back, I
don’t want others to miss out".
5.3.2 Personal benefits. One participant mentioned that the site
would beneft them because they can use it while teaching, to make
sure their students have access and can understand the content
fully. Many participants mentioned the site would help them un-
derstand content better and go through content faster, since they
Abraham Glasser, Fyodor Minakov, and Danielle Bragg
wouldn’t have to spend time looking up specifc English words in a
separate interface and/or re-reading the English text multiple times.
Some participants said that this would also help them improve
their signing and presentation skills, since they could beneft from
watching their own videos, or pick up new signs for unfamiliar
words. For instance, P14 said "I can improve how best I can interpret
English in ASL", P4 similarly saying "If I record, I could beneft from
watching my own videos, I will see if I signed it clear and understood
it well. I would also beneft from reading myself, and others would
beneft by reading my videos that I contribute". When we asked them
what kind of content they would like to see on the website, they
mentioned things they were studying, e.g. P2 "related to my major,
tutorials on 3d design software, art, technology, art terminology, for
example gothic art history, etc.", things they were interested in, as
P11 brought up "nutrition, diets, women’s health, for example there’s
a lot of things that are related to hormones, specifc foods afecting
things, having ASL there would be nice", general news, information
and topics, with P8 saying "news, health, podcasts, could be a safe
place for community involvement, like an area for people to post
news around the world, gaming area, etc. Make subcommunities for
gamers, etc, same concept as Facebook groups, Reddit subreddits, etc.
But everyone is deaf and uses ASL", among several others. Many
participants were very supportive of the idea, and did not care what
kind of content is available, as long as a lot is available, with P19
saying "... every site should have this option, all kind of topics are
welcomed", P8 agreeing "as much as possible, no limits", P5 "there’s so
many topics to choose from, I would just pick the best and most infor-
mative articles for education", and P7 "not that I can think of, general
Wikipedia articles would be good". This shows that participants were
very supportive of the site.
5.3.3 Concerns. We asked the participants whether they had any
concerns while using a site like this. 7 out of 19 participants ex-
plicitly brought up the concern that there wasn’t control over the
quality of users’ submitted videos. For instance, many commented
that people may not have professional backgrounds, or that they
may have something inappropriate or unintentional (such as other
people) in the backgrounds of the videos they submit. People also
mentioned that users have varying devices and webcam technolo-
gies, so that the quality of the videos themselves may not be as good
as they’d like – perhaps the lighting would be bad, the video would
be choppy or blurry, etc. A few participants also mentioned that
the site may fnd users who misinterpret or inaccurately translate
content. P18 said "It can be misinterpreted easily if the translator is
not professional or a novice", P11 brought up that "not everyone can
translate well, so that would be my concern – there might be some bad
videos. I recommend having STEM topics assigned to people who are
specialists in that feld". A participant also brought up the issue of
privacy, stating that they are concerned about the privacy of their
data, and who would "own" it and who would be able to access it,
especially if it was public.
5.3.4 Participant Impression. Overall, participants said that they
enjoyed using the website, and that they thought it was "cool to
use". When asked "Would you want to use a website like this to
read content in the future?", 14 participants said "Yes", and 5 said
"No". The participants who said "No" said that they are already
comfortable with reading English text alone, and do not require
ASL Wiki
ASL for reading comprehension. Despite this, the participants, on
average, said 4.5 to the question "How likely are you to recom-
mend this website to others? (1-5: very unlikely – very likely)".
Participants suggested many diferent groups of people who they
would recommend this site to. They would recommend it to DHH
individuals, because of the communication barriers they face, as P2
said "I feel for DHH people, and others who are not good at English,
and have communication barriers and have a lack of education, they
can learn well through this site", P7 said "I would recommend it to
people who I know grew up signing and struggle with English, they
could improve their reading skills and understand content better", P16
"pretty much everyone with ASL especially for people who have weak
English skills", and P4 bringing up "international friends who don’t
know English very well, it would help understand English and ASL,
or other people whose frst language isn’t English".
Many participants mentioned they would recommend the site
to people who are learning ASL, since the site is bilingual and has
synchronized English and ASL content, as P3 says "friends who are
interpreters, on their own time learn ASL/translation, receptive skills,
signing skills", P15 "ASL students for learning and people who are
thriving to learn ASL", P17 "If the website has enough recordings or
gains popularly among users, I would recommend to a friend who
isn’t fuent in ASL", and P19 summarizing both ASL learners and
the DHH community "this would be great for people learning ASL.
They can practice their receptive skills, and learn how to follow the
ASL grammar structure and sign placements. This is also good for
every person in the deaf community who may prefer reading captions
only some days and ASL other days, or anyone who has a preference
in how they absorb information". Since the site has been seeded with
articles from Wikipedia, which are normally informative and neu-
tral, participants suggested people who often look up information,
or use information in their profession, such as researchers, with
P6 saying "school educational use, like for students to do research,
or college students/professionals to record videos, k-12, community
college, ...", P5 suggesting teachers "I would recommend it to teachers.
I think the website would be best for education and is very educational
rather than recreational, so teachers could use it to record content and
provide information online", and P8 recommending learners: "Maybe
people who want to learn more things, learners, people who typically
look stuf up and read things".
6 TRANSLATION QUALITY EXPLORATION
While our ASL Wiki site was designed to facilitate translation
contributions, how the interface design may impact translation
quality is unclear. Interpreters typically generate ASL translations
of English texts in large sections (e.g. paragraphs). In contrast, our
interface elicits of text segmented into sentences to enable readers
to access spot-translations within long texts. Our interface also
provides built-in mechanisms to facilitate the translation process
(e.g. marking completion progress within the text, providing the
text and recording interface in the same tool).
To explore the potential impact of the interface on translation
quality, we ran a small experiment comparing a set of recordings
generated through our interface to a comparable set generated
through state-of-the-art recording setups. Specifcally, we paid four
professional Deaf interpreters to record 20 articles in both setups,
ASSETS ’22, October 23–26, 2022, Athens, Greece
and then paid two fuent ASL experts to evaluate all the recordings,
and compared the results. Our results suggest that the quality of
translations created through ASL Wiki are comparable to those
created through standard state-of-the art setups, with potential
slight improvements to translation accuracy and recording quality.
6.1 Procedure
6.1.1 Video Generation. We paid four Certifed Deaf Interpreters
(CDIs)6 to translate a set of 20 Wikipedia articles twice – with
both our interface and with their standard translation setup. We
chose to work with professional Deaf interpreters in order to enable
comparison to state-of-the-art translations. Each CDI was assigned
5 articles to record twice. and we counterbalanced the procedure,
so that two CDIs started with our interface and then used their
standard setup, and the other two CDIs did the reverse.
In the standard recording procedure, the interpreters were given
access to the plain text, and asked to record a translation of the
text in sections. They were instructed to use their typical setup and
procedures for such jobs – for example, referencing the text and/or
personal notes and recording through a video camera app on their
laptop or smartphone. This is a standard type of translation job
taken on by professional ASL interpreters (e.g. to translate written
questions in a survey, or to translate consent form language).
Each CDI translated their own set of 5 Wikipedia articles. Each
set spanned a variety of topics, including both technical and non-
technical topics. In total, 17 topics were covered in these 20 articles
(identifed through topic modeling on the most popular 810 Eng-
lish Wikipedia articles): Geography, Entertainment, Sports, Deaf
Culture, History, Science, Mathematics, Medicine, Business, Poli-
tics, Technology, Military, Philosophy, Food, Books, Religion, Art.
Article length ranged from 105-627 words (avg 309), and from 4-29
sentences (avg 15). In total, we collected 308 recordings through
our ASL Wiki interface (corresponding to individual sentences),
and 111 recordings through state-of-the-art interpreter setups (cor-
responding to sections).
6.1.2 Video Evaluation. To compare the quality of the two record-
ing sets, we paid two fuent ASL linguists to evaluate each video
along fve dimensions. These dimensions capture the accuracy of
the translation from English to ASL (Q1), the quality of the ASL
independent of the English (Q2-Q3), and the completeness of the
data captured (Q4-Q5). The dimensions and exact questions that
the experts answered for each video are listed below. In addition,
the experts had the opportunity to enter additional notes for each
video, and we also engaged in a debrief meeting to gather their
feedback and observations about the video sets as a whole.
Q1) Translation accuracy: How well does the ASL recording con-
vey the meaning of the English? (Scale of 1-5)
Q2) Linguistic correctness: How correct is the ASL execution
(e.g. were there many mistakes with handshape, movement,
grammar, etc.)? (Scale of 1-5)
Q3) Signing naturalness: How natural is the ASL (i.e. how similar
is it to ASL you might run into in real life)? (Scale of 1-5)
Q4) Recording quality: How good is the recording quality (e.g. is
it blurry, is the lighting good, etc.)? (Scale of 1-5)
6 https://rid.org/rid-certifcation-overview/available-certifcation/cdi-certifcation/
ASSETS ’22, October 23–26, 2022, Athens, Greece
Q5) Signing captured: Is the full signing space captured in the
video (i.e. hands, torso, surrounding are)? (Yes/No)
6.2 Results
The expert evaluations of the recordings generated through our
interface and through the CDIs’ standard setups were comparable,
across all fve explored dimensions. Figure 4 shows the overall re-
sults – average score and standard error for Q1-Q4, and the percent
of videos that were evaluated as having captured the full signing
space for Q5. We ran two-sided Wilcoxon-Mann-Whitney tests with
Bonferonni correction to compare evaluations of the two interfaces
for Q1-4. For Q1 (Translation accuracy), Q2 (Linguistic correctness)
and Q3 (Signing naturalness), there was no statistically signifcant
diference (p>.0125). For Q4 (Recording quality), the test showed
statistical signifcance Q4 (U=83175.5, p<.005). We also ran a � 2
test to compare Q5-Signing captured, which was not statistically
signifcant (p>.05).
During our debriefng, the expert evaluators identifed some pat-
terns in the data. They noted that the recordings, in particular those
created through ASL Wiki, contained straight translations rather
than interpretations. For example, the interpreters did not tend to
elaborate on concepts from the text to ensure that the meaning in
ASL is clear, or to provide additional context not provided in the
text. Instead, they tended to stick to the exact text. They also noted
some examples where it seemed that the interpreters had not done
the full prep work to understand the content they were translating.
For example, this was evident to one expert in a translation of some
plant anatomy, which lacked the appropriate visual representation.
One expert also noted that they had expected to see a larger dif-
ference in quality between the recording sets, in particular due to
the diference in text segmentation lengths. They were surprised
that there was not a larger diference in translation accuracy and
quality for the longer and shorter excerpts.
7 DISCUSSION
Generally, the results of our exploratory studies suggest that it
may be possible to use specialized interfaces to crowdsource ASL
translations of English text, to provide valuable bilingual resources
to the community and to curate ASL data. Our user study results
suggest that users would fnd value in such a bilingual ASL and
English platform, and would be willing to contribute, especially if
incentivized. At the same time, our translation quality exploration
suggests that the interface enables high-quality translations. In this
section, we provide further discussion on our exploratory work,
the limitations of this initial work in this space, and related future
work.
7.1 User Experience
Because participants were not incentivized further for contributing
more videos during our user study, the majority of participants only
contributed until they fgured out and were satisfed with the user
interface and experience for the recording view, with an average of
11 sentences per user. It seemed that participants generally chose
to contribute to topics that were personally meaningful to them,
especially those who contributed a larger number of recordings. It is
possible that an expanded range of topics that interest more people
Abraham Glasser, Fyodor Minakov, and Danielle Bragg
would thus incentivize contributions from the community. Further
incentivization, such as credit for class or monetary payment could
also be benefcial to deployment at scale.
Participants indicated that the reading and recording interfaces
of our website design were easy to use. Even though participants all
thought the site was easy and intuitive to use, several would rather
only use it for reading bilingual content, rather than contributing
ASL videos. They thought it was helpful to view content in both
English and ASL, and mentioned several cases where they wish
they had this level of accessibility in media. They talked about some
of the challenges and strategies used while recording. The website
was strongly supported and all participants identifed populations
that they would recommend the site to. Participants also suggested
many diferent topics that could be added to the interface that would
beneft them and others in mind.
Even participants who commented that they were fuent in Eng-
lish and ASL still indicated that seeing content in a bilingual, bi-
modal form was useful. Even if it did not help them understand
the content itself better, some participants still mentioned that they
could pick up new signs or improve their signing and presentation
skills. Overall, participants enjoyed using the website, and identi-
fed several use cases and target audiences who they would highly
recommend the interface to.
During the interview portion of the user study, we collected
feedback from participants so that we could further iterate upon our
design. These feedbacks would also be useful for future researchers
who want to generalize our interface, and potentially use it for
other signed or written languages. While our exploratory user
study serves as a proof-of-concept, several research questions have
arised. We have identifed several research avenues and next steps
as a result of this work.
7.2 Translation Quality
In our translation quality exploration, it is possible that linguistic
correctness was slightly more reliable with our interface due to
reduced cognitive load. Our interface provided required shorter
excerpts of text to be translated. It also simplifed the recording
task by keeping track of where the user was within the text, auto-
progressing to the next excerpt, and providing the text and video
feedback in a single interface rather than requiring the interpreter
to manage two separate interfaces for these components. It is also
possible that the recording quality was slightly better on average
with our interface because the quality of the recording was less
dependent on the quality of apps that the interpreter has available
to them. While we did not provide hardware, we provided built-in
recording software in our website, unlike state-of-the-art setups
that are dependent on the recording software that interpreters have
access to and know how to use.
While our exploration suggests comparable translation quality
with ASL Wiki compared to state-of-the-art translation setups, it
still leaves open questions about the impact of isolated interface
components. For example, it would be interesting to examine the
efect of diferent text segmentations within our interface, possibly
ranging from individual words, to sentences, to paragraphs or sec-
tions. Similarly, it would be interesting to experiment with the efect
of diferent types of visual cues for orienting the translator within
ASL Wiki ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 4: Comparison of expert evaluations of ASL translations of 20 Wikipedia articles, recorded by CDIs through ASL Wiki
and a control state-of-the-art setup. For Q1-4, (Translation accuracy, Linguistic correctness, Signing naturalness, and Recording
quality), the bar chart shows the average and standard error of expert evaluation. For Q5 (Signing captured), the bar chart
shows the percent of recordings evaluated as having captured the full signing space.

a page of text. It is also possible that the impact of the interface on
the translation quality may vary depending on the experience or
fuency of the user.
7.3 Limitations and Future Work
There is a need for a larger, more longitudinal study to see how
users use the site over a period of time in the real world, rather
than a short 1 hour session where they use the site for the frst
time and answer survey and interview questions with a researcher.
Additionally, most of our participants already had a Bachelor’s
degree, which may have biased our results; as a result, it is impor-
tant for future studies to capture more diverse participants from
the DHH community. Such studies would allow for deeper insight
in user participation and behavior, and the additional data collec-
tion would enable deeper linguistic analysis and open up several
research questions.
Since some participants in our user study skipped sentences,
selecting nonconsecutive sentences to contribute, there are gaps
in the articles. Our user study participants supported the idea of
the website, said it was easy to use, but many of them said they
would personally not contribute themselves. To encourage users to
contribute in completeness, further research is needed to investigate
diferent incentivization methods. There are several ways we can
imagine this happening, such as strengthening the gamifcation
inside the website (emphasizing the experience points they earn as
they contribute, displaying a leaderboard of the top contributors),
or monetary compensation for some arbitrary milestone of amount
of ASL videos an user contributes. Another possible avenue to
investigate is educational tasks, e.g. ASL interpreting students could
contribute to gain credits for certifcation or program requirements.
For this user study, we chose to implement a stand-alone website
pre-populated with a sample of Wikipedia articles, limiting the type
of content available for participants. We chose this implementation,
rather than a web plug-in or other setup with broader content for
several reasons: ability to choose English texts that are open for pub-
lic use, utility to users in having a complete translation as opposed
to sparse translations across more content, and implementation
feasibility. Still, user study participants brought up many diferent
types of content and explained their experiences with other real-
world content. Consequently, other types of content, and expansive
interface designs including web plug-ins, should be explored. The
utility of our interface with other signed/written language pairs,
or exploring other potential user groups (e.g. those recommended
by participants, such as K-12 students) could also be investigated.
Diferent use cases may or may not require other interface changes,
which would be explored in this research avenue.
There were two major concerns brought up during our user study.
Users were concerned about the level of control over data quality –
since this is a crowdsourced approach, it is the contributors’ respon-
sibility to have a good background in their signed videos, ensure
there is good lighting, and that the video is not choppy or blurry.
The other major concern was privacy. This is a very complicated
topic ([25], [9], [7]), and more research is needed about privacy
concerns when it comes to crowdsourced ASL datasets. Another
data quality research question is whether the crowd would be able
to control the data quality at a bigger scale. We included an up-
vote/downvote button where participants could give feedback, but
ASSETS ’22, October 23–26, 2022, Athens, Greece
we did not study this further, since 12 out of 19 participants did not
use it. We also had a small number of sentences from each partici-
pant, but if a larger and more longitudinal study was conducted, it
could be investigated how users use this feature.
We have also run a small experiment comparing the quality of
translation recordings made through our interface and through a
state-of-the-art setup. This exploratory study suggested that the
quality of translations created through ASL Wiki are comparable
to those created through state-of-the-art setups, and potentially
might enable slight improvements. While this is promising, we have
not evaluated the crowd-generated dataset from our participants
(as we did not have a control dataset to compare to, since general
community members do not normally engage in translation). It
would be useful for future researchers to investigate this, as well as
to conduct in-depth linguistic analysis. For instance, it is possible
that our interface reduced the cognitive load of the signer, as well as
the technical requirements, which may have elicited more natural
and linguistically correct translations.
As mentioned above, privacy is another issue that may impact
the design and use of ASL Wiki and future work. The research
community has only recently begun to explore privacy concerns
related to sign language videos and thought about how they can
be addressed [9, 26]. This initial work began to explore the impact
of fltering videos, for example by blurring the video or anonymiz-
ing facial features. However, acceptability of these approaches is
poorly understood, and their technical implementation is limited.
Indeed, it is possible that the community might prefer diferent ap-
proaches altogether, for example protective licensing or enhanced
security and transparency of data use. Once a better understanding
of the privacy needs and appropriate solutions have been developed,
such techniques could be incorporated into ASL Wiki and similar
applications, and make a ripe area for future work.
8 CONCLUSION
The lack of sign language bilingual resources and the lack of sign
language datasets are difcult problems to solve, mainly due to the
cost, resources needed, and amount of human efort required to label
and annotate data. In this work we have addressed both of these
problems by presenting a novel interface. Our interface provides a
side-by-side ASL and English synchronized interface, streamlines
pre-labeled data collection, and enables a crowd to contribute to
piecemeal translation. We pioneer exploration of the question of
how to enable everyday signers to contribute to continuous content
translation eforts, and how DHH users would respond to crowd-
generated content.
ACKNOWLEDGMENTS
We thank Chinmay Singh for initial work on the prototype; Vanessa
Milan for design guidance; Bill Thies and Naomi Caselli for thought-
ful discussions; and Philip Rosenfeld, Paul Oka, and Mary Bellard
for general support for the project.
REFERENCES
[1] Oliver Alonzo, Matthew Seita, Abraham Glasser, and Matt Huenerfauth. 2020.
Automatic Text Simplifcation Tools for Deaf and Hard of Hearing Adults: Benefts
of Lexical Simplifcation and Providing Users with Autonomy. Association for
Abraham Glasser, Fyodor Minakov, and Danielle Bragg
Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/
3313831.3376563
[2] Oliver Alonzo, Jessica Trussell, Becca Dingman, and Matt Huenerfauth. 2021.
Comparison of Methods for Evaluating Complexity of Simplifed Texts among
Deaf and Hard-of-Hearing Adults at Diferent Literacy Levels. Association for
Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.
3445038
[3] Larwan Berke, Sushant Kafe, and Matt Huenerfauth. 2018. Methods for Eval-
uation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at
Diferent Reading Literacy Levels. In Proceedings of the 2018 CHI Conference
on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18).
Association for Computing Machinery, New York, NY, USA, 1–12. https:
//doi.org/10.1145/3173574.3173665
[4] Jefrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller,
Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White,
et al. 2010. Vizwiz: nearly real-time answers to visual questions. In Proceedings
of the 23nd annual ACM symposium on User interface software and technology.
333–342.
[5] Erin Brady, Meredith Ringel Morris, and Jefrey P Bigham. 2015. Gauging re-
ceptiveness to social microvolunteering. In Proceedings of the 33rd Annual ACM
Conference on Human Factors in Computing Systems. 1055–1064.
[6] Danielle Bragg, Naomi Caselli, John W. Gallagher, Miriam Goldberg, Courtney J.
Oka, and William Thies. 2021. ASL Sea Battle: Gamifying Sign Language Data
Collection. Association for Computing Machinery, New York, NY, USA. https:
//doi.org/10.1145/3411764.3445416
[7] Danielle Bragg, Naomi Caselli, Julie A. Hochgesang, Matt Huenerfauth, Leah
Katz-Hernandez, Oscar Koller, Raja Kushalnagar, Christian Vogler, and Richard E.
Ladner. 2021. The FATE Landscape of Sign Language AI Datasets: An Interdis-
ciplinary Perspective. ACM Trans. Access. Comput. 14, 2, Article 7 (jul 2021),
45 pages. https://doi.org/10.1145/3436996
[8] Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault,
Annelies Brafort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa
Verhoef, Christian Vogler, and Meredith Ringel Morris. 2019. Sign Language
Recognition, Generation, and Translation: An Interdisciplinary Perspective. In
The 21st International ACM SIGACCESS Conference on Computers and Accessibility
(Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New
York, NY, USA, 16–31. https://doi.org/10.1145/3308561.3353774
[9] Danielle Bragg, Oscar Koller, Naomi Caselli, and William Thies. 2020. Explor-
ing Collection of Sign Language Datasets: Privacy, Participation, and Model
Performance. In The 22nd International ACM SIGACCESS Conference on Com-
puters and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for
Computing Machinery, New York, NY, USA, Article 33, 14 pages. https:
//doi.org/10.1145/3373625.3417024
[10] Danielle Bragg, Kyle Rector, and Richard E Ladner. 2015. A user-powered Amer-
ican Sign Language dictionary. In Proceedings of the 18th ACM Conference on
Computer Supported Cooperative Work & Social Computing. 1837–1848.
[11] B Cartwright. 2017. Signing Savvy. https://www.signingsavvy.com/
[12] ASL Clear. 2022. https://aslclear.org/
[13] Helen Cooper, Eng-Jon Ong, Nicolas Pugeault, and Richard Bowden. 2012. Sign
Language Recognition Using Sub-Units. J. Mach. Learn. Res. 13, 1 (jul 2012),
2205–2231.
[14] ASL Core. 2022. https://aslcore.org/
[15] Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth
DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. 2021. How2Sign:
A Large-scale Multimodal Dataset for Continuous American Sign Language. In
Conference on Computer Vision and Pattern Recognition (CVPR).
[16] Karen Emmorey, Chuchu Li, Jennifer Petrich, and Tamar H. Gollan. 2020. Turning
languages on and of: Switching into and out of code-blends reveals the nature of
bilingual language control. Journal of Experimental Psychology: Learning, Memory,
and Cognition 46, 3 (2020), 443–454. https://doi.org/10.1037/xlm0000734
[17] Jens Forster, Christoph Schmidt, Thomas Hoyoux, Oscar Koller, Uwe Zelle, Justus
Piater, and Hermann Ney. 2012. Rwth-phoenix-weather: A large vocabulary
sign language recognition and translation corpus. In Proceedings of the Eighth
International Conference on Language Resources and Evaluation (LREC’12). 3785–
3789.
[18] Ann Grafstein. 2002. HandSpeak: A Sign Language Dictionary Online. https:
//www.handspeak.com/
[19] Julie Hochgesang, Onno Crasborn, and Diane Lillo-Martin. 2022. Sign Bank.
https://aslsignbank.haskins.yale.edu/
[20] Leala Holcomb and Jonathan McMillan. 2022. http://www.handsland.com/
[21] Matt Huenerfauth and Vicki Hanson. 2009. Sign language in the interface: access
for deaf signers. Universal Access Handbook. NJ: Erlbaum 38 (2009), 14.
[22] Deaf Studies Digital Journal. 2009. https://www.deafstudiesdigitaljournal.org/
[23] Walter Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato
Borrello, Raja Kushalnagar, and Jefrey Bigham. 2012. Real-time captioning by
groups of non-experts. In Proceedings of the 25th annual ACM symposium on User
interface software and technology. 23–34.
ASL Wiki
[24] Walter S Lasecki, Christopher D Miller, Raja Kushalnagar, and Jefrey P Bigham.
2013. Legion scribe: real-time captioning by the non-experts. In Proceedings of
the 10th International Cross-Disciplinary Conference on Web Accessibility. 1–2.
[25] Sooyeon Lee, Abraham Glasser, Becca Dingman, Zhaoyang Xia, Dimitris Metaxas,
Carol Neidle, and Matt Huenerfauth. 2021. American Sign Language Video
Anonymization to Support Online Participation of Deaf and Hard of Hearing
Users. Association for Computing Machinery, New York, NY, USA. https:
//doi.org/10.1145/3441852.3471200
[26] Sooyeon Lee, Abraham Glasser, Becca Dingman, Zhaoyang Xia, Dimitris Metaxas,
Carol Neidle, and Matt Huenerfauth. 2021. American Sign Language Video
Anonymization to Support Online Participation of Deaf and Hard of Hearing
Users. In The 23nd International ACM SIGACCESS Conference on Computers and
Accessibility.
[27] Colin Lualdi. 2022. Sign School. https://www.signschool.com/
[28] Matt Malzkuhn and Melissa Malzkuhn. 2022. The ASL App. https://theaslapp.
com/
[29] The Daily Moth. 2022. The Daily Moth. https://www.dailymoth.com/
[30] National Association of the Deaf. 2022. Position Statement On ASL and Eng-
lish Bilingual Education. https://www.nad.org/about-us/position-statements/
position-statement-on-asl-and-english-bilingual-education/
[31] Eng-Jon Ong, Helen Cooper, Nicolas Pugeault, and R. Bowden. 2012. Sign Lan-
guage Recognition using Sequential Pattern Trees. 2012 IEEE Conference on
Computer Vision and Pattern Recognition (2012), 2200–2207.
[32] Eng-Jon Ong, Oscar Koller, Nicolas Pugeault, and Richard Bowden. 2014. Sign
Spotting Using Hierarchical Sequential Patterns with Temporal Intervals. In
2014 IEEE Conference on Computer Vision and Pattern Recognition. 1931–1938.
https://doi.org/10.1109/CVPR.2014.248
[33] Jingnan Peng. 2020. Bringing light to the news, for those who can’t hear it
(video). https://www.csmonitor.com/The-Culture/2020/0731/Bringing-light-to-
the-news-for-those-who-can-t-hear-it-video
[34] Manaswi Saha, Michael Saugstad, Hanuma Teja Maddali, Aileen Zeng, Ryan
Holland, Steven Bower, Aditya Dash, Sage Chen, Anthony Li, Kotaro Hara, et al.
2019. Project sidewalk: A web-based crowdsourcing tool for collecting sidewalk
accessibility data at scale. In Proceedings of the 2019 CHI Conference on Human
Factors in Computing Systems. 1–14.
[35] Elliot Salisbury, Ece Kamar, and Meredith Morris. 2017. Toward scalable social
alt text: Conversational crowdsourcing as a tool for refning vision-to-language
technology for the blind. In Proceedings of the AAAI Conference on Human Com-
putation and Crowdsourcing, Vol. 5. 147–156.
[36] Elliot Salisbury, Ece Kamar, and Meredith Ringel Morris. 2018. Evaluating and
Complementing Vision-to-Language Technology for People who are Blind with
Conversational Crowdsourcing.. In IJCAI. 5349–5353.
[37] Zed Sevcikova Sehyr, Naomi Caselli, Ariel M Cohen-Goldberg, and Karen Em-
morey. 2021. The ASL-LEX 2.0 Project: A Database of Lexical and Phono-
logical Properties for 2,723 Signs in American Sign Language. The Jour-
nal of Deaf Studies and Deaf Education 26, 2 (02 2021), 263–277. https://
doi.org/10.1093/deafed/enaa038 arXiv:https://academic.oup.com/jdsde/article-
pdf/26/2/263/36643382/enaa038.pdf
[38] Ather Sharif, Paari Gopal, Michael Saugstad, Shiven Bhatt, Raymond Fok, Galen
Weld, Kavi Asher Mankof Dey, and Jon E. Froehlich. 2021. Experimental Crowd+
AI Approaches to Track Accessibility Features in Sidewalk Intersections Over
Time. In The 23rd International ACM SIGACCESS Conference on Computers and
Accessibility. 1–5.
[39] Carol Bloomquist Traxler. 2000. The Stanford Achievement Test, 9th
Edition: National Norming and Performance Standards for Deaf and
Hard-of-Hearing Students. The Journal of Deaf Studies and Deaf Ed-
ucation 5, 4 (09 2000), 337–348. https://doi.org/10.1093/deafed/5.4.337
arXiv:https://academic.oup.com/jdsde/article-pdf/5/4/337/9835826/337.pdf
[40] Wikipedia. 2001. https://www.wikipedia.org/
[41] Wikipedia. 2022. Deaf News. https://en.wikipedia.org/wiki/Deaf_News
[42] Alicia Wooten and Barbara Spiecker. 2022. https://www.atomichands.com/
A APPENDIX
A.1 Semi-structured user study interview
questions
Below is the semi-structured interview questions that were dis-
cussed with participants as part of the user study:
• Role/relation to ASL: What’s your role/relationship with
ASL? (e.g. native speaker, primary language, ASL teacher,
use ASL at work, etc...)
Reading
ASSETS ’22, October 23–26, 2022, Athens, Greece
• Did you primarily look at the ASL or English part? [Follow
up to estimate percentage (0% ASL 100% English vs 100%
ASL 0% English)]
• How did viewing diferent signers afect your experience?
(If applicable)
• On a scale from 1-5 (1- very difcult), how understandable
was the ASL content you viewed? Can you explain why you
chose this number?
• On a scale from 1-5 (1-very difcult), how understandable
was the English content you viewed? Can you explain why
you chose this number?
• Was it helpful to view the content in both English and ASL?
Why or why not?
• Did you use the upvote/downvote feature? Why or why not?
• How easy was the interface to use? (1-5: Very difcult –
Very easy) If difcult, did information overload contribute
to difculties?
• What did you like or dislike about the interface?
Recording
• Did you fnd any content challenging to record? If so, what
made it challenging?
• Did you use any strategies while recording content? If so,
what were they?
• On a scale from 1-5, how easy was the interface to use? (1-5:
Very difcult– Very easy)
• What did you like or dislike about the interface?
Desirability
• Do you wish more content online provided both English and
ASL? (1-5: Strongly disagree– Strongly agree) If so, can you
give some examples of when you wanted content provided
in both languages?
• Would you be interested in generating content available in
both English and ASL? (1-5: Strongly disagree– Strongly
agree)
• What benefts do you feel this site ofer to you as a user, if
any?
• What concerns do you have in using a website like this, if
any?
• How enjoyable was using the website, overall? What did you
like/dislike?
• Would you want to use a website like this to read content in
the future? Why or why not? Is there diferent content you
would want to read (e.g. movie scripts, podcast, etc.)?
• What type of *Wikipedia* content would you want translated
(i.e., picking from the list of topics on the Wikipedia landing
page – food, math, Deaf culture, etc.)?
• Would you want to use a website like this to contribute
recordings in the future? Why or why not?
• How likely are you to recommend this website to others? (1-5:
Very likely – Very unlikely) If so, who would you recommend
this to, and for what purpose (e.g. ASL students for learning,
people with certain English/ASL fuency, etc.)?
Empowering Blind Musicians to Compose and Notate Music with
SoundCells
William Payne Fabiha Ahmed Michael Zachor
New York University New York University New York University
USA USA USA
william.payne@nyu.edu fsa253@nyu.edu mfz226@nyu.edu

Michael Gardell Isabel Huey R. Luke DuBois

New York University New York University New York University
USA USA USA
mg6950@nyu.edu ijh234@nyu.edu dubois@nyu.edu

Amy Hurst
New York University
USA
amyhurst@nyu.edu

ABSTRACT
Commercial technologies for notating music pose usage barriers
to blind and visually impaired (BVI) musicians because they use
graphic user interfaces and only produce visual, print scores. How-
ever, more research to date has studied how to make existing scores
available in braille or large print rather than understand the needs
and workfows of BVI musicians who notate new music. To address
this gap, we conducted a six-week remote study in which six BVI
musicians with wide-ranging backgrounds wrote original music
culminating in a live performance. To create their scores, partici-
pants used SoundCells, a product of ongoing co-design and testing
with BVI musicians that uses text to generate audio, print, and
braille music. Across three interviews, participants ofered diverse
and nuanced views of how text input could facilitate creative ex-
pression. We uncovered how vision ability, music experience, and
assistive technology preference afected how music was accessed
and traversed. From this research, we provide design recommenda-
tions for improving SoundCells’ input and output systems, discuss
how visual cues embedded in SoundCells’ syntax make learning
and remembering harder for people who can’t view it, and refect
on how our chosen methods resulted in high engagement.
CCS CONCEPTS
• Human-centered computing → Accessibility technologies;
Sound-based input / output.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544825
KEYWORDS
accessibility, music technology, blindness, visual impairments,
braille, music notation, creative expression, longitudinal study
ACM Reference Format:
William Payne, Fabiha Ahmed, Michael Zachor, Michael Gardell, Isabel
Huey, R. Luke DuBois, and Amy Hurst. 2022. Empowering Blind Musicians
to Compose and Notate Music with SoundCells. In The 24th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22),
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 14 pages.
https://doi.org/10.1145/3517428.3544825
1 INTRODUCTION
Recent HCI and accessibility research has studied how blind and
visually impaired (BVI) people use technology to create content
across wide-ranging mediums including photography [9], 3D mod-
eling [64], making and fabrication [10, 15, 59], writing [16], web
design [35], and podcasts [63], but the practices of BVI people who
use computers to notate music are understudied. Music notations,
including western print notation taught in most American schools,
are predominantly visual and cannot be read by many BVI people.
Many researchers [13, 25, 26, 36, 37] and community eforts [48, 72]
have developed systems for converting visual notation into braille
music and specially-formatted large print that can be read by BVI
musicians. While this important work makes existing music more
widely accessible, i.e. scores written by deceased, male composers,
it only partially addresses the needs of BVI music makers, including
composers, arrangers, students, and educators, who notate new
music. Commercial music notation tools predominately use graphic
user interfaces (GUI’s) with “what-you-see-is-what-you-get” inter-
actions that range from completely inaccessible to complex and
difcult to navigate for BVI musicians, while specialized braille
music tools limit collaboration since they do not produce visual
print scores [50, 52].
This study depicts the creative processes of BVI musicians as
they learn and use SoundCells, a text-based music notation software
that renders visual and braille music scores simultaneously [50].
ASSETS ’22, October 23–26, 2022, Athens, Greece
SoundCells’ design was motivated by prior research with experi-
enced BVI musicians who discussed gaps in existing software and
identifed a need for converting between braille and print notations
without complex tool chains [52]. SoundCells emerged frst as a
text-based music notation prototype validated through a design
probe [4]. Then, it was co-developed with two experienced blind
musicians and evaluated with fve other blind musicians [50]. While
prior research gathered BVI users’ initial reactions to SoundCells’
text input and multimodal output, this is the frst evaluation of the
system in use over time. Six BVI Musicians, who range widely in vi-
sion ability and prior composition and technology experience, used
SoundCells for six weeks and wrote original, short compositions
that were played in a live performance.
The primary contributions of this research are as follows:
• We depict how BVI musicians with wide-ranging musical
backgrounds, vision abilities, and equipment engaged mean-
ingfully with SoundCells, a new system for creative expres-
sion.
• We compile participant ideas and design recommendations
impacting music, text, and multimodal interactions.
• We demonstrate how our study design presented opportuni-
ties for self-expression, motivated engagement with technol-
ogy towards a common goal, and utilized the web increasing
access across the six-week study.
We frst identify areas of related research that SoundCells builds
upon including music technology accessibility, braille music, music
coding, and accessibility of text interfaces (§2). Then, we provide a
brief overview of the current SoundCells system (§3). We describe
our methodology based around a six-week virtual class culminating
in a performance (§4). Finally, we describe how participants engaged
with SoundCells and gave us feedback for how to improve it (§5).
We discuss and connect these fndings to the design of text-based
systems, multimodal interactions, and tools for creative expression
(§6).
2 RELATED WORK
This study builds on recent eforts to broaden access to music tech-
nology for creation, performance, and education. Specifcally, we
draw on existing systems and workfows for producing braille mu-
sic notation, research in accessible programming and text editing,
and environments for programmatically making music.
2.1 Accessibility of Music Technology for Blind
and Visually Impaired People
Researchers are increasingly committed to improving and measur-
ing accessibility in music technology [38] and expanding designers’
understanding of when, how, and why people who identify as liv-
ing with a disability or access requirement use music technology
[21]. BVI music learners who need accessible notation are well-
studied, e.g. [3, 41, 44], and recent work depicts how online support
networks and toolchains made up of commercial and custom hard-
ware/software enable creative practices [27, 52, 63].
A 2019 review of accessible digital musical instruments found
that few interactions have been designed and tested with BVI mu-
sicians compared to other populations with disabilities [20]. Past
Payne et al.
research exploring multisensory alternatives includes tactile inter-
actions with MIDI [28], audio navigation [71], audio equalization
[34], and tabletop collaboration [47]. Groove Pizza, a web-based
drum machine, explores keyboard input and auditory output, in-
cluding sonifcation and text-to-speech [51]. We build on past work
by closely examining how BVI music makers with diverse expertise
incorporate assistive technology and utilize multiple modalities to
create original music.
2.2 Braille Music Notation and Access
Braille music is harder to write and obtain than print music. De-
spite its long history and standardization [24, 65], braille music
is not typically sold by publishers or available in public domain
catalogues such as the International Music Score Library Project
(IMSLP) [53]. For example, IMSLP contains a catalog of more than
650,000 PDF scores compared to the NLS collection of instructional
music materials and scores made up of about 25,000 braille, large
print, and audio fles [46]. If a braille version of a score cannot
be obtained, schools, agencies, or end-users may translate print
music to braille following a multi-phase, rigorous process: One or
more sighted people copy a PDF in notation software, export to
MusicXML, convert to braille with a commercial or open-source
program [13, 14, 25, 26, 36, 37], and fnally send to a braille music
expert to fnalize layout and correct errors caused by human or com-
puter error [33]. Braille music notation software such as the Braille
Music Editor and Braille Music Notator [18, 61] and generic braille
translation software such as Duxbury [17] can be used to notate
braille scores directly, but the above steps are required in reverse
to generate print for sighted musicians. Lime Aloud is capable of
producing both print and braille music [14], but it provides a limited
view of the braille score during editing, only runs on Windows, and
is expensive. SoundCells ofers an alternative to existing eforts. It
incorporates multiple forms of notation during composition instead
of requiring additional work, software, and expertise to convert a
score after it has been completed.
2.3 Code and Text Interface Accessibility
ABC text [73] is a shorthand ASCII music notation that can be
directly interpreted by screenreaders. Many commercial music ap-
plications use GUIs that are impossible or unwieldy to navigate with
screen readers [52]. While text input may improve usability, it can
introduce new challenges. As with developing software with code,
notating music with text requires memorizing and understanding
syntax, debugging mistakes, running mental calculations to track
small and large structures, and planning approaches to open-ended
problems. For example, Payne et. al interviewed two blind musi-
cians who used LilyPond [23], a text-based music notation language
comparable to LATEX for musical scores [52], but found the syntax
challenging and the functionality geared mainly towards producing
PDF fles. Because blind developers may experience difculty nav-
igating and conceptualizing code structures [42, 68], researchers
have designed interfaces more suited for screen readers. For ex-
ample, StructJumper and AudioHighlight present approaches for
skimming structured documents [5, 8], SodBeans and Wicked Au-
dio Debugger use audio cues to aid error debugging [66, 67], and
Empowering Blind Musicians to Compose and Notate Music with SoundCells
CodeTalk adds screen reader features to Visual Studio [54]. Sound-
Cells is analogous to the Web-ALAP LATEX environment designed
for screen reader users [6]. While in Web-ALAP, users write math
expressions, in SoundCells, users notate music with ABC text. In
this study, we explore how BVI musicians understand and work
with specialized text notation, and propose design opportunities
for improving text-based environments.
2.4 Computational Music Making for Novices
Computer music environments are no longer only used by experts
who possess signifcant music and code knowledge [74]. Today,
many introductory computational music environments provide ed-
ucational opportunities to make and remix music within diferent
programming languages and musical styles. For example, SonicPi,
a live-coding language based on Ruby, is used in algoraves, taught
in primary school classrooms, and included in the Rasberry Pi
OS [2]. EarSketch uses Python or Javascript to control and alter
playback of hip-hop grooves and samples within a Digital Audio
Workstation (DAW) [19]. DAWs tend to use highly visual interfaces
[52, 63]. Finally, JythonMusic has been used in cross-disciplinary
college courses to teach music and programming [40]. Each of these
systems use text for editing music, but very few studies have in-
corporated BVI musicians who may beneft from improved screen
reader access to text instead of a GUI. Sonic Pi was adapted by
developers of the tangible coding environment Code Jumper, but
brief tests found the syntax unapproachable for children ages 7–11
who used screen readers or text enlargement [43]. Furthermore,
SonicPi uses a custom integrated development environment (IDE)
that appears to support screen readers [1], but its accessibility has
not been formally evaluated. This study takes place in a learning
context and mainly engages novices who have little music tech-
nology, notation, and/or code experience. Accessible interaction
fndings are applicable to other music coding environments that
use diferent languages and styles of music.
3 SOUNDCELLS SYSTEM
SoundCells is a web application for quickly notating print and
braille music scores with text [4, 50]. SoundCells can be accessed
across desktop and mobile operating systems, browsers, and screen
readers. Built on open-source software, SoundCells’ source code is
available on GitHub [56], a stable prototype is deployed via Heroku
[55], and we published a detailed technical description [50]. We
describe the fundamental features of SoundCells relevant to this
research below.
3.1 Text Input
3.1.1 ABC Text. SoundCells separates musical content from its
presentation making it unique from most other notation software.
Music is generated from plain ASCII text written in a syntax called
ABC [62] that can be read directly by screen readers. ABC, along
with other text-based music notations such as LilyPond [23], is
akin to LATEX, while SoundCells is akin to Web-ALAP [6], a web-
based LATEX editor with built-in accessibility features and domain-
specifc tools and commands. In SoundCells, users edit text, while
in Lime Aloud, a popular notation software developed and used by
blind people [14], users edit a rendering of print music directly and
ASSETS ’22, October 23–26, 2022, Athens, Greece
generate the complete braille score outside the editing environment.
ABC text is simpler than LilyPond in part because it is not intended
for engraving print music and provides limited functionality for
visual formatting and layout. Many online libraries contain free
music in ABC text, especially folk tunes [12]. While learning ABC
text has been suggested to be potentially useful for blind musicians,
students, and novices [62], existing ABC editors do not support
braille output and are not designed for screen reader users.
3.1.2 ABC Parser. SoundCells uses a custom parser that locates
syntax errors and lists unsupported text fragments and suggested
fxes in a screen-reader accessible diagnostics view. It tracks mea-
sure completeness, whether a measure has the appropriate number
of notes, and lists incomplete measures in the diagnostics view.
Finally, it connects the user’s cursor position in the ABC text with
their position in the generated score, enabling text-to-speech and
music feedback described below (§3.2). SoundCells uses music21
[13], an open-source Python library, for fnal braille and print con-
version.
3.1.3 Editing Environment. The coding editor in SoundCells is built
on CodeMirror 6, an open-source project designed to better support
screen reader and keyboard users in web-based text editing [29, 31].
Keyboard commands and the diagnostics view are built directly in
the editor. SoundCells includes an alternative HTML text area as
VoiceOver users have reported bugs in CodeMirror 6 [30]. However,
the HTML text area lacks the same features as the CodeMirror
editor requiring the user navigate elsewhere on the page to access
similar information conveyed via key commands.
3.2 Multi-Modal Output
We designed SoundCells to leverage audio and tactile output during
both composition and production.
Music Playback: As a user types, they hear corresponding notes
played back immediately making the QWERTY keyboard act like
a MIDI piano keyboard. Audio playback refects the current state
of the score, not just the character typed, so if the user types D
below a line indicating a key signature of Db, then they will hear Db.
Additionally, the entire score may be played, or a single measure
may be looped indefnitely as a user edits.
Text-To-Speech: “Tell-me” commands or question mark com-
mands trigger local descriptions of the score based on cursor loca-
tion. A user types a ? followed by a character to request information
about a note, symbol, or region. For example, typing ? - m describes
the current measure number and completeness.
Braille Music: The generated braille score may be viewed in the
interface or felt through a connected refreshable braille display. In
SoundCells, braille music can be displayed in both Unicode and
ASCII formats.
Print Score: A visual print score located below the text editor
updates as the user writes ABC text. Large print scores can be made
using a slider to simultaneously modify multiple elements, includ-
ing note size, font size, measures per system (row), and systems per
page.
3.2.1 Sharing and Extensibility. SoundCells does not support ad-
vanced formatting and customization of braille or print, but it ex-
ports fles in standard formats. For example, An ASCII braille fle
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 1: SoundCells System: Main interface shown on the right (with documentation, text editor, diagnostics view, playback
controls, and print score) showing how ABC text yields print music and braille notation on a braille display. The Settings and
Save menus are shown expanded on the left.
may be opened in Duxbury [17]. A PDF can be downloaded in stan-
dard page sizes (8.5x11 or 11x17) and layouts (portrait or landscape),
with enlargement settings applied. A MusicXML fle can be opened
in commercial applications such as Sibelius [7]. Raw ABC text may
also be downloaded or copy/pasted into an email or chat message.
4 METHODS
In this section, we describe our methods in which six BVI partici-
pants used SoundCells to notate music during a six-week course,
and were interviewed three times to share their experiences and
ideas to improve SoundCells.
4.1 Course Design
To learn SoundCells, participants enrolled in a six-week virtual
course through The Filomen M. D’Agostino Greenberg (FMDG)
Music School. The course, entitled “Composing Music with ABC
and SoundCells,” was advertised to students age 16 and older as
an opportunity to learn a new technology for notating braille and
large print scores. Enrollment was free, and the advertisement
email made clear that participating in research was optional and
not required. The course met once per week for one hour remotely
over Zoom. Audio recordings of sessions were shared with stu-
dents as a learning aid. These class recordings are not used for
research because we wanted the learning environment to be a
safe space where participants could make mistakes and ask ques-
tions. Instead, we noted topics and conversations from class to ask
Payne et al.
follow-up questions during the interviews. Finally, we designed
a supplementary course website following web accessibility best
practices [75] to post course topics and share participant work. It
remains online (with participant names removed) as a resource for
learning SoundCells [57].
The course was structured around learning ABC text and the
SoundCells interface (Table 1). The frst four weeks covered new
topics, while Week 5 was an open-work period to prepare for a
fnal performance in Week 6, in which a musician with low vision
from FMDG School performed participant works live on soprano
saxophone. Our goal of this research and the course design was
to provide an authentic music making environment and clear mo-
tivation to become adept at using SoundCells. This research was
not aiming to measure the efectiveness of the curriculum or the
extent to which participants learned music theory or composition
concepts.
4.1.1 Take-Home Challenges. To increase engagement, we pro-
vided three options for participants to use SoundCells outside of
class through notation, remix, and composition challenges. Nota-
tion challenges were prompts to practice new syntax (e.g. debug-
ging incorrect excerpts, notating common chord progressions, etc.),
remix challenges were prompts to integrate new syntax into ex-
isting work, and composition challenges ofered guided activities
for writing new music using the most recent syntax. Five of the six
participants submitted work each week via email or an embedded
form, and participants chose to work on diferent activities. For
Empowering Blind Musicians to Compose and Notate Music with SoundCells
Table 1: Class and Interview Schedule
Wk. Main Concepts
1 Notes and Pitches
2 Rhythms
3 Chords
4 Decorations (Other Musical Symbols)
5 Work Day
6 Performances
example, P5 worked primarily on the Notation Challenges fnding
composition “daunting,” while P3 worked primarily on her fnal
composition extending it and integrating new syntax each week.
4.1.2 Final Performance. Participants were tasked with notating
a short, 12–24 measure composition that they could perform, or
would be performed by a professional musician who has low vision
on soprano saxophone. Our requirements indicated that the pieces
had to be complete without mistakes and had to include certain
information like a composer and title. Five participants chose to
have the professional play their piece, and P3 chose to perform
herself because she notated chords to be played on piano. The
professional musician was not enrolled in the course. They asked
for three systems, or rows of measures, per page using 11x17 paper
in landscape orientation. Researchers used SoundCells to create
these scores from ABC text submitted by the participants. We used
the magnifcation slider to adjust the size of the notation until it
reached three systems per page. The musician requested we shrink
P1’s score from fve to four pages allowing it to ft on two music
stands preventing a page turn. The performance of each piece was
followed by short discussion. All participants attended along with
the musician and the head instructor from FMDG School.
4.2 Interview Protocols
We conducted three semi-structured interviews with all participants
at the beginning, middle, and end of the course to capture initial re-
actions, in-progress experiences, and refections. We wrote a generic
script for each interview along with individualized questions de-
termined by participant characteristics, prior interview responses,
and/or class interactions. For example, we asked P5 about print
music and not braille music which she does not read, we asked
P2 to compare SoundCells to other music notation software given
his signifcant prior experience, and we asked P4 repeatedly to dis-
cuss her workfow for notating rhythm which she indicated feeling
challenged by.
4.2.1 Interview 1. We asked participants to describe their musical
backgrounds, prior experience notating music with technology or
by hand, and motivations for enrolling. Questions were adapted
from previous interview studies with blind musicians [4, 52]. Par-
ticipants new to SoundCells (P3, P4, P5) were asked to share initial
reactions and where they felt challenged. Participants who con-
tributed to SoundCells’ design (P1) or tested SoundCells in an earlier
study (P2, P6) were asked to consider their frst experience with
SoundCells.
ASSETS ’22, October 23–26, 2022, Athens, Greece
SoundCells Features Interview
Interface Overview, Music Playback -
“Tell-Me” Commands 1
Settings and Save Menus -
Diagnostics View 2
- -
- 3
4.2.2 Interview 2. This interview emphasized the 3–4 weeks of
challenge activities participants had completed up to that point.
We frst asked participants to refect on what they had learned so
far and what they still hope to accomplish. Most of this interview
centered around a specifc excerpt chosen by us or the participant.
We then asked them to describe their process and to either to fnd
and correct an existing mistake in the excerpt, such as an incomplete
measure, or to demonstrate how they would.
4.2.3 Interview 3. We asked participants to discuss their entire
experience using SoundCells including their process working on
the fnal compositions, feelings after the live performance, ideas
for improving SoundCells, and vision for using it in the future.
4.3 Artifacts
In addition to the interviews, we gathered materials created by par-
ticipants across the course including exercises, improvisations, and
fnal compositions. A form incorporated into SoundCells enabled
participants to report bugs, issues, or questions, but most elected
to communicate via email.
4.4 Participants
Six BVI musicians enrolled in the course, and all elected to partici-
pate in the study and engage in all three interviews. As shown in
Table 2, the participants have diverse experience. P1 contributed
to the design of SoundCells [50] and has a degree in Music Com-
position, and P2 teaches Music Technology and has decades of
experience composing and performing professionally. Others had
never composed original music and had little or no prior experience
using music software (P4, P5, P6). Two participants identify as male
(P1, P2) and the remaining four identify as female. P5 uses tech-
nology with magnifcation and currently reads large print music,
while the the others use screen readers and read braille music.
Participants used a range of AT to access SoundCells. P3 and
P6 used both a screen reader and braille display referencing the
braille music often as they notated. P2 and P4 only used screen
readers. P4 made heavy use of SoundCells features including tell-
me commands, the diagnostics view, music playback, and looping,
while P2 primarily played his composition back while he worked.
P1 primarily used a braille display to toggle back and forth between
braille music, and rarely enabled audio. P5 used a large screen with
system magnifcation.
ASSETS ’22, October 23–26, 2022, Athens, Greece Payne et al.

Table 2: Participants

Prior SoundCells Notation Technology

Participant School Role Notation Experience Identifcation Assistive Technology
Experience Experience
Technologist, College degree in composition. Lime, Duxbury. JAWS,
P1 Co-Designer Blind
Braille Music Transcriber Braille music expert. Uses daily 40-cell display

Decades of experience composing Lime, Sibelius. VoiceOver,

P2 Music Technology Instructor Usability Tester Blind
and arranging print and braille. Uses multiple days a week 40-cell display

Piano Instructor, Teaches braille music Lime. Visually JAWS,

P3 None
Braille Music Instructor using Perkins Brailler. Used over 20 years ago. Impaired 40-cell display

P4 Adult Student None None None Blind NVDA

Hand-notated music when

P5 Adult Student None None Low Vision Magnifcation
much younger before vision loss.

Learning braille music Lime. VoiceOver,

P6 High School Student Usability Tester Blind
for about 3 years. 4 months, once per week. 32-cell display

4.5 Analysis
We conducted a Thematic Analysis [11] on interview data gath-
ered, recorded, and transcribed verbatim to identify trends and id-
iosyncrasies in how SoundCells was used and understood. Themes
were identifed, coded, and analyzed by the research team: Two
researchers conducted each interview, while a third listened to the
recording asynchronously. After each round the research team dis-
cussed the interviews and thematic trends. Once all 18 interviews
across the three rounds were completed, the team generated themes
relating participant experiences to features of SoundCells. As this
study builds on prior analysis of SoundCells [4, 50], we used a com-
bination of deductive and inductive coding. Initial codes such as
“Audio Interface,” “Mistake Correction,” and “Design Recommenda-
tion,” seeded the coding process, while new codes, such as “Reaction
to Performance” and “Perception of ABC Text,” arose out of data
gathered during this study. Finally, we refned themes and selected
especially relevant examples and quotations for this publication.
We considered the musical excerpts submitted by participants and
measured the complexity of fnal compositions by counting musical
events and unique instances of notation (shown below in Table 3).
Challenges or works-in-progress sent to us by participants during
the six weeks were not evaluated because participants shared their
work voluntarily. We used these measurements to understand what
was notated with SoundCells, not as a way to evaluate efort or
creativity.
5 FINDINGS
Participants interacted with SoundCells diferently and had a range
of feedback, likely informed by their assistive technologies, prior
experience, and personal preferences. Despite the diversity across
participants, all successfully used SoundCells to compose original
short works that were performed live in the fnal class. In addi-
tion to the valuable feedback about the design of this tool (§§5.1.4,
5.2.5), this research illustrates the value of making tools for creative
expression accessible to an underserved population. For example,
participants were enthusiastic about the physical copies of their
scores and the live performance. On reading her embossed score for
the frst time, P6 exclaimed “I’m like, oh my goodness, this is my
own piece. Oh wow, this is actually really nice.”, and then on the
experience of hearing the saxophone player interpret her piece that
lacked rests, she said, “I was like that sounds really good, but wow I
didn’t give him time to breathe.” Final compositions varied greatly,
and as shown in Table 3, prior music technology and notation ex-
perience roughly correlated with music complexity, as estimated
by the length of music and range of notation used. Multiple partici-
pants commented on hearing each other’s works and recognizing
the wide range of skill, experience, and aesthetic. For example, P5
said, “I enjoyed that everybody’s pieces were very diferent. Each
person had a very diferent way of expressing themselves. And also,
you know I think it showed diferent peoples’ profciency with the
system.” Final compositions including ABC text, print, braille, and
recorded audio may be accessed online [58].
In the remainder of this section, we describe how participants in-
teracted with SoundCells’ text input (§5.1) and multimodal outputs
(§5.2.1), and we report their suggestions for improving SoundCells
to support future use (§5.3).
5.1 Input: Using Text as Notation
Most participants enjoyed learning ABC text and found SoundCells
to be an approachable tool for notating music. Surprisingly, we saw
very few excerpts submitted with incorrect syntax, the most com-
mon error being mismatched brackets. When asked, participants
expressed little concern about syntax stating that they tended to
notice if they typed the wrong thing, and felt the diagnostics view
did a good job highlighting incorrect characters. Many challenged
themselves to incorporate as many unique features of ABC text into
their fnal compositions as possible. Throughout our interviews,
participants refected on how text could facilitate or complicate
universal music notation processes – conceiving of musical ideas,
inputting notes, and writing full, complete, and rhythmically inter-
esting measures.
5.1.1 Perceptions of ABC Text as Music Notation. Our participants
had a range of experience notating braille or print music, with
P1, P2, and P3 possessing signifcant experience, P6 with some
experience, and P4 and P5 with little or no experience. Only P1
had previously notated original music using ABC text. We believe
Empowering Blind Musicians to Compose and Notate Music with SoundCells
Figure 2: P1’s Final Composition. Left: ABC text excerpt from the middle of his piece. Middle: Corresponding notation in large
print with three systems on 11x17 paper and the entire score printed on 8.5x11 paper. Right: Embossed Braille score on large,
11x11.5 paper.
this experience informed participants’ initial positive and negative
reactions to using ABC text. P3 expressed difculty remembering
syntax, “What is hard is to memorize what each key is, how would
I get fat, sharp, what key to press.” P5 noted surprise at using text
to represent music, “strange experience to type out letters and then
get the music from that rather than writing notes on a staf.” P6
expressed enthusiasm for learning a new paradigm, “one of the
things I actually like is the text editor.” As the course progressed,
many felt more comfortable and able to express their ideas, but all
participants, except for P1, felt that after six weeks they were not
experts in ABC text.
Participants identifed specifc aspects of ABC text that were
unintuitive and difcult to learn. Many highlighted pitch notation:
In ABC text, a letter’s case describes a note’s starting octave, while
characters placed before or after raise or lower it. For example, a'
is an “A” in ffth octave raised to sixth octave while C,, is a “C” in
fourth octave lowered to second octave. P2 understood the syntax
but questioned the behavior of specifc characters, “Octave-down,
octave-up is comma, apostrophe. What’s the relationship between
those keys, you know?” P3 also brought up these symbols, “comma,
apostrophe is not used in music. It’s used in lyrics.” Furthermore,
P5, the participant who reported feeling most overwhelmed in the
class “reached [her] saturation point and level at Class 3.” She com-
pletely avoided expression markings covered in Class 4 (Table 3).
In contrast, P1, a programmer and experienced SoundCells user,
found that text presented exciting musical possibilities because
“ABC fundamentals allow you to expand the rhythmic complexity
in a way that you just can’t do with standard Western notation, like
large prime fractions like seven over eleven.” For him, the current
syntax lacked the precision to notate microtones, or notes in be-
tween notes, so he left a comment in his fnal score to communicate
what he was not able to notate, “approximation of Arabic ‘maqam
sabah’ without 3/4 tones”.
5.1.2 QWERTY Keyboard as Input. Participants difered in how
they viewed the mechanical process of typing musical notes on
a computer keyboard. P1, P3, and P6 thought typing was simple
and intuitive. For example, P6 explained her strategy: “Sometimes
I improvise, mainly it’s like a no right or wrong type of thing...
If I have something that comes in my head I’m like, ‘You know
ASSETS ’22, October 23–26, 2022, Athens, Greece
what, I hear something, let me type it.’ ” In contrast, P2, P4, and
P5 described how typing ABC text remained tedious and required
concentration after six weeks. Their attention to locating correct
characters overwhelmed their ability to consider larger musical
structures. For example, P2 said, “Most notation programs, we use
a musical keyboard, we are playing the notes... You don’t have
to think so much about typing octaves, sharps and fats. You are
thinking about melody line, harmony line,” and later asked “Am I
concentrated on the music or just on how to write the notation?”
The difculties typing syntax led some participants to come up
with new strategies or cut back on elaborate composition goals. P5
worked iteratively and divided her fnal composition process into
multiple steps. First, she wrote down all the pitches, and later she
incorporated other musical elements. “I started playing with what
I had written down, and adding durations and rests and holds and
ties, and actually that sort-of brought it all together for me.”
When asked to compare learning SoundCells to other systems,
participants with prior experience reported that SoundCells re-
quires memorizing syntax while GUI-based systems require memo-
rizing keyboard shortcuts. P3 refected on the difculty of learning
shortcuts in an early version of Lime Aloud more than 20 years ago
and preferred SoundCells. In contrast, P2 had a strong preference
towards keyboard shortcuts in GUI-based systems because key-
board shortcuts can be chosen based on physical location and not
only their function in entering text. For example, P2 highlighted
how Lime Aloud maps many important functions to the top row of
the keyboard. While P2 generally preferred keyboard shortcuts, he
appreciated how chords in SoundCells are typed out note-by-note
instead of played simultaneously, which may be difcult for some.
5.1.3 Challenges Notating Rhythms. Nearly all participants shared
exercises or drafts of their fnal compositions with incomplete mea-
sures in which the combined durations of notes made a measure
too long or too short. The difculty notating rhythm became ap-
parent as the study progressed. With quarter notes in 4/4 time the
extent of rhythm covered Class 1, all measures consist of four notes
surrounded by two barlines, e.g. | a b c d |. However, when
writing rhythms, expressed as fractions or multipliers, one can’t
simply count to 4, but instead needs to track the sum of all written
durations, e.g. | a1/3 b2/3 c1/4 d1/4 e1/2 f3/2 g1/2 | is a
ASSETS ’22, October 23–26, 2022, Athens, Greece Payne et al.

Table 3: Characteristics of Final Compositions: This breakdown provides a rough indication of notation complexity and is not a
measure of participant creativity or efort.

Unique Expression3 Tempo, Meter,

ID Measures Notes/Chords/Rests Accidentals1 Comments
Durations2 Markings Key Changes
P1 13 173/5/0 44 8 30 5 5
P2 16 70/0/0 15 5 0 0 0
P3 16 230/39/9 8 5 8 0 0
P4 18 103/6/1 5 4 4 0 0
P5 8 24/0/1 0 5 0 0 1
P6 13 76/0/0 0 3 4 0 0
1. Accidentals are sharp, fat, or natural symbols that raise and lower notes, and can indicate melodic complexity.
2. Durations are the lengths of notes, e.g. quarter note, eighth note, and can indicate rhythmic complexity.
3. Expression markings are instructions to perform a note a specifc way, e.g. accented, short, long, etc.

valid measure because the fractions sum to 4. None of the partici-
pants had any difculty interpreting the syntax or adding numbers,
but both tasks combined appeared to be signifcantly challenging.
For example, P4 said, “people, everybody, not just me, are going
crazy trying to fgure out what the character is on the line, and
then they have to make sure that they’re still keeping track, that
they don’t lose their train of thought.” As evidence of her statement,
Both P2 and P5 told us they avoided complex rhythm in their fnal
compositions fnding the efort not worth it. In her fnal composi-
tion, P4 requested help after two failed attempts to make a measure
complete because, complicating things further, the grace notes she
notated needed to be excluded from the count. P6 emailed us an
early draft of her fnal composition with signifcant duration errors
and chose to restart rather than revise after we highlighted the
mistakes. P1 was the only participant who did not submit music
with incorrect rhythms. His fnal composition also used the most
complicated rhythms and meters (Table 3, Figure 2). As we discuss
further below, P1 attributes his accuracy to strategies that incorpo-
rate text-to-speech (§5.2.2) and braille output (§5.2.3), derived from
prior braille notation and programming experience.
5.1.4 Participant Recommendations for Improving Text Input. Our
participants suggested several alternative modes of input to address
these challenges. Those with more experience using music software
(P1 and P2) suggested “musical typing,” a feature common in GUI-
based music software, where the arrangement of white and black
keys is superimposed on the QWERTY keyboard. If SoundCells
supported a “musical typing mode” the ‘a’ key would typically
produce a letter ‘C’ or middle C, while the ‘w’ key would produce
‘_D’ or D-fat. When asked if separating keys from their designated
character could get confusing, P2 responded that he didn’t think
so, “I’m not thinking about the letter. I’m just thinking about the
layout of the keyboard.” However, participants who were new to
notation (P4 and P5), liked the current keyboard input, but wished
the interface could ofer more aid with note entry. For example, P5
said, “The idea of the piano as a keyboard was useless to me, but
perhaps having some sort of popups asking what you want and just
being able to click, that would make it easier to notate.” Separately,
to support fuent braille music readers, P3 requested the ability to
edit braille directly citing an experience, “I found my mistake and
I wanted to delete a wrong note while I was in the Show-Braille
mode. I was unable to do that. I had to go back to the letter.”
5.2 Outputs: When and How Participants Used
Diferent Modalities
Participants used audio and tactile outputs while editing to track
progress and identify mistakes. We defne “mistakes” in this context
loosely as moments that break musical rules, such as measures that
are incomplete (§5.1.3), or moments that don’t sound “right.” These
mistakes exclude syntax errors, which as indicated above (§5.1),
were not a common issue for participants. Participants reported that
real-time feedback and visual/braille score renderings were useful
aids beyond the raw syntax. However, orientiation in complex
scores and mistake correction are nontrivial tasks in SoundCells.
For example, P4 said, “the more complex it gets the longer it takes
for me to explore it and to see what I can do to make it presentable.”
Participants identifed afordances of diferent output modalities.
For example, P6 said, “Braille music, I feel like it’s good whenever
to read rhythms. Screen reader to check what bar you are on and
whether it is incomplete. Music playback for rhythm check, and
adding the notes you want to add.” The remainder of this section
highlights features of each output before moving on to participant
suggestions.
5.2.1 Music Output: Listening for Creative Expression. All partici-
pants used SoundCells’ music playback, and many highlighted the
instantaneous output, in which notes sound as they are typed, as
their favorite feature. For example, P3 said “Well, when you type
something and you hear right away what you typed, it really moti-
vates you and you hear it and it brings me in such a mood that, ‘Oh,
I hear it, I will now do more and more.’ ” Most participants played
their entire score back often to inform their notation process, while
P1 tended only to use it near the end of completing a challenge:
“Sometimes I use the audio if I want to know what it sounds like.” P4
and P6 are both musicians with perfect pitch, the ability to discern
the exact pitch and octave of a note just by hearing its frequency.
They found note playback especially helpful since it conveyed most
information depicted in text-to-speech. P4 was the only participant
who experimented with Loop Mode, in which a measure repeats
indefnitely as it is edited, stating, “I love loop mode. Something
Empowering Blind Musicians to Compose and Notate Music with SoundCells ASSETS ’22, October 23–26, 2022, Athens, Greece

to meditate on.” Though we asked, none of the screen reader users easy with small projects to just count measures and fgure out how
indicated difculty hearing note playback, a fnding from an earlier things are within because I can view multiple layers at one time.
study of SoundCells [50], or experimented with the option to de- It’s not like only using speech and having to scroll through and
lay notes until after a screen reader speaks the text. Despite their try to analyze each note each time, and try to query where you are
enthusiasm, participants identifed some limitations in the built-in at any given time with the question mark commands.” P1 further
synthesizer. P2 stated that it does not sound as good as other pro- refected on using a braille display to track indentations in Python
grams, while P1 and P3 found it less audible in lower frequencies. scripts where “speech was just too hard.”
P1 and P4 requested that we incorporate diferent instruments. While they seemed to aid navigation, braille displays did not
universally solve difculties. In Interview 2, we asked P3 to try and
5.2.2 Text-To-Speech Feedback. All participants, except P5, oper-
identify the location of an incomplete measure in an excerpt she
ated SoundCells with screen readers. P1 only used a screen reader
submitted. P3 opened the braille music view and tracked measures
once while writing a complex part of his fnal composition. Mul-
across multiple lines. She told us how difcult it was to remember
tiple participants noted how difcult it could be to discern long
the measure she was on, since she only had access to one line of
sections of ABC text when voiced out loud, character by character
music at a time, and each line could have a diferent number of
or meaningless word, through their screen readers. For example, P4
measures depending on their length. Ultimately, she could not fnd
expressed frustration, saying, “the screen reader reads letters, and
the subtle mistake until we suggested she open the diagnostics
it doesn’t read notes, and it doesn’t read music signs. It just reads
view. P2, who described himself as a slower braille reader compared
even the whole words that are like nonsense.” She further compared
to some of the other participants, had a braille display, but chose
the relationship between raw ABC text and it’s musical meaning to
not to use it in SoundCells or any notation software. Generally, he
letters that constitute words in English, saying “It would be helpful
fnds that lengthy braille scores can provide unnecessary detail,
if the screen reader would read actual names of notes and what kind
“Sometimes half the line is just the explanation. But where’s the
of note... In a regular word document, if you wrote ‘I went to the
note? I want to know the note... So, I heard some people just make
store and I bought apples, bananas, clementines,’ it would say that.
their own [custom Braille Music scores], just get to rid of those
So it would be helpful if it read ‘A eighth note - A dotted quarter -
letters on the music line.”
Barline.’ You know, that kind of thing.” P6 felt similar wishing that
her screen reader would convey meaning beyond raw syntax, but
5.2.4 Print (Visual) Music. P5, a participant who identifes as hav-
she did appreciate that VoiceOver recognizes fractions, “which is
ing low vision and who uses SoundCells with a screen magnifer,
very helpful because it helps with the math side of music.”
was the only participant to read the print rendering in conjunction
Tell-me commands (§3.2) were thought to be helpful under cer-
with music. She described: “Mostly looked at the text. I listened
tain circumstances. P4 used the diagnostics view and tell-me com-
to the score, and if I felt there was something wrong I went back
mands frequently. For example, when challenged to fnd an incom-
to the text. I looked at the score if there were errors too, see if a
plete measure during Interview 2, P4 opened the diagnostics view,
measure was underflled or overflled and see how it looked as a
iterated to the frst warning, jumped to the corresponding measure,
score.” At times, the print score would reveal mistakes visible to P5
extended one note’s duration, and fnally used a tell-me command
that were not immediately apparent in the text, “Obvious when I
to verify the fx. She thought that the tell-me command for describ-
looked at the score, not obvious looking at the ABC.” Because of
ing an entire measure was useful, but the command for describing
P5’s system-wide magnifcation preferences, she did not change
the current note was unnecessary since she understood the syntax:
the font size or score size within SoundCells. P5 felt that the layout
“I have no problem remembering the notes and stuf like that.” P1,
of the webpage, in which the print score is located below the text,
who did not use a screen reader for most of the challenge activities,
resulted in too much scrolling.
indicated using tell-me commands for a very complex section of
his composition saying, “that was actually a big example of where
5.2.5 Participant Recommendations for Improving Outputs. Partici-
I did need to use the question mark to fgure out where I was or
pants ofered many suggestions to improve outputs, and focused
if the measure was full or not because those were long measures.
on the modality they primarily used. Regarding music output, P2
Yeah, with a complex meter I think it’s very useful.” P6 used the
requested that a metronome or click track play along to convey
HTML text area which didn’t support tell-me commands (§3.1.3),
durations relative to the beat and time signatures, which can’t
but instead she often navigated below the editor where the same
always be determined from hearing the music alone. Regarding
information is displayed.
text-to-speech, P4 requested more detailed information when using
5.2.3 Braille Music Accessed via Braille Displays. P1, P3, and P6 the tell-me command to debug incomplete measures: “It should
found that braille displays were crucial to their notation processes. specify WHY it’s underflled, instead of making people go crazy
For example, P6 used her braille display to track her location in her trying to fgure it out.” Regarding braille, P3 requested a view of the
own compositions and in others’ work, “I’ve tried using it without braille music better suited to single-line braille displays, in which
a braille display and I can’t tell what notes are in there already. I measures are preceded by measure numbers and separated by new
can still use my ear, but if I was working on something very hard lines instead of spaces so that navigation keys iterate measure by
it’s going to be a lot more complicated.” P1, who as indicated above measure. Finally regarding visual output, P5 requested the ability
( §5.1.3) used a braille display to notate rhythms, understood the to re-position the visual score and zoom in on the selected measure
difculties faced by other participants to be exacerbated by their to prevent needing to repeatedly scroll up and down. An additional
screen readers: “When I’m using a braille display, it’s relatively suggestion made by P2 and P5 was for feedback to occur the instant
ASSETS ’22, October 23–26, 2022, Athens, Greece
they make a mistake, e.g. an error sound upon entering a note that
is too long.
5.3 Envisioned Usefulness of SoundCells
Five participants envisioned continuing to use SoundCells in the
future. They gave us suggestions for who else may use it and what
features would be necessary to support themselves and others.
P1 and P2 highlighted diferent benefts of the tool, including the
shareability of text and the capacity to export diferent fle types.
Each hoped to keep testing SoundCells and contributing to its
development. P2 thought SoundCells could be useful for notating
short music and exercises in a classroom environment, however for
large projects, he intends to keep using Lime Aloud. Similarly, P3
noted that most BVI musicians use Lime Aloud for notation, but
that she fnds SoundCells easier, and envisions using SoundCells to
transcribe piano music excerpts for her students. P6 not only aims
to continue using SoundCells, she shared it with a blind musician
in a diferent country directing them to the built-in tutorial. P4,
told us she is continuing to extend her fnal composition, and felt
she motivated after learning SoundCells to attend a webinar on
accessibility and code and the American Printing House (APH)
National Coding Symposium. Finally, P5, who enrolled in the course
to learn something new, told us that she enjoyed the experience,
but does not plan to continue using SoundCells or learning any
music software, saying “I doubt it, but who knows?”
While SoundCells worked well as a tool for novices to create
new compositions, more experienced participants found they could
accomplish less in SoundCells than in the notation software Lime
Aloud. These experiences indicate large potential for SoundCells if it
supports generating more advanced braille and print music notation,
or suggest that it could be better suited for novices who transition
to more advanced tools. Currently, SoundCells supports single-line,
single-instrument scores, while participants requested the ability
to notate for multiple voices and instruments. For example, P3, a
piano instructor, needs to be able to notate left-hand and right-hand
piano parts along with fngerings that convey to students where
to place their fngers. P1 and P6 identifed a few musical symbols
covered in Week 4 that appeared incorrectly or not at all in the
braille music score. P1, a braille music expert, sent us a braille fle
containing his fnal composition in which he manually adapted
the version generated by SoundCells. His fle contains stylistic
improvements including more measure numbers and the removal
of some unnecessary text. Additionally, P1’s composition could not
be processed by the music21 converter because a few very long
measures in 10/4 time contained so many notes and symbols they
could not ft on a 40-cell line of braille. (The embossed score in
Figure 2, splits each 10/4 measure into fve 2/4 measures.) P1 used
this as an example of the limits of automation in braille music
transcription: “According to the code book, the break should occur
at a logical position in the measure.” Since what is logical is often
subjective and context-dependent, in this case based on the cultural
tradition P1’s rhythm draws upon, “that’s where [his] editing comes
in handy.”
Payne et al.
6 DISCUSSION
Participants overwhelmingly enjoyed composing music in Sound-
Cells as they incorporated their own workfows and accessibility
preferences. In this section, we suggest design recommendations,
refect on how the structure of the syntax within SoundCells hin-
dered usability, and report how our methodology facilitated creative
expression.
6.1 Input: Design Considerations for Text Entry
and Navigation Inspired by Visual Music
Software
SoundCells was designed to be an alternative to GUI-based music
software immediately accessible to screen reader and braille display
users. However, participants expressed diferent comfort levels with
text (§5.1.2) while the unstructured editing environment introduced
new challenges, including difculties understanding and remember-
ing notation (§5.1.1) and typing measures with the correct number
of notes (§5.1.3). Most participants suggested improved outputs
that aid in navigation and identifying mistakes, but new features
for structured editing could help reduce challenges.
Blind producers who work in digital audio workstations (DAWs)
commonly access musical data (MIDI) with an event view, or long
list of MIDI events [52]. SoundCells’ diagnostics view could be
extended to support a wider range of information beyond errors
and warnings. Going beyond what DAWs support, SoundCells could
provide diferent flters, e.g. notes, measures, errors, etc., verbosity
options for how much detail to provide about the note, and the
option to play back music and/or text-to-speech. Additional context-
specifc commands drawn from keyboard shortcuts present in other
tools, like doubling/halving a duration or transposing a note, might
support users’ workfows for editing music. An understanding of
how users work between raw text and flterable data could have
implications for other semi-structured text environments.
To aid in entering rhythms, participants suggested an error
sound/message (§5.2.5), but GUI-based tools, in contrast with text
editors, prevent incorrect measures from occurring in the frst place.
For example, Lime Aloud automatically flls an incomplete measure
with rests and ignores notes that would make a measure too long.
SoundCells could auto-complete underflled measures with rests
and prevent the entry of text that would overfll a measure.
New additions to SoundCells that aid text editing, such as an
event view or automatic error prevention, could have implications
for other coding environments. A signifcant trend in educational
coding environments like Scratch [60] has been to use a visual
canvas with draggable code blocks, but blocks-based environments
introduce many accessibility barriers [39, 45]. Structured text in-
teractions inspired by other tools for creative expression promote
usability and accessibility within creative coding environments.
6.2 Designing Interactions Across Multiple
Output Modalities
The incorporation of cross-modal music representations in Sound-
Cells enabled accessibility and fexible use while introducing chal-
lenges. Specifcally, we observed difculties navigating and inter-
preting screen reader output.
Empowering Blind Musicians to Compose and Notate Music with SoundCells
6.2.1 Adapting Braille and Print Representations for Improved Navi-
gation. SoundCells displays entire braille and print scores similar
to how Overleaf displays a full document next to its LATEX source.
However, this adherence to replicating physical format impeded
access and navigation in browsers. Instead, altered or reduced views
of music may address difculties. For example, P3 suggested a dig-
ital braille score containing one measure per line prefaced by its
measure number (§5.2.5). This would beneft braille display users
since they only read one line at a time, and buttons for iterating
line by line would apply to iterating measure by measure. In the
visual output, a reduced score containing only the current measure
or local group of measures could be displayed. This miniature score
would always include important context, like measure number and
key signature, that can be obscured by enlargement.
Furthermore, designers should consider not only how users nav-
igate within outputs but also between outputs. We observed how
mistake detection could lead to complex workfows - P3 described
hearing a mistake, then navigating multiple lines of braille music
on a single line braille display to fnd it, and then fnally returning
to the text to correct it (§5.2.3). Ultimately, systems designed to
support multiple modalities, must consider the afordances of each
and ensure users are not constantly in a state of reorientation.
6.2.2 Context-Aware Screen Reader Output. Screen reader output
presented additional challenges. Participants highlighted the con-
centration and extra efort required to understand streams of raw
text spoken through their screen readers and interpret the state
of their music accordingly (§5.2.2). Accessibility researchers devel-
oping systems for novice BVI coders [69, 70] have encountered
a similar challenge in code debugging workfows and developed
new screen reader paradigms. For example, the “talking debugger”
inside SodBeans can describe behaviors during program execution,
e.g. stating the underlying value of a variable rather than its name
[67]. SoundCells similarly, could interpret syntax and account for
musical context after an event is typed. For example, ‘ffth octave
B fat quarter note’ could be read when _b is typed in C major or b
is typed in F major. Such a feature may require implementing cus-
tom screen reader behaviors into SoundCells. However, SoundCells,
unlike SodBeans, is developed for browsers where screen readers
typically relay the content the user is focused on, and it isn’t clear
how web developers might access system accessibility API’s.
6.2.3 Customization for Print and Embossed Music. Customization
options for physical scores (printed and embossed) could provide
signifcant performance and educational value. The musician that
we partnered with for the fnal performance requested music with
three systems per 11x17 page (Figure 2), but in one case requested
we shrink the music to ft on four pages (§4.1.2). While the Sound-
Cells’ score size slider made this possible, we still needed to use
“guess and check” to resize the music and then reference the result-
ing layout of the PDF. Instead, end-users should be able to specify
systems-per-page and fne tune the layout based on an estimated
page count before needing to check the PDF. Prior music education
research has shown that blind musicians do not engage with music
the same way that sighted musicians do, and instead use notation as
a memorization tool often along with audio recordings [22, 32, 49].
Variations of a score may be desirable across diferent stages of
learning, memorization, and rehearsal. Furthermore, being able to
ASSETS ’22, October 23–26, 2022, Athens, Greece
generate multiple versions may have educational value in settings
where braille music is being taught. For example, a teacher may
want to remove expression markings before a student has learned
them, or, as P3 suggested, add optional piano fngerings at the start
of a learning process.
6.3 Ocularcentrism in ABC Syntax
While most participants did not express major concerns understand-
ing ABC text within SoundCells, many found it to be unintuitive
and hard to remember. Upon refection, we think this is a conse-
quence of ABC text’s fundamental design. P2 and P3 questioned
why seemingly random characters – commas/apostrophes, under-
scores/carets – are used to lower and raise notes. These characters
have no inherent musical meaning. Instead, their spatial position
visually signifes their operations. For example, characters placed
low relative to a pitch letter communicate lowering operations
(__F,), while higher characters communicate raising (^a''). This
visual cue benefts sighted readers but is lost in braille and text-
to-speech output. Not only can syntax convey meaning diferently
across visual, auditory, and tactile modalities, it presents usability
implications when inputted on a QWERTY keyboard. P2 described
how in other GUI-based music software, the most common key
commands for editing notes lie in one easy-to-reach part of the
keyboard, e.g. the top row in Lime Aloud [14]. While ,/' and _/^
characters have parallel efects in ABC, they are located across the
keyboard, and some require holding shift to input. The scattered
layout of common symbols may contribute to the slow text entry
P2 and others perceived while notating music in SoundCells (§5.1.2).
While the fnal compositions were relatively short, 24–230 notes
(Table 3), many classical compositions contain thousands of notes.
The ability to type quickly and efciently is crucial to advanced
notation workfows. ABC text was chosen for SoundCells in part
due to its 20-year existence and widespread use [73]. An alternative
notation would need to covert to ABC to make it universally share-
able and would not beneft from existing online resources. P1 and
P2 proposed alternative entry methods such as connecting a MIDI
keyboard, or mapping notes on the QWERTY keyboard to resemble
a piano (§5.1.2). While aiding speed, new entry methods would not
aid legibility. Looking ahead, an important consideration for domain
specifc languages, is that visual cues alone should not defne syntax
behavior, and that keyboard position and other intuitive properties
of characters should be considered. For example, P2 suggested that
fats, sharps, and natural signs be added with -/+/= respectfully.
Not only are these keys adjacent on the QWERTY keyboard, they
are meaningfully connected with their musical operations.
6.4 High Engagement Through Creative
Expression, Clear Motivation, and Remote
Study
Despite the potential for many challenges, this study yielded high
engagement across diverse participants. We refect on three features
of our experimental design, 1) giving participants opportunities for
self-expression, 2) positioning technology interactions towards a
clear goal, and 3) conducting research remotely. Each of these fea-
tures, we think, contributed to the success of this experiment. First,
our study revolved around open-ended music expression, while
ASSETS ’22, October 23–26, 2022, Athens, Greece
weekly creative prompts provided guidance and constraints. All
participants, busy adults and one high school student, attended
every class and devoted hours of efort working independently.
This was entirely voluntary, as participants were only compensated
$75 for research activities consisting of three interviews. We view
the high engagement as validation that music making is something
participants enjoy, but also that our bare-bones curriculum, largely
documentation with weekly challenges (Table 1), gave them fexi-
bility to learn and work on their own terms. This design expanded
the amount of material we could cover during interviews and re-
duced pressure during synchronous interactions for participants
to “be creative” under observation. Instead, they discussed ideas,
shared complete and incomplete projects, and demonstrated issues
they had run into. Second, the fnal concert motivated participants
to test the boundaries of new technology and notate pieces that
refected their aesthetics. By recruiting a professional musician,
the performance gave the course credibility and weight, but was
not overwhelmingly high-stakes since it was online and internal.
Finally, we chose to run this experiment online despite the fact
that our community partner FMDG School had returned to con-
ducting many activities in-person. Being online reduced barriers,
and enabled participation from individuals who would not have
joined otherwise. P4 and P5, who had never notated original music
or used music software before, participated just because the topic
seemed interesting. Being remote meant we had to ensure partici-
pants could use SoundCells independently. This included making
course materials screen reader accessible and recording zoom ses-
sions for later listening. Ultimately, we felt remote research worked
well in facilitating longitudinal, high-commitment activities with
synchronous and asynchronous interactions.
7 LIMITATIONS AND FUTURE WORK
While this study was set within a learning environment, we did
not evaluate our curriculum or student learning. More research is
needed to evaluate educational gains. SoundCells currently sup-
ports a limited range of what ABC text and braille music can rep-
resent. For example, it supports music for solo string or wind in-
struments but not music for orchestra or choir. We saw signifcant
participant interest in writing more complex compositions, espe-
cially for notating piano music and more expression markings.
To support advanced notation, music21, the open-source Python
library SoundCells uses for ABC parsing and braille conversion
[13], would need further development. In the same way that this
study extended prior SoundCells research, we would like to further
test and evaluate the capabilities of text and multimodal output in
SoundCells, following future system improvements, by commis-
sioning BVI composers to notate even more substantial works with
more time.
8 CONCLUSION
In this work, we shared fndings from a six-week remote study in
which six BVI musicians used SoundCells to notate music leading
to a fnal live performance of their work. SoundCells proved to be
accessible to participants who possessed a range of vision ability,
music experience, AT preference, and composition style. However
participants identifed difculties working within the text editor,
Payne et al.
like typing unintuitive syntax and mentally calculating complex
rhythms, while navigating and identifying mistakes across modali-
ties could prove demanding. We discussed design recommendations
for improving text input and multimodal output in SoundCells, iden-
tifed visual signifers as a cause of difculty learning ABC syntax,
and refected on how our methods led to high engagement across
all participants.
ACKNOWLEDGMENTS
We thank the FMDG School faculty and students for their support
and insight into the design of SoundCells. We also thank members
of NYU Ability Project and Vertically Integrated Projects, especially
Katrina Lee.
REFERENCES
[1] Samuel Aaron. 2020. Sonic Pi v3.2 Released! https://www.patreon.com/posts/
sonic-pi-v3-2-34414152
[2] Samuel Aaron, Alan F Blackwell, and Pamela Burnard. 2016. The development
of Sonic Pi and its use in educational partnerships: Co-creating pedagogies for
learning computer programming. Journal of Music, Technology & Education 9, 1
(2016), 75–94.
[3] Joseph Michael Abramo and Amy Elizabeth Pierce. 2013. An ethnographic
case study of music learning at a school for the blind. Bulletin of the Council
for Research in Music Education 195, 195 (2013), 9–24. https://doi.org/10.5406/
bulcouresmusedu.195.0009
[4] Fabiha Ahmed, Dennis Kuzminer, Michael Zachor, Lisa Ye, Rachel Josepho,
William Christopher Payne, and Amy Hurst. 2021. Sound Cells: Rendering
Visual and Braille Music in the Browser. In The 23rd International ACM SIGAC-
CESS Conference on Computers and Accessibility (Virtual Event, USA) (ASSETS ’21).
Association for Computing Machinery, New York, NY, USA, Article 89, 4 pages.
https://doi.org/10.1145/3441852.3476555
[5] Ameer Armaly, Paige Rodeghero, and Collin McMillan. 2018. AudioHighlight:
Code Skimming for Blind Programmers. In 2018 IEEE International Conference
on Software Maintenance and Evolution (ICSME) (Madrid, Spain). IEEE, 206–216.
https://doi.org/10.1109/ICSME.2018.00030
[6] Safa Arooj, Shaban Zulfqar, Muhammad Qasim Hunain, Suleman Shahid, and
Asim Karim. 2020. Web-ALAP: A Web-based LaTeX Editor for Blind Individuals. In
The 22nd International ACM SIGACCESS Conference on Computers and Accessibility.
1–6.
[7] Avid. 2021. Music Notation Software - Sibelius. Retrieved May 1, 2021 from
https://www.avid.com/sibelius
[8] Catherine M. Baker, Lauren R. Milne, and Richard E. Ladner. 2015. StructJumper:
A Tool to Help Blind Programmers Navigate and Understand the Structure of Code.
Association for Computing Machinery, New York, NY, USA, 3043–3052. https:
//doi.org/10.1145/2702123.2702589
[9] Cynthia L. Bennett, Jane E, Martez E. Mott, Edward Cutrell, and Meredith Ringel
Morris. 2018. How Teens with Visual Impairments Take, Edit, and Share Photos
on Social Media. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing
Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173650
[10] Cynthia L. Bennett, Abigale Stangl, Alexa F. Siu, and Joshua A. Miele. 2019.
Making Nonvisually: Lessons from the Field. In The 21st International ACM
SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA)
(ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 279–285.
https://doi.org/10.1145/3308561.3355619
[11] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative Research in Psychology 3, 2 (1 2006), 77–101. https://doi.org/10.1191/
1478088706qp063oa
[12] Chambers, John. 2021. JC’s ABC Tune Finder. Retrieved May 1, 2021 from
http://trillian.mit.edu/~jc/cgi/abc/tunefnd
[13] Cuthbert, Michael Scott. 2021. music21 Braille Translate. Retrieved May 1, 2021
from https://web.mit.edu/music21/doc/moduleReference/moduleBrailleTranslate.
html
[14] Dancing Dots. 2020. Dancing Dots: Accessible Music Technology for Blind and
Low Vision Performers since 1992. https://www.dancingdots.com/
[15] Maitraye Das, Katya Borgos-Rodriguez, and Anne Marie Piper. 2020. Weaving
by Touch: A Case Analysis of Accessible Making. Association for Computing
Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3313831.3376477
[16] Maitraye Das, Anne Marie Piper, and Darren Gergle. 2022. Design and Evaluation
of Accessible Collaborative Writing Techniques for People with Vision Impair-
ments. ACM Trans. Comput.-Hum. Interact. 29, 2, Article 9 (jan 2022), 42 pages.
https://doi.org/10.1145/3480169
Empowering Blind Musicians to Compose and Notate Music with SoundCells
[17] Duxbury Systems. 2021. Duxbury DBT: Braille Translation Software. Retrieved
May 1, 2021 from https://www.duxburysystems.com
[18] Associazione Giuseppe Paccini ETS. 2021. Braille Music Editor. Retrieved May
1, 2021 from https://braillemusiceditor.com/
[19] Jason Freeman, Brian Magerko, Tom McKlin, Mike Reilly, Justin Permar, Cameron
Summers, and Eric Fruchter. 2014. Engaging underrepresented groups in high
school introductory computing through computational remixing with EarSketch.
In Proceedings of the 45th ACM technical symposium on Computer science education.
85–90.
[20] Emma Frid. 2019. Accessible digital musical instruments—A review of musical
interfaces in inclusive music practice. Multimodal Technologies and Interaction 3,
3 (2019). https://doi.org/10.3390/mti3030057
[21] Emma Frid and Alon Ilsar. 2021. Reimagining (Accessible) Digital Musical In-
struments: A Survey on Electronic Music-Making Tools. In Proceedings of the
International Conference on New Interfaces for Musical Expression. Shanghai, China,
Article 28, 20 pages. https://doi.org/10.21428/92fbeb44.c37a2370
[22] Danni Gilbert. 2018. “It’s Just the Way I Learn!”: Inclusion from the Perspective of
a Student with Visual Impairment. Music Educators Journal 105, 1 (2018), 21–27.
https://doi.org/10.1177/0027432118777790
[23] GNU Project. 2021. LilyPond. Retrieved May 1, 2021 from https://lilypond.org
[24] David Goldstein. 2000. Music pedagogy for the blind. International Journal of Mu-
sic Education 35, 1 (5 2000), 35–39. https://doi.org/10.1177/025576140003500112
[25] D. Goto, T. Gotoh, R. Minamikawa-Tachino, and N. Tamura. 2007. A Transcription
System from MusicXML Format to Braille Music Notation. Eurasip Journal on
Advances in Signal Processing 2007, 1 (Jan. 2007), 152. https://doi.org/10.1155/
2007/42498
[26] Toshiyuki Gotoh, Reiko Minamikawa-Tachino, and Naoyoshi Tamura. 2008. A
Web-Based Braille Translation for Digital Music Scores. In Proceedings of the
10th International ACM SIGACCESS Conference on Computers and Accessibility
(Halifax, Nova Scotia, Canada) (Assets ’08). Association for Computing Machinery,
New York, NY, USA, 259–260. https://doi.org/10.1145/1414471.1414527
[27] Lucy Green and David Baker. 2017. Digital Music Technologies: The Changing
Landscape. In Insights in Sound: Visually Impaired Musicians’ Lives and Learning.
Routledge, Chapter 9, 158–173. https://doi.org/10.4324/9781315266060
[28] Thomas Haenselmann, Hendrik Lemelson, Kerstin Adam, and Wolfgang Efels-
berg. 2009. A tangible MIDI sequencer for visually impaired people. In Proceedings
of the seventeen ACM international conference on Multimedia - MM ’09. ACM Press,
New York, New York, USA, 993. https://doi.org/10.1145/1631272.1631485
[29] Marijn Haverbeke. 2018. Code editor screen reader accessiblity survey. Retrieved
May 1, 2021 from https://discuss.codemirror.net/t/code-editor-screen-reader-
accessiblity-survey/1790
[30] Marijn Haverbeke. 2021. Screen reader response for backspace is faky. Retrieved
Dec 1, 2021 from https://github.com/codemirror/codemirror.next/issues/563
[31] Marijn Haverbeke. 2022. Screen Reader Demo. Retrieved May 1, 2021 from
https://codemirror.net/6/screenreaderdemo/
[32] Virginia A. Jacko, Jin Ho Choi, Audrey Carballo, Brian Charlson, and J. Elton
Moore. 2015. A New Synthesis of Sound and Tactile Music Code Instruction in a
Pilot Online Braille Music Curriculum. Journal of Visual Impairment & Blindness
109, 2 (2015), 153–157. https://doi.org/10.1177/0145482X1510900212
[33] Nadine Jessel. 2015. Access to musical information for Blind People. In Interna-
tional Conference on Technologies for Music Notation and Representation. Institut
de Recherche en Musicologie (IReMus), 232–237.
[34] Aaron Karp and Bryan Pardo. 2017. HaptEQ. In Proceedings of the 12th Interna-
tional Audio Mostly Conference on Augmented and Participatory Sound and Music
Experiences. ACM, New York, NY, USA, 1–4. https://doi.org/10.1145/3123514.
3123531
[35] Claire Kearney-Volpe and Amy Hurst. 2021. Accessible Web Development: Op-
portunities to Improve the Education and Practice of Web Development with a
Screen Reader. 14, 2, Article 8 (2021), 32 pages. https://doi.org/10.1145/3458024
[36] Lang, Mario. 2009. FreeDots: MusicXML to Braille Music translation. Retrieved
May 1, 2021 from https://delysid.org/freedots.html
[37] Matthias Leopold. 2006. HODDER–a fully automatic braille note production sys-
tem. In International Conference on Computers for Handicapped Persons. Springer,
6–11.
[38] Alex Michael Lucas, Miguel Ortiz, and Dr. Franziska Schroeder. 2019. Be-
spoke Design for Inclusive Music: The Challenges of Evaluation. Proceedings
of the International Conference on New Interfaces for Musical Expression (2019),
105–109. https://novationmusic.com/keys/launchkey%0Ahttp://www.nime.org/
proceedings/2019/nime2019_021.pdf
[39] Stephanie Ludi and Mary Spencer. 2017. Design Considerations to Increase
Block-based Language Accessibility for Blind Programmers Via Blockly. 3, 1
(July 2017), 119–124. https://doi.org/10.18293/vlss2017-013
[40] Bill Manaris, Blake Stevens, and Andrew R. Brown. 2016. JythonMusic: An
environment for teaching algorithmic music composition, dynamic coding and
musical performativity. 9, 1 (May 2016), 33–56. https://doi.org/10.1386/jmte.9.1.
33_1
[41] Christina Matawa. 2009. Exploring the musical interests and abilities of blind and
partially sighted children and young people with Retinopathy of Prematurity.
ASSETS ’22, October 23–26, 2022, Athens, Greece
British Journal of Visual Impairment 27, 3 (9 2009), 252–262. https://doi.org/10.
1177/0264619609106364
[42] Sean Mealin and Emerson Murphy-Hill. 2012. An exploratory study of blind soft-
ware developers. In 2012 IEEE Symposium on Visual Languages and Human-Centric
Computing (VL/HCC). 71–74. https://doi.org/10.1109/VLHCC.2012.6344485
[43] Cecily Morrison, Nicolas Villar, Anja Thieme, Zahra Ashktorab, Eloise Taysom,
Oscar Salandin, Daniel Cletheroe, Greg Saul, Alan F Blackwell, Darren Edge,
Martin Grayson, and Haiyan Zhang. 2018. Torino: A Tangible Programming
Language Inclusive of Children with Visual Disabilities. 35, 3 (Oct. 2018), 191–239.
https://doi.org/10.1080/07370024.2018.1512413
[44] Frederick W Moss, Jr. 2009. Quality of Experience in Mainstreaming and Full
Inclusion of Blind and Visually Impaired High School Instrumental Music Stu-
dents.
[45] Aboubakar Mountapmbeme and Stephanie Ludi. 2021. How Teachers of the
Visually Impaired Compensate with the Absence of Accessible Block-Based
Languages. In The 23rd International ACM SIGACCESS Conference on Computers
and Accessibility (Virtual Event, USA) (ASSETS ’21). Association for Computing
Machinery, New York, NY, USA, Article 4, 10 pages. https://doi.org/10.1145/
3441852.3471221
[46] National Library Service for the Blind and Print Disabled. 2021. Music Instruc-
tional Materials and Scores. Retrieved April 1, 2022 from https://www.loc.gov/
nls/about/services/music-instructional-materials-scores/
[47] Shotaro Omori and Ikuko Eguchi Yairi. 2013. Collaborative music application
for visually impaired people with tangible objects on table. In Proceedings of the
15th International ACM SIGACCESS Conference on Computers and Accessibility
- ASSETS ’13. ACM Press, New York, New York, USA, 1–2. https://doi.org/10.
1145/2513383.2513403
[48] OpenScore. 2018. OpenScore: One Year On. Retrieved May 1, 2021 from
https://openscore.cc/blog/2018/8/20/openscore-one-year-on
[49] Hyu-Yong Park and Mi-Jung Kim. 2014. Afordance of Braille music as a media-
tional means: signifcance and limitations. British Journal of Music Education 31,
2 (2014), 137–155. https://doi.org/10.1017/S0265051714000138
[50] William Payne, Fabiha Ahmed, Michael Gardell, R. Luke DuBois, and Amy Hurst.
2022. SoundCells: Designing a Browser-Based Music Technology for Braille
and Print Notation. In 19th Web for All Conference (Lyon, France) (W4A ’22).
Association for Computing Machinery, New York, NY, USA, Article 11, 12 pages.
https://doi.org/10.1145/3493612.3520464
[51] William Payne, Alex Xu, Amy Hurst, and S. Alex Ruthmann. 2019. Non-visual
beats: Redesigning the Groove Pizza. In ASSETS 2019 - 21st International ACM
SIGACCESS Conference on Computers and Accessibility. Association for Computing
Machinery, Inc, 651–654. https://doi.org/10.1145/3308561.3354590
[52] William Christopher Payne, Alex Yixuan Xu, Fabiha Ahmed, Lisa Ye, and Amy
Hurst. 2020. How Blind and Visually Impaired Composers, Producers, and
Songwriters Leverage and Adapt Music Technology. In The 22nd International
ACM SIGACCESS Conference on Computers and Accessibility. 1–12.
[53] Petrucci Music Library. 2021. IMSLP: Sharing the world’s public domain music.
Retrieved May 1, 2021 from https://imslp.org/
[54] Venkatesh Potluri, Priyan Vaithilingam, Suresh Iyengar, Y. Vidya, Manohar
Swaminathan, and Gopal Srinivasa. 2018. CodeTalk: Improving Programming
Environment Accessibility for Visually Impaired Developers. In Proceedings of
the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC,
Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA,
1–11. https://doi.org/10.1145/3173574.3174192
[55] Ability Project. 2021. SoundCells. Retrieved Dec 1, 2021 from https://soundcells.
herokuapp.com/
[56] Ability Project. 2021. SoundCells GitHub Repository. Retrieved Dec 1, 2021
from https://github.com/Huriphoonado/soundcells
[57] Ability Project. 2022. SoundCells Class at Community Music School for the Blind.
Retrieved Mar 1, 2022 from https://huriphoonado.github.io/sc-workshop
[58] Ability Project. 2022. SoundCells Class at Community Music School for the Blind:
Final Compositions. Retrieved Mar 1, 2022 from https://huriphoonado.github.
io/sc-workshop/class/update/2022/04/03/week-6.html
[59] Lauren Race, Joshua A. Miele, Chancey Fleet, Tom Igoe, and Amy Hurst. 2020.
Putting Tools in Hands: Designing Curriculum for a Nonvisual Soldering Work-
shop. In The 22nd International ACM SIGACCESS Conference on Computers and
Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing
Machinery, New York, NY, USA, Article 78, 4 pages. https://doi.org/10.1145/
3373625.3418011
[60] Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn
Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian
Silverman, et al. 2009. Scratch: programming for all. Commun. ACM 52, 11 (2009),
60–67.
[61] Toby W Rush. 2019. Braille Music Notator. Retrieved May 1, 2021 from
https://tobyrush.com/braillemusic/notator/
[62] Marc Sabatella. 2013. ABC For Blind Musicians. Retrieved May 1, 2021 from https:
//accessiblemusicnotation.wordpress.com/2013/08/21/abc-for-blind-musicians/
[63] Abir Saha and Anne Marie Piper. 2020. Understanding Audio Production Practices
of People with Vision Impairments. In The 22nd International ACM SIGACCESS
ASSETS ’22, October 23–26, 2022, Athens, Greece
Conference on Computers and Accessibility. 1–13.
[64] Alexa F. Siu, Son Kim, Joshua A. Miele, and Sean Follmer. 2019. ShapeCAD:
An Accessible 3D Modelling Workfow for the Blind and Visually-Impaired Via
2.5D Shape Displays. In The 21st International ACM SIGACCESS Conference on
Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for
Computing Machinery, New York, NY, USA, 342–354. https://doi.org/10.1145/
3308561.3353782
[65] Lawrence R. Smith, Karin Auckenthaler, Gilbert Busch, Karen Gearreald, Dan
Geminder, Beverly McKenney, Harvey Miller, and Tom Ridgeway. 2015. Braille
Music Code. American Printing House for the Blind, Louisville, Kentucky.
[66] Andreas Stefk, Roger Alexander, Robert Patterson, and Jonathan Brown. 2007.
WAD: A Feasibility study using the Wicked Audio Debugger. In 15th IEEE In-
ternational Conference on Program Comprehension (ICPC ’07). 69–80. https:
//doi.org/10.1109/ICPC.2007.42
[67] Andreas Stefk, Andrew Haywood, Shahzada Mansoor, Brock Dunda, and Daniel
Garcia. 2009. SODBeans. In 2009 IEEE 17th International Conference on Program
Comprehension. 293–294. https://doi.org/10.1109/ICPC.2009.5090064
[68] Andreas Stefk, Christopher Hundhausen, and Robert Patterson. 2011. An empir-
ical investigation into the design of auditory cues to enhance computer program
comprehension. International Journal of Human-Computer Studies 69, 12 (2011),
820–838. https://doi.org/10.1016/j.ijhcs.2011.07.002
Payne et al.
[69] Andreas Stefk, Richard E. Ladner, William Allee, and Sean Mealin. 2019. Com-
puter Science Principles for Teachers of Blind and Visually Impaired Students. In
Proceedings of the 50th ACM Technical Symposium on Computer Science Education
(Minneapolis, MN, USA) (SIGCSE ’19). Association for Computing Machinery,
New York, NY, USA, 766–772. https://doi.org/10.1145/3287324.3287453
[70] Andreas M. Stefk, Christopher Hundhausen, and Derrick Smith. 2011. On the
Design of an Educational Infrastructure for the Blind and Visually Impaired
in Computer Science. In Proceedings of the 42nd ACM Technical Symposium
on Computer Science Education (Dallas, TX, USA) (SIGCSE ’11). Association for
Computing Machinery, New York, NY, USA, 571–576. https://doi.org/10.1145/
1953163.1953323
[71] Atau Tanaka and Adam Parkinson. 2016. Haptic Wave. (2016), 2150–2161.
https://doi.org/10.1145/2858036.2858304
[72] The DAISY Consortium. 2021. Music Braille. Retrieved July 1, 2021 from
https://daisy.org/activities/projects/music-braille/
[73] Chris Walshaw. 2021. abc notation. Retrieved May 1, 2021 from https://
abcnotation.com/
[74] Ge Wang. 2012. A History of Programming and Music. Cambridge University
Press, 55–71.
[75] World Wide Web Consortium. 2021. Web Content Accessibility Guidelines
(WCAG) Overview. Retrieved May 1, 2021 from https://www.w3.org/WAI/
standards-guidelines/wcag/
 Designing Gestures for Digital Musical Instruments: Gesture
Elicitation Study with Deaf and Hard of Hearing People
Ryo Iijima Akihisa Shitara Yoichi Ochiai
ryoiijima@digitalnature.slis.tsukuba.ac.jp theta- wizard@digitalnature.slis.tsukuba.ac.jp
University of Tsukuba akihisa@digitalnature.slis.tsukuba.ac.jp University of Tsukuba
Tsukuba, Japan University of Tsukuba Tsukuba, Japan
Tsukuba, Japan

castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65) pellet drum (AR=0.29)

tam-tam (AR=0.05) jaw harp (AR=0.05) recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40)

Figure 1: Most frequent gesture proposals for each referent. (AR = agreement rate)
ABSTRACT
When playing musical instruments, deaf and hard-of-hearing (DHH)
people typically sense their music from the vibrations transmit-
ted by the instruments or the movements of their bodies while
performing. Sensory substitution devices now exist that convert
sounds into light and vibrations to support DHH people’s musi-
cal activities. However, these devices require specialized hardware,
and the marketing profles assume that standard musical instru-
ments are available. Hence, a signifcant gap remains between DHH
people and their musical performance enjoyment. To address this
issue, this study identifes end users’ preferred gestures when us-
ing smartphones to emulate the musical experience based on the
instrument selected. This gesture elicitation study applies 10 instru-
ment types. Herein, we present the results and a new taxonomy of
musical instrument gestures. The fndings will support the design
of gesture-based instrument interfaces to enable DHH people to
more directly enjoy their musical performances.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544828
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility.
KEYWORDS
Deaf, hard of hearing, music, mobile, gesture elicitation study
ACM Reference Format:
Ryo Iijima, Akihisa Shitara, and Yoichi Ochiai. 2022. Designing Gestures for
Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard
of Hearing People. In The 24th International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece.
ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3517428.3544828
1 INTRODUCTION
Musical activities have been reported to provide a wide range of
cognitive, social, emotional, and physical benefts [64]. Similarly,
smartphones already provide users with opportunities to incorpo-
rate musical activities into their daily lives. Owing to the release
of digital audio applications for mobile devices and desktop music,
anyone can experience the process of creating music and virtually
playing an instrument. Although deaf and hard-of-hearing (DHH)
people can enjoy music in various ways [24], they typically fnd
it difcult to enjoy digital musical instrument applications. When
DHH people play real instruments, they typically feel the sound
and rhythm from the vibrations transmitted from the instrument
and their own body movements. However, this is not possible with
smartphone digital musical instrument applications because they
ASSETS ’22, October 23–26, 2022, Athens, Greece
cannot currently provide the same depth of vibrations as that pro-
vided by a real instrument. Furthermore, most applications are
controlled using a touch screen, rendering it difcult to employ the
body in ways benefcial to music appreciation. Previous research
has helped DHH people play musical instruments using sensory
substitution devices that convert sound into light or vibrations [52].
However, such approaches assume that a traditional instrument is
available, and special hardware is required. As a result, a gap re-
mains between DHH people and their daily musical satisfaction. To
fll this gap, this study focuses on augmenting smartphone digital
musical instrument applications accordingly.
In our previous work [26], we developed a smartphone-based mu-
sical instrument prototype for DHH users based on motion gestures,
enabling users to feel the music through smartphone vibrations.
This concept has been well received by DHH users. However, the
design space remains ambiguous, and there is a lack of taxonomical
and control parameters. This paper describes the empirical results
of a gesture elicitation study (GES) of the physical expressions dis-
played when using a smartphone as a musical instrument. In our
experiment, given the opportunity to obtain vibrational feedback
through the smartphone from gestures, participants were asked to
identify the most suitable motions to accompany the music. This
work provides a quantitative and qualitative characterization of
these gestures and a user-defned gesture set and taxonomy. Fur-
thermore, design implications are summarized.
2 RELATED WORK
2.1 Music and DHH People
DHH people can and do enjoy music [16, 49]. A detailed study
on how DHH people relate to music was conducted by Darrow
et al. [6]. Glenny [23] and Roebuck [55] are DHH people who
work as professional musicians. Beethoven, who is famous for his
many masterpieces, sufered from hearing loss as well [11]. These
examples suggest that people who are born deaf or have lost their
hearing during life can still enjoy music. Few organizations are
working toward making music more enjoyable and accessible to
DHH people. “Music and the Deaf” 1 encourages DHH people to
participate in musical activities. A musical activity called the “White
Hand Chorus” 2 values the participation of children who are hearing
and speech impaired or autistic. These projects also contribute to
bringing DHH people closer to music. This study aimed to create
digital musical instruments that can be enjoyed by DHH people
using only existing devices. Therefore, it is expected to facilitate
the daily musical enjoyment of DHH individuals.
2.2 Accessible Digital Musical Instruments
(ADMIs)
To make music-making available to a more diverse population,
ADMIs are gaining interest in the feld of computerized music
applications [27]. Previous research has resulted in systems that
allow people with neurodevelopmental [38, 47, 54] and mental
disorders [48], as well as motor [2, 3] and visual disabilities [32,
46, 50, 63, 78] to enjoy music. However, ADMI research for DHH
1 https://matd.org.uk/
2 https://www.elsistemajapan.org/whitehands
Iijima, et al.
people is scarce; the review by Ilsar et al. [27] reported that only
four ADMIs for DHH people have been developed as of 2020.
Music and speech are fundamentally diferent in their wave
forms; thus, it is difcult to enjoy music with hearing aids designed
for speech [5]. Additionally, cochlear implants make it easier to
listen to human conversations; however, they are not yet suitable
for music [41]. Researchers have worked to improve such devices
by designing accompanying music training programs and games
[18, 37, 81]. However, melody and timbre identifcation remains
difcult [9, 19]. Several approaches allow DHH users to perceive
music without relying on their ears via sensory substitution [35].
Numerous sensory substitution systems have been developed
to convert auditory information into visual [15, 17, 31, 42, 53, 68,
79, 81] and tactile cues [28–30, 33, 43, 51, 52]. Sensory substitution
systems faces the perennial problem of the difculty of designing
intuitive mappings [52].
This study is not an approach that allows DHH people to imitate
the way hearing people enjoy music. The main objective was to
create an application that allows DHH people to enjoy making
vibrations such that they can appreciate the nuances of timbres and
melodies.
2.3 Creating Music with Mobile Devices
Owing to their widespread use worldwide, mobile phones have the
potential to help people overcome cultural and economic barriers.
In 2004, Tanaka proposed a system for creating music with mobile
devices [62], and development communities have emerged [20].
Since the development of the pocket Gamelan [59] in 2006,
wherein a mobile phone is used with music by moving it around,
several projects [13, 22, 36, 56, 57, 70, 71] dedicated to realizing
musical interaction in this way have emerged. Smartphone mu-
sic applications have enabled multiplayer performances. Notably,
the Stanford Mobile Phone Orchestra [45] was founded in 2007,
followed which an ensemble at the University of Michigan was
founded [12]. However, to the best of our knowledge, there are no
examples of DHH people being involved in these activities.
Several mobile music projects address the importance of tactile
feedback [21, 25, 61]. Unfortunately, haptic feedback tends to be
monotonous on smartphones and does not directly provide the
depth of musical structure required for DHH people. In our ap-
proach, instead of using a smartphone’s haptic sensations as an
adjunct to music, we use them for purposes of musical expression.
In summary, several smartphone sensory output methods are being
applied to solo and group musical performances, and the inclusion
of DHH people in these activities is required.
2.4 GES Method
The GES method is used to design gesture-enabled user interfaces
that refect the behavioral preferences of end users. In doing so,
researchers and designers seek to identify the gestures that corre-
spond to certain system operations (i.e., referents). Often, designers
devise diferent gestures that correspond to all referents. However,
by involving the end user in this process, we expect to build a
gesture set that is easier to remember and use. In this study, the
pleasing nature of movement is a novel application of a GES.
Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People
The GES was frst introduced by Wobbrock et al. in 2005 [73].
Since then, GESs have been realized using various systems, and
216 peer-reviewed papers were compiled in 2020 [66]. GES has
been applied to various contexts including driving [44, 77], drone
operations [10, 60], and augmented-reality scenarios [1, 67, 72].
In addition to freehand types [69, 75, 76], smartphone [14, 34, 80]
and smartwatch gestures [4, 8, 39] have been studied. Our research
addresses the musical instrument context for DHH users, noting
that GES methods have been underutilized among the ASSETS
community, and we hope this study will inspire them.
3 EXPERIMENT
3.1 Participants
We recruited 11 participants (fve females, fve males, and one
“other”) through email and social media channels (see Table 1).
The eligibility criteria required DHH individuals with an iPhone
8 or a newer model, for which we designed the stimuli and study.
This hardware limitation was necessitated by the requirement of
the smartphone’s Core Haptics engine and framework, which is
a linear resonant actuator that provides customized vibrational
feedback at fne levels of detail. The criteria did not include any
musical experience. The participants were 25.4 years old on average
(SD = 5.39, range = 19-38). Eight participants were profoundly deaf,
and the others noted hearing difculties that interfered with their
daily lives. Participants were paid ¥860 for a 1-h study period. The
experiment reported in this study was approved by the authors’
university’s ethics committee.
3.2 Study Application
We created an iOS application using Xcode and Swift. The appli-
cation frst intakes the user consent to participate and verifes the
presence and functionality of the Core Haptics engine of the Ap-
ple smartphone. When launched, image buttons appear with the
names of 10 musical instruments, as shown in Fig. S1. When the
user makes a selection, haptic feedback is provided.
The 10 instruments used as stimuli were representative of the
four main Hornbostel–Sachs musical classifcation hierarchy ele-
ments (i.e., idiophone, membranophone, chordophone, and aero-
phone)(see Fig. S1). The reason for this was to investigate how
gestures were diferent depending on the category of instruments.
This hierarchy was included such that the gestures could be coded
and analyzed heuristically.
The vibrational pattern of each instrument was designed empiri-
cally to capture the characteristics of the instrument and make it
easy to distinguish (see Appendix A for details).
3.3 Experimental Process
The study was conducted remotely using Zoom, owing to the
COVID-19 pandemic and Japan’s social-distancing restrictions. The
iOS application was delivered to participants through TestFlight 3 .
The experiment began with demographic questions. Prior to the
experiment, participants were asked about their preferred com-
munication channel (i.e., text chat or verbal), and we proceeded
according to their preferences.
3 https://developer.apple.com/jp/testfight/
ASSETS ’22, October 23–26, 2022, Athens, Greece
First, the participants were provided instructions on how to use
the application. After they confrmed that all 10 instruments were
working properly, they were asked to identify the instruments with
which they were unfamiliar. Thereafter, they were presented with
the referents and asked to invent gestures for the given instruments
as the vibrations were created. They demonstrated the gestures over
Zoom. To mitigate ordering efects, the referents were presented in
a random order (Appendix B).
Following the study by Ruiz et al. [58], the participants were
instructed to treat their smartphone as a “magic brick” that could
recognize any gesture they might wish to perform. Hence, they
were less infuenced by the sensing and recognition technologies in
creating the gestures. After demonstrating their gestures for each
referent, the participants were asked to evaluate those gestures
using a fve-point Likert scale of questions [7]: (1) Goodness of ft:
“The gesture I performed is a good match for its purpose,” (2) Ease
of use: “The gesture I performed is easy to perform.”
Additionally, the experimenter(s) delved into how the partic-
ipants produced the gesture and why they selected it. Although
many GESs have employed a think-aloud protocol, it was difcult to
apply this method as some preferred to communicate via text chat.
Therefore, we asked them the questions immediately after they
demonstrated their gestures, resulting in a more concise taxonomy.
3.4 Data Analysis
We evaluated user gesture preferences using the agreement rate
(AR) for each referent obtained using the formula provided by
Vatavu and Wobbrock [65] (see Appendix C) and the thinking time
in seconds required by participants to propose a gesture for a given
referent after being asked.
4 RESULTS
4.1 User-Defned Gesture Set
With 11 participants and 10 referents, 110 total gestures were cre-
ated and clustered into groups of similar types according to the
criteria described in Appendix D. Of the 110 gestures obtained,
55 were distinct. Fig. 1 summarizes the ARs and most frequently
proposed gestures for each referent. The mean AR value was 0.27,
which is a typical value compared with those reported in previous
studies. The ARs for jaw harps and tam-tams were 0.05, and this
value is considered small [65]. In contrast, similar gestures were
proposed for maracas, with an AR of 0.65.
4.2 Correlation Analysis
The mean AR across all referents was 0.27 (SD = 0.18). The high-
est rate was obtained for maracas, and the lowest rate (0.05) was
obtained for the tam-tam and jaw harp (see Fig. 1). The average
thinking time was 62 s (SD = 59.1), goodness of ft was 4.02 (SD =
0.91), and ease of use was 4.12 (SD = 0.94).
Three pairs of signifcant correlations among the metrics are
shown in Fig. S2. We found a signifcant negative correlation be-
tween ease of use and thinking time (Pearson’s r (N =10) = −0.63, p =
0.05; see Fig. S2 (a)). We also found a signifcant positive correlation
between goodness of ft and AR (Pearson’s r (N =10) = −0.66, p =
0.03) and ease of use and AR (Pearson’s r (N =10) = −0.63, p = 0.05;
see Figs. S2 (b) and S2 (c)).
ASSETS ’22, October 23–26, 2022, Athens, Greece Iijima, et al.

Table 1: Demographics of the DHH participants for the GES.

ID Age Gender Hearing loss Onset age Hearing device Musical Experiences
P1 23 Male Unknown 9 years None piano
P2 38 Female Profound 3 years Hearing Aids chorus, piano
P3 29 Female Profound Birth Hearing Aids chorus, percussion, piano
P4 26 Male Profound 1 year Hearing Aids chorus, drum, melodica, recorder, xylophone
P5 28 Other Profound Birth Hearing Aids chorus, clapping, piano, xylophone
P6 21 Female Profound 2 years Hearing Aids chorus
P7 21 Female Unknown 16 years None accordion, chorus, piano, recorder, xylophone
P8 25 Male Profound Birth None drum, fute, melodica, piano, recorder
P9 19 Male Profound 1 year Cochlear Implants chorus, koto, piano, recorder
P10 21 Female Unknown 2 years Hearing Aids chorus, recorder
P11 28 Male Profound Birth Hearing Aids chorus, piano

4.3 Gesture Taxonomy for Musical Instruments
Several GES taxonomies have been developed, including surface
gestures [74] and mobile interaction types [58]. Taxonomies provide
a means of understanding design spaces and, in this case, explaining
and analyzing music appreciation gestures. Our taxonomy is the
frst developed for DHH gesture-based instrument performance,
and it is manually classifed into three dimensions of Nature, Form,
and Function. Each dimension is further classifed into smaller cate-
gories (see Fig. 2). The classifcation of all instruments together and
those of each instrument are shown in Figs. S3 and S4, respectively.
The Nature dimension classifes gesture-meaning relationships.
For the instruments, 70% of the gestures were classifed as Motion
and 22% were Vibration, accounting for 92% of the Nature ratings.
Motion accounted for more than 60% of all instruments, apart from
the jaw harp. The highest percentage of Vibration classifcations
was observed with the tam-tam (36%) and jaw harp (55%), which
were the instruments with the lowest ARs.
The Form dimension distinguishes whether the smartphone’s
position changes and whether the position of the hand not holding
the smartphone moves. Overall, Smartphone, Hand, and Both ac-
counted for 86% of the total. Thus, most gestures were accompanied
by user arm movements. Both was the most common gesture for
cymbals (64%) and maracas (45%), as both are commonly played by
holding objects of the same shape in both hands.
The Function dimension classifes the way in which a smart-
phone is observed. The instruments with most gestures classifed
as Non-sonorous are tam-tam (55%), guitar (55%), triangle (45%),
and guiro (36%). These four are played by holding and moving
non-soronous objects, such as a percussion mallet for the tam-tam
and triangle, pick for the guitar, and stick for the guiro, with one
hand. The instruments played by moving sonorous objects have a
high percentage of gestures classifed as Sonorous: castanets (100%),
cymbal (64%), maracas (82%), and pellet drum (91%).
5 DISCUSSION
5.1 Interpretation of Quantitative Analysis
5.1.1 Subjective ratings and thinking time. There was a negative
correlation between ease of use and thinking time. This can be
interpreted in two ways. First, when participants spend more time
thinking about a task, they may come up with gestures that are
more fexible and easier to demonstrate. Second, corresponding
gestures are quickly imagined when the musical instrument is easy
to play.
5.1.2 Subjective ratings and agreement rate. There was a positive
correlation between goodness of ft and agreement rate and be-
tween ease of use and agreement rate. It can be interpreted that the
gestures that most participants produce tend to be easy to perform
and suitable for the task.
5.2 Future Work
5.2.1 User Studies for Playing to the Rhythm. The objective of this
study was to use gestures to play music and not to select which
instrument to play. To this end, this study aimed to defne the
gestures as preliminary to playing a piece of music to a rhythm. For
example, when learning how to play the drums, a series of steps is
normally applied (e.g., hold the drumsticks, hit the drums, and play
to a rhythm). Of these, holding the drumsticks and hit the drums
are the gestures we investigated, as they precede rhythmic playing.
Future work should include experiments with playing to a rhythm
using gestures.
5.2.2 Cultural diferences. Previous research [40] has discussed
user-defned gestures while focusing on cultural diferences. Hence,
some diferences may appear between Deaf and other cultures. For
example, P3 mimicked a gesture that meant “sutra” in sign language,
associating the vibration of a wooden fsh used in a Buddhist temple
with that of the cymbal. Future research should include GESs for
playing musical instruments with hearing people and other user
groups, as it may provide deeper insights into inclusion.
5.3 Implications for Gesture Designs and
Implementations
5.3.1 Instruments Played with Both Hands. The instruments played
with the same object in both hands were cymbals and maracas. In
the Form dimension of cymbals and maracas, Both accounted for
64% and 45%, respectively. When playing a musical instrument ap-
plication, one hand holds the smartphone as the other creates sound.
This form is asymmetric and considerably difers from the actual
instrument. Nevertheless, our participants preferred to perform the
same movement with both hands. Other instruments played with
Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 2: Taxonomy of gestures for musical instruments based on 110 gestures.

Dimension Category Description

Motion Gesture that mimics the motion of playing a real instrument.
Vibration Gesture that indicates something associated with the feel of vibration.
Nature
Abstract The mapping between instruments and gestures is arbitrary.
Name Gesture that indicates something associated with the name of the instrument.
Smartphone User moves the smartphone.
Hand User moves the hand that is not holding the smartphone.
Form
Both User moves the smartphone and hand that is not holding the smartphone at the same time.
Pose User performs the gesture keeping the smartphone and hands at one location.
Sonorous The smartphone serve as a sonorous object.
Function Non-sonorous The smartphone serve as a non-sonorous object, such as hand and stick.
Others The smartphone serve as something other than the above two cases.

both hands include wooden clappers and several types of drums.
For those, it is recommended that the hand not holding the smart-
phone be able to move as well. In that case, the hand not holding the
smartphone would have difculty receiving vibrotactile sensations.
Notably, P3 stated that he would like to feel the vibrations with
both hands.
5.3.2 Hand Poses are not Important in Motion Gestures. Partici-
pants did not insist on hand poses when demonstrating gestures
that were not classifed as Pose in the Form dimension. For example,
all those who demonstrated the gesture of tapping the smartphone
with a fnger said that it was acceptable to tap it with another
fnger. When devising the gesture of swinging downward with
the smartphone, P1 said, “I hold my smartphone this way, but I
think other users should hold their smartphone in the way they
prefer.’ Motion gestures for playing a musical instrument should in-
clude various hand poses. Additionally, when developing a motion
gesture recognizer, diverse hand poses should be assumed.
5.3.3 Association from Vibration. Gestures falling into the Vibra-
tion category were created by imitating daily actions that evoke
pleasant feelings. The participants failed to agree on any gestures
in the Vibration category. For example, P7 associated the vibration
of the tam-tam with the motion of hitting a ball in tennis, whereas
P3 associated it with the motion of accelerating a motorcycle. De-
signing a gesture based on vibrations may result in low agreement;
hence, other design methods (e.g., Motion, Abstract, and Name in
Nature dimensions) should be prioritized.
5.3.4 Influence of Daily Smartphone Usage. As mentioned in Sec-
tion 3.3, participants were instructed to assume that the smartphone
was a magic brick that could recognize any gesture. However, sev-
eral participants were inspired by their daily use of smartphones.
For example, when creating a gesture for the triangle instrument,
P7 mimicked the action of capturing a picture with a smartphone
such that the position of the smartphone would be stable. Because
smartphones are devices that are used on a daily basis, the way they
are manipulated has been optimized to ensure user comfort. Thus,
it is recommended that gestures be designed to not disrupt the
comfortable holding of the smartphone in normal situations, such
as when holding it vertically or horizontally or when capturing
pictures.
5.3.5 Smartphone Cases and Accessories. There are ring-type ac-
cessories on the market that attach to the back of a smartphone to
make it easier to hold by inserting a user’s fnger through it. P7 and
P8 emphasized that such accessories would allow certain gestures
to be executed more safely. Some gestures are difcult to perform
with notebook-type smartphone cases. When employing gestures
that include vigorous movements, it may be safer to encourage the
user to install a ring-type accessory or remove the notebook cover.
5.3.6 Referents with Low Agreemnet.
Unknown instruments: jaw harp. No participants were familiar
with the jaw harp. With other instruments, participants often cre-
ated gestures that imitated how they imagined they would play the
real one. However, this approach did not work with the jaw harp.
It is likely that the resulting gestures were diverse for this reason.
Instruments that can be played with only one hand: tam-tam. The
agreement rate of the tam-tam was 0.05; however, all other idio-
phones had values greater than 0.2. This may be attributed to the
fact that the real tam-tam does not require the performer to hold
the sonorous object, and that it can be played with one hand. All
other idiophones are two-handed instruments; thus, the roles of
the hand holding the device and the other hand are easily fxed. Be-
cause this is not the case with the tam-tam, the gesture parameters
appear to have been distributed in various ways depending on the
smartphone being held by one or both hands, and whether it was
moved vertically or horizontally.
6 CONCLUSION
We conducted a study that elicited motion gestures from 11 DHH
participants based on 10 diferent musical instruments, using their
smartphones as instruments. Based on the commonalities of user-
defned gestures, we created the frst taxonomy of gestures for
DHH people such that developers of musical-instrument-emulating
smartphone applications will have design guidelines. Our fndings
provide several implications that will eventually help DHH people
enjoy various instruments on a daily basis.
ACKNOWLEDGMENTS
This work was supported by JST CREST Grant Number JPMJCR19F2,
Japan.
ASSETS ’22, October 23–26, 2022, Athens, Greece
REFERENCES
[1] Christopher R. Austin, Barrett Ens, Kadek Ananta Satriadi, and Bernhard Jenny.
2020. Elicitation study investigating hand and foot gesture interaction for im-
mersive maps in augmented reality. Cartography and Geographic Information
Science 47, 3 (2020), 214–228. https://doi.org/10.1080/15230406.2019.1696232
arXiv:https://doi.org/10.1080/15230406.2019.1696232
[2] Amal Dar Aziz, Chris Warren, Hayden Bursk, and Sean Follmer. 2008. The
Flote: An Instrument for People with Limited Mobility. In Proceedings of the 10th
International ACM SIGACCESS Conference on Computers and Accessibility (Halifax,
Nova Scotia, Canada) (Assets ’08). Association for Computing Machinery, New
York, NY, USA, 295–296. https://doi.org/10.1145/1414471.1414545
[3] Emeline Brulé. 2016. Playing Music with the Head. In Proceedings of the 18th
International ACM SIGACCESS Conference on Computers and Accessibility (Reno,
Nevada, USA) (ASSETS ’16). Association for Computing Machinery, New York,
NY, USA, 339–340. https://doi.org/10.1145/2982142.2982146
[4] Thisum Buddhika, Haimo Zhang, Samantha W. T. Chan, Vipula Dissanayake,
Suranga Nanayakkara, and Roger Zimmermann. 2019. FSense: Unlocking the
Dimension of Force for Gestural Interactions Using Smartwatch PPG Sensor. In
Proceedings of the 10th Augmented Human International Conference 2019 (Reims,
France) (AH2019). Association for Computing Machinery, New York, NY, USA,
Article 11, 5 pages. https://doi.org/10.1145/3311823.3311839
[5] Marshall Chasin. 2003. Music and hearing aids. The Hearing Journal 56, 7 (July
2003), 36–38.
[6] Alice-Ann Darrow. 1993. The Role of Music in Deaf Culture: Implications for
Music Educators. Journal of Research in Music Education 41, 2 (1993), 93–110.
https://doi.org/10.2307/3345402 arXiv:https://doi.org/10.2307/3345402
[7] Nem Khan Dim, Chaklam Silpasuwanchai, Sayan Sarcar, and Xiangshi Ren.
2016. Designing Mid-Air TV Gestures for Blind People Using User- and Choice-
Based Elicitation Approaches. In Proceedings of the 2016 ACM Conference on
Designing Interactive Systems (Brisbane, QLD, Australia) (DIS ’16). Association for
Computing Machinery, New York, NY, USA, 204–214. https://doi.org/10.1145/
2901790.2901834
[8] Tilman Dingler, Rufat Rzayev, Alireza Sahami Shirazi, and Niels Henze. 2018.
Designing Consistent Gestures Across Device Types: Eliciting RSVP Controls for
Phone, Watch, and Glasses. Association for Computing Machinery, New York,
NY, USA, 1–12. https://doi.org/10.1145/3173574.3173993
[9] Ward R Drennan and Jay T Rubinstein. 2008. Music perception in cochlear
implant users and its relationship with psychophysical capabilities. Journal of
rehabilitation research and development 45, 5 (2008), 779—789. https://doi.org/10.
1682/jrrd.2007.08.0118
[10] Jane L. E, Ilene L. E, James A. Landay, and Jessica R. Cauchard. 2017. Drone &
Wo: Cultural Infuences on Human-Drone Interaction Techniques. In Proceedings
of the 2017 CHI Conference on Human Factors in Computing Systems (Denver,
Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY,
USA, 6794–6799. https://doi.org/10.1145/3025453.3025755
[11] George Thomas Ealy. 1994. Of ear trumpets and a resonance plate: early hearing
aids and Beethoven’s hearing perception. 19th-Century Music 17, 3 (Spring 1994),
262–273.
[12] Georg Essl. 2010. The Mobile Phone Ensemble As Classroom. In Proceedings of
the International Computer Music Conference (ICMC), Stony Brooks/New York.
[13] Georg Essl and Michael Rohs. 2007. ShaMus – A Sensor-Based Integrated Mobile
Phone Instrument. In Proceedings of the International Computer Music Conference
(ICMC). 27–31.
[14] Sharif A. M. Faleel, Michael Gammon, Yumiko Sakamoto, Carlo Menon, and
Pourang Irani. 2020. User Gesture Elicitation of Common Smartphone Tasks for
Hand Proximate User Interfaces. In Proceedings of the 11th Augmented Human
International Conference (Winnipeg, Manitoba, Canada) (AH ’20). Association for
Computing Machinery, New York, NY, USA, Article 6, 8 pages. https://doi.org/
10.1145/3396339.3396363
[15] David Fourney. 2012. Can Computer Representations of Music Enhance En-
joyment for Individuals Who Are Hard of Hearing?. In Proceedings of the 13th
International Conference on Computers Helping People with Special Needs - Volume
Part I (Linz, Austria) (ICCHP’12). Springer-Verlag, Berlin, Heidelberg, 535–542.
https://doi.org/10.1007/978-3-642-31522-0_80
[16] David W. Fourney. 2015. Making the invisible visible: visualization of music and
lyrics for deaf and hard of hearing audiences. https://doi.org/10.32920/ryerson.
14664129.v1
[17] David W. Fourney and Deborah I. Fels. 2009. Creating access to music through
visualization. In 2009 IEEE Toronto International Conference Science and Technology
for Humanity (TIC-STH). 939–944. https://doi.org/10.1109/TIC-STH.2009.5444364
[18] Qian-Jie Fu and John J Galvin. 2007. Computer-Assisted Speech Training for
Cochlear Implant Patients: Feasibility, Outcomes, and Future Directions. Seminars
in hearing 28, 2 (May 2007). https://doi.org/10.1055/s-2007-973440
[19] John J. Galvin III, Qian-Jie Fu, and Robert V. Shannon. 2009. Melodic
Contour Identifcation and Music Perception by Cochlear Implant
Users. Annals of the New York Academy of Sciences 1169, 1 (July
2009), 518–533. https://doi.org/10.1111/j.1749-6632.2009.04551.x
Iijima, et al.
arXiv:https://nyaspubs.onlinelibrary.wiley.com/doi/pdf/10.1111/j.1749-
6632.2009.04551.x
[20] Lalya Gaye, Lars Erik Holmquist, Frauke Behrendt, and Atau Tanaka. 2006. Mobile
Music Technology: Report on an Emerging Community. In Proceedings of the
2006 Conference on New Interfaces for Musical Expression (Paris, France) (NIME
’06). IRCAM — Centre Pompidou, Paris, FRA, 22–25.
[21] Günter Geiger. 2006. Using the Touch Screen as a Controller for Portable Com-
puter Music Instruments. In Proceedings of the 2006 Conference on New Interfaces
for Musical Expression (Paris, France) (NIME ’06). IRCAM — Centre Pompidou,
Paris, FRA, 61–64.
[22] Nicholas Gillian, Sile O’Modhrain, and Georg Essl. 2009. Scratch-Of : A Gesture
Based Mobile Music Game with Tactile Feedback. In Proceedings of the Inter-
national Conference on New Interfaces for Musical Expression. Zenodo, 308–311.
https://doi.org/10.5281/zenodo.1177553
[23] Evelyn Glennie. 2015. Hearing Essay. https://www.evelyn.co.uk/hearing-essay/.
(Accessed on 07/10/2022).
[24] Rumi Hiraga and Kjetil Falkenberg Hansen. 2013. Sound Preferences of Persons
with Hearing Loss Playing an Audio-Based Computer Game. In Proceedings of the
3rd ACM International Workshop on Interactive Multimedia on Mobile & Portable
Devices (Barcelona, Spain) (IMMPD ’13). Association for Computing Machinery,
New York, NY, USA, 25–30. https://doi.org/10.1145/2505483.2505489
[25] Euyshick Hong and Jun Kim. 2017. Webxophone: Web Audio Wind Instrument.
In Proceedings of the International Conference on Algorithms, Computing and
Systems (Jeju Island, Republic of Korea) (ICACS ’17). Association for Computing
Machinery, New York, NY, USA, 79–82. https://doi.org/10.1145/3127942.3127954
[26] Ryo Iijima, Akihisa Shitara, Sayan Sarcar, and Yoichi Ochiai. 2021. Smartphone
Drum: Gesture-Based Digital Musical Instruments Application for Deaf and Hard
of Hearing People. In Symposium on Spatial User Interaction (Virtual Event, USA)
(SUI ’21). Association for Computing Machinery, New York, NY, USA, Article 25,
2 pages. https://doi.org/10.1145/3485279.3488285
[27] Alon Ilsar and Gail Kenning. 2020. Inclusive Improvisation through Sound
and Movement Mapping: From DMI to ADMI. In The 22nd International ACM
SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece)
(ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article
49, 8 pages. https://doi.org/10.1145/3373625.3416988
[28] Maria Karam, Carmen Branje, Gabe Nespoli, Norma Thompson, Frank A. Russo,
and Deborah I. Fels. 2010. The Emoti-Chair: An Interactive Tactile Music Exhibit.
In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (Atlanta,
Georgia, USA) (CHI EA ’10). Association for Computing Machinery, New York,
NY, USA, 3069–3074. https://doi.org/10.1145/1753846.1753919
[29] Maria Karam, Gabe Nespoli, Frank Russo, and Deborah I. Fels. 2009. Modelling
Perceptual Elements of Music in a Vibrotactile Display for Deaf Users: A Field
Study. In Proceedings of the 2009 Second International Conferences on Advances in
Computer-Human Interactions (ACHI ’09). IEEE Computer Society, USA, 249–254.
https://doi.org/10.1109/ACHI.2009.64
[30] Maria Karam, Frank Russo, Carmen Branje, Emily Price, and Deborah I. Fels. 2008.
Towards a Model Human Cochlea: Sensory Substitution for Crossmodal Audio-
Tactile Displays. In Proceedings of Graphics Interface 2008 (Windsor, Ontario,
Canada) (GI ’08). Canadian Information Processing Society, CAN, 267–274.
[31] Jeeeun Kim, Swamy Ananthanarayan, and Tom Yeh. 2015. Seen Music: Ambient
Music Data Visualization for Children with Hearing Impairments. In Proceedings
of the 14th International Conference on Interaction Design and Children (Boston,
Massachusetts) (IDC ’15). Association for Computing Machinery, New York, NY,
USA, 426–429. https://doi.org/10.1145/2771839.2771870
[32] Joy Kim and Jonathan Ricaurte. 2011. TapBeats: Accessible and Mobile Casual
Gaming. In The Proceedings of the 13th International ACM SIGACCESS Conference
on Computers and Accessibility (Dundee, Scotland, UK) (ASSETS ’11). Association
for Computing Machinery, New York, NY, USA, 285–286. https://doi.org/10.
1145/2049536.2049609
[33] Bruno La Versa, Isabella Peruzzi, Luca Diamanti, and Marco Zemolin. 2014. MU-
VIB: Music and Vibration. In Proceedings of the 2014 ACM International Sym-
posium on Wearable Computers: Adjunct Program (Seattle, Washington) (ISWC
’14 Adjunct). Association for Computing Machinery, New York, NY, USA, 65–70.
https://doi.org/10.1145/2641248.2641267
[34] Huy Viet Le, Sven Mayer, Maximilian Weiß, Jonas Vogelsang, Henrike Weingärt-
ner, and Niels Henze. 2020. Shortcut Gestures for Mobile Text Editing on Fully
Touch Sensitive Smartphones. ACM Trans. Comput.-Hum. Interact. 27, 5, Article
33 (aug 2020), 38 pages. https://doi.org/10.1145/3396233
[35] Charles Lenay, Stephane Canu, and Pierre Villon. 1997. Technology and Per-
ception: The Contribution of Sensory Substitution Systems. In Proceedings of
the 2nd International Conference on Cognitive Technology (CT ’97) (CT ’97). IEEE
Computer Society, USA, 44.
[36] Yang Kyu Lim and Woon Seung Yeo. 2014. Smartphone-based Music Conduct-
ing. In Proceedings of the International Conference on New Interfaces for Musical
Expression. Zenodo, 573–576. https://doi.org/10.5281/zenodo.1178851
[37] Charles J. Limb and Alexis T. Roy. 2014. Technological, biological, and acoustical
constraints to music perception in cochlear implant users. Hearing Research 308
(2014), 13–26. https://doi.org/10.1016/j.heares.2013.04.009 Music: A window
Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People
into the hearing brain.
[38] Joana Lobo, Soichiro Matsuda, Izumi Futamata, Ryoichi Sakuta, and Kenji Suzuki.
2019. CHIMELIGHT: Augmenting Instruments in Interactive Music Therapy
for Children with Neurodevelopmental Disorders. In The 21st International ACM
SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA)
(ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 124–
135. https://doi.org/10.1145/3308561.3353784
[39] Meethu Malu, Pramod Chundury, and Leah Findlater. 2018. Exploring Acces-
sible Smartwatch Interactions for People with Upper Body Motor Impairments.
Association for Computing Machinery, New York, NY, USA, 1–12. https:
//doi.org/10.1145/3173574.3174062
[40] Dan Mauney, Jonathan Howarth, Andrew Wirtanen, and Miranda Capra. 2010.
Cultural Similarities and Diferences in User-Defned Gestures for Touchscreen
User Interfaces. In CHI ’10 Extended Abstracts on Human Factors in Computing Sys-
tems (Atlanta, Georgia, USA) (CHI EA ’10). Association for Computing Machinery,
New York, NY, USA, 4015–4020. https://doi.org/10.1145/1753846.1754095
[41] Hugh J. McDermott. 2004. Music Perception with Cochlear Implants: A Review.
Trends in Amplifcation 8, 2 (January 2004), 49–82. https://doi.org/10.1177/
108471380400800203 arXiv:https://doi.org/10.1177/108471380400800203 PMID:
15497033.
[42] Jorge Mori and Deborah I. Fels. 2009. Seeing the music can animated lyrics
provide access to the emotional content in music for people who are deaf or hard
of hearing?. In 2009 IEEE Toronto International Conference Science and Technology
for Humanity (TIC-STH). 951–956. https://doi.org/10.1109/TIC-STH.2009.5444362
[43] Suranga Nanayakkara, Elizabeth Taylor, Lonce Wyse, and S H. Ong. 2009. An
Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music
Display and a Haptic Chair. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for
Computing Machinery, New York, NY, USA, 337–346. https://doi.org/10.1145/
1518701.1518756
[44] Vijayakumar Nanjappan, Rongkai Shi, Hai-Ning Liang, Kim King-Tong Lau, Yong
Yue, and Katie Atkinson. 2019. Towards a Taxonomy for In-Vehicle Interactions
Using Wearable Smart Textiles: Insights from a User-Elicitation Study. Multimodal
Technologies and Interaction 3, 2 (2019). https://doi.org/10.3390/mti3020033
[45] Jieun Oh, Jorge Herrera, Nicholas J. Bryan, Luke Dahl, and Ge Wang. 2010. Evolv-
ing The Mobile Phone Orchestra. In Proceedings of the International Conference
on New Interfaces for Musical Expression. Zenodo, 82–87. https://doi.org/10.5281/
zenodo.1177871
[46] Shotaro Omori and Ikuko Eguchi Yairi. 2013. Collaborative Music Application
for Visually Impaired People with Tangible Objects on Table. In Proceedings of
the 15th International ACM SIGACCESS Conference on Computers and Accessibility
(Bellevue, Washington) (ASSETS ’13). Association for Computing Machinery,
New York, NY, USA, Article 42, 2 pages. https://doi.org/10.1145/2513383.2513403
[47] Deysi Helen Ortega, Franceli Linney Cibrian, and Mónica Tentori. 2015. Bend-
ableSound: A Fabric-Based Interactive Surface to Promote Free Play in Chil-
dren with Autism. In Proceedings of the 17th International ACM SIGACCESS
Conference on Computers & Accessibility (Lisbon, Portugal) (ASSETS ’15). As-
sociation for Computing Machinery, New York, NY, USA, 315–316. https:
//doi.org/10.1145/2700648.2811355
[48] Mikel Ostiz-Blanco, Alfredo Pina, Miriam Lizaso, Jose Javier Astráin, and Gonzalo
Arrondo. 2018. Using the Musical Multimedia Tool ACMUS with People with
Severe Mental Disorders: A Pilot Study. In Proceedings of the 20th International
ACM SIGACCESS Conference on Computers and Accessibility (Galway, Ireland)
(ASSETS ’18). Association for Computing Machinery, New York, NY, USA, 462–464.
https://doi.org/10.1145/3234695.3241016
[49] Carol A Padden and Tom Humphries. 1988. Deaf in America. Harvard University
Press.
[50] William Payne, Alex Xu, Amy Hurst, and S. Alex Ruthmann. 2019. Non-Visual
Beats: Redesigning the Groove Pizza. In The 21st International ACM SIGACCESS
Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19).
Association for Computing Machinery, New York, NY, USA, 651–654. https:
//doi.org/10.1145/3308561.3354590
[51] Benjamin Petry, Thavishi Illandara, Don Samitha Elvitigala, and Suranga
Nanayakkara. 2018. Supporting Rhythm Activities of Deaf Children Using Music-
Sensory-Substitution Systems. Association for Computing Machinery, New York,
NY, USA, 1–10. https://doi.org/10.1145/3173574.3174060
[52] Benjamin Petry, Thavishi Illandara, and Suranga Nanayakkara. 2016. MuSS-Bits:
Sensor-Display Blocks for Deaf People to Explore Musical Sounds. In Proceedings
of the 28th Australian Conference on Computer-Human Interaction (Launceston,
Tasmania, Australia) (OzCHI ’16). Association for Computing Machinery, New
York, NY, USA, 72–80. https://doi.org/10.1145/3010915.3010939
[53] Michael Pouris and Deborah I. Fels. 2012. Creating an Entertaining and Infor-
mative Music Visualization. In Proceedings of the 13th International Conference
on Computers Helping People with Special Needs - Volume Part I (Linz, Austria)
(ICCHP’12). Springer-Verlag, Berlin, Heidelberg, 451–458. https://doi.org/10.
1007/978-3-642-31522-0_68
[54] Grazia Ragone. 2020. Designing Embodied Musical Interaction for Children with
Autism. In The 22nd International ACM SIGACCESS Conference on Computers and
ASSETS ’22, October 23–26, 2022, Athens, Greece
Accessibility (Virtual Event, Greece) (ASSETS ’20). Association for Computing
Machinery, New York, NY, USA, Article 104, 4 pages. https://doi.org/10.1145/
3373625.3417077
[55] Janine Roebuck. 2007. I am a deaf opera singer. https://www.theguardian.com/
theguardian/2007/sep/29/weekend7.weekend2. (Accessed on 07/10/2022).
[56] Michael Rohs and Georg Essl. 2007. CaMus²: Optical Flow and
Collaboration in Camera Phone Music Performance. In Proceedings of the 7th
International Conference on New Interfaces for Musical Expression (New York, New
York) (NIME ’07). Association for Computing Machinery, New York, NY, USA,
160–163. https://doi.org/10.1145/1279740.1279770
[57] Michael Rohs, Georg Essl, and Martin Roth. 2006. CaMus: Live Music Performance
using Camera Phones and Visual Grid Tracking. In Proceedings of the International
Conference on New Interfaces for Musical Expression. Zenodo, 31–36. https:
//doi.org/10.5281/zenodo.1176997
[58] Jaime Ruiz, Yang Li, and Edward Lank. 2011. User-Defned Motion Gestures for
Mobile Interaction. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Comput-
ing Machinery, New York, NY, USA, 197–206. https://doi.org/10.1145/1978942.
1978971
[59] Greg Schiemer and Mark Havryliv. 2006. Pocket Gamelan: Tuneable Trajectories
for Flying Sources in <i>Mandala 3</i> and <i>Mandala 4</i>. In Proceedings
of the 2006 Conference on New Interfaces for Musical Expression (Paris, France)
(NIME ’06). IRCAM — Centre Pompidou, Paris, FRA, 37–42.
[60] Matthias Seuter, Eduardo Rodriguez Macrillante, Gernot Bauer, and Christian
Kray. 2018. Running with Drones: Desired Services and Control Gestures. In
Proceedings of the 30th Australian Conference on Computer-Human Interaction
(Melbourne, Australia) (OzCHI ’18). Association for Computing Machinery, New
York, NY, USA, 384–395. https://doi.org/10.1145/3292147.3292156
[61] Bradley Strylowski, Jesse Allison, and Jesse Guessford. 2014. Pitch Canvas:
Touchscreen Based Mobile Music Instrument. In Proceedings of the International
Conference on New Interfaces for Musical Expression. Zenodo, 171–174. https:
//doi.org/10.5281/zenodo.1178947
[62] Atau Tanaka. 2004. Mobile Music Making. In Proceedings of the 2004 Conference
on New Interfaces for Musical Expression (Hamamatsu, Shizuoka, Japan) (NIME
’04). National University of Singapore, SGP, 154–156.
[63] Stephanie Valencia, Dwayne Lamb, Shane Williams, Harish S. Kulkarni, Ann
Paradiso, and Meredith Ringel Morris. 2019. Dueto: Accessible, Gaze-Operated
Musical Expression. In The 21st International ACM SIGACCESS Conference on
Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for
Computing Machinery, New York, NY, USA, 513–515. https://doi.org/10.1145/
3308561.3354603
[64] Maria Varvarigou, Susan Hallam, Andrea Creech, and Hilary McQueen. 2012.
Benefts experienced by older people in group music-making activities. Journal
of Applied Arts and Health 3 (08 2012), 183–198. https://doi.org/10.1386/jaah.3.2.
183_1
[65] Radu-Daniel Vatavu and Jacob O. Wobbrock. 2015. Formalizing Agreement Anal-
ysis for Elicitation Studies: New Measures, Signifcance Test, and Toolkit. In
Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing
Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machin-
ery, New York, NY, USA, 1325–1334. https://doi.org/10.1145/2702123.2702223
[66] Santiago Villarreal-Narvaez, Jean Vanderdonckt, Radu-Daniel Vatavu, and Ja-
cob O. Wobbrock. 2020. A Systematic Review of Gesture Elicitation Studies: What
Can We Learn from 216 Studies? Association for Computing Machinery, New
York, NY, USA, 855–872. https://doi.org/10.1145/3357236.3395511
[67] Panagiotis Vogiatzidakis and Panayiotis Koutsabasis. 2020. Mid-Air Gesture
Control of Multiple Home Devices in Spatial Augmented Reality Prototype.
Multimodal Technologies and Interaction 4, 3 (2020). https://doi.org/10.3390/
mti4030061
[68] Quoc V. Vy, Jorge A. Mori, David W. Fourney, and Deborah I. Fels. 2008. EnACT: A
Software Tool for Creating Animated Text Captions. In Computers Helping People
with Special Needs, Klaus Miesenberger, Joachim Klaus, Wolfgang Zagler, and
Arthur Karshmer (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 609–616.
[69] Benjamin Walther-Franks, Tanja Döring, Meltem Yilmaz, and Rainer Malaka.
2019. Embodiment or Manipulation? Understanding Users’ Strategies for Free-
Hand Character Control. In Proceedings of Mensch Und Computer 2019 (Hamburg,
Germany) (MuC’19). Association for Computing Machinery, New York, NY, USA,
661–665. https://doi.org/10.1145/3340764.3344887
[70] Ge Wang. 2014. Ocarina: Designing the iPhone’s Magic Flute. Computer
Music Journal 38, 2 (06 2014), 8–21. https://doi.org/10.1162/COMJ_a_00236
arXiv:https://direct.mit.edu/comj/article-pdf/38/2/8/1855988/comj_a_00236.pdf
[71] Gil Weinberg, Mark Godfrey, and Andrew Beck. 2010. ZOOZbeat: Mobile Music
Recreation. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems
(Atlanta, Georgia, USA) (CHI EA ’10). Association for Computing Machinery,
New York, NY, USA, 4817–4822. https://doi.org/10.1145/1753846.1754238
[72] Adam S. Williams and Francisco R. Ortega. 2020. Understanding Gesture and
Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality
Using Unconstrained Elicitation. arXiv:2009.06591 [cs.HC]
ASSETS ’22, October 23–26, 2022, Athens, Greece
[73] Jacob O. Wobbrock, Htet Htet Aung, Brandon Rothrock, and Brad A. Myers.
2005. Maximizing the Guessability of Symbolic Input. In CHI ’05 Extended Ab-
stracts on Human Factors in Computing Systems (Portland, OR, USA) (CHI EA
’05). Association for Computing Machinery, New York, NY, USA, 1869–1872.
https://doi.org/10.1145/1056808.1057043
[74] Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-
Defned Gestures for Surface Computing. In Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association
for Computing Machinery, New York, NY, USA, 1083–1092. https://doi.org/10.
1145/1518701.1518866
[75] Huiyue Wu, Jinxuan Gai, Yu Wang, Jiayi Liu, Jiali Qiu, Jianmin Wang, and Xi-
aolong(Luke) Zhang. 2020. Infuence of cultural factors on freehand gesture
design. International Journal of Human-Computer Studies 143 (2020), 102502.
https://doi.org/10.1016/j.ijhcs.2020.102502
[76] Huiyue Wu, Weizhou Luo, Neng Pan, Shenghuan Nan, Yanyi Deng, Shengqian
Fu, and Liuqingqing Yang. 2019. Understanding freehand gestures: a study of
freehand gestural interaction for immersive VR shopping applications. Human-
centric Computing and Information Sciences 9, 1 (2019), 43. https://doi.org/10.
1186/s13673-019-0204-7
[77] Huiyue Wu, Yu Wang, Jiayi Liu, Jiali Qiu, and Xiaolong (Luke) Zhang. 2020.
User-defned gesture interaction for in-vehicle information systems. Multimedia
Tools and Applications 79, 1 (2020), 263–288. https://doi.org/10.1007/s11042-019-
08075-1
[78] Ikuko Eguchi Yairi and Takuya Takeda. 2012. A Music Application for Visually
Impaired People Using Daily Goods and Stationeries on the Table. In Proceedings
of the 14th International ACM SIGACCESS Conference on Computers and Accessibil-
ity (Boulder, Colorado, USA) (ASSETS ’12). Association for Computing Machinery,
New York, NY, USA, 271–272. https://doi.org/10.1145/2384916.2384988
[79] Hui-Jen Yang, Y.-L. Lay, Yi-Chin Liou, Wen-Yu Tsao, and Cheng-Kun. Lin.
2007. Development and evaluation of computer-aided music-learning
system for the hearing impaired. Journal of Computer Assisted Learn-
ing 23, 6 (2007), 466–476. https://doi.org/10.1111/j.1365-2729.2007.00229.x
arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1365-2729.2007.00229.x
[80] Zhican Yang, Chun Yu, Fengshi Zheng, and Yuanchun Shi. 2019. ProxiTalk:
Activate Speech Input by Bringing Smartphone to the Mouth. Proc. ACM Interact.
Mob. Wearable Ubiquitous Technol. 3, 3, Article 118 (sep 2019), 25 pages. https:
//doi.org/10.1145/3351276
[81] Yinsheng Zhou, Khe Chai Sim, Patsy Tan, and Ye Wang. 2012. MOGAT: Mobile
Games with Auditory Training for Children with Cochlear Implants. In Pro-
ceedings of the 20th ACM International Conference on Multimedia (Nara, Japan)
(MM ’12). Association for Computing Machinery, New York, NY, USA, 429–438.
https://doi.org/10.1145/2393347.2393409
A DESIGN OF VIBRATIONAL FEEDBACK
Core Haptics provides two pattern types: transient (i.e., short and
fxed duration) and continuous (i.e., customizable duration). The
intensity and sharpness can be fnely controlled to provide fne
attenuation of intensity, frequency, and tempo. The vibrational pat-
terns of the instruments were determined empirically using 4 , a
graphical user interface tool used to generate haptic feedback. Hap-
trix for the Mac operating system provides graphically transient
and continuous parameters with time variations. The vibration pat-
terns were iteratively designed, tested, and improved in a laboratory
environment with multiple researchers.
We designed the vibration profle of each instrument following
the pattern guidelines provided by Apple 5 in the Apple Haptic and
Audio Pattern fle format, which can be rendered using the Core
Haptics framework.
B RANDOMIZE PROCESS
At the beginning of each experiment, variables were assigned to the
10 instruments, shufed into a uniform distribution, and presented
to the participants.
4 https://www.haptrix.com/
5 https://developer.apple.com/design/human-interface-guidelines/ios/user-
interaction/haptics/
Iijima, et al.
C AGREEMENT RATE
The agreement rate (AR) for each referent can be obtained using
Vatavu and Wobbrock’s formula [65]:
i <j δ i, j
Í
AR(r ) =
n · (n − 1)/2
(
1, i-th and j-th participants are in agreement over referent r
δi, j =
0, otherwise
where n is the number of participants.
D GESTURE GROUPING CRITERIA
(1) Tapping on a smartphone is considered the same gesture,
even when using diferent or multiple fngers. For example,
“tapping with the index fnger” and “tapping with the index
and middle fnger at the same time” are the same.
(2) The hand(s) used for gesturing are distinguished. For exam-
ple, if a smartphone is held in the right hand, touching the
screen with the right-hand thumb is diferent from touching
the screen with the left-hand thumb.
(3) Proximity of the smartphone to the body is distinguished.
For example, striking a stationary smartphone with one’s
moving hand is diferent from striking a stationary hand
with a moving smartphone.
(4) The contact locations on the phone include the touch screen,
back, and sides. It is inconsequential where on the screen
the participant taps.
E OTHER INSTRUMENTS PARTICIPANTS
WOULD LIKE TO PLAY
After the experiment, participants were asked if there were any
other instruments they would like to play on their smartphones.
The following instruments were suggested: cello (P2); drum (P9,
P10, P11); fute (P7); handbell (P3, P4); harp (P6, P11); metallophone
(P6, P9); piano (P7, P9); sampler (P1); tambourine (P2); trombone
(P7); trumpet (P6); violin (P2, P6, P9, P10); and xylophone (P6, P9,
P11). P5 stated that he would like to see the inclusion of rare and
unusual instruments that are unavailable on the market.
Few participants who wanted to make use of samplers and other
electronic music handlers. This study focused on simple, typical
acoustic instruments and did not cover instruments that produce
electronic music (e.g., synthesizers), which many smartphone users
can currently access. Therefore, exploring the designs of those
instruments will be an interesting challenge in future.
 Accessible Blockly: An Accessible Block-Based Programming
Library for People with Visual Impairments
Aboubakar Mountapmbeme Obianuju Okafor Stephanie Ludi
Department of Computer Science and Department of Computer Science and Department of Computer Science and
Engineering, University of North Engineering, University of North Engineering, University of North
Texas, Denton, Texas, USA Texas, Denton, Texas, USA Texas, Denton, Texas, USA
aboubakarmountapmbeme@my.unt.edu obianujuokafor@my.unt.edu stephanie.ludi@unt.edu

ABSTRACT
The visual and mouse-centric nature of block-based programming
environments generally make them inaccessible and challenging
to use by users with visual impairments who rely on assistive tech-
nologies to interact with computers. This prevents these users from
participating in programming activities where these systems are
used. This paper presents a prototype of an accessible block-based
programming library called Accessible Blockly that allows users to
create and navigate block-based code using a screen reader and a
keyboard. This is an attempt to make the famous Blockly library
accessible through a screen reader and keyboard. In this paper, we
present the design and implementation of Accessible Blockly. We Figure 1: Blockly’s editor showing the toolbox and workspace
also discuss the evaluation of the library for block-based code navi-
gation in a study with 12 blind programmers. Analysis of the study
results shows that Accessible Blockly efectively aids users with
Athens, Greece. ACM, New York, NY, USA, 15 pages. https://doi.org/10.1145/
reading and understanding block-based code. Participants found
3517428.3544806
Accessible Blockly easy to use and less frustrating for navigating
block-based programs. The participants also expressed enthusiasm
and interest in using the keyboard and screen reader to navigate 1 INTRODUCTION
block-based code and in the accessibility of block-based program- Block-based programming is a type of programming where users
ming. code by snapping together visual elements called blocks [1], [2]. In
a typical block-based programming environment (BBPE), a toolbox
CCS CONCEPTS contains blocks that are usually grouped by categories. Users select
• Human-centered computing → Accessibility; Accessibility; blocks from the toolbox and drag them onto the workspace, where
Accessibility systems and tools; Accessibility; Empirical studies in they are connected to one another to form a program (Figure 1).
accessibility; • Software and its engineering → Software creation Researchers and practitioners argue that these systems are suit-
and management. able for teaching coding and computational thinking skills to novice
students because their intrinsic design allows users to focus on
KEYWORDS concepts rather than language syntax [2] [3]. This has made block-
Block-based programming, Accessibility, Keyboard Navigation, based programming increasingly used in k-12 to introduce novices
Screen Reader, Students with Visual Impairments, Blind Program- to programming [4]. However, the intrinsic design of block-based
mers, Blockly programming environments presents a considerable obstacle for
ACM Reference Format:
those who cannot see or who cannot use the mouse. Block-based
Aboubakar Mountapmbeme, Obianuju Okafor, and Stephanie Ludi. 2022. programming, in its mouse-centric and visual nature, is inaccessible
Accessible Blockly: An Accessible Block-Based Programming Library for to people with visual impairments [5] [6] [4]. As a result, novices
People with Visual Impairments. In The 24th International ACM SIGACCESS with visual impairments often miss the opportunity to enjoy the
Conference on Computers and Accessibility (ASSETS ’22), October 23–26, 2022, learning facilities brought by these systems. Teachers of students
with visual impairments (TVIs) usually have to look for alternatives
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed for their students as reported in [7].
for proft or commercial advantage and that copies bear this notice and the full citation Blockly [8] [9] [10] is the de-facto library on top of which most
on the frst page. Copyrights for components of this work owned by others than ACM if not all mainstream block-based programming environments are
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a built. Popular BBPEs such as Microsoft MakeCode, Scratch 2.0, Ap-
fee. Request permissions from permissions@acm.org. pInventor, and many others are built using the Blockly Framework.
ASSETS ’22, October 23–26, 2022, Athens, Greece The Blockly library itself is not accessible to people with visual
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00 impairments, and as a result, all the mainstream BBPEs that rely on
https://doi.org/10.1145/3517428.3544806 Blockly are not accessible [7]. Recent eforts by the Blockly team,
ASSETS ’22, October 23–26, 2022, Athens, Greece
including collaboration with our research group, have led to the
addition of a keyboard interaction mechanism to Blockly.
This paper builds on our previous work [11]. It presents the
design and implementation of Accessible Blockly; a block-based
programming library augmented with keyboard and screen reader
interaction mechanisms. Accessible Blockly maintains the same
visual and mouse-based interaction as the original Blockly and uses
WAI-ARIA guidelines [12] to add screen reader and keyboard in-
teractions. Accessible Blockly is a natural extension to Blockly and
can be used in similar ways as Blockly. We believe having one sys-
tem accessible to all would allow sighted and students with visual
impairments to work on the same platforms, foster collaboration,
and remove feelings of exclusion in settings where these systems
are used. Accessible Blockly was designed with two main goals.
1. Enable keyboard interaction as an alternative to the mouse
for navigating, creating, and editing block-based code.
2. Leverage the screen reader as an alternative output channel
on Accessible Blockly.
We formulated the following research question to evaluate Ac-
cessible Blockly for block-based code navigation.
RQ1. How well does the keyboard and screen reader navigation
strategy in Accessible Blockly aid people with visual impair-
ments navigate and understand block-based code?
Our contribution in this paper includes a prototype of an accessibil-
ity library, an analysis and results of the evaluation of the library
for navigating block-based code, and fnally, refections on how
to improve the accessibility of block-based programming environ-
ments.
2 RELATED WORK
Research on the accessibility of block-based programming to peo-
ple with visual impairments is a growing feld and dates back to
2015. In [6], the author points out that block-based programming
is inaccessible to people with visual impairments mainly because
of its visual and mouse-centric design. The author discusses design
considerations to improve the accessibility of BBPEs through key-
board and screen reader interactions. The authors in [4] further
present the need to make these systems accessible to people with
visual and motor impairments, given their increased popularity in
introductory computer science courses.
In [5], the authors found that students with visual impairments
lack access to mainstream block-based programming environments
and rely on hybrids environments such as Swift Playgrounds. They
also reported that students with visual impairments face several
challenges, including difculties in navigating and editing block-
based code. Due to the inaccessibility of BBPEs, TVIs often have to
resort to tangible alternatives to block-based programming [7] [13].
They, however, argue that these alternatives only support basic
functionalities, unlike the mainstream BBPEs [7].
Blocks4All[3] is an accessible touchscreen-based block-based
programming environment designed with students with visual im-
pairment in mind. It runs on the iPad and allows users to build and
run block-based code using touchscreen interactions and a screen
reader. Blocks4All was designed to address the accessibility barriers
in block-based programming environments, including accessing the
output, accessing elements in the coding environments, conveying
Aboubakar Mountapmbeme et al.
program structure and type information, and moving blocks [3] [5].
In [14], the authors discuss the extension of Blocks4All to support
functions and variables. Our work difers from this in that while
Blocks4All is a complete coding environment where users can build
and run block-based code, Accessible Blockly as a natural extension
to Blockly is a library that allows people to develop an accessible
block-based programming environment. It is an extension of the
Blockly library that adds keyboard and screen reader interactions.
One of the primary objectives of developing Accessible Blockly is
to provide a mechanism for making existing mainstream BBPEs
that run on Blockly accessible.
Das et al. [15] present an accessible solution that uses Blockly’s
Keybaord interaction mechanism and custom speech rather than a
screen reader. The authors generate custom speech that describes
the blocks in Blockly. Our work defers from this in two ways: 1)
Accessible Blockly supports screen reader output, and 2) we ofer
a novel keyboard interaction mechanism diferent from Blockly’s
keyboard interaction scheme.
The keyboard and the screen reader have been used as alterna-
tive interaction mechanisms for people with visual impairments
in text-based programming environments. Baker et al. developed
Structjumper [16], which helps blind programmers navigate code
in Eclipse integrated development environment (IDE) using key-
board shortcuts and screen reader output. CodeMirror-Blocks [17]
is similar to Structjumper and allows blind programmers to edit
and navigate text-based programs. It also ofers a block view of
text-based code where code sections are converted into visual ele-
ments to facilitate navigation. The bulk of research in improving
the accessibility of programming environments for people with
visual impairments has primarily focused on text-based program-
ming [18]. In [18], the authors discuss the evolution of research in
this feld and present a summary of all the accessibility tools that
have so far been developed.
3 SYSTEM DESIGN AND IMPLEMENTATION
Before getting into the details of the design, it is important to briefy
describe the morphology of a typical block and how block-based
programs are constructed. Blocks are special visual elements that
represent a particular programming construct. The shape of the
block depends on the type of programming construct that it repre-
sents and on its intended functionalities. We can generally identify
two types of blocks based on shape: non-container blocks and con-
tainer blocks. As their name implies, container blocks are blocks
that can accommodate other blocks within them. Examples include
blocks that represent control fow statements such as the “If” blocks
and “repeat” blocks (Figure 1 and Figure 2 a). A block will have
connection points (Figure 2). Connection points on a block allow
other blocks to be attached to it or they enable the block to connect
to other compatible blocks. The type and number of connection
points also depend on the intended functionality of a block. A typi-
cal block-based program grows vertically downwards and left to
right. Users select blocks from the toolbox and connect them on a
workspace to form a program (Figure 1). Constructing block-based
programs typically occurs via drag-and-drop operations using the
mouse.
Accessible Blockly: An Accessible Block-Based Programming Library for People with Visual Impairments
Figure 2: Block samples
Figure 3: Components of Accessible Blockly
Designing accessible BBPEs requires providing access to the
rich visual information characteristic of BBPEs and also enabling
an alternative interaction mechanism other than the mouse. We
established fve primary conditions while designing Accessible
Blockly:
1. It should be usable by both students with visual impairments
and sighted students alike. This includes collaboration be-
tween the two.
2. It should communicate information that is otherwise avail-
able visually in an alternate format such as speech and audio.
3. It should support keyboard interactions as an alternative to
the mouse.
4. The library should be easy to integrate into existing main-
stream block-based programming environments to enable
accessibility on these systems.
5. It should be easy to learn and use.
Figure 3 shows the main components in Accessible Blockly.
3.1 Blockly Library
The frst component is the Blockly library [8], on top of which acces-
sibility is being added. As stated earlier, Blockly is an open-source
library developed by Google for creating block-based programming
environments. Blockly became popular in 2016 after former US
president Barack Obama introduced CS for All using the platform
to code [19]. Blockly is built using JavaScript and runs on the web.
This makes it a good candidate for use since Accessibility guide-
lines for Web Applications are well established through the W3C
WAI group [20]. In addition, interactive coding platforms that use
ASSETS ’22, October 23–26, 2022, Athens, Greece
Blockly run on the web. With Blockly being open-source and widely
adopted, anyone is free to add to or extend the Blockly framework.
This also makes it suitable for our case. Another reason for using
Blockly is that it already supports complex syntax such as functions
and variables. Therefore, adding accessibility would allow users
to have access to these rich sets of features by default. In [7], the
authors report that TVIs fnd current accessible programming envi-
ronments to be too basic to use at some point because they only
contain basic coding concepts.
3.2 Keyboard Module
The second component is the Keyboard Module. Blockly, in its
initial design, only supported mouse interactions. The Keyboard is
an alternative to the mouse for people with visual impairments and
temporary disabilities. We designed Accessible Blockly to support
keyboard interactions with the following principal guidelines.
1. Allow a user to browse block categories and individual blocks
in the toolbox using keyboard inputs.
2. Enable drag-and-drop-like operations via keystrokes alone.
3. Mimic spatial navigation by allowing a user to navigate a
program top-down, left-right, and transverse in and out of
nested or container blocks.
We believe adhering to these principles is essential to produce one
platform that is usable by people with visual impairments and
sighted people alike. We defned a virtual cursor on the program-
ming environment controlled by keystrokes. This cursor can be on
the toolbox or the workspace at any time. When the focus is on
the toolbox, a user can open a fyout category by using keystrokes,
move the cursor between categories, and explore the blocks within
a category in the same way they would use a screen reader and a
keyboard to access HTML menus on the Web.
The keyboard module also allows users to perform drag-and-
drop-like interactions from the toolbox to the workspace. Prior
research [3] and our experimental work [11] have shown that there
can be more than one way to perform drag-and-drop-like operations
with a keyboard. We designed the drag-and-drop-like functions to
closely mimic the actions performed by a user using the mouse.
To move a block from the toolbox to the workspace, a user frst
selects a location on the workspace where they want to attach a
new block using the virtual cursor. Next, they go to the toolbox
and choose the desired block to be added, and then with a fnal
keystroke, the selected block is placed at the previously determined
location on the workspace. This method is commonly referred to
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 4: The virtual cursors in Accessible Blockly
as the location-frst select, then block select method[3]. To improve
the user experience and minimize confusion, once a user selects a
connection point on the workspace and goes to the toolbox, blocks
that are incompatible with the selected connection point are marked
as disabled and communicated to the user via the speech output.
The keyboard module operates in two modes: Navigate mode and
Edit mode. Navigate mode allows a user to move the virtual cursor
through the blocks or categories of blocks on the toolbox or to
navigate the blocks that make up a program on the workspace. The
navigation allows a user to explore a program on the workspace
spatially. The virtual cursor determines the current location of
focus of the user. The cursor can be moved vertically downwards or
upwards across blocks. It can also be moved horizontally between
blocks connected at the same level. Figure 4a shows the cursor as a
yellow outline around the “repeat 3 times” block. The cursor can
also be caused to move directly into the body of a container block,
such as an “if” block. This allows a user to continue exploration
within the body of the container block.
Edit mode is used to construct a new program or modify an
existing program on a workspace. In this mode, the virtual cursor is
used to select a connection point of a block on the workspace. The
virtual cursor only moves between the connection points of the
highlighted block. It allows a user to circle through the connection
points and select a connection point at which to attach a new block.
Upon choosing a connection point, the user goes back to the toolbox
and selects a compatible block that is automatically snapped at the
connection point on the workspace. The cursor is taken back to
the workspace onto the newly added block. Figure 4b shows the
virtual cursor as a black outline on the top connection of the “repeat
3 times” block.
The Keyboard Module uses the WASD keys as the primary keys
for interaction (Table 1). These keys were selected because the
WASD keys are common among gamers for navigation and can
allow users to interact with the system one-handed. We chose to
use keys other than the arrow keys for navigation to avoid conficts
with existing screen reader functionalities and other applications.
Aboubakar Mountapmbeme et al.
As an example of using the WASD keys to navigate the program,
consider Figure 4a, and suppose the virtual cursor is on the “if”
block. Pressing W would take the virtual cursor up to the “set count
to 7” group of blocks. Pressing S at this location would take the
cursor back down to the “if” block. Now if we are still on the “if”
block, pressing F takes the cursor inline or horizontally right to the
“count >= 5” block. The F-key causes the virtual cursor to move to
the frst property block of the current block. In Blockly terminology,
these blocks are referred to as value blocks. These are blocks that
cannot start a statement on their own and must exist as properties
to other blocks. Another example of using the F-key is if we are on
the “repeat 3 times” blocks as shown in Figure 4a. Pressing F would
take the virtual cursor to the number block “3”. Pressing A while at
the number block “3” would move the cursor back to the “repeat 3
times” block. Pressing D while the virtual cursor is on the “repeat 3
times” block would cause the virtual cursor to move into the body
of this container block to the “print 10” block. Pressing A causes
the virtual cursor to move out back to the “repeat 3 times” block.
3.3 Screen Reader Module
This component was designed to serve as an alternative output
channel to the visual channel. One of the ways in which people
with visual impairments interact with computers is through screen
readers. A screen reader is a program that reads out loud content on
the screen. The screen reader module converts and conveys infor-
mation otherwise only accessible visually into verbal descriptions.
By default, blocks which are visual elements are not accessible
through the screen reader. Unlike HTML images tag, for example,
which can easily be made accessibly by adding the alt text, blocks
are JavaScript drawings created at run time. Currently, there is no
standard mechanism like the alt text to make blocks easily accessi-
ble through the screen reader. When the virtual cursor is focused
on a block, the screen reader will output verbal descriptions based
on what the block represents. Text descriptions for each block were
carefully designed based on previous experiments [21] and using
previously established guidelines for describing computer programs
Accessible Blockly: An Accessible Block-Based Programming Library for People with Visual Impairments ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Keyboard navigation shortcuts for Accessible Blockly

Key Action
W Go vertically up to the previous block.
Go to the previous connection point (in Edit mode).
A Go horizontally left to the block on the left.
Go out of the body of a container block to the container block.
S Go vertically down to the next block.
Go to the next connection point (in Edit mode)
D Go into the body of the container block to the frst child block
E Toggle between Edit Mode and Navigate Mode
F Go horizontally right to the frst inline block or frst block that is a property of the current block.
J Jump to the frst block on the workspace
R Repeat current focus or location
C Open the toolbox
ENTER Add selected block to workspace
Delete Delete selected block
Ctrl + Z Undo Delete

in speech or audio form [22]. Navigating the toolbox also allows the
screen reader to read out the categories of blocks and each block
within a category. As the virtual cursor moves from one location
to another in the coding environment, the screen reader outputs
a corresponding speech that describes the focused element. For
example, if the focus is on the “repeat block” in Figure 4a, the user
will hear “repeat three times block” Another example is if the focus
is on the connection point shown in Figure 4b, the user will hear
“top connection.”
The speech for each block is designed to closely convey what
the block represents. An aria-label text template is defned for each
block type. The text label is updated at run time to refect the actual
state of the block. For example, there is an aria-label text “repeat
N times block” for the repeat block. For the example in Figure 4a,
the fnal text will be “repeat 3 times block” where N is replaced by
3. The screen reader will then output this information to the user.
Aria-labels at run time can also come from a combination of two
templates. Consider the “set count to 7” group of blocks in Figure 4.
The resulting aria-label “set count to 7” comes from the template
“set VARIABLE to block” and “NUMBER block.” “VARIABLE” is
replaced by “count” and “NUMBER” by “seven”.
Accessible Blockly has been tested with NVDA, VoiceOver, and
JAWS screen readers. With NVDA and JAWS, it works in focus
mode or pass-key through mode. One of the design constraints
was to maintain compatibility with existing applications while
adding accessibility to Blockly. Having the screen reader work
in these modes minimizes the chances of key conficts between
Accessible Blockly functions and other screen reader or browser
functionalities.
Blocky is mainly based on JavaScript. Our Keyboard and Screen
Reader modules are also implemented in JavaScript. The system
listens for keypresses and calls special event handling functions
to execute the requested actions and generate the speech output.
Our code is open source. The repository alongside the wiki are
located at [23]. This repository was forked from the ofcial Blockly
repository.
4 EXPERIMENT DESIGN
To answer the research question, we conducted a study in which 12
blind programmers accomplished navigation tasks using Accessible
Blockly and Blockly’s alternate navigation scheme as control. Our
institution’s IRB approved the study. This evaluation is the frst
phase in our evaluation series and involves navigation tasks only.
We decided to focus on the navigation task because:
1. This is the frst attempt to design an accessible block-based
programming library that uses the keyboard and screen
reader for interaction. The Blockly’s alternate navigation
scheme originated from work by the Blockly team with a
collaboration from our research group.
2. Navigation is crucial in writing code or building block-based
programs. Therefore, we wanted to fully evaluate the tool
for this activity before considering other coding tasks. Acces-
sibility of code navigation is commonly assessed in isolation
[16], [17].
Blockly [8] currently does not support screen readers. It only of-
fers the alternate keyboard interaction mechanism. To use the
Blockly’s alternate keyboard navigation as control in our experi-
ment, we replicated this navigation scheme in our codebase since
this already supports screen reader interactions. Unlike Accessible
Blockly, which ofers two navigation modes as discussed above,
Blockly’s alternate keyboard navigation, referred to as the “Default
Cursor” on the ofcial website, has only one mode of interaction. In
this single-mode interaction, the user uses the WASD keys to move
the virtual cursor between blocks and connection points of blocks
alike. Detail description of Blockly’s alternate keyboard navigation
scheme can be found at [24]. This work will act as one that eval-
uates both navigation schemes through a formal user study. We,
however, chose to use this as a control because it is the navigation
scheme that is shipped with the ofcial Blockly library.
4.1 Participants
A total of 12 people took part in the evaluation study. There were
ten males and two females. Participants were recruited via mailing
ASSETS ’22, October 23–26, 2022, Athens, Greece Aboubakar Mountapmbeme et al.

Table 2: Summary of participants’ background information

Gender Age Location Programming Screen Screen Reader Keyboard

experience Reader profciency Profciency
used
P1 Male 51 US More than 3 years NVDA Expert Expert
P2 Male 51 US 2 to 3 years NVDA Advanced Advanced
P3 Male 40 Europe More than 3 years NVDA Expert Advanced
P4 Female 56 US More than 3 years NVDA Expert Expert
P5 Male 28 India More than 3 years NVDA Advanced Expert
P6 Male 57 US More than 3 years NVDA Moderate Advanced
P7 Male 26 Middle East More than 3 years NVDA Advanced Expert
P8 Female 40 Europe less than 1 year NVDA Expert Expert
P9 Male 22 India less than 1 year NVDA Expert Expert
P10 Male 29 Europe More than 3 years NVDA Expert Expert
P11 Male 60 US More than 3 years JAWS Expert Expert
P12 Male 35 Hong Kong 2 to 3 years NVDA Advanced Advanced

lists, snowballing, and personal contacts. Participants had varying
levels of experience with programming ranging from professional
programmers to novices just learning how to code. Of the 12 partic-
ipants, ten reported being blind, and the remaining two said they
had no useful vision and relied on screen readers to interact with
computers. Two of the participants were regular braille users and
the rest used screen readers daily. The average age of the partici-
pants was 41 (SD = 13). Only one participant (P1) out of the twelve
had heard of block-based programming before the study. P1 had
tried Scratch, Blockly and MakeCode and reported these to be only
partially accessible. Every participant signed an informed consent
form before the study and received $50 Amazon gift card as com-
pensation after the study session. Table 2 shows the summary of
participants demographic data.
4.2 Set-up
All study sessions were conducted remotely via Zoom. Participants
were encouraged to use their regular screen reader settings and
other settings related to their daily computer usage. Participants
shared their screen and computer audio throughout the study ses-
sion, and the researchers watched as they completed the tasks.
Sessions were recorded for later analysis and insights.
4.3 Procedure
Participants were asked to provide demographic information in
a pre-study survey hosted on SurveyMonkey. This survey asked
participants to provide information regarding their level of vision,
programming experience, experience with block-based program-
ming, and with screen readers (Table 2).
The study was organized in two phases based on the type of
navigation tool used, i.e., Accessible Blockly vs. Blockly’s alternate
keyboard navigation. Each phase included a training task, two ex-
ercise tasks, and qualitative questions regarding the navigation
tool used in that phase. After completing both phases, participants
were asked open-ended questions regarding their overall experience
using both navigation schemes. Before starting each session, the
participants were briefy introduced to block-based programming.
Participants were briefed on the morphology of a block and how
block-based programs are built. Participants were also informed
about the usefulness of block-based programming and the impor-
tance of making block-based programming accessible to all.
An identical program was used for training purposes in each
phase. This program contained non-container blocks with assign-
ment operations and a container block (see Appendix A for details).
During training, participants were taught how to navigate a pro-
gram using the keyboard shortcuts for each navigation scheme.
Participants executed the keyboard commands following the in-
structions from the researchers. Participants were given time to
familiarize themselves with the navigation schemes after the in-
struction stage. Once participants felt comfortable using the navi-
gation scheme in each phase, we moved on to complete the exercise
tasks.
We designed two sets of similar programs for use during the
exercise tasks. Each group comprised of one simple block-based
program without any container block. This program performed
some basic arithmetic operations and updated the value of a named
variable at the end of the program. The second program in each set
had two container blocks: an “if” block and a “repeat” block. The
program also included two print blocks. The participants completed
the tasks on one set of programs using Accessible Blockly and
on another set of programs using Blockly’s alternate keyboard
navigation scheme. Appendix A contains details about the study
tasks. The link to the study tasks can be found at [25].
An efective navigation scheme should allow a user to get to
desired locations on the code while building an understanding of the
code and an awareness of their location within the program. This
should also ofer an acceptable navigation time. The two exercise
tasks were designed to evaluate Accessible Blockly on these criteria.
For each set of programs, the two exercise tasks included:
1. Determine the fnal value of a variable: Navigate the program
and determine the fnal value of the variable “X.” Here, X had
diferent names on each set of programs, and the arithmetic
operations were diferent on each set.
Accessible Blockly: An Accessible Block-Based Programming Library for People with Visual Impairments
2. a) What is the expected output? Navigate the program and
say what you believe will be printed on the screen if this
program was executed. This program contained print blocks
within container blocks.
b) Conditions: how will the program’s output in (2a) above
change if the initial value of a variable in the program was
set to a diferent value?
Task 2b was designed to test how well participants understood
what the code was doing after navigating the program in Task 2a.
The question asked participants to state how the program’s output
in Task 2a changes when the initial value of a variable used on the
“if” block was set to a new value. The modifcation was such that it
would negate the value of the Boolean condition and thus alter the
program’s output.
Participants were given 10 minutes to complete each task, except
for Task 2b, a follow-up question to Task 2a. A timer was started
after the researcher had fnished reading out the question to the
participant and when the participant indicated they were ready to
start. The timer was stopped the moment the participant said they
found an answer. All answers were recorded. Participants were
allowed to take down notes as they completed the tasks. Eleven
participants used NVDA, and one participant used JAWS during
the study sessions.
All tasks were scored. Task 1 was scored on a scale of 0-1. A
participant received 1 if they could determine the correct value
of the named variable. Task 2a was scored on a scale of 0-2. A
participant received full credit if they determined the correct output
of the program. They received 1 if their answer was incomplete but
partially correct. Task 2b was scored on a scale of 0-1. A participant
received 1 if they correctly answered the follow-up question and
0 otherwise. For the timed tasks, participants received a 0 if they
could not provide an answer within the allotted time.
At the end of each phase, participants were asked to rate their ex-
perience using the navigation scheme in that phase on a 5-point se-
mantically anchored scale. We asked the following three questions.
Similar questions have been used in studies evaluating accessible
navigation tools in text-based programming [16] [17].
1. On a scale of 1 to 5, 1 being very hard and 5 being very easy,
rate how easy the tasks were to complete.
2. On a scale of 1 to 5, 1 being very frustrating and 5 being
not frustrating at all, rate how frustrating the tasks were to
complete.
3. On a scale of 1 to 5, 1 being I had no idea where I was and 5
being I always knew where I was, rate how well you know
where you were in the code while completing the tasks.
After a participant had fnished both phases of the study, they were
asked to share their experiences with the two navigation schemes
and state if they had a preference for any and, if so, why. Partici-
pants were also asked to share suggestions regarding improving
the accessibility of block-based programming languages.
4.4 Design and Analysis
We designed a 2x2 within-subjects factorial design experiment with
factors of the program sets and the navigation scheme used. Our ex-
periment was modeled following similar experiments that evaluate
accessible navigation tools for people with visual impairments [16]
ASSETS ’22, October 23–26, 2022, Athens, Greece
[17]. Each participant completed two tasks using each navigation
scheme. The order of the navigation scheme encountered frst was
counterbalanced between participants. However, each navigation
scheme was fxed to a particular set of programs because the pro-
grams were designed to be very similar. How this design afects
the study results are discussed under limitations. Each participant
completed four tasks for the entire experiment for a total of 48
tasks. A single factor ANOVA was used to analyze tasks completion
time. Descriptive statistics were used to analyze the semantically
anchored scale data.
5 EVALUATION RESULTS
We analyzed the results and compared the efectiveness of the two
navigation schemes based on task completion time, the correctness
of answers to the tasks, and the participants’ experience. Overall,
participants were faster when completing the tasks with Accessible
Blocky. Participants also scored higher on average with Accessible
Blockly than with the alternate navigation scheme. Although most
participants found both navigation schemes equally accessible, they
tended to prefer accessible Blockly because they found it to be easy
and less frustrating to use.
5.1 Task completion time
Participants were faster when completing the tasks using the Acces-
sible Blockly navigation scheme, with an average task complement
time of 1 minute 30 seconds against 2 minute 55 seconds for the
alternate navigation scheme (Table 3). This diference was statisti-
cally signifcant (P-value 0.0035). We note that fxing one navigation
scheme to a particular set of programs throughout the study could
also have an infuence on the time diference and is addressed as a
limitation. Participants completed task 1 with Accessible Blocky in
an average of 1m30s vs. 2m44s for the alternate navigation scheme.
The same pattern was observed on task 2a with an average task
completion time of 1m30s for Accessible Blockly against 3m5s for
the alternate navigation scheme.
All participants completed each task within the allotted 10 min-
utes. Longer task completion times came from participants who had
less experience with programming. Figure 5 and Figure 6 compare
the task completion time on both navigation schemes by partici-
pants for Task 1 and Task 2a respectively.
5.2 Task Score
The maximum score possible for completing tasks correctly with
each navigation scheme was four. As shown on Table 4, partici-
pants were more successful in completing the tasks accurately with
Accessible Blockly (avg. score = 3.17) than with the alternate nav-
igation scheme (avg. score = 2.75). Looking at the score by tasks,
participants scored higher on Task 1 when using Accessible Blockly
(avg. 0.83) than with the alternate navigation scheme (avg. 0.67).
For both navigation schemes, some participants felt intimidated
by the mathematical expressions in Task 1 and did not take time
to properly evaluate the arithmetic expressions despite correctly
navigating the programs.
The performance in Task 2a was similar. On average, participants
scored higher with Accessible Blockly (avg. 1.58) than with the
alternate navigation scheme (avg. 1.33). Eight participants scored
ASSETS ’22, October 23–26, 2022, Athens, Greece Aboubakar Mountapmbeme et al.

Table 3: Task completion time

Accessible Blockly Nav Blockly Alternate Nav

Mean SD Mean SD
Task 1 -Time 1m30s 59s 2m44s 1m43s
Task 2a - Time 1m30s 37s 3m5s 2m30s
Avg. Time 1m30s 48s 2m55s 2m6s

Figure 5: Task 1 task completion time by participants for both navigation schemes

Figure 6: Task 2a task completion time by participants for both navigation schemes

full points on task 2a using Accessible Blockly against fve for the
alternate navigation scheme (Table 5). Participants who did not
get the full points in Task 2a when using either navigation scheme
forgot to explore the body of a container block in the program.
Some participants attributed the miss to their unfamiliarity with
the jargon on block-based programming. While it was easier to
determine whether a block was a container or non-container block
with Accessible Blockly, it was not the same with the alternate
navigation scheme. With the alternate navigation scheme, some
participants got confused by the connection points, which made
them lose track of the level of nesting in the code. Because to go
one level into the body of the container block using the alternate
Accessible Blockly: An Accessible Block-Based Programming Library for People with Visual Impairments
Table 4: Task score
Accessible Blockly Nav
Mean
Task1 - Score 0.83
Task 2a - Score 1.58
Task 2b Score 0.75
Avg. Score 3.17
Table 5: Number of participants who got full points per task
Accessible Blockly Nav
Task 1 10
Task 2a 8 5
Task 2b 9 9*
*After they were given hints and completed task 2a
navigation, you must go through the connection points, it was
difcult for participants to keep track of their level of nesting at
the same time while remembering the location of the virtual cursor
through the connection points. Participants who got partial points
in Task 2a thought one of the container blocks was nested inside the
other, which was not the case. Again, participants with experience
in programming did not have this confusion.
For both navigation schemes, participants who did not get Task
1 or Task 2a correct after the timer was stopped were given hints to
complete the task. All participants who failed at the frst attempt suc-
cessfully completed the tasks with recommendations. However, this
was not accounted for in the task score and task completion time
above. For Task 2b, participants on average scored equally using
Accessible Blockly (Avg. 0.75) and the alternate navigation scheme
(avg. 0.75). This suggests participants had a good understanding of
the programs they navigated. However, more participants success-
fully completed Task 2a on frst trail with the timer on when using
Accessible Blockly than with the alternate navigation scheme. The
three participants (P2, P8, and P11) who did not respond correctly
to Task 2b while using either navigation scheme reported they
had limited programming experience. In addition, P8 and P11 were
seasoned braille users and mentioned having a hard time keeping
up with the screen reader output. These could also have infuenced
their performance at the tasks.
While observing participants complete the tasks, we noticed that
all participants could explore all the blocks within the programs
and within the allotted time. No participant gave up in the middle
of completing a task. For each navigation scheme, all participants
could explore all the blocks on the programs for each task. We no-
ticed excitement on the part of some participants, which made them
impatient to refect on their answers properly. In this precipitation
to give an answer, some participants made mistakes. This was more
apparent with task 1 which involved arithmetic operations.
5.3 Participant Experience
Participants also shared their experience regarding the level of dif-
fculty of the tasks, their degree of frustration while completing the
ASSETS ’22, October 23–26, 2022, Athens, Greece
Blockly Alternate Nav
SD Mean SD
0.39 0.67 0.49
0.67 1.33 0.65
0.45 0.75* 0.45
1.19 2.75 1.22
Blockly Alternate Nav
8
task, and about their awareness of their location on the program
while completing the tasks. A higher number is better as it indicated
they found the tasks easy, not frustrating, and were more aware
of their location within the program. Overall, participants felt Ac-
cessible Blockly was easier and less frustrating to use. Participants
explained that it required few keystrokes to get to a desired loca-
tion, and the fact that they did not have to deal with the connection
points while navigating made their experience less frustrating. The
participants also thought they were more aware of their location
within the programs while navigating with Accessible Blockly.
The largest diference in the semantically anchored scale data
between the navigation schemes was on the level of difculty. Par-
ticipants overall rated Accessible Blockly to be easier to use (avg.
4.33) than the alternate navigation scheme (avg. 3.75) (Figure 7). We
note that all participants found both navigation schemes equally
accessible and usable for navigating block-based programs.
6 QUALITATIVE RESULTS
After participants had completed the tasks with Accessible Blockly
and the alternate navigation scheme, we asked participants open-
ended questions regarding their experience learning and using
both navigation schemes. Participants were also asked whether
they preferred any of the two navigation schemes. Participants
were also allowed to provide suggestions regarding improving the
usability of the navigation schemes and the accessibility of block-
based programming in general.
Multiple steps of inductive analysis were applied to the tran-
scripts from each session. Codes were identifed from each tran-
script through multiple coding rounds. Using the afnity diagram-
ming technique, these codes were grouped into themes. Several
themes were identifed and are discussed below.
6.1 Easy to Use and to Learn.
Analyses of the open-ended questions revealed that participants
found Accessible Blockly easy to use and learn. Nine out of the
12 participants explicitly stated that accessible Blockly was easier
to use to navigate the programs. The participants attributed this
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 7: Average score across participants on the semantically anchored questions
ease of use to the fact that you do not have to worry about the
connection points while navigating the program with accessible
Blockly:
“I prefer the navigation two [Accessible Blockly]. Be-
cause I think it’s easiest to navigate because you don’t
have to worry about the connection. So, I prefer the
second one, phase two one.” (P3)
“Obviously, this [Accessible Blockly] is quite a bit
more some, like simplifed because you don’t have all
the connection points to worry about so this is just
simpler.” (P10)
On the other hand, fve participants found the alternate naviga-
tion scheme challenging and less intuitive. Participants explained
that navigating the connection points at the same time as blocks
made learning and using Blockly’s alternate navigation scheme less
intuitive.
“It [Blockly’s alternate navigation scheme] is easy not
very easy. I mean here, the training right, how to use
this navigation is much more important because it’s not
very intuitive. It’s a bit difcult. So yeah, a lot of things
to kind of understand properly.” (P9)
Other participants thought the alternate navigation scheme has a
steep learning curve and that its ease of use could improve with
experience. Because each block has its morphology and type of
connection points, it can be overwhelming for the users at the
beginning.
I guess my umm I really feel like I wanna make it com-
plicated but that’s because I’m just learning it [Blockly’s
alternate navigation scheme]. I think you know I think
after time this would be easier like I said because it ac-
tually goes left and right and I would be able to explain
Aboubakar Mountapmbeme et al.
it better to other people you know like what I was doing.
In one sense it’s more complicated to learn, I think. (P1)
6.2 Shorter Navigation Time
The quantitative results already showed that Accessible Blockly was
faster for navigating the programs. Additionally, six participants
also mentioned in the open-ended discussions that they found
Accessible Blockly to be faster when navigating the programs. The
participants said they could quickly get to their target on the block-
based code because they had fewer keystrokes to execute. This is
because Accessible Blockly in navigation mode does not go through
the connection points, saving extra keystrokes from the user.
“. . . I mean the other thing that I thought was diferent
I thought the second one [Accessible Blockly] felt like
it had a little less steps may be cause like you know
when going to the hollow block before we did D and
for this particular question there was one like one of 2
choices I didn’t’ really have to go deeper so in that aspect
it was kind of like less layers to go into in the second
session whereas in the frst session [Blockly alternate
navigation scheme] when we did the hollow blocks it
seemed like there was more like cause I had to go into
the do ummm connection and had to go in there and
then come back out and come back out.” (P2)
In addition to being faster to navigate, participants thought acces-
sible Blockly provided quick review of the programs.
“Phase 2 (Accessible Blockly) in terms of reviewing a program is
easier because it allows you to navigate the programs quickly. You
don’t have to go into the details of the connection points as with the
Phase 1 [Blockly alternate navigation scheme].” (P6)
On the other hand, participants thought using the alternate naviga-
tion scheme to read the programs made it longer to navigate the
Accessible Blockly: An Accessible Block-Based Programming Library for People with Visual Impairments
code. This longer navigation time is also evident in the quantitative
data presented above.
“Getting into the diferent connections made it a
bit longer than the frst one, phase one [Accessible
Blockly].” (P3)
6.3 Less Cognitively Demanding
While refecting on their experiences, participants also found that in
contrast to Accessible Blockly, the alternate navigation scheme was
more cognitively demanding. 6 out of the 12 participants explained
that using the alternate navigation scheme to read the programs
increased their memory constraints and required extra attention.
“umm I think it could umm I mean I know when I was
doing it, it [Blockly alternate navigation] required a
little bit more time to go in there and then I had to
focus a little bit more on where am I at, coming back
out you know like like ummm It felt like the second
one [Accessible blockly] I didn’t have to worry, . . . as
soon as I was in, I was back out. Whereas the frst time
[alternate navigation scheme] I was in then I was in
then I was like okay when I came out where am I at
when am I gonna come out again.” (P2)
Having to remember the connection points while navigating to get
to a target location was mostly cited as the reason for the increased
cognitive load. Participants felt that the alternate navigation scheme
exposes too much information during navigation, making it more
challenging to explore the code. Exposing connection points during
navigation increased navigation time and had participants retain
more in their memory.
“In the phase one [Blockly’s alternate navigation
scheme] you have to keep track of the connection points
which you don’t need if you are not editing the program.
I don’t know how it might be in terms of editing but the
phase 2 [Accessible Blockly] was straight forward.” (P6)
Although some participants made remarks regarding the increased
memory requirements, they thought the detailed information pro-
vided could be more helpful when navigating complex programs.
P5, while providing his rating for the level of frustration with the
alternate navigation scheme noted:
“I gave 4.5 because I had to process bit more of an in-
formation but I believe like to thoroughly understand
a complex program I might need all this information.”
(P5)
On the other hand, fve participants thought it was the opposite
when using Accessible Blockly because they did not have to worry
about the connection points when reading the programs.
“... Because it [Accessible Blockly] just gives you uhh
enough information you want to understand the pro-
gram. uhh it did not give you more details which can
make things a bit uhhh like which can increase the cog-
nitive load. The things you need to intake are more in
the second phase [Blockly alternate navigation scheme].
So, in frst phase [Accessible Blockly] like the cognitive
load was very much less, and so as frustration and un-
derstanding would be a bit more. So, for a novice or for
ASSETS ’22, October 23–26, 2022, Athens, Greece
uhh like student uhhmm learning frst phase was very
much good.” (P5)
One participant mentioned it was easier to memorize the navigation
keys with Accessible Blockly than with the alternate navigation
scheme.
“I have a preference for navigation scheme applied to
phase one [Accessible Blockly], because that’s much
more easier. Okay, it’s only few keys. So that’s much
more easier to memorize because being a blind individ-
ual, always you need to memorize key shortcuts. So it’s,
I mean, it should be as simple as that. So, in that case, I
have a preference with phase one [Accessible Blockly].”
(P9)
6.4 Suitable for Novices
As participants thought Accessible Blockly was straightforward,
faster, and less cognitively demanding, they also stated that it
seemed more suitable for novices and appropriate for simple pro-
grams. P5 explicitly mentions this in the previous paragraph. One
of the primary objectives of block-based programming is to make
learning programming by novices easier by removing the heavy
syntax imposed by text-based languages [26] [13]. Looking at it
from this perspective further strengthens the observations made
by participants. We also observed from the qualitative data that
participants with less programming experience performed better
with Accessible Blockly than with the alternate navigation scheme,
suggesting it was easier for them to learn and use.
On the other hand, participants who had more experience with
programming thought the alternate navigation scheme was more
appropriate for navigating complex programs because it provided
more information which would give them more control especially
when it comes to editing.
“ So, for reading the program it was easier with the
way it worked in phase one [Accessible Blockly], umm I
would imagine when it came to editing you would need
more control like what you have in phase 2 [Blockly’s
alternate navigation scheme].” (P3)
6.5 Participant Preference
Participants were explicitly asked if they had a preference for one
of the navigation schemes over the other and their responses were
interesting. Five participants explicitly stated that they preferred Ac-
cessible Blockly over Blockly’s alternate navigation scheme. Three
participants said they liked both navigation schemes citing various
use cases in which they fnd both to be valuable and appropriate.
The most common reason for preferring Accessible Blockly was its
ease of use and design to favor novices. Blockly’s alternate naviga-
tion scheme on the other hand was mostly preferred by experienced
programmers who stated it was because it provided more informa-
tion that might be valuable when navigating complex programs.
One participant, P10, stated they preferred the alternate navigation
scheme over accessible Blockly. What was more intriguing was
the three participants who said they liked none of the navigation
schemes. P12 explicitly stated this:
ASSETS ’22, October 23–26, 2022, Athens, Greece
“I don’t have a preference for any. I feel they end up
trying to map the WASD keys to the Arrow keys.” (P12)
Overall, we observed that both navigation schemes are equally us-
able and accessible for navigating and understanding block-based
programs, with Accessible Blockly being preferred over the alter-
nate navigation scheme because of its ease of use, faster navigation
time, and its tendency to favor novices.
7 DISCUSSION
7.1 Participants with Programming Experience
and Experience Using Screen Readers
Perform Better.
The individual reports and quantitative data show that participants
with good programming experience performed best, having the
fastest task completion time and highest score with both navigation
schemes. These participants also found the tasks to be easier to
complete. This was the contrast with those who had little or no
programming experience. The participants with limited program-
ming experience scored low on the tasks. When watching the video
recordings of participants who scored low on the tasks, we noticed
that most missed exploring the body of the container blocks or
control fow blocks in Task 2. Five participants missed exploring
the body of container blocks in Task 2a, and this was more com-
mon when they completed the task with the alternate navigation
scheme. When asked why they missed, some cited their limited
programming knowledge explaining that they did not know these
blocks had blocks nested within them. This was the case for P2 and
P8.
It could also be that these participants had not formed an accu-
rate mental model of block-based code and their conceptualization
of block-based code was still infuenced by their text-based program
reading strategy. At least the discussion with three participants sug-
gested this. P4, who missed exploring the container blocks in both
phase 1 and phase 2 said her familiarity with procedural languages
infuenced her thought process.
“Yeah, I think it’s just my training as you know umm
because normally in a procedural language the thing to
be repeated would be on the next line and I just haven’t
gotten used to looking inside the block for what should
be repeated.” (P4)
We believe that in addition to adding accessibility to block-based
programming, training students with visual impairment to form an
accurate mental model of how block-based code works would be
essential in increasing their success in using block-based program-
ming environments. Training students to use screen readers and
other accessibility technologies would also improve their perfor-
mance at block-based coding[5].
7.2 Enthusiasm Regarding both Methods and on
the Accessibility of BBPEs in General
We observed that almost all participants (9 out of 12) were enthu-
siastic as they used both navigation methods to complete tasks
and welcomed the idea of adding accessibility to block-based pro-
gramming environments. Participants displayed positive emotions
and frequently smiled as they completed the tasks and refected
Aboubakar Mountapmbeme et al.
upon their experience. Most participants expressed interest in using
the navigation schemes, and some even requested more complex
programs as they were curious and eager to see how they work
on larger programs. Overall, it was a feeling of enthusiasm, satis-
faction, and enjoyment. For the participants who had not heard
of block-based programming before, it felt like a new and exciting
experience. The only participant familiar with the accessibility is-
sues in mainstream BBPEs [5], [7] stated they were happy to see
eforts being made to add accessibility to these systems. P1, who
also teaches students with visual impairments how to code and has
tried some mainstream BBPEs stated:
“I really like this though you guys have done [. . .]. This
is much better than the Google outline method that they
had. Umm what was it? I don’t know how many years
ago? Ummm.” (P1)
P1 was referring to a text-based version of Blockly that was de-
signed by the Blockly team to be accessible to people with visual
impairments. The problem with this was that it was a separate sys-
tem and thus not suitable in situations where students with visual
impairments and sighted students work together.
7.3 Suggestions to Improve the Accessibility of
Block-Based Code
As discussed in the previous section, all participants in the study
expressed enthusiasm and gave feedback regarding how we could
improve the navigation strategy and the accessibility of block-based
code in general. We examined all the suggestions made and noticed
that they were similar across participants. Most of the recommen-
dations had to do with the feedback provided by the screen readers.
All participants found the keyboard navigation strategies accessible.
However, Participants thought the verbal descriptions provided
by the screen reader could be improved and enriched with more
feedback regarding the actions taking place on the coding envi-
ronment. Participants suggested providing hints about container
blocks. Participants thought letting the screen reader alert them
that a block is a container block would improve their success at
navigation and reading block-based code.
Participants also requested a quick review mode that would allow
one to listen to the description of the whole program or part of a
program without navigating the program. This would give the user
a head start before they begin exploring the individual blocks on
the program.
We note here that adequate feedback from the programming
environment is crucial to an efective and fully accessible program-
ming environment for people with visual impairments. Therefore,
researchers and practitioners should explore how to increase the
amount of feedback a user can obtain from such interactive visual
environments through alternate interaction channels. Sound and
auditory cues have been used for similar purposes in text-based
programming environments [22] [27] [28].
Accessible Blockly: An Accessible Block-Based Programming Library for People with Visual Impairments
8 LIMITATIONS
One obvious limitation of this work is the number of participants
who took part in the study. Due to the difculty in recruiting par-
ticipants, we could only evaluate Accessible Blockly with 12 partic-
ipants. Although 12 is high when compared to similar studies [3]
[16].
As stated before, each navigation scheme was fxed to one set of
programs throughout the study. This might have also infuenced the
quantitative results and their statistical signifcance. However, the
programs were designed to be very similar in length and content and
we believe this confguration had negligible infuence on the results.
Additionally, during the open-ended discussions most participants
felt Accessible Blockly was faster for navigation.
9 FUTURE WORK
This study mainly evaluated Accessibly Blockly for navigating and
reading block-based code. This was the frst study in our series of
evaluation studies. Part of the next steps involves looking at how
Accessible Blockly allows people with visual impairments to create
and edit block-based code. Participants were very eager about this
and almost all of them wanted to create block-based code.
Updating the screen reader output by improving the verbal de-
scriptions and increasing the amount and type of information a user
would get through the screen reader is also part of our future work.
Participants gave suggestions regarding the generated speech, and
we believe we can address these in our future iterations. Audio cues
have been used to increase the amount of feedback and information
that users get when navigating text-based programs [22] [28]. We
plan to explore these mechanisms and other ways to improve the
feedback generated by the screen reader.
10 CONCLUSION
We have presented a prototype of an accessible block-based pro-
gramming library that uses the keyboard and screen reader for
interaction. This library is a modifcation of the Blockly library
used to build most mainstream block-based programming envi-
ronments. The library was designed to improve the accessibility
of block-based programming environments for people with visual
impairments. Using the library, a user can navigate, create, and
edit block-based code using the screen reader and keyboard only.
This paper also reported a user study evaluating the library on
code navigation tasks. The results show that people can efectively
navigate block-based code with a keyboard and screen reader. Par-
ticipants in the study found Accessible Blockly to be fast, easy to
use, and less frustrating. They also showed enthusiasm about this
initiative to improve the accessibility of block-based programming
environments. Participants also expressed interest in future studies
to evaluate the library for code creation and editing.
ACKNOWLEDGMENTS
We thank all participants who took part in the study and Google’s
Blockly team for their collaboration.
REFERENCES
[1] D. Weintrop, “Block-based programming in computer science education,” Com-
mun. ACM, vol. 62, no. 8, pp. 22–25, Jul. 2019, doi: 10.1145/3341221.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[2] D. Weintrop and U. Wilensky, “Comparing block-based and text-based program-
ming in high school computer science classrooms,” ACM Trans. Comput. Educ.,
vol. 18, no. 1, pp. 1–25, 2017.
[3] L. R. Milne and R. E. Ladner, “Blocks4All: overcoming accessibility barriers to
blocks programming for children with visual impairments,” in Proceedings of the
2018 CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–10.
[4] L. R. Milne and R. E. Ladner, “Position: Accessible Block-Based Programming:
Why and How,” in Proceedings - 2019 IEEE Blocks and Beyond Workshop, B and B
2019, Oct. 2019, pp. 19–22, doi: 10.1109/BB48857.2019.8941230.
[5] A. Mountapmbeme and S. Ludi, “Investigating Challenges Faced by Learners with
Visual Impairments using Block-Based Programming / Hybrid Environments,” in
The 22nd International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’20), 2020.
[6] S. Ludi, “Position paper: Towards making block-based programming accessible
for blind users,” in 2015 IEEE Blocks and Beyond Workshop (Blocks and Beyond),
2015, pp. 67–69.
[7] A. Mountapmbeme and S. Ludi, “How Teachers of the Visually Impaired Compen-
sate with the Absence of Accessible Block-Based Languages; How Teachers of the
Visually Impaired Compensate with the Absence of Accessible Block-Based Lan-
guages,” 23rd Int. ACM SIGACCESS Conf. Comput. Access., doi: 10.1145/3441852.
[8] “Blockly | Google Developers.” https://developers.google.com/blockly (accessed
Apr. 09, 2022).
[9] N. Fraser, “Ten things we’ve learned from Blockly,” Proc. - 2015 IEEE
Blocks Beyond Work. Blocks Beyond 2015, pp. 49–50, Dec. 2015, doi:
10.1109/BLOCKS.2015.7369000.
[10] N. C. C. Brown, J. Mönig, A. Bau, and D. Weintrop, “Panel: Future directions of
block-based programming,” in Proceedings of the 47th ACM Technical Symposium
on Computing Science Education, 2016, pp. 315–316.
[11] S. Ludi and M. Spencer, “Design Considerations to Increase Block-based Language
Accessibility for Blind Programmers Via Blockly,” J. Vis. Lang. Sentient Syst., vol.
3, no. 1, pp. 119–124, 2017.
[12] “Accessible Rich Internet Applications (WAI-ARIA) 1.2.” https://www.w3.org/TR/
wai-aria-1.2/ (accessed Sep. 09, 2020).
[13] A. C. Pires, F. Rocha, A. J. De Barros Neto, H. Simão, H. Nicolau, and T. Guer-
reiro, “Exploring accessible programming with educators and visually impaired
children,” in Proceedings of the Interaction Design and Children Conference, IDC
2020, Jun. 2020, pp. 148–160, doi: 10.1145/3392063.3394437.
[14] J. S. Y. Ong, N. A. O. Amoah, A. E. Garrett-Engele, M. I. Page, K. R. McCarthy,
and L. R. Milne, “Expanding Blocks4All with Variables and Functions,” in The
21st International ACM SIGACCESS Conference on Computers and Accessibility,
2019, pp. 645–647.
[15] M. Das, D. Marghitu, M. Mandala, and A. Howard, “Accessible block-based pro-
gramming for k-12 students who are blind or low vision,” Lect. Notes Comput. Sci.
(including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 12769
LNCS, pp. 52–61, 2021, doi: 10.1007/978-3-030-78095-1_5/FIGURES/7.
[16] C. M. Baker, L. R. Milne, and R. E. Ladner, “Structjumper: A tool to help blind
programmers navigate and understand the structure of code,” in Proceedings of
the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015,
pp. 3043–3052.
[17] E. Schanzer, S. Bahram, and S. Krishnamurthi, “Accessible AST-Based Program-
ming for Visually-Impaired Programmers,” in Proceedings of the 50th ACM Tech-
nical Symposium on Computer Science Education, 2019, pp. 773–779.
[18] Mountapmbeme Aboubakar, Okafor Obianuju, and Ludi Stephanie, “Addressing
Accessibility Barriers in Programming for People with Visual Impairments: A
Literature Review,” ACM Trans. Access. Comput., vol. 15, no. 1, pp. 1–26, Mar. 2022,
doi: 10.1145/3507469.
[19] “Computer Science For All | whitehouse.gov.” https://obamawhitehouse.archives.
gov/blog/2016/01/30/computer-science-all (accessed Apr. 13, 2022).
[20] “WAI-ARIA Overview | Web Accessibility Initiative (WAI) | W3C.” https://www.
w3.org/WAI/standards-guidelines/aria/ (accessed Apr. 09, 2022).
[21] S. Ludi, J. Simpson, and W. Merchant, “Exploration of the use of auditory cues
in code comprehension and navigation for individuals with visual impairments
in a visual programming environment,” in ASSETS 2016 - Proceedings of the 18th
International ACM SIGACCESS Conference on Computers and Accessibility, Oct.
2016, pp. 279–280, doi: 10.1145/2982142.2982206.
[22] A. Stefk, C. Hundhausen, and R. Patterson, “An empirical investigation into the
design of auditory cues to enhance computer program comprehension,” Int. J.
Hum. Comput. Stud., vol. 69, no. 12, pp. 820–838, 2011.
[23] “RITAccess/blockly: The web-based visual programming editor.” https://github.
com/RITAccess/blockly (accessed Jul. 06, 2022).
[24] “@blockly/keyboard-navigation Demo.” https://google.github.io/blockly-
samples/plugins/keyboard-navigation/test/ (accessed Apr. 09, 2022).
[25] “Accessible block-based programs study.” https://mountapmbeme.com/study/
home.html (accessed Jul. 07, 2022).
[26] D. Weintrop and U. Wilensky, “To block or not to block, that is the question:
students’ perceptions of blocks-based programming,” in Proceedings of the 14th
international conference on interaction design and children, 2015, pp. 199–208.
[27] S. A. Brewster, “Using nonspeech sounds to provide navigation cues,” ACM Trans.
Comput. Interact., vol. 5, no. 3, pp. 224–259, Sep. 1998, doi: 10.1145/292834.292839.
ASSETS ’22, October 23–26, 2022, Athens, Greece Aboubakar Mountapmbeme et al.

[28] J. Hutchinson and O. Metatla, “An initial investigation into non-visual code struc- A.1.4 Task 2b. How will the output of the program above change
ture overviews through speech, non-speech and spearcons,” Conf. Hum. Factors if the value of count was initially set to 2?
Comput. Syst. - Proc., vol. 2018-April, Apr. 2018, doi: 10.1145/3170427.3188696.

A.2 Phase 2: Using Blockly’s Alternate

APPENDICES
Navigation scheme
A THE STUDY TASKS A.2.1 Training. The following program will be used to teach the
A.1 Phase 1: Using Accessible Blockly Keyboard basic navigation keys for about 15 mins
Navigation Scheme https://mountapmbeme.com/googles/blockly/
accessibleblockly/study/training.html
A.1.1 Training Task. The following program will be used to teach
the basic navigation keys for about 15 mins A.2.2 Task 1. What is the value of the variable named count after
https://mountapmbeme.com/study/accessible/blockly/ the following program is executed? Use the keyboard and Screen
accessibleblockly/study/training.html Reader to read the program.
https://mountapmbeme.com/googles/blockly/
accessibleblockly/study/taska.html
A.2.3 Task 2a. What is the output of the following program? Use
the keyboard and Screen Reader to navigate and read the program.
https://mountapmbeme.com/googles/blockly/
accessibleblockly/study/taskb.html
A.2.4 Task 2b. How will the output of the program above change
if the value of temp was initially set to 20?

Screenshot 1: Training Task

A.1.2 Task 1. What is the value of the variable named result after
the following program is executed? Use the keyboard and Screen
Reader to read the program.
https://mountapmbeme.com/study/accessible/blockly/
accessibleblockly/study/task1.html

Screenshot 3: Task 2a with Accessible Blockly

Screenshot 2: Task 1 with Accessible Blockly

A.1.3 Task 2a. What is the output of the following program? Use
the keyboard and Screen Reader to navigate and read the program.
https://mountapmbeme.com/study/accessible/blockly/
accessibleblockly/study/task2.html
Screenshot 4: Training Task
Accessible Blockly: An Accessible Block-Based Programming Library for People with Visual Impairments ASSETS ’22, October 23–26, 2022, Athens, Greece

Screenshot 5: Task 1 with Blockly Alternate Navigation Scheme

Screenshot 6: Task 2a with Blockly Alternate Navigation Scheme

 CodeWalk: Facilitating Shared Awareness in Mixed-Ability
Collaborative Sofware Development
Venkatesh Potluri∗ Maulishree Pandey∗
vpotluri@cs.washington.edu maupande@umich.edu
Paul G. Allen School of Computer University of Michigan School of
Science and Engineering at University Information
of Washington Ann Arbor, Michigan, USA
Seattle, Washington, USA
Michael Barnett
mbarnett@microsoft.com
Microsoft Research
Redmond, Washington, USA
ABSTRACT
COVID-19 accelerated the trend toward remote software devel-
opment, increasing the need for tightly-coupled synchronous col-
laboration. Existing tools and practices impose high coordination
overhead on blind or visually impaired (BVI) developers, imped-
ing their abilities to collaborate efectively, compromising their
agency, and limiting their contribution. To make remote collab-
oration more accessible, we created CodeWalk, a set of features
added to Microsoft’s Live Share VS Code extension, for synchronous
code review and refactoring. We chose design criteria to ease the
coordination burden felt by BVI developers by conveying sighted
colleagues’ navigation and edit actions via sound efects and speech.
We evaluated our design in a within-subjects experiment with 10
BVI developers. Our results show that CodeWalk streamlines the
dialogue required to refer to shared workspace locations, enabling
participants to spend more time contributing to coding tasks. This
design ofers a path towards enabling BVI and sighted developers
to collaborate on more equal terms.
CCS CONCEPTS
• Social and professional topics → People with disabilities;
• Software and its engineering → Collaboration in soft-
ware development; • Human-centered computing → Syn-
chronous editors.
KEYWORDS
software developers, collaboration, blind or visually impaired, ac-
cessibility, sound efects, workspace awareness
∗ Both
authors contributed equally to this research.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544812
Andrew Begel
abegel@andrewbegel.com
Microsoft Research
Redmond, Washington, USA
Scott Reitherman
scott.reitherman@gmail.com
Microsoft Research
Redmond, Washington, USA
ACM Reference Format:
Venkatesh Potluri, Maulishree Pandey, Andrew Begel, Michael Barnett,
and Scott Reitherman. 2022. CodeWalk: Facilitating Shared Awareness in
Mixed-Ability Collaborative Software Development. In The 24th Interna-
tional ACM SIGACCESS Conference on Computers and Accessibility (ASSETS
’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 16 pages.
https://doi.org/10.1145/3517428.3544812
1 INTRODUCTION
Synchronous software engineering activities like pair program-
ming and code walkthroughs are useful for developers to share
knowledge, improve and refactor the source code, and debug the
code together. Developers have to remain closely synced to achieve
efective collaboration and communication in these activities. If a
developer moves to a new location in the code i.e. line, function or
fle, their collaborator should follow them immediately; real-time
edits should become apparent right away to enable quick feedback.
When collocated, sighted developers work together on one sys-
tem to observe and discuss the source code without expending
additional efort to stay on the same page. However, referencing a
collaborator’s screen is inaccessible for blind or visually impaired
(BVI) developers, often requiring them to drive the collaboration
on their computers [53].
This screen inaccessibility is magnifed in remote synchronous
collaboration. Developers typically either use screen shares or inte-
grated development environments (IDEs) with integrated collabora-
tion support (e.g. VS Code, JetBrains, Floobits, CodeTogether, etc.) to
work synchronously (see Figure 1). These approaches assume that
everyone can see their screens [53, 73]. However, BVI developers
cannot access the screen share video or the visual awareness cues
in IDEs through assistive technologies such as screen readers. They
have to constantly request that their sighted colleagues speak code
locations, such as line numbers, functions, fle names, etc., out loud,
in order to stay in sync. Much like collocated collaboration, BVI
developers end up driving the activity. Sometimes, they even hand
of their computer’s control to sighted colleagues in refactoring and
debugging tasks, which reduces their own agency.
That task of providing accessible awareness information lies at
the heart of facilitating efective remote collaborations in mixed-
ability contexts [25, 34, 53]. Research has begun to explore making
ASSETS ’22, October 23–26, 2022, Athens, Greece
Following anonymous_2
anonymous_2
Figure 1: VS Code is an IDE that ofers integrated collaboration support through its Live Share extension. Live Share enables
developers to work together on source code through document sharing and co-editing in their respective IDEs. It represents
collaborators’ location and selection in the source code through colorful cursors.
shared workspaces accessible to BVI users [18, 38]. However, these
solutions are intended for general-purpose document co-editing;
they do not cater to the unique needs of software engineering tasks
like pair programming and code walkthroughs.
Prior to our work, no programming environment with accessible,
remote, synchronous co-editing support was publicly available. We
have created and released such an environment, in accordance
with four design criteria (see §2.4), which include maintaining the
agency of BVI developers and reducing their burden to drive the
collaboration.
In this paper, we present CodeWalk, a set of features added to Mi-
crosoft’s Live Share VS Code extension1 (available to all Live Share
users since November 2021), with support for remote, synchronous
code review and refactoring tasks. Our design derives from an in-
vestigation of relevant literature in remote collaboration and a set
of formative design activities led by a BVI developer and researcher
on the team. During our design process, we compared techniques
for capturing a collaborator’s navigation, editing, and referential
(i.e., pointing at or highlighting parts of the code) activities and
presenting them to BVI users using a combination of sound efects
and speech (see §4.1). We evaluated CodeWalk in a within-subjects
controlled experiment involving 10 BVI professional developers (see
§5). Our results show that CodeWalk increased study participants’
awareness of their collaborator’s actions and reduced the coordi-
nation overhead required to sync on code locations. Participants
strongly preferred CodeWalk over the baseline — the unextended
version of VS Code with Live Share which provides awareness cues
visually (see §6).
Our work is an end-to-end demonstration of how to improve the
accessibility of an IDE’s remote collaboration features. We make
the following contributions to the HCI, accessibility, and software
engineering design communities.
(1) A design for supporting tightly-coupled synchronous pro-
gramming activities (see §3 and §4).
(2) CodeWalk, an implementation of a set of features added to
Microsoft’s Live Share VS Code extension that makes syn-
chronous programming tasks accessible to BVI developers
(see §4).
(3) Validation of our design’s capability to increase shared
awareness and facilitate efcient synchronization during
1 https://docs.microsoft.com/en-us/visualstudio/liveshare/use/enable-accessibility-
features-visual-studio-code
Potluri and Pandey, et al.
remote collaboration, meeting our design criteria (see §5 and
§6).
The COVID-19 pandemic has exacerbated the need for collabo-
rative programming environments that can enable BVI developers,
one of the largest physical disability groups of software develop-
ers [67], to participate in remote work at par with their sighted
peers. CodeWalk addresses this timely need and provides a foun-
dation for future software engineering tools to facilitate accessible
collaborations.
2 BACKGROUND AND RELATED WORK
2.1 Awareness for Sighted People in Remote
Collaborations
Groupware is only efective when it supports collaboration across
time and space constraints in a shared workspace [32]. Buxton
describes a shared workspace in terms of (1) person space [14], (2)
task space [14], and (3) reference space [13]. Person space ofers a
strong sense of copresence with remote collaborators. For instance,
teleconferencing platforms combine video, audio, and even chat to
convey facial expressions, gestures, and spoken messages. Sharing
the task space refers to being copresent in the context of the task
itself. In collaborative software development activities, the shared
source code forms the task space. Reference space is where the per-
son and task overlap [13], allowing remote participants to gesture
and point to reference one another as well as the task at hand. An
example is using text highlighting during screen shares to direct
collaborators’ attention to specifc details.
Dourish and Bellotti identifed two approaches to present aware-
ness information in shared workspaces — active and passive [20].
Active approaches include role assignment (e.g. owner, reviewer,
editor, etc), audit trails, annotations, and messaging. Active mech-
anisms require explicit action on collaborators’ part (e.g. leaving
a comment on the shared artifact like document or source code).
Conversely, conveying a collaborator’s whereabouts and edits au-
tomatically in real-time is a passive approach — collaborators do
not have to make explicit eforts to communicate their actions.
Scholars and practitioners have blended diferent kinds of shared
workspaces with active and passive approaches to communicate
awareness information in remote synchronous software develop-
ment. Consider the example of real-time co-editing of source code.
A primary concern is that the shared activity should not introduce
CodeWalk: Facilitating Shared Awareness in Mixed-Ability Collaborative Sofware Development
bugs, preventing the code from successfully compiling [23]. IDE
plugins like FASTDash [7] and Syde [26] summarize the real-time
activity of collaborators to help them avoid editing conficts. They
also allow developers to leave annotations to inform collaborators
of their actions. Systems like AtCoPE [21] and Collabode [23] allow
programmers to concurrently edit the source code, enabling collab-
oration in a shared task space. CodePilot [76] extends the activities
supported in the shared task space to allow collaboration across
the software development process — editing, testing, debugging,
and even version control.
Real-time activities like pair programming and code walk-
throughs impose an additional requirement on developers to remain
closely coordinated [74]. These activities often rely on explicit role
assignment. In one common form of pair programming, one pro-
grammer drives (leader) the session and writes or explains the code;
the other programmer follows (follower) to ofer feedback or heed
the explanations. To collaborate efciently and maintain mutually
recursive awareness of one another (also known as shared inten-
tionality [74]), the participating developers must look at the same
regions within the source code. Saros [60], an Eclipse plug-in, dis-
plays each programmer’s text cursor to communicate their location
in the source code and provides a Follow mode for programmers
to sync their IDE viewports during pair programming and code
walkthroughs. D’Angelo and Begel used a novel gaze visualization
technique to communicate the lines of code a collaborator was look-
ing at during pair programming [16]. The visualization changed
color when both programmers’ gaze overlapped, communicating
awareness and co-presence passively in shared task space. Their
evaluation of the gaze visualization technique revealed that when
the collaborators’ eye gaze was in sync, they more efciently spoke
about their source code using deictic references (e.g. terms such as
this, here, that, etc.)
The research discussed above has focused solely on collabo-
rations among sighted developers; they do not report anything
about the needs of BVI developers. The tools noted above rely
heavily on visual information, which leads to signifcant accessibil-
ity problems in mixed-ability contexts [36, 54], including software
development [53].
2.2 Inaccessibility for BVI Programmers in
Software Development
Initial empirical studies identifed accessibility issues with program-
ming tools, like seeking information in IDEs and challenges in doing
UI development [1, 42, 50]. Albusays et al. found that IDEs rely on
visual aids like indentation and syntax highlighting to structure
the source code, which favor navigation for sighted developers [2].
On the other hand, command line interfaces (CLIs) present text in
unstructured form [61], which limit BVI developers in navigating
the text efciently. Researchers have designed tools to improve nav-
igation [3, 4, 30, 62] and address the accessibility challenges in tasks
like code editing [57] and debugging [68]. Potluri et al. designed
CodeTalk, a Visual Studio extension, to improve code glanceability,
navigation and debugging by combining UI enhancements, speech,
and sound efects [56].
Accessibility challenges are further complicated by workplace dy-
namics and project management practices [28, 58]. BVI developers
ASSETS ’22, October 23–26, 2022, Athens, Greece
often reach out to their sighted colleagues to solve breakdowns in
programming tools, especially selecting teammates who understand
workfows with assistive technologies [71]. However, help-seeking
in the workplace can be complicated by the team and organization’s
attitudes towards accessibility and inclusion [53].
Accessibility and HCI research has primarily studied BVI de-
velopers’ as individual contributors, resulting in limited under-
standing about the accessibility of collaborative programming
activities. The exceptions are the research eforts focused on
creating inclusive learning experiences for novice BVI program-
mers [29, 35, 41, 48, 49, 69]. Pandey et al. were the frst to report
on the accessibility of collaborative software development [53].
They highlight the sociotechnical challenges in activities like pair
programming, code reviews, and code writing that limit the contri-
butions of BVI developers during collaboration. The paper reported
that BVI developers have to perform additional work, often invis-
ible to their sighted colleagues [12], to address the accessibility
challenges in the workplace [53].
2.3 Accessibility for Mixed-Ability
Programmers in Remote Software
Development
Given the limited research on awareness needs of BVI developers,
we turn to the larger accessibility literature to identify how sighted
and visually impaired people achieve real-time, remote collabora-
tion. A common approach is asking collaborators to describe their
actions, but sighted people often forget to verbalize the relevant
details, resulting in incomplete collaborator awareness [17, 53]. BVI
people hesitate to repeatedly request information to avoid slowing
down the pace of the collaboration or imposing on their sighted
collaborators [17].
In another workaround, collaborators work on their respective
computers with the BVI developer sharing their screen using a
video calling application to enable the sighted developer to follow
them [53]. Collaborators rely on chat features to copy-paste text and
share line numbers, etc., to collaborate more accessibly. While this
workaround allows a sighted developer to track a BVI developer’s
location, the information is not reciprocated to that BVI developer.
It also places the onus of driving the collaboration session on BVI
developers. BVI people occasionally have to relinquish control of
their computers to let the sighted colleague control their screen and
make changes in real-time [73]. Since BVI and sighted computer
users navigate interfaces diferently [8, 55], the approach causes
the screen reader to change focus unexpectedly without feedback
to the BVI person, impinging on their agency, and raising privacy
concerns [73]. Latency issues also lead to unwieldy drag and click
interactions for sighted collaborators.
Another strategy is to use NVDA Remote [75] or JAWS Tan-
dem [63], screen reader addons that transmit announcements in-
stead of simply relaying the video of the collaborator’s screen during
screen share [9, 73]. Unfortunately, these addons often sufer from
long latency, causing the BVI person to receive announcements
after 15-30 seconds [73]. Plus, sighted collaborators have to set up
the screen reader and the addon with matching confgurations at
their ends, giving them additional invisible coordination work and
adding to the total collaboration time [53]. Tools like Sinter address
ASSETS ’22, October 23–26, 2022, Athens, Greece
the latency issues and strict confguration requirements of remote
screen reading but have not yet been evaluated in collaborative
contexts [10].
Das et al. designed auditory representations to support asyn-
chronous collaborator awareness for activities such as commenting
and editing in shared text documents [18]. The evaluation of their
design elements revealed that the use of non-speech audio, voice
modulation, and contextual presentation could improve awareness
of BVI authors. Recently, CollabAlly [38] and Co11ab [19], both
Google Docs extensions, have been designed to support synchronous
collaborator awareness in shared document editing. The extensions
use spatial audio and voice fonts to represent the actions of collab-
orators joining and leaving the document, addition and deletion of
comments, and movement of their cursor into or away from the BVI
user’s paragraph. Co11ab also uses a variant of Follow mode [60] to
sync collaborators’ viewports. Shared document editing difers from
pair programming and code walkthroughs in an important way,
however. Shared document collaborators need real-time awareness
to actively avoid each other’s cursors in order to prevent overwrit-
ing collisions. Software development collaborators, on the other
hand, intentionally work on the same lines of code together, re-
quiring close and immediate coordination for extended periods of
time.
In summary, the workarounds discussed above insufciently
convey collaborator awareness, sufer from delays, place a dispro-
portionate burden on BVI people for driving the collaboration, and
compromise their agency. Furthermore, some of the proposed non-
visual techniques [18, 38], which are designed for collaborative
writing, will not fulfll the unique needs of synchronous collabora-
tive programming. We begin to address accessibility of collaborative
software development by designing design criteria for CodeWalk.
2.4 Design Criteria
Based on this literature review, we synthesize the following design
criteria for CodeWalk. We annotate each criterion with citations to
the literature that inspired them.
• D1. to minimize the cognitive load on the BVI developer [18]
(e.g., maintaining accessible workspace awareness [19, 38]
while minimizing confict with collaborators’ conversations)
during synchronous programming activities
• D2. to maintain agency of BVI developers [65] in mixed-ability
collaboration [17, 53]
• D3. to reduce the burden on BVI developers [73] of driving the
collaboration session to accessibly collaborate [53]
• D4. to support tightly-coupled collaboration between all col-
laborators [74]
3 DESIGN
In this section, we describe the formative design activities we con-
ducted that led to the design of CodeWalk.
3.1 Formative Design Activity 1: Choosing a
Baseline IDE
Our frst design activity was to choose a good baseline IDE to
build upon, one that was already accessible by BVI developers and
had facilities for collaborative co-editing support. Several popular
Potluri and Pandey, et al.
Table 1: Descriptions of our code walkthroughs. Each walk-
through occurred between a pair of sighted and/or BVI de-
velopers along with a sighted observer watching a shared
screen or listening to a BVI developer’s screen reader.
ID Leader Follower Sighted Observer Task
CW1 Sighted BVI Watched sighted developer’s screen
CW2 BVI Sighted Watched sighted developer’s screen
CW3 Sighted BVI Listened to BVI developer’s screen reader
CW4 BVI Sighted Watched BVI developer’s screen
IDEs that ofer collaborative co-editing support, such as JetBrains
CodeWithMe [33], Sublime [24] and Atom [22], are unfortunately
difcult to use by BVI developers. A few require BVI developers to
perform additional setup steps and others even lack accessibility
support for screen reader users to be able to perform basic code
editing [72].
By contrast, we found Microsoft’s Visual Studio Code IDE (VS
Code) to be both accessible and easily extensible. Its accessible
command palette makes it easy for screen reader users to fnd
commands. In addition, we learned that the VS Code team speaks
with the BVI developer community regularly to improve its acces-
sibility [44, 45]. VS Code supports collaborative work through its
Live Share extension. Similar to Saros [60], Live Share supports
synchronous collaboration through a Follow mode feature, which
draws the leader’s cursor in all of the followers’ IDEs and keeps it
in sync as the leader moves around the document. Live Share also
supports co-editing, keeping a shared view of the source code in
sync between the connected parties. Though there is little informa-
tion on the accessibility of Live Share’s features for BVI developers,
there are enough features to make it a good choice for our project’s
baseline IDE.
3.2 Formative Design Activity 2: Code
Walkthroughs
To assess the accessibility of VS Code with Live Share for teams
with BVI and sighted developers, three of the authors (one of whom
is also a BVI developer) conducted four code walkthroughs (see
Table 1). Two walkthroughs were led by a sighted researcher and
two were led by the BVI researcher-developer. We tried to cover all
combinations of abilities in a pair along with each taking on a leader
or follower role. All of the walkthroughs involved mixed ability
teams (CW1, CW2, CW3, and CW4). In all of the walkthroughs, a
third sighted researcher observed the shared screen, however in
code walkthrough CW3, the sighted observer simply tried to listen
to and comprehend the (slowed down) screen reader audio used by
the BVI developer.
Each code walkthrough looked at diferent example source code
from VS Code’s library of extensions. Each example extension con-
sisted of multiple fles written in TypeScript, several fles of which
were read through during each walkthrough. Each walkthrough
was 60 to 90 minutes long.
In addition to recording the code walkthrough sessions, the
sighted observer took detailed notes during each code walkthrough,
noting down accessibility breakdowns and workarounds. They
CodeWalk: Facilitating Shared Awareness in Mixed-Ability Collaborative Sofware Development
minimized their interruptions, limiting questions to clarifcations
of leader and follow actions to locate one another in each fle and to
help work around any non-obvious accessibility barriers. In total,
the sighted observer took six pages of notes. Immediately after
each code walkthrough, both the leader and the follower memoed,
refecting on their experiences [59]. As a group, the entire research
team rewatched the sessions from the recording and discussed the
memos and notes in their weekly meetings.
We observed that conversations between leader and follower pri-
marily focused on code discussions and clarifcation questions dur-
ing breakdowns in accessibility. When in Live Share’s Follow mode,
the IDE drew each developer’s cursor and synced their viewports
on everyone’s screen. Unfortunately, since this information was
only visual, the BVI follower was not aware of any of it and was fre-
quently lost. Consequently, the sighted leader had to speak their lo-
cation out loud to the BVI follower to help facilitate tightly-coupled
collaboration (Design Criterion D4). This active approach was error-
prone because the sighted leader sometimes forgot to mention their
location, especially when they were navigating quickly around
the source code. The BVI developer initiated another workaround,
asking clarifcation questions to sync with the leader. This often
put the burden on the BVI developer (Design Criterion D3) to re-
quest enough accessible information to follow along in the code
walkthrough.
When the BVI developer led the code walkthrough, they never
got lost. However, they often became unsure whether the Follow
mode had really synced the pairs’ viewports, and had to ask their
sighted follower to confrm they could see the expected code in
their window. Finally, the sighted collaborator often talked at the
same time as the BVI developer was trying to listen to their screen
reader. This made it difcult for the BVI developer to listen to either
audio stream. The research team discussed that some of the audio
overlaps could be avoided if the sighted developer knew when the
BVI developer’s screen reader was speaking. But, revealing the use
of AT is a sensitive issue for many screen reader users, thus we de-
cided to designed CodeWalk’s features to judiciously and carefully
make use of audio efects and speech to reduce the cognitive load
(Design Criterion D1) experienced by the BVI developer.
3.3 Formative Design Activity 3: Synthesizing
Code Walkthrough Scenarios
Inspired by our literature review and our code walkthroughs, we
created 15 scenarios comprising short events that occurred (or we
wished had occurred if the IDE were more accessible) across our
code walkthroughs. In addition, we considered both sighted and
BVI developers as leaders, but skipped scenarios involving solely
sighted developers. Some scenarios explore possible communica-
tion mechanisms between leaders and followers (e.g., non-verbal,
notifying the leader, notifying all collaborators, or not notifying
at all and syncing up after the session). All of these scenarios are
listed in Table 2.
Here is an illustration of Scenarios 1 and 2, which expose some
inaccessible features of the baseline IDE and the design features we
explored to address them. Blake, a sighted developer, wants to refac-
tor a piece of code. He asks Mia, a BVI developer and his colleague
for advice. They set up an audio call to verbally discuss the code as
ASSETS ’22, October 23–26, 2022, Athens, Greece
they view the code in a Live Share collaboration session hosted by
Blake. Blake shares the session link with Mia, who joins the session
and is presented with Blake’s code in her IDE. Mia invokes the
Follow mode command to stay in sync with Blake’s viewport. As
Blake navigates in the IDE, Mia’s IDE shows a copy of Blake’s cur-
sor in a distinct color, which unfortunately is not accessible to Mia.
Though the viewport changes, Mia’s cursor remains untethered
from Blake’s. Therefore, Mia has to occasionally interrupt Blake to
ask him to speak his line numbers and keywords out loud so that
she can navigate there herself and use her screen reader to read the
code that Blake is referring to.
This scenario exposes the limitations and asymmetry of current
IDEs in supporting tightly-coupled collaboration and shared aware-
ness (Design Criterion D4) among sighted and BVI developers. To
address the asymmetry, CodeWalk automatically tethers a BVI de-
veloper’s cursor to the leader’s (section 4.1), so that their cursors
move in unison whenever the leader initiates the navigation ac-
tion. However, this only happens in Follow mode. Now, Mia should
normally have no doubts about being in sync with Blake, but can
still detach (i.e., turn of Follow mode) from the leader if she wants
to explore the code on her own. The feature can also be useful in
Scenarios 9 through 13 where she leads the collaboration. She does
not have to worry about the correct code segment being displayed
in Blake’s IDE.
Furthermore, to reduce the burden on BVI developers (Design
Criterion D3), to preserve their agency (Design Criterion D2), and
minimize cognitive load, we designed several features in CodeWalk
to convey a collaborator’s location, navigation, and edit actions
accessibly to BVI developers using a passive, automated approach.
We describe the detailed implementation of these features next.
4 CODEWALK
CodeWalk is a set of features released with Microsoft’s Live Share
VS Code extension that supports accessible, remote, synchronous
code review and refactoring activities. We describe the cursor teth-
ering and audible feedback features that power its capabilities and
discuss some of the implementation details that we found needed
careful design.
4.1 Features
4.1.1 Cursor Tethering. Live Share’s Follow mode yokes each col-
laborator’s editor viewport together, a passive visual mechanism
that is inaccessible to BVI developers. CodeWalk facilitates tightly-
coupled collaboration (Design Criterion D4) by tethering BVI collab-
orators’ cursors with the host of a Live Share session. In designing
this feature, we explored several options to toggle tethering along
with various levels of autonomy, ranging from always tethering
cursors to only tethering cursors when the user toggles Follow
mode.
Always tethering the BVI developer’s cursor to their collaborator
minimizes their cognitive load (Design Criterion D1), but reduces
their agency (violating Design Criterion D2), as the sighted col-
league would have total control over their BVI colleague’s cursor.
BVI and sighted developers navigate code and interfaces difer-
ently [2, 55, 56]; they may want to read a part of the code that their
sighted collaborator is talking about by character or by word, a kind
ASSETS ’22, October 23–26, 2022, Athens, Greece Potluri and Pandey, et al.

Table 2: Mixed ability code walkthrough scenarios that informed the design requirements for CodeWalk. Each scenario was
inspired by at least one code walkthrough. Sighted+ and BVI+ indicates more than one developer. Following or watching “on
the side” splits the VS Code editor and puts one in Follow mode.

Scenario Leader Follower Walkthrough Activity

1 Sighted BVI CW1, CW3 Follower joins collaboration session hosted by leader.
2 Sighted BVI CW1, CW3 Follower tethers cursor to leader.
3 Sighted BVI CW1 Follow leader “on the side” without tethering.
4 Sighted BVI CW1 Follower restarts tethering after watching leader “on the side”
5 Sighted BVI CW1, CW3 Follower tells leader they are lost.
6 Sighted BVI CW1, CW3 Follower takes notes during collaborative session.
7 Sighted BVI CW1, CW3 Follower fails to notice what command the leader just used.
8 Sighted BVI CW1, CW3 Follower asks leader about the command they just used.
9 BVI Sighted CW2, CW4 Leader invites follower to join collaboration session.
10 BVI Sighted CW2, CW4 Leader jumps to follower’s cursor, answers the follower’s question,
and jumps back.
11 BVI Sighted CW2 Leader asks follower to show them something.
12 BVI Sighted CW2 Leader asks follower a question to test if they are lost.
13 BVI Sighted CW2, CW3 Follower asks for help using a “I need help” command.
14 BVI Sighted, BVI CW2, CW3, CW4 Leader gets follower’s cursor location from VS Code.
15 BVI Sighted+, BVI+ CW4 Leader gets approximate location of multiple followers from VS
Code.

of fne-grained navigation a non-screen reader user has no idea
about. To support this need, we support temporarily untethering
the BVI follower’s cursor whenever they move it around, giving
them control to move the cursor to the code they want to read.
After 10 seconds of inactivity, CodeWalk retethers the cursors.
4.1.2 Conveying Collaborator Actions via Audio. CodeWalk uses
a combination of sounds and speech to passively communicate a
tethered collaborator’s location and their navigation and edit ac-
tions, reducing the burden on BVI developers to ask about them
(Design Criterion D3). We explored several sound designs, drawing
inspiration from audio cues used by popular accessible navigation
apps and operating systems. We experimented with futuristic ar-
tifcial sound efects as well as skeumorphic sounds of keyboard
clicks and scroll wheels. We felt that since BVI developers were
already familiar with the sounds of standard computer hardware,
the skeuomorphic sound efects would be the best one to convey
navigation actions. Navigation distance and direction lacked obvi-
ous skeumorphic analogs, so we designed a set of artifcial rising
and falling tone sound efects to be played a short time after navi-
gation activity ends. If the user clicks the mouse somewhere else
in the codebase, CodeWalk plays an artifcial “teleportation” sound
instead of a mouse click to make it more obvious that something
drastic has happened to the cursor location, which may invalidate
the mental model the BVI developer has of the region where they
thought the cursor was located.
In designing using speech and sound efects, our primary focus
is to minimize cognitive load relative to the frequency and speci-
fcity of the information to be conveyed. We draw inspiration from
accessible data visualization and programming eforts [40, 66, 70]
and use speech to announce highly specifc information like line
numbers, which is needed less frequently. We use sound efects
to convey less specifc information, such as the actions performed
and navigation direction - these actions occur at a much higher
frequency during collaboration. The use of speech and sound ef-
fects has shown to improve awareness between collaborators [43].
Sound efects minimize cognitive load on BVI developers (Design
Criterion D1) because they do not require conscious interpretation
and can be heard even when a screen reader is actively speaking.
However, they do not give enough information about a collabora-
tor’s location. To address this, after navigation activity has stopped
for 1.5 seconds, CodeWalk uses computer-generated speech to tell
the BVI developer what line of code (and fle name if it changed)
they are now on. Most sound efects are around 200 ms (though
one is longer, at 550 ms). Similarly, speech announcements are
kept short and precise. The complete business logic for CodeWalk’s
sound efects and speech can be seen in Table 3.
Sighted collaborators commonly use visual reference space ges-
tures such as cursor location, text highlighting, and mouse waving
to refer to code [16], gestures largely unavailable to BVI develop-
ers [36, 54]. CodeWalk supports selection awareness by speaking
the portion of code highlighted by a collaborator. This simplifes the
process of understanding what a collaborator wants to talk about
and reduces the burden on BVI developers to ask sighted colleagues
to verbally announce their selections. An example illustrating these
sound efects and speech can be seen in Figure 2.
When a collaborator edits the code with their keyboard, sharper,
shorter key click sound efects are played. If the collaborators are un-
tethered (i.e., Follow mode is of), then they may both be editing the
document simultaneously. The baseline VS Code Live Share gives
no indicator that collaborators’ edits may collide, other than draw-
ing the two cursors near one another. This, of course, is inaccessible
to BVI developers. In CodeWalk, whenever the collaborators are
editing within 5 lines of one another, CodeWalk speaks a warning,
“your collaborator is editing nearby.” If the collaborator is on the
same line, the warning repeats, “your collaborator is editing the
same line as you,” which should hopefully cause the collaborators to
CodeWalk: Facilitating Shared Awareness in Mixed-Ability Collaborative Sofware Development ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 3: Use of audio cues to convey awareness indicators in Follow mode (unless specifed otherwise in the row)

Non-Speech Speech Built-in

Awareness Non-Speech Speech
Indicator Indicator Visual
Information Indicator Indicator
Frequency Frequency Indicator
Viewport scrolls Click wheel Every scroll None None Screen scrolls
sound event
Scroll direction Falling or rising When scrolling None None Can be inferred
tone depending stops from the scrolling
on direction viewport
Current None None “Lines X to Y on When scrolling Visible on screen
Viewport screen” stops
Cursor moves by Keyboard click Every cursor “Line N” 1.5 seconds after Cursor moves on
single line to line move cursor moves end screen
N
Cursor moves Falling or rising Every event “Line N” 1.5 seconds after Cursor moves on
multiple lines to tone depending cursor moves end screen
line N on move
direction
Cursor moves by Falling or rising Every event “Line N” Every event Cursor moves on
multiple lines to tone depending screen
line N on move
direction
Selection Depends on Every event “Selection on line 1.5 seconds after Selection visible
selection (key- N” selection is made on screen
board/mouse)
Edits on Keyboard type For every None None Cursor moves on
follower’s line character typed screen; edits visi-
ble on screen
Edits on Keyboard type For every “<collaborator> is As long as edits Cursor moves on
follower’s line character typed editing the same continue on the screen; edits visi-
(Follow mode of ) line as you” same line ble on screen
Edits within 5 Proximity sound For every “<collaborator> is As long as edits Cursor moves on
lines of follower character typed editing nearby” continue on the screen; edits visi-
(Follow mode of ) same line ble on screen
Follow status Pull and push When follower “You are now fol- Every event None
sound starts and stops lowing <collabo-
following leader rator>”

Figure 2: Image shows BVI developer’s code editor as she follows a sighted leader. CodeWalk tethers the cursors of collaborators
in Follow mode. When the sighted leader uses arrow keys to navigate, CodeWalk plays skeuomorphic keyboard sounds for each
line moved. When they stop navigation at line 19, CodeWalk plays an artifcial falling tone to indicate downward movement
followed by line number announcement. Similarly, when they highlight a word, CodeWalk announces the selection.

stop what they are doing and negotiate their next actions together,
verbally.
4.2 System Implementation
The basic architecture of CodeWalk can be seen in Figure 3. Each de-
veloper runs an instance of the VS Code IDE, extended by CodeWalk.
CodeWalk extends four extensibility points provided by VS Code
and Live Share (i.e., programming APIs enabling third-party devel-
opers to enhance specifc features of the IDE), which we illustrate
in the following scenario walkthrough.
Mia, a BVI developer (User 2 in Figure 3), installs CodeWalk
along with Blake, a sighted colleague (User 1 in Figure 3). Blake
and Mia enter into a joint collaboration session facilitated by VS
Code Live Share’s existing co-editing support. Various extension
points are triggered as they collaborate. The frst triggers when
ASSETS ’22, October 23–26, 2022, Athens, Greece
IDE for User 1: Sighted Developer
Display User 1’s cursor in User 2’s IDE
Existing Co-editing Support
Display User 2’s cursor in User 1’s IDE
API Extension Points
e.g. Text Cursor API to track user 1’s locations,
Document Change API to track user 1’s edits, etc. Audio cue filename; Announcement text
e.g. keyclick.wav; “Line 22”
Figure 3: System Architecture Diagram for CodeWalk
Blake changes his cursor location. It sends the new location to Mia’s
IDE along with a tag explaining what action caused it. CodeWalk
then runs through its business logic (described in Table 3) to de-
termine the kind of audio feedback to play (sounds and/or speech)
and queues them for playback on Mia’s computer. Each of Blake’s
navigation actions may trigger a tuple of one or two sounds along
with a spoken message, each separated by a delay. Typically, the
frst sound is skeuomorphic (i.e. for key clicks, mouse clicks, or
the scroll wheel). It is followed by a 1.5-second delay, a falling or
rising tone (to indicate navigation direction), and an spoken an-
nouncement of the new line number. The 1.5 second delay avoids
spamming Mia with additional sounds and speech if Blake pauses
momentarily during his actions (e.g. pausing to adjust mouse wheel
when scrolling through a fle). As there are no cross-platform APIs
for asking screen readers to generate custom announcements, we
generate CodeWalk’s spoken announcements using the Microsoft
Azure Cognitive Services Text-to-Speech (TTS) API. If too many
sounds are requested to be played in a row, queued sounds and
speech may be delayed. If they are delayed over one second, it is
considered out of date and CodeWalk ejects it from the playback
queue. Additionally, CodeWalk categorizes sounds into notifcations
and warnings. Events associated with the former are interruptible,
meaning if a second event comes in before the frst one is done
playing, it will cancel the frst and start the second right away.
Warning sounds are uninterruptible. They are reserved only for
edit actions to prevent co-editors from overwriting one another’s
changes.
The second extension point tethers the co-editors’ cursors to-
gether. When Mia follows Blake, her cursor will move automatically
wherever Blake’s cursor goes. When tethering is turned on, all
of Blake’s edits will always happen on the same line as Mia’s, so
we suppress any spoken warnings. When tethering is turned of,
edit sounds and announcements only play when Blake is editing
within fve lines of Mia, else the sheer quantity of sounds would
overwhelm her.
A third extension point tracks and conveys selection actions
between the co-editors. Mia hears a verbal announcement of the
selection whenever Blake selects some text in his editor, as long as
she is tethered to Blake.
Potluri and Pandey, et al.
IDE for User 2: BVI Developer
Existing Co-editing Support
API Extension Points
Announcement Text
CodeWalk
Speech
Microsoft Cognitive
Services API
Load and play audio cue + speech
User 2 Speaker
The fnal extension point queues sounds to be played whenever
Mia toggles Follow mode on or of. Similar to what happens in Zoom
or Microsoft Teams, a sound is played whenever a co-editor joins
or leaves the collaboration session, followed by an announcement
of the co-editor’s name and their cursor location.
5 EVALUATION STUDY
We conducted a within-subjects study to understand and compare
the efectiveness of VS Code Live Share with CodeWalk features
against our baseline, plain VS Code Live Share [46]. Our study
aimed at answering the following research questions: (1) How well
does CodeWalk improve coordination during remote synchronous
collaboration between sighted and BVI developers? (2) How does
it afect the communication between developers about the source
code? (3) How does it shape BVI developers’ perceptions of their
collaborative experience?
5.1 Participants
Eligible participants had to be 18 years or older, identify as blind or
visually impaired, be comfortable with using screen readers, have
at least a year of programming experience in one of the following
languages: C, C++, C#, Python, JavaScript, TypeScript, or Java (i.e.
the programming languages into which we translated our tasks),
have collaborated on code, and be able to communicate about code
in spoken English. We recruited participants by posting on social
media platforms and mailing lists (e.g., Program-L) that primarily
comprised BVI developers.
Our study accepted 10 BVI developers (P1–P10). Nine partici-
pants identifed as male; one as female. Participants were between
21 and 47 years old (average age 33.6; median age 31.5). Table 4
summarizes the details of participants’ demographics, country of
residence, and current job title. Each participant was compensated
with USD $100 (or its equivalent in local currency) for their partici-
pation in the study.
5.2 Tasks
We employed a 2x2 within-subjects experimental design. Each par-
ticipant, in collaboration with a sighted confederate (one of the
CodeWalk: Facilitating Shared Awareness in Mixed-Ability Collaborative Sofware Development ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 4: Demographic characteristics of the participants and their study session details.

Programming
ID Gender Age Country Profession Condition 1 Condition 2
Language
P1 Male 34 India Senior Software Engineer CodeWalk (Set B) Baseline (Set A) JavaScript
P2 Male 24 India Software Development Engineer CodeWalk (Set A) Baseline (Set B) Python
P3 Male 27 India Technology Analyst CodeWalk (Set B) Baseline (Set A) JavaScript
P4 Male 47 USA Software Engineering Manager Baseline (Set B) CodeWalk (Set A) JavaScript
P5 Female 29 USA Data and Applied Scientist CodeWalk (Set A) Baseline (Set B) Python
P6 Male 44 USA Senior Program Manager Baseline (Set B) CodeWalk (Set A) C#
P7 Male 21 USA Software Engineering Intern Baseline (Set A) CodeWalk (Set B) Python
P8 Male 46 Sweden Software Developer Baseline (Set B) CodeWalk (Set A) C#
P9 Male 35 USA Senior Software Developer Baseline (Set A) CodeWalk (Set B) Java
P10 Male 29 Netherlands Freelance Developer CodeWalk (Set A) Baseline (Set B) Python

authors and the study coordinator), performed a series of tasks
without CodeWalk (the baseline condition) and another set of
tasks with CodeWalk (experimental condition). Like prior HCI stud-
ies [11, 31, 51], the confederate was instructed to strictly and con-
sistently follow the study protocol with all participants, which is
known to lead to more generalizable results [27].
We developed two sets of tasks (henceforth, set A and set B) for
the study. We randomized the order of task sets and the conditions
across the participants. Each set comprised three tasks, resulting
in a total of six tasks. We designed the tasks to range from easy
to difcult within each set, enabling participants to ease into the
programming environment. In both sets, the frst task was based
on editing a string; the second task required editing code central to
program execution; the third task required refactoring a specifed
set of lines into a function. The research team conducted multiple
rounds of discussion to ensure that both task sets were of equivalent
levels of difculty.
The confederate led the code walkthrough and asked the partici-
pant to follow them during the tasks. They asked the participant to
recommend changes and solutions to complete each task. Partici-
pants could explore, edit, or verify the actions of the confederate
as they wished. This made the collaboration feel more natural.
All tasks were based on Hangman,2 a common text-based game.
The tasks were representative of software development activities
and required participants to perform code reviews, bug fxing, and
code refactoring. We downloaded publicly available source code for
Hangman in C#, Java, Python, and JavaScript so that participants
could perform the tasks in their preferred programming language.
For internal validity, we selected code samples with similar lengths
and modifed their source code to have similar fle names, func-
tion names, variable names, and code structure. All code samples
included (1) a main code fle representing the game’s logic, (2) a
text fle containing 851 words to play the game, and (3) a text fle
listing both sets of tasks in the order determined for the participant.
The C# code sample included an additional fle that represented
the game’s UI; this fle was referenced for the string editing task.
The code samples in JavaScript included HTML/CSS fles which
were not required for the study. Table 4 lists the order of conditions
and task sets for each participant, along with the programming
language used in their study session.
2 https://en.wikipedia.org/wiki/Hangman_(game)
We conducted a pilot study with one BVI developer (not includ-
ing in the main study) to ensure that each task set was possible to
complete within 20 minutes and the total study time did not exceed
90 minutes. Based on their feedback, we found we needed only to
improve two things: our instructions on how to connect remotely
and the description of the extensions’ features in both conditions.
5.3 Procedure
We conducted the studies remotely over Microsoft Teams or Zoom,
as per the participant’s preference. Participants were not required
to turn on their cameras for the study. Before the main tasks, the
study coordinator explained the key features of the IDE used in the
study, the baseline condition, and CodeWalk. We asked participants
to share their screen without including the system audio so that
the confederate would not hear the participant’s screen reader or
CodeWalk audio output. We used the video conferencing tool’s
recording feature to capture the conversation between the partici-
pant and the study coordinator, which we referred to during our
analysis.
To facilitate switching between study conditions, we created a
Windows 10 virtual machine (VM) with two diferent versions of
VS Code — one version with the baseline condition and another
augmented with CodeWalk. Both versions had the same features,
keyboard shortcuts, and UI settings. We installed JAWS (version
2020) and NVDA (version 2021) on the remote VM. We also set up
Code Factory Eloquence [15], a popular text-to-speech (TTS) syn-
thesizer used to customize screen reader voice and speech. Before
each study session, we set NVDA as the default screen reader.
Participants connected to the VM using Microsoft Remote Desk-
top software. Upon login, we informed participants that they could
modify the screen reader settings. Only P1 and P4 used JAWS; the
others performed the tasks with modifed settings for NVDA. We
also turned on screen recording within the VM to record the screen
reader speech. Due to technical glitches with screen recording soft-
ware, we were unable to record the screen reader usage for P4 and
missed a portion of the screen reader usage for P1 and P2.
Participants were instructed to switch to the IDE window for
the frst condition (see Table 4 for the order) and invite the study
coordinator (referred to as the ‘confederate’ in this paragraph) to the
collaboration session. The confederate was under strict instructions
to hide the participant’s shared screen to not look at their IDE
ASSETS ’22, October 23–26, 2022, Athens, Greece
contents. Participants could ask questions about the IDE features or
share their comments about the baseline and CodeWalk during the
study. We believed this approach allowed the collaboration and the
conversation to proceed more naturally. After twenty minutes, the
participant and the confederate switched to the other experimental
condition to perform the next set of tasks.
After each condition, participants verbally responded to a 12
statement Likert-scale questionnaire (see Table 6). The question-
naire was adapted from existing scales [16, 64] and assessed partici-
pants’ opinions regarding awareness and collaboration. Participants
had to indicate on a fve point scale whether they strongly disagreed
(1) or strongly agreed (5) with the statements. The study concluded
with an informal interview about participants’ experience with
CodeWalk and a short questionnaire about their personal and pro-
gramming background.
5.4 Data Analysis
The confederate wrote analytic memos [59] after each study ses-
sion to refect on how each condition shaped their awareness and
collaboration. One researcher reviewed all the video recordings
and conversation transcripts to highlight the timestamps of sync
operations, analyzed using a Poisson regression (see §6.1). Section
6.2 discusses how we adapted an existing list of codes from [16] to
analyze the conversation between the confederate and the partic-
ipants. Section 6.3 details our analysis of participants’ responses
to Likert-scale questionnaire . Lastly, two authors used descriptive
coding [47] to analyze the interviews and organized the codes into
themes around collaboration and feedback (see §6.3).
6 STUDY RESULTS
6.1 How well did CodeWalk improve
coordination during collaboration?
To analyze how well the participants could follow the confederate,
we compared the number of times they attempted to sync their lo-
cation with the confederate’s location. We operationalized location
syncing as attempts, including successful attempts, by participants
to move their cursor to the confederate’s location using one of the
following: (1) moving from one fle to another (2) going from one
line to another (3) using the fnd tool to search for a specifc word
to navigate to its location (4) toggling the tether command if unsure
of the tether status of cursors. Participants synced their locations
to read the code that the confederate was referring to and to follow
them during the collaboration. We hypothesized that participants
would require fewer sync operations in CodeWalk because they
would feel less lost compared to the baseline. We analyzed the screen
share recording and participants’ screen reader speech to calculate
the total number of sync attempts.
The median value for the number of times participants tried
syncing in CodeWalk was 1, compared to the median value of 8 in
baseline, a huge drop. Figure 4a a visualizes the number of sync
attempts for each participant in both conditions. As recommended
for integral data with possibility of rare occurrences, we ft a Pois-
son regression [77]. We found that participants made signifcantly
fewer attempts to sync locations in CodeWalk condition (p = .000875
< 0.01). The result indicates that CodeWalk enabled the participants
and the confederate to stay closely coordinated during collaboration.
Potluri and Pandey, et al.
Table 5: Reference Codes and Descriptions
Code Description
Deictic When a participant or the confederate uses a deictic
reference such as this or here, e.g., “Let’s start with this
task.”
Anaphora When a participant or the confederate refers to a past
action or location, e.g., “Can you go back?”
Abstract When a participant or the confederate uses a broad cat-
egory to refer to an object, e.g., “We need to understand
the function.”
Reading When a participant or the confederate loudly reads a
portion of the code, generally done when approximate
location of the collaborator is known, e.g., “Press Enter
to leave the game!”
Typing When a participant or the confederate is referring to the
text being typed, e.g., “Let me confrm what you wrote.”
Specifc When a participant or the confederate uses a specifc
name to describe an object, e.g., “Let’s go to didGuessCor-
rect().”
Line number When a participant or the confederate uses a specifc
line number, e.g., “I am on line 31.”
Note, P7’s outlier value in the CodeWalk condition. The followup
interview and his screen share recording revealed that he had not
realized that his cursor was tethered to the confederate’s. He inter-
preted the sounds and speech in CodeWalk as locations he should
move to, resulting in similar behavior across both conditions.
6.2 How did CodeWalk afect communication
about the source code?
Since CodeWalk conveyed information on a co-editor’s location
and actions, we hypothesized that CodeWalk would enable the
participants and the confederate to converse about code using more
abstract and deictic references compared to the baseline. We also
hypothesized that they would use more line numbers and specifc
names in the baseline condition compared to CodeWalk. Both our
hypotheses were informed by D’Angelo and Begel [16].
To analyze how the confederate and the participant referred to
locations in the code, we adapted and extended the list of referents
from D’Angelo and Begel [16] by making two additions (anaphora
and reading). Table 5 shows the codes of all 7 referents along with
their defnitions and examples. Each time a participant or the con-
federate made a reference to the code, we recorded the referent and
its category. We calculated the total number of referents uttered
by the confederate and the participant in each study session. Thus,
we ended with 14 total referent counts (7 for the confederate; 7 for
the participant) in each condition per session. We normalized the
data in each condition by calculating the percentage of referents in
each category. We did not include P2’s data because he experienced
signifcant lags in screen reader speech causing the confederate
and P2 to verbalize and read code aloud for a large portion of the
study (an unlikely scenario for collaboration in outside the study).
We visualize the fraction of referents in each category for each
condition in Figure 4b. The fgure shows that specifc referents
were used most heavily during the tasks in both conditions. We
also note a greater usage of deictic and abstract referents in the
CodeWalk: Facilitating Shared Awareness in Mixed-Ability Collaborative Sofware Development ASSETS ’22, October 23–26, 2022, Athens, Greece

(a) Number of times participants attempted to sync locations with (b) Percentage of referents of each type uttered in each condition. An
confederate in each condition. * is shown above referent types that are signifcantly diferent across
conditions

Figure 4: Results from video and conversation analysis

CodeWalk condition compared to the baseline. We carried out a Table 6: Statements in the Likert-scale questionnaire along
one-way ANOVA to compare the percentage of referents in both with their p values. All statements had equal or higher me-
conditions. The analysis revealed no signifcant diference in the dian value in the CodeWalk condition. * beside the state-
percentage values of any category except the abstract referents (p ment code indicates p < 0.05.
= .037 < 0.05), which were greater in the CodeWalk condition.
# Statement p value
6.3 How did participants perceive their S1 I was keenly aware of everything in my environment. .0009*
collaboration experience with CodeWalk? S2 I was conscious of what is going on around me. .0035*
6.3.1 Responses to Likert-Scale Qestionnaire. Table 6 lists the S3 I was aware of what my teammate did and how it hap- .0029*
statements in the Likert questionnaire along with the p value of pened.
participants’ responses in both conditions. All statements had an S4 I was aware that my teammate is aware of my actions. .0197*
S5 I am aware of how well we performed together in the .0118*
equal or higher median value in the CodeWalk condition. A higher
team.
median indicates more agreement among the participants, implying S6 I felt like my teammate and I were on the same page .0118*
a better overall experience with CodeWalk. most of the time.
A one-tailed Wilcoxon signed-rank test indicated that the value S7 I could tell what my teammate was thinking .0328*
for responses to ten out of twelve statements were signifcantly about/looking at/talking about most of the time.
higher in the CodeWalk condition (p < 0.05). Their statement codes S8 I felt like we shared common subgoals as we worked .0294*
are followed by an asterisk in Table 6. The signifcantly diferent on the task.
values confrm that participants felt more aware of the confederate’s S9 My teammate communicated clearly during the task. .1284
locations and actions with CodeWalk. Furthermore, participants S10 I communicated clearly with my teammate during this .0169*
felt that the shared awareness was reciprocated by the confederate task.
S11 It was fun to work with my teammate on this task. .0294*
when using CodeWalk i.e. the participants felt that the confederate
S12 My teammate worked efectively with me to accomplish .1284
was also aware of their actions (S4 in Table 6). This indicates greater the task.
shared intentionality with CodeWalk.
Two statements (S9 and S12) were not signifcantly diferent
across conditions. These focused on participants’ perceptions of
the confederate’s communication style and efectiveness in collabo-
highlight code or read code aloud to direct the confederate’s at-
ration during the tasks. Since the confederate remained unchanged
tention. In addition, we observed that the confederate could easily
in both conditions, participants may have felt that their commu-
keep track of the participant’s cursor with CodeWalk’s tethering
nication style remained consistent across conditions. Participants’
feature. The participant’s cursor was always visible in the confeder-
responses to S9 and S12 may have also been subject to demand
ate’s viewport, and if the participant moved out of the viewport to
characteristics, cues that shape participants’ desire to form a posi-
read code, the confederate would scroll to keep track. On the other
tive impression on the experimenter [52]. Participants may have
hand, participants reported that they “leaned on the communication”
wanted to appear polite in their responses about the confederate’s
with the confederate “pretty heavily” (P7) in the baseline condition.
communication and collaboration abilities.
They had to either wait for the confederate to verbally announce
6.3.2 Interview Results. Video analysis revealed that the partici- their location using a line number or function name or request the
pants felt aware of being in the confederate’s vicinity. They would location information to sync cursors, also indicated by Figure 4a.
ASSETS ’22, October 23–26, 2022, Athens, Greece
We noted instances where participants used CodeWalk’s teth-
ering feature to direct the confederate. For example, P5 asked the
confederate to take her “to the line again” to revisit the source code.
The confederate moved their cursors to the location P5 had speci-
fed; the sounds confrmed arrival for P5. Later on in the interview,
P5 shared that the “auto move [of cursors] was really useful”. Simi-
larly, P6 directed the confederate to move their cursor to various
lines during the refactoring task. After each move, he would ex-
plore the code at the destination line, make recommendations for
improving the code, and then instruct the confederate to take him
to the next location.
Participants used CodeWalk’s sound efects extensively to main-
tain awareness of the confederate’s actions. For instance, after the
confederate fnished typing, P9 mentioned, “Yeah, I can tell you are
done ’cause the typing noise stopped”, and then went on to verify the
changes made by the confederate. Participants used the speech an-
nouncements to keep track of location changes. Many participants
phrased this as being aware that “things were happening” (P8). Even
on the occasions when the confederate moved quickly, leading to a
succession of sounds, participants felt that “at least [CodeWalk] con-
veyed a sense of movement” (P4). The increased awareness seemed
to positively shape the participants’ feelings about collaboration
and assuaged their worries about feeling lost: “Because I could just
snap to wherever you were, I wasn’t worried about wandering of” —
P4.
Furthermore, participants liked the design choice of primarily
using audio cues to convey the confederate’s actions and relying
on speech sparingly. They shared that the audio cues “packed a lot
of info” (P7) without seeming verbose. In addition, participants
did not seem to mind when the audio cues played simultaneously
with the screen reader speech, but they indicated a preference for
shorter sounds. Most participants were able to quickly map the
skeuomorphic audio cues to their awareness indicators. It took a
few participants longer to associate the non-skeuomorphic audio
cues with their intended meaning of direction changes. However,
they acknowledged that they had not “used it [CodeWalk] enough”
(P6) to remember the sounds and believed that “some more sessions”
(P1) would enable them to map all the audio cues to their respective
meanings.
Every participant told us that they would like to use CodeWalk
to collaborate with their teammates. P5 mentioned that using Code-
Walk in code reviews would enable her to be on the “same page
without lagging behind.” P7 shared that CodeWalk would be “ab-
solutely instrumental” in his pair programming assignments, and
he would “install it immediately” if it were released. P9 felt that it
would allow him to mentor junior developers by letting them drive
collaboration sessions: “When I’m collaborating, I’m the one driving
and I share my screen and they look at it. It’s just easier that way [...] I
would be much more likely with an extension like this to let them drive
more often.” Participants also appreciated that CodeWalk was built
for VS Code, a mainstream and accessible IDE that sighted “people
might have” (P5). Upon its launch, they could use it without asking
their colleagues to switch to a new IDE. These quotes suggest that
CodeWalk can enable BVI developers to participate in collaborative
activities without requiring them to manually manage the sessions
on their own.
Potluri and Pandey, et al.
6.4 Threats to Validity
Our study employed a sighted research team member as a confeder-
ate for all study sessions. Employing a single confederate across all
participants is common in HCI [11, 31, 51] and is recommended for
maintaining the internal validity of the experiment [27]. Despite fol-
lowing the study protocol strictly, the confederate may have gained
experience and improved as a communicator with each session.
Thus, it is likely that later participants’ collaboration experience
may have been better than the former, resulting in fewer diferences
in metrics between conditions. We believe the within-subjects de-
sign choice would have addressed any learning efects on the part
of the confederate.
Due to their research experience in accessibility, the confederate
may better understand participants’ awareness needs than sighted
people unused to collaborating with BVI developers. Participants
commented that the confederate was “a very good communicator”
(P7), also confrmed by the lack of signifcant diference in responses
to S9 on the Likert-scale questionnaire (see Table 6). The confeder-
ate’s communication may have suppressed diferences in referent
counts between conditions. Therefore, in real-world conditions
with more typical collaborators, CodeWalk may show even more
improvement in communication metrics over the baseline.
We deployed CodeWalk on a cloud-based virtual machine (VM)
to simplify the installation for our participants. Using screen readers
through remote VM may have increased latency. Some participants
reported lags in screen reader playback which may have impacted
their experience with the extension and shaped their feedback. The
latency issues are unlikely to occur in real-world conditions, since
the extension would be installed on the user’s own home system.
Thus, we expect the experience of CodeWalk to be better upon its
release.
7 DISCUSSION
Overall, we fnd CodeWalk successfully translates and conveys refer-
ence space gestures from sighted developers to their BVI colleagues,
extending Buxton’s model [13] for efective remote collaboration
to mixed-ability collaborations. In this section, we summarize our
fndings, consider the role of interdependence in our design, and
relate our results to the two projects that are most similar to ours.
We then refect on our own research practices and propose future
work.
7.1 Summary of Findings
Our study results show that signifcantly fewer attempts were
needed by our BVI participants to sync locations with CodeWalk
than in the baseline condition. This suggests that the coordina-
tion burden (Design Criterion D3), which often requires explicit
communication of awareness cues between collaborators, is re-
duced through CodeWalk’s sound efects and speech. Automating
the transmission of code location and navigation actions helps to
ensure that the sighted colleague also benefts from a reduced co-
ordination burden, since they need not remember to convey those
actions verbally either. The participants’ increased use of abstract
referents to code locations showed a corresponding decrease in the
number of more specifc referents (using line numbers and function
CodeWalk: Facilitating Shared Awareness in Mixed-Ability Collaborative Sofware Development
names). This suggests a reduction in cognitive load (Design Crite-
rion D1) on the part of the BVI developer. It also shows an increased
sense of shared awareness and shared intentionality between the
participants, which ensures that the BVI developer had the capabil-
ity to contribute equitably according to their ability rather than be
sidelined by inaccessible collaboration tools.
From the Likert scale statements, we learned that participants
felt that CodeWalk improved their awareness of their environment,
their teammate, and their teammate’s actions. They also felt that
they were more likely to be on the same mental page most of the
time and were able to work efectively together. BVI developers
were more likely to highlight text on the screen using their key-
board, confdent in the knowledge that their sighted colleagues
would be able to notice it and react to it. BVI participants also could
tell from the sound efects when their colleagues were navigating
or when they had stopped, concluding that they were now free to
engage in conversation and explore the source code. Not only did
this increase improved their communication, it also minimized the
cognitive load (Design Criterion D1) of trying to intuit what their
sighted colleague might be doing without any audible feedback.
CodeWalk’s cursor tethering was designed to support tight cou-
pling (criterion D4) between participants at the task level, so that
when one navigated through the code or edited some text, the other
would immediately be made aware and be able to respond. Some
participants responded by directing the confederate to move to
additional locations, showing increased agency (Design Criterion
D2), looking around the code with their own screen reader, and
then driving the confederate to the next code location.
CodeWalk’s skeumorphic sounds (e.g., key clicks and scroll wheel
sounds) were straightforward for the participants to understand
with no training. However, some sound efects, e.g., rising tone and
falling tone, used to convey directionality of movement, were not
immediately obvious to the listeners. While participants got better
at distinguishing these during their study session, others may need
more time to get better at this.3
CodeWalk’s spoken sentences were necessary to orient the BVI
participants after their sighted colleagues navigated to new areas in
the code. Sometimes, however, these spoken words collided with the
participant’s own screen reader speech. An early version of Code-
Walk played its sounds and speech by extending the NVDA screen
reader, which enabled us to detect overlapping speech utterances
and cancel one of them. However, to ensure CodeWalk worked with
multiple screen readers on multiple platforms (including Mac and
Linux), we used Microsoft’s Azure Cognitive Services to generate
speech and platform-specifc sound APIs to play it. It is possible
to address the overlapping audio, however due to time constraints,
we were unable to program CodeWalk to cancel our audio while
the screen reader was talking. We encourage screen reader and op-
erating system manufacturers to ofer extensible platform-agnostic
APIs for integrated systems like CodeWalk with screen readers.
One interesting form of spoken collision remains. Sighted col-
leagues receive no indicators when BVI users are listening to their
screen readers, and thus do not realize to stop talking to the BVI
3 See Cat_ToBI (http://prosodia.upf.edu/cat_tobi/en/ear_training/listening.html) to
practice distinguishing rising and falling tones from one another.
ASSETS ’22, October 23–26, 2022, Athens, Greece
user to avoid overlapping with the screen reader or CodeWalk. Par-
ticipants expressed a need for avoiding “double-speak” (P8) between
the collaborator and speech announcements by their screen readers
and CodeWalk. We plan to explore designs of a visual indicator to
non-screen reader users of CodeWalk to let them know when screen
reader speech is active for any of their collaborators. This feature
should require BVI users to opt-in before it is turned on because
BVI users’ opinions of whether to reveal their use of AT to col-
leagues varies by culture [39] and may have signifcant workplace
consequences [5].
7.2 Accessible Co-Editing
Lee et al. [37]’s CollabAlly tool developed similar sound efect
and speech-based feedback for BVI writers in common co-editing
environments (e.g. Google Docs). CollabAlly found success in iden-
tifying collaborators’ ongoing work and comments in the document,
enabling collaborator awareness to avoid overwrites synchronously
and asynchronously. Our work examined the awareness needs
found in synchronous tasks of code walkthroughs and reviews and
found the timeliness of push-based notifcations vital to enable
BVI collaborators to stay in sync with their sighted colleagues for
extended periods of time. Many coding tasks fuidly switch be-
tween asynchronous and synchronous modes leaving unanswered
how to best support users’ cognitive load by conveying awareness
information simultaneously using pull and push-based modalities.
Das et al. [18, 19]’s Co11ab work supporting mixed ability co-
editing stops short of handling collaborators editing near one an-
other. Prior to CodeWalk, these kinds of close edits would preferen-
tially disadvantage the BVI collaborator as their sighted colleagues
could see the impending collisions and take their own steps to avoid
them. CodeWalk’s use of non-interruptible warning messages as
colleagues get too close served to encourage all parties to commu-
nicate using alternate, more accessible, channels (e.g. a concurrent
audio call) in order to appropriately synchronize their edits and
avoid conficts.
7.3 Interdependence
Bennett et al.’s reframing of the goals of assistive technology as
interdependence instead of independence ring true in CodeWalk’s
scenarios [6]. Collaboration between colleagues of mixed abilities
encourages each to play to their own strengths, while requiring
that each cede some of their own power and control to cooperate
efectively with others. Working together in a code walkthrough
or code review, a BVI developer who might have special expertise
in accessibility can disseminate that knowledge to sighted non-
specialists in situ and create a better result for their customers. As
shown by Pandey et al. [53], long-term mixed ability collaborators
establish mutual reliance by learning how to work together by
paying attention, responding to, and adapting to one another’s task-
related behaviors, habits, and needs. CodeWalk’s sound efects and
speech events make a colleague’s navigation and edit work visible
to BVI collaborators, enabling them to be used by a BVI collaborator
as an essential assistive technology for remote collaborative work.
Finally, CodeWalk challenges the established hierarchy of sighted
participants controlling the task, enabling BVI developers to lead
ASSETS ’22, October 23–26, 2022, Athens, Greece
code walkthroughs and reviews instead of meekly defaulting to
follow.
7.4 Researcher Refections
This work improves our own practice to communicate accessibly
and to advocate for our own accessibility when participating in
collaborative software development activities. For example, the
BVI member of the research team now always asks sighted col-
laborators to verbalize code locations. A sighted member realized
that he needed to remember to stop speaking every so often to
allow his BVI collaborator to “read” the code for themselves using
their screen reader. The study coordinator recognized that each
BVI developer’s access and communication needs are diferent and
that expressing these needs can be tricky when collaborating with
someone for the frst time. They have become mindful of attuning
their communication to the preferences of their BVI collaborators.
Finally, as we collaboratively author this paper using the Overleaf
Latex editor, we yearn for it to make use of auditory feedback in
order to fully include our BVI co-author in our writing eforts.
7.5 Future Work
In the future, we would like to explore how to combine the lessons
learned from CodeWalk, CollabAlly, and Co11ab in supporting
mixed ability remote collaboration, whether it be for document
co-editing, software co-editing, or additional collaborative software
development tasks. In particular, future designs should explore
ways that BVI collaborators can most efectively and equitably lead
interactions, in one-to-one and one-to-many scenarios, including
collaborations involving two or more BVI developers.
Many participants said we should ensure that CodeWalk was
accessible to deaf-blind programmers and usable with Braille dis-
plays. They felt that its reliance on the audio medium could exclude
deaf-blind programmers. In future, we will extend our design to
support communication of awareness information through tactile
media.
We recommend that application design standards, such as ATAG
(Authoring Tool Accessibility Guidelines) and WCAG (Web Content
Accessibility Guidelines) be extended to support mixed ability teams
and provide non-visual information about collaborators’ location,
navigation, and edit operations. This could increase the use of
accessibility practices in the design of collaborative authoring tools.
8 CONCLUSION
Existing tools to facilitate tightly-coupled software development
tasks rely on visual cues and create accessibility barriers to equi-
tably collaboration for BVI developers. To address this accessibility
gap, we designed, developed and evaluated CodeWalk, a set of fea-
tures added to Microsoft’s VS Code Live Share extension that makes
a collaborator’s location in a code fle and their actions accessible
through cursor tethering, as well as sound efects and speech. Code-
Walk’s features improved coordination between BVI developers
and their sighted peers while reducing the explicit efort that BVI
developers need to put to stay coordinated. We hope CodeWalk can
serve as an exemplar for IDE manufacturers to make their envi-
ronments more accessible to blind and visually impaired software
developers.
Potluri and Pandey, et al.
REFERENCES
[1] Khaled Albusays and Stephanie Ludi. 2016. Eliciting Programming Challenges
Faced by Developers with Visual Impairments: Exploratory Study. In Proceedings
of 9th International Workshop on Cooperative and Human Aspects of Software
Engineering. ACM, Austin, TX, 82–85. https://doi.org/10.1145/2897586.2897616
[2] Khaled Albusays, Stephanie Ludi, and Matt Huenerfauth. 2017. Interviews
and Observation of Blind Software Developers at Work to Understand Code
Navigation Challenges. In Proceedings of the 19th International ACM SIGAC-
CESS Conference on Computers and Accessibility (Baltimore, Maryland, USA)
(ASSETS ’17). Association for Computing Machinery, New York, NY, USA, 91–100.
https://doi.org/10.1145/3132525.3132550
[3] Ameer Armaly, Paige Rodeghero, and Collin McMillan. 2018. Audiohighlight:
Code skimming for blind programmers. In Proceedings of 2018 IEEE International
Conference on Software Maintenance and Evolution. IEEE, Madrid, Spain, 206–216.
https://doi.org/10.1109/ICSME.2018.00030
[4] Catherine M. Baker, Lauren R. Milne, and Richard E. Ladner. 2015. StructJumper: A
Tool to Help Blind Programmers Navigate and Understand the Structure of Code.
In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing
Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machin-
ery, New York, NY, USA, 3043–3052. https://doi.org/10.1145/2702123.2702589
[5] Florian Beijers. 2019. How to Get a Developer Job When You’re Blind: Advice
From a Blind Developer Who Works Alongside a Sighted Team. https://www.
freecodecamp.org/news/blind-developer-sighted-team/
[6] Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence
as a Frame for Assistive Technology Research and Design. In Proceedings of the
20th International ACM SIGACCESS Conference on Computers and Accessibility
(Galway, Ireland) (ASSETS ’18). Association for Computing Machinery, New York,
NY, USA, 161–173. https://doi.org/10.1145/3234695.3236348
[7] Jacob T Biehl, Mary Czerwinski, Greg Smith, and George G Robertson. 2007.
FASTDash: a visual dashboard for fostering awareness in software teams. In
Proceedings of the SIGCHI conference on Human factors in computing systems.
ACM, San Jose, CA, 1313–1322.
[8] Jefrey P. Bigham, Anna C. Cavender, Jeremy T. Brudvik, Jacob O. Wobbrock, and
Richard E. Ladner. 2007. WebinSitu: A Comparative Analysis of Blind and Sighted
Browsing Behavior. In Proceedings of the 9th International ACM SIGACCESS
Conference on Computers and Accessibility (Tempe, Arizona, USA) (Assets ’07).
Association for Computing Machinery, New York, NY, USA, 51–58. https://doi.
org/10.1145/1296843.1296854
[9] Syed Masum Billah, Vikas Ashok, Donald E Porter, and IV Ramakrishnan. 2017.
Ubiquitous accessibility for people with visual impairments: Are we there yet?.
In Proceedings of the 2017 CHI conference on human factors in computing systems.
ACM, Denver, CO, 5862–5868.
[10] Syed Masum Billah, Donald E Porter, and IV Ramakrishnan. 2016. Sinter: Low-
bandwidth remote access for the visually-impaired. In Proceedings of the Eleventh
European Conference on Computer Systems. ACM, London, UK, 1–16.
[11] Jeremy Birnholtz, Nanyi Bi, and Susan Fussell. 2012. Do you see that I see?
Efects of perceived visibility on awareness checking behavior. In Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems. ACM, Austin,
TX, 1765–1774.
[12] Stacy M. Branham and Shaun K. Kane. 2015. The Invisible Work of Accessibility:
How Blind Employees Manage Accessibility in Mixed-Ability Workplaces. In
Proceedings of the 17th International ACM SIGACCESS Conference on Computers
& Accessibility (Lisbon, Portugal) (ASSETS ’15). Association for Computing Ma-
chinery, New York, NY, USA, 163–171. https://doi.org/10.1145/2700648.2809864
[13] William Buxton. 2009. Mediaspace - Meaningspace - Meetingspace. In Media
Space: 20+ Years of Mediated Life, S. Harrison (Ed.). Springer, London, UK, 217–
231.
[14] William A. S. Buxton. 1992. Telepresence: Integrating Shared Task and Person
Spaces. In Proceedings of the Conference on Graphics Interface ’92 (Vancouver,
British Columbia, Canada). Morgan Kaufmann Publishers Inc., San Francisco,
CA, USA, 123–129.
[15] Code Factory. 2021. Eloquence for Windows. Code Factory. https://
codefactoryglobal.com/app-store/eloquence-for-windows/
[16] Sarah D’Angelo and Andrew Begel. 2017. Improving Communication Between
Pair Programmers Using Shared Gaze Awareness. Association for Computing
Machinery, New York, NY, USA, 6245–6290. https://doi.org/10.1145/3025453.
3025573
[17] Maitraye Das, Darren Gergle, and Anne Marie Piper. 2019. “It Doesn’t Win You
Friends”: Understanding Accessibility in Collaborative Writing for People with
Vision Impairments. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 191
(Nov. 2019), 26 pages. https://doi.org/10.1145/3359293
[18] Maitraye Das, Anne Marie Piper, and Darren Gergle. 2022. Design and Eval-
uation of Accessible Collaborative Writing Techniques for People with Vision
Impairments. ACM Transactions on Computer-Human Interaction 29, 2 (2022),
1–42.
[19] Das, Maitraye and McHugh, Thomas B. and Piper, Anne Marie and Gergle, Darren.
2022. Co11ab: Augmenting Accessibility in Synchronous Collaborative Writing
CodeWalk: Facilitating Shared Awareness in Mixed-Ability Collaborative Sofware Development
for People with Vision Impairments. In Proceedings of the 2022 CHI Conference
on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22).
Association for Computing Machinery, New York, NY, USA, 1–18.
[20] Paul Dourish and Victoria Bellotti. 1992. Awareness and coordination in shared
workspaces. In Proceedings of the 1992 ACM conference on Computer-supported
cooperative work. ACM, Toronto, ON, Canada, 107–114.
[21] Hongfei Fan, Chengzheng Sun, and Haifeng Shen. 2012. ATCoPE: Any-Time
Collaborative Programming Environment for Seamless Integration of Real-Time
and Non-Real-Time Teamwork in Software Development. In Proceedings of the
17th ACM International Conference on Supporting Group Work (Sanibel Island,
Florida, USA) (GROUP ’12). Association for Computing Machinery, New York,
NY, USA, 107–116. https://doi.org/10.1145/2389176.2389194
[22] Github. 2022. Teletype for Atom. Microsoft, Redmond, WA. https://teletype.atom.
io/
[23] Max Goldman, Greg Little, and Robert C. Miller. 2011. Real-Time Collaborative
Coding in a Web IDE. In Proceedings of the 24th Annual ACM Symposium on User
Interface Software and Technology (Santa Barbara, California, USA) (UIST ’11).
Association for Computing Machinery, New York, NY, USA, 155–164. https:
//doi.org/10.1145/2047196.2047215
[24] Geof Greer and Matt Kaniaris. 2020. Floobits real-time collaboration plugin for
Sublime Text 2 and 3. Floobits. https://github.com/Floobits/foobits-sublime
[25] Carl Gutwin and Saul Greenberg. 2002. A Descriptive Framework of Workspace
Awareness for Real-Time Groupware. Computer Supported Cooperative Work
(CSCW) 11, 3 (Sept. 2002), 411–446. https://doi.org/10.1023/A:1021271517844
[26] Lile Hattori and Michele Lanza. 2010. Syde: A tool for collaborative software
development. In Proceedings of the 32nd ACM/IEEE International Conference on
Software Engineering-Volume 2. ACM/IEEE, Cape Town, South Africa, 235–238.
[27] Scott Highhouse. 2009. Designing experiments that generalize. Organizational
Research Methods 12, 3 (2009), 554–566.
[28] Earl W. Huf, Kwajo Boateng, Makayla Moster, Paige Rodeghero, and Julian
Brinkley. 2020. Examining The Work Experience of Programmers with Visual
Impairments. In 2020 IEEE International Conference on Software Maintenance and
Evolution (ICSME). IEEE, Online, 707–711. https://doi.org/10.1109/ICSME46990.
2020.00077
[29] Earl W. Huf, Kwajo Boateng, Makayla Moster, Paige Rodeghero, and Julian
Brinkley. 2021. Exploring the Perspectives of Teachers of the Visually Impaired
Regarding Accessible K12 Computing Education. In Proceedings of the 52nd ACM
Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE
’21). Association for Computing Machinery, New York, NY, USA, 156–162. https:
//doi.org/10.1145/3408877.3432418
[30] Joe Hutchinson and Oussama Metatla. 2018. An Initial Investigation into Non-
visual Code Structure Overview Through Speech, Non-speech and Spearcons. In
Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing
Systems - CHI ’18. ACM Press, New York, New York, USA, 1–6. https://doi.org/
10.1145/3170427.3188696
[31] Jennifer Hyde, Sara Kiesler, Jessica K Hodgins, and Elizabeth J Carter. 2014.
Conversing with children: Cartoon and video people elicit similar conversational
behaviors. In Proceedings of the SIGCHI conference on human factors in computing
systems. ACM, Toronto, ON, Canada, 1787–1796.
[32] Hiroshi Ishii and Naomi Miyake. 1991. Toward an open shared workspace:
computer and video fusion approach of TeamWorkStation. Commun. ACM 34,
12 (1991), 37–50.
[33] JetBrains. 2020. Meet Code With Me (EAP) — a tool for collaborative development
by JetBrains. JetBrains, Prague. https://blog.jetbrains.com/blog/2020/09/28/code-
with-me-eap/
[34] Sasa Junuzovic, Prasun Dewan, and Yong Rui. 2007. Read, write, and navigation
awareness in realistic multi-view collaborations. In 2007 International Conference
on Collaborative Computing: Networking, Applications and Worksharing (Collabo-
rateCom 2007). IEEE, New York, NY, 494–503. https://doi.org/10.1109/COLCOM.
2007.4553880
[35] Claire Kearney-Volpe and Amy Hurst. 2021. Accessible Web Development: Op-
portunities to Improve the Education and Practice of Web Development with a
Screen Reader. ACM Trans. Access. Comput. 14, 2, Article 8 (July 2021), 32 pages.
https://doi.org/10.1145/3458024
[36] Richard E Ladner and Kyle Rector. 2017. Making your presentation accessible.
Interactions 24, 4 (2017), 56–59.
[37] Cheuk Yin Phipson Lee, Zhuohao Zhang, Jaylin Herskovitz, JooYoung Seo, and
Anhong Guo. 2021. CollabAlly: Accessible Collaboration Awareness in Document
Editing. Association for Computing Machinery, New York, NY, USA, 1–4. https:
//doi.org/10.1145/3441852.3476562
[38] Cheuk Yin Phipson Lee, Zhuohao Zhang, Jaylin Herskovitz, JooYoung Seo, and
Anhong Guo. 2022. CollabAlly: Accessible Collaboration Awareness in Document
Editing. In Proceedings of the 2022 CHI Conference on Human Factors in Computing
Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery,
New York, NY, USA, 1–17.
[39] Franklin Mingzhe Li, Di Laura Chen, Mingming Fan, and Khai N. Truong. 2021.
“I Choose Assistive Devices That Save My Face”: A Study on Perceptions of
Accessibility and Assistive Technology Use Conducted in China. In Proceedings
ASSETS ’22, October 23–26, 2022, Athens, Greece
of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama,
Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA,
Article 372, 14 pages. https://doi.org/10.1145/3411764.3445321
[40] Stephanie Ludi, Jamie Simpson, and Wil Merchant. 2016. Exploration of the Use
of Auditory Cues in Code Comprehension and Navigation for Individuals with
Visual Impairments in a Visual Programming Environment. In Proceedings of the
18th International ACM SIGACCESS Conference on Computers and Accessibility
(Reno, Nevada, USA) (ASSETS ’16). Association for Computing Machinery, New
York, NY, USA, 279–280. https://doi.org/10.1145/2982142.2982206
[41] Stephanie Ludi and Mary Spencer. 2017. Design considerations to increase block-
based language accessibility for blind programmers Via Blockly. Journal of Visual
Languages and Sentient Systems 3, 1 (2017), 119–124.
[42] Sean Mealin and Emerson Murphy-Hill. 2012. An exploratory study of blind
software developers. In 2012 IEEE Symposium on Visual Languages and Human-
Centric Computing (VL/HCC). IEEE, Innsbruck, Austria, 71–74. https://doi.org/
10.1109/VLHCC.2012.6344485
[43] Oussama Metatla, Nick Bryan-Kinns, and Tony Stockman. 2018. “I Hear You”:
Understanding Awareness Information Exchange in an Audio-Only Workspace.
In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
(Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New
York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3174120
[44] Microsoft. 2020. Accessibility in Visual Studio Code. Microsoft. https://code.
visualstudio.com/docs/editor/accessibility#_screen-readers
[45] Microsoft. 2020. Microsoft/vscode-a11y - Gitter. Microsoft. https://gitter.im/
Microsoft/vscode-a11y
[46] Microsoft. 2022. Visual Studio Live Share. Microsoft, Redmond, WA. https:
//visualstudio.microsoft.com/services/live-share/
[47] Matthew Miles, A. Michael Huberman, and Michael Saldaña. 2013. Qualitative
Data Analysis: A Methods Sourcebook. Sage Publications, Thousand Oaks, CA.
[48] Lauren R. Milne and Richard E. Ladner. 2018. Blocks4All: Overcoming Accessi-
bility Barriers to Blocks Programming for Children with Visual Impairments. In
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
(Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New
York, NY, USA, 1–10. https://doi.org/10.1145/3173574.3173643
[49] Cecily Morrison, Nicolas Villar, Anja Thieme, Zahra Ashktorab, Eloise Taysom,
Oscar Salandin, Daniel Cletheroe, Greg Saul, Alan F Blackwell, Darren Edge, Mar-
tin Grayson, and Haiyan Zhang. 2020. Torino: A tangible programming language
inclusive of children with visual disabilities. Human–Computer Interaction 35, 3
(2020), 191–239.
[50] Aboubakar Mountapmbeme, Obianuju Okafor, and Stephanie Ludi. 2022. Address-
ing Accessibility Barriers in Programming for People with Visual Impairments:
A Literature Review. ACM Transactions on Accessible Computing (TACCESS) 15, 1
(2022), 1–26.
[51] Katja Neureiter, Martin Murer, Verena Fuchsberger, and Manfred Tscheligi. 2013.
Hand and eyes: how eye contact is linked to gestures in video conferencing. In
CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, Paris,
France, 127–132.
[52] Austin Lee Nichols and Jon K Maner. 2008. The good-subject efect: Investigating
participant demand characteristics. The Journal of general psychology 135, 2
(2008), 151–166.
[53] Maulishree Pandey, Vaishnav Kameswaran, Hrishikesh V. Rao, Sile O’Modhrain,
and Steve Oney. 2021. Understanding Accessibility and Collaboration in Program-
ming for People with Visual Impairments. In Proceedings of the CSCW Conference
on Computer Supported Cooperative Work (Virtual) (CSCW ’21). Association for
Computing Machinery, New York, NY, USA, 30 pages.
[54] Yi-Hao Peng, JiWoong Jang, Jefrey P Bigham, and Amy Pavel. 2021. Say It All:
Feedback for Improving Non-Visual Presentation Accessibility. In Proceedings
of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama,
Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA,
Article 276, 12 pages. https://doi.org/10.1145/3411764.3445572
[55] Venkatesh Potluri, Tad Grindeland, Jon E. Froehlich, and Jennifer Mankof. 2021.
Examining Visual Semantic Understanding in Blind and Low-Vision Technology
Users. In Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery,
New York, NY, USA, 14 pages.
[56] Venkatesh Potluri, Priyan Vaithilingam, Suresh Iyengar, Y. Vidya, Manohar
Swaminathan, and Gopal Srinivasa. 2018. CodeTalk: Improving Programming
Environment Accessibility for Visually Impaired Developers. In Proceedings of
the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC,
Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA,
1–11. https://doi.org/10.1145/3173574.3174192
[57] T. V. Raman. 1996. Emacspeak—Direct Speech Access. In Proceedings of the Second
Annual ACM Conference on Assistive Technologies (Vancouver, British Columbia,
Canada) (Assets ’96). Association for Computing Machinery, New York, NY, USA,
32–36. https://doi.org/10.1145/228347.228354
[58] Gema Rodríguez-Pérez, Reza Nadri, and Meiyappan Nagappan. 2021. Perceived
diversity in software engineering: a systematic literature review. Empirical
Software Engineering 26, 5 (2021), 1–38.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[59] Johnny Saldaña. 2016. The coding manual for qualitative researchers (3rd ed.).
Sage, London, UK.
[60] Stephan Salinger, Christopher Oezbek, Karl Beecher, and Julia Schenk. 2010.
Saros: an eclipse plug-in for distributed party programming. In Proceedings of the
2010 ICSE Workshop on Cooperative and Human Aspects of Software Engineering.
ACM/IEEE, Cape Town, South Africa, 48–55.
[61] Harini Sampath, Alice Merrick, and Andrew MacVean. 2021. Accessibility of
Command Line Interfaces. Association for Computing Machinery, New York, NY,
USA, 1–10. https://doi.org/10.1145/3411764.3445544
[62] Emmanuel Schanzer, Sina Bahram, and Shriram Krishnamurthi. 2019. Accessible
AST-Based Programming for Visually-Impaired Programmers. In Proceedings of
the 50th ACM Technical Symposium on Computer Science Education (Minneapolis,
MN, USA) (SIGCSE ’19). Association for Computing Machinery, New York, NY,
USA, 773–779. https://doi.org/10.1145/3287324.3287499
[63] Freedom Scientifc. 2022. Jaws Tandem Quick Start Guide. https://support.
freedomscientifc.com/JawsHQ/JawsTandemQuickStart
[64] Chirag Shah and Gary Marchionini. 2010. Awareness in collaborative information
seeking. Journal of the American Society for Information Science and Technology
61, 10 (2010), 1970–1986.
[65] Kristen Shinohara and Jacob O Wobbrock. 2016. Self-conscious or self-confdent?
A diary study conceptualizing the social accessibility of assistive technology.
ACM Transactions on Accessible Computing (TACCESS) 8, 2 (2016), 1–31.
[66] Alexa Siu, Gene S-H Kim, Sile O’Modhrain, and Sean Follmer. 2022. Supporting
Accessible Data Visualization Through Audio Data Narratives. In Proceedings of
the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans,
LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA,
Article 476, 19 pages. https://doi.org/10.1145/3491102.3517678
[67] Stack Overfow. 2021. Stack Overfow Developer Survey 2021. https://insights.
stackoverfow.com/survey/2021#section-demographics-disability-status
[68] Andreas Stefk, Andrew Haywood, Shahzada Mansoor, Brock Dunda, and Daniel
Garcia. 2009. Sodbeans. In 2009 IEEE 17th International Conference on Program
Comprehension. IEEE, Vancouver, BC, Canada, 293–294.
Potluri and Pandey, et al.
[69] Andreas Stefk and Richard Ladner. 2017. The Quorum Programming Language
(Abstract Only). In SIGCSE Technical Symposium (Seattle, Washington, USA)
(SIGCSE ’17). ACM, New York, NY, USA, 641.
[70] Andreas M. Stefk, Christopher Hundhausen, and Derrick Smith. 2011. On the
Design of an Educational Infrastructure for the Blind and Visually Impaired
in Computer Science. In Proceedings of the 42nd ACM Technical Symposium
on Computer Science Education (Dallas, TX, USA) (SIGCSE ’11). Association for
Computing Machinery, New York, NY, USA, 571–576. https://doi.org/10.1145/
1953163.1953323
[71] Kevin M. Storer, Harini Sampath, and M. Alice Merrick. 2021. “It’s Just Ev-
erything Outside of the IDE that’s the Problem”: Information Seeking by Soft-
ware Developers with Visual Impairments. In Proceedings of the SIGCHI Con-
ference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21).
Association for Computing Machinery, New York, NY, USA, 12 pages. https:
//doi.org/10.1145/3411764.3445090
[72] Sublime users. 2020. A request for the implementation of accessibility. Issue #3392.
sublimehq/sublime_text. GitHub. https://github.com/sublimehq/sublime_text/
issues/3392
[73] John Tang. 2021. Understanding the Telework Experience of People with Disabil-
ities. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 30 (apr 2021), 27 pages.
https://doi.org/10.1145/3449104
[74] Josh Tenenberg, Wolf-Michael Roth, and David Socha. 2016. From I-Awareness
to We-Awareness in CSCW. Computer Supported Cooperative Work (CSCW) 25, 4
(Oct. 2016), 235–278. https://doi.org/10.1007/s10606-014-9215-0
[75] Christopher Toth and Tyler Spivey. 2018. Documentation NVDA Remote Access.
https://nvdaremote.com/docs/
[76] Jeremy Warner and Philip J Guo. 2017. Codepilot: Scafolding end-to-end col-
laborative software development for novice programmers. In Proceedings of the
2017 CHI Conference on Human Factors in Computing Systems. ACM, Denver, CO,
1136–1141.
[77] Jacob O Wobbrock and Matthew Kay. 2016. Nonparametric statistics in human–
computer interaction. In Modern Statistical Methods for HCI. Springer, Berlin,
Germany, 135–170.
 Designing a Customizable Picture-Based Augmented Reality
Application For Therapists and Educational Professionals
Working in Autistic Contexts
Tooba Ahsen Christina Yu Amanda O’Brien
tooba.ahsen@tufts.edu christina.yu@childrens.harvard.edu amanda_obrien@g.harvard.edu
Computer Science Autism Language Program & Speech and Hearing Biosciences and
Tufts University Augmentative Communication Technology
Medford, MA, USA Program Harvard University
Boston Children’s Hospital Cambridge, Massachusetts, USA
Boston, Massachusetts, USA

Ralf W Schlosser Howard C Shane Dylan Oesch-Emmel

Ralf.Schlosser@Northeastern.edu
Communication Sciences and
Disorders
Northeastern University
Boston, Massachusetts, USA
howard.shane@childrens.harvard.edu dylan.oesch_emmel@tufts.edu
Boston Children’s Hospital Computer Science
Boston, Massachusetts, USA Tufts University
Harvard Medical School Medford, Massachusetts, USA
Boston, Massachusetts, USA

Eileen T. Crehan Fahad Dogar

eileen.crehan@tufts.edu fahad@cs.tufts.edu
Eliot-Pearson Department of Child Computer Science
Study & Human Development Tufts University
Tufts University Medford, Massachusetts, USA
Medford, Massachusetts, USA
Department of Psychiatry
Rush University Medical Center
Chicago, Illinois, USA

ABSTRACT when trying to incorporate picture-based AR in practical therapy

This paper presents the design and evaluation of CustomAR – a
customizable Augmented Reality (AR) application, designed in col-
laboration with therapists, that allows them to create and customize
picture-based AR experiences for use in an autistic context. Using a
2-week diary study, we gauge whether the application’s customiza-
tion options and features are sufcient to allow therapists and
educational professionals to create AR experiences for the various
learning activities they conduct with autistic children, and what
challenges they face in this regard. We fnd that participants think
the application would be suitable for creating AR experiences for a
wide range of learning activities, such as choice-making and teach-
ing daily living skills, and think that the application’s freeze feature
can be helpful when working with children with limited attention.
Towards the end of the paper we discuss the challenges users face
This work is licensed under a Creative Commons Attribution International
4.0 License.
exercises, and how they can be addressed.
CCS CONCEPTS
• Human-centered computing → Mixed / augmented reality;
User studies; • Social and professional topics → People with
disabilities.
KEYWORDS
Augmented Reality; Picture-Based; Marker-Based; Autism Spec-
trum Condition; Education; Therapy; Customization;
ACM Reference Format:
Tooba Ahsen, Christina Yu, Amanda O’Brien, Ralf W Schlosser, Howard
C Shane, Dylan Oesch-Emmel, Eileen T. Crehan, and Fahad Dogar. 2022.
Designing a Customizable Picture-Based Augmented Reality Application
For Therapists and Educational Professionals Working in Autistic Contexts.
In The 24th International ACM SIGACCESS Conference on Computers and
Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New
York, NY, USA, 16 pages. https://doi.org/10.1145/3517428.3544884

ASSETS ’22, October 23–26, 2022, Athens, Greece 1 INTRODUCTION

© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10. Autism Spectrum Condition (ASC) is a neurodevelopmental con-
https://doi.org/10.1145/3517428.3544884 dition that is often characterized by difculty in communication,
ASSETS ’22, October 23–26, 2022, Athens, Greece
(a) The CustomAR Application. The fgure shows the
creation window. Users can associate audio, videos, or
3D models with a target image to create an AR experi-
ence. The white plane to the center-left of the screen
shows the image the user has uploaded. The user has
added a Lego block 3D model to this AR experience and
is customizing its position.
(b) The AR-View window. Users can view their AR ex-
perience by going to the AR-View window. The user
has printed the target image onto a piece of paper, and
is holding up their device camera to the target image to
elicit the associated AR experience. The 3D Lego block
has appeared on their screen.
Figure 1
social-emotional reciprocity, and repetitive behavior [34]. Autis-
tic children1 often have varied learning and behavioral profles –
some may have more difculty with expressive and/or receptive
language, others may have difculty understanding and regulat-
ing emotions and behaviors, and many demonstrate very specifc
interests [34, 52].
While there is no one-size fts all approach to teaching, therapists
and teachers on an autistic child’s support team often adopt learning
strategies that utilize visual supports (pictures, icons, photographs
etc.), as they have been shown to be efective in autistic contexts
[50, 56]. Picture-based Augmented Reality (AR) is a logical extension
to learning activities that rely on visual supports as it allows pictures
to be superimposed with additional information or virtual content
(audio, videos, 3D models etc.) - the virtual content appears on
the screen when the picture comes into view of the user’s device
camera. Augmented Reality, in general, has shown to increase focus
and engagement in children with autism [30] and Picture-based AR,
1A
note on language: Adolescent and adult self-advocates in our studies and our
advisory board have shared that they strongly prefer identity-frst language (e.g.,
autistic person) [15, 46]
Tooba, et al.
in particular, has helped autistic individuals complete daily living
tasks more independently [22] and has helped them improve their
communication [40] and emotion recognition skills [19].
However, practical use of these previously designed AR appli-
cations may be limited in therapeutic or learning environments
because they lack support for customization. Therapists and edu-
cational professionals often need to customize learning exercises
and content according to an autistic child’s interests [37, 41], to
keep them engaged and encourage positive behaviors [16, 57]. Cus-
tomization is key, and the lack of support for customization could
make it difcult for therapists and educational professionals to inte-
grate picture-based AR into the day-to-day learning activities they
conduct. Therefore, there is a gap between research into AR appli-
cations and the practical use of those applications, and a scarcity
of work that bridges this gap.
Therapists and educational professionals are also key players
in the adoption of assistive or learning technology. Prior to intro-
ducing new technology into an autistic child’s educational plan,
they must use their clinical and educational experience to evaluate
the technology and determine how to efectively utilize it in an
intervention [27]. Therefore, to design AR applications that are
useful in practical autistic contexts, it is important for the research
community to understand the customization needs of therapists
and educational professionals, how they envision using AR in the
learning activities they conduct with autistic children, and what
features will be most useful in this regard.
Our contribution is as follows; We frst highlight the design and
implementation of CustomAR; a mobile AR application, created
in collaboration with therapists, that allows them to create and
customize picture-based AR experiences from scratch (refer to fg-
ure 1). We highlight key lessons from each of the two phases of
the design process - from engaging in participatory design with
therapists to create the frst version of the application for a specifc
symbolic development use-case, to modifying the application into
a customizable authoring tool. The latter could potentially allow
therapists and educational professionals to create AR experiences
for a variety of learning contexts, and for a diverse population of
autistic children.
Secondly, we conduct a 2-week diary study and follow-up semi-
structured interviews with special education teachers, occupational
therapists, and speech language pathologists, to gauge whether they
think the application’s customization options and other features are
suitable for use in autism-related learning contexts. Moreover, we
shed light on the challenges that could hamper the use of picture-
based AR in practical therapeutic or educational settings. Our study
reveals the following:
• Therapists/teachers fnd the application’s customization op-
tions sufcient to create AR experiences for a myriad of
learning activities, ranging from choice-making and teach-
ing daily living skills, to teaching lessons on emotion recog-
nition and for collaboration and group-work. Details of these
learning activities and the customization options that sup-
ported their development are provided in section 6.2 and
6.3.
• Therapists/teachers appreciate the ability to ‘freeze’ AR expe-
riences on the screen and think that it would be useful when
Designing a Customizable Picture-Based AR Application For Therapists Working in Autistic Contexts
working with children with limited focus, or when show-
ing AR experiences to children in groups. This is discussed
further in section 6.4.
• Therapists/teachers are concerned about generalizing AR
experiences or creating them ‘just-in-time’ during therapy
sessions (section 6.5). Allowing users to share AR experiences
with others, and recommending AR content to users may
alleviate these concerns. This is discussed further in sections
7 and 9.1 respectively.
Our work provides valuable insight into the features and cus-
tomization options that will help therapists and educational pro-
fessionals introduce AR into the learning activities they conduct.
Although autistic individuals did not take part in this study, our
fndings may be useful for future studies that position autistic
children as content creators and investigate which features/UI are
suitable to allow them to independently create AR experiences. In
the subsequent sections of this paper, we summarize related work,
provide details about the two-phased design process and the user
study, present and discuss our qualitative fndings, and highlight
limitations and future directions for this research.
2 RELATED WORK
This section leverages prior work to highlight how therapists and
educational professionals use visual supports in learning exercises,
how picture-based AR can extend them, and the signifcance of
customization and interest-based learning in autistic contexts.
2.1 Using Visual Supports in Learning
Exercises & How Picture-Based AR Can
Extend Them
Therapists and teachers frequently use visual supports, such as
pictures, symbols or photographs in learning activities involving
autistic children as they often exhibit strong visual processing abil-
ities [50]. For example, teachers or therapists may tape pictures
that represent the day’s activities (visual schedules) to the walls
of a classroom to make it easier for autistic children to transition
from one activity to another [5, 25]. Similarly, they may print pic-
tures depicting the steps involved in a daily living task, and tape
them somewhere in the environment as reminders [1, 50]. Visual
supports are also used during choice-making activities, reinforcer
assessment activities, or activities involving speech and language
[3]
Picture-based AR is a logical extension to some of these learn-
ing activities that rely on pictures/symbols, and can add layers of
information and helpful hints on top of them. When users hold
their devices over a picture to elicit the associated AR content, the
context of the activity (the picture/photograph, the tabletop etc.)
is preserved, and users can view additional information without
shifting their focus away from the task at hand. For example, Cihak
et al. took a traditional tooth-brushing exercise in which a picture
of the steps in the task was taped to the bathroom mirror, and
superimposed the picture with a video clip depicting how each step
was performed. Autistic children accessed this additional informa-
tion by holding their device cameras up to the picture and could
replicate the steps using the toothbrush, toothpaste, and paper cups
in front of them. The study found that autistic children performed
ASSETS ’22, October 23–26, 2022, Athens, Greece
the tooth-brushing task more independently when AR was used
[22]. Prior studies have also used picture-based AR to enhance the
learning efects of social stories, [20], to teach emotion recogni-
tion [17, 19], and to improve children’s communication abilities by
reinforcing the meaning of pictures/symbols found in the Picture
Exchange Communication System (PECS) [40, 47]. Picture-based
AR, therefore, can be an efective teaching tool in an autistic context
and as such it is important to understand what tools and features
could facilitate therapists and teachers in using picture-based AR
during the learning exercises they conduct.
2.2 Autism, Interest-Based Learning &
Customizable AR
Studies have shown that customizing learning exercises accord-
ing to the unique learning needs and interests of autistic children
[33, 35] leads to positive behaviors [16, 28, 57]. The ability to cus-
tomize, therefore, may be invaluable to therapists and educational
professionals working with autistic individuals. However, a recent
survey study revealed that current computer-based technologies de-
signed for autistic individuals lack robust support for customization
[44]. This includes studies under the umbrella of ‘augmented and
virtual reality’ - although AR interventions have shown to support
autistic individuals in various domains [38, 39, 45] including com-
munication [40, 58, 59], daily-living [22], pretend play [12, 13, 26],
and social and emotional learning [29, 43, 54], research into cus-
tomizable AR is lacking [44].
Very few studies have provided users with the ability to cus-
tomize picture-based AR experiences [24, 47], and even then users
can only customize within the confnes of an activity, e.g., being
able to change the virtual content associated with predefned PECS
symbols, but being unable to create an experience around new sym-
bols [47]. Moreover, there is only one AR application on the Google
play store (for android devices) that allows users to create their own
picture-based AR experiences [2], and this application has not been
created with the needs of therapists/teachers and autistic children
in mind.
In contrast, we involved therapists in the design process and
drew on their clinical experience in working with autistic children.
Subsequently, we created an application that not only gives them
the agency to create and customize picture-based AR experiences
from scratch using their own images and virtual content, for learn-
ing activities that they themselves have conceptualized, but that
also contains features that could potentially facilitate the use of
AR in autistic contexts. Moreover, we investigate the practical chal-
lenges they could encounter when trying to inculcate picture-based
AR in autism-related learning activities - something that no prior
work has looked into thus far.
3 RESEARCH GOALS & HIGHER LEVEL
RESEARCH QUESTIONS
Our frst goal is to provide therapists and educational profession-
als with an application that allows them to create and customize
picture-based AR experiences from scratch. Section 4 describes
the two design phases that culminated in the development of our
customizable AR application.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Our second goal is to evaluate this application by conducting user
studies with therapists and educational professionals. We aim to
understand; (1) whether therapists can sufciently use the applica-
tion’s creation and customization features to create AR experiences
for their autistic students; (2) whether the freeze feature – a feature
that was developed based on feedback from therapists during the
design phase (section 4.2.3), and which allows them to freeze AR
experiences on the screen while keeping the experiences interactive
– is useful in the context of autism; (3) what challenges therapists
face when trying to integrate picture-based AR in the learning
activities they conduct.
The specifc research questions (RQ) are listed below:
• RQ 1: Are the application’s customization options suf-
fcient to allow therapists/educational professionals to
create AR experiences to meet the varied interests and
learning needs of the autistic population they work
with?
• RQ 2: How does the application’s freeze feature facili-
tate the use of AR in the context of autism?
• RQ 3: What are the challenges in using picture-based
AR in practical therapy/learning settings?
Section 5 highlights the results of the user study we conducted
to answer these research questions.
4 APPLICATION DESIGN
We designed and developed our application in two phases. In phase
I (section 4.2), we designed the application in collaboration with 4
researchers/Speech Language Pathologists (SLPs) from the Boston
Children’s Hospital’s outpatient clinic in Waltham, Massachusetts.
The goal was to use AR to enhance the symbolic knowledge for the
moderate to severely autistic children that visited the clinic. Our
collaborating SLPs informally tested the application and realized
that the lack of customization options was hindering its use. We
then re-designed the application in phase II (section 4.3), to allow
users to create/customize picture-based AR experiences for a variety
of learning contexts and autistic individuals, not just those specifc
to the clinic. In this section, we highlight our design journey; We
describe our initial symbolic development use-case, features of the
frst prototype, and the lessons we learnt that spurred the redesign
of the application into a more customizable tool. We then describe
the features of the latest version of the application.
4.1 The Design Team & Participatory Design
Process
Our multi-disciplinary design team comprised of computer scien-
tists and 4 researchers/SLPs from the Boston Children’s Hospi-
tal’s outpatient clinic in Waltham, Massachusetts. These clinicians
primarily work with minimally verbal children with moderate to
severe autism and provide diagnostic, and speech and language
evaluations and treatments. At the time of introduction the SLPs
were exploring AR in play contexts, but had no prior experience
with picture-based AR.
The design process was as follows; the frst author visited the
clinic weekly for a month, and then bi-weekly over the next 3
months for design meetings. The frst month was dedicated to
outlining the features of the application, and the content of the AR
Tooba, et al.
experiences (e.g., the target images and 3D models used, the audio
and animations to create for each model etc.). Over the remaining
months, the frst author coded the application’s features and UI.
When a feature was coded, the frst author would demonstrate
the feature during the bi-weekly in-person meetings. The SLPs
would provide feedback and suggestions for improvement or re-
design, and modifcations would be made accordingly. The SLPs
also informally tested the application amongst themselves (at the
end of Phase I) and provided feedback that spurred the redesign of
the application into a more cutomizable tool (Phase II).
4.2 Phase I - Designing an AR Application for a
Specifc Clinical Context
4.2.1 Symbolic Development Use-case: In phase I, the goal
was to design an application that could enhance symbolic knowl-
edge for the minimally verbal children with moderate to severe
autism, that visited the clinic. The SLPs often centered therapy
around play, and these children were familiar with numerous toys
present at the clinic.
Let’s consider a common scenario at the clinic; Sam is a 7 year
old male who regularly visits the clinic. He has moderate-severe
autism and very little functional speech (a few intelligible spoken
words). Therapists at the clinic want to teach Sam how to requests
for objects (toys, food etc) by pointing towards pictures of those
objects, instead of trying to grab the objects themselves, physically
dragging an adult towards where the objects are being kept, or
showing disruptive behavior. When the therapist presents Sam
with two objects that he likes, a toy (a Lego block) and a snack
(goldfsh crackers), he consciously points towards the object he
wants. The therapist then puts the objects out of sight, and presents
Sam with a picture of the Lego block and goldfsh crackers instead.
Sam is unable to point towards the picture of the object he wants. He
does not understand that the 2D pictures are symbols that represent
the 3D objects.
Since pictures and symbols are key to numerous alternate com-
munication strategies [14, 31, 32, 49], it is important for these chil-
dren to understand the symbolic relationship between 2D pictures
and their 3D referents. In our example, Sam does not have a clear
understanding of this relationship. We hypothesized that if we
could momentarily convert the 2D into 3D, that is, superimpose
the 2D pictures of the objects with highly similar 3D models of the
objects, Sam may understand the 2D-3D symbolic relationship and
be able to make a conscious choice between the available options.
4.2.2 Content & Features of the Application:
Basic AR Experiences: The fst version of the application was
very simple. We pre-programmed a few images and corresponding 3D
models into the application (details present in the next bullet point).
Therapists would open the application on an iPad and hold the device
camera over a print-out of one of the pre-programmed images. The
associated 3D models would then appear on the screen. Figure 2 shows
a picture of Lego blocks and goldfsh crackers superimposed with
3D models of the same objects. Names of the objects would appear
on small banners beside the 3D models. When the 3D models were
tapped, an audio recording of the object’s name would play out. It
is important to note that these AR experiences were built into the
Designing a Customizable Picture-Based AR Application For Therapists Working in Autistic Contexts
(a) Target image of Lego block and
Goldfsh crackers
(b) 3D models of the Lego block and
goldfsh crackers appearing over the
target image
Figure 2: Basic AR Experience
application. The therapists could not alter the target images or the 3D
models associated with them.
3D Objects and Target Images: The initial application was
designed while keeping in mind the objects (e.g., toys) that the children
at the clinic interacted with the most. The design team picked 6 objects
- a bubble wand, a ‘Thomas the train’ toy, a large Lego block, a packet
of goldfsh crackers, a soccer ball and a pair of socks. Since the children
were prompted to make a choice between 2 objects at a time, we took
images of pairs of objects. These served as the 2D images the children
would be presented with and the target images that the application
would recognize. Examples of diferent pairs are shown in Figure 3.
Animations: Based on discussions with the therapists, we cre-
ated an animation for each 3D model to promote engagement. For
example, bubbles would appear in front of the bubble wand, the Lego
blocks would start stacking on top of each other, and a few pieces of
goldfsh crackers would appear outside their packet. (Figure 4a). Each
animation was paired with relevant audio, such as a popping sound
for the bubble wand, a stacking sound for the Lego blocks and the
sound of a packet opening for the goldfsh crackers.
ASSETS ’22, October 23–26, 2022, Athens, Greece
(a) Bubble wand and ‘Thomas’
(b) Socks and soccer ball
Figure 3: Target Images
Hiding Target Images: Sometimes when 3D models were super-
imposed over target images, there would be too many elements on the
screen. Some individuals on the spectrum could perceive this as visual
clutter or sensory overload [21]. To minimize this, we added a ‘hide’
feature to the application. A white plane would appear over the target
image (Figure 4b), temporarily hiding the target image and drawing
focus to the 3D models on top.
4.2.3 Informal Testing & Design Considerations Moving For-
ward: Our collaborating SLPs informally tested the application
amongst themselves and provided feedback. They were given a
hi-fdelity prototype of the application and instructed to trigger the
various in-built AR experiences, try out the animations for each 3D
model, and comment on the strengths and weaknesses of the appli-
cation. After using the application for several months, and due to
the overnight shift to remote learning practices due to the Covid-19
pandemic [53], the SLPs faced various challenges. These challenges,
and their implications on application design, are outlined below.
Predefned Content and Varied Interests - The Need for
Customization Options: We tried to ensure that the target im-
ages and 3D models found within the application were representative
of (and highly similar to) the objects found at the clinic that were
interesting to the clinic’s autistic population. However, the move to
ASSETS ’22, October 23–26, 2022, Athens, Greece Tooba, et al.

(a) Animations - Lego blocks stack and gold-

fsh crackers appear
(a) Creation window

(b) Target image hidden by creating a white

plane over the image
(b) Users can use the slide-in menu to add content to an AR
Figure 4: Animations and ‘Hide target Image’ Feature experience

online therapy due to the Covid-19 pandemic presented a challenge.

If a child did not have these specifc objects at home or did not fnd
them interesting, the application would not be engaging/useful for
them [41]. This feedback made it clear that our AR application had to
be more customizable, in order to be usable outside the clinic and to
cater to children’s diverse interests.
Dynamic Therapy Environments and Children’s Limited
Focus - The Need to Freeze Content: Owing to the nature of AR
technology, the AR experience is only visible on the screen when the
device camera has the target image in its sight. However, constantly
holding the iPad over the target image proved to be difcult for the (c) The selected target image appears in the middle of the
SLPs, as therapy environments are dynamic; the child may be unable workspace
to sit at a table (and therefore near the target image) for long periods
of time, or they may get distracted by another item in the room and Figure 5
wander of. The therapist may have to abandon the iPad to follow
after the child and may fnd it difcult to make the child sit still
long enough to trigger the AR experience again. We therefore needed experiences according to the needs of the autistic population they
to ‘freeze’ the AR experience on the screen, that is, retain the AR work with. We also added the freeze feature to make it easier to
experience on the screen even after the target image was no longer in view a completed AR experience. In this section, we provide details
view of the device camera. about these features and the UI of the application.
4.3.1 Customization Options:
4.3 Phase II - Re-designing the Application to Users can avail the following customization options:
Make it More Customizable • Custom Target Images - Users can select custom images
In Phase II of the design process, we added various customization to serve as targets for an AR experience by either taking
options to the application to give therapists and educational pro- an image in real time, or uploading an image from their
fessionals the ability to create and customize picture-based AR device gallery. This gives each user the fexibility to create
Designing a Customizable Picture-Based AR Application For Therapists Working in Autistic Contexts ASSETS ’22, October 23–26, 2022, Athens, Greece

AR experiences around the specifc visual supports they use

during learning/therapy exercises.
• Audio Prompts - Users can record an audio prompt for
an AR experience. The audio prompt plays automatically
when the target image is recognized by the application. Since
users will decide on the contents of the audio recording
themselves, they may choose to record short hints or longer,
more detailed prompts, depending on the activity they have
in mind.
• Videos - Users can associate videos with an AR experience
by either recording them in real time, or uploading videos
from their device gallery. We posit that this may be an im-
portant customization option as video modelling is widely
(a) Users can upload existing videos or take them in real
used as an instructional method when working with autistic time. Tapping on the video object (shown in the center of
children [11, 23, 51]. the screen), plays or pauses the video.
• 3D Models & animations - Users can add one or more
3D models to their AR experience from an in-built library
of 3D models that contains 40 models spanning the cate-
gories of food, hygiene, toys, etc. Each model has a default
audio prompt (the model’s name) and at least one default
animation associated with it. Users can record their own
audio prompts for a model, and choose between animations
if more than one is available for a particular model. Although
we initially added 3D models because they were essential
for the symbolic development use-case described in section
4.2.1, we were interested to know how users utilize them in
other use-cases or learning activities.
To simplify the process of creating and viewing an AR experience, (b) Users can select one of the two animations (stacking or
we decoupled the creation and viewing windows. The customiza- lining up the Lego blocks), using the animation window
shown on the right.
tion options described above can be accessed through the creation
window. The two windows are described below.

4.3.2 Creation Window (Authoring Tool):

The creation window (Figure 5a) allows users to create their own
picture-based AR experiences from scratch. The user must frst
upload a target image, which appears in the middle of the work-
space (Figure 5c). Users can use the slide-in menu to add either
audio, video, 3D models, or a combination of the three, to any AR
experience (Figure 5b). The creation window shows what the target
image and virtual content would look like if users were viewing
them head-on, instead of from above. This is similar to how the AR
experience will appear in the AR-view and helps users adjust the
position of the virtual content in relation to the target image.
(c) Users can re-position, re-size and rotate 3D models and
videos, using the window on the right.
Miscellaneous options: Users can change an animation asso-
ciated with a 3D model (Figure 6b), and can re-position, resize, and
Figure 6
rotate video objects or 3D models within their AR experience (Figure
6c). They can also clear the work-space, load/edit previously created
AR experiences, and shift the camera to look at the work-space from cameras (Figure 7). This window contains features that facilitate
above. Moreover, they can temporarily hide content that is present the use of AR within therapeutic/learning contexts.
within their AR experience to facilitate adding and customizing more
content. Users can also access video-based tutorials that explain how Freeze Feature: The freeze button allows users to freeze the video
to create AR experiences and how to use all the miscellaneous fea- background on the screen (Figure 7a), while keeping the AR content
tures/options just described. interactive. This allows them to retain the context of the AR experience
(the video background and the target image), and move the device
4.3.3 AR-View: The AR-View is where users can view their AR away from the target image without the AR experience disappearing.
experiences by bringing their target images in front of their device This could be useful in dynamic therapy/learning environments where
ASSETS ’22, October 23–26, 2022, Athens, Greece Tooba, et al.

a child is unable to stay near the tabletop. The therapist could activate
the AR experience, freeze it, and then bring the iPad/device closer to
the child for better viewing. Prior work has also shown the efectiveness
of such a feature in mobile-AR contexts [42].

Setings Menu & On-screen Butons: The settings menu allows

users to change the speed of animations and the light intensity, hide
target images and switch between ‘sound mode’ and ‘animation mode’
for the 3D models (Figure 7c). The latter determines what happens
when a user taps a 3D model in an AR experience - either the associated
audio or the chosen animation plays out. In situations where a user
has added more than one type of content (audio, video, 3D models)
to an AR experience, they can toggle through the content using the
(a) The 3D Lego block is superimposed over the target im-
‘Next’ and ‘Prev’ prompt buttons present on the bottom of the main age. The Freeze and Next and Prev prompt buttons are
screen (Figure 7a). By default, the 3D model appears frst, then video, present at the bottom of the screen
and then audio. Users can reverse this order using the settings menu.

This version of the application, at the end of phase II, was used
when conducting formal user studies with therapists and educational
professionals (section 5). .

4.4 Implementation Details

The application was created using Unity [55] and C# scripting. We
used the ARCore [8] and ARKit [36] XR plugins to provide the
necessary AR functionality for both the android and iOS version
of application respectively. We used the interface provided by AR
Foundation [6], within Unity, to communicate with these two plu-
gins. This allowed us to use the same code-base to build both an (b) A video object superimposed over the target image
Android and an iOS version of the application.

5 USER STUDY
To answer our research questions (section 3), we conducted a user
study that was approved by our university’s Institutional Review
Board. Recruitment was done over the course of fve months, from
June 2021 to October 2021. The details are as follows:

5.1 Participants
We recruited participants using word-of-mouth and snowball sam-
pling. All our participants were therapists or educational profes-
sionals who had prior experience working with autistic individuals.
(c) The diferent settings options
We recruited a total of 10 participants in the study; 7 participants
completed the study, 1 participant withdrew because she was not
Figure 7
comfortable with downloading the application on her personal
device, and 2 of them did not complete the study.
For simplicity, we refer to participants using ID numbers P1 to
P7. P1 and P2 belonged to the same clinic as the SLPs that helped Please note, no autistic children participated in this study. Our
design the frst version of the application (Phase I). However, these goal was to get feedback from therapists and educational profes-
participants were not present during the design discussion in phase sionals frst and refne the application, before testing it with autistic
I and did not have access to the application prior to their enrollment children in future studies.
in the study. Since the frst version of the application was created
for a specifc use-case for the clinic, we wanted to see how the latest, 5.2 Study Method
more customizable version could be useful for the same use-case. Each participant was enrolled in a diary study spanning 2 weeks.
Recruiting therapists who had negligible involvement in the design Participants frst took part in a one-on-one information session
and creation of the application, but were familiar with the use-case, via Zoom [9], where they were introduced to the application and
allowed us to get an unbiased opinion. downloaded it on their preferred devices. They were instructed to
Designing a Customizable Picture-Based AR Application For Therapists Working in Autistic Contexts
use the application over the next 2 weeks to create their own AR
experiences, and provide feedback about its strengths/weaknesses
and what learning activity or contexts of use they had in mind. We
built feedback options into the application to encourage partici-
pants to provide feedback. The types of feedback they provided are
discussed in section 5.4.
At the end of their 2 weeks, participants took part in a one-on-one
semi-structured interview with a member of the research team. This
helped us get an in-depth understanding of how the participants
used the application and in what contexts, what challenges they
faced, and what improvements they would like to see. At the end
of the interview, participants were asked to permanently delete the
application from their devices.
5.3 Procedures
5.3.1 Information Session: In the information session, a mem-
ber of the research team frst explained what picture-based aug-
mented reality was and then played demo/screen capture videos
of the application to explain its layout and the features. Interested
participants were taken through an informed consent process in
the presence of a witness. The participants then downloaded the
application on their preferred devices. All our participants used
iOS devices. For iOS users, we distributed the application through
Apple’s Testfight [7].
5.3.2 Diary Study and Follow-up Interview: During the 2-week
diary study, we sent emails twice a week to remind and encourage
our participants to continue testing the application and provide
feedback. At the end of the 2 weeks, we conducted a one-on-one
interview with them via Zoom. These interviews typically lasted
an hour and were either audio or audio/video recorded based on
the participants’ preferences (indicated in the consent forms).
5.4 Data Collection & Analysis
5.4.1 Types of Data Collected: We collected the following qual-
itative data:
• Written feedback - Participants provided written feedback
by flling out a short Qualtrics survey [4] that opened up
in a browser when they clicked the ‘Survey’ button in the
application. This short survey asked participants to indicate
which features they used, how they used them, and any other
comments they had.
• Voice notes - Participants provided verbal feedback through
the application by recording voice notes and pressing the
‘Send Voice Note’ button to upload them to an AWS simple
storage bucket [10]. Similar to the survey, participants had to
elaborate on which features they used, and in what contexts.
• Interview recordings - The audio/video recordings of in-
terviews were manually transcribed. In the interviews, we
asked our participants to describe their overall experience
with the application, what AR experiences they created and
for which learning exercises, what features may be more
useful in autistic contexts etc.
5.4.2 Analysis Methods: We performed a thematic analysis on
the written feedback, and the transcripts of the voice notes and
interview recordings. Braun & Clarke’s [18] approach to thematic
ASSETS ’22, October 23–26, 2022, Athens, Greece
analysis was used. A member of the research team frst transcribed
the data, generated initial codes, gathered data/quotes pertaining
to each code, collated them into themes, discussed the themes with
other members of the research team, and then reviewed and refned
them. A second coder (from outside the initial study team) validated
the themes and the data belonging to each particular theme.
6 RESULTS
We asked our participants to explore the application while keeping
in mind the autistic population they work with, and the learning
exercises they conduct. As such, some participants only explored
features relevant to the context they were developing their AR
experiences for, while others explored as many features as possible.
In this section, we frst highlight the participants’ general experi-
ence/progress with the application and provide details about the
learning activities they had in mind. We then analyze their feedback
and interview responses to answer our research questions.
6.1 Participants’ Progress With the Application
All our participants encountered a learning curve when testing
the application, but some made more progress than others. These
participants either got better after repeatedly using the application,
were able to intuitively fgure out what to do, or took advantage of
the in-app tutorials.
6.1.1 Geting Beter Afer Repeated Use: Some participants
mentioned getting better at using the application over time. For
example, P1 said that she was able to ‘fgure things out’ after she
‘played around with it’ and that with repeated use, she would be
‘more comfortable’ with the application. Similarly, P2 got faster at
using the application over time. She stated, ‘I defnitely got faster at
it. And so once I was able to do it like a couple times, it was pretty
seamless.’
6.1.2 Using One’s Intuition: P5 mentioned that the initial tuto-
rial in the one-on-one information session was enough for her to
use the application ‘independently’, and she ‘didn’t really have to
go back to the tutorials that are built in’. She thought the buttons
‘were clear’, ‘made sense’ and if she was not sure what to do then
as an ‘intuitive user of apps like this’, she was able to ‘fgure it out’.
6.1.3 Leveraging the In-app Video Tutorials: P1 mentioned
that the in-app tutorials were ‘defnitely helpful’ and P7 said that
they were ‘great reminders’. P6 had trouble remembering what
the miscellaneous buttons on the screen did (e.g. the ‘load’ but-
ton) and sought help from the in-app tutorials. She stated that she
was not ‘good at remembering steps’ so she watched the tutorials
multiple times, which were ‘very clear and very easy to understand’.
Participants who made less progress either failed to notice the
in-app video tutorials, or were unclear on how to trigger the AR
experiences they had created.
6.1.4 Lack of Prior Experience With AR & Failure to Notice
In-app Tutorials: P4 enjoyed ‘exploring’ the application, but was
‘new to AR’ and so ‘had some difculty’ with creating full AR
experiences and viewing them. She did not notice the ‘tutorial’
button in the application and was therefore unable to get help.
ASSETS ’22, October 23–26, 2022, Athens, Greece
6.1.5 Confusion About How to Trigger an AR Experience:
P3 misunderstood how to trigger an AR experience and instead
of using the target images to trigger the experience, she tried to
use the actual objects she took images of. For example, instead of
holding her camera to the picture of her driveway, she held it up to
the actual driveway and had a hard time making sure the camera
was in the ‘right place’ to trigger the experience. This negatively
impacted her progress with the application. Similarly, P6’s AR ex-
periences did not appear on the screen because her target images
were so big that the entire image was not in view of the device
camera. We identifed and corrected this issue during the interview,
and she was able to trigger her AR experiences right then. That
said, both participants provided feedback based on the features they
successfully explored, and brainstormed improvements. Despite
encountering issues, P6 ‘really liked’ using the application and, at
the end of the study, said that it was ‘too bad’ she could not have it
again.
In section 7.3 we discuss some improvements we can make to
help reduce the application’s learning curve.
6.2 Details of Learning Activities
Our participants envisioned creating AR experiences for a range of
learning activities. Table 1 provides background information about
each participant, the features they explored, and what learning
contexts they had in mind.
P1 and P2 considered the same symbolic development use-case
that was described in section 4.2.1 – they wanted to help children
make intentional choices using pictures of objects (toys, food etc.)
by superimposing them with 3D models of the objects to strengthen
the link between 2D images and their 3D referents. P3 wanted to use
the application to give children situational cues – such as how to
behave when entering a classroom. P3 and P4 both wanted to teach
daily living skills, such as ‘hair-brushing’ or tooth-brushing’ by
using the application to superimpose pictures representing the steps
of an activity with videos of the activity being performed. Children
would trigger these videos when they forgot the steps in a task. P5
was confdent that her high-school students could independently
use the application during group-work to create an AR battle game
using picture cards. Students would collaborate to make the rules of
the game, would decide which pictures to use, and would associate
actions with each picture (using audio or 3D models). Players could
trigger these actions by holding their iPads over each picture. This
spurred the discussion on whether any changes need to be made to
the UI so that autistic children could create AR experiences on their
own (section 7.4). P6’s school used worksheets (called ‘News2You’)
that paired action words, such as kicking, with pictures depicting
the action. She wanted to superimpose these static pictures with
videos showing what the actions looked like. Lastly, P7 envisioned
using the application to give ‘reminders or second prompts or visual
cues’ to children during group activities or if a paraprofessional
was not nearby to repeat instructions. He also wanted to reinforce
the visual schedules in his classroom with audio prompts. Children
could then ‘independently go up and check the schedule’ by holding
their iPads over the images in the schedule.
Tooba, et al.
6.3 Participants’ Experiences With The Various
Customization Options
As mentioned in 6.2, our participants envisioned using the appli-
cation for a plethora of learning activities, for autistic children
ranging from younger kids with severe autism, to those who were
independently navigating high-school. In this section, we discuss
our participants’ experiences with the application’s customization
options and answer RQ 1.
6.3.1 Recording Custom Audio: Participants could record au-
dio prompts that would either play automatically when a target
image was recognized, or would play when the associated 3D model
was tapped. P1 added the sound of a train honking from a Youtube
clip to the 3D model of ‘Thomas the train’ for her choice-making ac-
tivity and said that she could see ‘a kid really enjoying it because it’s
so realistic’. P5 highlighted that children ‘may be stimulated by hav-
ing their voice recorded’ and wanted her students to record audio
prompts for actions associated with each card in their battle game.
For example, a card could say, ‘You lost fve points’. P6 and P7 used
videos for their primary use-cases (teaching language and creating
reminders), but used audio for secondary activities. They imagined
using audio to pair instructions for a math lesson, or sounds of
musical instruments with worksheets and pictures respectively.
Children could hold up their devices to the worksheets/pictures to
listen to the instructions or the musical notes.
6.3.2 Adding A Custom Video: Participants liked the option to
add videos into the AR experience because videos either grasp a
child’s ‘attention’ better than ‘static’ content (P1) or because autistic
individuals ‘imitate video models more’ than if a person performed
the same action in front of them (P2). The latter especially could
help with teaching daily living skills. P2 added a video of someone
brushing their teeth, over an image of a tooth brush, to teach this
daily living task and P6 imagined using videos to demonstrate
the meaning of action words and emotions. For example, when
conducting a lesson on frustration, she could superimpose an image
of a frustrated kid with a video showing what ‘frustration looks
like’. P7’s goal was to foster independence in the children he worked
with - If stuck, they could hold their iPads up to their worksheets
to trigger a video of P7 giving them instructions, such as ‘please
write three complete sentences’. While an audio prompt may have
achieved the same purpose, it seemed the participants valued the
visual nature of videos. Participants who had trouble using this
feature (P4) or who found it irrelevant to their use-case (P5) also
thought that it had ‘potential’.
6.3.3 Adding 3D Models: Some participants used 3D models as
an integral part of their AR experience (e.g., P1 and P2 used the
train, bubble wand, Lego block etc. for choice making activities),
while others explored them out of curiosity (P3 to P7). P5 and P6
liked the ability to add multiple 3D models to an AR experience - P5
could then use diferent 3D models at the same time to demonstrate
a theme, for example adding a cofee cup and toast together to rep-
resent ‘breakfast’, and P6 could add multiple 3D models to an AR
experience for a math activity (e.g., 2 + 5). That said, participants
agreed that having ‘more options’ (P1) or a ‘broader range’ (P2) of
3D models would be helpful. P2 worked with kids‘ in play contexts’
and would have liked to see 3D models of other toys and P5 said
Designing a Customizable Picture-Based AR Application For Therapists Working in Autistic Contexts ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Participants’ Backgrounds

ID Participant Occupa- Population in Mind Learning Activity Features Tested Intended Users of the Applica-
tion tion
P1 Speech Language Younger kids with Choice making & sym- Audio, Video, 3D The therapist holds the device
Pathologist moderate-severe autism bolic development models (a few) & an- and shows the screen to the
who are minimally verbal imations, Freeze child, the child may interact
with the virtual content them-
selves or with the therapist’s
help
P2 Speech Language Younger kids with Choice making & Audio, Video, 3D Same as above
Pathologist moderate-severe autism symbolic development, models (a few) & an-
who are minimally verbal teaching daily living imations, Freeze
skills
P3 Speech Language 7th/8th graders with Teaching daily living Video, 3D models (a The therapist creates the AR ex-
Pathologist Autism who have artic- skills few) periences beforehand, and the
ulation or signifcant kids use the AR-View to view
communication difculties and interact with the virtual
content independently
P4 Special Education High-school kids with Teaching daily living 3D models —
Teacher moderate-severe Autism skills
P5 Speech Language High-school kids who are Group work / collabo- Audio, Video, 3D The therapists or the kids in-
Pathologist independently navigating a rative learning, creating models (a few) & an- dependently create their own
public school setting AR games together imations, Freeze AR experiences and use the AR-
view to show others what they
have created
P6 Occupational Ther- 14 y/o with Down’s syn- Worksheets with action Audio, Video, 3D The therapist creates the AR
apist drome & a child who suf- sentences, connecting models (a few) & an- experiences, and the kids view
fered stroke in utero - kids words and language, imations them independently
who have difculty speak- teaching emotions,
ing, low cognition and are teaching math concepts
emerging readers
P7 Special Education autistic children in 4th Giving children re- Audio, Video and The therapist creates the AR
Teacher grade minders or directions 3D models (a few) experiences, and the kids view
about individual or them independently
group-related class
activities, visual
schedules and pairing
musical notes with
images of musical
instruments

that categories related to daily living, such as ‘transportation’ or
‘morning routine’, and for her use case, categories like ‘battle ships’
would be helpful. P7 typically puts clipart on his slides to show chil-
dren what they need for an activity and could have done the same
with AR if 3D models related to school supplies had been available
in the in-app library, such as rulers, pencils, and crayons. Therefore,
providing users with the ability to add 3D models themselves or
search from a larger database of 3D models would be helpful. We
discuss this more in section 6.5.1.
6.3.4 Using Animations: Some 3D models had simple anima-
tions, such as an apple rotating, while others had functional anima-
tions, such as a pair of hands getting soap from a soap dispenser.
Participants found functional animations - animations that show
the function of a particular 3D model – to be more useful. For ex-
ample, if the toy train had an animation of the train gliding over,
P1 could use it to teach concepts like ‘pushing the train’. Similarly,
P2 stated that the animations would be great for teaching ‘choice-
making’, as they depict the function of the object. Although the
3D models of day-to-day objects, such as the toothpaste, had func-
tional animations, P2 thought that having a sequence of animations
that depict the item’s use (such as all the steps involved in putting
toothpaste onto a toothbrush) could be helpful in teaching daily
living skills. Thus, while simple animations attract children’s atten-
tion, functional animations are more useful from a teaching point
of view. While animations may not be useful for every child - P3
stated that they would not be necessary for the 7th/8th graders she
works with – they can be ‘really important to have’ when teaching
ASSETS ’22, October 23–26, 2022, Athens, Greece
children with severe autism, or to make the application useful for
‘a wide variety of clients’ (P3).
6.3.5 Are The Given Customization Options Suficient? Re-
call that RQ1 considered whether the application’s customization
options were sufcient to create AR experiences for a range of autis-
tic children and learning exercises. While participants suggested
improvements for existing customization features, they did not
fnd anything ‘missing’, and although participants were consider-
ing diferent learning activities, they were each able to fnd some
customization options that were useful for their context. P6, for
example, stated the following, ‘I could defnitely use the audio. I
could defnitely use video. I could defnitely use an image. I can use
all your features to teach a lesson on an emotion’. There is, however,
a question of how much customization is enough? P5 thought the
application provided a good balance of ‘fexibility’, without the
‘possibilities being endless’. P5 envisioned a scenario where her
students would create AR experiences themselves, instead of the
teacher/therapist. To that end she stated that some previous tools
she had used with her students (e.g., 3D printing software) had ‘too
many options’, and students got ‘bogged down in the details’, were
‘too frustrated’, or ‘working too independently without consulting
each other’, thus defeating the purpose of group exercises. Our
application was simple enough that she envisioned her students
‘running with it’ and even ‘coming up with something more cre-
ative’ than her game idea, but without getting too bogged down.
That said, a user may come up with a unique or niche use-case
which requires more customization than what is currently available
in the application. Future studies will need to investigate this. In
the next subsection, we attempt to answer RQ 2.
6.4 Can the Freeze Features Facilitate the Use
of AR in an Autistic Context?
Our second goal was to understand whether the freeze feature could
facilitate the use of AR with autistic individuals. Participants liked
the freeze feature because it removed the need to continuously
hold the iPad/device over a target image to keep an AR experience
interactive. P5 saw its utility in group settings - she could ‘capture’
the AR experience on the screen and then show it to children who
were sitting further away from her in a group. She also thought that
taking a screenshot of the frozen AR experience and putting it in
a visual schedule would help prepare children for future activities
involving the application. Moreover, it could be helpful in scenarios
where children were unable to focus or stay still. P1 mentioned that
‘attention is sometimes very impacted’ for children on the spectrum,
so they may ‘wander the room’ or ‘walk away’ when she tries to
show them something on the iPad. The freeze feature would be
‘useful’ in these situations as she could trigger the AR experience,
leave the target image on the desk, and go where the child is. P3
echoed this sentiment and said that having the AR experience not
‘disappear once you move to a diferent spot’ would be helpful.
Therefore, it seems that the freeze feature may be especially
important to have when using AR in an autistic context. In contrast
to the freeze feature, most participants skipped over the settings
options in the AR-View or only tried a few options. When asked
why they did so, some participants said that they either did not feel
the need to edit the default settings, or they were not needed for
Tooba, et al.
the activity being considered. In the next subsection, we talk about
some of the practical challenges that users may face when using
picture-based AR in day-to-day settings (RQ 3).
6.5 Practical Challenges of Using AR in
Therapy / Learning Settings
When thinking of using AR in day-to-day settings, our participants
were concerned about being able to access a wide range of 3D
models, being able to create AR experiences on-the-fy or ‘just-in-
time’ and generalizing AR Experiences. We discuss these concerns
below.
6.5.1 Access to a Wider Range of 3D Models: As mentioned in
section 6.3.3, most of our participants thought that having a larger
library of 3D models at their disposal would help them create AR
experiences for a wider audience. P3 thought that being able to add
one’s own 3D models would be helpful. For example, participants
could search for and download 3D models from the internet. How-
ever, these models may not have sounds or animations built-in; we
had to build the animation and sound associated with each model
in the application’s library. Moreover, other participants, may feel
the same as P2 who stated that this study was her ‘frst time re-
ally exploring AR’, and she was unaware that one could search for
and download 3D models from the internet. Therefore, building a
library of models into the application, or hosting one online that
users can access from within the application, may be preferable. If
users are unable to fnd a relevant 3D model, they could potentially
make requests for 3D models and associated animations through
the application, which would then be fulflled by members of the
research team.
6.5.2 Creating AR Experience On-the-fly or Just-in-time:
Preparing resources or creating exercise material according to each
child’s interests can be a difcult and time-consuming process. Be-
ing able to ‘customize an AR experience on-the-fy’ during a therapy
session would make a ‘huge diference’ as it would reduce the ‘prep
time’ participants have to put in beforehand (P1). P3 echoed this sen-
timent, stating that one must ‘make things quickly’ while working
with some children, and that it currently requires a ‘lot of planning’
to create relevant AR experiences. Therefore, some degree of in-
telligence / automation may be helpful. For example, P2 thought
that it would be helpful if the application could detect the contents
of a target image and quickly provide options for appropriate 3D
models and audio prompts that label the objects. This would al-
low therapists and educational professionals to quickly create AR
experiences during a therapy session, based on what the child is
interested in at that moment. In section 9.1 we present a prototype
for ‘just-in-time’ content creation as a proof of concept.
6.5.3 Generalizing AR Experiences: P3 pointed out that ‘gen-
eralization is a big area of focus when working with people on the
spectrum’. It involves learning a skill or concept and using it in any
environment, not just the one in which it was taught. P3 thought
it would be great if an AR experience could be generalized. For
example, in the context of hand-washing, the application would
recognize that a sink was present and would pull up the AR content
(audio, hand-washing videos etc.) that she had previously associ-
ated with another sink. While she was focused more on recognizing
Designing a Customizable Picture-Based AR Application For Therapists Working in Autistic Contexts
3D objects and using those as triggers for AR experiences instead
of pictures, her idea could be extrapolated to work for pictures as
well. Moreover, she thought that if a clinician could create an AR
experience and share it so that ‘parents can use it at home’, it would
help generalize the skills taught in school to a home setting. We
discuss this further in section 7.
7 DISCUSSION
While users found the application’s features and customization
options to be useful, there is room for improvement. In this section,
we use insights from our study to propose design implications
for future applications (AR or otherwise) that target therapists and
autistic contexts. We also discuss ways of reducing the application’s
learning curve and position autistic children as content creators.
7.1 Sharing AR Experiences & Just-in-Time
Creation - Implications for Content
Creation Apps
Sharing AR experiences was initially brought up in the context
of generalization (6.5.3), but could also facilitate just-in-time con-
tent creation and collaboration/group-work. For example, thera-
pists/teachers could share their AR experiences with parents, or
vice versa, by uploading them to an online portal. This would allow
children to be exposed to the same prompts/AR experiences in difer-
ent environments, which could help with generalizing ideas/skills.
Additionally, therapists could download and edit pre-made AR ex-
periences or templates to speed up the creation process and reduce
their own time and efort. Moreover, in class/group activities, teach-
ers could upload a half-fnished AR experience for their students to
download and edit. Students could also share AR experiences with
each other to collaborate or show what they have created so far.
Therefore, applications (AR or otherwise) that involve content cre-
ation and target therapists/teachers and learning settings, should
provide support for content-sharing and collaboration.
7.2 How Much Customization is Enough -
Implications for Autism-Focused
Customizable Applications
Recall that in section 6.3.5, we discussed whether or not our cus-
tomization options were sufcient. Improvements to 3D models
notwithstanding, perhaps one reason our approach to customiza-
tion was successful was because we provided sufcient fexibility
without the application becoming too overwhelming. For example,
users only had the option of adding audio, video or 3D models but
could decide themselves which of the three they wanted to add to
an AR experience and what the content of the audio and/or videos
should be. This allowed them to create AR experiences for a vari-
ety of learning contexts. Moreover, for future AR application that
position autistic children as content creators, it might be helpful
to provide a limited set of options but provide fexibility within
those options, so that children can be creative without getting too
‘bogged down’ in the details. Of course some learning contexts may
require support for a lot more customization, and by no means
do we claim that our approach fts every context. However, as a
ASSETS ’22, October 23–26, 2022, Athens, Greece
starting point, future applications might consider providing fex-
ibility within a limited set of customization options, or gradually
introducing customization options to users. The latter is discussed
in section 7.3 and 7.4.
7.3 Reducing the Application’s Learning Curve
- Tutorials & Gradually Exposing
Functionality
Recall that some participants made more progress in using the
application’s features than others. Although we added in-app video
tutorials, interactive tutorials where the application helps the user
create and trigger their frst AR experience and points out useful
features, may further reduce the learning curve. P7 also advocated
for gradually exposing the application’s functionality to users. He
stated that a lite version of the application, where users can only add
audio to a target image, may be helpful for users who are not very
technologically savvy. He stated that ‘a broader range of people
are more comfortable’ with taking a photo or recording something,
so limiting the customization options can help ease them into the
creation process. Users can later upgrade to a version with videos
and 3D models.
7.4 Autistic Children as Content Creators -
Will Our Current UI Sufce?
We initially envisioned therapists and educational professionals as
the main creators of AR experiences, while the autistic children
they worked with would simply view the AR content. P5, however,
stated that her high-school students could easily navigate the fea-
tures of the application to create their own AR experiences. This
begs the question: is the application’s UI sufcient to allow autistic
children to create their own AR experiences, or are modifcations
necessary? For example, some children may wish to hide some
buttons on the screen to reduce visual input. Other children may
have difculty remembering the steps involved in creating an AR
experience and may beneft from an interface that sequentially
takes them through each step of the creation process. P7’s idea of
gradually exposing functionality to users, as mentioned in section
7.3, may be useful here as well; children can take their time and
become comfortable with using one customization option before
moving onto another. To make the application accessible to a wide
range of content creators, we need to conduct user studies with
autistic individuals, particularly those who can independently nav-
igate similar applications and are interested in AR. We leave this
for future work.
8 KEY TAKEAWAYS
The following bullet points summarize this work’s key takeaways:
• The customization options we provided (audio, video and 3D
models with animation) allowed therapists/educational pro-
fessionals to create AR experiences for a variety of learning
exercises. Participants appreciated our approach of provid-
ing fexibility within a limited set of customization options
as it prevented users from getting too bogged down. Future
researchers/designers creating applications in an AR and
ASSETS ’22, October 23–26, 2022, Athens, Greece Tooba, et al.

autistic domain may beneft from using this approach to

customization (as a starting point).
• The freeze feature can be helpful when working with autistic
students with limited attention, or when working in group
settings. Future researchers/designers creating AR applica-
tions should consider providing a way to freeze AR expe-
riences on the screen to make it easier to use with autistic
children.
• Therapists and educational professionals may need to create
content just-in-time during therapy sessions or may need to
generalize the same content to diferent situations. Future
researchers/designers who are developing content-creation
applications for autistic contexts (AR or otherwise) should
(a) Sequential Interface. A user has uploaded a target im-
provide avenues for creating content quickly, or sharing age of fruits. The application used the AWS Rekognition
content/AR experiences with others. service to provide the following labels for the contents of
the image: ‘Plant’, ‘Fruit’, ‘Food’ etc. The labels appear in
the window at the bottom-right and the user can select the
most appropriate label.
9 LIMITATIONS & FUTURE WORK
Our study had some limitations: Firstly, participants had a limited
selection of 3D models. Future prototypes should allow users to
upload their own 3D models, or provide a larger library to choose
from. Secondly, we cannot make any claims about whether the
application’s current UI will be sufcient for autistic children to
create their own AR experiences. We posit that a lite version, or a
version that either gradually exposes functionality or systematically
takes the user through each step of the creation process, might be
helpful. That said, we have yet to conduct user studies with autistic
children to confrm this hypothesis. The next subsection discusses
a proof-of-concept prototype that could be used in future studies
to answer some of these questions.
(b) 3D Model Recommendations. The application has rec-
ommended several 3D models to the user based on the label
9.1 Just-In-Time Creation and Sequential they chose in the previous step (‘Fruits’). The recommenda-
tions are shown in window on the left. The user can explore
Interface all the other categories of 3D models using the buttons at
the top-right of the screen.
Based on the discussion in 6.5.2 and 7.4 about just-in-time con-
tent creation and the benefts of a sequential interface, we made
Figure 8
some modifcations to the application’s UI. The third version of the
application takes users through each step of the creation process
sequentially. Users can move between steps using the buttons at
the bottom of the screen (fgure 8).
Other features, such as sharing AR experiences through an online
The application uses the AWS Rekognition [48] service to iden-
portal, are more complex and require further thought.
tify labels for the object, scene or actions present in a target image
(fgure 8a). Based on the label that a user selects, the application
provides recommendations for 3D models (fgure 8b). Users can 10 CONCLUSIONS
still explore all the categories manually but we posit that providing This paper describes the design and evaluation of CustomAR, a
them with some recommendations might speed up the creation mobile application that allows therapists to create picture-based
process. Moreover, we use the AWS Simple Storage Service (S3) [10] AR experiences from scratch for use in autism-based learning exer-
to host a larger library of 3D models. Users can download these cises. A diary and interview study revealed that therapists found
3D models through their application. This allows us to provide the application’s customization options to be sufcient for creat-
a larger library of 3D models without encroaching on the user’s ing AR experiences for a variety of learning exercises, ranging
device storage. from choice-making activities to reminders, visual schedules and
This third prototype is a small proof-of-concept. We hope to use groupwork. We posit that our approach was successful because we
it in future design meetings/studies with therapists, educational provided a fair amount of fexibility, without inundating users with
professionals, and autistic children to understand whether they customization options. Participants also thought that the freeze
prefer UIs where the customization options are exposed gradu- feature would be useful when working with children with limited
ally/sequentially or all at once, and whether the recommendations attention. Moreover, our study highlighted some challenges to us-
given by the application are helpful when creating AR content. ing picture-based AR in practical therapy settings, such as fnding
Designing a Customizable Picture-Based AR Application For Therapists Working in Autistic Contexts
3D models that capture the children’s interests, generalizing con-
cepts, and creating AR experiences quickly during therapy sessions.
Moving forward, application developers looking to create customiz-
able applications for autistic contexts, should allow users to share
their AR experiences or create experiences quickly – the latter may
require incorporating some degree of automation or intelligence
into the application.
ACKNOWLEDGMENTS
We thank Hedi Skali, a high-school intern who worked with us
during the summer of 2019. We also thank the reviewers for their
eforts in reviewing this work and providing constructive feedback.
REFERENCES
[1] 2017. Daily living skills: Strategies to help sequence & achieve personal hy-
giene tasks. https://www.toolstogrowot.com/blog/2017/08/09/daily-living-
skills-strategies-to-help-sequence-achieve-personal-hygiene-tasks
[2] 2021. AugmentedClass! augmented reality for education - apps on Google Play.
https://play.google.com/store/apps/details?id=com.AugmentedClass.AClass
[3] 2021. The benefts of visual supports for children with autism. https://www.
autismparentingmagazine.com/benefts-of-autism-visual-supports/
[4] 2021. Qualtrics XM - experience management software. https://www.qualtrics.
com/
[5] 2021. What is Visual Scheduling? https://www.appliedbehavioranalysisprograms.
com/faq/what-is-visual-scheduling/#:~:text=Visualschedulingisasystematic,
andachievesuccessinlife.
[6] 2022. About AR Foundation: AR FOUNDATION: 4.1.7. https://docs.unity3d.
com/Packages/com.unity.xr.arfoundation@4.1/manual/index.html
[7] 2022. Apple. https://testfight.apple.com/
[8] 2022. Build new augmented reality experiences that seamlessly blend the digital
and physical worlds. https://developers.google.com/ar
[9] 2022. Video conferencing, Cloud Phone, WEBINARS, Chat, Virtual EVENTS:
ZOOM. https://zoom.us/
[10] 2022. What is Amazon S3? - amazon simple storage service. https://docs.aws.
amazon.com/AmazonS3/latest/userguide/Welcome.html
[11] Mohammed Alzyoudi, AbedAlziz Sartawi, and Osha Almuhiri. 2015. The impact
of video modelling on improving social skills in children with autism. British
Journal of Special Education 42, 1, 53–68.
[12] Zhen Bai, Alan F Blackwell, and George Coulouris. 2013. Through the looking
glass: Pretend play for children with autism. In 2013 IEEE International Symposium
on Mixed and augmented reality (ISMAR). IEEE, 49–58.
[13] Zhen Bai, Alan F Blackwell, and George Coulouris. 2015. Exploring expressive
augmented reality: The FingAR puppet system for social pretend play. In Proceed-
ings of the 33rd Annual ACM Conference on Human Factors in Computing Systems.
1035–1044.
[14] Andy Bondy and Lori Frost. 2011. A picture’s worth: PECS and other visual
communication strategies in autism. Woodbine House.
[15] Monique Botha, Jacqueline Hanlon, and Gemma Louise Williams. 2021. Does lan-
guage matter? Identity-frst versus person-frst language use in autism research:
A response to Vivanti. Journal of Autism and Developmental Disorders, 1–9.
[16] Brian A Boyd, Maureen A Conroy, G Richmond Mancil, Taketo Nakao, and Peter J
Alter. 2007. Efects of circumscribed interests on the social behaviors of children
with autism spectrum disorders. Journal of autism and developmental disorders
37, 8, 1550–1561.
[17] Jorge Brandão, Pedro Cunha, José Vasconcelos, Vítor Carvalho, and Filomena
Soares. 2015. An augmented reality gamebook for children with autism spectrum
disorders. In The International Conference on E-Learning in the Workplace. 1–6.
[18] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative research in psychology 3, 2, 77–101.
[19] Chien-Hsu Chen, I-Jui Lee, and Ling-Yi Lin. 2016. Augmented reality-based video-
modeling storybook of nonverbal facial cues for children with autism spectrum
disorder to improve their perceptions and judgments of facial expressions and
emotions. Computers in Human Behavior 55, 477–485.
[20] Chi-Hsuan Chung and Chien-Hsu Chen. 2017. Augmented reality based social
stories training system for promoting the social skills of children with autism.
In Advances in ergonomics modeling, usability & special populations. Springer,
495–505.
[21] Seungwon Chung and Jung-Woo Son. 2020. Visual perception in autism spectrum
disorder: A review of neuroimaging studies. Journal of the Korean Academy of
Child and Adolescent Psychiatry 31, 3, 105.
[22] David F Cihak, Eric J Moore, Rachel E Wright, Don D McMahon, Melinda M
Gibbons, and Cate Smith. 2016. Evaluating augmented reality to complete a
ASSETS ’22, October 23–26, 2022, Athens, Greece
chain task for elementary students with autism. Journal of Special Education
Technology 31, 2, 99–108.
[23] Blythe A Corbett and Maryam Abdullah. 2005. Video modeling: Why does it work
for children with autism? Journal of Early and Intensive Behavior Intervention 2,
1, 2.
[24] Camilla Almeida da Silva, António Ramires Fernandes, and Ana Paula Grohmann.
2014. STAR: speech therapy with augmented reality for children with autism
spectrum disorders. In International Conference on Enterprise Information Systems.
Springer, 379–396.
[25] Sarah Dettmer, Richard L Simpson, Brenda Smith Myles, and Jennifer B Ganz.
2000. The use of visual supports to facilitate transitions of students with autism.
Focus on autism and other developmental disabilities 15, 3, 163–169.
[26] Mihaela Dragomir, Andrew Manches, Sue Fletcher-Watson, and Helen Pain.
2018. Facilitating pretend play in autistic children: results from an augmented
reality app evaluation. In Proceedings of the 20th International ACM SIGACCESS
Conference on Computers and Accessibility. 407–409.
[27] Yao Du, LouAnne Boyd, and Seray Ibrahim. 2018. From Behavioral and Commu-
nication Intervention to Interaction Design: User Perspectives from Clinicians. In
Proceedings of the 20th International ACM SIGACCESS Conference on Computers
and Accessibility. 198–202.
[28] Carl J Dunst, Carol M Trivette, and Tracy Masiello. 2011. Exploratory inves-
tigation of the efects of interest-based learning on the development of young
children with autism. Autism 15, 3, 295–305.
[29] Lizbeth Escobedo, David H Nguyen, LouAnne Boyd, Sen Hirano, Alejandro
Rangel, Daniel Garcia-Rosas, Monica Tentori, and Gillian Hayes. 2012. MOSOCO:
a mobile assistive tool to support children with autism practicing social skills in
real-life situations. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems. 2589–2598.
[30] Lizbeth Escobedo, Monica Tentori, Eduardo Quintana, Jesus Favela, and Daniel
Garcia-Rosas. 2014. Using augmented reality to help children with autism stay
focused. IEEE Pervasive Computing 13, 1, 38–46.
[31] Lori Frost and Andy Bondy. 2002. The picture exchange communication system
training manual. Pyramid Educational Products.
[32] Jennifer B Ganz, Theresa L Earles-Vollrath, Amy K Heath, Richard I Parker,
Mandy J Rispoli, and Jaime B Duran. 2012. A meta-analysis of single case research
studies on aided augmentative and alternative communication systems with
individuals with autism spectrum disorders. Journal of autism and developmental
disorders 42, 1, 60–74.
[33] Barbara C Gartin and Nikki L Murdick. 2005. Idea 2004: The IEP. Remedial and
Special Education 26, 6, 327–331.
[34] Martin Guha. 2014. Diagnostic and statistical manual of mental disorders: DSM-5.
Reference Reviews.
[35] Individual Education Plan IEP. 1978. Special educational needs.
[36] Apple Inc. 2022. ARKit - augmented reality. https://developer.apple.com/
augmented-reality/arkit
[37] Chloe Jennifer Jordan and Catherine L Caldwell-Harris. 2012. Understanding
diferences in neurotypical and autism spectrum special interests through internet
forums. Intellectual and developmental disabilities 50, 5, 391–402.
[38] Kamran Khowaja, Dena Al-Thani, Bilikis Banire, Siti Salwah Salim, and Asadullah
Shah. 2019. Use of augmented reality for social communication skills in children
and adolescents with autism spectrum disorder (ASD): A systematic review. In
2019 IEEE 6th International Conference on Engineering Technologies and Applied
Sciences (ICETAS). IEEE, 1–7.
[39] Kamran Khowaja, Bilikis Banire, Dena Al-Thani, Mohammed Tahri Sqalli,
Aboubakr Aqle, Asadullah Shah, and Siti Salwah Salim. 2020. Augmented reality
for learning of children and adolescents with autism spectrum disorder (ASD): A
systematic review. IEEE Access 8, 78779–78807.
[40] I Kurniawan et al. 2018. The improvement of autism spectrum disorders on chil-
dren communication ability with PECS method Multimedia Augmented Reality-
Based. In Journal of Physics: Conference Series, Vol. 947. IOP Publishing, 012009.
[41] Aaron Lanou, Lauren Hough, and Elizabeth Powell. 2012. Case studies on using
strengths and interests to address the needs of students with autism spectrum
disorders. Intervention in School and Clinic 47, 3, 175–182.
[42] Gun A Lee, Ungyeon Yang, Yongwan Kim, Dongsik Jo, Ki-Hong Kim, Jae Ha Kim,
and Jin Sung Choi. 2009. Freeze-Set-Go interaction method for handheld mobile
augmented reality environments. In Proceedings of the 16th ACM Symposium on
Virtual Reality Software and Technology. 143–146.
[43] I-Jui Lee, Chien-Hsu Chen, Chuan-Po Wang, and Chi-Hsuan Chung. 2018. Aug-
mented reality plus concept map technique to teach children with ASD to use
social cues when meeting and greeting. The Asia-Pacifc Education Researcher 27,
3, 227–243.
[44] Roberto E Lopez-Herrejon, Oishi Poddar, Gerardo Herrera, and Javier Sevilla.
2020. Customization support in computer-based technologies for autism: A
systematic mapping study. International Journal of Human–Computer Interaction
36, 13, 1273–1290.
[45] Anabela Marto, Henrique A Almeida, and Alexandrino Gonçalves. 2019. Using
augmented reality in patients with autism: A systematic review. In ECCOMAS The-
matic Conference on Computational Vision and Medical Image Processing. Springer,
ASSETS ’22, October 23–26, 2022, Athens, Greece
454–463.
[46] MegEvans. 2022. Identity-frst language. https://autisticadvocacy.org/about-
asan/identity-frst-language/
[47] Esteban Menéndez and MD Lopez De Luise. 2018. Augmented reality as visual
communication for people with ASD. In Proc. 9th Int. Conf. Soc. Inf. Technol.(ICSIT).
28–32.
[48] Abhishek Mishra. 2019. Machine learning in the AWS cloud: Add intelligence
to applications with Amazon Sagemaker and Amazon Rekognition. https:
//aws.amazon.com/rekognition/
[49] Debora Nunes and Mary Frances Hanline. 2007. Enhancing the alternative
and augmentative communication use of a child with autism through a parent-
implemented naturalistic intervention. International Journal of Disability, Devel-
opment and Education 54, 2, 177–197.
[50] Shaila M Rao and Brenda Gagie. 2006. Learning through seeing and doing: Visual
supports for children with autism. Teaching exceptional children 38, 6, 26–33.
[51] Christopher Stephen Rayner. 2010. Video-modelling to improve task completion
in a child with autism. Developmental Neurorehabilitation 13, 3, 225–230.
[52] Mikle South, Sally Ozonof, and William M McMahon. 2005. Repetitive behavior
profles in Asperger syndrome and high-functioning autism. Journal of autism
and developmental disorders 35, 2, 145–158.
Tooba, et al.
[53] Elizabeth A Steed and Nancy Leech. 2021. Shifting to Remote Learning Dur-
ing COVID-19: Diferences for Early Childhood and Early Childhood Special
Education Teachers. Early childhood education journal, 1–10.
[54] Issey Takahashi, Mika Oki, Baptiste Bourreau, Itaru Kitahara, and Kenji Suzuki.
2018. FUTUREGYM: A gymnasium with interactive foor projection for children
with special needs. International Journal of Child-Computer Interaction 15, 37–47.
[55] Unity Technologies. 2022. https://unity.com/
[56] Catherine Tissot and Roy Evans. 2003. Visual teaching strategies for children
with autism. Early Child Development and Care 173, 4, 425–433.
[57] Laurie A Vismara and Gregory L Lyons. 2007. Using perseverative interests to
elicit joint attention behaviors in young children with autism: Theoretical and
clinical implications for understanding motivation. Journal of Positive Behavior
Interventions 9, 4, 214–228.
[58] Cheng Zheng, Caowei Zhang, Xuan Li, Xin Liu, Chuqi Tang, Guanyun Wang,
Cheng Yao, Fan Zhang, Wenjie Xu, and Fangtian Ying. 2017. Toon-chat: a cartoon-
masked chat system for children with autism. In ACM SIGGRAPH 2017 Posters.
1–2.
[59] Cheng Zheng, Caowei Zhang, Xuan Li, Fan Zhang, Bing Li, Chuqi Tang, Cheng
Yao, Ting Zhang, and Fangtian Ying. 2017. KinToon: a kinect facial projector for
communication enhancement for ASD children. In Adjunct Publication of the 30th
Annual ACM Symposium on User Interface Software and Technology. 201–203.
 State of the Art in AAC: A Systematic Review and Taxonomy
Humphrey Curtis Timothy Neate
Department of Informatics Department of Informatics
King’s College London King’s College London
London, UK London, UK
humphrey.curtis@kcl.ac.uk timothy.neate@kcl.ac.uk
ABSTRACT
People with complex communication needs (CCNs) can use high-
tech augmentative and alternative communication (AAC) devices
and systems to compensate for communication difculties. While
many use AAC efectively, much research has highlighted chal-
lenges – for instance, high rates of abandonment and solutions
which are not appropriate for their end-users. Presently, we lack
a detailed survey of this feld to comprehend these shortcomings
and understand how the accessibility community might direct its
eforts to design more efective AAC. In response to this, we con-
duct a systematic review and taxonomy of high-tech AAC devices
and interventions, reporting results from 562 articles identifed in
the ACM DL and SCOPUS databases. We provide a taxonomical
overview of the current state of AAC devices – e.g. their interac-
tion modalities and characteristics. We describe the communities
of focus explored, and the methodological approaches used. We
contrast fndings in the broader accessibility and HCI literature to
delineate future avenues for exploration in light of the current tax-
onomy, ofer a reassessment of the norms and incumbent research
methodologies and present a discourse on the communities of focus
for AAC and interventions.
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility; Accessibility technologies.
KEYWORDS
AAC, Alternative and Augmentative Communication, Systematic
Review, Taxonomy, Accessibility.
ACM Reference Format:
Humphrey Curtis, Timothy Neate, and Carlota Vazquez Gonzalez. 2022.
State of the Art in AAC: A Systematic Review and Taxonomy. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
22 pages. https://doi.org/10.1145/3517428.3544810
1 INTRODUCTION
Approximately 2.2 million people in the UK and 1.3% of the US
population experience a form of communication impairment [18].
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544810
Carlota Vazquez Gonzalez
Department of Informatics
King’s College London
London, UK
carlota.vazquez_gonzalez@kcl.ac.uk
Critically, challenges in human social and communication skills
subject individuals to many risks such as: negative social interac-
tions [122], employment challenges [156], educational access [48],
mental health disorders [76] and a myriad other challenges [131].
Equally, communication and freedom of speech is protected un-
der UN legislature [120] and revered as the very “essence of hu-
man life” [105]. Aided and unaided AAC strategies and systems
serve to remediate communication difculties experienced by in-
dividuals and communities with complex communication needs
(CCNs) [126]. In particular, high-tech aided AAC devices encap-
sulate the most advanced electronic AAC technology for example
speech generating devices (SGDs) or voice output communication
aids (VOCAs) [42]. Data on the prevalence of specifcally high-tech
AAC is limited, however there will be an increase in the number of
individuals requiring AAC interventions [9, 107, 135]. Also, there
has been sustained academic research into high-tech AAC devices
and interventions since the formation of the International Society
for AAC (ISAAC) in 1983 and the AAC journal in 1985 [155, 183].
Since then, high-tech AAC has been developed to support wide age
ranges [106] and serve many communities [135]. In many cases,
high-tech AAC devices and interventions have contributed to pos-
itive and successful outcomes for individuals with CCNs whilst
advances in computer technology, machine learning (ML) and ar-
tifcial intelligence (AI) hold much promise for future high-tech
AAC [140, 185].
1.1 High-tech AAC Device Abandonment and
Fellow Systematic Reviews
Numerous HCI researchers, including Bircanin et al. [20], Ibrahim
et al. [81], Norrie et al. [140] have noted that high-tech AAC de-
vices too often experience a high rate of abandonment amongst
their target community. The reasons for abandonment of high-tech
AAC devices are far ranging and vary dependent on the community
and intervention circumstances [140]. However, research has found
that high-tech AAC devices can be frustrating to use, unreliable,
slow [92, 140] and inefective for certain common communication
interactions leading to breakdowns and misalignments [81, 92].
Other problems noted by research include that high-tech AAC
devices carry a stigma [128], are too expensive [65], hard to pro-
gram [33] and inconsiderate of cultural factors [100] – making
high-tech AAC devices simultaneously challenging for their users,
close caregivers, specialists and wider communities to adopt. At
the same time high-tech AAC has often been found to inadequately
adapt to their users communication strengths and weakness or
ofer pathways for multimodal or embodied forms of communica-
tion [81]. In light of these criticisms, we believe it is important to
refect on the full body of high-tech AAC research with a systematic
review (SR) of the literature and taxonomic overview of devices
ASSETS ’22, October 23–26, 2022, Athens, Greece
fostering novel pathways for improving high-tech AAC devices,
interventions and research.
There are several pre-existing SRs into AAC, which typically fo-
cus on research interventions for specifc communities and groups.
Beukelman et al., studied AAC interventions for adults with neu-
rological conditions [17], Biggs et al., reviewed interventions for
children with CCNs [19], both Holyfeld et al., and Logan et al., ex-
amined AAC interventions amongst people with autism (ASD) [77,
111], van der Meer et al., analysed interventions for individuals
with developmental disabilities (IDDs) [182] and Simacek et al.,
focused on AAC interventions amongst individuals with multiple
disabilities [165]. Lastly, Moorcroft et al., assessed research on bar-
riers in the provision of low-tech and unaided AAC [130]. Some
SRs have even focused specifcally on high-tech AAC devices and
interventions. Baxter et al., conducted two reviews of high-tech
AAC interventions, barriers and facilitators [11, 12]. Whilst, the re-
maining high-tech AAC reviews focus on interventions in specifc
communities and groups. Morin et al., provided a meta-analysis of
AAC interventions for people with IDDs [131], Still et al., exam-
ined interventions for people with ASD [170] and lastly Ju et al.,
considered high-tech AAC interventions amongst ICU patients [89].
Despite this previous research, at present there is not a broad
survey of the literature including analysis of the earliest and latest
high-tech AAC research. Consequently, this review and taxonomy
will look to supplement the following three research gaps vacated
by the previous SRs. Firstly, previous SRs were focused on clinical
practice over HCI and device-orientated development – failing
to establish a taxonomy of high-tech AAC, which supports the
development of future generations of devices. Secondly, previous
SRs did not track the prevailing research methods and capture data
on the studies undertaken. Thirdly, previous SRs have not captured
broad data on which communities have comparatively currently
received the most high-tech AAC research. Additionally, the SRs
predate the recent criticisms of high-tech AAC (c.f. [20, 81, 140]) and
ofer no comparison with the Mack et al. fndings from reviewing
accessibility research [112].
1.2 Research Questions and Contributions
To focus our contribution we initially developed three research
questions:
RQ1: What is the current taxonomy and dominant characteristics
of high-tech AAC?
RQ2: What research methods are used to contribute towards the
design and study of high-tech AAC devices and interven-
tions?
RQ3: Who does high-tech AAC devices and interventions focus
on?
The three research questions support the review and analysis
of previous high-tech AAC research – quantitatively evaluating
the design of devices, methodologies and communities of focus.
The research questions capture data to enable the identifcation
of notable research gaps, development of strategic directions for
future research and support cross-comparison with the Mack et
al., research [112]. Thereby supporting more novel, successful and
long-term high-tech AAC interventions. To answer these three
Curtis et al.
research questions, mapped in Figure 1, we have four contributions
in this paper:
(1) We have built an open-source data-set of 562 coded papers
from 1978-2021 focused on research into high-tech AAC de-
vices and interventions encompassing peer-reviewed articles
using 2021 PRISMA guidelines [153].
(2) We present the frst SR and taxonomy of high-tech aided
AAC devices within the ACM literature. The taxonomy of
high-tech AAC was developed in three ways: (a) collecting
an inventory of the interaction experience through coding
high-tech AAC devices’ input and outputs (b) understanding
the features of high-tech AAC devices through coding the
interface layout and scalar attributes (c) understanding the
communication facilitated via coding the communication
model/type facilitated by the device.
(3) We provide an analysis of the methodologies, roles, and
communities of focus within high-tech AAC interventions
and research. This data is then cross-analysed with the Mack
et al. SR of accessibility research [112], which we consider to
be normative standards for accessible computing research –
to gain an understanding of similarities and diferences with
wider accessibility research.
(4) We provide implications for future research on high-tech
AAC devices and interventions. In the discussion, we identify
directions for future scientifc investigations and high-tech
AAC development. With almost 40 years of preceding re-
search, we consider this contribution signifcant for new
researchers to improve the development of future high-tech
AAC devices and decrease the current high rates of aban-
donment [81].
2 BACKGROUND AND RELATED WORK
We outline the key related work on SRs of high-tech AAC and frame
our work in communication research; discussing the models and
types of communication for developing the codebook.
2.1 Related Work in High-tech AAC
Although no SR and taxonomy of high-tech AAC has been pub-
lished within the ACM literature, previous research has explored the
design and barriers of high-tech AAC [11, 12, 166]. Plus, previous
research has considered specifc sub-areas including high-tech AAC
interventions for aphasia [181], autism [170] and ICU patients1 [89].
2.1.1 Related Taxonomies and Investigations of AAC Design and
Barriers. The closest contribution to a taxonomy for high-tech AAC
is a paper by Belani presenting a usability requirements taxonomy
for mobile AAC services [14]. Belani’s taxonomy inherits from sys-
tems engineering and focuses on accessible software principles: the
context of AAC usage, user relations and the principles of simplicity,
supplementing and trustworthiness [14]. Furthermore, this taxon-
omy tries to develop a set of augmentative requirements difering
from our systematic investigation of the pre-existing literature and
consequent device taxonomy2 [14, 30]. Barriers and facilitators of
1 Critical
research has considered the efcacy of high-tech AAC through consumer
perspectives [24] and as an evidence based practice [131].
2 See Brudy et al.’s, taxonomy of the cross-device computing domain [30].
State of the Art in AAC
Figure 1: Flow diagram presenting an overview of the fg-
ures and tabular contribution towards each research ques-
tion. 1.2: Figure 4 and Tables 2, 3, 4. 1.2: Figures 5, 6 and Ta-
bles 6, 7, 8. 1.2: Tables 9, 10, 11.
high-tech AAC are explored in 2 SRs by Baxter et al. [11, 12]. In
their frst SR of 27 papers, they identifed several key themes – 11
papers described the limited reliability of high-tech AAC devices, 8
papers highlighted the difculties of learning how to operate de-
vices and 6 papers discussed inappropriate voices/words generated
by devices within specifc cultural contexts3 [11]. Turning to their
second and more extensive SR of 65 papers and interventions from
2000-10 [12]. In order of frequency, Baxter et al. report that the
most common community of focus for high-tech AAC interventions
were 14 studies focused on aphasia from non-progressive causes, 13
focused on interventions with people with autism disorder (ASD),
12 considered interventions for adults with cerebral palsy whilst
other communities faced a smaller subset4 [12].
2.1.2 Sub-areas of High-tech AAC. SRs considering sub-areas of
high-tech AAC interventions to a greater extent report optimistic
results. For people with aphasia (PWA), Sandt-Koenderman’s SR
emphasised the need for AAC devices to be “tailor made”, taking
advantage of PWAs residual language skills and communicative
strengths [181]. In contrast, Still et al. perform an SR focused on
3 Secondary themes from the research include needs for staf training, limitations of
technical support, decision difculties faced by families whilst selecting an AAC device
and negative communication rates with the AAC device [11].
4 Other fndings of the research include that devices were found to be benefcial in
enhancing communication across a broad range of diagnoses and age ranges [12].
ASSETS ’22, October 23–26, 2022, Athens, Greece
high-tech AAC interventions for individuals under the age of six-
teen with ASD [170]. They found portable SGDs have been fre-
quently favoured for interventions – in particular, iPod and iPad-
based applications and intervention results were positive for teach-
ing requesting skills [170]. Within a hospital setting for voiceless
ICU patients, Ju et al. performed an SR considering adoption and
efects of a high-tech AAC intervention [89]. From the 18 stud-
ies qualitatively synthesised with the TAM model, they found that
high-tech AAC was easy to learn and use in most studies – with
customisation and portability of devices most important for pa-
tients [89]. For future positive development of high-tech AAC tech-
nologies, they encouraged further collaboration directly with ICU
staf and patients [89]. Although not specifcally high-tech AAC
research, broader accessibility scholarship has ofered meaningful
refection that should shape future high-tech AAC interventions.
Scholars have accepted the importance of not “medicalizing” dis-
ability5 [45, 112]. In this vein, AAC devices should not serve as
assistive technologies to fx people living with communication dis-
orders. Bennett et al. [15] have efectively conceptualised this form
of technology through a notion of independence – in which the
high-tech AAC device itself incorrectly serves as a dependent in-
terface to communicating with the environment and others6 [15].
Instead, we believe high-tech AAC should be guided by Bennett
et al.’s interdependence framing of assistive technology (AT) – re-
framing high-tech AAC as an assistive technology (AT) that all can
freely engage and leverage to interface and communicate with the
environment [15].
2.2 Models and Types of Communication
High-tech AAC is developed based upon our understanding of com-
munication and models of successful communication exchanges
between human parties. Here, we discuss three recognised models
of communication and two types of communication – this scholar-
ship directly infuenced the categories and sub-codes used in our
taxonomy of devices and high-tech AAC codebook.
2.2.1 Models of Communication. The most basic model of com-
munication that high-tech AAC can support is the original model
of communication i.e. Shannon-Weaver sender-receiver or Linear
model [161]. The model consists of just four parts7 and mirrors the
functioning of radio and telephone technology – for instance the
sender delivers a message using a high-tech AAC device, the chan-
nel is the AAC device speaker, and the receiver hears the sender’s
message [161]. Within this model, information is solely transmitted
between the two parties. Yet, Ibrahim et al., have criticised high-
tech AAC devices that exclusively support sender-receiver and
linear communication [81]. Emphasising Kraat et al.’s research [98]
, Ibrahim et al. argue that high-tech AAC communication should
be much more dynamic made up of people, the setting, rules of
language use and situated within the context [81]. In contrast, the
Interactive model ofers more dynamic communication. Here, high-
tech AAC supports the feedback fow between sender and receiver
5 Mankof et al., has called for greater representation of disabled people in accessibility
research – in the same vein as Mack et al., we provide some empirical metrics of
representation within our dataset [112, 114].
6 Limiting the AAC users’ autonomy, perpetuating social stigma and increasing
marginalization by exacerbating social diferences.
7 These parts are: (1) sender, (2) message, (3) channel, and (4) receiver.
ASSETS ’22, October 23–26, 2022, Athens, Greece
– feedback can be verbal (i.e., “yes” or "no" or nonverbal i.e., a nod
or smile) [190]. However, the feedback provided by the high-tech
AAC is not simultaneous and can be potentially slow or indirect.
Lastly, the Transactional model is the most dynamic – senders and
receivers are now considered communicators [10]. Now, high-tech
AAC communication is a co-created process with instantaneous
feedback between parties [10]. Although challenging, high-tech
AAC must look to support interactive and even transactional com-
munication where communication is an embodied, meaningful and
multimodal experience for its users [81].
2.2.2 Types of Communication. Broadly there are two exclusive
types of communication, which high-tech AAC can enrich: non-
verbal and verbal8 . Verbal communication is the use of words to
convey a message whether that be written or dialogue [84]. Ex-
amples of verbal communication include oral forms i.e., speech
and written forms e.g., letters and text messages [84]. In contrast,
non-verbal communication explains the processes that convey a
message in a form of non-linguistic representations [75]. Examples
of non-verbal include gestures, sign language, facial expression,
and eye contact [75]. Most high-tech AAC devices do not equally
support both types of communication. High-tech AAC to augment
and enhance users’ verbal communication has taken precedence
over supporting non-verbal forms [180]. Instead, high-tech AAC
should consider and enhance both types of communication for
successful outcomes [80]. For instance, Valencia et al.’s physical
expressive objects successfully increase augmentative communica-
tors (ACs) agency in conversations with unfamiliar partners using
solely non-verbal messages and signals [52, 180].
3 METHODS
In this paper we followed PRISMA 20219 procedures for SRs [153]
with secondary support from Siddaway et al. [162] and Silva et al.’s
SR guidelines [164]. Firstly, we consider the scope of our investiga-
tion via defning high-tech AAC. Secondly, we discuss methods for
creating a dataset using PRISMA guidelines. Thirdly, we describe
the qualitative, quantitative and programmatic analysis methods of
this study.
3.1 Scope
We start by presenting our defnition of high-tech AAC devices and
interventions. AAC comes in many forms to support a variety of
communities and needs - therefore a clear defnition for the scope
is signifcant [18]. Unaided AAC interventions use no equipment
i.e., signing or body language [130]. For unaided AAC research
there are several pre-existing SRs, such as: [130]. By contrast, aided
AAC interventions encompasses any enabling aids or technologies
used to support or replace communication for those with CCNs
thereby enriching the production or comprehension of communi-
cation [126]. However, this equipment comes in a wide variety of
form factors [18]. Aided AAC devices are categorised as no, low,
medium and high-tech [18]. Low-tech AAC devices do not require
electricity or battery power [130]. Typically, they are simple props
8 Sometimes re-conceptualised with more types of communication considered such as
written or visual.
9 The PRISMA acronym stands for Preferred Reporting Items for Systematic reviews
and Meta-Analyses.
Curtis et al.
to foster communication such head/mouth sticks to enable point-
ing, pen, paper, and eraser boards for drawing or writing to even
customised analogue systems such as communication boards and
books [130]. Medium-tech AAC devices encapsulate very simple
electronic devices with basic technology [142]. Examples include
battery powered switches or buttons that communicate one or two
messages, an LED light, or appropriated technology devices such
as motion toys, radios, and fans [142]. Therefore, high-tech aided
AAC devices encapsulate the most advanced electronic AAC tech-
nology with multi-message vocabularies – for example, SGDs or
VOCAs [17]. High-tech AAC devices vary in shape and size, poten-
tially with dynamic displays for depicting letters, words, phrases,
pictures and/or symbols that the communicator navigates to ex-
press messages [4]. They difer in size, weight, and portability as
well as access methods – either direct selection of a screen/keyboard
with a body part, pointer, or eye gaze adapted mouse/joystick or
indirect selection forms such as switches and scanning [18, 46, 142].
Our operational defnition of high-tech AAC is established by the
following three requirements:
Req1: Our scope addresses AAC research. AAC encompasses tools
and strategies that individuals use to supplement communi-
cation [42]. Communication is multimodal and takes many
forms i.e., speech, glance, text, gestures, facial expressions,
sign language, symbols, pictures, SGDs etc. contingent upon
the context and communication partner [98].
Req2: The paper must research aided AAC interventions and de-
vices. Aided AAC serves as hardware or software that is
used to supplement, enrich, or replace communication. Aided
AAC can take many forms and is categorised as low, medium
or high-tech.
Req3: The paper must address high-tech aided AAC interventions.
High-tech AAC refers to any AAC hardware, device, tools,
software or technologies powered by electricity that permits
the storage and retrieval of multiple electronic messages to
support or enrich the users communication.
Requirements 3.1 to 3.1 specify the scope of our investigation. We
wanted to focus on AAC research for individuals with CCNs. How-
ever, within this broad domain, we narrowed towards aided AAC
tools and devices deliberately designed to enrich verbal and nonver-
bal communication rather than unaided forms or systems. Equally,
we wanted to focus specifcally on high-tech aided interventions,
meaning our scope avoids appropriated devices or low-tech forms
of AAC.
3.2 Dataset Establishment
There are many ways to conduct SRs, but to derive solutions for our
three research questions (i.e., 1.2– 1.2) we harnessed methods de-
tailed in PRISMA 2021 guidelines for reporting meta-analyses [153].
We describe in detail the following: identifcation, screening, eligi-
bility and snowballing depicted in Figure 3.
3.2.1 Identification. The role of the identifcation stage is to cap-
ture work that addresses the three research questions by performing
queries of the relevant scientifc databases. We chose the ACM DL10
10 https://dl.acm.org
State of the Art in AAC ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 2: Flow diagram providing the scope of our investigation on high-tech AAC devices and interventions. For greater detail
please refer to 3.1, 3.1 and 3.1 detailed within the text.

Figure 3: The PRISMA diagram [153] illustrating the reference frequencies of identifcation, screening, eligibility, snowballing
and inclusion stages of our SR and taxonomy of high-tech AAC devices and interventions.

and SCOPUS11 as two major electronic databases for Computer Sci-
ence and AAC research. Initially, we performed some preparatory
searches to broadly investigate the search space – before commit-
ting to a defnitive search strategy. From this initial investigation,
we identifed that the ACM database provided essential HCI papers
yet failed to include key research papers from the academic liter-
ature – such as the AAC journal. In contrast, SCOPUS ofered a
broader search space containing literature from the AAC journal
and several other prominent venues for high-tech AAC literature.
For deciding keywords, we performed an analysis of the research
11 https://www.scopus.com/
questions and extracted initial keywords – utilising these, we then
collectively iterated several times to produce synonyms for more
keywords including acronyms and common named high-tech AAC
devices e.g., voice output communication aids and speech generating
devices. We then developed methods to avoid returning high quan-
tities of false-positive papers – for instance, we decided to search
just title, abstract and keywords. Additionally, within the search
string we concatenated the keywords with OR plus AND connec-
tors to ensure the ordering of the terminology was coherent and
asterisks were liberally used to enable multiple and plural forms of
the keywords. For the ACM DL, the fnal query was:
ASSETS ’22, October 23–26, 2022, Athens, Greece Curtis et al.

"query": were then only removed from the dataset once the two authors had
{
Title: ((aac) OR (augment* AND alternat* AND communicat*) OR (comput* AND
come to agreement. This screening process resulted in 1,288 papers
assist* AND communicat*) OR (voice* AND output* AND communicat* AND aid*) remaining in the dataset.
OR (alternat* AND augment* AND communicat*) OR (speech* AND generat* AND device*)) OR
Abstract: ((aac) OR (augment* AND alternat* AND communicat*) OR (voice* AND output*
AND communicat* AND aid*) OR (alternat* AND augment* AND communicat*) OR (speech*
3.2.3 Eligibility. Following PRISMA guidelines – we formulated a
AND generat* AND device*)) OR Keyword ((aac) OR (augment* AND alternat* AND communicat*) set of four eligibility criteria (3.2.3 through to 3.2.3) to further flter
OR (comput* AND assist* AND communicat*) OR (alternat* AND augment* AND communicat*) OR
(voice* AND output* AND communicat* AND aid*) OR (speech* AND generat* AND device*)) out work irrelevant to the scope of our systematic literature review
} and domain:
The ACM query returned 462 results. For the SCOPUS database EC1: Availability of full-text. The full-text research paper is avail-
we queried solely the title feld because initial testing with the able and written in English to be accessible for the research
abstract and keyword felds resulted in too many false positives – team.
meaning featureless, large search spaces exceeding 20,000 papers. EC2: Peer-reviewed academic research. The research must be peer
Additionally, we performed an abstraction based on venue for the reviewed literature and academic. E.g., journal articles, con-
SCOPUS query to limit the state space and number of papers. The ference papers, and PhD dissertations. Other media forms
fnal relevant selected venues were titled with: AAC, technology, would be excluded.
aphasia, language and communication disorder. Meaning the fnal EC3: Aided AAC research and interventions. The paper had to
SCOPUS query was: clearly focus on aided AAC interventions – an AAC aid
( is any device that is electronic or not, which is harnessed
TITLE-ABS-KEY(aac) OR TITLE-ABS-KEY(voice* AND output* AND communicat* AND aid*)
OR TITLE-ABS-KEY(speech* AND generat* AND device*) OR
to transmit and receive messages. In contrast, unaided AAC
TITLE-ABS-KEY(alternat* AND augment* AND communicat*) OR TITLE-ABS-KEY(augment* interventions are those that do not require external tools
AND alternat* AND communicat*)) AND (
LIMIT-TO ( EXACTSRCTITLE,"AAC Augmentative And Alternative Communication" ) e.g., sign languages.
OR LIMIT-TO ( EXACTSRCTITLE,"Lecture Notes In Computer Science Including Subseries EC4: High-tech AAC interventions. The paper must include re-
Lecture Notes In Artificial Intelligence And Lecture Notes In
Bioinformatics" ) search on high-tech aided AAC interventions. As defned
OR LIMIT-TO ( EXACTSRCTITLE,"Augmentative And Alternative Communication" )
OR LIMIT-TO ( EXACTSRCTITLE,"Disability And Rehabilitation Assistive Technology" )
earlier, high-tech refers to the most advanced forms of aided
OR LIMIT-TO ( EXACTSRCTITLE,"Communication Disorders Quarterly" ) AAC – that being electronic devices that enable the retrieval
OR LIMIT-TO ( EXACTSRCTITLE,"Assistive Technology" )
OR LIMIT-TO ( EXACTSRCTITLE,"International Journal Of Language And Communication Disorders" )
and storage of electronic messages to support and enrich
OR LIMIT-TO ( EXACTSRCTITLE,"Advances In Intelligent Systems And Computing" ) communication. There are a variety of high-tech AAC de-
OR LIMIT-TO ( EXACTSRCTITLE,"Conference On Human Factors In Computing Systems Proceedings" )
OR LIMIT-TO ( EXACTSRCTITLE,"Topics In Language Disorders" ) vices in a multitude of diferent forms: such as a dedicated
OR LIMIT-TO ( EXACTSRCTITLE,"Technology And Disability" ) device like a DynaVox e.g., [4], tablet and smartphone apps
OR LIMIT-TO ( EXACTSRCTITLE,"Aphasiology" )
OR LIMIT-TO ( EXACTSRCTITLE,"Assistive Technology Research Series" ) e.g., [124], brain computer interfaces (BCI) e.g., [59] and even
OR LIMIT-TO ( EXACTSRCTITLE,"Communications In Computer And Information Science" )
OR LIMIT-TO ( EXACTSRCTITLE,"Journal Of Special Education Technology" )
wearables e.g., [54]. We wanted to incorporate as much of
OR LIMIT-TO ( EXACTSRCTITLE,"Studies In Health Technology And Informatics" ) this emergent literature as possible to provide an encompass-
OR LIMIT-TO ( EXACTSRCTITLE,"Communication Sciences And Disorders" )
OR LIMIT-TO ( EXACTSRCTITLE,"Pervasivehealth Pervasive Computing Technologies For Healthcare" )
ing analysis of high-tech AAC research.
OR LIMIT-TO ( EXACTSRCTITLE,"ACM Transactions On Accessible Computing" )
OR LIMIT-TO ( EXACTSRCTITLE,"International Journal Of Speech Technology" )
The eligibility process involved the frst author initially labelling
) papers as in, out 12 or unsure. Consequently from the 1288 papers
The SCOPUS query returned 1,531 papers in total. All queries – 490 papers were included, 723 papers were excluded for failing
were run on the 05/11/2021 and returned a combined 1,993 papers to meet eligibility criteria (3.2.3 to 3.2.3) and 75 labelled as unsure.
in total from 1978-2021. Checks on the labelling were performed by the second author and
followed by lengthy discussions on the unsure papers. Then from
3.2.2 Screening. A quick scan of dataset titles revealed that some the unsure papers 44 were included with a further 31 papers ex-
of the 1,993 were out of scope and required manual removal during cluded. Once all procedures were completed, 534 papers remained
the subsequent process of screening. Firstly, we discovered 39 du- in the dataset for manual qualitative coding meaning in total 754
plicates and eliminated them from our dataset. Secondly, we read papers had been excluded.
titles, abstracts and keywords – removing papers not focused on
3.2.4 Snowballing. Snowballing identifed a further 28 papers on
high-tech AAC research. Consequently, 595 of the results were
the topic of high-tech AAC consisting of peer reviewed literature
not papers within the research area and focused on factors such
from 1992-2021. Following guidance from Wohlin, we performed
as: advanced audio coding e.g., [38], digital content distribution
snowballing iterations until dead ends were reached and no new
e.g., [95], communication routing algorithms e.g., [139] and pro-
candidate papers were identifed from forward or backward snow-
teins amino acid composition e.g., [188]. A further 67 results were
balling every paper [196]. The process of snowballing took 3 itera-
clearly not full-text papers i.e., conference proceedings and 4 pa-
tions until Wohlin’s efciency metric reached 0% [196]. Through
pers were removed for not being in English. Throughout screening,
this, we identifed 44 potential candidate papers from titles and
an explanation fle was developed where the frst author provided
abstracts, yet upon further inspection all did not meet our eligi-
an index, reason, and explanation for the suggested removal of a
bility criteria for inclusion. After applying these, just 28 papers
paper from the dataset. Subsequently, the second author reviewed
were added to our dataset. Backward snowballing gave 18 papers
the dataset and explanation fle for quality control purposes – to
by studying the references of selected papers. Forward snowballing
jointly discuss any highlighted papers promulgating uncertainty
and verify removal suggestions made by the frst author. Papers 12 Each paper labelled as out – failed eligibility criteria was noted.
State of the Art in AAC
presented 10 papers by studying the works cited utilising Google
Scholar. After snowballing our dataset was 562 papers in total.
3.3 Analysis
Analysis involved qualitative coding of the entire dataset and cal-
culating Fleiss’s Kappa inter-rater reliability (IRR)13 [53] to reach
agreement between the three authors. Following this, we program-
matically examined paper counts over the 43-year period, per-
formed quantitative data extractions for participant counts, de-
veloped an inventory of key high-tech AAC devices and analyzed
corresponding results found by the Mack et al., dataset [112].
3.3.1 Qalitative Analysis of the Dataset. We qualitatively coded
all 562 papers from 1978-2021. The process of coding and analysing
the dataset was performed by 3 scholars. We note that the proce-
dure of building and analyzing the dataset involved subjectivity
and we appreciate that our scholarship refects our own biases and
beliefs. Authors and coders identifed as male and female of Western
and Southern European backgrounds, with no accessibility needs.
The codebook is synthesized to provide data for our three research
questions (i.e., 1.2, 1.2 and 1.2). Shown in Table 1, the fnal code-
book included 16 categories with 2-10 subcodes each. Out of the
16 categories, 8 were developed by the research team and 8 based
upon the Mack et al., codebook for accessibility research [112]. The
8 categories adapted from the Mack et al., were to cross-analyse
high-tech AAC research with current normative accessibility stan-
dards14 . In contrast, the 8 categories from the research team focused
on establishing a taxonomy and understanding of the dominant
characteristics of high-tech AAC devices. We wanted to code the
features and role of the device – its commercialism, inputs/outputs,
scalar attributes, communication types/model and scenarios of us-
age. This data would taxonomise and provide an understanding of
the current characteristics of high-tech AAC devices. The codes
were iterated by two authors using an iterative inductive analysis
approach – in which codes were independently developed and f-
nally agreed between both parties. Indeed, the two authors would
regularly meet – to refne and eliminate existing codes or add new
codes. Qualitative coding took three months and IRR was calculated
between the three authors – to mitigate against bias and fatigue.
The second and third author retrospectively15 coded a random sam-
ple of 10% (N=60) of the dataset to provide a Fleiss Kappa IRR and
provide an opportunity for disagreements to be resolved through
consensus.
3.3.2 Qantitative and Programmatic Analysis. For quantitative
analysis, we programmatically analyzed paper counts for the full
43 year period and developed novel visualisations to demonstrate
our fndings. Following this, we performed an analysis of partici-
pants counts for user studies – calculating mean, median, interquar-
tile range and standard deviation providing analysis for the entire
dataset and each community of focus. Then we constructed an
inventory of key devices regularly mentioned within the research
13 An extension of Cohen’s Kappa for three raters or more [53].
14 The identifcation of diferences, advantages and limitations of high-tech AAC versus
wider accessibility standards we believe will improve future high-tech AAC devices
and interventions.
15 The third author was new to the dataset and not involved in the code development
process – resulting in a marginally lower IRR versus other papers [112].
ASSETS ’22, October 23–26, 2022, Athens, Greece
and analyzed corresponding results from the Mack et al., SR into
accessibility research [112]. However, we accept that a limitation
of the comparison is that the Mack et al., dataset is from a more
recent period of 2010-9.
4 RESULTS
4.1 A Taxonomy of High-tech AAC Devices
and Interventions
We present results from the following 7 categories within our
dataset of 562 papers: interaction input, output modality, scalar
attributes, layout, scenarios/communication partners, communica-
tion model and communication type.
4.1.1 Input and Output Interaction. We successfully coded the in-
puts and outputs of high-tech AAC devices within the dataset (see
Figure 4). Starting with inputs, the most frequent locus of research
is mechanical (N=216, 38.4%) followed by tactile (N=184, 32.7%).
Since the earliest high-tech AAC systems, mechanical inputs have
come in a wide variety of forms. Examples include, switches [180],
keyboards [157], button presses [125], mechanical pointing de-
vices [125], trackballs [187], and joysticks [157]. Tactile inputs
serve as a key input modality for touchscreen controlled hard-
ware e.g., smartphones [124], tablets [124] and smartwatches. In
contrast, camera (N=77, 13.7%) and gestural input (N=48, 8.5%) tech-
niques have been explored to a lesser extent. However, cameras
have been leveraged for diferent interactions: primarily to enable
eye gaze [17], blink activated, head motion controlled high-tech
AAC systems [17], equally to better design and confgure AAC
systems, to provide gesture recognition for sign language [70] and
for photography to support contextual word discovery [91, 141],
personalisation of VSDs [192] and even storytelling. Body ges-
tures have been leveraged in diferent ways to control high-tech
AAC devices. Systems have been designed that are controlled by
tongue movement [143], musculature contractions [21], breath con-
trolled [46], heart rate signals and brain computer interfaces (BCIs).
BCIs have been developed for diferent brain signals such as EEG
and EMG signals [86]. Recent advances have been made using BCI’s
such as P300-based, the RSVP Keyboard and BrainGate to enable
individuals with a physical disability to use assisted communication
interfaces [59, 148, 149, 178].
Contextual (N=31, 5.5%), and verbal inputs (N=29, 5.2%) are less
explored in high-tech AAC. Contextual enables smart leveraging
of environmental knowledge to act as an input to the high-tech
AAC device [22, 99]. Knowledge of the context i.e., environment,
location or communication partner have been harnessed in high-
tech AAC systems for: natural language generation (NLG) [152],
synthesised vocabulary searches, discourse prediction, improved
adaptation to topics and to capture experiences [90, 91]. Verbal
inputs can be used as another input for high-tech AAC devices –
to provide utterance recognition [174], perform NLP on the com-
munication partners dialogue [195], function as a voice input voice
output device [72] and ofer prosody on the wearer’s dialect. Lastly,
orientational input (N=7, 1.2%) serves as an underexplored input
technique. Accelerometers have been harnessed to calibrate and im-
prove the conversation rate of people with motor impairments [66],
develop wearables to translate sign language in real time [8], for
ASSETS ’22, October 23–26, 2022, Athens, Greece Curtis et al.

Table 1: The fnal codebook is represented by 85 subcodes across 16 code categories. Retrospective qualitative Fleiss Kappa
IRR calculated across the subcodes for each category between the three authors. IRR ranged from almost perfect to fair with
a high pairwise agreement throughout.

Category Codes Pairwise Fleiss Kappa Level

agreement IRR
Participatory design Yes; No 90% 0.8 Almost perfect
Interaction input Verbal; Camera; Tactile; Gestural; Mechanical; 94.8% 0.789 Substantial
Orientational; Contextual
Ability-based comparison Yes; No 85.6% 0.709 Substantial
Output modality Audio; Visual; Motion; Gustation; Thermocep- 91.3% 0.693 Substantial
tion
Participant groups No user study; People with disabilities; People 88.4% 0.667 Substantial
without disabilities; Specialists; Caregivers
Communication model Linear; Interactive; Transactional 85.9% 0.644 Substantial
Communication type Verbal; Non-verbal 81.7% 0.622 Substantial
Use of commercial AAC Yes; No 77.8% 0.553 Moderate
Contribution type Empirical; Artifact; Methodological; Theoreti- 84.3% 0.543 Moderate
cal and opinion; Dataset; Survey
User study method Controlled experiment; Randomized control tri- 90.4% 0.501 Moderate
als; Survey; Usability testing; Interviews; Fo-
cus groups; Case study; Field study; Workshop;
Other
Use of proxies Yes; No 75% 0.5 Moderate
Community of focus BVI; DHH; Motor/physical impairment; Autism; 88.2% 0.462 Moderate
IDD; Other cognitive; Older adults; General dis-
ability; Other
Location No user study; Near/at researchers lab; Home, 85.2% 0.455 Moderate
residence or school; Neutral; Online/remote;
Other
Scalar attributes Morphable; Customisability; Automaticity; Ex- 88.9% 0.447 Moderate
pressivity; Adaptive; Practicality; Combined;
Parallel
Interface layout Symbols; Pictographic; Text; Animation; Grid; 80% 0.442 Moderate
VSD; Novel; No layout
Scenarios and communication Fellow AC; Family/friends; Professionals; 87.9% 0.388 Fair
partners Groups; Strangers; Anyone; Virtual; Unclear

speech therapy, and to make apps more usable amongst children
with disabilities [25]. Unlike inputs, outputs for high-tech AAC
devices have been less widely explored. Audio signals dominant
communication output (N=353, N=62.8%) – high-tech AAC devices
tend to serve as SGDs, VOCAs and for speech synthesis. High-tech
AAC outputting visual signals for the purpose of communication
have also been well explored (N=157, 27.9%) – including print-
able text interfaces, direct screens with captions, photographs, and
graphics for the communication partner [40, 150]. More abstract
visual signal systems have been researched including LED lights
and graphics [166]. Motion-based AAC is an emerging research
area (N=11, 2%). For instance, the motion of robots to communi-
cate [85] – including in LEGO adaptable forms for children [1] and
co-designed novel sidekicks [180]. In addition, haptic feedback has
been briefy explored for individuals with deaf-blindness [171]. Our
dataset did not include AAC that uses taste or heat as a modality
for communication.
4.1.2 Typical Features of High-tech AAC. In Table 2, the feature
space of high-tech AAC is rich – here we coded for scalar attributes
and the interface layouts of high-tech AAC. We found that ac-
cessibility researchers have prioritised developing high-tech AAC
that is customisable (N=310, 55.2%) and automatic (N=262, 46.6%).
Customisability has been honed as a signifcant factor for making
high-tech AAC more usable and personalized thereby increasing
adoption and acceptance of the high-tech AAC [34]. Automation has
been a well-explored and key area for high-tech AAC – automation
of keystrokes [61], predictive text, abbreviation expansion [158]
and leveraging AI [22] has been researched to improve users’ com-
munication rates. Following these two subcodes, high-tech AAC
has been developed to be expressive (N=110, 19.6%) for diferent
genders [68], cultural groups [82], and age ranges [35]. To a lesser
extent AAC has been developed to receive combinations of inputs
(N=57, 10.1%) to improve usability. Research has considered making
high-tech AAC more practical (N=45, 8%) – through making the
State of the Art in AAC ASSETS ’22, October 23–26, 2022, Athens, Greece

Inputs Outputs

Mechanical - 216 (38.4%) - 140 (24.9%)

Audio - 353 (62.8%) - 280 (49.8%)

Tactile - 184 (32.7%) - 111 (19.8%)

Visual - 157 (27.9%) - 87 (15.5%)

Camera - 77 (13.7%) - 34 (6%)

Gestural - 48 (8.5%) - 28 (5%) Motion - 11 (2%) - 4 (0.7%)

Contextual - 31 (5.5%) - 5 (0.9%)

Thermoception - 0 (0%) - 0 (0%)
Verbal - 29 (5.2%) - 8 (1.4%)
Gustation - 0 (0%) - 0 (0%)
Orientational - 7 (1.2%) - 3 (0.5%)

Figure 4: Sankey diagram proportionally representing inputs and outputs of AAC found in the review. The frst number indi-
cates the frequency, the second the number or instances this code was found without other inputs/outputs.

Table 2: Frequency of applied codes for high-tech AAC scalar attributes and high-tech AAC interface layouts mentioned in
dataset.

Scalar attributes Papers w/code This code only Interface layout Papers w/code This code only
Customizable 310 (55.2%) 81 (14.4%) Text 322 (57.3%) 84 (14.9%)
Automatic 262 (46.6%) 39 (6.9%) Symbols 223 (39.7%) 18 (3.2%)
Expressive 110 (19.6%) 8 (1.4%) Grid format 157 (27.9%) 2 (0.4%)
Combined 57 (10.1%) 3 (0.5%) Pictures/drawn 127 (22.6%) 8 (1.4%)
Practical 45 (8%) 4 (0.7%) No layout 29 (5.2%) 29 (5.1%)
Adaptive 42 (7.5%) 3 (0.5%) VSDs 29 (5.2%) 2 (0.4%)
Parallel 17 (3%) 0 (0%) Video or animation 23 (4.1%) 4 (0.7%)
Morphable 14 (2.5%) 0 (0%) Novel 5 (0.2%) 2 (0.4%)

device discreet, wearable, small and lightweight [193]. Accessibility
researchers have considered leveraging AI systems to improve the
adaptiveness (N=42, 7.5%) of the AAC system – to improve feature
recognition i.e., eye gaze/landmark detection, to orient to the user
over time [6] and for improving frequency prediction algorithms
of vocabulary dependent on location and usage history [62].
A small amount of research has considered making high-tech
AAC parallel (N=17, 3%) and morphable (N=14, 2.5%). Examples of
parallel AAC includes a high-tech device that receives eye-gaze and
simultaneous input suggestions from the communication partner to
increase communication rates during exchanges [51]. Limited high-
tech AAC is morphable thereby physically adaptive according to
the consumer preferences or designed to run on diferent hardware
and devices. Some examples, include do it yourself (DIY) AAC
kits [71], Lego robots physically confgurable by the child [2] and
an AAC device that changes form factor by running on either a wrist
wearable and head worn display [193]. In terms of interface, we
fnd that most high-tech AAC interfaces use text (N=322, 57.3%) and
symbols (N=223, 39.7%). Historically, grid formats (N=157, 27.9%)
for AAC often with accompanying symbols have been popular [4].
To a lesser extent high-tech AAC has leveraged pictures or been
drawn (N=127, 22.6%). Few papers had no or a limited interface
meaning the AAC operated of sensors (N=29, 5.2%) [151]. A notably
small number of papers (N=29, 5.2%) provided research of high-tech
AAC with a visual scene display (VSD). Lastly, a small selection
of papers (N=5, 0.2%) had entirely novel interfaces including LED
morse code [78], augmented reality [94], and tactile surfaces [171].
4.1.3 Communication Supported by High-tech AAC. High-tech
AAC is designed for diferent scenarios and contexts of usage (see
Table 3) – here we code if papers explicitly mentioned contexts in
which the device could be used. Signifcantly, devices have mainly
ASSETS ’22, October 23–26, 2022, Athens, Greece
Table 3: Frequency of applied codes for scenarios of high-
tech AAC usage mentioned within dataset.
Scenario Papers w/code This code only
Professionals 315 (56%) 155 (27.8%)
Family/friends 221 (39.3%) 56 (10%)
Strangers 65 (11.6%) 14 (2.5%)
Groups 39 (6.9%) 1 (0.2%)
Virtual 37 (6.6%) 21 (3.7%)
Fellow AAC users 11 (2%) 1 (0.2%)
Anyone 7 (1.2%) 5 (0.9%)
Unclear 22 (3.9%) 22 (3.9%)
been built for professional contexts typically with specialists, teach-
ers or carers (N=315, 56%). Additionally, high-tech AAC has often
been designed to be used with family and friends (N=221, 39.3%) at
home or in social settings. A smaller number of papers (N=65, 11.6%)
mentioned designing high-tech AAC for use with strangers or unfa-
miliar communication partners. In addition, an even smaller number
of papers discussed high-tech AAC to be used with groups (N=39,
6.9%) and virtual communication (N=37, 6.6%). Group scenarios
involve several people – making the communication environment
pose potentially signifcantly more challenges for the high-tech
AAC device [180]. Also high-tech AAC will need to facilitate user’s
self-expression and communication in virtual environments (N=37,
6.6%) – a small subset of papers has considered this for phone
calls [50], videoconferencing [96], and online gaming [169]. Lastly,
we fnd little high-tech AAC aware of supporting communication
exchanges between fellow AAC users (2%) with the same or a dif-
ferent high-tech AAC device – yet it is potentially common for
two high-tech AAC users to directly communicate with each other
regularly e.g., within special needs schools [81, 140]. In Table 4, we
fnd that most high-tech AAC is designed to augment and enrich
verbal communication (N=519, 92.3%) – high-tech AAC to just sup-
port users non-verbal forms of communication is neglected in the
literature (N=8, 1.4%) [52]. Furthermore, we fnd limited high-tech
AAC devices operating at communication rates beyond interactive.
Instead, linear is by far the most common communication model
(N=491, 87.4%) – here using the device to provide feedback is re-
strained and the AAC user must take signifcant time to construct
messages [11]. Interactive high-tech AAC devices (N=61, 10.9%)
increase the communication rate by enabling the AAC user to ofer
feedback in a restrained manner. We fnd some high-tech AAC
(N=8, 1.4%) could potentially ofer pathways to transactional com-
munication for its user. Here, high-tech AAC leverages snippets
of discourse utilising advanced technologies to take in contextual
information [63, 195] signifcantly diminishing the reciprocity gap
between communicators [27].
4.2 The Incumbent Methodologies Used to
Design and Study High-tech AAC
4.2.1 High-tech AAC Contribution Counts and Types. Depicted in
Figure 5, paper counts for high-tech AAC has grown steadily over
the period of 1978-2021 yet this growth surges with counts over 25
Curtis et al.
for each year since 2011. This replicates wider trends noting the in-
creasing prominence of accessibility research [112]. Notable market
available hardware innovations have also fed this growth. High-tech
AAC devices have become more readily available with designers
able to shift from dedicated hardware e.g., Dynavox DynaMyte 3100
(1999) to increasingly developing apps e.g., Proloquo2Go (2009) for
accessible touch-screen devices such as smartphones e.g., iPhone
(2007) and smart-tablets e.g., iPad (2010). Further analysing the
input/output data over time in Table 5, we fnd trends have to a
greater extent reciprocated evolution’s in the market available hard-
ware and a general increase over time16 . Predictably, in terms of
inputs camera (0% to 22.4%) and tactile touchscreen technology has
increased with time (0% to 43.3%).
Whilst, mechanical inputs have steadily fallen over time (100%
to 22.4%). In Table 6, we found the majority of high-tech AAC con-
tributions were empirical (N=410, 73%) and artifact (N=224, 39.9%)
contributions. The two contributions regularly occurred in conjunc-
tion (N=169, N=30.1%). Similarly, in the Mack et al. dataset these
were both the two most popular contribution types [112]. Equally,
the following four contribution types occurred comparatively less
with Survey (N=60, 10.7%), methodological (N=43, 7.7%), theoreti-
cal and opinion (N=19, 3.4%) and Dataset (N=6, 1.1%). However, a
notable diference with the Mack et al., was that literature survey
contributions were much higher for high-tech AAC (+10.1%) and
typically published within the AAC journal (N=31/60, 51.6%) – lit-
erature surveys are a regular practice to provide essential refection
on current medical practices [155].
4.2.2 Location of User Studies. Incorporating locations beyond the
lab may enable accessibility researchers to have more users and test
if high-tech AAC functions in settings where it will be used day-
to-day. Our results noted in Table 7, quite closely replicate trends
found in the Mack et al., research [112]. A substantial proportion
of studies take place at home, residence or school (N=181, 43.6%)
and near or in the researchers laboratory (N=178, 42.8%). High-tech
AAC researchers have been successful at recruiting participants
for studies that take place at home or schools – a place where par-
ticipants are comfortable and visit frequently [55]. Furthermore,
user studies in these locations may broaden participation amongst
vulnerable groups e.g., testing high-tech AAC with children [55].
Lab studies are almost as popular, enabling the research team to
carefully control the variables and environment for testing – sup-
porting the collection of observation data [173]. Neutral locations
(N=71, 17.1%) feature relatively prominently and are diverse such
as: day programs [39], intervention camps [32], clinics, medical
centres [163], fast food restaurants [44], extracurricular clubs and
community centres [191]. Some of these locations have the ad-
vantage of testing the high-tech AAC in live, realistic and natural
conditions that are hard to replicate within a lab. Often neutral
locations are selected by the participants themselves due to per-
sonal preference [118]. Furthermore, these settings may imbue the
user with confdence that the AAC can be used in a public location.
Online and remote participation (N=50, 17.1%) has been used to
16 We also note that this review was done approximately 1/5 the way through the 2020s,
so accounts for a smaller number of papers. Extrapolating the number (i.e. multiplying
the number of papers in the 2020s by 5), we get 335 papers – suggesting a continuing
increase in the number of papers on AAC.
State of the Art in AAC ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 4: Frequency of applied codes for high-tech AAC communication type and model within dataset.

Comms type Papers w/code This code only Comms model Papers w/code This code only
Verbal 519 (92.3%) 517 (92%) Linear 491 (87.4%) 430 (76.5%)
Non-verbal 8 (1.4%) 6 (1.1%) Interactive 61 (10.9%) 0 (0%)
Transactional 8 (1.4%) 0 (0%)

40
37
36 36
35
32
31
30 30 30
30
28
27
26
25
22
Paper counts

21
20
20
17 17
16
15
15

10 9
8
7 7 7
6 6 6 6
5 5 5 5
5
3 3
1 1 1
0 0 0 0 0 0 0 0
0
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
Year

Figure 5: Frequency of 562 paper counts by year for high-tech AAC within dataset from 1978-2021.

test with large participant numbers and also may remove travel
burdens for participants [168].
4.2.3 Methods in User Studies. Refecting on the user-centred na-
ture of high-tech AAC, overwhelmingly user studies are favoured
(N=415, 73.8%) – despite the potential difculties of obtaining partic-
ipants with CCNs who often cannot easily provide verbal consent.
In contrast, studies which do not incorporate user studies (N=147,
26.2%) involve artifacts with no user testing (N=57, 10.1%) survey
(N=53, 9.4%), methodological (N=25, 4.4%), theoretical (N=15, 2.7%),
empirical (N=7, 1.2%) and dataset (N=4, 0.7%) contributions. These
studies without formal user testing – typically involve prototype
development [177], exploratory studies [108] and analyses of user-
studies conducted in other research [154]. Noted in Table 8, the
dominant preference for studying high-tech AAC is controlled
experiments (N=188, 45.3%) and usability testing (N=119, 28.7%). In-
terviews (N=67, 16.1%) rank to a lesser extent and typically involves
interviewing speech and language therapists (SLTs) and family
members due to the challenges of directly interviewing someone
with CCNs [132]. Field studies (N=75, 18.1%) and case studies (N=48,
11.6%) also rank quite highly – researchers often try to learn of the
success of deployment by observing in naturalistic settings high-
tech AAC usage [67]. Lastly, we found no contributions using ran-
domized control trials (N=0, 0%) (RCTs). Surveys and questionnaires
(N=65, 15.7%) have been regularly deployed to assess over large
groups of users and gain an understanding of potential patterns
and commonalities [168]. Furthermore, surveys enable high-tech
AAC users to communicate at there preferred rates without feel-
ing pressure to provide feedback quickly [133] and AAC is a well
established community with foundations/charities with extensive
mailing lists [47, 121, 183]. Focus groups (N=21, 5.1%) and work-
shops (N=4, 1%) feature quite lowly – due to the complexities of
gaining feedback from users with communication barriers [16, 90].
Comparing to the broader Mack et al. dataset [112], high-tech AAC
research comparatively favours more case studies (+7.6%) and con-
trolled experiments (+10.7%) at the expense of interviews (-26%),
workshops (-17.4%) and usability testing (-13%).
4.2.4 Participatory Methods. Participatory design (PD) provides a
method for involving users of technology direct in its design [57,
ASSETS ’22, October 23–26, 2022, Athens, Greece Curtis et al.

Table 5: Incremental 10 year binning of frequency of applied codes for input/output data of AAC over time. Percentage calcu-
lation involves dividing number of papers with code by paper count within period. Data is weighted towards AAC from the
2000s and 2010s, with a steady year-on-year increase over time. We see a gradual difusion in output types, away from only
mechanical to a more equal split of form factors. We see audio and visual dominating as outputs consistently over time.

Input 1970s 1980s 1990s 2000s 2010s 2020s

Paper count 1 11 49 136 298 67
Verbal (%) 0 9.1 2 8.8 5 0
Camera (%) 0 0 0 7.4 17.5 22.4
Tactile (%) 0 0 0 27.9 39.3 43.3
Gestural (%) 0 18.2 2.1 6.6 9.7 10.5
Mechanical (%) 100 81.8 71.4 55.2 27.2 22.4
Orientational (%) 0 0 0 0 1.7 3
Contextual (%) 0 9.1 2.1 4.4 6 7.5
Output 1970s 1980s 1990s 2000s 2010s 2020s
Audio (%) 0 45.5 65.3 71.3 59.1 64.2
Visual (%) 100 45.5 28.6 22.8 28.9 29.9
Motion (%) 0 0 0 0 2.4 6
Gustation (%) 0 0 0 0 0 0
Thermoception (%) 0 0 0 0 0 0

Table 6: Frequency of applied codes for high-tech contribution types within dataset versus Mack et al. accessibility data [112].

Contribution types Papers w/code % dif. This code only % dif.

Empirical 410 (73%) +12.7 226 (40.2%) +6.4
Artifact 224 (39.9%) -15.6 52 (9.2%) -26.8
Survey 60 (10.7%) +10.1 51 (9.1%) +9.1
Methodological 43 (7.7%) +4.5 18 (3.2%) +2.8
Theoretical and opinion 19 (3.4%) -5.3 12 (2.1%) +0.9
Dataset 6 (1.1%) -0.3 6 (1.1%) +0.7

Table 7: Frequency of applied codes for location for 415 user study papers within dataset versus Mack et al. accessibility
data [112].

Study location Papers w/code % dif. This code only % dif.

Home, residence or school 181 (43.6%) +14.7 139 (33.5%) +15.7
Near/at researchers lab 178 (42.9%) +15.6 145 (34.9%) +15.4
Neutral location 71 (17.1%) +10.4 37 (8.9%) +5.8
Online/remote 50 (12%) -8.5 34 (8.2%) -1.9

112]. However, PD methods have not been widely adopted in high-
tech AAC user-study papers (N=20, 4.8%) – lower than the Mack
et al. dataset (-5.5%). Indeed, for users with CCNs – traditional PD
methods are more inaccessible compared to other communities
and groups [28, 136]. PD is cognitively demanding and requires
for people with CCNs to have high levels of speech and language
profciency [136, 194]. Prior research has outlined challenges in
engaging populations with autism [57], aphasia [28, 90], Parkin-
son’s/dementia [26] and IDD [136] in PD. Researchers have de-
veloped solutions to mitigate against this through proxies [28] –
e.g using SLTs for the PD of high-tech AAC. Other studies that
did not use co-design engaged in other rich design activities such
as user-centred design [186], intervention programs [187], use of
probes [129] and academic workshops [195].
4.2.5 Category of Devices. Figure 6, shows that research has regu-
larly provided interventions using expensive standalone commer-
cial high-tech AAC devices including DynaVox models (N=63) e.g.,
the Dynavox 3100 (N=9) and their eyetracking technologies the To-
bii models e.g., Tobii T60 (N=5), Tobii X120 (N=1). Other standalone
high-tech AAC devices used in research includes the Lightwriter
(N=14), Pathfnder (N=14) and Liberator (N=13). Yet, recently re-
search has increased with downloadable software e.g, EZ keys (N=6),
SentenceShaper (N=7) or apps e.g., Proloquo2Go (N=16), Go Talk
Now (N=11), which converts laptops, smartphones, iPad’s (N=46)
State of the Art in AAC ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 8: Frequency of applied codes for methods for 415 user study papers within dataset versus Mack et al. accessibility
data [112].

Method Papers w/code % dif. This code only % dif.

Controlled experiment 188 (45.3%) +10.7 128 (30.8%) +19.3
Usability testing 119 (28.7%) -13 45 (10.8%) +1.2
Field studies 75 (18.1%) +0.3 25 (6%) +1.4
Interviews 67 (16.1%) -26 18 (4.3%) -1.4
Surveys and questionnaires 65 (15.7%) -9.9 23 (5.5%) +4.2
Case studies 48 (11.6%) +7.6 32 (7.7%) +7.5
Focus groups 21 (5.1%) -0.8 7 (1.7%) +0.9
Workshops 4 (1%) -17.4 1 (0.2%) -2.9
Other 2 (0.5%) -15.6 2 (0.5%) -0.3
Randomized control trials 0 n/a 0 (0%) n/a

Vanguard models 5
Vantage models 6
Minspeak icon system 6
TouchTalker models 6
EZ keys software 6
SentenceShaper software 7
AAC devices and technologies

BIGmack switches 7
Dynavox 3100 9
GoTalk Now app 11
DeltaTalker models 11
Tablet 11
DecTalk synthesizer 13
Liberator models 13
Pathfinder models 14
Lightwriter models 14
Proloquo2go app 16
Tobii T60 and eye tracking technologies 21
iPad high-tech AAC 46
Dynavox related devices 63
0 10 20 30 40 50 60 70
Frequency of papers

Figure 6: Inventory of 19 key devices and technologies with a frequency of above 5 papers within dataset.

or tablets (N=11) into a high-tech AAC device has grown in promi-
nence. Elsewhere, we fnd a slight preference towards testing with
commercial (N=227, 54.7%) versus non-commercial (N=188, 45.3%)
high-tech AAC within user-study papers. Indeed, for SLTs and
non-computer science research groups it is often easier to program
and customise pre-existing available commercial high-tech AAC
devices than develop new non-commercial high-tech AAC. Other
advantage of commercial high-tech AAC is that they are a more
reliable intervention as the device is maintained by an external cor-
poration [4]. Non-commercial oferings involve the research team
actively building and developing new high-tech AAC solutions for
the end users within the study. Advantages of the non-commercial
solutions is that the high-tech AAC is often free to its user, and
does not involve external purchase, subscription and maintenance
fees [186].
4.3 Communities of Focus and Participants for
High-tech AAC Research
4.3.1 The Communities of Focus. Shown in Table 9, high-tech AAC
research has largely focused on users categorised as Other (N=219,
39%) and motor impairments (N=182, N=32.4%). Users categorised
as “Other" do not belong to a specifc community of focus and
ASSETS ’22, October 23–26, 2022, Athens, Greece
Table 9: Frequency of applied codes for community of focus within dataset versus Mack et al. accessibility data [112].
Method Papers w/code
Other 219 (39%)
Motor impairments 182 (32.4%)
Other cognitive 74 (13.2%)
Autism 71 (12.6%)
IDD 67 (11.9%)
DHH 14 (2.5%)
BVI 11 (2%)
Older adults 4 (0.7%)
General disability 3 (0.5%)
the research is to contributing to a broad community of high-
tech AAC users e.g., “children" or “AAC users" – this is despite
each disabled group having very specifc requirements17 . Motor
impairments have also received a signifcant research contribu-
tion (N=182, 32.4%). In particular, disabilities such as cerebral palsy
(N=107, 19.1%) and ALS (N=20, 3.6%) have been a specifc focus for
the motor-impairment high-tech AAC research. Groups that have
received less research include cognitive impairments (N=74, 13.2%),
autism (N=71, 12.6%) and IDD (N=67, 11.9%). Within other cognitive
impairments, the dominant contribution has been towards people
living with aphasia (N=47, 8.4%) and people with TBIs (N=10, 1.8%).
Many of the autism and IDD contributions have built high-tech
AAC with child and adolescent users e.g., [55]. Groups with little
research include BVI (N=11, 2%), DHH (N=14, 2.5%), Older adults
(N=4, 0.7%) and General disabilities (N=3, 0.5%) – these are not
mainstream users of high-tech AAC with CCNs. Our data versus
the Mack et al. dataset for accessibility research reveals some difer-
ences [112]. In broader accessibility research BVI is the dominant
focus (-41.5%), whilst in high-tech AAC it is Other (+29.9%) and
motor impairments (+18.2%). Also, Autism (+6.5%), IDD (+9.1%) and
Other cognitive impairments (+4.1%) received a slightly higher pro-
portion of contributions within our dataset. However, DHH (-8.8%),
Older adults (-8.2%) and General disability (-8.6%) received a slightly
lower contribution within our dataset.
4.3.2 Study Participants. Table 10 shows the participant data for
user studies. The majority of studies include people with disabili-
ties (N=305, 73.5%) albeit in these studies the median participant
counts are predominantly low (N=5) and imbalanced with a high
standard deviation (SD) and range. Equally, fewer papers include
people with just disabilities only (N=178, 42.9%). A proportion of
the papers include caregivers (N=75, 18.1%) and specialists (N=62,
14.9%) particularly often to complement other users involved in
the user-study18 . A sizeable proportion of the user studies include
people without disabilities (N=130, 31.3%) with even N=82 papers
(19.8%) only with people without disabilities. For studies including
people without disabilities we fnd participant counts are much
higher with a median of N=18. Due to the difculties designing
for users with CCNs – proxies are often favoured [28]. Indeed, it
17 E.g.,
The high-tech AAC requirements of a user living with aphasia is very diferent
to a user living with cerebral palsy.
18 Indeed, only N=15 papers are coded with just specialists and N=6 coded with just
caregivers.
Curtis et al.
% dif. This code only % dif.
+29.9 209 (37.2%) +33.2
+18.2 146 (26%) +14.3
+4.1 59 (10.5%) +4.8
+6.5 46 (8.2%) +4
+9.1 29 (5.2%) +3.6
-8.8 5 (0.9%) -7.6
-41.5 2 (0.4%) -40.2
-8.2 0 (0%) -5.7
-8.6 1 (0.2%) -5.9
is even quite common for papers to test with proxies to gain ini-
tial data on high-tech AAC before considering a disabled group
of users [28]. Albeit, sometimes it makes legitimate sense to use
participants without disabilities because the research is examining
perspectives on high-tech AAC – monitoring its social acceptance
amongst peers e.g., [101].
A signifcant proportion of high-tech AAC user-study papers
include specialists (14.9%) and caregivers (18.1%). Additionally, in
these studies, the median participant count positively increases
for specialists (N=9.5) and caregivers (N=8.5). Indeed, this is to
be expected as people with CCNs in user-studies sometimes re-
quire caregivers and specialists to assist with communicating and
expressing themselves [136]. Equally, these groups can provide
key insights and feedback on the high-tech AAC within the user-
study [28]. Comparing these fndings versus the wider Mack et
al. research we fnd a lower number of user studies coded with
people with disabilities (-11.2%) and higher number of user stud-
ies using people without disabilities (+8.2%). In addition, we fnd
a marginally lower number of papers with specialists (-2.1%) but
much higher usage of caregivers (+8.7%). In terms of community of
focus, participant counts are median highest for the non-specifc
category of Other (N=16). However, median participant counts are
lower for Motor impairments (N=6) and Other cognitive (N=6),
yet participant counts are very low for high-tech AAC user-study
papers involving people with IDD (N=5) and Autism (N=4)19 . Over-
all, the low median participant counts, high ranges and standard
deviations refect the methods favoured in high-tech AAC research
– very large surveys versus interventions with just one or a small
group of users.
4.3.3 Usage of Proxies and Ability-based Comparisons. From our
415 user-study papers, we identifed 86 papers where proxies were
used (N=86, 20.7%) – higher than the Mack et al. dataset (+12.7%).
To a certain degree, this trend was expected as proxies are often
used to circumvent communication barriers [28]. However many
of the proxy papers (34.9%, N=30/86) were for motor impairments,
which compromises the validity of fndings as it is difcult for a
proxy user to accurately replicate the role of a motor impaired
user [113]. Sometimes a member of the research team has acted
as a proxy high-tech AAC user to obtain social perspectives from
19 Typically, these studies can often include vulnerable children users of high-tech
AAC resulting in a lower-participant count
State of the Art in AAC ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 10: Frequency of applied codes for study participants within 415 user study papers versus Mack et al. accessibility
data [112].

Participant group Papers w/code % dif. This code only % dif.

People with disabilities 305 (73.5%) -11.2 178 (42.9%) -2
People without disabilities 130 (31.3%) +8.2 82 (19.8%) +18.8
Caregivers 75 (18.1%) +8.7 6 (1.4%) +0.6
Specialists 62 (14.9%) -2.1 15 (3.6%) +1.7

Table 11: Quantitative analysis of participant counts for 415 user-study papers by community of focus and participant groups.

Group Median IQR SD Mean Range Total papers

Older 78 79 80.3 94.7 24–182 3
People without disabilities 18 20.25 84.6 32.7 1–893 130
Other 16 27.5 1188.5 146.1 1–12,776 121
Specialists 9.5 18.75 63.8 31.6 1–320 62
DHH 9.5 37.8 3678.5 1085.6 1–12,776 12
Caregivers 8.5 26 87.2 37.6 1–624 75
General 8 4.5 4.7 6.3 1–10 3
All user-studies 7 17 635.1 54.2 1–12,776 415
Other cognitive 6 9.5 12.4 10.5 1–75 63
Motor impairments 6 12 1024.3 105.4 1–12,776 158
People with disabilities 5 11 735.7 62.5 1–12,776 305
IDD 5 8 1719.9 243.9 1–12,776 55
Autism 4 8.75 1735.8 248 1–12,776 54
BVI 2 20 22 14.9 1–60 8

peers on the device and its user [176] and proxies are used for initial
pilot testing of devices to quickly determine the efectiveness and
usability of the device [197]. We identifed just 25 cases of ability
based comparisons (N=25, 6%) in which disabled and non-disabled
participants were compared in the user-study papers – sometimes
this is to have able participants act as the control group for deter-
mining if the high-tech AAC SGD ofers the same intelligibility as
natural human vocals [127] and whether the communication rate is
equivalent [88]. Additionally, testing has been performed in dyads
to determine cross-task performance [159]. Comparing to the Mack
et al., ability based comparisons are used less frequently (-7.6%)
within our dataset [112].
5 DISCUSSION
Despite four decades of research, abandonment of high-tech AAC
devices and interventions continues [140, 185]. Therefore, this frst
SR and taxonomy of ACM high-tech AAC research serves as a
refection for more successful future high-tech AAC development
and intervention outcomes. Based on our taxonomy and results,
we formulate directions for further research in high-tech AAC and
interventions in light of our three research questions: 1.2 to 1.2.
5.1 Expanding the Characteristics of High-tech
AAC Devices and Interventions
Our frst research question – presents what is the pre-existing
taxonomy and dominant characteristics of high-tech AAC. From
our examination and codifcation of the literature we fnd that me-
chanical (38.4%) was the dominant input modality for standalone
high-tech AAC devices before a surge in tactile (32.7%) inputs with
the decentralisation of high-tech AAC through capacitive touch-
screens (e.g., smartphones, tablets) and easily downloadable apps
(e.g. through app stores). However, audio (62.8%) and visual (27.9%)
have remained the most well explored output modalities. In re-
sponse to these fndings, high-tech AAC designers should more
systematically explore the input/output interaction possibilities of
using high-tech AAC. For example, with regards to inputs: camera,
gestural, verbal and contextual inputs serve as a still largely un-
explored input modalities (5.2% to 13.7%), which we believe hold
much promise for future high-tech AAC. For instance, Fiannaca
et al., increased gaze-based and eye-tracking communication rates
leveraging contextual inputs with the AACrobat [51]. Additionally,
Wisenburn et al., used verbal inputs from partner speech recog-
nition providing contextually relevant utterances and increased
communication rates [195]. Lastly, with regards to gestural inter-
action Ascari et al., explored non-invasive personalised gestural
interaction for the purposes of communication [7] whilst BCI-AAC
remains an emerging research area, which enables practitioners to
entirely bypass users motor system – albeit the successful deploy-
ment and calibration of BCIs is currently challenging [119, 148, 149].
High-tech AAC devices that have leveraged novel input charac-
teristics have ofered promising results for their user groups, for
instance using camera input for eye-tracking [97], storytelling [3]
and understanding the environment [141]. Harnessing pre-existing
ASSETS ’22, October 23–26, 2022, Athens, Greece
interdisciplinary AI research such as natural language process-
ing (NLP) [73] or smart environments [23, 36, 64], might support
the development of contextual and verbal inputs for high-tech
AAC to improve communication rates, enable easier interactions,
more agency and autonomy. Complementary research on outputs,
fnds that high-tech AAC typically outputs audio (62.8%) and visual
(27.9%) channels to communicate – future work might explore the
feasibility of the motion (2%) and even thermoception (0%) as out-
puts. Indeed, these more subtle and less information dense channels
can still be leveraged to support communication and engaging hu-
man to human interaction, particularly non-verbal communication
(c.f. “sidekicks" [179] and physical expressive objects [180]).
With respect to common scalar attributes and interface layouts of
high-tech AAC, we fnd that AAC is often customisable (55.2%) and
ofers automation (46.6%) typically for prediction to foster increased
communication rates [74]. Whilst high-tech AAC layouts often use
text (57.3%), symbols (39.7%), grids (27.9%) and pictures (22.6%) –
for instance the DynaVox 3100 serves as a typical grid, symbol
and text based interface [4]. To better expand these characteristics,
future AAC that just ofers customisation and automation is likely
not enough. Indeed, too often using high-tech AAC does not feel
natural but restrained for the end user’s autonomy resulting in
abandonment [81, 140] – the voice of the device does not refect their
personhood [117], the device is impractical to carry around day-to-
day [11], indiscreet limiting social acceptance [145], difcult to use
with few modes of input [20] and not adaptive to the constantly
evolving communication abilities of the user [20, 80]. Additionally,
future AAC research could explore more VSDs, video and animation
– these are under researched interfaces with the ability to convey
meaning [83]. Importantly, high-tech AAC designers must be aware
that text-based interfaces are challenging for some groups – e.g.,
people with aphasia [189].
Turning to scenarios of usage, communication types and mod-
els – we fnd that high-tech AAC devices are typically designed to
be used with professionals (56%) and families/friends (39.3%) sup-
porting verbal communication (92.3%). However, high-tech AAC
designers must be aware that users with CCNs often have larger
social groups than anticipated [180], users also require the ability to
use their high-tech AAC to communicate virtually (6.6%) on social
media [34, 123] or as widely appreciated during the pandemic, on
videoconferencing software [96, 137, 180]. Little high-tech AAC
is designed to function in group settings (6.9%), particularly if the
communication partner is a fellow AAC user (2%) with the same or
a similar high-tech AAC device – research has shown that in this
case breakdowns are common unless both users have analogous
vocabulary libraries pre-installed prior to the exchange [20, 81]. In
addition, there has been an under representation of high-tech AAC
that empowers the users non-verbal communication (1.4%) – this
compounds into most high-tech AAC ofering linear communica-
tion interactions (87.4%) [180]. Indeed, high-tech AAC must show
an appreciation of the users pre-existing communication abilities
to increase communication rates beyond linear – encouraging em-
bodied forms of communication and re-framing high-tech AAC as
an accepted interdependent assistive technology [15, 81]. Too often,
high-tech AAC restrains communication by acting as a physical
independent barrier – restraining the user from freely employing
Curtis et al.
there own unique communication skills including non-verbal sig-
nals to provide feedback more quickly [15, 52, 81].
5.2 Reassessment of Incumbent Research
Contributions and Methodologies in
High-tech AAC Research and Interventions
The second research question of this paper focused on what meth-
ods are used to contribute towards high-tech AAC research and
interventions. Versus other contributions, dataset contributions
(1.1%) are comparatively under-explored, thus high-tech AAC re-
search could certainly have more contributions focused on AI and
ML datasets. As proposed by Mack et al. and Kane et al., looking
to the future, AI and ML datasets should certainly be rooted in ac-
cessibility research [112] and equally have the potential to greatly
enrich the efectiveness of high-tech AAC research and interven-
tions [92, 140]. Already, work at ASSETS’21 by Theodorou et al.
[172] explores the concept of a “Disability-First” dataset, which
supports an accessible method for visually impaired users to create
their own datasets for personal object recognition. Furthermore,
as rightly noted by Park et al., there is a defnitive lack of data
from disabled populations and consequently they have provided a
robust set of design guidelines for developers looking to gather data
from disabled populations [147]. Indeed, the properly considered
deployment of AI within high-tech AAC might support quicker,
empathetic and more seamless communication opportunities for
people living with CCNs [140]. For instance, Vertanen et al., suc-
cessfully deployed crowd-sourced data-sets to ofer improved word
prediction and better high-tech AAC communication rates – yet
critically, the underlying data did not originate specifcally from
AAC users and disabled communities [184].
User-study papers (73%) compromise a sizable proportion of our
dataset and we believe this must continue and even increase for suc-
cessful high-tech AAC outcomes [140]. The location of user studies
is signifcant for high-tech AAC – researchers have successfully
tested widely in: residences (43.6%), labs (42.8%), neutral (17.1%) and
even remote (12%) locations. Nonetheless, researchers must look to
test high-tech AAC performance in the very conditions where it
will be deployed. Key reasons preventing high-tech AAC adoption,
include that the devices promise an unrealistic “magic wand” [20],
failing to adapt to the contextual environmental challenges (e.g.,
noisy environments) [96, 117] or consider peer perceptions of the
high-tech AAC device [5] – the potential accompanying stigmas and
its efect in conversational power dynamics [15, 101]. Indeed, the
high-tech AAC device should be shaped by its user and context of
usage [20] – high-tech AAC performance cannot be assumed on the
basis of unnatural controlled conditions rather than the actual con-
texts of daily usage [140]. With regards to high-tech AAC research
methods, we found no cases of randomized control trials (RCTs)
(N=0, 0%) within our dataset. Indeed in 2018, Kent-Walsh et al. report
that no RCTs have been published in the AAC journal [93]. How-
ever, further searching outside of the realm of this SR within the
medical literature20 we do fnd some RCTs with AAC e.g., [79, 134].
Furthermore, Todman et al. have strongly advocated for their usage
20 Beyond the scope of our investigation, RCTs with AAC were found via medical
literature databases i.e., https://pubmed.ncbi.nlm.nih.gov/ and Google Scholar.
State of the Art in AAC
as AAC devices are categorised as medical equipment requiring ran-
dom assignment procedures for treatment efcacy [175]. Therefore,
we suggest further RCTs to mitigate against bias and the ensure
clinical efcacy of high-tech AAC interventions [176].
Elsewhere, usability testing (28.7%), interviews (16.1%), work-
shops (1%) and PD methods (4.8%) all rank lower than the Mack et
al. dataset for the general accessibility community. Consequently,
we fnd a smaller use of methods in which AAC users with CCNs
are directly engaged and the development process is cooperative
e.g., [23, 90]. We accept that it is difcult to recruit, engage in
interviews, participatory design, usability testing [16] and work-
shops [57] with users that have severe CCNs, struggle to communi-
cate independently [90] and provide consent [28, 136]. Nonetheless,
we echo Mankof et al. [114] – we must, as a community, actively
strive to include end users within the design and development pro-
cess of AAC. Researchers have, in summary, not received enough
feedback21 from empowered end users during the development pro-
cess. Future research should build upon prior PD which has focused
on motor-impairments [41], autism [57, 167], dementia [109], apha-
sia [60, 138, 194] and older individuals [110]. Further participatory
designed high-tech AAC solutions for people with CCNs could re-
sult in better outcomes and more long-term adoption [136]. Indeed,
examples of PD found within our high-tech AAC dataset include
Kane et al.’s TalkAbout - a context-aware, adaptive AAC system
co-designed in collaboration with 5 adults with aphasia [90] and
de Faria Borges et al.’s PD4CAT customized communication device
developed in collaboration with a child with cerebral palsy [41].
5.3 Engaging with Communities and Greater
Representation in High-tech AAC Research
and Interventions
For the third research question and understanding who does high-
tech AAC research focus upon, we get very diferent results from the
Mack et al. research. In contrast, within our dataset Other (39%) and
Motor impairments (32.4%) feature prominently versus the Mack et
al. yet comparatively BVI takes up a much smaller minority of re-
search (-41.1%). Directly commenting on these results, pre-existing
high-tech AAC research has perhaps lacked specifcity regarding
its targeted community of focus as the majority of contributions
within our dataset is towards Other [112]. Investigating further, we
fnd that the Other community of focus frequently in papers is rep-
resented as diluted communities such as “AAC users" e.g., [56] or
“Children" e.g., [144]. This lack of specifcity has perhaps resulted in
high-tech AAC that lacks focus or curated design for specifc target
groups, disabilities, communities and their unique set of needs [112].
Therefore, a clear direction for research is that we implore future
high-tech AAC development to be directly contributing towards a
specifc community of focus with CCNs. Elsewhere, we fnd that
Other cognitive (13.2%), ASD (12.6%) and IDD (11.9%) could re-
ceive more high-tech AAC contributions and are under represented
versus Motor impairments (32.4%) – indeed Other cognitive [146],
Autism [37] and IDD [115] are prominent communities within so-
ciety. Lastly, Older adults have a very small contribution (0.7%) –
21 Bircanin et al., have even suggested the formation of AAC publics
to promote greater
discourse on high-tech AAC to ensure more empowering high-tech AAC solutions are
innovated [20].
ASSETS ’22, October 23–26, 2022, Athens, Greece
nonetheless high-tech AAC could feasibly contribute towards peo-
ple living with dementia and other CCNs that will become more
prominent from an ageing global population [116].
With regards to participant groups, people with disabilities fea-
ture most prominently (73.5%), followed by people without disabili-
ties (31.3%). Although our high-tech AAC dataset has less people
with disabilities (-11.2%) and more people without (+8.2%) versus
the Mack et al. – we believe that this is because our dataset oc-
curs over a broader period (1978-2021) meaning that this change
is perhaps symptomatic of evolving standards of practice for ac-
cessibility research in particular a focus on increased engagement
with participants with disabilities as a consequence of the social
model of disability [45]. Therefore, we echo Mack et al.’s encourage-
ment of more direct engagement with people with disabilities [112].
Furthermore, high-tech AAC research has shown a propensity to
engage with caregivers (18.1%), specialists (14.9%) and peers – in
the right circumstances this should be encouraged. Indeed, care-
givers, specialists (i.e., SLTs) and peers can often work as good
communication partners in user studies with people with CCNs –
supporting the high-tech AAC users’ expression when engaging
with the study [29] or for dyads [103]. Equally, if the study involves
children, parents or guardians they can also help support the child
with CCNs refection and potentially provide feedback on the un-
derlying efectiveness of the intervention [43]. Indeed, Delarosa et
al., rightly note that the successful integration of a high-tech AAC
system requires strong commitment from parents and other family
members [43].
Non-disabled participants are regularly used within our dataset
(31.3%), typically for either proxy usage (20.7%) or ability-based
comparisons (6%). Despite difculties designing and recruiting par-
ticipants with CCNs [160], we advocate not using just proxies as
Mack et al., note these high-tech AAC user studies run the risk
of reinforcing normalist beliefs [112]. Furthermore, proxy users
can often not accurately replicate a disabled user for testing the
usability of a high-tech AAC device – particularly for users with
motor-impairments [113]. Instead, proxies can be used jointly to
perform initial testing [28], triangulate fndings [58] and perhaps
learn of the diferences in social perceptions versus a disabled user
with high-tech AAC [13]. Although ill-frequent within our dataset,
ability-based comparisons are an imperfect heuristic for determin-
ing the performance of a high-tech AAC device – other qualitative
fndings concerning preference and satisfaction from disabled high-
tech AAC users are more useful for determining if an intervention
is successful [11, 87]. Indeed, even if the high-tech AAC success-
fully ofers pathways for communication that is not defnitively a
pre-requisite for long-term adoption [20].
6 LIMITATIONS
As in all SRs, we had to limit its scope. Despite a robust PRISMA
guided methodology, with snowballing, our approach would have
not covered all AAC papers. Indeed we only used two scientifc
databases: the ACM DL and Scopus for constructing our SR dataset
- there are other sources (e.g., IEEE, Elsevier, Thompson Reuters
etc.) which will likely reveal more papers on the topic of high-
tech AAC. Additionally, for manual coding of such a large dataset
there are consequently areas of potential inconsistency, subjectivity
ASSETS ’22, October 23–26, 2022, Athens, Greece
and errors. Consequently, some researchers have argued in favour
of using quantitative and automated analyses to mitigate against
this concern [49, 69, 102, 104]. Our comparison of fndings with
the Mack et al. [112] dataset has limitations – we are a diferent
research team and therefore have introduced subjective biases into
our review and our dataset covers a wider breadth of literature.
Indeed, future systematic reviews of AAC could look to be more
focused – for instance, explicitly focus on commercial devices or
AAC developed within a smaller period of inclusion. Much like other
research [31, 112], we also encountered highly varied language use
across papers, which impacted our classifcation of codes.
7 CONCLUSION
AAC is an essential support for many, yet is under-adopted and
frequently abandoned. To provide deeper insight and directions
for future research, we provide a taxonomic overview of the cur-
rent state of high-tech AAC devices, the methodologies, contribu-
tions and communities of focus – compiling a paper count and
inventory of the key devices used. Our results suggest three future
research directions. Firstly, more research is needed to explore high-
tech AAC inputs/outputs, the role of high-tech AAC in supporting
non-verbal communication, high-tech AAC communication with
groups, fellow AAC users and virtual scenarios. Secondly, in terms
of specifcally methods, future high-tech AAC research could use
PD methods and RCTs. Thirdly and lastly, with respect to commu-
nities of focus limited high-tech AAC research focuses on Other
Cognitive, Autism and IDD with low median participant counts
for studies incorporating people with disabilities (N=5). We hope
that our contribution will foster new and more optimally directed
AAC research, towards promulgating greater adoption and more
successful long-term intervention outcomes.
ACKNOWLEDGMENTS
We would like to thank Dr Rita Borgo for her support, comments
and advice throughout the establishment and writing of the paper.
Many thanks to Dr Claudia Daudén Roquet for their helpful critique
of an earlier draft of this paper. This work was supported in part
by a UKRI EPSRC Studentship.
REFERENCES
[1] Kim Adams and Al Cook. 2016. Using robots in “hands-on” academic activi-
ties: a case study examining speech-generating device use and required skills.
Disability and Rehabilitation: Assistive Technology 11, 5 (2016), 433–443.
[2] Kim D Adams and Albert M Cook. 2013. Programming and controlling robots
using scanning on a speech generating communication device: A case study.
Technology and Disability 25, 4 (2013), 275–286.
[3] Abdullah Al Mahmud, Rikkert Gerits, and Jean-Bernard Martens. 2010. XTag:
designing an experience capturing and sharing tool for persons with aphasia.
In Proceedings of the 6th Nordic Conference on Human-Computer Interaction:
Extending Boundaries. 325–334.
[4] In North America, Pure Ofces Plato Close, Tachbrook Park, and Leamington
Spa. 2014. DynaVox T10/T15 User’s Guide. (2014).
[5] Kate Anderson, Susan Balandin, and Sally Clendon. 2011. “He cares about me
and I care about him.” Children’s experiences of friendship with peers who use
AAC. Augmentative and Alternative Communication 27, 2 (2011), 77–90.
[6] Rúbia EO Schultz Ascari, Roberto Pereira, and Luciano Silva. 2020. Computer
vision-based methodology to improve interaction for people with motor and
speech impairment. ACM Transactions on Accessible Computing (TACCESS) 13,
4 (2020), 1–33.
[7] Rúbia EO Schultz Ascari, Luciano Silva, and Roberto Pereira. 2019. Personalized
interactive gesture recognition assistive technology. In Proceedings of the 18th
Brazilian Symposium on Human Factors in Computing Systems. 1–12.
Curtis et al.
[8] S Aswin, Ayush Ranjan, and KV Prashanth. 2019. Smart Wearable Speaking Aid
for Aphonic Personnel. In International Conference On Computational Vision
and Bio Inspired Computing. Springer, 179–186.
[9] Jon Baio. 2014. Prevalence of autism spectrum disorder among children aged
8 years-autism and developmental disabilities monitoring network, 11 sites,
United States, 2010. (2014).
[10] Dean C Barnlund. 2017. A transactional model of communication. In Commu-
nication theory. Routledge, 47–57.
[11] Susan Baxter, Pam Enderby, Philippa Evans, and Simon Judge. 2012. Barriers
and facilitators to the use of high-technology augmentative and alternative com-
munication devices: a systematic review and qualitative synthesis. International
Journal of Language & Communication Disorders 47, 2 (2012), 115–129.
[12] Susan Baxter, Pam Enderby, Philippa Evans, and Simon Judge. 2012. Interven-
tions using high-technology communication devices: a state of the art review.
Folia Phoniatrica et Logopaedica 64, 3 (2012), 137–144.
[13] Ann Beck, Stacey Bock, James Thompson, and Kullaya Kosuwan. 2002. Infu-
ence of communicative competence and augmentative and alternative com-
munication technique on children’s attitudes toward a peer who uses AAC.
Augmentative and Alternative Communication 18, 4 (2002), 217–227.
[14] Hrvoje Belani. 2012. Towards a usability requirements taxonomy for mobile
AAC services. In 2012 First International Workshop on Usability and Accessibility
Focused Requirements Engineering (UsARE). IEEE, 36–39.
[15] Cynthia L Bennett, Erin Brady, and Stacy M Branham. 2018. Interdependence
as a frame for assistive technology research and design. In Proceedings of the
20th International ACM SIGACCESS Conference on Computers and Accessibility.
161–173.
[16] Cynthia L Bennett, Burren Peil, and Daniela K Rosner. 2019. Biographical
prototypes: Reimagining recognition and disability in design. In Proceedings of
the 2019 on Designing Interactive Systems Conference. 35–47.
[17] David R Beukelman, Susan Fager, Laura Ball, and Aimee Dietz. 2007. AAC
for adults with acquired neurological conditions: A review. Augmentative and
alternative communication 23, 3 (2007), 230–242.
[18] David R Beukelman and Pat Mirenda. 2013. Augmentative & alternative com-
munication: Supporting children and adults with complex communication needs.
Paul H. Brookes Pub.
[19] Elizabeth E Biggs, Erik W Carter, and Carly B Gilson. 2018. Systematic review
of interventions involving aided AAC modeling for children with complex
communication needs. American Journal on Intellectual and Developmental
Disabilities 123, 5 (2018), 443–473.
[20] Filip Bircanin, Bernd Ploderer, Laurianne Sitbon, Andrew A Bayor, and Margot
Brereton. 2019. Challenges and opportunities in using augmentative and alterna-
tive communication (AAC) technologies: Design considerations for adults with
severe disabilities. In Proceedings of the 31st Australian Conference on Human-
Computer-Interaction. 184–196.
[21] Alexandre Luís Cardoso Bissoli, Yves Luduvico Coelho, and Teodiano Freire
Bastos-Filho. 2016. A system for multimodal assistive domotics and augmenta-
tive and alternative communication. In Proceedings of the 9th ACM International
Conference on PErvasive Technologies Related to Assistive Environments. 1–8.
[22] Rolf Black, Per Ola Kristensson, Jianguo Zhang, Annalu Waller, Sophia Bano,
Zulqarnain Rashid, and Christopher Norrie. 2016. ACE-LP: Augmenting com-
munication using environmental data to drive language prediction. In Commu-
nication Matters-CM2016 National Conference.
[23] Rolf Black, Annalu Waller, Ross Turner, and Ehud Reiter. 2012. Supporting
personal narrative for children with complex communication needs. ACM
transactions on computer-human interaction (TOCHI) 19, 2 (2012), 1–35.
[24] Sarah W Blackstone, Michael B Williams, and Mick Joyce. 2002. Future AAC
technology needs: consumer perspectives. Assistive Technology 14, 1 (2002),
3–16.
[25] Jamie B Boster and John W McCarthy. 2018. Designing augmentative and
alternative communication applications: The results of focus groups with speech-
language pathologists and parents of children with autism spectrum disorder.
Disability and Rehabilitation: Assistive Technology 13, 4 (2018), 353–365.
[26] Aikaterini Bourazeri and Simone Stumpf. 2018. Co-Designing Smart Home
Technology with People with Dementia or Parkinson’s Disease. In Proceedings
of the 10th Nordic Conference on Human-Computer Interaction (Oslo, Norway)
(NordiCHI ’18). Association for Computing Machinery, New York, NY, USA,
609–621. https://doi.org/10.1145/3240167.3240197
[27] LouAnne E Boyd, Alejandro Rangel, Helen Tomimbang, Andrea Conejo-Toledo,
Kanika Patel, Monica Tentori, and Gillian R Hayes. 2016. SayWAT: Augmenting
face-to-face conversations for adults with autism. In Proceedings of the 2016 CHI
Conference on Human Factors in Computing Systems. 4872–4883.
[28] Jordan L Boyd-Graber, Sonya S Nikolova, Karyn A Mofatt, Kenrick C Kin,
Joshua Y Lee, Lester W Mackey, Marilyn M Tremaine, and Maria M Klawe. 2006.
Participatory design with proxies: developing a desktop-PDA system to support
people with aphasia. In Proceedings of the SIGCHI conference on Human Factors
in computing systems. 151–160.
[29] Alisa Brownlee and Lisa M Bruening. 2012. Methods of communication at end
of life for the person with amyotrophic lateral sclerosis. Topics in Language
State of the Art in AAC
Disorders 32, 2 (2012), 168–185.
[30] Frederik Brudy, Christian Holz, Roman Rädle, Chi-Jui Wu, Steven Houben,
Clemens Nylandsted Klokmose, and Nicolai Marquardt. 2019. Cross-device
taxonomy: Survey, opportunities and challenges of interactions spanning across
multiple devices. In Proceedings of the 2019 chi conference on human factors in
computing systems. 1–28.
[31] Emeline Brulé, Brianna J Tomlinson, Oussama Metatla, Christophe Joufrais,
and Marcos Serrano. 2020. Review of Quantitative Empirical Evaluations of
Technology for People with Visual Impairments. In Proceedings of the 2020 CHI
Conference on Human Factors in Computing Systems. 1–14.
[32] Joan Bruno and David Trembath. 2006. Use of aided language stimulation
to improve syntactic performance during a weeklong intervention program.
Augmentative and Alternative Communication 22, 4 (2006), 300–313.
[33] Jessica Caron, Janice Light, Beth E Davidof, and Kathryn DR Drager. 2017.
Comparison of the efects of mobile technology AAC apps on programming
visual scene displays. Augmentative and Alternative Communication 33, 4 (2017),
239–248.
[34] Jessica Caron, Janice Light, and Kathryn Drager. 2016. Operational demands of
AAC mobile technology applications on programming vocabulary and engage-
ment during professional and child interactions. Augmentative and Alternative
Communication 32, 1 (2016), 12–24.
[35] Richard Cave and Steven Bloch. 2021. Voice banking for people living with motor
neurone disease: Views and expectations. International Journal of Language &
Communication Disorders 56, 1 (2021), 116–129.
[36] Rosanna Yuen-Yan Chan, Eri Sato-Shimokawara, Xue Bai, Motohashi Yukiharu,
Sze-Wing Kuo, and Anson Chung. 2019. A context-aware augmentative and al-
ternative communication system for school children with intellectual disabilities.
IEEE Systems Journal 14, 1 (2019), 208–219.
[37] Flavia Chiarotti and Aldina Venerosi. 2020. Epidemiology of autism spectrum
disorders: a review of worldwide prevalence estimates since 2014. Brain sciences
10, 5 (2020), 274.
[38] Simone Ciccia, Alberto Scionti, Giacomo Vitali, and Olivier Terzo. 2020.
QuadCOINS-Network: A Deep Learning Approach to Sound Source Localization.
In Conference on Complex, Intelligent, and Software Intensive Systems. Springer,
130–141.
[39] Lauren Cooper, Susan Balandin, and David Trembath. 2009. The loneliness
experiences of young adults with cerebral palsy who use alternative and aug-
mentative communication. Augmentative and alternative communication 25, 3
(2009), 154–164.
[40] Elke Daemen, Pavan Dadlani, Jia Du, Ying Li, Pinar Erik-Paker, Jean-Bernard
Martens, and Boris de Ruyter. 2007. Designing a free style, indirect, and inter-
active storytelling application for people with aphasia. In IFIP Conference on
Human-Computer Interaction. Springer, 221–234.
[41] Luciana Correia Lima de Faria Borges, Lucia Vilela Leite Filgueiras, Cristiano
Maciel, and Vinicius Carvalho Pereira. 2012. Customizing a communication
device for a child with cerebral palsy using participatory design practices:
contributions towards the PD4CAT method. In Proceedings of the 11th Brazilian
Symposium on Human Factors in Computing Systems. 57–66.
[42] Denise C DeCoste. 1997. The handbook of augmentative and alternative commu-
nication. Cengage Learning.
[43] Elizabeth Delarosa, Stephanie Horner, Casey Eisenberg, Laura Ball, Anne Marie
Renzoni, and Stephen E Ryan. 2012. Family impact of assistive technology scale:
Development of a measurement scale for parents of children with complex
communication needs. Augmentative and Alternative Communication 28, 3
(2012), 171–180.
[44] L Scott Doss, Peggy Ann Locke, Susan Johnston, Joe Reichle, Jef Sigafoos, Paul
Charpentier, and Dulce Foster. 1991. Initial comparison of the efciency of a
variety of AAC systems for ordering meals in fast food restaurants. Augmentative
and Alternative Communication 7, 4 (1991), 256–265.
[45] Elizabeth Ellcessor. 2010. Bridging disability divides: A critical history of web
content accessibility through 2001. Information, Communication & Society 13, 3
(2010), 289–308.
[46] Yasmin Elsahar, Sijung Hu, Kaddour Bouazza-Marouf, David Kerr, and Annysa
Mansor. 2019. Augmentative and alternative communication (AAC) advances:
A review of confgurations for individuals with a speech disability. Sensors 19, 8
(2019), 1911.
[47] P Enderby, S Judge, S Creer, and A John. 2013. Examining the need for, and
provison of, AAC in the United Kingdom. (2013).
[48] Annette Estes, Vanessa Rivera, Matthew Bryan, Philip Cali, and Geraldine Daw-
son. 2011. Discrepancies between academic achievement and intellectual ability
in higher-functioning school-aged children with autism spectrum disorder.
Journal of autism and developmental disorders 41, 8 (2011), 1044–1052.
[49] Hans J Eysenck. 1994. Systematic reviews: Meta-analysis and its problems. Bmj
309, 6957 (1994), 789–792.
[50] Torsten Felzer and Rainer Nordmann. 2008. Using intentional muscle contrac-
tions as input signals for various hands-free control applications. In Proceedings
of the 2nd International Convention on Rehabilitation Engineering & Assistive
Technology. 87–91.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[51] Alexander Fiannaca, Ann Paradiso, Mira Shah, and Meredith Ringel Morris. 2017.
AACrobat: Using mobile devices to lower communication barriers and provide
autonomy with gaze-based AAC. In Proceedings of the 2017 ACM Conference on
Computer Supported Cooperative Work and Social Computing. 683–695.
[52] Alexander J Fiannaca, Ann Paradiso, Jon Campbell, and Meredith Ringel Morris.
2018. Voicesetting: voice authoring UIs for improved expressivity in augmenta-
tive communication. In Proceedings of the 2018 CHI Conference on Human Factors
in Computing Systems. 1–12.
[53] Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and
the intraclass correlation coefcient as measures of reliability. Educational and
psychological measurement 33, 3 (1973), 613–619.
[54] Amanda Fleury, Gloria Wu, and Tom Chau. 2019. A wearable fabric-based
speech-generating device: system design and case demonstration. Disability
and Rehabilitation: Assistive Technology 14, 5 (2019), 434–444.
[55] Margaret Flores, Kate Musgrove, Scott Renner, Vanessa Hinton, Shaunita
Strozier, Susan Franklin, and Doris Hil. 2012. A comparison of communication
using the Apple iPad and a picture-based system. Augmentative and Alternative
Communication 28, 2 (2012), 74–84.
[56] Richard Foulds, Mathijs Soede, and Hans van Balkom. 1987. Statistical disam-
biguation of multi-character keys applied to reduce motor requirements for
augmentative and alternative communication. Augmentative and alternative
communication 3, 4 (1987), 192–195.
[57] Christopher Frauenberger, Julia Makhaeva, and Katta Spiel. 2017. Blending
methods: Developing participatory design sessions for autistic children. In
Proceedings of the 2017 conference on interaction design and children. 39–49.
[58] Melanie Fried-Oken, Lynn Fox, Marie T Rau, Jill Tullman, Glory Baker, Mary
Hindal, Nancy Wile, and Jau-Shin Lou. 2006. Purposes of AAC device use for
persons with ALS as reported by caregivers. Augmentative and Alternative
Communication 22, 3 (2006), 209–221.
[59] Melanie Fried-Oken, Aimee Mooney, Betts Peters, and Barry Oken. 2015. A
clinical screening protocol for the RSVP keyboard brain–computer interface.
Disability and Rehabilitation: Assistive Technology 10, 1 (2015), 11–18.
[60] Julia Galliers, Stephanie Wilson, Jane Marshall, Richard Talbot, Niamh Devane,
Tracey Booth, Celia Woolf, and Helen Greenwood. 2017. Experiencing EVA
park, a multi-user virtual world for people with aphasia. ACM Transactions on
Accessible Computing (TACCESS) 10, 4 (2017), 1–24.
[61] Nestor Garay-Vitoria and Julio Abascal. 2004. A comparison of prediction
techniques to enhance the communication rate. In ERCIM Workshop on User
Interfaces for All. Springer, 400–417.
[62] Luís Filipe Garcia, Luís Caldas De Oliveira, and David Martins De Matos. 2015.
Measuring the performance of a location-aware text prediction system. ACM
Transactions on Accessible Computing (TACCESS) 7, 1 (2015), 1–29.
[63] Paola García, Eduardo Lleida, Diego Castán, José Manuel Marcos, and David
Romero. 2015. Context-aware communicator for all. In International Conference
on Universal Access in Human-Computer Interaction. Springer, 426–437.
[64] Miguel Gea-Megías, Nuria Medina-Medina, María Luisa Rodríguez-Almendros,
and María José Rodríguez-Fórtiz. 2004. Sc@ ut: Platform for communication
in ubiquitous and adaptive environments applied for children with autism. In
ERCIM Workshop on User Interfaces for All. Springer, 50–67.
[65] Amy Goldman. 2008. Funding AAC. Perspectives on Augmentative and Alterna-
tive Communication 17, 1 (2008), 33–35.
[66] Isabel Gómez, Pablo Anaya, Rafael Cabrera, Alberto Molina, Octavio Rivera,
and Manuel Merino. 2010. Augmented and alternative communication system
based on dasher application and an accelerometer. In International Conference
on Computers for Handicapped Persons. Springer, 98–103.
[67] Carol Goossens’. 1989. Aided communication intervention before assessment:
A case study of a child with cerebral palsy. Augmentative and Alternative
Communication 5, 1 (1989), 14–26.
[68] Daniel Gorenfo and Carole Gorenfo. 1997. Efects of synthetic speech, gender,
and perceived similarity on attitudes toward the augmented communicator.
Augmentative and Alternative Communication 13, 2 (1997), 87–91.
[69] Sebastian Götz. 2018. Supporting systematic literature reviews in computer sci-
ence: the systematic literature review toolkit. In Proceedings of the 21st ACM/IEEE
International Conference on Model Driven Engineering Languages and Systems:
Companion Proceedings. 22–26.
[70] Zahid Halim and Ghulam Abbas. 2015. A kinect-based sign language hand
gesture recognition system for hearing-and speech-impaired: a pilot study of
Pakistani sign language. Assistive Technology 27, 1 (2015), 34–43.
[71] Foad Hamidi, Melanie Baljko, Toni Kunic, and Ray Feraday. 2014. Do-It-Yourself
(DIY) assistive technology: a communication board case study. In International
conference on computers for handicapped persons. Springer, 287–294.
[72] Mark S Hawley, Pam Enderby, Phil Green, Stuart Cunningham, and Rebecca
Palmer. 2006. Development of a voice-input voice-output communication aid
(VIVOCA) for people with severe dysarthria. In International Conference on
Computers for Handicapped Persons. Springer, 882–885.
[73] D Jefery Higginbotham, Gregory W Lesher, Bryan J Moulton, and Brian Roark.
2012. The application of natural language processing to augmentative and
alternative communication. Assistive Technology 24, 1 (2012), 14–24.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[74] D Jefery Higginbotham, Howard Shane, Susanne Russell, and Kevin Caves.
2007. Access to AAC: Present, past, and future. Augmentative and alternative
communication 23, 3 (2007), 243–257.
[75] Robert A Hinde and Robert Aubrey Hinde. 1972. Non-verbal communication.
Cambridge University Press.
[76] Anthony Hogan, Megan Shipley, Lyndall Strazdins, Alison Purcell, and Elise
Baker. 2011. Communication and behavioural disorders among children with
hearing loss increases risk of mental health disorders. Australian and New
Zealand journal of public health 35, 4 (2011), 377–383.
[77] Christine Holyfeld, Kathryn DR Drager, Jennifer MD Kremkow, and Janice Light.
2017. Systematic review of AAC intervention research for adolescents and adults
with autism spectrum disorder. Augmentative and alternative communication
33, 4 (2017), 201–212.
[78] Ming-Che Hsieh and Ching-Hsing Luo. 1999. Morse code typing training of
an adolescent with cerebral palsy using microcomputer technology: case study.
Augmentative and Alternative Communication 15, 4 (1999), 216–221.
[79] Li Huang, Szu-Han Kay Chen, Shutian Xu, Yongli Wang, Xing Jin, Ping Wan,
Jikang Sun, Jiming Tao, Sicong Zhang, Guohui Zhang, et al. 2021. Augmentative
and alternative communication intervention for in-patient individuals with
post-stroke aphasia: study protocol of a parallel-group, pragmatic randomized
controlled trial. Trials 22, 1 (2021), 1–9.
[80] Ms Seray Ibrahim, Asimina Vasalou, and Michael Clarke. 2017. Rethinking
technology design for and with children who have severe speech & physical
disabilities. (2017).
[81] Seray B Ibrahim, Asimina Vasalou, and Michael Clarke. 2018. Design opportu-
nities for AAC and children with severe speech and physical impairments. In
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
1–13.
[82] Rabia Jafri, Ameera Masoud Almasoud, Reema Mohammed Taj Alshammari,
Shahad Eid Mohammed Alosaimi, Raghad Talal Mohammed Alhamad, and
Amzan Abdullah Saleh Aldowighri. 2020. A Low-Cost Gaze-Based Arabic
Augmentative and Alternative Communication System for People with Severe
Speech and Motor Impairments. In International Conference on Human-Computer
Interaction. Springer, 279–290.
[83] Vinoth Jagaroo and Krista Wilkinson. 2008. Further considerations of visual
cognitive neuroscience in aided AAC: The potential role of motion perception
systems in maximizing design display. Augmentative and Alternative Communi-
cation 24, 1 (2008), 29–42.
[84] Roman Jakobson. 1972. Verbal communication. Scientifc American 227, 3 (1972),
72–81.
[85] Kyung Hea Jeon, Seok Jeong Yeon, Young Tae Kim, Seokwoo Song, and John
Kim. 2014. Robot-based augmentative and alternative communication for non-
verbal children with communication disorders. In Proceedings of the 2014 ACM
International Joint Conference on Pervasive and Ubiquitous Computing. 853–859.
[86] S Jirayucharoensak, A Hemakom, W Chonnaparamutt, and P Israsena. 2011.
Design and evaluation of a picture-based P300 AAC system. In Proceedings of the
5th International Conference on Rehabilitation Engineering & Assistive Technology.
1–4.
[87] Jeanne M Johnson, Ella Inglebret, Carla Jones, and Jayanti Ray. 2006. Perspectives
of speech language pathologists regarding success versus abandonment of AAC.
Augmentative and Alternative Communication 22, 2 (2006), 85–99.
[88] Rachel Kay Johnson, Monica Strauss Hough, Kristin Ann King, Paul Vos, and
Tara Jefs. 2008. Functional communication in individuals with chronic severe
aphasia using augmentative communication. Augmentative and Alternative
Communication 24, 4 (2008), 269–280.
[89] Xin-Xing Ju, Jie Yang, and Xiao-Xin Liu. 2021. A systematic review on voiceless
patients’ willingness to adopt high-technology augmentative and alternative
communication in intensive care units. Intensive and Critical Care Nursing 63
(2021), 102948.
[90] Shaun K Kane, Barbara Linam-Church, Kyle Althof, and Denise McCall. 2012.
What we talk about: designing a context-aware communication tool for people
with aphasia. In Proceedings of the 14th international ACM SIGACCESS conference
on Computers and accessibility. 49–56.
[91] Shaun K Kane and Meredith Ringel Morris. 2017. Let’s Talk about X: Combining
image recognition and eye gaze to support conversation for people with ALS.
In Proceedings of the 2017 Conference on Designing Interactive Systems. 129–134.
[92] Shaun K Kane, Meredith Ringel Morris, Ann Paradiso, and Jon Campbell. 2017.
"At times avuncular and cantankerous, with the refexes of a mongoose" Under-
standing Self-Expression through Augmentative and Alternative Communica-
tion Devices. In Proceedings of the 2017 ACM Conference on Computer Supported
Cooperative Work and Social Computing. 1166–1179.
[93] Jennifer Kent-Walsh and Cathy Binger. 2018. Methodological advances, op-
portunities, and challenges in AAC research. Augmentative and Alternative
Communication 34, 2 (2018), 93–103.
[94] Chutisant Kerdvibulvech and Chih-Chien Wang. 2016. A new 3D augmented
reality application for educational games to help children in communication
interactively. In International Conference on Computational Science and Its Appli-
cations. Springer, 465–473.
Curtis et al.
[95] Aggelos Kiayias. 2011. On the efects of pirate evolution on the design of
digital content distribution systems. In International Conference on Coding and
Cryptology. Springer, 223–237.
[96] Wooseok Kim and Sangsu Lee. 2021. “I Can’t Talk Now”: Speaking with Voice
Output Communication Aid Using Text-to-Speech Synthesis During Multiparty
Video Conference. In Extended Abstracts of the 2021 CHI Conference on Human
Factors in Computing Systems. 1–6.
[97] Susan Koch Fager, Melanie Fried-Oken, Tom Jakobs, and David R Beukelman.
2019. New and emerging access technologies for adults with complex communi-
cation needs and severe motor impairments: State of the science. Augmentative
and Alternative Communication 35, 1 (2019), 13–25.
[98] Arlene W Kraat. 1987. Communication interaction between aided and natural
speakers: A state of the art report. (1987).
[99] Per Ola Kristensson, James Lilley, Rolf Black, and Annalu Waller. 2020. A de-
sign engineering approach for quantitatively exploring context-aware sentence
retrieval for nonspeaking individuals with motor disabilities. In Proceedings of
the 2020 CHI Conference on Human Factors in Computing Systems. 1–11.
[100] Saili S Kulkarni and Jessica Parmar. 2017. Culturally and linguistically diverse
student and family perspectives of AAC. Augmentative and Alternative Commu-
nication 33, 3 (2017), 170–180.
[101] Joanne Lasker and David R Beukelmanoe. 1999. Peers’ perceptions of storytelling
by an adult with aphasia. Aphasiology 13, 9-11 (1999), 857–869.
[102] Joseph Lau, John PA Ioannidis, and Christopher H Schmid. 1997. Quantitative
synthesis in systematic reviews. Annals of internal medicine 127, 9 (1997), 820–
826.
[103] Emily Laubscher, Janice Light, and David McNaughton. 2019. Efect of an
application with video visual scene displays on communication during play:
Pilot study of a child with autism spectrum disorder and a peer. Augmentative
and Alternative Communication 35, 4 (2019), 299–308.
[104] Hang Li, Harrisen Scells, and Guido Zuccon. 2020. Systematic review automation
tools for end-to-end query formulation. In Proceedings of the 43rd International
ACM SIGIR Conference on Research and Development in Information Retrieval.
2141–2144.
[105] Janice Light. 1997. “Communication is the essence of human life”: Refections
on communicative competence. Augmentative and Alternative Communication
13, 2 (1997), 61–70.
[106] Janice Light and Kathryn Drager. 2007. AAC technologies for young children
with complex communication needs: State of the science and future research
directions. Augmentative and alternative communication 23, 3 (2007), 204–216.
[107] Janice Light and David McNaughton. 2012. The changing face of augmentative
and alternative communication: Past, present, and future challenges. , 197–
204 pages.
[108] Janice C Light, Kathryn DR Drager, and Jessica G Nemser. 2004. Enhancing the
appeal of AAC technologies for young children: Lessons from the toy manufac-
turers. Augmentative and Alternative Communication 20, 3 (2004), 137–149.
[109] Stephen Lindsay, Katie Brittain, Daniel Jackson, Cassim Ladha, Karim Ladha, and
Patrick Olivier. 2012. Empathy, participatory design and people with dementia.
In Proceedings of the SIGCHI conference on Human factors in computing systems.
521–530.
[110] Stephen Lindsay, Daniel Jackson, Guy Schofeld, and Patrick Olivier. 2012. En-
gaging older people using participatory design. In Proceedings of the SIGCHI
conference on human factors in computing systems. 1199–1208.
[111] Kristy Logan, Teresa Iacono, and David Trembath. 2017. A systematic review of
research into aided AAC to increase social-communication functions in children
with autism spectrum disorder. Augmentative and Alternative Communication
33, 1 (2017), 51–64.
[112] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich,
and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? A
Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to
2019. In Proceedings of the 2021 CHI Conference on Human Factors in Computing
Systems. 1–18.
[113] Jennifer Mankof, Holly Fait, and Ray Juang. 2005. Evaluating accessibility by
simulating the experiences of users with vision or motor impairments. IBM
Systems Journal 44, 3 (2005), 505–517.
[114] Jennifer Mankof, Gillian R Hayes, and Devva Kasnitz. 2010. Disability studies as
a source of critical inquiry for the feld of assistive technology. In Proceedings of
the 12th international ACM SIGACCESS conference on Computers and accessibility.
3–10.
[115] Pallab K Maulik, Maya N Mascarenhas, Colin D Mathers, Tarun Dua, and
Shekhar Saxena. 2011. Prevalence of intellectual disability: a meta-analysis
of population-based studies. Research in developmental disabilities 32, 2 (2011),
419–436.
[116] Auriel A May, Shakila Dada, and Janice Murray. 2019. Review of AAC in-
terventions in persons with dementia. International Journal of Language &
Communication Disorders 54, 6 (2019), 857–874.
[117] M Shannon McCord and Gloria Soto. 2004. Perceptions of AAC: An ethnographic
investigation of Mexican-American families. Augmentative and alternative
communication 20, 4 (2004), 209–227.
State of the Art in AAC
[118] Miechelle Mckelvey, David L Evans, Norimune Kawai, and David Beukelman.
2012. Communication styles of persons with ALS as recounted by surviving
partners. Augmentative and Alternative Communication 28, 4 (2012), 232–242.
[119] Deirdre McLaughlin, Betts Peters, Kendra McInturf, Brandon Eddy, Michelle
Kinsella, Aimee Mooney, Trinity Deibert, Kerry Montgomery, and Melanie
Fried-Oken. 2021. Decision-making for access to AAC technologies in late stage
ALS. Augmentative and Alternative Communication: Challenges and Solutions
(2021), 169–199.
[120] Sharynne McLeod. 2018. Communication rights: Fundamental human rights
for all. International Journal of Speech-Language Pathology 20, 1 (2018), 3–11.
[121] David McNaughton, David Beukelman, and Patricia Dowden. 1999. Tools
to support international and intercommunity collaboration in AAC research.
Augmentative and Alternative Communication 15, 4 (1999), 280–288.
[122] David McNaughton, Diane Bryen, Sarah Blackstone, Michael Williams, and
Pamela Kennedy. 2012. Young adults with complex communication needs: Re-
search and development in AAC for a “diverse” population. Assistive Technology
24, 1 (2012), 45–53.
[123] David Mcnaughton and Diane Nelson Bryen. 2007. AAC technologies to enhance
participation and access to meaningful societal roles for adolescents and adults
with developmental disabilities who require AAC. Augmentative and alternative
communication 23, 3 (2007), 217–229.
[124] David McNaughton and Janice Light. 2013. The iPad and mobile technology
revolution: Benefts and challenges for individuals who require augmentative
and alternative communication. , 107–116 pages.
[125] David McNaughton, Tracy Rackensperger, Elizabeth Benedek-Wood, Carole
Krezman, Michael B Williams, and Janice Light. 2008. “A child needs to be given
a chance to succeed”: Parents of individuals who use AAC describe the benefts
and challenges of learning AAC technologies. Augmentative and alternative
communication 24, 1 (2008), 43–55.
[126] Sally Millar, Janet Scott, et al. 1998. What is augmentative and alternative
communication? An introduction. Augmentative Communication in Practice 2
(1998).
[127] Pamela Mitchell and Carolyn Atkins. 1989. A comparison of the single word
intelligibility of two voice output communication aids. Augmentative and
Alternative Communication 5, 2 (1989), 84–88.
[128] Karyn Mofatt, Golnoosh Pourshahid, and Ronald M Baecker. 2017. Augmenta-
tive and alternative communication devices for aphasia: The emerging role of
“smart” mobile devices. Universal Access in the Information Society 16, 1 (2017),
115–128.
[129] Aimee Mooney, Steven Bedrick, Glory Noethe, Scott Spaulding, and Melanie
Fried-Oken. 2018. Mobile technology to support lexical retrieval during activity
retell in primary progressive aphasia. Aphasiology 32, 6 (2018), 666–692.
[130] A Moorcroft, N Scarinci, and C Meyer. 2019. A systematic review of the barriers
and facilitators to the provision and use of low-tech and unaided AAC systems
for people with complex communication needs and their families. Disability
and Rehabilitation: Assistive Technology 14, 7 (2019), 710–731.
[131] Kristi L Morin, Jennifer B Ganz, Emily V Gregori, Margaret J Foster, Stephanie L
Gerow, Derya Genç-Tosun, and Ee Rea Hong. 2018. A systematic quality review
of high-tech AAC interventions as an evidence-based practice. Augmentative
and Alternative Communication 34, 2 (2018), 104–117.
[132] Robert R Morris, Connor R Kirschbaum, and Rosalind W Picard. 2010. Broad-
ening accessibility through special interests: a new approach for software cus-
tomization. In Proceedings of the 12th international ACM SIGACCESS conference
on Computers and accessibility. 171–178.
[133] Joan Murphy, Ivana Marková, Eleanor Moodie, Janet Scott, and Sally Boa. 1995.
Augmentative and alternative communication systems used by people with
cerebral palsy in Scotland: Demographic survey. Augmentative and Alternative
Communication 11, 1 (1995), 26–36.
[134] Elizabeth Murray, Patricia McCabe, and Kirrie J Ballard. 2012. A comparison
of two treatments for childhood apraxia of speech: Methods and treatment
protocol for a parallel group randomised control trial. BMC pediatrics 12, 1
(2012), 1–9.
[135] NCBI. 2017. Augmentative and Alternative Communication and Voice Products
and Technologies. The Promise of Assistive Technology to Enhance Activity and
Work Participation; The National Academies Press: Washington, DC, USA (2017),
209–310.
[136] Timothy Neate, Aikaterini Bourazeri, Abi Roper, Simone Stumpf, and Stephanie
Wilson. 2019. Co-created personas: Engaging and empowering users with
diverse needs within the design process. In Proceedings of the 2019 CHI conference
on human factors in computing systems. 1–12.
[137] Timothy Neate, Vasiliki Kladouchou, Stephanie Wilson, and Shehzmani Shams.
2021. Just Not Together”: The Experience of Videoconferencing for People with
Aphasia during the Covid-19 Pandemic. In Just Not Together”: The Experience of
Videoconferencing for People with Aphasia during the Covid-19 Pandemic. ACM.
[138] Timothy Neate, Abi Roper, Stephanie Wilson, Jane Marshall, and Madeline
Cruice. 2020. CreaTable content and tangible interaction in Aphasia. In Pro-
ceedings of the 2020 CHI Conference on Human Factors in Computing Systems.
1–14.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[139] Zeyun Niu, Wenbing Yao, Qiang Ni, and Yonghua Song. 2007. Dereq: a qos
routing algorithm for multimedia communications in vehicular ad hoc networks.
In Proceedings of the 2007 international conference on Wireless communications
and mobile computing. 393–398.
[140] Christopher S Norrie, Annalu Waller, and Elizabeth FS Hannah. 2021. Establish-
ing context: AAC device adoption and support in a special-education setting.
ACM Transactions on Computer-Human Interaction (TOCHI) 28, 2 (2021), 1–30.
[141] Mmachi G Obiorah, Anne Marie Piper, and Michael Horn. 2017. Independent
Word Discovery for People with Aphasia. In Proceedings of the 19th International
ACM SIGACCESS Conference on Computers and Accessibility. 325–326.
[142] University of Wisconsin Hospital and Clinics. 2010. AAC Glossary of Terms.
[143] Bernard O’Keefe, Lina Brown, and Reinhard Schuller. 1998. Identifcation and
rankings of communication aid features by fve groups. Augmentative and
Alternative Communication 14, 1 (1998), 37–50.
[144] Judith Oxley and Janet Norris. 2000. Children’s use of memory strategies:
Relevance to voice output communication aid use. Augmentative and Alternative
Communication 16, 2 (2000), 79–94.
[145] Amanda M O’Brien, Ralf W Schlosser, Howard Shane, Oliver Wendt, Christina
Yu, Anna A Allen, Jacqueline Cullen, Andrea Benz, and Lindsay O’Neill. 2020.
Providing visual directives via a smart watch to a student with Autism Spectrum
Disorder: an intervention note. Augmentative and Alternative Communication
36, 4 (2020), 249–257.
[146] Ricardo Pais, Luís Ruano, Ofélia P Carvalho, and Henrique Barros. 2020. Global
cognitive impairment prevalence and incidence in community dwelling older
adults—a systematic review. Geriatrics 5, 4 (2020), 84.
[147] Joon Sung Park, Danielle Bragg, Ece Kamar, and Meredith Ringel Morris. 2021.
Designing an online infrastructure for collecting AI data from people with
disabilities. In Proceedings of the 2021 ACM Conference on Fairness, Accountability,
and Transparency. 52–63.
[148] Shailaja Arjun Patil. 2009. Brain gate as an assistive and solution providing
technology for disabled people. In 13th International Conference on Biomedical
Engineering. Springer, 1232–1235.
[149] Kevin M Pitt and Jonathan S Brumberg. 2021. Evaluating the perspectives
of those with severe physical impairments while learning BCI control of a
commercial augmentative and alternative communication paradigm. Assistive
Technology (2021), 1–9.
[150] Tracy Rackensperger, Carole Krezman, David Mcnaughton, Michael B Williams,
and Karen D’silva. 2005. “When I frst got it, I wanted to throw it of a clif”: The
challenges and benefts of learning AAC technologies as described by adults who
use AAC. Augmentative and alternative communication 21, 3 (2005), 165–186.
[151] Joseph Reddington and Nava Tintarev. 2011. Automatically generating stories
from sensor data. In Proceedings of the 16th international conference on Intelligent
user interfaces. 407–410.
[152] Ehud Reiter, Ross Turner, Norman Alm, Rolf Black, Martin Dempster, and
Annalu Waller. 2009. Using NLG to help language-impaired users tell stories
and participate in social dialogues. In Proceedings of the 12th European Workshop
on Natural Language Generation (ENLG 2009). 1–8.
[153] Melissa L Rethlefsen, Shona Kirtley, Siw Wafenschmidt, Ana Patricia Ayala,
David Moher, Matthew J Page, and Jonathan B Kofel. 2021. PRISMA-S: an ex-
tension to the PRISMA statement for reporting literature searches in systematic
reviews. Systematic reviews 10, 1 (2021), 1–19.
[154] Laura Roche, Jef Sigafoos, Giulio E Lancioni, Mark F O’Reilly, and Vanessa A
Green. 2015. Microswitch technology for enabling self-determined respond-
ing in children with profound and multiple disabilities: A systematic review.
Augmentative and Alternative Communication 31, 3 (2015), 246–258.
[155] MaryAnn Romski, Rose A Sevcik, Andrea Barton-Hulsey, and Ani S Whitmore.
2015. Early intervention and AAC: What a diference 30 years makes. Augmen-
tative and Alternative Communication 31, 3 (2015), 181–202.
[156] Robert J Ruben. 2000. Redefning the survival of the fttest: communication
disorders in the 21st century. The Laryngoscope 110, 2 (2000), 241–241.
[157] Anna-Liisa Salminen, Helen Petrie, and Susan Ryan. 2004. Impact of computer
augmented communication on the daily lives of speech-impaired children. Part
I: Daily communication and activities. Technology and Disability 16, 3 (2004),
157–167.
[158] Igor Schadle. 2004. Sibyl: AAC system using NLP techniques. In International
Conference on Computers for Handicapped Persons. Springer, 1009–1015.
[159] Jennifer M Seale, Ann M Bisantz, and Jef Higginbotham. 2020. Interaction
symmetry: Assessing augmented speaker and oral speaker performances across
four tasks. Augmentative and Alternative Communication 36, 2 (2020), 82–94.
[160] Andrew Sears and Vicki Hanson. 2011. Representing users in accessibility
research. In Proceedings of the SIGCHI conference on Human factors in computing
systems. 2235–2238.
[161] Claude Elwood Shannon. 2001. A mathematical theory of communication. ACM
SIGMOBILE mobile computing and communications review 5, 1 (2001), 3–55.
[162] Andy P Siddaway, Alex M Wood, and Larry V Hedges. 2019. How to do a
systematic review: a best practice guide for conducting and reporting narrative
reviews, meta-analyses, and meta-syntheses. Annual review of psychology 70
(2019), 747–770.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[163] Jef Sigafoos, Robert Didden, and MARK O’REILLY. 2003. Efects of speech output
on maintenance of requesting and frequency of vocalizations in three children
with developmental disabilities. Augmentative and Alternative Communication
19, 1 (2003), 37–47.
[164] Rodrigo Silva and Fran Neiva. 2016. Systematic Literature Review in Computer
Science - A Practical Guide. (11 2016). https://doi.org/10.13140/RG.2.2.35453.
87524
[165] Jessica Simacek, Brittany Pennington, Joe Reichle, and Quannah Parker-
McGowan. 2018. Aided AAC for people with severe to profound and mul-
tiple disabilities: A systematic review of interventions and treatment intensity.
Advances in Neurodevelopmental Disorders 2, 1 (2018), 100–115.
[166] Kiley Sobel, Alexander Fiannaca, Jon Campbell, Harish Kulkarni, Ann Paradiso,
Ed Cutrell, and Meredith Ringel Morris. 2017. Exploring the Design Space of
AAC Awareness Displays. In Proceedings of the 2017 CHI Conference on Human
Factors in Computing Systems. 2890–2903.
[167] Katta Spiel, Laura Malinverni, Judith Good, and Christopher Frauenberger. 2017.
Participatory evaluation with autistic children. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems. 5755–5766.
[168] Roger J Stanclife, Sheryl Larson, Karen Auerbach, Joshua Engler, Sarah Taub,
and K Charlie Lakin. 2010. Individuals with intellectual disabilities and augmen-
tative and alternative communication: Analysis of survey data on uptake of aided
AAC, and loneliness experiences. Augmentative and alternative communication
26, 2 (2010), 87–96.
[169] Stephen Steward. 2009. Designing AAC interfaces for commercial brain-
computer interaction gaming hardware. In Proceedings of the 11th international
ACM SIGACCESS conference on Computers and accessibility. 265–266.
[170] Katharine Still, Ruth Anne Rehfeldt, Robert Whelan, Richard May, and Simon
Dymond. 2014. Facilitating requesting skills using high-tech augmentative
and alternative communication devices with individuals with autism spectrum
disorders: A systematic review. Research in Autism Spectrum Disorders 8, 9
(2014), 1184–1199.
[171] Arthur Theil, Lea Buchweitz, James Gay, Eva Lindell, Li Guo, Nils-Krister Pers-
son, and Oliver Korn. 2020. Tactile board: a multimodal augmentative and
alternative communication device for individuals with Deafblindness. In 19th
International Conference on Mobile and Ubiquitous Multimedia. 223–228.
[172] Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily Morri-
son, Ed Cutrell, Matthew Tobias Harris, and Katja Hofmann. 2021. Disability-frst
Dataset Creation: Lessons from Constructing a Dataset for Teachable Object
Recognition with Blind and Low Vision Data Collectors. In International ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS). ACM. https:
//www.microsoft.com/en-us/research/publication/disability-frst-datasets/
[173] John Todman. 2000. Rate and quality of conversations using a text-storage AAC
system: Single-case training study. Augmentative and Alternative Communica-
tion 16, 3 (2000), 164–179.
[174] John Todman, Norman Alm, Jef Higginbotham, and Portia File. 2008. Whole
utterance approaches in AAC. Augmentative and alternative communication 24,
3 (2008), 235–254.
[175] John Todman and Pat Dugard. 1999. Accessible randomization tests for single-
case and small-n experimental designs in AAC research. Augmentative and
alternative communication 15, 1 (1999), 69–82.
[176] John Todman, Leona Elder, and Norman Alm. 1995. Evaluation of the content of
computer-aided conversations. Augmentative and Alternative Communication
11, 4 (1995), 229–235.
[177] Bálint Tóth, Géza Németh, and Géza Kiss. 2004. Mobile devices converted into
a speaking communication aid. In International Conference on Computers for
Handicapped Persons. Springer, 1016–1023.
[178] Kathryn Tringale, Daniel Bacher, and Leigh Hochberg. 2012. Towards the
optimal design of an assistive communication interface with neural input. In
2012 38th Annual Northeast Bioengineering Conference (NEBEC). IEEE, 197–198.
[179] Stephanie Valencia, Michal Luria, Amy Pavel, Jefrey P Bigham, and Henny
Admoni. 2021. Co-designing Socially Assistive Sidekicks for Motion-based AAC.
In Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot
Interaction. 24–33.
[180] Stephanie Valencia, Mark Steidl, Michael Rivera, Cynthia Bennett, Jefrey
Bigham, and Henny Admoni. 2021. Aided Nonverbal Communication through
Physical Expressive Objects. In The 23rd International ACM SIGACCESS Confer-
ence on Computers and Accessibility. 1–11.
[181] Mieke van de Sandt-Koenderman. 2004. High-tech AAC and aphasia: Widening
horizons? Aphasiology 18, 3 (2004), 245–263.
[182] Larah van der Meer, Jef Sigafoos, Mark F O’Reilly, and Giulio E Lancioni. 2011.
Assessing preferences for AAC options in communication interventions for
individuals with developmental disabilities: A review of the literature. Research
in Developmental Disabilities 32, 5 (2011), 1422–1431.
[183] Gregg C Vanderheiden. 2003. A journey through early augmentative communi-
cation and computer access. Journal of rehabilitation research and development
39, 6; SUPP (2003), 39–53.
[184] Keith Vertanen and Per Ola Kristensson. 2011. The imagination of crowds:
conversational AAC language modeling using crowdsourcing and large data
Curtis et al.
sources. In Proceedings of the 2011 Conference on Empirical Methods in Natural
Language Processing. 700–711.
[185] Annalu Waller. 2019. Telling tales: unlocking the potential of AAC technologies.
International journal of language & communication disorders 54, 2 (2019), 159–
169.
[186] Annalu Waller, Rolf Black, David A O’Mara, Helen Pain, Graeme Ritchie, and Ruli
Manurung. 2009. Evaluating the standup pun generating software with children
with cerebral palsy. ACM Transactions on Accessible Computing (TACCESS) 1, 3
(2009), 1–27.
[187] Annalu Waller and Alan F Newell. 1997. Towards a narrative-based augmenta-
tive communication system. International Journal of Language & Communication
Disorders 32, S3 (1997), 289–306.
[188] Shunfang Wang, Zicheng Cao, Mingyuan Li, and Yaoting Yue. 2019. G-DipC: an
improved feature representation method for short sequences to predict the type
of cargo in cell-penetrating peptides. IEEE/ACM Transactions on Computational
Biology and Bioinformatics 17, 3 (2019), 739–747.
[189] Janet Webster, Julie Morris, Carli Connor, Rachel Horner, Ciara McCormac, and
Amy Potts. 2013. Text level reading comprehension in aphasia: What do we
know about therapy and what do we need to know? Aphasiology 27, 11 (2013),
1362–1380.
[190] Bruce H Westley and Malcolm S MacLean Jr. 1957. A conceptual model for
communications research. Journalism Quarterly 34, 1 (1957), 31–38.
[191] Mary Wickenden. 2011. Talking to teenagers: Using anthropological methods
to explore identity and the lifeworlds of young people who use AAC. Commu-
nication Disorders Quarterly 32, 3 (2011), 151–163.
[192] Krista M Wilkinson and Janice Light. 2014. Preliminary study of gaze toward
humans in photographs by individuals with autism, Down syndrome, or other
intellectual disabilities: Implications for design of visual scene displays. Aug-
mentative and Alternative Communication 30, 2 (2014), 130–146.
[193] Kristin Williams, Karyn Mofatt, Denise McCall, and Leah Findlater. 2015. De-
signing conversation cues on a head-worn display to support persons with
aphasia. In Proceedings of the 33rd Annual ACM Conference on Human Factors in
Computing Systems. 231–240.
[194] Stephanie Wilson, Abi Roper, Jane Marshall, Julia Galliers, Niamh Devane,
Tracey Booth, and Celia Woolf. 2015. Codesign for people with aphasia through
tangible design languages. CoDesign 11, 1 (2015), 21–34.
[195] Bruce Wisenburn and D Jefery Higginbotham. 2008. An AAC application using
speaking partner speech recognition to automatically produce contextually rele-
vant utterances: Objective results. Augmentative and Alternative Communication
24, 2 (2008), 100–109.
[196] Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies
and a replication in software engineering. In Proceedings of the 18th international
conference on evaluation and assessment in software engineering. 1–10.
[197] Xiaoyi Zhang, Harish Kulkarni, and Meredith Ringel Morris. 2017. Smartphone-
based gaze gesture communication for people with motor disabilities. In Pro-
ceedings of the 2017 CHI Conference on Human Factors in Computing Systems.
2878–2889.
 AAC with Automated Vocabulary from Photographs:
Insights from School and Speech-Language Therapy Setings
Mauricio Fontana de Vargas Jiamin Dai
mauricio.fontanadevargas@mail.mcgill.ca jiamin.dai@mail.mcgill.ca
School of Information Studies School of Information Studies
McGill University McGill University
Montréal, Canada Montréal, Canada
ABSTRACT
Traditional symbol-based AAC devices impose meta-linguistic and
memory demands on individuals with complex communication
needs and hinder conversation partners from stimulating symbolic
language in meaningful moments. This work presents a prototype
application that generates situation-specifc communication boards
formed by a combination of descriptive, narrative, and semantic
related words and phrases inferred automatically from photographs.
Through semi-structured interviews with AAC professionals, we
investigate how this prototype was used to support communication
and language learning in naturalistic school and therapy settings.
We fnd that the immediacy of vocabulary reduces conversation
partners’ workload, opens up opportunities for AAC stimulation,
and facilitates symbolic understanding and sentence construction.
We contribute a nuanced understanding of how vocabularies gen-
erated automatically from photographs can support individuals
with complex communication needs in using and learning symbolic
AAC, ofering insights into the design of automatic vocabulary gen-
eration methods and interfaces to better support various scenarios
of use and goals.
KEYWORDS
Augmentative and Alternative Communication, autism, just-in-
time, automatic
ACM Reference Format:
Mauricio Fontana de Vargas, Jiamin Dai, and Karyn Mofatt. 2022. AAC with
Automated Vocabulary from Photographs: Insights from School and Speech-
Language Therapy Settings. In The 24th International ACM SIGACCESS
Conference on Computers and Accessibility (ASSETS ’22), October 23–26, 2022,
Athens, Greece. ACM, New York, NY, USA, 18 pages. https://doi.org/10.1145/
3517428.3544805
1 INTRODUCTION
Symbol-based Augmentative and Alternative Communication
(AAC) leverages the relative strengths in visual processing of in-
dividuals with complex communication needs such as children
with autism spectrum disorder. As with other forms of language
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544805
Karyn Mofatt
karyn.mofatt@mcgill.ca
School of Information Studies
McGill University
Montréal, Canada
acquisition, learning symbolic AAC demands a linguistic-rich envi-
ronment, with frequent opportunities for receiving and producing
language through the symbolic modality. Conversation partners
have a crucial role in this process: they need to ensure that the
AAC tool is programmed with relevant symbols and then model
language use with the tool as conversation opportunities naturally
arise [24].
However, the traditional hierarchical organization of symbol-
based AAC tools imposes substantial meta-linguistic and memory
demands on users searching to fnd desired words [35, 46], and re-
quires a great amount of time and efort from conversation partners
to select and pre-program relevant vocabulary [2]. Consequently,
tools are often programmed with only a small set of words that
cannot scale to unplanned situations, drastically limiting the op-
portunities for symbolic language use and acquisition.
One promising approach to alleviate the navigation and pre-
programming demands of traditional symbol-based AAC is combin-
ing Visual Scene Displays (VSDs) with “just-in-time” (JIT) program-
ming [4, 8, 37, 46]. This approach associates language concepts with
a photograph or image of a naturally occurring scene. Conversation
partners can program these concepts with the participation of AAC
users while the interaction takes place, i.e., “on the fy.” For example,
while at an amusement park, a family member can take a photo-
graph of the roller coaster and program concepts such as “high,”
“scared,” and “scream” on a page displaying the photograph. This en-
ables the conversation partner to model those concepts quickly, and
the individual with complex communication needs to interact with
the concepts simultaneously with the relevant real-world referents.
While this approach can capitalize on teachable moments [36], and
increase symbolic communication turns [21], it still requires efort
to manually select and program appropriate vocabularies that is
difcult to accomplish in unexpected or emergent situations.
Automated vocabulary generation techniques have been pro-
posed for constructing JIT vocabularies without human assistance,
using diferent types of contextual information as seeds [46]. Re-
searchers have explored the use of geographical locations [19, 44],
identifcation of conversation partners’ speech [55, 56], and a combi-
nation of diferent sensor data [7, 45, 53] for generating or retrieving
contextually relevant vocabularies. Photographs have also been ex-
plored for supporting people with aphasia ordering dinner [43]
and retelling past activities [39]. By applying image captioning
and optical character recognition (OCR), Obiorah et al.’s proto-
types [43] were able to translate photographs of food items and
menus of local restaurants into interactive symbols during labora-
tory experiments. However, their approach is limited to labeling
items directly depicted in the photograph and cannot be used to
ASSETS ’22, October 23–26, 2022, Athens, Greece
generate additional related concepts. To generate a set of ten words
related to a scene photographed, the system by Mooney et al. [39]
processed user-generated comments from a fctitious social media.
Although their approach showed promising results for supporting
people with aphasia in retelling past activities, the approach cannot
provide instantaneous support as it is dependent on other people
frst commenting on the photo.
To date, there has been no research on the creation of AAC
tools that automatically generate vocabulary from photographs for
use in a broad variety of communication contexts. The design of
such tools, both in terms of generation methods and interactive
interfaces, and the factors of the dynamics between individuals
with complex communication needs, their conversation partners,
and automated language support are unexplored. Consequently,
the exact kind of support and how such tools could be integrated
into real-life settings is unknown.
In this work, we present Click AAC, a prototype tool that gener-
ates situation-specifc communication boards organized in a VSD-
like layout and formed by a combination of descriptive, narrative,
and semantic related words and phrases inferred automatically from
photographs based on the technique proposed by de Vargas and
Mofatt [18]. Through our analysis of semi-structured interviews
with AAC professionals, we investigate how these professionals and
their clients with complex communication needs used Click AAC
during their routine therapy and school activities. We contribute a
deep understanding of how vocabularies generated automatically
from photographs can support individuals with complex communi-
cation needs using and learning symbolic AAC. We ofer additional
insights into the design of automatic vocabulary generation meth-
ods and interactive interfaces to provide adequate support across
scenarios of use and goals.
2 BACKGROUND AND RELATED WORK
2.1 AAC interventions by communicator
profles
Individuals with complex communication needs have an extensive
range of expressive communication abilities. Professionals such
as speech-language pathologists (SLPs) and assistive technology
evaluators are responsible for selecting tools that can adequately
attend to the specifc communicator’s evolving needs. The mapping
between available tools and users can be described according to
the three broad profles of communicators, as classifed by the
speech-hearing community: independent, context-dependent, and
emergent communicators [9]1 .
Independent communicators have literacy skills on par with
same-age peers and are able to generate completely spontaneous
messages about any topics or contexts while interacting with famil-
iar and unfamiliar partners, usually through text-based or robust
AAC systems—those containing a very large symbol set (e.g., 2000+)
organized hierarchically and with consistent arrangement for sup-
porting motor planning, in addition to allowing morphological
1 This classifcation is also used in Dynamic AAC Goals Grid-2 (DAGG-2) , a tool
for assessment and measurement of an individual’s current level of communication
popular in the clinical community.
changes (e.g., past and plural forms), programming of full sen-
tences, and access to keyboard. In contrast, emergent and context-
dependent communicators focus on gaining symbolic communi-
cation skills, and are the groups most relevant to the focus of our
work.
Context-dependent communicators can use symbolic com-
munication reliably but are still limited to certain contexts. Indi-
viduals with this profle often use dynamic displays containing
larger vocabularies organized hierarchically, but still do not take
advantage of all capabilities of a robust AAC system (e.g., com-
prehensive vocabulary and syntax modifers). They are starting to
compose two or more symbol messages, but their interactions are
still dependent on familiar partners, who must facilitate communi-
cation, selecting and programming words and messages for them,
or helping navigating vocabulary [34]. Intervention goals for this
group include increasing access to vocabulary, building literacy
skills, and expanding the communicator’s ability to interact with
more partners and contexts.
Emergent communicators use mostly body language, such
as gestures, and non-symbolic modalities that are often not easily
understandable by unfamiliar partners, and communicate primarily
about the current context. AAC interventions for such individu-
als focus on establishing more reliable communication through
symbolic expression and increasing opportunities for communica-
tion interactions. To support emergent communicators learning
the associations between real-world objects or actions with their
symbolic representation, professionals often pair these individuals
with single button communicators or static communication boards
composed of a few symbols (e.g., 4–20 on GoTalk series) repre-
senting very common words (i.e., core words). Recently, VSDs have
been proposed as alternative support for this group. VSD tools
associate language concepts with photographs taken or uploaded,
either as embedded “hot-spots” in the photograph that reveal the
concept when selected, or as a dedicated panel attached to the
photograph [4, 8].
2.2 Challenges learning and using symbolic
AAC
Not diferently from spoken language acquisition, learning sym-
bolic communication requires regular exposure to a rich linguistic
environment and frequent opportunities for language use. SLPs and
family members have a crucial role in this process [49]. They must
immerse learners in environments rich in AAC language, ensur-
ing the availability of relevant vocabulary and actively performing
aided language stimulation2 during meaningful and motivating op-
portunities [24]. In this technique, a conversation partner models
language on the learner’s device while they speak. This includes
describing their own actions while they engage in parallel play
with the learner, describing the learner’s actions, providing an ex-
ample of target production, and repeating the learner’s utterances
with additional words to create more semantically or syntactically
complete sentences [5]. Aided language stimulation has proved
efective in increasing learner’s semantic understanding of sym-
bols, [14, 15], number of communication turns, and syntax under-
standing complexity [47], and therefore, researchers recommend
2 Also known as aided language modeling or aided language input.
AAC with Automated Vocabulary from Photographs
that conversation partners should perform it in at least 70% of
interaction opportunities [15].
However, the design of traditional symbol-based AAC devices
hinders such frequent exposure to relevant symbolic communi-
cation. The main challenge with such tools is the difculty in or-
ganizing a large number of symbols needed for the spontaneous
creation of sentences in a manner that allow individuals with com-
plex communication needs and their conversation partners to easily
access desired language concepts when needed. Traditional tools
display symbols out of context, arranged in grid-based displays that
are organized hierarchically following linguistic (e.g., nouns and
verbs) or hierarchically-based (i.e., superordinate → ordinate, like
“food” → “dessert”) categories, imposing signifcant meta-linguistic
and memory demands [35, 46].
To facilitate the availability of relevant vocabulary and reduce
navigational demands, conversation partners can create topic-
specifc communication boards by selecting words related to a
topic they deem as useful and grouping them on a single page.
Nonetheless, this strategy does not scale to unexpected situations
and imposes a heavy workload on conversation partners, who must
anticipate learners’ vocabulary needs and dedicate time to program
that vocabulary into the devices. Consequently, vocabulary avail-
ability tends to be restricted to a small set of topics or a series of
frequent words that can be used across most contexts (e.g., want,
go). Conversation partners are not able to capitalize on naturally
occurring opportunities for language learning, further hindering
symbolic communication learning and learners independent use of
AAC.
2.3 Automated JIT support for AAC
The concept of “just-in-time” (JIT) support, as used in the AAC
feld, refers to the programming and availability of language con-
cepts at the moment they are needed, through technologies that
allow the easy creation of VSDs or other AAC content within in-
teractions [46]. This includes either mentor-generated JITs, such as
the creation of a hotspot on a VSD when a certain activity is hap-
pening, or automated JITs, which do not require additional human
assistance, such as playing a video that demonstrates how to wash
hands when a learner enters the bathroom. The benefts of JIT sup-
port are hypothesized based on conceptual underpinnings related
to working memory demands, situated cognition, and teachable
moments [46].
Context-aware computing has demonstrated value as an enabling
approach for automated JIT vocabulary support. Through a par-
ticipatory design involving people with aphasia, researchers [29]
explored the concept design of an AAC system that would adapt
the vocabulary presented according to the user location or conver-
sation partner to facilitate word fnding. In a Wizard of Oz study
in a local aphasia center, vocabulary was manually pre-assigned to
diferent contexts (e.g. doctor’s ofce) and presented to participants
while they were imagining using the device in that location. Al-
though their study showed the usefulness of providing vocabulary
tailored to the user’s context, the technical challenges of building
algorithms capable of generating those context-related words were
not addressed.
ASSETS ’22, October 23–26, 2022, Athens, Greece
In an attempt to generate vocabularies that attend to emergent
user needs in various locations, researchers [19] have applied infor-
mation retrieval and natural language generation (NLG) techniques
on internet-accessible corpora such as websites, dictionaries, and
Wikipedia pages related to user’s current location or conversation
topic. Although this approach was useful for augmenting a base
vocabulary with context-specifc terms during a laboratory experi-
ment simulating two locations and two topics, it is unclear how this
approach would behave in naturalistic settings or how to extend it
to personal situations (e.g., telling someone about last weekend’s
trip) for which internet-accessible corpora are unlikely to exist.
Storytelling vocabulary has been successfully generated for sup-
porting children with complex communication needs in recounting
“how was school today” to their families [53]. This method clusters
unstructured sensor data (e.g., RFID tags determining the user’s lo-
cation within the school) and transforms it into narrative sentences
using a knowledge base containing the school’s timetable and RFID
mapping information.
More recently, researchers have started exploring photographs
as the contextual information input. Obiorah et al. [43] designed
prototypes aimed at supporting people with aphasia in ordering
meals in restaurants by providing automated captioning of scenes
using images from the internet and making text-based information
and menus interactive through OCR. Mooney et al. [39] utilized
comments from a simulated social network to generate context-
related words of personally relevant events to support people with
primary progressive aphasia in retelling their past events. Although
participants in their study demonstrated increased lexical retrieval
during controlled experiments, this approach is dependent on the
availability of user-comments and thus cannot be used immediately
after the photograph is taken.
In this work, we build of of recent research by de Vargas and
Mofatt [18], which proposed a novel method for generating story-
telling vocabulary automatically from photographs for use in AAC.
Their method generates a rank of key words and short narrative
phrases from a single input photo, by matching against the Visual-
Storytelling Dataset (VIST) [27] and then expanding this initial
word list to include related ideas using the SWOW model of the
human lexicon [17]. Their performance evaluation using a subset of
VIST as groundtruth vocabulary (1,946 photos and 9,730 narrative
sentences) has shown that this method can provide relevant vocab-
ulary for creating narrative sentences. However, it is unclear how
well the technique performs when integrated into an interactive
application and evaluated by users in real-world contexts.
3 INTERACTIVE APP DESIGN
To explore the usefulness of automatic JIT vocabulary from pho-
tographs in supporting symbol-based AAC, we designed Click AAC,
an interactive mobile application that integrates diferent tech-
niques for generating context-related words and phrases. Click
AAC runs on Android and Apple smartphones and tablets.
The design of Click AAC is rooted in evidence-based recom-
mendations from HCI and AAC literature, including the design of
well-established AAC tools. Throughout the design process, the
frst author volunteered for eight months in a local aphasia center,
and integrated lessons from frst-hand communications with SLPs
ASSETS ’22, October 23–26, 2022, Athens, Greece
and people with aphasia into the design of Click AAC. Before the
launch of this user study, hundreds of AAC professionals working
directly with individuals with complex communication needs in-
formally checked the design and overall concept of the prototype
through a post on specialized social media groups. They confrmed
that the prototype design was suitable to be tested with end users
during therapy and school activities.
We detail the design rationale and important facets of Click
AAC’s vocabulary generation and user interface below.
3.1 Vocabulary Generation
AAC tools must support users in a variety of communication func-
tions across diferent contexts, such as commenting, describing,
asking and answering questions, and engaging socially [23, 33].
Therefore, Click AAC employs a combination of three generation
methods (descriptive, related, narrative) that provide vocabulary
spanning the main parts of speech for symbolic AAC (i.e., pronouns,
nouns, verbs, and adjectives).
The frst step for all methods consists of creating a set of candi-
date description tags and a human-like description sentence (i.e.,
caption) for the input photograph using the computer vision tech-
nique from Fang et al. [22]3 , as done in the work from de Vargas and
Mofatt [18]. By applying captioning rather than pure object detec-
tion and labelling, Click AAC obtains abstract concepts represent-
ing the interactions between the objects, people, and environment
depicted in the photograph (e.g., “playing”, “angry”).
This initial vocabulary is then used with distinct goals in each
method:
(1) Descriptive: Simple description of the scene. It includes
lemmas of all description tags, as well as the description
phrase.
(2) Related (Expanded): Words semantically related to the
elements in the scene. It includes lemmas of all description
tags plus lemmas of the three words most strongly connected
in SWOW—a model of the human mental lexicon constructed
from word-association experiment data—for each description
word4 .
(3) Narrative: Words and phrases used for creating narratives
about the scene photographed, obtained through the tech-
nique proposed by de Vargas and Mofatt [18]. This technique
selects vocabulary associated with similar photographs (i.e.,
having semantically similar captions) from the visual story-
telling dataset Vist [27], which contains 16,168 stories about
65,394 photos created by 1,907 mechanical Turk workers.
By default, the fnal set of vocabulary presented to the user is
the combination of all methods, limited to 20 verbs, 20 nouns, 15
adjectives, and 6 phrases, and fxed pronouns (I, you, he, she, they,
and we). We chose this combination of values to maximize the
number of vocabulary items displayed while keeping symbols size
similar to current tools, and minimizing scrolling. Vocabularies
from the Descriptive method have the highest priority, followed by
the Related (Expanded).
3 Microsoft
Azure API implementation.
4 The
words human, person, man, men, woman and women are not expanded with
SWOW vocabulary.
The app categorizes words by their parts of speech applying the
NLTK library’s tagger. Users can enable and disable each method in
the settings menu. Finally, symbols representing the vocabulary are
retrieved from ARASAAC, a repository containing more than 11,000
AAC symbols5 . If the language set in the application is diferent
from English, generated words and phrases are translated to the
target language through the Google Translate API.
3.2 Interface
Our mobile application is composed of three main screens designed
to provide direct access to its main features, as shown in Fig. 1: (1) a
home screen from which the user can import existing photos from
the device’s gallery, take a new photo, or view their album, (2) an
album screen from which the user can navigate through all their
previously imported photos and open associated communication
boards, and (3) the vocabulary page screen that presents the vocab-
ulary generated for an individual photo. The smartphone version
consists of the same three screens, with minor diferences in the
Vocabulary Page, as detailed next.
3.2.1 Vocabulary Page. Click AAC borrows the overall layout con-
cept and key features from VSDs, a state-of-art AAC support for
early symbolic communicators and individuals with cognitive and
linguistic limitations [4, 8, 12, 36, 37]: vocabulary is organized in
communication boards around a center topic represented by the
main photograph (e.g., “eating quesadillas”), rather than in hierar-
chical categories representing abstract concepts (e.g., “actions” or
“foods”).
We follow evidence-based guidelines for the design of VSDs and
grid displays for children with developmental disabilities and adults
with acquired conditions given by Light et al. [37].
First, the set of words is displayed in a grid layout with symbols
grouped and colored according to their part of speech, following
the Modifed Fitzgerald Key [38] color coding, as in popular AAC
tools. We chose this confguration over embedding vocabulary in
the photograph itself through “hotspots” to allow a larger number
of symbols to be displayed without navigation to other pages, and
to facilitate the transition between Click AAC and other popular,
grid-based tools. Each generated sentence is displayed as a single
button containing the symbols of its content words. Users can
trigger synthesized audio output by tapping on the vocabulary
buttons. Scrolling up or down possibly reveals items hidden due to
lack of available space on the screen.
Second, tablet users can navigate to other vocabulary pages by
selecting thumbnails of the signature photos, available via the navi-
gation bar on the left of the communication board currently open, a
strategy demonstrated benefcial by clinical researchers [4, 37, 54].
Selected words are displayed in a message bar on top of the screen,
allowing users to compose sentences combining individual symbols,
as in typical AAC devices. On smartphones, due to the restricted
screen size, the message bar and navigation bar are not displayed.
The main photograph is displayed on the top of the screen, with
the associated vocabulary in the bottom. Thus, smartphone users
must swipe left or right, or tap on arrows located on the sides of
the photo to navigate to other vocabulary pages.
5 ARASAAC is maintained by the Department of Culture, Sports and Education of the
Government of Aragon (Spain): https://arasaac.org/
AAC with Automated Vocabulary from Photographs ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 1: Click AAC’s Home Screen, Album, and Vocabulary Page containing words and phrases generated automatically from
a photograph. Within a Vocabulary page, users can navigate to other photos through the vertical panel on the left, or interact
with symbols. Taping on a symbol activates text-to-speech and adds the concept to the message bar on the top. Users can
reorder, remove, edit the symbol associated with a word, and add new words and sentences. The size of all elements and the
number of vocabulary items generated are customizable.

3.2.2 Editing Vocabulary Generated. To support user’s agency dur-
ing communication [52], Click AAC allows the editing of the initial
vocabulary set generated automatically to correct errors and enter
missing items. To edit vocabulary generated, users must enter the
edit mode by tapping on the main photograph and holding it for at
least 500 ms. In line with the design of other AAC apps6 , we chose
a non-obvious interaction for editing to prevent unintentional ac-
tivation. The editing mode displays a new menu bar next to the
photo with options for reordering and removing words and phrases,
adding new words, and editing symbols associated with the words.
To perform one of those actions (e.g., remove a word), users must
select the option (remove) and then select the item that will receive
the action for at least 500 ms.
3.3 Personalization settings
To maximally support a range of diferent user profles, Click AAC
allows personalization of several aspects of vocabulary generation
and interface: type of vocabulary (generation method), maximum
number of words generated for each part of speech, number of
phrases generated, language, number of columns used for each
part of speech (automatically adjusting the vocabulary buttons
to ft in the available space), and size of interface’s components
(main photo in the vocabulary page, phrases panel, words grid,
and menu bar). Users can also modify the font size (or remove
fonts completely, as suggested by Light et al. [37]), the spacing
6 Popular tools employ three-fnger swipe or a small button in the corners of screen
between vocabulary buttons, colours attributed to each part of
speech, number of columns in the photo album, voice type, rate,
and pitch. The interface in Fig. 1 shows the default confguration.
4 METHODS
We conducted a user study involving AAC professionals and their
clients with complex communication needs who used Click AAC
in their routine practices of therapy sessions or school activities.
Through questionnaires and semi-structured one-on-one online
interviews with AAC professionals, we investigated an overarching
question:
How can situation-specifc vocabularies auto-
matically generated from photographs support
communication and language learning for indi-
viduals with complex communication needs?
We explore this research question in terms of professionals’ re-
fections on their experiences with our prototype, as well as broader
factors and concepts envisioned through their experiences. This ap-
proach allowed us to understand the broad application of automatic
generation of vocabularies from photographs, without limiting
use scenarios or introducing artifcial ones. As a long-established
practice, AAC interventions must consider not just the needs of
individuals that require AAC, but also those of their conversation
partners [5, 34, 36]. These professionals regularly try novel AAC
technologies and combine multiple tools to accommodate emerging
needs dependent on the situation and client profle, in addition to
ASSETS ’22, October 23–26, 2022, Athens, Greece
practicing symbolic communication with clients and instructing
family members on how to support AAC at home. Therefore, their
expertise can provide unique higher-level perspectives than indi-
vidual users. This broad perspective was particularly pertinent to
this stage of our longer term research. HCI researchers working in
similar contexts, e.g., designing technologies for dementia care [20],
have also noted the value of working with clinicians.
In this exploratory study, our goal was not to specifcally eval-
uate Click AAC, but rather to understand the use and expand the
design space regarding automatic JIT vocabulary from photographs
in AAC. Engaging directly with users and observing them using
the application on defned tasks might bring valuable insights for
designing an application but would ofer little support for such
exploration. Nonetheless, during all interviews, care was taken to
ensure that the participants not only shared their perspectives but
also relayed the experiences of their clients. The virtual format of
the interviews also enabled us to reach a broad set of use cases
across learning, therapy, and cultural contexts, without geographic
constraints.
As a secondary investigation, we looked into user experiences
with Click AAC to understand its overall usability in naturalistic
settings. This investigation was not intended to obtain performance
metrics of Click AAC in comparison to existing approaches through
controlled experiments, but rather to ensure that the app had a
reasonable usability and to provide evidence to help us interpret
fndings that are directly infuenced by our particular implemen-
tation. This analysis could also shed light on how to improve the
interactive vocabulary support to inform future designs of such
tools.
4.1 Participants
We made Click AAC publicly available through mainstream app
store platforms, and recruited AAC professionals through a mes-
sage displayed in its initial screen. This message prompted SLPs,
who were trying or expecting to try Click AAC with one or more
individuals with complex communication needs, as well as AAC con-
sultants or evaluators, who assessed the app independently based
on their professional expertise, to enter their contact information if
they were interested in participating in the study. Eighty-four (84)
individuals agreed to participate, and 53 answered a preliminary
questionnaire regarding their experience with AAC, the profes-
sional setting of use, and their expected timeline for trying the app
with individuals with complex communication needs.
Within this group, 14 SLPs used Click AAC with their clients on
private therapy sessions or with their students in special education
for at least four weeks, and additional 6 consultants/evaluators
tested the app by themselves. This study includes the data from
these 20 professionals. Through them, we reached a variety of
settings and user profles (detailed in Table 1). We refer to these
clients and students as AAC learners throughout the paper.
4.2 Procedure
Since this study aimed to understand the use in naturalistic settings,
participants were not instructed on how or where to use the app,
but rather asked to use or continue to use it in their routine prac-
tices in the ways they judged to be most appropriate. Accordingly,
professionals used their own expertise and judgment in selecting
which clients to try the application with.
For participants who tested with AAC learners, frequency of use
ranged from a few sessions spread over four weeks to continuous
use during approximately two months. The time using the app
within their routines also greatly varied because most of the usage
occurred as the need and opportunities arose, rather than during
time slots dedicated to testing the app. Consultants/evaluators who
tested the app by themselves used the app less extensively, given
that their evaluation mostly consisted of uploading several pho-
tographs and investigating vocabulary generated without engaging
in specifc activities with AAC learners. Most participants used only
the tablet version, with the exception of P20 and P7 who also used
the smartphone one.
We interviewed the professionals through online video meetings
once they deemed their evaluation was complete and they were
ready to provide feedback. Each interview took approximately 20–
50 minutes. The semi-structured interview was guided by eight
questions, covering scenarios of use, profle of users, comparison
against current AAC tools, adequacy of the tool in professionals
and learners routines, and strengths and weaknesses of the current
prototype.
Professionals who used the application with their clients or
students also responded to a 5-point Likert scale questionnaire
containing 16 questions about interaction, vocabulary, and usage
factors (Fig. 2). Four participants (P16–P19) were uncomfortable
with communicating in English and were instead interviewed in
their preferred language (i.e., French (1), Spanish (2), Italian (1)) by
email. Interview questions and participant answers were translated
with third-party services and checked by the frst author who has
basic knowledge of those languages. Each participant received a
10$ honorarium.
4.3 Data Analysis
We conducted a refexive thematic analysis [10, 11] on the inter-
view transcripts within MAXQDA20227 . The frst author performed
inductive open coding, guided by our overarching research ques-
tion. The open codes were iteratively developed into themes and
sub-themes through axial coding, followed by selective coding. All
authors discussed the inductive codes as they were evolving, until
reaching an agreement on the themes and their interpretations.
5 OVERALL USABILITY
To help with the interpretation of the thematic analysis, we frst
present the results regarding the overall usability. These illustrate
how users perceived the quality of diferent elements in the app,
such as the vocabulary generated and the overall interaction style.
It also revealed necessary improvements, new features, and possible
avenues for improved interactive language support that can help
inform the design of future applications with automatic vocabulary
from photographs.
Fig. 2 shows the post-questionnaire answers as a diverging
stacked bar chart, with the count of participants’ answers8 rep-
resented in the x-axis. Horizontal bars are aligned by the center
7 https://www.maxqda.com/new-maxqda-2022
8 P5 and P13 did not answer the post-questionnaire.
AAC with Automated Vocabulary from Photographs ASSETS ’22, October 23–26, 2022, Athens, Greece

Current
AAC Users
ID Profession Years Exp. Caseload Setting User Profles
in this Study
(AAC Users)

non-verbal and min-verbal children; 3 with ASD, 1

P1 SLP 6 250 PT 4
with cerebral palsy
non-verbal and min-verbal children; severe sensory
P2 SLP 7 20 SES 5
dysregulation

P3 SLP 20 1 PT 1 5 yo with ASD and apraxia ; non-verbal

child with down syndrome; literate; dysarthirc

P4 SLP 6 many PT 1
speech

P5 SLP 43 25 PT 3 2 teenagers and 1 adult, all non-verbal

P6 SLP 10 4 SES 1 non-verbal child; fne-motor skill difculties

3–22 yo non-verbals and min-verbals; intellectual

P7 SLP 30 50 EC 20–25
disabilities; some with fne-motor skill difculties
non-verbal child with ASD; 2 young adults
P8 SLP 5 39 SES 3
min-verbal
6 yo with ASD, min-verbal, sensory needs; teen
P9 SLP 20 2 SES 2
min-verbal, apraxia
SLP, AAC 8 and 16 yo with signifcant cognitive and social
P11 20 3 PT 2
consultant issues, dependent on conversation partner

P13b SLP 16 50 PT, SES 2 min-verbal child; literate min-verbal 17 yo

verbal and non-verbal children with intellectual

P17ac SLP 20 16 PT 16
disabilities

P18ac SLP 15 22 PT 6 verbal and non-verbal children with ASD

SLP, AAC
P19ae 25 180 PHI 1 non-verbal child with ASD
specialist

SLP, AAC
P10 11 80–150 PS 0 diverse intellectual disabilities
specialist
SLP, AT
P12 20 40 PT 0 diverse disabilities
consultant
AT 300
P14 28 PS 0 diverse disabilities
specialist classrooms
SLP, AAC illiterate individuals with language impairment or
P15 25 3 PT 0
researcher intellectual disabilities
SLP, AAC 2800 emerging communicators and AAC experts;
P16ad 20 CATI 0
advisor subscribed aphasia, diverse cognitive impairments
SLP, AAC anyone in people with difculty naming objects and
P20ac 24 PHI 0
advisor the country navigating vocabulary
SES: Special ed. school PT: Private therapy PS: Public school CATI: Center for AT innovation PHI: Public health institute
aEmail interview b Hebrew app c Spanish app d French app e Italian app

Table 1: Participants in our user study.

ASSETS ’22, October 23–26, 2022, Athens, Greece

Strongly disagree Disagree Neutral Agree Strongly agree

1. The symbol set used was appropriate
2. The voice output quality was appropriate
3. Users could easily select desired vocabulary
4. Users could easily remove undesired vocabulary
5. Users could easily find a desired vocabulary page
6. Users could easily create a new vocabulary page
7. Users tended to access old vocabulary pages
8. Users tended to access new vocabulary pages
9. Vocabulary generated included desired words
10. Vocabulary generated included undesired words
11. Vocabulary order was adequate
12. Users enjoyed using the app
13. Users demonstrated willingness to use the app
14. Users operated the app independently
15. Users were more communicative using the app
16. Users would benefit from an app based on our prototype
12 11 10 9 8 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12
Count of participants' answers

Figure 2: Post-questionnaire scores for 12 professionals who used Click AAC with AAC learners during their practices. Original
questions are provided as supplementary material.

of the “Neutral” category. In general, professionals were satisfed
with the design of Click AAC and the support it provided. Most
users could operate the application without major issues (Q1–8).
Some participants commented that some learners’ motor abilities
required additional methods of access, such as external switches
or scanning options that were not supported by Click AAC, and
thus could not easily select desired items on a vocabulary page
(Q3). Three participants also disagreed that removing undesired
vocabulary was easy to perform. During the interviews, they ex-
plained that they had not fgured out how to access the editing
options, but once they were instructed, it became straightforward
to remove undesired items, highlighting the importance of more
evident instructions.
Regarding the perceived quality of vocabulary generated (Q9–
11), seven out of the nine participants who used the app in English
agreed that generated vocabulary included desired words. However,
most participants also agreed that the vocabulary set included non-
relevant words. Participants who had more issues with the quality of
vocabulary generated commented during interviews that Click AAC
was not recognizing the photographs they wanted to talk about,
and thus vocabulary generated was mostly irrelevant. Participants
who used translated vocabulary were least satisfed with its quality,
with two disagreeing that generated options were relevant and one
being neutral. This was not unexpected given the simplistic way
translation was handled.
The answers covering the app usage (Q12–16) indicate that, al-
though independent use of app occurred , professionals mostly
operated the app together with the learner or by themselves to
accommodate learners’ needs (e.g., physical access, level of prof-
ciency with symbolic communication). Also, participants strongly
agreed that learners demonstrated willingness and enjoyed using
the app, and that learners would beneft if there was a complete,
commercially ready application based on our prototype.
5.0.1 Importance of personalization. Besides vocabulary editing,
participants often had to perform other personalization actions to
accommodate learner’s needs (e.g., “I made some reduced choice
boards for students that couldn’t handle as many choices” (P14)).
The importance of limiting the number of vocabulary items
for each part of speech category and modifying the layout to in-
crease symbol buttons or the main photograph were highlighted
among participants. P3 attributed the observed improvement on
her learner’s communication to the ability of personalizing the tool
to match the learner’s profle:
P3: I think your board was what was able to start her
on more modalities of communication, in more areas
of her day, then what I had been previously doing with
her . . . because I was able to make it more specifc for
her needs and for her level of communication . . . and
I like how easy it is to add and take of a lot of the
icons, so that it’s not so overwhelming
5.0.2 Improving interactive language support. Participants identi-
fed two main improvements required for providing better inter-
active language support. First, they noted that, although the app
was highly customizable, they also need to be able to add familiar
people as pronouns symbols and that automatic identifcation of
the names of people photographed “would be an amazing feature”
(P5), speeding up the process. The second relates to the availability
of frequent words, i.e. “core” vocabulary, for all photographs. Par-
ticipants highlighted that would be “extremely useful” (P3) to have
a personalized core vocabulary present on all pages, and displayed
in the same location on the screen to leverage motor planning.
AAC with Automated Vocabulary from Photographs
6 FINDINGS
Our thematic analysis revealed three main themes that together
answer our overarching research question. The frst theme describes
the situations and ways in which Click AAC was incorporated in
school and therapy activities. It details the kinds of support provided
for diferent learner profles, in addition to presenting envisioned
use cases. The second theme interprets how people benefted from
the immediacy of vocabulary provided by Click AAC during those
activities. The third theme explores the dynamics between AI and
users, weighting the benefts and issues introduced by automation
and revealing the importance of keeping humans in the loop.
6.1 Click AAC ofers a fexible, complementary
AAC tool for a wide range of user profles
6.1.1 A wide range of learner profiles can benefit. Professionals se-
lected learners for trying Click AAC through the feature-matching
process [25, 48], the gold standard for AAC evaluations, in which
they consider the “learner profle, the environment they are in, and
the tasks they need to do” (P14) to select appropriate tools from
their “toolbox” for communication support or language instruction.
Overall, professionals felt that a wide range range of learner pro-
fles may beneft from technology similar to Click AAC. Selected
learners were, in majority, emerging or context-dependent commu-
nicators that were non- or minimally-verbal children with diverse
developmental disabilities (e.g., ASD, cerebral palsy). Professionals
also used Click AAC with a smaller number of children and young
adults with functional communication and some literacy skills. Pro-
fessionals described how a wide range of user profles could beneft
from auto JIT from photographs depending on how the professional
incorporates it in their practice. For example, P2 explained their
optimism about Click AAC potentially benefting more profles as
it extended the expressive capabilities for kids along the verbal and
nonverbal spectrum in terms of visual cues and structures, working
in tandem with existing AAC devices.
P2: I don’t know if there’s one profle that [Click AAC
is] better for. I think it’s pretty open to whatever pro-
fle you’re working with and tweaking . . . based on
what the kid’s goals are . . . [For kids who] are ver-
bal . . . , this really helps them visually . . . build those
sentences . . . to expand their expressive language and
hopefully generalize outside of just the app. [For]
completely nonverbal kids . . . this is really awesome
because . . . you’re limited to the amount . . . you can
talk about so it gives them more structure. So, hope-
fully then [they’ll] be able to build their language
in whatever communication app that they’re using,
that has the 3000+ words . . . not just whatever picture
they’ve taken.
Professionals also envisioned the use of such technology for
other populations such as adults with aphasia and dementia. P2,
for example, described how people with Broca’s aphasia9 , who
have “telegraphic speech” and often “want to be talking about their
9 Individuals with Broca’s aphasia have trouble speaking fuently but their comprehen-
sion can be relatively preserved. This type of aphasia is also known as non-fuent or ex-
pressive aphasia. The National Aphasia Association. https://www.aphasia.org/aphasia-
resources/brocas-aphasia/
ASSETS ’22, October 23–26, 2022, Athens, Greece
favorite things”, such as a visit to the “museum or zoo over the
weekend”, could “take pictures of” such an event and “bring it to
like, a family party and talk about it”. P15, who has experience with
people with aphasia also envisioned such use cases, but warned that
the user would need to have “pretty good reading comprehension”
to discern whether the vocabulary generated was adequate or not.
Otherwise, the app would get the user “into trouble because [users]
would be selecting messages that were maybe not appropriate or
relevant to the photo and not realize it.”
6.1.2 A complementary tool to talk about past and present contexts,
“giving them a voice” and “facilitating language development” . Pro-
fessionals viewed Click AAC as a complementary tool to facilitate
language development and enable communication about specifc
topics. They reported a variety of main goals for using the app,
including to “expand expressive language” (P2), “augment and fa-
cilitate speech language development” (P4), and reduce prompt
dependency (P8), “build[ing] up towards using it as an alternative
form of communication” (P4).
They did not see the tool as a substitute to existing robust sys-
tems because of the uncertainty of vocabulary that will be available
and because it does not give access to all language concepts the user
may need at all times—which are limitations inherent to the con-
cept of automatic JIT vocabulary and topic-specifc communication
boards, respectively. In addition to those limitations, some aspects
of our specifc design further hindered the adoption of the tool as
substitute AAC device, such as the the lack of a fxed core vocab-
ulary set across all pages and a variable arrangement of symbols
that does not promote motor planning.
On the other hand, the ability to easily access language in situa-
tions where they “want[ed] to talk about something that’s unique or
[to] tell a story” (P12) supported learners in diferent ways towards
the intervention goals. While learners used the app independently
in some instances (as reported by P2, P9, and P17), on most occa-
sions professionals operated the app in conjunction with learners
due to the learner’s cognitive and motor difculties, in line with
their regular practice involving other AAC tools. They used the app
for: i) taking photos of the current context (e.g., “desk space”, toys,
“table top activity”, and doing horticulture); ii) uploading generic
photos obtained from the internet about topics personally relevant
to learners (e.g., favourite toys and cartoon characters); iii) upload-
ing photos of past activities or events (e.g., “past vacation”, “last
weekend, and “cooking”).
Then, professionals used the vocabulary generated to work on
diferent activities that encouraged symbolic AAC, such as per-
forming aided language stimulation (i.e., modeling language) with
emerging communicators while describing, asking questions, or
making comments about the scene photographed. With context-
dependent communicators, professionals extended the activity by
instructing learners to construct their own sentences. For example,
P20 explained how she guided her student to compose sentences by
using the part of speech categorization, and P8 commented on how
Click AAC enabled her to model language faster and supported
one learner in constructing his own sentences about an activity
previously performed in the school, as he was telling a story:
P20: [I] used it for sentence construction. At frst it
was me who used it to teach my students. Then they
ASSETS ’22, October 23–26, 2022, Athens, Greece
learned how to do it and they are the ones who choose
the picture and the pictures to compose the sentence.
I guided them to put the subject frst, then the verb
and fnally the complements.
P8: The speaker is able to [model] quicker. It’s like an
easier application to be able to have [language] mod-
eled and then have them either replicate it or have
them generate their own sentence from it. [later] ev-
eryone was cooking . . . for the week, . . . making que-
sadillas. And so, . . . we just generated sentences based
on that and . . . one of them . . . could structure it to
where it was almost like he was telling a story. Like,
he could say: frst, I added the cheese, and then we
used the cheese, . . . kind of a sequencing story
P11 further illustrated how she used Click AAC in therapy as a
bridge between the auditory-verbal realm and the symbolic con-
cepts for emerging communicators, and discussed the potential of
auto JIT vocabularies from photographs for enabling communica-
tion about past events to family after therapy:
P11: It can be used fexibly either way. . . . So these are
not individuals with cognitive ability that you can just
hand [Click AAC] to them, and they can start talking
and modify it . . . They’re dependent on me to present
something that’s relevant to them, and they may not
have the physical capabilities to access it . . . [Click
AAC] is a great bridge for me to use as a therapy tool
so that I’m not just existing in the auditory-verbal
realm with them, because then I can become over
narrating everything, and I’m not anchoring them
to any concepts. So when I have something that’s
concrete, I can use that as my anchor to bridge what
I’m saying, and . . . the concepts I’m wanting to teach,
and they’re learning. . . . If you have somebody that
can . . . take a picture of . . . this event you’re seeing, and
when you go home, you have the picture, and you
can have some way of communicating that one time
event you saw, I see that as being very, very powerful.
In fact, P6 and her learner used the app for a scenario similar to
the one envisioned by P11. They relied on the generated vocabulary
to start talking about a past visit to the dentist when the learner’s
main AAC device could not provide support:
P6: So I used [Click AAC] to download a photo of
the a doctor or dentist, and I asked [a learner] where
she had been . . . and when we pulled up the dentist
picture, we got a few more words about it. So, we used
it as a supplement to an AAC that she’s already using
. . . it did help us kind of open up a conversation about
the dentist which we didn’t have easy access to use
in her standard device.
6.1.3 Theme summary. This theme revealed that a range of user
profles can beneft from automated generation of vocabulary from
photographs, and that Click AAC was used as a complementary tool
to facilitate symbolic language learning and to enable communica-
tion about specifc things, addressing some previously unmet needs.
These fndings signal potential directions to expand the existing
design to better attend users needs across the naturalistic activities
within therapy and school settings. The next theme explains how
the immediacy of vocabulary benefted and can beneft users during
those activities.
6.2 Immediacy of vocabulary facilitates
communication and language learning “on
the spot” with reduced workload
Our fndings in this theme reveals how learners and professionals
benefted and may beneft from immediate availability of situation-
specifc vocabulary from photographs across the diferent activities
and intervention goals described in the frst theme.
6.2.1 Reduced workload opening up opportunities for AAC stimu-
lation. Professionals stressed the importance of selecting and pro-
gramming appropriate fringe vocabulary to support learners in
the various situations encountered in their routines. They further
described how this is typically an arduous task, but that vocab-
ulary generated automatically from photographs can alleviate it.
Not surprisingly, conversation partners were not only overloaded
by the need of selecting and programming vocabulary on current
tools, but also unable to plan and perform these tasks for all situa-
tions encountered by learners. The instant availability of relevant
vocabulary allowed participants to increase the frequency of mo-
ments in which they could model language or engage learners with
AAC in general (which is fundamental for successful AAC inter-
ventions, as introduced in Section 2.2). For example, P7 pointed
out the challenges of helping multiple students with diferent tasks
concurrently in teaching routines. She then commented how she
was currently able to provide only core vocabulary throughout
the school day, and how auto JIT from photographs encouraged
communication on the spot by providing easy access to relevant
fringe vocabulary:
P7: One of the big drawbacks in teaching in that kind
of environment . . . is you don’t know what every-
body’s doing . . . I could be outside doing pruning and
snipping and lopping, or I could be in working with
somebody on hand washing, or somebody with feed-
ing . . . [Later] You always want a child to have the
ability to communicate, but the time for teachers to
do that is very limited. . . . I just I can’t keep up with
fringe [vocabulary]. With something like [Click AAC],
a teacher could take a picture and could encourage
that communication and they could do it quickly and
they could do it easily . . . So this is brilliant.
Continuing, P6 commented that Click AAC ofered “more spe-
cifc words” to talk about what was happening in their environ-
ment (school room) than the learner’s main AAC device, and P7
described one episode in which the instant generation of vocabu-
lary supported an unexpected situation for which she did not have
vocabulary material prepared in advance, enabling her to engage
the learner with AAC:
P7: The other day in horticulture . . . I had some fringe
vocabulary, but I had not taken pictures of pruners,
and loppers and snipers [for low-tech AAC] . . . So, I
AAC with Automated Vocabulary from Photographs
was able to take a picture of those three tools [with
Click AAC] [and] start talking about those tools.
They also postulated that auto JIT from photographs might par-
ticularly beneft families, who are less experienced with AAC tech-
nologies and vocabulary selection than professionals, and there-
fore face challenges in creating adequate on-topic communication
boards.
P1: [It] takes me probably like a few minutes to be able
to create a new page in someone’s communication
application, but that’s because I do that . . . fve days a
week [for] seven years . . . but imagine a family who
either is not tech savvy [or] just trying to keep up
with their child who has special needs. It takes them
like hours . . . they just don’t have that time. . . . the
convenience and quickness of being able to program
information into it is the most impressive thing that
I’ve seen.
The signifcantly reduced programming workload required by
Click AAC could “engage more families” to implement AAC at home,
as well as teachers in classrooms. P14, who works on selecting and
recommending assistive technologies, commented that she shared
the app to families and teachers in her county as an attempt to
encourage them to use AAC more often at home and in classroom:
P14: We have a weekly that we share to families with
free apps on it where we give information and [Click
AAC is] one of the tools for a student that’s start-
ing with communication where we want to . . . get the
family engaged with, this would be nice because it’s
not a lot of heavy programming . . . We have a lot of
families that are not super comfortable with technol-
ogy. So, if it’s easier, then we’re more likely to see
implementation . . .
In addition, P7 discussed how the easiness of having immediate
relevant vocabulary can help parents ofering opportunities for
immersing learners in AAC, and thus keeping learners engaged
during a variety of scenarios where is not be possible to have other
AAC tools to support communication:
P7: So, for the parent who wants to communicate
with their child, but they’re not going to carry around
15 core boards with the possibility of what might be
there, this is terrifc! I mean, you could be at Sea world
and take a picture of Shamu and you’re going to get
great stuf to talk to your kid about. You could be at
at a restaurant and take a picture of the food and be
able to talk about what you’re doing or while you’re
standing there waiting in line. Endlessly . . . to go to the
Harry Potter word, you could take diferent pictures
of things . . . keep them engaged .
6.2.2 Facilitated symbolic understanding and sentence construction.
Professionals discussed how Click AAC benefted learners and
themselves during aided language stimulation and sentence con-
structions activities. Having “something visual [to] anchor some
concepts” was deemed particularly important for modelling lan-
guage for emerging communicators because a picture taken was
“live” and “connecting [the symbols with] something very physical”
ASSETS ’22, October 23–26, 2022, Athens, Greece
(P13). Professionals explained how the immediate creation of sym-
bols from real world concepts support teaching learners on “how
to use symbolic communication to communicate something more
specifc” (P11). Because learners could see and use the symbols at
the same time as they engaged with the associated object or con-
cept, it was easier to “understand that a referential symbol replaced
the presence of that object” when composing messages on the AAC
tool, “such as words [do] in oral language,” as P20 described:
P20: The person can have a pictographic representa-
tion of an object or scene immediately, and therefore,
have at his disposal the symbolic representation that
he or she must learn to use communicatively to refer
to similar objects and scenes.
Participants noticed that having a concise set of vocabulary
displayed next to a photograph setting the communication context
in “real-time” supported learners in engaging in the formulation
of spontaneous sentences. P13 commented that this strategy gives
“situational context cues”, “making it easier to compose sentences
and to make it more of a conversation” with minimum navigation.
As P14 mentioned, “sometimes the navigation between boards,
as you’re learning to build sentences, it’s like a lot together”. P8
described how the layout particularly supported users with limited
attention spans:
P8: [It’s] an easier application to . . . have them either
replicate [sentences] or have them generate their own
sentence from it. It’s already right there as opposed to
needing to return back to the picture or go down to
the choices. Everything is all right there, which keeps
them focused . . . if they’re limited by their attention
span [or] cognition . . . Everything is all right there in
front of them.
Spontaneous sentence construction was also supported by the
automatic classifcation of vocabulary into part-of-speech columns.
Participants found the organization of vocabulary into columns
following the “modifed Fitzgerald coloring system” helpful because
it was “easier to read” (P9), and “one of the most common ones, so
the kids [were] familiar with that color coding” (P2), “match[ing
with] another couple of AAC” (P6). P3 further illustrated the ease
of use and the consistency of this organization:
P3: It was very easy . . . with the categories where you
have them lines up, like, the pronouns are on the
left . . . then the verbs . . . it makes it easier to follow
through when you’re trying to formulate sentences
or questions. I think it’s an easier fow, and it was
consistent across all the boards.
6.2.3 Support for communicating personal interests. Professionals
noted that instant situation-related vocabulary from photographs
enabled children who relied on nonverbal communication and had
major difculties navigating traditional AAC systems to initiate
communication about personally relevant topics, allowing profes-
sionals to “expand and build on whatever modality [learners were]
using” (P2). P6 explains how nonverbal learners often want to
initiate communication about topics interesting to them but are
ASSETS ’22, October 23–26, 2022, Athens, Greece
hindered due to the lack of easy access to relevant vocabulary, com-
paring how auto JIT from photographs provided better support in
relation to existing tools:
P6: My main purpose for it would be that “on the
go,” when I have a student that needs to talk about
something that is just too frustrating to fnd the words
for on their device. . . . right now it’s snowing, we could
take a picture of the snow and the playground, and
then the language that comes up about that is concise
and related. And, the Proloquo, the Snap Scene . . . we
have to dig and dig and dig and fnd “go”, back to the
page that has “playground”, and go forward to the
page that has “weather”. And I go back to the page
that has “clothes” . . .
The ability to choose a communication topic by selecting a pho-
tograph and having related vocabulary instantly available was also
deemed important because it allows users to talk about things that
were popular among other children, as P5 detailed:
P5: What my young people are screaming at me about
. . . is that [they] can’t say what [they] want to say, and
talk like the other kids that are out there (their peers
without disabilities) And this application gives them
the ability to do that, if the vocabulary that’s being
generated is right, ’cause they can pull up the things
that are popular, the things that are of interest to them.
6.2.4 Potential impact on motivation and confidence. Although our
user study did not focus on measuring language outcomes, our
fndings provided preliminary evidence that such technology may
improve motivation and confdence for some learners, particularly
for those who had been least successful with current tools. Some
professionals commented that learners were receptive to the tech-
nology and motivated to used it. P9 also observed improved com-
munication in a particular moment for a student who was more
confdent to speak words thanks to the support provided by Click
AAC.
P9: I’ve seen the most improvement with the one
who’s minimally verbal . . . This actually happened just
yesterday . . . We had used the app the whole session
with a puzzle, . . . so I would push the thing to say my
turn and take a turn and then I would prompt him to
do the same and he started doing that. . . . But then,
after just about a couple of minutes, he decided to be
verbal, so I would say my turn and then he would just
verbally say my turn instead of pushing the button.
So I think that kind of gave him like a little bit of
confdence . . . and an understanding of . . . I see what I
need to do . . . I’m okay with being verbal with with
this part. So it was it was a really cool moment I
thought for him to take that app in and start with it,
. . . and he’s shown some emergence of that kind of
skills a little bit through therapy.
P13 commented that her learner, who was working on literacy
skill and had no interest in engaging in language learning activities
with other tools, got motivated by trying Click AAC:
P13: [For] example, [a boy] had no interest... he [tried
Click AAC] and it was very, very emotional ’cause
. . . Once we started it till now, he’s like, I can’t read,
I don’t know, I’ll never know . . . but when I started
talking to him about the app . . . [he] started saying,
yeah, I’m going to learn to read and learn to write.
It . . . got him . . . motivated to even try, which is very
new for him.
6.2.5 Theme summary. This theme detailed the impact of having
vocabularies generated instantly from photographs, revealing four
main benefts in terms of how such vocabularies: (1) reduced the
workload for selecting and programming situation-specifc vocabu-
lary for professionals, which led to increased opportunities for AAC
practice , (2) facilitated the immersion of learners in symbolic com-
munication during language modeling and sentence construction
activities, (3) supported the communication of personal interests,
and (4) impacted on motivation and confdence engaging with sym-
bolic AAC.
6.3 Biases introduced did not compromise
support but highlighted the importance of
AI-human cooperation
Automation of vocabulary selection proved helpful and led to posi-
tive outcomes, but participants’ experiences highlighted the impor-
tance of keeping humans in the loop and revealed new aspects and
challenges intrinsic to human-AI cooperation for AAC. This theme
frst demonstrates how participants’ perceptions of the vocabulary
quality was related to the type of photographs they used as input
and the context of use. Then, it shows common biases and errors
caused by the algorithms powering Click AAC and revealed how
participants cooperated with the AI not only to overcome those
issues, but also to achieve improved support that would not be
possible by the AI or themselves alone.
6.3.1 Qality of vocabulary was directly related to the photograph’s
content, failing for some relevant situations. Our analysis revealed
common patterns in the quality of vocabulary generated across dif-
ferent input photographs, signaling the system’s high dependency
on the input photograph’s content.
Overall, professionals judged individual words generated to be
mostly relevant and requiring only a few modifcations, when the
scene photographed was correctly identifed (which users could
verify through the descriptive phrase displayed at the top option).
Participants positively noted that words were “not limited,” “not too
predictable,” and included not only the names of objects depicted
in the photo but also a broader set of words related to the scene,
“expand[ing] language.” However, they noticed that some words
were unrelated and deemed the quality of the narrative phrases as
inappropriate to support communication in the naturalist settings
they experimented in, as the instances described by P14 below:
P14: We took a picture of rainbow fsh, and that did
pull up a lot of really good vocabulary about stufed
animals that it . . . had all the colors, . . . things like draw.
And things . . . that were good to go with the book. But
then there were things that showed up that we were
like, I’m not sure how this fts in. [Later] the sentences
AAC with Automated Vocabulary from Photographs
often didn’t go with the activity that I was putting
together.
When the scene was not properly identifed, vocabulary was
generally not useful and the application was left aside. Some par-
ticipants got frustrated due to the misidentifcation, but then, by
experimenting, learned the kinds of pictures the app was able to
properly identify.
P14: [There was] this little learning period where I
was . . . a little frustrated with the AI, because it was
misreading the pictures, but once we got kind of fg-
uring it out, I was very happy with it.
In general, participants reported that Click AAC was able to
correctly identify the scene in the majority of photographs (“most
of the time it picks up what you’re doing” (P7)). However, although
identifcation on “cluttered pictures”, or with very specifc elements
or details occurred in some instances, such as specifc gardening
tools (e.g., pruners and loppers (P7)), TV characters (White, Rue,
and Bea from Golden girls (P5)), facial expressions (“straight face”
(P10), and age-related attributes (“historic” (P7)), Click AAC often
misinterpreted photographs relevant for learner’s common activi-
ties.
Participants who encountered the most difculties cited input
photographs containing “two-dimensional” “images that are not
real”, such as “cartoon’s characters”, “specifc toys”, a “door knob”,
a “smiley” face (to talk about emotions), “super heroes”, “ax throw-
ing place”, “play-doh”, “bubbles”, “holiday tree Tu BiShvat”, “body
parts for Mr. Potato head”, and “Peppa Pig”. P2, for example, dis-
cussed how learners wanted to take photos of cartoon characters or
games, but the incorrect identifcation of the specifc toy resulted
in unusable vocabulary:
P2: A lot of the pictures they’re wanting to take are
fat pictures . . . of Spiderman or, of Dora the Explorer
. . . one of the car mats, so it’s like the carpets and it
just has the cars that you can drive on it . . . But then
the vocabulary that came up, I couldn’t really talk
about it at all cause [Click AAC] thought it was a
box of cards . . . so those are the kind of pictures that
we’re ending up taking. It’s not of your normal three
dimensional objects
Besides the lack of specifcity in the identifcation of “uncommon”
scenes, participants noticed some items being constantly identifed
as other similar, but totally unrelated objects. For example, P10
discussed how “random cylinder objects” were being recognized as
soda containers, and P7 experienced “laundry soaps . . . and some
softeners . . . ” being ‘identifed as food’ ”. Some participants also
acknowledge how tricky it is to correctly identify some photos,
given potential similarities. For example, P15 described an instance
where Click AAC identifed a goat as a dog, expressing “but to be
fair, he does kind of look like a dog in this picture”.
Part of the vocabulary that participants judged inappropriate
were gender or language style-related biases introduced by the
datasets used when training the machine learning models adopted
in our generation methods. Identifying people with long hair as
woman was a common comment among participants. Another
common issue reported was that the language was not adequately
matched to user’s age-group. For example, P6 noted: “I would want
ASSETS ’22, October 23–26, 2022, Athens, Greece
to make sure that when the water bottle came up, [Click AAC]
doesn’t show me ‘wine’ ” Professionals—especially those using
Click AAC in languages other than English—also noticed difer-
ences in the vocabulary style of our generated options and their
community norms, describing these as “not adapted to our country”
(P18). P10 further expressed the need for recognizing subtle dif-
ferences between apparent synonyms, especially in language and
cultural contexts:
P10: I don’t think that my community [would] really
use the word “flth” in that way. We would use the
word “dirty” So, just being able to tweak that would
be helpful.
6.3.2 Errors and biases introduced did not compromise support, and
efort correcting and complementing the AI “was worth it”. Despite
the aforementioned errors and biases introduced by AI, participants
noted that the automation still facilitated performing language stim-
ulation with the learners during meaningful activities. In addition,
the great majority of participants found that the efort fltering,
complementing, and correcting the AI was worthwhile in compari-
son with the amount of work needed for programming the current
tools (“It takes less time to create a few boxes than to recreate a
complete page” (P6)).
Participants found it easy to edit and add individual words once
they had learned how to perform those actions, either through the
app’s embedded tutorial or by asking for researcher instructions:
“It was easy for me to move it around, take of what I didn’t want
and add what I did want” (P3). They highlighted the importance of
being able to quickly edit vocabulary for professionals and learners,
as P5 elaborated below:
P5: When you’re busy, when you’re programming
. . . and also for users who are doing their own pro-
gramming, [editing] needs to be streamlined to be as
easy as they can go in and do it . . . So, the least amount
of efort is the best thing.
In instances where professionals were mostly working on aided
language stimulation, professionals just mentally ignored irrelevant
symbols and focused on relevant ones to maximize the immediacy
of the symbolic representation, as P9 discussed: “I don’t delete
[anything]. I can . . . go through and determine which ones I like
best”.
In most situations though, participants edited the vocabulary
prior to engaging with learners, as a preparation for a specifc ac-
tivity, or in conjunction with the learner when the communication
was taking place. P9 described how she and her learners worked
together to add missing words:
P9: I think it’s really easy to add words, . . . I can do it
in real time during the session [when] we really need
this word. So, “let me put it in real quick.” . . . and my
other student who can already program the words on
his own . . . if he comes up with a word, I’ll say, “oh,
that’s a great idea. Why don’t you add that in there?”
6.3.3 Cooperation led to extended support. In most cases, once
Click AAC displayed a new vocabulary page, participants checked
if the overall scene identifcation was correct, and scanned (with
learners in some instances) through the items to remove undesired
ASSETS ’22, October 23–26, 2022, Athens, Greece
items and/or add missing words. Professionals reported that during
this scanning process, the initial set of words generated by the AI
often “served the role of a prime,” stimulating them to think of
new relevant words that they would have not thought if they were
selecting the vocabulary by themselves, as P15 discussed:
P15: I might see something that was generated by
the app that makes me think: “Oh, that’s a good
idea.” . . . this would also be appropriate and I might
not have thought of that before. . . . when it comes
to . . . vocabulary development, it’s kind of the difer-
ence between a blank slate, where you’re thinking,
okay where do I start? What do I? How do I come up
with something that’s relevant that . . . and having the
app generate some stuf for you, based on a relevant
picture, and then that triggers more ideas. So then
you might think of other things that you would try
programming to see if that would work for the client.
P11 discussed a similar efect, highlighting “the endless poten-
tial” of having vocabulary that is not strictly related to the input
photograph, using “imaginative” sentences as a starting point for
stimulating conversation through other forms of AAC:
P11: [When I tried with a photo of] my dog, it said “a
dog standing on a wood foor,” and then it came up
with something imaginative like “He decided to dress
up his dog.” So, then I could take that and run with
[the learner]. I wouldn’t have thought of that myself,
and I would be “What a great idea!” I could go to the
markup tool and on the iPhone or the iPad and start
dressing my dog up in diferent things. So, it could
be a springboard to something . . . I see that as being
endless potential.
P7 also illustrated that the mutual collaboration between users
and AI led to novel levels of support. She discussed how she adapted
her communication to incorporate words ofered by Click AAC and
expanded the interactions with the learner:
P7: [After uploading a photo of a dog,] if a child is not
really scanning, but they touch “mammal”, I can go
ahead and talk about that and I can say: yeah, she’s
a mammal, let’s think of some other mammals. Let’s
see . . . animals that have “fur” (points to the vocab
button). . . . You can really expand just with a handful
of vocabulary like that, that you would go. . . . Why
would I want the word “mammal” on a fringe board?
. . . that’s exactly why! So, you can go ahead and ex-
pand on language so that no matter what they touch,
I can go further with them . . .
6.3.4 Theme summary. This theme demonstrated how the per-
ceived quality of vocabulary was directly related to the photograph
content, informing future selection of machine learning models and
training dataset for improved scene recognition. It also explained
how participants cooperated with the AI to overcome the errors
and biases introduced, providing insights into how this cooperation
can be leveraged to reach improved support.
7 DISCUSSION
Our fndings revealed insights into the potential for automatic gen-
eration of context-related vocabulary from photographs to support
AAC, as well as on aspects specifc to our implementation in school
and speech-language therapy settings. We now discuss how the
observed and envisioned benefts of such technology relate to the
conceptual underpinning of JIT support introduced by Schlosser
et al. [46], moving to the implications for the design of such tools
for the variety of user profles and contexts of use identifed in
our analysis. We conclude with refections on the study design
employed and directions for future research.
7.1 Conceptual underpinning for the benefts
from immediacy of vocabulary
The benefts of being able to immediately generate vocabulary as
needs arise, as revealed in the second theme, included reduced
workload leading to increased opportunities for AAC stimulation,
facilitated symbolic understanding and sentence construction, sup-
port for communicating personal interests, and potential impact
on motivation and confdence. These benefts are all tightly related
to the conceptual foundations of the JIT support: working memory
demands, situated cognition, and teachable moments [46].
When communicating with the aid of a traditional dynamic grid
display, learners must keep the desired concept in mind while si-
multaneously remembering the page where that symbol is located,
how to navigate to that page, and the location of the desired symbol
on the target page, while avoiding distractions that may arise dur-
ing this process. When forming sentences, users must go through
this process several times [51]. With the combination of automated
generation of vocabulary from photographs and VSD-like interface,
users do not need to hold in memory the symbols previously se-
lected nor to remember how to navigate to a desired symbol while
constructing sentences, reducing memory demands. Our partici-
pants emphasized how this was particularly useful for constructing
sentences to model language because learners can focus on the
language concepts rather than being burdened with the navigation
task.
Our approach enabled users to have symbols representing the
real world concepts they were engaging with readily available,
which can not only alleviate working memory demands but also
facilitate situated cognition. Cognition and learning are inher-
ently dependent on the social and cultural contexts in which they
occur, and this is no diferent for language learning and comprehen-
sion [13]. Associating language elements with perceived referents
while a situation takes place is crucial for learners to comprehend
and use language. The immediacy of symbolic representation helps
to clarify the relation between objects, symbols, events, and agents
participating in that situation [1]. By providing related vocabu-
lary instantly without requiring users to anticipate the situation,
our approach can increase the frequency of moments for which
the learning of symbolic representation through aided language
stimulation is possible.
This relates to the third conceptual underpinning of JIT support,
teachable moments. According to the education literature [28],
teachable moments are those opportunities that emerge when stu-
dents are excited, engaged, and primed to learn. Adults must provide
AAC with Automated Vocabulary from Photographs
activities to children according to their level of development, al-
lowing them to “learn what they want and when they are ready to
learn” [28]. The provision of automatic JIT vocabulary can support
conversation partners capitalizing on those teachable moments by
being able to adapt the ofered support to emerging and unforeseen
situations quickly, and to engage in topics of interest of the learner,
which can activate background knowledge about those contexts,
consequently promoting comprehension [26]. Our fndings indeed
demonstrated how the auto generation of vocabulary in Click AAC
enabled or facilitated communication in those teachable moments,
even when the generated vocabulary had missing or irrelevant
words. Participants explained how the app could provide relevant
vocabulary during unplanned, very specifc activities (e.g., horti-
culture), or when fnding the words on the main device was too
frustrating for the learner (e.g., visit to dentist).
Bringing these three concepts together we can see that auto
JIT vocabulary from photographs not only reduced the workload
of AAC professionals, but also enabled them to take advantage of
teachable moments that arose during school or therapy activities, fa-
cilitating the use of situated cognition in stimulating symbolic AAC.
These benefts also speak to the need of considering modeling of
AAC use and the creation of AAC user-friendly environments when
designing new technologies, as highlighted by Bircanin et al. [6], as
well as some of the key facilitators (increased availability of techni-
cal solutions, motivated community of stakeholders) and barriers
(complex technologies, resource restrictions) for AAC adoption
and support in special education setting revealed on the ethno-
graphic study by Norrie et al. [42]. Looking beyond our results, the
impact of reduced workload on families, as hypothesized by our
participants, could be even more remarkable. Learners naturally
spend most of their time with family, but family members often
lack the expertise and time for selecting and programming vocab-
ulary needed to perform aided language stimulation sufciently
throughout their daily routines [36]. To underscore the magnitude
of this workload, note that it’s been recommended that at least 70%
of interaction opportunities should be modeled through the aid
tool [15]. With automatic vocabulary from photographs, families
have the opportunity to greatly increase the frequency of teach-
able moments without adding workload to their routines. Future
research can investigate the use of such tools with family members,
potentially revealing new facets of the cooperation between AI and
users, and new naturalist use cases unexplored in therapy or school
settings.
7.2 Expanding auto-generation of vocabulary
from photographs for other populations
The fact that auto JIT vocabulary from photographs can support
a wide range of profles opens up opportunities for broader user
groups. As envisioned by our participants, for example, people with
aphasia or dementia could use such an app independently as an
alternative mode of communication, which would likely result in
very diferent dynamics of cooperation between AI and users.
First, independent use of the tool would require individuals with
complex communication needs themselves to judge the relevance
of generated vocabulary. In our study, professionals acted as a fl-
ter, either by editing the vocabulary or ignoring unrelated words.
ASSETS ’22, October 23–26, 2022, Athens, Greece
That dynamic would be impacted during independent use of the
tool, and arguably would be very diferent across individuals with
autism, aphasia, and dementia, or other disabilities. In those cases,
a generation method with much higher precision may be required
to minimize the frustrations and confusion that the provision of
unrelated or not-so-related vocabulary may cause. For example, as
emerged from our fndings, some narrative sentences generated
were very imaginative, not exactly ftting the scene, but nonethe-
less were useful as a springboard to initiate communication. Such
serendipitous prompts can potentially better support agency for
people with aphasia in creative activities [40, 50] and people with
dementia in art therapy [32] and social sharing [16, 31].
Second, although professionals judged the editing interactions
as easy to perform once they had learned how, and the smartphone
form-factor facilitated one-hand operation, it is unclear how our de-
sign would support individuals with complex communication needs
operating the app to edit vocabulary and access it when needed.
More research is needed to investigate the design of interaction ges-
tures and layout confgurations for supporting independent editing
of vocabulary. For example, people with aphasia often have motor
defcits that would dictate diferent layout or interaction capabil-
ities. Another interesting venue for investigation is the trade-of
between vocabulary precision and editing efort across individuals.
Some people may fnd it easier to have a larger set of vocabulary
generated automatically, and then scan through it deleting unde-
sired items; others may fnd this task too difcult and require a
smaller, but more precise set of words.
Third, it is unclear exactly how other populations would make
use of the vocabulary generated. AAC use in aphasia ranges from
prompting to compensation [3]; for example, in the study by Obio-
rah et al. [43], involving the use of AI for helping people with
aphasia ordering dinner, a participant used the support provided by
a prototype to rehearse what he wanted to say rather than having
the system speak out loud or automatically place the restaurant
order for him. Literate users profcient of AAC, who do not need
the symbolic representation of vocabulary, may use the generated
words as a supplement for the next-word prediction mechanism
running in their current devices, potentially increasing the commu-
nication rate. Future research is needed to understand the impact of
removing the conversation partner and increasing the participation
of the individual with complex communication needs in the coop-
eration with the AI in tools that provide automatically generated
vocabulary from photographs.
7.3 Designing for specifc use cases
Our fndings from the frst theme provide insights into the scenarios
in which automatic JIT vocabulary from photographs can provide
support, as well as how people used the support ofered across
these situations. This enables future research to narrow down the
design of tools such as Click AAC. Since our study was exploratory
in nature, we designed Click AAC as a generic tool aimed at sup-
porting a wide set of contexts. Future research can now explore
how to leverage the capabilities of automatic JIT vocabulary from
photographs to facilitate the specifc activities identifed, including
language modeling, sentence construction, language expansion,
and past event recount.
ASSETS ’22, October 23–26, 2022, Athens, Greece
For example, researchers can explore diferent interfaces for
facilitating single word modeling for emergent communicators,
such as providing only the symbol of the main object identifed in
the scene, maximized in the display. Continuing, future work can
look into the design of tools that facilitate the practice of sentence
construction using language concepts extracted from photographs.
This may include exercises for flling gaps in sentences related to
the identifed scene, in which sentences and available options are
generated automatically. For example, taking a photograph with
a boy playing soccer as input, the application could automatically
generate the sentence “the boy is playing”, and ask the user to
complete it from a option list including baseball, tennis, and soccer.
To support language expansion, future directions include probing
new interactive interfaces and organization strategies that allow
easy exploration of semantically related words. For example, words
semantically related to the concepts appearing in the photographs,
generated by the related-expanded method, could be displayed in a
secondary level that would appear only when the user selects the
main concepts in the photograph. Finally, we propose studying how
to generate more meaningful sentences to retell a past event, in
addition to facilitating the presentation and editing of such phrases
for maximum personalization. The exploration how to combine
multiple photos of the same event for providing support is another
possibility, given that people often capture diferent moments and
angles of personally relevant events.
Another avenue for future research is to study how to create a
robust AAC that integrates automatic vocabulary photographs. Our
fndings pointed to some design opportunities, such as the use of
a customizable core vocabulary board across all pages, consistent
spacial arrangement of items to support motor planning, access to
a keyboard, and possibility to do morphological infections (e.g.,
plural and past tense).
The understanding of the usefulness of such tool on school set-
tings also raises questions on the other form factors that such
application could be created for. The use of tabletop displays and
smart boards, for example, may provide new opportunities for pro-
viding a “shared communication space” among the entire classroom,
potentially increasing the participation of peers in the interaction.
7.4 Improving quality of vocabulary generated
Our study did not focus on evaluating the quality of vocabulary gen-
erated through controlled experiments. Nonetheless, our fndings
are able to provide insights into some common, general patterns in
the quality of vocabulary generated in relation to the photograph
content, in addition to the use cases for such technology, informing
i) the future selection of machine learning models and training
dataset for improved scene recognition, ii) context-related vocabu-
lary generation methods, and iii) the selection of adequate datasets
for evaluating generation methods during early stages of system
design.
Future research can integrate existing techniques for identifying
cartoon’s characters [41, 58] and person re-identifcation [57], for
example, and study whether these models are able to attend the
needs of AAC professionals and learners during their routine activ-
ities. Another thread of research can study forms of cooperation
between AAC users, professionals, and AI to achieve enhanced sup-
port. This includes, for example, new techniques that incorporate
corrections on the image descriptions and vocabulary set generated
made by all users for retraining or reinforcing the image identifca-
tion model and/or vocabulary generation method over time, aiming
at improving their overall accuracy and precision.
The fndings that emerged in the third theme also inform how
novels methods for expanding the image description into a set of
contextually related terms following user’s own style are needed.
The narrative method used by Click AAC used corpora from adults
in the USA. This was insufcient, leading to mismatch between
users language styles and support ofered. Future research should in-
vestigate generation methods for AAC that accommodate regional
styles, and more importantly, that provide children and teenagers
with language that sounds like their peers’. One possible avenue is
to reproduce the user language style by applying the lexicon terms
manually associated with a certain photograph to new photographs
containing similar elements (as judged by the AI) during the gener-
ation process. Other strategy could be to reinforce the generation
method with vocabulary selected during communication.
The necessity of running performance evaluations of AAC sys-
tems on datasets has been discussed in the feld [18, 30]. Obtaining
quantitative fndings that are statistically signifcant and can inform
the fne-tuning of internal components for optimizing the system,
and anticipating faws before testing the system with end users
are the main reasons. In the initial evaluation of the storytelling
generation method by de Vargas and Mofatt [18], authors found
that the method was robust to variations in the input photograph.
However our fndings revealed that the technique for identifying
the scene failed for several AAC use cases, leading to unrelated
vocabulary and lack of support. Our fndings on what kind of pho-
tographs professionals and learners want to use the technology
with inform the construction of new datasets for this frst stage
of system evaluation that better represents AAC use. A possible
next step would be to extend the VIST dataset with photos and
vocabulary for cartoon characters, popular people, school objects,
and toys.
7.5 Limitations
Our approach to recruiting professionals interested about the con-
cept uncovered perspectives essential for exploring the broad use
of automated vocabulary from photographs for AAC. Professionals
expertise allowed us to understand the unique needs of users when
learning symbolic AAC, and how auto JIT vocabulary can be inte-
grated into their existing practices to support the learning process.
However, it hindered direct investigation into how AAC learners
interact with the tool.
Future work could perform on-site observations during therapy
and school sessions to better understand the interactions between
professionals, learners, and the tool, as well as assessing the level
of language support provided for diferent situations through more
controlled experiments. A possible approach is to employ a single-
subject treatment design to measure the diference in the individuals
lexical retrieval skills when using Click AAC and other tools to
support the person retelling past activities, such as in the study by
Mooney et al. [39].
AAC with Automated Vocabulary from Photographs
8 CONCLUSION
The immense potential of the “iPad and mobile technology revolu-
tion” for benefting AAC users has been discussed for more than a
decade, but current symbol-based tools still have not realized the ad-
vantages brought by recent advancements in artifcial intelligence
and context-aware computing. In this work, we integrated computer
vision and machine learning techniques proposed by de Vargas and
Mofatt [18] to create Click AAC—a mobile application that gener-
ates situation specifc communication boards automatically from
photographs. We conducted a user study with AAC professionals
and their clients with complex communication needs who used the
application in their routine practices for therapy sessions or school
activities. We contribute a nuanced understanding of how situation-
specifc vocabularies automatically generated from photographs
can support communication and language learning for individuals
with complex communication needs, ofering new insights into the
design of automatic vocabulary generation methods and interactive
interface to provide adequate support across naturalistic scenarios
of use and goals.
ACKNOWLEDGMENTS
We would like to thank all participants, who generously shared
their time and expertise in the development of this work. This
research was funded by the Fonds de Recherche du Québec - Nature
et Technologies (FRQNT), the Natural Sciences and Engineering
Research Council of Canada (NSERC) [RGPIN-2018-06130], the
Canada Research Chairs Program (CRC), and by AGE-WELL NCE,
Canada’s technology and aging network.
REFERENCES
[1] Lawrence W Barsalou. 1999. Language comprehension: Archival memory or
preparation for situated action? (1999).
[2] David Beukelman, Jackie McGinnis, and Deanna Morrow. 1991. Vocabulary
selection in augmentative and alternative communication. Augmentative and
alternative communication 7, 3 (1991), 171–185.
[3] David R Beukelman, Susan Fager, Laura Ball, and Aimee Dietz. 2007. AAC
for adults with acquired neurological conditions: A review. Augmentative and
alternative communication 23, 3 (2007), 230–242.
[4] David R Beukelman, Karen Hux, Aimee Dietz, Miechelle McKelvey, and Kristy
Weissling. 2015. Using visual scene displays as communication support options
for people with chronic, severe aphasia: A summary of AAC research and future
research directions. Augmentative and Alternative Communication 31, 3 (2015),
234–245.
[5] David R Beukelman, Pat Mirenda, et al. 1998. Augmentative and alternative
communication. Paul H. Brookes Baltimore.
[6] Filip Bircanin, Bernd Ploderer, Laurianne Sitbon, Andrew A Bayor, and Margot
Brereton. 2019. Challenges and opportunities in using Augmentative and Alterna-
tive Communication (AAC) technologies: Design considerations for adults with
severe disabilities. In Proceedings of the 31st Australian Conference on Human-
Computer-Interaction. 184–196.
[7] Rolf Black, Joseph Reddington, Ehud Reiter, Nava Tintarev, and Annalu Waller.
2010. Using NLG and sensors to support personal narrative for children with
complex communication needs. In Proceedings of the NAACL HLT 2010 Workshop
on Speech and Language Processing for Assistive Technologies. 1–9.
[8] Sarah Blackstone, J Light, D Beukelman, and H Shane. 2004. Visual scene displays.
Augmentative Communication News 16, 2 (2004), 1–16.
[9] Nancy C Brady, Kandace Fleming, Kathy Thiemann-Bourque, Lesley Olswang,
Patricia Dowden, Muriel D Saunders, and Janet Marquis. 2012. Development of
the communication complexity scale. (2012).
[10] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative research in psychology 3, 2 (2006), 77–101.
[11] Virginia Braun and Victoria Clarke. 2019. Refecting on refexive thematic analysis.
Qualitative research in sport, exercise and health 11, 4 (2019), 589–597.
[12] Kris Brock, Rajinder Koul, Melinda Corwin, and Ralf Schlosser. 2017. A com-
parison of visual scene and grid displays for people with chronic aphasia: A
ASSETS ’22, October 23–26, 2022, Athens, Greece
pilot study to improve communication using AAC. Aphasiology 31, 11 (2017),
1282–1306.
[13] John Seely Brown, Allan Collins, and Paul Duguid. 1989. Situated cognition and
the culture of learning. Educational researcher 18, 1 (1989), 32–42.
[14] Joan Bruno and David Trembath. 2006. Use of aided language stimulation to
improve syntactic performance during a weeklong intervention program. Aug-
mentative and Alternative Communication 22, 4 (2006), 300–313.
[15] Shakila Dada and Erna Alant. 2009. The efect of aided language stimulation on
vocabulary acquisition in children with little or no functional speech. (2009).
[16] Jiamin Dai and Karyn Mofatt. 2020. Making Space for Social Sharing: Insights
from a Community-Based Social Group for People with Dementia. In Proceedings
of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–13.
[17] Simon De Deyne, Steven Verheyen, Amy Perfors, and Daniel J Navarro. 2015.
Evidence for widespread thematic structure in the mental lexicon.. In CogSci.
[18] Mauricio Fontana de Vargas and Karyn Mofatt. 2021. Automated Generation of
Storytelling Vocabulary from Photographs for use in AAC. In Proceedings of the
59th Annual Meeting of the Association for Computational Linguistics and the 11th
International Joint Conference on Natural Language Processing (Volume 1: Long
Papers). 1353–1364.
[19] Carrie Demmans Epp, Justin Djordjevic, Shimu Wu, Karyn Mofatt, and Ronald M
Baecker. 2012. Towards providing just-in-time vocabulary support for assistive
and augmentative communication. In Proceedings of the 2012 ACM international
conference on Intelligent User Interfaces. 33–36.
[20] Emma Dixon and Amanda Lazar. 2020. Approach Matters: Linking Practitioner
Approaches to Technology Design for People with Dementia. In Proceedings of
the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–15.
[21] Kathryn DR Drager, Janice Light, Jessica Currall, Nimisha Muttiah, Vanessa Smith,
Danielle Kreis, Alyssa Nilam-Hall, Daniel Parratt, Kaitlin Schuessler, Kaitlin
Shermetta, et al. 2019. AAC technologies with visual scene displays and “just in
time” programming and symbolic communication turns expressed by students
with severe disability. Journal of intellectual & developmental disability 44, 3
(2019), 321–336.
[22] Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr
Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015.
From captions to visual concepts and back. In Proceedings of the IEEE conference
on computer vision and pattern recognition. 1473–1482.
[23] Brenda Fossett and Pat Mirenda. 2007. Augmentative and alternative communi-
cation. Handbook on developmental disabilities (2007), 330–348.
[24] Carol Goossens’. 1989. Aided communication intervention before assessment: A
case study of a child with cerebral palsy. Augmentative and Alternative Commu-
nication 5, 1 (1989), 14–26.
[25] Jessica Gosnell, John Costello, and Howard Shane. 2011. Using a clinical ap-
proach to answer “What communication apps should we use?”. Perspectives on
augmentative and alternative communication 20, 3 (2011), 87–96.
[26] Audrey L Holland. 1975. Language therapy for children: Some thoughts on
context and content. Journal of Speech and Hearing Disorders 40, 4 (1975), 514–
523.
[27] Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya
Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv
Batra, et al. 2016. Visual storytelling. In Proceedings of the 2016 Conference of the
North American Chapter of the Association for Computational Linguistics: Human
Language Technologies. 1233–1239.
[28] Eunsook Hyun and J Dan Marshall. 2003. Teachable-moment-oriented curriculum
practice in early childhood education. Journal of Curriculum Studies 35, 1 (2003),
111–127.
[29] Shaun K Kane, Barbara Linam-Church, Kyle Althof, and Denise McCall. 2012.
What we talk about: Designing a context-aware communication tool for people
with aphasia. In Proceedings of the 14th international ACM SIGACCESS conference
on Computers and accessibility. 49–56.
[30] Per Ola Kristensson, James Lilley, Rolf Black, and Annalu Waller. 2020. A Design
Engineering Approach for Quantitatively Exploring Context-Aware Sentence
Retrieval for Nonspeaking Individuals with Motor Disabilities. In Proceedings of
the 2020 CHI Conference on Human Factors in Computing Systems. 1–11.
[31] Amanda Lazar, Caroline Edasis, and Anne Marie Piper. 2017. Supporting people
with dementia in digital social sharing. In Proceedings of the 2017 CHI Conference
on Human Factors in Computing Systems. 2149–2162.
[32] Amanda Lazar, Jessica L. Feuston, Caroline Edasis, and Anne Marie Piper. 2018.
Making as Expression: Informing Design with People with Complex Communi-
cation Needs through Art Therapy. In Proceedings of the 2018 CHI Conference on
Human Factors in Computing Systems. ACM, 1–16.
[33] Janice Light. 1988. Interaction involving individuals using augmentative and
alternative communication systems: State of the art and future directions. Aug-
mentative and alternative communication 4, 2 (1988), 66–82.
[34] Janice Light. 1989. Toward a defnition of communicative competence for individ-
uals using augmentative and alternative communication systems. Augmentative
and alternative communication 5, 2 (1989), 137–144.
[35] Janice Light, Kathryn Drager, John McCarthy, Suzanne Mellott, Diane Millar,
Craig Parrish, Arielle Parsons, Stacy Rhoads, Maricka Ward, and Michelle Welliver.
ASSETS ’22, October 23–26, 2022, Athens, Greece
2004. Performance of typically developing four-and fve-year-old children with
AAC systems using diferent language organization techniques. Augmentative
and Alternative Communication 20, 2 (2004), 63–88.
[36] Janice Light, David McNaughton, and Jessica Caron. 2019. New and emerging
AAC technology supports for children with complex communication needs and
their communication partners: State of the science and future research directions.
Augmentative and Alternative Communication 35, 1 (2019), 26–41.
[37] Janice Light, Krista M Wilkinson, Amber Thiessen, David R Beukelman, and
Susan Koch Fager. 2019. Designing efective AAC displays for individuals with
developmental or acquired disabilities: State of the science and future research
directions. Augmentative and Alternative Communication 35, 1 (2019), 42–55.
[38] Eugene T McDonald and Adeline R Schultz. 1973. Communication boards for
cerebral-palsied children. Journal of Speech and Hearing Disorders 38, 1 (1973),
73–88.
[39] Aimee Mooney, Steven Bedrick, Glory Noethe, Scott Spaulding, and Melanie
Fried-Oken. 2018. Mobile technology to support lexical retrieval during activity
retell in primary progressive aphasia. Aphasiology 32, 6 (2018), 666–692.
[40] Timothy Neate, Abi Roper, Stephanie Wilson, and Jane Marshall. 2019. Empower-
ing expression for users with aphasia through constrained creativity. Proceedings
of the 2019 CHI Conference on Human Factors in Computing Systems, 1–12.
[41] Nhu-Van Nguyen, Christophe Rigaud, and Jean-Christophe Burie. 2017. Comic
characters detection using deep learning. In 2017 14th IAPR international confer-
ence on document analysis and recognition (ICDAR), Vol. 3. IEEE, 41–46.
[42] Christopher S Norrie, Annalu Waller, and Elizabeth FS Hannah. 2021. Establishing
Context: AAC Device Adoption and Support in a Special-Education Setting. ACM
Transactions on Computer-Human Interaction (TOCHI) 28, 2 (2021), 1–30.
[43] Mmachi God’sglory Obiorah, Anne Marie Marie Piper, and Michael Horn. 2021.
Designing AACs for People with Aphasia Dining in Restaurants. In Proceedings
of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
[44] Rupal Patel and Rajiv Radhakrishnan. 2007. Enhancing Access to Situational
Vocabulary by Leveraging Geographic Context. Assistive Technology Outcomes
and Benefts 4, 1 (2007), 99–114.
[45] Ehud Reiter. 2007. An architecture for data-to-text systems. In Proceedings of the
Eleventh European Workshop on Natural Language Generation (ENLG 07). 97–104.
[46] Ralf W Schlosser, Howard C Shane, Anna A Allen, Jennifer Abramson, Emily
Laubscher, and Katherine Dimery. 2016. Just-in-time supports in augmentative
and alternative communication. Journal of Developmental and Physical Disabilities
28, 1 (2016), 177–193.
[47] Samuel C Sennott, Janice C Light, and David McNaughton. 2016. AAC model-
ing intervention research review. Research and Practice for Persons with Severe
Disabilities 41, 2 (2016), 101–115.
[48] H Shane and J Costello. 1994. Augmentative communication assessment and the
feature matching process. In Mini-seminar presented at the annual convention of
the American Speech-Language-Hearing Association, New Orleans, LA.
[49] Martine M Smith. 2015. Language development of individuals who require aided
communication: Refections on state of the science and future research directions.
Augmentative and Alternative Communication 31, 3 (2015), 215–233.
[50] Carla Tamburro, Timothy Neate, Abi Roper, and Stephanie Wilson. 2020. Ac-
cessible Creativity with a Comic Spin. ASSETS 2020 - 22nd International ACM
SIGACCESS Conference on Computers and Accessibility, 1–11.
[51] Jennifer J Thistle and Krista M Wilkinson. 2013. Working memory demands of
aided augmentative and alternative communication for individuals with develop-
mental disabilities. Augmentative and Alternative Communication 29, 3 (2013),
235–245.
[52] Nava Tintarev, Ehud Reiter, Rolf Black, and Annalu Waller. 2014. Natural language
generation for augmentative and assistive technologies. In Natural Language
Generation in Interactive Systems. Cambridge University Press, 252–277.
[53] Nava Tintarev, Ehud Reiter, Rolf Black, Annalu Waller, and Joe Reddington.
2016. Personal storytelling: Using Natural Language Generation for children
with complex communication needs, in the wild. . . . International Journal of
Human-Computer Studies 92 (2016), 1–16.
[54] Sarah E Wallace and Karen Hux. 2014. Efect of two layouts on high technology
AAC navigation and content location by people with aphasia. Disability and
Rehabilitation: Assistive Technology 9, 2 (2014), 173–182.
[55] Bruce Wisenburn and D Jefery Higginbotham. 2008. An AAC application using
speaking partner speech recognition to automatically produce contextually rele-
vant utterances: Objective results. Augmentative and alternative communication
24, 2 (2008), 100–109.
[56] Bruce Wisenburn and D Jefery Higginbotham. 2009. Participant evaluations of
rate and communication efcacy of an AAC application using natural language
processing. Augmentative and Alternative Communication 25, 2 (2009), 78–89.
[57] Di Wu, Si-Jia Zheng, Xiao-Ping Zhang, Chang-An Yuan, Fei Cheng, Yang Zhao,
Yong-Jun Lin, Zhong-Qiu Zhao, Yong-Li Jiang, and De-Shuang Huang. 2019. Deep
learning-based methods for person re-identifcation: A comprehensive review.
Neurocomputing 337 (2019), 354–371.
[58] Yi Zheng, Yifan Zhao, Mengyuan Ren, He Yan, Xiangju Lu, Junhui Liu, and Jia Li.
2020. Cartoon face recognition: A benchmark dataset. In Proceedings of the 28th
ACM international conference on multimedia. 2264–2272.
LaMPost: Design and Evaluation of an AI-assisted Email Writing
Prototype for Adults with Dyslexia
Steven M. Goodman Erin Buehler Tifanie N. Horne
smgoodmn@uw.edu Patrick Clary Michal Lahav
University of Washington Andy Coenen Robert MacDonald
Seattle, Washington, USA
Google Research
Aaron Donsbach Rain Breaw Michaels
ebuehler@google.com tifanieh@google.com
Seattle, Washington, USA
pclary@google.com mlahav@google.com
andycoenen@google.com bmacdonald@google.com
donsbach@google.com rainb@google.com
Google Google
Mountain View, California, USA Mountain View, California, USA

Ajit Narayanan Lei Shi Meredith Ringel Morris

Mahima Pushkarna Rachel Sweeney merrie@google.com
Joel Riley Phil Weaver Google Research
Seattle, Washington, USA
Alex Santana Ann Yuan
ajitnarayanan@google.com leileilei@google.com
mahimap@google.com rachelsweeney@google.com
joel.c.riley@gmail.com pweaver@google.com
alexsantana@google.com annyuan@google.com
Google Google
Mountain View, California, USA Mountain View, California, USA

Figure 1: The LaMPost interface. The system augments a typical browser-based email editor with three AI-powered features:
(left) users can generate an outline of the email’s main ideas (a) with the option for a related subject line (b); (center) users can
generate suggestions for possible changes to a selected passage, and (right) users can generate rewritten text for a selected
passage based on a human- or machine-written instruction.
ABSTRACT word retrieval technologies to address these challenges. However,
Prior work has explored the writing challenges experienced by peo- the capabilities for natural language generation demonstrated by
ple with dyslexia, and the potential for new spelling, grammar, and the latest class of large language models (LLMs) highlight an oppor-
tunity to explore new forms of human-AI writing support tools. In
this paper, we introduce LaMPost, a prototype email-writing inter-
face that explores the potential for LLMs to power writing support
This work is licensed under a Creative Commons Attribution International
4.0 License.
tools that address the varied needs of people with dyslexia. LaM-
Post draws from our understanding of these needs and introduces
ASSETS ’22, October 23–26, 2022, Athens, Greece novel AI-powered features for email-writing, including: outlining
© 2022 Copyright held by the owner/author(s). main ideas, generating a subject line, suggesting changes, rewriting
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3544819 a selection. We evaluated LaMPost with 19 adults with dyslexia,
ASSETS ’22, October 23–26, 2022, Athens, Greece
identifying many promising routes for further exploration (includ-
ing the popularity of the “rewrite” and “subject line” features), but
also fnding that the current generation of LLMs may not surpass
the accuracy and quality thresholds required to meet the needs
of writers with dyslexia. Surprisingly, we found that participants’
awareness of the AI had no efect on their perception of the system,
nor on their feelings of autonomy, expression, and self-efcacy
when writing emails. Our fndings yield further insight into the
benefts and drawbacks of using LLMs as writing support for adults
with dyslexia and provide a foundation to build upon in future
research.
CCS CONCEPTS
• Human-centered computing → Accessibility technologies; Em-
pirical studies in accessibility.
KEYWORDS
large language models, dyslexia, writing
ACM Reference Format:
Steven M. Goodman, Erin Buehler, Patrick Clary, Andy Coenen, Aaron
Donsbach, Tifanie N. Horne, Michal Lahav, Robert MacDonald, Rain Breaw
Michaels, Ajit Narayanan, Mahima Pushkarna, Joel Riley, Alex Santana,
Lei Shi, Rachel Sweeney, Phil Weaver, Ann Yuan, and Meredith Ringel
Morris. 2022. LaMPost: Design and Evaluation of an AI-assisted Email
Writing Prototype for Adults with Dyslexia. In The 24th International ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS ’22), October
23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 18 pages. https:
//doi.org/10.1145/3517428.3544819
1 INTRODUCTION
Dyslexia refers to a cluster of symptoms that result in challenges
with word recognition, reading fuency, spelling, and writing that
impacts up to 20% of the population [1, 32, 59]. While some adults
with dyslexia may learn and adopt compensatory strategies for read-
ing difculties over time [39, 42, 44], the combination of reading,
comprehension, and planning skills required to carry out writing
tasks may lead to on-going difculties [42, 55]. In addition to low-
level obstacles such as spelling and grammar, writers with dyslexia
report a variety of high-level challenges (e.g.. [13, 42, 47]), such as
ordering and expressing their ideas, choosing language to match
their desired tone, and writing with clarity and precision. To over-
come these obstacles, they report using a variety of strategies—such
as speech-to-text tools to dictate ideas, templates to match style,
and revising feedback from friends and family—but these can add
further complexity and time to their writing process [13, 46].
Prior work in accessibility has explored a number of approaches
to overcome the reading challenges associated with dyslexia, such
as experimenting with various forms of text presentation [19, 50])
and synonym substitution for complex words [51]. However, work
targeting dyslexia’s associated writing challenges has primarily
focused on low-level interventions, including automatic suggestions
to support word retrieval [37, 41, 49] and specialized spellcheck
tools (e.g., [37, 45, 52, 66]). AI-based eforts, when present, have
continued this thread; for example, Wu et al. evaluated a dyslexia-
tuned Neural Machine Translation model for spelling and grammar
support on social media posts [66]. However, tools that can lend
support to people with dyslexia for important high-level aspects of
Goodman, et al.
writing—such as organization, expression, and voice—are absent
from accessibility literature. This gap highlights an opportunity to
explore the potential for AI-powered writing support tools that use
state-of-the-art neural language models.
Neural language models are neural networks that are trained
to predict the next word in a sequence given the previous words.
We use “large language models,” or LLMs, to refer to the recent
class of neural language models (e.g., GPT-3 [7]) that have been
trained using the Transformer neural architecture [64] and are
capable of generating long passages of text that human evaluators
perceive as human-written [15]. With few-shot learning to enable
controllable text generation, LLMs hold potential to drive new
technologies that bolster written expression [69]. This functionality
may provide signifcant value to writers with dyslexia by alleviating
common difculties and simplifying their existing workfow, but
questions arise over the correct approach for their implementation.
For example, although automatic text generation could help some
writers with dyslexia to conquer their “fear of the blank page” [47],
machine-powered writing may raise concerns over the authors’
control and autonomy in the writing process [27].
In this paper, we introduce LaMPost, an LLM-based prototype
to support adults with dyslexia with writing emails. LaMPost im-
plements LaMDA [60], an LLM for dialog applications, to augment
a standard email editor with AI-powered outlining, subject gener-
ation, suggestion, and rewriting features. We evaluated LaMPost
with 19 adult participants with dyslexia. Our fndings indicate en-
thusiasm among this demographic for high-level writing support
features, including rewriting passages in a particular tone or style
(e.g., “more formal”, “more concise”) and generating summative
content such as subject lines based on an email’s body. However,
we also found that accuracy and quality issues in the current gen-
eration of LLMs present obstacles to a reliable and trustworthy
writing-support experience. Further, efectively utilizing LLMs for
writers with dyslexia may require HCI innovations to manage trade-
ofs, such as autonomy vs. cognitive load and personalization vs.
privacy. Knowledge that our writing-support tool contained AI
did not have a signifcant efect on participants’ perception of the
system and written work. Our fndings highlight opportunities and
challenges of AI-assisted writing support for people with dyslexia
and provide a foundation for future work as the capabilities of gen-
erative language models—and our understanding of their risks and
trade-ofs—mature.
2 RELATED WORK
Our research is informed by and builds upon: work on dyslexia and
associated writing challenges, prior accessibility research with this
population, and AI-assisted writing tools.
2.1 Writers with Dyslexia
Dyslexia is a multifaceted condition characterized by difculties
with word recognition, reading fuency, spelling, and/or writing [59].
According to van Schaik [63], the complete defnition of dyslexia
varies according to the lens studying it. Through a medical lens,
dyslexia is defned as a cognitive defciency that is associated with
persistent difculties with reading, spelling, short-term/working
memory, and day-to-day organization (e.g., [13, 24]). Through a lens
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
of neurodiversity, however, dyslexia is defned by heightened spa-
tial and perceptual abilities, interconnected and dynamic reasoning,
and narrative and holistic thinking—alongside commonly defned
defcits [4]. Critical scholars defne dyslexia as a person’s failure
to meet the socially constructed expectations of timelines, literacy,
and communication that are embedded in one’s broader social and
cultural context [17, 29]. While dyslexia impacts up to 20% of the
population [1], structural disparities including gender, class, and
race [42, 63] cause many cases to go undiagnosed—leading many
to be unaware of the cause of their reading and writing difculties.
While many dyslexia-related challenges remain into adulthood [8],
individuals with dyslexia may learn and adopt compensatory strate-
gies for reading difculties over time [39, 42, 44]. Although reading
may still prove challenging, some adults with dyslexia report that
writing tasks tend to provide their greatest difculties [42, 55].
Writing challenges are wide-ranging, but some are commonly re-
ported [13, 18, 42, 47, 61]: on a high level (overall plan and structure),
these include organizing and expressing one’s thoughts, structuring
and ordering ideas, and overcoming a “fear of the blank page” [47].
At a lower level (sentence and word), challenges can include word
retrieval, sentence composition, appropriate tone and concision,
grammar, spelling, punctuation, and proofreading. As with read-
ing, writers with dyslexia may adopt strategies to assist in their
writing—such as preferred spell-checkers, text-to-speech and dicta-
tion software, and support from friends and family—but these can
add complexity and time to their writing process [13, 46].
In this paper, we contribute fndings regarding the needs and
challenges experienced by adults with dyslexia in email-writing,
and we explore how these might be addressed with AI interventions.
2.2 Dyslexia & Accessible Technology Design
Because text readability is impacted by the visual display of text, re-
searchers have explored how to alleviate reading challenges through
text presentation, such as typography choices [50], word segmenta-
tion [3], background colors [53], and increased font size and margin
space [19, 54]. To support reading comprehension, one promising
approach involves text simplifcation [49]. Rello et al. [51] found
promising results among readers with dyslexia when displaying ba-
sic synonyms alongside complex words, although readers struggled
when a simpler word was substituted automatically. In a study of
web searchers with dyslexia [41], participants sought pages utiliz-
ing multimedia whilst containing minimal visual clutter and large
text blocks (preferring headings and bullets instead). In the design
of our email-writing prototype, we draw from elements of this body
of work to address usability challenges—including guidelines for
text presentation and visual clutter—and explore text simplifcation
as a form of writing support.
Work in accessibility targeting writers with dyslexia has primar-
ily focused on low-level interventions, such as specialized spellcheck
tools (e.g., [37, 45, 52, 66]). While common spellcheck tools provide
value to this population, specialized versions are motivated by
the high occurrence of "real word" errors (e.g., “hear” and “here”)
among people with dyslexia that most common tools cannot recog-
nize [45, 52]. Text suggestions can also provide value when writing
to overcome word retrieval difculties [37, 41, 49]. PoliSpell [37]
was an early attempt to design a spellcheck and autocomplete tool
ASSETS ’22, October 23–26, 2022, Athens, Greece
for people with dyslexia, but it was never evaluated. Wu et al. ex-
amined the experience of writers with dyslexia on social media,
fnding challenges associated with not only the writing task, but
concerns over social self-presentation [55]—and used their results
to build a dyslexia-tuned Neural Machine Translation model for
spelling and grammar support on Facebook posts [66].
So far, accessibility researchers have not explored the high-
level challenges associated with dyslexia during text construc-
tion, nor have they begun exploring AI-powered solutions to these
challenges—a gap our work aims to address.
2.3 Large Language Models
The most recent class of large language models such as GPT-3 [7]
demonstrate signifcant advances in natural language generation.
At their core, these models have a simple API: given a string of text,
known as a prompt [55], they return plausible continuations for
that string. For example:
prompt: A healthy lunch includes
language model: fruits, vegetables, protein, and whole grains.
Prompts can also be written with exemplars of a desired response,
such that the model ends up performing a specifc task by continu-
ing the text. In this way, they are capable of few-shot learning [7]
which is shown to be more accurate than the zero-shot example
above [71]. This example prompts the language model for a piece
of clothing and an accessory to plan for the weather:
prompt: When it’s sunny, I need:
shorts and sunscreen
When it’s raining, I need:
rain boots and an umbrella
When it’s snowing, I need:
language model: mittens and a shovel
Researchers have begun exploring many end-user facing applica-
tions powered by LLMs, including chatbots and conversational
agents [68], code generation [2, 14, 70], creative writing [16, 28, 69],
and even accessibility applications such as keystroke-saving abbre-
viations expansions to accelerate eye-gaze typing by users with
motor disabilities [10].
While LLMs exhibit impressive performance on many tasks and
have many potential applications, these models also have draw-
backs. Of particular note is that such models risk generating factu-
ally incorrect, ofensive, or stereotyped text since they are trained
on content from the internet [5, 65]. “Memorization” (e.g., regur-
gitating existing text rather than producing novel content) is also
a risk of current LLMs [12]. The risks of erroneous or inappropri-
ate output from LLMs carry additional ethical challenges when
embedded in systems used by vulnerable audiences, such as users
with dyslexia, who may experience challenges in interpreting the
output’s quality [25, 40]. Mitigating the risks associated with LLMs
is an active area of research; for example, LaMDA (the LLM under-
pinning LaMPost) uses fne-tuning to improve model safety [60].
2.4 AI-Assisted Writing
Human-AI co-creation in the writing domain (for general audiences)
has been widely studied, and applications such as Gmail’s Smart
Reply feature [35] have already been deployed to massive audiences.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Buschek et. al. [9] explored the impact of multiple suggested text
continuations when writing emails, fnding benefts to ideation at
the cost of writing speed. Gero et. al. [26, 27] studied automatic
synonym and metaphor generation and found both features enabled
greater expression during the writer’s process, but questions arose
over autonomy and ownership over the produced text [27]. Further
questions arise over how the algorithms powering these systems
should be presented to the user, as users’ perceptions toward an
AI system and desire to use it can be impacted by this choice of
presentation [33, 36]. We explore these questions further in this
work.
Wordcraft [16, 69] explored LLMs incorporated into the writ-
ing process: users collaborate with the model to write a story
through a variety of operations—including inflling, elaboration,
and rewriting—as well as open-ended dialog. The system augments
a traditional text editor with a set of integrated LLM-powered con-
trols driven by novel prompting techniques that enabled users to
build their own custom controls, such as "rewrite the text to be more
melodramatic". In this paper, we adapt Wordcraft’s approach to
provide LLM-powered controls for writers with dyslexia, building
an email editor with features for automatic outlining, subject gener-
ation, rewriting, and suggestions. We also focus specifcally on how
AI-assisted writing can support the needs of adults with dyslexia
(rather than general audiences).
3 LAMPOST: AN LLM-POWERED PROTOTYPE
FOR EMAIL WRITING SUPPORT
Prior to developing the LaMPost system, our interdisciplinary re-
search team engaged in over a year of participatory research with
the dyslexia community. This included participatory design ses-
sions and workshops with partner organizations with expertise in
reading disabilities1 , and culminated in a brainstorming workshop
on AI-assisted writing support with experts in accessibility and
dyslexia (including team members with lived experience of dyslexia
and other visual processing and reading disabilities).
As a key step in our formative work, we conducted a 90-minute
formative study to motivate the design of an AI-powered writing
system. We recruited seven adults with dyslexia to join the two-
part study for (1) individual interviews on writing practices and
challenges, and (2) a group assessment of possible ideas for AI writ-
ing support. Individual interviews highlighted several challenges,
such as: planning how to order ideas, expressing ideas in clear
and concise wording, writing with appropriate tone, and fnding
proofreading help. Further, the group interviews highlighted an
overall interest in AI writing support: revising feedback could help
with clarity, verbosity, and tone; summarization could validate an
intended meaning; and visual organization could help to order and
structure ideas. However, concerns arose over users’ capability to
address AI feedback, maintaining autonomy and control over their
work, and their privacy.
Informed by our formative inquiries and prior work [42, 46, 47],
we built LaMPost, a web application and LLM-powered prototype
for email writing support for writers with dyslexia (Figure 1). We
chose email writing as a constrained—yet highly practical—use case
1 Partner
organizations included the British Dyslexia Association, Understood.org,
Madras Dyslexia Association, and the Landmark School in Prides Crossing, MA.
Goodman, et al.
to demonstrate and compare diferent approaches for building a
text editor infused with generative writing support.
In the following sections, we explain how LaMPost works and
describe its design (including accessibility considerations to support
users with dyslexia) and functionality (including the key motiva-
tions from our formative work for LaMPost’s LLM-based features).
3.1 Email-Writing Support Through Few-Shot
Learning
LaMPost is powered by LaMDA, a neural language model [60],
and adapts the few-shot prompting methods introduced by the
Wordcraft system [16, 69] (a LaMDA-powered tool for creative
story-writing by general audiences). When the user selects one
of LaMPost’s operations, the system constructs a custom few-shot
learning prompt and sends it to the language model.
Prompt performance is highly sensitive to word choice, for-
matting, and the content of the exemplars [71]. Writing efective
prompts requires rigorous testing and iteration to achieve reliable
and accurate responses from the model. When building LaMPost,
we experimented with diferent prompting methods for several
possible LLM-based features before settling on the three in our fnal
system. We describe our iterative development process for LaMPost
and refect on lessons learned in Section 5.3.
3.2 LaMPost’s Design and Functionality
LaMPost’s interface consists of a main panel that resembles a stan-
dard email editor, including sections for the email’s recipients, sub-
ject, and body; and “undo”, “redo”, and “clear” buttons near the
bottom. A secondary panel on the right is reserved for three LLM-
powered features: identifying main ideas (with the option to gener-
ate an email subject), rewriting a selection, and suggesting how to
rewrite a selection. These three features were inspired by fndings
from our formative work with organizations and participants with
dyslexia, as we describe in more detail below.
3.2.1 LLM Feature 1: Identify Main Ideas. Users can generate a
visual outline of their email with the main idea from each para-
graph. Additionally, they can choose to generate a new subject
line from this outline (Figure 2). This feature was motivated by
feedback from our formative study with dyslexic adults, in which
participants noted that to overcome difculties with organizing
ideas and making them understandable to readers, visual organi-
zation and automatic summarization were desired technological
supports. The visual outline can make it easier to parse sections of a
long text, while simplifed content can make it easier to understand
that text [49]. The option to generate a subject line allows users
to ask the AI to simplify the content for them. By displaying the
AI’s interpretation of the email’s salient points, we imagined this
feature could also show users how the email’s main ideas might be
extracted by another reader.
3.2.2 LLM Feature 2: Suggest Possible Changes. Users can select
a word, phrase, or paragraph and ask the AI for suggestions on
how to rewrite it (Figure 3). Participants in our formative study
described feeling unsure about the kinds of adjustments needed for
their writing, and were interested in automatic suggestions for high-
level language characteristics like tone and clarity. Results from
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 2: The Identify Main Ideas feature. Users can click ‘identify main ideas’ (1) and ‘include a new subject’ (2) to generate an
outline of their email based on the main ideas of each paragraph (4) with a subject line generated on top (3). Hovering over
each item of the outline highlights that respective paragraph in the email body.

Figure 3: The Suggest Possible Changes feature. A user can select a passage of text (1) and click ‘suggest how to rewrite’ (2).
Several suggestions from the AI for changing the passage will populate in the right-hand panel (3). Users can choose to exit the
operation and rewrite the text themselves (4), or choose to have individual suggestions ‘read aloud’, discarded, or to used as a
prompt for the ‘Rewrite My Selection’ feature (5).

the LLM appear as several suggestions for changing the selected
passage; for example, “Rewrite this sentence to be less business-like”.
Users can take these suggestions into consideration to guide their
own revisions, or use a preferred suggestion as a prompt to gener-
ate rewritten passages in a follow-up operation (described in the
following section).
To implement this feature, the few-shot prompt included sev-
eral examples containing: a passage from an email (i.e., the user’s
selected), the full email to provide context, and a suggestion for
improving the passage (the ideal response from the model). When
users press the ‘suggest how to rewrite’ button, their current selec-
tion and full email is appended to the end of the few-shot prompt
and sent to the model; the model responds with several suggested
changes to the passage for users to consider. We adapted the meta-
prompting method from [69] to allow users to optionally use a
suggestion as a precursor for the stand-alone rewriting feature
described below.
3.2.3 LLM Feature 3: Rewrite My Selection. Users can select a word,
phrase, or paragraph and provide an instruction to the AI to rewrite
the text in an arbitrary way (Figure 4); for example, ‘rewrite this:
to be shorter’. Participants in our formative study shared common
difculties with appropriate tone and style, while prior work shows
that people with dyslexia will often rely on a thesaurus [47] or
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 4: The Rewrite My Selection feature. A user can select a piece of text (1), provide a custom instruction for changing it (2),
and click ‘rewrite my selection’ (3). Several rewritten choices from the AI will populate in the right-hand panel (4). Highlighting
a choice with show a preview in the editor (5). For each choice, users can have it ’read aloud’, discard it, or apply it to replace
the original passage (6).
templates [13, 46] to achieve desired phrasing. Through custom
instructions, users with dyslexia can specify their intentions for
a passage and call upon the AI to select wording that meets that
intention. Rewritten passages from the LLM were returned as sev-
eral choices in an efort to maintain users’ autonomy over the fnal
passage.
To implement this feature, we adapted the related example prompt-
ing method from [69]: rather than anticipating every possible user
instruction and including these in the few-shot learning prompt, our
prompt only contained a collection of examples related to instruc-
tions that we anticipated would be relevant to users with dyslexia
for emails. Related examples are generally able to steer the model
to completing an unseen, user-generated task [69]. Our examples
included instructions for conciseness (‘to be simpler’), tone (‘to be
more polite’), audience (‘to be more formal’), and precision (‘to be
more clear’). Each example also contained the user’s selected text
paired with some text before and after the passage to provide con-
text, and an ideal way to rewrite that passage according to the given
instruction. When users press the ‘rewrite my selection’ button, their
current selection and surrounding text is appended to the end of
the few-shot prompt and sent to the model; the model responds
with several rewritten passages, shown to the user as choices.
3.3 Accessibility Considerations
Building a text-editing tool for users who fnd it difcult to parse and
manipulate text presents an inherent design challenge. Although
our primary goal for the LaMPost system was to demonstrate the
functionality of LLMs for writing to users with dyslexia, we recog-
nized that usability issues may impact their ability to fully evaluate
the system. To mitigate this, we made several design choices to
maximize usability for this population, and tested iterations of our
design among members of our team who identify as having dyslexia.
We used a sans serif font throughout the system because these have
Goodman, et al.
been shown to be the most readable and preferred by users with
dyslexia [50]. To further improve readability, we incorporated siz-
ing recommendations from prior work [54] suggesting a large font
(18 points or more) with line spacing near the default value (1.0
units or 120% of the font size); based on feedback from our team,
we chose an 18pt font size with 140% line spacing for the main
editor panel. To support visual referencing, we paired most buttons
with icons and added highlighting to the sentence surrounding the
insertion point cursor (visible in Figure 4-1). Finally, for users that
felt more comfortable listening to on-screen text than parsing it
visually, we used the Web Speech API 2 to include a “read aloud”
feature for the email’s body, the generated outline, and each choice
returned by the LLM.
4 LAMPOST EVALUATION
We evaluated the LaMPost prototype in a hands-on demonstration
and practical email writing exercise. Our primary goals were to
explore the potential ways that LLMs can be incorporated into
the email-writing process of writers with dyslexia, and to assess
users’ perceptions of each of LaMPost’s writing support features.
In addition, we had secondary goals of understanding users’ feel-
ings of satisfaction, self-expression, self-efcacy, autonomy, and
control while writing with LLMs, and to assess how exposure to AI
terminology may impact these feelings.
4.1 Method
4.1.1 Participants. We recruited 32 participants via a survey shared
with a large sampling pool maintained by our institution; 19 com-
pleted the study. All were based in the U.S. (N =16) and Canada
(3); all said English was their preferred language to write. The
recruiting survey asked if they had a dyslexia diagnosis, their email-
ing habits (Figure 5), and a series of demographic questions. We
2 https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
Figure 5: Email-writing habits for the evaluation’s 19 participants, showing frequency and duration when writing (a) new
emails and (b) replies. Most participants reported writing new emails and replies multiple times per day and spending 20
minutes or less on each one.
screened for experience writing emails (at least one per year) and
self-reported challenges associated with dyslexia, but we did not
require a dyslexia diagnosis to accommodate individuals without ac-
cess to formal screening procedures. Fourteen participants reported
having a formal dyslexia diagnosis and four reported discovering
their dyslexia on their own; one participant did not specify. We
aimed for balanced representation across gender and age categories,
but we attained neither due to cancellations. Four participants iden-
tifed as female, 14 as male, and one as non-binary. One participant
was 18-24 years old, seven were 25-34 years old, and 11 were 35-54
years old. Participants were compensated with a $100 gift card for
their time.
4.1.2 Procedure Overview. The evaluation procedure was split into
three parts during a 75-minute period and conducted remotely due
to the ongoing COVID-19 pandemic. First, we asked a few back-
ground questions to understand each participant’s email-writing
workfow, then provided a hands-on demonstration of the LaMPost
system.3 Second, we conducted an informal writing exercise in
which participants freely used the LaMPost system to write at least
one realistic email. Third, we asked semi-structured follow-up ques-
tions about the experience, then asked them to fll out rating scales
evaluating the system’s usefulness, consistency, and their own feel-
ings while using the system—including satisfaction, self-expression,
self-efcacy, autonomy, and control.
3 Two participants (P12, P18) were unable to access the system due to an unknown
technical issue. These participants dictated the content of the email and directions for
using each feature to the researcher via a shared screen.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Maintaining feelings of autonomy during AI-assisted writing
was a key design goal for LaMPost, following user concerns ex-
pressed during our formative study and in prior work [27]. However,
as illustrated by emerging work showing the efect that metaphor
choice can have on user perceptions [33, 36], our choices for LaM-
Post’s presentation may infuence the expectations, evaluations,
and attitudes among users. Overt AI presence might be viewed
as reducing autonomy, while obscured AI could resemble tradi-
tional computer-aided writing (e.g., spelling/grammar check). To
better understand how framing and the presence of AI metaphors
can impact perceptions of an LLM-powered writing tool among
adults with dyslexia, we segmented the system evaluation into two
between-subjects conditions:
(1) With AI metaphors (N =9): We introduced LaMPost as an
“AI-powered” email editor and used language throughout the
session to present LaMPost’s LLM as a personifed AI agent
providing writing assistance. For example, “You can get a
few suggestions from the AI for how it thinks you should write
this diferently”, “Hang on, the AI is thinking...”, and “This is
feedback from the AI on how to improve your selection.”
(2) Without AI metaphors (10): We introduced LaMPost as an
“enhanced” email editor and used language throughout the
session to obscure the presence of an AI/LLM. The examples
above in this condition: “You can get a few suggestions for how
to write this diferently”, “Hang on while the system loads...”,
and “This is feedback on how to improve your selection.”
The interface did not reference the system’s underlying LLM mech-
anism, and we left it unchanged for both conditions.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Before the session, we asked participants to prepare two ideas
for emails that they intended to write and felt comfortable sharing
with us. To ensure email ideas were of sufcient length to conduct
a substantive test of the system, we included an example: “family
newsletter on key events from 2021”. We provided assurance that
the writing exercises were only meant to provide a realistic expe-
rience of using the system, and that their performance would not
be evaluated during the session. With participants’ permission, all
evaluation sessions were recorded for later viewing and analysis.
4.1.3 Part 1: Background and Demo (25 min). The frst part of the
study was used to learn more about each participant’s current ap-
proach to email writing through a small set of interview questions
and rating scales, and to demonstrate the LaMPost system’s func-
tionality. To begin the interview, we asked participants why they
write emails, to share successes and challenges experienced when
emailing, to recall a past instance of confusion among an email
recipient, and to walk us through their email-writing process. Next,
we asked participants to rate their confdence in their emailing
ability (relates to self-efcacy [57]), ability to express themselves
and their ideas (self-expression; e.g., [38]), and their overall satis-
faction with their emails. The three ratings were predicated by a
positive statement (e.g., “I am confdent in the emails that I write.” )
and followed by a 7-point scale from “Completely disagree” to “Com-
pletely agree”. For privacy, we shared the scales with participants
via a linked form and ofered to read each statement aloud if they
desired: two participants opted for rating statements read aloud
throughout the study; one read the statement aloud themselves; 16
read silently.
After participants had completed the rating scales, the researcher
introduced the functionality of the system according to their as-
signed AI-metaphor condition in a hands-on demonstration. Partic-
ipants opened the LaMPost system in a new tab and shared their
screen with the researcher, who began by walking them through
each element of the main editor panel: input for the email’s recip-
ient and subject, space for the body text, and buttons for “undo”,
“redo”, “read aloud”, and “clear”. To make sure participants were
following along, we asked participants to move their mouse to each
element before it was introduced. To aid in demonstrating the LLM
features, we included an additional “insert sample text” button that
added a two-paragraph sample email into the body of the editor.
Next, we introduced each of the three LLM features (Identify
Main Ideas, Rewrite My Selection, Suggest Possible Changes): we
explained the feature’s intended function, asked participants to
try using it on the sample email, and explained any follow-up
functionality associated with that feature. For the Main Ideas feature,
this functionality included hovering over an idea in the structure
to highlight its associated paragraph, clicking on an idea to hear it
read aloud, and using the checkbox to generate a new subject line.
For the Rewrite feature, this included selecting diferent options to
see previews in the editor, hearing an option read aloud, deleting
undesired options, and choosing an option to replace the selected
passage. We gave a similar explanation for the options generated by
the Suggest feature, with an additional note that a chosen suggestion
could be sent as an instruction for the Rewrite feature if desired.
After introducing each feature, we asked participants to share their
Goodman, et al.
immediate thoughts about it, any concerns with it, and if they
thought they might use it when writing emails.
4.1.4 Part 2: Writing Exercises (25 min). After participants had
been introduced to LaMPost, they used the system to write emails
based on the ideas that we had requested for the session. To mini-
mize stress associated with time constraints among writers with
dyslexia [11, 30, 46] and challenges associated with learning an
unfamiliar system, we did not require participants complete their
email during the 25 minutes allotted. Instead, we told participants
to write as much as they were able in the time provided and to
freely use the LLM features as desired; if they fnished with one
email, they could try writing another.
We asked participants to “think aloud” throughout the writing
exercise, and to freely voice any questions, observations, sugges-
tions, or concerns about the system. If their actions were unclear
for any reason, the researcher prompted them for an explanation.
The researcher also provided limited answers to questions about
the system’s functionality with the appropriate language for each
participant’s AI-framing condition. To ensure that the participants
had experience with each LLM feature (Main Ideas, Rewrite, Sug-
gest), the researcher allowed them to write and use the system for at
least fve minutes before prompting them to try an unused feature,
repeating until all features had been used at least once. We logged
participants’ complete use of the system, including: all typed addi-
tions, changes, and deletions; all buttons and LLM features used;
all responses from the LLM; and all accepted LLM responses added
to the document.
4.1.5 Part 3: Follow-up Interview and Rating Scales (25 min). Fol-
lowing the email writing exercises, we discussed the experience
via semi-structured interview questions and rating scales. We be-
gan with questions to learn about participants’ overall experience,
asking them to compare their use of the system to their typical
experience writing emails, and to share anything they found eas-
ier or more difcult than usual. Next, we asked for their opinions
about the LaMPost system, including what they liked most about it,
what needed improvement, and if they had any ideas for additional
features that could assist them with writing. We used post-use rat-
ing scales to assess their overall impressions of the system and
measure the impact of the AI-framing manipulation. The ratings
targeted several concepts, including: usefulness and consistency
of each LLM feature and the system overall; satisfaction with the
system and the emails produced with it; and personal feelings of
self-efcacy [57], self-expression (e.g., [38]), autonomy, and control
(e.g., [22, 62]) while using the system. Each rating was predicated
by a positive statement and followed by a 7-point scale from “Com-
pletely disagree” to “Completely agree”. After they had privately
flled in the scales related to each concept, we asked a follow-up
question related to the concept to capture the reason for each rating.
4.1.6 Analysis. We analyzed our qualitative data following the
thematic coding process of Braun and Clarke [6] using a combined
inductive and deductive approach. Prior to the study, we produced
a set of deductive codes to categorize: existing email-writing prac-
tices; positives, negatives, and desired changes for each feature
and the overall system; and feelings of self-efcacy, self-expression,
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
autonomy, and control during and after system use. During data col-
lection, three researchers produced session notes and observations
and generated a set of inductive codes through analytic connection
across participants. We used both sets to produce a fnal codebook
containing a 3-level hierarchy: level-1 included high-level codes for
writing practices, the overall system, and each feature; level-2 in-
cluded positives, negatives, concerns for each; and level-3 included
low-level codes based on our expectations following the formative
study (deductive) and unexpected themes emerging from system
use (inductive). One researcher used the fnal codebook to indepen-
dently code transcripts for each of the 19 sessions, and resulting
themes were organized into subsections and constructed to form
our narrative. Our fnal codebook is provided as Supplementary Ma-
terial. For quantitative data, we used pre-use ratings to characterize
participants’ existing feelings about writing emails. To compare
how our between-subjects manipulation of AI framing shaped par-
ticipants’ perception of the system, we used a Mann-Whitney U test
(two-tailed) to test for signifcance in post-use ratings for usefulness,
consistency, satisfaction, self-expression, self-efcacy, autonomy,
and control.
4.2 Findings
The core goal of our study was to understand the potential ways
that LLMs can be incorporated into the email writing process of
writers with dyslexia. In the following sections, we describe their
current experience writing emails, reactions to our system and each
feature, and the (lack of) an efect with the AI-metaphors condition.
4.2.1 Current Email-Writing Experience. We briefy explored par-
ticipants’ thoughts on writing emails to understand opportunities
and challenges specifc to this genre of writing. Figure 5 shows par-
ticipants’ email-writing habits; most participants said they wrote
both new emails and replies daily (P5: “In my busy season, up to 100
a day”), though a few wrote less often (P19: “once or twice a month”).
Participants said they primarily wrote emails for work communi-
cation; some also wrote personal emails to connect with family
and friends or conduct service inquiries (e.g., P8: “doctor’s appoint-
ments”). In rating their level of agreement towards statements about
writing emails (Figure 6), participants generally felt a strong sense
of self-efcacy when emailing: the majority of participants (N =15)
felt confdent in their ability to write emails (avg.=5.05, SD=1.22).
Participants also generally felt satisfed with their emails (avg.=4.89,
SD=1.10) and that they could express themselves when emailing
(avg.=4.79, SD=1.36), although the overall agreement towards each
of these statements was more mixed.
When discussing their experiences writing emails, participants
described similar challenges and mitigation strategies as we had
heard about in our formative work and focus group, as well as
those discussed in prior work (e.g., [13, 42, 47]). Over half of partic-
ipants (N =10) said they liked to draft emails outside of an emailing
platform (e.g., Microsoft Word, pen and paper) due to “habit” (P7),
“personal preference” (P6), or “to separate it from the anxiety of having
to respond” (P2). Within their preferred platform, participants de-
scribed a common frustration of trying to convert information from
their mind into writing—or, “Putting what I’m trying to say in my
brain into words” (P13). Verbal communication was said to be easier
than writing, and six participants relied on speech-to-text tools to
ASSETS ’22, October 23–26, 2022, Athens, Greece
dictate their thoughts; others had abandoned it due to errors: “Soft-
wares that I’ve used have had a slightly higher [error] percentage than
my own [typed] inaccuracies” (P5). For typing, nine participants
mentioned predictive text, such as GMail’s Smart Compose [35],
as being particularly helpful to fnd their desired wording: “I like
the feature when it fnishes what you are thinking; when you don’t
have to type it all out” (P7). Common typed approaches included
the “word faucet” strategy (N =5) identifed during our formative
study—“I get all of my thoughts out so [...] I’ve got this blob of text”
(P10)—or bullet points (3) to map out high-level details. The re-
maining participants described unique initial drafting processes;
for example, P6 described a linear, spontaneous method: “I wing it.
[...] I start the frst sentence, then it’s like, ‘Okay, now I know what
the second sentence will be.”’ In contrast, P8 preferred to write in a
non-linear fashion: “I start with just drafting the middle paragraph
because I know that’s the one that has the most information.”
After participants had moved their ideas to writing, additional
challenges emerged during revising and proofreading. Spelling and
grammar was the most common issue, and ten participants said
they relied on a trusted spell checker, such as Grammarly, over their
email platform to catch “real word” errors (P10: “‘there’ and ‘their”’)
and other mistakes. Cutting overly verbose, or “wordy” (P16), drafts
to a succinct email was another frequent struggle (N =7). Partici-
pants recalled instances when they had used language that was
misinterpreted by the email’s recipient, most often due to a lack
of clarity (N =7). For example, P13 recalled a recipient “calling me
on the phone that evening [because] he had no idea of what I was
asking.” Tone was another common source of misunderstanding
(N =6): “They thought that I was just writing to them all mad and
pissed of, when in reality, I was just explaining myself” (P8). To
check the email before sending, some participants (4) said they
asked someone else to read the email, while others (3) said they
listened to the message via text-to-speech. Several participants (7)
mentioned they struggled to fnd enough time to adequately revise
and proofread, especially when responding to urgent emails.
4.2.2 Reactions to the LaMPost Prototype. In the following sec-
tion, we describe participants’ responses to LaMPost’s three LLM-
powered features and the overall system based on usefulness ratings
and feedback provided throughout the evaluation.
We asked participants to rate their level of agreement towards
the usefulness of LaMPost’s three LLM-powered features and the
overall system. To begin, we used a two-tailed Mann-Whitney U
test to compare usefulness ratings between the study’s two AI-
framing conditions—i.e., with (N =9) and without (10) AI-related
metaphors—but we did not fnd a signifcant diference for any of
the ratings (p>0.05). Figure 7 shows results as a whole as well as
results for each framing condition; we return to implications of the
AI-framing experiment in the Discussion (Section 5.4).
Of LaMPost’s three features, the Rewrite My Selection feature
was rated highest for usefulness on average (avg.=5.26, SD=1.26);
13 participants agreed that it was useful for writing emails4 and
nine selected it as LaMPost’s most useful feature. Participants said
the primary beneft of the Rewrite feature was its capability to fnd
satisfying and appropriate wording for an intended idea: “You’re
able to get a start on what you’re going to say and you can tweak your
4 Ratings ≥ 4 on 7-point agreement scale.
ASSETS ’22, October 23–26, 2022, Athens, Greece Goodman, et al.

Figure 6: Results of rating scales for feelings of self-efcacy, self-expression, and satisfaction during participants’ existing
email-writing process. Generally, most participants felt confdent about email-writing, but had slightly more mixed feelings
about self-expression and satisfaction.

Figure 7: Post-use rating scales among all participants, and across each of the AI framing conditions. We measured the usefulness
(left) and consistency (center) of the system and each feature, and several of their feelings about using the system (right). In
general, most found the LaMPost system useful while writing emails, appreciating the “Rewrite my selection” feature most of
all.

writing from there” (P18). P6 liked the feature because the synonyms
and alternative wording helped them to understand the meaning of
their own writing—a similar beneft has been shown in prior work
[49, 51]. While testing the Rewrite feature, participants saw further
value for mitigating problems with language precision (N =7) and
tone (4). For example, P1 said the feature would help to make his
ideas more compact—“I end up typing like three or four sentences to
explain something simply”—while P13 liked that he could “sculpt
the email to the audience” via instructions to rewrite the text to be
more “business-like” or “laid-back”. However, several participants
took issue with the feature’s implementation and functionality: the
most common concerns were inaccuracy and noise from the model
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
(N =13) and its overly numerous choices (11); four participants
selected Rewrite as LaMPost’s least useful feature as a result. We
outline these and other concerns in greater detail in the following
sections.
The Identify Main Ideas feature was rated the second high-
est on average in terms of its usefulness (avg.=5.11, SD=1.82); 13
participants agreed that it was useful to some extent, but only four
participants selected it as the most useful of LaMPost’s three core
features. These participants identifed the primary value of the fea-
ture’s visual outline as validating that their writing contained the
intended meaning for the email. For example, P4 said the feature
was the system’s “biggest selling point” because it would allow “the
ability to see, and make an independent verifcation, that I’m hitting
the points that I want to hit.” While the feature’s automatic sum-
marization was sufcient to capture the “gist” (P4, P16) of each
paragraph, four participants desired key details added to the out-
line: “It defnitely helped me recognize what I was talking about, other
than the fact that it missed time-sensitive information” (P2). Some
participants (N =8) also voiced concerns over the summarizations
feeling “sterile” (P17), or missing the emotion contained in the text.
For example, P9 felt dissatisfed when a three-sentence paragraph—
written to thank his project collaborators and share a draft of their
production—was reduced to an outline item that stated, “We have
a video.” Nine participants selected the Main Ideas feature as the
least useful overall, including four who said that they could not
see a practical use for it in their writing process: “It seems very
obvious” (P6).
Notably, the option to generate a subject line received a very
positive response. Because LaMPost created the subject line from
the Main Ideas outline returned by the LLM, we chose to pair the
subject as an optional checkbox attached to the feature and did not
ask participants to rate its usefulness separately. However, several
participants—who otherwise felt tepid about the visual Main Ideas
outline—saw value in a separate automatic subject feature: “I always
leave the subject line blank. [...] Nothing fts what I would be thinking”
(P19). Most participants thought LaMPost provided accurate subject
lines for their emails, but a few (N =3) found issues with the framing
of the subject. For example, P5 wondered why LaMPost added
the subject line “Invitation to [Name]” for his email informing the
individual that they would not be receiving an invitation: “It makes
sense, but given the context, that’s not quite right. The idea I’m trying
to get across is quite the opposite: something about bad news, or
‘Information about Upcoming Event’.”
Reactions to the Suggest Possible Changes feature were mixed
in terms of its usefulness (avg.=5.05, SD=2.03); six participants
said it was the most useful feature, and six said it was the least
useful feature. Twelve participants at least partially agreed that the
feature was useful, and they identifed the feature’s primary beneft
as support for fxing a detected, but unclear issue—i.e., “Times where
I’m like, ‘Something doesn’t sound right”’ (P15). For example, in an
email to his wife, P19 used the Suggest feature on a sentence that
“didn’t feel natural”, yielding a positive result: “Here we go, ‘Rewrite
this phrase to be more romantic’. That’s kind of what I was getting
at.” A few participants further identifed the Suggest feature’s value
in tandem with the Rewrite feature to provide possible boundaries
for the latter’s custom instructions: “It helped narrow down the
options that [Rewrite] could do” (P12). However, not all participants
ASSETS ’22, October 23–26, 2022, Athens, Greece
agreed that the Suggest feature was helpful. Two participants were
unsatisfed with feature’s limited scope, saying it, “Raises more
questions than answers” (P16), and desired further explanations of
the suggested changes: “It’s saying, ‘Rewrite this sentence to be less
wordy’. Is it telling me that the sentence is wordy? And why is it at
the top of the list? [...] And is it really wordy? There’s two sentences
in the paragraph.” (P10). Other frequent issues identifed for the
Suggest feature were similar to those for the Rewrite feature: several
participants (N =9) were concerned about inaccuracy and noise in
the results, while others (5) mentioned the “overwhelming” (P19)
quantity of choices. We describe these issues further in the following
sections.
Finally, while responses to individual features were varied, most
participants identifed at least one feature that was useful to them.
As a result, the usefulness of the LaMPost system overall was
rated fairly high following the email-writing exercise (avg.=5.53,
SD=1.38). P13 summed up the system’s utility as, “It allowed me to
validate for myself that the point I’m trying to make is actually getting
across, and it gives me the opportunity to rewrite it if it’s not.” In gen-
eral, participants were fond of LaMPost’s capabilities for automatic,
content-specifc support (P19: “like it’s an extra person helping you”)
and being able to direct that support to the scale of their choosing:
“I could do sentence-by-sentence, or paragraph-by-paragraph” (P10).
However, participants were split on how the system’s current capa-
bilities would help during day-to-day emailing. For example, P14
explained the contexts where he saw it having value: “Where I’m
having an informal dialogue with a couple teammates—I don’t need to
spend this amount of time rewriting three sentences. [...] [But] I can see
the beneft of using something like this when writing emails to a VIP,
or if I’m trying to convey a complex topic.” In total, fve participants
mentioned LaMPost as potentially increasing the time required to
write emails, but eight participants saw an overall time-saving ben-
eft: “I don’t have to rewrite it, and rewrite it, and rewrite it. [...] I get
fve minutes back of my life” (P4). Notably, one participant disagreed
that the overall system was useful for his needs: “It’s more about
potentiality versus reality. If you mean in its current state, I would
say not very useful. If you mean if I see how it could be extremely
useful, then it’s on the complete opposite end of the spectrum” (P5).
In the following sections, we outline participants’ concerns with
the system’s current state in greater detail—highlighting their con-
cerns over accuracy and choices in particular—and discuss fur-
ther improvements they requested to better assist with their email-
writing process.
4.2.3 Concerns Over Accuracy and Noise. One of the most com-
mon issues highlighted by participants throughout the evaluation
was unhelpful, inaccurate, or “nonsensical” (P8, P11) results they
received from the model. Anticipating the potential for instability
in our few-shot learning prompts when given unseen tasks [71],
we included an additional consistency rating for each feature and
the overall system.5 A two-tailed Mann-Whitney U test to compare
each consistency rating between the evaluation’s AI-framing con-
ditions did not yield a signifcant diference (p>0.05) for any rating;
Figure 7 shows results for each consistency rating as a whole, as
well as results for each framing condition.
5A 7-point rating scale measuring level of agreement toward “The [feature / system]
worked how I expected it would.”
ASSETS ’22, October 23–26, 2022, Athens, Greece
Poor results were most apparent from the Rewrite and Suggest
features, where our few-shot prompts tasked the LLM with provid-
ing new suggestions for rewriting or changing the text; generative
tasks that are more unpredictable than the relatively constrained
Main Ideas summarization. This unpredictability sometimes led to
striking “hallucinations”—factually incorrect or non-existent con-
tent generated by the LLM [20, 48]. For the Suggest feature, the
hallucinations amounted to suggested changes that were irrelevant
for the selected passage, such as “Rewrite this sentence to not use so
many commas” given to P11: “There’s not a comma in it.” Some par-
ticipants were able to tolerate a few irrelevant suggestions (P5: “It
sparks other ideas for how to rewrite things”), but others could not:
“If this is consistent, I’m going to think, ‘Why am I bothering to use
[this feature] in the frst place?”’ (P10)
In contrast, hallucinations within the Rewrite feature amounted
to seemingly relevant yet imaginary details added to the rewritten
passage. For example, when evaluating a choice containing the
phrase, “Maybe that nice patio you were telling me about,” P12 noted,
“The original text doesn’t mention a patio.” Although he removed the
option from his remaining choices, he wondered where the system
had found this information, why it had chosen to include it in the
email, and whether or not he would be able to catch more “wrong
info” in the future. In one instance, the hallucinated details carried
a deeper personal implication: P6 described feeling unsettled by an
option containing the phrase, “I’ve also posted on local Facebook,”
for an email informing a friend about schools in the area: “That’s
kind of one of those ‘gaslight’-y inaccuracies that unnerve me because
I would never be on Facebook. Although now, I’m like, ‘Wow, wait a
minute. Duh. Of course that would be a place to fnd out about this
stuf.’ [...] It’s almost like a suggestion or an assistant telling me to go
do that. That’s not necessary.”
Hallucinations weren’t the only source of inaccuracy for the
Rewrite feature. Four participants commented on choices that did
not satisfy the instruction that they had given; for example, after
the instruction “Rewrite this text: to sound more detailed” gave a
few concise results, P2 noted, “To me, this [instruction] implies more
text than I wrote.” Although the system removed repeated sugges-
tions in the fnal choices displayed to each user, three participants
mentioned seeing overly similar results: “This one is just the same
sentence that I’ve typed” (P3). Two participants noticed that some
rewritten choices had removed important details contained in the
original text. Although many participants were ultimately able to
use the Rewrite and Suggest features to fnd their desired wording or
query for helpful changes, the process of sifting through inaccurate
or irrelevant results was both “time wasteful” (P7) and cognitively
demanding (P14: “I need to put in a lot of focus”).
4.2.4 “The Paradox of Choice”. The LaMPost system included two
extremes with regard to choice: the Rewrite and Suggest features
displayed numerous choices to users (i.e., 15 responses returned
from the model, minus duplicates), while the Main Ideas and subject
line returned the model’s frst response each time. In this section,
we discuss participants’ concerns with each extreme, and their
suggestions for improvement.
Twelve participants across both the Rewrite and Suggest features
voiced concerns over the sheer volume of choices to parse. Two
Goodman, et al.
participants described this as “the paradox of choice” (P5, P6), re-
ferring to the psychological concept that large sets of choices can
feel overwhelming and lead to poor or unsatisfying fnal selections
[43, 56]. “My curiosity would just lead me into spiraling and read-
ing hundreds of these possible choices. And maybe there’d be more
self-doubt from that spiraling” (P6). Yet others found numerous op-
tions were helpful—even necessary—to fnd a correct option amidst
undesirable or inaccurate results from the model. “One that was
suggested was pretty much on point. The rest of them are either a little
of, or would require a bit of rewriting” (P4). Still, most participants
agreed that the number of options displayed at a time should be con-
siderably reduced, suggesting around “three to four options” (P19),
or “four to fve; nothing I need to scroll through” (P2). One promising
idea for fltering through a reduced number of options came from
P6: “The choices should be sorted by word count, just fve or six choices,
tops. And if I ‘X’ out one of those, it gives me a new one.”
Additional tensions around user choice emerged from the open-
ended possibilities for the Rewrite feature’s free-form instructions.
Six participants spoke of the immense value in being able to write
their own instructions, such as P10: “I don’t tend to think in the way
that commonly is spoke, so being able to put down my thoughts and
have an AI [respond]— [...] It’s like having a thesaurus for someone’s
thought." Others voiced concerns about this capability, including
doubts towards complex inputs (N =3), its handling of misspellings
(P8, P13), and potential dangers with misguided instructions: “If it’s
two in the morning and I’m emailing back and forth with support, I
might want to type something like, ‘Make it sound mad”’ (P4). These
concerns led to six participants requesting pre-written instructions
of general changes included alongside the open-ended input—and
distinct from the context-specifc instructions that could be queried
from the Suggest feature. For example, P13 said it seemed tedious
to type certain instructions: “I had to go to my spell checker on my
phone to fgure out how to spell the word ‘formal’ before I could even
use it. [I want] a drop-down of some of the most basic ones: [...] ’make
it shorter’, ‘make it longer’, ‘more professional’, ‘more casual’.”
The other choice extreme—the single result of the Main Ideas and
subject line feature—also raised concerns, although less frequently.
While pointing out that key details were missing from the visual
Main Ideas outline, two participants wondered why they were un-
able to choose the main ideas themselves. For example, P1 criticized
the feature for “missing the forest for the trees”—concentrating on
one thing while ignoring the rest—and ofered a solution: “An option
where I could highlight a section and then tell it, ‘This is what I want
you to work your identifcation around.”’ Three participants ques-
tioned why the generated subject line was returned as single result,
such as P8 after several repeated attempts: “It would be helpful to
see a list of options.”
4.2.5 Feelings About Email-Writing with LLMs. The secondary goal
of our evaluation was to understand how writing with LLMs can
impact personal feelings of satisfaction, self-expression, autonomy,
and control among writers with dyslexia. We asked participants to
rate their level of agreement towards these feelings within the con-
text of the LaMPost system (Figure 7); a two-tailed Mann-Whitney
U test to compare each consistency rating between the evaluation’s
AI-framing conditions did not yield a signifcant diference (p>0.05)
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
for any rating. In the following section, we briefy discuss each feel-
ing based on the results of the rating scales and relevant comments
provided throughout the study.
Participants generally felt satisfed with the system (avg.=5.26,
SD=1.14), but they voiced several concerns over the LLM’s accuracy
and our implementation of choices (described in the sections above).
Most participants also felt satisfed with what they wrote using the
system (avg.=5.16, SD=1.52), but this rating varied depending on
the complexity of their chosen emailing task: the majority (N =12)
were able to fnish a full email within the 25-minute writing period
(P8: “I would actually send this out to one of my friends”), while the
rest (7) were unable to complete their task due to limited time. This
variation was also apparent in their ratings of personal autonomy
(avg.=5.26, SD=1.88), where participants had ranging levels of agree-
ment about whether or not they had achieved their emailing goals.
Autonomy ratings generally aligned with email completion; e.g.,
P11: “I got my point across” vs. P6: “I still had a few more ideas to
express.” However, two participants that did not fnish writing their
emails said the system still helped them to achieve other personal
goals: “staying on task” (P16) and “keeping focus” (P2).
Participants’ sense of self-efcacy using LaMPost was fairly high
(avg.=5.26, SD=1.41), and 14 participants at least partially felt conf-
dent writing emails with the system: “It would be like a having a
proof-reader along with me” (P3). Those that felt less confdent ex-
pressed doubts towards the system’s accuracy: “I think the AI would
start breaking down if it really had to compute more and more” (P1).
Participants generally rated their sense of self-expression with LaM-
Post high (avg.=5.53, SD=1.34), particularly due to the suggestions
provided by the Rewrite feature: “I was able to look and say, ‘Well,
this is more of how I would speak,’ or, ‘This is more of how I would
want the email to sound”’ (P18). However, four participants did not
fnd LaMPost helpful for expressing their ideas, such as P8, who
gave a neutral rating: “The AI is limited on what it receives from the
user, and it cannot explain much of the email if I don’t give it the
[information]. [...] And I don’t always know what to write.”
AI-assisted writing systems can introduce complex questions
around whether the user or agent is ultimately in control over
the produced work [27]. Participants largely felt in control over
their email content while using LaMPost (avg.=5.58, SD=1.56), and
positive responses were closely associated with the ability to flter
through the model’s results and make fnal decisions over changes
to the text. For example, P15 felt a strong command over his work:
“Even though the system gave suggestions, in the end, I’m the one
that’s deciding what it’s going to say.” However, three participants
disagreed with the statement, citing limited control over the gener-
ated subject line (P5), the opaque source of the writing suggestions
(P17), and the inability to troubleshoot after undesirable results: “It
wasn’t acting on it’s own or anything, [...] but there was no actual,
refned control over each process once I put it into motion” (P1).
While not included in our rating scales, our evaluation also
elicited feelings around privacy and trust. For privacy, seven par-
ticipants voiced concerns about the system reading and storing
their personal data: “What is it doing with the information that I
put in there? Where does that go?” (P17) To protect their sensitive
information, P9 desired “the option to not have it read it” while P4
requested a clear explanation of the system’s data storage policy.
With regard to trust, a few participants highlighted the importance
ASSETS ’22, October 23–26, 2022, Athens, Greece
of building their trust in the system, especially if they were relying
on it for support in challenge areas. P13 used his own difculty
with determining writing’s tone to demonstrate the issue: “If you
wrote the same statement three diferent ways—one professional, one
casual and one romantic— [...] I would read them and they’d all look
exactly the same to me. [...] So if [the system is] saying, ‘I can take this
sentence and make it more businesslike’, I’m going to accept whatever
you’re ofering me.”
4.2.6 Additional Features and System Improvements. Participants
suggested several changes for the LaMPost system. In addition to
overall improvements to the system’s accuracy, six participants
thought the system should include personalization–either “learning
through prior conversations” (P6) or “narrow[ing] down what words
a user would use” (P19) by learning from choices over time. Some
element of personalization was needed to capture the writer’s voice,
according to P17, “Rather than it being a canned response from the
computer.” For the LaMPost interface, the most common requests
related to how each of the LLM-based features presented results
to the user. Participants desired fewer choices for the Rewrite and
Suggest features (N =12), and more choices for the Main Ideas fea-
ture (3). Five participants requested more features to track their
progress and the system’s results during each writing session, in-
cluding a “changelog” (P6), the ability to save and favorite individual
choices (P18), and creating action items from the Suggest results
(P2). Further requests for the Suggest feature included an “explana-
tion” button for each result (P10, P16) and suggestions generated
in real time (P4).
Drawing from their preferred approach of writing emails from
bullet points, four participants wondered if the system could gener-
ate the body of an email from a given outline (i.e., implementing
the Main Ideas feature in reverse): “I write the three main topics that
I want and the system writes [an email] around them” (P9). Three
participants also thought the visual Main Ideas outline would be
useful for reading other people’s emails, such as, “If I’m late to a
meeting and they’re referring to an email that I have not read” (P8).
Six participants appreciated the “Read aloud” feature (P12: “I wish
more things could just be clicked on and read back to me”), but they re-
quested options to change the speed (P14, P15) and improvements to
make the “mechanical” (P6, P12) voice sound more natural. Finally,
four participants requested improvements to spelling and grammar
detection, preferring to use existing “autocorrect” implementations
on mobile operating systems and native email clients.
4.2.7 Summary. All participants identifed one of LaMPost’s fea-
tures as being potentially useful to them when writing emails, and
the Rewrite My Selection feature was chosen as the most useful over-
all. However, accuracy and quality concerns limited the practical
usefulness of many features, despite LaMPost using today’s state of
the art models and prompting techniques. Quality concerns (and a
desire to support writer autonomy) led us to present generated text
as many choice for users to select from; however, a top-N approach
rather than a top-1 approach can be particularly challenging for
end-users with dyslexia due to the additional reading and cognitive
load challenges.
ASSETS ’22, October 23–26, 2022, Athens, Greece
5 DISCUSSION
The LaMPost system provided a testbed to explore the feasibility
and potential of LLMs as AI-powered writing support tools through
three features to meet high-level writing needs: Identify Main Ideas
(+ subject line generation), Rewrite My Suggestion, and Suggest Possi-
ble Changes. Our results indicate that LLMs with optimal output
hold potential to assist with email-writing tasks, and our evalua-
tion highlighted several promising routes for future explorations
of AI-assisted writing—including the popularity of the controllable
Rewrite and subject line features. However, our features as-is did
not surpass participants’ accuracy and quality thresholds, and as
a result, we conclude that state-of-the-art LLMs (as of early 2022)
are not yet ready to fulfll the real-world needs of writers with
dyslexia. Surprisingly, we found no efect in the use (or non-use)
of AI metaphors on perceptions of the system, nor on feelings of
autonomy, expression, and self-efcacy when writing emails. As a
whole, our fndings yield insights on the benefts and drawbacks of
using LLMs as writing support for adults with dyslexia. Below, we
discuss implications of our fndings and opportunities for future
work.
5.1 Implications for Designing Dyslexia
Support with LLMs
When evaluating LaMPost, we found that some users with dyslexia
desire personalized writing support, where the system would learn
the user’s preferred diction and tone from past writing samples and
apply it when producing rewrites and suggestions for their current
work. A personalized system may enable writers with dyslexia to
express themselves more naturally, and results flled with familiar
vocabulary and phrasing may be easier to parse. However, since
personalization requires access to the data of individual users, it
may also run in confict with their privacy choices [31]—a concern
that was expressed by participants in our work. What degree of
personalization is preferred, and how much writing data is needed
to implement it? Should the system collect this data automatically
(e.g., from an email platform’s “Sent” folder), or are users willing
to identify and share specifc writing examples themselves? Can
complex privacy policies be made more accessible through built-in
text-to-speech or writing with simple phrasing that meets visual
preferences for reading? There are opportunities for future work to
incorporate personalization in AI-powered writing tools to support
the needs of writers with dyslexia, but researchers must consider
the potential trade-ofs between these systems and the privacy
preferences of users.
Confrming and extending prior work on ownership in AI-assisted
writing [27], we found that it was important for users with dyslexia
to feel in control of AI-produced writing. To make the fnal decision
over changes to the text, participants with dyslexia in our formative
study requested writing suggestions be delivered as several options,
and we incorporated these fndings in the design of the LaMPost
system’s Rewrite and Suggest features (but overlooked choices for
the subject line). Our evaluation reiterated the importance of user
choice, but also highlighted usability issues with LaMPost’s large
list of options: making a choice—especially from similar and/or
lengthy options—was time-consuming, cognitively demanding, and
overwhelming to some users. Our fndings suggest these issues can
Goodman, et al.
be reduced by displaying fewer options, adding more variation, sort-
ing among displayed options (e.g., by length), and allowing users to
save and return to options later on. Although the Rewrite feature’s
open-ended instruction was identifed as a valuable mechanism
for expressing users’ intention, it led to further usability issues:
some users had difculty fnding the words to express their desired
changes while others encountered spelling and grammatical errors.
Additional scafolding to overcome these issues while instilling
a sense control could include pre-written instructions, dictation
options for users that prefer speaking, or probing questions to help
users narrow down their revising goals. Our work highlights possi-
ble usability pitfalls with control-enabling mechanisms, and future
work should seek to balance each users’ sense of control with their
ability to fully leverage the system’s features.
Assessing the results of an AI-enabled assistive technology can
present obstacles for primary users when the data is inherently
not accessible [25]; likewise, assessing the quality of text-based
results from an AI-enabled writing system presents challenges for
users with dyslexia. Additional assessment mechanisms may be
needed to support this population: LLMs often produce factually
incorrect “hallucinations” [20, 48], and have been shown to inherit
biases present in their training data (e.g., internet posts) [21]. We
did not encounter ofensive results in our work, but certain “non-
sensical” results prompted LaMPost users to request explanations
(e.g., question mark buttons) to assist with determining the suit-
ability of results for their writing. User feedback options to fag
incorrect or unacceptable language may ofer a promising solution
for developers to make targeted fxes. Increasing the transparency
of an automated system can foster trust among users and drive con-
tinued use [23], and further trust can be gained from performance
improvements. However, our work suggests that with sufcient
trust and confdence in the system, users with dyslexia may feel
less inclined to invest their time and energy towards analyzing
each result—increasing the risk for harm. To address this, future
work should explore potential safeguarding methods; for example,
before sending an email, a system could perform a fnal check of
all machine-written text, and ask the user to verify that this text
seems accurate.
5.2 New Datasets May Improve Support
Our LaMPost evaluation highlighted users’ limited tolerance for
inaccurate or unhelpful LLM results when writing emails. We chose
to use a pre-trained model as a generalized base for the LaMPost
prototype, and used few-shot prompting [16, 69] to demonstrate
each task from several exemplars of the task performed on generic
email text. Although our exemplars contained text pulled from real
emails that covered a range of topics, these were completed emails
that had been produced by writers of unknown lexical ability—
examples that may not have refected the characteristics of early
drafts produced by participants in our study. Further work is needed
to understand if state-of-the-art LLMs show the same limitations
when employed as email-writing support tools among the general
population, and whether or not LaMPost’s outlining, rewriting, and
suggesting features (or similar) can beneft this audience.
If we had constructed prompts with a more representative base-
line of participants’ writing, the quality of LaMPost’s results may
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
have improved during our study. To to our knowledge, however, a
public corpus of writing samples produced by adults with dyslexia
does not exist. To maximize the potential of AI-assisted writing
tools for this demographic, future work could collect samples of
writing from users with dyslexia. A small dataset could be added
to few-shot learning prompts to improve results from pre-trained
LLMs, while a sufciently large corpus could be used to train smaller,
specialized models for high performance on constrained tasks (e.g.,
summarizing main ideas). An ideal dataset should include snap-
shots from diferent times during the writing process, and give
consideration to the varied writing approaches described by par-
ticipants in our study. As an example, an early email draft from a
user that prefers to start writing by outlining key points is very
diferent from one dictated via a speech-to-text tool, and each draft
will develop diferently over time—yet users desire a tool that can
support their needs across all stages of writing. Constructing this
dataset would be a complex task, but it may be achievable through
usage logging during future prototype evaluations with a diverse
population of participants.
5.3 Lessons Learned in Few-Shot Prompting for
LLMs
Before settling on the core features in LaMPost, we experimented
with other high-level writing features identifed during formative
research as having potential beneft to writers with dyslexia; we
present them here to show the limits of few-shot prompting meth-
ods and potential design opportunities for future work.
Following positive responses toward personifed “digital writing
companions” in our formative study, we explored how LLMs could
be used for a collaborative, conversational writing experience. We
developed a prototype for emails drafted through an instant messag-
ing interface that could leverage the LaMDA model’s dialog-based
prompting format. After users supplied a short statement explain-
ing the purpose of the email, the LLM-as-chatbot would generate
probing follow-up questions to capture each detail to be included in
the message; in a separate panel, users could watch as the LLM grad-
ually constructed an email from each piece of relayed information.
However, chatting with the LLM exposed its tendency to propose
operations that it could not functionally perform (e.g., “Do you want
me to remove that information from the email?”), and we could not
determine a reliable method to limit these misleading responses.
We faced further issues with transforming the question-and-answer
chat stream into a meaningful email.
Inspired by an individual in our formative study that preferred
to write by expanding an initial bulleted list of key points (a prac-
tice also described in our evaluation), we explored a feature that
would automatically draft an email from a given outline. We cre-
ated several exemplars containing an outline of 3-7 bulleted ideas,
and a complete email containing the outline’s ideas reordered and
expanded into full sentences. While the exemplary email added
transition language and some light rephrasing, it did not include
any new information; despite this, the model tended to “hallucinate”
additional information that did not exist in the original bulleted list
(e.g., making up names of individuals when none were given; adding
unrelated information from the prompt itself). We attempted to
constrain the language of each exemplary email to more closely
ASSETS ’22, October 23–26, 2022, Athens, Greece
match that of the bulleted ideas, but this reached an extreme when
the expanded “emails” were word-for-word reproductions of the
given outline; while it successfully constrained the LLM’s imag-
ination, we decided this functionality would be of little value to
users. Future eforts could try adding empty spaces throughout the
exemplary email in place of any new language: the LLM will likely
copy this stylistic choice and users may feel comfortable flling in
specifc details themselves.
The third attempt drew from the “word faucet” approach de-
scribed in our formative study by exploring automatic ordering
and structuring for a long, disorderly text produced by, e.g., speech-
to-text dictation. We intended for the feature to improve a given
“block” of text by arranging ideas into logical order and separating
the writing into discrete paragraphs. Here, we ran into issues with
the model’s limited context window: because it accepts prompts
with a fnite length, we could include just one or two exemplars to
demonstrate the structuring task—leading to inconsistent results.
To overcome this, we tried breaking up the task via a prompt-
chaining process [67]: frst, summarizing the key ideas contained
in the “block” (creating a paragraph structure); next, connecting
each sentence with one of the key ideas (building each paragraph’s
content); and fnally, arranging the paragraph’s sentences into a
logical order. We decided this feature was not practical for our user
study: the chaining process required several minutes to complete,
and deconstructing the input in this way stripped each sentence
of meaningful surrounding context. However, as LLM context win-
dows increase in size and chaining paradigms mature, this feature
may be achievable in the future.
5.4 Limitations of Framing with and without AI
Metaphors
Our evaluation explored whether or not the presence of AI metaphors
impacted users’ perception of the tool, but we did not fnd statistical
signifcance for any rating. Based on prior work showing the efect
of diferent conceptual metaphors on perceptions of automatic sys-
tems [33, 36] and concerns over autonomy in human-AI writing
[27], we had hypothesized that a user’s knowledge that LaMPost
was AI-powered could reduce their sense of autonomy during use.
A lack of a signifcant result for any of the subjective ratings in our
AI metaphors conditions may be taken as positive outcome for pub-
lic attitudes towards AI writing support tools; knowledge of the AI
proved neutral for users’ sense of ownership over the text, and for
each of our other measured feelings. However, this result may have
also been caused by systematic error in the form of small sample
sizes for each condition (N =9 vs. 10) or study fatigue [58] (most
rating scales were administered at the end of a 75-minute video
conferencing session). We did not attempt to measure participants’
prior experience with AI; per our institution’s recruiting guidelines,
participants were not afliated with the technology industry and
we did not expect them to have deep familiarity with AI before the
study. Yet a few participants in the without AI group assumed the
system was powered by AI and described it as such without our
mentioning it; this could further indicate that our manipulation was
not designed correctly (e.g., AI was implied despite non-specifc vo-
cabulary), or it may simply refect an increasing awareness among
the public toward the capabilities of AI and the likelihood that AI
ASSETS ’22, October 23–26, 2022, Athens, Greece
is powering many new products and experiences—despite many
lay users not understanding which specifc technologies are AI-
infused [34]. Future research should attempt a deeper exploration of
the efects of presenting writing tools as AI-forward vs. obscured to
better characterize the possible impacts of each on user attitudes—
particularly end-users with dyslexia—and provide design guidelines
for of-the-shelf systems that begin to incorporate this technology.
5.5 Limitations
We identifed several issues as a result of conducting a remote
lab evaluation amidst the ongoing COVID-19 pandemic. We re-
cruited 32 participants, but only 19 completed the evaluation; this
caused an unbalanced demographic representation in our data and a
smaller-than-planned sample size for each condition of our between-
subjects AI-framing experiment. The remote nature of the study
also limited our ability to control the testing environment. An un-
known technical issues prevented two participants from accessing
the system and had to dictate the email’s content to the researcher
via screenshare. Other participants required support with setup and
troubleshooting, which reduced their time using the system during
the writing exercise, and added variation to the average duration
spent writing LaMPost. While all participants tested each of LaM-
Post’s features during the exercise, seven participants were unable
to complete their writing in the time provided. Finally, participants
were unable to experiment with LaMPost for diferent email topics
and audiences (e.g., work vs. personal), potentially skewing their
responses according to the complexity their chosen email writing
task.
6 CONCLUSION
In this paper, we introduced LaMPost, an email-writing interface
that explored the potential for large language models to power
writing support tools that address the varied needs of people with
dyslexia. LaMPost introduced AI-assisted writing features inspired
by the needs of adults with dyslexia, including rewrite my selec-
tion, identify main ideas (with subject line generation), and suggest
possible changes. Additionally, we contributed insights from an
evaluation of LaMPost with 19 adults with dyslexia. Our study iden-
tifed many promising routes for further exploration—including
the popularity of the “rewrite” and “subject line” features—but also
found that state-of-the-art LLMs (as of early 2022) may not yet have
sufcient accuracy and quality to meet the needs of writers with
dyslexia. Surprisingly, we found no efect in the use (or non-use) of
AI metaphors on users’ perceptions of the system, nor on feelings of
autonomy, expression, and self-efcacy when writing emails. Our
fndings yield further insight into the benefts and drawbacks of
using LLMs as writing support for adults with dyslexia and provide
a foundation to build upon in future research.
REFERENCES
[1] 2017. Dyslexia FAQ - Yale Dyslexia. https://dyslexia.yale.edu/dyslexia/dyslexia-
faq/.
[2] 2021. GitHub Copilot: Your AI pair programmer. https://copilot.github.com/.
[3] Damien Appert and Philippe Truillet. 2016. Impact of Word Presentation for
Dyslexia. In Proceedings of the 18th International ACM SIGACCESS Conference on
Computers and Accessibility (Reno, Nevada, USA) (ASSETS ’16). Association for
Computing Machinery, New York, NY, USA, 265–266.
Goodman, et al.
[4] Thomas Armstrong. 2011. The Power of Neurodiversity: Unleashing the Advan-
tages of Your Diferently Wired Brain (published in Hardcover as Neurodiversity).
Hachette Books.
[5] Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret
Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be
Too Big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability,
and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing
Machinery, New York, NY, USA, 610–623.
[6] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative research in psychology 3, 2 (2006), 77–101.
[7] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan,
Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jefrey Wu, Clemens Winter,
Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin
Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya
Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners.
(May 2020). arXiv:2005.14165 [cs.CL]
[8] Maggie Bruck. 1990. Word-recognition skills of adults with childhood diagnoses
of dyslexia. Dev. Psychol. 26, 3 (May 1990), 439–454.
[9] Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple
Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native
and Non-Native English Writers. (Jan. 2021). arXiv:2101.09157 [cs.HC]
[10] S Cai, S Venugopalan, K Tomanek, A Narayanan, M R Morris, and M Brenner.
2022. Context-Aware Abbreviation Expansion Using Large Language Models. In
Proceedings of NAACL.
[11] H E Cameron. 2016. Beyond cognitive defcit: the everyday lived experience of
dyslexic students at university. Disabil. Soc. 31, 2 (2016), 223–239.
[12] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-
Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson,
Alina Oprea, and Colin Rafel. 2020. Extracting Training Data from Large Lan-
guage Models. (Dec. 2020). arXiv:2012.07805 [cs.CR]
[13] Christine Carter and Edward Sellman. 2013. A view of dyslexia in context:
implications for understanding diferences in essay writing experience amongst
higher education students identifed as dyslexic. Dyslexia 19, 3 (Aug. 2013),
149–164.
[14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de
Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg
Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf,
Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail
Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter,
Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo-
tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex
Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shan-
tanu Jain, William Saunders, Christopher Hesse, Andrew N Carr, Jan Leike, Josh
Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles
Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei,
Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large
Language Models Trained on Code. (July 2021). arXiv:2107.03374 [cs.LG]
[15] Elizabeth Clark, Tal August, Sofa Serrano, Nikita Haduong, Suchin Gururangan,
and Noah A Smith. 2021. All That’s ’Human’ Is Not Gold: Evaluating Human
Evaluation of Generated Text. In Proceedings of the 59th Annual Meeting of the
Association for Computational Linguistics and the 11th International Joint Con-
ference on Natural Language Processing (Volume 1: Long Papers). Association for
Computational Linguistics, Online, 7282–7296.
[16] Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan. 2021.
Wordcraft: a Human-AI Collaborative Editor for Story Writing. (July 2021).
arXiv:2107.07430 [cs.CL]
[17] Craig Collinson and Claire Penketh. 2010. ‘Sit in the corner and don’t eat the
crayons’: postgraduates with dyslexia and the dominant ‘lexic’ discourse. Disabil.
Soc. 25, 1 (Jan. 2010), 7–19.
[18] V Connelly, E J Sumner, and A Barnett. 2014. Dyslexia and writing: Poor spelling
can interfere with good quality composition. Brookes eJournal of Learning and
Teaching 6, 2 (Dec. 2014).
[19] Vagner Figueredo de Santana, Rosimeire de Oliveira, Leonelo Dell Anhol Almeida,
and Maria Cecília Calani Baranauskas. 2012. Web accessibility and people with
dyslexia: a survey on techniques and guidelines. In Proceedings of the International
Cross-Disciplinary Conference on Web Accessibility (Lyon, France) (W4A ’12, Article
35). Association for Computing Machinery, New York, NY, USA, 1–9.
[20] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding.
(Oct. 2018). arXiv:1810.04805 [cs.CL]
[21] Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruk-
sachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. BOLD: Dataset and Met-
rics for Measuring Biases in Open-Ended Language Generation. (Jan. 2021).
arXiv:2101.11718 [cs.CL]
[22] Steven P Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L Schwartz,
and Scott R Klemmer. 2011. Parallel prototyping leads to better design results,
LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia
more divergence, and increased self-efcacy. ACM Trans. Comput.-Hum. Interact.
17, 4 (Dec. 2011), 1–24.
[23] Mary T Dzindolet, Scott A Peterson, Regina A Pomranky, Linda G Pierce, and
Hall P Beck. 2003. The role of trust in automation reliance. Int. J. Hum. Comput.
Stud. 58, 6 (June 2003), 697–718.
[24] John Everatt. 1997. The abilities and disabilities associated with adult develop-
mental dyslexia. J. Res. Read. 20, 1 (Feb. 1997), 13–21.
[25] Leah Findlater, Steven Goodman, Yuhang Zhao, Shiri Azenkot, and Margot Hanley.
2020. Fairness issues in AI systems that augment sensory abilities. SIGACCESS
Access. Comput. 125 (March 2020), 1.
[26] Katy Ilonka Gero and Lydia B Chilton. 2019. How a Stylistic, Machine-Generated
Thesaurus Impacts a Writer’s Process. In Proceedings of the 2019 on Creativity and
Cognition (San Diego, CA, USA) (C&C ’19). Association for Computing Machinery,
New York, NY, USA, 597–603.
[27] Katy Ilonka Gero and Lydia B Chilton. 2019. Metaphoria: An Algorithmic Com-
panion for Metaphor Creation. In Proceedings of the 2019 CHI Conference on
Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19, Paper 296).
Association for Computing Machinery, New York, NY, USA, 1–12.
[28] Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez:
an Interactive Poetry Generation System. In Proceedings of ACL 2017, System
Demonstrations (Vancouver, Canada). Association for Computational Linguistics,
Stroudsburg, PA, USA.
[29] Dené Granger. 2010. A tribute to my dyslexic body, as I travel in the form of a
ghost. Disabil. Stud. Q. 30, 2 (April 2010).
[30] Noel Gregg, Chris Coleman, Mark Davis, and Jill C Chalk. 2007. Timed essay
writing: implications for high-stakes tests. J. Learn. Disabil. 40, 4 (July 2007),
306–318.
[31] Foad Hamidi, Kellie Poneres, Aaron Massey, and Amy Hurst. 2018. Who Should
Have Access to My Pointing Data? Privacy Tradeofs of Adaptive Assistive
Technologies. In Proceedings of the 20th International ACM SIGACCESS Conference
on Computers and Accessibility (Galway, Ireland) (ASSETS ’18). Association for
Computing Machinery, New York, NY, USA, 203–216. https://doi.org/10.1145/
3234695.3239331
[32] IDA Editorial Contributors. 2020. Dyslexia Basics. https://dyslexiaida.org/
dyslexia-basics/.
[33] Ji-Youn Jung, Sihang Qiu, Alessandro Bozzon, and Ujwal Gadiraju. 2022. Great
Chain of Agents: The Role of Metaphorical Representation of Agents in Con-
versational Crowdsourcing. In Proceedings of the 2022 CHI Conference on Hu-
man Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Asso-
ciation for Computing Machinery, New York, NY, USA, Article 57, 22 pages.
https://doi.org/10.1145/3491102.3517653
[34] Shaun K. Kane, Anhong Guo, and Meredith Ringel Morris. 2020. Sense and
Accessibility: Understanding People with Physical Disabilities’ Experiences
with Sensing Systems. In The 22nd International ACM SIGACCESS Conference
on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Associ-
ation for Computing Machinery, New York, NY, USA, Article 42, 14 pages.
https://doi.org/10.1145/3373625.3416990
[35] Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins,
Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, and
Vivek Ramavajjala. 2016. Smart Reply: Automated Response Suggestion for Email.
In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (San Francisco, California, USA) (KDD ’16). Associa-
tion for Computing Machinery, New York, NY, USA, 955–964.
[36] Pranav Khadpe, Ranjay Krishna, Li Fei-Fei, Jefrey T Hancock, and Michael S
Bernstein. 2020. Conceptual Metaphors Impact Perceptions of Human-AI Collab-
oration. Proc. ACM Hum.-Comput. Interact. 4, CSCW2 (Oct. 2020), 1–26.
[37] Alberto Quattrini Li, Licia Sbattella, and Roberto Tedesco. 2013. PoliSpell: An
Adaptive Spellchecker and Predictor for People with Dyslexia. , 302–309 pages.
[38] Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J Cai.
2020. Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative
Models. In Proceedings of the 2020 CHI Conference on Human Factors in Computing
Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery,
New York, NY, USA, 1–13.
[39] Sônia Maria Pallaoro Moojen, Hosana Alves Gonçalves, Ana Bassôa, Ana Luiza
Navas, Graciela de Jou, and Emílio Sánchez Miguel. 2020. Adults with dyslexia:
how can they achieve academic success despite impairments in basic reading
and writing abilities? The role of text structure sensitivity as a compensatory
skill. Ann. Dyslexia 70, 1 (April 2020), 115–140.
[40] Meredith Ringel Morris. 2020. AI and accessibility. Commun. ACM 63, 6 (May
2020), 35–37.
[41] Meredith Ringel Morris, Adam Fourney, Abdullah Ali, and Laura Vonessen. 2018.
Understanding the Needs of Searchers with Dyslexia. In Proceedings of the 2018
CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada)
(CHI ’18, Paper 35). Association for Computing Machinery, New York, NY, USA,
1–12.
[42] Tilly Mortimore and W Ray Crozier. 2006. Dyslexia and difculties with study
skills in higher education. Studies in Higher Education 31, 2 (April 2006), 235–251.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[43] Antti Oulasvirta, Janne P Hukkinen, and Barry Schwartz. 2009. When more
is less: the paradox of choice in search engine use. In Proceedings of the 32nd
international ACM SIGIR conference on Research and development in information
retrieval (Boston, MA, USA) (SIGIR ’09). Association for Computing Machinery,
New York, NY, USA, 516–523.
[44] Henriette Folkmann Pedersen, Riccardo Fusaroli, Lene Louise Lauridsen, and
Rauno Parrila. 2016. Reading Processes of University Students with Dyslexia -
An Examination of the Relationship between Oral Reading and Reading Compre-
hension. Dyslexia 22, 4 (Nov. 2016), 305–321.
[45] Jennifer Pedler. 2007. Computer Correction of Real-word Spelling Errors in Dyslexic
Text. Ph. D. Dissertation. Birkbeck College, London University.
[46] Marco Pino and Luigina Mortari. 2014. The inclusion of students with dyslexia
in higher education: a systematic review using narrative synthesis. Dyslexia 20,
4 (Nov. 2014), 346–369.
[47] Geraldine A Price. 2006. Creative solutions to making the technology work: three
case studies of dyslexic writers in higher education. ALT-J 14, 1 (March 2006),
21–38.
[48] Alec Radford, Jefrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever,
and Others. 2019. Language models are unsupervised multitask learners. OpenAI
blog 1, 8 (2019), 9.
[49] Maria Rauschenberger, Ricardo Baeza-Yates, and Luz Rello. 2019. Technologies
for Dyslexia. In Web Accessibility: A Foundation for Research, Yeliz Yesilada and
Simon Harper (Eds.). Springer London, London, 603–627.
[50] Luz Rello and Ricardo Baeza-Yates. 2016. The Efect of Font Type on Screen
Readability by People with Dyslexia. ACM Trans. Access. Comput. 8, 4 (May 2016),
1–33.
[51] Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013. Simplify
or help? text simplifcation strategies for people with dyslexia. In Proceedings of
the 10th International Cross-Disciplinary Conference on Web Accessibility (Rio de
Janeiro, Brazil) (W4A ’13, Article 15). Association for Computing Machinery, New
York, NY, USA, 1–10.
[52] Luz Rello, Miguel Ballesteros, and Jefrey P Bigham. 2015. A Spellchecker for
Dyslexia. In Proceedings of the 17th International ACM SIGACCESS Conference
on Computers & Accessibility (Lisbon, Portugal) (ASSETS ’15). Association for
Computing Machinery, New York, NY, USA, 39–47.
[53] Luz Rello and Jefrey P Bigham. 2017. Good Background Colors for Readers: A
Study of People with and without Dyslexia. In Proceedings of the 19th International
ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland,
USA) (ASSETS ’17). Association for Computing Machinery, New York, NY, USA,
72–80.
[54] Luz Rello, Martin Pielot, and Mari-Carmen Marcos. 2016. Make It Big! The Efect
of Font Size and Line Spacing on Online Readability. In Proceedings of the 2016
CHI Conference on Human Factors in Computing Systems (San Jose, California,
USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA,
3637–3648.
[55] Lindsay Reynolds and Shaomei Wu. 2018. “I’m never happy with what I write”:
Challenges and strategies of people with dyslexia on social media. In Twelfth
International AAAI Conference on Web and Social Media.
[56] Barry Schwartz. 2003. The Paradox of Choice: Why More Is Less. Harper Collins.
[57] Ralf Schwarzer and Matthias Jerusalem. 1995. General self-efcacy scale. Applied
Psychology: An International Review (1995).
[58] Laure M Sharp and Joanne Frankel. 1983. Respondent Burden: A Test of Some
Common Assumptions. Public Opin. Q. 47, 1 (Jan. 1983), 36–53.
[59] S E Shaywitz, M D Escobar, B A Shaywitz, J M Fletcher, and R Makuch. 1992.
Evidence that dyslexia may represent the lower tail of a normal distribution of
reading ability. N. Engl. J. Med. 326, 3 (Jan. 1992), 145–150.
[60] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kul-
shreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang
Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali,
Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen,
Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao,
Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett,
Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel
Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben
Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Ol-
son, Alejandra Molina, Erin Hofman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar,
Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen,
Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak,
Ed Chi, and Quoc Le. 2022. LaMDA: Language Models for Dialog Applications.
(Jan. 2022). arXiv:2201.08239 [cs.CL]
[61] W Tops, C Callens, E Van Cauwenberghe, J Adriaens, and M Brysbaert. 2013.
Beyond spelling: the writing skills of students with dyslexia in higher education.
Read. Writ. 26, 5 (May 2013), 705–720.
[62] Kristen Vaccaro, Dylan Huang, Motahhare Eslami, Christian Sandvig, Kevin
Hamilton, and Karrie Karahalios. 2018. The illusion of control. In Proceedings of
the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC
Canada). ACM, New York, NY, USA.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[63] Monica van Schaik. 2021. “ACCEPT THE IDEA THAT NEURODIVERSE KIDS
EXIST”: DYSLEXIC NARRATIVES AND NEURODIVERSITY PARADIGM VISIONS.
Ph. D. Dissertation. Wilfrid Laurier University.
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you
need. In Proceedings of the 31st International Conference on Neural Information
Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates
Inc., Red Hook, NY, USA, 6000–6010.
[65] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Grifn, Jonathan Uesato,
Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac
Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba
Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean
Legassick, Geofrey Irving, and Iason Gabriel. 2021. Ethical and social risks of
harm from Language Models. (Dec. 2021). arXiv:2112.04359 [cs.CL]
[66] Shaomei Wu, Lindsay Reynolds, Xian Li, and Francisco Guzmán. 2019. Design
and Evaluation of a Social Media Writing Support Tool for People with Dyslexia.
In Proceedings of the 2019 CHI Conference on Human Factors in Computing Sys-
tems (Glasgow, Scotland Uk) (CHI ’19, Paper 516). Association for Computing
Goodman, et al.
Machinery, New York, NY, USA, 1–14.
[67] Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jef Gray, Alejandra Molina,
Michael Terry, and Carrie J Cai. 2022. PromptChainer: Chaining Large
Language Model Prompts through Visual Programming. (March 2022).
arXiv:2203.06566 [cs.HC]
[68] Jing Xu, Arthur Szlam, and Jason Weston. 2021. Beyond Goldfsh Memory:
Long-Term Open-Domain Conversation. (July 2021). arXiv:2107.07567 [cs.CL]
[69] Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft:
Story Writing With Large Language Models. In 27th International Conference on
Intelligent User Interfaces (Helsinki, Finland) (IUI ’22). Association for Computing
Machinery, New York, NY, USA, 841–852.
[70] Wojciech Zaremba, Greg Brockman, and OpenAI. 2021. OpenAI Codex. https:
//openai.com/blog/openai-codex/.
[71] Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate
Before Use: Improving Few-Shot Performance of Language Models. (Feb. 2021).
arXiv:2102.09690 [cs.CL]
 Exploring Smart Speaker User Experience for People Who
Stammer
Anna Bleakley Daniel Rough Abi Roper
University College Dublin University of Dundee City, University of London
Ireland Scotland, UK England, UK
anna.bleakley@ucdconnect.ie drough001@dundee.ac.uk Abi.Roper.1@city.ac.uk

Stephen Lindsay Martin Porcheron Minha Lee

University of Glagow Swansea University Eindhoven University of Technology
Scotland, UK Wales, UK Netherlands
Stephen.Lindsay@glasgow.ac.uk m.a.w.porcheron@swansea.ac.uk M.Lee@tue.nl

Stuart Nicholson Benjamin Cowan Leigh Clark

Dyson Institute of Engineering and University College Dublin Swansea University
Technology Ireland Wales, UK
England, UK benjamin.cowan@ucd.ie l.m.h.clark@swansea.ac.uk

ABSTRACT ACM Reference Format:

Speech-enabled smart speakers are common devices used for numer-
ous tasks in everyday life. While speech-enabled technologies are
widespread, using one’s voice as a computing modality introduces
new accessibility challenges for people with speech disfuencies
such as stammering (also known as stuttering). This paper investi-
gates the smart speaker user experiences of people who stammer
over three weeks. We conducted diary studies and semi-structured
interviews with 11 individuals to identify their daily routines, dif-
fculties with successful interactions, and strategies to overcome
these barriers. Our analysis demonstrates key factors such as de-
vice location, its afordances, and the structure of commands had a
strong impact on user experience. Participants highlighted difer-
ent linguistic strategies to try and overcome interaction difculties
and discussed the potential of using smart speakers for speech and
language therapy. We emphasise the need to further understand
the experiences of people who stammer in smart speaker design to
increase their accessibility.
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility; User studies; Empirical studies in HCI.
KEYWORDS
stammer, stutter, inclusivity, accessibility, speech interface, voice
interface, conversational user interface
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544823
Anna Bleakley, Daniel Rough, Abi Roper, Stephen Lindsay, Martin Porcheron,
Minha Lee, Stuart Nicholson, Benjamin Cowan, and Leigh Clark. 2022. Ex-
ploring Smart Speaker User Experience for People Who Stammer. In The
24th International ACM SIGACCESS Conference on Computers and Accessibil-
ity (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY,
USA, 10 pages. https://doi.org/10.1145/3517428.3544823
1 INTRODUCTION
The rapid evolution of speech technology has led to increasing
use of speech as a modality to engage with applications through
devices such as smart speakers, smartphone voice assistants, and
wearables [45]. Smart speakers in particular have become a popular
way of accessing applications and services, being owned by over 95
million people in the US alone, with 50% of device owners stating
that they use these devices on a daily basis [28]. There are concerns
about how users with diverse speech patterns like stammering
may successfully interact with speech-enabled devices [11]. These
users may fnd interaction with voice-only systems such as smart
speakers challenging in their current form, and at worst they may
be potentially excluded from smart speaker use altogether. Recent
research eforts have highlighted that these technologies create
accessibility challenges for others, including older adult users [43],
those who are deaf or hard of hearing [7], and people who are blind
or have visual impairments [2, 3].
Stammering (also known as stuttering) is classifed as a neuro-
logical condition that impacts the rhythmic fow of speech [39].
Disruptions to speech can include repetition, prolongation or block-
ing of specifc sounds and words. Blocking describes audible or
silent moments when people are unable to produce a specifc sound
or word despite intending to produce them [34]. Recent estimations
suggest 8% of children and 2% of adults experience some form of
stammer [53]. Given the global increase of smart speaker interac-
tions and speech interface use more generally, there is a risk that
people who stammer may not have successful interactions with
these devices, and their experiences are not commonly included in
the design and testing of such interactions [36]. While technical
ASSETS ’22, October 23–26, 2022, Athens, Greece
developments in improving speech recognition are ongoing (e.g.
[31, 49]), an understanding of how people who stammer engage
with these devices is still limited [11].
To address this research gap, this paper explores how 11 peo-
ple who stammer interacted with commercially available smart
speakers over a three-week period in their homes. Through diary
studies and semi-structured interviews, we investigated: 1) how
smart speakers were used and implemented into daily lives and
routines, and 2) the opportunities and challenges that emerged in
these interactions. Our fndings show common themes in partic-
ipants regularly using devices as part of their routines, yet these
were hindered by the presence of other people during interactions
leading to some participants preferring private smart speaker use.
The study also highlights the potential of using smart speakers to
practice difcult speech patterns in a controlled environment and
the possibility of using these devices to access speech and language
therapy. However, our fndings also showed a number of partici-
pants had interaction difculties with specifc sounds and words
(including device wake words), and a lack of appropriate error re-
covery measures. These were compounded by fatigue, anxiety, and
past negative experiences. Based on these fndings, we emphasise
the importance of examining speech interface experiences for peo-
ple who stammer, and including them in the design process, to
complement technical developments.
2 RELATED WORK
2.1 Speech as an Accessible Modality
Prior research on speech as a modality in human-computer inter-
action (HCI) has demonstrated its benefts in improving access to
interactive systems [12]. Through speech, people with limited hand
dexterity can gain hands-free access to mobile devices [13], and
users with motor difculties can employ voice as an alternative to
mice or touch-pad cursor control[22]. Speech interfaces, in particu-
lar those that produce synthetic speech output, can also support
people who cannot speak or have temporarily lost the ability to
speak, afording them the opportunity to engage in conversation
with others through assistive dialogue interfaces [4]. Additionally,
speech interfaces have also been shown to ofer support to deaf
users to privately engage with medical professionals when in the
presence of an interpreter [40].
Speech is a critical access modality for blind and low vision
people. A review of speech technologies demonstrated the wide
array of speech-enabled devices that can be used to improve ac-
cessibility for healthcare, educational tools, communication, and
daily living [17]. Speech-enabled screen readers such as Jobs Access
With Speech (JAWS) [51] are publicly available for people including
those who are blind or have low vision, although there is current
debate about whether these meet necessary standards for web con-
tent accessibility [58]. Speech commands have also been used to
help improve the accessibility of using visual content like images
and emojis [61].
While speech as a modality allows access to technology previ-
ously constrained to motor input devices, improving the accessibil-
ity of speech interfaces themselves is a burgeoning research area.
Human-centered design (HCD) approaches [16] have been imple-
mented to examine how people who are deaf or hard of hearing can
Bleakley et al.
better access and use smart speakers [7]. Additional visual modali-
ties, such as the use of light and customisation of smart assistant
voices to match individual needs can improve accessibility. HCD
approaches have also been used by Glasser et al. [19] and Mande
et al. [35] to understand the challenges and preferences of smart
speaker use by those who are deaf and hard-of-hearing (DHH).
Their work highlights preferences for waking up devices, as well
as the potential for sign language interaction, but also notes the
very limited regular use, or even attempts to use, smart speakers
by DHH users.
Work on older adults’ use of speech interfaces also identifes
ongoing challenges and barriers to interaction, how the systems
should speak and the potential benefts and drawbacks of anthro-
pomorphism [50]. While the spoken nature of interactions may
initially lower the barrier to using technology more generally for
older adults [43], there remain concerns about the reliability of both
the device and the information it provides. Despite the ongoing chal-
lenges, existing work highlights the importance of understanding
people’s needs and wishes as a means of improving their interac-
tions with speech interfaces.
2.2 Stammering and Speech Interfaces
While work focusing on the wider user experience is scarce, cur-
rent work on speech impairments and speech interfaces, such as
stammering or dysarthria [37], have presented technical improve-
ments in the use of automatic speech recognition (ASR). Progress
has been made by the development of appropriate datasets. For
instance, The SEP-28k dataset [31] has been shown to improve
stammering event detection performance by tagging speech events
of blocks, prolongations, sound repetitions, word repetitions, and
interjections. Other approaches include University College Lon-
don’s Archive of Stuttered Speech (UCLASS) [25], the LibriStutter
dataset of synthesised stuttered speech[29], and the FluencyBank
[44]. While improvements to speech recognition are observed when
using these datasets, they are still relatively small [52], and do not
include those systems that are not made publicly or commercially
available [31].
Adaptations for diverse speech patterns like stammering have
started to emerge in commercial smart speakers. For example, the
Amazon Alexa application has recently introduced a feature to
enable longer waiting times for Alexa when people are speaking
[6]. Google have also introduced a communication tool to improve
speech comprehension for a range of abilities [10], albeit in a beta
testing stage.
While advancements in stammering datasets and speech inter-
face design are promising, they are often restricted to closed-source
systems that limit the wider coverage of accessibility they can
provide [11]. More signifcantly, there is a lack of research where
people who stammer are included in the design and testing process
of technology that can support them [36]. As such, there is an op-
portunity to draw inspiration from existing HCD approaches, in
exploring how people who stammer use speech interfaces through
devices such as smart speakers and how they may be improved to
better accommodate such users.
Exploring Smart Speaker User Experience for People Who Stammer
2.3 Research Questions
Following similar research on home-based smart speaker deploy-
ments [42], this study deployed smart speakers in the homes of
people who stammer, with an aim to answer two questions:
(1) How do people who stammer interact with smart speakers
in their daily routines?
(2) What opportunities and challenges do smart speakers pro-
vide for people who stammer?
In doing so, our goal was to create a foundation of knowledge
in an under-explored demographic and present actionable insights
for the wider research and design community. We achieved this
through qualitative analysis using a combination of diary studies
and semi-structured interviews.
3 METHOD
3.1 Recruitment and Participants
12 participants were recruited to participate in the study through the
STAMMA charity based in the United Kingdom, via social media
channels (Facebook, Twitter), emails and the STAMMA charity
website. After an introductory session, one participant dropped out
of the study, leaving 11 participants (N=11, M=7, W=4; Mean age=33
yrs; SD=10.3 yrs). Of these, 5 were daily users of smart speakers,
1 used them multiple times a week, 2 used them once a month, 1
had only used them once or twice, and the remaining 2 had no
prior usage. 64% (N=7) of participants in the study owned a smart
speaker, with most (N=5) owning an Amazon Echo device. When
asked about previous experiences using smart speakers, almost
all participants (N=9) mentioned difculties interacting with these
devices. Stammering severity was not collected as this is typically
done using clinical instruments [47].
3.2 Diary & Optional Interview
For our home-based deployment, we asked participants to under-
take a diary study in two phases, lasting 11 days and 10 days re-
spectively. The diary study method is an established method in HCI
research, being a data gathering practice that prioritises daily logs
from participants’ perspectives [46].
In the frst phase of the diary study (D1; Mean entries: 8; SD: 2.23
entries) participants were asked the following questions to refect
on their barriers to interaction:
1) What experiences did you have interacting with your smart
speaker (Google Nest) today?
2) What difculties have you had interacting with your smart
speaker?
3) How did you overcome these difculties?
For the second phase (D2; Mean entries: 6.8; SD: 5 entries), ques-
tions were tailored to understand how these devices were now
integrated into participants’ daily lives:
1) How were your interactions with your Google Nest today?
What did you fnd easy and what did you fnd difcult?
2) If you had difculties, what aspects of your speech might have
afected this?
ASSETS ’22, October 23–26, 2022, Athens, Greece
3) Have you used new strategies to overcome difculties inter-
acting with the Google Nest? If you have, please elaborate on this
below.
For both phases, participants were encouraged to log at least one
diary entry a day in a Microsoft Form1 that was sent to them via
email between 8:00 AM and 10:00 AM. Midway through the diary
study (day 11), a set of unmoderated tasks (UT; N=10) were also
introduced as a probe to delineate diferences between diferent
task interactions. Participants were asked to complete three tasks:
1) add an event to their calendar; 2) participate in an interactive
game of their choice (e.g. quiz); and 3) add items to a shopping list.
At the end of phase two of the diary study, an optional refec-
tive interview invited participants to expand on their experiences.
In these interviews, participants were asked about their overall
experiences, difculties and strategies when interacting with the
smart speaker, and any improvements they believed could be im-
plemented in future smart speaker design. Questions were tailored
to individual participants’ activities they had logged during the
diary studies. Those that participated (either in written form or
verbally) in this Refective Interview (RF; N=9) demonstrated a mix
of cumulative negative and positive responses and they additionally
received a €25 honorarium.
3.3 Procedure
Th study received ethical clearance from Swansea University’s
ethics board. As mentioned, participants were recruited through
a charity based in the United Kingdom as well as via social media
channels (Facebook, Twitter), emails and the charity website. The
study was conducted between the months of July - September 2021.
Upon signing up for the study, and before the diary study phases
commenced, participants received a Google Nest Mini [21] to install
in their home. One participant expressed extreme difculties in
speaking, and so received a Google Nest Hub [20] that allows for
tactile and gestural input along with visual output via a display.
The researchers made this decision to ensure a more inclusive and
diverse set of experiences could be captured.
Upon commencing phase one of the diary study, participants
were given a 20-minute introductory information session via Zoom,
giving participants the opportunity to ask questions and allowing
the researchers to further explain any aspects that were unclear
about the study structure and tasks. In this introductory session,
participants were provided any help and advice they required to set
up their smart speakers, and how to complete the diary studies. Be-
yond confrming the smart speakers were set up, actual interactions
with the device were left to participants after this session. As audio
recordings were not collected as part of this study, participants
were also sent material detailing how to delete audio recordings
the smart speakers stored online, if they wished to do so. During
this session, participants also flled out a demographic question-
naire and answered questions relating to their past experiences
with smart speakers. Participants had the option to answer these
questions verbally or in written form. Only one participant declined
to answer these questions but did participate in the diary studies
(II; N=10). At the end of the study, participants were then thanked
for taking part and were debriefed as to the aims of the work. They
1 https://forms.ofce.com/
ASSETS ’22, October 23–26, 2022, Athens, Greece
were then paid an honorarium for participation and were also given
the Google Nest devices permanently as a thank you for taking part
in the study. This brand was selected due to ongoing popularity
and an expected largest smart speaker share by 2025 [27].
3.4 Data Analysis
After data collection was completed and optional interviews were
transcribed by a member of the research team, the same researcher
conducted inductive thematic analysis on the data [8] using NVivo
12.6.0 [32]. Codes were then exported to the Miro2 platform, where
an afnity diagram [24] was used to outline the initial themes along
with their defnitions. These themes were then discussed among
members of the project team in relation to overlap and divergences
between them and adjusted following refection.
4 FINDINGS
We found three distinct themes in our collected study data which
highlight the experience of smart speaker usage among people
who stammer: 1) "Using Devices in Daily Routines & Speech Ther-
apy Practice"; 2) "Interplay Between Contextual and Technological
Barriers"; 3) "Adapting to Device Limitations". These themes are
discussed in the following subsections.
4.1 Using Devices in Daily Routines & Speech
Therapy Practice
In this section, we discuss how our participants integrated the
Google Nest devices in their day-to-day lives, including how partic-
ipants envisioned the use of these devices to improve their stammer
in professional speech therapy contexts. The diferent activities un-
dertaken by participants and their frequency can be seen in Table 1.
Such usage is closely in line with recent work on typical behaviour
of smart speaker users, both generally [5], with older users [26]
and with DHH users [7].”
Table 1: Participant activities with their devices and the fre-
quency in which these activities were conducted.
Activity Frequency
Information Queries 13%
Music 15%
Weather 11%
Time 8%
News 8%
Home Appliances 4%
Travel & Commuting Queries 4%
Miscellaneous 37%
4.1.1 Device Activity & Routines. Participants’ Google Nest devices
quickly integrated into their daily lives, assisting them in morning
routines by allowing them to ask for the weather, listen to the news,
and interact with home appliances.
“It got integrated into not so much my routine but my
life in the sense that it became one of the main devices
I would use to play music.” [P10 - RI]
2 https://miro.com/
Bleakley et al.
The routines participants developed were infuenced by the loca-
tion of the device in their home. Participants that placed the device
in their home ofce [P4, P12, P11, P2] integrated the Nest into
their work routines - asking the device to assist with work-related
queries, play music during breaks, and turn on the kettle to begin
their morning routine:
“I used it through the day so like to play music while I’m
working or on my lunch break or something. I also have
to do like some maths calculation type things.” [P12 -
RI]
“And we have a plug in the kitchen and the kettle on
with working from home at the moment. So I generally
would do that three times in the morning to make a cup
of cofee because that’s my general routine.” [P2 - RI]
Some participants used the device to set alarms or prepare for
their commute by checking the weather; these behaviors were again
work related.
“I use it to set the forecast in the morning and through
the day, particularly if I was going into work that day
because I work in [a capital city] so it’s good to know,
you know if I need a coat.” [P12 - RI]
“Helping me fnd the right time to leave with trafc, or
whether I should take a coat or umbrella.” [P1 - RI]
Unsurprisingly, opportunities to interact with the smart speaker
were determined by its proximity, which thus determined the rou-
tines it became part of. Interactions with devices placed in ofce
environments complemented work routines in the home, whereas
those in communal spaces such as kitchens assisted their users
during the "leaving for work" routine.
4.1.2 Leveraging Device Afordances to Support Speech Therapy.
During the introductory interview, two participants mentioned
that they intended to use the Google Nest in a "therapeutic way"
[P10 - II] and "to essentially practice and essentially encouraging [...]
communicative experiences" [P8 - II].
“going to be about just putting myself out there in terms
of difcult sounds.” [P8 - II]
Exploring this theme in the refective interviews, we found that
the majority of participants considered smart speakers to be poten-
tially benefcial to their speech therapy. P2 suggested their place in
building speech confdence:
“if you were at a point where you were at your stammer-
ing journey where you wanted to get some confdence
[...] it could ask you to say certain words that you know
you fnd are difcult.” [P2 - RI]
P11 and P9 further suggested how smart speaker applications,
informed by speech therapists and people who stammer, could be
used to complement certain techniques taught in therapy clinics
and reduce reliance on in-person sessions:
“I would actually love that. Especially in the [local med-
ical service] it’s really hard to have a speech language
therapist.” [P11 - RI]
“there are techniques they’ll teach us and it wouldn’t
be difcult to code that and a device to assist the person
Exploring Smart Speaker User Experience for People Who Stammer
and because these devices are cheap as well it’s good
because anyone could get one.” [P9 - RI]
Indeed, interacting with the device provided opportunities for
participants to practise in the home, a "liminal space between talk-
ing to yourself and talking to a human being" [P10-RI]. Participants
described themselves as “more anxious” [P11 - RI] when talking to a
person, citing that the device “wouldn’t judge” - [P10 - RI], which in
turn made them less likely to stammer. A further advantage of inter-
acting in a controlled environment was avoiding eye contact, [P9-RI,
P8-RI] - a pressing concern during in-person communication:
“active efort to maintain eye contact when I’m having
a conversation in person, over the telephone that doesn’t
matter and obviously speaking to the device.” [P8 - RI]
Overall, participants found benefts in interacting with their
devices, citing the opportunities for such interactions to improve
aspects of their speech and practise techniques learned in therapy.
The controlled space the device aforded in their homes allowed
them to interact in contexts where certain barriers to in-person
interaction could be lowered.
4.2 Interplay Between Contextual and
Technological Barriers
Users’ varying speech patterns meant that difculties varied in
segments of the interaction. They had to overcome barriers in their
speech as well as learn the unwritten rules of interaction to create
successful utterances for the device.
4.2.1 Social Pressure, Privacy & Device Locations. Although the
Nest allowed participants to easily engage with the device in a
controlled environment, the presence of other people determined
whether users could interact without pressure. Difculties in in-
teraction were attributed to other people in the room, as users
stammered more on commands that were said fuently when others
were not around. These events occurred in communal areas such
as the kitchen or living rooms.
“like since before the study, I fnd it oddly easy to talk
to assistant devices, especially when there aren’t other
people around.” [P10 - D2]
“with the speaker because it was in my room, I stammer
less than when I’m by myself [...] we actually have a
Home Pod in our kitchen, so every time I try to use that,
I stammer a lot more even if I were to say the same
command.” [P6 - RI]
“I don’t stammer because of a device, it’s because there’s
added pressure sort of urgency from the other people.” [P9
- RI]
This has resulted in some users avoiding using the device in
communal spaces in the home, stating that it was “less of a problem”
[P6 - RI] if the command was unsuccessful. Although privacy was
related to the presence of other people, one participant expressed
concerns about how the device collected data, and the potential
impact on their stammer.
“I am a little apprehensive that the anxiety that I have
around privacy is going to afect whether I stammer
with the Google Nest.” [P10 - II]
ASSETS ’22, October 23–26, 2022, Athens, Greece
4.2.2 Mood: Stress, Fatigue & Anxiety. Participants reported that
moods such as stress [P12-D2, P8-RI, P5-D1, P4-D2], anxiety [P12-
RI, P1-D2, P6-RI], and fatigue [P12-RI, P10-D1] negatively impacted
their fuency when interacting with the device.
“when I was very tired, I defnitely did stammer more
with it.” [P10 - RI]
These factors often changed on a day-to-day basis as participants
reported "good speech days" [P12-D1, P2-D1] where they are less
likely to stammer frequently and "bad or worse speech days" [P12-
RI] where their stammer (e.g. a block) would be more prominent.
These were often clearly defned external factors, e.g., “Stress at
work making me block more.” [P4 - D2], but sometimes there was
no discernible reason:
“So it can be specifc things like if I’m stressed or if
I’m tired or if I’m quite anxious or nervous maybe, but
sometimes there isn’t a particular reason.” [P12 - RI]
4.2.3 Interaction Length & Interactivity. Participants reported in-
creased likelihood of stammering as utterance length increased;
"short and simple" [P3 - D1] interactions were more successful. We
further explored this in our diary study probe, where participants
were assigned preset tasks such as a quiz to complete.
“That was easier because I could just answer A,B, and
C. So I would just answer one word.” [P6 - RI]
Upon refection, some participants reported that they stammered
more when completing tasks in the diary study probe [P6 - RI, P10
- RI], mentioning that they often had to resort to repetition due
to the "longer phrases" [P6 - RI] that were required. Others stated
that shorter phrases and the specifcity of words added additional
"pressure" [P8-RI, P9-RI] that was not present in the other daily
interactions they had with the device.
“It does add an element of pressure because you have
to say certain words as [opposed to] on a day-to-day
basis.” [P9 - RI]
4.2.4 Dificult Sounds & Words. Participants exhibited varied dif-
fculties with diferent sounds and words, particularly the wake
word that initiates interaction with the smart speakers. For exam-
ple, one participant reported less difculty saying "Hey Google" in
comparison to "Alexa" due to the starting sound of each wake word
[P2]. Conversely, another participant had increased difculties due
to the starting sounds for "h" and "g" [P12]. This was particularly
problematic, as all commands began using the wake word.
“Wake word - getting the initial sound out to get Google
to interact with me.” [P1 - D1]
These patterns also persisted in other words in a command.
Diferences between soft, hard sounds [P6 - RI], hard stop letters
[P9 - D1], and harsh [P12 - RI] sounds were used to explain these
difculties.
“The letters O, K and G are hard starts and can be
difcult to say.” [P9 - D1]
Difculties relating to sounds also revolved around delays and
prolongations of certain letters. Although devices would sometimes
recognise users’ requests in these contexts, this experience was
inconsistent.
ASSETS ’22, October 23–26, 2022, Athens, Greece
“It seems to recognise certain elongation of words e.g
if I elongated “s” at the start of a word it mostly seems
to understand the word. Whereas if I elongated “m” at
the start of word, it wouldn’t recognise the word.” [P12
- D1]
“I think I asked it again and I could say the ‘b’, but it
did actually wait a while for me to do that.” [P6 - RI]
When commands to the device were not registered, users became
frustrated, which created negative interaction experiences, in turn
increasing the likelihood of the device not recognising them. Addi-
tional pressure was added as users repeated commands, analogous
to interaction experiences with other people.
“Whatever when you’re out and about and you actu-
ally think you’ve said that really well and you’re really
happy and you’re like “yeah”, but then the person will
say like “sorry, I didn’t hear that” [...] that kind of pres-
sure when you have to say it again and it intensifes
making it worse” [P2 - RI]
Being forced to give up after such an increasingly negative series
of interactions was described as a “debilitating” [P8-RI] experience.
4.2.5 Past Negative Experiences. Prior experiences with the Nest in-
fuenced users’ propensity to stammer. As participants began their
day with the device in the diary study, their frst communicative
experience often infuenced subsequent interactions throughout
the day. Moreover, past negative experiences with the Nest de-
termined users’ speech patterns, with some mentioning a loop
wherein thoughts about past stammering-related interactions with
the Google Nest would create difculties on words they did not
previously stammer on [P2-RI]. This happened as they prepared
themselves for the lack of recognition from the device, expecting
to be in a " difcult situation" [P10-RI].
“I think the main motivation was the awareness that
I have a fairly severe speech problem and that I must
do something to counteract it. On occasion this led to a
feedback loop of enhanced awareness causing a more
severe speech block.” [P3 - RI]
“If you’re having those negative thoughts then it would
lead to more block even on words that you wouldn’t
necessarily expect on an average day to block on.” [P2 -
RI]
4.2.6 Timing Out. The device timing out was a key issue, with
many participants mentioning that longer waiting times would
have improved their interactions with the device. However, a few
expressed that waiting times with the Nest were longer compared
to other devices [P12 - D1, P9 - RI], creating positive interaction ex-
periences. During unsuccessful interactions, users often struggled
to continue due to device interruption as they felt additional pres-
sure to produce utterances. Participants that reported blocking also
mentioned the device timing out before the following utterance or
the device ignoring a repetitive command after a block. Similarities
were reported among those that elongated sounds (also mentioned
in 4.2.4).
“When blocking, either no sound or a repetitive sound
comes out so the nest doesn’t always identify what you
Bleakley et al.
are saying or thinks you have fnished speaking.” [P12
- D2]
“I get stuck saying OK Google as I end up saying "OOO
OOOOOOOOOOOOOOOOOK Google" but it stops lis-
tening during my prolonged "O".” [P1 - RI]
Users stated that one of the main disadvantages of communicat-
ing with the devices is the lack of awareness it had to their stammer.
Unlike human-human interactions where interlocutors would ad-
just their turns to accommodate their speaking patterns, devices
would cut users of. Moreover, participants mentioned not being
able to disclose their stammer, which provides a level of ease and
comfort in human-human interactions not available with smart
speakers.
“People don’t stop listening when I can’t get a word out,
so just taking my time with people is fne.” [P1 - RI]
“if there’s a way to detect that the user stammers [...]
Then maybe it can recognize that or maybe give the
user more time to actually fnish their command to say
people that don’t stammer.” [P6 - RI]
4.2.7 Limited Scafolding for Error Recovery. Recognition was one
of the biggest barriers in interacting with the Google Nest. Often,
the device did not assist with users’ error recovery, leaving them
unsure how to correct utterances - "it just takes a long time to
realize or know what the Nest wants" [P10 - RI] and that they often
"wasn’t/not sure why" [P12 - D2, P9 - D1, P2 - D1] their initial
command was not recognized:
“Mostly frustrating - Didn’t recognise a request I made -
had to ask it 3 times . Think fuency was the same each
time so not sure why !.” [P2 - D1]
“It was just all of a sudden hit a wall and it no longer
recognizing and I thought we were friends – I thought
we were close.” [P8 - RI]
One participant [P10] explicated error recovery in detail, men-
tioning that as the device repeated the perceived utterance back
to them, they would be able to understand strategies to repair seg-
ments of the phrase. This is in contrast to situations where the
device would execute a command based on what it perceived the
user said.
“if you asked it to - where’s the best place to buy chips
and it said “here’s the best place to buy sheep”, you
would know that the word it didn’t understand correctly
was chips, you would make [...] some kind of efort to
make the Nest understand that word the next time” [P10
- RI]
4.3 Adapting to Device Limitations
As users tried to overcome these barriers, they deployed strategies
both old and new. However, issues such as error recovery limited
the strategies deployed in addition to defning the qualities of in-
teraction among users who stammer.
4.3.1 Repetition. To overcome errors they encountered with the
Nest devices, users would often repeat their commands, due to
the lack of alternative error recovery methods available. Although
Exploring Smart Speaker User Experience for People Who Stammer
at times repetition allowed them to eventually execute their com-
mands successfully, when unsuccessful, users would not be able to
delineate the cause of the error, due to lack of device feedback (see
4.2.7).
“Over time, I found ways to overcome my initial chal-
lenge of not being able to say the wake word, mainly
through repetition...” [P1 - RI]
“I suppose that’s what happened when I put something
in the calendar. I had to repeat it even though I originally
thought it went well...” [P2 - RI]
In these situations, participants either persisted until they were
successful, or simply gave up using these difcult commands. In-
deed, most users mentioned that their patterns of use remained
unchanged, often sticking to the same commands for the duration
of the diary study to avoid negative stammering experiences.
“That’s the command I know I won’t stammer and I
know I can say.” [P6 - RI]
“I am trying to be more aware of the sounds I struggle
with as I do avoid then habitually now.” [P8 - D1]
Repetition became characteristic of participants’ interactions
with the device. One participant explained how other household
members had "fewer interactions" [P3 - RI] with the device, whereas
they had difculties "being understood the frst or even second time
[P3 - RI]. Other participants described "persistence" [P1 - D1,P5 - D1]
and having "persevered" [P2 - D1] in relation to how they overcame
these challenges. As users repeated commands to the device, they
also employed methods to improve certain aspects of their speech.
These are discussed in more detail in the following subsections.
4.3.2 Planning Uterances. As users reformulated utterances, they
became more aware of the limitations in recognition that the Nest
had. They often tried to opt for other words to overcome blocking
or stammering, paralleling strategies used in in-person interaction.
“I think once I did actually I did have a way around
that command. Saying “what events are there?” or some-
thing. Because one of these words are much easier for
me to say, because events begins with ‘e’.” [P6 - RI]
However, at times, this paradoxically caused them to stammer,
due to previously formed expectations.
“I thought about day 1 more, and I defnitely had dif-
fculty with thinking that I was about to stutter and
planning for it in the way that meant I actually meant
I ended up messing up the sentence and confusing the
device....” [P10]
4.3.3 Controlled Speaking. The most commonly reported strategy
[P1, P12, P3, P9] was speaking in a "conscious" [P3], or "contrived"
[P1] manner. This meant that users were sometimes less relaxed
speaking to the device than when speaking to another person, yet
this strategy did not always result in successful interactions.
“Again, I tried to talk in a controlled and not very re-
laxed way. However it didn’t always work...” [P1 - D1]
Speaking slowly allowed participants to fuently produce words
that previously resulted in errors, and fow into words they previ-
ously stammered with [P12]. Additionally, controlling breathing or
ASSETS ’22, October 23–26, 2022, Athens, Greece
taking "deep breaths" [P1-D2] avoided issues such as timing out or
device interruption.
“I just found that speaking at a much more measured
pace never really gave it the opportunity to stop you
[...]Once I kind of, not mastered that but once I imple-
mented that it was much easier, much more success-
ful” [P8-RI]
At times, participants also mentioned increasing the volume in
which they spoke to ensure the device could pick up their com-
mands, while also moving closer to the device to increase recogni-
tion [P10 - D1 & D2, P9 - D1 & D2]. This became difcult in less
private and noisier environments in the home.
“It helps if I’m on my own when speaking to the device
the quieter the room the better as noise can cause a
direction and make speaking diferent.....” [P9 - D1]
“repeating, or going closer to the speaker....” [P10 - D1]
5 DISCUSSION
Smart speakers and other intelligent personal assistants (IPAs) are
becoming more ubiquitous with a strong presence in the home
and other environments. Yet, users that stammer have not been
explicitly accounted for in the design of these interfaces. In this
study, we highlight three distinct themes that refect the experiences
of users that stammer. We build on previous work [11] highlighting
the lack of accessibility in IPA use through smart speakers for users
with diverse speech patterns.
5.1 Navigating Difcult Interactions
While participants had some success in interacting with the smart
speakers, and even embedded them into their daily routines, there
were frequent indications that the devices’ design was insufcient
to accommodate their stammering. This included an inability to
initiate the wake word required to start interactions, difculties
with specifc sounds or phrases, and the device timing out for par-
ticipants, for example when blocking occurred.
Participants engaged in strategies in attempts to overcome dif-
culties with the devices understanding and correctly interpreting
their speech. These included repetition, planning utterances, speak-
ing in a more controlled manner, and adjusting pitch. However,
participants encountered mixed results. Repeating commands, for
example, would sometimes create additional pressure on producing
speech, which mirrored some participants’ experiences of talking to
other people. Unsuccessful use of strategies that already attempted
to navigate system recognition errors could leave participants want-
ing to avoid interacting with the devices again. Such challenges
to regular adoption are evident in related work with DHH users
[19, 35] who feel excluded from the speech-centred design process.
First impressions are vital in the subsequent adoption of a tech-
nology, and we recommend that some future ASR eforts focus on
maximising the success of those commands that are most frequently
used in this and previous work.
While using and possibly promoting these strategies may be
useful in developing more successful interactions, relying on them
rather than being able to use one’s own speech may be detrimental.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Prior work has identifed that openly stammering and being com-
fortable in speaking is a critical value that people who stammer
would like to see recognised in the tools they use [9], and it is likely
that smart speakers are no diferent.
Additionally, the variable success of the above strategies can be
compounded by how people’s stammers were manifesting at the
point of interaction. Daily life stress, anxiety, and fatigue had an
impact on people’s speech, which in turn could make successful in-
teractions with the smart speakers more problematic. Additionally,
participants felt further pressure from the presence of others, re-
sulting in less successful device interactions. Prior work has demon-
strated the reluctance of people to engage with speech interfaces in
public settings [14, 33], due to a sense of social embarrassment or
awkwardness [1, 14]. Although groups such as families may engage
collaboratively with smart speakers in home settings [41], the in-
creased potential of stammering because of others being co-located
with smart speakers may impact on this embedding of devices in
collaborative interactions and daily routines.
5.2 Fostering Error Recovery
A lack of error recovery support in the design of smart speakers
was a major issue for participants.One suggestion for improving
error recovery was to provide feedback on what the device had
interpreted from people’s speech, which does not currently occur
transparently. Similar fndings were observed in studies with L2
English speakers and IPAs, in which using visual feedback (e.g.
through smartphones) allowed people to understand the cause of
communication breakdowns [60]. While this may be one solution,
it is not always possible in hands-busy/eyes-busy scenarios where
people do not have the ability to access a screen [11]. While device
logs are available that indicate what a device has interpreted, these
are not readily accessible because they require people to actively
look up this information online via their personal accounts.
Similarly, error recovery may need to more clearly indicate why
an error has occurred. It was not always clear to participants why
difculties were emerging. Participants were sometimes left won-
dering if something about their speech pattern at the time of inter-
action was creating difculties, or whether it was the nature of the
query itself. This echoes fndings from [7] with DHH users who
were unable to understand why an error had occurred. In addition
to the visual ‘training wheels’ recommended in this previous work,
we recommend a “what went wrong?” command that would allow
any user to know why their previous command caused an error.
This would further help to bridge the “gulf of expectation” [11] that
is especially pertinent to users who stammer, who may consider
their stammer a “speech error” [36] and incorrectly attribute the
error to themselves, rather than the limitations of smart speakers.
While this lack of clarity is true irrespective of fuency [60], the
participants in this study were more inclined to blame their own
speech rather than the device’s performance, resulting in nega-
tive self-perception. There is advice on how to talk to people who
stammer (e.g. [54]), although it is unclear how this may feasibly
be transferred to smart speaker interactions [11]. While improve-
ments may need to draw on existing research on how to make
smart speaker interactions progress [15], this may not address the
Bleakley et al.
inability to conduct the initial interactions for people who stam-
mer. Improvements could also draw on web accessibility guidance,
including creating additional time to speak (as introduced with
Amazon Alexa [6]), capturing and re-using smaller parts of speech,
and the use of additional modalities when appropriate [48].
5.3 The potential for smart speakers to support
speech and language therapy
Our study highlights the opportunities that smart speakers aford
people who stammer. Over the course of the study, some partic-
ipants utilised their devices to practice speaking techniques and
sounds they know they have difculty with, to build confdence
for future social interactions. The lack of perceived judgement and
the nature of the controlled environment of interaction reduced
speaking anxiety for some participants. Additionally, there was
potential seen in using smart speakers as a means of delivering
speech and language therapy. This echoes the prior use of smart
speakers to deliver other forms of therapy or coaching in the past
[23] and applied to help build confdence for public speaking [55].
Additionally, this resonates with speech development services like
withVR [56] that provide customisable virtual reality speaking en-
vironments, where the speaker is in full control of the situation.
Similar approaches may be adoptable in commercial smart speakers
to provide cheaper and more readily available speaking practice.
The potential for using voice assistant technologies is shared by
recent work on how speech and language therapists (SLTs) use or
want to use these devices as part of their practice [30]. The authors
indicated some SLTs were already using commercial devices to help
their clients with daily tasks and improving and practising speech.
Smart speaker integration into daily routine and their use of voice
aligns them with the emerging discipline of healthcare coaching,
which often uses regular, short conversations to educate and mo-
tivate to deliver its interventions [38]. However, as discussed in
[31], existing algorithms that “smooth over” stammering support
ASR, but not detection of stammering events. Explicating such
events is necessary for both meaningful error recovery and poten-
tial speech therapy uses (discussed below). We recommend a focus
on improving the accuracy of event detection and development
of smart speaker-specifc datasets, collected through user-centred
studies such as this, or building on the work of [29] by synthesising
common stammering events in existing smart speaker datasets.
Further work should also focus on exploring the experiences
and expectations of both SLTs and people who use their services
with IPAs in this context. Such work could then inform profes-
sional standards on their use, as well as ways to improve the design
of the technology to support speech and language therapy. By
learning from the experiences of people who stammer, along with
healthcare practitioners, we can develop more inclusive design
guidelines for usage of smart speakers by people with specifc com-
municative needs, and further explore their use in SLT practice.
Indeed, work by authors of the StammerApp [36] shows support for
real-time feedback on speech from users who stammer. While the
use of augmented reality (AR) technologies such as Google Glass
were suggested for this, we see clear potential for such feedback
in smart speakers. We thus urge the third-party development of
‘Skills’ (Alexa) or ‘Actions’ (Google Assistant) that provide such
Exploring Smart Speaker User Experience for People Who Stammer
feedback to assist in identifying areas for further practice that are
not currently available.
5.4 Limitations and Future Work
This paper identifes some of the opportunities and challenges that
people who stammer face when interacting with commercially avail-
able smart speakers in their homes over a three-week period. We
emphasise that our fndings are primarily limited to experiences of
speech only smart speakers. Future research should seek to examine
how people’s experiences difer when using other speech-capable
devices in and outside of the home environment, as well as how
multi-modal forms of smart speakers impact the fndings presented.
Additionally, further work would beneft from exploring how peo-
ple who stammer in other languages interact with smart speakers,
and how stammering may also interact non-L1 English speakers’
experiences [57, 59]. While the participants in our study were all
adults, a higher number of children experience some form of stam-
mer [53]. Further research in the developing area of younger users
of smart speakers [18] may be required to create a more comprehen-
sive insight into the experiences of people who stammer, alongside
other diverse speech patterns. Further quantitative research would
also create additional insights by understanding how many people
who stammer own and use smart speakers, and how frequently
recognition errors emerge when interacting with them.
6 CONCLUSION
In this paper, we examined the use of smart speakers in the home
by people who stammer though three-week diary studies and semi-
structured interviews. We undertook our work to focus on how we
could explore the user experience of these technologies and make
them more inclusive. Our participants used smart speakers for a
range of features, placing the devices in diferent places within the
home and accordingly using them as part of diferent day-to-day
routines. Although our participants found benefts of using their
devices and highlighted the potential of using them for speech and
language therapy, external social factors such as the co-presence of
others negatively impacted their experiences. Crucial for consider-
ing smart speaker design, our fndings identifed how the length
and characteristics of interaction (including specifc sounds), time-
outs, and insufcient scafolding for error recovery, impeded device
use which our participants overcame through greater planning,
repetition, and consciously controlling their speaking. Stammering
is a diverse phenomenon that impacts people diferently, thus while
we propose improvements for smart speaker devices, these must
be applied carefully to avoid exclusion or negative side efects.
ACKNOWLEDGMENTS
We would like to thank STAMMA for their help in developing this
study. This work was supported by the Engineering and Physical
Sciences Research Council grant EP/M022722/1.
REFERENCES
[1] Ali Abdolrahmani, Ravi Kuber, and Stacy M Branham. 2018. " Siri Talks at You"
An Empirical Investigation of Voice-Activated Personal Assistant (VAPA) Usage
by Individuals Who Are Blind. In Proceedings of the 20th International ACM
SIGACCESS Conference on Computers and Accessibility. 249–258.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[2] Ali Abdolrahmani, Kevin M Storer, Antony Rishin Mukkath Roy, Ravi Kuber,
and Stacy M Branham. 2020. Blind leading the sighted: drawing design insights
from blind users towards more productivity-oriented voice interfaces. ACM
Transactions on Accessible Computing (TACCESS) 12, 4 (2020), 1–35.
[3] Dragan Ahmetovic, Cole Gleason, Chengxiong Ruan, Kris Kitani, Hironobu
Takagi, and Chieko Asakawa. 2016. NavCog: a navigational cognitive assistant for
the blind. In Proceedings of the 18th International Conference on Human-Computer
Interaction with Mobile Devices and Services. 90–99.
[4] Norman Alm, John Todman, Leona Elder, and Alan F Newell. 1993. Computer
aided conversation for severely physically impaired non-speaking people. In
Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in
Computing Systems. 236–241.
[5] Tawfq Ammari, Jofsh Kaye, Janice Y Tsai, and Frank Bentley. 2019. Music,
Search, and IoT: How People (Really) Use Voice Assistants. ACM Trans. Comput.
Hum. Interact. 26, 3 (2019), 17–1.
[6] Steven Aquino. 2021. Exclusive: Amazon Adds New Speech Set-
ting To Alexa App To Help Stutterers Finish Commands, Queries.
https://www.forbes.com/sites/stevenaquino/2021/10/05/exclusive-amazon-
adds-new-speech-setting-to-alexa-app-to-help-stutterers-fnish-commands-
queries/ Accessed 2nd April 2022.
[7] Johnna Blair and Saeed Abdullah. 2020. It Didn’t Sound Good with My Cochlear
Implants: Understanding the Challenges of Using Smart Assistants for Deaf and
Hard of Hearing Users. Proceedings of the ACM on Interactive, Mobile, Wearable
and Ubiquitous Technologies 4, 4 (2020), 1–27.
[8] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative research in psychology 3, 2 (2006), 77–101.
[9] Robin N Brewer and Vaishnav Kameswaran. 2018. Understanding the power
of control in autonomous vehicles for people with vision impairment. In Pro-
ceedings of the 20th International ACM SIGACCESS Conference on Computers and
Accessibility. 185–197.
[10] Julie Cattiau. 2021. A communication tool for people with speech impairments.
https://blog.google/outreach-initiatives/accessibility/project-relate/ Accessed
2nd April 2022.
[11] Leigh Clark, Benjamin R Cowan, Abi Roper, Stephen Lindsay, and Owen Sheers.
2020. Speech diversity and speech interfaces: Considering an inclusive future
through stammering. In Proceedings of the 2nd Conference on Conversational User
Interfaces. 1–3.
[12] Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl,
Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin Ed-
wards, and Benjamin R Cowan. 2019. The State of Speech in HCI: Trends,
Themes and Challenges. Interacting with Computers 31, 4 (09 2019), 349–371.
https://doi.org/10.1093/iwc/iwz016 arXiv:https://academic.oup.com/iwc/article-
pdf/31/4/349/33525046/iwz016.pdf
[13] Eric Corbett and Astrid Weber. 2016. What can I say? addressing user experience
challenges of a mobile voice user interface for accessibility. In Proceedings of the
18th international conference on human-computer interaction with mobile devices
and services. 72–82.
[14] Benjamin R. Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke,
Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. "What Can i Help
You with?": Infrequent Users’ Experiences of Intelligent Personal Assistants. In
Proceedings of the 19th International Conference on Human-Computer Interaction
with Mobile Devices and Services (Vienna, Austria) (MobileHCI ’17). Association
for Computing Machinery, New York, NY, USA, Article 43, 12 pages. https:
//doi.org/10.1145/3098279.3098539
[15] Joel E. Fischer, Stuart Reeves, Martin Porcheron, and Rein Ove Sikveland. 2019.
Progressivity for Voice Interface Design. In Proceedings of the 1st International
Conference on Conversational User Interfaces (Dublin, Ireland) (CUI ’19). Asso-
ciation for Computing Machinery, New York, NY, USA, Article 26, 8 pages.
https://doi.org/10.1145/3342775.3342788
[16] International Organization for Standardization. 2010. Ergonomics of Human-
system Interaction: Part 210: Human-centred Design for Interactive Systems. ISO.
https://www.iso.org/standard/77520.html
[17] Diamantino Freitas and Georgios Kouroupetroglou. 2008. Speech technologies
for blind and low vision persons. Technology and Disability 20, 2 (2008), 135–156.
[18] Radhika Garg, Hua Cui, Bo Zhang, Spencer Selingson, Martin Porcheron, Leigh
Clark, Benjamin R Cowan, and Erin Beneteau. 2022. The Last Decade of HCI
Research on Children and Voice-based Conversational Agents. In CHI Conference
on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22).
Association for Computing Machinery, New York, NY, USA, 19 pages. https:
//doi.org/10.1145/3491102.3502016
[19] Abraham Glasser, Vaishnavi Mande, and Matt Huenerfauth. 2021. Understand-
ing deaf and hard-of-hearing users’ interest in sign-language interaction with
personal-assistant devices. In Proceedings of the 18th International Web for All
Conference. 1–11.
[20] Google. 2022. Google Nest Hub (2nd Gen). https://store.google.com/gb/product/
nest_hub_2nd_gen?hl=en-GB Accessed 4th April 2022.
[21] Google. 2022. Google Nest Mini - A Smart Speaker for Any Room. https:
//store.google.com/gb/product/google_nest_mini?pli=1&hl=en-GB Accessed 4th
ASSETS ’22, October 23–26, 2022, Athens, Greece
April 2022.
[22] Susumu Harada, Jacob O Wobbrock, Jonathan Malkin, Jef A Bilmes, and James A
Landay. 2009. Longitudinal study of people learning to use continuous voice-
based cursor control. In Proceedings of the SIGCHI conference on Human Factors
in Computing Systems. 347–356.
[23] Ahmed Hassoon, Jennifer Schrack, Daniel Naiman, Dina Lansey, Yasmin Baig,
Vered Stearns, David Celentano, Seth Martin, Lawrence Appel, et al. 2018. In-
creasing physical activity amongst overweight and obese cancer survivors using
an alexa-based intelligent agent for patient coaching: protocol for the physical
activity by technology help (PATH) trial. JMIR research protocols 7, 2 (2018),
e9096.
[24] Karen Holtzblatt and Hugh Beyer. 1997. Contextual design: defning customer-
centered systems. Elsevier.
[25] Peter Howell, Stephen Davis, and Jon Bartrip. 2009. The University College
London Archive of Stuttered Speech (UCLASS). Journal of Speech, Language, and
Hearing Research 52, 2 (2009), 556.
[26] Sunyoung Kim and Abhishek Choudhury. 2021. Exploring older adults’ per-
ception and use of smart speaker-based voice assistants: A longitudinal study.
Computers in Human Behavior 124 (2021), 106914.
[27] Brett Kinsella. 2019. Loup Ventures Says 75% of U.S. Households Will
Have Smart Speakers by 2025, Google to Surpass Amazon in Market Share.
https://voicebot.ai/2019/06/18/loup-ventures-says-75-of-u-s-households-will-
have-smart-speakers-by-2025-google-to-surpass-amazon-in-market-share/
[28] Brett Kinsella. 2022. The Rise and Stall of the U.S. Smart Speaker Market – New
Report. https://voicebot.ai/2022/03/02/the-rise-and-stall-of-the-u-s-smart-
speaker-market-new-report/
[29] Tedd Kourkounakis, Amirhossein Hajavi, and Ali Etemad. 2020. FluentNet:
end-to-end detection of speech disfuency with deep learning. arXiv preprint
arXiv:2009.11394 (2020).
[30] Pranav Kulkarni, Orla Dufy, Jonathan Synnott, W George Kernohan, Roisin
McNaney, et al. 2022. Speech and Language Practitioners’ Experiences of Com-
mercially Available Voice-Assisted Technology: Web-Based Survey Study. JMIR
Rehabilitation and Assistive Technologies 9, 1 (2022), e29249.
[31] Colin Lea, Vikramjit Mitra, Aparna Joshi, Sachin Kajarekar, and Jefrey P Bigham.
2021. Sep-28k: A dataset for stuttering event detection from podcasts with people
who stutter. In ICASSP 2021-2021 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP). IEEE, 6798–6802.
[32] QSR International Pty Ltd. 2018. Nvivo (Version 12). https://www.
qsrinternational.com/nvivo-qualitative-data-analysis-software/try-nvivo
[33] Ewa Luger and Abigail Sellen. 2016. "Like Having a Really Bad PA": The
Gulf between User Expectation and Experience of Conversational Agents. In
Proceedings of the 2016 CHI Conference on Human Factors in Computing Sys-
tems. Association for Computing Machinery, New York, NY, USA, 5286–5297.
https://doi.org/10.1145/2858036.2858288
[34] Gerald A Maguire, Christopher Y Yeh, and Brandon S Ito. 2012. Overview of
the diagnosis and treatment of stuttering. Journal of Experimental & Clinical
Medicine 4, 2 (2012), 92–97.
[35] Vaishnavi Mande, Abraham Glasser, Becca Dingman, and Matt Huenerfauth. 2021.
Deaf Users’ Preferences Among Wake-Up Approaches during Sign-Language
Interaction with Personal Assistant Devices. In Extended Abstracts of the 2021
CHI Conference on Human Factors in Computing Systems. 1–6.
[36] Roisin McNaney, Christopher Bull, Lynne Mackie, Floriane Dahman, Helen
Stringer, Dan Richardson, and Daniel Welsh. 2018. StammerApp: Designing
a Mobile Application to Support Self-Refection and Goal Setting for People
Who Stammer. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems. 1–12.
[37] NHS. 2019. Dysarthria (difculty speaking). https://www.nhs.uk/conditions/
dysarthria/ Accessed 2nd April 2022.
[38] Jeanette M Olsen and Bonnie J Nesbitt. 2010. Health coaching to improve healthy
lifestyle behaviors: an integrative review. American Journal of Health Promotion
25, 1 (2010), e1–e12.
[39] World Health Organization. 2010. ICD-10 Version:2010. http://apps.who.int/
classifcations/icd10/browse/2010/en#/F98.5 Accessed 23rd Feb 2020.
[40] Anne Marie Piper and James D Hollan. 2008. Supporting medical conversations
between deaf and hearing individuals with tabletop displays. In Proceedings of
the 2008 ACM conference on Computer supported cooperative work. 147–156.
[41] Martin Porcheron, Joel E Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice
Interfaces in Everyday Life. In Proceedings of the 2018 ACM Conference on Human
Factors in Computing Systems (CHI ’18). Association for Computing Machinery,
New York, NY, USA, Article 640, 12 pages. https://doi.org/10.1145/3173574.
3174214
[42] Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019. " Phantom Friend" or"
Just a Box with Information" Personifcation and Ontological Categorization of
Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM
on Human-Computer Interaction 3, CSCW (2019), 1–21.
[43] Alisha Pradhan, Amanda Lazar, and Leah Findlater. 2020. Use of intelligent
voice assistants by older adults with low technology use. ACM Transactions on
Computer-Human Interaction (TOCHI) 27, 4 (2020), 1–27.
Bleakley et al.
[44] Nan Bernstein Ratner and Brian MacWhinney. 2018. Fluency Bank: A new
resource for fuency research and practice. Journal of fuency disorders 56 (2018),
69–80.
[45] Juniper Research. 2018. Digital Voice Assistants in Use to Triple to 8 Billion by
2023, Driven by Smart Home Devices. shorturl.at/dgoGL. Accessed 22nd Feb
2020.
[46] John Rieman. 1993. The diary study: a workplace-oriented research tool to guide
laboratory eforts. In Proceedings of the INTERACT’93 and CHI’93 conference on
Human factors in computing systems. 321–326.
[47] Glyndon Riley and Klaas Bakker. 2009. SSI-4: Stuttering severity instrument.
PRO-ED, an International Publisher.
[48] Abi Roper, Stephanie Wilson, Timothy Neate, and Jane Marshall. 2019. Speech
and Language. In Web Accessibility. Springer, 121–131.
[49] Frank Rudzicz, Aravind Kumar Namasivayam, and Talya Wolf. 2012. The TORGO
database of acoustic and articulatory speech from speakers with dysarthria.
Language Resources and Evaluation 46, 4 (2012), 523–541.
[50] Sergio Sayago, Barbara Barbosa Neves, and Benjamin R Cowan. 2019. Voice as-
sistants and older people: some open issues. In Proceedings of the 1st International
Conference on Conversational User Interfaces. 1–3.
[51] Freedom Scientifc. 2022. Jaws - Freedom Scientifc. https://www.
freedomscientifc.com/products/software/jaws Accessed 23rd Feb 2022.
[52] Shakeel Ahmad Sheikh, Md Sahidullah, Fabrice Hirsch, and Slim Ouni. 2021.
Machine Learning for Stuttering Identifcation: Review, Challenges & Future
Directions. arXiv preprint arXiv:2107.04057 (2021).
[53] Stamma. 2020. Stammer in the Population. https://stamma.org/news-features/
stammering-population Accessed 23rd Feb 2020.
[54] Stamma. 2020. Talking With Someone Who Stammers. https://stamma.org/about-
stammering/talking-someone-who-stammers Accessed 23rd Feb 2020.
[55] Jinping Wang, Hyun Yang, Ruosi Shao, Saeed Abdullah, and S Shyam Sundar.
2020. Alexa as coach: Leveraging smart speakers to build social agents that
reduce public speaking anxiety. In Proceedings of the 2020 CHI conference on
human factors in computing systems. 1–13.
[56] withVR. 2022. withVR - A safe space to speak. https://withvr.app Accessed 2nd
April 2022.
[57] Yunhan Wu, Justin Edwards, Orla Cooney, Anna Bleakley, Philip R Doyle, Leigh
Clark, Daniel Rough, and Benjamin R Cowan. 2020. Mental workload and lan-
guage production in non-native speaker IPA interaction. In Proceedings of the
2nd Conference on Conversational User Interfaces. 1–8.
[58] Yan Wu, Stephen Lindsay, Leighton Evans, Martin Porcheron, Leigh Clark, and
Rhys Jones. 2021. Enabling Digital First: A Case Study of Sight-Impaired Users in
Wales. Technical Report. Swansea University, Swansea, UK. https://cronfa.swan.
ac.uk/Record/cronfa58304/
[59] Yunhan Wu, Martin Porcheron, Philip Doyle, Justin Edwards, Daniel Rough, Orla
Cooney, Anna Bleakley, Leigh Clark, and Benjamin R. Cowan. 2022. Compar-
ing Command Construction in Native and Non-Native Speaker IPA Interaction
through Conversation Analysis. In Proceedings of the 4th International Conference
on Conversational User Interfaces.
[60] Yunhan Wu, Daniel Rough, Anna Bleakley, Justin Edwards, Orla Cooney, Philip R
Doyle, Leigh Clark, and Benjamin R Cowan. 2020. See what I’m saying? Com-
paring intelligent personal assistant use for native and non-native language
speakers. In 22nd international conference on human-computer interaction with
mobile devices and services. 1–9.
[61] Mingrui Ray Zhang, Ruolin Wang, Xuhai Xu, Qisheng Li, Ather Sharif, and Ja-
cob O Wobbrock. 2021. Voicemoji: Emoji Entry Using Voice for Visually Impaired
People. In Proceedings of the 2021 CHI Conference on Human Factors in Computing
Systems. 1–18.
 Beyond Subtitles: Captioning and Visualizing Non-speech
Sounds to Improve Accessibility of User-Generated Videos
Oliver Alonzo∗ Hijung Valentina Shin Dingzeyu Li
oa7652@rit.edu vshin@adobe.com dinli@adobe.com
Rochester Institute of Technology Adobe Research Adobe Research
Rochester, NY, USA Cambridge, MA, USA Seattle, WA, USA

ABSTRACT
Captioning provides access to sounds in audio-visual content for
people who are Deaf or Hard-of-hearing (DHH). As user-generated
content in online videos grows in prevalence, researchers have
explored using automatic speech recognition (ASR) to automate
captioning. However, defnitions of captions (as compared to sub-
titles) include non-speech sounds, which ASR typically does not
capture as it focuses on speech. Thus, we explore DHH viewers’
and hearing video creators’ perspectives on captioning non-speech
sounds in user-generated online videos using text or graphics. For-
mative interviews with 11 DHH participants informed the design
and implementation of a prototype interface for authoring text-
based and graphic captions using automatic sound event detection,
which was then evaluated with 10 hearing video creators. Our fnd- Figure 1: The non-speech sound [TRAIN BEEPS] is cap-
ings include identifying DHH viewers’ interests in having important tioned to provide access to this sound for DHH viewers in
non-speech sounds included in captions, as well as various criteria a frame from the TV show Castle. (obtained from [35]).
for sound selection and the appropriateness of text-based versus
graphic captions of non-speech sounds. Our fndings also include
hearing creators’ requirements for automatic tools to assist them 1 INTRODUCTION
in captioning non-speech sounds.
Closed captioning provides access to the audio of audio-visual con-
tent for Deaf or Hard-of-hearing (DHH) people . Using automatic
CCS CONCEPTS speech recognition (ASR) to support automatic captioning of online
• Human-centered computing → Empirical studies in HCI; videos has been increasingly explored, with several online plat-
Empirical studies in accessibility. forms supporting its use (e.g. YouTube [34]). However, auditory
content includes a richer array of sounds beyond speech such as
KEYWORDS music, background noises, or other non-speech sounds like laughter.
accessibility, audio tagging, automatic captions, non-speech sounds Thus, automatic speech recognition alone is not enough to create
complete closed captions that include such non-speech sounds [35].
ACM Reference Format: To the best of our knowledge, the use of automatic sound event
Oliver Alonzo, Hijung Valentina Shin, and Dingzeyu Li. 2022. Beyond Sub- detection when captioning user-generated videos has not been ex-
titles: Captioning and Visualizing Non-speech Sounds to Improve Accessi- plored (except for a blog post from YouTube [12]). Guidelines for
bility of User-Generated Videos. In The 24th International ACM SIGACCESS manual or professional captioning include suggestions for including
Conference on Computers and Accessibility (ASSETS ’22), October 23–26, 2022,
non-speech sounds (e.g. guidelines provided by 3Play Media1 , Web
Athens, Greece. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/
3517428.3544808
Accessibility Initiative2 , Described and Captioned Media Program3
and BBC4 ). Thus, professionally-produced captions often include
non-speech sounds. However, published research on automatic
∗ This work was conducted during a summer internship at Adobe Research. captioning for user-generated content mostly focuses on spoken
content [35]. Thus, in this work, we explore the interests and per-
spectives of DHH adults on the inclusion of non-speech sounds in
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed the context of user-generated videos. We also investigate how to
for proft or commercial advantage and that copies bear this notice and the full citation support content creators in captioning non-speech information and
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
sounds by using automatic sound event detection tools.
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
1 https://www.3playmedia.com/blog/captioning-sound-efects-in-tv-and-movies/
ASSETS ’22, October 23–26, 2022, Athens, Greece
2 https://www.w3.org/WAI/media/av/captions/
© 2022 Association for Computing Machinery.
3 https://dcmp.org/learn/602-captioning-key---sound-efects-and-music
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544808 4 https://bbc.github.io/subtitle-guidelines/#Sound-efects
ASSETS ’22, October 23–26, 2022, Athens, Greece
Guidelines for captioning non-speech sounds typically suggest
the use of verbal descriptions enclosed in brackets (Figure 1). How-
ever, considering that non-speech sounds can also be visualized
graphically, we also explore the use of graphic captions.
To this end, we conducted formative interviews with 11 DHH
participants about their experiences with online videos, captions,
and with non-speech sounds in online videos. We asked questions
about their interests in having non-speech sounds captioned in
online videos, including which sounds would be of interest, and
how those sounds should be captioned (e.g. through text-based or
graphic captions). Our results suggest interest in having important
non-speech sounds captioned in online videos. How to caption
those sounds may vary based on the type of video content, the
sound type, and the intended audience. We also identifed trade-
ofs between text-based and graphic captions.
Our formative study informed the design and implementation of
a high-fdelity prototype to caption or visualize non-speech sounds
using automatic sound event detection tools. We then conducted
a study with 10 hearing video creators, asking them about their
experiences creating online videos, adding or not adding captions,
and about their thoughts on captioning or visualizing non-speech
sounds, or how automatic tools can support them in this process.
Then, they interacted with our prototype to caption and visualize
non-speech sounds in three sample videos. Hearing video creators
wanted automatic systems to be selective about the sounds iden-
tifed and suggested that the appropriateness of graphic captions
may vary based on the video content. Accurate time stamps and
general descriptions were highlighted as important for automatic
systems to provide.
The contributions of our work include:
(1) Empirical evidence of DHH viewer’s preferences for what
non-speech sounds to caption and how to caption them,
and hearing video creators’ perspectives about what kind of
support they want for captioning non-speech sounds.
(2) Guidance for designers of captioning technologies and re-
searchers in audio-visual analysis felds investigating tech-
nologies that may support the captioning of non-speech
sounds in user-generated videos.
(3) A high-fdelity prototype for captioning or visualizing non-
speech sounds using automatic sound event detection tools.
2 BACKGROUND AND RELATED WORK
Captioning provides access to audio content as text and is often
used to provide access to auditory content for DHH people [35].
While the terms captions and subtitles are often used interchange-
ably (e.g. [10]), their purposes may be diferent: subtitles display the
language of the audio-visual content for people who do not know
that language (e.g., a non-English speaker watching an English
movie with subtitles in their language), while captions display the
audio for people who do not have access to it (e.g., some people
who are DHH). Thus, defnitions of and guidelines for captioning
include non-speech information and sounds such as speaker infor-
mation, environmental noises and sounds, sound efects, music, etc.
Captioning is legally required for content streamed on live TV in
countries such as the U.S. [16], but no such requirements exist for
Alonzo et al.
online videos. Thus there is a vast diference in the availability and
quality of captions across online platforms.
Research has explored various aspects of the user experience
(UX) and personalization of subtitles (e.g. [10, 11, 14, 15, 19]), as
well as the needs of diverse users (e.g. [1]. However, little work
has focused specifcally on the perspectives of DHH viewers and
hearing creators on the inclusion of non-speech sounds using text-
based or graphic captions.
2.1 ASR and Automatic Captions
Video authoring and communication tools, as well as online and
social media platforms, are increasingly adopting ASR to support
automatic captions (e.g. Premiere Pro5 , Zoom6 , YouTube7 , Insta-
gram8 , TikTok9 ). However, research on the use of ASR for automatic
captions is still on-going.
Prior work has examined the preferred appearances of captions
among DHH adults, fnding great diversity in preferences towards
various visual characteristics (e.g. font, background color) [4], and
preferences towards the use of punctuation in automatically gener-
ated captions [20]. Prior work has also examined how to evaluate
ASR systems among DHH adults, fnding that the literacy levels
of participants (which are diverse among DHH adults [29, 30, 32])
afect the efectiveness of metrics typically employed for caption-
ing evaluation [5]. More recent work explored which genres DHH
adults prioritize for accurate captioning, fnding that news and
politics, education, tech and science, and flm and animation were
among the top priorities [6]. Finally, semi-automatic captioning
approaches using crowd-sourcing have also been explored [31].
However, prior work on the use of ASR for automatic captions
has mostly focused on speech. To the best of our knowledge, the
only exception is a blog post from YouTube [12] which describes
the inclusion of three non-speech sounds (music, applause, and
laughter), but does not provide details about the user study support-
ing that decision. Thus, in this work, we explore the perspectives
of DHH adults on the inclusion of non-speech sounds when using
automatic tools for captioning user-generated videos.
2.2 Non-Speech Sounds in Real Life and VR
Research has explored using automatic sound event detection to
create sound-awareness applications for DHH users in physical
environments. Prior work includes investigations of which sounds
are of importance, what aspects of those sounds are of importance,
where sound awareness is more important, approaches to visualize
sounds, and the appropriateness of various devices for visualizing
sounds in physical environments (e.g. [7, 18, 22]). Findings from
prior work reveal urgent and safety-related sounds as sounds of
general interest for sound awareness [18]. However, participants’
hearing ability may afect those interests [18]. Some characteristics
of sounds, such as a sound’s source, identity and location, may be
more important than other characteristics such as its volume [18].
Recent work has also explored non-speech sounds in virtual
reality (VR). Jain et al. developed a taxonomy of sounds as a starting
5 https://helpx.adobe.com/premiere-pro/using/speech-to-text.html
6 https://blog.zoom.us/update-on-live-transcription-for-free-accounts/
7 https://googleblog.blogspot.com/2009/11/automatic-captions-in-youtube.html
8 https://about.fb.com/news/2020/09/new-automated-captions-powered-by-ai/
9 https://newsroom.tiktok.com/en-us/introducing-auto-captions
Beyond Subtitles
a b
Figure 2: Examples from prior work on diferent visualization techniques, including: a) dynamic text that varies in size to
indicate the volume of non-speech sounds [33], b) a physics-based approaches that indicate how sound would move through
physical materials [27], and c) using colors and icons for visualizing emotion in spoken content [25].
point to explore sound awareness in VR based on two dimensions
of sounds: their source and intent [21]. As the need for separate
exploration of non-speech sound awareness in VR highlights, the
fndings from one domain (e.g., physical spaces) may not necessarily
translate to others. However, to the best of our knowledge, the
inclusion of non-speech sounds in the context of authoring captions
for user-generated online videos has not been explored.
2.3 Visualizing Sounds
Sounds can be visualized in several ways that vary how they relate
to properties of the sounds or their level of semantic meaning. As
shown in Figure 2a, text with dynamic size has been explored to in-
dicate volume when captioning non-speech sounds [33]. Prior work
has also explored visualizing other non-speech information, such
as emotion, using icons and colors (Figure 2c) [17, 25]. Researchers
in [17] explored the use of icons and colors to augment existing
speech-oriented text-based captions, which was described as po-
tentially childish by participants. Furthermore, researchers have
explored visualizations as a way to supplement tactile information
when exploring the use of tactile feedback to provide non-speech
information to DHH viewers, fnding benefts when using both
combined such as improved recall of non-speech information [24].
However, in that work, researchers did not explore participants’
preferences for visualizations alone, as they were only explored
alongisde tactile feedback.
Research has also explored physics-based approaches, such as cy-
matics (Figure 2b), which imitate the movement of physical matter
among the waves produced by sounds [27]. These approaches can
be useful in settings such as audio editing or visual explorations of
characteristics of sounds. However, they are not semantically mean-
ingful. Thus, prior research on applications of sound awareness [26]
or visualization of non-speech sounds (e.g., in video games [13])
typically includes more semantically meaningful approaches such
as icons to represent a sound’s source.
GIFs and animated stickers, which can be overlaid on videos,
have been growing in popularity on social media and may provide
new ways for visualizing sounds in semantically meaningful ways
that are separate from text-based captions (see Figure 3), but to the
best of our knowledge, no work has explored their use as graphic
captions for non-speech sounds. Thus, in this work, we also explore
ASSETS ’22, October 23–26, 2022, Athens, Greece
c
the use of such graphic captions as a potential alternative to text-
based captions in the context of user-generated videos.
3 RESEARCH QUESTIONS
Based on the gaps identifed above, in this, study we investigate
the following research questions:
First, focusing on DHH viewers, we investigate:
• RQ1. What are the experiences of DHH viewers with online
videos (e.g. what type of content do they watch, what do
they like and dislike about it), and with closed captions?
• RQ2. What are DHH viewers’ perspectives on the inclusion
of text-based and graphic captions for non-speech sounds in
online videos?
Then, focusing on hearing video creators, we investigate:
• RQ3. What are the current practices of hearing creators with
captioning online videos?
• RQ4. How do hearing creators perceive the use of a pro-
totype tool based on automatic sound event detection for
captioning non-speech sounds using text-based and graphic
captions in online videos?
4 INTERVIEWS WITH DHH PARTICIPANTS
To answer RQs 1 and 2, and inform the design of a prototype used
for RQs 3 and 4 (described in section 6.1), we conducted a formative
study with DHH participants. This section describes the study’s
method, participants, and results.
4.1 Method
We conducted semi-structured interviews with DHH participants,
and our electronic appendix includes our full questionnaire. We
began by asking about their experiences watching online videos,
their experiences with captions in online videos, and their thoughts
about having non-speech sounds captioned or visualized.
Then, we prompted participants with three videos illustrating
text-based and graphic captions for non-speech sounds. We picked
three videos that varied in: their format (i.e. landscape or portrait);
their genre, which we selected from prior work identifying genre
priority for accurate captioning in online videos (more specifcally,
we selected one genre from each priority level identifed: sports,
news and entertainment) [6]; and their source, which included
YouTube (Figure 3a), BBC (Figure 3b), and TikTok (Figure 3c). Our
ASSETS ’22, October 23–26, 2022, Athens, Greece
Sports
Source: YouTube
Duration: 35 seconds
Baseline
Without non-
speech sounds
It’s good to see the smile, isn’t it?
Text-based
Non-speech
sounds
It’s good to see the smile, isn’t it? [laughter]
Graphic
Non-speech
sounds
It’s good to see the smile, isn’t it?
Figure 3: Frames from the video stimuli used in our studies. Each column illustrates the three videos, while each row illustrates
the conditions the videos were shown in. The text-based captions have been enlarged for readability.
video selection was not meant to illustrate all possible combinations
of those three aspects. Instead, we wanted some diverse combina-
tions to prompt participants to think about these diferent aspects.
Thus, our three videos consisted of a YouTube video of a sports
scene in landscape format, a BBC news video in landscape format,
and a TikTok entertainment video in portrait format. More details
about these videos are provided as part of our electronic appendix.
We showed each video to participants in three conditions (illus-
trated in Figure 3): a) without captioned non-speech sounds as a
baseline, b) using text-based captions for non-speech sounds, and c)
using graphic captions for non-speech sounds. The demonstrations
for these conditions were created manually using Premiere Pro,
using GIPHY10 stickers for the graphic captions in condition c.
We always started with our baseline (condition a), and then we
rotated the order of conditions b and c across participants. We also
rotated the order of the videos using a Latin Square schedule. After
watching all three conditions for each video, we asked participants
what they liked or disliked about each version, about their perspec-
tives on text-based and graphic captions for non-speech sounds,
and whom they think these technologies may beneft most.
4.2 Participants
We recruited 11 participants through online advertisements posted
on social media groups on Reddit and Facebook, including groups
10 https://giphy.com
Alonzo et al.
News Entertainment
Source: BBC Source: TikTok
Duration: 22 seconds Duration: 11 seconds
[trumpet playing] [beep]
targeted for DHH people in the general population, and one group
from a large university for DHH people in the U.S. Participants’
mean age was 30 (range = 18 to 47, SD = 9.68), self-identifying
as male (N = 4), female (N = 6) and non-binary (N = 1). Five self-
identifed as Deaf (Deaf, with a capital D, is usually employed to
refer to members of Deaf culture [28]) and six as Hard-of-hearing.
4.3 Procedure and Data Analysis
Participants received a consent form via e-mail before the study and
met with a researcher via Zoom. The appointments lasted 55 min-
utes on average. In the end, participants flled out a demographics
form and were compensated with a USD$30 Amazon gift card.
We conducted the interviews in English, accommodating partic-
ipants’ self-selected communication preferences, which included
using American Sign Language (ASL) interpreters (N = 3), profes-
sional captioners (N = 1), automatic captions (N = 3), text-based
chat (N = 3) or spoken English alone (N = 1).
The transcripts, obtained using Zoom’s automatic transcription,
contained 4770 words on average. In the interviews supported by
ASL interpreters, their voicing of participants’ signing was tran-
scribed. The frst author then conducted a thematic analysis using
an inductive approach to identify codes which were then grouped
into themes, as described by Braun and Clarke in [8]. We also follow
the best practices for reporting our fndings as discussed in [9].
Beyond Subtitles
5 FORMATIVE INTERVIEW RESULTS
5.1 Online Videos and Captions
The most commonly mentioned video platform was YouTube. Oth-
ers also mentioned watching videos on social media platforms,
including Instagram, Facebook, and TikTok, as well as streaming
platforms, such as Netfix, Amazon Prime, Hulu, HBO Max, and Dis-
ney+. Participants cited three main purposes for watching videos
online: entertainment, educational, and informational purposes.
5.1.1 Participants like the control they have in online videos, but
there is a lack of high-quality captions. Control was an aspect par-
ticipants liked about online videos, including control over what to
watch, the playback, volume, and the ability to turn captions on or
of. For instance, P11 said “When I’m watching videos online, I can
pause it and if I miss something, I can rewind it. I also have control of
the volume.” Some also highlighted having visual information as a
beneft, and how captioning may be benefcial in learning vocab-
ulary. Participants also highlighted how captioning technologies
for online videos, including automatic captioning, have improved
overall.
However, captioning issues were prominent among what partic-
ipants did not like about online videos, including inaccuracies in
automatic captions and the difculty of fnding well-captioned con-
tent. P3 mentioned that well-captioned content may be expensive,
citing subscription-based streaming platforms (which often caption
non-speech sounds, according to P3) as examples. Finally, partic-
ipants also mentioned the loss of captions when copying videos
(e.g., videos reposted to social media), and the lack of support for
captions in live videos (e.g., Twitch) or social media platforms.
5.1.2 Participants sought workarounds to understand uncaptioned
content, but were mindful of others’ experiences. Participants re-
ported several workarounds for uncaptioned content such as ask-
ing someone, trying to understand by re-watching the videos, or
fnding the same content in writing. Some hard-of-hearing par-
ticipants also indicated buying headphones or speakers to play
videos louder, including P11 who said “I put up a lot of volume in the
sound bar, like I spent extra money to get extra sound.”. Some Deaf
participants also mentioned reading people’s lips. However, many
participants indicated often leaving videos if one of the methods
above fails, especially if a person is clearly talking in the video. For
instance, P6 said “If I encounter a video that I need to rewatch or
go back several times, I’ll probably just move on to another thing.”
Notably, P4 and P11 mentioned not wanting to afect others’ ex-
periences by asking them to interpret the content or by making
sounds louder. For uncaptioned non-speech sounds, in addition to
the workarounds mentioned above, participants indicated recogniz-
ing missing sounds through the behavior of people in the videos, or
by noticing a gap in the story. While some mentioned instinctively
recognizing that something is missing, others also acknowledged
that they may still inadvertently miss non-speech sounds, including
P1 who said “I may miss out what happens beyond what people say
without my knowledge.”.
5.2 Text-based vs. Graphic Captions.
5.2.1 Advantages and disadvantages of graphic captions. Many par-
ticipants liked the ability of graphic captions to provide more details
ASSETS ’22, October 23–26, 2022, Athens, Greece
about a sound such as its source, location, tone of voice, volume or
emotion. Participants found graphic captions easier to understand
and see, which may make them more universal and beneft viewers
who cannot read text-based captions (e.g., children), with some
fnding the visual nature of graphic captions more “ASL friendly.”
However, participants worried about their potential for distracting,
blocking the content of the video, or taking away from the expe-
rience of watching online videos by impacting their emotional or
visual feel. Some also worried that they may feel childish and an-
noy adults as it may feel like “talking down” to them. Participants
sometimes found the graphic captions hard to distinguish from
other graphics or visual efects in videos. Finally, most participants
suggested that graphic captions should be provided on demand,
giving viewers control over whether they see them.
5.2.2 Advantages and disadvantages of text-based captions. Partici-
pants liked the familiarity of text-based captions, including their
location in standard places in the video, and the symbols used to
indicate non-speech sounds (i.e., brackets). Participants liked that
they “do the job” without disrupting the video. Participants saw
the potential for text-based captions to include aspects of sounds
that they liked that graphic captions included (e.g., changes, source,
source location, and timing). For instance, P5 mentioned that text-
based captions could use verbs to indicate changes (e.g. [applause
fading]), P10 suggested that arrows could show where the sounds
come from, and P3 suggested using additional symbols to layer
these details.
However, participants mentioned that while text-based captions
make videos accessible, they are often not interesting and lack
context. Furthermore, an emergent theme was that it may be hard
to verbally describe sounds for people who do not know those
sounds. For example, P11, who identifes as hard-of-hearing, shared
an anecdote of having to explain what "[roaring]" meant while
watching a movie with Deaf friends. While P11 indicated people
are now more familiar with brackets signifying non-speech sounds,
they still worried about the difculties of describing sounds with
words, and suggested the creation of a glossary to support this.
5.2.3 How to choose between text-based vs. graphic captions. Vari-
ous factors related to the videos, the sounds and the viewers were
discussed as criteria to select between text-based and graphic cap-
tions. First, three aspects of a video were discussed, including its
type, length, and how visually busy it is. Specifcally, participants
found graphic captions more appropriate for entertainment videos
(e.g., in social media), whereas text-based captions appeared more
ftting for more serious content. For instance, P6 said “If I’m on
social media [a] balloon is totally appropriate, right? But if I saw
that same balloon representation in like BBC, not so much.” Graphic
captions also seemed more appropriate for shorter videos, whereas
text-based captions may be better for visually busy scenes as well
as videos that have spoken content. Second, participants discussed
specifc aspects of the sounds, including the visibility of the sound’s
source (i.e,. whether it is on screen) and dynamicity (i.e., whether a
sound changes). Participants had mixed opinions about which was
more appropriate for visible sounds, but graphic captions seemed
better for dynamic sounds (e.g., an applause visualization decreas-
ing in size as its volume decreases). Finally, participants brought up
two aspects of viewers as criteria: their age and hearing abilities.
ASSETS ’22, October 23–26, 2022, Athens, Greece
P11 suggested that graphic captions seemed more appropriate for
younger audiences, while P9, for instance, suggested that text-based
captions may be better for people with slight hearing loss who, as
P11 put it, may already have an understanding of the sounds.
A few participants also suggested that graphic and text-based
captions could be used simultaneously, with one complementing
the other, which could have the beneft of standardizing the location
of the graphic captions if they were placed around the text-based
captions to provide complementary information, or as P6 said, it
would all be “located in the area that I’m expecting to see that infor-
mation”. P5 also suggested using emojis for “textifying”the captions
(i.e., making them closer to the informal language commonly used
in text messaging aided by emojis to convey emotion), which could
emphasize emotion. Other participants, in turn, expressed indifer-
ence for the format, as long as the information is provided, such as
P4 who believed that “access is the key.”.
5.3 Sounds and Information to Include
5.3.1 Caption sounds that are important to the storyline of the video,
and consider viewers’ preferences. Participants expressed strong
interest in having non-speech sounds included when captioning
online videos. Most participants conditioned their preferences for
what non-speech sounds to include on whether the sound is impor-
tant to the video, as opposed to specifc types of sounds. As P4 put
it: “specifcally if it’s relevant to the story. So for example, if a guy is
going into a house and he’s not talking but he hears something in the
house, then that’s important. But if for example, a woman is drinking
water and you can hear her swallow, that’s probably not relevant.”
This selectivity was in part motivated by the fact that there can be
too much going on, especially when a video also contains spoken
content. Other criteria related to the content of the video such as the
number of people talking and what they’re doing, and the length
of the video, are also important to consider when selecting which
sounds to visualize. Specifcally, participants mentioned that as the
number of people talking in a video increases, or if the level of
activity of activity of people in videos increases, the need for being
selective becomes more important. P9 also suggested being able to
select only the sounds that are outside of a viewer’s hearing range
such as low frequencies (e.g., bass sounds) in the case of P9. Some
participants also mentioned how some frequencies may interfere
with their hearing aids and thus it may be useful to have them cap-
tioned (and perhaps silenced). Notably, many participants talked
about the diversity within the DHH community, often referring to
it as a “spectrum,” and how this diversity may be refected in their
preferences for what sounds should be included. One participant,
for instance, mentioned not liking music because they felt music is
not compatible with Deaf culture. Thus, what counts as “important”
sounds may have a high level of user-dependency.
5.3.2 Including details about the sounds is important. When talking
about what aspects of sounds to include, participants highlighted
details such as the source, its location, and temporal changes within
the sounds. P4 also mentioned it is important to know how sounds
interact with people in videos by detailing who can hear the sounds,
for example. However, P11 also mentioned it is important to balance
the level of detail and the length of the descriptions, especially for
text-based captions, to consider slow readers and the fast-moving
Alonzo et al.
nature of captions. Finally, P2 highlighted the importance of details
especially for members of Deaf culture, for whom details and de-
scriptors are not only important, but can also be useful for learning
vocabulary and learning about sounds:
“Because in Deaf culture, we usually like descriptors. We
like to understand what vocabulary is applied to certain
sounds so that if it’s explained we can understand what’s
happening in English. So, that helps us as individuals
to understand what hearing people are hearing. The
same can be said, same could be true about Deaf people
explaining our experience to hearing people, they may
not understand so we have to be more detailed.”
5.4 Benefts of Captioning Non-Speech Sounds.
Participants commented that the inclusion of non-speech sounds
would beneft DHH people, as well as people with auditory process-
ing disorders. Some also highlighted the importance of considering
Deaf-blind people when incorporating these into videos, especially
with graphic captions. Participants also refected on how captioning
non-speech sounds can also improve the co-viewing experience of
DHH and hearing people, since DHH people would not need to ask
hearing people for clarifcation to understand content.
Ultimately, the benefts of captioning non-speech sounds in-
cluded understanding or knowing “what’s going on” in videos,
knowing the whole story, or “being at the same level” as everyone
else. Participants indicated how sounds can convey information,
set up scenes or the tone for a scene, give more depth, and provide
spatial awareness, with P6 describing sounds as "condiment" for
videos. Some participants commented how these may be realized
subconsciously and thus seem unimportant to hearing people, but
these functions are important for fully understanding “what’s going
on.” Finally, P2 highlighted that text-based captions of non-speech
sounds may help distinguish captioned videos with no dialogue
from uncaptioned videos. In other words, text-based captions of
non-speech sounds in a video without spoken content can indicate
that the video is captioned (whereas an uncaptioned video may or
may not contain spoken content). However, graphic captions alone
may not have this same function as it may be hard to distinguish
graphic captions for sounds from general visual efects in videos.
5.4.1 Tools that support captioning non-speech sounds may benefit
hearing and DHH creators, as well as DHH viewers. While many
participants indicated that tools to support including non-speech
sounds as text-based or graphic captions would support hearing
creators in making their videos more accessible, some also saw
potential utility for DHH creators who would like to accessibly
include non-speech sounds in their videos, but cannot hear those
sounds themselves. Some participants also envisioned how the
inclusion of non-speech sounds in fully automatic captioning would
be useful for viewers, as viewers often do not have the choice to
make videos accessible themselves when creators choose not do so.
6 PROTOTYPE STUDY
Based on the results from our formative interview study, we de-
signed a second study to investigate how creators can be supported
by automatic tools to caption or visualize non-speech sounds. To
this end, we developed a prototype for authoring text-based and
Beyond Subtitles
graphic captions of non-speech sounds for videos, and a video
demonstration of this prototype is included as part of our electronic
appendix. We conducted semi-structured interviews where par-
ticipants interacted with our prototype and answered questions
about their experiences before and after their interactions. This
section describes our prototype, followed by our interview method,
participants and data analysis.
6.1 Prototype
While we drew inspiration from existing captioning interfaces for
our general interface (e.g. YouTube and Adobe Premiere Pro), our
design was mostly function-driven, guided by the need to author
(section 6.1.1) and preview (section 6.1.2) the captions. Moreover,
our formative interview results also informed specifc design deci-
sions as highlighted below.
To use the prototype, users start by loading a video into the
authoring interface. Our system displays a list of non-speech sounds
in the video as would be detected by an automatic algorithm (Figure
4a), as well as a preview of the video (Figure 4g). Each occurrence of
a non-speech sound is represented by a card user-interface element,
which includes a text description of the sound, a time-stamp of
when the sound occurs, and optionally a graphic that can be used
as a graphic caption for the sound (Figure 4a). Motivated by the
fndings in our formative study, if the sound event card is generated
by an automatic algorithm, it also includes a label suggesting the
user to consider adding more details to the suggested description.
6.1.1 Authoring non-speech sound captions. Each sound event card
is added as a caption to the video. The text description is used for the
text-based caption, the graphic for the graphic caption, and the time
stamp indicates when the caption will appear in the video. Users
can add, delete or edit the sound card to customize the captions.
They can change the text description or the timestamp by simply
typing over the corresponding felds. To change the visuals, a user
can click on "Add visuals," which opens a modal search box. The
search box, powered by the GIPHY API for developers11 , supports
searching for three types of visualizations: GIFs (animated images),
Stickers (animated vector drawings or image cutouts) and Emojis
(more specifcally, stickers containing emojis). Users can specify
the location, size and rotation of the visuals through direct manip-
ulation in the video player while previewing the results. Finally,
as participants in our formative study indicated that they were
interested in important sounds, our prototype supports marking a
subset of sounds as important by toggling the star icon on the top
right corner of a sound event card.
6.1.2 Previewing the captions. Users can preview the captions in
the video player by selecting the "Descriptions" option under the
video player. Similarly, to preview graphic captions, the user selects
the "Visuals" option (Figure 4b). By default, all sound event cards
are included as captions in the preview. Users can select Starred, to
only preview captions for the starred sound events.
6.1.3 Implementation. We built the front-end of the prototype us-
ing the React JavaScript library for creating the HTML components
and functionality, and the Bootstrap library for the styling of the
11 https://developers.giphy.com
ASSETS ’22, October 23–26, 2022, Athens, Greece
components. As noted above, the visualizations provided were ob-
tained using the GIPHY API for developers, which provides options
for requesting GIFs and Stickers. The Emoji search in our prototype
was thus a search for stickers with the word “emoji” appended.
6.2 Method
The study began by asking participants about their experiences
creating online videos, including how often they create videos for
posting online and what types of videos they create. We then asked
participants about their experiences captioning videos, which in-
cluded questions about how often they caption their videos, how
they make the decision to caption or not caption their videos, as
well as what tools they use and what content they consider when
they do caption their videos. We then asked them questions related
to captioning non-speech sounds, including their current consider-
ations of including non-speech sounds in captions.
We then introduced the prototype to participants through a 3-
minute demo that introduced the video player, the sound events
tab, and all the functions of the prototype. Then, participants inter-
acted with the prototype to caption or visualize the three videos we
had used in the formative interview study, as described in section
4.1 under three conditions: 1) only adding sound events manually
(i.e. without using the Wizard-of-Oz automatic sound detection
system); 2) a Wizard-of-Oz condition using results that were cre-
ated by a member of our team to produce error-free output; and
3) using the direct output from an actual automatic sound event
detection system [23], which contained errors. This structure al-
lowed us to prompt participants for comparisons between manual
additions and the Wizard-of-Oz automatic system, but also between
human-quality output and automatic results containing errors. The
automatic results included errors such as missing sounds, misla-
beling sounds and labeling a sound multiple times. However, the
time stamps for the sounds that were detected were often close to
the actual timing of the sounds. All the labels included in these
conditions and their time stamps are provided in our electronic
appendix.
All participants interacted with the prototype in all three condi-
tions. We asked participants to do the manual-only condition frst
to ask them to imagine what they would want from an automatic
system to support them in this task. We rotated the other two condi-
tions using the Wizard-of-Oz automatic systems across participants.
We also rotated the videos using a Latin Square schedule.
We asked participants to think out loud as they captioned or
visualized non-speech sounds in the videos. Once participants were
satisfed with the results, they were exported as JSON fles contain-
ing metadata for the text-based and graphic captions participants
added so that we can replicate their work. After fnishing each
video, we asked about their experiences captioning or visualizing
the non-speech sounds for each video. After fnishing all three
videos, we asked participants to refect on their overall experiences,
and to compare the use of text-based versus graphic captions as
well as manual captioning versus automatic systems.
6.3 Participants
Participants were recruited from two sources: internal communica-
tion channels at an industry research lab, as well as special-interest
ASSETS ’22, October 23–26, 2022, Athens, Greece
a Individual Sound
Events
b Timestamps
c Text-based Caption
d Graphic Caption
Figure 4: Our prototype provides an interface to manage individual sound events (a) with their respective timestamps (b),
enabling authoring text-based captions (c) and graphic captions (d). The user can highlight (f) or remove (e) these sound events.
A a video player allows users (g) to preview the captions by selecting the appropriate setting (h) for text-based ("Descriptions")
or graphic ("Visuals") captions. The fgure shows "Visuals" selected and thus graphic captions are previewed (i).
social media groups, focused on video creation and university
groups local to one of our co-authors. Our recruitment criteria
included having experience creating videos for posting online, but
we did not specify specifc levels of experience so that we could get
diverse levels and thus a diverse set of perspectives.
We recruited ten participants, which included six who self-identifed
as male, three as female and one as non-binary. Participants’ aver-
age age was 26 (range = 18 to 29, SD = 6.6). Participants self-reported
levels of experience included developing (N = 3), competent (N =
5), advanced (N = 1) and expert (N = 1). Participants reported cap-
tioning the videos they create rarely (N = 3), occasionally (N = 2),
often (N = 4) and always (N = 1).
6.4 Procedure
Participants were contacted via e-mail to receive a consent form
ahead of the study. Participants then met with a researcher via
Zoom for a 60-minute appointment. At the end of the appointment,
participants flled out a demographics form and were compensated
with a $20 Amazon gift card for their participation.
6.5 Data Analysis
The video recordings of the interviews were an average of 55 min-
utes long. Zoom’s automatic transcription system was used through-
out all the interviews to obtain transcripts of these videos, obtaining
an average length of 7680 words per transcript. As we only collected
open-ended comments from participants, our analysis was similar
to our formative user study. Namely, the data from this study was
analyzed by the frst author using a thematic analysis approach,
frst using an inductive approach to identify codes in the data which
were then grouped into themes following the approach described
in [8, 9].
7 PROTOTYPE STUDY RESULTS
In this section, we summarize the results from our prototype study,
which address RQs 3 and 4.
7.1 Previous Captioning Experiences
We had a mix of participants who either caption their video con-
tent regularly or rarely. Participants discussed motivations for
Alonzo et al.
e Delete
Graphic
Caption i
f Star Preview
g
Preview Settings h
captioning content, which included personal experiences bene-
fting from captions for comprehension or for understanding peo-
ple with diferent accents. While some indicated only captioning
videos when it is required for work or academic assignments, others
were motivated to do so for ensuring their videos are accessible,
especially when making videos for classes with classmates with
cognitive disabilities or who are DHH. P5 also mentioned adding
captions because of stylistic purposes to add personality (e.g. to
emphasize something with specifc fonts).
7.1.1 Participants tend to produce higher quality captions when
the stakes are higher. Some participants indicated producing high-
quality captions only for academic- or work-related videos. Par-
ticipants indicated manually captioning videos, starting from an
automatic system and editing the output, or paying an expert to
do it. Most participants indicated relying on automatic systems to
do some of the work, or an expert to do it for them, because of
how tedious the manual captioning process is. For personal videos,
however, many participants indicated using automatic captioning
tools alone, even when they may be inaccurate. For instance, P1
said “At work, the stakes are higher [. . . ] whereas for personal videos
it’s a little more of a laid back approach [. . . ] I’m doing things on
YouTube and if I’m only going to get 10 to 100 views, then the auto
captions are more than enough.”
7.1.2 Most participants only captions videos with spoken content.
Many participants indicated only captioning videos that have spo-
ken content. For example, P4 said “I’d say if there’s spoken content,
I’ll do closed captions.” Some participants mentioned not captioning
videos when a video only contains music, text or pictures, or if there
is no spoken content. Most participants indicated never having in-
cluded non-speech sounds when captioning videos (which means
they were creating, by defnition, subtitles). However, 3 participants
indicated having occasionally included them, including P5 who said:
“I wouldn’t say consistently, every single time I encounter non-speech
sounds, but I have captioned them before.” P9 also mentioned adding
a song’s title if a video only contains music, or information about
non-speech sounds if they are the main point of a video. P9 has
also uploaded videos to social media platforms that do not sup-
port manually adding captions, but found workarounds to caption
Beyond Subtitles
non-speech sounds such as commenting, replying or using a text
overlay or a sticker.
7.2 Perspectives on Text-Based and Graphic
Captions
The themes we identifed related to hearing creators’ perspectives
on text-based and graphic captions included challenges for creating
descriptions of non-speech sounds for text-based captions, as well as
the appropriateness and benefts of graphic captions. The following
subsections summarize these themes.
7.2.1 Challenges for creating text-based captions for non-speech
sounds included trade-ofs between completeness and concision. Many
participants expressed doubts about the best way to describe sounds
for text-based captions, the best wording, and what level of detail to
include. Participants discussed trade-ofs between length, complex-
ity, accuracy, and how interesting they could make the descriptions.
For example, P6 said “I’m trying to think of how to put that across in
a short, interesting way.” Considering these trade-ofs, participants
commented that on-screen sounds may need fewer details as the
sound source is already visible. For instance, P5 said: “I think [’trum-
pet’] gets the point, especially if there’s a guy playing the trumpet
so I might leave that just for conciseness.” Describing ambiguous
sounds (i.e. sounds with unclear sources) was also challenging for
some participants. For example, when describing a beeping noise
from a human in one of the sample videos, P8 expressed confusion
by saying: “I didn’t understand how to describe it. So, it was a bit
confusing but I thought car beeping noise would be the most accurate
description because I think everybody knows about a beeping noise.”
Considering these challenges, some participants indicated that
guidance for creating text-based captions would be useful. P4 indi-
cated that for the text-based captions, guidance could be structural
(e.g. using verbs vs. nouns, or what number of words to use). P1
and P5, in turn, suggested that because coming up with good de-
scriptions may require a wider English vocabulary, having some
support for description alternatives may also be useful.
7.2.2 The appropriateness of graphic captions varies with the type
of video and certain characteristics of scenes. Most participants sug-
gested that the appropriateness of graphic captions varies depend-
ing on the type of content. For example, most suggested that the
visualizations available in our study were not appropriate for the
BBC video because those tend to be more “serious.” However, the
visualizations seemed appropriate for the TikTok video. Both P2
and P3 specifcally described the appropriateness depending on the
“place and purpose” of the video. There was disagreement, however,
about whether the formality of the visualizations themselves would
afect their appropriateness. P2, for example, suggested that if the
BBC created their own “more formal looking” set of visualizations,
graphic captions could potentially be more appropriate. However,
P5 indicated that “even if we were to have more formal visuals for
something like the BBC, I think it’s just not in their guidelines to use
[visual] efects.” Finally, participants discussed the utility of graphic
captions for sounds that are of screen, with many suggesting they
would not add graphic captions for sounds that are already on
screen as those are already visually available. Considering these
variations of appropriateness, participants suggested that a system
ASSETS ’22, October 23–26, 2022, Athens, Greece
could provide guidance such as when to add graphic captions for
a sound, why someone might want to add a graphic caption, and
which visualizations to include.
Participants talked about how it would be useful to break down
the timestamps by scene for sounds that linger over scene changes
when employing graphic captions, as their appropriateness may
vary by scene. P5 suggested that it would be useful if the system
could automatically detect scene changes and suggest time stamps
for those scenes within a specifc sound event. P4 also suggested
that the system could automatically identify visual objects in the
video to pin graphic captions to so that if the object moves, the
graphic caption moves along with it.
7.2.3 Graphic captions may be beneficial, but also distracting. Many
participants envisioned benefts from graphic captions, such as
being able to add humor to entertainment or social media videos.
For example, P7 said: “The social media crowd, you know people
who are using Facebook, TikTok and Instagram, will defnitely get
a kick out of this.” However, some participants were also wary of
the potential for visuals to “take away” or distract from the main
content. For example, P10 noted: “If I were to add any visuals, it
would take away from the actual subject matter of the video”.
7.3 Automatic Systems Should Identify
Important Sounds
When asked about what participants wanted from an automatic
system that could identify non-speech sounds for them, most indi-
cated they would want a system to identify important sounds. Some
even highlighted the prototype as doing “a good job” of fltering
important sounds when asked if they noticed any missing sounds
in the results. P3, for instance, said: “The car engine humming in the
background, that was missing but I think that’s good.” Participants’
inclinations to only include important sounds were also evident
in their lack of interactions with the star function (a function de-
scribed in Section 6.1). Participants found the purpose of the star
unclear as they had only included important sounds already, and
thus marking them with a star seemed redundant. P4, for example,
said “when you were explaining the starring feature in the tutorial
earlier, I thought ‘I don’t know if that’s something I would use’ because
I would only include sounds that are important.” When discussing
what constitutes an important sound, participants indicated sounds
that are “noticeable,” that “stood out,” or that provided context for
spoken content.
Many participants were not sure about whether “background”
or “insignifcant” sounds should also be included. However, P2 ac-
knowledged the importance of including all sounds for someone
who may not have the “privilege” to select which sounds to pay
attention to. Thus, after engaging with the prototype, P2 decided
to include all of the sounds in a video and star the important ones.
Notably, when talking about what to do with “sounds that do not
convey information” (P9), participants drew comparisons to alterna-
tive text (i.e. text added to images for screen readers to read). Both
P4 and P9 talked about how guidelines for alternative text suggest
marking images that do not convey information as “decorative.”
Thus, both participants considered sounds that are not important
to be akin to decorative images.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Finally, many participants also talked about what information
about a sound they would like an automatic system to identify.
Participants mentioned wanting only general descriptions of what
the sound may be as too much specifcity would be likely to in-
troduce errors. On the other hand, some participants highlighted
the importance of obtaining accurate time stamps of the sounds
from the system, which we explore in more detail in the next sub-
section. Most participants, however, indicated wanting both the
descriptions and the timestamps for the sounds.
7.4 Dealing with Errors from Automatic
Systems
In the conditions using the automatic system, some scenarios in-
cluded both descriptions and timing errors. In some cases, partici-
pants suggested that the automatic system ended up being unhelp-
ful when containing errors, with some deleting all the suggestions
provided and starting from scratch instead. For instance, P10 com-
mented: “I could argue that [the automatic system] actually made
things a little bit slower than just putting that things in manually.”
However, participants’ discussions of errors suggest that their ex-
pectations difered when talking about the labeling of the sound
versus the time stamps for the sounds.
7.4.1 The accuracy of timestamps may be the most important. Many
participants highlighted accurate timestamps as an important fea-
ture of an automatic system. For example, P5 said “Having the
timings already sorted for you beforehand is very convenient. It takes
out one of the biggest time-consuming parts of a process.” Thus, par-
ticipants seemed more sensitive to timing errors than errors in
describing the sounds. For instance, P9 attributed the unhelpful-
ness of errors specifcally to timestamps: “Erroneous timestamps
are not as useful.” P1 also mentioned that timing errors would be
time-consuming in longer videos: “What if I had a 40-minute video?
Am I gonna have to go through and look at every second?” However,
many participants agreed that having a “template” or “framework”
with accurate timestamps is useful even when the descriptions of
the sounds are incorrect.
7.4.2 Ambiguous sounds may be dificult to identify and describe,
but strategies can help. Ambiguous sounds were at the core of par-
ticipants’ discussions of errors in the automatic labeling of the
sounds. Many acknowledged that dealing with ambiguous sounds
is difcult even for humans. For example, P2 said: “I can see how the
computer that’s identifying the noise really has to be smart because [it
is hard] even for me.” For instance, one of the errors introduced by
the automatic system was labeling an of-screen whining child as a
“horse.” Some participants actually trusted that this was a horse. P4
and P7 mentioned that even though they understood that a horse
did not make sense in that context, the suggestion primed them
and they could not think of the sound as something else. How-
ever, participants also discussed strategies to disambiguate certain
sounds. For example, P8 found an error that labeled a trumpet as
a mosquito understandable because visual information from the
video was needed to disambiguate that sound: “To tell the diference
that this is a trumpet, you need a little bit of visual along with the
sound to know.” Others also talked about using spoken content in
the video to disambiguate sounds. For example, P2 talked about
Alonzo et al.
correctly identifying that the sound labeled as a “horse” was a child
using the spoken content from the video, which referred to a child.
“I guess it is confrmation that it is a baby,” P2 said.
7.4.3 The ability to adjust the system’s sensitivity may help in deal-
ing with errors. When discussing errors from the automatic system,
a few participants believed those errors were an issue of sensitivity.
P6, for example, suggested that the errors were caused because
the system may have been “too sensitive.” Thus, these participants
suggested adding a “reanalyze button” (P1) or a “sensitivity slider”
(P7) that could help to reduce the errors from the system. P7 defned
sensitivity in terms of the number of sounds identifed, but also in
terms of how “obvious” the sounds that the system identifed are.
8 DISCUSSION AND TAKEAWAYS
This section summarizes the takeaways from our studies for those
interested in captioning or visualizing non-speech sounds in user-
generated content, including researchers investigating automatic
captions. Considering that our fndings include the perspectives of
DHH viewers on new approaches to visualize non-speech sounds
(i.e. graphic captions), some of our takeaways may also be insight-
ful for industry professionals interested in visualizing non-speech
sounds in professionally-produced content.
Be selective about non-speech sounds. The results from both of
our studies suggest that both DHH and hearing users valued being
selective about which non-speech sounds to include. DHH partici-
pants expressed interest in only having sounds that are important,
as including every single detail may provide too much information.
Hearing creators, in turn, described important sounds as what they
would want an automatic system to identify because captioning all
sounds in a video may be too time-consuming. These fndings align
with guidelines for creating closed captions suggests including non-
speech sounds, which suggest only doing so when necessary (e.g.
the BBC’s captioning guidelines12 ). Our work further encourages
researchers working on automatic sound event detection to con-
sider importance estimations when detecting non-speech sounds in
user-generated videos, and we provide insights about what consti-
tutes an important sound. For instance, hearing creators discussed
the criteria of whether a sound afects the spoken content of a video.
Finally, estimations of sound importance or other sound qualities
(e.g. volume) could be used to adjust the “sensitivity” of automatic
systems and narrow down their results.
Include details, but balance with potential for distraction.
Our results suggest that DHH participants are interested in hav-
ing detailed information about non-speech sounds (e.g. the source,
source location and changes in sounds) in text-based or graphic
captions, which aligns with prior work on sound visualization [18]
and the use visual-tactile feedback for non-speech information in
captions [24]. However, DHH participants in our study worried that
too many details in text-based captions, and graphic captions in
general (which naturally included more details), could be distract-
ing. Thus, those interested in captioning or visualizing non-speech
sounds in their videos should consider including details, but be
12 https://bbc.github.io/subtitle-guidelines/#Intonation-and-emotion
Beyond Subtitles
mindful of the potential for distracting viewers from the main con-
tent. A possible design intervention to support creators would be
providing guidance within the user interface on how to structure
the text-based captions (e.g. how many words to use, and which
types of words) and how much detail to include.
Consider properties about the video, sound, and audience
for choosing between text-based and graphic captions, and
their respective benefts. Our fndings reveal diferent factors to
consider when choosing text-based or graphic captions. First, the
type of video appeared to determine the appropriateness of graphic
captions. The results from both studies suggested that graphic cap-
tions may be more appropriate for entertainment videos, while
text-based captions may be more appropriate for “serious” videos.
Future work can further explore whether varying the design of
graphic captions to better match the content would afect viewers’
and creators’ preferences for the use of graphic captions. Other
factors, such as the visibility of a sound on screen and demographic
factors of the viewers (e.g. their age and hearing ability) are impor-
tant too. While future work could explore each of these factors in
more detail, our fndings provide guidance for designers of these
technologies to consider these factors about the videos, the sounds
and their audiences. Our fndings also shed light on what text-based
and graphic captions may signify. The latter may serve to illustrate
recognizable sounds and indicate the location of the sources, the
precise timing of the sounds, as well as changes in the sounds.
Text-based captions, on the other hand, may better describe harder-
to-visualize abstract sounds where a greater level of verbal detail
may be required.
Graphic captions should be optional and may need standard-
ization. DHH participants suggested that graphic captions should
be optional for viewers instead of being embedded in videos, an ap-
proach analogous to close captioning (as opposed to open captions,
which are embedded in videos). Furthermore, DHH participants
also suggested that graphic captions may be difcult to distinguish
from other graphics in online videos. The ability to overlay them on
demand may also support viewers in distinguishing which graphics
are part of the video as opposed to graphic captions. However, cur-
rent captioning technologies only support text (or animated text).
Thus, new captioning or media formats, such as BBC’s proposal for
Object-Based Media [2, 3, 19], may need to be developed to support
the addition and standardization of optional graphic captions.
Text-based captions of non-speech sounds may help distin-
guish uncaptioned videos. DHH participants also suggested that
using text-based captions for non-speech sounds can support view-
ers to distinguish videos without spoken content from uncaptioned
videos, as there are times when DHH viewers cannot tell if a video
is not captioned or if it simply does not contain spoken content.
If a video only contains non-speech sounds, and those sounds are
captioned, viewers may conclude that the video does not contain
spoken content (although there is still a possibility that a creator
only captions non-speech sounds in a video with spoken content).
It is also possible that a video may contain unimportant (or dec-
orative) non-speech sounds. Hearing creators’ comparisons with
alt-text, and the respective guidelines for decorative images that
ASSETS ’22, October 23–26, 2022, Athens, Greece
do not provide any information13 , suggest providing indications
when videos only contain decorative sounds may help reduce the
ambiguity of uncaptioned videos.
Providing accurate timestamps to creators is important, but
general descriptions are helpful, too. Our results suggest that
accurate time stamps are important when using an automatic sys-
tem, as our hearing creators suggested that identifying these is
the most time-consuming aspect of captioning in general. While
descriptions were also important to many of our participants, gen-
eral descriptions seemed more useful than specifc ones given that
specifcity may introduce errors, especially when dealing with am-
biguous sounds. Our results also suggest ways in which ambiguity
can be reduced in automatic systems, which was a source of dif-
fculty highlighted by YouTube for including non-speech sounds
in their automatic captions [12]. More specifcally, participants
highlighted the use of semantic information from both visual and
spoken content to reduce ambiguity, which aligns with current
trends in multi-modal analysis and understanding.
9 CONCLUSION, LIMITATIONS, AND
FUTURE WORK
In this paper, we presented two studies with DHH and hearing
participants to explore their perspectives on captioning non-speech
sounds, using text-based or graphic captions, in user-generated
videos. Our fndings include DHH participants’ interests in hav-
ing important non-speech sounds in these videos, while hearing
creators also indicated an inclination toward only including im-
portant sounds when captioning non-speech sounds. Our fndings
also include trade-ofs between text-based and graphic captions
for captioning non-speech sounds, and potential factors for de-
termining their appropriateness. Finally, we explored the use of
automatic tools to support hearing creators when captioning non-
speech sounds and identifed guidance for future work in this area.
There were several limitations in our work, and avenues for
future work. First, our work was qualitative, with a small sample
size and a small selection of videos. We identifed potential factors
at play when determining the appropriateness of graphic captions
(including the content and sound type, the sound location, viewers’
demographic factors, and the graphic captions’ style). However,
future work should explore these factors using a larger selection of
carefully controlled videos among a larger sample size of viewers.
Similarly, in our prototype study, participants were using sample
videos provided by us. A study with content creators editing their
own videos could reveal further insights relevant to real use cases.
Our participants also had diverse levels of skills and experience. Fu-
ture work investigating the preferences of participants with specifc
levels of skills or experience may yield further insights.
Our formative study suggested that automatic sound event de-
tection may be helpful for DHH creators to caption non-speech
sounds in their videos, which may introduce diferent challenges.
Thus, future work should also explore its use among DHH creators
and its implications.
13 https://webaim.org/techniques/alttext/#decorative
ASSETS ’22, October 23–26, 2022, Athens, Greece
ACKNOWLEDGMENTS
We thank Justin Salamon for helping with sound event detection
and the Center for Accessibility and Inclusion Research (CAIR) at
RIT for feedback on a early draft of the paper. This material is based
upon work supported by the National Science Foundation under
awards No. 1822747 and 2125362.
REFERENCES
[1] Mike Armstrong, Andy Brown, Michael Crabb, Chris J Hughes, Rhianne Jones,
and James Sandford. 2016. Understanding the diverse needs of subtitle users in a
rapidly evolving media landscape. SMPTE Motion Imaging Journal 125, 9 (2016),
33–41.
[2] Mike Armstrong and Michael Crabb. 2017. Exploring ways of meeting a wider
range of access needs through object-based media-workshop. In Conference on
Accessibility in Film, Television and Interactive Media, York, UK.
[3] BBC Research & Development. [n.d.]. Object-Based Media.
https://www.bbc.co.uk/rd/object-based-media. Accessed: 2022-06-01.
[4] Larwan Berke, Khaled Albusays, Matthew Seita, and Matt Huenerfauth. 2019.
Preferred Appearance of Captions Generated by Automatic Speech Recognition
for Deaf and Hard-of-Hearing Viewers. In Extended Abstracts of the 2019 CHI
Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk)
(CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6.
https://doi.org/10.1145/3290607.3312921
[5] Larwan Berke, Sushant Kafe, and Matt Huenerfauth. 2018. Methods for Evaluation
of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Diferent Reading
Literacy Levels. Association for Computing Machinery, New York, NY, USA, 1–12.
https://doi.org/10.1145/3173574.3173665
[6] Larwan Berke, Matthew Seita, and Matt Huenerfauth. 2020. Deaf and Hard-
of-Hearing Users’ Prioritization of Genres of Online Video Content Requiring
Accurate Captions. In Proceedings of the 17th International Web for All Conference
(Taipei, Taiwan) (W4A ’20). Association for Computing Machinery, New York,
NY, USA, Article 3, 12 pages. https://doi.org/10.1145/3371300.3383337
[7] Danielle Bragg, Nicholas Huynh, and Richard E. Ladner. 2016. A Personalizable
Mobile Sound Detector App Design for Deaf and Hard-of-Hearing Users. In
Proceedings of the 18th International ACM SIGACCESS Conference on Computers
and Accessibility (Reno, Nevada, USA) (ASSETS ’16). Association for Computing
Machinery, New York, NY, USA, 3–13. https://doi.org/10.1145/2982142.2982171
[8] Virginia Braun and Victoria Clarke. 2006. Using thematic
analysis in psychology. Qualitative Research in Psychology 3,
2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
arXiv:https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa
[9] Virginia Braun and Victoria Clarke. 2022. Thematic Analysis. https://www.
thematicanalysis.net. Accessed: 2022-03-01.
[10] Andy Brown, Rhia Jones, Mike Crabb, James Sandford, Matthew Brooks, Mike
Armstrong, and Caroline Jay. 2015. Dynamic Subtitles: The User Experience. In
Proceedings of the ACM International Conference on Interactive Experiences for TV
and Online Video (Brussels, Belgium) (TVX ’15). Association for Computing Ma-
chinery, New York, NY, USA, 103–112. https://doi.org/10.1145/2745197.2745204
[11] Andy Brown, Jayson Turner, Jake Patterson, Anastasia Schmitz, Mike Armstrong,
and Maxine Glancy. 2017. Subtitles in 360-Degree Video. In Adjunct Publication
of the 2017 ACM International Conference on Interactive Experiences for TV and
Online Video (Hilversum, The Netherlands) (TVX ’17 Adjunct). Association for
Computing Machinery, New York, NY, USA, 3–8. https://doi.org/10.1145/3084289.
3089915
[12] Sourish Chaudhuri. 2017. Adding sound Efect information to Youtube captions.
https://ai.googleblog.com/2017/03/adding-sound-efect-information-to.html Ac-
cessed: 2021-06-01.
[13] Karen Collins and Peter J. Taillon. 2012. Visualized sound efect icons for improved
multimedia accessibility: A pilot study. Entertainment Computing 3, 1 (2012),
11–17. https://doi.org/10.1016/j.entcom.2011.09.002
[14] Michael Crabb, Rhianne Jones, and Mike Armstrong. 2015. The Development of
a Framework for Understanding the UX of Subtitles. In Proceedings of the 17th
International ACM SIGACCESS Conference on Computers & Accessibility (Lisbon,
Portugal) (ASSETS ’15). Association for Computing Machinery, New York, NY,
USA, 347–348. https://doi.org/10.1145/2700648.2811372
[15] Michael Crabb, Rhianne Jones, Mike Armstrong, and Chris J. Hughes. 2015.
Online News Videos: The UX of Subtitle Position. In Proceedings of the 17th
International ACM SIGACCESS Conference on Computers & Accessibility (Lisbon,
Portugal) (ASSETS ’15). Association for Computing Machinery, New York, NY,
USA, 215–222. https://doi.org/10.1145/2700648.2809866
[16] Sofa Enamorado. 2018. CVAA & FCC closed Captioning requirements for online
video. https://www.3playmedia.com/blog/fnal-cvaa-and-fcc-online-video-
Alonzo et al.
closed-captioning-rules/ Accessed: 2021-06-01.
[17] Deborah I Fels, Daniel G Lee, Carmen Branje, and Matthew Hornburg. 2005.
Emotive Captioning and Access to Television. https://doi.org/10.1145/3173574.
3173665
[18] Leah Findlater, Bonnie Chinh, Dhruv Jain, Jon Froehlich, Raja Kushalnagar, and
Angela Carey Lin. 2019. Deaf and Hard-of-Hearing Individuals’ Preferences for
Wearable and Mobile Sound Awareness Technologies. Association for Computing
Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300276
[19] Benjamin M. Gorman, Michael Crabb, and Michael Armstrong. 2021. Adaptive
Subtitles: Preferences and Trade-Ofs in Real-Time Media Adaption. In Proceedings
of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama,
Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA,
Article 733, 11 pages. https://doi.org/10.1145/3411764.3445509
[20] Michael Gower, Brent Shiver, Charu Pandhi, and Shari Trewin. 2018. Leveraging
Pauses to Improve Video Captions. In Proceedings of the 20th International ACM
SIGACCESS Conference on Computers and Accessibility (Galway, Ireland) (ASSETS
’18). Association for Computing Machinery, New York, NY, USA, 414–416. https:
//doi.org/10.1145/3234695.3241023
[21] Dhruv Jain, Sasa Junuzovic, Eyal Ofek, Mike Sinclair, John Porter, Chris Yoon,
Swetha Machanavajhala, and Meredith Ringel Morris. 2021. A Taxonomy of
Sounds in Virtual Reality. In Designing Interactive Systems Conference 2021 (Virtual
Event, USA) (DIS ’21). Association for Computing Machinery, New York, NY,
USA, 160–170. https://doi.org/10.1145/3461778.3462106
[22] Dhruv Jain, Angela Lin, Rose Guttman, Marcus Amalachandran, Aileen Zeng,
Leah Findlater, and Jon Froehlich. 2019. Exploring Sound Awareness in the Home for
People Who Are Deaf or Hard of Hearing. Association for Computing Machinery,
New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300324
[23] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D.
Plumbley. 2020. PANNs: Large-Scale Pretrained Audio Neural Networks for Audio
Pattern Recognition. IEEE/ACM Transactions on Audio, Speech, and Language
Processing 28 (2020), 2880–2894. https://doi.org/10.1109/TASLP.2020.3030497
[24] Raja S. Kushalnagar, Gary W. Behm, Joseph S. Stanislow, and Vasu Gupta. 2014.
Enhancing Caption Accessibility through Simultaneous Multimodal Information:
Visual-Tactile Captions. In Proceedings of the 16th International ACM SIGACCESS
Conference on Computers & Accessibility (Rochester, New York, USA) (ASSETS
’14). Association for Computing Machinery, New York, NY, USA, 185–192. https:
//doi.org/10.1145/2661334.2661381
[25] Daniel G. Lee, Deborah I. Fels, and John Patrick Udo. 2007. Emotive Captioning.
Comput. Entertain. 5, 2, Article 11 (April 2007), 15 pages. https://doi.org/10.1145/
1279540.1279551
[26] Tara Matthews, Janette Fong, F. Wai-Ling Ho-Ching, and Jennifer Mankof. 2006.
Evaluating non-speech sound visualizations for the deaf. Behaviour & Information
Technology 25, 4 (2006), 333–351. https://doi.org/10.1080/01449290600636488
arXiv:https://doi.org/10.1080/01449290600636488
[27] John McGowan, Grégory Leplâtre, and Iain McGregor. 2017. CymaSense: A Real-
Time 3D Cymatics-Based Sound Visualisation Tool. In Proceedings of the 2017 ACM
Conference Companion Publication on Designing Interactive Systems (Edinburgh,
United Kingdom) (DIS ’17 Companion). Association for Computing Machinery,
New York, NY, USA, 270–274. https://doi.org/10.1145/3064857.3079159
[28] Carol Padden and Tom Humphries. 2005. Inside Deaf Culture. Harvard University
Press. http://www.jstor.org/stable/j.ctvjz83v3
[29] S. J. Parault and H. M. Williams. 2010. Reading Motivation, Reading Amount,
and Text Comprehension in Deaf and Hearing Adults. Journal of Deaf Studies
and Deaf Education 15, 2 (2010), 120–135. https://doi.org/10.1093/deafed/enp031
[30] C. B. Traxler. 2000. The Stanford Achievement Test, 9th Edition: National Norming
and Performance Standards for Deaf and Hard-of-Hearing Students. Journal of
Deaf Studies and Deaf Education 5, 4 (Jan 2000), 337–348. https://doi.org/10.1093/
deafed/5.4.337
[31] M. Wald. 2011. Crowdsourcing Correction of Speech Recognition Captioning Errors.
Association for Computing Machinery, New York, NY, USA. https://doi.org/10.
1145/1969289.1969318
[32] Dawn Walton, Georgianna Borgna, Marc Marschark, Kathryn Crowe, and Jes-
sica Trussell. 2019. I am not unskilled and unaware: deaf and hearing learners’
self-assessments of linguistic and nonlinguistic skills. European Journal of Spe-
cial Needs Education 34, 1 (2019), 20–34. https://doi.org/10.1080/08856257.2018.
1435010 arXiv:https://doi.org/10.1080/08856257.2018.1435010
[33] Fangzhou Wang, Hidehisa Nagano, Kunio Kashino, and Takeo Igarashi. 2015.
Visualizing video sounds with sound word animation. In 2015 IEEE International
Conference on Multimedia and Expo (ICME). 1–6. https://doi.org/10.1109/ICME.
2015.7177422
[34] Noah Wang. 2017. Visualizing sound efects. https://youtube-eng.googleblog.
com/2017/03/visualizing-sound-efects.html Accessed: 2021-06-01.
[35] Sean Zdenek. 2015. Reading sounds: Closed-captioned media and popular culture.
University of Chicago Press.
Nothing Micro About It: Examining Ableist Microaggressions on
Social Media
Sharon Heung Mahika Phutane
Cornell Tech Cornell Tech
New York, New York, USA New York, New York, USA
Megh Marathe∗
Michigan State University
East Lansing, Michigan, USA
ABSTRACT
Ableist microaggressions are subtle forms of discrimination that
disabled people experience daily, perpetuating inequalities and
maintaining their ongoing marginalization. Despite the importance
of understanding such harms, little work has been done to examine
how disabled people are discriminated against online. We address
this gap by investigating how disabled people experience ableist
microaggressions on social media and how they respond to and
cope with these experiences. By conducting interviews with 20
participants with various disabilities, we uncover 12 archetypes
of ableist microaggressions on social media, reveal participants’
coping mechanisms, and describe the long-term impact on their
wellbeing and social media use. Lastly, we present design recom-
mendations, re-evaluating how social media platforms can mitigate
and prevent these harmful experiences.
KEYWORDS
microaggressions, ableism, social media, disability
ACM Reference Format:
Sharon Heung, Mahika Phutane, Shiri Azenkot, Megh Marathe, and Aditya
Vashistha. 2022. Nothing Micro About It: Examining Ableist Microaggres-
sions on Social Media. In The 24th International ACM SIGACCESS Conference
on Computers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens,
Greece. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3517428.
3544801
1 INTRODUCTION
Disabled people1 routinely experience diferent forms of discrimi-
nation, despite increased awareness about disability and legislation
to protect disabled people [1]. One form of discrimination is ableist
∗ Marathe and Vashistha are joint senior authors.
1We use the term disabled people, since identity-frst language is preferred by dis-
ability rights groups, the disability justice movement, and mimics the language our
participants used. We recognize that being a disabled person is a part of one’s identity
and experiences, and that a person is disabled by society, technology, and the built
environment.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544801
Shiri Azenkot
Cornell Tech
New York, New York, USA
Aditya Vashistha∗
Cornell University
Ithaca, New York, USA
microaggressions which are defned as subtle remarks or insults
that are fueled by negative stereotypes of disability [50]. These
“micro” forms of discrimination perpetuate inequalities, ableism,
and stereotypes against disabled people while maintaining their
ongoing marginalization [50].
Several scholars have examined disabled people’s experiences
with ableist microaggressions in everyday settings and documented
the harms emerging from such experiences. For example, Keller
and Galgay examined diferent types of microaggressions disabled
people encountered in-person and presented a framework to inter-
pret such microaggressive experiences [29]. Xiong developed scales
to measure diferent types of microaggressive behaviors [56]. Such
microaggressive experiences cause negative health outcomes (e.g.,
a greater likelihood of depression and anxiety) and afect academic
performance (e.g., reduced academic self concept and lower grade
satisfaction) [28, 32, 35]. While these research advances provide the
necessary foundation to study ableist microaggressions towards
disabled people, all the work focuses on microaggressive behaviors
in ofine settings.
In recent years, social media has witnessed an unprecedented
growth in users worldwide. An increasing number of social interac-
tions occur through social media for both disabled and non-disabled
people alike. With more disabled people turning to social media
as a means to socialize with others remotely and form online com-
munities, it’s important to understand feelings of exclusion and
discrimination that occur on these platforms [3, 11–13, 25, 36, 48].
To date, little is known about the types of microaggressions disabled
people experience on social media, and how these experiences difer
across online and ofine settings. To fll this critical gap, we sought
to answer two research questions:
• RQ1: How do disabled people experience microaggressions
on social media?
• RQ2: How do disabled people respond to and cope with
microaggressions?
To answer these questions, we conducted semi-structured inter-
views with 20 participants who had a variety of disabilities. During
the interviews, we asked them about a time when they experienced
subtle forms of discrimination and felt excluded on social media,
probing around their perception of what had happened and how
they responded. We gained insight on how these platforms mediate
such interactions between the disabled person and the perpetra-
tors as well as how these microaggressive experiences afected the
disabled person’s social media use.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Our fndings revealed 12 microaggression archetypes that dis-
abled people experience on social media. Participants reported
routinely receiving ableist posts, messages, and comments that
made them feel patronized and infantilized. Participants had to
endure invasive, personal questions regarding their disability, rela-
tionships, and intimacy. They also felt excluded when people on
social media ignored their posts or accessibility needs and when
users and platforms censored content posted by them. While some
microaggressive experiences bore similarity to those happening
in ofine settings (e.g., denial of disability identity), we uncovered
new forms of microaggressive experiences that are unique to so-
cial media (e.g., being ignored or ghosted). Participants described
microaggressions as harmful experiences that hampered their self-
esteem and confdence, left a lasting impact, and forced them to
change the way they use social media. They responded to and coped
with microaggressive behaviors in diferent ways. While some par-
ticipants responded to microaggressions to educate the perpetrator
or retaliate, many used platform features to report the ofending
comment and block the perpetrator.
Drawing on these fndings, we discuss how microaggressions
manifest diferently on social media and present design recommen-
dations on how social media platforms could mediate and prevent
these harmful experiences. We make several contributions to the
accessibility community and the conversation around online inclu-
sion:
• We provide a comprehensive understanding of various types
of microaggressive behaviors disabled people experience on
social media, their coping mechanisms and response strate-
gies, and the resulting impact on their wellbeing, self-worth,
and social media use.
• We explicate the diferences in ableist microaggressions in
online and ofine settings.
• We discuss design recommendations to reimagine social me-
dia that is more inclusive and welcoming.
2 RELATED WORK
The term microaggressions was originally coined by Chester Pierce,
a psychiatrist who studied these “subtle, stunning, often automatic,
and non-verbal exchanges” towards African Americans [42, 50].
Since then, microaggressions have been re-defned as “brief slights
and insults targeting persons of oppressed identities” [4, 50]. The
use of the term has expanded to other marginalized populations, in-
cluding the disability community. Similar to other structural forms
of oppression (e.g., racism and sexism), microaggressions experi-
enced by disabled people stem from structural ableism; hence the
term ableist microaggressions.
To date, scholarship on ableist microaggressions has primarily
focused on their occurrences in ofine and in-person settings. Keller
and Galgay [29] were the frst to systematize the types of microag-
gressions uniquely experienced by disabled people. Based on focus
groups with twelve disabled people, they developed a framework
to categorize ableist microaggressions into ten distinct patterns.
They found microaggressions about “denial of personal identity”
when people ignored all aspects of a disabled person’s identity
other than disability. Disabled people also encountered “denial of
disability experience” when microaggressions minimized or denied
Heung et al.
their disability itself or experiences related to their disability. Mi-
croaggressions also consisted of “denial of privacy” experiences
when people solicited private information relating to a person’s
disability or “desexualization” when disabled people were assumed
to be uninterested in or incapable of sex. Disabled people also en-
countered “helplessness” when non-disabled people insisted on
helping them even when no help was necessary or tried to rescue
them from their disability. Disabled people were often a target of
“patronization” when people lauded their unremarkable everyday
actions as achievements, “infantilization” when people treated them
as immature and dependent, and “second-class citizen” when they
were treated as inferior or more burdensome than non-disabled
people. Keller and Galgay also found a“spread efect” when people
made assumptions due to their disability (e.g., assuming that a blind
person’s sense of taste or smell are more powerful than those of
sighted people).
The foundational work by Keller and Galgay has been expanded
upon by numerous scholars who have examined ableist microag-
gressions in ofine contexts. For example, Bell [5] conducted an
interview study with people who identifed as having visible disabil-
ities, confrming Keller and Galgay’s patterns of ableist microaggres-
sions and developing new insights on the impact of microaggres-
sions for visibly disabled people. Bell found that microaggressions
place disabled people in a “disability double bind,” which causes an
internal confict between a disabled person asserting their identity
as a capable person worthy of equal treatment, while also having to
request accommodations for access purposes. Bell further described
fve strategies that visibly disabled people use when responding
to microaggressions and noted that the ten patterns proposed in
Keller and Galgay’s framework are not mutually exclusive; a single
microaggression can embody more than one pattern.
Olkin et al. [40] conducted a mixed-methods study with disabled
women and found two new types of ableist microaggressions in
addition to confrming the patterns identifed by Keller and Galgay.
They found that disabled women encountered microaggressions
when their health-related symptoms were not believed by medical
practitioners or when they were told they looked too healthy or
attractive to be disabled. Xiong [56] developed a prototype scale
to measure ableist microaggressions. The 93 items in this scale
combine Keller and Galgay’s patterns with microaggressions not
specifc to a disability context. The scale has been reviewed by dis-
abled experts, although it remains to be validated [56]. While these
research advances provide the necessary foundation to study ableist
microaggressions towards disabled people, all the work discussed
thus far focuses on microaggressive behaviors in ofine settings.
A growing body of HCI research has examined online harass-
ment and discrimination targeted towards various marginalized
communities [7, 27, 31, 37]. For example, scholars have studied
harassment experienced by Black people [22, 38, 55], Asian peo-
ple [23], women [39, 51, 54], LGBTQIA+ people [8, 21, 22, 51],
and Muslims [15, 30]. Although studies of online hate speech and
disability disclosure have uncovered some instances of ableism
[6, 7, 16, 37, 38], scholars have yet to examine the specifc kinds of
discrimination disabled people experience online. We extend the
scholarship on (1) online harassment and (2) ableist microaggres-
sions by examining new forms of microaggressions that disabled
people encounter on social media.
Nothing Micro About It: Examining Ableist Microaggressions on Social Media
3 METHODS
To examine ableist microaggressions on social media, we conducted
interviews with 20 disabled people.
3.1 Participant Recruitment
We recruited participants through various sampling methods, in-
cluding convenience sampling (recruiting those we knew), snow-
ball sampling (referrals from recruited participants), and strat-
ifed sampling (recruiting for a diverse balance of gender and
disability)[2, 45]. Participants were screened through a Qualtrics
survey to ensure they ft our criteria to include those who self-
identify as having a permanent or long-term disability, have a
visible disability profle (disability disclosed online), and use social
media on a regular basis (multiple times per week or everyday). We
required participants to disclose their disability because we sought
to understand disability-specifc microaggressions. This disclosure
could occur via participants’ posts, bios, and/or photos.
3.2 Data Collection
We conducted semi-structured interviews with participants via
Zoom. Due to access needs, three participants responded in text
through the chat feature and one participant used a combination
of voice and text. All interviews were conducted in English, lasted
approximately 60 minutes, and were audio-recorded with the par-
ticipants’ consent. Participants received Amazon gift cards in recog-
nition of their time and expertise.
Each interview began with introductions, a review of the consent
form, and reiteration that the participant could take a break or skip
questions at any time. Questions were open-ended to empower
participants to share their experiences. The interview included
questions about social media use, microaggression experiences,
and avenues for improvement of social media platforms. We asked
participants to share what platforms they use, what content they
post that reveal aspects of their identities to help contextualize
their experiences with microaggressions (e.g., how much does your
profle reveal about yourself and your disability identity?). We then
asked participants to tell us about a time when they were treated
diferently on social media, when they felt unwelcome on social
media, and when they experienced subtle discrimination or mi-
croaggressions on social media. We intentionally did not use the
term microaggression earlier in the interview in case participants
were unfamiliar with the term. From there we asked follow-up
questions to better understand what happened (e.g., when did this
happen?) and how it afected their social media use (e.g., did this
change the types of content you shared?). Finally, we asked par-
ticipants to refect on how microaggressions might be unique to
people with disabilities (e.g., do you think people with disabilities
are more or less likely to experience microaggressions on social
media?) and how the platform and other users could help during
microaggressive experiences. We conducted interviews until theo-
retical saturation was reached.
3.3 Data Analysis
Our data consisted of approximately 17 hours of audio recordings
and detailed notes collected during the interviews. After transcrib-
ing the interviews, we used thematic analysis [9] to conduct open
ASSETS ’22, October 23–26, 2022, Athens, Greece
coding to discover new categories of microaggressions and deduc-
tive coding to draw from existing in-person, ableist microaggres-
sions [29, 52]. The three coders frst coded two interviews together,
discussing discrepancies in codes and creating a preliminary code-
book. The coders then coded the rest of the interviews separately,
communicating when new codes arise by annotating the code-
book. Throughout the analysis, we held multiple discussions to
iteratively refne the codes and reconcile disagreements through
peer-debriefng [17] to ensure that our themes comprehensively
represent the data. In the end, we had 182 codes (e.g., response =
no response) and 16 themes (e.g., “responding is pointless”).
3.4 Participant Demographics
Table 1 presents details about participant demographics and social
media use. This information uses participants’ own words. We re-
cruited 20 participants (11 male, 7 female, 1 gender non-conforming,
and 1 preferred not to disclose) with varying long-term disabilities
(e.g., blind or visually impaired, deaf and hard of hearing, and neu-
rodivergent). Participant ages ranged from 19 to 35 years (mean =
26.7). The majority (n=17) of our participants were from the United
States, while three participants resided in India, Ireland, and the
UK.
3.5 Positionality
Given the sensitive nature of discussing online discrimination and
microaggressions, we believe it is crucial to refect on our stance.
Our team consists of authors with a rich experience of working
with disabled people and authors who have experienced ableist
microaggressions. Although we present individual accounts of mi-
croaggressions, we take a disability studies perspective in striving
for structural changes that mitigate and ideally dismantle ableism
[34]. Our goal is to bring forward the voices of disabled people in
how they are excluded online; and intervene in conversations on
online governance and content moderation.
4 FINDINGS
We present the diferent forms of microaggressions that the disabled
participants experienced on social media (Section 4.1). We then
share the aftermath of these experiences, discussing ways in which
disabled people responded to (Section 4.2.1) and coped with these
microaggressions (Section 4.2.2). Finally, we describe participants’
views on the long-term impact of microaggressions and ways to
help prevent and mitigate the harm (Section 4.2.3).
4.1 Types of Ableist Microaggressions
Although two participants could only recall experiences of
overt discrimination, the majority of our participants described
microaggression-related experiences in detail. Participants often
referred to these experiences as “backhanded comments” (P11)
or somebody "hid[ing] the fact that they’re trying to discriminate
[against] me” (P16). Some added that microaggressions were “usu-
ally unintentional” and, due to their subtlety, these experiences
could lead to “overanalyzing” the event (P3). We present our fnd-
ings in the form of 12 archetypes that closely mirror the actual
ASSETS ’22, October 23–26, 2022, Athens, Greece Heung et al.

Table 1: Participant Demographics and Social Media (SM) Use.

P Gen. Age Disability* Ethnicity* SM Use SM Platforms Disability Disclosure*

P1 F 19 Several Physical Disabilities Caucasian Daily Facebook, TikTok, Snapchat, Insta- via public profle (Infuencer)
gram, dating apps
P2 M 29 100% Vision Impairment Indian LinkedIn, WhatsApp via posts
P3 M 22 Duchenne Muscular Dystrophy Indian-American Bi-weekly Facebook, Instagram via personal chats on Messenger
P4 X 21 Neurodivergent, "Cocktail of disabil- Caucasian Daily Twitter, Instagram, Tumblr, Snapchat, via smaller groups ie. discord servers, via
ities" Discord, Leage of Legends chat description
P5 F 22 Blind, Fiber Myalgia, PTSD, Lupus, Caucasian Daily Facebook, Messenger, Reddit via posts, groups, and service animal
Autism
P6 M 30 Mild Vision Issues, Speech-related African American Facebook via posts of hobbies that indirectly re-
vealed disability
P7 M 26 Visually Impaired Black American Daily Facebook, Twitter, Reddit, Instagram, via photos
LinkedIn, WhatsApp, Snapchat
P8 F 30 Depression (since childhood) Caucasian Daily Facebook via public posts, writing articles
P9 M 25 Speech Disability Black Daily Facebook, Twitter, Reddit via personal profle
P10 M 32 Speech-related Disability, Attention African American Daily Facebook, Twitter, Whatsapp, Tele- via public profle, bio ("disability is not in-
Defcit gram, Instagram ability")
P11 M 21 Cerebral Palsy, Wheelchair User Caucasian
P12 F 26 Autism Black American Daily Facebook, Twitter, Instagram, What- via posts, stories
sApp
P13 M 35 Epilepsy African American Daily Facebook, Twitter, Instagram, Dating via personal chats
apps, WhatsApp
P14 M 35 Leg Problem, User of Manual White Daily Facebook, Twitter, Instagram, What- via public pictures, posts
Wheelchair, Standing Stick sApp, TikTok
P15 M 35 Hard of Hearing and Other Physical Black Weekly Facebook, Twitter, Instagram, TikTok via private status, photos for friends
Impairments
P16 M 25 Visually Impaired (low vision) Black American Daily Facebook, Twitter, Instagram, via posts, pictures for friends
LinkedIn, WhatsApp
P17 F 32 Epilepsy, Autism (since childhood) Black American Daily Facebook, Instagram, WhatsApp, via bio description, writing articles
Linkedln
P18 F 24 Autism Black Daily Twitter, Instagram, LinkedIn, What- via pictures, posts
sApp
P19 X 21 Ehlers-Danlos Syndrome, En- White Daily Facebook, Twitter, TikTok, Youtube via public description, bio, content
dometriosis, Neurological Disorder,
Chronic Pain
P20 F 23 Wheelchair user Pakistani & Irish Daily Twitter, Instagram via bio ("wheelchair user")

*are self-identifed and self-dictated by participants

manifestation of a microaggression. These archetypes were in- “I hate that so much. I’m not inspirational for going to
tentionally chosen to preserve the visceral impact, nuance, and class and expressing to my friends how much I don’t
emotional intensity that accompany instances of microaggressions. want to be in class. That’s not inspirational, that is what
every other college student does.”

4.1.1 Patronization and Infantilization. Participants shared sev-
eral instances of microaggressions where they felt patronized and
infantilized.
You’re so inspirational. Patronizing comments were the most
common microaggression that our participants experienced. More
specifcally, this manifested in comments where other users ex-
aggerated the participant’s routine activities as inspirational or
even sometimes “glorif[ied]” a disabled person for living a normal
life. According to P2, "[people] keep posting ‘oh, this is so cool’ and
‘you’re so inspiring’ just because you might have done something
very normal." P11 expressed the same sentiment saying the most
frequent microaggression is “oh my God you’re doing this thing
independently like that’s amazing [and] I can’t believe you’re func-
tioning as a human person. . . phrased in 1000 diferent ways.” P1 also
received comments like “you’re so inspirational” and explained why
this particular microaggression was upsetting:
Other instances of patronization were more subtle. P20 received
patronizing comments on two occasions. In one instance, when she
posted about a night out, she received comments like, “you seem
so happy being out” and “that’s great for you,” while non-disabled
people received “usual comments like emojis” (P20). On another
occasion, P20 posted about going on a hike, people commented,
“you look so happy.” “People don’t expect me to be able to hike because
I use a wheelchair,” P20 explained.
P20 connected her online experience with face-to-face patron-
ization. Her friends patted her head and said “good girl, good boy.”
While she did not think they had ill-intent, P20 thought this be-
havior was inappropriate, stating that she’s not a “dog” and no one
likes to be “touched without permission.”
Some participants expressed discomfort when others glorifed
disabled people. For example, P20 found videos that glorifed asking
a disabled person to prom, characterizing it as a “heroic thing.”
Nothing Micro About It: Examining Ableist Microaggressions on Social Media
Similarly, P2 saw this on a professional networking site, where
someone commended himself for changing the life of a disabled
person:
"I think about six months back there was some random
post on linkedin... and this person was a stockbroker
and he said I gave an internship to someone with a
disability and he [the intern] made a proft... and now
he [the intern] also has a girlfriend or something like
that. I got really pissed... where did the disability thing
and the girlfriend come into play.” (P2)
Where’s your mom? Several participants expressed another
type of microaggression where people infantilized a disabled person,
treating them like a child, perceiving them as “naive,” and often
discounting their opinions and life experiences. For example, when
P1 posted videos about her daily life on TikTok, she experienced "a
lot of infantilization" like “oh where’s your mom?” and “oh you live
by yourself?” She felt that people believed that she was incapable
of being independent and needed assistance in doing daily chores.
While she understood that this might be because of ableism or
people’s limited understanding of disabilities, she felt frustrated
when people commented that “grocery shopping and doing laundry
is cute”(P1).
4.1.2 Disability as Inability. Several participants experienced mi-
croaggressions when people made assumptions about what a dis-
abled person can and cannot do. In extreme cases, some microag-
gressions questioned disabled people’s ability to contribute to soci-
ety and their very existence.
Can someone like you do that/wear that? While some mi-
croaggressions glorifed disability, others directly assumed a lack of
ability. P16 recalled “rude comments” from people making assump-
tions about his disability, such as “oh sorry, you can’t do this because
you’re visually impaired.” Others shared ableist microaggressions
that shamed them in participating in social media challenges. P12
explained,
“I made a dance video on a popular song and posted it
on my WhatsApp [status]. . . someone on my friend list
said I don’t have to follow all the challenges and make
dance videos when I can’t move my body according to
the beats.”
These ableist assumptions carried over to what a disabled person
can do with their professional and career goals. For instance, P13
partook in a Whatsapp group to talk about business ideas. After
P13 shared a proposal someone asked“can someone like you do
that?” On the other hand, when a disabled person was perceived
as successful, perpetrators invalidated their success by attributing
their achievement to charity. As a TikTok infuencer and performing
artist, P1 explained how people second-guess her success in acting:
"People think that when I get hired for a job in the
entertainment industry it’s simply out of pity...I get a
lot of people who genuinely think that everything is
handed to me because I’m disabled. . . people are like
‘wow I bet the government paid [for you getting the
work]’, which is funny because the government doesn’t
pay for anything.”
ASSETS ’22, October 23–26, 2022, Athens, Greece
Beyond comments about ability, a few participants shared instances
when people commented on how disabled people should look and
dress. For example, P18 described how her disabled friend, who
has a physical disability with “one side of her hand is shorter than
the other,” received a comment on Instagram about her wardrobe.
The perpetrator asked her to “wear something [with] more covering
because of her body and the hand” (P18).
In a more severe case, P13 shared thoughts about the COVID-19
pandemic on Facebook, and a “friend” on the platform expressed
surprise that P13 had not died due to his disability.“Even you sur-
vived this?” the perpetrator asked.
I would kill myself if I was disabled. Some participants ex-
perienced extreme microaggressions with eugenic undertones. P1
received a comment saying “wow you’re so brave, I would kill my-
self if I was disabled.” Such microaggressions were also instances
of patronization. Others recalled microaggressions that indicated
severe stigmatization of a disabled person. P17 received comments
from family members on Facebook referring to her as a “mistake,”
and P14 received a comment from a stranger on Facebook saying,
“you are disturbing me with your disability.” Recalling an in-person
encounter, P1 shared that “old white women” tried “to pray the
disability out of [her].”
4.1.3 Denial of Disability Identity and Experience. Participants ex-
perienced a range of microaggressions where others accused them
of faking their disability or questioned their disability experience.
You’re lying about your disability. While some microaggres-
sions stemmed from assumptions about disability, one type of mi-
croaggression involved questioning whether participants were dis-
abled in the frst place and doubting the degree of their disability. P1
shared instances when “being disabled comes into question” with ac-
cusations of faking the need for a wheelchair. Similarly, P5 recalled
being accused of faking her disability due to lack of awareness of
assistive technology for people with visual impairments:
“Somebody will [comment] ‘how are you on Facebook,
or how are you on reddit if you have vision issues’ sug-
gesting that you’re lying about having vision issues,
because you wouldn’t be on social media if you [have]
vision issues. Like literally it makes absolutely no sense.
This one actually happened today.”
Such microaggressions were alike in both ofine and online set-
tings. For example, P5 described how she struggled with medical
professionals who did not believe her when she described aspects
of her disability. P5 experienced a severe migraine episode for sev-
eral days where she became completely dehydrated. As she was
wheeled onto an ambulance, the EMTs were “actively denying” the
possibility of her having light sensitive migraines because she is
legally blind. Similarly, P19 recalled an instance that they described
as “medical gaslighting.” Health professionals assumed the pain was
“in [their] head,” leading to misdiagnosis of endometriosis which
“nearly killed” them.
That’s not a disability. Although not as extreme as accusing a
person of faking a disability, participants recalled microaggressions
when aspects of their disability experiences were invalidated. Per-
petrators had their own assumptions of what “counts” as a disability
experience. For instance, P4 had an invisible physical disability and
ASSETS ’22, October 23–26, 2022, Athens, Greece
recognized that the hidden nature of their disability caused “in-
validation from both non-disabled and disabled people.” On Tumblr,
they experienced microaggressions from the disability community
along the lines of “oh well that’s not what we meant by disability.”
P4 fretted that disclosing their disability “invited ableism” and not
disclosing it resulted in invalidation.
Some participants felt invalidated when they were compared
to people who have other forms of disability. For example, P19
shared that people often commented that she is not disabled because
she is “not in a wheelchair." P8 described a similar situation when
others invalidated her mental-health disability. P8 was diagnosed
with depression at a young age and used Facebook to write about
her experiences. She recalled an incident where a Facebook friend
accused her of lying:
"There was a time that I wrote about myself being preg-
nant and being depressed... There’s some people [who]
actually think that depression is. . . someone’s person-
ality, they don’t believe that depression is actually a
disability, so I was trying to tell everybody how I was
doing and how I was taking care of myself.. A [Face-
book] friend was like I should stop lying. . . stop saying
things that are not real. . . I should stop doing what I’m
doing because people would not appreciate what I’m
doing.” (P8)
While some participants received invalidating comments from peo-
ple, P5 felt invalidated when her posts were removed by the platform
as a result of other users fagging her posts. She shared that at least
fve of her posts that disclosed her disability were reported and
taken down on Reddit. For example, one post described an outing
with her service dog and another was about her preference in canes
that were being given by her local government agency. Given the
lack of visibility on why the post is reported, she felt frustrated
that not only was her disability experience invalidated, but also en-
tirely removed from the platform. Like P5, P1’s posts were reported
on TikTok by "trolls" who wanted to cause the platform to lock
her account without any reason. P1 felt that these experiences are
frequent for disabled activists who “have had their whole accounts
removed and completely erased because [they] talked about issues
that people are uncomfortable with."
You’re abusing your service dog. All participants who used
a service dog were accused of animal mistreatment. For instance,
after posting a video of her and her service dog, P1 had a person
accuse them of “abusing her dog and say[ing] [her dog] doesn’t
like to work” (P1). As a service dog handler, P11 echoed a similar
misconception: disabled people “force” dogs to work. In addition,
P5 has been accused of faking the need for a service dog. She was
denied access to a service dog-related Facebook group because the
administrators thought she was too young to have one.
4.1.4 Invasion of Privacy and Denial of Meaningful Relationships.
Participants recalled experiencing microaggressions in the form
of invasive, personal questions regarding their disability and their
sexual activity. In addition participants received questions about
their romantic relationship, indicating an assumption that a disabled
person cannot form meaningful, intimate relationships.
Heung et al.
What happened to you? Many participants reported being
asked personal questions which they “didn’t feel comfortable an-
swering” (P3). For example, someone on Reddit asked P5, “if you’re
blind, how do you know when you’re on your menstrual cycle”. Some
of these invasive questions relate to participants’ disabilities, for
example, about “how [they are] functioning in a wheelchair” (P18)
and “what happened to your eyes and why are you [wearing] blocky
glasses” (P16). P3 emphasized that these questions stem from a
“place of curiosity, but at the same time it’s not always good to be
curious about things like that... There are boundaries people have."
P11 called this "the line between being curious [and] being ofensive.”
Can you have sex? Another type of microaggression partici-
pants shared was being asked inappropriate and crude questions
about their sexual health and activity. P11 described instances of
being asked personal questions about sex both in-person and on
social media:
"Random strangers inquire about my sexual function..
they see a person in a wheelchair, and they think.. if
their reproductive system works properly and it’s weird
but it’s happened several times. Sometimes in the rare
instances where I have had conversations with strangers
on social media in DMs which isn’t something I do a
lot but I’ve done it a few times. . . People have questions
about things that it just makes me stop and think for a
second. . . you can ask me any question in the world and
why is that the frst thing you want to know about me."
P1 also recalled experiences of being asked about her sexual
activities. Even in public places like a grocery store, she was asked:
“can you have sex?” Participants believed that the perpetrators ob-
jectifed disabled people and took liberties, often crossing lines. P1
recalled how she was asked such questions before she turned 18
and her immediate thoughts were “why are you asking a 16 year old
girl these questions?”
Are you sure your husband loves you? Participants described
instances when the perpetrators assumed that disabled people are
incapable of forming intimate relationships. P8 explained that when
she got married, she received comments on Facebook that ranged
from disbelief to accusations of her lying about being married.
She described how insensitively some people reacted, including
comments like “You got married really? Are you sure your husband
loves you? Are you sure you are not lying?” Similarly, P1 recounted
how her disabled friend, who is a social media infuencer, received
harsh comments from people when she announced her relationship
with a non-disabled person. P1 detailed that the trolls did not view
it as a “normal human relationship,” and commented that her friend
“must be paying” the boyfriend to be in the relationship.
4.1.5 Being Ignored and Excluded Online. Participants felt a target
of microaggressions when people on social media purposely ignored
or “ghosted” them, making them feel unwelcome. Participants also
shared experiences of feeling excluded due to the inaccessibility of
social media, social media challenges, and online content in general.
Being ignored. Although the majority of microaggressions were
specifc actions, this type of microaggression was the lack of action
or inactivity on social media. P3 labeled being ghosted or “left on
read” (when the perpetrator sees the message and does not respond)
as a microaggression. During this time, his friends kept making
Nothing Micro About It: Examining Ableist Microaggressions on Social Media
excuses not to talk or hang out with him. P3 eventually concluded
that his friends didn’t want to be friends anymore because “they
think I’m diferent [from] other people...It’s not direct [aggression] but
more of a subtle thing.” Instead of being ghosted, P10 experienced a
similar microaggression where he felt ignored when people did not
comment on his Facebook post:
“I posted a recording of myself talking about ways that
the disabled can be considered in society but. . . I didn’t
get anyone who could support me in the comment sec-
tion so I felt so bad that people with disabilities can be
easily ignored on Facebook. I expected some comments
about that so that I know people care about the disabled.
I don’t [usually] delete my posts but I had to since. . . no
one cared.”
Exclusion via inaccessibility & moderation A few participants
experienced microaggressions when the perpetrators purposely
shared inaccessible content with them. P5 described how her room-
mate changed the text color to orange in a group chat on Facebook
messenger, fully knowing that the color would make the chat in-
accessible to P5. Another participant, P14, found the behavior to
be microaggressive when people tagged him in social media chal-
lenges (e.g., dancing challenges) that he cannot do because of his
disability.
Many participants mentioned that inaccessibility of social me-
dia content was a source of feeling unwelcome and excluded. For
instance, P2 felt unwelcome on social media when people in his
WhatsApp groups posted pictures without captions and alt text and
then engaged in conversations about them. Similarly, P1 found it
frustrating when content creators on TikTok and YouTube uploaded
videos without closed captions. P1 also expressed resentment on
exclusionary, content moderation systems that are ill-equipped
to meet the needs of disabled people. She described how TikTok
fagged her videos and banned her from live streaming believing
her to be “underaged because of [her] dwarfsm.” Participants also
reported experiencing microaggressions when there was a delay
addressing issues they experienced online. For example, while P1
recognized that she was blocked as a measure for minor safety, she
found it frustrating that her appeal remained unaddressed for over
a year.
4.2 The Aftermath of Microaggressions
Having described the diferent types of microaggressions disabled
people experience online, we now dive deeper into the aftermath of
experiencing a microaggression, sharing how participants reacted,
responded, and coped on the social media platform. We then share
participants’ perceptions of the long-term impact and ideas of what
social media can do to mitigate these experiences.
4.2.1 Responding to Microaggressions. There was a wide spectrum
of reactions to microaggressions. Some participants claimed “social
media is not meant for the disabled” (P14) and some “want[ed] to
delete [their] whole online existence” (P19). Others felt lonely at
frst (P3) and some were “speechless” (P5) and “so upset to the point
they didn’t even know what to do” (P8). However, with time, the
way participants reacted to microaggressions also changed. For
example, P14 described how initially microaggressions on social
media would make him depressed and heartbroken, but later he
ASSETS ’22, October 23–26, 2022, Athens, Greece
would "calm himself down, leave social media for a week, and move
on with life.” P9 referred to microaggressions as “norms,” while
other participants expressed they are “used to it already” (P12) and
“unfortunately no longer surprised” (P1).
Whether to Respond Or Ignore. Regardless of their emotional
reaction, participants decided whether to respond to or ignore
the microaggression based on the identity of the perpetrator. P2
explained how if a patronizing microaggression came from a friend,
he “would sit them down and speak to them” (P2). However if the
perpetrator was “really old or from a previous generation,” he would
not put efort into changing their mindset because “he might not be
able to change them.” Similarly, P11 called out that he might be more
likely to respond to children or “younger people" who are just being
“curious.” Nonetheless, responding to microaggressions exerted an
emotional toll and one “might not be in the mood” to handle it (P2).
P20 elaborated on the trade-ofs:
“Some people will report those comments. I always try
to have an open conversation with them, because the
reason I am on social media is to try and educate people.
So it would be remiss if I just deleted them, but then I
am getting to the point where I am like how much of
my physical energy is this costing?”
Similar to P20, P1 appreciated her social media followers who
replied to ableist comments for her. Another participant recalled
being the one to defend and respond on behalf of a disabled friend.
P18 stated that she defended her disabled friends by engaging with
people who were the perpetrators of microaggressions. She replied
to a comment about clothes her friend should not wear and de-
fended her friend by commenting:“she’s disabled and that doesn’t
change who she is so I will be glad if you people don’t throw shit
on disabled people [because] we can actually be who we want to be.”
(P18)
Some participants saw a microaggression incident as an opportu-
nity to educate the perpetrator. P8 described how she “was so pissed
that [she] had to just respond.” Instead of expressing her anger, she
described her mental disability to the perpetrator and outlined the
lived experiences of people with such disabilities, in the hopes of
“proving” to him that “this exists.” Similarly, P1 described a mantra
she told to those who asked her invasive questions: “if you aren’t
going to say it to a non-disabled person, don’t say it to a disabled
person; it’s pretty simple." P19 described how they felt responsible
not only to educate the perpetrator but also their followers:
"If I think that it’s a learning opportunity [and] a com-
mon misconception I wil screenshot it and block out the
name and post it on my story. I like to explain why this
isn’t okay and why it’s damaging. But I try not to do
that because. . . even if you block out the name, people
can stalk your videos. . . so I’m not sure, because I don’t
want to send hate back. But I don’t want to let that
opinion exist.”
Although participants recognized the beneft of responding as a
means of education and advocacy, some wondered if it was worth
the efort. P1, a disability activist on social media, questioned the
efectiveness of educating about disability on social media and felt
that “people are so caught up in their [own] mindsets. . . there is no
point in talking to them." Other participants decided that responding
ASSETS ’22, October 23–26, 2022, Athens, Greece
is pointless because they cannot change the perpetrator’s views or
their own disability (P18). Therefore, some participants felt there
was nothing more to do “but to accept it and move on.” (P14)
Some participants decided not to respond to microaggressions
because they felt that doing so would add more fuel to the fre. P8
viewed the microaggressions as a “test” of their resilience; “social
media bullies who are looking for her to get mad.” P18 would “feel
more weak” responding to the perpetrator, failing the test. Similarly,
P1 viewed the perpetrators as people who are bored and found it
most efective to not “give them attention.”
A few participants responded with humor; P11 described how
he “dish[es] it right back to them”and P5 "often snapped back with
a sarcastic comment.” For instance, on a local Facebook group, P5
warned others of a road where she “almost sprained her ankle”
because of her “vision issues.” She described her response to the
patronizing comment:
“Somebody comments well you should have just driven
to your appointment then. . . I replied: if I can’t manage
to step of a curb correctly because of my depth percep-
tion I don’t think you want to trust me on a road.”
4.2.2 Coping with Microaggressions. Participants described several
coping strategies to reduce the risks and harms of microaggressions.
While some turned to distracting or distancing themselves from
the microaggression by engaging in a hobby or fnding refuge in
entertainment, several participants shared other ways they coped
with microaggressions, which involved the use of certain social
media features and changes in social media behaviors.
Deleting, Blocking, and Reporting. Rather than replying to
the perpetrator, some participants responded by taking action on
the platform either by deleting and reporting ofensive comments,
and blocking the ofenders. Five of our participants had deleted
comments or posts. Not only did they delete the comment to “let it
go” but also as a means to forget and recover.
Several participants opted to report the posts and comments
containing microaggressions. P9 described an incident where he
had posted a quote on Facebook and the perpetrator responded that
he is “not supposed to post that kind of stuf” because of his disability.
P9 decided to not respond because he was not in the “best mood”
and did not want to infringe on the perpetrator’s “freedom of speech.”
Because the perpetrator was a Facebook friend and someone he
knew from college, P9 was also reluctant to block him. Instead P9
took a “screenshot of the message and sent it directly to Facebook.”
P9 elaborated:
"It was a friend of mine that I do not want to lose. I don’t
want it to look like I blocked [him]... but there’s a need
to actually cope with that kind of stuf. . . to stop it from
recurring..I’m not that kind of aggressive human being
[that usually] blocks or deletes. . . because you can’t just
block everybody.”
Like P9, other participants also felt that blocking and unfriending
people they cared about could burn bridges and “make enemies”
(P7).
A few participants decided to block the perpetrator to “prevent”
the microaggression from happening again. P14 explained that,
“when it happened to me I just [had] to get the guy blocked and leave
Heung et al.
social media for a week.” Some participants found blocking peo-
ple and deleting comments to be a better recourse than reporting.
For example, P4 expressed frustration with the “current reporting
systems and algorithms" and believed that microaggressions go
unchecked because “there are no repercussions for the [perpetra-
tors’] actions.” Not only are the current processes and underlying
algorithms inadequate to counteract microaggressions, they often
amplify undesirable behaviors instead of curbing them. P4 reported
and blocked perpetrators on Twitter rather than responding to
prevent other users from seeing the original post:
"If [they] take the time to quote tweets and [the tweets]
get more popular then the algorithm will support [their]
posts and then people can see the original post. . . . [They
are] putting that message to a bunch of people who
haven’t necessarily consented to reading that [and]...
what if [they] boost this person’s post to someone who
actually agrees with it.”
Changing Social Media Use. Immediately after experiencing the
microaggression, many participants took a break from social media
and refected on what they could do to avoid experiencing such
negative slights and ofensive behavior. For example, immediately
after the microaggression, P12 did not post for several days, P15
reduced “visits to Facebook,” and P16 “backed of from social media"
to spend more time with family.
Although this time-of from social media was a short-term efect,
the microaggression experiences forced participants to set bound-
aries on how they use social media, when they use social media, and
what they post. For example, P1 and P19 described that to avoid
being afected by microaggressions, they stay away from social
media right after they wake up or right before bedtime. P2 even
deleted his social media apps every few days as a habit to protect
his mental health. P4 and P6 became “passive users” and decided to
share “less information” about themselves, including their disability.
Similarly, P16 described how he does not post pictures of himself
to avoid being a target of microaggressions:
“I don’t really post my personal pictures due to [my] dis-
ability. I sometimes post memes or funny pictures. I[do]
like other people’s posts, engage [with] comments, scroll
through my feed, [and] watch other people’s stories on
instagram... If I post private pictures...on my Facebook
I make it private to reduce discrimination and prevent
unwanted questions on vision impairment."
Like P16, several participants believed that they were a target
of microaggressions because of their disability and opted to hide
their disability identity. For example, P14 avoided posting on Tik-
Tok because “there is no way [he] can do a video without showing
[his] legs." By becoming passive users and limiting the amount of
disclosure of their disability, P14 explained that this will “avoid all
sorts of embarrassment or harassment on social media." P1 also took
specifc measures in the types of content she created to avoid going
viral on TikTok. Previously she had a TikTok video with 3 million
views and “90% of the comments [were] negative” with hate, which
made her consciously post content that would not go viral. She
highlighted how a viral video has impacted what she posts:
"I have realized that the videos that go viral for me are
ones where I present an issue surrounding disability and
Nothing Micro About It: Examining Ableist Microaggressions on Social Media
say this is why this is ofensive. . . I call non-disabled
people out for the things that they do to disabled people.
Those are the kinds of videos that go viral and. . . bring
me a lot of hate from non-disabled people who feel called
out. . . if I post about my daily life, those things don’t
go viral. People think that having a viral video is the
perfect idea. . . when in reality for marginalized groups,
it brings a lot more hate than it does good so a lot of
activists out there don’t want their videos to go viral
and. . . have left platforms.” (P1)
Although some participants explicitly said they post less fre-
quently, a few stopped posting altogether. P8 explained how these
harmful experiences have led her not to post anymore on social
media even though “she wants to post but then think[s] about what
people will say.” As someone who blogs about depression, she still
writes but does not share her writing on social media because she
does not “want to see bad comments.”
4.2.3 Mitigating the Harm of Microaggressions. Participants ac-
knowledged that coping does not repair the harm caused by mi-
croaggressions. P18 emphasized this by saying:
"If there is a person that wants to block the person for
your own good, then go ahead but what has been said
has already been said. . . a person can’t take back the
words and its efect on you.”
Long-Term Impact. Participants refected on the long-term
efect of microaggressions on their self-perception. P2 described
how patronizing microaggressions have changed the way he inter-
prets compliments on social media. He explained “being patronizing
because of disability in certain cases is super evident, but there are
cases when it might not be.” As a result, he second-guesses the com-
ments that say “he’s handsome.” Others more broadly shared how
the microaggressions have afected their self-esteem, adding to the
difculties of having a disability. P3 explained:
"It [microaggressions on social media] defnitely afects
your confdence and self-esteem. It [self-esteem] drops...
You already have it pretty low because of your physical
difculties and things you have to deal with normal
people don’t have to deal with and on top of that, you
have to deal with these issues like people not wanting
to be friends with you [and] ignoring you."
Some participants felt that they were unable to stop these mi-
croaggressions. P19, who is a disability activist, described how these
microaggressions left a lasting impact on her:
“Even though I do have the language to combat [mi-
croaggressions] now, I’m in such a state of shock when-
ever it happens because it brings back memories and
because it’s just a nasty thing to experience that I don’t
feel like I’m doing as good a job, as I could, of being an
advocate, which is another like layer of guilt.”
In addition to shaping self-perceptions, microaggressions
showed participants how mainstream society perceives disabil-
ity. For example, P11 highlighted how social media in particular
exposes these views since it “gives an opportunity to see the kinds
of people out there and people will say a lot through a screen [that]
they won’t say to your face.” P19 further explained that “disability is
ASSETS ’22, October 23–26, 2022, Athens, Greece
such a taboo topic [and] we’re so dehumanized that people don’t even
realize that it matters whether we’re here or not. . . They don’t think we
can feel pain or we’re intellectually disabled.” P1 revealed what she
learned about society’s view on disability through microaggression
experiences:
"Being disabled is publicized unfortunately if you have
a platform or not everything is publicized everybody
thinks that it’s public information. So your sex life, your
relationships with other people, how your family [and]
friends treat you people just think it’s charity. . . espe-
cially with having my platform grow on social media, [I
see] the ableism that is just so prevalent in our society."
Re-Imagining Social Media. Participants shared ways in which
social media platforms could prevent the microaggression from hap-
pening in the frst place. Some participants suggested social media
platforms to have stricter community guidelines that explicitly pro-
tect disabled people and a process of removing users if they do not
abide. P15 shared that social media platforms could ask all users
to watch a short video about discrimination. Other participants
wanted social media platforms to take responsibility in disability
awareness by educating users about disability, ableism, and disabil-
ity pride month. Taking this a step further, P11 recommended that
platforms educate users when the microaggression occurs.
“I think exposure is the only way to combat microag-
gressions. . . that’s starting to happen on social media
more now but it wasn’t until the past couple of years. . .
you have more disabled creators on social media trying
to combat those narratives but I defnitely think the best
way to do it is just to have proof of those things on social
media being wrong. So you go to the places where the
microaggressions are and prove them wrong." (P11)
Other participants thought of ways to mitigate the harm once
the microaggression has been said. For example, P16 suggested
having a bot to “automatically remove” the microaggression on his
behalf, delete “rude words,” or automatically “reply to [inconsiderate]
private messages.” P5 furthers this notion by proposing a bot to
automatically send an “informative video” when people ask “how
can you be blind."
Some participants wanted improvements to reporting and mod-
eration on social media. P4 explained that there is a lack of trans-
parency and accountability when reporting a post. With their per-
sonal experience reporting on Tumblr and Twitter, they were un-
aware if “someone who is reported has actually been punished.” How-
ever, their experience was diferent in League of Legends (an online
game) where the administrators promptly addressed the matter. P4
felt “good to some extent” that the perpetrators were “minutely in-
convenienced.” Participants recognized that the perpetrators found
workarounds that prevented their comment from being removed.
For example, P4 and P20 reported that the perpetrators often re-
placed letters with numbers or special characters in hate speech
to avoid being fagged by the underlying content moderation al-
gorithms. In addition, some participants felt frustrated when the
algorithms removed hateful posts without context. For instance,
on a Facebook group to talk about abuse and trauma, P5 had a
post removed because it included hateful words that was a part
of someone’s experience rather than being directed at anyone. P5
ASSETS ’22, October 23–26, 2022, Athens, Greece
explained how exasperating it was when Facebook kept removing
the post and every time they appealed the group got more “strikes.”
P1 also noted issues with the slow appeal process and wishing that
there was a better way to verify age. P19 explained why ableist
microaggressions are hard to automatically identify:
“With disability, the language is so vague. You can’t
really pick out a word that they use. . . that’s a microag-
gression and that’s a hate crime. Because it’s the tone.
There’s no button for saying they’ve implied that it’s
okay that I die. But clearly that’s not an okay thing to
say, but there’s no way of reporting that kind of thing."
5 DISCUSSION
This paper has described 12 archetypes of ableist microaggressions
experienced on social media, unpacked how disabled people re-
spond to and cope with these incidents, and explained how they
reimagine a more welcoming social media. We now discuss how
microaggressive experiences are unique to social media and present
design recommendations on how social media platforms could me-
diate and perhaps even prevent these harmful experiences.
5.1 Ableist Microaggressions on Social Media
5.1.1 Categorizing Microaggressions. “Being ignored” and “exclu-
sion via inaccessibility and moderation” are the two novel types
of microaggressions our study uncovered. “Being ignored” occurs
when a disabled person is neglected or ghosted by others. This is a
particularly interesting microaggression, because it is characterized
by the lack of action, whereas prior work has generally conceptu-
alized microaggressions as a problematic action. Thus, a pointed
silence when encountering information about a person’s disability
can be just as hurtful as a negative response. A more appropriate
response would involve sincere engagement with and a willingness
to learn about disability and ableism. “Exclusion via inaccessibility
and moderation” refers to two instances: 1) when people share inac-
cessible content with a disabled person despite knowing about their
access needs; and 2) when moderation algorithms ban disabled users
or fag their content due to normative expectations around what a
legitimate user looks like. Participants reported feeling frustrated
by exclusionary experiences that arise from content moderation
algorithms and policies, because the mechanisms intended to pro-
tect users result in making disabled people feel unwelcome and
excluded on social media.
We confrm that the ableist microaggressions identifed in prior
work also occur on social media. Five of our archetypes map di-
rectly onto previously-identifed ofine microaggressions. “You’re
so inspirational” corresponds to patronization, while “where’s your
mom?” embodies infantilization [29]. Similarly, “you’re lying about
your disability,” “that’s not a disability,” and “you’re abusing your
service dog” represent specifc manifestations of the denial of dis-
ability experience [29].
Five other archetypes could be categorized into multiple types
of existing microaggressions. The archetype “can someone like you
do that/wear that?” simultaneously makes assumptions about a
disabled person and ignores other aspects of a disabled person’s
identity. Hence, it embodies the spread efect and denial of personal
identity patterns [29]. The archetype “I would kill myself if I was
Heung et al.
disabled” is related to the helplessness pattern, in that it character-
izes a frantic urge on the part of a nondisabled person to cure or
rid another person of their disability [29]. The eugenic undertone
of this archetype also questions the worth and very existence of
disabled people, exemplifying second-class citizenship [29, 49]. The
archetypes “what happened to you,” “can you have sex,” and “are
your sure your husband loves you?” represent instances where
disabled people are asked invasive and ofensive questions about
a person’s life that are typically considered private. These three
archetypes map onto the denial of privacy pattern [29]. The ap-
parent comfort with which nondisabled strangers inquire into a
disabled person’s history or sexual activity betrays the understand-
ing that disabled people can be treated as objects of frank curiosity.
These two archetypes also embody an implicit sense of second-class
citizenship for disabled people [29]. Questioning the legitimacy of
intimate relationships denies the possibility that a disabled person
could be a sexual being and intimate partner, and thus corresponds
to the desexualization and spread efect patterns [29]. We support
Bell’s assertion that the types of microaggressions are not mutually
exclusive, as evidenced by the instances in which our archetypes
embodied more than one type [5].
5.1.2 Responding to Microaggressions. Keller and Galgay wrote
that “we understand least” the “strategies [disabled people] use to
deal with” microaggressions [29]. Our work addresses this critical
gap by examining: how disabled people cope with microaggressions
and the strategies they employ to counter, take action against, and
respond to microaggressions in the moment and over the longer
term. We distinguish between and describe three temporally distinct
responses to microaggressions – the reaction that immediately
follows a microaggression, coping mechanisms that people use
to move past a microaggression, and longer-term strategies that
people use to respond to microaggressions and prevent them from
occurring in the future.
Our fndings confrm known response strategies including ac-
tive self-advocacy around disability; parsing the intent behind a
microaggression to determine whether it was meant to be hurtful;
educating others to counter stereotypes and myths about disabil-
ity; responding via humor, sarcasm, or direct communication; and
ignoring microaggressions and avoiding perpetrators [5]. Addition-
ally, we uncover response strategies unique to social media that
disabled people routinely use. These include modulating the fre-
quency and timing of social media use; being cautious about the
topic and format of the content they share; blocking and report-
ing perpetrators or problematic content; and deleting social media
content – both their own and/or the ofender’s. These strategies
align with recent work on understanding harassment experienced
by content creators and Black women [38, 51]. We also unpack the
decision-making that goes into choosing which strategy to employ,
fnding that context and the perceived likelihood of being able to
change the perpetrator’s mind plays a key role in this decision.
Finally, our participants point to the labor that goes into experienc-
ing microaggressions. Not only does the recipient have to spend
time and energy fguring out whether an action truly was a mi-
croaggression, but they must also cope with their reaction, decide
how to respond, and perhaps advocate for themselves with the
Nothing Micro About It: Examining Ableist Microaggressions on Social Media
perpetrator or platform, thereby opening themselves up to more
microaggressions.
5.1.3 Comparing Experiences of Online and Ofline Microaggres-
sions. Extending the study of ableist microaggressions to social
media provides us with an opportunity to begin comparing how
microaggressions are experienced across online and ofine contexts.
In both online and ofine contexts, participants found it challenging
to defnitively establish an incident as an ableist microaggression.
Microaggressions are subtle and rarely contain slurs or curse words,
relying instead on the tone and form. For instance, a seemingly
harmless question or admiration make it difcult to identify the
perpetrator’s intent and make disabled people question their felt
experience: intentional hurt versus innocent curiosity and back-
handed remark versus compliment. Participants also reported that
ableist microaggressions are usually unintentional. Prior work has
interpreted the lack of intent to mean that the perpetrator does
not mean to cause harm [29]. However, our participants said that
perpetrators do intend what they say or do, but they typically have
not thought through the implications of their actions.
Participants described that ableist microaggressions are more
frequent online, perhaps due to the lack of consequences for actions
on social media and the anonymity aforded by some platforms. Fur-
ther, participants worried that the sheer scale and visibility of social
media means that online microaggressions are much more likely to
afect the views of observers. That is, online microaggressions are
more worrisome due to their ability to shape the views of many,
leading to larger-scale ramifcations in terms of how these users
will think about and treat disabled people in the future. Participants
also noted that perpetrators may continue harassing them on other
platforms. This cross-platform harassment makes it pointless to
delete individual posts and comments.
However, participants reported that online microaggressions
can be easier to ignore and move past than ofine ones. The rela-
tively public nature of online microaggressions means that there is
a higher chance that disabled people can rely on friends, disability
advocates, and allies to intervene on their behalf. Further, some
participants reported that perpetrators of online microaggressions
rarely have the power to actually deny disabled people a material
resource. This is unlike ofine microaggressions where the disabled
person may rely on the perpetrator for something such as food,
transportation, or medical treatment. In these cases, the perpetra-
tor may also occupy a position of authority, granting a veneer of
legitimacy to the microaggression and its underlying message.
Finally, the same type of microaggression can manifest difer-
ently in online and ofine contexts. The archetype “that’s not a
disability,” which corresponds to a denial of disability experience
[29], typically manifests ofine in the form of words or actions that
invalidate a person’s disability experience. This archetype takes on a
diferent form on social media, that of posts and users being fagged
as inappropriate by other users. Participants noted that posts dis-
cussing their own disability experience or sharing disability-related
content were often reported and removed on social media platforms
without any explanation. Participants also reported cases of dis-
abled activists having their accounts banned for sharing disability
advocacy content. This resonates with Haimson et al. who found
that transgender and Black social media users often had content
ASSETS ’22, October 23–26, 2022, Athens, Greece
referring to systemic marginalization removed despite adhering to
platform content policies [22]. We propose that scholars continue
to develop this comparison between online and ofine microaggres-
sions for various minoritized groups, due to its potential to yield
broader insights about marginality, intersectionality, and power
dynamics in the digital world.
5.2 Design Recommendations
Currently, most major social media platforms incorporate moder-
ation tools (e.g., reporting, community policies) [41] and educate
their users on hate speech and harassment [14]. Therefore, it is
important to consider how social media platforms might account
for ableist microaggressions. We discuss difculties in considering
microaggressions within online moderation and recommend that
social media designers and researchers educate users on disability
and ableism.
5.2.1 The Challenge of Microaggressions Within Online Modera-
tion. Prior work on social media moderation documented faws
within current online moderation systems, such as unfair and false
reporting [22, 27, 46], mislabelling harassment [7], and lack of trans-
parency in the harassment reporting process [7]. Our participants
echoed current faws in moderation, leading us to wonder how mi-
croaggressions should ft within these systems. Several participants
in our study felt uncomfortable blocking or reporting perpetrators,
due to the unintentional nature of microaggressions and to avoid
severing personal relationships (see Section 4.2.1). Others felt that
everyone was entitled to their opinion, and blocking was unneces-
sary. This aligns with prior work viewing online harassment with a
restorative justice lens: removing perpetrators of microaggressions,
or banning their profle may not serve justice to disabled people
[47], or align with their preferences and beliefs. Overall current
moderation tools (i.e., blocking and reporting) are designed to ac-
count for overt cases of harassment. Future work should continue
to investigate preferences for handling microaggressions and more
subtle forms of discrimination.
Due to their inherent subtlety and nuance, microaggressions
may be difcult to detect automatically. Microaggressions are often
personal, and may be ofensive given prior history and type of
relationship with the perpetrator. Moderators or models may not
be privy to this information, and hence may be unable to classify
some microaggressions.
Online moderators may lack interpersonal and social context of
microaggressions. For instance, microaggressions like “what hap-
pened to you?” or “where’s your mom?” are difcult to identify as
microaggressions without additional context. On the other hand a
microaggression such as "I would kill myself if I was disabled" can be
clearly labelled as an ableist microaggression. Since the moderator
lacks interpersonal context and relationship, we recognize that they
may be unable to reduce the emotional toll that microaggressive
experiences can have. Our participants also shared that they relied
on friends and other users to support them by responding to the
perpetrator. Mahar et al. have also used an approach that recruits
friends to help flter messages during harassment attacks [33]. Since
microaggressions are highly contextual to interpersonal history and
ASSETS ’22, October 23–26, 2022, Athens, Greece
styles of interaction, crowdsourced and even friendsourced mod-
erators may be unable to identify some microaggressions without
further context.
Similarly, microaggressions are more complex to pinpoint by
models compared to hate speech. Existing models are mostly trained
to detect hate speech [19, 20, 24, 44]; however, some researchers
are developing machine learning techniques to identify microag-
gressions (based on gender, race, and sexuality) [10]. We encourage
researchers to continue investigating ways of identifying ableist
microaggressions online.
5.2.2 Designing for Disability Education and Awareness. Partici-
pants suggested that social media should educate users about dis-
ability and ableism, as a preventive measure to reduce microag-
gressions. For example, some wanted improvements in current
community guidelines, wishing for explicit policies around ableism.
Such community norms can educate and disincentivize perpetrators
from being ableist [41]. One participant proposed that the platform,
instead of other users, can combat microaggressions when they
occur. Similarly, Bennett calls on online dating platforms to dispel
the notion that disabled people are asexual [6]. Although education
around disability and ableism should extend beyond online set-
tings, social media platforms have a unique opportunity to combat
harassment and discrimination by educating their users.
In the event that ableist microaggressions could be detected,
social media platforms can combat ableist microaggressions proac-
tively by nudging the perpetrator, asking them to refect if their
comment or post has ableist microaggressions. Perhaps this nudge
can activate only when there are indicators of ableism. This nudge
can educate perpetrators on the potential harms before they engage
with disabled users on the platform. Other scholars have explored
educating users through nudges to alter future behaviors [53] and
such lightweight interventions while sharing have been found to
be efective in reducing the propagation of fake news [26]. For
example, Jahanbakhsh et al. explored the use of nudges in fact-
checking information on social media and experimented with a
variety of behavioral nudges; such as using checkboxes to assess
the information or implementing text with rationales on why the in-
formation is inaccurate. We recommend that designers explore how
such lightweight interventions can be adapted to prevent ableist
microaggressions from being sent in the frst place.
Social media platforms also need reactive approaches to mitigate
the harms of ableist microaggressions. Consider the microaggres-
sion: “can disabled people have sex?” The platform can present
information that: 1) answers the microaggressive question and
replies for the disabled person reducing the emotional labor, and 2)
debunks the assumption that disabled people are uninterested in or
incapable of sex. If the ableist microaggression is a post, there could
be a public correction, much like credibility indicators to combat
misinformation and fake news [26, 46]. A public correction would
not censor or augment the original post, but include more informa-
tion about the content, educating all users on ableist stereotypes
and misconceptions. The proposal to contradict an existing dis-
criminatory comment on social media is similar to counter-speech,
which is used to combat hate-speech online [18, 23]. A recent study
has found that counter-speech reduces the probability of other
Heung et al.
users becoming hateful [23]. This further motivates a design oppor-
tunity for social media platforms to contradict narratives behind
ableist microaggressions. Similar to traditional media, inaccurate
portrayals of disability on social media perpetuate ableism [43].
In this study, we were limited to self-reported experiences of
ableist microaggressions. Future work should quantitatively inves-
tigate the breadth, frequency, and impact of these microaggressions,
and examine disabled people’s preferences in moderating microag-
gressions online. Visibly disabled people and disabled women are
known to experience more ableist microaggressions [4, 28], but
other aspects of a person’s identity might infuence the kinds and
frequency of microaggressions that they are targeted for. We en-
courage researchers to consider intersecting identities of those who
may be more susceptible to these harmful experiences.
6 CONCLUSION
Our work is the frst attempt to examine how disabled people ex-
perience and cope with ableist microaggressions on social media.
From our interviews with 20 disabled people, we uncovered 12 mi-
croaggression archetypes, validating in-person microaggressions
and introducing manifestations that are unique to social media.
Some participants responded to perpetrators using platform fea-
tures, while others either ignored them or engaged with them in
the hopes of changing the perpetrators’ mindset about disability.
Overall participants felt that experiencing ableist microaggressions
afected their wellbeing and heavily transformed the way they used
social media. We see our work as a starting point in examining
exclusionary experiences of disabled people on social media. As we
work towards inclusion, we call upon researchers and designers to
consider disability as a facet of diversity along with race, gender,
and sexuality.
ACKNOWLEDGMENTS
We are thankful to our participants who openly shared their expe-
riences. We thank Gillian Hayes and our anonymous reviewers for
helping improve the paper. This work was supported in part by the
National Science Foundation Graduate Research Fellowship and
the University of California President’s Postdoctoral Fellowship.
REFERENCES
[1] 1990. Americans With Disabilities Act. https://www.ada.gov/pubs/adastatute08.
htm
[2] 2011. The Sage Handbook of Qualitative Research.
[3] Raya Al-Jadir. 2019. The damaging efect of social media positivity on disabled
people. https://disabilityhorizons.com/2019/10/the-damaging-afect-of-social-
media-positivity-on-disabled-people/
[4] Deniz Aydemir-Döke and James T. Herbert. 2021. Development and Validation
of the Ableist Microaggression Impact Questionnaire. Rehabilitation Counseling
Bulletin (2021). https://doi.org/10.1177/00343552211014259
[5] Ayoka Bell. 2013. Nothing about us without us: A qualitative investigation of
the experiences of being a target of ableist microaggressions. (2013). https:
//www.proquest.com/docview/1536397558
[6] Cynthia L. Bennett. 2017. Disability-Disclosure Preferences and Practices in
Online Dating Communities. XRDS 24, 2 (dec 2017), 30–33. https://doi.org/10.
1145/3155120
[7] Lindsay Blackwell, Jill Dimond, Sarita Schoenebeck, and Clif Lampe. 2017. Clas-
sifcation and Its Consequences for Online Harassment: Design Insights from
HeartMob. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 24 (dec 2017),
19 pages. https://doi.org/10.1145/3134659
[8] Lindsay Blackwell, Jean Hardy, Tawfq Ammari, Tifany Veinot, Clif Lampe, and
Sarita Schoenebeck. 2016. LGBT Parents and Social Media: Advocacy, Privacy,
and Disclosure during Shifting Social Movements. In Proceedings of the 2016 CHI
Nothing Micro About It: Examining Ableist Microaggressions on Social Media
Conference on Human Factors in Computing Systems (San Jose, California, USA)
(CHI ’16). Association for Computing Machinery, New York, NY, USA, 610–622.
https://doi.org/10.1145/2858036.2858342
[9] Virginia Braun and Victoria Clarke. 2006. Using thematic
analysis in psychology. Qualitative Research in Psychology 3,
2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
arXiv:https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa
[10] Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding
Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social
Media Posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural
Language Processing and the 9th International Joint Conference on Natural Lan-
guage Processing (EMNLP-IJCNLP). Association for Computational Linguistics,
Hong Kong, China, 1664–1674. https://doi.org/10.18653/v1/D19-1176
[11] Carol J. Gill Carmit-Noa Shpigelman. 2014. "Facebook Use by Persons with
Disabilities". Journal of Computer-Mediated Communication 19 (2014), 610–624.
Issue 3. https://doi-org.proxy.library.cornell.edu/10.1111/jcc4.12059
[12] Jessica Caron and Janice Light. 2016. “Social Media has Opened a World of ‘Open
communication:’” experiences of Adults with Cerebral Palsy who use Augmenta-
tive and Alternative Communication and Social Media. Augmentative and Alter-
native Communication 32, 1 (2016), 25–40. https://doi.org/10.3109/07434618.2015.
1052887 arXiv:https://doi.org/10.3109/07434618.2015.1052887 PMID: 26056722.
[13] Sue Caton and Melanie Chapman. 2016. The use of social media and people with
intellectual disability: A systematic review and thematic analysis. Journal of
Intellectual & Developmental Disability 41, 2 (2016), 125–139. https://doi.org/10.
3109/13668250.2016.1153052 arXiv:https://doi.org/10.3109/13668250.2016.1153052
[14] Meta Transparency Center. 2022. "Hate Speech". (2022). https://transparency.fb.
com/policies/community-standards/hate-speech/
[15] Mohit Chandra, Manvith Reddy, Shradha Sehgal, Saurabh Gupta, Arun Balaji
Buduru, and Ponnurangam Kumaraguru. 2021. "A Virus Has No Religion": Ana-
lyzing Islamophobia on Twitter During the COVID-19 Outbreak. In Proceedings
of the 32nd ACM Conference on Hypertext and Social Media (Virtual Event, USA)
(HT ’21). Association for Computing Machinery, New York, NY, USA, 67–77.
https://doi.org/10.1145/3465336.3475111
[16] Dasom Choi, Uichin Lee, and Hwajung Hong. 2022. “It’s Not Wrong, but I’m
Quite Disappointed”: Toward an Inclusive Algorithmic Experience for Content
Creators with Disabilities. In Proceedings of the 2022 CHI Conference on Human
Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association
for Computing Machinery, New York, NY, USA, Article 593, 19 pages. https:
//doi.org/10.1145/3491102.3517574
[17] John W. Creswell and Dana L. Miller. 2000. Determining Validity in Qualitative
Inquiry. Theory Into Practice 39, 3 (2000), 124–130. https://doi.org/10.1207/
s15430421tip3903_2 arXiv:https://doi.org/10.1207/s15430421tip3903_2
[18] Mithun Das, Binny Mathew, Punyajoy Saha, Pawan Goyal, and Animesh Mukher-
jee. 2020. Hate Speech in Online Social Media. SIGWEB Newsl. Autumn, Article
4 (nov 2020), 8 pages. https://doi.org/10.1145/3427478.3427482
[19] Paula Fortuna and Sérgio Nunes. 2018. A Survey on Automatic Detection of
Hate Speech in Text. ACM Comput. Surv. 51, 4, Article 85 (jul 2018), 30 pages.
https://doi.org/10.1145/3232676
[20] Jennifer Golbeck, Zahra Ashktorab, Rashad O. Banjo, Alexandra Berlinger, Sid-
dharth Bhagwan, Cody Buntain, Paul Cheakalos, Alicia A. Geller, Quint Gergory,
Rajesh Kumar Gnanasekaran, Raja Rajan Gunasekaran, Kelly M. Hofman, Jenny
Hottle, Vichita Jienjitlert, Shivika Khare, Ryan Lau, Marianna J. Martindale, Shal-
mali Naik, Heather L. Nixon, Piyush Ramachandran, Kristine M. Rogers, Lisa
Rogers, Meghna Sardana Sarin, Gaurav Shahane, Jayanee Thanki, Priyanka Ven-
gataraman, Zijian Wan, and Derek Michael Wu. 2017. A Large Labeled Corpus
for Online Harassment Research. In Proceedings of the 2017 ACM on Web Science
Conference (Troy, New York, USA) (WebSci ’17). Association for Computing Ma-
chinery, New York, NY, USA, 229–233. https://doi.org/10.1145/3091478.3091509
[21] Oliver L. Haimson. 2017. Digital and Physical Barriers to Changing Identities.
XRDS 24, 2 (dec 2017), 26–29. https://doi.org/10.1145/3155118
[22] Oliver L. Haimson, Daniel Delmonaco, Peipei Nie, and Andrea Wegner. 2021.
Disproportionate Removals and Difering Content Moderation Experiences for
Conservative, Transgender, and Black Social Media Users: Marginalization and
Moderation Gray Areas. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article
466 (oct 2021), 35 pages. https://doi.org/10.1145/3479610
[23] Bing He, Caleb Ziems, Sandeep Soni, Naren Ramakrishnan, Diyi Yang, and Srijan
Kumar. 2021. Racism is a Virus: Anti-Asian Hate and Counterspeech in Social
Media during the COVID-19 Crisis. In Proceedings of the 2021 IEEE/ACM Interna-
tional Conference on Advances in Social Networks Analysis and Mining (Virtual
Event, Netherlands) (ASONAM ’21). Association for Computing Machinery, New
York, NY, USA, 90–94. https://doi.org/10.1145/3487351.3488324
[24] Jackson Houston, Peter Peterson, and Catherine Reich. 2020. Hate Speech De-
tection in Twitter: A Selectively Trained Ensemble Method. Master’s thesis. USA.
Advisor(s) Richard, Maclin,. AAI27997273.
[25] Amanda Hynan, Juliet Goldbart, and Janice Murray. 2015. A grounded theory
of Internet and social media use by young people who use augmentative and
alternative communication (AAC). Disability and Rehabilitation 37, 17 (2015),
1559–1575. https://doi.org/10.3109/09638288.2015.1056387
ASSETS ’22, October 23–26, 2022, Athens, Greece
[26] Farnaz Jahanbakhsh, Amy X. Zhang, Adam J. Berinsky, Gordon Pennycook,
David G. Rand, and David R. Karger. 2021. Exploring Lightweight Interventions
at Posting Time to Reduce the Sharing of Misinformation on Social Media. Proc.
ACM Hum.-Comput. Interact. 5, CSCW1, Article 18 (apr 2021), 42 pages. https:
//doi.org/10.1145/3449092
[27] Shagun Jhaver, Sucheta Ghoshal, Amy Bruckman, and Eric Gilbert. 2018. Online
Harassment and Content Moderation: The Case of Blocklists. 25, 2, Article 12
(mar 2018), 33 pages. https://doi.org/10.1145/3185593
[28] Shanna Katttari. 2020. Ableist Microaggressions and the Mental Health of
Disabled Adults. Community Mental Health Journal 56 (2020), 1170–1179.
https://doi.org/10.1007/s10597-020-00615-6
[29] Richard Keller and Corinne Galgay. 2010. Microaggressive Experiences of People
with Disabilities. Microaggressions and Marginality: Manifestation, Dynamics, and
Impact (2010), 241–267.
[30] Ugur Kursuncu, Manas Gaur, Carlos Castillo, Amanuel Alambo, Krishnaprasad
Thirunarayan, Valerie Shalin, Dilshod Achilov, I. Budak Arpinar, and Amit Sheth.
2019. Modeling Islamist Extremist Communications on Social Media Using
Contextual Dimensions: Religion, Ideology, and Hate. Proc. ACM Hum.-Comput.
Interact. 3, CSCW, Article 151 (nov 2019), 22 pages. https://doi.org/10.1145/
3359253
[31] Amanda Lenhart, Michele Ybarra, Kathryn Zickur, and Myeshia Price-Feeney.
2016. Online Harassment, Digital Abuse, and Cyberstalking in America. Data &
Society (2016). https://www.datasociety.net/pubs/oh/Online_Harassment_2016.
pdf
[32] Kayla Lett, Andreea Tamaian, and Bridget Klest. 2020. Impact of ableist microag-
gressions on university students with self-identifed disabilities. Disability &
Society 35, 9 (2020), 1441–1456. https://doi.org/10.1080/09687599.2019.1680344
arXiv:https://doi.org/10.1080/09687599.2019.1680344
[33] Kaitlin Mahar, Amy X. Zhang, and David Karger. 2018. Squadbox: A Tool to
Combat Email Harassment Using Friendsourced Moderation. In Proceedings of
the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC,
Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA,
1–13. https://doi.org/10.1145/3173574.3174160
[34] Jennifer Mankof, Gillian R. Hayes, and Devva Kasnitz. 2010. Disability Studies as
a Source of Critical Inquiry for the Field of Assistive Technology. In Proceedings of
the 12th International ACM SIGACCESS Conference on Computers and Accessibility
(Orlando, Florida, USA) (ASSETS ’10). Association for Computing Machinery,
New York, NY, USA, 3–10. https://doi.org/10.1145/1878803.1878807
[35] Ming Cui Mathew D. Gayman, Robyn Lewis Brown. 2011. Depressive symptoms
and bodily pain: the role of physical disability and social stress. Stress & Health
27 (2011), 52–63. https://doi.org/10.1002/smi.1319
[36] Hannah Miller, Heather Buhr, Chris Johnson, and Jerry Hoepner. 2013. Aphasi-
aWeb: A Social Network for Individuals with Aphasia. In Proceedings of the 15th
International ACM SIGACCESS Conference on Computers and Accessibility (Belle-
vue, Washington) (ASSETS ’13). Association for Computing Machinery, New York,
NY, USA, Article 4, 8 pages. https://doi.org/10.1145/2513383.2513439
[37] Mainack Mondal, Leandro Araújo Silva, and Fabrício Benevenuto. 2017. A
Measurement Study of Hate Speech in Social Media. In Proceedings of the
28th ACM Conference on Hypertext and Social Media (Prague, Czech Republic)
(HT ’17). Association for Computing Machinery, New York, NY, USA, 85–94.
https://doi.org/10.1145/3078714.3078723
[38] Tyler Musgrave, Alia Cummings, and Sarita Schoenebeck. 2022. Experiences of
Harm, Healing, and Joy among Black Women and Femmes on Social Media. In
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
(New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New
York, NY, USA, Article 240, 17 pages. https://doi.org/10.1145/3491102.3517608
[39] Fayika Farhat Nova, MD. Rashidujjaman Rifat, Pratyasha Saha, Syed Ishtiaque
Ahmed, and Shion Guha. 2019. Online Sexual Harassment over Anonymous
Social Media in Bangladesh. In Proceedings of the Tenth International Conference
on Information and Communication Technologies and Development (Ahmedabad,
India) (ICTD ’19). Association for Computing Machinery, New York, NY, USA,
Article 1, 12 pages. https://doi.org/10.1145/3287098.3287107
[40] Rhoda Olkin, H’Sien Hayward, Melody Schaf Abene, and Goldie VanHeel. 2019.
The Experiences of Microaggressions against Women with Visible and Invisible
Disabilities. Journal of Social Issues 75 (2019), 757–787. https://doi.org/10.1111/
josi.12342
[41] Jessica A. Pater, Moon K. Kim, Elizabeth D. Mynatt, and Casey Fiesler. 2016. Char-
acterizations of Online Harassment: Comparing Policies Across Social Media
Platforms. In Proceedings of the 19th International Conference on Supporting Group
Work (Sanibel Island, Florida, USA) (GROUP ’16). Association for Computing Ma-
chinery, New York, NY, USA, 369–374. https://doi.org/10.1145/2957276.2957297
[42] Chester M. Pierce, Jean V. Carew, Diane Pierce-Gonzalez, and Deborah Wills.
1977. An Experiment in Racism: TV Commercials. Education and Ur-
ban Society 10, 1 (1977), 61–87. https://doi.org/10.1177/001312457701000105
arXiv:https://doi.org/10.1177/001312457701000105
[43] Rebecca Renwick. 2016. Rarely Seen, Seldom Heard: People with Intellectual
Disabilities in the Mass Media. Palgrave Macmillan UK, London, 61–75. https:
//doi.org/10.1057/978-1-137-52499-7_5
ASSETS ’22, October 23–26, 2022, Athens, Greece
[44] N.D.T. Ruwandika and A.R. Weerasinghe. 2018. Identifcation of Hate Speech
in Social Media. In 2018 18th International Conference on Advances in ICT for
Emerging Regions (ICTer). 273–278. https://doi.org/10.1109/ICTER.2018.8615517
[45] Matthew J. Salganik. 2017. Bit by Bit: Social Research in the Digital Age (open
review edition ed.). Princeton University Press, Princeton, NJ.
[46] Emily Saltz, Claire R Leibowicz, and Claire Wardle. 2021. Encounters with Visual
Misinformation and Labels Across Platforms: An Interview and Diary Study to
Inform Ecosystem Approaches to Misinformation Interventions. Association for
Computing Machinery, New York, NY, USA. https://doi-org.proxy.library.cornell.
edu/10.1145/3411763.3451807
[47] Sarita Schoenebeck, Oliver L Haimson, and Lisa Nakamura. 2021. Drawing
from justice theories to support targets of online harassment. New Media
& Society 23, 5 (2021), 1278–1300. https://doi.org/10.1177/1461444820913122
arXiv:https://doi.org/10.1177/1461444820913122
[48] Woosuk Seo and Hyunggu Jung. 2017. Exploring the Community of Blind or
Visually Impaired People on YouTube. In Proceedings of the 19th International
ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland,
USA) (ASSETS ’17). Association for Computing Machinery, New York, NY, USA,
371–372. https://doi.org/10.1145/3132525.3134801
[49] Sharon L Snyder and David T Mitchell. 2006. Cultural Locations of Disability.
University of Chicago Press.
[50] Derald Wing Sue, Christina M Capodilupo, Gina C Torino, Jennifer M Bucceri,
Aisha Holder, Kevin L Nadal, and Marta Esquilin. 2007. Racial microaggressions
in everyday life: implications for clinical practice. American Psychologist 62, 4
(2007), 271.
Heung et al.
[51] Kurt Thomas, Patrick Gage Kelley, Sunny Consolvo, Patrawat Samermit, and
Elie Bursztein. 2022. “It’s Common and a Part of Being a Content Creator”:
Understanding How Creators Experience and Cope with Hate and Harassment
Online. In Proceedings of the 2022 CHI Conference on Human Factors in Computing
Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery,
New York, NY, USA, Article 121, 15 pages. https://doi.org/10.1145/3491102.
3501879
[52] Stefan Timmermans and Iddo Tavory. 2012. Theory Construction in Qual-
itative Research: From Grounded Theory to Abductive Analysis. Sociologi-
cal Theory 30, 3 (2012), 167–186. https://doi.org/10.1177/0735275112457914
arXiv:https://doi.org/10.1177/0735275112457914
[53] Nicole A. Vincent and Emma A. Jane. 2017. Beyond law: protecting victims through
engineering and design. Number 30 in Routledge Studies in Crime and Society.
Routledge, Taylor and Francis Group, United Kingdom, 209–223.
[54] Jessica Vitak, Kalyani Chadha, Linda Steiner, and Zahra Ashktorab. 2017. Identify-
ing Women’s Experiences With and Strategies for Mitigating Negative Efects of
Online Harassment. In Proceedings of the 2017 ACM Conference on Computer Sup-
ported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW
’17). Association for Computing Machinery, New York, NY, USA, 1231–1245.
https://doi.org/10.1145/2998181.2998337
[55] Qunfang Wu, Louisa Kayah Williams, Ellen Simpson, and Bryan Semaan. 2022.
Conversations About Crime: Re-Enforcing and Fighting Against Platformed
Racism on Reddit. Proc. ACM Hum.-Comput. Interact. 6, CSCW1, Article 54 (apr
2022), 38 pages. https://doi.org/10.1145/3512901
[56] Susan Xiong. 2016. The Development of the Disability Microaggressions Scale.
(2016). https://hdl.handle.net/10027/20980
 Authoring accessible media content on social networks
Letícia Seixas Pereira José Coelho André Rodrigues
lspereira@fc.ul.pt jrocoelho@fc.ul.pt afrodrigues@fc.ul.pt
LASIGE, Faculdade de Ciências, LASIGE, Faculdade de Ciências, LASIGE, Faculdade de Ciências,
Universidade de Lisboa Universidade de Lisboa Universidade de Lisboa
Portugal Portugal Portugal

João Guerreiro Tiago Guerreiro Carlos Duarte

jpguerreiro@fc.ul.pt tjvg@di.fc.ul.pt caduarte@fc.ul.pt
LASIGE, Faculdade de Ciências, LASIGE, Faculdade de Ciências, LASIGE, Faculdade de Ciências,
Universidade de Lisboa Universidade de Lisboa Universidade de Lisboa
Portugal Portugal Portugal

ABSTRACT
User-generated content plays a key role in social networking, al-
lowing a more active participation, socialisation, and collaboration
among users. In particular, media content has been gaining a lot
of ground, allowing users to express themselves through diferent
types of formats such as images, GIFs and videos. The majority of
this growing type of online visual content remains inaccessible to
a part of the population, in particular for those who have a visual
disability, despite available tools to mitigate this source of exclusion.
We sought to understand how people are perceiving this type of
online content in their networks and how support tools are being
used. To do so, we conducted a user study, with 258 social network
users through an online questionnaire, followed by interviews with
20 of them – 7 blind users and 13 sighted users. Results show how
the diferent approaches being employed by major platforms may
not be sufcient to address this issue properly. Our fndings reveal
that users are not always aware of the possibility and the bene-
fts of adopting accessible practices. From the general perspectives
of end-users experiencing accessible practices, concerning barri-
ers encountered, and motivational factors, we also discuss further
approaches to create more user engagement and awareness.
CCS CONCEPTS
• Human-centered computing → Accessibility; Social media.
KEYWORDS
accessibility, social media, visual content, user-generated content
ACM Reference Format:
Letícia Seixas Pereira, José Coelho, André Rodrigues, João Guerreiro, Tiago
Guerreiro, and Carlos Duarte. 2022. Authoring accessible media content on
social networks. In The 24th International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece.
ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3517428.3544882
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3544882
1 INTRODUCTION
Social networks have permeated every facet of modern society’s
daily life. Following the recent events of the COVID-19 pandemic,
their usage is at record levels [33]. Facebook has reported an in-
crease of approximately 6% users over the previous year reaching
1.93 billion daily active users in the third quarter of 2021 [10]. The
possibility to engage with one another, while physically distant,
is at the moment more relevant than it ever was. For people with
disabilities, these platforms also play an important role in disability
advocacy, as it provides a vehicle for meeting new contacts with
disabilities, learning about issues and news related to it, and dis-
cussing accessibility challenges and solutions for improving social
media inclusion [12, 36, 38].
Despite the contributions and improvements promoted by social
networks in recent years [37], main platforms still present substan-
tial accessibility barriers for users with disabilities. The complexity
of their interfaces compared to many typical websites comes from
the fact that they are primarily composed of user-generated content.
As such, for these platforms to be truly accessible they have to go
beyond ensuring the content they control and produce is accessible:
they need to ensure the content their users produce is also accessi-
ble. This is especially relevant for people with visual impairments
given the prevalence of user-generated content that is mostly visual
(e.g., images, GIFs, videos). As observed by Voykinska et al. [36],
in order to fully engage with visual content, blind users need to
overcome several challenges, in particular, the frequent lack of al-
ternative descriptions in photos, essential to provide them proper
contextual information. Most of them rely on workarounds such
as searching for meta-data (author, geo-localization and even com-
ments posted by other users), or reaching out to a nearby friend or
family member to assist them. Conversely, in a survey conducted by
Mathur et al. [24], friends and family members of visually impaired
users conveyed that writing alternative text is time consuming and
requires more thought than inaccessible uploading practices.
Eforts have been undertaken by the social network’s services
themselves to improve the accessibility of visual content, such as
automatic captions [9], or providing text inputs for alternative de-
scriptions [35]. However, these approaches are not yet sufcient
to provide sufcient contextual information, or to ensure that end-
users are aware of these features and therefore they are not widely
used. In addition, little is known about the barriers and motivations
ASSETS ’22, October 23–26, 2022, Athens, Greece
impacting the creation of accessible content by most end-users,
which is at the root of the problem. In this research, we sought to
explore the current context of the accessibility of visual content
in social networks. We provide further analysis on the factors hin-
dering the creation of accessible content by end-users, considering
people with and without a visual impairment, on major social media
platforms. Furthermore, we also aimed to uncover what does or can
motivate people to create accessible content. We set the following
research questions:
• RQ1: What are the motivations for social network users to
create accessible media content?
• RQ2: Which barriers social network users encounter to share
and author accessible media content?
• RQ3: What are the requirements for social network users to
create accessible media content?
To answer these questions, we conducted a two-step user study
including an online survey and user interviews. Our fndings sug-
gest that users are interested in providing accessible content on their
social networks, but most of them are not aware of the steps needed
in order to improve their practices and the impact it could have.
Our work leads to a better understanding of the current state of
media accessibility, in particular, of accessibility awareness among
social network users. The insights collected also highlight gaps and
opportunities to enable a better interaction for blind people and
more engagement in accessible practices by other end-users.
2 RELATED WORK
Our research is related to prior work on (1) existing accessibility
approaches employed by major platforms and its impact on visually
impaired users’ media interpretation, (2) practices on accessible
content sharing by end-users, and (3) current advances and gaps
on image captioning.
2.1 Accessibility on social networks
Social networking services, such as Facebook, have a high adoption
rate among blind users [39]. This also happens with Twitter, which
evolved from a very simple text-based interface to one that is now
flled with multimedia content [3]. The widespread usage of camera
equipped mobile phones contributes to the growth in publications
containing visual content. However, this user-generated content
pushes social networks to become increasingly inaccessible [3].
Numerous eforts have been made to improve accessibility of
visual content on social networks with some of these initiatives
coming from the service providers themselves. For instance, in 2016,
Twitter included a feature allowing users to compose their own
alternative descriptions for their images. However, users claim that
this initial feature had its drawbacks as it had to be enabled by the
users themselves, was hard to fnd, and to understand [34]. Even
though this changed in 2020 [35] – at the moment this resource
is active and available by default – the impact of this measure
has not yet been assessed nor discussed. Meanwhile, Facebook
made a choice to use automatic descriptions by tagging each image
uploaded using image detection and recognition algorithms and
enabling the user to edit the automatically provided alternative
description [9].
Pereira, et al.
Despite these eforts, as described in the most recent studies
on this context, blind and visually impaired users still encounter
signifcant barriers in interpreting visual content [4, 11, 23, 28, 38].
2.2 Content sharing
Approaches to improving the accessibility of media content have
been previously discussed in the literature. For instance, Morash et
al. [26] investigated approaches for web workers to author descrip-
tions of STEM (science, technology, engineering, and mathematics)
images. Still on the context of guiding users on how to provide
better descriptions, Mack et al. [22] proposed two types of proto-
type interfaces to facilitate authoring of alternative descriptions for
images in Power Point presentations, as well as providing feedback
for automatic alternative descriptions. In a similar direction, Gurari
et al. [17] analyze a dataset of images and alternative descriptions
focused on improving current machine generation approaches to-
wards real end users needs. Another approach focused on online
content is presented by Guinness et al. [15] which, using a fully
automated approach, searches for possible descriptions already
available on the web for a given image. This strategy addresses
a current gap, since much content present on social networks is
shared and widely disseminated on other pages - such as memes
and GIFs. However, another type of content particular to social
networks are personal images and photos - which are unlikely to be
available on other pages. Considering this last aspect, one great chal-
lenge still remains, the engagement of users in providing accessible
content. Through an analysis of a million images posted on Twitter,
Gleason et al. [11] observed that only 0.1% contained an alternative
text. Twitter itself may be accountable for this low number by not
enabling by default the feature that allows the inclusion of the
alternative description for many years. However, Gleason et al. [11]
also observed that even users who enabled this feature to provide
alternative text descriptions did not always write them. Visually
impaired users engage in major photo-related activities as other
users, considered by them as part of the social network experience.
However, practices such as taking and editing photos, and provid-
ing an alternative text, often involve undertaking workarounds or
getting help from trusted sighted people [1, 24, 36, 39].
Mathur et al. [24] observed that friends and family members of
visually impaired users engage frequently in accessible practices.
Also, according to Wu et al. [39], users with visual impairments are
much more likely to have friends who are also visually impaired.
These two factors seem to play a role in the increased accessibility of
images that are accessed by visually impaired people (in comparison
to those accessed by sighted people) [24]. However, users also report
that writing alternative text is time consuming and requires more
thought than inaccessible uploading practices, which explains – at
least, in part – the low percentage of images with an alternative
description [11].
Concerning users’ engagement, previous works suggest that
users who currently provide alternative descriptions on their images
are mainly motivated by personal connections to someone with a
disability or by a general matter of inclusion [11, 30]. These works
also reinforce the need of educating users in providing alternative
descriptions as well as investigate further approaches on using
automated description techniques to improve descriptions provided
Authoring accessible media content on social networks
by authors. While these fndings provide important insights into
authoring and sharing practices for accessible content, they leave
some unanswered questions around the perspective of end users
who are not yet aware of current accessibility approaches and the
specifc needs of users with disabilities. In this paper, we focus on
further exploring the challenges these users encounter when trying
to engage in such activities for the frst time and how to motivate
them to create accessible media content.
2.3 Image captioning
The low user compliance in providing alternative text descriptions
is a common web accessibility problem and therefore some alter-
native methods are usually employed to fll this gap. Stangl et
al. [32] classifed some of the existing approaches to generate im-
age descriptions as human-powered approaches, automated image
descriptions approaches and hybrid image description technologies.
Human-powered are recognised by users for their accuracy and
quality of responses. Techniques such as Crowdsourcing may have
slow response times for real-time needs and a high fnancial cost.
Friendsourcing may improve the quality and trustworthiness of the
answers received, as friends would better understand the question
asked, while also removing fnancial costs of the service [3]. How-
ever, the social costs of exposing one’s problems and vulnerability
are a serious concern for these users as they may appear or feel less
independent [3]. As for automated approaches, unlike the previous
technique, they are fast and cheap, allowing platforms to deploy
them at scale [8, 21]. While there are signifcant eforts undertaken
over the past years on image understanding and automated cap-
tioning, the accuracy of these captions is not yet sufcient. Caption
and phrasing models have an important impact in scepticism as
blind and visually impaired people may rely more on automatically
generated captions than on their intuition, making decisions based
on misinformation [23, 31]. Besides that, the adequacy of these
systems used in the open world is still limited, especially when
captioning the wide variety of images posted to social media [31].
In order to fll the gap in image captions, hybrid image descrip-
tion technologies propose a combination of automatic techniques
and human intervention to investigate a trade-of between both
techniques, as explored in [16, 21, 27, 31].
Despite prior eforts in evolving alternative interpretations for
visual content, user-generated content still has a great impact on the
accessibility of media content in social networks. For this research,
we collected feedback from users about their difculties and possible
motivations in accessible authoring practices. From that, we provide
insights into how this interaction fow can be improved to support
their needs, as well as to better engage them in the authoring of
accessible media content.
3 METHOD
In order to address our research questions, i.e., to better understand
the barriers faced by end-users to author accessible media content
and the motivational factors to engage in such practices, the study
was structured into two diferent phases. Ethical approval to run
the study was granted by our university’s Ethics committee.
ASSETS ’22, October 23–26, 2022, Athens, Greece
3.1 Online survey
An online survey was conducted to gather information about usage
of social networks, motivations and barriers for authoring acces-
sible media content, and awareness of accessible practices. The
questionnaire used took about 15 minutes to complete, and it was
built using Microsoft Forms in four diferent languages (English,
French, Portuguese, and Spanish) in order to reach a diverse sample
of participants. This questionnaire was divided in three sections:
demographic questions; social networks usage - considering fre-
quency, devices, and type of content of access, posting, and sharing
activities; and social networks accessibility practices - consider-
ing current practices but also accessibility awareness and further
motivations to engage in such practices.
Pilot study interviews to inform survey design: As a frst step, we
conducted pilot interviews to identify potential problems with, and
improvements to, the questionnaire. Participants were recruited
through the research team’s network. The only inclusion criterion
was to be a user of at least one social network. The pilot study in-
cluded 7 social network users, including three people who are blind.
Participants had a variety of occupations as well as diferent levels
of accessibility awareness. Some of them were accessibility practi-
tioners, but also high school teachers with no previous knowledge
of digital accessibility. Sessions were remotely conducted, taking
between 30 and 40 minutes. During the sessions, a preliminary
version of the online questionnaire was used. Participants were
encouraged to propose suggestions and improvements to enhance
the overall understanding of the survey. At this stage, participants
responses were not collected rather than their thoughts on the in-
formation contained on the questionnaire; i.e., their answers were
not considered in the fnal analysis. Participants provided impor-
tant feedback on the questionnaire, such as questions that were
difcult to understand or suggestions for new response options to
closed-ended questions. Based on their contributions, a new version
of the questionnaire was built, including two new questions and
some adjustments in the wording of other questions.
Questionnaires: In order to focus on social network users, we
disseminated the questionnaire through diferent social media chan-
nels. The call for participation was shared by the research team,
their university and research unit, and fellow organisations (such
as disability-related ones). The questionnaire was online during 3
months and we gathered a total of 258 answers from participants
aged from 17 to 73 years old (Mean=37.35, Median=31, IQR=23)
with 64 (25%) of them self-reporting having some kind of disability,
such as visual, hearing, motor and/or cognitive impairments. In
particular, 34 participants were blind, 12 had low vision, and 1 was
colourblind.
In addition to the specifc questions concerning social media
usage and accessible practices, participants were also invited to
share additional thoughts. Through this questionnaire, participants
also stated their availability to be contacted for the next phase of
this research.
3.2 User interviews
In a frst moment, participants stating their availability in the previ-
ous phase were contacted, and we followed up with semi-structured
ASSETS ’22, October 23–26, 2022, Athens, Greece
interviews with 20 of them – all reported being frequent social
network users in the questionnaire. From these participants, 7 self-
reported being blind and 13 of them were sighted users without a
disability. Furthermore, half of them stated not frequently under-
taking accessible practices, as presented in Table 1.
Prior to scheduling the interview, we asked participants to post
media content on their usual social networks in an accessible way.
Participants were instructed to post, at least, three diferent media
contents in order to familiarize themselves with the authoring
processes (in case they were not). In addition, they were also asked
to take notes reporting in detail the activities conducted and their
opinions and difculties encountered in this process. Participants
were invited to conduct these activities over a two-week period.
This time frame was intended to include accessible practices in
their sharing and posting routine for two weeks, rather than asking
for a specifc task to create accessible content.
After two weeks of study (i.e., after the initial contact), the semi-
structured interviews were conducted. We asked questions about
their experience in accessible practices in social networks, further
motivations for accessible content authoring, and potential sugges-
tions or additional thoughts on how to improve this process and
to more fully commit end-users to accessibility practices. These
interviews were also an opportunity to further discuss the answers
provided by participants in the questionnaire. All interviews were
conducted remotely over the phone, Skype, or Zoom, lasted 20 to
30 minutes and were recorded with participants’ prior consent.
3.3 Data analysis
The frst step of the data analysis consisted of a quantitative analysis
of the answers to the closed-questions in the online questionnaire.
A preliminary analysis was made to ensure that all answers pro-
vided were valid. For that, we searched for answers out of scope
and found that all the answers provided by the participants were
consistent with the corresponding question. In addition, all ques-
tions included in the questionnaire were optional, resulting in all
entries being considered valid. In summary, all responses provided
by all participants were considered in the data analysis. It is also
important to highlight that several questions were multiple choice,
allowing participants to choose one or multiple options from a list
of possible answers.
Next, all conducted interviews were transcribed and, along with
answers provided for the open-ended questions, analysed through
an inductive coding approach [25]. First, two researchers reviewed
independently a subset of this data, conceptualizing a set of codes
based on this subset. The coders compared their initial set of codes
and their categorisations in order to develop a unifed list of codes.
Then, the two coders reviewed a new subset of transcriptions taking
into account the consolidated and revised codes in order to reach
an agreement, identifying a total of 150 distinct codes, organized in
two levels, 21 codes and 129 sub-codes. Following that, the coding
of all data collected was performed. The codes and their sub-codes
are available online1 .
1 The codebook is available at: https://osf.io/anmd7/?view_only=
547d72f78553489498af9c1df7af1a58
Pereira, et al.
4 FINDINGS
In this section, we frst present key fndings identifed through the
quantitative analysis of the data gathered through the question-
naire. Next, we present the fndings obtained from the qualitative
analysis of the information gathered during the user interviews.
This information was divided by the following topics: (1) accessibil-
ity unawareness, (2) lack of know-how, (3) the cost of the additional
efort, (4) complying with and without guidelines or features, (5)
inaccessibility, (6) and accessibility motivations and concerns.
4.1 Online survey
This section provides detailed information obtained through the
online survey, containing questions regarding authoring and shar-
ing practices on social networks. The following data concerns the
258 total answers gathered.
4.1.1 Device access. We asked participants how often they interact
with social networks, considering access and posting activities,
according to device types.
In both groups – sighted and visually impaired participants –
mobile devices were more popular for accessing and posting content
on social networks. However, in both scenarios, visually impaired
participants tend to carry these activities on desktop or laptop
devices more than sighted participants, even though, the diference
is more signifcant in posting activities, as presented in Figure 1.
Figure 1: Devices used by participants to access and post con-
tent on social networks
4.1.2 Social networks used. Concerning the social networking plat-
forms accessed by our questionnaire participants, Facebook was
the frst choice for the majority, followed by Twitter. Only 2% of
visually impaired participants declared accessing Instagram, while
25% of sighted participants reported having Instagram as their main
social network.
Sighted participants reported to post most of their content on
Instagram followed by Facebook and Twitter. Regarding visually
impaired participants, they declared to prefer to post their content
on Facebook, followed by Twitter and a very low percentage of
them declared using Instagram as their main social network to post
content. This information is presented in Figure 2. It is important to
highlight that other social networks were also mentioned by both
Authoring accessible media content on social networks ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Demographics of interviewees including age, most accessed social networks, social networks users most post their
own content, and social networks users most share their content (i.e., post content created by others, for example, sharing on
Facebook and retweeting on Twitter).

Most accessed Social networks users Social networks users

ID Age Accessible practices
social network most post their content most share their content
Blind participants
BP1 20 Yes Facebook Facebook Facebook
BP2 63 Yes Facebook Facebook Facebook
BP3 53 Yes Twitter Facebook Facebook
BP4 52 Yes Twitter Twitter Twitter
BP5 50 Yes Twitter Twitter Twitter
BP6 21 No WhatsApp WhatsApp WhatsApp
BP7 17 Yes Twitter Messenger Twitter
Sighted participants
SP1 73 Yes Twitter Twitter Twitter
SP2 32 No WhatsApp WhatsApp Instagram
SP3 30 No Facebook Facebook Facebook
SP4 57 Yes Twitter Facebook Twitter
SP5 30 No WhatsApp Instagram Instagram
SP6 25 No Instagram Instagram Instagram
SP7 30 No Twitter Messenger Messenger
SP8 33 No Facebook Facebook Facebook
SP9 41 No Facebook Facebook -
SP10 27 Yes Instagram Instagram Instagram
SP11 29 No WhatsApp WhatsApp Instagram
SP12 34 Yes Instagram Instagram Instagram
SP13 30 No WhatsApp WhatsApp WhatsApp

groups, such as LinkedIn, WhatsApp, and Tiktok, however, with
less expressive numbers.
they still encounter several challenges using these apps [1], which
may contribute to their low participation levels in image-centric
social networks, such as Instagram. In particular, among other types
of visual content, GIFs and Memes were highly unpopular among
visually impaired participants, with all of them reporting never
sharing this type of content. As explored in previous works, these
kind of media represents a challenge for accessibility. Besides not
being properly supported by major platforms [30], they carry some
cultural context or hidden meaning related to an emotional tone or
humorous aspect [13, 14], being heavily dependent on alternative
descriptions provided by the authors themselves.

4.1.4 Accessible practices. The vast majority of visually impaired

Figure 2: Social networks used by participants
4.1.3 Type of content. We also asked participants about the con-
tent they most post on social networks. Although the numbers
obtained for text, audio, and video content do not show a signif-
cant diference, the same was not observed for visual-only content.
Sighted participants reported posting 5% of image content, while
this number is 2% for visually impaired participants. Even though
users with a visual impairment edit and post photos on social media,
participants (70%) reported providing an alternative description for
their last shared media content, while most of sighted participants
(71%) declared not having provided it.
We also asked participants that did provide an alternative de-
scription for their last three posted media content how they did it.
Most of them (25%) used the functionality provided by the social
network, while 23% chose to integrate the description in the text of
the post.
Among those not providing alternative descriptions, when asked
about the reasons for not engaging in such activity, most of them
declared not knowing that it was possible, followed by those who
declared not knowing where to write an alternative description
(Figure 3).
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 3: Reasons to not provide alternative description
We also observed through questionnaire responses a correlation
between adopting accessible practices and the familiarity with the
needs of people with disabilities. The majority of sighted partici-
pants (56%) had no idea about accessibility practices adopted by
people around them, while more than half of visually impaired
participant (51%) stated having friends or family members posting
or sharing accessible content.
Figure 4 summarises participants’ opinions on why social net-
work users in general do not provide alternative text descriptions
for media content they post or share. Most participants consider
that other people don’t know that this is possible, followed by those
considering that other people don’t think it has any impact.
Figure 4: Reasons for others not providing alternative de-
scription
4.2 Interviews with Social Network Users
In this section, we present the information collected during the
interviews conducted with 20 social network users. Participants
were asked about their experience with accessible practices in social
networks, further motivations for accessible content authoring, and
potential suggestions or additional thoughts on how to improve
this process.
4.2.1 Accessibility unawareness. Considering the data obtained, we
observed an overall lack of awareness among sighted participants,
but also, some stigma associated with people with disabilities
Pereira, et al.
using technology, in particular, about blind people accessing vi-
sual content. Further investigation conducted through interviews
allowed us to observe that most of them were not aware of the use
of technology by blind people and how this content is being con-
sumed by them, therefore becoming more difcult to understand
what accessible content really means:
“I am describing this image, theoretically for a blind
person, but then how will the platform use it? Does
the person have the possibility to listen? Is there going
to be a...? I don’t know how this is used in the end, it
seems silly, but how does it reach the person? How is
it accessible in fact, at the end of the experience?” –
SP13
A frst layer of this unawareness is refected by the lack of
knowledge about what they should do to improve the accessi-
bility of their content. Most sighted participants, although frequent
users of social networks, only discovered this possibility during
this study.
Some sighted participants also felt that it was not necessary to
provide an alternative access for their content as they do not know
anyone with a disability, underestimating the reach of authoring
and sharing activities. However, while some sighted participants
were used to privately sharing some family pictures and may follow
this line of argument, this misjudgement about the impact and
needs of accessibility may go further, reinforcing the stigma, by
implying that visual content is not for blind users:
“I also share a bit of this stigma, which is that, if
Instagram is highly visual, a blind person will use it?
[...] I don’t know if there is this motivation on the part
of the blind people to use a tool that is so visual.” –
SP5
Under the assumption that blind people are not interested or mo-
tivated in accessing visual content, the accessibility of this content
is not taken into account. This is particularly evident in image-
centric social networks such as Instagram. Consequently, blind
participants reported being excluded from accessing this content
and thus from using these platforms.
From the perspective of two blind interviewees, some people
just do not care enough about the subject. Partly because many of
them don’t know someone with a disability, so they do not refect
about it, but also because some may be focused on just reaching
out to as many people as possible, neglecting minority user groups:
“Other people don’t care, no matter how much we tell
them.” – BP5
4.2.2 Lack of know-how. During the interviews, participants who
embed alternative descriptions in the content of their post evoked
having discovered this practice through other users’ publications,
and, therefore, used it as a model example to complete this task.
This behaviour was also observed by Gleason et al. [11] in a previ-
ous study. Although not ideal, it may increase the visibility of issues
related to content accessibility. However, this choice was mainly
driven by sighted interviewees not perceiving nor discovering the
specifc accessibility feature provided by the platform. Interviewees
further described the difculty to discover where to write an al-
ternative text description as being related to the non user-friendly
Authoring accessible media content on social networks
and hard-to-fnd accessibility approaches currently employed by
social network service providers:
“If they told me ‘Oh yes, you can make these things
accessible, you just have to take a lot of very compli-
cated steps to get it done’. Obviously, this will make it
very difcult for people who are interested in doing
it. Accessibility features are difcult to fnd and not
very discoverable, they’re hidden.”- SP8
Furthermore, sighted interviewees also discussed their difculty
in not knowing how to write a suitable alternative description.
While they convey it was a challenge to represent an image through
text, they also stated fnding no guidance or information on what
is considered a good alternative description:
“My biggest difculty was this: not knowing if there
was any standard or not, what would be necessary to
include or not, if I should put more information, less
information, if I should be more specifc, give more
details. . . ” – SP13
It was a consensus among all our interviewees that, as a frst
step, it is essential to create approaches to raise awareness about
accessible practices and their benefts among social network users.
Most of them suggest that one way to achieve equity in social media
is by emphasising accessibility features as, currently, they do not
draw enough attention of these users to this matter:
“They have to be more visible in the authoring process
[. . . ], even if they weren’t hidden, they could also be
promoted, which they aren’t either.” – SP7
Besides, accessibility must be a part of the authoring process.
Interviewees consider that an edit feld for the alternative descrip-
tion should be provided within the authoring fow with a content
warning not only to make it mandatory but, especially, to get
people used to providing this information.
Some participants have also reported that having a support fea-
ture to suggest alternative descriptions would help them, frst by
providing an example of what is considered an appropriate alterna-
tive description, and second by providing the opportunity for them
to improve machine generated descriptions:
“This could become standard [...] show the descrip-
tion and say: "Do you think this is a good description
for this image?" because in this case, it [Facebook]
simply decides the description itself and also, if it is
not correct, I have no way of indicating it.” – SP8
4.2.3 The cost of the additional efort. The third most popular rea-
son mentioned by questionnaire participants for not providing an
alternative description for their media content was the time it takes
to do it. While further exploring this subject during the interviews,
some sighted interviewees also declared that accessible practices
are currently not integrated into their current publishing inter-
action fow. Providing an alternative description for their images
adds a new task that would involve considerable time for refection
and make their activities much more time-consuming, especially
given the spontaneous nature of social media content sharing for
personal usage:
“Depending on the elements that are in the picture
[it] can be a bit exhausting, especially if you have
ASSETS ’22, October 23–26, 2022, Athens, Greece
many people and I have to be describing each person
. . . ” – SP5
Blind participants shared the additional burden of often having
to be the activist in their social circles, bringing awareness and
recalling others to ensure they start producing accessible content.
This leads some to the feeling of being obnoxious for having to
constantly remind their acquaintances they are not able to fully
understand the content shared:
“When we have a disability and we’re faced with it,
we’re tired of saying ‘hey, don’t forget, publish some-
thing accessible’.” – BP5
As reported by our blind interviewees and more broadly dis-
cussed in previous works [23, 27, 32, 36, 38, 40], alternative de-
scriptions currently provided by major platforms are not providing
enough contextual information for visually impaired users to prop-
erly interpret media content. Therefore, they reinforced the impor-
tance of promoting user engagement to convey this additional
context and the users’ own intentions and purposes in publishing
this content through human description:
“When it comes to being accessible, you can’t take it
out of somebody’s hands [. . . ] Let them decide what
they want to say, [...] how they want to describe it,
whatever they were thinking of, why they were post-
ing, that image should be what they’re posting.” –
BP4
4.2.4 Lack of standardisation. Major platforms are employing dif-
ferent approaches to support accessible media authoring. This lack
of standardisation requires users to identify and learn how to use
each of the accessible features provided by every platform used. In
Facebook, only two participants publishing accessible content for
the frst time were able to discover this feature, and, as stated by
one of them, after a thorough research.
“On Facebook, I really, really had to go search, I tried
a lot, I tried and eventually I had to search what Face-
book had for accessibility because I could not fnd it.”
– SP7
They also reported not fnding this feature available in the
native app, only being able to edit the automatically generated
description on their desktop through the web interface. One par-
ticipant also shared that, even though being blind and a frequent
Facebook user, he was not aware of the possibility of editing these
descriptions.
As for Twitter, the feature enabling the user to provide an alter-
native description for their own images was considered the best
approach by one blind user. However, as described by other par-
ticipants, its efective use is jeopardised by not being part of the
standard Twitter authoring process:
“Twitter’s way of making a feld for people to put in
the alt text is the best way to do it, but I can’t tell you
how many people can fnd it.” – BP4
Only one sighted participant tried to explore the accessibility
feature provided by Instagram during this study, and also mentioned
the difculty in fnding it:
ASSETS ’22, October 23–26, 2022, Athens, Greece
“In Instagram, the frst thing I think I had heard about
is the alternative text, but I tried to fgure it out and I
found it out, but with some difculty, because it was
in some advanced settings in the post, it’s not very
visible.” – SP7
During the interviews, Facebook, Twitter, and Instagram were
the most cited social networks among participants. Each of these
platforms adopted a diferent approach for providing alternative
descriptions for user-generated content. Twitter provides an input
feld for users, while Facebook and Instagram rely on machine-
generated descriptions. As observed by Sacramento et al. [30], ma-
jor social networks are not consistent also when it comes to
providing accessibility features and compliance among diferent
platforms, such as mobile/desktop interfaces or iOS/Android mobile
applications. However, the non-user-friendly interface is a common
ground between them.
Therefore, concerning the roles of diferent stakeholders in-
volved in the accessibility context, most interviewees consider that
these platforms have the largest infuence and, currently, they are
not fully engaged in promoting accessibility:
“The platform is responsible for ensuring a good user
experience and they have a good percentage of users
who have these specifc needs, and they are the ones
who have to ensure this experience also applies to
them.” – SP7
Some interviewees also considered this responsibility should
go further and shared with the users themselves. As they play an
essential role in this process, they have to be more committed to
accessible practices.
Governments’ actions and the legal context of digital media reg-
ulation was also mentioned during the interviews as a possible way
forward. Two participants also believe it involves providing better
public policies for social inclusion in general. For instance, one par-
ticipant made a comparison with current accessibility legislation
for physical structures, such as ramps and handrails.
4.2.5 Inaccessibility. The inaccessibility of accessibility features is
a paradox currently encountered in these services. Just like sighted
participants, blind interviewees also reported having difculties
identifying the proper feature provided by major platforms even
though they were familiar with its availability. This difculty is fur-
ther reinforced by the constant updates of these systems, requiring
the acquisition of new knowledge about the structure of these new
interfaces, as also pointed out by Voykinska et al. [36].
Furthermore, while most blind questionnaire participants de-
clared creating accessible content on their social networks, in the
interviews it was possible to observe that this practice is not neces-
sarily being enabled by accessible features provided by platforms.
During the interviews, they mentioned asking others for help in
order to confrm the elements contained in the media to be pub-
lished, making them dependent of sighted friends or family
members. Moreover, one of them stated only sharing content al-
ready accessible. Therefore, these users declared missing a feature
to assist them in creating their own descriptions for their images:
Pereira, et al.
“In this case [for this study], I had to ask my brother
how the photos were like for me to describe them.” –
BP6
This lack of support for blind content authors reinforces the
social cost, also observed by Brady et al. [3], and their sense of
exclusion as they are unable to fully experience this aspect of social
networks:
“I don’t think it’s because we’re blind or have any
other type of disability that we lose the right to ex-
press ourselves, or to send any kind of joke, let’s say,
those little stickers... when talking, we who are blind,
are excluded from doing it.” – BP1
As mentioned by a blind participant, the constant arising of new
media content such as GIFs, Stickers, Memes, and images with
embedded text or screenshots enabled by major platforms pushes
these social networks to be less and less accessible for them. This
diversity also comprises stories, a new format that has become
quite popular on major platforms. This format supports a mix of
images, videos, text, and even stickers. This type of content raises
another challenge for users as, once more, they don’t know how to
provide accessibility for this type of content. Another issue raised
by participants was concerning links that, when posted on social
networks, often generate a preview of the website and often do
not contain any information about it. This diverse range of content
often fails to match the accessibility features available, which allows
us to better understand why none of the questionnaire participants
reported posting this type of content.
4.2.6 Accessibility motivations & concerns. Participants currently
publishing accessible content are mostly driven by having a dis-
ability themselves or having acquaintances with disabilities. Two
sighted participants, not used to share accessible content, declared
that they would be more engaged in these practices if they had a
personal connection to someone with a disability.
Sighted participants declaring publishing accessible content
agreed the main reason for them to engage in these practices is
because it is the right thing to do. Sighted interviewees not used
to share accessible content shared their interest in contributing to
the inclusion of people with disabilities and to make information
reach as many people as possible.
Most interviewees also convey that providing platform sup-
port is an important factor to create more user engagement. One
blind participant, who frst discovered accessibility features during
the study, stated that now that he is aware of it, he has already
included alternative descriptions for all his previously posted pic-
tures and intends to adopt this practice in the future. Nevertheless,
the difculty encountered may discourage some users, causing the
opposite efect on people who, at frst, are more likely to give it a
try.
It was also possible to observe that some sighted interviewees
may consider accessibility approaches as creating setbacks to
their experience, reinforcing the unawareness about accessible prac-
tices – in particular the behaviour of screen readers and alternative
descriptions.
The previously mentioned strategy of embedding an alternative
description text in the post’s content, while it may be perceived
Authoring accessible media content on social networks
by some as an example of a good practice, was mentioned by one
participant as a downside of accessibility, as it makes a post very
long and somehow redundant for those not using a screen reader.
For that, she considers that this information should be embedded
in a way that is only perceived by screen reader users. Loading
and scroll speed was also considered as a setback by other sighted
interviewee, as accessible practices concerns including additional
information to be loaded by apps and websites:
“It occurred to me, people could describe just like
I did, or record a short audio to be heard by these
people, but this will take a lot out of the dynamism
that people have gotten used to in Instagram. [. . . ] it
would be great for them, but I think it represents a
setback for non-visually impaired people.” – SP5
5 DISCUSSION
In what follows, we frst discuss how these fndings can be used to
answer our research questions followed by further contributions
provided by this work.
5.1 Research questions
RQ1: What are the motivations for social network users to create
accessible media content? Our participants were mainly motivated
by doing the right thing and promoting inclusion for people
with disabilities. Although some of them are used to share media
content only for a private audience, such as family and close friends,
many of them are interested in enabling access to information for
other people as well.
Considering the willingness to include people with diferent abil-
ities and cultural contexts shown by interviewees, some strategies
may be applied in order to motivate more users to create accessible
media content. Participants tended to be more aware when in con-
tact with a person with a disability, but as well when confronted
with current accessibility approaches. They showed curiosity and in-
terest in knowing more about the subject and they are motivated to
understand how their content is being consumed by blind users. One
major challenge identifed among interviewees was the unaware-
ness of how and why blind people use social networks. Making
end-users part of this process, integrating accessibility features
more prominently on platforms authoring fow, educating them
about accessible practices and alternative access, and provid-
ing tutorials or scenarios of people with disabilities using the Web,
for instance, may increase awareness and thus encourage them to
become more frequently engaged in such practices. From that, we
reinforce the conclusions reached by previous studies [11, 30] on
the need of additional tooling and training for social network users
on all major platforms.
RQ2: Which barriers social network users encounter to share and au-
thor accessible media content? First, it is important to highlight that
the barriers encountered and reported by participants are strongly
related to their familiarity with people with disabilities, assistive
technologies, and accessibility features in general. For this reason,
it is extremely important to educate people about accessibility, in-
cluding how diferent disabilities afect the way people interact
with technology, their challenges, and how users can publish ac-
cessible, inclusive digital content. While the society at large should
ASSETS ’22, October 23–26, 2022, Athens, Greece
be responsible for seeking and providing such education, social
networking sites may leverage their platform to educate users and
enable more inclusive sharing practices. Instead, accessible prac-
tices and features remain unknown to most users. Most sighted
participants were not aware of the steps they can take to make
their content more accessible and they found no guidance on
major platforms to assist them in this process. In addition, even
when they actively search for guidance, they report difculty in
learning about or in fnding accessibility features. Given this con-
text, accessibility practices are being perceived by many sighted
users as an activity that requires a signifcant additional efort
on their part.
We also identifed that some sighted users still have a certain
stigma associated with accessibility, arguing that accessibility
should be employed only when necessary or, even worse, that
accessibility compliance may compromise their current experience
on social media. Platforms not making this a requirement, or not
providing a proper prompt warning, may also be contributing to
this frst line of thought. On the other hand, there is still a lack
of support for blind users to create accessible media content as
they fnd no features to assist them in this activity. Even though
they recognise the latest advances promoted by major platforms,
accessibility issues have been present in these services for a long
time [5, 6] and blind users are still highly dependent on other
people to participate in social networks, reinforcing the social cost
constantly experienced by them.
RQ3: What are the requirements for social network users to create ac-
cessible media content? As a frst step, major platforms must provide
a more user-friendly and accessible interface in order to make
accessibility features more noticeable and easier to use. An-
other critical issue identifed concerns the diferent approaches
adopted by these platforms to provide accessibility features, mak-
ing it difcult for users to identify and learn how to use each one of
these resources. For that, we suggest that accessible approaches
should be standardised among platforms in order to create
an easily recognised pattern towards an accessible posting and
sharing inherent routine. Moreover, machine-generated descrip-
tions introduced by some platforms often do not provide blind
users with enough information or context, so they properly un-
derstand the corresponding media content, as also observed in
previous works [23, 27, 32, 36, 38, 40]. One promising avenue is
investing in hybrid solutions: exploring the balance of the bene-
fts of technological advances in automatic image recognition and
machine-generated descriptions, and involving users to fll the gap
concerning context details and, in particular, their own intention
and purpose in posting a certain media content. This approach also
harnesses automatic descriptions to guide and support users in
creating their own description, reducing the work and time cost
perceived by some of them. Furthermore, blind users authoring
accessible content will also beneft from it, reducing the social cost
previously discussed.
5.2 Perspectives on accessible social media
content authoring
While this study was not intended to be a comprehensive study on
the accessibility features or barriers of social networks as a whole,
ASSETS ’22, October 23–26, 2022, Athens, Greece Pereira, et al.

the diferent user perspectives on accessible practices in social questions also apply to diferent kinds of impairments, in particular
networks gathered in our study allowed us to identify potential concerning accessibility awareness and motivational strategies.
avenues for future research in accessible media content authoring.
Although machine-generated descriptions are not currently pro- 6 CONCLUSIONS
viding sufcient contextual information of visual content, their
In this study, we presented an overview of the usage and the ac-
main advantage is that they can potentially be deployed at a large
cessible practices employed by 258 inquired social network users.
scale. For that, research on text alternatives best practices and users
These fndings suggest that people with disabilities are interacting
preferences [29, 31, 32] may be used to further improve these de-
in social networks as much as users without disabilities. However,
scriptions so that their quality becomes acceptable. From a hybrid
our blind participants showed being more frequently engaged with
perspective, providing suggestions of appropriate alternative de-
non-visual content, such as text and audio, than with visual-only
scriptions may be useful to educate users and, therefore, to create
content, such as images, in particular GIFs and memes. This may be
more engagement on accessible practices.
partly explained by another fnding: most sighted participants are
Another opportunity suggested by some of our sighted intervie-
not sharing accessible media content because, as reported by most
wees is providing users with diferent ways of including alternative
of them, they were not aware of this possibility until participating
descriptions, such as audio descriptions. Previous studies [7, 20]
in this study.
identifed that human narration – besides being faster – allows
Following that, we conducted interviews with 20 of these partic-
better image comprehension by blind users and it helps to establish
ipants in order to better explore their experiences, challenges and
a connection between user and content author. Marques et al. [7]
motivations concerning accessibility and visual content in social
also suggest a scenario where blind users could send a request to
networks. While sighted interviewees were interested in being more
the author of an image so that he would record an audio descrip-
engaged in accessible practices, most of them did not fnd proper
tion. While this solution possibly reinforces the existing burden
assistance or support on major platforms to guide them to enhance
imposed on blind people, collaborative approaches on alternative
the accessibility of their content. At the same time, blind users are
descriptions are also an interesting aspect to be further explored.
not being provided with proper alternative descriptions whether
As suggested by Sacramento et al. [30], providing users the possi-
by authors or by the machine-generated descriptions provided by
bility of including alternative descriptions for visual content they
some major platforms. Moreover, the burden of educating others
encounter may be employed by sighted users who are already moti-
and promoting the authoring and sharing of accessible content falls,
vated and currently engaged in accessible practices. This approach
unfairly, upon their ability to compel others to act.
could be also a complementary action to create more awareness
This research does not aim to provide a thorough analysis of
among social media users.
current accessibility features rather than provide insights into the
Moreover, this research is not intended to scrutinize any par-
current status of accessibility awareness among social networks
ticular social network. The fndings and discussions presented
users, and potential future directions. Our work complements pre-
relied solely on the data collected, in which Facebook, Twitter,
vious research on visual content in social networks by providing
and Instagram were the social networks most used by study
insights on how sighted users are experiencing accessible practices,
participants, corroborating earlier studies on similar areas of re-
but also brings up the importance of employing hybrid solutions
search [1, 3, 13, 32, 36, 39]. Further research with participants using
to fll the current gap. Platforms must go beyond just deploying
other social networks to create accessible content, in particular
accessibility features and must be more invested in approaches that
those involving diferent forms of interaction, such as instant mes-
create user awareness, engaging more people in adopting accessible
sengers, would yield important insights into this context. Another
practices in their daily posting routine.
interesting perspective to be further investigated and discussed is
the current status of compliance of these platforms with the current
guidelines, in particular with the WCAG (Web Content Accessibility ACKNOWLEDGMENTS
Guidelines) and ATAG (Authoring Tools Accessibility Guidelines) - This work was supported by the European Commission through
which are often overlooked but might also be very ftting to this project SONAAR (grant agreement LC-01409741), and by FCT
context. Considering, for instance, the case of Facebook and Insta- through the LASIGE Research Unit, ref. UIDB/00408/2020 and ref.
gram that, even though they provide an alternative description for UIDP/00408/2020.
each image, as discussed earlier, it does not necessarily entail ac-
cessible content. This context is especially challenging since these REFERENCES
companies are not currently required by law to comply with any [1] Cynthia L. Bennett, Jane E, Martez E. Mott, Edward Cutrell, and Meredith Ringel
of these guidelines, leaving each of these platforms to decide the Morris. 2018. How Teens with Visual Impairments Take, Edit, and Share Photos
level of commitment they are willing to assume. on Social Media. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems - CHI ’18, Vol. 2018-April. ACM Press, New York, New York,
Finally, even though this work focused on visual disabilities, ac- USA, 1–12. https://doi.org/10.1145/3173574.3173650
cessibility in media content also has an impact on the interaction [2] Larwan Berke, Matthew Seita, and Matt Huenerfauth. 2020. Deaf and hard-of-
hearing users’ prioritization of genres of online video content requiring accurate
of people with other disabilities, for instance, video captions for captions. In Proceedings of the 17th International Web for All Conference, W4A
users with hearing impairments [2] or alternative descriptions for 2020. https://doi.org/10.1145/3371300.3383337
other screen reader users, such as people with cognitive impair- [3] Erin Brady, Yu Zhong, Meredith Ringel Morris, and Jefrey P. Bigham. 2013. Inves-
tigating the appropriateness of social network question asking as a resource for
ments [18, 19]. Therefore, most topics discussed in our research blind users. Proceedings of the ACM Conference on Computer Supported Cooperative
Work, CSCW (2013), 1225–1236. https://doi.org/10.1145/2441776.2441915
Authoring accessible media content on social networks
[4] Julian Brinkley and Nasseh Tabrizi. 2017. A Desktop Usability Evaluation of
the Facebook Mobile Interface using the JAWS Screen Reader with Blind Users.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting 61, 1
(sep 2017), 828–832. https://doi.org/10.1177/1541931213601699
[5] Maria Claudia Buzzi, Marina Buzzi, and Barbara Leporini. 2011. Web 2.0: Twitter
and the blind. In Proceedings of the 9th ACM SIGCHI Italian Chapter International
Conference on Computer-Human Interaction: Facing Complexity. 151–156.
[6] Maria Claudia Buzzi, Marina Buzzi, Barbara Leporini, and Fahim Akhter. 2010. Is
Facebook really" open" to all?. In 2010 IEEE International Symposium on Technology
and Society. IEEE, 327–336.
[7] João Marcelo dos Santos Marques, Luiz Fernando Gopi Valente, Simone Bacellar
Leal Ferreira, Claudia Cappelli, and Luciana Salgado. 2017. Audio Description
on Instagram: Evaluating and Comparing Two Ways of Describing Images for
Visually Impaired. In Proceedings of the 19th International Conference on Enterprise
Information Systems. SCITEPRESS - Science and Technology Publications, 29–40.
https://doi.org/10.5220/0006282500290040
[8] Carlos Duarte, Carlos M. Duarte, and Luís Carriço. 2019. Combining Semantic
Tools for Automatic Evaluation of Alternative Texts. In Proceedings of the 16th
Web For All 2019 Personalization - Personalizing the Web. ACM, New York, NY,
USA, 1–4. https://doi.org/10.1145/3315002.3317558
[9] Facebook. 2020. How do I edit the alternative text for a photo on Facebook?
https://www.facebook.com/help/214124458607871
[10] Facebook. 2021. Facebook Reports Third Quarter 2021 Results.
https://investor.fb.com/investor-news/press-release-details/2021/Facebook-
Reports-Third-Quarter-2021-Results/default.aspx
[11] Cole Gleason, Patrick Carrington, Cameron Cassidy, Meredith Ringel Morris,
Kris M Kitani, and Jefrey P Bigham. 2019. “It’s almost like they’re trying to
hide it”: How User-Provided Image Descriptions Have Failed to Make Twitter
Accessible. In The World Wide Web Conference on - WWW ’19. ACM Press, New
York, New York, USA, 549–559. https://doi.org/10.1145/3308558.3313605
[12] Cole Gleason, Patrick Carrington, Lydia B. Chilton, Benjamin Gorman, Hernisa
Kacorri, Andrés Monroy-Hernández, Meredith Ringel Morris, Garreth Tigwell,
and Shaomei Wu. 2020. Future research directions for accessible social media.
ACM SIGACCESS Accessibility and Computing 127 (jul 2020), 1–12. https://doi.
org/10.1145/3412836.3412839
[13] Cole Gleason, Amy Pavel, Himalini Gururaj, Kris Kitani, and Jefrey Bigham. 2020.
Making GIFs Accessible. In The 22nd International ACM SIGACCESS Conference
on Computers and Accessibility. ACM, New York, NY, USA, 1–10. https://doi.org/
10.1145/3373625.3417027
[14] Cole Gleason, Amy Pavel, Xingyu Liu, Patrick Carrington, Lydia B. Chilton,
and Jefrey P. Bigham. 2019. Making memes accessible. ASSETS 2019 - 21st
International ACM SIGACCESS Conference on Computers and Accessibility (2019),
367–376. https://doi.org/10.1145/3308561.3353792
[15] Darren Guinness, Edward Cutrell, and Meredith Ringel Morris. 2018. Caption
Crawler: Enabling Reusable Alternative Text Descriptions Using Reverse Image
Search. In Proceedings of the 2018 CHI Conference on Human Factors in Computing
Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery,
New York, NY, USA, 1–11. https://doi.org/10.1145/3173574.3174092
[16] Darren Guinness, Edward Cutrell, and Meredith Ringel Morris. 2018. Caption
Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image
Search. In Proceedings of the 2018 CHI Conference on Human Factors in Computing
Systems - CHI ’18, Vol. 2018-April. ACM Press, New York, New York, USA, 1–11.
https://doi.org/10.1145/3173574.3174092
[17] Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. 2020. Cap-
tioning Images Taken by People Who Are Blind. In Computer Vision – ECCV
2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.).
Springer International Publishing, Cham, 417–434. https://doi.org/10.1007/978-
3-030-58520-4_25
[18] Bronwyn Hemsley, Stephen Dann, Stuart Palmer, Meredith Allan, and Susan
Balandin. 2015. "We defnitely need an audience": Experiences of Twitter, Twitter
networks and tweet content in adults with severe communication disabilities
who use augmentative and alternative communication (AAC). Disability and
Rehabilitation 37, 17 (2015), 1531–1542. https://doi.org/10.3109/09638288.2015.
1045990
[19] Amanda Hynan, Juliet Goldbart, and Janice Murray. 2015. A grounded theory
of Internet and social media use by young people who use augmentative and
alternative communication (AAC). Disability and Rehabilitation 37, 17 (2015),
1559–1575. https://doi.org/10.3109/09638288.2015.1056387
[20] Lawrence H Kim, Abena Boadi-Agyemang, Alexa Fay Siu, and John Tang. 2020.
When to Add Human Narration to Photo-Sharing Social Media. In The 22nd
International ACM SIGACCESS Conference on Computers and Accessibility. ACM,
New York, NY, USA, 1–3. https://doi.org/10.1145/3373625.3418013
[21] Christina Low, Emma McCamey, Cole Gleason, Patrick Carrington, Jefrey P.
Bigham, and Amy Pavel. 2019. Twitter A11y: A Browser Extension to Describe
Images. In The 21st International ACM SIGACCESS Conference on Computers
and Accessibility - ASSETS ’19. ACM Press, New York, New York, USA, 551–553.
https://doi.org/10.1145/3308561.3354629
ASSETS ’22, October 23–26, 2022, Athens, Greece
[22] Kelly Mack, Edward Cutrell, Bongshin Lee, and Meredith Ringel Morris. 2021.
Designing Tools for High-Quality Alt Text Authoring. The 23rd International
ACM SIGACCESS Conference on Computers and Accessibility, 1–14. https://doi.
org/10.1145/3441852.3471207
[23] Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell.
2017. Understanding blind people’s experiences with computer-generated cap-
tions of social media images. Conference on Human Factors in Computing Sys-
tems - Proceedings 2017-May (2017), 5988–5999. https://doi.org/10.1145/3025453.
3025814
[24] Reeti Mathur and Erin Brady. 2018. Mixed-Ability Collaboration for Accessible
Photo Sharing. In Proceedings of the 20th International ACM SIGACCESS Confer-
ence on Computers and Accessibility - ASSETS ’18. ACM Press, New York, New
York, USA, 370–372. https://doi.org/10.1145/3234695.3240994
[25] Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and
inter-rater reliability in qualitative research: Norms and guidelines for CSCW
and HCI practice. Proceedings of the ACM on Human-Computer Interaction 3
(2019). Issue CSCW. https://doi.org/10.1145/3359174
[26] Valerie S. Morash, Yue-Ting Siu, Joshua A. Miele, Lucia Hasty, and Steven Lan-
dau. 2015. Guiding Novice Web Workers in Making Image Descriptions Using
Templates. ACM Trans. Access. Comput. 7, 4, Article 12 (nov 2015), 21 pages.
https://doi.org/10.1145/2764916
[27] Meredith Ringel Morris, Jazette Johnson, Cynthia L. Bennett, and Edward Cutrell.
2018. Rich representations of visual content for Screen reader users. Conference
on Human Factors in Computing Systems - Proceedings 2018-April (2018), 1–11.
https://doi.org/10.1145/3173574.3173633
[28] Meredith Ringel Morris, Annuska Zolyomi, Catherine Yao, Sina Bahram, Jefrey P.
Bigham, and Shaun K. Kane. 2016. "With most of it being pictures now, I rarely use
it": Understanding Twitter’s Evolving Accessibility to Blind Users. In Proceedings
of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, New
York, NY, USA, 5506–5516. https://doi.org/10.1145/2858036.2858116
[29] Helen Petrie, Chandra Harrison, and Sundeep Dev. 2005. Describing images on
the web: a survey of current practice and prospects for the future. In Proceedings
of Human Computer Interaction International (HCII), Vol. 71.
[30] Carolina Sacramento, Leonardo Nardi, Simone Bacellar Leal Ferreira, and João
Marcelo dos Santos Marques. 2020. #PraCegoVer: Investigating the description
of visual content in Brazilian online social media Carolina. In Proceedings of the
19th Brazilian Symposium on Human Factors in Computing Systems. ACM, New
York, NY, USA, 1–10. https://doi.org/10.1145/3424953.3426489
[31] Elliot Salisbury, Ece Kamar, and Meredith Ringel Morris. 2017. Toward Scalable
Social Alt Text: Conversational Crowdsourcing as a Tool for Refning Vision-to-
Language Technology for the Blind. Aaai Hcomp 17 Hcomp (2017), 147–156.
[32] Abigale Stangl, Meredith Ringel Morris, and Danna Gurari. 2020. "Person, Shoes,
Tree. Is the Person Naked?" What People with Vision Impairments Want in
Image Descriptions. In Proceedings of the 2020 CHI Conference on Human Factors
in Computing Systems. ACM, New York, NY, USA, 1–13. https://doi.org/10.1145/
3313831.3376404
[33] The New York Times. 2020. The Coronavirus Revives Facebook as News Power-
house. https://www.nytimes.com/2020/03/23/technology/coronavirus-facebook-
news.html Accessed: 2020-08-12.
[34] Twitter. 2016. Accessible images for everyone. https://blog.twitter.com/en_us/a/
2016/accessible-images-for-everyone.html
[35] Twitter. 2020. Twitter Accessibility. https://twitter.com/TwitterA11y/status/
1265689579371323392
[36] Violeta Voykinska, Shiri Azenkot, Shaomei Wu, and Gilly Leshed. 2016. How
Blind People Interact with Visual Content on Social Networking Services. In
Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work
& Social Computing - CSCW ’16, Vol. 1. ACM Press, New York, New York, USA,
1582–1593. https://doi.org/10.1145/2818048.2820013
[37] WebAIM. 2017. Screen Reader User Survey. https://webaim.org/projects/
screenreadersurvey7/
[38] Gill Whitney and Irena Kolar. 2020. Am I missing something? Universal Access in
the Information Society 19, 2 (jun 2020), 461–469. https://doi.org/10.1007/s10209-
019-00648-z
[39] Shaomei Wu and Lada A. Adamic. 2014. Visually impaired users on an online
social network. In Proceedings of the 32nd annual ACM conference on Human
factors in computing systems - CHI ’14. ACM Press, New York, New York, USA,
3133–3142. https://doi.org/10.1145/2556288.2557415
[40] Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2017. The efect
of computer-generated descriptions on photo-sharing experiences of people with
visual impairments. Proceedings of the ACM on Human-Computer Interaction 1,
CSCW (2017), 1–22. https://doi.org/10.1145/3134756
Support in the Moment: Benefits and use of video-span selection
and search for sign-language video comprehension among ASL
learners
Saad Hassan∗ Diego Navarro
Akhter Al Amin∗ National Institute for the Deaf
Caluã de Lacerda Pataca∗ Rochester Institute of Technology
Computing and Information Science Rochester, NY, USA
Rochester Institute of Technology don6763@rit.edu
Rochester, NY, USA
{sh2513,aa7510,cd4610}@rit.edu
ABSTRACT
As they develop comprehension skills, American Sign Language
(ASL) learners often view challenging ASL videos, which may con-
tain unfamiliar signs. Current dictionary tools require students to
isolate a single sign they do not understand and input a search query,
by selecting linguistic properties or by performing the sign into
a webcam. Students may struggle with extracting and re-creating
an unfamiliar sign, and they must leave the video-watching task
to use an external dictionary tool. We investigate a technology
that enables users, in the moment, i.e., while they are viewing a
video, to select a span of one or more signs that they do not under-
stand, to view dictionary results. We interviewed 14 American Sign
Language (ASL) learners about their challenges in understanding
ASL video and workarounds for unfamiliar vocabulary. We then
conducted a comparative study and an in-depth analysis with 15
ASL learners to investigate the benefts of using video sub-spans
for searching, and their interactions with a Wizard-of-Oz prototype
during a video-comprehension task. Our fndings revealed benefts
of our tool in terms of quality of video translation produced and
perceived workload to produce translations. Our in-depth analy-
sis also revealed benefts of an integrated search tool and use of
span-selection to constrain video play. These fndings inform future
designers of such systems, computer vision researchers working
on the underlying sign matching technologies, and sign language
educators.
CCS CONCEPTS
• Human-centered computing → Accessibility systems and tools;
Graphical user interfaces; Empirical studies in interaction design; User
interface programming.
∗ These authors contributed equally to this research.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544883
Alexis Gordon
Sooyeon Lee
Matt Huenerfauth
School of Information
Rochester Institute of Technology
Rochester, NY, USA
{aag7593,slics,matt.huenerfauth}@rit.edu
KEYWORDS
American Sign language, Sign Languages, Continuous Signing, Sign
Language Videos, Sign Look-up, Video Selection, Search Interface,
Integrated Search, Sign Language Learning, ASL Learning
ACM Reference Format:
Saad Hassan, Akhter Al Amin, Caluã de Lacerda Pataca, Diego Navarro,
Alexis Gordon, Sooyeon Lee, and Matt Huenerfauth. 2022. Support in the Mo-
ment: Benefts and use of video-span selection and search for sign-language
video comprehension among ASL learners. In The 24th International ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS ’22), Oc-
tober 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 14 pages.
https://doi.org/10.1145/3517428.3544883
1 INTRODUCTION
Over 70 million Deaf and Hard of Hearing (DHH) people worldwide
use one of the over 300 sign languages recognized by the World
Federation of the Deaf [12, 48]. In the U.S., increasing numbers
of DHH and hearing individuals are motivated to learn American
Sign Language (ASL), which is used by about 500,000 people as
a primary form of communication [46]. There are nearly 200,000
students in ASL classes [15] at schools or universities, and ASL has
one of the fastest growing enrollments among language classes [17].
Learning ASL can promote interactions between DHH and hearing
individuals, to support greater inclusion, mutual understanding,
and participation across society. Further, if DHH children cannot
access spoken language nor learn sign language during critical
developmental years, they may experience language deprivation
[20]. As most DHH children are born to hearing parents, their
family or teachers are motivated to learn sign languages [53, 61].
Students trying to understand a challenging video is part of
sign language education, to develop comprehension skills [18, 29,
38]. While there have been advances in machine translation to
automatically convert an ASL video into an English text, such
technology is still under development [51], and its use would bypass
the educational activity of students working to understand a video
themselves. Technology is needed to support learners during a
video-comprehension task without fully automating the process.
In foreign-language-learning contexts, dictionaries are a valuable
tool for students when faced with an unknown word in a text or
audio recording. However, students learning sign languages face
challenges when encountering a sign whose meaning they do not
know, given the lack of a standard writing system or an easy way
ASSETS ’22, October 23–26, 2022, Athens, Greece
for students to search for the meaning of a sign based on its visual
appearance. Existing search tools are not sufcient [3, 24, 25], and
it is difcult for students to use websites that ask them to enter
linguistic features of the sign and browse a list of results to fnd the
sign that matches what they see [1, 6, 7, 40, 45, 56, 59]. Based on
automatic ASL-recognition technology for video matching, recent
tools have been researched to enable students to submit a video
of a single sign to conduct a search within an ASL dictionary [3,
5, 11, 13, 26, 37, 59, 63]. However, students trying to understand
an ASL video may not be able to accurately extract just one sign
nor replicate the sign themselves into a webcam to initiate a search
[3, 26]. For this reason, we investigate technologies for enabling
users to quickly select a span of a video of ASL signing that contains
one or more signs that they do not understand, and then trigger
a video-analysis search that will return a set of dictionary results
with likely matches to the signs contained within the span.
Rather than considering separately the tasks of watching a video
and looking up the meaning of an unknown sign, we instead focus
on the overall task. Based on feedback from ASL students dur-
ing an interview study, we investigate an integrated video-player
and sign-search tool. In our second study, some students watched
videos containing some signs that they were not familiar with,
while using a Wizard-of-Oz prototype for playing videos, selecting
spans of video, and performing searches for signs. Other students
used a baseline prototype without the searching functionality. This
study shed light on how ASL learners interact with such a system.
This study also investigated the beneft of providing integrated
dictionary search to ASL learners (hearing university students) for
video-comprehension, in comparison to their use of an existing
dictionary website. Our contributions include:
• We present an interview-based study with ASL learners
about their experiences watching challenging ASL videos.
Our novel fndings reveal their desire to view videos of
various genres from multiple platforms, factors that lead
to challenges in video comprehension, and their current
workarounds when facing unfamiliar signs.
• In the task context of ASL learners translating difcult ASL
videos, we present the frst comparative study between our
video-player prototype with integrated dictionary-search,
in comparison to an existing search-by-feature dictionary
website. Our fndings reveal benefts in terms of quality of
translations produced and reduced workload.
• We present the frst observational study of ASL learners en-
gaged in the task of translating difcult ASL videos while
using search technology, specifcally a Wizard-of-Oz pro-
totype of our proposed system. Our analysis revealed how
users selected sub-spans and conducted searches, and we
characterize how users benefted from an integrated tool
that presented search results alongside the video (enabling
checking of results in context). We found that usage varied
depending on the genre of signing video, and we observed
unexpected use of the sub-span selection tool for the purpose
of constraining the video play-head.
Hassan, et al.
2 BACKGROUND AND RELATED WORK
Background on sign-language linguistics is provided below to ex-
plain key terminology, and prior work on sign-language pedagogy
is discussed, to contextualize our work within that domain. Next,
section 2.2 discusses the current state of sign-language look-up
technologies, to illuminate key limitations of existing resources.
2.1 Sign Language Comprehension
ASL linguistic phenomena contribute to challenges students may
face in comprehending an ASL video. Although students can browse
dictionaries that show videos of an ASL sign’s citation form, i.e.,
the standard way in which a sign may appear when produced in iso-
lation, when signs are produced during sentences in a continuous
manner, the appearance may difer. For instance, there is natural
diversity in the production of signs across individual signers, which
may be based on demographic or geographic regional variation.
Further, two or more ASL signs may linguistically combine into
a compound sign [41]; novice ASL learners may have difculty
segmenting them appropriately to look up a meaning in an ASL
dictionary [8]. Coarticulation, broadly, refers to how the produc-
tion of one sign may afect the way in which other nearby signs are
produced in continuous signing [19, 54], e.g., the ending location or
handshape of one sign may afect the location or handshape of the
next. Coarticulation efects may lead to the production of a sign in
context to difer from its citation form. When ASL signers produce
rapid sequences of handshapes during fngerspelling, i.e., when
specifc words are spelled alphabetically, coarticulation efects are
also possible [35], leading to the fngerspelled word not being a sim-
ple concatenation of the individual alphabet handshapes. Finally,
ASL signing may include depiction, in which particular linguis-
tic constructions, often referred to as “classifers," convey spatial
information about the position, movement, or shape of entities [58].
Most prior work on sign-language video comprehension has
focused on Deaf users. Little prior work–and no prior observa-
tional studies–have investigated the behavior of hearing people
when watching a challenging ASL video nor their workarounds for
unknown signs, e.g., using sign look-up tools. A recent review of
prior eye-tracking studies with DHH participants [2] discussed a
study that examined diferences in gaze patterns between Deaf and
hearing individuals when looking at a live signer [55]. Although
understanding sign language in person is diferent then watching a
video (and no sign look-up technologies had been used), that study
characterized gaze patterns of hearing people when trying to com-
prehend sign language. Observational studies, with eye-tracking or
other means, may lead to insights about behaviors of ASL learners
during video comprehension, especially given limited prior work.
In contrast, substantial prior literature exists on non-native learners
of various spoken languages engaging in video comprehension, and
there has been work on spoken/written language translation tasks
while learners use various electronic resources [4, 16, 27, 44, 57, 64].
2.2 State of Sign Language Lookup Resources
Dictionaries are an important tool used by second language learners
when looking up an unfamiliar word. However, when someone
encounters a sign whose meaning they do not know when viewing
sign language, it is more difcult to look up the word, since sign
Support in the Moment
languages lack a common writing system and users cannot use a
text-search or alphabetical listing to search for a sign [5, 28].
Some sign-language dictionary systems expect users to recall
linguistic properties of the sign that they are looking for–e.g., hand
confguration, orientation, location, movement–and enter these
properties into a search-query interface to obtain a list of matching
signs [1, 6, 7, 40, 45, 56, 59]. Prior work on such search-by-feature
systems has revealed that they are challenging for ASL students
[6]. Other proposed sign-language dictionary systems expect users
to submit a video of a single sign that they have extracted from a
longer video – or to recall and perform a sign into a webcam [5, 7, 11,
13, 37, 59, 63]; sign-recognition technology performs a video search
against a dictionary to provide potential matches. Even if a student
were able to remember and produce a sign into a webcam, there are
technical challenges in recognizing signs from video due to various
factors [52, 62]. Despite recent advancements [47, 51], state-of-
the-art continuous sign-recognition software is still imperfect. To
mitigate inaccuracies in the video-to-sign matching, some proposed
dictionaries provide users with post-query fltering options, to
narrow the set of results that are returned [26]. Overall, existing
dictionary systems face several limitations: They expect that the
ASL learner can recall linguistic properties of the desired ASL sign
or accurately perform the sign from memory. In addition, systems
assume that a user is able to precisely identify the starting and
ending of a sign they encountered in a video or in a conversation.
Fast signing speed or various linguistic factors (section 2.1) make it
difcult for ASL learners to precisely select signs in videos. Finally,
the user must launch an additional task of querying a dictionary,
while engaging in a video-watching and comprehension task. Using
separate tool to perform the search may cause users to lose context
from the video they were watching.
In contrast to prior work, we investigate a dictionary-search
system that enables the user to select a span (of potentially multiple
signs) from a video of continuous sign language, as a basis for
a query to search for potential matching signs from a dictionary
system, with the results presented in an integrated video-player
and search-results interface. This approach may mitigate the need
for users to recall specifc linguistic properties of the unknown
sign, mitigate the need to identify the specifc start/end of signs
in a continuous video, and enable the user to remain in context in
their video-watching-and-comprehension task.
Some recent research has investigated ASL learners interacting
with Wizard-of-Oz prototype systems for ASL dictionary search,
to identify factors that afect users’ satisfaction [3, 23–26]. Method-
ologically, our studies also employ a Wizard-of-Oz prototype of
an ASL dictionary-search system to understand users’ interaction
and potential benefts. However, in those prior studies, users had
been shown a stimulus video of native signer performing a single
isolated sign (in citation form), and the user was asked to use a
dictionary system to identify the sign’s meaning. In contrast, our
studies examine how ASL learners engage in a search task while in
the midst of a video-watching-and-comprehension task.
2.3 Research Questions
There has been limited prior research on the experience of ASL
learners who are engaged in the educational activity of watching
ASSETS ’22, October 23–26, 2022, Athens, Greece
a sign-language video that is difcult for them, especially how
technologies for ASL dictionary look-up may beneft these users.
In addition, no prior work has examined how users might beneft
from an integrated tool for viewing ASL videos, with users able
to select spans of the video as the basis for dictionary search. To
address these gaps, we investigate the following research questions:
RQ1 What are the challenges that ASL learners currently expe-
rience when trying to understand a difcult sign-language
video, and what workarounds do they employ?
RQ2 Comparing the experience of users who used our tool and
those who used an existing feature-based ASL-English re-
verse dictionary, is there a diference between translation
quality or perceived workload to produce translation?
RQ3 How do users interact with a Wizard-of-Oz prototype for
viewing an ASL video and conducting dictionary-search
on selected spans of video, during the video-watching and
comprehension task?
3 STUDY 1: INTERVIEW STUDY
This paper presents two studies: The goal of study 1 was to un-
derstand ASL learners’ challenges with video comprehension and
current workarounds they use. The fndings from study 1 informed
the study design, videos, and prototypes included in study 2.
3.1 Study Design
This IRB-approved study was conducted in person or remotely
based on the preferences of participants during the COVID-19 pan-
demic. After informed consent was obtained, the semi-structured
interview began with questions about participants’ prior experi-
ences watching ASL videos. They were asked about the type of
videos they watch, their experiences when they have difculty un-
derstanding, and any workarounds they use. To provide context
for later questions about how difcult it may be for participants
to select an individual sign or span of multiple signs they do not
understand, we displayed several example videos to participants as
a basis for discussion. These videos were taken from advanced ASL
or ASL-English interpreting classes, conversational videos between
expert signers on YouTube, signing performances at theatres, and
interpreted poetry and music. (Video details appear in electronic
supplementary fles.) Videos contained a variety of linguistic phe-
nomena discussed in section 2.1. Participants were asked how hard
it would be to select a sub-span containing one or multiple signs
and how they would select a time-range of a video. The average
length of each interview was 36.5 minutes (σ =5.46 minutes).
3.2 Participants and Recruitment
Participants were recruited by posting an advertisement on an
ASL Reddit channel and by contacting professors of introductory
ASL courses, who shared an advertisement by email with their
students, containing two screening questions: “Are you currently
learning American Sign Language?” or “Have you completed an
introductory or intermediate ASL course in the past fve years?”
Participants were recruited if they responded with yes to at least one
question. We recruited a total of 14 participants for our frst study,
which included, 2 men, 11 women, and 1 non-binary individual.
The median age was 21 (σ = 3.67). Participants had studied ASL for
ASSETS ’22, October 23–26, 2022, Athens, Greece
a mean of 3.4 years, and all participants confrmed that they had
taken fewer than 3 years of formal ASL classes.
3.3 Analysis and Findings
We employed a mixture of deductive and inductive approaches in
our qualitative data analysis. To become familiar with the interview
transcripts, two authors read all 14, then during a subsequent read-
ing, they individually took notes to produce initial codes, which they
collated and collapsed into two individual code-books. Each of the
authors then investigated underlying patterns among their codes
and formed initial categories, and they consulted the interviewers
to get feedback on their initial categories and further improved
them. The authors then met to review all of their initial categories,
to identify similarities and diferences. During two three-hour meet-
ings, the authors performed an initial thematic grouping, which
lead to fnal high-level categories. These high-level categories were
then presented to the rest of the team to arrive at a fnal set of
themes and sub-themes presented in this section.
3.3.1 Prior experiences and challenges associated with watching
signed video content. Participants discussed various motivations
for viewing signing content. Twelve participants mentioned en-
gaging with ASL videos during classroom-related or homework
activities. Ten participants also mentioned watching signed con-
tent outside of the classroom for their own enrichment or personal
exposure to other types of signing, e.g., Deaf theatre or ASL songs.
P11 said, "It’s both in-class, we have diferent assignments the teacher
will give us, and then I also do it on my own time, if I’m looking for a
deeper understanding about things, or if I’m looking for specifc signs.
And I also follow some deaf content creators as well."
Participants also discussed how their lack of familiarity with
regional or dialectical variation in signing, such as Black ASL
[43] used among some African-American signers in the U.S., led to
challenges in understanding videos. P5 described their experience
in understanding signing among various communities: "I know some
white people in the community, [but the] black Deaf community and
the interpreter community, I still fnd hard."
Participants discussed how various linguistic types of signs
posed comprehension challeges. For instance, P1 described needing
to consciously "switch my brain from a sign to actually each letter"
when encountering fngerspelling. P11 discussed challenges with
"fngerspelling, classifers, compound words, any of that kind of stuf...
fngerspelling is defnitely a little tougher for me." Participants dis-
cussed how fngerspelled names were challenging to understand
in a video, especially when there were multiple individuals with
similar names. Participants also described challenges with under-
standing numbers, e.g., P5 said, "numbers are hard for me, for some
reason, I don’t know why." Participants also discussed challenges
with compound signs, e.g., P13 said, "I wasn’t sure if that was one or
two separate signs. So there were defnitely points in the video where
they were blending together a little bit, and I wasn’t sure."
Overall, participants discussed how diferent content sources
or genres pose challenges for comprehension. Participants men-
tioned viewing signed content on various streaming services, e.g.,
YouTube and Netfix, as well as on social media, e.g., Instagram and
TikTok. P14 said, "I watch ASL videos when I am going through In-
stagram because I follow some Deaf creators." Participants discussed
Hassan, et al.
how the signed content on social media is shorter and more un-
predictable in nature, with the topic of the video not always well
defned, which poses challenges for comprehension. Participants
also discussed how factual signing, e.g., in a documentary, was dif-
fcult, due to complex vocabulary or increased use of fngerspelling.
Other participants mentioned watching videos of ASL poetry and
ASL translations of popular songs, contexts in which they described
signers as using more depiction and having "their own fow, and they
have their own rhythm" (P11). Participants described how videos
with multiple signers, e.g., Deaf theatre, pose challenges, as P8 de-
scribed, "my brain is used to practicing with one signer." Similarly,
participants mentioned how natural conversations were difcult to
understand, e.g., P6 discussed how signing in such videos tends to
be "quicker, and they’re a little bit more relaxed."
3.3.2 Workarounds. Participants mentioned several workarounds
that were useful in understanding challenging signed video content.
For instance, several participants discussed using the context of a
video to understand unknown signs. Participants would consider
the description or title of the video, such as on YouTube, or they
would consider what was said before or after any unknown signs.
For instance, P3 described a situation in which they fgured out the
sign for a citrus fruit by considering the context of the surrounding
signing, which had mentioned lemons. P3 discussed how under-
standing later signing may clarify a portion of signing that had
not been previously understood, explaining how if they become
confused then they "really focus on the next thing they’re saying, so I
can piece together what they might have said, so I can understand it."
Participants discussed various strategies that involved controlling
the fow of the video player:
• Periodically pausing was a strategy among several partic-
ipants. For instance, participants discussed how they paused
videos in-between conversational turns in videos with mul-
tiple signers; P8 described how they "pause in between each
speaker... just enough time to grasp" what had been said.
• Backtracking and replaying was another common ap-
proach, as P12 explained, "pausing it and replaying it." P3
also discussed how they will "backtrack the video" if needed.
• Slowing down the video was also popular, if possible within
the video player. For instance, P7 explained how they will
"slow down the fngerspelling if...it’s on YouTube. If I could
alter the speed, I might try to slow it down."
Regarding current strategies for seeking the meaning of an un-
known sign, six participants mentioned using English-to-ASL
dictionaries, i.e., guessing English meanings of the sign they did
not understand to look up that English word in the dictionary to see
if the sign displayed visually matched the sign that had not been
understood. Participants also mentioned using ASL-to-English
“reverse” dictionaries, i.e., websites that allow someone to enter
linguistic properties to search for the English translation of a sign.
Participants discussed challenges, e.g., P7 said, "if I think I have an
idea of what the sign is I might use Handspeak, or there’s another
one I use... [It’s] hard to specify handshape in current dictionaries."
P6 discussed struggling to enter linguistic properties when con-
structing a query: "I defnitely tried using the reverse dictionary stuf
online. Usually it doesn’t end up being successful, and I have to just
end up moving on. Because, the way it’s structured, you have the
Support in the Moment
handshape, and the movement, and the location. Sometimes it’s a
little ambiguous, especially if you don’t actually know what that sign
is; so, it’s hard to end up looking [it] up." Participants also expressed
their frustration with having to launch a web-dictionary in another
window while trying to understand a video. P11 said, "It’s pretty
frustrating sometimes when I’m trying to fnd a specifc sign, I have
to like go to Google and...then go through all the diferent pages... If I
could just scroll and have the source material right there, I think it
would be much more efcient."
Rather than use a specifc dictionary website, other participants
mentioned typing descriptions of what a sign looked like into a
Google search, e.g., P11 said, "I’ve defnitely tried to Google it before,
but it’s so hard to sometimes describe what it is that you’re looking
for. I end up being very vague... It’s very rare that I go to Google and
fnd what I’m looking for as far as trying to describe a sign." Finally,
several participants mentioned that, if other people are available,
they may ask a teacher or a peer. As P09 said, "If I’m in class I
would ask the teacher. If it’s for a class I would either look it up online
or if it’s in a vocabulary learning unit."
4 STUDY 2: PROTOTYPE STUDY
While study 1 motivated the need for a tool that provided an inte-
grated video-playing and sign-search experience, study 2 investi-
gated how users would interact with such a tool and whether there
are benefts as compared to existing systems. In this prototype
study, users interacted with a Wizard-of-Oz prototype, at a desktop
computer in a lab setting. Findings from study 1 informed both the
design of the prototypes used and the selection of videos included
in both the prototypes, as described below.
4.1 Prototype Design
4.1.1 Integrated Search. Since the focus of this study was on users’
interaction and behavior, an interactive Wizard-of-Oz prototype
was designed (Figure 1), in which the underlying sign-recognition
technology was simulated, without any automatic video analysis.
On this web-based prototype, participants entered a participant
ID on the frst screen. Next, they were provided a calibration screen
to ensure that the size and aspect ratio of their browser window
was consistent. As shown in Figure 1, the interface displayed an
ASL video with a play/pause button at the bottom-left corner of the
screen. On a video-timeline at the bottom of the screen, a vertical
white line indicated the video playhead (the current position of
the video). The users can select a video span by dragging yellow
edges of a selection bar on this timeline, and they can press a "Play
selection" button to play only the portion of the video in that span.
Once satisfed with their span selection, users can click on the
yellow "Search selection" button on the top right corner of the
screen to search for the signs. The results were displayed in a scroll-
able window on the right side of the screen. Each result consisted
of a video of a sign from the American Sign Language Lexicon
Video Dataset (ASLLVD) [50] and a label below showing the closest
English gloss for that sign. When clicking on a "more information"
icon for each search result, linguistic properties for the item were
displayed, as illustrated on the right side of Figure 1.
Study 1 fndings informed the prototype design: For instance,
participants expressed frustration with needing to leave the context
ASSETS ’22, October 23–26, 2022, Athens, Greece
of watching a video to use electronic dictionaries, and this informed
our decision to display dictionary results on the same page. In addi-
tion, participants’ explanation of workaround strategies they used
when encountering difcult video, e.g., re-playing and backtrack-
ing a video, led us to provide the "Play selection" button, which
played the video while the playhead was constrained to the selected
span–to support users in replaying a short segment of the video.
Findings from study 1 also informed the selection of videos
shown to participants during the study. Based on participants’ com-
ments about how the genre or linguistic phenomena in a video
relates to how challenging it is, we selected three genres of videos
to display in this study, including educational videos, conversa-
tional videos with at least two signers, with turn-taking between
them, and Deaf theatre and poetry videos. Since participants had
discussed how they face particular challenges when encountering
fngerspelling or compound signs, we ensured that the videos in-
cluded instances of these types of signing. In addition, we selected
videos that included multiple signers engaged in conversational
signing, as well as a video with diferent regional dialects of signing,
since both had been discussed by participants in study 1. The videos
used in this study were obtained from online sources and material
from fourth-year ASL interpreting courses. Videos were an average
of 23.7 seconds in length, and details about each video are provided
in electronic supplementary fles.
4.1.2 Selection of signs appearing in the results list. Given the rapid
pace of advancement in the feld of sign-recognition technology, and
since the purpose of this observational study was to understand
participants’ behavior and interaction, we selected a Wizard-of-
Oz approach to simulate an automatic search-recognition system.
Therefore, our system returned a pre-determined list of results,
in which the actual sign appeared somewhere in the results list.
The selection of signs that appeared on the results was based on
pre-processing of videos. The protocol is described below:
(1) A Deaf member of our team with native ASL fuency watched
all 9 of the ASL videos in advance to identify the sequence
of signs appearing in each video, along with the starting and
ending time-stamps of each sign.
(2) For each sign, a set of dictionary-search results were manu-
ally prepared, to simulate the type of results someone would
see if using a real automatic dictionary-search system in the
future. Specifcally, for each sign, a native signer carefully
selected the closest match and 11 other signs that were simi-
lar in appearance to the sign, from a collection of signs from
the ASLLVD [50]. The researcher prioritized selecting signs
for this “match list" with as many properties in common as
possible to the given sign, i.e., the same handshape, number
of hands, movement, and location.
(3) When a participant selects a span of video, it is possible that
the start or end of the span is within the duration of a sign in
the video, rather than precisely at a boundary between signs.
We established a rule that the prototype would consider a
sign to be within a span selected by the participant if at least
half of the sign appears within the selected span.
(4) Since a selected span may contain multiple signs, the list of
dictionary-search results displayed combined results from
the match lists for the all signs within that span, as follows:
ASSETS ’22, October 23–26, 2022, Athens, Greece Hassan, et al.

Figure 1: Screen image of the prototype, displaying labels for the various interface regions, including the video player at the
top left, a text box where translations can be typed below, a video timeline with a span-selection interface along the bottom,
and a dictionary-search panel on the right.

From among the match lists for all signs in the span, one
match list was selected randomly, and the top item from
that list was taken (without replacement) for inclusion on
the combined results list. This process was repeated until all
match lists were empty, to produce a combined list of results
that contained the union of the original match lists. The frst
50 items from this combined list were displayed to users.
4.1.3 Baseline Prototype. We also designed a “baseline" prototype
identical to the one described above but without the integrated
dictionary-search option, as shown in Figure 2(a). When viewing (a)
a video with this baseline, participants were instructed to open a
second web-browser window to use the handSpeak reverse dictio-
nary 1 , as shown in Figure 2(b). That site enables users to look for
an isolated ASL sign by selecting text labels that represent various
linguistic properties of a sign, e.g., handshape, hand location, move-
ment, and orientation. Based on the query options selected, results
appear as a list of English gloss labels for matching signs. Partici-
pants were not allowed to visit other websites or other resources.

4.2 Study Design and Analysis Plan (b)

After providing informed consent in this IRB-approved study, par-
ticipants were asked to view a video and produce an English transla-
tion text for it using the integrated-search prototype or the baseline
prototype. Using a sample video, the researcher frst demonstrated
the prototype (details in section 4.2). After indicating that they
understood the prototype, each participant viewed 9 videos. The
order of videos was randomized. Their interaction was recorded:
(1) The software prototype was designed to automatically record
the starting and ending points on the video timeline of every
Figure 2: (a) The prototype for the baseline condition in
Study 3, identical to the one from Study 2 but without any
dictionary-search ability. (b) The handSpeak ASL-English
reverse dictionary website, which participants used in the
Study-3 baseline condition. As users click on linguistic prop-
erties, the list of English gloss labels at the bottom of the
window updates to list matching signs.

1 https://www.handspeak.com/word/asl-eng/
Support in the Moment
span the participant selected, the number of signs within
each span, whenever the participant triggered a search, and
the text of the English translation typed by the participant.
(2) Each participant’s face was approximately 65cm from a 19-
inch monitor, to which a Tobii Nano [49] 60Hz screen-based
eye-tracking device was attached. iMotions (v9.1) [30] soft-
ware recorded each participant’s gaze.
(3) A researcher, who was a fourth-year English-ASL interpret-
ing student at the university, sat 2m away and took observa-
tional notes during the experiment. The iMotions software
enabled the researcher to monitor the participant’s gaze on
the user-interface in real-time on a secondary display.
At the end of the entire session, a debriefng interview was con-
ducted to gather participants’ impressions of the system, perception
of how they interacted with the device, and other recommenda-
tions. The interview data was transcribed and coded using the same
methodology as in study 1.
Qualitative analysis of the data listed above was performed by
two members of our research team who reviewed and coded this
data, from the perspective of identifying typical sequences of inter-
action behavior during each video session. They reviewed record-
ings of the screen and eye-gaze, plotted eye-gaze patterns, analyzed
data captured by the software prototype, and reviewed the ob-
server’s notes. The researchers discussed their notes and agreed
upon a categorization of the behaviors observed, as presented in
Findings section 4.5. After viewing and translating each video for
both conditions, participants’ English translation texts were saved,
and participants completed a NASA TLX [21, 22].
4.3 Participants and Recruitment
We recruited a total of 15 ASL students for study 2, using recruit-
ment criteria and approaches identical to study 1. 8 participants
were assigned to the integrated search condition whereas 7 were
assigned to the baseline condition.
The median age of participants in the integrated search condition
was 20, and this included 7 women and 1 non-binary individual.
Participants had studied ASL for a mean of 3.5 years, and all partic-
ipants confrmed that they had taken fewer than 3 years of formal
ASL classes.
The median age of participants who used the baseline prototype
was 21, and they included 4 women and 3 men. A single recruitment
process was conducted and participants were randomly assigned to
either the prototype-with-dictionary-search or baseline-prototype
condition. Participants using the baseline prototype had studied
ASL for a mean of 3.7 years, and all participants confrmed that
they had taken fewer than 3 years of formal ASL classes.
4.4 Findings: Comparative Study
To assess translation quality, we adapted a prior approach [9], in
which a human judge looked for translation errors in a text (e.g.,
wrong or omitted words) and then assigned an overall translation-
accuracy score (out of 10). In our study, a fourth-year ASL inter-
preting student, who had completed a university course on ASL
linguistics, analyzed the transcripts to identify errors and assign
translation-accuracy scores—without knowing which translations
had been produced using the dictionary-search prototype and
ASSETS ’22, October 23–26, 2022, Athens, Greece
which had been produced using the baseline prototype. The average
translation-accuracy score assigned by the researcher was 8.03 for
translations produced using the dictionary-search prototype and
6.67 for the baseline prototype. The distributions in scores between
the two conditions difered signifcantly (Mann–Whitney U = 10,
n1=8, n2=7, P = 0.0424 < 0.05 two-tailed, r = 0.24).
Figure 3 shows scaled mean raw NASA-TLX sub-scores (physical
demand, temporal demand, performance, efort, and frustration)
and results of two-tailed Mann-Whitney U tests comparing sub-
scores across conditions. Participants who used the prototype with
dictionary-search gave signifcantly lower values for mental de-
mand (how much mental and perceptual activity was required),
temporal demand (how much time pressure was felt), and frustra-
tion (how insecure, discouraged, irritated, or stressed they felt). The
NASA TLX instrument and details of these scales appear in [21, 22].
4.5 Findings: In-depth Analysis
We present an in-depth analysis of how participants interacted with
the integrated search prototype. Since prior work had investigated
the experience of students with existing dictionary-search websites,
e.g., [6, 26], we do not present a detailed observational analysis of
users interacting with the baseline, given the limited novelty of
such an analysis, amid prior literature.
4.5.1 Using the span selection to constrain the playhead. Six par-
ticipants used the span selection tool to constrain the portion of
the video that would play at one time, to enable them to progress
incrementally through the videos. Participants selected spans of
average duration 11.43s for viewing videos in this manner, and they
progressively selected spans of video as they typed the English
translation text. Figure 4(a) illustrates this trend, by plotting the
positions of spans a participant selected over time. Span 1, shown
at the bottom of the image, indicates the frst span selected.
Comments from debriefng interviews also support this observa-
tion. P7 described how they selected a span of a particular width,
and then dragged it along the video, to watch portions of video
progressively: "I like how you can just maintain the length and you
just drag it over so you’re getting the same length of a chunk of the
video; that was easy to use."
4.5.2 Approaching task linearly, sometimes afer initial overview.
Participants viewed the videos in a linear manner and produced
transcripts as they watched short segments of video. In some cases,
participants frst viewed the entire video, and then they returned to
the beginning of the video to progressively view short segments of
video in a linear manner, as illustrated in Figure 4(b), which shows
a full-video span prior to progressive short spans.
4.5.3 Using dictionary search to inform translation. In 62 out of
72 video sessions, participants made use of the dictionary-search
feature to look up the meaning of unknown signs in the video. As
illustrated in Figure 5, the results of the search tool informed par-
ticipants’ translation decisions as they linearly progressed through
the video. During the debriefng interview, P6 described how the
tool helped: “I knew what he was saying in general, I just couldn’t
think of the exact English words and that one came up right away."
P7 discussed the benefts of the tool during fast signing, “It was
defnitely useful, especially when the signers were going really fast
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 3: NASA TLX sub-scale scores from participants in the dictionary-search (n=8) and baseline prototype (n=7) conditions,
with scores scaled to a 0-to-100 range. For all sub-scales, lower scores are better, i.e., indicating less perceived demand, less efort
needed, less frustration, or better sense of performance success. Signifcance testing results from two-tailed Mann-Whitney U
tests are also presented on top of the bars.
because then I could double check to make sure that what I thought I
saw was actually what I saw."
4.5.4 Gradually making the span shorter prior to search. When
encountering a difcult portion of video, participants often reduced
the size of their span before initiating a dictionary search. When
selecting a span to view a portion of video (and not to conduct a
search), the average span width was 8.17 seconds (10.83 signs), while
the length of spans immediately prior to a search request was 2.33
seconds (3.25 signs). If a search result did not enable a participant to
identify the meaning of signs in a particular span, some participants
progressively narrowed the span width, to more precisely select
the specifc video portion where they were confused, and then
requested additional dictionary searches, as shown in Figure 6(a).
While adjusting spans, participants’ gaze alternated between the
main video region and the span-selection control, as shown in 6(b).
In debriefng interviews, participants described watching longer
segments to understand the context and then narrowing in on a
shorter span that was difcult to understand. P5 characterized their
approach as “narrowing it down and then pressing search." Other
participants discussed the benefts of beginning with a search of a
wider span initially, e.g., as P7 explained, “if I was just trying to get
the general idea of a section, it was helpful that sometimes there were
more signs in the up results besides like the specifc signs that I had
selected because it gave more context and it was easier to understand."
4.5.5 Using dictionary-search to confirm results afer initial transla-
tion. Among the 62 video sessions in which participants made use
of the dictionary-search tool, in 40 cases, we observed participants
using the search tool after they had already completed a full trans-
lation of the entire video. As illustrated in Figure 7, after writing a
translation for the whole video, the participant reviewed specifc
Hassan, et al.
earlier portions, using the search tool, to confrm that they had
correctly understood specifc signs.
During debriefng interviews, participants talked about various
ways in the search tool was useful in producing a more accurate
translation. For instance, P7 discussed how the search results moti-
vated them to adjust their wording in the translation, e.g., saying,
“I would go back and use the tool to make my translation more precise
I guess. So, I could fx the sentences and the wording." Other times,
the search results simply boosted their confdence in how they had
understood the video, e.g., with P7 saying “sometimes that helped
to confrm what I thought I saw."
4.5.6 Participants struggled to find a sign if a diferent version was
being signed in the results. In several instances, participants were
confused when the citation-form of the sign displayed in the search
results did not match the variation of the sign in the main video,
often in the case of compound signs and depiction. In Figure 8, P5
performed a search, but the specifc appearance of the sign in the
video difered from the citation form shown in the dictionary-search
results, which led the participant to glance back and forth between
them to compare. Ultimately, the participant did not produce the
correct translation for that portion of the video, suggesting that
they did not realize this was a match to what they had seen.
In debriefng interviews, participants how matching dictionary-
search results to signs in the video was more difcult for some video
genres. For instance, P4 discussed how when a sign was produced
with great emotion, e.g., in a theater video, then it was difcult
to match it to a dictionary-search result with more neutral afect.
P4 described their difculty with a video in which the signer “was
showing emotion and then you would go in the searches and they
wouldn’t. So it’s like, I guess you can get mixed up about the emotion."
Participants suggested that the dictionary-search system could be
Support in the Moment
(a)
(b)
Figure 4: In (a), P5 viewed a video in a linear manner, us-
ing the span selection to progressively view short video seg-
ments. Horizontal bars indicate spans selected with respect
to the total duration of the video. The frst span selected is
at the bottom of the y-axis, with subsequent spans appear-
ing higher on the y-axis. Spans in blue dotted lines are those
for which the participant pressed the “Search” button to con-
duct a dictionary search. The black lines at the bottom show
the actual signs on the video timeline. In (b), P8 frst selected
the entire video, played it, and then reduced the span to a
smaller duration and moved the span forward progressively.
improved by providing dialectical variations of each sign result and
an example of each sign’s use in a sentence, e.g., P3 said, “It would
be nice to see the sign in a context with more facial expressions or in
a sentence like you have in other dictionaries."
4.5.7 Diferences in span selections across genres of videos. As de-
scribed above, the 9 videos in the study were from three genres:
natural conversations, educational videos, and theatre/poetry per-
formances. An analysis of the span-selection data captured by the
prototype revealed that participants selected wider spans for the-
ater videos, as illustrated in Figure 9. When users were selecting
a span simply to constrain the playhead to view a portion of the
video, a Kruskal-Wallis test [H(2)=24.28, p<0.00001, η 2 = 0.043 (Small
Efect)] with Mann-Whitney post-hoc testing with Bonferroni cor-
rections, revealed that users selected wider spans for theatre videos.
Similarly, when users were selecting a span as input to a search,
the testing [H(2)=24.3031, p<0.00001, η 2 = 0.12 (Medium Efect)]
revealed that users also selected wider spans for theatre videos.
ASSETS ’22, October 23–26, 2022, Athens, Greece
(a)
(b)
Figure 5: P5 using the tool to produce a translation: (a) brows-
ing search results for a span and (b) after identifying the
meaning of the sign FIGURE-OUT, typing into the transla-
tion text box to continue the sentence: "They give books to
learn from and fgure out..."
In debriefng interviews, participants discussed why they se-
lected spans of diferent widths for diferent genres. P6 discussed
how theatre/poetry videos were challenging to understand gener-
ally, “This one had more of a poetic meaning and display; so, it did
take more focus to understand it for the translation.". P5 described
selecting spans while watching conversational videos in contrast
to the theatre/poetry performances: “the conversational ones, when
I chose these subspans were shorter, because they go back and forth a
lot [between signers]. But the poetry ones I feel are more conceptual...
so you can watch longer pieces. You don’t need to cut it down."
P7 discussed how the width of selected span depended on the
overall signing pace, but longer spans were generally needed for
theatre/poetry videos due to the style of signing: “Some of the ones
that were very visual, like the mushroom one and the moon one and all
those ones that were ASL storytelling type of very fgurative language...
It’s typically slower paced, and sometimes there’s a lot of repeated
signs. Or there’s a lot of just a depiction that’s very visual and doesn’t
have a lot of strictly vocabulary to go with it, but it’s more classifers.
I found that I would sometimes need a longer chunk in order to use
the tool and actually get relevant results of what was being signed."
Participants were free to select spans that did not align precisely
with the boundaries of when one sign ends and the next begins;
however, the results of dictionary-search could be controlled more
precisely if spans were selected more accurately. An analysis of the
mean error (in seconds), between each span selection boundary
and the nearest actual sign boundary, reveled that users were less
accurate when selecting spans during theatre/poetry videos. The
mean error for natural conversation videos was 0.19 seconds, for
educational videos was 0.23 seconds, and for theatre/poetry videos
was 0.55 seconds. A Kruskal-Wallis test [H(2)=5.2174, p=0.02236,
η 2 = 0.041 (Small Efect)] and Mann-Whitney post-hoc testing with
ASSETS ’22, October 23–26, 2022, Athens, Greece
(a)
(b)
Figure 6: P1 adjusting span width over time, while perform-
ing repeated search requests. (a) Blue dotted lines indicate
spans for which dictionary-search was requested, and red
solid lines indicate spans for which no search was requested.
For spans 7 to 10, the width reduces over time. (b) When fne-
tuning span width, participant 8’s gaze moves between the
video region and the span control. The yellow lines going
from the span to the video are showing the gaze pattern.
Figure 7: P7 had already completed an English translation
for the entire video, and then they returned to a few ear-
lier regions of the video and requested dictionary searches
to confrm their translation for specifc segments of video.
Bonferroni correction, revealed that the error in the case of the-
atre/poetry videos was higher than for the other two genres. In
debriefng interviews, P2, P6, and P7 discussed how aligning span
selection with actual sign boundaries was more challenging for
theatre/poetry videos. P7 explained that the type of signs within
Hassan, et al.
Figure 8: P5 glanced between a dictionary result for RE-
FLECT and corresponding portion of the video, where the
sign had been produced in a diferent manner, leading to P5
to believe they were diferent signs and ultimately produc-
ing an incorrect translation. The yellow lines going from the
span to the video are showing the gaze pattern.
such videos was a factor, “because of depiction... there weren’t clear
boundaries as much in the signs because it was using classifers."
5 DISCUSSION
While prior second-language pedagogical research had investigated
the challenges faced by students trying to understand texts with
difcult vocabulary [16], no prior study had investigated ASL stu-
dents’ experiences and challenges through frsthand interview and
observational methodologies. Our participants described how di-
alectical variation, linguistic types of signs, and various genres of
ASL content led to comprehension challenges. We investigated how
students currently approach understanding ASL videos contain-
ing unknown signs, revealing how students turn to online video
streaming, video sharing, and social media sites in order to gain
experience at understanding more diverse and natural examples of
signing. While the proliferation of video social media and stream-
ing services has diversifed and increased available ASL content
[60], most prior ASL pedagogical research on video comprehension
predates the ubiquity of such sources, e.g., [14]. Our fndings moti-
vate the need for a tool to (a) support students viewing videos they
seek from diverse sources and (b) with challenging content to
support developing comprehensions skills.
Study 1 also investigated students’ current workarounds for
videos with difcult signing, revealing their dissatisfaction with ex-
isting ASL dictionary resources. These fndings aligned with prior
work, e.g., [6], on the need for better tools to enable students in
identify the meaning of an unknown sign. A unique focus of our
study was that participants considered dictionary-searching chal-
lenges within the context of trying to understand a difcult video:
We found students were frustrated at needing to leave their video
to look up a sign in a separate website and their use of workarounds
like repeated pausing and rewinding of videos. These fndings
specifcally motivate HCI research on tools that support: (a) view-
ing and repeating short spans of video and (b) integrating the
dictionary-search tool into the video-playing experience.
Study 2 investigated the potential beneft of these tools. It
frst compared the full prototype and a baseline prototype in which
span selection was still available for constraining the playhead, yet
the students had to use an external, existing ASL dictionary website
Support in the Moment ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 9: Box and whisker plots illustrating span widths for each video genre, in two cases: spans selected immediately prior
to a search (top of graph) and spans selected not immediately prior to a search (bottom of graph). In each case, participants
selected wider spans for videos in the theater genre. Median are indicated by vertical bars within each box, means indicated
by x, and outliers by dots. Signifcant pairwise diferences are marked with asterisks: ∗∗ if p < 0.01, and ∗ ∗ ∗ if p < 0.001.

as a reference tool. Our fndings revealed that the integrated search
tool led to translations of greater accuracy and participants rated
the workload lower. Prior work had examined at how using in-
situ dictionaries improve comprehension for non-native speakers
[36], but this was the frst study to explore this in the sign-language
context. Study 2 also investigated how users would actually interact
with a such a system, using a Wizard-of-Oz prototype methodology.
While prior HCI research had investigated students interactions
with ASL dictionary-search interfaces, e.g., [3, 6, 25, 26], that work
had focused on students trying to look up a specifc sign from
their memory or search using a video of an isolated sign. Our
novel observational fndings include how students engage with a
dictionary search tool within the context of the overall video
comprehension task, e.g., replaying and comparing the portions
of the original video to potential sign matches. These observational
fndings in Study 2 aligned with the interview-based fndings from
Study 1, in which students had expressed frustration at losing their
video-watching context when using a separate dictionary tool.
Our observational fndings also revealed how students engaged
in a dual use of the video player’s span-selection interface: (a) to
constrain the playhead to progressive portions of video and (b)
to specify input for dictionary search. Notably, usage (a) served
as a probe within our analysis to identify the “window size" of
video that students viewed as they worked through the challenging
video. While some prior research on video analysis or annotation
tools for linguists analyzing ASL videos had incorporated a span-
selection interface for labeling regions of videos, e.g., [39, 50], no
prior research had investigated span selection in the context of ASL
learners viewing video. Our fndings therefore motivate further
HCI research on span-selection interfaces within this new context.
Our analysis also revealed that video players with span-selection
interfaces may be a useful probe during studies in which partici-
pants use such a tool to view videos may reveal their comprehension
strategies. For instance, our analysis revealed diferences in span
widths that users selected for diferent genres, and participants
discussed how their selection of span width related to the linguistic
properties, with wider spans for challenging theater/poetry videos.
Similarly, when span-selection boundaries were compared to actual
sign boundaries in videos, we observed more error during the-
ater/poetry videos. Thus, post-study analysis of the spans selected
by individuals who view ASL videos may reveal insights for ASL
linguistics or education researchers, and real-time analysis of spans
could enable adaptive educational software capable of identifying
when students are currently struggling while viewing a video.
Overall, the fndings of this paper are relevant to several au-
diences: The current experience and workarounds of students
that motivate research on integrated tools for students viewing
challenging ASL videos will inform the work of accessibility and
HCI researchers. Students’ perspective on factors that lead to chal-
lenges in ASL video comprehension, as well as the potential of
span-selection video players as a research tool, will be relevant to
ASL linguistics and education researchers. In addition, our fndings
on the benefts of ASL dictionary search using extracted spans of
continuous-signing video informs the work of computer-vision
researchers. Specifcally, our fndings reveal a whole new sign-
recognition task of using a sub-segment of a continuous video as
ASSETS ’22, October 23–26, 2022, Athens, Greece
input for ASL recognition. Traditionally, when ASL recognition
researchers consider processing continuous ASL videos, it is in
the context of machine-translation to English text. Rather than
providing fully automatic translation of a video, our fndings have
revealed interest and benefts for students in attempting to under-
stand a challenging ASL video on their own with some integrated
sign-searching supports. In addition, compared to recognition of
videos of entire utterances, there are unique challenges when the
input is a fragment extracted from a longer video. The selected
span may not perfectly align with sign boundaries, the signs at
the boundary may have been subject to co-articulation efects of
signing beyond the span selection, and there will be less contextual
information for the recognition system to consider.
More broadly, our fndings speak to the literature on users inter-
acting with videos, especially when carefully scrutinizing video,
e.g., when interacting with specialized video-editing software. While
span-selection is less prevalent in video-player systems, several
commercial video-editing systems, e.g., [31–33], allow users to se-
lect a segment of video. Some prior observational research had
investigated users selecting spans while engaged in a video-editing
task [34], and our study has extended span-selection interaction
to the ASL education and video-search context. Other prior work
had investigated educational software tools for students to select a
segment of a spoken-language lecture videos while taking notes
[10, 42] or integrated approaches to editing, sharing, and control-
ling spoken-language educational lecture videos [10, 57]. While
there are diferences between the task of understanding an ASL
video and understanding an educational lecture video in spoken
language, our fndings on the benefts of span-selection interfaces
for constraining the playhead and serving as a basis for integrated
search tools may be relevant to that domain.
6 LIMITATIONS AND FUTURE WORK
Video stimuli in study 2 varied in frame-rate, bit-depth, compres-
sion, and frame scan (i.e., interlaced vs. progressive); these factors
can afect video comprehension [29]. Since the videos were consis-
tent in both prototype conditions in study 2, these factors did not
afect our results, but future research could investigate the experi-
ence of ASL students viewing videos that vary in these dimensions.
While our research has focused on ASL learners and videos, future
work could be extended to learners of other sign languages.
While the Wizard-of-Oz dictionary-search output in our proto-
type simulated a single level of sign-recognition output accuracy,
future research could examine how variations in the output qual-
ity would afect users’ experience to inform computer vision re-
searchers on the level of accuracy required. Analogous research has
been conducted on isolated-sign search-by-video systems [3, 24, 25].
While our work revealed benefts of integrated, span-based
dictionary-search during ASL video comprehension, future work
could further explore the design space of span selection or presen-
tation of search results. Further, while study 2 revealed benefts in
translation quality and lower workload scores, it did not investigate
whether engaging in the task of watching a challenging video led
to measurable learning efects among students; future work could
examine potential short- and long-term benefts of such activity.
Hassan, et al.
Finally, our studies have focused on ASL learners, but the video-
playing and span-searching tool could be useful for other user
groups, e.g., linguists annotating videos of ASL, or experienced ASL
interpreters translating complex technical videos. Future research
could investigate the design of tools for these related tasks.
7 CONCLUSION
Students trying to understand a challenging video is a key part of
comprehension-skill development among ASL learners. However,
there are limitations with existing tools for looking-up signs. Our
interview study with ASL learners revealed users’ frsthand per-
spective on challenges they face during comprehension of such
videos and their existing workarounds. These fndings motivated
and suggested the design of a Wizard-of-Oz video-player prototype
with an integrated dictionary-search, based on span selection. Our
study revealed diferences between this prototype and a baseline
(use of an existing ASL dictionary website) revealed benefts for stu-
dent’s accuracy of translation of ASL videos and subjective rating
of workload. An analysis of how users interacted with the inte-
grated system also revealed diferences in use based on the genre
and linguistic complexity of the video, benefts of the integrated
design, and a dual use of span selection both as input to dictionary
search and for constraining the video playhead.
Our work motivates research and tools to address the needs
of ASL learners engaged in comprehension of challenging videos,
and our fndings inform future designers of ASL systems, com-
puter vision researchers working on sign-matching technologies,
and sign-language educators or linguists. These fndings may also
inform the design of video comprehension tools for other contexts.
ACKNOWLEDGMENTS
This material is based upon work supported by the National Science
Foundation under Grant No. 1763569, 2212303, and 2125362.
REFERENCES
[1] Alikhan Abutalipov, Aigerim Janaliyeva, Medet Mukushev, Antonio Cerone, and
Anara Sandygulova. 2021. Handshape Classifcation in a Reverse Dictionary of
Sign Languages for the Deaf. In From Data to Models and Back, Juliana Bowles,
Giovanna Broccia, and Mirco Nanni (Eds.). Springer International Publishing,
Cham, 217–226.
[2] Chanchal Agrawal and Roshan L Peiris. 2021. I see what you’re saying: A lit-
erature review of eye tracking research in communication of Deaf or Hard
of Hearing Users. In The 23rd International ACM SIGACCESS Conference on
Computers and Accessibility (Virtual Event, USA) (ASSETS ’21). Association
for Computing Machinery, New York, NY, USA, Article 41, 13 pages. https:
//doi.org/10.1145/3441852.3471209
[3] Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. 2019. Efect of automatic
sign recognition performance on the usability of video-based search interfaces for
sign language dictionaries. In The 21st International ACM SIGACCESS Conference
on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association
for Computing Machinery, New York, NY, USA, 56–67. https://doi.org/10.1145/
3308561.3353791
[4] Stavroula Sokoli Athens and Stavroula Sokoli. 2007. Stavroula Sokoli ( Athens )
Learning via Subtitling ( LvS ) : A tool for the creation of foreign language learn-
ing activities based on flm subtitling. In MuTra 2006 – Audiovisual Translation
Scenarios: Conference Proceedings. MuTra, Copenhagen, Denmark, 8 pages.
[5] Vassilis Athitsos, Carol Neidle, Stan Sclarof, Joan Nash, Alexandra Stefan, Ashwin
Thangali, Haijing Wang, and Quan Yuan. 2010. Large lexicon project: American
Sign Language video corpus and sign language indexing/retrieval algorithms.
In Workshop on the Representation and Processing of Sign Languages: Corpora
and Sign Language Technologies (CSLT), Vol. 2. European Language Resources
Association (ELRA), Valletta, Malta, 11–14.
[6] Danielle Bragg, Kyle Rector, and Richard E. Ladner. 2015. A user-powered
American Sign Language dictionary. In Proceedings of the 18th ACM Conference
Support in the Moment
on Computer Supported Cooperative Work and Social Computing (Vancouver, BC,
Canada) (CSCW ’15). Association for Computing Machinery, New York, NY, USA,
1837–1848. https://doi.org/10.1145/2675133.2675226
[7] Fabio Buttussi, Luca Chittaro, and Marco Coppo. 2007. Using web3D technologies
for visualization and search of signs in an international sign language dictionary.
In Proceedings of the Twelfth International Conference on 3D Web Technology
(Perugia, Italy) (Web3D ’07). Association for Computing Machinery, New York,
NY, USA, 61–70. https://doi.org/10.1145/1229390.1229401
[8] Naomi K. Caselli, Zed Sevcikova Sehyr, Ariel M. Cohen-Goldberg, and Karen Em-
morey. 2017. ASL-LEX: A lexical database of American Sign Language. Behavior
Research Methods 49, 2 (01 Apr 2017), 784–801. https://doi.org/10.3758/s13428-
016-0742-0
[9] Sheila Castilho, Stephen Doherty, Federico Gaspari, and Joss Moorkens. 2018.
Approaches to human and machine translation quality assessment. Springer Inter-
national Publishing, Cham, 9–38. https://doi.org/10.1007/978-3-319-91241-7_2
[10] Konstantinos Chorianopoulos and Michail N. Giannakos. 2013. Usability design
for video lectures. In Proceedings of the 11th European Conference on Interactive
TV and Video (Como, Italy) (EuroITV ’13). Association for Computing Machinery,
New York, NY, USA, 163–164. https://doi.org/10.1145/2465958.2465982
[11] Christopher Conly, Zhong Zhang, and Vassilis Athitsos. 2015. An integrated
RGB-D system for looking up the meaning of signs. In Proceedings of the 8th ACM
International Conference on PErvasive Technologies Related to Assistive Environ-
ments (Corfu, Greece) (PETRA ’15). Association for Computing Machinery, New
York, NY, USA, Article 24, 8 pages. https://doi.org/10.1145/2769493.2769534
[12] Eberhard, David M., Gary F. Simons, and Charles D. Fennig (eds.). 2021. Sign
language. https://www.ethnologue.com/subgroups/sign-language
[13] Ralph Elliott, Helen Cooper, John Glauert, Richard Bowden, and François
Lefebvre-Albaret. 2011. Search-by-example in multilingual sign language
databases. In Proceedings of the Second International Workshop on Sign Language
Translation and Avatar Technology (SLTAT). SLTAT, Dundee, Scotland, 8 pages.
[14] Karen Emmorey, Robin Thompson, and Rachael Colvin. 2009. Eye gaze during
comprehension of American Sign Language by native and beginning signers.
Journal of deaf studies and deaf education 14, 2 (2009), 237–243.
[15] National Center for Education Statistics (NCES). 2018. Digest of education
statistics number and percentage distribution of course enrollments in languages
other than English at degree-granting postsecondary institutions, by language
and enrollment level: Selected years, 2002 through 2016. https://nces.ed.gov/
programs/digest/d18/tables/dt18_311.80.asp
[16] Susan M Gass, Jennifer Behney, and Luke Plonsky. 2020. Second language acqui-
sition: An introductory course (5 ed.). Routledge, New York. 774 pages.
[17] David Goldberg, Dennis Looney, and Natalia Lusin. 2015. Enrollments in lan-
guages other than English in United States Institutions of Higher Education, Fall
2013.
[18] Debbie B Golos and Annie M Moses. 2011. How teacher mediation during video
viewing facilitates literacy behaviors. Sign Language Studies 12, 1 (2011), 98–118.
[19] Michael Andrew Grosvald. 2009. Long-distance coarticulation: A production and
perception study of English and American Sign Language. University of California,
Davis, 1 Shields Ave, Davis, CA 95616.
[20] Wyatte C Hall, Leonard L Levin, and Melissa L Anderson. 2017. Language de-
privation syndrome: A possible neurodevelopmental disorder with sociocultural
origins. Social psychiatry and psychiatric epidemiology 52, 6 (2017), 761–776.
[21] Sandra G Hart. 2006. NASA-task Load Index (NASA-TLX); 20 years later. In
Proceedings of the human factors and ergonomics society annual meeting, Vol. 50.
Sage Publications Sage CA, Sage publications, Los Angeles, CA, 904–908.
[22] Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task
Load Index): Results of empirical and theoretical research. In Advances in psy-
chology. Vol. 52. Elsevier, Amsterdam, Netherlands, 139–183.
[23] Saad Hassan. 2022. Designing and experimentally evaluating a video-based
American Sign Language look-up system. In ACM SIGIR Conference on Hu-
man Information Interaction and Retrieval (Regensburg, Germany) (CHIIR ’22).
Association for Computing Machinery, New York, NY, USA, 383–386. https:
//doi.org/10.1145/3498366.3505804
[24] Saad Hassan, Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. 2020.
Efect of ranking and precision of results on users’ satisfaction with search-
by-video sign-language dictionaries. In Sign Language Recognition, Translation
and Production (SLRTP) Workshop-Extended Abstracts, Vol. 4. Computer Vision –
ECCV 2020 Workshops, Virtual, 6 pages.
[25] Saad Hassan, Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. 2021. Efect
of Sign-Recognition Performance on the Usability of Sign-Language Dictionary
Search. ACM Trans. Access. Comput. 14, 4, Article 18 (oct 2021), 33 pages. https:
//doi.org/10.1145/3470650
[26] Saad Hassan, Akhter Al Amin, Alexis Gordon, Sooyeon Lee, and Matt Huener-
fauth. 2022. Design and Evaluation of Hybrid Search for American Sign Language
to English Dictionaries: Making the Most of Imperfect Sign Recognition. In CHI
Conference on Human Factors in Computing Systems (New Orleans, LA, USA)
(CHI ’22). Association for Computing Machinery, New York, NY, USA, Article
195, 13 pages. https://doi.org/10.1145/3491102.3501986
ASSETS ’22, October 23–26, 2022, Athens, Greece
[27] Saad Hassan, Aiza Hasib, Suleman Shahid, Sana Asif, and Arsalan Khan. 2019.
Kahaniyan - Designing for Acquisition of Urdu as a Second Language. In Human-
Computer Interaction – INTERACT 2019, David Lamas, Fernando Loizides, Lennart
Nacke, Helen Petrie, Marco Winckler, and Panayiotis Zaphiris (Eds.). Springer
International Publishing, Cham, 207–216.
[28] Robert J Hofmeister. 2000. A piece of the puzzle: ASL and reading comprehension
in deaf children. Mahwah, N.J. : Lawrence Erlbaum Associates, New Jersey, USA.
143–163 pages.
[29] Simon Hooper, Charles Miller, Susan Rose, and George Veletsianos. 2007. The
efects of digital video quality on learner comprehension in an American Sign
Language assessment environment. Sign Language Studies 8, 1 (2007), 42–58.
[30] iMotions A/S. 2019. iMotions Biometric Research Platform. imotions. https:
//imotions.com/academy/
[31] Adobe Inc. 2008. Adobe Premiere Pro. https://www.adobe.com/products/
premiere.html. [Online; accessed 03-March-2022].
[32] Apple Inc. 2008. Apple Finalcut. http://aiweb.techfak.uni-bielefeld.de/content/
bworld-robot-control-software/. [Online; accessed 03-March-2022].
[33] Apple Inc. 2008. Apple iMovie. https://www.apple.com/imovie/. [Online;
accessed 03-March-2022].
[34] Tero Jokela, Minna Karukka, and Kaj Mäkelä. 2007. Mobile Video Editor: Design
and Evaluation. In Proceedings of the 12th International Conference on Human-
Computer Interaction: Interaction Platforms and Techniques (Beijing, China)
(HCI’07). Springer-Verlag, Berlin, Heidelberg, 344–353.
[35] Jonathan Keane, Diane Brentari, and Jason Riggle. 2012. Coarticulation in ASL
fngerspelling.
[36] Annette Klosa-Kückelhaus and Frank Michaelis. 2022. The Design of Internet
Dictionaries. The Bloomsbury Handbook of Lexicography 1 (2022), 405.
[37] Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, and Debi Prosad Dogra. 2018.
A position and rotation invariant framework for sign language recognition (SLR)
using Kinect. Multimedia Tools and Applications 77, 7 (2018), 8823–8846.
[38] Marlon Kuntze, Debbie Golos, and Charlotte Enns. 2014. Rethinking literacy:
Broadening opportunities for visual learners. Sign Language Studies 14, 2 (2014),
203–224.
[39] The language archive. 2018. ELAN - The Max Planck Institute for Psycholinguis-
tics. https://archive.mpi.nl/tla/elan
[40] J. Lapiak. 2021. Handspeak. https://www.handspeak.com/
[41] Scott K Liddell and Robert E Johnson. 1986. American Sign Language compound
formation processes, lexicalization, and phonological remnants. Natural Language
& Linguistic Theory 4, 4 (1986), 445–513.
[42] Ching (Jean) Liu, Chi-Lan Yang, Joseph Jay Williams, and Hao-Chuan Wang.
2019. NoteStruct: Scafolding Note-Taking While Learning from Online Videos.
In Extended Abstracts of the 2019 CHI Conference on Human Factors in Comput-
ing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing
Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312878
[43] Carolyn McCaskill, Ceil Lucas, Robert Bayley, and Joseph Christopher Hill. 2011.
The hidden treasure of Black ASL: Its history and structure. Gallaudet University
Press Washington, DC, Gallaudet University Press, 800 Florida Avenue, NE,
Washington, DC 20002-3695.
[44] John Milton and Vivying S. Y. Cheng. 2010. A toolkit to assist L2 learners
become independent writers. In Proceedings of the NAACL HLT 2010 Workshop on
Computational Linguistics and Writing: Writing Processes and Authoring Aids (Los
Angeles, California) (CL&amp;W ’10). Association for Computational Linguistics,
USA, 33–41.
[45] Daniel Mitchell. 2021. British Sign Language BSL dictionary. https://www.
signbsl.com/
[46] Ross Mitchell, Travas Young, Bellamie Bachleda, and Michael Karchmer. 2006.
How many people use ASL in the United States? Why estimates need updating.
Sign Language Studies 6 (03 2006). https://doi.org/10.1353/sls.2006.0019
[47] Anshul Mittal, Pradeep Kumar, Partha Pratim Roy, Raman Balasubramanian, and
Bidyut B Chaudhuri. 2019. A modifed LSTM model for continuous sign language
recognition using leap motion. IEEE Sensors Journal 19, 16 (2019), 7056–7063.
[48] J Murray. 2020. World Federation of the deaf. http://wfdeaf.org/our-work/
[49] Tobii Pro Nano. 2014. Tobii Pro Lab. Tobii Technology. https://www.tobiipro.com/
[50] Carol Neidle and Christian Vogler. 2012. A new web interface to facilitate access
to corpora: Development of the ASLLRP data access interface (DAI). In Proc. 5th
Workshop on the Representation and Processing of Sign Languages: Interactions
between Corpus and Lexicon, LREC. Citeseer, OpenBU, Istanbul, Turkey, 8 pages.
https://open.bu.edu/handle/2144/31886
[51] Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera. 2021. Sign language recog-
nition: A deep survey. Expert Systems with Applications 164 (2021), 113794.
https://doi.org/10.1016/j.eswa.2020.113794
[52] Kishore K Reddy and Mubarak Shah. 2013. Recognizing 50 human action cate-
gories of web videos. Machine vision and applications 24, 5 (2013), 971–981.
[53] Jerry Schnepp, Rosalee Wolfe, Gilbert Brionez, Souad Baowidan, Ronan Johnson,
and John McDonald. 2020. Human-Centered design for a sign language learning
application. In Proceedings of the 13th ACM International Conference on PErva-
sive Technologies Related to Assistive Environments (Corfu, Greece) (PETRA ’20).
Association for Computing Machinery, New York, NY, USA, Article 60, 5 pages.
ASSETS ’22, October 23–26, 2022, Athens, Greece
https://doi.org/10.1145/3389189.3398007
[54] Jérémie Segouat. 2009. A study of sign language coarticulation. SIGACCESS
Accessible Computing 1, 93 (Jan 2009), 31–38. https://doi.org/10.1145/1531930.
1531935
[55] Zatorre RJ Shiell MM, Champoux F. 2014. Enhancement of visual motion detection
thresholds in early Deaf people. PloS one 9, 2 (2014), e90498. https://doi.org/10.
1371/journal.pone.0090498
[56] ShuR. 2021. SLintoDictionary. http://slinto.com/us
[57] Namrata Srivastava, Sadia Nawaz, Joshua Newn, Jason Lodge, Eduardo Velloso,
Sarah M. Erfani, Dragan Gasevic, and James Bailey. 2021. Are You with Me?
Measurement of Learners’ Video-Watching Attention with Eye Tracking. In
LAK21: 11th International Learning Analytics and Knowledge Conference (Irvine,
CA, USA) (LAK21). Association for Computing Machinery, New York, NY, USA,
88–98. https://doi.org/10.1145/3448139.3448148
[58] Ted Supalla. 1982. Structure and acquisition of verbs of motion and location in
American Sign Language. Ph.D. Dissertation. University of California, San Diego.
[59] Nazif Can Tamer and Murat Saraçlar. 2020. Improving keyword search perfor-
mance in sign language with hand shape features. In Computer Vision – ECCV
2020 Workshops, Adrien Bartoli and Andrea Fusiello (Eds.). Springer International
Publishing, Cham, 322–333.
Hassan, et al.
[60] Carolina Tannenbaum-Baruchi and Paula Feder-Bubis. 2018. New sign language
new (S): the globalization of sign language in the smartphone era. Disability &
society 33, 2 (2018), 309–312.
[61] Kimberly A. Weaver and Thad Starner. 2011. We need to communicate! Helping
hearing parents of Deaf children learn American Sign Language. In The Pro-
ceedings of the 13th International ACM SIGACCESS Conference on Computers and
Accessibility (Dundee, Scotland, UK) (ASSETS ’11). Association for Computing
Machinery, New York, NY, USA, 91–98. https://doi.org/10.1145/2049536.2049554
[62] Polina Yanovich, Carol Neidle, and Dimitris Metaxas. 2016. Detection of major
ASL sign types in continuous signing for ASL Recognition. In Proceedings of the
Tenth International Conference on Language Resources and Evaluation (LREC’16).
European Language Resources Association (ELRA), Portorož, Slovenia, 3067–
3073. https://www.aclweb.org/anthology/L16-1490
[63] Zahoor Zafrulla, Helene Brashear, Thad Starner, Harley Hamilton, and Peter
Presti. 2011. American Sign Language recognition with the Kinect. In Proceedings
of the 13th International Conference on Multimodal Interfaces (Alicante, Spain)
(ICMI ’11). Association for Computing Machinery, New York, NY, USA, 279–286.
https://doi.org/10.1145/2070481.2070532
[64] Mikhail A. Zagot and Vladimir V. Vozdvizhensky. 2014. Translating Video:
Obstacles and challenges. Procedia - Social and Behavioral Sciences 154 (2014),
268–271. https://doi.org/10.1016/j.sbspro.2014.10.149
 A Dataset of Alt Texts from HCI Publications
Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in Scientific
Sanjana Chintalapati∗
sanjanac@cs.washington.edu jbragg@allenai.org
University of Washington Allen Institute for AI
Seattle, WA, USA Seattle, WA, USA
ABSTRACT
Figures in scientifc publications contain important information
and results, and alt text is needed for blind and low vision readers
to engage with their content. We conduct a study to characterize
the semantic content of alt text in HCI publications based on a
framework introduced by Lundgard and Satyanarayan [30]. Our
study focuses on alt text for graphs, charts, and plots extracted from
HCI and accessibility publications; we focus on these communities
due to the lack of alt text in papers published outside of these
disciplines. We fnd that the capacity of author-written alt text to
fulfll blind and low vision user needs is mixed; for example, only
50% of alt texts in our sample contain information about extrema
or outliers, and only 31% contain information about major trends
or comparisons conveyed by the graph. We release our collected
dataset of author-written alt text, and outline possible ways that
it can be used to develop tools and models to assist future authors
in writing better alt text. Based on our fndings, we also discuss
recommendations that can be acted upon by publishers and authors
to encourage inclusion of more types of semantic content in alt
text.
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility.
KEYWORDS
accessibility, scientifc documents, alt text, dataset
ACM Reference Format:
Sanjana Chintalapati, Jonathan Bragg, and Lucy Lu Wang. 2022. A Dataset
of Alt Texts from HCI Publications: Analyses and Uses Towards Producing
More Descriptive Alt Texts of Data Visualizations in Scientifc Papers. In
The 24th International ACM SIGACCESS Conference on Computers and Acces-
sibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York,
NY, USA, 12 pages. https://doi.org/10.1145/3517428.3544796
∗ Work
conducted during internship at Allen Institute for AI
This work is licensed under a Creative Commons Attribution International
4.0 License.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3544796
Papers
Jonathan Bragg Lucy Lu Wang
lucylw@uw.edu
Allen Institute for AI; University of
Washington
Seattle, WA, USA
1 INTRODUCTION
Alternative text (or “alt text”) describes the content of a visual
graphic or image to those who cannot see them. As such, alt text
is an important component of accessible design. Most scientifc
documents use graphics to communicate information alongside
text; scientifc documents can be especially difcult to make acces-
sible to BLV readers [5], with a large majority of these paper PDFs
lacking usable alt text [47]. Mack et al. [31] conducted a study of
BLV users and what they need from alt text, and found that graphs
and charts are of special importance to these users, as they can be
especially important for conveying results. This, coupled with the
fnding that the vast majority of scientifc fgures lack alt text alto-
gether, suggests that even if the rest of the text in a scientifc paper
were accessible to a BLV reader, that a signifcant portion of the
informational content of these works (fgures) remain inaccessible,
which can negatively impact reader experience.
Though there have been progress and attempts in automatically
generating alt text descriptions of images on web and social me-
dia platforms [17, 35, 48], these methods do not apply as well to
scientifc images. From a machine learning perspective, much of
the advancement in image recognition and scene understanding
in recent years have derived from training neural models on large-
scale labeled image datasets (such as ImageNet [42] and Google
Open Images [25]), datasets that are primarily composed of natu-
ral images, which represent only a small proportion of the types
of images found in scientifc publications. Figures from scientifc
papers run the gamut of image types, including but not limited to
natural images, medical images, diagrams, schematics, a wide array
of graphs and charts, as well as combinations of these types, e.g., a
medical image annotated with a histogram. Correspondingly, many
established image understanding models cannot be directly or easily
adapted for the scientifc domain. Hybrid crowdsourcing solutions
that integrate human experience with machine functionality may
ofer useful alternative solutions [17, 19, 39, 43]. For example, Qian
et al. [39] advocate for a hybrid approach to image captioning where
machines generate caption units and humans perform stitching.
To support these solutions for scientifc alt text, we need to better
understand the current status of alt text content, and develop tools
that can support authors in writing more useful descriptions of
scientifc fgures.
Several prior studies have attempted to quantify the availability
of alt text in scientifc documents [7, 26, 47], though none have in-
vestigated the content of author-written alt texts and whether they
convey adequate information about fgures to blind and low vision
ASSETS ’22, October 23–26, 2022, Athens, Greece
(BLV) readers. In this work, we extract and analyze the content of
author-written alt text from papers published by the accessibility
and HCI communities, and provide recommendations toward en-
couraging the inclusion of more types of descriptive information in
alt text that may be useful to BLV readers. By extracting realistic
author-written alt text, we also provide a useful data resource that
can be used to support authors in writing better alt text and study
how image understanding models and crowd-authoring techniques
can be adapted to more efectively produce fgure alt text in the
scientifc domain.
We process and extract author-written alt text from over 25K pub-
lications in the domains of accessibility and HCI, identifying nearly
3.4K pieces of valid alt text from 899 papers. To assess the type
of information contained in these alt texts, we use the framework
introduced by Lundgard and Satyanarayan [30], which accounts
for four diferent levels of semantic content that may be conveyed
by graphical data visualizations. We assess the semantic content
present in the alt text corresponding to fgures of graphs, charts,
and plots (data visualizations), images that are prevalent in sci-
entifc papers, and for which the alt text content can be suitably
represented using the levels introduced in the Lundgard and Satya-
narayan [30] framework. We fnd that though most alt text contain
basic information about the graph type, axes labels, and what is
plotted, far fewer contain information beyond this. For example,
only 50% of alt text in our sample discuss extrema or outliers in the
data, and only 31% discuss trends or comparisons. The lack of this
type of semantic content in alt text can make it difcult for a BLV
user to understand these kinds of images in the way they desire, as
found by Lundgard and Satyanarayan [30].
Our contributions in this work can therefore be summarized as:
• An assessment of the semantic information conveyed by
author-written alt text of graph and chart fgures extracted
from papers published in venues representing work in ac-
cessibility, HCI, and related areas. We found that levels of
covered content are inadequate, even at accessibility and HCI
conferences, which have alt text requirements and writing
guidelines.
• A dataset of 3386 author-written alt text from HCI publi-
cations, of which 547 have been annotated with semantic
levels.1 The methods used to construct this dataset can be
extended to study trends in scientifc fgure alt text more
broadly, and our dataset can be used develop tools and mod-
els to support alt text authoring. For example, we experiment
with training a classifer that identifes semantic levels in
text, which could be used to provide feedback to authors as
they are writing alt text. We discuss and explore additional
opportunities in Section 5.
2 RELATED WORK
We briefy discuss related work on how to write useful scientifc
alt text (Section 2.1), resources and methods for scientifc fgure
understanding (Section 2.3), resources and methods for automatic
alt text generation (Section 2.4), and other methods for improving
fgure accessibility (Section 2.5).
1 The dataset and annotations are available at https://github.com/allenai/hci-alt-texts.
S. Chintalapati, J. Bragg, and L.L. Wang
2.1 Guidelines for writing scientifc alt text
The Web Content Accessibility Guidelines (WCAG) [9, 13] contain
guidance on when alt text should be provided and the suggested
content for the alt text. The National Center for Accessible Media
(NCAM) has published guidelines including high-level recommen-
dations for writing alt text for graphs, suggesting that a complete
description should include text describing (i) the layout of the graph,
(ii) the location of variables on the graph, and (iii) for static graphs,
the overall trends presented, and for dynamic graphs, summary
information such as the range of the axes.2 The Benetech Diagram
Center also provides image description guidelines with the goal of
making it easier, cheaper, and faster to create and use accessible dig-
ital images.3 The referenced documentation includes both general
best practices concerning aspects such as style and language that
apply to every type of image, along with specifc considerations
for bar graphs, pie graphs, line graphs, and scatter plots such as
listing the numbers in a pie graph from smallest to largest, and
focusing on the change of concentration in scatter plots. Several
academic publishers have also provided research-based guidelines
for improving the accessibility of digital media. For example, the
Association for Computing Machinary (ACM) strongly encourages
authors to provide alt text for images and charts, and includes in-
structions for authors such as not duplicating the caption text and
providing keywords.4
Lundgard and Satyanarayan [30] introduced a four-level con-
ceptual model describing the semantic content of information that
should be present in alt text descriptions of scientifc data visualiza-
tions. Level 1 includes construction details such as the type of fgure
(e.g., bar plot or line plot), and labels of the axes. Level 2 includes
statistics about the fgure data, such as extremes and correlations.
Level 3 includes larger takeaways, such as trends and patterns in
the data. Level 4 includes domain-specifc insights and societal con-
text for the fgure data. The authors conducted studies including
BLV users, and found that these users gained the most information
from textual descriptions conveying information from semantic
levels 1–3 [30]. This fnding corresponds to the recommendations
of alt text content made by the NCAM. Though this framework is
not a guideline document per se, we adopt it for this study in order
to evaluate the quality of author-written alt text for graphs and
charts found in scientifc publications. It has been validated through
studies with BLV participants, and we are unaware of alternative
frameworks.
Towards alt text preferences, Bennett et al. [4] conducted a study
on best practices for describing race, gender, and disability sta-
tus in alt text, and found that people in photographs preferred to
be described with the language that they use to talk about them-
selves, and that descriptions of concrete visual details were more
appropriate than language around identities. Though this does not
apply directly to scientifc fgures, alt text written by someone
other than the authors of a paper may want to consider how the
authors intended for the fgure to be understood as well as the
language used by the authors in the rest of the publication. Af-
ter reviewing BLV people’s experiences with digital image types
2 https://www.wgbh.org/foundation/ncam/guidelines/accessible-digital-media-
guidelines
3 http://diagramcenter.org/making-images-accessible.html
4 https://authors.acm.org/proceedings/production-information/describing-fgures
A Dataset of Alt Texts from HCI Publications
such as news articles and employment websites, Stangl et al. [46]
found that a one-size-fts-all approach for image descriptions is
not ideal. Similarly, through interviews with screen reader users,
Mack et al. [31] found that diferent BLV users had varying pref-
erences about the level of detail that they found to be most help-
ful in alt text, although most users concluded that both brevity
and the availability of detailed information were desirable traits.
Such fndings should be kept in mind when authoring efective
alt text.
2.2 Interfaces for authoring alt text
Morash et al. [33] developed interfaces to guide novice web workers
in writing descriptions of scientifc images. The authors queried
workers for information about select image attributes based on the
NCAM guidelines, on attributes such as image type, title, and units
shown. They found that the templated query method was preferred
by the workers and produced better image descriptions.
Mack et al. [31] built a prototype interface for authoring alt text,
and measured the quality of alt text on a four-point scale based on
three interface variations: the current PowerPoint interface; a free-
form interface, where suggestions were presented as a bulleted list;
and a template interface, where each prompt was listed separately
and included a text box to respond to that prompt. Participants who
use screen readers were asked to rank the quality of alt text written
under the PowerPoint, free-form, and template interfaces, they
found that, in general, the free-form interface encouraged authors
to write alt text that is more closely aligned with the preferences
of screen reader users.
2.3 Automated methods for scientifc fgure
understanding
Scientifc fgure understanding tasks such as fgure classifcation,
visual question-answering (VQA), or image captioning have re-
ceived signifcant attention from the AI community in recent years
[10, 11, 20, 21, 23, 24, 27, 32, 34, 38, 40, 45]. In Table 1, we describe a
number of datasets that have been introduced to train models and
evaluate their performance on these tasks.
Datasets introduced for scientifc image classifcation include
FigureSeer [45], DocFigure [21], and SlideImages [34]. All three
datasets include realistic images extracted from scientifc papers,
along with labels to classes such as graph, medical image, or natural
image. These datasets have been used to train models that can detect
fgure type, which is an essential piece of information that should
be available in alt text. However, fgure type is only one of many
pieces of information that BLV users may need to understand the
content of an image, and therefore these datasets are of limited use
in the alt text generation setting.
Towards more detailed fgure understanding, datasets such as
FigureQA [24], DVQA [23], and PlotQA [32] have been introduced.
These datasets are made up of synthetically generated graphs and
charts, along with associated questions and answers about the graph
contents. For example, questions may be related to the graph title,
axes, the � and �-axes values associated with specifc data series,
or the names of each data series. In PlotQA [23], many questions
also go beyond the structure of the plot and may require data re-
trieval or additional reasoning (e.g. What is the average diference
ASSETS ’22, October 23–26, 2022, Athens, Greece
between two data series?) Models trained on these datasets have
shown improving performance [24, 27], though because the plots
in these datasets are generated synthetically, they may not transfer
well to graphs found in actual scientifc publications, which are
signifcantly more noisy, diverse, and variable than those found in
these datasets. Also, though fgure understanding through VQA is
related to the task of producing alt text, the task of VQA itself does
not produce a coherent textual description of the fgure, which is
the desired outcome for alt text. The FigCAP project [10, 11] at-
tempts to bridge this divide by deriving fgure descriptive text from
the questions and answers of the FigureQA [24] dataset; though
FigCAP refers to itself as a fgure captioning dataset, the “captions”
they provide are more similar to the notion of alt text, including
descriptions about the fgure structure, axes labels, data values,
etc.
Recent work has aimed to automatically generate captions for
both natural images [12, 44] and fgures [20, 38]; however, cap-
tion generation and alt text generation are not the same task, and
could be said to have difering goals. Captions are intended to
be consumed by all readers, and contain information that com-
plements the content of the image; while alt text is meant to ex-
plain the informational content of the image for users who cannot
see it. SciCap [20] introduces a large dataset of graph fgures and
captions derived from arXiv; captions are those originally writ-
ten by authors, and are post-processed to remove tokens corre-
sponding to numbers and equations. The ImageClefMed Caption-
ing task released a relatively much smaller dataset focused on
captioning of medical images derived from scientifc publications
[38].
As far as we know, there are no datasets available for studying
scientifc fgure alt text generation with realistic, author-written
alt text. Though FigCAP [10, 11] and FigJAM [40] explore text
generation in the alt text setting, the images and target texts used
are synthetic (derived from FigureQA [24]) and not representative
of the analogous task in a realistic setting.
2.4 Automated methods for alt text generation
Ofce 365 generates alt text for any image or fgure pasted into
Microsoft PowerPoint.5 Though easy to use, the feature shows
limited performance on scientifc fgures, usually only describing
the type of the fgure. For example, for the fgures in Table 2, the
corresponding alt texts generated by Ofce 365 are, respectively:
“Chart, bar chart,” “Chart, line chart,” and “Chart.” Though Ofce
365 usually identifes the correct type of chart, no other information
about the fgure content is generated, and the resulting alt text is
of limited use to the reader.
Qian et al. [40] created synthetic datasets for fgure alt text gen-
eration by adapting the FigureQA [24] and DVQA [23] datasets
for fgure VQA. Figures in both datasets are synthetic (not from
scientifc papers), and alt text units are derived from the data and
information used to construct each fgure. The authors then trained
the FigJAM model, which generates alt text descriptions based on
the multimodal inputs of the raw fgure image and fgure metadata
[40]. Though performance on synthetic data was shown to be good,
5 https://www.microsoft.com/en-us/microsoft-365/powerpoint
ASSETS ’22, October 23–26, 2022, Athens, Greece S. Chintalapati, J. Bragg, and L.L. Wang

Dataset Domain Realistic/ Task Size of dataset

Synthetic
FigureSEER [45] Scientifc fgures Realistic Image classifcation 60K fgures
(6 classes)
DocFigure [21] Scientifc fgures Realistic Image classifcation 33K images
(28 classes)
SlideImages [34] Educational Realistic Image classifcation 3K images
illustrations (8 classes)
FigureQA [24] Scientifc graphs Synthetic VQA 100K images
and charts
DVQA [23] Bar charts Synthetic VQA 300K images
PlotQA [32] Plots Synthetic plots; VQA 224K images
Realistic data
SciCap [20] Scientifc fgures Realistic Image captioning 416K fgures
ImageClefMed Caption [38] Medical images Realistic Image captioning 5K images
FigCAP [10, 11] Bar charts, pie Synthetic alt text generation 110K fgures
charts, line plots
Table 1: Datasets for scientifc fgure understanding

the model may not generalize to realistic fgures found in scien-
tifc papers, which exhibit signifcantly more variability than those
evaluated by Qian et al. [40].6
Researchers have developed tools for generating alt text infor-
mation for images on non-scientifc social platforms. Gleason et
al. [17] addressed the accessibility barrier on Twitter by creating a
browser extension which adds alt text to Twitter using six methods,
such as reverse image searching and automatic image captioning.
However, this project focused on Twitter images, which difer from
realistic scientifc fgures. Additionally, Wu et al. [48] deployed
an automatic alt text system to identify faces, objects, and themes
for photos on Facebook in order to make them more accessible to
screen reader users. The domain of natural images on social media
again signifcantly difers from our domain of scientifc graphs and
charts.
2.5 Other methods for improving fgure
accessibility
Researchers have also developed alternatives to alt text for improv-
ing fgure accessibility. ChartSense [22] and PlotDigitizer7 are chart
data extraction methods which convert chart images into structured
data tables. However, charts of the same type are too diverse in
style to apply a single extraction algorithm, algorithms have trouble
interpreting overlapped visual entities, and there is no text-region
detection algorithm for chart images with sufcient accuracy [22].
Auditory graphs, tactile graphs, and various multimedia approaches
have also been introduced to improve the accessibility of graphs
and charts [15, 36].
Crowdsourcing, in which a group of non-experts completes a
task that is currently infeasible to accomplish via automated meth-
ods, has also been used to good efect for making images more
6 The
authors do not release their dataset or pretrained models, so we are unable to
perform a comparative analysis on realistic fgures derived from scientifc papers.
7 http://plotdigitizer.sourceforge.net/
accessible. Salisbury et al. [43] explored a novel approach in which
crowdworkers were paired together to create image descriptions.
Crowdworkers were asked questions to extract desired details about
an image such as the location of the image and what emotions the
image evoked. Platforms such as VizWiz [6] or Be My Eyes8 con-
nect BLV users to sighted crowdworkers and volunteers via an app
for assistance with questions and daily tasks. Building upon the suc-
cesses of the VizWiz Grand Challenge [19], a similar solution could
be created to connect BLV researchers with volunteers that can pro-
vide suitable descriptions or answers to questions about scientifc
fgures. Mack et al. [31] built user interfaces for both authoring alt
text and providing feedback on automatic alt text, which could be
theoretically connected to a crowdsourcing platform and adapted
to collect alt text for scientifc fgures based on user demand.
Lastly, researchers have developed methods for surfacing alt text
that do not involve manual efort from humans or automated alt
text generation. Guinness et al. [18] found that many images appear
in several places across the web, and used this insight to develop
Caption Crawler. This system uses reverse image search to fnd alt
text from similar images available on the web and surfaces these
alternate descriptions to the user. This method works quite well in
the domain of natural images, where many similar images of places
or things can be found on the open internet.
3 METHODS
We extract and analyze the presence and content of alt text for
graphs, charts, and plots in scientifc papers. These fgures are
typically used to visualize data and results, are of special importance
to BLV readers [31], and are the types of images for which Lundgard
and Satyanarayan [30] have defned semantic levels. The Lundgard
and Satyanarayan [30] framework describes four levels of semantic
content, organized by increasing complexity:
8 https://www.bemyeyes.com/
A Dataset of Alt Texts from HCI Publications
Level 1: enumerating visualization construction details (e.g.,
type, marks, and encodings)
Level 2: identifying statistical concepts and relations (e.g.,
extremes and correlations)
Level 3: characterizing perceptual and cognitive phenomena
(e.g., trends and patterns)
Level 4: articulating domain-specifc insights or societal con-
text.
Lundgard and Satyanarayan [30] found that levels 1–3 were re-
ported most useful by blind and low vision readers. Level 4, which
incorporates signifcantly more subjective information, was found
to be less essential to understanding; in fact, a majority of blind
readers in their study (63%, n=19) believed that fgure alt text should
not contain level 4 content.
3.1 Sampling papers and extracting
author-written alt text
Our goal is to construct a dataset of author-written alt text by au-
tomatically sampling and extracting alt text from papers. We start
with the set of papers from two conferences: ACM CHI Confer-
ence on Human Factors in Computing Systems (CHI) and ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS)
published in the years 2010-2020. We identify these papers using
the ACM’s reported DOIs, and link these to PDFs in the Seman-
tic Scholar corpus [1]. This yields 5218 PDFs. We then iteratively
extend our paper sample to include all papers written by authors
who have published in CHI and/or ASSETS. We prioritize authors
based on the frequency of their publications in CHI and ASSETS,
and retrieve these authors’ other publications using the Semantic
Scholar API.9 From these queries, we assemble a further sample of
20000 paper PDFs to process and extract alt text.
CHI and ASSETS are premiere conferences on human-computer
interaction and accessible computing, and both conferences have a
history of soliciting accessible paper submissions10 and requiring
authors to provide alt text for fgures. We therefore expect a much
higher percentage of papers published at these venues to contain
valid alt text. Similarly, our rationale for sampling additional papers
published at other venues by authors who have published at CHI
and ASSETS is based on our hypothesis that such authors are more
likely to write alt text in general, even if another publishing venue
may not require alt text for submission. Initially, we intended to
extract alt text from a stratifed random sample of papers repre-
senting all felds of study, but a pilot attempt showed that random
sampling would yield virtually no alt text. We arrived at this con-
clusion after processing a stratifed sample of 5000 PDFs from 2010
to 2020 and extracting only a single piece of descriptive alt text.
Given the time and expense of processing PDFs to extract alt text
at scale, we decided that the stratifed random sample strategy was
untenable.
For each paper in our sample, we followed a three step process
to extract alt text. First, we processed the PDF using Adobe Acrobat
Pro11 to convert the PDF to HTML using the Adobe Acrobat Pro
9 https://api.semanticscholar.org/
10 CHI directs authors to the SIGCHI guidelines for an accessible submission:
https://sigchi.org/conferences/author-resources/accessibility-guide/ and ASSETS pro-
vides these instructions: https://assets21.sigaccess.org/creating_accessible_pdfs.html
11 https://www.adobe.com/acrobat/acrobat-pro.html
ASSETS ’22, October 23–26, 2022, Athens, Greece
Action Wizard. We used Adobe Acrobat Pro rather than an alter-
nate programmatic approach because we were unable to identify
a PDF processing library capable of extracting alt text.12 Second,
we extracted alt text from the converted HTML document. Third,
we fltered the extracted alt text according to a set of heuristics.
Our pilot attempts at extracting alt text revealed that most of the
extracted alt text consisted of uninformative short descriptions like
“Image” or fle paths like “C:\\path_to\fgure1.jpeg,” so we defned a
set of fltering criteria to remove these. We also fltered out alt text
shorter than 80 characters since many shorter alt text fall under the
category of uninformative short descriptions. To determine this 80-
character threshold, we analyzed a sample of 100 extracted alt text
to fnd a limit that maximizes recall without sacrifcing precision.
We refer to the alt text that pass these fltering criteria as “valid alt
text.”
For all valid alt texts, we identify those that are likely to corre-
spond to graphs, charts, and plots. We iteratively defned another
set of heuristics: a list of words and phrases that correspond to
graphs and charts (e.g., “graph”, “chart”, “error bar”; the full list of
terms are provided in Appendix A). For each fgure, we search for
token matches in the alt text and image caption against this list of
terms, and retain only fgures and alt text matching at least one
term. The alt text of the matching fgures are then annotated with
the Lundgard and Satyanarayan [30] semantic levels.
3.2 Annotation of alt text semantic levels
To study what types of content are present in author-written alt text,
we ask annotators to assess the semantic content levels present
in each sentence of each piece of alt text. We split alt text into
sentences using the scispaCy NLP library [37]. Annotators are
shown these sentences along with the corresponding fgure caption
and a link to view the fgure.
Six label options were provided for each sentence:
• Level 1: Figure logistics
• Level 2: Statistical properties and comparisons
• Level 3: Complex trends and patterns in data
• Level 4: Domain-specifc insights or societal concepts to help
explain Level 3 trends
• This alt text contains no levels of content
• This image is not a graph or chart
If a fgure is not a graph or chart, the annotator is instructed to se-
lect the last option. Otherwise, up to three semantic levels could be
selected for each sentence. These label options were adapted from
the level descriptions given by Lundgard and Satyanarayan [30]
and shortened and simplifed to make them easier for annotators to
understand. Annotators were given an additional instruction docu-
ment that provides more detail on each of the label options, along
with examples of sentences corresponding to each label option.
The annotators were instructed to exhaustively label each sen-
tence with all of the levels it contains. We recruited two annotators
12We experiment with several other widely available PDF libraries and con-
version tools, including PDFTOHTML (http://pdftohtml.sourceforge.net/),
PDFMiner (https://github.com/euske/pdfminer), pdf2xml
(https://sourceforge.net/projects/pdf2xml/) etc., and found that none of these
tools allowed access to the embedded alt text. Scaling in Adobe Acrobat Pro was the
only working solution we were able to identify.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 1: The proportion of PDFs in our analyzed sample which contain valid alt text over time, with 95% confdence intervals
computed through bootstrap resampling.
through the UpWork platform.13 The annotators had undergraduate-
level education in math, statistics, and materials science, and had
previous experience reading graphs, plots, and other scientifc fg-
ures. The alt text retained after the fltering steps described in the
previous section were split into sets of 100 alt texts each for an-
notation. Two individuals annotated an initial sample of 100 alt
texts (298 sentences) to refne the task and ensure high annota-
tor agreement. The inter-annotator agreement computed over this
sample was 87.6%, with � = 0.80, indicating very good agreement.
We further clarifed the instructions following a discussion of dis-
agreements. Given the high agreement level, a single annotator
annotated the remaining alt text. For the fnal analysis, the frst
annotator’s annotations are used for the sample that was doubly
annotated. Examples of extracted alt text and the corresponding
annotated semantic levels are provided in Table 2.
4 RESULTS
We analyze the semantic content of the extracted alt text, and
attempt to answer the following research questions:
• RQ1: What is the distribution of semantic content in
author-written alt text?
We want to determine the proportion of alt text contain-
ing level 1, 2, 3, and 4 content. Of these, what proportion
contains levels 1–3 content, which satisfes most BLV user
needs? Correspondingly, which semantic levels are most
often missing?
• RQ2: How does the distribution of semantic content in
alt text change over time? We want to determine whether
the presence of levels 1-3 semantic content is increasing
over time, and by how much. We expect that with improve-
ments in alt text awareness and workfows over time, that
the amount of content available in alt text should correspond-
ingly increase.
• RQ3: How does length of alt text correlate with seman-
tic levels?
Mack et al. [31] fnd that there is tension between detail
and brevity in alt text. The ideal alt text may vary based on
user needs, but should balance length and completeness of
13 https://www.upwork.com/
S. Chintalapati, J. Bragg, and L.L. Wang
semantic content. We determine the relationship between
length and presence of semantic levels using our data.
4.1 Descriptive statistics
We process 25218 total paper PDFs to extract alt text. Of these, 19500
(77.3%) are successfully converted to HTML by Adobe Acrobat Pro.
Only 897 (4.6%) of these converted documents contain at least one
piece of valid alt text. Around 2048 pieces of alt text corresponded
to fle paths and 2545 alt texts did not meet our length criteria, and
were fltered out; all other alt texts that were fltered out did not
have any content besides “image.” After this fltering, the 897 papers
contain 3386 valid author-written alt texts. Using our keyword
heuristics, we determine that 1085 of these alt texts are likely to
correspond to graphs or charts. Based on a cursory examination,
the alt texts that were fltered out using these keyword heuristics
consist primarily of natural images and diagrams.
We ask annotators to assess the semantic levels present in each
sentence of the remaining 1085 alt text. Of these, 547 fgure alt texts
(consisting of 2127 sentences) are labeled as belonging to graphs,
charts, and plots by our annotators, indicating that our keyword
heuristics have approximately 50% precision. Alt texts of fgures
corresponding to these data visualizations are further annotated
for semantic content. Several examples of author-written alt text
and semantic level annotations are given in Table 2. In Table 3, we
provide the numbers of PDFs processed and alt text retained after
each fltering step.
Figure 1 shows the proportion of PDFs in our sample that contain
any valid alt text, and how this proportion changes in our sample
over the last decade. There is a slight increase in alt text coverage
in 2014; this is the same year that CHI specifed that alt text is
required in submissions. We also observe that the proportion of
papers with valid alt text has improved over time, especially in the
past few years, although the overall proportion is still quite low
(below 15%). We note that Figure 1 does not indicate the actual
proportions of papers from these years that have valid alt text (we
do not process the version of record for all papers in our sample
due to the difculty in ascertaining these versions and copyright
challenges in obtaining them; we also cannot guarantee that the
pipeline we use succeeds in extracting alt text for all of the papers
A Dataset of Alt Texts from HCI Publications ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure & Source Author-written alt text with annotations

“Figure 4: Materials participants reported wanting to read in ASL
text (L1). This fgure presents a bar chart, with separate bars for
DHH (light blue) and hearing (dark blue) populations (L1). Y-axis is
% participants, ranging from 0-70 (L1). X-axis is Material Desired in
ASL Text (L1). sorted by DHH popularity (most popular frst):
Website content, Printed content, Email, Texts/SMS, Video
captions, Other, None (L1).”

Figure 4 reproduced from Bragg et al. [8]

“A line chart showing the average time it took participants on all

Figure 4 reproduced from Baker et al. [2]
tasks (L1). The y-axis of the chart is time in seconds (ranges from 0
to 60), the x-axis of the chart is session number (ranges from 1 to 6)
(L1). There is a line for the three modes: Silent, Verbal and Finger
Pointing (L1). They all appear to be going down, but there is a big
spike in the Verbal mode line at session 4 (L3). In general, the
Finger Pointing mode is the highest (takes the most time), the
Silent mode is next and the Verbal takes the least amount of time,
although in the fourth and sixth sessions, the Verbal line is above
the Silent one (L3). There is a dot corresponding to the Braille
mode at Session 6, it is between the Verbal and Silent modes (L1).”

“Plot of mean proportion of image pixels diferentiable

(Figure 7 reproduced from Reinecke et al. [41])
(independent) for 0% - 100% of the population (dependent) for
websites and infographics (L1). Increasing from 0% of the
population, both plots start at 100% diferentiable and gradually fall
to 80% diferentiable at 75% of the population (L2, L3). Plots begin
to diverge here as they both fall of more quickly until a
discontinuity plateau is reached at 88% of the population (websites
= 60% diferentiable, infographics = 50% diferentiable) (L2, L3).
Plateau gradually declines to 99% of population (websites = 55%
diferentiable, infographics = 45% diferentiable), and then both
plots fall to 0% diferentiable for 100% of the population (L2, L3).”

Table 2: Example fgures and author-written alt text with annotated semantic levels added to the end of each sentence, with the
prefx L, in parentheses and colored (blue).

we process). Rather, the fgure describes our success rate in creating
this dataset, and provides some sense of the trend towards more
valid alt text in recent years.
4.2 Analysis of alt text semantic content
Distribution of semantic content in alt text (RQ1). In Figure 2 (left),
we present the maximum level of content found in each fgure alt
text in our sample. We observe a fairly evenly distribution between
max levels 1, 2, and 3. Recall that in Lundgard and Satyanarayan
[30], BLV users found levels 1–3 content to be most useful. Although
some fgures in our sample have alt text with level 3 content, over
two-thirds of the fgure alt texts that we examined do not contain
level 3 content (level 3 content describes trends and patterns). At
the same time, over one-third of the fgure alt texts lack both level
2 and level 3 content, meaning that there is no information on
extrema and outliers in addition to trends and patterns. The lack of
such content can make it more difcult for BLV users to acquire
the information that they need from these graphs and charts.
In Figure 2 (right), we show the total number of levels of content
present in all fgure alt text we analyzed. We fnd that the vast
ASSETS ’22, October 23–26, 2022, Athens, Greece
Processing step
Total PDFs processed
PDFs successfully converted to HTML
Papers with at least one valid alt text
Pieces of valid alt text
Number of fgure alt text annotated (heuristically fltered to likely be graphs or charts)
Number of annotated fgure alt text that correspond to graphs or charts
Table 3: The numbers of papers and fgure alt text that remain after each fltering step.
Figure 2: Distribution of the maximum level (left) and total number of levels (right) of semantic content found in the sample of
annotated author-written alt text.
majority of alt text in our sample only contain one or two levels. This
means that though the maximum levels of content are somewhat
evenly distributed between levels 1, 2, and 3, only one or two of
these levels are present in most fgure alt texts.
Semantic content in alt text over time (RQ2). Figure 3 shows the
proportion of alt text from each year that contain text of each of
the semantic levels. Though the vast majority of alt text contain
level 1 information, a much lower proportion contain level 2 and 3
information. Over time, there have not been signifcant changes to
the proportion of alt text that contains level 2 and 3 information.
Relationship between alt text length and content (RQ3). In Figure 4,
we show the relationship between the length of alt text and the
number of levels of content that it contains. Alt text containing
more levels of information tend to be longer, though alt text of
comparable length can have diferent numbers of levels present
(as indicated by the overlapping boxes across levels). This suggests
that alt text does not have to be longer in order to have more
levels of content. In regards to the balance between brevity and
detail, authors may want to optimize for the amount of information
included in alt text without resorting to writing something overly
long.
5 DATASET USES
Alongside our analysis, we release our dataset of 3386 alt text col-
lected from 897 HCI publications.14 Of these, the contents of 547
alt texts (2127 sentences) are annotated with the semantic levels
introduced by [30]. This dataset can be used to develop tools to
14 Data and analyses are available at https://github.com/allenai/hci-alt-texts.
S. Chintalapati, J. Bragg, and L.L. Wang
Count
25218
19500 (77.3%)
897 (4.6%)
3386
1085
547
support authors in writing better alt text, or to develop authoring
tools or models for producing alt text where none are available. We
discuss some of these potential applications below.
5.1 Improving author-written alt text &
supporting reading interfaces
To improve authoring of alt text, we can develop tools to help
identify potentially missing content in alt text and prompt authors
to add such information during the authoring and editing process.
For graphs and charts, the semantic levels serve as a proxy for
content. Based on the fndings by Lundgard and Satyanarayan [30],
BLV users found levels 1–3 information most benefcial, and a tool
aimed to improve alt text for graphs and charts could assess alt
text quality based on the presence or absence of information at
these levels. In other words, we could train a classifer based on the
semantic level annotations in our dataset to detect which levels are
present and build a tool that prompts authors for the missing levels,
similar to interfaces that have been built in the past for other tasks
like providing peer feedback [16]. Such a classifer could also be
used to automate trend monitoring for alt text content, enabling
expanded and continually updating versions of our analysis with
reduced human labor.
To test out the viability of this theory, we train several classif-
cation models using our collected annotations. We approach this
as a multi-class, multi-label classifcation problem. The classes are
the four semantic levels, and up to four labels can be assigned to
a single piece of text. The input to the model is a single sentence
of alt text, and the output is a distribution of labels over the four
classes. We experiment with a Random Forest classifer using tf-idf
A Dataset of Alt Texts from HCI Publications
Figure 3: Proportion of alt text containing text of each semantic level over time with 95% confdence intervals computed using
bootstrap resampling.
Figure 4: Relationship between length of alt text (character
count) and the number of semantic levels of information
represented. Though length correlates with the number of
levels, there are many longer alt texts that do not necessarily
contain more levels of semantic content.
word representations, as well as classifers based on BERT [14] and
SciBERT [3]. We use 5-fold cross-validation to train and evaluate
all models. The training data is split into folds preserving each alt
text as a unit (547 instances), while text and labels are provided to
the model at the sentence level (2127 sentences). We report mean
accuracy and F1 over all fve folds for each model in Table 4.
Baseline performance of these models suggests that they are rea-
sonably good at identifying the correct semantic levels present in an
ASSETS ’22, October 23–26, 2022, Athens, Greece
Model Accuracy Micro-F1
Random Forest (tf-idf ) 0.689 (0.021) 0.782 (0.011)
BERT-base 0.912 (0.016) 0.824 (0.032)
SciBERT-base 0.910 (0.010) 0.819 (0.021)
Table 4: Model performance averaged over fve folds, shown
with standard deviations in parentheses.
alt text sentence. Performance could be improved further through
additional model tuning or data annotation. The outputs of these
models can feasibly be used to provide feedback to authors who
are writing alt text, to indicate when content of certain semantic
levels may be lacking.
In addition to helping to improve author-written alt text, such a
classifer could be used to support improved reading experiences
of existing alt text. For example, semantic level classifcation could
enable users to make informed decisions about whether and how
to read alt text by fltering for semantic levels that may be more
relevant to their needs. Morash et al. [33] described similar types
of personalized reading experiences, which could be enabled for alt
text that has been written using standardized templates. We leave
the implementation of these authoring and reading interfaces, as
well as explorations on user interface design, to future work.
5.2 Training and evaluating NLP models for alt
text generation
Recent developments in multimodal image-language pretraining
[28, 29, 49] hold promise towards the eventual automatability of
scientifc fgure alt text generation. Currently, fgure alt text genera-
tion is hampered by the lack of realistic training and evaluation data.
Though the size of our dataset is small and insufcient for training
neural models, it may still be useful to help scale the collection of
training data. Alt text from this dataset can be used to provide high-
quality examples to annotators. Additionally, a classifer trained
to predict alt text semantic levels such as the one introduced in
Section 5.1 could be used to ofer feedback to annotators during
ASSETS ’22, October 23–26, 2022, Athens, Greece
the annotation process, e.g., by indicating when information of a
certain content level is missing in the annotation. Instruction spe-
cifc to the missing level could be provided to the annotator to elicit
further description information, as in the techniques employed by
Morash et al. [33]. Several works propose a hybrid approach that
combines machine learning model output with human writers to
create better image descriptions [19, 31, 39]. Our dataset and classi-
fers could be used in collaboration with human writers to produce
more descriptive alt text.
The alt texts in our dataset could also serve as part of a viable
evaluation corpus. Though not all alt text in the dataset contain
semantic level annotations, the texts themselves are written by the
original paper authors, and are therefore more likely to be faithful
to the original intent and content of the paper. We release all 3386
valid alt text extracted from our sample of papers, which includes alt
text in addition to the 547 alt texts belonging to data visualizations
which we annotate for semantic content. These alt texts can be
used to assess pieces of information that authors found important
enough to include in the image description. The output of a general-
purpose scientifc alt text generation model can be evaluated against
the information contained in the original author-written alt text
associated with these fgures.
6 DISCUSSION & CONCLUSION
In regards to scientifc alt text, availability is still the primary issue.
However, in the alt text we were able to extract from HCI publica-
tions, we observed that for papers where authors have taken the
time to write alt text, the content and level of detail available in
these alt text is also worth considering. What does it mean to write
useful alt text? What does it mean to include enough detail such
that the content of an image can be understood by a BLV user? For
graphs and charts, we propose that authors leverage the framework
introduced in Lundgard and Satyanarayan [30] to ensure that some
basic semantic information is provided, enough such that BLV users
can understand the structure of the graph, its extrema and outliers,
as well as the obvious trends and comparisons that can be made. In
our current analysis, we fnd that many author-written alt text are
not yet meeting these thresholds.
We recognize the limitations of our techniques. The alt text and
fgures included in our dataset make up a biased sample, containing
only papers from CHI and ASSETS and from the authors publishing
in these venues. They are not representative of fgures in all schol-
arly documents. Though we would have liked to construct a more
representative sample of scientifc fgures, the overwhelming lack
of fgure alt text in scholarly publications prevents us from doing
so. As more authors from other disciplines begin including alt text
and the barriers to adding alt text to scientifc fgures decreases, we
hope that it will become easier to create such a dataset.
Additionally, our analysis and annotations are limited to fgures
containing data visualizations (graphs, charts, and plots), which
are only one of many types of images present in scientifc publi-
cations. Our results on the suitability or missingness of semantic
content in author-written alt text cannot generalize beyond these
image types. Going beyond graphs, frameworks other than that
introduced in Lundgard and Satyanarayan [30] may be needed to
capture the availability and distribution of informational content.
S. Chintalapati, J. Bragg, and L.L. Wang
We emphasize that the lack of certain semantic levels in alt text is
not equivalent to an assessment about the alt text’s quality. Difer-
ent users may want diferent information out of alt text, and there
is no one-size-fts-all solution [31]. Rather, we use the framework
as a proxy for content availability, which can be used to elicit dif-
ferent kinds of descriptive information that may be missing in an
author’s original alt text. Finally, in Lundgard and Satyanarayan
[30], the authors assessed whether the levels were useful for BLV
users, but not whether all of levels 1–3 were necessary for an alt text
to be considered complete. Further study is needed to determine
the appropriate balance of depth of information, completeness, and
brevity in relation to the usefulness of alt text.
We propose in Section 5 several uses of this dataset towards
improving author and publisher workfows around writing fgure
alt text. A classifer trained to detect semantic levels can be used to
provide feedback to authors who are writing alt text, and can be
used to elicit alt text containing more levels of information. The
dataset can also be used to help develop machine learning models
that can generate better alt text based on the image itself. We believe
that coupling machine learning models with crowdsourced image
descriptions may provide a reasonable solution to problems around
alt text availability, and we plan to explore such solutions in future
work.
Policy clearly matters. We faced signifcant challenges when
attempting to extract alt text from a broad swathe of scientifc pub-
lications, and had to limit ourselves ultimately to HCI publishing
venues such as ASSETS and CHI. There is no doubt in our minds
that the alt text we were able to extract are only there because of
the eforts of the accessibility and HCI research community and
the importance that members have placed on digital accessibil-
ity. Signifcant work remains to encourage researchers outside of
these communities to participate in making their work accessible.
Within the community, there are also ways that we can improve
fgure accessibility, by providing information that are described
by BLV users as being more relevant or more important towards
interpreting these images.
ACKNOWLEDGMENTS
We thank Donal Fitzpatrick, Doug Downey, the reviewers, and
members of the Semantic Scholar research team for their valuable
feedback. We thank Bailey Kuehl and Ihsan Allah Rakha for their
assistance in the annotation process.
REFERENCES
[1] Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Craw-
ford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu A.
Ha, Rodney Michael Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler C. Murray, Hsu-
Han Ooi, Matthew E. Peters, Joanna L. Power, Sam Skjonsberg, Lucy Lu Wang,
Christopher Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni.
2018. Construction of the Literature Graph in Semantic Scholar. In NAACL.
[2] Catherine M. Baker, Lauren R. Milne, Jefrey Scofeld, Cynthia L. Bennett, and
Richard E. Ladner. 2014. Tactile graphics with a voice: using QR codes to access
text in tactile graphics. Proceedings of the 16th international ACM SIGACCESS
conference on Computers & accessibility (2014).
[3] Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A Pretrained Language
Model for Scientifc Text. In EMNLP.
[4] Cynthia L. Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jefrey P. Bigham,
Anhong Guo, and Alexandra To. 2021. “It’s Complicated”: Negotiating Accessibil-
ity and (Mis)Representation in Image Descriptions of Race, Gender, and Disability.
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
A Dataset of Alt Texts from HCI Publications
(Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York,
NY, USA, Article 375, 19 pages. https://doi.org/10.1145/3411764.3445498
[5] Jefrey P. Bigham, E. Brady, Cole Gleason, Anhong Guo, and D. Shamma. 2016.
An Uninteresting Tour Through Why Our Research Papers Aren’t Accessible.
Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in
Computing Systems (2016).
[6] Jefrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Rob
Miller, Rob Miller, Aubrey Tatarowicz, Brandyn Allen White, Samuel White, and
Tom Yeh. 2010. VizWiz: nearly real-time answers to visual questions. Proceedings
of the 23nd annual ACM symposium on User interface software and technology
(2010).
[7] E. Brady, Y. Zhong, and Jefrey P. Bigham. 2015. Creating accessible PDFs for
conference proceedings. Proceedings of the 12th Web for All Conference (2015).
[8] Danielle Bragg, Raja S. Kushalnagar, and Richard E. Ladner. 2018. Designing an
Animated Character System for American Sign Language. Proceedings of the 20th
International ACM SIGACCESS Conference on Computers and Accessibility (2018).
[9] B. Caldwell, M. Cooper, Loretta Guarino Reid, and G. Vanderheiden. 2008. Web
Content Accessibility Guidelines (WCAG) 2.0.
[10] Chen Chen, Ruiyi Zhang, Sungchul Kim, Scott D. Cohen, Tong Yu, Ryan A. Rossi,
and Razvan C. Bunescu. 2019. Neural caption generation over fgures. Adjunct
Proceedings of the 2019 ACM International Joint Conference on Pervasive and
Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium
on Wearable Computers (2019).
[11] Charles C. Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott D. Cohen, Tong
Yu, Razvan C. Bunescu, and Razvan C. Bunescu. 2019. Figure Captioning with
Reasoning and Sequence-Level Training. ArXiv abs/1906.02850 (2019).
[12] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta,
Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft COCO Captions: Data
Collection and Evaluation Server. ArXiv abs/1504.00325 (2015).
[13] W. Chisholm, G. Vanderheiden, and Ian Jacobs. 2001. Web content accessibility
guidelines 1.0. Interactions 8 (2001), 35–54.
[14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding. In
NAACL.
[15] Christin Engel and Gerhard Weber. 2017. Improve the Accessibility of Tactile
Charts. In INTERACT.
[16] C. Ailie Fraser, Tricia J. Ngoon, Ariel S. Weingarten, Mira Dontcheva, and
Scott Klemmer. 2017. CritiqueKit: A Mixed-Initiative, Real-Time Interface For
Improving Feedback. In Adjunct Publication of the 30th Annual ACM Sympo-
sium on User Interface Software and Technology (Québec City, QC, Canada)
(UIST ’17). Association for Computing Machinery, New York, NY, USA, 7–9.
https://doi.org/10.1145/3131785.3131791
[17] Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington,
Kris M. Kitani, and Jefrey P. Bigham. 2020. Twitter A11y: A Browser Extension
to Make Twitter Images Accessible. Proceedings of the 2020 CHI Conference on
Human Factors in Computing Systems (2020).
[18] Darren Guinness, Edward Cutrell, and Meredith Ringel Morris. 2018. Caption
Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image
Search. Proceedings of the 2018 CHI Conference on Human Factors in Computing
Systems (2018).
[19] Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman,
Jiebo Luo, and Jefrey P. Bigham. 2018. VizWiz Grand Challenge: Answering
Visual Questions from Blind People. 2018 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (2018), 3608–3617.
[20] Ting-Yao Hsu, C. Lee Giles, and Ting-Hao Kenneth Huang. 2021. SciCap: Gener-
ating Captions for Scientifc Figures. In EMNLP.
[21] K. V. Jobin, Ajoy Mondal, and C. V. Jawahar. 2019. DocFigure: A Dataset for Scien-
tifc Document Figure Classifcation. 2019 International Conference on Document
Analysis and Recognition Workshops (ICDARW) 1 (2019), 74–79.
[22] Daekyoung Jung, Wonjae Kim, Hyunjoo Song, Jeongin Hwang, Bongshin Lee,
Bo Hyoung Kim, and Jinwook Seo. 2017. ChartSense: Interactive Data Extraction
from Chart Images. Proceedings of the 2017 CHI Conference on Human Factors in
Computing Systems (2017).
[23] Kushal Kafe, Scott D. Cohen, Brian L. Price, and Christopher Kanan. 2018. DVQA:
Understanding Data Visualizations via Question Answering. 2018 IEEE/CVF
Conference on Computer Vision and Pattern Recognition (2018), 5648–5656.
[24] Samira Ebrahimi Kahou, Adam Atkinson, Vincent Michalski, Ákos Kádár, Adam
Trischler, and Yoshua Bengio. 2018. FigureQA: An Annotated Figure Dataset for
Visual Reasoning. ArXiv abs/1710.07300 (2018).
[25] Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan
Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander
Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020. The Open Images Dataset
V4. International Journal of Computer Vision 128 (2020), 1956–1981.
[26] J. Lazar, E. Churchill, T. Grossman, G. V. D. Veer, Philippe A. Palanque, J. Morris,
and Jennifer Mankof. 2017. Making the feld of computing more inclusive.
Commun. ACM 60 (2017), 50 – 59.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[27] Matan Levy, Rami Ben-Ari, and Dani Lischinski. 2021. Classifcation-Regression
for Chart Comprehension.
[28] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.
2019. VisualBERT: A Simple and Performant Baseline for Vision and Language.
ArXiv abs/1908.03557 (2019).
[29] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining
Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.
In NeurIPS.
[30] Alan Lundgard and Arvind Satyanarayan. 2022. Accessible Visualization via
Natural Language Descriptions: A Four-Level Model of Semantic Content. IEEE
Transactions on Visualization and Computer Graphics 28 (2022), 1073–1083.
[31] Kelly M. Mack, Edward Cutrell, Bongshin Lee, and Meredith Ringel Morris. 2021.
Designing Tools for High-Quality Alt Text Authoring. The 23rd International
ACM SIGACCESS Conference on Computers and Accessibility (2021).
[32] Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. 2020.
PlotQA: Reasoning over Scientifc Plots. 2020 IEEE Winter Conference on Applica-
tions of Computer Vision (WACV) (2020), 1516–1525.
[33] Valerie S. Morash, Yue-Ting Siu, Joshua A. Miele, Lucia Hasty, and Steven Lan-
dau. 2015. Guiding Novice Web Workers in Making Image Descriptions Using
Templates. ACM Transactions on Accessible Computing (TACCESS) 7 (2015), 1 –
21.
[34] David Morris, Eric Müller-Budack, and Ralph Ewerth. 2020. SlideImages: A
Dataset for Educational Image Classifcation. Advances in Information Retrieval
12036 (2020), 289 – 296.
[35] Meredith Ringel Morris, Jazette Johnson, Cynthia L. Bennett, and Edward Cutrell.
2018. Rich Representations of Visual Content for Screen Reader Users. Proceedings
of the 2018 CHI Conference on Human Factors in Computing Systems (2018).
[36] Azadeh Nazemi and Iain Murray. 2013. A Method to Provide Accessibility for
Visual Components to Vision Impaired.
[37] Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy:
Fast and Robust Models for Biomedical Natural Language Processing. ArXiv
abs/1902.07669 (2019).
[38] Aaron Nicolson, Jason Dowling, and Bevan Koopman. 2021. AEHRC CSIRO at
ImageCLEFmed Caption 2021. In CLEF.
[39] Xin Qian, Eunyee Koh, Fan Du, Sungchul Kim, and Joel Chan. 2020. A Formative
Study on Designing Accurate and Natural Figure Captioning Systems. In Extended
Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems
(Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New
York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3382946
[40] Xin Qian, Eunyee Koh, Fan Du, Sungchul Kim, Joel Chan, Ryan A. Rossi, Sana
Malik, and Tak Yeon Lee. 2021. Generating Accurate Caption Units for Figure
Captioning. Proceedings of the Web Conference 2021 (2021).
[41] Katharina Reinecke, David R. Flatla, and Christopher Brooks. 2016. Enabling
Designers to Foresee Which Colors Users Cannot See. Proceedings of the 2016
CHI Conference on Human Factors in Computing Systems (2016).
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean
Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein,
Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition
Challenge. International Journal of Computer Vision 115 (2015), 211–252.
[43] Elliot Salisbury, Ece Kamar, and Meredith Ringel Morris. 2017. Toward Scalable
Social Alt Text: Conversational Crowdsourcing as a Tool for Refning Vision-to-
Language Technology for the Blind. In HCOMP.
[44] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Concep-
tual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic
Image Captioning. In ACL.
[45] Noah Siegel, Zachary Horvitz, Roie Levin, Santosh Kumar Divvala, and Ali
Farhadi. 2016. FigureSeer: Parsing Result-Figures in Research Papers. In ECCV.
[46] Abigale Stangl, Meredith Ringel Morris, and Danna Gurari. 2020. "Person, Shoes,
Tree. Is the Person Naked?" What People with Vision Impairments Want in Image
Descriptions. Association for Computing Machinery, New York, NY, USA, 1–13.
https://doi.org/10.1145/3313831.3376404
[47] Lucy Lu Wang, Isabel Cachola, Jonathan Bragg, Evie (Yu-Yen) Cheng,
Chelsea Hess Haupt, Matt Latzke, Bailey Kuehl, Madeleine van Zuylen, Linda M.
Wagner, and Daniel S. Weld. 2021. Improving the Accessibility of Scientifc Doc-
uments: Current State, User Needs, and a System Solution to Enhance Scientifc
PDF Accessibility for Blind and Low Vision Users. ArXiv abs/2105.00076 (2021).
[48] Shaomei Wu, Jefrey Wieland, Omid Farivar, and Julie Schiller. 2017. Automatic
Alt-text: Computer-generated Image Descriptions for Blind Users on a Social
Network Service. Proceedings of the 2017 ACM Conference on Computer Supported
Cooperative Work and Social Computing (2017).
[49] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and
Jianfeng Gao. 2020. Unifed Vision-Language Pre-Training for Image Captioning
and VQA. ArXiv abs/1909.11059 (2020).
ASSETS ’22, October 23–26, 2022, Athens, Greece S. Chintalapati, J. Bragg, and L.L. Wang

A HEURISTICS FOR IDENTIFYING GRAPHS • data

AND CHARTS • points
• error
The full list of words and phrases used to identify fgures corre-
• error bar
sponding to graphs and charts include:
• trial
• graph • trials
• chart • bar plot
• plot • bar
• scatter plot • venn
• scatter • mean
• distribution • average
 Low-Cost Tactile Coloring Page Fabrication on a Cuting
Machine
Assembly and user experiences of cardstock-layered tangible pictures
Nicole E. Johnson*
University of Colorado, Boulder, USA, University of Colorado, Boulder, USA,
Nicole.Johnson-1@colorado.edu Tom.Yeh@colorado.edu
ABSTRACT
Tactile images are an important mode of information for the blind
and low vision community but many of the common methods have
high acquisition and material costs. This design inquiry looks at
the use of a Cricut cutting machine to layer cardstock for creating
tangible pictures and coloring pages. This report discusses the devel-
opment of the technique followed by the corresponding production
considerations, design iterations, pilot feedback, and limitations.
We believe that in certain contexts this technique could be a viable
method for conveying tactile information, especially in the context
of early learning activities where practice is required for gaining
life-long tactile graphicacy skills that can translate to reading maps
and information graphics later in life. Additionally, the low acquisi-
tion cost of this method has value to parents and educators of blind
and low vision children as a cheap DIY way to make tactile content.
CCS CONCEPTS
• Human-centered computing; • Accessibility technologies;
Accessibility systems and tools;; • Social and professional
topics; • People with disabilities.;
KEYWORDS
CNC machine, tactile graphic, paper, pictures, touch, blindness,
illustrations, images, tactile, early learning, technique
ACM Reference Format:
Nicole E. Johnson*, Tom Yeh, and Ann Cunningham. 2022. Low-Cost Tac-
tile Coloring Page Fabrication on a Cutting Machine: Assembly and user
experiences of cardstock-layered tangible pictures. In The 24th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22),
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 5 pages.
https://doi.org/10.1145/3517428.3551353
1 INTRODUCTION
The proliferation of the internet has opened a landscape of informa-
tion access to many populations; however, visual imagery remains
a challenge for blind and low vision (BLV) communities. Common
alternative modes for conveying visual information include audio
descriptions and tactile representations, ideally paired together
This work is licensed under a Creative Commons Attribution-NonCommercial
International 4.0 License.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3551353
Tom Yeh Ann Cunningham
Sensational Books, USA,
Ann@acunningham.com
[1, 2]. Tactile images in particular hold benefts regarding spatial
cognition [3] but also challenges regarding availability and literacy
[4–6]. This work addresses availability and literacy development
by leveraging a low-cost technology to make tactile coloring pages
for BLV early literacy content.
This report summarizes three years of work investigating a com-
mercially available CNC cutting machine, the Cricut Maker, as a
viable medium for creating tactile pictures. We will discuss the
development of our cardstock layering technique followed by the
corresponding production considerations, design iterations, pilot
feedback, and limitations. Cardstock layered pictures are a cost-
efective alternative to other traditional tactile image techniques
and are ideal for coloring activities, a skill that helps build tactile
graphicacy in early learning settings.
2 RELATED WORK
2.1 Production of Tactile Images
In terms of image production, most techniques remain analog with a
high time and resource cost due to the specialty demand of the tools
and need for design mediation. The most common methods used
today for tactile images include microcapsule paper (swelled lines),
thermoform (vacuum-formed plastic), craft supplies, 3D printing,
paper embossing (raised dot matrix on paper), and refreshable pin-
matrix devices (matrix of pins on a refreshable tablet) [7–9].
Each of these methods has pros and cons across a range of spec-
trum including acquisition cost, production time, production cost,
image resolution, ease of use/production, and whether or not the
image is dynamic [9] as well as accessibility of production. All meth-
ods with the exception of rudimentary craft supplies involve high
acquisition costs that are between $1,400 for swell, thermoform, and
low-volume embossers and over $14,000 for refreshable pin matrix
displays and high-volume embossers. Cricut cutting machines have
a lower acquisition cost ($200-$400) and has the advantage of being
able to use regular cardstock and other craft materials as opposed
to swell paper that costs $1.50-$2.50 per sheet.
2.2 Cognition of Tactile Images
In order to consider the cognitive challenges in reading tactile im-
ages, one must consider the fundamental diferences between the
visual and tactile modality; vision is global and immediate whereas
touch is local and sequential [10, 11]. For this reason, tactile design
is not a one-to-one process. Visual embellishments must be stripped
away, where spacing and textural distinguishability become the
most important elements [12, 13]. Understanding communication
through tactile images requires tactile graphicacy, as vision re-
quires visual graphicacy. Tactile graphicacy is a learned skill that
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 1: (Left) Middle layer of lobster design having registration holes fed into dowels. (Right) Top of lobster design fed into
dowels
takes practice [14–19] and consequently, access to a wide range of
materials and pictures that are not as immediately available as in
vision [13, 19, 20]. The long-term benefts of tactile reading skill
development have been shown widely across areas such as spatial
reasoning and working memory [13, 20–23], as well as being an
indicator of future employment [24, 25].
3 CARDSTOCK LAYERING TECHNIQUE
DEVELOPMENT
The initial idea for this cardstock layering technique came from
co-author Ann Cunningham and is centered around the premise
of registration holes, which line up the image layers and allow for
streamlined duplication. Registration holes are a common method
in printmaking to line up multi-block prints. The general cardstock
layering technique is outlined in Figure 1.
The idea for layered image components stems from the process
of stacking clay layers to create bas-relief images. Bas-relief is a
style of sculpture where the subject of the image is protruding out
of a fat surface. Types of this image include low-relief, high-relief,
and sunken relief [26] The peg board was created using a thick
plastic base with holes drilled in for two dowels. The spacing of the
dowels corresponds to the outer holes of the standard three-hole
punch (7mm holes spaced 70mm from center of hole). Each design
includes all three holes for easy storage in three ring binders.
3.1 Early Design Phase
The following research questions guided the early phases of this
design inquiry:
1. How can we leverage a Cricut cutting machine to create
tactile pictures?
2. How can we create engaging tactile content for kids?
3. How can we make tactile pictures and activities collabora-
tive?
The inquiry started with familiarizing to the machine and creat-
ing prototype designs over a period of months to experiment with
various paper thicknesses and textures. Designs were drawn in the
vector drawing program Adobe Illustrator and saved as .SVG fles.
Nicole Johnson et al.
For the designs themselves, the top layer is always the broadest out-
line of the object, then the middle paper layer(s) include important
details of the object.
One draw to this technique was the possibility for duplication
and streamlined production. Our frst large-scale picture distribu-
tion of around 80 pictures was a tiger design by Cunningham. All
three authors participated in the frst assembly-line style produc-
tion technique, with one gluing the layers and transferring them
for assembly at the peg boards. Working together we were able
to complete the 80 pictures with 160 paper layers in one hour. We
theorized the assembly could go very fast with between 5-10 vol-
unteers, each with a specifc job such as gluing, transferring paper
layers, multiple peg board assembly stations, transferring assem-
bled pictures to the drying rack, etc. The task also works with just
two people, with one person gluing and the other assembling. It is
possible to do alone but does add time switching between gluing
and assembling.
The tiger design was distributed in the art room at the National
Federation of the Blind Colorado 2019 Regional Conference for pilot
feedback. These sessions yielded informal feedback from both blind
and sighted children and adults about the tiger picture, coloring
activity, as well as brainstorming subject matter the viewers would
be interested in.
At this stage in the inquiry, we learned that the thickness of
paper was more important than a linen textured paper. The thicker
paper we used was 110lb/300gsm weight and created a more dis-
tinct edge for the design. The linen paper texture was not very
discernable from the regular paper texture and was too thin for
cutting, but moving forward we continued to use it with designs as
the uncut backing page. The Cricut Maker is able to cut a variety of
materials such as fabric, foam, plastics, cardboard, vinyl, and others.
This ofers tactile designers many options for adding areal textures
to their images. The best adhesive we found is the solvent-based
product Super 77 Spray by 3M. Assembly requires an extremely
well-ventilated area and aerosol masks.
Low-Cost Tactile Coloring Page Fabrication on a Cuting Machine ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 2: Ten designs clockwise from top left: stegosaurus, girafe, sea turtle, grasshopper, Saturn, rhino, seahorse, rocket,
honey bee, and elk.

3.2 Late Design Phase Table 1: Material costs for each picture
After establishing a papercut picture workfow and getting pre-
liminary feedback from the BLV community, frst author Nicole Material Cost
Johnson engaged in an extended period of design work over a pe- Super 77 Spray $0.04 per paper layer
riod of months, creating over 80 designs intended for early learning Heavy cardstock $0.15 per sheet
settings. Images include subject matter such as animals, coloring Regular cardstock $0.08 per sheet
pages, maze, matching activities, and cut-outs (Figure 2). Cut-outs Colored cardstock $0.08 per sheet
of shapes and objects have also been shown to help in tactile under- Newsprint backing $0.02 per sheet
standing [7], a bi-product of the top outline page of our designs as Linen textured backing $0.07 per sheet
well as possibilities for collage cut-out activities made on a Cricut. Total cost per picture $0.14-$0.64

a mid-range price per picture compared to swell paper at $1.50-

4 TECHNIQUE IN PRACTICE
4.1 Cost & Time Breakdown
As mentioned above, this layered cardstock technique is cheaper
than traditional tactile picture techniques in terms of acquisition
and materials. Material cost breakdown is listed in Table 1.
One of the limitations to this technique is time required cut-
ting the pages, which varies depending on material thickness and
amount of detail. Table 2 includes a breakdown of production time
for 6 designs with production data averaged based on the number
of copies. Data includes total cutting time of all layers, gluing time
per picture, trimming time per picture, braille label time (Including
punching with a slate and stylus and adhering to the picture), and
the total production time for each picture.
Each cardstock layered picture costs between $0.14-$0.64 de-
pending on the amount and material of the paper layers. This is
$2.50 per picture, thermoform at $0.13-$0.36 and braille paper for
embosser graphics at $0.05 [27]. While thermoform and embosser
graphics have a lower cost per picture, the acquisition cost is con-
siderably higher at around $1,000-$4,000 for thermoform machines
and $1,300-$6,000 for graphic capable, low-volume embossers [28].
4.2 Feedback
Due to the COVID-19 pandemic there was a period of months where
it was difcult to collect feedback from users. The designs in this
phase were instead guided largely by production considerations
from the sighted designer with an extensive background in tactile
design. Ideally there would be more intermittent user feedback
directly from the BLV community to guide the designs. We are
currently in the process of collecting a range of feedback, discussed
in the following sections.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Table 2: Production time to produce one picture.
Design Total cut time Glue
Heart Sheet 2:35 min 1:15 min
Hands with Hearts 1:55 min 1:15 min
Flower Scene 4:00 min 0:56 sec
Sea Turtle 2:55 min 1:05 min
Butterfy 2:13 min 1:00 min
Unique Hands 2:20 min 1:00 min
4.2.1 Picture Exchange Program. Our team paired with two sepa-
rate tactile content creators who provide families and teachers of
the visually impaired (TVIs) with monthly subscription boxes. They
agreed to share two of our pictures per month in the boxes over
a period of 3 and 4 months. Interviews were also conducted with
the creators about their background and experiences producing
BLV learning materials. One topic brought up in both interviews
was the need for parents to be engaged and informed about how
to support their children’s learning outside of the classroom and
what DIY tools are available for creating content.
The box for K-5 ages reached around 30-50 families over a period
of 4 months, distributing around 300 pictures. The box for the 3-6
age range reached around 25-35 families over a period of 3 months,
distributing around 200 pictures. Each month included a short sur-
vey link for the pictures, ofering more pictures in exchange for
feedback. In the process of developing prize incentives, two 10-page
themed coloring books were developed titled “In the Garden” and
“In the Ocean” based on the design library and altered slightly for
added braille titles.
Preliminary picture feedback from parents was largely positive,
all stating that the tactile sensation was good. One parent suggested
colored images for low-vision kids. Another family had fun with the
pictures by using them as a backing guide to color over with crayons
on another sheet of printer paper. One aspect of these designs that
make them ideal for coloring is the barrier of the papercut that
creates a guide for crayons.
4.2.2 Open-Source Design Share. In addition to dissemination of
assembled pictures, all design fles are shared as open-source. File
variations for the images include: .SVG for layered cardstock, .PDF
of all layers for swell paper and embossers, stencils fles for hand
embossing, and the original .AI design fle. Ofering a variety of fle
types for each design increases availability based on what the user
has access to, whether it be a Cricut, swell machine, embosser, or
low-tech craft supplies. Files are available in the meta tactile graphic
library BTactile [29], which catalogs tactile graphic libraries all over
the world, consolidating the resources into a single source. Putting
the fles into community hands ofers new learning experiences
and insights about this medium and its practical applications.
4.2.3 Tactile Drawing Club. In collaboration with the BTactile
team, we shared the layered cardstock method to TVIs in Colombia
and Argentina on two occasions. First, during an hour presentation
about the cardstock paper layering technique with a question-and-
answer session with around 60 participants.
Nicole Johnson et al.
Trim Braille labels Total time per pic
0:45 sec 1:00 min 5:35 min
0:45 sec 1:00 min 4:55 min
0:43 sec 0:53 sec 6:32 min
0:44 sec 0:53 sec 5:37 min
0:40 sec 0:53 sec 4:46 min
0:40 sec 0:53 sec 4:53 min
The second was a more in-depth look at the technique with 9
participants in a Tactile Drawing Club. The technique was repli-
cated in all phases and worked well for the tactile literacy process
required to replicate an image through tactile drawing. Four of
the drawing club participants were blind TVIs and ofered helpful
insights about the technique. One of the biggest issues with the
technique is the lack of accessibility in the production stages. One
positive that was mentioned is the debossed nature of the image,
which stores better and does not wear down like a raised embossed
image. Another comment that was brought up was the possibility to
deconstruct the image by leaving the parts unglued in a binder and
fipping through them individually to better understand how they
ft together. This method of the deconstructed layers also allowed
participants to utilize as stencils when recreating the image, which
some participants especially liked. The coloring activity angle was
also received positively as a skill development tool to help with the
metacognition of tactile image processing.
5 LIMITATIONS
This design inquiry had limitations in terms of feedback opportuni-
ties due to COVID-19 restrictions. We recognize the importance of
participatory design and in the future would prefer a more inter-
woven feedback and design system and are actively working to
build one. We are currently working to collect feedback from broad
and diverse sources to provide more insights into this new tactile
image technique. The feedback we have received indicates this
medium is a viable tactile picture alternative but further testing
scenarios are needed.
As mentioned, one of the biggest limitations to the technique is
the lack of accessibility in the production process. Another draw-
back in the production process is the time required for each picture,
the need for a well-ventilated area when gluing, and the mid-range
cost per picture.
6 FUTURE WORK
There are many avenues for possible future work on this topic. One
possible direction of the work is more rigorous comparison studies
between this technique and other common tactile image methods
such as raised-line or embosser graphics. We would also like to
better assess the accessibility of the Cricut interface and the broader
cardstock layering technique. Another direction that was suggested
by BTactile is to use this layering technique with new materials
such as acetate to create a master design that can be embossed on
regular braille paper through a press.
Low-Cost Tactile Coloring Page Fabrication on a Cuting Machine
7 CONCLUSION
Tactile images have value to the blind and low-vision community as
a mode of information, helping with spatial reasoning and working
memory. Production techniques of these images have many limita-
tions, mostly due to their analog nature. Lack of availability hinders
information access as well as tactile graphicacy skills. We propose a
cardstock layering technique fabricated on a Cricut Maker cutting
machine that uses registration holes for streamlined production.
Based on preliminary feedback we believe this method has value
as a tactile image mode but further testing is needed.
As discussed in this report, the layered cardstock technique
carries both benefts and drawbacks. The main benefts include low
acquisition and material costs, streamlined assembly, wear-resistant
storage, and the use as coloring activities in early learning settings
and to develop tactile graphicacy. Main drawbacks include the lack
of accessibility in the production process, moderate production
time, and the need for a well-ventilated area when gluing.
ACKNOWLEDGMENTS
This work was supported by National Science Foundation under
Grant No. IIS-1453771 and DGE 1650115. We thank Maria Zuniga-
Zabala and John Guerra-Gomez for collaborating as well as hosting
the design fles on BTactile. We also thank Stacey Chambers with
the ECC and Me, Melisa Matthews with Eye Am LLC, the Tactile
Drawing Club participants, and all anonymous feedback sources
for their valuable perspectives and insights. Appreciation to Gary
Paynter for production assistance.
REFERENCES
[1] Don Parkes. 1998. Tactile Audio Tools for Graphicacy and Mobility "A circle is
either a circle or it is not a circle." The British Journal of Visual Impairment, 16,
3, 99-104. DOI: https://doi.org/10.1177/026461969801600304
[2] Timo Götzelmann. 2018. Visually Augmented Audio-Tactile Graphics for Visually
Impaired People, ACM Transactions on Accessible Computing, 11, 2, 1-31. DOI:
https://doi.org/10.1145/3186894.
[3] Yvette Hatwell, Arlette Streri, and Edouard Gentaz. 2003. Touching for know-
ing: Cognitive psychology of haptic manual perception. John Benjamins Pub,
Philadelphia, PA.
[4] Kim Zebehazy and Adam Wilton. 2014a. Charting success: The experience of
teachers of students with visual impairments in promoting student use of graphics.
Journal of Visual Impairment & Blindness, 108, 4, 263-274. DOI: https://doi.org/
10.1177/0145482X1410800402
[5] Kim Zebehazy and Adam Wilton. 2014b. Straight from the source: Perceptions of
students with visual impairments about graphic use. Journal of Visual Impairment
& Blindness, 108, 4, 275-286. DOI: https://doi.org/10.1177/0145482X1410800403
[6] Penny Rosenblum and Tina S. Herzberg. 2015. Braille and Tactile Graphics: Youths
with Visual Impairments Share their Experiences. Journal of Visual Impairment
& Blindness, 109,. 3, 173-184. DOI: https://doi.org/10.1177/0145482X1510900302
[7] Amy Kalia, Rose Hopkins, David Jin, Lindsay Yazzolino, Svena Verma, Lotf
Merabet, Flip Phillips, Pawan Sinha. 2014. Perception of tactile graphics: Em-
bossings versus cutouts. Multisensory Research, 27, 2, 111-125. DOI: https:
//doi.org/10.1163/22134808-00002450
[8] Sandra Jehoel, Simon Ungar, Don McCallum, and Jonathan Rowell. 2005. An evalu-
ation of substrates for tactile maps and user preferences. Journal of Visual impair-
ment & Blindness, 99, 85-95. DOI: https://doi.org/10.1177/0145482X0509900203
ASSETS ’22, October 23–26, 2022, Athens, Greece
[9] Denise Prescher, Jens Bornschein, J., and Gerhard Weber. 2017. Consistency of a
tactile pattern set. ACM Transactions on Accessible Computing, 10(2), 1-29. DOI:
https://doi.org/10.1145/3053723
[10] Ira Puspitawati, Ahmed Jebrane, and Annie Vinter. 2014. Local and Global Pro-
cessing in Blind and Sighted Children in a Naming and Drawing Task. Child Dev,
85, 1077-1090. DOI: https://doi.org/10.1111/cdev.12158
[11] Richa Gupta, P. V. M. Rao, M. Balakrishnan, M., & Steve Mannheimer. 2019.
Evaluating the use of variable height in tactile graphics. 2019 IEEE World Haptics
Conference (WHC), 121-126. DOI: https://doi.org/10.1109/WHC.2019.8816083
[12] Braille Authority of North America [BANA]. 2010. Guidelines and Standards for
Tactile Graphics. http://www.brailleauthority.org/tg/web-manual/index.html
[13] Megan Lawrence and Amy Lobben. 2011. The design of tactile thematic symbols.
Journal of Visual Impairment & Blindness, 105, 10, 681-691. DOI: https://doi.org/
10.1177/0145482X1110501014
[14] Krish Sathian. 2000. Practice makes perfect: Sharper tactile perception in the
blind. Neurology, 54, 12, 2203. DOI: https://doi.org/10.1212/WNL.54.12.2203
[15] Amede D’Angiulli, John Kennedy and Morton Helle. 1998. Blind children rec-
ognizing tactile pictures respond like sighted children given guidance in ex-
ploration. Scandinavian Journal of Psychology, 39, 3, 187x-190, DOI: https:
//doi.org/10.1111/1467-9450.393077
[16] Frances Aldrich, Linda Sheppard and Yvonne Hindle. 2003. First Steps Towards
a Model of Tactile Graphicacy. Cartographic Journal, vol. 40, 3, 283-287. DOI:
https://doi.org/10.1179/000870403225013014
[17] Annie Vinter, Viviane Fernandes, Oriana Orlandi, and Pascal Morgan. 2012.
Exploratory procedures of tactile images in visually impaired and blindfolded
sighted children: How they relate to their consequent performance in drawing.
Research in Developmental Disabilities, 33, 6, 1819–1831. DOI: https://doi.org/10.
1016/j.ridd.2012.05.001
[18] David Dulin and Yvette Hatwell. 2006. The efects of visual experience and
training in raised-line materials on the mental spatial imagery of blind persons.
Journal of Visual Impairment & Blindness, 100, 7, 414-424. DOI: https://doi.org/
10.1177/0145482X0610000705
[19] Miriam Ittyerah. 2009. Hand ability and practice in congenitally blind children.
Journal of Developmental and Physical Disabilities, 21, 5, 329-344. DOI: https:
//doi.org/10.1007/s10882-009-9146-8
[20] Susanna Millar and Miriam Ittyerah. 1992. Movement imagery in young and
congenitally blind children: Mental practice without visuo-spatial information.
International Journal of Behavioral Development, 15, 1, 125-146. DOI: https:
//doi.org/10.1177/016502549201500107
[21] David Dulin and Coline Serrière. 2009. Special needs education in the blind
population: Efects of prior expertise in raised line drawings on blind people’s
cognition. Procedia - Social and Behavioral Sciences, 1, 1, 549-553. DOI: https:
//doi.org/10.1016/j.sbspro.2009.01.099
[22] Lora T. Likova and Laura Cacciamani. 2018. Transfer of learning in people who
are blind: Enhancement of spatial-cognitive abilities through drawing. Journal
of Visual Impairment & Blindness, 112, 4, 385-397. DOI: https://doi.org/10.1177/
0145482X1811200405
[23] Athina Panotopoulou, Xiaoting Zhang, Tammy Qiu, Xing-Dong Yang, and Emily
Whiting. 2020. Tactile line drawings for improved shape understanding in blind
and visually impaired users. ACM Transactions on Graphics, 39, 4, 89:1-89:13.
DOI: https://doi.org/10.1145/3386569.3392388
[24] Darleen Bogart and Alan J. Koenig. 2005. Selected Findings from the First In-
ternational Evaluation of the Proposed Unifed English Braille Code. Journal of
Visual Impairment & Blindness, 99, 4, 233.
[25] Natalie Martiniello and Walter Wittich. 2019. Employment and visual Impairment,
in The Routledge Handbook of Visual Impairment (1st ed.). Ravenscroft, J. (Ed.),
Routledge, 415-437.
[26] “Relief Sculpture,” Encyclopedia of Sculpture. [Online]. Available: http://www.
visual-arts-cork.com/sculpture/relief.htm. [Accessed: 22-Jun-2022].
[27] “Braille Printers, Tactile Graphics & Supplies,” American Thermoform. [Online].
Available: http://www.americanthermoform.com/. [Accessed: 16-Jun-2021]
[28] “Braille Embossers,” National Library Service for the Blind and Print
Disabled (NLS) | Library of Congress, 2020. [Online]. Available:
https://www.loc.gov/nls/resources/blindness-and-vision-impairment/devices-
aids/braille-embossers/#sources-braille-embossers. [Accessed: 16-Jun-2021]
[29] BTactile Graphics MetaLibrary, 2021. [Online]. Available: https://btactile.com/.
[Accessed: 16-Jun-2021]
 Animations at Your Fingertips: Using a Refreshable Tactile
Display to Convey Motion Graphics for People who are Blind or
have Low Vision
Holloway, Leona Swamy Ananthanarayan Matthew Butler
Monash University, Australia Monash University, Australia Monash University, Australia
leona.holloway@monash.edu swamy.ananthanarayan@monash.edu matthew.butler@monash.edu

Madhuka De Silva Kirsten Ellis Cagatay Goncu

Monash University, Australia Monash University, Australia Monash University, Australia
madhuka.desilva@monash.edu kirsten.ellis@monash.edu cagatay.goncu@monash.edu

Kate Stephens Kim Marriott

Monash University, Australia Monash University, Australia
kate.stephens@monash.edu kim.marriott@monash.edu

Figure 1: Touch readers exploring images on a Refreshable Tactile Display

ABSTRACT
People who are blind rely on touch and hearing to understand the
world around them, however it is extremely difcult to understand
movement through these modes. The advent of refreshable tactile
displays (RTDs) ofers the potential for blind people to access tactile
animations for the very frst time. A survey of touch readers and
vision accessibility experts revealed a high level of enthusiasm for
tactile animations, particularly those relating to education, mapping
and concept development. Based on these suggestions, a range of
tactile animations were developed and four were presented to 12
touch readers. The RTD held advantages over traditional tactile
graphics for conveying movement, depth and height, however there
were trade-ofs in terms of resolution and textural properties. This
work ofers a frst glimpse into how refreshable tactile displays can
best be utilised to convey animated graphics for people who are
blind.
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility; Accessibility technologies; Accessibility systems and
tools.
KEYWORDS
blind, tactile graphics, refreshable displays, accessible graphics
ACM Reference Format:
Holloway, Leona, Swamy Ananthanarayan, Matthew Butler, Madhuka De
Silva, Kirsten Ellis, Cagatay Goncu, Kate Stephens, and Kim Marriott. 2022.
Animations at Your Fingertips: Using a Refreshable Tactile Display to Con-
vey Motion Graphics for People who are Blind or have Low Vision. In
ASSETS ’22: ACM SIGACCESS Conference on Computers and Accessibility,
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 16 pages.
https://doi.org/10.1145/3517428.3544797

Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544797
1 INTRODUCTION
A recent resurgence in refreshable tactile displays (RTDs) with
larger tactile surfaces ofers new opportunities for the touch read-
ing community. Along with the obvious advantages of storage space,
reduction of hard copy materials and immediate access, RTDs ofer
the brand new possibility of animated tactile images. Until now,
if you were blind or had low vision (BLV), access to dynamic graph-
ics such as an animation showing how a bird faps its wings was
provided by a sequence of tactile graphics, with successive graphics
ASSETS ’22, October 23–26, 2022, Athens, Greece
showing key frames in the animation. Tactile animations ofer an
exciting alternative.
While there has been considerable research into technologies
for building refreshable displays, e.g. [44, 72], there has been little
research into the usefulness and efectiveness of animated tactile im-
ages for presenting dynamic graphics to BLV people. Prior research
has focused on specifc applications–a blind soccer match [37, 54],
freworks display [62, 63] and zooming/panning on a map [75, 76].
Here we investigate the efectiveness and design of animated tactile
images in general. Our specifc contributions are threefold:
Contribution 1: Identifcation of potential application areas
and diagrams that touch readers would like to see displayed
as animated tactile images on RTDs. As a frst step we con-
ducted an online survey of 19 touch readers and seven accessibility
experts asking what animated tactile images would be of interest
to touch readers. This revealed a high level of interest in animated
tactile images. Learning support for children and information for
adults were ranked as more important than sport and entertainment
applications.
Contribution 2: Design considerations for creating ani-
mated tactile images for touch readers. The research team, one
of whom is blind, collaboratively designed a variety of animated
tactile images. These were tested on a Graphiti RTD1 . Design con-
siderations include taking account of the lower resolution of most
RTD displays, and tailoring the refresh rate to the complexity of the
image. A limitation of the Graphiti display (like many RTDs), is a
slow refresh rate of up to 5 seconds for a full page. This meant that
the animations are generally experienced as a sequence of discrete
frames rather than as fuid animations.
Contribution 3: Comparison of animated tactile images with
a sequence of tactile graphics. We conducted a user study with
12 BLV participants comparing animated tactile images shown on
the Graphiti versus traditional tactile graphics. We presented four
image sequences in both formats. Overall, participants were better
able to understand change and gain a sense of movement using the
RTD and drew parallels to animation and movies. However, tactile
graphics were generally preferred for understanding of a static
diagram due to higher resolution and superior textural and colour
contrast. An unexpected advantage of RTDs over traditional tactile
graphics is that images can be immediately modifed in response to
viewer feedback.
The research presented here is a frst step in exploring the po-
tential of tactile animations. We hope that it motivates greater ex-
ploration of this new media for touch readers. Our fndings provide
guidance for future designers creating content for touch readers on
RTDs. They also provide guidance on the design of future RTDs,
suggesting that increasing the refresh rate is less important than
increasing the resolution.
2 RELATED WORK
2.1 Animated Graphics
There are three standard approaches to visually showing change
over time. These rely on creating a sequence of frames, each frame
a snapshot of the scene at a particular instant in time. The frst
1 http://www.orbitresearch.com/product/graphiti/
Holloway et al.
approach is to arrange the frames temporally. This results in an
animated graphic in which the frames are shown to the viewer, one
after another. Typically the frames are shown in rapid succession
so that the viewer’s visual system does not perceive the individual
frames but rather perceives objects to be moving smoothly. In
the second approach, called small multiples, selected frames are
arranged spatially in a sequence or grid. In the third approach, a
few frames may be overlaid and arrows or other annotations used
to show the direction of movement.
There have been many studies comparing the efectiveness of
instructional materials showing change over time using animation
with those using small multiples [7, 11, 33, 70]. The results are
mixed. The disadvantage of an animation is that it is difcult to
track multiple simultaneous changes and that if the viewer wishes
to compare two frames they must remember one of those frames
and mentally compare it with the current frame. On the other
hand, it seems to be easier to track selected objects if the surrounds
are also changing in an animation [5]. There is also evidence that
human body movements are easier to learn from an animation than
small multiples [7]. Another application in which animation has
proved efective is to animate transitions between diferent views
of the data in data visualisation [30].
2.2 Tactile Graphics
Tactile graphics, also known as raised line drawings, are recom-
mended as the best way for people who are blind or have low
vision to access and understand diagrams with spatial informa-
tion [17, 42, 65]. The frst tactile graphics were produced manually
using collage and copied using pressed wet paper (e.g. [40]) or ther-
moforming, and collage remains the most popular technique used
by vision specialist teachers in schools [58]. Since the widespread
adoption of digital production in the 1980s, swell paper and em-
bossing have become the most popular tactile graphics production
techniques [66]. Discriminability of textures and lines are the most
important factors determining user preference for tactile graphics,
with variable heights and visual contrast as welcome additions [4].
While there are many internationally recognised and detailed
guidelines on the design of tactile graphics (e.g. [17, 25, 65]), they
provide little or no advice on how to represent moving images.
Instead, there is an assumption that tactile graphics will be based
on static images, usually from text books. These may use small
multiples or annotations like arrows to convey change and move-
ment (Fig. 2a&b). Due to the high storage and/or material costs of
some tactile graphics, when a large number of diagrams is required,
as with small multiples, they are often reduced in size. This can
negatively impact their legibility by touch.
Another approach to movement in tactile graphics, employed
mainly for young children, has been the addition of moving parts
to a base diagram. For example, a cardboard clock hand may be
attached to a tactile graphic of a clock using a split pin (Fig. 2c), or
a bead may be attached to a thread placed across the tactile image
to denote movement along a route [23, 25]. Whilst providing a high
level of engagement, these graphics must be handmade, they are
often bulky and they must be stored in hard copy.
Animations at Your Fingertips ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 2: Examples of traditional approaches to conveying movement with tactile graphics: (a) small multiples depicting solar
eclipse stages (b) using arrows to signal the movement of melted sulfur during the Frasch process (c) moving parts (clock
hands attached with a split pin) on a tactile graphic and (d) a manipulative to accompany a static tactile diagram (Fleximan by
Hungry Fingers)

2.3 Refreshable Tactile Displays (RTDs) limitations for touch input and made subsequent suggestions re-
Refreshable tactile displays date back as far as 1931 [1], and some garding the design of refreshable graphics to be used for this pur-
such as the Optacon have even been used to covert images to pose [59, 61].
refreshable pins on a small display [19, 22]. Since the advent of
maker technologies and an injection of funding spurred by the 2.4 Applications of RTDs
DAISY Consortium’s Transforming Braille Project [69], a range of
The most basic research into RTDs examines perception based on
competing technologies have come to market or are in prototype
pin density, height and other factors like vibration [27, 37, 54]. Initial
phase. The majority of refreshable tactile displays consist of a grid
research into their use has focused on the display of static images
of pins controlled by electro-mechanical actuators [72]. Resulting
in the felds of art [29], maps [18, 34, 43, 51, 67, 75] and textbook
from the HyperBraille project2 , Metec have been selling refreshable
diagrams [36, 52, 57]. Most notably, O’Modhrain and colleagues ex-
graphics displays since 20123 . More recently, Orbit Research has
plored the possibilities and potential pitfalls of refreshable graphics
released Graphiti, a tactile display of 60×40 refreshable pins that can
displays from the perspective of vision impaired users [55]. They
be raised to four diferent heights4 . Looking forward, Humanware
point out that most refreshable tactile displays have low fdelity,
is working in partnership with the American Printing House for the
therefore images must be simplifed and carefully designed to be
Blind to develop a Dynamic Tactile Display (DTD) similar in size
readable on a refreshable tactile display.
to Graphiti and spaced comparably with braille [24]. Meanwhile,
There has also been considerable interest in using RTDs to give
Dot Inc is developing Dot Pads, a refreshable pin display with
real-time feedback to assist people who are BLV when creating
the capability to zoom and pan5 . Also in development, the Blitab
graphics. They have been used for drawing [13, 38, 53], plotting
Android Tablet has 14 rows of 23 6-dot braille cells6 and is designed
mathematical charts [3, 39] and for 3D modelling [47, 68]. Similarly,
for displaying both text and images, and TTPAT are developing
Bornshein, Prescher and colleagues have investigated the use of
the TouchPad Pro with dots raising to varying heights and a colour
touch-sensitive refreshable displays for collaborative creation of
display7 .
tactile graphics [14–16]. The advantages of being able to update
Fixed grids of movable pins or braille dots are not the only ap-
graphics or store multiple versions on an RTD have also been
proach for creating refreshable tactile displays. Some researchers
explored [18, 43, 45].
have attached a small grid of pins to a mouse; as the mouse is
moved, the pins are updated refecting the mouse’s position on a
large virtual grid [41]. Disney Research used water jets on a fexible 2.5 Animated Tactile Images
screen [62, 63]. Surface haptic devices such as the TeslaTouch [9] There has been much less research into the use of RTDs to show
use electrostatic resistance on a fat screen, however detection is change over time by presenting animated tactile images.
limited to very simple shapes using this method [71]. Weber and colleagues have investigated zooming and panning
The touch-feedback feature of tactile displays can also been of a map and UML diagrams on an RTD [48, 60, 75, 76]. A pilot
used to support audio feedback. Researchers have explored the evaluation suggested BLV participants found it difcult to explore
a map using panning [76].
2 http://hyperbraille.de/project/ Ohshima, Kobayashi and colleagues developed and tested a sys-
3 https://metec-ag.de/produkte-graphik-display.php
tem to represent a football match on an RTD [37, 54]. They reported
4 http://www.orbitresearch.com/product/graphiti/
5 https://pad.dotincorp.com/ that of their seven participants “several participants commented
6 https://blitab.com/ that they could distinguish the detailed movement of the play-
7 https://tppat.com/the-touchpad-pro/ ers” [54]. Jung and colleagues presented animated games like pong
ASSETS ’22, October 23–26, 2022, Athens, Greece Holloway et al.

on a small prototype RTD. Users held their hand fat on the de-
vice and were able to locate stimuli within a 13mm a margin of
error [35].
A range of other methods have also been trialled to create tactile
animations. Disney Research investigated the use of their water-jet
display for visualisation of freworks displays, which was evaluated
as enjoyable by most of their BLV participants [62, 63]. The Phan-
tom device has been used to haptically explore 3D objects moving
in virtual reality, e.g. [10, 47]. Slide-tone and Tile-tone use motors
to move a user’s fnger along a graph line [26] and Robographics
uses motorised robots to indicate key areas on a graphic [28]. How-
ever, none of these methods are applied to representational tactile
graphics.
To date there has not been a systematic investigation of using
animations to show change in tactile graphics over time. Here we
address three fundamental questions: When and for what types of
graphics are they useful? What are design guidelines for creating
them? And how do they compare with traditional tactile graphic
presentations?
3 CONSULTATION
We began by consulting with the vision impaired community to
determine whether tactile animation is of interest and, if so, what
type of animations would be of most value.
3.1 Method
A list of possible animations was derived through multiple brain-
storming sessions with the research team, cross-referencing against
school textbooks, and in consultation with touch readers. This list,
as given in Table 1 but with accompanying examples, was presented
in survey format. Respondents were asked to indicate which they
would be interested in having made available and to provide further
examples of tactile animations that they would recommend.
The survey was distributed online using Google Forms. It was
tested for accessibility and keyboard shortcuts were provided. The
survey link was shared through a participants pool, social media
and listservs.
that the respondents are not a homogeneous group, potentially
difering in their priorities.
3.3 Results
As shown in Table 1, there is a high level of interest in moving tactile
images. In general, learning support for children and information
for adults were seen as more important than sport and entertain-
ment applications. More specifcally, the most popular selections
were astronomy (e.g. planetary orbits, eclipse, the big bang), ge-
ographic movement (e.g. shifting of the continents over time,
tectonic plate movement, erosion, waves), maps with movement
along a route, rotation (e.g. geometric shapes rotating, the earth
spinning), biology concepts (e.g. plant growth, embryo growth,
mitosis and meiosis), physics animations (e.g. projectile motion,
pulleys, pendulum movement, airplane wings), maps with move-
ment in space (e.g. stage directions, dance positions, etc.) and hu-
man movement (e.g. dance moves, exercise moves, sports moves,
swimming strokes).
In addition, participants made a number of further suggestions
for moving graphics for education (analogue and digital output
from oscilloscopes, plotters and curve tracers; CAD/CAM software
for schematics and CNC machining work; chemical reactions; dy-
namic charts for maths and statistics; historical maps; sex educa-
tion), for orientation and mobility (transport routes with move-
ment), sports (team sport positions; martial arts), for entertain-
ment (movies, tv or videos), and to assist with content creation
(videos; presentation slides). Some further suggestions were given
for static images.
4 INITIAL DESIGN EXPLORATION
4.1 Graphiti

3.2 Participants
A total of 26 people responded to the survey. Nineteen were touch
readers and seven were sighted members of the BLV community –
parents of touch readers(n=2), accessible formats producers(n=3)
and vision specialist teachers(n=3). The average age was 52 years
(sd=13.7), ranging from 20 to 75. The vast majority resided in Aus-
tralia (n=17), with the remainder in North America (n=7) or New
Zealand (n=1). There was a reasonable gender balance, with 14
women, 10 men and 1 non-binary. Education level was high, with
17 respondents (68%) holding a Bachelor’s degree or higher, com-
pared with 24% of the Australian adult population in 2016 [6]. Of
the touch readers, 16 were totally blind and 2 were legally blind.
Twelve were blind from birth and two had acquired their vision
impairment at age 15 or over. The majority of the touch readers Figure 3: Graphiti tactile display consisting of 60×40 refre-
(n=12) considered themselves up-to-date with new technology and shable pins, showcasing an ocean wave
a further fve were early adopters. Overall, the self-selected sample
are well-educated and technology savvy, potentially representing The test images were presented using a Graphiti tactile display,
the future early adopters of RTDs. However, it is acknowledged commercially available from Orbit Research [56] for USD$25,000.
Animations at Your Fingertips ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Interest in moving diagrams by survey respondents who are BLV and others (parents, educators and accessible format
producers

BLV Others Total

Suggested topic
(n=19) (n=7) (n=26)
Education and science
Astronomy 18 7 25
Geographic movement 17 6 23
Rotation 16 5 21
Physics animations 15 5 20
Biology concepts 15 5 20
Orientation & Mobility
Map with route 17 7 24
Map with movement 15 6 21
Trafc movement 15 4 19
Concept development
Modes of fying 16 5 21
Human movement 15 6 21
Animals in motion 14 5 19
News and information
Weather 14 6 20
Emergency mapping 14 5 19
Disease transmission modelling 11 4 15
Migration patterns of people or animals 10 4 14
Entertainment and games
Simple computer games 10 7 17
GIFs 7 6 13
Sport
Players and ball on a football feld 9 5 14
Players and ball on a tennis court 8 6 14
Race on a straight track 8 4 12
Race around a circuit 7 5 12

The Graphiti device measures 29.5×26.9× 4.1 cm and provides an
array of 60 (horizontal)×40 (vertical) refreshable pins spaced 2.1
mm apart (Fig. 3). Each pin can be raised to one of four heights (0.5
mm, 1 mm, 1.5 mm, and 2.0 mm) allowing images to be displayed
in relief. The pins are noticeably larger than standard braille dots.
Below the tactile display, the Graphiti has eight input keys, a
space bar, and a navigation pad with four directional buttons (up,
down, left, right) along with a select button. These controls were
not used in our study as the images were loaded through a HDMI
connection to a host PC.
The frame rate on the HDMI connection can be confgured to
display an image every 1 to 15 seconds. Each line of the image
refreshes progressively from the top down. Up to fve seconds are
required for a refresh, but usually less as only the pins being raised
or lowered are refreshed.
4.2 Creation of Sample Graphics
We began by creating a broad range of image sequences to exper-
iment with how best to design images for an RTD and confrm
which types of graphics might work best as tactile animations (Fig.
4). All graphics were created in accordance with tactile graphics
standards (e.g. [17, 65]) by Holloway, an experienced tactile graph-
ics producer. A range of diferent software programs were used to
create and edit the graphics: Graphiti PC Utility, AseSprite, Libre-
Sprite, Excel, Adobe Illustrator and MS Paint. For each sequence
of RTD images, corresponding tactile graphics were created using
capsule paper or collage as these techniques are most popular in
production houses [66] and schools [58].
These test images were explored by Stephens, who is blind and a
competent touch reader. Improvements to the graphics were made
as a result of her feedback, for example simplifying or enlarging.
As described by O’Modhrain, we found that the low fdelity of the
RTD means that graphics must be greatly simplifed [55].
4.2.1 Map with Route. One of the frst test images was a map of the
route to our workplace. A few methods were trialled to dynamically
show movement along the route (Fig.4f). The favoured option was
to show the full route on the base map then move a single-pin
blinking cursor alongside the route as it was verbally described.
This allows a touch reader to follow the description but also to
independently explore the route. Using the blinking cursor to draw
attention was preferable to the invasive practice of hand-over-hand
direction of touch [21]. We were also able to use the blinking cursor
to aid discussion of other landmarks and routes not already marked
ASSETS ’22, October 23–26, 2022, Athens, Greece Holloway et al.

Figure 4: Some of the sample graphics created for presentation on an RTD in the development phase: (a) map of a complex
multi-level train station (b) alignment of the sun, earth and moon prior to a solar eclipse (c) cat twitching its tail (d) plant
growth (e) Elvis Presley dance moves (f) workplace map (g) cat with moving eyes (h) fetal development. With the exception of
the maps, each graphic is part of a sequence of images to show movement or change.

Figure 5: Input images (top row) and RTD images (bottom row) representing a gull in fight. This sequence was very difcult
to understand by touch.

on the map. This aligns with prior work fnding that blinking pins
were the best method of highlighting a static route and important
landmarks on a map presented on an RTD [34].
4.2.2 Birds in Flight. Two attempts were made to illustrate a bird in
fight. Firstly, a fying gull was depicted in profle view in a sequence
of four images (Fig. 5). However, the concept was not conveyed
successfully as there was a big change in wing position from one
image to the next. This was exacerbated by a slow refresh rate
from the top to the bottom of the page, such that the gull may have
two sets of wings during the refresh. Moreover, even though the
wings were shown at a higher level than the body, it was difcult
to distinguish them when held in line with the body.
A second attempt depicted an owl in fight, seen from the front.
This eliminated the problem of occlusion, however it was difcult
to understand the horizontal spreading of the wings when they
were beating down compared with the folding of the wings when
they were lifting upwards. The number of images in the sequence
was greatly increased to a total of 10 per wing cycle, allowing the
sequence to be viewed at a faster rate without the refresh direction
causing confusion. The size of the image was also reduced so that
the full width of the wings could more easily be felt under the hands
and the refresh was faster.
4.2.3 Human Movement. Two sets of diagrams were created to
demonstrate human movement. The frst illustrated sequential
Animations at Your Fingertips
poses of Elvis Presley dancing (Fig. 4e). It provided a good im-
pression of the dance moves that Stephens had heard about but not
understood until seeing it on the RTD.
While Elvis Presley’s dance moves were interesting, they served
purely as entertainment. We next considered human movement for
instructional purposes, creating a series of six diagrams to illustrate
the moves in a Tai Chi sequence. After some experimentation, we
chose to show the eye(s) and nose on the face to give an indicator of
the direction of the head; raised dots (buttons) down the centre front
of the body to show its direction; and limbs shown with lower dots
if they were further away than the body or higher dots if they were
closer than the body (Fig. 9). This representation was successful
and Stephens liked being able to look at the pose, try it herself and
get verbal feedback regarding any minor adjustments that were
required. This was much preferred over having someone try to
fully describe the poses or physically guide her into the correct
positions.
4.2.4 Solar Eclipse. Astronomy was the most wanted requested
application for moving tactile images in our survey. To represent a
solar eclipse, we began with a representation of the light from the
sun, which was blocked by the path of the moon over a series of
26 steps shown in quick succession. While this sequence gave an
adequate representation of the phenomenon of a solar eclipse, it did
little to explain the causes. We next produced an 8-image sequence
depicting the movement of the earth around the sun (slowly) and
the moon around the earth (more quickly) up to the point where
the moon blocks the light of the sun in a solar eclipse (Fig 4b).
When used together, the two diagram sequences enabled a good
understanding.
4.2.5 Science. Growth and change through time are a common
topic of inquiry in science, especially biology and earth sciences.
To depict plant growth from a seed, we began with a horizontal
(landscape) alignment, however it was difcult to show all of the
important details so we turned the image (and RTD) on its side to
portrait orientation to use more of the display space. The soil was
initially depicted as low dots but it was difcult to distinguish from
the seeds and roots, so the diagram was simplifed to show only
the line of the top of the soil. This greatly improved contrast and
understanding.
The formation of a wave was confgured for the RTD as a series
of seven diagrams with the water represented as the highest level
dots and the sand at level 2. A row of blank dots was reserved
between the water and the sand to aid tactile distinction. A single
wave changed shape and moved from the ocean on the left to the
beach on the right in each successive diagram (Fig 6).
The formation of a waterfall over time was based on a series of
fve simple print graphics that could be translated quite closely on
the RTD. Water was shown as level 1 dots, hard rock as level 4 and
soft rock as level 3. Once again, a row of blank dots was reserved
between elements to assist with tactile distinction (Fig. 7.
Both the wave and waterfall formation series were considered
highly successful – they were simple to understand and the move-
ment was easy to follow, in part because it followed a predictable
path in a discrete area of the display.
Stephens requested a representation of human fetal development,
thinking that it might be useful for blind parents. However, as the
ASSETS ’22, October 23–26, 2022, Athens, Greece
baby curls up in the later months it was impossible to distinguish
the limbs (Fig 4h). Refnement of the images using diferent dot
heights may have helped to some extent.
4.2.6 GIFs. Two of the original test images were animated GIFs
from social media–one of a cat looking from side to side (Fig. 4g),
and another of a cat twitching its tail (Fig.4c). They were selected for
trial because the movement was simple and fast. However, while
the graphics themselves were clear, the purpose of the GIFs to
convey emotional states (puzzlement and annoyance) was unsuc-
cessful because touch readers have a more limited visual vocabulary.
Very simplifed facial expressions may be easier to understand, and
therefore more in keeping with the original intention of GIFs.
4.3 Learnings
4.3.1 Capitalising on Height to Convey Information. Several of the
diagrams were unsuccessful on the RTD due to a difculty in distin-
guishing between diferent areas of the diagram, such as the wings
on the gull, the eyes on the cat and limbs on the fetus. The gull
was produced using standard image software then converted auto-
matically by the Graphiti software. Direct editing of the individual
pixels and their heights may have given a better result. Likewise,
the fetus diagrams could have been improved with greater use of
contrasting heights, with level 1 for the base shape and level 3 or 4
to highlight the the most important features.
4.3.2 Resolution. The low resolution of the RTD meant that less
detail could be included in a single RTD diagram compared with a
tactile graphic. For example, more braille labels and an additional
feature (stairs) could be included on the tactile graphic version
of a city square map. Furthermore, curved shapes were difcult
to convey with a fxed pin matrix, and prior research has found
that tactile shape recognition is worse for pins compared with
continuous lines [27].
4.3.3 Refreshing from the Top of the Page. The Graphiti display
refreshes its pins progressively from the top row to the bottom.
This afected reading of the graphics. Stephens’ favourite graphic
sequence depicted the formation of a waterfall, beginning with
water running from left to right along the top of the display then
dropping further down as the ground erodes. She was able to follow
the water as the waterfall dropped downwards. Conversely, she
had a lot of difculty interpreting a side view graphic of gull in
fight because when the wing was raised, the previous wing was
still visible while the display was refreshing. Images that changed
from side to side, such as a wave forming from left to right, were
less impacted by the refresh from top to bottom. When using RTDs
that refresh line-by-line, the refresh direction is a limiting factor
and design consideration for the display of sequential images.
4.3.4 Procedure for Presenting Moving Images on a Refreshable Tac-
tile Display. A procedure for best presenting moving tactile images
was developed through trial and error while exploring prototype
image sequences with Stephens. Guidelines for teaching with tactile
graphics state that an overview should be given frst [4, 23, 31]. Ac-
cordingly, we began by giving a brief but precise verbal description
of the frst graphic in the sequence, beginning with an overview
and then giving the location and height of the key features.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 6: Sequence of seven graphics illustrating the movement of a wave for presentation on a refreshable tactile display,
progressing from top left to bottom right, and equivalent collage tactile graphic
5 EVALUATION
In order to better understand the relative benefts and considera-
tions for using an RTD for tactile animations, we ran a user study
comparing RTDs with traditional tactile diagram representations.
5.1 Materials
Four tactile animations were selected for the user study. They repre-
sented the most popular topics from the online survey and showed
a range of diferent applications. Each image sequence was also
produced as corresponding tactile graphics, created in a manner
that was considered to be typical of standard practice to provide an
ecologically valid comparison. Descriptive text to accompany the
graphics was scripted and kept as consistent as possible between
the two presentation modes (Appendix A).
5.1.1 Wave. The formation and movement of a wave was based
on a print graphic showing a cross-section of the ocean and sea bed
with several waves on a single diagram. As described in Section
4.2.5, this was translated to a 7-image sequence of a single wave
moving towards the beach on the RTD for the sake of simplicity
and because each individual image has no associated material costs.
The tactile graphic could show more detail in one diagram and was
therefore based much more closely on the original print graphic. It
was constructed using collage with water represented using blue
cardboard covered in a smooth clear plastic, and sand represented
with a fne-grained sandpaper. The wave formation test materials
(Fig. 6) were chosen for the user study because the RTD animation
was thought to give a good sense of movement and animation.
5.1.2 Waterfall. Waterfall formation was chosen for the user study
because it was thought to be a successful example that gave a
fairly straightforward translation of educational materials. The
corresponding tactile graphics were likewise very similar to the
original print graphics. They were created on fve separate A4 sheets
of swell paper with water represented as horizontal lines, hard rock
as solid fll and soft rock as a textured fll. The test materials are
shown in Fig. 7.
5.1.3 Map. A new map was created to represent Federation Square,
a complex and well-known outdoor area. Buildings were shown as
high (level 4) dots and the roads were represented as low (level 1)
dots. Space allowed for only two braille labels. The route was not
shown on the base graphic, but instead represented by a moving
Holloway et al.
cursor. The corresponding tactile graphic was produced on A4
size swell paper with straight lines for roads and solid flled areas
for buildings. Due to the higher fdelity of swell paper, there was
enough space to include two additional street names and striped
areas for stairs. The route was shown as a dotted line created with
a spur wheel. The test materials are shown in Fig. 8. A map was
included in the user study because it is one of the most common
types of graphics used by people who are blind or have low vision
and because it allowed us to explore the use of a fashing cursor as
a diferent means of conveying movement on an RTD.
5.1.4 Tai Chi. As described in Section 4.2.3, a series of diagrams
were created to represent a series of Tai Chi movements. The dia-
gram was shown sideways (in portrait) on the RTD to allow details
such as the facial features and shirt buttons to indicate rotation.
The corresponding tactile graphics were created on swell paper,
with the addition of spur wheel lines for the shirt buttons. Three
fgures were able to be placed on each page, albeit at a smaller size
than on the RTD, and limbs were shown with a dashed line if they
were further away than the body. We began with a sequence of six
diagrams, however after showing them to the frst two participants,
we realised that this number was unnecessary and time-consuming
to interpret, so the number of diagrams was reduced to three for
the remaining participants. The test materials are shown in Fig. 9.
5.2 Participants
A total of twelve blind adults took part in the user evaluations,
as detailed in Appendix B. Aged from 23 to 68 (x̄=46.9), half were
totally blind and half were legally blind. Seven had past or present
visual experience that they could draw upon to understand visual
concepts, including three participants who had enough residual
vision to assist in their understanding of the high contrast tactile
graphics. All were braille touch readers. While none of the partic-
ipants used tactile graphics more frequently than monthly, only
four considered themselves beginner users needing support in the
use of tactile graphics. The remainder considered themselves pro-
fcient (n=3) or expert (n=5) in exploring and interpreting tactile
graphics. The majority (n=10) considered themselves up-to-date
with technology, while one was an early adopter and one needs
support or encouragement to try new technologies.
Animations at Your Fingertips
Figure 7: Five stages in the formation of a waterfall for presentation on a refreshable tactile display (top) and on swell paper
(bottom)
Figure 8: Map of Federation Square for presentation on a re-
freshable tactile display (left) and on swell paper (right)
Figure 9: Sequence of six Tai Chi poses for presentation on
a refreshable tactile display (top) and swell paper (bottom)
5.3 Method
Participants were shown all four tactile animations and the corre-
sponding tactile graphics. The order of presentation was counter-
balanced with participants split into four groups to adjust both the
order of diagrams and the order of tactile graphic versus RTD.
The diagram description was delivered verbally so that other
factors would not infuence the results, such as braille reading
ability, available space for braille labels, or diferent mechanisms
for audio labels on the refreshable display compared with the tactile
graphic.
The diagrams were shown one at a time until the touch reader
was ready to go to the next. Once they had understood the static di-
agrams, the images on the refreshable display were run in sequence
with a cycle speed between 2 and 8 seconds, as predetermined by
ASSETS ’22, October 23–26, 2022, Athens, Greece
Stephens. After viewing one cycle, the rate was adjusted until the
touch reader was satisfed.
After the participant had seen each diagram (either as a tactile
graphic or on the RTD), they were asked whether the static image
was understandable and whether they could detect what changed.
After they had seen both diagrams on a single topic, they were asked
which was better in terms of initial understanding, detecting what
had changed, getting a sense of motion, and which they preferred
overall. They were also asked how the images might be improved,
whether they had learned anything, and whether an RTD is useful
for that type of diagram.
After the participant had explored all of the diagrams, they were
asked a series of open-ended questions about their preferred ani-
mations and suggestions for future use. They were also asked eight
questions from the System Usability Scale [20]. The term ‘system’
was replaced with ‘refreshable tactile display’ as the SUS tool has
been found to be robust to such wording substitutions [8].
Sessions were conducted at the University or the participant’s
home and took around 90-120 minutes to complete. Video and
audio recording was conducted. The audio was transcribed with
additional notes regarding observed hand positions.
5.4 Results
Participant preferences for the RTD sequences compared with the
tactile graphics are given in Table 2. Overall, there was a moderate
preference for the tactile graphic for gaining an initial understand-
ing of the static display (RTD=17; same=10; TG=21), whereas the
RTD was preferred for detecting change (RTD=24; same=7; TG=17),
giving a sense of direction and motion (RTD=20; same=13; TG=10)
and overall (RTD=25; same=6; TG=17). However, there was a lot
of nuance in these preferences and the fndings difered for each
diagram.
5.4.1 Representing Static Images on Refreshable Tactile Displays.
Comparing the RTD with an equivalent tactile graphic for the
presentation of static images, the most successful image was the
Tai Chi moves, for which the static image was considered easier
to understand (RTD=7, same =1, TG=4). This is because the use of
pin heights was intuitive and aided in understanding of the body
position (11 comments) much more than the use of solid and dashed
lines on the tactile graphics (5 comments). Participant 1 said of the
RTD diagram, “This has heights. It makes it more real.” This fnding
was also refected in responses to the map: “If you’re thinking 3D,
ASSETS ’22, October 23–26, 2022, Athens, Greece Holloway et al.

Table 2: Preferred format for understanding the static image Table 3: Average desired frame rate on the Graphiti display
(‘Understanding’), detecting the change from one image to (secs), and number of participants (n=12) who learned some-
the next (‘Change’), giving a sense of direction and motion thing new from viewing the graphics (in either format) and
(‘Movement’) and preference overall (‘Overall’). thought that RTDs are useful for this type of graphic.

Refreshable Tactile Graphic Frame rate Learned RTD useful?

Preference Same None
Display Graphic
Wave 2.25 10 11
Understanding 17 10 21 0 Waterfall 7.17 10 10
Wave 3 3 6 0 Map n/a 9 11
Waterfall 3 3 6 0 Tai Chi 9.18 6 11
Map 4 3 5 0
Tai Chi 7 1 4 0
5.4.2 Representing Sequences and Moving Images on Refreshable
Change 24 7 17 0
Tactile Displays. The materials for this study were chosen specif-
Wave 12 0 0 0
cally to examine the representation of change and movement, where
Waterfall 6 3 3 0
change was defned as being able to detect what had changed from
Map 0 2 10 0
one image to the next. The RTD was rated as somewhat better in
Tai Chi 6 2 4 0
conveying change (RTD=24, same =7, TG=17).
Movement 20 13 10 5 Movement was defned as giving a sense of the moving object’s
Wave 7 4 1 0 direction and motion. Again, the RTD performed better than tactile
Waterfall 4 6 2 0 graphics for conveying movement (RTD=20, same=13, TG=10). The
Map 3 3 5 1 wave diagram was the most successful in terms of giving a sense
Tai Chi 6 0 2 4 of movement and animation (RTD=7, same=4, TG=1).
Overall 25 6 17 0 “Now this is the power of the Graphiti, because you
Wave 8 1 3 0 can feel it coming, it’s like looking at a motion picture.”
Waterfall 6 1 5 0 [P5]
Map 5 3 4 0 This success may be attributed to the simplicity of the diagram
Tai Chi 6 1 5 0 with changes localised to a small area and minor changes from one
diagram to the next, allowing the sequence to be shown quickly,
with an average preferred speed of 2.25 seconds per frame. Unlike
the buildings are taller. It’s more intuitive” [P5]. Thus, RTDs with the formation of a waterfall, waves moving towards the beach was
variable heights may be particularly well suited to diagrams where a sequence that could be viewed almost in real time.
height or distance is important. The waterfall was also successful in terms of giving a sense of
The Tai Chi graphics were also more successful on the RTD movement on the RTD, although there was only a slight preference
because the larger size enabled a clearer distinction of the nose over the tactile graphics for this purpose (RTD=4, same=6, TG=2).
and eyes (4 comments). While this size diference is due to a design Because the areas of change were larger and slightly more complex,
decision made by the researchers, we believe it refects real-world the waterfall was viewed at a slower frame rate of 7.1 seconds on
practicalities – space is conserved on expensive swell paper whereas average. The direction of refresh from top to bottom matched the
there is no cost per page for display on an RTD. movement of the water and rocks from the top of the waterfall
By contrast, the static RTD was considered less successful than down to the plunge pool, which added to the sense of animation (1
the tactile diagrams for both the wave and waterfall diagrams comment).
(RTD=3, same = 3, TG = 6). The tactile graphics were preferred The Tai Chi movements were the least successful in terms of
because the areas on the diagram were considered more distinct giving a sense of movement. Even though the RTD was preferred
(21 comments) and intuitive (8 comments). The use of a smooth over the tactile graphic (RTD=6, same=0, TG=2), four participants
surface to represent water and sandpaper to represent sand on said that neither format gave a sense of movement and they had not
the collage wave diagram was praised for being tactually distinct thought about the poses as being part of a sequence: “I didn’t think
and meaningful. “This [water] feels gorgeous. This [sandpaper] they were connected. It didn’t occur to me” [P5]. This problem may
represents sand well” [P2]. Additionally, the colour contrast on the have been due to the complexity of the diagram, combined with the
tactile graphics assisted people with low vision. The RTD provided large changes from one diagram to the next: “I can’t really follow
less contrast, particularly the separation of hard rock (level 4) and what’s going on here, there’s too much going on as it changes” [P6].
soft rock (level 3) on the waterfall diagram. Participant 9 suggested The sequence was viewed on the RTD at a slow frame rate of 9.2
using a pattern such as every second dot to represent the soft rock, seconds on average. In response to suggestions from participants,
and two other touch readers agreed that this provided a much better we created two additional diagrams with intermediate poses to
tactual contrast. It is clear that a greater efort must be made in the be viewed between the original diagrams on the RTD. The two
design of RTDs to ensure that there is adequate distinction between participants who viewed these diagrams agreed that it helped to
diferent regions of the graphic. connect the poses and give a better sense of movement.
Animations at Your Fingertips
Movement was defned diferently on the map, using a moving
cursor on the RTD compared with a static route on the tactile
graphic. There was no clear preference for one format over the
other to give a sense of movement on the map (RTG=3, same=3,
TG=5). The experience of following the cursor was enjoyed for
being novel and interactive (5 comments): “I liked following the
dot” [P2]; “it was fun” [P1]. However, it was sometimes difcult
to fnd the cursor (10 comments) and following a static route was
defnitely easier (10 comments).
We asked participants whether the sound of pins refreshing
helped them to interpret the diagrams or movement. Only two
people agreed, stating that the sound helped them to locate the
moving pins. A further fve participants suggested that the sound of
the pins refreshing served more as an indication that something was
changing, “like the sound of a curtain when you switch between
scenes in a play” [P9].
5.4.3 Hand Movements. Hand movements were central to the de-
tection of change. Participants were asked about their hand move-
ments and the researchers made independent observations. Seven
of the twelve participants reported that they had adjusted their
hand movements for accessing graphics on the RTD. When using
the RTD they would hold their hands in the region where they
expected the change to occur so that they could feel the change
under their hands (10 mentions), however this process relied on
memory of the frst image. By contrast, the tactile graphic provided
the opportunity for parallel comparisons by exploring two images
simultaneously (4 mentions), with one hand on one image and the
other hand on a second image that was placed either beside or
underneath the the frst.
The wave and waterfall were considered best for detecting
changes on the RTD because the region of change was predictable
and in a relatively small region. When following the cursor for the
map route, participants would use the fnger pads on both hands,
lined up in the direction they expected the cursor to travel based on
the verbal description. Detecting changes in the Tai Chi poses was
much more difcult on the RTD because the arms and legs were
so spread out that they could not all be felt at once (6 comments).
Stephens reported that it was easier to use a large area of the hand
to detect change on the RTD because the pins are so prominent.
She tended to use the fnger pads to focus on the detail of what is
changing (for simple diagram) or the key area of interest (for more
complex diagrams), combined with the fat of both hands to get an
overview.
It was noted that the swell paper moved around while the partic-
ipants were examining the diagrams; this was particularly problem-
atic for a braille reader with use of only one hand. The RTD’s weight
was advantageous in this context as it was completely stable.
5.4.4 Practical Considerations. Beyond user experiences, practical
considerations are also factors in the likely adoption and use of
RTDs. When asked about the disadvantages of RTDs, participants
mentioned the limited contrast between diferent areas (7 partici-
pants), the low resolution or wide spacing between the dots (n=5),
the high cost (n=5), unreliability with the potential for individual
dots to fail or for the whole device to break down (n=3), the weight
(n=2), the hard feeling of tall dots on the fngers (n=2), and the slow
refresh rate (n=1). While some of these issues may be overcome
ASSETS ’22, October 23–26, 2022, Athens, Greece
with advances in the technology, others are likely to remain as
limitations inherent to the device. We did not ask explicitly about
advantages of the RTD, but participants commented on the fact
that each electronic diagram shown on an RTD has zero associated
material costs, allowing for access to a greater number of images
(5 comments). In terms of usability, the average SUS score was
79.9 (sd=6.2), which can be interpreted as ‘good’ [8]. However, this
result is difcult to interpret as the participants were not operating
the RTD themselves. Note also that scores were adjusted for the
removal of questions 7 and 8 relating to the system integration and
consistency, as several participants were unsure how to interpret
or answer these questions. The SUS is robust for the removal of
one question [46] but results with two questions removed should
be interpreted with caution.
As researchers and accessible format producers, one of the
biggest advantages of the RTD for us was the ability to quickly
and easily change diagrams while discussing them with the touch
reader. For example, we were able to move lines further away from
each other on the map, test whether tactile distinction was easier
or more difcult with a blank line of separation between areas, and
trial diferent dot patterns. By contrast, it was much more dif-
cult for us to go back to the ofce to edit and re-print the tactile
graphics. Four of the participants likewise commented on the value
of quick editing on the RTD: “one of the strengths of refreshable
displays is the ability to refresh and show what is needed at the time
rather than predetermined” [P4]. This immediacy is of particular
value for the blind and low vision community to cater for individu-
alised needs, as there is wide diversity in terms of eye conditions,
co-occurrence with other disabilities, and user skills and experi-
ence with tactile graphics. RTDs therefore address the difculty of
modifying traditional tactile graphics, which has been identifed as
problematic in prior work [2, 49, 64, 73].
5.4.5 Potential Application Areas for Refreshable Tactile Displays.
Participants in the user study reported learning about the subject
matter from viewing the images presented as tactile graphics and on
the RTD. As shown in Table 3, the majority of participants agreed
that they had learned something about the subject matter from all
of the diagrams except Tai Chi, for which only half agreed. Often,
what they learned was basic concepts such as the shape of a wave,
the structure of a waterfall, and an overview of an area that they
had visited but not understood as a whole. For example, “I had sort
of imagined a waterfall as a perfectly smooth curve whereas in
reality it wouldn’t be because of the rocks underneath it” [P8].
As a novel device able to give the frst access to tactile animations
without input from the user, it is not surprising that the RTD was
found to be engaging (15 comments): “I get very excited when
there are toys to play with” [P2]. This could help with student
engagement and therefore learning in the classroom, particularly
as it provides a tangible means for exploring visual narratives [74].
As one participant commented, “I could sit here for ages watching
it. So cool” [P6].
When asked what applications they might have for animated
tactile graphics in their life, all but one participant was able to give
examples. These included maps (n=5); for collaborating with charts
and process diagrams at work (n=3); for fun applications such as
following live sports (n=2), learning dance steps or exercise moves
ASSETS ’22, October 23–26, 2022, Athens, Greece
(n=2), playing games (n=1), watching demonstration videos (n=2)
or watching the weather radar (n=1); and for concept development
such as learning about print notation (n=3), visual communication
such as hand gestures and body language (n=2) and trafc move-
ment for orientation and mobility (n=1). While only one of the
participants was a current student, many of them also suggested
that RTDs would be especially useful for students to gain access to
educational materials (n=7).
6 DISCUSSION
6.1 Potential of Tactile Animations on
Refreshable Tactile Displays
Feedback from participants in our studies makes it clear that there
is a high level of enthusiasm for access to tactile animations. RTDs
can be used to show change over time and for some images they
hold advantages over traditional tactile graphics for conveying
movement, depth and height. However, this was not true for all
images and there were trade-ofs in terms of resolution and textural
properties.
In the best case, the changes to the image were localised and
allowed the touch reader to feel the changes under their hand.
The wave and waterfall were favoured on the RTD because touch
readers knew where to look for the changes, which they could then
follow as animations. Conversely, the Tai Chi diagram was the most
difcult to follow because it was not possible to follow the changes
to the four limbs at the same time.
6.2 Design Considerations for Animated
Graphics Shown on Refreshable Tactile
Displays
The process of creating, using and sharing graphics on an RTD
provided insights into design considerations that are either specifc
to RTDs or more important when using this format.
• Simplify: As the images need to be read quickly and on a
low-fdelity screen, moving tactile images should be very
simple, with less detail than on an equivalent tactile graphic
and much less detail than the original print graphic. This
aligns with O’Modhrain’s guidelines for tactile graphics on
refreshable displays [55].
• Use height to distinguish diferent components within the
diagram. Specifcally, pin height can signify the height of ob-
jects seen from above, or to signify distance by representing
closer elements with higher pins. Pin heights of 0.5mm, 1mm
and 2mm were clearly distinguishable from one another.
• Texture patterns, such as stripes or alternative pins in a
grid, can also be used to clearly distinguish large areas on
the diagram.
• Use blinking pins to direct attention to important areas of
the graphic. This recommendation is in line with prior work
with RTDs and travel route planning [34].
• Ideally, movement should be restricted to one region
of the diagram, especially since simultaneous movement in
multiple regions can be difcult to follow by touch.
• The ideal display duration depends on the complexity of
the diagram and the amount of movement between frames –
Holloway et al.
more complex diagrams require longer display times, and less
change between sequential diagrams allows faster display
times.
• If the display refreshes one line at a time, consider the di-
rection of the refresh when designing sequences of mov-
ing images. Movement in the opposite direction to the refresh
can be difcult to interpret.
• Make use of the ability to edit images while they are be-
ing used. This study benefted from the ability to immedi-
ately adjust diagrams based on feedback from touch read-
ers, either to improve the diagram or to cater to individual
needs and preferences. On-the-spot editing could also be
used to gradually add features (or remove braille labels) as
the learner’s understanding advances. Such an approach has
been proposed as best practice [12] but implementation is
difcult using traditional tactile graphics.
6.3 Considerations for Refreshable Tactile
Display Technology
Our study reinforces that a limitation of current RTDs is their low
resolution. Even with 2,400 pins, less detail could be conveyed
on the Graphiti display compared with an equivalent sized tactile
graphic. Moreover, the wide spacing between pins meant that it
was difcult to accurately perceive areas across the display simulta-
neously. These fndings align with a previous study in which shape
recognition on RTDs was found to improve when the pin spacing
is decreased and pin array size is increased [27].
Labels and accompanying descriptions are always impor-
tant to support understanding of tactile images due to the bottom-up
nature of tactile image consumption [32, 50]. As there is less space
available for braille on RTDs compared with tactile graphics, other
strategies should be built into the devices, such as touch-triggered
audio labels or an accompanying braille display.
Use of high contrast or colour is of great value to touch read-
ers who have some residual vision. White pins against a black
background did not provide sufcient contrast because it was dif-
cult to see whether they were raised. Designers of future devices
should consider further measures to provide visual cues, such as
the coloured lights on the TouchPad Pro.
6.4 Limitations and Future Work
During our study, we wanted participants to focus on exploring
the tactile animations frame by frame in order to gather rich and
detailed feedback. Consequently, we did not ask participants to
operate the RTD themselves or independently access the description
of the graphic. When these tasks are added in an individual home
setting, touch readers may lose their place in the graphic. A key
question is how best to support independent interaction while
maintaining relative positioning of the hands and fngers. Future
usability studies are required to better understand ease of use in
real-world settings.
Another avenue of potential research lies in the drawing and
collaboration capabilities of RTDs. This was of interest to several
of our participants. Although there is prior work in this area [3, 38,
39, 53], the use of touch sensitive RTDs with larger tactile surfaces,
Animations at Your Fingertips
audio feedback, and network capabilities ofers new possibilities
and warrants further research.
Lastly, design guidelines for tactile graphics are an integral tool
for the transcription process. Even in our brief study, we have
already identifed a number of ways graphics need to be designed
diferently for use on an RTD. These guidelines need to be expanded
for a wider range of diagrams and users.
6.5 Conclusion
Given the considerable cost of refreshable tactile displays, inde-
pendent studies like this provide important information to help
potential consumers decide whether an RTD is a worthwhile in-
vestment. To our knowledge, this is the frst study that has sought
to gain understanding of the most suitable use cases and efcacy
of RTDs for tactile animations.
Our results suggest that RTDs have considerable potential for
use in education, orientation and mobility training, and collabora-
tion in the workplace. While their use for entertainment was of
some interest, support for education was seen as much more im-
portant. Unsurprisingly, RTDs demonstrated a particular strength
for conveying motion, however some previously unconsidered ben-
efts also emerged, such as allowing for on-the-spot editing and
customisation for individualised needs. Further work is needed to
obtain a nuanced understanding of what diagrams are most suited
to the RTD.
This work provided the basis for a series of preliminary guide-
lines and design considerations relating to both the graphics being
presented on RTDs and also the design of RTD technology. We
hope that these can be adopted to further explore development of
this important technology as well as determine best practice for
portraying graphic material on these RTDs.
ACKNOWLEDGMENTS
Thanks are extended to the touch readers who contributed their
time and expertise to the user study. Thanks also to Ann Cunning-
ham for sharing her expert advice on diagram design for under-
standing through touch.
REFERENCES
[1] 1931. Blind Can Read Any Book with Aid of Electric Eye. Popular Science Monthly
(1931). https://archive.org/details/blindcanreadanyb0000unse
[2] Dragan Ahmetovic, Niccolò Cantù, Cristian Bernareggi, João Guerreiro, Sergio
Mascetti, and Anna Capietto. 2019. Multimodal Exploration of Mathematical
Function Graphs with AudioFunctions.web. In W4A ’19: Proceedings of the 16th
International Web for All Conference. Article 8, 1–2. https://doi.org/10.1145/
3315002.3332438
[3] Peter Albert. 2006. Math Class: An Application for Dynamic Tactile Graphics.
In ICCHP: International Conference on Helping People with Special Needs (Linz,
Austria). Springer, 1118–1121.
[4] Frances K. Aldrich and Linda Sheppard. 2001. Tactile graphics in school education:
perspectives from teachers. British Journal of Visual Impairment 19, 3 (2001),
93–97. https://doi.org/10.1177/026461960101900303
[5] Daniel Archambault and Helen C. Purchase. 2016. Can animation support the
visualisation of dynamic graphs? Information Sciences 330 (2016), 495–509.
[6] Australian Bureau of Statistics. 2017. Educational Qualifcations in Aus-
tralia. https://www.abs.gov.au/ausstats/abs@.nsf/Lookup/by%20Subject/
2071.0~2016~Main%20Features~Educational%20Qualifcations%20Data%
20Summary%20~65
[7] Paul Ayres, Juan C. Castro-Alonso, Mona Wong, Nadine Marcus, and Fred Paas.
2019. Factors that impact on the efectiveness of instructional animations. Ad-
vances in cognitive load theory: Rethinking teaching (2019), 180–193.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[8] Aaron Bangor, Philip T. Kortum, and James T. Miller. 2009. Determining What
Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of
Usability Studies 4, 3 (2009), 114–123.
[9] Olivier Bau, Ivan Poupyrve, Ali Israr, and Chris Harrison. 2010. Teslatouch:
Electrovibration for Touch Surfaces. In Proceedings of the 23rd Annual ACM
Symposium on User Interface Software and Technology. ACM, New York, NY, USA.
https://doi.org/10.1145/1866029.1866074
[10] Cristian Bernareggi, Dragan Ahmetovic, and Sergio Mascetti. 2019. µ Graph:
Haptic Exploration and Editing of 3D Chemical Diagrams. In ASSETS ’19: The
21st International ACM SIGACCESS Conference on Computers and Accessibility
(Pittsburgh, PA, USA). ACM, New York, NY, USA. https://doi.org/10.1145/3308561.
3353811
[11] Sandra Berney and Mireille Bétrancourt. 2016. Does animation enhance learning?
A meta-analysis. Computers & Education 101 (2016), 150–167.
[12] Donna Bogner, Ben Wentworth, and David Hurd. 2011. Visualizing Science and
Adapted Curriculum Enhancements (ACE): Resource Manual. (2011). https:
//sites.google.com/site/mcrelace
[13] Jens Bornschein, Denise Bornschein, and Gerhard Weber. 2018. Blind Pictionary:
Drawing Application for Blind Users. In CHI Conference on Human Factors in
Computing Systems. ACM, New York, NY, USA. https://doi.org/10.1145/3170427.
3186487
[14] Jens Bornschein and Denise Prescher. 2014. Collaborative Tactile Graphic Work-
station for Touch-Sensitive Pin-Matrix Devices. In TacTT ’14 (Dresden, Germany).
[15] Jens Bornschein, Denise Prescher, and Gerhard Weber. 2015. Collaborative
Creation of Digital Tactile Graphics. In ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS ’15) (Lisbon, Portugal). ACM, New York, NY, USA, 117–
126. https://doi.org/10.1145/2700648.2809869
[16] Jens Bornschein, Denise Prescher, and Gerhard Weber. 2015. Inclusive Production
of Tactile Graphics. In INTERACT 2015: IFIP Conference on Human-Computer
Interaction (Bamburg, Germany). Springer, 80–88.
[17] Braille Authority of North America (BANA). 2010. Guidelines and Standards for
Tactile Graphics. The Braille Authority of North America, USA. http://www.
brailleauthority.org/tg/
[18] Luca Brayda. 2018. Updated Tactile Feedback with a Pin Array Matrix Helps
Blind People to Reduce Self-Location Errors. Micromachines (Basel) 14, 9 (2018),
351. https://doi.org/10.3390/mi9070351
[19] Stephen A. Brewster, Steven A. Wall, Lorna M. Brown, and Eve E. Hoggan.
2008. Tactile Displays. In The Engineering Handbook of Smart Technology for
Aging, Disability, and Independence. John Wiley & Sons, Ltd, 339–352. https:
//doi.org/10.1002/9780470379424.ch18
[20] John Brooke. 1986. SUS: a "quick and dirty" usability scale. Taylor and Francis,
London, England.
[21] C. A. Cook Walker. 2015. Hands On? Hands Of! Future Refections Winter
(2015). https://nfb.org//sites/www.nfb.org/fles/images/nfb/publications/fr/fr34/
1/fr340101.htm
[22] James C. Craig and Carl E. Sherrick. 1982. Dynamic tactile displays. Tactual
perception: A sourcebook (1982), 209–233.
[23] Louise Curtin, Leona Holloway, and Debra Lewis. 2019. Documenting Tactile
Graphicacy. JSPEVI Journal of the South Pacifc Educators in Vision Impairment
12, 1 (2019), 82–98.
[24] Anne Durham. 2021. APH is Ready for a Braille Revolution. American Printing
House for the Blind website (2021). https://www.aph.org/aph-is-ready-for-a-
braille-revolution/
[25] Polly K. Edman. 1992. Tactile Graphics. AFB Press, Arlington, VA, USA.
[26] Danyang Fan, Alexa F. Siu, Wing-Sum Adrienne Law, Raymond Ruihong Zhen,
Sile O’Modhrain, and Sean Follmer. 2022. Slide-Tone and Tilt-Tone: 1-DOF Haptic
Techniques for Conveying Shape Characteristics of Graphs to Blind Users. In
CHI ’22: CHI Conference on Human Factors in Computing Systems. ACM, New
York, NY, USA. https://doi.org/10.1145/3491102.3517790
[27] Nadia Garcia-Hernandez, N. G. Tsagarakis, and D. G. Caldwell. 2011. Feeling
through Tactile Displays: A Study on the Efect of the Array Density and Size on
the Discrimination of Tactile Patterns. IEEE Transactions on Haptics 4, 2 (2011).
[28] Darren Guinness, Annika Muehlbradt, Daniel Szafr, and Shaun K. Kane. 2019.
RoboGraphics: Dynamic Tactile Graphics Powered by Mobile Robots. In ASSETS
’19: The 21st International ACM SIGACCESS Conference on Computers and Acces-
sibility. ACM, New York, NY, USA, 318–328. https://doi.org/doi.org/10.1145/
3308561.3353804
[29] Stanislav Gyoshev, Dimitar Karastoyanov, Nikolay Stoimenov, Virginio Cantoni,
Luca Lombardi, and Alessandra Setti. 2018. Exploiting a Graphical Braille Display
for Art Masterpieces. In Computers Helping People with Special Needs: 16th In-
ternational Conference, ICCHP (Linz, Austria), Klaus Miesenberger and Georgios
Kouroupetroglo (Eds.). Springer International Publishing, Switzerland, Part II,
237–245. https://doi.org/10.1007/978-3-319-94274-2_35
[30] Jefrey Heer and George Robertson. 2007. Animated transitions in statistical data
graphics. IEEE transactions on visualization and computer graphics 13, 6 (2007),
1240–1247.
[31] Morton A. Heller, Jefrey A. Calcaterra, Lynetta L. Burson, and Lisa A. Tyler. 1996.
Tactual picture identifcation by blind and sighted people: Efects of providing
ASSETS ’22, October 23–26, 2022, Athens, Greece
categorical information. Perception & Psychophysics 58 (1996), 310–323. https:
//doi.org/10.3758/BF03211884
[32] R. A. L. Hinton and D. G. Ayres. 1987. The development of tactile diagrams for
blind biology students. Journal of Visual Impairment & Blindness 81, 1 (1987),
24–25.
[33] Tim N Höfer and Detlev Leutner. 2007. Instructional animation versus static
pictures: A meta-analysis. Learning and instruction 17, 6 (2007), 722–738.
[34] Mihail Ivanchev, Francis Zinke, and Ulrike Lucke. 2014. Pre-journey Visualization
of Travel Routes for the Blind on Refreshable Interactive Tactile Displays. In
ICCHP: International Conference on Computers Helping People with Special Needs
(Paris, France), Klaus Miesenberger, Deborah Fels, Dominique Archambault,
Petr Peňáz, and Wolfgang L. Zagler (Eds.), Vol. Part II. Springer International,
Switzerland, 81–88.
[35] Jingun Jung, Sunmin Son, Sangyoon Lee, Yeonsu Kim, and Geehyuk Lee. 2021.
ThroughHand: 2D Tactile Interaction to Simultaneously Recognize and Touch
Multiple Objects. In CHI ’21: Proceedings of the 2021 CHI Conference on Human
Factors in Computing Systems. ACM, New York, NY, USA. https://doi.org/10.
1145/3411764.3445530
[36] Seondae Kim, Yeongil Ryu, Jinsoo Cho, and Eun-Seok Ryu. 2019. Towards
Tangible Vision for the Visually Impaired through 2D Multiarray Braille Display.
Sensors 19, 23 (2019). https://doi.org/10.3390/s19235319
[37] Makoto Kobayashi, Yoshiki Fukunaga, and Shigenobu Shimada. 2018. Basic
Study of Blind Football Play-by-Play System for Visually Impaired Spectators
Using Quasi-Zenith Satellites System. In ICCHP: International Conference on
Computers Helping People with Special Needs, Karl Miesenberger and Georgios
Kouroupetroglou (Eds.). Springer, Switzerland, 23–27.
[38] Makoto Kobayashi and Tetsuya Watanabe. 2004. Communication System for
the Blind Using Tactile Displays and Ultrasonic Pens – MIMIZU. In ICCHP:
International Conference on Computers Helping People with Print Disabilities (Paris,
France). Springer, Switzerland, 731–738.
[39] Christopher Alexander Kopel. 2021. Accessible SVG Charts with AChart Creator
and AChart Interpreter. Thesis.
[40] Martin Kunz. c.1890. Abbildungen fur Blinde (Pictures for the Blind. Blind Institute,
Illzach, Germany.
[41] Ki-Uk Kyung, Seung-Chan Kim, and Dong-Soo Kwon. 2007. Texture display
mouse: vibrotactile pattern and roughness display. IEEE/ASME Transactions on
Mechatronics 12, 3 (2007), 356–360. https://doi.org/10.1109/TMECH.2007.897283
[42] Steve Landau. 2013. An Interactive web-based tool for sorting textbook images
prior to adaptation to accessible format: Year 1 Final Report. (2013). http://
diagramcenter.org/decision-tree.html
[43] Fabrizio Leo, Tania Violin, Alberto Inuggi, Angelo Raspagliesi, Elisabetta Capris,
Elena Cocchi, and Luca Brayda. 2019. Blind Persons Get Improved Sense of
Orientation and Mobility in Large Outdoor Spaces by Means of a Tactile Pin-Array
Matrix. In CHI ’19 Workshop on Hacking Blind Navigation (Glasgow, Scotland).
ACM, New York, NY, USA.
[44] Daniele Leonardis, Loconsole Claudio, and Antonio Frisoli. 2017. A survey on
innovative refreshable braille display technologies. In International Conference
on Applied Human Factors and Ergonomics. Springer, 488–498.
[45] Vincent L’evesque, Grégory Petit, Aude Dufresne, and Vincent Hayward. 2012.
Adaptive level of detail in dynamic, refreshable tactile graphics. In IEEE Haptics
Symposium (HAPTICS) (Vancouver, BC, Canada). IEEE, 1–5. https://doi.org/10.
1109/HAPTIC.2012.6183752
[46] James R. (Jim) Lewis and Jef Sauro. 2017. Can I Leave This One Out? The Efect
of Dropping an Item From the SUS. Journal of Usability Studies 13, 1 (2017),
38–46.
[47] Sebastian Lieb, Benjamin Rosemeier, Thorsten Thormählen, and Knut Buettner.
2020. Haptic and Auditive Mesh Inspection for Blind 3D Modelers. In ASSETS
International SIGACCESS Conference on Computers and Accessibility (Greece
(virtual event)). ACM, New York, NY, USA. https://doi.org/doi.org/10.1145/
3373625.3417007
[48] Claudia Loitsch and Gerhard Weber. 2012. Viable Haptic UML for Blind People.
In ICCHP: International Conference on Helping People with Special Needs (Linz,
Austria), Klaus Miesenberger, Arthur Karshmer, Petr Penaz, and Wolfgang L.
Zagler (Eds.). Springer, Switzerland, Part II, 509–516.
[49] David McGookin, Euan Robertson, and Stephen A. Brewster. 2010. Clutching at
straws: using tangible interaction to provide non-visual access to graphs. In CHI
’10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
ACM, New York, NY, USA, 1715–1724. https://doi.org/10.1145/1753326.1753583
[50] S. Millar. 1999. Memory in touch. Psicothema 11, 4 (1999), 747–767.
[51] Tatsuo Motoyoshi, Sota Mizushima, Kei Sawai, Takumi Tamamoto, Hiroyuki
Masuta, Ken’ichi Koyanagi, and Toru Oshima. 2018. Prototype Development
of a Shape Presentation System Using Linear Actuators. In ICCHP: Interna-
tional Conference on Computers Helping People with Special Needs (Linz, Austria),
Karl Miesenberger and Georgios Kouroupetroglo (Eds.). Springer International,
Switzerland, Part II, 226–230. https://doi.org/10.1007/978-3-319-94274-2_31
[52] Rahul Kumar Namdev and Pattie Maes. 2015. An interactive and intuitive stem
accessibility system for the blind and visually impaired. In PETRA: International
Conference on PErvasive Technologies Related to Assistive Environments (Corfu,
Holloway et al.
Greece). ACM, 1–7. https://doi.org/10.1145/2769493.2769502
[53] Atsushi Nishi and Ryoji Fukuda. 2006. Graphic Editor for Visually Impaired
Users. In ICCHP: International Conference on Computers Helping People with
Special Needs (Linz, Austria), Klaus Miesenberger, Joachim Klaus, Wolfgang L.
Zagler, and Arthur I. Karshmer (Eds.). Springer, Switzerland, 1139–1146.
[54] Hiroyuki Ohshima, Makoto Kobayashi, and Shigenobu Shimada. 2021. Devel-
opment of Blind Football Play-by-play System for Visually Impaired Specta-
tors: Tangible Sports. In 2021 CHI Conference on Human Factors in Comput-
ing Systems (Yokohama, Japan (virtual event)). ACM, New York, NY, USA, 1–6.
https://doi.org/10.1145/3411763.3451737
[55] Sile O’Modhrain, Nicholas A. Giudice, John A. Gardner, and Gordon E. Legge.
2015. Designing Media for Visually-Impaired Users of Refreshable Touch Displays:
Possibilities and Pitfalls. IEEE Transactions on Haptics 8, 3 (2015), 248–57. https:
//doi.org/10.1109/TOH.2015.2466231
[56] Orbit Research. 2016. Graphiti® - a Breakthrough in Non-Visual Access to All
Forms of Graphical Information. http://www.orbitresearch.com/product/graphiti/
[57] Grégory Petit, Aude Dufresne, Vincent Levesque, Vincent Hayward, and Nicole
Trudeau. 2008. Refreshable Tactile Graphics Applied to Schoolbook Illustrations
for Students with Visual Impairment. In SIGACCESS Conference on Computers
and Accessibility - ASSETS ’08 (Nova Scotia, Canada). ACM, 89–96.
[58] Mahika Phutane, Julie Wright, Brenda Veronica Castro, Lei Shi, Simone R. Stern,
Holly M. Lawson, and Shiri Azenkot. 2021. Tactile Materials in Practice: Under-
standing the Experiences of Teachers of the Visually Impaired. ACM Transactions
on Accessible Computing (TACCESS) (2021). https://doi.org/10.1145/3508364
[59] Christopher Power. 2006. On the Accuracy of Tactile Displays. In ICCHP: Interna-
tional Conference on Computers Helping People with Special Needs (Linz, Austria),
Klaus Miesenberger, Joachim Klaus, Wolfgang L. Zagler, and Arthur I. Karshmer
(Eds.). Springer, Switzerland, 1155–1162.
[60] Denise Prescher and Gerhard Weber. 2017. Comparing Two Approaches of
Tactile Zooming on a Large Pin-Matrix Device. In INTERACT 2017: Human-
Computer Interaction – INTERACT 2017, Vol. 10513. Springer, 173–186. https:
//doi.org/10.1007/978-3-319-67744-6_11
[61] Hrishikesh Rao and Sile O’Modhrain. 2019. Multimodal Representations of
Complex Spatial Data. In CHI Conference on Human Factors in Computing Systems
(Glasgow, Scotland). ACM.
[62] Dorothea Reusser, Espen Knoop, Roland Siegwart, and Paul Beardsley. 2017. Feel-
ing Fireworks. In UIST ’17 Symposium on User Interface Software and Technology
(Québec City, Canada). ACM.
[63] Dorothea Reusser, Espen Knoop, Roland Siegwart, and Paul Beardsley. 2019.
Feeling Fireworks: An Inclusive Tactile Firework Display. In CHI Conference on
Human Factors in Computing Systems (Glasgow, Scotland). ACM, New York, NY,
USA.
[64] Patrick Roth, Hesham Kamel, Lori Petrucci, and Thierry Pun. 2002. A Compar-
ison of Three Nonvisual Methods for Presenting Scientifc Graphs. Journal of
Visual Impairment & Blindness 96, 6 (2002), 420–428. https://doi.org/10.1177/
0145482X0209600605
[65] Round Table on Information Access for People with Print Disabilities Inc.
2005. Guidelines on Conveying Visual Information. Round Table on Informa-
tion Access for People with Print Disabilities Inc., Lindisfarne, Tasmania, Aus-
tralia. https://printdisability.org/guidelines/guidelines-on-conveying-visual-
information-2005/
[66] Jonathan Rowell and Simon Ungar. 2003. The world of touch: An international
survey of tactile maps. Part 1: production. British Journal of Visual Impairment
21, 3 (2003), 98–104. https://doi.org/10.1177/026461960302100303
[67] Bernhard Schmitz and Thomas Ertl. 2012. Interactively Displaying Maps on a
Tactile Graphics Display. In SKALID 2012 Spatial Knowledge Acquisition with
Limited Information Displays (Germany). 13–18.
[68] Alexa F. Siu, Son Kim, Joshua A. Miele, and Sean Follmer. 2019. shapeCAD:
An Accessible 3D Modelling Workfow for the Blind and Visually-Impaired Via
2.5D Shape Displays. In The 21st International ACM SIGACCESS Conference on
Computers and Accessibility (Pittsburgh, USA). ACM, New York, NY, USA, 342–
354. https://doi.org/10.1145/3308561.3353782
[69] Larry Skutchan. 2016. Transforming Braille. http://transformingbraille.org/blog/
[70] Barbara Tversky, Julie Bauer Morrison, and Mireille Betrancourt. 2002. Animation:
can it facilitate? International Journal of Human-Computer Studies 57, 4 (2002),
247–262.
[71] Cheng Xu, Ali Israr, Ivan Poupyrev, Olivier Bau, and Chris Harrison. 2011. Tactile
display for the visually impaired using TeslaTouch. In CHI Conference on Human
Factors in Computing Systems (Vancouver, BC, Canada). ACM, New York, NY,
USA, 317–322. https://doi.org/10.1145/1979742.1979705
[72] Wenzhen Yang, Jinpen Huang, Ruirui Wang, Wen Zhang, Haitao Liu, and Jianliang
Xiao. 2021. A Survey on Tactile Displays For Visually Impaired People. IEEE
Transactions on Haptics 14, 4 (2021), 712–721.
[73] Koji Yatani, Nikola Banovic, and Khai N. Truong. 2012. SpaceSense: repre-
senting geographical information to visually impaired people using spatial
tactile feedback. In CHI ’12: Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems. ACM, New York, NY, USA, 415–424. https:
//doi.org/10.1145/2207676.2207734
Animations at Your Fingertips
[74] Bilal Yousuf and Owen Conlan. 2018. Supporting Student Engagement Through
Explorable Visual Narratives. IEEE Transactions on Learning Technologies 11, 3
(2018), 307–320. https://doi.org/10.1109/TLT.2017.2722416
[75] Limin Zeng, Mei Miao, and Gerhard Weber. 2015. Interactive Audio-haptic Map
Explorer on a Tactile Display. Interacting with Computers 27, 4 (2015), 413–429.
https://doi.org/10.1093/iwc/iwu006
[76] Limin Zeng, Gerhard Weber, and Ulrich Baumann. 2012. Audio-haptic you-are-
here maps on a mobile touch-enabled pin-matrix display. In IEEE International
Workshop on Haptic Audio Visual Environments and Games (HAVE 2012) (Munich,
Germany). IEEE. https://doi.org/10.1109/HAVE.2012.6374428
A VERBALISED DESCRIPTIONS OF SAMPLE
GRAPHICS
These descriptions were provided verbally to accompany the sample
graphics as they were explored tactually by touch readers in the
Evaluation phase. Pauses were inserted while the graphic was being
explored, and additional guidance was given if needed to ensure
that the key features had been identifed.
A.1 Wave
Collage: This diagram shows how waves form as they come in
towards the beach. The diagram is a cross section showing the
water (raised smooth area) and sand (sandpaper on the lower right).
RTD: This is the frst of seven diagrams showing a wave as it
forms and comes in towards the beach. The diagram is a cross
section showing the water (highest dots) and sand (lower dots on
the lower right, in a low triangular shape).
Both formats: The wind is blowing from the ocean (left) to the
beach (right). Starting from the left, the frst waves are beginning to
form as low lumps. The air pressure creates the waves, with upper
layer of air sink on either side of the wave. The wave gets higher as
the sea foor gets higher. Friction from the beach slows the lower
part of the wave but the upper part continues to move forward,
forming a curve. After the wave breaks, swash moves up the beach
and backwash moves down.
A.2 Waterfall
Both formats: This is the frst of fve diagrams showing the creation
of a waterfall over time.
Swell paper: The diagram shows a cross section of the earth, with
water at the top (horizontal stripes). The water is fowing down
from left to right. There is a triangle of hard rock on the left, just
below the water. It is shown as solid fll. Below the hard rock there
is a large area of soft rock, shown with textured fll.
RTD: The diagram shows a cross section of the earth, with water
at the top (low dots). The water is fowing down from left to right.
There is a triangle of hard rock on the left, just below the water. It
is shown with high dots. Below the hard rock there is a large area
of soft rock, shown with slightly lower dots.
Both formats: Waterfalls are often formed where a layer of harder
rock overlays a layer of softer rock. Diagram 2: As the river passes
over the softer rock, it is able to erode it at a faster rate, forming a
step in the river bed. The water is lower on the right now, where
the soft rock has eroded. Diagram 3: Erosion continues, and cuts
underneath the hard rock. Diagram 4: The tip of the harder rock
has collapsed and fallen into the plunge pool, because there was not
enough support underneath it. Diagram 5: The rocks and boulders
ASSETS ’22, October 23–26, 2022, Athens, Greece
under the plunge pool and the hydraulic action have further eroded
the plunge pool and notch.
A.3 Map
Note that all participants were familiar with the area depicted on
the map, but did not know all of the buildings.
Both formats: This is a map of Federation Square, seen from
above.
Swell paper: Roads are shown as solid lines. Buildings are shown
as flled shapes. Stairs are shown as striped blocks. A walking route
is shown with a dotted spur wheel line.
RTD: Roads are shown as low lines. Buildings are shown as taller
blocks. The position I am describing will be indicated by a blinking
pin.
Both formats: Flinders street runs along the top of the map,
Swanston street is to the left, Russel street is on the right, and
the Yarra River at the bottom.
Young and Jackson’s pub is at the top left corner. This is where the
famous painting, Chloe, hangs. Moving right across Swanston street,
we fnd St Paul’s Cathedral. Further along Flinders street we come to
Hosier Lane, famous for its street art. After Hosier Lane is the Forum
Theatre. Moving down across Flinders street, there is NGV Australia.
Below the NGV is ZINC, a venue for events like weddings. Moving
left from ZINC we come to BMW Edge, a multi-purpose theatre
that is open to the atrium (the space to the left of the NGV). Moving
left from the atrium we come to ACMI, the Australian Centre for
the moving image. On the corner is the Melbourne Visitor Centre,
which is currently undergoing construction to form an entrance
to the new Town Hall underground train station. Crossing over
Swanston street, we come to Flinders street station. In the bottom
left corner, there is the Transport Hotel.
When you are ready, let’s trace the route on the map. The route
goes from Flinders Street Station to the replacement buses for the
Frankston, Pakenham and Cranbourne lines. Let’s start at the corner
steps of Flinders street station. First we head to the pedestrian
crossing at the tram stop on Swanston street. We cross the road
from Flinders street station to Federation square. The Information
centre is on our left. We follow around the information centre to
the footpath on Flinders street. We pass by ACMI and the atrium
on our right. We turn right when we reach Russell street. The bus
stop is behind NGV Australia.
A.4 Tai Chi
Both formats: This series of three diagrams shows a sequence of
movements for Tai Chi. There are three fgures per page, progress-
ing from left to right. Each fgure has a rectangular body with arms,
legs, feet, and a head with eyes and a nose. There may be one or
two eyes, depending on the angle of the head. There is a dotted line
on the body, representing shirt buttons down the centre front of
the body.
Swell paper: Solid lines for the arms and legs indicate that they
are positioned in front of the body. Dotted lines for the arms and
legs indicate that they are positioned behind the body.
RTD: The body in the centre is at height 2. Lower pins indicate
that the arms or legs are further away than the body. Higher pins
indicate that the arms of legs are closer than the body.
ASSETS ’22, October 23–26, 2022, Athens, Greece Holloway et al.

B EVALUATION PARTICIPANT PROFILES

Table 4: Profles for participants in the evaluation study, giv-
ing their age range, level of blindness (totally blind or legally
blind), onset of vision impairment, and self-rated level of
competency using tactile graphics.

ID age blindness onset TG competency

1 40s total acquired profcient
2 40s legal acquired beginner
3 50s total congenital beginner
4 50s total congenital expert
5 50s legal congenital beginner
6 20s legal acquired profcient
7 20s legal acquired expert
8 60s total congenital expert
9 60s total congenital profcient
10 30s legal acquired profcient
11 40s total congenital profcient
12 50s legal acquired beginner
 Quantifying Touch: New Metrics for Characterizing What
Happens During a Touch
Junhan Kong Mingyuan Zhong Jacob O. Wobbrock
junhank@uw.edu James Fogarty wobbrock@uw.edu
The Information School, myzhong@cs.washington.edu The Information School,
University of Washingtonl jfogarty@cs.washington.edu University of Washington
Seattle, WA, USA Paul G. Allen School of Computer Seattle, WA, USA
Science & Engineering DUB Group,
University of Washington
Seattle, WA, USA
ABSTRACT 1 INTRODUCTION
Measures of human performance for touch-based systems have With the proliferation of modern touch-enabled devices such as
focused mainly on overall metrics like touch accuracy and target smartphones, tablets, watches, and other surfaces, touch input has
acquisition speed. But touches are not atomic—they unfold over become perhaps the most prevalent form of input to computer
time and space, especially for users with limited fine motor systems. As with mouse pointing and text entry [37, 48, 49, 59, 60],
function, for whom it can be difficult to perform quick, accurate understanding human performance with touch-based systems can
touches. To gain insight into what happens during a touch, we reveal opportunities for improved device and interface design.
offer 15 target-agnostic touch metrics, most of which have not To date, most measures of human performance with touch-based
been mathematically formalized in the literature. They are touch systems have mainly focused on overall performance like accuracy
direction, variability, drift, duration, extent, absolute/signed area and target acquisition speed (e.g., [3, 4, 19, 23, 24, 44]). However,
change, area variability, area deviation, area extent, absolute/signed a touch is not an atomic event; it unfolds over space and time
angle change, angle variability, angle deviation, and angle extent. [43]. Therefore, examining what happens during a touch might
These metrics regard a touch as a time series of ovals instead of yield insights into users’ touch behaviors and any underlying
a mere (G, ~) coordinate. We provide mathematical definitions causes of touch inaccuracy. It might also afford designers the
and visual depictions of our metrics, and consider policies for chance to improve device or interface designs, or enable software to
calculating our metrics when multiple fingers perform coincident usefully adapt to users’ touch behaviors at runtime. Particularly for
touches. To exercise our metrics, we collected touch data from accessibility, measuring what happens during a touch could help
27 participants, 15 of whom reported having limited fine motor designers make touch interactions more accessible. For example, if
function. Our results show that our metrics effectively characterize a system measures how much a user “drifts” on the screen while
touch behaviors including fine-motor challenges. Our metrics their finger is down before lifting, the system can enlarge their
can be useful for both understanding users and for evaluating widgets at runtime to accommodate.
touch-based systems to inform their design. A similar idea motivated prior work by MacKenzie et al. [37],
who formulated measures for what happens during a mouse
CCS CONCEPTS pointing movement, rather than just relying on overall speed
• Human-centered computing → Touch screens. and accuracy to understand pointing performance (Figure 1).
MacKenzie et al. formulated seven new accuracy measures that
KEYWORDS captured properties of the cursor’s path of movement. Building
Touch input, touch screens, touch metrics, human performance, on this work, Keates et al. [33] used MacKenzie et al.’s metrics
limited fine motor function. with people with motor impairments, introducing six new path
measures along the way. Hwang et al. [26] built upon this work
ACM Reference Format:
yet further, conducting submovement analyses of motion-impaired
Junhan Kong, Mingyuan Zhong, James Fogarty, and Jacob O. Wobbrock. 2022.
users with 12 additional metrics. In each case, examining the path
Quantifying Touch: New Metrics for Characterizing What Happens During
a Touch. In The 24th International ACM SIGACCESS Conference on Computers of movement by formalizing new quantitative metrics revealed
and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, why overall speed and accuracy differences emerged.
New York, NY, USA, 13 pages. https://doi.org/10.1145/3517428.3544804 Inspired by this prior work, we aim to understand why human
performance with touch screens emerges the way it does. Certainly,
some of our touch metrics have been previously used in individual
This work is licensed under a Creative Commons Attribution International 4.0 License. studies or in ad hoc ways, as was the case for some of the
aforementioned cursor measures. But few of our metrics have
ASSETS ’22, October 23–26, 2022, Athens, Greece been formalized mathematically so as to be operationalized in a
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
reusable, generalizable way. Furthermore, our metrics have not,
https://doi.org/10.1145/3517428.3544804 to the best of our knowledge, been applied to the study of touch
behaviors by people with and without limited fine motor function.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 1: (a) MacKenzie et al. [37] defined metrics for quantifying what happens during a mouse pointing movement, revealing
underlying causes of speed and accuracy outcomes. For example, the variability (”wiggliness”) of a movement can be quantified.
(b) In an analogous fashion, we formulate 15 metrics for quantifying what happens during a touch, which is not atomic but
unfolds over time and space, even if only briefly. For example, a user might land outside a target, then slide into the target and
lift, creating touch ovals with various areas, orientations, and dimensions along the way.
To these ends, we formalized 15 target-agnostic touch metrics,
many of which are analogues inspired by MacKenzie et al. [37].
They are: touch direction, variability, drift, duration, extent,
absolute/signed area change, area variability, area deviation, area
extent, absolute/signed angle change, angle variability, angle
deviation, and angle extent. Instead of treating a touch as an atomic
event or single (G, ~) coordinate, our metrics regard a touch as a
time series of ovals approximating a finger’s contact area from
finger-down to finger-up. Thus, a “touch process” [43] can contain
movement of the touch oval centroid and change of the touch oval
size, orientation, and shape. For each of our 15 metrics, we provide
a mathematical formula and intuitive description. We also include
visual depictions of some of our metrics to convey their purposes.
In addition, ambiguity arises when multiple fingers touch
concurrently. Although our metrics are defined on a per-finger
basis, we address the issue of how to handle multiple concurrent
touches. Specifically, we discuss three policies for defining the touch
process in multi-finger situations. The three policies are first-down,
longest-lived, and sum-of-all. These three policies converge to the
same definition in the case of a single-finger touch. Our study
results also show that all three policies are viable and result in
similar conclusions, even if their specific values change.
To put our metrics through their paces, we conducted a user
study to collect touch data on a Microsoft Perceptive Pixel (PPI)
interactive tabletop display. Our study collected complete touch
oval information from 27 participants, 15 of whom reported
having limited fine motor function with different specific fine
motor challenges. We then computed all 15 of our touch metrics
on our data and ran both descriptive and inferential statistics.
Our results show that our metrics effectively characterize users’
touch behaviors. Specifically, we found that our metrics uncover
differences among people with no, moderate, and severe fine
motor challenges. We also found that each metric correlates with a
different subset of specific fine motor challenges. Additionally, we
found that for multi-finger touches, the longest-lived policy and the
sum-of-all policy yielded the same conclusions in characterizing
users’ touch processes, while the sum-of-all policy magnifies the
differences between user groups.
Importantly, none of our 15 metrics are “target-aware” [2, 11];
rather, our metrics are “target-agnostic” [56], calculable using only
the touch data itself without knowledge of target locations or
dimensions. Being target-agnostic is an immensely useful property
Kong, et al.
because, in practice, it is generally infeasible for software to know
what a target is and whether a user was intending to touch [11, 12]
(This challenge becomes trivial in an artificial testbed, but not at
runtime with software “in the wild.”) Despite being target-agnostic,
our touch metrics can nonetheless explain why overall touch
accuracy is low or high. Thus, our metrics can help understand
users’ touch abilities and inform better designs of touch-based
systems that can adapt to these abilities.
The major contributions of this work are:
• Formalizing 15 target-agnostic touch metrics to characterize
what happens during a touch;
• Proposing and investigating three policies for handling
inadvertent multi-finger situations during a touch;
• Exercising our metrics in a formal study of 15 participants
with and 12 participants without limited fine motor function.
2 RELATED WORK
Our work attempts to “pry the cover” off a touch to characterize
what happens “inside” it. As noted above, prior work by MacKenzie
et al. [37] pried the cover off mouse pointing to understand more
than just speed and accuracy, but how pointing unfolds over time
and space. To this end, MacKenzie et al. devised accuracy measures
that include movement variability, offset, error, axis crossings,
and more for evaluating mouse pointing, revealing differences
among input devices. Keates et al. [33] extended MacKenzie et al.’s
measures to better understand pointing by people with limited fine
motor function. Through a study of mouse movements by users
with motor impairments, Keates et al. characterized movement
distributions and submovement characteristics, finding significant
differences between people with and without upper-body motor
impairments. Hwang et al. [26] further analyzed submovements by
people with motor impairments. These papers inspired the current
work1 and point to the value of characterizing input processes,
not just outcomes. That said, none of the metrics from MacKenzie
et al. [37], Keates et al. [33], or Hwang et al. [26] can be applied
directly to touch. For example, most of their metrics require a
known target and a task axis between the mouse position and
1 The current work also builds upon an ACM ASSETS 2021 poster [34]. This offered
only a subset of metrics and exercised only four of them on a previously collected data
set [14] that lacked complete touch-oval information (e.g., no oval major or minor
axes, no areas or orientations). The current work formulates more metrics, collects an
original data set, and exercises all metrics on complete touch-oval information.
Quantifying Touch: New Metrics for Characterizing What Happens During a Touch
target center; moreover, a mouse cursor only occupies a single (G, ~)
position at one time, whereas a touch occupies an area, abstracted
as an oval. Therefore, we developed custom metrics pertaining to
touch; furthermore, our metrics are target-agnostic, making them
practical to deploy in actual systems [2, 11, 12, 56].
Holz and Baudisch [23, 24] sought to understand factors affecting
touch performance by modeling how users perceive their own touch
input, specifically their intended versus actual point of contact.
They did not formulate a set of metrics for quantifying touch, as
we do here, but they devised the generalized perceived input point
model [23] and the projected center model [24] to help characterize
perceptual aspects of the touch process. Their work examined users’
mental models through user studies, interviews, and analyses based
on a series of pointing studies, identifying sources of inaccuracy
in touch devices as a result of the parallax between the top and
bottom of the finger. Bi and Zhai [6], using a different approach,
conceptualized touch input in target selection as an uncertain
process, and proposed a Bayesian touch criterion to statistically
model the probabilistic distribution of touch selections. In other
work, Bi et al. [5] proposed FFitts law to model finger-touch based
inputs on touch screens, which reflects relative precision due to
the speed-accuracy tradeoff and captures the absolute precision of
finger touch.
Beyond understanding, characterizing, and modeling touch
input, researchers have sought to improve touch performance
through the invention and evaluation of new touch input techniques.
In early work, Potter et al. [45] evaluated three strategies for target
selection on touch screens, finding lift-off with an offset cursor to be
most accurate. Sears and Shneiderman [47] evaluated a stabilization
technique for improving touch accuracy. More recently, Vogel and
Baudisch presented Shift [50], which uses an offset lens to magnify
touch targets when multiple targets are beneath the finger. Cao et
al. proposed ShapeTouch [7] to utilize touch regions and motion to
infer virtual contact forces, enabling pseudo-force-based interaction
techniques. Benko et al. [3] designed high-precision touch selection
techniques by utilizing two fingers that together dynamically adjust
the control-display (C-D) gain. Harrison et al. [21] used machine
learning classification to distinguish the sounds made by finger tips,
pads, nails, and knuckles, extending the “interaction vocabulary”
of touch-based systems. Holz and Baudisch [25] went even further
with Fiberio to detect fingerprints on a touch table with a custom
fiber optic plate. On the output side, Wigdor et al. [54] created
Ripples, which are visualizations accompanying users’ touches to
provide vital feedback and reduce touch errors. These are just some
of the many interaction techniques developed to improve human
performance with touch-based systems. By contrast, our metrics
contribute to understanding human performance at a fine-grained
level. Indeed, they could be used to explain why and how some of
these touch-improvement techniques succeed.
Some work has also attempted to improve touch accuracy,
not through novel interaction techniques, but through advanced
machine learning. For example, Weir et al. [52, 53] used Gaussian
process regression to improve touch accuracy. Kumar et al. [35]
collected capacitive touch images and trained convolutional neural
networks (CNNs) to improve touch accuracy. Mayer et al. [39] used
CNNs to infer finger orientation from capacitive touch images.
The accessibility of touch-based systems for people with
disabilities has also been a focus of prior work, particularly because
ASSETS ’22, October 23–26, 2022, Athens, Greece
touch-based systems are often inaccessible [8]. For example,
Findlater et al. [13] studied users’ target acquisition performance
on touch screens, finding that touch accuracy and usability issues
still persist for people with limited fine motor function. To improve
systems’ accessibility, Kane et al. [28] used accessible touch and
gesture interactions in SlideRule to create the first finger-driven
screen reader. Kane et al. [30] also developed Access Overlays for
interactive tabletops—software-based “layers” that impose various
interaction techniques on displayed content to make it more
accessible to blind users. This approach was extended to physical
documents using computer vision in Access Lens [29]. Similarly,
Guo et al. [20] used computer vision and crowdsourcing in StateLens
to model dynamic touch screen interfaces, and provide guidance
for blind users while using existing inaccessible touch screens.
Using hardware, Kane et al. [31] created Touchplates to provide
physical-tactile landmarks when interacting with tabletop displays.
In a related fashion, Zhang et al. [61] created Interactiles to provide
physical-tactile interactions on smartphones. Wacharamanotham
et al. [51] presented Swabbing for improving target selection
accuracy for users with tremor. A similar technique proposed by
Mertens et al. [40] called TRABING enables users to continuously
move over desired targets, improving accuracy for people with
tremor. Guerreiro et al. [19] studied characteristics of mobile
touch screen interfaces to provide tools for better interface design
for motor-impaired users. Montague et al. [41, 42] investigated
shared user models and user interface adaptivity for making
touch applications more accessible, and examined motor-impaired
touchscreen interactions in the wild. Sarcar et al. [46] proposed the
Touch-WLM model for making predictions of how users with given
abilities enter text, then optimized touch screen layouts for users
with tremor and dyslexia. Mott et al. created both Smart Touch [43]
and Cluster Touch [44] to resolve intended touch locations by people
with motor impairments through template matching of segmented
“frames” of the touch process and clustering, respectively. Cluster
Touch was also directed towards people incurring situational
impairments [55] due to walking; similarly, Goel et al. created
WalkType [17] and ContextType [18] for improving touch accuracy
on smartphone keyboards while walking. Although some of our
metrics (e.g., touch drift) were used in an ad hoc fashion in prior
work, they were not formalized mathematically, leaving them open
to variations of implementation and interpretation.
Relatedly, researchers have analyzed touch behaviors to detect
specific diseases that relate to impaired motor function. Mastoras
et al. [38] extracted statistical features from keystroke sequences to
detect depressive disorder. Giancardo et al. [16] and Arroyo-Gallego
et al. [1] used hold time (e.g., the interval between key press and
release) and flight time (e.g., the interval between one key’s release
and the next key’s press) to distinguish early-stage Parkinson’s
disease (PD). Iakovakis et al. [27] used normalized flight time and
normalized pressure to detect declines in fine motor function in
early PD patients. Kay et al. [32] created PVT-Touch to implement
the Psychomotor Vigilance Task [10] as a robust clinical tool for
assessing issues related to sleep loss. These projects demonstrate
the ability to use touch behaviors for disease- or condition-specific
diagnoses and tracking. In contrast, our touch metrics are general,
capable of illuminating touch behaviors that could be useful in a
variety of applications (e.g., we provide examples in Section 7).
ASSETS ’22, October 23–26, 2022, Athens, Greece Kong, et al.

Variability Low High High High Very High

Drift Medium Low High Low High
Extent Medium Medium High High High
Table 1: Example traces of touch oval centroids and their
touch variability, drift, and extent. This table was inspired by
Figure 2: (a) A sequence of = touch ovals %0, ..., %=−1 from Figure 5 from MacKenzie et al. [37].
finger-down to finger-up along a time axis. (b) The first touch
oval in the sequence has centroid (G 0, ~0 ), angle \ 0 , major mathematical definition uses radians.
axis 0 0 , and minor axis 1 0 . The last touch oval has centroid ⎧ tan −1 [(~=−1 − ~0 )/(G=−1 − G 0 )], if G 0 < G=−1 ⎫
(G=−1, ~= −1 ), angle \=−1 , major axis 0=−1 , and minor axis 1=−1 .
⎪
⎪ ⎪
⎪
⎨ c/2, if G 0 = G=−1 , ~0 > ~=−1 (straight up)
⎪
⎪ ⎪
⎬
⎪
Direction =
3 OUR TOUCH METRICS ⎪
⎪ 3c/2, if G 0 = G=−1 , ~0 < ~=−1 (straight down) ⎪
⎪
⎪ undefined, if G 0 = G=−1 , ~0 = ~=−1 (no movement)
⎪ ⎪
In this section, we formalize 15 touch metrics for characterizing
⎪
⎩ ⎭
what happens during a touch. Although some of our metrics are ∈ [0, 2c)
admittedly not new (e.g., touch duration), most have never been
3.2.2 Touch Variability. Touch variability is the total distance
previously conceptualized or given to mathematical formalism.
covered by successive touch inputs from finger-down to finger-up.
As a general principle, our metrics result in values closer to zero
Intuitively, this metric indicates how much distance was covered
when touches are closer to “perfect” (i.e., when a single finger
during a touch (e.g., its “jitter”).
lands and lifts instantaneously from the same place). Similarly, our
−1 q
metrics generally increase in magnitude as touches are further from =Õ

“perfect” (i.e., as a finger moves, rotates, changes area, or persists on Variability = (G8 − G8 −1 ) 2 + (~8 − ~8 −1 ) 2 ∈ [0, ∞)
the screen). We begin by conceptualizing the “anatomy of a touch”
8=1

as a time series of ovals.
3.1 Anatomy of a Touch
Our touch metrics are designed to capture what happens during a
touch process, which, in the very least, consists of a finger-down
oval and a finger-up oval (Figure 2). Additional ovals might be
created if the finger moves during the touch or if the touch software
samples the finger position at regular intervals. Our metrics are
defined assuming only one finger is in contact with the screen, an
assumption we revisit in Section 3.5.
Let = be the total number of touch input events captured from
finger-down to finger-up, inclusive. A touch process can be defined
as a sequence of touch ovals %0, ..., %=−1 , where finger-down is %0
and finger-up is %=−1 . Then, for the 8 C oval:
• (G8 , ~8 ) is the location of its centroid;
• 08 , 18 are the lengths of its major and minor axes, respectively;
• \8 is the angle, in radians, of the major axis relative to the
+G axis (i.e., straight right on the screen);
• (8 is the oval’s size, calculated as c08 18 /4;
• )8 is the oval’s timestamp.
3.2.3 Touch Drift. Touch drift is the Euclidean distance from
finger-down to finger-up. Intuitively, this metric indicates how far
away a touch finished from where it started.
q
Drift = (G=−1 − G 0 ) 2 + (~=−1 − ~0 ) 2 ∈ [0, ∞)
3.2.4 Touch Duration. Touch duration is the time elapsed from
finger-down to finger-up. Intuitively, this metric indicates how long
a touch persisted.
Duration = ()=−1 − )0 ) ∈ [0, ∞)
3.2.5 Touch Extent. Touch extent is the Euclidean distance
between the most distant two oval centroids. Intuitively, this
metric indicates the spatial range of the touch process.
q
Extent = max (G 9 − G8 ) 2 + (~ 9 − ~8 ) 2 ∈ [0, ∞)
8,9 ∈0,...,=−1
3.3 Metrics Based on Touch Area
In our second of three categories, we formalize five touch metrics
based on touch area. The metrics are absolute area change, (signed)
area change, area variability, area deviation, and area extent.

3.2 Metrics Based on Touch Location and Time
In our first of three categories, we formalize five touch metrics based
on location and time. These metrics are touch direction, variability,
drift, duration, and extent.
3.2.1 Touch Direction. Touch direction is the angle formed
between the centroids of the finger-down and finger-up ovals.
Intuitively, this metric indicates the overall direction that a touch
moved. Conceptually, we define straight right (+G) from the
finger-down centroid as 0◦ , straight left (−G) as 180◦ , straight
up (−~) as 90◦ , and straight down (+~) as 270◦ , although our
3.3.1 Touch Absolute Area Change. The absolute area change of
a touch is the change in area between finger-up and finger-down
ovals. Intuitively, this metric indicates the difference in area
between the start and end of a touch.
Absolute Area Change = |(=−1 − ( 0 | ∈ [0, ∞)
3.3.2 Touch Area Change. The (signed) area change of a touch is the
same as above, but positive if finger-up is larger than finger-down;
negative or zero otherwise. Intuitively, this metric indicates whether
a touch grew (+) or shrank (-), and by how much.
Area Change = ((=−1 − ( 0 ) ∈ (−∞, ∞)
Quantifying Touch: New Metrics for Characterizing What Happens During a Touch ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 3: Depictions of (a) touch variability, (b) touch drift, and (c) touch extent. The blue ovals centered at the black and red
dots represent the first and the last touch ovals of the sequence, respectively. The dark-blue line represents the centroid trace,
and the yellow line represents the respective metric.
3.3.3 Touch Area Variability. The variability in touch area is the 3.4.2 Touch Angle Change. The (signed) angle change of a touch is
cumulative change in area over all touch ovals from finger-down the same as above, but positive when counterclockwise and negative
to finger-up. Intuitively, this metric indicates how “stable” the size when clockwise. Intuitively, this metric indicates whether the finger
of a touch was during its lifetime. rotated counterclockwise or clockwise, and by how much.
−1
=Õ Angle Change = (\=−1 − \ 0 ) ∈ (−c, c]
Area Variability = |(8 − (8 −1 | ∈ [0, ∞)
8=1 3.4.3 Touch Angle Variability. The variability in touch angle is the
cumulative change of the major axis angle over the duration of the
3.3.4 Touch Area Deviation. The deviation in touch area is the
touch. Intuitively, this metric indicates how “jittery” the orientation
standard deviation of touch oval area over the duration of the
of the finger was during a touch.
touch. Intuitively, this metric indicates how much the area changed
−1
from one oval to the next during the touch process. =Õ
s Angle Variability = |\8 − \8 −1 | ∈ [0, ∞)
Í= − 1 ¯2 =Õ−1
8 =0 ((8 − ( )
8=1
Area Deviation = ∈ [0, ∞), where (¯ = (8 /=
=−1 3.4.4 Touch Angle Deviation. The deviation in touch angle is the
8=0
standard deviation of the major axis angle over the duration of
3.3.5 Touch Area Extent. The extent of a touch’s area is the the touch. Intuitively, this metric indicates how much the finger
difference between the largest and smallest touch areas from orientation changed from one oval to the next during a touch.
finger-down to finger-up. Intuitively, this metric indicates the s
range of touch areas that occurred during the touch process.
Í=−1 ¯2 =−1
8=0 (\ 8 − \ )
Õ
Angle Deviation = ∈ [0, ∞), where \¯ = \8 /=
=−1
   
Area Extent = max (8 − min ( 9 ∈ [0, ∞) 8=0
8 ∈0,...,=−1 9 ∈0,...,=−1
3.4.5 Touch Angle Extent. The extent of a touch’s angle is the angle
3.4 Metrics Based on Touch Angle between the two major axes that are farthest apart in orientation.
In our third of three categories, we formalize five touch metrics Intuitively, this metric indicates the range of angles that the major
based on oval angles, that is, the angle of the major axis relative to axis covered throughout the touch.
the +G-axis (straight right), which is defined as 0◦ . These metrics
 
are absolute angle change, (signed) angle change, angle variability, Angle Extent = max |\ 9 − \8 | ∈ [0, c]
8,9 ∈0,...,=−1
angle deviation, and angle extent.
For a theoretical “perfect touch,” where the finger-down
Note that two touch angles with greatly different values can, in
oval exactly coincides with the finger-up oval and the touch
fact, be quite close to each other (e.g., 359◦ and 1◦ are only 2◦ apart).
is instantaneous, 14 metrics will equal zero and touch direction will
Therefore, to make our metrics accurately reflect the amount of
be undefined. In practice, of course, there should be some elapsed
angle change during a touch, we define the angle change from \ 1
time between finger-down and finger-up, with distinct ovals for
to \ 2 as
each captured event. Of course, the number of ovals and their
\ 2 − \ 1 =
\ 1,2
properties one receives depend upon the software and hardware
where
\ 1,2 is the angle between \ 1 and \ 2 in (−c, c]. Also, comprising the sensing platform one is using. For example, some
counter-clockwise is considered positive and clockwise is considered Android smartphones and tablets do not reliably report major and
negative. minor axis information. In our study, described below, we used a
3.4.1 Touch Absolute Angle Change. The absolute angle change of Microsoft Perceptive Pixel (PPI) display with custom software that
a touch is the change in the major axis angle from finger-down to we wrote to ensure all necessary oval information was reported for
finger-up. Intuitively, this metric indicates how much the finger calculation of our 15 touch metrics.
orientation changed between the start and end of a touch. Table 1 offers a few example touch centroid traces to illustrate the
intuitive differences among three of our touch metrics. Recall that
Absolute Angle Change = |\=−1 − \ 0 | ∈ [0, c]
touch variability is the total distance covered by successive touch
ovals; touch drift is the distance from finger-down to finger-up; and
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 4: Depictions of (a) touch angle change, (b) touch angle variability, (c) touch angle extent. The blue ovals represent the
touch oval sequence, the orange & yellow lines represent the major axes of ovals, and the green arcs represent measured angles.
touch extent is the distance between the two most distant touch
ovals that occurred during the touch process.
3.5 Handling Multiple Concurrent Fingers
Our 15 metrics assume that a single finger is performing a touch.
Indeed, future research could explore additional metrics specifically
meant to characterize multi-touch behaviors. But our metrics can
nonetheless be used in multi-touch situations, and inadvertent
multi-touch events can and do occur, especially for people with
limited fine motor function or for people in impairing situations.
Therefore, we address the question of which “policy” to adopt for
our metrics when multiple concurrent touches occur. To this end,
we explore three policies: first-down, longest-lived, and sum-of-all.
In discussing these three policies, we regard a touch trace as the
successive touch ovals associated with a single finger, and a touch
process as the collection of touch ovals from the first finger-down
to the last finger-up, even if they are different fingers. Under this
view, a single touch process can contain multiple touch traces if
multiple fingers touch the screen.
3.5.1 First-Down Policy. As a baseline, we examined the first-down
policy, which defines a touch process by whichever finger is first
to land on the touch-sensitive display. If other fingers land on the
display while the first finger is still present, they are simply ignored.
This policy is obviously naïve and sometimes incorrect, but serves
as a useful point of comparison.
3.5.2 Longest-Lived Policy. With the longest-lived policy, a touch
process is defined by whichever finger persists on the touch-sensitive
display the longest. Thus, all fingers are tracked during the touch
process, and once the process has ended, the finger with the
greatest duration is used to calculate our 15 touch metrics. This
policy has the benefit that short-lived fingers are often the result of
unintentional contact with the display surface.
3.5.3 Sum-Of-All Policy. With the sum-of-all policy, we avoid
trying to discern which finger is most “indicative” of the user’s
intended touch process, and instead include all touch traces in
the calculation of our metrics. As unintentional finger-touches are
often short-lived and produce relatively few touch events, they
naturally contribute little to the sum of touch oval data than the
finger indicative of the user’s actual intentions. For people with
limited fine motor function, multiple fingers contacting the screen
can even be an indicator of their fine motor abilities and can be
reflective of their challenges using a device. Thus, for this policy, we
first calculate our 15 metrics for the touch traces of all fingers that
Kong, et al.
appear, and then simply sum these over the entire touch process.
The above three policies provide a pragmatic approach to
handling cases where multiple fingers appear in a touch process
that only requires a single finger. As we show below, all three
policies resulted in similar empirical conclusions in our user study,
which only required single-finger touches and no swipe or pinch
gestures to be performed. As for metrics to specifically characterize
multi-touch behaviors, we leave their formulation to future work.
4 STUDY METHOD
In this section, we describe a formal user study to put our 15 metrics
through their paces. We conducted this study using custom software
we built for the Microsoft Perceptive Pixel (PPI) display, which is
capable of giving complete oval information for touch down, move,
and up events. The goal of our study was to understand whether
and how our metrics might characterize users’ touch behaviors,
particularly any differences between users with and without limited
fine motor function.
4.1 Participants
We recruited 27 participants, 15 of whom reported having limited
fine motor function (LFMF), using email solicitations, phone
calls, convenience sampling, and snowball sampling (Table 2). On
average, participants with LFMF were 63.8 years old (( = 18.8),
with 8 women and 7 men. Reported health conditions included
essential tremor (5), arthritis (4), Charcot-Marie-Tooth disease (1),
enhanced physiological tremor (1), idiopathic tremor (1), referral
nerve pain (1), spinal cord injury (1), tendinitis (1), and traumatic
brain injury (1). Reported specific fine motor challenges included
tremor (53.3%), spasm (40.0%), numbness (6.7%), stiffness (53.3%),
pain (46.7%), rapid fatigue (26.7%), poor coordination (13.3%),
low strength (33.3%), slow movements (20.0%), difficulty gripping
(33.3%), difficulty lifting (20.0%), difficulty holding (20.0%), difficulty
holding still (26.7%), difficulty forming hand postures (26.7%), and
difficulty controlling movement direction (13.3%) and distance
(13.3%). Participants without LFMF were, on average, 34.8 years old
(( = 22.0), with 8 women and 4 men.
4.2 Apparatus
To collect touch data, we developed a custom testbed program for
use on a Microsoft Perceptive Pixel (PPI) 55-inch tabletop display
connected to a PC running Windows 10. The testbed ran a Universal
Windows Platform (UWP) application we wrote in C#. The purpose
of the testbed was to present participants with a series of crosshairs
Quantifying Touch: New Metrics for Characterizing What Happens During a Touch ASSETS ’22, October 23–26, 2022, Athens, Greece

Participants w/ Limited Fine Motor Function (LFMF) Participants w/o LFMF

ID Gender Age Diagnosis Fine Motor Challenges ID Gender Age
A1 woman 22 Tendinitis in wrist Stiffness, rapid fatigue B1 woman 21
A2 man 68 N/A Low strength B2 woman 23
Stiffness, pain, slow movements,
difficulty gripping, difficulty holding,
A3 man 77 Arthritis B3 man 23
difficulty holding still,
difficulty forming hand postures
Pain, poor coordination,
Mild referral
A4 man 59 difficulty forming hand postures, B4 woman 25
nerve pain
difficulty controlling movement distance
Tremor, spasm, stiffness, pain, rapid fatigue,
Arthritis, poor coordination, low strength,
A5 woman 75 Charcot-Marie-Tooth slow movements, difficulty gripping, B5 woman 26
disease difficulty lifting, difficulty holding still,
difficulty forming hand postures
A6 man 71 Essential tremor Tremor, slow movements B6 woman 29
A7 woman 67 Essential tremor Tremor, spasm, pain B7 man 23
Tremor, spasm, stiffness, low strength,
A8 woman 68 Essential tremor B8 woman 37
difficulty gripping, difficulty holding
Tremor, stiffness, pain,
A9 woman 81 Arthritis B9 woman 80
difficulty gripping, difficulty lifting
Tremor, stiffness, pain, rapid fatigue,
low strength, difficulty holding still,
A10 woman 71 Essential tremor B10 woman 24
difficulty controlling movement direction,
difficulty controlling movement distance
A11 man 78 Essential tremor Tremor B11 man 24
A12 man 34 Enhanced physiological tremor Spasm, pain, rapid fatigue B12 man 82
Tremor, spasm, stiffness, low strength,
Arthritis, Samilial
A13 woman 78 difficulty lifting, difficulty holding still,
idiopathic tremor
difficulty controlling movement direction
Spasm, difficulty gripping, difficulty holding,
A14 man 32 Spinal cord injury
difficulty forming hand postures
A15 woman 76 Traumatic brain injury Numbness, stiffness
Table 2: Demographics of participants with and without limited fine motor function (LFMF).

Figure 6: (a) A participant using a finger to tap on a crosshairs

target. (b) The sitting posture for participants who preferred
it. (c) The standing posture for participants who preferred it.

Figure 5: Our custom tabletop testbed. Four buttons in
the top-left enabled the experimenter to control the study:
start session, resume session, restart session, and undo. The
participant ID and trial number were displayed at the top. The
large white rectangular region was defined by the participant
to indicate their comfortable reach area. Crosshairs were
randomly generated within this region.
that they touched as accurately as they could. All touch events
(finger down, move, and up) were logged as touch ovals. Each oval
was reported with its centroid location, orientation, major and
minor axis lengths, and a timestamp.
To ensure participants could reach the crosshairs presented to
them, before the trials began, participants were asked to indicate the
area of the tabletop that they could comfortably reach by touching
the four corners of a rectangle of their choosing (Figure 5). During
trials, crosshairs were generated at random (G, ~) locations only
within this rectangle. The crosshairs themselves had a line thickness
of 1 pixel and a radius of 10 pixels, and participants were told to
touch the crosshairs’ centers as accurately as they could. The PPI
display was placed on a desk for participants who preferred to stand,
or on a coffee table for participants who preferred to sit (Figure 6).
4.3 Procedure
Study sessions were conducted in-person following all COVID-19
health and safety protocols, with approval from our university’s
Institutional Review Board (IRB). After a brief introduction to
the study and the collection of demographic information via a
questionnaire (see Table 2), participants were asked to draw a
rectangular area on the screen indicating the region in which they
were comfortable reaching. Each participant then completed 10
practice trials to familiarize themselves with the study. A single
ASSETS ’22, October 23–26, 2022, Athens, Greece
trial consisted of touching a crosshairs that was drawn at a random
(G, ~) location within the comfortable reach area. After the practice
trials, each participant completed 10 blocks of 20 trials for a
total of 200 trials. (With 27 participants, our data set consisted
of 200 × 27 = 5400 touch trials in all.) Short breaks were offered
between each block of trials. Upon completing all pointing trials,
participants were asked to rate how challenging they thought the
trials were on a Likert-type scale ranging from “1=very easy” to
“7=very difficult.” They were also asked about anything they found
challenging during the trials, and any difficulties they have using
touch screen devices in their daily lives. The study session took ∼45
minutes for participants with limited fine motor function (LFMF),
and <30 minutes for participants without LFMF. Participants with
LFMF were compensated $40 USD and participants without LFMF
were compensated $15 USD.
4.4 Design and Analysis
The purpose of our study was to exercise our touch metrics and see
how they differ for participants with and without limited fine motor
function (LFMF). In formal experiment terms, our study involved
a single between-subjects factor indicating whether someone
reported having LFMF or not. That said, during our study sessions,
we observed that the touch abilities of participants with LFMF
varied a great deal, as some participants experienced significant
difficulty touching crosshairs while others had almost no difficulty.
To ensure that our analysis took into account such differences, we
further grouped our participants into three levels having different
degrees of fine motor challenge: None, Moderate, and Severe.
The None group included participants who did not report having
LFMF and who did not show any observable signs of having
limited fine motor function during the study. The Moderate group
included participants who (1) reported having LFMF, but reported
that they were having a “good day” on the day of their study
session, or (2) reported having LFMF, but reported that they found
ways of accommodating their fine motor challenges such that
their touch screen use was not affected, or (3) did not report
having LFMF, but showed some observable signs of having limited
fine motor function during the study. Finally, the Severe group
included the participants who both reported having LFMF and
showed observable signs of having fine motor challenges. These
participants also described having difficulty operating touch screen
devices in their everyday lives. In total, we had 11 participants
in the None group, 10 participants in the Moderate group, and 6
participants in the Severe group.
For inferential statistics, because most of our metrics were
conditionally non-normal, we used the rank transform procedure
of Conover and Iman [9] to apply midranks to our 15 touch metrics
before conducting our analyses of variance using linear mixed
models [15, 36], resulting in nonparametric analyses. Our single
factor was Impairment with three levels: None, Moderate, and Severe,
as described above. In addition, we ran separate analyses with each
specific fine motor challenge (e.g., “tremor”) as a dichotomous fixed
effect. As is customary, all models included Participant as a random
factor to account for repeated measures [36]. Any post hoc pairwise
comparisons following significant or marginal omnibus tests were
corrected with Holm’s sequential Bonferroni procedure [22].
Kong, et al.
5 RESULTS
In this section, we present the results of our study. These results
include an examination of the effects of Impairment and of each
specific fine motor challenge on our 15 touch metrics. The boxplots
in Appendix A show each of our 15 touch metrics by the three
levels of Impairment. We also show associations between specific
self-reported fine motor challenges and the 15 touch metrics (see
Table 3). But first, we discuss our choice of touch policy for resolving
inadvertent multiple touches (see Section 3.5).
5.1 Choosing a Touch Policy
Overall, 1.2% of trials had more than one finger touch the surface.
By level of Impairment, multiple fingers occurred in 0.1% (None),
0.5% (Moderate), and 4.4% (Severe) of trials.
We calculated our 15 touch metrics using the policies discussed
in Section 3.5. In conducting our analyses, we found that the results
from all three touch policies generally agreed. Specifically, our
analyses revealed that in 77.2% of touch processes that had multiple
fingers, the finger to land first on the table was also the longest-lived
finger, explaining the convergence of results for the first-down and
longest-lived policies. The sum-of-all policy also resulted in the
same conclusions, but with the differences magnified, owing to
the inclusion of all fingers that touched the surface during a touch
process. Given the convergence of the three policies, we proceed
using the sum-of-all policy for the remainder of our analyses.
5.2 How Our Metrics Reveal Touch Behaviors
Appendix A shows means and standard deviations for all 15 metrics
by level of Impairment. Visually, it is apparent that the Severe
group tended to have somewhat higher values for many of our
touch metrics. Non-parametric analysis indicates a significant main
effect of Impairment on touch variability (j2,# 2
=5400 = 15.55, ? <
2 2
.001), drift (j2,# =5400 = 11.58, ? < .01), duration (j2,# =5400 =
2
10.85, ? < .01), extent (j 2,# =5400 = 12.86, ? < .01), and area
2
change (j2,#
=5400 = 6.46, ? < .05). Furthermore, Impairment had a
marginal effect on absolute area change (j2,#2
= 5400 = 4.78, ? = .092).
Post hoc pairwise comparisons among levels of Impairment
revealed that the Severe group had significantly higher touch
variability (C 24 = −3.76, ? < .01), drift (C 24 = −3.27, ? < .01),
duration (C 24 = −3.28, ? < .01), and extent (C 24 = −3.43, ? < .01)
than the None group, and marginally higher area change (C 24 =
−2.51, ? = .057) than the None group. The Severe group also had
significantly higher touch variability (C 24 = −3.31, ? < .01), drift
(C 24 = −2.80, ? < .05), and extent (C 24 = −2.98, ? < .05) than the
Moderate group. Thus, it seems that some of our metrics are indeed
capable of revealing differences in touch behaviors among our
three groups.
Along with examining the effects of Impairment on our 15 touch
metrics, we also examined correlations between specific fine motor
challenges and our metrics. Recall that at the start of our study,
participants indicated whether or not they experienced various
fine motor challenges (see Section 4.1), namely tremor, spasm,
numbness, stiffness, pain, rapid fatigue, poor coordination, low
strength, slow movements, difficulty gripping, difficulty lifting,
difficulty holding, difficulty holding still, difficulty forming hand
postures, difficulty controlling movement direction, and difficulty
Quantifying Touch: New Metrics for Characterizing What Happens During a Touch ASSETS ’22, October 23–26, 2022, Athens, Greece

Poor Difficulty Difficulty For- Difficulty Con-

Difficulty
Tremor Spasm Stiffness Pain Coordi- Holding ming Hand trolling Movement
Lifting
nation Still Postures Distance
Direction – – 3.08 · – – 5.66 ∗ 3.91 ∗ – –
Variability 8.07 ∗∗ – 3.38 · 4.95 ∗ – 3.89 ∗ – – –
Drift 6.39 ∗ – 2.74 · 4.45 ∗ – 3.46 · – – –
Duration 8.27 ∗∗ 3.57 · – – – – – – 3.11 ·
Extent 7.04 ∗∗ – 3.02 · 4.64 ∗ – 3.52 · – – –
Absolute Area Change 3.94 ∗ – – – – – – – –
Area Change 3.90 ∗ – – – – – – – –
Area Variability 2.73 · – – – – – – – –
Area Deviation 2.71 · – – – – – – – –
Area Extent 3.04 · – – – – – – – –
Absolute Angle Change – – – 3.72 · – – – 4.01 ∗ –
Angle Change 2.86 · – – – 6.00 ∗ – – – 2.77 ·
Angle Variability – – – 3.51 · – – – 3.08 · –
Angle Deviation – – – – – – – 3.50 · –
Angle Extent – – – 3.30 · – – – 3.23 · –
Table 3: Chi-square (j 2 ) statistics for all significant (? < .05) and marginal (? < .10) effects of each fine motor challenge on the
touch metrics for people with vs. people without the given challenge. All j 2 -statistics are for j2,# 2
=5400 . Significance indicators:
·? < .10, ∗? < .05, ∗∗ ? < .01.

controlling movement distance. These challenges were indicated in
a dichotomous fashion (“yes” or “no”) at participants’ discretion.
All significant (? < .05) and marginal (? < .10) correlations
between these specific fine motor challenges and our touch metrics
are presented in Table 3.2 This table reveals, for each specific fine
motor challenge, the touch metrics with which it significantly (or
marginally) correlates compared to people without that challenge.
For example, people who self-reported experiencing tremor showed
a significant correlation with touch variability, drift, duration,
extent, and absolute area change; and marginal correlations with
area variability, area deviation, area extent, and angle change
compared to people who did not report experiencing tremor. Taken
as a whole, then, we can use these results to understand how
people experiencing different fine motor challenges tended to
exhibit certain corresponding touch behaviors, as revealed by our
touch metrics.
6 DISCUSSION
Overall, touch variability, drift, duration, extent, and (signed) area
change distinguished well among our three Impairment groups,
especially between the None and Severe groups. To a lesser extent,
the same can be said of absolute area change. Our results match
our expectations that people with limited fine motor function
might experience more challenges while using touch screens,
and therefore would have higher values for some metrics. Our
findings also comport with those of Findlater et al. [13], which
showed increased pointing errors on touch screens and a high
frequency of spurious touches for people with limited fine motor
function. Moreover, our results also align with what we observed
during our study sessions, namely that participants in the Severe
group experienced greater difficulty touching accurately than those
in either the None or Moderate groups. Furthermore, the touch
2 Fine
motor challenges mentioned in Section 4.1 but not shown in Table 3 exhibited
no detectable correlations with our touch metrics; they are omitted for clarity of
presentation.
behaviors of participants in the None and Moderate groups were
often difficult to distinguish during study sessions, and so we
are not surprised to find few statistically detectable differences
between them.
Many or all of touch variability, drift, duration, and extent were
associated with tremor, stiffness, pain, and difficulty lifting. Touch
duration was also associated with spasm and difficulty controlling
movement distance. Touch direction was associated with stiffness,
difficulty lifting, and difficulty holding still. Touch extent was
associated with tremor, stiffness, pain, and difficulty lifting. Touch
angle change was associated with tremor, poor coordination, and
difficulty controlling movement distance. These associations, once
revealed by our analyses, generally are quite intuitive.
Although most of the area and angle metrics did not show
significant differences between Impairment groups, they were
nonetheless useful in characterizing different specific fine motor
challenges. The touch area metrics were positively correlated
with tremor (absolute area change, signed area change, area
variability, area deviation, and area extent). Certain touch angle
metrics were also positively correlated with tremor (signed angle
change), pain (absolute angle change, angle variability, angle
extent), poor coordination (signed angle change), difficulty forming
hand postures (absolute angle change, angle variability, angle
deviation, and angle extent), and difficulty controlling movement
distance (signed angle change).
Among all specific fine motor challenges, tremor had an impact
on the most individual touch metrics. This finding is intuitive,
as tremor directly affects people’s finger movements and use of
touch screen devices. Also, stiffness was correlated with touch
direction, variability, drift, and extent, which also comports with our
expectations, as stiffness might result in reduced finger movement.
Although challenges like pain, difficulty lifting, and difficulty
forming hand postures also had associations with a few of our
metrics, their expected effects on our touch metrics are less directly
predictable. Regardless, our findings on the whole show that our
ASSETS ’22, October 23–26, 2022, Athens, Greece
metrics are useful in characterizing a number of different fine
motor challenges, and can be helpful for understanding users’
different behaviors and challenges while using touch screens.
Our findings also match findings in the earlier ACM ASSETS
work which analyzed a subset of the metrics [34], that people who
self-reported as having limited fine motor function tended to have
significantly larger touch variability, drift, duration and extent than
people who did not self-report thusly. We believe that our metrics
can benefit future accessibility research, by providing additional
information for understanding motor-impaired touch behaviors
in empirical studies, as well as enabling inventions of accessible
systems with our metrics – for example, ability-aware systems that
dynamically adapt to users’ touch abilities (Section 7).
6.1 Limitations
As with any study, ours had limitations. To acquire full touch-oval
information, our study was run on a Microsoft Perceptive Pixel
(PPI) tabletop display, which required us to be co-located with
our participants in a lab setting. A more ecologically valid touch
data set could be obtained from people’s everyday smartphone use,
although based on our testing, many Android smartphones fail to
report complete touch-oval data, making this approach challenging.
Future smartphone models might remedy this limitation.
Although we were able to recruit 15 participants who reported
having limited fine motor function, and 12 participants otherwise,
our sample of 27 participants was still relatively small considering
the large variety and severity of fine motor challenges. With
more participants, we would probably be able to identify stronger
and more correlations between fine motor challenges and our 15
touch metrics, likely resolving many of our marginal results into
statistically significant ones. Furthermore, many of our participant
volunteers were quite tech-savvy, and might have found ways
to adapt their touch behaviors to accommodate their fine motor
challenges. Our distinguishing of participants with moderate and
severe fine motor challenges enabled us to detect performance
differences between them, but having more participants in each
group would enable some statistical trends to further emerge.
Finally, our goal was to isolate participant touches using a testbed
showing crosshairs as, effectively, single-pixel targets. It remains to
be seen whether participants’ touch behaviors would be the same
on “real” user interfaces with, for example, buttons, hyperlinks,
scroll bars, and menus, to name a few. A future version of our study
could not only include crosshairs as targets, but also genuine user
interfaces whose targets are typical of touch-based applications.
7 FUTURE WORK
In addition to addressing the limitations above, strategic directions
for future work are numerous. This work provided an initial
scientific exploration into 15 touch metrics and their association
with limited fine motor function. Going forward, systems could
be implemented that observe a user’s touch behaviors and offer
possible adaptations for improving the usability or accessibility
of their user interfaces, even at runtime. For example, if a system
detects high touch variability, drift, extent, area change, area
variability, and angle change, it might be reasonable for it to
hypothesize that its user experiences tremor and could benefit from
Kong, et al.
larger target sizes, an idea consistent with ability-based design
[57, 58]. Note that such a hypothesis could be drawn without
knowing anything about the specific targets in the user interface
or the particular user’s intention, as all 15 of our touch metrics are
target-agnostic [56]. This property of our metrics makes them easy
to implement in a way that avoids the vast complications of having
to identify targets in a user interface [2, 11, 12, 56]. Besides adaptive
or personalized user interfaces, our metrics might also be used by
software designed to diagnose or track human motor functioning.
For example, a personal health application of our metrics might
allow a user to self-track their touch performance with our 15
metrics to see whether they are affected by medication, time of
day, fatigue, therapy, or progressive symptoms. Yet another future
application of this work is in studies comparing touch screens,
user groups, or user situations. We anticipate that numerous future
research and development projects might benefit from measuring
more than just touch accuracy and speed, using our metrics to
understand why overall touch performance is what it is.
8 CONCLUSION
Current human performance measures for use with touch-based
systems have focused mainly on overall performance like touch
accuracy and target acquisition speed. But to understand what
happens during a touch, we formalized 15 target-agnostic touch
metrics: touch direction, variability, drift, duration, extent, absolute
/ signed area change, area variability, area deviation, area extent,
absolute / signed angle change, angle variability, angle deviation,
and angle extent. Instead of treating a touch as an atomic event, our
metrics regard a touch as a time series of touch ovals approximating
a finger’s contact area from finger-down to finger-up. For each of
our 15 metrics, we provided the mathematical formula and intuitive
description of what the metric means. We also included visual
depictions of some of our metrics to aid understanding. We also
described three policies to handle cases where multiple fingers
inadvertently contact the screen during a touch.
To exercise our 15 metrics, we built a custom testbed for an
interactive tabletop and collected complete touch-oval data from
27 participants, 15 of whom reported having limited fine motor
function. Our analysis showed that our metrics could effectively
distinguish unimpaired from impaired touch behaviors. Our metrics
were also significantly associated with different self-reported fine
motor challenges. Thus, our metrics can shed light on the underlying
causes of touch inaccuracy, and can further help understand users’
touch abilities. Conceivably, our metrics might inform the design of
touch-based systems. Touch devices and touch accuracy will remain
important for many years to come; our metrics can contribute to a
better understanding of both.
ACKNOWLEDGMENTS
This work was supported in part by Google, by the University
of Washington Center for Research and Education on Accessible
Technology and Experiences (CREATE), and by National Science
Foundation grant #IIS-1702751. Any opinions, findings, conclusions
or recommendations expressed in our work are those of the authors
and do not necessarily reflect those of any supporter.
Quantifying Touch: New Metrics for Characterizing What Happens During a Touch
REFERENCES
[1] Teresa Arroyo-Gallego, María Jesus Ledesma-Carbayo, Alvaro Sánchez-Ferro, Ian
Butterworth, Carlos S Mendoza, Michele Matarazzo, Paloma Montero, Roberto
López-Blanco, Veronica Puertas-Martin, Rocio Trincado, et al. 2017. Detection of
motor impairment in Parkinson’s disease via mobile touchscreen typing. IEEE
Transactions on Biomedical Engineering 64, 9 (2017), 1994–2002.
[2] Ravin Balakrishnan. 2004. “Beating” Fitts’ law: virtual enhancements for pointing
facilitation. International Journal of Human-Computer Studies 61, 6 (2004),
857–874.
[3] Hrvoje Benko, Andrew D Wilson, and Patrick Baudisch. 2006. Precise selection
techniques for multi-touch screens. In Proceedings of the ACM SIGCHI Conference
on Human Factors in Computing Systems. 1263–1272.
[4] Joanna Bergstrom-Lehtovirta, Antti Oulasvirta, and Stephen Brewster. 2011. The
effects of walking speed on target acquisition on a touchscreen interface. In
Proceedings of the 13th International Conference on Human Computer Interaction
with Mobile Devices and Services. 143–146.
[5] Xiaojun Bi, Yang Li, and Shumin Zhai. 2013. FFitts law: modeling finger touch
with fitts’ law. In Proceedings of the ACM SIGCHI Conference on Human Factors in
Computing Systems. 1363–1372.
[6] Xiaojun Bi and Shumin Zhai. 2013. Bayesian touch: a statistical criterion of target
selection with finger touch. In Proceedings of the 26th Annual ACM Symposium
on User Interface Software and Technology. 51–60.
[7] Xiang Cao, Andrew D Wilson, Ravin Balakrishnan, Ken Hinckley, and Scott E
Hudson. 2008. ShapeTouch: Leveraging contact shape on interactive surfaces. In
2008 3rd IEEE International Workshop on Horizontal Interactive Human Computer
Systems. IEEE, 129–136.
[8] Sinead Carew. 2009. Touch-screen gadgets alienate blind. Reuters, January 8
(2009).
[9] William J Conover and Ronald L Iman. 1981. Rank transformations as a bridge
between parametric and nonparametric statistics. The American Statistician 35, 3
(1981), 124–129.
[10] David F Dinges and John W Powell. 1985. Microcomputer analyses of performance
on a portable, simple visual RT task during sustained operations. Behavior research
methods, instruments, & computers 17, 6 (1985), 652–655.
[11] Morgan Dixon, James Fogarty, and Jacob Wobbrock. 2012. A general-purpose
target-aware pointing enhancement using pixel-level analysis of graphical
interfaces. In Proceedings of the ACM SIGCHI Conference on Human Factors in
Computing Systems. 3167–3176.
[12] Abigail Evans and Jacob Wobbrock. 2012. Taming wild behavior: The input
observer for obtaining text entry and mouse pointing measures from everyday
computer use. In Proceedings of the ACM SIGCHI Conference on Human Factors in
Computing Systems. 1947–1956.
[13] Leah Findlater, Karyn Moffatt, Jon E Froehlich, Meethu Malu, and Joan Zhang.
2017. Comparing touchscreen and mouse input performance by people with
and without upper body motor impairments. In Proceedings of the ACM SIGCHI
Conference on Human Factors in Computing Systems. 6056–6061.
[14] Leah Findlater and Lotus Zhang. 2020. Input Accessibility: A Large Dataset and
Summary Analysis of Age, Motor Ability and Input Performance. In Proceedings of
the 22th International ACM SIGACCESS Conference on Computers and Accessibility.
1–6.
[15] Brigitte N Frederick. 1999. Fixed-, Random-, and Mixed-Effects ANOVA Models:
A User-Friendly Guide for Increasing the Generalizability of ANOVA Results.
(1999).
[16] Luca Giancardo, Alvaro Sanchez-Ferro, Teresa Arroyo-Gallego, Ian Butterworth,
Carlos S Mendoza, Paloma Montero, Michele Matarazzo, José A Obeso, Martha L
Gray, and R San José Estépar. 2016. Computer keyboard interaction as an indicator
of early Parkinson’s disease. Scientific Reports 6, 1 (2016), 1–10.
[17] Mayank Goel, Leah Findlater, and Jacob Wobbrock. 2012. WalkType: using
accelerometer data to accomodate situational impairments in mobile touch screen
text entry. In Proceedings of the ACM SIGCHI Conference on Human Factors in
Computing Systems. 2687–2696.
[18] Mayank Goel, Alex Jansen, Travis Mandel, Shwetak N Patel, and Jacob O
Wobbrock. 2013. ContextType: using hand posture information to improve
mobile touch screen text entry. In Proceedings of the ACM SIGCHI Conference on
Human Factors in Computing Systems. 2795–2798.
[19] Tiago Guerreiro, Hugo Nicolau, Joaquim Jorge, and Daniel Gonçalves. 2010.
Towards accessible touch interfaces. In Proceedings of the 12th International ACM
SIGACCESS Conference on Computers and Accessibility. 19–26.
[20] Anhong Guo, Junhan Kong, Michael Rivera, Frank F Xu, and Jeffrey P Bigham.
2019. Statelens: A reverse engineering solution for making existing dynamic
touchscreens accessible. In Proceedings of the 32nd Annual ACM Symposium on
User Interface Software and Technology. 371–385.
[21] Chris Harrison, Julia Schwarz, and Scott E Hudson. 2011. TapSense: enhancing
finger interaction on touch surfaces. In Proceedings of the 24th Annual ACM
Symposium on User Interface Software and Technology. 627–636.
[22] Sture Holm. 1979. A simple sequentially rejective multiple test procedure.
Scandinavian Journal of Statistics (1979), 65–70.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[23] Christian Holz and Patrick Baudisch. 2010. The generalized perceived input point
model and how to double touch accuracy by extracting fingerprints. In Proceedings
of the ACM SIGCHI Conference on Human Factors in Computing Systems. 581–590.
[24] Christian Holz and Patrick Baudisch. 2011. Understanding touch. In Proceedings of
the ACM SIGCHI Conference on Human Factors in Computing Systems. 2501–2510.
[25] Christian Holz and Patrick Baudisch. 2013. Fiberio: a touchscreen that senses
fingerprints. In Proceedings of the 26th Annual ACM Symposium on User Interface
Software and Technology. 41–50.
[26] Faustina Hwang, Simeon Keates, Patrick Langdon, and John Clarkson. 2004.
Mouse movements of motion-impaired users: A submovement analysis. In
Proceedings of the 5th International ACM SIGACCESS Conference on Computers
and Accessibility. 102–109.
[27] Dimitrios Iakovakis, Stelios Hadjidimitriou, Vasileios Charisis, Sevasti
Bostantzopoulou, Zoe Katsarou, and Leontios J Hadjileontiadis. 2018.
Touchscreen typing-pattern analysis for detecting fine motor skills decline in
early-stage Parkinson’s disease. Scientific Reports 8, 1 (2018), 1–13.
[28] Shaun K Kane, Jeffrey P Bigham, and Jacob O Wobbrock. 2008. Slide Rule: making
mobile touch screens accessible to blind people using multi-touch interaction
techniques. In Proceedings of the 10th International ACM SIGACCESS Conference
on Computers and Accessibility. 73–80.
[29] Shaun K Kane, Brian Frey, and Jacob O Wobbrock. 2013. Access lens: a
gesture-based screen reader for real-world documents. In Proceedings of the
ACM SIGCHI Conference on Human Factors in Computing Systems. 347–350.
[30] Shaun K Kane, Meredith Ringel Morris, Annuska Z Perkins, Daniel Wigdor,
Richard E Ladner, and Jacob O Wobbrock. 2011. Access overlays: improving
non-visual access to large touch screens for blind users. In Proceedings of the 24th
Annual ACM Symposium on User Interface Software and Technology. 273–282.
[31] Shaun K Kane, Meredith Ringel Morris, and Jacob O Wobbrock. 2013. Touchplates:
low-cost tactile overlays for visually impaired touch screen users. In Proceedings of
the 15th International ACM SIGACCESS Conference on Computers and Accessibility.
1–8.
[32] Matthew Kay, Kyle Rector, Sunny Consolvo, Ben Greenstein, Jacob O Wobbrock,
Nathaniel F Watson, and Julie A Kientz. 2013. PVT-touch: adapting a reaction
time test for touchscreen devices. In 2013 7th International Conference on Pervasive
Computing Technologies for Healthcare and Workshops. IEEE, 248–251.
[33] Simeon Keates, Faustina Hwang, Patrick Langdon, P John Clarkson, and Peter
Robinson. 2002. Cursor measures for motion-impaired computer users. In
Proceedings of the 4th International ACM SIGACCESS Conference on Computers
and Accessibility. 135–142.
[34] Junhan Kong, Mingyuan Zhong, James Fogarty, and Jacob O Wobbrock. 2021.
New Metrics for Understanding Touch by People with and without Limited
Fine Motor Function. In Proceedings of the 23th International ACM SIGACCESS
Conference on Computers and Accessibility. (To apprear).
[35] Abinaya Kumar, Aishwarya Radjesh, Sven Mayer, and Huy Viet Le. 2019.
Improving the input accuracy of touchscreens using deep learning. In Extended
Abstracts of the ACM SIGCHI Conference on Human Factors in Computing Systems.
1–6.
[36] Ramon C Littell, PR Henry, and Clarence B Ammerman. 1998. Statistical analysis
of repeated measures data using SAS procedures. Journal of Animal Science 76, 4
(1998), 1216–1231.
[37] I Scott MacKenzie, Tatu Kauppinen, and Miika Silfverberg. 2001. Accuracy
measures for evaluating computer pointing devices. In Proceedings of the ACM
SIGCHI Conference on Human Factors in Computing Systems. 9–16.
[38] Rafail-Evangelos Mastoras, Dimitrios Iakovakis, Stelios Hadjidimitriou, Vasileios
Charisis, Seada Kassie, Taoufik Alsaadi, Ahsan Khandoker, and Leontios J
Hadjileontiadis. 2019. Touchscreen typing pattern analysis for remote detection
of the depressive tendency. Scientific reports 9, 1 (2019), 1–12.
[39] Sven Mayer, Huy Viet Le, and Niels Henze. 2017. Estimating the finger orientation
on capacitive touchscreens using convolutional neural networks. In Proceedings
of the ACM International Conference on Interactive Surfaces and Spaces. 220–229.
[40] Alexander Mertens, Nicole Jochems, Christopher M Schlick, Daniel Dünnebacke,
and Jan Henrik Dornberg. 2010. Design pattern TRABING: touchscreen-based
input technique for people affected by intention tremor. In Proceedings of the
ACM SIGCHI Symposium on Engineering Interactive Computing Systems. 267–272.
[41] Kyle Montague, Vicki L Hanson, and Andy Cobley. 2012. Designing for
individuals: usable touch-screen interaction through shared user models. In
Proceedings of the 14th International ACM SIGACCESS Conference on Computers
and Accessibility. 151–158.
[42] Kyle Montague, Hugo Nicolau, and Vicki L Hanson. 2014. Motor-impaired
touchscreen interactions in the wild. In Proceedings of the 16th international ACM
SIGACCESS conference on Computers & accessibility. 123–130.
[43] Martez E Mott, Radu-Daniel Vatavu, Shaun K Kane, and Jacob O Wobbrock. 2016.
Smart touch: Improving touch accuracy for people with motor impairments with
template matching. In Proceedings of the ACM SIGCHI Conference on Human
Factors in Computing Systems. 1934–1946.
[44] Martez E Mott and Jacob O Wobbrock. 2019. Cluster Touch: Improving touch
accuracy on smartphones for people with motor and situational impairments.
In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing
Systems. 1–14.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[45] Richard L Potter, Linda J Weldon, and Ben Shneiderman. 1988. Improving the
accuracy of touch screens: an experimental evaluation of three strategies. In
Proceedings of the ACM SIGCHI Conference on Human Factors in Computing
Systems. 27–32.
[46] Sayan Sarcar, Jussi PP Jokinen, Antti Oulasvirta, Zhenxin Wang, Chaklam
Silpasuwanchai, and Xiangshi Ren. 2018. Ability-based optimization of
touchscreen interactions. IEEE Pervasive Computing 17, 1 (2018), 15–26.
[47] Andrew Sears and Ben Shneiderman. 1991. High precision touchscreens: design
strategies and comparisons with a mouse. International Journal of Man-Machine
Studies 34, 4 (1991), 593–613.
[48] R William Soukoreff and I Scott MacKenzie. 2003. Metrics for text entry research:
An evaluation of MSD and KSPC, and a new unified error metric. In Proceedings
of the ACM SIGCHI Conference on Human Factors in Computing Systems. 113–120.
[49] R William Soukoreff and I Scott MacKenzie. 2004. Towards a standard for
pointing device evaluation, perspectives on 27 years of Fitts’ law research in HCI.
International Journal of Human-Computer Studies 61, 6 (2004), 751–789.
[50] Daniel Vogel and Patrick Baudisch. 2007. Shift: a technique for operating
pen-based interfaces using touch. In Proceedings of the ACM SIGCHI Conference
on Human Factors in Computing Systems. 657–666.
[51] Chat Wacharamanotham, Jan Hurtmanns, Alexander Mertens, Martin
Kronenbuerger, Christopher Schlick, and Jan Borchers. 2011. Evaluating
swabbing: a touchscreen input method for elderly users with tremor. In
Proceedings of the ACM SIGCHI Conference on Human Factors in Computing
Systems. 623–626.
[52] Daryl Weir. 2012. Machine learning models for uncertain interaction. In Adjunct
proceedings of the 25th Annual ACM Symposium on User Interface Software and
Technology. 31–34.
[53] Daryl Weir, Simon Rogers, Roderick Murray-Smith, and Markus Löchtefeld.
2012. A user-specific machine learning approach for improving touch accuracy
on mobile devices. In Proceedings of the 25th Annual ACM Symposium on User
Interface Software and Technology. 465–476.
[54] Daniel Wigdor, Sarah Williams, Michael Cronin, Robert Levy, Katie White, Maxim
Mazeev, and Hrvoje Benko. 2009. Ripples: utilizing per-contact visualizations to
improve user interaction with touch displays. In Proceedings of the 22nd Annual
ACM Symposium on User Interface Software and Technology. 3–12.
Kong, et al.
[55] Jacob O Wobbrock. 2019. Situationally-induced impairments and disabilities. In
Web Accessibility. Springer, 59–92.
[56] Jacob O Wobbrock, James Fogarty, Shih-Yen Liu, Shunichi Kimuro, and Susumu
Harada. 2009. The angle mouse: target-agnostic dynamic gain adjustment based
on angular deviation. In Proceedings of the ACM SIGCHI Conference on Human
Factors in Computing Systems. 1401–1410.
[57] Jacob O Wobbrock, Krzysztof Z Gajos, Shaun K Kane, and Gregg C Vanderheiden.
2018. Ability-based design. Commun. ACM 61, 6 (2018), 62–71.
[58] Jacob O Wobbrock, Shaun K Kane, Krzysztof Z Gajos, Susumu Harada, and Jon
Froehlich. 2011. Ability-based design: Concept, principles and examples. ACM
Transactions on Accessible Computing (TACCESS) 3, 3 (2011), 1–27.
[59] Shumin Zhai. 2004. Characterizing computer input with Fitts’ law
parameters—the information and non-information aspects of pointing.
International Journal of Human-Computer Studies 61, 6 (2004), 791–809.
[60] Shumin Zhai, Michael Hunter, and Barton A Smith. 2002. Performance
optimization of virtual keyboards. Human–Computer Interaction 17, 2-3 (2002),
229–269.
[61] Xiaoyi Zhang, Tracy Tran, Yuqian Sun, Ian Culhane, Shobhit Jain, James Fogarty,
and Jennifer Mankoff. 2018. Interactiles: 3D printed tactile interfaces to enhance
mobile touchscreen accessibility. In Proceedings of the 20th International ACM
SIGACCESS Conference on Computers and Accessibility. 131–142.
A DISTRIBUTION OF VALUES OF TOUCH
METRICS BY LEVEL OF IMPAIRMENT
The 15 boxplots below show distribution of values of all 15 touch
metrics by level of Impairment. The plots are vertically organized
such that the first column shows metrics based on touch location
and time (Section 3.2), the second column shows metrics based on
touch area (Section 3.3), and the third column shows metrics based
on touch angle (Section 3.4).
Quantifying Touch: New Metrics for Characterizing What Happens During a Touch ASSETS ’22, October 23–26, 2022, Athens, Greece

1250
15
•• Q)
• Q)
Ol
250 •
g11000 ffi 200
• Cl)

2
C
••
I .c
(_)
750
.c
(_)

150 ••• I I

~
..!!:!
10
••
Cl)
• ~ Ol
(.)

-~
• <(
500
C
I I

111
0 Q) I <( 100
-s Q)
-s
$ $ ••
5 0Cf) •
250 0Cf) 50

0
.0
<(
0
___L_ _L _l_ .0
<(
0
None Moderate Severe None Moderate Severe None Moderate Severe

I 1200 • •
40 • 200
•
Q)
• I
•
•
g30
Q)
Ol 800 Ol
100 •
• C
••
C

+++
Cl) Cl)
:0
-~ 20
•• .c
(_)
.c
(_)
Cl)
I Cl)
400 I Q)
o,
0
> ~
I C
<(
•
.L
<(

_l_ j_ ~
10

+ •
-100
0
0 -I-
•
•
I I
•
I
-200
None Moderate Severe None Moderate Severe None Moderate Severe

15 •• 1500 • 600 • • •
g • •
I
£ •
•• • 15Cl) 1000 ~ 400
I
•
••
•
.;:::
10
•• •I -~ •
•I
-~
•I II •

l
~ >Q)
I
>Cl)
• •• I

..L l J_
500 g,200
~

l_
5
<( I

_l_
<(

0 0
....J..._ _J_
•
~ 0
None Moderate Severe None Moderate Severe None Moderate Severe

1250
• • 150 •
400 ••• •I
C
0
~:J
1000

750 •
•
•
2
C

Cl)
·;;:
300
I
C
0
~ 100
·;;:
Q)
I
•
I
•
I
I
••
•

11
Q)
0
I
~
500 0 200
0 Cl) I Q)
o, 50
~
• • I
l_
C
250 <( 100 <(

0
~ $ 0
•
_J_ _J_ ...L 0
None Moderate Severe None Moderate Severe None Moderate Severe

15 •• • • I •
• 150 •
•• -C 1000 c • ••
•• • • Q)
•
c Q)
• •
10
•I •
I
x • ;E 100
••
I• I I
2 w
X I Q)
w Cl)
~
•• o,

..L l ~
I 500 •
5 <( • I
C
<( 50

0 0
_J_ .....L _L 0 ~ j_ _L
None Moderate Severe None Moderate Severe None Moderate Severe
 Creating 3D Printed Assistive Technology Through Design
Shortcuts: Leveraging Digital Fabrication Services to Incorporate
3D Printing into the Physical Therapy Classroom
Erin L. Higgins∗ William Easley
University of Maryland, Baltimore University of Maryland, Baltimore
County County
erinh2@umbc.edu williameasley3@gmail.com
Amy Hurst
New York University
amyhurst@nyu.edu
ABSTRACT
Digital fabrication methods have been shown to be an efective
method for producing customized assistive technology (AT). How-
ever, the skills required to utilize these tools currently require a
high level of technical skill. Previous research showed that integra-
tion of these skills within physical therapy training is appropriate
but that the level of technical difculty required can be an issue.
We worked to address these issues by introducing a group of PT
students to maker concepts and having them develop custom AT
for real end users with the help of makers. We present three consid-
erations when integrating making into PT curriculum: 1) including
all stakeholders, 2) developing interdisciplinary competencies for
PTs and makers, and 3) leveraging academic training programs to
connect makers and PT students. In this paper, we contribute to
knowledge on how to facilitate the 3D printing of customized ATs
for PT students by connecting them with a community organization
that provides digital fabrication services and technical expertise.
By connecting multiple stakeholders (i.e., PT students, digital fabri-
cators, and AT users), we ofer an approach to overcome time and
capacity constraints of PT students to utilize advanced fabrication
technologies to create customized ATs through connecting them to
professional makers.
CCS CONCEPTS
• B7; Human-centered computing → Accessibility.
KEYWORDS
3D Printing, Assistive Technology, Physical Therapy, Makerspaces,
Education, Digital Fabrication
∗ Corresponding author
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544816
Karen Gordes
University of Maryland Baltimore,
Graduate School
karen.gordes@umaryland.edu
Foad Hamidi
University of Maryland, Baltimore
County
foadhamidi@umbc.edu
ACM Reference Format:
Erin L. Higgins, William Easley, Karen Gordes, Amy Hurst, and Foad Hamidi.
2022. Creating 3D Printed Assistive Technology Through Design Shortcuts:
Leveraging Digital Fabrication Services to Incorporate 3D Printing into the
Physical Therapy Classroom. In The 24th International ACM SIGACCESS
Conference on Computers and Accessibility (ASSETS ’22), October 23–26, 2022,
Athens, Greece. ACM, New York, NY, USA, 16 pages. https://doi.org/10.1145/
3517428.3544816
1 INTRODUCTION
Previous research has shown that consumer-grade fabrication meth-
ods, such as 3D printing and laser cutting, can improve Assistive
Technology (AT) development and production [1-3]. In a recent
article, Mankof et al. provided an overview of the possibilities and
challenges of using consumer-grade fabrication methods to develop
customized AT solutions [1]. They found that these fabrication
methods have great potential for creating small-batch customized
devices that may better meet the needs of a user and for involv-
ing users more closely in the design and creation of their own
technology. Despite this, the research also identifed that current
fabrication tools and processes are not inclusive of people without
prior technical expertise or knowledge and that without stakeholder
involvement at all levels of fabrication design and implementation,
important challenges in integrating these techniques into therapy
and medicine remain [1]. In the last few decades, several online
communities of makers interested in creating customized ATs have
formed [2, 4]. While successful outcomes from these communities
are documented, previous research has also shown a tension be-
tween the priorities of hobbyists and makers and those of clinicians
and therapists [5]. The tension, manifesting in concerns about the
safety and practicality of ATs customized or created by makers,
may be due to the diference between the clinical ethos of “do not
harm” and the making ethos of “help where you can” [5].
Previous research has explored how to better connect digital
fabrication and physical therapy skill sets. McDonald et al. con-
ducted a study on how to introduce Physical Therapy (PT) students
to 3D printing [3, 6]. The study was motivated by the potential
of enhancing clinician’s ability to use digital fabrication skills to
develop customized, usable, and safe devices by providing them
with the needed maker skills and experience. While the research
ASSETS ’22, October 23–26, 2022, Athens, Greece
showed that 3D printing can be a viable and useful addition to PT
students’ current curriculum, it also identifed several issues with
integrating it. Specifcally, it identifed the signifcant amount of
time required for clinicians to learn Computer-Aided Design (CAD)
skills necessary to model a 3D printable object or specifc and de-
tailed features of an AT device. Furthermore, the study identifed
the lack of consistent access to fabrication tools and the challenge
of getting end users to return for multiple visits required to tweak
and evaluate the designs, as barriers for PTs using digital fabrica-
tion for AT design and creation [3]. Overall, this previous research
shows that while digital fabrication tools have great potential for
supporting PTs to create and customize ATs for their clients, it
is unclear how to make them accessible without requiring PTs to
become experts in 3D printing or CAD.
To investigate how to make digital fabrication processes acces-
sible to PT students without the need for them to become experts
at using them, we conducted a study that connected PT students
to staf at a community makerspace that provides 3D printing and
scanning services to local organizations and individuals. By lever-
aging the services and skills of expert makers, rather than learning
complex and time-consuming 3D modeling and printing skills, PT
students could focus on understanding and translating their clients’
needs to a physical design to be communicated for realization. Fur-
thermore, by including real end users in the design and fabrication
process, we investigated how to form small stakeholder groups
consisting of a group of clinicians, an end user, and a community
maker, who would communicate and work together to create a
customized AT. This approach allowed us to investigate the com-
munication possibilities and challenges involved in this process to
design future communication protocols that may alleviate the need
for PTs to learn digital fabrication skills and instead use external
fabrication services and resources to create customized ATs.
In this study, we seek to understand (1) if consumer-grade fabrica-
tion with 3D printers can be used to produce customized and usable
ATs for real end users in the context of a graduate PT classroom, (2)
if resulting AT designs could be accurately communicated by PT
students to a youth-stafed community makerspace for fabrication,
and (3) what information would be needed by makers and PTs to
successfully communicate with each other about this process.
We investigated these questions by integrating six 3D printing
class sessions into the curriculum of 58 PT graduate students over
two semesters. The students were frst given an introduction to 3D
printing, then they designed AT devices for 5 simulated end users
leveraging the resources of a community makerspace, and then
repeated the process for creating AT devices for 12 real end users.
In this paper, we contribute to knowledge on how to facilitate
the 3D printing of customized ATs for PT students by connecting
them with a community organization that provides digital fabri-
cation services and technical expertise. By connecting multiple
stakeholders (i.e., PT students, digital fabricators, and AT users),
we ofer an approach to overcome time and capacity constraints of
PT students to utilize advanced fabrication technologies to create
customized ATs through connecting them to professional makers.
We describe our observations, including feedback and reactions
from PT students who worked with real end users in a classroom
setting to design and realize customized AT products. Our fndings
provide insights on how the AT design process can be improved
Erin Higgins et al.
for PT students, how the communication between PT students and
professional makers can be strengthened, including when to use
clay modeling and when measurements and sketching might be
a better option, and how future PT training programs can better
integrate interdisciplinary skills and competencies related to digital
fabrication, including those related to 3D scanning and printing
practice.
We will contextualize our study using previous research into
AT abandonment, the use of consumer-grade fabrication tools by
people with disabilities, the practices of online DIY communities,
specifcally focused on designing ATs, and previous eforts to use
3D printing in the context of PT education. We then describe how
we designed and implemented class sessions on 3D printing and
fabrication into a graduate PT course and describe our data collec-
tion and analysis procedures. Next, we report fndings from our
study and describe each end user’s resulting customized AT device.
We conclude with a discussion of PT students’ experiences with
respect to the diferent aspects of the project and suggest directions
for future work.
2 RELATED WORK
Our work builds on the existing body of research that has explored
the possibilities and challenges of using digital fabrication processes,
such as 3D printing, for AT development and customization. In this
section, we provide an overview of this previous research, including
work on AT abandonment that motivates eforts to customize ATs
to better meet the needs of end users, studies of Do-it-yourself
(DIY) and bespoke approaches to AT design, and the role of AT
development in physical therapy education.
2.1 AT Abandonment and Fit Issues
According to the Center for Disease Control and Prevention, 61
million adults in the United States live with a disability, equating
to 26% or 1 in 4 adults [7]. Worldwide, an estimated one billion
individuals need ATs. This number is expected to increase to 2
billion by 2030 [8]. Without appropriate AT, individuals are often
unable to fully participate in society and live active, independent
lives. Therefore, it is increasingly important for AT procurement to
become accessible by a large number of people.
AT-focused projects tend to have very high social impact but
a low economic impact [9] and related proft margins. Because of
these relatively small proft margins, it is uncommon for companies
to fund research and development that advances the feld of AT to
meet the growing need. This leads to many issues.
One issue is the high abandonment rate of ATs. Of all ATs that are
prescribed, 20 to 30 percent are abandoned by end users [10]. Poorly
designed or ftted devices that do not meet the needs of the end users
is a key factor related to abandonment [11, 12]. Though improved
service techniques have been shown to reduce abandonment, there
is a growing concern that the quality and reliability of ATs are
decreasing over time [13-16]. Reduced quality may be due in part to
the reduced reimbursements for ATs and related services [14, 15, 17],
which leads to a second major issue in AT – a lack of fnancial
incentive for companies to provide post-launch support.
These issues have motivated eforts to develop ways for end
users to be more involved in the design and customization of ATs
Creating 3D Printed Assistive Technology Through Design Shortcuts: Leveraging Digital Fabrication Services to
Incorporate 3D Printing into the Physical Therapy Classroom
to reduce poor ft and increase user technology buy-in. Our study
contributes to this space, by investigating how to connect PTs,
makers, and end users to allow for the creation of assistive devices
that are customized based on user needs and are, therefore, less
likely to be abandoned.
2.2 Consumer Grade Fabrication Tools for
People with Disabilities
Over the past decades much research has investigated the possi-
bilities of using consumer grade fabrication tools and prototyping
materials for creating customized and Do-It-Yourself Assistive Tech-
nologies (DIY-ATs) [4, 18-20]. Much of this efort has focused on
understanding how creating their own technologies can lead to
empowerment for people with disabilities [18, 19, 21]. Previous
studies have shown that AT users fnd it empowering to create
their own AT, which could lead to fewer cases of abandonment.
Hurst and Tobias showed that individuals with disabilities were
motivated by increased control over design elements, passion, and
cost to create and use their own DIY-ATs [18]. Meissner et al. found
that participants with disabilities in DIY-AT workshops reported
several empowerment-related outcomes, including viewing maker
skills as extensions of their own accessibility hacking abilities and
as tools to gain recognition in their community. Other eforts have
shown that creating DIY-ATs can lead to community building both
ofine (e.g., [22]) and online (see next subsection).
A number of barriers have been identifed in relation to DIY-ATs.
For example, many of the tools that are used in makerspaces, along
with makerspaces themselves, are not accessible. This greatly limits
the ability for individuals with disabilities to work in these spaces
independently. Recommendations have been made for special edu-
cation makerspaces [23, 24] but this has yet to become the norm for
these settings. The design of CAD tools, informed by HCI research,
could greatly increase the inclusiveness of consumer-grade fabricat-
ing [25] but has yet to produce meaningful changes. Furthermore,
Hook et al. found considerable skill and time are required in creat-
ing DIY-ATs, putting additional pressures on parents or teachers
[26].
Recognizing the need for diverse expertise in this space, research
eforts have brought together multiple stakeholders (e.g., end-users,
digital fabricators/makers, therapists) in this space [5, 27]. For ex-
ample, Afatoony and Lee conducted a study that brought together
an end-user with physical disabilities with four occupational ther-
apists, and four industrial designers in a series of workshops to
co-design ATs [27, 28]. They found that working together resulted
in knowledge exchanges and mutual learning that resulted in ap-
plying combined expertise to co-designing novel and advanced AT
solutions. They further identifed the lack of tools, methods and
materials to co-design ATs.
Our work builds on these eforts to leverage the potential of
consumer grade fabrication tools to create ATs with input from end-
users. While it is important to continue making digital fabrication
tools accessible, creating processes that bring together stakeholders
with complementing skills and lived experiences can enrich the
space of AT development and reduce the burden of learning new
skills. While some individuals will enjoy the process of design, there
ASSETS ’22, October 23–26, 2022, Athens, Greece
should not be a requirement to also be an engineer to create or
customize appropriate AT devices for users in a PT setting.
2.3 DIY Online Communities for AT
In addition to hands-on localized eforts to create DIY-AT (described
above), many studies have looked at how to create or leverage online
communities to connect makers interested in developing ATs, end
users, and other stakeholders together. For example, Buehler et al.
studied technologies related to AT posted on the online community
of Thingiverse [2]. While they found that the platform housed a
rich collection of AT designs for free usage and customization, they
also found that a majority of the AT designers using it did not have
disabilities nor any training in AT.
While Thingiverse is a general-purpose fabrication design repos-
itory that houses AT designs in addition to many other designs,
other communities have formed that focus specifcally on DIY-ATs.
For example, the e-NABLE community was created specifcally to
support makers that focused on prosthetic devices [4]. Their open-
sourced designs are shared online and can be printed by volunteers
for those who need them. Despite its promise and potential to ofer
customized ATs to a wide range of users across the world, previous
research has identifed several challenges in this community. For
example, Parry-Hill et al. showed that while there are a number of
e-NABLE chapters around the world, there are only a few mem-
bers who can design devices as opposed to those who can fabricate
and deliver them. Furthermore, there is an issue when someone
receives a device of not being able to get back in contact with the
volunteer if something goes wrong [4]. At a workshop that brought
together various stakeholders of this community, including makers
and clinicians, serious tensions were identifed between the “do no
harm” culture of clinicians and the “help where you can” culture
of makers [5]. Furthermore, the workshop surfaces concerns about
the ability of consumer grade materials and design tools to produce
artifacts and customizations specifc enough for clinical practice.
Previous research into online DIY-AT communities provides
motivation to develop better communication protocols and mecha-
nisms that bring together the expertise of digital fabricators, profes-
sional PTs, and the lived experiences of end users in the design and
customization of ATs. These approaches may ensure device safety
and medical appropriateness by including feedback from PTs while
strengthening device acceptance by incorporating feedback from
end users.
2.4 Digital Fabrication for AT Development in
Clinical Education
Until recently, there were few projects that studied how to bring in
digital fabrication, including 3D printing, into the physical and oc-
cupational therapy classrooms. A recent systematic review of using
3D printing in biological education, found only one paper focused
on PT [29]. In this pioneering project, McDonald et al. developed a
course to introduce PT students to 3D printing [3]. They identifed
several opportunities for connecting PTs and makers in the future,
including that (1) PTs already perform making tasks and presenting
them with new DIY tools could expand this existing practice, (2)
PTs bring important and complementary medical expertise to the
ASSETS ’22, October 23–26, 2022, Athens, Greece
table when engaging with DIY practice and could ensure that DIY-
AT products are created to be safe and appropriate for extended
use, and (3) PTs had access to actual end users who could inform
the DIY maker process. They also identifed multiple barriers in
introducing PTs to digital fabrication. These included PTs having
limited time to learn new digital fabrication skills, the availability
of easily purchased devices that most PTs can default to rather than
creating new artifacts because of their limited time, and the lack of
tools to ensure standardized reliability of DIY-AT products, leading
to concerns about their liability.
More recently, several eforts have explored interdisciplinary
approaches for integrating digital fabrication in PT education. For
example, Wagner et al. described a project which connected PT and
OT students and faculty with librarians operating 3D printers on a
college campus [30]. The librarians were able to connect OT and PT
students to a group of biomedical engineering students to help with
3D modeling tasks. School faculty collaborated with the librarians
to develop assistive technology assignments for students that were
then printed at the library. Over three years, 78 students collabo-
rated with librarians and faculty to 3D print ATs. The stakeholders
used in-person and virtual meetings as well as email and shared
spreadsheets to keep track of projects [30]. This case study shows
that leveraging on services (e.g., librarians trained in 3D printing)
can be an efective way to introduce 3D printing in OT and PT edu-
cation through the creation of ATs. In another recent project, Davis
and Gurney assessed the impact of using 3D printing in project-
based OT learning and found that compared with a control group,
students who used 3D printing in their projects had increased tech-
nology self-efcacy [31]. While students were asked to consider
a “real-world” user of their designs, they did not interact or work
iteratively on their work and following a month-long module in 3D
printing were asked to design and print devices for their projects.
While both of these projects provide evidence that 3D printing can
be successfully introduced in the PT and OT classroom, they did not
provide details on challenges faced in implementing the programs
and communication strategies that proved successful or otherwise
with respect to specifc student projects.
Other work by Chen et al. elicited positive attitudes in OT clini-
cians towards integrating 3D printing into their practice, especially
with respect to the potential for reducing cost of customized items
and the potential for the customizability of digital fabrication to
fll in a gap in creating ATs [32]. The clinicians stressed that the
time to learn how to use 3D printing efectively can be a barrier
to adoption. While most previous eforts have focused on general
applications of 3D printing for PT and OT, Paterson et al. developed
and evaluated a computer-aided software design specifcally for
creating wrist splints [33]. The software and associated workfow
allow practitioners to customize a base model to match a client’s 3D
scanned wrist model and create a customized model of an aestheti-
cally pleasing splint. In interviews with 10 clinicians, they found
that users were able to navigate and use the software to create
models but also were concerned about capturing patient scan data
(needed as input into the software tool) and the material and cost
suitability of the designs (since the models need to be created on
higher end fabrication devices) [33].
Finally, several studies have focused on the existing practices
of OTs and PTs when adapting ATs. For example, in an interview
Erin Higgins et al.
study with 17 OTs, Afatoony and Shenai investigated how clini-
cians customize and adapt ATs using low-tech materials to better
ft the needs of their clients [34]. They found that while most OTs
tried to use of-the-shelf ATs, adaptations were often necessary to
ft a client’s needs. Most OTs described using low-cost craft ma-
terials (e.g., foam, duct tape, etc.) and techniques (e.g., gluing or
cutting to enlarge or reduce size, or to attach items together) to
make adaptations. They described a process where they would as-
sess a client’s need, make an adaptation for them, and then iterate
on it with the client to troubleshoot and refne it over time. Addi-
tionally, OTs mentioned having concerns around liability and safety
of modifed devices that makes them try to do minimal mechanical
adaptations and, for some, to avoid adapting ATs with electronic
components all together. In another study, where authors worked
with four OTs as digital fabricators over a period of four-months,
Hofmann et al. found a misalignment between the priorities of ATs
and digital fabricators/makers: specifcally, OTs prioritized client
safety over taking risks in fabrication and aimed to come up with
efective adaptations for their clients in one shot (rather than us-
ing iterations) [35]. Furthermore, they identifed a need to make
it clear what resources are available and what is feasible to the
clients, so they are not overreaching when using digital fabrication
methods. These fndings show that adapting and customizing ATs
is an important part of OTs practice and could be supported by
utilizing emerging technological tools and processes. Furthermore,
clinicians have concerns and priorities that may difer from makers
and digital fabricators.
Our work builds on this previous research by both recognizing
the potential of 3D printing to enhance the learning of OT and PT
students and by studying an approach to limit the need for time and
technical expertise on PT students when using digital fabrication for
AT development through connecting them to professional services
provided by staf at a community makerspace.
3 METHODS
3.1 Educational Series Design
We designed and implemented a digital fabrication educational
series for a single cohort of PT students that consisted of six face-
to-face learning sessions. This course series was modeled after an
earlier course designed by McDonald et al. [3] but there were three
major diferences. First, in contrast to the earlier course, CAD was
not attempted to be taught in this course, but simply introduced to
allow the PTs the ability to communicate with makers efectively.
Second, after the design with simulated end users, PT students in
our course designed with real end users. Finally, instead of the
research team 3D printing the design, they were fabricated by a
local community makerspace.
The frst session was a 1.5-hour introduction to AT with the
subsequent fve sessions involving AT development (modeling, re-
vising, reviewing) with the later fve sessions each spanning 3 hours
in length. All six sessions were conducted with the same cohort of
PT students over a total of 2 semesters within the PT educational
curriculum. During the initial sessions (2, 3, and 4), students de-
signed AT for simulated end users and during the later sessions
(5 and 6) they worked with real end users on AT devices. We will
describe the format and content of each phase next.
Creating 3D Printed Assistive Technology Through Design Shortcuts: Leveraging Digital Fabrication Services to
Incorporate 3D Printing into the Physical Therapy Classroom
Table 1: Simulated AT Design Cases (adapted from McDonald, et al. 2016)
Case Number Age Gender Diagnosis
1 62 Male Left Hemorrhagic stroke
2 45 Female Traumatic Brain Injury (TBI)
3 32 Male Humeroradial fracture
4 60 Female Right adhesive capsulitis
5 8 Male Cerebral Palsy
The fabrication for the educational learning series took place at
a local community makerspace stafed by youth employees with
expertise in 3D modeling and CAD, 3D scanning, 3D printing, and
other digital fabrication methods (e.g., laser cutting). The mak-
erspace houses a 3D print shop with multiple consumer-grade
mid-range printers (e.g., LulzBots, Prusas, and Ultimakers) that use
common plastic flaments including PLA and NinjaFlex fexible
flament.
3.1.1 Phase 1: Building Foundational Skills with Simulated End
Users. The six session, in-person educational series introduced PT
students to 3D printing and designing custom assistive technolo-
gies across two semesters of the PT curriculum, with the frst four
sessions being incorporated into a foundational PT course and the
last two sessions embedded into a clinical focused PT course.
The frst four sessions were embedded into a PT course designed
to bridge foundational sciences with introductory clinical skills.
For instance, in this course PT learners are exposed to principles
of biomechanics and the application of these concepts to assistive
devices. Session One began with a defnition of additive manufac-
turing and a description of current applications of 3D printing with
specifc examples of PT products (such as handgrips). The students
were then given an overview of the common types of 3D printers
(including SLA, SLS, and FDM) and their pros and cons. The stu-
dents then went through a detailed description of the process of
designing for 3D printing and a detailed description of additive
3D printers and how they work. Students were fnally asked to
complete a thought exercise in which they came up with an assis-
tive device that could be augmented by 3D printing. The intended
outcome of this session was an understanding of the process and
the limitations of 3D printing and to brainstorm some practical uses.
In the second session, the PT students were separated into 5 teams
and designed assistive devices for imaginary end user cases (Table
1). In these cases, the PT students were presented with scenarios
in which an end user needed an assistive technology device that
was no larger than 5 by 5 by 5 inches. This limitation was to ensure
that it could be printed on consumer-grade 3D printers that were
available at the makerspace.
The imaginary design cases were presented as shown in Table
1 below and adapted from McDonald et al. They were created
by the PT instructor and focused on designing augmentations to
ASSETS ’22, October 23–26, 2022, Athens, Greece
Specifcations
Requires use of hemiwalker; limited ability to grasp
with right hand
Requires use of a quad cane; limited by right wrist
fexor synergy
Requires use of auxiliary crutches for recent ankle
sprain; limited by right elbow fexion range
Requires use of straight cane for balance; limited by
shoulder fexion range
Balance issues; limited by fxed trunk fexion to 20
degrees; fxed trunk right side bend/rotation to 10
degrees
pre-existing devices. The scenarios refected the students’ learning
objectives in the course and, therefore, centered on walking aids
such as canes, crutches, and walkers. The cases refect common
pathologies and frequently used assistive devices. They provided
an opportunity for practical application of the student’s existing
knowledge and skills in the new domain of 3D printing. By utilizing
familiar content, we reduced cognitive overload.
The second session used the specifcations provided in the table
to fll out order forms (Table 2) that would be sent to the local mak-
erspace where staf printed prototypes based on their specifcations.
Question fve and question nine were added to this form based on
feedback after phase 1. In the third session, PT students presented
the frst version of their designs to the class and had an opportunity
to make fnal revisions by resubmitting revised order forms. In the
fourth session, the 3D printed models were delivered and evaluated
by the PT students.
3.1.2 Phase 2: Designing with End Users. For sessions fve and six,
PT learners were asked to apply knowledge from the frst phase of
the course to design AT devices for 12 volunteer end users (Table
3). Sessions 5 and 6 were incorporated into a course designed to
teach advanced concepts of clinical care across patient populations,
specifcally, with medically-complex patient populations. In session
fve, the PT students were broken into 12 teams and assigned to do
design and modeling sessions for a particular end user.
For these sessions, we kept the constraint of the maximum 5
by 5 by 5-inch dimension. Based on our observations from the
frst sessions, we reevaluated and refned the order form. The main
changes were to add two questions: one regarding the physical
properties and how the object should feel and one clarifying if this
object was to be attached to another object. This fnal version of
the order form (Table 2) was used to provide specifcations to the
community makerspace.
During this phase, students’ groups had the option of using
clay models to specify AT designs. If the group used clay models,
they were left to dry and then brought back to the print shop for
scanning using a NextEngine 3D Laser Scanner. Once the 3D model
was captured and refned by the staf, it was 3D printed. Some
groups had their designs printed based of of sketches and given
dimensions through the order form. Before the fnal review session,
all 12 teams were allowed an opportunity to make modifcations to
ASSETS ’22, October 23–26, 2022, Athens, Greece Erin Higgins et al.

Table 2: Questions from the Final Order Form Used in the Course

Question Number Question Text

1 Client description: age, diagnosis, expected use of device, expected impact of device use
2 Object Drawing: Please provide a drawing of your fnal object design (with dimensions in mm).
3 Surface Details: Are there any decorative aspects (surface details) that are not important to your
fnished print?
4 Material (circle one): STIFF FLEXIBLE
a. Please provide a detailed description for why your object should be stif or fexible?
b. Will this 3D printed object come in contact with your client’s skin? (Circle one): Yes/No
c. If yes, please describe where the client will touch it, and how much weight you think they will
put on it.
5 (not included in phase 1) Will the 3D printed object attach to an existing object (circle one) Yes/No
a. If yes, please describe that object and provide a drawing showing how the 3D printed part will
touch the other part
6 Density (circle one): HOLLOW PARTIALLY FILLED SOLID
b. How much weight will your object hold (e.g., in pounds)? How much stress will it be under?
7 Size: Should your printed object be the same size as your clay model? (Circle one): YES/NO
c. If no, should your model be scaled up or down? (Maximum dimensions of 139mm x 139mm x
139mm)
8 Color: What color is your object? (Circle one): Black/White
9 (not included in phase 1) Physical Properties: How should your printed object feel? Describe another object that has similar
physical properties (strength/fexibility/texture)
10 Additional notes

Table 3: End Users Information and Description of Diagnosis and AT Design and Use Description

Case Number Age Need for AT Expected Use Expected Impact

6 56 Grip issues when writing Increase size of pen/pencil Decrease efort exerted while
with pencil writing
7 74 Flexor Synergy Increase left wrist extension Increase wrist extension and
decrease fexor synergy
8 68 No Diagnosis To stabilize wrist Help her type, eat, shower, use
stairs
9 61 Cerebrovascular accident Finger extension Improve left fnger and wrist
(CVA) extension
10 66 Right stroke and left To allow left hand to open and Would allow user to restore some
hemiparesis supinate restoring some function functional use with grip strength
11 78 Right CVA Holding eating utensil Achievement of goal of using left
hand more
12 68 Decreased grip strength To open caps and lids Able to open jars and can
independently
13 72 Severe bilateral collapsed Daily in-shoe Reduce pain by straightening ankle
arch and supinate in his foot; relieve
pressure on bone spur callous
14 70 Extension of right Assist with opening of right hand Improve door opening and cooking
metocarpophalangeal joints
and proximal
interphalangeal joints
15 66 Lower grip strength and Wear on wrist to improve grip on Improved grip and steadiness
control due to stroke left side
16 63 Stroke; fexor synergy Extend fngers More functional use of arms
17 61 Stroke Help cut vegetables when cooking Helps with activities of daily living
Creating 3D Printed Assistive Technology Through Design Shortcuts: Leveraging Digital Fabrication Services to
Incorporate 3D Printing into the Physical Therapy Classroom
their models through the order forms again. When necessary, prints
were brought back to the makerspace and modifed or reprinted.
Almost all of the communication was through the order forms or
face-to-face with a minimal amount of communication happening
over email.
The fnal session of the second phase consisted of the fnal review
and AT delivery session with therapists and end users. During this
session, 3Doodler Pens were provided to student teams for any fnal
customizations. These pens allow users to draw a raised graphic
using PLA flament on an object’s surface. This allowed the students
to quickly add small additional changes to their devices without any
modeling or measuring. They could directly “draw” new additions
onto their devices. Only a small number of groups (between 1 and 3)
utilized this tool. They were also provided with extra components
such as Velcro which is common practice to include with AT devices
in the PT profession.
3.2 Participants
58 Physical Therapy (PT) students participated in the educational
series. Sessions 1-4 were conducted during the Spring semester of
their frst year of the PT curriculum and sessions 5-6 were completed
during the following Fall semester, correlating to the second year
of the PT program.
Twelve end users also participated in sessions 5 and 6 of the
series (Table 2). Finally, 12 youth (6 female, age range 14-18) staf
members at the community maker space participated in the study.
The youth had received training in 3D printing, 3D modeling, 3D
scanning, and other aspects of digital fabrication (e.g., laser cutting)
prior to this project and worked collaboratively on a part-time basis
in the community maker space primarily on 3D printing tasks for
between 3-12 months prior to this project. In this paper, we focus
on data collected from the PT students and will discuss data from
makers in the future.
3.3 Data Collection and Analysis
Our data consisted of PT student surveys to gain feedback about
their feelings on 3D printing being integrated into their profession
from both phases of the course, AT model order forms, AT design
documentation, and observations of the courses by researchers. We
have described the order forms and AT design documentation pre-
viously and, in this subsection, will describe the surveys collected
from the students.
During the frst phase, at the end of each session, the PT students
flled out a survey. Each survey served to evaluate how each student
felt about the diferent aspects of sessions. It helped to ascertain
their reservations and expectations about the 3D printing process.
It also contained their evaluations of the fnal products.
During the second phase of the course, because the PT students
had already completed a similar process with simulated end users,
only two sets of surveys were collected for this study. One was
used to collect data about the student’s experience in comparison
to simulated end users as well as how they feel about 3D printing
in their feld overall. The other survey was used to garner feedback
from the PT students and end users about the fnal products. Ob-
servations were utilized throughout both phases to capture verbal
ASSETS ’22, October 23–26, 2022, Athens, Greece
questions that the students raised throughout the course that were
not refected in the survey responses.
The data was analyzed using qualitative content analysis with an
inductive approach, as well as thematic analysis [36]. All collected
data was stored in a spreadsheet and coded according to repeated
themes brought up by PT students. The themes were abstracted
and labeled with a code. Fourteen diferent codes were used and
identifed. These 14 categories were used to identify six themes that
addressed most of the topics brought up by the PT students.
4 FINDINGS
Our fndings consisted of device design outcomes and PT students’
feedback on their experience with the learning material presented in
the sessions. We will frst present end products that were designed
by the PT students throughout the course followed by qualitative
fndings from student surveys and observations. It should be noted
that in direct participant quotes throughout the paper, PT students
refer to the end users as “patients” which is consistent with their
educational context. However, we will use the term “end user” in
our writing as it better refects a social model of disability.
4.1 Device Outcomes
At the conclusion of each course, all 5 student teams from sessions
2-4 for simulated patients (case numbers 1 through 5) and all 14
student teams from sessions 5-6 for end users (case numbers 6
through 17) had successfully created customized AT designs (Table
4). Small changes were required in almost every case mostly relating
to the attachment mechanisms or smoothness of the devices, as
seen in Table 4.
Overall, the designs were successfully printed in large part due
to the sketching skill of the PT students as seen in Figure 1. The PT
students were able to provide incredibly detailed technical sketches
with precise measurements to assist the makers in developing their
designs. This skill from the PT profession allows makers to easily
understand specifcations needed in the end product. Figure 1 shows
the sketch for case 2 that was the frst step in the development of
the printed device shown in Figure 2.
However, even with their sketching talents, PT students learned
the value of iteration when some of the devices were unusable after
their frst order form. For example, the frst print of their sketch in
Figure 1 was entirely unusable, as seen in the left-hand image of
Figure 2.
This failure was caused by attempting to use cheetah flament for
printing. They originally thought this would be an ideal material
because of its fexibility, but it led to the failed print. They had
to re-evaluate their techniques to better align with the skills of
the makers and the use case of their design and ended up with a
successful product at the end of phase 1 as seen in the right-side
image of Figure 2 by changing the flament type.
Despite all end devices being determined as successful, some
devices, for example as seen in the top right image of Figure 3,
were printed in such a way that they could not fully be used by
the end user due to lack of ft. However, the devices in the top left
and bottom images of Figure 3 were all more appropriate and ftted
properly.
ASSETS ’22, October 23–26, 2022, Athens, Greece Erin Higgins et al.

Table 4: Final product description and PT student comments on the 3D prints for imaginary (phase 1) and real (phase 2) users

Case Product Created PT Student Comments End user comments

Number
1 Custom ftting hand grip for a
hemi-walker
2 A bowl for someone to use a cane with
their elbow
3 Custom elbow splint that can attach to
crutches
4 Shoe wrap to improve balance
5 Forearm rest on a walker
6 Custom grip for pencil
7 Curved splint with holes for straps
8 Wrist splint
9 Finger straps to help increase
fexibility
10 Wrist splint with holes for straps
11 Larger handle that attaches to fork for
eating
12 Y-shaped handle that attaches to a jar
that requires lateral force instead of
twisting to open
13 A shoe insert for heel
14 Prints that ft over joints on the hand
to allow for training to open hand
wider
15 Cylinder weights that allow patient to
strength build throughout their day
16 Splint to assist in extending fngers
17 Customized extended grip holder that
can ft silverware
Worried gravity efected their clay model
and that the device wouldn’t ft on the
hand of a hemiwalker
Worried it is not strong enough to be
weight bearing
The overall size would need to be bigger
to ft an adult patient
Side loops for the Velcro aren’t big
enough
It is only 70% of the size of their model so
the dimensions are of and the clay model
captured many imperfections
Would round edges more if given time;
would want a “squishier” material
Round the edges and cut more room for
the thumb
Smooth rough edges; design change to
hollow if given time
The fngers were not wide enough; wrong
color
Slightly too small
The angles did not match the design so it
wasn’t properly ergonomic
The material needs to be more fexible
and a hole needs to be drilled to attach a
handle
The material was too rough and there
were concerns about breakdowns with
continuous use
Smoothness was great just needed bigger
holes for attachments
Better print than they were expecting
Need a hole for attachments
Need to adjust hole size for attachments
N/A – simulated case
N/A – simulated case
N/A – simulated case
N/A – simulated case
N/A – simulated case
He is hopeful that this will relieve the
tiredness and aching in his hand
allowing him to write better
N/A – did not share feedback
This will give her more confdence in
holding the railing while going up
stairs; needs smoothed for comfort
The stretch felt so good; needed to be
smoothed for comfort
Too small and does not believe it can
sustain enough pressure to pull him
out of pronation; generally, very
curious about 3D printing, however,
and willing to try again
If the design is not correct in 8 weeks,
he would be uninterested
She is hopeful that it will allow her to
open all jars and bottles easily
because that is tough for her
Looking forward to the potential pain
relief; needs smoothed for comfort
Really excited; exactly what she
wanted
Comfortable and stable enough to
wear and not interfere with his
activities; ideas are worth printing as
a prototype to see if the idea is
feasible
N/A – did not share feedback
Exactly what was expected and she is
hopeful that it will help her efectively
cut fruits and vegetables safely
Creating 3D Printed Assistive Technology Through Design Shortcuts: Leveraging Digital Fabrication Services to
Incorporate 3D Printing into the Physical Therapy Classroom ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 1: Example of sketching skills from Group 2 in phase 1 depicting a bowl designed to be comfortable for an elbow and
attachable to a cane.

Figure 2: (Left) This image shows a failed cheetah print of an elbow bowl for case 2 in phase 1 and (Right) This image shows the
successfully printed version of the image on the left.

4.2 PT Student Reactions to Working with Real with real end users made the process more meaningful in compar-
End Users ison to simulated users for the PT students and made them more
motivated to produce successful results. Student 20 stated, how-
After completing designs for simulated and real end users, the stu-
ever, that this added some stress by saying, “There’s more pressure!
dents flled out a survey asking for their opinions on the process.
[You] don’t want to disappoint. An emotional aspect has been added”.
When asked to compare the experience to designing for simulated
While many felt this emotional pressure, real end users helped the
patients, most PT students said that designing with a real end user
learning process. Student 33 stated, “It is defnitely more challenging
was helpful to their design process and led to more impactful AT
working with a real patient but it allows us to get real feedback on
device creation. For example, Student 24 stressed the importance of
things that will beneft the patient”, showing how feedback from end
the relationship with the end user stating, “I am more motivated to
users was found to be very important. Many PT students also found
make an object that will really make a diference for my patient be-
cause I am forming a relationship with him.” Furthermore, working
ASSETS ’22, October 23–26, 2022, Athens, Greece Erin Higgins et al.

Figure 3: (Top Left) The pen grip designed for and being used by end user 1. (Top right) the hand splint designed and being used
for end user 5. (Bottom Left) The strength building arm device designed for and being used by end user 10. (Bottom right) The
device to train users to open their hand wider being used by end user 9.

that designing for real end users helped them to form a more real-
istic understanding about what designs could actually be created
utilizing the methods that they were learning. Student 49 stated,
“Some things that seem functional or well designed in simulation don’t
work out or show faws when applied to a real patient. Working with
a patient shows how to create a more realistic design”.
4.3 Challenges Experienced by PT Students
Communicating with Makers
PT students identifed a series of challenges in using both paper
forms (Figure 4) and clay models (Figure 5) to communicate with
makers. Many of these comments point to a need for face-to-face
meeting and discussion with makers. While CAD was introduced
to the PT students in initial sessions of the project, PT students
found it incredibly difcult to learn how to efectively communicate
about CAD designs in a short time. Therefore, all PT students relied
entirely on drawings and clay models for the development of AT
with end users.
With respect to the paper forms, most of the feedback pointed to
not knowing the best way to communicate what needed to be done
to the makers because of lack of personal connection and shared
language. Student 2 stated after phase 1, “It really seems like some
groups benefted from having their design tweaked by a CAD program.
So without a good working relationship with a computer engineer or
a personal knowledge of CAD program this seems like a difcult and
clumsy venture”. As seen in Figure 4, the forms contained places
for drawings, measurements, materials, etc. but the PT students
felt that the drawings and precise measurements were the most
important. Student 2 stated, “I don’t think the designers were able to
use our drawings, measurements, and written descriptions. I would
like to have seen what could be produced by our written plans. If I had
to guess, more views of the designed device may have been useful”.
PT students also felt that face time with the makers would have
greatly benefted them, student 19 stating, “I think you just need to
be as specifc as possible. Maybe it would help to talk directly to the
individual printing”.
A specifc example of a communication breakdown can be seen
in Group 4. This group’s printed project had fnger holes that were
too small. A possible reason for this was because in their order form
they simply gave a list of diameters in centimeters. Not only were
dimensions asked for in millimeters, but they also never specifed
if this was to be in the inner or outer diameter of the fnger holes.
Their sketch is shown with these specifcations in Figure 4 below.
With this lack of understanding, the makers made assumptions that
were incorrect.
We found that using clay for 3D modeling was a great way to engage
the end users and provided an easy way to get exact measurements
in the moment. However, there were major issues with digitizing
the clay model and turning it into a 3D printed object. A major
issue was the shrinking and deforming of clay models while drying.
For example, Student 6 explained, “I found that the clay was not
super helpful in the transition to the 3D printer. The clay tends to
lose its shape during the drying process”. There were also complaints
about the surface issues created by scanning clay. When using
Creating 3D Printed Assistive Technology Through Design Shortcuts: Leveraging Digital Fabrication Services to
Incorporate 3D Printing into the Physical Therapy Classroom ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 4: The sketch depicting a drawing of a hand with measurements for each fnger hole that was provided by Group 4 to
guide the makers in printing

Figure 5: (Left) Clay model of an attachable device for a walker that has deformed slightly because of the method used for
modeling an object that is heavy. (Right) A clay model of a pencil grip that has held its shape well because of the small
dimensions and uncomplicated nature of the item.

clay, modeling smaller devices, such as the pen grip seen in the
right image of Figure 5, were very successful because they did not
sufer some of these deforming issues. Whereas bigger objects, such
as the walker addition see in the left image of Figure 5, sufered
from some of the issues with shrinking while drying. These issues
led to measurements and sketches being the most reliable way to
communicate their device needs.
Overall, we saw that while the paper forms and clay models each
ofered diferent possibilities for engaging PT students, end users,
and makers but they had limitations that ultimately led to a lack of
a robust shared language and communication approach. Because of
the lack of face-to-face time and shared language, communication
between the PTs and the makers required a large number of specifcs
communicated through the forms and clay objects. Some of the
most successful products utilized multiple communication methods
to develop exact specifcations for the makers. The clay model seen
in Figure 5 on the left-hand side can be seen in Figure 6 being used
to generate exact measurements for a sketch on the paper form.
This combination of sketching, measuring, and clay modeling
allowed the students to create a very smooth and successful fnal
product as shown in the right-hand images of Figure 6. While the
clay model deformation led to a product that was unusable, the
sketching and modeling together helped to form a usable fnal
product.
Overall, the PT students felt there was a disconnect between
what they thought they were communicating to the 3D printer oper-
ators and what they interpreted. PT student suggestions to address
this were to have more face-to-face time with makers, more fre-
quent communication, and a better way to communicate. Another
suggestion was that more knowledge of CAD might be helpful for
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 6: (Left) The failed clay model from fgure 5 is being measured and used to create a detailed sketch also shown in the
image. (Right) The fnal 3D printed item that is smooth and attached to a walker device.
communication. This lack of competency with technology also led
to mismatches between what the PTs were imagining and what
was possible to achieve with 3D printers.
4.4 Observations on Design Process
Several PT students identifed the mismatch between the engineer-
ing design process and the PT clinical process because in practice,
to get feedback for each design iteration, they would be required to
have repeated end user visits. PT students observed that they felt it
would be difcult to bill for multiple iterations of the same product
or repeat visits for iterating on their design. Student 16 summarized
the issues stating, “The design process is very similar to the traditional
engineering design process...time constraints and the number of avail-
able patient visits (insurance-wise) could be a challenging aspect of
actually using 3D printing for devices for real patients”. The iterative
nature of the making process is at odds with the time-pressured
and insurance-run nature of healthcare. It led to some frustrations
amongst the groups when they were not able to actually achieve
their full vision. Some PT students ofered suggestions about what
might help ease this tension including ensuring that end users were
more communicative in the short time they spent together. Student
7 stated, “I think having a patient that explicitly states what they
want help in will help in the creative process”. The lack of end user
availability means that the communication between the PTs and
the end users’ needs to be informed and accurate the frst time so
that they are able to fully execute their design with the makers.
After being introduced to the concepts of 3D modeling in phase
1 of the course, many PT students had a moment of realization
that the CAD modeling might be too difcult to learn on top of
their other curriculum but that there are experts out there whose
services can be leveraged. Student 18 stressed the importance of
these experts stating, “I’ve felt like 3D technology was incredibly
Erin Higgins et al.
useful for PT throughout all workshops. Seeing our failed attempts
made me realize how detail-oriented 3D printing is and how difcult
it can be. I’ve also learned that even though there’s a lot of obstacles it
most defnitely is possible to perfect.” Perfecting technology literacy
to facilitate communication with makers as well as ensuring that
PT students are aware of the possibilities with 3D printing could be
an important next step in furthering the potential of 3D printing in
AT. Student 5 expressed that the knowledge of what is possible is
an essential next step for the relationships between AT and making
to fourish stating, “Yes, it is a cool tool that we can hopefully use
to help patients in the future. I would like to see examples of more
products and become more familiar with items that are possible to see
what works. The failed prints allowed us to see what doesn’t work or
what should be reconsidered.”.
After the course, we found an interesting diference between PT
students in their knowledge of the prospects of 3D printing and
how many materials were available. For example, Student 17 stated
that, “There are many materials available, which allows us to tailor
our device to the likes of our patient”. However, Student 30 stated
“3D printing has limitations and I wish there were more options to
incorporate other materials”. This mismatch in understanding of the
availability of materials is an indication that there might be some
additional information about materials that would be helpful to
impart to the PT students so that they all have a full understanding
of what can currently be done with this process.
4.5 Final Products and Appropriate Use Cases
PT Students provided detailed feedback on the process of using
digital fabrication to develop and create AT devices. Many of them
saw the potential but just didn’t feel that the digital technology
was where they needed it to be to make exactly what they wanted
due to the difculty of the software and the time needed to iterate
Creating 3D Printed Assistive Technology Through Design Shortcuts: Leveraging Digital Fabrication Services to
Incorporate 3D Printing into the Physical Therapy Classroom
and create projects. Some acknowledged the limitations of the
technology but instead ofered some solutions for them. Nine PT
students had specifc suggestions for appropriate contexts and how
3D printing could be used in the meantime. For example, Student
3 was very specifc in stating, “I think that 3D technology is most
applicable for patients that require customization for the grip and fne
motor control but not for general adaptive technology. For example,
I think our elbow cup was not a good device to 3D print whereas
the customized walker grip on case number 1 was ideal and patient
centered”.
Multiple PT students brought up how the printing might have
made more sense as an addition to already created devices. For
example, Student 2 stated, “Perhaps attaching the device to the cane
would be more efective using some sort of commercially available
hardware”. This could mean having AT on hand to build of of
and specify from or printing something similar and then adding
these customized components as Student 3 stated would be helpful,
“I think our device could beneft from a non-3D printed component
to afx it to the quad cane. I would have liked to modify the AT in
addition to just afxing the device – we really wanted to screw into the
cane itself”. All of them said it was benefcial to have the old device
with them for reference, usually along the lines of how Student
18 stated, “I think [makerspace staf] having access to the assistive
devices we used to make our design [would be] incredibly helpful”.
Considering this feedback, it might be helpful to encourage an
understanding of of-the-shelf devices for the makers so that they
can just add additions to standards instead of having to create a
device entirely by themselves.
Despite the challenges, 37 of the PT students expressed that they
were excited to see where the use of digital fabrication in AT was
going but not as excited about their current outcomes, with Student
49 expressing, “[3D printing] will be more and more useful as printers
and materials improve”. They seemed to believe that only signifcant
improvements in technology could get them where they wanted
to be in terms of designing items with 3D printing but saw the
potential. For example, Student 56 stated, “Your imagination is your
limit”. Overall, following the course the PT students were able to
identify what the shortcomings for this type of device development
were and ofered suggestions of how to make more appropriate
projects.
4.6 Expected Material and Time Costs
The PT students were asked to share what they would be willing
to pay for 3D printing services and the amount of time that they
were willing to wait for the fabrication process to be completed.
This question was not asking about the amount of time the PTs
themselves would spend, but the amount of time they were com-
fortable waiting for a device to be printed. Twelve PT students
said they would be willing to pay any cost for a device that was
perfectly suited to the needs of their end user. Otherwise, they said
that they would be willing to pay 40.81 dollars on average with the
median response being 25 dollars. Student 56 who said there was
no limit, stated, “It could change our patients’ lives. She said within
minutes that the mid/stretch felt ‘so good’. That is invaluable”. Of the
PT students included, on average they said they would be willing
to spend 345.34 hours with a median of 168 hours to develop these
ASSETS ’22, October 23–26, 2022, Athens, Greece
products. PT students generally do not have much formal education
about the cost of customization and it is something they learn from
real-world clinical practice. Therefore, the PT students did not have
much practical knowledge on which to base these estimates.
End users also gave feedback on the amount they would pay and
the time they were comfortable spending. Four end users stated
that they would spend any amount of time necessary to have a
custom device. However, most end users had a cost limit that was
based of of previously paid for AT.
4.7 Liability Concerns
During observations, several PT students expressed concerns about
liability issues stemming from the types of materials available to
3D print with. However, no students brought up liability in their
written responses, possibly due to the format of our surveys. The
questions brought up in the classroom were in regards to concerns
about who is liable if these products were to break or injure an end
user. In one survey, Student 46 also expressed fears about device
failures due to materials and ill-ftting devices stating that in regards
to 3D printing in PT, “[Their] opinion has grown, but I feel specifcs
are very important to ensure a reliable print with limited failures.
Failures could become costly for patients”. There was an apparent
fear that the materials available were not exactly appropriate in the
medical context. Though not much data was collected in regards to
overall student opinions about liability, the concerns brought up
by the students are valid and need to be addressed in order to fully
integrate 3D printing into PT practice.
5 DISCUSSION
Conducting the course provided insights into the possibilities and
challenges that exist when PT students, makers, and end users work
together to use digital fabrication tools to create customized AT. We
found that working with end users and outsourcing the fabrication
was overall successful at providing PT students with an opportunity
to utilize digital fabrication tools and techniques in their learning.
Compared to previous work, where PT students learned how to use
3D modeling and printing tools [3], outsourcing the digital design
and fabrication to a community makerspace addressed many of
the issues PT students faced. By creating this connection between
makers and PTs, our approach eliminated the need for either group
to become experts in each other’s felds. Instead, they developed a
shared language to communicate their needs to each other. While
our study showed that developing and efectively using this shared
language is non-trivial, it is a promising direction and can equip
all stakeholders with relevant interdisciplinary competencies. Out-
sourcing the 3D modeling adds an overhead of communication,
however, and training clinicians in fabrication skills has value that
needs to be balanced with other factors when outsourcing fabri-
cation. Based on these fndings, a promising future direction to
focus on is developing training programs that provide both PT
students and makers with a shared interdisciplinary knowledge
and language that can be leveraged for using efective and efcient
communication protocols. In the following subsections, we will
discuss lessons learned from the educational series in more detail.
ASSETS ’22, October 23–26, 2022, Athens, Greece
5.1 Involving All Stakeholders
A key aspect of our approach was including real end users in the
second phase of the project. We found including end users efective
at motivating PT students in creating functional and safe AT devices.
After iterations, all AT devices designed in the sessions incorpo-
rating end users were evaluated as successful. Furthermore, PT
students described working with actual clients as valuable, mean-
ingful, and “real”. While previous research has shown that clinicians
have concerns about device safety and liability if injury occurs and
if the materials used in digital fabrication is inappropriate for ther-
apeutic or medical use [3, 5], we saw that these considerations
were heightened for our participants in the later sessions of the
educational series when they were working with real users.
To address concerns about safety and liability, AT project se-
lection needs to be carefully considered. If a design case has the
potential, if broken or used over time, to cause injury, the appro-
priateness of using 3D printing to create it needs to be approached
with caution. In this study, the project that was the most successful
was also happened to have the lowest liability (e.g., the pen grip).
Working with real users can emphasize the importance of AT de-
vice safety, not only in the context of the PT classroom but in any
context, such as online communities or community makerspaces,
where digital fabrication methods may be used to create ATs. In
the future, more input from PT experts on the type of ATs that are
appropriate for digital fabrication could help inform future itera-
tions of similar courses and programs, and also inform community
DIY-AT eforts.
5.2 Developing Interdisciplinary Competencies
One of the key fndings from our study was the need to develop an
interdisciplinary shared language between makers and PTs. While
in the fnal sessions of the course, all AT devices were evaluated
as successfully designed, throughout the process, PT students ex-
pressed challenges in communicating the characteristics of the
devices they designed to makers. Compared to previous research
which required PT students to use CAD software and engage with
3D printing directly [3], our study facilitated the process of digital
fabrication for PT students and allowed them to focus more on AT
specifcations through methods they are already familiar with (e.g.,
sketching and drawing on paper). PT students were very skilled
and able to provide very detailed sketches to the makers. PT lead-
ers are very hands-on and much of their training focuses on the
development of psychomotor skills in manipulating objects with
their hands. They were able to utilize these skills to make use of
the clay for modeling as well. However, efectively communicat-
ing the design with clay in precise enough language to produce a
perfect product remained challenging. This paper identifes smaller
objects that are molded directly to a patient as more appropriate for
clay modeling while large objects, especially with unequal weight
distributions, were less successful and might be better suited for
sketching and measurements.
These challenges can be addressed in the future in several ways:
frst, more detailed information about the fabrication and design
process can be provided to PT students as part of the training. For
example, a set of sample 3D printed ATs, including failed ones, can
be used to demonstrate the possibilities and challenges of using
Erin Higgins et al.
these techniques for AT development. Conversely, an overview of
common PT terminology used to describe and evaluate ATs can be
provided to makers to help develop a common language. Makers
would need to have an understanding of what of-the-shelf AT
looks like and the basics of the PT practice in order to more quickly
understand what they are being asked to create.
Second, a combination of digital and paper forms combined with
a mechanism for continuous asynchronous communication (e.g.,
a Slack channel [37]) between makers and PT students can be es-
tablished to provide detailed and frequent feedback on designs and
fabrication iterations. These forms can provide a template with ex-
plicit felds for the most important measurements and descriptions.
Another possible facilitator to communication could be creating
a shared VR workspace that would allow the makers and the PTs
to send CAD fles to each other and interact with them in a more
tangible way before printing.
Third, in line with what McDonald et al. also recommended
previously, creating a base set of 3D printed AT designs to start
from and build of, in combination with detailed documentation
about each design’s purpose, consideration, and possible variations
would be incredibly benefcial [3]. In addition to capturing existing
knowledge, this document can be informed by expert clinician and
maker perspectives.
5.3 Leveraging Academic Training Programs to
Connect Makers and Clinical Students
An important aspect of our project was to connect university PT
training programs to community organizations. Our project pro-
vides an example of how community resources, such as makerspaces
or other youth technical learning programs, can connect with uni-
versity programs to form mutually-benefcial relationships. Our
research team comprises experts in physical therapy, digital fabrica-
tion, and community engagement and the interdisciplinary nature
of our collaboration as well as the range of resources and relation-
ships available to use facilitated the design and implementation of
the project. Several co-authors had a long-term working relation-
ship with the community makerspace, and another research team
member was a core faculty member, with lead teaching responsi-
bility for the PT training program. Researchers and practitioners
considering setting up similar future programs should consider
the need for long-term relationships between all organizations in-
volved and identify clear roles, timelines, and expected outcomes
for everyone involved. In our case, working out these details were
essential to project success.
While community engagement was central to our project, an
alternative confguration can leverage the increasing number of
universities that are creating makerspaces on their campuses [38].
These spaces provide ample opportunities for interdisciplinary col-
laboration across diferent academic programs, such as PT and
engineering, and can be sites of AT innovation and development.
Introducing engineers to the AT design process can ofer mean-
ingful and motivating experiences for diferent students in these
programs and lead to more sophisticated designs that draw on
skills from multiple engineering disciplines, for example to also
add sensors to AT devices. Such collaborative programs can beneft
Creating 3D Printed Assistive Technology Through Design Shortcuts: Leveraging Digital Fabrication Services to
Incorporate 3D Printing into the Physical Therapy Classroom
by ongoing input from existing medical professionals in interdis-
ciplinary roles such as Assistive Technology Professionals (ATPs)
or experts who generally have both engineering and medical skills.
Further collaborations can include connections with hospitals and
therapy centers.
6 LIMITATIONS AND FUTURE WORK
This course was run with one group of PT students over one aca-
demic year and needs to be verifed with more students in the
future. The specifc characteristics of the community makerspace
that was stafed by trained youth may have impacted some of the
project outcomes and in future work we plan to study how these
factors may be diferent if collaborations are with other types of
community organizations or digital fabrication services. The com-
munity makerspace we worked with was stafed by youth makers
who while trained and knowledgeable in fabrication were not as
experienced as professional industrial designers, engineers, or other
professional fabricators.
Future work can explore how working with other organizations,
for example makerspaces at universities or colleges or online profes-
sional services, such as Shapeways, would impact the process and
outcomes. Given the PT student feedback on wanting to work with
more materials and fabrication techniques, in the future, it would
be helpful to collaborate with multiple makerspaces or fabrication
facilities that may provide a wider range of options for the students
to work with. For example, the use of thermoplastics or two-part
modeling materials could alleviate some issues with clay shrinking.
Furthermore, this study took place before the COVID-19 pan-
demic and most design communication between makers and PT
students was done through order forms or face-to-face. In future
iterations of the course, the learned familiarity with online confer-
encing tools (such as Zoom) can potentially help to facilitate these
interactions. For example, video conferencing tools can be lever-
aged to communicate about designs. This will be especially helpful
in the future if individuals choose to design projects at a larger
scale or that contain external components or perform mechanical
functions.
A challenge of 3D printing in the PT classroom is that it is difcult
to get end-users back for multiple iterative cycles, especially in this
study as the end users were volunteers, many of whom had limited
access to transportation and technology. Increasing the frequency
of in-person visits would have created an undue burden on the
volunteers and digital communication as an alternative would have
been a challenge. In the future, we would like to explore facilitating
more collaboration between makers, PT students, and end-users.
We also plan to gather more feedback from all three stakeholders
in order to get a fuller picture of the whole design process.
Finally, in this study, we worked with PT students rather than
expert clinicians and while student perspectives provide valuable
insights into what may be relevant in PT practice, inquiring into
the perspectives of expert clinicians in the future can further enrich
this research area.
7 CONCLUSIONS
This study has continued investigating the efectiveness of cus-
tomized assistive technology developed through a collaboration
ASSETS ’22, October 23–26, 2022, Athens, Greece
between PT students and makers. It has demonstrated the potential
for this collaboration as one way to address many issues encoun-
tered when attempting to teach PTs to design and fabricate these
devices independently [3]. This study further worked to identify
specifc barriers to 3D-printing adoption in PT and some acceptable
use cases. Specifcally, we highlight the need to involve all stake-
holders in the process of custom AT development and the need
to develop interdisciplinary competencies to further facilitate this
relationship. We also discussed an important fnding in regards
to the best communication methods for this design process. We
highlight some cases in which clay modeling is the most efcient
and in which paper forms and measurements are the most efcient
method of communication. In all cases, however, we point to the
importance of face-to-face communication is for makers and PTs
and how these connections might be facilitated by universities. In
the future, we hope to continue teaching 3D printing classes to PT
students, as well as expand to a variety of medical professionals. We
also hope to develop the tools and competencies needed to make
3D printing a seamless part of medical practice.
ACKNOWLEDGMENTS
This work is supported by the National Science Foundation under
Grants DRL-2005502, DRL-2005484, and EEC-1623490. We would
like to thank the PT students, instructors, and clinicians from UMB
who participated in our study. We would like to thank the Digital
Harbor Foundation for providing the space, materials, machines,
and student workers to facilitate printing the devices for this study.
We would also like to thank the Adaptive Design association, the
Shield, and the Ability Project, Anita Perr, Shawn Grimes, and
Darius McCoy.
REFERENCES
[1] Mankof, J., et al., Consumer-Grade Fabrication and Its Potential to Revolutionize
Accessibility. Communications of the Acm, 2019. 62(10): p. 64-75.
[2] Buehler, E., Branham, S., Ali, A., Chang, J.J., Hofmann, M.K., Hurst, A., and Kane,
S.K. Sharing is caring: Assistive technology designs on Thingiverse. in 33rd Annual
ACM Conf. on Human Factors in Computing Systems. 2015.
[3] McDonald, S., Comrie, N., Buehler, E., Carter, N., Dubin, B., Gordes, K., McCombe-
Waller, S., and Hurst, A. Uncovering challenges and opportunities for 3D printing
assistive technology with physical therapists. in 18th Intern. ACM SIGACCESS Conf.
Computers and Accessibility. 2016.
[4] Parry-Hill, J., Shih, P.C., Mankof, J., and Ashbrook, D. Understanding volunteer
AT fabricators: Opportunities and challenges in DIY-AT for others in e-NABLE.
in 2017 Conf. Human Factors in Computing Systems. 2017. ACM.
[5] Hofmann, M., Burke, J., Pearlman, J., Fiedler, G., Hess, A., Schull, J., Hudson,
S.E., and Mankof, J. Clinical and maker perspectives on the design of assistive
technology with rapid prototyping technologies. in 18th Intern. ACM SIGACCESS
Conf. Computers and Accessibility. 2016. ACM.
[6] Gordes, K.L.M.W., S. , Novel partnerships for interprofessional education: A pilot
education program in 3D technologies for human centered computing students
and physical therapy students. Journal of Interprofessional Education & Practice,
2019. 15: p. 15-18.
[7] Okoro, C.A., et al., Prevalence of Disabilities and Health Care Access by Disability
Status and Type Among Adults - United States, 2016. MMWR Morb Mortal Wkly
Rep, 2018. 67(32): p. 882-887.
[8] Assistive Technology. 2018; Available from: https://www.who.int/news-room/fact-
sheets/detail/assistive-technology.
[9] Investing in Innovation – Government Funding R&D. 2010; Available from: https:
//www.oecd.org/site/innovationstrategy/45188215.pdf.
[10] Phillips, B. and H. Zhao, Predictors of assistive technology abandonment. Assist
Technol, 1993. 5(1): p. 36-45.
[11] Hocking, C., Function or feelings: factors in abandonment of assistive devices1.
Technology and Disability, 1999. 11(1-2): p. 3-11.
[12] Sugawara, A.T., et al., Abandonment of assistive products: assessing abandonment
levels and factors that impact on it. Disabil Rehabil Assist Technol, 2018. 13(7): p.
ASSETS ’22, October 23–26, 2022, Athens, Greece
716-723.
[13] Presti, A.L., Scherer, M.J., and Corradi, F., Measuring the assistive technology match,
in Assistive Technology Handbook, Second Edition. 2017, CRC Press. p. 53-70.
[14] Toro, M.L., et al., Type and Frequency of Reported Wheelchair Repairs and Related
Adverse Consequences Among People With Spinal Cord Injury. Arch Phys Med
Rehabil, 2016. 97(10): p. 1753-60.
[15] Worobey, L., et al., Increases in wheelchair breakdowns, repairs, and adverse con-
sequences for people with traumatic spinal cord injury. Am J Phys Med Rehabil,
2012. 91(6): p. 463-9.
[16] Worobey, L., et al., Diferences between manufacturers in reported power
wheelchair repairs and adverse consequences among people with spinal cord
injury. Arch Phys Med Rehabil, 2014. 95(4): p. 597-603.
[17] McClure, L.A., et al., Wheelchair repairs, breakdown, and adverse consequences for
people with traumatic spinal cord injury. Arch Phys Med Rehabil, 2009. 90(12): p.
2034-8.
[18] Hurst, A.T., J. Empowering individuals with do-it-yourself assistive technology. in
13th international ACM SIGACCESS conference on Computers and accessibility
(ASSETS ’11). 2011. Association for Computing Machinery.
[19] Meissner, L.J., Vines, J., McLaughlin, J., Nappey, T., Maksimova, J., and Wright, P.
Do-It-Yourself Empowerment as Experienced by Novice Makers with Disabilities. in
Proceedings of the 2017 Conference on Designing Interactive Systems. 2017.
[20] Hamidi, F., et al., TalkBox: a DIY communication board case study. Journal of
Assistive Technologies, 2015. 9(4): p. 187-198.
[21] Profta, H.P., Stangl, A., Matuszewska, L., Sky, S., and Kane, S. K. Nothing to
Hide: Aesthetic Customization of Hearing Aids and Cochlear Implants in an
Online Community. in Proceedings of the 18th International ACM SIGACCESS
Conference on Computers and Accessibility (ASSETS ’16). 2016.
[22] Hamidi, F., Mbullo, P., Onyango, D., Hynie, M., McGrath, S., and Baljko, M. Par-
ticipatory design of DIY digital assistive technology in Western Kenya. in Second
African Conference for Human Computer Interaction: Thriving Communities
(AfriCHI ’18). 2018. Association for Computing Machinery.
[23] Buehler, E., Kane, S.K., and Hurst, A. ABC and 3D: Opportunities and obstacles to
3D printing in special education environments. in 16 th Intern. ACM SIGACCESS
Conf. Computers & Accessibility. 2014. ACM.
[24] Buehler, E., et al., Investigating the Implications of 3D Printing in Special Education.
Acm Transactions on Accessible Computing, 2016. 8(3): p. 1-11.
[25] Lindtner, S., Bardzell, S., and Bardzell, J. Reconstituting the utopian vision of
making: HCI after technosolutionism. in 2016 Conf. Human Factors in Computing
Systems. 2016.
Erin Higgins et al.
[26] Hook, J., Verbaan, S., Durrant, A., Olivier, P., and Wright, P. A study of the chal-
lenges related to DIY assistive technology in the context of children with disabilities.
in DIS’14. 2014.
[27] Afatoony, L., and Lee, S. Codea: a Framework for Co-Designing Assistive Tech-
nologies With Occupational Therapists, Industrial Designers, and End-Users
With Mobility Impairments. in Design Society: DESIGN Conference. 2020.
[28] Afatoony, L.L., S. AT makers: a multidisciplinary approach to co-designing
assistive technologies by co-optimizing expert knowledge. in 16th Participatory
Design Conference 2020. 2020.
[29] Hansen, A.K., et al., Exploring the Potential of 3D-printing in Biological Education:
A Review of the Literature. Integr Comp Biol, 2020. 60(4): p. 896-905.
[30] Wagner, J.B., et al., Three professions come together for an interdisciplinary
approach to 3D printing: occupational therapy, biomedical engineering, and
medical librarianship. J Med Libr Assoc, 2018. 106(3): p. 370-376.
[31] Davis, K. and L. Gurney, Impact of 3D Printing on Occupational Therapy Student
Technology Efcacy. International Journal of Technology in Education and Science,
2021. 5(4): p. 571-586.
[32] Chen, D., et al., Three-Dimensional Printing in OT: Increasing Integration Into
Practice by Countering Barriers. American Journal of Occupational Therapy, 2021.
75.
[33] Paterson, A.M., et al., Computer-aided design to support fabrication of wrist splints
using 3D printing: A feasibility study. Hand Therapy, 2014. 19(4): p. 102-113.
[34] Afatoony, L. and S. Shenai, Unpacking the Challenges and Future of Assistive
Technology Adaptation by Occupational Therapists, in CHItaly 2021: 14th Bian-
nual Conference of the Italian SIGCHI Chapter. 2021, Association for Computing
Machinery. p. 1-8.
[35] Hofmann, M., Williams, K., Kaplan, T., Valencia, S., Hann, G., Hudson, S.E.,
Mankof, J., and Carrington, P. . "Occupational Therapy is Making" Clinical Rapid
Prototyping and Digital Fabrication." in Proceedings of the 2019 CHI Conference on
Human Factors in Computing Systems. 2019.
[36] Miles, M., Huberman, A. & Saldana, J. , Qualitative Data Analysis: A Methods
Sourcebook. 2014, Thousand Oaks, CA: Sage.
[37] Easley, W., Hamidi, F., Lutters, W. G., & Hurst, A. Shifting Expectations: Un-
derstanding Youth Employees’ Handofs in a 3D Print Shop. in ACM Human
Computer Interaction CSCW. 2018.
[38] Barrett, T., Pizzico, M., Levy, B. D., Nagel, R. L., Linsey, J. S., Talley, K. G., Forest,
C. R., and Newstetter, W. C. A Review of University Maker Spaces. in ASEE Annual
Conference. 2015. Seattle, WA.
 BentoMuseum: 3D and Layered Interactive Museum Map for
Blind Visitors
Xiyue Wang Seita Kayukawa
Miraikan – The National Museum of Miraikan – The National Museum of
Emerging Science and Innovation Emerging Science and Innovation
Tokyo, Japan Tokyo, Japan
wang.xiyue@lab.miraikan.jst.go.jp seita.kayukawa@lab.miraikan.jst.go.jp

Hironobu Takagi Chieko Asakawa

Miraikan – The National Museum of Miraikan – The National Museum of
Emerging Science and Innovation Emerging Science and Innovation
Tokyo, Japan Tokyo, Japan
hironobu.takagi@miraikan.jst.go.jp chieko.asakawa@miraikan.jst.go.jp

Figure 1: BentoMuseum, a 3D and layered design of a museum map that makes information accessible to visually impaired
visitors. (a) All foors can be stacked or separated. (b) A user taps the interactive label, which responds with an audio guide when
the foor is overlayed on an iPad app. (c) A user explores a structural attraction with fngers (a circular walkway named Oval
Bridge which goes around a “globe-like” display named Geo-Cosmos). (d) The Oval Bridge and Geo-Cosmos in the museum.
ABSTRACT CCS CONCEPTS
Obtaining information before a visit is one of the priority needs and • Human-centered computing → Accessibility systems and
challenges for blind museum visitors. We propose BentoMuseum, a tools; • Hardware → Tactile and hand-based interfaces; • Social
layered, stackable, and three-dimensional museum map that makes and professional topics → People with disabilities.
complex structural information accessible by allowing explorations
on a foor and between foors. Touchpoints are embedded to provide KEYWORDS
audio-tactile interactions that allow a user to learn the museum’s information access, 3D structure, audio-tactile, touch screen
exhibits and navigation when one foor is placed on a touch screen.
Using a tour design task, we invited 12 frst-time blind visitors to ACM Reference Format:
Xiyue Wang, Seita Kayukawa, Hironobu Takagi, and Chieko Asakawa. 2022.
explore the museum building, chose exhibits that attracted them,
BentoMuseum: 3D and Layered Interactive Museum Map for Blind Visitors.
and built a mental map with exhibit names and directions. The In The 24th International ACM SIGACCESS Conference on Computers and
results show that the system is useful in obtaining information that Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New
links geometric shapes, contents, and locations to then build a rough York, NY, USA, 14 pages. https://doi.org/10.1145/3517428.3544811
mental map. The connected foors and spatial structures motivated
users to explore. Moreover, having a rough mental map enhanced
1 INTRODUCTION
orientation and confdence when traveling through the museum.
As audience-centered institutions with a range of educational and
social roles, museums are, more than ever, aware of the impor-
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed tance of delivering equality, diversity, and inclusion, as well as their
for proft or commercial advantage and that copies bear this notice and the full citation power to bring about positive change in society [13, 33, 34]. In-
on the frst page. Copyrights for components of this work owned by others than ACM creased attention is being paid to the development of equal access
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a and multidimensional access, which not only refers to physical
fee. Request permissions from permissions@acm.org. access but also to multi-sensory, intellectual, fnancial, emotional,
ASSETS ’22, October 23–26, 2022, Athens, Greece cultural, educational as well as information access [2, 12, 24], all
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00 of which are important to the visually impaired community. At
https://doi.org/10.1145/3517428.3544811 present, many barriers impede the attempts of people with visual
ASSETS ’22, October 23–26, 2022, Athens, Greece
impairments to visit a museum. Alongside exhibition accessibility
and mobility assistance [2, 3], the limited provision of information
and orientation before a visit can lead to a negative overall experi-
ence and emotional isolation [2, 10]. Nevertheless, making detailed
and usable information available would create a welcoming and
encouraging environment to blind visitors [30].
The museum’s unique environment brings challenges to blind
people in accessing information, diferent from other public spaces
such as neighborhoods, parks, and hallways in a building. First, con-
temporary museums have distinctive architecture, internal design,
and “inter-foor structures” (i.e., stairs and walkways connecting the
foors, see an example in Fig. 1d) as a part of their exhibitions [31, 43].
Blind visitors have difculty grasping the museum’s complex “mul-
tidimensional information”, such as the overall geometric shape of
the building, the inter-foor structures, and each exhibit section’s
name, description, approximate shape and size, and location. Sec-
ond, even though some museums have a minimal structure with
a pre-defned route, more museums contain freely arranged ex-
hibits that may or may not have clear route indication [14]. In such
museums, visitors explore the museum and choose the exhibits
they examine based on their interests. By encouraging such “free
explorations”, these museums efectively trigger a sighted visitor’s
curiosity, but conventional posted information may cause blind
visitors access difculties and orientation frustrations.
Accessible maps are the means for visually impaired visitors to
learn about a site. Tactile maps are often available in public spaces
and institutions to provide information and assist in navigation,
with the aim of helping the user to build a mental map before going
to a new place [45, 46]. Since the efectiveness and understandabil-
ity of a tactile map largely depend on the user’s tactile skills and
abilities [38], three-dimensional (3D) maps with volumetric symbols
have been developed for ease of understanding. Moreover, audio-
tactile labels have been proposed for seamless and autonomous
operation. The user is thus largely freed from either shifting atten-
tion between the map and the braille legend [25] or asking others for
explanations [16]. The current 3D-printed audio-tactile maps show
thrilling possibilities, but limitations persist. These maps usually
present a simple one-foor layout, which is insufcient to support a
structural mental map of a multidimensional museum.
Due to the fact that a museum contains a large amount of mul-
tidimensional information, and it is not a frequently visited place,
blind visitors might feel it’s particularly challenging to obtain infor-
mation, orient themselves, and build a mental map. Consequently,
they refuse to do this and give up the idea of a self-reliant visit. How
can museums that have a complex 3D structure and freely arranged
exhibitions provide information access to blind people before a
visit? To bridge the gap between museums and blind visitors in
terms of information access, and to investigate the suitable format
of an accessible and inclusive museum map, the following research
questions emerge:
• RQ1. How can we make the vast amount of needed infor-
mation (e.g., architecture and interior structures, exhibits,
facilities, locations, and route-fnding) accessible and under-
standable on a museum map?
• RQ2. Is building a mental map possible and signifcant in the
museum context?
Wang, et al.
Previous literature either developed foor plans in isolation [20,
29] or reproduced external structures [26, 50], in which complex
multi-foor structures such as museums were rarely explored. Our
core concept and innovation is the design of stackable 3D foors to
capture the complex multidimensional nature of a museum. The
multidimensional information includes the external and internal
structures, exhibits and facilities, and their locations as well as
route-fnding. Using a participatory and user-centered approach,
we designed BentoMuseum, a 3D and layered museum map with
audio-tactile interactions, to support blind users in obtaining in-
formation and understanding the 3D attractions through tactile
explorations (Fig. 1). The system contains two main elements: the
3D and layered foors (Fig. 1a), which can be either interlocked to
allow vertical exploration between foors (Fig. 1c) or separated to
support horizontal exploration of a single foor; interactive touch-
points on the foor that allow audio feedback by touch (Fig. 1b).
When one foor is placed on a touch screen, diferent levels of infor-
mation and tactile navigation with audio support can be triggered
by tapping. Novel designs we propose include stackable foors, 3D
and 2D attributes that represent diferent types of contents, and
simulated navigation by tracing paths and intersections.
We invited 12 participants with severe vision impairment to be
museum tour designers and instructed them to use the system as
part of an authentic museum tour. We let them explore as much
as possible, obtain information, choose exhibits of their interest,
and try to build a mental map. Participants expressed their map
exploration styles and elaborated on their needs for information
access before, during, and after the tour. Our results suggest: (1)
Using the system, the participants were able to actively obtain
information that links shape, location, and content. Consequently,
they were able to choose exhibits of interest and build a rough
mental map. (2) Touching inter-foor structures motivated blind
users to explore the museum map. Along with the navigation, it
supported them in building a 3D mental map. (3) Building a rough
mental map beforehand was benefcial for the subsequent visit. It
provided orientation, enhanced the sense of safety and confdence
that they would not get lost while traveling through the museum,
and led to a positive and inclusive museum experience.
2 BACKGROUND AND RELATED WORK
2.1 Museum Accessibility
Museums are not only institutions for the collection, preservation,
and display of valued objects but also audience-centered spaces
with a wide range of social roles and responsibilities [2, 6, 27, 48].
Sandell suggested that museums should contribute to social inclu-
sion on individual, community, and society levels by supporting
creativity and confdence; empowering independence, decision-
making processes, and democratic structures; fostering acceptance
and respect; and challenging stereotypes [41].
However, barriers exist when people with visual impairments
attempt to visit a museum. It was noted that a blind person’s visit
experience begins even before entering the museum site [48]. The
provision of information has been a priority need in terms of ser-
vice accessibility [11, 23, 48]. A United Kingdom survey concluded
that the basic accessibility information provided by a majority
of museum websites could not address the access needs of blind
BentoMuseum: 3D and Layered Interactive Museum Map for Blind Visitors
or partially sighted persons who want to plan an independent
visit [10]. Argyropoulos et al. found that complex museum architec-
ture/interior design also hindered access to a museum. The overall
inaccessibility led to a lack of motivation and negative emotions [2].
It was noted that providing haptic, touchable, and multi-sensory
objects is one of the most anticipated and efective ways of improv-
ing museum accessibility [2, 9, 51]. In terms of accessing informa-
tion and orienting visually impaired visitors, Vas et al. suggested
presenting the museum space and exhibition through audio at the
beginning of the visit [49]. Tactile maps and 3D models have also
been developed to the needs of visually impaired visitors. Urbas et
al. explored 3D printing techniques to develop physical foor plans
to be mounted on the museum walls for touch [47]. Holloway et
al. created 3D maps with distinct icons for an event, and they sug-
gested that large maps require time to explore and thus should be
made available before an event or at the entrance in a comfortable
environment [26]. Leporini et al. pointed out that being able to
explore and familiarize themselves with the structure and details of
a large cultural site is crucial for orienting visually impaired people.
This allows them to gain a global understanding and an overall
impression [29].
2.2 Accessible Maps for the Visually Impaired
Maps that enable tactile explorations are designed for the visu-
ally impaired to learn the environment. One of the most common
methods is tactile maps created with a set of accessibility guide-
lines [4, 35, 36]. Such maps represent features with raised lines, sym-
bols, keys, and orientation [38, 55], often with the same elevation
(around 0.5mm) on swell paper or a greater height diference (up to
2-3cm) when thermoformed [25, 39]. Most tactile maps are created
for information provision in a digestible form that can be easily
understood [39, 40, 55] and for mobility and navigation [25, 40, 46].
Although they are commonly used, tactile maps propose a set of
challenges to visually impaired users. They are limited in creating
3D structures and a variety of heights [25]. The understandability of
a tactile map also largely depends on the user’s tactile ability [38, 40]
and training in the skill of reading tactile graphics [1, 45, 46]. These
maps also contain a relatively small amount of information, and nor-
mally a large area needs to be divided into diferent sections [38, 55].
Furthermore, the tactile maps need supplementary means to access
the information, for example, a symbols glossary or legend [38].
To mitigate the understandability issues, research has explored
maps with more distinct 3D structures. Voigt and Marten suggested
the use of 3D models of buildings to facilitate spatial orientation
and build a mental map [50]. Leporini et al. developed foor plans
of indoor monuments of a cultural site to help both visually im-
paired and sighted people explore and familiarize themselves with
elements before a visit [29]. Gual et al. found it was easier to mem-
orize 3D volumetric symbols than 2D symbols [18]. Comparing
two tactile maps, one with only 2D elements and the other also
including volumetric symbols, Gual et al. found the use of 3D vol-
umetric symbols signifcantly reduced both location-fnding time
and discrimination errors [19]. Holloway further compared the
readability of diferent 3D icons on a map and proposed guidelines
for icon design [26]. Pistofdis et al. tested a number of parameters
related to 3D shape and haptic performance [37]. Holloway et al.
ASSETS ’22, October 23–26, 2022, Athens, Greece
compared 3D printed maps with tactile maps and found that 3D
maps were preferred. The use of more easily understood icons and
relative heights of map elements facilitated improved memory in
short-term recall [25]. Gual et al. designed urban maps containing
both volumetric attributes and relief attributes and demonstrated
the value of the maps in terms of interpreting, memorizing, and
understanding. However, they also found that the maps required
verbal support to be used autonomously [20, 21].
2.3 Touch Sensing and Audio-Tactile Labels
Audio-tactile maps using touch sensing or buttons have been pro-
posed to further support understanding in addition to the tactile
sensations. Brock et al. designed an interactive map composed of
a multitouch screen, a raised-line overlay, and audio output. Com-
pared with tactile maps, they showed that replacing braille with
audio-tactile interaction signifcantly improved efciency and sat-
isfaction [7]. Comparing tactile maps with interactive small-scale
models (SSMs) for learning, Giraud et al. showed that the interac-
tive SSMs improved both space and text memorization and were
also adaptable to diferent situations and needs [16]. Several stud-
ies demonstrated that perceptible buttons that invoked diferent
levels of audio content promoted an interactive, autonomous explo-
ration [25, 29] and increased emotional engagement [52]. Utilizing
printing technologies, automatic creation of 3D printable tactile
graphics, and touch screens, research has developed the ability to
instantly produce tactile-audio representations on a printed map
by implementing the map on a touch screen [17, 44, 53].
The previous literature has built a strong foundation for devel-
oping 3D maps with audio-tactile labels. However, those works
focused on one-foor settings with relatively small amounts of in-
formation. To the best of our knowledge, few research works have
explored accessible maps for complex multi-foor structures. We
fll this research gap by proposing stackable foor maps to access
both the internal and external structures of an entire museum.
3 PARTICIPATORY SYSTEM DESIGN
The design concept is implemented in a science museum, Miraikan
– The National Museum of Emerging Science and Innovation1 , with
distinctive structure and symbolic interior attractions. It is a seven-
foor building with a large-area atrium (with the 2nd, 4th, and 6th
foors mainly atrium space) and structural attractions such as a
series of escalators that directly connect all the foors (Fig. 3a), a
walkway called Oval Bridge that goes around a “globe-like” display
named the Geo-Cosmos (Fig. 1d)2n, and a Dome Theater with half
of it inside the building and the other half extended into the exterior
(Fig. 3b). It also lacks maps that can be perceived by touch.
We employed a participatory and user-driven methodology to
design a map adapted to the museum. The design sessions include
seven interviews with the blind designer (once in prototype 1, three
times in prototype 2, and three times in preparing for the fnal
design), one event that involved twenty blind museum visitors
and three staf members, and one group meeting with those staf
members.
1 https://www.miraikan.jst.go.jp/en/
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 2: The printout of an early iteration and the fnal design. (a) The realistic model of the 3rd foor that was used in
Prototype 1 to start the discussion (Section 3.2). (b) The 3rd foor of the fnal model, with simplifed designs, audio-tactile labels
painted in black, and control buttons on a touch screen. (c) The fully stacked fnal model that shows the external structure.
3.1 Motivation: “What if I can open the model
and get information about the foors?”
One of the designers, P0, is an adult female who has complete
blindness, as well as being an interaction designer and researcher.
After being presented a 3D model of the museum, she expressed the
need to understand the interior: “I have heard about the symbolic
globe-like display and the Oval Bridge around it. But it’s so hard
to imagine them just through descriptions. I wish I could open the
model and touch them” 2 . This was the initial attribute of the map we
hoped to investigate: a 3D model that contains internal structures.
3.2 Prototype 1: Feedback of a Realistic Model
An early version of the map was a realistic 3D print of the museum
foor (Fig. 2a). We sliced the 3D model into foors and encapsulated
the detailed information such as the walls and tables at each exhibit
to a 24cm×14.5cm×1.2cm miniature foor. A tactile map resembling
the 3D map’s layout was developed and printed on swell paper
for comparison. During a two-day event called Inclusion Week,
two maps along with other 3D prints were explored in the wild
by 20 blind visitors, for 5 to 10 minutes each person. From their
comments, we learned the following needs to satisfy in developing
an understandable map:
• Content: Simplifed and categorized forms were needed.
Users highly praised the understandable form of the Oval
Bridge on the 3D map but also pointed out that the detailed
depictions of exhibits were not digestible.
• Tactile exploration: A relatively smooth surface without
acute edges was preferred. Small and pointy objects (i.e.,
walls and tables) on the 3D map hindered hand scanning.
• Explanations: Automated audio-tactile interactions were
desired. Both maps were not understandable unless the mu-
seum staf gave explanations.
The two maps were then tested by P0 during a 30-minute interview.
Further requirements were confrmed based on her knowledge of
the museum and expertise in design:
2 All communication with the participants was in their native language. In this paper,
we present any translated content in the form of “translated content.”
Wang, et al.
• Facilities: While the current maps were highly focused on
the exhibits, the basic map elements such as restrooms, ele-
vators, and escalators also needed to be included.
• Orientation and navigation: The map should support iden-
tifying the entrance, main route, each exhibit, and how to
move around the space. These elements support the devel-
opment of a “mental map,” which is crucial for blind people.
The feedback revealed that the 3D map was good at delivering
structural impressions while the tactile map preserved scanning,
thus confrming the previous research [25]. Previous literature
suggested 3D maps with audio-tactile labels ofer clear advantages,
including understandability, memorization, and efectiveness, all of
which are important for museums. Nevertheless, we did not fnd
a clear advantage to using the tactile map for the museum, so we
were motivated to focus on 3D maps. We utilize the advantages
of both volumetric attributes and relief attributes, learning from
previous practices [18, 19, 25, 26, 29] while making our own design
innovations and adaptations.
3.3 Prototype 2: 3D Floors and Audio-tactile
Interactions
Based on the feedback, we categorized the museum’s multidimen-
sional information into the following three types of information,
and we provided design criteria for each of them:
• Structural attractions include inter-foor structures and
symbolic spatial structures (Fig. 1c, 1d, 3a, 3b). Our design
choices include: (1) Simplifying structures into primary forms
with understandable relative scales. For example, the parallel
escalators and stairs were made simpler into one slope with
textures (Fig. 3a). (2) Simplifying prominent walls into 1mm
tall and 3mm wide cuboids. (3) Embedding magnets to sup-
port easy stacking and lining up of foors, which has proved
to be efective in developing 3D objects for the blind [15].
Floors can be partially stacked to simulate how to walk
between them (Fig. 3a) or fully stacked to show a facade
(Fig. 2c).
• The exhibits included booths, wall-divided spaces, and arti-
facts placed in open spaces. Our design choice was to simplify
them into outlines that were proportional to the real space
BentoMuseum: 3D and Layered Interactive Museum Map for Blind Visitors
Figure 3: Designs for diferent types of information. (a) The escalators and stairs run parallel in the museum (left) and their
representations on several foors (right). (b) The Dome Theater (left) and its representations on several foors (right). (c) One
exhibit area (left) and it’s representation on one foor (right). (d) Eight symbols that represent museum facilities.
they took (Fig. 3c). This design supports clear separation, dif-
ferentiation, and scanning. The outlined shape was hollowed
to enable audio-tactile interaction (described later).
• The facilities in the museum were summarized into eight
frequent items. We represented them using volumetric sym-
bols (Fig. 3d), with design guidelines from previous work
[18, 19, 26]. For those facilities that take a large space (e.g.,
lobby and restaurant), their outlines were hollowed out to
show the area and enable touch interaction.
To support orientation and navigation on the map, we further de-
fned paths and intersections to indicate how the user can travel.
• The path is similar to the tactile paving [32], which suggests
a route on open ground. According to the actual layout and
fow, we defned a main path in the center of the exhibition
space and sub-paths as routes that connect each exhibit’s
entrance to the main path. All of the paths are represented
by 1mm wide embossed lines (Fig. 2b).
• The intersection is represented as a 10mm × 10mm hol-
lowed square located at the crossing of the paths, which is
distinguishably smaller than the exhibit areas (Fig. 2b).
To automate the explanations with diferent levels of detail, we
implemented audio-tactile labels using capacitive sensing on a
touch screen. A 12.9-inch iPad Pro was used as a platform to sense
touch. When a foor is placed on it, a touch can be sensed directly on
the hollowed exhibits. On a structural attraction with a geometric
shape (Oval Bridge, Geo-Cosmos, Dome Theater), the audio-tactile
label was implemented by redirecting touch from the screen to the
surface of the shape using conductive ink, following touch screen
redirection technical guidelines [42, 54]. A 3.5mm wide tube was
cut out in the geometric shape, flled with the conductive ink, and
had its top and bottom painted with conductive ink. We also pasted
a 4mm wide circular tactile sticker at the center to indicate the
ASSETS ’22, October 23–26, 2022, Athens, Greece
touchpoint (see an example in Fig. 1c). To tactually distinguish
hollowed interactable exhibits from the atrium, we attached a pa-
per with textures on the back of the foor (Fig. 2b). An app that
processes touch and provides voiceover information was developed
in Unity. As learned from the previous work [7] and through our
own testing, we mainly adopted a double tap as the recognized
touch to prevent accidental triggering during the exploration. Two
modes were developed to serve the needs of free exploration and
route-fnding:
• In the Exploration mode, double-tapping a touchable area
triggers the audio explanations.
• In the Navigation mode, the user double-taps two exhibits
to select the destination and the start. A route with a start,
destination, and a number of intersections in-between is
generated. Next, The user is instructed to move a fnger
to the entrance of the start place, which is the location in
the exhibit reached by the path. Once the user moves there
(without any tapping), she is directed to trace the path to the
next location of the route until she reaches the destination.
The fnal foors are created with a stacked area of approximately
32cm × 20cm × 13cm stacked, and 2.5cm tall, 1.5mm thick each foor
(Fig. 2c), which was at a 1/400 scale of the actual museum (see
specifc sizes and details in Fig. 4). This is the largest size that can
ft onto an iPad to support audio-tactile exploration. It is designed
in Autodesk Fusion 360, and printed with Formlabs Form 3L SLA
3D printer, using Clear Resin material.
3.4 Final Design: Content and Customization
We then conducted a 1.5-hour group meeting with 3 museum staf
members, who are not only profcient as museum guides but also
experienced in guiding blind users. We decided to include the fol-
lowing contents:
ASSETS ’22, October 23–26, 2022, Athens, Greece Wang, et al.

Figure 4: The layout of all the 3D foors. The x and y axes are in cm. Diferent colors marked the foors, facilities, and paths.

• The audio guide for the 3D structure or the exhibit, in Table 1. They were recruited via an e-newsletter for people with
which speaks at one of two levels of detail in turn when visual impairments, and compensated $75 plus travel expenses for
tapped. The frst level contains name, keyword (e.g., uni- their time. All of them were frst-time visitors who held minimal
verse, earth, life), and accessibility info (e.g., “Over there, you preset knowledge about this museum where the study took place.
can touch a 3D model of the rocket engine.” ) The second level Six participants were frequent museum visitors who visited other
contains a 15-second description about it. museums more than once a year. Three visited other museums
• The audio guide for an intersection, which speaks the every two to three years, and three rarely visited a museum. All
surrounding information when tapped (e.g., “This intersection of them had experience with tactile materials, including tactile
is connected to an earth-type exhibit on the top and a universe- graphics (P2, P4–P12) and 3D models (P1–P11).
type exhibit on the bottom. Eight exhibits are on the left. Five
are on the right.” ) Table 1: Demographic information of the participants.
The following updates were made to enable user customization:
• Three physical buttons (stop the voiceover, modify the speed, Blind Navigation Visiting
ID Age
and change Exploration/Navigation mode) were developed. since Aid other Museums
They are clipped onto the iPad and can be triggered at any
P1 58 41 Guide Dog Once every 2–3 years
time using a double-tap (Fig. 2b).
P2 49 45 White Cane 2–3 times/year
• In the Navigation mode, the route explanation style can be
P3 63 50 Guide Dog Once every 2–3 years
switched between the turn-by-turn (default) and the north-
P4 60 45 White Cane Once every 2–3 years
up navigation.
P5 42 32 White Cane 2–3 times/year
In summary, all elements in our proposed map are as follows: P6 49 16 White Cane Never
(1) the 3D and layered foors, which include inter-foor structural P7 57 53 White Cane 2–3 times/year
attractions, the outlined exhibits, and facilities shown by volumetric P8 24 3 White Cane Once/year
symbols; (2) the audio-tactile interactions, which include the two- P9 71 60 White Cane 2–3 times/year
level audio guide of exhibits, an audio guide at the intersections, P10 43 3 White Cane A few times
and navigation by tracing the paths and intersections. P11 61 56 White Cane 4 times/year
P12 68 35 Guide Dog Never
4 USER STUDY
We conducted a user study at the science museum1 to investigate
our research questions and evaluate the efectiveness of our pro-
posed system. The staf who joined the fnal design process (Sec-
4.2 Task and Procedure
tion 3.4) stressed that visitors came with diferent interests and 4.2.1 Pre-Interview. The frst part of the study, tour design, took
expectations. A fxed task and a rigorous evaluation of the perfor- place in a guest room located on the frst foor of the museum.
mance might discourage the participants, who are also important Before presenting them the system, we conducted a roughly 10-
stakeholders. We came to agree that a tour design task should be minute pre-tour interview, hearing about their tactile experience
fexible to refect diferent user styles and support curiosity and and previous preparations before going to other museums.
autonomy, which are the museum’s important social roles. We in- 4.2.2 Structural Exploration. Next, we presented the fully stacked
cluded a tour after the map exploration to help visitors generate BentoMuseum 3D model and allowed the participant to explore
feedback towards a real museum visit. Each individual study took freely by touch as we introduced the basic external structures. We
two to three hours in an order of tour design, conducting the tour, informed participants that the museum had the shape of a boat, and
and post-tour interview. the front “bow” should be kept on the left-hand side for a consistent
orientation. We then introduced the “Bento Box” characteristics
4.1 Participants and encouraged the participant to take the foors apart one by one.
We recruited 12 blind participants (male = 5, female = 7) with ages Next, the foors were stacked back one by one, and we encouraged
ranging from 24 to 71 years old (mean = 53.8, SD = 13.1), as listed the participant to touch the inter-foor structures (e.g., escalators
BentoMuseum: 3D and Layered Interactive Museum Map for Blind Visitors
and Oval Bridge) to learn how foors are connected. Finally, the
participants were primed with a list of 10 icons (eight in Fig. 3d,
escalator and wall), which is separately prepared on a sheet. This
phase took roughly 10 minutes.
4.2.3 Training Phase. A training phase was conducted to familiar-
ize participants with the audio-tactile interactions. The participant
was shown the 1st foor map on the touch screen in Exploration
mode. The following steps were taken: (1) double-tap the special
exhibit zone to listen to the name; (2) double-tap it again to hear
the details; (3) double-tap the intersection to hear information of
surroundings; (4) fnd the guest room by double-tapping; (5) double-
tap the speed buttons to adjust voiceover speed; (6) double-tap
the cancel button to stop the audio; (7) double-tap the navigation
button to change to the Navigation mode; (8) double-tap to set the
special exhibit zone as the goal and the guest room as the start, and
trace the route following the audio guide. This entire process took
around 10 minutes.
4.2.4 Tour Design Task. A loosely structured tour design task was
conducted. The individual participant was asked to imagine the
following real-world scenario: The system is placed at the entrance
of the museum, and they are using this system to select the exhibits
of interest and design a unique tour for themselves. With the help
of the staf, they can place any foor on the touch screen. From a
total of 28 exhibits, they were asked to select 6 (equivalent to a two-
hour tour) and to try to build a mental map with routes connecting
the spots within 45 minutes. Considering the real-world scenario,
they were also free to take notes. During the task, a researcher was
taking notes of the selected spots for later evaluation. We video and
audio recorded the session and saved app log data for later analysis.
When time was up or the participant was fnished, they were
asked to orally explain the (1) name and (2) orientation and location
of each spot. Based on their explanations after the task and during
the tour, we determined which level of mental map they possessed.
In this study, we defned fve levels of the mental map:
• Level 1: Hardly remember any exhibits they chose.
• Level 2: Remember some of the exhibits they chose.
• Level 3: Remember all of the exhibits they chose.
• Level 4: Remember all of the exhibits they chose and which
foor each exhibit is on.
• Level 5: Remember all of the exhibits they chose and the
location of each exhibit.
We also asked the participants to give a self-evaluation of what
level of the mental map was needed.
4.2.5 Conducting the Tour. To validate their mental map in a real-
world setting and gather feedback on the important factors before
visiting the exhibitions, we invited the participant to experience the
designed tour. We shortened the two-hour tour to approximately 15
minutes, the time allotted for walking over the designed tour with
a museum guide and listening to elaborated guidance of a chosen
exhibit to get a taste of the museum. We encouraged participants to
concentrate on validating their mental map, and they were allowed
to fully explore the museum after the study.
4.2.6 Post-Tour Interview. A roughly 30-minute interview was con-
ducted after the participant was settled back in the guest room. The
ASSETS ’22, October 23–26, 2022, Athens, Greece
interview included two forms: a seven-point Likert rating, from
strongly disagree (score = 1) to strongly agree (score = 7), and
free responses. Four sections composed the interview: (1) rating
the overall experience of using the system (Q1–Q3 in Fig. 5); (2)
rating the overall system usability (Q4–Q6 in Fig. 8); (3) rating the
3D foors and audio-tactile interactions, which were further divided
into eight specifc elements in terms of A. understandability and B.
usefulness (Q7.A–Q14.B in Fig. 9); (4) free responses about using the
map prior to the visit, the strengths and limitations of the system,
applications, and other fndings after the tour. For all of the ratings,
we asked participants to consider or imagine accessing information
by audio means, such as reading a homepage when preparing for a
visit, as a baseline (score = 4).
5 RESULTS
5.1 Preparations before Going to the Museums
Eight out of nine who had visited museums said they would gather
information ahead of time through web pages and other means.
The information they hoped to collect included exhibit information
(all), foor information (P2, P3, P7, P8, P10, P11), the detailed route
to get to the exhibits (P7, P8, P11), and facility information such as
restaurants (P4, P5) and opening hours (P5). Two participants said
it was difcult to acquire information they needed through web
pages, so they made calls (P5, P7).
5.2 Performance of Information Access
5.2.1 Overall Experience. All participants successfully fnished the
tour design task within the allowed time (45 minutes). The ratings
related to information access through the task are summarized in
Fig. 5 (Q1–Q3). The participants strongly agreed that by using the
system they could get an overall image of the museum (median =
7). They also agreed that they were able to grasp the details of the
museum (median = 6). All participants agreed (median = 7) that it
is important to decide on their own where to go. Participants were
excited about having good control of information and being able to
design the tour independently based on their own interests:
A1:“The museum visits are precious parts of my life. I really
don’t want to miss anything interesting. Thus I want both inde-
pendent exploration and recommendations." P9
A2:“It might be nice if there were a recommended course, but I
would still like to explore it myself. There is a sense of security
to control where I go." P11
5.2.2 User Exploration Styles. By analyzing the double-tap log data
during the Exploration mode, we identifed several hand movement
styles when the participants were exploring the foor. Research has
found that touch readers were taught to frst systematically scan in
a circular pattern, and this efective scan strategy was used with
both tactile map and 3D models [25]. We were interested in learn-
ing whether did they naturally perform this exploration strategy.
Although all participants had tactile experience, we identifed that
three participants (P8, P9, P12) under-explored the foor (see ex-
ample in Fig. 6a). They only touched some of the exhibits, and no
circular pattern was formed. Three participants (P3, P5, P11) over-
explored the foor (see example in Fig. 6b). Their tapping covered
most exhibits, but their fngers traveled randomly by long distances
ASSETS ’22, October 23–26, 2022, Athens, Greece Wang, et al.

Figure 5: Questionnaire results of the overall experience of our system (Q1–Q3).

(a) Under-exploration (P8, 3rd foor) (b) Over-exploration (P3, 3rd foor)

(c) Typical (P4, 3rd Floor) (d) Ideal (P4, 5th foor)

Figure 6: Diferent exploration styles. The location of frst level touches and their sequences during the Exploration mode are
plotted on a 2D foor map. The arrow indicates the consequence, and the color shows the touch order.

on the map. A clear, circular pattern was hardly observed. The rest
of the participants showed a typical style (see example in Fig. 6c).
Even though they had slightly long traveling, they were able to build
a relatively circular and systematic pattern. In latter exploration,
one participant (P4) also showed an ideal style (see example in
Fig. 6d) with a very systematic circular pattern. After one exhibit
was explored, he moved to the closest exhibit. This style exhibited
the “circular and complete” scanning strategy, which was noted as
one of the most efcient strategies for exploring a map [25].
We next analyzed the relationship between the identifed styles
and the quantitative performance, and the latter was used to identify
exploration efectiveness on tactile graphics [5]. These quantitative
measures include coverage percentage of the exhibitions, tap counts
(perceived as a more reliable indication of time in our case compared
to the clock time), and exploration distance. We computed a Univari-
ate ANOVA with a Bonferroni Pairwise post-hoc test to compare
the results. Mean coverage for under-exploration was signifcantly
lower than those for typical (p < .01) and over-exploration (p <
.001) (Fig. 7a). There was no signifcant diference among the three
styles in terms of tap count (Fig. 7b). Mean exploration distance
for over-exploration was signifcantly higher than those for typical
(p < .01) and under-exploration (p < .01) (Fig. 7c). The results show
BentoMuseum: 3D and Layered Interactive Museum Map for Blind Visitors
Figure 7: Mean (a) coverage, (b) tap count, and (c) exploration distance for the typical, over-explored and under-explored groups.
p: p-value of the Univariate ANOVA with a Bonferroni Pairwise post-hoc test (*** and ** indicate the 0.001 and 0.01 levels of
signifcance, respectively).
that among all the styles, those participants who hold a typical style
are having high coverage and short exploration distance, which
indicated relatively efcient explorations.
5.3 Performance of Mental Map Building
Among twelve participants, nine participants (75%) remembered
all the exhibits they chose and their locations (level 5), two (16.7%)
remembered the exhibitions and their corresponding foors (level
4), and one (8.3%) remembered a part of the chosen exhibits (level
2). Level 5 participants could explain the general location of each
exhibit (e.g., “Exhibit A is on the upper-left side of the 5th foor
or “Exhibit B is located to the left as you exit Exhibit C), and we
determined that they had built a “rough” mental map.
All participants noted that there could be a clear diference be-
tween with and without a mental map, and some commented that
3D and layered foors and the Navigation mode of the proposed sys-
tem were efective for building a 3D mental map (see Section 5.4.2
and Section 5.4.3). The participants noted that there was an improve-
ment in orientation (P2, P6, P9, P10) and confdence that they were
safe and would not get lost (P7–P9, P11) with a rough mental map
during the visit, compared to their previous museum experiences
without a mental map.
A3:“ Without the mental map, I didn’t understand where I was
walking. Controlling where I was going using the map, I walked
with a sense of accomplishment, and everywhere I went became
a lot more fun." P2
A4:“ Without a rough mental map, it just feels like being pulled
around and it’s simply boring and tiring." P9
A5:“ If I don’t have a mental map [before following a course], I
don’t remember where I went, I don’t know how long I will walk.
Now when I notice where I am, I can calculate back from the
mental map, and I feel a completely diferent level of security." P8
A6:“ It feels safe to decide where I go, understand the relative
locations, and have a structure of the museum in my mind. It
makes the tour and the discussion easier. I might make mistakes
about the route, but I can soon integrate the new information
and easily correct my map." P7
On the other hand, all participants contented themselves with the
current level of the mental map they built. The participants thought
building a higher level of mental map, which means remembering
the route clearly, would be unnecessary for the following reasons:
(1) the museum is not a frequently visited place (P4); (2) there is too
much intellectual information to remember (P9, P11, P12); and (3)
ASSETS ’22, October 23–26, 2022, Athens, Greece
they felt the complex environment is not yet ready to allow them
to travel alone (P1, P5, P6).
A7:“It was easier to remember the routes than the exhibit
names. I wish the names could be replaced with another format,
such as numbers.” P9
5.4 System Usability
5.4.1 Overall Usability. The results of three ratings related to us-
ability are summarized in Fig. 8 (Q4–Q6).
All participants agreed that the system was useful (Q4, median
= 6.5) and enjoyable compared with getting information from the
homepage (Q6, median = 6.5). In particular, seven participants (P1,
P3, P4, P5, P7, P10, P12) commented that linking geometric or
outlined shape, location, and content using the 3D model and audio-
tactile labels was both efective and enjoyable.
A8:“I’m not good at building mental maps, but I somehow
managed to do it by touching and listening." P1
A9:“Processing the audio-tactile information, I could under-
stand the content and arrangement of the exhibit booths." P10
A10:“Compared to merely reading the homepage, touching and
listening made me excited about the following trip." P4
Three participants (P2, P11, P12) were also excited about the inde-
pendence they obtained in the exploration.
A11:“The best thing is that I could explore independently with-
out asking for help." P11
The participants other than P12 leaned toward giving a rating
that the system was easy to use (score equal to or greater than 5).
Six participants commented that the double-tap was not easy at
frst (P2, P4, P6, P8, P9, P12), which infuenced their score on Q5.
The participants pointed out that there was a learning curve, and
it largely depended on the user’s profciency with mobile devices
and voiceover controls.
A12:“I wasn’t used to double-tapping and didn’t know where
I could tap at frst. But I got profcient after I spent more time
with it." P6
Three participants hoped to use a more explicit and seamless touch
to trigger the audio (P2, P6, P8), although they acknowledged that
a single-tap would trigger unnecessary sounds (P2, P8).
A13:“The double-tap also reacted to other fngers during ex-
ploration. I think touch with a stronger force is better than
double-tap. It would respond to more conscious movements." P6
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 8: Questionnaire results of the overall usability of our system (Q4–Q6).
Figure 9: Questionnaire results of the usability of 3D and layered foor-related elements (Q7–Q10) and audio-tactile interface-
related elements (Q11–Q14). A: [The element] is easy to understand, and B: [It] is useful in exploration and tour design.
5.4.2 Usability of 3D and Layered Floors. The results of ratings
related to 3D and layered foors are summarized in Fig. 9 (Q7.A–
Q10.B). In general, all 3D and layered foor-related elements were
rated as understandable and useful (median >= 6). The fully stacked
3D building (Q7) and partially stacked 3D interlocking foors (Q8)
received especially positive ratings (median = 7). The participants
felt it was especially benefcial and enjoyable to be able to stack and
touch structural attractions in the building, such as the Oval Bridge,
the Dome Theater, the atrium, and the series of long escalators (P2,
P4–P8, P10). They noted that they were attracted by the “Bento”
characteristics, which motivated them to learn structural details.
A14:“When building a mental map before, I could only make a
fat map for one foor. But using this system, I had a stronger
impression of 3D movement. I was so excited to walk [with a
fnger] in the 3D space." P4
A15:“By exploring the structures in order, such as the entrance
and escalator, I feel like I was walking in the museum. It gave
me the sense of being immersed into the museum." P6
A16:“The building can be separated, and it is easy to under-
stand the details. In the case of a tactile map, it is difcult to
understand how to move from foor to foor." P8
Wang, et al.
Participants commented that touching the structural attraction’s
geometric shape on the model was the best way to understand its
actual structure (P2, P12). As a result, all of the participants chose
the Oval Bridge to be a part of the tour they designed.
A17:“It would be impossible to understand the structure of the
Oval Bridge without the 3D model." P7
About the outlined exhibits, not every participant associated
them with the actual size and outline of the exhibition area. How-
ever, when they noticed the association, they were very positive
about this kind of information being provided.
A18:“One exhibit was a narrow and long chamber. When I went
in, I was like ‘That’s it!’ I remembered the shape clearly with
my fngers. The impression would not be that strong if I had
only heard from the voice guide that it was narrow and long."
P4
The volumetric symbols were understandable but not perceived
as especially useful due to the fact that the task was designing a tour.
They acknowledged that even though it was not very much used, it
would be absolutely important in the actual visit (P4). Participants
also noted that facilities, especially the restroom and front desk,
possibly needed touch interactions (P4, P10, P11).
BentoMuseum: 3D and Layered Interactive Museum Map for Blind Visitors
A19:“I want to know more about the ticketing information and
where the fush button is in the restroom." P11
The study did not fnd particular challenges in stacking and
orienting the foors due to the model’s irregular shape, the magnets’
support of a specifc lockup, and the maintained model orientation.
However, some participants reported difculties in recognizing
separable 3D structures. They noted that even though the inter-
foor structures (e.g., Oval Bridge and escalators) on the bottom
foor were self-explanatory, it was difcult to locate them on the
top foor alone (P1, P3, P6).
5.4.3 Usability of Audio-tactile Interface. The results of ratings
related to audio-tactile interactions are summarized in Fig. 9 (Q11.A–
Q14.B). The exhibition’s two-level audio information (Q11 and
Q12) and Navigation mode (Q14) were rated as understandable and
useful (median >= 6.5). After obtaining information, all participants
included at least one exhibit with accessible content in their tour.
Two participants (P4, P11) chose the north-up navigation style, and
the rest used the default turn-by-turn style. Participants mentioned
that the navigation was helpful for them to develop the route in
their minds (P1–P4, P7, P11) and that using a fnger to trace the
route was an enjoyable experience (P1, P3, P4, P7, P10).
A20:“Thanks to the Navigation mode, I could learn relative
locations and build a mental map." P1
A21:“It was fun to grasp the location relationship by tracing the
route with a fnger." P3
Nevertheless, the need for the Navigation mode might depend
on the tactile and orientation ability:
A22:“I don’t need the Navigation mode now because the layout
is easy. I can understand and remember it just by touch." P8
Participants somehow agreed that double-tapping the intersec-
tion was easy (Q13.A, median = 5) and useful (Q13.B, median =
4.5). Four participants reported that the intersection was small for
a double-tap (P5, P7, P9, P10). In terms of usefulness, we received a
variety of comments and suggestions.
A23:“I can touch the exhibit to learn the needed information,
thus I didn’t use the intersection." P8
A24:“I might have touched a number of exhibits, but I don’t
know which are untouched. I hope the system tells me what area
hasn’t been explored. Probably it is best at an intersection." P6
5.4.4 Free Comments and Suggestions. Participants freely expressed
where they wanted to use the system. The answers are categorized
as follows: locations containing many points of interest, such as
museums (P2, P5, P6, P7, P9, P11), amusement parks (P5, P11, P12),
and department stores (P5, P9, P11); large and complex places, such
as convention halls (P6) and airports (P2, P10); and frequently vis-
ited places, such as train stations (P3, P8, P10, P11), hospitals (P1,
P11), city halls (P4, P11), schools (P4, P6), and in the train (P4).
Participants also raised a variety of hopes and suggestions of
what the system could ofer.
A25:“I want to bring the system with me, and let it explain
things to me just like a tour guide. Like ‘We arrived here. It
is about...’ I hope we can hold a discussion in-depth about the
exhibition. It would also be nice to tell me where it is crowded
and where it isn’t." P5
ASSETS ’22, October 23–26, 2022, Athens, Greece
A26:“The museum has a shop, right? I want to learn what are
popular souvenirs. I also want to know the restaurant menu. I
want to hear a lot of options in this museum!" P11
A27:“I want to use it at the entrance when I come again. But
it’s not only for the visually impaired. Foreigners, children, it
can be useful for everybody!" P12
6 DISCUSSION
6.1 RQs: Efectiveness in Information
Provision and Mental Map Building
6.1.1 RQ1. How can we make the vast amount of needed information
accessible and understandable on a museum map? While contem-
porary museums contain a massive amount of multidimensional
information, through the participatory design with stakeholders
we categorized the needed information into structural information
(layered and stackable), exhibition (name, detailed description, and
accessibility), facilities, and their locations on the map.
Our proposed method, BentoMuseum, has been proved efective
in obtaining the above types of information (Section 5.4.1, Q1 and
Q2). It helped the participants to build knowledge that integrates
the structural information, location, and contents. All of the ratings
of the purposed system outperformed the baseline of accessing
information by reading a web page, the current prevalent method.
The multi-sensory method was noted to be more helpful than using
either a tactile map or an audio guide alone (A8 and A9). Participants
especially praised the innovative “Bento Box” characteristics of the
map as being curiosity-arousing, enjoyable and understandable
(Section 5.4.2, Q7 and Q8). Enabling the users to travel from one
foor to another by touch when the map is stacked gave them a sense
of immersion (A14 and A15). It also made the structural attraction’s
geometric shape, especially those crossing several foors, accessible
and understandable in a concrete way (A16 and A17).
Aside from the provision of information, we also found a set
of advantages that BentoMuseum contributes toward the social
inclusion of the museum. It empowers users to achieve independent
exploration and decision-making, which are reported to be valuable
(Section 5.2.1, A1 and A2) [41]. At the end of the study, participants
showed their gratitude for the museum becoming more accessible
and inclusive. Frequent museum visitors (P5, P11) ideated further
customization (Section 5.4.4, A25 and A26), and those who rarely
visited a museum (P12) expressed the desire to come again for an in-
depth exploration (A27). The fndings indicate that the system can
help to bridge the understanding between the blind visitors and the
information provider and contribute to the social role of museums,
that is, to welcome blind visitors and challenge stereotypes.
6.1.2 RQ2. Is building a mental map possible and significant in the
museum context? The results from a non-rigorous fve-level evalua-
tion show that most participants could build a rough mental map
using our system (level 5 in Section 5.3). The proposed system was
helpful in the following ways: In addition to the other elements that
allowed them to explore the exhibits of interest, Navigation mode
was benefcial for drawing location relationships on a foor (A20 and
A21), and the touchable inter-foor structures helped them to con-
nect foors and build a 3D mental map (A14–A16). Diferent from
frequently visited places where Orientation and Mobility (O&M)
ASSETS ’22, October 23–26, 2022, Athens, Greece
training has been conducted, the visitors exhibited resistance to
building a detailed mental map of the museum (Section 5.3). The re-
sults demonstrate the ability of our system to help build the current
type of rough mental map (A20 and A21) and indicate its ability to
support route memorization (A7). Even though creating and main-
taining a fner mental map imposes a cognitive load in addition
to information access and tour design, it is necessary for visually
impaired people’s independence [22]. A future work is to rigorously
measure spatial memory to support O&M training.
On the other hand, when we asked participants’ thoughts and im-
pressions about the mental map after the museum tour, all of them
explicitly stated that the current rough mental map was benefcial
for their tour. First, it supported their orientation, which made their
tour meaningful (A3 and A4). Second, it gave them a sense of safety
and the confdence that they will not get lost (A5 and A6). This
confdence is an objective of navigation for the visually impaired
in an unfamiliar environment, which supports autonomy and self-
reliance [8, 28]. Through such comments, we infer that the rough
mental map is signifcant in improving the museum experience.
6.2 Limitations and Next Steps
Some usability issues are related to system design, which can be
addressed by further improvement: (1) Enlarge the small touch area
(e.g., the intersections) to ft diferent fnger sizes. (2) Refne 3D
details to make clear, inter-foor structures available on both top
and bottom foors. (3) Refne the audio content. The audio-tactile in-
formation at the intersection was noted as not useful (A23); instead,
it can be used to report the exhibit coverage as P6 suggested (A24).
Some participants also found double-tap difcult. This input method
was used to prevent unintentional touches but might be unnatural
when all fngers are used for exploration. Seamless touch interac-
tions with less learning efort (e.g., force recognition suggested by
P6 in A13) need to be further investigated.
To make the system a part of the museum facility, the display
and communication methods need to be further examined: (1) Self-
serve foor-changing. The participants did not confuse the order of
the foors because we handed them each foor upon request. We
also instructed the participants to keep the same orientation. To
order and orient the foors in the wild by the users themselves, clear
labels, verbal instructions, and tactile indications need to be tested.
(2) Automated instructions. Our study proved that instructions
delivered by museum staf (Section 4.2.2 and Section 4.2.3) were
benefcial to understanding the external structure and learning
the system. Automating those instructions is needed to reduce the
staf expertise for system operation. (3) Time and interest-based
instructions for efciency. The tour design task took approximately
45 minutes, which was a considerable amount of time that not every
visitor can aford. Our log data show that the movement styles were
linked to efciency, and the most efcient style was not performed
naturally (Section 5.2.2). Since museum visitors might have varied
tactile skills and needs, customizable instructions that support an
efcient exploration need to be investigated.
6.3 Generalizability and Lesson Learned
Many participants hoped that the method would become avail-
able in museums. Interestingly, they also suggested applying the
Wang, et al.
methods to other locations that troubled them, attracted them, or
required their confdence (Section 5.4.4).
Even though the design method is currently implemented in one
particular museum, it could be generalized to a variety of locations,
especially buildings with irregular external or internal structures
and complex information. We suggest the following design con-
siderations in applying our method: (1) Categorize the complex
information of a building into three types: structural attractions, ex-
hibits (or informative attractions), and facilities. Simplify the shape
of each type into 3D elements, 2D relief elements, and volumetric
symbols on the foor, respectively. (2) Make foors stackable and en-
sure the inter-foor 3D elements can be understood by touch when
foors are stacked. (3) Add audio labels to the points of interest
and ensure they can be recognized by touch. Such a design can
be implemented with relatively inexpensive materials: 3D printers,
touch screens, and conductive ink.
Through our study of introducing an entire museum to frst-time
visitors, another important lesson we learned was that instruction,
guidance, and encouragement could motivate the users and promote
efective information provision when presenting a complex map.
In our study, we instructed the users to travel through inter-foor
structures using fngers and consequently learned they especially
enjoyed these structures. If we failed to do so, some participants
might overlook them due to the large number of 3D and 2D elements
on each foor. The methods to communicate the map and motivate
the users should be a part of the map design.
6.4 Toward Universal Access
Participants showed an interest in having the BentoMuseum as an
“ask-me-anything” box (A25 and A26). Further investigations (e.g.,
conversational agent) should be made to fulfll individual needs
while maintaining simple operation. Some participants also ex-
pected it to support the needs of those beyond themselves (A27).
Indeed, it can potentially share information between diferent stake-
holders: the blind visitors, museum staf members and service
providers, domestic visitors, foreign visitors, and other visitors
with disabilities to create universal access. For example, the infor-
mation that is now decoded to audio might be presented in sign
language to support hearing impaired visitors in gaining informa-
tion access. Diferent services can be connected through the map
to extend access throughout the visit.
7 CONCLUSION
This work investigated how a museum with a massive amount
of multidimensional information could provide accessible maps
to blind visitors. We designed 3D and layered museum maps for
each foor of a science museum which can be stacked or placed
on a touch screen to learn diferent levels of detail. An authentic
tour design task with 12 blind frst-time museum visitors showed
our system’s efectiveness in obtaining information and building a
rough mental map. Through user feedback, we learned the potential
of our system to contribute to a positive and inclusive museum
experience. Our next steps include expanding this design method
to other museums and attractions, making it smarter to support
diferent needs, and making it available along with other means of
assistive technologies to support autonomous museum exploration.
BentoMuseum: 3D and Layered Interactive Museum Map for Blind Visitors
ACKNOWLEDGMENTS
We thank all of the participants who took part in our user study. We
also thank Sakiko Tanaka, Bunsuke Kawasaki, and Kotaro Osawa,
the Science Communicators at Miraikan – The National Museum
of Emerging Science and Innovation, for their contributions to the
system design and feld study.
REFERENCES
[1] Frances Aldrich, Linda Sheppard, and Yvonne Hindle. 2002. First steps towards
a model of tactile graphicacy. British Journal of Visual Impairment 20, 2 (2002),
62–67.
[2] Vassilios S Argyropoulos and Charikleia Kanari. 2015. Re-imagining the mu-
seum through “touch”: refections of individuals with visual disability on their
experience of museum-visiting in Greece. Alter 9, 2 (2015), 130–143.
[3] Saki Asakawa, João Guerreiro, Dragan Ahmetovic, Kris M. Kitani, and Chieko
Asakawa. 2018. The Present and Future of Museum Accessibility for Peo-
ple with Visual Impairments. In Proceedings of the 20th International ACM
SIGACCESS Conference on Computers and Accessibility (Galway, Ireland) (AS-
SETS ’18). Association for Computing Machinery, New York, NY, USA, 382–384.
https://doi.org/10.1145/3234695.3240997
[4] Japanese Standards Association. 2021. JIS T 0922:2007 Guidelines for older
persons and persons with disabilities – Information content, shapes and display
methods of tactile guide maps [In Japanese]. https://webdesk.jsa.or.jp/books/
W11M0090/index/?bunsyo_id=JIS+T+0922%3A2007
[5] Sandra Bardot, Marcos Serrano, Bernard Oriola, and Christophe Joufrais. 2017.
Identifying How Visually Impaired People Explore Raised-Line Diagrams to
Improve the Design of Touch Interfaces. In Proceedings of the 2017 CHI Conference
on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17).
Association for Computing Machinery, New York, NY, USA, 550–555. https:
//doi.org/10.1145/3025453.3025582
[6] Graham Black. 2012. The engaging museum: Developing museums for visitor
involvement. Routledge, London.
[7] Anke M Brock, Philippe Truillet, Bernard Oriola, Delphine Picard, and Christophe
Joufrais. 2015. Interactivity improves usability of geographic maps for visually
impaired people. Human–Computer Interaction 30, 2 (2015), 156–194.
[8] Kangwei Chen, Victoria Plaza-Leiva, Byung-Cheol Min, Aaron Steinfeld, and
Mary Bernardine Dias. 2016. NavCue: Context Immersive Navigation Assistance
for Blind Travelers. In The Eleventh ACM/IEEE International Conference on Human
Robot Interaction (Christchurch, New Zealand) (HRI ’16). IEEE Press, 559.
[9] Anne Chick et al. 2017. Co-creating an accessible, multisensory exhibition
with the National Centre for Craft & Design and blind and partially sighted
participants. REDO Cumulus Conference Proceedings (2017).
[10] Matthew Cock, Molly Bretton, Anna Fineman, Richard France, Claire Madge,
and Melanie Sharpe. 2018. State of Museum Access 2018: does your museum
website welcome and inform disabled visitors? http://vocaleyes.co.uk/state-of-
museum-access-2018/
[11] Eugenia Devile and Elisabeth Kastenholz. 2018. Accessible tourism experiences:
the voice of people with visual disabilities. Journal of Policy Research in Tourism,
Leisure and Events 10, 3 (2018), 265–285.
[12] Jocelyn Dodd, Richard Sandell, et al. 2001. Including museums: perspectives on
museums, galleries and social inclusion. University of Leicester, Leicester, UK.
[13] Fatma Faheem and Mohammad Irfan. 2021. Museums for Equality: Diversity and
Inclusion–A New Concept of Future Museums. IAR Journal of Humanities and
Cultural Studies 2, 1 (2021), 12–13.
[14] Natália Filová, Lea Rollová, and Zuzana Čerešňová. 2022. Route options in
inclusive museums: Case studies from Central Europe. Architecture Papers of the
Faculty of Architecture and Design STU 27, 1 (2022), 12–24.
[15] Uttara Ghodke, Lena Yusim, Sowmya Somanath, and Peter Coppin. 2019. The
Cross-Sensory Globe: Participatory Design of a 3D Audio-Tactile Globe Prototype
for Blind and Low-Vision Users to Learn Geography. In Proceedings of the 2019
on Designing Interactive Systems Conference (San Diego, CA, USA) (DIS ’19).
Association for Computing Machinery, New York, NY, USA, 399–412. https:
//doi.org/10.1145/3322276.3323686
[16] Stéphanie Giraud, Anke M Brock, Marc J-M Macé, and Christophe Joufrais. 2017.
Map learning with a 3D printed interactive small-scale model: Improvement of
space and text memorization in visually impaired students. Frontiers in psychology
8 (2017), 930.
[17] Timo Götzelmann. 2016. LucentMaps: 3D Printed Audiovisual Tactile Maps
for Blind and Visually Impaired People. In Proceedings of the 18th International
ACM SIGACCESS Conference on Computers and Accessibility (Reno, Nevada, USA)
(ASSETS ’16). Association for Computing Machinery, New York, NY, USA, 81–90.
https://doi.org/10.1145/2982142.2982163
[18] Jaume Gual, Marina Puyuelo, and Joaquim Lloveras. 2014. Three-dimensional
tactile symbols produced by 3D Printing: Improving the process of memorizing a
ASSETS ’22, October 23–26, 2022, Athens, Greece
tactile map key. British Journal of Visual Impairment 32, 3 (2014), 263–278.
[19] Jaume Gual, Marina Puyuelo, and Joaquim Lloveras. 2015. The efect of volumetric
(3D) tactile symbols within inclusive tactile maps. Applied Ergonomics 48 (2015),
1–10.
[20] Jaume Gual, Marina Puyuelo, Joaquim Lloveras, et al. 2011. Universal design and
visual impairment: Tactile products for heritage access. In Proceedings of the 18th
International Conference on Engineering Design (ICED 11). Lyngby/Copenhagen,
Denmark, 155–164.
[21] Jaume Gual, Marina Puyuelo, Joaquim Lloverás, and Lola Merino. 2012. Visual im-
pairment and urban orientation. Pilot study with tactile maps produced through
3D printing. Psyecology 3, 2 (2012), 239–250.
[22] João Guerreiro, Dragan Ahmetovic, Kris M. Kitani, and Chieko Asakawa. 2017.
Virtual Navigation for Blind People: Building Sequential Representations of the
Real-World. In Proceedings of the 19th International ACM SIGACCESS Conference on
Computers and Accessibility (Baltimore, Maryland, USA) (ASSETS ’17). Association
for Computing Machinery, New York, NY, USA, 280–289. https://doi.org/10.
1145/3132525.3132545
[23] Kozue Handa, Hitoshi Dairoku, and Yoshiko Toriyama. 2010. Investigation of
priority needs in terms of museum service accessibility for visually impaired
visitors. British journal of visual impairment 28, 3 (2010), 221–234.
[24] Kevin Hetherington. 2000. Museums and the visually impaired: the spatial politics
of access. The Sociological Review 48, 3 (2000), 444–463.
[25] Leona Holloway, Kim Marriott, and Matthew Butler. 2018. Accessible Maps for
the Blind: Comparing 3D Printed Models with Tactile Graphics. In Proceedings of
the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC,
Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA,
1–13. https://doi.org/10.1145/3173574.3173772
[26] Leona Holloway, Kim Marriott, Matthew Butler, and Samuel Reinders. 2019. 3D
Printed Maps and Icons for Inclusion: Testing in the Wild by People Who Are
Blind or Have Low Vision. In The 21st International ACM SIGACCESS Conference
on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association
for Computing Machinery, New York, NY, USA, 183–195. https://doi.org/10.
1145/3308561.3353790
[27] Eilean Hooper-Greenhill, Richard Sandell, Theano Moussouri, Helen O’Riain,
et al. 2000. Museums and social inclusion: The GLLAM report. University of
Leicester, Leicester, UK.
[28] Sulaiman Khan, Shah Nazir, and Habib Ullah Khan. 2021. Analysis of Navigation
Assistants for Blind and Visually Impaired People: A Systematic Review. IEEE
Access 9 (2021), 26712–26734. https://doi.org/10.1109/ACCESS.2021.3052415
[29] Barbara Leporini, Valentina Rossetti, Francesco Furfari, Susanna Pelagatti, and
Andrea Quarta. 2020. Design Guidelines for an Interactive 3D Model as a Sup-
porting Tool for Exploring a Cultural Site by Visually Impaired and Sighted
People. ACM Trans. Access. Comput. 13, 3, Article 9 (aug 2020), 39 pages.
https://doi.org/10.1145/3399679
[30] Nina Levent and Christines Reich. 2012. How Can My Museum Help Visitors With
Vision Loss? Museum. American Association of Museums July-August (2012), 21–
22. http://ww2.aam-us.org/docs/default-source/museum/visitors-with-vision-
loss.pdf?sfvrsn=0
[31] Georgia Lindsay. 2020. Contemporary Museum Architecture and Design. Routledge,
New York.
[32] Marianne Loo-Morrey. 2005. Tactile Paving Survey. https://www.hse.gov.uk/
research/hsl_pdf/2005/hsl0507.pdf
[33] ICOM The International Council of Museums. 2020. International Museum Day
2020 - Museums for Equality: Diversity and Inclusion. https://imd.icom.museum/
past-editions/2020-museums-for-equality-diversity-and-inclusion/
[34] ICOM The International Council of Museums. 2022. International Museum Day
2022: The Power of Museums. https://imd.icom.museum/international-museum-
day-2022-the-power-of-museums/
[35] The Braille Authority of North America. 2010. Guidelines and Standards for
Tactile Graphics. http://www.brailleauthority.org/tg/web-manual/index.html
[36] Round Table on Information Access for People with Print Disabilities Inc. 2005.
Guidelines on Conveying Visual Information. https://printdisability.org/
guidelines/guidelines-on-conveying-visual-information-2005/
[37] Petros Pistofdis, George Ioannakis, Fotis Arnaoutoglou, Natasa Michailidou,
Melpomeni Karta, Chairi Kiourt, George Pavlidis, Spyridon G Mouroutsos, De-
spoina Tsiafaki, and Anestis Koutsoudis. 2021. Composing smart museum exhibit
specifcations for the visually impaired. Journal of Cultural Heritage 52 (2021),
1–10.
[38] Jonathan Rowell and Simon Ongar. 2003. The world of touch: an international
survey of tactile maps. Part 2: design. British Journal of Visual Impairment 21, 3
(2003), 105–110.
[39] Jonathan Rowell and Simon Ungar. 2003. The world of touch: an international
survey of tactile maps. Part 1: production. British Journal of Visual Impairment
21, 3 (2003), 98–104.
[40] Jonathan Rowell and Simon Ungar. 2005. Feeling our way: tactile map user
requirements-a survey. In International Cartographic Conference, La Coruna. 652–
659.
[41] Richard Sandell. 2002. Museums, society, inequality. Routledge, London.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[42] Martin Schmitz, Mohammadreza Khalilbeigi, Matthias Balwierz, Roman Lisser-
mann, Max Mühlhäuser, and Jürgen Steimle. 2015. Capricate: A Fabrication
Pipeline to Design and 3D Print Capacitive Touch Sensors for Interactive Objects.
In Proceedings of the 28th Annual ACM Symposium on User Interface Software &
Technology (Charlotte, NC, USA) (UIST ’15). Association for Computing Machin-
ery, New York, NY, USA, 253–258. https://doi.org/10.1145/2807442.2807503
[43] Jonathan Sweet. 2007. Museum architecture and visitor experience. Museum
Marketing: Competing in the global marketplace (2007), 226–237.
[44] Brandon Taylor, Anind Dey, Dan Siewiorek, and Asim Smailagic. 2016. Customiz-
able 3D Printed Tactile Maps as Interactive Overlays. In Proceedings of the 18th
International ACM SIGACCESS Conference on Computers and Accessibility (Reno,
Nevada, USA) (ASSETS ’16). Association for Computing Machinery, New York,
NY, USA, 71–79. https://doi.org/10.1145/2982142.2982167
[45] Simon Ungar. 2018. Cognitive mapping without visual experience. In Cognitive
Mapping: Past Present and Future. Routledge, London, 221–248.
[46] Simon Ungar, Mark Blades, and Christopher Spencer. 1993. The role of tactile
maps in mobility training. British Journal of Visual Impairment 11, 2 (1993),
59–61.
[47] Raša Urbas, Matej Pivar, and Urška Stankovič Elesini. 2016. Development of
tactile foor plan for the blind and the visually impaired by 3D printing technique.
Journal of graphic engineering and design 7, 1 (2016), 19–26.
[48] Roberto Vaz, Diamantino Freitas, and António Coelho. 2020. Blind and Visually
Impaired Visitors’ Experiences in Museums: Increasing Accessibility through
Assistive Technologies. International Journal of the Inclusive Museum 13, 2 (2020).
[49] Roberto Vaz, Diamantino Freitas, and António Coelho. 2020. Perspectives of
Visually Impaired Visitors on Museums: Towards an Integrative and Multisensory
Framework to Enhance the Museum Experience. In 9th International Conference
Wang, et al.
on Software Development and Technologies for Enhancing Accessibility and Fight-
ing Info-Exclusion (Online, Portugal) (DSAI 2020). Association for Computing
Machinery, New York, NY, USA, 17–21. https://doi.org/10.1145/3439231.3439272
[50] Andreas Voigt and Bob Martens. 2006. Development of 3D tactile models for
the partially sighted to facilitate spatial orientation. In 24th eCAADe Conference
Proceedings. CUMINCAD.
[51] Diana Walters. 2009. Approaches in museums towards disability in the United
Kingdom and the United States. Museum management and curatorship 24, 1
(2009), 29–46.
[52] Xi Wang, Danny Crookes, Sue-Ann Harding, and David Johnston. 2022. Stories,
journeys and smart maps: an approach to universal access. Universal Access in
the Information Society 21, 2 (2022), 419–435.
[53] Zheshen Wang, Baoxin Li, Terri Hedgpeth, and Teresa Haven. 2009. Instant
Tactile-Audio Map: Enabling Access to Digital Maps for People with Visual
Impairment. In Proceedings of the 11th International ACM SIGACCESS Con-
ference on Computers and Accessibility (Pittsburgh, Pennsylvania, USA) (As-
sets ’09). Association for Computing Machinery, New York, NY, USA, 43–50.
https://doi.org/10.1145/1639642.1639652
[54] Neng-Hao Yu, Sung-Sheng Tsai, I-Chun Hsiao, Dian-Je Tsai, Meng-Han Lee,
Mike Y. Chen, and Yi-Ping Hung. 2011. Clip-on Gadgets: Expanding Multi-Touch
Interaction Area with Unpowered Tactile Controls. In Proceedings of the 24th
Annual ACM Symposium on User Interface Software and Technology (Santa Barbara,
California, USA) (UIST ’11). Association for Computing Machinery, New York,
NY, USA, 367–372. https://doi.org/10.1145/2047196.2047243
[55] Limin Zeng, Gerhard Weber, et al. 2011. Accessible maps for the visually impaired.
In Proceedings of IFIP INTERACT 2011 Workshop on ADDW, CEUR, Vol. 792. 54–60.
 Depending on Independence
An Autoethnographic Account of Daily Use of Assistive
Technologies
Felix Fussenegger
Katta Spiel
felix.fussenegger@igw.tuwien.ac.at
katta.spiel@tuwien.ac.at
HCI Group – TU Wien
Vienna, Austria
ABSTRACT
Assistive technologies (AT) are a necessity for a person with a
severe disability to be able to lead a self-determined life within
modern societies. Thus, these technologies fulfl an important soci-
etal role and have a signifcant impact on the lives of those afected.
This experience report provides insights on the lived experiences
surrounding the necessary use of AT from the point of view of a dis-
abled person. Using an autoethnographic approach, we determine
the function and relevance of AT in everyday life and illustrate the
intended and unintended efects of AT as well as the subsequently
arising socio-technical dependencies through representative exam-
ples. The results show how the resulting dependencies on AT pose
a risk for users, especially in the event of a technological failure.
Furthermore, a deployment of AT without the necessary refection
and preparation of backup strategies in case of failure may lead
to unexpected and inadvertent, potentially harmful, side-efects.
Based on these observations, we elaborate on the implications for
diferent stakeholder groups involved with the design, development,
deployment and daily use of AT. We deem the key factors for suc-
cess to lie in a deeper understanding of the application context, the
integration of afected people in the development process as well
as a fundamentally refective approach by everyone involved with
AT.
CCS CONCEPTS
• Social and professional topics → People with disabilities;
• Human-centered computing → Accessibility design and eval-
uation methods; Empirical studies in accessibility; Accessibility
technologies.
KEYWORDS
assistive technologies, dependence, interdependence, independence,
disability studies, autoethnography
This work is licensed under a Creative Commons
Attribution-NonCommercial-ShareAlike International 4.0 License.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3551354
ACM Reference Format:
Felix Fussenegger and Katta Spiel. 2022. Depending on Independence An
Autoethnographic Account of Daily Use of Assistive Technologies. In The
24th International ACM SIGACCESS Conference on Computers and Accessibil-
ity (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY,
USA, 6 pages. https://doi.org/10.1145/3517428.3551354
1 INTRODUCTION
Assistive technologies (AT) have a signifcant place in the lives of
people with disabilities and are paramount to their participation
in modern societies. For my1 personal condition as an impaired
person with a high spinal cord injury, I can confrm that I would
not be able to survive on my own without AT or at least I would be
permanently and constantly dependent on the help of other people.
Thus, I am glad to live in this day and age in which we can rely on a
fast advancing variety of technologies at our disposal that support
me in leading a self-determined life [2]. But precisely because AT
are gaining traction and becoming so fundamentally important in
terms of enabling people with disabilities, the impacts they have on
their lives must be scrutinised and well understood. Otherwise, the
disposition of AT with the resulting dependencies can and will lead
to unexpected and inadvertent, potentially harmful, side-efects.
With this experience report, I want to provide an insight into my
personal use of AT and how they infuence me, my health, my
social interactions and my surroundings; how they are shaped
by the needs of my specifc embodiment and shape these needs
in return [17]. Or, more generally speaking, I detail their impact
on my interdependence. Interdependence can be seen as a com-
plementary frame for AT researchers, previously introduced to
Human Computer Interaction (HCI) by Bennett et al. The concept
considers access to be relational and simultaneous, highlights the
contributions of people with disabilities, and challenges traditional
hierarchies of abilities [3]. My report stems from such a relational
understanding of disabilities, self-determination and access.
I will describe my individual experience with AT from my per-
spective as a person living with a severe disability, but also the
drawbacks of the lived experiences, from which I draw when think-
ing through issues of dependence in AT. The chosen examples
are intended to guide an understanding of the importance of AT
for disabled persons, the impacts AT have on their everyday lives
1 Thisexperience report is predominantly written from the perspective of Felix as the
frst author with Katta, the second author, guiding and supporting in the development
and writing processes for this report. Hence, whenever there is a reference from a frst
person singular perspective, it refers to Felix if not stated otherwise.
ASSETS ’22, October 23–26, 2022, Athens, Greece
and what it is like to depend on them from a personal perspective.
Further, I elaborate on the implications of the insights I make in
observing my own practices and involvements with AT to make
them relevant for diferent stakeholders (such as researchers, medi-
cal professionals, policy makers, and disabled people themselves).
In that, I operate loosely from within the notions of frst-person
research methods [4].
In this context, I deem it necessary to briefy explain my own im-
pairment and my perception of disability. I live with my physical
impairment since a snowboarding accident in 2003, in which I
sufered a spinal cord injury to the sixth cervical vertebra. This
resulted in a complete sensory and motor paralysis of my body ap-
proximately from half of the shoulder down. Besides these medical
limitations, there are also physical and social barriers in society
that have a substantial infuence on my disability. Therefore, I see
disability not only defned by my own impairment, but also by the
lack of opportunities that society ofers me to employ my abili-
ties, which means I operate from a predominantely social model of
disability also in my use of AT [10].
I advise that this text is based on my subjective experiences and
describes purely my personal insights and convictions. I do not
represent the position of any organisation or other people with
disabilities – either similar or dissimilar to mine and operates from
a specifc locality and situatedness [7]. Their experiences and opin-
ions may well difer from mine and result in diferent implications
for research.
2 ASSISTIVE TECHNOLOGIES AND ME
This section describes my personal experiences with AT - the good
as well as the not so pleasant. I provide a rough overview of the
most important technologies that support me in my everyday life.
I describe how I use them and what I need them for. In the frst
section, I list the specifc AT that I need to master a typical day and
explain for which activities I use them. In the following sections,
I illustrate how AT has infuenced me and my life through three
specifc examples. These examples are chosen in such a way that
they 1) clearly describe the contexts and efects of the use of AT, 2)
are easily comprehensible and 3) have a representative potential as
they relate to other AT.
2.1 A Day in the Life
Table 1 contains an overview of some of the AT I use and what I
need them for. The enumeration of the AT outlines a common daily
routine of a common day in my life. I identifed them by tracking
my daily interactions with objects and technologies for several days
and refecting on my notes, a common procedure within autoethno-
graphic approaches [20]. The mentioned AT are things, products or
systems that support and help me with my impairments to perform
functions that might otherwise be difcult or impossible for me to
achieve. These devices support me to improve or maintain my daily
quality of life by easing or compensating for my disability and/or
the ways my environments are built without having my disability
in mind. This can be a very simple object like an eating aid or a
more complex application like speech recognition on a computer.
The technical complexity, however, says nothing about how impor-
tant or useful an artefact or application is for me. The third column
Felix Fussenegger and Kata Spiel
of Table 1 lists the activities that the AT enable me to carry out
independently. On the one hand, this shows what the AT empower
me to do, but on the other, it also illustrates how dependent I am
on them and what activities I lose or have to manage drastically
diferently in case of failure.
2.2 Example 1: Early Voice Recognition vs Pen
The accident that caused my disability happened in my fnal year
of high school. After about a year of hospitalisation and intensive
therapy in a rehabilitation centre, I returned to school with the aim
of graduating the same year. However, there turned out to be one
problem: it was unclear how I could perform my written exams and
exercises. With my limited hand function, I could not write fast and
well enough, neither by hand nor on the computer, to get my work
done in what would have been deemed a reasonable amount of
time. Despite the long and intensive rehabilitation, my handwriting
was like that of someone just starting their schooling and learning
to write. The recommended solution from my therapists and from
school was that I should use a speech recognition software and
dictate my work and exams from now on. Following their advice, I
bought (very expensive) software and started working with it. But
in 2003, such software was not yet very sophisticated and I was
deeply unsatisfed with the reusults. But the therapists and teachers
in their positions as “experts” explained to me that the software
had to get to know me, and I had to get to know the software in
return before I could expect it to function more seamlessly. And if I
just trained regularly, it would get better and better with time (and
patience).
So I trained - for hours and hours, but the progress was sluggish.
During the training, I began to scribble my spoken sentences on a
piece of paper with a pen. This helped me to formulate the spoken
language more in line with what would be expected from a text
in written language. At the beginning the scribble was nothing
more but a wobbly line. But after many hours, during which I
trained with the software, I noticed that the scribbles developed
more and more into a recognisable script. While I intended to train
the software to recognise my voice better and perform more in
line with my expectations, the process had, in return, unexpectedly
trained my own body to perform the task I meant to relegate to an
AT. After a while, my progress in handwriting was greater than
with the software. Hence, at some point, I only concentrated on my
handwriting skills. And lo and behold - after several fully written
notebooks, my handwriting skills were sufcient enough for me
to take the fnal graduation exams on a regular basis without the
support of any software. This bafed both, my therapists and my
teachers. Until this day, I am grateful for the poorly functioning
language software that inadvertently returned my handwriting
skills back to me. I am convinced that this would not have happened
with today’s much more sophisticated software and that I still would
not be able to write a text by hand or fll out a form on my own.
This example shows me how much the use of AT can promote or
hinder one’s own abilities – even if this might be contrary to the
initially intended purpose of use.
Depending on Independence ASSETS ’22, October 23–26, 2022, Athens, Greece

Device Function/Explanation Enablement/Dependency

holding bar over bed
wheelchair fxing device
manual wheelchair
shower wheelchair
elevator
applications in the the car
voice recognition software
- for PC
- for smartphone
eating aids
mobile phone with assistance
and emergency numbers
credit card with NFC payment
hybrid manual wheelchair
smart thermostat
Table 1: List of AT I use during a normal day, what it does and which activities it enables
- hold on without fnger function
- pull up torso
- holding mechanism to fxate the
wheelchair to the bed
- ultra lightweight
- foldable
- extra strong grip on push rims
- water solid
- cutouts in the seat area
- big enough to ft a wheelchair
- reachable buttons
- transfer device
- hand throttle
- hand brake
- steering aid
- dictate long texts
- dictate input for chats and mails
- execute frequently used commands
- specially formed fork, knife and
spoon
- easy call when I unexpectedly need
assistance
- easy call in case of emergency
- easy payment of small charges
- senses my activity on the
push rim
- provides adequate support
with an electric drive
- digitally programmable, network-
compatible thermostats
- sit up in bed
- turn my self over
- get dressed and undressed
- get out of the bed
- facilitates transfer
between bed and wheelchair
- reduces the risk of falling
- covering medium distances on a level
ground
- loading into and out of the car
- Body wash / shower
- using the toilet
- get to and from my fat (2nd foor)
- get to the garage
- perform a safe transfer
- load the wheelchair on/of the passenger
seat
- steering the car safely
- writing documents for work
and university
- participate in digital communication
- eating without help
- get help when needed
- handling small purchases
e.g. at the grocery store
- covering long distances on a semi
level ground
- locomotion that is easy on the joints
- regulation of temperature at home

2.3 Example 2: A Small Hook with a Serious when I am not at the peak level of my ftness, for example, due to
Catch an illness. Therefore, a few years ago, I constructed an individually
ftted mechanism together with my therapist, with which I can
With my degree of disability, transferring between a wheelchair
frmly connect the wheelchair to the bed. This makes the transfer
and my bed comprises a big challenge. At the same time, the abil-
easier and safer and I can manage it even if I am not entirely ft. In
ity to master the transfer on my own means a great degree of
the beginning, I only used the mechanism when I really needed it.
independence. That is why I trained specifcally in the course of
But as time went by, I used it for every transfer, because it makes
my rehabilitation to tackle this hurdle. I’m glad I succeeded, even
the process so much easier and is therefore a relief in everyday
though I have to keep myself ft all the time so that I don’t lose
life, allowing me to use my energy for other tasks [15]. When the
this ability. Moreover, it is difcult for me to carry out the transfer
holding mechanism was once unavailable due to damage incurred,
ASSETS ’22, October 23–26, 2022, Athens, Greece
I realised that I had lost the ability to transfer between bed and
wheelchair without using the mechanism. The device could be re-
paired quite easily, but the fact of having lost such a hard-earned
function without noticing until it was too late hit me hard. For me,
this means that if I spend the night in a bed diferent from the one
at home (e.g. when travelling), I now need help to get in or out of
the bed when previously, I could just more easily plan for these
aspects of travel. I have decided to specifcally train this skill again
during my next stay at the rehabilitation centre to counteract the
loss of function due to the AT. After that, I plan to only use the
mechanism when I absolutely need it.
This example should demonstrate how quickly you may become un-
intentionally dependent on AT. As in this case, it can lead to the loss
of important functions and, thus, to a certain loss of independence
when the AT become unavailable – potentially without anyone
noticing that this loss has occurred until it becomes a signifcant
issue.
2.4 Example 3: A Hot Winter Night
Having a high spinal cord injury my body’s own temperature reg-
ulation is signifcantly reduced. This means that I get too cold or
too hot quite quickly, depending on the ambient temperature of
the environment I am in. Some of this can be countered with ap-
propriate clothing, but at home I try to keep the room temperature
in an optimised range for me. To achieve this, I installed digitally
programmable, network-compatible thermostats (so-called "smart
thermostats") on my radiators. This enables me to set the desired
temperature very precisely and individually for each room. If the
temperature still fails to suit me, I can quickly and easily make the
desired change via my smartphone. In general, the system works
quite well for me and I am happy with the usability and the overall
improved energy balance for my body.
But last winter, for no obviously apparent reason, two of the ther-
mostats encountered a problem and stopped working. The heating
valves were completely open and the thermostats could not be
adjusted either via the buttons on the unit or via the smartphone.
My fat quickly became very hot and since it was already late in
the evening, I had no option available to organise anyone to reset
or dismantle the thermostats (both can only be performed with
specifc tools and full fnger function). This lead to me having to
endure a sleepless night, during which I tried to keep the tempera-
ture within a bearable range by constantly closing and opening the
windows (thankfully, it was cold enough outside for this to work),
until I fnally could get help the next morning.
After this experience, I constructed my own personal device with
which I can reset the thermostats on my own, echoing practices
of empowerment for disabled makers more generally [13]. Addi-
tionally, on the radiators that are out of reach for me, I reinstalled
the old-fashioned but reliable mechanical thermostats. This means
that the temperature control in the fat is no longer as efective and
pleasant, but it is safer and more consistently within my control.
The example is intended to illustrate how important the reliability
of AT is, even when a technology is not recognised as assistive
in all circumstances of use, and that the exit strategy (e.g. reset)
must similarly be accessible without barriers. Furthermore, I should
have thought about the consequences of a malfunction in advance
Felix Fussenegger and Kata Spiel
and only perform the change when I have an exit strategy or a
backup plan that works for me. However, the question remains
whose responsibility it is to account for alternative modes in the
case of malfunctioning or nonfunctioning of technological artefacts
in assistive contexts.
3 PERSONAL REFLECTION
It was my intent that the explanations in the previous section
provide a comprehensible insight into my life experience with AT
as a person with a disability. It is important to me to emphasise
the extent and importance of AT, but also to create awareness
of the interconnections, the arising dependencies as well as the
scope of possible consequences of use, functioning, bad functioning,
malfunctioning and nonfunctioning [17].
Without AT, my life would look very diferent and I would never
have access to some of the experiences that I treasure in my every-
day life such as:
• living on my own in the area of my choice (regardless of
the location of my family or the regional supply situation of
caregivers),
• having a job (as a product developer of medical devices) that
I choose based on my interests and talents and not defned
by my impairment,
• enjoying hobbies like cycling, bird watching, etc., and
• being social with friends, family and community.
All in all, I am very grateful (and a little proud) to be able to live
a self-determined, satisfying and fulflling life, despite my severe
disability. Table 1 provides practical insight into how much AT
supports me in my everyday life. At the same time, it also shows
how much I depend on it and what fundamental activities I lose if
one of the ATs fails. In addition, the examples described have shown
that the application of AT can also be associated with unintended
advantages making the AT at least partially obsolete as well as
veritable disadvantages.
4 DISCUSSION
In this section, I discuss what these observations may mean for
dealing with AT and disabilities and how the insights can be used by
the diferent types of people involved with AT, be they researchers,
designers, policy makers, therapists or disabled people themselves.
At frst, the fundamental objectives of AT for people with disabili-
ties should be elaborated, which requires us to challenge existing
convictions and the general focus on “doing good” [5] or a singular
evaluation oriented solely on the ‘usefulness’ and potential of AT,
but also on possible side efects, as has been argued for general
purpose technological research previously [8, 22]. The goal is not
just to compensate the impaired functions of a disabled person but
the objectives have to be thought of in a more holistic way.
4.1 Implications for Researchers and Designers
It is important to have a basic knowledge of diferent models of
disability and to position specifc works within those – at the very
least with a basic understanding of the big two, namely the medical
and the social models of disability [10]. Findings from disability
studies can help to develop a better understanding for these [9, 12,
18, 19, 21]. Such a position requires the proper comprehension of
Depending on Independence
the context of the application and the fundamental objectives of
an AT that is to be researched or designed. One way to achieve
this is to involve disabled stakeholders in the whole development
process. This means not only taking part in user surveys, but also
employing people from the “target population” as project members
to help shape the developments. Mankof et al. already stated how
relevant for good AT it is to include disabled people directly in
technology research about them [11], drawing on a long tradition
of disability rights activism also echoed in a notion of “rights not
charity” [16] – or adapted for technology research: “access not
benevolence”. The list of fundamental activities enabled by AT
shown in Table 1 together with the in-depth examples I provided
reveal how serious a failure of one of the used AT can be. Therefore,
the reliability and quality of AT is a fundamental requirement.
The scenario for a failure should be considered and the possible
consequences analysed. If necessary, exit strategies or a backup
plan must be developed along with the product itself. One strategy
achieving this can be to have diferent levels of support included in
the technologies so that people might be able to choose between
comfort and training where applicable. Besides the intended efects
of AT, possible side efects should be considered equally – including
those that may make technological support unnecessary. Again
it is helpful to have a clear understanding of the context of the
application and to work in close cooperation with the intended
(and even unintended) users.
Due to the rapid development of technologies and the intrinsic
motivation of researchers and developers who want to improve
the lives of people with disabilities with all their commitment and
inventive spirit, there is a danger that the use of AT becomes an
end in itself. Despite good intentions, the original goals of a cer-
tain AT are sometimes not defned clearly enough or abandoned
during the process [6] and a sober consideration of the advantages
and disadvantages (or efects and side-efects) may not be feasible
anymore, if overridden by other constraints.
4.2 Implications for Educators, Carers, Medical
Professionals and Policy Makers
For people who work in one of these areas as professionals, it is
equally benefcial to be aware of the fundamental objectives of spe-
cifc AT and the previously referred to models of disability. Before
providing specifc AT to people with disabilities, providers need
to be aware of the intended efect and the range of potential side
efect of these technologies as they relate to the situated contexts
of the disabled person they are implementing AT for. Therefore, it
is necessary to have a good knowledge of the technologies and to
practise open communication to ensure that identifed advantages
and disadvantages are communicated to all stakeholders accord-
ing to their communicative preferences. In the course of providing
certain technologies to people with disabilities, it is essential to
consider their individual living situations and to understand the
individual application contexts and socio-technical ecologies of
care [1]. Further, policy makers have to provide the framework
conditions that enable individualised care along the preferences
and desires of the disabled person themselves. We identify here the
risk of overriding personally established strategies or use intentions
ASSETS ’22, October 23–26, 2022, Athens, Greece
disabled people might have with the assumed potential of use often
persuasively communicated by technologists.
4.3 Implications for Disabled People
Disabled people have a nuanced understanding of their own life sit-
uation as long as they are allowed to share it on their own terms and
along their individual communicative preferences (see, for example,
for autism contexts [14]). Therefore, it is especially important for
them to become aware of the diferent kinds of impact specifc AT
can have on their lives. An open and honest communication from
all involved can make the diference, even and particularly if these
conversations might be fraught with power dimensions that make
the refusal of AT difcult to communicate and follow through. An
already widely used form of experience exchange wit AT is to pro-
vide self-organized peer counseling through social networks. I hope
that more and more people will participate in this type of commu-
nication and develop collective strategies in having conversations
around AT. In my opinion, users have a certain responsibility in
the interaction with AT as well. We need to actively take on our
expert roles and refect on the consequences and possible side ef-
fects before we integrate a new technology or technique into our
lives. I would like to further encourage anyone who is interested
in AT to get involved in research, development, design, supply or
communication - be it as a regular collaborator, as a testimonial or
in providing constructive feedback. In line with Mankof et al. [11],
it is my conviction that the involvement of people with disabilities
to represent their own interests is an essential contribution to the
development of AT.
5 CONCLUSION
This experience report was written with the motivation to con-
tribute to the understanding and discourse about Assistive Tech-
nologies in the context of disabilities. I intended to accomplish
this by communicating my personal experiences and insights to a
broad audience of AT stakeholders. Therefore, I described my own
impairment and explained my understanding of disability. Subse-
quently, I listed the AT most relevant to my everyday life, illustrated
in which form I use them and which activities they enable me to
do. By discussing some representative examples, I could illustrate
the potential infuences and efects the application of AT have for
me specifcally. On the basis of this understanding, I identifed
relevant implications for diferent stakeholder groups concerned
with assistive technologies. In conclusion, the implementation and
advancement of AT for people with disabilities holds huge opportu-
nities but we need to turn more honestly towards their risks as well,
particularly to their risks of failure or malfunctioning. We expect
that a deeper understanding of the socio-technical aspects of the
intended application context, the analysis of diferent dependencies
AT might introduce or amplify, the integration of afected people in
the development process as well as a responsible approach by ev-
eryone involved could help to utilize the opportunities AT promises
and minimize the risks that might come along with them.
ACKNOWLEDGMENTS
Part of this work has been funded by the Austrian Science Fund
(FWF) project T 1146-G.
ASSETS ’22, October 23–26, 2022, Athens, Greece
REFERENCES
[1] Mark S Ackerman, Ayşe G Büyüktür, Pei-Yao Hung, Michelle A Meade, and
Mark W Newman. 2018. Socio-technical design for the care of people with spinal
cord injuries. In Designing Healthcare That Works. Elsevier, 1–18.
[2] Valéria Baldassin, Helena Eri Shimizu, and Emerson Fachin-Martins. 2018. Com-
puter assistive technology and associations with quality of life for individuals
with spinal cord injury: a systematic review. Quality of Life Research 27, 3 (2018),
597–607.
[3] Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence
as a Frame for Assistive Technology Research and Design. In Proceedings of the
20th International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’18). Association for Computing Machinery, New York, NY, USA, 161–173.
https://doi.org/10.1145/3234695.3236348
[4] Audrey Desjardins, Oscar Tomico, Andrés Lucero, Marta E. Cecchinato, and
Carman Neustaedter. 2021. Introduction to the Special Issue on First-Person
Methods in HCI. ACM Trans. Comput.-Hum. Interact. 28, 6, Article 37 (dec 2021),
12 pages. https://doi.org/10.1145/3492342
[5] Murray Edelman. 1974. The political language of the helping professions. Politics
& Society 4, 3 (1974), 295–310.
[6] Jean D Hallewell Haslwanter and Geraldine Fitzpatrick. 2017. Why do few
assistive technology systems make it to market? The case of the HandyHelper
project. Universal Access in the Information Society 16, 3 (2017), 755–773.
[7] Donna Haraway. 1988. Situated Knowledges: The Science Question in Feminism
and the Privilege of Partial Perspective. Feminist Studies 14, 3 (1988), 575–599.
[8] Brent Hecht, Lauren Wilcox, Jefrey P Bigham, Johannes Schöning, Ehsan Hoque,
Jason Ernst, Yonatan Bisk, Luigi De Russis, Lana Yarosh, Bushra Anjum, et al.
2018. It’s time to do something: Mitigating the negative impacts of computing
through a change to the peer review process. arXiv preprint arXiv:2112.09544
(2018).
[9] Eve Lacey. 2014. Alison Kafer, Feminist Queer Crip (Indiana: Indiana University
Press, 2013), pp. 258, ISBN: 9780253009340, £16.99, paperback. Studies in the
Maternal 6 (Jan. 2014). https://doi.org/10.16995/sim.11
[10] Richard E Ladner. 2011. Accessible technology and models of disability. In Design
and use of assistive technology. Springer, 25–31.
[11] Jennifer Mankof, Gillian R. Hayes, and Devva Kasnitz. 2010. Disability Studies as
a Source of Critical Inquiry for the Field of Assistive Technology. In Proceedings of
Felix Fussenegger and Kata Spiel
the 12th International ACM SIGACCESS Conference on Computers and Accessibility
(Orlando, Florida, USA) (ASSETS ’10). Association for Computing Machinery,
New York, NY, USA, 3–10. https://doi.org/10.1145/1878803.1878807
[12] Deborah Marks. 1997. Models of disability. Disability and Rehabilitation 19, 3
(Jan. 1997), 85–91. https://doi.org/10.3109/09638289709166831 Publisher: Taylor
& Francis.
[13] Janis Lena Meissner, John Vines, Janice McLaughlin, Thomas Nappey, Jekaterina
Maksimova, and Peter Wright. 2017. Do-It-Yourself Empowerment as Experienced
by Novice Makers with Disabilities. In Proceedings of the 2017 Conference on
Designing Interactive Systems (Edinburgh, United Kingdom) (DIS ’17). Association
for Computing Machinery, New York, NY, USA, 1053–1065. https://doi.org/10.
1145/3064663.3064674
[14] Damian EM Milton. 2014. Autistic expertise: A critical refection on the production
of knowledge in autism studies. Autism 18, 7 (2014), 794–802.
[15] Christine Miserandino. 2017. The spoon theory. In Beginning with Disability.
Routledge, 174–178.
[16] Jenny Morris. 2014. Pride against prejudice: Transforming attitudes to disability.
The Women’s Press.
[17] Ingunn Moser. 2006. Disability and the promises of technology: Technology,
subjectivity and embodiment within an order of the normal. Information, com-
munication & society 9, 3 (2006), 373–395.
[18] Mike Oliver. 1990. THE INDIVIDUAL AND SOCIAL MODELS OF DISABILITY.
(1990), 7.
[19] KEVIN PATERSON and BILL HUGHES. 1999. Disability Studies and Phenomenol-
ogy: The carnal politics of everyday life. Disability & Society 14, 5 (Sept. 1999),
597–610. https://doi.org/10.1080/09687599925966 Publisher: Routledge _eprint:
https://doi.org/10.1080/09687599925966.
[20] Amon Rapp. 2018. Autoethnography in human-computer interaction: Theory
and practice. In New directions in third wave human-computer interaction: Volume
2-methodologies. Springer, 25–42.
[21] Tom Shakespeare. 2006. Disability Rights and Wrongs. Routledge, London.
https://doi.org/10.4324/9780203640098
[22] Rua M. Williams and Juan E. Gilbert. 2019. Cyborg Perspectives on Computing
Research Reform. In Extended Abstracts of the 2019 CHI Conference on Human
Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association
for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/
3290607.3310421
“I Used To Carry A Wallet, Now I Just Need To Carry My Phone”:
Understanding Current Banking Practices and Challenges
Among Older Adults in China
Xiaofu Jin
IIP (Computational Media and Arts Thrust)
The Hong Kong University of Science and Technology
Hong Kong SAR, China
xjinao@connect.ust.hk
ABSTRACT
Managing fnances is crucial for older adults who are retired and
may rely on savings to ensure their lives’ quality. As digital banking
platforms (e.g., mobile apps, electronic payment) gradually replace
physical ones, it is critical to understand how they adapt to digital
banking and the potential frictions they experience. We conducted
semi-structured interviews with 16 older adults in China, where
the aging population is the largest and digital banking grows fast.
We also interviewed bank employees to gain complementary per-
spectives of these help givers. Our fndings show that older adults
used both physical and digital platforms as an ecosystem based on
perceived pros and cons. Perceived usefulness, self-confdence, and
social infuence were key motivators for learning digital banking.
They experienced app-related (e.g., insufcient error-recovery sup-
port) and user-related challenges (e.g., trust, security and privacy
concerns, low perceived self-efcacy) and developed coping strate-
gies. We discuss design considerations to improve their banking
experiences.
CCS CONCEPTS
• Human-centered computing → Accessibility; • Social and
professional topics → Seniors.
KEYWORDS
Older adults, elderly, seniors, aging, banking, virtual bank, elec-
tronic payment, mobile banking, accessibility, technology use, digi-
tal inclusion, digital equity
∗ Corresponding
Author
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544820
Mingming Fan∗
Computational Media and Arts Thrust
The Hong Kong University of Science and Technology
(Guangzhou)
Guangzhou, China
Division of Integrative Systems and Design
Department of Computer Science and Engineering
The Hong Kong University of Science and Technology
Hong Kong SAR, China
mingmingfan@ust.hk
ACM Reference Format:
Xiaofu Jin and Mingming Fan. 2022. “I Used To Carry A Wallet, Now I
Just Need To Carry My Phone”: Understanding Current Banking Practices
and Challenges Among Older Adults in China. In The 24th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22),
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 16 pages.
https://doi.org/10.1145/3517428.3544820
1 INTRODUCTION
Banking, as an indispensable service of daily life, is undergoing un-
precedented changes under the trend of the digital economy. With
the increasing penetration of high-speed internet and electronic
devices (e.g., tablets, smartphones), digital banking has become
increasingly pervasive [3, 21, 115, 126]. In addition to banking web-
sites and mobile banking apps, electronic payment services (e.g., Ali-
Pay [4], ApplePay [10], Google Pay [93], PayPal [94], Venmo [113],
WeChat Pay [119], Yunshanfu [131], ZellePay [134]) and virtual (i.e.,
online-only) banks (e.g., Netbank [84], Webank [118], Monzo [80])
recently emerged as alternative digital banking platforms. More-
over, this trend has been accelerated by the COVID-19 pandemic.
Many banks have shut down their physical branches and replaced
them with more digital banking platforms [12, 52, 100, 109].
As face-to-face interactions in physical banks have been grad-
ually replaced by digital user interfaces on websites and mobile
apps, researchers have investigated people’s experiences and atti-
tudes toward online and mobile banking [46, 57, 128] and digital
payment [66, 135]. However, such studies primarily focused on
young adults. Compared to young adults, older adults tend to use
technologies to a lesser extent and feel more reluctant to adopt new
technologies [25, 88, 111]. They also tend to encounter more dif-
culties when using new technologies in general due to factors such
as age-related declines (e.g., cognitive, physical) [44, 78, 132], gener-
ation/cohort efects [132], digital divide/barriers [20, 90], and fewer
educational opportunities to keep up with the technology [98].
Consequently, older adults might face more challenges when
adopting digital banking.
On the other hand, compared to young adults, older adults have
accumulated more experiences with a variety of technologies over
decades and may have diferent criteria for “good technology”. As a
ASSETS ’22, October 23–26, 2022, Athens, Greece
result, understanding what older adults have to say about technol-
ogy would be benefcial for designing more inclusive technology,
not just for themselves but also for everyone. After all, aging is a
process that everyone is experiencing [53].
Thus, it is critical to understand older adults’ banking expe-
riences to improve the accessibility of emerging digital banking
platforms.
Researchers have already begun to explore older adults’ digital
banking experiences recently [20, 38, 81, 89, 95, 99, 136]. While
informative, these studies primarily focused on specifc aspects of
digital banking, such as the user experience of bank websites on
desktop computers [38], strategies to build trust in internet bank-
ing [99], and the efect of self-efcacy and anxiety on internet
banking [95]. A recent survey study took a step further to un-
derstand older adults’ overall banking practices in light of recent
technological development and found that although the majority
of older adults still used physical banks, a small percentage of them
started to adopt digital banking and digital payments [48]. However,
this survey study mainly provided a quantitative overview of older
adults’ banking practices, and it remains largely unknown why and
how older adults choose to use diferent baking platforms and the
challenges they encounter.
Inspired by this line of work, we took a step further to explore
the following research questions (RQs):
• RQ1: How do older adults use physical and digital banking
platforms? What are the perceived pros and cons of these
banking platforms?
• RQ2: What are older adults’ motivations for learning digital
banking?
• RQ3: What are the challenges that older adults encounter
when using digital banking platforms?
To answer RQs, we conducted in-depth semi-structured inter-
views with 16 older adults living in fve cities in China. China has
the largest older adult population [82, 91] and has been under-
going a fast growth in digital banking. For example, China is a
world leader in the adoption of contactless mobile payments with
81.1% usage penetration [30, 46] and has the highest growth rate
of electronic-payment transactions among all countries [21]. Un-
der this rapid transition to digital banking, older adults in China
have opportunities to experience both physical and digital banking
platforms and may encounter challenges associated with various
digital banking platforms.
Furthermore, we also conducted semi-structured interviews with
bank employees to gain an understanding of older adults’ banking
experiences from their complimentary perspectives as help-givers.
These bank employees had direct interactions with older adults and
accumulated experiences when assisting older adults with their
banking needs.
Our fndings show that older adults used both physical (e.g.,
bank counters, and ATMs) and digital banking (e.g., website, mobile
bank apps, electronic payment, virtual bank apps) platforms. They
performed diferent types of transactions on diferent platforms
based on their perceived pros and cons.
Moreover, our study uncovered three motivations for learning
digital banking, which was perceived usefulness, self-confdence,
Jin and Fan
and social infuence(e.g., being motivated by people in social cir-
cles). Furthermore, our study found that older adults encountered
app-related challenges (e.g., ambiguous afordance, low informa-
tion scent, insufcient error recovery support, lack of feedback)
and user-related challenges (e.g., anxiety caused by intangibility,
trustworthiness, security, and privacy, low perceived self-efcacy
and memory concerns) when using digital banking platforms.
We found that although older adults encountered challenges,
they showed a strong willingness to learn and apply their accumu-
lated experiences and knowledge to tackle problems before seeking
help from those they trust. Taken all together, we further discuss
the implications of our key fndings and propose design considera-
tions and future work for assisting older adults with overcoming
challenges in using digital banking and for supporting help-givers
to better assist older adults.
In sum, we make the following contributions based on both older
adults’ and bank employees’ perspectives:
• An understanding of how older adults use physical and digi-
tal banking platforms and their perceived pros and cons of
these platforms;
• An understanding of the motivations to learn digital banking;
• An understanding of banking challenges and design consid-
erations to improve older adults’ banking experiences.
2 BACKGROUND AND RELATED WORK
2.1 Banking Trend and Technology Adoption
for Older Adults
Banking services are increasingly digitized with the development of
information and communication technology (ICT) [5]. At the same
time, physical banks also constantly reduce the number of branches
and staf to save money [74]. While digital banking could bring
convenience to people who are adept at digital technology, such a
rapid technological shift may pose challenges to people who are
accustomed to traditional physical banking, such as older adults.
Prior research suggested that older adults tend to adopt new
technologies (e.g., computers, internet, tablets) slower and may be
less likely to use technologies in general than young adults [25,
88, 111]. Olson et al. conducted a survey study with 430 younger
adults and 251 older adultsand found that older adults tend to
be frequent users of long-standing technologies (e.g., telephone)
and less frequent users of more recent technologies (e.g., Internet,
ATMs) [88]. Digital banking was also found to be not as popular
among older adults as it was among the young in Asia [73].
Possible reasons for this age diference could be the age-related
health declines and the digital divide among older adults. Age-
related perception declines like vision impairments may make it
more difcult for older adults to perceive small icons when using
technology devices [26, 79], and age-related physical declines like
reduction in fne motor skills can cause older adults to encounter
more motor issues such as tapping and scrolling when using mobile
apps [129], and age-related cognitive issues, such as the reduced
speed of learning and memory difculties, may also slow down
their learning of digital technologies [28]. Moreover, older adults
were exposed to a diferent set of technological products when they
were in the workforce compared to younger generations [20, 90],
Understanding Banking Practices and Challenges Among Older Adults in China
which could potentially contribute to the digital divide between
older and young adults.
As a result, as an emerging new technology, digital banking
might impose new challenges for older adults, especially under the
current COVID-19 pandemic, people in many countries are advised
to avoid using in-person banking. To ensure the inclusiveness of
digital banking for older adults, it is critical to understand how older
adults perform banking activities on digital banking platforms and
the associated challenges.
In this work, we seek to explore this question with older adults
in the context of China. China has the world’s largest and fastest-
growing aging population [82, 91]. It is estimated to have 280 million
older adults by 2025, which will represent about one-ffth of its
total population [70, 91]. At the same time, the development of
the digital banking and e-commerce market in China is among the
fastest-growing in the world with a volume of 1.94 trillion USD
in 2019 [29, 46] and has almost become a cashless economy with
the fastest growing electronic payment [21, 54, 68]. As a result,
older adults in China have been experiencing the fastest shift from
traditional physical banking to digital banking and may experience
more challenges when learning and using new banking platforms.
By studying the banking practices and challenges that older adults
in China experience, we hope to reveal design opportunities to
better improve the banking experience for the aging population.
2.2 Older Adults’ Banking Practices
Physical banks (e.g., bank counters) are commonly used by older
adults [1, 2, 44, 89], and older adults were found to prefer to visit
bank branches over conducting banking transactions online [14,
72, 89]. In-person customer service ofered in physical banks was
rated as one of the top desired services from fnancial institutions
among older adults [1]. Despite the familiarity with physical banks,
older adults complained about the inconvenience associated with
visiting bank branches, such as the long wait time [48, 89].
The automated teller machine (ATM) was a technological innova-
tion to traditional banking products [20, 85], and the adoption rate
of ATMs among older adults has been slowly increasing over the
past decades [27, 41, 92, 102, 133]. Nonetheless, many older adults
still do not use ATMs [48]. One critical reason was that older adults
felt uncomfortable and less in control of their fnances when using
an ATM [27]. O’Brien et al. further identifed more factors infuenc-
ing ATM adoption among older adults, which include usefulness,
compatibility, complexity, technology generation, and relative ad-
vantage of a technology [92]. Compared to traditional ATMs, newer
versions of self-service banking machines, such as Cash Recycling
System (CRS) machines, could handle more banking transactions.
These CRS machines typically have a big touchable screen and can
handle many banking transactions. In this paper, we treat all of
them as ATMs.
Digital banking platforms, such as bank websites and mobile
banking apps, have recently emerged as alternatives to physical
banks and ATMs [5]. Researchers investigated whether and to what
extent older adults adopted digital banking platforms (e.g., [20, 38,
81, 89, 95, 99, 136]). Compared to young adults, older adults tend to
use digital banking to a lesser extent and preferred to use traditional
banking platforms (e.g., telephones, physical banks) [73, 88, 89].
ASSETS ’22, October 23–26, 2022, Athens, Greece
Furthermore, researchers investigated the factors to afect older
adults’ adoption of digital banking platforms, including the user
experience of banking websites [38], trust [99], fear [136], and self-
efcacy and anxiety [95].
In sum, prior research primarily focused on the adoption of a
particular type of banking platforms (e.g., physical banks [1], ATMs
[27, 41, 92, 102, 133], or digital banking platforms [17, 19, 38, 72, 81,
88, 89, 95, 99]) among older adults. However, it remains unknown
the practices and challenges of banking on both physical and digital
platforms (e.g., banking activities on each platform) among older
adults and how they make trade-ofs between these platforms.
Furthermore, electronic payment services (e.g., AliPay, ApplePay,
Google Pay, PayPal, Venmo, WeChat Pay, Yunshanfu, ZellePay)
and virtual banks (e.g., Netbank, WeBank, Monzo) are the latest
technological development in banking. Although a few studies
began to explore the use of electronic payment services and virtual
banks among young adults [66, 135], few focused on older adults.
Recently, a survey study showed that electronic payment had a
higher adoption rate than mobile banking and even ATMs among
older adults in China [48]. This has motivated us to understand
the practices and challenges of using electronic payment and virtual
banks among older adults as well as other banking methods. Such
an understanding may potentially explain why electronic payment,
despite coming around the same time as mobile banking and virtual
banks [124, 125, 127] and about 30 years later than ATMs [85], has
gained popularity in such a short time.
In this work, we seek to understand how older adults use physical
and digital banking platforms, how they learn to use digital banking
platforms, and the challenges encountered under current hybrid
digital and physical banking services.
3 METHOD
To answer RQs, we conducted IRB-approved semi-structured inter-
views with older adults to better understand their banking expe-
riences. Moreover, we also conducted semi-structured interviews
with bank employees who had experience helping older adults to
gain a complementary understanding of the difculties older adults
encountered.
3.1 Participants
Older Adults. Sixteen (N=16) participants aged 60 or older were re-
cruited through our social network and snowball sampling. Table 1
shows their demographic information. Nine were self-identifed as
female and seven as male. They resided in diferent tier-ed cities:
nine were in frst-tier cities, fve in second-tier cities, one in a third-
tier city, and one in a fourth-tier city. The tier system is roughly
based on the level of economic development, and the frst-tier cities
are considered to have the highest levels of economic develop-
ment in China, such as Beijing and Shanghai [121]. All participants
(Md=65, SD=7) used smartphones. Five also had a computer and
one also used an iPad.
Bank Employees. After interviewing older adults, we found
that although they wished to be assisted by their children, they
tended to go to banks in person for reasons like worrying about
their children not having the time or relevant knowledge, not living
with their children, and not wanting to reveal their fnancial info
ASSETS ’22, October 23–26, 2022, Athens, Greece Jin and Fan

Table 1: Older adults’ demographic information 3.2 Procedure

We obtained approval to conduct this research from the Institu-
Id Age Sex Location Devices used tional Review Board of our university. Interviews were held through
P1 61 F Shenyang computer, android phone phone calls, video calls, or in-person at participants’ places of choice
P2 61 M Shenyang computer, android phone by one researcher. All local COVID prevention protocols were
P3 60 M Huainan android phone strictly followed. All participants spoke Mandarin, and we also
P4 63 M Shenyang android phone conducted the interviews in Mandarin. While some participants
P5 62 M Jining android phone voluntarily showed us their apps to explain their points, our studies
P6 65 F Shanghai android phone did not ask for or record any personally identifable information.
P7 60 F Zhengzhou android phone The interviews lasted about an hour ranging from 47 minutes to 76
P8 76 M Shenyang android phone minutes and were audio-recorded. Each participant was compen-
P9 60 F Shanghai android phone sated with 70 CNY.
P10 63 F Shanghai android phone For older adult participants, they were frst asked to share their
P11 60 F Shanghai android phone banking practices including the activities they performed on various
P12 77 M Shenzhen computer, iPad, iPhone banking platforms (e.g., physical banks, ATMs, mobile banking
P13 79 F Shanghai android phone apps, virtual banks, and electronic payments) and the reasons for
P14 71 F Shanghai android phone their choices. Following a semi-structured format, we asked them
P15 62 M Shanghai computer, iPhone to freely talk about their usage patterns among these platforms
P16 60 F Shanghai computer, iPhone and the challenges they encountered as well as the strategies they
used to overcome the challenges. Specifcally, we asked participants
to explain their problems by showing the issues with their apps
for both in-person and video-call interviews. For phone-call-only
to their children. In contrast, they felt bank employees had worked interviews, we installed the same app as the participants to follow
with many older adults and had more experience helping them. their descriptions of the problems. Lastly, we asked about their
Thus, we also interviewed bank employees to gain their perspec- expectations of banking platforms.
tives. Seven (N=7) bank employees who had experience helping For banking employee participants, they were frst asked to
older adults were recruited through our social network and snow- estimate the proportion of older adult customers and their queuing
ball sampling. Table 1 shows banking employees’ demographic status(e.g., whether they need to queue, how long they line up), and
information. Five worked as lobby managers, who were responsible to share their observations and experiences regarding older adults’
for guiding and answering customer inquiries. Two worked as f- practices and challenges of using physical counters and digital
nancial managers, who were responsible for fnancial management banking methods. They were also asked to talk about the types of
transactions. While lobby managers could assist customers with transactions that they often help older adult customers with. After
most of the transactions, they will hand over fnancial manage- that, they were asked to share the difculties they encountered
ment requests to fnancial managers. In China, lobby managers when helping older adult customers and their banks’ policies for
are the ones who have the most contact with customers. Generally older adults. Finally, they were asked to share how they felt banking
speaking, they are the frst front to serve customers and then guide systems could be improved to better help older adults.
customers to queue or suggest trying digital methods without wait-
ing. They are also responsible for providing help on how to use
3.3 Data Analysis
digital banking platforms. Unlike bank tellers, who are only respon-
sible for transactions at the counters with a fxed process, lobby The interview recordings were auto transcribed, and one native
managers have more diverse and extensive experience in helping Mandarin-speaking author reviewed and corrected the transcripts.
older adults, which is the reason that we choose to interview lobby Two Mandarin-speaking authors frst familiarized themselves with
managers instead of bank tellers. the transcripts and then coded them independently using an open
coding method [22]. They met to discuss their codes and rationales,
revise their codes, and resolve disagreements to gain consensus on
Table 2: Bank employees’ demographic information the codes. After that, all the codes were translated into English, and
all the researchers performed afnity diagramming [13] to cluster
Id Sex Location Title the codes and categorize emerging themes related to our research
S1 M Tianjin Lobby manager questions with an inductive approach. We followed an iterative
S2 M Shanghai Lobby manager process to constantly challenge the groupings until we reached a
S3 F Tianjin Lobby manager consensus on the fnal groupings and their themes. re
S4 M Beijing Lobby manager
S5 M Shenyang Financial manager 4 FINDINGS
S6 M Dalian Lobby manager Our analysis revealed three main themes about practices and chal-
S7 F Beijing Financial manager lenges of banking among older adults: 1) Activities and Practices
on Diferent Banking Platforms, 2) Motivations to Learning Digital
Banking Platforms, and 3) Challenges in Using Digital Banking
Understanding Banking Practices and Challenges Among Older Adults in China
Platforms. We refer to older adult participants as “participants” and
bank employee participants as “bank employees” in the rest of the
section.
4.1 Activities and Practices on Diferent
Banking Platforms
4.1.1 Banking Transactions on Physical and Digital Platforms. Fig-
ure 1 shows participants’ banking platforms and transactions. Phys-
ical banking platforms include both bank counters (i.e., physical
banks) and ATMs. Digital banking platforms include mobile bank-
ing apps, virtual banks, and electronic payment. Next, we present
the details of the transactions on each platform.
Physical Platforms. Participants primarily conducted the fol-
lowing transactions in physical bank counters: deposit money, with-
draw money, manage wealth, and transfer money. Likewise, partic-
ipants also conducted a similar set of transactions on ATMs though
they tended to use ATMs more often to check balances and details
of their bank accounts.
Bank employees estimated that older adult customers took up
30% to 90% of all the customers visiting physical banks, depending
on the bank branch’s location, the date (e.g., day of pension release),
and the time of day. Our analysis found that the wide range ([30%-
90%]) was mainly related to banks’ locations and types. Banks on
the higher end of the range tended to be state-owned big banks
with a long trustworthy history and locate in residential areas.
The ones on the lower end tended to be relatively small banks or
locate in business districts. Moreover, bank employees reported that
older adult customers tended to food to the banks around the time
when pension payments are released, which crowded the banks and
overburdened the bank employees. Furthermore, bank employees
reported that older adults who did not use smartphones faced extra
challenges during the COVID-19 pandemic because banks required
their customers to scan the QR code to reveal their health code
on their phones to prove they are healthy before being allowed to
enter the banks, which were complained a lot by them.
Digital Platforms. Participants used three main forms of digital
banking platforms: mobile banking apps, virtual banks, and elec-
tronic payment. Participants used two types of mobile banking apps
to conduct fnancial transactions remotely. One type was provided
by traditional banks (see the left UI in the middle of Fig. 1), and they
used this type of mobile app to transfer money, manage wealth,
and conduct transactions related to a credit card. Bank employees
confrmed that mobile banking apps provided by traditional banks
provided the majority of the non-cash banking transactions.
The other type was provided by or in collaboration with stock
exchange (see the right UI in the middle of Fig. 1), such as Eastmoney
and Dazhihui [103, 105], and participants used such apps mostly to
perform wealth management, such as buying stocks or funds.
Virtual (online-only) banks are the ones that do not have physi-
cal branches but ofer banking services remotely [123]. Participants
used virtual banks’ apps to deposit money to their electronic pay-
ment account (e.g., WeChat Pay [119]—an electronic payment app),
pay bills, and manage wealth. Furthermore, participants also used
the virtual bank as a bridge to transfer money between accounts in
diferent physical banks. They would frst transfer money from one
bank account into the virtual bank and then transfer the money
ASSETS ’22, October 23–26, 2022, Athens, Greece
again out to another bank account. In such a way, they could avoid
a transaction fee associated with transferring money directly be-
tween two bank accounts because virtual banks have the beneft of
waiving online transferring transaction fees.
Electronic payment options include digital wallets (e.g., Alipay
Wallet, WeChat Wallet), contactless payment methods (e.g., Alipay,
WeChat Pay), etc. To use electronic payment, participants needed to
create a virtual account and bind one bank account with the virtual
account so that they could transfer money from their bank account
into the virtual account. The two most commonly used electronic
payment services among participants were WeChat Pay [119] (Fig. 1
right shows two UIs) and Alipay [4], and participants used them
regularly for daily activities, such as buying groceries, shopping
online, paying for meals, taking bus or taxi, and transferring money.
4.1.2 Perceived Pros and Cons of Banking Platforms. Physical
bank counters. Safety was the major perceived pros of physical
bank counters. In addition to a safe physical environment, being
able to correct errors with the assistance of bank employees was
another reason for perceived safety. “I would run into big trouble
if I accidentally typed 10 instead of 5 years for the deposit time on
my phone if I’m alone. But I could correct it right away with bank
employees.”- P3.
On the other hand, the long wait time was a frequently men-
tioned disadvantage of physical bank counters. Bank employees
reported that wait time was infuenced by the bank’s location, date,
and the time of day. For example, wait time would be longer if a
bank branch was the only one in the local area, or if it was the
pension release day. Especially on the day of pension release, some-
times even before the bank branch is open, people have already
fled a long line at the bank gate. Moreover, they mentioned that to
comply with improving regulatory requirements, the transaction
processes became longer than before, which further increased wait
time.
ATMs. ATMs were perceived to be convenient and time-saving
for certain banking transactions, such as withdrawing money. How-
ever, participants complained that they were unable to fnd out
whether ATMs had enough cash until they inserted their cards and
were several steps into the process. What’s more, participants were
concerned about the legibility of the ATM’s screen and often had to
deliberately bring reading glasses to overcome this problem. “Who
would always remember to bring the reading glass when going out?”-
P1. Indeed, bank employees also shared similar feedback from older
adult customers and pointed out some bank branches started to
provide reading glasses to customers. Moreover, participants were
also concerned about the fact that others could spot their passwords
over their shoulders when they used ATMs.
Furthermore, bank employees mentioned that the latest versions
of ATMs allowed customers to sign on the screen using a digital
pen. However, many older adults had hand tremors, which made it
difcult for them to sign on the screen.
Digital Platforms. One common advantage of all digital bank-
ing platforms was convenience. First, with digital platforms, partici-
pants could save physical eforts while still satisfying their banking
needs at home. “I could pay utility bills using AliPay (a virtual bank)
with a few clicks and don’t have to go out”-P2; “Those unnecessary
ASSETS ’22, October 23–26, 2022, Athens, Greece Jin and Fan

Figure 1: Participants’ banking platforms and transactions. An arrow from “traditional bank accoun” points to both physi-
cal banking platforms and digital banking platforms, which indicates that these platforms could be bound with traditional
bank accounts. Another arrow from “virtual bank accoun” points to virtual banks and electronic payments, which indicates
that these services could also be bounded with a virtual bank account. Diferent colors and types of the lines indicate the
connections between the banking platforms and the corresponding transactions that older adult participants performed.

human labors or human errors are saved by the information technol-
ogy.”-P5 Second, participants did not need to carry and deal with
banknotes, which were susceptible to both hygiene and counterfeit
issues. In particular, participants appreciated that they did not have
to carry their purses or wallet to store cash but just needed to carry
their smartphones, which almost all of them would carry anyway
when going out. “I used to carry a wallet and need another bigger bag
to store it along with my phone. Now, I only need to carry a small bag
to store my phone.” -P6; “It helps to avoid touching paper bills and is
more hygienic; this is especially important during the pandemic.” -P5.
Third, participants could transfer the money instantaneously on
digital platforms. P5 preferred to use pocket money (i.e., a function
of WeChat Pay [119]) to transfer money to his daughter, who stud-
ied far away, over traditional money transfers in banks because the
money would arrive at his daughter instantaneously to ensure her
fnancial needs without delay.
On the other hand, participants expressed privacy and security
concerns about digital banking platforms. Because of the security
concern about the electronic payment (e.g., WeChat Pay), P7 did
not directly bind her bank account with WeChat Pay. Instead, she
usually withdrew more cash from physical banks than she needed
and exchanged the extra cash with her trusted relatives so that they
could transfer money to her WeChat Pay. Moreover, participants
also had security concerns about ofine transactions. When paying
a person ofine (i.e., in-person) using an electronic payment app, the
app typically generates a barcode and displays it on their phone’s
screen for the person to scan. However, participants worried that
passers-by might be able to scan the barcode easily from a distance
and steal their money. “A Passer-by could easily use their phone to
scan the barcode on my phone from several meters away without my
awareness, so I never use it ofine.” -P7
4.2 Motivations to Learning Digital Banking
Platforms
We identifed three primary motivations for learning digital banking
platforms: perceived usefulness, self-confdence, and social infu-
ence.
Perceived Usefulness. Participants felt that it was convenient
to complete transactions for daily activities, such as paying gro-
ceries and transportation fees, on their smartphones, especially
for daily activities. Bank employees also reported that many older
adults were willing to try digital banking platforms even if they
had to wait in lines for a long time (e.g., over an hour).
Moreover, they were also motivated by economic benefts, such
as discount activity on online shopping platforms. “After hearing
from others that fruits on Pinduoduo (an e-commerce platform) were
cheap, I started to learn to use WeChat Pay.”-P16. Similarly, they
were also motivated by the fact that many ofine shops ofered dis-
counts when customers pay with electronic payment. “Yunshanfu
Understanding Banking Practices and Challenges Among Older Adults in China
(an electronic payment app) collaborates with local shops to ofer
discounts on their groceries, such as tofu, so I was motivated to learn
to use it.” -P8. Additionally, one participant was motivated by higher
interest rates of wealth management products ofered through mo-
bile banking apps. “Some wealth management products have higher
interests, but they are only available in mobile banking apps, so it is
worth learning.” -P14.
Another perceived usefulness of learning digital banking plat-
forms was to keep their brains sharp. “I’m getting old, and if I don’t
use it, I’ll lose it.” -P14.
Self-Confdence. Overall, participants expressed confdence in
their ability to learn digital banking. “I might be slower than others,
but just because I’m slow doesn’t mean I can’t learn it.” -P6; “Although
I am slower than young people, I could still manage to learn it.” -P14;
Moreover, their prior experiences with technology seemed to
help them gain confdence in learning digital banking platforms. P12
was used to learning things by himself because he did not live with
his children for a long time. “Many years ago I bought a computer
and started to learn to type, and I got used to this learning process
over time, so it’s not too hard for me to learn these operations on my
phone.”-P12. Similarly, P15 learned how to do wealth management
on the website using a computer and felt it was relatively easy to
transit conducting on mobile banking apps.
Besides, they tended to describe learning digital banking as trou-
blesome instead of difcult. “It was troublesome to use mobile banking
for wealth management because I only did it once per year but still
had to remember the steps.” -P4. Last but not least, they were able to
aford the learning eforts in particular if they could receive some
help along the way. “Someone taught me how to use it, and I felt it
was not that hard to learn.” -P1.
Social Infuence. All participants mentioned that they were
motivated by people in their social circles, such as their children,
grandchildren, and friends, who almost all used digital banking
platforms. P15 mentioned that he decided to learn to use digital
banking because he envied his colleagues for being able to use
digital banking apps to buy breakfast much faster. “I was reluctant
and resistant to learn it, but later I noticed that my younger colleagues
used their phones to pay for breakfast by simply scanning a code. In
contrast, I had to carry change to pay for my breakfast, which was
much slower. I really envied them and wanted to learn. Later, they
taught me how to use it.”-P15. Bank employees also reported that
there are few older adults who would like to proactively propose to
use digital banking methods to replace manual counters although
most of the banking transactions could be conducted via digital
banking platforms. Therefore, bank employees provide active guid-
ance and advertisement to encourage and lead older adults to use
digital banking tools like mobile banking apps. While there were
still a large proportion of older adults who would rather wait a long
time for the manual counters, which consists with the fndings of
the study [48]. Some older adults were indeed motivated by their
words and tried these new methods with their guidance.
Moreover, participants were motivated to learn because they
were involuntarily connected with digital banking platforms
through their social connections. P16 started to learn digital banking
because her friend transferred digital cash to her when paying her
back. She thought that she had to fnd a way to spend the money in
ASSETS ’22, October 23–26, 2022, Athens, Greece
the digital bank, so she gradually learned to use electronic payment
and now she could use it for online shopping.
Furthermore, participants felt that digital banking represented a
societal trend and they needed to keep up with the fast-changing
world and not become obsolete. “Society is moving forward, and new
technology is benefcial for societal development.”-P5; “Now every-
thing is online, so if you don’t know how to do it online, you will be
outdated.” -P6.
Lastly, participants also felt proud of being able to use digital
banking among their peers. P8 and P14 regarded being able to use
digital banking as a fashion. “I think learning digital banking is a
kind of pride and fashionable.” -P8.
4.3 Challenges in Using Digital Banking
Platforms
When using digital banking platforms, participants encountered
app-related challenges and user-related challenges.
4.3.1 App-Related Challenges. As for app-related challenges, P6’s
complaint may shout out many older adults’ thoughts, “It is not
designed for our older adults, it is designed for the young“-P6. Besides
participants, bank employees confrmed that many older adult cus-
tomers complained about the poor design and defective accessibility
of the banking app. There were four types of app-related chal-
lenges: ambiguous afordance, low information scent, insufcient
error recovery support, and lack of feedback or confrmation.
Figure 2: UIs for checking bills in a virtual bank (We-
Bank [118]): (a) a clickable option besides “All types of trans-
actions”; once clicked, a pop-up window shows up; (b) The
pop-up windows allows users to select and view the transac-
tions of a specifc type.
Ambiguous Afordance. Participants did not realize certain
functions existed due to the ambiguous afordance of UI elements.
For example, Fig. 2 a shows a list of “all types of transactions” in
March 2021. There was a gray triangle icon on the right side of
the title. When clicked, the interface pops up a window (Fig. 2 b),
which includes an “All” option for users to check their spending on
all types of transactions and other options, such as “Red packet”,
“Transfer”, “Pay credit card,” for them to check the spending of
a specifc type. However, the triangle icon was not perceived as
clickable. As a result, P1 believed that there was no way to check
her total spending: “WeBank shows all transactions together. When
ASSETS ’22, October 23–26, 2022, Athens, Greece
I had too many transactions, I needed to check my spending on a
specifc category. But I could not do it.”
Low Information Scent. One symptom of low information
scent was that participants had no idea certain functions existed in
the app. P5 was frustrated with not receiving real-time notifcations
about his spending when using the credit function of a virtual bank.
When the credit bill was due after a month, he often found he spent
more money than he realized: “It wasn’t transparent and didn’t send
any notifcation in time. I only realized the bill was higher than I
expected after a month.” This confusion was caused because P5 did
not realize that the real-time notifcation needed to be enabled in
the settings.
Moreover, participants were also frustrated about being redi-
rected from page to page. “ATM has clear steps, but the app does not.
You have to click here to enter a page, then click here to go to another
page, and so on and so forth.” -P1. Bank employees refected that it
did happen commonly during the use of banking apps and it was a
kind of burden for them to lead older adults to the exact page as
well.
Insufcient Error Recovery Support. The digital banking
apps did not provide sufcient guidance for participants to recover
from a mistaken operation. “I once clicked on a wrong thing and was
taken to a completely diferent place. I had no idea how to return to
the previous step and became panicked. In other situations, I would
have restarted my phone, but this was related to money and I was
afraid of losing money if I restarted my phone.” -P6. Thus, P6 went to
the bank to ask for help instead, but she felt this approach was too
costly to be a regular solution.
Moreover, insufcient error recovery support could also happen
at the operating system level. P8 reported that he once could not
fnd the electronic payment app after he accidentally switched to
other apps with an unintended touch gesture. “I didn’t know how
I switched to a diferent app [from the WeChat Pay app]. I tried to
swipe in diferent directions to get it back but I couldn’t.” -P8
Actually, participants expressed that they would like to do trial
and error by themselves because they could learn without troubling
others. Moreover, they felt that learning independently would help
them gain a deeper understanding and remember it longer. “If I
learn and do it myself, I would remember the process much better. My
wife always asked our kid to teach her and she still couldn’t remember
the steps. Thus, getting help from others is no better than learning it
by me.” -P2. After receiving a text message with a link from his bank,
P2 clicked the link on his phone, downloaded the banking app, and
tried to bind his bank card to it. However, without sufcient error
recovery support, he felt reluctant to try those unfamiliar functions
in the app.
Lack of Feedback or Confrmation. Participants also men-
tioned that their apps did not provide confrmation or feedback
when they fnished typing their bank account number and submit-
ting it. Consequently, they were unsure if they were successful or
not and therefore were hesitant to continue the following steps.
“The app has no confrmation after I enter the amount to be transferred.
As a result, I have to check many times to make sure I have entered
the correct information. Because of this, I am afraid of transferring
any large amount.” -P9
4.3.2 User-Related Challenges. Participants encountered the fol-
lowing user-related challenges: digital banking’s intangibility,
Jin and Fan
trustworthiness, security and privacy, overspending concerns, low
perceived self-efcacy, and memory concerns.
Intangibility. Unlike physical banks that had a physical location
and could provide printed receipts for transactions, digital banks
were perceived to be intangible. “I could not see what I buy in virtual
banks.”-P6. However, the worry about abstraction seemed to go
away as they gained more experience with digital banking over
time. “I was hesitant to use digital banks in the beginning, but I
gradually accepted it after using it for a while.”-P3.
Trustworthiness. The trustworthiness of digital banking plat-
forms also made participants worry. Unlike physical banks, in par-
ticular national banks (state-owned capital holding banks), that
had been there in their entire lives, digital banks were new. They
tended to think of digital banks the same as small local banks and
worried about their trustworthiness. They mentioned stories about
the bankruptcy of small banks. “Small local banks (whose the con-
trolling shareholder is the local government or private capital) may
run into cash fow issues and could even bankrupt like Haifa bank,
so although I use digital banks, I still worry about not being able to
get my money out.”-P5. Participants’ doubts in small banks were
reasonable. None of the nation-wise banks have bankrupted in
China. Although rare, one local bank (Hainan Development Bank)
did bankrupt in 1998 [60]. What’s more, digital platforms tend to be
riskier than physical ones. For example, hundreds of digital peer-to-
peer lending platforms were found to be fraudulent in 2018 [122].
The rampant fraud targeting older adults also made older adults
more cautious and anxious about digital banking.
Security and Privacy. The security and privacy of digital bank-
ing also caused anxiety. Participants mentioned that they heard
about online hackers and data breaches in the news. Moreover,
they also worried about losing money due to mistakes in electronic
payment. “What if there was a mistake in electronic payment and
others swiped my card?”-P3. Lastly, unlike physical banks that ofer
printed receipts, they felt virtual banks did not provide receipts
that they could use to go and get the money back if unexpected
things happened. “I don’t have any physical receipts. What if the
digital bank disappears all of a sudden like the app is gone, what do I
do with my money?”-P6. Bank employees echoed that some older
adults even did not trust the receipts printed by ATMs and only
trusted the ones from a bank counter.
Participants also mentioned privacy breaches though they were
unsure whether it was due to physical or digital banks. “I received a
lot of spam calls from all over the country. I didn’t know who leaked
my phone number but I suspected it was the banks since many calls
are for loaning stuf.” -P2
Overspending Concerns. Participants felt that it was harder
to keep track of their spending when using digital banking and
consequently they tended to overspend. they. “Sometimes I was
surprised how much I had spent when I was reviewing my monthly
bill.” -P6; “It (Digital banking) was indeed convenient. But I ran out of
budget much more easily [using it] than using cash. With cash bills, I
was prompted to think again about how much I would spend.” -P8
Low Perceived Self-Efcacy. Another user-related issue was
caused by low perceived self-efcacy. One recurring symptom was
being afraid of making mistakes. This was because making mistakes
would likely cause them to lose money. “When buying stocks in
mobile apps, I am extremely worried about making mistakes. For
Understanding Banking Practices and Challenges Among Older Adults in China
example, it is very easy to type an additional zero by accident.”-
P1 The concern about making mistakes was, even more, severer
when the mistakes were perceived as nonrecoverable. Reasons for
low perceived self-efcacy included the lack of confdence in their
literacy level and their declining physical conditions, such as poor
eyesight. “I don’t have much education, and it is common for me
to press the wrong button.” -P3; “I recommended WeChat Pay to my
sister. However, she resisted it because she was worried about making
mistakes and losing money due to her poor eyesight.” -P2 In this case,
they preferred to learn to use digital banking with assistance from
their close peers (e.g., colleagues, classmates, friends), their family
members (e.g., children, relatives), banking employees, and classes
ofered by local communities and volunteers, that made they felt
reassurance about learning.
When deciding from whom to seek help, participants reported
considering both the trustworthiness and expertise of the help
givers. Regarding trustworthiness, participants preferred to be as-
sisted by people who knew them well. “My children know my person-
ality and knowledge level. Thus, they would teach me at my level. But
others might start from a higher level. They may be kind and helpful,
but I won’t be able to understand it.” -P8. Moreover, participants also
worried that help givers might lose patience and assist them less
carefully, especially when there were other people waiting to get
help from the same help giver.
Indeed, bank employee participants also reported that in peak
hours they could only tell customers where to click with little
time for an explanation so as not to keep other customers waiting
for too long. In some cases, they would even operate directly on
older adults’ phones instead of guiding them even though this
was discouraged by bank regulations for privacy concerns. “The
pressure of serving all customers in line is high, so it is challenging
to be always patient and meticulous. In fact, more often than not,
the regulation of never operating directly on customers’ devices was
not well followed.” -S6. From older adults’ point of view, many were
willing to let bank employees operate on their phones, except for
inputting the password, to save time and efort. “Many older adults
mentioned that they had a bad memory and might forget it easily
after being taught, so it was better to operate it for them directly.” -S1.
Furthermore, some participants preferred help givers to monitor
their operation process and only give feedback when they ran into
trouble. “I’d like the bank employee to stand nearby to monitor my
progress while I’m trying to do it on my phone. I hope that he would
help me only when I run into trouble. Otherwise, I won’t be able to do
it next time." -P6. Bank employee participants confrmed that many
older adult customers wished to have them oversee their operations
to spot mistakes and provide answers, such as “I am going to tap on
this button. Are you sure I tap on the right button?” -S2. Moreover,
bank employees reported that many older adults had the ability to
conduct transactions in banking apps by themselves, “The majority
of them (older adults) did the right operations all the time, but they
just wanted me there to confrm each operation.”-S2.
Eventually, they still hoped to become as independent as possible
to avoid troubling others. “Young people are busy with their jobs. I try
to avoid bothering them and learn by myself”-P6. To help themselves
become independent, participants felt it helpful to have a voice
assistant whom they could ask for help. “It would be very helpful to
have a voice assistant. Whenever I didn’t understand it, I could just ask
ASSETS ’22, October 23–26, 2022, Athens, Greece
for help. I don’t need to go to banks to ask their employees. I don’t want
to bother others.” -P7. Meanwhile, bank employees expressed their
concerns that they might get impatient during busy hours with
older adults asking repetitive questions, as S6 refected, “If some
older adult customers keep asking me the same question while there
are so many people waiting in lines, my voice may sound impatient”.
This might also explain why older adults feel bad about troubling
others and want to be as independent as possible.
Besides learning with others’ assistance, during the usage of
digital banking platforms, participants developed four strategies to
manage the risk of losing money. First, they set an upper limit for
the amount of money that they would feel comfortable with even if
they would lose it. “I only put in 1000 CNY. Even if I became a victim
of a fraud, I would only have that much to lose, which is manageable
for me.” -P3. Second, they would not directly bind their regular bank
accounts with digital platforms. Instead, they bind a dedicated non-
primary bank account with digital banking platforms. In this way,
they could avoid the risk of losing money in their primary bank
accounts. The third strategy was to practice and get familiar with
digital banking platforms with a small amount of money frst. After
several successful trials, they would gradually increase the amount.
For example, when P6 started to use her virtual bank, she was
concerned about whether it was easy to withdraw money from it.
After successfully withdrawing money several times, she became
more confdent about it and started to put more money into it. “I
just wanted to withdraw money from AliPay (a virtual bank) to test
if it worked. After realizing it worked well, I would deposit the money
back.” -P6. Nonetheless, participants still tended to set a maximum
amount for digital platforms. Lastly, they believed that “avoiding
petty discounts” could also help them avoid falling for online fraud.
Memory Concerns. Another category of the user-related issue
was about forgetting various types of information related to digital
banking. First, they were concerned about forgetting the steps
of completing a bank transaction. “I followed the bank employee’s
instructions to walk through the process step by step. However, if I
didn’t write it in my notebook, I would forget about the steps.” -P11
In addition, they also worried about not being able to remember
how to use many functions at the same time. Two of them men-
tioned that after mastering basic functions, such as how to use
the electronic payment to pay bills, they worried that they might
forget how to use basic functions if they continued to learn more
advanced functions. “I can’t put too much information in my head.
Otherwise, I would mess it up and forget about it altogether.” -P13
Second, they were also concerned about forgetting passwords.
P11 reported that she forgot the password and did not know how to
fnd it back. The mobile banking app asked her to set the password
to be a combination of numbers and alphabets to ensure its security.
However, such requirements increased the burden of memorization
for older adults. For example, after many failed attempts, P11 fnally
had to work with a bank employee to get her money back and never
used the mobile banking app afterward.
5 DISCUSSION
Prior research investigated how older adults use physical banks [1],
ATMs [83], and online banks [89]. Although some researchers
studied how older adults use diferent banking platforms, they
ASSETS ’22, October 23–26, 2022, Athens, Greece
primarily ofered a quantitative overview of their usage via sur-
veys [48, 72, 88]. Moreover, electronic payment services (e.g., AliPay,
ApplePay, and WeChat Pay) have been increasingly integrated into
our societies in recent years. In particular, China has the high-
est growth rate of electronic transactions [21] and becomes the
current world leader in the usage of proximity mobile payments,
among which 81.1% of smartphone users adopt proximity mobile
payments [31]. Although a few recent studies explored how elec-
tronic payment services were used, they mainly focused on young
adults [66, 67, 135]. While a recent study provided a quantitative
understanding of how older adults use electronic payment through
a survey study [48], it did not explain why older adults chose to use
it over other banking methods. Building on top of prior work, our
work presents an in-depth qualitative understanding of why and
how older adults use both physical and digital banking platforms
in today’s technology landscape, how they learned to use digital
banking platforms, and the challenges that they encounter in a
country that has been experiencing one of the fastest growth of
electronic transactions. Furthermore, our work also provides com-
plementary perspectives from bank employees to better understand
older adults’ banking practices and challenges. We discuss our key
fndings, the design implications, and potential future work in this
section.
5.1 Banking Activities and Practices
Front-line bank employees reported that almost all older adults
came to the banks by themselves, showing their intention to be
independent. Older adults also had their own usage patterns with
consideration of diferent platforms’ pros and cons. Although their
understandings of diferent platforms varied in some aspects, they
generally agreed that security and timely in-person assistance are
two advantages of physical banks with a series of visible secure
checks and professional staf. On the other hand, the complexity
of the secure process combined with other factors like the bank
branch’s location and the date (e.g., the pension releasing day) also
caused inconvenience, such as the long wait time in physical banks.
In contrast, using digital banking platforms could mitigate this issue
and bring convenience for either conducting banking transactions
online or paying ofine without cash. However, it remains an open
question of how to design a hybrid online and ofine ecosystem to
integrate the advantages of the two.
Bank employees mentioned that almost all banking transactions
can be conducted through digital banking platforms except cash-
related ones. Prior studies pointed out that the two most common
transactions conducted by older adults were money transfers and
account/transaction inquiries [15, 48, 89]. This is consistent with
what we observed in our study. Participants frequently mentioned
their experiences of transferring money and checking balances or
details. They also refected challenges related to these two types of
transactions (Sec. 4.3), which suggests that even the most common
transactions still need to be improved. Future work could investigate
ways to improve the accessibility of older adults’ commonly used
transactions as a stepstone to increase their adoption of digital
banking platforms.
Furthermore, both older adults and banking employees summa-
rized the characteristics of banking transactions that older adults
Jin and Fan
conducted as being relatively simple and having a relatively low
risk. Reasons for not conducting more complex and risky transac-
tions among our participants include the concerns of losing money
and the lack of fnancial knowledge. Thus, future work should in-
vestigate ways to help older adults confdently learn to perform
more complex transactions.
5.2 Potential Cultural Efects
Our study was conducted with older adults living in China, and
their banking practices may have been infuenced by the Chinese
culture that difers from those of western countries. Our partic-
ipants reported some practices that were not found in western
countries. For example, older adults in China widely use digital
payment functions in WeChat, a social media app, to pay for daily
expenses (e.g., groceries, and transportation). In contrast, we have
not found a study reporting a similar trend in western contexts.
During Chinese traditional festivals, older generations often gift
younger generations a physical “red packet”, which is a red enve-
lope with cash in it. Interestingly, WeChat allows them to continue
practicing this tradition by sending a digital “red packet” with a
similar visual efect to each other conveniently.
On the other hand, there are common practices among older
adults in western countries, which are uncommon in China. For
example, older adults in western countries (e.g., the UK, the US,
Spain, and Canada) regularly use personal cheques [1, 58, 114].
However, people in China in general rarely use them. Moreover,
while it is common for older adults in western countries to use credit
cards, many major banks in China impose age restrictions on credit
card applications and exclude older adults [117]. This restriction
might explain why older adults in China have less experience with
credit cards [48]. One reason for rejecting older adult applicants is
that banks lack confdence in their ability to pay credit bills [117].
However, such a concern was not reported for western contexts in
prior studies. Furthermore, older adults in Canada were reported to
be comfortable with sharing online banking credentials with their
close ties [58]. In contrast, our participants tended not to want to
reveal their credentials to their close ties, who would then know
their fnancial status. Instead, they were comfortable sharing their
credentials with bank employees when seeking help because they
felt that bank employees’ work ethics would not allow them to
abuse their credentials. In sum, although our research scratched the
surface of the potential efects of culture on older adults’ banking
practices and challenges, more systematic and substantial follow-up
research is needed to further investigate the cultural efects.
5.3 Motivations to Learn Digital Banking
We found that older adults learned digital banking with three pri-
mary motivations: perceived usefulness, social infuence, and self-
confdence. These three motivations relate to several dimensions in
the UTAUT2 (Unifed Theory of Acceptance and Use of Technology)
model [112]. Specifcally, the “perceived usefulness” relates to both
“expectation of performance” and “price value” in UTAUT2, the “so-
cial infuence” relates to “social infuence”, and the “self-confdence”
relates to “expectation of efort”. Unlike the motivation of learning
social technologies for communication such as reducing loneliness,
Understanding Banking Practices and Challenges Among Older Adults in China
which happens more prevalent in older adults [87, 106], partici-
pants learned digital banking because they felt that digital banking
was convenient and ofered economic benefts. These motivations
aligned with younger adults’ acceptance of online banking and
electronic payments, which included “perceived ease of use, per-
ceived values, and perception of utility” [46, 66]. In addition, older
adults felt learning digital banking could keep their brains sharp.
This echoes the fnding of the previous study that older adults play
digital games to help treat age-related cognitive disorders [23] and
learn computer programming to keep their brains challenged [43].
Compared to the literature, the motivation to learn new technolo-
gies such as digital banking platforms for keeping brains sharp
is perhaps a unique motivation among older adults, while young
adults rarely mention it as a motivation.
Social infuence was found to afect younger adults’ adoption
and continued use of mobile payment services recently [67]. Our
work extends this fnding and confrms that it also afects older
adults’ adoption of digital banking platforms. Participants were
motivated by the adoption of digital banking among people in their
close social circles and felt they should learn it to keep up with the
social norms. This motivation echoes the previous research that
older adults are motivated and encouraged by people around them
such as their children and grandchildren to learn to use mobile
phones and computer devices [69] and do online shopping [65].
However, in contrast, previous research showed that, unlike
other ICTs, the intention to learn internet banking among older
adults is not signifcantly impacted by social infuence since users
tend to access banking services alone considering security is-
sues [11, 130]. One potential reason for this diference might be that
the new generation of digital banking services ofers more social
opportunities, such as sending digital money (e.g., a red packet) to
their social circles as fnancial and emotional support or seeing oth-
ers pay for goods with a single contactless tap, that was unavailable
decades ago. As our participants mentioned, they were motivated
to learn digital banking after they felt envy when seeing people
around them use it or received electronic money transferred from
others. Although “social infuence” mostly played a positive role in
increasing the adoption of digital banking among older adults, a
few participants also mentioned that their family members tried to
steer them away from digital banking to avoid potential fnancial
loss. This raises an open question of how to help older adults adopt
digital banking while minimizing potential fnancial loss.
In general, our participants felt that they might be slow in learn-
ing or need some help along the way just as the previous research
pointed out [25, 50, 79, 88, 108, 110, 129], but they believed they
could still do it. They also felt proud of being able to use digital
banking.
5.4 Challenges in Using Digital Banking
We discuss design considerations to address the challenges older
adults encounter when using digital banking.
5.4.1 App-Related Challenges. We uncovered four types of app-
related challenges: ambiguous afordance, low information scent,
insufcient error recovery support, and lack of feedback or confr-
mation. The frst two challenges were also found in a recent study
about older adults using digital maps [129]. The common challenges
ASSETS ’22, October 23–26, 2022, Athens, Greece
shared across drastically diferent apps (i.e. digital banking in our
case vs. mobile maps [129]) suggest that mobile app designs sufer
from common design faws for older adults.
Furthermore, modern app designs also tend to use icons to indi-
cate functions [9]. Although such designs can reduce text, our study
suggests that icon-based designs may lead to ambiguous afordance
issues. Instead of just focusing on solving a particular type of app
issue, we, as a community, should conduct more research to care-
fully scrutinize whether general mobile app design practices and
guidelines are suitable for older adults, and more importantly how
they might need to be adapted based on older adults’ experiences.
Toward this goal, we discuss some potential design solutions.
Minimize the number of visual elements per screen. When
reviewing digital banking apps, we noticed that they tended to show
many options on one screen. Although app designers often use dif-
ferent font sizes and colors to provide grouping and hierarchical
information to distinguish diferent options (e.g., [7, 8]), which is a
good practice in general, our study and other ones together provide
evidence that some UI elements with less visual salience would
likely go unnoticed if there are too many visual elements. As a
result, reducing the number of visual elements per screen would
likely increase the chance of each element being noticed by older
adults. What functions should be shown on the screen? Previous
research shows that older adults might prefer a multi-layered inter-
face design, which provides reduced functions for initial learning
and then progressively increases the interface complexity, over a
full-function interface [61]. However, one challenge with multi-
layered interface design is how to present additional functions to
older adults. One common approach is to hide them in deeper lay-
ers, which will result in a “deep and narrow” design. This raises
the “discoverability” issue. Indeed, our participants mentioned that
they did not realize the existence of certain functions that required
many steps to reach. It remains an open question of how to balance
the “visual salience” and “discoverability” of a function in mobile
apps.
Understand older adults’ interpretations of common
icons. One cause of the ambiguous afordance issue was the mis-
interpretation of graphical icons. For example, although a triangle
icon might be interpreted as clickable by young adults, our par-
ticipants did not always perceive it as clickable. Indeed, Berget
G. and Sandnes F.E. found that icons were not universally known
by all, and age had a positive correlation on the recognition of
the aged icons vs timeless icons [39]. Older adults were found
to have more problems using existing mobile device icons than
younger adults [62]. A recent study also showed that older adults
might misinterpret icons and interactive elements in online visual-
izations [35]. Furthermore, Rock et al. summarized that four icon
characteristics—semantically close meaning (i.e. natural, a close link
between depicted objects and associated function), familiar, labeled
and concrete (i.e. those depicting real-world objects)—improved
its usability for older adults, and they suggested allowing users to
choose an icon from a set of potentially suitable icons [62]. Similar
to icons, we also found misinterpretations of mobile app UI ele-
ments. Understanding how older adults perceive icons or common
mobile app UI elements would provide insights for app designers
to design ones that match older adults’ expectations. Thus, future
work should systematically explore older adults’ interpretation of
ASSETS ’22, October 23–26, 2022, Athens, Greece
banking icons or UIs. In addition to universal icons, icons with a
specifc cultural element can also be designed for older adults living
in that culture.
Explore interactive intelligent agents to assist older
adults. One way to combat the “low information scent” and “in-
sufcient error recovery support” is to allow older adults to ask
questions when they are confused. As voice-based intelligent assis-
tants (e.g., iOS Siri [107], Amazon Echo [120], Google Home [42])
continue being improved, researchers began to investigate their
potential in assisting older adults [51, 104]. Indeed, our participants
also expressed the need to ask others when learning digital banking.
Thus, it is worth investigating whether and how to design voice-
based intelligent assistants to help older adults overcome challenges
when using mobile apps (e.g., banking) when they have no idea
what to do with an interface (low information scent) or need to
recover from an error. Toward this goal, older adults’ think-aloud
verbalization and voice features can be used to build AI models
to detect when they encounter problems [32, 33, 36]. Furthermore,
bank employees mentioned that some pioneer banks started to
add voice interaction into mobile banking apps. However, for older
adults with poor hearing, the devices could not automatically in-
crease their voice to an appropriate level and react like a real bank
employee. Future work should explore how to design voice assis-
tants that are not only able to detect the problems older adults
encounter but also interact with them like real bank employees.
Combination of human support and advanced technolo-
gies. In addition to the aforementioned automatic voice assistants
with intelligence and voice-based interfaces, we can also consider
combining automatic features with human support to help older
adults. For example, when older adults frst learn to use digital
banking, it might be more helpful for them to interact with a real
remote bank teller, who can empathize better than a pure automatic
agent. Recent technologies, such as virtual and augmented reality,
might also allow older adults to interact with a live remote bank
teller through embodied avatars and shared visual cues. Moreover,
future research could explore ways to design features that enable
help-givers to draw visual guidance on older adults’ phones to sup-
port them in conducting banking transactions and even support
them to perform trial-and-error, which is efective for learning new
technologies but is often challenging for older adults to do [34, 35].
Another approach is to design a remote collaborative tool that al-
lows help-givers to demonstrate how to complete a banking task
with interactive guidance for older adults.
5.4.2 User-Related Challenges. We propose the following design
considerations to address three types of user-related challenges:
anxiety, low perceived self-efcacy, and memory concerns.
Increase older adults’ understanding of digital banking
with social help. The main causes for anxiety were related to the
intangibility, trustworthiness, and security and privacy concerns of
digital banking platforms. Previous research found that older adults
are concerned about security(e.g., [65, 77]) and privacy(e.g., [75])
issues when they access digital technology such as smartphones
and health monitor systems and they also experience challenges
with managing online security behaviors (e.g., [6, 37, 47, 86]) and
privacy settings (e.g., [37, 45, 86]) on their own. More recently, the
social support approach provided by older adults’ social networks
Jin and Fan
especially close-ties either family or peer friends has been found
efective to help older adults manage the security and privacy of
smartphones [56, 76, 116]. Compared to general smartphone us-
age, digital banking is more sensitive to older adults with possible
economic risks and privacy leaks, which may lead older adults to
feel more anxious while using. Thus, more research is warranted to
investigate ways to better inform older adults how digital banking
works, help them debunk false impressions, and build confdence
with social support while considering how to avoid invasion of
privacy during the support.
In the meantime, banking entities should also pay more attention
to security issues and further improve the mechanism of vulnerabil-
ity patching (e.g., provide various channels to collect faws reported
by users) [18] to fundamentally alleviate older adults’ anxiety about
digital banking platforms.
Integrate the feeling of companionship into the design of
digital banking. On the other hand, concerns about digital bank-
ing’s security and privacy were not unique to older adults. Younger
adults were also shown to worry about the perceived risk and pri-
vacy of digital banking services [46, 57]. One potential reason was
that online services lack physically present security personnel as in
physical banks [55]. This suggests the importance of the compan-
ionship of a trustworthy person for creating a safety feeling
of digital banking platforms. This was echoed by both the fact that
older adults would rather wait a long time to receive assistance
from bank employees [48] and in bank employees’ observations that
older adults felt safe when talking to them and receiving printed
receipts of the transactions. The sympathy that helps givers express
when interacting with older adults is what current digital services
lack, as S6 articulated: “Machines are not as smart as bank employ-
ees, but more importantly, not as sympathetic as them.” However,
bank employee participants felt that it was challenging to keep
being sympathetic and patient when they had too many customers
waiting to be served. One potential future direction perhaps is to
integrate companionship and sympathy into the design of voice-
based assistants to not only ofer help but also express sympathy
and companionship.
Increase older adults’ perceived self-efcacy. Our study
found that the main causes for “low perceived self-efcacy” were
being afraid of making mistakes and lack of confdence in their lit-
eracy level and their declining physical conditions. One common
underlying factor was the potential fnancial loss associated
with mistakes or misoperation. This suggested that older adults
might be more sensitive to “loss aversion” [49] when dealing with
digital banking platforms. This was evident in the approaches par-
ticipants took to minimize their potential fnancial loss: set an upper
limit for the amount of money put in digital banking platforms; bind
a dedicated non-primary bank account with the digital banking
platforms; practice with a small amount of money. Future work
should investigate ways to help older adults cope with loss aversion
by boosting their confdence when using digital banking platforms.
One potential approach is to deliver multi-model confrma-
tion instead of just visual confrmation. Current mobile app confr-
mation design heavily depends on visual feedback. However, older
adults often have declining eyesight and may not be sensitive to
all sorts of visual feedback (e.g., popup boxes). Instead, the app
Understanding Banking Practices and Challenges Among Older Adults in China
could deliver the confrmation in multiple channels, for example,
by popping up a box and reading the confrmation out [59].
Another approach is to design better error recovery mecha-
nisms. Current error-recovery mechanisms often require users
to accurately press a series of buttons (e.g., a back button placed
on the top left corner) to return to a previous step. However, it is
not uncommon for older adults to encounter touch-related motor
issues [24, 96, 97], such as tapping and swiping. Thus, touch-based
error recovery mechanisms might even lead to more errors along
the way. Furthermore, it is also challenging for older adults to fg-
ure out which step they should backtrack to. This raises an open
question of how best to help older adults recover from an error. One
possible direction is to leverage artifcial intelligence to infer the
step where older adults start to deviate from a correct completion
path and later guide them to backtrack to the step.
Lastly, our fndings revealed common “memory concerns”, such
as forgetting passwords or doing a required action. There is a body
of literature investigating ways to alleviate these memory issues,
such as designing more memorable authentication mechanisms
(e.g., [101]) and reminder systems (e.g. [16, 63]). Moreover, par-
ticipants were also afraid of forgetting the steps of completing a
bank transaction and wrote the steps down in a notebook. However,
writing the steps down is not a scalable approach to learning. One
potential solution is to leverage AI technology to learn the steps
for completing a task and alleviate older adults from needing to
remember the steps. For example, Li et al. showed an approach
for the mobile app to learn from the user how to complete a task
and then automate the process for the user for the same and even
other similar tasks [64]. Future work could investigate similar ap-
proaches to help older adults complete tasks without needing them
to remember the exact steps.
6 LIMITATIONS AND FUTURE WORK
Our study presents a qualitative understanding of why and how
older adults in China use both physical and digital banking plat-
forms, how they learn to use digital banking platforms and the
challenges that they encounter. Although our study included older
adults from rural areas (e.g., third-tier and fourth-tier cities), the ma-
jority of the participants lived in middle- and big-sized cities. As the
general technology development and the availability of digital bank-
ing services might difer in diferent-sized cities, our fndings might
not refect the banking practices and challenges of older adults
living in regions with diferent levels of economical development.
China has been experiencing one of the fastest growth in elec-
tronic transactions. Thus, older adults in China might have felt
stronger peer pressure to adopt digital banking compared to coun-
tries where digital banking is yet to gain popularity. Moreover, the
culture, household income, the occupational and educational back-
ground of older adults may also afect their mindsets about money
and their fnancial management strategies. Future work should
investigate older adults’ banking practices in diferent countries to
better understand cultural impacts and derive common and unique
challenges.
Furthermore, our study did not compare the user experience of
diferent banking apps. A controlled experiment with older adults
ASSETS ’22, October 23–26, 2022, Athens, Greece
using diferent banking apps would be able to understand the pros
and cons of diferent app designs.
Last but not least, our study shows that older adults are not
helpless but instead are quite willing to learn new digital banking
technologies and apply their accumulated experiences and knowl-
edge to come up with creative solutions before seeking help from
someone whom they trust. Inspired by this fnding and the advocate
for positive aging [40, 53], we argue that we should design interac-
tive learning approaches for older adults so that they could play a
more active and leading role in exploring new technological solu-
tions that still allow them to gain support from their close ties (e.g.,
children, grandchildren); This could also be a great opportunity to
foster a stronger intergenerational bind, which might potentially
reduce ageism [71].
7 CONCLUSION
Understanding why and how older adults use current banking plat-
forms is key to improving their experiences of managing fnancial
activities. Toward this goal, we have conducted a semi-structured
interview study with sixteen (N=16) older adults living in diferent
tiered cities in China to understand why and how they use physical
and digital banking platforms, how they learn digital banking, and
the challenges encountered. We have also interviewed bank employ-
ees to understand older adults’ banking practices and challenges
from their perspectives as help givers. Our fndings show that older
adults used both physical and digital banking platforms and per-
formed diferent types of transactions based on their perceived pros
and cons. They liked the familiarity and safety of physical platforms
and the convenience and instantaneity of digital platforms. They
tended to use digital platforms for everyday transactions involving
a small amount of money and physical platforms for transactions
involving a large amount of money.
Our study found three motivations for learning digital banking:
perceived usefulness, self-confdence, and social infuence. More-
over, our study revealed app-related and user-related challenges
that older adults encounter when using digital banking platforms
Specifcally, app-related challenges included ambiguous afordance,
low information scent, insufcient error recovery support, and lack
of feedback or confrmation. User-related challenges included anxi-
ety caused by intangibility and trustworthiness of digital banking,
lower perceived self-efcacy related to the fear of making mistakes
and the lack of confdence in their ability, and memory concerns.
Taken all together, we further discussed the design implications of
the fndings and potential design considerations, and future work
for assisting older adults with learning digital banking and address-
ing the challenges.
REFERENCES
[1] Maya Abood, Karen Kali, Robert Zdenek, et al. 2015. What can we do to help?
Adopting age-friendly banking to improve fnancial well-being for older adults.
Technical Report. Federal Reserve Bank of San Francisco.
[2] AARP Age UK. 2016. Age-friendly banking-What it is and how you do
it. Retrieved Jan 20, 2021 from https://www.ageuk.org.uk/globalassets/age-
uk/documents/reports-and-publications/reports-and-briefngs/money-
matters/rb_april16_age_friendly_banking.pdf
[3] Adel M Aladwani. 2001. Online banking: a feld study of drivers, development
challenges, and expectations. International journal of information management
21, 3 (2001), 213–225.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[4] Alipay. 2021. Alipay | Accessible digital payments for everyone. https://global.
alipay.com/platform/site/ihome
[5] Faris Alshubiri, Syed Ahsan Jamil, and Mohamed Elheddad. 2019. The impact of
ICT on fnancial development: Empirical evidence from the Gulf Cooperation
Council countries. International Journal of Engineering Business Management 11
(2019). https://doi.org/10.1177/1847979019870670
[6] Catherine L. Anderson and Ritu Agarwal. 2010. Practicing Safe Computing: A
Multimedia Empirical Examination of Home Computer User Security Behavioral
Intentions. MIS Q. 34 (2010), 613–643.
[7] Apple. 2021. Color - Visual Design - iOS - Human Interface Guidelines - Apple
Developer. https://developer.apple.com/design/human-interface-guidelines/ios/
visual-design/color/.
[8] Apple. 2021. Navigation - App Architecture - iOS - Human Interface Guide-
lines - Apple Developer. https://developer.apple.com/design/human-interface-
guidelines/ios/app-architecture/navigation/.
[9] Apple. 2021. System Icons - Icons and Images - iOS - Human Interface Guide-
lines - Apple Developer. https://developer.apple.com/design/human-interface-
guidelines/ios/icons-and-images/system-icons/.
[10] ApplePay. 2021. Apple Pay - Apple. https://www.apple.com/apple-pay/
[11] Jorge Arenas Gaitán, Begoña Peral Peral, and Mª Ramón Jerónimo. 2015. Elderly
and internet banking: An application of UTAUT2. Journal of Internet Banking
and Commerce, 20 (1), 1-23. (2015).
[12] BBC. 2020. Coronavirus: Bank branches close as virus afects access - BBC News.
https://www.bbc.com/news/business-52021246
[13] Hugh Beyer and Karen Holtzblatt. 1997. Contextual Design: Defning Customer-
Centered Systems. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
[14] Silvio Camilleri and Gail Grech. 2017. The Relevance of Age Categories in
explaining Internet Banking Adoption Rates and Customers’ Attitudes towards
the Service. Journal of Applied Finance and Banking 7 (01 2017), 29–47.
[15] Silvio John Camilleri and Gail Grech. 2017. The relevance of age categories in
explaining internet banking adoption rates and customers’ attitudes towards
the service. Camilleri, SJ, and G. Grech (2017), 29–47.
[16] Samantha WT Chan, Thisum Buddhika, Haimo Zhang, and Suranga
Nanayakkara. 2019. ProspecFit: In Situ Evaluation of Digital Prospective Mem-
ory Training for Older Adults. Proceedings of the ACM on Interactive, Mobile,
Wearable and Ubiquitous Technologies 3, 3 (2019), 1–20.
[17] Deepak Chawla and Himanshu Joshi. 2018. The moderating efect of demo-
graphic variables on mobile banking adoption: an empirical investigation. Global
Business Review 19, 3_suppl (2018), S90–S113.
[18] Sen Chen, Ting Su, Lingling Fan, Guozhu Meng, Minhui Xue, Yang Liu, and
Lihua Xu. 2018. Are mobile banking apps secure? what can be improved?. In
Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering
Conference and Symposium on the Foundations of Software Engineering. 797–802.
[19] Jyoti Choudrie, Chike-Obuekwe Junio, Brad Mckenna, and Shahper Richter.
2018. Research Archive Citation for published version: Understanding and
Conceptualising the Adoption, Use and Difusion of Mobile Banking in Older
Adults: A Research Agenda and Conceptual Framework. Journal of Business
Research 88 (2018), 449–465. https://doi.org/10.1016/j.jbusres.2017.11.029
[20] Jyoti Choudrie, Chike Obuekwe Junior, Brad McKenna, and Shahper Richter.
2018. Understanding and conceptualising the adoption, use and difusion of
mobile banking in older adults: A research agenda and conceptual framework.
Journal of Business Research 88 (2018), 449–465.
[21] McKinsey & Company. 2021. The 2020 McKinsey Global Payments Report.
https://www.mckinsey.com/~/media/mckinsey/industries/fnancialservices/
ourinsights/acceleratingwindsofchangeinglobalpayments/2020-mckinsey-
global-payments-report-vf.pdf
[22] Juliet Corbin and Anselm Strauss. 2014. Basics of qualitative research: Techniques
and procedures for developing grounded theory. Sage publications.
[23] Túlio Teixeira Cota, Lucila Ishitani, and Niltom Vieira. 2015. Mobile Game
Design for the Elderly. Comput. Hum. Behav. 51, PA (oct 2015), 96–105. https:
//doi.org/10.1016/j.chb.2015.04.026
[24] Alma Leora Culén. 2013. Touch-Screens and Elderly users: A Perfect Match. In
ACHI 2013, The Sixth International Conference on Advances in Computer-Human
Interactions. 460465.
[25] Sara J Czaja, Neil Charness, Arthur D Fisk, Christopher Hertzog, Sankaran N
Nair, Wendy A Rogers, and Joseph Sharit. 2006. Factors predicting the use of
technology: Findings from the center for research and education on aging and
technology enhancement (CREATE). Psychology and aging 21, 2 (2006), 333.
[26] Sara J Czaja and Chin Chiin Lee. 2009. Information technology and older adults.
Human-computer interaction: Designing for diverse users and domains (2009),
17–32.
[27] Ulrike Darch and Nerina J Caltabiano. 2004. Investigation of automatic teller
machine banking in a sample of older adults. Australasian Journal on Ageing 23,
2 (2004), 100–103. https://doi.org/10.1111/j.1741-6612.2004.00024.x
[28] Julie A. Delello and Rochell R. McWhorter. 2017. Reducing the Digital Divide:
Connecting Older Adults to iPad Technology. Journal of Applied Gerontology
36, 1 (jan 2017), 3–28. https://doi.org/10.1177/0733464815589985
Jin and Fan
[29] EMarketer. 2019. Global Ecommerce 2019. Retrieved Jan 20, 2021 from https:
//www.emarketer.com/content/global-ecommerce-2019
[30] eMarketer. 2019. Global Mobile Payment Users 2019. Retrieved April,13, 2021
from https://www.emarketer.com/content/global-mobile-payment-users-2019
[31] EMarketer. 2019. Global Mobile Payment Users 2019. Retrieved July 20, 2021
from https://www.emarketer.com/content/global-mobile-payment-users-2019
[32] Mingming Fan, Yue Li, and Khai N. Truong. 2020. Automatic Detection of
Usability Problem Encounters in Think-Aloud Sessions. ACM Trans. Interact.
Intell. Syst. 10, 2, Article 16 (may 2020), 24 pages. https://doi.org/10.1145/3385732
[33] Mingming Fan, Jinglan Lin, Christina Chung, and Khai N. Truong. 2019. Concur-
rent Think-Aloud Verbalizations and Usability Problems. ACM Trans. Comput.-
Hum. Interact. 26, 5, Article 28 (jul 2019), 35 pages. https://doi.org/10.1145/
3325281
[34] Mingming Fan and Khai N Truong. 2018. Guidelines for Creating Senior-Friendly
Product Instructions. ACM Transactions on Accessible Computing (TACCESS) 11,
2 (2018), 1–35.
[35] Mingming Fan, Yiwen Wang, Yuni Xie, Franklin Mingzhe Li, and Chunyang
Chen. 2022. Understanding How Older Adults Comprehend COVID-19 Inter-
active Visualizations via Think-Aloud Protocol. International Journal of Hu-
man–Computer Interaction 0, 0 (2022), 1–17. https://doi.org/10.1080/10447318.
2022.2064609 arXiv:https://doi.org/10.1080/10447318.2022.2064609
[36] Mingming Fan, Qiwen Zhao, and Vinita Tibdewal. 2021. Older Adults’ Think-
Aloud Verbalizations and Speech Features for Identifying User Experience Prob-
lems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing
Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery,
New York, NY, USA, Article 358, 13 pages. https://doi.org/10.1145/3411764.
3445680
[37] Alisa Frik, Leysan Nurgalieva, Julia Bernd, Joyce S. Lee, Florian Schaub, and
Serge Egelman. 2019. Privacy and Security Threat Models and Mitigation
Strategies of Older Adults. In Proceedings of the Fifteenth USENIX Conference
on Usable Privacy and Security (Santa Clara, CA, USA) (SOUPS’19). USENIX
Association, USA, 21–40.
[38] Chrysoula Gatsou, Anastasios Politis, and Dimitrios Zevgolis. 2017. Seniors’
experiences with online banking. In 2017 Federated Conference on Computer
Science and Information Systems (FedCSIS). IEEE, 623–627.
[39] Berget Gerd and Sandnes Frode Eika. 2015. On the Understandability of Pub-
lic Domain Icons: Efects of Gender and Age. In Universal Access in Human-
Computer Interaction. Access to Today’s Technologies, Margherita Antona and
Constantine Stephanidis (Eds.). Springer International Publishing, Cham, 387–
396.
[40] Mary M Gergen and Kenneth J Gergen. 2001. Positive aging: New images for a
new age. Ageing International 27, 1 (2001), 3–23.
[41] Mary C. Gilly and Valarie A. Zeithaml. 1985. The Elderly Consumer and
Adoption of Technologies. Journal of Consumer Research 12, 3 (1985), 353–
357. http://www.jstor.org/stable/254379
[42] Google. 2021. Discover what Google Assistant is. https://assistant.google.com/
[43] Philip J. Guo. 2017. Older Adults Learning Computer Programming: Motivations,
Frustrations, and Design Opportunities. In Proceedings of the 2017 CHI Conference
on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17).
Association for Computing Machinery, New York, NY, USA, 7070–7083. https:
//doi.org/10.1145/3025453.3025945
[44] Michael Harris, K Chris Cox, Carolyn Findley Musgrove, and Kathryn W Ernst-
berger. 2016. Consumer preferences for banking technologies by age groups.
International Journal of Bank Marketing (2016).
[45] Dominik Hornung, Claudia Müller, Irina Shklovski, Timo Jakobi, and Volker
Wulf. 2017. Navigating Relationships and Boundaries: Concerns around ICT-
Uptake for Elderly People. In Proceedings of the 2017 CHI Conference on Human
Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association
for Computing Machinery, New York, NY, USA, 7057–7069. https://doi.org/10.
1145/3025453.3025859
[46] Guangying Hua. 2008. An Experimental Investigation of Online Banking Adop-
tion in China. https://aisel.aisnet.org/amcis2008/36 AMCIS 2008 Proceedings.
36.
[47] Mengtian Jiang, Hsin-yi Tsai, Shelia Cotten, Nora Rifon, Robert Larose, and
Saleem Alhabash. 2016. Generational Diferences in Online Safety Perceptions,
Knowledge, and Practices. Educational Gerontology 42 (06 2016). https://doi.
org/10.1080/03601277.2016.1205408
[48] Xiaofu Jin, Emily Kuang, and Mingming Fan. 2021. ”Too Old to Bank Digitally?
”: A Survey of Banking Practices and Challenges Among Older Adults in China.
Association for Computing Machinery, New York, NY, USA, 802–814. https:
//doi.org/10.1145/3461778.3462127
[49] Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.
[50] Muath Alhussain Khawaji. 2017. Overcoming challenges in smart phone use
among older adults in Saudi Arabia. (2017).
[51] Sunyoung Kim. 2021. Exploring How Older Adults Use a Smart Speaker–Based
Voice Assistant in Their First Interactions: Qualitative Study. JMIR mHealth and
uHealth 9, 1 (2021), e20427.
Understanding Banking Practices and Challenges Among Older Adults in China
[52] Sharon Kimathi. 2020. Third wave of COVID-19 forces Hong Kong banks to close
branches - FinTech Futures. https://www.fntechfutures.com/2020/07/third-
wave-of-covid-19-forces-hong-kong-banks-to-close-branches/
[53] Bran Knowles, Vicki L. Hanson, Yvonne Rogers, Anne Marie Piper, Jenny Way-
cott, Nigel Davies, Aloha Hufana Ambe, Robin N. Brewer, Debaleena Chat-
topadhyay, Marianne Dee, David Frohlich, Marisela Gutierrez-Lopez, Ben Je-
len, Amanda Lazar, Radoslaw Nielek, Belén Barros Pena, Abi Roper, Mark
Schlager, Britta Schulte, and Irene Ye Yuan. 2021. The Harm in Confating
Aging with Accessibility. Commun. ACM 64, 7 (jun 2021), 66–71. https:
//doi.org/10.1145/3431280
[54] KPMG. 2020. Banking in the new reality: China. Retrieved Jan 20, 2021
from https://home.kpmg/cn/en/home/insights/2020/07/banking-in-the-new-
reality-china.html
[55] Umbas Krisnanto. 2018. Digital banking made transaction more trusted and
secured? International Journal of Civil Engineering and Technology 9 (12 2018),
395–407.
[56] Jess Kropczynski, Zaina Aljallad, Nathan Jefrey Elrod, Heather Lipford, and
Pamela J. Wisniewski. 2021. Towards Building Community Collective Efcacy
for Managing Digital Privacy and Security within Older Adult Communities.
Proc. ACM Hum.-Comput. Interact. 4, CSCW3, Article 255 (jan 2021), 27 pages.
https://doi.org/10.1145/3432954
[57] Sylvie Laforet and Xiaoyan Li. 2005. Consumers’ attitudes towards online and
mobile banking in China. International Journal of Bank Marketing 23, 5 (aug
2005), 362–380. https://doi.org/10.1108/02652320510629250
[58] Celine Latulipe, Ronnie Dsouza, and Murray Cumbers. 2022. Unofcial Proxies:
How Close Others Help Older Adults with Banking. In Proceedings of the 2022
CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA)
(CHI ’22). Association for Computing Machinery, New York, NY, USA, Article
601, 13 pages. https://doi.org/10.1145/3491102.3501845
[59] Ju-Hwan Lee, Ellen Poliakof, and Charles Spence. 2009. The efect of multimodal
feedback presented via a touch screen on the performance of older adults. In
International conference on haptic and audio interaction design. Springer, 128–
135.
[60] Legalweekly. 2013. Hainan Development Bank: The frst case of bank failure
in China(in Chinese). Retrieved July 7,2021 from http://fnance.sina.com.cn/
money/bank/20130626/072415917875.shtml
[61] Rock Leung, Leah Findlater, Joanna McGrenere, Peter Graf, and Justine Yang.
2010. Multi-layered interfaces to improve older adults’ initial learnability of
mobile applications. ACM Transactions on Accessible Computing (TACCESS) 3, 1
(2010), 1–30.
[62] R. Leung, J. McGrenere, and P. Graf. 2011. Age-related diferences in the initial
usability of mobile device icons. Behaviour & Information Technology 30 (2011),
629 – 642.
[63] Franklin Mingzhe Li, Di Laura Chen, Mingming Fan, and Khai N Truong. 2019.
FMT: A Wearable Camera-Based Object Tracking Memory Aid for Older Adults.
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Tech-
nologies 3, 3 (2019), 1–25.
[64] Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017. SUGILITE: creating
multimodal smartphone automation by demonstration. In Proceedings of the
2017 CHI conference on human factors in computing systems. 6038–6049.
[65] Jiunn-Woei Lian and David C. Yen. 2014. Online Shopping Drivers and Barriers
for Older Adults. Comput. Hum. Behav. 37, C (aug 2014), 133–143. https:
//doi.org/10.1016/j.chb.2014.04.028
[66] Francisco Liébana-Cabanillas, Inmaculada García-Maroto, Francisco Muñoz-
Leiva, and Iviane Ramos-de Luna. 2020. Mobile payment adoption in the age of
digital transformation: The case of Apple Pay. Sustainability 12, 13 (2020), 5443.
[67] Kuan-Yu Lin, Yi-Ting Wang, and Travis K. Huang. 2020. Exploring the an-
tecedents of mobile payment service usage: Perspectives based on cost–beneft
theory, perceived value, and social infuences. Online information review 44, 1
(2020), 299–318.
[68] Alex Lipton, David Shrier, and Alex Pentland. 2016. Digital banking manifesto:
the end of banks? Massachusetts Institute of Technology.
[69] Katrien Luijkx, Sebastiaan Peek, and Eveline Wouters. 2015. “Grandma, you
should do it—It’s cool” Older Adults and the Role of Family Members in Their
Acceptance of Technology. International journal of environmental research and
public health 12, 12 (2015), 15470–15485.
[70] Guoping Mao, Fuzhong Lu, Xuchun Fan, and Debiao Wu. 2020. China’s ageing
population: The present situation and prospects. In Population change and
impacts in Asia and the Pacifc. Springer, 269–287.
[71] Sibila Marques, João Mariano, Joana Mendonça, Wouter De Tavernier, Moritz
Hess, Laura Naegele, Filomena Peixeiro, and Daniel Martins. 2020. Determinants
of ageism against older adults: A systematic review. International journal of
environmental research and public health 17, 7 (2020), 2560.
[72] Minna Mattila, Heikki Karjaluoto, and Tapio Pento. 2003. Internet banking adop-
tion among mature customers: Early majority or laggards? Journal of Services
Marketing 17 (09 2003), 514–528. https://doi.org/10.1108/08876040310486294
[73] Mckinsey. 2015. Digital Banking in Asia:What do consumers really want?
Retrieved Jan 18,2021 from https://www.mckinsey.com/~/media/mckinsey/
ASSETS ’22, October 23–26, 2022, Athens, Greece
industries/fnancial%20services/our%20insights/capitalizing%20on%20asias%
20digital%20banking%20boom/digital_banking_in_asia_what_do_consumers_
really_want.pdf
[74] Mckinsey. 2019. Rewriting the rules in retail banking. Retrieved April 7,2021
from https://www.mckinsey.com/industries/fnancial-services/our-insights/
rewriting-the-rules-in-retail-banking#
[75] Andrew McNeill, Pam Briggs, Jake Pywell, and Lynne Coventry. 2017. Func-
tional Privacy Concerns of Older Adults about Pervasive Health-Monitoring
Systems. In Proceedings of the 10th International Conference on PErvasive Tech-
nologies Related to Assistive Environments (Island of Rhodes, Greece) (PE-
TRA ’17). Association for Computing Machinery, New York, NY, USA, 96–102.
https://doi.org/10.1145/3056540.3056559
[76] Tamir Mendel. 2019. Social Help: Developing Methods to Support Older Adults
in Mobile Privacy and Security (UbiComp/ISWC ’19 Adjunct). Association for
Computing Machinery, New York, NY, USA, 383–387. https://doi.org/10.1145/
3341162.3349311
[77] Tracy L Mitzner, Julie B Boron, Cara Bailey Fausset, Anne E Adams, Neil Char-
ness, Sara J Czaja, Katinka Dijkstra, Arthur D Fisk, Wendy A Rogers, and Joseph
Sharit. 2010. Older adults talk technology: Technology usage and attitudes.
Computers in human behavior 26, 6 (2010), 1710–1721.
[78] Tracy L Mitzner, Jyoti Savla, Walter R Boot, Joseph Sharit, Neil
Charness, Sara J Czaja, and Wendy A Rogers. 2018. Technol-
ogy Adoption by Older Adults: Findings From the PRISM Trial.
The Gerontologist 59, 1 (09 2018), 34–44. https://doi.org/10.1093/
geront/gny113 arXiv:https://academic.oup.com/gerontologist/article-
pdf/59/1/34/27456596/gny113.pdf
[79] Hazwani Mohd Mohadisdudis and Nazlena Mohamad Ali. 2014. A study of
smartphone usage and barriers among the elderly. In 2014 3rd International
Conference on User Science and Engineering (i-USEr). IEEE, 109–114.
[80] Monzo. 2021. Banking made easy. https://monzo.com/
[81] Nkosikhona Msweli and Tendani Mawela. 2020. Enablers and Barriers for Mobile
Commerce and Banking Services Among the Elderly in Developing Countries: A
Systematic Review. 319–330. https://doi.org/10.1007/978-3-030-45002-1_27
[82] Sonia Mukhtar. 2020. Psychological impact of COVID-19 on older adults. Current
medicine research and practice (2020).
[83] Ahmad Nedaei Fard. 2013. Barriers and Facilitators of Iranian Elderly in Use of
ATM Machines: A Qualitative Research in the Way of Cultural Probes. Salmand:
Iranian Journal of Ageing. (01 2013), 17–24.
[84] NetBank. 2021. NetBank - CommBank. https://www.commbank.com.au/digital-
banking/netbank.html
[85] BBC News. 2007. The man who invented the cash machine. Retrieved Jan 18,2021
from http://news.bbc.co.uk/2/hi/business/6230194.stm
[86] James Nicholson, Lynne Coventry, and Pamela Briggs. 2019. "If It’s Im-
portant It Will Be A Headline": Cybersecurity Information Seeking in Older
Adults. Association for Computing Machinery, New York, NY, USA, 1–11.
https://doi.org/10.1145/3290605.3300579
[87] Magnhild Nicolaisen and Kirsten Thorsen. 2014. Who are lonely? Loneliness in
diferent age groups (18–81 years old), using two measures of loneliness. The
International Journal of Aging and Human Development 78, 3 (2014), 229–257.
[88] Katherine E Olson, Marita A O’Brien, Wendy A Rogers, and Neil Charness. 2011.
Difusion of technology: frequency of use for younger and older adults. Ageing
international 36, 1 (2011), 123–145.
[89] Funmilola O Omotayo and Tolulope A Akinyode. 2020. Digital Inclusion and
the Elderly: The Case of Internet Banking Use and Non-Use among older Adults
in Ekiti State, Nigeria. Covenant Journal of Business and Social Sciences 11, 1
(2020).
[90] Tiago Nascimento Ordonez, Mônica Sanches Yassuda, and Meire Cachioni. 2011.
Elderly online: efects of a digital inclusion program in cognitive performance.
Archives of Gerontology and Geriatrics 53, 2 (2011), 216–219.
[91] World Health Organization et al. 2015. China country assessment report on
ageing and health. (2015).
[92] Marita A O’Brien, Katherine E Olson, Neil Charness, Sara J Czaja, Arthur D Fisk,
Wendy A Rogers, and Joseph Sharit. 2008. Understanding technology usage in
older adults. Proceedings of the 6th International Society for Gerontechnology,
Pisa, Italy (2008).
[93] Google Pay. 2021. Google Pay - Learn What the Google Pay App Is & How To
Use It. https://pay.google.com/intl/usa/about/
[94] Paypal. 2021. Send Money, Pay Online or Set Up a Merchant Account - PayPal.
https://www.paypal.com/us/home
[95] Begoña Peral-Peral, Ángel F Villarejo-Ramos, and Jorge Arenas-Gaitán. 2019.
Self-efcacy and anxiety as determinants of older adults’ use of Internet Banking
Services. Universal Access in the Information Society (2019), 1–16.
[96] Ryan M Peters, Monica D McKeown, Mark G Carpenter, and J Timothy Inglis.
2016. Losing touch: age-related changes in plantar skin sensitivity, lower limb
cutaneous refex strength, and postural stability in older adults. Journal of
neurophysiology 116, 4 (2016), 1848–1858.
ASSETS ’22, October 23–26, 2022, Athens, Greece Jin and Fan

[97] Andraž Petrovčič, Sakari Taipale, Ajda Rogelj, and Vesna Dolničar. 2018. Design [125] Wikipedia. 2021. NetBank. https://en.wikipedia.org/wiki/NetBank
of mobile phones for older adults: An empirical analysis of design guidelines [126] Wikipedia. 2021. Online Shopping. https://en.wikipedia.org/wiki/Online_
and checklists for feature phones and smartphones. International Journal of shopping.
Human–Computer Interaction 34, 3 (2018), 251–264. [127] Wikipedia. 2021. PayPal. https://en.wikipedia.org/wiki/PayPal
[98] Jakub Pikna, Nikoleta Fellnerova, and Michal Kozubík. 2018. INFORMATION [128] Jiaqin Yang, Li Cheng, and Xia Luo. 2009. A comparative study on e-banking
TECHNOLOGY AND SENIORS. CBU International Conference Proceedings 6 (09 services between China and USA. International Journal of Electronic Finance 3,
2018), 702. https://doi.org/10.12955/cbup.v6.1236 3 (2009), 235–252.
[99] Naimot Popoola and Md Razib Arshad. 2015. Strategic Approach to Build [129] Ja Eun Yu and Debaleena Chattopadhyay. 2020. “Maps are hard for me”: Identi-
Customers Trust in Adoption of Internet Banking in Nigeria. Journal of Internet fying How Older Adults Struggle with Mobile Maps. In The 22nd International
Banking and Commerce 20 (04 2015). ACM SIGACCESS Conference on Computers and Accessibility. 1–8.
[100] PTI. 2020. COVID-19 crisis: Credit Suisse to close 37 bank branches in Switzer- [130] Yee Yen Yuen, Paul HP Yeow, Nena Lim, and Najib Saylani. 2010. Internet
land. https://www.businesstoday.in/sectors/banks/covid-19-crisis-credit- banking adoption: Comparing developed and developing countries. Journal of
suisse-to-close-37-bank-branches-in-switzerland/story/414037.html Computer Information Systems 51, 1 (2010), 52–61.
[101] Karen Renaud and Judith Ramsay. 2007. Now what was that password again? A [131] Yunshanfu. 2021. Yunshanfu. https://yunshanfu.unionpay.com/
more fexible way of identifying and authenticating our seniors. Behaviour & [132] Salifu Yusif, Jefrey Soar, and Abdul Hafeez-Baig. 2016. Older people, assistive
Information Technology 26, 4 (2007), 309–322. technologies, and the barriers to adoption: A systematic review. International
[102] Wendy A. Rogers, Elizabeth F. Cabrera, Nef Walker, D. Kristen Gilbert, and Journal of Medical Informatics 94 (07 2016). https://doi.org/10.1016/j.ijmedinf.
Arthur D. Fisk. 1996. A Survey of Automatic Teller Machine Usage across the 2016.07.004
Adult Life Span. Human Factors 38, 1 (1996), 156–166. https://doi.org/10.1518/ [133] Valarie A Zeithaml and Mary C Gilly. 1987. Characteristics afecting the ac-
001872096778940723 PMID: 8682517. ceptance of retailing technologies: A comparison of elderly and nonelderly
[103] Eastmoney International Securities. [n.d.]. Eastmoney International Securities. consumers. Journal of retailing (1987).
https://www.emsec.hk/index.html [134] Zelle. 2021. Zelle® | A fast, safe and easy way to send and receive money.
[104] Shradha Shalini, Trevor Levins, Erin L Robinson, Kari Lane, Geunhye Park, https://www.zellepay.com/
and Marjorie Skubic. 2019. Development and comparison of customized voice- [135] Yu Zhao and Sherah Kurnia. 2014. Exploring Mobile Payment Adoption in
assistant systems for independent living older adults. In International Conference China.. In PACIS. 232.
on Human-Computer Interaction. Springer, 464–479. [136] M. Švecová and E. Odlerová. 2018. Smartphone and mobile application usage
[105] Ltd. Shanghai Dazhihui Information Technology Co. [n.d.]. Dazhihui. http: among seniors in Slovakia. European Journal of Science and Theology 14 (12
//www.dzh.com.cn/#page1 2018), 125–133.
[106] Tamara Sims, Andrew E Reed, and Dawn C Carr. 2017. Information and com-
munication technology use is related to higher well-being among the oldest-old.
The Journals of Gerontology: Series B 72, 5 (2017), 761–770.
[107] Siri. 2021. Siri does more than ever. Even before you ask. https://www.apple.
com/tw/siri/
[108] Aaron Smith. 2014. Older adults and technology use.
[109] Lauren SullivanMahum Tofq. 2020. Shuttered for COVID-19, some US
banks consider closing branches for good | S&P Global Market Intelligence.
https://www.spglobal.com/marketintelligence/en/news-insights/latest-news-
headlines/shuttered-for-covid-19-some-us-banks-consider-closing-branches-
for-good-58537195
[110] Michele Van Volkom, Janice C Stapley, and Vanessa Amaturo. 2014. Revisiting
the digital divide: Generational diferences in technology use in everyday life.
North American Journal of Psychology 16, 3 (2014), 557–574.
[111] Eleftheria Vaportzis, Maria Giatsi Clausen, and Alan J Gow. 2017. Older adults
perceptions of technology and barriers to interacting with tablet computers: a
focus group study. Frontiers in psychology 8 (2017), 1687.
[112] Viswanath Venkatesh, James Y. L. Thong, and Xin Xu. 2012. Consumer Ac-
ceptance and Use of Information Technology: Extending the Unifed Theory
of Acceptance and Use of Technology. MIS Quarterly 36, 1 (2012), 157–178.
http://www.jstor.org/stable/41410412
[113] Venmo. 2021. Venmo - Share Payments. https://venmo.com/
[114] John Vines, Mark Blythe, Stephen Lindsay, Paul Dunphy, Andrew Monk, and
Patrick Olivier. 2012. Questionable Concepts: Critique as Resource for Designing
with Eighty Somethings. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for
Computing Machinery, New York, NY, USA, 1169–1178. https://doi.org/10.
1145/2207676.2208567
[115] Christian Von Haldenwang. 2004. Electronic government (e-government) and
development. The European journal of development research 16, 2 (2004), 417–432.
[116] Zhiyuan Wan, Lingfeng Bao, Debin Gao, Eran Toch, Xin Xia, Tamir Mendel,
and David Lo. 2019. AppMoD: Helping Older Adults Manage Mobile Security
with Online Social Help. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.
3, 4, Article 154 (dec 2019), 22 pages. https://doi.org/10.1145/3369819
[117] Kexian Wang. 2015. A Study on Credit Card Application Restrictions for Older
Adults (in Chinese). Legal and Economy 3 (2015).
[118] WeBank. 2021. WeBank. https://www.webank.com/
[119] Wechatpay. 2021. WeChat Pay. https://pay.weixin.qq.com/index.php/public/
wechatpay
[120] Wikipedia. 2021. Amazon Echo. https://en.wikipedia.org/wiki/Amazon_Echo
[121] Wikipedia. 2021. Chinese city tier system. https://en.wikipedia.org/wiki/
Chinese_city_tier_system.
[122] Wikipedia. 2021. The collective failure of China’s online lend-
ing platform in 2018 (in Chinese). Retrieved July 7,2021 from
https://zh.wikipedia.org/wiki/2018%E5%B9%B4%E4%B8%AD%E5%9C%
8B%E7%B6%B2%E7%B5%A1%E5%80%9F%E8%B2%B8%E5%B9%B3%E5%8F%B0%
E9%9B%86%E9%AB%94%E5%80%92%E9%96%89%E4%BA%8B%E4%BB%B6
[123] Wikipedia. 2021. Direct bank. https://en.wikipedia.org/wiki/Direct_bank
[124] Wikipedia. 2021. Mobile banking. https://en.wikipedia.org/wiki/Mobile_
banking#cite_note-3
 Mobile Phone Use by People with Mild to Moderate Dementia:
Uncovering Challenges and Identifying Opportunities
Mobile Phone Use by People with Mild to Moderate Dementia
Emma Dixon∗
University of Maryland
eedixon@umd.edu
Yu Zhong
Google Inc.
yuzhong@google.com
Robin Brewer∗
University of Michigan
rnbrew@umich.edu
ABSTRACT
With the rising usage of mobile phones by people with mild de-
mentia, and the documented barriers to technology use that exist
for people with dementia, there is an open opportunity to study
the specifcs of mobile phone use by people with dementia. In this
work we provide a frst step towards flling this gap through an
interview study with fourteen people with mild to moderate demen-
tia. Our analysis yields insights into mobile phone use by people
with mild to moderate dementia, challenges they experience with
mobile phone use, and their ideas to address these challenges. Based
on these fndings, we discuss design opportunities to help achieve
more accessible and supportive technology use for people with
dementia. Our work opens up new opportunities for the design
of systems focused on augmenting and enhancing the abilities of
people with dementia.
CCS CONCEPTS
• Human-centered computing → Accessibility; Empirical stud-
ies in accessibility; Human computer interaction (HCI); Empirical
studies in HCI; Accessibility; Accessibility theory, concepts and
paradigms.
KEYWORDS
Dementia, Mobile Phones, Accessibility
ACM Reference Format:
Emma Dixon, Rain Michaels, Xiang Xiao, Yu Zhong, Patrick Clary, Ajit
Narayanan, Robin Brewer∗ , and Amanda Lazar. 2022. Mobile Phone Use by
People with Mild to Moderate Dementia: Uncovering Challenges and Iden-
tifying Opportunities: Mobile Phone Use by People with Mild to Moderate
∗ Also
afliated with Google Inc.
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3544809
Rain Michaels Xiang Xiao
Google Inc. Google Inc.,
rainb@google.com xiangxiao@google.com
Patrick Clary Ajit Narayanan
Google Inc. Google Inc.,
pclary@google.com ajitnarayanan@google.com
Amanda Lazar
University of Maryland
lazar@umd.edu
Dementia. In The 24th International ACM SIGACCESS Conference on Comput-
ers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM,
New York, NY, USA, 16 pages. https://doi.org/10.1145/3517428.3544809
1 INTRODUCTION
Dementia is a condition which involves changes in cognition and
afects the ability to engage in daily tasks and activities [74]. De-
mentia is typically progressive, with people in more mild stages of
dementia experiencing less changes to their functioning than those
in more advanced stages [74]. Many everyday technologies are not
designed to meet the access needs of individuals with dementia
[32, 36–38, 79]. At the same time, research shows a trend towards
greater mobile phone use by people with mild cognitive impairment
and mild dementia, with almost half using smartphones [42, 57].
The rising usage of mobile phones and the documented barriers
that exist lead to the opportunity to study the specifcs of mobile
phone use by people with dementia. This usage, as well as barriers
to use, are key to understand in order to design apps and features
that are useful and accessible for people with dementia. In this
study we address the following research questions:
• For what purposes do people with dementia use their mobile
phones?
• What challenges, if any, exist with mobile phone use?
• What opportunities do people with dementia envision to
support them when they encounter challenges with their
mobile phone use?
Through semi-structured interviews with fourteen people with
mild to moderate dementia, we learned that individuals used mobile
phones in everyday life not only to accommodate changes in cogni-
tive ability and emotional regulation, but also to stay productive and
manage their health. We uncovered three major challenges with
mobile phone use: 1) navigating to apps and features; 2) task execu-
tion in moments of high stress, fatigue, and time pressure; and 3)
re-learning task fows after updates and upgrades. To address these
challenges, participants described ideal interactions with mobile
phones, including customizable user interfaces, activity-based cus-
tomization, proactive technology assistance, and extended modali-
ties for voice-based interactions.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Based on these fndings, our work makes four primary contri-
butions to the literature. First, this paper provides an empirical
account of how fourteen people with mild to moderate dementia
use mobile phones and the challenges they face with mobile phone
use. Second, it describes participants’ ideas to address challenges
with mobile phone use. Third, abstracting from participants’ ideas
for future more accessible interactions with mobile phones, this pa-
per discusses design opportunities to help achieve more accessible
and supportive technology use for people with dementia. Finally,
this work opens up new opportunities for the design of systems
focused on augmenting and enhancing the abilities of people with
dementia.
2 RELATED WORK
The following section describes research on mobile phone use by
neurodiverse users. Following this, we describe research on barriers
to general technology use for people with dementia as well the pro-
vision of technical support when people with dementia experience
challenges with technology use.
2.1 Mobile Phone Use by Neurodiverse Users
Researchers have begun to investigate how neurodiverse users, such
as those living with traumatic brain injury, Down syndrome, and
autism use mobile phones. For example, researchers have found in-
dividuals with traumatic brain injury heavily relied on their phones
for reminders out of concern for forgetting upcoming events and
that they only used limited features as the more advanced features,
which may have been of use, were considered too complex [25].
Research with individuals with developmental disabilities broadly,
including people with Down syndrome, autism and other unspeci-
fed developmental disabilities, found that mobile phones played
a key role in safety [28], increasing independence [28], providing
entertainment [58], social connectedness [28, 58], and reminders
and scheduling support [29]. Researchers have noted specifc phone
features which can be barriers for individuals with developmental
disabilities (e.g., small buttons and complex menus [28]), as well
as how features can be better designed to support neurodiverse
users (e.g., large buttons, icons with titles, and a single level menu
structure for individuals with traumatic brain injury [73]).
Past researchers have designed and developed prototype apps
for tablets and mobile phones to support people across the stages
of dementia (e.g., to promote safe walking with GPS tracking on
mobile devices [46, 61, 100, 101]). Apps have also been designed
to support self-care and health management by people with mild
dementia [40, 52, 54]. Much of this past work focuses on the design
and evaluation of new prototype applications to support users
with mild to moderate dementia in important aspects of life. Given
the increasing number of people with mild dementia now using
mobile phones in their daily lives [42, 57], there is an opportunity to
understand how to design for people with dementia from another
angle: by studying existing use.
2.2 Barriers to Technology Use by People with
Dementia
Past research has identifed barriers that people living with mild
to moderate dementia experience with technology use, such as
Emma Dixon et al.
the cost of devices [56], ethical issues [39, 56], attitudinal aspects
[39, 56, 84], condition-related challenges [39, 56, 66, 78, 85], and
technology-related challenges [39, 56, 78].
In regards to specifc attitudinal aspects, researchers have argued
that people with dementia face difculties with using technology
due to low digital self-efcacy (belief in one’s capacity to execute
technology-related tasks) [6, 39, 78, 79]. Two studies compared digi-
tal self-efcacy of people with mild dementia to older adults without
dementia, fnding participants with mild dementia perceived using
technology to be more difcult than participants of the same age
[66, 85]. This low digital self-efcacy may impact the uptake of new,
potentially useful devices [18].
This low digital self-efcacy may be due, at least in part, to
changes that dementia brings which include memory, sensitiv-
ity to stress, orientation to place and time, and interpreting and
understanding information [78]. Past work investigating barri-
ers to technology use largely focuses on cognitive challenges
[39, 56, 66, 78, 85]. Though researchers have also described the
unique changes to sensory abilities that can cause barriers to tech-
nology, such as changes in speech patterns and language patterns
and difculty identifying diferent sounds [30]. In addition, re-
searchers have pointed to how the progressive nature of the condi-
tion afects technology use [39, 56, 84].
Past research has also investigated difculties people living with
dementia experience due to the design of various technologies,
such as websites [35, 43, 55], devices for in-home monitoring and
support [9, 12, 18, 21], and computers and cell phones [78]. These
difculties include challenges with familiarity [18], conspicuous
devices [18, 21], and complex interfaces [40, 43, 55]. Unlike past
work with other neurodiverse users [25, 28, 58], little work has
investigated the accessibility of mobile phones for people with mild
to moderate dementia. One exception reports difculty some tech-
savvy participants with mild to moderate dementia have typing on
their mobile phone due to small key size and identifying notifcation
ringtones and sounds [31]. With this limited understanding of the
technologically-related challenges people with mild to moderate
dementia experience, in this work, we set out to investigate the
specifc barriers people with mild to moderate have with mobile
phone use. Understanding these barriers is essential to designing
future systems to provide the necessary technical assistance to
combat these barriers.
2.3 Technical Support for People with Dementia
With the challenges that people with dementia face with technology
use, research to date has largely discussed ways informal caregivers
can provide technical assistance [27, 37, 53, 54, 68, 72, 77, 84, 88].
Past work found that while assistance from others can be useful, it
was not always desirable by people with dementia, with individuals
going to great lengths to avoid assistance from other people to
avoid burdening loved ones and to avoid others taking away tasks
that they had difculty with [32]. Some participants in this study
suggested that technical assistance mediated by technology may be
more desirable than human assistance. Researchers have proposed
one alternative to provide necessary technical assistance without
relying on caregivers - automatic personalization (e.g., [97, 98]) in
combination with AI to determine instances when an individual
Mobile Phone Use by People with Mild to Moderate Dementia: Uncovering Challenges and Identifying Opportunities
may need assistance and automatically provide this assistance or
make suggestions according to the perceived need [31]. Although
this past work suggests alternative solutions to provide technical
support to people with dementia, researchers have not yet investi-
gated how people living with mild to moderate dementia envision
the provision of technical support. In this work, we investigate
the perspectives of people with mild to moderate dementia on
the optimal provision of technical support when challenges arise
with mobile phone use and future opportunities for technological
support.
3 METHODS
To understand how people with mild to moderate dementia are
using mobile phones (RQ1), the specifc challenges they have with
mobile phone use (RQ2), and opportunities for technology to sup-
port people with dementia when they encounter these challenges
(RQ3), we conducted semi-structured interviews with fourteen peo-
ple with mild to moderate dementia. Below, we present our ap-
proach to recruitment, procedures for data collection, participant
demographics, analytic approach, and limitations.
3.1 Recruitment
Participants with mild to moderate dementia were recruited
through convenience sampling and snow-ball sampling. The frst
author reached out to 20 potential participants with a brief email
describing the study and the eligibility criteria. People were eligible
to participate in the study if they self-reported a clinical diagnosis of
mild to moderate dementia, owned a mobile phone, and used their
mobile phone daily. Seventeen potential participants responded to
the email from the frst author expressing interest in participating
in the study. Three of these potential participants who initially
showed interest in participating in the study were not comfort-
able with the interview being recorded and therefore chose not to
participate. The other fourteen potential participants consented
to participate electronically, and completed a demographics ques-
tionnaire. Aligning with best practices when working with people
with mild to moderate dementia [44] as well as legal and ethical
best practices in some countries [59, 93], we assumed participants’
capacity to consent. Although people in the mild to moderate stages
of dementia can experience changes in cognitive ability [74], they
are generally able to participate independently in research studies
[44]. All interviews were conducted remotely in November and
December of 2021.
3.2 Procedure
We conducted remote, semi-structured video interviews that were
split into two segments of questions. The frst segment focused
on participants present mobile phone use and any challenges they
experienced. The second segment focused on opportunities for
future mobile device interactions. This is in response to Lewis,
Sullivan, and Hoehl’s call to include individuals with cognitive dis-
abilities in the design and development of future more accessible
mobile phones [60]. Speculative futuring in the dementia space has
primarily used co-design methods, where people with dementia
use sketching and paper prototyping to ideate future technologies
[61]. Due to changes in dexterity, other researchers have sketched
ASSETS ’22, October 23–26, 2022, Athens, Greece
participants’ design ideas as participants describe them, with the
participants actively critiquing the designs being drawn [14]. How-
ever, researchers have found these low-fdelity prototyping methods
are not always efective for people with dementia who sometimes
prefer to verbally describe their design ideas over sketching [33].
For this reason, in the second section, we frst asked participants
to verbally describe their ideas to accommodate the challenges
disclosed during the frst section of the interview. We used ideas
participants shared in earlier interviews to engage later participants
in discussions about future mobile phone use.
Past work has also shown how using high-fdelity prototypes
helps participants with mild to moderate dementia grasp abstract
concepts even with changes in abstract thinking ability [33, 45].
For this reason, after participants had shared their ideas for future
technologies, we used a publicly available Android smartphone
application as a technology probe [48], to inspire participants to
think of new kinds of technology to support their needs and desires.
This app was chosen as the technology probe because it was de-
signed for neurodiverse users with cognitive disabilities to provide
a simplifed user interface with customizable buttons which enable
multi-step tasks with a single click (see Figure 1). We demonstrated
some functions of the technology probe which we believed may
be of specifc use to people with dementia, such as providing di-
rections, making phone calls to specifc people, using smart-home
devices, and opening specifc YouTube Channels. See Figure 1 for a
visual of the mobile phone home screen demonstrated during the
interview as the technology probe.
The semi-structured nature of the interview allowed us to ask
further probing questions to pursue topics guided by the infor-
mants themselves. Following each interview, participants received
a $75 gift card as an incentive. Interviews ranged from 47 to 67
minutes (average = 54 minutes). The interviews were audio and
video recorded, resulting in 13 hours and 27 minutes of data. We
provide the full study protocol, including interview questions, in
the supplementary materials.
3.3 Participants
The research team conducted 14 semi-structured interviews with
people with mild to moderate dementia. This aligns with the aver-
age sample sizes for remote interview studies [19]. Table 1 provides
more information on participant demographics and mobile phone
use. All participants were familiar with voice-assistants and some
of their functionality. All participants resided in the U.S. with one
participant, Miranda, residing in Canada. Throughout the paper we
use pseudonyms for participants.
3.4 Analysis
We used a thematic analysis approach to analyze the interview data
[16]. To become familiar with the data, the frst author verifed
computer-generated interview transcripts from audio recorded in-
terviews. The frst author then coded each transcript to generate
initial codes. The frst author grouped initial codes into potential
themes and went back through all interview transcripts to gather
quotes relevant to those themes. The research team reviewed and
discussed themes to ensure that they were relevant to the codes and
quotes extracted. The frst author created a thematic map, grouping
ASSETS ’22, October 23–26, 2022, Athens, Greece Emma Dixon et al.

Table 1: Participants’ Demographic Information

Pseudonym Gender Age Education Racial or Ethnic Mobile Phone Regularly received assists
Group Group Operating from others with mobile
System phone use
Thomas Man 55-64 Bachelor’s degree White iOS Yes
Elenora Woman 55-64 Master’s degree White Android No
Kim Woman 45-54 Bachelor’s degree Black or African iOS No
American
Tristen Man 65-74 Some college credit, White Android Yes
no degree
Sylvia Woman 55-64 Some college credit, Black or African Android No
no degree American
Preston Man 55-64 Some college credit, White Android No
no degree
Miranda Woman 55-64 Some college credit, White iOS Yes
no degree
Kennith Man 65-74 Some college credit, White iOS Yes
no degree
Tina Woman 65-74 Master’s degree White iOS No
Malcolm Man 55-64 Master’s degree White iOS Yes
Josslyn Woman 65-74 Some college credit, White Android No
no degree
Hall Man 65-74 High school White iOS Yes
diploma or the
equivalent
Alecia Woman 55-64 Some college credit, White Android No
no degree
Sabella Woman 55-64 Some college credit, Black or African iOS Yes
no degree American

themes under each of the three research questions of the study. The
names of the themes within the thematic map were then refned
to more clearly describe the fndings in each section. Finally, the
research team selected the most vivid and compelling examples
that related back to the research questions to include in this report.
3.5 Limitations
This work has demographic limitations. First, most participants
(10/14) most likely had early onset dementia, representing 9% of
dementia cases world-wide [1]. This relatively younger group of
participants may be overrepresented in our research due to the
contacts we recruited from and the recruitment material language:
“we’ll discuss how you use your mobile phone in daily life, any
challenges you have with mobile phone use, and your ideas for
ways to make technology use easier for you.” As one study showed,
technophilia - high enthusiasm for new technologies - was asso-
ciated with lower age of people with dementia [42]. The second
demographic limitation is the lack of people of color. With research
showing a higher prevalence of dementia in non-white people [2]
and only three participants identifying as Black or African Ameri-
can, these demographics were underrepresented in our sample. The
third demographic limitation is that participants resided primarily
in the United States. Findings from this study are therefore limited
in terms of geographic and cultural settings, which afect healthcare
access, socioeconomic status, and network device coverage.
This study was also limited by the one-hour semi-structured
interview method, which was chosen to minimize the time com-
mitment and amount of work required of participants. This choice
limited the scope of the data collected, as opposed to a longitudinal
study design which could provide several weeks for participants to
report on mobile phone use and future ideas. Further, we choose
to conduct interviews, not co-design sessions, as past work has
demonstrated that some people with dementia may experience dif-
fculty sharing sketches over video-conference calls [33]. Instead,
we relied on participants’ verbal explanation of their future ideas,
where a UX Designer on our team later illustrated these ideas in
Figma to include in this report.
4 FINDINGS
Through our interviews we learned of the reliance, desire and inter-
est participants with dementia had with their mobile phones and the
individually meaningful activities they used their mobile phones for.
Even with the heavy use of and reliance on their mobile phones, we
also uncovered challenges participants had with navigating to apps
and features; time pressure, high stress and fatigue afecting task
execution; and difculty re-learning task fows after updates and
upgrades. To address these challenges participants outlined their
Mobile Phone Use by People with Mild to Moderate Dementia: Uncovering Challenges and Identifying Opportunities
Figure 1: An Android phone home screen showing customiz-
able single click buttons for common multi-step tasks spe-
cifc to the individual. Customizations include personalized
images, labels, and sizing.
ideal interactions with mobile phones, providing opportunities for
the design of more accessible mobile phone interactions for people
with dementia. Below we elaborate on each of these fndings.
4.1 Mobile Phone Use
All participants described their reliance on their mobile phone,
emphasizing how they “can’t live without this phone” [Malcolm].
Further, mobile phones enhanced participants’ quality of life, as
they described them as “their best and most famous friend” [Ken-
nith] and their “brain” [Thomas, Tristen, Preston]. Most participants
described their reliance on the mobile phone to support “memory
issues” [Thomas] they experienced afecting their executive func-
tioning skills, leading Kennith to refer to his mobile phone as “the
way I cope with my daily existence.” Their reliance on the phone for
scheduling and reminders of everyday activities resulted in several
participants sharing the adage: “if it’s [an activity or event] not on
the phone, it’s not going to happen” [Preston]. Other participants
used their phones as a security measure for moments when they
struggled to remember where they were. As Hall described, the
phone is “something that I have to have with me all the time if I
ASSETS ’22, October 23–26, 2022, Athens, Greece
go someplace. That way, [Hall’s wife] can keep track of me and/or
call me, or I can call them. So, it’s a safety catch for me.”
4.1.1 Overall Mobile Phone Use. Participants described their use of
their mobile phones for diverse and individually meaningful activi-
ties. They described their use of apps for social media, navigation,
online shopping, mobile banking, reading the news, listening to
music, games, entertainment, productivity, and health management.
Please see the appendix for the full list of apps participants reported
using. This list does not represent all uses of the mobile phone by
participants, only those that participants were able to recall during
the interview. Participants’ specifc uses of their mobile phones
for purposeful work and health management were particularly
interesting and unexpected. Therefore, we describe them further
below.
4.1.2 Mobile Phones Facilitating Purposeful Work and Connection.
Many participants described the difculty they had after receiving a
dementia diagnosis and subsequent retirement. Malcolm described
his experience as feeling like there “was such a void in my life. . .
I have to be busy. I don’t sit around much because I feel if I sit
around, I’m not productive and I like to be productive.” Participants
described ways that their “phone is the key factor” in “keeping
myself pretty busy” [Tina].
All participants described the importance of their calendar to
support general productivity through connecting with others and
keeping track of daily tasks. A few participants described produc-
tivity in relation to helping others through advocacy work. As
Miranda described “I do a lot of one-on-one support things with
other people with dementia. So quite often I do video calls on my
phone with someone who is in need.”
Still others worked on more social and creative activities, which
they considered to be purposeful part-time jobs. For example,
Thomas works part-time as a freelance court reporter. Sylvia “use[s]
my phone a lot on Facebook live. . . I show people how to make
natural herbal lip balm, and I tape the whole thing.” Malcolm started
to co-host a podcast, which he posts online from his phone. He
explained how his podcast “is sort of my salvation. It’s the thing
that without it I think I would probably be very depressed because
it’s the one thing that I so look forward to” [Malcolm]. Staying
productive through these various methods was as much about en-
joyment for participants as it was about “keep[ing] my brain active”
through “something to prepare for... I do it all pro bono, but it’s that
little job that I have. It is purposeful work, and it keeps me going”
[Malcolm].
Participants used their mobile phones to support general pro-
ductivity, meaningful hobbies, and for more creative activities - all
providing purposeful work for participants.
4.1.3 Mobile Phones Making Health Management Easier. Partici-
pants also described using their phones for health management
activities. For example, some participants used their phone for med-
ication reminders, asking Siri to “remind me to take my medication
every day at 7:00. So, I use that feature and it’s extremely helpful
with making sure I take my medication” [Kim]. For Miranda these
medication reminders included specifc days and dates because “I
often struggle with what day we’re on, what the day and date is,”
which is important for specifc medications she takes.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Participants described how valuable it was to be able to use apps
on their phone to keep track of medical results [Josslyn, Sabella]
and to collect medical data [Thomas]. Thomas, who is living with
type 2 diabetes, demonstrated his use of his “glucose monitor, which
I use with my phone” where all he has to do is place the phone up to
his left triceps to check his blood sugar. This data is then uploaded
to the cloud so physicians can access this information, sparing
Thomas from “hav[ing] to stick my fnger all the time. They were
just able to share the data and see on a day-to-day, hour-to-hour
way all my blood sugar.”
The phone was also used for recalling medical information dur-
ing doctor’s visits. Elenora described how during doctor’s visits
they “always ask the same standard information about your medi-
cations and surgeries and all this stuf. And I can just pull it up on
my phone. . . and I can share it with the doctors” [Elenora]. This
included notes on her phone with “specifc links to my prescription
list document my supplements document” [Elenora].
Participants also described how valuable the phone is to call
emergency medical services. For instance, when Miranda’s “phone
is upended or shaken too severely, it will send out an SOS1 and
they will actually come on and say, ‘are you okay? Do you need
help?”’ In addition to this feature, Miranda has used her phone “to
Call 9112 to have an ambulance come for me.”
Participants used their mobile phones for medication reminders,
keeping track of medical results, collecting medical data, to recall
medical information during doctor’s appointments and to be able
to call 911 in emergency medical situations.
4.2 Challenges with Mobile Phone Use
Although participants described their reliance on their mobile
phones and used their phones for a variety of individually mean-
ingful activities, they also described the challenges they had with
navigation, task execution, and relearning changed interfaces after
updates and upgrades.
4.2.1 Dificulty Navigating to Apps and Features. Participants de-
scribed experiencing challenges with navigating to apps and fea-
tures on their mobile phone. Thomas described this as “maneuver-
ing through the phone” where “as it becomes more useful I think it
can become more challenging because there’s more stuf squeezed
in there and fnding it all and maneuvering through it all can be a
challenge.”
Participants often knew exactly what they wanted to fnd on
their mobile phone but had difculty “sift[ing] through the vast
amount of information. . . zoom[ing] down into it” to get to what
they were looking for [Thomas]. For example, several participants
described their difculty with “navigating the contact lists” in high
stress environments [Miranda]. Preston described how “A lot of
people with dementia would forget to scroll. They would think I
only can see what’s on that page. . . Somebody with dementia that
gets overwhelmed or fnally realizes they’re lost. All a sudden your
anxiety is so high. It’s like, ‘Where do I make a phone call? Where’s
the number?”’ We can understand this example through the lens of
afordances - an interaction element that tells us what we can do
1 SOS
is an internationally recognized signal of distress in radio code used especially
by ships calling for help.
2 911 is a phone number used in North America to contact emergency services.
Emma Dixon et al.
with it [75]. Afordances are binary - they either exist or they don’t
- and contextual, meaning if we cannot perceive or understand
an afordance because of our situation, it does not exist. In the
example Preston describes where high anxiety makes scrolling to
fnd a phone number in his address book challenging. For Preston,
there is no identifable indication of how to interact with the phone
in those moments, though Preston is able to understand how to do
this in other contexts when he is less anxious.
In other instances, participants described difculty navigation
to specifc applications and content on their mobile phones. For
example, Sabella described an instance when her daughter
“made a post about me on her Snapchat. . . it was a
beautiful post about living with dementia and how
she feels about me living with dementia. And so, I was
trying to go back to fnd it to save it and I just could
not understand the concept of doing that... I wanted it
so bad, but I just couldn’t fgure out how to do it [get
back to the Snapchat post]. So, I lost it which was the
most frustrating for me.”
Participants described difculty with navigating to apps and
features due to the amount of information available on their phone
and challenges with remembering where items were saved on their
phone.
4.2.2 Time Pressure, High Stress and Fatigue Impairing Task Execu-
tion. Participants experienced challenges executing the necessary
steps to complete tasks with precision on their mobile phones
though they understood the necessary steps to complete tasks (i.e.,
the Gulf of Execution [76]). This challenge was primarily described
in instances of time pressure or when participants experienced
moments of high stress and fatigue.
Participants described feeling time pressure for tasks, which
made them more difcult to execute with precision. For example,
when inputting calendar events on their mobile phone, participants
described experiencing “the pressure of the person that you’re doing
business with kind of standing there going ‘Why is this so hard for
you?”’ [Alecia] as well as the “people behind you” in line [Josslyn].
In some instances, this time pressure can lead to the event not
“get[ting] on the calendar and I’m thinking, I thought I did it. And I
think I’m missing that last click” [Josslyn]. In other instances, this
time pressure led participants to input the exact month and time
of events incorrectly, therefore scheduling events “in the wrong
month” and consequently “show[ing] up at wrong appointments
at the wrong time because I’ve screwed them up” [Alecia].
Participants also described challenges with executing tasks in
moments of high stress and fatigue. For example, Elenora, who
lives in a large city, described a time when after a long day of
errands she “was very cognitively exhausted and I was waiting for
a specifc bus to come home”. This was the only way she knew
how to get home. But, the bus she was waiting for was delayed
for three hours, leaving her sitting at the bus stop into the night.
Although Elenora was able to check on her phone to see the bus
continued to be delayed, she “didn’t have the cognitive wherewithal
to fgure anything else out,” meaning use her phone to search for
other alternative ways to get home. Similarly, after a stressful day
of work and navigating transportation in a large, unfamiliar city,
Thomas described how on his phone he was “looking at this map
Mobile Phone Use by People with Mild to Moderate Dementia: Uncovering Challenges and Identifying Opportunities
of these train stations and it’s just not making any sense. And all
I could think of was the train somehow jumped the tracks into a
diferent line and they’d taken me to the wrong place.” He explained
this as “a situation where our brains aren’t working right,” making
it difcult to execute routine tasks, such as navigation, on their
mobile phones. This created high stress situations for participants
who were aware of the increased risk of the harm of being lost for
people living with dementia [86, 104].
In these instances of high stress and fatigue, participants were
either not willing to ask for help from other people or had dif-
culty communicating with others. Elenora described how in these
moments “I’m certainly not going to ask a stranger on the street
because when you get in a fog like that, as friendly as I tend to
be and willing to ask for help, when I’m that deep in the fog, I
really enclose in on myself.” Thomas was willing to ask for help,
but when he “asked the conductor, he was very gruf, and made me
feel stupid. . . and look[ed] at me like I was crazy” but his directions
just were “not computing to me.” In this instance, Thomas called his
husband who drove to pick him up, describing how “people with
challenges need the ability to reach out to a human even more than”
other people. Thomas is referring to reaching out to signifcant
others who understand his situation and can provide sympathetic
assistance rather than people, such as the conductor, who do not
understand his situation and therefore cannot provide the necessary
assistance.
Participants also described high stress interactions with voice-
based systems which impaired them from executing tasks on their
mobile phone. For example, Malcolm describes using his bank’s
interactive voice response system regularly to pay his bills. How-
ever, these systems do not always understand Malcolm where the
system will say “I didn’t hear you.” This could be due to changes in
speech patterns that people with dementia experience [71, 82] such
as slowed speech, a developed stutter, and greater pauses between
words. Preston posits this may be in part due to participants asking
“question[s] and say[ing] the wrong word or the wrong adjective
sometimes by mistake” [Preston]. In these instances, Malcolm then
responds by “yelling and screaming because they don’t hear my
voice,” which led to “a very high level of frustration. . . and I end up
hanging up the phone because it’s horrible.” This example shows
that execution of tasks on mobile phones may be impaired due to
voice assistants not understanding their verbal commands.
Participants described challenges with executing tasks on their
mobile phones when under time pressure and in moments of high
stress and fatigue, such as after a long day trying to navigate home
or when interacting with mobile phone voice-based systems.
4.2.3 Dificulty Re-learning Task Flows Afer Updates and Upgrades.
Similar to past work [50], participants expressed the challenge it
was to re-learn tasks fows after updates and upgrades because “this
is the new path I have to go through to do the things that I’m used
to doing. I’m not to the point where I can’t do it, but it defnitely
can give me pause when a program changes or updates” [Thomas].
For Tristen, upgrades were difcult because “nobody gives you
instructions that these are the changes that have taken place. It
just happens on the phone.” He explained that “sometimes I get
very anxious and frustrated, when changes [updates] take place.”
Notably, participants did not describe barriers with updates and
ASSETS ’22, October 23–26, 2022, Athens, Greece
upgrades in relation to changes made to software and/or interfaces
in order to correct accessibility barriers, as described in the Web
Content Accessibility Guideline [49]. Instead these barriers were
more so due to the change itself, rather than the substance of the
change, as well as the fact that these changes are not explicitly
called out by the system.
Several participants expressed fear over upgrading their phones
to a newer version. Tina described how she has “always kind of
updated or upgraded as needed. But now with my Alzheimer’s, I
think that if I got another phone that had more fancy things that
might make it more difcult. . . I don’t try to start all over again and
get a whole brand-new phone” because she was “afraid I’m gonna
mess up and then it’ll screw up my brain.” Malcolm shared this fear:
“it’s a fve-year-old [phone]. I’m afraid to get a new phone because
I’m afraid I’m not going to know how to use all of this stuf. So, I
keep the phone. It works fne.”
Participants also described not wanting to switch the operating
system they used because of the difculty they experienced with
learning new systems. Josslyn explained that when she retired “I
just kept [previous operating system] because I knew how to use it
and I didn’t want to have to relearn everything.” One participant,
Preston, intentionally got a new phone with a larger screen and
memory “with the hopes I probably never have to get a new phone
again” as he plans for this phone to last him the rest of his phone
use.
Updated, upgraded or switched operating systems of their mobile
phones complicated discovery for participants, forcing them to re-
learn mobile phone skills, which led to frustration and fear for some
participants.
4.3 Participants’ Ideas to Address Challenges
with Mobile Phone Use
In this section we describe participants’ ideas to address challenges
with mobile phone use, including customizable user interfaces,
activity-based customization, proactive technology assistance, and
extended use of voice-based interactions. These design opportuni-
ties are based on feedback provided during the speculative design
portion of the interview, where participants frst articulated their
ideal interactions with mobile phones and then saw the technology
probe demonstration.
4.3.1 Customizable User Interface on Mobile Phones. To address
challenges with navigating their mobile phones, participants
wanted to customize the size of apps and the icons so that they
could more easily recognize icons. For example, Thomas wanted
“to sort of mold the device to what you need.” Molding the device
included making app icons that are “recognizable. . . easy for me
to identify, even when I’m at my worst with my brain fog with
my dementia” [Miranda]. This also included “hide[ing] all those
other things that we don’t use or don’t want” [Miranda] and plac-
ing only the most used “app[s] on the home screen” [Thomas].
Malcom describes how he needs “less options. I don’t need 90% of
the things [apps]. . . Because I think that’s where I have problems,
when there’s a lot of options. Limit the options so I’m not searching
for things.”
ASSETS ’22, October 23–26, 2022, Athens, Greece Emma Dixon et al.

Relating to the challenges with re-learning task fows after up- action on their behalf without them asking for this assistance. For
dates and upgrades, participants wanted a pre-specifed easy ver- instance, Sylvia described her desire for a single button, that when
sion of their phone rather than having to make user interface cus- pushed, enabled a personal assistant that could help her during
tomizations themselves. Josslyn proposed the idea of being able doctor’s visits by “advocat[ing] for me” by “asking [questions] for
to “choose an easy format” where modifcations were “set for you” me” if she forgets to ask them. Sylvia also wanted the personal as-
making the phone similar to “these phones advertised for seniors sistant to tape her conversations with her doctor, because “if I can
with the simplicity things.” Though when asked if she had consid- record it then I can play it back,” supporting her memory changes.
ered buying and using one of these phones, Josslyn explained how In later interviews, we used Sylvia’s idea to probe other partici-
she would prefer “just a simplifed version of my phone” because pants’ perception of the line between helpful and disconcerting
this is what she is familiar with. with this kind of assistance from mobile phones. Tina described her
perspective: “I don’t think it’s creepy because you’re making the
4.3.2 Activity-based Customization on Mobile Phones. As another
choice to tell [the system] to start flming [recording]. . . It’s not
way to address challenges with navigating their mobile phones,
like she [the system] knows you just got into the doctor’s ofce and
participants described their need to bundle common activities -
now automatically the microphone’s going to show up.” However,
meaning to create specifc categories of activities - and store these
Sabella surfaced a legitimate concern: “the thing about the button
bundles in distinct, easily identifable places. For example, Josslyn
[to activate the personal assistant] is you have to remember to
described how she “had to have it [her calendar] compartmen-
tell the button to do it [provide assistance].” Because memory can
talized” where she “save[s] it [her phone calendar] for doctor’s
be increasingly challenging for people with dementia, Tina later
appointments, or hair appointments or something that I’m leaving
concluded: “I think[s] there’s going to be a time where we really
the house for. Not just like to remember to do the laundry. . . routine
need all those crazy voices [voice assistants] kind of helping us”
things are put on the paper calendar.” Josslyn made this modifca-
without being prompted.
tion to her calendar use after her diagnosis as she described “when
In addition to wanting more proactive assistance in doctor’s
I worked I had all that” on her phone calendar, “but once I had to
visits, participants also described how they would like their phone
simplify my life that’s how I simplifed it - paper calendar, phone
to track their activities on their mobile phone to play a part in
calendar.”
sustaining their relationships. Participants wanted their phone to
This idea of bundling common activities and storing them in
help them respond to others and initiate social interactions with
distinct places was applied to several envisioned futures for mobile
others - tasks that were becoming increasingly difcult. For example,
phone use. First, after seeing the technology probe could make a
Alecia wanted her phone to be “snooping in your text” to then
phone call using a single click button to a pre-specifed individual (a
remind you “‘hey, you haven’t checked in on or checked up on so
common feature on devices geared towards older adults and people
and so since such and such.”’, similar to existing Microsoft Outlook
with dementia [72, 90]), Tina described her desire for a “my tribe”
email reminders [69]. Though Alecia describes how it would be
button to “push the button that would have the name and the phone
important for the system to be “synced to your text” rather than
number of those seven people that are in my tribe so that I don’t
social media in order to only be tracking those more active social
have to look it up” in her contacts. Both iOS [5] and Android smart
connections.
phones [38] provide this service through groups.
One participant, Preston, proposed another instance when track-
In another example of activity-based customization by bundling
ing mobile phone use to provide proactive technology assistance
common activities, one participant, Sylvia, described her desire to
may be useful: in identifying unused but potentially useful features
present voice-assistants as support for specifc aspects of life. For
(e.g., a stylus). For example, Preston proposed the phone could iden-
example, Sylvia wanted her phone to “have fve buttons” each with
tify: “‘Hey, he really hasn’t used that stylus pen. Maybe we should
a diferent “personal assistant. . . one is for doctor’s appointments.
send him another opportunity.”’ and then provide “a tutorial” of
One is to help you with the grocery store. One is to [remind you
how to use that feature. When this idea was presented to partici-
to] take medication.” Practically this would be one voice-assistant
pants in later interviews it was well received as a potential solution
(e.g., Alexa, Google Assistant, Siri) but presented as diferent types
to facilitate learning new task fows after updates or upgrades.
of assistants to help with diferent bundled aspects of their life. We
Participants also wanted their phone to provide more proactive
found this concept compelling, and so proposed it in later interviews
assistance by learning and automating routine uses of their phone,
to get other participants’ feedback. This idea was well received,
which could further support them in executing tasks on their mo-
with several participants proposing diferent personal assistance
bile phones in stressful situations or when they experience fatigue.
(e.g., life coach assistant to support chores [Alecia, Josslyn], and
For example, participants described using specifc intervals of re-
technical support [Kennith]).
minders for every calendar event: “it’s always two days before the
Participants described their desire to have activity-based cus-
event, one day before the event, one hour before the event, and
tomization by bundling common activities and storing these bun-
maybe 15 or 20 minutes before the event, and then a fve-minute
dles in distinct, easily identifable places to address challenges with
reminder” [Preston]. Preston wanted his phone to be “smart enough
navigating their mobile phones.
to realize, I’m doing the same reminders every time. There should
4.3.3 Proactive Technology Assistance on Mobile Phones. To assist be an option where I can just say, ‘use your common reminders’.”
in moments of pressure, stress and fatigue that can hinder task Sabella wanted to completely automate the process of inputting
execution on mobile phones, participants described their desire for calendar events so that “when you call to say ‘Sabella, I’m going
more proactive technology, where their mobile phones take some to set us up a Zoom for so and so’ and it automatically just kind of
Mobile Phone Use by People with Mild to Moderate Dementia: Uncovering Challenges and Identifying Opportunities
Figure 2: Mockup of adding a calendar event using voice-based interactions with Google Assistant on a mobile phone, including
proactive technology assistance with reminder intervals.
records it in some little strange way and it sets it on the calendar
right then. . . Not for me to do it [input a calendar invite] but its
own self.”
Proactive assistance could also take the form of voice-assistants
providing more structured prompts in a conversational style to help
people execute tasks on their mobile phone. For example, Elenora
wanted to be able to
“tell my phone while I’m out: ‘I’m confused’ or ‘I’m
not sure where to go.’ Then it could start asking me
questions. Not, ‘Do you want to do this?’ But literally
ask me questions to help me fgure out what it is I’m
trying to even do. And, I could just say ‘I’m trying
to get home’ and it could say, ‘Well, there’s all kinds
of ways. What do you prefer? Want to go home the
fastest way? The easiest way? The cheapest way?’
And then I can just pick one.”
This system would allow Elenora to “interact with it in a more
natural language. . . almost have a conversation with the system.”
This example refects the challenges of balancing 100% open-ended
conversation with structured prompting (as previous work with
older adults has highlighted [17]). In our fndings, participants
describe their need for structured prompting but in a more conver-
sational style and for voice-assistants to understand less structured
responses.
Participants described their desire for more proactive technology
assistance through unprompted assistance, more in-depth tracking
of their mobile phone use to support relearning task fows after up-
dates and upgrades, as well as automating routine tasks to provide
ASSETS ’22, October 23–26, 2022, Athens, Greece
support in moments of high stress or fatigue that impair mobile
phone use.
4.3.4 Extending Voice-based Interactions with Mobile Phones. Par-
ticipants described their desire to use voice-based interaction to
address challenges with navigating their phone. For instance, Tina
wanted to just be able to speak to her preferred social media app
to have it search for pictures of her and her daughter on vacation
rather than having to scroll through her past posts to fnd the album.
Many participants described their desire to “click on the calendar
[app] and use your voice and say, ‘doctor’s appointment, this date,
this time”’ and have it add an event [Josslyn]. Figure 2 provides an
example of adding a calendar event using voice-based interactions
with Google Assistant on the mobile phone, which also takes into
consideration more proactive technology assistance. Participants
also wanted to search for content across apps [Elenora, Josslyn,
Alecia, Sabella, Kim, Preston] (e.g., saved recipes on diferent so-
cial media apps [Kim], social connections with people [Thomas]).
Preston wanted to “name your phone, which would then turn it on
like Alexa” when you spoke its name.
Several participants described needing an avatar in combination
with a voice-based virtual assistant on their mobile phone to help
with attention, which would ultimately help them to execute tasks
on their mobile phones. For instance, Malcolm described how “I
don’t like people talking to me over the phone, it’s difcult for me.
This [points at the video conference call camera] is easier.” Preston
describes this as “a dual connection because you’re connecting with
your eyes and your ears, whereas if my screen is gone, we’ve only
got one connection and that’s just listening.” Preston believes this
ASSETS ’22, October 23–26, 2022, Athens, Greece
dual connection would help to “draw you in and you’re going to pay
more attention than a voice that’s just randomly talking.” Because
of this need for a dual connection, participants described the need
for voice-based interactions on their phone to have an “artifcial
person” or “an android person” [Preston]. Sylvia described wanting
to personalize her virtual assistant by having “an animated person”
where “I can pick his voice. I can pick the culture. Like, it’ll probably
be a person of color.” She explained this concept by relating to “the
Wii game you can describe your person. . . You can pick the hair,
you pick the glasses, the voice, the outft. I would like to pick my
virtual assistant” [Sylvia].
Participants also described clear lines for acceptable use of
avatars with virtual assistants for people with dementia, relating
back to the uncanny valley efect [70]. Preston notes that the system
needs to make it clear that “it’s not a real person” by having “in the
background saying ‘You’re speaking to Robby the Robot’ something
that is displayed. So that they know this person isn’t real because
for some people they could get confused.” One participant, Alecia,
was open to support from avatars in combination with virtual as-
sistants as long as it remained two dimensional where any physical
or three-dimensional robots were considered “scary” and “going
over the top.”
Even with the enthusiasm of many participants about extending
voice-based interactions with their mobile phone, several partic-
ipants also noted the importance of providing human assistance
in high stress or time sensitive instances where voice assistants
were not understanding them, which made executing tasks on their
mobile phones difcult. Thomas stated: “there are times when we
just need, particularly people with cognitive challenges, need the
ability to reach out and touch another person. Especially if we’re
in a situation where our brains aren’t working right.” Similarly, Mi-
randa describes how she “prefers person-centered help” especially
“when you’re talking to an artifcial intelligence [about] anything
more complex.” Though she elaborates “that doesn’t mean that I
don’t think things like [voice assistants] aren’t good. But I think
within those [having] the ability to easily access an actual person
is also important” [Miranda].
Participants described their desire to extend the capabilities of
voice-based interactions by: facilitating voice-based interactions
within and across apps to better support navigation, as well as
to support task execution by using avatars in combination with
voice assistants to support attention and further personalization;
and providing access to human assistance in instances when voice
assistants did not understand them.
5 DISCUSSION
Through interviews with fourteen people with mild to moderate
dementia, we address each of our three research questions. First, we
investigated how people with mild to moderate dementia use their
mobile phones, fnding they use a range of apps for: navigation,
mobile shopping, online banking, games, social media, news, en-
tertainment, communication, connection, productivity, and health
management. These fndings take a frst step towards flling an
empirical gap in the literature by detailing some ways people with
mild to moderate dementia use their mobile phones. Because past
Emma Dixon et al.
research in the development of mobile apps for people with demen-
tia have primarily focused on designing apps to improve memory
[40] or for GPS tracking to assist in navigation [46, 61, 100, 101],
we also intentionally highlight two common activities on mobile
phones, productivity and health management, as promising areas
for future development. Additionally, participants described some
ways their use of apps changed as they progressed with dementia.
For example, they experienced a new reliance on their calendar
apps to dictate their daily activities. They also described having to
compartmentalize calendars based on diferent aspects of life, as
well as the need for set intervals of reminders for each calendar
event. As our fndings report participants’ refections on how their
mobile phone use has changed, further research is needed to ob-
serve the longitudinal use of mobile phone applications as people
with dementia progress with the condition.
Second, we investigated the pain points and barriers to com-
pleting tasks on mobile phones that people with mild to moderate
dementia experience. We uncovered challenges with: 1) navigating
to apps and features; 2) task execution in moments of high stress, fa-
tigue, and time pressure, and 3) re-learning task fows after updates
and upgrades. These fndings expand on past work on barriers tech-
savvy people with mild to moderate dementia have with typing on
their mobile phone due to small key size and difculty identifying
the notifcation ringtones and sounds [31] by providing a more
thorough examination of the barriers people with mild to moderate
dementia experience with mobile phone use.
Addressing our third research question concerning opportuni-
ties for technology to support people with dementia when they
encounter challenges with their mobile phones, we uncover four
design opportunities based on participants envisioned future inter-
actions with mobile phones: customizing for accessibility, activity-
based customization, proactive technology assistance, and extended
modalities for voice-based interactions. These fndings demonstrate
the considerable creativity of people with dementia in generating
ideas for future technologies.
Many of these ideas incorporate AI and automation to support
more accessible interactions with mobile phones for people with
dementia. This provides a diferent perspective from past work in
AI and dementia, which has primarily focused on ways AI could
be used to detect and monitor the progress of dementia [8, 41, 87,
92, 96], in smart home environments to support care partners in
monitoring the activities of individuals with dementia [3, 24, 26, 63,
80], or to support therapy [13, 20, 95]. Therefore, our work opens
up new opportunities for the design of future AI systems to support
the abilities of people with dementia, as in ability-based design
[105].
In the remainder of the discussion, we describe design oppor-
tunities to support individuals with progressive disabilities and
tensions with automation for people with dementia.
5.1 Design Opportunities to Support Individuals
with Progressive Disabilities
One contribution of this work is taking a frst step towards under-
standing how to design for access needs for a group of neurodiverse
users that experience progressive changes in ability, going beyond
the traditional binary representation of disability. For instance, our
Mobile Phone Use by People with Mild to Moderate Dementia: Uncovering Challenges and Identifying Opportunities
fndings demonstrate the importance of adaptive user interfaces to
minimize navigation of mobile phone use, which can become in-
creasingly difcult for people with dementia as they progress with
the condition. Our fndings also demonstrate the importance of
providing increasingly proactive technological support for people
with dementia as they progress with the condition. In the following
section we discuss each of these directions for future design.
5.1.1 Adaptive User Interfaces for Progressive Changes in Abil-
ity. Participants described their desire for adaptive interfaces that
could simplify navigating their phones through customizable home
screens, and adjustable app sizes and personalizable icons (aligning
with guidance from the WCAG Cognitive Accessibility Task Force)
[99]. But, they also described challenges with learning new interac-
tion patterns, which are inevitable with adaptive interfaces. As one
way to navigate this tension, designers could ensure that users are
made aware of any changes with updates and upgrades and then
provide training on new task fows.
Another potential way to navigate this tension is to use temporal
dimensions for adaptive user-interfaces for people with dementia,
building of of previous work on ephemeral adaptations - where
only the most used menu items are displayed abruptly and then all
other menu items gradually fade in [34]. In the case of mobile phone
accessibility for people with dementia, it may be necessary for the
less used apps to gradually fade out and no longer be displayed due
to changes in visual ability with the progression of the condition,
including selecting an object from a visually busy environment
[11, 91]. To be clear, we are not advocating for systems that strip
away functions of a device to the bare minimum due to the inability
of people with dementia to understand complex functions of devices
(as in past work [4, 18, 43, 55, 62, 72, 88]). Rather, we are proposing
systems which display the most used features in a way that is easy
for people with mild to moderate dementia to navigate to, while
still providing access to less used features if/when they want them.
Still another potential future direction is the design of adaptable
systems in combination with activity-centric thinking [7], which
shifts away from traditional application-centric computing and
towards human goal-oriented activities, cutting across systems
boundaries [7]. Participants wanted their user interfaces to bundle
common activities and store these bundles in distinct, easily iden-
tifable places. For instance, participants bundled their calendars
by type of activity (e.g., in-home activities vs. out of the house
activities). Adapting user interfaces to refect bundles of common
activities may be one way to make systems more accessible to
people with dementia as they experience progressive changes in
ability.
These are just two examples of potential areas to explore in
future work on adaptive user interfaces to support people with
dementia. There is room for much further exploration concerning
adaptive user-interfaces to support more accessible interactions
with technology for people with progressive changes in ability.
5.1.2 Proactive Technological Support. One way to provide more
proactive technological support is through context-aware comput-
ing. Participants described the need for their phone to recognize
their location as well as their conversational partner (as in the
TalkAbout System for people with Aphasia [51]) to assist them
ASSETS ’22, October 23–26, 2022, Athens, Greece
in executing tasks in moments of high stress and fatigue. This as-
sistance could be provided through context-aware prompting for
tasks, as in [22, 23] which provided prompts for users with cogni-
tive impairments to complete tasks. Though, as participants in our
study mentioned, there may come a time with the progression of
the condition where prompting will not be enough and the system
will need to provide more active assistance. For example, assistance
could be provided automatically when the system recognized the
context the person with dementia was in (e.g., in a doctor’s appoint-
ment speaking with their doctor). One participant even wanted
their mobile phones to listen into their phone conversations to
automatically add calendar events as they were confrmed during
phone calls, as in past work [64, 65]. Future systems should consider
utilizing context-aware computing to provide more active support
to people with dementia.
Another way proactive technological support could be provided
is through tracking technology use. For example, several partici-
pants described their desire for their phone to monitor their text and
emails to prompt them to respond to and initiate social interactions
with others. Participants also wanted their mobile phones to track
what features they used and propose new or previously unused
features that may be helpful to them based on their historic phone
usage. Current systems provide nudges and notifcations for new
features once rather than providing additional reminders as these
additional reminders are assumed to be unhelpful and annoying
to users [81]. Our fndings suggest that for people with dementia
they may need to be reminded about features more than once, due
to the progressive nature of the condition and their changes in
memory overtime. Future systems could integrate more proactive
technological support (e.g., context-aware proactive smart-speakers
[83, 102, 103]) by highlighting potentially useful features and provid-
ing regular reminders of these features if they begin to go unused.
Although these design directions may provide necessary sup-
port to prolong mobile phone use for people with dementia, they
also introduce privacy concerns due to the level of data collection
necessary to provide this support. As past work has outlined [67],
such tracking could be used as a mediator of coercive control and
abuse. To manage this tension between the need for more proactive
technology support and privacy concerns, we urge researchers,
designers and developers to keep privacy considerations central to
their work.
5.2 Tensions in Designing Technologies for use
by People with Dementia
Participants described extensive future applications for automation,
surfacing tensions with autonomy and who would be in control of
initiating support, the person or the system. For instance, partici-
pants in our study described ways that automation could be used
to improve their task execution on their mobile phones, centering
their own role in completing the task (e.g., having a button where
they could prompt voice assistants). This may be one solution to
the concern of some people with dementia in past work towards
receiving support from AI to assist with managing their daily life
out of concern for the loss of autonomy [32]. However, participants
also described how with the progression of the condition, they will
need systems to eventually act autonomously to perform tasks on
ASSETS ’22, October 23–26, 2022, Athens, Greece
their behalf (e.g., start audio recording in doctor’s visits). Partici-
pants described these progressive changes in abilities due to the
condition necessitates less autonomy with technology to support
them to continue to be active in everyday activities (e.g., attending
doctor’s visits independently). However, with more proactive tech-
nological support and less autonomy, this could pose additional
risks for abuse facilitated via the proactive systems (as described in
[67]). With evidence of elder abuse through automation in the form
of smart home technologies [15], we join in the call of past work
[94] to design automated systems for use by people with dementia
rather than on people with dementia.
Participants also described instances where their reliance on
technology negatively afected them. For example, moments of in-
tense brain fog can render technology assistance useless (e.g., when
navigating), requiring them to reach out to another person for sup-
port. These fndings suggest future systems should be designed to
keep humans in-the-loop (as proposed in [89]) around users with
dementia to provide support when necessary. Importantly, partic-
ipants needed to be connected to sympathetic human assistance.
Therefore, future systems could sense frustration or negative emo-
tion from voice interactions, and then automatically connect the
user to a friend or loved one to provide human assistance.
Still another tension which emerged from our work concerns
the use of a visual representation for voice assistance. Although
some participants noted a visual representation may help with
attention when interacting with voice assistants, they also noted
how such visuals may be confusing or disturbing to some people
with dementia who may not be able to discern that the avatar is
not a real person. Future work is needed to understand how people
with dementia perceive diferent types of visual representations of
voice assistants (e.g., disembodied agents, artifcial embodied agent,
and photorealistic embodied agents [10]) and if their perceptions of
visual representations change with the progression of the condition.
We recognize that this is not a full investigation of all possible
tensions that may arise with the design of technologies for use
by people with dementia (e.g., trust, explainability). Further work
is needed to investigate these tensions and ways for designers,
developers, and researchers to better navigate these tensions.
6 CONCLUSION
This work details ways people with mild to moderate dementia use
their mobile phones surfaced through an analysis of interviews with
fourteen people with mild to moderate dementia. Findings from
this study showed three major challenges with mobile phone use:
1) navigating to apps and features; 2) task execution in moments of
time pressure, high stress and fatigue, and 3) re-learning task fows
after updates and upgrades. To address these challenges participants
described their ideal interactions with their mobile phones, which
included customizing for accessibility, activity-based customization,
proactive technology assistance, and extended modalities for voice-
based interactions. This paper contributes to the literature by 1)
providing an empirical account of how fourteen people with mild to
moderate dementia use mobile phones and the challenges they face
with mobile phone use; 2) uncovering design opportunities to help
achieve more accessible mobile phone use for people with dementia;
and 3) providing new directions for the design of future systems to
Emma Dixon et al.
augment and enhance the abilities of people with dementia. With
the pervasive use of mobile phones in our society, these fndings will
help researchers and creators of technology design environments
that assure societal inclusion [47] for people with mild to moderate
dementia.
ACKNOWLEDGMENTS
Thank you to participants, as well as Shaun Kane, Hernisa Kacorri,
Jonathan Lazar, Gregg Vanderheiden, Matthew Goupell and the
anonymous reviewers who provided feedback on versions of this
paper. This work was supported, in part, by grant 90REGE0008,
U.S. Admin. for Community Living, NIDILRR, Dept. of Health &
Human Services, NSF Grant IIS-2045679, and the National Science
Foundation Graduate Research Fellowship Program under Grant
No. DGE 1840340. Opinions expressed do not necessarily represent
ofcial policy of the Federal government.
REFERENCES
[1] Alzheimer’s Association. Younger/Early-Onset Alzheimer’s. Alzheimer’s Dis-
ease and Dementia. Retrieved March 16, 2020 from https://alz.org/alzheimers-
dementia/what-is-alzheimers/younger-early-onset
[2] Alzheimer’s Association. 2021. 2021 Alzheimer’s Disease Facts and Figures.
Alzheimers Dement, Chicago, IL.
[3] Mohsen Amiribesheli and Abdelhamid Bouchachia. 2015. Smart Homes De-
sign for People with Dementia. In 2015 International Conference on Intelligent
Environments, 156–159. https://doi.org/10.1109/IE.2015.33
[4] Claire Ancient, Alice Good, Clare Wilson, and Tineke Fitch. 2013. Can Ubiqui-
tous Devices Utilising Reminiscence Therapy Be Used to Promote Well-Being
in Dementia Patients? An Exploratory Study. In Universal Access in Human-
Computer Interaction. Applications and Services for Quality of Life (Lecture Notes
in Computer Science), 426–435. https://doi.org/10.1007/978-3-642-39194-1_50
[5] Apple Inc. 2022. Create and manage groups of contacts on iCloud.com. Apple
Support. Retrieved June 24, 2022 from https://support.apple.com/guide/icloud/
create-and-manage-groups-mmfba73c71/icloud
[6] Norm Archer, Karim Keshavjee, Catherine Demers, and Ryan Lee. 2014. Online
self-management interventions for chronically ill patients: Cognitive impair-
ment and technology issues. International Journal of Medical Informatics 83, 4:
264–272. https://doi.org/10.1016/j.ijmedinf.2014.01.005
[7] Jakob E. Bardram, Steven Jeuris, Paolo Tell, Steven Houben, and Stephen Voida.
2019. Activity-centric computing systems. Communications of the ACM 62, 8:
72–81. https://doi.org/10.1145/3325901
[8] Flavio Bertini, Davide Allevi, Gianluca Lutero, Danilo Montesi, and Laura Calzà.
2021. Automatic Speech Classifer for Mild Cognitive Impairment and Early
Dementia. ACM Transactions on Computing for Healthcare 3, 1: 8:1-8:11. https:
//doi.org/10.1145/3469089
[9] Inga-Lill Boman, Stefan Lundberg, Sofa Starkhammar, and Louise Nygård. 2014.
Exploring the usability of a videophone mock-up for persons with dementia and
their signifcant others. BMC Geriatrics 14, 1: 49. https://doi.org/10.1186/1471-
2318-14-49
[10] Michael Bonfert, Nima Zargham, Florian Saade, Robert Porzel, and Rainer
Malaka. 2021. An Evaluation of Visual Embodiment for Voice Assistants on
Smart Displays. In CUI 2021 - 3rd Conference on Conversational User Interfaces,
1–11. https://doi.org/10.1145/3469595.3469611
[11] François-Xavier Borruat. 2013. Posterior Cortical Atrophy: Review of the Recent
Literature. Current Neurology and Neuroscience Reports 13, 12: 406. https://doi.
org/10.1007/s11910-013-0406-8
[12] Ann L Bossen, Heejung Kim, Kristine N Williams, Andreanna E Steinhof, and
Molly Strieker. 2015. Emerging roles for telemedicine and smart technologies
in dementia care. Smart homecare technology and telehealth 3: 49–57. https:
//doi.org/10.2147/SHTT.S59500
[13] Eleni Boumpa, Ioanna Charalampou, Anargyros Gkogkidis, and Athanasios
Kakarountas. 2017. Home Assistive System for Dementia. In Proceedings of the
21st Pan-Hellenic Conference on Informatics (PCI 2017), 1–6. https://doi.org/10.
1145/3139367.3139435
[14] Aikaterini Bourazeri and Simone Stumpf. 2018. Co-designing smart home tech-
nology with people with dementia or Parkinson’s disease. In Proceedings of the
10th Nordic Conference on Human-Computer Interaction - NordiCHI ’18, 609–621.
https://doi.org/10.1145/3240167.3240197
[15] Bonnie Brandl, Carmel Bitondo Dyer, Candace J. Heisler, Joanne Marlatt Otto,
Lori A. Stiegel, and Randolph W. Thomas. 2006. Elder Abuse Detection and
Intervention: A Collaborative Approach. Springer Publishing Company.
Mobile Phone Use by People with Mild to Moderate Dementia: Uncovering Challenges and Identifying Opportunities
[16] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychol-
ogy. Qualitative Research in Psychology 3, 2: 77–101. https://doi.org/10.1191/
1478088706qp063oa
[17] Robin Brewer, Raymundo Cornejo Garcia, Tedmond Schwaba, Darren Gergle,
and Anne Marie Piper. 2016. Exploring Traditional Phones as an E-Mail Interface
for Older Adults. ACM Transactions on Accessible Computing 8, 2: 6:1-6:20. https:
//doi.org/10.1145/2839303
[18] Suzanne Mary Cahill, Emer Begley, Jon Paul Faulkner, and I. Hagen. 2007. “It
gives me a sense of independence” – Findings from Ireland on the use and
usefulness of assistive technology for people with dementia. Technology and
Disability 19, 2–3: 133–142. https://doi.org/10.3233/TAD-2007-192-310
[19] Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the
2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), 981–992.
https://doi.org/10.1145/2858036.2858498
[20] Mariona Carós, Maite Garolera, Petia Radeva, and Xavier Giro-i-Nieto. 2020.
Automatic Reminiscence Therapy for Dementia. In Proceedings of the 2020
International Conference on Multimedia Retrieval, 383–387. Retrieved March 4,
2022 from https://doi.org/10.1145/3372278.3391927
[21] Filippo Cavallo, Michela Aquilano, and Marco Arvati. 2015. An Ambient Assisted
Living Approach in Designing Domiciliary Services Combined With Innovative
Technologies for Patients With Alzheimer’s Disease: A Case Study. American
Journal of Alzheimer’s Disease & Other Dementias® 30, 1: 69–77. https://doi.org/
10.1177/1533317514539724
[22] Yao-Jen Chang, Wan Chih Chang, and Tsen-Yung Wang. 2009. Context-aware
prompting to transition autonomously through vocational tasks for individu-
als with cognitive impairments. In Proceedings of the 11th international ACM
SIGACCESS conference on Computers and accessibility (Assets ’09), 19–26. https:
//doi.org/10.1145/1639642.1639648
[23] Yao-Jen Chang, Shih-Kai Tsai, and Tsen-Yung Wang. 2008. A context aware
handheld wayfnding system for individuals with cognitive impairments. In
Proceedings of the 10th international ACM SIGACCESS conference on Computers
and accessibility (Assets ’08), 27–34. https://doi.org/10.1145/1414471.1414479
[24] Gibson Chimamiwa, Marjan Alirezaie, Hadi Banaee, Uwe Köckemann, and
Amy Loutf. 2019. Towards Habit Recognition in Smart Homes for People with
Dementia. In Ambient Intelligence (Lecture Notes in Computer Science), 363–369.
https://doi.org/10.1007/978-3-030-34255-5_29
[25] Yi Chu, Pat Brown, Mark Harniss, Henry Kautz, and Kurt Johnson. 2014. Cog-
nitive support technologies for people with TBI: current usage and challenges
experienced. Disability and Rehabilitation: Assistive Technology 9, 4: 279–285.
https://doi.org/10.3109/17483107.2013.823631
[26] Dagoberto Cruz-Sandoval and Jesus Favela. 2016. Human-robot interaction to
deal with problematic behaviors from people with dementia. In Proceedings of
the 10th EAI International Conference on Pervasive Computing Technologies for
Healthcare (PervasiveHealth ’16), 274–275.
[27] Richard Davies, Chris Nugent, Mark Donnelly, Marike Hettinga, Franka Meiland,
Ferial Moelaert, Maurice Mulvenna, Johan Bengtsson, David Craig, and Rose-
Marie Dröes. 2009. A user driven approach to develop a cognitive prosthetic to
address the unmet needs of people with mild dementia. Pervasive and Mobile
Computing 5, 3: 253–267. https://doi.org/10.1016/j.pmcj.2008.07.002
[28] Melissa Dawe. 2006. Desperately seeking simplicity: how young adults with cog-
nitive disabilities and their families adopt assistive technologies. In Proceedings
of the SIGCHI Conference on Human Factors in Computing Systems, 1143–1152.
https://doi.org/10.1145/1124772.1124943
[29] Melissa Dawe. 2007. Understanding mobile phone requirements for young
adults with cognitive disabilities. In Proceedings of the 9th international ACM
SIGACCESS conference on Computers and accessibility (Assets ’07), 179–186.
https://doi.org/10.1145/1296843.1296874
[30] Emma Dixon, Jesse Anderson, and Amanda Lazar. 2022. Understanding How
Sensory Changes Experienced by Individuals with a Range of Age-Related
Cognitive Changes can Afect Technology Use. ACM Transactions on Accessible
Computing. https://doi.org/10.1145/3511906
[31] Emma Dixon and Amanda Lazar. 2020. The Role of Sensory Changes in Ev-
eryday Technology use by People with Mild to Moderate Dementia. In The
22nd International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’20), 1–12. https://doi.org/10.1145/3373625.3417000
[32] Emma Dixon, Anne Marie Piper, and Amanda Lazar. 2021. “Taking care of myself
as long as I can”: How People with Dementia Confgure Self-Management Sys-
tems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing
Systems, 1–14. https://doi.org/10.1145/3411764.3445225
[33] Emma Dixon, Ashrith Shetty, Simone Pimento, and Amanda Lazar. 2021. Lessons
Learned from Remote User-Centered Design with People with Dementia. In
Proceedings of the 2021 Dementia Lab Conference (Design for Inclusion).
[34] Leah Findlater, Karyn Mofatt, Joanna McGrenere, and Jessica Dawson. 2009.
Ephemeral adaptation: the use of gradual onset to improve menu selection
performance. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems, 1655–1664. https://doi.org/10.1145/1518701.1518956
[35] Ed Freeman, Linda Clare Dr, Nada Savitch, Lindsay Royan, Rachael Litherland,
and Margot Lindsay. 2005. Improving website accessibility for people with
ASSETS ’22, October 23–26, 2022, Athens, Greece
early-stage dementia: A preliminary investigation. Aging & Mental Health 9, 5:
442–448. https://doi.org/10.1080/13607860500142838
[36] Grant Gibson, Claire Dickinson, Katie Brittain, and Louise Robinson. 2015. The
everyday use of assistive technology by people with dementia and their family
carers: a qualitative study. BMC Geriatrics 15, 1: 1–10. https://doi.org/10.1186/
s12877-015-0091-3
[37] Grant Gibson, Claire Dickinson, Katie Brittain, and Louise Robinson. 2019.
Personalisation, customisation and bricolage: how people with dementia and
their families make assistive technology work for them. Ageing & Society 39, 11:
2502–2519. https://doi.org/10.1017/S0144686X18000661
[38] Google Inc. 2022. View, group & share contacts - Android - Contacts Help.
Google Support - Contacts Help. Retrieved June 24, 2022 from https://support.
google.com/contacts/answer/30970?hl=en&co=GENIE.Platform%3DAndroid
[39] Estefanía Guisado-Fernández, Guido Giunti, Laura M. Mackey, Catherine Blake,
and Brian Michael Caulfeld. 2019. Factors Infuencing the Adoption of Smart
Health Technologies for People With Dementia and Their Informal Caregivers:
Scoping Review and Design Framework. JMIR Aging 2, 1: e12192. https://doi.
org/10.2196/12192
[40] Yuqi Guo, Fan Yang, Fei Hu, Wei Li, Nicole Ruggiano, and Hee Yun Lee. 2020. Ex-
isting Mobile Phone Apps for Self-Care Management of People With Alzheimer
Disease and Related Dementias: Systematic Analysis. JMIR Aging 3, 1: e15290.
https://doi.org/10.2196/15290
[41] Chathurika Palliya Guruge, Sharon Oviatt, Pari Delir Haghighi, and Eliza-
beth Pritchard. 2021. Advances in Multimodal Behavioral Analytics for Early
Dementia Diagnosis: A Review. In Proceedings of the 2021 International Con-
ference on Multimodal Interaction, 328–340. Retrieved March 4, 2022 from
https://doi.org/10.1145/3462244.3479933
[42] Jose Guzman-Parra, Pilar Barnestein-Fonseca, Gloria Guerrero-Pertiñez, Peter
Anderberg, Luis Jimenez-Fernandez, Esperanza Valero-Moreno, Jessica Marian
Goodman-Casanova, Antonio Cuesta-Vargas, Maite Garolera, Maria Quintana,
Rebeca I García-Betances, Evi Lemmens, Johan Sanmartin Berglund, and Fermin
Mayoral-Cleries. 2020. Attitudes and Use of Information and Communication
Technologies in Older Adults With Mild Cognitive Impairment or Early Stages
of Dementia and Their Caregivers: Cross-Sectional Study. Journal of Medical
Internet Research 22, 6: e17253. https://doi.org/10.2196/17253
[43] Bart Hattink, Rose-Marie Droes, Sietske Sikkes, Ellen Oostra, and Afna W
Lemstra. 2016. Evaluation of the Digital Alzheimer Center: Testing Usability
and Usefulness of an Online Portal for Patients with Dementia and Their Carers.
JMIR Research Protocols 5, 3. https://doi.org/10.2196/resprot.5040
[44] Soumya Hegde and Ratnavalli Ellajosyula. 2016. Capacity issues and decision-
making in dementia. Annals of Indian Academy of Neurology 19, Suppl 1: S34–S39.
https://doi.org/10.4103/0972-2327.192890
[45] Niels Hendriks, Liesbeth Huybrechts, Andrea Wilkinson, and Karin Slegers.
2014. Challenges in doing participatory design with people with dementia. In
Proceedings of the 13th Participatory Design Conference on Short Papers, Industry
Cases, Workshop Descriptions, Doctoral Consortium papers, and Keynote abstracts
- PDC ’14 - volume 2, 33–36. https://doi.org/10.1145/2662155.2662196
[46] Kristine Holbø, Silje Bøthun, and Yngve Dahl. 2013. Safe walking technology for
people with dementia: what do they want? In Proceedings of the 15th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’13), 1–8.
https://doi.org/10.1145/2513383.2513434
[47] Kat Holmes. 2018. Mismatch: How Inclusion Shapes Design. MIT Press, Cambridge,
MA, USA.
[48] Hilary Hutchinson, Wendy Mackay, Bo Westerlund, Benjamin B. Bederson,
Allison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy,
Helen Evans, Heiko Hansen, Nicolas Roussel, and Björn Eiderbäck. 2003. Tech-
nology probes: inspiring design for and with families. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems (CHI ’03), 17–24.
https://doi.org/10.1145/642611.642616
[49] W3C Web Accessibility Initiative (WAI). Web Content Accessibility Guidelines
(WCAG) Overview. Web Accessibility Initiative (WAI). Retrieved January 9, 2021
from https://www.w3.org/WAI/standards-guidelines/wcag/
[50] Shaun K. Kane, Chandrika Jayant, Jacob O. Wobbrock, and Richard E. Ladner.
2009. Freedom to roam: a study of mobile device adoption and accessibility for
people with visual and motor disabilities. In Proceedings of the 11th international
ACM SIGACCESS conference on Computers and accessibility (Assets ’09), 115–122.
https://doi.org/10.1145/1639642.1639663
[51] Shaun K. Kane, Barbara Linam-Church, Kyle Althof, and Denise McCall. 2012.
What we talk about: designing a context-aware communication tool for people
with aphasia. In Proceedings of the 14th international ACM SIGACCESS conference
on Computers and accessibility - ASSETS ’12, 49. https://doi.org/10.1145/2384916.
2384926
[52] Yvonne Kerkhof, Ad Bergsma, Maud Graf, and Rose-Marie Dröes. 2017. Se-
lecting apps for people with mild dementia: Identifying user requirements for
apps enabling meaningful activities and self-management. Journal of Reha-
bilitation and Assistive Technologies Engineering 4: 2055668317710593. https:
//doi.org/10.1177/2055668317710593
ASSETS ’22, October 23–26, 2022, Athens, Greece
[53] Yvonne Kerkhof, Gianna Kohl, Melanie Veijer, Floriana Mangiaracina, Ad
Bergsma, Maud Graf, and Rose-Marie Dröes. 2020. Randomized controlled
feasibility study of FindMyApps: frst evaluation of a tablet-based interven-
tion to promote self-management and meaningful activities in people with
mild dementia. Disability and Rehabilitation: Assistive Technology 0, 0: 1–15.
https://doi.org/10.1080/17483107.2020.1765420
[54] Yvonne Kerkhof, Myrna Pelgrum-Keurhorst, Floriana Mangiaracina, Ad
Bergsma, Guus Vrauwdeunt, Maud Graf, and Rose-Marie Dröes. 2019. User-
participatory development of FindMyApps; a tool to help people with mild de-
mentia fnd supportive apps for self-management and meaningful activities. DIG-
ITAL HEALTH 5: 205520761882294. https://doi.org/10.1177/2055207618822942
[55] Helianthe S. M. Kort and Joost van Hoof. 2014. Design of a website for home
modifcations for older persons with dementia. Technology and Disability 26, 1:
1–10. https://doi.org/10.3233/TAD-140399
[56] Clemens Scott Kruse, Joanna Fohn, Gilson Umunnakwe, Krupa Patel, and
Saloni Patel. 2020. Evaluating the Facilitators, Barriers, and Medical Out-
comes Commensurate with the Use of Assistive Technology to Support Peo-
ple with Dementia: A Systematic Review Literature. Healthcare 8, 3. https:
//doi.org/10.3390/healthcare8030278
[57] Haley M LaMonica, Amelia English, Ian B Hickie, Jerome Ip, Catriona Ireland,
Stacey West, Tim Shaw, Loren Mowszowski, Nick Glozier, Shantel Dufy, Alice A
Gibson, and Sharon L Naismith. 2017. Examining Internet and eHealth Practices
and Preferences: Survey Study of Australian Older Adults With Subjective Mem-
ory Complaints, Mild Cognitive Impairment, or Dementia. Journal of Medical
Internet Research 19, 10: e358. https://doi.org/10.2196/jmir.7981
[58] Jonathan Lazar, Libby Kumin, and Jinjuan Heidi Feng. 2011. Understanding
the computer skills of adult expert users with down syndrome: an exploratory
study. In The proceedings of the 13th international ACM SIGACCESS conference
on Computers and accessibility (ASSETS ’11), 51–58. https://doi.org/10.1145/
2049536.2049548
[59] Raphael J. Leo. 1999. Competency and the Capacity to Make Treatment Deci-
sions: A Primer for Primary Care Physicians. Primary Care Companion to The
Journal of Clinical Psychiatry 1, 5: 131–141.
[60] Clayton Lewis, James Sullivan, and Jefery Hoehl. 2009. Mobile Technology
for People with Cognitive Disabilities and Their Caregivers – HCI Issues. In
Universal Access in Human-Computer Interaction. Addressing Diversity (Lecture
Notes in Computer Science), 385–394. https://doi.org/10.1007/978-3-642-02707-
9_44
[61] Stephen Lindsay, Katie Brittain, Daniel Jackson, Cassim Ladha, Karim Ladha, and
Patrick Olivier. 2012. Empathy, participatory design and people with dementia.
In Proceedings of the 2012 ACM annual conference on Human Factors in Computing
Systems - CHI ’12, 521. https://doi.org/10.1145/2207676.2207749
[62] Philippe Lopes, Maribel Pino, Giova Carletti, Sofana Hamidi, Sylvie Legué,
Helene Kerhervé, Samuel Benveniste, Guillaume Andéol, Pierre Bonsom, S
Reingewirtz, and Anne-Sophie Rigaud. 2016. Co-Conception Process of an
Innovative Assistive Device to Track and Find Misplaced Everyday Objects for
Older Adults with Cognitive Impairment: The TROUVE Project. IRBM 37, 2:
52–57. https://doi.org/10.1016/j.irbm.2016.02.004
[63] Ahmad Lotf, Caroline Langensiepen, Sawsan M. Mahmoud, and M. J. Akhlagh-
inia. 2012. Smart homes for the elderly dementia suferers: identifcation and
prediction of abnormal behaviour. Journal of Ambient Intelligence and Human-
ized Computing 3, 3: 205–218. https://doi.org/10.1007/s12652-010-0043-x
[64] Kent Lyons, Christopher Skeels, and Thad Starner. 2005. Providing support for
mobile calendaring conversations: a wizard of oz evaluation of dual–purpose
speech. In Proceedings of the 7th international conference on Human computer
interaction with mobile devices & services (MobileHCI ’05), 243–246. https://doi.
org/10.1145/1085777.1085821
[65] Kent Lyons, Christopher Skeels, Thad Starner, Cornelis M. Snoeck, Benjamin
A. Wong, and Daniel Ashbrook. 2004. Augmenting conversations using dual-
purpose speech. In Proceedings of the 17th annual ACM symposium on User
interface software and technology (UIST ’04), 237–246. https://doi.org/10.1145/
1029632.1029674
[66] Camilla Malinowsky, Ove Almkvist, Anders Kottorp, and Louise Nygård. 2010.
Ability to manage everyday technology: a comparison of persons with dementia
or mild cognitive impairment and older adults without cognitive impairment.
Disability and Rehabilitation: Assistive Technology 5, 6: 462–469. https://doi.org/
10.3109/17483107.2010.496098
[67] Dana McKay and Charlynn Miller. 2021. Standing in the Way of Control: A Call
to Action to Prevent Abuse through Better Design of Smart Technologies. In
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems,
1–14. https://doi.org/10.1145/3411764.3445114
[68] Franka Meiland, Ans Bouman, Stefan Sävenstedt, Sanne Bentvelzen, Richard
Davies, Maurice Mulvenna, Chris Nugent, Ferial Moelaert, Marike Hettinga,
Johan Bengtsson, and Rose-Marie Dröes. 2012. Usability of a new electronic
assistive device for community-dwelling persons with mild dementia. Aging &
Mental Health 16, 5: 584–591. https://doi.org/10.1080/13607863.2011.651433
[69] Microsoft. 2022. Send an email message with a follow-up reminder. Microsoft
Support. Retrieved June 24, 2022 from https://support.microsoft.com/en-
Emma Dixon et al.
us/ofce/send-an-email-message-with-a-follow-up-reminder-740a3b9e-
e837-4711-938a-08dd0ea5ac64
[70] Masahiro Mori, Karl F. MacDorman, and Norri Kageki. 2012. The Uncanny
Valley [From the Field]. IEEE Robotics Automation Magazine 19, 2: 98–100. https:
//doi.org/10.1109/MRA.2012.2192811
[71] Kimberly D. Mueller, Bruce Hermann, Jonilda Mecollari, and Lyn S. Turk-
stra. 2018. Connected speech and language in mild cognitive impairment and
Alzheimer’s disease: A review of picture description tasks. Journal of Clini-
cal and Experimental Neuropsychology 40, 9: 917–939. https://doi.org/10.1080/
13803395.2018.1446513
[72] Maurice Mulvenna, Suzanne Martin, Stefan Sävenstedt, Johan Bengtsson, Franka
Meiland, Rose Marie Dröes, Marike Hettinga, Ferial Moelaert, and David Craig.
2010. Designing & evaluating a cognitive prosthetic for people with mild de-
mentia. In Proceedings of the 28th Annual European Conference on Cognitive
Ergonomics - ECCE ’10, 11. https://doi.org/10.1145/1962300.1962306
[73] David Nandigam, Judith Symonds, Nicola Kayes, and Kathryn McPherson. 2010.
Mobile phone user interface design for patients with traumatic brain injury.
In Proceedings of the 11th International Conference of the NZ Chapter of the
ACM Special Interest Group on Human-Computer Interaction (CHINZ ’10), 69–72.
https://doi.org/10.1145/1832838.1832850
[74] National Institute of Aging. 2021. Alzheimer’s Disease Fact Sheet. National
Institute on Aging. Retrieved October 11, 2021 from http://www.nia.nih.gov/
health/alzheimers-disease-fact-sheet
[75] Donald Norman. 1999. Afordance, conventions, and design. Interactions 6: 38–42.
https://doi.org/10.1145/301153.301168
[76] Donald A. Norman and Stephen W. Draper (eds.). 1986. User Centered System
Design: New Perspectives on Human-computer Interaction. CRC Press, Hillsdale,
NJ.
[77] Chris D. Nugent, Richard J. Davies, Mark P. Donnelly, Josef Hallberg, Mossaab
Hariz, David Craig, Franka Meiland, Ferial Moelaert, Johan E. Bengtsson, Stefan
Savenstedt, Maurice Mulvenna, and Rose-Marie Droes. 2008. The development of
personalised cognitive prosthetics. In 2008 30th Annual International Conference
of the IEEE Engineering in Medicine and Biology Society, 787–790. https://doi.org/
10.1109/IEMBS.2008.4649270
[78] Louise Nygård and S Starkhammar. 2007. The use of everyday technology by
people with dementia living alone: Mapping out the difculties. Aging & mental
health 11: 144–55. https://doi.org/10.1080/13607860600844168
[79] Siobhan O’Connor, Matt-Mouley Bouamrane, Catherine A. O’Donnell, and
Frances Mair. 2016. Barriers to Co-Designing Mobile Technology with Persons
with Dementia and Their Carers. Nursing Informatics. https://doi.org/10.3233/
978-1-61499-658-3-1028
[80] Roger Orpwood, Chris Gibbs, Timothy Adlam, Ricahrd Faulkner, and D. Meega-
hawatte. 2005. The design of smart homes for people with dementia—user-
interface aspects. Universal Access in the Information Society 4, 2: 156–164.
https://doi.org/10.1007/s10209-005-0120-7
[81] Martin Pielot, Karen Church, and Rodrigo de Oliveira. 2014. An in-situ study of
mobile phone notifcations. In Proceedings of the 16th international conference
on Human-computer interaction with mobile devices & services - MobileHCI ’14,
233–242. https://doi.org/10.1145/2628363.2628364
[82] Matthew L. Poole, Amy Brodtmann, David Darby, and Vogel. 2017. Motor Speech
Phenotypes of Frontotemporal Dementia, Primary Progressive Aphasia, and Pro-
gressive Apraxia of Speech. Journal of Speech, Language, and Hearing Research
60, 4: 897–911. https://doi.org/10.1044/2016_JSLHR-S-16-0140
[83] Leon Reicherts, Nima Zargham, Michael Bonfert, Yvonne Rogers, and Rainer
Malaka. 2021. May I Interrupt? Diverging Opinions on Proactive Smart Speakers.
In CUI 2021 - 3rd Conference on Conversational User Interfaces (CUI ’21), 1–10.
https://doi.org/10.1145/3469595.3469629
[84] Merja Riikonen, Eija Paavilainen, and Hannu Salo. 2013. Factors supporting the
use of technology in daily life of home-living people with dementia. Technology
& Disability 25, 4: 233–243. https://doi.org/10.3233/TAD-130393
[85] Lena Rosenberg, Anders Kottorp, Bengt Winblad, and Louise Nygård. 2009.
Perceived difculty in everyday technology use among older adults with or
without cognitive defcits. Scandinavian Journal of Occupational Therapy 16, 4:
216–226. https://doi.org/10.3109/11038120802684299
[86] Meredeth A. Rowe and Vikki Bennett. 2003. A look at deaths occurring in persons
with dementia lost in the community. American Journal of Alzheimer’s Disease
& Other Dementias® 18, 6: 343–348. https://doi.org/10.1177/153331750301800612
[87] Yoichi Sakai, Yuuko Nonaka, Kiyoshi Yasuda, and Yukiko I. Nakano. 2012. Lis-
tener agent for elderly people with dementia. In Proceedings of the seventh
annual ACM/IEEE international conference on Human-Robot Interaction (HRI ’12),
199–200. https://doi.org/10.1145/2157689.2157754
[88] Vardit Sarne-Fleischmann, Noam Tractinsky, Tzvi Dwolatzky, and Inbal Rief.
2011. Personalized reminiscence therapy for patients with Alzheimer’s disease
using a computerized system. In Proceedings of the 4th International Conference
on PErvasive Technologies Related to Assistive Environments (PETRA ’11), 1–4.
https://doi.org/10.1145/2141622.2141679
[89] Ben Shneiderman. 2022. Human-Centered AI. Oxford University Press.
Mobile Phone Use by People with Mild to Moderate Dementia: Uncovering Challenges and Identifying Opportunities
[90] Andrew Sixsmith, Grant Gibson, Roger Orpwood, and Judith Torrington. 2007.
Developing a technology ‘wish-list’ to enhance the quality of life of people with
dementia. Gerontechnology 6. https://doi.org/10.4017/gt.2007.06.01.002.00
[91] Aida Suárez-González, Susie M. Henley, Jill Walton, and Sebastian J. Crutch.
2015. Posterior Cortical Atrophy: An Atypical Variant of Alzheimer Disease.
Psychiatric Clinics 38, 2: 211–220. https://doi.org/10.1016/j.psc.2015.01.009
[92] Hiroki Tanaka, Hiroyoshi Adachi, Norimichi Ukita, Takashi Kudo, and Satoshi
Nakamura. 2016. Automatic detection of very early stage of dementia through
multimodal interaction with computer avatars. In Proceedings of the 18th ACM
International Conference on Multimodal Interaction (ICMI ’16), 261–265. https:
//doi.org/10.1145/2993148.2993193
[93] The Health Research Authority. 2005. Mental Capacity Act. Retrieved June 29,
2021 from https://www.hra.nhs.uk/planning-and-improving-research/policies-
standards-legislation/mental-capacity-act/
[94] Federico Tiersen, Philippa Batey, Matthew J. C. Harrison, Lenny Naar, Alina-
Irina Serban, Sarah J. C. Daniels, and Rafael A. Calvo. 2021. Smart Home Sensing
and Monitoring in Households With Dementia: User-Centered Design Approach.
JMIR Aging 4, 3: e27047. https://doi.org/10.2196/27047
[95] Esther Y. C. Tiong, David M. W. Powers, and Anthony J. Maeder. 2018. Dementia
virtual assistant as trainer and therapist: identifying signifcant memories and
interventions of dementia patients. In Proceedings of the Australasian Computer
Science Week Multiconference (ACSW ’18), 1–9. https://doi.org/10.1145/3167918.
3167953
[96] Kelvin KF Tsoi, Max WY Lam, Christopher TK Chu, Michael PF Wong, and
Helen ML Meng. 2018. Machine Learning on Drawing Behavior for Dementia
Screening. In Proceedings of the 2018 International Conference on Digital Health
(DH ’18), 131–132. https://doi.org/10.1145/3194658.3194659
[97] Gregg Vanderheiden, Jonathan Lazar, J. Bern Jordan, Yao Ding, and Rachel E.
Wood. 2020. Morphic: Auto-Personalization on a Global Scale. In Proceedings of
the 2020 CHI Conference on Human Factors in Computing Systems, 1–12. Retrieved
January 27, 2022 from https://doi.org/10.1145/3313831.3376204
ASSETS ’22, October 23–26, 2022, Athens, Greece
[98] Gregg Vanderheiden, Jutta Treviranus, José Angel Martinez Usero, Evangelos
Bekiaris, Maria Gemou, and Amrish Chourasia. 2012. Auto-Personalization:
Theory, Practice and Cross-Platform Implementation. Proceedings of the Human
Factors and Ergonomics Society Annual Meeting 56: 926–930. https://doi.org/10.
1177/1071181312561193
[99] W3C Working Group. 2021. Making Content Usable for People with Cognitive
and Learning Disabilities. Retrieved March 4, 2022 from https://www.w3.org/
TR/coga-usable/#dfn-cognitive-and-learning-disabilities
[100] Lin Wan, Claudia Müller, Dave Randall, and Volker Wulf. 2016. Design of A
GPS Monitoring System for Dementia Care and its Challenges in Academia-
Industry Project. ACM Transactions on Computer-Human Interaction 23, 5: 1–36.
https://doi.org/10.1145/2963095
[101] Lin Wan, Claudia Müller, Volker Wulf, and David William Randall. 2014. Address-
ing the subtleties in dementia care: pre-study & evaluation of a GPS monitoring
system. In Proceedings of the 32nd annual ACM conference on Human factors in
computing systems - CHI ’14, 3987–3996. https://doi.org/10.1145/2556288.2557307
[102] Jing Wei, Tilman Dingler, and Vassilis Kostakos. 2021. Developing the Proactive
Speaker Prototype Based on Google Home. In Extended Abstracts of the 2021
CHI Conference on Human Factors in Computing Systems, 1–6. Retrieved March
4, 2022 from https://doi.org/10.1145/3411763.3451642
[103] Jing Wei, Tilman Dingler, and Vassilis Kostakos. 2022. Understanding User
Perceptions of Proactive Smart Speakers. Proceedings of the ACM on Interactive,
Mobile, Wearable and Ubiquitous Technologies 5, 4: 185:1-185:28. https://doi.org/
10.1145/3494965
[104] Eleanor Bantry White and Paul Montgomery. 2015. Dementia, walking outdoors
and getting lost: incidence, risk factors and consequences from dementia-related
police missing-person reports. Aging & Mental Health 19, 3: 224–230. https:
//doi.org/10.1080/13607863.2014.924091
[105] Jacob O. Wobbrock, Shaun K. Kane, Krzysztof Z. Gajos, Susumu Harada, and
Jon Froehlich. 2011. Ability-Based Design: Concept, Principles and Examples.
ACM Transactions on Accessible Computing 3, 3: 9:1-9:27. https://doi.org/10.1145/
1952383.1952384
ASSETS ’22, October 23–26, 2022, Athens, Greece Emma Dixon et al.

A APPENDICES
Table 2: Table of Apps and Features from Participants Self-reported Mobile Phone Usage

Type of App or Feature App or Feature Participants who Reported Using that App or Feature Number of
Participants
Social Media Facebook Thomas, Elenora, Kim, Tristen, Sylvia, Kennith, Tina, Malcolm, Josslyn, Hall, 11
Alecia
YouTube Sylvia, Preston, Kennith, Hall 4
Instagram Elenora, Tina 2
SnapChat Alecia, Sabella 2
TikTok Sabella 1
LinkedIn Thomas 1
Twitter Thomas 1
Navigation Waze Tristen, Kennith, Malcolm 3
Google Maps Elenora, Hall 2
Apple Maps Thomas 1
Life360 Thomas 1
Uber Elenora 1
Lift Elenora 1
Unspecifed Kim, Miranda 2
Online Shopping Amazon Kim, Josslyn, Alecia 3
Sams App Kim 1
Ebay Alecia 1
Mobile Banking Capital One App Elenora 1
Apple Wallet Thomas 1
Venmo Alecia 1
unspecifed Kim, Tristen, Sylvia, Sabella 4
News BBC Malcolm 1
CNN Malcolm 1
Fox Malcolm 1
NPR Malcolm 1
unspecifed Josslyn, Hall 2
Music Pandora Elenora, Hall 2
Shazam Alecia 1
unspecifed Thomas, Tristen, Tina, Alecia, Sabella 5
Communication Email Thomas, Elenora, Tristen, Sylvia, Miranda, Tina, Malcolm, Josslyn, Sabella 9
phone calls Kim, Tristen, Preston, Kennith, Josslyn, Hall, Sabella 7
Messenger Kim, Kennith, Tina, Malcolm, Hall, Sabella 6
SMS Tristen, Sylvia, Preston, Josslyn, Alecia 5
FaceTime Miranda, Tina, Malcolm, Sabella 4
Zoom Sylvia, Alecia 2
Google Meet Miranda 1
Evite Kennith 1
Blogs Tina 1
WhatsApp Elenora 1
Games Words with Friends Kennith 1
Solitaire Hall 1
memory games Sabella 1
unspecifed Kim, Alecia 2
Entertainment Photos Thomas, Elenora, Sylvia, Preston, Hall, Alecia 6
Audible Kim, Hall, Alecia 3
Bible app Kim, Hall 2
Deer Cast Hall 1
Apple TV Sabella 1
Planter - Garden Planner Kim 1
app
ESPN Hall 1

Frameo Alecia 1
GoFan Hall 1
E-books Thomas 1
Productivity Reminders Thomas, Elenora, Tristen, Sylvia, Miranda, Malcolm, Josslyn, Hall 8
Google Search Preston, Miranda, Kennith, Tina, Malcolm, Josslyn, Sabella 7
Calendar Thomas, Kim, Preston, Kennith, Tina, Hall, Alecia 7
Google Calendar Elenora, Tristen, Sylvia, Miranda, Malcolm, Josslyn 6
Calculator Thomas, Elenora, Kim, Alecia 4
Weather Kim, Tina, Hall, Alecia 4
KeepNote Elenora, Sylvia, Alecia 3
Clock Elenora, Tristen, Alecia 3
Google Workspace Thomas, Sylvia 2
Timer Kim 1
Notes Kim 1
Outlook Alecia 1
Evernote Elenora 1
Flashlight Thomas 1
Health Management Fitbit Alecia 1
pharmacy apps Kim 1
unspecifed
patient portals Sabella 1
glucose monitoring app Elenora 1
fall detection apps Miranda 1
 Freedom to Choose: Understanding Input Modality Preferences
of People with Upper-body Motor Impairments for Activities of
Daily Living
Franklin Mingzhe Li
Carnegie Mellon University
Pittsburgh, Pennsylvania, USA
mingzhe2@cs.cmu.edu
Yang Zhang
University of California, Los Angeles
Los Angeles, California, USA
yangzhang@ucla.edu
ABSTRACT
Many people with upper-body motor impairments encounter chal-
lenges while performing Activities of Daily Living (ADLs) and
Instrumental Activities of Daily Living (IADLs), such as toileting,
grooming, and managing fnances, which have impacts on their
Quality of Life (QOL). Although existing assistive technologies en-
able people with upper-body motor impairments to use diferent
input modalities to interact with computing devices independently
(e.g., using voice to interact with a computer), many people still
require Personal Care Assistants (PCAs) to perform ADLs. Mul-
timodal input has the potential to enable users to perform ADLs
without human assistance. We conducted 12 semi-structured in-
terviews with people who have upper-body motor impairments
to capture their existing practices and challenges of performing
ADLs, identify opportunities to expand the input possibilities for
assistive devices, and understand user preferences for multimodal
interaction during everyday tasks. Finally, we discuss implications
for the design and use of multimodal input solutions to support
user independence and collaborative experiences when performing
daily living tasks.
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility.
KEYWORDS
People with upper-body motor impairments, multimodal, assistive
technology, activity of daily living
ACM Reference Format:
Franklin Mingzhe Li, Michael Xieyang Liu, Yang Zhang, and Patrick Carring-
ton. 2022. Freedom to Choose: Understanding Input Modality Preferences of
People with Upper-body Motor Impairments for Activities of Daily Living.
This work is licensed under a Creative Commons Attribution International
4.0 License.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3544814
Michael Xieyang Liu
Carnegie Mellon University
Pittsburgh, Pennsylvania, USA
xieyangl@cs.cmu.edu
Patrick Carrington
Carnegie Mellon University
Pittsburgh, Pennsylvania, USA
pcarrington@cmu.edu
In The 24th International ACM SIGACCESS Conference on Computers and
Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New
York, NY, USA, 16 pages. https://doi.org/10.1145/3517428.3544814
1 INTRODUCTION
Activities of Daily Living (ADLs) is a term used to collectively de-
scribe fundamental tasks that are required for a person to care
for themselves, independently [18]. They are often broken up into
two categories: 1) basic ADLs, such as eating, bathing, and groom-
ing, and 2) Instrumental Activities of Daily Living (IADLs), such
as managing fnances, shopping, house cleaning, and managing
communication. Throughout this paper, we will use “ADLs” to re-
fer to tasks in both of these categories. Collectively, these tasks
form a base from which society understands a person’s essential
independence and begins to evaluate Quality of Life (QOL) [22].
Since many of these tasks involve interacting with objects and one’s
environment using hands and arms, people with upper-body motor
impairments can experience difculties completing these activities
[56]. Prior work has shown that people generally adopt one of
two solutions to help with ADL tasks: digital assistance through
assistive technologies [24, 74]) or human assistance provided by
Personal Care Assistants (PCAs) [50]. Although there has been
prior work to develop assistive technologies that can empower peo-
ple with disabilities to interact with diferent computing devices
by leveraging alternate input modalities, many users still require
human assistance to complete tasks efectively [50].
The reliance on human assistance is due, in part, to a lack of
knowledge on how to efectively leverage various input modali-
ties, which have become increasingly common among emerging
computing devices. We have not fully explored the potential for
multimodal input to support people with upper-body motor im-
pairments in ADL tasks that do not completely rely on computing
technology like mobile devices and laptops. Furthermore, we do
not fully understand the collaborative role of technology and PCAs
in completing ADL tasks.
To begin addressing these gaps, we conducted an interview study
with 12 people with upper-body motor impairments. We frst aimed
to understand the current practices of people with upper-body mo-
tor impairments while performing ADLs, along with the challenges
they encounter. We then explored the potential uses of diferent
ASSETS ’22, October 23–26, 2022, Athens, Greece
input modalities during specifc daily living tasks. Finally, we high-
lighted potential applications and implications for the design of
multimodal assistive technologies to facilitate user input during
ADL tasks. Our research questions are:
RQ1: What are the current practices and challenges of people with
upper-body motor impairments during ADLs?
RQ2: How and why would various input modalities beneft people
with upper-body motor impairments during ADLs?
RQ3: How can we leverage multimodal input to support people
with upper-body motor impairments during ADLs?
In the sections that follow, we frst describe upper-body motor
impairments and ADLs (Section 2.1). We then summarize related lit-
erature on both common and emerging input modalities for people
with upper-body motor impairments (Section 2.2) as well as prior
research examining multimodal interactions for assistive applica-
tions (Section 2.3). Next, we describe our semi-structured interview
study, the subsequent analysis, and our fndings. Specifcally, we
describe practices and challenges of existing approaches toward
ADLs (Section 4). We interrogate users’ preferences for applying
individual inputs and combinations of inputs to support specifc
daily living tasks (Section 5). We then describe reasons for indi-
vidual input modality and multimodal input alternatives (Section
6). Finally, we provide design recommendations for future input
solutions to support people with upper-body motor impairments
during ADLs (Section 7), including creating multimodal designs
that consider collaborative experiences in ADLs, diferentiating be-
tween interaction with computing devices and systems to support
traditional ADLs (e.g., toileting), and consideration for actuation
and human-robot interaction during multimodal interactions.
2 BACKGROUND AND RELATED WORK
In this section, we frst provide background information regarding
upper-body motor impairments and the importance of performing
ADLs (Section 2.1). Next, we describe existing literature that ex-
plored diferent input modalities to assist people with upper-body
motor impairments (Section 2.2). Finally, we show existing research
on multimodal input and how it may beneft ways of interactions
(Section 2.3).
2.1 Upper-body Motor Impairments and ADLs
The term Upper-body motor impairment usually refers to motor im-
pairments that afect the upper extremities, which are often caused
by spinal cord injury, cerebral palsy, muscular dystrophy, etc [69].
People with upper-body motor impairments may have diferent
mobility conditions of their upper limbs, such as fne motor and
gross motor impairments. When rehabilitation specialists assess
the QOL for people with upper-body motor impairments, they usu-
ally evaluate the capability of performing basic and instrumental
activities of daily living (i.e., ADLs and IADLs) [22]. Basic ADLs in-
volve essential tasks such as grooming, cooking, toileting, dressing,
and showering, and instrumental ADLs (IADLs) refer to activities
that require more planning and thinking, such as using a phone
and managing fnances [33]. According to existing research, many
people with upper-body motor impairments rely on PCAs for ADLs
[50] or use inputs to control devices (e.g., voice) to overcome the
Li et al.
limitations of upper-body motor impairments [10, 12, 13, 28]. In or-
der to further support the independence of people with upper-body
motor impairments in ADLs and improve the QOL, it is important to
frst understand how people with upper-body motor impairments
perform diferent ADLs and associated challenges.
2.2 Input Modalities for People with
Upper-body Motor Impairments
Prior research also explored various input modalities for people
with upper-body motor impairments to interact with diferent sys-
tems to compensate the upper-body motor impairments [28, 36],
such as controlling mobile devices [15, 19, 41], performing text-
entry tasks [60, 71], interacting with TVs [64], using robotic arms
[5], and playing games [21]. These approaches either rely on readily
available devices (e.g., [19]) or customized technologies (e.g., [5, 64]).
To interact with these devices, prior research explored various in-
put modalities for people with upper-body motor impairments—
touch input [11, 60, 65], hand or arm gestures [3, 4, 58], voice input
[6, 24, 57], eye-based input [19, 20, 31, 74], head-movement input
[15, 55], brain-computer input [17, 46], facial or mouth-based ges-
tures [23, 45, 67], and biometrics [30, 32, 47]. Table 1 summarizes
input modalities presented in prior work, from which we took much
inspiration in generating our user studies.
Touch Input: Existing research has explored touch input meth-
ods and customization on computing devices to support people who
have upper-body motor impairments. For example, Vatavu and Un-
gurean [65] conducted stroke-gesture analyses from a dataset of
9681 gestures collected from 70 participants with motor impair-
ments to outline the research roadmap for accessible gesture input
on touchscreens. Similar explorations on touch input have also
been conducted using various devices, such as trackballs [71, 72],
joysticks [60], smartphones [44, 48], tablets [65], smartwatches [41],
head-mounted displays [42], and customized touchpads [11].
Voice-based Input: Voice-based input has been studied to help
people with upper-body motor impairments in controlling diferent
IoT devices (e.g., [53]), substituting inaccessible input techniques
(e.g., [6]), or performing specifc tasks (e.g., drawing [24], program-
ming [57]). For example, Rosenblatt et al. [57] demonstrated that
using vocal input could help people with upper-body motor impair-
ments navigate and edit code in programming.
Eye-based Gesture: Furthermore, we found that eye-based ges-
tures, such as using eye-gaze fxations [74] and eyelid gestures
[19], could support people with upper-body motor impairments
in interacting with digital interfaces. For example, Zhang et al.
[74] demonstrated decoding eye gestures (e.g., looking up or look-
ing down) into commands that can be used to enable people with
upper-body motor impairments to control mobile devices.
Head-movement Input: Similar to eye-gaze directions, prior
research also leveraged head movements and orientations to control
the pointer on a device [15, 55]. For instance, Cicek et al. [15] pro-
posed a calibration-free head-tracking input mechanism for mobile
devices that allows people with upper-body motor impairments to
achieve pixel-level pointing precision on small screens.
Face-based or Mouth-based Input: Besides using eye-based
or head-based input, we found that existing research also explored
face-based or mouth-based gestures as an input modality to help
Understanding Input Modality Preferences of People with Upper-body Motor Impairments ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Input modalities for people with upper-body motor impairments.

Category Description Diagnosis of Motor Impairments Reference

Touch Input Touch control through joystick, track- Spinal cord injuries, spina bifda, [11], [41], [42], [44],
ball, smartphone, tablet, smartwatch, orthostatic tremor, cerebral palsy, [48], [60], [65], [71],
or customized touchpads. muscular dystrophy, multiple scle- [72]
rosis, osteogenesis imperfecta, juve-
nile rjeumatoid arthritis, radial nerve
injury, spastic quadriplegia, hemor-
rhagic stroke
Voice-based Input Use voice to control diferent IoT Spinal cord injury, cerebral palsy, [6], [24], [53], [57]
devices, substituting inaccessible in- stroke, spinal dysmorphism
put techniques, or performing specifc
tasks.
Eye-based Input Use eye-gaze position or eyelid ges- Spinal cord injury, amyotrophic lateral [19], [74]
tures to control devices. sclerosis
Head-movement Input Use head movements and orientations Non-specifc upper body [15], [55]
to control a pointer on devices.
Face-based or Mouth-based Use facial expressions or mouth-based Non-specifc upper body [23], [45], [67]
Input input (e.g., teeth tapping, sip-and-puf)
to interact with devices.
Hand or Arm Gesture Use personalized hand or arm gestures Cerebral palsy, hydrocephalus, [3], [4]
to interact with computing systems de- quadriplegia, spastic quadriplegia,
pending on the upper-body mobility. static encephalitis, pseudobulbar
palsy
Brain-Computer Interface Use Electroencephalogram (EEG) or Muscular dystrophy, stroke [16], [17], [46]
Electromyography (EMG) to allow
people with upper-body motor impair-
ments to interact with devices or to
understand their needs.
Biometric Input Use biometric information (e.g., fnger- Cerebral palsy [30], [47]
print, voice) for security and privacy
purposes.
Automatic Recognition or Use automatic recognition to respond Non-specifc upper body [8]
Other Input to the user without explicit input (e.g.,
automatic door opener) or other input
modalities.

people with motor impairments to interact with their devices [23,
45, 61, 67]. For example, Wang et al. [67] introduced the use of
facial expressions as controls in games, such as Super Mario Bros.,
and Grewal et al. [23] showed the approach of using sip-and-puf
systems to place commands to a power wheelchair.
Hand or Arm Gesture: From existing literature, we learned
that people with upper-body motor impairments might have vari-
ous levels and conditions of controlling their upper extremity to
interact with their devices (e.g., fne motor, gross motor) [48]. Thus,
prior research explored various recognition approaches that allow
detection of personalized hand or arm gestures for interaction with
computing systems based on the abilities of people with upper-body
motor impairments [3, 4].
Brain-Computer Interface: In addition, existing research also
explored EEG (Electroencephalography) or EMG (Electromyog-
raphy) approach that allows people with upper-body motor im-
pairments to interact with technologies through brain-computer
interfaces (BCI) or to understand their needs [16, 17, 46, 49]. For
example, Neuper et al. [49] introduced the approach of EEG-based
brain-computer interface to help people with severe motor im-
pairments accomplish tasks like selecting fne-grain letters, where
people need to precisely specify the start and end boundary of the
selection.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Biometric Input: As people with upper-body motor impair-
ments are becoming more conscious of privacy and security through
daily activities [32], researchers further proposed approaches to
leverage biometric information for people with upper-body motor
impairments [30, 34, 47], such as fngerprints. For instance, Lewis
and Venkatasubramanian [34] mentioned the existing approaches
of using fngerprint and face recognition to assist people with upper-
body motor impairments in completing the authentication process
to control their devices.
Overall, we introduced existing research that explored various in-
put modalities to help people with upper-body motor impairments
interact with their computing devices for diferent accomplishments
and under certain environments (e.g., [28]). Existing work mostly
focused on input modalities for computing devices (e.g., computers),
which does not fully address the problem of high reliance on PCAs
for ADLs (e.g., toileting, dressing). We chose to focus on practices
and challenges of leveraging diferent input modalities in various
ADLs. We are also interested in exploring which input modalities
are more preferred from the perspectives of people with upper-
body motor impairments, such as whether people prefer using
voice control or head gestures to tweak the water temperature.
2.3 Multimodal Input Methods for People with
Upper-body Motor Impairments
Existing research has studied the overall benefts of multimodal
input in human activities, such as achieving low-false positive
rates [63] and the accommodation of tasks and context changes
[54]. More importantly, Reeves et al. [54] also implied the poten-
tial benefts of multimodal input to adapt to individual diferences
in mobility conditions (e.g., sensor or motor impairments). Simi-
lar to what Reeves et al. [54] projected, prior research leveraged
multiple input modalities to support people with upper-body mo-
tor impairments to interact with their computing devices (e.g.,
[7, 16, 26, 39, 40, 62, 67, 68]). For example, Wang et al. [67] combined
eye input gestures (e.g., eye movements to the left and right, double
blink) with facial expressions (e.g., smile, open and close mouth)
to enable people with upper-body motor impairments to provide
input for VR games. As another example, Dupres et al. [16] com-
bined hand input with brain-computer interfaces for both better
control of applications in daily life (e.g., web browser, video game)
and enabling researchers to understand behaviors of people with
upper-body motor impairments. Finally, Tomari et al. [62] proposed
leveraging multimodal input by combining momentary switch and
head recognition to control the direction and orientation of smart
wheelchairs.
Although several works leveraged the benefts of combining mul-
tiple input modalities to better assist people with upper-body motor
impairments [70], there exist gaps between people with upper-body
motor impairments in ADLs and how multimodal modalities may
beneft the overall input experiences. We are interested in compre-
hensively investigating multimodal input as opposed 12ptto only a
few niche interactions and aim to investigate the applications of
multimodal input in ADLs. As a result, our research provides guide-
lines to HCI and Accessibility researchers on designing multimodal
input for people with upper-body motor impairments to help with
their ADLs.
Li et al.
3 METHOD
In this work, we conducted semi-structured interviews with people
who have upper-body motor impairments to learn about their ex-
isting practices and challenges when performing ADLs, and their
preferences for various input modalities. In addition, we inves-
tigated opportunities for multimodal input to help people with
upper-body motor impairments in ADLs.
3.1 Participants
We recruited 12 participants with upper-body motor impairments
to participate in our study (Table 2). Participants were recruited
through online platforms (e.g., Reddit, Twitter, Facebook) and snow-
ball sampling. To participate in our study, participants must be 18
years or older, have upper-body motor impairments, have expe-
riences with assistive technologies, and be able to communicate
in English. Among the 12 participants we recruited, three of them
were female, and nine were male (Table 2). They had an average
age of 31.6 (SD = 8.0). Four participants stated that they had spinal
cord injuries, four had cerebral palsy, one had stroke, one had pri-
mary lateral sclerosis, one had arthrogryposis multiplex congenita,
and one had muscular dystrophy. The study took around 75 to
90 minutes per participant. Participants were compensated with a
$20 Amazon gift card. The recruitment and study procedure was
approved by the Institutional Review Board (IRB).
3.2 Study Procedure
3.2.1 Demographic Background. In our semi-structured interviews,
we frst asked the demographic background of our participants (e.g.,
age, gender, descriptions of upper-body motor impairments).
3.2.2 Current Practices and Challenges of ADLs. We then asked
our participants about their current practices of performing certain
ADLs [51] (Figure 1) and associated challenges across diferent
ADLs (e.g., rely on PCAs, additional).
3.2.3 Input Modality Preferences. Afterwards, we introduced par-
ticipants with existing input modalities for people with upper-
body motor impairments from literature (e.g., head-movement in-
put [15, 55], eye-based input [19, 74], brain-computer interface
[16, 17, 46, 49]) (Table 1). To ensure participants understood the
various input modalities and to reduce bias, we created introduction
slides of each input modality by including fgures from existing
research (e.g., [74]) and commercially available products (e.g., [2]).
After introducing each input modality, we confrmed with partici-
pants to make sure they understood diferent input modalities and
associated applications. If they still had difculties understanding
the diferent input modalities, we provided video demonstrations to
participants for better understanding. We then asked participants
to describe how the diferent input modalities may beneft their ex-
periences (Table 1) completing each ADLs (Figure 1), and associated
reasons.
3.2.4 Multimodal Input Preferences. Finally, we asked participants
for their opinions and preferences on combining diferent input
modalities for each ADL (Figure 1) and associated reasons.
Understanding Input Modality Preferences of People with Upper-body Motor Impairments ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 2: Participants’ demographic information.

Participant Gender Age Upper-body Motor Impairments Details

P1 Female 41 Arthrogryposis multiplex congenita Range of motion and strength of arm and leg are limited. Cannot
lift arm or stretch.
P2 Female 23 Cerebral palsy Gross motor and fne motor difculties.
P3 Male 35 Spinal cord injury (C5) Paralyzed from shoulder down, no fnger fexion.
P4 Female 38 Spinal cord injury (C5) Some wrist function and no hand dexterity.
P5 Male 47 Muscular dystrophy Strength is limited and bicep is extremely weak, cannot lift the
arm without gravity.
P6 Male 27 Spinal cord injury (C4/C5) Wrist extension on one side, no fnger mobility, no tricep control,
no fne motor.
P7 Male 24 Cerebral palsy Paralyzed left arm.
P8 Male 33 Spinal cord injury (C5) Have use of bicep, no triceps, can do to mid way point of the
bicep, no sensation further down. No fne motor function on
either hand.
P9 Male 22 Stroke Right arm cannot go past 45 degrees.
P10 Male 34 Cerebral palsy Difculty moving wrist, hand, and cannot fex arm.

P11 Male 20 Cerebral palsy Floppy limbs due to cerebral palsy. Cannot use the left arm at
all. Bicep and tricep functionality are limited.
P12 Male 35 Primary lateral sclerosis Have difculty holding objects and moving around.

using a Miro board [29] to cluster the codes and identify emergent
themes.

4 FINDINGS: CURRENT PRACTICES AND

CHALLENGES OF ADLS BY PEOPLE WITH
UPPER-BODY MOTOR IMPAIRMENTS
In this section, we frst show existing practices of ADLs by our
participants (Section 4.1). We then present associated challenges in
ADLs from people with upper-body motor impairments (Section
4.2).

4.1 Practice of ADLs by People with

Figure 1: Activities of Daily Living [51]. (Bathing, Dress-
ing, Grooming, Oral Care, Toileting, Transferring, Moving
Around, Eating, Shopping, Cooking, Managing Medications,
Using the Phone, Housework, Laundry, Driving, Managing
Finances, Leisure and Other Activities)
3.3 Data Analysis
The semi-structured interviews were conducted through Zoom [27]
and all interviews were audio-recorded and transcribed. After the
interviews, two researchers independently performed open cod-
ing [14] on the transcripts. Then the coders met to discuss their
codes and resolve any conficts (e.g., missing codes, disagreement
on codes). After the two researchers reached a consensus and con-
solidated the list of codes, they performed afnity diagramming [25]
Upper-body Motor Impairments
In terms of current practices of ADLs, we learned that participants
rely heavily on Personal Care Assistant (PCA) or their fam-
ily members to help with diferent ADLs (e.g., dressing, bathing,
toileting, driving) (Table 3). Based on the responses to each activity
of daily living and statistical analysis, we found that about 67.2%
(SD = 29.7%) of participants require PCAs for each ADL on average
(Table 3). P3 commented on this situation:
“...Many of the activities still require my personal as-
sistant for help...tasks like doing my laundry, toileting,
and shopping...I have a very limited range of motion,
and current technologies are not there yet to support
me living independently...”
We also found that there is a high disparity between ADLs
that involves computing devices (e.g., managing fnances) and
basic ADLs in the reliance on PCAs (e.g., toileting, dressing).
Only 10% of our participants require other people for assistance
ASSETS ’22, October 23–26, 2022, Athens, Greece Li et al.

Table 3: The number of participants that leveraged technologies or PCAs for each ADL and associated sample tasks.

Activities of Number of Sample Tasks through Personal Assistance Number of Sample Tasks through Technology
Daily Living Participants Participants
Required Use
PCAs Technology
Bathing 10 Transfer between wheelchairs and shower 2 Turn on/of the water, control water tem-
bench, control water temperature, dry after a perature
shower, turn on/of the water
Dressing 11 Help with zipping, help with small buttons, feel 1 Use tablet to select clothes
the clothes to make sure it is fat or smooth
Grooming 8 Comb the hair, trim nails, shaving 6 Customized electric hair removing reduce
the required range of motion, automatic
hand-wave dispenser
Oral Care 9 Turn on/of electric toothbrush, cleaning 11 Electric toothbrush, electric fosser
Toileting 12 Transfer, fushing, cleaning 3 Automatic height adjusting hydraulics,
voice control fushing systems
Transferring 9 Transfer from wheelchair to bed 8 Seat elevation, joystick and remote control
to change into a comfortable position
Moving 3 Open/close doors 9 Diferent input modalities to control the
Around wheelchair moving direction and speed
Eating 11 Choose the food on the plate, lift utensils, cut 3 Omnibot to scoop and feed the person,
the food spoon with accelerometer that could level
itself
Shopping 9 Overhead reach of products, put food in bags, 7 Online shopping through accessible smart
carry food devices
Cooking 11 Slicing, getting hot plates of the stove, everyday 4 Voice control microwave, electric cooker,
cooking automatic peeler and cutter
Managing 8 Pill refll, picking up drugs 6 Pill organizer, reminders
Medication
Use the Phone 1 Plug in the charger 10 Smaller phone for touch with less range
of motion, use voice to reduce the need of
motion
Housework 10 Mobbing the foor, dish washing, vaccuming 7 Robot sweeper, roomba, voice-based light
control, TV control
Laundry 11 Folding clothes, general laundry 6 Customized dials and buttons to control
the washer, voice-assisted camera systems
to place commands and instructions
Driving 9 Change settings that are hard to reach, general 5 Hand control of A/C, touchscreen for radio,
driving
Managing 4 Accessing mailbox, write the information on a 11 Keep records on computers, camera to de-
Finances check, count cash posit checks, online banking
Leisure and 1 Horse riding 8 Play video games, walk dog, control TV
other activities
with using the phone and leisure activities (Table 3). However, “...Traditional daily activities or essential activities are
for activities like toileting, dressing, and cooking, over 90% of our usually restricted based on my motion capability and
participants still require other people for assistance (Table 3). P5 also the inaccessible environment. One example is
explained this: that my washer and dryer are in the basement, which
forced me to use PCAs for assistance...”
Understanding Input Modality Preferences of People with Upper-body Motor Impairments
Among all technologies that are being used to help people with
upper-body motor impairments in ADLs, we found that the majority
of existing input systems for traditional ADLs require addi-
tional efort for installation, adjustment, and modifcation
supports for individuals with upper extremity capabilities.
For example, P12 explained how his multimodal shower control
system was installed based on the mobility of his upper extremity:
“...My cousin helped me to use my voice to control
the temperature and added a big button on the left
side about my shower seat height to allow me to use
my less afected left hand to turn on or of the wa-
ter...He had to seal all wires, the button, and the mi-
crophone...He also needs to program and map my
preferred way of controlling the water...”
Furthermore, we realized that some of our participants leverage
both technology and PCAs to collaboratively accomplish cer-
tain ADLs. For example, P11 explained how he could use the elec-
tric toothbrush by himself, but needed someone to put toothpaste
on the brush and press the on/of button:
“...I have limited motion control of my arm, which
made me hard to press the on/of button on the elec-
tric toothbrush, and it is a tiresome process to put
toothpaste on the brush as well. But I could use the
limited motion control to brush my teeth after the
power is on. So I usually have my personal assistant
add toothpaste for me and turn on the toothbrush, so
she can focus on other things while I am brushing my
teeth...”
4.2 Challenges with ADLs by People with
Upper-body Motor Impairments
According to what we showed above, we found that there exist
unique practices for people with upper-body motor impairments
across diferent ADLs. Thus, we further show challenges of diferent
existing approaches toward ADLs and associated challenges. From
the interview, we learned that participants in our study rely heavily
on PCAs or their family members to help with diferent ADLs (e.g.,
dressing, bathing, toileting, driving). Despite the usefulness and
necessity of having PCAs for ADLs, they also mentioned limitations
and concerns of having other people assist with ADLs. For example,
P2 mentioned that always relying on PCA for help with ADLs
may afect her choice of time to do certain ADLs. P2 further
elaborated on this:
“...My PCA helped me with basically everything in my
daily activities. I appreciate everything. However, this
forced me to only be able to do certain activities while
my PCA was around. For example, toileting, bathing,
and dressing. It would be impossible if I wanted to
go out and visit my friend at a certain time without
having my PCA put my clothes on...”
Furthermore, P6 complained about the fnancial burden of
hiring personal assistants:
“...I am already low income, and hiring personal as-
sistants is a huge part of my monthly cost. If I can
do some tasks myself with my voice or eye, it would
ASSETS ’22, October 23–26, 2022, Athens, Greece
reduce the number of times that I need personal as-
sistants per month...”
We also learned that our participants have concerns about pri-
vacy while relying on other people for assistance in certain ADLs,
such as dressing, toileting, and bathing. P2 further commented:
“...Although having a personal assistant is the only
way to help me with some activities like toileting and
bathing, sometimes I feel embarrassed when they see
me using the bathroom or having a shower. Especially
when a new PCA comes...”
Moreover, we found that there also exists communication bar-
riers between PCAs and people with upper-body motor impair-
ments, such as language barriers (P2) and difculties in verbally
describing the detailed instructions if the person with upper-body
motor impairments cannot be physically in front of certain appli-
ances (P2, P4). P2 further commented on this:
“...If a PCA doesn’t know English well, it’s hard for me
to communicate with the PCA regarding my needs,
especially for doing the laundry where my washer
is in the basement that is not accessible...like putting
items that shrink separately, and putting bras in the
bra bags...”
Besides the concerns and challenges while working with PCAs,
fve of our participants also expressed concerns about the inability
to contribute to housework when their family members are
busy. P8 commented on his situation:
“...I live with my family. My sister and parents usually
help with various daily activities, such as doing laun-
dry and bathing. I know they are very busy with other
stuf too, and I really want to help them sometimes.
For example, I would like to cook them some meal
before they come back home, but existing technology
does not support me to do so...”
In terms of their current challenges of existing technologies, all of
the participants expressed limited input options they have obtained
and the willingness to try new input modalities. Specifcally, four
participants mentioned that existing input systems mostly rely on
a fxed single input modality for a particular task. They mentioned
the concerns of convenience, efort, and reachability for a
fxed input modality (P3, P6, P7, P9). For example, P7 explained
the limitation of using hand waving to control the A/C of his car
when he does not sit on the drivers’ seat:
“...I had my car customized by allowing me to use
hand-waving gestures to control my A/C while driv-
ing. However, this only works when I am in the dri-
ver’s seat. It becomes unreachable for me while I am
sitting in the passenger’s seat...”
Beyond the fxed input modality, three participants mentioned
concerns about the fxed mapping after installation between
certain input and specifc tasks, and it is almost impossible to modify
the mapping by themselves. P5 commented on her customized dial
pads to control the washer:
“...I had a technician install this dial pad to control my
washer a couple of years ago. At that time, I was able
to reach the top level of buttons to control the water
ASSETS ’22, October 23–26, 2022, Athens, Greece
level and the heat, but because of the muscle decline,
reaching the top row became hard for me last year,
and I did not know how to reprogram the pads...”
Compared with having personal assistance for ADLs, the use of
technology could potentially address concerns from participants
regarding privacy concerns, time and efort requirements from
PCAs, and maintain self-confdence in social interactions. Overall,
we showed the current limitations of existing technologies in ADLs
and found that all of our participants showed strong needs and
preferences for being able to use various input modalities in ADLs
towards independence. Therefore, it is important to uncover their
preferences among diferent input modalities in ADLs and how
these input modalities may help people with upper-body motor
impairments in accomplishing ADLs by maintaining independence.
5 FINDINGS: APPLICATIONS OF INDIVIDUAL
INPUT MODALITY AND MULTIMODAL
INPUT IN ADLS
In this section, we frst present preferred applications of individual
input modality in ADLs for people with upper-body motor impair-
ments (Section 5.1). We then show participants’ preferences on
applications of diferent combinations of input modalities (Section
5.2).
5.1 Applications of Individual Input Modality
In our interview, we asked participants whether they would like to
change their existing ways of doing ADLs and what input modali-
ties (Table 1) they would prefer to use for specifc ADLs (Fig. 1). On
average, almost all (11/12) participants mentioned that they would
like to change their existing method for ADLs (Table 3). Among
the participants who had already been using technologies in ADLs,
nearly all of them would like to change the technology involved by
having new input modalities. Only one participant did not want to
change the technology involved for toileting, transferring, house-
work, or leisure and other activities. For participants who do not
use technologies in certain ADLs, we found that all of them prefer
having new input modalities that enable them to accomplish these
ADLs independently or collaboratively, especially for the ADLs that
they may fully rely on PCA or family members, such as dressing,
bathing, toileting, and eating.
While we examined each ADL, it became clear that certain input
types are favored over others. Across all tasks, we found that touch
(e.g., joysticks, touchscreens) and voice inputs are highly desirable
compared with others (Table 4). On average, seven participants
prefer using touch or voice input as an input modality to substitute
their existing ways of handling ADLs. Specifcally, we found that
our participants mostly prefer using touch input for bathing (10),
toileting (8), and cooking (8). Furthermore, our participants would
also like to use voice input for cooking (10) and driving (10) (Table
4). For example, P8 mentioned that he would prefer using touch
input rather than the existing shower knob:
“...I usually have my dad or sister help with the bathing
process because I cannot rotate the shower knob due
to the lack of control with my fngers and hands. How-
ever, I can ‘tap’ with my palm, I can defnitely use the
Li et al.
touch interface to turn on/of the water or control the
water temperature...”
Beyond touch or voice input, we learned that hand-only ges-
ture (e.g., waving hands) is the third preferred input method by
our participants, which has about three participants prefer using
hand-only gestures as an efcient input method for each ADLs.
Specifcally, we found fve participants preferred hand-only ges-
tures for cooking, and four participants chose hand-only gestures
for grooming, toileting, and housework (Table 4). P3 commented
on his preferences of waving his hand to fush the toilet or control
cooking appliances:
“...I have spinal cord injuries, and it is hard for me to
push or press buttons with my fngers. I think waving
hands is a very compelling way for me to control or
interact with my devices. Especially for toileting, I
always have trouble pressing the button to fush the
toilet after use, it would be better to wave my hand
after use to fush it...For cooking, I also have trouble
using the knob to control the time and temperature.
Thus, using waving to control a scrolling panel would
be a great option for me...”
Eye-based input (e.g., eye gaze, eyelid gestures), head-movement
input, and brain-computer input all have the same number of
selections (2) by participants who prefer to use these methods for all
ADLs. By further analyzing the data quantitatively, we found that
participants highly prefer eye-based input (6) and head-movement
input (5) for shopping (Table 4). P10 explained her existing way of
shopping and why he prefers eye-based input:
“I do not often go shopping physically, I usually just
use online platforms such as Amazon, for most of the
things. The main barrier is that I cannot buy things
at a store by myself. If it is possible, I want to use
eye-based input to dwell at a product, and a robotic
arm can grab that product for me...”
Finally, we learned that our participants prefer to use facial ges-
tures, biometric input, and automatic recognition or other
input only for limited tasks, which ends up being the least selected
input modalities among all ADLs (1). For instance, we found partic-
ipants mostly prefer using biometric input for managing fnances,
because biometric information could be used for authentications.
In terms of automatic recognition or other input, seven participants
prefer it when in motion, P6 commented on the difculty of moving
physical obstacles (e.g., door):
“Opening my door is the hardest thing ever, I am
always in my wheelchair, and I do not have much
control of my upper body, which is not sufcient to
open my front door, which forced me to ask my family
members to open the door for me all the time. This is
why I want an automatic door that can open when I
approach...”
5.2 Applications of Multiple Modality Inputs
We further identify participants’ preferences for applications of
diferent combinations of input modalities. The combination means
Understanding Input Modality Preferences of People with Upper-body Motor Impairments ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 4: Input modality preferences for specifc ADLs among all 12 participants.

Touch Input Hand-only Voice Input Eye-based Head-movement Brain-computer Facial Gestures Biometrics Others
Gestures Input Input Input
Bathing 10 3 9 2 3 3 5 0 0
Dressing 4 3 8 2 2 0 2 0 1
Grooming 7 4 6 3 3 1 3 1 2
Oral Care 5 2 2 3 3 2 4 2 1
Toileting 8 4 6 0 3 2 3 0 2
Transferring 5 2 5 1 2 1 1 0 1
Moving Around 6 2 9 1 3 2 0 6 7
Eating 7 2 6 4 2 2 0 0 1
Shopping 8 3 7 6 5 3 0 0 0
Cooking 8 5 10 2 1 1 0 0 1
Managing Medications 7 1 7 2 0 3 0 1 2
Uses the Phone 7 3 8 5 3 2 1 4 0
Housework 6 4 9 3 1 2 0 0 0
Laundry 8 2 6 0 0 2 0 1 1
Driving 8 2 10 2 3 1 1 2 0
Managing Finances 6 0 5 2 1 0 0 8 1
Leisure and Other Activities 8 2 6 3 3 4 0 0 0

participants either prefer using several input methods for redun-
dancy purposes or use these input methods to accomplish diferent
sections of a specifc task. For example, people may prefer using
both touch input and voice-based input for switching the water
temperature during showering. On the other side, people may have
eye-based input to target an object and use voice-based input to
open or close the object.
In our paper, if a certain combination of input modalities got
selected more than fve times by our participants, we defne it as a
popular combination. Based on participants’ responses, we found
that there is a majority popular combination (selection > 5 among
all participants) between touch and voice input for the majority
of ADLs, which include bathing, grooming, toileting, eating,
cooking, managing medications, use the phone, housework,
laundry, driving, and leisure and other activities (Table 4). We
found moving around has the most number of input modalities
in popular combinations (selection > 5 among all participants),
which includes touch input, voice-based input, biometric input,
and automatic recognition or other input. From what we uncov-
ered in the interviewees’ responses, we found moving around
usually involves more complex environments. Having multiple
input modalities would allow people to easier accommodate the
complexity of such interactions. We also realized people prefer
combining touch input, voice-based input, and eye-based input (se-
lection > 5 among all participants) for shopping specifcally (Table
4). Furthermore, we found that managing fnances has a popular
combination (selection > 5 among all participants) between touch
input and biometric input.
6 FINDINGS: CHOOSING INPUT
ALTERNATIVES FOR PERFORMING ADLS
We showed preferred applications of both individual input modal-
ity and multimodal input in the previous section. In this section,
we frst show the reasons for choosing individual input modality
alternatives in ADLs (Section 6.1). We then present the reasons for
choosing multimodal input alternatives (Section 6.2).
6.1 Reasons for Individual Input Modality
Alternatives
In the interview, we asked our participants why they would prefer
one input modality over the other based on their responses to input
preferences that we showed in the prior section. We uncovered
fve main factors (i.e., usability, efciency, consequences, personal-
ization, and context) that may afect how people with upper-body
motor impairments choose a specifc input modality, and we present
each of them in detail.
Usability: We learned that participants highly value reliability
and confdence while choosing a specifc input modality (P6, P8,
P10, P11). For example, P10 mentioned that he prefers touch input
over voice-based input for some ADLs because he knows what
the outcome of the touch interaction will be, and it makes him
become more confdent with providing reliable output. P11 further
explained the involvement of biometric information in driving to
ensure reliability and reduce false positives:
“...I can only use my right side of the body to drive.
This makes driving a hard task for me because there
are so many diferent functions, and I do not have
control of my left hand or foot, which causes the huge
concern of accidentally touching and safety issues.
That is why I prefer to use fngerprint as a verifcation
approach to make sure I do not accidentally touch
somewhere and prevent false activation...”
Another important factor for choosing an input modality is in-
put precision, which indicates how precise the user can place the
command with certain input modalities (P4, P8, P9). For instance,
P9 mentioned that he prefers using touch input for some activities
because it can make precise commands. P4 further commented on
why she likes using voice or joystick compared with hand-only
gestures for certain ADLs:
“...For tasks that need you to set up a specifc time or
temperature, using my voice or joystick would be eas-
ier. You can simply say, ‘turn the water temperature
ASSETS ’22, October 23–26, 2022, Athens, Greece
to 39 degrees,’ but it would be very difcult if you use
hand gestures to indicate such information...”
Furthermore, we found that learnability and intuitiveness
afect how people with upper-body motor impairments choose to
use a certain input modality for ADLs (P4, P9, P10). This means
that they prefer the input modality have a lower learning curve
(P10). P9 further compared voice-based input with hand gestures
in new skill training eforts:
“...Voice is more straightforward to me. If I need to
place commands for my kitchen appliances, I have
to learn diferent hand gestures that map to various
commands. However, using voice would not require
me to spend time to learn and remember certain com-
mands...”
Similar to learnability, we also uncovered that our participants
highly value the importance of adaptability and compatibility,
which means using a single input modality on a specifc system
should be able to interact with diferent devices for various purposes
(P8, P9, P10, P12). P10 commented on how he likes to use the same
touchscreen system to both control the kitchen appliances at home
and help with navigation in grocery stores:
“...I hate switching input devices for diferent purposes
due to the difculty of switching the hardware and
adapting new methods based on mobility conditions.
Therefore, I would like to control or place commands
from a single unit with a single method, such as hav-
ing my touchscreen connect to my instant pot while
cooking and also accessibility maps of the grocery
store for navigation. This compatibility will reduce
the efort of changing devices for diferent purposes...”
Efciency: From the interview, we found that our participants
mentioned the necessity of ensuring efciency for a specifc input
modality. First of all, response time is a key factor for certain input
modalities to be used for some specifc ADLs (e.g., temperature
change in showering) (P4, P6, P7, P8, P11). P6 further elaborated
on the importance of quickly changing temperature while taking a
shower for people with spinal cord injury:
“People with spinal cord injuries care about skin tem-
perature a lot because of the lack of sensation. I prefer
hand gestures over voice while changing the water
temperature, because using voice can take a long time,
including activation, placing specifc commands, and
waiting for the responses from the system. I might
get burned with hot water by that time.”
Moreover, some people with upper-body motor impairments
(e.g., spinal cord injury, cerebral palsy) have a hard time making
movements both for the upper limbs and also their lower limbs
as well. Therefore, they mentioned the importance of less phys-
ical efort (P3, P4, P5, P6, P9). P3 explained the necessity of less
movement required for certain ADLs:
“...I have a C5 level of spinal cord injury and have to
stay in my wheelchair all the time. Currently, most of
the devices now require me to physically move close
to them and touch them to place commands. This situ-
ation is frustrated because some spaces are too narrow
Li et al.
for me to move around and take turns. My bathroom
is a good example, that is why I would prefer to use
eye-based input or brain-computer input as an input
compared with touch input for these interactions that
requires me to move around...”
Besides requiring physical efort, we learned that our participants
also mentioned the preferences of simpler level of interactions
according to the mobility conditions when performing the input
(P1, P4, P8). P8 commented on this:
“...Compared with diferent hand gestures, I like sim-
ple input methods, just like eye blinking or voice. I
might not be able to perform certain gestures, and it
will end up with poor performance...”
Consequences: Our participants mentioned the concerns of
various consequences after using certain input modalities, which
might change their minds about continuing with that specifc input
method. First, both P6 and P9 commented on the importance of
mess prevention while using certain input modalities (e.g., touch)
in ADLs (e.g., bathing, cooking). P6 explained his concerns about
using touch interfaces, but rather hand gestures or voice while
having a shower:
“...I like using touch screens because they can provide
me with accurate output. However, it can also be prob-
lematic when the touch interfaces already have water
on them, or my hand has shampoos. For this situation,
I will just use hand gestures or voice input in a better
way without creating a mess in the bathroom...”
Next, we also learned that safety is an important factor for
people with upper-body motor impairments when choosing input
modalities for ADLs (P1, P3). For example, P3 mentioned the im-
portance of having hands-free interactions because he only has
control of the arm and no sensations of the hand or fnger. P3 further
commented on this:
“I cannot tell if my fnger gets burned or not due to the
loss of sensation from spinal cord injury, so I would
prefer not using touch input while cooking or boiling
water.”
Beyond safety concerns, our participants also mentioned the
importance of confdentiality and security for input modalities
(P1, P6, P8, P11). P8 mentioned the concerns of confdentiality while
checking out as a person with upper-body motor impairments and
why he prefers biometric input for payment:
“...I often have a hard time pulling out my credit card
and paying at the cashier. Not just that, due to the
difculty of moving my upper body, it can take a long
time to fnish the payment process, and other people
behind me may see my personal belongings in my
wallet, which can be dangerous to me. That is why I
think biometric input could reduce the consequences
of breaching my personal identities...”
Personalization: Participants in our study had diferent motor
impairments afecting their bodies in diferent ways, including their
upper and lower limbs. To account for these diferences, personal-
ized input modalities are preferred when selecting a specifc input
Understanding Input Modality Preferences of People with Upper-body Motor Impairments
method. For example, P7 and P9 mentioned the importance of tak-
ing physical ability and mobility into consideration. P7 further
explained:
“...The limited mobility of my hand afects my choices
of input methods, which means I would prefer the
input method that can take my personal situations
into consideration. This is the reason why I prefer eye
movement over touch...”
Furthermore, we learned that familiarity is another factor in
choosing personalized input modalities (P1, P5, P9). We found that
our participants preferred certain input modalities because they
had already used them for other ADLs. P1 further elaborated on
her existing experiences with the joystick, which made her prefer
touch input:
“...Touch input would be good for me to do other tasks
if the technology is available. Because I am already
familiar with diferent ways of using joystick, touch-
screen, and trackball. I have been driving with joystick
for many years...”
Similar to what we have mentioned previously (Section 4.2), our
participants prefer using input modalities that can enable indepen-
dence for specifc ADLs based on individual circumstances (P2, P4).
P2 further commented:
“...I prefer input methods that can ensure indepen-
dence when I am taking a shower, eating, toileting
or dressing. More specifcally, I do not have to wait
for my PCA all the time, I can then eat whenever I
want...”
Context: We identifed that the context of specifc environments
also afects how people perceive diferent input modalities. For
instance, six participants mentioned that environmental infu-
ences afect how they choose a specifc input method. The major
consideration was using voice as input while the surrounding envi-
ronment is noisy, which may afect the accuracy. P3 commented
on his preferences during gaming:
“...My body limits me from using touch input for aim-
ing in playing FPS games...I do not like voice input
when playing games because either the game sound
is noisy and the recognition system cannot detect my
voice correctly, or I have to use my voice to communi-
cate with my teammates from the game. Therefore, I
would go a lot with eye-based input and simple touch
commands as a confrmation for shooting...”
In addition, P6 further mentioned the importance of reachabil-
ity in a space by using certain input modalities, which includes
considerations of both the space and scalability:
“...Some input methods require certain space for in-
teractions, such as eye tracking, head tracking, and
brain-computer interfaces. I do not have a huge space
at home. This brings the concerns of whether I should
use a certain input for daily tasks...”
Finally, participants also expressed concerns regarding the social
acceptability of using certain input modalities during interactions
with or around other people (P5, P6, P8, P9). Three of them explicitly
mentioned that using voice input in a store or during shopping
ASSETS ’22, October 23–26, 2022, Athens, Greece
can be embarrassing and attract unwanted attention. P6 further
explained this:
“...It is less socially acceptable to use voice input at a
store, and I feel it is weird to have someone see me
yelling at my wheelchair or a specifc product in a
store. That is why I would prefer to use eye-based or
touch interfaces that have gentle reactions from other
people...”
6.2 Reasons for Multimodal Input Alternatives
We asked them about their preferences for combining various input
modalities together. We illustrate reasons for two main uses of mul-
tiple modality input: multi-modality for input redundancy (Section
6.2.1) and multi-modality for input variability (Section 6.2.2).
6.2.1 Multi-modality for Input Redundancy. Based on our partici-
pants’ responses about why they prefer having multimodal input
for redundancy, we uncovered four main factors (i.e., reliability,
convenience, customization, appropriateness) that afect how and
why our participants choose multiple input modalities for the same
purpose vs. just having a single input modality.
Reliability: we learned that our participants prefer having re-
dundant input methods for the same purposes to make sure they
can accomplish tasks in a reliable way. The frst reason our partici-
pants mentioned is that having a backup input modality could
enable people with upper-body motor impairments to accomplish
certain tasks disregarding the physical and mobility limitations (P2,
P4, P5, P8). P5 further commented on this:
“...I have muscular dystrophy, I continuously lose my
muscles every day. On the one hand, having multiple
input methods for the same purpose could allow me
to accommodate my muscle decline as time passes.
On the other hand, having two or three options could
allow me to fnish the tasks with the most suitable way
of input based on my physical situation. For example,
I could use voice to turn on the water remotely if I
want to have a bath while I am in my wheelchair, this
just gives me more options for specifc tasks...”
Another reason is that having multiple inputs for redundancy
could ensure task completion through environmental changes
(P1, P2, P4). We mentioned this in Section 6.1 as the reason why they
prefer certain input modalities. P4 further commented on having
redundant input while taking a shower:
“...The beneft of having multiple inputs could beneft
the showering experiences for sure. Because the envi-
ronment could be very noisy at that time, if I only used
voice, it might not work nicely. That is why having
touch input as an alternative is a great option...”
Beyond always having a backup and ensuring task completion
through environmental changes, we found that our participants
consider redundant inputs as having better accuracy and preci-
sion for various tasks (P4, P10). P4 further commented on this:
“...Diferent input methods usually have diferent pre-
cision and accuracy. Take showering as an example, I
would like to have both voice control and joystick as
the input methods to control the water. However, a
ASSETS ’22, October 23–26, 2022, Athens, Greece
joystick is more in control and more precise than just
using voice...”
Convenience: we found that fve participants claimed that they
like redundant input mainly for convenience. First, P4, P6, and P12
commented on the importance of maintaining multi-purposing
during complex situations, which allows them to choose their pre-
ferred input method freely. For example, P4 mentioned her experi-
ences of having two or more things at a time:
“...I sometimes have multiple things happening at the
same time, such as I may have to answer the phone
while I am driving through touch. Therefore, having
multiple input methods for diferent tasks ofers me
great fexibility in daily activities...”
Moreover, we found redundant input ofers better reachability
for participants with upper-body motor impairments due to mo-
bility issues. It allows people to control their devices at a certain
distance with comfort. P1 commented on her preference of combin-
ing the joystick with the voice to control the seat elevation while
transferring:
“...I use seat elevation to do some of the transfers,
such as transferring from my wheelchair to my bed.
After I raise my wheelchair, I need someone else to
put the wheelchair back down for transferring back
because I can only control the seat elevation through
the joystick on my wheelchair. If I can use voice to
control my wheelchair, I won’t need someone else for
assistance in transferring...”
Customization: we learned that our participants preferred hav-
ing redundant input for customized input opportunities to com-
promise their own mobility conditions. P6 commented on her
situation of only being able to perform a portion of gestures through
her hands even though she prefers using her hands for practice and
exercises:
“...Having more input options allows me to choose
the way that I feel most comfortable with based on
my body conditions. For example, although I can only
move my elbow to move my hand to tap on touch
screens, which limits the input I can do with my hands,
I still want to use my hand for practice because my
doctor told me to practice my hands often. That is
why I need more than one input method for tasks...”
Furthermore, we found that our participants also like to use mul-
tiple input methods to cope with inaccessible environments.
This indicates that having more input methods could allow peo-
ple to access technology or interact with their devices in a less
accessible environment. P11 further commented:
“...It takes efort to modify the home layout, I some-
times stay at my parents’ or my sister’s houses. Some
of their bathrooms or stairs are not that accessible
to me. A good system should have more than one
input method that allows me to interact with it under
complex environments...”
Appropriateness: by analyzing participants’ responses, we found
that having multiple input modalities reduces unwanted atten-
tion during social interactions (P3, P4). P4 commented on her
Li et al.
feedback of using multiple input methods to control a robot to help
her eat:
“...Having multiple input methods allow me to prevent
awkward times, such as I do not want to talk with my
devices while I am having dinner with someone...”
In addition to reducing unwanted attention, we also learned
that having redundant input modalities could potentially increase
self-confdence of people with upper-body motor impairments in
social interactions. P10 commented on this:
“...Having multiple ways of controlling my devices is
really cool! I could show of to my friends not only I
can fnish tasks independently, but also I could inter-
act with my devices ‘intelligently’...”
6.2.2 Multi-modality for Input Variability. During the interview,
we also asked our participants about the perceived benefts and
concerns of having multiple inputs through task completion, which
means combining diferent input modalities to accomplish various
sub-tasks of a complete task. We uncovered three main factors and
the usefulness of having varied inputs throughout task completion
(i.e., usability, efciency, personalization).
Usability: using multiple input modalities for sub-tasks enables
participants to have a natural mapping between input modali-
ties and the actual meaning of the interface for complemen-
tary purposes (P4, P5). P4 explained why she likes to use varied
inputs for diferent sub-tasks:
“...Voice control is really bad with placing commands
which require multiple steps. Therefore, all I want to
control my Roomba in the future is just saying ‘start
cleaning Roomba,’ and if I need to select specifc places
to clean, I would just either use touch or eye-based
input to make a more precise command...”
Our participants mentioned that they prefer having multiple
input modalities for sub-tasks to prevent false activation and
provide a more efective activation approach. P3 further elab-
orated on why she prefers to use hand waving as an activation
approach for voice-based systems:
“...If I have a smart bathroom with all the input meth-
ods you just showed, I would like to use voice because
I can control it remotely. However, I would like to use
hand waving or eye-based gestures as an activation
approach, because I found that the current voice in-
put accuracy and precision are not sufcient, it might
falsely activate through conversations...”
Having multiple input modalities to assist people with upper-
body motor impairments for sub-tasks allows additional input
dimensions for various tasks, because certain input modality does
not have sufcient input dimensions, or people with upper-body
motor impairments are restricted to using a subset of them (P1, P2,
P3, P7, P12). P8 commented on his limited options in gaming:
“...I spend about ten hours in front of my computer,
gaming becomes an important leisure activity for me.
However, I can only play chess by using the light
pointer controlled by my head. It would be great if I
could have more input methods that could support
me to play RPG or FPS games...”
Understanding Input Modality Preferences of People with Upper-body Motor Impairments
Efciency: besides providing additional input dimensions, we
learned that using multiple inputs could reduce the efort on each
single input modality, thus reducing the complexity of learning
eforts. P8 further commented on this:
“...I do not have full control of my fngers, I can def-
initely use touch to complete tasks, like document
editing, but it would be really slow and can easily
make me feel tired. Especially for complex tasks, that
is why I prefer combining eye, head, touch, and brain
control to form a hybrid control system...”
By reducing the efort of a single input modality, we further
learned that it would make people with upper-body motor im-
pairments feel more comfortable (P5, P9). P9 elaborated on the
reason for extra comfort:
“...Let me use gaming as an example, I have to hold
a certain body posture while playing the game just
by using the touch input. Having more input vari-
eties allow me to relax and change postures during
gaming...”
Beyond reducing the efort of a single input modality, P10 and
P11 also mentioned the benefts of reducing the time cost by
combining multiple input modalities. P10 further explained:
“...Having multiple input methods on small tasks could
allow me to accomplish the task faster than just using
a single input. For example, I can use head or eye
gestures to control where I want to shoot and then
use a touch button to shoot. It would be a lot faster
than I use a joystick to aim and shoot...”
Privacy and Security: we mentioned in Section 6.1 that our
participants have privacy and confdentiality concerns about using
existing methods in ADLs (e.g., shopping). Our participants men-
tioned the use of additional input for verifcation and security
purposes. P1 commented on combining biometric information
with a joystick for security purposes:
“...My current power wheelchair does not have a good
authentication process, which means anyone can eas-
ily drive it by using the built-in joystick. That is why
I prefer future power wheelchairs to have biometric
verifcation for ease of use and security...”
Finally, we also learned that having multiple input modalities
for diferent purposes ensures and extends the independence
in social activities. P10 commented on this:
“...A combination of multiple input methods for dif-
ferent small tasks could help me to become more in-
dependent while moving outside. For example, using
a joystick allows me to move my wheelchair, using
biometric information allows me to lock my door, and
using voice can allow me to answer phone calls while
my hand is occupied by controlling the joysticks. This
allows me to have a complete trip independently...”
ASSETS ’22, October 23–26, 2022, Athens, Greece
6.3 Summary of Individual versus Multimodal
Input Approaches
In this section, we presented specifc reasons for choosing individ-
ual input modality (Section 6.1) and multimodal input alternatives
(Section 6.2) for helping people with upper-body motor impair-
ments in ADLs. By comparing both reasons for individual and
multimodal input alternatives, we found there exist similarities,
such as design for reliability, reachability, and efciency. The use
of multimodal input could further extend these individual input
preferences either through input redundancy (e.g., reliability, reach-
ability) or variability (e.g., efciency, privacy). On the other hand,
we also found that multimodal input also brings opportunities that
complement individual input modality. For instance, multimodal
input could enhance self-confdence in social interactions for input
redundancy and prevent false activation through input variability.
Overall, we present how multimodal inputs provide opportunities
through upper-extremity mobility limitations and adapting the com-
plexity of contexts (Section 6.2). We should also consider existing
concerns that might also be applicable to certain multimodal inputs,
such as fnancial burden and additional efort of installation and
mapping (Section 4).
7 DISCUSSION
In the previous section, we presented the overall practices and
challenges of diferent approaches in ADLs (Section 4), preferred
applications of individual input modality and multimodal input (Sec-
tion 5), and associated reasons for choosing certain individual input
modality and multimodal input alternatives (Section 6). Refecting
on these fndings, we will discuss future research opportunities
and design recommendations for more accessible applications of
multimodal input systems.
7.1 Multimodal Input Towards Collaborative
Experiences in ADLs
In our fndings, we showed that our participants already adopted
the experiences of co-accomplishing certain ADLs with their PCAs.
For instance, P11 mentioned that he used the electric toothbrush
by himself, but needed a PCA to put toothpaste on the brush and
press the on/of button (Section 4.1). Understanding the prefer-
ences for multimodal input may reduce the efort and time cost
of PCAs for assisting people with upper-body motor impairments
through ADLs. Furthermore, our fndings of the needs and pref-
erences of collaboratively accomplishing ADLs may further bring
more opportunities for enhancing collaborative experiences [9]
through multimodal input systems. For example, fve of our par-
ticipants expressed the desire to contribute to housework when
their family members are busy (Section 4.2), such as preparing meat
and vegetables before their family members come back home and
cook (P8). Thus, our paper proposes the following questions for
future research to consider while leveraging multimodal input to
support the collaborative experiences in ADLs: 1) how to support
communications and interactions between PCAs and multimodal
input systems? 2) how to set up collaborative tasks based on upper-
extremity capabilities and diferent ADLs? 3) how to enable certain
multimodal input systems to adapt to diferent collaborative tasks
among ADLs with PCAs?
ASSETS ’22, October 23–26, 2022, Athens, Greece
7.2 Diferentiate Computing Devices with
Sensing Systems for Traditional ADLs
From the fndings, we recognized the disparity between technology
adoption and reliance on PCAs across computing-based activities
(e.g., managing fnances) and traditional ADLs (e.g., toileting). Prior
research uncovered the practices of multimodal input on comput-
ing devices (e.g., gaming context) [70] and design considerations
of sensing systems [28]. The fndings from our paper uncovered
the demand for taking consideration of consequences and context
of use while developing multimodal input systems (e.g., safety, pri-
vacy). For example, input systems for computers would require
more modifcations to be able to adapt to the environment in the
bathroom or for cooking scenarios [37], such as location, water
resistant, and surrounding objects.
From our results, we showed that our participants prefer having
various input options for diferent ADLs (Section 4.2). This would
likely lead to increased efort to install sensing units for various
devices when used in the home context [28]. However, home envi-
ronments may have unique layouts, spaces, and requirements for
installment, which require designers to provide more customized so-
lutions. Furthermore, participants expressed concerns regarding the
maintenance and replacement of assistive technologies. Therefore,
to reduce the efort required to exchange or repair devices and sys-
tems, it is important to explore ultra-low power or self-sustaining
solutions (e.g., [66]).
7.3 Consider Actuation and Human-Robot
Interaction (HRI) during Multimodal
Interactions
We also found that the signifcance of input modality can depend
on actuation techniques when it comes to physical systems. In the
context of ADLs, actuation techniques [75] include a wide spec-
trum of mechanisms that change the physical world, ranging from
interacting with the environment, automatic doors, to connected
appliances and service robots [73]. Working together with input
modalities, these actuation techniques close the loop of interaction
paradigms, examples of which can be found in P12’s case of the
voice-controlled shower system, P10’s robotic arm, P6’s automatic
door, P8’s omnibot for eating and P4’s Roomba in our study (Ta-
ble 3). Based on our fndings on challenges of existing practices
(Section 4.2), leveraging actuation and robots could further support
independence and reduce concerns of privacy and fnancial eforts.
Especially, having robots could potentially beneft traditional ADLs
(e.g., toileting, showering, and cooking) and reduce the reliance
on upper-extremity mobility. Moreover, existing HRI research also
explored how users could leverage multimodal input to interact
with robots for complex tasks [1, 43, 52].
We propose the following research directions for HRI and sup-
porting people with upper-body motor impairments in ADLs. First,
existing research in HRI has explored how to support robot learning
from multiple modalities [43]. Future research could further com-
bine our fndings of preferred input modalities with robot learning
research to support robot learning from customized multiple modal-
ities by people with upper-body motor impairments. Second, end-
user robot programming is also important through the interaction
with robots, which enables end-users to overcome the complexities
Li et al.
of specifying robot motions [1]. The support of end-user robot pro-
gramming for people with upper-body motor impairments would
reduce the efort for users with upper-body motor impairments
and further support the robot for complex ADLs. Third, we showed
the existing high reliance on PCAs for ADLs in general. Extending
from robot learning through multiple modalities [43], supporting
active learning of robots on PCAs would strongly reduce the efort
from end-users. Finally, we mentioned that some of our participants
prefer working collaboratively on ADLs with PCAs or their fam-
ily members. Thus, robots may become a medium to support the
collaborative experiences between people with upper-body motor
impairments in social interactions.
7.4 Personalize Multimodal Designs Based on
Socially Categorized ADLs
We learned that people with upper-body motor impairments have
diferent preferences for input modalities depending on the social
context in which they will be used (Section 6.1), which correlates to
prior research which proposed designing assistive technologies for
social interactions [38, 59]. For example, our participants mentioned
the necessity of combining biometric input with joystick input for
security purposes in public. Thus, future designers may addition-
ally consider categorizing daily tasks with more social dimensions.
Based on our fndings, we ofer three examples of sub-categories
that may be considered based on the relative social expectations
in each. One might be group tasks by location and consider which
inputs are more appropriate depending on whether a location is
private (e.g., a bathroom) or public (e.g., a restaurant). A second
category might be based on the relative expectations of particu-
lar events or activities. For example, bathing is a private activity
that has relatively low expectations of social interactions, while
shopping is often a completely public activity with high expecta-
tions of social connections (albeit direct or indirect contact). The
third dimension we ofer is based on the sensitivity or expectation
of privacy for an individual in a given scenario. For this example,
one might consider how a wheelchair user may again feel that the
expectation of privacy in highly social activities like shopping is
conversely low, whereas managing one’s fnances, no matter the
environment or occasion, carries high expectations or desire for
privacy. We ofer these few examples as a starting point for future
researchers and developers to consider when designing multimodal
systems for use in diferent contexts.
8 LIMITATION AND FUTURE WORK
In our study, we chose to interview people with upper-body mo-
tor impairments to understand all three research questions (RQ1 -
RQ3). Although we are confdent about our current contributions
to the HCI and Accessibility community, there might be more op-
portunities by conducting contextual inquiries to further explore
the detailed interactions in-depth. Beyond showing participants
with diferent input modalities, future research could leverage tech-
nology probes to actually deploy in the living environments [35]
of people with upper-body motor impairments to uncover more
specifc designs of multimodal input systems for ADLs.
Understanding Input Modality Preferences of People with Upper-body Motor Impairments
9 CONCLUSION
In this paper, we describe the results of an interview study involv-
ing 12 people with upper-body motor impairments, which aimed to
understand their current and potential future use of emerging input
techniques for ADLs. We highlight the signifcance of incorporating
new input modalities to potentially decrease reliance on PCAs and
increase opportunities for independence. We assert that by under-
standing these opportunities based on the social- and task-based
preferences of people with upper-body motor impairments, future
research and development eforts can better utilize diferent input
modalities in ADLs (Section 6). Overall, we believe our fndings
contribute opportunities to support end-users’ ability to choose
how technology can best adapt to their unique preferences, abilities,
and goals for independent and collaborative achievements of ADLs.
REFERENCES
[1] Gopika Ajaykumar and Chien-Ming Huang. [n.d.]. Multimodal Robot Program-
ming by Demonstration: A Preliminary Exploration. ([n. d.]).
[2] Apple. 2021. Use Dictation on your iPhone, iPad, or iPod touch - Apple Support.
https://support.apple.com/en-us/HT208343. (Accessed on 03/27/2022).
[3] Rúbia EO Schultz Ascari, Roberto Pereira, and Luciano Silva. 2020. Computer
Vision-based Methodology to Improve Interaction for People with Motor and
Speech Impairment. ACM Transactions on Accessible Computing (TACCESS) 13, 4
(2020), 1–33.
[4] Rúbia EO Schultz Ascari, Luciano Silva, and Roberto Pereira. 2020. Personalized
gestural interaction applied in a gesture interactive game-based approach for
people with disabilities. In Proceedings of the 25th International Conference on
Intelligent User Interfaces. 100–110.
[5] Tommaso Lisini Baldi, Giovanni Spagnoletti, Mihai Dragusanu, and Domenico
Prattichizzo. 2017. Design of a wearable interface for lightweight robotic arm for
people with mobility impairments. In 2017 International Conference on Rehabili-
tation Robotics (ICORR). IEEE, 1567–1573.
[6] Jef Bilmes, Xiao Li, Jonathan Malkin, Kelley Kilanski, Richard Wright, Katrin
Kirchhof, Amarnag Subramanya, Susumu Harada, James Landay, Patricia Dow-
den, et al. 2005. The Vocal Joystick: A voice-based human-computer interface for
individuals with motor impairments. In Proceedings of Human Language Tech-
nology Conference and Conference on Empirical Methods in Natural Language
Processing. 995–1002.
[7] Pradipta Biswas and Patrick Langdon. 2012. Developing multimodal adapta-
tion algorithm for mobility impaired users by evaluating their hand strength.
International Journal of Human-Computer Interaction 28, 9 (2012), 576–596.
[8] G Bourhis and P Pino. 1996. Mobile robotics and mobility assistance for people
with motor impairments: Rational justifcation for the VAHM project. IEEE
transactions on Rehabilitation Engineering 4, 1 (1996), 7–12.
[9] Stacy M Branham and Shaun K Kane. 2015. Collaborative accessibility: How
blind and sighted companions co-create accessible home spaces. In Proceedings
of the 33rd Annual ACM Conference on Human Factors in Computing Systems.
2373–2382.
[10] Patrick Carrington, Jian-Ming Chang, Kevin Chang, Catherine Hornback, Amy
Hurst, and Shaun K Kane. 2016. The gest-rest family: Exploring input possibilities
for wheelchair armrests. ACM Transactions on Accessible Computing (TACCESS)
8, 3 (2016), 1–24.
[11] Patrick Carrington, Amy Hurst, and Shaun K Kane. 2014. The gest-rest: a pressure-
sensitive chairable input pad for power wheelchair armrests. In Proceedings of
the 16th international ACM SIGACCESS conference on Computers & accessibility.
201–208.
[12] Patrick Carrington, Amy Hurst, and Shaun K Kane. 2014. Wearables and
chairables: inclusive design of mobile input and output techniques for power
wheelchair users. In Proceedings of the SIGCHI Conference on human factors in
computing systems. 3103–3112.
[13] Patrick Alexander Carrington. 2017. Chairable Computing. University of Mary-
land, Baltimore County.
[14] Kathy Charmaz. 2006. Constructing grounded theory: A practical guide through
qualitative analysis. sage.
[15] Muratcan Cicek, Ankit Dave, Wenxin Feng, Michael Xuelin Huang, Julia Kather-
ine Haines, and Jefry Nichols. 2020. Designing and Evaluating Head-Based
Pointing on Smartphones for People with Motor Impairments. In The 22nd Inter-
national ACM SIGACCESS Conference on Computers and Accessibility. 1–12.
[16] Alban Duprès, François Cabestaing, José Rouillard, Vincent Tifreau, and Charles
Pradeau. 2019. Toward a hybrid brain-machine interface for palliating motor
handicap with Duchenne muscular dystrophy: A case report. Annals of physical
ASSETS ’22, October 23–26, 2022, Athens, Greece
and rehabilitation medicine 62, 5 (2019), 379–381.
[17] Alban Duprès, José Rouillard, and François Cabestaing. 2014. Hybrid BCI for
palliation of severe motor disability. In Proceedings of the 26th Conference on
l’Interaction Homme-Machine. 171–176.
[18] Peter F Edemekong, Deb L Bomgaars, Sukesh Sukumaran, and Shoshana B Levy.
2021. Activities of daily living. In StatPearls [internet]. StatPearls Publishing.
[19] Mingming Fan, Zhen Li, and Franklin Mingzhe Li. 2020. Eyelid Gestures on
Mobile Devices for People with Motor Impairments. In The 22nd International
ACM SIGACCESS Conference on Computers and Accessibility. 1–8.
[20] Mingming Fan, Zhen Li, and Franklin Mingzhe Li. 2021. Eyelid gestures for
people with motor impairments. Commun. ACM 65, 1 (2021), 108–115.
[21] Sergio García-Vergara, Yu-Ping Chen, and Ayanna M Howard. 2013. Super pop
VR TM: An adaptable virtual reality game for upper-body rehabilitation. In
international conference on virtual, augmented and mixed reality. Springer, 40–49.
[22] Robbert J Gobbens. 2018. Associations of ADL and IADL disability with physical
and mental dimensions of quality of life in people aged 75 years and older. PeerJ
6 (2018), e5425.
[23] Harkishan Singh Grewal, Aaron Matthews, Richard Tea, Ved Contractor, and
Kiran George. 2018. Sip-and-Puf Autonomous Wheelchair for Individuals with
Severe Disabilities. In 2018 9th IEEE Annual Ubiquitous Computing, Electronics &
Mobile Communication Conference (UEMCON). IEEE, 705–710.
[24] Susumu Harada, Jacob O Wobbrock, and James A Landay. 2007. Voicedraw: a
hands-free voice-driven drawing application for people with motor impairments.
In Proceedings of the 9th international ACM SIGACCESS conference on Computers
and accessibility. 27–34.
[25] Rex Hartson and Pardha S Pyla. 2012. The UX Book: Process and guidelines for
ensuring a quality user experience. Elsevier.
[26] Andreas Holzinger and Alexander K Nischelwitzer. 2006. People with motor
and mobility impairment: Innovative multimodal interfaces to wheelchairs. In
International Conference on Computers for Handicapped Persons. Springer, 989–
991.
[27] Zoom Video Communications Inc. 2012. Video Conferencing, Cloud Phone, We-
binars, Chat, Virtual Events | Zoom. https://zoom.us/. (Accessed on 07/20/2021).
[28] Shaun K Kane, Anhong Guo, and Meredith Ringel Morris. 2020. Sense and accessi-
bility: Understanding people with physical disabilities’ experiences with sensing
systems. In The 22nd International ACM SIGACCESS Conference on Computers
and Accessibility. 1–14.
[29] Andrey Khusid and Oleg Shardin. 2011. An Online Visual Collaboration Platform
for Teamwork | Miro. https://miro.com/. (Accessed on 06/02/2021).
[30] Ki-Hong Kim, Jae-Kwon Yoo, Hong Kee Kim, Wookho Son, and Soo-Young Lee.
2006. A practical biosignal-based human interface applicable to the assistive
systems for people with motor impairment. IEICE transactions on information
and systems 89, 10 (2006), 2644–2652.
[31] Vinay Krishna Sharma, Kamalpreet Saluja, Vimal Mollyn, and Pradipta Biswas.
2020. Eye gaze controlled robotic arm for persons with severe speech and motor
impairment. In ACM Symposium on Eye Tracking Research and Applications. 1–9.
[32] Jonathan Lazar, Brian Wentz, and Marco Winckler. 2017. Information privacy
and security as a human right for people with disabilities. In Disability, Human
Rights, and Information Technology. University of Pennsylvania Press, 199–211.
[33] Carol Levine, Susan Reinhard, Lynn Friss Feinberg, Steven Albert, and Andrea
Hart. 2003. Family caregivers on the job: Moving beyond ADLs and IADLs.
Generations 27, 4 (2003), 17–23.
[34] Brittany Lewis and Krishna Venkatasubramanian. 2021. “I... Got my Nose-Print.
But it Wasn’t Accurate”: How People with Upper Extremity Impairment Authen-
ticate on their Personal Computing Devices. (2021).
[35] Franklin Mingzhe Li, Di Laura Chen, Mingming Fan, and Khai N Truong. 2019.
FMT: A wearable camera-based object tracking memory aid for older adults. Pro-
ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
3, 3 (2019), 1–25.
[36] Franklin Mingzhe Li, Di Laura Chen, Mingming Fan, and Khai N Truong. 2021.
“I Choose Assistive Devices That Save My Face” A Study on Perceptions of
Accessibility and Assistive Technology Use Conducted in China. In Proceedings
of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
[37] Franklin Mingzhe Li, Jamie Dorst, Peter Cederberg, and Patrick Carrington. 2021.
Non-Visual Cooking: Exploring Practices and Challenges of Meal Preparation
by People with Visual Impairments. In The 23rd International ACM SIGACCESS
Conference on Computers and Accessibility. 1–11.
[38] Franklin Mingzhe Li, Franchesca Spektor, Meng Xia, Mina Huh, Peter Cederberg,
Yuqi Gong, Kristen Shinohara, and Patrick Carrington. 2022. “It Feels Like Taking
a Gamble”: Exploring Perceptions, Practices, and Challenges of Using Makeup
and Cosmetics for People with Visual Impairments. In CHI Conference on Human
Factors in Computing Systems. 1–15.
[39] Michael Xieyang Liu, Aniket Kittur, and Brad A. Myers. 2022. Crystalline: Low-
ering the Cost for Developers to Collect and Organize Information for Decision
Making. In Proceedings of the 2022 CHI Conference on Human Factors in Computing
Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA.
https://doi.org/10.1145/3491102.3501968 event-place: New Orleans, LA, USA.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[40] Michael Xieyang Liu, Andrew Kuznetsov, Yongsung Kim, Joseph Chee Chang,
Aniket Kittur, and Brad A. Myers. 2022. Wigglite: Low-cost Information Collection
and Triage. In The 35th Annual ACM Symposium on User Interface Software and
Technology (UIST ’22). Association for Computing Machinery, New York, NY,
USA. https://doi.org/10.1145/3526113.3545661 event-place: Bend, OR, USA.
[41] Meethu Malu, Pramod Chundury, and Leah Findlater. 2018. Exploring accessible
smartwatch interactions for people with upper body motor impairments. In
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
1–12.
[42] Meethu Malu and Leah Findlater. 2015. Personalized, wearable control of a head-
mounted display for users with upper body motor impairments. In Proceedings
of the 33rd Annual ACM Conference on Human Factors in Computing Systems.
221–230.
[43] Anahita Mohseni-Kabir, Changshuo Li, Victoria Wu, Daniel Miller, Benjamin
Hylak, Sonia Chernova, Dmitry Berenson, Candace Sidner, and Charles Rich.
2019. Simultaneous learning of hierarchy and primitives for complex robot tasks.
Autonomous Robots 43, 4 (2019), 859–874.
[44] Martez E Mott and Jacob O Wobbrock. 2019. Cluster Touch: Improving touch
accuracy on smartphones for people with motor and situational impairments. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
1–14.
[45] Imad Mougharbel, Racha El-Hajj, Houda Ghamlouch, and Eric Monacelli. 2013.
Comparative study on diferent adaptation approaches concerning a sip and puf
controller for a powered wheelchair. In 2013 Science and Information Conference.
IEEE, 597–603.
[46] John E Muñoz, Ricardo Chavarriaga, and David S Lopez. 2014. Application of
hybrid BCI and exergames for balance rehabilitation after stroke. In Proceedings
of the 11th Conference on Advances in Computer Entertainment Technology. 1–4.
[47] Syifaun Nafsah, Oyas Wahyunggoro, and Lukito Edi Nugroho. 2016. Evaluating
the usage of short-time energy on voice biometrics system for cerebral palsy.
In 2016 8th International Conference on Information Technology and Electrical
Engineering (ICITEE). IEEE, 1–6.
[48] Maia Naftali and Leah Findlater. 2014. Accessibility in context: understanding
the truly mobile experience of smartphone users with motor impairments. In
Proceedings of the 16th international ACM SIGACCESS conference on Computers &
accessibility. 209–216.
[49] Christa Neuper, Gernot R Müller, Andrea Kübler, Niels Birbaumer, and Gert
Pfurtscheller. 2003. Clinical application of an EEG-based brain–computer inter-
face: a case study in a patient with severe motor impairment. Clinical neurophys-
iology 114, 3 (2003), 399–409.
[50] U.S. Department of Labor. [n.d.]. Personal Assistance Services | U.S. Depart-
ment of Labor. https://www.dol.gov/agencies/odep/program-areas/employment-
supports/personal-assistance-services. (Accessed on 08/31/2021).
[51] PBS.org and the AARP. 2021. Activities of Daily Living Checklist & Assessments.
https://www.payingforseniorcare.com/activities-of-daily-living. (Accessed on
07/26/2021).
[52] Dennis Perzanowski, Alan C Schultz, William Adams, Elaine Marsh, and Magda
Bugajska. 2001. Building a multimodal human-robot interface. IEEE intelligent
systems 16, 1 (2001), 16–21.
[53] Alisha Pradhan, Kanika Mehta, and Leah Findlater. 2018. " Accessibility Came
by Accident" Use of Voice-Controlled Intelligent Personal Assistants by People
with Disabilities. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems. 1–13.
[54] Leah M Reeves, Jennifer Lai, James A Larson, Sharon Oviatt, TS Balaji, Stéphanie
Buisine, Penny Collings, Phil Cohen, Ben Kraal, Jean-Claude Martin, et al. 2004.
Guidelines for multimodal user interface design. Commun. ACM 47, 1 (2004),
57–59.
[55] Syed Asad Rizvi, Ella Tuson, Breanna Desrochers, and John Magee. 2018. Sim-
ulation of Motor Impairment in Head-Controlled Pointer Fitts’ Law Task. In
Proceedings of the 20th International ACM SIGACCESS Conference on Computers
and Accessibility. 376–378.
[56] Young Hak Roh. 2013. Clinical evaluation of upper limb function: Patient’s
impairment, disability and health-related quality of life. Journal of exercise
rehabilitation 9, 4 (2013), 400.
[57] Lucas Rosenblatt, Patrick Carrington, Kotaro Hara, and Jefrey P Bigham. 2018.
Vocal programming for people with upper-body motor impairments. In Proceed-
ings of the Internet of Accessible Things. 1–10.
[58] David M Roy, Marilyn Panayi, Roman Erenshteyn, Richard Foulds, and Robert
Fawcus. 1994. Gestural human-machine interaction for people with severe speech
and motor impairment due to cerebral palsy. In Conference companion on Human
factors in computing systems. 313–314.
[59] Kristen Shinohara and Jacob O Wobbrock. 2011. In the shadow of misperception:
assistive technology use and social interactions. In Proceedings of the SIGCHI
conference on human factors in computing systems. 705–714.
[60] Young Chol Song. 2010. Joystick text entry with word prediction for people with
motor impairments. In Proceedings of the 12th international ACM SIGACCESS
conference on Computers and accessibility. 321–322.
Li et al.
[61] Wei Sun, Franklin Mingzhe Li, Benjamin Steeper, Songlin Xu, Feng Tian, and
Cheng Zhang. 2021. Teethtap: Recognizing discrete teeth gestures using motion
and acoustic sensing on an earpiece. In 26th International Conference on Intelligent
User Interfaces. 161–169.
[62] Mohd Razali Md Tomari, Yoshinori Kobayashi, and Yoshinori Kuno. 2012. Devel-
opment of smart wheelchair system for a user with severe motor impairment.
Procedia Engineering 41 (2012), 538–546.
[63] Matthew Turk. 2014. Multimodal interaction: A review. Pattern Recognition
Letters 36 (2014), 189–195.
[64] Ovidiu-Ciprian Ungurean and Radu-Daniel Vatavu. 2021. Coping, Hacking,
and DIY: Reframing the Accessibility of Interactions with Television for People
with Motor Impairments. In ACM International Conference on Interactive Media
Experiences. 37–49.
[65] Radu-Daniel Vatavu and Ovidiu-Ciprian Ungurean. 2019. Stroke-Gesture Input
for People with Motor Impairments: Empirical Results &amp; Research Roadmap.
In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
1–14.
[66] Anandghan Waghmare, Qiuyue Xue, Dingtian Zhang, Yuhui Zhao, Shivan Mittal,
Nivedita Arora, Ceara Byrne, Thad Starner, and Gregory D Abowd. 2020. Ubiqui-
Touch: Self sustaining ubiquitous touch interfaces. Proceedings of the ACM on
Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1–22.
[67] Ker-Jiun Wang, Quanbo Liu, Yifan Zhao, Caroline Yan Zheng, Soumya Vhasure,
Quanfeng Liu, Prakash Thakur, Mingui Sun, and Zhi-Hong Mao. 2018. Intelligent
wearable virtual reality (VR) gaming controller for people with motor disabilities.
In 2018 IEEE International Conference on Artifcial Intelligence and Virtual Reality
(AIVR). IEEE, 161–164.
[68] Ker-Jiun Wang, Caroline Yan Zheng, and Zhi-Hong Mao. 2019. Human-centered,
ergonomic wearable device with computer vision augmented intelligence for
VR multimodal human-smart home object interaction. In 2019 14th ACM/IEEE
International Conference on Human-Robot Interaction (HRI). IEEE, 767–768.
[69] WebAIM. 2012. Types of Motor Disabilities. https://webaim.org/articles/. (Ac-
cessed on 07/26/2021).
[70] Johann Wentzel, Sasa Junuzovic, James Devine, John Porter, and Martez Mott.
2022. Understanding How People with Limited Mobility Use Multi-Modal Input.
In CHI Conference on Human Factors in Computing Systems. 1–17.
[71] Jacob Wobbrock and Brad Myers. 2006. Trackball text entry for people with
motor impairments. In Proceedings of the SIGCHI conference on Human Factors in
computing systems. 479–488.
[72] Jacob O Wobbrock and Krzysztof Z Gajos. 2008. Goal crossing with mice and
trackballs for people with motor impairments: Performance, submovements, and
design directions. ACM Transactions on Accessible Computing (TACCESS) 1, 1
(2008), 1–37.
[73] Muna Khalil Yousef. 2001. Assessment of metaphor efcacy in user interfaces
for the elderly: a tentative model for enhancing accessibility. In Proceedings of
the 2001 EC/NSF workshop on Universal accessibility of ubiquitous computing:
providing for the elderly. 120–124.
[74] Xiaoyi Zhang, Harish Kulkarni, and Meredith Ringel Morris. 2017. Smartphone-
based gaze gesture communication for people with motor disabilities. In Pro-
ceedings of the 2017 CHI Conference on Human Factors in Computing Systems.
2878–2889.
[75] Michael Zinn, Bernard Roth, Oussama Khatib, and J Kenneth Salisbury. 2004.
A new actuation approach for human friendly robot design. The international
journal of robotics research 23, 4-5 (2004), 379–398.
 It’s Enactment Time!: High-fidelity Enactment Stage for Accessible Automated
Driving System Technology Research
Aaron Gluck Hannah Solini Julian Brinkley
Clemson University Clemson University Clemson University
Clemson, South Carolina Clemson, South Carolina Clemson, South Carolina
amgluck@g.clemson.edu hsolini@g.clemson.edu jbrinkl@g.clemson.edu

ABSTRACT
Automated driving system (ADS) technology has been incorporated into critical driving
functions (e.g., adaptive cruise control and autonomous braking) for over two
decades. Now companies like GM and Google are developing and testing fully
autonomous vehicles (AVs). However, the current design of AVs is for individuals who
can drive rather than those who cannot, those who would benefit most from these
vehicles. As the technology is still primarily experimental, conducting research with
AVs and older adults or people with disabilities is infeasible. Therefore, to explore
the accessibility of AVs, we conduct user enactment studies, as this method works well
with technologies that participants have little to no experience utilizing. This pictorial
describes the iterative process of developing a high-fidelity enactment environment, the
space and objects with which participants interact. The aim is to encourage the HCI
community to utilize high-fidelity enactment environments for conducting accessible
future technology research.
Author Keywords
User enactment; Automated driving system; High-fidelity enactment stage
CSS Concepts
• Human-centered computing • Human-computer interaction (HCI) • HCI design and
evaluation methods • Usability testing

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and
the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.

ASSETS ’22, October 23–26, 2022, Athens, Greece

© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10…$15.00.
DOI: https://doi.org/10.1145/3517428.3551351
INTRODUCTION
Automated driving system (ADS) technology holds the
promise of redefining the way we drive, minimizing
roadway accidents and injuries, and saving lives
[10,17,22]. While important, the potential for mobility
independence that ADSs provide for people who cannot
drive (e.g., older adults and people who are blind) is
arguably more significant [2]. However, current ADS
development trends appear to focus on occupants who can
presently drive conventional motor vehicles [16]. This
lack of accessible development may be due to designers
waiting for ADS technology to reach SAE International’s
Level 5 of driving automation, in which the autonomous
vehicle (AV) can handle all driving situations without
occupant input [18,23]. The problem with this approach Figure 1: Messy teenager’s room stage [20]. Figure 2: Autonomous vehicle stage [21].
is that design patterns for human-machine interfaces
(HMIs) are being developed without standardization and
without implementing accessibility features from the
beginning [3,5]. This problem is exacerbated by the
unavailability of Level 5 AVs to explore accessible HMI
and vehicle design for older adults and people with
disabilities with participant stakeholders. For instance,
Google and GM are designing and testing Level 5 AVs,
but these vehicles are currently only available for
ridesharing and in limited, controlled environments
[11,25]. These vehicles are unavailable to be brought into
the research lab or interacted with by participants outside
specific locations (e.g., San Francisco, California [7,26]
and Phoenix, Arizona [26]).
Due to the inability to conduct research in a real-world
environment, accessible ADS technology research Figure 3: Smart-home stage [6]. Figure 4: Virtual home office stage [15].
studies require participants to imagine interacting with
the AV and its HMI systems. This imaginative method out their decisions [19]. User enactment sessions are User enactment stages have been used to study topics
can be complicated when exploring technologies that built around one or more scenarios, developed in advance including a messy teen’s bedroom (Figure 1) [20],
participants do not have prior experience using [21]. This to explore the study’s research questions and consider the autonomous vehicle design (Figure 2) [21], smart-home
situation is the perfect research environment to conduct amount of participant control and the level of fidelity assistants (Figure 3) [6], and understanding burglary
user enactment studies. User enactment (UE) is a process required to set the stage [20]. The enactment stage (Figure 4) [15]. These enactment stages showcase how
by which individuals can explore emerging and future provides the setting where the scenarios are acted out to UE can provide researchers with a better understanding of
technologies through simulated scenarios [4,19]. The UE give researchers a deeper understanding of the needs, participants’ needs and interactions across many research
process combines brainstorming, expressed by opinions, and limitations participants experience while areas.
participants speaking aloud their thoughts and actions, attempting to physically and mentally complete the tasks
and bodystorming, where participants physically act presented in the scenario [20].
EVOLUTION OF THE ENACTMENT STAGE
The design and development of our enactment stage evolved throughout a series of user
enactment studies conducted by our research lab. After each study, we held internal
debriefing meetings to discuss our enactment stage’s positive and negative aspects based
on participant feedback and our research staff’s observations. We used the results of
these debriefings to iterate on the design of the enactment stage.
First Enactment Study
Our first user enactment study [13,14] created a low fidelity enactment stage. We
outlined an average-sized three-row sport utility vehicle (SUV) using blue painter’s tape
with a length of 190 inches and a width of 75 inches. We completed the stage by
including six chairs representing seating options within the vehicle. Ten participants,
divided into two groups of five, participated in the study. Participants acted and spoke
aloud while interacting with the enactment stage: entering and exiting the vehicle, taking
part in in-vehicle activities (e.g., talking on the phone, listening to music, and sleeping),
and moving and removing chairs from the stage to fit their needs.
Planned enactment stage changes after reviewing the first study:
• Add a method for tracking proposed changes to the enactment stage
• Add front and rear of vehicle sections to delineate the vehicle’s main cabin to
reflect participant preference for a smaller vehicle.
• Add props to represent items that participants may choose to bring along
while traveling.
Second Enactment Study
In preparation for our second enactment study [12,14], we provided a selection of
physical props (e.g., phone, purse, and book) and foam core board pieces to write the
name of items that participants requested but that we did not have (e.g., golf clubs).
Additionally, we used pieces of foam core board to write down changes to the enactment
stage to track these modifications between participant groups. Finally, two tri-fold
display boards were used to represent the front and rear of the ADS vehicle.
We began this study by taping off the same SUV footprint as the first study and placing
six chairs into the enactment stage. Thirty older adult participants (60+) participated in
this study. Participants were divided into seven groups of three to six individuals. The
first group was presented with the final design from the first enactment study and iterated
upon it to develop their design. Each following group began with the previous group’s
ending design and continued the iteration. User enactment for this study focused on three
phases of interacting with the ADS vehicle: the arrival of the vehicle to pick up
passengers and vehicle ingress, riding in the vehicle, and the vehicle’s arrival at the
passengers’ destination and vehicle egress. As in the first study, participants acted out
and spoke aloud about their interactions with the enactment stage.
Planned enactment stage changes after reviewing the second study:
• Rework tracking system to make changes more visible to all participants
• Inclusion of a communication system to allow for more in-depth studies using
the Wizard-of-Oz method
• A better system for delineating the vehicle’s interior spaces
• A way to place a human-machine interface (HMI) for requested features in a
specific location within the enactment stage
 HIGH-FIDELITY ENACTMENT STAGE DEVELOPMENT
After discussing participant responses and observations made
during the second study, we began planning the development of
an updated enactment stage. We were inspired to create a high-
fidelity enactment stage by the ridesharing ADS vehicles
currently under development and testing. As these vehicles are
in limited production, there were physical vehicle models on
which to base our design. We compared the vehicle designs of
Google’s Waymo One (Figure 5) [26] and GM’s Cruise series
of AVs (Figure 6 and Figure 7) [7], as each of these companies
was conducting on-road testing of their vehicles at the time of
Figure 5: Waymo One autonomous vehicle [26].
our development. Ultimately, we decided to base our high-
fidelity enactment stage model on the GM Cruise Origin
ridesharing AV [7]. Deciding factors were doors that can support
fully motorized wheelchair ingress and egress, the ability to
easily manipulate the seating floor plan, and more surface space
for exploring multiple HMI placements and interactions.
Development Plan
Development of the high-fidelity enactment stage began with
finding reference images of the GM Cruise Origin [7] to
determine its approximate dimensions. Next, rough blueprints
were developed to determine the final layout and plan materials.

Figure 6: First Generation Cruise AV [7].

Figure 7: The Cruise Origin ridesharing autonomous vehicle [7].

HIGH-FIDELITY ENACTMENT STAGE CONSTRUCTION
The main structure of the enactment stage was built from 2-inch by 4-inch wood boards and ¾” plywood sheets and
covered with a foam core board shell. Screen door material was used to represent portions of the vehicle composed of
tinted glass. Structural elements in screened areas were painted black to show this would be glass in a production
vehicle. The overall dimensions of the high-fidelity enactment stage are 14 feet long, 7 feet high (with an internal height
of 6 feet), and 6 feet wide. It has two sets of sliding doors on either side of the vehicle and a working trunk.
Building for Accessibility
This project was designed to ensure that older adults and people with disabilities could interact with the high-fidelity
enactment stage, which dictated certain aspects of our development. First, the base floor was over-engineered to support
the weight of four individuals in powered wheelchairs. A subfloor structure of 2-inch by 4-inch wood boards was set
every foot, with ¾” plywood laid on top to create the floor, providing over 600 pounds per square foot of support [1].
Second, as the enactment stage floor is 4 ¼” above the floor, a metal ramp can be added to access the interior without
stepping up. The ramp is 5 feet long and 3 feet wide to create the required rise over run and width for access by an
individual in a wheelchair [9].
HIGH-FIDELITY ENACTMENT STAGE FOR RESEARCH
The primary use of the high-fidelity enactment stage is to research automated driving
system technologies for older adults and people with disabilities, so we needed to provide
the ability for our participants to design and modify the stage. The interior walls of the
stage are covered with metallic paper [8], and we developed foam core magnet widgets
in distinctive shapes to represent different interaction devices one might encounter in an
ADS (e.g., levers, buttons, keypads, and microphones). Each widget has its name printed
in both text and Braille. Additionally, we have a supply of blank widgets readily available
if a participant requests a previously uncreated widget. The new widget’s name is added
via a sticky note. The metallic paper and the magnetic widgets allow participants to place,
rearrange, and remove various ADS human-machine interfaces on the stage’s interior
walls to reflect their needs, preferences, and ideas.
In addition to the widgets, our high-fidelity enactment stage has other features that allow
for design modifications in future research studies. The seating is moveable, allowing
researchers to create different scenarios and for participants to modify the seating layout
as desired. There is also sufficient space in the front and rear of the vehicle to install
computers or other digital devices to study various systems. Additionally, as the
enactment stage is constructed from lumber and foam core board, modifying the internal
structures is as simple as removing or adding a few screws. Finally, multiple locations
are available within the enactment stage for placing recording equipment data analysis.
INTERNAL PILOT STUDY
We explored the usability and accessibility of the high-fidelity enactment stage while
researching accessible autonomous vehicles for people with disabilities as a semi-finalist
in the U.S. Department of Transportation’s (DOTs) Inclusive Design Challenge (IDC)
[24]. We made project-specific modifications to the enactment stage to present a usable
touch-screen HMI that can be interacted with directly or through a wearable device, the
glasses worn in the images below. We also installed a video projector to explore using
the ADS vehicle’s floor as an additional monitor displaying the vehicle’s position on a
map. One member of our research team with low vision that had not previously interacted
with the enactment stage was brought in to interact with the enactment stage and our
final IDC project. We asked a series of qualitative questions about her experience
interacting with the high-fidelity enactment stage, both with and without the IDC glasses.
She had no problems entering or exiting the enactment stage and reported finding the
environment accessible, easy to interact with, and use. The results of this internal pilot
demonstrated the usability of the high-fidelity enactment stage and that it could quickly
and easily be modified to explore different features and assistive technologies for
different research studies and conditions. While additional testing is required, the initial
findings show the benefit of using a high-fidelity enactment stage to research accessible
ADS technology and vehicles.
CONCLUSION
In this pictorial, we described the preliminary user
enactment studies that led to the design and development
of the high-fidelity enactment stage for researching the
accessibility of ADSs. The main factor driving the
development of this enactment stage is the unavailability
of self-driving ADS technology for research studies. The
high-fidelity enactment stage provides an environment
where participants can imagine interacting with the ADS
technology and providing their needs and opinions for the
required features and design to make the system
accessible for older adults and people with disabilities.
The high-fidelity enactment stage is far from complete,
especially as ADS technology is still new. Instead, it will
continue to morph and change based on future study
requirements, findings from research conducted on the
enactment stage, and ADS technology advancements.
Additionally, we plan to explore new high-fidelity add-
ons to the enactment stage (e.g., augmented reality and
projectors) and human-machine interface systems and
applications.
Finally, this research was best presented as a pictorial due
to the highly visual nature of the studies. The pictorial
format allows HCI researchers to see the processes that
led to the high-fidelity enactment stage’s development
and how participants can interact with it.
ACKNOWLEDGMENTS
We want to thank the many participants who have
participated in the associated studies and the US
Department of Transportation, who partially funded the
development of the high-fidelity enactment stage through
their Inclusive Design Challenge for Autonomous
Vehicle Accessibility.
REFERENCES
[1] Appliance Analysists. 2021. How Much Weight
Can Plywood Hold? Free Calculator. Appliance
Analysts. Retrieved June 23, 2022 from
https://applianceanalysts.com/plywood-weight-
capacity/
[2] Robin N. Brewer and Vaishnav Kameswaran.
2018. Understanding the Power of Control in
Autonomous Vehicles for People with Vision
Impairment. In Proceedings of the 20th
International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’18),
ACM, New York, NY, USA, 185–197.
DOI:https://doi.org/10.1145/3234695.3236347
[3] Julian Brinkley, Earl W. Huff, Kwajo Boateng, and
Suyash Ahire. 2020. Autonomous Vehicle Design
Anti-Patterns: Making Emerging Transportation
Technologies Inaccessible by Design. Proceedings
of the Human Factors and Ergonomics Society
Annual Meeting 64, 1 (December 2020), 1038–
1042.
DOI:https://doi.org/10.1177/1071181320641249
[4] Marion Buchenau and Jane Fulton Suri. 2000.
Experience prototyping. In Proceedings of the 3rd
conference on Designing interactive systems:
processes, practices, methods, and techniques (DIS
’00), Association for Computing Machinery, New
York City, New York, USA, 424–433.
DOI:https://doi.org/10.1145/347642.347802
[5] Oliver Carsten and Marieke H. Martens. 2019.
How can humans understand their automated cars?
HMI principles, problems and solutions. Cogn
Tech Work 21, 1 (February 2019), 3–20.
DOI:https://doi.org/10.1007/s10111-018-0484-0
[6] Yi-Shyuan Chiang, Ruei-Che Chang, Yi-Lin
Chuang, Shih-Ya Chou, Hao-Ping Lee, I-Ju Lin,
Jian-Hua Jiang Chen, and Yung-Ju Chang. 2020.
Exploring the Design Space of User-System
Communication for Smart-home Routine
Assistants. In Proceedings of the 2020 CHI
Conference on Human Factors in Computing
Systems, ACM, Honolulu HI USA, 1–14.
DOI:https://doi.org/10.1145/3313831.3376501
[7] Cruise LLC. 2022. We’re Cruise, a self-driving car
service designed for the cities we love. cruise.
Retrieved June 21, 2022 from
https://www.getcruise.com/technology
[8] Drytac. 2021. Ferro metal paper. Drytac. Retrieved
June 23, 2022 from
https://www.drytac.com/product/ferro-metal-paper/
[9] EZ-ACCESS. 2022. Ramp Incline Calculator |
ADA Ramp Slope. EZ-ACCESS. Retrieved June
23, 2022 from https://www.ezaccess.com/?
[10] Daniel J. Fagnant and Kara Kockelman. 2015.
Preparing a nation for autonomous vehicles:
opportunities, barriers and policy
recommendations. Transportation Research Part A:
Policy and Practice 77, (July 2015), 167–181.
DOI:https://doi.org/10.1016/j.tra.2015.04.003
[11] General Motors. 2018. 2018 Self-Driving Safety
Report. Detroit, MI. Retrieved June 22, 2022 from
https://www.gm.com/content/dam/company/docs/u
s/en/gmcom/gmsafetyreport.pdf
[12] Aaron Gluck, Kwajo Boateng, Earl W. Huff Jr.,
and Julian Brinkley. 2020. Putting Older Adults in
the Driver Seat: Using User Enactment to Explore
the Design of a Shared Autonomous Vehicle. In
12th International Conference on Automotive User
Interfaces and Interactive Vehicular Applications,
ACM, Virtual Event DC USA, 291–300.
DOI:https://doi.org/10.1145/3409120.3410645
[13] Aaron Gluck, Earl W. Huff, Mengyuan Zhang, and
Julian Brinkley. 2020. Lights, Camera, Autonomy!
Exploring the Opinions of Older Adults Regarding
Autonomous Vehicles Through Enactment.
Proceedings of the Human Factors and Ergonomics
[18] NHTSA. 2022. Automated Vehicles for Safety |
Society Annual Meeting 64, 1 (December 2020),
1971–1975.
DOI:https://doi.org/10.1177/1071181320641475
[14] Earl W. Huff, Mengyuan Zhang, and Julian
Brinkley. 2020. Enacting into Reality: Using User
[19] William Odom, John Zimmerman, Scott Davidoff,
Enactment to Explore the Future of Autonomous
Vehicle Design. Proceedings of the Human Factors
and Ergonomics Society Annual Meeting 64, 1
(December 2020), 1561–1565.
DOI:https://doi.org/10.1177/1071181320641373
[15] Amy Meenaghan, Claire Nee, Jean-Louis Van
Gelder, Marco Otte, and Zarah Vernham. 2018.
Getting Closer to the Action: Using the Virtual
[20] William Odom, John Zimmerman, Jodi Forlizzi,
Enactment Method to Understand Burglary.
Deviant Behavior 39, 4 (April 2018), 437–460.
DOI:https://doi.org/10.1080/01639625.2017.14071
04
[16] National Federation of the Blind. 2016. 2016
Resolutions | National Federation of the Blind.
National Federation of the Blind. Retrieved June
23, 2022 from https://nfb.org/resources/speeches-
and-reports/resolutions/2016-resolutions
[21] Ingrid Pettersson and Wendy Ju. 2017. Design
[17] National Highway Traffic Safety Administration.
2020. Automated Driving Systems | NHTSA.
United States Department of Transportation.
Retrieved June 18, 2022 from
https://www.nhtsa.gov/vehicle-
manufacturers/automated-driving-systems
NHTSA. United States Department of
Transportation. Retrieved June 23, 2022 from
https://www.nhtsa.gov/technology-
innovation/automated-vehicles-safety
Jodi Forlizzi, Anind K. Dey, and Min Kyung Lee.
2012. A fieldwork of the future with user
enactments. In Proceedings of the Designing
Interactive Systems Conference on - DIS ’12,
ACM Press, Newcastle Upon Tyne, United
Kingdom, 338.
DOI:https://doi.org/10.1145/2317956.2318008
Hajin Choi, Stephanie Meier, and Angela Park.
2012. Investigating the presence, form and
behavior of virtual possessions in the context of a
teen bedroom. In Proceedings of the 2012 ACM
annual conference on Human Factors in
Computing Systems - CHI ’12, ACM Press,
Austin, Texas, USA, 327.
DOI:https://doi.org/10.1145/2207676.2207722
Techniques for Exploring Automotive Interaction
in the Drive towards Automation. In Proceedings
of the 2017 Conference on Designing Interactive
Systems - DIS ’17, ACM Press, Edinburgh,
United Kingdom, 147–160.
DOI:https://doi.org/10.1145/3064663.3064666
[22] Mike Ramsey. 2015. Self-Driving Cars Could Cut
90% of Accidents - WSJ. The Wall Street Journal.
Retrieved June 23, 2022 from
https://www.wsj.com/articles/self-driving-cars-
could-cut-down-on-accidents-study-says-
1425567905
[23] SAE International. Taxonomy and Definitions for
Terms Related to Driving Automation Systems for
On-Road Motor Vehicles. SAE International.
DOI:https://doi.org/10.4271/J3016_201806
[24] U.S. Department of Transportation. 2022. DOT
Inclusive Design Challenge. U.S. Department of
Transportation. Retrieved June 23, 2022 from
https://www.transportation.gov/accessibility/inclusi
vedesign
[25] Waymo LLC. 2021. Waymo Safety Report.
Retrieved June 22, 2022 from
https://storage.googleapis.com/waymo-
uploads/files/documents/safety/2021-12-waymo-
safety-report.pdf
[26] Waymo LLC. 2022. Waymo One. Waymo.
Retrieved June 21, 2022 from
https://waymo.com/waymo-one/
 Where Are You Taking Me? Reflections from Observing
Ridesharing Use By People with Visual Impairments
Earl W. Huf Jr. Robin N. Brewer
School of Information School of Information
The University of Texas at Austin University of Michigan
Austin, TX, USA Ann Arbor, MI, USA
ewhuf@utexas.edu rnbrew@umich.edu
ABSTRACT
Ridesharing services have become a popular mode of transportation
that holds signifcant benefts for people with disabilities unable
to operate conventional motor vehicles. Prior work shows how
these services enable people with vision impairments to travel in-
dependently without the use of public transportation or walking.
We conducted a study in which we observed 17 blind or visually
impaired participants using the Uber ridesharing service to explore
the social and accessibility dynamics they perceived during their
experiences. This paper presents a case study of the process used for
our research, refecting on aspects that considerably impacted the
study. Key takeaways include study site considerations, recruiting
participants, balancing realism with participant safety, and the un-
intended side efects of observations in a ridesharing context. The
refection points provided will help readers in considering impor-
tant study aspects when conducting observations with participants
with disabilities, particularly in a ridesharing setting.
CCS CONCEPTS
• Human-centered computing → Accessibility; • Applied com-
puting → Transportation; • Social and professional topics →
People with disabilities.
KEYWORDS
ridesharing, observation study, people with visual impairments
ACM Reference Format:
Earl W. Huf Jr., Robin N. Brewer, and Julian Brinkley. 2022. Where Are You
Taking Me? Refections from Observing Ridesharing Use By People with
Visual Impairments. In The 24th International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece.
ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3517428.3551355
1 INTRODUCTION
Research has indicated that as many as 20% of people with disabili-
ties in the U.S. face transportation barriers, with 45% of this group
lacking access to a personal passenger vehicle [16]. This problem
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specifc permission and/or a
fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3551355
Julian Brinkley
School of Computing
Clemson University
Clemson, SC, USA
julianbrinkley@clemson.edu
especially applies to persons with visual disabilities. Unlike am-
putees or persons with partial paralysis, there are no commercially
available assistive technologies that enable persons who are blind
[14] or those with signifcant low vision to operate conventional
motor vehicles [7]. Ridesharing is an increasingly popular form
of transportation that has played an important part in reshaping
public transportation. These real-time services, led by companies
like Uber and Lyft, are an increasingly utilized on-demand mobility
option that has seen increased use in the United States from 15% to
36% over the past four years [10]. While these services generally
add to the growing transportation options for society, they also
hold signifcant mobility potential for disabled persons unable to
operate conventional motor vehicles.
Ridesharing, therefore, may be a viable option for people with
disabilities to travel independently without reliance upon friends,
family, public transportation, or walking. However, there is a grow-
ing but arguably insufcient body of research focused on the ac-
cessibility of popular ridesharing services for persons with visual
disabilities. Existing work has looked at trust-building, driver-rider
collaboration, situational awareness, and the roles of both driver
and passengers in ridesharing [3, 4, 12]. These studies classifed
the means through which blind or visually impaired (BVI) people
establish trust and engage in collaborative navigation with the dri-
ver in situational awareness and location verifcation. While these
recent studies provide relevant data about past experiences with
ridesharing services, they primarily rely on self-report research
techniques. Such methods may incur response and recency bias
or lack of details about anyone’s experience. Our paper presents a
case study of how we conducted a series of quasi-naturalistic simu-
lations of the end-to-end process for using a ridesharing service by
people with visual impairments to better understand the real-world
experiences of BVI people in the ridesharing context. The remain-
der of this paper is organized as follows: section 2 provides some
background on ridesharing and people with disabilities; section
3 describes in detail the study setup and data collection process;
section 4 details our observations regarding the study execution,
participant behavior, and main takeaways from our work; section
5 provides future directions and conclusion.
2 BACKGROUND
2.1 People with Disabilities in Ridesharing
Prior research on ridesharing services has investigated the expe-
rience from both driver and rider perspectives. From the rider’s
perspective, prior work has looked at the benefts of ridesharing,
such as providing lower-cost transportation alternatives to taxis,
and challenges in using their services [4, 8, 11, 12]. More recently,
ASSETS ’22, October 23–26, 2022, Athens, Greece
research has focused on ridesharing services for people with vi-
sual impairments. The current research on the topic examined how
visually impaired persons can beneft from ridesharing services
and what challenges may afect the adoption of their use. Results
revealed that independence was the most signifcant positive of us-
ing ridesharing services; the increased mobility without relying on
sighted people to take them to their destinations [4, 12]. The biggest
challenges in using ridesharing services were 1) the inaccessibility
of the ridesharing app, 2) locating the right vehicle upon its arrival,
3) getting assistance to the destination upon arrival, and 4) drivers’
lack of experience working with people with disabilities [4, 12].
The studies provide a strong foundation for future discussions on
ridesharing experiences for people with vision impairments and
serve as the basis for our quasi-naturalistic approach by collect-
ing observation data as participants use a ridesharing service in a
real-world setting.
2.2 Factors Impacting Adoption of
Transportation
Prior work on people’s adoption of emerging transportation tech-
nologies reveals the importance of trust and its efect on their
willingness to adopt [5, 6, 8, 9, 13, 15].Choi and Ji’s study revealed
that trust and perceived usefulness were the most signifcant factors
motivating people’s adoption of autonomous vehicles [6]. Further
studies also support the fndings of Choi and Ji by reporting that the
reliability of autonomous systems is an additional factor for tech-
nology adoption [13, 15]. Trust is also a signifcant factor when it
comes to the adoption of using ridesharing services. Related works
reveal how transparency and safety assurance is vital to fostering
trust between riders and drivers [8, 9]. Dillahunt’s work reports on
the mistrust of Uber due to past negative experiences [8]. Although
prior work dealt with sighted people, the same concerns and bar-
riers can afect visually impaired riders. In our study, we wanted
to examine the interaction between drivers and visually impaired
riders to understand the social dynamics in a more naturalistic
setting.
2.3 People with Disabilities in HCI Research
Within the HCI community, we seek to understand users and their
context to obtain insights into how they use technology. We achieve
such goals through user research methods such as surveys and in-
terviews. However, an area of concern in HCI research is the lack of
inclusion of people with disabilities in such studies. They are often
excluded because, depending on their disability, they would require
adapting existing methods to accommodate their needs and the “im-
plicit bias against disabled subjects as producers of knowledge”[17].
Such exclusion promotes exclusive technology design and further
alienates people with disabilities from participating actively in the
community. Within the current literature, however, observations
are a method to include persons with disabilities as producers of
knowledge in a way that researchers can collect in-depth informa-
tion about their personal experiences with technology that cannot
be captured using self-report techniques [1, 2]. The emphasis on
allowing participants to carry out their natural routine in a non-
laboratory setting while still gaining some understanding of their
lives using technology was why we decided to use observations
Earl W. Huf et al.
as our preferred data collection method. Using observations, our
goal was to understand how BVI participants used ridesharing in a
real-world experience.
3 STUDY METHOD
We conducted a quasi-naturalistic study where participants initiated
and completed a ride to a pre-defned destination using the Uber
ridesharing service, after which they were interviewed about their
experience. For the context of this paper, we do not include the
fndings of the post-study interviews as we present an account
of the process used to conduct this study. The purpose of this
case study is to present readers with a narrative of how specifc
decisions in our methodology impacted the research execution and
produced certain outcomes and side efects. We believe these details
are essential for researchers to know when considering conducting
observations in ridesharing research with people with disabilities.
3.1 Recruitment
We partnered with an organization serving persons with visual
impairments in North-Central Florida to recruit participants. In-
terested individuals could participate in the study if they fulflled
the following criteria: 1) age 18 or above and 2) considered them-
selves to be a visually impaired person based on an accompanying
defnition describing visual impairment as blindness or limited vi-
sion not correctable by conventional correction such as glasses
or contact lenses. Additionally, we screened participants before
the consent and scheduling process by telephone. As part of the
screening process, they expressed an absence of physical or cog-
nitive disabilities that would prevent the use of a mobile device,
challenges with ingress or egress from a passenger vehicle, or a
history of motion sickness. Lastly, we required participants to have
used an Android or Apple smartphone within the past six months
from the date of the study session. The frst author’s Institutional
Review Board approved the study, and each participant provided
written informed consent on the day of their study session. We
compensated participants with a $40 prepaid gift card for their
participation.
3.2 Participant
We recruited 17 people to participate in the study. In summary,
participants were 20 to 88 years old (mean = 55). Regarding vehicle
ownership, 70.6% of participants indicated that they own a vehicle.
When asked about modes of transportation, 59% indicated getting
a ride from others in diferent forms of passenger vehicles, while
41% indicated using public transportation (i.e., bus, taxi, shuttle,
and ridesharing). During the screening and scheduling process,
we briefy presented participants with functional defnitions of
blindness and low vision. We asked them to choose the defnition
that best characterized their degree of vision loss. On the day of
their study session, we again presented them with these defnitions
and asked them to indicate their medically diagnosed visual acuity
in their better-seeing eye with conventional correction if known.
Three participants self-identifed as blind, and 14 self-identifed
as low vision. Many of the participants (41%) have been visually
impaired for less than ten years, 35.3% have been visually impaired
Where Are You Taking Me? Reflections from Observing Ridesharing Use By People with Visual Impairments
between 10 to 25 years, and 23.7% have been visually impaired since
birth.
3.3 Tools
We purchased two smartphones for the study. Due to technical
issues, three phones, two Android and one Apple were ultimately
used for each trial. All phones had the Uber ridesharing application
installed, and each phone had a preconfgured payment option to
cover the anticipated cost of the participant’s trip. We asked partic-
ipants to wear a GoPro Hero 7 Black using a harness accessory to
record the interaction between the participant and the ridesharing
driver for data collection and safety purposes. We used a set of
hand-held communication devices for communication between the
research team and the participants. We instructed the participant
that if they felt ill, unsafe, or otherwise desired to stop participating,
they should use the communication device to contact a research
team member to request a ride back to the study staging area. None
of the participants fell ill or decided to stop participating in our
study.
3.4 Procedure
Each trial lasted approximately an hour and a half, with a procedure
identical to each session. After the consent form and demographic
questionnaire were completed, we provided participants with the
following: (1) an Android or Apple smartphone with accessibility
features confgured to their specifcations, depending upon prefer-
ence and availability, and (2) a two-way communication device to
communicate with the research team during the ridesharing trip,
and (3) a GoPro action camera harnessed to their torso to record
their experience during the trip. Before the trip, one research team
member trained the participant on how to use the two-way com-
munication device and explained how the GoPro camera worked.
We provided participants with details on the study scenario and
instructed them to begin their tasks. The two-way communica-
tion device and the GoPro camera were provided to 1) maintain
communication with the participant, 2) maintain awareness of the
participant’s current status in terms of health and location, and 3)
ensure the participant’s safety during the trip. We informed the
participant that a research team member would be following them
in a separate vehicle for their safety. The participant was instructed
to use the two-way communication device at any time during the
ride if they wished to stop participating or required medical atten-
tion. Participants were asked to attempt a one-way ride using the
Uber mobile application as part of a four-task scenario. Participants
were asked to: 1) Request a ride to a pre-defned location, 2) Locate
the Uber vehicle upon arrival at the point of origin, 3) Enter the
vehicle, 4) Exit the vehicle when they arrive at the destination.
We instructed participants to request the ride by selecting the
“UberX” option within the Uber mobile application to start an in-
dividual ride (as opposed to carpool options) and were assisted, as
needed, in entering the destination and selecting the UberX vehicle.
Once a ride was confrmed, participants were led outside to wait
for the Uber driver. Once the Uber vehicle arrived, participants
began interacting with the driver and entered the vehicle. During
the trip, a member of the research team, equipped with a paired
two-way communication device, followed the ridesharing vehicle
ASSETS ’22, October 23–26, 2022, Athens, Greece
in a separate vehicle. This measure was to ensure the participant’s
safety and to aid the Uber driver in navigation in the event they
get lost along the route. Upon arrival at the fnal destination, this
research team member sought written informed consent from the
Uber driver and asked them to complete a photo/video release form.
We collected informed consent from the driver after the trip because
of concern that informing the driver about the study before the
trip would cause the driver to act in a manner unnatural in their
regular working routine, potentially taking away from the natural
interaction between the driver and the participant. The research
team member walked with the study participant to the staging area,
where recording equipment was removed, and the post-scenario
interview was conducted.
4 OBSERVATIONS
We present an account of our observations during the ridesharing
study. We examine the pros and cons of the diferent aspects of
each phase of the study.
4.1 Participants and Study Site
With any accessibility research, recruiting participants with dis-
abilities, especially focusing on specifc disabilities, is a common
challenge. An even more signifcant challenge is recruiting partici-
pants with disabilities in one geographic location to conduct a feld
study of ridesharing experiences. One solution to this challenge is
contacting local advocacy groups or chapters of national organiza-
tions serving people with disabilities who may have a contact list
of members from whom they help recruit. In our case, we reached
out to a school for the blind in North Central Florida that could
help us recruit blind or visually impaired participants for our study.
We selected a school in Florida because a team member had prior
experience working and built a good rapport with them. As a result
of the arrangement, the team agreed to conduct the study at the
school since the participants will generally be school attendants.
The general area in which the school resided provided an ideal
site for the study. There were many public landmarks to serve as
the destination for the trips; most roads had a speed limit of no
more than 45 MPH, and the roads contained multiple lanes, which
was ideal for having a team member follow the ridesharing vehi-
cle. In addition to the road conditions, the school ofered sufcient
space for conducting the pre-ride and post-ride phases of the study.
We used four rooms within the school: one for the pre-ride phase,
another room to prepare the participants with the necessary equip-
ment for their ride, and two rooms to conduct one-on-one post-ride
interviews. We used two rooms for interviews because of the possi-
bility that one session may not end before the next session begins,
and two participants may be interviewed concurrently.
4.2 Pre-Ride Setup and Procedure
The initial setup for the study involved preparing the tools the
participants would be using. For our study, we used two smart-
phones, one with the Android operating system and the other using
Apple’s iOS. We set up each phone with the necessary assistive
technologies for participants to use them. For Apple, VoiceOver
is their built-in mobile screen reading feature; for Android-based
devices, TalkBack (at the time of the study) is their screen reading
ASSETS ’22, October 23–26, 2022, Athens, Greece Earl W. Huf et al.

application. We intentionally selected the Uber ridesharing applica- impaired participants. One way we could have alleviated this issue
tion because of the higher availability of drivers in the area than was to run the sessions at diferent times of the day to increase the
Lyft at the time. We confgured accounts with a credit card saved odds of getting new drivers. Similarly, we could have scheduled the
in the Payments section. We also saved the destination location sessions on diferent days rather than consecutive days to avoid
in the app to make it easier for participants to select it during the making a pattern of our ridesharing runs. The challenge lies in the
ride-booking process. We instructed participants to go through the unknown of what and when drivers will be in the vicinity of the
process of ordering a ride from the Uber application using their site.
phone of choice. During this phase, we experienced some technical
issues, such as inconsistencies in the payment being processed for 4.4 Ridesharing Experience
rides. There was a problem with the account we created for the
After being equipped with the tools for the study and booking a ride,
phones and using the payment options to book the rides. The issue
participants made their way to the building entrance to wait for
was less prevalent when using an Android-based smartphone than
their ride to come. Some participants noted that the screen reader
the Apple smartphone (although we believe the issue was not spe-
would speak out the name of the driver but not the make or model
cifc to a mobile O.S.). As a working solution, we decided to use an
of their vehicle nor the license plate number, which was a concern
Android-based smartphone; however, we knew it would be an issue
for them. When the vehicle arrived, a team member approached the
for participants who were accustomed to using Apple smartphones.
driver to inform them that the participant was the intended rider
However, they were able to adjust quickly, with some assistance
but did not tell them it was part of a research study. A team member
from the research team. As participants began using the application,
followed the ridesharing vehicle in a separate car during the ride.
there were difculties navigating the user interface (U.I.). At times,
The participant was aware of this but not the driver until they
they could not move from the map interface to the Uber service
reached their destination. We did this so that if the participant did
selection menu at the bottom of the screen. Other times, they had
not want to participate in the study anymore or experienced motion
challenges navigating from selecting the Uber vehicle to the con-
sickness, they could contact the team member via the two-way radio
frmation of payment menu. These issues were brought up again
to end the study, and the team member would take the participant
in the post-study interviews. Some participants mentioned they
back to their vehicle. Additionally, if the driver got lost getting
would have preferred the ability to use voice to schedule the ride
to the destination entrance, the team member could assist with
rather than interacting with the U.I. Another tool was the GoPro
navigation. There were instances where the driver had difculties
Hero 7 Black camera mounted onto a harness that participants
fnding the entrance to the destination, and the participant would
wore for the trip. The harness wraps around the user’s upper body
contact the driving team member to assist. Once at the destination,
and sits on their chest. The range of adjustment for the harness
both vehicles parked, the team member approached the driver to
was limited, which made it uncomfortable for some participants.
explain the purpose of the study and their involvement and asked
While it captured much of the interaction with the research team
the driver to sign an informed consent to agree to be a part of the
and ridesharing driver, the camera positioning did have a limited
study and a media release form for permission to use the recording
range of video coverage.
from the camera. Additionally, the drivers were compensated with
an appropriate tip for their participation. No driver declined to
4.3 Consent Process for Driver and Passenger participate or refused to have the recording released. Afterward,
the participant was assisted into the team member’s vehicle and
As in general research practice, we obtained informed consent from
driven back to the study site. Upon refection, we believe a possible
the BVI participants before we began each session. However, for the
faw in the study design was the safety measures for the participant,
drivers involved in our study, we decided to collect their informed
namely the camera given to the participant, the two-way radio,
consent after completing the trip with the participant passengers.
and the team member driving alongside the ridesharing vehicle.
We needed their consent in order to use the video from the partici-
Those elements may have given the participant a greater sense of
pant’s camera and also to obtain their acknowledgment that they
safety and security than a typical ride without them and could have
willfully took part in the experience. We intentionally waited until
impacted their natural behavior and interaction with ridesharing
after the trip to maintain the natural interaction between driver
drivers. Another consideration is the time of day when the study
and rider. We felt the drivers would act out of their typical behav-
was conducted. All sessions were conducted in the morning and
ior if they discovered the nature of our research before the ride.
early afternoon when there was less trafc. However, the time of day
However, this proved to be a moot point as a few of the drivers
may have impacted the availability of drivers present for the study,
stayed in the area near the study site to accept subsequent ride
as noted by the a few of the drivers that repeatedly accepted our ride
requests. We believed this to be the result of the incentive ofered
requests by staying within the area once they knew about the study.
to drivers where we provided a $25 tip (this was considered relative
We believe staging the sessions during evening or morning rush
to the suggested tip by the app for the short trip requested) as
hour times may have yielded more drivers (although not guaranteed
compensation. This led some drivers to stay within an area close
diferent drivers every time).
to the study site to increase their chances of being selected for
subsequent requests. Further, because the same drivers would pick
up the participants and know the purpose of the study, the natural 4.5 Takeaways
interaction with the passenger was lost; instead, an incentive-based We provide several takeaways from our experience conducting our
bias may have infuenced the drivers’ behaviors toward the visually study to inform the community who may be interested in carrying
Where Are You Taking Me? Reflections from Observing Ridesharing Use By People with Visual Impairments
out a similar study with the visually impaired or other persons with
disabilities.
4.5.1 Feasibility of Recruitment. The feasibility of our study rests
on the convenience of fnding a location where our population of
interest would be available, in our case, a school for the blind or
visually impaired. When conducting an observation under quasi-
naturalistic or naturalistic settings with people with disabilities,
being able to stage the study at a location that services or can
help recruit the intended participants is one of the most important
areas to consider. Even though no team member resides in the state
of Florida, we agreed that the close ties with the school and the
increased availability of participants with visual impairments made
the site an ideal area to host our research. We suggest contacting
local agencies serving people with disabilities, advocacy groups, or
local chapters of disability organizations to fnd an ideal location to
host the research and if they can assist with recruiting participants.
4.5.2 Technology Barriers. A major obstacle in conducting our
study was the tools we used, specifcally smartphones. The techni-
cal issues we encountered using the Uber application forced us to
use one type of smartphone device (Android). While we had multi-
ple Android smartphones, participants preferred using an Apple
smartphone. As a result, participants had an additional period to
adjust to using the Android device. However, we did discover partic-
ipants’ perceptions of using the Uber application during the setup
process. While some participants found the application generally
accessible, others preferred multimodal interaction support as they
found the U.I. challenging to navigate, even with their assistive
technologies. We advise readers to 1) thoroughly examine and test
all equipment before the study and 2) research and provide options
for assistive technologies when working with participants with
disabilities. We suggest running a series of mock simulations to
ensure the equipment will work as planned.
4.5.3 Considering Roles of Stakeholders. An implication of our
research was the role of the stakeholders in our research. While
the BVI participants engaged in the entire process for most of the
study, we did not fully consider the role of the drivers outside of
driving the participants to the destination. We decided to withhold
information regarding the study and the participant’s role until
the trip’s conclusion to avoid unnatural interactions between the
driver and passenger. However, we did not consider the case of
what may happen after we informed the driver about our study and
compensated them. For example, one driver remained in the area
closest to the study site to increase their chance of receiving subse-
quent requests and earn additional compensation. While canceling
their ride was an option, we could not guarantee another available
driver to take the request. Hence, we believe the reoccurrence of
the same drivers, infuenced by an incentive-based bias, could have
negatively impacted the genuine interaction with other passengers.
Considerations should be made regarding running the study at
diferent times of the day to help fnd diferent drivers.
4.5.4 Safety vs Realism. An unforeseen circumstance was the vari-
ous safety measures used to protect the participant. Using a cam-
era, two-way radio, and a following vehicle from the team would
provide a measure of security for the participants; however, that
may also have afected their natural behavior and interaction in a
ASSETS ’22, October 23–26, 2022, Athens, Greece
ridesharing vehicle. The camera alone could have made the driver
hesitant to engage since they were being recorded. When deciding
on the safety precautions, we prioritized the participants’ safety at
the expense of some of the natural progression of a ridesharaing
simulation. Another concern was approval from our institution’s
IRB. It might not have been possible to achieve approval if we did
not include the measures we did. When conducting observations of
people with disabilities, safety should always be the priority. The
challenge becomes ensuring their safety while mitigating the loss of
their natural interaction in the studied context. We advise readers to
carefully consider all possible side efects of any props, equipment,
or process used in the study and how they may intentionally or
unintentionally afect any aspect of the user experience.
5 CONCLUSION
We presented a case study of our experience conducting quasi-
naturalistic observations of blind or visually impaired people using
the Uber ridesharing service. Refecting on our experience, the
study ofered a lot of relevant fndings regarding how passengers
with visual impairments engage in ridesharing trips with their
drivers and how ridesharing generally compares with other modes
of transportation they use. The process we used was not without
its drawbacks, such as technical issues with the equipment, the
unintended side efects of being present during the trip, and the use
of certain safety equipment. The lessons learned from our study will
help us plan a future study involving observations of people with
disabilities using ridesharing services. We present to readers several
takeaways to consider when conducting their observations within
a ridesharing context to avoid some of the challenges encountered
within our work.
REFERENCES
[1] Khaled Albusays, Stephanie Ludi, and Matt Huenerfauth. 2017. Interviews
and Observation of Blind Software Developers at Work to Understand Code
Navigation Challenges. In Proceedings of the 19th International ACM SIGAC-
CESS Conference on Computers and Accessibility (Baltimore, Maryland, USA)
(ASSETS ’17). Association for Computing Machinery, New York, NY, USA, 91–100.
https://doi.org/10.1145/3132525.3132550
[2] Pranjal Protim Borah and Keyur Sorathia. 2019. Direct Observation of Tactile
Geometric Drawing by Visually Impaired and Blind Students. In Proceedings of
the 10th Indian Conference on Human-Computer Interaction (Hyderabad, India)
(IndiaHCI ’19). Association for Computing Machinery, New York, NY, USA, Article
11, 10 pages. https://doi.org/10.1145/3364183.3364185
[3] Robin N. Brewer, Amy M. Austin, and Nicole B. Ellison. 2019. Stories from
the Front Seat: Supporting Accessible Transportation in the Sharing Economy.
Proc. ACM Hum.-Comput. Interact. 3, CSCW (Nov. 2019), 95:1–95:17. https:
//doi.org/10.1145/3359197
[4] Robin N. Brewer and Vaishnav Kameswaran. 2019. Understanding Trust, Trans-
portation, and Accessibility through Ridesharing. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glas-
gow, Scotland Uk, 1–11. https://doi.org/10.1145/3290605.3300425
[5] Julian Brinkley, Brianna Posadas, Julia Woodward, and Juan E. Gilbert. 2017.
Opinions and Preferences of Blind and Low Vision Consumers Regarding Self-
Driving Vehicles: Results of Focus Group Discussions. In Proceedings of the 19th
International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS
’17. ACM Press, Baltimore, Maryland, USA, 290–299. https://doi.org/10.1145/
3132525.3132532
[6] Jong Kyu Choi and Yong Gu Ji. 2015. Investigating the Importance of Trust on
Adopting an Autonomous Vehicle. International Journal of Human–Computer
Interaction 31, 10 (Oct. 2015), 692–702. https://doi.org/10.1080/10447318.2015.
1070549
[7] Audrey Demmitt. [n. d.]. The Transportation Problem: Finding Rides
When You Can’t Drive - Visually Impaired: Now What? - Vision-
Aware. https://www.visionaware.org/blog/visually-impaired-now-what/the-
transportation-problem-fnding-rides-when-you-can%E2%80%99t-drive/12
ASSETS ’22, October 23–26, 2022, Athens, Greece
[8] Tawanna R. Dillahunt, Vaishnav Kameswaran, Linfeng Li, and Tanya Rosenblat.
2017. Uncovering the Values and Constraints of Real-time Ridesharing for Low-
resource Populations. In Proceedings of the 2017 CHI Conference on Human Factors
in Computing Systems - CHI ’17. ACM Press, Denver, Colorado, USA, 2757–2769.
https://doi.org/10.1145/3025453.3025470
[9] Tawanna R. Dillahunt and Amelia R. Malone. 2015. The Promise of the Sharing
Economy Among Disadvantaged Communities. In Proceedings of the 33rd Annual
ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, New
York, NY, USA, 2285–2294. https://doi.org/10.1145/2702123.2702189 event-place:
Seoul, Republic of Korea.
[10] Jinging Jiang. 2019. More Americans are using ride-hailing apps.
https://www.pewresearch.org/fact-tank/2019/01/04/more-americans-are-
using-ride-hailing-apps/ Library Catalog: www.pewresearch.org.
[11] Vaishnav Kameswaran, Lindsey Cameron, and Tawanna R. Dillahunt. 2018.
Support for Social and Cultural Capital Development in Real-time Rideshar-
ing Services. In Proceedings of the 2018 CHI Conference on Human Factors in
Computing Systems (CHI ’18). ACM, New York, NY, USA, 342:1–342:12. https:
//doi.org/10.1145/3173574.3173916 event-place: Montreal QC, Canada.
[12] Vaishnav Kameswaran, Jatin Gupta, Joyojeet Pal, Sile O’Modhrain, Tifany C.
Veinot, Robin Brewer, Aakanksha Parameshwar, Vidhya Y, and Jacki O’Neill.
2018. ’We Can Go Anywhere’: Understanding Independence Through a Case
Study of Ride-hailing Use by People with Visual Impairments in Metropolitan
Earl W. Huf et al.
India. Proc. ACM Hum.-Comput. Interact. 2, CSCW (Nov. 2018), 85:1–85:24. https:
//doi.org/10.1145/3274354
[13] J. Lee, H. Chang, and Y. I. Park. 2018. Infuencing Factors on Social Acceptance
of Autonomous Vehicles and Policy Implications. In 2018 Portland International
Conference on Management of Engineering and Technology (PICMET). 1–6. https:
//doi.org/10.23919/PICMET.2018.8481760
[14] National Federation of the Blind. 2019. Blindness Statistics. https://nfb.org/
resources/blindness-statistics
[15] Arulanandam Jude Niranjan and Geert de Haan. 2018. Public Opinion About
Self-Driving Vehicles in the Netherlands. In Proceedings of the 36th European
Conference on Cognitive Ergonomics (ECCE’18). ACM, New York, NY, USA, 19:1–
19:4. https://doi.org/10.1145/3232078.3232080
[16] U.S. Department of Transportation. [n. d.]. Transportation Difculties Keep
Over Half a Million Disabled at Home | Bureau of Transportation Statis-
tics. https://www.bts.gov/archive/publications/special_reports_and_issue_
briefs/issue_briefs/number_03/entire
[17] Anon Ymous, Katta Spiel, Os Keyes, Rua M. Williams, Judith Good, Eva Hornecker,
and Cynthia L. Bennett. 2020. "I Am Just Terrifed of My Future" — Epistemic
Violence in Disability Related Technology Research. In Extended Abstracts of
the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI,
USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA,
1–16. https://doi.org/10.1145/3334480.3381828
 A Collaborative Approach to Support Medication Management
in Older Adults with Mild Cognitive Impairment Using
Conversational Assistants (CAs)
Niharika Mathur Kunal Dhodapkar Tamara Zubatiy
nmathur35@gatech.edu kunal.dhodapkar2012@gmail.com tzubatiy3@gatech.edu
School of Interactive Computing School of Interactive Computing School of Interactive Computing
Georgia Institute of Technology Georgia Institute of Technology Georgia Institute of Technology
Atlanta, Georgia, USA Atlanta, Georgia, USA Atlanta, USA

Jiachen Li Brian D.Jones Elizabeth D. Mynatt

jli986@gatech.edu brian.jones@imtc.gatech.edu
Georgia Institute of Technology Institute for People and Technology
Atlanta, USA Georgia Institute of Technology
Atlanta, USA
ABSTRACT
Improving medication management for older adults with Mild Cog-
nitive Impairment (MCI) requires designing systems that support
functional independence and provide compensatory strategies as
their abilities change. Traditional medication management inter-
ventions emphasize forming new habits alongside the traditional
path of learning to use new technologies. In this study, we navi-
gate designing for older adults with gradual cognitive decline by
creating a conversational “check-in” system for routine medication
management. We present the design of MATCHA - Medication Ac-
tion To Check-In for Health Application, informed by exploratory
focus groups and design sessions conducted with older adults with
MCI and their caregivers, alongside our evaluation based on a two-
phased deployment period of 20 weeks. Our results indicate that
a conversational “check-in” medication management assistant in-
creased system acceptance while also potentially decreasing the
likelihood of accidental over-medication, a common concern for
older adults dealing with MCI.
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility.
KEYWORDS
mild cognitive impairment, older adults, medication management,
conversational assistants
This work is licensed under a Creative Commons Attribution International
4.0 License.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3544830
e.mynatt@northeastern.edu
Northeastern University
Boston, USA
ACM Reference Format:
Niharika Mathur, Kunal Dhodapkar, Tamara Zubatiy, Jiachen Li, Brian
D.Jones, and Elizabeth D. Mynatt. 2022. A Collaborative Approach to Sup-
port Medication Management in Older Adults with Mild Cognitive Im-
pairment Using Conversational Assistants (CAs). In The 24th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22),
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 14 pages.
https://doi.org/10.1145/3517428.3544830
1 INTRODUCTION
In this research, we explore the potential of supporting medica-
tion management in older adults with Mild Cognitive Impairment
(MCI) through the use of Conversational Assistants (CAs) which
include smart speakers and assistants such as the Google Assistant,
Siri, Amazon Alexa or Microsoft Cortana. They provide a natural
language based support to enable access to information and ser-
vices for everyday interactions [7]. As of 2019, 20% of people over
the age of 60 owned a smart speaker in their homes [18]. While
adoption rates by older adults are lower, many older adults have
reported feeling “supported” and “empowered” while interacting
with any form of smart assistant in their homes [21, 46]. Assistants
equipped with distinctive personality characteristics such as voice
modulations, learning behaviors, and advanced natural language
processing contribute to the anthropomorphization of these devices
by users, particularly older adults, who tend to draw upon their
experiences and behaviors cultivated over their lifetime. Our work
is motivated by these observations, seeking to foster a welcome
connection between users and CAs that can positively infuence
healthy behaviors such as efective medication management.
Most medication management interventions support building
habits over time by providing an integrative system for reminders
or alerts [5, 31]. An additional challenge we address in our work
is designing an interactive medication management system for an
aging population diagnosed with gradual cognitive decline. Mild
Cognitive Impairment is defned as an intermediate stage of cog-
nition situated between the expected decline due to aging and the
more signifcant decline associated with dementia and Alzheimer’s
disease [33]. It is estimated that between the years 2012 and 2050,
the US population over the age of 60 is expected to double [29],
ASSETS ’22, October 23–26, 2022, Athens, Greece
and about 16.6% of the population above 65 will develop MCI [34].
Symptoms of MCI, which include varying degrees of memory loss,
language problems, and loss of attention among many others, have
shown to gradually become worse, and very few patients show a sig-
nifcant cognitive improvement. It is estimated that approximately
14.9% of individuals with MCI will progress to dementia in over 2
years and 38% will progress to dementia in 5 years [16]. Therefore,
while we do want to encourage and support habit-building in older
adults with MCI through our design, we also attempt to under-
stand an optimum level of persistence and support required. We are
aiming to fnd a balance between a system that delivers repeated
notifcations (which may be suitable for people with dementia or a
more advanced cognitive decline than MCI) and one that takes into
account the varying cognitive strengths of people with MCI and
remains optimally attentive to sustain healthy medication goals,
including preventing accidentally taking the same medication twice
(referred as “over-medication” throughout this paper). With our
work, we aim to ground MCI as a critical period to empower older
adults to develop habits and compensatory strategies that they can
rely on if or when their cognitive abilities decline further.
A key foundation for our work are existing studies with older
adults with MCI and their usage of CAs for daily activities, such as
in [46]. CAs have shown to provide useful and usable support for
older adults with MCI across a variety of functions, from informa-
tion searching to calendaring, especially when sufcient training is
provided. For the purpose of our study, we chose to use Google’s
CA, Google Home Hub, a visual assistant with an 8 x 6 inch touch
screen and a built-in Google Assistant. A deciding factor in choos-
ing this device was the display of voice commands and responses on
the screen, eliminating the need for our participants to remember
what they said. In this paper, we frst describe our exploratory user
research process, which informed our design of an interactive medi-
cation assistant system built within the Google Home Hub. We then
discuss the deployment of this system in households that included
older adults with MCI, focusing on their usage and engagement
patterns over the course of 20 weeks divided between 2 Phases. In
this two-phase research study, we explore the impact of ‘learning
from use’ and inform design revisions based on usage and feedback
from Phase 1. Finally, we summarize our fndings from both phases
and frame opportunities for future work.
1.1 Contributions
With this work, we aim to ofer the following contributions to the
larger research community:
• We ofer insights into the existing medication management
strategies and habits of older adults with MCI and their
expectations for an interactive medication management sys-
tem that integrates with these strategies. We also identify
opportunities for this system to address gaps with existing
strategies as well as limitations to traditional alarm and re-
minder based interventions.
• We present the design of an interactive system built within
a CA for fexible medication management and deploy the
design in a two-phase research study. Our system design is in-
formed by possible usage scenarios drawn from exploratory
Mathur, et al.
user research sessions with older adults with MCI and their
carepartners.
• We articulate how building a medication system which prompts
refection from its users and focuses on “checking-in” rather
than traditional reminders or alarms led to increased en-
gagement with the system over time while also discouraging
accidental over-medication in older adults with MCI.
2 RELATED WORK
In this section, we discuss why medication management in older
adults with MCI remains a challenging area of research and the
various factors that can lead to the inefectiveness of existing inter-
ventions in the context of older adults. We also discuss the additional
and peripheral cognitive challenges that they face, and highlight
the need for more empathetically designed systems. We then re-
view and discuss the adoption of CAs by older adults and identify
potential areas of opportunities.
2.1 Medication adherence in older adults: A
persistent challenge
Many past studies have sought to reduce barriers to efective medi-
cation management in aging populations. In [25], Martin et al. dis-
cuss various psychological needs and requirements for medication-
related artifacts representing information to older adults, such as
text size, language used, and amount of information presented. Ful-
mer et al. note that cognitive power and mental health conditions
such as depression can signifcantly impact medication behaviors
[13]. Studies report that 25 to 59% older adults above 65 are not
able to take medications as prescribed and report higher instances
of complications arising as a result of those. [4, 39].
These challenges are further complicated by neurological issues
associated with MCI and aging, such as dexterity limitations, vision
and memory loss. Some older adults with MCI also struggle with
reduced awareness of their circadian rhythms, placing additional
stress on their ability to adhere to routines [8, 22]. Low adherence
is also attributed to lack of personalized and alternate ways to
keep track of medications and efcient contextual reminders de-
spite feeling a need for it [14]. Medication management forms an
integral component of the daily lives of older adults, they have
scheduled medications not only for specifc health issues, but also
for preventive and maintenance use, such as vitamin and dietary
supplements. Remembering to take medications multiple times a
day can be challenging for older adults with MCI and also for their
caregivers, who often have a parallel medication schedule of their
own [2]. Medication management for older adults remains an es-
sential area of research, as poor adherence to prescribed medicines
is considered to be a signifcant health challenge [15] and one of
the leading causes for emergency complications arising from issues
related to heart disease, diabetes, hypertension and mental health
[4, 15].
2.2 Role of technology in Medication
management in older adults
A number of research prototypes and commercial products aim to
incorporate technology into systems for medication management.
However, the bulk of studies with older adults and their interactions
Medication Management in Older Adults with MCI Using Conversational Assistants
with technology currently focus on the roadblocks stemming from
the technological stereotypes associated with older adults, who are
often “defned by their defcits rather than capabilities” [32, 44].
Currently, most commonly-available tech-based interventions to
support medication management are aided by the use of smart-
phones [41], alarms, reminders [23], automated pill dispensers [28],
and digital calendars [3]. Similar to other technologies oriented to
the consumer market, there are many challenges associated with
using these interventions with older adults. Some older adults, par-
ticularly those with a relatively recent diagnosis of MCI, report
feeling distressed by a lack of control over their ability to main-
tain a schedule. This fear of losing control and independence also
becomes a contributing factor in their hesitation to adopt newer
technologies [32]. There is a need to develop systems that provide
a degree of functional independence to individuals with MCI while
also supporting them through the journey of cognitive transition
stemming from MCI. The other challenges with technological in-
terventions arise primarily out of the friction associated with their
costs, on-boarding and technical complexity [31]. Patel et al. in [30]
reviewed these challenges in a usability and workload compari-
son of 21 commercially-available electronic medication adherence
products with older adults and caregivers such in [9]. They re-
ported that the usability of these products is signifcantly lower
than the national average usability scale score [30]. An analysis of
3 of-the-shelf commercially-available automatic pill dispensers -
Hero, Pria and Medacube - revealed an average annual cost of more
than $300 with additional on-boarding charges. The set up and
maintenance requirements of pill dispensers are also challenging
for older adults [37]. Additionally, most of the existing strategies,
consisting of reminders and alarms, introduce the risk of accidental
over-medication by simply reminding to take medication. When
reminded about taking medication, older adults with MCI have a
high chance of not being able to recall if they have already taken
the medication and take it again [2]. Reminders and alarms also
run the risk of becoming too persistent by requiring the user to
keep snoozing or asking it to stop [38], leading to a drop in en-
gagement levels over time resulting from alarm fatigue [32]. While
there is some research about specialized medication management
systems specifc to older adults [31], very little, if any, research
proposes the integration of such systems within existing practices
of older adults with cognitive defcits, while also involving them in
the design process. There is a potential to develop systems which
integrate and utilize the connection that older adults form with CAs
and provide them a more coherent experience [36]. Given this, our
work strongly advocates and aims to represent an assets-based de-
sign approach [19] which efectively integrates within the existing
strategies used by older adults to manage medications.
2.3 Adoption and use of CAs by older adults
In recent years, there has been a substantial amount of research
that explores the potential of CAs, either embedded within smart-
phones or as standalone devices, in helping older adults for health
and well-being, as well as for entertainment purposes given their
interactive and multimodal input capability [6, 11, 45, 46]. In [27],
Morrow et al. develop a framework adopting the use of CAs from
pedagogical purposes to promoting self-care for older adults. Their
ASSETS ’22, October 23–26, 2022, Athens, Greece
framework highlights the assistant’s need to function dynamically
through the evolving stages of any type of cognitive defcit, from
acceptance to establishing routines and fnally sustaining engage-
ment with the system. While most of the existing work focuses
on older adults and their interactions with CAs, the challenge of
designing for compensatory cognitive support is underexplored,
with the exception of a few studies that address cognitive behaviors
as a result of MCI [20, 45, 46]. These studies emphasize following a
user-centered approach for personalization given the diversity in
the MCI spectrum, accounting for hearing and dexterity limitations
and to incorporate pleasant interactions that are able to sustain
engagement beyond the study. Based on analyses from studies that
have explored long-term interactions between older adults and CAs
[10, 17, 46], it is observed that the initial interactions were mainly
related to entertainment (music, searching, etc.), cooking related
reminders, and alarms related to healthcare monitoring, highlight-
ing the potential for CAs to aid the maintenance of routine tasks.
An important aspect of older adults and their usage of CAs is their
tendency to anthropomorphize the device as a companion within
their home environment [6, 35]. They are likely to interact posi-
tively with the system through socially engendered responses as
a result of the voice-based interaction that can facilitate a longer
retention of healthcare related information, as demonstrated by
Azevedo et al. in [1]. Waterschoot et al. [42] highlight the need to
think about multi-stakeholder interactions with the CA such as the
older adults, their caregivers and healthcare professionals with a
controlled sharing of information. Designing for gradual behavioral
change as a result of aging through multimodal interactions in the
form of a conversational coach and a text-based system is explored
in [12] and the results indicate that a system that provides multiple
interaction afordances was positively accepted by the participants.
Finally, a critical aspect of designing an interactive system for older
adults with MCI is to retain a degree of autonomy and functional
independence related to personal health routines and decisions
through customizable interaction options [40].
We defne our research and design goals to use these insights
from existing studies and gaps identifed within them to design a
medication management system which reduces the complexity of
the set up and maintenance process, works with the existing strate-
gies that older adults are used to, takes the cognitive decline and
the specifc nature of the decline into account while also retaining a
degree of control over their own system, is optimally persistent and
not too repetitive, and fnally, is designed to reduce the instances
of over-medication to the extent possible.
3 OVERVIEW OF RESEARCH APPROACH
This research study is conducted within the context of a larger
comprehensive cognitive program in a hospital for older adults di-
agnosed with MCI. This program provides lifestyle and therapeutic
interventions with a focus on exercise, nutrition, functional inde-
pendence, group therapy and compensatory cognitive strategies
working with older adults with MCI and their caregivers, who are
mostly their spouses or adult children. The program aims to "em-
power” older adults through an array of restorative activities such
as yoga, meditation, group sessions, nutrition counseling, smart
home installations, etc. with the aim of slowing cognitive decline
ASSETS ’22, October 23–26, 2022, Athens, Greece
and protecting overall brain health. The program refers to individu-
als with MCI as “members”, not patients, and calls their caregivers
as “carepartners” to signify a mutual partnership and an active
commitment to each other. Within this context, the program has
ongoing studies about the use of CAs to support members and
carepartners in their daily lives. All the participants in our study
were enrolled in this year-long program and had an existing di-
agnosis of MCI. More details about participant characteristics are
mentioned in sections 4 and 6.
To gain a better understanding of the day-to-day experience of
older adults living with MCI, we attended 4 weekly group sessions
ofered by the program. During these sessions, "dyads" of members
and their carepartners interact with each other and the program
staf via a Zoom call, keeping each other updated on their progress
and occasionally sharing personal life events. During two of these
sessions in Month 1, we introduced them to our study goals and
scheduled user research sessions with interested participants. All
the participating dyads were located in the same city, however,
the research sessions were conducted remotely via Zoom due to
COVID-19 research protocols. All members had a diagnosis of MCI
from their neurologists and were actively enrolled in the program
activities. As active participants, all the dyads in the study had the
Google Home Hub installed in their homes for over a month and
were aware of the keywords required to initiate conversation with
it. We also worked through a Privacy Impact Assessment to under-
stand privacy questions and concerns addressed by MacLeod et al.
in [24] and provided a summary of data collection and management
protocols to each participant. The purpose of these research ses-
sions was to gain a better understanding of the existing strategies
for medication management as well as the expectations from CAs
in the context of medication management. In the next section, we
describe these research sessions and the insights gathered, which
informed the design goals for our system. We then deployed the
system with 7 dyads, in 2 phases for a total period of 20 weeks. We
evaluated usage mid-deployment and iterated on our design be-
tween the two phases. The Phase 1 deployment of our study lasted
for 4 weeks. Following initial evaluation of usage and incorporat-
ing design revisions based on our fndings, we then deployed the
system for an additional period of 16 weeks (Phase 2). Finally at the
end of Phase 2, we again conducted interaction logs analysis and
interviews with the participants, which informed our fnal study
takeaways. A visual timeline of the study is shown in Fig. 1.
4 EXPLORATORY USER RESEARCH
To understand existing medication practices and expectations for
the system, we conducted two focus group sessions and one scenario-
based design session with groups of older adults with MCI and their
carepartners. The insights gathered from these user research ses-
sions helped us to formulate concrete design goals for our system. In
the following sections, we describe these sessions and our analysis.
The participants for these sessions were all recruited and sampled
from the cognitive program introduced in section 3. All partici-
pating dyads had a greater than 2 daily medication frequency and
used a variety of strategies to manage medications that we were
interested in learning more about.
Mathur, et al.
4.1 Focus Group Sessions
We conducted 2 focus groups with our participants to understand
their existing medication management strategies and to introduce
them to our study goals. The sessions were conducted remotely
via Zoom and adhered to COVID-19 research protocols in place at
the time. Verbal consent was obtained from all the participants to
record the session and share the data with the researchers on the
team as described in the approved Institutional Review Board (IRB)
protocol. The participants were all recruited from the cognitive
program introduced in section 3. The 1st session had an attendance
of 18 dyads, out of which 17 members had their respective spouses
as carepartners and 1 member was in a member-daughter dyad
structure (36 total participants). The second session was conducted
in a similar format with 12 dyads, with 11 members with their
respective spouses as carepartners and 1 member in a member-
child structure with 3 adult children (1 male; 2 female), located in
diferent cities (26 total participants). We divided the 45-minute
sessions into 3 sections, shown to the participants through a shared
screen presentation. The 1st section had multiple choice questions
that prompted the participants to indicate how often they took
medications, where in their home medications were located, and
where the CA was situated in their house. In the 2nd section, we
focused on facilitating a discussion among the participants around
their personal habits or tips for remembering medications. This
section also prompted a dialog among the carepartners and their
involvement in the medication management routine of the members.
In the 3rd section, we informally recruited members to be involved
in subsequent design activities as well as the eventual deployment.
After the sessions, we made an afnity diagram from our notes
and transcripts to extract key insights from the data gathered. This
inductive approach helped us to group the learnings from the ses-
sion into four categories (Expectations, Habits, Concerns, Current
Techniques), presented in Fig. 2. This analysis then helped us to
formulate the following design goals from this session:
(1) The traditional use of alarms or reminders that lack context
or labels to signify the purpose of that alarm or reminder,
often accompanied with a repetitive sound, is considered to
be unreliable and inefectual. Participants reported that they
often had trouble recollecting what an alarm or a reminder
was originally set for. This insight helped us understand the
need for specifc and unambiguous messaging associated
with an alarm or reminder.
(2) Physical or digital spreadsheets, pillboxes, and paper cal-
endars were the most common ways of keeping track of
medications. Some members also reported relying on their
carepartners to be in control of their medications. While
these methods provided a certain level of robustness for the
dyads, they also expressed interest in exploring how their
CAs can help them streamline this process by reducing the
need to remember to check their calendars or sheets every
day, often multiple times in a day, and to also reduce the
carepartner burden.
(3) Participants did not want to completely abandon their ex-
isting medication management methods such as using pill-
boxes, sticky notes, etc., since they had been a part of their
routines for many years now. This suggested that any new
Medication Management in Older Adults with MCI Using Conversational Assistants
Figure 1: Month-wise research study timeline
digital medication management tools that we design for them
should work with, and not replace, these existing methods.
4.2 Scenario-Based Design Session
To probe deeper into the learnings from the focus groups and to
understand the expectations of the participants for CAs supporting
medication tasks, we conducted a 45-minute design session with
the goal of involving the participants in the design process. The ses-
sion was conducted remotely via Zoom due to COVID-19 protocols
and had 18 participants, all of whom also participated in the focus
groups previously. The participants for this session were also re-
cruited and sampled from the cognitive program and had a greater
than 2 daily medication frequency. 8 members attended the session
with their carepartners and 2 members attended unaccompanied.
While designing the narrative structure of the session, we decided
to partially anthropomorphize the CA as a humanoid robot called
Rosey, inspired from the popular 60s animated sitcom The Jetsons,
who worked as the Jetson family’s housekeeper. This depiction
helped spur engagement with participants during the session by
grounding the discussions in a playful, familiar metaphor [43]. We
divided the session into 3 parts. In the 1st part, we introduced the
session structure and presented example scenarios, such as Rosey
reminding them to call their kids. Building on this example, in the
second part, we presented the participants with a storyboarded
scenario of Rosey interacting with them at home. We kept the
conversational aspects of the storyboard interactive using empty
speech bubbles and giving the participants the chance to design
the conversation between them and Rosey by talking about their
dialog preferences and typing them in speech bubbles. This exercise
helped us to understand their expectations for the system. In the 3rd
part, we touched upon the degree of involvement that they expect
from the system by presenting multiple scenarios through story-
boards in which the robot varies its level of persistence with the
reminders by asking once or twice or continually until an answer.
We then asked the participants to think aloud and discuss their
preferences for these interaction levels. This exploratory design
session helped us in understanding their attitude and perception
ASSETS ’22, October 23–26, 2022, Athens, Greece
towards the involvement of CAs. We present the codes developed
based on the analysis of insights from this session in Table 1 that
later informed our system design. Using the codes in Table 1, we
extracted the following tangible design goals from this session:
(1) In most scenarios, the participants hinted at their preference
for a system that checks in with them as opposed to an alarm
or reminder. In traditional alarms and reminders, the systems
generally do not prompt the members to answer, since they
are worded as “Time for medicine/take medicine” or similarly.
An individual with MCI might not remember that they have
taken the medication already, and in the moment, might
unknowingly take it again. They could also unknowingly
take the medication for the next day as a result of being told
by the system to take it. Given that “over-medication” is also
a challenge for them, our system needs to “check-in” and
ask the member if they took the medication or not, and not
“remind”. This check-in introduces the possibility to prompt
the member to think about whether they took the medication
or not, go and check their pillbox or calendar if they need
to, and then report back to the system.
(2) Another crucial expectation from the system is for it to have
an understanding of the member’s routines and dialog pref-
erences. If the timing of initiation conficts with an ongoing
activity, it should have the functionality to check in again
later, or allow the member to schedule it for a later time.
If the assistant receives no response after initiating, there
should be multiple channels of notifcations to confrm the
check-in. Suggestions for these channels included phone
notifcations, text messages, emails, and calls.
(3) When the member has already taken the medication, and
if they choose to notify the system of it, it should generate
positive feedback or reinforcement to motivate the user. This
positive recognition can reinforce feelings of achievement
of having accomplished the task beforehand.
ASSETS ’22, October 23–26, 2022, Athens, Greece Mathur, et al.

Figure 2: Insights from the focus group sessions categorized into expectations, concerns, habits and current techniques

Table 1: Codes and sample quotes from the scenario-based design session conducted with participants

Codes Sample Quotes From the Scenario-Based Design Session ("She" refers to Rosey)
Positive “Maybe it can play my favorite song after I answer it back..”
feedback “How about a movie star’s voice to applaud me? Maybe James Earl Jones!”
Lacking in “Sometimes when I hear my phone alarm, I take the medicine and then later my wife tells me
current that I’d taken it before already”
systems “I have to remember to check the calendar every day, my wife helps me with that.."
“We set alarms for our medicines but it’s hard to remember what the alarm was set for because
we have many alarms”
“Check-in” “She should ask me whether I have taken the pills because that way I can think about it or ask
and my wife, rather than telling me frmly that I should”
not remind “It should ask me to go check, its hard but it can help me making sure that I’m just not taking the
same pill again”
Dialog “I hope she does not talk like the reminder in my phone, it’s dull..”
expectation “She should be respectful and not nosey like my alarm I keep snoozing”
Multiple “If we’re not at home, can she send a message to our phone?”
channels of “What if I’m outside walking the dog when she wakes up?”
notifcation “Maybe it can send a message to my daughter’s mobile saying that I have taken the medicine for
the morning”
Check again “I expect her to come back inside and check again after we’re done with yoga”
“But I hope she knows when is a good time. Can I tell her when to come back?”

5 DESIGNING A GOOGLE HOME ACTION FOR Console and can be integrated into any device that supports Google
MEDICATION Assistant.
We listed the scenarios and structured the dialog fow based
From the insights generated during the exploratory user research
on the design goals informed by the discussions during the focus
sessions, we designed our Google Action, “Medication Action To
group and exploratory design sessions, with appropriate paths for
Check-In for Health Application” (MATCHA), using Google’s Ac-
each scenario. Table 2 presents possible scenarios and the resulting
tion Console, a web-based tool to manage the development, reg-
MATCHA response, as well as the source of the insights that led
istration, confguration and analysis of Google Actions. A Google
to our designated response. Our system recognizes more than 200
Action is an applet for Google Assistant that provides additional
possible unique responses from the dyads. The assistant’s volume of
and extended functionality. Google Actions are coded in the Action
Medication Management in Older Adults with MCI Using Conversational Assistants ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 2: Possible scenarios, MATCHA response and informing insight

No. Possible Scenario MATCHA Response Corresponding Source of Insight

S1 Taken the medication be- Plays cheering sounds and ap- The need to foster positive engage-
fore MATCHA checks-in plauds them with positive afrma- ment with MATCHA and reinforce-
tion ment of medication management.
S2 Not taken the medication Asks if they would like to repeat The need for the check-in to be per-
before MATCHA checks- the check-in at a later time and sistent and let the member specify a
in guides them to specify a time pe- later time for check-in
riod after which MATCHA should
check-in again
S3&4 Does not remember if they Asks to check and waits for them The expectation to reduce cognitive
have taken the medication to respond again and responds fur- burden on the member to remember
or not before MATCHA ther according to scenario 1 or 2 whether they have taken the medica-
checks-in AND/OR Needs tion. Goal with the pillbox is to catch
to check the pillbox to con- missing medications but also reduce
frm risk of over-medication through spe-
cifc reminders.
S5 Says something Google Lets them know that it has sent The preference to utilize the exist-
does not understand - un- the feedback to the research team ing connection between the members
known response to rectify the error and we will be and the conversational assistant to
in touch soon foster support and feedback
S6 Not around to respond or Sends a notifcation on the mem- The expectation to engage multiple
MATCHA receives no re- ber and the care partner’s phones channels of notifcations to confrm
sponse until it is timed out to notify that the routine has the check-in
started

operation as well as the physical location of the Google Home Hub
was infuenced by house layout and preferences. Some members
keep their medications closer to their Google Home Hub, while oth-
ers keep the two on diferent levels in the house. Our data reported
that the kitchen and the bathroom were the most common locations
for the Google Home Hub and the medications respectively. These
specifcs of the location factored into the medication assistant’s
pitch, volume and time functions for the dyads. The assistant also
addressed the member with a greeting and their name with every
check-in.
We conducted the deployment in two successive phases. We
frst introduced MATCHA to the participating dyads by conducting
training sessions at their houses and also provided them with a set of
printed training materials explaining the purpose and interaction
fow of MATCHA, and ways to contact us in case of questions.
We then personalized the medication schedules for each dyad by
obtaining medication routines during a pre-deployment interview
and incorporated these individual routines into MATCHA through
the Google Assistant backend on the Google Home app. After Phase
1, we conducted interviews and interaction logs analysis to inform
design revisions for Phase 2. At the end of Phase 2, we conducted
a fnal set of interviews and interaction logs analysis to inform
overarching takeaways from the study.
6 LEARNING FROM USE: DEPLOYMENT
PHASE 1
In Phase 1, we deployed MATCHA to the Google Home Hubs of 7
dyads for a period of 4 weeks. These 7 dyads were a subset of the
dyads from the research sessions and were sampled on the basis
of their willingness to participate in the study and their level of
engagement with the CA. The dyads primarily belonged to subur-
ban communities, and had a diverse professional background with
3 members having retired from technical professions, 2 from an
art background, 1 from a clinical support background and 1 from
academia. MATCHA triggered at specifc times each day depending
on the member’s medication schedule. The average frequency of
MATCHA reminders for the 7 dyads was twice a day, with the
lowest frequency being once in a day and the highest being 5 times
in a day. The medication frequency is the total number of times per
day that a member takes a set of medicines, for example, if once in
the morning and once in the evening, the medication frequency is 2.
A summary of the demographic data including sex, age, medication
frequency is provided in Table 3.
After Phase 1, we collected interaction log data from the 4 weeks
and manually transcribed them from the “My Activity” toolbar
in the Google Home app. We also sent a modifed version of the
System Usability Scale (SUS) to the members and carepartners to
respond to, reworded to better represent our study context, with
the goal of calibrating our assessment of the interaction log data.
The SUS results gave us a usability score of 84.66 from the mem-
bers, and 86.16 from the carepartners. Overall, the log data and SUS
indicated sustained use for most participants but presented poten-
tial areas for improvement and revisions. Based on that input, we
conducted qualitative interviews with the 7 dyads via Zoom lasting
for 30-45 minutes, to gain an understanding of their experience
with MATCHA so far and to contextualize the patterns seen in the
interaction logs and SUS.
ASSETS ’22, October 23–26, 2022, Athens, Greece
6.1 Phase 1 Analysis
The insights from the interviews, interaction logs and the SUS
score after Phase 1 resulted in a set of design revisions that we
incorporated before the deployment of Phase 2. Below, we present
some interview quotes from Phase 1 analysis that, along with logs
and SUS analysis, resulted in the design revisions as described in
Section 6.2:
(1) M6: "To be honest, the loud cheers were a little overwhelming.
It kept going on and on."
(2) CP1: "Sometimes Google doesn’t understand what I am saying,
she keeps asking me to repeat."
(3) CP2: "I would like to get a text or email whenever he has taken
a medicine."
(4) M5: "There are days when I take the pill after Google tells me,
does she know that?"
A breakdown of scenario-wise interactions, including the "Tak-
ing Now" scenario and engagement rate for Phase 1 is discussed
further in the Results section.
6.2 Design revisions after Phase 1
Major design revisions informed by the interaction log analysis,
SUS results and the interviews included tamping down the level of
positive feedback from MATCHA - while some members found the
length and nature of the positive reinforcement uplifting, others
found it “overwhelming”. To be specifc, the frst positive feedback
started with a 4-second loud cheering and celebratory sounds, fol-
lowed by a 4-second clapping sound and ended with the assistant
verbally appreciating the user saying "Yay! Good Job! I’ll check-in
at the next medication time". The feedback was overwhelming for
some members, particularly for those with multiple medications
through the day therefore, in Phase 2, we revised an adequate level
of positive feedback, with a 2-second moderate cheer accompanied
by the same verbal praise from the assistant. We also included a
touch-based response option in which the members can press the
corresponding button on the touch screen in lieu of answering ver-
bally. We added 4 touch buttons - “Yes I did”, “No I did not”, “I don’t
remember” and “Taking Now” in Phase 2 based on the insights from
the interviews. This modifcation also provided an alternate way of
providing a response in the case of mufed speech or other speech-
based issues. To distinguish between responses that were recorded
through touch buttons and ones that were recorded through verbal
response in the interaction logs, we added an emoji to every button
text to recognize this response type in the logs.
Additionally, we increased the system time-out period to receive
a response, and added an additional ‘Taking now’ scenario in which
the member informs MATCHA of taking the medication right now
as a result of the check-in. We noticed a large number of such
responses and separated those from the unknowns and counted
them manually through the backend under a new category called
"Taking Now". We also revised the settings for phone notifcations
to be delivered every time MATCHA is triggered, as opposed to only
when it receives no response. This was informed by the carepartners
expressing the need to know every time the member has responded
to the system. We further explain and contextualize these revisions
with participant quotes in the results section (Section 8).
Mathur, et al.
7 LEARNING FROM USE: DEPLOYMENT
PHASE 2
For Phase 2, the revised version of MATCHA was deployed to the
Google Home Hubs of 5 dyads for a period of 16 weeks, with 2
dyads wishing to drop out of the study between Phase 1 and 2. One
dyad with a medication frequency of 5 times in a day found the
frequent check-ins tiresome, and the other dyad was unavailable
due to travel through the summer. We further plan to address the
inactivity of the 2 dyads that dropped out of the study by exploring
more about personal preferences and their incorporation into the
system in our future work. A summary of the demographic data,
including if they continued to Phase 2 is provided in Table 3.
We analyzed the interaction logs for the 16 weeks of the Phase 2
deployment to examine the continuing engagement patterns of the
dyads resulting from the design revisions. Finally, we conducted
interviews with the 5 dyads via Zoom for 45 minutes to understand
their experience and contextualize patterns in the interaction logs.
We used the inductive coding process to develop major themes
across our interviews and usage data, grounded by our initial ex-
ploratory design sessions and their fndings. For each interview,
a team of 3 coders, all researchers with a background in Human-
Computer Interaction, clustered the codes together and as patterns
began to emerge, developed distinctive themes for these clusters. Af-
ter initial coding, a process of inter-rater reliability was conducted
to agree on the fnal themes. At the end of Phase 2, we verifed and
refned the existing themes by looking across the interviews from
both phases and generated concise fndings. We discuss the results
from the interaction log analysis and the interviews from the two
phases in the next section.
8 RESULTS
8.1 Interaction Log and Engagement Analysis
During Phase 1, MATCHA generated 476 initiations and we recorded
a total of 84 interactions (responses) over the course of 4 weeks. The
remaining initiations were left answered. This ratio results in an
engagement rate of around 18% for Phase 1. These interactions are
mapped on a normalized scale of 30 interactions per week for each
scenario in Fig. 3. This normalization per week was done to account
for the diference in the length of the phases. During the analysis,
we noted that there were some responses in which the members
indicated that they have not yet taken the medication but will take
it now as a result of the MATCHA check-in. This scenario extended
the existing functionality of MATCHA as a reminder in addition
to the original check-in function. This scenario, which we call the
“Taking Now” scenario, was not incorporated into MATCHA be-
fore Phase 1, and it prompted an “I don’t understand what you
just said” response from it. This also accounts for the signifcant
number of unknown responses in Phase 1. We manually counted
these interactions from the logs to defne an additional 7th scenario
for the design revision in Phase 2. To be clear, the “taking now”
responses formed a part of the unknown responses by the dyads
and as we had access to the backend of the system, we were able to
count those separately. The unknown interactions do not include
the responses that pertain to the “taking now” scenario and hence
the separated unknowns are labelled as "other unknowns" in Fig. 3.
Medication Management in Older Adults with MCI Using Conversational Assistants ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 3: Normalized number of interactions by scenario in Phase 1 for 7 dyads (normalized per week)

Figure 4: Normalized number of interactions by scenario in Phase 2 for 5 dyads (normalized per week)

Figure 5: Weekly Engagement Rate Trend for all Dyads for 20 Weeks
ASSETS ’22, October 23–26, 2022, Athens, Greece
Table 3: Summary table of participant demographics.
Daily
Member & Member Member
Medication
Carepartner ID Age Sex
Frequency
M1, CP1 62.6 Male
M2, CP2 78 Male
M3, CP3 75.4 Male
M4, CP4 85.3 Male
M5, CP5 70.8 Female
M6, CP6 75.4 Male
M7, CP7 74.3 Female
During Phase 2, MATCHA generated 1120 initiations and we
recorded a total of 760 interactions (responses) over the course of
16 weeks. This ratio of interactions to all initiations results in an
engagement rate of around 67% for Phase 2, showing an increase
from the engagement rate of 18% from Phase 1. We attribute this
increase in engagement to the design revisions we incorporated to
improve the user experience and also an increase in the comfort
level and familiarity with MATCHA forming a part of their daily
routine over time within their CA. If we treat dyads 3 and 4 (from
Phase 1) as outliers that did not adopt the use of MATCHA, we
nevertheless still see an uptick in engagement from Phase 1 to
Phase 2, from 27% (without outliers) to 67%. These interactions are
mapped on a normalized scale of 30 interactions per week for each
scenario in Fig. 4.
To account for the variability in study phase duration and medica-
tion frequency for each dyad, we calculated the weekly engagement
rate for the dyads over the course of the study shown in Fig. 5. This
weekly engagement rate was calculated by adding all the times that
MATCHA initiated for all dyads in 1 week (using each member’s
medication frequency per day), adding all the interactions counted
for all dyads in that week from the log data, and then calculating the
engagement rate for every week of the study. It can be seen from
the fgure that the weekly engagement rate increased signifcantly
right at the start of Phase 2, was sustained during Phase 2 and then
increased towards the end of Phase 2, indicating an overall positive
attitude and a sustained level of comfort with the system over the
course of the study. The slight dip in engagement around Week
10 coincides with a holiday weekend (4th of July). Some dyads
were away from home and only getting reminders on their phones,
which they may or may not have had the time to respond to. The
slight uptick of engagement in the fnal weeks is also correlated
with scheduling interviews at the conclusion of the study.
8.2 Findings from Interviews
In addition to analyzing data from interaction logs, we also con-
ducted interviews with the members and carepartners at the con-
clusion of the study with the goal to understand their perception
towards the system. We asked questions that prompted them to
talk through their experience of using MATCHA and whether they
would like to continue using it. We present the following fndings
from synthesizing our design insights, observed usage and mid
deployment interviews, and integrating our fnal set of participant
interviews with the 5 dyads at the end of Week 20.
Mathur, et al.
Relation to Carepartner Continued
member Age to Phase 2?
3 Spouse 60.9 Yes
2 Spouse 75 Yes
5 Spouse 70.6 No
2 Spouse 76.3 No
2 Daughter 45.7 Yes
1 Spouse 76.7 Yes
2 Spouse 74.6 Yes
8.2.1 The system induced feelings of “confidence” and support to
the dyads while being suficiently persistent in its interactions.
The dyads reported feeling “confdent” about their medication rou-
tines as a result of the medication assistant being integrated into
their existing Google Home Hub routines. This kind of support
also reduced the need to set up external systems such as automatic
medication dispensers that are signifcantly more expensive and
have complex set-up as highlighted in section 2.2. The integration
of the medication system within the CA reduced the on-boarding
friction experienced by our participants with other technologies.
The conversational and unambiguous nature of the check-in as
opposed to the non-contextual and repetitive nature of alarms and
reminders led to a relatively faster adoption of MATCHA through
the course of the study.
• M5: “I like that Google kind of feels like a part of the house,
and not something that I have to keep answering to all the time
like my morning alarm..”
• M7: “I feel assured that if I were to forget my pill, she [MATCHA]
will ask me and that’s important because I feel less pressure to
remember my routine..”
Some members also recollected instances when MATCHA prompted
them to think about or confrm the medication status before re-
sponding as a result of the check-in functionality. In some cases,
they went and checked their pillboxes before reporting completion
if they did not remember taking the medication. The positive afr-
mation from MATCHA after reporting completion also generated
assurance of having a degree of control over their own memory
and subsequently getting rewarded for it.
• M5: “When I hear Google praise me, I feel good and like I have
accomplished something and that she is happy with me..”
All the participating dyads in Phase 2 expressed the desire to con-
tinue using MATCHA in their Google Home Hubs after the com-
pletion of the study. Their use, now at around 55 weeks, continues
through the writing of this paper, which is a signifcant result for
us given the need to sustain long-term engagement of technology
with older adults, as mentioned in section 2.2 in [38]. We discuss
the long-term engagement requirement further in the discussion
section.
8.2.2 The medication assistant provided an alternate way for the
carepartner to monitor member’s medication schedule.
We noted that most members reported being supported by their
carepartner in managing their medication schedules. During the
Medication Management in Older Adults with MCI Using Conversational Assistants
interviews, the carepartners reported having MATCHA as a way
for them to use its capabilities to support their goals for medication
management of their partner. They also reported instances when
they were not around or were unavailable. In such cases, MATCHA
supported an efective integration with external channels such as
phone notifcations and calls at the medication time by sending
notifcation alerts to the member as well as the carepartner’s phones.
While carepartners expressed that they would still like to sometimes
physically check-in with the member as they do regularly, they
also followed it up by feeling a sense of peace that if they were to
someday forget checking-in, there was a system in place to do it
for them.
• CP6: “I’m usually outside when it’s his medication time in the
evening but I get the alert on my phone which is good because
then I can know that he has taken the medications..”
• CP5: “It is great that I get mom’s medication notifcations on
my phone and can keep a check on her. I also double check
with her sometimes just in case..”
8.2.3 Members and carepartners liked that the system was aware of
their preferences and individual medication schedules.
Based on the diversity in medication schedules and preferences
revealed during pre-deployment interviews with the dyads, we
attempted to personalize some parameters of the assistant such as
the volume, location, personalized greeting for time of the day, etc.
This personalization was appreciated because it took into account
individual cognitive behaviors to some extent and incorporating
these individual variations into the system helped in making the
experience more pleasant for the dyads. For example, while some
dyads preferred a lively and playful interaction, others wanted a
more straightforward and simple check-in and the positive feedback.
In addition, some dyads also expressed their desire to have access to
their reported medication data and have it shared with their clinical
teams at difering intervals.
• M2: “I like that it’s kind of playful and talks to me but also
not too much or too loud that it starts to annoy me..”
• M1: “I think it’s helpful because it reminds me every night be-
fore I go upstairs that I need to get the medicine out. Sometimes
I keep the medicine out on the counter for the next day too. In
the morning, I’m usually sitting right next to it in the kitchen
so I can respond directly to it while I’m having cofee.”
• M2: “Our doctor always asks us if we have been taking our
medicines regularly and we always feel like we have to say yes
whether we did or not because it’s hard to recall perfectly, so it
would be useful if Google could tell him that directly..”
8.2.4 Adding touch-butons to provide an alternate way of interac-
tion in addition to voice interaction was appreciated.
After adding touch-buttons on the Google Home Hub screen as
an alternate way to account for Phase 1 feedback, we saw that the
number of touch-based interactions hovered at a range of 20-35%
of total interactions for each dyad through the study. While some
dyads used the buttons more than others, a combination of the two
forms of responses was appreciated and used consistently by the
dyads. It also helped to alleviate the requirement to speak clearly
during every interaction.
ASSETS ’22, October 23–26, 2022, Athens, Greece
• CP1: “We did notice that you added the buttons there, they
weren’t on there before. I quite like them, specially with the
smileys. We can now touch the buttons to answer and I do that
a lot because I sit very close to it mostly..”
• M7: “Sometimes Google does not catch what I’m saying, I don’t
know if it’s my fault or hers, but when that happens, I go and
touch the button and she can hear me fne again. So I defnitely
like having the buttons..”
9 DISCUSSION
In this section, we refect upon our results and discuss their impor-
tance from the context of the gaps and opportunities identifed at
the end of section 2 and from user research.
9.1 Personalization Matters
A key insight from related work and the user research conducted
with older adults with MCI is that the lived experience of MCI is
highly diverse. It is shaped and characterized by various factors
such as age, personality, caregiving network and the existence of a
supporting environment, both physically and emotionally. MCI is
a diferent and a unique experience for everyone and incorporat-
ing individual preferences for interaction to the extent possible is
critical for improving user experience and system acceptance. The
result of these individual circumstances is a customized functional-
ity incorporated into the assistant that works in accordance with
individual preferences identifed through following a user-centered
approach to research. As highlighted in [20], user research should
be conducted in a way that facilitates this level of personalization
in volume, medication routines for diferent types of medications,
preferred mode of greeting, etc. Before deploying MATCHA, we
conducted a series of pre-deployment interviews with the dyads
to understand their medication schedules and behaviors and the
location of their Google Home Hubs and pillboxes within the house.
These schedules revealed the diversity of medication habits based
on personal preferences. Integrating these preferences in the system
helped in providing a coherent support to their existing medica-
tion habits and strategies eliminating the need to abandon them
entirely. Adequate personalization also helped in navigating the
initial hesitation to adoption of new technologies by older adults
[32], and provided a compensatory support system that resulted in
faster acceptance of it during the course of the study. With respect
to the need for personalization, as the system deployment expands,
we plan on studying the interaction patterns based on frequency
and also the scenarios of usage based on the type of interactions,
with the aim of informing frameworks that can be used for groups
of users. This can be a way to ensure practical scalability while also
retaining personalization to the extent possible.
9.2 Sustaining long-term engagement
It is crucial that any technical intervention aiming to provide
routine-based support during MCI to older adults needs to be ef-
fective in the long-term and also account for the often aggravating
decline. In most cases, the primary caregiver supporting their part-
ner with MCI is also of an advanced age, and have medical issues
of their own. This can induce feelings of anxiety related to their
ASSETS ’22, October 23–26, 2022, Athens, Greece
ability to continue providing support to their partner. As identi-
fed in [32, 38], sustaining long-term engagement beyond study
duration is a critical issue that leads to the failure of most interven-
tions deployed for older adults. These failures could be a result of
poor user experience that does not provide error prevention, alarm
fatigue as a result of frequent reminders or inadequate amount
of training and functional support to the older adults and their
partners while introducing the system. Increased technical com-
plexity and difculty in understanding the instructions also leads to
drop in engagement after a while for most commercial medication
management devices [37]. Our study worked towards addressing
these issues by incorporating the medication system within the
Google Home Hubs, CAs that the participants already felt a degree
of comfort towards and also reduced the interaction complexity
by having an interactive voice as the primary afordance. We also
highlight the need to provide comprehensive written and printed
training materials and personal support to older adults at every
step in the process of deploying such systems, and creating a safe
environment for them to ask questions and have their concerns
addressed. Error prevention was addressed through transmission of
unknown responses to the research team, and a feedback response
from the system requesting the user to repeat their last sentence.
Given the high-risk nature of medication mishaps, we also made
sure that the system had alternate ways to notify to carepartner of
the medication time (phone notifcations) and regularly checked
the backend to make sure the check-ins were generated at the right
time for each dyad. Finally, the most pivotal aspect of MATCHA that
contributed towards the long-term engagement was the clear and
unambiguous check-in functionality as opposed to the traditional
reminders and alarms. As discussed in section 4, over-medication is
often a result of non-contextual reminders and is a critical concern
for older adults with MCI, and the check-in functionality of the
assistant helped to alleviate this concern by prompting refection at
the medication time, notifying the carepartner and incorporating
existing strategies such as prompting them to check their pillbox
as part of the interaction.
9.3 User autonomy and freedom
Given the occurrence of MCI at the juncture of normal aging and
dementia, its diagnosis can be accompanied by feeling a lack of
control [26] as a result of altered memory capacity and functional
abilities while also retaining some cognitive power. As a result of
this, the cognitive program as well as the individuals with MCI
and their carepartners feel a signifcant desire to maintain a cer-
tain level of autonomy and functional independence, while also
being open to compensatory support. The interventions need to
support fexibility of medication routines as a result of changing life
circumstances. Dyads, specifcally the carepartners, expressed the
willingness to get more training to be able to provide the needs of
their partner to the assistant in a way which makes the interaction
more efective. There are days when they know in advance that
their schedule is going to be diferent from usual, and having the
control to schedule their own check-ins a day before will make
the experience more independent. We demonstrated this to them
through the Google Home App. Any intervention for medication
management needs to strike a balance between the identical basic
Mathur, et al.
structure of the conversational fow for all participants, while also
providing the participants the option to personalize check-in tim-
ings, and notifcation preferences for their individual uses. Another
way to facilitate more autonomy in the system is by generating
transparent explanations regarding the check-in purpose by the
assistant and the nature of specifc medication information given to
it [40]. Additionally, it is also important to provide users the func-
tionality to record, share and review medication data. While this
ability is not currently supported by Google’s actions console, there
are workarounds to incorporate this feature into the medication
assistant that our team is currently exploring, discussed more in
section 9.5. Recording and sharing of medication routine history
over a period of time can also be benefcial to the clinical staf to
monitor adherence and efciency of a new medication routine in
order to make informed clinical decisions.
9.4 Multiple interactions modalities for
improved compensatory support
While this forms a part of designing for personalized interactions
in 9.1, it is important to address this separately as an important
construct in providing individualized support to older adults with
MCI. Multimodal interactions were positively accepted by partici-
pants in [12], as a result, we explored the addition of touch-based
feedback in the system in addition to voice response. The Google
Home Hub has a screen to provide visual feedback to the user
and the members have a strong mental model that stems from the
touchscreen capabilities of their smartphones and tablets that they
extended to the CA. The touch-buttons also provided an alternate
interaction option to the participants in the cases when their speech
was not recognized by the assistant or they had issues with verbal
communication. However, we highlight the need to make these
multiple interaction modalities clear to the users by having them
designed to be adequately visible to avoid multiple responses and
the resulting confusion.
9.5 Adaptive functionality and external
integrations
While MATCHA relied primarily on the check-in functionality,
there were also instances in Phase 1 when the participants used it
as a reminder to inform the assistant of taking medication as a result
of the check-in, efectively making the check-in work as an efective
reminder to take medication. Recurring instances of this adaptive
functionality, although initially prompting an unknown response
from the assistant, was then incorporated into Phase 2 as a separate
interaction stream and the assistant provided positive feedback by
playing cheering sounds leading to increased interaction success
between the member and the assistant.
Finally, as a closing contribution of the study and to set up future
directions for this work, we recommend the need to incorporate
external integration with the CA to efectively extend its use for
health and well-being of older adults with MCI. A proposed external
integration for the purpose of recording medication behavior as well
as to verify ground truth of medication ingestion are smart pillboxes.
These pillboxes would integrate with the medication assistant in a
way that the presence of the pill in the pillbox compartment would
be known to the assistant through sensors. This will also lead to
Medication Management in Older Adults with MCI Using Conversational Assistants
lesser medication check-ins throughout the day for individuals
with more than average number of medications in a day. The smart
pillbox and the CA would work together in the same ecosystem
to notify the users in the case of a delayed or missed medication
and report it whenever asked. There are also other ways in which
the pillbox status can be verifed with the medication assistant
such as smart buttons pinned to the pillbox, an integration that
we are hoping to explore in future studies. Added integrations can
be with smart home devices, smart phones, tablets, sensors, etc.
which provide a more robust functionality to the CA by creating a
networked interaction ecosystem.
10 LIMITATIONS
While we are encouraged to see our study results, specifcally
the sustained engagement of the medication system, we would
like to address some limitations of our work. The development of
MATCHA was a contributing part of a larger ongoing study that
is aimed at understanding the usage of CAs to support functional
independence and provide compensatory support to the individ-
uals with MCI and their carepartners. As a result of this, most of
our participants had been interacting with a CA for over a month
before our study. It would require more research to analyze the
engagement and usage when the system is newly introduced to
the participants. Additionally, we would also like to point out that
recruitment for the participants for the purpose of this study was
limited to the participating dyads in the cognitive program who
are able to aford long-term therapeutic healthcare and primarily
belong to upper-middle and high-income households. While we are
currently in the process of working towards extending the study
with CAs to other healthcare institutions serving a more broader
population, the study results at the time of this paper are only
representative of the dyads within this cognitive program. In the
future, we also plan on reaching out to assisted living facilities and
establishing connections with the community partners who work
in those facilities. The commercial nature of the system, i.e., being
a Google product, ensures that the technological infrastructure is
well supported and can be accessed through a central Google email.
Finally, we would also like to point out that all of our participating
dyads included a 2-person team - the member and the carepartner
(spouse or adult child). Our goal for future research is to understand
multi-carepartner teams consisting of more than one adult child or
extended family members in diferent locations.
11 CONCLUSION
As older adults with MCI continue to age and advance through
varying levels of cognitive progression, and with increasing num-
bers of new diagnoses of MCI every year, it is crucial to design
new technologies and adapt existing ones to support aging adults
in important tasks such as medication management. In this study,
we explored the idea of supporting these tasks using CAs and de-
signed a medication management system while keeping the needs
and cognitive behaviors of our participant population central to
our design. We observed that a design centered on specifc medi-
cation management scenarios, personalized to individual routines,
and broadly focused on "checking-in" regarding medication actions
in contrast to narrow alerts and reminders, generated sustained
ASSETS ’22, October 23–26, 2022, Athens, Greece
and positive engagement. Working from this foundation, partici-
pants saw additional avenues for future work including integration
with smart devices and augmented pillboxes, and the creation of
medication records to share with clinical care teams.
REFERENCES
[1] Renato FL Azevedo, Dan Morrow, James Graumlich, Ann Willemsen-Dunlap,
Mark Hasegawa-Johnson, Thomas S Huang, Kuangxiao Gu, Suma Bhat, Tarek
Sakakini, Victor Sadauskas, et al. 2018. Using conversational agents to explain
medication instructions to older adults. In AMIA annual symposium proceedings,
Vol. 2018. American Medical Informatics Association, 185.
[2] Renée L Beard and Tara M Neary. 2013. Making sense of nonsense: experiences
of mild cognitive impairment. Sociology of health & illness 35, 1 (2013), 130–146.
[3] William S Bond and Daniel A Hussar. 1991. Detection methods and strategies
for improving medication compliance. American journal of hospital pharmacy 48,
9 (1991), 1978–1988.
[4] Marie T Brown and Jennifer K Bussell. 2011. Medication adherence: WHO cares?.
In Mayo clinic proceedings, Vol. 86. Elsevier, 304–314.
[5] Noll L Campbell, Malaz A Boustani, Elaine N Skopelja, Sujuan Gao, Fred W
Unverzagt, and Michael D Murray. 2012. Medication adherence in older adults
with cognitive impairment: a systematic evidence-based review. The American
journal of geriatric pharmacotherapy 10, 3 (2012), 165–177.
[6] Chen Chen, Janet G Johnson, Kemeberly Charles, Alice Lee, Ella T Lifset, Michael
Hogarth, Alison A Moore, Emilia Farcas, and Nadir Weibel. 2021. Understanding
Barriers and Design Opportunities to Improve Healthcare and QOL for Older
Adults through Voice Assistants. In The 23rd International ACM SIGACCESS
Conference on Computers and Accessibility. 1–16.
[7] Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin
Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu,
et al. 2019. What makes a good conversation? Challenges in designing truly
conversational agents. In Proceedings of the 2019 CHI Conference on Human Factors
in Computing Systems. 1–12.
[8] Andy Cochrane, Ian H Robertson, and Andrew N Coogan. 2012. Association
between circadian rhythms, sleep and cognitive impairment in healthy older
adults: an actigraphic study. Journal of neural transmission 119, 10 (2012), 1233–
1239.
[9] David M Cutler and Wendy Everett. 2010. Thinking outside the pill-
box—medication adherence as a priority for health care reform. New England
Journal of Medicine (2010).
[10] Melisa Duque, Sarah Pink, Yolande Strengers, Rex Martin, and Larissa Nicholls.
2021. Automation, Wellbeing and Digital Voice Assistants: Older People and
Google devices. Convergence 27, 5 (2021), 1189–1206.
[11] Mira El Kamali, Leonardo Angelini, Maurizio Caon, Giuseppe Andreoni,
Omar Abou Khaled, and Elena Mugellini. 2018. Towards the NESTORE e-Coach:
a tangible and embodied conversational agent for older adults. In Proceedings of
the 2018 ACM International Joint Conference and 2018 International Symposium
on Pervasive and Ubiquitous Computing and Wearable Computers. 1656–1663.
[12] Mira El Kamali, Leonardo Angelini, Denis Lalanne, Omar Abou Khaled, and
Elena Mugellini. 2020. Multimodal conversational agent for older adults’ behav-
ioral change. In Companion Publication of the 2020 International Conference on
Multimodal Interaction. 270–274.
[13] Terry T Fulmer, Penny Hollander Feldman, Tae Sook Kim, Barbara Carty, Mark
Beers, Maria Molina, and Margaret Putnam. 1999. An intervention study to
enhance medication compliance in community-dwelling elderly individuals.
[14] Lynn Gitlow. 2014. Technology use by older adults and barriers to using technol-
ogy. Physical & Occupational Therapy in Geriatrics 32, 3 (2014), 271–280.
[15] Bradi B Granger and Hayden Bosworth. 2011. Medication adherence: emerging
use of technology. Current opinion in cardiology 26, 4 (2011), 279.
[16] D Heister, James B Brewer, Sebastian Magda, Kaj Blennow, Linda K McEvoy,
Alzheimer’s Disease Neuroimaging Initiative, et al. 2011. Predicting MCI outcome
with clinically available MRI and CSF biomarkers. Neurology 77, 17 (2011), 1619–
1628.
[17] Sunyoung Kim et al. 2021. Exploring How Older Adults Use a Smart Speaker–
Based Voice Assistant in Their First Interactions: Qualitative Study. JMIR mHealth
and uHealth 9, 1 (2021), e20427.
[18] Bret Kinsella. 2019. Voice Assistant Demographic Data-Young Consumers More
Likely to Own Smart Speakers While Over 60 Bias Toward Alexa and Siri.
[19] Deborah Klee, Marc Mordey, Steve Phuare, and Cormac Russell. 2014. Asset
based community development–enriching the lives of older citizens. Working
with Older People (2014).
[20] Kathrin Koebel, Martin Lacayo, Madhumitha Murali, Ioannis Tarnanas, and Arzu
Çöltekin. 2021. Expert Insights for Designing Conversational User Interfaces as
Virtual Assistants and Companions for Older Adults with Cognitive Impairments.
In International Workshop on Chatbot Research and Design. Springer, 23–38.
ASSETS ’22, October 23–26, 2022, Athens, Greece Mathur, et al.

[21] Jarosław Kowalski, Anna Jaskulska, Kinga Skorupska, Katarzyna Abramczuk, [44] John Vines, Gary Pritchard, Peter Wright, Patrick Olivier, and Katie Brittain. 2015.
Cezary Biele, Wiesław Kopeć, and Krzysztof Marasek. 2019. Older adults and An age-old problem: Examining the discourses of ageing in HCI and strategies
voice interaction: A pilot study with google home. In Extended Abstracts of the for future research. ACM Transactions on Computer-Human Interaction (TOCHI)
2019 CHI Conference on human factors in computing systems. 1–6. 22, 1 (2015), 1–27.
[22] Glenn J Landry and Teresa Liu-Ambrose. 2014. Buying time: a rationale for exam- [45] Pierre Wargnier, Giovanni Carletti, Yann Laurent-Corniquet, Samuel Benveniste,
ining the use of circadian rhythm and sleep interventions to delay progression of Pierre Jouvelot, and Anne-Sophie Rigaud. 2016. Field evaluation with cognitively-
mild cognitive impairment to Alzheimer’s disease. Frontiers in aging neuroscience impaired older adults of attention management in the embodied conversational
6 (2014), 325. agent louise. In 2016 IEEE International Conference on Serious Games and Applica-
[23] Matthew L Lee and Anind K Dey. 2014. Real-time feedback for improving tions for Health (SeGAH). IEEE, 1–8.
medication taking. In Proceedings of the SIGCHI Conference on Human Factors in [46] Tamara Zubatiy, Kayci L Vickers, Niharika Mathur, and Elizabeth D Mynatt.
Computing Systems. 2259–2268. 2021. Empowering Dyads of Older Adults With Mild Cognitive Impairment And
[24] Haley MacLeod, Maia Jacobs, Katie Siek, Kay Connelly, and Elizabeth D Mynatt. Their Care Partners Using Conversational Agents. In Proceedings of the 2021 CHI
2016. Ethical considerations in pervasive health research. In Proceedings of Conference on Human Factors in Computing Systems. 1–15.
the 10th EAI International Conference on Pervasive Computing Technologies for
Healthcare. 326–329.
[25] Aqueasha M Martin, Jessica N Jones, and Juan E Gilbert. 2013. A spoonful
of sugar: understanding the over-the-counter medication needs and practices
of older adults. In 2013 7th International Conference on Pervasive Computing
Technologies for Healthcare and Workshops. IEEE, 93–96.
[26] Kerry Mills and Jennifer Brush. 2014. Coaching care partners who support loved
ones with MCI. Perspectives on Gerontology 19, 2 (2014), 50–56.
[27] Daniel G Morrow, H Chad Lane, and Wendy A Rogers. 2021. A framework
for design of conversational agents to support health self-care for older adults.
Human Factors 63, 3 (2021), 369–378.
[28] Gift Arnold Mugisha, Faith-Michael Uzoka, and Chinyere Nwafor-Okoli. 2017. A
framework for low cost automatic pill dispensing unit for medication manage-
ment. In 2017 IST-Africa Week Conference (IST-Africa). IEEE, 1–10.
[29] Jennifer M Ortman, Victoria A Velkof, Howard Hogan, et al. 2014. An aging
nation: the older population in the United States. (2014).
[30] Tejal Patel, Jessica Ivo, Sadaf Faisal, Aidan McDougall, Jillian Carducci, Sarah
Pritchard, Feng Chang, et al. 2020. A prospective study of usability and workload
of electronic medication adherence products by older adults, caregivers, and
health care providers. Journal of medical Internet research 22, 6 (2020), e18073.
[31] Jessica Pater, Shane Owens, Sarah Farmer, Elizabeth Mynatt, and Brad Fain.
2017. Addressing medication adherence technology needs in an aging population.
In Proceedings of the 11th EAI International Conference on Pervasive Computing
Technologies for Healthcare. 58–67.
[32] Belén Barros Pena, Rachel Elizabeth Clarke, Lars Erik Holmquist, and John Vines.
2021. Circumspect Users: Older Adults as Critical Adopters and Resistors of
Technology.. In CHI. 84–1.
[33] Ronald C Petersen. 2004. Mild cognitive impairment as a diagnostic entity. Journal
of internal medicine 256, 3 (2004), 183–194.
[34] Ronald C Petersen, Oscar Lopez, Melissa J Armstrong, Thomas SD Getchius,
Mary Ganguli, David Gloss, Gary S Gronseth, Daniel Marson, Tamara Pring-
sheim, Gregory S Day, et al. 2018. Practice guideline update summary: Mild
cognitive impairment: Report of the Guideline Development, Dissemination,
and Implementation Subcommittee of the American Academy of Neurology.
Neurology 90, 3 (2018), 126–135.
[35] Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019. " Phantom Friend" or"
Just a Box with Information" Personifcation and Ontological Categorization of
Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM
on Human-Computer Interaction 3, CSCW (2019), 1–21.
[36] Alisha Pradhan, Amanda Lazar, and Leah Findlater. 2020. Use of intelligent
voice assistants by older adults with low technology use. ACM Transactions on
Computer-Human Interaction (TOCHI) 27, 4 (2020), 1–27.
[37] Blaine Reeder, George Demiris, and Karen D Marek. 2013. Older adults’ satisfac-
tion with a medication dispensing device in home care. Informatics for Health
and Social Care 38, 3 (2013), 211–222.
[38] Yara Rizk, Vatche Isahagian, Merve Unuvar, and Yasaman Khazaeni. 2020. A
snooze-less user-aware notifcation system for proactive conversational agents.
arXiv preprint arXiv:2003.02097 (2020).
[39] Assumpta Ann Ryan. 1999. Medication compliance and older people: a review of
the literature. International journal of nursing studies 36, 2 (1999), 153–162.
[40] Jamie Sanders and Aqueasha Martin-Hammond. 2019. Exploring autonomy in
the design of an intelligent health assistant for older adults. In Proceedings of the
24th International Conference on Intelligent User Interfaces: Companion. 95–96.
[41] Takuo Suzuki and Yasushi Nakauchi. 2014. A smartphone mediated portable
intelligent medicine case for medication management support. In 2014 36th
Annual International Conference of the IEEE Engineering in Medicine and Biology
Society. IEEE, 3642–3645.
[42] Jelte Barachia van Waterschoot, IHE Hendrickx, Arif Khan, Catia Cucchiarini,
Helmer Strik, LFM ten Bosch, and Rob Tieben. 2021. Spoken Conversational
Agents for Older Adults. Who Are the Stakeholders and What Do They Expect?
(2021).
[43] Bert Vandenberghe and Karin Slegers. 2016. Anthropomorphism as a strategy
to engage End-Users in health data ideation. In Proceedings of the 9th Nordic
Conference on Human-Computer Interaction. 1–4.
 Designing Post-Trauma Self-Regulation Apps for People with
Intellectual and Developmental Disabilities
Krishna Venkatasubramanian∗
The University of Rhode Island
Kingston, RI, USA
krish@uri.edu
ABSTRACT
In the US people with intellectual and developmental disabilities
(I/DD) comprise one of the most likely groups to experience trau-
matic life events. These experiences often produce negative efects
(e.g., stress, anxiety, grief, numbing, etc.) that need to be managed.
Methods such as emotional self-regulation are often used to help
people cope when these efects present themselves post-trauma.
In recent years mobile-computing-devices-based apps have been
increasingly used to help the general population with autonomous
self-regulation. However, none of these is designed for people with
I/DD or is cognizant of the trauma they experience in their lives.
We interviewed eight (8) practitioners at a trauma services organi-
zation that, among other things, helps people with I/DD learn and
practice post-trauma self-regulation. The goal of the interviews is to
understand what it would take to build post-trauma self-regulation
apps for people with I/DD. Based on the interview responses we
argue for a set of guidelines, based on the social work practice of
trauma-informed care, to design post-trauma self-regulation apps
for people with I/DD.
CCS CONCEPTS
• Human-centered computing → Accessibility technologies.
KEYWORDS
trauma-informed care, intellectual disability, developmental disabil-
ity, trauma, design
ACM Reference Format:
Krishna Venkatasubramanian and Tina-Marie Ranalli. 2022. Designing Post-
Trauma Self-Regulation Apps for People with Intellectual and Developmen-
tal Disabilities. In The 24th International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece.
ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3517428.3544798
1 INTRODUCTION
Trauma is a sudden, potentially deadly experience that often leaves
lasting, troubling memories [25]. In the US, a signifcant number of
∗ Both
authors contributed equally to this paper.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9258-7/22/10. . . $15.00
https://doi.org/10.1145/3517428.3544798
Tina-Marie Ranalli∗
Unafliated
Providence, RI, USA
ranalli@gmx.edu
people with intellectual and developmental disabilities (I/DD)1 are
suspected to have experienced trauma in their lifetime [51]. The
negative efects of trauma (e.g., anxiety, depression, complicated
grief) experienced by people with I/DD are often misdiagnosed be-
cause of what is known as diagnostic overshadowing, where health
practitioners assume that the negative efects of trauma are merely
an aspect of the person’s I/DD and/or other disability [50]. There-
fore, the majority of people with I/DD who experience trauma may
not get appropriate help to cope with the negative efects of the
trauma and to improve their quality of life [86]. Moreover this
disregard for, and negation of, the post-trauma reality of people
with I/DD may also constitute another traumatic event, thereby re-
traumatizing them [12]. Untreated trauma can result in severe and
long-term impairments that negatively afect one’s general well-
being and emotional, social, academic, and physical development
[51]. Therapeutic treatment often involves processing the traumatic
experience. The methods to do so include, cognitive behavior ther-
apy (CBT) and eye movement desensitization and reprocessing
(EMDR), both of which have been shown to be efective for people
with I/DD [50], especially EMDR [51]. However, there are often
substantive systemic and personal barriers that prevent or impede
people with I/DD from accessing the relevant mental health ser-
vices [84]. Engaging in self-regulation - coping with the negative
efects of trauma (more on this in Section 1.1) - outside of therapy
has been shown to play a major role in facilitating the process of
recovering from trauma in people with I/DD [51]. It is, thus, im-
portant to empower people with I/DD by giving them the means to
autonomously cope with the negative efects of trauma in their daily
lives, that is, self-regulate, whether or not they also pursue therapeutic
treatment.
The last two decades have seen the near pervasive availability of
mobile computing devices (e.g., smartphones, tablets, and wearables).
Such mobile computing devices are increasingly being used to help
people who have experienced trauma autonomously cope with the
day-to-day negative efects of trauma. In the past decade, the US
federal government has developed several trauma-focused apps that
help military veterans and US-Department-of-Defense-afliated
military and civilian personnel seek help, understand, and manage
the negative trauma efects they experience [58, 61, 68, 81]. How-
ever, to the best of our knowledge, no post-trauma self-regulation
(PTSR) app exists that has been specifcally designed for the needs
of people with I/DD. Recent years have seen the regular use of mo-
bile computing devices (like smartphones, tablets, etc.) by people
1 Based on the defnition from the American Association of Intellectual and Develop-
mental Disabilities, I/DD can be thought of as a set of disabilities that negatively afect
the trajectory of an individual’s intellectual, emotional, and/or physical development.
I/DD appear in childhood and are likely to be present throughout life [1].
ASSETS ’22, October 23–26, 2022, Athens, Greece
with I/DD in the US [56, 80]. Therefore a mobile-computing-device-
based app would provide an excellent mechanism for empowering
people with I/DD to self-regulate post-trauma. In this collaborative
paper, which was written by an HCI and a humanities scholar, we
aim to explore the design of post-trauma self-regulation apps for
people with I/DD that can complement therapeutic interventions.
Consequently, in this paper we frst defne the notion of self-
regulation in the context of its use by people with I/DD. We then
present an interview study where we interviewed eight (8) practi-
tioners at an organization in the United States (US) that provides
trauma services specifcally for people with I/DD. As part of their
services, among other things, they teach people with I/DD how to
use self-regulation to cope with trauma. Such eforts are intended
to be complementary to any therapeutic interventions that the sur-
vivor may pursue. Our aim with the user study is to understand
the following: (1) What does post-trauma self-regulation for people
with I/DD entail? (2) Do people with I/DD use apps/technologies
for post-trauma self-regulation and if so, how? and (3) What should
designers keep in mind when designing post-trauma self-regulation
apps for someone with I/DD?
We then present a new framework for designing for post-trauma
self-regulation that leverages the notion of trauma-informed care
[46]. Trauma-informed care was originally developed in the feld of
social work for support organizations that help trauma survivors.
The practice of trauma-informed care avoids pathologizing trauma
and focuses on validating the post-trauma experiences of the in-
dividual. This prevents retraumatization of the individual while
empowering them to develop a sense of safety and reducing any
shame or guilt surrounding their past trauma [39]. In this paper,
we refect on how the notion of trauma-informed care can be applied
to the design of PTSR apps for people with I/DD and propose several
design guidelines to this end.
This paper makes the following contributions: demonstrating
the lack of existing self-regulation solutions that meet the PSTR
needs of people with I/DD (Section 2); an interview study with
stakeholders to understand what to keep in mind when designing
PSTR apps (Section 3); and proposed guidelines for designing PTSR
apps for individuals with I/DD (Sections 5).
1.1 Trauma and post-trauma self-regulation
for people with I/DD
Before we delve into the details of designing post-trauma self-
regulation apps, we present a short overview of how trauma afects
people with I/DD. Trauma afects people with I/DD in profound
ways. After trauma, people with I/DD can experience: (1) impaired
working memory for tasks at hand; (2) a variety of negative ef-
fects from the trauma (e.g., depression, anxiety, etc.) that vary from
individual to individual; (3) excessive distraction, even more so
than non-I/DD individuals experience; (4) limited ability to avoid
triggers2 because of limited control over their environment than
those without I/DD (e.g., a group living situation); and (5) limited
ability to get appropriate help to manage the negative efects of
their trauma [86].
2 Triggers are things like sights, sounds, smells, situations, or depictions that are similar
to and in some way evoke a person’s traumatic experience and which cause the person
to uncontrollably recall and relive the trauma [42].
Venkatasubramanian and Ranalli
The negative efects of trauma on an individual with I/DD need to
be managed on an on-going basis. In this work, we do not look at the
negative efects of trauma from a medical, diagnostic standpoint
that seeks to determine the presence/absence of post-traumatic
stress disorder (PTSD). Rather we approach the post-trauma state
from the standpoint of what some humanities scholars term dis-
ease [14, 72, 77]. This state of dis-ease, or a lack of ease, can present
itself in many ways, including: a general sense of grief, depression,
anxiety, sleep disturbance, exaggerated tendency to startle, self-
harming behavior, and being stressed by triggers [51]. We do not
view the state of dis-ease as something caused by triggers alone,
though triggers can provoke or exacerbate it.
Note that we do not claim that self-regulation should be the only
source of dealing with post-trauma dis-ease for people with I/DD.
Therapeutic approaches, such as cognitive behavior therapy (CBT)
and eye-movement desensitization and reprocessing (EMDR), are
powerful, have been shown to work for people with I/DD [51],
and are indeed useful. That being said, when it comes to the I/DD
population, there is also a need to think more holistically beyond
therapeutic approaches alone. This is because of: (1) a lack of thera-
pists who will work with people with I/DD; (2) frequent diagnostic
overshadowing by therapists and providers; (3) a lack of knowledge
about mental health issues and the stigmas associated with people
with I/DD; and (4) logistical problems related to the lifestyles of
people with I/DD (in the US), such as a lack of suitable transporta-
tion and a lack of privacy in group living situations, which often
preclude them from pursuing therapy [83].
Therefore, we defne post-trauma self-regulation (PTSR) as
engaging in one or more activities that regulate one’s emotions to
help cope with the state of dis-ease in the moment, outside of (but
complementary to) any long-term therapy [55]. Broadly speaking,
self-regulation usually requires performing activities that focus
one’s attention to temper the negative emotions and feelings in a
given situation [71]. This paper does not concern itself with the
design of specifc PTSR activities within an app environment to be
used by people with I/DD. Rather, this paper is about the larger
design considerations of an app that provides PTSR for people with
I/DD. The decision of whether or not to engage in any particular
type of PTSR activity is up to the individual with I/DD. Given that
people with I/DD are avid users of mobile computing devices [80],
apps are one of the easiest ways of providing people with I/DD the
ability to perform PTSR.
2 RELATED WORK
In this section, we provide an overview of the work done in re-
search and commercial spaces around mobile-computing-based
self-regulation apps. None of the prior work has focused on the use of
technology for PTSR for people with I/DD. We categorize the extant
work on self-regulation into four broad groups, which we describe
below.
2.1 Mobile technologies for mental health
Recent years have seen an increasing number of technologically-
oriented mental health solutions that have become available. These
include: connecting people with help online [16, 45], conversational
Designing Post-Trauma Self-Regulation Apps for People with I/DD
agents [26, 57, 73], chatbots [30, 62, 78], and virtual reality (VR) sys-
tems [27]. Often these health technologies focus on helping people
with issues, such as depression and anxiety, by providing treatment
and therapies (such as cognitive behavioral therapy (CBT) or expo-
sure therapy) [88]. Such technological approaches to therapy have
become necessary due to a lack of easy access to mental health
services, stigma, and low mental health literacy in the general popu-
lation [53] (much like the I/DD community). Consequently, consid-
erable design work is now underway to systematically understand
the users of these technologies and build better and more usable
mental health technologies [8–10, 22, 41, 45, 76, 88]. These technolo-
gies, though important, are diferent from our work in important
ways. (1) The focus of these technologies is often on therapeu-
tic approaches, which are diferent from, though complementary
to self-regulation. (2) These technologies have not been designed
for people with I/DD whose lived experiences, motivations, and
struggles can be quite diferent from non-I/DD individuals.
2.2 Research apps for non-trauma
self-regulation
Mobile-computing-device-based apps for self-regulation have been
investigated in the academic research space as well. These include
using a device to record one’s emotions, location-tagging them,
and sharing them (anonymously, if needed) with others [35]. Other
approaches include using slow, repetitive actions to maintain the
user’s attention [4], providing gentle feedback via soft fascinations3
[2], and promoting an efortless refection on any loss of attention
[69]. These ideas of repetitive actions and soft fascinations have
been extended from monitoring one’s thoughts to performing mind-
ful physical activities (e.g., yoga) as well [60]. These apps, not having
been designed for someone with I/DD, provide a limited repertoire
of often abstract self-regulation activities. Someone with limited
working memory (as many people with I/DD have post-trauma)
would likely fnd them difcult to use. Further, if any of the activ-
ities/actions/sounds in these apps constitutes a trigger for users
who have experienced trauma, the apps do not ofer any alternative
activities or any form of recourse to deal with the ramifcations.
2.3 Commercial apps for non-trauma
self-regulation
Numerous mobile-computing-device-based apps are now available
that provide a means of self-regulation, primarily through guided
meditation practices (e.g., Calm and HeadSpace). We looked at 28
popular self-regulation apps listed on websites like Healthline [33],
Positive Psychology [64] and Women’s Health Weekly [85]. We
provide the list of the 28 apps along with the URL where we found
them in the appendix. Usually designed to manage stress, these
apps require their users to observe their current state of attention
and adjust it, often based on spoken instructions. Most of these apps
appear not to be cognizant of the lives of people with I/DD or those
who have experienced trauma. These apps tend to blithely ofer
activities unproblematically, unaware that certain characteristics
of the activities and/or the way in which they are presented can
3 Soft fascinations are feedback, such as nature sounds, which are meant to be efortless
to observe. These are often used in Attention Restoration Theory as a way to recover
from mental fatigue [38].
ASSETS ’22, October 23–26, 2022, Athens, Greece
include aspects that trigger trauma survivors. For instance, just the
act of asking a trauma survivor to close their eyes for meditation
can be triggering and make them feel unsafe and potentially retrau-
matized [66]. These apps impose a very high cognitive load on the
user and require signifcant working memory, as they often contain
an enormous number of self-regulation exercises from which to
choose. For instance, the app Insight Timer boasts of having tens of
thousands of videos alone, not to mention all of its live content [36].
Moreover, these apps also provide limited instructions and often
poor recommendation support. Many of these apps have substantial
subscription fees, which can be difcult for many to aford (e.g.,
Calm [15] and Headspace [32]) [40]. As mentioned above, most of
these apps are heavily reliant on some form of meditation practice
involving listening to someone guide the activity through spoken
cues, which are usually vague and tailored to users who already
have experience with meditation and do not require explicit instruc-
tions. Not surprisingly, these meditations have been found to be
difcult to use for many people who fnd it hard to keep up with the
pace of the instructions [69]. On the whole, these apps constitute
part of what is referred to as “self-tracking culture” in [47], which is
only fully available to those with sufcient socioeconomic privilege.
Thus, those who lack adequate privilege (e.g., older adults, those
with less education or lower incomes, people with disabilities and
chronic health problems, people living in rural and remote areas,
etc.) cannot fully beneft from these technologies and are left be-
hind. In short, whether consciously or not, these apps are designed
for highly motivated, non-traumatized, privileged people who have
the means to pay for them; they are not intersectionally inclusive.
2.4 Non-commercial trauma-focused
self-regulation apps
In recent years, an entire class of free apps has been developed
by the US federal government primarily for veterans, members
of the military, civilian personnel, and their families to help them
cope with the negative efects of trauma. These include apps that:
teach mindfulness and meditation practices and routines over time
[49], provide a variety of ways to deal with the negative efects
of trauma [61, 81], help healthcare providers deal with secondary
traumatic stress [67], help families of trauma survivors [24], and
help survivors of sexual assault [58, 68]. These apps have many
useful features that could be leveraged for people with I/DD (as
we shall see from time-to-time in this paper). However, as none
of these apps has been designed for people with I/DD, they have
several drawbacks in terms of being accessible for our target popu-
lation. Most of these apps are quite reading-heavy in nature (often
necessarily so, as they want their audience to understand trauma,
take it seriously, and get help), which can be difcult for people
with I/DD, many of whom have limited literacy [34]. Moreover,
these apps typically lack efective recommendation services for
activities to do, given the often large set of available activities. We
believe that these aspects, among others, make these apps not only
difcult for people with I/DD to navigate but also difcult for them
to operate, especially when used autonomously.
ASSETS ’22, October 23–26, 2022, Athens, Greece
ID Freq. of interaction w/ people with I/DD
P1 2 days/week
P2 Daily
P3 Daily
P4 Weekly
P5 Weekly
P6 Daily
P7 3 days/week
P8 Daily
Table 1: Demographic information for the practitioners surveyed in our study. All participants identifed as female.
2.5 Trauma-informed computing
In a very recent paper [17], the idea of trauma-informed comput-
ing was introduced. The authors, coming from the perspective of
intimate partner violence (IPV) and the role of technology is per-
petuating it, describe the need for trauma-informed computing via
three fctional scenarios they constructed, which refect their expe-
rience working with IPV survivors. The authors then turn to the
notion of trauma-informed care, which we explore in more detail
below. In basic terms, trauma-informed care asks organizations pro-
viding services to trauma survivors to treat these survivors in a way
that conveys respect and compassion [21, 31, 46]. This recent paper
adapts the notion of trauma-informed care into a broad framework
they refer to as trauma-informed computing. The scope of the paper
is broad and discusses applying their framework to topics, such as:
UX research and design, security and privacy, AI and machine learn-
ing, and corporate culture. Therefore the guidelines provided in
the paper are designed to be broadly applicable rather than directly
relate to the needs of a specifc class of technology or to address the
needs of a specifc community (such as the I/DD community). We
similarly apply the notion of trauma-informed care to the develop-
ment of technology (in the present paper): specifcally PTSR apps
for people with I/DD. We consider our contemporaneous work as
complementary to the one proposed in [17]. We adapt the idea of
trauma-informed care specifcally to meet the targeted needs of
designing PTSR apps for people with I/DD, based on interviews
with stakeholders. Consequently, our view of how trauma-informed
care maps to technology design difers somewhat from the view
presented in [17].
3 INTERVIEW STUDY
Our aim in this paper is to explore the design of post-trauma self-
regulation (PTSR) apps for people with I/DD that complements
therapeutic interventions. In this regard, we wanted to develop
a broad understanding of PTSR within the I/DD community. To
wit, we interviewed practitioners at an organization in the US
that provides trauma services to people with I/DD. As part of their
services they help and teach people with I/DD to use self-regulation
to cope with their past trauma. Our aim with the interviews was to
answer three core research questions. (RQ1) What does post-trauma
self-regulation for people with I/DD entail? (RQ2) Do people with
I/DD use apps/technologies for post-trauma self-regulation and if
so, how? and (RQ3) What should designers be mindful of when they
design post-trauma self-regulation apps for someone with I/DD?
Below we describe the study methods of our interview.
Venkatasubramanian and Ranalli
Duration working w/ people with I/DD
4 years
25 years
3 years
4 years
3 months
5 years
10 years
17 years
3.1 Study methods
We interviewed practitioners who are afliated with an adult pro-
tective services agency in the US and who help people with I/DD
with trauma services. Among the services they provide is helping
people with I/DD learn how to engage in PTSR. These practitioners
have a global perspective about coping with trauma specifcally
for the I/DD community. They can therefore provide us with the
larger context regarding self-regulation practices within the com-
munity. All interviews were conducted over Zoom because of the
COVID-19 pandemic. The interview protocol was approved by the
institutional review board (IRB), the ethics board, at the University
of Rhode Island.
3.1.1 Study design. We conducted semi-structured interviews with
participants recruited for our study. We had a script with open-
ended questions and interviewees were allowed to wander in
their responses. We opened with a brief introduction of our aims
followed by questions in four categories: the participant’s back-
ground, the services their organization provides related to enabling
self-regulation for individuals with I/DD, their understanding of
the types of PSTR practiced by individuals with I/DD, and their
thoughts on ways to more efectively facilitate PSTR among individ-
uals with I/DD. A total of eight (8) practitioners participated in the
interviews. All eight interact with people with I/DD on a regular
basis and have experience helping them learn and practice self-
regulation. The individuals with I/DD with whom our practitioners
work have mild to moderate I/DD. Table 1 shows the demographic
information for the practitioners.
3.1.2 Study analysis. After the user study, the collected Zoom
recordings were transcribed. We applied Braun and Clark’s 6-step
recursive approach to thematic analysis, as described in [13]. The
coding and analysis were completed in a collaborative manner
between the two authors, aiming to achieve a richer interpretation
of meaning than attempting to achieve consensus would produce.
The coding and theme development were done inductively and
evolved throughout the analytic process. Table 2 lists the codes that
were generated during the thematic analysis. The results of our
analysis are summarized in the fndings below.
3.1.3 Limitations. The methodology of our study had three main
limitations that we briefy discuss. First, all practitioners we in-
terviewed identify as female, though we did not set out to solicit
opinions exclusively from female practitioners. We do not believe
that this afected the observations in the paper; however, a more
Designing Post-Trauma Self-Regulation Apps for People with I/DD ASSETS ’22, October 23–26, 2022, Athens, Greece

Code Name Defnition

Current self-regulation activities
Potential self-regulation activities
Ersatz apps for self-regulation
Suitability of ersatz apps
Designing for PTSR
Self-regulation activities in which individuals with I/DD engage
Categories of self-regulation activities that should work for individuals with I/DD
Apps that individuals with I/DD turn to for self-regulation (since there is no existing PTSR app for them to use)
Appropriateness of the ersatz self-regulation apps for PTSR for individuals with I/DD
What the PTSR apps for individuals with I/DD should consider in their design

Table 2: The codes used in our thematic analysis of the interviews.

diverse practitioner population could have provided additional per-
spectives that we might have missed here. Second, all of our partici-
pants were from the United States and, therefore, their perspectives
and experience may difer from those of practitioners from other
regions. Third, we did not talk to individuals with I/DD about how
they self regulate on a day-to-day basis. This is because, at this
point in our work, the aim was to get a community-level picture of
what self-regulation for people with I/DD entails. This is something
that we plan to work on in the immediate future.
3.2 Study fndings 1: PTSR for people with
I/DD should promote empowerment and
include diverse activities.
In our interviews, we began by trying to understand what PTSR
for people with I/DD entails. We found two main themes in this
regard, which we describe next. We provide verbatim quotations,
which are edited for brevity and clarity using ellipses and brackets,
respectively.
3.2.1 PTSR should enable people with I/DD to gain control over their
lives as a way to mitigate some of the efects of their trauma. People
with I/DD often lead, as P2 puts it, a highly “managed life.” For
instance, they live in group homes where they share personal space
and resources with others: “I also think, acknowledging the disability
component... that’s something that’s unique to [people with I/DD]. Like
a [person with I/DD] will talk about, in one way or another, how... they
might not be able to access certain things because of their disability,
like privacy [or lack there of, in their daily lives].” (P3). Furthermore,
the opinions of people with I/DD are often discounted by others: “...
people don’t believe people with [I/DD] are reliable narrators of their
experiences.” (P3).
Moreover, if people with I/DD are in distress, they are often
blamed for displaying such behavior: “if someone [with I/DD] is
distressed, it’s like they’re doing something wrong or they’re having a
behavior that’s wrong.” (P2). These experiences often mean that they
lack power over their words, body, and surroundings, which con-
tributes to their feeling traumatized on a day-to-day basis, which
is line with extant literature [52, 86]. Given the pervasive presence
of traumatizing events in the lives of people with I/DD the partici-
pants felt that one of the main tenets of PTSR should be to allow
people with I/DD to regain control over their lives. They brought
up three broad ways of empowering individuals with I/DD in this
regard:
• Helping people with I/DD to observe and determine their
current feelings/emotions in the moment: “being able to em-
power them and teach them the skills that they need to be
able to [check in and see how they are feeling] in a practical
way in their life would be a really amazing goal.” (P7). This is
essential because knowing one’s feelings or emotions in the
moment is often healing in itself: “[people with I/DD] enjoy
learning [about and] identifying what their feelings are or
maybe what they’re going through. Even the difcult feelings
but they’ll come to terms with [them]... in my understanding
and observation, it’s very healing and it’s very empowering.”
(P8).
• Enabling people with I/DD to make decisions about their
lives, including the type of self-regulation activities that
work for them: “So it’s important to make sure that they’re
taking charge in... any of the decisions being made... allowing
people to pick their own [self-regulation activities].” (P6).
• Providing a means of supporting them as they make these
decisions: “... decrease any barriers or any complexity to them
getting whatever support that they might need.” (P8). The idea
is to enable people with I/DD to lead a healthy and positive
life by taking in account things like “what are their goals,
their dreams, their interests, their preferences, and how can
they get those things.” (P2).
3.2.2 Activities used for PTSR should be diverse to give them more
choices. Every person has diferent needs when it comes to self
regulating to deal with the negative efects of trauma: “I think it’s
good to have choices, like a variation of choices because that’s really a
big thing too, is that one coping skill might not work for [one] person
[while for another person] it might be the best coping skill ever.” (P7).
Consequently, when it comes to PTSR, a variety of activities should
be ofered. Our participants listed a large variety of options that
they have seen work for people with I/DD to cope with their trauma.
These can be broadly categorized into 6 categories (next to each
categorical entry, we list the number of participants who suggested
that category): audio-visual activities (e.g., music, video games, and
video clips) (N = 4), outdoor activities (e.g., gardening/yard work,
walks, and traveling) (N = 4); establishing social connections (N =
3); identifying one’s emotions/feelings (N = 3); breathing, yoga, and
mindfulness practices (N = 2); viewing positive afrmations and
messages (N = 2); and activities that engage a person’s creativity
(e.g., coloring books and crafts) (N = 1). These categories are, of
course, not comprehensive but they do give a sense of how diverse
PTSR activities can be. Interestingly one participant stated that, for
some people with I/DD, activities involving abstract imagery or
visualizations can be distressing instead of calming: “[People with
I/DD] feel like [if] it’s visually too imaginary [it’s] distressing rather
than calming. In that same vein, it makes me think that something
that can be useful is being a bit more concrete... being less abstract and
less imaginary or visual. Like the idea of visualizations is probably
not the best route, usually.” (P2).
ASSETS ’22, October 23–26, 2022, Athens, Greece
3.3 Study fndings 2: People with I/DD use
existing apps for PTSR but these are not
well-suited to their needs.
Next, we wanted to understand what apps or technologies people
with I/DD use for PTSR purposes. Further, we also wanted to under-
stand how well-suited these solutions were for our population. We
describe our fndings below. Again, we provide verbatim quotations,
which are edited for brevity and clarity using ellipses and brackets,
respectively.
3.3.1 People with I/DD use a variety of ersatz apps for PTSR. Ac-
cording to our participants, people with I/DD use a variety of apps
for PTSR purposes. These can be grouped into four broad categories
(as before, we list the number of participants who suggested each
category). They include the use of: audio-visual apps (e.g., YouTube,
Netfix, Spotify, video games) (N = 5); social media apps (e.g., In-
stagram, TikTok, Facebook, and Pinterest) (N = 3); creativity apps
(e.g., coloring books, puzzles, creating music playlists) (N = 2); a
digital companion (e.g., Care.Coach) (N = 1); viewing pictures of
cute animals (N = 1).
These data show that the apps used by people with I/DD, accord-
ing to our participants, roughly match the diversity of the PTSR
activities that they recommended as potentially useful. The only
categories not found here, as compared to the section above, in-
clude outdoor activities and observing one’s feelings/emotions. It
is unsurprising that the former is not included here, since outdoor
activities are not suited to implementation on an app modality. It is
also unsurprising that the latter is not present here, since all of the
apps in this section are essentially ersatz PTSR apps. This means
that were not explicitly designed for self-regulation, which often
involves refecting on how one is feeling in the moment. Going
forward we refer to the app categories listed in this section as er-
satz apps. Interestingly, none of our participants mentioned people
with I/DD using of any of the trauma-focused self-regulation apps
developed for conditions like PTSD (as mentioned in Section 2).
3.3.2 None of the ersatz apps is well-suited for people with I/DD.
Although people with I/DD were using a variety of apps for PTSR
purposes, they were using apps that were neither explicitly designed
for this population nor were they using apps that are designed
for coping with trauma. When it came to these ersatz apps, the
participants felt that the apps were not well suited for PTSR for
people with I/DD. The practitioners explicitly mentioned three
categories of ersatz apps when discussing their unsuitability for
PTSR for people with I/DD:
• Commercial non-trauma self-regulation apps: The practition-
ers felt that commercial apps were not designed to be usable
for people with I/DD because of the cognitive load they
impose: “I would not recommend any of those [commercial
self-regulation] apps to any of my clients. I would never. The
meditations are weird... You need to know how to do too much
independent navigation to use any of those apps. It’s too much...
There were no pictures on anything... everything should be writ-
ten at most a ffth-grade reading level. If my client, who’s never
meditated before, is going to use a guided meditation, like, ’No!’
... I need what’s an intro level. I need like a video or something.
My clients, they don’t use this stuf on a regular basis.” (P3).
Venkatasubramanian and Ranalli
• Non-commercial trauma-focused apps: The practitioners felt
that these apps were not designed to be accessible for people
with I/DD, despite being designed for helping with trauma:
“I remember the military app like defning PTSD. That’s like a
clinical defnition of PTSD, right? I don’t need that... it’s also
there’s not a lot of personalization, which I think is like the
core part of the work we do... I think you just can’t personalize
any of those apps the way that they need to be personalized
for our clients.” (P3).
• Social media apps: These apps were viewed as being too risky
for being used for PTSR purposes: “God no! Social media isn’t
designed for anybody dealing with trauma. No one.” (P4).
3.4 Study fndings 3: The design of PTSR apps
for people with I/DD needs to promote
autonomy, be accessible, and enable social
connections
We next asked our participants what designers of PTSR apps should
be mindful of when designing for someone with I/DD. The partic-
ipants’ responses can be categorized into three broad categories,
which we describe below. Once again, we provide verbatim quo-
tations, which are edited for brevity and clarity using ellipses and
brackets, respectively.
3.4.1 Support the autonomy of people with I/DD in design to make
decisions about their self-regulation. As mentioned before, one of
the most signifcant sources of trauma in the lives of people with
I/DD was not having control over their body, words, and lives in
general. Therefore, one of the main design elements that PTSR apps
should espouse is the provision of control, that is, helping people
with I/DD control the type of self-regulation that they want to do:
“I think people want to have the option of what type of self care that
they engage in and what it’s relating to... I’ve noticed having topics
people like... food, TV, exercise, and/or trauma and coping.” (P1). In
this regard, the app should not try to tell people with I/DD what to
do. Rather it should guide them to identify what they need: “I guess
listing or being able to identify what it is their needs are, for one, and
that’s going to vary from individual to individual. So a section... for
them to be able to pinpoint or identify, ‘okay, I need this’ or... ’these
are my basic needs’....” (P8).
Furthermore, supporting autonomy also includes understanding
that some (perhaps most) people will not want to talk or engage with
the trauma when doing PTSR: “... framing the app in a way where
the individual, if they want to use and talk about their trauma, they
can but it’s not a requirement to use the app... sometimes somebody
might, say, want a self-care activity, might want to do something but
doesn’t want to specifcally think about their sexual assault, doesn’t
want to specifcally think about why they need that self care, just
they need a self care activity.” (P6). Moreover, there is a common
misconception that PTSR requires engaging with the trauma: “but
we talk about skills... you don’t have to talk about trauma to respond
to it or heal from it. I think that’s a big misconception that people
have.” (P3). This notion that PTSR need not involve engaging with
the trauma at all builds on our earlier idea from this section where
the person with I/DD ultimately gets to decide what works for them
when it comes to PTSR.
Designing Post-Trauma Self-Regulation Apps for People with I/DD
Finally, supporting the autonomy of a person with I/DD also
means being aware that what triggers someone in a given moment
is often unpredictable to designers beforehand: “I think you never
know what [someone’s trigger] is... because I have a client who was
triggered by the word ‘guys,’ right?... and saying the word ‘guys’ and
being perceived as not feminine was a big trigger for that client and I
would mess up. I would have to write myself a note to... remember not
to say the word but that’s just a small example of what it can look like.”
(P3). Since triggers cannot be predicted the design should include
appropriate provisions to mitigate the efects of such triggers by:
(1) making sure that users do not feel blamed for their triggers: “You
don’t want something that would inadvertently make a person feel
blamed or shamed.” (P2), and (2) allowing the person using the app
to individualize (customize) the app to their personal situation and
needs.
3.4.2 Be accessible and intersectionally inclusive in design. In terms
of the eventual design of the app, given that it is ultimately being
designed for someone with I/DD, it must be accessible to them.
Our participants specifed several key properties for accessibility.
These included: the use of simple and easy-to-understand language
that is concrete rather than abstract. The idea is to tend toward
over-explaining things within the app: “I think [the activities] would
have to be really broken down. So I know sometimes if I will go on
like meditation apps, if I listen to people talk, they sometimes will
use very large words, talk about like the space, the heavens. But for
some people, that’s really not accessible and so just something that’s
very simple, very clear. If we’re breathing in: ‘breathe in for three
seconds. I’m going to count with you: one, two, three,’ things like that.
Over-explaining to where you think it’s not necessary but for someone
with a disability, that could be really helpful because again there’s a
diference between telling somebody ‘breathe in’ and ‘we’re going to
breathe in for three seconds through our nose and we’re going to try
and feel it in our chest or in our stomach.”’ (P6).
An interesting point on this topic raised by one of the participants
was that the app should be accessible to people with I/DD without
calling obvious attention to that fact. In other words, people with
I/DD want to be treated like everyone else, as a person frst instead
of being approached in terms of their disability: “an app that’s
accessible without screaming, ‘I’m accessible,’... So not necessarily of
the bat, you sign on to the app and it’s like, do you need large font?...
I just think that puts folks with disabilities of; they’ve told me it has.
So I just think it needs to look like an app that someone without a
disability could use but creatively having [the team] who’s creating it
know that it does need to be accessible and these are some particular
things for folks with intellectual and developmental disabilities that
you need to know....” (P1).
The participants also suggested that the design of the material
used in the app should be more inclusive, in terms of the types
of people shown, than apps have traditionally tended to be. This
means refecting and respecting diferences in lived experience in
terms of things like language, ethnicity, culture, gender expression,
sexuality, religion, and more of the target I/DD community: “I think
inclusion in making sure the app represents folks on a lot of diferent
identities and skin colors. I think also [that] the disability work can
be really whitewashed.” (P3). In other words, participants suggested
that we take an intersectional approach to the problem of designing
ASSETS ’22, October 23–26, 2022, Athens, Greece
for our target community. When we use the term intersectional
approach to design, we mean design that is aware of the multitude
of factors that afect the lived experience of a person’s life: “People
[with I/DD] haven’t just experienced ableism but have also experienced
sexism, racism, and homophobia. I think it’s important to [consider
this intersectionality] for how we [think about] helping somebody.”
(P2). Therefore an intersectionally inclusive design of PTSR apps
should include content that: accommodates diferent literacy levels
(to promote I/DD accessibility); employs gender-neutral language
and avoids both gender binarism and gender stereotypes (to avoid
biases like transphobia, homophobia, and sexism); uses images of
people with diverse skin tones and body types (to avoid typical
content whitewashing and fxations on certain notions of beauty),
etc.
3.4.3 Include some form of social connection in the design. An in-
teresting pattern that we observed in the use of the ersatz apps
was that people with I/DD were using them to try to engage in
social connection in some form or another. This is particularly true
for social media apps: “I know a lot of [people with I/DD]... [use]
social media for self [regulation]. A lot of them will do that... for
social connection, talking to people, feeling like they’re connected,
they’re not isolated.” (P6). Additionally, the participants also men-
tioned social connection in the context of using music for PTSR by
sharing playlists with others: “... the person I was talking about who
has a communication device... his device connects to YouTube and he
shares music videos all the time.” (P2). In the same vein, a recurring
theme in the design of PTSR apps was the need to promote social
connection for people with I/DD: “I just envision an app more of like
a dialog between people with I/DD as opposed to some of [the] apps
that you would see now.... [For PTSR, people with I/DD]... really like...
the dialog and the socialization, especially during a time like this. So
if they have an app that... really flls the void of the socialization piece
that none of [the clients] have right now because they’re so isolated.
They want to feel like they’re talking to somebody and if they feel
like they’re talking to their peer, that’s, to me, even better.” (P1). The
participants suggested several alternatives for establishing social
connection for people with I/DD as part of PTSR.
• Sending and receiving positive afrmations among people
with I/DD as a way to support one another: “People [with
I/DD] really like words of wisdom and hearing things from
their peers... I’m wondering if they could have like a daily
positive quote from someone who’s been through something
that they have, that they just check it out because that kind of
gets addicting, right? Like I’m waking up this morning, I want
to hear what one of my peers has to say about this morning
and it’s like a positive quote to get their day going. I think
things like that would be cool just coming from their peer or if
you wake up having a crappy day, ’here’s what I’ve done. If
you click on this, you can do it for yourself too.”’ (P1).
• Utilizing voice-based interactive systems that help the person
feel connected (even if that’s not with another person but
rather with a bot): “If there’s voice activation, you don’t want
something that’s bossing you around... You want something
that makes you feel positive about yourself and also helps you
feel like you’re in control.” (P2).
ASSETS ’22, October 23–26, 2022, Athens, Greece
• The words and lived experience of people with I/DD should
be conveyed as part of the interaction: “If [people with I/DD’s]
words and experiences are somehow captured in the app, I just
think that’s a catch... I think the folks are more inclined to
tune in when it’s... coming from somebody that’s lived the
experience and has a suggestion that worked for them.” (P1).
4 TRAUMA-INFORMED CARE (TIC)
Based on the fndings of our interview study, we posit that one way
of approaching the design of PTSR for people with I/DD is based on
the notion of trauma-informed care (TIC). TIC is an idea obtained
from social work practice that is used by organizations to support
traumatized individuals. Before we delve into how this can be done
(in the next section), in this section we provide a brief overview of
TIC, its suitability for helping trauma survivors with I/DD, and its
fve main criteria.
4.1 Overview of trauma-informed care
The main idea behind TIC is that organizations providing trauma
services should consider the roles of trauma and its lingering efects
on the lives of individuals who have experienced trauma. Hence,
the idea is for the entire culture of an organization that deals with
trauma survivors to be aware of the trauma their clients have ex-
perienced [21, 46]. Organizations providing TIC services rely on
their staf’s knowledge about trauma in responding to clients in
ways that convey respect and compassion, foster the autonomy
of the survivor, and aid in building coping strategies [21, 46]. TIC
is diferent from trauma-focused therapy (e.g., exposure therapy),
as its primary goal is not to directly address or process the past
trauma (i.e., the root of the problem per se) but to provide a process
to deal with what the client presents as a result of their traumatic
experience(s) [46].
In recent years, there has been an increased understanding of
the need for TIC for people with I/DD. Given that people with I/DD
often need organizational support to manage their daily lives and
require such support throughout their lives, TIC has been seen as a
natural way of helping people with I/DD manage their situation
[39]. It has been recognized that, through TIC practice, people with
I/DD can: develop a sense of safety; feel reduced shame and guilt;
and be empowered and engaged to minimize the impact of the
traumatic experience [21].
4.2 Trauma-informed care and the postmodern
model of disability
Even though TIC did not necessarily develop in the context of
helping people with disabilities, it aligns with the postmodern model
of disability as described in [48].
4.2.1 Models of disability. Mankof et. al provided a beautiful
overview models of disability in this seminal paper [48]. In it they
described three models of disability, which we quickly summarize
here for context. Traditionally, disability has been conceptualized
frst as something to cure, as part of what is known as the medical
model of disabilities. This model conceives of people with disabil-
ities as lacking and in need of being “fxed" to bring them to some
notion of “normal". However the medical model rests uncomfortably
Venkatasubramanian and Ranalli
on an assumption that the goal is normality - that is, the elimination
of disability. Conceptualizing disability using the medical model can
have drastic negative consequences on the autonomy and decision-
making power of people with disabilities [48]. This was followed in
the 1970s by the social model, which shifts the narrative from a need
for cure to a need for care where patients, not clinicians, become
the leaders in managing their conditions. The social model and the
associated independent living movement promote self-advocacy
and peer support as the frst steps toward full participation in so-
ciety, citizenship, and leadership development. The social model,
however, also has some limitations. For example, if disability is only
defned at the level of society, then the individual’s experience is
to some extent invalidated and the question of accommodations at
the level of the individual sidelined [48]. Among the models that
have been proposed to move beyond the medical and social models,
the postmodern model is, to us, most apt. Based on the concept
of postmodernity from the humanities, this model incorporates
some aspects of the medical and social models that can still be
considered relevant. Rather than simply claiming that these former
approaches no longer have anything to ofer: it includes a focus
on the individual’s experience of disability (similar to the medical
model, though signifcantly expanded) as well as a consideration of
how society constructs disability (from the social model). The post-
modern model privileges each individual’s unique lived experience,
complete with the complexity and nuance of everyday life [48].
4.2.2 Trauma-informed care and the postmodern model of disability.
Trauma-informed care embodies the postmodern model of disabil-
ity for many reasons. First, TIC does not view trauma purely as
a medical issue to be cured. It achieves this by viewing negative
efects of trauma as coping mechanisms for the trauma survivor as
opposed to seeing them as some form of pathology, thus highlight-
ing the survivor’s resilience. This reframing of negative trauma
efects as adaptations ultimately allows support organizations to
help the survivor develop healthier coping mechanisms without
stigma or guilt [39]. Second, TIC considers traumatic experiences
(e.g., a feeling of loss of bodily autonomy) not as past events but as
defning experiences that shape the core of the survivor’s current
identity. Therefore, TIC fosters the survivor’s autonomy, control,
and say over their own well-being [39]. Third, TIC is careful not to
ascribe the cause of negative trauma efects to existing disabilities.
Consequently, organizations implementing TIC take extreme care
not to use language, behaviors, scenarios, or practices that can po-
tentially retraumatize a trauma survivor [39]. Overall, TIC, being
cognizant of the negative efects of trauma on the lives of people, is
by its very nature an ideal framework to meet the needs of people
with I/DD afected by trauma.
4.3 The fve criteria of trauma-informed care
The Substance Abuse and Mental Health Services Administration
(SAMHSA) qualifes that any organization seeking to provide TIC
services must satisfy fve core criteria: safety, trust, choice, col-
laboration, and empowerment [31]. These criteria were originally
defned for the general population of trauma survivors. However, in
recent years these criteria have been seen as applicable to the I/DD
community as well [39, 86]. Based on the work by John Keesler
Designing Post-Trauma Self-Regulation Apps for People with I/DD
[39], in this section we defne the fve criteria of TIC in terms of
people with I/DD:
Safety: The organization should recognize that people with I/DD
are likely to have a history that includes trauma. Consequently, it
should explicitly provide an environment where the person with
I/DD feels both physically and emotionally safe. Another way of
putting it is that the environment should be such that it does not
retraumatize the person with I/DD.
Trust: Trust is essential to the care process to ensure that the
person with I/DD is able to develop the autonomy, initiative, and
competence required to empower themselves to manage with the
efects of their trauma. From an organizational standpoint, this
criterion relates to a clarity of expectations for the person with
I/DD in terms of receiving consistent service delivery across the
organization.
Choice: Oftentimes trauma causes a person with I/DD to lose
their sense of control over their lives and their bodies. The concept
of choice within TIC is an attempt to enable the person with I/DD
to make their own decisions and gain a sense of control over their
recovery and life.
Collaboration: When a person with I/DD receives services for
their trauma from a support organization, a power diferential
can exist in that relationship. If not carefully equalized, such a
power imbalance (often subtle and insidious) can increase feelings
of vulnerability in the trauma survivor. An essential feature of TIC
is equalizing the potential power diferential between the survivor
and any support organization (and their staf) in decision-making
around care. For instance, people with I/DD are often prone to
the seek acceptance of others and are particularly vulnerable to
instinctive compliance to authority. Therefore, the collaboration
criterion requires that we remind trauma survivors with I/DD that
they have the right to ask questions, decline services, and make
requests.
Empowerment: This criterion essentially does two things. First, it
recognizes the behavior of a trauma survivor with I/DD as a legiti-
mate way for them to deal with past trauma. It therefore validates
their current experience, thus empowering trauma survivors to
take their well-being in their own hands. Second, it recognizes the
individual strengths, skills, and abilities of the person with I/DD
and aids them every step of the way. This focus on the survivor’s
existing strengths and skills is seen as an integral way to help them
realize that they may already possess the resources necessary to
develop a solution for any obstacles they face in dealing with the
efects of the trauma.
5 INCORPORATING TRAUMA-INFORMED
CARE INTO THE DESIGN OF PTSR APPS
FOR PEOPLE WITH I/DD
In this section, we explore the design space for PTSR apps for
people with I/DD using the notion of trauma-informed care (TIC)
discussed in Section 4. To the best of our knowledge no prior work
has attempted to look at applying TIC to app design. In the rest of
this section we discuss how each of the TIC criteria can be adapted
to the design of PTSR apps by exploring important design guidelines
related to each criterion. We derived these design guidelines from
the responses from the interviews we conducted as well as our years
ASSETS ’22, October 23–26, 2022, Athens, Greece
of experience designing technologies for people with I/DD. Next
we describe each of these design guidelines, which are summarize
in Table 3. In this section, we use the term user(s) to specifcally
mean individual(s) with I/DD.
5.1 Designing for safety
The frst criterion we consider is that of safety, where the idea
is to provide users with a sense of physical and emotional safety.
Likewise we defne safety within PTSR apps as designing with the
intent to minimize any retraumatization of users and to provide a
means for them to cope with negative trauma efects. We suggest
three guidelines for ensuring the safety criterion for PTSR apps.
Mitigate known negative trauma efects. The design of PTSR apps
should be cognizant of the dis-ease that its users may be experienc-
ing and ensure that it does not exacerbate it. Consequently, when
designing PTSR apps, similar eforts must be made to help users
to manage known negative trauma efects. For instance, the app
should be cognizant of the fact that, for some people with I/DD, self-
regulation activities that are abstract and involve the imagination,
like visualization activities, can be distressing rather than calming.
Similarly, closing one’s eyes and going right into a meditation with
no preparatory safety measures is not ideal for those who have
experienced trauma (including people with I/DD). Thus, the app
should give users options of various safety measures they can take
before and/or during meditation (like keeping their eyes open or
frst check that the doors are locked) to make sure they feel safe
and in control in their surroundings.
Be actively supportive of the lived experience of people with I/DD.
The I/DD community is diverse. Thus the design of PTSR apps
should be considerate of the diferences in lived experience in terms
of things like language, ethnicity, culture, gender expression, sex-
uality, religion, and more of the target I/DD community. An app
that does not actively represent a worldview that is supportive of
the diversity of people with I/DD essentially rejects non-normative
body and mental states, which is oppressive and traumatizing, thus
negatively impacting user safety [59]. For instance, any voice com-
munication in PTSR apps designed to be inclusive of trans and
nonbinary people with I/DD should include more than just stereo-
typically cis male and female genders in its vocal elements.
Assume that “potentially anything can trigger”: For someone who
has experienced trauma, anything that reminds them of their trau-
matic experience can trigger negative efects. Therefore, designers
need to be aware of the fact that even self-regulation activities can
potentially trigger users. Triggers are inherently personal and are
not the same for any two people [42]. It is thus often very difcult
for designers to know a priori what can trigger a user when using
PTSR apps. We thus believe that PTSR apps should be designed
with the “potentially anything can trigger” mindset, which means
that it should: (1) use non-judgmental language and features in the
app and (2) include a variety of self-regulation activities from which
to choose, including ofine activities that people with I/DD like
(some examples that participants in our study mentioned include:
sit with a pet, gardening, coloring, taking a walk, mowing the lawn,
being outdoors, yard work, and crafting).
ASSETS ’22, October 23–26, 2022, Athens, Greece Venkatasubramanian and Ranalli

TIC criterion Guidelines

1. Mitigate known negative trauma efects
Safety 2. Be actively supportive of the lived experience of people with I/DD
3. Assume that potentially anything can trigger
4. Ensure authentication to protect access to user-generated personal data within the app
Trust
5. Never take away recognition for prior successes
6. Provide a diversity of choices by designing for a variety of abilities
Choice 7. Guide users through the choice-making process
8. Give users an easy-to-understand way to express their current feelings/emotions
9. Provide the ability to interact with others
Collaboration
10. Provide tools that converse directly with users
11. Provide accessible information on trauma
Empowerment
12. Provide customized, pithy messages of support

Table 3: A summary of guidelines obtained for incorporating trauma-informed care into the design of PTSR apps
5.2 Designing for trust
The second criterion is to develop trust with people with I/DD.
When it comes to PTSR apps, we defne trust as an app’s ability
to foster confdence in users, such that they keep using the app to
the fullest extent. We suggest two guidelines for ensuring the trust
criterion for PTSR apps.
Ensure authentication to protect access to user-generated personal
data within the app. PTSR apps provides services and features that
are intensely personal and should not be available for anyone else
to see without explicit permission from the user. For instance, one
could imagine an app that allows a user to journal or check in about
their emotions and feelings at diferent points in time. For a user to
be able to avail themselves of the benefts of journaling/check-ins,
they have to feel safe in doing so honestly. Providing a way to
secure this within an app from any unauthorized access is a power-
ful way to incentivize users to engage with such features. In this
regard, one of the main features an app should have is authentica-
tion, especially for any personal data. Authentication would ensure
that only the user can access their personal information. Indeed,
password-based authentication is used by the Safe Helpline app
[68] to control access to any user-generated information. However,
it frequently requires one’s password to be retyped (approximately
once every fve minutes), which can make using the app tedious and,
for people with I/DD, it can make using the app difcult. Designing
an authentication solution that works for people with I/DD in the
context of protecting the content of PTSR apps is a challenging
problem, given that people in this population often need to share
their passwords with close family members [80].
Never take away recognition for prior successes. Having incentives
within the app is a powerful way to encourage users to use PTSR
apps over time. The idea of giving people incentives for using
the app came up during the interviews: “So maybe having some
sort of incentive or like... “Oh, you completed this series of coping
skills, you get a badge”... because I also have a background in applied
behavior analysis, so I think creating incentive to show that they
accomplished something, it would be helpful for them to wanting to
be engaged in it.” (P7). Often incentives within apps (in general)
are designed to be negative in nature, where prior rewards are
rescinded if certain goals are not met. This can be clearly seen in
apps where an anthropomorphic virtual being becomes happier and
sadder with app use and disuse, respectively [18, 29, 63]. However,
in our prior work we found that incentives designed for people with
I/DD should never penalize and should only add to prior successes
[34]. Consequently, any incentive elements in PTSR apps should
therefore accumulate in-app successes. We believe that this would
allow users to trust the motives of PTSR apps (e.g., that the app is
not negatively judging the users’ performance) and therefore use it
more fully.
5.3 Designing for choice
The third criterion is that of choice, where people with I/DD are
given the chance to make choices and gain control over their re-
covery and life. PTSR apps that incorporates the choice criterion
should provide their users with a variety of ways to achieve their
self-regulation goals. We suggest three guidelines for ensuring the
choice criterion for PTSR apps.
Provide a diversity of choices by designing for a variety of abilities.
One way to increase the choices for users of PTSR apps is to ensure
that a given self-regulation activity is available at diferent levels of
difculty to accommodate the diferent cognitive abilities of people
with I/DD. For instance, a jigsaw puzzle activity designed for PTSR
could provide several diferent versions of the same puzzle with
difering degrees of difculty. For instance, in addition to allowing
users to use hints or not, this activity could also ofer things like
diferent numbers of pieces or diferent degrees of intricacy in the
image. Ofering several difculty levels for an activity would allow
users to select a level not only based on their general abilities but
also according to what they feel capable of doing on a given day or
in a given situation.
Guide users through the choice-making process. So far we have
argued for the need for a diversity of choices for self-regulation ac-
tivities. However, as the number of self-regulation activities within
an app increases, it is not enough for an app simply to provide a
list of all of the options. PTSR apps should include a recommender
system to ofer users self-regulation activity suggestions in order
to help facilitate the task of selecting an activity and thus reduce
the cognitive load imposed on users. However, given the negative
efects of trauma on individuals with I/DD (as described in Section
1.1), such as impaired working memory and a tendency to be easily
distracted, the recommender system may have to considerably limit
the number of options shown at a time. That being said, any PTSR
app’s recommender system should be careful just to suggest options
and not dictate what the user should do.
Designing Post-Trauma Self-Regulation Apps for People with I/DD
Give users an easy-to-understand way to express their current
feelings/emotions. How to design a recommender system in PTSR
apps for people with I/DD is an open question. One crucial input for
the recommender system is the current emotional state of the user,
which is something that extant trauma-focused self-regulation apps
often ofer. For instance, PTSD Coach [66] provides a barometer
from 1 to 10 that can be used to indicate one’s level of distress4 . This
is a good approach for populations without I/DD but it involves an
abstract way of expressing how one is feeling in the moment, which
can be difcult for people with I/DD to determine about themselves
[74]. Consequently, an app designed for people with I/DD needs to:
(1) explicitly guide the user through the process of determining how
they are feeling in that moment, perhaps through an illustrated,
guided emotional check-in process [37] and (2) make the manner of
expressing the user’s level of distress less abstract, such as through
the use of a scale with faces to represent the diferent emotions, as
in [65].
5.4 Designing for collaboration
Traditionally, the collaboration criterion is about equalizing the
power diferential between the individual with I/DD and the staf
at a support organization. When it comes to PTSR apps, it is clear
that collaboration needs to be defned diferently. Given the nature
of mobile computing devices, in that they are typically devices
for individual use, working with mobile apps is usually a solitary
activity. Furthermore, self-regulation, which is the focus of this
paper, can increase the user’s sense of isolation even more. As P6
pointed out (in Section 3.4.3), one of the biggest sources of difculty
comes from being isolated from others. Therefore, PTSR apps, as
part of the collaboration criterion, should actively allow the user to
communicate with others in a meaningful fashion and to engage in
collaborative activities and community formation [6]. We suggest
two guidelines as part of the collaboration criterion for PTSR apps.
Provide the ability to interact with others. We believe that PTSR
apps for our population should provide a means for users to interact
with others for social support. This can be friends and family, allies,
a designated trusted person, therapy or support groups, and other
people with I/DD. This support can take many forms, including
informational support, emotional support, personal network sup-
port, self-esteem support, and even tangible in-person support [19].
For instance, several trauma-focused self-regulation apps, such as
those designed by for US Department of Defense personnel [61, 68],
provide their users with a means to call someone to talk. However,
these apps primarily focus on phone-call-based communication.
One way of promoting the ability to communicate with others for
people with I/DD is to leverage the paradigm of social VR. Social
VR would allow individuals with I/DD, who otherwise live highly
managed lives, to move around and communicate with others in a
simulated, immersive space in imaginative ways. VR would thus
allow them not only to perform PTSR but also to interact with oth-
ers on their own terms: the virtual environment would allow them
to socialize without having to wait for care assistants to arrange
transportation, etc., as they often do when physically socializing
4 PTSD Coach uses the catchall term distress to signify one or more of a host of
psychological and physiological states (“anger, sadness, fear, pain, stress, worry, or
anything negative they are feeling”).
ASSETS ’22, October 23–26, 2022, Athens, Greece
with others. Giving this community unmediated access to socializ-
ing would reduce some of the very sources of repeated trauma that
they endure.
Provide tools that converse directly with users. PTSR apps need
not only promote social collaboration involving other individuals.
One can also imagine PTSR apps interacting directly with users to
simulate social interactions, similar to AI-assisted chatbots or digital
voice assistants. Such interactive tools can help people with I/DD
feel less isolated in the absence of others with whom to interact.
However, for interactive tools to be truly benefcial to individuals
with I/DD, they should react and talk like a peer (i.e., another person
with I/DD). At a minimum, existing chatbots and voice assistants
have to become more inclusive and be trained on data emanating
from people with I/DD.
5.5 Designing for empowerment
In an organizational context the empowerment criterion is about
validating the individual’s trauma experience and focusing on their
abilities5 . In terms of PTSR app design, we leverage the notion of
knowledge is power and defne empowerment as accessibly pro-
viding the user with information about trauma, its manifestations
and efects, and diferent ways to cope and heal. The availability
of information surrounding trauma is a powerful way to validate
users’ experiences, remove any stigma surrounding trauma [28],
and better foster their ability to engage in self-regulation. We sug-
gest two guidelines for ensuring the empowerment criterion for
PTSR apps.
Provide accessible information on trauma. Apps focused on trauma
for people without I/DD often provide a lot of information (in the
form of reading material) about trauma and why it is important to
address it [61, 68]. Many non-trauma commercial self-regulation
apps similarly provide a lot of information about the importance and
benefts of meditation practices [75, 82]. However, such a reading-
intensive approach is not appropriate for an I/DD population. Prior
work in HCI/accessibility has identifed several approaches for
teaching, informing, and skill-building for the I/DD community.
Some of the major fndings include recommendations to: (1) use im-
ages [11, 54], interactive content, and videos [5, 23]; (2) use concrete,
large, diferentiated icons and symbols [7]; (3) make the content
accessible by supporting audio descriptions of images [3, 79] and
accommodating diferent levels of literacy [20]; (4) reduce infor-
mation overload [20]; (5) use positive reinforcement for successful
engagement with the content [11, 43]; and (6) provide self-paced
learning capabilities [43, 44, 54, 70]. These approaches should be
adapted to convey information about trauma and self-regulation to
people with I/DD within PTSR apps.
Provide customized, pithy messages of support. Another way to
empower users via PTSR apps is to provide short afrmations that
validate their current experience and reduce any stigma surround-
ing trauma. Once again these messages have to be designed with
the capabilities of users in mind. For instance, in our prior work
[34], we developed an abuse-recognition tool for people with I/DD
5 The empowerment criterion in this context lends itself naturally to the notion of
ability-based design in HCI [87]. We believe that an entire app, in order to be accessible
to people with I/DD, has to be designed with empowerment in mind. Therefore, we do
not list ability-based design as a specifc guideline in this section, as we view it as a
meta-guideline that holds for any app designed for people with I/DD.
ASSETS ’22, October 23–26, 2022, Athens, Greece
that frequently reminds users that “abuse is not their fault.” It is
understood that people with I/DD often need repetition to fully
appreciate a concept [34]. Consequently, such empowering, easy-
to-understand messages should appear passim throughout an app
and should be repeated often. Moreover, for such messages to be
efective for a variety of people with I/DD, the messages should be
presented via multiple modalities (such as text, pictures/symbols,
audio, and video) and should be in the voices of people with I/DD.
6 CONCLUSION
People with I/DD are some of the most traumatized individuals
in the US. We posit that mobile-computing-based apps have the
potential to help people with I/DD deal with the negative efects of
trauma. In recent years a plethora of research, commercial, and non-
commercial self-regulation apps have become available but they
have not been designed for people with I/DD. Thus this community
has turned to cobbling together aspects of ersatz apps for self-
regulation. However, the apps this population has been using were
never designed for them, nor are they appropriate for coping with
trauma. Consequently, we interviewed eight (8) practitioners at a
trauma services organization in the US to better understand what it
would take to develop post-trauma self-regulation apps for people
with I/DD. Based on the interview responses, we then developed
a set of guidelines, based on the social work practice of trauma-
informed care, to design post-trauma self-regulation apps for people
with I/DD. In the future, we plan to instantiate a prototype of a
PTSR app based on our design guidelines and then assess the app
through user studies involving members of the I/DD community.
ACKNOWLEDGMENTS
We would like to thank our reviewers and participants for their
invaluable help with this paper. We would also like to thank Nancy
Alterio, Mariah Freark, Priyankan Kirupaharan, Andrew Laraw
Lama, and Brittany Lewis for their help with this work. This work
was made possible by generous funding from the Massachusetts
Disabled Persons Protection Commission.
REFERENCES
[1] AAIDD 2020. American Association of Intellectual and Developmental Disabili-
ties. AAIDD. Retrieved Feb 27, 2020 from https://www.aaidd.org/intellectual-
disability/defnition
[2] Jesper J Alvarsson, Stefan Wiens, and Mats E Nilsson. 2010. Stress recovery
during exposure to nature sound and environmental noise. International journal
of environmental research and public health 7, 3 (2010), 1036–1046.
[3] Andrew A. Bayor. 2019. HowToApp: Supporting Life Skills Development of Young
Adults with Intellectual Disability. In The 21st International ACM SIGACCESS
Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19).
Association for Computing Machinery, New York, NY, USA, 697–699. https:
//doi.org/10.1145/3308561.3356107
[4] Herbert Benson and Miriam Z Klipper. 1975. The relaxation response. Morrow,
New York, NY.
[5] Petra Boström and Eva Eriksson. 2015. Design for self-reporting psychological
health in children with intellectual disabilities. In Proceedings of the 14th Interna-
tional Conference on Interaction Design and Children. Association for Computing
Machinery, New York, NY, USA, 279–282.
[6] Alice V. Brown and Jaz Hee-jeong Choi. 2017. Towards Care-Based Design:
Trusted Others in Nurturing Posttraumatic Growth Outside of Therapy. In Pro-
ceedings of the 8th International Conference on Communities and Technologies
(Troyes, France) (C&T ’17). Association for Computing Machinery, New York,
NY, USA, 56–63. https://doi.org/10.1145/3083671.3083703
[7] Erin Buehler, William Easley, Amy Poole, and Amy Hurst. 2016. Accessibility
Barriers to Online Education for Young Adults with Intellectual Disabilities. In
Proceedings of the 13th Web for All Conference (Montreal, Canada) (W4A ’16).
Venkatasubramanian and Ranalli
Association for Computing Machinery, New York, NY, USA, Article 27, 10 pages.
https://doi.org/10.1145/2899475.2899481
[8] Eleanor R. Burgess. 2019. Collaborative Self-Management of Depression. In
Conference Companion Publication of the 2019 on Computer Supported Cooperative
Work and Social Computing (Austin, TX, USA) (CSCW ’19). Association for Com-
puting Machinery, New York, NY, USA, 38–42. https://doi.org/10.1145/3311957.
3361851
[9] Eleanor R. Burgess, Kathryn E. Ringland, Jennifer Nicholas, Ashley A. Knapp,
Jordan Eschler, David C. Mohr, and Madhu C. Reddy. 2019. "I Think People
Are Powerful": The Sociality of Individuals Managing Depression. Proc. ACM
Hum.-Comput. Interact. 3, CSCW, Article 41 (nov 2019), 29 pages. https://doi.org/
10.1145/3359143
[10] Eleanor R. Burgess, Alice Renwen Zhang, Jessica L. Feuston, Madhu C. Reddy,
Sindhu Kiranmai Ernala, Munmun De Choudhury, Stephen Schueller, Adrian
Aguilera, and Mary Czerwinski. 2020. Technology Ecosystems: Rethinking
Resources for Mental Health. In Extended Abstracts of the 2020 CHI Confer-
ence on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA
’20). Association for Computing Machinery, New York, NY, USA, 1–8. https:
//doi.org/10.1145/3334480.3375166
[11] Luke Buschmann, Lourdes Morales, and Sri Kurniawan. 2014. Online Learning
System for Teaching Basic Skills to People with Developmental Disabilities. In
Proceedings of the 16th International ACM SIGACCESS Conference on Computers &
Accessibility (Rochester, New York, USA) (ASSETS ’14). Association for Comput-
ing Machinery, New York, NY, USA, 271–272. https://doi.org/10.1145/2661334.
2661391
[12] Lisa D Butler, Filomena M Critelli, and Elaine S Rinfrette. 2011. Trauma-informed
care and mental health. Directions in Psychiatry 31, 3 (2011), 197–212.
[13] David Byrne. 2022. A worked example of Braun and Clarke’s approach to refexive
thematic analysis. Quality & quantity 56, 3 (2022), 1391–1412.
[14] Lucille Cairns. 2015. Bodily Dis-ease in Contemporary French Women’s Writing:
Two Case Studies. French Studies 69, 4 (2015), 494–508.
[15] Calm 2022. Calm. Calm. Retrieved June 20, 2021 from https://www.calm.com/
[16] Prateek Chanda, Amogh Wagh, Jemimah A. Johnson, Swaraj Renghe, Vageesh
Chandramouli, George Mathews, Sapna Behar, Poornima Bhola, Girish Rao,
Paulomi Sudhir, T. K. Srikanth, Amit Sharma, and Seema Mehrotra. 2021. MIND-
NOTES: A Mobile Platform to Enable Users to Break Stigma around Mental
Health and Connect with Therapists. In Companion Publication of the 2021 Con-
ference on Computer Supported Cooperative Work and Social Computing (Virtual
Event, USA) (CSCW ’21). Association for Computing Machinery, New York, NY,
USA, 213–217. https://doi.org/10.1145/3462204.3482895
[17] Janet X. Chen, Allison McDonald, Yixin Zou, Emily Tseng, Kevin A Roundy, Acar
Tamersoy, Florian Schaub, Thomas Ristenpart, and Nicola Dell. 2022. Trauma-
Informed Computing: Towards Safer Technology Experiences for All. In Proceed-
ings of the 2022 CHI Conference on Human Factors in Computing Systems (New
Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York,
NY, USA, Article 544, 20 pages. https://doi.org/10.1145/3491102.3517475
[18] Simone Colombo, Franca Garzotto, Mirko Gelsomini, Mattia Melli, and Francesco
Clasadonte. 2016. Dolphin Sam: A Smart Pet for Children with Intellectual
Disability. In Proceedings of the International Working Conference on Advanced
Visual Interfaces (Bari, Italy) (AVI ’16). Association for Computing Machinery,
New York, NY, USA, 352–353. https://doi.org/10.1145/2909132.2926090
[19] Constantinos K Coursaris and Ming Liu. 2009. An analysis of social support
exchanges in online HIV/AIDS self-help groups. Computers in Human Behavior
25, 4 (2009), 911–918.
[20] Vagner Figueredo de Santana, Rodrigo Laiola Guimarães, and Andrea Britto
Mattos. 2016. Identifying Challenges and Opportunities in Computer-Based
Vocational Training for Low-Income Communities of People with Intellectual
Disabilities. In Proceedings of the 13th Web for All Conference (Montreal, Canada)
(W4A ’16). Association for Computing Machinery, New York, NY, USA, Article 2,
8 pages. https://doi.org/10.1145/2899475.2899480
[21] Denise E Elliott, Paula Bjelajac, Roger D Fallot, Laurie S Markof, and Beth Glover
Reed. 2005. Trauma-informed or trauma-denied: Principles and implementation
of trauma-informed services for women. Journal of community psychology 33, 4
(2005), 461–477.
[22] Hayley Evans, Udaya Lakshmi, Hue Watson, Azra Ismail, Andrew M. Sherrill,
Neha Kumar, and Rosa I. Arriaga. 2020. Understanding the Care Ecologies of
Veterans with PTSD. Association for Computing Machinery, New York, NY, USA,
1–15. https://doi-org.uri.idm.oclc.org/10.1145/3313831.3376170
[23] Anya S Evmenova and Michael M Behrmann. 2014. Enabling access and enhanc-
ing comprehension of video content for postsecondary students with intellectual
disability. Education and Training in Autism and Developmental Disabilities 49, 1
(2014), 45–59.
[24] Family 2021. PTSD Family Coach. Family. Retrieved August 20, 2021 from
https://www.ptsd.va.gov/appvid/mobile/familycoach_app.asp
[25] Charles R Figley. 2012. Encyclopedia of trauma: An interdisciplinary guide. Sage
Publications, Los Angeles, CA.
[26] Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering
cognitive behavior therapy to young adults with symptoms of depression and
Designing Post-Trauma Self-Regulation Apps for People with I/DD
anxiety using a fully automated conversational agent (Woebot): a randomized
controlled trial. JMIR mental health 4, 2 (2017), e7785.
[27] Eivind Flobak, Jo D. Wake, Joakim Vindenes, Smiti Kahlon, Tine Nordgreen, and
Frode Guribye. 2019. Participatory Design of VR Scenarios for Exposure Therapy.
In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
(Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New
York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300799
[28] Edna B Foa. 1997. Psychological processes related to recovery from a trauma and
an efective treatment for PTSD. Annals of the New York Academy of Sciences 821
(1997), 410–424.
[29] Stuart Iain Gray, Tom Metcalfe, Kirsten Cater, Peter Bennett, and Chris Bevan.
2020. The Sugargotchi: An Embodied Digital Pet to Raise Children’s Awareness
of Their Dental Health and Free Sugar Consumption. In Extended Abstracts of the
2020 Annual Symposium on Computer-Human Interaction in Play (Virtual Event,
Canada) (CHI PLAY ’20). Association for Computing Machinery, New York, NY,
USA, 242–247. https://doi.org/10.1145/3383668.3419874
[30] Hee Jeong Han, Sanjana Mendu, Beth K Jaworski, Jason E Owen, and Saeed
Abdullah. 2021. PTSDialogue: Designing a Conversational Agent to Support Individ-
uals with Post-Traumatic Stress Disorder. Association for Computing Machinery,
New York, NY, USA, 198–203. https://doi-org.uri.idm.oclc.org/10.1145/3460418.
3479332
[31] M Harris and R Fallot. 2001. Creating cultures of trauma-informed care (CCTIC):
A self-assessment and planning protocol. Jossey-Bass, New York, NY.
[32] HeadSpace 2022. HeadSpace. HeadSpace. Retrieved June 20, 2021 from https:
//www.headspace.com/
[33] Healthline 2021. 20 Best Meditation Apps of 2021. Healthline. Retrieved June
2, 2021 from https://www.healthline.com/health/mental-health/top-meditation-
iphone-android-apps
[34] Thomas Howard III, Krishna Venkatasubramanian, Jeanine L. M. Skorinko,
Pauline Bosma, John Mullaly, Brian Kelly, Deborah Lloyd, Mary Wishart, Emi-
ton Alves, Nicole Jutras, Mariah Freark, and Nancy A. Alterio. 2021. De-
signing an App to Help Individuals with Intellectual and Developmental Dis-
abilities to Recognize Abuse. In Proceedings of the 23rd International ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS’21). Associa-
tion for Computing Machinery, New York, NY, USA, Article 373, 13 pages.
https://doi.org/10.1145/3441852.3471217
[35] Yun Huang, Ying Tang, and Yang Wang. 2015. Emotion Map: A Location-
Based Mobile Social System for Improving Emotion Awareness and Regula-
tion. In Proceedings of the 18th ACM Conference on Computer Supported Co-
operative Work & Social Computing (Vancouver, BC, Canada) (CSCW ’15). As-
sociation for Computing Machinery, New York, NY, USA, 130–142. https:
//doi.org/10.1145/2675133.2675173
[36] Insight Timer 2021. Insight Timer. Insight Timer. Retrieved September 5, 2021
from https://insighttimer.com/
[37] Jen Ryan and Eve Ashwood and Shannan Puckeridge. 2010. The Development of
Anger Management Skills in Adults with Moderate Intellectual Disability. ASID.
Retrieved August 20, 2021 from https://www.asid.asn.au/Portals/0/Conferences/
45thBrisbane/Conference%20Papers/Ryan_et_al_THU_1555_Behaviour77.pdf
[38] Stephen Kaplan. 1995. The restorative benefts of nature: Toward an integrative
framework. Journal of environmental psychology 15, 3 (1995), 169–182.
[39] John M Keesler. 2014. A call for the integration of trauma-informed care among
intellectual and developmental disability organizations. Journal of Policy and
Practice in Intellectual Disabilities 11, 1 (2014), 34–42.
[40] M Kellen and Deepak Saxena. 2020. Calm my headspace: Motivations and barriers
for adoption and usage of meditation apps during times of crisis. In Proceedings
of The 20th International Conference on Electronic Business. Elsevier, Amsterdam,
Netherlands, 5–8.
[41] Rachel Kornfeld, Renwen Zhang, Jennifer Nicholas, Stephen M. Schueller, Scott A.
Cambo, David C. Mohr, and Madhu Reddy. 2020. "Energy is a Finite Resource":
Designing Technology to Support Individuals across Fluctuating Symptoms of De-
pression. Association for Computing Machinery, New York, NY, USA, 1–17.
https://doi-org.uri.idm.oclc.org/10.1145/3313831.3376309
[42] Francesca Laguardia, Venezia Michalsen, and Holly Rider-Milkovich. 2017. Trig-
ger warnings. Journal of Legal Education 66, 4 (2017), 882–903.
[43] Rodrigo Laiola Guimarães and Andrea Britto Mattos. 2015. Exploring the Use of
Massive Open Online Courses for Teaching Students with Intellectual Disability.
In Proceedings of the 17th International ACM SIGACCESS Conference on Computers
and Accessibility (Lisbon, Portugal) (ASSETS ’15). Association for Computing Ma-
chinery, New York, NY, USA, 343–344. https://doi.org/10.1145/2700648.2811370
[44] Rodrigo Laiola Guimarães, Andrea Britto Mattos, and Carlos Henrique Cardonha.
2016. Investigating Instructional Pacing Supports for Teaching Students with
Intellectual Disability. In Proceedings of the 2016 CHI Conference Extended Abstracts
on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA
’16). Association for Computing Machinery, New York, NY, USA, 2171–2177.
https://doi.org/10.1145/2851581.2892342
[45] Reeva Lederman, Greg Wadley, John Gleeson, Sarah Bendall, and Mario Álvarez
Jiménez. 2014. Moderated Online Social Therapy: Designing and Evaluating
Technology for Mental Health. ACM Trans. Comput.-Hum. Interact. 21, 1, Article
ASSETS ’22, October 23–26, 2022, Athens, Greece
5 (feb 2014), 26 pages. https://doi.org/10.1145/2513179
[46] Jill Levenson. 2017. Trauma-informed social work practice. Social Work 62, 2
(2017), 105–113.
[47] Deborah Lupton. 2014. Self-Tracking Cultures: Towards a Sociology of Personal
Informatics. In Proceedings of the 26th Australian Computer-Human Interaction
Conference on Designing Futures: The Future of Design (Sydney, New South Wales,
Australia) (OzCHI ’14). Association for Computing Machinery, New York, NY,
USA, 77–86. https://doi.org/10.1145/2686612.2686623
[48] Jennifer Mankof, Gillian R. Hayes, and Devva Kasnitz. 2010. Disability Studies as
a Source of Critical Inquiry for the Field of Assistive Technology. In Proceedings of
the 12th International ACM SIGACCESS Conference on Computers and Accessibility
(Orlando, Florida, USA) (ASSETS ’10). Association for Computing Machinery,
New York, NY, USA, 3–10. https://doi.org/10.1145/1878803.1878807
[49] MC 2021. Mindfulness Coach. MC. Retrieved August 20, 2021 from https:
//www.ptsd.va.gov/appvid/mobile/mindfulcoach_app.asp
[50] L Mevissen and A De Jongh. 2010. PTSD and its treatment in people with
intellectual disabilities: A review of the literature. Clinical psychology review 30,
3 (2010), 308–316.
[51] Liesbeth Mevissen, Robert Didden, and Ad de Jongh. 2016. Assessment and
Treatment of PTSD in People with Intellectual Disabilities. Springer International
Publishing, Cham. 1–15 pages. https://doi.org/10.1007/978-3-319-08613-2_95-2
[52] A Mitchell and Jennifer Clegg. 2005. Is post-traumatic stress disorder a helpful
concept for adults with intellectual disability? Journal of Intellectual Disability
Research 49, 7 (2005), 552–559.
[53] David C Mohr, Joyce Ho, Jenna Dufecy, Kelly G Baron, Kenneth A Lehman, Ling
Jin, and Douglas Reifer. 2010. Perceived barriers to psychological treatments
and their relationship to depression. Journal of clinical psychology 66, 4 (2010),
394–409.
[54] Lourdes M. Morales-Villaverde, Karina Caro, Taylor Gotfrid, and Sri Kurniawan.
2016. Online Learning System to Help People with Developmental Disabilities
Reinforce Basic Skills. In Proceedings of the 18th International ACM SIGACCESS
Conference on Computers and Accessibility (Reno, Nevada, USA) (ASSETS ’16).
Association for Computing Machinery, New York, NY, USA, 43–51. https://doi.
org/10.1145/2982142.2982174
[55] Neema Moraveji, Athman Adiseshan, and Takehiro Hagiwara. 2012. Breath-
Tray: Augmenting Respiration Self-Regulation without Cognitive Defcit. In CHI
’12 Extended Abstracts on Human Factors in Computing Systems (Austin, Texas,
USA) (CHI EA ’12). Association for Computing Machinery, New York, NY, USA,
2405–2410. https://doi.org/10.1145/2212776.2223810
[56] John T Morris, Michael L Jones, and W Mark Sweatman. 2016. Wireless Technology
Use by People with Disabilities: A National Survey. Technical Report. California
State University, Northridge.
[57] Nasim Motalebi and Saeed Abdullah. 2018. Conversational Agents to Provide
Couple Therapy for Patients with PTSD. In Proceedings of the 12th EAI Interna-
tional Conference on Pervasive Computing Technologies for Healthcare (New York,
NY, USA) (PervasiveHealth ’18). Association for Computing Machinery, New York,
NY, USA, 347–351. https://doi.org/10.1145/3240925.3240933
[58] MST 2021. Beyond MST. MST. Retrieved August 20, 2021 from https://www.ptsd.
va.gov/appvid/mobile/beyondMST.asp
[59] Andrea Nicki. 2001. The abused mind: Feminist theory, psychiatric disability,
and trauma. Hypatia 16, 4 (2001), 80–104.
[60] Kavous Salehzadeh Niksirat, Chaklam Silpasuwanchai, Peng Cheng, and Xiangshi
Ren. 2019. Attention Regulation Framework: Designing Self-Regulated Mind-
fulness Technologies. ACM Trans. Comput.-Hum. Interact. 26, 6, Article 39 (Nov.
2019), 44 pages. https://doi.org/10.1145/3359593
[61] Jason E Owen, Beth K Jaworski, Eric Kuhn, Kerry N Makin-Byrd, Kelly M Ramsey,
and Julia E Hofman. 2015. mHealth in the wild: using novel data to examine the
reach, use, and impact of PTSD coach. JMIR mental health 2, 1 (2015), e3935.
[62] SoHyun Park, Anja Thieme, Jeongyun Han, Sungwoo Lee, Wonjong Rhee, and
Bongwon Suh. 2021. “I Wrote as If I Were Telling a Story to Someone I Knew.”:
Designing Chatbot Interactions for Expressive Writing in Mental Health. Association
for Computing Machinery, New York, NY, USA, 926–941. https://doi-org.uri.
idm.oclc.org/10.1145/3461778.3462143
[63] Nintendo 2020. The Pocket Pikachu. Nintendo. Retrieved March 18, 2021 from
https://www.nintendo.co.jp/n09/pokepika/t
[64] Positive Psychology 2021. Top 14 Apps For Meditation and Mindfulness (+ Reviews).
Positive Psychology. Retrieved June 2, 2021 from https://positivepsychology.
com/mindfulness-apps/
[65] Therapists Aid 2021. Printable Emotion Faces. Therapists Aid. Retrieved August 20,
2021 from https://www.therapistaid.com/therapy-worksheet/printable-emotion-
faces
[66] ptsdcoach 2021. ptsdcoach. ptsdcoach. Retrieved Sept 4, 2021 from https:
//mobile.va.gov/app/ptsd-coach
[67] Resilience 2021. Provider Resilience. Resilience. Retrieved August 20, 2021
from https://www.health.mil/Military-Health-Topics/MHS-Toolkits/Provider-
Resilience-Toolkit
[68] safehelpline 2021. safehelp. safehelpline. Retrieved August 20, 2021 from
https://www.safehelpline.org/app
ASSETS ’22, October 23–26, 2022, Athens, Greece
[69] Kavous Salehzadeh Niksirat, Chaklam Silpasuwanchai, Mahmoud Mohamed
Hussien Ahmed, Peng Cheng, and Xiangshi Ren. 2017. A Framework for Interac-
tive Mindfulness Meditation Using Attention-Regulation Process. In Proceedings
of the 2017 CHI Conference on Human Factors in Computing Systems (Denver,
Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY,
USA, 2672–2684. https://doi.org/10.1145/3025453.3025914
[70] Maria Saridaki and Michalis Meimaris. 2018. Digital Storytelling for the Em-
powerment of People with Intellectual Disabilities. In Proceedings of the 8th
International Conference on Software Development and Technologies for Enhanc-
ing Accessibility and Fighting Info-Exclusion (Thessaloniki, Greece) (DSAI 2018).
Association for Computing Machinery, New York, NY, USA, 161–164. https:
//doi.org/10.1145/3218585.3218664
[71] Corina Sas and Rohit Chopra. 2015. MeditAid: A Wearable Adaptive
Neurofeedback-Based System for Training Mindfulness State. Personal Ubiquitous
Comput. 19, 7 (Oct. 2015), 1169–1182. https://doi.org/10.1007/s00779-015-0870-z
[72] Susan Scheckel. 2017. Home-Sickness, Nostalgia, and Therapeutic Narrative in
Poe’s" The Fall of the House of Usher". Poe Studies 50, 1 (2017), 12–25.
[73] Jessica Schroeder, Chelsey Wilkes, Kael Rowan, Arturo Toledo, Ann Paradiso,
Mary Czerwinski, Gloria Mark, and Marsha M. Linehan. 2018. Pocket Skills:
A Conversational Mobile Web App To Support Dialectical Behavioral Therapy.
Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.
org/10.1145/3173574.3173972
[74] Nokuthula Shabalala and Aimee Jasson. 2011. PTSD symptoms in intellectually
disabled victims of sexual assault. South African Journal of Psychology 41, 4 (2011),
424–436.
[75] SimplyBeing 2021. The SimpleBeing App. SimplyBeing. Retrieved August 20,
2021 from https://www.meditationoasis.com/simply-being-app
[76] Katarzyna Stawarz, Chris Preist, Deborah Tallon, Laura Thomas, Katrina Turner,
Nicola Wiles, David Kessler, Roz Shafran, and David Coyle. 2020. Integrating
the Digital and the Traditional to Deliver Therapy for Depression: Lessons from a
Pragmatic Study. Association for Computing Machinery, New York, NY, USA,
1–14. https://doi-org.uri.idm.oclc.org/10.1145/3313831.3376510
[77] Erin Sullivan. 2016. Beyond melancholy: Sadness and selfhood in Renaissance
England. Oxford University Press, Oxford, UK.
[78] Colm Sweeney, Courtney Potts, Edel Ennis, Raymond Bond, Maurice D. Mulvenna,
Siobhan O’neill, Martin Malcolm, Lauri Kuosmanen, Catrine Kostenius, Alex
Vakaloudis, Gavin Mcconvey, Robin Turkington, David Hanna, Heidi Nieminen,
Anna-Kaisa Vartiainen, Alison Robertson, and Michael F. Mctear. 2021. Can
Chatbots Help Support a Person’s Mental Health? Perceptions and Views from
Mental Healthcare Professionals and Experts. ACM Trans. Comput. Healthcare 2,
3, Article 25 (jul 2021), 15 pages. https://doi.org/10.1145/3453175
[79] Juan C. Torrado, Ida Wold, Letizia Jaccheri, Susanna Pelagatti, Stefano Chessa,
Javier Gomez, Gunnar Hartvigsen, and Henriette Michalsen. 2020. Developing
Software for Motivating Individuals with Intellectual Disabilities to Do Outdoor
Physical Activity. In Proceedings of the ACM/IEEE 42nd International Conference
on Software Engineering: Software Engineering in Society (Seoul, South Korea)
(ICSE-SEIS ’20). Association for Computing Machinery, New York, NY, USA, 81–84.
https://doi.org/10.1145/3377815.3381376
[80] Krishna Venkatasubramanian, Jeanine L. M. Skorinko, Mariam Kobeissi, Brittany
Lewis, Nicole Jutras, Pauline Bosma, John Mullaly, Brian Kelly, Deborah Lloyd,
Mariah Freark, and Nancy A. Alterio. 2021. Exploring A Reporting Tool to
Empower Individuals with Intellectual and Developmental Disabilities to Self-
Report Abuse. In Proceedings of the 2021 CHI Conference on Human Factors in
Computing Systems. Association for Computing Machinery, New York, NY, USA,
Article 373, 13 pages. https://doi.org/10.1145/3411764.3445150
[81] Vet 2021. Vet Change. Vet. Retrieved August 20, 2021 from https://www.ptsd.va.
gov/appvid/mobile/VetChange_app.asp
[82] Wakingup 2021. The WakingUp App. Wakingup. Retrieved August 20, 2021 from
https://wakingup.com/
Venkatasubramanian and Ranalli
[83] Erin Louise Whittle, Karen R Fisher, Simone Reppermund, Rhoshel Lenroot, and
Julian Trollor. 2018. Barriers and enablers to accessing mental health services for
people with intellectual disability: a scoping review. Journal of Mental Health
Research in Intellectual Disabilities 11, 1 (2018), 69–102.
[84] Erin Louise Whittle, Karen Raewyn Fisher, Simone Reppermund, and Julian
Trollor. 2019. Access to mental health services: The experiences of people with
intellectual disabilities. Journal of Applied Research in Intellectual Disabilities 32,
2 (2019), 368–379.
[85] Women’s Health Weekly 2021. The 12 Best Meditation Apps For 2020, According
To Experts. Women’s Health Weekly. Retrieved June 2, 2021 from https://www.
womenshealthmag.com/health/g25178771/best-meditation-apps/
[86] Sarah Wigham and Eric Emerson. 2015. Trauma and life events in adults with
intellectual disability. Current Developmental Disorders Reports 2, 2 (2015), 93–99.
[87] Jacob O Wobbrock, Shaun K Kane, Krzysztof Z Gajos, Susumu Harada, and Jon
Froehlich. 2011. Ability-based design: Concept, principles and examples. ACM
Transactions on Accessible Computing (TACCESS) 3, 3 (2011), 1–27.
[88] Renwen Zhang, Kathryn E. Ringland, Melina Paan, David C. Mohr, and Madhu
Reddy. 2021. Designing for Emotional Well-Being: Integrating Persuasion and
Customization into Mental Health Technologies. In Proceedings of the 2021 CHI
Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21).
Association for Computing Machinery, New York, NY, USA, Article 542, 13 pages.
https://doi.org/10.1145/3411764.3445771
APPENDIX
The following are the 28 commercial apps that we looked
at in 2021 as part of this work. Most of these apps are
paid (21) or freemium (5). The names of the apps and the
URLs where we obtained them are as follows: Calm (https:
//www.calm.com), Breethe (https://breethe.com), Headspace
(https://www.headspace.com), My Life (https://my.life), Bud-
dhify (https://buddhify.com), Inspace (https://inscape.life),
Breathe+ (https://dynamicappdesign.com/#Breathe), Oak (https:
//www.oakmeditation.com), Whil (https://www.whil.com), Sim-
ple Habit (https://www.simplehabit.com), Petit Bambou (https:
//www.petitbambou.com/en/), Waking up (https://wakingup.com),
Prana Breath (https://pranabreath.info), The Mindfulness App
(https://themindfulnessapp.com), Sattva (https://www.sattva.life),
Insight Timer (https://insighttimer.com), Meditation Studio
(https://meditationstudioapp.com), Let’s Meditate (https://play.
google.com/store/apps/details?id=com.meditation.elevenminute),
Happy Not Perfect (https://happynotperfect.com/), Omvana
(https://www.omvana.com/), Welzen (https://welzen.app/), Relax-
ing Melodies (https://www.relaxmelodies.com/), 10% Happier
(https://www.tenpercent.com/), Simply Being (https://apps.apple.
com/us/app/simply-being-guided-meditation/id347418999), Aura
(https://www.aurahealth.io), Unplug (https://www.unplug.com),
Enso (https://www.ensomeditationtimer.com), and Meditation
Nest (https://appadvice.com/app/meditation-nest/1460053458).
"I Should Feel Like I’m In Control": Understanding Expectations,
Concerns, and Motivations for the Use of Autonomous
Navigation on Wheelchairs
JiWoong Jang Yunzhi Li Patrick Carrington
jiwoongj@cs.cmu.edu yunzhil@cs.cmu.edu pcarrington@cmu.edu
Human-Computer Interaction Human-Computer Interaction Human-Computer Interaction
Institute, Carnegie Mellon University Institute, Carnegie Mellon University Institute, Carnegie Mellon University
USA USA USA

Figure 1: The relationship between user-centric Factors (and their respective qualities), Attitudes, and Intent to Use Au-
tonomous Navigation on wheelchairs.
ABSTRACT
Autonomous navigation on wheelchairs promises to be a signifcant
frontier in the evolution of the power wheelchair as an assistive en-
abling device, and is increasingly explored among researchers for its
potential to unlock more accessible navigation for wheelchair users.
While developments on path-planning methods for wheelchairs
is ongoing, there is a relative paucity of research on autonomous
wheelchair navigation experiences which accommodate potential
users’ needs. In this work, we present preliminary design consider-
ations for the user experience for autonomous wheelchair naviga-
tion derived from a semi-structured interview with ten (10) current
wheelchair users about their willingness to use and applicability of
an autonomous navigation function. From this, nine (9) expressed
a willingness to use autonomous navigation in the near future in
a range of contexts, while expressing attitudes like expectations,
concerns, and motivations for intent to use. To better understand
the impetus for such attitudes, we conducted thematic analysis
to reveal three high-order factors and associated qualities which
together serve as a framework to help understand participants’
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550380
intent. Finally, we highlight three critical areas of focus to high-
light opportunities and challenges for developers of a user-centered
autonomous navigation experience for wheelchairs.
CCS CONCEPTS
• Human-centered computing → Empirical studies in accessibil-
ity.
KEYWORDS
wheelchairs, assistive technology, social navigation, interaction
design
ACM Reference Format:
JiWoong Jang, Yunzhi Li, and Patrick Carrington. 2022. "I Should Feel Like
I’m In Control": Understanding Expectations, Concerns, and Motivations for
the Use of Autonomous Navigation on Wheelchairs. In ASSETS ’22: The 24th
International ACM SIGACCESS Conference on Computers and Accessibility,
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 5 pages.
https://doi.org/10.1145/3517428.3550380
1 INTRODUCTION AND RELATED WORK
Autonomous navigation on wheelchairs arguably furthers the promise
of powered wheelchairs. Powered wheelchairs have enabled in-
creased independence from both the necessity of a companion and
from the physical toll of manual movement. Meanwhile, power
wheelchairs generate their own challenges due to larger wheelchair
sizes and inaccessibility concerns, and a dependence on battery
power which has led to concerns of "range anxiety" from users [16].
ASSETS ’22, October 23–26, 2022, Athens, Greece
Autonomous wheelchairs have the potential of addressing chal-
lenges arising from power wheelchair use by unburdening users
from managing the minutiae of navigation, and allowing the user
to automate difcult maneuvering, such as turning tight corners
[12].
Exploration of autonomous navigation function in wheelchairs
is timely due to advancements in LIDAR-based technology[7] and
reinforcement-learning based navigational techniques in contexts
like household robots [9][10]. It is accompanied by signifcant de-
velopments in autonomous vehicle navigation, which are poten-
tially transferable to the wheelchair navigation context [7] [15].
Self-driving wheelchairs are already in development by several
commercial wheelchair makers and technology companies, includ-
ing WHILL [8], Hitachi [11], and Panasonic [3], as well as by re-
search eforts at Massachusetts Institute of Technology [14] and
the United Kingdom’s National Institute of Health [1]. With con-
tinued development and proliferation of "smart" features in today’s
power wheelchairs [2] [16] [17], autonomous navigation serves as
a convergence point for both trends.
Prior research sought to address technical challenges of devel-
oping autonomous function for wheelchairs [1][7][15], and some
dealt with human-robotics interaction-related problems of social
navigation [9] and shared control [4][5]. Despite analogous explo-
rations in the smart wheelchair context [13][17], there has yet to be
an exploration into an autonomous wheelchair experience which
caters to users’ preferences and expectations. In this paper, we
motivate such eforts by demonstrating what contexts and factors
motivate adoption of autonomous wheelchair navigation, as well
as provide guidance for future work for developing user-centered
autonomous wheelchair navigation experiences.
2 STUDY METHODOLOGY FOR
UNDERSTANDING ATTITUDES TOWARD
AUTONOMOUS NAVIGATION
To better understand motivating use cases, and attitudes like con-
cerns and expectations for autonomous navigation functionality for
wheelchairs, we conducted a semi-structured interview study with
ten (n = 10) current wheelchair users over a period of three months.
Participants were recruited from public messages on Reddit, par-
ticularly in specifc subreddits (e.g. r/wheelchairs, r/cerebralpalsy,
and r/spinalcordinjuries). We required that participants be over 18
years of age, and to be able to demonstrate prior experience with
wheelchairs of any kind for a period of at least 3 months to guaran-
tee some minimum familiarity with operating a wheelchair. The
study procedure was done in agreement with an Institutional Re-
view Board (IRB). Of our participants, 3 were manual wheelchair
user, 4 had not had prior exposure to autonomous navigation in any
context, and overall had an average of 9.4 years of experience with
signifcant variance (σ = 10.02). We present specifc demographics
of our participants in Table 1 of the Appendix.
2.1 Interview Procedure
Each participant engaged in an hour-long, recorded, semi-structured
interview in video-conferencing conditions, in which they were
compensated at the conclusion of the study period. The interviewer
Jang, et al.
encouraged participants to communicate and envision any expec-
tations, concerns, or potential uses about a future autonomous
navigation function in a think-aloud session. To ground the dis-
cussion, the interviewer devised a scenario where the participant
was to consider whether or not to purchase a wheelchair in the
near future with autonomous navigation built in. Finally, the in-
terviewer asked for participants’ opinions about issues regarding
social navigation, shared/cooperative control, and expectation for
the autonomous system to exhibit identical behavior to participants’
current driving habits.
2.2 Interview Analysis
Two researchers conducted thematic analysis [6] over the text
transcript generated from participant interviews. Each researcher
independently generated a codebook from three (3) transcripts,
whereafter the codebooks were merged after agreement was found
between both researchers. The remaining transcripts were then
evenly divided amongst the researchers to independently code.
Each code and its associated part of the interview transcript were
then afnity diagrammed and organized until larger patterns and
relationships became apparent.
3 FINDINGS AND EMERGENT FACTORS
Overall, we found that 9 out of 10 participants expressed some in-
tention to use a wheelchair with autonomous navigation function -
8 (P1-P4, P6, P7, P9, P10) reported willingness to use in an everyday
fashion, while P8 expressed conditional willingness to use the func-
tion in limited circumstances where involuntary hand tremors from
her cerebral palsy would otherwise inhibit her ability to operate her
wheelchair. Despite almost all of our participants communicating a
willingness to use, each expressed difering attitudes like expecta-
tions, concerns, and motivations, about the overall user experience
of an autonomous navigation function. This motivated the thematic
analysis, in order to better understand the basis of such attitudes
and to generalize our participants’ thoughts.
As part of analysis, 46 codes were used to label the text tran-
scripts. Overall, the codes were best organized by grouping into
three diferent themes: intrinsic, extrinsic, and chair-centered
factors.
In general, each high-level factor is associated with respective
qualities. We further observed that these qualities infuence at-
titudes like expectation, concern, and motivation to use, where
attitudes collectively shaped the overall willingness of a user to
adopt autonomous navigation functionality. Here, we defne the
three primary factors and provide examples of associated quali-
ties to illustrate their connection to attitudes and ultimately to
participants’ intent to use.
3.1 Intrinsic Factors
Intrinsic factors refected innate, personal characteristics of poten-
tial users of the autonomous wheelchair. From our analysis from
our interviews, we highlight particpants’ prior awareness of
automation, extent of disability, and risk appetite as notable
intrinsic qualities.
• Prior awareness of automation, opinions of AI-based fea-
tures, and openness to novel technology writ large had direct
"I Should Feel Like I’m In Control":
Understanding Expectations, Concerns, and Motivations for the Use of Autonomous Navigation on Wheelchairs
links to attitudes and intent to use. For example, P2 brought
up that he was inspired by Tesla’s autonomous vehicle navi-
gation and expressed that he expected the riding experience
of the autonomous wheelchair to be as "perfect" as Tesla’s
Autopilot equipped cars. Additionally, he expressed hesita-
tion to adopt if the wheelchair could not provide a similar
quality of experience.
• Disability provided a unique point of view which afected
attitudes towards autonomous navigation. For example, P3
relayed that due to his difculties with spatial memory and
cognition, this provided him motivation for adopting au-
tonomous navigation in contexts where he would need to
return to a previously visited location.
• Risk appetite also appeared to afect attitudes, with the
propensity to engage with the maximum speed allowed by
the wheelchair, as one example. While many participants
reported adjusting their risk tolerance depending on extrin-
sic factors, several (P4, P7, P9) reported engaging with the
maximum speed setting frequently while several others (P1,
P2, P3, P8) rarely needed to do so, showcasing a range of
diferent innate preferences to actions which could carry risk.
This was associated with varying levels of concern at the pos-
sibility that the chair would go faster than what participants
would attempt in their current wheelchairs.
3.2 Extrinsic Factors
Extrinsic factors relates to elements and contexts in the environ-
ment around the wheelchair. We found physical location and
social interaction with other people to be important qualities
throughout interviews.
• Physical location encapsulates the characteristics of the
built environment which the wheelchair is occupying - this
includes potential obstacles which impact path-planning
and the presence of safe wheelchair-accessible routes. For
instance, at least three participants (P1, P2, P10) remarked
that the surface on which a wheelchair drives on has a large
efect on the riding experience, and that circumstances where
road conditions are hazardous, such as with large potholes,
can provide severe navigational and safety challenges. These
participants expressed a mix of concern and expectation
that an autonomous system would be capable of successfully
recognizing and adjust to such scenarios.
• Social interaction represents the need to account for those
around the wheelchair who may exhibit a range of diferent
navigational behavior, including a personal care assistant
(PCA) following closely behind or besides the wheelchair,
or unrelated passerby or groups going in a diferent way
from the wheelchair user. Depending on the social context,
participants expressed their expectation that an autonomous
navigation function would be capable of distinguishing these
roles, and navigating environments like a shopping mall with
a lot of people.
3.3 Chair-Centered Factors
Chair-centered factors include participants’ difculties stemming
from operating their current wheelchairs, desire for sense a
ASSETS ’22, October 23–26, 2022, Athens, Greece
sense of control, and system transparency needs. This directly
led to how participants perceived they could beneft from develop-
ments in autonomous navigation functionality in wheelchairs.
• Participants’ difculties with their current wheelchairs,
such as P9’s difculty with maneuvers requiring fne-motor
control, or physical efort needed to move for manual wheelchair
users (P1, P3, P5), directly led to motivation to potentially
resolve these challenges with autonomous navigation, and
then to willingness to use such a function.
• Participants’ desire for sense a sense of control, and the
implementation of the control scheme were a source of con-
cern for many participants, with many (8) stating that they
would strongly prefer to have traditional joystick controls
on autonomous chairs. This concern also manifested as a
desire for the ability to immediately override the planned
path or next action of the wheelchair. Eight (8) out of ten
participants showed strong reservations against shared or
cooperative control schemes in which they did not have over-
riding capability, with P4 describing a lack of such a feature
"a huge problem", and P7 calling it a "show stopper."
• System transparency-related qualities dealt with partic-
ipants’ expectations to view the system’s planned path, as
well as the system’s understanding of the world, such as with
a planner displaying the intended path and perceived obsta-
cles overlayed on a map. Participants relayed that knowing
both elements were crucial in scenarios with many elements
of concern as to determine whether to course-correct. Vi-
sual/touch, and sound/voice-based interfaces were cited most
often (6 and 3 participants, respectively), including both in
conjunction (2), as the preferred method to give input and
receive communication to and from the system. Several par-
ticipants (P2, P3, P6, P7), however, admitted that they would
likely not consult the system often, as they would rather pay
attention to their smartphones or surroundings. In this case,
participants expected a reliable alert system which would
provide a haptic or sonic feedback as to scenarios where
the wheelchair user would need to intervene. P2 even went
so far as to suggest that the reliability of the alert system,
and that it consistently provided sufcient time for response,
mattered more than the reliability of the planner itself.
4 DESIGN IMPLICATIONS AND FUTURE
WORK
We believe fndings from the previous section’s thematic analysis
highlight three immediate areas of focus to be explored in future
research, which we explain briefy.
(1) Create Adaptable Intercommunication Between the Sys-
tem and User
The chair-centered quality of desire for a sense of control
places emphasis on the intercommunicative abilities of the
user interface and input implementation. The extent partici-
pants felt they needed to know about the system state varied
on intrinsic qualities like disability, as well as situational
extrinsic qualities like the complexity of the environment.
These implicate research questions designing system UI and
input modalities which are adaptable to these variables.
ASSETS ’22, October 23–26, 2022, Athens, Greece
(2) Incorporate Diferences in Ability, Environment, and
Social Preferences in Navigation
Likewise, variations arising from intrinsic, chair-centered,
and extrinsic factors give rise to a need for developing naviga-
tion systems which are responsive to these potential changes.
For example, participants expressed expectation that nav-
igation in or around moving groups of people be handled
smoothly and in accordance with their current driving be-
havior.
(3) Scafold and Ease In to Build Trust
Given the variation of the awareness of autonomous naviga-
tion, we encourage inquiry into opportunities for potential
users to learn about the capabilities and limitations of au-
tonomous system prior and after deployment to engender
grounded trust. To help users form mental models for use
of autonomous navigation, we suggest exploring discretiza-
tion of functions, such as a separate mode for autonomously
aligning into a tight hallway. Finally, given that driving speed
was expressed by participants as an important intermedi-
ating variable to build trust, we advocate for exploring the
possibility of acclimating users by navigating with relatively
slower speeds until the user had developed confdence in the
system’s ability.
5 CONCLUSION
This work aims to provide preliminary understanding of wheelchair
users’ motivations and potential uses for autonomous navigation
functions. It delineates intrinsic, extrinsic, and chair-centered fac-
tors as a guiding framework to understand the source of observed
attitudes. It highlights notable qualities for each factor, and connects
them to attitudes toward autonomous navigation like expectation,
concern, and motivation. By outlining areas of focus, it seeks to
guide future eforts of designers, engineers, and researchers on
developing nuances in autonomous navigation on wheelchairs. Fi-
nally, as preliminary work, we acknowledge limitations to the study
methodology, including the limited sample size interviewed, which
preclude any claims of statistical signifcance.
ACKNOWLEDGMENTS
We gratefully acknowledge Franklin Mingzhe Li’s feedback while
preparing this work.
REFERENCES
[1] André R. Baltazar, Marcelo R. Petry, Manuel F. Silva, and António Paulo Moreira.
2021. Autonomous wheelchair for patient’s transportation on healthcare institu-
tions. SN Applied Sciences 3, 3 (Feb. 2021). https://doi.org/10.1007/s42452-021-
Jang, et al.
04304-1
[2] Patrick Carrington, Amy Hurst, and Shaun K. Kane. 2014. Wearables and
Chairables: Inclusive Design of Mobile Input and Output Techniques for Power
Wheelchair Users. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Com-
puting Machinery, New York, NY, USA, 3103–3112. https://doi.org/10.1145/
2556288.2557237
[3] Panasonic Holdings Corporation. 2020. Panasonic begins trial of Mobility Service
using autonomous tracking robots at Takanawa Gateway station. Panasonic
Newsroom Global (2020). https://news.panasonic.com/global/topics/2020/79848.
html
[4] Samuel Echefu. 2016. Towards Natural Human Control and Navigation of Au-
tonomous Wheelchairs.
[5] Chinemelu Ezeh, Pete Trautman, Catherine Holloway, and Tom Carlson. 2017.
Comparing Shared Control Approaches for Alternative Interfaces: A Wheelchair
Simulator Experiment. In 2017 IEEE International Conference on Systems, Man,
and Cybernetics (SMC) (Banf, AB, Canada). IEEE Press, 93–98. https://doi.org/
10.1109/SMC.2017.8122584
[6] Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating rigor using
thematic analysis: A hybrid approach of inductive and deductive coding and
theme development. International journal of qualitative methods 5, 1 (2006),
80–92.
[7] Harkishan Grewal, Aaron Matthews, Richard Tea, and Kiran George. 2017. LIDAR-
Based Autonomous Wheelchair. In 2017 IEEE Sensors Applications Symposium
(SAS) (Glassboro, NJ, USA). IEEE Press, 1–6. https://doi.org/10.1109/SAS.2017.
7894082
[8] WHILL Inc. 2019. WHILL Expands Airport Trials of Self-Driving
Personal Mobility Devices to North America. WHILL News (2019).
https://whill.inc/us/whill-expands-airport-trials-of-self-driving-personal-
mobility-devices-to-north-america/
[9] Beomjoon Kim and Joelle Pineau. 2016. Socially adaptive path planning in
human environments using inverse reinforcement learning. International Journal
of Social Robotics 8, 1 (2016), 51–66.
[10] Lucia Liu, Daniel Dugas, Gianluca Cesari, Roland Siegwart, and Renaud Dubé.
2020. Robot navigation in crowded environments using deep reinforcement
learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and
Systems (IROS). IEEE, 5671–5677.
[11] Hitachi Ltd. 2017. Autonomous mobility: Focus on the problems of
people that can be solved by autonomous mobility. Hitachi Research
(2017). https://www.hitachi.com/rd/research/design/vision_design/future/
autonomous_mobility/chair/index.html
[12] Corey Montella, Timothy Perkins, John R. Spletzer, and Michael Sands. 2012. To
the Bookstore! Autonomous Wheelchair Navigation in an Urban Environment.
In FSR.
[13] Yoichi Morales, Atsushi Watanabe, Florent Ferreri, Jani Even, Tetsushi Ikeda,
Kazuhiro Shinozawa, Takahiro Miyashita, and Norihiro Hagita. 2015. Including
human factors for planning comfortable paths. In 2015 IEEE International Confer-
ence on Robotics and Automation (ICRA). IEEE. https://doi.org/10.1109/icra.2015.
7140063
[14] Massachusetts Institute of Technology News. 2017. Featured video: A self-driving
wheelchair. MIT News (2017). https://www.csail.mit.edu/node/5962
[15] Hye-Yeon Ryu, Je-Seong Kwon, Jeong-Hak Lim, A-Hyeon Kim, Su-Jin Baek,
and Jong-Wook Kim. 2022. Development of an Autonomous Driving Smart
Wheelchair for the Physically Weak. Applied Sciences 12, 1 (2022). https://doi.
org/10.3390/app12010377
[16] Richard C Simpson. 2005. Smart wheelchairs: A literature review. Journal of
rehabilitation research and development 42, 4 (2005), 423.
[17] Bingqing Zhang, Giulia Barbareschi, Roxana Ramirez Herrera, Tom Carlson, and
Catherine Holloway. 2022. Understanding Interactions for Smart Wheelchair
Navigation in Crowds. In Proceedings of the 2022 CHI Conference on Human
Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association
for Computing Machinery, New York, NY, USA, Article 194, 16 pages. https:
//doi.org/10.1145/3491102.3502085
"I Should Feel Like I’m In Control":
Understanding Expectations, Concerns, and Motivations for the Use of Autonomous Navigation on Wheelchairs ASSETS ’22, October 23–26, 2022, Athens, Greece

A APPENDIX: PARTICIPANT
DEMOGRAPHICS TABLE
ID Years of Experience Type of Wheelchair Prior Exposure to Autonomous Navigation
P1 1 manual Yes
P2 0.5 power Yes
P3 2.5 manual No
P4 18 power Yes
P5 8 manual Yes
P6 1 power No
P7 1 power No
P8 29 power Yes
P9 18 power No
P10 15 power Yes
Table 1: Self-reported Participant Demographics. Prior expo-
sure relates to participants’ awareness of autonomous navi-
gation in any context prior to the interview.
 “What Makes Sonification User-Friendly?” Exploring Usability
and User-Friendliness of Sonified Responses
Ather Sharif∗ Olivia H. Wang∗
asharif@cs.washington.edu wang4@cs.washington.edu
Paul G. Allen School of Computer Paul G. Allen School of Computer
Science & Engineering | DUB Group, Science & Engineering,
University of Washington University of Washington
Seattle, Washington, USA Seattle, Washington, USA
ABSTRACT
Sonifcation is a commonly used technique to make online data
visualizations accessible to screen-reader users through auditory
means. While current sonifcation solutions provide plausible utility
(usefulness) to screen-reader users in exploring data visualizations,
they are limited in exploring the quality (usability) of the soni-
fed responses. In this preliminary exploration, we investigated the
usability and user-friendliness of data visualization sonifcation
for screen-reader users. Specifcally, we evaluated the Pleasantness,
Clarity, Confdence, and Overall Score of discrete and continuous
sonifed responses generated using various oscillator waveforms
and synthesizers through user studies with 10 screen-reader users.
Additionally, we examined these factors using both simple and com-
plex trends. Our results show that screen-reader users preferred
distinct non-continuous responses generated using oscillators with
square waveforms. We utilized our fndings to extend the function-
ality of Sonifier—an open-source JavaScript library that enables
developers to sonify online data visualizations. Our follow-up inter-
views with screen-reader users identifed the need to personalize
the sonifed responses per their individualized preferences.
CCS CONCEPTS
• Human-centered computing → Empirical studies in visu-
alization; Accessibility systems and tools; Empirical studies
in accessibility.
KEYWORDS
sonifcation, audio graphs, waveforms, visualizations, screen-reader
users
ACM Reference Format:
Ather Sharif, Olivia H. Wang, and Alida T. Muongchan. 2022. “What Makes
Sonifcation User-Friendly?” Exploring Usability and User-Friendliness of
Sonifed Responses. In The 24th International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece.
ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3517428.3550360
∗ These authors contributed equally to this work.
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550360
Alida T. Muongchan
alidatm@uw.edu
Human Centered Design and
Engineering,
University of Washington
Seattle, Washington, USA
1 INTRODUCTION
With the increasing data representation through visualizations
comes the critical need to make these visualizations accessible to
people who may not be able to extract information using visual
means (e.g., screen-reader users) [6, 16, 18, 21]. According to recent
fndings from Sharif et al., due to the inaccessibility of visualiza-
tions, screen-reader users extract information 62% less accurately
and spend 211% more interaction time with online data visualiza-
tions compared to non-screen-reader users [21]. Researchers and
developers have implemented several approaches and strategies
to make online data visualizations accessible [8, 10, 15, 22, 24, 25].
Among these techniques is sonifcation, often referred to as “au-
dio graphs,” a widely-used approach in conveying data through
auditory channels to screen-reader users.
Several prior works have utilized sonifcation to improve the ac-
cessibility of online data visualizations [2, 3, 7, 13, 14, 19, 24, 26, 29].
However, current solutions are focused on the utility (usefulness)
of sonifcation to screen-reader users and provide limited insights
into the quality (usability and user-friendliness) of the sonifed re-
sponses. Therefore, in this work, we sought to examine and improve
the usability and user-friendliness of sonifed responses generated
from online data visualizations created using JavaScript libraries
(e.g., D3 [4]). Prior research has explored the “pleasantness” of soni-
fed responses [1, 9, 20]. However, the most relevant research to
our work is the recent exploration by Wang et al. [27], in which
they examined the impact of various auditory channels (e.g., pitch,
volume) on users’ perception of data and visualization. We build on
their work by investigating the efects of diferent oscillator wave-
forms and synthesizers on the pleasantness and users’ confdence
in interpreting simple and complex sonifed responses.
To perform our investigation, we developed several sonifcation
prototypes using the Tone.js library [17], incorporating diferent
confgurations for Sound Types (oscillator waveforms and synthe-
sizers), Continuity Levels (interval for sounds between data points),
and Trend Types (simple and complex). We fnalized six prototypes
through Wizard-of-Oz [5, 12] and pilot studies with users, employ-
ing a user-centered iterative design process. We evaluated these
prototypes by collecting subjective ratings for Pleasantness, Clarity,
Confdence, and Overall Score through user studies with 10 screen-
reader users. We utilized our fndings to extend the functionalities
of Sonifier—a recently-introduced open-source JavaScript library
that generates sonifed responses for two-dimensional single-series
data [23, 24]. Additionally, we conducted follow-up interviews with
our participants to solicit feedback on improving their experiences
with sonifed responses.
ASSETS ’22, October 23–26, 2022, Athens, Greece Ather Sharif, Olivia H. Wang, and Alida T. Muongchan

Figure 1: Average scores per Continuity Level and Trend Type for each Sound Type for (a) Pleasantness, (b) Clarity, (c) Conf-
dence, and (d) Overall Scores. Baseline was a sawtooth waveform OmniOscillator, Prototype 1 was a square waveform OmniOscil-
lator, and Prototype 2 was a square waveform MonoSynthesizer. Subjective ratings were collected using a Likert scale ranging
from 1 (worst) to 7 (best).

In this work, we contribute the empirical fndings from our
preliminary explorations. Specifcally, we provide the quantitative
and qualitative results from our user studies with 10 screen-reader
users. Additionally, we provide enhancements to the open-source
Sonifer library and identify avenues for future work.
2 USER STUDY
To assess the usability and user-friendliness of sonifed responses,
we conducted a user study with 10 screen-reader users, subse-
quently interviewing them to gather insights on further improving
the sonifed responses.
2.1 Prototypes
We created several sonifcation prototypes employing diferent
combinations for oscillator waveforms, synthesizers, and partial
counts (number of harmonics to generate the waveform). We de-
veloped these prototypes using the Tone.js JavaScript library
[17]—a widely-used framework to generate sounds in the browser.
Then, we eliminated the prototypes that users found undesirable
through Wizard-of-Oz [5, 12] and pilot studies. After elimination,
our sound types comprised an oscillator (termed “OmniOscilla-
tor” by Tone.js) and a synthesizer (termed “MonoSynthesizer”
by Tone.js), with both using the square waveform and a partial
count of 8. We created a continuous (minimal time interval be-
tween sounds for each data point) and discrete (clear time interval
between sounds for each data point) prototype for each sound type.
We used the default settings from the Sonifier library as our base-
line measure. Since the baseline only supports discrete responses,
we developed our ffth prototype that generated a continuous re-
sponse for the baseline to account for balanced conditions. Our
fnal set contained 12 prototypes (including the baseline), six for
each Trend Type (simple and complex).
2.2 Participants & Procedure
Our participants were 10 screen-reader users (Appendix A, Table
1). Four identifed as women, fve as men, and one as non-binary.
Their average age was 48.4 (SD=14.4) years. We compensated our
participants with a $10 Amazon gift card for 30 minutes of their
time. We supervised our user studies online using Zoom.
For each participant, we played six sonifed responses (fve pro-
totypes + baseline) for simple trends, randomizing the order to
account for learning efect. At the end of each sonifcation, we
collected their subjective ratings for Pleasantness (timbre), Clarity
(assessment of the sound to identify the trend clearly), Confdence
(user’s confdence in understanding the overall trend), and Overall
Score (subjective assessment of the sound overall, including any
“What Makes Sonification User-Friendly?”
factors not mentioned above). We used a Likert scale for subjec-
tive ratings, ranging from 1 (worst) to 7 (best). Then, we asked
follow-up questions from our users to gather insights on the areas
of improvement. We followed the same steps for complex trends.
Our study sessions, on average, took approximately 30 minutes
from start to fnish.
2.3 Design & Analysis
The experiment was a 3×2×2 within-subjects design with the fol-
lowing factors and levels:
• Sound Type (S): {Baseline, OmniOscillator, MonoSynthesizer}
• Continuity Level (C): {discrete, continuous}
• Trend Type (T): {simple, complex}
We used Pleasantness (PL), Clarity (CL), Confdence (CF), and
Overall Score (OS) as our dependent variables. In our analysis, all
our dependent variables were ordinal (on a scale of 1-7). Therefore,
we conducted our analysis using an ordinal logistic regression
[11, 28]. We also included Subjectr as a random factor to account
for repeated measures. We tested our participants over 3×2×2=12
conditions, resulting in 12×10=120 total trials.
2.4 Results
We present the results of our user studies assessing Pleasantness
(PL), Clarity (CL), Confdence (CF), and Overall Score (OS) of soni-
fed responses. Additionally, we present our fndings on areas of
improvement for sonifed responses from our follow-up interviews.
2.4.1 Pleasantness (PL). Our results show a signifcant main ef-
fect of S (χ 2 (2, N =10)=12.15, p<.05, Cramer’s V =.42) on PL overall,
indicating that PL was signifcantly diferent between the three S
groups. Overall, the sonifed responses from OmniOscillator with
square waveform and discrete continuity level outperformed the
other prototypes (M=4.65). C (p ≈.844) and T (p ≈.122) did not have
a signifcant main efect on PL. Figure 1 and Table 2 show average
PL scores for each independent variable.
2.4.2 Clarity (CL). We found a signifcant main efect of C (χ 2 (1,
N =10)=11.03, p<.001, Cramer’s V =.40) and T (χ 2 (1, N =10)=22.83,
p<.001, Cramer’s V =.57) on CL overall. This result indicates that
CL was signifcantly diferent between discrete and continuous
sounds and also between simple and complex trends. On average,
the Baseline had the best scores (M=5.15). S (p ≈.476) did not have
a signifcant main efect on CL. We show the average scores for CL
in Figure 1 and Table 2 for each independent variable.
2.4.3 Confidence (CF). C (χ 2 (1, N =10)=6.36, p<.05, Cramer’s V =.30)
and T (χ 2 (1, N =10)=42.34, p<.001, Cramer’s V =.78) had a signif-
cant main efect on CF , indicating that CF signifcantly difered be-
tween simple and complex trends as well as discrete and continuous
sounds. Similar to PL, the sonifed responses from OmniOscillator
with square waveform and discrete continuity level had the best
overall average scores (M=5.20). We did not fnd a signifcant efect
of S on CF (p ≈.726). The average scores for CF are shown in Figure
1 and Table 2 for each independent variable.
2.4.4 Overall Score (OS). Our results show a signifcant main efect
of T on OS overall (χ 2 (1, N =10)=7.16, p<.05, Cramer’s V =.32). This
fnding indicates that OS signifcantly varied between the simple
ASSETS ’22, October 23–26, 2022, Athens, Greece
and complex trends. OmniOscillator with square waveform and
discrete continuity level performed the best compared to the other
prototypes (M=4.65) on average, similar to PL and CF . We only
found a marginal efect of S (p ≈.090) and C (p ≈.098) on OS. We
display the average scores for OS in Figure 1 and Table 2 for each
independent variable.
2.5 Areas of Improvement
Our participants recognized three areas of improvement for sonifca-
tion: (1) Personalization (customizing the auditory output for factors
including speed and frequency); (2) Identifcation of extrema/outliers
(identifying the maximum and minimum data points); and (3) Multi-
modality (using a combination of diferent instrument sounds to-
gether with diferent frequencies to amplify the distinctions be-
tween data points). Future work can incorporate our fndings to
improve the usability and user-friendliness of sonifed responses
for screen-reader users.
3 SONIFIER LIBRARY ENHANCEMENTS
Sonifier is an open-source JavaScript library that generates a
sonifed response from two-dimensional single-series data, recently
developed by Sharif et al. [23, 24]. Utilizing the fndings from our
user studies with screen-reader users, we enhanced the Sonifier
library by (1) improving its scalability to support several sound
types; and (2) modifying its default settings to those from our best-
performing prototype.
We refactored and modularized the code to enable developers to
create sonifed responses using additional sound types, including
MonoSynthesizers and Envelopes. (Currently, the Sonifier library
only supports the creation of sonifcation using an OmniOscillator.)
We further improve the customization for OmniOscillator by adding
more confguration options, including “sourceType” (source of the
oscillator; e.g., am), “baseType” (waveform of the oscillator; e.g.,
sawtooth), and “partialCount” (number of harmonics to generate
the waveform, ranging from 1-32).
Additionally, we modifed the default settings of the library based
on our fndings. Specifcally, we changed “sourceType” to am, “base-
Type” to square, and “partialCount” to 8. We made our code publicly
available at the Sonifier library’s open-source repository [23].
4 DISCUSSION & CONCLUSION
In this preliminary exploration, we investigated the usability and
user-friendliness of sonifed responses for screen-reader users via
user studies. Our results show that screen-reader users preferred
distinct non-continuous sonifed responses generated using oscilla-
tors with square waveforms. Additionally, our follow-up interviews
showed that our participants identifed the need to personalize the
sonifed responses. We utilized our fndings to enhance the capa-
bilities of Sonifier [23]—an open-source library that generates
sonifed responses from two-dimensional data.
While the prototype generated using an oscillator with square
waveform overall outperformed the rest of the prototypes, inter-
estingly, the Baseline (using a sawtooth waveform) had the best
average scores for Clarity. (However, Clarity did not have a sta-
tistically signifcant on Sound Type.) Our results identify avenues
for future work to investigate the efects of diferent waveforms
ASSETS ’22, October 23–26, 2022, Athens, Greece
on diferent aspects of user experiences. Additionally, our fndings
revealed that our participants preferred to personalize the sonifed
responses, depending on factors such as trend type and data cardi-
nality. Future work can utilize our fndings to generate a central
confguration system, enabling screen-reader users to customize
the responses per their needs.
Our work identifes avenues for future research to improve the
quality of sonifed responses. Specifcally, our work highlights that
utilizing the knowledge from domains outside of Human-Computer
Interaction, such as Music Theory, can improve the user experiences
for people who rely on auditory channels to extract information
from online data visualizations in a personalized manner. We hope
our work will inspire researchers to conduct an in-depth investiga-
tion of the usability and user-friendliness of sonifed responses and
improve the experiences of screen-reader users with the sonifca-
tion of online data visualizations.
ACKNOWLEDGMENTS
This work was supported by the University of Washington Center
for Research and Education on Accessible Technology and Expe-
riences (CREATE). We thank Jacob O. Wobbrock and Katharina
Reinecke for their helpful comments and suggestions. Finally, we
thank and remember our recently-departed team member Zoey for
her feline support, without which the purrusal of this work would
not have been as efective. May she cross the rainbow bridge in
peace and fnd her way to cat heaven.
REFERENCES
[1] Dragan Ahmetovic, Federico Avanzini, Adriano Baratè, Cristian Bernareggi,
Gabriele Galimberti, Luca A Ludovico, Sergio Mascetti, and Giorgio Presti. 2019.
Sonifcation of rotation instructions to support navigation of people with visual
impairment. In 2019 IEEE International Conference on Pervasive Computing and
Communications (PerCom. IEEE, 1–10.
[2] Dragan Ahmetovic, Cristian Bernareggi, João Guerreiro, Sergio Mascetti, and
Anna Capietto. 2019. Audiofunctions. web: Multimodal exploration of math-
ematical function graphs. In Proceedings of the 16th International Web for All
Conference. 1–10.
[3] Apple. n.d.. Audio Graphs | Apple Developer Documentation. https://developer.
apple.com/documentation/accessibility/audio_graphs. (Accessed on 08/01/2021).
[4] Michael Bostock, Vadim Ogievetsky, and Jefrey Heer. 2011. D3 data-driven
documents. IEEE transactions on visualization and computer graphics 17, 12 (2011),
2301–2309.
[5] Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz stud-
ies—why and how. Knowledge-based systems 6, 4 (1993), 258–266.
[6] Joel J Davis. 2002. Disenfranchising the Disabled: The Inaccessibility of Internet-
Based Health Information. Journal of Health Communication 7, 4 (2002), 355–367.
https://doi.org/10.1080/10810730290001701
[7] Danyang Fan, Alexa Fay Siu, Wing-Sum Adrienne Law, Raymond Ruihong Zhen,
Sile O’Modhrain, and Sean Follmer. 2022. Slide-Tone and Tilt-Tone: 1-DOF Haptic
Techniques for Conveying Shape Characteristics of Graphs to Blind Users. In
CHI Conference on Human Factors in Computing Systems. 1–19.
[8] Connor Geddes, David R Flatla, Garreth W Tigwell, and Roshan L Peiris. 2022.
Improving Colour Patterns to Assist People with Colour Vision Defciency. In
CHI Conference on Human Factors in Computing Systems. 1–17.
[9] Andrea Gerino, Lorenzo Picinali, Cristian Bernareggi, Nicolò Alabastro, and
Sergio Mascetti. 2015. Towards large scale evaluation of novel sonifcation tech-
niques for non visual shape exploration. In Proceedings of the 17th International
ACM SIGACCESS Conference on Computers & Accessibility. 13–21.
[10] Nicholas A. Giudice, Hari Prasath Palani, Eric Brenner, and Kevin M. Kramer. 2012.
Learning Non-Visual Graphical Information Using a Touch-Based Vibro-Audio
Interface. In Proceedings of the 14th International ACM SIGACCESS Conference on
Computers and Accessibility (Boulder, Colorado, USA) (ASSETS ’12). Association
for Computing Machinery, New York, NY, USA, 103–110. https://doi.org/10.
1145/2384916.2384935
[11] Pedro Antonio Gutiérrez, Maria Perez-Ortiz, Javier Sanchez-Monedero, Francisco
Fernandez-Navarro, and Cesar Hervas-Martinez. 2015. Ordinal regression meth-
ods: survey and experimental study. IEEE Transactions on Knowledge and Data
Ather Sharif, Olivia H. Wang, and Alida T. Muongchan
Engineering 28, 1 (2015), 127–146.
[12] Melita Hajdinjak and France Mihelic. 2004. Conducting the Wizard-of-Oz Exper-
iment. Informatica (Slovenia) 28, 4 (2004), 425–429.
[13] Highcharts. n.d.. Sonifcation | Highcharts. https://www.highcharts.com/docs/
accessibility/sonifcation. (Accessed on 08/01/2021).
[14] Leona M Holloway, Cagatay Goncu, Alon Ilsar, Matthew Butler, and Kim Marriott.
2022. Infosonics: Accessible Infographics for People who are Blind using Soni-
fcation and Voice. In CHI Conference on Human Factors in Computing Systems.
1–13.
[15] Edward Kim and Kathleen F McCoy. 2018. Multimodal deep learning using
images and text for information graphic classifcation. In Proceedings of the 20th
International ACM SIGACCESS Conference on Computers and Accessibility. 143–
148.
[16] Bongshin Lee, Arjun Srinivasan, Petra Isenberg, John Stasko, et al. 2021. Post-
WIMP Interaction for Information Visualization. Foundations and Trends® in
Human-Computer Interaction 14, 1 (2021), 1–95.
[17] Yotam Mann. n.d.. Tone.js. https://tonejs.github.io/. (Accessed on 08/02/2021).
[18] Kim Marriott, Bongshin Lee, Matthew Butler, Ed Cutrell, Kirsten Ellis, Cagatay
Goncu, Marti Hearst, Kathleen McCoy, and Danielle Albers Szafr. 2021. Inclusive
data visualization for people with disabilities: a call to action. Interactions 28, 3
(2021), 47–51.
[19] David K. McGookin and Stephen A. Brewster. 2006. SoundBar: Exploiting Multiple
Views in Multimodal Graph Browsing. In Proceedings of the 4th Nordic Conference
on Human-Computer Interaction: Changing Roles (Oslo, Norway) (NordiCHI ’06).
Association for Computing Machinery, New York, NY, USA, 145–154. https:
//doi.org/10.1145/1182475.1182491
[20] Giorgio Presti, Dragan Ahmetovic, Mattia Ducci, Cristian Bernareggi, Luca A.
Ludovico, Adriano Baratè, Federico Avanzini, and Sergio Mascetti. 2021. Iterative
Design of Sonifcation Techniques to Support People with Visual Impairments
in Obstacle Avoidance. ACM Trans. Access. Comput. 14, 4, Article 19 (oct 2021),
27 pages. https://doi.org/10.1145/3470649
[21] Ather Sharif, Sanjana Shivani Chintalapati, Jacob O Wobbrock, and Katharina
Reinecke. 2021. Understanding Screen-Reader Users’ Experiences with Online
Data Visualizations. In The 23rd International ACM SIGACCESS Conference on
Computers and Accessibility. 1–16.
[22] Ather Sharif and Babak Forouraghi. 2018. evoGraphs — A jQuery plugin to create
web accessible graphs. In 2018 15th IEEE Annual Consumer Communications
Networking Conference (CCNC). IEEE, Las Vegas, NV, USA, 1–4. https://doi.org/
10.1109/CCNC.2018.8319239
[23] Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and
Jacob O. Wobbrock. 2022. Sonifer: JavaScript library that converts a two-
dimensional data into a sonifed response. https://github.com/athersharif/sonifer.
(Accessed on 06/12/2022).
[24] Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and
Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible
with an Interactive JavaScript Plug-In. In CHI Conference on Human Factors in
Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing
Machinery, New York, NY, USA, Article 478, 19 pages. https://doi.org/10.1145/
3491102.3517431
[25] Lei Shi, Idan Zelzer, Catherine Feng, and Shiri Azenkot. 2016. Tickers and Talker:
An Accessible Labeling Toolkit for 3D Printed Models. In Proceedings of the
2016 CHI Conference on Human Factors in Computing Systems. Association for
Computing Machinery, New York, NY, USA, 4896–4907. https://doi.org/10.1145/
2858036.2858507
[26] Alexa Siu, Gene SH Kim, Sile O’Modhrain, and Sean Follmer. 2022. Supporting
Accessible Data Visualization Through Audio Data Narratives. In CHI Conference
on Human Factors in Computing Systems. 1–19.
[27] Ruobin Wang, Crescentia Jung, and Yea-Seul Kim. 2022. Seeing Through Sounds:
Mapping Auditory Dimensions to Data and Charts for People with Visual Impair-
ments. In The 24th EG/VGTC Conference on Visualization (EuroVis’ 22), Rome, Italy,
13-17 June, 2022. Eurographics-European Association for Computer Graphics.
[28] Christopher Winship and Robert D Mare. 1984. Regression models with ordinal
variables. American sociological review (1984), 512–525.
[29] Haixia Zhao, Catherine Plaisant, Ben Shneiderman, and Jonathan Lazar. 2008.
Data sonifcation for users with visual impairment: a case study with georefer-
enced data. ACM Transactions on Computer-Human Interaction (TOCHI) 15, 1
(2008), 1–28.
“What Makes Sonification User-Friendly?” ASSETS ’22, October 23–26, 2022, Athens, Greece

A PARTICIPANT DEMOGRAPHICS

Gender Age Screen Vision-Loss Level Diagnosis

Reader
P1 M 57 JAWS Lost vision gradually Retinitis Pigmentosa
P2 F 38 JAWS Blind since birth Leber Congenital Amaurosis
P3 F 65 JAWS Lost vision gradually Retinitis Pigmentosa
P4 F 69 Fusion Lost vision gradually, Partial vision Juvenile Macular Degeneration
P5 M 33 NVDA Blind since birth Peters Anomaly
P6 M 37 JAWS Blind since birth Leber Congenital Amaurosis
P7 F 52 JAWS Blind since birth Retinopathy
P8 M 58 JAWS Lost vision gradually Cataracts and Glaucoma
P9 M 49 JAWS Lost vision gradually Leber Congenital Amaurosis
P10 NB 26 VoiceOver Partial vision Corneal damage
Table 1: Screen-reader participants, their gender identifcation, age, screen reader, vision-loss level, and diagnosis. Under the
Gender column, M = Male, F = Female, and N B = Non-binary.

B SUBJECTIVE RATINGS

Average Scores
Sound Type (S) Continuous PL CL CF OS
No 3.35 5.15 5.10 3.90
OmniOscillator with sawtooth waveform (Baseline)
Yes 3.95 4.55 4.70 3.75
No 4.65 5.05 5.20 4.65
OmniOscillator with square waveform
Yes 4.30 4.50 4.85 4.05
No 3.75 4.95 5.15 4.20
MonoSynthesizer with square waveform
Yes 3.20 4.15 4.58 3.80
Table 2: Overall average scores for each sound type per continuity level. PL represents Pleasantness, CL represents Clarity, CF
represents Confdence, and OS represents Overall Score. Highest average scores for PL, CL, CF , and OS are shown in bold.
 “What’s going on in Accessibility Research?” Frequencies and
Trends of Disability Categories and Research Domains in
Publications at ASSETS
Ather Sharif
asharif@cs.washington.edu
Paul G. Allen School of Computer
Science & Engineering | DUB Group,
University of Washington
Seattle, Washington, USA
Ruoxi Shang
rxshang@uw.edu
Human Centered Design and
Engineering | DUB Group,
University of Washington
Seattle, Washington, USA
ABSTRACT
ACM SIGACCESS Conference on Computers and Accessibility (AS-
SETS) is considered one of the premium forums for research on
accessibility. Recently, Mack et al. shed light on the demograph-
ics, goals, research methodologies, and evolution of accessibility
research over time. We extend their work by exploring the fre-
quencies and trends of disability categories and computer science
research domains in publications at ASSETS (N =1,678). Our re-
sults show that disability categories and research domains varied
signifcantly across the publication years. We found that in the
past 10 years, publications targeting Mental-Health-Related disabili-
ties and the research domain of AR/VR show an increasing trend.
In opposition, Gaming, Input Methods/Interaction Techniques, and
User Interfaces domains portray a decreasing trend. Additionally,
our results show that the majority of the publications utilize the
AI/ML/CV/NLP domain (19%) and focus on people with visual dis-
abilities (42%). We share our preliminary exploration results and
identify avenues for future work.
CCS CONCEPTS
• Human-centered computing → Accessibility theory, con-
cepts and paradigms; Accessibility design and evaluation meth-
ods; • Social and professional topics → People with disabili-
ties.
∗ These authors contributed equally to this work.
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550359
Ploypilin Pruekcharoen∗ Thrisha Ramesh∗
ploypp@uw.edu thrisha@cs.washington.edu
Human Centered Design and Paul G. Allen School of Computer
Engineering | DUB Group, Science & Engineering,
University of Washington University of Washington
Seattle, Washington, USA Seattle, Washington, USA
Spencer Williams Gary Hsieh
sw1918@uw.edu garyhs@uw.edu
Human Centered Design and Human Centered Design and
Engineering | DUB Group, Engineering | DUB Group,
University of Washington University of Washington
Seattle, Washington, USA Seattle, Washington, USA
KEYWORDS
accessibility, disability categories, research domains, frequency,
trend, assets
ACM Reference Format:
Ather Sharif, Ploypilin Pruekcharoen, Thrisha Ramesh, Ruoxi Shang, Spencer
Williams, and Gary Hsieh. 2022. “What’s going on in Accessibility Research?”
Frequencies and Trends of Disability Categories and Research Domains in
Publications at ASSETS. In The 24th International ACM SIGACCESS Con-
ference on Computers and Accessibility (ASSETS ’22), October 23–26, 2022,
Athens, Greece. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/
3517428.3550359
1 INTRODUCTION
With the increase in disability awareness, research focusing on ac-
cessible and assistive technology for disabled people has drastically
increased over the past few years [3, 11]. While several venues
exist for accessibility-related publications, the ACM SIGACCESS
Conference of Computing and Accessibility (ASSETS) is amongst
the premium forums for publications focusing on the design, evalu-
ation, use, and education related to computing for disabled people
[1]. Given its reputation for publishing top-tier work, ASSETS is
the go-to conference for both seasoned and new researchers.
Understanding the evolution of accessibility research over time at
ASSETS can reveal crucial information, including norms, gaps, and
adoptions of technical and societal concepts in academia [6, 8, 11].
Such information is benefcial for researchers to refect on the
growth of accessibility as a research feld and can guide them in
identifying avenues for future work. Prior work has conducted
literature surveys for sub-demographics within the disability de-
mographic [4, 5, 10, 11, 13, 16–18]. Most relevant to our work is
the recent exploration by Mack et al. [11] that seeks to understand
the demographics, goals, research methodologies, and evolution
of accessibility research over time. We build on their work by (1)
extending their identifed communities of focus to incorporate more
nuanced disability categories; (2) investigating the utilization of
computer science research domains over time; and (3) conducting
ASSETS ’22, October 23–26, 2022, Athens, Greece Ather Sharif, Ploypilin Pruekcharoen, Thrisha Ramesh, Ruoxi Shang, Spencer Williams, and Gary Hsieh

Table 1: Overview of keyword counts and percentages, and trend analyses for each disability category and research domain
in publications at ASSETS overall (2000-2021) and in recent years (2012-2021). N is the total keyword count and % is the per-
centage compared to the total keyword count for all the categories. τ is the measure of the ordinal association between cat-
egories/domains and their ratios (+ve τ values mean increasing trend and -ve τ values mean decreasing trend). p shows the
statistical signifcance of the trend (α=.05).

Overall (2000-2021) Past 10 Years (2012-2021)

N % τ p N % τ p
Disability Category
Auditory 256 17% .57 <.001 183 18% .32 .243
Chronic Illness 11 1% -.07 .740 6 1% -.08 .838
Cognitive 188 12% .03 .871 116 11% -.07 .858
Learning 89 6% .39 .019 67 7% -.09 .788
Mental-Health-Related 13 1% .60 <.001 12 1% .60 .026
Mobility 175 11% .08 .650 116 11% -.11 .721
Neurological 55 4% .01 .998 37 4% -.32 .243
Older Adults 98 6% -.08 .626 58 6% .09 .788
Visual 643 42% .15 .381 419 41% .20 .474
Research Domain
3-D Representation 71 3% .54 <.001 62 4% -.07 .858
AI/ML/CV/NLP 498 19% -.32 .051 280 18% .33 .211
AR/VR 75 3% .32 .055 61 4% .54 .039
Educational/Methodological/Theoretical 401 16% .22 .194 256 16% .47 .074
Gaming 103 4% .23 .163 75 5% -.51 .049
Hardware Tools 281 11% -.11 .516 163 10% -.07 .858
Input Methods/Interaction Techniques 252 10% -.39 .019 139 9% -.56 .032
Media/Graphics/Visualizations 283 11% .28 .098 180 11% .24 .371
Security/Privacy 44 2% .31 .067 35 2% .11 .721
Software Tools 299 12% -.10 .559 180 11% -.09 .788
User Interfaces 159 6% -.55 <.001 68 4% -.51 .049
Wearables 89 3% .55 <.001 78 5% .45 .088

Figure 1: Trends for each disability category in ASSETS publications overall (2000-2021).

empirical statistical analyses to explore the frequencies and trends
of each disability category and research domain.
To shed more light on the frequencies and trends in accessibil-
ity research, frst, we scraped the keywords from ASSETS papers
published since 2000, including poster and demonstration papers.
Then, we manually categorized the keywords into nine disability
categories and 12 research domains to analyze the frequencies and
trends for each category and domain over time. We further ana-
lyzed our data by fltering the publications to only include those
published recently (past 10 years; 2012-2021). Overall, we extracted
3,234 keywords from 1,678 papers.
We found that disability categories and research domains vary
signifcantly across publication years. Visually disabled people were
the largest targeted audience (42% overall and 41% in the past 10
years), and publications on Mental-Health-Related disabilities show
“What’s going on in Accessibility Research?”
Figure 2: Trends for each research domain in ASSETS publications overall (2000-2021).
an increasing trend overall and in recent publications. Additionally,
the majority of the publications employed the research domain of
AI/ML/CV/NLP (19% overall and 18% in the past 10 years), with
publications on AR/VR showing an increasing trend in recent years.
In contrast, Gaming, Input Methods/Interaction Techniques, and User
Interfaces domains showed a decreasing trend in recent years.
We contribute the empirical fndings from our preliminary ex-
plorations. Specifcally, we provide the results from our statistical
analyses of frequencies and trends of (1) disabilities categories and
(2) research domains over the past (1) 20 years (2000-2021; ASSETS
skipped publications in 2001 and 2003) and (2) 10 years (2012-2021).
Additionally, we identify avenues for future work.
2 FREQUENCIES AND TRENDS IN
PUBLICATIONS AT ASSETS
We studied the frequencies and trends of disability categories and
computer science research domains by extracting 3,234 keywords
from 1,678 publications at ASSETS. We present our methodology,
analysis, and quantitative results.
2.1 Data Collection & Procedure
We queried the ACM Digital Library to collect the author keywords
of all the papers published at ASSETS (N =1,678) since its inception
in 1994, similar to prior work [10, 11]. However, unlike these works,
our data set included the poster, panel, and demonstration papers.
Due to missing author keywords in publications from years before
2000 (only 59% had author keywords), we reduced our data set to
publications from 2000 onward (90% had author keywords). After
reduction, our data set included 3,234 keywords from 1,603 papers.
We provide our collected keywords with their overall counts in the
supplementary materials.
First, we defned the factor levels for disability categories and
research domains. We selected the disability categories from prior
work [15]. For the computer science research domains, we fnal-
ized the set through discussions with well-published researchers
at ASSETS. Then, we manually categorized each keyword into ap-
propriate disability categories and research domains (a keyword
could belong to multiple categories and research domains). For
research domains, categorization qualifcations included applied
ASSETS ’22, October 23–26, 2022, Athens, Greece
work. We eliminated broad (e.g., “accessibility”) and ambiguous
(e.g., “Germany”) keywords from our data set. After categorization,
we calculated ratios for each category and domain per publication
year. (Ratio was the frequency of the total papers containing a
given category/domain’s keywords divided by the total number of
publications with at least one author keyword per year.) Our fnal
set comprised a total of 1,885 keywords.
At least three researchers participated in the keyword catego-
rization process to account for accurate categorization. We resolved
any disagreements through mutual discussions.
2.2 Analysis & Results
Our goal was to examine the frequencies and trends of disability
categories and research domains in overall (2000-2021) and recent
(2012-2021) publications at ASSETS. Our preliminary analysis using
Anderson-Darling [2] tests of normality showed that the ratios were
conditionally non-normal. Therefore, we used a generalized linear
model [7, 14] with Gamma distribution and log link function to
investigate frequencies, as our data was positive and right-skewed.
Category (C) and Domain (D) were the independent variables for
analyzing disability categories and research domains, respectively,
whereas Ratio (R) was the dependent variable. Additionally, we
used Mann-Kendall [9, 12] test to evaluate temporal trends for each
category and their respective ratios across the publication years.
2.2.1 Disability Categories. Our results show a signifcant main
efect of Category (C) on R (χ 2 (1, N =180)=363.83, p<.001, Cramer’s
V =.50), with 42% of the publications on the disability category Vi-
sual. Filtering the publication years to include only the past 10 years
(2012-2021) yields similar results, showing a signifcant main efect
of C on R (χ 2 (1, N =90)=253.32, p<.001, Cramer’s V =.59) and 41% fo-
cusing on people with Visual disabilities. These results indicate that
the disability categories vary signifcantly in ASSETS publications.
Figure 3 (Appendix A) and Table 1 show the percentages.
Our trend analysis for each disability category identifes an in-
creasing trend for disability categories Auditory (τ =.57, p<.001),
Learning (τ =.39, p<.05), and Mental-Health-Related (τ =.60, p<.001)
overall. However, in recent publications, only Mental-Health-Related
category shows an increasing trend (τ =.60, p<.05). Figure 1 shows
the trends and Table 1 displays the statistical results.
ASSETS ’22, October 23–26, 2022, Athens, Greece Ather Sharif, Ploypilin Pruekcharoen, Thrisha Ramesh, Ruoxi Shang, Spencer Williams, and Gary Hsieh
2.2.2 Research Domains. Domain (D) also had a signifcant main
efect on R for both overall (χ 2 (1, N =240)=334.89, p<.001, Cramer’s
V =.36) and in recent publications (χ 2 (1, N =120)=207.43, p<.001,
Cramer’s V =.44). Similar to C, these results indicate that the re-
search domain employed in publications at ASSETS varied signif-
cantly. The majority of the publications employed the domain of
AI/ML/CV/NLP overall (19%) and in recent years (18%). Figure 4
(Appendix A) and Table 1 show the percentages per domain.
The results from our trend analysis for each research domain
reveal that 3-D Representation (τ =.54, p<.001) and Wearables (τ =.55,
p<.001) had an increasing trend overall, whereas only AR/VR had
an increasing trend in recent publications. In contrast, Gaming
(τ =-.51, p<.05) had a decreasing trend in recent publications. Input
Methods/Interaction Techniques (τ =-.39, p<.05) and User Interfaces
(τ =-.55, p<.001) had a decreasing trend both overall and in recent
publications. We show the trends in Figure 2 and statistical results
in Table 1 for each research domain.
3 DISCUSSION & CONCLUSION
In this preliminary exploration, we examined the frequencies and
trends of disability categories and computer science research do-
mains in overall and recent publications at ASSETS, extending the
fndings from Mack et al. [11]. To achieve this goal, we extracted
3,234 keywords from 1,678 papers, including the poster and demon-
stration papers, and manually categorized them into nine disability
categories and 12 research domains. Our results show that the focus
on disability categories and research domains signifcantly varies
across publication years. Additionally, we conducted trend analyses
to identify the trends for each category and domain overall and in
recent publications at ASSETS.
Similar to the fndings by Mack et al. [11], our results showed
that publications focusing on visually disabled people are dispropor-
tionately higher than those focused on other disability categories.
Interestingly, our analyses did not identify a statistical increase or
decrease in the trend for publications focused on this demographic,
likely indicating consistency in the higher focus. On the other hand,
the Mental-Health-Related category shows an increasing trend in
both overall and recent publications, possibly attributing to the
recently increasing awareness of mental-health-related matters.
Future work can explore the correlation between public aware-
ness of specifc disability categories and their respective focus in
publications at ASSETS to investigate knowledge difusion.
Similarly, our analyses revealed an increasing trend of AR/VR
in publications at ASSETS over the recent years, likely attributing
to its growing technological focus within and outside academia.
Surprisingly, the adoption of 3-D Representation in publications
shows an overall increasing trend but not for the recent publications.
Our work provides avenues for researchers to explore and gather
further insights into technology difusion in accessibility research.
Since our work is a preliminary exploration, we only performed
categorization using the author-identifed keywords. Additionally,
while ASSETS is considered the top forum for accessibility research,
other venues for accessibility-related research, including the ACM
Conference on Human Factors in Computing Systems (CHI), Web
for All Conference (W4A), and other academic journals are also
prevalent. Therefore, future work can employ our methodology
and extend our work to analyze publications from other venues.
We hope our work will inspire researchers to explore frequencies
and trends in accessibility research, providing further insights into
the growth of this feld.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their helpful comments and
suggestions. Additionally, we thank and remember our recently-
departed team member Zoey for her feline support, without which
the purrusal of this work would not have been as efective. May she
cross the rainbow bridge in peace and fnd her way to cat heaven.
REFERENCES
[1] ACM. n.d.. ASSETS Conference - Home. https://dl.acm.org/conference/assets.
(Accessed on 06/08/2022).
[2] Theodore W. Anderson and Donald A. Darling. 1954. A test of goodness of ft.
Journal of the American statistical association 49, 268 (1954), 765–769.
[3] GVU Center at Georgia Tech. n.d.. Workbook: Weaving CHI - Top Keyword Top-
ics. https://public.tableau.com/views/WeavingCHI\protect\discretionary{\char\
hyphenchar\font}{}{}TopKeywordTopics/TopKeywordTopics. (Accessed on
06/08/2022).
[4] Alexy Bhowmick and Shyamanta M Hazarika. 2017. An insight into assistive
technology for the visually impaired and blind people: state-of-the-art and future
trends. Journal on Multimodal User Interfaces 11, 2 (2017), 149–172.
[5] Emeline Brulé, Brianna J Tomlinson, Oussama Metatla, Christophe Joufrais,
and Marcos Serrano. 2020. Review of Quantitative Empirical Evaluations of
Technology for People with Visual Impairments. In Proceedings of the 2020 CHI
Conference on Human Factors in Computing Systems. 1–14.
[6] Faye Ginsburg and Rayna Rapp. 2013. Disability worlds. Annual Review of
Anthropology 42, 1 (2013), 53–68.
[7] John M Grego. 1993. Generalized linear models and process variation. Journal of
Quality Technology 25, 4 (1993), 288–295.
[8] Alan M Jette, Marilyn J Field, et al. 2007. The future of disability in America.
(2007).
[9] Maurice George Kendall. 1948. Rank correlation methods. (1948).
[10] Lior Levy, Qisheng Li, Ather Sharif, and Katharina Reinecke. 202. Respectful
Language as Perceived by People with Disabilities. In The 23rd International
ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA)
(ASSETS ’21). Association for Computing Machinery, New York, NY, USA, Article
83, 4 pages. https://doi.org/10.1145/3441852.3476534
[11] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich,
and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? A
Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019.
In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York,
NY, USA, Article 371, 18 pages. https://doi.org/10.1145/3411764.3445412
[12] Henry B Mann. 1945. Nonparametric tests against trend. Econometrica: Journal
of the econometric society (1945), 245–259.
[13] Aboubakar Mountapmbeme, Obianuju Okafor, and Stephanie Ludi. 2022. Address-
ing Accessibility Barriers in Programming for People with Visual Impairments:
A Literature Review. ACM Transactions on Accessible Computing (TACCESS) 15, 1
(2022), 1–26.
[14] John Ashworth Nelder and Robert WM Wedderburn. 1972. Generalized linear
models. Journal of the Royal Statistical Society: Series A (General) 135, 3 (1972),
370–384.
[15] Ather Sharif, McCall Aedan L., and Bolante Kianna R. 2022. Should I Say “Disabled
People” or “People with Disabilities”? Language Preferences of Disabled People
Between Identity- and Person-First Language. In The 23rd International ACM
SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS
’22). Association for Computing Machinery, New York, NY, USA, To Appear.
[16] Katta Spiel, Christopher Frauenberger, Os Keyes, and Geraldine Fitzpatrick. 2019.
Agency of autistic children in technology research—A critical literature review.
ACM Transactions on Computer-Human Interaction (TOCHI) 26, 6 (2019), 1–40.
[17] METTE Warburg. 2001. Visual impairment in adult people with intellectual
disability: literature review. Journal of intellectual disability research 45, 5 (2001),
424–438.
[18] Rua M Williams and Juan E Gilbert. 2020. Perseverations of the academy: A
survey of wearable technologies applied to autism intervention. International
Journal of Human-Computer Studies 143 (2020), 102485.
“What’s going on in Accessibility Research?” ASSETS ’22, October 23–26, 2022, Athens, Greece

A PUBLICATIONS PER DISABILITY CATEGORIES AND RESEARCH DOMAIN PER YEAR

Figure 3: Percentage of ASSETS publications per year for each disability category.

Figure 4: Percentage of ASSETS publications per year for each research domain.
A Participatory Design Approach to Explore Design Directions for
Enhancing Videoconferencing
Experience for Non-signing Deaf and Hard of Hearing Users
Yeon Soo Kim
Industrial Design, KAIST
Daejeon, Repuplic of Korea
ykim18@kaist.ac.kr
ABSTRACT
The breakout of the COVID-19 pandemic shifted people’s daily ac-
tivities from in-person to video-mediated ones. Many people with
hearing loss encounter cognitive overload due to inefective visu-
als of the videoconferencing interface and therefore fnd meeting
contents difcult to comprehend. This research incorporates a par-
ticipatory design methodology to investigate the Deaf and Hard of
Hearing (DHH) users’ tacit needs. DHH users demonstrated ways
of mitigating their hardships in the workshop, such as emphasiz-
ing the visual hierarchy or assigning visual cues to fxed positions.
These fndings are used in developing design directions for creating
a more inclusive online environment.
CCS CONCEPTS
• Human-centered computing → Graphical user interfaces.
KEYWORDS
Hard of hearing, Accessibility, Videoconferencing
ACM Reference Format:
Yeon Soo Kim, Sunok Lee, and Sangsu Lee. 2022. A Participatory Design
Approach to Explore Design Directions for Enhancing Videoconferencing
Experience for Non-signing Deaf and Hard of Hearing Users. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
4 pages. https://doi.org/10.1145/3517428.3550375
1 INTRODUCTION
The breakout of the COVID-19 pandemic pressured communities
and businesses around the globe to physically isolate and familiarize
themselves with the virtual world, enabling various professional
work to be done without having to relocate. The current work-
from-home nature of jobs seems to ofer a potential solution for the
work limitations in the DHH community, such as the barriers in
daily commute, which can lead to job termination[4, 12]. However,
the sound-dependent system of current videoconferencing services
is not properly designed for people with hearing loss and does not
support an inclusive nor compelling work environment the DHH
workers need, leaving most DHH workers in distress [1, 13]. This
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550375
Sunok Lee Sangsu Lee
Industrial Design, KAIST Industrial Design, KAIST
Daejeon, Repuplic of Korea Daejeon, Repuplic of Korea
sunoklee@kaist.ac.kr sangsu.lee@kaist.ac.kr
work focuses on the Deaf and hard of hearing (DHH) people’s use
of the videoconferencing tools in a work environment to address
these restraints.
There are ongoing eforts to make videoconferencing inclusive
of the DHH community, and they have learned to adapt to it in a
few ways, like reading captions, interpreting lip movements, or hav-
ing sign-language interpreters [7, 24]. However, these approaches
do not resolve the fundamental restraints. Advancement of auto-
captioning technology[16] attempt to assist the DHH users in some
ways but are often incomplete, incorrect, or delayed. Lip-readers
fnd the digital screen size and quality not adequate for lip-reading
[15]. DHH users are frequently left with fractional information in-
adequate for a complete assistance, and their eforts to gather these
fragmental data for comprehension leave them with an excessive
cognitive load [18]. These technical problems are not always solved
by having signers available. Merely providing signers in an online
meeting is counterproductive to making a videoconferencing en-
vironment accessible because only 1% of DHH people know sign
language, and even those who know sign language may prefer to
understand hearing people without a signer [11].
This research seeks to address an area that has not been discussed:
DHH people’s videoconferencing occasions without sign language.
We aim to create an inclusive environment for the DHH community
with varying preferences of communication strategies. Our research
goals are: 1) Understand DHH user’s pain points and needs for
communicating with coworkers when sign language is not the
main communication method, and 2) Explore design directions to
enhance these user’s video-mediated working experiences
For our study, we applied a participatory design method [3] with
a close interview to present qualitative results showing pain points
and the possible design directions to enhance videoconferencing
experiences for DHH users in non-signing situations.
2 RELATED WORKS
There are prior researches conducted pursuing accessible com-
munication services for DHH people. There are lists of common
difculties in DHH people’s videoconferencing experiences and
guidelines for mitigating those, such as providing live caption and
transcripts[10] or providing visual and haptic feedback [19]. While
these studies provide important fndings for the DHH community
and suggest some foundational guidelines in making videoconfer-
encing accessible, these works consider sign language a primary
communication method. On the other hand, our work aims to pin-
point DHH people’s experiences in a non-signing context. Another
research integrates ASR (automated speech recognition system) to
ASSETS ’22, October 23–26, 2022, Athens, Greece
Table 1: List of Participants
Hearing Loss Level
Participant Gender (dB HL) Afliation
P1 Female Moderately severe (56-70) Student
P2 Male Moderately severe (56-70) Store manager Partial hearing, Lip-reading
Lip-reading, Reading facial
P3 Other Moderately severe (56-70) Music producer expressions and body language Reading captions, Typing
P4 Male Severe (56-70) Engineer
aid conversation between Deaf and hearing pairs [20], primarily fo-
cusing on text-based conversation, which is one of many necessary
communication strategies for DHH users. This study intents to look
at multiple conversational techniques combined as a whole (e.g. lip-
reading, reading facial expressions and body language, using text,
etc.), for many DHH people use combination of communication
method for fuller comprehension.There were also approaches using
live captions for DHH people’s comprehension in real-life situations
by using digital aid [25] or using smart glasses [17]. These works
illustrate valuable data on DHH people’s cognitive interaction with
live captions; however, they are confned to real-life situations and
possibly not be the case in virtual settings.
Moreover, current studies of DHH users’ videoconferencing ex-
perience largely depend on verbal interviews, and an approach to
discovering their tacit needs is uncommon. It is difcult for the
researchers to relate to the experiences of users with impairments
because their experiences widely vary [21] and the experiences
relating to their disabilities may be hard to articulate [5]. Under-
standing end-user experiences are crucial to align with the actual
needs of the user [9] when the issues are driven by their limitations
[8]. Although some studies incorporate co-design workshops with
DHH users to do so[17, 19], common ability bias during co-design
workshops can bias end-users’ real feedback and alter their au-
thentic insights [2]. Therefore, conducting a participatory design
activity with DHH users is central when we lack knowledge about
their interaction with digital technologies [14].
3 METHOD
The user study is divided into two consecutive sessions. First, inter-
view session explores the real-life difculties users face, discovers
the essential areas for improvement, and triggers users’ memories
from past videoconferencing experiences. Participants are asked
to point out the aspects of current video-mediated services that
need improvement in their perspectives. Second, a participatory
design workshop is conducted over Google Slides [22] to have users
directly involved in rearranging the main UI components of Zoom.
In this session, the users discover their ways of optimizing their
comprehension in an online environment and seek solutions to
make it accessible for themselves. We anticipate these design de-
cisions in the workshop to uncover users’ tacit needs that may be
difcult to verbalize.
The study is conducted one-on-one in a private setting for each
participant to enhance freer discussion on disability. The user study
Yeon Soo Kim, Sunok Lee, and Sangsu Lee
Communication Method Communication Method
(In-person) (Online)
Lip-reading ASL, Lip-reading, Reading captions
Partial hearing, Lip-reading,
Reading captions
Partial hearing, Lip-reading,
ASL, Lip-reading, Writing ASL, Lip-reading, Reading captions
takes place on Zoom[26], the most commonly used and preferred
videoconferencing tool for all recruited participants. The users par-
ticipated in the remote user study in their usual settings, which also
acted to provoke their past memories of video-meeting experiences
for the user study.
3.1 Participants
We recruited four participants with moderately severe to severe
hearing loss who were frequently involved (more than three times
a week) in a videoconference. The participants were often involved
in work related video meetings without using sign language (refer
to Table 1 for detailed information on recruited participants).
3.2 Procedure
For efective discussion, we frst show a short video of a real cor-
porate Zoom meeting with six workers to provoke users’ past ex-
periences. We then proceed with our interview session and ask
the users questions regarding the general use of online meetings,
their change in communication techniques between in-person and
online conversation, and their approaches to understanding others
better online. Our questions are solely used to guide the discussion,
and participants are encouraged to express anything they like.
For participatory design workshops, users design iterations based
on their personal experiences to create the desired interpretation of
Zoom. The participants are provided Zoom components on Google
Slides for modifcation (Fig. 1a). The components include a GIF
image of the speaker, GIFs of fve non-speakers, text for captions,
and a shared screen. The meeting scene used for the workshop com-
prises six total members; according to statistics, most work-related
meetings include 4-6 people[23]. Possible modifcations of these
components are resizing, relocating, cropping, changing bright-
ness or contrast, or recoloring them (Fig. 1b, 1c). There are three
tasks for designing: 1) to optimize understanding with captions,
2) to optimize understanding with lip-reading, and 3) to optimize
understanding with additional shared material.
4 FINDINGS AND DISCUSSION
All data were collected and analyzed by coding and afnity tech-
nique to extrude current problems and the corresponding solutions
found from the users. The users’ approaches for mitigating barriers
and optimizing their comprehension were discovered by looking at
their design decisions: the components they decided to change and
how they changed them. We grouped the users’ solutions and our
A Participatory Design Approach to Explore Design Directions for Enhancing Videoconferencing
Experience for Non-signing Deaf and Hard of Hearing Users ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 1: a) Google slides set-up for the workshop include GIF of the speaker (pink dashed outline), GIFs of other meeting
members (blue fll) and subtitles (yellow solid outline). Participants are given common Zoom interface image as their workplace
(green dotted outline), b) Tutorial slide for creating and decorating text and inserting shapes, and c) Tutorial slide for adjusting
the GIF.

Figure 2: a) P2’s placement of the presenter (pink dashed outline) and the live captions (yellow solid outline), b) P4’s placement
of the presenter and live captions with a shared screen (blue fll), and c) P2’s design of zoomed in screen next to the speaker
(green dotted outline).

design directions under the corresponding problems: 1) Limitations
of lip-reading leaves users highly dependent on captions, and 2)
Unclear identifcation of speaker makes users neglected from the
meeting.
4.1 Limitations of Lip-reading Leaves Users
Highly Dependent on Captions
The diference in lip-reading experience between online and in-
person communication was considered the biggest challenge of
video-mediated communication. For the face-to-face conversations,
all participants preferred to lip-read, and "there is no need to use text
to communicate in-person"(P2). On the other hand, in a videocon-
ference, the captions are a necessity (P2, P3), forcefully shifting the
users’ attention from looking at the speaker to focusing on the cap-
tions, confning the DHH users to a text-oriented communication
method. However, these participants emphasized the importance
of keeping eye contact, looking at the speaker’s gestures, facial
expressions, and precise lip movements to follow the conversation
as a whole, and many did not prefer to rely on captions as an only
communicative method.
Users’ Approach: Emphasized Visual Hierarchy for Diverse Com-
munication Techniques. We observed the users setting visual hier-
archy and emphasizing information based on its signifcance. For
example, the speaker’s screen is usually considered the most im-
portant and is enlarged to be the largest component on the screen
(Fig. 2a). However, in lip-reading scenarios, the zoomed-in portion
of the speaker becomes the largest (Fig. 2c). The captions come
next in the hierarchy, occupying ample space and located next to
the speaker. Non-speaking participants are placed smaller than the
speaker but are big enough for users to see their facial expressions
(Fig. 2) because users consider them important for suggesting the
overall ambiance of the meeting (P1). In the case of meetings with
an additional screen being shared, the shared screen and the speaker
takes up the majority of the screen (Fig. 2b), and "those who don’t
talk can be eliminated on occasion to reduce visual distraction" (P4).
The amount of captions is reduced in this case because having more
text causes confusion when another material is being shared (P2).
Our users’ comprehension method seemed far more complicated
than those of signers; we observed non-signing DHH individuals
to switch communication method within the meeting according
to meeting context, speech-rate, or their momentary capability of
taking in cognitive load.
Design Directions: An accessible design should not confne the
user to a certain type of communication method and ofer all pos-
sible ones. Additional screens of speaker’s face or lips should be
available to support a clearer view, and the visual information must
be prioritized accordingly. The size of the videoconferencing screen
is often limited, and securing the right amount of space is vital when
designing for DHH users. Videoconferencing interface should ofer
diferent communication modes: for lip-reading, for reading body
language or facial expressions, for just caption reading, etc. This fea-
ture is especially vital for non-signers because signing DHH people
have tendency to focus on the signer and the speaker throughout
the whole meeting, whereas non-signers need more freedom in
choosing the right source of information to meet their dynamic
needs.
4.2 Unclear Identifcation of Speaker Make
Users Neglected From the Meeting
Video meeting participants often come across occasions when mul-
tiple people are speaking at the time or when the speaker change
is abrupt. It is difcult for our DHH users to see other participants
ASSETS ’22, October 23–26, 2022, Athens, Greece
when their eyes are focused on the current speaker. The inability of
speaker identifcation is problematic because being unsure of the
current speaker leads to difculty fnding the right time to speak
up and, most of the time, they decide not to speak at all(P3, P4).
The participants claimed the assistance from the host as unhelpful
because "it’s very hard for the presenter to realize, understand, or get
to know how many people ... have been left out."(P4) in an online
setting.
Users’ Approach: Dedicated Locations for Each Information for
Instant Identifcation. Our DHH users naturally found two solutions
to mitigate their cognitive load and keep up with meeting contents:
1) having essential components closer together and 2) consistently
placing the essential components. Our participants assigned the
crucial components close to each other and in a place where they
could easily focus. Moreover, all participants asked to keep speaker
and captions and other components to be "always fxed in their
position" (P1, P2, P3, P4) with captions always close to the speaker
screen (P1, P2, P3, P4) (Fig 2a). Although the participants have
diferent preferences of visual arrangement (e.g. left-bottom for
P2, top-center for P3), they expressed the common need to have a
specifed location for these elements, preventing them from chasing
around the speaker like they currently have to.
Design Directions: Decreasing split visual attention [6] to reduce
cognitive overload is commonly discussed; however, we recom-
mend a novel implication, which is to provide a designated location
for informative elements. The current "speaker mode" in Zoom at-
tempts to fx the speaker in one place. However, this feature makes
other non-speakers’ screens very small and thus considered un-
suitable; it does not consider that these users often want to clearly
see other participants during the meeting. Also, speaker screen
is not the only important element during videoconferences, and
all elements must be carefully and coherently placed. By doing so,
users can efortlessly access visual information all in one picture
and not lose optic focus by having to hunt for wanted elements on
a given screen.
5 CONCLUSION
Through our user study, we aimed to understand the videoconfer-
encing experiences from DHH people’s perspectives and pinpointed
critical insights that need to be considered to resolve the current
issues. The study tackles an unexplored perspective of not sign
language oriented online communication of the DHH community.
We propose how their techniques for better comprehension can be
utilized to provide a design direction that needs to be considered
to make videoconferencing interface more accessible.
REFERENCES
[1] 2020. Deaf at work during the pandemic: Making gains with fexibility, optimism,
and resilience. https://www.nationaldeafcenter.org/news/deaf-work-during-
pandemic-making-gains-fexibility-optimism-and-resilience
[2] Ashley Ashley. 2022. The diference between co-design and participatory
design. https://uxdesign.cc/diference-between-co-design-participatory-design-
df4376666816#:~:text=Participation%20allows%20for%20more%20input,the%
20project%20goals%20and%20outcomes.
[3] Liam J Bannon and Pelle Ehn. 2012. Design: design matters in Participatory
Design. In Routledge international handbook of participatory design. Routledge,
57–83.
[4] Gary R Bettger and Timothy J Pearson. 1989. Accommodating deaf and hard-of-
hearing persons on public transportation systems in Massachusetts. Transporta-
tion research record 1209 (1989), 16–18. http://dx.doi.org/
Yeon Soo Kim, Sunok Lee, and Sangsu Lee
[5] David Braddock, Mary C Rizzolo, Micah Thompson, and Rodney Bell. 2004.
Emerging technologies and cognitive disability. Journal of Special Education
Technology 19, 4 (2004), 49–56.
[6] Matthew W Dye, Peter C Hauser, and Daphne Bavelier. 2008. Visual attention
in deaf children and adults. Deaf cognition: Foundations and outcomes (2008),
250–263.
[7] Richard S Hallam and Roslyn Corney. 2014. Conversation tactics in persons with
normal hearing and hearing-impairment. International journal of audiology 53, 3
(2014), 174–181.
[8] Shawn Lawton Henry, C Law, and K Barnicle. 2001. Adapting the design process
to address more customers in more situations. In UPA (Usability Professionals’
Association) 2001 Conference.
[9] Ilpo Koskinen, Katja Battarbee, and Tuuli Mattelmeaki. 2003. Empathic design.
IT press.
[10] Raja S. Kushalnagar and Christian Vogler. 2020. Teleconference Accessibility and
Guidelines for Deaf and Hard of Hearing Users. In The 22nd International ACM
SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece)
(ASSETS ’20). Association for Computing Machinery, New York, NY, USA, Article
9, 6 pages. https://doi.org/10.1145/3373625.3417299
[11] Susan Lacke. [n.d.]. Do all deaf people use sign language? https://www.
accessibility.com/blog/do-all-deaf-people-use-sign-language
[12] Petrus Ng and Angela Tsun. 1999. Work Experience of People Who are Deaf or
Hard of Hearing in Hong. 32, 3 (1999), 35–49.
[13] Kiri O’Brien. 2020. How coronavirus is making life harder for deaf work-
ers. https://www.drutherssearch.com/how-coronavirus-is-making-life-harder-
for-deaf-workers/
[14] Sushil K Oswal. 2014. Participatory Design : Barriers and Possibilities. (2014),
14–19.
[15] John Oswald. 2020. I’m deaf, and this is what happens when I get on a
zoom call. https://www.fastcompany.com/90565930/im-deaf-and-this-is-what-
happens-when-i-get-on-a-zoom-call
[16] Otter.ai. 2021. Otter.ai. https://Otter.ai.
[17] Yi-Hao Peng, Ming-Wei Hsi, Paul Taele, Ting-Yu Lin, Po-En Lai, Leon Hsu,
Tzu-chuan Chen, Te-Yen Wu, Yu-An Chen, Hsien-Hui Tang, and Mike Y. Chen.
2018. SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-
of-Hearing People in Group Conversations. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems (Montreal QC, Canada)
(CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–10.
https://doi.org/10.1145/3173574.3173867
[18] Filipa M Rodrigues, Ana Maria Abreu, Ingela Holmström, and Ana Mineiro. 2022.
E-learning is a burden for the deaf and hard of hearing. Scientifc Reports 12, 1
(2022), 9346. https://doi.org/10.1038/s41598-022-13542-1
[19] Jazz Rui Xia Ang, Ping Liu, Emma McDonnell, and Sarah Coppola. 2022. “In This
Online Environment, We’re Limited”: Exploring Inclusive Video Conferencing
Design for Signers. In Proceedings of the 2022 CHI Conference on Human Factors in
Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing
Machinery, New York, NY, USA, Article 609, 16 pages. https://doi.org/10.1145/
3491102.3517488
[20] Matthew Seita, Sooyeon Lee, Sarah Andrew, Kristen Shinohara, and Matt Huen-
erfauth. 2022. Remotely Co-Designing Features for Communication Applications
Using Automatic Captioning with Deaf and Hearing Pairs. In Proceedings of the
2022 CHI Conference on Human Factors in Computing Systems (New Orleans,
LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA,
Article 460, 13 pages. https://doi.org/10.1145/3491102.3501843
[21] Karin Slegers, Pieter Duysburgh, Helma van Rijn, and Niels Hendriks. 2012. Par-
ticipatory Design for Users with Impairments Afecting Cognitive Functions and
Communication Skills. In Proceedings of the 12th Participatory Design Conference:
Exploratory Papers, Workshop Descriptions, Industry Cases - Volume 2 (Roskilde,
Denmark) (PDC ’12). Association for Computing Machinery, New York, NY, USA,
141–142. https://doi.org/10.1145/2348144.2348190
[22] Google Slides. 2021. Google Slides. https://www.google.com/slides/.
[23] Louis Turmel. 2020. Meeting statistics - stats on costs & time spent in meetings.
https://bettermeetings.expert/meeting-statistics/
[24] Nancy Tye-Murray, Suzanne C Purdy, and George G Woodworth. 1992. Reported
use of communication strategies by SHHH members: Client, talker, and situational
variables. Journal of Speech, Language, and Hearing Research 35, 3 (1992), 708–717.
[25] Elissa Weeden and Sharon Mason. 2020. An Initial Survey of Deaf and Hard-of-
Hearing Student Use of a Composite Screen Solution Utilizing Web Conferencing
Software. In Proceedings of the 21st Annual Conference on Information Technol-
ogy Education (Virtual Event, USA) (SIGITE ’20). Association for Computing
Machinery, New York, NY, USA, 46–49. https://doi.org/10.1145/3368308.3415459
[26] Zoom. 2021. Video conferencing. http://zoom.us/.
 Co-designing a Bespoken Wearable Display for People with
Dissociative Identity Disorder
Patricia Piedade∗
ITI/LARSyS, Instituto Superior
Técnico, Universidade de Lisboa
Portugal
patricia.piedade@tecnico.ulisboa.pt
Nikoletta Matsur∗
ITI/LARSyS, Instituto Superior
Técnico, Universidade de Lisboa
Portugal
nikoletta.matsur@tecnico.ulisboa.pt
Catarina Rodrigues∗
ITI/LARSyS, Instituto Superior
Técnico, Universidade de Lisboa
Lisbon, Portugal
catarina.rebelo.rodrigues@tecnico.ulisboa.pt

Francisco Cecilio∗ Afonso Marques∗ Rings of Saturn

ITI/LARSyS, Instituto Superior ITI/LARSyS, Instituto Superior Lisbon, Portugal
Técnico, Universidade de Lisboa Técnico, Universidade de Lisboa
Portugal Lisbon, Portugal
franciscocecilio@tecnico.ulisboa.pt afonso.m.marques@tecnico.ulisboa.pt

Isabel Neto∗ Hugo Nicolau

INESC-ID, ITI/LARSyS, Instituto
Superior Técnico, Universidade de
Lisboa
Lisbon, Portugal
isabel.neto@tecnico.ulisboa.pt
ITI/LARSyS, Instituto Superior
Técnico, Universidade de Lisboa
Lisbon, Portugal
hugo.nicolau@tecnico.ulisboa.pt

Figure 1: Stages of the co-design process of creating a bespoken technology with a person living with DID. Left: unfnished
storyboard that the co-designer had to fll-in (last three slots). Middle: paper prototype used during the experience prototyping
activity and MoSCoW matrix resulting from the card sorting activity. Right: fnal concept, a pendant and necklace that displays
a representation of the fronting personality and pronouns.
ABSTRACT display (WhoDID) to facilitate in-person social interactions. The
Dissociative Identity Disorder (DID) is characterized by the pres- prototype aims to be used as a necklace and enable the user to
ence of at least two distinct identities in the same individual. This make their fronting personality visible to others. Thus, facilitating
paper describes a co-design process with a person living with DID. social encounters or sudden changes of identity. We refect on the
We frst aimed to uncover the main challenges experienced by the design features of WhoDID in the broader context of supporting
co-designer as well as design opportunities for novel technolo- people with DID. Moreover, we provide insights on co-designing
gies. We then engaged in a prototyping stage to design a wearable with someone with multiple (sometimes conficting) personalities
regarding requirement elicitation, decision-making, prototyping,
∗ Authors contributed equally to this research. and ethics. To our knowledge, we report the frst design process
Permission to make digital or hard copies of part or all of this work for personal or with a DID user within the ASSETS and CHI communities. We aim
classroom use is granted without fee provided that copies are not made or distributed to encourage other assistive technology researchers to design with
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
DID users.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s). CCS CONCEPTS
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550369 • Human-centered computing → Accessibility.
ASSETS ’22, October 23–26, 2022, Athens, Greece
KEYWORDS
Dissociative Identity Disorder, Co-design, Wearable, Social Encoun-
ters
ACM Reference Format:
Patricia Piedade, Nikoletta Matsur, Catarina Rodrigues, Francisco Cecilio,
Afonso Marques, Rings of Saturn, Isabel Neto, and Hugo Nicolau. 2022.
Co-designing a Bespoken Wearable Display for People with Dissociative
Identity Disorder. In The 24th International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece.
ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3517428.3550369
1 INTRODUCTION
Dissociative Identity Disorder (DID), formerly known as multiple
personality disorder, is a clinical condition involving a “disruption of
identity characterized by two or more distinct personality states"[1].
It consists of a sudden and temporary discontinuity in the sense of
agency and self [1]. DID is a complex and controversial diagnosis
[2, 6], often associated with childhood trauma and abuse [4], that
allows the host, who may be unable to deal with overwhelming
stress, to pass control of their body (i.e., switch) to an alternative
identity, the alter. The term ‘alter’ is often used to describe other
identities within the body. Alters can difer “in afect, behavior,
cognition, consciousness, memory, perception, and sensory-motor
functioning"[1].
People living with DID experience daily challenges such as
stigma, fear, and maintaining relationships [9]. Although previ-
ous work has proposed adopting a lens from disability studies and
incorporating the expertise of assistive technology researchers to
support people with psychosocial disabilities [7], to the best of
our knowledge, there is no previous research exploring the role of
technology in supporting people living with DID.
In this paper, we describe the process of designing a bespoke
technology to enhance social interactions by engaging in a co-
creation process with a person living with DID. We contribute with
1) a report on the design process, which ofers preliminary practical
and ethical considerations to design with people living with DID;
and 2) the design of a wearable display (WhoDID) to be used by
the sixth author of this paper.
2 THE DESIGN OF WHODID
This section describes two 2-hour in-person co-design sessions with
a person living with DID. We took inspiration from the Double
Diamond design process by frst aiming to understand and defne
the challenge and then develop and test diferent solutions. The
co-designer is a fve-personality DID system (i.e., group of host
and alters). The host, Onyx, is 20-year-old and has been diagnosed
with DID for a year. They share their body with Ana, a 20-year-
old protector (alter who handles stressful situations), Oliver, a 17-
year-old caretaker (alter responsible for assuring the physical and
mental needs of the body), Thalia, a 26-year-old caretaker, and
Charlotte, a 7-year-old alter. This system lives in relative harmony,
communicating and making decisions collectively.
2.1 Exploring the Problem Space
We began our co-design process by investigating the daily struggles
of living with DID. We conducted a semi-structured interview
Piedade, et al.
in a relaxed environment to create a safe space for learning and
getting to know our co-designer. For added comfort, only two team
members were present: an interviewer and a notetaker.
We inquired about demographics, DID, and technology. We
started by explaining the project’s goal and then used directed sto-
rytelling [3] to harness detailed descriptions of our co-designer’s
experiences. We then asked follow-up open-ended questions.
This session enlightened us on many of the challenges faced by
our co-designer. Social encounters were one of the main emergent
discussion topics. The co-designer expressed awkwardness and
embarrassment in communicating which alter is fronting (i.e., in
charge of the body). Moreover, having people misidentify alters was
a source of frustration and misunderstandings. We thereby focused
on facilitating in-person communication through externalizing who
in the system is fronting. Our co-designer showed great interest in
using a wearable device to indicate the fronting alter.
2.2 Exploring the Solution Space
Having a clear problem statement, we conducted a co-design work-
shop to craft the details of the wearable. We documented this session
through video recordings, photos and notetaking.
We crafted two unfnished storyboards representing challeng-
ing in-person communication scenarios referenced by our co-designer
(Figure 1). The frst storyboard illustrated the co-designer hanging-
out with a friend and being disrupted by a switch. The second
storyboard portrayed meeting a friend on the street when the host
was not fronting. We discussed how we might facilitate these situa-
tions using technology and asked the co-designer to fnish flling
both storyboards accordingly.
We also conducted a card-sorting activity using the MoSCoW
prioritization [8] technique to understand the co-designer’s views
on collectively brainstormed features catering to their needs, deter-
mining which should be included (or excluded) in the prototype.
The cards represented the features and the co-designer categorized
them into one of the four MoSCoW priority groups: Must have,
Should have, Could have, and Will not have. The resulting MoSCoW
matrix informed the prototyping of a wearable necklace display
and its companion mobile app - WhoDID.
To simulate the use of WhoDID, we crafted a low-fdelity pro-
totype of our wearable using cardboard and post-its (Figure 1).
We recruited a recent friend of the host, unacquainted with the
whole system, to conduct an experience prototyping activity [5].
We simulated a social interaction between the co-designer and
their friend, during which a switch occurred. When the co-designer
touched the "screen", a Wizard would switch the post-it to display
the now fronting alter. During this session, an actual switch oc-
curred. This helped solidify the robustness of the WhoDID concept,
as we avoided mentioning the switch, yet the friend was aware
with whom they were interacting.
Finally, to complement the wearable, we prototyped a high-
fdelity companion app using Figma, which enabled the set up
of alters’ profles and control of the appearance of the wearable
display. We demoed the app and discussed its features with the
co-designer.
Co-designing a Bespoken Wearable Display for People with Dissociative Identity Disorder
3 FINDINGS AND DISCUSSION
In this section, we highlight the main fndings regarding the design
of WhoDID and lessons learned about designing with someone
living with DID.
3.1 WhoDID Design
Supporting In-person Social Interactions. One of the co-designer’s
main challenges was their difculty conveying the fronting al-
ter after a switch without awkwardness and unpleasant feelings.
While conveying a switch through digital technologies is relatively
straightforward using existing features of communication plat-
forms (e.g., social media profles, PluralKit Discord bot), there is not
a similar tool for in-person communication. Thus, the co-designer
explicitly focused on this challenge and fnding new ways of en-
hancing in-person communication by easing the externalization of
the alter in charge.
Designing a Wearable Display. We explored several alterna-
tives of wearable form factors (e.g., bracelet, watch, rings) for dis-
playing the fronting alter. The co-designer strongly preferred a
pendant in a necklace due to its fexibility to be highly visible or
hidden at will. During the experience prototyping activity, we ob-
served that it was natural for the co-designer to perform swipe
gestures on the display due to its familiarity and robustness to
accidental taps.
Privacy Concerns. The main safety concern emerging during
the design process was related to using biometric sensing to detect
a switch automatically. While this feature could reduce the inter-
action with the device and seemly transition between alters, the
co-designer reached a consensus that each alter should have the
freedom to choose when to identify themselves. This identifcation
process should not attract attention from onlookers; fashing lights
or sound cues should not be used.
Companion App Features. Being able to customize the visual
representation of each alter is a key feature. WhoDID supports the
personalization of background color and image. Interestingly, the
co-designer preferred to display the alter’s pronouns over their
names as a balance to provide enough information for a respectful
social interaction without disrupting the norms of meeting someone
for the frst time (i.e., asking for their name). A central concern
was the possibility of a given alter making destructive changes to
another alter’s profle (e.g., deleting it). As a risk mitigation measure,
WhoDID supports an edit window of 15 days, which allows the
user to undo any action. Although this feature does not prevent
destructive behaviors, it is a coping mechanism. We believe these
challenges are fertile ground for usable privacy/security research.
Lastly, to tackle the awkwardness when a friend of the system
frst encounters an unknown alter by chance, WhoDID features
proximity notifcations, allowing the system to choose from trusted
individuals who should be informed of who is fronting when facing
a possible chance encounter.
3.2 Lessons Learned from the Design Process
Wait for Consensus As the system contains multiple alters within
the body, our co-designer expressed that it is vital to allow them
enough time to reach a consensus when decision-making, as dif-
ferent alters may have diferent opinions. Thus, design processes
ASSETS ’22, October 23–26, 2022, Athens, Greece
must account for time (possibly even over several days) and allow
for “internal" discussions.
Consider Diversity Throughout the design process, our co-
designer stressed the diversity of DID systems, from the number
of alters to the ease of communication among alters. Though we
focused on a solution tailored to our co-designer, the fnal design
is highly customizable in terms of the number of alters, displays.
For instance, from our co-design process until the time of writing
this paper, a new alter appeared within the co-designer system.
When discussing initial research with our co-designer, even in
simple aspects such as DID-specifc terminology, we found that
our co-designer had their unique perspective and preferences. The
diversity in the experiences of people with DID and in the informa-
tion relating to this diagnosis calls for added double-checking of all
researched information with the specifc co-designer.
Ponder Ethics Triggering switches are one of the main ethi-
cal concerns when designing with individuals with DID. Switches
produce both physical and mental changes, possibly even pain.
Thus, there are critical implications for designers trained in partic-
ipatory practices of including stakeholders in the design process.
Participatory activities that require decision-making, discussion,
and consensus (e.g., prioritizing features) can be highly problematic
as they can trigger several switches. Moreover, testing prototypes
with the various alters may require a longer time frame and the use
of asynchronous/remote methods to leverage naturally occurring
switches rather than trigger them.
4 CONCLUSION AND FUTURE WORK
We have presented the co-design of a bespoken wearable display
to support in-person social interactions for people with DID. We
highlight fndings related to the design of WhoDID and the design
process itself. Results show that to co-design with people with
DID, it is not feasible to make decisions based on opinions from one
alter; we should aim for consensus of the diferent alters and double-
check whether all voices are heard. We also refect on the ethical
concerns of this activity as switches impact the individual comfort
and well-being. Future work will develop a functional prototype
and evaluate its efectiveness in supporting our co-designer’s needs
in the wild.
ACKNOWLEDGMENTS
This work was supported through FCT’s projects UIDB/50009/2020
and SFRH/BD/06452/2021.
REFERENCES
[1] American Psychiatric Association. 2013. Diagnostic and statistical manual of mental
disorders: DSM-5. Vol. 5. American psychiatric association Washington, DC.
[2] David Blihar, Elliott Delgado, Marina Buryak, Michael Gonzalez, and Randall
Waechter. 2020. A systematic review of the neuroanatomy of dissociative identity
disorder. European Journal of Trauma & Dissociation 4 (9 2020), 100148. Issue 3.
https://doi.org/10.1016/j.ejtd.2020.100148
[3] Shelley Evenson. 2006. Directed storytelling: Interpreting experience for design.
Design Studies: Theory and research in graphic design (2006), 231–240.
[4] Eva Irle, Claudia Lange, Godehard Weniger, and Ulrich Sachsse. 2007. Size abnor-
malities of the superior parietal cortices are related to dissociation in borderline
personality disorder. Psychiatry Research: Neuroimaging 156 (11 2007), 139–149.
Issue 2. https://doi.org/10.1016/j.pscychresns.2007.01.007
[5] Ken Keane and Valentina Nisi. 2014. Experience Prototyping. , 224-237 pages.
https://doi.org/10.4018/978-1-4666-4623-0.ch011
ASSETS ’22, October 23–26, 2022, Athens, Greece
[6] M. M. McAllister. 2000. Dissociative identity disorder: a literature review. Journal
of Psychiatric and Mental Health Nursing 7 (2 2000), 25–33. Issue 1. https://doi.
org/10.1046/j.1365-2850.2000.00259.x
[7] Kathryn E Ringland, Jennifer Nicholas, Rachel Kornfeld, Emily G Lattie, David C
Mohr, and Madhu Reddy. 2019. Understanding Mental Ill-Health as Psychosocial
Disability: Implications for Assistive Technology. The 21st International ACM
SIGACCESS Conference on Computers and Accessibility, 156–170. https://doi.org/
Piedade, et al.
10.1145/3308561.3353785
[8] Kelly Waters. 2009. Prioritization using moscow. Agile Planning 12 (2009), 31.
[9] Melissa Zeligman, Jennifer H. Greene, Gulnora Hundley, Joseph M. Graham, Sarah
Spann, Erin Bickley, and Zachary Bloom. 2017. Lived Experiences of Men With
Dissociative Identity Disorder. Adultspan Journal 16 (10 2017), 65–79. Issue 2.
https://doi.org/10.1002/adsp.12036
 Co-designing the automation of Theatre Touch Tours
Alexandra Tzanidou Sami Abosaleh Vasilis Vlachokyriakos
a.tzanidou2@newcastle.ac.uk a.h.s.abosaleh2@newcastle.ac.uk Open Lab, Newcastle University
Open Lab, Newcastle University Open Lab, Newcastle University Newcastle upon Tyne, UK
Newcastle upon Tyne, UK Newcastle upon Tyne, UK Vasilis.Vlachokyriakos1@newcastle.ac.uk

Figure 1: Co-design WS - Phase 1: Creating Tangible Objects for the performance

ABSTRACT
This paper presents the fve-step co-design process of a prototype
system for the automation of Touch Tours (TTs) service in theatre
performances.
By grounding our research on the Social Model of Disability, we
motivate our initial design by examining the outputs of our col-
laboration with the frst inclusive professional theatre ensemble in
Greece. We present our refections on the discourse created during
the co-design process, and we contribute some initial fndings of
the process.
CCS CONCEPTS
• Human-centered computing → Accessibility technologies;
• General and reference → Design; • Applied computing →
Performing arts;
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550370
KEYWORDS
Co-design, Inclusion, Accessibility, Social Model of Disability, Touch
Tours
ACM Reference Format:
Alexandra Tzanidou, Sami Abosaleh, and Vasilis Vlachokyriakos. 2022. Co-
designing the automation of Theatre Touch Tours. In The 24th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22),
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 5 pages.
https://doi.org/10.1145/3517428.3550370
1 INTRODUCTION
Individuals who are experiencing disabilities have not enjoyed equal
access to traditional theatre. Modern theatre has challenged this,
and is trying to accommodate access with equity to the theatre
experience. As a part of an ongoing research project that examines
inclusion and accessibility in theatre based on the social model of
disability [17], this paper focuses, specifcally, on the service of
Touch/Sensory Tours.
Touch/Sensory Tours (TTs) services are ofered by theatre venues
that ofer the service of audio description during the performances
[16], to create stimuli that substitute for the visual ones for the
blind or visually impaired (B/VI) members of the audience, through
the sense of touch [19]. Although the value of the TTs is recognised
ASSETS ’22, October 23–26, 2022, Athens, Greece Tzanidou, Abosaleh, Vlachokyriakos.

[19, 20],the availability of the service is not common according to lack of visibility of such work in Greece, where this research has
our research, mostly because of organisational and timing reasons taken place. While the co-design process is still ongoing, this paper
that we will briefy mention. presents the initial stages of a co-design process with a community
of experts in inclusive theatre.
2 RELATED WORK
This work approaches accessible design based on the social model 3.1 Ethnographic Research
of disability (SMoD) as described by Oliver [17].Oliver’s model The idea to automate TTs services was one of the outcomes of the
states that ‘impairment’ is dysfunction, while ‘disability’ is the lim- 18-month ethnographic research engagement of the frst author
itation created by the dysfunction because society fails to provide with the THEAMA ensemble. The scope of those engagements was
the infrastructure needed to include people with dysfunctions. Ac- focused on the observation and the understanding of the inclusive
cording to this distinction, the problem is created by society, which methodologies and patterns that the group is following, going a
emphasises the weaknesses of people with dysfunctions [16, 17]. In step outside of the traditional HCI comfort zone.
this work, inclusion is defned based on the disabled rights move-
ment’s slogan ‘Nothing about us, without us’, which requests the 3.2 Focus Group (FG) - Understanding
inclusion of disabled people in all the steps of the decision-making co-designers needs
process [10]. This slogan, in our case, is interpreted as the inclusion
Our initial step was to share our fndings and ideas about the TTs’
of disabled people in all the steps of the design process.
automation with the director of the group, who saw our idea as an
Inclusive theatre is defned as the form of artistic expression that
opportunity to enhance the accessibility services already provided
includes artists of all colours, genders, races, religions, nationalities
by THEAMA ensemble. Thus, we initiated this process by discussing
and ages who are experiencing disability or not, while having as its
the topic with the members of THEAMA. They shared with us their
main ideological approach the equal participation and integration of
thoughts, perspectives and ideas about introducing such a service
all people without exclusions in art and consequently in society [6,
in Greece. The most important factors of this focus group were
21]. The inclusive theatre approach ofers inclusion in participation,
that 1) all the participants (fve professional actors – referred to as
and furthermore, aims to enhance the accessibility ofered on and of
FGP1–5) knew and had already experienced theatre TTs at least
the stage [2]. Theatre accessibility includes all the factors connected
in one performance as spectators; and 2) they were not informed
to physical and content accessibility [3].
about our idea of incorporating the TTs’ service in their work,
which ofered us the opportunity to collect their very frst reactions
2.1 Disability - The Greek Context
and thoughts.
Accessibility is a painful issue in Greece, due to societal barriers.
The lack of funding and relevant legislation prevent inclusion with
equity and independent living of people experiencing disability
[5, 14, 18]. The Hellenic Greek parliament voted for the adoption
of the United Nations Convention on disabled rights in September
2017. This vote took place just a few months after the ‘able-bodied’
requirement for access to tertiary-level drama education by the
Greek Ministry of Culture was removed in February 2017 [1].

2.2 Our partners THEAMA & ISON

To explore our research questions, based on desk research for in-
clusive theatre groups supporting the social model of disability in
Greece, we chose to work with the THEAMA ensemble, founded
in 2010 by Vasilis Oikonomou 1 . THEAMA’s target was to fll the
void that existed in the sector of professional training and the
career-oriented approach for people experiencing disability who
are actively involved in the performing arts. ISON 2 is the inclusive
education workshop founded by THEAMA and the inclusive dance
group EXIS.
3 CO-DESIGN
In this project, we employ a constructivist approach towards the co-
designing of systems to support inclusion in theatre. The existence
and use of TTs services in theatre were unknown to the researchers
at the beginning of this project (other than using touch to enhance
accessibility in museums or art installations [4, 11, 13]), due to the
1 https://theamatheater.gr/en/intro/
2 https://ison.com.gr/
Figure 2: Pre-workshop activity package
3.3 Pre-workshop Activity (PWA)
For the PWA, we prepared ten paper bags with materials connected
to the senses and experiences discussed by the participants of the
FG. Those materials allowed the participants to create artefacts for
the workshop. The bags were distributed to actors of THEAMA
and students of ISON who had an interest in the project. A fyer
explaining the scope and the description of the activity, along with
our contact information, was included. On the outer side of the
bag, we placed a sticker with a QR code, which contained a link
to an online version of the fyer, to be leveraged by screen reader
users. The phrase ‘Scan here’ was written around the QR code in
English, Greek, English Braille and Greek Braille, creating in this
way a ‘Braille frame’ around the QR code.
Co-designing the automation of Theatre Touch Tours ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 3: The frst prototype developed for TTs automation.

3.4 TTs Research Workshop - Research

through making together
The activity was aimed at the people who received the PWA bag,
hoping to gather the perspectives of the co-designers. This was a
double-scoped workshop, aiming to collect data regarding the im-
portant elements of the TTs from the point of view of the artists and
the diferent ways of communicating them, but also to collectively
observe the benefts we could get out of the procedure and the expe-
rience [7]. In the workshop, nine people participated (referred to as
WP1–9); two of them were B/VI. We frst held a discussion on our
previous work and fndings on TTs and the participants presented
and communicated their creations. Afterwards, we refected on the
previous parts of the workshops and the way that the sense of touch
enriched our experience; while in the fnal part, the participants
were expected to experience and refect on the prototype.

3.5 Prototype System

The prototype system was developed based on data gathered dur-
ing the FG, along with prior related work [11]. Thus, the system
combined diferent elements to provide a diversity of stimuli to
the participants. The prototype system consisted of a 3D printed
bust of Einstein, a barber’s mannequin head, a scarf, two candles
of diferent fragrances, and a 3D diorama of the hypothetical stage
with small-scale 3D printed objects in it. All the aforementioned
elements were connected to a Raspberry Pi with the help of con-
ductive ink. Each of the elements was connected to a specifc piece
of text that could be read through the printed text, through a smart-
phone using the QR code or through a Bluetooth speaker when the
corresponding element was touched.
4 ANALYSIS
From the engagements mentioned above, we collected qualitative
data in the form of audio recordings from interviews and workshops
that we later transcribed, in combination with ethnographic notes.
To analyse such a corpus of data, we used Braun and Clarke’s
thematic analysis[8]; we initially coded the data descriptively (and
inductively) [9] and we later created categories of codes and formed
initial themes refectively, in respect to the social model of disability
(and thus deductively).
Figure 4: Performing and experiencing the prototype during
the fnal part of the workshop
4.1 Reasons for Incorporating TTs and
Prerequisites
Our initial aim was to understand if the automation of such a ser-
vice, usually conducted by human professionals, would make sense.
During the FG, [P1] and [P3], who have also directorial duties
within the team, mentioned instances of deaf, hard of hearing and
B/VI people contacting them in the past, and asking them for short
descriptions of the characters or other materials that could help
them have a better theatre experience. We also learned about the
importance of bringing the spectator into a specifc experience, and
the connection that touch and the incorporation of other senses
have with experience creation. As [P2] expressed, “If the actor wears
a winter coat and the spectator touches a piece of the coat, this au-
tomatically means [for them] winter, cold ...”. All the participants
ASSETS ’22, October 23–26, 2022, Athens, Greece
shared the same thoughts on the introduction of TTs, and the signif-
icant added value that TTs would bring to the accessibility services
currently ofered. However, three of them were particularly worried
about letting other people touch them, their clothes or their faces
for various reasons, including the fear of COVID [P4], or the fact
that some of the actors might feel upset due to such close contact
with strangers [P2, P5]. Finally, all the actors agreed that the last
hour before the show is important for them to calm down and
concentrate, while it is also the time in which the TTs usually take
place.
4.2 False Understandings – No need for
accuracy
Throughout the workshop, the two groups chose to name them-
selves based on performances of their preference, Troades and
Macbeth. Team Troades chose to create objects that the actors held
while performing, and parts of their costumes, and provide seman-
tic meanings for their creations. Team Macbeth created miniature
dolls that they held while they were performing together. When
asked about their choices and the procedure they followed, the
actors told us that, in art, fdelity or accuracy are not important in
the creation of objects. Thus, we conclude that, in theatre, the right
approach is the utilisation of the accessibility services experts for
the transfer of semiotics and meanings, and triggering the imag-
ination rather than the image. As [WP1] notes: [we should] “use
stimuli to let imagination free, the description shouldn’t reveal the
plot, we watch the same play, we don’t see the same thing, the details
frame the imagination.”
4.3 Experience Enhancement, Activism
Based on the notion of creating stimuli rather than providing details,
participants noticed that our representations were based on realistic
details and expressed that they might not choose this approach,
now that they knew that they could prepare objects by themselves
while rehearsing. As [WP3] said, a service like this would beneft
all the people involved in a performance as “Some people have this
special ability to connect tangible objects with their own personal
experiences and create new meanings out of them”.
While experiencing the prototypes, there was a lot of discourse
around them and their connection with the activist approach of the
group. As they explained, a big priority for them is to make a state-
ment by incorporating this accessibility service. Their approach to
using accessibility services is twofold: for practical reasons (theatre
access with equity), as well as to bring visibility to an issue, as
“Accessibility is not a thing of our everyday life, thus activism is a
need right now” [WP1].
5 FUTURE WORK
In the next steps of this research, we will focus on the understanding
of stimuli creation through technology, respecting the restrictions
that can be created by the visual culture [11]. We plan to do so
by following a reverse commissioning process (examples of such
a process is also talked about in [12, 15]). By reverse commission-
ing we refer to the community (in this case the inclusive theatre
group) being put in the position of ‘commissioning’ research and
design/development work from the university. This reverses the
Tzanidou, Abosaleh, Vlachokyriakos.
conventional power relations between academia and community
(or ‘research participants’), by giving control to the community to
tell us what they need to know or make. In that sense, the knowl-
edge we collected from ethnographic and participatory work with
inclusion experts (the community) will be used to start conversa-
tions with HCI and design experts about what can be made, what
the limits of current technology are and how we can creatively con-
fgure it. Then, these conversations and design ideas will be brought
back to the community to adjust and confgure. As such, we, as the
researchers involved in both community and academia, have the
role of mediating and facilitating inclusive design across expertise
(inclusion and technology design) and space (remote collaboration
across the UK and Greece) [10].
ACKNOWLEDGMENTS
This research was funded by the EPSRC CDT in Digital Civics
(EP/L016176/1). The authors thank our collaborators THEAMA and
ISON for the time and expertise they devoted to this research study.
REFERENCES
[1] 2019. https://www.disabilityartsinternational.org/resources/greece-country-
profle/
[2] Talleri Adkins Mcrae and Mickey Rowe. 2019. The Future of Theatre is Accessible.
https://howlround.com/series/future-theatre-accessible
[3] Austin Allison. 2018. Accessibility in the Theatre. Honors Thesis. Ouachita Baptist
University.
[4] Vassilios S. Argyropoulos and Charikleia Kanari. 2015. Re-imagining the mu-
seum through “touch”: Refections of individuals with visual disability on
their experience of museum-visiting in Greece. Alter 9, 2 (2015), 130–143.
https://doi.org/10.1016/j.alter.2014.12.005
[5] Stathis Balias and Pandelis Kiprianos. 2014. Disability in Greece: Social Perception
and Educational Policies. The Review of Disability Studies: An International Journal
1 (2014).
[6] Stephanie Barton Farcas. 2017. Disability and Theatre (1 ed.). Routledge.
[7] Cynthia L. Bennett, Burren Peil, and Daniela K. Rosner. 2019. Biographical
Prototypes: Reimagining Recognition and Disability in Design. In Proceedings
of the 2019 on Designing Interactive Systems Conference (San Diego, CA, USA)
(DIS ’19). Association for Computing Machinery, New York, NY, USA, 35–47.
https://doi.org/10.1145/3322276.3322376
[8] Virginia Braun and Victoria Clarke. 2006. Using thematic
analysis in psychology. Qualitative Research in Psychology 3,
2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
arXiv:https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa
[9] Virginia Braun and Victoria Clarke. 2021. Can I use TA? Should I use
TA? Should I not use TA? Comparing refexive thematic analysis and
other pattern-based qualitative analytic approaches. Counselling and Psy-
chotherapy Research 21, 1 (2021), 37–47. https://doi.org/10.1002/capr.12360
arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/capr.12360
[10] James I. Charlton. 1998. Nothing About Us Without Us: Disability Oppression
and Empowerment. University of California Press. https://doi.org/doi:10.1525/
9780520925441
[11] Han-Xing Chen and Wen Huei Chou. 2020. Exploratory Design Research
for the Blind and Visually Impaired Visitor in Exhibitions. The Design Jour-
nal 23, 3 (2020), 395–417. https://doi.org/10.1080/14606925.2020.1744257
arXiv:https://doi.org/10.1080/14606925.2020.1744257
[12] Andrew Garbett, Rob Comber, Edward Jenkins, and Patrick Olivier. 2016. App
Movement: A Platform for Community Commissioning of Mobile Applications.
Association for Computing Machinery, New York, NY, USA, 26–37. https://doi.
org/10.1145/2858036.2858094
[13] Gunnar Jansson, Massimo Bergamasco, and Antonio Frisoli. 2003. A new option
for the visually impaired to experience 3D art at museums: manual exploration
of virtual copies. Visual Impairment Research 5, 1 (2003), 1–12. https://doi.org/
10.1076/vimr.5.1.1.15973 arXiv:https://doi.org/10.1076/vimr.5.1.1.15973
[14] Lefkothea Kartasidou, Ioanna Dimitriadou, Elisavet Pavlidou, and Panagiotis
Varsamis. 2013. Independent living and interpersonal relations of individuals with
intellectual disability: The perspective of support staf in Greece. International
Journal of Learner Diversity and Identities 19, 1 (2013), 59–73.
[15] Kellie Morrissey, Andrew Garbett, Peter Wright, Patrick Olivier, Edward Ian Jenk-
ins, and Katie Brittain. 2017. Care and Connect: Exploring Dementia-Friendliness
Through an Online Community Commissioning Platform. In Proceedings of the
Co-designing the automation of Theatre Touch Tours
2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado,
USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA,
2163–2174. https://doi.org/10.1145/3025453.3025732
[16] Mala D Naraine, Margot R Whitfeld, and Deborah I Fels. 2018. Who’s devising
your theatre experience? A director’s approach to inclusive theatre for blind and
low vision theatregoers. Visual Communication 17, 1 (2018), 113–133. https://doi.
org/10.1177/1470357217727678 arXiv:https://doi.org/10.1177/1470357217727678
[17] Michael Oliver. 1990. Politics of disablement. Macmillan International Higher
Education.
[18] Maria Poli. 2020. Greece, Tourism and Disability. In Strategic Innovative Mar-
keting and Tourism, Androniki Kavoura, Efstathios Kefallonitis, and Prokopios
Theodoridis (Eds.). Springer International Publishing, Cham, 667–675.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[19] J.P. Udo and D.I. Fels. 2010. Enhancing the entertainment experience
of blind and low-vision theatregoers through touch tours. Disability &
Society 25, 2 (2010), 231–240. https://doi.org/10.1080/09687590903537497
arXiv:https://doi.org/10.1080/09687590903537497
[20] Margot Whitfeld and Deborah I. Fels. 2013. Inclusive Design, Audio
Description and Diversity of Theatre Experiences. The Design Journal
16, 2 (2013), 219–238. https://doi.org/10.2752/175630613X13584367984983
arXiv:https://doi.org/10.2752/175630613X13584367984983
[21] Roger Wooster. 2009. Creative inclusion in community theatre: a journey with
Odyssey Theatre. Research in Drama Education: The Journal of Applied Theatre
and Performance 14, 1 (2009), 79–90. https://doi.org/10.1080/13569780802655814
arXiv:https://doi.org/10.1080/13569780802655814
 Creating Personas for Signing User Populations: An
Ability-Based Approach to User Modelling in HCI
Amelie Nolte
amelie.nolte93@gmail.com
nolte@imis.uni-luebeck.de
Ergosign GmbH
Hamburg, Germany
Institute for Multimedia and Interactive Systems
University of Luebeck
Luebeck, Germany
Dieter Wallach
dieter.wallach@ergosign.de
Ergosign GmbH
Saarbruecken, Germany
ABSTRACT
Personas allow designers to build empathy towards users, avoiding
reliance on stereotypes or assumptions. When it comes to creat-
ing personas for people with disabilities, a common procedure is
to integrate impairments as dedicated characteristics. However,
when considering Deaf and Hard of Hearing (DHH) individuals
as target group, we argue that such an approach is incompatible
with the latter’s own ability-based identifcation concepts. Thus,
we propose a modifcation of personas for DHH users by defn-
ing them in terms of ability ranges and driving or keeping factors
within specifc contexts. Starting with assumption personas based
on previous research, we conducted focus groups with DHH in-
dividuals from which we derived the data and perspectives that
formed our modifcation. With this approach, we want to promote
an adequate representation of DHH users among designers while
also demonstrating an application of Ability-Based Design to HCI
methods.
CCS CONCEPTS
• Human-centered computing → User models.
KEYWORDS
Personas, Ability-Based Design, Ability-based user modelling, Deaf,
Hard of Hearing, Sign language users
ACM Reference Format:
Amelie Nolte, Karolin Lueneburg, Dieter Wallach, and Nicole Jochems.
2022. Creating Personas for Signing User Populations: An Ability-Based
Approach to User Modelling in HCI. In The 24th International ACM SIGAC-
CESS Conference on Computers and Accessibility (ASSETS ’22), October
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550364
Karolin Lueneburg
Ergosign GmbH
Berlin, Germany
Edinburgh College of Art
The University of Edinburgh
Edinburgh , United Kingdom
Nicole Jochems
jochems@imis.uni-luebeck.de
Institute for Multimedia and Interactive Systems
University of Luebeck
Luebeck, Germany
23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 6 pages. https:
//doi.org/10.1145/3517428.3550364
1 INTRODUCTION
Personas as a method within HCI-design [7, 8] ofer a way of ad-
dressing and refecting upon user needs, as the method aims to
"create empathic descriptions of the user" [39, p. 5]. As such, per-
sonas are intended to prevent designers from having to rely on their
own assumptions or stereotypes and, instead, see things from the
user’s perspective [17, 39]. A common procedure for applying this
approach to people with disabilities is to integrate medical impair-
ments as a dedicated characteristic. However, considering Deaf and
Hard of Hearing (DHH) sign language users, research shows that
they do not understand themselves to be impaired or disabled, but
instead view their Deafness as cultural identifcation [14, 18, 46].
Therefore, in order to achieve adequate representations of this tar-
get group, personas should refect their ability-based perspective—a
requirement, as we argue, that cannot be met by integrating im-
pairments but demands a shift in the creation of personas towards
a more explicit and dynamic modelling of abilities, contexts, and
infuential factors. Thus, this paper proposes a modifcation of the
persona method by defning users in terms of ability ranges and
driving or keeping factors within specifc contexts. The result is
a persona design for DHH sign language users that pursues an
ability-based perspective, as put forth by Wobbrock et al. [49, 50].
Although initially defned for the context of travelling, we believe
that our approach can be applied and adapted to other contexts or
target groups as needed. The contribution of this paper lies, there-
fore, not only in a novel perspective on the design of personas for
DHH users but in the creation of ability-based personas altogether.
2 RELATED WORK
Personas are proposed by previous research as one solution to
address the needs and requirements of either diverse disabilities
groups [16, 19–21, 27, 37, 44, 47] or specifc groups, e.g., elderly
users [5, 31, 41], dementia patients [24], or DHH individuals [22].
ASSETS ’22, October 23–26, 2022, Athens, Greece
A common approach among these studies in regard to creating
personas for specifc, accessibility-concerned target users is to in-
tegrate medical impairments into the otherwise unchanged per-
sona structure, i.e., by modelling aspects such as degree of vision
loss, diagnosed illnesses or hearing status. These aspects and, thus,
the resulting persona designs are captured in static, determined
descriptions—a procedure that is criticised by Edwards and col-
leagues [12, 13] as being insufciently representative for any type
of user group.
The relevance of this criticism in regard to DHH individuals is
highlighted by a variety of studies [1, 2, 10, 23, 29, 30, 34, 35, 45].
These address cultural and ethical aspects of working with or design-
ing for sign language users, emphasising repeatedly the dilemma
within DHH communities of being treated as disabled persons,
but not identifying as such, i.e., receiving rigid, infexible attribu-
tions that do not correspond to their own ability-based, diverse
self-concept. Recent approaches apprehend such gaps between
designers and users by demanding further consideration of the con-
cepts of empathy [3] and identity [33] within personas, but fail to
provide solutions that model the complexity of ability-based diver-
sity among sign language users. Microsoft [36] has also addressed
the demand for diversity in personas by developing the Persona
Spectrum method which ofers a ways of overcoming designer’s
ability bias by considering situational, temporary, and permanent
impairments. However, it associates the absence of certain abilities
as a defcit, i.e., as something caused by or linked to the individual
instead of its surrounding environments. Therefore, our approach
seeks to challenge this perspective on personas by building on the
Persona Spectrum as well as on the discussed fndings from pre-
vious research on DHH identifcation concepts and diversity and
applying an ABD perspective [49, 50] to the creation of personas
for DHH sign language users.
3 OUR APPROACH
To establish a frst understanding of the target group within the
team of our research project AVASAG 1 , we designed a common
assumption persona [43] based on the experiences and perspec-
tives ofered by the Deaf sign language experts within the team
while also considering the results of previous studies on the eval-
uation of sign language avatars with DHH users [11, 25, 26, 28].
This assumption persona contained a list of characteristics that
were grouped into tasks, goals, communication preferences, impair-
ments, pain points and social needs. When comparing our modeled
assumptions with the statements and discussions we witnessed in
the sign language accessible focus groups we conducted with DHH
individuals (n=10)2 , we realised that integrating the category of
impairment descriptions did not comply with the target group’s
self-concept, seeing that it attributed them with a "disabled" label
which they are determined to avoid and reject [29, 30, 34]. Thus, we
modifed our assumption persona according to the perspectives and
1 The research project AVASAG [4, 40, 48] focuses on the development of an algorithm
that automatically translates German text into German Sign Language with the trans-
lation being carried out by an avatar and the initial project scope focused on the topic
of travel information and services.
2 We conducted three focus group sessions, each with three to four native signers in
order to identify experienced barriers when travelling as a Deaf person and also when
using diferent technologies. A more detailed documentation of our conducted focus
groups and their results can be found in [48].
Amelie Nolte, Karolin Lueneburg, Dieter Wallach, and Nicole Jochems
insights gained throughout the focus group sessions and created
an ability-based persona for DHH sign language users in travelling
contexts as depicted in Figure 1 (see Appendix A). This persona con-
sists of three components which are integrated into one coherent
template3 :
(1) Persona type: The persona is characterised through a de-
scriptive label, an image, and a quote of something the persona
would say about themselves. We deliberately discarded the idea of
specifying demographic aspects (e.g., name, age, profession, marital
status) to minimise the risk of stereotypical assumptions connected
with specifc aspects. In addition, we chose a sketch as an image,
seeing that sketches have proven to reduce less stereotypes than
photographs [42]. In this way we formed the Enthusiastic Traveller
as persona type for whom travelling is a substantial and important
part of life and who is concerned mostly with being independent
and unrestricted in their travelling activities. We chose this type
based on our focus groups, which indicated a high passion for
travelling throughout all participants.
(2) Environmental drivers and keepers: The middle part of
the persona contains information on relevant contexts or environ-
ments, each complemented with factors (external or internal) which
either motivate (i.e., drivers) or block (i.e., keepers) the persona.
When considering technology design for travel information and
services, both, the travel context as well as social environments
should be analysed in order to understand where a design can ofer
support or compensation. For the Enthusiastic Traveller, we identi-
fed independence and adequate technology design as drivers when
travelling, with lack of information and untrained staf as respec-
tive keepers. Within social environments the persona is motivated
through personal relations in- and outside of DHH communities,
while being frustrated by hearing people’s lack of consideration,
their focus on hearing loss or the own fear of appearing helpless
infront of them.
(3) Ability ranges: The third component focuses on contextu-
ally relevant abilities of the users and how these are afected by
external factors. For this, we defned per ability a set of enabling
and disabling factors as well as a range indicator, positioning the
individual ability spectra of the persona. Relevant abilities of the
Enthusiastic Traveller thus contain 1) Sign language abilities, which
can be emphasised by skilled communication partners and good
lighting conditions, while being inhibited by being restricted on
fngerspelling or one-handed signing; 2) Spoken language abilities,
enabled by lip-reading, but disabled through a general dismissal of
using spoken languages instead of sign language; 3) Written lan-
guage abilities which are enhanced by short and well-structured
texts, but are reduced through unknown words or the own fear of
making mistakes; and 4) Technology profciency, enhanced through
well-structured content or video calls as an available channel of
communication, but weakened by visually cluttered interfaces.
Depending on the defned persona type, the types of factors
and their infuence on the individual ability ranges can vary. This
3 We chose to deviate from the classical narrative format [7, 39] and instead inserted
bullet points containing either full or partial sentences in order to focus on the essentials
while keeping the overall artifact as clear as possible—an approach that has also been
suggested by Constantine [6] who criticised the narrative of personas for their risk of
potentially distracting details, confusing the reader on "what matters and what does
not" [6, p. 505].
Creating Personas for Signing User Populations
parallelism of allowing individual diferences between persona
types, but also within one single persona while at the same time
ensuring an ability-based perspective is what we argue as the main
contribution of our modifcation.
4 DISCUSSION & FUTURE WORK
Persona design for DHH sign language users poses critical chal-
lenges to HCI designers in terms of capturing DHH perspectives
and needs in a way that corresponds to the latter’s own ability-
based self-concepts. Previous studies [13] demand that designers
must fnd ways to create personas that are not static or fnite, but
instead address the complex and diverse nature of users. We ar-
gue that our approach of an ability-based persona for DHH sign
language users takes a frst step into this direction by recognising
dynamics in the user’s abilities, understanding the latter to be dy-
namic areas infuenced by various factors in the physical, social,
and technological environment. Such an approach, besides comply-
ing to the user’s own ability-based identifcation, allows designers
to recognise opportunity areas for their solutions by integrating
and making use of enabling factors while avoiding disabling factors.
If one wants to e.g., integrate communication mechanisms into a
solution for the Enthusiastic Traveller, they can apprehend from the
persona design which communication abilities are pronounced and
how diferent abilities can be enhanced.
Even though we have drafted this persona for DHH sign language
users, we consider it possible to transfer and apply the template to
other target groups as well, pursuing, thereby, the goals set out by
the Ability-Based Design approach [49, 50]. This approach argues
that if a shift in the designer’s attention can be achieved and abili-
ties are considered in the early conceptual phases of development
processes, resulting solutions should be more accommodating to
such abilities of users. However, despite being cited as infuential
on various projects, the concept still lacks practical methods and
characterisations of how to implement its required focus, i.e., how
to perceive and model users’ abilities within an articulated design
process. Our modifed persona approach addresses this method-
ological gap by providing an example of how the general notion of
ABD can be applied to existing HCI design methods.
Although the persona itself was designed by two hearing project
members, a tight integration of feedback from members of DHH
communities into the design workfows was constantly ensured
(i.e., through the conduction of focus groups as well as regular
feedback sessions with DHH project team members). However, a
thorough evaluation of our fnalised approach has yet to be carried
out. Seeing that such a summative evaluation could address diferent
hypotheses, we argue that future research should explicitly address
the following issues:
Adequate representation format. Considering accurate represen-
tations of DHH sign language users as the major aim of this paper,
researchers should examine the degree to which a broad spectrum
of target group members feels adequately represented by the pro-
posed design format, especially in regard to the idea of not mod-
eling disabilities at all. Although our approach has been approved
by Deaf project members, potential drawbacks such as the risk
of diminishing lived experiences and knowledge, or oppressing
someone’s identifcation with and advocacy for a disability need
ASSETS ’22, October 23–26, 2022, Athens, Greece
yet be discussed with a larger sum of target group members. A
respective study should inspect both, the general characterisation
scheme (e.g., discarding medical aspects in terms of the hearing sta-
tus completely to focus, instead, on skills) and the selected abilities,
enabling factors, and disabling ones. Researchers examining the
latter (i.e., the selection of abilities) could, in addition, analyse the
question of the appropriate amount, degree, and types of abilities
within such a user representation. An adequate evaluation format
addressing such issues could include either a comparative study,
as suggested above, evaluating the novel concept in contrast to
existing approaches. Alternatively, only the novel approach could
be presented to participants, requiring feedback and opinions on its
core design and components. Results of both approaches, if indicat-
ing a preference for the novel design, could help the approach gain
momentum and receive support from within DHH communities
and, thereby, potentially also decrease the mistrust that DHH users
often show towards hearing designers or researchers [1, 34].
Impacts on stereotyping. In addition, seeing that the avoidance
of false or negative stereotyping is another aim of the approach
brought forth in this paper, future work should explore if and in
what ways this is actually achieved. Studies could apply a research
design similar to [42] or [32], who evaluated stereotypical percep-
tion of personas through the Stereotype Content Model (SMC)—a
framework based on research on social perception [9, 15] that di-
vides stereotyping into two dimensions: competence and warmth.
Regarding the evaluation of our persona design, researchers could
present participants with a specifc persona designed in both ways
(i.e., the novel design as proposed in this paper and existing designs
as defned by [7] or [38, 39]), in order to measure and compare
stereotypical perception of both designs.
Practical applicability. Finally, to evaluate the actual feasibility
of the approach, researchers should explore if and how designers
are able to apply the proposed persona design within projects.
Qualitative measurements could be used to compare the usage of
this novel persona design to traditional approaches [7, 38, 39]. Such
an evaluation format should aim to investigate the ways in which
both approaches either support or inhibit the design work, i.e., allow
designers to build empathy towards users and identify adequate,
optimised design solutions for their target group. For this, a variety
of testing scenarios should be considered so as to not only highlight
advantages or disadvantages of the novel approach but also identify
whether specifc types or scopes of projects are better suited with
one or the other approach.
Methodological perspectives. In addition to these evaluative chal-
lenges, future research should also examine the idea of applying
the novel approach to other target groups and contexts in order to
understand how much of its potential is motivated by the unique
nature of DHH community members and to what degree this poten-
tial can be exploited for other user groups as well. In this context, it
would also be helpful to discuss if or to which degree the approach
presented here can actually be defned as a (modifed) persona or
whether it should be labeled as a diferent method to better justify
or explain the deviations it takes from common persona models.
ASSETS ’22, October 23–26, 2022, Athens, Greece
5 CONCLUSION
In this paper, we have presented a novel approach to designing
ability-based personas for DHH sign language users. Having iden-
tifed the necessity of coming up with modelling solutions that
capture the ability-based identifcation concepts of DHH users as
well as their diversity, we propose a modifcation of common per-
sona templates that includes abilities instead of disabilities and
external factors infuencing the persona. The template recognises
ability as a positive attribute and dynamic area that is infuenced
by a variety of factors from physical, social, and technological en-
vironments. Although defned for DHH sign language users in the
context of travelling, we believe that our approach can be applied
and adapted to other contexts or target groups as needed. Thus,
the contribution of this paper lies not only in a novel perspective
on the design of personas for DHH users but on the integration of
Ability-Based Design into common HCI-methods altogether.
ACKNOWLEDGMENTS
The work was conducted as part of the research project AVASAG
and as such, funded by the German Federal Ministry of Education
and Research. In addition, the work was in part supported by the
Edinburgh College of Art, University of Edinburgh. The authors
wish to thank all projects, partners, and people who were involved
in and supported this work. Any opinions, fndings, conclusions or
recommendations expressed in this work are those of the authors
and do not necessarily refect those of any supporter.
REFERENCES
[1] Melissa L Anderson, Timothy Riker, Stephanie Hakulin, Jonah Meehan, Kurt
Gagne, Todd Higgins, Elizabeth Stout, Emma Pici-D’Ottavio, Kelsey Cappetta, and
Kelly S Wolf Craig. 2020. Deaf ACCESS: Adapting Consent Through Community
Engagement and State-of-the-Art Simulation. The Journal of Deaf Studies and
Deaf Education 25, 1 (2020), 115–125.
[2] H-Dirksen L Bauman. 2004. Audism: Exploring the Metaphysics of Oppression.
Journal of Deaf Studies and Deaf Education 9, 2 (2004), 239–246.
[3] Cynthia L. Bennett and Daniela K. Rosner. 2019. The Promise of Empathy: Design,
Disability, and Knowing the "Other". In Proceedings of the 2019 CHI Conference
on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1–13.
https://doi.org/10.1145/3290605.3300528
[4] Lucas Bernhard, Fabrizio Nunnari, Amelie Unger, Judith Bauerdiek, Christian
Dold, Marcel Hauck, Alexander Stricker, Tobias Baur, Alexander Heimerl, Elis-
abeth André, Melissa Reinecker, Cristina España-Bonet, Yasser Hamidullah,
Stephan Busemann, Patrick Gebhard, Corinna Jäger, Sonja Wecker, Yvonne
Kossel, Henrik Müller, Kristofer Waldow, Arnulph Fuhrmann, Martin Misiak,
and Dieter Wallach. 2022. Towards Automated Sign Language Production: A
Pipeline for Creating Inclusive Virtual Humans. In Proceedings of the 15th Inter-
national Conference on PErvasive Technologies Related to Assistive Environments
(PETRA ’22). Association for Computing Machinery, New York, NY, USA, 260–268.
https://doi.org/10.1145/3529190.3529202
[5] Roberto Casas, Rubén Blasco Marín, Alexia Robinet, Armando Roy Delgado, Ar-
mando Roy Yarza, John McGinn, Richard Picking, and Vic Grout. 2008. User Mod-
elling in Ambient Intelligence for Elderly and Disabled People. In Computers Help-
ing People with Special Needs, David Hutchison, Takeo Kanade, Josef Kittler, Jon M.
Kleinberg, Friedemann Mattern, John C. Mitchell, Moni Naor, Oscar Nierstrasz,
C. Pandu Rangan, Bernhard Stefen, Madhu Sudan, Demetri Terzopoulos, Doug
Tygar, Moshe Y. Vardi, Gerhard Weikum, Klaus Miesenberger, Joachim Klaus,
Wolfgang Zagler, and Arthur Karshmer (Eds.). Vol. 5105. Springer Berlin Heidel-
berg, Berlin, Heidelberg, 114–122. https://doi.org/10.1007/978-3-540-70540-6_15
Series Title: Lecture Notes in Computer Science.
[6] Larry Constantine. 2006. Users, Roles and Personas. In The Persona Lifecycle:
Keeping People in Mind Throughout Product Design, John Pruitt and Tamara Adlin
(Eds.). Morgan Kaufmann, San Francisco, 498–519.
[7] Alan Cooper. 1999. The inmates are running the asylum: Why high-tech products
drive us crazy and how to restore the sanity (2 ed.). Sams Publishing, Indianapolis.
[8] Alan Cooper, Robert Reimann, Dave Cronin, Reinhard Engel, and Alan Cooper.
2010. About Face: Interface- und Interaction-Design (1. auf ed.). mitp, Heidelberg
Amelie Nolte, Karolin Lueneburg, Dieter Wallach, and Nicole Jochems
München Landsberg Frechen Hamburg.
[9] Amy J. C. Cuddy, Susan T. Fiske, Virginia S. Y. Kwan, Peter Glick, Stéphanie
Demoulin, Jacques-Philippe Leyens, Michael Harris Bond, Jean-Claude Croizet,
Naomi Ellemers, Ed Sleebos, Tin Tin Htun, Hyun-Jeong Kim, Greg Maio, Judi
Perry, Kristina Petkova, Valery Todorov, Rosa Rodríguez-Bailón, Elena Morales,
Miguel Moya, Marisol Palacios, Vanessa Smith, Rolando Perez, Jorge Vala,
and Rene Ziegler. 2009. Stereotype content model across cultures: Towards
universal similarities and some diferences. British Journal of Social Psychol-
ogy 48, 1 (2009), 1–33. https://doi.org/10.1348/014466608X314935 _eprint:
https://onlinelibrary.wiley.com/doi/pdf/10.1348/014466608X314935.
[10] Tatiany X. de Godoi, Deógenes P. da Silva Junior, and Natasha M. Costa Valentim.
2020. A Case Study About Usability, User Experience and Accessibility Prob-
lems of Deaf Users with Assistive Technologies. In Universal Access in Human-
Computer Interaction. Applications and Practice, Margherita Antona and Con-
stantine Stephanidis (Eds.), Vol. 12189. Springer International Publishing, Cham,
73–91. https://doi.org/10.1007/978-3-030-49108-6_6 Series Title: Lecture Notes
in Computer Science.
[11] Sarah Ebling and John Glauert. 2016. Building a Swiss German Sign Language
avatar with JASigning and evaluating it among the Deaf community. Universal
Access in the Information Society 15, 4 (Nov. 2016), 577–587. https://doi.org/10.
1007/s10209-015-0408-1
[12] Emory Edwards, Kyle Lewis Polster, Isabel Tuason, Emily Blank, Michael Gilbert,
and Stacy Branham. 2021. "That’s in the eye of the beholder": Layers of Inter-
pretation in Image Descriptions for Fictional Representations of People with
Disabilities. In The 23rd International ACM SIGACCESS Conference on Comput-
ers and Accessibility. ACM, Virtual Event USA, 1–14. https://doi.org/10.1145/
3441852.3471222
[13] Emory James Edwards, Cella Monet Sum, and Stacy M. Branham. 2020. Three
Tensions Between Personas and Complex Disability Identities. In Extended Ab-
stracts of the 2020 CHI Conference on Human Factors in Computing Systems. ACM,
Honolulu HI USA, 1–9. https://doi.org/10.1145/3334480.3382931
[14] Johannes Fellinger, Daniel Holzinger, Rudolf Schoberberger, and Gerhard Lenz.
2005. Psychosoziale Merkmale bei Gehoerlosen: Daten aus einer Spezialambulanz
fuer Gehoerlose. Der Nervenarzt 76, 1 (2005), 43–51.
[15] Susan T. Fiske, Amy J. C. Cuddy, and Peter Glick. 2007. Universal dimensions of
social cognition: warmth and competence. Trends in Cognitive Sciences 11, 2 (Feb.
2007), 77–83. https://doi.org/10.1016/j.tics.2006.11.005
[16] Kristin Skeide Fuglerud, Trenton Schulz, Astri Letnes Janson, and Anne Moen.
2020. Co-creating Persona Scenarios with Diverse Users Enriching Inclusive
Design. In Universal Access in Human-Computer Interaction. Design Approaches
and Supporting Technologies, Margherita Antona and Constantine Stephanidis
(Eds.). Vol. 12188. Springer International Publishing, Cham, 48–59. https://
doi.org/10.1007/978-3-030-49282-3_4 Series Title: Lecture Notes in Computer
Science.
[17] Kim Goodwin. 2011. Designing for the Digital Age: How to Create Human-Centered
Products and Services. John Wiley & Sons Ltd., West Sussex, UK. Google-Books-ID:
yH6Aqr5zKJEC.
[18] Iris Groschek. 2008. Unterwegs in eine Welt des Verstehens. Gehörlosenbildung
in Hamburg vom 18. Jahrhundert bis in die Gegenwart. Hamburger Historische
Forschungen, Vol. 1. Hamburg University Press, Hamburg, Germany.
[19] Jo E. Hannay, Kristin Skeide Fuglerud, and Bjarte M. Østvold. 2020. Stakeholder
Journey Analysis for Innovation: A Multiparty Analysis Framework for Startups.
In Universal Access in Human-Computer Interaction. Applications and Practice,
Margherita Antona and Constantine Stephanidis (Eds.). Vol. 12189. Springer
International Publishing, Cham, 370–389. https://doi.org/10.1007/978-3-030-
49108-6_27 Series Title: Lecture Notes in Computer Science.
[20] Alexander Henka and Gottfried Zimmermann. 2017. PersonaBrowser. In Human-
Computer Interaction - INTERACT 2017 (Lecture Notes in Computer Science), Regina
Bernhaupt, Girish Dalvi, Anirudha Joshi, Devanuj K. Balkrishan, Jacki O’Neill,
and Marco Winckler (Eds.). Springer International Publishing, Cham, 54–63.
https://doi.org/10.1007/978-3-319-67684-5_4
[21] Shawn Lawton Henry. 2007. Just Ask: Integrating Accessibility Throughout Design.
Lulu.com, North Carolina, USA. Google-Books-ID: hRnpXbFB06cC.
[22] Dar’ya Heyko. 2021. Supporting d/Deaf and Hard of Hearing Employees in Their
Workplaces Through Technology, Design, and Community. Ph. D. Dissertation.
University of Guelph, Guelph, Ontario, Canada.
[23] Paul C. Higgins. 1979. Outsiders in a Hearing World: The Deaf Community.
Urban Life 8, 1 (1979), 3–22. Publisher: SAGE Publications.
[24] Charlotte Jais. 2018. Designing for dementia: personas to aid communication
between professionals developing built environments for people with dementia.
thesis. Loughborough University. https://doi.org/10.26174/thesis.lboro.8248934.
v1
[25] Hernisa Kacorri, Matt Huenerfauth, Sarah Ebling, Kasmira Patel, Kellie Men-
zies, and Mackenzie Willard. 2017. Regression Analysis of Demographic and
Technology-Experience Factors Infuencing Acceptance of Sign Language Ani-
mation. ACM Transactions on Accessible Computing 10, 1 (April 2017), 3:1–3:33.
https://doi.org/10.1145/3046787
Creating Personas for Signing User Populations
[26] Hernisa Kacorri, Matt Huenerfauth, Sarah Ebling, Kasmira Patel, and Macken-
zie Willard. 2015. Demographic and Experiential Factors Infuencing Accep-
tance of Sign Language Animation by Deaf Users. In Proceedings of the 17th
International ACM SIGACCESS Conference on Computers & Accessibility (AS-
SETS ’15). Association for Computing Machinery, Lisbon, Portugal, 147–154.
https://doi.org/10.1145/2700648.2809860
[27] Sebastian Kelle, Alexander Henka, and Gottfried Zimmermann. 2015. A Persona-
based Extension for Massive Open Online Courses in Accessible Design. Procedia
Manufacturing 3 (2015), 3663–3668. https://doi.org/10.1016/j.promfg.2015.07.772
[28] Michael Kipp, Quan Nguyen, Alexis Heloir, and Silke Matthes. 2011. Assessing
the deaf user perspective on sign language avatars. In The proceedings of the 13th
international ACM SIGACCESS conference on Computers and accessibility. ACM
Press, Dundee, Scotland, UK, 107–114.
[29] Jessica Korte, Leigh Ellen Potter, and Sue Nielsen. 2017. The impacts of deaf
culture on designing with deaf children. In Proceedings of the 29th Australian
Conference on Computer-Human Interaction. ACM, Brisbane Queensland Australia,
135–142. https://doi.org/10.1145/3152771.3152786
[30] Harlan Lane. 2005. Ethnicity, Ethics, and the Deaf-World. Journal of Deaf Studies
and Deaf Education 10, 3 (2005), 291–310.
[31] Joong Hee Lee, Yong Min Kim, Ilsun Rhiu, and Myung Hwan Yun. 2021. A Persona-
Based Approach for Identifying Accessibility Issues in Elderly and Disabled
Users’ Interaction with Home Appliances. Applied Sciences 11, 1 (Jan. 2021), 368.
https://doi.org/10.3390/app11010368 Number: 1 Publisher: Multidisciplinary
Digital Publishing Institute.
[32] Nicola Marsden and Maren Haag. 2016. Stereotypes and Politics: Refections
on Personas. In Proceedings of the 2016 CHI Conference on Human Factors in
Computing Systems. ACM, San Jose California USA, 4017–4031. https://doi.org/
10.1145/2858036.2858151
[33] Nicola Marsden and Monika Proebster. 2019. Personas and Identity: Looking at
Multiple Identities to Inform the Construction of Personas. In Proceedings of the
2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow
Scotland Uk, 1–14. https://doi.org/10.1145/3290605.3300565
[34] Michael McKee, Deirdre Schlehofer, and Denise Thew. 2013. Ethical Issues in
Conducting Research With Deaf Populations. American Journal of Public Health
103, 12 (2013), 2174–2178.
[35] Tan Ching Ying Michelle. 2017. Exploring inclusive design partnerships through
an IDEA framework to support deaf or hard of hearing australian children in
design process participation. In Proceedings of the 29th Australian Conference on
Computer-Human Interaction. ACM, Brisbane Queensland Australia, 433–437.
https://doi.org/10.1145/3152771.3156151
[36] Microsoft. 2018. Inclusive Activities. https://www.microsoft.com/design/
inclusive/
[37] Timothy Neate, Aikaterini Bourazeri, Abi Roper, Simone Stumpf, and Stephanie
Wilson. 2019. Co-Created Personas: Engaging and Empowering Users with
ASSETS ’22, October 23–26, 2022, Athens, Greece
Diverse Needs Within the Design Process. In Proceedings of the 2019 CHI Confer-
ence on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1–12.
https://doi.org/10.1145/3290605.3300880
[38] Lene Nielsen. 2004. Engaging Personas and Narrative Scenarios. Doctoral Thesis.
Copenhagen Business School, Frederiksberg.
[39] Lene Nielsen. 2019. Personas - User Focused Design. Springer London, London.
https://doi.org/10.1007/978-1-4471-7427-1
[40] Fabrizio Nunnari, Judith Bauerdiek, Lucas Bernhard, Cristina España-Bonet,
Corinna Jäger, Amelie Unger, Kristofer Waldow, Sonja Wecker, Elisabeth André,
Stephan Busemann, Christian Dold, Arnulph Fuhrmann, Patrick Gebhard, Yasser
Hamidullah, Marcel Hauck, Yvonne Kossel, Martin Misiak, Dieter Wallach, and
Alexander Stricker. 2021. AVASAG: A German Sign Language Translation System
for Public Services (short paper). In Proceedings of the 1st International Workshop
on Automatic Translation for Signed and Spoken Languages (AT4SSL). Association
for Machine Translation in the Americas, Virtual, 43–48. https://aclanthology.
org/2021.mtsummit-at4ssl.5
[41] Lyydia Pertovaara. 2021. Framework for Creating Ethical Older People Personas for
the Development of Ambient Assisted Living Technologies. Master’s thesis. Laurea
University of Applied Sciences, Vantaa, Finland.
[42] Monika Proebster, Julia Hermann, and Nicola Marsden. 2019. Personas and
Persons - An Empirical Study on Stereotyping of Personas. In Proceedings of
Mensch und Computer 2019. ACM, Hamburg Germany, 137–145. https://doi.org/
10.1145/3340764.3340771
[43] John Pruitt and Tamara Adlin. 2006. The Persona Lifecycle. Elsevier, San Francisco.
[44] Trenton Schulz and Kristin Skeide Fuglerud. 2012. Creating Personas with
Disabilities. arXiv:2003.11875 [cs] 7383 (2012), 145–152. https://doi.org/10.1007/
978-3-642-31534-3_22 arXiv: 2003.11875.
[45] Jenny Singleton. 2014. Toward Ethical Research Practice With Deaf Participants.
Journal of Empirical Research on Human Research Ethics 9, 3 (2014), 59–66.
[46] Andrew Solomon. 2012. Far From the Tree: Parents, Children and the Search for
Identity. Simon and Schuster, New York, USA.
[47] Nicky Sulmon, Karin Slegers, Karel Van Isacker, Maria Gemou, and Evangelos
Bekiaris. 2010. Using Personas to capture Assistive Technology Needs of People with
Disabilities. CSUN, Northridge.
[48] Amelie Unger, Dieter P. Wallach, and Nicole Jochems. 2021. Lost in Translation:
Challenges and Barriers to Sign Language-Accessible User Research. In The
23rd International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’21). Association for Computing Machinery, New York, NY, USA, 1–5.
https://doi.org/10.1145/3441852.3476473
[49] Jacob O. Wobbrock, Krzysztof Z. Gajos, Shaun K. Kane, and Gregg C. Vander-
heiden. 2018. Ability-based design. Commun. ACM 61, 6 (May 2018), 62–71.
https://doi.org/10.1145/3148051
[50] Jacob O. Wobbrock, Shaun K. Kane, Krzysztof Z. Gajos, Susumu Harada, and
Jon Froehlich. 2011. Ability-Based Design: Concept, Principles and Examples.
ACM Transactions on Accessible Computing 3, 3 (April 2011), 9:1–9:27. https:
//doi.org/10.1145/1952383.1952384
ASSETS ’22, October 23–26, 2022, Athens, Greece Amelie Nolte, Karolin Lueneburg, Dieter Wallach, and Nicole Jochems

A APPENDIX: ABILITY-BASED PERSONA

Figure 1: Ability-based persona of sign language users in the context of travel information and services. The template consists
of (1) the persona type with label, image and quote), (2) a list of context-specifc needs and pains and (3) a display of ability
ranges, each with respective enabling and disabling factors. (Illustration credits: Severin Arnold)
 Designing a Data Visualization Dashboard for Pre-Screening
Hong Kong Students with Specific Learning Disabilities
Ka Yan Fung Zikai Alex Wen∗ Haotian Li
Individualized Interdisciplinary Computational Media and Arts Department of Computer Science
Program, The Hong Kong University Thrust, The Hong Kong University of and Engineering, The Hong Kong
of Science and Technology Science and Technology (Guangzhou) University of Science and
Hong Kong SAR, China Guangzhou, China Technology
kyfungag@connect.ust.hk zikaiwen@ust.hk Hong Kong SAR, China
haotian.li@connect.ust.hk

Xingbo Wang Shenghui Song∗ Huamin Qu

Department of Computer Science Department of Electronic and Department of Computer Science
and Engineering, The Hong Kong Computer Engineering, The Hong and Engineering, The Hong Kong
University of Science and Kong University of Science and University of Science and
Technology Technology Technology
Hong Kong SAR, China Hong Kong SAR, China Hong Kong SAR, China
xingbo.wang@connect.ust.hk eeshsong@ust.hk huamin@cse.ust.hk

ABSTRACT 1 INTRODUCTION AND BACKGROUND

Students with specifc learning disabilities (SLDs) often experience
reading, writing, attention, and physical movement coordination
difculties. However, in Hong Kong, it takes years for special ed-
ucation needs coordinators (SENCOs) and special-ed teachers to
pre-screen and diagnose students with SLDs. Therefore, many stu-
dents with SLDs missed the golden time for special interventions
(i.e., before six years old). In addition, although there are screen-
ing tools for students with SLDs in Chinese and Indo-European
languages (e.g., English and Spanish), they did not provide a stu-
dent data visualization dashboard that could help teachers speed
up the pre-screening process. Therefore, we designed a new visual-
ization dashboard for Hong Kong SENCOs and special-ed teachers
to assist them in pre-screening students with SLDs. Our formative
study showed that our current design met teachers’ need to quickly
identify a student’s specifc under-performing tasks and efectively
collect evidence about how the student was afected by SLDs. Fu-
ture work will further test the efcacy of our design in real life.
ACM Reference Format:
Ka Yan Fung, Zikai Alex Wen, Haotian Li, Xingbo Wang, Shenghui Song,
and Huamin Qu. 2022. Designing a Data Visualization Dashboard for Pre-
Screening Hong Kong Students with Specifc Learning Disabilities. In The
24th International ACM SIGACCESS Conference on Computers and Accessi-
bility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York,
NY, USA, 5 pages. https://doi.org/10.1145/3517428.3550361
∗ Zikai Alex Wen and Shenghui Song are the corresponding authors.
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550361
Specifc learning disabilities (SLDs) afect how people process and
learn language-based information [21], but they do not afect peo-
ple’s intelligence quotient [20]. As a result, SLDs can signifcantly
impact learning skills and the acquisition of literacy skills [30, 33].
SLDs are used to cover a range of frequently co-occurring learn-
ing difculties, most commonly known as dyslexia (reading), dys-
graphia (writing), and dyspraxia (physical co-ordination) [2]. Re-
search [6] suggested that students with SLDs who receive special
interventions during the golden time (i.e., before six years old) can
develop better learning skills as they grow up.
To help special education students, including those with SLDs,
Hong Kong Education Bureau has recruited special education needs
coordinators (SENCOs) and special-ed teachers for public schools
since 2017 [4]. However, because of the limited number of SEN-
COs and special-ed teachers, one special-ed faculty has to support
nearly one thousand students. In this case, many students with
SLDs in Hong Kong have to wait over 2 to 3 years to fnish their
assessment [9, 22].
To help the Hong Kong special-ed faculty speed up assessing
their students, Fung et al. [10] developed an automatic pre-screening
tool for dyslexia in Chinese, dysgraphia in Chinese, and dyspraxia.
Fung et al.’s pre-screening tool collected valuable student data, in-
cluding correctness rate, answer time, handwriting process, and
the video of the student interacting with the touchscreen. Teach-
ers can analyze the data to understand why the student’s perfor-
mance is below average and what difculties the student encoun-
tered. However, it is still quite time-consuming for special-ed teach-
ers to analyze the raw data.
Currently, the student data dashboards of existing pre-screening
tools [3, 8, 12, 29] only provide student performance statistics. For
example, the dashboard of Dyscreen Dyslexia Screener [29] only
provides individual student performance scores and scores com-
pared with the student’s related age group. If we were to adopt
the existing student data dashboard designs, teachers would have
ASSETS ’22, October 23–26, 2022, Athens, Greece
faced the same problem: They cannot efciently analyze and un-
derstand students’ performance with SLDs.
Therefore, our work aims to design a new student data visual-
ization dashboard that processes the raw data collected by Fung
et al.’s pre-screening tool [10]. Without collecting the raw data of
handwriting and videos, special-ed teachers cannot discover how
exactly the student’s performance was afected by SLDs. Therefore,
although our dashboard visualizes the raw data provided by Fung
et al.’s tool, it is potentially helpful for any pre-screening tools de-
signed to meet special-ed teachers’ needs. Our tool allows special-
ed teachers to quickly narrow down the list of students who might
have SLDs in Chinese from thousands of student records and ef-
fciently gather evidence that indicates how the students were af-
fected by SLDs. To verify whether our design meets our goals and
improves our design, we conducted a formative study with two
SENCOs, one special-ed teacher, and one special-ed student-teacher.
The study showed that our current design met special-ed teach-
ers’ need to quickly identify a student’s specifc under-performing
tasks and efectively collect evidence about how the student was
afected by SLDs. Our future work will test the efcacy of our dash-
board design in special-ed teachers’ daily jobs. We will also ex-
plore how our design may beneft SLDs pre-screening for other
languages (e.g., Indo-European languages such as English).
2 DASHBOARD DESIGN
The design of our tool consists of 4 panels (as shown in Figure 1):
(1) an overview of all students’ tests performance (Panel 1) for the
teacher to quickly narrow down the list of students who might
have SLDs; (2) overview of a selected student’s test performance
(Panel 2) for the teacher to identify the specifc under-performing
tests quickly; (3) performance statistics of individual questions in
a test (Panel 3) for the teacher to identify which question to look
into; (4) student performance when the student answered the ques-
tion (Panel 4) for the teacher to investigate how the student was
afected by SLDs. Each one of the panels is designed based on a de-
sign requirement by our target users. In the following paragraphs,
we will elaborate on each design requirement.
Design Requirement 1: The system should help teachers narrow
down the list of students who might have SLDs in Chinese. SENCOs
and special-ed teachers in Hong Kong need to care for around
a thousand students. They need an assistive tool to quickly nar-
row down the list of students who might have SLDs. Many pre-
screening tools [1, 7, 12, 18, 19, 32] provide information about which
students potentially face mild, moderate, or severe learning dif-
culties. Therefore, we designed Panel 1, which allows teachers to
flter the student records by choosing the options of four learning
difculty levels: acceptable, mild, moderate, and severe. We also
added a red line that denotes the average score of all students so
that teachers can explicitly compare how much a student lags be-
hind the average student performance.
Design Requirement 2: The system should help teachers identify
the specifc under-performing tasks. After the teacher decides which
student profle to look into, they want to determine which test-
ing tasks are under-performing. The existing SLDs pre-screening
Ka Yan Fung, Zikai Alex Wen, Haotian Li, Xingbo Wang, Shenghui Song, and Huamin Q
tools [11, 14, 15, 17, 27, 28] provide information about the per-
formance of each task in three testing categories: word recogni-
tion, reading, and writing. Therefore, we designed Panel 2, which
presents all one student’s task scores. Panel 2 also provides infor-
mation about how much a student lags behind the average student
performance in each task. Reviewing Panel 2 allows teachers to
decide which testing tasks to investigate quickly.
Design Requirement 3: The system should help teachers decide
which sub-question to investigate. In many SLDs pre-screening
tools [13, 24–26, 31], students need to answer multiple sub-questions
to complete one task. As a result, students potentially afected by
SLDs may spend too much time on one sub-question (even though
they might answer it correctly). Therefore, teachers need a con-
venient way to determine whether the student was in the above
situation and further investigate how the student answered this
sub-question.
We designed Panel 3 to meet the teachers’ needs. Panel 3 presents
the following student data for each sub-question: the correctness
rate (including correct, incorrect, and no answer), the time spent on
a sub-question, the average time spent on all sub-questions of the
same task, and the average time that all students spend on the same
sub-question. In addition, we designed a new feature for handwrit-
ing tasks - animating the playback of the entire handwriting pro-
cess. Previous work for pre-screening dysgraphia in English only
provided the fnal screenshot of a student’s handwriting [23]. How-
ever, research [10] suggested that analyzing the Chinese character
stroke sequence to pre-screen dysgraphia in Chinese is essential.
Furthermore, it is also critical to pre-screen dyspraxia in Chinese
by identifying slow and curly handwriting process [36]. Therefore,
our handwriting animation (as shown in Figure 1, 3.1) helps teach-
ers to pre-screen dysgraphia and dyspraxia in Chinese.
Design Requirement 4: The system should help teachers investigate
how the student was afected by SLDs. Fung et al.’s pre-screening
tool [10] collected more student data than many pre-screening tools
for SLDs in an Indo-European language (e.g., English, Spanish) [3,
8, 12, 29, 34]: screenshots of every handwriting Chinese character
stroke and videos of the student interacting with the pre-screening
tool. As with any disability, no two individuals experience the same
difculties, and some may exhibit signs of more than one SLD.
Therefore, this information is helpful for teachers to collect explicit
evidence about the common characteristics of SLDs (i.e., short at-
tention span and inadequate memory retrieval). That being said,
it is time-consuming for teachers to go through all the videos, es-
pecially for Hong Kong special-ed teachers, because they have to
manage many student cases.
Special-ed teachers care most about students’ specifc spatial
movement patterns (e.g., starting to look around the room). How-
ever, because the temporal information is rich, we faced a design
challenge to break down all the temporal information to reduce
visual clutter and avoid visual occlusion.
Inspired by previous timeline designs [16, 35] for analyzing tem-
poral evolution of body movement we broke down the student
movement information into three dimensions (i.e., limbs, heads,
and body movements) as shown in Figure 1, 4.1. We also visualized
the average student movement range and put it alongside the indi-
vidual student’s movement visualization, as shown in Figure 1, 4.2.
The special-ed teachers can then capture the signals in the timeline
Designing a Dashboard for Pre-Screening Hong Kong Students with Specific Learning Disabilities ASSETS ’22, October 23–26, 2022, Athens, Greece

Figure 1: Student Data Visualization Dashboard. Panel 1 shows the overview of student performance. Panel 2 shows the
overview of one student’s testing tasks. Panel 3 shows the student data in answering all sub-questions in one task. Panel
3.1 shows the playback of the handwriting recording. Finally, panel 4 shows the visualization of the movement data (Panel
4.1), the comparison between individual and average movements (Panel 4.2), each stroke of the Chinese character in the writ-
ing tasks (Panel 4.3), and the video recording of the student movement (Panel 4.4).

visualization that indicate a student might have a short attention
span compared with general education students.
Panel 4 also visualizes when and how the student wrote each
Chinese character stroke in the writing tasks (as shown in Figure 1,
4.3). This visualization can highlight the situation when a student
spent a signifcant amount of time fnishing one Chinese character
stroke, which informs the teacher that the student might experi-
ence inadequate memory retrieval. The teacher can then go to the
corresponding video playback time (as shown in Figure 1, 4.4) to
watch how exactly the student behaved.
3 FORMATIVE STUDY: TEACHER FEEDBACK
To form our design, we conducted a formative study with two SEN-
COs, one special-ed teacher, and one special-ed student-teacher for
students with SLDs (3 females, 1 male, between 3 to 30 years of ex-
perience in teaching students with SLDs from primary school stu-
dents to undergraduates, aged from 20 to 55 years old). They have
either an education certifcate or a bachelor degree in special ed-
ucation. We started by interviewing the participants because they
are the key users of the learning analytic visualization tool and
have experience in using relevant systems to pre-screen SLDs stu-
dents. We anticipated that the participants would be able to use
their SLDs pre-screening experience to ofer valuable insights into
our current design. During the study, we demonstrated our proto-
type to the participants. Our participants’ working languages are
Chinese and English, so the system language of our prototype sup-
ports these two languages.
The data presented in our dashboard prototype are real data col-
lected by Fung et al. [10] when they invited students to play a game
for pre-screening SLDs in Chinese. We post-processed the video
ASSETS ’22, October 23–26, 2022, Athens, Greece
recordings of student actions using the OpenPose AI model [5] to
highlight the students’ heads, arms, and hands. We also wrote a
script to visualize how a student writes each stroke of a Chinese
character by processing the handwriting data (i.e., the coordinates
of screen touching).
As we showed the dashboard prototype to our study partici-
pants, we asked them to give feedback on the design of the data
visualization dashboard to help them pre-screen SLDs. We screen-
recorded and then transcribed all interviews. We coded the tran-
scriptions and found two design highlights and one design im-
provement summarized from the teacher feedback.
All SENCOs and special-ed teachers mentioned that the student
data visualization dashboard could facilitate their pre-screening
job. In addition, they appreciated the design of Panel 3 for analyz-
ing dysgraphia and dyspraxia in Chinese and the design of Panel
4 for analyzing the issues of short attention span or inadequate
memory retrieval. One special-ed teacher explained why animat-
ing the order of handwriting strokes helps, “The stroke order is a
characteristic of reading and writing [Chinese words]. We can [use
it to] analyze the student’s sense of spacing. It can provide more
evidence for teachers to judge students’ reading and writing prob-
lems.” Another special-ed teacher shared a similar opinion, “Chil-
dren with SLDs might not be aware of the writing grid. [So] their
handwriting is sometimes big or small.”
One special-ed and one SENCO pointed out that Panel 4 helped
them quickly and explicitly collect the evidence of students having
SLDs because “it can record students’ problematic situations like
easily distracted, and sitting awkwardly” according to the special-
ed teacher. All participants agreed that the time range of a stu-
dent interacting with the touchscreen (as shown in Figure 1, 4.2)
in Panel 4 helped them understand the student thinking process.
One special-ed teacher explained, “If the student thinks for a long
time but can write a word correctly. The teacher will wonder if
the question is too difcult, or the student cannot extract the word
because of inadequate memory retrieval.”
According to one SENCO and one special-ed teacher, we can im-
prove our tool to help them better analyze the student behaviours
when the student looks left and right while halfway fnishing the
writing task. Noticing that our tool provides both handwriting an-
imation (as shown in Figure 1, 3.1) and video recording (as shown
in Figure 1, 4.4), they suggested we synchronously display these
two types of playback side by side in Panel 4.
4 CONCLUSION AND FUTURE WORK
We presented a new design of a student data visualization dash-
board for SLDs pre-screening. Our design aims to assist Hong Kong
SENCOs and special-ed teachers with their SLDs pre-screening job
so their students can receive SLDs assessment and special inter-
ventions in time. We invited two SENCOs, one special-ed teacher,
and one special-ed student-teacher for a formative study. Our de-
sign met their needs for quick pre-screening of students with SLDs
in Chinese, especially our designs that fulfll their design require-
ments of identifying a student’s specifc under-performing tasks
quickly and collecting evidence about how the student was afected
by SLDs. The study participants also suggested a minor design im-
provement of the visualization dashboard.
Ka Yan Fung, Zikai Alex Wen, Haotian Li, Xingbo Wang, Shenghui Song, and Huamin Q
In the future, we will recruit more SENCOs and special-ed teach-
ers to test the efcacy of our dashboard in their daily job. We
will also explore how our dashboard design may beneft SLDs pre-
screening for other languages (including Indo-European languages).
Finally, our work will inspire educational technology designers
and developers to leverage our design to provide a better student
data visualization dashboard for teachers of students with SLDs.
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their sugges-
tions and our study participants for their valuable feedback.
REFERENCES
[1] GL Assessment. 2022. Dyslexia and Dyscalculia Screeners Digital.
(2022). https://support.gl-assessment.co.uk/media/1764/gla176-dyslexia-and-
dyscalculia-screener-sample-report.pdf
[2] The Dyslexia Association. [n.d.]. Specifc Learning Difculties (SpLDs).
https://www.dyslexia.uk.net/specifc-learning-difculties/
[3] Nathanael Bucher, Mirjam Voegeli, and Rebecca Gretler. [n.d.]. Welcome
to Dybuster Calcularis. https://dybuster.com/wp-content/uploads/2018/12/
WorkingCards_Calcularis_2018.pdf
[4] The Hong Kong Education Bureau. 2017.08. 關愛基金《向普通學校撥款以安
排特殊教育需要統籌主任》試驗計劃成效檢討報告 (Report on the Effective-
ness Review of the Community Care Fund’s Pilot Scheme on "Appropriation of
Funds to Ordinary Schools to Arrange Special Educational Needs Coordinators").
(2017.08). https://sense.edb.gov.hk/uploads/page/professional-support/special-
educational-needs-coordinator/Evaluation_LegCo_Chi.pdf
https://www.dyslexia.uk.net/specific-learning-difficulties/
[5] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2021.
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields.
IEEE Transactions on Pattern Analysis & Machine Intelligence 43, 01 (2021), 172–
186.
[6] Helen Arkell Dyslexia Charity. [n.d.]. When should a Child be Assessed for
Dyslexia? https://www.helenarkell.org.uk/about-dyslexia/parents/faq-for-
parents.php#: :text=Dyslexia%20can%20be%20identified%20by,symptoms%20are
%20not%20always%20obvious.
[7] The Psychological Corporation. 2008. DAS-II Sample Report - Pearson Clinical.
(2008). https://www.pearsonclinical.com.au/files/DASII_Sample_Report(1).pdf
[8] Houston Independent School District. [n.d.]. Dyslexia Universal Screener 2021-
2022. https://www.houstonisd.org/Page/178674
[9] The Hong Kong Society for Community Organization. 2018. 《學
前 有 特 殊 教 育 需 要 兒 童 過 渡 至 學 齡 階 段 的 服 務 需 要》 質
性 研 究 調 查 報 告 (A Qualitative Research Report on the Service
Needs of Preschool Children with Special Educational Needs in
Transition to School Age). (2018). https://www.legco.gov.hk/yr17-
18/chinese/panels/ed/papers/ed20180302cb4-679-1-c.pdf
[10] Ka Yan Fung, Fung Kwong Chiu, Chan Aidan, and Yu Yi Ching. 2021. A Digital
Tool to Provide Pre-Screening to Dyslexia in Hong Kong. In Proceedings of the
2021 IEEE International Conference on Engineering, Technology & Education. IEEE,
755–761.
[11] Louise Ho. 2000. Hong Kong Writing and Writing Hong Kong. World Englishes
19, 3 (2000), 381–386.
[12] Children’s Learning Institute. [n.d.]. Dyslexia Screeners on CLI Engage.
https://public.cliengage.org/tools/assessment/dyslexia-screener/
[13] Ruchira Kariyawasam, Madhuka Nadeeshani, Tuan Hamid, Inisha Subasinghe,
Pradeepa Samarasinghe, and Pasangi Ratnayake. 2019. Pubudu: Deep Learning
based Screening and Intervention of Dyslexia, Dysgraphia and Dyscalculia. In
Proceedings of the 2019 14th Conference on Industrial and Information Systems.
IEEE, 476–481.
[14] Catherine Lam. 2018. Children Assessment Service Epidemiology and Research
Bulletin. (2018). https://www.dhcas.gov.hk/fle/caser/CASER15.pdf
[15] Che Kan Leong, Pui Wan Cheng, and Li Hai Tan. 2005. The Role of Sensitivity
to Rhymes, Phonemes and Tones in Reading English and Chinese Pseudowords.
Reading and Writing 18, 1 (2005), 1–26.
[16] Haotian Li, Min Xu, Yong Wang, Huan Wei, and Huamin Qu. 2021. A Visual
Analytics Approach to Facilitate the Proctoring of Online Exams. In Proceedings
of the 2021 CHI Conference on Human Factors in Computing Systems. 1–17.
[17] Catherine McBride-Chang, Fanny Lam, Catherine Lam, Sylvia Doo, Simpson WL
Wong, and Yvonne YY Chow. 2008. Word Recognition and Cognitive Profles of
Chinese Pre-school Children at Risk for Dyslexia through Language Delay or
Familial History of Dyslexia. Journal of Child Psychology and Psychiatry 49, 2
(2008), 211–218.
Designing a Dashboard for Pre-Screening Hong Kong Students with Specific Learning Disabilities
[18] A Jockey Club Learning Support Network. 2010. The Hong Kong Be-
haviour Checklist of Specifc Learning Difculties in Reading and Writ-
ing for Primary School Students (Second Edition) Manual (BCL-P(II)).
(2010). https://hksld.eduhk.hk/%E8%B3%87%E6%BA%90/%E8%AD%98%E5%
88%A5%E5%B7%A5%E5%85%B7
[19] NWEA. 2022. Dyslexia Screener Matrix Report. (2022). https://teach.mapnwea.
org/impl/maphelp/Content/ReadFluency/Reports/DyslexiaScreenerMatrix.htm
[20] Learning Disabilities Association of America (LDA). [n.d.]. Types of Learning
Disabilities. https://ldaamerica.org/types-of-learning-disabilities/
[21] University of Kent. [n.d.]. What are Specifc Learning Difculties?
https://www.kent.ac.uk/teaching/networks/ltn/documents/2014-15/18-
Sept-2014-SpLD-handout-L-Regan.pdf
[22] Legislative Council of the Hong Kong Special Administrative Region. 2014.
立 法 會 十 題： 教 育 心 理 學 家 (LCQ10: Educational psychologists). (2014).
https://www.info.gov.hk/gia/general/201405/14/P201405140403.htm
[23] Md Abdur Rahman, Elham Hassanain, Md Mamunur Rashid, Stuart J Barnes,
and M Shamim Hossain. 2018. Spatial Blockchain-based Secure Mass Screening
Framework for Children with Dyslexia. IEEE Access 6 (2018), 61876–61885.
[24] Maria Rauschenberger, Christian Lins, Noelle Rousselle, Andreas Hein, and Se-
bastian Fudickar. 2019. Designing a New Puzzle App to Target Dyslexia Screen-
ing in Pre-readers. In Proceedings of the 5th EAI International Conference on Smart
Objects and Technologies for Social Good. 155–159.
[25] Luz Rello, Ricardo Baeza-Yates, Abdullah Ali, Jefrey P Bigham, and Miquel Serra.
2020. Predicting Risk of Dyslexia with an Online Gamifed Test. Plos One 15, 12
(2020), e0241687.
[26] Luz Rello, Miguel Ballesteros, Abdullah X Ali, Miquel Serra, Daniela Alarcón
Sánchez, and Jefrey P Bigham. 2016. Dytective: Diagnosing Risk of Dyslexia
with a Game. In Pervasive Health. 89–96.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[27] Luz Rello, Clara Bayarri, and Azuki Gorriz. 2012. What is Wrong with this Word?
Dyseggxia: a Game for Children with Dyslexia. In Proceedings of the 14th Inter-
national ACM SIGACCESS Conference on Computers and Accessibility. 219–220.
[28] Luz Rello, Clara Bayarri, Yolanda Otal, and Martin Pielot. 2014. A Computer-
based Method to Improve the Spelling of Children with Dyslexia. In Proceedings
of the 16th International ACM SIGACCESS Conference on Computers & Accessibil-
ity. 153–160.
[29] Dyscreen Dyslexia Screener. [n.d.]. Reading Assessment & Progress Monitoring,
Simplifed. https://dystech.com.au/
[30] Hong Kong EP Services. [n.d.]. Specifc Learning Difculties (SLD).
https://www.hkep.org/specifc-learning-difculties-sld/
[31] Abdul Samad Shibghatullah. 2017. Dleksia Game: A Mobile Dyslexia Screening
Test Game to Screen Dyslexia using Malay Language Instruction. Asian Journal
of Information Technology 16, 1 (2017), 1–6.
[32] Hong Kong Specifc Learning Difculties Research Team. 2015. The Hong Kong
Reading Ability Teacher Observation Checklist for Preschool Children(TOC-K).
(2015). https://hksld.eduhk.hk/
[33] Zikai Alex Wen, Erica Silverstein, Yuhang Zhao, Anjelika Lynne Amog, Kather-
ine Garnett, and Shiri Azenkot. 2020. Teacher Views of Math E-learning Tools
for Students with Specifc Learning Disabilities. In Proceedings of the 22nd Inter-
national ACM SIGACCESS Conference on Computers and Accessibility. 1–13.
[34] Zikai Alex Wen, Yuhang Zhao, Erica Silverstein, and Shiri Azenkot. 2021. An
Intelligent Math E-Tutoring System for Students with Specifc Learning Disabili-
ties. In Proceedings of the 23rd International ACM SIGACCESS Conference on Com-
puters and Accessibility. 1–4.
[35] Haipeng Zeng, Xingbo Wang, Yong Wang, Aoyu Wu, Ting-Chuen Pong, and
Huamin Qu. 2022. GestureLens: Visual Analysis of Gestures in Presentation
Videos. IEEE Transactions on Visualization and Computer Graphics (2022).
[36] 李瑩玓. 2004. 寫字困難學生寫字特徵之分析 (An Analysis of the Writing
Characteristics of Students with Writing Difculties). 師大學報: 教育類 (2004).
 Digital Accessibility in Iran: An Investigation Focusing on Iran’s
National Policies on Accessibility and Disability Support
Laleh Nourian
ln2293@rit.edu
Department of Computing and
Information Sciences Rochester Institute of Technology
Rochester Institute of Technology
Rochester, New York, USA
ABSTRACT
Digital accessibility has become an important topic in the feld of
HCI, but when looking at accessibility on a global scale, we fnd that
the representation of accessibility research is mostly centered in the
Global North with countries that are WEIRD (Western, Educated,
Industrialized, Rich, and Democratic). Our paper explores digital
accessibility in Iran, focusing exclusively on its national policies
on accessibility. Iran is a non-WEIRD country located in the Global
South, with no reports on its digital accessibility status from the
Global Initiative for Inclusive Information and Communication
Technologies (G3ict). We found that there is not enough focus on
accessibility in Iran’s regulations and we conclude our paper by
recommending directions for improving this situation such as HCI
and disability organizations in Iran cooperating with G3ict.
CCS CONCEPTS
• Human-centered computing → Accessibility.
ACM Reference Format:
Laleh Nourian, Kristen Shinohara, and Garreth W. Tigwell. 2022. Digital
Accessibility in Iran: An Investigation Focusing on Iran’s National Policies
on Accessibility and Disability Support. In The 24th International ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS ’22), October
23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 5 pages. https://doi.
org/10.1145/3517428.3550385
1 INTRODUCTION
The term postcolonial computing emerged to encourage refection
of cultural infuence on the design and evaluation of technology
due to increased globalization [21], and such refection must also
encompass accessibility [44]. Digital accessibility typically ensures
that all disabled people1 can use digital content and services (e.g.,
websites, mobile applications, etc.), while also providing wider
benefts to anybody who might encounter contextual challenges.
However, prior studies demonstrate a bias in HCI and accessi-
bility research publications highlighting that the fndings of the
published papers are mostly from specifc regions of the globe (e.g.,
1 We will be writing with identify-frst language in this paper [5]
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550385
Kristen Shinohara Garreth W. Tigwell
kristen.shinohara@rit.edu garreth.w.tigwell@rit.edu
School of Information School of Information
Rochester Institute of Technology
Rochester, New York, USA Rochester, New York, USA
Global North2 [7] and WEIRD countries (Western, Educated, In-
dustrialized, Rich, and Democratic) [26]). The existing bias within
our accessibility and HCI communities might have consequences
for digital accessibility when we consider that research from pub-
lished work often goes on to infuence many sectors, government
legislation, and industry practice [4, 12].
There is still a need to increase the representation of people from
diferent parts of the world and the efort from the Global Initia-
tive for Inclusive Information and Communication Technologies
(G3ict) partly fulflls this need by developing the Digital Accessibil-
ity Rights Evaluation Index (DARE Index). The DARE Index assesses
how countries incorporate accessibility in digital content [14] and
the assessment is completed with the cooperation of Disabled Peo-
ple’s International (DPI) Assemblies, and the experts and national
organizations of disabled people in the countries. However, not all
countries are represented in the DARE Index.
We present our initial work on the status of digital accessibility
in Iran to identify the extent that digital accessibility is codifed in
Iran’s national policies. Our investigation could be valuable since
there is no work on the status of Iran’s digital accessibility in the
DARE Index and the fact that it is a non-WEIRD country located in
the Global South makes it generally under-represented in HCI and
accessibility research [7, 26] (i.e., there are only a few HCI studies
investigating Iranian website accessibility [18, 32], with others
focusing mainly on usability [11, 31]). Moreover, prior studies have
shown that the policies of a country can infuence accessibility
awareness in society [6, 28] and the results of this investigation
ascertain an initial understating of accessibility regulations and
awareness in Iran. Thus, we seek to answer the following questions:
RQ1: What regulations, if any, does Iran set for accessibility and
supporting disabled people in digital spaces?
RQ2: What accessibility guidelines, if any, do the national regula-
tions require or suggest that designers use?
We provide the frst review of Iran’s policies on accessibility
and disability, showing that digital accessibility is not extensively
codifed in its national laws and there are no regulations about
adhering to digital accessibility guidelines. We call on the HCI com-
munity and disability support organizations in Iran, and the Global
Initiative for Inclusive ICTs (G3ict) to cooperate and conduct ex-
tensive investigations on digital accessibility in Iran. A cooperative
efort will result in a deeper understanding of digital accessibil-
ity knowledge within Iran and identify knowledge gaps for future
accessibility research to address.
2 Global North countries are typically richer, economically developed, and located in
northern hemisphere (including Australia and New Zealand) [16].
ASSETS ’22, October 23–26, 2022, Athens, Greece
2 RELATED WORK
2.1 Accessible Digital Design
Despite the availability of accessibility guidelines, methods, and
tools [23], many mobile and web services remain inaccessible [6, 17,
22, 35, 38]. One of the most widely used international accessibility
resources is the W3C’s Web Content Accessibility Guidelines [9].
However, some designers are critical of the W3C’s Guidelines be-
cause they are difcult to understand [40], not to mention that they
are yet to be translated into all ofcial languages (e.g., there is no
Persian translation, which would provide better support for Iranian
designers).
Although accessibility research has grown in the past 24 years [29],
inaccessible design remains a concern, likely caused by numerous
factors [37], such as inadequate education, inappropriately scoped
and budgeted projects, lack of accessibility awareness and expe-
rience, and a need for improved design tools or guidelines that
support accessible design (see: [2, 10, 24, 34, 39–43]). This lack of
access is a serious issue because website inaccessibility is globally
prevalent [1, 6, 19, 22, 35], with many studies focusing on govern-
ment websites because they are usually the primary communication
platforms to citizens and businesses [3], and typically are under the
jurisdiction of accessibility laws.
2.2 Digital Accessibility in a Global Context
Prior work has discussed accessibility in the global context showing
the dominance of some countries in HCI publications. For example,
recent work from Barbareschi et al. [7] argued that most of the fnd-
ings at the intersection of disability and technology in HCI venues
are from the Global North while 80% of disabled people in the world
live in the Global South, making it crucial that accessibility becomes
an important issue to consider on a global scale. They also argued
that there is a lack of literature on accessibility and disability stud-
ies in the Global South which results in a lower level of awareness
about accessible and inclusive technology for disabled people in
this region [7]. Moreover, only around 8% of all countries are re-
sponsible for 80% of the papers published in CHI [8]), and a majority
of the fndings at CHI belong to WEIRD countries (Western, Edu-
cated, Industrialized, Rich, and Democratic) [26]. One contributing
factor to this issue is that the majority of HCI researchers are lo-
cated in Western countries [25] which in general, indicates serious
inequalities in the structure of research communities [7]. One of
the resources of digital accessibility in the global context is the
DARE Index [14] which reports on the progress made by 137 coun-
tries that are parties to the Convention on the Rights of Persons
with Disabilities (CRPD) [14] by measuring three main factors: 1)
countries’ digital accessibility regulations and programs, 2) their
capacity and resources to implement accessibility considerations,
3) and the efects of digital accessibility on disabled people.
In our study, we focused on the frst factor from the DARE Index:
the role that a country’s regulations play in digital accessibility,
focusing specifcally on Iran. Previous studies suggest that a factor
contributing to inaccessible websites in a country is insufcient laws
and policies [6, 28]. Thus, investigating accessibility regulations
in Iran provides the opportunity to learn about the state of digital
accessibility in this country and help to improve the existing gaps.
Nourian et al.
3 METHODS
The goal of our study was to fnd the extent to which digital acces-
sibility was codifed in Iran’s national policies and if there were any
requirements or suggestions about using accessibility guidelines in
digital design. One important reason to explore accessibility-related
laws in a country is that regulations to support disabled people can
infuence other aspects of society that intersect with the work of
designers and can impact the output of their work [6, 28].
Since the lead author of this paper is fuent in the Persian lan-
guage and literature, we used certain search terms and keywords in
both Persian and English to investigate existing regulations. We uti-
lized a combination of full-text search, short terms, and stemming
in both languages (Table 1) and searched through the constitution
of Iran and the Iranian national website for all regulations called
National Laws Portal (qavanin.ir). We also used three search engines:
Google (www.google.com), DuckDuckGo (duckduckgo.com), and
Parsijoo (parsijoo.ir)—an Iranian search engine—to obtain more in-
formation about accessibility/disability in Iran from the perspective
of international organizations.
On the national laws portal (qavanin.ir), frst, we categorized
the laws by searching in their titles manually using certain key-
words (English root words and full words and the corresponding
Persian words for them) and found the ones that met our criteria.
The key words were the root words: “Disable= ÈñÊªÓ , à@ ñKAK ” and

“Access= €QƒX ” and the full words are “Disability= IJËñʪÓ
 ,úG @ñKAK ”

and “Accessibility= úæ… QƒX IJÊK

 . A¯ ,øQKYK €QƒX” 
Afterward, we searched the full text of the selected laws using
the search terms in Table 1.
4 FINDINGS
After investigating Iran’s Constitution, we found that out of 177
Articles, only one of them (Article 29) slightly pointed out the gov-
ernment’s responsibility for providing social support for disabled
people. However, no more details were provided on this matter [36].
On the Iranian national laws portal, we found 32 laws that spoke
of disability, mostly about physical disability that were linked with
martyrdom, and only one law that addressed digital Accessibility.
We identifed the sections of the laws related to disability and acces-
sibility using our search terms (Table 1) and read through them to
fnd how, if any, each law codifed accessibility in the digital spaces.
The governmental “Comprehensive Law on Protection of the Rights
of Persons with Disabilities” was ratifed 2004 which included 16
articles [20]. In 2018, the “Law to Protect the Rights of the Disabled”
was published [13, 36] and replaced the previous law aiming to
support disabled people by ensuring their access to national ser-
vices [13]. The new law [13] is a Federal law for the public, private,
and governmental organizations. The “Law to Protect the Rights
of the Disabled” [13] includes accessibility in the physical world
and work environments, healthcare insurance for disabled peo-
ple, education and employment, etc. Additionally, it mentions that
accessibility should be included in technological spaces, but no
more detail or articles were written on that topic, and it does not
explicitly prohibit discrimination [46]. In 2009, Iran acceded to
the Convention on the Rights of Persons with Disabilities (CRPD),
but it is unclear to what extent Iran is willing to comply with its
Digital Accessibility in Iran ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Search Terms in both Persian and English

Search Terms Accessibility Laws and Regulations

á ËñÊªÓ †ñ®k  IKAÔ  g á K@ ñ¯
 
€QƒX ,úæ… QƒX IJÊK  . A¯ ,øQKYK  €QƒX
Persian
, IKA‚
éKAÓAƒ  .ð ,
à@ ñKAK ,ÈñÊªÓ ,úG@ ñKAK àñKA¯


¹JKð QºË@ øA’¯
Disability Law/Act/policy
Disability, Disable, Disability Policy
Access, Accessible
English
Accessibility Law
Accessibility Regulation, Accessibility policy
Web, Website, Digital

obligations under the convention, as it claimed that “it does not
consider itself bound by any provisions of the Convention which may
be incompatible with its applicable rules.” [33].
The results of our investigation demonstrated that accessibility
and disability support is not sufciently codifed in the national
regulations of Iran in either of physical or digital spaces. Many
disabled people in Iran face discrimination, humiliation, and in-
accessibility in public spaces [46] and are unable to participate
in social activities independently [46]. Unsurprisingly, there were
also no specifc regulations for accessibility in digital spaces or
websites in any of the national policies. It is important to mention
that there were some standards relating to accessibility for disabled
people, but they are only restricted to “movement disabilities” [45].
Accordingly, since insufcient laws and policies [6, 28] can be a
factor contributing to the existence of inaccessible websites, we
anticipate that Iranian developers and designers may not be aware
of accessibility of digital content to a good extent.
5.1 Future Work
Laws and policies are one of the assets to spread accessibility knowl-
edge and insufcient laws can create barriers to informing the de-
signers about accessibility in digital spaces [6, 28]. As one of our
future plans, we aim to conduct interviews with Iranian designers
to understand their accessibility awareness and any challenges that
they face.
Moreover, another important aspect to take into account when
discussing accessibility are the universities that play an important
role in serving governmental agencies [4], in training experts, and
in bringing new ideas to a society [12]. Accessibility education
and training are crucial if we want designers to fully understand
and create accessible content [27, 30, 47] Thus, we are motivated
to focus on education settings in Iran to fnd out whether they
impart knowledge on accessibility by connecting to instructors and
students in Iranian universities.

5 DISCUSSION AND FUTURE WORK
The main implication of our work is that it can inform Iranian
policymakers to refect more on limited accessibility regulations and
motivate the HCI community and organizations of disabled people
in Iran to help the government in supporting disabled people, thus
increasing the awareness of accessibility among Iranian designers.
Despite the Iranian government not having sufcient policies
for accessibility, there are Iranian organizations such as the Iranian
Society of People with Disabilities (www.iransdp.com), and the Ira-
nian Disability Support Association (www.iraniandsa.org), which
indicates an interest and need for increased disability support.
As such, since there is no indication about Iranian organizations
cooperating with G3ict, we recommend that a frst step would be
for the HCI community and the organizations of people with dis-
ability in Iran to work with the government and cooperate with
international organizations like G3ict to improve accessibility regu-
lations. Our study is benefcial since it helps to provide an initial
background of accessibility and disability support in Iran. Having a
source of information about the status of accessibility in Iran can
help identify opportunities in promoting and implementing digital
accessibility in this country [15].
6 CONCLUSION
We investigated the state of digital accessibility in Iran by exploring
its government policies and we wanted to know to what extent
accessibility was addressed in Iran’s laws and regulations. We pre-
sented the frst study on the above criteria in Iran showing that
accessibility is not considered an essential topic in this country.
Our results suggest that there is a possible lack of accessibility
awareness in Iran and this work could be benefcial for inform-
ing the Iranian policymakers, designers, and HCI and disability
communities of the importance of digital accessibility. Our study
with our long-term plans is a starting point for improving digital
accessibility in Iran.
REFERENCES
[1] Abdulmohsen Abanumy, Ali Al-Badi, and Pam Mayhew. 2005. e-Government
Website accessibility: in-depth evaluation of Saudi Arabia and Oman. The Elec-
tronic Journal of e-government 3, 3 (2005), 99–106. https://ueaeprints.uea.ac.uk/
id/eprint/22182
[2] Hayfa Y Abuaddous, Mohd Zalisham Jali, and Nurlida Basir. 2016. Web ac-
cessibility challenges. International Journal of Advanced Computer Science and
Applications (IJACSA) (2016). https://doi.org/10.14569/IJACSA.2016.071023
[3] Basel Almourad and Faouzi Kamoun. 2013. Accessibility Evaluation of Dubai
e-Government Websites: Findings and Implications. Journal of E-Government
Studies and Best Practices (09 2013), 1–15. https://doi.org/10.5171/2013.978647
ASSETS ’22, October 23–26, 2022, Athens, Greece
[4] Nabil Amara, Mathieu Ouimet, and Réjean Landry. 2004. New evidence on
instrumental, conceptual, and symbolic utilization of university research in
government agencies. Science communication 26, 1 (2004), 75–106. https:
//doi.org/10.1177/1075547004267491
[5] Erin E Andrews, Anjali J Forber-Pratt, Linda R Mona, Emily M Lund, Carrie R
Pilarski, and Rochelle Balter. 2019. #SaytheWord: A disability culture commentary
on the erasure of “disability”. Rehabilitation psychology (2019). https://doi.org/
10.1037/rep0000258
[6] Muhammad Bakhsh and Amjad Mehmood. 2012. Web accessibility for disabled:
a case study of government websites in Pakistan. In 2012 10th International
Conference on Frontiers of Information Technology. IEEE, 342–347. https://doi.
org/10.1109/FIT.2012.68
[7] Giulia Barbareschi, Manohar Swaminathan, Andre Pimenta Freire, and Catherine
Holloway. 2021. Challenges and Strategies for Accessibility Research in the
Global South: A Panel Discussion. In X Latin American Conference on Human
Computer Interaction (Valparaiso, Chile) (CLIHC 2021). Association for Computing
Machinery, New York, NY, USA, Article 20, 5 pages. https://doi.org/10.1145/
3488392.3488412
[8] Christoph Bartneck and Jun Hu. 2009. Scientometric Analysis of the CHI Pro-
ceedings. (2009), 699–708. https://doi.org/10.1145/1518701.1518810
[9] Ben Caldwell, Michael Cooper, L Guarino Reid, and Gregg Vanderheiden. 2008.
Web content accessibility guidelines (WCAG) 2.0. WWW Consortium (W3C)
(2008). http://www.w3.org/TR/2008/REC-WCAG20-20081211/
[10] Michael Crabb, Michael Heron, Rhianne Jones, Mike Armstrong, Hayley Reid,
and Amy Wilson. 2019. Developing Accessible Services: Understanding Current
Knowledge and Areas for Future Support. In Proceedings of the 2019 CHI Confer-
ence on Human Factors in Computing Systems (CHI ’19). Association for Computing
Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300446
[11] Iman Dianat, Pari Adeli, Mohammad Asgari Jafarabadi, and Mohammad Ali
Karimi. 2019. User-centred web design, usability and user satisfaction: The
case of online banking websites in Iran. Applied Ergonomics 81 (2019), 102892.
https://doi.org/10.1016/j.apergo.2019.102892
[12] Henry Etzkowitz. 2003. Innovation in innovation: The triple helix of university-
industry-government relations. Social science information 42, 3 (2003), 293–337.
https://doi.org/10.1177/05390184030423002
[13] Center for Human Rights in Iran. 2020. Law to Protect the Rights of the Dis-
abled. Retrieved June 7, 2021 from https://iranhumanrights.org/2020/01/english-
translation-irans-law-to-protect-the-rights-of-the-disabled/
[14] G3ict. 2020. Digital Accessibility Rights Evaluation Index (DARE Index). Re-
trieved January 3, 2022 from https://g3ict.org/upload/accessible_DARE-Index-
2020-Global-Progress-by-CRPD-States-Parties-ENGLISH.pdf
[15] G3ict. 2022. Digital Accessibility Rights Evaluation Index (DARE Index). Retrieved
June 16, 2022 from https://g3ict.org/digital-accessibility-rights-evaluation-index/
[16] Danny Haelewaters, Tina A Hofmann, and Adriana L Romero-Olivares. 2021.
Ten simple rules for Global North researchers to stop perpetuating helicopter
research in the Global South. PLoS Computational Biology 17, 8 (2021), e1009277.
https://doi.org/10.1371/journal.pcbi.1009277
[17] Vicki L. Hanson and John T. Richards. 2013. Progress on Website Accessibility?
ACM Trans. Web 7, 1, Article 2 (March 2013), 30 pages. https://doi.org/10.1145/
2435215.2435217
[18] Mohammad Hassanzadeh and Fatemeh Navidi. 2010. Web site accessibility
evaluation methods in action: A comparative approach for ministerial web sites
in Iran. The Electronic Library (2010). https://doi.org/10.1108/02640471011093499
[19] Chaomeng James Huang. 2003. Usability of e-government web-sites for people
with disabilities. In 36th Annual Hawaii International Conference on System Sci-
ences, 2003. Proceedings of the. IEEE, 11–pp. https://doi.org/10.1109/HICSS.2003.
1174330
[20] Iran. 2004. Comprehensive Law on Protection of the Rights of Persons with Disabili-
ties. Retrieved June 7, 2021 from http://www.ilo.org/dyn/natlex/natlex4.detail?
p_lang=en&p_isn=91491&p_country=IRN&p_count=168
[21] Lilly Irani, Janet Vertesi, Paul Dourish, Kavita Philip, and Rebecca E. Grinter.
2010. Postcolonial Computing: A Lens on Design and Development. Association
for Computing Machinery, New York, NY, USA, 1311–1320. https://doi.org/10.
1145/1753326.1753522
[22] Joanne M Kuzma. 2010. Accessibility design issues with UK e-government sites.
Government Information Quarterly 27, 2 (2010), 141–146. https://doi.org/10.1016/
j.giq.2009.10.004
[23] Jonathan Lazar, Daniel F. Goldstein, and Anne Taylor. 2015. Ensuring Digital
Accessibility through Process and Policy (1st ed.). Morgan Kaufmann Publishers
Inc., San Francisco, CA, USA. https://doi.org/doi/book/10.5555/2815674
[24] Junchen Li, Garreth W. Tigwell, and Kristen Shinohara. 2021. Accessibility of
High-Fidelity Prototyping Tools. In CHI Conference on Human Factors in Com-
puting Systems (CHI ’21) (CHI ’21). Association for Computing Machinery, New
York, NY, USA. https://doi.org/10.1145/3411764.3445520
[25] Jonathan Ling and Paul Van Schaik. 2002. The efect of text and background
colour on visual search of Web pages. Displays 23, 5 (2002), 223–230.
[26] Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus
Opwis, and Katharina Reinecke. 2021. How WEIRD is CHI?. In CHI Conference
Nourian et al.
on Human Factors in Computing Systems (CHI ’21) (CHI ’21). Association for
Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.
3445488
[27] Stephanie Ludi, Matt Huenerfauth, Vicki Hanson, Nidhi Rajendra Palan, and
Paula Conn. 2018. Teaching Inclusive Thinking to Undergraduate Students in
Computing Programs. In Proceedings of the 49th ACM Technical Symposium on
Computer Science Education (Baltimore, Maryland, USA) (SIGCSE ’18). Association
for Computing Machinery, New York, NY, USA, 717–722. https://doi.org/10.
1145/3159450.3159512
[28] Sergio Luján-Mora, Rosa Navarrete, and Myriam Peñafel. 2014. Egovernment
and web accessibility in South America. In 2014 First International Conference
on eDemocracy & eGovernment (ICEDEG). IEEE, 77–82. https://doi.org/10.1109/
ICEDEG.2014.6819953
[29] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and
Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? A Literature
Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019. Association
for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.
3445412
[30] Rachel Menzies, Garreth W. Tigwell, Mandar Tamhane, and Annalu Waller.
2019. Weaving Accessibility Through an Undergraduate Degree. In The 21st
International ACM SIGACCESS Conference on Computers and Accessibility (AS-
SETS ’19). Association for Computing Machinery, New York, NY, USA, 526–529.
https://doi.org/10.1145/3308561.3354611
[31] Sedigheh Mohamadesmaeil and Somaye Koohbanani. 2012. Web Usability Eval-
uation of Iran National Library Website. Collnet Journal of Scientometrics and
Information Management 6 (06 2012), 161–174. https://doi.org/10.1080/09737766.
2012.10700931
[32] Sedigheh Mohamadesmaeil and Mahrokh Nassehi Oskouei. 2015. Comparative
evaluation of the hospital library websites accessibility in Iran. (2015), 83–96.
https://qje.ntb.iau.ir/article_516853.html
[33] United Nations. 2008. Convention on the Rights of Persons with Disabilities. Re-
trieved January 8, 2022 from https://treaties.un.org/pages/ViewDetails.aspx?src=
IND&mtdsg_no=IV-15&chapter=4&clang=_en#9
[34] Rohan Patel, Pedro Breton, Catherine M. Baker, Yasmine N. El-Glaly, and Kris-
ten Shinohara. 2020. Why Software is Not Accessible: Technology Profession-
als’ Perspectives and Challenges. In Extended Abstracts of the 2020 CHI Con-
ference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA
’20). Association for Computing Machinery, New York, NY, USA, 1–9. https:
//doi.org/10.1145/3334480.3383103
[35] Manas Ranjan Patra, Amar Ranjan Dash, and Prasanna Kumar Mishra. 2014. A
Quantitative Analysis of WCAG 2.0 Compliance for some Indian Web Portals.
International Journal of Computer Science, Engineering and Applications 4, 1 (2014),
9. https://doi.org/10.48550/arXiv.1710.08788
[36] Iran’s National Laws Portal. 2018. Law to Protect the Rights of the Disabled.
Retrieved June 17, 2021 from https://qavanin.ir/Law/TreeText/261251
[37] Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O. Wobbrock. 2017.
Epidemiology as a Framework for Large-Scale Mobile Application Accessibility
Assessment. In Proceedings of the 19th International ACM SIGACCESS Confer-
ence on Computers and Accessibility (ASSETS ’17). Association for Computing
Machinery, New York, NY, USA, 2–11. https://doi.org/10.1145/3132525.3132547
[38] Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O. Wobbrock.
2018. Examining Image-Based Button Labeling for Accessibility in Android
Apps through Large-Scale Analysis. In Proceedings of the 20th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’18). As-
sociation for Computing Machinery, New York, NY, USA, 119–130. https:
//doi.org/10.1145/3234695.3236364
[39] Kristen Shinohara, Saba Kawas, Amy J. Ko, and Richard E. Ladner. 2018. Who
Teaches Accessibility? A Survey of U.S. Computing Faculty. In Proceedings of
the 49th ACM Technical Symposium on Computer Science Education (SIGCSE ’18).
Association for Computing Machinery, New York, NY, USA, 197–202. https:
//doi.org/10.1145/3159450.3159484
[40] David Swallow, Christopher Power, Helen Petrie, Anna Bramwell-Dicks, Lucy
Buykx, Carlos A Velasco, Aidan Parr, and Joshue O Connor. 2014. Speaking the
Language of Web Developers: Evaluation of a Web Accessibility Information
Resource (WebAIR). In International Conference on Computers for Handicapped
Persons. Springer, 348–355. https://doi.org/10.1007/978-3-319-08596-8_54
[41] Garreth W. Tigwell. 2021. Nuanced Perspectives Toward Disability Simulations
from Digital Designers, Blind, Low Vision, and Color Blind People. In CHI Con-
ference on Human Factors in Computing Systems (CHI ’21) (CHI ’21). Association
for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.
3445620
[42] Garreth W. Tigwell, David R. Flatla, and Neil D. Archibald. 2017. ACE: A Colour
Palette Design Tool for Balancing Aesthetics and Accessibility. ACM Trans. Access.
Comput. 9, 2 (Jan. 2017). https://doi.org/10.1145/3014588
[43] Garreth W. Tigwell, Rachel Menzies, and David R. Flatla. 2018. Designing for
Situational Visual Impairments: Supporting Early-Career Designers of Mobile
Content. In Proceedings of the 2018 Designing Interactive Systems Conference
(DIS ’18). Association for Computing Machinery, New York, NY, USA, 387–399.
Digital Accessibility in Iran
https://doi.org/10.1145/3196709.3196760
[44] Garreth W. Tigwell, Kristen Shinohara, and Laleh Nourian. 2021. Accessibility
Across Borders. In CHI ’21 Workshop: Decolonizing HCI Across Borders (CHI
Workshop ’21). 1–4. https://arxiv.org/abs/2105.01488
[45] Human Rights Watch. 2018. "I Am Equally Human” Discrimination and
Lack of Accessibility for People with Disabilities in Iran. Retrieved June
19, 2021 from https://www.hrw.org/report/2018/06/27/i-am-equally-human/
discrimination-and-lack-accessibility-people-disabilities-iran#_ftn168
ASSETS ’22, October 23–26, 2022, Athens, Greece
[46] Human Rights Watch. 2018. Iran: People with Disabilities Face Discrimination and
Abuse. Retrieved August 20, 2021 from https://www.hrw.org/news/2018/06/26/
iran-people-disabilities-face-discrimination-and-abuse
[47] Qiwen Zhao, Vaishnavi Mande, Paula Conn, Sedeeq Al-khazraji, Kristen Shi-
nohara, Stephanie Ludi, and Matt Huenerfauth. 2020. Comparison of Meth-
ods for Teaching Accessibility in University Computing Courses. In The 22nd
International ACM SIGACCESS Conference on Computers and Accessibility (AS-
SETS ’20). Association for Computing Machinery, New York, NY, USA. https:
//doi.org/10.1145/3373625.3417013
 Exploring Accessibility Features and Plug-ins for Digital
Prototyping Tools
Urvashi Kokate Kristen Shinohara
uk4890@rit.edu kristen.shinohara@rit.edu
School of Information School of Information
Rochester Institute of Technology Rochester Institute of Technology
Rochester, NY, USA Rochester, NY, USA
ABSTRACT
Many digital systems are found to be inaccessible and a large part of
the issue is that accessibility is not considered early enough in the
design process. Digital prototyping tools are a powerful resource
for designers to quickly explore both low and high fdelity design
mockups during initial stages of product design and development.
We evaluated 10 popular prototyping tools to understand their built-
in and third-party accessibility features. We found that accessible
design support is largely from third-party plug-ins rather than pro-
totyping tools’ built-in features, and the availability of accessibility
support varies from tool to tool. There is potential to improve ac-
cessible design by increasing the potential for accessibility to be
consider earlier in the design process.
CCS CONCEPTS
• Human-centered computing → Accessibility.
ACM Reference Format:
Urvashi Kokate, Kristen Shinohara, and Garreth W. Tigwell. 2022. Exploring
Accessibility Features and Plug-ins for Digital Prototyping Tools. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
4 pages. https://doi.org/10.1145/3517428.3550391
1 INTRODUCTION AND RELATED WORK
Accessible design should not be an afterthought in the design pro-
cess [4]. When accessibility issues are checked later in the design
process, it requires more time, money, and efort in redesigning
and reprogramming the product [9, 13, 23, 24]. Therefore, it is ben-
efcial to move accessible design as early as possible in the design
process [16].
Accessibility tools are an essential part of the design process
to support people meeting accessible design standards set by the
Web Content Accessibility Guidelines (WCAG) [12], and mobile
app design guidelines also incorporate WCAG criteria [2, 7, 17] due
to an increasing need for mobile accessibility [1]. Designers and
developers have access to a multitude of accessibility tools [5, 10].
However, they generally fall into two categories: those to be used
during design and development, and those that run evaluations after
development. Automated accessibility checking generally results
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550391
Garreth W. Tigwell
garreth.w.tigwell@rit.edu
School of Information
Rochester Institute of Technology
Rochester, NY, USA
in missing some issues compared to manual assessment [14], and
there is also a lack of consistency among accessibility evaluation
tools [6, 18]. However, there is potential with accessibility tools that
support people in meeting WCAG (e.g., color contrast checkers),
especially if they are designed to be used earlier in the design
process (e.g., the Accessible Colour Evaluator, which was developed
through a User-Centered Design process [22]).
Prototyping is a design method used early in the design process
to lay down ideas and also support quicker evaluations [11, 20].
There are many approaches to facilitating prototyping practice rang-
ing from working ofine (e.g., pen and paper) to using low-cost,
readily available software (e.g. Microsoft PowerPoint) to high-end
professional software (e.g., Adobe XD). We anticipated that proto-
typing tools are likely not providing enough feature support for
users since prior work already found prototyping tools themselves
are not made to be accessible [15]. Some prototyping software
also supports the use of plug-ins, and there is potential with this
approach to ofer accessibility plug-ins (e.g., Adee [8]).
There are many scenarios where prototyping tools could support
accessible design. For example, notifying the user when color pairs
are inaccessible due to not meeting WCAG minimum contrast crite-
ria, warning when button target sizes may be too small for certain
devices, recommending accessible fonts styles and font sizes.
We formally evaluated 10 popular design tools that can be used
to support digital prototyping to understand what accessibility sup-
port they provide. We found that only a few tools have built-in
features to support accessible design and most of the tools rely on
third party plug-ins that provide features to check for accessibil-
ity. The majority focused on color contrast checking and ofered
problematic color blind simulations. We recommend that compa-
nies and the HCI community focus more on providing a range of
accessibility check features to support accessible design during on
of the earliest points of the design process.
2 EVALUATION OF DESIGN PROTOTYPING
TOOLS
Selection of tools. We selected 10 UI design tools from the 2020 design
tools survey conducted by UX tools [19]. Our selection included:
Adobe Illustrator, Adobe Photoshop, Adobe XD, Afnity Designer,
Axure, Figma, Framer, InVision Studio, Sketch, and UXPin.
Evaluation method. Our evaluation process had three steps.
Step 1: Find and document any accessibility-related support re-
ported in the prototyping tool’s documentation and/or website. We
frst browsed through the website and then did a keyword search
to look up for specifc sections of the page. We selected keywords
related to visual design, accessibility, and guidelines since interface
ASSETS ’22, October 23–26, 2022, Athens, Greece
prototyping does not often have the ability for audio (see list of key-
words in supplementary materials). We frst checked the top-level
menus and landing page of each tool’s website. Then we did a key-
word search in the help/support/documentation, and blog section
of each website. Step 2: Check for any built-in accessible design
features in the tools. We checked the top menu, tool bar, and right
clicked on design elements to identify any accessibility features
the prototyping tools ofer. Step 3: Check for third-party plug-ins
for each tool that supports accessible design. We searched plug-in
managers with Step 1’s list of keywords. We read the description of
each plug-in and noted down its features. We also checked whether
it was free, free with paid features, or paid.
2.1 Findings
2.1.1 Searching through each prototyping tool’s documentation and
website. We found two out of 10 tools (Afnity Designer, Framer)
did not have any content related to accessibility. Six prototyping
tools (Adobe XD, Axure, Figma, InVision Studio, Sketch, UXPin)
featured blogs based on diferent topics of accessibility such as
‘what is accessibility’, ‘design practices for accessibility’ and ‘why
designers should think about accessibility’. Adobe Illustrator and
Adobe Photoshop mention about their inbuilt color blind simula-
tion features in their documentation. UXPin highlights the built-in
features of color contrast checker and color blind simulation in
the ‘features’ section of their websites. Adobe XD and Figma also
highlight the Stark plug-in which provides diferent accessibility
check features to the users such as color contrast checker, color
blind simulator, and focus order.
A majority of the tools provide information on why and how to
include accessibility in design, but only half of the tools mention
how the tool itself can support them in designing or checking for
accessibility. Also, the information provided is limited and hard to
fnd on the website. None of the tools have a section in the top-level
menu for accessibility on their website. To look up content, a user
needs to search for the content using diferent keywords. ‘Accessi-
bility’ and ‘contrast’ were the keywords that returned informative
content about accessibility for seven of the prototyping tools. We
based how easy it was to search for information about accessibility
on the number of results that were returned and the time taken to
fnd relevant results out of the total returned results. More time
was taken to look up content which returned more results. Figma
proved more most difcult to fnd relevant content because search-
ing for ‘accessibility’ in the help center gave over 200 results out
of which only 3 included the word accessibility since most of the
results were for the word ‘access’. It was easiest to look up for
content on UXPin because it returned fewer results and the built-in
features were highlighted on the feature page of the website, which
was easier to spot.
Table 2 in Supplementary Materials provides a breakdown of
accessibility content for each prototyping tool.
2.1.2 Evaluating tool’s built-in accessibility features. Afnity De-
signer and Sketch did not have any built-in accessibility features.
We were able to identify fve categories that built-in accessibility
features would ft. The categories were: (1) color contrast checker
(found in UXPin); (2) adding audio feedback and speech output
(found in Adobe XD); (3) color blind simulation (found in Adobe
Kokate et al.
Illustrator, Adobe Photoshop, UXPin); (4) voice prototyping (found
in Adobe XD); and (5) adding keyboard navigation (found in Adobe
XD, Axure, Figma, Framer, InVision Studio, UXPin).
UX Pin is the only tool that had a built-in color contrast checker
that automatically identifes text colors with insufcient contrast
with respect to WCAG guidelines. It also had a color blind simulator
that allowed users to view their designs in eight diferent types of
color blindness, as well as a prototyping feature to add keyboard
shortcut as a trigger to transition across screens. Adobe Illustrator
and Adobe Photoshop both provided color blind simulation features,
but they only allowed users to view in Protanopia or Deuteranopia.
Adobe XD was the only prototyping tool that allows users to add
audio feedback and speech output in the design to provide guidance
for users who can access the prototype with only sound. Adobe
XD also supports voice prototyping, which allows users to design
transitions through voice commands. Axure, Figma, Framer, and
InVision also had the prototyping feature to add keyboard shortcuts
for transition. Table 3 in Supplementary Materials provides an
overview.
2.1.3 Evaluating third-party accessibility plug-ins. Afnity Designer,
Axure, and UX Pin did not support any third-party plug-ins. We
were able to identify three general categories that plug-in accessi-
bility features would ft, and we provide examples for each. Table
4 in Supplementary Materials provides an overview of the plug-
in accessibility features category type found for each prototype
tool, While Tables 5-11 provide details on each prototyping tool’s
plug-ins.
Plugins used for accessibility checking: Color contrast checker:
checks contrast between two color layers (foreground text color and
background color); Touch target size checker: checks touch target
size with respect to devices and shows if it violates guidelines; and
Epilepsy checker: checks if images and animated GIFs in designs are
safe for people with photosensitive epilepsy to view.
Plugins used to enhance accessibility: Alt text generator: adds
alt-text for images to share with the developers; Focus orderer: adds
focus order for keyboard navigation; Screen reader support: adds
ARIA roles, ARIA properties and tab index in designs; and Text
resizer: creates legible texts with respect to screen size.
Plugins used to support understanding impairments: Color
blind simulation: tests designs with diferent color blind simulation;
and Visual Impairment Simulation: checks elements against difer-
ent types of visual impairments such as central loss, blind spots,
hemianopia, peripheral loss, retinal detachment, ocular albinism.
We list all third party accessibility plug-ins in Table 1 by proto-
typing tool. Figma ofered the most number of plug-ins equal—17 in
total. Though Figma had relatively more plug-ins, 11 out of the 17
plug-ins actually only included features to check colors accessibility.
Table 4 in Supplementary Materials shows the 17 plug-ins cover
seven accessibility feature categories. Most of the plug-ins were
free to use. The Stark plug-in is common across the top 3 most
popular tools—Adobe XD, Figma, and Sketch. Stark provides color
contrast checking, a color blindness simulator, and focus orderer,
but there is limited functions within the free version of Stark. The
paid features provide smart suggestion of colors to use if the cur-
rent colors in the design do not adhere to accessibility guidelines.
None of the free plug-ins provide color suggestions to the users.
Exploring Accessibility Features and Plug-ins for Digital Prototyping Tools ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: List of all third party plug-ins. Details of each tool and plug-in are found in supplementary materials.
Prototyping tools List of plug-ins
Figma Able, Zebra, Contrast, A11y - Color Contrast Checker, Color blind, Epilepsy Blocker, Stark,
Color Contrast Grid, Cards for Humanity, A11y - Focus Orderer, Adee Comprehensive
Accessibility Tool, Contrast, Contrast Grid, HCL Easy, Color Contrast, Low Vision
Sketch Stark, Adee Comprehensive Accessibility Tool, Cluse, Color Contrast Analyzer, Check Con-
trast, Color Blindless, Sketch WCAG
Adobe XD Stark, Colorsinspo, Dopely colors
Adobe Illustrator Pantone Connect
Adobe Photoshop Pantone Connect, Check Contrast Ratio
InVision Studio Contrast
Framer Color Contrast checker, Color check, Accessibility Tool Kit

Epilepsy blocker is a paid plug-in available only on Figma. It was
the only plug-in to allow users to check if images and animated
GIFs in designs are safe for people with photosensitive epilepsy.
The Adee Comprehensive Accessibility tool plug-in provided the
maximum number of features and is available to install in Figma
and Sketch only. It was totally free and included features such as alt
text generator, touch target size checker, a color blindness simulator
and a contrast checker. It also allows users to generate a report for
any guideline violations found in a design with respect to touch
target size and color contrast. The plug-ins Visual Impairment Sim-
ulation and Epilepsy checker were only available for Figma. Screen
reader support feature was provided by a plug-in available only
on Framer. Color blindness simulators and color contrast checkers
were the most popular features ofered by plug-ins. Six out of the
Seven prototyping tools that support third party plug-ins had color
contrast checkers and color blindness simulators.
3 DISCUSSION
We found that although prototyping tools do ofer some level of
support for accessible design, there is still room for improvement.
We found evidence of prototyping tool websites providing informa-
tion about accessible design practice and info on support their tools
ofered. However, the information was hard to fnd since none of
the websites had a homepage that linked to an accessibility features
support page.
Furthermore, we found the built-in accessibility features in pro-
totyping tools to be limited. The common features were the ability
to add keyboard navigation, check color contrast, and run a color
blindness simulator. In fact, the majority focused on color contrast
checking and ofered color blind simulations. We want to acknowl-
edge that color blindness simulations are often limited in accuracy
and disability simulations are problematic [3, 21]. It is concerning
that so many of the tools and plug-ins focused on color blindness
simulations without adequate information on the limitations of
using such a feature [21]. Although use of color is a prominent as-
pect of design, we fnd that more efort needs to be directed toward
creating accessibility features for other accessible design criteria.
It would be useful to conduct follow-up work to understand user
perspective on what features to prioritize and how.
While we found fewer accessibility features provided within
the tools themselves, we did notice a decent ofering from third
party plug-ins that either supported the creation or evaluation of
accessible design. It is clear that Figma and Sketch have benefted
from opening their platforms up to the design community to allow
designers and developers to contribute new prototyping features.
Figma had 17 accessibility plug-ins and Sketch had seven accessi-
bility plug-ins both with a variety of features, whereas the other
prototyping tools had only one to three plug-ins that provided basic
color contrast checks and color blindness simulations.
However, if prototyping tools are relying on third party devel-
opers, rather than adding built-in accessibility features, then we
do want to refect on the potential limitations of this. If a specifc
accessibility feature is only available as a third party plug-in, then
the user has to actively seek it out by searching through the plug-in
store. A new user of the prototyping tool may not know about being
able to install plug-ins. Finally, there is a question of validity. There
is no clear vetting process and what determines a plug-ins success
is likely community based (e.g., ratings, comments, feedback). The
prototyping tool companies are potentially in a better position to
develop accessibility features that accurately conform to current
standards and best practice guidelines.
For future work, we will employ qualitative methods to inter-
view diferent stakeholders (e.g., designers and prototyping tool
companies) to understand their attitudes and concerns for built-in
and plug-in accessibility features. We also plan to run additional
evaluations of the accessibility features to understand how they are
used within the design process and individual workfow styles to
identify whether there are opportunities to improve usability.
4 CONCLUSION
Designers should include accessibility practices in the early phase
of design such as while creating design prototypes. Prototyping
tools can be used to provide good assistance to designers to verify
design accessibility. We evaluated 10 design prototyping tools to
research on the current accessibility assistance provided by these
tools. There is support provided by prototyping tools largely in the
form of third-party plug-ins, but minimal assistance in the form of
built-in features. Also, the availability of these features varies from
tool to tool. We argue that there is potential to improve accessible
design by increasing the potential for accessibility to be consider
earlier in the design process, but the current approach needs more
refnement.
REFERENCES
[1] Shadi Abou-Zahra, Judy Brewer, and Shawn Lawton Henry. 2013. Essential
Components of Mobile Web Accessibility. In Proceedings of the 10th International
ASSETS ’22, October 23–26, 2022, Athens, Greece
Cross-Disciplinary Conference on Web Accessibility (Rio de Janeiro, Brazil) (W4A
’13). Association for Computing Machinery, New York, NY, USA, Article 5, 4 pages.
https://doi.org/10.1145/2461121.2461138
[2] Apple. n.d.. Human Interface Guidelines: Visual Design. https://developer.
apple.com/design/human-interface-guidelines/ios/overview/themes/. Accessed:
2020-07-18.
[3] Cynthia L. Bennett and Daniela K. Rosner. 2019. The Promise of Empathy: Design,
Disability, and Knowing the "Other". In Proceedings of the 2019 CHI Conference
on Human Factors in Computing Systems. Association for Computing Machinery,
New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300528
[4] Vicente Luque Centeno, Carlos Delgado Kloos, Martin Gaedke, and Martin Nuss-
baumer. 2005. Web Composition with WCAG in Mind. In Proceedings of the 2005
International Cross-Disciplinary Workshop on Web Accessibility (W4A) (Chiba,
Japan) (W4A ’05). Association for Computing Machinery, New York, NY, USA,
38–45. https://doi.org/10.1145/1061811.1061819
[5] Lisa Dziuba. 2019. Accessibility tools for designers and develop-
ers. https://uxdesign.cc/accessibility-tools-for-designers-and-developers-
ea400a415c0a. Last Accessed: 2022-6-11.
[6] Tânia Frazão and Carlos Duarte. 2020. Comparing Accessibility Evaluation Plug-
Ins. In Proceedings of the 17th International Web for All Conference (Taipei, Taiwan)
(W4A ’20). Association for Computing Machinery, New York, NY, USA, Article
20, 11 pages. https://doi.org/10.1145/3371300.3383346
[7] Google. n.d.. Material Design. https://material.io. Accessed: 2020-07-18.
[8] Samine Hadadi. 2021. Adee: Bringing Accessibility Right Inside Design Tools. In
The 23rd International ACM SIGACCESS Conference on Computers and Accessibility
(Virtual Event, USA) (ASSETS ’21). Association for Computing Machinery, New
York, NY, USA, Article 101, 4 pages. https://doi.org/10.1145/3441852.3476478
[9] Shawn Lawton Henry and Andrew Arch. 2012. Financial factors in de-
veloping a web accessibility business case for your organization. W3C.
http://www.w3.org/WAI/bcase/fn#decreasing. Accessed: 2014-08-16..
[10] Shayna Hodkin. 2019. 8 tools that make accessible design easier. https://www.
invisionapp.com/inside-design/accessibility-tools/. Last Accessed: 2022-6-11.
[11] Gopinaath Kannabiran and Susanne Bødker. 2020. Prototypes as Objects of Desire.
Association for Computing Machinery, New York, NY, USA, 1619–1631. https:
//doi.org/10.1145/3357236.3395487
[12] Andrew Kirkpatrick, Joshue O’Connor, Alastair Campbell, and Michael Cooper.
2018. Web Content Accessibility Guidelines (WCAG) 2.1.
[13] Chris Law, Julie Jacko, and Paula Edwards. 2005. Programmer-Focused Website
Accessibility Evaluations. In Proceedings of the 7th International ACM SIGACCESS
Conference on Computers and Accessibility (Baltimore, MD, USA) (Assets ’05).
Association for Computing Machinery, New York, NY, USA, 20–27. https://doi.
org/10.1145/1090785.1090792
Kokate et al.
[14] Jonathan Lazar, Patricia Beere, Kisha-Dawn Greenidge, and Yogesh Nagappa.
2003. Web accessibility in the Mid-Atlantic United States: a study of 50 homepages.
Universal Access in the Information Society 2, 4 (2003), 331–341. https://doi.org/
10.1007/s10209-003-0060-z
[15] Junchen Li, Garreth W. Tigwell, and Kristen Shinohara. 2021. Accessibility of
High-Fidelity Prototyping Tools. In Proceedings of the 2021 CHI Conference on
Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association
for Computing Machinery, New York, NY, USA, Article 493, 17 pages. https:
//doi.org/10.1145/3411764.3445520
[16] Adriana Martín, Alejandra Cechich, and Gustavo Rossi. 2011. Accessibility at
Early Stages: Insights from the Designer Perspective. In Proceedings of the Inter-
national Cross-Disciplinary Conference on Web Accessibility (Hyderabad, Andhra
Pradesh, India) (W4A ’11). Association for Computing Machinery, New York, NY,
USA, Article 9, 9 pages. https://doi.org/10.1145/1969289.1969302
[17] Microsoft. n.d.. Universal Windows Platform documentation. https://docs.
microsoft.com/en-gb/windows/uwp. Accessed: 2020-07-18.
[18] Marian Padure and Costin Pribeanu. 2020. Comparing six free accessibility
evaluation tools. Informatica Economica 24, 1 (2020), 15–25. https://doi.org/10.
24818/issn14531305/24.1.2020.02
[19] Taylor Palmer and Jordan Bowman. 2020. 2020 Design Tools Survey. https:
//uxtools.co/survey-2020#conclusion. Last Accessed: 2022-6-11.
[20] K. Schneider. 1996. Prototypes as assets, not toys. Why and how to extract
knowledge from prototypes. (Experience report). In Proceedings of IEEE 18th
International Conference on Software Engineering. 522–531. https://doi.org/10.
1109/ICSE.1996.493446
[21] Garreth W. Tigwell. 2021. Nuanced Perspectives Toward Disability Simulations
from Digital Designers, Blind, Low Vision, and Color Blind People. In Proceedings
of the 2021 CHI Conference on Human Factors in Computing Systems. Association
for Computing Machinery, New York, NY, USA, Article 378, 15 pages. https:
//doi.org/10.1145/3411764.3445620
[22] Garreth W. Tigwell, David R. Flatla, and Neil D. Archibald. 2017. ACE: A Colour
Palette Design Tool for Balancing Aesthetics and Accessibility. ACM Trans. Access.
Comput. 9, 2, Article 5 (Jan. 2017), 32 pages. https://doi.org/10.1145/3014588
[23] Shari Trewin, Brian Cragun, Cal Swart, Jonathan Brezin, and John Richards.
2010. Accessibility Challenges and Tool Features: An IBM Web Developer
Perspective. In Proceedings of the 2010 International Cross Disciplinary Confer-
ence on Web Accessibility (W4A) (Raleigh, North Carolina) (W4A ’10). Asso-
ciation for Computing Machinery, New York, NY, USA, Article 32, 10 pages.
https://doi.org/10.1145/1805986.1806029
[24] Brian Wentz, Paul T Jaeger, and Jonathan Lazar. 2011. Retroftting accessibility:
The legal inequality of after-the-fact online access for persons with disabilities
in the United States. First Monday 16, 11 (Nov. 2011). https://doi.org/10.5210/fm.
v16i11.3666
 Does XR introduce experience asymmetry in an
intergenerational seting?
Vibhav Nanda Hanuma Teja Maddali
Independent Researcher Department of Computer Science,
vibhavnanda1996@gmail.com University of Maryland College Park,
USA
hmaddali@umd.edu
ABSTRACT
Intergenerational social interactions are benefcial for bridging gen-
erational gaps, strengthening family bonds, and improving social
cohesiveness. However, opportunities for in-person intergenera-
tional social interactions are decreasing as families become increas-
ingly geographically dispersed. Researchers are examining how
technology might support these interactions. Extended Reality (XR)
is an emerging technology that has shown potential for support-
ing immersive remote interactions but might cause an “experience
asymmetry” in an intergenerational setting. In this poster we con-
trast the user experience of younger and older participants engaging
in remote gardening sessions with our social XR prototypes. We
present systemic infuence factors that afected user experience of
participants from diferent age groups diferently with our XR pro-
totypes. We discuss potential approaches to mitigate their efects
based on observational learning and becoming aware of designer
biases.
CCS CONCEPTS
• Human-centered computing → Human computer interaction
(HCI); Accessibility; Empirical studies in accessibility.
KEYWORDS
Extended Reality, Intergenerational Interaction, Gardening, Acces-
sibility
ACM Reference Format:
Vibhav Nanda, Hanuma Teja Maddali, and Amanda Lazar. 2022. Does XR
introduce experience asymmetry in an intergenerational setting?. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
4 pages. https://doi.org/10.1145/3517428.3550373
1 INTRODUCTION AND RELATED WORK
Intergenerational social interactions are benefcial for bridging gen-
erational gaps [7], strengthening family bonds [7], improving social
cohesiveness [18], and reducing social isolation [14]. A growing
body of literature is studying how to support remote intergener-
ational interactions using conventional technology (e.g., laptops,
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550373
Amanda Lazar
College of Information Studies,
University of Maryland College Park,
USA
lazar@umd.edu
tablets) [14] and remote socialization mechanisms such as on on-
line platforms [1]. However, these technologies and mechanisms
(e.g., video chat, social media) may not preserve the dynamic of
real-world social interactions, but extended reality (XR) can help
maintain this dynamic which can inspire joy [1] among older adults
and positively infuence their social well-being [12] during remote
interactions. XR is an umbrella term for a set of emerging tech-
nologies (augmented, virtual, mixed-reality) that allow diferent
compositions of real and virtual objects in the user’s feld-of-view
[19]. Immersion, Imagination, and Multimodal Interaction (3i’s)
[3] are core features of XR that, in view of their potential benefts
for older adult users, calls for the examination of XR-mediated
intergenerational social interactions.
The focus of designers trying to support remote intergenera-
tional interactions also tends to be game-centric [7, 14, 18], the
competitive nature of which may not appeal to older adults [14].
Here, we examine an approach that instead focuses on casual in-
teractions that could be better suited to support daily social partic-
ipation. The benefts of activities that are more representative of
daily social participation, such as gardening, can be very impactful.
We use gardening activity since it possesses health benefts [10], is
a popular everyday activity among older adults [20, 21], and has
been found to increase intergenerational social interactions [20].
This poster presents a secondary analysis of data from inter-
views with 13 gardener dyads about the usability of our social XR
prototypes in supporting remote learning and social interaction
in a garden setting. We contribute an understanding of systemic
factors of infuence that could negatively afect older adults’ (65
years and above) user experience with social XR. We examined
an “experience asymmetry” where some older participants’ user
experience was negatively afected, and younger participants’ user
experience was positive or not visibly afected.
2 DATA, METHODS, AND ANALYSIS
Our data was collected over the course of two studies to identify
perceptions of XR and evaluate social XR prototypes for remote
teaching, learning, and connecting in informal settings with friends,
family, other hobbyists. We analyzed 15 hours of transcripts from
interviews with 13 gardener dyads. This included 13 dyads (average
age diference = 22 years) with 6 older adults (average age = 70.3
years; 5 females; 1 male) and 21 relatively younger adults and teens
(average age = 30.7 years; 14 females; 7 males). The prototypes
evaluated represented an XR system where a remote user using a
VR headset can view and interact (e.g., annotate) with a garden of
an on-site user who uses an AR device.
ASSETS ’22, October 23–26, 2022, Athens, Greece Vibhav Nanda et al.

Figure 1: From Study 1, frst-person-view of “remote” participant wearing Oculus quest (left) and “remote” user wearing VR
headset and onsite participant using tablet as AR device (right).

Figure 2: From Study 2, a physical garden (top-left), its 3D model from photogrammetry (top-right), frst-person views of dyad
walking through a study scenario using the 3D model (bottom-left and bottom right)

In Study 1, 9 gardener dyads walked through three scenarios
based on frequent activities when teaching/learning gardening (e.g.,
garden planning) in a lab-simulated remote setting and assessed
the potential strengths and drawbacks of XR prototypes when facil-
itating key interactions for remote instruction as well as informal
social interactions. Participants worked together in a 360-image
representation of familiar garden using an Oculus Quest (remote
VR user) or tablet (onsite AR user) (Figure 1). Although only 1 dyad
in this study had an older adult, information from this dyad helped
with contrasting older adults’ user experience with that of younger
participants in other dyads. Study 2 involved 4 gardener dyads with
at least one older adult (above 65 years) and a younger gardener
who was at least relatively one generation younger. The dyads
walked through scenarios from Study 1 in a naturalistic setting
with one participant in a physical garden. Participants were also
able to use basic avatars and a photogrammetrically reconstructed
digital twin (Figure 2) of a pre-selected garden-site instead of a 360
image. This allowed for detailed feedback on practical challenges
with using the system and how the virtual representation of the
activity space and participants might impact remote interactions in
an intergenerational setting.
Constructivist grounded theory [4] was used to analyze data
from the transcripts of the 2 studies. For study 2, the frst author
engaged in an iterative process of open coding, memo writing,
pattern identifcation, and axial coding to fnd emergent themes.
Through team discussions, interesting themes were identifed, and
associated codes were fnalized, which were then used for focused
coding transcripts from study 1. After completing focused coding,
we looked for infuence factors by identifying and examining ex-
perience asymmetry: when systemic factors have opposite infuence
on the user experience of two diferent user groups. We decided to
focus more on factors that negatively afected older adult partici-
pants’ experience than younger participants’ because commercially
available XR may be designed around younger users’ and designers’
preferences. This analysis can also serve to become aware of design
(or methodology) biases that can arise due to study prototype de-
signers being younger adults rather than attributing it to a defcit
model of aging that tends to position older adults as technophobes.
Does XR introduce experience asymmetry in an intergenerational seting?
3 FINDINGS
In this section, we identify three systemic infuence factors and
present example instances of where they infuenced older adults’
user experience diferently when compared with younger partici-
pants using our social XR prototypes.
3.1 XR immersion felt diferent for some older
and younger participants
We fnd that the immersive capability of XR negatively afected
some older participants’ user experience while simultaneously posi-
tively afecting younger participants’ user experience. For example,
S1P7 (older participant) chose not to standup from her chair when
she was immersed in the virtual environment created by the VR
headset. When the second author explained that she could navigate
the virtual environment by moving in the physical world, S1P7 said
that even though “I understand that intellectually. That doesn’t mean
that I want to do it. (With) my luck, the thing would fail.” A fear of
“crashing” into the wall while immersed in the virtual environment
negatively afected S1P7s’ user experience as it prevented her from
engaging with the technology’s main purpose. On the other hand,
S2P2 (younger participant) felt excited by the immersion and de-
scribed a positive experience with the immersion because according
to her while immersed “you can actually feel like you’re there [in
the garden].”
3.2 Interactions that change the virtual
environment can be diferently interpreted
Our prototypes allowed users to interact with a virtual represen-
tation of the garden area in various ways, one of which was to
manipulate virtual proxies for certain physical objects in the envi-
ronment in a way that isn’t possible in the real world. However, this
capability was harder to interpret for some older participants which
negatively afected their user experience, but positively afected
younger participants’ user experience. For example, a prototype
in study 2 allowed the participants to move the sun in the virtual
environment by using a slider which further changed the distribu-
tion of sunlight across the virtual garden. After attempting to move
the sun, S2P1 (older participant) stated that “I’d just rather step out
my back and see what the sun is doing.” This suggestion to rather
observe the efects of the sun in the physical world disregarded the
utility of user interactions to manipulate the virtual environment
and was indicative of the difculty S2P1 had interpreting this capa-
bility in a virtual environment. While talking about her experience
trying to move the sun to manipulate the virtual environment, S2P1
said that “I don’t feel very comfortable being on this [VR headset]
. . . with regard to gardening, when I can just easily step out my back
door and see what I need to see [the sun].” This feedback from S2P1
linked the negative afect on her user experience, in the form of
discomfort, to the difculty she had interpreting this capability.
However, S2P2 (younger participant) found the ability to change
the amount of sun light “interesting” as it enabled her to “know
what it [the garden] looks like throughout the day,” and access sunny
weather even if it was raining outside. The capability to change the
virtual environment through interactions positively afected S2P2’s
user experience.
ASSETS ’22, October 23–26, 2022, Athens, Greece
3.3 Diferent reactions to sensory limitations
of XR
Participants had noted that all fve senses were used for garden-
ing in the physical world. However, lack of smell, taste, and touch
couldn’t be recreated in the prototypes’ virtual environments, due
to the limitations of current commercially available XR. This limi-
tation negatively afected some older participants’ user experience
but did not seem to have a signifcant negative efect on younger
participants’ user experience, which remained positive overall. For
instance, S1P7 (older participant) talk about how “I could do just
as much if we had a computer screen and remote because I could
draw on the computer screen. . . and I really wouldn’t need the virtual
reality”. The missing experience of “smell” and “feel [touch]” in
the virtual garden environment and the common elements of sight
and sound as in conventional computer made her question the ad-
vantage of remote sessions in XR and seemed to negatively impact
her user experience. S1P5 (younger participant) also identifed that
the sense of touch was missing from his experience in the virtual
environment. He explained that in the physical world he uses touch
to assess the soil quality, saying that he “feels” for “how well it [soil]
sticks together, how it feels in my hand,” and added that he couldn’t
use touch in the virtual environment as he would in the physical
world. However, S1P5 still indicated a more positive attitude to-
wards XR and mentioned “I think it’s [prototype’s] cool.” Sensory
limitations of XR did not have a signifcant negative impact on his
user experience.
4 DISCUSSION AND FUTURE WORK
Our fndings presented three systemic infuence factors that cre-
ated an experience asymmetry for intergenerational participants
by negatively afecting some older adults’ user experience with the
prototypes and positively or not visibly afecting younger partic-
ipants. We see that for older adults, these factors were related to
the sensory limitations of the XR and the core features of an XR
experience such as immersion and interpretability of interactions
with virtual objects.
While we only identifed some factors causing experience asym-
metry in this poster, further empirical data is required to understand
the possible mechanisms underlying the causes and efects of these
factors and identify optimal user-centered approaches to mitigate
their negative efects. For example, it is possible that these factors
are linked to afnity for novel technologies or past experience with
XR rather than age. Regardless, our data indicates that these po-
tential barriers, and particularly those related to immersion and
multimodal interactions with the virtual environment, are areas
of interest for future work and generalizable to more application
domains and settings.
Future work could explore approaches that balance the experi-
ence asymmetry, caused by these factors, during the introduction
to XR devices and technology. This could be done during the par-
ticipatory design process or during user’s initial onboarding for
the device. Unlike manuals or solo training, learning from a known
partner while onboarding could provide opportunities for remote
social interaction, as we found in our own gardener dyad sessions.
Past work has also shown that observational learning [15, 16] dur-
ing onboarding can help older adults learn new technology related
ASSETS ’22, October 23–26, 2022, Athens, Greece
skills. Such interventions can also help older adults feel positively
about their ability [5, 8, 9, 16] to engage with the 3i’s of XR [3]
because self-belief is essential to actualizing the benefts of tech-
nology [2, 8, 9]. Alternatively, we can lean into this asymmetrical
experience aspect and present diferent interfaces for users with
diferent attitudes towards XR. This approach, rather than seeing
older participants’ attitudes towards XR technology as a problem in
need of fxing, would mean that developers slow down on pursuing
novel experiences to focus on ensuring that all users can access,
and are interested in, cutting edge technologies.
In this poster, we chose gardening as an activity to provide a
setting for casual social interaction and to represent a meaningful
everyday activity that is outside of the game-focused approach
to past remote technology-mediated intergenerational literature
[7, 14, 18]. Future work can similarly look to understand other
experience asymmetry factors that might be visible in less multi-
sensory, or even learning focused, intergenerational settings (e.g.,
remote storytelling [6, 11, 13, 17]).
ACKNOWLEDGMENTS
This work was supported in part by grant 90REGE0008, U.S. Admin-
istration for Community Living, National Institute on Disability,
Independent Living and Rehabilitation Research, Department of
Health Human Services. Opinions expressed do not necessarily
represent ofcial policy of the Federal government.
REFERENCES
[1] Steven Baker, Jenny Waycott, Romina Carrasco, Thuong Hoang, and Frank Vet-
ere. 2019. Exploring the Design of Social VR Experiences with Older Adults.
In Proceedings of the 2019 on Designing Interactive Systems Conference (DIS
’19), Association for Computing Machinery, New York, NY, USA, 303–315.
DOI:https://doi.org/10.1145/3322276.3322361
[2] A. Bandura. 1977. Self-efcacy: toward a unifying theory of behavioral change.
Psychol Rev 84, 2 (March 1977), 191–215. DOI:https://doi.org/10.1037//0033-295x.
84.2.191
[3] Grigore C. Burdea and Philippe Coifet. 2003. Virtual Reality Technology. John
Wiley & Sons.
[4] Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through
Qualitative Analysis. SAGE.
[5] Adeline Chu and Beth Mastel-Smith. 2010. The outcomes of anxiety, confdence,
and self-efcacy with Internet health information retrieval in older adults: a pilot
study. Comput Inform Nurs 28, 4 (August 2010), 222–228. DOI:https://doi.org/10.
1097/NCN.0b013e3181e1e271
[6] Jennifer Healey, Duotun Wang, Curtis Wigington, Tong Sun, and Huaishu
Peng. 2021. A Mixed-Reality System to Promote Child Engagement in Re-
mote Intergenerational Storytelling. In 2021 IEEE International Symposium on
Vibhav Nanda et al.
Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 274–279. DOI:https:
//doi.org/10.1109/ISMAR-Adjunct54149.2021.00063
[7] Eng Tat Khoo, Tim Merritt, and Adrian David Cheok. 2009. Designing physical
and social intergenerational family entertainment. Interacting with Computers 21,
1–2 (January 2009), 76–87. DOI:https://doi.org/10.1016/j.intcom.2008.10.009
[8] Luciana Lagana‘. 2008. Enhancing the Attitudes and Self-Efcacy of Older Adults
Toward Computers and the Internet: Results of a Pilot Study. Educ Gerontol 34, 9
(September 2008), 831. DOI:https://doi.org/10.1080/03601270802243713
[9] Luciana Laganà, Taylor Oliver, Andrew Ainsworth, and Marc Edwards. 2011.
Enhancing computer self-efcacy and attitudes in multi-ethnic older adults: a
randomised controlled study. Ageing Soc 31, 6 (August 2011), 911–933. DOI:https:
//doi.org/10.1017/S0144686X10001340
[10] Chhian Hūi Lêng and Jung-Der Wang. 2016. Daily home gardening improved
survival for older people with mobility limitations: an 11-year follow-up study in
Taiwan. Clin Interv Aging 11, (July 2016), 947–959. DOI:https://doi.org/10.2147/
CIA.S107197
[11] Cun Li, Jun Hu, Bart Hengeveld, and Caroline Hummels. 2019. Story-Me: Design
of a System to Support Intergenerational Storytelling and Preservation for Older
Adults. In Companion Publication of the 2019 on Designing Interactive Systems
Conference 2019 Companion (DIS ’19 Companion), Association for Computing
Machinery, New York, NY, USA, 245–250. DOI:https://doi.org/10.1145/3301019.
3323902
[12] Charles Xueyang Lin, Chaiwoo Lee, Dennis Lally, and Joseph F. Coughlin. 2018.
Impact of Virtual Reality (VR) Experience on Older Adults’ Well-Being. In Human
Aspects of IT for the Aged Population. Applications in Health, Assistance, and Enter-
tainment, Jia Zhou and Gavriel Salvendy (eds.). Springer International Publishing,
Cham, 89–100. DOI:https://doi.org/10.1007/978-3-319-92037-5_8
[13] Hayes Rafe, Glenda Revelle, Koichi Mori, Rafael Ballagas, Kyle Buza, Hiroshi
Horii, Joseph Kaye, Kristin Cook, Natalie Freed, Janet Go, and Mirjana Spasoje-
vic. 2011. Hello, is grandma there? let’s read! StoryVisit: family video chat and
connected e-books. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’11), Association for Computing Machinery, New York,
NY, USA, 1195–1204. DOI:https://doi.org/10.1145/1978942.1979121
[14] Logan Reis, Kathryn Mercer, and Jennifer Boger. Technologies for fostering
intergenerational connectivity and relationships: Scoping review and emergent
concepts | Elsevier Enhanced Reader. DOI:https://doi.org/10.1016/j.techsoc.2020.
101494
[15] Colleen D. Story. 2010. Observational Learning among Older Adults Living in
Nursing Homes. ProQuest LLC.
[16] Doreen Struve and Hartmut Wandke. 2009. Video Modeling for Training Older
Adults to Use New Technologies. ACM Trans. Access. Comput. 2, 1 (May 2009),
4:1-4:24. DOI:https://doi.org/10.1145/1525840.1525844
[17] Duotun Wang, Jennifer Healey, Jing Qian, Curtis Wigington, Tong Sun, and
Huaishu Peng. 2021. Lets Make A Story Measuring MR Child Engagement. Re-
trieved May 18, 2022 from http://arxiv.org/abs/2104.06536
[18] Z. Zhou, A.D. Cheok, S.P. Lee, L.N. Thang, C.K. Kok, W.Z. Ng, Y.K. Cher, M.L.
Pung, and Y. Li. 2005. Age Invader: human media for natural social-physical
inter-generational interaction with elderly and young. In Proceedings of the 2005
International Conference on Active Media Technology, 2005. (AMT 2005)., 203–204.
DOI:https://doi.org/10.1109/AMT.2005.1505308
[19] XR Access Symposium Report. Google Docs. Retrieved June 8, 2022 from
https://docs.google.com/document/d/131eLNGES3_2M5_roJacWlLhX-
nHZqghNhwUgBF5lJaE/edit?usp=sharing&usp=embed_facebook
[20] 2014-NGA-Garden-to-Table.pdf. Retrieved June 8, 2022 from https://garden.org/
special/pdf/2014-NGA-Garden-to-Table.pdf
[21] Gardening for older people - Better Health Channel. Retrieved May 19, 2022
from https://www.betterhealth.vic.gov.au/health/healthyliving/gardening-for-
older-people
 Flexible Activity Tracking for Older Adults Using Mobility Aids
— An Exploratory Study on Automatically Identifying Movement
Modality
Dimitri Vargemidis
dimitri.vargemidis@kuleuven.be
KU Leuven
Leuven, Belgium
Luc Geurts
luc.geurts@kuleuven.be
KU Leuven
Leuven, Belgium
ABSTRACT
Wearable activity trackers are inaccessible to older adults who use
mobility aids (e.g., walker, wheelchair), because the accuracy of
trackers drops considerably for such movement modalities (MMs).
As an initial step to address this problem, we implemented and
tested a minimum distance classifer to automatically identify the
used MM out of seven modalities, including movement with or
without a mobility aid, and no movement. Depending on the test
setup, our classifer achieves accuracies between 82 % and 100 %.
These fndings can be leveraged in future work to combine the
classifer with algorithms tailored to each mobility aid to make
activity trackers accessible to users with limited mobility.
CCS CONCEPTS
• Human-centered computing → Accessibility; • Computing
methodologies → Machine learning.
KEYWORDS
wearables, activity tracking, mobility aids, accessibility, machine
learning
ACM Reference Format:
Dimitri Vargemidis, Kathrin Gerling, Luc Geurts, and Vero Vanden Abeele.
2022. Flexible Activity Tracking for Older Adults Using Mobility Aids —
An Exploratory Study on Automatically Identifying Movement Modality.
In The 24th International ACM SIGACCESS Conference on Computers and
Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New
York, NY, USA, 5 pages. https://doi.org/10.1145/3517428.3550371
1 INTRODUCTION
Wearables for activity tracking are routinely leveraged to measure
physical activity (PA) in medical and research settings (e.g., [6,
7]), and can be used for self-refection on activity routines [12].
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550371
Kathrin Gerling
kathrin.gerling@kuleuven.be
KU Leuven
Leuven, Belgium
Vero Vanden Abeele
vero.vandenabeele@kuleuven.be
KU Leuven
Leuven, Belgium
Particularly in the context of older adults, activity tracking supports
physicians and therapists to follow up and better understand the
rehabilitation progress of their patients [3, 6], e.g., while recovering
from surgery or injuries after a fall, or coping with back pain and
other chronic conditions. In terms of personal exercise, activity
trackers help older adults to keep an automated record of their levels
of PA, which can make them more aware of their daily activity,
ftness, and physical wellbeing in general [16].
However, currently available wearable tracking systems do not
adapt well to this audience. We identifed two main problems. (1)
The accuracy of step counting for older adults is lower than for
younger adults. A lower walking speed and divergent gait are the
most important reasons for this problem [4, 14], and result, in most
cases, in an underestimation of the number of steps [14]. (2) Existing
activity trackers are not accessible to older adults who use mobility
aids, as these movement modalities are not supported, resulting
in even less accurate results [10]. Considering the likelihood of
older adults needing such mobility aids in late life due to sudden
(e.g., surgery, rehabilitation) or gradual changes (e.g., loss of muscle
strength, aching joints), staying physically active and being aware
of their own levels of PA is particularly important [5].
Several studies discuss and compare the accuracy of activity
trackers in diferent settings. Activity trackers are designed to op-
erate within a set parameter range. For example, walking slowly or
running fast negatively impacts the accuracy of the computed step
count [13]. For older adults, with a slower gait in general, consumer-
grade activity trackers can have accuracies ranging from 18 % to
78 %, depending on the tracker’s brand and the user’s walking speed
[14]. Younger adults’ step count accuracies are on average 90 %
[1]. Mobility aids further impede the accuracy of existing activity
trackers, rendering them unreliable for use in clinical contexts [8].
Because of these known accuracy issues, stand-alone algorithms
have been designed to address this, focusing on one MM at a time.
For example, with an improved algorithm tailored to rollator use,
accuracies of about 84 % can be achieved, compared to 10 % with
out-of-the-box wearable trackers [11]. Moreover, studies discuss
the implementation of an activity tracking system that has sepa-
rate algorithms for walking and running activities, using machine
ASSETS ’22, October 23–26, 2022, Athens, Greece Vargemidis, et al.

learning to decide which algorithm should be applied to each situa- Table 1: Overview of all data recording setups for each partic-
tion. For example, Bagui et al. report an accuracy of 93 % with this ipant as a combination of MM, sensor position, and location.
approach [2]. For each setting, we made two recordings
With a similar strategy, we aspire to build a fexible activity
tracking system for older adults to (1) recognise their MM, and (2) Movement modality Sensor position Location
apply an algorithm for that specifc modality to accurately com- Inside
Mobility aid
pute the achieved levels of PA. In this study, we focus on the frst Outside
Cane
objective, and consider six MMs: walking with a cane, crutches, Inside
Wrist
walker, rollator, using a wheelchair, or walking without a mobility Outside
aid (Figure 1). Our goal is to investigate the possibility of identi- Inside
Mobility aid
fying the user’s MM based on sensor data from a single inertial Outside
Crutches
measurement unit (IMU). Since this is part of an exploratory study, Inside
Wrist
we recorded a dataset with only three younger participants (two Outside
persons in the age range of 25-35, and one in 60-65). This allowed Inside
Mobility aid
us to avoid exposing older adults to health risks in light of the Outside
Walker
ongoing COVID-19 pandemic. Next, we implemented a minimum Inside
Wrist
distance classifer, and used the dataset to verify the accuracy of Outside
our classifer’s prediction. Inside
Mobility aid
Outside
Rollator
2 DATASET Inside
Wrist
Outside
We gathered 3-axes accelerometer and gyroscope data from three
participants engaging in the six MMs we considered in this study. In Inside
Mobility aid
addition to these MMs, we wanted to distinguish between a person Outside
Wheelchair
being in motion or not. Therefore, we also collected data while Inside
Wrist
the participants were sitting down calmly. For each modality, we Outside
recorded sensor data in a combination of diferent setups: having Inside
No mobility aid Wrist
the IMU attached to either the participant’s wrist or to the mobility Outside
aid being used, and recording the data either inside (smooth foor) Sitting Wrist Inside
or outside (uneven pavement). For each specifc setup, we made
Table 2: Three derived types of IMU data
two recordings (Table 1): the frst one to train our classifcation
model, the second one to verify the accuracy of its predictions.
Type of data Symbol
Each recording is about 60 seconds long, and data is sampled q
at a rate of approximately 20Hz. We used an Adafruit Flora micro- ®
Magnitude of angular velocity (ω) ωx2 + ωy2 + ωz2 = ω
controller and a BNO055 IMU to register all activity and send the
q
Magnitude of acceleration (®
a) a x2 + ay2 + az2 = a
sensor data in real time to a laptop, where everything was stored
in a CSV fle. We recorded movement data for each setup of Table Magnitude of acceleration (® a) along a® · д® = aд
1. The entire data collection process took about three to four hours ®
the gravitational force’s direction (д)
per participant. None of the participants used mobility aids in their
Table 3: Overview of all considered, mostly statistical, base
daily lives. However, for this exploratory study, having experience
features
with all aids is less important, as we investigate whether each aid
afords a sufciently distinct pattern of movement to automatically
recognise it using a machine learning algorithm. Base feature Abbreviation
Frequency of the signal f
Minimum value of the signal min
3 IMPLEMENTATION OF THE CLASSIFIER Maximum value of the signal max
We opted to use a minimum distance classifer because of its simplic- The diference between max and min range
ity and low use of resources, e.g., memory and computing power, Average value of the signal mean
and trained a model based on a feature set that highlights the most Standard deviation of all values of the signal sd
pronounced diferences between parameters of MMs. We derived Median value of the signal median
three types of data samples from the IMU, i.e., angular velocity mag- First quartile, 25th percentile of all values Q1
nitude, acceleration magnitude, and magnitude of the acceleration
Third quartile, 75th percentile of all values Q3
component along the gravitational force’s direction (Table 2). For
The diference between Q3 and Q1 IQR
each of these data types, we considered 11 possible base features
Total energy of the data signal e
(Table 3). Thus, to train our classifer’s model, we can compose a
feature set from a selection of 33 diferent features in total.
To train our minimum distance classifer, we select a feature
set, group all participants’ data per MM, and compute the mean
Flexible Activity Tracking for Older Adults Using Mobility Aids ASSETS ’22, October 23–26, 2022, Athens, Greece

Algorithm 1: Pseudo-code of the train function

for each movement modality do
for each selected feature do
compute mean feature value
end for
end for

Algorithm 2: Pseudo-code of the predict function

compute feature values of given data sample
for each movement modality do
compute Mahalanobis distance
(a) (b) end for
return movement modality with smallest distance

Table 4: Confusion matrix showing the classifer’s predic-

tions (only wrist-worn IMU) as a relative ratio (in %). The
mean accuracy is 88 %, and all results in this table are
obtained using the following features: ωQ 3 , ω I Q R , ωr anдe ,
ωmedian , ω f , aQ 1 , a I Q R , asd , amean , amedian , a f , aд,Q 3 , aд, I Q R ,
aд,mean , aд,median , aд,e

Actual

Wheelchair
Crutches

Rollator

Walker

Sitting
No aid
Cane
(c) (d)

Cane 67 0 8 8 0 0 0
Crutches 25 92 0 8 0 0 0
Predicted

Rollator 0 0 92 0 0 8 0
Walker 0 0 0 84 8 0 0
Wheelchair 0 8 0 0 92 0 0
No aid 8 0 0 0 0 92 0
Sitting 0 0 0 0 0 0 100

(e)
Figure 1: Mobility aids supported by the fexible track-
ing system: (a) cane, (b) crutches, (c) walker, (d) rollator,
(e) wheelchair. The sixth movement modality, i.e., walking
without a mobility aid, is not depicted here
value for each selected feature of all MMs (see Algorithms 1 and 2).
Once a model is trained, the algorithm predicts the user’s MM by
comparing the feature values of a given data sample to the mean
feature values of all MMs stored in the model. The smallest distance
between these values results in the classifer’s MM prediction. To
compensate for potential diferences in range and variance among
the selected features, we used the Mahalanobis distance instead of
the Euclidean distance [9].
4 RESULTS
Here, we consider four diferent subsets of our data to verify our
classifer. For each of these cases, we obtained a feature set through
forward selection, i.e., iteratively adding features that lead to the
biggest increase in prediction accuracy until it no longer increases.
Table 4 shows the resulting confusion matrix when we consider all
data recordings where participants wore the IMU on their wrist,
and train our classifer model with 16 selected features. The mean
prediction accuracy for this setup is 88 %. When the IMU is attached
to the mobility aid instead of the participant’s wrist, the prediction
accuracy is 82 % with four features selected to train the model
(Table 5). Note that no mobility aid is used for sitting and walking
without an aid. Therefore, only for those MMs, we included the data
recordings during which the IMU was attached to the participants’
wrist (see Table 1).
In a setting where an older adult would use no mobility aid to
walk short distances inside, and a rollator or a walker for longer dis-
tances outside, the classifer achieves a prediction accuracy of 96 %
(Table 6). Similarly, if a person only walks either with or without a
cane, the classifer always predicts the correct MM (Table 7).
ASSETS ’22, October 23–26, 2022, Athens, Greece Vargemidis, et al.

Table 5: Confusion matrix showing the classifer’s predic- distinguishing two diferent activities [2], we found that predictions
tions as a relative ratio (in %) (IMU attached to mobility aid, are still accurate when we include up to seven MMs, including MMs
except for No aid and Siting). The mean accuracy is 82 %, using fve mobility aids. Note that in a more realistic setting, one
and all results in this table are obtained using the following older adult is unlikely to use all considered types of mobility aids
features: ωmin , amin , aд,Q 3 , aд,r anдe on a regular basis. We can leverage this knowledge, e.g., by let-
ting users select which aids they regularly use through a one-time
Actual setup. This will further increase the classifer’s prediction accuracy,

Wheelchair
mounting to 96 % or higher.
Crutches

Rollator Additionally, the results suggest that the IMU’s location also

Walker

Sitting
No aid
impacts the prediction accuracy. Here, we found more favourable
Cane

results for a wrist-worn IMU. However, sensor placement is not

Cane 84 8 0 0 0 0 0 trivial [15, 17], and needs to be subject of follow-up studies.
Crutches 8 66 0 0 0 0 0
6 LIMITATIONS AND FUTURE WORK
Predicted

Rollator 0 0 50 0 0 0 0
Walker 8 8 0 92 0 0 0 This initial study allowed us to investigate the feasibility and poten-
Wheelchair 0 8 33 8 33 0 0 tial of a fexible tracking system. An important limitation is that we
No aid 0 8 0 0 0 100 0 recruited three participants only, and no older adults were involved
Sitting 0 0 17 0 17 0 100 for recording our dataset. We took this decision because the work
was carried out during the COVID-19 pandemic, and we did not
Table 6: Confusion matrix showing the classifer’s predic- want to put older adults at risk of exposure to collect preliminary
tions as a relative ratio (in %) for a subset of MMs (only wrist- data. Future studies need to verify whether the results found in
worn IMU). The mean accuracy is 96 %, and all results in this this study still apply to similar setups with older adults, specifcally
table are obtained using the following features: ωmean , a f , those who are familiar with using mobility aids in their daily lives,
aд,Q 3 , aд,sd and what adjustments would need to be made. Moreover, these re-
sults allow for continuing the fexible tracking approach, i.e., linking
Actual a MM prediction to a tailored algorithm for accurately computing
the user’s level of PA, e.g., step count or distance travelled.
Rollator

Walker

Sitting
No aid

7 CONCLUSION
Rollator 83 0 0 0 In this exploratory study, we examined fexible tracking as an ap-
proach to make activity tracking more accessible to older adults
Predicted

Walker 0 100 0 0 using mobility aids. As an initial step, we implemented and evalu-
No aid 17 0 100 0 ated a minimum distance classifer, achieving prediction accuracies
of more than 82 %. Future work should focus on evaluating this ap-
Sitting 0 0 0 100
proach with movement data of older adults, and linking predictions
to algorithms that can accurately track activity.
Table 7: Confusion matrix showing the classifer’s predic-
tions as a relative ratio (in %) for a subset of MMs where we REFERENCES
consider only sitting, and walking with or without a cane [1] Hyun-Sung An, Gregory C. Jones, Seoung-Ki Kang, Gregory J. Welk, and Jung-
(only wrist-worn IMU). The mean accuracy is 100 %, and all Min Lee. 2017. How valid are wearable physical activity trackers for mea-
results in this table are obtained using the following fea- suring steps? European Journal of Sport Science 17, 3 (March 2017), 360–368.
https://doi.org/10.1080/17461391.2016.1255261 Publisher: Routledge _eprint:
tures: aд,Q 3 , aд,sd https://doi.org/10.1080/17461391.2016.1255261.
[2] Sikha Bagui, Xingang Fang, Subhash Bagui, Jeremy Wyatt, Patrick Houghton,
Joe Nguyen, John Schneider, and Tyler Guthrie. 2022. An improved step count-
Actual ing algorithm using classifcation and double autocorrelation. International
Sitting
No aid

Journal of Computers and Applications 44, 3 (March 2022), 250–259. https:

Cane

//doi.org/10.1080/1206212X.2020.1726006 Publisher: Taylor & Francis _eprint:

https://doi.org/10.1080/1206212X.2020.1726006.
[3] Jens Barth, Jochen Klucken, Patrick Kugler, Thomas Kammerer, Ralph Steidl, Jür-
Cane 100 0 0 gen Winkler, Joachim Hornegger, and Björn Eskofer. 2011. Biometric and mobile
Predicted

No aid
Sitting
5 DISCUSSION
The results suggest that predicting diferent MM is possible, and
accuracies above 82 % can be achieved with straightforward ma-
chine learning algorithms. While Bagui and colleagues report on
0 100
0 0
gait analysis for early diagnosis and therapy monitoring in Parkinson’s disease.
0 In 2011 Annual International Conference of the IEEE Engineering in Medicine and
Biology Society. 868–871. https://doi.org/10.1109/IEMBS.2011.6090226
100 [4] Lynne Clay, Megan Webb, Claire Hargest, and Divya Bharatkumar Adhia. 2019.
Gait quality and velocity infuences activity tracker accuracy in individuals
post-stroke. Topics in Stroke Rehabilitation 26, 6 (Aug. 2019), 412–417. https:
//doi.org/10.1080/10749357.2019.1623474 Publisher: Taylor & Francis _eprint:
https://doi.org/10.1080/10749357.2019.1623474.
[5] Conor Cunningham, Roger O’ Sullivan, Paolo Caserotti, and Mark A. Tully.
2020. Consequences of physical inactivity in older adults: A systematic re-
view of reviews and meta-analyses. Scandinavian Journal of Medicine & Sci-
ence in Sports 30, 5 (2020), 816–827. https://doi.org/10.1111/sms.13616 _eprint:
Flexible Activity Tracking for Older Adults Using Mobility Aids
https://onlinelibrary.wiley.com/doi/pdf/10.1111/sms.13616.
[6] Andreas Ejupi, Matthew Brodie, Yves J. Gschwind, Daniel Schoene, Stephen Lord,
and Kim Delbaere. 2014. Choice stepping reaction time test using exergame tech-
nology for fall risk assessment in older people. In 2014 36th Annual International
Conference of the IEEE Engineering in Medicine and Biology Society. 6957–6960.
https://doi.org/10.1109/EMBC.2014.6945228
[7] Orestis Giotakos, Katerina Tsirgogianni, and Ioannis Tarnanas. 2007. A virtual
reality exposure therapy (VRET) scenario for the reduction of fear of falling and
balance rehabilitation training of elder adults with hip fracture history. In 2007
Virtual Rehabilitation. 155–158. https://doi.org/10.1109/ICVR.2007.4362157
[8] Paul Kooner, Taran Schubert, James L Howard, Brent A Lanting, Matthew G
Teeter, and Edward M Vasarhelyi. 2021. Evaluation of the Efect of Gait Aids,
Such as Canes, Crutches, and Walkers, on the Accuracy of Step Counters in
Healthy Individuals. Orthopedic Research and Reviews 13 (Jan. 2021), 1–8. https:
//doi.org/10.2147/ORR.S292255
[9] Konstantinos Koutroumbas and Sergios Theodoridis. 2008. Pattern Recognition.
Academic Press.
[10] Jonas Lauritzen, Adolfo Muñoz, Jose Luis Sevillano, and Anton Civit. 2013. The
usefulness of activity trackers in elderly with reduced mobility: a case study.
Studies in Health Technology and Informatics 192 (2013), 759–762.
[11] Denys J.C. Matthies, Marian Haescher, Suranga Nanayakkara, and Gerald Bieber.
2018. Step Detection for Rollator Users with Smartwatches. In Proceedings of the
Symposium on Spatial User Interaction (SUI ’18). Association for Computing Ma-
chinery, New York, NY, USA, 163–167. https://doi.org/10.1145/3267782.3267784
[12] Tara O’Brien, Meredith Troutman-Jordan, Donna Hathaway, Shannon Armstrong,
and Michael Moore. 2015. Acceptability of wristband activity trackers among
ASSETS ’22, October 23–26, 2022, Athens, Greece
community dwelling older adults. Geriatric Nursing 36, 2 (March 2015), S21–S25.
https://doi.org/10.1016/j.gerinurse.2015.02.019
[13] Tifany Sears, Elmer Alvalos, Samantha Lawson, Ian McAlister, L. Eschbach, and
Jennifer Bunn. 2017. Wrist-Worn Physical Activity Trackers Tend To Underesti-
mate Steps During Walking. International Journal of Exercise Science 10, 5 (Aug.
2017), 764–773. https://digitalcommons.wku.edu/ijes/vol10/iss5/12
[14] Salvatore Tedesco, Marco Sica, Andrea Ancillao, Suzanne Timmons, John Barton,
and Brendan O’Flynn. 2019. Accuracy of consumer-level and research-grade
activity trackers in ambulatory settings in older adults. PLOS ONE 14, 5 (May
2019), e0216891. https://doi.org/10.1371/journal.pone.0216891 Publisher: Public
Library of Science.
[15] Dimitri Vargemidis, Kathrin Gerling, Vero Vanden Abeele, Luc Geurts, and Katta
Spiel. 2021. Irrelevant Gadgets or a Source of Worry: Exploring Wearable Ac-
tivity Trackers with Older Adults. ACM Transactions on Accessible Computing
(TACCESS) 14, 3 (Aug. 2021), 28. https://doi.org/10.1145/3473463 Publisher: ACM
PUB27 New York, NY, USA.
[16] Rachel K. Walker, Amanda M. Hickey, and Patty S. Freedson. 2016. Advantages
and Limitations of Wearable Activity Trackers: Considerations for Patients and
Clinicians. Clinical Journal of Oncology Nursing 20, 6 (Dec. 2016), 606–610.
https://doi.org/10.1188/16.CJON.606-610
[17] Clint Zeagler. 2017. Where to wear it: functional, technical, and social considera-
tions in on-body location for wearable technology 20 years of designing for wear-
ability. In Proceedings of the 2017 ACM International Symposium on Wearable Com-
puters. ACM, Maui Hawaii, 150–157. https://doi.org/10.1145/3123021.3123042
 Improving Image Accessibility by Combining Haptic and
Auditory Feedback
Mallak Alkhathlan
Worcester Polytechnic Institute
Worcester, United States
malkhathlan@wpi.edu
Lane Harrison
Worcester Polytechnic Institute
Worcester, United States
ltharrison@wpi.edu
ABSTRACT
Advancements in accessibility have led to mobile applications that
help blind or low vision people (BLV) access surrounding infor-
mation independently. Unfortunately, accessibility of visual infor-
mation such as images remains limited. Previous research demon-
strated that spatial interaction can help BLV users build a mental
model of the relative locations of image objects. As haptics has
recently become a core component of modern smartphones, we
extend this prior research by designing three prototypes that use
haptic feedback to reveal object location in images. We evaluate
these techniques in terms of experience and ability of BLV users to
locate multiple objects in images. Evaluation results in a prelimi-
nary study with seven BLV users suggest that the proposed haptic
feedback prototype with auditory notifcations to identify people
and auditory caption can provide a more accessible and engaging
image experience.
CCS CONCEPTS
• Human-centered computing → Accessibility systems and
tools; Empirical studies in interaction design; haptic feedback.
KEYWORDS
Haptics, Touchscreens, Smartphones, Screen readers, Visual impair-
ment, Accessibility
ACM Reference Format:
Mallak Alkhathlan, ML Tlachac, Lane Harrison, and Elke Rundensteiner.
2022. Improving Image Accessibility by Combining Haptic and Auditory
Feedback. In The 24th International ACM SIGACCESS Conference on Com-
puters and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece.
ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3517428.3550362
1 INTRODUCTION AND RELATED WORK
Images are widely used in mobile applications. Given the prolifer-
ation of online images, it is increasingly critical for people with
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550362
ML Tlachac
Worcester Polytechnic Institute
Worcester, United States
mltlachac@wpi.edu
Elke Rundensteiner
Worcester Polytechnic Institute
Worcester, United States
rundenst@wpi.edu
blindness or low vision (BLV) to be able to understand digital im-
ages and their properties. Unfortunately, most image content is
largely inaccessible [1, 7, 24] to the more than 1.3 billion people
worldwide who have a vision disability [26]. Previous work has
started exploring strategies to capture more image details. For ex-
ample, in a study [18] that used image tags to capture the relative
spatial positions of objects within an image, 71.4% of BLV partic-
ipants expressed enjoyment at being able to discover the relative
locations of objects.
Recent technology advances in smartphones can ofer more en-
gaging experiences that simultaneously convey more relevant in-
formation to BLV users. Smartphones contain actuators which can
operate with low voltage and provide a high-speed touch sensa-
tions. The physical reaction created by a touchscreen interface is
called haptic feedback, or “haptics”. Force feedback and tactile hap-
tic feedback are two major components of haptic technology [9].
Force feedback relates to muscles and tendons, which can produce
physical characteristics such as force and mechanical compliance
[4, 16]. In contrast, tactile feedback can be described based on the
human senses and is in response to manual input [19, 27]. Tactile
feedback systems have been studied in many diferent contexts,
and have been used to inform the design of techniques within the
HCI community [2, 3, 5, 6, 8, 15–17, 20, 22, 23, 25, 28].
In this study, we thus aim to verify and extend previous studies
in the current context of touchscreen mobile phones. Our research
goal is to improve BLV understanding of images through tactile
haptic feedback, as such feedback has the potential to solve certain
accessibility issues. Our focus is to explore emerging haptic capabil-
ities to communicate to BLV users where objects appear spatially
within the image. We designed and tested various prototypes to
explore the feasibility of providing haptic feedback to allow BLV
users to form a mental model of the images, thereby gaining a
more in-depth understanding of the images. As haptics alone can
only convey spatial information, some of our prototypes paired
the haptic notifcations with auditory image descriptions to form a
more complete user experience, geared towards improving image
accessibility. Our study designs and evaluates three prototypes:
Prototype (F1) has only haptic feedback to indicate object and
no-object regions, prototype (F2) has haptic feedback to compare
object and no-object regions as well as auditory notifcations to
identify people, and prototype (F3) has haptic feedback to compare
object and no-object regions as well as auditory notifcations to
ASSETS ’22, October 23–26, 2022, Athens, Greece
identify people as well as auditory captions describing the image
scenario.
Our research is most similar to that of Morris et al. [18] in that
we also explore how BLV users spatially explore images. Since
the standard used for image descriptions since 1995 is outdated,
Morris et al. [18] developed a prototype for Android phones that
allows direct interactions with an image when touching a specifed
region. Yet even more engaging descriptions can be experienced
with current technologies, such as haptics.
Microsoft’s Seeing AI app1 attempts to create a more engaging
experience for BLV users. Seeing AI primarily uses auditory feed-
back to describe an image object the user is touching on the screen.
Recently, Seeing AI introduced a haptic notifcation when users
found the object. However, it remains unclear how users interact
with such haptics or whether haptics create a more engaging ex-
perience. How to address challenges involved in image navigation
are also unclear, such as situations when images contain multiple
objects.
Thus, our study explores possibilities beyond Morris et al. [18]
and Seeing AI. We extend the spatial interaction style introduced by
Morris et al. [18] by incorporating haptic feedback to assist image
navigation. We design prototypes that leverage haptic notifcations
to distinguish between object and no-object image regions. Our
prototypes extend the haptic ideas explored by Seeing AI which use
one type of haptic notifcation. The expressiveness of haptics and
increased capabilities in modern smartphones suggest that more
information might be communicated with appropriate design. We
also conduct a usability test with our prototypes to evaluate how
users interact with the haptic notifcations alone as well as when
the haptic notifcations are paired with complementary auditory
image description. Our research explores how combining haptics
with auditory image descriptions can promote spatial awareness to
make images more accessible for BLV users.
2 SYSTEM DESIGN
We prototype three novel interactions based on previous research
[18] that illustrate the relative location of the components in an
image. We implement our prototypes as an iOS application. The
app works across devices with diferent screen size. Our prototypes
consist of a feedback generator (UIFeedbackGenerator superclass
that available from UIKit and iOS 10 or later) that can play the pre-
defned haptic patterns [12]. The haptic notifcations [13] facilitate
image navigation by allowing users to distinguish between difer-
ent regions in the image. To trigger feedback, notifcations informs
users when tasks have been successfully completed or failed. Only
a single fnger interacting with the image is required to trigger the
object and no-object notifcations. Object and no-object regions are
activated by the use of the touchend event which helps to prevent
unintentional actions through the UINotifcationFeedbackGener-
ator subclass [13]. Specifcally, the prototypes deliver a medium
haptic notifcation to indicate that the user has successfully found
the object while the prototypes deliver a heavy haptic notifcation
to indicate that the user has failed to fnd the object [10]. In noti-
fcations of object, we use impact haptics to provides a medium
physical metaphor that indicates a collision between one fnger
1 https://www.microsoft.com/en-us/ai/seeing-ai
and a object region (object regions exist where there is a person in
the image). Comparatively, For notifcations of no-object, we use
impact haptics to provide a heavy physical metaphor that indicates
a collision between one fnger and a no-object region. These tac-
tile responses intend to reinforce understanding of spatial objects
within the image[11]. We select the two types of impacts to indicate
the efect of collision, however, we intend to experiment with more
feedback patterns in the future as more expressive technology is
developed.
While our prototypes can relate spatial knowledge regarding any
objects, we focused on images that contain people because a prior
study [21] found that questions about people were more frequent
than other image questions in accessibility contexts. Therefore, un-
like prototype F1, we incorporate facial recognition for prototypes
F2 and F3 by syncing the audio indicators with the haptics. In proto-
type F3, a screen reader also reads the image caption aloud. For the
captions, we add descriptions in the button’s content description
feld to label an image-based button to interact properly with the
screen reader.
For each of the three prototype, we presented users with four
diferent scenarios, a, b, c, and d, which respectively contain 2, 3,
4, and 5 objects. These scenarios test the ability of participants to
locate multiple objects within images. No participant experienced
the same image more than once. Briefy, the prototypes are as
follows:
(1) Prototype (F1) has only haptic feedback to compare object
and no-object regions (Appendix Fig. 1).
(2) Prototype (F2) has haptic feedback to compare object and
no-object regions as well as auditory notifcations to identify
people (Appendix Fig. 2).
(3) Prototype (F3) has haptic feedback to compare object and no-
object regions, auditory notifcations to identify people, and
auditory captions to describe the image scenario (Appendix
Fig. 3).
3 PRELIMINARY EVALUATION
As a proof-of-concept, we conduct an evaluation study of the proto-
types (IRB-approved). Using Twitter advertisements, we recruited
7 legally blind adults with diferent levels of haptic experience (age
range 21-51 years old; mean age = 32 years old; 4 reported male; 3
female). Participants used their personal smartphones. Each par-
ticipant completed a 15-minute tutorial, allowing them to practice
using the prototypes on sample images and ask questions to guar-
antee understanding. For these evaluations, prototype order and
scenario order randomized across participants using a latin square
methodology to mitigate order efects. All sessions were recorded.
Evaluation measures include four components: memory recall,
time measurement, sensitivity level, and preference. 1) Memory
recall: A previous study [14] indicates that one of the difculties
that BLV encountered when using touch screens is learning where
objects are located on the screen and remembering the location
of on-screen objects. We address this problem by testing memory
recall. For each prototype, we asked “Do you remember the sequence
of the scenarios in same order given?” participants would then state
the sequence they remember, for example, “There were 4, 3, 2, and 5
objects.” to answer our questions. When comparing the accuracy of
Improving Image Accessibility by Combining Haptic and Auditory Feedback
participant recall, the prototypes showed statistically signifcant
diferences (� 2 (2, N = 7) = 16.88, P = .0003) while scenarios did not
difer. Thus, the quantity of feedback channels (haptics and audio)
appears to impact recall, while the number of objects (between two
and fve) in the image does not appear to impact recall. Overall, the
integration of the three channels may prove in understanding the
image.
2) Time measurement: For each image, we recorded the time
taken by participants to fnd each object in seconds. The time it
took to fnd each individual object is the dependent variable for
this experiment. The predictors of the model were categorical vari-
ables for scenario, prototype, and object number. In our repeated
measures design, the result of the outcome variable is continuous
and each participant having multiple observations, thus a linear
mixed model was used to analyze the data. No statistically signif-
cant results were found for the model that contained an interaction
between prototype and scenario. However, there was a diference in
completion time between object 2 and object 3 (using the data from
scenarios b, c and d) for prototype F1 compared to the diference
between object 2 and object 3 (using the data from scenarios b, c
and d) for prototype F2. The model that contained an interaction
between prototype and object showed no signifcant diference
(� 2 (8, N = 7) = 14.48, P = .07). Thus, the completion time was simi-
lar for two, three, and four objects. However, the completion time
appears increased when a ffth object was added to the image. We
did not explore the completion time for images with more than
fve objects, though future studies might explore whether there are
ceiling efects for the number of efective objects.
3) Sensitivity level: At the end of the session, participants were
asked to rate the appropriateness of the strength of the haptic
feedback experienced when exploring the images. The level of
haptic notifcation was efectively used to measure comfort. Our
prototypes contained two diferent haptic notifcations; a heavy
strength notifcation was used to indicate failure at fnding an
object, and a medium strength notifcation was used to indicate
success at fnding an object. Notably, all but one participant reported
that the haptic feedback notifcation level was appropriate. The
majority of participants thus approved of the strength of both types
of notifcations.
4) Preference: After having reviewed all three prototypes, partici-
pants were asked to rate their preferences regarding each prototype.
Six participants reported lower preference scores for prototype F1
when compared to prototype F2. One participant had no change
in preference between these two prototypes. A Wilcoxon signed-
rank test indicated that this diference was statistically signifcant,
z = -2.311, P = .02 and exact test probability of P = .03. All seven
participants reported lower preference scores for prototype F1 as
compared to prototype F3. The Wilcoxon signed-rank test indicated
that this diference was statistically signifcant, z = -2.53, P = .01
and exact test probability of P = .02. Similarly, all seven participants
reported lower preference scores for prototype F2 as compared to
prototype F3. The Wilcoxon signed-rank test indicated that this
diference was statistically signifcant, z = -2.41, P = .02. From these
results, we deduce that participants preferred the prototype F3 that
conveyed information using haptic feedback with both auditory
notifcations to identify people and auditory captions describing
the scene.
ASSETS ’22, October 23–26, 2022, Athens, Greece
4 CONCLUSION
In this study, we explore how BLV users can beneft from haptic
feedback when navigating images. We developed three prototypes
that incorporate varying degrees of details to enhance digital im-
age understanding. With diferent haptic notifcations for object
and no-object regions, our prototypes are designed so that BLV
users can navigate the spatial location of objects within images.
To evaluate our three prototypes, we conduct a usability test with
seven BLV participants. The participants preferred the haptic feed-
back prototype with auditory notifcations to identify people and
auditory caption. Overall, the results revealed that tactile haptic
feedback has the potential to improve image understanding and
user experience of BLV smartphone users. Our fndings can enable
a better understanding of tactile feedback rendering strategies and
support the adoption of haptic feedback as an accessibility feature
in smartphones.
5 ACKNOWLEDGMENT:
We thank Imam Abdulrahman Bin Faisal University (IAU) and Saudi
Arabian Cultural Mission to the USA (SACM) for fnancial support.
REFERENCES
[1] Mallak Alkhathlan, ML Tlachac, Lane Harrison, and Elke Rundensteiner. 2021.
“Honestly I Never Really Thought About Adding a Description”: Why Highly En-
gaged Tweets Are Inaccessible. In IFIP Conference on Human-Computer Interaction.
Springer, 373–395.
[2] Christopher S Campbell, Shumin Zhai, Kim W May, and Paul P Maglio. 1999.
What you feel must be what you see: adding tactile feedback to the trackpoint.
In In: Proc. of INTERACT’99: 7th IFIP Conference on Human Computer Interaction.
Citeseer.
[3] Géry Casiez, Nicolas Roussel, Romuald Vanbelleghem, and Frédéric Giraud. 2011.
Surfpad: riding towards targets on a squeeze flm efect. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems. 2491–2500.
[4] Heather Culbertson, Samuel B Schorr, and Allison M Okamura. 2018. Haptics:
The present and future of artifcial touch sensation. Annual Review of Control,
Robotics, and Autonomous Systems 1 (2018), 385–409.
[5] Jack Tigh Dennerlein, David B Martin, and Christopher Hasser. 2000. Force-
feedback improves performance for steering and combined steering-targeting
tasks. In Proceedings of the SIGCHI conference on Human factors in computing
systems. 423–429.
[6] Nicholas A Giudice, Hari Prasath Palani, Eric Brenner, and Kevin M Kramer. 2012.
Learning non-visual graphical information using a touch-based vibro-audio
interface. In Proceedings of the 14th international ACM SIGACCESS conference on
Computers and accessibility. 103–110.
[7] Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington,
Kris M Kitani, and Jefrey P Bigham. 2020. Twitter A11y: A browser extension
to make Twitter images accessible. In Proceedings of the 2020 CHI Conference on
Human Factors in Computing Systems. 1–12.
[8] Cagatay Goncu and Kim Marriott. 2011. GraVVITAS: generic multi-touch presen-
tation of accessible graphics. In IFIP Conference on Human-Computer Interaction.
Springer, 30–48.
[9] Mitchell L. Gordon and Shumin Zhai. 2019. Touchscreen Haptic Augmentation
Efects on Tapping, Drag and Drop, and Path Following. In Proceedings of the
2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland
Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12.
https://doi.org/10.1145/3290605.3300603
[10] 2022 Apple Inc. 2022. Human Interface Guidelines. Retrieved Jun 20, 2022
from https://developer.apple.com/design/human-interface-guidelines/patterns/
playing-haptics
[11] 2022 Apple Inc. 2022. Human Interface Guidelines. Retrieved Jun 20, 2022
from https://developer.apple.com/design/human-interface-guidelines/patterns/
feedback/
[12] 2022 Apple Inc. 2022. uikit. Retrieved Jun 20, 2022 from https://developer.apple.
com/documentation/uikit/uifeedbackgenerator
[13] 2022 Apple Inc. 2022. uikit. Retrieved Jun 20, 2022 from https://developer.apple.
com/documentation/uikit/uinotifcationfeedbackgenerator
[14] Shaun K Kane, Jefrey P Bigham, and Jacob O Wobbrock. 2008. Slide rule: making
mobile touch screens accessible to blind people using multi-touch interaction
ASSETS ’22, October 23–26, 2022, Athens, Greece

techniques. In Proceedings of the 10th international ACM SIGACCESS conference

on Computers and accessibility. 73–80.
[15] Ernst Kruijf, Saugata Biswas, Christina Trepkowski, Jens Maiero, George Ghinea,
and Wolfgang Stuerzlinger. 2019. Multilayer haptic feedback for pen-based
tablet interaction. In Proceedings of the 2019 CHI Conference on Human Factors in
Computing Systems. 1–14.
[16] Karon E MacLean. 2008. Haptic interaction design for everyday interfaces.
Reviews of Human Factors and Ergonomics 4, 1 (2008), 149–194.
[17] Giuseppe Melf, Karin Müller, Thorsten Schwarz, Gerhard Jaworek, and Rainer
Stiefelhagen. 2020. Understanding what you feel: A mobile audio-tactile system
for graphics used at schools with students with visual impairment. In Proceedings
of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.
[18] Meredith Ringel Morris, Jazette Johnson, Cynthia L Bennett, and Edward Cutrell.
2018. Rich representations of visual content for screen reader users. In Proceedings
of the 2018 CHI conference on human factors in computing systems. 1–11.
[19] Alvaro Pascual-Leone and Roy Hamilton. 2001. The metamodal organization of
the brain. Progress in brain research 134 (2001), 427–445.
[20] Benjamin Poppinga, Charlotte Magnusson, Martin Pielot, and Kirsten Rassmus-
Gröhn. 2011. TouchOver map: audio-tactile exploration of interactive maps. In
Proceedings of the 13th International Conference on Human Computer Interaction
with Mobile Devices and Services. 545–550.
[21] Elliot Salisbury, Ece Kamar, and Meredith Morris. 2017. Toward scalable social
alt text: Conversational crowdsourcing as a tool for refning vision-to-language
technology for the blind. In Proceedings of the AAAI Conference on Human Com-
putation and Crowdsourcing, Vol. 5.
[22] Brandon T Shrewsbury. 2011. Providing haptic feedback using the kinect. In The
proceedings of the 13th international ACM SIGACCESS conference on Computers
and accessibility. 321–322.
[23] Francesca Sorgini, Renato Caliò, Maria Chiara Carrozza, and Calogero Maria
Oddo. 2018. Haptic-assistive technologies for audition and vision sensory dis-
abilities. Disability and Rehabilitation: Assistive Technology 13, 4 (2018), 394–421.
[24] Abigale Stangl, Meredith Ringel Morris, and Danna Gurari. 2020. " Person, Shoes,
Tree. Is the Person Naked?" What People with Vision Impairments Want in
Image Descriptions. In Proceedings of the 2020 CHI Conference on Human Factors
in Computing Systems. 1–13.
[25] Eric Vezzoli, Thomas Sednaoui, Michel Amberg, Frédéric Giraud, and Betty
Lemaire-Semail. 2016. Texture rendering strategies with a high fdelity-capacitive
visual-haptic friction control device. In International Conference on Human Haptic
Sensing and Touch Enabled Computer Applications. Springer, 251–260.
[26] World Health Organization (WHO). 2020. Blindness and vision impairment.
Retrieved October 8, 2019 from https://www.who.int/news-room/fact-sheets/
detail/blindness-and-visual-impairment
[27] Thomas Wolbers, Roberta L Klatzky, Jack M Loomis, Magdalena G Wutte, and
Nicholas A Giudice. 2011. Modality-independent coding of spatial layout in the
human brain. Current Biology 21, 11 (2011), 984–989.
[28] Yang Zhang and Chris Harrison. 2015. Quantifying the targeting performance
beneft of electrostatic haptic feedback on touchscreens. In Proceedings of the
2015 International Conference on Interactive Tabletops & Surfaces. 43–46.
Improving Image Accessibility by Combining Haptic and Auditory Feedback ASSETS ’22, October 23–26, 2022, Athens, Greece

APPENDICES
PROTOTYPE SPECIFICATIONS
Table 1: Haptic feedback legend for prototypes F1, F2, and F3. No-object regions do not contain people, and when touched with
one fnger, it triggers a no-object notifcation. Object regions contain people, and when touched with one fnger, it triggers an
object notifcation. We use heavy and medium impact haptics for the no-object and object notifcations, respectively.

No-object region Notifcation of no-object Object region Notifcation of object

(a) Image with 2 objects. (b) Image with 3 objects. (c) Image with 4 objects. (d) Image with 5 objects.

Figure 1: Prototype (F1): has only haptic feedback; prototype. (a) shows scenario a: image with two objects, (b) shows scenario b:
image with three objects, (c) shows scenario c: image with four objects, and (d) shows scenario d: image with fve objects.

(a) Image with 2 objects. (b) Image with 3 objects. (c) Image with 4 objects. (d) Image with 5 objects.

Figure 2: Prototype (F2): has haptic feedback with auditory notifcations to identify people. We use audio feedback for each
object to indicate people. Note, the audio feedback is depicted as text and the images do not contain text. (a) shows scenario
a: image with two objects; frst object audio is “Here is Denzel Washington”, and second object audio is “Here is Viola Davis”.
(b) shows scenario b: image with three objects; frst object audio is “Here is Edward Carstens”, second object audio is “Here is
Virginia Madsen”, and third object audio: “Here is Elaine Madsen”. (c) shows scenario c: image with four objects; frst object
audio is “Here is Laura Harris”, second object audio is “Here is Aubrey dollar”, third object audio is “Here is Angie Harmon”,
and fourth object audio is “Here is Paula Newsome”. (d) shows scenario d: image of fve objects; frst object audio is “Here is
Benjamin”, second object audio is “Here is Olivi”, third object audio is “Here is Isabella”, f ourth object audio is “Here is Emma”,
and ffth object audio is “Here is Sophia”.
ASSETS ’22, October 23–26, 2022, Athens, Greece

(a) Image with 2 objects. (b) Image with 3 objects. (c) Image with 4 objects. (d) Image with 5 objects.

Figure 3: Prototype (F3): has haptic feedback with auditory notifcations to identify people and auditory caption. We use audio
feedback for each object to indicate people. Note, the audio feedback is depicted as text and the images do not contain text. (a)
shows scenario a: image with two objects; frst object audio is ”Here is Denzel Washington”, and second object audio is ”Here is
Viola Davis” ; the caption is ”Actors atend the opening night”. (b) shows scenario b: image with three objects; frst object audio is
”Here is Edward Carstens”, second object audio is ”Here is Virginia Madsen”, and third object audio: ”Here is Elaine Madsen” ;
caption is ”Actor, dramatist and person arrive at the premiere ”. (c) shows scenario c: image with four objects; frst object audio is
”Here is Laura Harris”, second object audio is ”Here is Aubrey dollar”, third object audio is ”Here is Angie Harmon”, and fourth
object audio is ”Here is Paula Newsome” ; caption is ”Actors atend the upfront presentation”. (d) shows scenario d: image of fve
objects; frst object audio is ”Here is Benjamin”, second object audio is ”Here is Olivi”, third object audio is ”Here is Isabella”,
f ourth object audio is ”Here is Emma”, and ffth object is”Here is Sophia” ; caption is ”Happy children and adults with a shopping
cart inside retail”.
 Inter-rater Reliability of Command-Line Web Accessibility
Evaluation Tools
Eryn Rachael Kelsey-Adkins
ekelseyadkins@mines.edu
Colorado School of Mines
Golden, Colorado, USA
ABSTRACT
This study compares four command-line interface (CLI) web acces-
sibility tools, examining if one CLI tool is sufcient for automated
accessibility evaluation. The four tools were: Axe-core/cli, IBM
Equal Access NPM Accessibility Checker (Accessibility Checker),
Pa11y-ci, and the A11y Machine. Inter-rater reliability was calcu-
lated using Gwet’s alpha coefcient 2 (AC 2 ), and the results indicate
very poor reliability between tools.
CCS CONCEPTS
• Human-centered computing → Accessibility design and eval-
uation methods.
KEYWORDS
accessibility, command-line interface, inter-rater reliability
ACM Reference Format:
Eryn Rachael Kelsey-Adkins and Robert Thompson. 2022. Inter-rater Reli-
ability of Command-Line Web Accessibility Evaluation Tools. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
4 pages. https://doi.org/10.1145/3517428.3550395
1 INTRODUCTION AND BACKGROUND
Tim Berners-Lee, inventor of the World Wide Web, said, "it is critical
that the Web be usable by anyone, regardless of individual capa-
bilities and disabilities" [1] at the launch of the Web Accessibility
Initiative (WAI) in 1997. WAI promotes Web accessibility through
the creation of accessibility standards such as the Web Content
Accessibility Guidelines (WCAG). WCAG 2.0, published in 2008,
outlines accessibility success criteria across four principles: that
Web content must be Perceivable, Observable, Understandable, and
Robust (POUR) [11]. These guidelines have been the basis of acces-
sibility and non-discrimination legislation in nineteen countries
and the European Union [12].
Web accessibility can be evaluated through manual testing or
with the aid of automated tools. Automated testing cannot replace
manual testing, but it is an important part of accessibility evaluation
of websites, both after deployment and during development. The
need for accelerated development and delivery of website features
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550395
Robert Thompson
rthompson@mines.edu
Colorado School of Mines
Golden, Colorado, USA
has led many organizations to incorporate Continuous Integra-
tion/Continuous Deployment (CI/CD) practices that can eliminate
or reduce the need for many manual tasks [10]. Incorporating auto-
mated Web accessibility evaluation in CI/CD practices can alert de-
velopers to egregious accessibility failures. Command-line interface
(CLI) automated Web accessibility evaluation tools are particularly
suited to integration with CI/CD. WAI maintains a list of nineteen
CLI evaluation tools, but makes no claims to the reliability of any
of them [13].
A common reliability measurement is Inter-Rater Reliability
(IRR), which quantifes the rate of agreement or disagreement be-
tween two or more raters [4]. Statistics to calculate IRR include
Cohen’s kappa, Fleiss’s kappa, Krippendorf’s alpha, and Gwet’s al-
pha coefcient 2 (AC 2 ). Both Krippendorf’s alpha and Gwet’s AC 2
can be calculated with missing data and can use diferent weight
systems for diferent types of ratings. Gwet’s AC 2 was developed to
address paradoxes in kappa-like statistics: the prevalence problem,
which can cause kappa estimates to be erroneously low, and the
bias problem, which can cause kappa estimates to be erroneously
high [5]. This makes Gwet’s AC 2 well suited to evaluate IRR of
Web accessibility evaluation tools where not every tool rates every
HTML element. Gwet’s AC 2 values range from -1 to 1. A result of
-1 shows near perfect disagreement, whereas a result of 1 signifes
unanimous agreement. A result of 0 is equivalent to each rater
randomly assigning ratings.
Previous studies have compared automated evaluation tools [2,
3, 6, 7, 9], but very few have evaluated IRR between tools [2, 7] and
no studies have focused on IRR in CLI evaluation tools. This study
uses Gwet’s AC 2 to evaluate IRR of four automated evaluation tools:
Axe-core/cli, IBM’s Accessibility Checker, Pa11y-ci, and the A11y
Machine.
2 METHODOLOGY
2.1 Inclusion and Exclusion Criteria for
Evaluation Tools
The frst step of the analysis was to identify the inclusion and exclu-
sion criteria for the evaluation tools. For the desired use case, the
tools should have a command line interface, be open source, and be
able to evaluate groups of websites for compliance with W3C Web
Content Accessibility Guidelines 2.0 and Section 508, the US federal
procurement standard. Any tool that is a more specifc version
of another tool should be excluded in favor of the more general
version. The World Wide Web Consortium (W3C) maintains a list
of Web Accessibility Evaluation Tools that can be fltered by type
of tool, guidelines, license, and language, among others [13]. Using
the inclusion criteria, fve flters were identifed: Command Line
Interface, Open Source, WCAG 2.0 — W3C Web Content Accessibility
ASSETS ’22, October 23–26, 2022, Athens, Greece Kelsey-Adkins and Thompson

Table 1: Node.js packages of each evaluation tool

Package Name Developer Version Release Date Last Update

@axe-core/cli Deque Systems, Inc. 4.4.2 01/10/2015 06/01/2022
accessibility-checker IBM Accessibility 3.1.30 05/18/2020 05/10/2022
pa11y-ci Nature Publishing Group 3.0.1 04/08/2015 11/26/2021
the-a11y-machine Liip 0.9.3 01/29/2016 02/08/2017

Table 2: Rating classifcation by evaluation tool

Rating A11y machine Accessibility Checker Axe-core/cli Pa11y-ci

pass N/A N/A pass N/A
warning notice, warning potential violation incomplete notice, warning
violation error violation violation error

Guidelines 2.0, Section 508, US federal procurement standard, and Gwet’s AC 2 was calculated following the general form:
Automatically checks groups of web pages or web sites. pa − pe
Applying these flters resulted in fve evaluation tools. After α= (1)
1 − pe
applying the exclusion criterion, four evaluation tools met all cri- where pa is the weighted percent agreement between evaluation
teria: Axe-core/cli, IBM Equal Access NPM Accessibility Checker tools and pe the weighted percent chance agreement. As there is a
(Accessibility Checker), Pa11y-ci, and the A11y Machine. The most ranked order to the ratings, such that pass < warning < violation,
current version of each evaluation tool was installed with Node an ordinal weight function was calculated:
Package Manager (NPM) version 8.1.4. See Table 1 for versions and
the most recent update of each evaluation tool. All tools were run (
with Google Chrome version 102.0.5005.61 in headless mode. 1, if k = l
w kl = #{(i, j),min(k,l )≤i <j ≤max (k,l )} (2)
1− wmax , if k , l
where #{(i, j), min(k, l) ≤ i < j ≤ max(k, l)} is the number of
2.2 Data Collection pairs (i, j), i < j, that can be formed between min(k, l) and max(k, l),
In order to compare the efectiveness of each evaluation tool, the and wmax the maximum value over all k and l [4].
frst 5 higher education institutions of the United States listed in
the ranking Webometrics [14] were used as the sample: Harvard 3 RESULTS
University, Stanford University, Massachusetts Institute of Technol- Using the methods described in section 2, the accessibility ratings
ogy, University of California Berkeley, and University of Michigan. for 81 HTML elements across fve websites by the four evaluation
The homepage of each institution was evaluated, as it is the most tools were analyzed. Axe-core/cli rated the highest number of ele-
representative page and is the web page frequently analyzed in web ments at 78, while the A11y Machine rated the least (see Figure 1).
accessibility studies [8]. Reports from each evaluation tool were
saved as JSON fles.
Each HTML element evaluated by one or more tools was entered
into a table with all of its associated ratings from any of the evalua-
tion tools. As each evaluation tool has its own rating system, these
ratings were divided into pass, warning, and violation (see Table 2).
Note that although both Pa11y-ci and Axe-core/cli use a pass rating,
only Axe-core/cli explicitly rates specifc HTML elements as "pass".
A system of four ratings, not evaluated, pass, warning, and viola-
tion, was considered but rejected, as it is unclear from the reports
if any specifc element has been evaluated by that specifc tool. As
Gwet’s AC 2 is based on the computation of pairwise values, any
element that had been rated by only one tool was discarded. See
Table 3 for a truncated copy of the rating table.
The rating table was converted into an agreement table that
records how often each element received each rating. The cells in Figure 1: HTML Element Rating by Tool
this table, r i j , are the number of times the evaluation tools assigned
element i rating category j. See Table 4 for a truncated copy of the Calculating Gwet’s AC 2 gave an observed weighted percent
agreement table. agreement, pa , of 0.87 and a chance weighted percent agreement,
Inter-rater Reliability of CLI Accessibility Evaluation Tools
Table 3: Example of the ratings table
HTML Element A11y machine
<a class="home-hero-link" ... </a> warning
<a href="https://coronavirus ... </a> violation
<button type="submit" ...</button> violation
<form id="navbar-search" ... </form> violation
<p>2022/23 season</p> violation
Table 4: Example of the agreement table
Element pass warning violation
1 0 3 1
2 0 0 4
3 0 0 4
4 1 1 1
pe , of 0.88. The weighted inter-rater reliability of all four evaluation
tools was -0.05, indicating near randomness. Inter-rater reliability
was then recalculated excluding the A11y Machine ratings; the
A11y Machine met all inclusion and exclusion criteria, but has been
archived with no updates since 2019. Calculating Gwet’s AC 2 on
the remaining three evaluation tools resulted in a score of -0.17
(pa = 0.82, pe = 0.85).
4 DISCUSSION AND FUTURE WORK
Gwet’s AC 2 shows there is very little inter-rater reliability between
these evaluation tools when analyzing ratings for individual HTML
elements. The diference between two tools may be minor if one
tool marks an element as a violation and another tool marks that
element as a potential violation, if the developers that rely on these
tools in a CI/CD environment follow up on both ratings. However,
a false pass rating could allow an accessibility standard violation to
make it into production. This reinforces the idea that incorporating
more than one evaluation tool or method in accessibility testing is
necessary.
Usability can be highly subjective and dependent on individual
use cases, so the four evaluation tools were only informally assessed
for their ease of use during this study. The criteria were ease of use,
confguration options, and the level of detail included in reports.
Of the four, the A11y Machine was the easiest to setup and
run. This tool can crawl all Web pages within a website (up to 128
pages), which is useful for a comprehensive accessibility overview
of a website when a site map is not available. The A11y Machine
outputs the results to an easily read local Web interface that can
be fltered by violation level; a summary of the evaluation is saved
to a JSON fle for integration with CI/CD environments. However,
the A11y Machine has the least options for confguration according
to specifc use case needs and is no longer maintained. Any issues
with the package will likely not be resolved in the future.
The other three evaluation tools are highly confgurable and eas-
ily connected to CI/CD environments. These confguration options
include enabling or disabling specifc rules or rule sets, specifying
ASSETS ’22, October 23–26, 2022, Athens, Greece
Accessibility Checker Axe-core/cli Pa11y-ci
violation warning warning
violation violation violation
pass warning
violation violation violation
violation violation
report formats, and enabling page load delays and timeout lim-
its. IBM’s Accessibility Checker not only notes the specifc HTML
element fagged due to a violation or potential violation, but also re-
ports that element’s specifc location in the DOM tree. Axe-core/cli
reports include a list of standards each HTML element passed,
which can reduce manual evaluation times, and links to more in-
formation for each standard. Both Accessibility Checker and Axe-
core/cli seem to be better at fagging appropriate use of aria tags
such as focusable elements tagged aria-hidden=true, but also had a
higher number of false positive contrast errors. Pa11y-ci runs on
the HTML Code Snifer accessibility engine by default, but can be
confgured to run on the axe-core engine as well, which increases
its fexibility.
This study compared the ratings given to individual HTML el-
ements. However, analyzing ratings for WCAG success criteria
rather than individual elements may provide useful insight into
the IRR of these tools. Further, these tools were not evaluated for
intra-rater reliability or accuracy. An accurate Web accessibility
evaluation tool must have few false positives and negatives, and
must produce consistent results. Future work should analyze the
correctness, completeness and specifcity of the results of each of
these CLI evaluation tools. As Pa11y-ci can be confgured to run
on either the HTML Code Snifer or the axe-core accessibility en-
gines, future work could analyze the inter-rater reliability between
Pa11y-ci confgured for the axe-core rule set and Axe-core/cli.
5 CONCLUSION
Four command-line interface Web accessibility evaluation tools
were analyzed for inter-rater reliability with Gwet’s AC 2 . Each eval-
uation tool was used to assess the accessibility of fve university
website home pages. Each individual HTML element and the as-
sociated number of pass, warning, or violation ratings it received
from these evaluations were collated into an agreement table. As
pass < warning < violation, a ordinal weight function was used
to calculate Gwet’s AC 2 . An initial result of -0.05 shows very poor
inter-rater reliability, near equivalent to random guessing. There
were notable diferences both in which HTML elements each eval-
uation tool fagged, as well as in the ratings assigned to the same
elements be diferent tools. Each evaluation tool prioritizes and tests
for each accessibility standard through diferent criteria and no one
of these tools should be assumed to provide complete coverage of
the WCAG standards that can be tested through automation.
REFERENCES
[1] Tim Berners-Lee. 1997. Press Release: W3C Launches Web Accessibility Initiative.
W3C. Retrieved June 5, 2022 from https://www.w3.org/Press/WAI-Launch.html
ASSETS ’22, October 23–26, 2022, Athens, Greece
[2] Maria Björkman. 2022. Inter-tool Reliability of Three Automated Web Accessibil-
ity Evaluators. USCCS 15 (2022), 15–25.
[3] Giorgio Brajnik. 2004. Comparing accessibility evaluation tools: a method for
tool efectiveness. Universal Access in the Information Society 3, 3 (2004), 252–263.
https://doi.org/10.1007/s10209-004-0105-y
[4] Kilem L. Gwet. 2015. On the Krippendorf’s alpha coefcient. (2015).
https://www.researchgate.net/profle/Kilem-Gwet/publication/267823285_
On_Krippendorf’s_Alpha_Coefcient/links/60e3bf0892851ca944ae25d6/On-
Krippendorfs-Alpha-Coefcient.pdf
[5] Kevin A. Hallgren. 2012. Computing Inter-Rater Reliability for Observational
Data: An Overview and Tutorial. Tutorials in Quantitative Methods for Psychology
8, 1 (2012), 23–34. https://doi.org/10.20982/tqmp.08.1.p023
[6] Rita Ismailova and Yavuz Inal. 2022. Comparison of Online Accessibility Evalua-
tion Tools: An Analysis of Tool Efectiveness. IEEE Access 10 (2022), 58233–58239.
https://doi.org/10.1109/ACCESS.2022.3179375
[7] Ashli M. Molinero, Frederick G Kohun, and R. Morris. 2006. Reliability in au-
tomated evaluation tools for web accessibility standards compliance. Issues in
Information Systems 7, 2 (2006), 218–222.
[8] Carlos Máñez-Carvajal, Jose Francisco Cervera-Mérida, and Rocío Fernández-
Piqueras. 2021. Web accessibility evaluation of top-ranking university Web sites
in Spain, Chile and Mexico. Universal Access in the Information Society 20, 1
Kelsey-Adkins and Thompson
(March 2021), 179–184. https://doi.org/10.1007/s10209-019-00702-w
[9] Marian Pădure and Costin Pribeanu. 2019. Exploring the diferences between
fve accessibility evaluation tools. In Proceedings of RoCHI 2019-International
Conference on Human-Computer interaction. MATRIX ROM, Bucharest, Romania,
87–90. http://rochi.utcluj.ro/articole/7/RoCHI2019-Padure.pdf
[10] Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. 2017. Continuous
Integration, Delivery and Deployment: A Systematic Review on Approaches,
Tools, Challenges and Practices. IEEE Access 5 (2017), 3909–3943. https://doi.
org/10.1109/ACCESS.2017.2685629
[11] W3C Web Accessibility Initiative (WAI). 2008. Web Content Accessibility Guide-
lines (WCAG) 2.0. W3C Web Accessibility Initiative (WAI). Retrieved June 5,
2022 from https://www.w3.org/TR/WCAG20/#guidelines
[12] W3C Web Accessibility Initiative (WAI). 2018. Web Accessibility Laws & Policies.
W3C Web Accessibility Initiative (WAI). Retrieved June 5, 2022 from https:
//www.w3.org/WAI/policies/
[13] W3C Web Accessibility Initiative (WAI). 2022. Web Accessibility Evaluation Tools
List. W3C Web Accessibility Initiative (WAI). Retrieved June 5, 2022 from
https://www.w3.org/WAI/ER/tools/
[14] Webometrics. 2022. United States of America | Ranking Web of Universities: Webo-
metrics ranks 30000 institutions. Webometrics. Retrieved June 5, 2022 from https://
www.webometrics.info/en/North_america/United%20States%20of%20America
 Investigating How People with Disabilities Disclose Dificulties
on YouTube
Shuo Niu
shniu@clarku.edu
Clark University
Worcester, Massachusetts, USA
ABSTRACT
Video-sharing platforms such as Youtube are increasingly used by
people with disabilities (PWDs) to share their experiences and con-
cerns in their lives. However, there is no systematic examination
of how and why YouTubers disclose their challenges publicly on
YouTube. This poster presents a preliminary analysis of 257 video
clips made by YouTubers with disabilities. The most common dis-
closed difculties are related to social support and societal attitudes.
PWDs also use YouTube to share knowledge about accessibility
and advocate for public changes.
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility; Empirical studies in collaborative and social computing.
KEYWORDS
online videos, YouTube, video data, disability, barrier
ACM Reference Format:
Shuo Niu, Jaime Garcia, Summayah Waseem, and Li Liu. 2022. Investigating
How People with Disabilities Disclose Difculties on YouTube. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
5 pages. https://doi.org/10.1145/3517428.3550383
1 INTRODUCTION
People with disabilities (PWDs) use video-sharing platforms such
as YouTube to share everyday life stories, present creative work,
and exchange information [2]. PWDs present their daily activities
and opinions in videos, which contain the challenges and difcul-
ties they have met with social circles, technologies, and the envi-
ronment. Understanding PWDs’ unique needs center accessibility
research. Recent research has leveraged online videos to examine
PWDs’ interaction barriers and identify design opportunities. HCI
researchers studied online videos to understand PWDs’ particular
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550383
Jaime Garcia
Summayah Waseem
Li Liu
jaime.garciagarcia.455@my.csun.edu
summayah.waseem.616@my.csun.edu
lliu@csun.edu
California State University, Northridge
Northridge, California, USA
needs and challenges when doing daily activities [1, 7, 11, 12], bar-
riers with the video interactions [3, 8], and ways to have playful
experiences [4]. However, there is no systematic examination of
how YouTubers with disabilities disclose challenges and difcult
experiences. This work presents a preliminary analysis of 257 video
clips that mention disability challenges to understand how and why
PWDs use YouTube to discourse disabilities.
We collected numerous YouTube videos posted by PWDs and
sampled 1,000 video clips containing difcult words. Then the au-
thors performed an analysis of disability difculties under the envi-
ronmental barrier framework [6]. Our results indicate that YouTube
is primarily used to disclose social support pressures and societal
attitudes. PWD YouTubers use the platform to share knowledge
and experience with assistive technologies and the environment.
YouTube is also used for advocating changes and calling for in-
clusion. This work builds a foundation for our future quantitative
analysis of how PWDs discourse difculties on video-sharing plat-
forms.
2 DATA AND ENCODING
We took three steps to collect video data. In the frst step, we iden-
tifed disability keywords for fve disability categories to search
channels of YouTubers with disabilities (Table 1) with YouTube Data
API. Then the channels were selected by a mix of programmatic
and manual fltering. The included channels must contain at least
one disability keyword in the channel description and at least one
video. We also exclude channels with “center,” “organization,” “asso-
ciation,” “group,” or “mission” in the channel description to remove
channels of groups and organizations. Then we manually verifed
the channel to ensure the videos were about individual PWDs or
their caregivers. In the second step, we collected all videos posted
by these channels. We keep the videos with closed captions and
at least a difculty keyword in the closed caption (e.g., “difcult”,
“difculty”, “disappoint”, “challenge”, “hard”, “impossible”, “inconve-
nience”, 96 words in total). This step yielded 16,710 videos made
by 431 creators. In the third step, we segment videos into small
video clips by the timestamps denoted in the closed caption fle
(SubRip fle). Each SubRip timestamp has a start and ending time
and the video’s speech text for around a 5-second video segment.
We merged 24 consecutive timestamps into one video clip. The
start of each clip is the beginning of the frst timestamp, and the
end is the ending of the last timestamp. All speech texts for a clip
are concatenated. Then we only keep video clips with at least one
ASSETS ’22, October 23–26, 2022, Athens, Greece Niu, et al.

Disability Search Keywords

Vision blindness, low vision, blind, loss of vision, visual impairment
Speech speech disorders, language disorders, tourette syndrome, aphasia, speech impairment, loss of speech, speech disability, communication
disability
Mobility Amputation, arthritis, cerebral palsy, Charcot-Marie-Tooth disease, Huntington’s disease, juvenile rheumatoid arthritis, multiple
sclerosis, muscular dystrophy, scleroderma, scoliosis, spina bifda, spinal cord injury, mobility disability, physical disability, wheelchair
user, physically disabled
Hearing deafness, hard of hearing, deaf, hearing loss, hearing impairment
Cognitive & anxiety disorders, Asperger syndrome, attention defcit hyperactivity disorder, ADHD, autism, autism spectrum disorders, bipolar
Neural disorder, down syndrome, fetal alcohol syndrome, intellectual disabilities, mental retardation, learning disabilities, mental health,
obsessive compulsive disorder , OCD, post-traumatic stress disorder, PTSD, Williams syndrome, mental health disability
Table 1: Keywords used to search YouTube channels of PWD

difcult keyword in the speech text. For this poster, we sampled
1,000 video clips for qualitative coding with the disability barrier
framework [6].
3 DATA ENCODING
We used the disability barrier framework [6] to guide the genera-
tion of barrier sub-categories. The framework suggests six types of
barriers of PWD: IT access, social support and societal attitudes, sys-
tems and policies, economic, built environment, natural environment,
assistive technology, and transportation. Four authors split the 1,000
video clips and annotate whether it mentions the six barrier themes
(multi-categorical annotation). After categorization, IT access and
assistive technology clips are combined into one barrier theme
since many YouTubers mention assistive software and technologies.
The built environment, natural environment, and transportation
barriers are all related to environmental factors, and they have 22
clips in total. Therefore, we combine the clips of those three themes
into one theme. Then the four authors used afnity diagramming
to group the clip notes. The fnal sub-themes can be seen in Table 2.
4 RESULTS
257 clips from 223 videos contain at least one difculty under the
barrier framework [6]. The rest of the clips are either not a dif-
culty statement (e.g., “no problem” or “accept a challenge”), general
difculties unrelated to disabilities (e.g., “a hard math problem”),
or general hardship with the disability (e.g., “my problems with
bipolar disorder”). All speech in this section are paraphrased for
privacy protection [5].
Social Support and Societal Attitudes. The largest sub-theme
is disclosing or discussing the self-stigma with disabilities or the
shame or negativity caused by the disability. For example, in one
video, a YouTuber with a physical disability mentioned that “Physi-
cal disabilities are aggravating since they make you feel like you’re
bothering everyone else.” Some other videos call for stopping self-
stigma, such as one video mentions “We should probably stop apolo-
gizing because you aren’t truly sorry. It’s simply a means of defusing
the issue and making it less awkward.” Some YouTubers talk about
the problems with other people’s lack of awareness of their disabili-
ties. For example, one YouTube discloses that “It’s quite aggravating
for me because I’m dealing with a variety of issues relating to my
ataxia that few people understand.” Some difculties are caused by
others’ stigma, making PWDs feel excluded or isolated. A YouTuber
shared their experience “I believe that new individuals in my life
may feel unable to accept me due of my troubles, which may be rather
scary to them.” YouTubers with disabilities also mentioned the lack
of support or communities. Like one video mentions “When no one
is at home, this is how I travel around. Because of my Ehlers-Danlos
Syndrome and back problems, it’s a little challenging.” Thirteen clips
are about social problems caused by communication impairment.
For example, “People who are hard of hearing have no idea how to
speak in a public situation with someone who does not have a hear-
ing disability.” YouTubers also share their challenges with COVID.
In one video about how the coronavirus afects blind people, the
YouTuber says “Hand sanitizer was becoming increasingly difcult
to hold. It was becoming a serious issue.”
IT Access and Assistive Technologies. 49 video clips are about
problems with IT or assistive technologies. The most mentioned
technologies are assistive hardware (13 clips) and software (12 clips).
For example, a YouTuber with a motor disability shows how they
take a bath and mention the chair texture can hurt skin (Figure 2a).
In another video, a YouTuber with vision impairment demonstrates
the problem with the magnifer feature (Figure 2b). Nine clips are
about the accessibility barriers in video games. For example, a
game YouTuber with vision impairment mentions the fog efect
makes the text hard to read (Figure 2c). Seven clips discuss problems
with closed captioning. For example, one YouTuber argues that
YouTube should not remove community captions because “You’re
just alienating a large population of people by saying things like, ‘uh
catch you later,’ or ‘uh farewell, we don’t need you viewing the movies.”
Five video clips mention problems with communication tool and
AAC such as hearing aids and sign languages. Three clips mention
mobile or web software problems that contain barriers (Figure 2d).
System and Policies. System and policy barriers relate to the
availability of and access to the systems to support participation. In
our data, 14 videos mention barriers within the education system,
12 mention issues with the healthcare system, and nine complaints
about public agencies or policy problems. For example, one YouTu-
ber with vision impairment mentions their struggles in a new se-
mester that “The frst week could be difcult since you’re getting used
to learning a new way around somewhere, or making sure the new
equipment works, and that everything is in working order.” Another
YouTuber with bipolar disorder comments about the healthcare that
Investigating How People with Disabilities Disclose Dificulties on YouTube ASSETS ’22, October 23–26, 2022, Athens, Greece

Sub-theme Defnition
Video game Accessibility issues with video games
Closed captioning Issues caused closed captioning or the lack of captioning
IT & assistive
technologies

Comm. and AAC* tool Hindrance of communication caused by technologies

Assistive software Problems with assistive features in software (e.g., screen reader and magnifer)
Web & app Accessibility issues with websites, mobile apps, or computer software
Assistive device Barriers with assistive hardware, devices, and tools (e.g., wheelchair and hearing aid)
Social stigma Stigma of disability; PWDs being isolated or excluded due to discrimination
Social support &

Disability awareness Social issues caused by neglecting, misconceptions of, and nonrecognition of disabilities
societal attitudes

Social support Difculties caused by lack of social support or communities

Communication Social barriers caused by communication difculties or impairments
Self-stigma Difculties caused by PWDs’ self-stigma, shame, or negativity caused by the disability
COVID Difculties caused COVID-19, social-distancing, quarantine, and PPE
Healthcare Lack of medical treatment or information, or the poor quality of treatment and healthcare
Education Problems, discrimination, or inaccessible learning in education and childcare
& policies
Systems

Government Problems with the authorities such as government, legislation, and public policies
Employment Barriers in job market, employment, and workplace
Business & service Barriers in commercial services, business, and recreational services
Residential Problems with private and residential space, including building environment and home appliances
Environment &
transportation

Public facility Barriers found in a public environment or commercial facilities

Transportation Barriers with vehicles and transportation systems
Natural environment Problem caused by the natural environment, weather, or climate
Econ. Problems caused by inadequate economic or allocation of fnancial resources
Table 2: The sub-themes of disability barriers in [6]. *AAC standards for Augmentative and Alternative Communication

Figure 1: The distribution of videos in each of the sub-themes.

they need “additional information on the therapy’s disadvantages (a compound found in marijuana). We live in a place where none of
and criticisms so that we can see both sides” About public policies, it is yet legal.” Employment and business and services have four
one video mentions that “For her (the daughter’s) back issues and videos mentioning them.
muscle problems, I was really interested in getting some CBD oil cream
ASSETS ’22, October 23–26, 2022, Athens, Greece Niu, et al.

(a) A YouTube explains the texture of (b) A video shows a problem with (c) A YouTube gamer mentions the fog (d) A kid is struggling with using an
bathing chair can hurt skin. magnifers when reading comments. can make text hard to read. iPad with the left hand.

(e) A YouTuber tells a story about (f) A YouTuber shows it is difcult to (g) An interviewee mentions they lost
hitting a stone and overturn the use a regular ladder to fx the smoke their vision and it is hard to take
wheelchair. detector. busses.

Figure 2: Top: Example videos showing difculties with IT Access and Assistive Technologies. Bottom: Example videos showing
environment and transportation difculties

Environment and Transportation. 22 video clips mention
problems with the environment. In this theme, barriers with public
and commercial facilities are mentioned the most with 12 clips (e.g.,
Figure 2e). Six clips show their problems at home when using tools
to complete daily tasks (e.g., Figure 2f). Three clips talked about
challenges taking transportation or navigating the environment
(e.g., Figure 2g). One video mentions the challenge of the cold
weather.
Economic. Eleven videos mention economic difculties. For
example, one video mentioned a fnancial issue with social security
that “money is very tight for a lot of people, especially those people
living on a fxed income.”
5 DISCUSSION AND FUTURE WORK
YouTube for Knowledge-Sharing. YouTube is commonly used
for sharing personal experiences with technologies and public and
private environments. Disclosing the difculties may help other
people to avoid similar challenges. YouTubers share their expe-
riences with assistive tools and software features. Gamers speak
about the accessibility challenges when playing games. PWDs also
share their difculties with public facilities and everyday tools. For
other people with similar disability challenges, these videos are
valuable experiences they can learn from. Therefore, video-sharing
platforms could recommend such videos to help PWDs avoid prob-
lems or obtain solutions. For designers, YouTube can be a data
source to discover accessibility issues with their products and de-
sign. Design framework and theories are needed to guide analysis
of online videos for accessibility examination.
YouTube as a Place for Disclosing Social Pressure. Our pre-
liminary analysis suggests that the most common difculties dis-
closed by YouTubers are problems with social support and societal
attitudes [10]. PWDs reveal the self-stigma and negativity caused
by disabilities, indicating YouTube is a place for disclosing social
pressure. Our work fnds six challenges related to social support and
societal attitudes: self-stigma, disability awareness, social stigma,
lack of social support, communication challenges, and COVID-19
challenges. The prevalence of social difculty disclosure indicates
that future HCI and accessibility research need to pay attention to
PWD’s social and emotional needs. Accessibility technologies need
to be designed to ofer dignity and shield against stigma. YouTube
videos can be used to examine the social situation and societal
attitudes that hurt PWD. Future research could leverage YouTube
videos to investigate PWDs’ need for social respect and mental
support. But it should also be noted that using YouTube data to
infer the commonality of PWD difculties and accessibility barriers
may introduce statistical bias since whether YouTube creators have
the same disability distribution as the overall population needs
future investigation.
Public Advocacy with Online Videos. Our result suggests that
YouTube is also used to advocate for solving accessibility problems
[9]. YouTubers with disabilities discuss their issues with education,
healthcare, government, and employment systems to share their
struggles and raise public awareness of accessibility challenges.
The discussion of technological and environmental problems in-
dicates that PWDs want their voice heard. YouTube has a culture
of creating community and opinion leaders. Disclosing challenges
with systems and policies could promote public changes. Future
research should examine the role of online videos as a new way of
accessibility advocacy and their efects on public awareness. It will
be interesting to study how YouTubers use video-sharing commu-
nities to call for action and solve common accessibility challenges.
Meanwhile, researchers need to examine the challenges of acces-
sibility advocacy. It will be interesting to investigate whether and
how creators exaggerate video content and use other attractive
techniques to engage the viewers.
Investigating How People with Disabilities Disclose Dificulties on YouTube
REFERENCES
[1] Katya Borgos-Rodriguez, Kathryn E Ringland, and Anne Marie Piper. 2019.
MyAutsomeFamilyLife: Analyzing Parents of Children with Developmental Dis-
abilities on YouTube. Proc. ACM Hum.-Comput. Interact. 3, CSCW (11 2019).
https://doi.org/10.1145/3359196
[2] Barbara E Bromley. 2008. Broadcasting Disability: An Exploration of the Ed-
ucational Potential of a Video Sharing Web Site. Journal of Special Education
Technology 23, 4 (12 2008), 1–13. https://doi.org/10.1177/016264340802300401
[3] Dasom Choi, Uichin Lee, and Hwajung Hong. 2022. “It’s not wrong, but I’m
quite disappointed”: Toward an Inclusive Algorithmic Experience for Content
Creators with Disabilities. In CHI Conference on Human Factors in Computing
Systems. 1–19.
[4] Jared Duval, Ferran Altarriba Bertran, Siying Chen, Melissa Chu, Divya Subramo-
nian, Austin Wang, Geofrey Xiang, Sri Kurniawan, and Katherine Isbister. 2021.
Chasing Play on TikTok from Populations with Disabilities to Inspire Playful and
Inclusive Technology Design. In Proceedings of the 2021 CHI Conference on Human
Factors in Computing Systems (CHI ’21). Association for Computing Machinery,
New York, NY, USA. https://doi.org/10.1145/3411764.3445303
[5] Casey Fiesler, Nathan Beard, and Brian C Keegan. 2020. No Robots, Spiders,
or Scrapers: Legal and Ethical Regulation of Data Collection Methods in Social
Media Terms of Service. Proceedings of the International AAAI Conference on Web
and Social Media 14, 1 SE - Full Papers (5 2020), 187–196. https://ojs.aaai.org/
index.php/ICWSM/article/view/7290
[6] Joy Hammel, Susan Magasi, Allen Heinemann, David B Gray, Susan Stark, Pamela
Kisala, Noelle E Carlozzi, David Tulsky, Sofa F Garcia, and Elizabeth A Hahn. 2015.
Environmental Barriers and Supports to Everyday Participation: A Qualitative
Insider Perspective From People With Disabilities. Archives of Physical Medicine
ASSETS ’22, October 23–26, 2022, Athens, Greece
and Rehabilitation 96, 4 (2015), 578–588. https://doi.org/10.1016/j.apmr.2014.12.
008
[7] Franklin Mingzhe Li, Franchesca Spektor, Meng Xia, Mina Huh, Peter Cederberg,
Yuqi Gong, Kristen Shinohara, and Patrick Carrington. 2022. “It Feels Like Taking
a Gamble”: Exploring Perceptions, Practices, and Challenges of Using Makeup
and Cosmetics for People with Visual Impairments. In CHI Conference on Human
Factors in Computing Systems (CHI ’22). Association for Computing Machinery,
New York, NY, USA. https://doi.org/10.1145/3491102.3517490
[8] Xingyu Liu, Patrick Carrington, Xiang ’Anthony’ Chen, and Amy Pavel. 2021.
What Makes Videos Accessible to Blind and Visually Impaired People?. In
Proceedings of the 2021 CHI Conference on Human Factors in Computing Sys-
tems (CHI ’21). Association for Computing Machinery, New York, NY, USA.
https://doi.org/10.1145/3411764.3445233
[9] Shuo Niu, Cat Mai, Katherine G McKim, and D Scott McCrickard. 2021.
#TeamTrees: Investigating How YouTubers Participate in a Social Media Cam-
paign. Proc. ACM Hum.-Comput. Interact. 5, CSCW2 (10 2021). https://doi.org/10.
1145/3479593
[10] Shuo Niu, Katherine G McKim, Hess Danielle, and Kathleen P Reed. 2022. Educa-
tion, Personal Experiences, and Advocacy: Examining Drug-Addiction Videos on
YouTube. Proc. ACM Hum.-Comput. Interact. 5, CSCW2 (11 2022).
[11] Woosuk Seo and Hyunggu Jung. 2021. Understanding the community of blind
or visually impaired vloggers on YouTube. Universal Access in the Information
Society 20, 1 (2021), 31–44. https://doi.org/10.1007/s10209-019-00706-6
[12] Johann Wentzel, Sasa Junuzovic, James Devine, John Porter, and Martez Mott.
2022. Understanding How People with Limited Mobility Use Multi-Modal Input.
In CHI Conference on Human Factors in Computing Systems (CHI ’22). Association
for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3491102.
3517458
 ProAesthetics: Changing How We View Prosthetic Function
Susanna Abler
University of Maryland, Baltimore County, USA
sabler1@umbc.edu
ABSTRACT
There are multiple perspectives that must be considered while
designing a limb: The engineering requirements, the visual and
aesthetic appeal, the needs and wants of the client, and how these
designs impact users physically, socially, and mentally. Historically,
the main focus of design has been on the engineering requirements
while neglecting the individual’s needs or desires for aesthetics
which are seen as secondary concerns. However, these aspects still
hold important roles in how the user views and connects with their
own limb. In order to better understand the impact that aesthetic
design has on the individual, this exploratory case study aimed
to create custom-designed prosthetic limb covers for lower limb
amputees. This poster includes the fndings of the research and
design process, including related works, the interview, design, and
prototyping processes, as well as issues that occurred during the
project and how they were either overcome or could be addressed
in the future.
CCS CONCEPTS
• Human-centered computing → Accessibility.
KEYWORDS
3D Printing, Assistive Technology, Co-Design, Aesthetics
ACM Reference Format:
Susanna Abler and Foad Hamidi. 2022. ProAesthetics: Changing How We
View Prosthetic Function. In The 24th International ACM SIGACCESS Con-
ference on Computers and Accessibility (ASSETS ’22), October 23–26, 2022,
Athens, Greece. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/
3517428.3550386
1 INTRODUCTION AND BACKGROUND
In Design Meets Disability, Graham Pullin argues strongly for the
need to explore how design and disability can inform and inspire
each other, and in particular why it is important to explore the
incorporation of aesthetic design into the construction of varying
types of assistive technology [11]. When it comes to the design of
prosthetic limbs, the main area of focus has often been on improv-
ing the mechanical capabilities of the device to replicate biological
capabilities as closely as possible. While this is not inherently a
bad thing, it can lead to some level of neglect towards the visual
and aesthetic aspects of the prosthesis, often viewing the appear-
ance as secondary to restoration of movement. Furthermore, the
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550386
Foad Hamidi
University of Maryland, Baltimore County, USA
foadhamidi@umbc.edu
creation of the prosthesis often involves the end-user minimally,
outside of measurements for size and tests for comfort much of the
prosthetic limb is already prefabricated. There is great potential
in exploring how users view the appearance of a prosthetic limb
and how involvement in the design process of a cosmetic limb or
limb cover can lead to designs that are personalized and refect
aspects of a user’s identity. In this project, we adopted a participa-
tory interdisciplinary approach to study how input from prosthetic
users can be combined with technical knowledge of the prosthetics
manufacturing process as well as an eye for visual composition
to create aesthetically pleasing prosthetic designs. We engaged in
the prototyping and fabrication of three customized lower-limb
prosthetics that focused on aesthetic appeal and personalization
and found that users had innovative ideas for how to customize
their devices.
This project is informed and inspired by previous research that
has studied the impact of aesthetic appearance on individuals with
limb loss as well as efective methods for pursuing the design pro-
cess. Customizing prosthetics and assistive technologies (ATs), more
broadly, has been studied for the past few decades in the context
of the work of Do-It-Yourself (DIY) AT communities [4, 5, 8, 9].
Of particular relevance to this project is the work of Profta et al.
that showed customization practices within the deaf community
that went beyond functionality and used aesthetic aspects of hear-
ing aids to represent aspects of identity [10]. Other projects have
shown how online communities create and share customized ATs
[4, 9] and also identifed the challenges of balancing practical, aes-
thetic and political considerations when creating customized ATs
[7]. Other eforts have focused on developing co-design methods
where people with disabilities, AT experts, and designers come
together to create customized and personalized designs (e.g., [1, 6]).
Finally, Bekrater-Bodmann et al. developed a Prosthesis Embodi-
ment Scale [2, 3] used to describe how prosthesis-users connect
with their prosthetic limb and how to evaluate that connection.
These projects motivated the current exploratory case study that
focuses on inquiring into aesthetic preferences and considerations
of lower-limb prosthetic users to augment existing devices to make
them more appealing. Our fndings show that prosthetic users may
have personally meaningful preferences for what their devices look
and feel like that if incorporated into prosthetic design may make
them more appealing and aligned with one’s identity.
2 METHODS
Five individuals with lower-limb amputations participated in semi-
structured interviews regarding their interest in cosmetic covers
as well as how they felt about their current limb appearance. Each
interview was approximately an hour, occurring during a regularly
scheduled appointment at the clinic where they receive care for
their prosthetic limb. Three women and two men were interviewed,
all were white and above the age of thirty. Notes were taken during
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 1: "Jersey Devil" limb cover (digital 3D model top Left, 3D printed model bottom Left), and 3D printed "Flame On" cover
design (Right)
the interview process as recording devices were prohibited. From
these notes key words and quotes were taken out to provide context
for their relationship to their current prosthetic device as well as
providing insight into what their interests were regarding cosmetic
covers. The data from these interviews was then used to design
aesthetically focused limb covers, beginning with digital sketches
and 3D models of the limbs using Autodesk’s Fusion 360. The
participants had each given diferent criteria for their desired covers,
whether it be certain colors to use or avoid, sports teams they liked,
or activities that they enjoyed. Due to time constraints amplifed
by COVID-19 restrictions, we were unable to get feedback from the
participants regarding the initial sketches. After initial designs were
fnalized, the models were printed utilizing additive 3D printing
technology. Due to the varying aesthetic desires of each participant,
each limb cover had its own unique challenges and prototyping
needs that we will describe in the next section. Our university’s
Institutional Review Board (IRB) ofce approved the study before
data collection began.
3 FINDINGS
Of the fve interviews conducted, two of the participants stated
that they currently had minimal to no interest in cosmetic covers
while three stated that they had previously considered altering the
appearance of their prosthetic. The three who had some previous
interest in altering the appearance of their limb had varying levels
of current interest, one was no longer interested in altering the
appearance of their limb, another was still interested in it, but it was
simply less of a priority, and the third was receptive to the concept
of altering the appearance of their limb as it was something they
Susanna Abler and Foad Hamidi
had thought of but had not really pursued. These answers were con-
sistent with data from the literature review; some individuals were
very narrowly focused on realism, others had an interest in more
robotic or abstract appearances, and some were happy with their
limbs the way that they were [5, 6]. From there they were further
interviewed about what they might like to see in a limb cover for
themselves, with varying answers given between each individual.
Some were more interested in a realistic looking cover, while others
had varying interest in less lifelike designs. Answers as to why they
felt the way they did about their preferences varied. One participant
responded “everyone wants to look normal” when asked about her
preference for a realistic limb cover. Another participant mentioned
she is around children often and is concerned about her prosthetic
scaring them, she wants to make sure her limb is something that the
children can connect with as well and a realistic limb cover would
raise fewer concerns with these younger children. Two participants
with bionic knees mentioned the importance of a cover acting as
protection for the mechanical components inside their prosthetic
limb and that such a focus was often considered more important
than aesthetics due to insurance covering protective covers rather
than aesthetic covers.
An overarching issue regarding prosthetic limbs is the expense
of the limbs themselves with the additional cost of the amputation
procedure if applicable. Many people feel like they cannot aford the
additional cost of cosmetic changes, especially if they are paying
out of pocket. Financial stressors can impact well-being and mental
health and further medical costs of occupational therapy and limb
maintenance can impact continued use of a limb [7]. While aford-
ability and access to limbs is an important factor in both designing a
ProAesthetics: Changing How We View Prosthetic Function
Figure 2: "Peace and Love" prosthetic limb cover, front (Left) and back (Center) of the model, detail of DIY glitter snow globe
(Right)
prosthetic limb and continued acceptance of the limb into the body
identity of the individual, this research is focused more narrowly
on the aesthetics of the limb and how that impacts the user, areas
relating to public policy and insurance our outside of the scope of
the research in this project.
From the fve designs described by participants, we selected a
total of three design concepts, based on their practicality and possi-
ble aesthetic appeal, to proceed to the design and prototyping stage.
The designs ranged from more simple colors with minor additions
to requiring diferent types of materials and separate prototyp-
ing artifacts. The most simplistic design was the one titled "Jersey
Devil," (Figure 1, Left) as all that was requested was something in
red and black that contained the logo of the participant’s favorite
hockey team. This design required little additional prototyping and
was fairly simple to model and print, the most difcult part was
modeling the logo to be visible but also solidly connected to the
body of the cover.
The second most complicated cover was the "Flame On" design
(Figure 1, Right). The cover itself was simple to model as it was
solid, but due to the fact that it was designed for an above-the-knee
amputee it was much larger than the other two designs and had to
be printed in two parts. There were several issues that occurred with
the printing of this cover in which prints would fail in the middle
of the print, taking extra time and troubleshooting to complete.
For the fame design, a few diferent avenues were explored. Vinyl
stickers were explored as an option; however, it was discovered
they do not stick well to the PLA material and would eventually
peel of. Further research was done into how they do fame decals
for cars and motorcycles and made some attempts with painting on
the design. From this research the vinyl fame stickers were then
repurposed to act as a stencil for the fames, using tape to further
defne the edges of the shape. Glossy black spray paint was used
over the remainder of the cover.
ASSETS ’22, October 23–26, 2022, Athens, Greece
The most time-intensive cover was the "Peace and Love" design
(Figure 2). The creation of the cutouts for this model was fairly
time intensive due to limited features in the Fusion 360 program.
Fusion360 does not yet have support for wrapping a sketch around
a model, so shapes had to be created on individual sketch planes
and cut out to prevent stretching. Nylon was used for this cover due
to its fexibility and because it is a dyeable material. Nylon test dye
sticks were printed to evaluate dying time needed for the material
to achieve the most vibrant colors in the requested pink and purple.
These sticks were then dipped in the dye for ffteen seconds to two
minutes to see what time would produce the most ideal color for
the covers.
This cover came with an additional area of prototyping due
to the inclusion of a "glitter snow globe heart" in the cover. This
snow globe was made using acrylic sheets, water resistant adhesive,
glitter, and vegetable glycerin. A heart shaped metal cookie cutter
and a heat gun to heat gun was used to cut the sheets clear acrylic,
those that the cutter could not cut through fully were cut with
scissors. These shapes created the top and bottom of the snow
globe. Long thin strips of acrylic were cut out and heated up to
then mold along the inside of the cookie cutter, creating the middle
portion of the snow globe. These were then adhered to the heart
cutout shapes and flled with vegetable glycerin and glitter. Varying
concentrations of the glycerin were evaluated for fow and sustained
movement for the glitter.
4 DISCUSSION
The results from the interviews show that some participants had
an interest in cosmetic limb covers and creative and personal ideas
for what they could look like. Each participant had their own reac-
tions to the idea of custom cosmetic limb covers, whether it was
immediate interest or indiference. And each came in with their
own reasons for that reaction, be it fnancial, cultural, or personal.
Additionally, we found that it is feasible to create custom limb
ASSETS ’22, October 23–26, 2022, Athens, Greece
cover prototypes for individuals utilizing commercially available
materials, making this a more accessible avenue for the user.
Involving the end-user in the process of design and understand-
ing the impact of their involvement is important for future devel-
opment of assistive technology. There is great value in the end user
not only being involved in the design of their limb but learning how
to design for themselves, having more agency in the appearance
of their limb [8]. The interviews showed that individuals do have
an interest in custom cosmetic covers and while follow up on the
covers was not acquired, the criteria given by the participants was
fulflled through the project.
Another refection that our research points to but needs further
future exploration is questioning the assumption that a prosthetic
limb is necessary for all individuals with limb diferences. Much of
the research shows that individuals with a congenital limb difer-
ence are less likely to want or need a prosthetic limb to function in
daily life [5]. The same can go for amputees as over time they may
fnd they are more comfortable, happier, and more mobile without
a prosthetic limb. The purpose of this research is not to imply that
this method is the only way for prosthesis users to experience the
world or fnd comfort. Rather, the research is aimed at those who do
want to use a prosthetic limb and fnd they are not happy with their
device, specifcally if that dissatisfaction stems from the appearance
of the prosthesis. The goal of this research is to help individuals
feel happier with themselves and how they present themselves to
the world, and those who feel happier without a prosthesis have
already reached that point.
This research also highlighted the need for an interdisciplinary
lens when approaching prosthetic design. The task of designing
these prosthetic limb covers required the use of disciplines like me-
chanical engineering, visual arts, psychology, and disability studies
to both designs aesthetically pleasing and functional covers as well
as provide contextual understanding of the micro and macro im-
pacts that a prosthetic limb and disability has on the individual.
These disciplines each worked to create a broader understanding
of that context, beginning with the design process and needs of the
limb itself to understanding the societal structures that infuence
the design of the device. Having a broad understanding of the fac-
tors that can impact both the user and their prosthetic device is
important in the design of future assistive technology as the many
physical, mental, and social factors that impact the individual’s life
will dictate the type of assistive technology they need. While this
research took on an interdisciplinary approach, it was limited by
the knowledge base of our team, a limitation that can be remedied
by working with a broader team of experts in the future.
5 CONCLUSION AND FUTURE WORK
In this exploratory research, we studied how to co-design and
prototype aesthetic elements for prosthetics using DIY method and
consumer-grade prototyping materials, such that the designs refect
users’ personal preferences and desires and go beyond functionality.
We designed and prototyped three designs based on participant
specifcations.
There are several directions for future research. First, we can
build on the current work to develop improved co-design models
that bring together multiple perspectives into the design process.
Susanna Abler and Foad Hamidi
Further research into this topic can also examine the role of pub-
lic policy and regulations in how aesthetic aspects of assistive
technologies are viewed and how does this interact with devices’
afordability and functionality. We focused only on lower-body
prosthetics in this study. Future work can explore devices designed
for other parts of the body and with other fabrication methods.
Finally, and more importantly, future research can seek feedback
from individuals with disabilities on co-designed prosthetics in an
iterative process.
ACKNOWLEDGMENTS
We would like to thank our participants and the clinic staf for their
time and input. We would also like to thank Steven McAlpine, Jamie
Gurganus, Symmes Gardner, and Drew Holladay. This project is
partially supported by National Science Foundation (NSF) under
Grants DRL-2005502 and DRL-2005484.
REFERENCES
[1] Leila Afatoony and Su Jin (Susan) Lee. 2020. AT makers: A multidisciplinary ap-
proach to co-designing assistive technologies by co-optimizing expert knowledge.
In Proceedings of the 16th Participatory Design Conference 2020 - Participation(s)
Otherwise - Volume 2, ACM, New York, NY, USA. DOI:https://doi.org/10.1145/
3384772.3385158
[2] Robin Bekrater-Bodmann. 2020. Factors Associated With Prosthesis Embodiment
and Its Importance for Prosthetic Satisfaction in Lower Limb Amputees. Front.
Neurorobot. 14, (2020), 604376.
[3] Robin Bekrater-Bodmann. 2020. Perceptual correlates of successful body–
prosthesis interaction in lower limb amputees: psychometric characterisation
and development of the Prosthesis Embodiment Scale. Sci. Rep. 10, 1 (August
2020), 1–13.
[4] Erin Buehler, Stacy Branham, Abdullah Ali, Jeremy J. Chang, Megan Kelly Hof-
mann, Amy Hurst, and Shaun K. Kane. 2015. Sharing is Caring. In Proceedings of
the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI
’15, ACM Press, New York, New York, USA. DOI:https://doi.org/10.1145/2702123.
2702525
[5] Erin Buehler, Amy Hurst, and Megan Hofmann. 2014. Coming to grips: 3D
printing for accessibility. In Proceedings of the 16th international ACM SIGACCESS
conference on Computers & accessibility (ASSETS ’14), Association for Computing
Machinery, New York, NY, USA, 291–292.
[6] Megan Hofmann, Jefrey Harris, Scott E. Hudson, and Jennifer Mankof. 2016.
Helping hands: Requirements for a Prototyping Methodology for Upper-limb
Prosthetics Users. In Proceedings of the 2016 CHI Conference on Human Factors
in Computing Systems, ACM, New York, NY, USA. DOI:https://doi.org/10.1145/
2858036.2858340
[7] Megan Hofmann, Kristin Williams, Toni Kaplan, Stephanie Valencia, Gabriella
Hann, Scott E. Hudson, Jennifer Mankof, and Patrick Carrington. 2019. Occupa-
tional therapy is making: : Clinical Rapid Prototyping and Digital Fabrication. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems,
ACM, New York, NY, USA. DOI:https://doi.org/10.1145/3290605.3300544
[8] Jennifer Mankof, Megan Hofmann, Xiang “anthony”Chen, Scott E. Hudson, Amy
Hurst, and Jeeeun Kim. 2019. Consumer-grade fabrication and its potential to
revolutionize accessibility. Commun. ACM 62, 10 (September 2019), 64–75.
[9] Jeremiah Parry-Hill, Patrick C. Shih, Jennifer Mankof, and Daniel Ashbrook.
2017. Understanding Volunteer AT Fabricators: Opportunities and Challenges
in DIY-AT for Others in e-NABLE. In Proceedings of the 2017 CHI Conference
on Human Factors in Computing Systems (CHI ’17), Association for Computing
Machinery, New York, NY, USA, 6184–6194.
[10] Halley P. Profta, Abigale Stangl, Laura Matuszewska, Sigrunn Sky, and Shaun
K. Kane. 2016. Nothing to Hide: Aesthetic Customization of Hearing Aids and
Cochlear Implants in an Online Community. In Proceedings of the 18th Interna-
tional ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’16),
Association for Computing Machinery, New York, NY, USA, 219–227.
[11] Graham Pullin. 2011. Design Meets Disability. MIT Press, London, England.
 Towards Visualization of Time–Series
Ecological Momentary Assessment (EMA) Data on Standalone
Voice–First Virtual Assistants
Yichen Han1 , Christopher Bo Han3 , Chen Chen1 , Peng Wei Lee2 ,
Michael Hogarth5 , Alison A. Moore5 , Nadir Weibel1 , Emilia Farcas4
1 Computer Science and Engineering, 2 Electrical and Computer Engineering,
3 Departmentof Mathematics, 4 Qualcomm Institute, 5 School of Medicine
University of California San Diego, La Jolla, California, United States
{y4han,cbhan,chenchen,pwlee,mihogarth,alm123,efarcas,weibel}@ucsd.edu

Figure 1: With touchscreen based standalone voice–frst virtual assistant, older adults are able to query and visualize the
time–series based Ecological Momentary Assessment (EMA) data (e.g., the quality and time of the sleep).
ABSTRACT
Population aging is an increasingly important consideration for
health care in the 21th century, and continuing to have access and
interact with digital health information is a key challenge for ag-
ing populations. Voice-based Intelligent Virtual Assistants (IVAs)
are promising to improve the Quality of Life (QoL) of older adults,
and coupled with Ecological Momentary Assessments (EMA) they
can be efective to collect important health information from older
adults, especially when it comes to repeated time-based events.
However, this same EMA data is hard to access for the older adult:
although the newest IVAs are equipped with a display, the efective-
ness of visualizing time–series based EMA data on standalone IVAs
has not been explored. To investigate the potential opportunities
for visualizing time–series based EMA data on standalone IVAs, we
designed a prototype system, where older adults are able to query
and examine the time–series EMA data on Amazon Echo Show
— a widely used commercially available standalone screen–based
IVA. We conducted a preliminary semi–structured interview with
a geriatrician and an older adult, and identifed three fndings that
should be carefully considered when designing such visualizations.
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550398
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility.
KEYWORDS
Gerontechnology, Accessibility, Health – Well-being, User Experi-
ence Design, Older Adults, Voice User Interfaces, EMA
ACM Reference Format:
Yichen Han1 , Christopher Bo Han3 , Chen Chen1 , Peng Wei Lee2 ,, Michael
Hogarth5 , Alison A. Moore5 , Nadir Weibel1 , Emilia Farcas4 . 2022. Towards
Visualization of Time–Series Ecological Momentary Assessment (EMA)
Data on Standalone Voice–First Virtual Assistants. In The 24th International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22),
October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA, 4 pages.
https://doi.org/10.1145/3517428.3550398
1 INTRODUCTION
Population aging is a global health consideration in the 21th cen-
tury [8], especially when it comes to keeping track of older adults’
daily health data. To more efectively support older adults and their
clinicians to collect daily health check-ins, studies have shown the
efectiveness of using Ecological Momentary Assessments (EMA),
a well–known method to more conveniently assess specifc recur-
ring behaviors and track specifc health states. An efective EMA
strategy could allow clinicians to constantly monitor older adults
and ensure their physical and mental well-being, hopefully leading
to an increase in their Quality of Life (QoL) [9, 18].
However, such strategies often necessitates tremendous amounts
of time and efort; and further issues, such as communication chal-
lenges, could arise [20]. While a wide variety of technologies have
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 2: Two sample demonstrations on visualizing time-series data on Amazon Echo Show with voice input (a–b) and a semi-
structured interview exploring their utility (c). Faces are pixelized to protect participants’ personally identifable information.
been used to keep track of the older adults’ conditions [16, 19], many
older adults are still facing obstacles while using smart devices,
most notably those with complicated Graphical User Interfaces
(GUI) [13, 14].
Voice based conversational user interfaces ofers an promising
alternative interaction channel for older adults to interact with
their digital health information. Such modality allows older adults
to naturally and easily use voice based conversation–like com-
mands to delegate tasks to, or query information from their health-
care providers. While Trajkova et al. [21] demonstrates the limita-
tions of today’s voice assistants among aging users, others (e.g., [3–
5, 15, 17]) show the feasibility and potential usefulness of using
standalone voice–based Intelligent Virtual Assistants (IVAs) for
conducting EMA among older adults patients.
Despite the potential of deploying EMAs through IVAs, due
to the nature of ambiguity of conversational voice user interfaces,
Chen et al. [5, 6] pointed out the necessity of also integrating simple
visual components for facilitating better interaction with the EMAs
deployed on the device. The same group of researchers [6] also
investigated how to visualize simple interaction elements, such as
the texts of EMA questions and input buttons on Amazon Echo
Show [10], a widely used commercially available standalone IVAs.
However, while data collection through EMA is important, previous
work did not yet explore how to designs and visualize the data
collected through EMAs on the IVA device itself, especially when
it comes to display time–series visualizations.
In this paper we present the frst preliminary exploratory efort
to design time–series data visualizations on screen based voice–
frst IVAs. We chose Amazon Echo Show [10] as the standalone
voice–based IVA device with a built–in touchscreen due to its dom-
inant market share [1], however, many of our fndings could be
transferred to other screen–based voice–frst IVAs. We prototyped
a system that enables older adults to examine their past EMA data
on the built–in touchscreen of Echo Show. By observing interaction
with the IVA visualization, and especially older adults’ past data
and corresponding trends, we found that older adults might be able
to make in-situ decisions to address possible unhealthy lifestyle.
We conducted a remote co–design workshop with two repre-
sentative stakeholders – an older adult and a geriatrician. After
thematic analysis of the recordings, we present fndings and poten-
tial future development.
Han et al.
2 PRELIMINARY DESIGN
Our preliminary designs consist of two example scenarios demon-
strating how time–series EMA data can be visualized and are bene-
fcial to older adults (see Figs. 2a and b). In both cases, we assume
that the EMA data has already been collected and stored in a secure
Electronic Health Record (EHR). We considered scenarios of one–
measure visualization, where we only visualize one measure
(e.g., the hour of sleep as shown in Fig. 2a) over the time domain,
and two–measure visualization, that places two one–measure
graphs with the same dimension vertically, which might indicate
possible correlations between two dependent variables. Visualiza-
tions with more than two measures are out of our scope due to the
consideration of the screen size and participants’ cognitive load.
Scenario 1 – One–Measure Visualization: Fig. 2a demonstrates
a one–measure visualization for hours of sleep, one important time-
series EMA data used in geriatrics to help tracking older adults’
QoL [7]. A voice skill could be initiated by a simple query: “Hey
Alexa, what’s my average hours of sleep everyday in the week?” The
mean hours of sleep by days in a week will then be computed, yield-
ing a line plot shown on the IVA’s screen. The trend of the graph
could help older adults recall the common activities on particular
days in a week that possibly led to an irregularity of sleeping hours.
Scenario 2 – Two–Measure Visualization: Older adults should
also be able to visualize two–measure plots with the same time
span to examine any potential correlations. As an example, Fig. 2b
demonstrates how older adults could query about their sleep hours
and alcohol consumption in the past 10 days. This plot can visually
guide the user to fnd the correlation between the daily alcohol
consumption and sleep, so that the user can properly in–situ adjust
to a healthier lifestyle.
Prototype: We prototyped the system based on Amazon Alexa
and Amazon Web Services (AWS). The overall system architecture
is shown in Fig. 3, and consists of three major components: (1) a
front–end converting users’ utterances to queries, (2) a middle–tier
that handles the queries and requests the graphs, and (3) a back–
end to generate the graphs. In the front–end, speech recognition
technology in the IVAs, such as Alexa Voice Service (AVS) in an
Echo Show, are used to recognize the raw audio input. An AWS
Towards Visualization of Time–Series
Ecological Momentary Assessment (EMA) Data on Standalone Voice–First Virtual Assistants
Figure 3: When older adults request a graph from voice assistant Alexa. AWS Lambda handles this request by asking EC2 to
generate a graph and send it back to Echo show, paired with Alexa.
Lambda Function is then triggered and passes through the tran-
scribed text to the back-end server via RESTful API which will
then generate a time-series graph. Through Amazon Presentation
Language (APL) [12], the plot is then rendered on the touchscreen
as a background image. Through this process, older adults are able
to view their time–series healthcare information as a graph on their
standalone devices.
3 DESIGN AND USER EVALUATION
To understand how data visualization can help the aging population,
we conducted a remote 30–minute co–design workshop through
Zoom [22], and interviewed an older adult (P1) from the retirement
community "Vi" in La Jolla, California and a geriatric provider (P2)
from the University of California San Diego Health System. Our
study has been approved by the Institutional Review Board (IRB).
We showed the interviewees our two initial visualization prototypes
(see Fig. 2a and b) and solicited feedback, encouraging them to
propose alternatives. Specifcally, the participants were asked to
express their opinions regarding (a) the potential functionality and
efects of the proposed EMA visualizations, as well as (b) how useful
these visualizations would be to their daily routine. Fig. 2c shows a
screenshot of our Zoom session with the two stakeholders.
4 PRELIMINARY FINDINGS
We recorded the complete Zoom session and analyzed the discus-
sion using analysis [11]. In this section we summarize our fndings
in terms of three aspects.
(1) Selections of Measures to be Visualized Should be Care-
fully Considered.
Participants agreed that mean values, as visualized in Fig. 2a, could
provide clear insights while retrospecting the older adults’ activities
over the previous weeks. However, participants also expressed a
preference for seeing additional measures to help evaluating the
spread of the measured data. For example, P2 mentioned that if
“error bars” existed in Fig. 2a,“the preciseness of the number of sleep-
ing hours” could be shown. This revision would help visualize the
consistency of sleeping time among aging populations.
(2) The Span of Horizontal and Vertical Axis Should be Care-
fully Designed.
Participants suggested to have more fexibility in terms of the range
for the vertical axis. As shown in Fig. 2a, the range of the vertical
axis is a four–hour diference. Changing the range of the vertical
axis to be more fexible would allow users to display a more ac-
curate variability of the number of sleeping hours. P2 mentioned
ASSETS ’22, October 23–26, 2022, Athens, Greece
that having a set vertical span has the limitation of being “bounded
by the responses they [the users] have” and emphasized that hav-
ing a fexible vertical range could be implemented “so people don’t
potentially freak out”.
Participants also emphasized the importance of carefully design-
ing the time span (i.e., visualized as horizontal axis). In Figs. 2a and
b, the graphs show data within only a short time span (e.g., the
time span of Fig. 2a is a week and the time span of Fig. 2b is 10
days). However, healthcare providers could request data from a
longer time span, such as from the past several months or years.
P2 in fact questioned the efectiveness of graphing only tens days
of data in Fig. 2b and suggested to “make the axis longer so you[the
visualization] show more” health data. Future designs should adapt
the time span according to the user’s demand.
(3) The Types of Data to be Visualized Should be Carefully
Chosen.
Currently, the prototypes only display quantitative data, but there
are numerous essential qualitative data points and types that could
be included in the graphs, as suggested by the participants. The
quality of sleep, an example of qualitative data, can provide a clearer
picture on how alcohol is afecting the user’s sleep. This data type,
however, is difcult to be implemented in the visualizations. One
method is to “build it in as a color ... to overlay the number of [sleep-
ing] hours with quality”, as suggested by P2. The addition of "setting
goals" perhaps overlaid on top of the quantitative data would also
make the visualizations more personal to the user, since meeting
their goal encourages the user to maintain a better lifestyle. P1
mentioned that “you might set your goal fairly low... so if you’re
consistently meeting your lower goal, then that gives you a chance to
consider whether you want to increase the goal or not”. In the graphs
showing the correlation between the amount of consumed alcohol
and the sleeping time, the time diference between when the user
last consumed alcohol and the time they fell asleep the same day is
also a factor that afects sleep. Therefore, this variable could also
be considered to add into the visualizations.
5 CONCLUSION AND FUTURE WORK
To help older adults visualize their time-series EMA data through
voice input, we designed and implemented a prototype that can
generate one–measure and two–measure graphs on standalone
voice–frst virtual assistants. We explored opportunities of the sys-
tem and summarized needs on 3 major components of this visual-
ization, which are measures, types, and time span of data. Although
some of these opportunities can be easily implemented, others like
providing a better picture of the user’s conditions and goals, can be
ASSETS ’22, October 23–26, 2022, Athens, Greece
difcult to integrate into the visualizations and interpreted correctly
by the user and the healthcare providers.
This work is only scratching the surface in terms of how to design
graphical visualization of EMAs on IVAs, and outline a number of
challenges that still need to be resolved. In future work, we plan
to conduct user studies with older participants to fully explore
opportunities and challenges faced by this a system like the one we
have prototyped. Meanwhile, integrating some straight-forward
suggested refnements as part of our existing prototype will already
make visualizations more adaptive to context and representation
of qualitative data.
ACKNOWLEDGMENTS
This work is part of project VOLI [2] and was supported by NIH/NIA
under grant R56AG067393. Co-author Michael Hogarth has an
equity interest in LifeLink Inc. and also serves on the company’s
Scientifc Advisory Board. The terms of this arrangement have been
reviewed and approved by the UC San Diego in accordance with
its confict of interest policies. We appreciate insightful feedback
from the anonymous reviewers, Manas Satish Bedmutha, Mary
Draper and fellow colleagues from the Design Lab and Computer
Science and Engineering at UC San Diego, as well as residents from
the Vi in La Jolla.
REFERENCES
[1] 2019. Alexa Devices Maintain 70% Market Share in U.S. according to sur-
vey. https://marketingland.com/alexa-devices-maintain-70-market-share-in-u-
s-according-to-survey-265180
[2] 2022. Project VOLI at UC San Diego. http://voli.ucsd.edu
[3] Rebecca Adaimi, Ka Tai Ho, and Edison Thomaz. 2020. Usability of a Hands-
Free Voice Input Interface for Ecological Momentary Assessment. In 2020 IEEE
International Conference on Pervasive Computing and Communications Workshops
(PerCom Workshops). IEEE, 1–5.
[4] Kemeberley Charles, Chen Chen, Janet G. Johnson, Alice Lee, Ella T. Lifset,
Michael Hogarth, Nadir Weibel, Emilia Farcas, and Alison A. Moore. 2021. How
might an intelligent voice assistant address older adults’ health-related needs?. In
Journal of the American Geriatrics Society, Vol. 69. Wiley 111 River St, Hoboken
07030-5774, NJ USA, S243–S244.
[5] Chen Chen, Janet G Johnson, Kemeberly Charles, Alice Lee, Ella T. Lifset, Michael
Hogarth, Alison A. Moore, Emilia Farcas, and Nadir Weibel. 2021. Understanding
Barriers and Design Opportunities to Improve Healthcare and QOL for Older
Adults through Voice Assistants. In The 23rd International ACM SIGACCESS
Conference on Computers and Accessibility (Virtual Event, USA) (ASSETS ’21).
Association for Computing Machinery, New York, NY, USA, Article 9, 16 pages.
https://doi.org/10.1145/3441852.3471218
[6] Chen Chen, Khalil Mrini, Kemeberly Charles, Ella T. Lifset, Michael Hogarth,
Alison A. Moore, Nadir Weibel, and Emilia Farcas. 2021. Toward a Unifed
Metadata Schema for Ecological Momentary Assessment with Voice-First Virtual
Assistants. In CUI 2021 - 3rd Conference on Conversational User Interfaces (Bilbao
Han et al.
(online), Spain) (CUI ’21). Association for Computing Machinery, New York, NY,
USA, Article 31, 6 pages. https://doi.org/10.1145/3469595.3469626
[7] Kate Crowley. 2011. Sleep and sleep disorders in older adults. Neuropsychology
review 21, 1 (2011), 41–53.
[8] Carl-Johan Dalgaard, Casper Worm Hansen, and Holger Strulik. 2022. Physio-
logical aging around the World. PloS one 17, 6 (2022), e0268276.
[9] Lorraine S Evangelista, Jung-Ah Lee, Alison A Moore, Marjan Motie, Hassan
Ghasemzadeh, Majid Sarrafzadeh, and Carol M Mangione. 2015. Examining the
efects of remote monitoring systems on activation, self-care, and quality of life
in older patients with chronic heart failure. The Journal of cardiovascular nursing
30, 1 (2015), 51.
[10] Amazon Inc. 2020. Echo Show 8 – HD smart display with Alexa – stay connected
with video calling. https://www.amazon.com/Echo-Show-Pantalla-inteligente-
Alexa/dp/B07PF1Y28C/ref=sr_1_1?dchild=1&keywords=echo+show&qid=
1598674780&sr=8-1
[11] Helene Jofe. 2012. Thematic analysis. Qualitative research methods in mental
health and psychotherapy 1 (2012), 210–223.
[12] Alexa Presentation Language. 2022. Echo Show 8 – HD smart display with Alexa
– stay connected with video calling. https://developer.amazon.com/en-US/docs/
alexa/alexa-design/apl.html
[13] Rock Leung, Charlotte Tang, Shathel Haddad, Joanna Mcgrenere, Peter Graf, and
Vilia Ingriany. 2012. How older adults learn to use mobile devices: Survey and
feld investigations. ACM Transactions on Accessible Computing (TACCESS) 4, 3
(2012), 1–33.
[14] Qingchuan Li and Yan Luximon. 2020. Older adults’ use of mobile device: us-
ability challenges while navigating various interfaces. Behaviour & Information
Technology 39, 8 (2020), 837–861.
[15] Ella T. Lifset, Kemeberley Charles, Emilia Farcas, Nadir Weibel, Michael Hogarth,
Chen Chen, Janet G. Johnson, and Alison A. Moore. 2020. Can an Intelligent
Virtual Assistant (IVA) Meet Older Adult Health-Related Needs in the Context of
a Geriatric 5Ms Framework?. In Journal of the American Geriatrics Society, Vol. 70.
Wiley 111 River St, Hoboken 07030-5774, NJ USA, S245–S246.
[16] W Ben Mortenson, Andrew Sixsmith, and Ryan Woolrych. 2015. The power (s)
of observation: Theoretical perspectives on surveillance technologies and older
people. Ageing & Society 35, 3 (2015), 512–530.
[17] Khalil Mrini, Chen Chen, Ndapa Nakashole, Nadir Weibel, and Emilia Farcas.
2021. Medical Question Understanding and Answering for Older Adults. Southern
California Machine Learning and Natural Language Processing Symposium (2021).
[18] Martin Salzmann-Erikson and Henrik Eriksson. 2012. Panoptic power and mental
health Nursing—Space and surveillance in relation to staf, patients, and neutral
places. Issues in Mental Health Nursing 33, 8 (2012), 500–504.
[19] Andrew Sixsmith, Gloria Gutman, et al. 2013. Technologies for active aging. Vol. 9.
Springer.
[20] Annelie J. Sundler, Hilde Eide, Sandra van Dulmen, and Inger K. Holmström. 2016.
Communicative challenges in the home care of older persons – a qualitative explo-
ration. Journal of Advanced Nursing 72, 10 (2016), 2435–2444. https://doi.org/10.
1111/jan.12996 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/jan.12996
[21] Milka Trajkova and Aqueasha Martin-Hammond. 2020. “Alexa is a Toy”: Ex-
ploring Older Adults’ Reasons for Using, Limiting, and Abandoning Echo. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York,
NY, USA, 1–13. https://doi.org/10.1145/3313831.3376760
[22] Matin Yarmand, Chen Chen, Danilo Gasques, James D. Murphy, and Nadir Weibel.
2021. Facilitating Remote Design Thinking Workshops in Healthcare: The Case
of Contouring in Radiation Oncology. In Extended Abstracts of the 2021 CHI
Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI
EA ’21). Association for Computing Machinery, New York, NY, USA, Article 40,
5 pages. https://doi.org/10.1145/3411763.3443445
 Understanding and Improving Information Extraction From
Online Geospatial Data Visualizations for Screen-Reader Users
Ather Sharif Andrew M. Zhang Anna Shih
asharif@cs.washington.edu azhang26@cs.washington.edu annas55@cs.washington.edu
Paul G. Allen School of Computer Paul G. Allen School of Computer Paul G. Allen School of Computer
Science & Engineering | DUB Group, Science & Engineering, Science & Engineering,
University of Washington University of Washington University of Washington
Seattle, Washington, USA Seattle, Washington, USA Seattle, Washington, USA

Jacob O. Wobbrock Katharina Reinecke

wobbrock@uw.edu reinecke@cs.washington.edu
The Information School | Paul G. Allen School of Computer
DUB Group, Science & Engineering | DUB Group,
University of Washington University of Washington
Seattle, Washington, USA Seattle, Washington, USA

Figure 1: Interactions with a geospatial data visualization showing COVID-19 cases per US state, using our enhancements to
VoxLens. “Q” represents questions that screen-reader users can verbally ask using our enhancement, and “A” represents the
answers they would hear via their screen readers.

ABSTRACT remain unexplored. In this work, we study the interactions of and

Prior work has studied the interaction experiences of screen-reader
users with simple online data visualizations (e.g., bar charts, line
graphs, scatter plots), highlighting the disenfranchisement of screen-
reader users in accessing information from these visualizations.
However, the interactions of screen-reader users with online geospa-
tial data visualizations, commonly used by visualization creators
to represent geospatial data (e.g., COVID-19 cases per US state),
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550363
information extraction by screen-reader users from online geospa-
tial data visualizations. Specifcally, we conducted a user study with
12 screen-reader users to understand the information they seek
from online geospatial data visualizations and the questions they
ask to extract that information. We utilized our fndings to generate
a taxonomy of information sought from our participants’ interac-
tions. Additionally, we extended the functionalities of VoxLens—an
open-source multi-modal solution that improves data visualization
accessibility—to enable screen-reader users to extract information
from online geospatial data visualizations.
CCS CONCEPTS
• Human-centered computing → Information visualization;
Accessibility systems and tools.
ASSETS ’22, October 23–26, 2022, Athens, Greece
KEYWORDS
geospatial, visualization, accessibility, screen reader, blind, voice
assistant, map
ACM Reference Format:
Ather Sharif, Andrew M. Zhang, Anna Shih, Jacob O. Wobbrock, and Katha-
rina Reinecke. 2022. Understanding and Improving Information Extraction
From Online Geospatial Data Visualizations for Screen-Reader Users. In
The 24th International ACM SIGACCESS Conference on Computers and Acces-
sibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York,
NY, USA, 5 pages. https://doi.org/10.1145/3517428.3550363
1 INTRODUCTION
Online data visualizations efectively communicate large volumes
of data to their audience [8] and enable users to identify outliers,
recognize patterns, and explore oddities in the data that may be
challenging to determine from a simple table [25]. The benefts of
data visualizations are especially applicable to map-based visual-
izations that assist users in exploring, summarizing, and analyzing
geospatial data [2, 26, 27]. Indeed, geospatial data visualizations dis-
playing information on current events, including COVID-19 cases
and vaccination rates per US state and country, were amongst the
top 10 most popular interactive data visualizations in 2021 [21, 24].
However, screen-reader users—who may not be able to interact
with online visualizations fully using sight—are inherently disen-
franchised from extracting information from online visualizations
due to the inaccessibility of the visualizations [4, 11, 15, 22]. Sev-
eral prior works have identifed the need for accessible online data
visualizations, shedding light on the disenfranchisement caused by
inaccessible visualizations for screen-reader users [10, 13, 15, 22, 28].
Most recently, Sharif et al. [22] reported that even when an alter-
native text (“alt-text”) exists, screen-reader users spend 211% more
time and are 61% less accurate in extracting information from online
data visualizations compared to non-screen-reader users. However,
their work only explored simple online data visualizations, such as
bar charts, line graphs, and scatter plots. We build on their work by
understanding interaction experiences and information extraction
by screen-reader users from online geospatial data visualizations.
To understand the information screen-reader users seek from
online geospatial data visualizations and the questions they ask
to extract that information, we conducted a Wizard-of-Oz [3, 6]
user study with 12 screen-reader users. We found that our partici-
pants grouped and fltered geospatial data through categorization
and ranking in addition to extracting and comparing individual
data points. We utilized these fndings to develop a taxonomy of
information sought by our screen-reader users during their explo-
rations. Finally, using the taxonomy, we extended the function-
alities of VoxLens [23]—an open-source JavaScript plug-in that
improves the accessibility of online data visualizations using a
multi-modal approach—by supporting information extraction from
online geospatial data visualizations.
In this work, we contribute the: (1) taxonomy of information
sought by our screen-reader users in their explorations of online
geospatial data visualizations; and (2) enhancement of VoxLens [23]—
an open-source JavaScript plug-in to make online data visualiza-
tions accessible—to support information extraction from online
geospatial data visualizations.
Ather Sharif, Andrew M. Zhang, Anna Shih, Jacob O. Wobbrock, and Katharina Reinecke
2 USER STUDY
We conducted a Wizard-of-Oz [3, 6] user study with 12 screen-
reader users to understand their information extraction experiences
with online geospatial data visualizations, subsequently generat-
ing a taxonomy of their interactions. We acted as the “wizards”
and simulated responses from a hypothetical screen reader, follow-
ing recommendations from prior work [9, 12, 16]. We present our
methodology, results, and the taxonomy development process.
2.1 Participants, Materials, & Procedure
Our participants were 12 screen-reader users (M=50.3 years, SD=13.6;
see Appendix A, Table 2). Seven participants self-identifed as
women and fve as men. We compensated participants with a $20
Amazon gift card for one hour of their time. Our data set included
three geospatial data visualizations (curated based on the search
results for “most popular map visualizations 2021” on Google): (1)
US trafc congestion in 2021; (2) Latest COVID-19 vaccination per-
centages per country; and (3) Percentage of US workers at or below
minimum wage in 2021 per state.
We conducted our studies via Zoom and used its built-in fea-
tures for recording and transcribing sessions. First, we presented
participants with a holistic overview of the visualization generated
using VoxLens’ Summary mode. Then, we asked our participants
to explore the data in the visualization by verbally asking questions,
replicating the behavior of the Q-&-A mode of VoxLens. Each par-
ticipant interacted with all three visualizations. We randomized the
order of the visualizations across participants.
2.2 Analysis & Results
We used semantic thematic analysis [14, 19] and employed Braun
and Clarke’s “essentialist” method [1], focusing on the “surface
meanings of the data.” At least two researchers independently coded
the transcripts and identifed 18 initial thematic codes, resolving our
disagreements through mutual discussions. We combined our 18
initial codes into eight axial codes and classifed the axial codes into
two broader categories. Our inter-rater reliability (IRR), expressed
as percentage agreement [7], was 89.4%, demonstrating a high level
of agreement between raters [5, 7].
We found that besides extracting and comparing individual data
points, our participants performed additional actions in their in-
teractions with online geospatial data visualizations, which we
classifed into two high-level categories: (1) Categorization; and (2)
Ranking. Specifcally, we found that screen-reader users categorize
data by (in the order of frequency): regional, political, climate-related,
population-related, and spoken-language-related. For example, P4
inquired about alcohol consumption diferences between diferent
regions of Asia: Are there geographic diferences, like, between eastern
Asia versus western Asia or southern Asia? Our participants also
ranked data based on their values. Specifcally, they sought (in the
order of frequency): top and bottom X data points and X data points
surrounding the average; where X is an arbitrary number that varied
across participants. For example, P3 wanted to fnd the top three
countries in 2021 for COVID-19 vaccination percentages: So, Hong
Kong was the highest, but who is in the second and third position?
Information Extraction From Online Geospatial Data Visualizations ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Taxonomy of information screen-reader users seek when exploring online geospatial data visualizations to extract and
compare data points. Information types within each category are in descending order based on their sought frequency. For each
information type, the “Query” column shows some of the questions that our participants asked to extract that information.

Category Information Type Query

Is there a diference between the east and west, or the north and south?
Regional
Tell me the values of the Southern states as opposed to the Northwest.
How do the trends during the Democratic presidential campaigns com-
Political pare to the trends during Republican presidential campaigns?
Do socialist countries have higher rates?
Categorization
Do warmer climate states have higher values?
Climate-Related
How are colder places compared to warmer places?
Can you compare these two states by population?
Population-Related
Do the states with larger population have higher trafc rates?
Spoken-Language-Related Can we compare Spanish speaking countries to the English ones?
What are the fve top countries in Western Europe?
Top
I’d like to see them all in order from the highest to the lowest.
What are the bottom 10 countries in the graph?
Ranking Bottom
What about the second and third lowest?
Which ones are in the middle?
Surrounding Average
What are the three countries that are closest to the average?

2.3 Taxonomy Development Specifcally, we implemented regional categorization and ranking

Using these fndings, we developed a taxonomy of information
sought by screen-reader users containing three tiers: (1) Category,
the broader categories; (2) Information Type, the axial codes; and
(3) Query, questions that our participants asked to extract a given
information type. We show the taxonomy in Table 1, organizing the
categories and the information types in the order of their frequency.
3 ENHANCEMENTS TO VOXLENS
Drawing on the results from the taxonomy from our user study with
12 screen-reader users, we extended the capabilities of VoxLens by
supporting information extraction from geospatial data visualiza-
tions.
3.1 Brief Overview of VoxLens
VoxLens is an open-source JavaScript plug-in that improves the
accessibility of online data visualizations for screen-reader users us-
ing a multi-modal approach [23], requiring only a single line of code
for integration from developers. VoxLens supports three modes: (1)
Q-&-A (verbal interaction); (2) Summary (holistic overview of the
data); and (3) Sonifcation (sonifed version of the data). However,
VoxLens is currently limited to simple visualizations, such as bar
charts, line graphs, and scatter plots.
the data by the top- and bottom-most values. To identify the chart
type and regional classifcation to accurately process users’ queries,
we extended the existing confguration options for developers to
include two more parameters: “chartType” and “dataModule.” Four
values for “chartType” are possible: (1) bar; (2) line; (3) scatter; and
(4) map. For “dataModule,” two values are possible: (1) state; and
(2) country. (No additional data or confguration was required from
the developers.)
3.2.1 Regional Categorization. For data involving the states in the
US, our participants categorized the data by US regions (e.g., east
coast); for countries of the world, they grouped the data by conti-
nents (e.g., Asia). Therefore, we implemented two data modules:
state and country. State allows the data to be grouped and fltered
by US region, whereas country does so by continent. Developers
can enable regional categorization by using map as the “chartType”
and specifying the appropriate “dataModule.” For example, in a
graph representing COVID-19 cases per US state, the user can ask
region-related questions, such as: “how is the east coast vs. the
west coast?” (Figure 1). We chose the US regions using National
Geographic Society’s [18] classifcation of US regions. Additionally,
we made our modules scalable, enabling straightforward additions
and modifcations to the list of our regions.

3.2.2 Support for Ranking. Our enhancements enable the users to

3.2 Our Additions to VoxLens
Our objective was to extend the functionality of VoxLens’ Q-&-
A mode by enabling screen-reader users to extract information
from online geospatial data visualizations. To this end, we selected
the most frequently sought information types from our taxonomy.
obtain the top X and bottom X data points, where X represents any
number of data points. For example, users can ask for the top-seven
or the bottom-fve data points. Our algorithm, currently, only rec-
ognizes specifc keywords to rank the data (e.g., “top” or “bottom”).
We plan on extending the vocabulary in our future iteration.
ASSETS ’22, October 23–26, 2022, Athens, Greece
4 DISCUSSION & CONCLUSION
In this work, we presented a taxonomy of information sought by
screen-reader users in their interactions with online geospatial data
visualizations, generated using the fndings from our user study
with 12 screen-reader users. Our work is the frst to understand and
improve the information extraction of screen-reader users from
online geospatial data visualizations. We found that screen-reader
users perform regional categorization to extract and compare data
points. Additionally, screen-reader users rank the data based on the
values of the dependent variable, arranging it by the highest, lowest,
and nearest-to-the-average. Utilizing our fndings, we extended the
capabilities of VoxLens [23], enabling screen-reader users to extract
information from online geospatial data visualizations.
In our study, a recurring yet unsurprising observation was that
each participant exhibited a distinct way of interacting with online
geospatial data visualizations. Although we found high-level simi-
larities in their interactions, their word choices and verbosity levels
for the questions they asked to extract information were unique.
Therefore, we recommend using personalized designs [17, 20] that
cater to the individualized preferences of users by identifying us-
age patterns (e.g., input queries issued to extract information) to
improve the interaction experiences of screen-reader users.
We plan on conducting task-based user studies with screen- and
non-screen-reader users to assess the performance of our enhance-
ment to VoxLens using a mixed-methods approach. Additionally,
we intend to extend the functionality of VoxLens to include more
complex data visualizations, such as multi-line graphs. Future work
can employ our methodology to build systems that use voice as-
sistants for screen-reader users to improve their information ex-
traction. We hope that by providing insights into the screen-reader
users’ interactions with geospatial data visualizations and open-
sourcing our code, this work will inspire researchers and developers
to make online data visualizations more accessible to screen-reader
users and reduce the information access disparity between screen-
and non-screen-reader users caused by inaccessible visualizations.
ACKNOWLEDGMENTS
This work was supported by the University of Washington Center
for Research and Education on Accessible Technology and Expe-
riences (CREATE). Finally, we thank and remember our recently-
departed team member Zoey for her feline support, without which
the purrusal of this work would not have been as efective. May she
cross the rainbow bridge in peace and fnd her way to cat heaven.
REFERENCES
[1] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative research in psychology 3, 2 (2006), 77–101.
[2] Martin Breunig, Patrick Erik Bradley, Markus Jahn, Paul Kuper, Nima Mazroob,
Norbert Rösch, Mulhim Al-Doori, Emmanuel Stefanakis, and Mojgan Jadidi. 2020.
Geospatial data management research: Progress and future directions. ISPRS
International Journal of Geo-Information 9, 2 (2020), 95.
[3] Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz stud-
ies—why and how. Knowledge-based systems 6, 4 (1993), 258–266.
[4] Joel J Davis. 2002. Disenfranchising the Disabled: The Inaccessibility of Internet-
Based Health Information. Journal of Health Communication 7, 4 (2002), 355–367.
https://doi.org/10.1080/10810730290001701
[5] Matthew Graham, Anthony Milanowski, and Jackson Miller. 2012. Measuring
and Promoting Inter-Rater Agreement of Teacher and Principal Performance
Ratings.
Ather Sharif, Andrew M. Zhang, Anna Shih, Jacob O. Wobbrock, and Katharina Reinecke
[6] Melita Hajdinjak and France Mihelic. 2004. Conducting the Wizard-of-Oz Exper-
iment. Informatica (Slovenia) 28, 4 (2004), 425–429.
[7] Donald P Hartmann. 1977. Considerations in the choice of interobserver reliability
estimates. Journal of applied behavior analysis 10, 1 (1977), 103–116.
[8] Jake Holland. 2017. New York Times’ Upshot editor discusses data visualiza-
tion, storytelling. https://dailynorthwestern.com/2017/05/03/campus/new-york-
times-upshot-editor-discusses-data-visualization-storytelling/. (Accessed on
03/05/2022).
[9] Todd Hunt. 1982. Raising the Issue of Ethics through Use of Scenarios. The
Journalism Educator 37, 1 (1982), 55–58.
[10] Mario Konecki, Charles LaPierre, and Keith Jervis. 2018. Accessible data visual-
ization in higher education. In 2018 41st international convention on information
and communication technology, electronics and microelectronics (MIPRO). IEEE,
Institute of Electrical and Electronics Engineers, New York, NY, USA, 0733–0737.
[11] Bongshin Lee, Arjun Srinivasan, Petra Isenberg, John Stasko, et al. 2021. Post-
WIMP Interaction for Information Visualization. Foundations and Trends® in
Human-Computer Interaction 14, 1 (2021), 1–95.
[12] Peng Liang and Onno De Graaf. 2010. Experiences of using role playing andwiki
in requirements engineering course projects. In 2010 5th International Workshop
on Requirements Engineering Education and Training. IEEE, Institute of Electrical
and Electronics Engineers, New York, NY, USA, 1–6.
[13] Alan Lundgard, Crystal Lee, and Arvind Satyanarayan. 2019. Sociotechnical
considerations for accessible visualization design. In 2019 IEEE Visualization
Conference (VIS). IEEE, Institute of Electrical and Electronics Engineers, New
York, NY, USA, 16–20.
[14] Moira Maguire and Brid Delahunt. 2017. Doing a thematic analysis: A practical,
step-by-step guide for learning and teaching scholars. All Ireland Journal of
Higher Education 9, 3 (2017), 3351–3364.
[15] Kim Marriott, Bongshin Lee, Matthew Butler, Ed Cutrell, Kirsten Ellis, Cagatay
Goncu, Marti Hearst, Kathleen McCoy, and Danielle Albers Szafr. 2021. Inclusive
data visualization for people with disabilities: a call to action. Interactions 28, 3
(2021), 47–51.
[16] Randall B Martin. 1991. The assessment of involvement in role playing. Journal
of clinical psychology 47, 4 (1991), 587–596.
[17] Esther Nathanson. 2017. Native voice, self-concept and the moral case for per-
sonalized voice technology. Disability and rehabilitation 39, 1 (2017), 73–81.
[18] National Geographic Society. n.d.. United States Regions | National Geographic
Society. https://www.nationalgeographic.org/maps/united-states-regions/. (Ac-
cessed on 03/29/2022).
[19] Michael Quinn Patton. 1990. Qualitative evaluation and research methods. SAGE
Publications, Inc., Thousand Oaks, CA, USA.
[20] Silvia Quarteroni and Suresh Manandhar. 2007. User modelling for personalized
question answering. In Congress of the Italian Association for Artifcial Intelligence.
Springer, Springer Berlin Heidelberg, Berlin, Heidelberg, 386–397.
[21] Nick Routley. 2021. Our Top 21 Visualizations of 2021 - Visual Capitalist. https:
//www.visualcapitalist.com/our-top-21-visualizations-of-2021/. (Accessed on
06/18/2022).
[22] Ather Sharif, Sanjana Shivani Chintalapati, Jacob O. Wobbrock, and Katharina
Reinecke. 2021. Understanding Screen-Reader Users’ Experiences with Online
Data Visualizations. In The 23rd International ACM SIGACCESS Conference on
Computers and Accessibility (Virtual Event, USA) (ASSETS ’21). Association for
Computing Machinery, New York, NY, USA, Article 14, 16 pages. https://doi.
org/10.1145/3441852.3471202
[23] Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and
Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible
with an Interactive JavaScript Plug-In. In CHI Conference on Human Factors in
Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing
Machinery, New York, NY, USA, Article 478, 19 pages. https://doi.org/10.1145/
3491102.3517431
[24] Terence Shin. 2021. The 10 Best Data Visualizations of 2021 | by Terence
Shin | Towards Data Science. https://towardsdatascience.com/the-10-best-data-
visualizations-of-2021-fec4c5cf6cdb?gi=13652b78af45. (Accessed on 06/18/2022).
[25] Antony Unwin. 2020. Why is Data Visualization Important? What is Important
in Data Visualization? · Issue 2.1, Winter 2020. https://hdsr.mitpress.mit.edu/
pub/zok97i7p/release/3. (Accessed on 03/05/2022).
[26] Jia Yu, Zongsi Zhang, and Mohamed Sarwat. 2018. Geosparkviz: a scalable geospa-
tial data visualization framework in the apache spark ecosystem. In Proceedings of
the 30th international conference on scientifc and statistical database management.
1–12.
[27] Xianfeng Zhang and Micha Pazner. 2004. The icon imagemap technique for multi-
variate geospatial data visualization: approach and software system. Cartography
and Geographic Information Science 31, 1 (2004), 29–41.
[28] Jonathan Zong, Crystal Lee, Alan Lundgard, JiWoong Jang, Daniel Hajas, and
Arvind Satyanarayan. 2022. Rich Screen Reader Experiences for Accessible Data
Visualization. In The 24th EG/VGTC Conference on Visualization (EuroVis’ 22),
Rome, Italy, 13-17 June, 2022. Eurographics-European Association for Computer
Graphics.
Information Extraction From Online Geospatial Data Visualizations ASSETS ’22, October 23–26, 2022, Athens, Greece

A PARTICIPANT DEMOGRAPHICS

Table 2: Screen-reader participants, their gender identifcation, age, screen reader, vision level, and diagnosis. Under the “G”
(Gender) column, M = Male, F = Female, and N B = Non-binary.

G Age Screen Vision-Loss Level Diagnosis

Reader
P1 M 36 JAWS Blind since birth, Complete blindness Leber Congenital Amaurosis
P2 M 58 JAWS Complete blindness, Lost vision gradually Cataracts and Glaucoma
P3 M 49 JAWS Complete blindness, Lost vision gradually Leber Congenital Amaurosis
P4 M 32 NVDA Blind since birth, Complete blindness Peters Anomaly
P5 F 32 NVDA Blind since birth, Complete blindness Retinopathy of Prematurity
P6 F 65 JAWS Complete blindness, Lost vision gradually Retinitis Pigmentosa
P7 F 68 Fusion Lost vision gradually, Partial blindness Stargaart’s Maculopathy
P8 F 69 JAWS Blind since birth, Complete blindness Retinopathy of Prematurity
P9 F 52 JAWS Blind since birth, Complete blindness Retinopathy of Prematurity
P10 F 38 JAWS Blind since birth, Complete blindness Leber Congenital Amaurosis
P11 F 47 JAWS Complete blindness, Lost vision gradually Meningitis, Optic Neuropathy
P12 M 57 JAWS Complete blindness, Lost vision gradually Retinitis Pigmentosa
Understanding Design Preferences for Sensory-Sensitive Earcons
with Neurodivergent Individuals
Lauren Race Kia El-Amin
Twitter Inc., San Francisco, CA, U.S. Twitter Inc., San Francisco, CA, U.S.
Andrew Hayward Amber James
Twitter UK Ltd., London, U.K Twitter Inc., San Francisco, CA, U.S.
Audrey Davis
Twitter Inc., San Francisco, CA, U.S.
ABSTRACT
Earcons are a critical auditory modality for those who perceive
information best through sound. Yet earcons can trigger sensory
sensitivities with neurodivergent individuals, causing pain or dis-
comfort and creating barriers to information access. They must be
carefully designed with neurodivergent representation in the design
process to minimize the harm they impose. To address these chal-
lenges, we conduct a study on Twitter, a social media platform with
frequent earcons, to understand how to design sensory-sensitive
earcons for neurodivergent individuals. We present the results of
our qualitative interviews with nine neurodivergent Twitter users,
uncovering six key themes for designing sensory-sensitive earcons.
Based on our fndings, we ofer a set of novel guidelines for practi-
tioners to design sensory-sensitive earcons for accessibility.
CCS CONCEPTS
• Human-centered computing; • Human computer interac-
tion (HCI);
KEYWORDS
Accessibility, Design, Neurodiversity, Sensory Sensitivities, Sound,
Earcons, Social Media
ACM Reference Format:
Lauren Race, Kia El-Amin, Sarah Anoke, Andrew Hayward, Amber James,
Amy Hurst, Audrey Davis, and Theresa Mershon. 2022. Understanding
Design Preferences for Sensory-Sensitive Earcons with Neurodivergent
Individuals. In The 24th International ACM SIGACCESS Conference on Com-
puters and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece.
ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3517428.3550365
1 INTRODUCTION
An estimated range of 7% [16] to 20% [11] of adults identify as
neurodivergent—a term describing individuals who have learning,
cognitive, and psychological disabilities. Neurodivergence can take
many forms, including Attention Defcit Hyperactivity Disorder
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550365
Sarah Anoke
Twitter Inc., San Francisco, CA, U.S.
Amy Hurst
New York University, New York, NY,
U.S.
Theresa Mershon
Twitter Inc., San Francisco, CA, U.S.
(ADHD), Autism Spectrum Disorder (ASD), or Generalized Anxiety
Disorder (GAD) and result in experiencing sensory sensitivities
[34]. Sensory sensitivities present challenges with sound, which
can cause distraction [33], pain, or discomfort [10]. Yet the Web
Content Accessibility Guidelines (WCAG) lack specifc guidance on
supporting sensory sensitivities and neurodivergence [28], while
cognitive and learning disability guidance is framed as supplemen-
tal [27] and, therefore, not required to meet compliance minimums.
For sound design, WCAG recommends 1) the ability to turn of
sound longer than three seconds [30]; 2) ensuring sound is not the
only way to perceive information [29]; and 3) making sure users
can separate speech from background sounds [31]. Earcons—or au-
dio notifcations—are commonly used on the web and 1) are often
shorter than three seconds [30]; 2) must work for those who require
sound to perceive information and have sensory sensitivities, such
as individuals with multiple disabilities [29]; and 3) are primary
sounds, not background sounds [31]. Earcons must be as accessible
for those who prefer to have them turned on but are sensitive to out-
put modalities. Thus, we propose sensory-sensitive earcons (Figure
1) that minimize the pain and discomfort triggered by sensory sensi-
tivities. We crafted them with neurodivergent representation in the
design process [26] to reduce neurotypical assumptions embedded
in the fnal artifact and improve overall design [8].
The goal of this research is to better understand how to design
sensory-sensitive earcons for neurodivergent users. We explore this
research goal on Twitter, a social media platform that is a source
of frequent notifcations. We describe our qualitative interviews
with nine neurodivergent Twitter users, present the fndings from
our study to understand design preferences for sensory-sensitive
earcons, and contribute a set of six novel, concrete design guidelines.
2 RELATED WORK
Past HCI literature studies the design of technologies to support neu-
rodivergent individuals who have sensory sensitivities. Researchers
have probed support methods for neurodivergent users managing
distracting and uncomfortable sensory experiences by using au-
dio [22], proprioceptive [24], haptic [25], and visual [18, 19, 32]
modalities. Zolyomi et al. found that neurodivergent adults relied
on coping strategies to manage distracting noises which imposed
cognitive load [33]. Damiani studied how sound intensity and pitch
can cause pain and discomfort [10]. To support sensory needs, schol-
ars have advocated for including a range of sensory preferences
ASSETS ’22, October 23–26, 2022, Athens, Greece
to accommodate varying individuals’ thresholds for sensory input
[4]. Though some design processes aim to incorporate a spectrum
of sensory preferences [21], Spiel and Gerling found that neuro-
divergent individuals are often excluded from the design process
[26]. Dalton argued for the involvement of neurodiverse designers
who can create from lived experience [9]. Thus, researchers have
identifed frameworks for practitioners to include neurodiverse per-
spectives throughout the design process [2, 16, 23]. With various
technologies competing for our attention [20], past research has
investigated how to improve earcon design to increase compre-
hension and decrease the cognitive load imposed. Scholars have
researched how customizing notifcations for sensory preferences
benefts those who prefer auditory output [3] and those who prefer
non-auditory notifcations [12]. Researchers have looked at how
earcons can assist in situations where visual interfaces cannot [5],
such as leveraging unique sounds users could learn for commonly
used words such as “thanks” [14]. The design of earcons is complex
since—though pitch and timbre can be leveraged to capture atten-
tion [1]—some end users have described earcons as “discordant”
and “too obtrusive”, advocating for more “aesthetically pleasant”
sounds [13].
Though prior HCI literature examines both neurodiversity and
earcons, there is a lack of research investigating how earcons afect
neurodivergent users or identifes the variables in earcons that
trigger sensory sensitivities with that user population. Further,
there are currently no design guidelines for earcons that minimize
the pain and discomfort triggered by sensory sensitivities. Our work
addresses this gap in accessibility and HCI literature by expanding
existing earcon design guidelines [6, 7, 15] to include the needs of
neurodivergent users.
3 UNDERSTANDING SENSORY SENSITIVE
EARCON DESIGN
Our research team included both neurodivergent and neurotypical
individuals to incorporate lived experience, while initially explor-
ing which types of earcons trigger sensory sensitivities and should
be avoided. We recruited participants with a screener survey sent
to 22,439 individuals to fnd Twitter users between the ages of 18-
65, asking them their age, gender, whether they self-identifed as
neurodivergent, and if they had used Twitter in the past month.
We received 201 qualifed responses before identifying nine Twitter
users (1 nonbinary, 1 trans male, 3 female, 4 male; ages 18-35) who
self-identifed as neurodivergent and had experience using audio
notifcations on their mobile devices. With informed consent, we
conducted qualitative, semi-structured interviews via teleconfer-
encing software. During the interviews, we asked key questions
such as: 1) Do you keep sound notifcations turned on or of? Why? 2)
What do you think makes a sound annoying? 3) What are some types
of sounds you would like to see less of in apps? 4) Tell me about a time
when you had a bad experience with sound notifcations. 5) What
sound notifcations do you like, or even love? 6) Was there ever a sound
notifcation that was fne at frst, but increasingly bothered you over
time? Can you describe that sound? We recorded video, transcribed
the audio, and stored the data on a secure cloud server. The frst and
second author reviewed the data and performed a thematic analysis.
Our fndings yielded a set of six sensory-sensitive earcon themes,
Lauren Race et al.
which we used to develop a concrete set of sensory-sensitive earcon
design guidelines. We used the guidelines to create an earcon for
the send tweet interaction on Twitter (Figure 1).
Figure 1: A sensory-sensitive earcon for send tweet. The sound
wave of a sensory-sensitive earcon is one second or less, min-
imizes wavelength space between repetitive notes, has an ∼
40-50 dB intensity, ∼ 1k Hz frequency, and uses a slow attack
and decay with no sustain and release for a smooth sound
envelope. Image credit: Listen.
4 DESIGN GUIDELINES FINDINGS
Our interviews uncovered six key themes for designing sensory-
sensitive earcons: 1) duration; 2) repetition; 3) sound envelope; 4)
intensity and frequency; 5) familiarity; and 6) purpose. Many (5
participants) described challenges with duration. They told us that
“ding” sounds were bothersome because they would linger, while
shorter sounds were more pleasant. When asked to provide specifc
duration guidance, they said that shorter than one second was ideal.
P6 told us: “Half a second to a second at most because, other than that,
they feel obtrusive.” Additionally, 6 participants reported sensory
sensitivities with repetition. They told us that hearing a sound mul-
tiple times annoyed them and was unnecessary for understanding
an earcon. P5 said, “I don’t need to hear it three times to know that
it happened.” Almost all (8 participants) reported issues with the
shape of the sound envelope. They told us that sudden earcons
annoyed them, while gentler earcons were appreciated. P9 said,
“Sounds are more pleasant when they’re a little softer or smoother,
whether it’s a gradual increase in volume or notes that fow together.”
Almost all (8 participants) highlighted issues with intensity and
frequency. High-pitched or loud notes caused discomfort with par-
ticipants, even if they had their computing device volume settings
turned down. P1 said: “I don’t like high-pitched noises at all. I fnd
them extremely uncomfortable and jarring.” They told us that more
balanced and soothing tones were better tolerated. Most (6 partic-
ipants) detailed their views with familiarity. They explained that
recognizable sounds could be more clearly mapped to an action. P1
said: “I like something that I can recognize right of the bat. That’s
another reason why I like the skeuomorphic sounds because I can iden-
tify what it is trying to tell me immediately.” Most (6 participants)
reported challenges with purpose. They said that excessive sound
that did not identify anything specifc could feel overwhelming. P9
said they did not like “Generic sounds that don’t relate to anything
or aren’t identifying anything specifc—they’re there to fll the space.”
They told us that they would like more careful and practical earcon
placement. P1 said: “If there has to be a sound to something, it doesn’t
necessarily have to be attached to every button that you press, but it
should be functional in some way.”
Understanding Design Preferences for Sensory-Sensitive Earcons with Neurodivergent Individuals
Table 1: Sensory Sensitive Earcon Design Guidelines
1. Keep duration to one second or less. 2. Minimize wavelength space between
repetitive notes to avoid percussive
sounds.
4. Keep intensity mid-range (∼ 40-50 dB) 5. Use recognizable, skeuomorphic
and the frequency close to the human sounds.
voice (∼ 1k Hz).
5 DISCUSSION & RECOMMENDATIONS
While prior earcon themes focus on increasing comprehension [4]
[6, 7, 15], the themes we uncovered focus on increasing accessibility
for neurodivergent individuals with sensory sensitivities. These
themes expand upon WCAG [28] sound design requirements for
turning of sound longer than three seconds [30], making sure
[5]
sound is not the only output modality [29], and sound clarity [31]
by minimizing pain and discomfort with shorter sounds the user
has enabled. This study underscores that WCAG compliance is the
[6]
minimum and unique use cases—such as earcons—need additional
considerations for accessibility. Based on the six key themes for [7]
designing sensory-sensitive earcons, we provide recommendations
with our guidelines (Table 1).
[8]
6 CONCLUSION & FUTURE WORK
Earcons can trigger sensory sensitivities with neurodivergent indi-
viduals, yet must be accessible for those who prefer to use them. To
make earcons more accessible, we wanted to learn how to design [9]
earcons for neurodivergent individuals. We described our qualita-
tive interviews with nine neurodivergent Twitter users to inform
the earcon designs. Our study identifed six key themes for design- [10]
ing sensory-sensitive earcons: duration, repetition, sound envelope,
intensity and frequency, familiarity, and purpose. Based on our fnd-
ings, we contributed six, novel concrete guidelines for practitioners [11]
to design sensory-sensitive earcons. We see opportunities to evalu-
ate future earcon concepts generated from the design guidelines [12]
by surveying a larger sample size of both neurodivergent and neu-
rotypical users, explore sensory-sensitive earcons on other social
media platforms, and research how sensory sensitivities impact
[13]
other output modalities in HCI.
ACKNOWLEDGMENTS
We thank our participants, Emmanuelle Bastien, Anita Butler, Rosie [14]
Cabreros, Gerard Cohen, Kristine Ibale, Brett Lewis, Listen, Katrina
Lofaro, Nada Rastad, Tess Sitzmann, Jason Sofonia, and Johanna
Stein.
[15]
REFERENCES
[1] David Beattie, Lynne Baillie, and Martin Halvey. 2017. Exploring How Drivers
Perceive Spatial Earcons in Automated Vehicles. Proc. ACM Interact. Mob. Wear- [16]
able Ubiquitous Technol. 1, 3, Article 36 (September 2017), 24 pages. DOI:https:
//doi.org/10.1145/3130901
[2] Laura Benton, Asimina Vasalou, Rilla Khaled, Hilary Johnson, and Daniel Gooch.
2014. Diversity for design: a framework for involving neurodiverse children in the
technology design process. In Proceedings of the SIGCHI Conference on Human [17]
Factors in Computing Systems (CHI ’14). Association for Computing Machinery, [18]
New York, NY, USA, 3747–3756. DOI:https://doi.org/10.1145/2556288.2557244
[3] Athina Bikaki and Andreas Floros. 2011. An RSS-feed auditory aggregator using
earcons. In Proceedings of the 6th Audio Mostly Conference: A Conference on
ASSETS ’22, October 23–26, 2022, Athens, Greece
3. Use a smooth sound envelope, with a slow
attack and decay, and no sustain or release.
6. Only use earcons for confrming something
has happened or indicating an error.
Interaction with Sound (AM ’11). Association for Computing Machinery, New
York, NY, USA, 95–100. DOI:https://doi.org/10.1145/2095667.2095681
LouAnne Boyd, Kendra Day, Ben Wasserman, Kaitlyn Abdo, Gillian Hayes, and
Erik Linstead. 2019. Paper Prototyping Comfortable VR Play for Diverse Sensory
Needs. In Extended Abstracts of the 2019 CHI Conference on Human Factors
in Computing Systems (CHI EA ’19). Association for Computing Machinery,
New York, NY, USA, Paper LBW1714, 1–6. DOI:https://doi.org/10.1145/3290607.
3313080
Stephen A. Brewster. 1998. Using earcons to improve the usability of tool palettes.
In CHI 98 Conference Summary on Human Factors in Computing Systems
(CHI ’98). Association for Computing Machinery, New York, NY, USA, 297–298.
DOI:https://doi.org/10.1145/286498.286775
Stephen Brewster, Peter Wright, and Alistair Edwards. 1995. Experimentally
derived guidelines for the creation of earcons.
Stephen A. Brewster, Peter C. Wright, and Alistair D. N. Edwards. 1993. An eval-
uation of earcons for use in auditory human-computer interfaces. In Proceedings
of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing
Systems (CHI ’93). Association for Computing Machinery, New York, NY, USA,
222–227. DOI:https://doi.org/10.1145/169059.169179
Jim A. Carter and David W. Fourney. 2007. Techniques to assist in developing
accessibility engineers. In Proceedings of the 9th international ACM SIGACCESS
conference on Computers and accessibility (Assets ’07). Association for Comput-
ing Machinery, New York, NY, USA, 123–130. https://doi.org/10.1145/1296843.
1296865
Nicholas Sheep Dalton. 2013. Neurodiversity & HCI. In CHI ’13 Extended
Abstracts on Human Factors in Computing Systems (CHI EA ’13). Associa-
tion for Computing Machinery, New York, NY, USA, 2295–2304. DOI:https:
//doi.org/10.1145/2468356.2468752
Luca M. Damiani. 2019. Hyper Sensorial – Human Computed Neurodivergent
Poem. In Extended Abstracts of the 2019 CHI Conference on Human Factors in
Computing Systems (CHI EA ’19). Association for Computing Machinery, New
York, NY, USA, Paper VS04, 1–2. DOI:https://doi.org/10.1145/3290607.3311779
Nancy Doyle, Neurodiversity at work: a biopsychosocial model and the impact
on working adults, British Medical Bulletin, Volume 135, Issue 1, September 2020,
Pages 108–125, https://doi.org/10.1093/bmb/ldaa021
Jose A. Gallud and Ricardo Tesoriero. 2015. Smartphone Notifcations: A Study
on the Sound to Soundless Tendency. In Proceedings of the 17th International
Conference on Human-Computer Interaction with Mobile Devices and Services
Adjunct (MobileHCI ’15). Association for Computing Machinery, New York, NY,
USA, 819–824. DOI:https://doi.org/10.1145/2786567.2793706
Stavros Garzonis, Simon Jones, Tim Jay, and Eamonn O’Neill. 2009. Auditory icon
and earcon mobile service notifcations: intuitiveness, learnability, memorability
and preference. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’09). Association for Computing Machinery, New York,
NY, USA, 1513–1522. DOI:https://doi.org/10.1145/1518701.1518932
Ellen Isaacs, Alan Walendowski, and Dipti Ranganathan. 2001. Hubbub: a wire-
less instant messenger that uses earcons for awareness and for "sound instant
messages". In CHI ’01 Extended Abstracts on Human Factors in Computing Sys-
tems (CHI EA ’01). Association for Computing Machinery, New York, NY, USA,
3–4. DOI:https://doi.org/10.1145/634067.634070
David K. McGookin and Stephen A. Brewster. 2004. Understanding concur-
rent earcons: Applying auditory scene analysis principles to concurrent earcon
recognition. ACM Trans. Appl. Percept. 1, 2 (October 2004), 130–155. DOI:https:
//doi.org/10.1145/1024083.1024087
Vivian Genaro Motti. 2019. Designing emerging technologies for and with neuro-
diverse users. In Proceedings of the 37th ACM International Conference on the
Design of Communication (SIGDOC ’19). Association for Computing Machin-
ery, New York, NY, USA, Article 11, 1–10. DOI:https://doi.org/10.1145/3328020.
3353946
Listen. Listen Homepage. Retrieved May 21, 2022 from https://wearelisten.com/
Jessica Navedo, Amelia Espiritu-Santo, and Shameem Ahmed. 2019. Strength-
Based ICT Design Supporting Individuals with Autism. In The 21st International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’19).
ASSETS ’22, October 23–26, 2022, Athens, Greece
Association for Computing Machinery, New York, NY, USA, 560–562. DOI:https:
//doi.org/10.1145/3308561.3354637
[19] A. Ould Mohamed, V. Courboulay, K. Sehaba, and M. Menard. 2006. Attention
analysis in interactive software for children with autism. In Proceedings of the
8th international ACM SIGACCESS conference on Computers and accessibility
(Assets ’06). Association for Computing Machinery, New York, NY, USA, 133–140.
DOI:https://doi.org/10.1145/1168987.1169011
[20] Eleftherios Papachristos, Timothy Robert Merritt, Tobias Jacobsen, and Jimmi
Bagger. 2020. Designing Ambient Multisensory Notifcation Devices: Managing
Disruptions in the Home. In 19th International Conference on Mobile and Ubiq-
uitous Multimedia (MUM 2020). Association for Computing Machinery, New
York, NY, USA, 59–70. DOI:https://doi.org/10.1145/3428361.3428400
[21] Angela M. Puccini, Marisa Puccini, and Angela Chang. 2013. Acquiring educa-
tional access for neurodiverse learners through multisensory design principles.
In Proceedings of the 12th International Conference on Interaction Design and
Children (IDC ’13). Association for Computing Machinery, New York, NY, USA,
455–458. DOI:https://doi.org/10.1145/2485760.2485848
[22] Lauren Race, Amber James, Andrew Hayward, Kia El-Amin, Maya Gold Pat-
terson, and Theresa Mershon. 2021. Designing Sensory and Social Tools for
Neurodivergent Individuals in Social Media Environments. In The 23rd Interna-
tional ACM SIGACCESS Conference on Computers and Accessibility (ASSETS
’21). Association for Computing Machinery, New York, NY, USA, Article 61, 1–5.
DOI:https://doi.org/10.1145/3441852.3476546
[23] Amon Rapp, Federica Cena, Christopher Frauenberger, Niels Hendriks, and
Karin Slegers. 2019. Designing Mobile Technologies for Neurodiversity: Chal-
lenges and Opportunities. In Proceedings of the 21st International Conference
on Human-Computer Interaction with Mobile Devices and Services (MobileHCI
’19). Association for Computing Machinery, New York, NY, USA, Article 75, 1–5.
DOI:https://doi.org/10.1145/3338286.3344427
[24] Kathryn E. Ringland, Christine T. Wolf, LouAnne Boyd, Jamie K. Brown, Andrew
Palermo, Kimberley Lakes, and Gillian R. Hayes. 2019. DanceCraft: A Whole-
body Interactive System for Children with Autism. In The 21st International
ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’19).
Association for Computing Machinery, New York, NY, USA, 572–574. DOI:https:
//doi.org/10.1145/3308561.3354604
Lauren Race et al.
[25] Will Simm, Maria Angela Ferrario, Adrian Gradinar, Marcia Tavares Smith,
Stephen Forshaw, Ian Smith, and Jon Whittle. 2016. Anxiety and Autism: Towards
Personalized Digital Health. In Proceedings of the 2016 CHI Conference on Hu-
man Factors in Computing Systems (CHI ’16). Association for Computing Machin-
ery, New York, NY, USA, 1270–1281. DOI:https://doi.org/10.1145/2858036.2858259
[26] Katta Spiel and Kathrin Gerling. 2021. The Purpose of Play: How HCI Games
Research Fails Neurodivergent Populations. ACM Trans. Comput.-Hum. Interact.
28, 2, Article 11 (April 2021), 40 pages. DOI:https://doi.org/10.1145/3432245
[27] W3C. Making Content Usable for People with Cognitive and Learning Disabilities.
Retrieved April 27, 2022 from https://www.w3.org/TR/coga-usable/
[28] W3C. Web Content Accessibility Guidelines (WCAG) 2.1. Retrieved April 27,
2022 from https://www.w3.org/TR/WCAG21/
[29] W3C. Web Content Accessibility Guidelines (WCAG) 2.1. Success Criterion 1.3.3
Sensory Characteristics. Retrieved April 27, 2022 from https://www.w3.org/TR/
WCAG21/#sensory-characteristics
[30] W3C. Web Content Accessibility Guidelines (WCAG) 2.1. Success Criterion 1.4.2
Audio Control. Retrieved April 27, 2022 from https://www.w3.org/TR/WCAG21/
#audio-control
[31] W3C. Web Content Accessibility Guidelines (WCAG) 2.1. Success Criterion 1.4.7
Low or No Background Audio. Retrieved April 27, 2022 from https://www.w3.
org/TR/WCAG21/#low-or-no-background-audio
[32] Victoria Yaneva, Irina Temnikova, and Ruslan Mitkov. 2015. Accessible Texts for
Autism: An Eye-Tracking Study. In Proceedings of the 17th International ACM
SIGACCESS Conference on Computers & Accessibility (ASSETS ’15). Association
for Computing Machinery, New York, NY, USA, 49–57. DOI:https://doi.org/10.
1145/2700648.2809852
[33] Annuska Zolyomi, Andrew Begel, Jennifer Frances Waldern, John Tang, Michael
Barnett, Edward Cutrell, Daniel McDuf, Sean Andrist, and Meredith Ringel
Morris. 2019. Managing Stress: The Needs of Autistic Adults in Video Calling.
Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 134 (November 2019), 29
pages. DOI:https://doi.org/10.1145/3359236
[34] Annuska Zolyomi and Jaime Snyder. 2021. Social-Emotional-Sensory Design Map
for Afective Computing Informed by Neurodivergent Experiences. Proc. ACM
Hum.-Comput. Interact. 5, CSCW1, Article 77 (April 2021), 37 pages. DOI:https:
//doi.org/10.1145/3449151
 Understanding How People with Visual Impairments Take
Selfies: Experiences and Challenges
Ricardo E. Gonzalez Penuela Paul Vermette Zihan Yan
Information Science, Cornell Tech, Cornell University, Ithaca, New York, Information Science, Cornell
Cornell University, New York, New USA University, Ithaca, New York, USA
York, USA prv25@cornell.edu zihanyan@zju.edu.cn.
reg258@cornell.edu

Cheng Zhang Keith Vertanen Shiri Azenkot

Information Science, Cornell Computer Science, Michigan Jacobs Technion-Cornell Institute,
University, Ithaca, New York, USA Technological University, Houghton, Cornell Tech, Cornell University,
chengzhang@cornell.edu Michigan, USA New York, New York, USA
vertanen@mtu.edu shiri.azenkot@cornell.edu

Figure 1: Participants demonstrating how they take a selfe with their smartphone (P3, P6, P5, and P9).

ABSTRACT We contribute design guidelines that researchers and designers can

Selfes are a pervasive form of communication in social media. implement for creating accessible selfe-taking applications.
While there has been some work on systems that guide people with
visual impairments (PVI) in taking photos, nearly all has focused CCS CONCEPTS
on using the camera on the back of the device. We do not know
• Human-centered computing → Accessibility; Empirical stud-
whether and how PVI take selfes. The aim of our work is to un-
ies in accessibility; Human computer interaction (HCI); Empirical
derstand (1) PVI selfe-taking experiences and challenges, (2) what
studies in HCI.
information do PVI need when taking selfes, and (3) what modali-
ties do PVI prefer (e.g., tactile, verbal, or non-verbal audio) to sup-
port selfe-taking. To address this gap, we conducted interviews KEYWORDS
with 10 PVI. Our fndings show that current selfe-taking applica- Selfes, Accessibility, Challenges Photo-Taking, Design Guidelines,
tions do not provide enough assistance to meet the needs of PVI. Photography, PVI

Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550372
ACM Reference Format:
Ricardo E. Gonzalez Penuela, Paul Vermette, Zihan Yan, Cheng Zhang,
Keith Vertanen, and Shiri Azenkot. 2022. Understanding How People with
Visual Impairments Take Selfes: Experiences and Challenges. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
5 pages. https://doi.org/10.1145/3517428.3550372
ASSETS ’22, October 23–26, 2022, Athens, Greece
1 INTRODUCTION & RELATED WORK
Like their sighted counterparts, people with visual impairments
(PVI) take photos, share, and recall memories [2, 12, 19, 23, 26].
Moreover, for PVI, taking photos can be an important way to get
information about their environment, whether it is from remote
friends, the crowd [14-16, 20, 22] or computer vision services [18].
However, since they cannot see the photos, it is difcult for them to
capture the target and ensure a photo is aesthetic and high quality
(e.g., well-lit, and properly framed).
Researchers developed methods that provide audio and tactile
guidance to assist PVI in taking photos [1-11]. For example, Jayant
et al. developed Easy Snap [2] a mobile device application that
helped PVI capture other people in a photo by providing verbal
instructions, vibrations, and audio cues. Similarly, Cutter et al. [9]
developed a method to help PVI capture a document. Their method
provided the user precise verbal instructions specifying the direc-
tion and distance to move the device. This body of work focused
on taking photos with the back camera (the camera on the back of
the device) and ensuring that a certain target was captured. Fur-
thermore, these systems have mainly focused on whether a PVI
can take a “compliant” photo with the provided guidance (e.g.,
Whether they can capture a document, people, or a scene in an
urban area). Only a small number of researchers have considered
the task of taking selfes, which is typically more open ended and
requires more detailed information to meet the user’s needs.
Today, people frequently take selfes (i.e., photos of themselves).
They share selfes to present themselves in a desired light (partic-
ularly in social media platforms). PVI want to participate in social
media just like their sighted counterparts, taking and posting self-
ies as well as other types of photos [11, 13, 19, 21, 26], so it is im-
portant to ensure that selfe-taking is accessible.
Only one project has proposed an accessible selfe-taking ap-
plication for people with visual impairments [24]. Yungjung et
al. [24] designed SelfeHelper, an application that provided verbal
feedback about the number of faces, sizes and their locations in
the screen. Although this application addressed some challenges
in the selfe-taking process (e.g., framing the user’s head in the
photo), participants mentioned that they desired more information
(i.e., their physical appearance, or the background). Fang et al. [25]
also designed an application that provides feedback throughout
the selfe-taking process, but it was not intended for PVI. These
applications did not provide holistic support for accessible selfe-
taking—they did not provide enough information to meet PVI
needs, or the feedback was not designed to be understood by PVI.
Thus, at present, we still do not know what specifc challenges PVI
experience when taking selfes, what type of support can best as-
sist them and what information they need to take a selfe.
To address this gap, we investigated the selfe-taking experi-
ences of PVI. Our research questions were: (1) What are the current
challenges that PVI face when taking selfes? (2) What information
do PVI want when taking selfes? (3) What modalities do PVI pre-
fer (e.g., tactile, verbal, audio non-verbal cues) in a hypothetical
accessible selfe-taking application?
In this paper, we describe an interview study to address the re-
search questions and contribute design guidelines for accessible
selfe-taking applications. Building on prior work, our research
Ricardo Gonzalez Penuela et al.
takes one more step towards making photography fully accessible
to PVI.
2 METHOD
We recruited 10 participants, all of whom had a visual impairment.
Their ages ranged from 22 to 68 (mean=38, SD=14). All partici-
pants were based in the United States and were recruited through
the National Federation of the Blind mailing list. Four identifed
as totally blind, fve identifed as legally blind, and one participant
identifed as low vision. Eight were female and two were male. All
participants were iPhone users.
We conducted remote interviews that were approximately one-
hour long via videoconferencing software. We frst asked partici-
pants to recall instances where they needed to use their phone’s
front-facing camera. Next, we asked them about their selfe-taking
behavior, e.g., “which applications do you use to take selfes?”.
Then, we asked participants to demonstrate how they take a selfe
with their phone while they “thought out loud” [17]. Finally, we
asked participants to provide feedback about a set of potential fea-
tures and modes of interaction for an accessible selfe-taking appli-
cation.
During the interviews, one of the researchers took notes of sig-
nifcant moments and relevant quotes. In addition, the interviews
were videorecorded and transcribed. Two researchers coded two of
the transcripts individually through an inductive coding process.
Then, the two researchers went through the generated codes to-
gether until there was an agreement on a single set of codes. The
resulting codebook was utilized by one of the researchers to code
the rest of the transcripts.
3 FINDINGS
3.1 Selfe-Taking Experiences and Challenges
Participants described their weekly selfe-taking behavior. Five
participants took selfes infrequently (1-4 every week), three fre-
quently (5-9) and two very frequently (10+). Participants took self-
ies for various reasons: to communicate with friends (P3, P8), to
take photos with their newborn (P6), for online dating (P1, P8, P9),
to remember an occasion with their peers (P1, P5, P10), and to share
how they looked (P4, P7).
All participants except P4 sent selfes to a sighted person for
validation before sharing them. P4 had extensive experience tak-
ing selfes, and she felt that if “you do not care too much about
how [the photo] looks” then she would send the photo without
validation.
Many participants expressed desire but also frustration over tak-
ing selfes, and using frontal camera features to take augmented
selfes, like face flters (P1, P4-10). Two participants mentioned us-
ing inaccessible camera applications in social media (e.g., Snapchat
and Facebook Messenger) solely because they are forced by social
circumstances (e.g., friends only use these platforms).
P5, who had a successful business with her online blog, de-
scribed how the inability to take good selfes independently could
afect her business:
“I see when people post those types of things (selfes),
you really get a feel for who they are and you get a
feel for who they are as a person, in addition to who
Understanding How People with Visual Impairments Take Selfies: Experiences and Challenges
Table 1: Participant demographics.
Participant ID Gender
1 Male
2 Female
3 Female
4 Female
5 Female
6 Male
7 Female
8 Female
9 Female
10 Female
they are as a business owner. And I feel like I’m kind
of missing out on that because I can’t express myself
in that way”.
P9 explained how she felt when she was creating a dating profle:
“[Taking a selfe for the dating app] was stressful. I
kept thinking it is probably going to be a really
weird shot where I’m either cut of, where it is go-
ing to look like I have a physical disability.”
3.2 Information Needs and Modality
Many participants pointed out how the lack of information in the
guidance provided by the camera they typically use to take selfes
(e.g., the iPhone camera) demotivated them entirely from taking a
selfe (P1, P2, P5, P7, P8, P10).
P8 described how the feedback given by Apple’s guidance sys-
tem was not helpful: “It will say if a photo is bright or dark, but
even photos that people send me when it’s a good photo or they
say it’s a good photo, it will describe it as dark. So, I don’t really
have a good sense of what that means”.
Participants desired to have more detailed and descriptive infor-
mation about their physical appearance to share selfes confdently
(P1, P2, P6, P7, P8). P1 provided an example of the kind of informa-
tion he would like to receive: “Bob is sitting at his desk in his new
vinyl desk chair, and he is wearing a brown sweater with three but-
tons, blue jeans, and black socks.”
Six participants had a strong preference for human-like conver-
sational guidance (P1, P4, P5, P7, P9, P10). Six participants cau-
tioned that verbal information should strive to be natural and in-
tuitive (P1, P2, P4, P5, P7, P9). They were especially concerned
with verbal cues that may lead to cognitive overload (e.g., “turn
the phone 45 degrees to the right”) because (1) they would need to
process that information and (2) such instructions would be harder
to reliably follow.
Half of the participants also liked the idea of having auditory
non-verbal guidance cues (P1, P4, P7, P8, P9). Many participants
recalled some form of “hot-cold” tone guidance. This meant that
the guidance tone would change its rhythm and volume to guide
their movements. P8 provided an example: “This is [in an applica-
tion] for taking pictures of documents with the back camera to read
them. It uses tonal guidance, it is louder and steadier when the image
is clear, and softer when you are too far away or wavery when you
are not steady. It is one of my favorite ways to use a camera.”
ASSETS ’22, October 23–26, 2022, Athens, Greece
Age Visual Impairment Onset
29 Totally blind Birth
44 Totally blind 1
22 Legally blind Birth
24 Legally blind Birth
32 Legally Blind Birth
31 Legally Blind 11
52 Totally blind Birth
38 Totally blind Birth
41 Low Vision. 37
68 Legally blind Birth
None of the participants were excited about using tactile cues
to convey information. On the contrary, some participants discour-
aged the use of vibrations because they thought that it would be
confusing or uncomfortable to have the phone vibrating while tak-
ing a photo (P1, P4). P4 elaborated on their reasoning: “The only
thing I can think of that your phone can do [with tactile cues] is buzz,
and that would shake the camera.”
4 DESIGN GUIDELINES & FUTURE WORK
In prior studies, involving the back camera, researchers considered
only one criterion for taking successful photos: the target object
was captured in the photo [1–3, 9, 11]. In contrast, our fndings
showed that taking selfes was more complex, and involved three
success criteria: PVI had to capture themselves, make sure they
looked as desired, and not capture anything unwanted in the back-
ground. Based on this observation and our fndings, we present
design guidelines for accessible selfe-taking applications:
•Tailor the system guidance to the goal of the user and
support all success criteria. For example, P4 mentioned that
sometimes she wanted to take a quick photo and they did not care
how she looked, but other times when she shared the photo on
social media, she cared about how she looked and the photo qual-
ity. This is consistent with prior fndings [24]. Most participants
wanted to know how they looked (e.g., their facial expressions and
clothing). In the frst scenario the iOS camera guidance provides
enough information since it helps PVI center their face and snap
a quick photo; but in the second scenario, PVI need more informa-
tion about their physical appearance (e.g., their facial expression,
whether their eyes are closed or open, clothing detected), about
background information (e.g., disorganized table, foor detected)
and the quality of the photo (e.g., whether the photo is blurry,
whether the faces in the photo can be seen clearly).
•Guide the user through human-like conversational
prompts and descriptions. As many participants mentioned, PVI
are accustomed to being assisted by sighted peers when they take
photos. Thus, conversational prompts and descriptions of photo
content and quality would be natural and easy to follow.
•Communicate low-level guidance with real-time feed-
back using non-verbal audio cues: As mentioned by partic-
ipants, PVI are familiar with systems that leverage the pitch,
tone, or rhythm of sounds to convey a change of state in the
system or progress of the task. This, combined with human-like
ASSETS ’22, October 23–26, 2022, Athens, Greece
conversational prompts, would provide PVI with the necessary in-
formation to make small adjustments to the camera position.
•Include settings to flter information: Consistent with the
literature in other domains about accessible technology for PVI
[27], we recommend providing an easy way to turn on or of infor-
mation provided by the guidance system (e.g., turn on/of guidance
about the photo’s background or people’s facial expressions). More
generally, information should be modifable with varying degrees
of specifcity to adapt to the needs and preferences of the user.
We hope that these guidelines will be used by researchers and
designers to create accessible selfe-taking applications. In a future
study, we will develop our own accessible selfe-taking application
that PVI could use to share photos about their daily lives.
ACKNOWLEDGMENTS
This research was supported in part by the National Science Foun-
dation under awards IIS-1909248 and IIS-1909930. We thank all par-
ticipants for their time and feedback.
REFERENCES
[1] Samuel White, Hanjie Ji, and Jefrey P. Bigham. 2010. EasySnap: real-time
audio feedback for blind photography. In Adjunct proceedings of the 23nd
annual ACM symposium on User interface software and technology (UIST
’10). Association for Computing Machinery, New York, NY, USA, 409–410.
DOI:https://doi.org/10.1145/1866218.1866244
[2] Chandrika Jayant, Hanjie Ji, Samuel White, and Jefrey P. Bigham. 2011.
Supporting blind photography. In The proceedings of the 13th interna-
tional ACM SIGACCESS conference on Computers and accessibility (ASSETS
’11). Association for Computing Machinery, New York, NY, USA, 203–210.
DOI:https://doi.org/10.1145/2049536.2049573
[3] Marynel Vázquez and Aaron Steinfeld. 2012. Helping visually impaired
users properly aim a camera. In Proceedings of the 14th international
ACM SIGACCESS conference on Computers and accessibility (ASSETS
’12). Association for Computing Machinery, New York, NY, USA, 95–102.
DOI:https://doi.org/10.1145/2384916.2384934
[4] Dustin Adams, Tory Gallagher, Alexander Ambard, and Sri Kurniawan. 2013. In-
terviewing blind photographers: design insights for a smartphone application. In
Proceedings of the 15th International ACM SIGACCESS Conference on Comput-
ers and Accessibility (ASSETS ’13). Association for Computing Machinery, New
York, NY, USA, Article 54, 1–2. DOI:https://doi.org/10.1145/2513383.2513418
[5] Marynel Vázquez and Aaron Steinfeld. 2014. An Assisted Photography Frame-
work to Help Visually Impaired Users Properly Aim a Camera. ACM
Trans. Comput.-Hum. Interact. 21, 5, Article 25 (November 2014), 29 pages.
DOI:https://doi.org/10.1145/2651380
[6] Roberto Manduchi and James M. Coughlan. 2014. The last meter: blind visual
guidance to a target. In Proceedings of the SIGCHI Conference on Human Fac-
tors in Computing Systems (CHI ’14). Association for Computing Machinery,
New York, NY, USA, 3113–3122. DOI:https://doi.org/10.1145/2556288.2557328
[7] Jan Balata, Zdenek Mikovec, and Lukas Neoproud. 2015. BlindCamera: Cen-
tral and Golden-ratio Composition for Blind Photographers. In Proceedings
of the Mulitimedia, Interaction, Design and Innnovation (MIDI ’15). As-
sociation for Computing Machinery, New York, NY, USA, Article 8, 1–8.
DOI:https://doi.org/10.1145/2814464.2814472
[8] Minju Kim and Jungjin Lee. 2019. PicMe: Interactive Visual Guidance for
Taking Requested Photo Composition. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems (CHI ’19). Associ-
ation for Computing Machinery, New York, NY, USA, Paper 395, 1–12.
DOI:https://doi.org/10.1145/3290605.3300625
[9] Michael P. Cutter and Roberto Manduchi. 2015. Towards Mobile OCR:
How to Take a Good Picture of a Document Without Sight. In Proceed-
ings of the 2015 ACM Symposium on Document Engineering (DocEng
’15). Association for Computing Machinery, New York, NY, USA, 75–84.
DOI:https://doi.org/10.1145/2682571.2797066
[10] Jongho Lim, Yongjae Yoo, Hanseul Cho, and Seungmoon Choi. 2019. Touch-
Photo: Enabling Independent Picture Taking and Understanding for Visually-
Impaired Users. In 2019 International Conference on Multimodal Interaction
(ICMI ’19). Association for Computing Machinery, New York, NY, USA, 124–134.
DOI:https://doi.org/10.1145/3340555.3353728
[11] Dragan Ahmetovic, Daisuke Sato, Uran Oh, Tatsuya Ishihara, Kris Kitani, and
Chieko Asakawa. 2020. ReCog: Supporting Blind People in Recognizing Personal
Ricardo Gonzalez Penuela et al.
Objects. Proceedings of the 2020 CHI Conference on Human Factors in Comput-
ing Systems. Association for Computing Machinery, New York, NY, USA, 1–12.
DOI:https://doi.org/10.1145/3313831.3376143
[12] Cynthia L. Bennett, Jane E, Martez E. Mott, Edward Cutrell, and Meredith Ringel
Morris. 2018. How Teens with Visual Impairments Take, Edit, and Share Photos
on Social Media. In Proceedings of the 2018 CHI Conference on Human Factors
in Computing Systems (CHI ’18). Association for Computing Machinery, New
York, NY, USA, Paper 76, 1–12. DOI:https://doi.org/10.1145/3173574.3173650
[13] Dustin Adams, Sri Kurniawan, Cynthia Herrera, Veronica Kang, and Natalie
Friedman. 2016. Blind Photographers and VizSnap: A Long-Term Study. In Pro-
ceedings of the 18th International ACM SIGACCESS Conference on Comput-
ers and Accessibility (ASSETS ’16). Association for Computing Machinery, New
York, NY, USA, 201–208. DOI:https://doi.org/10.1145/2982142.2982169
[14] Jefrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller,
Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual
White, and Tom Yeh. 2010. VizWiz: nearly real-time answers to visual questions.
In Proceedings of the 23nd annual ACM symposium on User interface software
and technology (UIST ’10). Association for Computing Machinery, New York,
NY, USA, 333–342. DOI:https://doi.org/10.1145/1866029.1866080
[15] J. P. Bigham, C. Jayant, A. Miller, B. White and T. Yeh, "VizWiz::LocateIt - en-
abling blind people to locate objects in their environment," 2010 IEEE Computer
Society Conference on Computer Vision and Pattern Recognition - Workshops,
2010, pp. 65-72, doi: 10.1109/CVPRW.2010.5543821.
[16] Michele A. Burton, Erin Brady, Robin Brewer, Callie Neylan, Jefrey P.
Bigham, and Amy Hurst. 2012. Crowdsourcing subjective fashion advice us-
ing VizWiz: challenges and opportunities. In Proceedings of the 14th inter-
national ACM SIGACCESS conference on Computers and accessibility (AS-
SETS ’12). Association for Computing Machinery, New York, NY, USA, 135–142.
DOI:https://doi.org/10.1145/2384916.2384941
[17] Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data.
the MIT Press.
[18] Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2018.
A Face Recognition Application for People with Visual Impairments: Un-
derstanding Use Beyond the Lab. In Proceedings of the 2018 CHI Con-
ference on Human Factors in Computing Systems (CHI ’18). Associa-
tion for Computing Machinery, New York, NY, USA, Paper 215, 1–14.
DOI:https://doi.org/10.1145/3173574.3173789
[19] Violeta Voykinska, Shiri Azenkot, Shaomei Wu, and Gilly Leshed. 2016. How
Blind People Interact with Visual Content on Social Networking Services. In
Proceedings of the 19th ACM Conference on Computer-Supported Cooperative
Work & Social Computing (CSCW ’16). Association for Computing Machinery,
New York, NY, USA, 1584–1595. DOI:https://doi.org/10.1145/2818048.2820013
[20] Erin L. Brady, Yu Zhong, Meredith Ringel Morris, and Jefrey P. Bigham. 2013. In-
vestigating the appropriateness of social network question asking as a resource
for blind users. In Proceedings of the 2013 conference on Computer supported
cooperative work (CSCW ’13). Association for Computing Machinery, New York,
NY, USA, 1225–1236. DOI:https://doi.org/10.1145/2441776.2441915
[21] Shaomei Wu and Lada A. Adamic. 2014. Visually impaired users on an online
social network. In Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems (CHI ’14). Association for Computing Machinery, New
York, NY, USA, 3133–3142. DOI:https://doi.org/10.1145/2556288.2557415
[22] D. Gurari et al., "VizWiz Grand Challenge: Answering Visual Questions from
Blind People," 2018 IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2018, pp. 3608-3617, doi: 10.1109/CVPR.2018.00380.
[23] Susumu Harada, Daisuke Sato, Dustin W. Adams, Sri Kurniawan, Hironobu
Takagi, and Chieko Asakawa. 2013. Accessible photo album: enhancing the
photo sharing experience for people with visual impairment. In Proceedings
of the SIGCHI Conference on Human Factors in Computing Systems (CHI
’13). Association for Computing Machinery, New York, NY, USA, 2127–2136.
DOI:https://doi.org/10.1145/2470654.2481292
[24] Lee, Y., & Oh, U. (2020). SelfeHelper: Improving Selfe Experiences for People
with Visual Impairments. ᆫ ᅡᆨ
ᄒ ᅮHCI ᆨ
ᄀ ᅡᄒ
ᄒ ᅬᅩ ᆫᅮ
ᄂ ᆫᅵ ᄌ, 15(3), 23-30.
[25] Fang, N., Xie, H., & Igarashi, T. (2018). Selfe Guidance System in Good Head
Postures. IUI Workshops.
[26] Yuhang Zhao, Shaomei Wu, Lindsay Reynolds, and Shiri Azenkot. 2017. The
Efect of Computer-Generated Descriptions on Photo-Sharing Experiences of
People with Visual Impairments. Proc. ACM Hum.-Comput. Interact. 1, CSCW,
Article 121 (November 2017), 22 pages. DOI:https://doi.org/10.1145/3134756
[27] Hugo Nicolau, André Rodrigues, André Santos, Tiago Guerreiro, Kyle Mon-
tague, and João Guerreiro. 2019. The Design Space of Nonvisual Word
Completion. In The 21st International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS ’19) Association for Computing Ma-
chinery, New York, NY, USA, 249–261. DOI: https://doi-org.proxy.library.
cornell.edu/10.1145/3308561.3353786
[28] Pradhan, A., & Daniels, G. (2021). Inclusive beauty: how buying and using cos-
metics can be made more accessible for the visually impaired (VI) and blind con-
sumer. Cosmetics and Toiletries, 136(4), DM4-DM15. Association, USA, Article
109, 1929–1948.
Voice-Enabled Blockly: Usability Impressions of a Speech-driven
Block-based Programming System
Obianuju Okafor
obianujuokafor@my.unt.edu
University of North Texas
Denton, Texas, USA
ABSTRACT
Block-based programming environments pose a challenge for peo-
ple with upper-limb motor impairments. This is because they are
highly dependent on the physical manipulation of a mouse or key-
board to drag and drop elements on the screen. Our research aims to
make the block-based programming environment Blockly, accessi-
ble to users with upper limb motor impairments by adding voice as
an alternative input modality. This voice-enabled version of Blockly
will reduce the need for the use of a pointing device, thus increasing
access for people with limited dexterity. The Voice-enabled Blockly
system consists of the Blockly application, a speech recognition
API, predefned voice commands, and a custom function. A us-
ability study was conducted using a prototype of Voice-enabled
Blockly. The results revealed that people with upper-limb motor
impairments can use the system. However, it also exposed some
shortcomings of the tool and gave some suggestions on how to fx
them. Based on the fndings, changes will be made to the system,
and then, it will be evaluated in another user study in the near
future.
CCS CONCEPTS
• Human-centered computing → Accessibility design and eval-
uation methods; Empirical studies in accessibility.
KEYWORDS
Motor Impairments, Blocked-based Programming, Accessibility,
Speech Recognition, Usability Testing
ACM Reference Format:
Obianuju Okafor and Stephanie Ludi. 2022. Voice-Enabled Blockly: Usability
Impressions of a Speech-driven Block-based Programming System. In The
24th International ACM SIGACCESS Conference on Computers and Accessibil-
ity (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY,
USA, 5 pages. https://doi.org/10.1145/3517428.3550382
1 INTRODUCTION
Block-based programming environments (BBPE) such as Scratch
1 , Blockly 2 , and Pencil Code 3 are visual programming interfaces
1 https://scratch.mit.edu/
2 https://developers.google.com/blockly/
3 https://pencilcode.net/
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550382
Stephanie Ludi
stephanieludi@unt.edu
University of North Texas
Denton, Texas, USA
where programs are constructed by dragging colorful blocks to a
workspace and snapping them together. These blocks represent
programming constructs such as conditional statements, loops,
functions, variables, etc. BBPEs were created to reduce the steep
learning curve associated with learning to program using traditional
text-based programming languages [6], [2], [25]. In text-based pro-
gramming languages, to create programs, users have to memorize
the syntax of that language. Additionally, when programming with
text-based programming languages, a lot of typing is required. All
of this could be challenging for novices trying to get into program-
ming, and may discourage them.
BBPEs reduce the need for the memorization of complex syn-
tactical statements, which encourages learners to focus more on
fundamental and general programming concepts [2], [25]. Hence,
BBPEs are often used to introduce programming concepts and
computational thinking to novices [5]. Although most block-based
programming environments were built for a younger population,
studies conducted revealed that block-based programming environ-
ments are benefcial to novice programmers of all ages, ranging
from elementary school students to high school students [22] [21],
to college students [23], and even professionals [16] [3]. BBPEs
also do not require much typing; hence, they could be a suitable
platform for novices with limited dexterity.
Although block-based programming environments have indeed
proven to be advantageous for teaching programming to novices,
they also have some limitations. One drawback they have is their
dependence on the use of a mouse or keyboard to drag and drop
elements on the screen [18]. This has made them inaccessible to
people with upper limb motor impairments (ULMI) such as cerebral
palsy, muscular dystrophy, multiple sclerosis, etc. People with ULMI
often sufer from paralysis, muscle weakness, and poor coordination
[13]. This makes performing actions like dragging and dropping
objects using a pointing device quite hard [24], an action often
performed in BBPEs. This limitation excludes a signifcant portion
of people from reaping the benefts associated with the use of
BBPEs.
Our research addresses the obstacles that people with ULMI
face in the BBPE Blockly, and presents a possible solution.
In our approach, we add speech as an alternative form of input in
Blockly, to make it accessible to people with ULMI who would like
to use Blockly. Similar work exist where they attempted to make
the BBPE Scratch, accessible to people with ULMI, by creating a
standalone tool called Myna. Myna allows people with ULMI to per-
form actions in Scratch using their voice [20], [18], [19]. In contrast
to Myna, to make Blockly accessible to people with ULMI, we added
speech as an alternative input modality in the system, and not as a
separate entity. Therefore, no download or installation is needed.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Figure 1: System Overview
The rationale behind doing this is that the user experience will
be more seamless and there will be room for collaboration among
children with and without dexterity skills. Additionally, Myna was
not evaluated by enough users in their target population. Only 2
people with a ULMI participated in their study. In the user study
that we conducted with our prototype, we had 9 participants with
ULMI.
Before implementing the prototype of our system, we conducted
a preliminary study [10], [11]. The study aimed to confrm the fea-
sibility of using voice as an input modality, particularly for people
with cerebral palsy who were our target audience at the time, as
they are known to sometimes have speech impediments [4], [9].
The study also helped us test the appropriateness of our voice com-
mands. The results of the preliminary study revealed the need to
broaden the target population to include anyone with an ULMI.
Additionally, the study showed the need to modify some voice
commands.
2 VOICE-ENABLED BLOCKLY
Our system is a voice-enabled version of the Blockly application,
hence the name. Blockly’s primary input method is a mouse or
keyboard. We added speech as an alternative input modality for
people with ULMI. Our goal is not to replace the mouse or keyboard
as a form of input in Blockly, but to provide an alternative for those
who cannot use them. In this section, we present the components
of Voice-enabled Blockly and how they work together. The system
comprises four main components:
(1) Blockly Application: Blockly is a browser-based block-based
programming environment. Originally, in Blockly, actions
are performed using a mouse or keyboard. We made some
modifcations to Blockly’s source code so the same actions
can also be performed using speech. Blockly’s source code
was made available online via GitHub 4 , and therefore the
code used in this research project was downloaded from
Blockly’s GitHub repository.
(2) Speech Recognition API: Speech recognition API is a robust
pre-built library that records speech in real-time, converts
it to text, and returns the text. As with every API, it can be
added to your website or application by adding a few lines
4 https://github.com/google/blockly
Obianuju Okafor and Stephanie ludi
of code. The speech recognition API that we chose was the
Web speech API 5 . This API was used in several research
studies [12], [7].
(3) Voice Commands: The voice commands are the words ut-
tered by users when performing actions in the system. They
are limited and predefned. This helps prevent any ambiguity
associated with more verbose speech recognition systems. A
voice command consists of one or two words, e.g., "delete",
"edit feld", etc. Each voice command has an action that they
perform, e.g., selecting, deleting, etc. For example, to delete
a block from the workspace, the user will say "delete".
The design of the voice commands was infuenced by four
things. The frst one was the design of keyboard shortcuts
for Blockly, which was done in a prior work [8], the second
one was the fndings from the preliminary study that we
conducted [10],[11], the third and fourth is empathy and
domain knowledge.
Voice commands can be broken down into 5 categories:
• Navigation Commands: These are the commands used to
navigate through menus, drop-down menus, and between
a stack of blocks in the workspace. Using these commands,
a user can control and move the cursor from one point to
another. e.g. "up", "down".
• Placement Commands: These commands are used to select
blocks in the menu and place them in the workspace. They
can also be used to remove blocks from the workspace.
These commands are synonymous with dragging and
dropping blocks using a mouse or keyboard. e.g. "select",
"delete".
• Control Commands: These set of commands are respon-
sible for controlling elements in the interface, such as
opening and closing menus, e.g. "menu", "close" etc.
• Edit Commands: These commands are used to edit a block’s
text value or to add a comment to a block. It can also be
used to change the option selected from the drop-down
menu, e.g., "edit feld", "save".
• Mode Commands: These commands are used to switch
between the 3 modes in our system. The modes are nav-
igation mode, connect mode, and edit mode. e.g. "edit",
"connect", etc.
5 https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API
Voice-Enabled Blockly: Usability Impressions of a Speech-driven Block-based Programming System ASSETS ’22, October 23–26, 2022, Athens, Greece

Participant ID Age Gender Condition Overall Experience (1-5) SUS Score (0-100)
P1 40 Male Cerebral Palsy 2 70
P2 24 Male Muscular Dystrophy 4 85
P3 46 Female Muscular Dystrophy 4 55
P4 28 Female Muscular Dystrophy 4 52.5
P5 63 Female Amyotrophic Lateral Sclerosis 2 50
P6 30 Female Cerebral Palsy 4 62.5
P7 20 Female Cerebral Palsy 3 45
P8 21 Female Cerebral Palsy 4 80
P9 36 Male Cerebral Palsy 5 95
Table 1: Participant information and their respective overall experience rating, and SUS score

(4) Custom Function: To perform actions in Blockly using speech,

we created a function that translates voice commands to ac-
tions. This function entails a switch statement [17]. In the
switch statement, each case is a voice command. Each voice
command is paired with a corresponding action function
to be executed. For instance, the voice command "delete" is
paired with the block deletion function.
Fig. 1 gives an overview of the system and its components. Look-
ing at Figure 1, we can see that these components interact in the
following way. The speech recognition API receives speech input
through the device’s microphone. The audio input is processed and
converted to text. The generated text is sent to the custom func-
tion. The custom function passes the text to the switch statement
to see if there is a case that matches the text. If there is a match,
the action associated with that case is executed, which triggers a
change to Blockly’s user interface. If there is no match in the switch
statements, the message "command doesn’t exist" is displayed in
the console, and no action is performed.
3 USABILITY STUDY
We created a prototype of our application and used it to conduct
a usability study. The study aimed to investigate the following: (i)
how easy the application is to use, (ii) the faws in the system, and
(iii) what improvements can be made.
3.1 Participants
We recruited 9 participants with ULMI, to use the prototype and
give feedback. The participants comprised of 6 females and 3 males.
The age range was from 20 to 63 years old (M=34.2, SD=13.087).
Over half of the participants had cerebral palsy, 3 had muscular
dystrophy, and 1 had amyotrophic lateral sclerosis. To recruit par-
ticipants, we post our study fyer on several Facebook and Discord
groups catering to people with motor impairments. Table 1 shows
information about each participant, such as age, sex, condition,
and two quantitative metrics of the study, which are the overall
experience rating and the SUS score.
3.2 Procedure
The study was conducted remotely over Zoom. Before the study,
we deployed the prototype online using Heroku 6 . Therefore, the
participants were able to access it through their browser. When
6 https://voice-enabled-blockly.herokuapp.com/blockly/accessibleblockly/interface.html
Figure 2: Blockly program created in task 3
using our prototype, they shared their screen using the Zoom screen
share feature.
During the session, participants were given a demo, and then
six tasks were assigned to them. The frst task was a training task
to familiarize themselves with the system. The second task was a
navigation task; the participants had to navigate through the user
interface. The 3rd task was a program creation task, they were
shown a program, and they were asked to recreate that program.
Fig. 2 shows the program that was created. The fourth task was
another navigation task; they were to navigate from the last block to
the frst block of the program that was created in task 3. In the ffth
task, participants were asked to add a comment to the frst block
in the workspace. In the fnal task, they were asked to delete the
frst block from the workspace and to undo the delete action. They
had to complete all tasks with their voice. While they performed
each task, they were observed. Each session lasted an average of 60
minutes.
At the end of the study, participants had to complete a survey.
The survey had a total of 23 questions. 10 out of the 23 questions
were the System usability scale (SUS) questions. SUS questions are
a quick and efective way to measure the usability of a system [14].
After completing the survey, they were sent a $25 Amazon gift card.
3.3 Result
We present the fndings of the study in this section. We break the
results down into 3 categories, the system’s usability, its drawbacks,
and some suggestions on how to improve it.
3.3.1 System Usability. We evaluated the usability of the system
based on two quantitative metrics, the overall experience rating
given by the participants and their SUS score. We also considered
ASSETS ’22, October 23–26, 2022, Athens, Greece
open-ended responses given in the survey. The SUS score was de-
rived from the response to the 10 SUS questions in the survey. The
score was graded on a scale of 0 to 100. Similarly, the overall expe-
rience rating was taken from the response to the survey question
asking them to rate their overall experience from 1 to 5. Table 1
shows the overall experience rating given by each participant and
their SUS score.
The mean SUS score was 66.1 (SD=17.4). The maximum score
was 95, while the minimum SUS score was 45. 44.4% had an SUS
score greater than 68, which is the 50th percentile. Furthermore,
almost 80% of the participants rated their overall experience above
average. Most of the participants rated their experience as 4 and
above. The average experience rating was 3.555. The highest overall
experience rating was 5, while the lowest experience rating was 2.
Additionally, when asked what they liked most about the system, 6
out of 9 participants stated that it was easy to use. Some of their
responses include the following:
It was easy to use and understand (P2)
Easy to use, mostly responsive, very intuitive (P9)
However, 2 out of the 6 gave the caveat that the system was easy
to use once they understood the commands. They stated:
After getting used to the commands, I could see know-
ing how to use this with more ease. The commands
are a little difcult to learn at frst (P3)
Easy to use once you know the commands (P4)
3.3.2 Drawbacks. One of the questions asked in the survey was "In
your own words, what did you dislike most about the prototype?".
The reported drawbacks can be grouped into fve main themes. The
frst and most common problem was that speech recognition did
not work properly. Almost 70% of the participants reported this.
In particular, there were certain words that the system found hard
to detect, e.g., "in" and "up". The second most reported issue was
that voice commands were difcult to learn, especially in the given
time frame. 5 out of 9 people reported this. A related issue to the
previous one was that some commands were confusing and can be
mixed up, e.g., "edit" and "edit feld". One third of people reported
this. Another command-related issue was that the commands were
not intuitive enough. Only P7 reported this. The last category of
problems mentioned involves the response time of the system. Two
people stated that the system took too long to respond. Due to this,
they had to wait before performing the next action.
We discovered some problems with the system while observing
the participants as they performed the tasks. One was that the
speech recognition system was afected by poor internet. Another
issue we observed was that the colors of the blocks in diferent menu
categories were similar. Due to this, participants were sometimes
confused about which menu category to go into when trying to
locate a block.
3.3.3 Suggestions. When asked what they would like to change,
the participants gave several suggestions. Most of the suggestions
were given as a solution to the challenge faced during the study. As
a solution to the voice recognition system having difculty in rec-
ognizing certain commands. P2 and P7 suggested that commands
Obianuju Okafor and Stephanie ludi
that are difcult to detect should have alternative words. An addi-
tional suggestion given that is an extension of this strategy, was to
add a feature that allows users to modify or add commands they
want to use. Three suggestions were given about improving the
learnability and comprehensibility of commands. One of them was
to use commands that are easier to understand. Another was to
give a list of commands before the study to allow the user time to
prepare. A third was that commands that were not distinct enough
should be changed. Regarding improving the intuitiveness of the
commands, P7 gave some examples of commands that need to be
changed to be more intuitive. For example, they stated that "in"
should be replaced by "inside". To address the system’s delay in
response, P1 stated that the time lag should be reduced to allow
him perform actions faster. Although not mentioned as a challenge,
some participants suggested ways to improve navigation in the
system. One was to add number labels to the menu categories. An-
other suggestion given was to allow for the skipping of disabled
blocks when navigating up or down a menu category.
On the basis of observation, we also decided to make some
changes. We will make the color of each menu category and their
blocks very distinct. In addition, we will add a section in the user
interface that displays the voice commands and their respective
functions. This section will help reduce the cognitive load users
face when using voice to program in Voice-enabled Blockly. Finally,
we will provide a highly detailed manual that can be used to teach
people how to use the system.
3.4 Discussion and Future Work
The results are promising. The overall experience rating, the SUS
score, and the open-ended responses show that on average people
with ULMI found the system easy to use. However, the result also
reveals that there is room for improvement. According to [15],
[14],[1] the average SUS score is 68, and anything less than that is
poor. In our result only 44.4% had an SUS score above 68. We aim
to make changes so that most, if not all, of our users have an SUS
score of 68 and above.
A huge part of why the system did not have higher SUS score val-
ues could be attributed to the steep learning curve of the commands.
As mentioned above, 2 participants, P3 and P4, reported that after
understanding the commands, they found the tool easier to use.
Another issue that may have ameliorated a participant’s experience
was the dysfunctional voice recognition system. There may be a
need to change to a more robust speech recognition system that
is better at detecting speech. Although these two things afect the
application, they are things that can be improved.
In future work, we will make the necessary modifcations to
address all of these issues. Some examples of the changes we plan
to make include, add alternative words for certain commands, make
the commands more intuitive and less ambiguous, increase the
system’s response speed, add number labels to menu categories,
change the color of some menu categories, add a new section on
the user interface for voice commands, etc. We are confdent that
with these changes, the overall usability of this application, as well
as the user experience, will improve greatly. After making these
changes, we will conduct another user study. We will compare the
results of the two studies to see if there is any improvement.
Voice-Enabled Blockly: Usability Impressions of a Speech-driven Block-based Programming System
REFERENCES
[1] Hadi Alathas. 2018. How to Measure Product Usability with the
System Usability Scale (SUS) Score. Retrieved "Apr 01, 2022"
from https://uxplanet.org/how-to-measure-product-usability-with-the-system-
usability-scale-sus-score-69f3875b858f
[2] David Bau, Jef Gray, Caitlin Kelleher, Josh Sheldon, and Franklyn Turbak. 2017.
Learnable Programming: Blocks and Beyond. Commun. ACM 60, 6 (may 2017),
72–80. https://doi.org/10.1145/3015455
[3] Georgios Fesakis and Kiriaki Serafeim. 2009. Infuence of the Familiarization
with "Scratch" on Future Teachers’ Opinions and Attitudes about Programming
and ICT in Education. SIGCSE Bull. 41, 3 (jul 2009), 258–262. https://doi.org/10.
1145/1595496.1562957
[4] Centers for Disease Control and Prevention. 2021. What is Cerebral Palsy?
Retrieved "Jan 13, 2022" from https://www.cdc.gov/ncbddd/cp/facts.html
[5] N. Humble. 2019. Developing Computational Thinking Skills in K-12 Education
Through Block Programming Tools. In ICERI2019 Proceedings (Seville, Spain) (12th
annual International Conference of Education, Research and Innovation). IATED,
4865–4873. https://doi.org/10.21125/iceri.2019.1190
[6] Surendheran Kaliyaperumal. 2019. Novice programmers’ attitude towards the
introduction of block-based coding in Virtual Reality programming ABSTRACT.
(06 2019).
[7] Phoebe Lin, Jessica Van Brummelen, Galit Lukin, Randi Williams, and Cynthia
Breazeal. 2020. Zhorai: Designing a Conversational Agent for Children to Explore
Machine Learning Concepts. Proceedings of the AAAI Conference on Artifcial
Intelligence 34 (04 2020), 13381–13388. https://doi.org/10.1609/aaai.v34i09.7061
[8] Stephanie Ludi and Mary Spencer. 2017. Design Considerations to Increase
Block-based Language Accessibility for Blind Programmers Via Blockly. Journal
of Visual Languages and Sentient Systems 3 (07 2017), 119–124. https://doi.org/10.
18293/VLSS2017-013
[9] National Institute of Neurological Disorders and Stroke. 2013. Cere-
bral Palsy: Hope Through Research. Retrieved "Jan 13, 2022"
from https://www.ninds.nih.gov/Disorders/Patient-Caregiver-Education/Hope-
Through-Research/Cerebral-Palsy-Hope-Through-Research
[10] Obianuju Okafor. 2022. Helping Students with Cerebral Palsy Program via Voice-
Enabled Block-Based Programming. SIGACCESS Access. Comput. 132, Article 2
(mar 2022), 1 pages. https://doi.org/10.1145/3523265.3523267
[11] Obianuju Okafor and Ludi Stephanie. 2022. Helping Students with Upper-body
Motor Disabilities Program via Voice-Enabled Block-based Programming (Lecture
Notes in Computer Science (LNCS)). Springer, New York, NY, USA.
[12] Lucas Rosenblatt, Patrick Carrington, Kotaro Hara, and Jefrey Bigham. 2018.
Vocal Programming for People with Upper-Body Motor Impairments. W4A ’18:
Proceedings of the Internet of Accessible Things, 1–10. https://doi.org/10.1145/
3192714.3192821
[13] International Neuromodulation Society. 2012. Motor Impairment. https://www.
neuromodulation.com/motor-impairment Accessed on 2022-01-12.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[14] Will T. 2017. Measuring and Interpreting System Usability Scale (SUS) - UIUX
Trend. Retrieved "Mar 28, 2022" from https://uiuxtrend.com/measuring-system-
usability-scale-sus/
[15] Usability.gov. 2013. System Usability Scale (SUS). https://www.usability.gov/
how-to-and-tools/methods/system-usability-scale.html [Online; accessed 28-
March-2022].
[16] Leticia Azucena Vaca-Cárdenas, Francesca Bertacchini, Assunta Tavernise,
Lorella Gabriele, Antonella Valenti, Diana Elizabeth Olmedo, Pietro Pantano,
and Eleonora Bilotta. 2015. Coding with Scratch: The design of an educational
setting for Elementary pre-service teachers. In 2015 International Conference on
Interactive Collaborative Learning (ICL). 1171–1177. https://doi.org/10.1109/ICL.
2015.7318200
[17] W3Schools. 2022. JavaScript Switch Statement. https://www.w3schools.com/js/
js_switch.asp [Online; accessed 14-April-2022].
[18] Amber Wagner and Jef Gray. 2015. An Empirical Evaluation of a Vocal User
Interface for Programming by Voice. Vol. 8. 47–63. https://doi.org/10.4018/IJITSA.
2015070104
[19] Amber Wagner and Jef Gray. 2017. An Empirical Evaluation of a Vocal User
Interface for Programming by Voice. https://doi.org/10.4018/978-1-5225-1759-
7.ch012
[20] Amber Wagner, Ramaraju Rudraraju, Srinivasa Datla, Avishek Banerjee, Mandar
Sudame, and Jef Gray. 2012. Programming by voice: a hands-free approach
for motorically challenged children. (05 2012). https://doi.org/10.1145/2212776.
2223757
[21] David Weintrop. 2019. Block-Based Programming in Computer Science Education.
Commun. ACM 62, 8 (July 2019), 22–25. https://doi.org/10.1145/3341221
[22] David Weintrop and Uri Wilensky. 2015. To Block or Not to Block, That is the
Question: Students’ Perceptions of Blocks-Based Programming. In Proceedings
of the 14th International Conference on Interaction Design and Children (Boston,
Massachusetts) (IDC ’15). Association for Computing Machinery, New York, NY,
USA, 199–208. https://doi.org/10.1145/2771839.2771860
[23] Joseph B. Wiggins, Fahmid M. Fahid, Andrew Emerson, Madeline Hinckle, Andy
Smith, Kristy Elizabeth Boyer, Bradford Mott, Eric Wiebe, and James Lester. 2021.
Exploring Novice Programmers’ Hint Requests in an Intelligent Block-Based
Coding Environment. In Proceedings of the 52nd ACM Technical Symposium on
Computer Science Education (Virtual Event, USA) (SIGCSE ’21). Association for
Computing Machinery, New York, NY, USA, 52–58. https://doi.org/10.1145/
3408877.3432538
[24] James Williamson. 2017. What I’ve learned about motor impairment. http:
//simpleprimate.com/blog/motor Accessed on 2022-01-12.
[25] Zhen Xu, Albert Ritzhaupt, Fengchun Tian, and Karthikeyan Umapathy. 2019.
Block-based versus text-based programming environments on novice student
learning outcomes: a meta-analysis study. Computer Science Education 29 (01
2019), 1–28. https://doi.org/10.1080/08993408.2019.1565233
The Landscape of Accessibility Skill Set in the Sofware Industry
Positions
Lilu Martin
marti400@wwu.edu
Western Washington University
Bellingham, Washington, USA
Kristen Shinohara
kristen.shinohara@rit.edu
Rochester Institute of Technology
Rochester, New York, USA
ABSTRACT
In the software industry, good design often translates into good
user experiences. As accessibility is a key component of usable soft-
ware, there are opportunities for software professionals to include
accessibility in their skill set. However, despite a push to motivate
more companies to include accessibility as a desired knowledge and
skill set, little is known about how much companies are seeking to
recruit employees with accessibility profciency. In this paper, we
investigated the extent to which companies seek software designer
and developer skills in accessibility by analyzing software job posts
on LinkedIn. Our results showed that the majority of job posts did
not require any accessibility skills, and that educating developers
and designers about accessibility was a required qualifcation for
many of the accessibility-focused software jobs.
CCS CONCEPTS
• Human-centered computing → Accessibility technologies.
KEYWORDS
accessibility skills, job posts, software developer, designer, tester
ACM Reference Format:
Lilu Martin, Catherine M. Baker, Kristen Shinohara, and Yasmine N. Elglaly.
2022. The Landscape of Accessibility Skill Set in the Software Industry Posi-
tions. In The 24th International ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM,
New York, NY, USA, 4 pages. https://doi.org/10.1145/3517428.3550389
1 INTRODUCTION
Along the years, software companies were blamed for producing
inaccessible technology, including websites and mobile applications
[17, 28]. Several researchers studied the barriers to creating accessi-
ble software [20]. Some researchers found that empathy is necessary
for creating accessible software, and discussions and solutions were
proposed to foster empathy [8, 27]. Tools and frameworks used in
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550389
Catherine M. Baker
catherinebaker@creighton.edu
Creighton University
Omaha, Nebraska, USA
Yasmine N. Elglaly
elglaly@wwu.edu
Western Washington University
Bellingham, Washington, USA
design and development were assessed to uncover whether they
support accessibility [1, 15]. Practices of accessibility requirements
in software development were also investigated [16]. Addition-
ally, researchers identifed how accessibility can be better taught
in undergraduate education [7]. Many companies are working on
improving the accessibility of their products [21]. Large software
companies that own enough resources created positions for acces-
sibility experts to facilitate the production of accessible products
[6]. However, the landscape of accessibility jobs in software com-
panies and the desired accessibility qualifcations in software jobs
is unknown. To cover this gap in literature, we studied the accessi-
bility skill set required by software companies, based on their job
posts, in various software roles, e.g., developer, designer, and tester.
Our goal was to determine what the software industry is seeking
in new hires in terms of accessibility skills; shedding some light
for tech professionals on what they need to learn with respect to
accessibility.
By analyzing a little more than 5900 software job posts, retrieved
from LinkedIn, we found that accessibility skill set is not in demand
in any of the general software engineering roles. Accessibility best
practices, without further explanation or specifcs, was required in
a relatively small number of the general developer, designer, and
tester roles. Job posts with “accessibility” in their title required a
large and diverse set of accessibility experiences that go beyond
the technical experience to teaching and advocacy experiences. In
essence, convincing and coaching the software development team
to implement accessibility will be falling on the shoulders of those
that will be hired in accessibility roles.
2 METHOD
We chose LinkedIn from which to collect publicly available job
posts as it has the best availability for Application Programming In-
terfaces (API) to do the scraping. Also, when we compared the data
on multiple jobs websites, e.g., Google Jobs, Indeed, and LinkedIn,
we found that most jobs were duplicated across these sites. We
searched for job types that are important in the development of
accessible technology. The search terms we used were: Web De-
veloper, UX Designer, UI Designer, Software Engineer, Software
Developer, Front End Developer, UIUX Accessibility, Software Ac-
cessibility, and Accessibility Tester. The location was set to the
USA. We collected the data in May-June 2021. These search terms
were plugged into the API called Phantombuster, frst with the
ASSETS ’22, October 23–26, 2022, Athens, Greece
LinkedIn Search Export to get the results of the diferent searches,
then with the LinkedIn Job Scraper to get the actual data in each of
the scrapes. In total there were 5,920 posts, after removing dupli-
cates, grabbed of of LinkedIn using these APIs. We considered job
posts with the Job Title feld containing the word “Accessibility” to
be accessibility-specifc jobs. Two researchers manually analyzed
the Job Description feld associated with the accessibility-specifc
jobs to identify the required accessibility qualifcations. We then
searched automatically via ctrl+F for the accessibility qualifcations
in the generic software job posts to check whether generic develop-
ment and design roles require accessibility knowledge. We created
Python scripts to calculate the frequency and percentage of these
accessibility keywords.
3 RESULTS AND DISCUSSION
Our dataset is comprised of 5920 posts from 2061 diferent compa-
nies. The main industry of 1551 companies is computer software
or information technology. The other companies’ main industry
includes fnancial services, marketing, and media, among others.
While we did not flter our search by company size, and there is a
range of small companies in our dataset, most of the companies are
large ones, such as Amazon, Apple, Microsoft, GitHub, Esri, Oracle,
Samsung, Google, Twitter, and IBM, in no particular order. 1886
posts (31.85%) were accessibility-focused, and 4034 posts (68.14%)
were not. Table 1 shows a comparison of the frequency and per-
centage of a sample of accessibility-related keywords in both acces-
sibility and non-accessibility job posts. The keywords “accessibility”
occurred in 66.9% of the accessibility job posts, while occurred in
5.68% of the non accessibility posts. The keywords “WCAG” [26],
“Accessibility Guidelines”, and “Section 508”[22] occurred in 23.6%
of the accessibility job posts, while occurred in only 0.97% of the
non accessibility posts. All keywords related to assistive technol-
ogy such as “Assistive Technologies”, “Screen Readers”, “JAWS”[14],
“NVDA”[19], “VoiceOver”, “Talkback”, “Zoomtext”, “Dragon Natu-
rally Speaking”[12], and “Voice Recognition” appeared in 19.7% of
the accessibility posts, and in 0.17% of the non accessibility posts.
This data about the job qualifcations for various software pro-
fessional roles portrays the image that accessibility knowledge is
generally not expected from software engineers, developers, de-
signers, or testers.
3.1 Non Accessibility-Specifc Job Posts
With the exception of the Engineer role, there were few job posts
that required accessibility skills in roles that are not accessibility-
specifc. For the Developer role, a few posts listed understanding
web accessibility and familiarity with ARIA[4] and WCAG 2.1 as
part of the job qualifcations. Usually, accessibility was referenced
in these posts as one of the software quality criteria. For example:
“Understanding of front-end development best prac-
tices (accessibility, 508/WCAG compliance, perfor-
mance optimization, SEO).”
Similarly, a few job posts for the Designer role included accessi-
bility skills in the qualifcations, such as familiarity with WCAG.
Other posts mentioned accessibility in generic and broad terms. For
example:
Lilu Martin, Catherine M. Baker, Kristen Shinohara, and Yasmine N. Elglaly
“A solid grasp of user-centered design, usability test-
ing methodologies and accessibility best practices.”
The Tester role was the one with the most frequent occurrence
of accessibility qualifcations in non-accessibility specifc job posts.
The accessibility statements were also generic in most cases, such
as knowledge of web standards and accessibility standards, and
performing accessibility testing. In one post by a federal contractor,
accessibility testing for Section 508 compliance was specifed.
3.2 Accessibility-Specifc Job Posts
The accessibility-specifc roles in job posts had titles such as Ac-
cessibility Support Engineer for the engineer role, and Accessi-
bility Designer and Product Designer - Accessibility for the de-
signer role. The accessibility-specifc tester role had a wide range of
job titles such as Accessibility Tester, Accessibility/A11y/ADA/508
Trusted Tester, Quality Assurance (QA) Tester with Accessibility
Testing experience, Accessibility Scrum Tester, Automation Acces-
sibility Tester, and WCAG Accessibility Tester. We did not fnd any
accessibility-specifc job posts for the developer role. We grouped
the skills and qualifcations required by the accessibility-specifc
job posts into the following categories:
3.2.1 Accessibility Guidelines and Implementation. Solid under-
standing of WCAG (2.0 and 2.1 AA) was one of the most prevalent
required qualifcations across all accessibility-specifc roles. It was
rare to see posts requiring experience in WCAG Level AAA com-
pliance. Other guidelines mentioned in the job posts were Apple
accessibility Guidelines, Microsoft Accessibility Standards, Android
accessibility guidelines, and inclusive design guidelines. Several
job posts required familiarity with accessibility implementation
techniques like proper use of Accessible Rich Internet Applications
(ARIA)[4] attributes, logical order, and correct semantics. Another
common qualifcation was localization and knowledge of multiple
platform and browser accessibility practices, including iOS and
Android mobile devices, and desktop operating systems: Windows,
MacOS and ChromeOS.
3.2.2 Accessibility Regulations. The majority of job posts with ref-
erence to regulations were for the Tester role, and required knowl-
edge of Section 508 and ADA. This is expected as our data was
limited to positions in the US. In a few instances, knowledge about
more specifc accessibility regulations were required, such as Sec-
tion 255 of the Telecommunications Act, the Communications and
Video Accessibility Act (CVAA)[11], and the European accessibility
standard EN 301 549[13]. Most of the job posts discussed knowledge
of accessibility regulations in terms of compliance.
3.2.3 Assistive Technology. In Design role job posts, experience
using assistive technology and designing for assistive technolo-
gies were common qualifcations. In Tester role job posts, solid
experience testing with assistive technology and confguring test
tools and assistive devices for testing was an in-high demand skill.
Assistive technologies often cited were Zoomtext, Dragon Natu-
rally Speaking[12], screen readers (JAWS[14], NVDA[19], MACIOS
VoiceOver, TalkBack, Narrator), keyboard only (navigation, focus,
input), magnifers and voice recognition products. Several job posts
mentioned in the job description that being a native assistive tech-
nology user is preferred. A blind person, for example, that uses a
The Landscape of Accessibility Skill Set in the Sofware Industry Positions ASSETS ’22, October 23–26, 2022, Athens, Greece

General Software Roles (4034 posts) Accessibility Specifc Roles (1886 posts)
Keyword Frequency Percentage Keyword Frequency Percentage
Accessibility 229 5.68% Accessibility 1262 66.9%
WCAG 30 0.74% WCAG 208 11%
Accessibility Guidelines 4 0.09% Accessibility Guidelines 126 6.7%
Section 508 5 0.12% Section 508 111 5.9%
Localization 14 0.3% Localization 75 3.9%
JAWS 1 0.02% JAWS 87 4.6%
NVDA 1 0.02% NVDA 71 3.7%
Assistive Technologies 2 0.05% Assistive Technologies 66 3.5%
VoiceOver 1 0.02% VoiceOver 42 2.2%
Talkback 1 0.02% Talkback 31 1.6%
Zoomtext 0 0% Zoomtext 30 1.6%
Screen Readers 1 0.02% Screen Readers 26 1.37%
Voice Recognition 0 0% Voice Recognition 10 0.5%
Dragon Naturally Speaking 0 0% Dragon Naturally Speaking 9 0.47%
Magnifers 0 0% Magnifers 4 0.2%
CVAA 0 0% CVAA 1 0.05%
Table 1: The frequency and percentage of the occurrence of a sample of accessibility keywords in general and accessibility
specifc job posts.

screen reader on daily basis is considered a native screen reader
user [18]. So, this job qualifcation implies that people with
disabilities, that use assistive technology on daily basis, will
be highly considered for the job.
3.2.4 Testing Strategies and Tools. There were various testing
strategies that were listed, mostly in the Tester role job posts, such as
accessibility audit, third-party audits, test and validate accessibility
acceptance criteria, creating automation scripts for WCAG compli-
ance, testing using assistive technology, evaluating and formulating
a Voluntary Product Accessibility Template (VPAT)[23], and per-
forming usability testing with reference to accessibility. Experience
with specifc testing tools were often cited in required qualifca-
tion such as Wave[25], Axe[5], CCA[9], Accessibility Management
Platforms (AMP)[2], Accessibility Inspector, ANDI[3], Contrast
Analyzer, and Color Blindness Simulator. In at least one job post,
experience with document accessibility testing, e.g. and PDF, Excel,
was required.
3.2.5 Accessibility Certificates. Certifcation in accessibility was
preferred in a handful of job posts in the Engineer and Tester roles.
Examples are Trusted Tester, Certifed Professional in Accessibility
Core Competencies (CPACC)[10], 508 Certifed Accessibility Tester,
and Certifed IAAP Web Accessibility Specialist (WAS)[24].
3.2.6 Coaching and Educating. Many Designer and Tester job posts
added to the qualifcations list the ability to train and coach others
on various accessibility topics. These topics ranged from accessibil-
ity acceptance criteria to inclusive design principles and practices,
and WCAG. The new hires are expected to provide consultative
support to others currently working in the company, e.g., designers,
product managers, developers, and engineers.
3.2.7 Advocacy. It was interesting to see that advocacy was a fre-
quent job qualifcation for many Designer and Tester job posts.
The required qualifcations included the ability to promote the im-
portance of accessibility, experience driving organizational change
through advocacy, advocate for inclusive design methodologies,
and help developers think with an accessible mindset.
“Talent for building relationships and trust within
teams who have been resistant to accessibility in the
past.”
3.2.8 Disability Knowledge and the Disability Communities. A few
posts specifed disability knowledge as required qualifcations. This
requirement was holistic and included understanding for all as-
pects of disability, including visual, mobility, hearing, cognitive,
and speech. Several job posts required active participation in the
accessibility community, empathy for people with disabilities, or
experience working directly with people and communities of con-
sumers with disabilities.
To sum, job posts for developers, designers, and testers
rarely mention accessibility in the job qualifcations. When
accessibility is mentioned, it is usually listed as one of the soft-
ware quality criteria, e.g., performance, or in generic terms, e.g.,
experience in accessibility best practices. On the other hand,
accessibility-specifc jobs require from applicants to have ex-
tensive technical accessibility knowledge, and to function as
accessibility educators and advocates within the company.
ASSETS ’22, October 23–26, 2022, Athens, Greece
Another insight from our data is that the majority of accessi-
bility qualifcations were related to accessibility testing, and
little attention was given to accessibility in the earlier stages
of software development, e.g., requirements and design. This
points to a broader structural issue with tech job posts as accessi-
bility is seen as an afterthought rather than a way to restructure
the system to beneft a wider variety of people.
4 CONCLUSION
We analyzed position announcements and job descriptions for acces-
sibility skills to determine the extent to which industry companies
are seeking such skills. We then distinguished between general
software positions that may require skills and knowledge in acces-
sibility from accessibility specialist positions. Our results indicate
that accessibility skills are rarely sought in general software roles,
while professionals in accessibility positions were tasked with ed-
ucating and mentoring software professionals in understanding
and implementing accessibility. The expectations of the required
accessibility knowledge in general roles and accessibility roles are
unbalanced. We hope that this snapshot of accessibility demand in
software jobs motivates companies to seek employees with accessi-
bility skills in general roles, and motivate software professionals to
acquire accessibility knowledge, e.g., through training. Finally, we
hope that accessibility as a skill and a mindset will be covered in
college education, so software professionals become well-equipped
to create accessible software. We recognize that the Job Description
on LinkedIn job posts may not consistently refect the accurate
and complete information about the advertised role. In the future,
we will further investigate the process of recruiting candidates
with accessibility skills, and whether there is specialization within
accessibility, e.g., accessibility specialists in particular disabilities.
ACKNOWLEDGMENTS
This material is based upon work supported by the United States
National Science Foundation under grants #2121606, #2121428,
#2121549, and Henry Luce Foundation - Clare Boothe Luce Fund.
REFERENCES
[1] Abdullah Alsaeedi. 2020. Comparing web accessibility evaluation tools and
evaluating the accessibility of webpages: proposed frameworks. Information 11,
1 (2020), 40.
[2] AMP 2022. The Accessibility Management Platform. Retrieved June 21, 2022 from
https://www.levelaccess.com/solutions/software/amp/
[3] ANDI 2022. Accessibility Testing Tool. Retrieved June 21, 2022 from https:
//www.ssa.gov/accessibility/andi
[4] ARIA 2022. Accessible Rich Internet Applications (ARIA). Retrieved June 21, 2022
from https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA
Lilu Martin, Catherine M. Baker, Kristen Shinohara, and Yasmine N. Elglaly
[5] Axe 2022. Axe The Standard in Accessibility Testing. Retrieved June 21, 2022 from
https://www.deque.com/axe/
[6] Shiri Azenkot, Margot J Hanley, and Catherine M Baker. 2021. How Accessibility
Practitioners Promote the Creation of Accessible Products in Large Companies.
Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–27.
[7] Catherine M Baker, Yasmine N Elglaly, and Kristen Shinohara. 2020. A systematic
analysis of accessibility in computing education research. In Proceedings of the
51st ACM Technical Symposium on Computer Science Education. 107–113.
[8] Cynthia L Bennett and Daniela K Rosner. 2019. The Promise of Empathy: Design,
Disability, and Knowing the" Other". In Proceedings of the 2019 CHI conference on
human factors in computing systems. 1–13.
[9] CCA 2022. Colour Contrast Analyser. Retrieved June 21, 2022
from https://www.accessibility-developer-guide.com/setup/helper-tools/colour-
contrast-analyser
[10] CPACC 2022. Certifed Professional in Accessibility Core Competencies. Re-
trieved June 21, 2022 from https://www.accessibilityassociation.org/s/certifed-
professional
[11] CVAA 2022. The 21st Century Communication and Video Accessibility Act
(CVAA). Retrieved June 21, 2022 from https://www.fcc.gov/consumers/guides/
21st-century-communications-and-video-accessibility-act-cvaa
[12] Dragon Naturally Speaking 2022. Dragon Naturally Speaking. Retrieved June 21,
2022 from https://www.nuance.com/dragon.html
[13] EAS 2022. European Accessibility Standard EN 301 549. Retrieved June 21, 2022
from https://ec.europa.eu
[14] JAWS 2022. Job Access With Speech (JAWS). Retrieved June 21, 2022 from
https://support.freedomscientifc.com/products/blindness/jawsdocumentation
[15] Michael Longley and Yasmine N Elglaly. 2021. Accessibility Support in Web
Frameworks. In The 23rd International ACM SIGACCESS Conference on Computers
and Accessibility. 1–4.
[16] Darliane Miranda and João Araujo. 2022. Studying industry practices of accessi-
bility requirements in agile development. In Proceedings of the 37th ACM/SIGAPP
Symposium on Applied Computing. 1309–1317.
[17] M Nagaraju, Priyanka Chawla, and Ajay Rana. 2019. A practitioner’s approach to
assess the wcag 2.0 website accessibility challenges. In 2019 Amity International
Conference on Artifcial Intelligence (AICAI). IEEE, 958–966.
[18] Native AT User 2022. Accessibility Testing by People with Disabilities. Re-
trieved June 21, 2022 from https://www.24a11y.com/2019/accessibility-testing-
by-people-with-disabilities/
[19] NVDA 2022. NonVisual Desktop Access (NVDA) Screen Reader. Retrieved June 21,
2022 from https://www.nvaccess.org/
[20] Rohan Patel, Pedro Breton, Catherine M Baker, Yasmine N Elglaly, and Kristen
Shinohara. 2020. Why software is not accessible: Technology professionals’
perspectives and challenges. In Extended abstracts of the 2020 CHI conference on
human factors in computing systems. 1–9.
[21] PEAT Report 2018. The Accessible Technology Skills Gap. The Partnership on
Employment & Accessible Technology (PEAT), report. Retrieved June 21, 2022 from
https://www.peatworks.org/infographic-the-accessible-technology-skills-gap/
[22] Section 508 2022. Section 508 of the Rehabilitation Act. Retrieved June 21, 2022
from https://www.section508.gov/
[23] VPAT 2022. Voluntary Product Accessibility Template (VPAT). Retrieved June 21,
2022 from https://www.section508.gov/sell/vpat/
[24] WAS 2022. Web Accessibility Specialist. Retrieved June 21, 2022 from https:
//www.accessibilityassociation.org/s/wascertifcation
[25] Wave 2022. WAVE Web Accessibility Evaluation Tool. Retrieved June 21, 2022
from https://wave.webaim.org/
[26] WCAG2.1 2018. Web Content Accessibility Guidelines (WCAG) 2.1. Retrieved
June 21, 2022 from https://www.w3.org/TR/WCAG21/
[27] Peter Wright and John McCarthy. 2008. Empathy and experience in HCI. In
Proceedings of the SIGCHI conference on human factors in computing systems.
637–646.
[28] Shunguo Yan and PG Ramachandran. 2019. The current status of accessibility in
mobile apps. ACM Transactions on Accessible Computing (TACCESS) 12, 1 (2019),
1–31.
 How people who are deaf, Deaf, and hard of hearing use
technology in creative sound activities
Keita Ohshiro
ko89@njit.edu
New Jersey Institute of Technology
Newark, New Jersey, USA
ABSTRACT
Creative sound activities, such as music playing and audio engi-
neering, are said to have been democratized with the development
of technology. Yet, the use of technology in creative sound activities
by people who are deaf, Deaf, and hard of hearing (DHH) has been
underexplored by the research community. To address this gap, we
conducted an online survey with 50 DHH participants to under-
stand their use of technology and barriers they face in their creative
sound activities. We fnd DHH people use four types of technology
— hearing devices, sound manipulation, sound visualization, and
speech-to-text — for three purposes — to improve sound perception
via auditory and visual means, to avoid hearing fatigue, and to bet-
ter communicate with hearing people. We also fnd their barriers
to technology: unknown availability, limited options, and limita-
tions that technology can solve. We discuss opportunities for more
inclusive design specifc to DHH people’s creative sound activities,
as well as facilitating access to information about technology.
CCS CONCEPTS
• Human-centered computing → Empirical studies in acces-
sibility.
KEYWORDS
accessibility, deaf, Deaf, hard of hearing, audio engineering, music
ACM Reference Format:
Keita Ohshiro and Mark Cartwright. 2022. How people who are deaf, Deaf,
and hard of hearing use technology in creative sound activities. In The 24th
International ACM SIGACCESS Conference on Computers and Accessibility
(ASSETS ’22), October 23–26, 2022, Athens, Greece. ACM, New York, NY, USA,
4 pages. https://doi.org/10.1145/3517428.3550396
1 INTRODUCTION
It is said that creative sound activities, such as music playing and au-
dio engineering, have been democratized with the development of
technology [13, 25]. Nowadays, these activities are not only limited
to professionals working with high-end studio equipment but also
opened to amateurs and novices using personal computers and au-
dio production software called Digital Audio Workstations (DAWs)
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proft or commercial advantage and that copies bear this notice and the full citation
on the frst page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
ASSETS ’22, October 23–26, 2022, Athens, Greece
© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9258-7/22/10.
https://doi.org/10.1145/3517428.3550396
Mark Cartwright
mark.cartwright@njit.edu
New Jersey Institute of Technology
Newark, New Jersey, USA
[29]. Those who engage in creative sound activities enjoy perform-
ing, producing, and distributing their works in audio-related felds
including music, flm, TV, radio, and podcasting [23].
However, they have not been democratized for all in the context
of accessibility and disability. With a growing interest in the acces-
sibility of audio and sound technologies [17], we have begun to see
research on creative sound activities by people with disabilities, for
example, by people who are blind or have low vision [42, 44]. Yet,
the research community has underexplored how people who are
deaf, Deaf, and hard of hearing (DHH) [8, 26] use technology in
creative sound activities. In 2021, the World Health Organization
reported that more than 1.5 billion people (1 in 5 people) worldwide
are afected by some degree of DHH, and they estimated that this
number will increase to nearly 2.5 billion people (1 in 4 people) by
2050 [40]. Given this increasing DHH population and that creative
sound activities are often primarily auditory, we feel it is important
to develop an understanding of the current state of accessibility by
DHH people in creative sound activities.
In this paper, we aim to understand how DHH people use tech-
nology in their creative sound activities and what barriers they
may face. We present our fndings from an online survey with 50
responses by DHH people who engage in creative sound activi-
ties. We conclude by discussing future research directions to make
technology more available and inclusive of DHH people.
2 RELATED WORK
DHH people often have difculty perceiving sound characteristics
such as pitch, loudness, timbre, and spatial information [15, 16, 48].
Some use medical hearing devices such as hearing aids (HA) or
cochlear implants (CI) to improve auditory perception, yet they
can cause additional challenges such as worsening pitch perception
or increasing noises [9, 14, 27, 30–32, 43]. Issues such as tinnitus,
hearing fatigue, and hearing fuctuation can also afect their quality
of hearing [24, 47]. To overcome these limitations of their auditory
system, DHH people often rely more on other senses such as vision
and touch to experience sound [46].
Prior work has ofered understanding of DHH people’s expe-
riences in creative sound activities. DHH musicians utilize visual
and physical cues as well as music theory to develop musical self-
efciency [18]. Social and cultural factors afect how DHH people
approach their creative sound activities as being DHH [11]. Stories
by DHH individuals show their unique experience in creative sound
activities as being DHH [2, 10, 15, 19, 22, 48]. For example, Richard
Einhorn shared how he was able to continue his professional audio
engineer career after becoming deaf later in his life [15]. There also
are organizations that support DHH people’s creative sound activ-
ities [1, 34–37, 39]. For example, the Frequalise Project by Music
ASSETS ’22, October 23–26, 2022, Athens, Greece
and the Deaf [34] demonstrated the efectiveness of technology to
provide positive music learning experience to 63 young DHH peo-
ple through 26 sessions of workshops [33]. In addition, research on
Accessible Digital Music Instruments (ADMI) has been exploring
the design and evaluation of how technology can be used for DHH
people to play music and collaborate with other DHH and hearing
people [6, 17], such as Music Aid [45] and Felt Sound [7].
While this related work gives an idea of how DHH people are
using technology in creative sound activities, it remains partial
and sparse because it consists mainly of individual stories, the use
of technology is not the primary focus, and its scope is limited to
music-specifc activities. Focusing on technology with a broader
scope, our work extends past work to understand the current state
of accessibility in creative sound activities by DHH people.
3 METHODS
We conducted an online survey targeting DHH people who en-
gage in creative sound activities. To understand their experience
with technology, we asked about their use of technology to make
their activities accessible and the challenges they face in using and
choosing the technology. Using snowball sampling, we recruited
survey participants via individuals and online communities related
to DHH, accessibility, music, and audio engineering. All partici-
pants were fuent in English and over 18 years old. The survey
took approximately 20 minutes. Participation was voluntary and
not compensated. The study received the New Jersey Institute of
Technology IRB approval.
50 participants completed the survey (Table 1). The degree of
hearing loss [12] of the participants varies from mild to profound,
and the time of becoming DHH varies from birth to adulthood.
Their creative sound activities largely consist of music playing and
audio engineering (Table 2). Music playing includes singing and
playing musical instruments. Audio engineering includes recording,
editing, mixing, mastering for music, radio, flm, podcast, as well
as transcribing and captioning. We analyzed the survey data using
a thematic analysis approach [4, 5, 28, 38].
4 FINDINGS
4.1 Use of Technology
We found participants use four types of technology — hearing de-
vices, sound manipulation, sound visualization, and speech-to-text
— and for three purposes — to better perceive sound through audi-
tory and visual means, to avoid hearing fatigue, and to communicate
with others and understand speech.
The most commonly reported technologies are hearing devices.
Many use one or a combination of medical and non-medical hear-
ing devices such as HA, Bone-Anchored HA [3], CI, headphones,
earbuds, and speakers. They use hearing devices primarily to im-
prove auditory perception. To hear more clearly, some use them
together with Bluetooth, noise cancelling features, and neck loops
(a loop of wire worn around the neck that transmits audio signals
to hearing devices). However, the use of hearing devices can also
induce hearing fatigue and physical and mental overload. Thus,
some people choose not to wear HA and CI all the time because they
become overwhelmed by the unwanted efects such as distortion
and mufing, especially when listening to music rather than speech.
Keita Ohshiro and Mark Cartwright
For example, P12 sometimes switches from HA to bone conduction
headphones to rest his ear.
Participants also commonly use sound manipulation technology
to improve auditory perception. Some use equalization to boost or
cut certain frequencies. P27, an advanced amateur musician, octave-
shifts the pitch with an octave pedal when writing songs with his
electric guitar and bass. To hear sound from their non-DHH side ear,
many of the single-sided DHH participants use technology such as
signal routing and mono-to-stereo conversion. Single-sided DHH
audio engineers also use stereo fipping (swapping the left and right
channels) to check stereo-feld panning and phase cancellation.
Participants use visualizations both to better perceive sound and
to avoid hearing fatigue. For example, participants who engage in
audio engineering commonly reported using waveform (i.e., time-
amplitude) and a few also use spectrum (i.e., frequency-amplitude)
and spectrogram (i.e., time-frequency) displays. Such visualizations
allow them not only to supplement their auditory perception, but
also to edit sound with their eyes so that they can rest their ears. P29,
a professional audio engineer for podcasts, said “I can sometimes
do edits with no headphones and a script, because I can edit sound
by sight.” For music playing, some use a tuner and metronome to
visually ensure that pitch and tempo were correct. Also, P34 uses a
piano roll to visually see the notes and P12 uses sound waves as a
metronome to see the click sound when it was difcult to hear.
With one exception, all participants who practice audio engi-
neering reported the use of DAWs. We assume participants who
reported the use of DAWs are also familiar with some use of the
technology mentioned above such as equalization and waveform
/ spectrum / spectrogram displays, as they are basic features in
DAWs. In contrast, the use of DAWs was typically not reported by
participants who only play musical instruments or sing but do not
practice audio engineering.
Some participants use speech-to-text technology for communi-
cation with hearing people during their activities, such as Google’s
Live Transcribe [21] and Otter.ai [41]. At online meetings, they also
use an auto caption feature in video conference software such as
Zoom [49] and Google Hangouts [20].
No participants use the technology specialized for tactile feed-
back. However, P42, an electric bass player who is Deaf, reported
the usefulness of tactile feedback with his instrument and amplifer.
He said “a fve strings bass helps me feel the pulsating bass line to the
backbeat... Musical amplifers are the best instrument to feel the beat
for me.”
4.2 Barriers
While some participants reported their use of technology, others
described barriers to technology such as unknown availability, lim-
ited options, and limited solutions that it can provide. In fact, when
we asked about the technologies that make their activities more ac-
cessible, more than one-third of participants (n=19) did not specify
any technologies.
Some participants reported not knowing if technology that would
make their activity accessible is available in the frst place. P8, an
intermediate amateur musician, expressed his frustration saying
“Don’t even know what’s available. Ignorance.” They feel difculty
in fnding the right technology for their hearing and activities. P4,
How people who are deaf, Deaf, and hard of hearing use technology in creative sound activities ASSETS ’22, October 23–26, 2022, Athens, Greece

Table 1: Participant Demographics. (Numbers in parentheses are the number of participants)

Gender Male (26), female (19), non-binary/non-conforming (3), transgender male (2)
Age 18-24 years old (6), 25-34 (18), 35-44 (8), 45-54 (6), 55-64 (6), 65-74 (4), 75-84 (1), 85+ (1)
DHH identity Hard of hearing (25 including three single-sided), deaf (18 including six single-sided), Deaf (4), others (3)

Table 2: Participants’ Activity and Level of Experience. (beg = beginner; int = intermediate; adv = advanced)

Activity Amateur beg Amateur int Amateur adv Professional N/A Total
Music playing 4 12 8 14 2 40
Audio engineering 1 4 7 12 1 25

a professional musician and audio engineer, said “I am not aware of
any technology/tools developed specifcally for my problem.” Whereas,
P16 expressed reluctance to try using technology: “Since I can’t hear
well, I haven’t even tried using most technology from a musical per-
spective.”
Even when the challenge seems solvable or improvable with tech-
nology, the limited variety of products do not solve the challenges
unique to each individual. P22, an advanced amateur musician and
audio engineer, expressed a sense of resignation by saying “the
idea... is so niche that it’s not something product designers tend to
think about or cater for.” Some wish to have more inclusive products.
A few use DIY solutions specifc to their hearing and activity. How-
ever, that is not a feasible solution for others who are unfamiliar
with technology: “Software is still a mystery to me.” (P6).
Participants reported limited solutions that current technology
may provide for certain tasks. P9, who became deaf after estab-
lishing his career as a professional audio engineer, said “I can’t
master as well as I used to and no tool will fx that.” P30, who started
audio engineering for podcasts after becoming deaf, said “There
are defnitely times when I just cannot do a specifc task — creating
certain kinds of sound efects or soundscapes, or I don’t always catch
if speech isn’t as clear as it should be or if it fully ‘matches’ the audio
from other actors, etc. It’s simply not a thing I can always do.”
With these barriers, technology still cannot ease participants’
lack of confdence. P2 said “Most of my challenges come from lack
of confdence that what I am hearing is accurate.” It also created a
concern for their professional career. P48, who works as a Digital
Signal Processing (DSP) engineer, said “I feel I am held back when
learning advanced DSP and flter design since I can’t properly hear...
(I) don’t feel like I can advance super far in that feld.” To deal with
the lack of confdence, many mentioned seeking hearing people’s
feedback as a workaround. P27 said “I must get a second opinion.
Must.” However, the availability of hearing people is a challenge.
They need to be familiar with creative sound activities to provide
valid feedback. Scheduling them is also not easy. Otherwise, partic-
ipants had to rely on guessing. P48 said “I’m always second guessing
my ears”. P16 said “(I’m) trusting it sounds ok... I have no idea how I
actually sound.”
5 DISCUSSION
Our study found participants’ current use of technology and the bar-
riers they face in creative sound activities. Aside from HA/CI, most
of the technology was not specifcally designed for their specifc
DHH situations and activities. For future research, we recommend
researchers to work closely with DHH users to identify specifc
challenges in their activity and design inclusive technologies. We
also see an opportunity for research that facilitates community
where both hearing and DHH people who engage in creative sound
activities can exchange knowledge and information about technol-
ogy. That would help raise awareness of available technologies and
how to use them efectively.
6 CONCLUSION
In this work, we provide early insight into DHH people’s use of
technology and its barriers in creative sound activities. To support
these activities, we found DHH people use four types of technology
— hearing devices, sound manipulation, sound visualization, and
speech-to-text — and for three purposes — to better perceive sound
through auditory and visual means, to avoid hearing fatigue, and to
communicate with others and understand speech. We found DHH
people also have barriers to technology in terms of its availability,
limited options, and limitations that technology can solve. In future
work, we plan on conducting interviews that focus on more specifc
tasks of creative sound activities that DHH people fnd challenging.
ACKNOWLEDGMENTS
We thank all participants, organizations, and communities that
contributed to this study.
REFERENCES
[1] Audiovisability. 2022. Audiovisability. Retrieved June 23, 2022 from https:
//www.audiovisability.com/
[2] Swann Barrat. 2020. I’m a sound technician. Losing my hearing
was devastating. CBC News (24 Nov. 2020). Retrieved June 23,
2022 from https://www.cbc.ca/news/canada/british-columbia/i-m-a-sound-
technician-losing-my-hearing-was-devastating-1.5813327
[3] Ricardo Ferreira Bento, Alessandra Kiesewetter, Liliane Satomi Ikari, and Rubens
Brito. 2012. Bone-anchored hearing aid (BAHA): indications, functional results,
and comparison with reconstructive surgery of the ear. International archives of
otorhinolaryngology 16, 03 (2012), 400–405.
[4] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qualitative research in psychology 3, 2 (2006), 77–101.
[5] Virginia Braun and Victoria Clarke. 2013. Successful qualitative research: A
practical guide for beginners. sage.
[6] Doga Cavdir. 2022. Touch, Listen,(Re) Act: Co-designing Vibrotactile Wearable
Instruments for Deaf and Hard of Hearing. In NIME 2022. PubPub.
[7] Doga Cavdir and Ge Wang. 2020. Felt sound: A shared musical experience for
the deaf and hard of hearing. In Proceedings of the 20th international conference
on new interfaces for musical expression (nime-20).
[8] Anna Cavender and Richard E Ladner. 2008. Hearing impairments. In Web
accessibility. Springer, 25–35.
ASSETS ’22, October 23–26, 2022, Athens, Greece
[9] Marshall Chasin and Frank A Russo. 2004. Hearing aids and music. Trends in
Amplifcation 8, 2 (2004), 35–47.
[10] Wendy Cheng and Au.D. Willia Horowitz. 2021. Making Music with a Hearing
Loss: Strategies and Stories (2nd ed.). AAMHL Publications, Gaithersburg, MD,
USA.
[11] Warren N Churchill. 2016. Claiming musical spaces: Stories of deaf and hard-of-
hearing musicians. Ph. D. Dissertation. Teachers College, Columbia University.
[12] John G Clark. 1981. Uses and abuses of hearing loss classifcation. Asha 23, 7
(1981), 493–500.
[13] Bill Crow. 2006. Musical creativity and the new technology. Music Education
Research 8, 1 (2006), 121–130.
[14] Harvey Dillon. 2008. Hearing aids. Hodder Arnold.
[15] Richard Einhorn. 2012. Observations from a musician with hearing loss. Trends
in Amplifcation 16, 3 (2012), 179–182.
[16] Leah Findlater, Bonnie Chinh, Dhruv Jain, Jon Froehlich, Raja Kushalnagar, and
Angela Carey Lin. 2019. Deaf and hard-of-hearing individuals’ preferences for
wearable and mobile sound awareness technologies. In Proceedings of the 2019
CHI Conference on Human Factors in Computing Systems. 1–13.
[17] Emma Frid. 2019. Accessible digital musical instruments—a review of musical
interfaces in inclusive music practice. Multimodal Technologies and Interaction 3,
3 (2019), 57.
[18] Robert Fulford, Jane Ginsborg, and Juliet Goldbart. 2011. Learning not to listen:
the experiences of musicians with hearing impairments. Music Education Research
13, 4 (2011), 447–464.
Keita Ohshiro and Mark Cartwright
[30] Valerie Looi, Hugh McDermott, Colette McKay, and Louise Hickson. 2008. The
efect of cochlear implantation on music perception by adults with usable pre-
operative acoustic hearing. International journal of audiology 47, 5 (2008), 257–
268.
[31] Sara MK Madsen and Brian CJ Moore. 2014. Music and hearing aids. Trends in
Hearing 18 (2014), 2331216514558271.
[32] Hugh J McDermott. 2004. Music perception with cochlear implants: a review.
Trends in amplifcation 8, 2 (2004), 49–82.
[33] Music and the deaf. 2016. The Frequalise Report: A project by Music and the
Deaf. Retrieved June 23, 2022 from https://network.youthmusic.org.uk/fle/
27500/download?token=4_rxCK37
[34] Music and the Deaf. 2022. Music and the Deaf | West Yorkshire | MatD. Retrieved
June 23, 2022 from https://www.matd.org.uk/
[35] Drake Music. 2022. Drake Music | Leaders in Music, Disability & Technology.
Retrieved June 23, 2022 from https://www.drakemusic.org/
[36] Youth Music. 2022. Youth Music Home Page. Retrieved June 23, 2022 from
https://youthmusic.org.uk/
[37] Deaf Professional Arts Network. 2022. D-PAN: Deaf Professional Artist Network.
Retrieved June 23, 2022 from https://d-pan.org/
[38] Kimberly A Neuendorf. 2018. Content analysis and thematic analysis. In Advanced
research methods for applied psychology. Routledge, 211–223.
[39] Association of Adult Musicians with Hearing Loss. 2022. Welcome! - Association
of Adult Musicians with Hearing Loss. Retrieved June 23, 2022 from https://www.