Quantitative and Qualitative Features

of Co-Speech Gesture in Verbally

Julia Leung, Zachary Dravis, Armen Bagdasarov, Ambar Muñoz Lavanderos, Brittney Mahoney, Leslie Adeoye, Amanda Riiff,
Fluent Individuals with ASD Alison Pomykacz, Katherine Bassanello, Aubrey Faggen, Samantha Plate, Meredith Cola, Victoria Petrulla, Leila Bateman, Juhi
Pandey, Julia Parish-Morris, Casey Zampella, Elizabeth Kim, Robert Schultz, and Ashley de Marchena

Introduction Methodology Participants Discussion

• Qualitative features of gesture, such as certainty
• Non-verbal hand gestures augment • Participants with IQ scores below 70, or who did Table 1 TDC (n=34) ASD (n=45) p-value and confidence, may be more robust markers of
social communication with a visual not use any representational gestures were ASD than quantitative features.
Age (years) 11.0 (3.0); 5.8 – 16.6 11.8 (2.6); 8.3 – 17.9 .215
accompaniment excluded (1 control and 6 ASD participants)
Full scale IQ 111.1 (10.0); 97 – 136 106.2 (18.8); 73 – 137 .871 • Surprisingly, participants with ASD added more
• Autism Spectrum Disorder (ASD) • Groups were matched on age, gender, and IQ
SRS-2 new information with the gestures they used.
• Participants completed two communication tasks
affects certain neuropsychomotor
47.0 (6.5); 38 – 65 69.5 (10.0); 51 – 91 <.001
Total t-Score • Quantitative aspects of gesture may be an
to elicit co-speech gesture: Box and Cartoon: effective strategy that youth with ASD have
functions that may underpin motor ADOS-2 CSS 1.29 (0.6); 1 – 3 7.2 (2.2); 1 – 10 <.001
identified to enhance communication
and gesture skills (Paquet et al., 2019) Note: Data presented as Mean (SD); Range; • Quantitative aspects of gesture may also
• Studies have suggested that toddlers BOX SRS-2 = Social Responsiveness Scale, Second Edition;
ADOS CSS = Autism Diagnostic Observation Scale, Second Edition, Calibrated
reflect gesture’s beneficial effects for the
speaker’s own cognitive processes
with ASD may have difficulty in Participants played Severity Score • Quantitative features may be less effective
gesture production and language with a shape-sorter when their communicative quality is lower
(Talbott et al., 2018) toy for 15 to 30
• Research on co-speech gestures in seconds, then
described how it
Results • The ASD group had lower qualitative ratings,
suggesting that their gestures, while information
verbally fluent individuals with ASD is Add Author Names and Information rich, were more difficult to interpret.
equivocal when it comes to how often worked. Table 2 TDC (n=34) ASD (n=45) p-value

gesture is used to supplement speech Include University or Department

New Info Names
0.2 (0.2); 0 – 0.8 if 0.4 (0.3); 0 – 1.0 .005 • Given the dynamic, intuitive, and subjective
nature of live social communication, qualitative
across a discourse (de Marchena & Needed Two-
CARTOON
0.5 (0.2); 0 – 0.9 0.5 (0.2); 0 – 0.9 .346 features may better capture these subtleties. The
handed
Eigsti, 2010; Choi et al., 2019) Certainty 1.8 (0.2); 1.4 – 2.0 1.7 (0.2); 1.3 – 2.0 .107 disconnect between quantitative and qualitative
• Here, we take a look at the individual Participants features of gesture may also reflect aspects of
Confidence 1.8 (0.2); 1.4 – 2.0 1.6 (0.2); 1.2 – 2.0 .003 gesture such as motor performance or timing,
contribution of each gesture watched a clip of a
Note: Data presented as Mean (SD); Range which could be accessible using automated
• We propose that each gesture can be Looney Tunes approaches.
analyzed in at least 2 ways: cartoon, then
described what
0.45

TDC ASD • Both quantitative and qualitative ratings were

happened. conducted exclusively by neurotypical, college-
0.35

QUANTITATIVELY QUALITATIVELY 0.25

aged students who are unfamiliar with the
0.15
* participants; these findings may not be illustrative
Look at how much Look at how well the of interpretations made by other interlocutors,

Mean Score
• •
Communication tasks were video-recorded
*

content the gesture gesture carries out its • 0.05

such as other individuals with ASD, or people

contributes to the intended purpose and coded for new information, two- -0.05
familiar with the participants.
interaction • More subtle, handedness, certainty, and confidence -0.15

• Can be quantified subjective measures

• Coders were neurotypical undergraduate
References
-0.25

objectively • Includes certainty and

• Include new confidence college students -0.35

information and two-

Coding Scheme
-0.45
Cassell, J., McNeill, D., & McCullough, K.-E. (1998). Speech-gesture mismatches: Evidence for one underlying
Quantitative Qualitative
handedness representation of linguistic and nonlinguistic information. Pragmatics and Cognition, 6, 1–24. doi:10.1075/pc.7.1.03cas
Figure: Quantitative vs. qualitative measures of gesture, by group.
Note: Quantitative features collapsed across (1) how often a gesture added new information to the discourse and (2) Choi, B., Shah, P., Rowe, M. L., Nelson, C. A., & Tager-Flusberg, H. (2019). Gesture development, caregiver

QUANTITATIVE QUALITATIVE how often gestures were executed using two hands. Qualitative features collapsed across (1) raters’ certainty about
responsiveness, and language and diagnostic outcomes in infants at high and low risk for autism. Journal of Autism and
Developmental Disorders. https://doi.org/10.1007/s10803-019-03980-8

how much content the how well the gesture is

where a movement was a gesture, and (2) raters’ confidence in the meaning of the gesture.
Error bars represent standard error. de Marchena, A., & Eigsti, I.-M. (2010). Conversational gestures in autism spectrum disorders: Asynchrony but not
gesture contributes executed decreased frequency. Autism Research, 3(6), 311–322. https://doi.org/10.1002/aur.159

Hypothesis Analysis
Paquet, A., Olliac, B., Golse, B., & Vaivre-Douret, L. (2019). Nature of motor impairments in autism spectrum disorder: A
comparison with developmental coordination disorder. Journal of Clinical and Experimental Neuropsychology, 41(1), 1–

New Information Certainty • All variables were z-score transformed

14. https://doi.org/10.1080/13803395.2018.1483486

Qualitative features of gesture Does this gesture add new information

that was not included in speech? (Cassell et
How clear is it that the hand movement is
a meaningful gesture rather than a fidget
• New information and two-handedness were collapsed as
Talbott, M. R., Young, G. S., Munson, J., Estes, A., Vismara, L. A., & Rogers, S. J. (2018). The developmental sequence and
relations between gesture and spoken language in toddlers with autism spectrum disorder. Child Development.

quantitative features
https://doi.org/10.1111/cdev.13203

are more pronounced markers al. 1998; note: representational gestures only)
• 0 = No new information
or other movement?
• 0 = Unclear • Certainty and confidence were collapsed as qualitative features
of ASD than quantitative
• 1 = New information added • 1 = Somewhat clear
• 2 = Clear • A mixed ANOVA yielded a significant Group*Measure Acknowledgements
interaction (p=.002) Many thanks to our participants, as well as the large support team involved in
features. Two-handedness
Was this gesture executed with one or two
Confidence
How well can you interpret the intended
• Post-hoc t-tests (Table 2) showed that data collection and coding. This project was supported by NIH National Award #
T32NS007413 National Institute of Neurological Disorders and Stroke (NINDS);
hands? meaning of the gesture? • Out of the quantitative features, the ASD group use a Maslow and Zolot Family Fund for Technology and Innovation; the Lurie Family
• Live social interaction is dynamic • 0 = One-handed • 0 = Cannot interpret
significantly higher proportion of representational gestures
Foundation; the Allerton Foundation, and NICHD Intellectual and Developmental
• 1 = Two-handed Disabilities Research Center (IDDRC) U54 to Schultz (U54HD086984; Sub Project
• Qualitative, subjective ratings made by human coders • 1 = Somewhat able to interpret
may better capture the nuances of non-verbal
Because two-handed gestures are more • 2 = Can interpret confidently that included new information compared to TDC Award: 8065). The content is the sole responsibility of the authors and does not
visually expansive, they may provide more necessarily represent the official views of these funding agencies.
communication content to the discourse. • Out of the qualitative features, confidence had significantly
lower ratings in the ASD group compared to TDC For further information about this study, please contact
Julia Leung at jleung2015@mail.usciences.edu