Joshua Gulab December 10, 2021

Essay on Auditory Data Representation

By the end of January 2018 the American market for smart speakers had almost doubled with
about one in every six persons owning one. Despite the rise in audio-first interactions and virtual
assistants there was no current standard in a design for sound experiences that were able to
covert datasets by ubiquitous computing particularly in VUIs (Voice user interface). Thus a need
to create new ways of representing data into auditory form to combat the rise of audio first VUIs
with data quantifying. This problem has plagued us for some time with initiatives such as
WebAim long pushing to make websites widespread through screen reader. Roger Molina and
Brian Foo in recent years aimed to develop innovative ways to explore data sonification as art.
But the real issue is how shall we learn to listen to data patterns as we have through the visual
world.
The research was based on the simple question of “How has audio successfully been used to
express data?” Search for literature was done with basic words like musical graph and auditory
data with prominent authors like Bruce Walker and John Flower as well as “The Sonification
Handbook” by an International Conference on Auditory Display (ICAD). It was noted that
extensive exploration and research had been done to sonify data and thus the report became a
study of “What types of visual graphs were sonified and how did they perform.”
The results were as follows;
Firstly, Distributions: Histogram and Box Plot; Flowers and Hauer's explored the capability of
auditory histograms and box plots to effectively communicate central tendency, variability and
shape. They found that these audio histograms consisted of strings of musical 17 notes in which
numeric values of observations were presented by musical pitch, and the frequency of scores in
each interval or bin was represented by the number of repetitions of a given note in the display.
The authors did however imply that both representations may be a useful alternative to
traditional visual graphics.
Secondly, Spatial Relationships: Choropleth Map and Pie Chart; Zhao et aleffort's aims to make
geo-referenced statistics data more accessible to visually impaired people. In his study, each state
was assigned one of five categorical values, each represented by a 200-millisecond value pitch.
Three continuous bells signal the completion of the sequence, and the "gist," or summary of all
stages, lasts roughly 25 seconds. While the authors concluded that study participants correctly
identified geographic data distribution patterns in both the enhanced table and auditory map, they
suggested that the study could be improved by using real data rather than simulated data and
using stronger sampling procedures.
Thirdly, Clusters: Scatter plot. Each of the six variables in Bly's six-dimensional data samples
was translated to sound properties such as pitch, duration, loudness. After dividing the data
samples into two groups, test volunteers were asked to identify which group the data came from
when given visually, aurally, or in a mix of both. While the "data sonograms" concept is
applicable to Bly's work, it was based on Madhyastha and Reed's sound scatter plots in which
they presented designs based on the relative location of each data point.

Lastly Trends: Line Graph; Flower's research did not include any participant testing, but they
did note a few design choices, such as balancing the MIDI to center around the temperature
range for the area of interest for artistic reasons, and choosing a grand piano MIDI #1 as the
timbre evokes the feeling of rainfall, rather than the octave arpeggio on the dulcimer MIDI #15
to indicate snow piling up. These sound graph were more flexible than tactile graphs, according
to the authors, because they allowed the user's hands to be freed for other tasks more crucially,
they can be used effectively independent of the individual's orientation or distance from the
source of sound. Walker and Lane of the Georgia Institute of Technology's Sonification Lab
worked on the directionality of slopes of the line through auditory representation, or "polarity,"
with both sighted and non-sighted participants.
Moreover, in the Analysis and Discussion, the interdisciplinary selection of papers
demonstrated great variety. Some demonstrated a more creative desire to provoke with more
memorable, musical designs, while others concentrated on correct, scientific representations. The
absence or rigor of the experimental process, as well as the quantity of stimuli employed, were
observed differences in the data sonification literature. However, there was a wide diversity in
sample sizes and methodologies among the studies that included experimental data, limiting the
generalizability of the findings. There were also disparities in whether the researchers matched
sounds to real or simulated data in their papers.
As seen in the Objective Vs Subjective Approaches some of the publications were explicitly
suggesting a new design, while others examined numerous designs to quantify their effects.
Some researchers conducted tests with sighted and low-to-no-vision individuals, depending on
their goals and level of rigor. Flowers, merely offered various design patterns without
undertaking any experimental evaluation in contrast to Tomlinson, who did a longitudinal
research to fully grasp the long-term ramifications of their work.
While Abstract Vs Functional Data various data sources were employed depending on the sort
of data representation being investigated. Flowers, for example, utilized weather pattern data to
see whether the design made sense, whereas Flowers used simulated data for histogram and box
skew to see if participants could perceive the differences in trends across the plots, despite the
fact that the data had no semantic value.
With data from the 2010 Census for population by state and data from the 2015 American
Community Survey for employment and unemployment by age and education their prototype
replicated three distinct strategies for data sonification. They also gave each participant a chance
to see the designs in a different order: three people saw the job data sonification first, and two
people saw the population data sonification first. They divided the population of the states into
five primary areas (excluding Hawaii, Alaska, Puerto Rico, and Washington, DC) and assigned
each one a pitch the higher the pitch, the bigger the population; the lower the pitch, the smaller
the population.
The results of this research participants remarked the labels and instruments were more important
than individual tones they could tell the facilitator exactly what each instrument signified, as well
as the trend, but they couldn't recall which pitches were higher. Despite this, all five participants
said that the designs were "interesting," "cool," or "powerful," and that there was something new
and intriguing about them since they had never "listened to data." What would the participants do
differently? While three participants noted that they appreciated listening to the data in the
background, two participants requested that the labels of state names be displayed initially to
make the map more functional, rather than just after zooming in to each region.
In conclusion, they discovered that there have been several attempts to sonify various sorts of
visual data representations or graphs as a result of our data sonification research. While they
were unable to correctly recollect the data in the designs, all participants stated that that was their
first time listening to data, and that they appreciated the experience and considered it enjoyable
and exciting. "Investigating Perceptual Congruence Between Data and Display Dimensions in
Sonification," a paper presented at CHI 2018, highlights that "listeners often have a mental
model of how a data variable should sound during sonification, and this model is not taken into
account in most data sound mappings."
Recommendations, the researchers suggested that they can assist make data accessible to persons
with weak numeracy if they can create aural data representations in a meaningful and scalable
fashion, similar to the Culturally Situated Design Tools (CSDT), a project they should look into
commencing soon. Also they encountered a major difficult while reviewing the literature as
much of it is decades old and lacked access to the audio files associated with each of the various
papers, this was particularly problematic when audio stimuli were created with simulated data.
Thus newer literature should be used in future studies.
Bibliography

Carney, M., & Lee, J. (n.d.). (rep.).