Johannes Lund With - Tangible & Embodied Interaction - Individual Essay - H22 semester

The Sight of a Scent: Engaging Crossmodal

Correspondences in Olfactory Displays

Introduction
Creating artifacts for displaying smell is an interesting challenge for interaction
designers, as there is no dominant display technology (like screens for sight or
speakers for hearing) - a challenge further compounded by the difficulties of describing
smell with language, the lack of a common notation system and high individual
variance in susceptibility to different smells (Niedenthal, 2012, pp. 5-6).
This leaves designers with the quite open question of how to make meaningful
smell-delivery-devices. Historical examples suggest that a way to do this is to conceive
the display as smell-first and subserving the other sensory modalities to its delivery.
This paper outlines the background and theory for such a view, before applying it to the
analysis of a design case.

Background: olfactory displays and their relation to multisensory outputs

Olfactory displays have to grapple with the technicalities of particular molecules having
to physically enter the user’s body - the storage, vaporization, blending and delivery of
scent material (see fig. 1), each of which can be done in many ways (Yanagida, 2012).
Deciding on these solutions is central to the design, as they provide constraints that
the designed artifact will have to be beholden to - on the other hand, this is only part of
the challenge: the same configuration of delivery-solutions might give rise to vastly
different artifacts and interactions. Eg. the same “smell cubes” have been used for
photograph categorization (Brewster et al. 2006) and reminders for home care
(Williamson et al., 2012).

Fig. 1. Overview of the components of smell-delivery (Yanagida, 2012. p. 1)

Johannes Lund With - Tangible & Embodied Interaction - Individual Essay - H22 semester

A key part of this difference is in the engagement with other sensory modalities.
Looking at historical smell-centric artifacts, they are often very active in their
engagement of other senses. For example, 17th century “rosewater eggs'' - blown-out
eggs filled with rosewater, made to be thrown around during dinner parties - combine
the kinesthetic thrill of throwing with the hygrosensory anticipation of getting wet as a
central part of the delivery (Niedenthal, 2012, p. 6-7).
Incense clocks, an ancient chinese way of telling time by the combustion rate of
incense, are another example of multisensory olfactory displays, wherein the sight of
the smoke, or even sound caused by bells dropped by incense sticks burning away
under them (as in fig. 2), contribute to both the artifacts’ function and aesthetic appeal
(Bedini, 1963, pp. 26-27).

Fig. 2. “Dragon”-style incense clock with attached bells (Bedini, 1963, p. 26)

More modern examples, however, have tended to posit smell as a secondary sense, to
enhance or add further immersion to an audiovisual experience. These include
cinematic experiments, like Scent-O-Vision, AromaRama or Green Aria, or games - some
early adventure games included Scratch-N-Sniff cards to be smelled at given times
(Niedenthal, 2012, pp. 7-11) while wargames manufacturer Games Workshop recently
launched a range of scented candles to accompany the various factions of their
Warhammer 40,000 franchise1.

This suggests that an avenue for creating more culturally impactful olfactory displays
might be keeping the smell and its delivery as the central factor in the motivation for
how to engage the other senses, rather than subserving it to them (eg. “adding
immersion” to an audiovisual experience). One successful example of such a display is
the SensaBubble by Seah et al., that uses visuals projected on soap bubbles containing
scented smoke (Seah et al. 2014).

Theory
A key term for understanding multisensory experiences is that of crossmodal
correspondences. These “are the non-arbitrary and consistent associations between

1
See https://www.wargamer.com/warhammer-40k/scented-candles-reveal
Johannes Lund With - Tangible & Embodied Interaction - Individual Essay - H22 semester

stimulus features in different sensory modalities that are shared among most of the
population” (Ward et al. 2021. p. 1) and are key in decision processes and appreciation,
while incongruent sense data negatively impacts enjoyment.
In relation to smell, Ward et. al. have found strong crossmodal
correspondences with some scents, like coffee and caramel being associated with
rounded shapes and bassy sounds, while lemon and peppermint were associated with
angular shapes and treble sounds - see the scores of various smells in fig. 3 (Ward et
al. 2021)

Fig 3. Crossmodal correspondences between different smells and other sensory stimuli
(Ward et. al. 2021, p. 5)

While this is important for designing for smell, the presence of these associations in
themselves tell us very little of how to employ them as designers. And merely positing
the answer as “stronger crossmodal correspondences equals better design” is overly
reductive, giving us only a single axis to appraise a design from, as well as implying the
assumption that strong crossmodal correspondences are always desirable - something
the rosewater eggs mentioned above cast doubt on, as a large part of their enjoyment
seems to also derive from the incongruence between what we see (an egg) and what
we smell (rosewater).

To broaden the problem but keep a smell-centric approach, a place to begin is in

looking at how to design for olfaction itself and review it from a crossmodal
perspective.
Johannes Lund With - Tangible & Embodied Interaction - Individual Essay - H22 semester

Maggioni et al. describes four major components of the olfactory design space:
chemical, emotional, spatial and temporal (Maggioni et al. 2020). Chemical factors
concern the physical properties of the smell itself (including source and intensity), but
also semantic associations (eg. banana-scent with “banana”) while emotional factors
include smell’s powerful relation to memory but also pleasantness and its salience, ie.
ability to make objects stand out. Spatial and temporal factors concern olfaction’s
unique relationship to these dimensions: the former in smell’s ability to propagate
through space and bring attention to particular places, the latter in both smell’s ability
to linger and the effects of habituation during continued exposure to a smell.
The paper discusses crossmodality to some extent, eg. in combining a lemon
scent with a treble sound for greater impact, but it is worthwhile to extend this notion
beyond individual 1-to-1 associations for specific smells, in regards to all four domains.
Chemical factors are closely tied to the given smell in question, but in general
terms, both source, intensity and semantic associations are possible touchpoints for
investigating crossmodality. For example, isoamyl acetate is a chemical compound
present in many fruits, but particularly prominent in the smell of banana, giving a rough
way to indicate the “banananess” of a given smell, something which can then be
reinforced or reinterpreted through semantic mappings - eg. pairing it with yellow hues
and high intensity to reinforce the banananess of it, or green ones and lower intensity
to lean into more generally fruity associations.
Emotional factors are difficult to quantify in terms of their crossmodal potential,
but memory, pleasantness and salience likewise provide points to work off. The delight
of the rosewater egg, for example, can be seen as a combination of the memory of
breaking eggs (a combined haptic, visual and auditory experience), the salience of a
thrown egg (implying immediate, messy breakage) with the ‘surprising’ pleasantness of
the scent (compared to the near-odorless slime of raw egg).
Spatial and temporal factors exist across all sensory modalities in some ways
and thus provide two additional axes to explore commonalities between the senses,
each providing rich sensory data that can be compounded or rejected by our other
senses. The pleasantness of wood-fire smoke, for example, is contingent on time and
place: cozy if by the campfire, highly alarming if the location of the source is not
known. Likewise, while pleasant while on a camping trip, looking at the flames and
hearing the crackle of the logs, it is less desirable afterwards when the odor lingers on
the clothes but is accompanied by everyday visual and auditory cues (Maggioni et al.
2020. pp. 1-9).
Johannes Lund With - Tangible & Embodied Interaction - Individual Essay - H22 semester

Fig. 4. Overview of design factors and their crossmodal touchpoints (based on Maggioni
et al. 2020)

This overview (summarized in fig. 4) is of course far from comprehensive, but provides
a starting point for how to open up smell-centric crossmodality. Below, these concepts
will be explored in relation to concrete design work to show how they can be used in
practice to assess the crossmodal strengths and weaknesses of a design.

Design Analysis
Ambiens is an ambient notification system primarily relying on olfaction to notify the
user. It consists of a wooden box with a water mister under patterned grating on one
end and a protruding bamboo-and-paper fan on the other, from which swirls of mist
emerge and are wafted away by the fan (see fig. 5). The mist was scented with a
kiwi-fruit fragrance.
Johannes Lund With - Tangible & Embodied Interaction - Individual Essay - H22 semester

Fig. 5. The “Ambiens” olfactory display

In regards to the display’s crossmodal qualities, the visual choices made in the design
both lend it certain strengths and some significant incongruences.
The visual qualities of the mist gives strong spatial and temporal crossmodal
anchorage of the smell. It is clearly visualized how and when the smell is released into
the space, with the qualities of the swirls giving an indication of how the smell
propagates into the surrounding space - for example, it is clearly visible if the
propagation of the smell was disturbed by a nearby drafty window.
Additionally, the refreshing hygrosensory qualities of the mist corresponded
nicely with the perceived “freshness” of the odor described by several test participants,
one person (nr. 6 in fig. 6) even being in doubt if it was the freshness of the mist or the
scent that they were experiencing.
Johannes Lund With - Tangible & Embodied Interaction - Individual Essay - H22 semester

Fig. 6. Overview of associations drawn by study participants

The choice of material, however, provided some strong incongruences that pulled in
the other direction and made the device’s olfactory-visual output less clear and more
confusing. The laser-cut plywood of the box had a quite distinctive burnt-wood smell
that was keenly noticed by all participants, to the point of almost entirely obscuring the
released scent. One explanation is that the wood-smell is simply more intense, but a
likely contributing factor is the strong crossmodal correspondence between
woodsmoke and the sight of the charred wood casing providing a stronger overall
impression than the mist, which suggested fresh, sweet and flowery scents.
Likewise, the smokey associations are probably compounded by the device’s
reminiscence of incense burners, which suggest the deeper, ‘bassier’ smell of incense,
and thus at odds with the fresh and flowery qualities of the scent that was released.
Overall, this all contributed to the released scent being perceived as overall
quite low-intensity and not too pleasant, though the delight of the swirls were remarked
on as a positive by most participants. In relation to the concepts outlined above, it can
be posited that the swirls in the mist gave strong temporal and spatial crossmodal
correspondence between the scent and the visual output, while the chemical and
emotive qualities of the delivered smell was at odds with the device’s aesthetic
qualities.

Further Work
What has been provided is far from a comprehensive map of this design space, but
rather a timid starting point from which further, more robust, investigations can be
undertaken. Nevertheless, going beyond merely demonstrating the presence of
crossmodal correspondences and moving on to engaging with how to actually use
them constructively within design practice seems an important question for making
more meaningful olfactory displays in the future.
Johannes Lund With - Tangible & Embodied Interaction - Individual Essay - H22 semester

Conclusion
The aim of this paper has been to open up the space surrounding smell-centric
crossmodal correspondences beyond simple pairings (eg. “lemon scent is bright and
yellow”) and to further nuance how a given display can successfully engage with
crossmodality in some ways while being confusing or unpleasant in others.
The distinction between chemical, emotional, temporal and spatial
correspondences is just one way of doing this, but one that proves to have some
measure of practical, analytical applicability. This conception keeps olfaction as the
primary engaged sense, as these categories are intimately tied to the nature of smell
itself.

Literature
Bedini, S. A. (1963). The Scent of Time. A Study of the Use of Fire and Incense for Time
Measurement in Oriental Countries. Transactions of the American Philosophical
Society, 53(5), 1-51. American Philosophical Society.

Brewster, S. A., McGookin, D. K. & Miller, C. A. (2006). Olfoto: Designing a Smell-Based

Interaction. CHI 2006, Montreal, Québec, Canada. April 22-27, 2006. Association
for Computing Machinery.

Maggioni, E., Hornbæk, K., Obrist, M. (2020). SMELL SPACE: Mapping out the Olfactory
Design Space for Novel Interactions. ACM Transactions on Computer-Human
Interaction, 27(5). Association for Computing Machinery.

Niedenthal, S. (2012). Skin Games: Fragrant Play, Scented Media and the Stench of
Digital Games. Eludamos. Journal for Computer Game Culture. 6(1), 1-3.
Septentrio Academic Publishing.

Seah, A. S., Plascencia, D. M., Bennett, P. D., Karnik, A., Otrocol, V. S., Knibbe, J.,
Cockburn, A., Subramanian, S. (2014). SensaBubble: a chrono-sensory mid-air
display of sight and smell. CHI ‘14: Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems. Toronto, Ontario, Canada. April 14 - May 1.
Association for Computing Machinery.

Ward, R. J., Marshall, A., Wuerger, S. M. (2021). Smelling sensations: olfactory

crossmodal correspondences. Journal of Perceptual Imaging, 4(2). Society for
Imaging Science and Technology.

Williamson, J. R., McGee-Lennon, M., Brewster, S. (2012). Designing Multimodal

Reminders for the Home: Pairing Content with Presentation. ICMI’12, Santa
Monica, California, USA. October 22-26, 2012. Association for Computing
Machinery.
Johannes Lund With - Tangible & Embodied Interaction - Individual Essay - H22 semester

Yanagida, Y. (2012). A survey of olfactory displays: Making and delivering scents.

Proceedings of IEEE Sensors, Taipei, Taiwan. October 28-31. Institute of Electrical
and Electronics Engineers.