Commentary on Morin 1

Visual languages and the problems with ideographies: A commentary on Morin

Neil Cohn and Joost Schilperoord

Tilburg University, Tilburg School of Humanities and Digital Sciences, Department of

Communication and Cognition

Tilburg University, Tilburg School of Humanities and Digital Sciences, Department of

Communication and Cognition, P.O. Box 90153, 5000 LE Tilburg, The Netherlands

Neil Cohn: neilcohn@visuallanguagelab.com

Joost Schilperoord: j.schilperoord@tilburguniversity.edu

Neil Cohn: www.visuallanguagelab.com

This is a preprint. Please cite the published version, which is:

Cohn, Neil, and Joost Schilperoord. 2023. "Visual languages and the problems with
ideographies: A commentary on Morin " Behavioral and Brain Science 46 (E240). doi:
10.1017/S0140525X23000778.

Abstract
Morin argues that ideographies are limited because graphic codes lack a capacity for
proliferating standardization. However, natural graphic systems display rich standardization and
can be placed in sequences using complex combinatorial structures. In contrast, ideographies are
not natural, and their limitations lie in their attempts to artificially force a graphic system to
behave like a writing system.
Commentary on Morin 2

In “The puzzle of ideography” Morin argues that ideographic systems of pictorial signs are
limited in their communicative capacities because “Graphic codes can only be standardized for a
limited number of meaning-symbol mappings.” While we agree that ideographies are limited
systems, we disagree with Morin’s reasons why, especially the notion that the graphic modality
itself is limited in its semiotic capacities. Specifically, ideographies attempt to artificially force a
graphic system to behave like a writing system, which itself is an adaptation of the vocal
modality into the graphic modality.

First, it’s important to recognize that “standardization” or “conventionality” is orthogonal to

symbolicity (Peirce, 1940). These notions are often conflated (de Saussure, 1972 [1916]), but
standardization or conventionality is how much a signal is patterned and recognized across
individuals. These signals (idiosyncratic or patterned) correspond to meanings through various
interfaces characterizing their signification (i.e., iconicity, indexicality, and symbolicity). To
clarify: a signal in sounds or graphics can be standardized or not, and how that signal
corresponds to conceptual structures characterizes its signification(s).

Despite their stereotype as “arbitrary symbols”, spoken languages display all types of
signification, as in Figure 1a (Clark, 1996; Ferrara & Hodge, 2018), and so does the graphic
modality (Figure 1b). Because all modalities use all types of signification, symbolicity is not the
issue, but rather the question is about standardization.

Contrary to Morin’s statements, the graphic modality displays voluminous standardization at

multiple levels of complexity. While small “pictographs” like hearts, stars, or peace or radiation
signs are easy to recognize as standardized (as in ideographies), all graphics use standardized
building blocks (Arts & Schilperoord, 2016; Cohn, 2013; Wilson & Wilson, 1977). Drawings are
constructed from low-level visual patterns, such as how people draw eyes, headshapes, houses,
and flowers, as in the conventionalized hands by three comic artists in Figure 1c. Graphics also
use classes of combinatorial signs, such as the inventory of elements that float above characters
heads or replace eyes, which use systematic and symbolic meaning-making (Figure 1d), in
addition to visual vocabulary like motion lines, impact stars, speech balloons, and other highly
standardized, culturally-variable, graphic codes. While these small visual “morphemes” can
combine into novel pictures, larger units can also be systematized, whether as templates of
abstract scenes (Figure 1e) or templates in reference to other scenes, which often invoke
symbolic meanings (Figure 1f). These observations contrast with the phenomenologically-based
idea that drawing and graphics are about articulating one’s idiosyncratic vision of what one sees,
despite this notion being unsupported by cognitive, cultural, and developmental research
(Wilkins, 2016; Wilson, 1988) and having erroneous origins (Willats, 2005).
Commentary on Morin 3

Figure 1. a) Vocal and b) graphic signification, and standardized c) components of drawings, d)

combinatorial signs, and scene templates e) without and f) with reference to other graphics.

Visual representations also allow a range of conventionalized sequencing. Many patterns of two-
unit sequences persist to show causative before-after relations, contrasts, or analogies
(Schilperoord & Cohn, 2022). Longer sequences often provide visual lists of related images,
Commentary on Morin 4

such as what is allowed in a park or on an airplane (Cohn & Schilperoord, 2022). Visual
narratives also use recursive combinatorial structures for sequential images displaying structural
features of linguistic grammars, but operating at a higher-level information structure than the
organization of nouns and verbs (Cohn, 2013; Cohn & Schilperoord, 2022). These visual
narrative sequences are natural productions of sequential images, and the specific sequencing
constructions they use have been shown to vary across cultures’ comics (Cohn, 2019), again
indicating culturally relative standardization, not universality. In addition, their processing
invokes the same neural responses as linguistic syntax and semantics, and the understanding of
these visual sequences requires proficiency that is acquired through exposure to and practice
with those cultural graphic systems (Cohn, 2020).

In contrast to these natural graphic systems, ideographies use a basic lexicon of simple graphics
attempting to have “word” levels of information, often created top-down by individuals, rather
than emerging from a language community. These graphics are intended to be sequenced through
a syntax, but since graphics do not naturally afford sentence-level combinatorics, they end up
parasitic to spoken languages. Writing systems themselves are adaptations of the spoken into the
graphic modality, but ideographies then assume graphics should behave like writing systems—
forced into sentence-level sequences—to take on “linguistic” properties. This is why
ideographies are largely invented by specific people (e.g., Blissymbolics, as in Morin’s Figure 4),
because they do not proliferate instinctively through human history or cultures.

These invented ideographies are thus systems that attempt to mimic the structures of
speech/writing which serve as people’s reference-point for what “linguistic graphics” should be
like. However, this denies the affordances and linguistic properties already displayed by natural
graphics in the first place. It should be no wonder then that ideographies don’t work.

There is a clear analogue to this in the bodily modality. Sign languages are natural linguistic
systems that optimize the affordances of the bodily modality and thereby do things in ways that
differ from the structure of speech (Liddell, 2003). Yet, attempts persist to force sign languages
to have the properties of spoken languages, like Manually Coded English, which maps the
lexicon and grammar of spoken English onto the body (Supalla, 1991). As adaptations of one
modality to another, these systems deny the bodily affordances that natural sign languages
display, just like ideographies deny the affordances apparent in natural graphic systems.

To conclude, ideographies are limited because they attempt to make the natural expressive
graphic modality behave like writing, itself a conversion of the spoken modality into graphics.
This quality undermines Morin’s claim that “understanding why ideography has not worked in
the past may help us understand how technology could make it work in the future”, because
ideographies’ unnaturalness will never “make it work.” Rather, this discussion raises the
importance of investigating the affordances of all our modalities and their meaning-making
capacities, and especially a greater integration of graphics into the study of the mind and
cognition.
Commentary on Morin 5

Competing Interests Statement

The authors have no competing interests to declare.

Funding Statement
This research received no specific grant from any funding agency, commercial or not-for-profit
sectors.

References
Arts, A., & Schilperoord, J. (2016). Visual Optimal Innovation. In C. Fernandes (Ed.),
Multimodality and Performance (pp. 61-81). Newcastle upon Tyne, UK: Cambridge
Scholars Publishing.
Clark, H. H. (1996). Using Language. Cambridge, UK: Cambridge University Press.
Cohn, N. (2013). The visual language of comics: Introduction to the structure and cognition of
sequential images. London, UK: Bloomsbury.
Cohn, N. (2019). Structural complexity in visual narratives: Theory, brains, and cross-cultural
diversity. In M. Grishakova & M. Poulaki (Eds.), Narrative Complexity: Cognition,
Embodiment, Evolution (pp. 174-199). Lincoln: University of Nebraska Press.
Cohn, N. (2020). Who understands comics? Questioning the universality of visual language
comprehension. London: Bloomsbury.
Cohn, N., & Schilperoord, J. (2022). Remarks on multimodality: Grammatical interactions in the
parallel architecture. Frontiers in Artificial Intelligence, 4, 1-21.
doi:10.3389/frai.2021.778060
de Saussure, F. (1972 [1916]). Course in General Linguistics (R. Harris, Trans.). Chicago, IL:
Open Court Classics.
Ferrara, L., & Hodge, G. (2018). Language as Description, Indication, and Depiction. Frontiers
in Psychology, 9. doi:10.3389/fpsyg.2018.00716
Liddell, S. K. (2003). Grammar, Gesture, and Meaning in American Sign Language. Cambridge:
Cambridge University Press.
Peirce, C. S. (1940). Logic as Semiotic: The Theory of Signs. In J. Buchler (Ed.), The
Philosophy of Peirce: Selected Writings (pp. 98-119). London: Kenga Paul, Trench,
Trubner & Co. Ltd.
Schilperoord, J., & Cohn, N. (2022). Before: Unimodal linguistics, After: Multimodal linguistics:
An expoloration of the Before-After construction. Cognitive Semantics, 8(1), 109–140.
doi:10.1163/23526416-bja10025
Supalla, S. (1991). Manually coded English: The modality question in signed language
development. In S. D. Fischer & P. Siple (Eds.), Theoretical issues in sign language
research. Volume 2: Psychology (pp. 85-109). Chicago: University of Chicago Press.
Wilkins, D. P. (2016). Alternative Representations of Space: Arrernte Narratives in Sand. In N.
Cohn (Ed.), The Visual Narrative Reader (pp. 252-281). London: Bloomsbury.
Willats, J. (2005). Making Sense of Children's Drawings. Mahwah, NJ: Lawrence Erlbaum.
Wilson, B. (1988). The Artistic Tower of Babel: Inextricable Links Between Culture and
Graphic Development. In G. W. Hardiman & T. Zernich (Eds.), Discerning Art:
Concepts and Issues (pp. 488-506). Champaign, IL: Stipes Publishing Company.
Commentary on Morin 6

Wilson, B., & Wilson, M. (1977). An Iconoclastic View of the Imagery Sources in the Drawings
of Young People. Art Education, 30(1), 4-12.