Bateman 2019 End of Disembodied Semiotics

International Journal of Semiotics and Visual Rhetoric
Volume 3 • Issue 2 • July-December 2019
Transmediality and the End of

Disembodied Semiotics
John A Bateman, Bremen University, Bremen, Germany
ABSTRACT
The phenomena of mixing, blending, and referencing media is a major topic in contemporary media
studies. Finding a sufficient semiotic foundation to characterize such phenomena remains challenging.
The current article argues that combining a notion of ‘semiotic mode’ developed within the field
of multimodality with a Peircean foundation contributes to a solution in which communicative
practices always receive both an abstract ‘discourse’-oriented level of description and, at the same
time, a biophysically embodied level of description as well. The former level supports complex
communication, the latter anchors communication into the embodied experience. More broadly, it
is suggested that no semiotic system relevant for human activities can be adequately characterized
without paying equal attention to these dual facets of semiosis.
Keywords
Embodiment, Multimodality, Peirce, Semiotic modes, Transmediality
INTRODUCTION: THE CHALLENGE OF TRANSMEDIALITY
The phenomena of combining, mixing, blending, referencing and re-using media within the context
of the use of other media has become a major line of research in both contemporary media studies (cf.
Bolter & Grusin, 2000; Jenkins, 2008; Grant & Wilkinson, 2009) and approaches to adaptation and
‘reworkings’ of many kinds (cf. Elliot, 2003; Hutcheon, 2006; Elleström, 2010a; Bruhn et al., 2013;
Hassler-Forest & Nicklas, 2015; Bruhn, 2016). Although media have arguably always been ‘mixed’
in many ways, the sheer flexibility, extent, and speed of change for combining media evident today
is unmatched. Attempts to engage theoretically with the phenomena arising in such situations are
consequently of growing importance and the literature on the topic is already extensive. When seen
from a semiotic perspective, however, the situation is very different and there are few semiotically
sophisticated accounts directly engaging with the phenomena, their semiotic foundations, and semiotic
consequences.
The development of semiotic descriptions of medially complex artifacts and performances began
in earnest when ‘popular’ communication, itself medially complex, was accepted in the 1960s as a
justified object of study. Starting from the seminal work of Roland Barthes on a variety of media (cf.
Barthes, 1977), considerable work followed focusing on visual semiotic systems, as in, for example,
Groupe μ (e.g., Groupe μ 1992), Saint-Martin (1987) and, on broadly Greimasian lines, in Floch
(1981). With few exceptions (e.g., Kawama, 1990), such work continued to build on the broadly
Saussuro-Hjelmslevian lines adopted by Barthes and, for several reasons that will be discussed below,
are as a consequence limited in their abilities both to characterize transmediality and to explain its
effectiveness. In addition, in a general move towards cultural studies within semiotic discussions,
analyses of all persuasions tended to consider restricted use cases, illustrating semiotic analyses on
the basis of very small collections of examples and sometimes even single cases (e.g., Nöth, 2011).
DOI: 10.4018/IJSVR.2019070101

Copyright © 2019, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.

1
Within discussions of this kind, several crucial notions underlying the ‘medialities’ that were
being combined remained unaddressed. The primary reason for this was that the concept of ‘medium’
itself was not available semiotically for discussion. To this day, semiotic accounts talking of signs
and their combinations have not, in general, been refined sufficiently to distinguish media and their
properties as particular kinds of meaning-making. Although there is now work that attempts to move
beyond this state of affairs – particularly, for example, in the theoretical position set out in detail by
Elleström (2014a, 2014b) or in approaches originating in narrative concerns (e.g., Ryan, 2005; Wolf,
2007; Rajewksy, 2010) – the accounts of media involved are still, it will be argued, limited with
respect to their semiotic foundations. The task taken up here will consequently be to set out a more
foundational view of media that semiotically re-constructs the notion of a medium drawing on more
recent developments in the area of multimodality. This will be suggested to enhance our abilities to
work with embodied materiality and its role in media combination.
Multimodality has emerged over the past two decades as a theoretical and practical branch of
applied semiotics that provides strong methodological principles for the analysis of artifacts and
performances of any semiotic complexity; a general introduction to the field is given in Bateman et
al. (2017). The central research question defining ‘multimodality’ is how materially and semiotically
diverse resources can combine productively in the service of meaning making. Multimodal research
asks, for example, how it is that a combination of written text and pictorial images both conveys ‘more’
and is often more effective than either form of contribution alone. Standard areas of communication
addressed in detail within multimodality studies include film, face-to-face dialogic interaction, graphic
novels, webpages and so on – in each case, attention to any one form of communication (language,
image, music, sound effects, layout and composition, etc.) would be insufficient for understanding
the communicative artifacts or performances as a whole. The present paper will introduce the core
theoretical construct within multimodality, that of semiotic mode, and show how this can be used to
generate a definition of media that is particularly supportive of discussions of transmedial phenomena
as well.
In addition, by explicitly relating the notion of semiotic mode developed within multimodality
studies to the dimensions of semiotic description articulated by Peirce, primarily in his ‘mature’
model of around 1903, it will be argued that the effectiveness of transmedial communication can be
characterized in a far more compelling fashion than has previously been the case. In short, it will
be suggested that current explanations of the functioning of transmedial communication in terms of
‘iconicity’ can be beneficially refined to consider the particular effects of embodied perception and
physical responses to materiality. Such an orientation is motivated primarily by addressing Peircean
qualisigns rather than simply iconicity. The paper then argues in conclusion that the strong binding
established between semiotic modes and qualisigns demands that all semiotic descriptions be anchored
in embodiment and that ‘disembodied semiotics’ should no longer be considered an available option.
The organization of the paper is as follows. First, a brief introduction to the theoretical challenges
and practical applications of ‘medium’ and ‘medialities’ will be offered. Second, the view of media
that can be generated within contemporary accounts of multimodality will be outlined. Third, the
relationship that this model of media constructs with Peircean semiotics and particularly with
embodied, distributed cognition and the role of qualisigns will be set out. The paper then turns to
discussion of specific cases showing the framework in action before concluding with a brief outlook
for future directions of research.
THE CHALLENGE OF TRANSMEDIALITY
The term transmediality will be used here in a deliberately broad fashion so as to bring into focus a
range of diverse, but nevertheless related, phenomena. The term will then receive a closer definition
below after the necessary semiotic groundwork has been laid. Transmediality is currently raised as
an issue in several disciplines. In sociology and media studies concerns with changes in society and
2
cultural formations going back to McLuhan (1964) and others have led to mediality receiving increased
attention as a foundational property of the human social world (e.g., Couldry & Hepp, 2017). In
literature, aesthetics and art studies, changing creative practices are increasingly working with and
theorizing artworks that operate across media boundaries (e.g., Clüver, 2007) and there is extensive
discussion of narrative beyond the traditional confines of the verbal (e.g., Ryan, 2004; Horstkotte,
2009; Wolf, 2007; Kulezic-Wilson, 2015; Bruhn, 2016). Similarly, in research areas focusing on the
study of interaction, language and communication, the role of the media involved similarly comes into
increasingly sharp relief as it becomes evident that the embedding of communication within media
is in many respects constitutive of the forms that develop; the study of the discourses supported by
‘new’ social media, for example, is already a major research direction in its own right (Brinker, 1985;
Holly, 2011; Schmitz, 2016). Much of the current emphasis on ‘media convergence’ and ‘remediation’
(Bolter & Grusin, 2000; Jenkins, 2008) rests on the growing interpenetrability of diverse media that
such technologies support.
Despite this broad take-up of the topic, accounts of media that engage seriously with the semiotic
foundations of communication are rare. Conversely, there are few discussions in semiotics that
explicitly address the theoretical status of media as well (cf. Withalm, 2006; Bateman, 2017). This
lack of connection between semiotics and media studies has itself occasionally been made a point
of discussion. Winfried Nöth, for example, draws attention to this issue with regard to the traditions
pursued in different cultures/traditions of semiotic research. As Nöth points out, whereas in France,
Spain and Brasil ‘media science’ and ‘media semiotics’ appear almost to operate as synonyms, in
Germany and most English-speaking environments, there is almost no contact (Nöth, 1997, pp. 6–7).
When media do come to be discussed, what is most commonly found are loose mixtures of
Saussure and Peirce that do not anchor the terms drawn upon back into their necessary definitional
contexts. Saussure is employed to discuss language, emphasizing arbitrariness and the signifier-
signified pairing, and Peirce brings iconicity for discussing pictorial media and, for media such as
photographs, indexicality. This unstable combination is then augmented with discussions of cultural
and social implications, often drawing on Barthes’ suggestion of a distinction between denotatative
and connotative meanings – a distinction going back to Hjelmslev’s (1943) definition of connotative
and denotative semiotics, although rarely presented as such by more recent writers. Discussions then
turn to individual ‘media’, typically mass media, and a relatively loose application of connotation,
iconicity and arbitrary signs (cf., e.g., Bignell, 2002; Danesi, 2002). The notions of ‘signs’ employed
in such discussions remain relatively independent of any claimed engagement with medium, although
this relationship is, it will be argued below, important and complex in its own right.
The situation is not particularly improved in discussions of researchers beginning more
fundamentally with Peirce rather than with Saussure. ‘Medium’ is not a category that is native to
Peirce’s model and so constructions of media within the Peircean tradition tend instead to rely on a
relatively simple, because unspecified, ‘multiplication’ of semiosis. That is: since media are complex
cultural constructions, they are taken to give rise to multiple, interwoven chains of semiosis – but
this is in itself hardly supportive of more precise analysis. Attention then also falls back more on
discussions of ‘individual’ media, such as photography, film, theatre, radio, advertising and so on.
This offers little advance over Peirce’s own few comments on ‘medium’ taken from manuscripts
from around 1906 and cited by Withalm (2006, p. 638), Jappy (2017, p. 56) and others as follows:
“All my notions are too narrow. Instead of ‘Sign’, ought I not to say Medium?” (Peirce MS R339
[1906], The Logic Notebook: seq 530 [283r]; Emphasis in original)
“A sign is plainly a species of medium of communication.” (Peirce MS R283 [1906], The Basis of
Pragmatism: 125)
“For the purposes of this inquiry a Sign may be defined as a Medium for the communication of a
Form.” (Peirce MS R793 [1906], On Signs: 1)
3
These proposals echo suggestions still common today that see ‘medium’ either as a mediating
channel or as a vehicle or carrier of communication (or both).
What is needed for progress beyond this state of affairs and towards a useful framework that
is not only semiotically well-founded, but also supportive of empirical investigations of all cases
of media mixing, blending, quoting and representation, is a far more detailed disentangling of the
notions of medium, sign, channel, code and, indeed, semiotic mode. These (and other) terms often
appear in discussions without clear characterizations of their mutual borders and of what their
particular respective contributions as distinct theoretical categories are intended to be (Withalm,
2006, pp. 638–639). As a consequence, the term ‘medium’ continues to find itself subjected to the
same stresses and strains found in discussions of the modes of multimodal communication: some
discussions move more to consider issues of the materiality of communication and the physical/material
carriers or channels of that communication, while others move more towards the particularly semiotic
contributions of abstract forms or modes of communication. None of these positions is sufficient in
its own right for constructing a foundation for media as a semiotic construct, however. Approaches
to date then remain somewhat promissory with respect to their support for empirical analysis and for
the fine-grained modeling that would be necessary for such analysis to take place.
This leaves many media researchers skeptical of the contribution that semiotics might make. The
proposal made by Withalm (2006, p. 636), for example, that what semiotics can offer to the study of
media is a “differentiated view of the communication process as sign process” proves less than inspiring
because characterizations of that sign process remain at too great a remove in abstraction from the
specifics of communicative situations to support empirical work. Moreover, the further one moves
away from the (already in many respects unrealistically abstracted) focus on verbal communication
still common in many areas, the less grasp of specifics is achieved. Multimodal communication, i.e.,
the use of diverse communicative forms, resources and materialities in the service of meaning making,
consequently leaves most existing semiotic models appearing relatively weak in their analytic hold of
the semiotic processes unfolding. The ability of semiotically detailed accounts to engage productively
with empirical studies is thus severely compromised, which in turn hinders development because
theoretical descriptions remain at too far a remove from actual cases to show their faults.
At this point, however, we can begin to clarify more of the beneficial methodological role that
questions of transmedial explorations can play. When considering how communicative practices can
move between and across materially very different forms of communication, it becomes important
to address far more specifically just what facets of semiosis are involved and how these respond
to differing conditions of production. One of the basic problems that continues to hinder research
on media from a semiotic perspective is then that media is simply ‘too large.’ That is: any specific
medium considered pre-theoretically appears to involve a combination of facets at several very
disparate levels of abstraction, ranging from the signs made in any specific materiality involved up
to entire institutional contexts of production, distribution and reception intrinsic to the existence of
a communicative form as a medium at all. Unraveling some of the necessary semiotic complexity
here is essential in order to place discussion on a firmer basis.
One of the most detailed accounts seeking to address such phenomena while also doing equal
justice to definitions of media and their semiotic import is that of Lars Elleström (Elleström, 2014a,
2014b, 2018). As Elleström clearly sets out:
“The problem is that intermediality has tended to be discussed without clarification of what a medium
actually is. …I find it as unsatisfying to continue talking about ‘writing’, ‘film’, ‘performance’,
‘music’ and ‘television’ as if they were like different persons that can be married and divorced as to
find repose in the belief that all media can be fundamentally blended in a hermaphroditical way.”
(Elleström, 2010b, pp. 11–12)
4
Elleström consequently offers a characterization of ‘medium’ that has a strong semiotic core,
usefully articulating the principal dimensions by which media can both be distinguished from one
another and related. Elleström defines four modalities of a medium (Elleström, 2010b, pp. 17-24,
36) thus:
• a material modality, described as the ‘latent corporeal interface’ of a medium

• a sensorial modality, described as the ‘physical and mental acts of perceiving the interface’
• a spatiotemporal modality, described as the ‘structuring of the sensorial perception of the material
interface into experiences and conceptions of space and time’
• a semiotic modality, described as the ‘creation of meaning … by different sorts of thinking and
sign interpretation’
Transmedial relationships are then characterized by comparing and contrasting media products
along these four dimensions.
The first two dimensions engage with embodied perception and interaction, thereby aligning
well both with several recent multimodal accounts of semiosis as we shall see below (cf. Kress & van
Leeuwen, 2001), while the third dimension specifically picks out the ‘spatiotemporal’ properties of any
such material. These three dimensions are labeled ‘pre-semiotic’ in contrast to the fourth dimension,
which Elleström characterizes in terms of Peircean iconicity, indexicality and symbolicity. This is
clearly an advance over previous semiotic discussions where media have appeared as undifferentiated
wholes or looser complexes or bundles of semiotic processes. It also permits a useful bridge to be
constructed to accounts originating in more recent views of multimodality, as we shall now see.
A MULTIMODAL PERSPECTIVE ON MEDIA AND MEDIALITY
In Bateman (2017), an alternative construction of ‘medium’ is offered anchored within multimodal

semiotics. Central to this research endeavor is the nature of the ‘semiotic modes’ that are combined
in order to give rise to ‘multimodality’ – i.e., just what does the ‘multi’ of ‘multimodality’ refer to?
Various positions have been adopted with respect to this issue, ranging from a restricted orientation
back to the physical sensory channels employed (sight, sound, touch, etc.), thereby overlapping with
some accounts of medium, to abstract communicative practices that may be employed independently of
medium or material carrier. As set out in Bateman (2017), there are many similarities to be drawn here
with discussions of the ‘media’ concept, which also range across these kinds of levels of abstraction.
As Elleström has explained, one reason for this is that both the concepts of ‘semiotic mode’ from
multimodality and ‘medium’ in media studies have remained sufficiently vague that their extensions
overlap, resulting in a muddied view of just what, if anything, each may contribute.
On the basis of detailed analyses of a broad range of specific kinds of communicative artifacts,
Bateman (2011, 2016) offers a definition of the core multimodal concept of ‘semiotic mode’ that
combines both attention to material details and the diverse affordances of varied materialities on
the one hand, and the use of regularities inscribed in such materials for semiotic purposes, on the
other. This extends earlier multimodal accounts of Kress, van Leeuwen and colleagues (e.g., Kress
& van Leeuwen, 2001; Kress, 2010) by taking ‘semiotic mode’ as a doubly stratified configuration
of semiotic resources together with the materialities that those resources engage: in contrast to many
previous approaches, these dual aspects are seen as inseparable and constitutive for any semiotic
mode. In addition, the semiotic resources are themselves internally stratified in a manner analogous
to the expression-content plane division proposed by Hjelmslev (1943).
This gives rise to the three-way stratified model shown graphically in Figure 1. First, on the
left-hand side of the figure, we see the material that is ‘claimed’ for expression of the distinctions
drawn in the semiotic mode: this can involve any materiality that allows perceptible differences to be
constructed and may range freely across sensory channels. Second, such differences are organized
5
into systems of alternatives – the ‘technical features’ or ‘structural regularities’ of the semiotic mode
– that are charged with the task of carrying the meanings of the semiotic mode; this is shown in the
centre of the diagram. And third, on the right-hand side of the figure, a level of ‘discourse semantics’
is introduced analogously to the level of abstraction suggested for language by Martin (1992). The
resources at this level of abstraction are responsible for correlating the structural configurations of
the previous level with contextual interpretations; examples of these mechanisms at work in the case
of some visual semiotic modes are given in Bateman and Wildfeuer (2014).
This model has several important consequences when conducting empirical research of complex
multimodal semiotic artifacts or performances.
First, since materialities are in general very rich, or ‘dense,’ it is possible for many distinct semiotic
modes to be operative at the same time and with respect to the same material. This can be seen even
for verbal language, where the acoustic signal is modified along several dimensions simultaneously
in order to carry segmental phonetic information, suprasegmental information such as intonation,
temporal phrasing and melody, as well as (independently) information concerning emotional state,
age, and so on. Precisely analogous multiple mode deployments occur on the static printed page,
where visually realized language, pictorial, diagrammatic and spatial layout may all operate together,
each taking partial responsibility for the overall material form of the artifact. Further examples of
such multimodality, showing how this is the usual situation for all meaning meaning, are given in
Bateman (2016).
Second, semiotic modes which are operative in specific artifacts or performances is still, at this
time, essentially an open empirical question. Whereas some semiotic modes appear relatively clear
– such as written language and spoken language – in fact the precise boundaries and range of modes
operative here still require further research. Whether, for example, gesture and facial expression
should be included as part of the semiotic mode of verbal language or as distinct supportive modes
remains a challenging issue. In general, however, it can already be noted that semiotic modes will
tend to be far ‘smaller’ than typically assumed when discussing either broader, sensory channel
views or largely pre-theoretical terms such as ‘image’. At the same time, these semiotic modes are
nevertheless ‘broader’ in the sense that they are not restricted to lie within single sensory modes.
Divisions along more traditional lines such as language, gesture, images, music and so on quickly
reveal their limitations as the multimodal artifacts or performances at issue are explored in more
detail. Again, Bateman (2016) provides details.
Figure 1. Abstract schematic representation of a semiotic mode following Bateman (2016, p. 47)
6
Third and finally, the anchoring in perceived materiality as a constitutive facet of each and
every semiotic mode requires the account to engage directly with issues of embodied perception and
emotional responses to the material regularities employed. It is no longer sufficient, for example,
simply to characterize an abstract semiotic distinction between large and small, or between loud
and quiet, as formal distinctions that might be played out independently of material: the biophysical
response to such distinctions also makes an inseparable contribution to any meanings made. It is not
then accidental that a loud and sudden noise in a film may be employed for the discourse purpose of
raising narrative tension: the articulation of the artifact relies upon the physical response just as much
as it does on the broader narrative strategies being pursued. Examples of this duality are discussed
further in Bateman (2019) and will play an important role below.
We can now make the relation between semiotic modes and medium and media explicit. As noted
above, any particular materiality may play host to a broad range of semiotic modes. The materialities
that are available are, however, generally determined by the medium with which we are concerned.
Materialities are not simple naturally occurring objects in the wild, but are constructed for the
purposes of communication within cultural and institutional contexts of production, distribution and
reception. Materialities of this kind are strongly dependent on technological circumstances, such as
the development of paper and the printing press, or on the telephone or the internet. As a consequence,
media provide the materialities that an institutionalized collection of semiotic modes may draw on for
semiotic purposes. The medium of graphic novels, for example, provides a page-based ‘canvas’ that
may be utilized by the semiotic modes of written language, of schematic pictorials, of page layout,
of color and framing and so on. Similarly, the medium of film provides a time-based ‘canvas’ that
may be utilized by the semiotic modes of spoken language, of sound design, of moving pictorial
images, of cuts, camera distances and angles, etc. The details of such ‘canvases’ therefore correspond
fairly directly to Elleström’s ‘pre-semiotic’ media modalities introduced above. Figure 2 suggests
this relationship between semiotic modes and media graphically. The vertical ‘slice’ on the left-hand
side of the figure running through the materialities of the contributing semiotic modes constitutes
the generalized ‘canvas’ of the medium.
A multimodally enriched characterization of media of this form offers complementary semiotic
perspectives on media and materiality that are particularly beneficial when pursuing empirical
application: i.e., media regularities can be triangulated both materially and semiotically. In effect, this
can then be seen as the beginnings of an ontology of multimodal semiotic artifacts and performances,
combining fine-grained descriptions in terms of semiotic modes and broader, culturally and
institutionally anchored descriptions of their use. Semiotic modes therefore offer a kind of ‘subatomic’
physics of media, pulling apart the contributing ways of making meaning that are socioculturally
combined at any point in time by the institutionalization of those media.
This then shows, on the one hand, how any claims made over the years concerning media
essentialism or ‘eternal’ properties of media (cf Greenberg, 1940) cannot hold while, on the other,
making it clear that at any particular point in their existence media may have determinate and distinctive
expressive properties inherited by virtue of the semiotic modes that they are mobilizing. Nevertheless,
to the extent that any semiotic mode is deployed in more than one medium, then those media will
naturally then ‘share’ expressive possibilities: such questions thereby become empirical issues in
their own right. This position also offers valuable clarification for discussions of many of the kinds
of ‘media’ in common use – such as, for example, the considerable debate within art history around
whether an ‘image’ is to be considered a mode, a medium, or mental construct or all or none of these
(cf., e.g., Boehm, 1995; Mitchell ,2005). Distinguishing the quite distinct semiotic work performed
by modes and media allows such questions to be posed (and answered) anew.
7
Figure 2. Abstract schematic representation of the semiotic mode and medium relationship following Bateman (2016, pp. 56, 61)
THE RELATION BETWEEN SEMIOTIC MODES AND PEIRCEAN SEMIOTICS
Our next step is to draw a close connection between the view of semiotic modes developed in
multimodality and Peirce’s highly developed account of semiosis. Whereas the concept of ‘medium’
remains difficult to relate to Peircean semiotics with the degree of detail that would be necessary to
make that combination do useful work in supporting analysis, the more fine-grained account of semiotic
modes as constitutive ‘components’ of any medium allow a rather direct connection to be drawn. We
can consequently use a semiotic re-construction of the multimodal notion of ‘semiotic mode’ within
a broadly Peircean scheme as a bridging tool for similarly reconceptualizing media from a Peircean
perspective. This will also allow us at the same time to make a further connection to several current
attempts to approach transmediality and intermediality from a more semiotically-informed position.
To begin, we return to the ‘tri-stratal’ organization of semiotic modes introduced above and
shown graphically in Figure 1. Each of the levels of abstraction as well as the principal mechanisms
posited for relating them in fact find a relatively natural home in Peirce’s model of the sign from
around 1903. First, all semiotic modes are established as conventionalized forms of practice within
specific communities of users and are therefore appropriately characterized as Peircean legisigns.
Although Peirce’s own comments on legisigns remained somewhat fragmentary, it is evident that they
also need to be considered as internally structured – and this raises important questions concerning
the nature of those structures. Moreover, such structures have to perform particular kinds of semiotic
work (cf. Short, 1982, p. 304): first, there is the task of distinguishing legisigns that otherwise might
be confused and, second, these distinctions need to be correlated with consequences for interpretation.
These kinds of semiotic work are strikingly analogous to the functions provided by the discourse
semantic stratum of a semiotic mode.
8
Semiotic modes are consequently to be interpreted as legisigns of a very particular kind – i.e.,
socioculturally established practices within which meanings can be attributed to material forms – and
with a particular internal structural organization – i.e., the three-way semiotic stratification. Moving
within this stratification, Peirce’s notion of entailments, or prescinding relationships, holding across his
distinct trichotomies of ways of being signs, is also strongly suggestive of the realizational relationships
posited by Hjelmslev to hold across semiotic strata: e.g., configurations within the discourse semantic
stratum must be realized in configurations in the stratum below. In more Peircean terms, the legisigns
of semiotic modes are only recognized via their instantial realizations in sinsigns, which in turn can
only be recognized by means of regularities in material forms, i.e., in configurations of qualisigns. In
short, the potential for signification given by a semiotic mode as legisign is related to the actualization
of that potential by means of the semiotic movement ‘down’ from legisign to sinsigns to qualisigns.
A graphical depiction of the proposed structural relationships between the two orientations is given
in Figure 3, adapting one of Peirce’s diagrams (MS R339, dated 7th August 1904) relating signs and
combining this with the graphical notation for semiotic modes introduced above. Detailed discussion
of this relation between current multimodality theory and Peirce can be found in Bateman (2018).
The connection drawn here between semiotic modes and legisigns is essential for binding together
sign use and discourse purposes (the highest semiotic stratum of a semiotic mode). As frequently noted
in discussions of Peirce, occurrences of signs are generally only intelligible when anchored against
the purposive use of those signs (cf. Lefebvre 2007). This then allows us to beneficially refine the
increasingly common proposal that a suitable foundation for characterizing how transfers of meanings
and communicative strategies may be ‘transferred’ across media can be found in the phenomenon
of iconicity (e.g., Elleström 2013; Emmorey 2014; Perniss & Vigliocco 2014; Mittelberg & Waugh
2014). As we shall argue in more detail as we proceed, any focus solely on iconicity in fact leaves
accounts with an inherent gap. It is not the presence of bare similarities of material possibilities or
patterns (i.e., iconicity) that support transmediality but rather the purposive deployment (i.e., legisigns
and the middle and lower semiotic strata of a semiotic mode) of any such material possibilities
in the service of congruent discourse goals. Nevertheless, within this overall view, the Peircean
requirement that qualisigns need to inhere in sinsigns insists that we also take the role of the lowest
semiotic stratum within a semiotic mode equally seriously because it is here that materiality – and a
sign-user’s embodied engagement with that materiality – is situated. These two facets together will
now be argued to be essential for any adequate account of transmediality.
Figure 3. Proposed paths of communication opened up between the Peircean semiotic framework (left) and semiotic modes as
defined (right)
9
MATERIALITY AND THE SEMIOTIC BASIS OF TRANSMEDIALITY
Building on the framework set out in the previous sections, we now suggest how some quite
challenging cases of mediality, and subsequently of transmediality, can begin to be accommodated.
For this, we can take perhaps the most embodied art form: dance. This will allow us to show rather
straightforwardly why semiotic modes, and our use of semiotic modes in the services of analysis,
need to operate with both discourse purposes and embodied expression of, and response to, those
discourse purposes. These levels of description are then also what is needed when considering any
transmediality involving semiotic modes. Transmediality as such will be argued to rely crucially on
the embodied nature of the semiotic modes that any of the media involved are deploying and their
use for discourse.
First, following the methodology for multimodal inquiry set out in Bateman (2016) and Bateman
et al. (2017), we need to decide on the medium that is to be investigated, which for current purposes
we might define illustratively as performed classical ballet. The materiality of this medium – including
stage, lighting, stage props, backdrops, music and the bodily movements and expressions of the
dancers – is supportive of a rich host of semiotic modes, although precisely which, as emphasized
above, is an empirical issue. Again, for current purposes, we will assume and subsequently focus on
the physical movements of the dancers as constituting one such semiotic mode. We will also focus first
of all on the ‘internal’ perspective of dancers, i.e., ballet from the perspective of personal embodied
movement. The materiality involved in this semiotic mode is therefore relatively straightforward:
the particular ‘slice’ at issue is the physical body of the dancer and all movements, including both
changes in position and posture, that that body can perform. This materiality includes in principle
all bodily expressed capabilities, including facial expression, gaze directions and so on but is also
nonetheless far from unconstrained: the human body can only move in particular ways given by its
physical construction, which has already been described by several detailed formal models. An initial
description of the regularities described here as belonging to the ‘middle’ semiotic stratum of the
semiotic mode is given in Maiorani (2017), while a cognitively motivated functional segmentation
is explored, for example, in Bläsing et al. (2009). In addition, and again solely for current purposes,
the discourse organization of the higher semiotic stratum will be assumed to be similar to that of
other narrative-capable modes, although in general this would need extensive empirical investigation
of its own.
Developing this further, Hsieh and Luciani (2005) set out a useful characterization of the semantics
of dance verbs in terms of direct models of qualities, energies and forces. By adopting a ‘physically-
based particle’ approach, the behavior of physical systems is captured in terms of various forces
acting between particles and changes in their energy states. Hsieh and Luciani use this approach to
implement descriptions of specific kinds of dance movements, naturally incorporating notions such
as centre of gravity, balance, rotation, jumps and projection (cf. Laws, 2002). This is illustrated well
in their dynamic model of the verb ‘to squat down’ shown in Figure 4. In this figure, the center of
gravity or ‘energy sink’ is identified by the basin and squatting down is then:
“the result of simple physical actions combination: one is the relaxed descent of the basin under
its weight and the other is the effect of an external force coming from the ground through the foot.
These two forces push together the knee, so that it operates spontaneously a horizontal translation.”
(Hsieh & Luciani, 2005, p. 4)
A view of the embodied materiality of the dancer of this kind naturally lends itself well not only
to depictions of dance actions from the ‘outside’, but also to characterizations of those actions from
the ‘inside’: i.e., in terms of the embodied sensations of the dancer performing the actions. Robotic
systems are in fact increasingly being fitted out with internal sensors that provide the equivalent of
proprioceptive feedback of this kind and so it becomes both possible and relevant to consider such
‘internal’ viewpoints as essential components of formal models of materiality wherever (particularly
embodied) movement is involved. For the embodied actor performing such movements, there is
10
Figure 4. Model of ‘squatting down’ from Hsieh and Luciani (2005)
naturally a ‘whole-body’ response due to the physical actions as such and this is an inherent component
of the materiality employed by the semiotic mode.
At the same time, a dancer may be performing sequences of actions that combine into an intended
narrative; this is particularly the case for classical dance, including ballet. Thus, at the discourse level
of description, there is an unfolding topic structure and narrative development (such as the ‘story’ of
Swan Lake), while at the material level that topic structure unfolds in already embodied movement
with all of the physical responses that that movement gives rise to, such as flight, panic, speed, and
so on. This is then what offers much of the power and ‘interest’ of performing the dance.
Extending the example so that we now consider an audience, or audience member, watching the
dance, it is known that observing performed actions will often give rise to sympathetic activations of
similar neural configurations to those required to perform those actions (cf. Rizzolatti & Craighero,
2004). Such ‘mirror-neuron’ systems have now been observed for a host of quite diverse kinds
of actions, ranging from simple movements and tool manipulations to musical performance and
social interactions. This offers an important candidate for explaining why observing certain medial
productions (either directly or remotely, or ‘mediated’ in a more traditional sense) may be so involving
because the perception of performed activities is by no means restricted (in the case of visual
perception) to the construction of a ‘visual’ image or representation. Instead, more or less abstract
neural simulations of activities are performed by observers (Gallese & Sinigaglia, 2011; Gallese &
Cuccio, 2015). This is what is then captured in the view of semiotic modes offered above and its
tri-stratal organization. A detailed discussion of some consequences of this for the semiotics and
aesthetics of film, which we will draw on below, is given in Bateman (2019). In terms of our Peircean
characterization, this suggests that the kind of embodied perception relating observed movements to
physical responses is ‘more’ than iconicity: rather than just ‘knowing’ or working out ‘more’ on the
basis of iconicity, one actually experiences more, due to the embodied simulation.
This can then be applied to all cases of transmediality or depictions or representations that are
taken across media and picks up on, within a semiotically-grounded perspective, several earlier
11
proposals or claims in the literature for strong ‘sensorial’ or ‘embodied’ responses. Eisenstein, for
example, writing on his ‘overtonal’ montage in film, talks of this technique “enhancing perception
from a melodically emotional colouring to a direct physiological sensation (Eisenstein, 1929, p. 191,
emphasis in original). Indeed, going further:
“Because, while a shot is a visual perception and a tone is a sound perception, both visual and
sound overtones are totally physiological sensations. And, consequently, they are of one of the same
kind, outside the sound of acoustic categories that serve merely as guides, paths to its achievement.
For the musical overtones (a beat) the term ‘I hear’ is no longer strictly appropriate. Nor ‘I see’ for
the visual. For both we introduce a new uniform familiar: ‘I feel’.” (Eisenstein 1929: 186; emphasis
in original)
This synesthetic quality of engagement with film is also taken up in Chion’s (1994, pp. 136–137)
discussion of ‘transsensorial perception’, discussed with many further illustrations concerning abstract
patterns such as rhythm and intensity in film by Kulezic-Wilson (2015).
Similarly, turning to an example discussed by Rajewsky (2005, pp. 56–57), consider a particular
‘dance’ that was performed in the narrow space between two vertically-placed transparent plates so
as to give the effect of a two-dimensional pictorial image. Here we can postulate that the kinds of
movements involved may trigger simulations of various kinds. There is first of all a limited kind
of dance movement; there is also, however, the embodied perception of a surface with its own sets
of (restricted and restricting) simulations. This then would support, and provide mechanisms for
explaining, Rajewsky’s proposal that:
“the means and instruments of dance theatre (bodies, costumes, movements, lighting, stage props,
etc.) are employed and fashioned in a way that corresponds to, and resembles, elements, structures
and representational practices of painting, thus creating an illusion of painterly qualities. …Hence,
the sequence as a whole constitutes itself (and is ‘received’ by the viewer) in relation to painting,
imitating, but at the same time – as is typical of intermedial references of this kind – also expanding
the representational modes of the medium being referred to …: it is as if the viewer sees a painting put
into motion, turned to life – a tableau vivant in the truest sense of the term.” (Rajewsky, 2005, p. 57)
The correspondence of media largely integrated and ‘co-spatialized’ by virtue of the shared
unfolding discourse organization of the dance, is driven at a ‘deeper’, more embodied level by means
of the shared simulations that automatically and necessarily characterize semiotic interpretation. In
the current case it would be necessary therefore not only to address the transmedial associations with
painting but also to allow the feelings of constraint and restriction to movement from the embodied
nature of being ‘contained’ within the space provided for the dance as an aesthetic component as
well. Investigations of the consequences of combining distinct kinds of simulations as required here
is a research direction very much in its infancy. However, this appears to be precisely where much
of the power of transmedial artifacts and performances resides.
Since mechanisms of this kind will now be taken to be operative in all semiotic modes, they are
naturally extremely broad in application. An illustration of a completely different area where we can
arguably find the very same process at work can be found in the verbal language of art history and
reflection on artworks. Here, for example, is how the artist Paul Klee is reported to describe one of
his paintings:
“A wide span from pole to pole lends space to breathe in and out deeply, which can even turn
into a wheezing struggle for air. A lesser span muffled the breath to a soto voce. Around the grey
area, it is only possible to whisper. Either you raise yourself above this position to the violins or sink
below it to the violoncellos” (Klee cited in Düchting, 1997, p. 60)
The work in question, Colour Table (In Grey Minor) (1930) consists of a ‘checkerboard’ of
variously colored but regular rectangles; there is no depiction of breathing of any kind and no musical
instruments. The question then is what gives rise to such a description.
Baxandall (1979) considers the language of art as raising challenging issues concerning how
it is that verbal language can be used for intrinsically visual phenomena at all. He consequently
12
characterizes the language of art history as principally ‘demonstrative’, i.e., guiding the attention of
a viewer to consider certain aspects of a work by means of “a sort of visual pointing”. Thus, rather
than engaging with art works by describing them, the language of the art critic or historian is instead
concerned with picking out aspects of the experience of engaging with any work in question and
attempting to share this. This is then quite similar to the position that Peirce adopts for all meaning-
making; that is, without indexicality and collateral experience, sign use is never sufficient to reach
interpretation.
It is commonly suggested that language of the above kind is then ‘metaphorical’, translating
from one area of experience to a description of another. In a purely formal sense, Klee’s language is
clearly metaphorical; however, in the terms now opened up with respect to the anchoring of semiotic
modes in embodied engagement, there is a further possibility. Klee can also be describing literally
his engagement with the work, either in responding to it or during his creation of the work. In other
contexts, as Düchting relates, Klee draws connections not with verbal descriptions but more directly
with music and, occasionally, even to (performed) dance. All such transmedial configurations speak
to shared or overlapping embodied materiality tied together by common discourse concerns for
orchestrating communicative effect.
In terms of theoretical accounts, striking similarities and parallels can now therefore be drawn
out in several accounts of multimodality and transmediality concerning the central role of materiality.
Most approaches have now argued at some point that characterizing communicative acts across
media will demand more engagement with embodied materiality. This is already strongly present
in Elleström’s account introduced above. His three pre-semiotic levels of description, ranging over
materiality, sensorial responses to materiality and spatiotemporality (in contrast to any iconic, indexical
and symbolic statuses played out by media artifacts), are where most of the work of transmediality
is located. The specifically ‘semiotic’ modes in Elleström’s terms are then only involved indirectly
in transmediation via their pre-semiotic anchoring. Thus, for example:
“A visual gesture may depict largeness through the magnitude of the area that the hands
circumscribe, and a musical sound may depict largeness through, for instance, a broad range of pitches
kept together in melodic form – suggesting expansion.” (Elleström, 2014b, p. 50)
It is the common embodied sense of space that relates the two forms of expression. In a
rather similar fashion, van Leeuwen (2016) in multimodality studies argues that certain semiotic
‘organizations’ result from general, amodal, or ‘cross-sensory’ principles that find different expressions
in different modalities. Van Leeuwen describes this in terms of synesthesia, identifying categories
such as ‘framing’, which may find expression equally in pictorial depictions (by means of visual
discontinuities) and in aural modalities (by means of pauses and other phenomena of rhythm). In
both cases, it is clear that properties are being identified that are not in themselves restricted to
particular materials, semiotic modes or media, but which are inalienably constitutive of materiality.
Moreover, both also correspond to the placement of materiality as the necessary lowest level of
semiotic abstraction in the definition of semiotic modes offered above.
Despite the similarities and parallels, however, the linkages between the distinct levels of
description involved are usually theorized to very limited degrees. Van Leeuwen draws on Lakoff
and Johnson’s (1980) and Fauconnier and Turner’s (2002) informal cognitive notions of conceptual
metaphor and blending, particularly in the application of image schemas, whereby structured mappings
are posited between embodied experience, on the one hand, perhaps represented in terms similar
to Gallese and Cuccio’s (2015) ‘body formatted’ information, and conceptual configurations, on
the other. Thus, a bodily formatted reaction to heat and cold may be ‘transcoded’ as a conceptual
organization in terms of color-terms. For van Leeuwen, therefore, ‘experiential metaphor’ is taken
to relate materiality to more traditional semiotic concerns and similarities where engagements with
materiality are characterized in terms of ‘integrative principles’ (e.g., of synesthesia); ‘embodied’
meaning constructions of this kind have now been proposed in several approaches to media (e.g., Cienki
& Müller, 2008; Kappelhoff & Müller, 2011; Hague, 2014; Fahlenbrach, 2016; Leman et al., 2018).
13
Elleström similarly insists on an inalienable interaction between the various levels of description
(Elleström, 2014b, pp. 48–50), the precise mechanisms of which remain to be explored, primarily by
neurocognitive and perceptual research. Several accounts make analogous distinctions (e.g., Sonesson,
2008, p. 55), but similarly remain less explicit on the issue of how the distinct levels of abstraction
relate. In the next section, we consider how we might now gain extra leverage on this challenge by
combining the semiotic stratification assumed for semiotic modes and our Peircean interpretation
of those semiotic modes.
CLEARING OUT THE SEMIOTIC BASEMENT
Elleström’s account relying on pre-semiotic media modalities also draws on iconicity to characterize
the similarities driving transmedial expressions. Thus, with the example of the transmedial expressions
of ‘size’ or ‘extent’ above, there are two dimensions of iconicity at work: one dimension relating
embodied response (e.g., ‘largeness’) to expressions (e.g., “a sweeping gesture”) and another among
materially differentiated embodied responses (e.g., the various but similar “notions of largeness”).
Both the visual gesture and the musical expression are then considered ‘iconic’:
…sound iconically representing largeness is clearly not exactly the same as visual impressions
iconically representing largeness, whereas the two notions of largeness are similar enough to be
regarded as equivalent, they are not identical. …the iconic aptitude of the target medium is modified
by the alteration of material, sensorial, and spatiotemporal modes. (Elleström, 2014b, p. 50)
This leads to a considerably extended notion of cross-modal iconicity (Elleström, 2017). Here in
addition to the more traditional, and now extensively researched, case of crossovers between sensory
modes, his spatiotemporal and material media traits are included as well as areas across which
similarities and differences can be found and deployed (Elleström, 2017, pp. 176–177). Although a
characterization of this kind broadly agrees with the kind of cross-material relationships suggested
above to play a central role for transmediality, it appears that the use of ‘iconicity’ here has not been
sufficiently clarified. Within Elleström’s model, iconicity is one of the possibilities of the semiotic
media mode and so is not directly accessible as a mechanism of the ‘pre-semiotic’ materialities.
In order to probe the interrelationships at issue here in more detail, it is necessary to consider the
semiotic foundations of iconicity more deeply.
First then, following Peirce, sign-relations functioning iconically are only actually found as
hypoicons, i.e., actually occurring investigable entities that may be attributed sign status, initially as
sinsigns. The iconic function of such sinsigns is then carried by the qualisigns inhering in them: it is
only here that we gain access to qualities or properties that may be drawn upon to drive iconic reasoning.
Now, qualisigns are properties or qualities in so far as such properties or qualities are attributed sign
status. And yet, even ‘before’ such attributions of sign statuses for ‘us’ as social human beings, the
‘stuff’ (or ‘purport’ in Hjelmslevian terms) already has properties of its own, regardless of their use
for signs. It is largely in this organization that the bases for embodied reactions to combinations of
qualisigns are laid.
Moreover, reasserting a more semiotic standpoint, it is clear that the internal organization of
purport may itself be considered as being ‘signful’, albeit for our bodies rather than for us as aware
social beings. The dependence of perception on our species-specific bodily endowments is well
developed in the philosophy of perception and empirically evident in the fact that different species
construct ‘basic’ notions – such as color – quite differently (cf., e.g., Osorio & Vorobyev, 2005).
The relationship to the bodily requirements of particular species is then explained evolutionarily. As
Matthen (2005) sets out in considerable detail, for example, the relative sensitivities of the cones
in the human eye result in a system that is excellent for discriminating red and green and it is then
14
arguably no accident that red-green separation is an ideal capability for recognizing, for instance,
berries against background vegetation. Similarly, ‘body’-centered viewpoints apply to our perception
of space in terms of ‘anthropomorphic geometry’ and of movement and action (cf. Zlatev, 1997;
Gallagher, 2005).
Thus, the organization of purports can itself be sensibly considered semiotically: the interpretants
at this ‘level’ of semiotic organization offer the basic building blocks that mesh with the semiotic
stratum of materiality set out above in the proposed model of semiotic modes. By these means we
achieve a thorough-going semioticization from embodied perception through to the most abstract
social meanings – as Peirce would certainly have wanted. However, the different levels are related to
one another as stratal, or recoding, correlations in Hjelmslev’s sense. They thus allow us to talk on the
one hand of signs for the body and on the other hand of signs for the social human being, but in a way
that allows the kinds of more differentiated discussion advocated with respect to embodiment by, for
example, Sonesson (2007), without also importing any dualism. The two levels (and their respective
internal organizations) are all co-descriptions of single phenomena rather than distinct realms of
‘mind’ and ‘body’. This may then also be related both to Smith’s (2012) important philosophical
characterization of transdisciplinary ‘triangulation’ as a method of research and to Hutto and Myin’s
(2017) proposal of non-representational cognition in radical enactivism.
The distinction drawn thus provides a means of talking usefully about the quite distinct kinds
of organization relevant for a more differentiated use of ‘iconicity’. And, for these reasons, we will
not be concerned with descriptions couched more in terms of physics, chemistry or biology when
discussing materiality: it is always ‘engaged whole-body perception’ that is at issue. There is clearly
very much more to be said here and much to be discovered. On the one hand, one might look to
notions of ‘conceptual spaces’ as developed by Gärdenfors (2000), or the dimensions of synesthesia
proposed in van Leeuwen (2016), for characterizations of the bodily organization of ‘qualia fields’.
On the other, one might see in the distinct levels further bases for differences between different kinds
of abductive inference as proposed in discussions such as those of Paavola (2011), where the rather
different apparent uses of abduction in perception and in other kinds of more abstract reasoning within
a single Peircean view are problematized.
Such a position then allows us to usefully distinguish cases of genuine experienced ‘sameness’ of
the kinds discussed above in terms of embodied simulations and more indirect cases of media depiction
or reference – thereby supporting and developing further Elleström’s concerns in characterizing
transmediality in terms of basic mechanisms that draw on neurocognitively anchored notions of
similarity and difference as examples of such mechanisms (Elleström, 2017). It also positions more
precisely the reoccurring suggestion made in the literature that iconicity is somehow a ‘graded’
phenomenon (e.g., Anderson, 1998, p. 72; Stjernfelt, 2000, p. 370; Bauer & Ernst, 2010, p. 43;
Colapietro, 2011; Elleström, 2013, p. 106): this is now to be seen as a property of the neurocognitive
mechanisms supporting perception of similarities and difference, and not as a property of the semiotic
notion of iconicity. This means that arguments and examples involving graded notions of similarity
and difference can be recognized, but only as applying to relations among bundles of qualia and not
to iconicity as such (and particularly not to Peirce’s subclasses of hypoicons).
Thus, whereas Elleström, for example, seeks to relate the sensory and the conceptual as lying
along a continuum defined in terms of graded iconicity, the relation drawn here is rather in terms of
interstratal realization following Hjelmslev (cf., e.g., Thibault, 2006). The sensory does not ‘blend’
into the conceptual, but rather sensory and conceptual commitments are both always co-present and
co-describing. This is what gives real force to the kinds of transmedial relationships illustrated above.
ONE FURTHER ANALYSIS
One final example drawn from Bateman (2019) will illustrate the more immediate connection drawn
here between embodied engagement with material and the use of such material in extended discourses,
15
particularly demonstrating how these levels are co-present, co-descriptions. The well-known segment
from Alfred Hitchcock’s Marnie (1964) summarized in Figure 5 shows the eponymous main character,
played by Tippi Hedren, stealing a sum of money from the safe of her employer. Hitchcock sets up the
scene so that the viewer is made aware that another person is in the office. We thus have the classic
configuration of a difference in states of knowledge of the characters and the spectators. Since we
know ‘more’ than the characters, suspense is often said to be constructed – here this is refined further
by the co-description of embodiment: it is not only the differential ‘knowing’ that leads to tension,
but rather the embodied nature of the projected consequences of that knowledge (cf., e.g., Vorderer
et al., 1996; Lehne & Koelsch, 2015).
More specifically: the opening shots in the current example establish caution, surreptitious
movements, and fear of being caught because the entire situation is constructed filmically as precarious.
This is managed through lighting choices, the repeatedly anxious looks of the main character, and
tracking camera movements showing point-of-view shots from her perspective – all of which will
activate embodied simulation, i.e., such shots do not simply ‘look like’ (i.e., show an iconic relation
to) someone moving anxiously, they are experienced as such by viewers (Gallese & Guerra 2012).
At the same time, however, the segment exhibits rich discoursal organization consisting of
parallel actions playing out in time, additional point-of-view shots, and more. The construction of
this discourse structure depends on the fine-grained information made available by the film and, in
this sense, requires close textual analysis. The topic structure that is constructed consequently follows
discourse principles to create characters, events, and evaluations of events (cf. Tseng, 2013) as well
as to signal goals and hindrances (cf. Bateman, 2007; Wildfeuer, 2014). Moreover, in addition to this,
the direct experiential reactions to the materiality being manipulated are also entailed by virtue of
the double articulation anchored in the semiotic mode in play. The tri-stratal definition of semiotic
modes given above supports a detailed characterization of this productive relation between textual and
material patterns: the discourse structures provide longer arcs, which are necessarily also bound into
material forms that involve physical properties and embodied responses. The two facets, the embodied
material and the discoursal, may therefore be combined to maximum effect. Their interrelationship
is summarized graphically in Figure 6; further details are given in Bateman (2019).
Combining discourse structures with embodied responses thus offers a way of bridging several
previous approaches, ranging from cognitive metaphor-based accounts (Kappelhoff & Müller, 2011)
and thematic organizations set out by Tan (1996), Grodal (1999) and others, right through to dual-
coding notions Paivio (1986) and Kahneman’s (2011) ‘fast and slow’ thinking. The Marnie segment
shows convincingly how filmic discoursal organization may push the embodied material responses
of the viewer to unfold in the manner that a film-maker intends. The discourse structure constructs
Figure 5. Frame stills from a sequence from Alfred Hitchcock’s Marnie (1946)
16
Figure 6. Discoursal/embodied material characterization of the Marnie sequence
spaces within which embodied simulations may be opened, blended, sequenced, extended or shortened
in duration, separated, and ended. In a very real sense, therefore, discourse development functions
to shape physical responses.
CONCLUSION AND OUTLOOK
In this paper, it has been suggested that the constitutive and essential role of experienced materiality
in semiotic processes of all kinds can be characterized by: (i) relating the view of semiotic modes
developed in multimodality studies with the Peircean account of semiosis; and (ii) constructing
embodied materiality in terms of neural simulation that is made directly accessible during perception.
The result then distinguishes between iconicity ‘proper’ and closer experienced relationships holding
among embodied simulation. The latter is suggested to be the real force behind effective transmediality.
This also serves to characterize the workings of various kinds of adaptation more appropriately.
As Gjelsvik (2013) argues, despite a widespread shift against essentialism in the study of media and
claims of broad inter-exchangeability, this position appears to go too far when suggesting that there
may be no difference in what distinct media can do. With the model set out here we can characterize
this situation in a more fine-grained fashion. First, there is certainly a difference with respect to
different conventions and habits concerning what is actually done with media due to their varying
institutional and cultural embeddings. Second, there has certainly been, at various times, unwarranted
restrictions assumed for what diverse media can do due to a lack of awareness of the workings of
discourse semantics. For example, whereas it has been suggested that a single image cannot express
that something is not present simply because it ‘shows’ positive states of affairs, when discourse
semantics is added this can be seen to be simply fallacious. All that is necessary is to establish
framings that can support the discourse relation of contrast; negation then follows pragmatically as
a matter of course. And third, even though this over-restriction has been made, it is still the case that
distinct semiotic modes will give rise to very different embodied responses.
This echoes the essence of Gjelsvik’s (2013) argument. Following Vivien Sobchack (e.g.,
Sobchack, 1992) and others the audiovisual film is never ‘just’ an image, and for this reason
watching a film is indeed very different from reading a story, even when the discourse relations
and topic structures established at the discourse level are largely congruent. The discoursal levels
17
of description define a web of ‘textual’ interconnections constituting and driving interpretations,

while the materially embodied levels of description provide a web of material distinctions anchored
into experiential responses and embodied experience. Semiotic modes then integrally link these two
aspects as necessarily related strata within any semiotic mode employed. By accepting that semiotic
modes and their deployment in semiosis is always necessarily embodied as argued here, such results
are to be expected.
18
REFERENCES
Anderson, E. R. (1998). A Grammar of Iconism. Madison, Teaneck: Farleigh Dickinson University Press.
Barthes, R. (1977). Image – Music – Text (ed. and trans. by S. Heath). London: Fontana Press.
Bateman, J. A. (2007). Towards a grande paradigmatique of film: Christian Metz reloaded. Semiotica, 167(1/4),
13–64.
Bateman, J. A. (2011). The Decomposability of Semiotic Modes. In K. L. O’Halloran & B. A. Smith (Eds.),
Multimodal Studies: Multiple Approaches and Domains, Routledge Studies in Multimodality (pp. 17–38).
London: Routledge.
Bateman, J. A. (2016). Methodological and theoretical issues for the empirical investigation of multimodality.
In N.-M. Klug & H. Stöckl (Eds.), Handbuch Sprache im multimodalen Kontext [Handbooks of Linguistics and
Communication Science] (pp. 36–74). Berlin: de Gruyter Mouton.
Bateman, J. A. (2017). Triangulating transmediality: A multimodal semiotic framework relating media, modes
and genres. Discourse. Context & Media, 20(Supplement C), 160–174. doi:10.1016/j.dcm.2017.06.009
Bateman, J. A. (2018). Peircean Semiotics and Multimodality: Towards a New Synthesis. Multimodal
Communication, 7(1).
Bateman, J. A. (2019). Multimodality and Materiality: The Interplay of Textuality and Texturality in the Aesthetics
of Film. Poetics Today, 40(2), 235–268.
Bateman, J. A., & Wildfeuer, J. (2014). A multimodal discourse theory of visual narrative. Journal of Pragmatics,
74, 180–218. doi:10.1016/j.pragma.2014.10.001
Bateman, J. A., Wildfeuer, J., & Hiippala, T. (2017). Multimodality – Foundations, Research and Analysis. A
Problem-Oriented Introduction. Berlin: Mouton de Gruyter.
Bauer, M., & Ernst, C. (2010). Diagrammatik: Einführung in ein kultur- und medienwissenschaftliches
Forschungsfeld. Bielefeld: Transcript. doi:10.14361/9783839412978
Baxandall, M. (1979). The language of art history. New Literary History, 10(3), 453–465. doi:10.2307/468922
Bignell, J. (2002). Media Semiotics: an introduction (2nd ed.). Manchester, NY: Manchester University Press.
Bläsing, B., Tenenbaum, G., & Schack, T. (2009). The cognitive structure of movements in classical dance.
Psychology of Sport and Exercise, 10(3), 350–360. doi:10.1016/j.psychsport.2008.10.001
Boehm, G. (Ed.). (1995). Was ist ein Bild? (2nd ed.). München: Wilhelm Fink Verlag.
Bolter, J. D., & Grusin, R. (2000). Remediation: Understanding New Media. Cambridge, MA: MIT Press.
Brinker, K. (1985). Linguistische Textanalyse: Eine Einführung in Grundbegriffe und Methoden. Berlin: Erich
Schmidt Verlag.
Bruhn, J. (2016). The Intermediality of Narrative Literature: Medialities Matter. Basingstoke: Palgrave Macmillan.
doi:10.1057/978-1-137-57841-9
Bruhn, J., Gjelsvik, A., & Hanssen, E. F. (Eds.). (2013). Adaptation Studies. New Challenges, New Directions.
London: Bloomsbury Academic.
Chion, M. (1994). Audio-vision: sound on screen (C. Gorbman, Trans.). New York: Columbia University Press.
Cienki, A., & Müller, C. (2008). Metaphor and Gesture. Amsterdam: John Benjamins. doi:10.1075/gs.3
Clüver, C. (2007). Intermediality and interarts studies. In J. Arvidson, M. Askander, J. Bruhn, & H. Führer
(Eds.), Changing borders: Contemporary positions in intermediality (pp. 19–37). Lund: Intermedia Studies Press.
Colapietro, V. (2011). Image, diagram and metaphor. Unmined resource and unresolved questions. In C.
Ljungberg, O. Fischer, & P. Michelucci (Eds.), Semblance and Signification (pp. 157–171). Amsterdam: John
Benjamins. doi:10.1075/ill.10.09col
19
Couldry, N., & Hepp, A. (2017). The Mediated Construction of Reality. Cambridge, UK and Malden, MA: Polity.
Danesi, M. (2002). Understanding Media Semiotics. London: Bloomsbury Academic.
Düchting, H. (1997). Paul Klee. Painting music. London: Prestel.
Eisenstein, S. (1988). The fourth dimension in cinema. In R. Taylor (Ed.), S.M. Eisenstein. Selected Works.
Volume I: Writings, 1922-34 (pp. 181–194). London: BFI Publishing and Indiana University Press. (originally
printed in 1929)
Elleström, L. (Ed.). (2010a). Media Borders, Multimodality and Intermediality. Basingstoke: Palgrave Macmillan.
doi:10.1057/9780230275201
Elleström, L. (2010b). The modalities of media: a model for understanding intermedial relations. In L. Elleström
(Ed.), Media Borders, Multimodality and Intermediality (pp. 11–50). Basingstoke: Palgrave Macmillan.
doi:10.1057/9780230275201_2
Elleström, L. (2013). Spatiotemporal aspects of iconicity. In L. Elleström, O. Fischer, & C. Ljungberg (Eds.),
Iconic Investigations (pp. 95–120). Amsterdam: John Benjamins. doi:10.1075/ill.12.09ell
Elleström, L. (2014a). Material and Mental Representation: Peirce Adapted to the Study of Media and Arts.
American Journal of Semiotics, 30(1–2), 83–138. doi:10.5840/ajs2014301/24
Elleström, L. (2014b). Media transformation: the transfer of media characteristics among media. Basingstoke:
Palgrave Macmillan. doi:10.1057/9781137474254
Elleström, L. (2017). Bridging the gap between image and metaphor through cross-modal iconicity. An
interdisciplinary model. In A. Zirker, M. Bauer, O. Fischer, & C. Ljungberg (Eds.), Dimensions in iconicity (pp.
167–190). Philadelphia: John Benjamins. doi:10.1075/ill.15.10ell
Elleström, L. (2018). A medium-centred model of communication. Semiotica, 224(224), 269–293. doi:10.1515/
sem-2016-0024
Elliot, K. (2003). Rethinking the Novel/Film Debate. Cambridge: Cambridge University Press.
Emmorey, K. (2014). Iconicity as structure mapping. Philosophical Transactions of the Royal Society of London.
Series B, Biological Sciences, 369(1651), 20130301. doi:10.1098/rstb.2013.0301 PMID:25092669
Fahlenbrach, K. (Ed.). (2016). Embodied Metaphors in Film, Television, and Video Games: cognitive approaches.
New York: Routledge.
Fauconnier, G., & Turner, M. (2002). The Way We Think: Conceptual Blending and the Mind’s Hidden
Complexities. New York: Basic Books.
Floch, J.-M. (1981). Sémiotique plastique et langage publicitaire: analyse d’une annonce de la campagne de
lancement de la cigarette “News.” Documents de recherche - Groupe de recherches sémio-linguistiques, 26(3).
Gallagher, S. (2005). How the body shapes the mind. Oxford: Clarendon Press. doi:10.1093/0199271941.001.0001
Gallese, V., & Cuccio, V. (2015). The Paradigmatic Body: Embodied Simulation Intersubjectivity, the Bodily
Self, and Language. Open MIND, 14(1), 1–22.
Gallese, V., & Guerra, M. (2012). Embodying movies: Embodied simulation and film studies. Cinema: Journal
of Philosophy and the Moving Image, 3, 183–210.
Gallese, V., & Sinigaglia, C. (2011). What is so special with embodied simulation? Trends in Cognitive Sciences,
15(11), 512–519. doi:10.1016/j.tics.2011.09.003 PMID:21983148
Gärdenfors, P. (2000). Conceptual spaces: the geometry of thought. Cambridge, MA: MIT Press. doi:10.7551/
mitpress/2076.001.0001
Gjelsvik, A. (2013). What novels can tell that movies can’t show. In J. Bruhn, A. Gjelsvik, & E. F. Hanssen
(Eds.), Adaptation Studies. New Challenges, New Directions (pp. 245–264). London: Bloomsbury Academic.
Grant, A. E., & Wilkinson, J. S. (Eds.). (2009). Understanding media convergence: the state of the field. Oxford,
NY: Oxford University Press.
20
Greenberg, C. (1940). Towards a newer Laocoön. Partisan Review (New York, N.Y.), 7.
Grodal, T. (1999). Emotions, cognitions, and narrative patterns in film. In C. Plantinga & G. M. Smith (Eds.),
Passionate views: film, cognition, and emotion (pp. 127–145). Baltimore: The John Hopkins University Press.
Groupe μ (1992). Traité du signe visuel: pour une rhétorique de l’image. Paris: Editions du Seuil.
Hague, I. (2014). Comics and the senses: a multisensory approach to comics and graphic novels. New York,
London: Routledge. doi:10.4324/9781315883052
Hassler-Forest, D., & Nicklas, P. (Eds.). (2015). The Politics of Adaptation: Media Convergence and Ideology.
Basingstoke: Palgrave Macmillan.
Hjelmslev, L. (1961). Prolegomena to a theory of language (F. J. Whitfield, Trans.). Madison, Wisconsin:
University of Wisconsin Press. (Original work published 1943)
Holly, W. (2011). Medien, Kommunikationsformen, Textsortenfamilien. In S. Habscheid (Ed.), Textsorten,
Handlungsmuster, Oberflächen: Linguistische Typologien der Kommunikation (pp. 144–163). Berlin: de Gruyter.
Horstkotte, S. (2009). Seeing or speaking: visual narratology and focalization, literature to film. In S. Heinen & R.
Sommer (Eds.), Narratology in the age of cross-disciplinary narrative research (pp. 170–192). Berlin: de Gruyter.
Hsieh, C.-M., & Luciani, A. (2005). Physically-based particle modeling for dance verbs. In Proceedings of the
International Conference Graphicon, Novosibirsk, Russia.
Hutcheon, L. (2006). A Theory of Adaptation. London: Routledge. doi:10.4324/9780203957721
Hutto, D. D., & Myin, E. (2017). Evolving Enactivism: Basic Minds Meet Content. London: MIT Press.
doi:10.7551/mitpress/9780262036115.001.0001
Jappy, T. (2017). Peirce’s Twenty-Eight Classes of Signs and the Philosophy of Representation. Bloomsbury
Advances in Semiotics. London: Bloomsbury.
Jenkins, H. (2008). Convergence Culture: Where Old and New Media Collide. New York: NYU Press.
Kahneman, D. (2011). Thinking Fast and Slow. London: Allen Lane.
Kappelhoff, H., & Müller, C. (2011). Embodied meaning construction. Multimodal metaphor and expressive
movement in speech, gesture, and in feature film. Metaphor and the Social World, 1(2), 121–153. doi:10.1075/
msw.1.2.02kap
Kawama, T. (1990). A semiotic approach to product forms. In T. A. Sebeok & J. Umiker-Sebeok (Eds.), The
Semiotic Web 1989 (pp. 625–638). Berlin: Mouton de Gruyter. doi:10.1515/9783110874099.625
Kress, G. (2010). Multimodality: a social semiotic approach to contemporary communication. London: Routledge.
Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: the modes and media of contemporary
communication. London: Arnold.
Kulezic-Wilson, D. (2015). The Musicality of Narrative Film. Basingstoke: Palgrave Macmillan.
doi:10.1057/9781137489999
Lakoff, G., & Johnson, M. (1980). The metaphorical structure of the human conceptual system. Cognitive
Science, 4(2), 195–208. doi:10.1207/s15516709cog0402_4
Laws, K. (2002). Physics and the Art of Dance: Understanding Movement. Oxford, NY: Oxford University Press.
Lefebvre, M. (2007). The Art of Pointing. On Peirce, Indexicality, and Photographic Images. In J. Elkins (Ed.),
Photography Theory (Vol. 2, pp. 220–244). London: Routledge.
Lehne, M., & Koelsch, S. (2015). Toward a general psychological model of tension and suspense. Frontiers in
Psychology, 6, 79. doi:10.3389/fpsyg.2015.00079 PMID:25717309
Leman, M., Maes, P.-J., Nijs, L., & van Dijk, E. (2018). What is Embodied Music Cognition? In R. Bader
(Ed.), Springer Handbook of Systematic Musicology (pp. 747–760). Berlin: Springer. doi:10.1007/978-3-662-
55004-5_34
21
Maiorani, A. (2017). Making Meaning through Movement: A Functional Grammar of Dance Movement. In M. G.
Sindoni, J. Wildfeuer, & K. L. O’Halloran (Eds.), Mapping Multimodal Performance Studies. London: Routledge.
Martin, J. R. (1992). English text: systems and structure. Amsterdam: Benjamins. doi:10.1075/z.59
Matthen, M. (2005). Seeing, doing, and knowing: a philosophical theory of sense perception. Oxford: Clarendon
Press. doi:10.1093/0199268509.001.0001
McLuhan, M. (2002). The medium is the message. In Understanding Media: the extensions of man, Routledge
Classics (pp. 7–23). London: Routledge. (Original work published 1964)
Mitchell, W. (2005). There are no visual media. Journal of Visual Culture, 4(2), 257–266.
doi:10.1177/1470412905054673
Mittelberg, I., & Waugh, L. (2014). Gestures and metonymy. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D.
McNeil, & J. Bressem (Eds.), Body – Language – Communication. An International Handbook on Multimodality
in Human Interaction (Vol. 2, pp. 1747–1766). Berlin, Boston: Mouton.
Nöth, W. (Ed.). (1997). Semiotics of the Media. State of the Art, Projects, and Perspectives. Approaches to
Semiotics. Berlin: Mouton de Gruyter. doi:10.1515/9783110803617
Nöth, W. (2011). Visual semiotics: key features and an application to picture ads. In E. Margolis & L. Pauwels
(Eds.), The SAGE Handbook of Visual Research Methods (pp. 298–316). London. doi:10.4135/9781446268278.
n16
Osorio, D., & Vorobyev, M. (2005). Photoreceptor spectral sensitivities in terrestrial animals: Adaptations
for luminance and colour vision. Proceedings. Biological Sciences, 272(1574), 1745–1752. doi:10.1098/
rspb.2005.3156 PMID:16096084
Paavola, S. (2011). Diagrams, iconicity, and abductive discovery. Semiotica, 186(1/4), 297–314.
Paivio, A. (1986). Mental Representations: A Dual Coding Approach. London: Oxford University Press.
Perniss, P., & Vigliocco, G. (2014). The bridge of iconicity: From a world of experience to the experience of
language. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 369(1651).
doi:10.1098/rstb.2013.0300 PMID:25092668
Rajewksy, I. O. (2010). Border talks: the problematic status of media borders in the current debate about
intermediality. In L. Elleström (Ed.), Media Borders, Multimodality and Intermediality (pp. 51–68). Basingstoke:
Palgrave Macmillan.
Rajewsky, I. O. (2005). Intermediality, intertextuality, and remediation: A literary perspective on intermediality.
Intermédialités, 6, 43–64.
Rizzolatti, G., & Craighero, L. (2004). The mirror-neuron system. Annual Review of Neuroscience, 27(1),
169–192. doi:10.1146/annurev.neuro.27.070203.144230 PMID:15217330
Ryan, M.-L. (2004). Toward a transmedial narratology. In M.-L. Ryan (Ed.), Narrative across Media: The
Languages of Storytelling (pp. 337–359). Lincoln, NB: University of Nebraska Press.
Ryan, M.-L. (2005). On the Theoretical Foundations of Transmedial Narratology. In J. C. Meister, T. Kindt,
W. Schermus, & M. Stein (Eds.), Narrative Beyond Literary Criticism (pp. 1–23). Berlin: De Gruyter.
doi:10.1515/9783110201840.1
Saint-Martin, F. (1990). Semiotics of visual language. Bloomington, IN: Bloomington University Press. (originally
published in 1987)
Schmitz, U. (2016). Multimodale Texttypologie. In N.-M. Klug & H. Stöckl (Eds.), Handbuch Sprache im
multimodalen Kontext (pp. 327–347). Berlin: de Gruyter Mouton.
Short, T. L. (1982). Life among the legisigns. Transactions of the Charles S. Peirce Society, 18(4), 285–310.
Smith, M. (2012). Triangulating aesthetic experience. In A. P. Shimamura & S. E. Palmer (Eds.), Aesthetic
Science: Connecting Minds, Brains, and Experience (pp. 80–106). London: Oxford University Press.
22
Sobchack, V. (1992). The address of the eye: a phenomenology of film experience. Princeton, NJ: Princeton
University Press.
Sonesson, G. (2007). From the meaning of embodiment to the embodiment of meaning: A study in
phenomenological semiotics. In T. Ziemke, J. Zlatev, & R. M. Frank (Eds.), Body, Language, and Mind:
Embodiment (Vol. 1, pp. 85–128). Berlin: Mouton de Gruyter.
Sonesson, G. (2008). Prolegomena to a general theory of iconicity considerations on language, gesture, and
pictures. In K. Willems & L. De Cuypere (Eds.), Naturalness and Iconicity in Language (pp. 47–72). Philadelphia:
John Benjamins. doi:10.1075/ill.7.05son
Stjernfelt, F. (2000). Diagrams as Centerpiece of a Peircean Epistemology. Transactions of the Charles S. Peirce
Society, 36(3), 357–384.
Tan, E. (1996). Emotion and the Structure of Narrative Film. Film as an Emotion Machine. Mahwah, NJ:
Lawrence Erlbaum.
Thibault, P. J. (2006). Brain, mind and the signifying body: an ecosocial semiotic theory. London: Continuum.
Tseng, C. (2013). Cohesion in Film: Tracking Film Elements. Basingstoke: Palgrave Macmillan.
doi:10.1057/9781137290342
van Leeuwen, T. (2016). A social semiotic theory of synesthesia? – a discussion paper. Hermes – Journal of
Language and Communication in Business, 55, 105-119.
Vorderer, P., Wulff, H. J., & Friederichsen, M. (Eds.). (1996). Suspense: conceptualizations, theoretical analyses,
and empirical explorations. Mahwah, NJ: Lawrence Erlbaum Associates.
Wildfeuer, J. (2014). Film Discourse Interpretation. Towards a New Paradigm for Multimodal Film Analysis.
Routledge Studies in Multimodality. London: Routledge. doi:10.4324/978020376620
Withalm, G. (2006). Semiotics of media. In K. Brown (Ed.), Encyclopedia of Language and Linguistics (2nd
ed., Vol. 7, pp. 631–642). Oxford: Elsevier. doi:10.1016/B0-08-044854-2/01426-7
Wolf, W. (2007). Description as a transmedial mode of representation: general features and possibilities of
realization in painting, fiction and music. In W. Wolf & W. Bernhart (Eds.), Description in Literature and Other
Media, number 2 in Studies in Intermediality (pp. 1–90). Amsterdam: Rodopi.
Zlatev, J. (1997). Situated embodiment: studies in the emergence of spatial meaning. Stockholm: Gotab.
John Bateman received his PhD in Artificial Intelligence from the University of Edinburgh in 1986 and has worked
since then in Japan, California, Germany and the UK specializing in functional, computational and multimodal
linguistics. Since 1999, he has been a full Professor of Applied Linguistics in the English and Linguistics Departments
of the University of Bremen, researching and teaching in the areas of functional linguistic approaches to multilingual
and multimodal document design, multimodal semiotics, situated human-machine dialogue systems, natural
language generation and analysis, and formal ontology. He has published widely in all these areas, including
several introductory and survey articles on natural language generation, systemic-functional linguistics, and the
theory and practice of multimodality research.
23

Bateman 2019 End of Disembodied Semiotics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Bateman 2019 End of Disembodied Semiotics

Uploaded by

Copyright:

Available Formats

International Journal of Semiotics and Visual Rhetoric

Volume 3 • Issue 2 • July-December 2019

Transmediality and the End of

INTRODUCTION: THE CHALLENGE OF TRANSMEDIALITY

THE CHALLENGE OF TRANSMEDIALITY

• a material modality, described as the ‘latent corporeal interface’ of a medium

A MULTIMODAL PERSPECTIVE ON MEDIA AND MEDIALITY

In Bateman (2017), an alternative construction of ‘medium’ is offered anchored within multimodal

THE RELATION BETWEEN SEMIOTIC MODES AND PEIRCEAN SEMIOTICS

MATERIALITY AND THE SEMIOTIC BASIS OF TRANSMEDIALITY

Figure 4. Model of ‘squatting down’ from Hsieh and Luciani (2005)

CLEARING OUT THE SEMIOTIC BASEMENT

ONE FURTHER ANALYSIS

Figure 6. Discoursal/embodied material characterization of the Marnie sequence

CONCLUSION AND OUTLOOK

of description define a web of ‘textual’ interconnections constituting and driving interpretations,

You might also like